The CPU, Instruction Cycle, and Pipeline

A CPU may operate billions of times per second, but that does not mean it finishes an entire program in one clock cycle. A program is divided into instructions, and each instruction is divided into several steps. To reason about performance, distinguish the time taken by one instruction from the number of instructions completed in a given interval.

From an instruction to a result

A typical teaching CPU processes an instruction through fetch, decode and register read, execution, memory access when needed, and register write-back. Fetch reads the instruction at the address held in the program counter. Decode determines which operation and operands to use.

For example, an addition instruction reads two register values, sends them to the ALU, and writes the result to a destination register. A load calculates an address in the ALU before reading memory. For a branch, the condition affects the address of the next instruction. Different instructions use different parts of the processor.

Overlapping work reduces the interval between completions

Consider five stages, each taking 1 ns. Processing each instruction from start to finish before starting another would take 500 ns for 100 instructions. With registers between the stages, different instructions can occupy different stages at the same time. Under ideal conditions, the first instruction finishes after 5 ns, and another finishes every 1 ns thereafter. The total for 100 instructions is 104 ns.

Ideal 5-stage pipeline, 1 ns per stage
Time for N instructions = (5 + N - 1) × 1 ns
100 instructions = 104 ns

Latency of 1 instruction is about 5 ns
Completion interval after the pipeline fills is about 1 ns

This calculation assumes equal stage times, no stalls or branch recovery, and no overhead from pipeline registers. In practice, the slowest stage and the overhead between stages constrain the clock period. Adding more stages does not automatically multiply performance by the number of stages.

When the previous result is not ready

add t0, a0, a1
sub t1, t0, a2

The second instruction needs t0, which the first instruction produces. Reading the old t0 before write-back would give the wrong answer. A data dependency becomes a data hazard when it conflicts with the timing of instruction processing. If the result is already available from an execution stage, forwarding can pass it directly to a later operation without waiting for register write-back.

When an instruction uses a value immediately after a load, that value may not arrive until the end of the memory-access stage. A teaching five-stage pipeline may need a one-cycle stall even with forwarding. The exact number of stall cycles depends on the hardware and on how memory responds.

Branches and competition for resources

With a conditional branch, the processor may not know the next fetch address until the condition is evaluated. It can wait, or it can proceed using a branch prediction. If the prediction is wrong, it discards work on the incorrect path and restarts on the correct one. The mechanisms that commit execution results preserve the behavior visible to the program.

A structural hazard occurs when instructions compete for a limited resource, such as a single memory port needed both for instruction fetch and for data access. Hardware organization can avoid some of these conflicts, for example by providing separate instruction and data ports. A hazard is not a programming mistake; it is a situation that requires coordination when operations overlap.

Breaking CPU time into factors

CPU time can be expressed as instruction count × average CPI × clock period. CPI is the average number of cycles per instruction. Even with the same instruction count, more cache misses or branch recoveries can increase CPI. Comparing clock frequency alone misses these delays.

Modern CPUs may issue multiple instructions at once and execute instructions out of order. The five-stage model is not a diagram of every real CPU. It is a starting point for explaining stalls in terms of dependencies, resources, and branches.

Check your understanding

Question 1: Does pipelining necessarily make each individual instruction five times faster? No. Its main benefit is greater throughput when a sequence of instructions flows through the processor.

Question 2: What is the CPU time for one million instructions, an average CPI of 2, and a clock period of 0.5 ns? It is 1,000,000 × 2 × 0.5 ns = 1 ms. Reducing stalls to achieve a CPI of 1.5 would reduce the time to 0.75 ms without changing the clock period.

To see why memory stalls occur, continue to Caches and the Memory Hierarchy.