Yesterday we explored tensor parallelism: multiple GPUs cooperate on the same Transformer layer by splitting its weight matrices. That can make a large model fit, but it introduces communication within almost every layer.
Today we'll examine another way to divide a model: pipeline parallelism (PP).
The intuition is:
Instead of making four GPUs work on one layer together, give each GPU a different stretch of layers—and keep them busy on different pieces of the batch.
This sounds like an assembly line. It is. The interesting question is how to prevent most of that assembly line from sitting idle.
Splitting layers is easy; keeping GPUs occupied is hard¶
Imagine a 32-layer Transformer divided across four GPUs:
GPU 0: layers 1–8
GPU 1: layers 9–16
GPU 2: layers 17–24
GPU 3: layers 25–32For one input, activations must pass through the stages in order:
input → GPU 0 → GPU 1 → GPU 2 → GPU 3 → outputUnlike tensor parallelism, each stage can execute its layers mostly locally. Communication happens at stage boundaries, where an activation tensor moves to the next GPU.
But with just one input, GPU 1 cannot start until GPU 0 has produced its output. GPU 2 waits for GPU 1, and so on. Simply distributing layers saves memory per GPU; it does not automatically make a single request four times faster.
The solution, especially during training, is to divide a large batch into microbatches and overlap their progress through the stages.
Microbatches turn the model into an assembly line¶
Suppose each stage takes one equal-length time slot to process a microbatch. With four stages and four microbatches, a forward-only schedule looks like this:
Time slot 1 2 3 4 5 6 7
GPU 0 A B C D · · ·
GPU 1 · A B C D · ·
GPU 2 · · A B C D ·
GPU 3 · · · A B C DThe dots are pipeline bubbles: idle slots while the pipeline fills or drains.
Once the pipeline is full, all four GPUs can work simultaneously—just on different microbatches. That is the critical difference from tensor parallelism, where GPUs work together on the same layer and often synchronize before proceeding.
A simple formula for pipeline efficiency¶
Let \(p\) be the number of pipeline stages and \(m\) the number of microbatches. Assume equal stage times, no communication overhead, and a forward-only pipeline.
The schedule takes
slots. Each stage does useful work for \(m\) of them, so ideal utilization is
and the bubble fraction is
This gives a useful result: more microbatches amortize pipeline startup and shutdown, while more stages create a longer pipeline to fill.
For \(p=4\) and \(m=8\):
About 27.3% of the ideal stage capacity is lost to bubbles. With \(m=32\), utilization rises to
That improvement does not mean arbitrarily tiny microbatches are best. Very small microbatches can make matrix multiplications inefficient and increase communication and scheduling overhead.
Worked example: throughput versus single-request latency¶
Suppose a model has four equally expensive stages, each taking 10 ms to process one microbatch.
A single microbatch needs four sequential stages, so its best-case latency is approximately
Now feed eight microbatches through the pipeline. The complete forward schedule takes
Running those eight microbatches through all four stages sequentially on one equally fast GPU would take approximately
So the idealized throughput improvement is
not \(4\times\), because of bubbles. Importantly, the first microbatch still takes about 40 ms to emerge. Pipeline parallelism mainly improves throughput across multiple microbatches, not the fundamental dependency chain of one token through the model.
These numbers assume perfectly balanced stages and ignore inter-GPU transfers. Real speedups are often lower.
Training adds a backward pass¶
Training is more complicated because every microbatch must travel forward to compute its loss and then backward to compute gradients.
One simple schedule, often called GPipe-style, performs the forward passes for all microbatches and then their backward passes. With equal forward and backward stage times, its ideal bubble fraction still scales approximately as
But GPipe-style scheduling may need to retain many microbatches' forward activations until their backward passes begin. That can consume substantial GPU memory.
A common alternative is 1F1B scheduling: after a warm-up period, each stage alternates one forward microbatch and one backward microbatch. This does not eliminate the fundamental pipeline bubbles, but it can substantially reduce the number of outstanding activations compared with an all-forward-then-all-backward schedule.
Microbatch gradients are accumulated into the gradient for the overall batch; the optimizer normally updates weights after the required microbatches finish. Appropriate loss scaling is needed so changing the microbatch count does not inadvertently change the effective gradient magnitude.
The communication and memory tradeoff¶
A pipeline boundary transfers activations, not the full model weights. For a BF16 activation tensor containing \(b\) sequences of \(s\) tokens with hidden width \(d\), the payload is approximately
For \(b=4\), \(s=2048\), and \(d=8192\):
That is one forward boundary tensor; training also involves backward gradient communication, and actual implementations may transmit additional tensors or use different representations.
Compared with tensor parallelism, pipeline parallelism typically communicates less frequently, but stage imbalance is dangerous. If one stage takes 15 ms while the others take 10 ms, the entire pipeline eventually runs at the slowest stage's pace. Splitting layers evenly is therefore not always the same as splitting compute evenly.
Why inference is different¶
Pipeline parallelism can also serve inference requests. Multiple requests or microbatches can occupy different stages simultaneously, helping a model too large for one GPU fit across several devices.
But autoregressive decode introduces a dependency: a request's next token cannot enter stage 0 until its previous token has passed through the final stage and been selected. Without enough independent requests, the pipeline may remain underfilled.
This is why tensor parallelism is often attractive for low-latency single-request inference on tightly connected GPUs, while pipeline parallelism can be appealing when model size or high concurrency makes layer-wise partitioning worthwhile. Production systems may combine both techniques, plus data parallelism, into a three-dimensional layout.
Mental model: an automobile assembly line¶
A car passes through body assembly, painting, interior installation, and inspection. Four stations cannot finish one car in one-quarter the time: that car must visit every station.
But once several cars are in flight, each station can work on a different car. The factory's output rate rises even though an individual car's journey is not dramatically shorter.
Microbatches are the cars. Pipeline stages are the stations. Bubbles are empty stations during startup and shutdown. Stage imbalance is the slow paint booth that holds up the entire factory.
The deeper lesson¶
Yesterday's tensor parallelism divided the work inside a layer. Today's pipeline parallelism divides the sequence of layers.
The central design question is not simply how many GPUs can hold the model. It is whether there is enough independent work to keep every stage occupied, without making microbatches too small or communication too expensive.
That is the same systems principle we have encountered throughout inference optimization: distributing work creates capacity, but scheduling determines how much of that capacity becomes useful throughput.