GPU topology — why the wires between accelerators shape LLM parallelism

Yesterday we built a Transformer pipeline across multiple GPUs.

Before that, we saw tensor parallelism, where several GPUs cooperate inside the same layer, and data parallelism, where replicas periodically synchronize gradients.

Those strategies sound like abstract software choices:

TP = 8
PP = 4
DP = 16

But on a real training cluster, the GPUs are not all equally connected.

Some may exchange data through extremely fast links inside one server. Others may communicate only through a slower network between servers.

That means a distributed LLM has a second architecture hiding underneath the Transformer:

The central idea today is simple:

Put the communication that happens most often on the fastest links.

That sounds obvious. At thousands of GPUs, it becomes one of the main determinants of whether a parallelization strategy works at all.

Not every GPU-to-GPU path is equal¶

Imagine eight GPUs in one server connected by a high-bandwidth accelerator fabric.

Another server contains eight more GPUs.

Conceptually:

Server A                         Server B

GPU0 ─┬─ GPU1                   GPU8 ─┬─ GPU9
GPU2 ─┼─ GPU3                   GPU10─┼─ GPU11
GPU4 ─┼─ GPU5                   GPU12─┼─ GPU13
GPU6 ─┴─ GPU7                   GPU14─┴─ GPU15
       │                               │
       └────── cluster network ────────┘

Communication inside a server may have much higher bandwidth and lower latency than communication between servers.

So although the cluster contains 16 GPUs, it is misleading to think of it as a perfectly uniform pool:

may be much cheaper than:

This matters because different forms of model parallelism communicate at very different frequencies.

Tensor parallelism is especially communication-hungry¶

Recall the idea of tensor parallelism.

Instead of putting an entire matrix multiplication on one GPU, we divide it across several GPUs.

For a simplified linear layer:

we might partition \(W\) across four GPUs.

Each GPU computes part of the result, but the partial results often need to be combined through collective operations such as:

all-reduce
all-gather
reduce-scatter

And this can happen repeatedly inside every Transformer layer.

If a model has 80 layers, a tensor-parallel group may communicate hundreds of times during a single forward/backward pass.

So tensor parallelism usually wants:

between its members.

That strongly suggests keeping a tensor-parallel group inside the fastest local topology when possible.

For an eight-GPU server, a natural configuration might therefore be:

All eight GPUs cooperate tightly, but their frequent communication remains inside the server.

Pipeline parallelism communicates differently¶

Pipeline parallelism has a different pattern.

A stage performs several complete layers and then transfers an activation tensor to the next stage:

stage 0
[layers 1–20]
      │
      │ activation
      ▼
stage 1
[layers 21–40]

The transfer can be large, but it occurs only at stage boundaries rather than repeatedly inside every layer.

That means pipeline stages can often tolerate a slower connection better than tensor-parallel peers can.

So suppose each server has eight GPUs.

We could map:

Server A = pipeline stage 0, TP=8
Server B = pipeline stage 1, TP=8
Server C = pipeline stage 2, TP=8
Server D = pipeline stage 3, TP=8

Now the frequent tensor-parallel collectives remain local:

inside Server A
inside Server B
inside Server C
inside Server D

Only pipeline activations cross the inter-server network.

This is an example of topology-aware parallelism.

A simple communication-time model¶

Suppose we need to transfer \(S\) bytes across a link.

A useful first approximation is:

where:

• \(L\) is link latency,
• \(B\) is effective bandwidth,
• \(S\) is message size.

This equation explains why both latency and bandwidth matter.

For a tiny message:

may be small, so latency dominates.

For a huge tensor, transfer time is mostly:

Suppose a 256 MiB activation must cross a link delivering an effective:

Ignoring latency:

If the next pipeline stage performs 100 ms of computation, 5 ms may be manageable.

But imagine a tensor-parallel collective that adds even 1 ms of exposed communication and occurs 160 times:

Communication that looks tiny in isolation becomes enormous when it sits inside the model's inner loop.

Frequency matters as much as message size.

Communication can sometimes hide behind computation¶

The previous calculation is deliberately pessimistic.

Modern distributed systems try to overlap communication with computation.

Instead of:

compute
   ↓
wait for communication
   ↓
compute

we would prefer:

compute A ────────────────┐
                          │
communication X ──────────┼── simultaneous
                          │
compute B ────────────────┘

If 5 ms of communication occurs entirely while the GPU is doing 20 ms of unrelated useful work, the 5 ms may add almost nothing to wall-clock time.

This leads to an important distinction:

The performance problem is the portion that cannot be hidden.

But dependencies limit overlap. If the next matrix multiplication requires the result of an all-reduce, the GPU cannot simply continue indefinitely.

Eventually the critical path reaches the communication barrier.

All-reduce is not one magical operation¶

When we say:

all-reduce gradients

it can sound as though every GPU sends its entire tensor to every other GPU simultaneously.

Real collective libraries use structured algorithms.

One classic example is a ring all-reduce.

With four GPUs:

GPU0 → GPU1 → GPU2 → GPU3
 ↑                       ↓
 └───────────────────────┘

The tensor is divided into chunks. GPUs circulate chunks around the ring while reducing them, then circulate the completed chunks so every GPU receives the final result.

For a large message of size \(S\) over \(N\) GPUs, a ring all-reduce moves roughly:

bytes through each participant's links.

As \(N\) becomes large:

That is surprisingly efficient in bandwidth terms.

But the operation also requires multiple sequential communication steps, so latency and topology still matter.

Other collective algorithms use trees, hierarchical structures, or topology-specific paths.

The communication library is effectively solving a routing problem underneath the model.

Hierarchical communication exploits the hardware hierarchy¶

Suppose we have 32 GPUs arranged as four eight-GPU servers.

A naive global collective might treat all 32 GPUs symmetrically.

But the physical system is hierarchical:

fast:
GPU ↔ GPU inside server

slower:
server ↔ server

A hierarchical collective can exploit this.

For example:

1. reduce within each 8-GPU server
2. exchange reduced data between servers
3. distribute the result within each server

Conceptually:

8 GPUs          8 GPUs          8 GPUs          8 GPUs
   │               │               │               │
 local             local           local           local
 reduce            reduce          reduce          reduce
   │               │               │               │
   └──────── inter-server collective ──────────────┘

This minimizes expensive cross-server traffic.

The same principle appears throughout distributed computing:

Now combine TP, PP, and DP¶

Suppose we have 128 GPUs.

Each server contains 8 GPUs, giving:

Imagine we choose:

The total is:

A topology-aware mapping might look like:

Data-parallel replica 0

Server 0: 8 GPUs = TP group = pipeline stage 0
Server 1: 8 GPUs = TP group = pipeline stage 1
Server 2: 8 GPUs = TP group = pipeline stage 2
Server 3: 8 GPUs = TP group = pipeline stage 3


Data-parallel replica 1

Server 4: 8 GPUs = TP group = pipeline stage 0
...

Why is this attractive?

The highest-frequency communication — tensor parallelism — stays inside each server.

Pipeline communication crosses servers, but only at stage boundaries.

Data-parallel synchronization spans replicas and therefore may cross many servers, but it occurs less frequently than tensor-parallel collectives and can often be bucketed or overlapped with backward computation.

The software parallelism hierarchy has been aligned with the hardware communication hierarchy.

The wrong mapping can destroy performance¶

Now imagine we accidentally construct each eight-way tensor-parallel group using one GPU from eight different servers:

TP group:

Server 0 → GPU0
Server 1 → GPU0
Server 2 → GPU0
Server 3 → GPU0
Server 4 → GPU0
Server 5 → GPU0
Server 6 → GPU0
Server 7 → GPU0

Every tensor-parallel collective now crosses the cluster network.

Meanwhile, GPUs sitting next to each other on the fastest local fabric may barely communicate.

The mathematical model is unchanged.

The parameter count is unchanged.

The FLOP count is almost unchanged.

Yet training throughput can collapse.

This is why statements such as:

We trained on 1,024 GPUs.

are incomplete descriptions of a training system.

We also need to know:

What kind of GPUs?
How are they grouped into servers?
How are servers connected?
What parallelism dimensions map to which links?
What collective algorithms are used?
How much communication is exposed?

Inference has topology problems too¶

This is not only a training issue.

Suppose a huge model requires eight-way tensor parallelism during inference.

Each generated token may require synchronization across those eight GPUs.

Decode is sequential:

so communication latency can sit directly on the critical path for every token.

If one token requires:

then throughput for that sequence is roughly determined by:

That corresponds to about:

If faster local interconnect reduces exposed communication to 0.5 ms:

giving:

A few milliseconds matter because the cost repeats for every generated token.

This connects directly to our earlier lesson on decode: sequential generation makes small per-token delays unusually important.

A concrete mental model: roads between factories¶

Imagine four factories.

Inside each factory, parts move on high-speed conveyor belts.

Between factories, they travel by truck.

You need to organize three kinds of work:

A: exchange parts 100 times per product
B: exchange parts 4 times per product
C: exchange one large shipment after many products

A sensible layout puts workers doing task A inside the same factory.

Task B can cross factories occasionally.

Task C can tolerate the truck network even better.

A terrible layout puts every tightly cooperating worker in a different factory and forces trucks to carry parts back and forth hundreds of times.

Nothing about the product design changed.

The factory layout changed the throughput.

For distributed LLMs:

conveyor belts ≈ fast local GPU fabric
trucks          ≈ inter-node network

task A          ≈ tensor parallelism
task B          ≈ pipeline parallelism
task C          ≈ less-frequent global synchronization

That is the essential topology-aware mental model.

The deeper lesson¶

We began this series thinking primarily about the Transformer:

At large scale, that is only half the system.

The other half is a communication graph:

Parallelism creates communication edges, and the cluster supplies physical links on which those edges must run.

Efficient distributed training and inference require matching the two.

So today's main principle is:

This also explains why scaling an LLM from 8 GPUs to 8,000 GPUs is not simply the same computation on a larger machine. As the system expands, communication becomes increasingly hierarchical, failures become normal rather than exceptional, and keeping thousands of accelerators doing useful synchronized work becomes a systems problem of its own.