Over the last few lessons, we have learned to split an LLM across GPUs along several axes.
Data parallelism splits examples.
Tensor parallelism splits matrix operations.
Pipeline parallelism splits layers.
Sequence/context parallelism splits tokens.
Today we add a different kind of split:
\[
\boxed{\text{different tokens can use different parameters}}
\]
This is the key idea behind Mixture-of-Experts (MoE) models and expert parallelism.
The surprising promise of MoE is:
A model can contain vastly more parameters than it uses for any one token.
That lets us increase model capacity without increasing per-token computation proportionally.
But the price is a new systems problem: tokens must be routed across the cluster to whichever experts they need.