Mixture-of-Experts — huge capacity without huge compute per token

A dense Transformer uses the same MLP parameters for every token.

Mixture-of-Experts (MoE) replaces a dense MLP with many expert MLPs plus a router.

             ┌─ Expert 1
             ├─ Expert 2
token → router├─ Expert 3
             ├─ Expert 4
             └─ ...

The router selects only a small number of experts per token.

Tiny router example¶

Suppose the router outputs logits:

\[ [1.2,-0.3,2.1,0.7] \]

Softmax might give:

\[ [0.23,0.05,0.57,0.15] \]

A top-2 router selects experts 3 and 1.

Only those experts execute for the token.

Capacity vs active parameters¶

Suppose: - 64 experts - each expert = 1B parameters - top-2 routing

Stored expert parameters:

\[ 64B \]

Active expert parameters/token:

\[ 2B \]

So:

\[ \boxed{\text{total parameters}\neq\text{active parameters}} \]

This lets the model store much more capacity than it computes for each token.

Why experts can specialize¶

Training may cause different experts to become more useful for different representation patterns — code-like, mathematical, linguistic, etc.

But specialization is distributed and messy; expert 17 is not necessarily “the Python expert.”

Router training¶

The router is just another learned network. Gradients flow into its parameters based on the final language-model loss.

Expert collapse¶

If one expert becomes slightly better, the router may send it more tokens, causing it to train faster, making it even more attractive.

This positive feedback can collapse utilization.

MoE training therefore uses load-balancing objectives or constraints.

Systems cost¶

Experts are often distributed across GPUs.

A token may need to travel to another device to reach its selected expert, so MoE commonly requires all-to-all communication.

Thus:

\[ \boxed{\text{model architecture}\rightarrow\text{network architecture}} \]

again.

Mental model¶

A dense model is a company where all employees attend every question.

An MoE model employs thousands of specialists but dispatches only a few per question.

Where MoE usually lives¶

In a Transformer block, attention mixes information across token positions, while the MLP performs a large per-token nonlinear transformation.

MoE typically replaces the MLP:

attention
   ↓
router
 ┌─┼─┐
E1 E2 E3 ...
 └─┼─┘
   ↓
residual stream

This is attractive because MLPs contain a large fraction of model parameters.

Expert parallelism¶

If experts are spread across GPUs, the router may send token 1 to GPU A and token 2 to GPU F.

Systems therefore: 1. route tokens, 2. exchange them across the network, 3. batch tokens destined for the same expert, 4. run expert MLPs, 5. send results back.

The all-to-all exchange can become a major bottleneck.

Capacity factor and overflow¶

An expert cannot accept unlimited tokens in a fixed batch. Systems often allocate expected expert capacity plus some margin.

If routing is badly imbalanced, one expert may overflow while others sit idle.

So routing quality affects both model quality and hardware utilization.

Capacity vs compute¶

Dense scaling ties capacity and compute together.

MoE loosens that relationship:

\[ \text{stored knowledge capacity}\uparrow \]

without proportionally increasing:

\[ \text{compute per token} \]

That is its central attraction.

Router probabilities become part of the computation graph¶

The router itself computes something like:

\[ r=softmax(W_rx). \]

Top-k selection chooses experts according to \(r\).

The selected expert outputs are then weighted and combined.

Because routing affects the final language loss, gradients can teach the router which experts tend to be useful for which token representations.

The network bottleneck in expert parallelism¶

Imagine each GPU holds a subset of experts.

After routing, a GPU may discover that most of its local tokens belong to experts on other GPUs.

Those token activations must be transmitted.

After expert computation, outputs must travel back.

Hence one MoE block can require two major communication phases around the expert computation.

The expert MLP may be extremely fast while the all-to-all exchange is the limiting factor.

This is why MoE scaling depends heavily on interconnect quality and balanced routing—not just the number of available FLOPs.

MoE changes parameter-count comparisons¶

When someone says an MoE model has “500B parameters,” that number can be misleading if only 30B are active for a token.

For serving cost, active parameters and routing overhead matter heavily.

For memory capacity and potential specialization, total stored parameters matter.

So model comparisons should distinguish at least:

\[ \text{total parameters} \]

from:

\[ \text{active parameters/token}. \]

This is why a nominally larger MoE model can sometimes have inference compute closer to a much smaller dense model.