Saved in this browser. Includes supplemental lessons.
LLM Daily
How language models work
75 lessons on training, inference, and the systems behind LLMs. New lessons are added daily.
Explore the lessons
What an LLM is actually learning
At its core, an LLM is trained to do one deceptively simple thing:
Attention — the core operation inside a Transformer
Yesterday we treated an LLM as a giant function:
Prefill vs. Decode — why LLM inference has two very different phases
Last time we got to the KV cache: during generation, the model saves the keys and values for previous tokens so it doesn't have to recompute them.
Tensor shapes in a real-ish Transformer
Let’s make the dimensions realistic enough to understand GPU memory, but small enough to follow.
Prefill worked example — Q, K, V step by step
The key thing to see is that during prefill, we compute Q, K, and V for every prompt token at once, then form a whole attention matrix in one large parallel…
MHA → GQA → MQA — why modern LLMs share their KV cache
With ordinary Multi-Head Attention (MHA), each attention head gets its own Q, K, and V projections.
Why training needs so much more GPU memory than inference
During inference, the model performs a forward pass and generates outputs. During training it must later reverse that computation and determine how every…
How a giant model is split across GPUs
There are three conceptually different axes for splitting training work:
Backpropagation — how one wrong prediction changes billions of weights
Suppose the final hidden state is:
Batches and gradient accumulation — what is a training step, really?
A gradient from one sentence is a noisy opinion about how the model should change. A gradient averaged over millions of tokens is a much more reliable vote.
Adam — why the optimizer remembers the past
Plain SGD updates:
Learning-rate schedules — why training starts cautiously and ends gently
The gradient says which way to move; the learning rate says how far.
Scaling laws — if you have 10× more compute, what should grow?
A useful dense-Transformer pretraining approximation is:
Mixture-of-Experts — huge capacity without huge compute per token
A dense Transformer uses the same MLP parameters for every token.
Quantization — how 16-bit weights become 4-bit weights
Quantization stores model numbers with fewer bits.
Speculative decoding — getting several tokens from one expensive pass
Autoregressive generation normally looks like:
Prefix caching — why repeated prompts need not be prefilling from scratch
Suppose a coding assistant prompt contains:
Disaggregated inference — why reading and writing may belong on different GPUs
Prefill and decode have very different hardware profiles.
RoPE — how attention knows where tokens are
Attention compares content through:
FlashAttention — how 100K-token attention fits in memory
For one head with 100K tokens:
Can we escape \(O(T^2)\) attention?
Dense causal attention lets each token access every earlier token directly.
RAG vs. long context vs. learned memory
There are three fundamentally different ways for an LLM to access information:
Post-training — how a next-token predictor becomes an assistant
Pretraining teaches:
Test-time compute — making a model smarter without changing its weights
A model can improve in two ways: 1. change the model, 2. let the same model do more work on this question.
Verifiers and process rewards — when checking is easier than solving
Yesterday we looked at test-time compute: instead of asking a model for one answer, we can generate several attempts, branch, search, and spend more compute…
Distillation — moving expensive reasoning into cheaper weights
Over the last few lessons, we built a powerful inference-time recipe:
The synthetic-data flywheel — can models create their own training data?
Last time we saw distillation:
Credit assignment — how one reward trains 2,000 reasoning tokens
Yesterday we built this loop:
GRPO — learning by comparing several attempts at the same problem
Yesterday we got to the credit-assignment problem. An LLM generates a 2,000-token solution, receives:
Why reasoning RL is so expensive — one GRPO batch, end to end
Yesterday we saw GRPO conceptually:
Does RL *create* reasoning, or discover reasoning the model already had?
Yesterday we ended with a puzzle.
What is actually stored in an LLM's weights?
Yesterday we described pretraining as building a huge repertoire of latent knowledge and computational behaviors that RL can later select and compose.
Mechanistic interpretability — can we read what an LLM is thinking?
Yesterday we said an LLM's weights aren't a database. They're closer to a gigantic compiled program containing distributed facts, concepts, and learned…
The residual stream — the Transformer's shared scratchpad
Yesterday we looked at mechanistic interpretability: trying to understand the internal features and circuits that make a Transformer produce an answer.
Transformer MLPs as learned key-value memories
Yesterday we built a useful picture of the residual stream as the Transformer's shared scratchpad. Attention heads read from that scratchpad, move…
Why LLM decoding is often memory-bandwidth bound
We have now built up three pieces of the Transformer’s internal computation:
Continuous batching — why an LLM server behaves like an operating system
Yesterday we saw why autoregressive decoding is often memory-bandwidth bound.
Why training needs an optimizer — from gradients to AdamW
In earlier lessons, we followed the forward pass through a Transformer and looked closely at inference: KV caches, batching, memory bandwidth, and scheduling.
Gradient accumulation — how a huge training batch fits on finite GPUs
Yesterday we saw what happens after backpropagation: AdamW turns gradients into parameter updates, while maintaining substantial optimizer state.
Activation checkpointing — trading extra compute for much less training memory
Yesterday we saw gradient accumulation separate the physical microbatch from the effective global batch.
Data parallelism — why copying the whole model can still make training faster
Over the last few lessons, we have built up the memory side of LLM training:
Fully sharded data parallelism — stop copying the whole training state
Yesterday we saw ordinary data parallelism:
Tensor parallelism — splitting one matrix multiplication across GPUs
Yesterday we saw fully sharded data parallelism (FSDP).
Pipeline parallelism — splitting Transformer layers across GPUs
Over the last three lessons, we have split LLM training in three different ways.
Sequence parallelism — splitting the token dimension across GPUs
Over the last few lessons, we have learned to cut a Transformer along several axes.
Expert parallelism — how Mixture-of-Experts models spend more parameters without spending all the FLOPs
Over the last few lessons, we have learned to split an LLM across GPUs along several axes.
Prefill vs. decode — why LLM inference is really two different workloads
Over the last several lessons, we have learned how large models are spread across GPUs: tensor parallelism, pipeline parallelism, context parallelism, and…
Speculative decoding — generating several tokens while paying for one verification pass
Over the last few lessons, we have learned to split an LLM across GPUs along several axes.
Quantization — why fewer bits can make LLM inference dramatically faster
Over the last several lessons, we have been building up the systems view of LLM inference. Yesterday’s speculative decoding lesson showed how we can get…
PagedAttention — why serving LLMs needs a virtual-memory system for the KV cache
Over the last few lessons, we have focused on inference.
Prefix caching — when repeated prompts can skip most of prefill
Over the last few lessons, we have built a fairly complete picture of LLM inference.
Continuous batching — why an LLM server should not wait for a batch to finish
Over the last several lessons, we have built the pieces of a modern inference server.
Disaggregated inference — why prefill and decode may belong on different GPUs
Over the last several lessons, we have treated LLM inference as two different workloads hiding behind one API call.
Activation checkpointing — why training can save memory by deliberately recomputing work
Recent lessons have focused heavily on inference: KV caches, quantization, PagedAttention, continuous batching, and the separation of prefill from decode.
ZeRO — how training a huge model stops replicating everything on every GPU
Yesterday we saw that training memory is not just the model weights. A training GPU may also hold gradients, optimizer state, and activations. Activation…
Tensor parallelism — how several GPUs cooperate on one Transformer layer
Yesterday we saw how ZeRO lets a training job shard persistent model state instead of storing a complete copy on every data-parallel worker.
Pipeline parallelism — turning a deep Transformer into an assembly line
Yesterday we looked at tensor parallelism: multiple GPUs cooperate on the matrix multiplications inside the same Transformer layer.
GPU topology — why the wires between accelerators shape LLM parallelism
Yesterday we built a Transformer pipeline across multiple GPUs.
Sequence parallelism — splitting the token dimension across GPUs
Over the last few lessons, we have been slicing large LLM computations in several directions.
Mixture-of-Experts — making a model wider without using all of it for every token
Over the last few lessons, we have focused on how to distribute dense Transformer computation across many GPUs.
Expert routing in practice — why MoE needs a traffic controller
Yesterday we introduced Mixture-of-Experts (MoE): instead of sending every token through one dense MLP, a router chooses a small number of expert MLPs for…
Speculative decoding — using a small model to make a large model faster
Over the last several lessons, we have focused on distributing work: across GPUs, across sequence positions, and across experts.
Disaggregated prefill and decode — why one LLM request needs two kinds of GPU work
Yesterday we explored speculative decoding, which tries to make the sequential part of inference faster by verifying several candidate tokens together.
Prefix caching — when an LLM can skip reading the same prompt twice
Yesterday we explored prefill–decode disaggregation: separate GPU pools can specialize in processing prompts and generating tokens, but the handoff requires…
PagedAttention — virtual memory for the KV cache
Yesterday we explored prefix caching: when two requests share an identical prompt prefix, an inference engine can reuse the corresponding key/value (KV)…
KV-cache quantization — making long-context inference fit in memory
Yesterday we explored PagedAttention: the inference runtime stores a request's KV cache in reusable blocks instead of reserving one large contiguous region.…
Continuous batching and chunked prefill — why the scheduler matters
Yesterday we looked at KV-cache quantization: reducing the bytes occupied by each request's attention history. Together with PagedAttention and prefix…
Disaggregated prefill and decode — why one GPU pool may not fit both jobs
Yesterday we explored speculative decoding: a cheap draft proposes several tokens, and an expensive target verifies them together. That optimization…
The roofline model — why LLM inference runs out of memory before math
Yesterday we explored disaggregated prefill and decode: separate GPU pools can serve prompt processing and token generation because the two phases stress…
Tensor parallelism — when faster math requires more communication
Yesterday we used the roofline model to explain why LLM inference can be limited by memory traffic rather than arithmetic. We saw that batching improves…
Pipeline parallelism — keeping multiple GPUs busy across layers
Yesterday we explored tensor parallelism: multiple GPUs cooperate on the same Transformer layer by splitting its weight matrices. That can make a large…
FSDP and ZeRO — training a model without replicating every optimizer byte
Yesterday we explored pipeline parallelism: dividing a Transformer into consecutive stages so multiple GPUs can work on different microbatches. Before that,…
Context parallelism — splitting the sequence across GPUs
Yesterday we explored activation checkpointing: saving GPU memory by recomputing intermediate activations during backward. Before that, FSDP and ZeRO showed…
Mixture of Experts — scaling model capacity without activating every parameter
Yesterday we explored context parallelism: splitting a long sequence across GPUs while preserving attention between its tokens. Before that, tensor and…
Training the MoE router — specialization without expert collapse
Yesterday we explored Mixture of Experts (MoE): a Transformer can contain many feed-forward experts while activating only a few for each token. This…
MoE token dispatch — when sparse computation becomes a network problem
Yesterday we explored training the Mixture-of-Experts router: the router must choose useful experts without overloading a few and starving the rest. Day 75…