Saved in this browser. Includes supplemental lessons.
LLM Daily
How language models work
32 lessons on training, inference, and the systems behind LLMs. New lessons are added daily.
Explore the lessons
What an LLM is actually learning
At its core, an LLM is trained to do one deceptively simple thing:
Attention — the core operation inside a Transformer
Yesterday we treated an LLM as a giant function:
Prefill vs. Decode — why LLM inference has two very different phases
Last time we got to the KV cache: during generation, the model saves the keys and values for previous tokens so it doesn't have to recompute them.
Tensor shapes in a real-ish Transformer
Let’s make the dimensions realistic enough to understand GPU memory, but small enough to follow.
Prefill worked example — Q, K, V step by step
The key thing to see is that during prefill, we compute Q, K, and V for every prompt token at once, then form a whole attention matrix in one large parallel…
MHA → GQA → MQA — why modern LLMs share their KV cache
With ordinary Multi-Head Attention (MHA), each attention head gets its own Q, K, and V projections.
Why training needs so much more GPU memory than inference
During inference, the model performs a forward pass and generates outputs. During training it must later reverse that computation and determine how every…
How a giant model is split across GPUs
There are three conceptually different axes for splitting training work:
Backpropagation — how one wrong prediction changes billions of weights
Suppose the final hidden state is:
Batches and gradient accumulation — what is a training step, really?
A gradient from one sentence is a noisy opinion about how the model should change. A gradient averaged over millions of tokens is a much more reliable vote.
Adam — why the optimizer remembers the past
Plain SGD updates:
Learning-rate schedules — why training starts cautiously and ends gently
The gradient says which way to move; the learning rate says how far.
Scaling laws — if you have 10× more compute, what should grow?
A useful dense-Transformer pretraining approximation is:
Mixture-of-Experts — huge capacity without huge compute per token
A dense Transformer uses the same MLP parameters for every token.
Quantization — how 16-bit weights become 4-bit weights
Quantization stores model numbers with fewer bits.
Speculative decoding — getting several tokens from one expensive pass
Autoregressive generation normally looks like:
Prefix caching — why repeated prompts need not be prefilling from scratch
Suppose a coding assistant prompt contains:
Disaggregated inference — why reading and writing may belong on different GPUs
Prefill and decode have very different hardware profiles.
RoPE — how attention knows where tokens are
Attention compares content through:
FlashAttention — how 100K-token attention fits in memory
For one head with 100K tokens:
Can we escape \(O(T^2)\) attention?
Dense causal attention lets each token access every earlier token directly.
RAG vs. long context vs. learned memory
There are three fundamentally different ways for an LLM to access information:
Post-training — how a next-token predictor becomes an assistant
Pretraining teaches:
Test-time compute — making a model smarter without changing its weights
A model can improve in two ways: 1. change the model, 2. let the same model do more work on this question.
Verifiers and process rewards — when checking is easier than solving
Yesterday we looked at test-time compute: instead of asking a model for one answer, we can generate several attempts, branch, search, and spend more compute…
Distillation — moving expensive reasoning into cheaper weights
Over the last few lessons, we built a powerful inference-time recipe:
The synthetic-data flywheel — can models create their own training data?
Last time we saw distillation:
Credit assignment — how one reward trains 2,000 reasoning tokens
Yesterday we built this loop:
GRPO — learning by comparing several attempts at the same problem
Yesterday we got to the credit-assignment problem. An LLM generates a 2,000-token solution, receives:
Why reasoning RL is so expensive — one GRPO batch, end to end
Yesterday we saw GRPO conceptually:
Does RL *create* reasoning, or discover reasoning the model already had?
Yesterday we ended with a puzzle.
What is actually stored in an LLM's weights?
Yesterday we described pretraining as building a huge repertoire of latent knowledge and computational behaviors that RL can later select and compose.
Mechanistic interpretability — can we read what an LLM is thinking?
Yesterday we said an LLM's weights aren't a database. They're closer to a gigantic compiled program containing distributed facts, concepts, and learned…