Attention — the core operation inside a Transformer

Yesterday we treated an LLM as a giant function:

Today, let's open the box. The central mechanism inside modern LLMs is attention.

The intuition is simple:

For each token, figure out which earlier tokens are relevant, then pull information from them.

Consider:

The animal didn't cross the street because it was too tired.

When processing it, the model needs information about animal, not street. Attention gives the model a learned mechanism for making that connection.

Tokens aren't words inside the model¶

Suppose our sequence is:

The animal was very tired

Each token is first converted into an embedding: perhaps a vector of 4,096 numbers in a large model.

So conceptually:

"The"     → [ 0.2, -0.7,  1.1, ...]
"animal"  → [-0.8,  0.4,  0.3, ...]
"was"     → [ 0.1,  0.9, -0.5, ...]
"very"    → [...]
"tired"   → [...]

Those numbers aren't manually assigned meanings. Training discovers representations that are useful for predicting tokens.

Now comes attention.

Query, Key and Value¶

For every token, the Transformer generates three new vectors:

where \(x\) is the token's current representation and \(W_Q,W_K,W_V\) are learned weight matrices.

The names are surprisingly useful.

Think of:

Query: What information am I looking for?

Key: What kind of information do I contain?

Value: What information should I provide if someone selects me?

Imagine the model is processing:

The animal didn't cross the street because it

The query generated by it might effectively be looking for something like:

"Which earlier token is the entity I refer to?"

The keys for earlier tokens might encode things resembling:

animal → entity / noun / singular
street → object / noun / singular
because → conjunction

The model compares the query for it with every preceding key.

The actual calculation is beautifully simple¶

The relevance score between two tokens is basically a dot product:

Higher dot product → stronger match.

For our simplified example, imagine:

                   attention score

The                  0.01
animal               0.72
didn't               0.02
cross                0.04
the                  0.01
street               0.12
because              0.08

A softmax converts these scores into probabilities that sum to 1.

The model then calculates:

So it gets a new representation containing information pulled primarily from animal.

That entire operation is usually written:

This equation is worth understanding because an astonishing amount of modern AI comes from repeatedly doing essentially this operation.

Why multiple attention heads?¶

A Transformer doesn't perform just one attention calculation.

It performs many in parallel: multi-head attention.

Different heads can learn different relationships.

For:

Alice gave Bob his book because he asked for it.

one head might learn something related to:

pronoun → person

another:

verb → subject

another:

object → owner

another might track something completely different that doesn't correspond neatly to a human linguistic concept.

If a model has, say, 64 attention heads, you can roughly imagine 64 different learned ways of asking:

What other tokens matter to me right now?

Their outputs are combined and passed onward.

Here's the important leap¶

Attention isn't simply retrieving words.

Each layer transforms the token representations.

Early in the network:

"Paris"
   ↓
token-ish representation

After several layers:

Paris
+ city
+ France
+ grammatical role
+ context

Later:

Paris
+ capital relationship
+ entity being asked about
+ likely answer to current question

These aren't literal fields inside the vector—the information is distributed across thousands of dimensions—but it's a useful mental model.

And Transformers stack many such layers.

tokens
   ↓
embedding
   ↓
attention + computation
   ↓
attention + computation
   ↓
attention + computation
   ↓
       ...
   ↓
attention + computation
   ↓
next-token probabilities

A large model might perform this process through dozens or even hundreds of layers.

The inference consequence that matters enormously¶

Suppose you've given the model 10,000 tokens of context and it is generating token 10,001.

The new token needs to attend to information from those previous tokens.

Naively, we'd repeatedly recompute the keys and values for all 10,000 old tokens.

But those don't change.

So inference systems store them.

This stored data is the KV cache:

token 1  → K₁, V₁
token 2  → K₂, V₂
token 3  → K₃, V₃
...
token 10000 → K₁₀₀₀₀, V₁₀₀₀₀

When generating the next token, the model only needs to compute its new query, key and value and compare its query against the cached keys.

This dramatically speeds up generation.

But there's a catch:

The KV cache consumes a lot of GPU memory, and it grows with context length.

That turns out to be one of the central engineering constraints of LLM serving.

So we now have our first bridge between model architecture and inference infrastructure:

Attention
   ↓
needs K and V for previous tokens
   ↓
KV cache
   ↓
GPU memory consumption
   ↓
limits batching / context length
   ↓
affects inference cost and throughput

That's why techniques such as multi-query attention (MQA) and grouped-query attention (GQA) matter: they reduce the number of distinct K/V heads that must be stored, shrinking the KV cache while retaining much of the benefit of many query heads.

The mental model to keep¶

Imagine every token entering a huge meeting.

It asks:

"Who in this room has information relevant to what I'm trying to understand?"

Its query describes what it needs.

Everyone else's keys advertise what information they might have.

Attention scores determine whom it listens to.

Their values contain the information it receives.

Then the token leaves the meeting with a richer representation of what it means in this particular context.

Stack that process dozens of times, across billions of learned parameters, and you have the computational heart of a modern LLM.