What is actually stored in an LLM's weights?

Yesterday we described pretraining as building a huge repertoire of latent knowledge and computational behaviors that RL can later select and compose.

That raises a deceptively difficult question:

When an LLM “knows” that Paris is the capital of France, where exactly is that fact stored?

The tempting mental model is a gigantic database:

France → Paris
Einstein → relativity
water → H₂O

But that is mostly the wrong picture.

A better intuition is:

They encode transformations that cause useful information to emerge when the right input passes through them.

Let's make that concrete.

Start with one tiny neuron¶

Suppose a hidden representation is:

and a neuron has weights:

It computes:

So:

Then an activation function transforms that value.

Already notice something important.

What does:

“mean”?

Nothing obvious.

You can't look at it and say:

Ah yes, this is where the model stores France.

Its meaning exists only in combination with:

• the representation entering the neuron,
• thousands of other neurons,
• earlier layers,
• later layers,
• attention,
• the output projection.

A parameter is useful because of the computation it participates in.

Now scale this to a Transformer¶

Suppose the prompt is:

The capital of France is

At the bottom of the network we begin with token representations.

After layer 1:

might encode some mixture of:

this is English
this looks factual
"France" is a country
"capital" implies a geographic relation

After more layers, attention lets the representation of the current position interact with representations of:

capital
France
is

MLPs transform those representations.

Eventually the final hidden state:

is projected into vocabulary logits:

Perhaps:

Paris        13.8
Lyon          8.1
Marseille     7.3
London        5.2
banana       -2.7

Softmax converts these into probabilities.

The model “knows” Paris because the entire learned computation maps this context to a hidden representation that aligns strongly with the output direction corresponding to Paris.

There isn't necessarily a single:

FRANCE_CAPITAL_PARAMETER = PARIS

anywhere.

How did the fact get there?¶

Imagine pretraining encounters many contexts:

Paris is the capital and largest city of France.

France's capital, Paris, lies on the Seine.

The French government is headquartered in Paris.

Q: What is the capital of France? A: Paris.

Every occurrence produces gradients.

For one occurrence, perhaps the model predicts:

Paris      0.20
Lyon       0.15
London     0.10

Target:

Paris

Cross-entropy produces gradients that slightly modify many parameters:

After many related examples:

Paris      0.93
Lyon       0.02
London     0.004

The fact emerges from the cumulative effect of enormous numbers of tiny updates.

So knowledge is often distributed.

But here's the surprising part: models also memorize¶

Suppose the training corpus contains a strange unique string:

The access code for Project Marmalade is Q7ZP-491X.

If the model sees this sufficiently often, it may later reproduce:

Q7ZP-491X

That's much closer to memorization.

So two things coexist:

and:

Consider these examples:

Example A¶

The capital of Italy is ___

The model can exploit a learned concept like:

Example B¶

The SHA-256 hash beginning a34f... continues ___

There may be no useful rule.

If the model can reproduce it, that's much more likely to depend on memorized training information.

Modern LLMs do both.

The interesting question is not:

Do they memorize?

They clearly can.

It's:

How much of their behavior comes from memorization versus learned structure that generalizes?

And that varies enormously by task.

Compression gives us a useful clue¶

Imagine a model has:

parameters stored in BF16.

That's roughly:

Now imagine its pretraining corpus contained:

tokens.

Even at just a couple of bytes per token, the raw training data is tens of terabytes.

So the model cannot simply contain a lossless copy of its entire training corpus.

Something much more interesting has happened.

Training has compressed regularities from an enormous dataset into a much smaller set of parameters.

For example, instead of storing separately:

dogs can run
cats can run
horses can run
children can run
athletes can run

the model can learn abstract representations around:

animate entities
movement
capabilities

and combine them.

This is why generalization is possible.

Compression forces the model to discover reusable structure.

A useful analogy: JPEG versus a program¶

But even “compression” is slightly misleading.

JPEG compresses an image so you can reconstruct approximately the same image.

An LLM does something more like compiling a gigantic dataset into a predictive program.

Given:

17 × 24 =

it doesn't need to have seen that exact string.

Its learned transformations can produce:

That's algorithm-like behavior.

So perhaps:

is better than:

Here's a beautiful example: addition¶

Suppose training includes many examples:

12 + 7 = 19
31 + 8 = 39
14 + 25 = 39
...

Now ask:

Perhaps that exact equation never appeared in training.

Yet the model can answer:

If so, the weights must encode some reusable computation approximating addition—not merely a lookup table containing every possible equation.

That learned algorithm may be imperfect, especially as numbers become longer.

But this illustrates what weights can contain:

which connects directly to yesterday's “latent cognitive subroutines.”

Why retrieval from weights differs from a database¶

Suppose a database contains:

employee_id: 48271
salary: $184,500

A query can return:

exactly.

The database has:

• explicit records,
• addresses,
• exact retrieval,
• easy updates.

An LLM has none of those guarantees.

Its knowledge is distributed through floating-point parameters optimized for predicting tokens.

So asking:

What was employee 48271's salary?

may produce:

$184,500
$185,000
$180,000
"I don't know"

depending on context.

That's why we described RAG as a different memory tier.

If information must be:

a database or retrieved document is usually a much better representation than model weights.

Updating one fact reveals the problem¶

Suppose a model knows:

CEO of Acme = Alice.

Tomorrow:

CEO of Acme = Bob.

In a database:

UPDATE company
SET ceo='Bob'
WHERE company='Acme';

Done.

But inside an LLM, the Alice relationship may be distributed across many parameters and connected to other representations.

Changing those parameters could affect:

Acme
Alice
Bob
CEO relationships
other companies
language behavior

This is related to catastrophic forgetting and the difficulty of model editing.

Weights aren't neat rows in a table.

They're an entangled computational substrate.

Yet individual components can specialize¶

“Distributed” doesn't mean every fact is uniformly spread over every parameter.

Research probing Transformers often finds:

• attention heads with recognizable behavioral patterns,
• MLP neurons/features associated with concepts,
• layers specializing in different kinds of transformations.

But these aren't usually clean one-to-one mappings.

A better picture is:

              feature A
             /         \
parameters ── feature B ──→ behavior
             \         /
              feature C

Many parameters contribute to features.

Many features contribute to behaviors.

This is called superposition: neural networks can encode more useful features than they have simple dedicated dimensions by overlapping representations.

That makes the network extremely efficient—and difficult to interpret.

Now connect this back to distillation¶

Remember:

Initially, the strategy:

Rewrite \(25\) as \(100/4\).

exists as tokens in a reasoning trajectory.

After training on enough examples, gradient descent modifies weights so that contexts involving multiplication by 25 naturally evoke that transformation.

What was once:

becomes:

That's what we meant by compiling inference into the model.

The model hasn't stored the training transcript like a file.

Training has altered the computational landscape so that useful behavior becomes easier to produce.

Today's mental model¶

Think about an experienced chess player.

Ask:

Where is your knowledge that knights are often strong on outposts stored?

They can't point to neuron 7,481 in their brain.

Nor are they mentally searching a database row:

KNIGHT_OUTPOST = GOOD

Years of games have changed a distributed network so that certain board positions simply produce:

That knight belongs there.

Some games may be remembered verbatim.

Some opening sequences are explicitly memorized.

But much of expertise has been compressed into pattern recognition and procedures.

An LLM's weights are analogous:

all encoded in an enormous distributed numerical program.