What an LLM is actually learning

At its core, an LLM is trained to do one deceptively simple thing:

Given the tokens so far, predict the next token.

Suppose the training text contains:

The capital of France is Paris.

The model might see:

The capital of France is

and assign probabilities something like:

Paris → 0.91
Lyon → 0.02
France → 0.01
thousands of other tokens → tiny probabilities

Training adjusts billions of numerical weights so that the probability of the correct next token goes up.

The important insight is that next-token prediction forces the model to learn much more than autocomplete. To reliably predict text across trillions of tokens, it becomes useful to internally represent grammar, facts, programming patterns, concepts, relationships between objects, and even rough models of how people reason.

What happens during one training step?¶

Conceptually:

Text → tokens → model → probabilities → error → update weights

For example:

Input:
"The cat sat on the"

Model predicts:
mat   0.35
floor 0.20
roof  0.08
...

Actual next token:
"mat"

The training system computes a number called the loss. For LLMs this is usually cross-entropy loss, which essentially says:

“How surprised were you by the correct answer?”

If the model gave mat probability 0.35, the loss for that token is:

\[ -\log(0.35) \approx 1.05 \]

If it had predicted mat with 0.95 probability:

\[ -\log(0.95) \approx 0.05 \]

Lower is better.

Then backpropagation figures out how every relevant weight contributed to that error, and an optimizer slightly changes those weights.

This happens over and over—across enormous amounts of text.

Training vs. inference¶

This distinction is fundamental.

Training¶

tokens
  ↓
forward pass
  ↓
prediction
  ↓
compare with correct answer
  ↓
loss
  ↓
backpropagation
  ↓
change weights

Inference¶

tokens
  ↓
forward pass
  ↓
prediction
  ↓
choose next token
  ↓
append token
  ↓
repeat

During ordinary inference, the weights do not change.

If you ask:

Why is the sky blue?

the model computes probabilities for the first answer token, chooses one, then recomputes for the next token, and keeps going.

That means generating a 1,000-token answer requires roughly 1,000 sequential token-generation steps.

This sequential nature is one major reason inference has some interesting performance constraints.

A useful mental model¶

Think of an LLM as an enormous function:

\[ f(\text{tokens}, \text{weights}) \rightarrow \text{probability distribution over next token} \]

Training discovers the weights.

Inference repeatedly evaluates the function.

The architecture defines the machinery; training fills in the numbers.

And that distinction leads directly to one of the most important concepts in modern AI:

What exactly are those billions of weights doing?

Why this deceptively simple objective creates broad capability¶

The model is never explicitly given labels such as “grammar,” “world knowledge,” “reasoning,” or “programming.” It only sees token sequences and is rewarded for predicting what comes next. But accurate prediction often requires those hidden capabilities.

If a sentence says:

The glass fell from the table and ___.

predicting shattered requires some model of physical events. If the input is:

for i in range(10):
    print(i)

predicting the next lines requires knowledge of Python syntax and program behavior. If the text contains a mathematical derivation, predicting the next token may require maintaining the algebraic state.

So next-token prediction is better thought of as a compression pressure: the model needs to discover whatever latent structure helps it predict a huge variety of text.

One sequence gives many training examples¶

For a tokenized sequence:

\[ x_1,x_2,x_3,\ldots,x_T \]

the model does not learn from only the final token. A single sequence supplies many next-token targets:

\[ P(x_2|x_1),\; P(x_3|x_1,x_2),\; \ldots,\; P(x_T|x_{<T}) \]

The causal mask lets these positions be trained in parallel even though the mathematical prediction for position \(t\) can only depend on tokens before \(t\).

This is one of the key reasons LLM pretraining can use GPUs so efficiently: the training sequence is already known, so thousands of next-token predictions can be evaluated simultaneously.

Parameters versus activations¶

It is useful to distinguish two things we will keep returning to.

Parameters are the learned weights that persist between requests. Training changes them.

Activations are the temporary values produced while a particular prompt flows through the model. They disappear after the computation, except for state deliberately retained during inference such as the KV cache.

A prompt does not normally “write a new fact into the weights.” It changes activations and context. Training is what changes the persistent model.