Mechanistic interpretability — can we read what an LLM is thinking?

Yesterday we said an LLM's weights aren't a database. They're closer to a gigantic compiled program containing distributed facts, concepts, and learned procedures.

That creates an obvious question:

Can we reverse-engineer that program?

This is the goal of mechanistic interpretability.

The basic idea is to stop treating the model as:

\[ \text{prompt}\rightarrow\boxed{\text{mysterious neural network}}\rightarrow\text{answer} \]

and instead ask what computation happens inside the box.

A useful analogy is reverse-engineering a binary when you don't have its source code.

Start with something we'd love to find¶

Imagine giving a model:

John drove from Boston to New York. He stopped in Connecticut for lunch. Where did John start his trip?

The model answers:

Boston.

Somewhere inside the Transformer it must preserve information roughly equivalent to:

John → starting location → Boston

We might hope to find a neuron whose activation means:

\[ \boxed{\text{Boston-is-start-location}} \]

Then interpretability would be easy.

Inspect neurons.

Label them.

Read the model.

Unfortunately, neural networks usually aren't organized so conveniently.

Why individual neurons are troublesome¶

Imagine one hidden layer has only three dimensions:

\[ h=[h_1,h_2,h_3]. \]

You might hope for:

h₁ = "is a city"
h₂ = "is a person"
h₃ = "is a country"

This is called a monosemantic representation: one component corresponds neatly to one interpretable feature.

But neural networks have strong incentives to reuse dimensions.

Instead, a neuron might respond to some strange mixture of:

cities
French words
quotation marks
software package names

depending on context.

This phenomenon is called polysemanticity.

Why would the network do something so inconvenient?

Because it lets the model pack more useful features into limited representational space.

Superposition¶

Suppose we have only two dimensions:

\[ h\in\mathbb R^2. \]

Naïvely, we'd expect to store only two independent features:

feature A → x-axis
feature B → y-axis

But suppose features are sparse—they're rarely active simultaneously.

Then the network can encode several features as different directions:

             B
             ↑
          C ↗│
             │
      ───────┼──────→ A
           ↙ │
          D  │

Perhaps:

\[ v_A=(1,0) \]

\[ v_B=(0,1) \]

\[ v_C=(0.7,0.7) \]

\[ v_D=(-0.7,-0.7). \]

We now have more features than dimensions.

When only a few features are active at once, the network can often disentangle them well enough to perform useful computation.

This is superposition.

And it explains something important from yesterday:

\[ \boxed{\text{a concept need not correspond to one neuron}} \]

A concept can correspond to a direction in activation space.

A concrete example¶

Suppose an internal activation has 4,096 dimensions.

We collect activations from millions of text examples:

"The Eiffel Tower is in Paris."
"def calculate_loss(..."
"She was furious with him."
"DNA is transcribed into RNA."
...

We want to discover meaningful recurring patterns.

One approach is to train another neural network—typically a sparse autoencoder, or SAE.

It takes:

\[ h\in\mathbb R^{4096} \]

and maps it into a much larger feature space:

\[ z\in\mathbb R^{65536}. \]

Why make the representation larger?

Because we impose sparsity:

\[ \boxed{\text{only a tiny number of the 65,536 features should activate at once}} \]

and train a decoder to reconstruct the original activation:

\[ h\approx W_{dec}z. \]

Conceptually:

messy 4096-D
neural activation
      ↓
SPARSE AUTOENCODER
      ↓
65,536 candidate features
      ↓
only ~dozens active

The hope is that those sparse features become more interpretable.

What might a feature look like?¶

Suppose feature 18,427 activates strongly on:

Paris is the capital of France.
Berlin is Germany's capital.
Tokyo, the capital of Japan...
Canberra is the Australian capital.

but weakly on:

Paris has excellent restaurants.
Berlin has many museums.
Tokyo has a large population.

We might label it:

\[ \boxed{\text{country-capital relationship}} \]

Another feature might activate on:

def foo():
class User:
import torch
return result

and get labeled:

\[ \boxed{\text{Python code}} \]

Another might respond to:

I'm devastated.
She began sobbing.
He was overwhelmed by grief.

perhaps:

\[ \boxed{\text{grief/sadness}} \]

Notice that nobody explicitly created these labels during model training.

The features emerged because they were useful for prediction.

Interpretability tries to discover them afterward.

Finding a feature isn't enough¶

Suppose feature \(z_{18427}\) activates on capital relationships.

We still don't know whether the model uses it.

Correlation is not causation.

So researchers can intervene.

Imagine the model is processing:

The capital of France is

and feature 18,427 becomes active.

We artificially set:

\[ z_{18427}=0. \]

Then run the model again.

If:

before intervention:
Paris 94%

after intervention:
Paris 41%

we've learned something much stronger.

The feature isn't merely correlated with capital questions.

It appears to participate causally in the computation.

This general technique is called activation intervention or ablation.

We can also push features¶

Instead of removing a feature:

\[ z_i\leftarrow0 \]

we can amplify it:

\[ z_i\leftarrow z_i+\alpha. \]

Suppose we've identified a feature associated with:

\[ \text{formal mathematical reasoning}. \]

Increasing its activation might shift the model's continuation toward more mathematical language or behavior.

Conceptually:

normal activation
       ↓
"Let's estimate..."

boost feature
       ↓
"Let x denote..."

This gives us a remarkable experimental capability:

Change an internal concept and observe how the model's behavior changes.

That's much closer to neuroscience than ordinary software debugging.

But a Transformer isn't just a bag of features¶

Remember the architecture we've built up:

\[ \text{residual stream} \rightarrow \text{attention} \rightarrow \text{MLP} \rightarrow \text{residual stream} \rightarrow\cdots \]

Features are being:

created,
copied,
combined,
transformed,
suppressed,
routed between token positions.

So the deeper goal isn't merely to create a dictionary:

feature 12,832 = France
feature 18,427 = capitals
feature 24,991 = geography

We want to understand circuits:

\[ \boxed{\text{which features interact to perform a computation?}} \]

A toy circuit¶

Take:

Alice gave the book to Bob. Later, Bob handed it to Carol. Who has the book?

Perhaps the model performs something approximately like:

detect possession relation
          ↓
Alice → book
          ↓
detect transfer
          ↓
Bob → book
          ↓
detect second transfer
          ↓
Carol → book
          ↓
answer Carol

Different attention heads might move information between token positions.

MLPs might detect concepts such as:

person
object
transfer
current owner

Later layers combine them.

The resulting computation could involve dozens or hundreds of interacting components.

That's a circuit.

Finding these circuits is much harder than identifying individual features.

Attention gives us one tempting clue¶

Remember attention:

\[ A=\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d}}\right). \]

We can inspect the attention matrix.

Perhaps while generating Carol, a head strongly attends to:

"Bob handed it to Carol"
                   ↑

It's tempting to conclude:

Aha! This head found the answer.

But attention weights tell us primarily where information is being read from, not necessarily what computation is being performed with that information.

A head can attend strongly to a token while contributing very little to the final output.

So:

\[ \boxed{\text{attention visualization}\neq\text{full explanation}} \]

Causal interventions are much stronger evidence.

Now connect this to reasoning RL¶

Over our last several lessons, we've said RL might teach a model behaviors like:

decompose
check assumptions
use tools
verify
backtrack

Mechanistic interpretability gives us a tantalizing possibility.

Train model:

\[ M_0 \]

with reasoning RL to obtain:

\[ M_1. \]

Then compare their internal activations.

Perhaps \(M_1\) has stronger or more reliable circuits associated with:

error detection
planning
verification
uncertainty

Maybe RL didn't create a completely new algorithm.

Perhaps it strengthened an existing latent circuit.

Or perhaps it caused previously separate features to interact in a new way.

Interpretability could potentially tell us.

This connects directly to yesterday's question:

Does RL create reasoning or uncover capabilities already latent in pretraining?

Behavioral benchmarks alone can't fully answer that.

Looking inside the network might.

Why this matters beyond scientific curiosity¶

Suppose an AI says:

I've checked the calculation and it is correct.

We'd like to distinguish:

A:
model actually performed verification

from:

B:
model generated language that sounds like verification

Those outputs can be linguistically identical.

If we understood the internal circuits involved in genuine checking, perhaps we could inspect whether they actually activated.

Similarly, imagine a model deliberately producing deceptive behavior.

Output monitoring only sees:

\[ \text{text}. \]

Mechanistic interpretability aims eventually to inspect:

\[ \boxed{\text{computation producing the text}}. \]

That's one reason the field matters for AI safety as well as basic science.

But we're nowhere near reading models like source code¶

A frontier model may have:

\[ 10^{11}\text{–}10^{12} \]

parameters and process millions of internal activations for a single sequence.

Even if an SAE discovers:

\[ 1,000,000 \]

features, we still have to determine:

what each feature means
how reliable that interpretation is
how features interact
how computations change by context
which features causally matter

And some concepts may themselves be distributed across many SAE features.

So today's state is much closer to:

We've invented microscopes.

than:

We've completely mapped the brain.

Today's mental model: reverse-engineering an alien CPU¶

Imagine archaeologists discover a working computer.

They have:

no source code,
no instruction manual,
billions of transistors,
the ability to provide inputs,
the ability to measure internal voltages,
the ability to disable or amplify individual components.

At first:

input → machine → output

is mysterious.

Eventually they notice:

These wires activate during multiplication.

Then:

This cluster seems to represent carries.

Then:

These components together implement something resembling an adder.

That's mechanistic interpretability.

For LLMs:

\[ \boxed{\text{neurons aren't necessarily concepts}} \]

because superposition mixes concepts together.

Sparse autoencoders try to recover:

\[ \boxed{\text{interpretable features}} \]

and causal interventions ask:

\[ \boxed{\text{does this feature actually affect behavior?}} \]

The ultimate goal is to move from:

\[ \text{billions of inscrutable floating-point numbers} \]

toward:

\[ \boxed{\text{a mechanistic account of the algorithms the model learned}}. \]