RAG vs. long context vs. learned memory

There are three fundamentally different ways for an LLM to access information:

These correspond roughly to internalized knowledge, long context, and RAG.

Long context¶

Put the entire source in the prompt.

Example: a 500-page handbook, perhaps 200K tokens, followed by a question.

Advantage: the model can decide during attention which parts matter and combine information across distant passages.

Cost: large prefill, large KV cache, expensive attention.

RAG¶

Search first, then give the model only selected passages.

A typical embedding retriever maps: - query → vector \(q\) - document chunks → vectors \(d_i\)

and compares, for example:

Maybe the relevant handbook passage scores 0.91 while unrelated chunks score 0.1–0.4.

The LLM now receives 1500 tokens rather than 200K.

Much cheaper.

But retrieval can fail.

A complex answer may require several policies that a naïve retriever did not fetch.

Learned memory in weights¶

Stable facts such as “Paris is the capital of France” can be compiled into model weights during training.

This is cheap at inference, but difficult to update exactly and difficult to cite.

If a company policy changes tomorrow, changing a document is easy; changing a distributed parametric association is not.

Memory hierarchy¶

A useful systems analogy:

model weights        → deeply internalized knowledge
current context/KV   → working memory
external retrieval   → library

Agents add more tiers: - recent history kept exactly, - important past facts stored structurally, - older conversations searchable externally.

The central architectural problem is deciding what belongs in which memory tier.

RAG is a two-stage reasoning system¶

In vanilla RAG, retrieval happens before the main LLM has deeply reasoned about the question.

That means the retriever is effectively making an early relevance decision:

query
  ↓
retriever
  ↓
chosen passages
  ↓
LLM reasoning

If the necessary evidence is not selected, the model may never get a chance to reason over it.

More advanced systems therefore iterate: 1. reason, 2. realize what information is missing, 3. retrieve, 4. continue reasoning.

That turns retrieval into a tool rather than a one-shot preprocessing step.

Exactness and provenance¶

External documents have a major advantage over weight memory: you can point to the exact source.

For tasks involving: - current policy, - financial data, - legal text, - citations, - user-specific records,

retrieval is often preferable even if the model “knows” roughly the same fact.

Context is expensive but flexible¶

Long context keeps evidence inside the model’s native attention mechanism. The model can combine subtle clues that an embedding retriever might not consider similar enough.

Hence the practical answer is rarely “RAG or long context.” It is often a layered combination.

Memory hierarchy analogy¶

Computer systems do not choose between registers or RAM or disk. They use a hierarchy.

Likewise capable AI systems increasingly combine: - weights, - context, - cache, - structured state, - retrieval, - tools.

Retrieval granularity matters¶

Documents are commonly split into chunks.

Chunks that are too small may omit necessary context.

Chunks that are too large reduce retrieval precision and put unnecessary tokens into the LLM context.

So RAG has its own systems hyperparameters: - chunk size, - overlap, - embedding model, - number of retrieved chunks, - reranking strategy.

Embedding similarity is not reasoning¶

A vector retriever is good at finding semantically related material, but some questions require logical joins rather than surface similarity.

For example:

Which employee works for the company acquired by the firm whose CEO wrote this memo?

No single chunk may look strongly similar to the entire question.

An agentic retrieval system can break the task into subqueries and retrieve iteratively.

Thus RAG ranges from simple nearest-neighbor lookup to multi-step information gathering.

The external memory system can itself become part of the reasoning policy.

Reranking adds a middle layer¶

A common architecture is:

cheap embedding retriever
        ↓
20 candidate chunks
        ↓
more expensive reranker
        ↓
top 5 chunks
        ↓
LLM

The first stage emphasizes recall: do not miss potentially relevant text.

The second stage spends more compute judging a small candidate set.

This is structurally similar to speculative decoding and reasoning search: use a cheap mechanism to propose broadly, then an expensive mechanism to discriminate more carefully.

The same propose–filter pattern appears repeatedly across modern AI systems.