1.1 · Knowledge gaps and RAG

Parametric knowledge and its gaps

What a model learns in training is broad, but some facts never reach its weights.

1.2 · Knowledge gaps and RAG

Hallucination and missing sources

A question in a knowledge gap still gets a fluent answer, and nothing in the answer shows where it came from.

1.3 · Knowledge gaps and RAG

Document-based applications

Many applications must answer from a specific set of documents.

1.4 · Knowledge gaps and RAG

Two ways to supply documents

The documents can go into the weights by training, or into the input at answer time.

1.5 · Knowledge gaps and RAG

Retrieval-augmented generation

RAG puts both kinds of knowledge to work: the model supplies the language ability, and the documents supply the facts.

Definition

RAG: a method that retrieves relevant passages from external documents and gives them to a language model, which answers from them.

Parametric

Written into the weights by training.

Non-parametric

Placed into the current input by retrieval.

1.6 · Knowledge gaps and RAG

Passage selection

A few pages fit into the prompt. Thousands of pages do not, so the system must select the passages that the question needs.

1.7 · Knowledge gaps and RAG

Two stages of a RAG system

Work that does not depend on the question runs once, before anyone asks.

Stage 1 · Preparation

The documents are divided and encoded. This runs when documents are added or changed.

Stage 2 · Answering

Retrieval and generation run for each question. The preparation keeps each answer fast.

1.8 · Knowledge gaps and RAG

Example question and cited answer

A company assistant answers employees’ questions from the employee handbook. The answer comes from a retrieved passage and cites it.

2.1 · Document preparation

Chunks

A question needs a few sentences, not a whole document, so the system stores and retrieves short passages.

Need

Real documents are long: the EU AI Act has about 330,000 characters.

Definition

Chunk: a short passage of a document. The system splits each document into chunks and retrieves chunks.

Record

Each chunk keeps its text, its position in the document, and its source.

2.2 · Document preparation

Chunk size and boundaries

A cut in the wrong place separates a sentence from the words that give it meaning.

2.3 · Document preparation

Chunk count

Overlap repeats text, so it increases the number of chunks.

L characters in the document

s characters at most in one chunk

o characters shared by neighboring chunks; each chunk after the first adds only s − o new characters

Measured on the EU AI Act text · L = 328,831, s = 1,500, o = 150

Formula

244

chunks if every chunk is full

Actual splitter

298

chunks: it first cuts at the 126 headings of articles and annexes, then at line ends, so most chunks stay below s

2.4 · Document preparation

Embeddings

An embedding model turns each text into a vector: a fixed-length list of numbers.

2.5 · Document preparation

Cosine similarity

Cosine similarity compares the directions of two vectors and ignores their lengths.

Reason

Embedding models are trained so that related texts have a small angle between them.

Numerator

The dot product is large when two vectors point the same way.

Denominator

Division by the lengths removes length, so only the angle remains.

For vectors of length 1, the cosine equals the dot product; vector stores use this to save computation.

2.6 · Document preparation

Indexing and the vector store

The index finds the nearest vectors quickly; the store keeps each vector with its text and source.

id        §3.1
vector    [0.021, -0.013, 0.044, ...]
text      "Employees in their first year receive
           15 days of annual leave."
metadata  {source: "Employee handbook",
           section: "3.1 Annual leave"}

3.1 · Retrieval and answer generation

Similarity scores

A score ranks the chunks for one question. It does not prove that a chunk is correct.

3.2 · Retrieval and answer generation

Top-k retrieval

The model cannot read every chunk, so the system passes only the best few.

3.3 · Retrieval and answer generation

Grounded generation

The model answers only from the retrieved passages, not from its own memory.

Answer the question using only the passages in the user message.
1. Do not add facts from your own knowledge.
2. After each claim, cite the passage ID in brackets, such as [§3.1].
3. If the passages do not contain the answer, say so and name what is missing.

Question: How many vacation days do new employees get?

Passages:
[§3.1] Employees in their first year receive 15 days of annual leave.
[§3.4] Leave requests must be submitted two weeks in advance.

New employees receive 15 days of annual leave in their first year [§3.1].

3.4 · Retrieval and answer generation

Original RAG model (Lewis et al., 2020)

The first RAG model joins a retriever and a generator, and weights each retrieved passage by its relevance.

Retriever · DPR

Finds the top-k passages among about 21 million Wikipedia passages. p_η(z | x) is the relevance of passage z to question x.

Generator · BART

Reads the question with one passage and writes the answer. p_θ(y | x, z) is the probability of answer y with that passage: the product of its token probabilities.

Training

The question encoder and the generator learn together from question–answer pairs. The passage index stays fixed.

Reading of the formula

For each retrieved passage: relevance of the passage × probability of the answer with that passage. The sum over the top-k passages gives the probability of the answer. One passage serves the whole answer (RAG-Sequence).

The RAG in this chapter trains nothing: it connects a ready-made embedding model and a ready-made chat model through a prompt.

4.1 · Retrieval errors and grounding errors

Two causes of a wrong answer

RAG has two stages, so a wrong answer has two possible points of failure.

4.2 · Retrieval errors and grounding errors

Recall at k

Recall at k measures retrieval on its own: the share of the needed chunks that appear in the top k.

Illustrative example

How many vacation days does an employee get in the third year?

Needed§3.1§3.2

Top-2§3.1 ✓§3.4 ✗also about leave

recall@2 = 1/2

Without §3.2, no passage gives the third-year number; any number that the model writes is a guess.

4.3 · Retrieval errors and grounding errors

Hybrid search and reranking

The first search can miss a needed passage. Two methods reduce such misses.

Hybrid search

Adds a keyword search. Embeddings handle paraphrases but can blur exact names, codes, and numbers; keywords match them directly.

Reranker

Reads the question and each candidate together, so it judges relevance more precisely. It is slow, so it only reorders the short list.

Pattern: a cheap, broad search first, then an expensive, careful one. A reranker cannot add a chunk that the first search missed.

4.4 · Retrieval errors and grounding errors

Grounding errors

A claim can sound right and still have no support in the passage.

5.1 · Agentic RAG and Self-RAG

Limit of a single search

Basic RAG searches once. Some questions reveal the next search term only in the first result.

5.2 · Agentic RAG and Self-RAG

Agentic RAG

The retriever becomes a tool. Each result is an observation, and the agent chooses the next search or the answer.

5.3 · Agentic RAG and Self-RAG

Fixed workflow and agent loop

A workflow always searches once; an agent decides how many searches a question needs.

Fixed workflow

Retrieve, then generate: one search for every question.

Agent loop

After each observation, the next search or the answer.

Cost

Each search adds a model call and time.

Stop rule

A limit on searches, so the loop cannot run without end.

5.4 · Agentic RAG and Self-RAG

Self-RAG

A prompted agent judges only as well as its prompt. Self-RAG trains the model itself to make these judgments.

5.5 · Agentic RAG and Self-RAG

Self-RAG process and limit

The decisions are learned in training, so they need labeled data and a trained model.

Simplified redraw after the Self-RAG authors’ overview (selfrag.github.io), CC BY-SA 4.0

Process

A retrieval decision, candidate generation, assessment of relevance, support, and utility, then selection of the text to keep.

Limit

Training data labeled with these tokens and a model trained on that data. Prompting alone cannot add Self-RAG to an existing chat model.

Sources

Sources