Chapter 5. Reflection & Evaluation

Correct a result with feedback · Measure a system on fixed cases

Learning objectives
  • Explain the three steps of reflection.
  • Explain what external feedback adds to a critique.
  • Name the parts of an evaluation.
  • Define an LLM judge, and select code or an LLM judge for a criterion.
  • Compare two versions of a system with an evaluation.

In Chapter 4, the agent loop returned the first final answer of the model. That answer can contain errors. This chapter adds two methods:

Part 1. Reflection

1.1 Why review a result

A person often finds errors in a draft when they read it again. The draft is complete, so the person can compare it with the task. A model can do the same. After the model writes a result, a second step gives the result back to the model. This step asks one question: what must change so that the result satisfies the task?

1.2 Reflection

Reflection

A workflow in which an LLM examines a result, writes feedback about it, and then uses that feedback to change the result.

Reflection has three steps:

  1. Generate. The model does the task and writes a first result, the draft.
  2. Critique. The model receives the task and the draft. It examines the draft against the task and lists the problems. This list is the critique.
  3. Revise. The model receives the critique and changes the draft.
Direct generation: Task, Generate, Submit. With reflection: Task, Generate, Critique, Revise, Submit.
Figure 1: Direct generation submits the first result. Reflection adds a critique and a revision before the system submits the result.

The same model can do all three steps. Each step is only a different prompt, and the model itself does not change (Madaan et al., 2023).

1.3 Feedback

The critique is one type of feedback: information about a result that the next step uses. A model writes the critique from the text of the draft. But some errors are not visible in the text. For example, a SQL query can look correct and still return wrong rows.

External feedback

Feedback that comes from outside the model: the output of a program that runs or draws the result.

Examples of external feedback:

  • the rows that a query returns,
  • an error message,
  • a chart that code draws.

The program gives this output to the model together with the draft. Then the critique can use facts about the result, not only its text. For this reason, reflection is most useful when a program can run or examine the result.

1.4 The reflection loop

The critique and the revision can repeat. Each round examines the latest version. The loop stops when the critique finds no problem, or when the number of rounds gets to a limit. Each round adds model calls, so the limit is usually small.

Task, Generate, a program runs the draft; its output (rows, error, chart) and the draft go to Critique; Revise; Result. The revised version runs again until no problem remains or the round limit.
Figure 2: Reflection with external feedback.

Check 1 · External feedback

What does external feedback give to the critique that the text of the draft does not give?

Read the answer

Facts about what the result does: for example, the rows that a query returns, an error message, or the chart that code draws.

Part 2. Evaluation

2.1 Why measure

Reflection can correct one result. But does it make the system better on most tasks? One good result does not answer this question. We must run the system on many tasks and count the correct results. This measurement is an evaluation.

Evaluation

A measurement of the outputs of a system against written criteria on a fixed set of cases.

2.2 The parts of an evaluation

Chapter 2 compared two prompts with an evaluation set and accuracy. This section names all parts of an evaluation.

An evaluation starts with a criterion: a written rule that a correct output obeys. For example: “The output is the payment due date of the invoice, in the form YYYY/MM/DD.”

Next, the evaluation needs cases. An evaluation set is a fixed set of inputs for the system. For each input, a person writes the correct output. This output is the reference answer. The system does not receive it.

A grader applies the criterion to each output and gives a grade. A metric summarizes the grades. The simplest metric is accuracy: the number of correct outputs divided by the number of cases.

Evaluation set with inputs and reference answers; the inputs go to the System, its Outputs go to the Grader (code or LLM judge), and the Grader gives the Metric (accuracy). The reference answers go only to the Grader.
Figure 3: The parts of an evaluation.

2.3 Two types of grader

Some criteria are exact. A date is equal to the reference date, or it is not. Code can grade these criteria, for example with pred == ref.

Other criteria need an interpretation of meaning. For example: “The essay answers an argument from the other side.” Two essays can do this in very different words. For these criteria, a model grades the output.

LLM judge

A model that grades an output against written criteria. It receives the task, the output, and a rubric: the criteria and the rules for the score.

An LLM judge is also a model, so it can make errors. Before you use it, compare its grades with the grades of a person on some cases.

2.4 Compare two versions

An evaluation lets you compare two versions of a system. Run both versions on the same evaluation set with the same grader. Change only one part, for example the system message, or add reflection. Then the difference in the metric comes from that change.

The same evaluation set goes to Version A and Version B (one change, for example reflection); the same grader gives Metric A and Metric B, which are compared.
Figure 4: Comparing two versions.

Also read the failed cases. They show what to change next.

Check 2 · Does reflection help?

You add reflection to a system. How do you find out if the system is better?

Read the answer

Run the system with reflection and without reflection on the same evaluation set. Grade both with the same grader, and compare the accuracy. Also compare the number of model calls, because reflection adds calls.

For regression tests and component tests, read the optional evaluation supplement.

Summary

  • Reflection has three steps: generate, critique, and revise. The same model can do all three steps.
  • External feedback, such as the output of a program, gives the critique facts about the result.
  • An evaluation has a criterion, an evaluation set with reference answers, a grader, and a metric.
  • Code grades exact criteria. An LLM judge with a rubric grades criteria that need an interpretation of meaning.
  • To compare two versions, use the same evaluation set and the same grader, and change one part.

Lab preparation: from concept to code

The lab uses the Agent class from the practice notebook. Each call to ask adds a user message to the list of the object and keeps the reply.

Concept Lab code (short form)
One agent for all steps agent = Agent(REFLECT_SYSTEM, [wikipedia_search]): the system message gives the role and the steps of reflection
Generate reply = agent.ask(TASK): the first reply is the draft
Critique and revise reply = agent.ask("Continue."): each reply is the next step, an evaluation or an improved essay
External feedback The results of wikipedia_search enter the list before the next version
Reflection loop Repeat agent.ask("Continue.") until the reply holds the heading FINAL ESSAY, with a limit of 8 rounds
Run one case json.loads(Agent(EXTRACT_SYSTEM, json_mode=True).ask(invoice))["due_date"]: a new object for each case; the JSON field is compared with the reference
Code grader date_matches(pred, ref)
LLM judge judge_essay(QUESTION, essay): a new Agent(JUDGE_SYSTEM, json_mode=True); the rubric is its system message

Lab

  1. Do the practice notebook first.
  2. Part 1: one agent writes a short science essay with Wikipedia citations, evaluates it, and improves it until it writes the final essay.
  3. Part 2: measure the accuracy on ten invoices with code, and test one change on the same cases.
  4. Part 3: check an LLM judge against the scores of a person, then measure the essays of Part 1.

Practice notebook in Colab · Lab notebook in Colab · Homework notebook in Colab: reflection on a rendered chart. Submit the homework on the LMS.

Materials and sources

  • Andrew Ng, Agentic AI: Module 2 (reflection, and the adapted workflow diagram) and Module 4 (evaluation).
  • Madaan et al., Self-Refine (2023): generation, feedback, and revision with one LLM.
  • Zheng et al., Judging LLM-as-a-Judge (2023): LLM judges and their agreement with human grades.