Week 05 — Reflection & Evaluation
| Material | Where |
|---|---|
| HTML chapter | Read online |
| Optional reading | Error analysis and component tests |
| Practice (first) | Open in Colab · Download notebook |
| Practice v2 | Open in Colab · Download notebook |
| Lab | Open in Colab · Download notebook |
| Homework | Open in Colab · Download notebook |
| Papers | Self-Refine (Madaan et al., 2023) (PDF) · Judging LLM-as-a-Judge (Zheng et al., 2023) (PDF) |
| Student presentation slides | Self-Refine · Judging LLM-as-a-Judge |
This week
The chapter has two parts.
- Reflection. A model examines a result, writes a critique, and revises the result. Feedback from a program, such as an error message, helps the critique.
- Evaluation. Measure a system on fixed cases with a written criterion. Use code for an exact rule and an LLM judge for meaning.
Do the work in this order:
- Practice. Do the practice notebook first. It shows one model call, the message list, JSON output, tool calls, the agent loop, ReAct, and the
Agentclass. Practice v2 shows the same patterns with one form for each call. - Lab. Every model is an
Agentobject. In Part 1, one agent writes a short science essay with Wikipedia citations, evaluates it, and improves it until it writes the final essay. In Part 2, an extractor returns JSON, and code measures its accuracy on ten invoices. In Part 3, you check an LLM judge against a person’s scores and then measure the essays of Part 1. - Homework. A coder object writes chart code. A critic object examines the chart image and revises the code. A judge object gives scores to both versions.
The ✍️ exercises leave only the key lines blank. We do not collect the practice notebooks or the lab.
Click Open in Colab in the table. Save your own copy first (File → Save a copy in Drive). For the homework, run the completed notebook from top to bottom and keep the outputs. Then select File → Download → Download .ipynb and upload the file to the LMS. Due date: see the LMS.