Week 12 — Inference Economics · Benchmarks

Material	Where
Lecture notes	published before class
Lab	published before class
Papers	Scaling LLM Test-Time Compute Optimally (Snell et al., 2024) (PDF) · FrugalGPT (Chen et al., 2023) (PDF) · AI Agents That Matter (Kapoor et al., 2024) (PDF) · τ-bench (Yao et al., 2024) (PDF)

This week

The inference an agent consumes — calls × tokens × model tier — placed on the same axis as accuracy: the cost–accuracy trade, compute-optimal allocation, and the levers (routing, cascades, caching, budgets). Then agent-level benchmarks on top of W5’s component evaluation: the AI-Agents-That-Matter critique, cost–accuracy Pareto curves, and consistency metrics (τ-bench, pass^k). The course eval harness that final presentations must defend against.

Lab plan

Weekly LangChain and LangGraph practice follows LangChain Academy and official RAG tutorials.