{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bc034117",
   "metadata": {
    "vscode": {
     "languageId": "python"
    }
   },
   "source": "# W2 Lab — Prompting and Reasoning: Chain-of-Thought and Examples\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ralbu85/stml_2026/blob/main/lectures/week02/W2_lab_prompting.ipynb)\n\n**Goal.** Measure with code when three prompting techniques change the model's accuracy.\n\n- **Chain-of-thought (CoT)** = making the model write its solution before the answer.\n- **Few-shot prompting** = putting worked examples in the prompt for the model to imitate.\n- **Self-consistency** = sampling several solutions and taking the majority answer.\n\nSections: setup (1) → answer-only vs worked solution (2) → reasoning that is not written (3) → a code-graded eval, baseline → format fix → your CoT prompt, target 11/12 (4) → worked examples (5) → self-consistency (6) → the limit of written reasoning (7) → in-class check (8).\n\n*Runtime:* Google Colab, top to bottom, about 90 minutes. Every call is written out as in W1: a message list, one `create` call, the reply at `response.choices[0].message.content`.\n\nThis lab is not collected; the graded part of the week is the separate homework notebook `W2_hw_prompting.ipynb`. ✍️ marks a cell that needs your own writing. *Try:* names a prompt change to make and rerun before moving on.\n"
  },
  {
   "cell_type": "markdown",
   "id": "56e3e9cd",
   "metadata": {},
   "source": "## 1. Setup\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "5f714fb9",
   "source": "### 1.1 Installation\n\n`aisuite` runs the same code on OpenAI or Anthropic keys.\n\n*Do:* run the cell (about 30 seconds, once per session).\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac40340a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q \"aisuite[openai,anthropic]\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b43cf834",
   "metadata": {},
   "source": "### 1.2 API key and model\n\n**API key** = the secret string that identifies your account and bills usage to it (issuing steps: the API Setup guide on the course site). Do not share the notebook with the key inside.\n\n*Do:* replace `PASTE-YOUR-KEY-HERE` with your key and run the cell. Nothing prints.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b97052b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"PASTE-YOUR-KEY-HERE\"\n",
    "\n",
    "MODEL = \"openai:gpt-4o-mini\"   # Anthropic accounts: MODEL = \"anthropic:claude-haiku-4-5\" and set ANTHROPIC_API_KEY instead"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2918f4d",
   "metadata": {},
   "source": "### 1.3 Client and a first call\n\nThe same call that opened W1. Every later cell repeats this shape; only the prompt string changes.\n\n*Do:* run the cell. The output must be exactly `ready`.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92d922a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import aisuite\n",
    "\n",
    "client = aisuite.Client()\n",
    "\n",
    "messages = [{\"role\": \"user\", \"content\": \"Reply with exactly: ready\"}]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "817a7062",
   "metadata": {},
   "source": "Any error here is a setup problem: recheck the API Setup guide before continuing.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 2. Answer-Only versus Worked Solution\n\nSame model, same problem, two prompts: one demands only the number, the other demands the solution first. If the scores differ, the difference comes from what the model wrote before the answer, not from the model.\n",
   "id": "7e5db2f6"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "ea87d701",
   "source": "### 2.1 The apple problem\n\nThe first example of the original chain-of-thought paper: a 2022 model answered 27 without a solution and 9 with one. This checks whether a current model still fails on it.\n\n*Do:* run both cells. Both should be correct.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "## Answer only\nAPPLES = \"A cafeteria has 23 apples. They use 20 for lunch and buy 6 more. How many apples do they have?\"\n\nmessages = [\n    {\"role\": \"user\", \"content\": APPLES + \" Reply with only the number.\"},\n]\nresponse = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\nprint(response.choices[0].message.content)\n",
   "id": "dd60e791"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "## Worked solution first\nmessages = [\n    {\"role\": \"user\", \"content\": APPLES + \" Write the solution step by step, then give the answer on the last line as ANSWER: <number>.\"},\n]\nresponse = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\nprint(response.choices[0].message.content)\n",
   "id": "f820dbf4"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "Both are correct: one intermediate step (23 − 20 = 3) fits in a single prediction. The next cells add steps and digits to find where answer-only breaks.\n",
   "id": "961f9f01"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "67d1f065",
   "source": "### 2.2 Larger numbers, graded by code\n\nSix problems of the same shape with a two-digit multiplication added. The question is a **template** = a string with named placeholders (`{a}`, `{b}`, ...) that `.format` fills from a table of numbers. Python computes each answer, so the score is exact. Expected: answer-only fails on most items, the worked solution passes.\n\n*Do:* run the three cells and compare the two scores. Each line prints the correct answer, then the model's reply.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "QUESTION = \"A warehouse holds {a} crates with {b} items each. {c} items are shipped out and {d} items are returned. How many items are in the warehouse now?\"\n\nnumbers = [\n    (47, 38, 519, 284),\n    (63, 29, 807, 356),\n    (84, 57, 2046, 173),\n    (92, 73, 3318, 907),\n    (54, 96, 1480, 622),\n    (79, 68, 4155, 231),\n]\n\nproblems = []\nfor a, b, c, d in numbers:\n    question = QUESTION.format(a=a, b=b, c=c, d=d)\n    answer = a * b - c + d\n    problems.append((question, answer))\n\nfor question, answer in problems:\n    print(answer, question)\n",
   "id": "78c4a43a"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "## Answer only\nINSTRUCTION = \"Reply with only the number.\"\n\nscore_answer_only = 0\nfor question, answer in problems:\n    messages = [\n        {\"role\": \"user\", \"content\": question + \" \" + INSTRUCTION},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    output = output.replace(\",\", \"\").strip()\n    print(answer, output)\n    if output == str(answer):\n        score_answer_only += 1\n\nprint(\"answer only:\", score_answer_only, \"/\", len(problems))\n",
   "id": "c9e2a2e6"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "## Worked solution first\nINSTRUCTION = \"Write the solution step by step, then give the answer on the last line as ANSWER: <number>.\"\n\nscore_worked = 0\nfor question, answer in problems:\n    messages = [\n        {\"role\": \"user\", \"content\": question + \" \" + INSTRUCTION},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    last_line = output.strip().splitlines()[-1].replace(\",\", \"\")\n    print(answer, last_line)\n    if str(answer) in last_line:\n        score_worked += 1\n\nprint(\"worked solution:\", score_worked, \"/\", len(problems))\n",
   "id": "0bbb71f2"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "The gap is the chain-of-thought effect: intermediate values written into the text before the answer.\n\n*Try:* make `crates` and `items` one-digit and rerun; the gap should close. Then make them three-digit; the worked solution begins to fail too, because each written step also has a size limit.\n",
   "id": "77d83873"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 3. Reasoning That Is Not Written\n\nTwo controls on the same six problems. Each keeps the instruction to reason but removes the reasoning from the text before the answer. If both score like answer-only, the accuracy in Section 2 came from the written text, not from the instruction to think.\n",
   "id": "46b84398"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "40918d39",
   "source": "### 3.1 Thinking in silence\n\nThe prompt asks the model to think carefully and write only the number.\n\n*Do:* run the cell. Expected score: the answer-only level.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "## Told to think silently\nINSTRUCTION = \"Think about it carefully in silence, do not write your reasoning, and reply with only the number.\"\n\nscore_silent = 0\nfor question, answer in problems:\n    messages = [\n        {\"role\": \"user\", \"content\": question + \" \" + INSTRUCTION},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    output = output.replace(\",\", \"\").strip()\n    print(answer, output)\n    if output == str(answer):\n        score_silent += 1\n\nprint(\"silent thinking:\", score_silent, \"/\", len(problems))\nprint(\"answer only:\", score_answer_only, \"  worked solution:\", score_worked)\n",
   "id": "d5542026"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "Reasoning that is not written is not performed.\n",
   "id": "0bc28ee4"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "0ddd3caf",
   "source": "### 3.2 Answer first, solution after\n\nThe prompt asks for the full solution, but after the answer line. Everything is written; nothing is written before the answer.\n\n*Do:* run the cell. Expected score: the answer-only level.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "## Answer first, then the solution\nINSTRUCTION = \"Give the answer on the first line as ANSWER: <number>, then explain the solution step by step.\"\n\nscore_answer_first = 0\nfor question, answer in problems:\n    messages = [\n        {\"role\": \"user\", \"content\": question + \" \" + INSTRUCTION},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    first_line = output.strip().splitlines()[0].replace(\",\", \"\")\n    print(answer, first_line)\n    if str(answer) in first_line:\n        score_answer_first += 1\n\nprint(\"answer first:\", score_answer_first, \"/\", len(problems))\nprint(\"answer only:\", score_answer_only, \"  worked solution:\", score_worked)\n",
   "id": "7d3f1c21"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "The answer is produced before any intermediate value exists, and a token already emitted cannot be revised. The original chain-of-thought paper reports the same. Rule for every prompt from here on: the solution goes before the answer, and the grader reads the answer from the end of the reply (an `ANSWER:` last line, or `<answer>` tags after `<thinking>`).\n\n*Try:* add `\"If the solution reaches a different number, end with CORRECTION: <number>.\"` to the answer-first prompt and count how often a correction appears.\n",
   "id": "0edd2874"
  },
  {
   "cell_type": "markdown",
   "id": "41f427f7",
   "metadata": {},
   "source": "## 4. A Code-Graded Evaluation\n\n**Code-graded evaluation** = running a prompt over a fixed test set with known answers and grading the outputs with code. The task: how many legs the animal in a statement has. Some statements are tricky (a fox that lost a leg and regrew two). The section improves one prompt in three versions and scores each.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "446fca92",
   "source": "### 4.1 The eval set\n\n**Eval set** = the fixed list of test items the prompt is scored on. Here: twelve statements, each with a **golden answer** = the known correct output.\n\n*Do:* run the cell and mark the statements you expect the model to miss.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f040e22e",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_data = [\n",
    "    {\"animal_statement\": \"The animal is a human.\", \"golden_answer\": \"2\"},\n",
    "    {\"animal_statement\": \"The animal is a snake.\", \"golden_answer\": \"0\"},\n",
    "    {\"animal_statement\": \"The fox lost a leg, but then magically grew back the leg he lost and a mysterious extra leg on top of that.\", \"golden_answer\": \"5\"},\n",
    "    {\"animal_statement\": \"The animal is a dog.\", \"golden_answer\": \"4\"},\n",
    "    {\"animal_statement\": \"The animal is a cat with two extra legs.\", \"golden_answer\": \"6\"},\n",
    "    {\"animal_statement\": \"The animal is an elephant.\", \"golden_answer\": \"4\"},\n",
    "    {\"animal_statement\": \"The animal is a bird.\", \"golden_answer\": \"2\"},\n",
    "    {\"animal_statement\": \"The animal is a fish.\", \"golden_answer\": \"0\"},\n",
    "    {\"animal_statement\": \"The animal is a spider with two extra legs\", \"golden_answer\": \"10\"},\n",
    "    {\"animal_statement\": \"The animal is an octopus.\", \"golden_answer\": \"8\"},\n",
    "    {\"animal_statement\": \"The animal is an octopus that lost two legs and then regrew three legs.\", \"golden_answer\": \"9\"},\n",
    "    {\"animal_statement\": \"The animal is a two-headed, eight-legged mythical creature.\", \"golden_answer\": \"8\"},\n",
    "]\n",
    "print(len(eval_data), \"items\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a89e7da",
   "metadata": {},
   "source": "### 4.2 The first prompt, on one item\n\nThe prompt is a template; `.format` fills the statement into the `<animal_statement>` tags. **Delimiters** = markers that separate pasted data from instructions. This cell sends one item (the fox) to see the raw reply before scoring the whole set.\n\n*Do:* run the cell. Note the reply's shape (bare number or sentence) and value (5 is correct).\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8386eaa5",
   "metadata": {},
   "outputs": [],
   "source": [
    "PROMPT_V1 = \"\"\"You will be provided a statement about an animal and your job is to determine how many legs that animal has.\n",
    "\n",
    "Here is the animal statement.\n",
    "<animal_statement>{statement}</animal_statement>\n",
    "\n",
    "How many legs does the animal have? Please respond with a number\"\"\"\n",
    "\n",
    "item = eval_data[2]   # the fox\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": PROMPT_V1.format(statement=item[\"animal_statement\"])},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "output = response.choices[0].message.content\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8a16307",
   "metadata": {},
   "source": "*Try:* set `item` to `eval_data[0]` and `eval_data[10]`. Does the reply's shape stay the same?\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "3e78df62",
   "source": "### 4.3 The whole set, graded\n\nThe loop repeats the call of 4.2 for every item; the second cell grades the replies.\n\n*Do:* run both cells.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a06b4fea",
   "metadata": {},
   "outputs": [],
   "source": "outputs_v1 = []\nfor item in eval_data:\n    messages = [\n        {\"role\": \"user\", \"content\": PROMPT_V1.format(statement=item[\"animal_statement\"])},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    outputs_v1.append(response.choices[0].message.content)\n\nfor item, output in zip(eval_data, outputs_v1):\n    print(\"golden\", item[\"golden_answer\"], \"| output:\", output)"
  },
  {
   "cell_type": "markdown",
   "id": "9b6bae0a",
   "metadata": {},
   "source": "The grader is one exact comparison. A correct number inside a sentence grades as wrong.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81842b37",
   "metadata": {},
   "outputs": [],
   "source": "score_v1 = 0\nfor item, output in zip(eval_data, outputs_v1):\n    correct = output.strip() == item[\"golden_answer\"]\n    score_v1 += correct\n    print(correct, \"golden\", item[\"golden_answer\"], \"| output:\", output)\n\nprint(\"score:\", score_v1, \"/ 12\")\n"
  },
  {
   "cell_type": "markdown",
   "id": "1e39f10c",
   "metadata": {},
   "source": "Two kinds of miss: sentences instead of bare numbers (format), and wrong numbers on tricky items (reasoning). The next two prompts fix them one at a time.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "ab42162a",
   "source": "### 4.4 Fixing the format\n\nOne changed last line: \"Respond only with a numeric digit, like 2 or 6, and nothing else.\"\n\n*Do:* run the cell. Format misses should disappear; tricky items should still miss.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aae88d47",
   "metadata": {},
   "outputs": [],
   "source": "PROMPT_V2 = \"\"\"You will be provided a statement about an animal and your job is to determine how many legs that animal has.\n\nHere is the animal statement.\n<animal_statement>{statement}</animal_statement>\n\nHow many legs does the animal have? Respond only with a numeric digit, like 2 or 6, and nothing else.\"\"\"\n\noutputs_v2 = []\nfor item in eval_data:\n    messages = [\n        {\"role\": \"user\", \"content\": PROMPT_V2.format(statement=item[\"animal_statement\"])},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    outputs_v2.append(response.choices[0].message.content)\n\nscore_v2 = 0\nfor item, output in zip(eval_data, outputs_v2):\n    correct = output.strip() == item[\"golden_answer\"]\n    score_v2 += correct\n    print(correct, \"golden\", item[\"golden_answer\"], \"| output:\", output)\n\nprint(\"score:\", score_v2, \"/ 12\")"
  },
  {
   "cell_type": "markdown",
   "id": "4aeeeb22",
   "metadata": {},
   "source": "*Try:* soften the last line to `\"Respond with the number.\"` and rerun. Which items come back as sentences?\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "a8db7ba6",
   "source": "### 4.5 Chain of thought, graded ✍️\n\nThe remaining misses are reasoning misses; Section 2's fix applies. The prompt must make the model reason inside `<thinking>` tags first and put only the integer inside `<answer>` tags. The grader then extracts the integer from the tags.\n\n**Starter** = the prefilled value of a fill-in; it runs but does not pass.\n\n*Do:* write `PROMPT_V3`: keep the task statement and `{statement}`; replace the closing question with the two-part instruction. Then run the three cells below.\n\nHints: name both tags; say what goes in each; state that `<answer>` holds the number and nothing else (the extraction takes the tag content verbatim, so \"5 legs\" fails).\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61a65368",
   "metadata": {},
   "outputs": [],
   "source": "### FILL IN (START) ###\n# Keep the first four lines of the prompt (task statement, <animal_statement> tags).\n# Replace the last line with two instructions:\n#   1. reason step by step inside <thinking> tags\n#   2. put the final answer, the integer alone, inside <answer> tags\nPROMPT_V3 = \"\"\"You will be provided a statement about an animal and your job is to determine how many legs that animal has.\n\nHere is the animal statement.\n<animal_statement>{statement}</animal_statement>\n\nHow many legs does the animal have?\"\"\"\n### FILL IN (END) ###"
  },
  {
   "cell_type": "markdown",
   "id": "9b7de2b5",
   "metadata": {},
   "source": "One item first. The reply should be a `<thinking>` block followed by an `<answer>` block.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6c2fd74",
   "metadata": {},
   "outputs": [],
   "source": [
    "item = eval_data[2]   # the fox\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": PROMPT_V3.format(statement=item[\"animal_statement\"])},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "output = response.choices[0].message.content\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caac7d12",
   "metadata": {},
   "source": "The reply can no longer be compared whole with `\"5\"`. **Regular expression** = a text pattern that finds strings of a given shape; here it takes what sits between the answer tags.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d15e082f",
   "metadata": {},
   "outputs": [],
   "source": "import re\n\nmatch = re.search(r\"<answer>(.*?)</answer>\", output, re.DOTALL)\nextracted = None\nif match:\n    extracted = match.group(1).strip()\n\nprint(match)\nprint(\"extracted:\", extracted)\n"
  },
  {
   "cell_type": "markdown",
   "id": "8168e15e",
   "metadata": {},
   "source": "The extraction is now inside the loop. Target: **at least 11 of 12**.\n\n*Do:* run the cell; edit `PROMPT_V3` and rerun the three cells until the target is reached.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d9fa767",
   "metadata": {},
   "outputs": [],
   "source": "outputs_v3 = []\nfor item in eval_data:\n    messages = [\n        {\"role\": \"user\", \"content\": PROMPT_V3.format(statement=item[\"animal_statement\"])},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    outputs_v3.append(response.choices[0].message.content)\n\ncot_score = 0\nfor item, output in zip(eval_data, outputs_v3):\n    match = re.search(r\"<answer>(.*?)</answer>\", output, re.DOTALL)\n    extracted = None\n    if match:\n        extracted = match.group(1).strip()\n    correct = extracted == item[\"golden_answer\"]\n    cot_score += correct\n    print(correct, \"golden\", item[\"golden_answer\"], \"extracted\", extracted, \"|\", item[\"animal_statement\"])\n\nprint(\"score:\", cot_score, \"/ 12   target 11\")"
  },
  {
   "cell_type": "markdown",
   "id": "cadb860e",
   "metadata": {},
   "source": "The sequence baseline → format fix → reasoning fix, each scored, is how prompts are improved in practice.\n\n*Try:* remove the words restricting `<answer>` to the number alone; the extraction picks up \"5 legs\". Then put the answer tags before the thinking tags; the tricky items return to their 4.4 values (Section 3.2).\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "f04f8aa0",
   "source": "### 4.6 The price of thinking\n\nEach response carries a `usage` field with the billed token counts. **Completion tokens** = the tokens the model generated in its reply. The two cells send the fox item under the direct prompt and under the CoT prompt.\n\n*Do:* run both cells and compare completion tokens.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6382533",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Direct answer\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": PROMPT_V2.format(statement=eval_data[2][\"animal_statement\"])},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "print(response.choices[0].message.content)\n",
    "response.usage.completion_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9454536",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Chain of thought\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": PROMPT_V3.format(statement=eval_data[2][\"animal_statement\"])},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "print(response.choices[0].message.content)\n",
    "response.usage.completion_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c227f6",
   "metadata": {},
   "source": "The accuracy of 4.5 was bought with output tokens. **Test-time compute** = accuracy bought with more generation at answer time, the weights unchanged. Section 6 spends more of it.\n"
  },
  {
   "cell_type": "markdown",
   "id": "268f6be7",
   "metadata": {},
   "source": "## 5. Learning from Examples\n\n**Few-shot prompting** = example question–answer pairs (**exemplars**) in the prompt; the model imitates their format, tone, and procedure. **Zero-shot** = no examples in the prompt.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "149e79cd",
   "source": "### 5.1 The parent bot\n\nAsked without an example, the model answers a child's question like an encyclopedia. One worked example should change the tone without any instruction describing it.\n\n*Do:* run both cells and compare the tone.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de87bfb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Asked cold\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"Will Santa bring me presents on Christmas?\"},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc46fd75",
   "metadata": {},
   "outputs": [],
   "source": [
    "## One worked example in the prompt\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"\"\"Please complete the conversation by writing the next line, speaking as \"A\".\n",
    "Q: Is the tooth fairy real?\n",
    "A: Of course, sweetie. Wrap up your tooth and put it under your pillow tonight. There might be something waiting for you in the morning.\n",
    "Q: Will Santa bring me presents on Christmas?\"\"\"},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a73a4f01",
   "metadata": {},
   "source": "One example fixed tone and length; no instruction described either.\n\n*Try:* rewrite the example answer as a curt parent (`\"A: No. The tooth fairy is a story.\"`) and rerun. Then add a second warm Q/A pair and check whether two examples hold the tone more firmly.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "e6b420ac",
   "source": "### 5.2 Email classification by examples ✍️\n\nCategories: (A) Pre-sale question, (B) Broken or defective item, (C) Billing question, (D) Other. The grader checks the **last character** of the output against the correct letter. The starter prompt does not name the categories, so it scores 0 of 4. Target: **4 of 4**.\n\n*Do:* rewrite `EMAIL_PROMPT` with the category list and a few example emails answered in the same closing line; run and iterate until all four read `PASS`.\n\nHints: two or three example emails (not the ones in `EMAILS`), each answered `The correct category is: B`; end the prompt with `The correct category is:` left open.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "327f9ee9",
   "metadata": {},
   "outputs": [],
   "source": "### FILL IN (START) ###\n# Replace the whole prompt. It must contain:\n#   1. the four categories (A)-(D)\n#   2. two or three example emails, each answered with the line: The correct category is: <letter>\n#   3. the placeholder {email}, then the same line left open: The correct category is:\nEMAIL_PROMPT = \"\"\"Please classify this email as either green or blue: {email}\"\"\"\n### FILL IN (END) ###\n\nEMAILS = [\n    \"Hi -- My Mixmaster4000 is producing a strange noise when I operate it. It also smells a bit smoky and plasticky, like burning electronics.  I need a replacement.\",  # (B) Broken or defective item\n    \"Can I use my Mixmaster 4000 to mix paint, or is it only meant for mixing food?\",  # (A) Pre-sale question OR (D) Other (please explain)\n    \"I HAVE BEEN WAITING 4 MONTHS FOR MY MONTHLY CHARGES TO END AFTER CANCELLING!!  WTF IS GOING ON???\",  # (C) Billing question\n    \"How did I get here I am not good with computer.  Halp.\",  # (D) Other (please explain)\n]\nANSWERS = [[\"B\"], [\"A\", \"D\"], [\"C\"], [\"D\"]]\n\nemail_score = 0\nfor email, accepted in zip(EMAILS, ANSWERS):\n    messages = [\n        {\"role\": \"user\", \"content\": EMAIL_PROMPT.format(email=email)},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    correct = output.strip()[-1] in accepted\n    email_score += correct\n    print(correct, \"expected\", accepted, \"| output:\", output)\n\nprint(\"score:\", email_score, \"/ 4\")"
  },
  {
   "cell_type": "markdown",
   "id": "f1b38020",
   "metadata": {},
   "source": "*Try:* delete the examples and keep only the category list and the closing line. Does the format survive? Then add a fifth email of your own to `EMAILS` and `ANSWERS`.\n\nExemplars whose answers are step-by-step solutions are few-shot CoT; the homework asks you to write them.\n"
  },
  {
   "cell_type": "markdown",
   "id": "7bcb751f",
   "metadata": {},
   "source": "## 6. Self-Consistency\n\n**Self-consistency** = sampling several reasoning paths at nonzero temperature and taking the majority of the extracted answers. Wrong paths scatter; correct paths agree.\n\n**Temperature** = the sampling-randomness parameter of generation; 0.0 always takes the most probable token, 1.0 samples from the model's distribution.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "c5102cca",
   "source": "### 6.1 One sample at temperature 1.0\n\nThe call of 4.5 on the fox with `temperature=1.0`.\n\n*Do:* run the cell three or four times and note the extracted answers.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "140ff9db",
   "metadata": {},
   "outputs": [],
   "source": "HARD = eval_data[2]   # the fox that lost a leg and regrew two\n\nmessages = [\n    {\"role\": \"user\", \"content\": PROMPT_V3.format(statement=HARD[\"animal_statement\"])},\n]\nresponse = client.chat.completions.create(model=MODEL, messages=messages, temperature=1.0)\noutput = response.choices[0].message.content\nmatch = re.search(r\"<answer>(.*?)</answer>\", output, re.DOTALL)\nextracted = None\nif match:\n    extracted = match.group(1).strip()\n\nprint(output)\nprint(\"extracted:\", extracted)"
  },
  {
   "cell_type": "markdown",
   "id": "b527e46f",
   "metadata": {},
   "source": "At temperature 1.0 the path differs from run to run, and on a tricky item the answer sometimes differs too.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "b3bb7a8e",
   "source": "### 6.2 Five samples\n\nThe loop repeats 6.1 five times and keeps the extracted answers.\n\n*Do:* run the cell.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e51d2ad9",
   "metadata": {},
   "outputs": [],
   "source": "N_SAMPLES = 5\n\nsamples = []\nfor i in range(N_SAMPLES):\n    messages = [\n        {\"role\": \"user\", \"content\": PROMPT_V3.format(statement=HARD[\"animal_statement\"])},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=1.0)\n    output = response.choices[0].message.content\n    match = re.search(r\"<answer>(.*?)</answer>\", output, re.DOTALL)\n    if match:\n        samples.append(match.group(1).strip())\n\nsamples"
  },
  {
   "cell_type": "markdown",
   "id": "d42b0f14",
   "metadata": {},
   "source": "### 6.3 The vote\n\n`Counter` tallies the samples; the most common value is the answer.\n\n*Do:* run the cell and compare the majority with the golden answer.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8c8c02c",
   "metadata": {},
   "outputs": [],
   "source": "from collections import Counter\n\nvotes = Counter(samples)\nmajority = None\nif votes:\n    majority = votes.most_common(1)[0][0]\n\nprint(\"votes:\", dict(votes))\nprint(\"majority:\", majority, \"  golden:\", HARD[\"golden_answer\"])\n"
  },
  {
   "cell_type": "markdown",
   "id": "afb96560",
   "metadata": {},
   "source": "One sample stands or falls with one path; the vote averages over paths at the price of five calls (test-time compute).\n\n*Try:* set `temperature=0.0` in 6.2 and rerun 6.2 and 6.3; the five samples collapse to one path. Restore 1.0 and set `N_SAMPLES = 9`. Then point `HARD` at `eval_data[0]`: on an item every path gets right, the vote is five calls for nothing.\n"
  },
  {
   "cell_type": "markdown",
   "id": "b88db902",
   "metadata": {},
   "source": "## 7. The Limit of Written Reasoning\n\n**Hallucination** = generating content that is plausible but not factual. Chain-of-thought repairs computation, not knowledge. When the model lacks a fact, writing more does not produce it; it invents a plausible premise and reasons on top of it. The question below has no reliable answer in any source.\n\n*Do:* run the three cells and compare what each prompt produces.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd1c1ea7",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Asked cold\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"Who is the heaviest hippo of all time?\"},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ba9a60d",
   "metadata": {},
   "outputs": [],
   "source": [
    "## With chain of thought\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"Who is the heaviest hippo of all time? \"\n",
    "                                \"Think step by step in <thinking> tags first, then give your answer.\"},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6e95f1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "## With an out\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": \"Who is the heaviest hippo of all time? \"\n",
    "                                \"Only answer if you know the answer with certainty.\"},\n",
    "]\n",
    "response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d61be6ce",
   "metadata": {},
   "source": "The chain-of-thought version reasons in good form toward a name it cannot know. **Giving an out** = an instruction that makes declining an acceptable answer. Checking a premise against the world needs a tool (W3).\n\n*Try:* replace the out with `\"If you are not sure, answer exactly: I don't know.\"` Then ask, with the same out, a question the model does know (\"What is the largest land animal alive today?\") and confirm the out does not suppress a known answer.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "18cdf1f3",
   "source": "## 8. In-class check\n\nThe two fill-ins of this lab, 4.5 and 5.2, are practice; nothing from this notebook is submitted. The cell prints their state.\n\n*Do:* run the cell.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "id": "5f7307b4",
   "execution_count": null,
   "outputs": [],
   "source": "checks = {\n    \"4.5 chain-of-thought prompt reaches >= 11/12\": cot_score >= 11,\n    \"5.2 email prompt reaches 4/4\":                  email_score == 4,\n}\n\nfor item, ok in checks.items():\n    if ok:\n        print(\"PASS \", item)\n    else:\n        print(\"FAIL \", item)\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "e272ae57",
   "source": "---\n\nHomework: `W2_hw_prompting.ipynb`, collected, due before the W3 class.\n\nW3: plain Python functions become tools the model can call.\n"
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [
    {
     "file_id": "https://github.com/ralbu85/stml_2026/blob/main/lectures/week02/W2_lab_prompting.ipynb"
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}