{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "864bba81",
   "source": "# W2 Homework — Prompting Techniques (graded)\n\n[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ralbu85/stml_2026/blob/main/lectures/week02/W2_hw_prompting.ipynb)\n\n**Goal.** Apply the four techniques of the W2 lab to new tasks: chain-of-thought by instruction, format pinned by a worked example, self-consistency written by you, and chain-of-thought by worked examples.\n\nSections: setup (1) → assignment 1, instruction (2) → assignment 2, exemplar (3) → assignment 3, the vote (4) → assignment 4, worked solutions (5) → completion and submission (6).\n\nEach assignment has one block marked `### FILL IN (START) ###` … `### FILL IN (END) ###` and a fixed check below it. Only the block changes. The check prints, for each test item, whether it passed, then one summary line. Grading reads the saved outputs, so run the notebook top to bottom before submitting.\n\n**This homework is collected.** Fill in Section 1.3, make every check pass, and follow Section 6 to submit to the LMS. Due date: see the LMS.\n"
  },
  {
   "cell_type": "markdown",
   "id": "a005c355",
   "metadata": {},
   "source": "## 1. Setup\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "058b9f43",
   "source": "### 1.1 Installation\n\n`aisuite` runs the same code on OpenAI or Anthropic keys.\n\n*Do:* run the cell (about 30 seconds, once per session).\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4922a6f8",
   "metadata": {},
   "outputs": [],
   "source": "%pip install -q \"aisuite[openai,anthropic]\""
  },
  {
   "cell_type": "markdown",
   "id": "addaabad",
   "metadata": {},
   "source": "### 1.2 API key and model\n\n**API key** = the secret string that identifies your account and bills usage to it (issuing steps: the API Setup guide on the course site). Do not share the notebook with the key inside.\n\n*Do:* replace `PASTE-YOUR-KEY-HERE` with your key and run the cell. Nothing prints.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35a681eb",
   "metadata": {},
   "outputs": [],
   "source": "import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"PASTE-YOUR-KEY-HERE\"\n\nMODEL = \"openai:gpt-4o-mini\"   # Anthropic accounts: MODEL = \"anthropic:claude-haiku-4-5\" and set ANTHROPIC_API_KEY instead"
  },
  {
   "cell_type": "markdown",
   "id": "0ba581b5",
   "metadata": {},
   "source": "### 1.3 Submission identity\n\nThe completion check in Section 6 prints these values; a blank name fails it.\n\n*Do:* fill in your name and student ID, run the cell.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66b9325c",
   "metadata": {},
   "outputs": [],
   "source": "STUDENT_NAME = \"\"\nSTUDENT_ID = \"\"\n\nprint(\"submitting as:\", STUDENT_NAME, STUDENT_ID)\n"
  },
  {
   "cell_type": "markdown",
   "id": "965ed114",
   "metadata": {},
   "source": "### 1.4 Client and a first call\n\nThe same call that opened W1. Every later cell repeats this shape; only the prompt string changes.\n\n*Do:* run the cell. The output must be exactly `ready`.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "307efc8d",
   "metadata": {},
   "outputs": [],
   "source": "import aisuite\n\nclient = aisuite.Client()\n\nmessages = [{\"role\": \"user\", \"content\": \"Reply with exactly: ready\"}]\nresponse = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\nresponse.choices[0].message.content"
  },
  {
   "cell_type": "markdown",
   "id": "2549798b",
   "metadata": {},
   "source": "Any error here is a setup problem: recheck the API Setup guide before continuing.\n"
  },
  {
   "cell_type": "markdown",
   "id": "70bc8093",
   "metadata": {},
   "source": "## 2. Assignment 1 — An instruction that elicits the solution ✍️\n\n`COT_INSTRUCTION` is appended after each word problem. The starter is empty, so the extraction finds no `ANSWER:` line. Both problems must pass.\n\n*Do:* write the instruction, run, iterate until both lines read `PASS`.\n\nHints: demand the solution step by step and the exact final line `ANSWER: <number>`.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6d98e3d",
   "metadata": {},
   "outputs": [],
   "source": "import re\n\nWORD_PROBLEMS = [\n    (\"A library has 4 shelves with 38 books each. During the day 47 books are checked out and 26 are returned. How many books are on the shelves now?\", \"131\"),\n    (\"A bakery bakes 12 trays of 24 muffins. 6 trays are sold whole and 37 more muffins are sold singly. How many muffins remain?\", \"107\"),\n]\n\n### FILL IN (START) ###\n# Write the instruction appended after each problem: demand the solution step by step\n# and the exact last line ANSWER: <number>\nCOT_INSTRUCTION = \"\"\n### FILL IN (END) ###\n\na1_passed = 0\nfor question, expected in WORD_PROBLEMS:\n    messages = [\n        {\"role\": \"user\", \"content\": question + \"\\n\" + COT_INSTRUCTION},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    match = re.search(r\"ANSWER:\\s*(\\d+)\", output)\n    extracted = None\n    if match:\n        extracted = match.group(1)\n    correct = extracted == expected\n    a1_passed += correct\n    print(correct, \"extracted\", extracted, \"expected\", expected, \"| output ended:\", output.strip()[-60:])\n\nassignment_1 = a1_passed == len(WORD_PROBLEMS)\nprint(\"assignment 1:\", a1_passed, \"/\", len(WORD_PROBLEMS))\n"
  },
  {
   "cell_type": "markdown",
   "id": "2dfa2b8d",
   "metadata": {},
   "source": "## 3. Assignment 2 — An exemplar that pins the format ✍️\n\nThe check accepts exactly one output shape per note: `DATE: 2026-03-10`, nothing before or after. Instructions leave something loose; a worked exemplar shows the shape. Both notes must pass.\n\n*Do:* add one or two worked note → answer exemplars inside `DATE_PROMPT` above the test note; iterate until both lines read `PASS`.\n\nHints: format each exemplar exactly as the output should look (a note about June 9th, 2026 answered `DATE: 2026-06-09` on its own line); keep the test note last. Exemplar notes must not be the test notes.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5189038d",
   "metadata": {},
   "outputs": [],
   "source": "TEST_NOTES = [\n    (\"Team offsite moved from March 3rd to March 10th, 2026.\", \"DATE: 2026-03-10\"),\n    (\"Deadline for the grant report extended from April 1st to April 22nd, 2026.\", \"DATE: 2026-04-22\"),\n]\n\n### FILL IN (START) ###\n# Insert one or two worked exemplars between the first line and the final Note line,\n# each as:  Note: <a note with two dates>  /  DATE: YYYY-MM-DD\nDATE_PROMPT = \"\"\"Extract the final date from the note.\n\nNote: {note}\"\"\"\n### FILL IN (END) ###\n\na2_passed = 0\nfor note, expected in TEST_NOTES:\n    messages = [\n        {\"role\": \"user\", \"content\": DATE_PROMPT.format(note=note)},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    correct = output.strip() == expected\n    a2_passed += correct\n    print(correct, \"got\", output.strip(), \"| expected\", expected)\n\nassignment_2 = a2_passed == len(TEST_NOTES)\nprint(\"assignment 2:\", a2_passed, \"/\", len(TEST_NOTES))\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "a04987f8",
   "source": "## 4. Assignment 3 — Self-consistency, written by you ✍️\n\nA statement the direct prompt tends to miss: a spider that lost three legs, golden answer 5. Two things are yours: the chain-of-thought prompt and the sampling loop.\n\n*Do:* (1) write `COT_PROMPT`, a template with the `{statement}` placeholder that makes the model reason inside `<thinking>` tags and put the integer alone inside `<answer>` tags, the prompt shape of lab Section 4.5; (2) write the loop that calls it five times at `temperature=1.0` and appends the text between the `<answer>` tags of each reply to `samples`. The vote below is given. Iterate until the line reads `PASS`.\n\nHints: the loop is the cell of lab Section 6.2 with the statement replaced. Keep `temperature=1.0`; at 0.0 the five samples are five copies of one path. The check requires at least five samples and a majority equal to the golden answer.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "id": "c66e7cb7",
   "execution_count": null,
   "outputs": [],
   "source": "from collections import Counter\n\nTRICKY = {\"animal_statement\": \"The animal is a spider that lost three legs.\", \"golden_answer\": \"5\"}\n\n### FILL IN (START) ###\n# 1. COT_PROMPT: keep the {statement} placeholder; replace the last line with two instructions:\n#    reason step by step inside <thinking> tags, then put the final answer, the integer alone,\n#    inside <answer> tags.\nCOT_PROMPT = \"\"\"You will be provided a statement about an animal and your job is to determine how many legs that animal has.\n\nHere is the animal statement.\n<animal_statement>{statement}</animal_statement>\n\nHow many legs does the animal have?\"\"\"\n\n# 2. The sampling loop: five calls of COT_PROMPT on TRICKY[\"animal_statement\"] at temperature=1.0;\n#    append the text between the <answer> tags of each reply to samples.\nsamples = []\n### FILL IN (END) ###\n\nvotes = Counter(samples)\nmajority = None\nif votes:\n    majority = votes.most_common(1)[0][0]\nassignment_3 = len(samples) >= 5 and majority == TRICKY[\"golden_answer\"]\n\nprint(\"samples:\", samples)\nprint(\"majority:\", majority, \"  golden:\", TRICKY[\"golden_answer\"], \"  samples:\", len(samples))\nif assignment_3:\n    print(\"PASS  assignment 3\")\nelse:\n    print(\"FAIL  assignment 3\")\n"
  },
  {
   "cell_type": "markdown",
   "id": "e26715c7",
   "metadata": {},
   "source": "## 5. Assignment 4 — Worked examples that elicit the solution ✍️\n\nSix arithmetic expressions in Roman numerals. The check requires the right number and a **last line** of exactly `ANSWER: <number>`. The starter gives the instruction alone, and the last-line check tends to fail. Target: **at least 5 of 6**; no evalset expression may appear in the prompt (the check reports a leak).\n\n*Do:* write `ROMAN_PROMPT` as few-shot CoT: at least two worked examples, each converting the numerals, computing, and closing with `ANSWER: <number>` on its own line, then the test expression laid out the same way. Iterate until `PASS`.\n\nHints: choose exemplars not in `ROMAN_EVALSET` (for instance `XII + VII`); write each solution the way the model's should look; keep `{expression}` last.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51d47b2f",
   "metadata": {},
   "outputs": [],
   "source": "import re\n\nROMAN_EVALSET = [\n    (\"XIV + IX\", \"23\"),\n    (\"XLII - XXIX\", \"13\"),\n    (\"CD + XC\", \"490\"),\n    (\"MMXXVI - MCMXCIV\", \"32\"),\n    (\"XCIX + I\", \"100\"),\n    (\"LXXX / XVI\", \"5\"),\n]\n\n### FILL IN (START) ###\n# Insert at least two worked examples between the first line and the {expression} line.\n# Each example: the expression, the numerals converted, the computation, then a line ANSWER: <number>\n# Do not use any expression from ROMAN_EVALSET.\nROMAN_PROMPT = \"\"\"Evaluate the expression written in Roman numerals. Give the result as ANSWER: <number>.\n\n{expression}\"\"\"\n### FILL IN (END) ###\n\nleaked = []\nfor expression, expected in ROMAN_EVALSET:\n    if expression in ROMAN_PROMPT:\n        leaked.append(expression)\nn_exemplars = len(re.findall(r\"^ANSWER:\\s*\\d+\\s*$\", ROMAN_PROMPT, re.M))   # answer lines of the worked examples\n\nroman_score = 0\nfor expression, expected in ROMAN_EVALSET:\n    messages = [\n        {\"role\": \"user\", \"content\": ROMAN_PROMPT.format(expression=expression)},\n    ]\n    response = client.chat.completions.create(model=MODEL, messages=messages, temperature=0.0)\n    output = response.choices[0].message.content\n    last_line = output.strip().splitlines()[-1].strip()\n    correct = last_line == \"ANSWER: \" + expected\n    roman_score += correct\n    print(correct, expression, \"| last line:\", last_line)\n\nassignment_4 = roman_score >= 5 and n_exemplars >= 2 and not leaked\nprint(\"assignment 4:\", roman_score, \"/ 6   exemplars:\", n_exemplars, \"  leaked:\", leaked)\nif assignment_4:\n    print(\"PASS  assignment 4\")\nelse:\n    print(\"FAIL  assignment 4\")\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "id": "44d6d2af",
   "source": "## 6. Completion and Submission\n\nCompletion: the identity line is filled and all four assignments pass. Grading checks these facts from the saved outputs and compares each fill-in block with the starter; it never grades prose.\n\n*Do:* run the notebook top to bottom once more so every output is saved, confirm every row below reads `PASS`, download the notebook (**File → Download → Download .ipynb**), and upload that file to the LMS assignment. Keep the file name and the check cells as they are.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "id": "b176ee90",
   "execution_count": null,
   "outputs": [],
   "source": "completion = {\n    \"name and student ID filled in (1.3)\":            bool(STUDENT_NAME.strip()) and bool(STUDENT_ID.strip()),\n    \"assignment 1: instruction elicits ANSWER on both problems\": assignment_1,\n    \"assignment 2: exemplar pins the format on both notes\":      assignment_2,\n    \"assignment 3: the vote finds the answer\":                   assignment_3,\n    \"assignment 4: worked examples reach >= 5/6\":                assignment_4,\n}\n\nprint(\"submitted by:\", STUDENT_NAME, STUDENT_ID)\nprint()\nfor item, ok in completion.items():\n    if ok:\n        print(\"PASS \", item)\n    else:\n        print(\"FAIL \", item)\n\nif all(completion.values()):\n    print(\"\\nHOMEWORK COMPLETE — download and submit to the LMS\")\nelse:\n    print(\"\\nNOT COMPLETE YET\")\n"
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [
    {
     "file_id": "https://github.com/ralbu85/stml_2026/blob/main/lectures/week02/W2_lab_prompting.ipynb"
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}