{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# W7 Homework — Your Topic Through the Workflow\n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ralbu85/stml_2026/blob/main/lectures/week07/W7_hw_own_topic.ipynb)\n",
    "\n",
    "**Goal.** Run the research workflow on a topic from your own research interests,\n",
    "extend its checklist with two checks of your own design, and submit the scored\n",
    "report. Nothing new to build — the first half exercised in your own domain.\n",
    "\n",
    "The path: setup → the condensed workflow (plan → research → draft → edit) → your\n",
    "topic ✍️ → your two checks ✍️ → the scored run → completion.\n",
    "\n",
    "*Runtime:* ~40 minutes plus reading your own report. Due before the W8 midterm.\n",
    "Reference answers: `labs/checkpoints/week07/solution.py`, published after the homework deadline.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 1. Setup\n\n*Do:* run the three cells; the last must print `ready`.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "%pip install -q \"aisuite[openai,anthropic]\"\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"PASTE-YOUR-KEY-HERE\"\n\nMODEL = \"openai:gpt-4o-mini\"          # alt: \"anthropic:claude-haiku-4-5\"\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "import aisuite\n\nclient = aisuite.Client()\n\n\ndef ask(prompt, system=None, temperature=0.0, **kwargs):\n    \"\"\"Single prompt -> reply text.\"\"\"\n    messages = ([{\"role\": \"system\", \"content\": system}] if system else []) + [\n        {\"role\": \"user\", \"content\": prompt}]\n    response = client.chat.completions.create(model=MODEL, messages=messages,\n                                              temperature=temperature, **kwargs)\n    return response.choices[0].message.content\n\n\nprint(ask(\"Reply with exactly: ready\"))\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 2. The Workflow, Condensed\n\nThe lab's four stages in one pass: a planner decomposes the topic, a research call\ngathers sources (keyless arXiv, with an offline fallback so the notebook runs\nanywhere), a writer drafts, and the editor critiques and revises. All prompts are\nthe published reference versions — the homework's fill-ins are the topic and the\nchecks, not the machinery.\n\n*Do:* run the cell; nothing prints yet — the run happens in Section 4.\n"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import json\n",
    "import re\n",
    "\n",
    "import requests\n",
    "import xml.etree.ElementTree as ET\n",
    "\n",
    "MOCK_SOURCES = [\n",
    "    {\"title\": \"A Survey of LLM Agents\", \"url\": \"https://arxiv.org/abs/2401.00001\"},\n",
    "    {\"title\": \"Measuring Agent Reliability\", \"url\": \"https://arxiv.org/abs/2402.00002\"},\n",
    "    {\"title\": \"Tool-Use Benchmarks\", \"url\": \"https://arxiv.org/abs/2403.00003\"},\n",
    "]\n",
    "\n",
    "\n",
    "def arxiv_search(query: str, max_results: int = 4) -> list:\n",
    "    \"\"\"Keyless arXiv search; falls back to a mock corpus when offline.\"\"\"\n",
    "    try:\n",
    "        r = requests.get(\"http://export.arxiv.org/api/query\",\n",
    "                         params={\"search_query\": f\"all:{query}\",\n",
    "                                 \"max_results\": max_results}, timeout=15)\n",
    "        atom = \"{http://www.w3.org/2005/Atom}\"\n",
    "        out = [{\"title\": e.find(atom + \"title\").text.strip(),\n",
    "                \"url\": e.find(atom + \"id\").text.strip()}\n",
    "               for e in ET.fromstring(r.text).findall(atom + \"entry\")]\n",
    "        if not out:\n",
    "            print(\"NOTE: arXiv returned nothing for this query — using mock \"\n",
    "                  \"fallback sources; broaden or rephrase your topic.\")\n",
    "        return out or MOCK_SOURCES\n",
    "    except Exception:\n",
    "        print(\"NOTE: arXiv unreachable — using mock fallback sources.\")\n",
    "        return MOCK_SOURCES\n",
    "\n",
    "\n",
    "PLANNER_PROMPT = (\n",
    "    \"Decompose the research topic into 3 ordered steps: one research step, one \"\n",
    "    \"drafting step, one editing step. Reply with a JSON list of step strings only.\\n\"\n",
    "    \"Topic: {topic}\")\n",
    "\n",
    "WRITER_PROMPT = (\n",
    "    \"Write a research report in Markdown on the topic below, using ONLY the \"\n",
    "    \"sources given. Requirements: '## ' section headings; every claim cites a \"\n",
    "    \"source with its full URL; a closing section naming open questions.\\n\\n\"\n",
    "    \"Topic: {topic}\\n\\nSources:\\n{sources}\")\n",
    "\n",
    "EDITOR_CRITIQUE_PROMPT = (\n",
    "    \"You are an editor for research reports. Return numbered, actionable feedback \"\n",
    "    \"on the report below, judged against structure (clear Markdown sections), \"\n",
    "    \"grounding (claims tied to cited URLs), and length (concise, no padding). \"\n",
    "    \"Do not rewrite.\\n\\nReport:\\n{report}\")\n",
    "\n",
    "EDITOR_REVISE_PROMPT = (\n",
    "    \"Revise the report to address every numbered point in the feedback. Keep all \"\n",
    "    \"citations with full URLs. Return the complete revised report only.\\n\\n\"\n",
    "    \"Report:\\n{report}\\n\\nFeedback:\\n{feedback}\")\n",
    "\n",
    "\n",
    "def run_workflow(topic, verbose=True):\n",
    "    \"\"\"Topic -> (plan, revised report). Four model calls, one search.\n",
    "\n",
    "    The plan is recorded for your report, not dispatched — the routing executor\n",
    "    lives in the lab; this condensed pipeline is fixed.\n",
    "    \"\"\"\n",
    "    raw_plan = ask(PLANNER_PROMPT.format(topic=topic))\n",
    "    steps = json.loads(re.sub(r\"^```[a-z]*\\s*|\\s*```$\", \"\", raw_plan.strip()))\n",
    "    sources = arxiv_search(topic)\n",
    "    source_text = \"\\n\".join(f\"- {s['title']} — {s['url']}\" for s in sources)\n",
    "    draft = ask(WRITER_PROMPT.format(topic=topic, sources=source_text))\n",
    "    feedback = ask(EDITOR_CRITIQUE_PROMPT.format(report=draft))\n",
    "    revised = ask(EDITOR_REVISE_PROMPT.format(report=draft, feedback=feedback))\n",
    "    if verbose:\n",
    "        print(\"PLAN:\", steps, \"\\n\")\n",
    "        print(\"FEEDBACK:\", feedback[:300], \"...\\n\")\n",
    "    return steps, revised\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Your Topic and Your Checks ✍️\n",
    "\n",
    "Two fill-ins.\n",
    "\n",
    "**The topic.** Replace the starter with a topic from your own research area —\n",
    "specific enough that the sources differ from your neighbor's (\"reward hacking in\n",
    "RLHF for code models\", not \"AI\").\n",
    "\n",
    "**Two checks of your own.** The given checklist verifies structure. Add two named\n",
    "binary checks that verify what *you* care about in a report on your topic.\n",
    "Requirements: each is `(\"name\", lambda report: <bool>)`; each must be checkable\n",
    "from the report text alone (notes Ch. 5: a critique question answerable from the\n",
    "output is what qualifies).\n",
    "\n",
    "Hints — shapes that work: a term that must appear (\"mentions at least one named\n",
    "benchmark\": `lambda r: bool(re.search(r\"bench\", r, re.I))`); a count (\"cites at\n",
    "least 3 distinct URLs\"); a structural fact (\"has an open-questions section\").\n",
    "\n",
    "A good check is one a competent report on your topic would pass — when a check\n",
    "fails, decide whether the prompt or the check is at fault before changing either.\n",
    "If the scored run prints the mock-fallback notice, your query found nothing on\n",
    "arXiv; broaden the topic phrasing.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "STARTER_TOPIC = \"test-time compute for LLM agents\"\n\n### FILL IN (START) ###\nTOPIC = STARTER_TOPIC          # starter — replace with your own research topic\n\nMY_CHECKS = [\n    # (\"name of the check\", lambda report: bool(...)),\n]\n### FILL IN (END) ###\n\nBASE_CHECKS = [\n    (\"report is substantial (>= 800 chars)\", lambda r: len(r) >= 800),\n    (\"has Markdown section headings\", lambda r: \"## \" in r),\n    (\"cites >= 2 distinct URLs\",\n     lambda r: len(set(re.findall(r\"https?://\\S+\", r))) >= 2),\n]\nCHECKS = BASE_CHECKS + list(MY_CHECKS)\nprint(f\"{len(CHECKS)} checks registered\")\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. The Scored Run\n",
    "\n",
    "One full pass on your topic, then every check, then the judge (a variant of the lab's rubric:\n",
    "grounding, structure, concision — 1 to 5). Target: **all checks PASS and judge ≥ 4**.\n",
    "\n",
    "*Do:* run the cell, then read your own report against the feedback — would you\n",
    "sign it?\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "plan_steps, report = run_workflow(TOPIC)\n\ncheck_results = {name: bool(fn(report)) for name, fn in CHECKS}\nfor name, ok in check_results.items():\n    print(f\"{'PASS' if ok else 'FAIL':4}  {name}\")\n\nJUDGE_PROMPT = (\n    \"Grade the research report on three criteria - grounding (claims tied to \"\n    \"cited URLs), structure (clear sections), concision. One sentence of \"\n    \"justification, then exactly 'SCORE: <n>' with n from 1 to 5.\\n\\nReport:\\n{report}\")\njudge_reply = ask(JUDGE_PROMPT.format(report=report))\nmatch = re.search(r\"SCORE:\\s*(\\d)\", judge_reply)\njudge_score = int(match.group(1)) if match else 0\nprint(f\"\\njudge: {judge_score}/5\")\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Completion Check\n",
    "\n",
    "Submit: run the notebook top to bottom, then **File → Download → Download .ipynb**\n",
    "and upload the file to the LMS.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "completion = {\n    \"own topic set (starter replaced)\": TOPIC != STARTER_TOPIC,\n    \"two own checks added\": len(MY_CHECKS) >= 2,\n    \"all checks PASS\": all(check_results.values()),\n    \"judge score >= 4\": judge_score >= 4,\n}\nfor item, ok in completion.items():\n    print(f\"{'PASS' if ok else 'FAIL':4}  {item}\")\nprint(\"\\nHOMEWORK COMPLETE\" if all(completion.values()) else \"\\nNOT COMPLETE YET\")\n"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}