{ "cells": [ { "cell_type": "markdown", "id": "c00", "metadata": {}, "source": [ "# W6 Lab — One Agent or a Team of Agents?\n", "\n", "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ralbu85/stml_2026/blob/main/lectures/week06/W6_lab_multiagent.ipynb)\n", "\n", "**Goal.** In Week 5, one agent wrote a science essay, evaluated it, and improved it. This lab divides the same work between two roles, a writer and an evaluator, and then adds a coordinator. At the end, it checks whether the essay of the team is better than the essay of the single agent, and what each one cost.\n", "\n", "Each role is one object of the `Agent` class from the practice notebook (v2). The class gains one thing this week: `total_tokens`, the tokens that each object used. A multi-agent system is several objects plus the code, or the agent, that passes messages between them.\n", "\n", "| Part | Question |\n", "|---|---|\n", "| 1. The single agent of Week 5 | What does one agent produce alone, and what does it cost? |\n", "| 2. Two roles | How do a writer and an evaluator pass the essay and the evaluation to each other? |\n", "| 3. A coordinator agent | Can an agent run the team itself, with the roles as its tools? |\n", "| 4. Is the team better? | Same question, same judge: scores, citations, and tokens |\n", "| 5. Your own words | What did each part show? |\n", "\n", "**Cell labels.** `# ✍️ TYPE` = write the code between the `FILL IN` lines yourself; `# ✍️ WRITE` = answer in words; `# ▶ RUN · example` = read, then run; `# ▶ RUN · setup` / `check` = just run.\n", "\n", "About 70 minutes. Not collected. The search results come from Wikipedia and the model replies come from your own run, so your text will differ from the examples." ] }, { "cell_type": "markdown", "id": "c01", "metadata": {}, "source": [ "## 0. Setup\n", "\n", "*Do:* run the four cells; paste your API key into the second. The last cell defines `show_messages` and the `Agent` class of the Week 5 lab (`json_mode=True` for JSON replies, `temperature=0` so repeated runs give similar results), with one addition for this week: `total_tokens`, the tokens that each object used." ] }, { "cell_type": "code", "execution_count": null, "id": "c02", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · setup: just run\n", "%pip install -q \"openai==1.109.1\"" ] }, { "cell_type": "code", "execution_count": null, "id": "c03", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · setup: just run\n", "import os\n", "\n", "OPENAI_API_KEY = \"PASTE-YOUR-KEY-HERE\" # paste your key between the quotes\n", "if OPENAI_API_KEY != \"PASTE-YOUR-KEY-HERE\":\n", " os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY # the client reads the key from here" ] }, { "cell_type": "code", "execution_count": null, "id": "c04", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · setup: just run\n", "import json\n", "from openai import OpenAI\n", "\n", "client = OpenAI()\n", "MODEL = \"gpt-4.1\"\n", "done = {} # the check cells record your results here\n", "messages = [{\"role\": \"user\", \"content\": \"Reply with exactly: ready\"}]\n", "response = client.chat.completions.create(model=MODEL, messages=messages)\n", "reply = response.choices[0].message.model_dump(exclude_none=True)\n", "messages.append(reply)\n", "reply[\"content\"]" ] }, { "cell_type": "code", "execution_count": null, "id": "c05", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · setup: just run\n", "# The course helpers and class, as in the Week 5 lab, with one addition: total_tokens.\n", "import inspect\n", "\n", "PYTHON_TO_JSON = {str: \"string\", int: \"integer\", float: \"number\", bool: \"boolean\"}\n", "\n", "def tool_from_function(function):\n", " \"\"\"Build a tool schema from a function's name, docstring, and type annotations.\"\"\"\n", " doc = inspect.getdoc(function)\n", " arg_help = {}\n", " for line in doc.split(\"\\n\"):\n", " line = line.strip()\n", " if \":\" in line and not line.endswith(\":\"):\n", " name, text = line.split(\":\", 1)\n", " arg_help[name.strip()] = text.strip()\n", " properties = {}\n", " for name, parameter in inspect.signature(function).parameters.items():\n", " properties[name] = {\"type\": PYTHON_TO_JSON[parameter.annotation], \"description\": arg_help[name]}\n", " return {\"type\": \"function\",\n", " \"function\": {\"name\": function.__name__, \"description\": doc.split(\"\\n\")[0],\n", " \"parameters\": {\"type\": \"object\", \"properties\": properties, \"required\": list(properties)}}}\n", "\n", "\n", "def show_messages(messages):\n", " \"\"\"Print each message: position, role, and content.\"\"\"\n", " for i in range(len(messages)):\n", " m = messages[i]\n", " if \"tool_calls\" in m: # a request: print each requested function\n", " if \"content\" in m and m[\"content\"] != \"\": # text that arrived together with the requests\n", " print(i, m[\"role\"], m[\"content\"])\n", " for request in m[\"tool_calls\"]:\n", " print(i, m[\"role\"], request[\"function\"])\n", " else:\n", " print(i, m[\"role\"], m[\"content\"])\n", " print()\n", "\n", "\n", "class Agent:\n", " \"\"\"A model with its own message list and, optionally, tools.\"\"\"\n", "\n", " def __init__(self, system, tool_functions=(), json_mode=False):\n", " self.messages = [{\"role\": \"system\", \"content\": system}] # SYSTEM message, once\n", " self.tools = [] # TOOL SCHEMAS\n", " self.functions = {} # name -> Python function\n", " for function in tool_functions: # () = no tools\n", " self.tools.append(tool_from_function(function))\n", " self.functions[function.__name__] = function\n", " self.response_format = {\"type\": \"text\"}\n", " if json_mode: # Week 5: every reply is valid JSON\n", " self.response_format = {\"type\": \"json_object\"}\n", " self.total_tokens = 0 # NEW this week: the tokens this object used\n", "\n", " def call_model(self):\n", " \"\"\"Call the model with the list; keep the reply and return it.\"\"\"\n", " if len(self.tools) > 0:\n", " response = client.chat.completions.create(model=MODEL, messages=self.messages, tools=self.tools,\n", " response_format=self.response_format, temperature=0)\n", " else: # no tools: the API rejects an empty list\n", " response = client.chat.completions.create(model=MODEL, messages=self.messages,\n", " response_format=self.response_format, temperature=0)\n", " self.total_tokens = self.total_tokens + response.usage.total_tokens # NEW this week\n", " reply = response.choices[0].message.model_dump(exclude_none=True)\n", " self.messages.append(reply) # ASSISTANT message\n", " return reply\n", "\n", " def run_tools(self, reply):\n", " \"\"\"Run each request in the reply and add each result as a tool message.\"\"\"\n", " for request in reply[\"tool_calls\"]:\n", " name = request[\"function\"][\"name\"]\n", " try:\n", " args = json.loads(request[\"function\"][\"arguments\"])\n", " result = self.functions[name](**args)\n", " except Exception as error: # unknown name or wrong arguments\n", " result = \"error: \" + str(error) # the error becomes the result\n", " self.messages.append({\"role\": \"tool\", \"tool_call_id\": request[\"id\"], \"content\": json.dumps(result)}) # TOOL message\n", "\n", " def ask(self, question, max_steps=6):\n", " \"\"\"Add the question, then loop until the model answers in text.\"\"\"\n", " self.messages.append({\"role\": \"user\", \"content\": question}) # USER message\n", " for step in range(max_steps):\n", " reply = self.call_model()\n", " if \"tool_calls\" in reply: # requests: run the tools\n", " self.run_tools(reply)\n", " else: # a text answer: done\n", " return reply[\"content\"]\n", " print(\"Stopped: no text answer after\", max_steps, \"steps\")\n", " return None # the step limit ended the loop\n", "\n", " def show(self):\n", " \"\"\"Print this object's messages.\"\"\"\n", " show_messages(self.messages)" ] }, { "cell_type": "markdown", "id": "c06", "metadata": {}, "source": [ "## 1. The single agent of Week 5\n", "\n", "This is the agent of the Week 5 lab: one object with the role of a science writer, the Wikipedia search as its tool, and the steps of reflection in its system message. The search function has one new line: it records every title that a search returns in `FOUND_TITLES`, for the citation check in Part 4.\n", "\n", "*Do:* run the two cells. The second cell prints the first line of every reply, the final essay, and the tokens." ] }, { "cell_type": "code", "execution_count": null, "id": "c07", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · example: read each line, then run\n", "import requests\n", "import html\n", "\n", "FOUND_TITLES = [] # every article title that a search returned in this notebook\n", "\n", "def wikipedia_search(query: str):\n", " \"\"\"Search Wikipedia and return the top three results: title and a short snippet.\n", "\n", " Args:\n", " query: Search words, for example \"black hole\".\n", " \"\"\"\n", " try:\n", " data = requests.get(\"https://en.wikipedia.org/w/api.php\",\n", " params={\"action\": \"query\", \"list\": \"search\", \"srsearch\": query,\n", " \"srlimit\": 3, \"format\": \"json\"},\n", " headers={\"User-Agent\": \"STML-course-lab/1.0\"}, timeout=10).json()\n", " lines = []\n", " for item in data[\"query\"][\"search\"]:\n", " snippet = item[\"snippet\"].replace('', \"\").replace(\"\", \"\") # drop highlight tags\n", " lines.append(item[\"title\"] + \" | \" + html.unescape(snippet)) # ' -> '\n", " FOUND_TITLES.append(item[\"title\"]) # NEW: record the title\n", " return \"\\n\".join(lines)\n", " except Exception as error: # a failed search is returned as text, not raised\n", " return \"error: \" + str(error)\n", "\n", "print(wikipedia_search(\"black hole\"))" ] }, { "cell_type": "code", "execution_count": null, "id": "c08", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · example: read each line, then run\n", "QUESTION = \"How do black holes form?\"\n", "TASK = f\"Write a short essay on this question: {QUESTION}\"\n", "WRITER_SYSTEM = \"\"\"You are a science writer. Use wikipedia_search to find facts. Cite each Wikipedia article that you use as [article title].\"\"\"\n", "REFLECT_SYSTEM = f\"\"\"{WRITER_SYSTEM}\n", "Work in steps, one step in each reply: write a draft, evaluate the draft, improve the essay, and evaluate again.\n", "Repeat until an evaluation finds nothing to improve. Then reply with the final essay under the heading FINAL ESSAY.\"\"\"\n", "\n", "single = Agent(REFLECT_SYSTEM, [wikipedia_search]) # ONE object writes and evaluates, in one list\n", "reply = single.ask(TASK)\n", "for step in range(8): # a limit, in case FINAL ESSAY never comes\n", " if \"FINAL ESSAY\\n\" in reply: # the heading on its own line, as in Week 5\n", " break\n", " reply = single.ask(\"Continue.\")\n", "single_essay = reply.split(\"FINAL ESSAY\\n\")[-1] # the text after the heading\n", "\n", "for i in range(len(single.messages)):\n", " m = single.messages[i]\n", " if m[\"role\"] == \"assistant\" and \"tool_calls\" not in m:\n", " print(i, m[\"content\"].split(\"\\n\")[0]) # the first line of each reply: a map of the steps\n", "print()\n", "print(single_essay)\n", "print(\"tokens:\", single.total_tokens)" ] }, { "cell_type": "markdown", "id": "c09", "metadata": {}, "source": [ "**Reading the output.** One list holds everything: the searches, every version of the essay, and every evaluation. The same object, with the same system message, wrote the essay and evaluated it. Every call sends the whole list again, so each step costs more tokens than the step before it." ] }, { "cell_type": "markdown", "id": "c10", "metadata": {}, "source": [ "## 2. Two roles\n", "\n", "A **role** is the responsibility of one agent: its own system message, its own tools, and its own list. We divide the work of Part 1 between two roles. The **writer** has the first line of the single agent's system message, without the steps. The **evaluator** is a second object. It never sees the searches or the drafts of the writer; it knows only what a message gives it.\n", "\n", "### 2.1 The writer drafts\n", "\n", "*Do:* run the cell and read the list of the writer." ] }, { "cell_type": "code", "execution_count": null, "id": "c11", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · example: read each line, then run\n", "writer = Agent(WRITER_SYSTEM, [wikipedia_search]) # ROLE: writer\n", "essay = writer.ask(TASK)\n", "writer.show()" ] }, { "cell_type": "markdown", "id": "c12", "metadata": {}, "source": [ "### 2.2 ✍️ The handoff to the evaluator\n", "\n", "A **handoff** gives the next agent the task and the information that it needs to continue. In code, it is a user message to the next object.\n", "\n", "*Do:* complete `handoff`: the essay question and the essay, each under a label. Then read the list of the evaluator: its searches and its evaluation." ] }, { "cell_type": "code", "execution_count": null, "id": "c13", "metadata": {}, "outputs": [], "source": [ "# ✍️ TYPE · exercise: write the code between the FILL IN lines yourself\n", "EVALUATOR_SYSTEM = \"\"\"You evaluate science essays. Use wikipedia_search to check the facts.\"\"\"\n", "\n", "evaluator = Agent(EVALUATOR_SYSTEM, [wikipedia_search]) # ROLE: evaluator\n", "### FILL IN (START) ###\n", "handoff = f\"\"\"Essay question: {QUESTION}\"\"\" # add the essay under a label\n", "### FILL IN (END) ###\n", "evaluation = evaluator.ask(handoff)" ] }, { "cell_type": "code", "execution_count": null, "id": "c14", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · check: just run\n", "evaluator.show()\n", "done[\"2.2\"] = QUESTION in handoff and essay in handoff\n", "done[\"2.2\"]" ] }, { "cell_type": "markdown", "id": "c15", "metadata": {}, "source": [ "**Reading the output.** The list of the evaluator starts with its own system message and your handoff: the question and the essay. It holds none of the searches or drafts of the writer. In our runs, the evaluator made its own searches and compared the statements of the essay with the results, one by one. The single agent of Part 1 did not make this check: its evaluations came from the same list that held its own draft. In our runs, the evaluator found no wrong fact. If your handoff carries only the question, the evaluator has no essay to check; read what it does then.\n", "\n", "### 2.3 The evaluation goes back to the writer\n", "\n", "The list of the writer already holds its essay, so this handoff carries only the evaluation.\n", "\n", "*Do:* run the cell and read the end of the list of the writer." ] }, { "cell_type": "code", "execution_count": null, "id": "c16", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · example: read each line, then run\n", "team_essay = writer.ask(f\"\"\"An evaluator wrote:\n", "{evaluation}\n", "\n", "Improve your essay.\"\"\")\n", "writer.show()" ] }, { "cell_type": "markdown", "id": "c17", "metadata": {}, "source": [ "**Reading the output.** The handoff carried only the evaluation, and the writer went back to its own task: it improved the essay, in our runs after one more search. Your code was the coordinator. It fixed the order before the run (writer, evaluator, writer) and then it stopped: this is the sequential structure. Each object has its own list, so no call sends the history of the other role." ] }, { "cell_type": "markdown", "id": "c18", "metadata": {}, "source": [ "## 3. A coordinator agent\n", "\n", "In Part 2, your code was the coordinator: it fixed the order (writer, evaluator, writer) and then it stopped. A coordinator agent decides the order itself, and it decides when the essay is finished. Each role becomes a tool: a function that gives a task to one `Agent` object and returns its answer.\n", "\n", "### 3.1 ✍️ The roles as tools\n", "\n", "*Do:* the cell makes a new writer and a new evaluator, so this run starts fresh, and the tool function `ask_writer`. Complete `ask_evaluator`: give the task to `evaluator_team` and return its answer." ] }, { "cell_type": "code", "execution_count": null, "id": "c19", "metadata": {}, "outputs": [], "source": [ "# ✍️ TYPE · exercise: write the code between the FILL IN lines yourself\n", "writer_team = Agent(WRITER_SYSTEM, [wikipedia_search])\n", "evaluator_team = Agent(EVALUATOR_SYSTEM, [wikipedia_search])\n", "\n", "def ask_writer(task: str):\n", " \"\"\"Give a task to the writer and return its essay.\n", "\n", " Args:\n", " task: The task for the writer.\n", " \"\"\"\n", " return writer_team.ask(task)\n", "def ask_evaluator(task: str):\n", " \"\"\"Give a task to the evaluator and return its evaluation.\n", "\n", " Args:\n", " task: The task for the evaluator.\n", " \"\"\"\n", " ### FILL IN (START) ###\n", " return \"\" # give the task to evaluator_team and return its answer\n", " ### FILL IN (END) ###" ] }, { "cell_type": "markdown", "id": "c20", "metadata": {}, "source": [ "### 3.2 The coordinator runs the team\n", "\n", "The system message of the coordinator names the two roles and the stopping rule of Part 1. The user message is the same task as before.\n", "\n", "*Do:* run the cell. It prints one line for each assignment of the coordinator: the role, the length of the task, and the first line of the task. Then it prints the final essay." ] }, { "cell_type": "code", "execution_count": null, "id": "c21", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · example: read each line, then run\n", "COORDINATOR_SYSTEM = \"\"\"You coordinate a writer and an evaluator to produce a short science essay. Use ask_writer and ask_evaluator.\n", "Repeat until an evaluation finds nothing to improve. Then reply with the final essay under the heading FINAL ESSAY.\"\"\"\n", "\n", "coordinator = Agent(COORDINATOR_SYSTEM, [ask_writer, ask_evaluator])\n", "answer = coordinator.ask(TASK, max_steps=12) # more steps: several assignments\n", "if answer is None: # the step limit ended the run: take the latest essay of the writer\n", " answer = writer_team.messages[-1][\"content\"]\n", "coordinator_essay = answer.split(\"FINAL ESSAY\\n\")[-1]\n", "\n", "assigned = []\n", "for m in coordinator.messages: # the requests of the coordinator are its assignments\n", " if \"tool_calls\" in m:\n", " for request in m[\"tool_calls\"]:\n", " task = json.loads(request[\"function\"][\"arguments\"])[\"task\"]\n", " print(request[\"function\"][\"name\"], \"|\", len(task.split()), \"words |\", task.split(\"\\n\")[0])\n", " assigned.append(request[\"function\"][\"name\"])\n", "print()\n", "print(coordinator_essay)\n", "done[\"3.1\"] = \"ask_evaluator\" in assigned and len(evaluator_team.messages) > 1\n", "done[\"3.1\"]" ] }, { "cell_type": "markdown", "id": "c22", "metadata": {}, "source": [ "**Reading the output.** Read the assignments. The length of a task shows what a handoff carried. In our runs, one task for the evaluator carried the essay (about 150 words), but other tasks had fewer than 20 words: \"Evaluate the revised essay ...\", without the essay. For each revision, the coordinator also wrote its own instructions for the writer, in place of the text of the evaluation. The stopping rule worked as in Part 1: the coordinator repeated the two roles several times and then wrote the final essay. A coordinator agent can make handoff mistakes that your code in Part 2 did not make." ] }, { "cell_type": "markdown", "id": "c23", "metadata": {}, "source": [ "## 4. Is the team better?\n", "\n", "To compare systems, give them the same task and grade them with the same criteria (Chapter 5). The judge is the judge of the Week 5 lab, with the same rubric. A code check adds an exact criterion: every cited title must be a title that a search returned in this notebook. The tokens show the cost.\n", "\n", "*Do:* run the two cells. Compare the three systems on scores, citations, and tokens. Then read the reasons of the judge." ] }, { "cell_type": "code", "execution_count": null, "id": "c24", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · example: read each line, then run\n", "JUDGE_SYSTEM = \"\"\"You grade short science essays. Score each criterion from 1 (weak) to 5 (strong) and give a short reason.\n", "1. explanation: the essay answers the question and explains the cause step by step.\n", "2. sources: each main fact has a citation of a Wikipedia article, written as [article title].\n", "3. clarity: a general reader can follow the essay, and technical terms are explained.\n", "Return JSON: {\"explanation\": n, \"sources\": n, \"clarity\": n, \"reasons\": \"...\"}\"\"\"\n", "CRITERIA = [\"explanation\", \"sources\", \"clarity\"]\n", "\n", "def judge_essay(question, essay):\n", " \"\"\"Score one essay with a new judge; return the JSON answer as a dictionary.\"\"\"\n", " judge = Agent(JUDGE_SYSTEM, json_mode=True) # a new judge for every essay\n", " return json.loads(judge.ask(f\"\"\"Essay question: {question}\n", "\n", "Essay:\n", "{essay}\"\"\"))\n", "\n", "def scores_of(result):\n", " scores = []\n", " for name in CRITERIA:\n", " scores.append(result[name])\n", " return scores\n", "\n", "def unsupported_citations(essay):\n", " \"\"\"Return the cited titles that no search returned.\"\"\"\n", " missing = []\n", " for part in essay.split(\"[\")[1:]: # the text after each \"[\"\n", " title = part.split(\"]\")[0] # up to the next \"]\"\n", " if title not in FOUND_TITLES and title not in missing:\n", " missing.append(title)\n", " return missing" ] }, { "cell_type": "code", "execution_count": null, "id": "c25", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · example: read each line, then run\n", "single_result = judge_essay(QUESTION, single_essay)\n", "team_result = judge_essay(QUESTION, team_essay)\n", "coordinator_result = judge_essay(QUESTION, coordinator_essay)\n", "\n", "team_tokens = writer.total_tokens + evaluator.total_tokens\n", "coordinator_tokens = coordinator.total_tokens + writer_team.total_tokens + evaluator_team.total_tokens\n", "\n", "print(\"one agent (Part 1):\", scores_of(single_result), \"| unsupported citations:\", unsupported_citations(single_essay), \"|\", single.total_tokens, \"tokens\")\n", "print(\"writer + evaluator, your code coordinates (Part 2):\", scores_of(team_result), \"| unsupported citations:\", unsupported_citations(team_essay), \"|\", team_tokens, \"tokens\")\n", "print(\"writer + evaluator + coordinator agent (Part 3):\", scores_of(coordinator_result), \"| unsupported citations:\", unsupported_citations(coordinator_essay), \"|\", coordinator_tokens, \"tokens\")\n", "print()\n", "print(\"ONE AGENT:\", single_result[\"reasons\"])\n", "print(\"TEAM, YOUR CODE:\", team_result[\"reasons\"])\n", "print(\"TEAM, COORDINATOR:\", coordinator_result[\"reasons\"])" ] }, { "cell_type": "markdown", "id": "c26", "metadata": {}, "source": [ "**Reading the output.** In our runs, the single agent and the team of Part 2 both scored 5 on every criterion, and no system cited an article that no search returned. The team of Part 2 used about a quarter to a third of the tokens of the single agent: its lists are short, and it ran one round. The coordinator team used the most tokens, two to three times the single agent, and its final essay scored 1 or 2 on sources, because it had no citations in the form `[article title]`. Run `writer_team.show()` to find where they were lost: in an essay of the writer, or in the final answer of the coordinator.\n", "\n", "So on this question, the division into two roles did not raise the scores: the single agent was already at the top of the scale. It lowered the cost, and it gave the evaluation its own searches. The coordinator agent added cost and a new place for errors. These are observations on one question, not a general rule.\n", "\n", "*Try:* change `QUESTION` in Part 1 to another science question, for example \"Why does the Moon cause tides on Earth?\", and run the notebook again. One question does not show which system is better in general." ] }, { "cell_type": "markdown", "id": "c27", "metadata": {}, "source": [ "## 5. Explain in your own words\n", "\n", "Write two or three sentences for each in `answers` (not checked).\n", "\n", "1. What does the evaluator know when it starts its work, and where does that come from?\n", "2. Who decided the order of the work and the moment to stop: in Part 1, in Part 2, and in Part 3?\n", "3. From Part 4: is the team better than the single agent on this question? Give the evidence for and against, with the cost.\n", "\n", "*Do:* write your answers, then run the last cell to see which exercises passed." ] }, { "cell_type": "code", "execution_count": null, "id": "c28", "metadata": {}, "outputs": [], "source": [ "# ✍️ WRITE · your answer in words (not checked)\n", "answers = \"\"\"\n", "1.\n", "\n", "2.\n", "\n", "3.\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": null, "id": "c29", "metadata": {}, "outputs": [], "source": [ "# ▶ RUN · check: just run\n", "for key in done:\n", " print(key, done[key])" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }