| Week 1Sept 10Shopping · overview |
What an AI agent is
- Isn't this just ChatGPT? It works in your folder rather than in a chat window
- It can use tools, run the code it writes, and read the errors back
- What it is good and bad at
- Demo: a city's 311 CSV → cleaned, analyzed, charted, live
|
Install, then redo the demo yourself
- Install Claude Code with step-by-step support; TA circulating
- Redo the demo on the same 311 dataset, with the same prompt, on your machine
- If time: try it on a different provided dataset (permits, inspections)
|
The eight steps of policy analysis, start to finish
- RequiredBardach & Patashnik, A Practical Guide for Policy Analysis, the eightfold-path overview chapter [chapter and edition to confirm; the same textbook as MPA 2445, Policy Analysis and Problem Solving, the MPA core course, also taught in the fall]
- OptionalOECD, Digital Government Outlook 2026, AI chapter; Stanford HAI, AI Index 2026, policy chapter; Margetts & Dorobantu (Nature 2019)
- Discussionone question: which of the eight steps do you think AI can help with, and which not? Vote and defend; the answers are recorded and revisited in Week 12.
|
Nothing due. Installation happens in class. |
| Teaching goalEveryone leaves having watched the agent write and run code, and with the tool installed. Students see it hit an error and fix it on its own. |
| Week 2Sept 17Shopping |
Directing the agent
- Git in 15 minutes, agent-first: a commit is a snapshot; the agent runs the commands, you read what it did
- Three ways to run the agent: normal (approve each step), auto-accept, plan mode (the agent writes a plan, you approve or edit it before anything runs)
- Demo: same task with a vague prompt, then with a directed one
- What a session costs, and how to spend less
|
Repo + first script + plan mode
- Create a repo; have the agent write a summary-statistics script on a provided dataset; commit it
- A second, bigger task (merge two files, chart the result) in plan mode: read the agent's plan, change one thing, then let it run
- Checkpoint: show a neighbor your script runs; explain it in your own words
- Late adders: setup guide + office hours, on your own; class does not repeat Week 1
|
Does AI actually make an analyst faster or better?
- RequiredDell'Acqua et al., Navigating the Jagged Technological Frontier (Harvard Business School Working Paper 24-013, 2023): a randomized experiment with 758 BCG consultants. AI raised speed and quality on tasks inside its competence, and made people more wrong on a task designed to sit just outside it.
- RequiredUK Department for Business and Trade, Evaluation of the M365 Copilot Pilot (2025), executive summary. A government ministry's own evaluation: real but small time savings, hallucinations observed throughout, and no evidence that saved time became higher productivity.
- OptionalBrynjolfsson, Mitchell & Rock (AEA P&P 2018), the pre-agent task rubric; GAO, Generative AI Use and Management at Federal Agencies (GAO-25-107653); Generative AI in Public Administration: A Quasi-Experimental Analysis of Bureaucratic Productivity (Government Information Quarterly 2026)
|
Nothing graded, but your setup must be working by the end of this week: the tool installed, a repository created, one committed script. If it is broken, come to office hours before Week 3. |
| Teaching goalEveryone can stage a multi-step task through plan mode instead of sending a vague prompt. The vague-vs-directed demo is where prompting skill becomes visible. |
| Week 3Sept 24Eightfold pathStep 1: define |
Why the agent is inconsistent, and the fix
- Every session starts blank; vague instructions produce variable output
AGENTS.md: project context, data conventions, output standards, safety rules
|
Working dataset + spec + the with/without-spec comparison
- Choose the dataset and policy question for the semester. This is the draft of your final project; the Week 9 proposal formalizes the same thread. Switching later is allowed, but the cost grows every week.
- The 3-item dataset checklist, verified in conference:
- programmatically collectable, meaning an API or a scrapable source (Week 4 needs this)
- a policy question you can chart and explain to a named audience (Week 6)
- a topic you are willing to carry all semester
Text data is NOT required; numeric-only projects are fully eligible.
- One paragraph: the question, labelled predictive / causal / descriptive
- Write the spec; run the same task with and without it, and compare the two outputs side by side
|
Step 1: define the problem, and why two analysts define it differently
- RequiredBardach & Patashnik, the Step 1 chapter, on defining the problem
- RequiredKahneman et al., Noise (HBR 2016)
- OptionalDeborah Stone, Policy Paradox, excerpt on problem definition; Gebru et al., Datasheets for Datasets
|
Check-in 1
- a one-paragraph question, labelled predictive, causal, or descriptive
- a named dataset that passes the three-item dataset test
- an AGENTS.md with all four sections
- the same task run with and without the spec, both outputs committed
- everything in the repository with a clear commit message
data/codewriting
Due before class, Thu Oct 1
Worth 5% of the grade, one point per checklist item |
| Teaching goalWithout the spec, the same task comes back with different choices each run; with the spec, the output follows the rules the student wrote. The side-by-side comparison is the evidence. |
| Week 4Oct 1Eightfold pathStep 2a: evidence |
Collecting data with the agent
- How an API call works; where public data lives (Census, BLS, Socrata)
- Scraping a table when there is no API; what a 403 tells you
- Keep the raw pull; log where it came from and when
- Subagents: when the pull is really 50 identical pulls (50 states, 10 years, 20 agencies), the agent sends one copy of itself per piece, waits, and assembles. Each copy needs written instructions: what to fetch, what format to return, what to do if the source fails
|
Build your data pipeline
- API pipeline from a starter template, pointed at your working dataset's domain
- One scraped table with its provenance log
- Extend the pipeline to several units (a few states or years) with subagents, one per unit
- README: what source, what parameters, how to rerun
- Last 20 min: swap repos, rerun a neighbor's pipeline from their README alone
|
Step 2: assemble the evidence, and write down where it came from
- RequiredLazer, Kennedy, King & Vespignani, The Parable of Google Flu (Science 2014), 3 pp.: 45 undocumented search terms, nobody could rerun it, wrong for 100 straight weeks
- OptionalLum & Isaac, To Predict and Serve? (2016), a different data problem: the records measure policing rather than crime, which a provenance log does not fix
|
Check-in 2
- running the script from a fresh copy of the repository produces the same output file, with no manual steps
- the downloaded file is saved unchanged in
data/, next to a short note of the URL and the date it was pulled
- changing one parameter (a different state or year) and rerunning produces new, correct output
- a README that tells someone else what to run and in what order
- committed to the repository
data/code
Due before class, Thu Oct 8
Worth 5% of the grade, one point per checklist item |
| Teaching goalHalf the neighbor-reruns fail on the first try, for the same reason Google Flu could not be rerun: the documentation is missing. A README that tells someone else what to run, and in what order, prevents this. |
| Week 5Oct 8Eightfold pathStep 2b: evidence |
Coding text at scale: making your measures
- Using the agent to turn text (311 complaints, public comments, permits) into variables
- The prompt is the instrument: freeze it, then run it
- The tier question: can you check every result yourself? If not, you need a gold standard
- Confusion matrix; a simple baseline for comparison
|
Build and validate your measure
- Hand-code 100 documents from the course-provided text dataset (311 complaint narratives); use your own data instead if your project has text
- Have the agent code 1,000 with the frozen prompt
- Compare: confusion matrix against your 100, and against a two-feature baseline
- Optional close: wrap your frozen prompt as a Skill
|
Step 2: build a measure, and check that it measures what you meant
- RequiredObermeyer et al. (Science 2019): the algorithm predicted health costs when the goal was health needs, so it understated how sick Black patients were
- RequiredDressel & Farid (Sci Adv 2018): two features match COMPAS
- OptionalDavidson et al. (ICWSM 2017)
|
Skill check: measurement
- your 100 hand-coded documents
- the agent's 1,000 codes, from the same frozen prompt
- the confusion matrix comparing the two
- the confusion matrix for the two-feature baseline, for comparison
data/code
Due before class, Thu Oct 15
Worth 10% of the grade |
| Teaching goalThe agent's coding looks correct until the confusion matrix. Expect real disagreement with the hand-coded 100, and the two-feature baseline comes close for some students. After this week nobody should treat unvalidated agent output as data. |
| Week 6Oct 15Eightfold pathSteps 2c + 8 |
Analyze, visualize, publish
- From coded data to charts a non-technical reader can use
- From a script to a page, and from a page to a public URL (GitHub Pages)
- When the data cannot leave your machine, and how to work that way
|
Analysis + a public page + the brief
- Analyze the data you collected in W4, folding in your W5 measure if it came from your own text: three charts with plain-language captions
- An interactive display: filter by geography, time, or group
- Deploy it: a public link anyone can open, committed in your README
- Draft the 2-page data brief to a named decision-maker. Every number in it must come from a script's output; none typed in by hand.
|
Step 2 and step 8: publish the evidence so others can check it
- RequiredYu & Robinson, The New Ambiguity of "Open Government" (2012): evidence the public can audit; transparency affects whether the evidence is credible, not only how it is presented
- OptionalO'Neil, Weapons of Math Destruction, ch. 1
|
Data brief
- 2 pages to a named decision-maker
- each number in it comes from a script's output, not typed by hand
- the public page linked
writing
Due before class, Thu Oct 22
Worth 10% of the grade |
| Teaching goalBy end of class every student's analysis is at a URL a stranger can open. The readings argue for transparency, and students apply it to their own work. Exploratory charts are Step 2 and the public page is Step 8; the agent makes publishing cheap, so enforce the rule that every number in the brief comes from a script, not from someone eyeballing a chart and typing a figure in. This is the fullest week of the implement arc; if it overruns, the brief deadline (before W7) absorbs it. |
| Week 7Oct 22Added to the pathVerify |
Checking the AI's work: facts and fairness
- Why a fluent model invents citations and figures; demo: five claims about RI poverty, checked against Census live
- Fairness metrics: parity, equalized odds, calibration, and why you cannot have all three
|
Audit your own outputs
- Fact-check: pull 10 numbers from your W4–6 outputs, trace each to a source; table: claim | AI number | actual | verdict
- Subgroup error audit: recompute the W5 confusion matrix group by group, using demographic groups if your data has them, otherwise geography, source, or time period; the skill is the same
- Short structured debate: which fairness definition should govern your measure?
|
Check the evidence: invented numbers, and errors that fall unevenly across groups
- RequiredNIST AI RMF 1.0, exec summary
- RequiredProPublica, Machine Bias, read against Corbett-Davies et al. (WaPo)
- OptionalBuolamwini & Gebru, Gender Shades; Cronkite News on fabricated citations in court
|
Audit
- 10 facts checked against a source
- a subgroup error audit
- 1 page of interpretation
data/codewriting
Due before class, Thu Oct 29
Worth 10% of the grade |
| Teaching goalMost students find at least one wrong number in their own earlier outputs. This is the densest session of the course: fact-check, fairness, and debate; cap the debate at 20 minutes. |
| Week 8Oct 29Eightfold pathSteps 3+4 |
Audit a government AI system: everything so far, applied
- What the government itself runs: the mandated public inventory of federal AI use cases
- M-25-21's checklist for high-impact systems repeats, item by item, what you already know: validated against what gold standard (W5)? numbers documented and rerunnable (W4, W7)? fairness-tested by group (W7)?
- No new coding; W4's collection skills, pointed at a governance target
|
Score a real federal system
- Pull the 2025 federal AI use-case inventory (3,611 rows)
- Collect the agency-hosted inventories the same way you collected your own data in Week 4, one subagent per agency
- Pick one high-impact system; score it against M-25-21's own
hi_* minimum practices
- Jigsaw report-out
- Before leaving: form arena groups and claim an arena. This is the policy-of-AI thread, about AI as a policy domain; it is separate from your own project's domain. Groups present in W10, two weeks of preparation.
|
Steps 3 and 4: alternatives and criteria, as the federal government already writes them
- RequiredOMB M-25-21 (operative; M-24-10 rescinded, used for comparison only): the
hi_* minimum practices are a criteria list already in use, Step 4 in practice
- RequiredEngstrom, Ho, Sharkey & Cuéllar, Government by Algorithm (2020), pp. 6–20: what agencies actually deployed, which is the set of alternatives available in practice
- OptionalBlueprint for an AI Bill of Rights (2022)
|
Policy of AI ①
- one high-impact federal system, chosen from the inventory
- a scorecard against the M-25-21 minimum practices
data/code
Due before class, Thu Nov 5
Worth 5% of the grade |
| Teaching goalThe integrative week: four weeks of practice on their own project become the criteria for judging someone else's system. Engstrom's team took 8 months to find 157 cases by hand; the class collects 3,611 in one session. Arena groups form here and present in W10, with two weeks of preparation. |
| Week 9Nov 5Eightfold pathStep 5: project |
Designing your final project
- What makes a policy project worth deploying: question, audience, upkeep
- Scoping: what fits in two build weeks
|
Proposal + deployed skeleton
- Project proposal: question, data, audience, what the project does
- Deploy an empty but working version, published the same way as the Week 6 page
- Instructor conferences rotate through the room while everyone builds
|
Step 5: project the outcomes before the policy is adopted
- RequiredMullainathan & Spiess (JEP 2017): what prediction can give policy, ŷ vs. β̂
- RequiredHofman, Sharma & Watts, Prediction and Explanation in Social Systems (Science 2017): the ceiling on predictability of social outcomes
- RequiredHewitt, Ashokkumar, Ghezae & Willer (Nature 2026): LLMs predict survey-experiment treatment effects at r = 0.85
- Discussionuse AI to predict your proposed project's outcomes, then argue about the consequences of relying on prediction: what would your agency be allowed to conclude from a simulated pilot?
|
Check-in 3: the proposal
- the question and a named audience
- the data, already in hand from the semester so far
- what the project will do
- three alternatives, including doing nothing
- predicted outcomes and the basis for the prediction
writing
Due before class, Thu Nov 12
Worth 5% of the grade, one point per checklist item |
| Teaching goalEvery project has a live URL before building starts. The prediction readings set the question each proposal has to answer: what will happen when this deploys, and on what basis do you claim that. |
| Week 10Nov 12Eightfold pathSteps 6+7 |
The six arenas of AI policy
- Opening lecture (~30 min): what people argue about when they argue about AI policy, and a quick tour of the three regimes (US preemption fight, EU AI Act + 2026 delay, China's layered rules)
- Shared pre-read for everyone, not just the regimes group: Bradford, Digital Empires, Introduction (~15pp) [in Zotero], which gives the whole class the market-driven / state-driven / rights-driven vocabulary before groups split
- Two arenas are not on the menu because the class already worked in them: present-harm accountability (W7) and government's own AI (W8)
|
Teach your arena (groups formed in W8)
- Each group, 8–10 min: what the fight is about, who is fighting, one concrete 2025–26 case, one core reading they chose, from the menu below or found with the agent
- Every presentation ends on the same question: what does this arena's fight mean for a government analyst office?
- Memo: draft a one-page internal AI-use policy for a government analyst office: weigh the trade-offs among the approaches your arena surfaced, decide, and cite your own experience this term for each rule
|
Steps 6 and 7: weigh the approaches to regulating AI, and decide (one arena per group)
- RequiredEvery student reads the shared Bradford introduction; every group reads and presents one core reading from its own arena.
- Frontier-model risk: International AI Safety Report; Backfiring Effect of Weak AI Safety Regulation (PNAS 2026, in Zotero; citekey pending a metadata fix)
- Jurisdiction & regimes: Nelson (Science 2026) [in Zotero]; the shared Bradford intro [in Zotero]; on China specifically, Sheehan, China's AI Regulations and How They Get Made (2023) [in Zotero] for how the rules get made, Migliorini, China's Interim Measures on Generative AI (2024) [in Zotero] for what they say (already compares to the EU AI Act); the Dec 2025 preemption EO
- Geopolitics & chips: export-control brief (not yet picked); Chip War excerpt (background only; it is chip-industry history rather than AI policy)
- Power & concentration: Kak & West, the sole candidate right now; Farrell's AI as Governance was read in full and dropped, because it is a political-theory research agenda rather than a case-based fight, so this arena needs a second reading before W10
- Information ecosystem: Kostka & Bogs, The Expanding Digital Reach of the Chinese State (2026) [in Zotero]: function creep as the mechanism (Health Codes repurposed against protesters, facial recognition onto Uyghurs and schoolchildren); China's 2025 labeling rules; EU AI Act Art. 50; state deepfake-election laws
- Labor & economy (incl. the regulation-vs-innovation fight): Brynjolfsson, Mitchell & Rock (optional in W2); Bradford, False Choice (2024); Draghi excerpt; Jia, Jin & Wagman (2021)
|
Policy of AI ②
- an arena presentation, delivered in class
- a 1-page memo that weighs the alternatives and decides
presentationwriting
Presented in class, Nov 12
Memo due before class, Thu Nov 19
Worth 10% of the grade |
| Teaching goalStudents teach the six arenas, with two weeks of preparation, so the class covers the range of AI policy debates. In the memo each student weighs the approaches, picks rules, and backs each rule with their own experience this term. |
| Week 11Nov 19Project week |
Reviewing someone else's repo
- What a review looks for when nobody typed the code
|
Build, then swap
- Build time with instructor support
- Swap repos; rerun a classmate's project using the README alone; write a one-page review
|
No reading, build week. |
Peer rerun review, counts toward Participation (10%)
presentation |
| Teaching goalIn Week 4 students reran a practice pipeline from a classmate's README. Here they do the same thing to each other's real final projects: can a classmate get your project running using only what you wrote down? Thanksgiving is next week, so this is the last class before presentations. A problem found here still has two weeks on the calendar to fix, but no more class time to get help with it. |
| Thu, Nov 26: Thanksgiving recess (Nov 25–29). No class. |
| Week 12Dec 3Eightfold pathStep 8: tell it |
What you can claim, and what to learn next
- Revisit the validation tiers from Week 5: what could you defend in a methods section?
- What the next step up costs, and when not to take it
|
Final presentations
- Present the deployed project
- Required in every talk: one thing the agent got wrong, and how you caught it
|
No reading, presentations. |
Final project
- the deployed project
- documentation that lets a classmate rerun it
- a presentation naming one agent error you caught
data/codewritingpresentation
Presented in class, Dec 3
Project and documentation due, Thu Dec 10
Worth 30% of the grade |
| Teaching goalEvery presentation includes the "one thing the agent got wrong" slide: nobody presents a project without presenting an audit of it. |