AI for Data Analysis II

Today

• Breaking work into clear steps.
• Coding data with the agent.
• Break.
• Verifying your results.
• Working faster: several agents at once.
• Discussion.

Where we left off

• You have a repository with a README, protected data, one script, and one commit.
• You know what an agent is: tools, a loop, and code execution.
• You wrote one sentence last night: the analysis you want to run.
• Today you do two things: run several agents at once to go faster, then direct one analysis carefully to get it right.

Five stages of any analysis

• Every empirical project moves through the same five stages:
  • Data preparation. Where does the data come from, and is it clean?
  • Exploratory analysis. What patterns are there?
  • Statistical modeling. Does the hypothesis hold?
  • Output and reporting. How do you present it?
  • Reproducible publication. Can someone else replicate it?
• Today you drive stages 1 through 4 on your own data. Stage 5 is Wednesday.

What the agent contributes at each stage

• The pattern in one line: the agent does the typing, and you make the choices.

Three ways to run the agent

• The agent runs at one of three levels of freedom:

• Claude Code and Codex both have these three; the names and the key to switch differ, so check your tool.
• Ask yourself how much you would let a brand-new RA do unsupervised.

Explore, Plan, Execute, Commit

• The workflow for any task with real stakes:

• Explore and Plan are cheap; a wrong plan costs one rewrite, and a wrong execution costs an afternoon.

Turning text into variables

• A lot of social-science data starts as text: open-ended answers, interview transcripts, documents, news.
• The agent can read each case and assign a code, the way a research assistant would.
• That turns text into a variable you can count and model.

How to set it up

• Write the codebook first: the categories, each with a clear rule and an example.
• Give the agent the codebook and a handful of cases, and read what it returns.
• Fix the codebook where it was unclear, until the agent codes the way you mean.

The catch: check it against human coding

• Hand-code a random sample yourself, then compare the agent's codes to yours.
• Report the agreement, the same way you would for two human coders.
• Low agreement means the codebook is vague or the task is too hard; fix that before you trust the rest.

Then code at scale

• Once agreement is high enough, let the agent code the full dataset.
• Keep the sample and the agreement number; they are your evidence the measure is sound.
• Spot-check a few more as it goes.

Why you always check

• The agent is fluent, and fluent output can still be wrong.
• It will not tell you when it guessed; a wrong number looks exactly like a right one.
• Nothing counts as a result until you have checked it yourself.

Ways to check a result

• Recompute one number a second way, in software you already trust, and see if it matches.
• Compare it against something you already know: a published figure, a total that must add up, or a number you expected.
• Read the code and the choices it made: which cases it dropped, how it treated missing values, whether it weighted.
• Look at the data at its edges: the row count, the missing counts, the smallest and largest values.
• Change one input, and check the output moves the way it should.
• Have a fresh session or a second agent redo the step, and compare the two.

Worked example: Monday's chart

• Monday's WVS chart looked convincing. It was probably wrong in at least one place:
  • Right variable? WVS has several politics items: interest, importance, action. Check the codebook.
  • Missing values? WVS codes "don't know / refused" as -1 to -5; averaging those in corrupts every country mean.
  • Weights? WVS has survey weights; unweighted means can mislead.
  • Scale direction? 1 = very important, 4 = not at all; the chart may be upside-down in meaning.
• Every one of these is invisible in the picture and visible in the code.

You can run more than one agent

• You are not limited to a single agent; you can open several at once.
• Give each one a separate job, and they work at the same time.
• One cleans the data, one draws a figure, one looks up the codebook, all at once.
• The jobs have to be independent, so none of them waits on another.

Or let one agent run the others

• You do not have to open and manage the agents yourself.
• You can tell one agent to do it: "clean the data, draw the figure, and look up the codebook, using a separate helper for each."
• It opens the helpers, hands them the jobs, and brings back the results.
• You will hear these helpers called subagents; the only difference is who manages them, you or the main agent.

What running them costs

• The tool charges by usage, measured in units called tokens, and each plan has a limit.
• Several agents at once, or one agent working for a long time, spend that usage faster.
• Check your usage now and then, because going faster spends it faster.

Discussion

• Where did the agent surprise you, for better and for worse?
• What did you catch when you read the code?
• Would you have caught it without being told to look?

Why long sessions drift

• As a session grows, the agent's working memory fills with files, outputs, and errors.
• Your instructions from the first hour get crowded out, and the agent slowly reverts to its own defaults.
• Tomorrow morning it remembers nothing at all.
• Tomorrow's session builds the fix: instructions that persist outside the conversation.