AI for Data Analysis II

Today

• Breaking work into clear steps.
• Verifying your results.
• Several agents at once.
• Turning text into data.
• Discussion.

Monday's Materials

• You have a repository. It has a README, protected data, one script, and one commit.
• You know what an agent is. It has tools, a loop, and code execution.
• You wrote one sentence last night, which was the analysis you want to run.

Five stages of analysis

• Every empirical project moves through the same five stages:
  • Data preparation. Where does the data come from, and is it clean?
  • Exploratory analysis. What patterns exist?
  • Statistical modeling. Is the hypothesis true?
  • Output and reporting. How do you show it?
  • Reproducible publication. Can another person do the same thing?
• Work through stages 1 to 4 today using your own data and Stage 5 is a build option on Friday.

Agent contributions by stage

• The agent types. You make the choices.

Unstated choices

• Real tasks have unstated choices. You must choose a variable, decide how to treat missing codes, decide whether to weight, and choose a sample.
• If you leave a gap, the agent fills it with a default you did not choose.
• Write the plan first, so the choices are yours.

Who writes the plan

• Three ways, all legitimate:
  • You plan. You break the work into stages yourself and hand the agent one stage at a time.
  • The agent plans. In plan mode, it reads your project, proposes a plan, and runs nothing until you approve it.
  • You draft, the agent refines. You write the plan and hand it over. The agent tells you what is missing, what order is wrong, and what it would cut.
• Today's exercise uses the third way, so use it on your own project.
• Press Shift+Tab to switch modes in both tools:
  • Claude Code cycles plan mode, ask-first, and auto-accept
  • Codex cycles Plan, Pair, and Execute
• You own the plan after you edit it, and this applies regardless of who drafts it. Do not approve any step that you cannot explain.

Why you check frequently

• The agent speaks well, but fluent output can still be wrong.
• It will not tell you when it guessed, and a wrong number looks exactly like a right one.
• Check every result yourself.

Ways to check a result

• Pick one number, recompute it using different software that you trust, and see if the result matches.
• Check it against what you already know and use a published figure, a total that must add up, or a number you expected.
• Read the code and its choices, see which cases it dropped, how it treated missing values, and if it used weights.
• Look at the basic data features and check the number of rows and the missing counts and find the smallest and the largest values.
• Change one input, and check the output moves the way it should.
• Start a new session or have a second agent do the step again, and compare the two.

Worked example: Monday's chart

• Monday's WVS chart looked convincing. It was probably wrong in at least one place:
  • Right variable? WVS has several politics items: interest, importance, action. Check the codebook.
  • Missing values? WVS uses codes from -1 to -5 for "don't know / refused," and averaging these values makes the mean for every country incorrect.
  • Weights? WVS uses survey weights and unweighted means can be misleading.
  • Scale direction? 1 means very important and 4 means not at all. The chart may go in the opposite direction of your expectation.
• All of these are invisible in the picture but visible in the code.

Run multiple agents

• Open many agents at once, give each agent a different job, and they work at the same time.
• All three read the same cleaned dataset: one writes the descriptive table, one draws the chart, one drafts the README.
• Jobs must be independent, and no job can wait on another. Cleaning and charting are not independent, and the chart needs the cleaned data.

Subagents: One agent manages others

• You do not have to open the agents yourself, and can tell one agent to do it instead. Tell it: "write the table, the chart, and the README from the cleaned data, using a separate helper for each."
• These helpers are called subagents. Each has its own context, does one task, and sends a short report back to the main agent.
• The main conversation only gets the reports, which keeps the helpers' working text from filling it up.
• Ask for them in plain language. Say "in parallel, using separate subagents." This is enough and you do not need to configure anything.

Which arrangement for which job

• The difference depends on who coordinates the work, and it is either you or the main agent.
• Open several sessions yourself if the tasks are independent and do this when you want to watch and steer each one.
• Use subagents if you need to combine results later. The main agent gets every report and can ask all subagents to use the same format.
• If the combining step requires judgment, do it yourself. Run separate sessions and merge the results.
• On Thursday, you use subagents on a real case, read five papers in parallel, and then do one synthesis.

Session messaging

• In both tools, one open session can send a message to another.
• Claude Code can list your reachable sessions and send a message to one of them, with no setup, including sessions on other machines. Its /help lists the command.
• Codex: A session can queue a message to another local or remote session.
• Use it to coordinate your open sessions, and the session that finishes cleaning tells the modeling session that the data is ready.
• Messages are for coordination in the moment.
• The durable record still lives in git, where a collaborator can read it later.

Two reasons to run multiple

• Speed is one reason. Split a task into five independent pieces and run five agents, so a five-minute wait becomes about one minute.
• If one piece needs output from another, it waits and the speedup is gone.
• Another reason is checking. Two subagents do the same job, and they cannot see each other, so comparing their results tells you something one run cannot.
• Research already works this way. Two coders code the same sample, and someone else runs a replication.

Plan with advanced model, execute with cheap one

• The advanced model plans better and it costs more. The cheap model is fast and it costs almost nothing.
• Planning needs a better model and execution is mechanical, so spend the expensive tokens on planning.
• One prompt does it:

• You set your own session's model with /model, and this applies to both tools.
• Tomorrow you will write this down one time. Put it in a file and the agent reads this file every time.

Running costs

• The tool charges based on usage, which is measured in units called tokens, and each plan has a limit.
• Using several agents at once or one agent working for a long time makes you spend faster.
• Check your usage now and then.

Turning text into variables

• Much social-science data begins as text. It includes open-ended answers, interview transcripts, documents, and news.
• The agent reads each case, assigns a code, and works like a research assistant.
• That turns text into a variable, and you can count and model it.
• In this unit, you build a codebook which contains the coding rules for the agent. To improve it, have two coders work alone and read the parts where they disagree.
• Scaling this up to thousands of cases is the last card of this unit.

The pilot: two coders, one codebook

• This is the main exercise of the unit.
• Two subagents use your codebook, and they each code the same ten cases and work independently.
• The codebook was unclear when they disagreed. Discuss the disagreements, fix the rules, run again, and watch agreement rise.
• What you keep at the end is the revised codebook.
• Be careful. The two subagents use the same model, and their agreement is like one coder coding twice. Your own hand codes are the reference.

Compare agent codes with yours

• Pick a random sample, hand-code it yourself, and compare your codes to the agent's codes.
• Report the agreement the same way you would for two human coders.
• Low agreement means the codebook is vague, and it can also mean the task is too hard. Fix those issues first, and do not trust the rest until then.

Coding at scale

• Today you develop the codebook in the conversation, on dozens of cases.
• Do not feed thousands of cases into the conversation:
  • it costs money
  • it uses up the context window
  • the session drifts as it fills
  • you cannot rerun a conversation
• The agent writes a script instead. The script sends each case to a cheap model over the internet, prompts it with the frozen codebook, and saves the labels to a CSV.
• The script can be rerun or resumed, and a run costs cents.
• That script has a frozen prompt and is your measurement instrument.

Discussion

• What unexpected things did you learn today?
• What mistakes did the agent make today and how did you find them?
• There are three tools today: plan mode, a second coder, and several agents at once, which one will you use on your own project this week?

Homework

• Finish any unfinished stages of your plan today and commit each stage.
• Write down the three instructions you found yourself repeating to the agent today. Tomorrow they become the first draft of your spec file.
• Write down one procedure you want to repeat on a future project and turn it into a Skill on Wednesday.

Why long sessions drift

• The session accumulates files, outputs, and errors as it grows.
• The first hour's instructions get crowded out, and the agent slowly reverts to its own defaults.
• It will remember nothing tomorrow morning.
• Tomorrow's session builds the fix and includes instructions that persist outside the conversation.