AI for text coding

Today

• Learn what text coding is and how research and government use it.
• Code 10 abstracts by hand with a given codebook, then revise the codebook.
• See why coding at scale needs a computer, and how computers code text.
• Write scripts that code the abstracts with a dictionary and with a language model.
• Test every method on a held-out sample of hand-coded abstracts: accuracy, precision, recall, and F1.
• Choose a coding method by its errors, its cost, and the rules for your data.
• See how topic models help when you do not yet know the categories.

From text to a measure

Today, code text by hand, ask a computer to code the same text, and check its codes against your own. Use the abstracts collected in Week 4 or suitable text from your individual project.

Evaluating AI text annotation

• Gilardi, Alizadeh & Kubli, “ChatGPT outperforms crowd workers for text-annotation tasks” (2023). Focus on how annotation performance is evaluated.
• Optional readings appear at the end of several sections below.

Turning text into a variable

• Much policy information comes as text. Administrative records such as rules, public comments, court filings, and inspection notes are mostly text.
• Surveys contain text too, whenever people answer a question in their own words.
• Text coding turns that text into a variable. You read each document and assign it one value from a fixed set of categories.
• A codebook is the written rule for that assignment. It names the variable, lists the categories, and defines each category with an example.

Text coding in research and government

• Researchers code text to build the variables their analysis needs.
  • The Manifesto Project codes the election platforms of more than 1,400 parties since 1945. Coders assign each sentence to one of 56 policy categories, such as “Free Market Economy”. The codebook is public.
  • The Comparative Agendas Project codes government activity in about 20 countries and regions into the same 21 policy topics. Researchers use it to compare what governments pay attention to.
• Governments code text to produce official statistics.
  • The American Community Survey asks people to describe their job in their own words. The Census Bureau codes each answer into one of about 570 occupation categories. Different answers for the same job, such as “nurse”, “RN”, and “registered nurse”, must get the same code.
  • The Bureau of Labor Statistics codes written descriptions of workplace injuries. Each case gets codes for the occupation, the injury, the part of the body, and what caused it.

A draft codebook for federal rules

• Today everyone codes the Federal Register abstracts collected in Week 4. Each of you chose a different agency, so the question must work for any agency.
• The question for each abstract: does this rule add regulation, reduce it, or neither?
• The Federal Register labels each document by type and agency. These labels do not answer the question. You need to read the abstract.
• I wrote a first codebook with three categories.
  • Stricter: the rule adds or tightens a requirement, limit, fee, or standard.
  • Looser: the rule removes, delays, exempts, or eases a requirement.
  • Technical: the rule corrects, clarifies, or extends a deadline without changing what anyone must do.
• This codebook is a first draft. In the exercise below, you test it on your own abstracts and improve it.

Exercise 1: Revising the draft codebook

• Code 10 abstracts from your agency with this codebook.
• If you were writing this codebook, how would you make it clearer? Rewrite it.

Exercise 2: A codebook for your own project

• Pick text from your own project, such as state laws, incident reports, or public comments. Write a short codebook for one variable.
• Swap with a classmate who does not know your topic. Your partner codes 5 of your documents using only the codebook. Do not explain anything while they code.
• Compare their codes with yours. Where you disagree, find the part of the codebook that misled them.
• A large language model reads your codebook the same way your partner did. It knows only what the codebook says.

The volume of policy text

• A federal agency can publish hundreds of rules in a year. In 2025 the EPA published 242 proposed rules.
• In 2012 the Census Bureau reported that it assigns more than 2 million industry and occupation codes to survey answers every year (Thompson, Kornbau & Vesely 2012).
• A public comment period can draw far more documents. The FCC received more than 22 million comments on net neutrality in 2017 (New York Attorney General 2021).
• At that scale, hand coding takes too long. Researchers and agencies use computers to code the text.

Two sources of difficulty

• A coding task can be hard for two separate reasons.
  • Each document is hard for a person to code. Two careful people using the same codebook often disagree. Exercise 2 tested this.
  • There are too many documents for people to code.
• The two reasons combine into four cases.

• A computer solves the scale problem. It does not make a hard document easier. If people disagree about a document, there is no correct code against which to check the computer.
• Davidson and colleagues (2017) found that hate speech is hard to separate from other offensive language, for people and for computers.

When the text does not hold the answer

• Some questions cannot be answered from the text at all.
• Was this public comment written by AI? The answer depends on how the comment was written. The text itself may show no sign of it.
  • OpenAI released a tool to detect AI-written text in January 2023. It withdrew the tool in July 2023 because of its low accuracy.
  • Liang and colleagues (2023) found that detectors of AI-written text labeled more than half of the essays by non-native English writers as AI-written.
• A better model does not help with such questions. The answer needs information from outside the text.

Coding a sample

• Some questions ask for a share. For example, what share of the 2025 rules tightened regulation?
• For a share, code a random sample by hand. With a random sample of 400 documents, the estimate is usually within 5 percentage points of the true share.
• Other questions need a code for every document. For example, list every rule that loosened a requirement. These questions need the computer methods in the next section.

Three common approaches

• The dictionary method uses keyword rules. You list keywords for each category. A document gets a category when it contains that category’s keywords.
  • RegData measures the amount of federal regulation by counting the words “shall”, “must”, “may not”, “required”, and “prohibited”.
• Supervised machine learning trains a model on a large number of hand-coded documents.
  • The Bureau of Labor Statistics trained its injury coder on past cases that its staff had coded by hand. The model now assigns most of the codes.
  • The Census Bureau codes occupations the same way. Answers the model is unsure about go to staff coders.
  • This course does not teach supervised machine learning.
• Large language models code text from a prompt. The prompt contains the codebook, the text, and a request for a category. There are two common ways to write the prompt.
  • Zero-shot: the prompt contains only the codebook and the text. It needs no coded examples.
  • Few-shot: the prompt also contains a handful of coded examples.
  • Agencies are now testing large language models for this work. In April 2026, Census Bureau researchers presented their work on coding occupations with large language models.
• Today you code with a dictionary and with a large language model.
• The reading by Gilardi and colleagues compares coding by ChatGPT with coding by crowd workers.

Four principles

• Grimmer and Stewart (2013) give four principles for coding text with computers.
  • Every model of language is wrong, but some are useful.
  • Computer methods help people read more text. They do not replace careful reading.
  • No method is best for every task.
  • Validate, validate, validate. Check every measure against careful human reading.
• The rest of today applies the fourth principle. Whatever method codes the text, you check its codes against hand codes in the same way.

Model APIs

• In Week 4 your script sent a request to the Federal Register API. The request named an agency and a year. The API sent back documents.
• A model API works the same way. Your script sends a request, and the API sends back the model’s answer.
• Here the request contains your codebook and one abstract. The answer is the model’s category for that abstract.

Commercial and open-source models

• Commercial models run on the company’s servers. You pay for each call, and your text goes to the company. Each company documents its API:
  • OpenAI API (GPT models)
  • Anthropic API (Claude models)
  • Google Gemini API
• A chat subscription such as ChatGPT Plus does not include API access. The API is billed separately.
• Open-source models can be downloaded and run by anyone.
  • Hugging Face hosts most open-source models, such as Llama, Qwen, and Gemma.
  • Ollama runs these models on your own computer and gives them a local API. Your data stays on your computer.
  • They are free to run, but usually less capable than the best commercial models.
• Find the API documentation for the model you want to use. Give the link to the agent when it writes the script.

The coding scripts

• Ask the agent to write two scripts. Each one codes every abstract and saves the category for each abstract in a table.
• The dictionary script uses keyword lists. Read the 10 pilot abstracts and list keywords for each category.
  • Looser: “withdraw”, “delay”, “rescind”.
  • Stricter: “require”, “establish”.
  • Technical: “correction”, “technical amendment”.
• The language model script sends your codebook and one abstract at a time to a model through its API. Make two versions.
  • Zero-shot: the prompt contains your codebook only.
  • Few-shot: add 2 or 3 of the pilot abstracts, with your codes, as examples.
• Run the scripts on the 10 pilot abstracts and compare their codes with your hand codes. Where they differ, read the abstract again. Revise the codebook or the keyword lists.
• Each call to a commercial API costs money. You pay for the text you send and the text you get back. Ask the agent to estimate the cost before you run the script.
• Record the exact model name and version. Companies update their models, and the same prompt can give different codes after an update.

Testing on new abstracts

• Every method needs the same test, whether it is a dictionary, a supervised model, or a language model.
• You revised the codebook and the keyword lists while looking at the 10 pilot abstracts. Each method’s score on those 10 abstracts is too high.
• Test each method on new abstracts that you did not use to write the codebook. These abstracts are the held-out sample.
• The steps:
  • Freeze the codebook, the prompts, and the keyword lists. After this point you do not change them.
  • Draw 20 new abstracts at random.
  • Code them by hand before you run any script.
  • Run each script on them and compare its codes with yours.
• Research projects hand-code about 100 to 300 documents for this test. Twenty abstracts are enough to practice.

The confusion matrix

• Take a classic example. A university has 10,000 students, and 10 of them have COVID. A test predicts who is infected.
• A confusion matrix compares the predictions with the truth. Every student falls into one of four cells:
  • True positive (TP): an infected student whom the test calls infected.
  • False negative (FN): an infected student whom the test misses.
  • False positive (FP): a healthy student whom the test calls infected.
  • True negative (TN): a healthy student whom the test calls healthy.

• For text coding, the truth is your hand codes. The test is the codes from a script, such as the dictionary or the language model.

Accuracy

• Accuracy is the share of all cases that the test gets right.
• Accuracy = (TP + TN) / (TP + FN + FP + TN).

Why accuracy misleads

• Take a test that calls every student healthy.

• Its accuracy is 9,990 / 10,000 = 99.9%.
• It finds none of the 10 infected students.
• When one category is rare, a test can reach high accuracy by never predicting it. This is called class imbalance.
• The test that always predicts the most common category is the majority-class baseline. Compare each method’s accuracy with this baseline.

Precision and recall

• Precision: of the students the test calls infected, what share are infected?
  • Precision = TP / (TP + FP).
• Recall: of the students who are infected, what share does the test find?
  • Recall = TP / (TP + FN).
• For the test that calls everyone healthy, recall is 0 / 10 = 0%. Its precision cannot be computed, because it never calls anyone infected.

The precision–recall trade-off

• Test 1 calls 5 students infected, and all 5 are infected.

• Accuracy = (5 + 9,990) / 10,000 = 99.95%.
• Precision = 5 / 5 = 100%. Every student it calls infected is infected.
• Recall = 5 / 10 = 50%. It misses half the infected students.

• Test 2 uses a lower bar for calling a student infected. It calls 13 students infected.

• Accuracy = (9 + 9,986) / 10,000 = 99.95%, the same as Test 1.
• Precision = 9 / 13 = 69%.
• Recall = 9 / 10 = 90%.
• Raising recall usually lowers precision. This is the precision–recall trade-off.
• Which one matters depends on the policy question.
  • To list every rule that loosened a requirement, you need high recall.
  • To report how many rules loosened a requirement, you also need high precision.
• In class: name a policy task where precision matters more, and one where recall matters more.

F1

• The F1 score combines precision and recall into one number.
• F1 = 2 × precision × recall / (precision + recall).
• Test 1: 2 × 1.00 × 0.50 / (1.00 + 0.50) = 0.67.
• Test 2: 2 × 0.69 × 0.90 / (0.69 + 0.90) = 0.78.
• F1 is 1 for a perfect test and 0 for a useless one. It is low when either precision or recall is low.
• A common target: F1 of 0.7 or more on the category you care about, before you code the full collection.

More than two categories

• Your codebook has three categories. Compute precision and recall for each category separately. For each one, treat that category as “positive” and the other two as “negative”.

• Cells on the diagonal, in bold, are agreements. Accuracy is 16 / 20 = 80%.
• Precision for Stricter: the script coded 8 abstracts Stricter, and 6 of them match yours. Precision is 6 / 8 = 75%.
• Recall for Stricter: you coded 7 abstracts Stricter, and the script found 6. Recall is 6 / 7 = 86%.
• The off-diagonal cells show which category the script mistook for which. Read the abstracts behind those cells.

Comparing the methods

• Compare the held-out results of the dictionary, zero-shot, and few-shot coding. If you tried more than one language model, compare those too.
• Look closely at precision and recall for the categories that matter most for your question.
• The dictionary method gives a simple benchmark. If the language model cannot beat a list of keywords, it is not worth its cost.

Consistency checks

• A good measure gives the same codes when nothing important changes. Check this in three ways.
  • Same script, same codebook: run the held-out sample twice. The two runs should give the same codes.
  • Different wording: your zero-shot and few-shot versions are two wordings of the same task. Small changes in wording can change the codes. This is called prompt sensitivity.
  • Different language model: run the held-out sample with a model from another company, or with an open-source model.
• If the codes change a lot, read the abstracts whose codes changed. The codebook may be unclear for those abstracts.

Cost and data rules

• Estimate the cost of coding your full collection.
• Check whether your data may be sent to a company’s servers. Restricted data needs a local model or an approved arrangement.
• Write down the chosen method, the model name and version if you chose a language model, the final codebook, and the held-out results.

Finding categories in the text

• So far you started from a codebook with fixed categories. Sometimes you do not know the categories yet. For example, you may have thousands of public comments and no idea what they discuss.
• A topic model finds groups of words that often appear together in the same documents. Each group is a topic. Each document gets a share of each topic.
• The model does not name the topics. A researcher reads the top words and some documents for each topic, then names it.
• Three common topic models:
  • LDA (latent Dirichlet allocation) is the standard topic model.
  • The structural topic model lets topic shares vary with facts about each document, such as its date or the respondent’s party. Roberts and colleagues used it on open-ended survey answers.
  • TopicGPT asks a large language model to read documents and propose topics. Each topic gets a name and a short description. The model then assigns each document to a topic.
• The topics can change when you rerun the model or change the number of topics.
• Use a topic model to draft the categories of a codebook. Then code with a script and test on a held-out sample, as above.

Text coding in your project

• Does your project need text coding like today’s, or could it use it?
• If so, which text would you code, and which variable would you build from it?
• Many projects rely mostly on numbers. Coding plays a small part in those projects.

Weekly memo · individual

• Write 1–2 specific observations about coding text. Use an actual coding decision or disagreement to explain what worked, what needed correction, and whether the resulting variable is useful for your policy question.
• If you have not finished the coding exercise, you may skip this memo and write one memo for Weeks 5 and 6 together, due Oct 29.
• Aim for 150–200 words total.
• Describe what you actually tried and observed in your own words. Do not use AI to write the memo.
• Submit to Canvas Discussions. See assignments and grading.

Due Oct 22 before class.