Algorithmic bias

Today

• See how a classifier’s errors can fall more often on one group.
• Compare fairness criteria and the trade-offs between them.
• Compute one confusion matrix per group for a risk score used in courts.
• Discuss where your own project could make more errors for one group than another.

Classification errors by group

• In Week 5 you coded Federal Register abstracts with a dictionary and with a language model. You checked both against your hand codes on a held-out sample.
• The confusion matrix showed which abstracts each method got wrong. It pooled all the abstracts together.
• A classifier can be accurate on average and still make most of its errors on one group of people.
• Two examples from text classification:
  • Liang and colleagues (2023) tested detectors of AI-written text. The detectors labeled more than half of the essays by non-native English writers as AI-written. People had written all of these essays.
  • Davidson, Bhattacharya, and Weber (2019) trained classifiers on five sets of tweets labeled for hate speech and abusive language. Every classifier flagged tweets written in African-American English as abusive more often than tweets in Standard American English.
• When a system’s errors fall more often on one group, people call it algorithmic bias.
• To say that a classifier is biased, you have to say which rate you compare across groups. The next section lists the main comparisons.

False positive rate and precision

• To say that a classifier is biased, you compute one rate from the confusion matrix for each group and compare. A fairness criterion names the rate that should be equal.
• The example is the study of detectors of AI-written text by Liang and colleagues (2023). They ran seven detectors on 91 essays by non-native English writers (TOEFL essays) and 88 essays by US eighth graders. People wrote all of these essays.

• The false positive rate is new today. You met precision in Week 5.
• The two rates can disagree. The COMPAS case below shows how.

The COMPAS risk score

• COMPAS is a commercial risk assessment tool. The company Northpointe made it. Northpointe is now called Equivant.
• A defendant answers a questionnaire when booked into jail. COMPAS turns the answers and the person’s record into a score from 1 to 10.
• A higher score means a higher predicted chance of a new offense within two years. The scores are grouped into three labels: Low, Medium, and High.
• Courts use the score in decisions about bail and sentencing.
• In State v. Loomis (2016), the Wisconsin Supreme Court ruled that a judge may consider a COMPAS score at sentencing, with cautions. The defendant had argued that he could not examine how the private tool computed his score.

ProPublica’s analysis and Northpointe’s reply

• In 2016 ProPublica obtained the COMPAS scores of people scored in Broward County, Florida, in 2013 and 2014. It then checked who was arrested again within two years.
• ProPublica compared false positive rates. Among defendants who were not arrested again, about 45% of Black defendants had been labeled Medium or High. The share for white defendants was about 23% (ProPublica’s methods article).
• Northpointe compared precision. Among defendants with the same label, Black and white defendants were arrested again at similar rates.
• Corbett-Davies and colleagues argued that both claims can be true at the same time. The base rate of arrest differs between Black and white defendants in the data.

COMPAS errors by group

• ProPublica published the risk scores it analyzed and its code in its COMPAS analysis GitHub repository.
• Compute the base rate, the precision, and the false positive rate for Black and white defendants. The steps are on the right.

Your results

• Fill in this table with your filtered sample. In the data, Black defendants are coded African-American and white defendants are coded Caucasian.

Base rates and the two criteria

• Here are the two numbers from the class data (6,172 defendants; Medium or High counts as flagged):

• Northpointe looked at precision. It is similar for the two groups, 65% and 60%. A Medium or High label means about the same chance of a new arrest for a Black or a white defendant. By this measure, Northpointe called the score fair.
• ProPublica looked at the false positive rate. Among defendants who were not arrested again, 42% of Black defendants were flagged, against 22% of white defendants. By this measure, ProPublica called the score unfair.
• Both calculations are correct. They measure different things.
• The two groups have different base rates, 52% and 39%. To keep precision similar, the score flags a larger share of Black defendants, 58% against 33%. More of the Black defendants who were not arrested again are then flagged too. That raises their false positive rate.
• Lowering that false positive rate would mean flagging fewer Black defendants. Precision would then no longer be equal.
• When base rates differ and the score is not perfect, both numbers cannot be equal across groups at the same time. You have to choose which one matters for the decision.
• Your own numbers may differ slightly from those in the ProPublica article, which used a different sample.
• Kleinberg, Mullainathan, and Raghavan (2016) and Chouldechova (2017) proved this result.

Bias in your own project

• Could your project work differently for different groups? For example:
  • a coding or classification step that is wrong more often for one group,
  • data that miss one group or record it less accurately,
  • a policy that helps or burdens one group more than another.
• If so, how would you diagnose it?
  • Name the groups.
  • Name the rate you would compare, such as the false positive rate.
  • Name the data you would need to compute it.

Reading · due Nov 5 before class

• Read these four readings. We will discuss them in class. Together they are about 25 pages.
• Two articles on COMPAS:
  • Angwin et al., “Machine Bias” (ProPublica, 2016). About 10 pages.
  • Corbett-Davies, Pierson, Feller & Goel, the response to ProPublica (Washington Post, 2016). 5 pages.
• Two readings on AI in the federal government:
  • The White House AI memo, issued by the Office of Management and Budget (Accelerating Federal Use of AI through Innovation, Governance, and Public Trust, M-25-21, April 2025), pp. 14–17 and 21–23: the definition of high-impact AI, the minimum risk-management practices, the uses presumed to be high-impact, and the list of risk factors.
  • Engstrom, Ho, Sharkey & Cuéllar, Government by Algorithm (2020), pp. 6–8, the executive summary.
• As you read the COMPAS articles, find the fairness criterion that each one uses.