Algorithmic Accountability: What Biased Sentencing Data Reveals About the Limits of Mathematical Fairness

Mathematics has long carried an aura of objectivity. Numbers, the reasoning goes, do not harbor prejudice. Yet when those numbers are trained on historically inequitable data, the resulting models can encode and even amplify the very biases they were presumed to eliminate. Nowhere is this tension more consequential than in the American criminal justice system, where algorithmic risk-assessment tools now influence decisions about bail, sentencing, and parole for hundreds of thousands of people each year.

For students pursuing careers in data science, software engineering, or public policy, the story of algorithmic sentencing is not a cautionary footnote. It is a foundational lesson in what it means to build mathematical systems responsibly.

The Rise of Risk Scores

Over the past two decades, courts across the United States have increasingly adopted what are known as risk-assessment instruments — statistical tools designed to predict the likelihood that a defendant will reoffend. In theory, these instruments offer a more consistent, evidence-based alternative to the unchecked discretion of individual judges. In practice, their track record has been deeply troubling.

Perhaps the most scrutinized of these tools is COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), used in states including Florida, Wisconsin, and New York. In 2016, the investigative outlet ProPublica published an analysis of COMPAS scores assigned to more than 7,000 defendants in Broward County, Florida. The findings were stark: Black defendants were nearly twice as likely as white defendants to be incorrectly flagged as high risk for future offenses. White defendants who did go on to reoffend were more frequently mislabeled as low risk.

The developers of COMPAS disputed the methodology behind ProPublica's analysis, and a subsequent exchange in the academic literature illustrated just how contested the mathematics of fairness can be. Researchers demonstrated that it is mathematically impossible for a predictive model to simultaneously satisfy several common definitions of fairness when base rates differ across demographic groups — a result known informally as the "impossibility theorem" of algorithmic fairness. The implication is unsettling: there is no neutral formula. Every design choice encodes a value judgment about whose errors matter most.

What the Data Inherits

Understanding why these models produce disparate outcomes requires tracing the data they are built upon. Risk-assessment instruments typically incorporate factors such as prior arrest history, neighborhood characteristics, employment status, and family criminal history. Each of these variables is a proxy — a stand-in for something harder to measure — and each carries the imprint of decades of unequal policing and prosecution.

Consider arrest history. If law enforcement has historically concentrated resources in predominantly Black and low-income neighborhoods, residents of those communities will accumulate more arrests per capita — not necessarily because they commit more crimes, but because they are watched more closely. A model trained on arrest data will interpret this surveillance disparity as a signal of individual risk, effectively laundering structural inequality into a mathematical prediction.

This is not a bug unique to criminal justice algorithms. It is a fundamental challenge in any domain where machine learning models are trained on data generated by imperfect human institutions. The lesson for students is that statistical fluency must be paired with sociological awareness. Knowing how to build a logistic regression model is insufficient preparation for deploying one responsibly.

Auditing as a Mathematical Practice

The emerging field of algorithmic auditing offers one response to these challenges. Auditors — often teams combining data scientists, legal scholars, and domain experts — systematically test models for disparate impact, probe their inputs for proxy discrimination, and evaluate whether stated accuracy metrics obscure group-level disparities.

Several universities and research organizations have developed open-source toolkits to support this work. IBM's AI Fairness 360, Google's What-If Tool, and Microsoft's Fairlearn library each provide frameworks for measuring and mitigating bias in machine learning systems. These tools are increasingly being incorporated into university-level data science curricula, and there is a strong argument for introducing their underlying concepts even earlier — at the high school level, where students are first learning to interpret statistical claims.

For educators, algorithmic auditing offers a compelling pedagogical bridge. It connects abstract concepts like false positive rates, confusion matrices, and conditional probability to decisions that affect real people. It asks students not only to compute but to question: Who designed this model? On what data was it trained? Whose definition of accuracy does it optimize?

Teaching the Ethics of the Equation

The mathematical education community in the United States has begun to grapple with these questions, but progress has been uneven. Advanced Placement Computer Science courses have expanded access to programming instruction, and data science courses are proliferating at the secondary level. Yet few of these curricula devote sustained attention to the ethical dimensions of algorithmic systems — the point at which mathematics intersects with power.

A more robust approach would integrate what some scholars call "critical data literacy" into STEM education from early on. This means teaching students to ask not only how a model works but what it was designed to do, who benefits from its deployment, and what recourse exists when it fails. It means examining real cases — COMPAS, predictive policing systems like PredPol, credit-scoring algorithms — as mathematical objects worthy of rigorous analysis.

It also means preparing students for careers in which these questions will be professionally unavoidable. As governments and corporations deploy increasingly sophisticated automated decision systems, demand is growing for technologists who can navigate the intersection of quantitative rigor and ethical accountability. Organizations including the Partnership on AI, the Algorithmic Justice League, and the AI Now Institute are actively shaping standards and practices in this space — and they need graduates who can contribute meaningfully to both the technical and normative dimensions of the work.

The Equation Is Never Just the Equation

The deeper lesson of algorithmic sentencing is one that applies across every domain in which mathematics is used to make consequential decisions: a model is always a simplification, and the choices embedded in that simplification are never value-neutral. When students understand this — when they can look at a dataset and ask what it does not capture, or examine a loss function and ask whose losses it discounts — they become more than competent technologists. They become responsible ones.

The goal of mathematics education has never been merely to produce students who can solve equations. It has been to produce students who understand what the equations mean. In an age when algorithms adjudicate liberty, that distinction has never mattered more.