Introduction: The Rise of Unfair Mathematical Systems

Cathy O'Neil grew up with a deep love for the order and logic of mathematics. As a child, she spent her time breaking down license plate numbers into their basic elements, and she eventually turned this passion into a career as a math professor. Later, she moved into the world of high finance to work for a major hedge fund. However, the 2008 financial crisis changed her perspective on the field. She realized that the mathematical formulas she once admired were being used to fuel global economic problems, such as the housing market collapse and mass unemployment.

In the years following the crisis, these mathematical techniques expanded into almost every part of daily life. Companies began using large sets of data to predict how people would behave as students, workers, and consumers. These systems were marketed as fair and objective because they used cold numbers instead of biased human judgment. However, O'Neil discovered that these models often reflected the prejudices and misunderstandings of the people who created them. She calls these harmful models Weapons of Math Destruction because they are secret, they affect millions of people, and they are almost impossible to challenge.

A clear example of this occurred in the Washington, D.C., school system through a teacher evaluation tool called IMPACT. The city wanted to improve schools by identifying and firing low-performing teachers. Sarah Wysocki, a fifth-grade teacher, was well-loved by her students' parents and received glowing reviews from her principal. Despite this, she was fired because an algorithm gave her a low score. The system tried to calculate her effectiveness by looking at her students' standardized test scores, but it failed to account for the many outside factors that influence a child's performance, such as poverty or family issues.

These types of mathematical models often lack a way to learn from their mistakes. In a healthy system, errors are used to improve the formula. However, when the school district fired teachers like Wysocki, they had no way of knowing if they were actually losing good employees. The system simply defined the fired teachers as failures and moved on, creating a destructive loop where the model's results are used to justify its own existence. Similar loops happen in employment, where companies use credit scores to screen job applicants, keeping people with low scores in poverty and seemingly confirming the model's original bias.

These secret systems tend to punish the poor while the wealthy continue to benefit from personal human interaction. While a rich student might get into a school based on a face-to-face interview, a person applying for a low-wage job is often filtered out by a machine. When victims of these models try to protest, they are often told that the math is too complex to explain. For many companies, the only feedback that matters is whether the model makes money, and if profits are high, they assume the model is working perfectly while ignoring the individuals whose lives are damaged.