Introduction to Data and Prediction
The first true information revolution began not with the microchip, but with the printing press. In 1440, Johannes Gutenberg’s invention transformed books from expensive luxury items into accessible tools for the masses. Because information grew much faster than the human ability to process it, the era was defined by mass confusion and sectarianism.
This historical pattern of technology outpacing understanding repeats in the modern era. During the 1970s and 1980s, the dawn of the computer age promised a new frontier of scientific and economic progress. Instead, it produced a productivity paradox where massive investments in technology failed to yield tangible gains because the complex computer models were based on flimsy assumptions.
Today, we are submerged in the era of Big Data, generating quintillions of bytes of information every single day. There is a seductive belief that the sheer volume of data will eventually make the scientific method obsolete by allowing the numbers to speak for themselves. This is a dangerous illusion because data has no voice of its own; humans are the ones who imbue it with meaning, often in highly selective ways.
Our biological makeup complicates our relationship with this wealth of information. Humans are evolutionarily wired to be pattern-recognition machines, a trait that helped our ancestors survive by identifying threats in the wild. However, in an information-saturated world, this instinct leads us to see meaningful signals in random noise. We are prone to information overload, a state where we simplify a complex world by picking out the data points we like and ignoring the rest.
The consequences of failing to distinguish between truth and distraction are visible in our most significant modern crises. We missed the warning signs leading up to the September 11 attacks not because we lacked data, but because we lacked a proper theory to connect the dots. Similarly, the 2008 global financial crisis was fueled by a blind faith in mathematical models that were built on fragile, self-serving assumptions. In the realm of biomedical research, the problem is so pervasive that nearly two-thirds of positive findings in peer-reviewed journals cannot be replicated.
Despite these failures, there are clear paths toward progress. In fields like baseball and weather forecasting, we have seen remarkable success by combining human judgment with rigorous data analysis. When analysts design systems to forecast player performance, the goal is not to find a perfect answer, but to outline a range of probable outcomes.
The solution to our prediction problem requires an attitudinal shift toward probability, embodied by Bayes’s theorem. This approach implies that we must think differently about our ideas, becoming more comfortable with uncertainty and more critical of the assumptions we bring to any problem. We must learn to separate the underlying truth from the distracting, irrelevant data that surrounds it. Prediction is not about achieving perfect certainty, but about the relentless pursuit of objective reality through careful observation.
Full summary available in the Readsome app
