Human Compatible

The Promise and Peril of Artificial Intelligence

Stuart Russell observes that the field of artificial intelligence has always aimed to create machines that match or exceed human capabilities. However, there has been surprisingly little discussion about what happens if that goal is actually met. If researchers succeed in creating superintelligent systems, it would represent a massive shift in human civilization. This transition could help solve major global catastrophes or extend human life, but it could also be humanity's final event if not managed with care.

The formal quest to build intelligent machines began in 1956 when scientists proposed that every feature of learning could be simulated. Early milestones included programs that could play checkers, but these successes were followed by periods of disappointment when the technology failed to handle complex real-world tasks. Over the decades, the field shifted from simple rule-based systems to complex mathematical models involving probability and statistics. By 2011, deep learning techniques began solving long-standing problems in speech and image recognition, attracting billions of dollars in investment.

The path toward superhuman intelligence requires several major breakthroughs that are difficult to forecast. History shows that scientific progress often happens much faster than experts anticipate, such as when physicists dismissed the possibility of atomic energy just one day before the nuclear chain reaction was invented. This sudden shift from impossible to solved serves as a warning against assuming that advanced artificial intelligence is too far in the future to worry about. Even if success is not immediate, the potential impact is so great that preparing for it now is an absolute necessity.

For decades, the field of artificial intelligence has operated under a specific definition of success called the standard model. In this framework, humans provide a fixed objective, and the machine optimizes its behavior to reach that specific goal. While this works for simple tasks, it becomes dangerous as machines become more capable because humans are often unable to specify their goals perfectly. If a machine is smarter than a human and pursues a flawed objective, it will pursue that goal relentlessly, leading to unintended and potentially disastrous consequences.