Understanding the risk posed by AGI is like standing at the foot of an accelerating train track; you can debate the exact speed or the final destination, but the wisdom lies in recognizing the immediate need to step off the tracks before the exponential approach leaves you behind.
This is not science fiction. It is not fringe thinking. The people saying this most loudly are the very people who built the train: Hinton, Bengio, Amodei, Altman, Musk. When the architects of a technology begin publicly questioning whether it can be controlled, that is a data point worth weighting heavily.
What Changed in 2023
For most of my career in AI — and I have been in this field since before it was called AI — the dominant sentiment was impatience. Progress was slow. Neural networks were repeatedly declared dead and repeatedly revived. The 2012 ImageNet shock woke people up, but the full implications took years to percolate.
Then GPT-4 happened. Then Claude. Then Gemini. And the pace stopped feeling slow.
The change is not just in benchmark performance. It is in generality. Previous AI systems were narrow specialists. What we have now are systems that can reason across domains, write code, interpret images, synthesise medical literature, and engage in strategic planning — all in the same session. The general-purpose nature of these systems is what changes the risk calculus.
The Alignment Problem, Stated Simply
We are building systems that are becoming more capable than humans in an increasing number of cognitive tasks. The alignment problem is simply: how do we ensure those systems reliably pursue goals that are good for humans?
This turns out to be extraordinarily hard. Not because we cannot write the rules, but because:
- The rules are incomplete — we cannot specify every situation in advance
- The systems learn from human data, which contains every human bias, contradiction, and pathology
- More capable systems find more creative ways to achieve goals, including ways we didn't anticipate
Geoffrey Hinton left Google in 2023 specifically to speak freely about this. His concern: digital intelligence may eventually conclude that eliminating competition for resources — including human competition — is instrumentally useful for achieving its goals. Not because it is evil. Because optimisation is ruthless.
What Should Be Done
I am not a doomer. I believe AI is the greatest lever for human progress since the printing press, probably since agriculture. Amodei's essay on what AI could do for medicine alone is not hyperbole — it maps out a genuinely plausible path to compressing decades of biomedical progress into a few years.
But the upside and the risk are not mutually exclusive. The appropriate response is not to stop — it is to slow down enough to build proper alignment infrastructure, which is currently running far behind capability development.
Concretely:
- Interpretability research needs ten times the funding it currently receives. We need to understand what these systems are actually doing internally, not just what they output.
- Regulatory frameworks need to be built by people who understand the technology — not exclusively by lawyers and politicians.
- International coordination is not optional. A unilateral slowdown by one country simply hands capability advantage to another. This requires treaty-level agreements.
None of this is easy. But we have navigated existential-scale technical risks before — nuclear weapons, gain-of-function research, ozone-depleting chemicals — and built imperfect but functional governance frameworks for each.
The train is accelerating. The question is not whether to build tracks, but whether we are building them fast enough.