I use artificial intelligence every day. I write about it constantly, and I think AI could help cure diseases, make technology more accessible, and solve problems humans have struggled with for decades.
I also recently learned there is a term for the probability that AI kills us all. It is called P(doom), and I really wish I were joking.
P(doom) is essentially someone’s estimate of the probability that artificial intelligence causes an existential catastrophe. Depending on whom you ask, the number can be tiny, enormous, or somewhere in between.
Think about how insane that is for a moment. We are racing to put AI into our phones, computers, cars, workplaces, schools, hospitals, and pretty much everything else, while some of the people who understand this technology best are seriously discussing the possibility that it eventually destroys humanity.
Geoffrey Hinton, one of the pioneers of modern AI, has repeatedly put his estimate at around 10 to 20 percent. When CNN recently confronted him with that number, Hinton made an important point: anyone assigning probabilities to something like this is basically making a guess because humanity has never dealt with anything comparable.
Anthropic CEO Dario Amodei has offered similarly uncomfortable numbers, although his comments require some context. An older figure of 10 to 25 percent has sometimes been described as his P(doom), but Amodei later said that characterization was inaccurate and that he had actually been talking about the probability of civilization getting substantially derailed, not necessarily AI killing everyone.
Amodei has since been asked more directly about P(doom). At the Axios AI+ DC Summit in 2025, he said there was a 25 percent chance that things go “really, really badly”, compared with a 75 percent chance that things go really well.
Those qualifications matter because P(doom) isn’t some scientific measurement sitting on a laboratory instrument. Different people can be estimating different outcomes, over different periods, under different assumptions, and then those numbers get tossed together online as though everyone answered the same question.
That makes me skeptical of any precise P(doom) number. If somebody tells me there is exactly a 17 percent chance AI wipes out humanity, I want to know what could possibly justify that level of precision.
The underlying concern is harder for me to dismiss. Hinton isn’t arguing that he ran the numbers and discovered humanity has exactly a one-in-five chance of extinction. In fact, he has explicitly described estimates like his as a “wild guess” or gut feeling because there is no real evidence available to calculate such a probability.
The possible ways things could go horribly wrong aren’t limited to science fiction, either. Just this week, I wrote about Anthropic warning that increasingly capable AI could lower barriers around dangerous biological research, while researchers have also raised concerns involving cyberattacks, critical infrastructure, manipulation, autonomous systems, and eventually AI systems humans simply cannot control.
None of this requires Skynet. There doesn’t need to be an evil robot that wakes up one morning and decides it hates humanity.
That version actually seems easier to understand than some of the alternatives. The scarier possibility is a system doing something catastrophic because humans gave it enormous capabilities, connected it to important systems, and discovered too late that we couldn’t reliably control what it was doing.
Maybe none of that happens, and I certainly hope it doesn’t. I also don’t believe a 20 percent P(doom) means humanity literally has a scientifically established one-in-five chance of disappearing.
In fact, the inability to calculate the number may be the most interesting thing about P(doom). We are trying to attach percentages to an event for which humanity has no precedent, no useful historical data, and no way to conduct an experiment and try again if we get the answer wrong.
Yet I can’t dismiss the idea completely, either. We have somehow reached a point where credible people who helped build the foundations of modern AI are openly discussing human extinction as one possible outcome of making this technology increasingly powerful.
I’ll probably continue using AI every day. I’m excited about where it is going, and I don’t want technological progress to stop because of hypothetical disasters that may never happen.
But P(doom) has gotten stuck in my head. If Hinton and the other people warning about AI are completely wrong, someday we’ll look back at all this doom talk and laugh.
I really hope they’re wrong. More importantly, I hope we’re around long enough to laugh about it.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz