Is Ten Percent a Lot?

In 2023 I was in Tasmania with a friend, trying to climb the Candlestick, a thin pillar of rock standing in the sea off Cape Hauy. He swam across the channel to set us up on the route, and somewhere in the crossing he dropped the rope. With it went any realistic way back. The water under him was choppy and cold, and he was getting hypothermic.

I abseiled down the cliff on the mainland side and threw him a line. He clipped in and zip-lined back across the gap to safety.

I’ve had a lot of moments like that. A night stuck on the side of a cliff. Racing the daylight in the Alps. Spindrift so thick I couldn’t see my hand in front of my face. I’ve never had a serious incident, because I’ve learnt to manage risk, quantify it and think quickly on the spot.

So when I heard an AI researcher say there’s more than a 10% chance AI kills everyone, I started doing the maths.

Rain and falling

This month Evan Hubinger, who leads alignment work at Anthropic, wrote publicly that he personally puts the chance of AI killing all humans at more than 10% within the next decade. He isn’t an outlier. The largest survey of AI researchers also published its latest results this month, and the median answer on AI causing human extinction, or something as permanent, was 10%.

Is 10% a lot? It depends on what’s at stake.

If the forecast says 20% chance of rain, I don’t bother with an umbrella. If a route had a 1% chance of me falling to my death, I wouldn’t climb it. The first number is twenty times bigger and matters far less, because I can dry off.

High consequence, low probability is a different thing entirely to low consequence, high probability, and extinction is as high-consequence as it gets.

What we build for

When the stakes are that high, we normally demand very small numbers.

The systems on an airliner that could bring it down are each designed to fail less than once in a billion flight hours. At 22, I spent a summer at the European Space Agency working out the fuel budget for a satellite called FLEX. My answer was that it would have enough propellant for its mission with 99.7% confidence. A 0.3% chance of running out felt low enough for an uncrewed satellite. With people on board, I’d have wanted more margin.

In a 2025 Australian survey, 94% of people said AI should be held to safety standards at least as strict as commercial aviation. Measured against those standards, 10% isn’t in the same universe. Neither is 1%.

Where do the numbers come from?

My PhD was built on Bayesian statistics, and this is a Bayesian problem.

Nobody has watched humanity end. There’s no record of past extinctions to count, the way there is for plane crashes or floods. So every one of these numbers is a degree of belief: someone’s best judgement, written as a percentage. They get produced in four main ways.

Surveys. Researchers are asked for their number and you report the middle. In the latest survey, half the respondents gave 10% or more. But only about one in ten of the people emailed replied, and three slightly different wordings of the same question gave medians anywhere from 5% to 10%.

Forecasting tournaments. In 2022 the Forecasting Research Institute put AI experts and “superforecasters”, people with proven records predicting world events, through months of structured argument. The AI experts put extinction from AI by 2100 at 3%. The superforecasters said 0.38%. After all that discussion, neither group moved much.

Breaking the argument into steps. The philosopher Joe Carlsmith split the case for AI catastrophe into a chain of steps, gave each a probability and multiplied them together. He got about 5%, and later revised it to more than 10%.

Personal judgement. This is what you hear on podcasts. In the recent Diary of a CEO debate, four guests gave numbers from roughly zero to 99%, sitting at the same table. Honestly, they all made good points.

Critics call these “vague intuitions... translated into pseudo-precise numbers”, and they have a point. Nobody will ever be scored on an extinction forecast, so there’s no feedback to separate good forecasters from bad ones. A probability here is a summary of what someone believes, given what they know. The spread matters as much as the median: informed, serious people range from close to zero to near certain, and we don’t know who’s right.

Knowing a risk and choosing it

On the rock, understanding a risk lowers it. When I tap a loose-looking hold before pulling on it, I’m actively changing my odds. That’s the trust side of risk: trust in your own ability to understand it and deal with it.

Familiarity has a catch, though. In 1993 Derek Hersey, described in the American Alpine Club’s accident report as one of the best free soloists in the world, died on the Steck-Salathé in Yosemite, probably on wet rock. The report called the route “an easy day” for him. The club notes that free-solo deaths usually happen “on well-trafficked moderate routes”.

With AI it’s the other way round. Most of the people carrying this risk never chose it. And the people closest to the technology tend to give the higher numbers: in that tournament, the AI experts put the risk about eight times higher than the superforecasters did.

How we’ve handled this before

During the Cuban Missile Crisis in 1962, John F. Kennedy later told his adviser Ted Sorensen he’d put the odds of nuclear war at “somewhere between one out of three and even”. The closest call came far from the White House, on a Soviet submarine called B-59. US ships were dropping practice depth charges to signal it to surface. The captain called for a nuclear torpedo launch, and one officer, Vasili Arkhipov, talked him down.

My number

If Steven Bartlett turned to me, I’d say I don’t know. We don’t have enough information to make a prediction that means much.

The space industry has a habit of adding margin “just in case”, stacking worst case on worst case instead of quantifying the risk properly. That summer at ESA, I was arguing for the statistics over the padding.

The difference with AI is that it can’t be undone. A lost satellite gets rebuilt. Even if the real number was 0.01%, that isn’t a risk I’d want to take on behalf of everyone alive.

What excites me most about AI is what it could do for science and medicine, and slowing down has a real cost. But right now we’re in an accelerating car with very little steering wheel and no real brake. I’d rather fit both, through an international body that can hold every lab and every country to the same rules, so we get across the finish line in one piece.

This year’s International AI Safety Report, written by more than a hundred experts, describes the bind well. Acting on limited evidence risks bad policy, “but waiting for stronger evidence could leave society vulnerable to the risks.”

The question I ask

At events, I usually ask people the same thing: what are you most excited about with AI, and what are you most scared of?

Most people are excited about productivity. Very few buy the idea of an AI utopia. And when they get to the second half, most people say a version of the same thing: they’re scared of losing control of it, and they don’t trust the intentions of the people driving.

What’s yours?