Taking Chances - Affix Space

Fear sells

There is a reason these numbers get repeated. A percentage chance of human extinction is the best advertising a technology has ever had. It says, in one breath, that the product is powerful enough to end the world and that the person selling it is honest enough to admit it. Doom is a pitch. In 2023 Sam Altman sat in front of Congress and asked for his own industry to be regulated, a move that doubles as a claim that the technology is powerful enough to need it. The same founder who puts his p(doom) at 25% is also raising money on the premise that the thing is about to change everything, and those two messages prop each other up.

Watch the story move and the marketing shows through. For two years the line was that AI would wipe out white-collar work, that whole professions would be gone by next year. Now that the layoffs have been smaller than promised, the same companies have started reassuring everyone that AI will not destroy jobs after all, that it only makes workers more productive. Both versions sell. The first sells urgency, the second sells comfort, and the product underneath is identical.

The reassurance arrives with an economics papers backing it up. The favorite is Jevons paradox. When a resource gets cheaper to use, we do not use less of it, we use far more. Cheaper steam engines burned more coal, not less. Cheaper code, the argument goes, means more software gets built, so demand for the people who build it climbs instead of falling. It might be true. It is also exactly what you would say if you needed the people whose work you are automating to keep feeling safe.

None of this means the risk is fake, and that is the trap. Treating extinction odds as a marketing dial does two kinds of damage. It inflates expectations until the bubble has to pop, and it burns the credibility we would need if one of these warnings ever turns out to be the true one. The numbers in the game above might be hype. They might also be right. We cannot yet tell, and selling them as either one is dangerous.

The other loaded guns

AI gets the headlines currently, but it is one gun among several. We are running a few of these bets at once and each is large enough to end most of what we care about. They differ in some details, like how fast they would act and how much warning we would get.

Nuclear weapons

This is the oldest gun. Around twelve thousand warheads exist, and roughly a third of them sit ready to launch within minutes. The danger has never been only a deliberate first strike. It is the false alarm: a training tape loaded into a live system, a flock of geese read as incoming missiles, sunlight on clouds mistaken for a launch plume.

A nice story often cited on documentaries about nuclear weapons is the story of Stanislav Petrov. In 1983, as a Soviet officer, he saw the early-warning system report five US missiles inbound. He decided, against protocol, that it was a glitch, and he was right. The system had mistaken sunlight reflecting off clouds for rocket exhaust. This gun has been loaded since 1945, and we have mostly survived it through luck and the occasional person willing to ignore the alarm.

Engineered pandemics

A natural outbreak is bad luck. An engineered one is a choice someone made. The same biology that lets us read and edit genomes lets a state program, or eventually a small and well-funded group, build a pathogen more transmissible or more lethal than anything that evolved on its own.

What makes this chamber spin faster every year is cost. Gene synthesis gets cheaper, published research gets more detailed, and the gap between "only a national lab could do this" and "a determined team could do this" keeps narrowing. Researchers have already rebuilt the 1918 pandemic flu and an extinct relative of smallpox from synthesized DNA. Covid showed how little it takes for one pathogen to stop the whole world. A designed one would not be constrained by what nature happened to produce.

Cyberwarfare

On its own this does not end the species. Its real danger is that it sits underneath everything else. Power grids, water treatment, hospitals, financial systems, and the early-warning networks that decide whether the nuclear chamber gets spun all run on code, and that code can be attacked.

Stuxnet, found in 2010, was a piece of software that physically destroyed centrifuges in an Iranian enrichment plant. It proved that code can reach across the gap into the physical world and break things. A serious attack during a crisis would not need to kill anyone directly. Blinding the systems that tell decision-makers what is happening is enough to turn a tense hour into a catastrophic one.

Autonomous weapons

These hand the trigger to software. AI is already making decisions about which targets to attack. Once a system can pick and engage targets without a human in the loop, and the speed of a conflict outruns the speed of a person approving each shot, anything is possible.

This is not a future scenario. A 2021 UN panel report describes a Turkish Kargu-2 drone hunting down retreating fighters in Libya in 2020 with no operator directing it, possibly the first time a machine chose to kill a human on its own.

The worry is not a single rogue machine. It is swarms of cheap, fast, expendable systems making lethal decisions at machine speed, with humans pushed further and further out of the loop because staying in it makes you slower than your opponent.

Climate damage

This is a slow projectile, shot from a distance, in a parabolic trajectory. It rarely kills directly, and that is exactly why it is easy to discount. What it does is pull away the things a stable society rests on: reliable harvests, fresh water, coastlines that stay where the cities were built.

A hotter, more crowded, more stressed world is a world more likely to reach for the other chambers. Crop failures and water shortages move people, and large movements of desperate people strain the institutions that are supposed to keep the nuclear and biological risks in check. The UN Security Council and the Pentagon both call it a threat multiplier: dangerous less as a single event and more as something that makes everything else here worse.

The chambers we unloaded

The numbers above are grim, so it is worth remembering that the odds are not fixed. We have looked at powerful technologies before and decided not to build them out. The chamber sits there, loaded, and we choose to leave it alone. It has happened more than once.

Human reproductive cloning is the clearest case. When Dolly the sheep was announced in 1996, cloning a person stopped being science fiction and became a matter of time and money. The response was fast. Dozens of countries wrote reproductive cloning into law as a crime, and the labs that could have raced ahead mostly did not. The technology exists. We decided we did not want what it would do to us.

Genetic engineering of people went the same way. In 2018 a Chinese scientist announced he had edited the genes of twin babies, and instead of a gold rush the field reacted with near-unanimous condemnation. He went to prison, and researchers across the world backed a moratorium on editing the human germline. The same community had done this to itself once before, at Asilomar in 1975, when the people inventing recombinant DNA called a halt to their own work until they understood the risks.

Weapons have been put back in the box too. Biological weapons were given up by treaty under the 1972 Biological Weapons Convention. Blinding laser weapons were banned in 1995 before they were ever deployed, one of the rare cases where a weapon was outlawed on the drawing board rather than after a war made the case for it.

The biggest win is the one nobody talks about because it worked. The chemicals eating the ozone layer were phased out under the 1987 Montreal Protocol, the first treaty every country on Earth signed, and the hole is on track to close by the middle of this century. None of these deals were perfect or self-enforcing. They show the move exists: see the chamber, and choose not to load it.

That is the only real lever here. The revolver does not have to stay loaded. Whether it joins cloning on the unloaded pile or stays on the table with the rest is still being decided.
AI is an amazing technology, and it is not going to stop being amazing, useful and game-changing on many fronts any time soon. What we decide to do with it now, and what kind of culture we build around it, will echo in the years to come.