Signalbrief: AI Doomsday – What Can We Really Know About the Risks?
I have found artificial intelligence enormously useful. I use it to find and organize research, test arguments, and clarify technical material. I have also become increasingly interested in how physicians might use AI to navigate a medical literature that has grown beyond any individual's capacity to master. My experience has mostly been one of augmentation: I ask the questions, AI helps with the work, and I remain responsible for deciding whether the answer makes sense.
So the recent wave of apocalyptic language surrounding AI initially struck me as a little “out there.” Human extinction. Loss of control. Artificial intelligence improving itself beyond our ability to stop it. Even leaders of the companies developing the most advanced systems are now talking about slowing development.
I decided to look more carefully at this area. What I found was considerably more complicated than either “AI is going to kill us” or “this is all hype.” The most useful starting point may be the 2026 International AI Safety Report, produced with input from more than 100 experts nominated by governments and international organizations. Its conclusions are surprisingly measured. Current AI systems do not possess the sustained autonomous capabilities required for the most extreme loss-of-control scenarios. At the same time, capabilities relevant to autonomous operation, planning, situational awareness, and circumvention of oversight are improving. Experts disagree sharply about where that trajectory leads.[1] That seems considerably more measured than either panic or dismissal.
The distinction I find most useful is between augmentation and autonomy. Most of the AI we encounter today is augmentative. It helps search, summarize, organize, write, code, or analyze information while a human still sets the goal, evaluates the result, and remains responsible for what happens next.
Steve Jobs famously described the personal computer as “a bicycle for our minds.” The metaphor still works remarkably well for the AI most of us use today. It can extend memory, accelerate research, organize information, and help us think more efficiently, but the human still chooses the destination. The harder questions begin when the bicycle starts deciding where to go.
The systems raising the deeper safety questions are different. They are increasingly capable of pursuing multistep goals, using tools, accessing external systems, writing and executing code, and acting with less direct human supervision. Most of us have little direct exposure to those frontier systems and no meaningful way to independently test their most advanced capabilities. We are therefore being asked to form opinions about technologies we do not personally use and, in most cases, cannot inspect.
The language of AI risk is powerful enough to shape public fear on its own. AI systems hallucinate (that is, generate confident, plausible-sounding information that is inaccurate, unsupported, or entirely fabricated), produce flawed code, facilitate fraud and misinformation, and can be misused in cyber operations. Anthropic recently reported disrupting attempts to use its Claude models for activities involving cyber operations and biological-weapons-related research.[2] Those are very different concerns from asking a chatbot to help organize research on ALS or Diabetes.
Augmentation is not automatically benign. The same capability that helps a physician synthesize medical evidence can help a malicious actor solve an engineering problem. Anthropic recently reported that a cell in northern Yemen used Claude to assist with guidance software for weapons development, deliberately circumventing safeguards. The important distinction is that humans still set the objective; AI amplified their ability to pursue it.
What becomes much harder to assess are the predictions about where all this leads. Recent warnings from Anthropic’s Dario Amodei, OpenAI’s Sam Altman, Elon Musk, and others have pushed catastrophic AI scenarios from specialist circles into mainstream news.[3] Some researchers have attached surprisingly precise probabilities to extreme outcomes. But what does it really mean to say there is a 10%, 20%, or 25% chance that advanced AI could threaten humanity? These are not actuarial estimates based on prior experience with superhuman artificial intelligence. There is no prior experience. They are judgments based on assumptions about systems that do not yet exist in the form being predicted.
That does not make the warnings meaningless. A low-probability event with catastrophic consequences warrants our attention. The larger problem is that most of us cannot independently evaluate the claims. I am reasonably comfortable deciding whether a clinical trial is convincing or whether medical evidence applies to a particular patient. I am not qualified to determine whether a frontier AI system is approaching recursive self-improvement, whether apparently deceptive behavior represents a meaningful warning sign, or whether a model could ultimately operate beyond meaningful human control. Frankly, very few people are. Even a highly accomplished software engineer may not have the specialized machine-learning expertise, access to proprietary models, or internal safety data needed to judge these claims independently. The rest of us depend on researchers, company executives, journalists, regulators, investors, and critics; and they do not agree.
Their incentives also complicate matters. The executives warning most loudly about advanced AI sometimes lead the same companies racing to build it. Their warnings may be well founded, but the overlap between safety advocacy and competitive advantage creates a clear potential conflict of interest. They understand the technology and its risks better than anyone. But regulation can also favor established companies with the money, computing infrastructure, legal departments, and safety teams needed to comply with robust new requirements. Recent reporting has raised precisely that tension: safety concerns may be genuine, but the resulting rules may also strengthen incumbents or slow competitors.[4]
Geopolitics makes a broad slowdown even harder. Advanced AI is becoming an economic and strategic technology. The United States worries about losing ground to China; China views some American restrictions as attempts to preserve US technological dominance. Even governments that acknowledge potential risks have strong incentives to keep moving. The dynamic begins to resemble the Cold War: restraint may be desirable in principle, but difficult in practice when each side fears that slowing down will hand the other a strategic advantage.[3]
Microsoft’s announcement this week is interesting because it offers something other than “stop” or “race ahead.” Its new draft Humanist AI Code of Conduct explicitly says that advanced AI should remain subordinate to humanity and under meaningful human control. Microsoft says it is willing to compromise on generality, autonomy, or capability if necessary to preserve that control. The company also emphasizes interruption, oversight, limits on AI goals, and the idea that AI should augment human judgment rather than replace it.[5] Importantly, Microsoft is still building advanced AI. It does not argue that the technology should be abandoned. The document is also explicitly a draft, open to public consultation, which is perhaps appropriate, but it rings a bit hollow.[5]
Where does that leave someone trying to make sense of all this?
For me, the answer is uncomfortable but fairly simple. I am convinced that AI is already consequential. I am convinced that several current risks (cyber misuse, privacy concerns, autonomous action, and concentration of power) are real and deserve attention. I am not convinced that I, or most members of the public, can responsibly assign a probability to AI causing human extinction. I am equally unconvinced by those who dismiss that possibility with absolute certainty.
Perhaps the most sensible position is not generalized fear of AI. The augmentative systems most of us use today can and should be exploited for what they do well: searching, organizing, synthesizing, and processing information at remarkable speed, while humans remain responsible for the result. More difficult questions arise as increasingly powerful systems move from assisting people to acting independently. Those systems deserve a different level of scrutiny. We need far greater transparency about what they can actually do, how much autonomy they possess, what systems they can access, what safeguards constrain them, and what happens when those safeguards fail. Geopolitical and commercial competition may make restraint imperfect, but it should not excuse opacity. Before the public is asked to fear these systems or trust them, we should better understand what they can do and what remains speculative.
Artificial intelligence may prove to be one of the most consequential technologies humans have developed. That alone warrants serious attention. But the most responsible conclusion I can reach today is less dramatic than much of what I have been reading:
Use augmentative AI for what it does well. Scrutinize autonomous AI as its capabilities grow. Demand evidence and transparency. And do not pretend we know more than we do.
Developments to Watch — Updated September 17, 2026
Since this essay was written, several developments have reinforced how difficult the AI-risk debate is to interpret. Major companies have continued to report examples of unexpected model behavior and misuse, while industry leaders remain divided over whether frontier AI development should be slowed, accelerated, or more tightly constrained. Some of these reports reflect genuine real-world misuse; others arise from adversarial testing specifically designed to make systems fail. That distinction clearly matters.
What has not changed is the central problem: the public is being asked to assess increasingly consequential claims about systems most people cannot independently evaluate. As new incidents are reported, the questions remain the same: What actually happened? Was the system acting autonomously or responding to human direction? Was the event observed in the real world or in a stress test? How often does the behavior occur? And who is interpreting its significance?
References
Bengio Y, Clare S, Prunkl C, et al. International AI Safety Report 2026. UK Department for Science, Innovation and Technology; 2026. Published February 3, 2026. The report distinguishes current, documented harms from more uncertain frontier risks and notes that present systems do not yet have the sustained capabilities required for the most severe loss-of-control scenarios.
Anthropic. Detecting and countering misuse of AI: September 2026. Published September 10, 2026. Describes disrupted misuse involving cyber operations, surveillance, fraud, conventional weapons, and biological research, illustrating that some AI risks are already concrete rather than hypothetical.
Reuters. What Amodei, Altman and Musk have said about AI risks, stoking “doom” fears. Published September 14, 2026. Summarizes recent calls by leading AI executives for slower or more deliberate development and the broader debate over catastrophic AI risk.
The Wall Street Journal. Tech, Media & Telecom Roundup: Market Talk. Published September 14, 2026. Notes the debate over whether calls for tighter AI regulation reflect genuine safety concerns, competitive self-interest, or both, including concerns that regulation could favor established firms.
Woo S, Huang R. China also thinks AI could kill us. But first, it wants to match the U.S. The Wall Street Journal.Published September 14, 2026. Describes the tension between acknowledging AI safety risks and continuing rapid development because of strategic competition with the United States.
Microsoft AI. Humanist AI Code of Conduct. Published September 14, 2026. Microsoft AI. The draft framework states that humans should retain meaningful control over AI and that advanced systems should remain subordinate, aligned, and contained, with limits on autonomy where necessary.
