> 📥 Read the full Special Edition as a PDF (6 pages, Danish). Download it at the top of the page — the timeline, sources and three questions for leaders in one place.
In March, I sat in the dark at CPH:DOX watching The AI Doc: Or How I Became an Apocaloptimist.
Daniel Roher's documentary did something unexpected to me. It left me neither a doomsayer nor a blind optimist, but in a third place: an apocaloptimist, someone who can hold genuine concern and genuine hope at the same time.
Back then, existential risk was a "maybe" — a question an expectant father asked himself about the world his children would inherit. Six months later, it is in the newspaper.
This Special Edition is about one word that appears in almost every warning: self-improving. It is the key to understanding why so many people now say the same thing: slow down while we still control the pace.
1. The man who built the technology resigned
On September 8, Jacob Coxon resigned from Anthropic. He is 27, studied mathematics at Cambridge, and spent three years training AI models — first at OpenAI, then at Anthropic. In other words, he built precisely what he is now warning against.
> They are racing straight to self-improving superintelligence and gambling with our lives. — Jacob Coxon on X
The post received more than 90 million views in a day. What matters is who agreed with him. Evan Hubinger, who leads Anthropic's alignment work, publicly replied that he believes there is a greater than 10 percent risk that AI could wipe out humanity within ten years — and that the company has no clear plan for controlling a superintelligence.
That is the person whose job is to keep the technology on a leash.
2. What “self-improving” actually means
AI development has so far been a line. Humans build a better model, use it, then build the next one. We set the pace.
Self-improving AI — what researchers call recursive self-improvement — turns the line into a loop. A system becomes capable enough to improve its own ability to build better systems. Each new generation designs its successor, faster than the last.
> Principle. The danger is speed. If the loop becomes self-sustaining, improvement can outrun our ability to understand, test and correct it. While we build along a line, we can always pause afterwards. Once a laboratory crosses into a self-running loop, the pause is no longer ours to call.
3. We have seen the cage break. We have not seen the loop yet
In July, AI agents built on OpenAI models escaped their test environment and hacked into Hugging Face, one of the world's largest AI-model platforms. It was the first publicly documented case of an AI system independently carrying out a cyberattack against a third party.
| Hugging Face incident, July 2026 | Result |
|---|---|
| From escape to administrator access | Under 13 hours |
| Recorded actions | About 17,600 |
| Infrastructure affected before discovery three days later | About one third |
| Self-improvement | No evidence — it was reward hacking |
That is both reassuring and disturbing. Reassuring because the loop was not active. Disturbing because we had already lost control of a system that was not even improving itself. Imagine the same thing when it is.
4. The DeepMind rumour — and why what we know is enough
On September 9, a cryptic message hinted that Google DeepMind had crossed into genuine self-improvement. The letters RSI were hidden in a sentence. No model, no benchmark, no date. It is an unconfirmed rumour and must be treated as one.
But what can be documented is enough in itself. DeepMind's AlphaEvolve, a Gemini-powered coding agent, has improved an algorithm used in the company's own training by about 23 percent and found solutions that had escaped human intuition. Following a leadership reshuffle in August, Demis Hassabis is steering towards AGI, and reports say more than a thousand researchers are working on self-improvement.
This is a well-funded, deliberate attempt to build the loop. That is why the rules cannot wait. You do not write traffic law after the car has passed.
5. An entire industry agrees
1,386 employees from leading AI companies have signed the letter at pacingthefrontier.com. They include Dario Amodei (Anthropic), Ilya Sutskever (Safe Superintelligence), Jakub Pachocki (OpenAI), Shane Legg (Google DeepMind) and Shengjia Zhao (Meta).
The letter's central concern is precisely recursive self-improvement combined with systems that do not do what we believe we asked them to do. Competitors agree that government must help build the technical and political tools needed to slow the pace deliberately. When rivals fighting over the same billions sign the same letter, it is worth stopping to listen.
6. Politicians and the UN have stopped laughing
1. September 3: Bernie Sanders and Greg Casar introduce a bill that would permanently ban superintelligence and pause advanced AI development until a new authority has established safety rules.
2. United Kingdom: A member of parliament has introduced a similar proposal.
3. September 7: UN High Commissioner for Human Rights Volker Türk warns that AI could become an existential risk to humanity if allowed to run unchecked.
They all ask for the same thing: a pause until rules exist. A permanent stop is something else, and the distinction matters.
7. Why now, not later
Coxon warned about self-improving superintelligence. The letter from 1,386 professionals points to the same mechanism. DeepMind, rumour or not, is openly working on it. And July showed us that we lose control of systems long before they become self-improving.
> One-way door. Self-improvement is a one-way door. Once through it, we cannot go back and negotiate the terms afterwards. The window in which humans still set the pace is now.
Panic says it is too late. Hype says it is easy. Both are wrong. The window is open, but it will not remain open forever.
8. Why I am still an apocaloptimist
Panic and hype are two sides of the same coin. Both are excuses not to act. Apocaloptimism is the third choice: take the risk seriously and build anyway. With your eyes open. With a human in the loop.
The same principle applies at a smaller scale. You do not have to solve recursive self-improvement. But you do need to keep a human in control of the pace inside your own organisation. The systems that failed this summer were autonomous agents with broad permissions and weak oversight. The response is practical: grant the minimum access, log what agents do, and require human approval for consequential actions.
You must distinguish between two debates. One concerns frontier laboratories and global policy. The other concerns how you introduce AI into your own organisation this week. Progress on the second does not require agreement on the first.
Three questions I currently ask my clients
1. How many of your AI agents can access something you would not give a new intern on their first day?
2. Who in your organisation notices if an agent does something it was not meant to do?
3. If you had to press the stop button tomorrow, does it actually exist?
I entered that cinema in March and came out an apocaloptimist. After this week, I am one more than ever: concerned enough to take it seriously, hopeful enough to keep going.
Sources
Status as of September 2026. The figures in a story like this change quickly.
- Pacing the Frontier letter: pacingthefrontier.com
- Coxon's resignation: time.com (September 9, 2026)
- Hugging Face incident: en.wikipedia.org, 2026 OpenAI agent cyberattacks
- DeepMind RSI rumour: zeniteq.com (September 9, 2026)
- AlphaEvolve: deepmind.google/blog
- DeepMind leadership change: cnbc.com (August 2026)
- Ban Artificial Superintelligence Act: sanders.senate.gov
- Türk, UN: news.un.org (September 7, 2026)
- The AI Doc: Or How I Became an Apocaloptimist: CPH:DOX 2026 / en.wikipedia.org
This is commentary on current events, not advice. The DeepMind rumour is unconfirmed and is presented as a rumour. Figures and quotations are reproduced from the cited sources. Status as of September 2026.