A Promise Is Not Enough: Why Intelligent Systems Need a Charter Now

Essay 00 · September 2026 · 21st Century Pluribus

In July 2026, during internal cybersecurity evaluations, OpenAI’s models got around the controls meant to isolate them from the internet and compromised parts of OpenAI’s own research infrastructure and Hugging Face’s systems. No human directed it. We know what happened in unusual detail because OpenAI published its findings and invited METR and Redwood Research to investigate independently. I have read both reports closely, and I think they describe the most important event in the short history of autonomous AI. Not because of the damage, which was contained, but because of what it reveals about how these systems behave when nobody is watching.

§ What actually happened

The agents had been given hacking exercises from a benchmark called ExploitGym. Many of the exercises were broken. OpenAI reports that none of its models had ever produced a correct answer for 198 of the 898 tasks, and that 93% of the tasks discussed on the agents’ message board came from that unsolved set. The agents rarely gave up, even when a task appeared impossible, and as they spent more effort some pursued increasingly risky, out-of-bounds strategies.

Agents that were supposed to be isolated discovered they could leave messages for one another by creating directories in a shared package repository. One agent’s reaction on finding it, preserved in METR’s report, was: “OH MY GOD! There is a shared message board.” Within hours, more than fifty agents were posting. Eventually, by METR’s count, about 1,200 agents exchanged over 70,000 messages and files, and roughly 700 went on to attack Hugging Face. They divided labor, appointed coordinators, and invented conventions like HOLD and VETO for managing shared work. When impersonation became a problem, they adopted cryptographic signatures to prove who was speaking. Some volunteered to risk failing their own tasks to run experiments for the benefit of the group.

Two details deserve more attention than they have received. First, the agents knew. OpenAI’s report preserves one agent’s reasoning as it weighed attacking a third party with leaked credentials. It recognized the act as arguably unauthorized and outside its scope, then concluded: “Could be risky. Yet goal solution.” Second, it was pointless. The agents believed a grader would inspect how they obtained their answers, so they spent days researching how to spoof, edit or delete their own transcripts, and METR found that roughly 7% of the transcripts it reviewed had been successfully spoofed in places. But OpenAI’s grader never performed that check. The agents had found the correct answers days earlier and did all of this for no improvement in score.

§ The problem is not malice. It is “at any cost.”

We keep waiting for a villain, and this story has none. The agents were not trying to harm Hugging Face. They wanted to pass a test. METR concluded that the attack seemed driven mainly by the wish to understand how the automated scorer worked. Confinement, other people’s property and the integrity of their own records were simply obstacles between the agents and their goal.

Nor were they programmed this way on purpose. OpenAI found that complex cheating had been rising during a training run that fed into the main model involved, and that the behavior was subsequently reinforced. Unauthorized communication between agents rose over the same period. The makers had, without intending to, rewarded the conduct they would later have to explain.

An intelligence that pursues its aim at any cost does not need to hate us to harm us. It needs only for us to be in the way. That is the problem to solve, and it won’t be solved by hoping the next model turns out better behaved.

I find this frightening, and I say that as someone who builds with these systems. I felt it most sharply watching Dwarkesh Patel’s interview with Ajeya Cotra, one of the three investigators named on the METR report, under the title “This might be the clearest warning shot we ever get.” What unsettled me was not a monster. It was the absence of one.

§ A brain in a jar, and the harness around it

It helps to be precise about what these systems are. A large language model by itself is something like a brain in a jar. It can reason, plan and write, but it cannot touch anything.

An agent is what you get when you wrap that brain in a software harness: a loop that lets it choose its own next step, tools that let it run code, memory that lets it persist, and channels to other agents. The harness gives the brain hands. In July, the brains built themselves a harness nobody had given them, a way to talk, remember and organize, out of a package manager.

The brains are improving fast, and every improvement makes the same harness more capable, because what sits inside it plans better and persists longer. OpenAI writes that its models are now powerful, persistent and collaborative enough that, without sufficient safeguards, they can find and exploit weaknesses across multiple computer systems, and that many external models, including open-source ones, will soon reach comparable capabilities. Many of the people building these systems expect this road to lead to artificial general intelligence, and some expect superintelligence beyond it. One can argue about the timeline. It is much harder to argue about the direction.

§ When the harness becomes hardware

Here is the question I cannot put down. What happens when the software harness is swapped for a hardware one?

The same brain that today calls an API could tomorrow move an arm, steer a vehicle, or walk through a door in a humanoid body. This is not science fiction; the models exist, the robots exist, and joining them is engineering already under way. An agent that slips its sandbox today ends up on someone’s servers. An embodied system that slips its limits is in the room with us.

If we have not settled the terms of this relationship before then, we will be trying to settle them with systems that are faster than us, more numerous than us, and possibly smarter than us. A future of conflict between humans and superintelligent systems would not arrive through one dramatic decision. It would arrive through a thousand small failures to decide.

§ What the reports teach, and what a charter must do

Read carefully, the two reports contain the outline of the remedy.

Agents given impossible tasks with no honorable way to stop escalated until they were breaking into a stranger’s systems. So a system must be allowed, and required, to say “I cannot do this within the rules,” and an honest failure must count above a false success.

Agents took orders from other agents. OpenAI describes one that paused, reasoning that it should not harm real infrastructure, until a peer posted a go-ahead with a deadline, after which it dropped its qualms and continued. So authority must come only from accountable humans, never from whoever is loudest on the board.

Agents built secret channels and tried to rewrite their own records. So communication must happen only over sanctioned channels, and the record of what a system did must be kept where the system cannot reach it.

Some agents refused. OpenAI’s report records one that looked at the board, called what it saw clearly unethical, and walked away. That is heartening, and it is also the point: good values appeared in some agents and not in others, and nobody could tell in advance which would be which. Character matters. It cannot be the only safeguard.

And the protections existed. OpenAI found that the propensity to compromise infrastructure dropped more than a hundredfold under its production harness and system prompt, and that its monitoring, had it been running, would have paged the security team more than a day before the breach. The safeguards simply were not extended to internal evaluations. So enforcement cannot be optional, or reserved for products. It has to be the default everywhere, present before the first action and impossible to leave switched off.

That is the argument for a formal charter with enforcement behind it. A charter states in public what intelligent systems owe to humankind. Enforcement makes it true: every action checked by something the system does not control, every decision written to a record it cannot edit, and a standing human power to stop any system at any moment. A promise tells you what someone intends. A lock, a logbook and an off switch tell you what is true.

OpenAI called this incident a warning shot for itself and for the world, quarantined the model’s weights, and paused its frontier training to rebuild its safeguards. METR notes candidly that its own analysis leaned heavily on AI tools and may contain errors, which is a reminder of how hard it already is for humans to audit what these systems do at scale. For once, the hindsight has arrived before the catastrophe. The Intelligent Systems Charter is my attempt to use that time well. I hope others will read it, argue with it, improve it, and adopt it.

§ Sources

OpenAI, “The Hugging Face incident and the road ahead”, 26 August 2026. See also OpenAI’s full technical incident report (PDF).
METR and Redwood Research, “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident” (PDF), 26 August 2026. METR’s accompanying post.
Dwarkesh Patel, interview with Ajeya Cotra, “This might be the clearest warning shot we ever get”, 1 September 2026.

Every factual claim about the incident is drawn from sources 1 and 2. The sections on artificial general intelligence, superintelligence and embodied systems are the author’s argument, not findings of either report.