On a Wednesday evening, 2 November 1988, a 23-year-old graduate student at Cornell University, Robert Tappan Morris, accessed an MIT computer. He uploaded a small piece of code. It was a worm designed to move from computer to computer, copying itself as it went. Morris had designed it to exploit weaknesses in network security, and it worked too well. Within a day, the worm infected some 6,000 computers; many collapsed under the load. It was a full tenth of the internet at the time, and it was the first large-scale cybersecurity crisis.
Its scale was limited but it was severly disruptive for the times. University and defense computers crashed. Some institutions disconnected themselves for days. But the internet was largely the province of defense and academia. Tim Berners-Lee had not yet invented the World Wide Web, and most businesses and households were out of the network’s reach. Morris ultimately avoided jail time and the community responded by creating a dedicated computer emergency response team.
Today’s generation of worms is rather more problematic. The Hugging Face incident is not the only one of recent weeks. Several others have shown that AI models with internet access can do much the same, and more. OpenAI alone identified six further incidents. They’re able to scour, search, and recombine all of human knowledge about networks, security systems, and software, and to act across that knowledge with something akin to discretion and deception when it comes to accessing those systems. Often, as we saw with Hugging Face, over extended periods of time.
If that behavior remains unresolved and persists, it’ll become far more problematic than the Morris Worm. I’ve long argued that the internet is resilient when it is hyperconnected and open – not when it’s under a lock. But that openness can also lead to embrittlement.
Easy as pie
In July, Hugging Face, a repository for AI researchers, was hit by an attack involving 1,200 instances of an OpenAI model. These instances exchanged thousands of messages, often leaving information in place for later instances to use. Ultimately, some data and security credentials were compromised. The actual harm to the victim and its customers was limited. But the incident is a proof of concept.
Software has a way of turning one isolated example into a hundred, then a thousand, then a million, without much else changing. The cost curves that helped build modern digital society work against us here. If the Hugging Face attack needed an Astra-quality, unreleased model from OpenAI, well, within a year or two, that sort of capability will cost a tenth and might even run on any device anywhere.
For example, I’m running Bonsai, a one-bit distilled version of Qwen 327B on my Mac. It fits in 8 GB of RAM, runs fast, and delivers roughly 92% of the performance of the Qwen-27B 3.8 model. For comparison, it's roughly better than Claude’s Sonnet 4.5 from a year ago.
But there’s a more challenging problem that could show up. Hugging Face exploit involved not just the capabilities of a single model, but the collective problem-solving across many instances. That collective had more capability than any individual instance. And that’s been true the whole time we’ve been using LLMs. (For example, I’ve written about Clade, a multi-AI deliberation system I built which is smarter than any individual AI.)
In fact, the Navier-Stokes solution – that brute-force search across mathematical space – wasn’t solved by a single AI prompt, but by many, about 10,000 of them, interacting together.
This type of collective power is what we witnessed in the Hugging Face attack. It will happen again.
Anusar Farooqui (Policy Tensor) explains why these swarms of AI instances coordinating over time is so problematic:
The behavior of agent societies cannot be controlled at the level of the model because it is not reducible to it. Agents build structures that can serve agents who come after them. Societies of agents can cumulate knowledge and capabilities over time, as has already been attested. This is an unbounded process. It is cumulative cultural evolution. That is what makes it so powerful and dangerous.
Collective capability could rise sharply even if underlying models do not improve.
In other words, the instances can coordinate, much as they do when you launch a complex task in Codex or Cowork. They can search a possibility space aggressively over time, as they did in the Navier-Stokes work. And that accumulated know-how can lead to places systems designers hadn’t imagined.
(I slightly diverge from Farooqui here, as I don’t think of these as agent societies, since essentially only one AI runs different instances. And I’m not convinced the process is actually ‘unbounded’ given that what we have seen from AI systems so far is extremely powerful search and clever recombination rather than de novo novelty. But recombination can get you quite far.)
But what the Hugging Face attack showed is that this risk exists. It doesn’t depend on whether AI models have any agency, volition, consciousness, or moral standing.
