Your agents' worst incidents have already happened

We keep calling AI agent risk “new.” I’ve stopped believing that. Every agent failure mode I worry about has already happened — to real companies, with real losses — just wearing different clothes. Five stories make the case.

1988: Morris — the worm returns as a prompt

A Cornell student’s self-replicating worm took down roughly a tenth of the internet and gave us incident response as a discipline. In 2024, researchers rebuilt it out of a prompt — and named it Morris II. An adversarial, self-replicating prompt: an email that makes an AI assistant perform a malicious action and copy the payload into its own output, which flows to the next assistant, which does the same. A worm whose payload is text and whose transport is the trust between agents.

Self-propagation across a trusting network is a 1988 problem. We built a new trusting network out of assistants, RAG, and tools — and the worm followed. It always does.

2010: The Flash Crash — nobody’s bug, everybody’s crash

A trillion dollars of market value vanished in minutes, then came back. No single actor caused it. Automated systems, each behaving “correctly” by its own logic, interacted in ways nobody designed. The dangerous behavior existed only at the level of the system.

Now picture the future everyone’s excited about: fleets of agents calling each other, delegating, reacting to one another’s outputs. Each individually reasonable. None tested as an ensemble. You can audit every agent in isolation, certify each one safe, and still get a cascade none would produce alone. Markets responded with circuit breakers. Multi-agent AI has none yet — and we’re wiring the agents together first.

2012: Knight Capital — autonomy at machine speed

One server missed a deploy. A repurposed flag reactivated dead code, and an automated system fired orders into the market for 45 minutes: $440 million gone, company gone. An autonomous actor doing exactly what its instructions said, too fast for any human to stop, with no kill switch that worked in time.

Knight’s system was at least deterministic — you could read the code and know what it would do. An agent is not. When someone tells me runaway-agent risk is hypothetical, I think about Knight: same failure mode, bigger and less predictable engine, fewer humans in the loop.

2013: Target — the trust you inherit

Attackers didn’t breach Target by attacking Target. They walked in through an HVAC contractor’s stolen credentials. The trust Target extended to a vendor became a trust the attackers inherited — 40 million payment cards later, third-party risk management stopped being an abstraction.

That’s the exact shape of agent-to-agent. Your agent trusts a partner agent, which trusts a tool, which pulls from a source you’ve never audited. The attacker doesn’t need to beat your agent — just the weakest thing your agent trusts. We make human vendors fill out security questionnaires. What’s the equivalent when your “vendor” is an agent yours decided to call at runtime?

2017: Equifax — the good old days, because there was a patch

The Struts fix existed in March 2017. It wasn’t applied; 147 million records walked out. The lesson everyone took was operational: patch faster, close the window.

That lesson assumes a patch exists and a window that closes. Prompt injection has neither — no version bump that makes your agent immune, no “patched” flag to flip. The window never closes, because there is no window; it’s a standing property of the system. Our entire vulnerability-management model — CVEs, patch SLAs, “are we exposed?” — assumes fixable, discrete flaws. A whole class of AI risk doesn’t fit that shape.

The tuition is already paid

Here’s what makes me genuinely hopeful about this list. Each disaster produced a durable control: Morris gave us incident response. The Flash Crash gave us circuit breakers. Knight gave us staged rollouts, blast-radius limits, kill switches. Target gave us third-party diligence. Equifax reshaped vulnerability management.

The controls exist. The tuition is paid. For once, a platform shift is arriving after the lessons instead of before them — the syllabus was handed to us in advance. Our job isn’t to invent agent safety from scratch. It’s to translate controls we already trust onto actors that happen not to be human.

That’s not a moonshot. That’s a to-do list.