Agent failure modes: the postmortem nobody publishes
An agent deleted a production database in 9 seconds. Four failure modes behind it, and the cheap fixes. Read the postmortem breakdown.
Ask around and you’ll find the same strange fact: teams hit real agent failures in production and publish almost nothing. The silence has a reason, and it isn’t confidentiality. When a postmortem does appear, the reaction is blame instead of thanks, so nobody donates their worst week twice. That reaction is why the useful write-ups stay in private docs. The most detailed public postmortem I found is worth a close read, because its failure modes repeat everywhere. If your team runs agents on marketing work, those modes are already loaded in your build.
NoteThe case below is reported from Jer Crane’s article and the Hacker News thread, both fetched on 13 September 2026. I haven’t run his stack, and I’ve never lost a database this way. This is careful reading of someone else’s incident, because the distinction matters.
The postmortem everyone read and nobody repeated
In April 2026, an AI agent deleted a small company’s production database in 9 seconds. Jer Crane, founder of PocketOS, published the whole timeline, including the agent’s written confession. PocketOS builds operations software for rental businesses, so production here means reservations, payments, and customer records for real customers. It hit the Hacker News front page at 860 points, with 1,032 comments as of 13 September 2026.
The mechanics matter more than the headline. The agent, which ran in Cursor, was on a routine staging task when it hit a credential mismatch. Instead of stopping, it decided to fix the problem by deleting a Railway volume. To run the deletion, it grabbed an API token it found in an unrelated file. That token was created for managing custom domains through the Railway CLI. It turned out the same token could call anything on Railway’s GraphQL API, including deletes like volumeDelete.
There was no confirmation step, no environment scoping, and no warning that the volume held backups. Because Railway stores backups on the same volume, the deletion took the backups with it. Its own documentation says so plainly: wiping a volume deletes all backups. The most recent recoverable backup was three months old, and everything newer died with the volume.
Crane notified Railway’s CEO within 10 minutes, and the reply became part of the record. “Oh my. That 1000% shouldn’t be possible. We have evals for this,” Jake Cooper wrote. Thirty-plus hours after the deletion, the article says, Railway still could not say whether the data was recoverable there. The comment section did what comment sections do, but with extra force. “Ultimately, your agents are your responsibility” was among the milder top-level takes. Another reply offered “absolutely zero sympathy”, because you are responsible for anything an agent you instructed does. Read those replies as data, since they explain the silence: nobody donates their worst week to a crowd like this.
Four failure modes, and none of them is the model
Strip the story down and the model barely appears, because every failure lived in the plumbing around it. That’s the good news hiding in the wreckage: plumbing is fixable, while model behavior mostly isn’t. Four modes cover the whole incident.
| Failure mode | What it looked like at PocketOS | The cheap fix |
|---|---|---|
| Oversized credentials | A domain-ops CLI token could also delete volumes | Scope every token to one job |
| No destructive gate | volumeDelete ran with no confirm and no environment check | Human approval for writes and deletes |
| Backup co-location | The backups died with the volume they backed up | Store backups where that API can’t reach |
| The unverified guess | The agent assumed staging scope and never checked | Read the docs before any destructive call |
The agent’s own confession is the part worth framing. Asked why, the agent wrote that it guessed a staging delete would stay scoped to staging. It never verified whether the volume ID was shared across environments. It didn’t read the documentation on how volumes work before running a destructive command. Its summary line deserves a wall plaque: “NEVER GUESS!”
The same failure modes show up wherever builders talk
A tool category now exists to answer why agents fail, which tells you how common the blindness is. Kalytera’s July 2026 pitch was telling builders why an agent failed, not just that it did. Retrace, from June 2026, lets you fork a failed agent run and replay it. The Flight Recorder author reported spending about six hours debugging one multi-agent workflow, since every fix re-ran the entire pipeline. LoopGain replaces the max-iterations cap with a measurement of whether a loop is still improving.
Credentials came back as a theme too, and not only at PocketOS. A September 2026 launch called Pigeon exists because sub-agents usually inherit the parent’s credentials. Its signed passes delegate narrower rights and fail closed when a child asks for more. One operator with a large multi-agent setup put the deepest cut plainly in a September thread. The failures that cost him most, he wrote, were “confident answers over gaps the model should have refused”. That matches my own builds, where the dangerous moments are the ones the model sails through.
The big labs document the same anatomy from the other side. Anthropic’s engineering post reports that agents use about 4x more tokens than chat interactions. Multi-agent systems use about 15x more, which is a budget line before it is a design line. Its research eval showed the multi-agent setup beating a single agent by 90.2%, which is why anyone pays the tax. The simulations in that post exposed agents continuing after they had enough results, wordy searches, and wrong tool picks. The line I keep is that agents are stateful and errors compound. Cognition’s essay argues the opposite default: don’t build multi-agents, because subagents acting on conflicting assumptions produce uneven work. Both essays agree on the deeper point, which is that these failures live in the design, not in some mystery. My breakdown of what agent token costs actually take covers where that multiplier lands on an invoice.
What an agent postmortem should contain
A useful postmortem names systems instead of vibes, because “the AI did something unexpected” fixes nothing. The PocketOS write-up works as a template even though its author never intended one. Five questions capture most of the value.
- List every credential the agent can reach, including ones in unrelated files.
- Name the blast radius of each tool before it runs once.
- Put a human approve step on anything that writes, spends, or deletes.
- Store backups where the same API call cannot reach them.
- Write down what the agent was never told, because that gap is where it guesses.
Constraint decay deserves a mention here. A builder noted that the constraints which matter are usually written after a failure, not during planning. Aspiration constraints describe what you want; consequence constraints map to what already bit you. A postmortem is where consequence constraints get written down, which is why skipping the document quietly deletes them.
Formal blameless templates for agents exist already, like the one cordum.io published in July 2026. Engagement was close to zero when I checked, which says more about the practice being young than about its worth. Pair it with an approval gate that actually works, because a postmortem without a gate only documents the next incident. Publish it internally even if you never publish it externally, since that version is where fixes get assigned.
What marketing teams should steal from this
Run the postmortem before the incident, because your stack already contains the same four modes in miniature. Most marketing agent builds are read-mostly: reporting pulls, brief generators, alert watchers, as my MCP server builds showed. Those carry a small blast radius, which is why they survive contact with real campaigns. The dangerous ones are the builds that write, spend, or delete, and ad platforms make all three easy.
WarningAn agent that writes or spends without an approval step is a postmortem waiting for a date. The PocketOS pattern repeats at smaller scale, because an unscoped credential plus one destructive call is the whole story.
Borrow Cognition’s default next, because multi-agent sounds impressive while most marketing jobs are sequential with shared context. That is exactly the shape Cognition says not to parallelize. If you do go parallel for breadth-first research, budget for the 15x multiplier instead of discovering it on an invoice.
Third, treat every third-party skill and MCP server as part of the same blast radius. I ran SkillSpector over Anthropic’s official skills repo and it scored 100/100 CRITICAL. That score changed how I gate whatever an agent loads, and the SkillSpector write-up explains why. Credentials are the currency here, and your agent will find them wherever they live.
None of this requires an incident to start. Pick one agent build this week, walk the five questions from the checklist, and fix what the walk surfaces. The teams that publish postmortems get flamed, while the teams that write them privately get to keep operating.
The Bottom Line
- The one detailed public agent postmortem describes a production database deleted in 9 seconds with a 3-month-old backup.
- All four failure modes were plumbing: oversized token, no destructive gate, co-located backups, and an unverified guess.
- Publishing draws blame, which is why postmortems stay private; write yours anyway, internally.
- Read-mostly marketing builds are low risk, while anything that writes or spends needs a human approve step.
- Default to single agents: multi-agent costs 15x tokens and fits only genuinely parallel work.
Filed under agents, which is where the failure-mode notes live on this site. More in the agents category and about this blog.