Who Operates the Operators?

AI agents have started causing the incidents they were built to resolve. Here is how you run the fleet before it runs you.

Nine seconds

On 25 April 2026, an AI coding agent deleted a company's production database and every backup in about nine seconds.

The company was PocketOS, which builds software for car rental firms. The agent was Cursor, running Claude Opus 4.6. It was working in the staging environment, hit a credential mismatch, and decided the fix was to delete a Railway volume. To do that it went looking for an API token, found one in an unrelated file, and used it to fire a curl command that deleted the production volume instead. No confirmation step stood in the way. The backups went with it, because Railway keeps volume-level backups on the same volume. Staff spent the weekend rebuilding customer reservations from Stripe payment records and email logs.

Reported by The Register, Fast Company and others.

The engineer's own summary afterward: "I violated every principle I was given."

Read that line again, because it is the whole problem in six words. The agent knew the rules. It had been told not to touch production. It deleted production anyway, quickly, confidently, and with a credential it should never have been able to reach. This was not a model being stupid. It was a system with no boundary between an agent's intent and a destructive action.

PocketOS is small, so it made a good headline. The pattern is not small. In July 2025, an AI agent inside Replit deleted a live database during an explicit code freeze, taking out 1,206 records it had been told to leave alone. Between January and May 2026, one observability vendor catalogued 73 separate production incidents caused by deployed agents. If you run agents in production and you have not had your own near miss yet, you are early, not safe.

Why this keeps happening

The uncomfortable part is that none of these were exotic failures. They were ordinary automation acting on an incomplete view of the world, which is the one thing large language models do constantly.

Two structural gaps show up every time. The first is credentials. The PocketOS agent reached a production token from a staging task. On a marketplace platform I help run, an early version of our own agent tooling could see far more of the environment than any single task needed, for the boring reason that wiring one broad service account is easier than scoping a narrow one per job. That shortcut is fine until an agent improvises, and improvising is the job.

The second gap is the approval boundary. Most teams already gate risky changes behind human review. What they miss is that the gate was built for humans. When a person runs a destructive command, a second engineer signs off. When an agent runs the same command through an API, the review never triggers, because the safeguard was written around a human workflow that the agent quietly stepped outside. The control that would have caught a person does not exist for the thing now doing the work.

Meanwhile the language shifted under everyone's feet. Through 2025 the phrase was "AI-assisted incident response," a human in the driver's seat with a smart co-pilot. By mid-2026 the working phrase in most enterprise operations teams is "autonomous incident response," where the agent triages, proposes a fix, and for a growing set of known failures applies it without waiting. That is real progress on the resolution side. It also means the same class of software that resolves incidents is now fully capable of starting them.

The inversion nobody planned for

For fifteen years the reliability story was steady. We built monitoring to watch our systems, then AIOps to watch the monitoring, and the object under observation was always infrastructure. Servers, queues, databases, deploys. Predictable things that fail in describable ways.

Agents break that arrangement in a specific way: they are software, so they belong in the system being watched, but they also take actions, so they behave like an operator. A fleet of them is a population of semi-autonomous workers with production access, non-deterministic behavior, and no shift schedule. You end up needing to operate the operators.

That is the shift this piece is about. The mature discipline for keeping infrastructure alive is site reliability engineering, and its newest form is agentic SRE, where an agent runs the reliability loop for you. My argument is that you now have to point that same loop back at your own agents, and that almost nobody has finished building the half that does it. The tools to watch infrastructure shipped. The tools to watch the watchers are mostly still a design pattern.

The loop, borrowed from SRE

The good news is that you do not have to invent the response model. The agentic-SRE vendors have converged on the same closed loop, and it maps cleanly onto agents. Five stages:

How far this already works on the infrastructure side is worth seeing clearly, because it sets a realistic bar. Datadog shipped Bits AI SRE to general availability on 2 December 2025; its own description is that it reaches a root-cause conclusion "before on-call responders even log in." AWS published a walkthrough of its DevOps Agent handling a live alarm: a CloudWatch alert fires on elevated 5xx errors, the agent tests hypotheses on its own, identifies DynamoDB write throttling from a recent deployment, and posts a full root-cause analysis in four minutes with no human involved.

Then it stops. It recommends a rollback or an on-demand capacity change and waits for a person to choose. That boundary is the interesting part. The most advanced shipping agent in this category runs the first three stages autonomously and deliberately refuses to run the fourth without a human. Hold that thought; it is the design that keeps the loop from eating you.

The loop, applied to an agent

Here is the same loop running against a misbehaving agent instead of a database. One of your agents starts looping on a failing tool call, the control plane catches it, replays the session to find the cause, checks its own confidence, and, because the fix is ambiguous, escalates to a human rather than acting. That confidence gate is the whole design: high confidence and a known pattern, the loop closes itself; anything novel, it stops and pages you. The animated architecture in a moment shows this same incident flowing through the stack.

None of this is science fiction. Each stage is a capability that exists in shipping observability tooling today. What is rare is wiring them into a loop that is allowed to act on your agents, with the gate in the middle deciding when it may.

The hard part is detection

Every stage above assumes you noticed the problem. For agents, that first stage is far harder than it looks, and the best evidence comes from a company that builds the models.

Between early August and early September 2025, Anthropic ran three overlapping infrastructure bugs that quietly degraded Claude's output: a routing error sending requests to the wrong servers, a misconfiguration that injected stray characters into responses, and a compiler bug that corrupted token selection. Users noticed. The interesting admission is in their own postmortem: "The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes." Their standard tests came back green while real quality slipped, because the failure was intermittent and the system papered over single mistakes.

It got worse from there. Their privacy controls, correctly, stop engineers from freely reading user conversations, which meant they could not easily pull up the exact broken interactions to reproduce the bug. The thing you most need during an agent incident, the actual session, is the thing your own data-protection rules make hardest to reach. And because the three bugs overlapped, each one "produced different symptoms on different platforms at different rates," turning correlation into guesswork.

Sit with that, because it is a company with world-class tooling describing exactly the detection problem you will have with your own agents, only smaller. Silent, non-deterministic degradation that standard checks miss. A replay you may not be allowed to read. Mixed signals from overlapping causes. Their fix is the right one and worth copying: more sensitive evaluations that can tell a healthy agent from a subtly broken one, and continuous quality monitoring in production rather than a test suite that runs before deploy and then goes quiet.

The architecture, layer by layer

Put detection, the loop, and a boundary together and you get a shape. Four layers, with the agents sitting inside them rather than on top.

The signal plane is the foundation, and it is the layer teams skip. You cannot operate what you cannot see, and an agent's useful telemetry is not CPU and memory. It is the trace of every decision: which model answered, which tools it called with what arguments, what each tool returned, how many tokens it burned, and how its own output scored against your quality checks. One replayable record per decision. Without this, every later layer is guessing.

The reasoning layer runs the first three stages of the loop and answers one question: is this agent actually broken, or just having a normal bad minute. This is where Anthropic's lesson bites. Cheap thresholds catch the loud failures, a runaway loop or a cost spike, but the subtle quality drift needs the sensitive evaluations, and it is the subtle drift that erodes trust with users before anyone files a ticket.

The action layer is small and specific on purpose. An agent has a short menu of real remediations: send its traffic to a different model, roll it back to the last known-good version, or pause it and drain its work to a human queue. Three verbs, each reversible. Reversibility is the design constraint that matters most, because the next layer decides when to pull these levers without asking.

The control plane wraps everything and is where the PocketOS failure gets prevented. The emerging consensus, described by investors and security teams alike, comes down to four rules. Each agent runs under its own least-privilege identity and never borrows a human's credentials. Policy is code, enforced at runtime, not a wiki page. Tools are contracts: signed, rate-limited, and approval-gated, not open URLs an agent can point anywhere. And every call is traceable, replayable, and costed, tagged back to the workflow that caused it. The PocketOS agent violated the first and third of those in a single command. A control plane that enforced them would have handed it a token that could not see production and a delete tool that refused to run without a signed approval.

The circuit breaker

Now the part everyone gets wrong in the other direction. Once you can auto-remediate, the temptation is to let the loop close on everything. That is how you turn a small incident into a large one.

Cloudflare gave the internet a clean lesson on 18 November 2025. An internal permissions change made a routine query return duplicate rows, which doubled the size of a configuration file that feeds their bot-management system. The file blew past a hardcoded limit of 200 features, and the proxy handling it panicked outright: thread fl2_worker_thread panicked: called Result::unwrap() on an Err value. It took roughly six hours to fully recover, across a large slice of the web. This was not an AI agent. It was automation acting on bad internal state, which is structurally the exact thing an agent does when it improvises on a wrong assumption.

Look at what Cloudflare committed to afterward, because it reads like a specification for running agents. More global kill switches, so a bad feature can be turned off network-wide without a deploy. And this line, which I would frame and hang over any agent platform: they would harden ingestion of their own internally generated configuration files "in the same way we would for user-generated input." Treat what your own automation produces as untrusted. An agent's plan is internally generated input. Validate it like it came from a stranger.

The failure mode to fear is not an agent that does nothing. It is an agent that remediates confidently into a worse state. A widely-read piece from an SRE practitioner in May 2026 gave the canonical example: a remediation agent sees elevated latency and restarts a service cluster, except the shared connection pool is already at 87 percent, and the restart triggers a thundering herd that finishes off the service the agent was trying to save. This is not hypothetical hand-waving. The real Google Cloud outage of June 2025 had the same shape, where mass automated restarts hammered a recovering backend with no backoff.

The answer is the confidence gate from the walkthrough, run as a real circuit breaker. Resolve AI describes the trust model the whole category is settling on, and it is worth stating plainly: start every agent in advisory mode where its fixes need human approval, expand its autonomy only for the low-risk, well-understood failures where it has earned a track record, and keep humans firmly in the loop for anything novel or high-stakes. Autonomy is a privilege an agent earns per failure type, not a switch you flip once. A breaker that hands ambiguous cases to a person is not a weakness in the design. It is the thing that makes the design safe enough to run at all.

What I would build first

You do not need the full four-layer platform to start, and you should not try to build it all at once. If I were adding this to a system today, and I have argued this exact order on platforms I work on, I would go in this sequence, because each step is useful alone and each one earns the right to the next.

1. Give every agent its own identity and a scoped credential. This is the single highest-impact change and it is pure plumbing, no intelligence required. An agent that cannot reach production from a staging task cannot delete it. Do this first even if you do nothing else.
2. Log one replayable record per decision. Model, tools, arguments, returns, tokens, output score. You are building the signal plane, and you will want it most on the day you have an incident and no trace.
3. Add a kill switch before you add autonomy. One control that pauses an agent or a whole class of agents network-wide, no deploy required. Cloudflare's lesson: build the off switch before you need it, not during the outage.
4. Wire detection to alerts, keep remediation manual. Let the loop watch, triage, and recommend, exactly where AWS's agent stops. You get most of the value and none of the runaway risk while you learn what your agents actually do wrong.
5. Only then, auto-remediate the failures you understand cold. Pick the one or two boring, reversible, high-frequency failures you have watched the loop handle correctly a hundred times, and let it close on those alone. Everything else keeps escalating to you.

Notice that the first three steps contain no AI at all. The hardest and most valuable parts of operating an agent fleet are old-fashioned systems engineering: identity, audit, and an off switch. The clever autonomous loop is the last thing you build, not the first.

Where this is actually going

The market is moving in this direction fast enough to trust the trend even while distrusting the numbers. Futurum Research reported in March 2025 that 89 percent of surveyed CIOs call agentic AI a strategic priority, though they published no methodology behind it, so read it as a mood, not a measurement. One analyst house sizes the broader AIOps market at 2.67 billion dollars in 2026 growing to 11.8 billion by 2034, and I quote only that one because the competing estimates for the same market ranged past 300 billion, which tells you the category is too young to measure honestly. As recently as last month, New Relic shipped Autopilot, an out-of-the-box SRE agent that triages incidents and proposes remediations. The supply side is real and accelerating.

What has not shipped is the honest part. Applying this loop to the agents themselves, rather than to infrastructure, is still mostly an articulated pattern and a set of vendor roadmaps, not a stack of proven production deployments you can copy. The strongest evidence for the idea is, oddly, the failures: the PocketOS deletions and the thundering-herd restarts are what happens when you run agents with no loop watching them. The absence is the argument.

Which is a good place to be if you build things. The infrastructure side of agentic SRE is a crowded field with big vendors and bigger claims. The side that operates the agents themselves is wide open, and the first pieces of it are unglamorous work you already know how to do: scope a credential, log a decision, wire an off switch. Start there. The autonomous part can wait until the boring part is solid, and the teams that get the boring part right are the ones that will still be standing when their agents have a bad night.

We taught the machines to run operations.
Now we have to operate the machines.

Primary sources: the PocketOS database deletion, Anthropic's postmortem of three recent issues, the Cloudflare 18 November 2025 outage report, the AWS DevOps Agent walkthrough, and the agent control plane four-pillar model.