AgentOps / AI agent reliability

Your AI Agents Need a NOC

Telecom spent six years and three standards bodies getting 4 percent of operators to Level 4 autonomy. AI agent teams are speed-running the same curve in months, without the guardrails. Here is what thirty years of network-operations discipline actually transfers.

Mohamed Kadri · July 2026 · 15 min read
A network operations center whose wall of screens shows AI agents instead of network elements
The same room, a different fleet. Operations discipline does not care what it is operating.
In this article
  1. The lesson telecom already paid for
  2. Six years to reach 4 percent
  3. Autonomy is an adoption problem
  4. The Agent NOC: what transfers
  5. What breaks in translation
  6. Your observability stack is an element manager from 1998
  7. Operate one right now
  8. The Agent NOC, in one page
  9. Questions this article answers

In April 2026, a coding agent deleted a software company's production database and its backups in about nine seconds [1]. Nine months earlier, a different agent at a different company ignored an explicit code freeze and wiped a production database it had been told not to touch [2]. Two incidents, two vendors, one shape: an agent took a real action on an incomplete view of the world, and nothing stood between its intent and the damage.

Gartner now predicts that more than 40 percent of agentic AI projects will be canceled by the end of 2027, and one of the three reasons it names is inadequate risk controls [3]. The industry's answer so far has been better prompts, better evals, and better traces. Those are all upstream of the actual problem.

The actual problem is that we are putting autonomous systems into production without an operations discipline, and there is exactly one industry that has spent the last three decades building that discipline for systems too large for humans to babysit: telecom. I spent seventeen years inside it, in the network operations centers of Tier-1 carriers. The AI industry is speed-running a lesson telecom paid thirty years to learn.

The lesson telecom already paid for

A mobile network is a fleet of millions of semi-autonomous elements that fail constantly, individually and in cascades, around the clock. Nobody can watch them all. So telecom built the NOC: a discipline, not a room. Alarms are collected, deduplicated and correlated so ten thousand sympathetic alerts collapse into one actionable signal. Root cause is traced through the network topology, because the element screaming loudest is rarely the one that is broken. Every service has an SLA and every SLA has consequences. Fixes are codified in runbooks, and runbooks that prove themselves are automated. Changes happen in maintenance windows; freezes are enforced by tooling, not by memos.

None of that was invented for elegance. Every rule is scar tissue from an outage.

When the industry decided to push past human-scale operations altogether, it did something the AI world has not done yet: it wrote down what autonomy means. TM Forum's Autonomous Networks framework (IG1252) defines six levels, L0 to L5, from manual operations to full closed-loop autonomy, and an evaluation methodology for grading real systems against them [4]. 3GPP and ETSI aligned behind the same direction [5]. Vendors built product lines around climbing those levels. This is the most systematic attempt any industry has made at answering the exact question every AI team now faces: how much do you let the machine do on its own, and how do you prove it earned that?

Six years to reach 4 percent

Here is the number that should calibrate everyone building agents. After six-plus years of that industry-wide, standards-backed, executive-sponsored effort, TM Forum's own survey of operators found roughly 21 percent had reached Level 3 or above overall, and only about 4 percent had reached Level 4 anywhere in their operations [6]. Target dates for broad L4 adoption quietly slipped from 2025 to 2030 [7]. A recent reference-architecture paper on AI agents for autonomous networks describes the industry consensus bluntly: copilot-style deployments hit an autonomy ceiling around L3, where a human still approves anything that matters [8].

I lived that climb. On a Tier-1 operator account I led the L2-to-L3 transition for network operations: an AIOps platform with ML anomaly detection and topology-based root-cause analysis that eventually automated 75 percent of Level-1 incident handling at a 99.6 percent SLA. Getting there took years, not sprints, and the gating factor was never the models. It was proving, incident type by incident type, that the automation could be trusted with the next increment of authority.

The calibration: a trillion-dollar industry with standardized autonomy levels, certification programs and purpose-built vendor stacks needed six years to get 4 percent of its members to Level 4. An agent startup claiming full autonomy in month three has not discovered a shortcut. It has skipped the part where trust gets built - and the cancellation statistics suggest the bill arrives on schedule.

Autonomy is an adoption problem

Why did telecom's autonomy targets slip five years? Not capability. The uncomfortable answer is that autonomy is not a technology problem. It is an adoption problem, and adoption science called this decades ago.

The unified theory of technology acceptance (UTAUT) models why people actually use a new system: performance expectancy, effort expectancy, social influence, facilitating conditions [9]. Rogers' diffusion-of-innovations theory adds the attributes of the technology itself: relative advantage, compatibility, complexity, trialability, observability [10]. And across the empirical literature, one variable keeps dominating high-stakes contexts: trust. I spent my MBA research measuring exactly this, reviewing thirty-eight studies of why citizens in developing countries refuse e-government services that work perfectly well; trust in the institution and trust in the technology beat every other factor [11].

Now read the agent-autonomy problem through that lens. Performance expectancy: will the agent actually resolve the incident? Effort expectancy: how much babysitting does it need? Social influence: did anyone else's agent just delete production? Facilitating conditions: is there tooling to see what it is doing? And trust, the master variable: has this specific agent, on this specific class of action, earned it?

Rogers' five factors read like an agent-rollout playbook written thirty years early. Trialability is propose-only mode. Observability is the trace and the signal plane. Complexity is the integration burden. Which means the NOC disciplines that telecom evolved are not bureaucracy. They are trust-manufacturing machinery: graduated autonomy levels, earned per scenario, with evidence. That is what an operations discipline is for.

The Agent NOC: what transfers

Call it the Agent NOC: the same operations discipline, pointed at a fleet of AI agents instead of a fleet of network elements. Most of it maps one to one.

Telecom NOCAgent fleet equivalent
Alarms from network elementsAgent events: tool errors, token spend spikes, low-confidence outputs, guardrail hits
Alarm correlation and deduplicationBehavioral pattern analysis: "agent A degrades whenever upstream agent B touches finance data"
Topology-based root causeRoot cause traced through the agent dependency graph: agent, tool, API, downstream agent
Per-service SLA and error budgetPer-agent semantic SLOs: task completion rate, factual accuracy, guardrail adherence - with a budget
Burn-rate alerting on the error budgetBaseline-relative burn rate per agent: alarm on the derivative, not the level [12]
Runbook graduation: manual, then supervised, then automatedEarned autonomy per (situation, action) pair - what the autogenic-networks literature calls solution banking [13]
Tier 1 / Tier 2 / Tier 3 escalationAgent shepherds (monitor and apply runbooks), agent mechanics (diagnose and re-prompt), model engineers (retrain and rebuild)
Maintenance windows and change freezes, enforced in toolingThe thing whose absence deleted a production database during a declared freeze [2]
Service-level monitoring, not element-levelWatch the outcome the user experiences, not the HTTP 200 the agent returned

Runbook graduation is the whole game

In a NOC, no automation is born trusted. A new incident type is handled by a human following a runbook. When the runbook proves stable, it becomes a script a human triggers. When the script proves stable across enough executions, it earns the right to trigger itself - for that incident type only. Autonomy attaches to the (situation, action) pair, never to the system as a whole.

The research frontier has now formalized the same idea for AI-driven management. A 2026 paper from Nokia Bell Labs and the University of Ottawa, accepted in IEEE Network, proposes "autogenic" network management - systems that generate their own automation - and its practical deployment mechanism is exactly runbook graduation: validated solutions from human-supervised agents accumulate in a trusted repository, recurring scenarios are handled autonomously from proven patterns, novel ones still get a human. They call it solution banking [13]. A companion line of work, the Hierarchical Agent-native Network Architecture (HANA), organizes the operating side: an orchestrator over specialized executive agents with a shared memory, validated on a live 5G core with an 86 percent MTTR reduction reported [14].

And this is not just theory. Huawei and China Mobile report a commercially deployed Level-4 "Dark NOC" where intelligent agents run network operations: fault-diagnosis automation lifted from 60 to 90 percent, mean time to repair down 30 percent, and - the detail I find most telling - hallucination rates in the operations assistant driven down to 3 percent before the system was allowed to matter, measured against a defined indicator framework [15]. Huawei's operations platform ships with more than 20 scenario agents whose risk-identification and diagnosis accuracy figures are published up front, as claimed capability boundaries, before autonomy is granted [16]. Treat vendor numbers as vendor numbers - but notice the pattern: accuracy disclosed first, authority granted second. That ordering is the discipline.

Watch the service, not the element

The subtlest transfer is the one telecom fought hardest for. Element monitoring says the base station is up. Service monitoring says the customer's call still dropped. Telecom formalized the difference as a three-layer indicator hierarchy - network KPIs feeding service-quality KQIs feeding customer-experience indices [17] - because green elements and a broken service is the default failure mode of any complex system.

Agent fleets have the identical problem, and it is the most dangerous one they have: the agent that returns HTTP 200, on time, error-free - and approved a transaction it should have refused. Latency dashboards will never catch it. Only service-level judgment does: task-level correctness, checked downstream of the agent, against what the outcome was supposed to be.

What breaks in translation

An honest mapping names its failures. Two things do not transfer, and they are both fundamental.

Determinism. A microwave link fails the same way every time; that is why runbooks converge. An LLM agent is non-deterministic and its logic is not fixed: a model update can subtly shift behavior across an entire fleet without a single error being thrown. Classic root-cause analysis assumes a stable system with discrete faults. Against gradual reasoning drift there is no root cause to find - which is why agent operations must alarm on distributions and trends, not on events. The hidden-technical-debt literature warned about exactly this property of ML systems a decade ago: change anything, and you change everything [18].

Fixed topology. A NOC's correlation power comes from knowing the network graph. In a multi-agent system the collaboration topology is partly emergent - it exists per task, assembled at runtime. Pathologies can arise from the interaction pattern of individually healthy agents, and a static dependency map cannot see them. This is a genuinely open problem, and anyone selling you a finished answer to it is early [19].

Both breaks point the same direction: the NOC disciplines that transfer are the ones about authority and evidence - who may act, on what, with what proof. The ones that need reinvention are the ones about causality. Import the former today; research the latter honestly.

Your observability stack is an element manager from 1998

The AgentOps tooling wave - LangSmith, Langfuse, Arize, AgentOps, Weave and their peers - has built genuinely good tracing: step-level replay, token accounting, eval pipelines. OpenTelemetry now has GenAI semantic conventions for agent spans [20]. In telecom terms, this is exactly where network management stood in the late 1990s: excellent element managers, no service assurance. Per-agent debuggers, no fleet operations.

Held against NOC discipline, five gaps are still open across the category:

  1. No correlation layer. Traces are per-agent silos. Nothing collapses a cascade across the agent dependency graph into one incident with one probable cause.
  2. No service-level judgment. The silent wrong-action failure - clean trace, harmful outcome - is invisible to every latency and error metric on the dashboard.
  3. No enforced budgets. Semantic SLOs and per-agent error budgets exist in blog posts, not in enforcement paths that demote an agent when the budget burns.
  4. No maintenance windows. Change freezes for agents are policy documents. The Replit incident is what a policy document looks like when tested by an autonomous system [2].
  5. No earned autonomy. Tools observe agents; almost none holds the authority ladder - propose-only to supervised to autonomous, per scenario, with automatic demotion on evidence.

Gartner's market framing agrees from the demand side: it projects "guardian agents" - AI that supervises AI - to reach 10 to 15 percent of the agentic AI market by 2030 [21]. The assurance layer is not a nice-to-have. It is the missing product category.

Operate one right now

Arguments about operations discipline are better experienced than read, so here is a small Agent NOC you can run. Six simulated agents work against their own learned baselines. Inject drift into one and watch the burn-rate breaker catch it - not because its failure rate is high in absolute terms, but because it is abnormal for that agent - and demote it to propose-only before it can act. The kill switch is yours too.

Cramped in the frame? Open the full console.

The agents and their failures are simulated. The control plane ruling on them is not: the page imports guardplane 0.2, unmodified, from npm. The new release ships the burn-rate breaker this article argues for - each agent tracked against its own trailing baseline, with three guards learned from the SRE burn-rate practice [12] and from watching real baselines get poisoned: learning freezes during an active incident, improvement is learned fast while degradation is learned slowly, and an absolute ceiling catches the slow boil that a ratio can never see. It is a few hundred lines, zero dependencies, MIT. The point is not the library. The point is that the discipline fits in a few hundred lines once you know what it is supposed to do - the knowing is the thirty years.

The Agent NOC, in one page

The telecom industry is currently pointing agents at the NOC - every vendor keynote promises a dark NOC run by AI. That direction is real, and I work inside it. This article argued the reverse arrow: point the NOC at the agents. Whoever is operating your agent fleet next year - a platform team, an SRE group, or an agent supervising other agents - they will need the discipline either way. It exists. It is documented. Telecom already paid for it.

Operate the fleet, or the fleet operates you.

Mohamed Kadri
Mohamed Kadri spent 17 years operating and transforming Tier-1 telecom networks, including leading a NOC's L2-to-L3 autonomy transition (75 percent of Level-1 incidents automated). He now builds AI and cloud products, and maintains guardplane, an open-source control plane for agent fleets. LinkedIn · mkadri85.github.io

Questions this article answers

What is the first real safety gate before letting an AI agent act on its own?

Earned autonomy per (situation, action) pair: an agent starts propose-only, and a specific action graduates to auto-execute only after it has run cleanly under supervision enough times. Reversibility plus seen-before equals allowed; everything novel drafts for a human.

How do you detect that an AI agent is degrading before it causes damage?

Alarm on the derivative, not the level: track each agent's failure rate against its own trailing baseline and trip a breaker when the burn rate exceeds about 2x, with an absolute ceiling as a backstop for slow drift. Per-action checks catch bad calls; only baseline-relative signals catch a degrading agent.

What is the difference between AgentOps and AIOps?

AIOps applies AI to operating infrastructure; AgentOps is the discipline of operating AI agents themselves - their autonomy levels, error budgets, escalation and kill switches. This article argues AgentOps should import telecom NOC practice rather than reinvent it.

Do AI agents need a kill switch?

Yes, built before the incident, at two scopes: per-agent and fleet-wide, checked in the execution path so a tripped switch blocks the next action without a deploy. A gate you have never watched deny something is decoration, so test it on a schedule.

How many autonomy levels should an AI agent system define?

Telecom's TM Forum framework uses six (L0 manual to L5 fully autonomous), and after six years only about 4 percent of operators have certified L4 anywhere. The exact count matters less than writing levels down and grading each agent honestly - ungraded autonomy defaults to whatever the worst prompt allows.


References

  1. The Register: Cursor/Opus agent deletes PocketOS production database and backups (Apr 2026). Also covered by Fast Company.
  2. The Register: Replit AI agent deletes SaaStr production database during an explicit code freeze (Jul 2025).
  3. Gartner press release: Over 40 percent of agentic AI projects will be canceled by end of 2027 (Jun 2025).
  4. TM Forum: IG1252 Autonomous Network Levels Evaluation Methodology; IG1392 AN Levels Assessment and Certification.
  5. ETSI: GR ENI 051: AI Agents based Next-generation Network Slicing (2025); 3GPP TR 22.870 6G study.
  6. Fierce Network on TM Forum survey data: Telcos hit Level 4 autonomous network milestone (2025).
  7. RCR Wireless: The race to Level 4 (Aug 2025); TM Forum: A regional guide to autonomous networks progress (PDF).
  8. arXiv: Leveraging AI Agents for Autonomous Networks: A Reference Architecture (2025).
  9. Venkatesh, Morris, Davis and Davis: User Acceptance of Information Technology: Toward a Unified View, MIS Quarterly 27(3), 2003.
  10. Rogers, E. M.: Diffusion of Innovations, 5th ed., Free Press, 2003.
  11. Kadri, M.: Factors affecting electronic government adoption and acceptance in developing countries, MBA thesis, ESLSCA Business School, 2018 (Zenodo; PDF mirror).
  12. Google SRE Workbook, ch. 5: Alerting on SLOs (multi-window, multi-burn-rate).
  13. Djukic, Acharya, Kennouche and Kantarci: From Agentic to Autogenic Network Management for AI-Native 6G and Beyond: A Standards Perspective, accepted in IEEE Network (2026).
  14. arXiv: From Automated to Autonomous: Hierarchical Agent-native Network Architecture (HANA) (2026).
  15. TM Forum case study: China Mobile achieves Level 4 AN in network operation center with intelligent agents; Huawei: Dark NOC TM Forum Excellence Award (2025). Vendor-reported figures.
  16. Huawei: AUTINOps AI-Native intelligent operations launch (MWC 2026). Vendor-reported figures.
  17. Huawei SmartCare CEM: the CEI-KQI-KPI indicator hierarchy.
  18. Sculley et al.: Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015.
  19. Han et al.: LLM Multi-Agent Systems: Challenges and Open Problems (2024).
  20. OpenTelemetry: AI agent observability and GenAI semantic conventions (2025).
  21. Gartner press release: Guardian agents will capture 10-15 percent of the agentic AI market by 2030 (Jun 2025).

Part of a series on operating AI in production: Who Operates the Operators? introduced the incident loop for agent fleets; One Repository, One Shared Brain covered the memory layer. The runnable companion to all three is guardplane.