Five Thousand Alarms, One Ticket

What an assurance platform does to a fiber cut between the first alarm and the one ticket that reaches the right team: filtering, deduplication, parent-child rules, cross-domain correlation, demarcation, and the arrival-order problem that can split one incident into many tickets. Part 2 of the topology series.

At 03:12 a backhoe finds a fiber on an unprotected metro section. The section carries two wavelengths, and behind them twenty IP links, forty microwave hubs and one hundred eighty base stations go dark within the same minute.

The platform receives about five thousand alarms. Not one of them says "fiber cut." Two minutes later the fiber team has one ticket with the segment, the priority and the impact on it. This article is about what happened in between.

The cut and the burst

In Part 1 I described how an assurance platform reconstructs the network map from raw exports and why alarm correlation is only as good as that map. This part follows one incident through the alarm pipeline that sits on top of it. The numbers are illustrative; the shape of the burst is what every operator sees.

Every layer alarms on its own, in its own vocabulary. The optical equipment raises loss of signal on the cut section, then forward defect and AIS indications on every wavelength and every client signal it carries [1]. Those indications exist precisely to suppress alarms further down the chain, but depending on how the equipment and its northbound interface are configured, many of them still reach the platform as records. The routers raise interface down on every affected link, then BFD session down [2], routing adjacency down, label session down, tunnel down and pseudowire down for every service on those interfaces that protection or rerouting does not save. The microwave hubs behind those routers raise uplink down. The base stations raise transmission link down, then S1 or NG link down, X2 or Xn down and cell unavailable per cell. The core raises a signaling association down for every one of those base stations. Five layers report one cause, each in its own dictionary.

Why does one cut raise five thousand alarms?

The IETF definition is the clearest one: an alarm "signifies an undesirable state in a resource that requires corrective action" [3]. The older ITU-T definition, which every alarm format still inherits, adds the fields that make it usable: probable cause, perceived severity, the managed object it refers to [4]. The 5G management specification carries the same fields today, keeps the list of correlated notifications that X.733 already defined, and adds a root cause indicator [5]. Both matter for this article.

Two terms get mixed up, so here is how this article uses them. Alarm correlation groups related alarm instances using their identity, timing, dependencies and fault rules. Root cause analysis weighs that evidence to name the underlying fault. A correlated group alone does not prove the cause. Demarcation names the likely fault domain; a test in the field confirms it.

Every element applies these definitions locally. A router does not know that its interface went down because a fiber was cut two layers below it; from where it stands, an interface down is an undesirable state that requires corrective action, so it alarms. Multiply by every layer and every service and you have the burst. The process industries have measured what a burst does to a human: their standards define an alarm flood as more than ten new alarms in ten minutes for one operator, and the commonly quoted operating benchmarks put six an hour as an acceptable average and twelve as the maximum manageable one, as workload guides rather than hard limits [6] [7]. Telecom alarms are no cleaner: in a study of close to twenty million alarms at a mobile operator, only 2.5 percent required manual action [15]. A fiber cut delivers five thousand in one minute. No screen and no person absorbs that, which is why the pipeline exists.

The pipeline shrinks the burst

Each alarm passes a fixed chain of rules before anyone sees it. The logical chain is the same across vendors; platforms order and place the stages differently, and the rule content differs. Figure 1 shows the chain with the counter for our fiber cut.

1. The collector

The collector is the same component that fills the inventory tables in Part 1, with a different rule set. It standardizes the vendor record into the common alarm format, keeping the vendor's own alarm identifier as a field for reference. It applies a mask list per vendor: alarms that are known noise on that platform ("clock source switched", a nightly self-test) are discarded or archived before they cost anything. It protects the rest of the platform against storms by throttling per source into a bounded queue that spools to disk for replay, reporting the lag, and reconciling active alarms with the source if anything was lost, before correlation or automatic closure relies on them. And it enriches the record with site, region, domain and network layer from the inventory, which is the first place the map is used. Five thousand becomes about three thousand eight hundred.

2. Deduplication and compression

A port that flaps raises and clears thirty times. Deduplication keys each alarm instance on its source, object, alarm type and any qualifier, and every repeat updates that one record: a count goes up, a last-occurrence time moves, nothing new appears. A clear changes the state of the instance; it is not another occurrence. Flap detection sits beside it and keeps the timestamps of each state change, so the history survives. The design is old; the classic event platforms document it as an identifier field that controls deduplication and an automatically maintained tally of inserts [8]. Delay-count rules build on the same mechanism: an alarm is only promoted to the operator after it repeats N times inside T seconds. Twenty-one hundred.

3. Transient suppression

Every new alarm is held for a short window; in the platforms I have run it was set somewhere between thirty seconds and two minutes, per alarm class. If the clear arrives inside the window, the alarm is archived as a message and is never shown as active. Loss-of-signal alarms skip the window, so the root is never delayed. The noise around them, protection switches that settle and sessions that re-establish, clears inside it. Nineteen hundred.

4. The discriminator

Not everything that arrives is an alarm in the RFC sense. Informational and low-severity classes go to a message channel: kept, searchable, exported, but not shown by default and never ticketed. Eleven hundred alarms remain; eight hundred messages sit beside them.

5. The planned outage check

If the affected objects fall inside an approved maintenance window, the alarm is tagged planned; it stays visible, but ticket dispatch and notification are suppressed. This stage is only as good as its input. Windows typed by hand are always late, so the useful implementation reads them from the change system automatically. Not the case tonight; eleven hundred stay.

Parent and child inside one domain

Now the biggest single reduction, and it still needs no network map: parent-child rules inside each element manager, using the object model the manager already has. The idea is thirty years old. Jakobson and Weissman defined alarm correlation in 1993 as "a conceptual interpretation of multiple alarms such that new meanings are assigned to these alarms," and listed the basic operations: compression, count, suppression, boolean patterns, generalization [9]. Correlation engines still implement versions of that list.

The rules are per vendor and per alarm type: a parent alarm type, a set of child alarm types, a time window, and an object hierarchy (same object or a child object). The router's own object model is enough to know that a BFD session belongs to an interface; no map is needed. Eleven hundred alarms become two hundred fifty-one parents: two optical, one at each end of the section; twenty IP, one per link; forty microwave; one hundred eighty radio; and nine core, because in this example the core manager is configured to group its per-site association alarms by tracking area, a local policy rather than a standard mapping. Every one of them is a real parent inside its own domain.

Cross-domain correlation on the map

This stage uses the map from Part 1. The engine walks the restored paths: the twenty IP links are carried by the two wavelengths on the section; the forty hubs and one hundred eighty base stations sit behind those links; the core alarms name exactly those base stations. The rule is a walk over the map: an alarm on a node whose upstream path contains a failed link, inside the time window, is a child of that link's alarm. The same walk handles rings, virtual functions on hosts, and optical client-to-line layers, over a different graph each time. The walk has to use current forwarding and protection state as well as inventory: a protected ring, an ECMP bundle or a rerouted service can survive a failed link, and a match the engine cannot confirm stays a candidate rather than a child.

The last operation is synthesis. Loss of signal at OTN A and loss of signal at OTN B are two vendor alarms for one cause, so the engine creates one synthetic root, "section down," and hangs both under it. Two hundred fifty-one parents become one synthetic root with two hundred fifty-one children. Nothing is deleted: all eleven hundred alarms stay active and linked to the root, and correlation changes what a person has to look at. The standards give this result a place to live: the IETF model recommends "a single alarm for the underlying problem" with the affected resources listed in it rather than separate alarms [3], the 3GPP alarm record carries a root cause indicator and a list of correlated notifications to record it [5], and the TM Forum alarm API has the same idea as a root-cause flag with parent and correlated alarm lists [11]. The fields record the diagnosis; they do not prove it, and building a synthetic root is an implementation choice.

Event and demarcation

The root, its children and the impact count become one event with a state: open, in progress, recovering, closed. Children that arrive late attach to the existing event; they never open a second one. Then demarcation decides who owns it, with a rule table:

The first row alone does not prove a cut. With automatic laser shutdown, a failed transmitter makes the far end turn its own laser off, and both ends can report loss of signal [16]. The equipment-alarm row is what carries the decision. Run the same table with loss of signal at one end only and a laser bias alarm on the far node, and the answer is a transmitter card on OTN A and the optical team. Run it with a clean optical layer and interface down on the routers, and the answer is IP. Demarcation is the step that turns "five thousand alarms" into "suspected fiber fault between site A and site B, fiber team, P1, one hundred eighty base stations affected." The fiber team confirms it with an OTDR trace before a crew digs. Without these rules, the step falls to a human on the phone.

One ticket, and the ordering problem

One ticket per event, not per alarm. The ticket number is written back to every alarm in the event as a tag, all eleven hundred of them, so whoever opens any of them sees the same ticket. Notification goes once, to the on-duty list of the owning team. When the root clears, the engine waits a stability timer, checks that the children cleared too, and closes the ticket by policy; a child that does not clear with the root is re-evaluated as a new root, because a second fault was hiding behind the first. A repeat on the same section within a few days opens a new trouble ticket for the outage and a linked problem ticket for the recurring cause.

Now the detail that breaks most implementations. The alarms do not arrive in cause order.

Radio alarms reach the platform in seconds; the radio manager's northbound interface is fast. Microwave takes twenty seconds, IP thirty, and the optical manager, which owns the actual cause, reports after a minute. For that minute the engine sees one hundred eighty base stations down with no root cause, which looks exactly like one hundred eighty independent faults. Two mechanisms make this survivable. Ticket dispatch is held for a delay longer than the slowest interface, so no child ticket is opened before the parent can arrive; this only works because loss-of-signal alarms skip the transient window, otherwise the hold would also have to cover it. And correlation is re-evaluated when the parent arrives, re-parenting the children retroactively; if a child ticket was opened anyway, it is merged into the parent ticket rather than left open. Set the hold shorter than the slowest interface and you get a second ticket for the same incident, and the radio team learns about the transmission team's outage from its own escalation.

None of this is new. Cisco filed a patent in 2000 for an aging queue that holds out-of-order side-effect alarms [13], and an AT&T patent filed in 2008 holds each symptom for a correlation life span and, if no root arrives, promotes it to root with its own ticket [14]. The 3GPP model even has a notification for a correlation that changes after the fact [5]. In my experience, what is rarely done is setting the hold from measurement. The TM Forum alarm API carries both the time an alarm was raised at its source and the time the owning system reported it [11]. Add the time the correlation engine received it, and you can measure how long the root arrives after the first symptom, per pair of interfaces, with clock quality checked. That relative delay is what the hold has to cover. Size the hold from the distribution of those delays, not from one sample, start it at the first qualifying alarm (here, 5 s plus 120 s gives dispatch at about 125 s), and keep re-correlating after dispatch, because no finite hold catches every late root.

For the process people: the whole chain is TM Forum's Resource Trouble Management. Stages 1 to 7 are "Survey and Analyze Resource Trouble," which the framework describes as filtering and correlating alarm event notifications; demarcation is "Localize Resource Trouble"; the ticket is "Create," "Track and Manage" and "Report Resource Trouble," and closure is its own step [10]. The framework names the steps and the handovers. The windows, keys and hold times are engineering choices that every vendor makes differently.

How do you know the pipeline works?

The fourth check deserves the last word. The process industries settled decades ago on alarm rate per operator as the health metric of an alarm system [6]. Telecom operations centers rarely publish theirs, but the same discipline applies: if you cannot say how many alarms came in yesterday, how many a person saw, and how many tickets left, you do not know whether your pipeline is working.

This is also what the autonomy talk comes down to. In TM Forum's autonomous network levels, fault management moves from Level 2 to Level 3 when the system owns the awareness, shares the analysis, and starts to take part in the decision [17]. Stages 1 to 7 are the awareness and most of the analysis; demarcation and the single ticket are the first part of the decision. That makes this pipeline the core of what Level 3 fault management needs; a formal level still comes from assessing a defined scenario. Agentic NOC projects start from the output of this pipeline, and the agents themselves need the same operational discipline, as I argued in Your AI Agents Need a NOC and Who Operates the Operators?. If the map is wrong or the alarms arrive out of order, an agent reasons confidently about the wrong ticket.

The seven-minute version

The animated version of this article, for a design review or a new joiner: the burst, the pipeline with its counter, the three parent-child trees, the walk on the map, the event and demarcation table, the arrival timeline with the dispatch hold, and the four checks. Vendor-neutral, free to reuse with attribution.

Five thousand in, one out. Every stage in between is a rule, and every rule needs an owner who checks its hit rate after each network or software change.

Mohamed Kadri operates and transforms Tier-1 telecom networks, and led a NOC's L2-to-L3 autonomy transition that automated 75 percent of Level-1 incidents. He now builds AI and cloud products, and maintains guardplane, an open-source control plane for agent fleets. LinkedIn · mkadri85.github.io

Quick questions and answers

Why does one fiber cut produce thousands of alarms?

Because every layer applies the definition of an alarm locally. The optical equipment reports loss of signal on the section and defect indications per wavelength and client signal; routers report interface, BFD, routing, label, tunnel and pseudowire failures per link and per service; microwave hubs, base stations and the core each report their own view. Five layers report one cause, each in its own alarm dictionary.

What is the difference between alarm filtering, deduplication and correlation?

Filtering decides which notifications are kept and shown (mask lists, low-severity classes); planned-outage handling keeps the alarms but suppresses dispatch. Deduplication collapses repeats of the same alarm on the same object into one record with a count. Correlation assigns new meaning to a group of alarms: which one is the cause and which are consequences, first inside one domain by parent-child rules, then across domains by walking the topology.

How does the platform know the fiber is the cause and not a router?

By walking the restored paths from Part 1: the IP links are carried by the optical section, the hubs and base stations sit behind those links, so their alarms become children of the optical alarm. Then demarcation applies a rule table: loss of signal at both ends with no equipment alarm on either node points at the medium as the leading suspect, to be confirmed by an OTDR trace (both conditions matter, because automatic laser shutdown can make a transmitter fault look like loss of signal at both ends); loss of signal at one end with a transmitter alarm points at a card.

Why is only one ticket created for hundreds of alarms?

Because tickets are created per event, not per alarm. The root alarm, its children and the impact count form one event; the ticket number is written back to every alarm in the group, notification goes once to the owning team, and the ticket closes when the root clears and the children have cleared too.

What is the ordering problem?

Alarms reach the platform in interface order, not cause order: radio in seconds, optical after a minute. For that minute the engine sees many base stations down with no root cause. Ticket dispatch must be held longer than the worst arrival delay across the northbound interfaces, and correlation must re-parent the children when the cause arrives, otherwise one incident produces a second ticket. The mechanism is old; setting the hold from measured interface latency is still rare.

How do I know whether my alarm pipeline is healthy?

Four numbers per day: alarms in, alarms shown to a person, events, tickets. Add rule hit counts per rule so a vendor upgrade that renames alarms shows up as a rule dropping to zero. The process industries use alarm rate per operator as the health metric; the same idea applies to a network operations center.

Sources and references

Standards are cited for the mechanisms named in the text. The fiber-cut scenario and its counts are illustrative, built from the layers that alarm in practice; the transient window values are the author's field practice.

1. ITU-T, Recommendation G.798: Characteristics of optical transport network hierarchy equipment functional blocks. Clause 8.10, generic layer fault processing: consequent actions "for suppression of unwanted downstream alarms", with forward defect indication insertion in the optical layers and ODU AIS under signal fail (checked against the 09/2023 edition). Supports: why one cut appears as loss of signal plus defect indications on every wavelength and client signal (section: The cut and the burst).
2. D. Katz and D. Ward, RFC 5880: Bidirectional Forwarding Detection (BFD), IETF, 2010. Supports: BFD detects failures in the forwarding path "including the interfaces, data link(s)", so a lost link also takes the BFD session down (sections: The cut and the burst; Parent and child).
3. S. Vallin and M. Bjorklund, RFC 8632: A YANG Data Model for Alarm Management, IETF, 2019, sections 3.1, 3.4 and 3.6. Supports: the definition of an alarm as an undesirable state requiring corrective action; alarm instance identity by resource and alarm type; the recommendation of a single alarm with affected resources rather than separate alarms (sections: Why does one cut raise five thousand alarms?; Cross-domain correlation).
4. ITU-T, Recommendation X.733: Alarm reporting function, 1992. Supports: the probable cause, perceived severity and managed object fields inherited by every alarm format (section: Why does one cut raise five thousand alarms?).
5. 3GPP / ETSI, TS 128 111 V18.5.1: 5G; Management and orchestration; Fault management, 2025, clauses 6.4, 6.10, 7.3.1, 7.3.4 and 7.4.1. Supports: perceived severity and probable cause in current standards; correlated notifications inherited from X.733 clause 8.1.2.9; the rootCauseIndicator attribute of the AlarmRecord; and the notification for a changed correlation (sections: Why does one cut raise five thousand alarms?; Cross-domain correlation; One ticket, and the ordering problem).
6. K. VanCamp, Alarm Management by the Numbers, Chemical Engineering, March 2016, pp. 1-2 (secondary; the EEMUA 191, ISA-18.2 and IEC 62682 texts are paywalled). Supports: the commonly quoted benchmarks of about six alarms per hour as acceptable and twelve as the maximum manageable per operator, and an alarm flood as more than ten new alarms in ten minutes (sections: Why does one cut raise five thousand alarms?; How do you know the pipeline works?).
7. EEMUA, Glossary: Alarm flood / alarm overload. Supports: the definition of an alarm flood as more alarms than a single console operator can physically address, per EEMUA 191 (section: Why does one cut raise five thousand alarms?).
8. IBM, Netcool/OMNIbus 8.1: alerts.status table reference. Supports: the identifier field that controls deduplication and the automatically maintained tally of inserts and updates, as the documented form of the deduplication mechanism described (section: Deduplication and compression).
9. G. Jakobson and M. D. Weissman, Alarm Correlation, IEEE Network, November 1993. Supports: the definition of alarm correlation as a conceptual interpretation of multiple alarms, the correlation operations (compression, count, suppression, boolean, generalization), and the role of the network configuration model in correlation rules (section: Parent and child inside one domain).
10. TM Forum, GB921 Business Process Framework (eTOM) suite, v25.0. Supports: Resource Trouble Management as the process the pipeline implements, and its Level 3 processes: Survey and Analyze Resource Trouble (filtering and correlation of redundant, transient or implied alarm event notifications), Localize, Create, Track and Manage, Report and Close Resource Trouble (section: One ticket, and the ordering problem).
11. TM Forum, TMF642 Alarm Management API, v5.0.1 specification (Apache 2.0). Supports: isRootCause, parentAlarm, correlatedAlarm and plannedOutageIndicator as standard alarm fields, and alarmRaisedTime ("the time ... at which the alarm occurred at its source") versus alarmReportingTime ("the time ... at which the alarm was reported by the owning OSS") (sections: Cross-domain correlation; One ticket, and the ordering problem).
12. Narration script of the video above: Five thousand alarms, one ticket, script with timings, 7 minutes 37 seconds.
13. Cisco Technology, US 6,694,364 B1: System and method for suppressing out-of-order side-effect alarms in heterogeneous integrated wide area data and telecommunication networks, filed June 2000, granted 2004. Supports: an aging queue that holds possibly leaking alarms until the cause arrives, with about 33 percent fewer alarms reported in a production network (section: One ticket, and the ordering problem).
14. AT&T Intellectual Property, US 8,018,859 B2: Method and apparatus for asynchronous alarm correlation, filed December 2008, granted 2011. Supports: a correlation life span during which a symptom waits for its root, and promotion of the symptom to root with a trouble ticket if none arrives (section: One ticket, and the ordering problem).
15. S. Wallin, Alarm and Service Monitoring of Large-Scale Multi-Service Mobile Networks, licentiate thesis, Luleå University of Technology, 2009, section 2.2.1.1. Supports: "Only 2.5 % of the alarms required manual actions" in a mobile operator database of close to twenty million alarms (section: Why does one cut raise five thousand alarms?).
16. ITU-T, Recommendation G.664: Optical safety procedures and requirements for optical transmission systems, 10/2012. Supports: automatic laser shutdown, which makes the far end switch its laser off so a single fault can appear as loss of signal at both ends (section: Event and demarcation).
17. TM Forum, IG1252 Autonomous Network Levels Evaluation Methodology, v1.2.0. Supports: autonomy levels L0 to L5 evaluated per scenario, such as fault management, across awareness, analysis, decision and execution (section: How do you know the pipeline works?).

Related reading, for the strategy view of the same problem: Stop Managing Alarms: An Incident-First Blueprint for Telecom AIOps (Towards Data Science, September 2026) and Alarm Correlation in the NOC (5G6G Academy, 2026). This article adds demarcation, the arrival-order problem, closure and the health checks.