MELLORCATalk to Mellorca
Discovery

Cloud, Infrastructure & Platform Operations

When monitoring produces more alerts than your team can triage

Cloud monitoring can generate hundreds of alerts a week, most needing no action. How alert noise hides real incidents, drives up cost, and how to cut it back to signals people act on.

Mellorca Discovery2026-10-07

A business moves its systems to the cloud and switches on monitoring for everything it can measure. Within a few months the operations channel receives hundreds of alerts a week. Most resolve themselves within minutes. Many arrive in clusters, twenty at a time, all describing the same underlying problem. The team stops reading them closely. Then a real problem arrives, looking exactly like the noise, and customers notice before anyone in IT does.

This is not a shortage of monitoring. It is the opposite: alerts are generated faster than people can read, judge and act on them. Buying more tools or hiring more staff rarely fixes it, because the volume grows with every new service that is switched on.

The problem in plain language

Every alert costs attention. Someone has to read it, decide whether it matters, find out what caused it and either act or dismiss it. When the number of alerts exceeds the time available for that work, people start skimming, muting channels and building private filters. The monitoring still runs, but the business has quietly stopped relying on it.

Typical signs include:

  • one incident produces dozens of separate alerts from different components;
  • alerts fire on thresholds such as CPU or memory that rarely affect customers;
  • planned maintenance and deployments trigger floods of alerts that everyone ignores;
  • alerts stay open long after the problem has cleared;
  • the team cannot say how many alerts last month led to a real action.

This article is about volume and signal quality. Who should receive each alert is a related but separate question, covered in our piece on why integration failures should not route to personal inboxes.

What the buyer is actually trying to solve

People search for phrases such as "too many monitoring alerts", "alert fatigue in operations", "how to reduce alert noise" or "why are our monitoring costs going up". Behind the search is usually a specific moment: an outage that was alerted but missed, a monthly observability bill that has jumped, or an engineer who has resigned partly because of out-of-hours interruptions.

What they need is a smaller number of alerts that each mean something, with the rest turned into dashboards, tickets or nothing at all.

Evidence and system mechanism

Good alerts are judged by precision as well as coverage

Google's site reliability engineering workbook chapter on alerting on service level objectives sets out four attributes for judging an alerting strategy: precision (the proportion of detected events that were significant), recall (the proportion of significant events that were detected), detection time and reset time. Most noisy alerting set-ups are tuned for recall alone: alert on everything in case something matters. The workbook's recommended approach ties alerts to how quickly a service is using up its agreed error budget, and sends urgent pages only for fast burn while slower problems become tickets.

Monitoring tools already include ways to collapse duplicates

The Prometheus documentation for Alertmanager describes three mechanisms aimed directly at this problem. Grouping combines alerts of a similar nature into a single notification, which matters when an outage triggers many alerts at once. Inhibition suppresses notifications for some alerts when a related, more important alert is already firing. Silences mute matching alerts for a set period, for example during maintenance.

Microsoft offers comparable controls. Its documentation on alert processing rules in Azure Monitor explains how to suppress notifications during planned maintenance or outside business hours, and how to apply notification logic in one rule rather than setting it separately on every alert rule as the number of rules grows. It also notes the weakness of the common workaround of disabling and re-enabling alert rules around maintenance windows: it is time-consuming and error-prone.

The mechanism behind the noise is therefore usually configuration, not tooling. The features to group, suppress and prioritise exist; they were simply never set up.

Problem owner and why now

The problem owner is normally the IT operations or infrastructure lead. The budget owner is the CIO, CTO or, in smaller businesses, the financial director who pays the observability and on-call bills.

The trigger is often a cost spike. Many monitoring and logging platforms charge by data volume, hosts or events, so the same habits that create noise, such as collecting everything and alerting on everything, also drive cost. Other triggers include a missed incident, a failed audit of incident response, or the loss of an experienced engineer.

Economic consequence

  • Missed incidents: real alerts are ignored because they look like routine noise, so outages last longer.
  • Wasted engineering time: skilled people spend hours acknowledging alerts that need no action.
  • Staff fatigue and turnover: constant interruptions, especially at night, wear people down. See the cost of constant interruptions and context switching.
  • Rising tool spend: more data and more rules mean higher bills, without better outcomes.

Estimate the cost for your own team by counting alerts per week, multiplying by the average minutes to triage each one, and comparing that with the number that led to a real fix.

Root cause

  • Alerts on causes, not symptoms: rules watch component metrics rather than what users experience.
  • Default rules left on: templates and vendor defaults were enabled at migration and never reviewed.
  • No grouping or suppression: one failure produces many notifications.
  • No retirement process: alerts are added after every incident and never removed.
  • No measure of alert quality: nobody tracks which alerts led to action.

Practical intervention

  1. Count and classify. Export the last 30 to 90 days of alerts. For each rule, record how often it fired and how often someone took a meaningful action.
  2. Delete or demote. Rules that never led to action should be removed or turned into dashboard items or daily reports.
  3. Alert on customer impact. For important services, define what "working" means to users, such as successful checkouts or response times, and alert when that degrades.
  4. Group and suppress. Configure grouping, inhibition and maintenance suppression so one failure produces one notification.
  5. Separate urgency levels. Only conditions needing immediate human action should interrupt someone; everything else becomes a ticket for working hours.
  6. Review monthly. Make alert quality a standing agenda item, with a target for the share of alerts that are actionable.

An illustrative example

Illustrative: a Cape Town online retailer running on a public cloud receives about 400 alerts a week after migration. A review shows that a handful of rules generate most of them, mainly disk and CPU thresholds on servers that scale automatically. The team removes those rules, adds grouping so a database failure produces one notification rather than thirty, and creates a single alert on failed checkout rate. Weekly alerts fall to a number the team can read properly, and the next real problem is spotted within minutes.

Diagnostic questions

  • How many alerts did your team receive last month, and how many led to action?
  • Does one outage produce one notification or dozens?
  • Are alerts suppressed during planned maintenance and deployments?
  • Do your most important alerts describe what customers experience?
  • When did someone last delete an alert rule?
  • Do you know which monitoring data drives most of your observability bill?

What good looks like

Every alert that interrupts a person is actionable, linked to a service that matters and accompanied by enough context to start work. Duplicate alerts are grouped. Maintenance is silenced automatically. Lower-urgency problems become tickets. The team reviews alert quality regularly, and the number of alerts grows more slowly than the number of services. Monitoring is trusted again, which is the point of having it. Our insight on why automated workflows need observability covers the same principle for automations.

Where Mellorca fits

Mellorca's managed digital operations service reviews existing monitoring and alert rules, removes noise, configures grouping and suppression, and defines service-level alerts tied to business outcomes. The work is practical: fewer, better alerts and a simple monthly review that keeps them that way.

Method note

This article draws on public documentation from Google's site reliability engineering material, the Prometheus project and Microsoft Azure. Specific features and pricing vary by monitoring platform. The example is illustrative and not drawn from a named client.

Sources and further reading

Related Mellorca reading

Article summaryCloud monitoring can generate hundreds of alerts a week, most needing no action. How alert noise hides real incidents, drives up cost, and how to cut it back to signals people act on.
Explore our solutionsStart a conversation