This guide sets up metric-threshold alerting end to end: getting host, application, and SNMP-polled network metrics in, defining a threshold that fires only when it means it, routing the resulting incident to on-call, and wiring the metric_alert.fired event into your own automation. The result is that a saturated disk, a latency budget blown, or a switch interface in trouble opens a real incident the analyst can explain — not a lonely email.
You can only alert on a series that exists, so the first step is making sure the metric you care about is arriving. There are three sources, and they all land in the same place.
Application metrics arrive over OpenTelemetry. If your services already export metrics via OTLP, point the exporter at the endpoint and your request rates, latencies, and business counters land as queryable series. Host metrics — CPU, memory, disk, and network for a server — come from the one-line Linux Sensor, which ships them automatically once installed. And network-device metrics — interface status, throughput, errors, and saturation — flow in when the collector polls a device over SNMP; the network onboarding guide walks through configuring that.
The important point is that, however a metric arrives, it becomes the same kind of object: a named series you can query and put a threshold on. A disk-full on a server, a latency budget on a service, and a saturating interface on a switch are all just series crossing lines, set up identically. Confirm the series you want to alert on is visible under Metrics before you build the alert — if it is not there, the alert has nothing to watch, and the fix is on the ingestion side, not the alerting side.
If you are not sure where to begin, a handful of metric alerts cover the failures that most reliably hurt, and they are worth setting on day one. Free disk under a floor on any host that stores data is the classic silent killer — a disk fills gradually, nothing alarms, and then everything that writes to it falls over at once; a threshold with a sensible sustain window turns that from a 2 a.m. outage into a daytime warning with hours of runway. Latency over a budget on the endpoints your users actually touch catches the slow degradations that status checks miss entirely. And error rate above a baseline on a critical service surfaces a bad deploy or a failing dependency before the support tickets do.
For hosts, a small set of saturation alerts — sustained high CPU, memory pressure, and a disk floor — covers the ways a server most commonly gets into trouble. For network devices, the equivalents are interface saturation and a rising error counter. None of these need to be elaborate; the value is in covering the obvious failure modes well, with honest severities and sensible durations, rather than in building a sprawling threshold for every conceivable metric. Start with the handful that map to real user or operational impact, and add more only when a specific incident teaches you that you needed one.
Two mistakes turn metric alerting from an asset into noise, and both are easy to make. The first is alerting on causes instead of symptoms. It is tempting to add a threshold for every internal counter, but most of them do not map to anything a human should act on, and each one is a future false page. Prefer to alert on the symptoms your users or your operations actually feel — slowness, errors, saturation, things running out — and let the analyst trace the cause when one of those fires. You do not need a threshold for the runaway log; you need one for the disk it is filling, and the investigation will find the log.
The second is dishonest severity. If everything is critical, nothing is, and the loud channels that should mean "wake a human now" become background noise people learn to swipe away. Reserve critical for genuine, act-now emergencies; route the merely-important to a channel that does not ring a phone; and send the routine to automation. The discipline of matching the loudness of the alert to the real stakes of the breach is what keeps the whole system trustworthy over time — and a trustworthy alert is the only kind that does its job when it finally matters.
A final practice worth adopting is to review your metric alerts periodically against reality. Thresholds set months ago drift out of relevance as your systems grow — a disk floor that was generous becomes tight, a latency budget that was strict becomes loose, a host that mattered gets decommissioned. A light quarterly pass to retire alerts that no longer fire usefully and adjust the ones that fire too often or too late keeps the set honest and the noise low. Use the analyst's dispositions as your guide: if a threshold keeps opening incidents that get closed as nothing, it is telling you to move the line. A small, current set of well-tuned alerts you trust beats a large, stale set you have learned to ignore, every time.
Taken together, these practices add up to a simple philosophy: a metric alert is a promise that crossing this line genuinely warrants attention, and every false page erodes that promise. Set durations so blips do not fire, severities so the loudness matches the stakes, thresholds on symptoms users feel, and review the set as your systems change — and the promise holds. When it holds, your team trusts the alerts, acts on them quickly, and is not slowly trained to swipe them away. That trust is the entire point of metric alerting, and it is worth a little ongoing care to protect.
A good metric alert is a series, a condition, and a duration — plus the severity and routing that decide how loudly it speaks.
Create a metric alert and give it five things: the series to watch, an optional filter to scope it to one host or device, the comparison that defines a breach, the duration the condition must hold, and a severity. A typical critical alert on a database host's free disk looks like this:
Series: host.disk.free_percent Filter: host = db-primary Condition: < 10 For: 5m Severity: critical Channels: voice + pagerduty (on-call)
The duration is the part teams most often get wrong by omitting. Without it, a single sample dipping below the line pages someone, and after a few false wake-ups the alert gets muted — the classic path to a useless alert. Requiring the condition to hold for a window (five minutes here) means you are alerted to a disk that is genuinely and persistently filling, not to a transient blip that healed itself. Set the duration to the timescale of the real problem: short for things that matter immediately, longer for slow trends where a brief excursion is harmless.
Severity is not cosmetic — it drives routing. A critical alert can ring a phone and hit your on-call tool; a warning can post quietly to a chat channel. Setting severity honestly, so that only genuine emergencies are critical, is what keeps the loud channels meaningful. The same threshold setup works identically for an SNMP- polled interface metric as for a host disk, so once you have built one, you have built them all.
When the condition holds, the alert opens a first-class incident — the same kind a failed check or a fired detection opens — which means everything the platform does with incidents applies. It routes through your escalation policy and on-call schedule to whoever is genuinely on call, climbs the chain if they do not acknowledge, lands on the topology map against the affected host or device, and groups with related incidents that share a root cause so one failure does not page you many times. You configure on-call once, and every metric alert obeys it.
A threshold crossing is a symptom; the cause is usually one timeline away. Because the incident lives beside your logs and events, the analyst can investigate it and return a root cause — why the disk filled, not merely that it did. That turns a 2 a.m. page into a decision rather than the start of an investigation, which is the whole point of metrics living in the same platform as everything else.
Every metric alert emits a structured, signed webhook event when it fires, so your own systems can react without a human in the loop:
// You receive a signed metric_alert.fired event:
{
"event": "metric_alert.fired",
"series": "host.disk.free_percent",
"host": "db-primary",
"value": 8.2,
"threshold": 10,
"incident_id": 5102,
"severity": "critical"
} Subscribe to it to trigger a runbook, feed a dashboard you own, or hand the incident to an agent that takes the first remediation step. The alert becomes a programmable building block, not a terminus — which is what lets you automate the routine breaches and reserve human attention for the judgement calls.
Because creating and testing metric alerts is part of the API, a new service can ship with its thresholds defined in the same change that deploys it, and an agent can tune them over time. Your alerting lives in version control alongside the things it watches.
Get host, app, and SNMP metrics flowing, set thresholds that fire only when they mean it, route them to on-call, and let the analyst explain the breach — with an event for whatever you want to automate.