The real test of monitoring isn't dashboards
Any team can build dashboards. The real test of a monitoring system is what happens at 3 AM when a pager goes off: does the engineer know exactly what to do, or do they spend the first 20 minutes wading through irrelevant graphs trying to find the actual problem? If it's the second, your monitoring is theatre.
This guide is a structured walk through monitoring that doesn't cry wolf, with the four golden signals at the foundation, SLI/SLO/error budgets in plain language, alert hygiene rules, and a checklist for auditing the monitoring you already have.
Outline
- The four golden signals
- SLI, SLO and error budgets without the jargon
- Alert hygiene — the rules that protect humans
- Logs vs metrics vs traces: when to use which
- Synthetic and real-user monitoring
- Picking a tool stack
- A sane on-call setup for small teams
- Audit checklist for existing monitoring
1. The four golden signals
For every user-facing service, measure these four and nothing else first:
- Latency — time to serve a successful request. Track p50, p95, p99.
- Traffic — requests per second, or business equivalent (orders/min).
- Errors — failed requests as a rate, broken down by category.
- Saturation — how full the system is (CPU, memory, queue depth, connection pools).
Get these four right and you've covered 80% of incidents. Add infrastructure-level metrics (disk, network, host health) only when they actually predict business impact.
2. SLI, SLO and error budgets in plain language
- SLI is what you measure: "% of requests served under 300 ms".
- SLO is the target you commit to: "99.5% of requests under 300 ms over 30 days".
- Error budget is what you're allowed to spend: 0.5% in this example.
The point of error budgets isn't punishment. It's the explicit trade-off between reliability and feature velocity. If you've burned this month's budget, you slow down releases and invest in stability. If you haven't, you can ship faster. This conversation must happen at the engineering-leadership level, not just inside the SRE team.
3. Alert hygiene — the rules that protect humans
Five rules, all non-negotiable:
- Every alert has a runbook. No runbook, no alert.
- Alerts are actionable. If the on-call can't do something within five minutes, it's a ticket, not an alert.
- Alerts are based on symptoms, not causes. "API latency p99 above 800 ms" beats "CPU on web-3 above 80%".
- Alerts respect business hours. Page only for things that hurt revenue or users right now. Everything else goes to a daytime queue.
- Alert volume is reviewed weekly. Any alert that fires without leading to action gets tuned or deleted.
A team that gets paged twice a week with real, actionable incidents is healthier than one that gets paged 30 times with noise.
4. Logs vs metrics vs traces: when to use which
| Signal | Best for | Worst for |
|---|---|---|
| Metrics | Alerting, SLOs, dashboards | Why a specific request failed |
| Logs | Forensics, audit, debug | High-cardinality dashboards (cost explodes) |
| Traces | Latency root cause across services | Cost-efficient long-term storage |
Don't log what should be a metric. Don't trace what should be a log. The three are complementary, not interchangeable.
5. Synthetic and real-user monitoring
Synthetic monitoring (scripted checks from external locations every minute) catches outages and certificate expirations before users notice. Real-user monitoring (RUM) tells you what users actually experience, including the slow networks and old browsers your staging environment doesn't have.
Run both. Alert from synthetic. Investigate from RUM.
6. Picking a tool stack
There is no single correct stack. The honest selection matrix:
- Prometheus + Grafana + Loki + Tempo — best total cost of ownership for teams with one engineer who likes operating this stuff. Worst for teams that don't.
- Zabbix — strong for infrastructure-heavy environments, on-premise estates, mixed Linux/Windows fleets. Quieter on application-level observability.
- Datadog / New Relic / Dynatrace — fastest time to value, highest sticker price, lock-in via custom data model. Often correct for a 20–100 person company that doesn't want to operate the platform.
- Hyperscaler-native (CloudWatch, Azure Monitor, GCP Cloud Monitoring) — convenient, weaker cross-cloud or cross-on-prem story.
Pick on three criteria: total cost over three years (including engineer time), data portability if you change your mind, and whether the on-call team actually wants to use it.
7. A sane on-call setup for small teams
- Minimum two on-call engineers. One is single-point-of-failure for humans.
- Primary rotates weekly. Secondary covers the primary. No 24/7 single-pager hero.
- Handover meeting every Monday: open incidents, top noisy alerts, what to fix this week.
- Compensation policy in writing.
- After every incident: a five-line postmortem in shared docs. Blameless, focused on systems and signals.
8. Audit checklist for existing monitoring
- Are the four golden signals in place for every user-facing service?
- Does every active alert have a written runbook?
- What's the weekly alert volume vs incidents that needed human action?
- When was the last successful synthetic monitor of your most critical flow?
- Are logs and metrics in the same place (or at least the same UI)?
- Does your tool of choice cost what it should in 12 months at projected growth?
- Is the on-call rotation documented, compensated and free of single-points-of-failure?
If you can't answer any of these in one sentence, that's where to start.
Key takeaways
- Build on the four golden signals, not on infrastructure trivia.
- Error budgets are a leadership conversation about velocity vs reliability.
- Every alert needs a runbook and an action; everything else is a ticket.
- Logs, metrics and traces complement each other — never substitute.
- Tool choice is a 3-year TCO decision, not a feature checkbox.
If you'd like an outside review of your current monitoring against this checklist, get in touch — you'll talk to an engineer, not a sales rep.