Separating real failures from rejected calls, setting error budgets, and treating "found by the weekly report, not an alarm" as a finding in itself.
A service can look healthy while quietly failing — if your dashboards count rejected calls as errors, or count nothing at all.
Three practices that made reporting honest
- Classify outcomes. A 4xx rejection, a timeout and a 5xx mean different things. Only genuine failures count against health; the rest are separate metrics with their own thresholds.
- Error budgets, even proposed ones. An SLO target makes the monthly failure allowance computable, and burn pace tells you whether you'll run out before month end. Publish targets marked proposed until owners agree.
- "Found by the report" is a finding. If the weekly review surfaces something no alarm caught, the gap is in alerting — log it and fix the monitor, not just the bug.
The lesson: health reporting is a product for your own team. If the numbers mix signal with noise, people stop trusting them — and the first casualty of distrusted metrics is the incident you don't see.