The Tech Lead Handbook · Leading Across the Team and Beyond
Agreeing Reliability Targets with Product
Use SLOs and error budgets to turn 'reliability vs features' from an argument into a shared rule.
Scenario
Example team: after two outages, the payments tech lead asks for a month of reliability work. Product refuses: a partner launch is due. Each side says the other doesn't understand the risk. The next outage happens during the launch.
What you see on the team
- Reliability work negotiated case by case, usually after incidents.
- Alerts on everything; on-call fatigue.
- "Five nines" requested casually, with no idea what it costs.
- Product and engineering with different definitions of "reliable enough".
The common mistake
Arguing about reliability in opinions and anecdotes rather than agreeing a measurable target and a rule for what happens when it's missed.
Why it happens
Without a shared target, every reliability request competes directly with features, and the loudest or most recent incident wins. Google's SRE book on service level objectives and the error budget policy example describe the alternative: agree what level of reliability users need, measure it, and let the remaining "budget" of allowed failure decide how much risk to take.
The playbook
1. Pick user-centred indicators (SLIs): e.g. share of payment requests that succeed; share completed in under 800 ms.
2. Agree targets (SLOs) with product: e.g. 99.9% of payment requests succeed over 28 days. That allows 0.1% failures — the error budget.
3. Write an error budget policy together, before the next incident:
Error budget policy - payments API (28-day window)
- Budget remaining > 50%: normal feature work; risky changes allowed with flags.
- Budget 0-50%: extra review for risky changes; top reliability item each sprint.
- Budget exhausted: feature releases pause (except fixes) until back within SLO;
postmortem actions take priority.
Signed: tech lead, product manager, engineering manager. Review quarterly.
4. Alert on SLO burn, not on every symptom.
5. Review monthly with product: budget spent, top causes, whether the target is still right.
What to say
Proposing an SLO:
"Instead of arguing case by case, can we agree what 'reliable enough' means for payments? I'd suggest 99.9% of payment requests succeeding over 28 days — that's about 40 minutes of full outage a month. If we burn that budget, we pause releases until we're back. If we don't, we can move faster."
Trade-offs
- Higher targets cost more: each extra nine roughly multiplies effort; choose what users actually need.
- Pausing releases is painful; the policy must be agreed in calm times or it won't hold.
- SLOs need good measurement; start with what you can measure reliably and improve it.
Signals it's working
- Reliability vs feature decisions reference the budget, not the latest outage.
- Fewer, more meaningful alerts.
- Product proposes reliability work when the budget is low.
- The SLO target is reviewed and sometimes changed with evidence.
Key points
- Define user-centred SLIs and agree SLO targets with product.
- The error budget (100% minus SLO) is how much failure you can spend on change and risk.
- Write the error budget policy before the next incident and get it signed.
- Alert on budget burn and review monthly with product.
Practice questions
Product refuses a month of reliability work before a launch. How do SLOs help?
Answer
An agreed SLO and error budget policy turn the decision into a rule: if the budget is exhausted, releases pause until reliability recovers; if not, feature work continues. The trade-off is agreed in advance instead of argued after each incident.
What is an error budget for a 99.9% monthly availability SLO?
Answer
0.1% of requests (or time) in the window may fail — roughly 43 minutes of full outage in a 30-day month. Spending it on risky changes is acceptable; exceeding it triggers the policy.
Why alert on SLO burn rate rather than on every error?
Answer
Burn-rate alerts fire when users are actually affected enough to threaten the target, reducing noise and on-call fatigue while still catching significant problems early.
Open in the app to mark lessons complete, take notes and track your progress.