Site Reliability Engineering Interview Questions

Summary

SRE is an engineering discipline, originated at Google, that applies software engineering to operations: reliability targets are defined with SLOs, risk is managed with error budgets, toil is automated away, and incidents are learned from. DevOps is a broader culture of shared ownership between development and operations; SRE is one concrete way to implement it.

Longer Answer

Google describes SRE as "what happens when you ask a software engineer to design an operations team". Core practices: SLIs and SLOs agreed with product owners, error budgets that balance reliability and release speed, a cap on operational toil (Google targets under 50% of an SRE's time), blameless postmortems, and engineering work that removes classes of failure.

The SRE workbook puts it as "class SRE implements interface DevOps": both value shared responsibility, automation, measurement and small changes; SRE adds prescriptive mechanisms. Job titles vary a lot between companies, from platform engineering to on-call operations, so ask what the role means locally.

Key Points

• Software engineering applied to operations.
• SLOs, error budgets, toil limits, postmortems.
• SRE is one implementation of DevOps.
• Titles vary between companies.

Summary

An SLI (service level indicator) is a measurement of service behaviour, such as the proportion of successful requests. An SLO (objective) is a target for an SLI over a window, such as 99.9% of requests successful over 28 days. An SLA (agreement) is a contract with customers that specifies consequences, usually credits, if a level is missed.

Longer Answer

SLIs are usually ratios of good events to valid events: availability (non-5xx responses / all responses), latency (requests faster than 300 ms / all requests), freshness, correctness, throughput.

SLOs are internal targets that drive engineering decisions; SLAs are business commitments and should be looser than SLOs so you get warning before breaching a contract. Many internal services have SLOs but no SLA. Measure SLIs as close to the user as practical (load balancer or client), because server-side metrics miss failures that never reach the server.

Key Points

• SLI = measurement, SLO = target, SLA = contract.
• SLIs as good/valid ratios.
• SLA looser than SLO.
• Measure close to the user.

Summary

Start from what users care about for each critical user journey: for request-driven services, availability and latency; for pipelines, freshness, coverage and correctness; for storage, durability and throughput. Express each as a ratio of good events to valid events, measured where it reflects user experience.

Longer Answer

Good SLIs are few (two to four per journey), meaningful to users, and directly actionable. Pitfalls: averaging latency (hides the tail; use thresholds such as "under 400 ms" or percentiles), counting health-check traffic, measuring only at the server (misses DNS, load balancer and network failures), and SLIs on internal components that users never notice.

Example for a checkout API: proportion of POST /checkout requests that return non-5xx in under 1 second, measured at the load balancer, excluding requests rejected for invalid input (4xx).

Key Points

• Based on user journeys.
• Good events / valid events.
• Latency via thresholds, not averages.
• Measure near the user.

Summary

The error budget is the amount of unreliability an SLO allows: with a 99.9% availability SLO over 30 days, 0.1% of requests (or about 43 minutes of full downtime) may fail. While budget remains, teams can ship and take risks; when it is spent, the agreed policy shifts effort to reliability.

Longer Answer

The budget turns the reliability-versus-velocity argument into data: both product and engineering agree on the SLO, so spending the budget on launches is legitimate, and exhausting it triggers a pre-agreed response rather than a negotiation.

An error budget policy typically says: if the budget is exhausted over the window, freeze non-critical releases (except fixes), prioritise postmortem action items, and require extra review for risky changes until the service is back within SLO. It also says what happens when a single incident consumes a large share (for example 20%): mandatory postmortem. A budget that is never spent can mean the SLO is too loose or the team is too cautious.

Key Points

• Budget = 1 - SLO.
• Spend it on velocity; protect it with a policy.
• Policy agreed in advance.
• Unspent budget is a signal too.

Summary

Burn rate is how fast you are consuming the error budget relative to the SLO: a burn rate of 1 uses exactly the budget over the window; 14.4 exhausts a 30-day budget in about 2 days. Alerting on burn rate pages only when the SLO is genuinely threatened, with urgency proportional to the speed of the burn.

Longer Answer

The SRE workbook recommends multiwindow, multi-burn-rate alerts. For example, page if 2% of a 30-day budget is consumed in 1 hour (burn rate 14.4) confirmed by a 5-minute window, and if 5% is consumed in 6 hours (burn rate 6); open a ticket for slower burns (10% in 3 days). The short window makes alerts reset quickly after recovery; the long window prevents paging on brief blips.

Compared with "error rate > 1% for 5 minutes", burn-rate alerts reduce false pages for low-impact blips and still catch slow leaks before the budget is gone.

Key Points

• Burn rate = speed of budget consumption.
• Page on fast burns, ticket on slow burns.
• Multiwindow: long to detect, short to reset.
• Fewer false pages, better coverage.

Summary

Latency, traffic, errors and saturation. If you can only measure four things about a user-facing system, measure these.

Longer Answer

• Latency: time to serve requests; track successful and failed requests separately (fast errors can make latency look good) and use percentiles.
• Traffic: demand, such as requests per second or concurrent sessions.
• Errors: rate of failed requests, explicit (5xx) and implicit (wrong content, too slow per policy).
• Saturation: how full the most constrained resource is (CPU, memory, I/O, queue depth, connection pools), ideally predictive ("disk full in 4 hours").

They complement SLIs: SLOs say whether users are happy; golden signals help explain why not.

Key Points

• Latency, traffic, errors, saturation.
• Separate latency of errors.
• Saturation of the constrained resource.
• Pair with SLOs.

Deep dive: Observability and Operations

Summary

A good page is urgent, actionable, and indicates real or imminent user impact. Alert on symptoms (users seeing errors or slowness, SLO burn) rather than causes (high CPU on one host), and route non-urgent problems to tickets. Alert fatigue, from too many noisy pages, makes people ignore the alert that matters.

Longer Answer

Every page should answer: is this hurting users or about to? Can the on-call engineer do something now? Is there a runbook? Cause-based alerts are still useful as dashboards or low-urgency tickets, and for imminent saturation (disk full in two hours).

Hygiene: review pages weekly, delete or downgrade alerts that were not actionable, deduplicate and group related alerts, and track pages per shift. Google's guidance is a small number of incidents per on-call shift so each gets proper attention and follow-up.

Key Points

• Pages must be urgent and actionable.
• Symptoms over causes.
• Tickets for non-urgent issues.
• Regularly prune noisy alerts.

Summary

Monitoring checks known failure modes with predefined metrics and alerts; observability is the ability to understand new, unknown problems by asking arbitrary questions of the system's telemetry. Metrics are cheap numeric time series for trends and alerts; logs are detailed event records; traces follow a single request across services.

Longer Answer

Metrics answer "is something wrong, and how much?"; traces answer "where in the request path is the time or error?"; logs answer "what exactly happened in this component?". Correlating them (a trace ID in every log line, exemplars linking metrics to traces) is what makes debugging fast.

High-cardinality, structured events (wide events with many attributes such as user ID, region, version) let you slice by any dimension, which is the practical heart of observability. OpenTelemetry provides a vendor-neutral standard for generating all three signals.

Key Points

• Monitoring: known questions; observability: new questions.
• Metrics, logs, traces have different strengths.
• Correlate with trace IDs.
• OpenTelemetry as the standard.

Deep dive: Observability and Operations

Summary

Use a lightweight incident command structure: an incident commander who coordinates and makes decisions, an operations lead who works on mitigation, a communications lead who updates stakeholders and status pages, and a scribe who keeps the timeline. In small incidents one person may hold several roles, but the commander role is always explicit.

Longer Answer

Principles from incident command systems:

• Declare early: a declared incident that turns out minor is cheap; an undeclared one that grows is expensive.
• Mitigate first, diagnose later: roll back, fail over, shed load or disable a feature to stop user impact; root cause comes afterwards.
• One coordination channel, a written live state document, regular updates at a stated cadence, and explicit hand-offs when people change.
• The commander does not debug; they keep the big picture, assign work and decide.

Afterwards: a blameless postmortem with tracked action items.

Key Points

• Commander, ops, comms, scribe.
• Declare early.
• Mitigate before root-causing.
• Single channel, regular updates, explicit hand-offs.

Deep dive: The Tech Lead Handbook

Summary

A written review after an incident that explains what happened, the impact, how it was detected and mitigated, the contributing factors, and the actions to prevent recurrence, while assuming everyone acted reasonably with the information they had. Blamelessness makes people share the truth, which is the only way to fix the system.

Longer Answer

An effective postmortem has: a timeline built from logs and chat, impact in user terms (and SLO budget consumed), contributing factors rather than a single 'root cause' (systems fail through combinations), what went well and where we got lucky, and a small number of specific, owned, prioritised action items with due dates.

"Human error" is never the conclusion: ask why the system allowed the error (missing guardrails, confusing tooling, unclear runbooks). Share postmortems widely, review them in a regular forum, and track action-item completion; unfinished items are how the same incident happens twice.

Key Points

• Blameless: assume good intent; fix the system.
• Timeline, impact, contributing factors, actions.
• No 'human error' conclusions.
• Track action items to completion.

Deep dive: The Tech Lead Handbook

Summary

Load shedding deliberately rejects some work when a service is overloaded so that the rest succeeds, instead of everything slowing down and failing. Graceful degradation keeps the core experience working by turning off expensive or optional features under stress.

Longer Answer

Techniques: limit concurrent requests per instance and return 429/503 quickly when full; prioritise traffic (checkout over recommendations, user requests over batch jobs); drop requests that have already exceeded their deadline; serve cached or simplified responses; and turn off non-essential features with flags.

The key insight: an overloaded server that accepts everything ends up serving nothing within its deadline. Rejecting early is cheaper than timing out late. Clients must back off when they receive these responses, or shedding just generates retries.

Key Points

• Reject early to protect the rest.
• Prioritise critical traffic.
• Degrade optional features.
• Clients must back off.

Deep dive: Resilience Patterns

Summary

A failure that grows by positive feedback: one component fails or slows, its load shifts to others or callers retry, those become overloaded and fail too, until the whole system is down. Typical triggers are overload, retry storms, resource exhaustion and slow dependencies tying up threads.

Longer Answer

Prevention:

• Limit load: load shedding, rate limits, bounded queues.
• Retry discipline: capped exponential backoff with jitter, retry budgets, no retries at every layer.
• Timeouts and deadlines propagated through the call chain so work for abandoned requests stops.
• Bulkheads: separate pools per dependency so one slow dependency cannot consume all threads.
• Circuit breakers to stop calling a failing dependency.
• Capacity headroom so losing one instance or zone does not overload the rest.

Recovery may require stopping traffic entirely, then ramping back up gradually, because a cold system cannot absorb the full load at once.

Key Points

• Positive-feedback failure spread.
• Overload, retries, resource exhaustion.
• Shedding, backoff, deadlines, bulkheads, breakers.
• Recover with gradual ramp-up.

Deep dive: Resilience Patterns

Summary

Retry only errors that are transient and requests that are safe to repeat (idempotent, or protected with an idempotency key); use capped exponential backoff with jitter; limit attempts; and enforce a retry budget so retries cannot multiply load during an outage. Retry at one layer, not at every layer.

Longer Answer

If each of three layers retries three times, one user request can become 27 backend requests during an incident. Jitter spreads retries so clients do not stampede in sync. A retry budget (for example, retries may be at most 10% of requests) caps amplification. Respect Retry-After and 429 responses, and do not retry errors that will not change (400, 403, validation).

Key Points

• Transient errors and idempotent requests only.
• Exponential backoff + jitter, capped.
• Retry budgets limit amplification.
• One layer retries.

Deep dive: Resilience Patterns

Summary

A canary sends a small share of traffic (or a few hosts) to the new version first, compares its health metrics with the baseline, and only continues the rollout if they match. Progressive delivery automates the stages (1%, 10%, 50%, 100%) with automatic rollback when metrics degrade.

Longer Answer

Most outages are caused by changes, so limiting the blast radius of each change is one of the most effective reliability practices. For a meaningful canary: compare the canary against a control group running the old version at the same time (not against last week), watch SLIs (errors, latency) and business metrics, run long enough to see real traffic patterns, and make rollback automatic and fast. Tools include Argo Rollouts, Flagger and cloud deployment services.

Database migrations and configuration changes need the same staged approach.

Key Points

• Small traffic share first.
• Compare against a concurrent control.
• Automated stages and rollback.
• Applies to config and schema changes too.

Summary

Recovery time objective is how long a service may be down after a disaster; recovery point objective is how much data (measured in time) may be lost. They decide the strategy and its cost: backup and restore (hours, cheapest), pilot light, warm standby, and multi-site active-active (near zero, most expensive).

Longer Answer

RPO drives replication and backup frequency (continuous replication for seconds, nightly backups for a day). RTO drives how much infrastructure is pre-provisioned and automated. Both must be agreed with the business per service, tested in real recovery drills, and include dependencies (a service with a 15-minute RTO that depends on a database with a 4-hour restore really has a 4-hour RTO). Protect backups from the same failure: separate accounts, immutability, and protection against ransomware or accidental deletion.

Key Points

• RTO = downtime allowed; RPO = data loss allowed.
• Strategies from backup/restore to active-active.
• Include dependencies.
• Test restores; isolate backups.

Summary

A readiness probe decides whether a pod receives traffic; a failing pod is removed from the Service endpoints. A liveness probe decides whether the container should be restarted. A startup probe protects slow-starting containers by disabling the other probes until startup succeeds.

Longer Answer

The classic mistake: a liveness probe that checks dependencies (database reachable). When the database blips, every pod fails liveness and restarts at the same time, turning a dependency problem into a full outage and a restart storm. Liveness should check only that the process itself is not stuck; readiness can reflect whether the instance can serve (warm-up done, not overloaded). Use generous thresholds and timeouts, and set up graceful shutdown (stop accepting, drain, then exit on SIGTERM) along with preStop hooks where load balancers are slow to deregister.

Key Points

• Readiness: receive traffic or not.
• Liveness: restart or not.
• Startup: protect slow starts.
• Never check dependencies in liveness.

Summary

First assess impact and mitigate (is the SLO burning? was there a recent change? roll back if likely). Then narrow down: which endpoints, regions, instances and customers are affected; when exactly it started; what changed then; and where the time goes (traces, dependency latency, saturation of CPU, memory, connection pools, database).

Longer Answer

A structured approach:

1. Scope: all traffic or a subset (one endpoint, one zone, one tenant, one version)?
2. Timeline: correlate the start with deploys, config and flag changes, traffic shifts, dependency incidents, cron jobs.
3. Breakdown: traces show whether time is in our code, the database or a downstream call.
4. Resources: USE method on the constrained resources; garbage collection pauses; thread or connection pool exhaustion (queueing).
5. Hypothesis and test: one change at a time, compare against a healthy baseline.

Communicate status throughout, and capture findings for the postmortem.

Key Points

• Impact and mitigation first.
• Scope, timeline, what changed.
• Traces and dependency breakdown.
• Saturation and queueing checks.

Summary

First confirm the business needs it, because it cuts allowed downtime from about 43 minutes to about 4 minutes a month. Then analyse where the current budget goes, and attack the largest causes: remove single points of failure (multi-zone, redundancy), make changes safer (canaries, automated rollback), speed up detection and mitigation (burn-rate alerts, automated failover), and degrade gracefully when dependencies fail.

Longer Answer

At four nines, humans are too slow to be the recovery mechanism: an engineer being paged, waking up and logging in already uses the monthly budget. So recovery must be automatic (health-check-based failover, automated rollback, load shedding), and changes must be staged so a bad one only affects a small slice. Check critical dependencies: you cannot exceed their availability without redundancy or fallbacks. Measure progress against the SLO, and price the cost (extra regions, engineering time) explicitly.

Key Points

• Confirm the need; 4 minutes/month.
• Spend analysis: fix the biggest budget consumers.
• Automated recovery; humans are too slow.
• Dependencies cap you.

Summary

Per 30 days: 99% allows about 7.2 hours, 99.9% about 43 minutes, 99.95% about 22 minutes, 99.99% about 4.3 minutes, and 99.999% about 26 seconds. Pick the target from what users need and what the business will pay for, not 'as high as possible'.

Longer Answer

Every extra nine costs much more: redundancy across zones or regions, faster detection and automated recovery, stricter change processes. It is also capped by dependencies: a service cannot be meaningfully more available than the critical services it calls (unless it degrades gracefully), and users' own networks and devices fail far more often than 99.99%.

Practical approach: measure current performance, set an SLO slightly below it, check that missing it would actually upset users, and tighten over time if needed. 100% is the wrong target for nearly everything.

Key Points

• ~43 min/month at 99.9%; ~4 min at 99.99%.
• Each nine costs much more.
• Bounded by dependencies.
• Start from current performance and user needs.

Summary

Toil is operational work that is manual, repetitive, automatable, tactical, without enduring value, and that grows linearly with the service. Examples: hand-run deployments, manually restarting stuck jobs, applying the same config change to each server, answering the same access requests. Reduce it by measuring it and automating or eliminating the highest-volume items.

Longer Answer

Not all operational work is toil: incident response and thoughtful capacity planning are valuable. Google's SRE teams aim to keep toil below 50% of their time so the rest goes to engineering that removes future toil.

Approach: log toil for a few weeks (what, how often, how long), rank by total time and risk, then fix the cause (self-healing, better defaults, self-service tooling) rather than writing a script that still needs a human to run it. Sometimes the right fix is in the product (removing the feature that generates tickets).

Key Points

• Manual, repetitive, automatable, no lasting value, scales with load.
• Cap at ~50% of SRE time.
• Measure, rank, remove causes.
• Self-service and self-healing.

Summary

Averages hide the tail: a mean of 120 ms can coexist with 5% of users waiting over 2 seconds. Percentiles (p50, p95, p99) describe what users actually experience, and the tail matters even more when one page load fans out to many backend calls. Percentiles cannot be averaged across hosts; aggregate histograms and compute percentiles from them.

Longer Answer

If a page makes 20 backend calls, the chance that at least one hits that backend's p99 is about 18%, so backend p99 becomes typical user experience. Prometheus histograms (or native/exponential histograms) store bucket counts that can be summed across instances, after which histogram_quantile estimates percentiles. Averaging per-host p99 values gives a meaningless number.

For SLOs, threshold-based SLIs ("proportion of requests under 300 ms") are often clearer than percentile targets and are easy to aggregate.

Key Points

• Averages hide the tail.
• Fan-out makes tails common.
• Aggregate histograms, never percentiles.
• Threshold SLIs aggregate cleanly.

Summary

Forecast demand (organic growth, launches, seasonal peaks), measure how much load each instance or resource can handle through load testing, and provision for peak plus a safety margin and the loss of a failure domain (N+1 or N+2, zone loss). Revisit regularly and automate scaling where possible.

Longer Answer

Steps: know the service's bottleneck resource and its saturation point (load tests, production measurements); model demand per business driver (requests per active user); plan for redundancy (survive losing a zone at peak); include lead times for things that cannot autoscale (database instances, quotas, reserved capacity, hardware). Watch cost: overprovisioning is also a failure. Track forecast accuracy and adjust.

Key Points

• Forecast demand including peaks and launches.
• Measure per-unit capacity with load tests.
• Plan N+1 / zone loss.
• Account for lead times and quotas.

Summary

Chaos engineering runs controlled experiments that inject failures (instance loss, latency, dependency errors, zone outages) to verify that the system behaves as expected, uncovering weaknesses before real incidents do.

Longer Answer

Method from the Principles of Chaos Engineering: define steady state with metrics (an SLI), form a hypothesis ("losing one cache node does not change the error rate"), inject a realistic failure, minimise blast radius, and compare. Start in staging or with a tiny production scope during working hours, with a stop button and on-call aware; expand as confidence grows. Tools: AWS Fault Injection Service, Chaos Mesh, LitmusChaos, Gremlin. Game days apply the same idea to people and processes.

Key Points

• Hypothesis-driven failure experiments.
• Measure steady state via SLIs.
• Small blast radius and a stop button.
• Start small; expand with confidence.

Summary

Load average counts runnable tasks plus tasks in uninterruptible sleep (usually waiting on disk or network I/O), averaged over 1, 5 and 15 minutes. Compare it with the CPU count, then check whether CPU is busy (top, mpstat), tasks are blocked on I/O (vmstat 'b' column, iostat -x), memory is short (swapping, free, vmstat si/so), or something else saturates.

Longer Answer

A quick sequence (similar to Brendan Gregg's "Linux performance analysis in 60 seconds"): uptime, dmesg | tail (OOM kills, errors), vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m, sar -n DEV 1, sar -n TCP,ETCP 1, top. High load with idle CPU usually means I/O wait or many blocked threads; high load with saturated CPU means not enough CPU (or a runaway process). In containers, also check CPU throttling from cgroup limits.

Key Points

• Load = runnable + uninterruptible tasks.
• Compare with CPU count.
• CPU vs I/O vs memory vs throttling.
• vmstat, mpstat, iostat, pidstat, dmesg.

Summary

Follow the error budget policy that product and engineering agreed in advance: normally, feature launches pause and the team works on reliability until the service is back within SLO. Exceptions should go through the escalation path the policy defines, with the risk made explicit, rather than being negotiated ad hoc.

Longer Answer

Make it constructive: show the data (what consumed the budget), propose the specific reliability work and its timeline, and look for lower-risk options (launch behind a flag to a small group, after the fix lands). If the SLO is routinely missed and nobody acts, revisit whether the SLO reflects what users and the business need, because an ignored SLO is worse than none.

Key Points

• Follow the pre-agreed policy.
• Exceptions via defined escalation.
• Show data; propose a plan.
• Revisit the SLO if constantly ignored.