What is toil, and how do you reduce it?

Summary

Toil is operational work that is manual, repetitive, automatable, tactical, without enduring value, and that grows linearly with the service. Examples: hand-run deployments, manually restarting stuck jobs, applying the same config change to each server, answering the same access requests. Reduce it by measuring it and automating or eliminating the highest-volume items.

Longer Answer

Not all operational work is toil: incident response and thoughtful capacity planning are valuable. Google's SRE teams aim to keep toil below 50% of their time so the rest goes to engineering that removes future toil.

Approach: log toil for a few weeks (what, how often, how long), rank by total time and risk, then fix the cause (self-healing, better defaults, self-service tooling) rather than writing a script that still needs a human to run it. Sometimes the right fix is in the product (removing the feature that generates tickets).

Key Points

• Manual, repetitive, automatable, no lasting value, scales with load.
• Cap at ~50% of SRE time.
• Measure, rank, remove causes.
• Self-service and self-healing.