A production deployment failed halfway and the site is partially broken. What do you do?

Summary

Mitigate first: stop the rollout and roll back to the last known good version (or forward with a known fix if rollback is unsafe), communicate status, and confirm recovery with metrics. Then investigate why the pipeline let it through and why it failed halfway, and add safeguards.

Longer Answer

Considerations: was a database migration already applied (is the old code compatible)? Are there mixed versions running? Are caches or queues holding data written by the new version? Afterwards, run a blameless postmortem: what signals could have stopped the rollout earlier (canary analysis, health checks), what test would have caught it, and how to make rollback faster and safer.

Key Points

• Stop the rollout; roll back.
• Check migrations and mixed versions.
• Communicate and verify recovery.
• Postmortem; add automated safeguards.

Deep dive: Site Reliability Engineering interview questions