Safe Deployments and Change Monitoring

Most incidents start with a change. Make changes observable and reversible: deploy markers on dashboards, canary or progressive rollouts compared against a baseline, automatic rollback on SLO regression, feature flags as kill switches, and change logs that include config and infrastructure changes — not just code.

# Argo Rollouts canary with analysis (excerpt)
strategy:
  canary:
    steps:
      - setWeight: 10
      - pause: { duration: 10m }
      - analysis: { templates: [{ templateName: error-rate-check }] }
      - setWeight: 50
      - pause: { duration: 10m }
---
# AnalysisTemplate: fail if canary 5xx ratio > 1%
metrics:
  - name: error-ratio
    failureCondition: result[0] > 0.01
    provider:
      prometheus:
        query: sum(rate(http_requests_total{rollout="checkout",status=~"5.."}[5m])) / sum(rate(http_requests_total{rollout="checkout"}[5m]))

Common pitfalls

• Config changes deployed without the same safety as code.
• Canary analysis on metrics too noisy to compare.
• No quick rollback path for database migrations.

Key points

• Every change is visible and reversible
• Canary + automated analysis
• Flags as kill switches

Further reading: Argo Rollouts