Leader Election — Distributed Systems for Backend Engineers

Some work must run on exactly one instance: schedulers, partition owners, compaction. Use leases from a consensus store (Kubernetes Lease objects via client-go/controller-runtime leader election, etcd, ZooKeeper, Consul) or database locks. The leader renews its lease; followers take over when it expires. Combine with fencing for correctness, and design leader work to be idempotent since failovers may repeat it.

Leader election options

Mechanism	Failover time	Dependencies
Kubernetes Lease	~LeaseDuration (seconds)	Kubernetes API
etcd / ZooKeeper	Session/lease TTL	Consensus cluster
DB advisory lock	On session loss	Database
Static config (one replica)	Manual	None (no HA)

lock := &resourcelock.LeaseLock{
    LeaseMeta: metav1.ObjectMeta{Name: "billing-scheduler", Namespace: "prod"},
    Client:    clientset.CoordinationV1(),
    LockConfig: resourcelock.ResourceLockConfig{Identity: podName},
}
leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
    Lock: lock, LeaseDuration: 15 * time.Second, RenewDeadline: 10 * time.Second, RetryPeriod: 2 * time.Second,
    ReleaseOnCancel: true,
    Callbacks: leaderelection.LeaderCallbacks{
        OnStartedLeading: func(ctx context.Context) { runScheduler(ctx) },   // stop when ctx is cancelled
        OnStoppedLeading: func() { log.Println("lost leadership") },
    },
})

-- Only the session holding the lock runs the scheduler; released automatically if the session dies
SELECT pg_try_advisory_lock(hashtext('billing-scheduler'));   -- true → we are leader

Common pitfalls

Leader work continuing after losing leadership (check context/lease).
Non-idempotent leader tasks.
Very short leases causing flapping.

Key points

Leases from a consistent store
Stop work when leadership is lost
Idempotent leader tasks

Leader election options

Common pitfalls

Key points

Practice questions