Zero-downtime Redis password rotation

TL;DR Link to heading

A Redis ACL user can hold two passwords at once, so a rotation needs no moment where anyone is locked out. Add the new password to every Redis and Sentinel process, publish it, restart the clients, then restart the Redis pods one by one. The restarts remove the old password; you can’t do it at runtime on the Bitnami chart, because the health probes still log in with it. Before restarting, move sentinel-pass to the new password as well, or the Sentinels lock each other out.

Two passwords at once Link to heading

ACL SETUSER default >new-password

> adds a password and leaves the existing one alone. From then on AUTH accepts either, so the rotation becomes: servers accept both, clients switch, servers drop the old one. Nothing is rejected at any step, as long as the servers accept the new password before any client can read it.

ACL changes don’t replicate, so run it on every node. redis-cli -x reads the last argument from stdin, which keeps the password out of your shell history and ps:

{ printf '>'; cat new-password-file; } \
  | kubectl exec -i redis-node-0 -c redis -- \
      sh -c 'REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli --no-auth-warning -x ACL SETUSER default'

Then try AUTH on each node three times: old password, new password, wrong password. The wrong one should return WRONGPASS and add one to acl_access_denied_auth in INFO stats. If it doesn’t, your check can’t fail. Once it does, that counter is your alarm for the rest of the rotation.

Sentinel is a server too Link to heading

The chart gives Sentinel the same requirepass as Redis, and anything that asks Sentinel for the master logs in to it. If haproxy sits in front of Redis, its health checks do exactly that every few seconds. Add the new password only on port 6379 and the first haproxy pod restarted onto it marks every Sentinel down and stops routing. Run the ACL SETUSER on port 26379 as well: six processes for a three-node cluster.

The probes stop you removing the old password Link to heading

ACL SETUSER default <old-password looks like the obvious last step. On the Bitnami chart it takes the cluster down. The liveness and readiness probes, and the metrics exporter, log in using the container’s environment variable, which still holds the old password until the container restarts. Remove it and every probe fails; with liveness every 5 seconds and a failure threshold of 5, each pod is killed about 25 seconds later.

A restart does the removal instead. On startup the chart builds requirepass, masterauth and sentinel.conf from the environment, which now holds the new password only. Roll the pods one at a time, replicas first, master last, and check each before the next.

Move the internal logins first Link to heading

Mid-roll, restarted pods accept only the new password, while the rest accept both but still send the old one when they talk to each other. So before the first restart, switch every internal login:

The third one is easy to miss. Without sentinel-pass, a Sentinel logs in to its peers with its own requirepass, and the chart never sets sentinel-pass. Restart one pod without it and the other two Sentinels mark the new one down:

SENTINEL SENTINELS mymaster
  ip=redis-node-2...  flags=s_down,sentinel  last-ok-ping-reply=289193

Clients carry on, since two Sentinels still make quorum. Restart the master in that state, though, and a split Sentinel group has to run the failover. Setting sentinel-pass on the other two clears it within a ping interval.

All three settings live only in memory and vanish on restart, which is fine: the restarted pod’s config already uses the new password.

Proving nobody is on the old password Link to heading

Behind a proxy, CLIENT LIST shows every application connection coming from the proxy’s pod IPs, so it can’t tell you which app still holds the old password. Prove it from the other side instead, starting with a complete client list. Search for workloads that mount the secret and for ones that read the synced Kubernetes Secret as an environment variable: a migrations deployment that only did the latter never shows up in a search for volume mounts.

• every client pod started after the new secret synced (run the same check on the Redis pods as a control; they’re older, so it should fail). Run it over the whole list even if your restart loop said it succeeded. A restart loop that dies partway leaves the last few deployments untouched, still on the old password, and nothing complains until you remove it;
• every mounted secret file and synced Kubernetes Secret has the new password’s SHA-256 digest;
• every client connection on the master is younger than the proxy restart.

Restart every client rather than working out which ones reread the file. Most cache the password in a connection pool, and a uniform restart gives you a uniform proof. Cronjobs need nothing: each run is a fresh pod, which also makes them your earliest warning.

One trap while watching: ACL LOG merges repeated failures into one entry and keeps only the latest client’s details. Two Sentinel failures followed by your own wrong-password test show up as count=3 from 127.0.0.1. Trust the counter, not the address.

Redis pods restart under you Link to heading

The runtime changes live only in memory, and Redis pods restart on their own: spot preemptions, and cluster autoscaler scale-downs, which the chart invites by annotating the pods cluster-autoscaler.kubernetes.io/safe-to-evict: "true". A pod that restarts before you publish comes back knowing only the old password and has lost the new one. A pod that restarts between publishing and switching the internal logins comes back knowing only the new one, while the other nodes still use the old one to reach it.

So check that every process still holds two passwords immediately before publishing, and switch the internal logins straight after. Once both are done, an unplanned restart is harmless: the pod comes back on the new password, and everything already uses it.

Checklist Link to heading

1. Generate the password. Keep the character set if something templates it into config; length is free.
2. ACL SETUSER default >new on every Redis and Sentinel. Test old, new and wrong.
3. Confirm every process still holds two passwords, then publish to the secret store.
4. Straight away: masterauth, auth-pass and sentinel-pass to the new password on every node. Check replication and that no Sentinel is s_down. Wait for the synced Secret to show the new digest.
5. Restart the proxy, then every client, found by mounts and by env references. Check every deployment’s pod start times, not the loop’s output, plus digests and connection ages.
6. Delete Redis pods one at a time, master last. After each: old rejected, new accepted, links up, Sentinels whole. To roll back, ACL SETUSER default >old on the restarted pods restores access instantly.
7. Watch the counters and client logs, then disable the old secret version. Your rollback reads the old password from that version, so re-enable it before rolling back.

Clients see graceful restarts and one failover, and not a single WRONGPASS.