Kubernetes: Restarting Pods

When making changes to config maps or other things - it’s tempting to restart all the pods for a particular service (select them all, delete, and let them come back up). This may work in the majority of cases, but if something goes wrong, it’ll be hard to tell if the recent change caused it or if it’s unrelated.

Recommendation:

• First check the logs in the pods to restart for any abnormal behavior
• Second restart a single pod before making any changes, make sure it comes up properly
• If the pod cannot restart without making any changes, debug the issue
• If the pod can restart, make the config change.
• Now restart (or kill) pods one at a time, ensuring each one enforces the new change.

The point here isn’t the specific process, but rather having checkpoints when changes are made. Ideally, by making only one change at a time and testing, it’s evident when something is broken which change caused it.