Kubernetes Isolate Crashed Pod
See also full blog post: Kubernetes: Isolate Crashed Pod
If you are running your pod (services / containers) on a Kubernetes cluster, and the pod runs into problems, you can decide to take it out of the load, but keep the pod running for further investigations.
To park or isolate a pod:
- Find your pod name.
- Find the pod selector labels in use.
- Update one of the selector labels.
- Debug.
- Delete the broken pod.
# find exact pod name:
kubectl --context=YOURCONTEXT get pods -n NAMESPACE \
| grep -e NAME -e PARTIAL-POD-NAME
# find labels:
kubectl --context=YOURCONTEXT get svc -n NAMESPACE SERVICE-NAME -o wide
# update label:
kubectl --context=YOURCONTEXT label -n NAMESPACE \
--overwrite pods FULL-POD-NAME somelabel=crashed-OLD-LABEL-VALUE
# do some debugging on the broken pod...
# and when all done, delete the broken pod:
kubectl --context=YOURCONTEXT delete -n NAMESPACE pod FULL-POD-NAME
Code language: Bash (bash)
In summary; by changing one of the labels on your pod, the Kubernetes service will no longer route traffic to that pod, and it will start a fresh instance to take it’s place to help the end users.
For more info, see the full blog post as mentioned at the top of this page.
The one thing to remember:
kubectl help label
Code language: Bash (bash)