[TBD]Troubleshooting Kubernetes

Run POD with multiple container with restartPolicy Never

It will help to get the log and debug

RestartPolicy for a POD

kind: Pod apiVersion: v1 metadata: name: marks-dummy-pod spec: containers: - name: marks-dummy-pod image: ubuntu restartPolicy: Never

Useful link: https://markhneedham.com/blog/2017/10/21/kubernetes-simple-example-pod-running/

Debugging Tips

something isn’t working? figure out which controller is responsible and look at its logs

Today one of my pods wasn’t getting scheduled. I thought about it for a minute, thought “hmm, the scheduler is in charge of scheduling pods”, and went to look at the scheduler’s logs. It turned out that I’d reconfigured the scheduler wrong so it wasn’t starting anymore!

The more I use k8s, the easier it is to figure out which component might be responsible when I have a problem!

Linux logs

• /var/log/syslog

• /var/log/dmesg

• /var/log/kern.log

• /var/log/auth.log

• systemctl -a -> kube error

• journalctl -a -> kube error

Kubernetes logs

kubectl get event -n <namespace>

Master

• /var/log/kube-apiserver.log - API Server, responsible for serving the API

• /var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions

• /var/log/kube-controller-manager.log - Controller that manages replication controllers

Worker Nodes

• /var/log/kubelet.log - Kubelet, responsible for running containers on the node

• /var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing

POD level logs

Kubernetes stores logs to STDOUT/STDERR at /var/log/containers folder. Naming is <POD_NAME>_<NAMESPACE>_<CONTAINER_NAME>. In fact, Kubectl logs command collect logs from this location.

Example output of /var/log/containers/

root@ubuntu-231:/cic_code/temp# ls /var/log/containers/

bind-57446cd868-r6fj9_default_bind-088985de6ddb8060212dc3a4f3cd4f419355a7969cf8d05688c3bf7b9dc98149.log

bind-57446cd868-r6fj9_default_bind-c189406cb62bbd5c0ee27826b34249749e74b05a0de35f4b1541d9f06a853b52.log

cpx-ns-7b46d_default_cic-776f22d1fdb8dd7716362a6abdcb0d5e07ac92a866bab3510000b92f3ba3ae73.log

cpx-ns-7b46d_default_cpx-23d1ee9f3e9c81588396e9f6d21fa82a6930017ee24f291971d63105de8e050e.log

kube-flannel-ds-zzb4s_kube-system_install-cni-a1193c30b7d50332dbc6ab17a9798547f0ba93bdee0c4774025c7fc689a8c808.log

kube-flannel-ds-zzb4s_kube-system_kube-flannel-b2bca1fe70f45b6b5a0e833505c3891e4618545c3fd3ac67a57f4d635d55a345.log

kube-flannel-ds-zzb4s_kube-system_kube-flannel-ca8981070181f0bae4a316085ded212420a9cc70ffd4071712f64e6c356911ad.log

kube-proxy-kcsp7_kube-system_kube-proxy-b2fcbf89d9ead14154db12495d6a4a0e18b72209811bcd6b62bf7c0e33d2bb2c.log

kube-proxy-kcsp7_kube-system_kube-proxy-f4ee57b717ba9540cd7f03224967390137d2f37c2d073b70c470469377882443.log

logging-app-pod_default_logging-app-0444ac97116de14aa3d0cbdd7197cb6efaa78e7d0d05e36cbfd65ffa266ab30a.log

two-containers_default_first-99c727af4f987639915db15196adfd0e26d2fc102989c56fd0f8d7d48d72aa0c.log

two-containers_default_second-81bbd6ca6865a37e2f2c19422f160d43fa715ddda3aa0059c45c3a73b0117874.log

web-backend-87sp2_default_web-backend-a1b64c84d7e43f050d897a2745eb655ab4b03a6486504f1811078ffb2f2ce95b.log

web-backend-dmlr4_default_web-backend-326ecf49a613ad37d8973cf088124736b74d06cc18c45261e24a792650426529.log

web-frontend-web-frontend-l9v8t_default_web-frontend-web-frontend-ee00361bb5eceb7afc1050d020916c3744d2a4962399bc7b9f9d6e127160844a.log

web-frontend-web-frontend-lkldk_default_web-frontend-web-frontend-20f0bf0afe23b337106c8ad2f55adf4467ca4603ea484c096bf41f3d4c732917.log

web-frontend-web-frontend-qqjtl_default_web-frontend-web-frontend-0f08097c0de78e5ce761d76657066e6acb6617208af1a87d8246ffc75e51c82b.log

Useful link: https://timber.io/blog/collecting-application-logs-on-kubernetes/

https://stackoverflow.com/questions/47915287/where-are-kubernetes-pods-logfiles

Reference

https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/#looking-at-logs

https://jvns.ca/blog/2017/06/04/learning-about-kubernetes/