How to Use kubectl debug in Kubernetes?

One of the challenges in Kubernetes is debugging a process in Pod.

Photo by Sigmund on Unsplash

I will deploy two pods (web1-pod, web2-pod), and one service (web1-service):

00-multiple-pods.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: web1-pod
  labels:
    app: web1-pod
spec:
  containers:
  - image: webratio/nodejs-http-server
    name: web1-container
---
apiVersion: v1
kind: Service
metadata:
  name: web1-service
spec:
  ports:
  - port: 1111
    targetPort: 8080
  selector:
    app: web1-pod
---
apiVersion: v1
kind: Pod
metadata:
  name: web2-pod
  labels:
    app: web2-pod
spec:
  containers:
  - image: webratio/nodejs-http-server
    name: web2-container

Apply:

Checking connectivity between web2-pod and web1-pod :

I will attach an ephemeral/debug container to debug the network activity on the web1-container on the web1-pod :

kubectl debug -it web1-pod --target=web1-container --image=ubuntu

I’ve attached an ephemeral/debug container to web1-pod . It can be seen in the output of kubectl describe pods/web1-pod

I installed net-tools and tcpdump packages in the ephemeral/debug container:

apt update && apt install net-tools tcpdump

Check for listening ports and pay attention to the PID:

Let’s enter the web1-container in the web1-pod and check the PID:

It is verified that the container we created for debugging purposes was attached to web1-container .

Note: We see the web1-container’s main process’ PID from the ephemeral/debug container thanks to the — target parameter.

I will send a few requests from web1-pod to web2-pod and try to observe network activities via tcpdump in the debug container.

Before this, I want to show that there is no tcpdumpon the web1-container :

So we don’t need tcpdumpon the web1-container to use tcpdump on the ephemeral container.

I will send a request from web2-pod to web1-pod and try to observe the request via tcpdump in the ephemeral/debug container:

Since the ephemeral/debug container and web1-container use the same network namespace, we can access the network activity of web1-container from the ephemeral/debug container (More details: What is the Purpose of the Pause Container in Kubernetes?).

Observe the process via strace in the ephemeral/debug container:

Why is the — target Parameter Important?

The ephemeral/debug container can use the same PIDnamespace as the web1-containerthanks to the targetparameter. Without the targettarget parameter, we cannot access the PID of the processes of the web1-container:

How do you attach the ephemeral/debug container and use it later?

When you use — attach=false , the ephemeral/debug container will be attached to the web1-pod . However, your terminal will not be attached web1-podimmediately.

You may connect the ephemeral container container later. However, when you exit the container, it will be terminated immediately:

More details: What is the Difference Between kubectl attach and kubectl exec?