How to Mirror Traffic to Another Pod in Kubernetes?

adil
3 min readApr 13, 2024

--

Mirroring network traffic in production has a lot of benefits, such as debugging, troubleshooting, testing, monitoring, etc.

***** You can follow me on LinkedIn *****

Photo by Jovis Aloor on Unsplash

Some Kubernetes plugins allow you to mirror traffic from one pod to another. In this tutorial, we will be using Istio, a well-known Kubernetes traffic management component.

Istio provides a clear and concise installation document. After installing istioctl on your computer,

Install istio with its CNI component:

cat <<EOF > istio-cni.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
components:
cni:
enabled: true
EOF
istioctl install -f istio-cni.yaml -y

*** Note: Istio CNI is required to mirror the network ***

Enable istio Sidecar in your namespace (I’ll use the default namespace):

kubectl label namespace default istio-injection=enabled

Check if Istio is operating correctly:

istioctl analyze

Traffic Mirroring

Deploy two pods.

One will function like a production app, while the other will behave like a production app with debug mode turned on.

00-pod.yaml

---
apiVersion: v1
kind: Pod
metadata:
labels:
app: payment
type: production
name: payment-app
spec:
containers:
- image: ailhan/web-debug
name: payment-app
imagePullPolicy: Always
ports:
- containerPort: 80
env:
- name: TEXT
value: "Hello from the Payment App"
---
apiVersion: v1
kind: Pod
metadata:
labels:
app: payment
type: debug
name: payment-app-debug
spec:
containers:
- image: ailhan/web-debug
name: payment-app-debug
imagePullPolicy: Always
ports:
- containerPort: 80
env:
- name: TEXT
value: "Payment App - DEBUG Mode On"

Apply:

➜  ~ kubectl apply -f 00-pod.yaml
pod/payment-app created
pod/payment-app-debug created

01-service.yaml

apiVersion: v1
kind: Service
metadata:
name: payment-app-svc
spec:
ports:
- port: 80
targetPort: 80
name: http-payment-app
selector:
app: payment

*** Note: A name must be added to the port The naming convention is <protocol>[-<suffix>]***

Apply:

➜  ~ kubectl apply -f 01-service.yaml
service/payment-app-svc created

Let’s test it:

The requests are randomly distributed between two pods. It works.

For mirroring, I’ll use the payment-app-debug pod.

02-destination-rule.yaml

The destination rules define the rules that virtual services will follow.

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: payment-app-destination-rule
spec:
host: payment-app-svc
subsets:
- name: main
labels:
type: production
- name: mirror
labels:
type: debug

Apply:

➜  ~ kubectl apply -f 02-destination-rule.yaml
destinationrule.networking.istio.io/payment-app-destination-rule created

03-virtual-service.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: payment-app-svc-virtual-service
spec:
hosts:
- payment-app-svc
http:
- route:
- destination:
host: payment-app-svc
subset: main
weight: 100
mirror:
host: payment-app-svc
subset: mirror
mirrorPercentage:
value: 100

Apply:

➜  ~ kubectl apply -f 03-virtual-service.yaml
virtualservice.networking.istio.io/payment-app-svc-virtual-service created

Let’s test it:

Istio’s virtual service intercepts the actual Service component. All requests are sent to the type: production app. Let’s examine the logs!:

Logs for the Production app:

Logs for the Debug app:

It works. I got a response from the production app. However, the debug app got the same requests as well.

--

--