How to Increase the Pod Limit on an Amazon EKS Node Using Prefix Mode?

As stated in the previous post, the number of pods that can run a Node is limited.

You can follow me on LinkedIn

Photo by Maciej Ruminkiewicz on Unsplash

The total number of ENIs and IP addresses per ENI are used to compute the number of pods that may operate a Node.

(More details here: Terminology Confusion on EKS: WARM_ENI_TARGET, WARM_IP_TARGET, MINIMUM_IP_TARGET)

However, Prefix Mode is available as of version 1.9.0 of the AWS VPC CNI (Container Network Interface).

What is the Prefix Mode?

Each EC2 instance type may have several ENIs, each with a set number of IP addresses. For example; t3.medium instances have 3 ENIs, each of which has 6 different IPv4 addresses.

When you enable the Prefix Mode, AWS VPC will allocate each ENI a /28 IPv4 block (2⁴ = 16 IP addresses) rather than a single IP address.

How Do I Enable the Prefix Mode?

You should create the node group with max-pods-per-node

Example:

eksctl create nodegroup --cluster adil-eks-cluster --region eu-west-1 --name adil-nodes --max-pods-per-node 110

If you create your node group with YAML:

...
managedNodeGroups:
  - name: adil-nodes
    instanceType: t3.medium
    desiredCapacity: 2
    minSize: 2
    maxSize: 4
    maxPodsPerNode: 110

After your nodes are created with a new limit (110), you can enable prefix mode:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Why is the limit 110?

Kubernetes supports a maximum of 110 pods per node. There is an ongoing debate about increasing the limit.

The relevant parameters on my aws-node pod:

➜  ~ kubectl exec -it aws-node-8djs2 -n kube-system -- /bin/bash
Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)

bash-4.2# env | grep -e _PREFIX -e WARM
ENABLE_PREFIX_DELEGATION=true
WARM_ENI_TARGET=1
WARM_PREFIX_TARGET=1

The node’s network settings:

2 pods are running on this node:

➜  ~ kubectl get pods --all-namespaces --field-selector spec.nodeName=ip-192-168-42-72.eu-west-1.compute.internal
NAMESPACE     NAME               READY   STATUS    RESTARTS   AGE
kube-system   aws-node-hg88p     1/1     Running   0          2m58s
kube-system   kube-proxy-mndsq   1/1     Running   0          5m23s

I will place 17 more pods in this node:

19 pods in total

Please notice that the majority of Pods are operating on the 192.168.39.144/28 block. Since all of the IPs in that block are in use, a new block (192.168.43.208/28) is added.192.168.46.112/28 added to be used in the future:

new IP blocks

The maximum number of partitions that could operate on a t3.medium before these settings was 17, now it is 110.