As stated in the previous post, the number of pods that can run a Node is limited.
The total number of ENIs and IP addresses per ENI are used to compute the number of pods that may operate a Node.
(More details here: Terminology Confusion on EKS: WARM_ENI_TARGET, WARM_IP_TARGET, MINIMUM_IP_TARGET)
However, Prefix Mode is available as of version 1.9.0 of the AWS VPC CNI (Container Network Interface).
What is the Prefix Mode?
Each EC2 instance type may have several ENIs, each with a set number of IP addresses. For example;
t3.medium instances have 3 ENIs, each of which has 6 different IPv4 addresses.
When you enable the Prefix Mode, AWS VPC will allocate each ENI a /28 IPv4 block (2⁴ = 16 IP addresses) rather than a single IP address.
How Do I Enable the Prefix Mode?
You should create the node group with
eksctl create nodegroup --cluster adil-eks-cluster --region eu-west-1 --name adil-nodes --max-pods-per-node 110
If you create your node group with YAML:
- name: adil-nodes
After your nodes are created with a new limit (110), you can enable prefix mode:
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
Why is the limit 110?
Kubernetes supports a maximum of 110 pods per node. There is an ongoing debate about increasing the limit.
The relevant parameters on my
➜ ~ kubectl exec -it aws-node-8djs2 -n kube-system -- /bin/bash
Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)
bash-4.2# env | grep -e _PREFIX -e WARM
The node’s network settings:
2 pods are running on this node:
➜ ~ kubectl get pods --all-namespaces --field-selector spec.nodeName=ip-192-168-42-72.eu-west-1.compute.internal
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-hg88p 1/1 Running 0 2m58s
kube-system kube-proxy-mndsq 1/1 Running 0 5m23s
I will place 17 more pods in this node:
Please notice that the majority of Pods are operating on the 192.168.39.144/28 block. Since all of the IPs in that block are in use, a new block (192.168.43.208/28) is added.192.168.46.112/28 added to be used in the future:
The maximum number of partitions that could operate on a t3.medium before these settings was 17, now it is 110.