Terminology Confusion on EKS: WARM_ENI_TARGET, WARM_IP_TARGET, MINIMUM_IP_TARGET

EKS provides a few unique settings for managing Kubernetes’ IP pool.
Before delving into those settings, let’s look at IP address management in Kubernetes.

Photo by Kelly Sikkema on Unsplash

There are three critical aspects of networking:

• Each pod has a unique IP address
• All pods communicate with one another without using NAT
• All nodes communicate with one another without using NAT

(More details here: Kubernetes Core Concepts: Pod)

EKS leverages VPC subnets and Elastic Network Interfaces to satisfy these criteria (ENIs). But, there is a limitation to EC2 instances. Each instance type may have several ENIs, each with a set number of IP addresses. (More details here).

For this post, I will use the t3.medium instance type

t3.medium instances have 3 ENIs and each of which has 6 different IPv4 addresses. There are a total of 18 IPv4 addresses in all.

One IP address will be designated for the node itself.

Since each pod must have its own IP address, I can run 17 Pods on a single node using the t3.medium instance type.

Why?

Because those IP addresses will need to be assigned to the pods. You will need to update the instance type if you wish to run many more Pods on a Node. Newer EKS versions have a trick called ‘Prefix Mode’. This is not the subject of this article.

How do I determine which instance type is capable of running how many containers?

You may want to look at this list: eni-max-pods.txt

You can use this formula:

NUMBER_OF_ENI * (NUMBER_OF_IPV4_PER_ENI - 1) + 2

t3.medium has 3 ENIs, each with 6 IP addresses:

3 * (6–1) + 2 = 17 pods can run on t3.medium

What is WARM_ENI_TARGET?

The Elastic Network Interface is automatically added to an EKS Node when it is created. Attempting to schedule a high number of pods will exhaust the node’s available IP addresses.

Since your prior Pods are already utilizing available IP addresses, EKS will need to add another ENI to the node. This might be a time-consuming procedure (creating a new ENI, assigning IP addresses, attaching it to the node, etc.).

Meanwhile, your pods will wait to be run.

Moreover, some Kubernetes clusters may communicate frequently with the Amazon API. As a result, you may eventually reach the Amazon API rate limits.

EKS has an option called WARM_ENI_TARGET that may be used to speed up the process of assigning a fresh IP address to the pod.

WARM_ENI_TARGET = 1 tells EKS to connect an available ENI to your node. For future usage, a new ENI is attached.

When the node needs more IP addresses, it does not have to wait for Amazon VPC instructions.

Example scenario:

1. You have just created a node with WARM_ENI_TARGET=1
2. There are two ENIs on the node. Primary ENI is now in-use, and secondary ENI is available (ready-to-use).
3. The primary ENI has 6 addresses.
4. The node has 1 pod (aws-node).
5. 1 IP address has been allotted for the node itself.
6. 2 IP addresses from the primary ENI in-use. 4 IP addresses available (ready-to-use) from the primary ENI.
7. You want to deploy 5 more pods to the node.
8. You need another ENI since you have 4 IP addresses available.
9. You do not need to wait to add another ENI since you have already set WARM_ENI_TARGET=1. You may begin using another one.
10. Through the secondary ENI, Pod 5 obtained an IP address.
11. ***Important***
    EKS will add another ENI to the Node. Hence, if you require extra IP addresses on this node, you may utilize an IP address from the third ENI.

The value of WARM_ENI_TARGET may be more than one. Assume its value is 2. EKS will add two more ENIs for future usage.

It should be noted that each instance type may have a set number of ENIs.

What is WARM_IP_TARGET?

It is pretty similar to WARM_ENI_TARGET. When you set WARM_IP_TARGET=1, EKS adds another IP address for future use. If the current ENI does not have an available IP address, a new ENI will be added to the node.

WARM_IP_TARGET’s value can be more than 1.

What is MINIMUM_IP_TARGET?

It is pretty similar to WARM_IP_TARGET. It can be used to specify the minimum number of IP addresses in use and available.

What is the difference between WARM_IP_TARGET and MINIMUM_IP_TARGET?

MINIMUM_IP_TARGET = 3

It means:

The number of available IP addresses + the number of IP addresses in use must be at least three.

WARM_IP_TARGET = 3

The number of available IP addresses must be at least three.

Example scenario:

1. MINIMUM_IP_TARGET = 5 and WARM_IP_TARGET = 2

5 IP addresses will be assigned to the node. When 5 pods are deployed, the node is assigned 2 new IP addresses.

The total number of available + in-use IP addresses is 7

2. MINIMUM_IP_TARGET = 2 and WARM_IP_TARGET = 5

5 IP addresses will be assigned to the node. When 5 pods are deployed, the node will get 5 new IP addresses.

The total number of available + in-use IP addresses is 10