Imagine that you have an application running on multiple EC2 instances.
The app has an internal network load balancer in front of it so that other lambda functions or apps in a different VPC can access it (via VPC peering).
Assume for the moment that you wish to call the application itself using the internal load balancer.
It will fail. Why?
Due to the internal load balancer's default behavior of keeping the source IP address.
How so?
Assume the following configuration:
If Instance 2 receives a request sent from Instance 1 to the load balancer, the request will be successful, and the network connections will appear as follows:
Tcpdump on Instance 1:
Tcpdump on Instance 2:
Instance 1 sent a request to the load balancer’s IP address and received a response back from the load balancer.
Instance 2 got a request from Instance 1’s IP address and responded to that address.
Why does Instance 1 appear to have received its response from the load balancer when Instance 2 has directly returned the response to Instance 1?t
Magic!
No, not really. This is what is called Client IP preservation. NLB can preserve clients' source IP addresses. The target server (Instance 2) can see the client’s (Instance 1) actual IP address rather than the load balancer’s IP address.
Since the target server is not aware of the load balancer’s IP address, the target server will try to send the request to the client directly.
However, AWS is aware of this fundamental issue. In their NAT Table, they modify the source IP address for Instance 1. In this way, Instance 1 thinks the answer is from the load balancer.
What happens if Instance 1's request goes to Instance 1 rather than Instance 2?
Tcpdump on Instance 1:
The request is sent to the load balancer. Because the NLB preserves the source IP and port, Instance 1 recognizes it as an incoming request.
The OS says, “I was expecting a response from 10.0.0.139:80, but instead, I got one from 10.0.0.140:39648. I didn’t expect this to happen.” In other words, the connection is never established and eventually times out.
How to fix this problem?
1) Disable Client IP preservation
The easiest way to fix it is to turn off client IP preservation:
Disable it:
2) Keep Client IP preservation on and enable NAT loopback/hairpinning
Sometimes, the client’s IP is an absolute need, and you want to keep the client’s IP preserved.
If the source and destination IP addresses are the same or belong to the same subnet, NAT loopback redirects the traffic before it reaches the load balancer:
iptables -t nat -A OUTPUT -p tcp --source 10.0.0.140 --dport 80 -d 10.0.0.139 -j DNAT --to-destination 10.0.0.140:80tcpdump on Instance 1 (when Instance 1 attempts to send a request to the load balancer):
The request was looped back thanks to the NAT rule.
tcpdump on instance 1 (when the lambda function sends a request to the load balancer):
The source IP address of the lambda function can be observed in Instance 1:
But I use Docker!
Even though Docker does not encourage modifying iptables rules, you may want to use this rule to fix the issue (use at your own risk):
sysctl -w net.ipv4.ip_forward=1
iptables -I PREROUTING -t nat -p tcp --dport 80 -d 10.0.0.139 -j DNAT --to-destination 10.0.0.140:80