It might be necessary to utilize several network interfaces within a single subnet. Separating application traffic from replication traffic on the services (like Kafka, Elasticsearch, etc.) is a common use case.
Linux uses the weak host model; hence, having multiple network interfaces might lead to unexpected network issues.
Table of Contents
· What is the host model?
· What is the ARP Flux problem?
· Why is having two MAC addresses a problem?
· Reproduce the ARP Flux Problem
· Let’s fix the multiple MAC addresses issue:
· Routing Problem
· Fix the routing issue on the multiple network interfaces
What is the host model?
Either the weak host model or the strong host model is used by modern operating systems.
Weak host model (a.k.a. Weak End System Model or Weak ES Model): If the network packet's destination IP address matches one of the IP addresses on its interfaces, the operating system will accept the packet.
TL; DR: The operating system acts like a router for its network interfaces.
Linux uses weak host model.
Strong host model (a.k.a. Strong End System Model or Strong ES Model): If the IP address does not match the IP address on the interface that the traffic was routed through, the operating system does not accept the packet.
TL; DR: The operating system does not act like a router for its network interfaces.
Newer Windows systems use strong host model.
What is the ARP Flux problem?
If the operating system gets an ARP request and there are multiple network interfaces on the same subnet, the MAC addresses of both will be returned due to the weak host model.
Why is having two MAC addresses a problem?
Having two MAC addresses isn’t an issue. The problem is that the OS returns multiple MAC addresses. The network routers could drop your packets. In particular, if the cloud provider’s source/destination check option is set, the packet will be dropped since the returned MAC address does not match the MAC address for the IP address.
Reproduce the ARP Flux Problem
The setup is as follows:
A virtual machine will act as a client.
Another virtual machine will act as a server.
The server has two interfaces:
eth0: 10.211.55.13
eth1: 10.211.55.14
gateway: 10.211.55.1
The client has one interface:
eth0: 10.211.55.7
gateway: 10.211.55.1
I can ping both IP addresses from the client:
Let’s look at the ARP packets sent by the server:
The server returns two MAC addresses.
Let’s fix the multiple MAC addresses issue:
I’ll activate arp_ignore on the server:
echo 2 > /proc/sys/net/ipv4/conf/all/arp_ignore
The server returns one MAC address for each IP address.
Routing Problem
Let’s examine the Tcpdump output on the server:
Notice the difference;
✅ ✅ When I sent a ping request to 10.211.55.13, the request reached the server via its interface eth0, and the response was sent from the server’s eth0 interface.
❗️❗️When I sent a ping request to 10.211.55.14, the request reached the server via its interface eth1, and the response was sent from the server’s eth0 interface
Fix the routing issue on the multiple network interfaces
Create a route table for each interface on the server:
echo "100 eth0rt" >> /etc/iproute2/rt_tables
echo "101 eth1rt" >> /etc/iproute2/rt_tablesAdd necessary rules to the tables on the server:
ip route add 10.211.55.0/24 dev eth0 src 10.211.55.13 table eth0rt
ip route add default via 10.211.55.1 dev eth0 src 10.211.55.13 table eth0rt
ip rule add from 10.211.55.13 table eth0rt
ip rule add to 10.211.55.13 table eth0rt
ip route add 10.211.55.0/24 dev eth1 src 10.211.55.14 table eth1rt
ip route add default via 10.211.55.1 dev eth1 src 10.211.55.14 table eth1rt
ip rule add from 10.211.55.14 table eth1rt
ip rule add to 10.211.55.14 table eth1rtScreenshot from the server:
Let’s test the ping:
It works
