ARP stale cache and a rather peculiar arp_notify behavior in Linux
We’ve got a minor bug in our monitoring systems. Check for Neutron’s DHCP agent availability occasionally generated false positives (reporting problem when everything is OK). We’re checking DHCP agents availability by simply pinging their IP addresses.
After extensive research I found that inside dhcp namespace, interface where dnsmasq listens, receives ICMP (from ping) but that ICMP has a wrong MAC destination address.
That was strange, but lately I found that local pings (in the same network segment) work fine. Issue was in routed packets. Router’s ARP table shows that it was assigned to wrong MAC address (the same wrong address I saw in tcpdump inside DHCP namespace). After some searching in older logs of Neutron I found that IP address was earlier assigned to other neutron port (instance, in fact, which was deleted not log before). Router learned MAC address from traffic of that instance, and used for ping to that IP.
To solve that problem I’ve tried enable net.ipv4.all.arp_notify=1, net.ipv4.default.arp_notify=1, but it failed. It was really peculiar.
What arp_notify=1 does?
It should send gracious ARP. It sometimes called ‘probe ARP’ and ‘grace ARP’. This is just a notification to all other member of network segment that there is a new IP address and it’s MAC address.
Amazingly, it didn’t work. I asked my friends, read the docs, and found, than, to my amusement, Linux sends grace ARP (when arp_notify=1) only in two cases:
- Interface was brought up
- Interface’s MAC address has been changed
If new IP address (even 1st IP address) is assigned, no grace IP would be send. If you want to use grace ARP, you need:
- Enable arp_notify
- Configure interface IP
- Bring it up
Only this sequence (albeit order of arp_notify and IP address operations) will guarantee grace ARP.
Neutron agent when it configures DHCP does the opposite: It brings interface up, and then assign IP address(es). This will not send grace ARP, and together with IP recycling and long ARP table live time on router this cause false positive.
Fixing
To fix it locally (for given DHCP agent one should or wait until ARP record on the router go away due to the old age, or send any network packet to the router. Some ping from within DHCP namespace would do the job.
To fix it permanently more severe changes should be done.
- I filed the bug to Openstack neutron: https://bugs.launchpad.net/neutron/+bug/1672433
- I filed the bug to Linux upstream. I think, that reasonably expects that Linux would send grace ARP on every new IP address, not only for up/down/new_mac events: https://bugzilla.kernel.org/show_bug.cgi?id=194879
For now it goes to our local knowledge base for operations department, plus we consider to write a patch for dhcp agent which would send grace ARP or perform any network activity on network namespace activation.
Update: Unfortunately, bugreport into kernel bugtracker was rejected with following explanation:
Linux kernel does uses a weak binding between addresses and interfaces, plus there are other sysctl's controlling arp behavior. A patch to change current behavior when value is 1 is unlikely to be accepted. You might get a patch accepted that adds a third state (value = 2) but it would be complex since with weak host model it is not easy to do.
