Don’t balance TCP traffic with HAProxy
Although the title is pretty bold, you should think twice before opting to balance TCP traffic with HAProxy. And yes, I know that HTTP actually uses TCP connection underneath, but what I refer to here are long-living TCP sessions, especially in the scenario described in my previous article. Such services like RabbitMQ or MySQL fall in this category.
As we deploy often, and services might be automatically brought up or shut down by Marathon, HAProxy configuration is also changed often and every time HAProxy should be reloaded for changes to take effect. Since we do graceful reload with -sf option, the newly launched HAProxy process asks the old process to stop listening for new connections and exit when its active long-living connections are closed. The new process then takes over and starts to listen for new connections while the old one is still up. Now the new process accepts new long-living connections and ends up being hanging after the next configuration reload and so on. After a few reloads we can see the following:
And it may get even worse. The normal situation is to always see the single HAProxy session running. It means that after the configuration reload all predecessors’ connections are closed and processes safely quit:
When it comes to load balance such a long-living connections, it’s better to refer to IPVS which is already built-in to almost every Linux kernel. It performs transport-layer load balancing, doesn’t have a separate process, nor it opens any sockets. Also, it’s lightning fast as it works on a very low level (netfilter module), one could even load balance HTTP traffic with it, but only if request contents analysis is not required (remember, it’s common for HTTP balancers to read request headers such as “Host” to decide on a destination backend?).
We used to load balance long-living RabbitMQ connections with HAProxy, but after we’ve noticed such a behavior, we immediately moved it to IPVS by writing a custom IPVS configuration daemon which communicates with Consul to get registered backend services. Now our HAProxy does its job by only balancing HTTP traffic leaving TCP for IPVS where it really shines.
