HAProxy and 503 HTTP errors with AWS ELB as a backend
Although AWS provides load balancer service in the form of Elastic Load Balancer (ELB), a common trick is to use HAProxy in the middle to provide SSL offloading, complex routing and better logging.
In this scenario, a public ELB is the frontier of all the traffic, HAProxy farm in the middle is managed by an Auto Scaling Group, and one (or more) internal backend ELBs stay in front of the Web farm.
I think that HAProxy does not need any introductions here. It is highly scalable and reliable piece of software. There is, however, a small caveat when you use it with domain names and not IP addresses. To speed up things, HAProxy resolves all the domain names during startup (during config file parsing in fact). Hence, when the IP of a domain changes, you end up with a lot of 503 (Service Unavailable).
Why is this important? In AWS, ELB’s IP can change over time, so it is recommended to use ELB’s domain name. Now, when you use this domain name in HAProxy’s backend, you can end up with 503s. ELB IPs do not change so often but still you would not want any downtimes.
The solution is to configure runtime resolvers in HAProxy and use them in the backend (it works only in HAProxy 1.6+). This way HAProxy will verify the IP address of the ELB DNS name every 10s (or whetever you put in the hold valid parameter).
