HAPRoxy — HTTPS Load Balancing on SNI
These notes are aimed at understanding what HAProxy offers to load balance HTTPS traffic and the difference between mode HTTP and mode TCP. It is aimed at admins switching from old load balancer implementation to new ones.
Before you get started, you’ll need to understand the basics behind HTTPS communications and principles of load balancers and HAProxy configuration files.
Context
We are in the process of improving our technology communication layer with clients and partners as well as targeting 99.99% service availability from 99.9%. A component in achieving this is our load balancer infrastructure. It now requires the capability to load balance TLS traffic based on subdomains and properly redirect traffic to the right backends.
After considering different solutions, we decided to keep HAProxy as we only had to upgrade to version 1.8 to improve SNI support and SNI health checks. This required a lot more research than anticipated to properly understand the TLS protocols and the difference between the 2 modes of operations (TCP vs HTTP) in HAProxy.
Here is a quick synopsis to help you decide which route you need or want to go with.
HAProxy modes: TCP vs HTTP
With HAProxy we have 2 options to load balance based on the server name indicator (SNI):
· SSL session termination at the load balancer (Mode HTTP)
· Transparent passthrough between the client and the server (Mode TCP)
SSL Termination — Mode HTTP
In the case of HAProxy, SSL session termination is done by using the HTTP mode and providing the load balancer with the proper certificates and associated chains. The traffic looks like this:
In this scenario, HAProxy allows you to decide if you want the traffic between the load balancer and the server to be encrypted or not. Encrypted traffic should be the norm even on secure network but some backend servers do not always support it especially in the case of legacy applications.
In HTTP mode, you can use the ssl_fc_sni indicator or the host header information (hdr(host)) to filter the incoming session and route it to the proper backend server.
The advantages of the solution are:
· Operates at layer 7 and offer more functionality on routing the traffic properly and rewriting data for the backend servers
· Legacy backend servers can be presented as TLS web sites
And, in our case, the big inconvenient to this method is:
· Client certificates are stopped at the load balancers and are not transmitted to the backend server as is.
Although HAProxy allows you to extract every bit of information from the client certificate and put it into the header of the data being sent to the server, that means your application needs to read header information instead of certificate information. For us, this meant spending a lot of time re-writing legacy code.
Transparent passthrough — Mode TCP
For passthrough, HAProxy needs to work on the TCP layer (mode TCP). The traffic looks like this:
Since HAProxy does not decrypt the HTTPS data, we still need to get the information we need to route properly. Luckily, the TLS protocol and communication is as follow:
Figure 1: Source: https://tools.ietf.org/html/rfc2246#page-31
And following RFC5246 (https://tools.ietf.org/html/rfc5246#page-41), the TLS protocol allows the ClientHello message to contain extensions. One such extension is the server_name extension. HAProxy can retrieve the SNI information from the ClientHello message:
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
acl acl_app1 req_ssl_sni -i app1.domain.com
use_backend bk_app1 if acl_app1
In this mode, you use the req_ssl_sni value to filter the incoming session and route it to the proper backend server. ssl_fc_sni is not available in TCP mode nor is the header (hdr(host)) as traffic is encrypted.
The advantages of the solution are:
· Client certificates are blindly sent to the proper backend servers based on SNI
· No need to deploy the server certificates and chains on the load balancers
· And, in our case, we do not need to upgrade legacy code
Choosing the TCP mode worked perfectly in our use case scenario and allowed us to support legacy code and authentication mechanisms (client certificate)
Our next steps
In our case, client certificates and custom PKIs require a high amount of resources for the code and the infrastructure. Thus, we have decided to slowly move to another authentication method.
The new solution provides us:
· Higher availability
· Reduced management for both us and our customers of the authentication component in our ecosystem.
· Reduced additional avenue of attacks by signing each request
· Enrollment capabilities even when losing datacenters
And, since this is an article on HAProxy, it improves the feature sets we can use by upgrading our load balancing layer to the HTTP mode.
Acknowledgements
I’d like to thank the community and the people on the IRC channel (freenode, #haproxy) that helped figuring this out so that the setup would properly work. The documentation while being very good sometimes does not provide the reasons for the different modes and options available.
Additional resources can be found on the web:
· Google results for: HAProxy req_ssl_sni vs ssl_fc_sni
· HAProxy 1.8 configuration manual: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html
· A good example of client certificate information rewriting: http://www.loadbalancer.org/blog/client-certificate-authentication-with-haproxy/