One of the key products of Tablehero is automatic creation and hosting of beautiful and mobile friendly websites for restaurants.

Hosting multiple websites

Tablehero is completely hosted on AWS. We started out with a typical setup to serve these websites — a set of application servers which fetch data from the database and render the restaurant website on the fly. These application servers decide which restaurant to render based on the hostname of the HTTP request. A classic Elastic Load Balancer (ELB) was configured with HTTP listener forwarding the traffic to the set of application servers.

This was all well and good, but we decided that all the restaurant websites served by us should be HTTPS. Because, its a good thing. Also, with more and more restaurants opting for Online orders, Merchandise sale, etc — their website essentially becomes an online store where customers transact and hence HTTPS is essential.

TLS and SNI

Now, the problem:

When making a TLS connection, the client (browser) requests a digital certificate from the web server. Once the server sends the certificate, the client examines it and compares the name it was trying to connect to with the name(s) included in the certificate. If a match occurs, the connection proceeds as normal. If a match is not found, the user may be warned of the discrepancy and the connection may abort as the mismatch may indicate an attempted man-in-the-middle attack. …

A server that is responsible for multiple hostnames is likely to need to present a different certificate for each name. …

Name-based virtual hosting allows multiple DNS hostnames to be hosted by a single server (usually a web server) on the same IP address. To achieve this, the server uses a hostname presented by the client as part of the protocol (for HTTP the name is presented in the host header). However, when using HTTPS, the TLS handshake happens before the server sees any HTTP headers. Therefore, it is not possible for the server to use the information in the HTTP host header to decide which certificate to present and as such only names covered by the same certificate can be served from the same IP address. …

SNI (Server Name Indication) addresses this issue by having the client send the name of the virtual domain as part of the TLS negotiation.This enables the server to select the correct virtual domain early and present the browser with the certificate containing the correct name. Therefore, with clients and servers that implement SNI, a server with a single IP address can serve a group of domain names for which it is impractical to get a common certificate.

— From the Wikipedia page on SNI (Server Name Indication)

Since 2006, web browsers have been adding support for SNI and now it is supported by most web browsers — on desktop and mobile devices.

Sounds well and good. The catch is that AWS ELB does not support SNI — at least not yet. So, we cannot have the ELB handle the SSL termination. Enter nginx with SNI support.

LetsEncrypt and nginx with SNI

It is straightforward to configure nginx to handle multiple virtual hosts using different server blocks in its configuration.

What about the SSL certificates? Anyone who has purchased multiple SSL certificates from traditional Certificate Authorities (CAs) know that the experience is far from smooth. Imagine purchasing and managing hundreds of SSL certificates.

Enter LetsEncrypt.

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit.

Sponsored by industry heavyweights, and with a deliberate focus on transparency, it certainly is the way to go.

The certificates issued by LetsEncrypt are valid for 3 months and hence certificate renewal needs to be dealt with almost immediately. EFF’s Certbot is a command line tool which makes getting new certificates and renewing them a relatively simple task.

certbot-auto — nginx -d {{ domain_name }} -n — redirect

When run the first time, it gets a new certificate for the domain and using the renew option, subsequent runs check if it is due for renewal and gets a new certificate if needed. All this while the nginx server is running — checkout the awesome ACME protocol.

Tying up

Next, to eliminate the single point of failure of the nginx instance, we added a new ELB in front of a set of instances running nginx. This ELB has a TCP listener (not an HTTP listener) and it simply transfers all TCP traffic on port 80 and 443 transparently to the nginx servers at its backend. The final picture looks like this.

There is still a problem to be taken care of. Forwarding requests from nginx to the HTTP ELB using a proxy_pass directive is not as straightforward as it sounds. This is because the IP address of the ELB endpoint may change quite frequently. And nginx does the DNS lookup once and uses the result till the configuration is reloaded. The way around this is to use the resolver directive as described here.

The remaining work was mainly writing Ansible roles and playbooks for doing tasks like adding new virtual hosts, getting and renewing certificates on the nginx instances and making it all available through good old Jenkins.

et voilà