Nginx and Amazon ELBs

Where I work we have just recently migrated our platform to Amazon Web Services. We use Nginx to good effect as a general routing layer for all our applications but came across a nuance which caught us out so thought I would share our solution.

The issue arose when we needed to configure part of our Nginx setup to use Amazon’s elastic load balancers (ELB) as an upstream proxy. Now by their very name sake we should have realised something was not going to be right when using these to proxy traffic.

At first we weren’t sure what the issue was, our Nginx would stop serving traffic and when we looked in the logs it was clear that it was marking these upstream services as bad.

With further investigation, we discovered the IP address Nginx was resolving for the ELB was no longer valid, so we had a DNS caching issue. What we discovered was even worse, once started, Nginx only did a DNS lookup once until restarted. ELB’s occasionally change their IPs (as they are elastic, go figure) so when Nginx sent traffic to our custom-domains.us-east-1.elb.amazonaws.com ELB it obviously didn’t work too well.

The solution was to add a resolver into our HTTP block. When using a resolver, by default Nginx has a 5 minute TTL for DNS lookups so in our case we decided to add in an extra parameter for a 30 second TTL.

Here is an example:

http {
resolver 10.1.1.1 valid=30s;
 include upstream/*.conf;
 server {
listen 80 default_server;
server_name au_forwarder;
proxy_pass http://custom-domains.us-east-1.elb.amazonaws.com;
}
 location / {
proxy_pass http://custom-domains.us-east-1.elb.amazonaws.com;
}
}

Now you may ask, “why am I hard coding this”, which would be a great question, especially in an AWS environment. For that, you will have to wait for my next post where I will introduce how we template our Nginx configuration to make it more flexible by using something like envtpl (https://github.com/andreasjansson/envtpl).

Show your support

Clapping shows how much you appreciated Justin Hennessy’s story.