Deploying AWS’s Web Application Firewall on CloudFront with dynamic content from an Elastic Beanstalk Rails app
This is Part 1 of 2 on deploying and configuring AWS’s WAF with CloudFront and Elastic Beanstalk. Part 1 is the guide to deployment and Part 2 will cover configuring AWS’s WAF effectively.
Like many startups, Primer uses AWS to power the majority of its infrastructure. So when it was time for us to start beefing up our security by installing a more advanced firewall, we immediately looked at AWS’s Web Application Firewall. It gives similar benefits of other WAF solutions without having to actually manage the servers, software, load balancer configurations and scaling. Whenever we can effectively take things off our team’s plate so we can focus on our core product, we consider it a big win! Sadly, we were quickly disappointed when we learned WAF only works with CloudFront.
However, we don’t give up easily and were determined to find a way to make it work. At the time, our infrastructure consisted of a Rails app deployed through Elastic Beanstalk. CloudFront served our static assets by caching them from our rails servers or S3. This lead to an idea: what if we could serve our entire site’s dynamic content through CloudFront to take advantage of AWS’s WAF. AWS advertises this use case on their info page, but without having actually tried it we were slightly skeptical it would work as we needed.
We pieced together some resources we found online (listed below), but none gave us a complete solution. There were also some tricky snags along the way, so we want to share our process for getting AWS WAF running with CloudFront to deliver an entire Rails app running on Elastic Beanstalk. Best of all we can deploy our new configuration without any downtime! We’d recommend testing out this deployment and setup on a staging server if you have any sort of traffic coming to your main site to make sure it fully works for your specific environment.
The first step is setting up a new CloudFront Web distribution to serve as the front line of the site. For the Origin Domain Name, use a new internal subdomain (ex: internal.myapp.com). We originally pointed the Origin domain directly to the load balancer for our Elastic Beanstalk servers. This caused a number of issues, especially when it came to SSL. The resulting solution is that the site’s origin must be on the same domain.
Next, under the cache settings, forward all headers, query strings, cookies, and set all TTLs to zero. This forces CloudFront to look to the Origin for the content, meaning users will always hit the app servers.
In the Distribution Settings, the Alternate Domain Name is where you specify the actual domain that you want to serve through CloudFront, in our case this was goprimer.com. Don’t forget if you use www. to also list this in the alternate domain names. Here we also set our SSL certificate. To access your SSL certificate in CloudFront it must be uploaded to AWS’s IAM. The Distribution Settings is also the spot to set a WAF if you have already created one, if not you can come back and update this setting later.
That’s it for the CloudFront distribution! Now, while the distribution is deploying (it usually takes about 15 minutes), we can setup the first part of the DNS settings in Route53. Currently your main domain should be pointing to the Elastic Load Balancer for your Elastic Beanstalk environment. Create a new A record for internal.myapp.com (the same we listed in the CloudFront origin) and set that Alias to the Elastic Load Balancer. Once the DNS propagates, navigating to the subdomain should display your site just as if you were accessing it from your main domain.
It might seem like switching over your main domain to your CloudFront distribution would be all that’s left, but there’s one more snag! Because users are now hitting CloudFront before your Rails app, Rails will actually return the IP address of the CloudFront endpoint instead of the user’s IP address. The Elastic Load Balancer always adds the IP of the CloudFront endpoint as the last value in the HTTP_X_FORWARDED_FOR value. Fortunately this is easily solvable by writing some Rails middleware! There are two ways we can solve this:
- CloudFront provides a list of IP addresses that match their endpoints in JSON format, so we could include that list in our `TRUSTED_PROXIES` variable of Rail’s RemoteIp.
- We could extend the RemoteIp middleware and tell Rails to always drop the last IP in HTTP_X_FORWARDED_FOR.
We chose to go with option 2 as that provides the least overhead and maintenance over time. However to deploy with no downtime, we also took advantage of CloudFront’s IP list, just until our DNS was fully upgraded and we could guarantee everyone was accessing our site through CloudFront (more on this soon).
The middleware we named CloudFrontRemoteIp and the code changes to install it in your application.rb are in this gist:
As we mentioned, we were able to deploy with no downtime, so we wanted to share the code to do that as well. The issue with switching over immediately is that DNS updates at different times for different users. So if we released the CloudFrontRemoteIp middleware as is above we’d potentially be stripping off an incorrect IP address for users who were still hitting our load balancer directly. The code in this gist only strips the IP if it’s in the list AWS provides. After several days, when we could ensure all users were hitting our site through our CloudFront distribution, we switched back to the original CloudFrontRemoteIp middleware so we didn’t have to worry about updating the CloudFront IP list as it changes. This change is really only needed for our production app, so if you are setting this up first on a staging environment, you can skip this step.
The last thing to do is direct your main domain to your new CloudFront distribution via Route53. To do this, update the A Record of your domain and point the alias to the CloudFront distribution, it should look similar to z123abcdefghij.cloudfront.net. This value can be found on the CloudFront section of AWS in the Domain Name field for your distribution.
That’s all! We now have the Primer site running with CloudFront and AWS’s WAF in front of our Elastic Load Balancer and Rails app. Since releasing, everything has been running smoothly. We’ve seen no issues with our web accessible dashboard or our API which is accessed from mobile devices all over the world. We also discovered some pretty cool use cases where we could serve high volume static or infrequently updated content directly on our main domain from s3 through CloudFront origins. Additionally, we’re really enjoying AWS’s WAF. It has allowed us to streamline our firewall and move it closer to the end user (previously we had used tools like the RackAttack middleware). We also don’t have to worry about scaling and maintaining the servers, a huge win for our small team! Part 2 of this series will cover correctly configuring AWS’s Web Application Firewall, so be on the lookout!
Resources that were helpful in getting this configuration setup:
- Full-Site Delivery with Cloudfront
- AWS Cloudfront Expiration Guide
- Whole Site Delivery with Cloudfront (AWS Presentation)
- How to get the client IP when using CloudFront and Nginx
This is just one of the many cool challenges we’re solving as our small engineering team at Primer scales up. We are building amazing things using AWS systems (like Kinesis), Rails and AngularJS, along with some breakthrough technology for SDKs in Objective-C and Java. We’re hiring! Check out our careers page.
We’d love to hear if you found this useful or any experience you have deploying your dynamic content through CloudFront, so please respond here and let us know!
Primer’s SDK enables fully-customizable native mobile app experiences, delivered from the server. Our customers are using this technology to grow their user base by converting more new users with compelling onboarding experiences.
