Configuring IPTABLES to work with AWS hosts (like *.crashlytics.com)

The problem

Some time ago I had to configure IPTABLES on a server which acts as the APN gateway for mobile phones used by my client. All traffic comes through this server and is filtered using IPTABLES service.

One of the required services which had to be available is crashlytics.com. I quickly figured out that simple rule below is not enough.

-A FORWARD -s xx.xx.xx.xx -d settings.crashlytics.com -j ACCEPT

Why’s that?

The reason is really simple. When you put a domain in -d parameter IPTABLES will resolve this domain against DNS and put resolved IP address in the place of the domain. So after IPTABLES restart and typing iptables-save command in console you’ll see above rule but slightly modified:

-A FORWARD -s xx.xx.xx.xx -d dns.resolved.ip.address -j ACCEPT

Why it’s wrong?

For some cases it might not be a problem — I mean the hosts which IP is constant, but for crashlytics.com and it’s subdomains — IP is not constant.

I ran dig settings.crashlytics.com several times and each time I got a different IP address!

dig settings.crashlytics.com
; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> settings.crashlytics.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34220
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 4, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;settings.crashlytics.com. IN A
;; ANSWER SECTION:
settings.crashlytics.com. 60 IN CNAME settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com.
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 54.197.241.124
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 54.225.212.15
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 50.19.89.237
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 54.225.152.187
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 23.23.254.42
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 23.23.96.14
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 50.19.85.98
settings-crashlytics-1410998606.us-east-1.elb.amazonaws.com. 60 IN A 50.16.244.33
;; AUTHORITY SECTION:
us-east-1.elb.amazonaws.com. 300 IN NS ns-934.awsdns-52.net.
us-east-1.elb.amazonaws.com. 300 IN NS ns-1793.awsdns-32.co.uk.
us-east-1.elb.amazonaws.com. 300 IN NS ns-235.awsdns-29.com.
us-east-1.elb.amazonaws.com. 300 IN NS ns-1119.awsdns-11.org.
;; Query time: 224 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Sep 08 14:32:48 CEST 2017
;; MSG SIZE rcvd: 388

Each time I ran this command I got a different results in ANSWER SECTION , but there was always one common thing — all A records hostnames have us-east-1.elb.amazonaws.com suffix which indicated that the service is hosted on AWS cloud and hosts are located at us-east-1 region.

It still didn’t help me to configure IPTABLES correctly but after a while, I came up with the idea for…

The solution

I did a little research and I found out that Amazon publishes IP addresses they use as JSON file which is modified from time to time — you can find it here with a small documentation how to use this file.

Based on that I thought that I can write a bash script which will run every night, check if there is new JSON published and in case there is — modify my IPTABLES rules to match with current IP addresses used by Amazon.

What steps have to be done by the script?

There are several steps that have to be done by the script to apply new IP addresses to the configuration file.

  1. Download current JSON file from AWS server.
  2. Check if date from current JSON differs from the previously fetched file. If it doesn’t — stop execution (there are no changes to apply).
  3. Use jq to fetch IP addresses from specified region and strip them from JSON syntax.
  4. Generate new rules for IPTABLES based on the IPs we generated in the previous step.
  5. Replace the section for auto-generated AWS rules in IPTABLES config file.
  6. Restart IPTABLES service.

As all of these sounds quite simple, there was actually one problem in step number 5.

IPTABLES configuration file structure

In step 5 we want to replace a section dedicated to AWS-related rules. The simplest solution would be to put this section at the end of the file and mark it by putting some kind of comment marker (e.g. # AWS CRON ). Then we could just find this marker, drop all the lines below and put new content at the end of the file. But with IPTABLES config it’s not that simple…

The IPTABLES configuration file contains sections for each table: mangle, filter, nat. Those sections look like the one below:

*nat
...
rules
COMMIT

It caused that the solution with replacing the end of file couldn’t be applied. Although I found a way to do this in the other way, but it requires to manually add the comment marks in IPTABLES config file before the first run of the script.

In filter table section I put those markers to indicate the place where I want this auto-generated rules to be written.

# AWS CRON
# AWS CRON END

With those, my script will look for those markers and replace them with the generated rules set (of course adding those markers and the beginning and the end of the set).

The code

This is the final code of the script. I setup CRON job to run this every night around 2am to update the rules and it works like a charm!

IPTABLES rule template was modified in the given script so it may not be exactly correct, but the most important part of this line (30) is the usage of $line variable. You can use it with your own rule template.

Feedback?

I’m not a professional system administrator and I don’t write a lot in Bash so please feel free to provide a feedback about the script or some other ways of handling the issue I faced. It’s also my beginnings with writing technical posts so feedback is also desired in this area.


Thank you for reading my post! I would really appreciate your feedback so please feel free to share one in the comment! Please also get familiar with my other writings.

You can also find me on:
LinkedIn | Twitter | GitHub | StackOverflow