Compliance at Velocity with AWS Config
When it’s time to lay down the law, you better make sure you close the loopholes. This is what we had to do recently when we noticed that our EC2 Security Groups were skirting compliance in our development environments. The rule in this case is don’t allow any open internet connections (0.0.0.0/0) on SSH port 22 or RDP port 3389. Any service or developer wanting an instance to be accessible via SSH or RDP had to make sure that the connection was locked down to a known Signiant network. We found that there were outlaws skirting these rules and thats where we sprang into action.
Creating Compliance
At re:Invent in 2014, AWS introduced the Config service. The service works as watchdog, auditing changes in your AWS account and alerting on whether your service configurations fall into a compliance with the standards of the environment. This is determined by rules that you create to track down what you consider compliant. A couple of years ago we decided to make use of this service to lock down EC2 Security to groups in an automated fashion. As mentioned earlier, we wanted to make sure that anyone creating Security Groups would need to make sure that SSH and RDP was locked down. To do this we created 2 config rules.
First we would tackle SSH. The setup of a rule is trivial with AWS Config’s libary of Managed Rules. When Clicking the “Rules” link in the Config dashboard and choosing “Add rule”, we are given the option of viewing managed rules. These rules are provided by AWS as common configuration audits us to use. There is already a restricted-ssh option for us to use:
Choosing this rule gives us everything we wanted and will alert on non-compliance should a Security Group be created with a wide open SSH inbound rule. The default behaviour for the rule’s frequency is to check for compliance every time a configuration change occurs on a Security Group within the account. From here we need a compliance officer to action non compliant groups and making everything on the level.
Reaping Rewards
The offenders have been identified by AWS Config but we still need to action that intel. AWS documentation suggests streaming config changes to SNS and then subscribing to that topic via email or any other alerting method of your choice. This is a good method to be informed of the issue but we needed to go further and handle the alert in automated fashion, shutting down the rule. For that we created a solution called the AWS Config Rule Port Reaper, a docker container that is ready to do the dirty work of shutting down rogue open ports. It is scheduled to run periodically in each of our AWS accounts, finding out what resource isn’t compliant and then removing the rule in question based on the type of infraction.
We had originally developed this before Lambda was being widely used at Signiant. Today this would most likely be a Lambda function from the get go, weeding out trouble and returning us to a fairly secure state.
Catching Them All
Until recently everything was working well with our security group Compliance. Users were more mindful on what ports they left opened and we rarely had the reaper doing working. But then we found a loophole.
We received an alert that the reaper ran and that there was a Security Group not in compliance. We expected to the Reaper to work as per normal but something was amiss. We continued to receive warnings about the same security group being not in compliance for SSH. This should have been handled by our ‘Lawbringer’ but it was not making the cut. Taking a closer look at the security group we found not every case was being considered. The group looked as follows:
SSH was wide open, and the Config rule had deemed it not compliant. It seems our Reaper tool was not robust enough to handle a port range over a port that was being monitored. Further to this, we had used the managed AWS config rule, restricted-rdp, to alert on RDP ports that open to the web. We learned this rule did not take into account that RDP can be over TCP and UDP, alerting only on TCP cases. The TCP security group rule was squashed by the reaper but the UDP rule was left behind in a state of non conformity. The oversight in the Reaper would require a quick change in checking for a range condition, but the restricted-rdp over UDP was going to require a Custom Rule.
There are a few ways to create a custom Config rule. You can go the manual route where you follow the AWS Config console creation wizard which includes you developing a Lambda from scratch ( or using an existing one ). It has to handle the Config event that gets fired by the rule for checks. The event includes a “Configuration Item” object that will includes all the configuration of a single AWS Resource in json. In our case it’s a Security Group configuration. As well you will be able to pull any parameters specified in the Config rule parameters section that can optionally be used with the evaluating code. The Lambda must return either COMPLIANT, NON_COMPLIANT, or NOT_APPLICABLE (for those times you are way off in your evaluation). Figuring out a manual deploy and linking the ARN back to the rule is cumbersome and we also have to think about how we are going to roll this out to multiple accounts and multiple regions. Luckily I was able to come across a speedy solution to my woes.
Ruling at Velocity
The stranger that came into town to clean up the mess ended up being the AWS Config Rules Development Kit provided by AWS Labs. The python based tool, available on to install on pip, allows you to ramp up rule creation in a flash. Once installed, you initialize your working directory where you’ll be writing the rule which also prepares an S3 bucket for the region you’ll be deploying the rule:
rdk -p dev -r us-east-1 initYour lambda code will also be stored in this bucket. Next you create a rule:
rdk -p dev create restrict-rdp-udp --runtime python2.7 --resource-types AWS::EC2::SecurityGroup --input-parameters '{"PORT":"3389"}'This automatically generates stub code in my language of choice (java, nodejs, or python), in this case python, and will be checking the SecurityGroup resource type. We also introduce the PORT input parameter. All that’s left to do is to look through the Configuration object and parameters to determine what I think would be complaint. The stub has a section in the evaluate_complaince method for you to determine what is complaint vs non complaint.
The code that I ended up using is detailed in my Custom Config Rule repository. You can test your code with an example resouce, test_event_template.json, that they provide. Once you’re satisfied, it’s time to deploy:
rdk -p dev -r us-east-1 deploy restrict-rdp-udpThis command uploads a compressed Lambda to the initialized Config bucket, and runs a CloudFormation Template to create the rule and deploy the Lambda. The use of this suite help me with the majority of the heavy lifting and the deploy was very fast and repeatable to multiple regions and/or accounts. I had started with the manual processes but easily abandoned it once I found this tool.
The AWS Config RDK is considered to be in “Open Beta” right now so you may want to evaluate it for your purposes. What’s locked down in the Lambda role creation may be something you want to review for a production environment. Otherwise I recommend it for the speed of compliance it allowed me for something that was missing.
A future tool I plan to evaluate is Watchmen, which allows you to centrally manage Config Rules in one account instead of having to deploy them to multiple regions and accounts. This looks like an ideal solution to speed up where your rules are deployed saving you some time on making sure your environments are on lockdown.
