Help! My AWS Server Has Been Hacked!

You’re sitting at your desk, enjoying your morning coffee, getting up to speed with the day’s latest news. You touch base with some colleagues and then quickly scan through your email inbox. You spot an email from Amazon Web Services: you open it and find out that some of your EC2 instances are being used to spread malware via your website.

Guess what? You’ve been hacked.

Bad actors always pose a threat to infrastructure on the internet. They might use your systems to spread viruses, phish for credit card information, or simply deface a website. Whatever the end result, hacking can have an impact on your business and on your pocket. If someone gains access to your AWS account, they could potentially spin up hundreds of EC2 instances, for example. As AWS uses a pay-as-you-go model, the costs to you can rack up very, very quickly.

It’s vital that you secure your AWS infrastructure and, despite what you might think, it’s relatively easy to take steps to heighten your security. The overall aim is to keep costs low and to recover quickly.

First thing’s first: secure your login. Enable multi-factor authentication on your root account. Go and do that right now before you read any further. Yes, really. Next, create an IAM user (also with MFA authentication) in your account, and use this new user for your day-to-day AWS use. Avoid using the root account if at all possible.

All done? Great. Let’s get into working through an incident response plan!

For this, I will use a compromised EC2 instance as the example. AWS have a number of recommended steps that you carry out should a security incident take place. I like to put them into three main groups:

• Observation
• Action
• Review

These are quite self explanatory. In Observation we set up tools to watch out for intrusions. In Action we can find the steps carried out after — or during — an attack. Finally, the Review phase is for looking at procedures and processes and ensuring they meet the requirements to secure a system and to react to a security incident.

Observation

Logging activity is the key message here. By using the resources outlined below you can prepare yourself well against a possible security attack.

Preparation

Encrypt your data, where possible, both in transit and at rest. AWS provides a large variety of encryption methods as well as a very useful Key Management Service (KMS). Once encrypted, your data will be protected, even if someone does somehow manage to get hold of it outside your system.

Encryption is the only tool we have to protect your data. “Dance like no one is watching. Encrypt like everyone is.”

— Werner Vogels, AWS CTO

Take backups and snapshots of everything you can: EBS volumes, EFS volumes, and RDS instances. In the event of a failure — not just an attack — you can use these to restore your systems to working order.

Monitoring and Detection

Use CloudWatch to send your application’s logs to AWS. If an attacker gains access to an application server, they can easily delete logs to cover their tracks. Once logs are sent to AWS, this becomes more difficult to do.

CloudTrail is used to log API activity and is very useful for watching logins and calls to AWS services. CloudTrail and CloudWatch can be used in conjunction to send you a message if an alarm is triggered.

VPC Flow Logs allow you to monitor traffic entering and leaving your VPC. This is a vital source of information for many attacks, particularly for Denial of Service attacks on your infrastructure.

AWS GuardDuty monitors your EC2 infrastructure and alerts you to threats on your system, such as Bitcoin miners, DoS attacks, and so on. It ranks them in severity, and you can also whitelist (and blacklist) known IPs for better control of your system. GuardDuty is one of my favourite security tools on AWS as it’s so easy to use and highly informative.

Finally, AWS Config allows you to monitor changes to your infrastructure. For example, if an EC2 instance is resized from, say, a t3.small to a t3.large, this will be recorded in Config. Config also allows you to audit your infrastructure: you can set up rules where you can say, for example, that instances with a “production” tag can only be a c5.medium or a c5.large. If Config spots an instance outside of the allowed values, it will let you know. Config is great for monitoring changes to your infrastructure, whether that be a resizing event or the creation of new resources, and is extremely useful from a security perspective.

Action

Okay, this is the business bit. You’ve just realised you’ve been hacked. What now?

Isolation

The first step is to isolate the compromised instance, and stop traffic entering or leaving the instance. To do this, remove the existing security groups from the instance, and add a single one that has one egress rule, allowing all traffic to go to 127.0.0.1 only. (There’s no such thing as a “deny” on a security group, but you can allow traffic to 127.0.0.1, which is the instance itself, thereby stopping traffic going elsewhere.)

Analysis

Immediately take a snapshot of the compromised EC2 instance’s EBS volume. You can use this later on for further analysis, if required. Do not shut down the instance.

Start up a new EC2 instance: this will be your forensics instance. When started and configured, add a rule to the security group you created above to allow access only from the new instance. You can now use the new instance to connect to the server and carry out a forensic analysis.

As well as taking a snapshot of the EBS volume, you should take a memory dump of all processes to see what’s doing what. Finally, take a copy of relevant log files (if you don’t have have them stored in CloudWatch or elsewhere).

Use this gathered information to try to determine what went wrong and how the instance as compromised. You will use this information later on in your Review phase.

Termination

Finally, terminate the compromised instance.

Restoration

Recreate your EC2 instance and configure it. CloudFormation is an excellent tool that allows you to describe your infrastructure in either JSON or YAML. If you have a template for your infrastructure, you can use it to rebuild it with CloudFormation. The AWS Cloud Development Kit (CDK) and other tools such as Terraform can also assist with this.

You may wish to attach a previously snapshotted EBS volume, or retrieve some data from other backups, at this point.

Review

By now, the attack has been stopped and your system is back up and running (hopefully!). Take time to review what happened and how it happened using the information you gathered during the analysis. Let this then inform your overall process for dealing with security incidents: ask yourself if any changes can be made to your infrastructure or your processes to help add security to your system.

Creating an Incident Response Plan

As you can see, knowing what to do in the event of an attack is vital: it can help keep the costs resulting from an attack down, and it can allow you to get back to a healthy system very quickly.

In your plan, write down what your vulnerabilities are and what the threats are. Like the example above, the vulnerability could be “instance may be running outdated and insecure software” and the threat could be “a hacker could take advantage of vulnerable software and use it to get root privileges on the instance”. Take your time to go through them.

Next, write down what tools you use and measures you take to protect your system already. You’ll find useful tools in the Observation section above, but you may do things differently.

Next, write down the steps you need to carry out in the event of an attack. Make it succinct and easy to read, but make sure you put in detail where required. Ensure your engineers have access to this document and understand it.

At this point you can do a practice run. In the industry, this is called a “game day” and is part of a concept known as Chaos Engineering (I love that phrase). Set up a copy of your environment (ideally in its own account, away from your “real” systems) and “hack” it: do something that should trigger an alert and see how your engineers react. Use this information resulting from this to iterate over your response plan and improve it.

Test, test, test. Break everything to see how your systems respond. Game Days and chaos engineering can be invaluable.

— Werner Vogels, AWS CTO

Finally, and this is an important part, make sure your incident response plan is in constant review. As technology changes, software is updated, and your infrastructure grows, your response plan will eventually become useless.

Hopefully I’ve given you some good tips and advice to get started in securing your AWS infrastructure!