How to detect root hackers on AWS and prevent the launching of unauthorized EC2 instances

10 min readJan 30, 2022

They usually want to spin up as many EC2 instances for cryptocurrency mining as possible then make it tedious to remove. Let’s stop them.

We’ll be using AWS Eventbridge, Config, Systems Manager, Simple Notification Service, Input transformer and then wrap it in CloudFormation. We’ll start by discussing some security best practices first.

Section one: Detection

This application will identify any and all API calls made by the root user. It will also identify login attempts which is our target purpose.

Why are you writing this?

I made my first appearance at Amazon Web Services working as a Technical Customer Service Associate with the Trust and Safety department. I would review complaints of abusive AWS resources and forward legitimate ones to customers. Most of the time the customer had no idea it was happening and was not intentionally defrauding the company. EC2 instances can become compromised from many different angles, but today we’re concerned with the actual AWS account.I no longer work for that organization but the practices and experiences remain. While I am an AWS Employee my opinions are my own.

This post is dedicated to my friends and former colleagues at AWS Trust and Safety.

AWS Security Best Practices

AWS Accounts are extremely powerful sandboxes that have thick and tall steel walls with electrified barbed wire fences, but none of these are turned on by default. Despite obvious public messaging there are still many individuals that use open ports on virtual machines, never enable CloudTrail, never enable Multi-Factor Authentication and worst of all — they use their root account for everyday tasks. Below are some best practices as published by the AWS Security Blog.

Create a strong password for your AWS resources.
Use a group email alias with your AWS account.
Enable MFA now.
Set up AWS IAM (Identify and Access Management) users, groups and roles for daily account access.
Delete the root account access keys.
Enable CloudTrail in all AWS regions.

I have some of my own best practices to add to this list.

Enable CloudTrail and pipe it to a secondary AWS account for logging with different credentials.
Never use the root account unless you are making your first IAM user or deactivating your service.
Your root account email should not be a commonly used address on the internet.
If possible put the MFA device in a safe and forget it exists.

As many of you know the shared responsibility model we must reinforce it. Platform, applications, Identity and Access Management are your responsibilty as the customer. If someone guesses your password and runs your bill then it is your bill to pay. It is on you to maintain your security posture, but hopefully I can help you with that.

Special Considerations

Attacking an IAM user’s login requires a username, password and account number, however, attacking a root user is easier. All you need is an email address and you can likely brute force by guessing passwords. This is actually pretty easy when someone is using the same email address for everything on the internet. If your email gets stolen in a data leak it’s likely someone will enumerate it across the most popular websites. If you had a Ticketfly account in 2018 then your email address and site password were stolen and posted on the public internet. Now if you also had an AWS account that had the same email and password without MFA then it’s safe to assume you are compromised. The understanding here is that root’s can be accessed easily and that’s why they need to be locked and not used. I really suggest using an AWS oriented email address for your account and never use it for anything else. Enable MFA, make an IAM administrator that can be traced, enable CloudTrail and then let’s track the Root. If someone gains access to an IAM user you can cycle its credentials or destroy it much easier than you can with a root account.

Lets get to it then

Enable MFA on your root user now if you have not done so previously.
Make a IAM user with administrative capabilities and start using it when you need “root power.”
Proceed below.

Basically everything I do on AWS is infrastructure as code. I do not like replicating steps or re-referencing how I accomplish tasks. I’m going to post the CloudFormation Template and we’ll walk through it.

Download both of the templates with the below git command;

git clone https://github.com/drampil/aegis-combat-systems

First we need to start the template with a basic description. There is no need to edit in your own email address here as it’s prompted when you launch the template.

In case you are wondering about the word,

“Aegis” is defined by the Merriam-Webster dictionary as “the power to protect, control, or support something or someone.”

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  Welcome to Aegis Combat Systems. This Cloudformation template will set up root API alerts on your account.
Parameters:
  Email:
      Type: String
      Description: Simple notification service will send you an actionable confirmation.
      Default: xxxxxxxxx@xxx.xxx

The following resources will be created on your account and should not incur any costs.

Simple Notification Service Topic
Simple Notification Service Topic Policy
EventBridge Rule

AegisInstanceSystems:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Endpoint: !Ref Email
          Protocol: email

This section is self-explanatory. It creates a subscription to the email address provided to the template. This must be confirmed from your inbox after launching the template.

AegisTopicPolicy:
    Type: 'AWS::SNS::TopicPolicy'
    Properties:
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sns:Publish'
            Resource: '*'
      Topics:
        - !Ref AegisInstanceSystems

Next we create a Topic Policy so SNS has permission to publish from Eventbridge. This is normally automatic when done from the console, but policies must be added manually when doing IaC (infrastructure as code).

AegisCombatSystem:
    Type: AWS::Events::Rule
    Properties:
      Description: "root user detection system"
      EventPattern:
        detail-type:
          - "AWS API Call via CloudTrail"
        detail:
          userIdentity:
            type:
              - "Root"
      State: "ENABLED"
      Targets:
        -
          Arn:
            Ref: "AegisInstanceSystems"
          Id: "OpsTopic"

Above it the first section of the event. The Event pattern is the important thing here. In fact this can be changed to to any user, but we are specifically targeting the dangerous one. If we sent this to our email it would look like a mess of JSON and we don’t want that so lets check out input transformer.

InputTransformer:
            InputPathsMap:
              MFA: "$.detail.additionalEventData.MFAUsed"
              action: "$.detail.eventName"
              arn: "$.detail.userIdentity.arn"
              loginrate: "$.detail.responseElements.ConsoleLogin"
              mobile: "$.detail.additionalEventData.MobileVersion"
              region: "$.detail.awsRegion"
              sector: "$.detail.additionalEventData.LoginTo"
              sourceip: "$.detail.sourceIPAddress"
              time: "$.detail.eventTime"
              user: "$.detail.userIdentity.userName"
              userAgent: "$.detail.userAgent"
            InputTemplate: |
              "Aegis Secure Combat Systems has detected an Root user API event."
              "API call: <action>"
              "Source IP: <sourceip>"
              "User Agent Detected: <userAgent>"
              "ARN: <arn>"
              "Region: <region>"
              "Time(UTC): <time>"
              "-------------------------------"
              "Login Information if applicable"
              "MFA status <MFA>"
              "Mobile Version: <mobile>"
              "Sector: <sector>"
              "Status: <loginrate>."

In input path we dictate which key value pairs we want from the event and then pass it to the template inside of <actions> tags. These absolutely need to be dictated with double quotes on each line. Feel free to change which parameters you think you need for your own email deliveries.

I attached an example of a recent incursion attempt.

This is what your notification will appear as when a failed root attempt is made. User is only shown when an IAM user is being used.

The above attack indicates someone wanted to go straight into the EC2 dashboard to spin up instances in a region I don’t even use. I have no idea what password they attempted to use. Attempted passwords are not shared since we don’t know who could have possible access to logs. This system does not prevent these dashboard hackers — MFA, security focused user hierarchy and password policies will though. This template will just notify you of the attempt.

Section 2: Prevent unauthorized EC2 Instances

The first section was about monitoring root API calls to determine if someone was trying to access the root user. This section will handle how to actively prevent these hackers from creating EC2 instances on your account, root user or otherwise.

This is handled by utilizing special rules, policies and remediation orders by AWS Config and Systems Manager. When there is an instance state change (or launch) a special configuration rule will evaluate whether or not a special required tag is present on the resource. If it is not then an execution order is passed through Systems Manager to stop the instance within a minute of it being fully initialized.

If you are wondering why I’m stopping instances rather than terminating them I have that answer for you as well. Mining hackers like to spin up as many instances as possible and prolong their lifespan by adding termination protection. This will require you to manually disable each one. It’s much easier to stop them in their tracks and terminate them later. There is still risk involved here as EBS volumes cost money and must be removed. That is left as your responsibility at your leisure if this happens to you.

The combined template with sections 1 and 2 is located here:

https://github.com/drampil/aegis-combat-systems/blob/main/barrier.yaml

The secret tag set is established in the parameters. Just use the same one on any EC2 resources you create, or they’ll be stopped by the application.

The resources involved in this template are as follows;

SNS Topic
SNS Policy
EventBridge Rule
Config Recorder
Config Delivery Channel
S3 Bucket
Config IAM Role
Config Rule
Config Remediation Configuration
Remediation IAM Role
Automation Role Policy

To roll back this template you must remove the S3 bucket and it’s contents as it’s recording logs of configuration changes. I have added screenshots to demonstrate the operation in real time.

Two instances are created. Only one has the special name tag.

AWS Config Reports that one of the two resources is out of compliance.

Remediation action parameters inside of Config.

Systems manager prepares to execute a stop on the non-compliant instance.

Conclusion

Security is a zero day conversation at AWS, but it’s important to remember the shared responsibility model. It is your responsibility to take care of the security in the Cloud and all of your applications within it. I must also explain that not all hackers are interested in EC2. Some may want to load tons of movies into a CloudFront distribution and illegally stream them around the world. Feel free to modify this template to any resource type.

Full Source Code Below

AWSTemplateFormatVersion: "2010-09-09"
Description: >-
  Welcome to Aegis Combat Systems. Specify alarm point and secret resource tags key.
Parameters:
  Email:
      Type: String
      Description: Simple notification service will send you an actionable confirmation.
      Default: xxxxxxxxx@xxx.xxx
  tag1Key:
      Type: String
      Description: Secret Tag Keyname
      Default: Name
  tag1Value:
      Type: String
      Description: Secret Tag Value
      Default: Secret
Resources:
  AegisInstanceSystems:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Endpoint: !Ref Email
          Protocol: "email"
  AegisTopicPolicy:
    Type: 'AWS::SNS::TopicPolicy'
    Properties:
      PolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sns:Publish'
            Resource: '*'
      Topics:
        - !Ref AegisInstanceSystems
  AegisCombatSystem:
    Type: AWS::Events::Rule
    Properties:
      Description: "root user detection system"
      EventPattern:
        detail-type:
          - "AWS API Call via CloudTrail"
        detail:
          userIdentity:
            type:
              - "Root"
      State: "ENABLED"
      Targets:
        -
          Arn:
            Ref: "AegisInstanceSystems"
          Id: "OpsTopic"
          InputTransformer:
            InputPathsMap:
              MFA: "$.detail.additionalEventData.MFAUsed"
              action: "$.detail.eventName"
              arn: "$.detail.userIdentity.arn"
              loginrate: "$.detail.responseElements.ConsoleLogin"
              mobile: "$.detail.additionalEventData.MobileVersion"
              region: "$.detail.awsRegion"
              sector: "$.detail.additionalEventData.LoginTo"
              sourceip: "$.detail.sourceIPAddress"
              time: "$.detail.eventTime"
              user: "$.detail.userIdentity.userName"
              userAgent: "$.detail.userAgent"
            InputTemplate: |
              "Aegis Secure Combat Systems has detected an Root user API event."
              "API call: <action>"
              "Source IP: <sourceip>"
              "User Agent Detected: <userAgent>"
              "ARN: <arn>"
              "Region: <region>"
              "Time(UTC): <time>"
              "-------------------------------"
              "Login Information if applicable"
              "MFA status <MFA>"
              "Mobile Version: <mobile>"
              "Sector: <sector>"
              "Status: <loginrate>."

  ConfigurationRecorder:
    Type: "AWS::Config::ConfigurationRecorder"
    Properties:
      RoleARN:
        Fn::GetAtt:
          - "IamRoleForAwsConfig"
          - "Arn"
      RecordingGroup:
        AllSupported: true
        IncludeGlobalResourceTypes: true
  DeliveryChannel:
    Type: "AWS::Config::DeliveryChannel"
    Properties:
      S3BucketName:
        Ref: "S3BucketForAwsConfig"
  S3BucketForAwsConfig:
    Type: "AWS::S3::Bucket"
    Properties: {}
  IamRoleForAwsConfig:
    Type: "AWS::IAM::Role"
    Properties:
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AWSConfigRole"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: ""
            Effect: "Allow"
            Principal:
              Service: "config.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: "allow-access-to-config-s3-bucket"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "s3:PutObject"
                Resource:
                  - Fn::Join:
                      - ""
                      -
                        - Fn::GetAtt:
                            - "S3BucketForAwsConfig"
                            - "Arn"
                        - "/*"
                Condition:
                  StringLike:
                    s3:x-amz-acl: "bucket-owner-full-control"
              - Effect: "Allow"
                Action:
                  - "s3:GetBucketAcl"
                Resource:
                  Fn::GetAtt:
                    - "S3BucketForAwsConfig"
                    - "Arn"
  ConfigRule:
    Type: "AWS::Config::ConfigRule"
    Properties:
      ConfigRuleName: "ec2-required-tags"
      Scope:
        ComplianceResourceTypes:
          - "AWS::EC2::Instance"
      Description: "Aegis Config rule that checks whether your resources have the special tags that you specify."
      InputParameters:
        tag1Key: !Ref tag1Key
        tag1Value: !Ref tag1Value
      Source:
        Owner: "AWS"
        SourceIdentifier: "REQUIRED_TAGS"
    DependsOn:
      - "ConfigurationRecorder"
  RemediationForConfigRule:
    Type: "AWS::Config::RemediationConfiguration"
    Properties:
      Automatic: true
      ConfigRuleName:
        Ref: "ConfigRule"
      MaximumAutomaticAttempts: 5
      RetryAttemptSeconds: 60
      TargetId: "AWS-StopEC2Instance"
      TargetType: "SSM_DOCUMENT"
      TargetVersion: "1"
      Parameters:
        AutomationAssumeRole:
          StaticValue:
            Values:
              - Fn::GetAtt:
                  - "AutoRemediationIamRole"
                  - "Arn"
        InstanceId:
          ResourceValue:
            Value: "RESOURCE_ID"
  AutoRemediationIamRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "ec2.amazonaws.com"
                - "events.amazonaws.com"
                - "ssm.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AmazonSSMAutomationRole"
      Policies: []
  AutomationPassRolePolicy:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: "passAutomationRole"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Action:
              - "iam:PassRole"
            Resource:
              Fn::GetAtt:
                - "AutoRemediationIamRole"
                - "Arn"
      Roles:
        - Ref: "AutoRemediationIamRole"
Metadata: {}
Conditions: {}