Marwan
Marwan
Aug 4 · 7 min read

As organisations becomes more focused on cloud and realising benefits of the cloud operating model, migrations and builds accelerate, in most cases, putting operational and architecture best practices at risk.

In a cloud operating model, security cannot be an afterthought, especially when attack surfaces and vectors are constantly evolving to exploit vulnerabilities we never thought existed.

Compliance plays a major role in the cloud game and enforcing compliance can prove challenging when the organisation has not gone through the DevOps change mentality.

Organisations face challenged on-boarding newly recruited DevOps engineer, who, in the spirit of being agile, get into workload deployments -sometimes-too soon, risking exposing sensitive information and IP.

AWS Inspector

With regards to instance vulnerability assessment, AWS introduced Inspector, a product that often flies under the radar.

Inspector is an AWS native security assessment service that helps improve the security and compliance of instances and instance backed applications deployed on AWS. It automatically assesses applications and cloud compute instances for exposure, vulnerabilities, and deviations from best practices.

As any vulnerability assessment platform, it produces a detailed list of security findings prioritised by level of severity and triggers SNS topics (and Lambda functions).

The key differentiator is the ability to trigger a Lambda function post scans, which in this example, can be used to help an organisation automatically remediate issues.

At a high level, Inspector encompasses the below

In this scenario, a DevOps engineer attempts to deploy an EC2 instance (or a container, running on an EC2 instance) without checking which AMIs are the corporate approved \ baked ones.

The EC2 instance chosen is missing critical patches. However, the organisation has been ahead of the curve and implemented AWS Inspector to run frequent scans and implemented measures built-in to the CI\CD.

The organisation pipeline, has a scan stage “post CodePipeline deploy” that triggers a lambda at the end of the orchestration process, which launches a scan of the EC2 produced in this case.

The scan will trigger an SNS topic with findings to a Lambda and payload parsing CVE Findings.

Lambda will parse the output, and based on specific logic, can stop or remediate the instance.

The key pre-requisites for this is:

  • IAM Roles for Inspector, Lambda and SNS
  • Inspector agent deployed on the target EC2 instances.

First, create the SNS trigger:

aws sns create-topic \--name sns-inspector-sandpit \--profile "my cli profile"

A simple AWS Inspector Assessment targets configuration looks like the below:

It can be created via code as well using the below snippet

aws inspector create-assessment-target \--assessment-target-name \ CVE \--profile "my cli profile"

If you do not define targets specifically, the default behaviour is all EC2 instances in the account will be included in the scan.

Inspector Assessment configuration looks like the below in my example, where i am scanning using all the run packages for optimum security. The scan is searching for:

  • AWS Best practices implemented
  • Network reachability of the instance
  • Runtime behavior analysis
  • Common Vulnerabilities
  • CIS benchmarks

through CLI, it can be configured as such:

aws inspector create-assessment-template \--assessment-target-arn arn:aws:inspector:ap-southeast-2:************:target/0-sOsiHl9z \--assessment-template-name CVE-Template \--duration-in-seconds 180 \--rules-package-arns "arn:aws:inspector:ap-southeast-2:454640832652:rulespackage/0-D5TGAxiR" "arn:aws:inspector:ap-southeast-2:454640832652:rulespackage/0-Vkd2Vxjq" "arn:aws:inspector:ap-southeast-2:454640832652:rulespackage/0-FLcuV4Gz" "arn:aws:inspector:ap-southeast-2:454640832652:rulespackage/0-asL6HRgN" "arn:aws:inspector:ap-southeast-2:454640832652:rulespackage/0-P8Tel2Xj" \--user-attributes-for-findings key=Name,value=CVE \--profile "my cli profile"

When done, make sure to associate the assessment template with an SNS topic that triggers the lambda.

aws inspector subscribe-to-event \--event ASSESSMENT_RUN_COMPLETED \--resource-arn arn:aws:inspector:ap-southeast-2:**********:target/0-sOsiHl9z/template/0-3QpY4vZc \--topic arn:aws:sns:ap-southeast-2:************:sns-inspector-sandpit \--profile "my cli profile"

For the purpose of this test, i created an initial lambda that shuts down an instance and hardcoded an instance ID into it.

In a real life scenario, you want to pass the payload as CVE and remediate or shutdown on that basis.

My Lambda looks like the below:

import boto3
region = 'ap-southeast-2'
instances = ['i-*****************']
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
ec2.stop_instances(InstanceIds=instances)
print ('stopped your instances: ' + str(instances))

You do not have to stop the ec2 instance, you may choose to run an SSM command to patch \ update the instance in place and force a reboot, or, simply remove the Elastic IP if the instance is exposed externally.

Or feel free to create lambdas to address all of OWAS’s Top 10 if you know what you are looking for.

Make sure lambda is triggered by the SNS topic when you workout the payload you want to use for your lambda.

When the scan is triggered, your findings report will look like the below:

AWS provides a comprehensive code that will automatically patch Linux instances if CVEs are reported, see code below, found here. The only remaining item for you to workout is the lambda payload from the SNS trigger.

import boto3
import json
import logging
import datetime
ssm = boto3.client('ssm')
inspector = boto3.client('inspector')
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# quick function to handle datetime serialization problems
enco = lambda obj: (
obj.isoformat()
if isinstance(obj, datetime.datetime)
or isinstance(obj, datetime.date)
else None
)
def lambda_handler(event, context):logger.debug('Raw Lambda event:')
logger.debug(event)
# extract the message that Inspector sent via SNS
message = event['Records'][0]['Sns']['Message']
logger.debug('Event from SNS: ' + message)

# get inspector notification type
notificationType = json.loads(message)['event']
logger.info('Inspector SNS message type: ' + notificationType)
# skip everything except report_finding notifications
if notificationType != "FINDING_REPORTED":
logger.info('Skipping notification that is not a new finding: ' + notificationType)
return 1

# extract finding ARN
findingArn = json.loads(message)['finding']
logger.info('Finding ARN: ' + findingArn)
# get finding and extract detail
response = inspector.describe_findings(findingArns = [ findingArn ], locale='EN_US')
logger.debug('Inspector DescribeFindings response:')
logger.debug(response)
finding = response['findings'][0]
logger.debug('Raw finding:')
logger.debug(finding)
# skip uninteresting findings
title = finding['title']
logger.debug('Finding title: ' + title)
if title == "Unsupported Operating System or Version":
logger.info('Skipping finding: ' + title)
return 1

if title == "No potential security issues found":
logger.info('Skipping finding: ' + title)
return 1

service = finding['service']
logger.debug('Service: ' + service)
if service != "Inspector":
logger.info('Skipping finding from service: ' + service)
return 1

cveId = ""
for attribute in finding['attributes']:
if attribute['key'] == "CVE_ID":
cveId = attribute['value']
break
logger.info('CVE ID: ' + cveId)

if cveId == "":
logger.info('Skipping non-CVE finding (could not find CVE ID)')
return 1

assetType = finding['assetType']
logger.debug('Asset type: ' + assetType)
if assetType != "ec2-instance":
logger.info('Skipping non-EC2-instance asset type: ' + assetType)
return 1

instanceId = finding['assetAttributes']['agentId']
logger.info('Instance ID: ' + instanceId)
if not instanceId.startswith("i-"):
logger.info('Invalid instance ID: ' + instanceId)
return 1

# if we got here, we have a valid CVE type finding for an EC2 instance with a well-formed instance ID

# query SSM for information about this instance
filterList = [ { 'key': 'InstanceIds', 'valueSet': [ instanceId ] } ]
response = ssm.describe_instance_information( InstanceInformationFilterList = filterList, MaxResults = 50 )
logger.debug('SSM DescribeInstanceInformation response:')
logger.debug(response)
instanceInfo = response['InstanceInformationList'][0]
logger.debug('Instance information:')
logger.debug(instanceInfo)
pingStatus = instanceInfo['PingStatus']
logger.info('SSM status of instance: ' + pingStatus)
lastPingTime = instanceInfo['LastPingDateTime']
logger.debug('SSM last contact:')
logger.debug(lastPingTime)
agentVersion = instanceInfo['AgentVersion']
logger.debug('SSM agent version: ' + agentVersion)
platformType = instanceInfo['PlatformType']
logger.info('OS type: ' + platformType)
osName = instanceInfo['PlatformName']
logger.info('OS name: ' + osName)
osVersion = instanceInfo['PlatformVersion']
logger.info('OS version: ' + osVersion)

# Terminate if SSM agent is offline
if pingStatus != 'Online':
logger.info('SSM agent for this instance is not online: ' + pingStatus)
return 1

# This script only supports remediation on Linux
if platformType != "Linux":
logger.info('Skipping non-Linux platform: ' + platformType)
return 1

# Look up the correct command to update this Linux distro
# to-do: patch only CVEs, or patch only the specific CVE
if osName == 'Ubuntu':
commandLine = "apt-get update -qq -y; apt-get upgrade -y"
elif osName == 'Amazon Linux AMI':
commandLine = "yum update -q -y; yum upgrade -y"
else:
logger.info('Unsupported Linux distribution: ' + osName)
return 1
logger.info('Command line to execute: ' + commandLine)

# now we SSM run-command
response = ssm.send_command(
InstanceIds = [ instanceId ],
DocumentName = 'AWS-RunShellScript',
Comment = 'Lambda function performing Inspector CVE finding auto-remediation',
Parameters = { 'commands': [ commandLine ] }
)

logger.info('SSM send-command response:')
logger.info(response)

Follow us on Twitter 🐦 and Facebook 👥 and join our Facebook Group 💬.

To join our community Slack 🗣️ and read our weekly Faun topics 🗞️, click here⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts

Marwan

Written by

Marwan

Tech head, early adopter of all things cloud, DevOps\automation, Security and Infrastructure, but first and foremost, a proud father!

Faun

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade