Part 2: AWS Monitoring Case Studies

Ashish Kurmi, Kaibo Ma, and Ankit Kumar, Security Engineering

Editor’s note: This is Part 2 in a series. Part 1 can be found here.

Uber’s multi-cloud environment combines the advantages of both public and on-premise infrastructure. It allows our teams to quickly innovate with a resilient, performance, and scalable foundation. In Part 1 of this series, we explained the framework behind our use of Hammer, an open source AWS security monitoring tool.

In this post, we share detailed case studies to illustrate some of the ways we use Hammer at Uber.

Developer education/Minimize Mean Time To Detect

One of our ongoing objectives as a security team is to maximize productivity by empowering individuals to be self-sufficient. We understand that developers are not necessarily security experts and they may accidentally make security mistakes. To educate them while minimizing mean time to detect/remediate, we share real-time feedback with cloud users as security issues are introduced and remediated. The platform tries to attribute security issues to CloudTrail events. it notifies relevant individuals in real-time so that they can learn from their mistakes and take corrective actions promptly. Here is an example scenario:

• A developer while trying to get access to an EC2 instance adds an inbound rule exposing the SSH port on the internet.
• Hammer picks up the CloudTrail event for this change and kicks off all Hammer rules applicable for security groups. The security group unrestricted access check detects that the port has been exposed on the internet and creates a security issue. Hammer then notifies CMON.

• CMON creates a Jira ticket with all relevant information. Using the AWS principal naming convention used by our internal AWS user access management system, it maps the AWS principal back to an Uber user and assigns the ticket to them.
• CMON onboards the ticket onto uNotify(a centralized notification system at Uber that leverages several communication channels such as email, slack, etc. and management escalations to get the vulnerabilities remedied in a timely manner) so that the service can follow up with the developer.

• Developer receives email/Slack notifications in real-time about the vulnerability. They access the runbook included in the Jira ticket and deploy one of the remediation options included in the runbook.

• Hammer picks up the CloudTrail event for the remediation action, verifies the security fix, updates its database and notifies CMON.
• CMON resolves the Jira ticket and leaves a comment.

This scenario is also applicable on vulnerabilities introduced by automated systems. In this case, as the monitoring platform cannot map the AWS identity back to a human user, it would only notify account security contacts.

Unauthenticated Vulnerability Scans

Since most cloud environments provide ephemeral endpoints by default, it’s challenging to use unauthenticated vulnerability scanners to scan a fixed set of IPs/IP ranges. Using our monitoring platform, we built a solution to scan all public endpoints in AWS as well as GCP using a third-party endpoint scanning solution and operationalized these findings like other cloud security vulnerabilities. For AWS, the solution leverages AWS Config Aggregator to discover all public endpoints. To discover all GCP public endpoints, the solution utilizes CSCC Asset Inventory. After discovering all public endpoints, the system uses third-party APIs to scan these endpoints and find updates. It then operationalizes these updates.

Acknowledgments

We’d like to thank the following:

• Dow Jones Product Security for all the collaboration with Hammer
• AWS Enterprise Account and Support Team for providing guidance throughout the process