BinaryAlert: Real-time Serverless Malware Detection

Today, Airbnb is proud to announce the open-source release of BinaryAlert: a serverless, real-time framework for detecting malicious files. BinaryAlert can efficiently analyze millions of files a day with a configurable set of YARA rules and will trigger an alert as soon as anything malicious is discovered! Organizations can deploy BinaryAlert to their private AWS account in a matter of minutes, allowing them to analyze internal files and documents within the confines of their own environment.

Example Use Cases

• Enterprise alerting: Find malware infections on laptops and servers by analyzing executable binaries.
• Email attachments: Identify malicious documents sent to your organization.
• User uploads: Detect when a user uploads a malicious file to your application.
• Security research: Quickly test new iterations of YARA rules against your own private collections of files using built-in retroactive analysis.

Leveraging YARA Rules

YARA is a powerful pattern-matching tool for binary analysis. Unlike a simple hash-based signature, YARA rules can classify entire families of malware according to common patterns. As YARA sees more widespread use within the security community, we wanted to find a way to leverage YARA rules to scan for malicious files across our entire organization.

Other security tools support YARA rule integration, but we could not find a private, low-cost, scalable, batteries-included solution that was easy to deploy and maintain. For example, VirusTotal supports YARA rule matching against file submissions, but it is a public service and not designed for analyzing internal files and documents with varying levels of confidentiality and sensitivity.

BinaryAlert is our solution: a serverless framework for scalable YARA analysis that you can deploy in your own AWS account cheaply and easily!

BinaryAlert Features

Real-time

Time is of the essence when responding to a threat, so BinaryAlert analyzes files almost immediately after being uploaded. In our deployment, analysis is usually completed within 1–2 minutes of file discovery.

Retroactive Analysis

When the YARA ruleset is updated, BinaryAlert will automatically re-analyze your entire file corpus to find any new matches. This allows you to identify threats in the past with information you receive in the future, and it provides an easy mechanism for testing the efficacy of new rules.

YARA Rules Included

BinaryAlert includes several of our own YARA rules and also makes it easy to clone rules from other open-source projects like YaraRules. Each included rule has been tested against more than 2 million executable binaries from Airbnb’s environment to verify its effectiveness.

Serverless Design

Like StreamAlert, BinaryAlert utilizes AWS Lambda functions for analysis instead of a traditional server. This provides a number of benefits, including stronger security (no servers to patch or maintain) and lower cost (pay only for what you use).

Infrastructure-as-Code

Again following StreamAlert’s example, BinaryAlert uses Terraform to manage its underlying infrastructure. This considerably simplifies the deployment process: a single command creates and configures all of the necessary AWS components. Deployments are simple, safe, and repeatable.

Metrics and Monitoring

BinaryAlert uploads custom metrics about its processing throughput and automatically creates CloudWatch alarms to monitor the health of your deployment. Alarm thresholds are easily configurable to accommodate different workloads.

Architecture

AWS Services

BinaryAlert utilizes several different AWS services:

• AWS CloudWatch: Stores logs and metrics and monitors service health.
• AWS DynamoDB: Stores YARA match information.
• AWS IAM: Manages permissions with role-based access control.
• AWS Lambda: Executes serverless computation (e.g. YARA analysis).
• AWS S3: Stores files uploaded to BinaryAlert for analysis.
• AWS SNS: Sends notifications for YARA matches and CloudWatch alerts.
• AWS SQS: Provides a queue to keep track of files awaiting analysis.

Fortunately, Terraform automatically configures all of these services so you don’t have to!

Analysis Flow

1. The organization collects files and delivers them to their BinaryAlert S3 bucket. Files of interest could include executable binaries, email attachments, documents, etc.
2. Every file uploaded to the S3 bucket is immediately queued for analysis.
3. A dispatching Lambda function runs every minute, grouping files into batches and invoking up to dozens of analyzers in parallel.
4. Each analyzer scans its files using a list of pre-compiled YARA rules.
5. YARA matches are saved to DynamoDB and an alert is sent to an SNS topic. We use StreamAlert to dispatch these alerts, but other organizations can instead consume the alerts via email or any other supported SNS subscription.
6. For retroactive analysis, a batching Lambda function enqueues the entire S3 bucket to be re-analyzed.
7. Configurable CloudWatch alarms will trigger if any BinaryAlert component is behaving abnormally. This will notify a different SNS topic than the one used for YARA match alerts.

Future Work

A future version of BinaryAlert will add support for file pre-processing, including decompressing and unpacking, prior to the YARA analysis. We can leverage AWS Step Functions to better orchestrate the different stages of the pipeline.

Airbnb is also committed to supporting the YARA community. We will continue to contribute our own YARA rules as well as to source, test, and provide feedback on rules from other open-source projects.

Concluding Thoughts

Serverless architectures have proven effective for security tools due to the lower cost, simpler management, and scalability associated with serverless designs. BinaryAlert represents our next contribution in the open-source serverless security space, allowing others to more quickly and easily detect malicious files within their own organization.

Visit github.com/airbnb/binaryalert to get started!

Security Team @ Airbnb

Contributors

BinaryAlert is brought to you by:

• austinbyers (architect and developer)
• mime-frame (concept, design and code review, YARA rules)
• fusionrace (YARA rules)
• ryandeivert, jacknagz, and chunyong-lin (code review)