Threat Hunting Using PolySwarm

Published in

PolySwarm

6 min readMay 22, 2019

Live & historical hunting features are available now!

(UPDATED, JULY, 30, 2019: PolySwarm now includes Threat Hunting capabilities accessible from your PolySwarm dashboard at polyswarm.network)

As of today, you’re able to leverage familiar YARA rules to query against PolySwarm incoming sample stream (live hunting) or against PolySwarm’s entire archive of samples* (historical hunting).

This is big news for researchers and incident response (IR) teams that have identified a threat and want to expand their knowledge of the threat by identifying variants of known-malicious samples.

Threat hunting features are exclusively available to Enterprise (and above) customers. Visit https://polyswarm.network today to see the PolySwarm Marketplace in action, then use the 30 Day Trial button to get started on your free Enterprise trial!

*Historical hunts search the last 6 months of samples. Currently, this covers the entire PolySwarm archive.

Rare photo of sharks using YARA rules to hunt fish, probably. Photo by Ibrahim Rifath on Unsplash.

Threat Hunting 101

So, you found some malware. Now you have more questions:

When was this malware first seen in the wild?
How well have AV engines / EDR agents detected this threat over time?
Are there variants to this malware?

PolySwarm’s threat hunting capabilities provide you with the tools to answer these questions, expanding your knowledge of the threat and helping you identify previously unknown threat variants.

Below is a crash course on using YARA to detect suspicious files as well as running YARA rulesets against PolySwarm’s historical database (historical hunts) and PolySwarm’s live feed (live hunts). UPDATE, July 30, 2019: In addition to Threat Hunting being supported by polyswarm-api, it is now also supported on https://polyswarm.network. The new UI makes it easy to add rule sets, and run historic and/or live hunts, all from your PolySwarm dashboard. (New UI pictured below.)

As an Enterprise customer, you’re allotted:

5 historical hunts (YARA ruleset executions) every 30 days. There is no limit to the number of YARA rulesets you can upload.
1 live hunt at any given time.

Live & historical hunt features can be used simultaneously. We’re working on enhancements to https://polyswarm.network that will allow you to visually manage your hunts!

These are not hard limits— please reach out if you’d like to chat about higher-volume usage and view our account options for more details.

You writing precision YARA rules & hunting on the PolySwarm marketplace. Photo by Will Porada on Unsplash.

Test Locally

Before you deploy your rulesets on the PolySwarm marketplace, we highly recommend that you test your rules locally. Testing locally ensures:

Your ruleset matches what you intend — and doesn’t match what you don’t intend. Results for any given ruleset are capped at 1000.
Your rulesets are sufficiently performant*.

*We reserve the right to throttle execution on overly broad or malicious rulesets that consume inordinate compute resources.

Install YARA

YARA installation instructions vary by platform. Please consult YARA’s official documentation for instructions on your platform.

Once installed, run YARA to verify your installation:

$ yara --version
3.10.0

YARA Matching Against the EICAR Test File

We’ll use Airbnb’s open source library of YARA rulesets to detect the EICAR test file.

Take a look at Airbnb’s YARA ruleset here. Their ruleset contains 2 rules:

eicar_av_test: the textbook definition of EICAR
eicar_substring_test: a looser interpretation of EICAR that simply substring searches the file for the EICAR string

Let’s run the entire ruleset against the canonical EICAR test file and against a file that is definitely not the EICAR test file:

$ wget   https://raw.githubusercontent.com/airbnb/binaryalert/master/rules/public/eicar.yara$ wget http://www.eicar.org/download/eicar.com$ yara eicar.yara eicar.com 
eicar_av_test eicar.com
eicar_substring_test eicar.com$ echo "this is not eicar" > not-eicar$ yara eicar.yara not-eicar

YARA will print one line of output per rule matched against the file(s) you specify. In the above, both the eicar_av_test and eicar_substring_test in Airbnb’s eicar.yara match against eicar.com. Neither match against our not-eicar file.

Protips

When developing your own YARA rulesets, optimize for precision and speed. You probably want to detect malware variants — and not benign files. There are many articles on optimizing your YARA rulesets for speed — avoid heavy features like loops and regex whenever possible.

PolySwarm Live Hunting

This fox is your YARA ruleset, on the lookout for new malware variants. Photo by Ray Hennessy on Unsplash.

PolySwarm’s “live hunting” feature allows you to install a YARA ruleset that is run against all incoming samples to the public PolySwarm marketplace.

There are only 2 steps:

Install your ruleset.
Query against your installed ruleset to retrieve any matched samples.

Okay, we kind of lied: you also need to have polyswarm-api installed. But you already had that installed, right?

Installing Your Live Hunt Ruleset

For convenience, set your POLYSWARM_API_KEY environment variable to match your API key:

$ POLYSWARM_API_KEY="<YOUR_API_KEY>"

Installing a live ruleset is as simple as:

$ polyswarm live install eicar.yara
Successfully submitted rules, rule_id: <YOUR_RULE_ID>

Write down your rule_id- we’ll use this to query for results!

Querying Against Your Ruleset

Gathering results is as simple as querying against the rule_id:

$ polyswarm live results --rule-id <YOUR_RULE_ID>
Found 24 samples in this hunt.
Match on rule eicar_substring_test
File cabf86b068c066b70e7defead09d252832490bc7fe8a1b14093b7e86560566dc
    File type: mimetype: application/x-dosexec, extended_info: PE32 executable (GUI) Intel 80386, for MS Windows
    SHA256: cabf86b068c066b70e7defead09d252832490bc7fe8a1b14093b7e86560566dc
    SHA1: e56b040814ecb0426b61319d4ac1433b7cb263c4
    MD5: 41faf9d72f3b3a588025f2c02011a51b
    Observed countries: 
    Observed filenames: cabf86b068c066b70e7defead09d252832490bc7fe8a1b14093b7e86560566dc
...

All polyswarm-api commands support the --fmt json option — which will deliver results in a machine-consumable JSON format.

PolySwarm Historical Hunting

PolySwarm samples, neatly organized for your hunting pleasure. Photo by Samuel Zeller on Unsplash.

PolySwarm’s historical hunting runs your YARA ruleset against the last 6 months of PolySwarm submissions.

Again, only two steps:

Execute the historical hunt.
Query the hunt for results.

Executing a Historical Hunt

Starting your historical hunt is as simple as:

$ polyswarm historical start eicar.yara 
Successfully submitted rules, rule_id: <YOUR_RULE_ID>

Querying for Results

Historical hunting takes time. You’re able to query for results at any time, but results may be incomplete if the hunt is still in progress. Future releases of polyswarm-api will notify when a hunt is in progress. For now, we recommend waiting at least an hour before considering the hunt complete.