Integrating Shuffle with Virustotal and TheHive — Open Source SOAR part 3
Misalignment between practitioners and leadership in Infosec is normal. Leadership may decide to buy a product or service the practitioners are not fond of, but demand an implementation. That means you need to integrate it fully into your processes and procedures. How do you do that however? It’s a hard issue and usually takes a lot of time and effort. But what if it doesn’t have to?
Shuffle gives you options to explore the platform from a programmatic point of view, whether you’re technical or not, and gives access to open standards to build implementations in minutes rather than months or years. In this post we’ll take a quick look at how one would implement a new service into Shuffle.
This is the third part of a series about Shuffle. Our previous post was about getting started, where we had a look at basic workflow, passing values between nodes and data usage. This post is a continuation which explains the app creation and implementation process in the form of a custom Virustotal app. This is an important step, as you can’t have accessibility without the right integrations. If you haven’t already, check out the previous posts here:
- Introducing Shuffle — an Open Source SOAR platform
- Getting started with Shuffle
- Integrating Shuffle with Virustotal and TheHive
- Real-time executions with TheHive, Cortex and MISP
- An advanced workflow walk-through
- The (not so distant) future of Shuffle (Mitre Att&ck, dynamic dashboards, asset management, search-ability, KPI’s and more)
To jump straight into app creation, check out the documentation: https://shuffler.io/docs/apps.
Services come and go, whether you’re a small SME or a huge enterprise. Change is the only constant. Our focus for the next ten minutes is to get through the process of creating an app for Shuffle and integrating it. The idea is to continue the workflow created in the previous post and build upon it with Virustotal and TheHive functionality. As Virustotal is not a part of Shuffle outside of our OpenAPI specification, we’ll have to start by creating the app itself. These are the steps we’ll follow.
- Create an app for Virustotal with the ability to get an IP report
- Add the Virustotal “Get IP report” action to our Workflow
- Set up TheHive (if you haven’t already)
- Create alert in TheHive with history of our IP from Virustotal
1. Create a Virustotal app
Assuming you’ve already got Shuffle installed, we’ll move on to the app selection at https://localhost:3443/apps. We’re presented with a view to create, edit, delete, search, download and everything else related to apps. This place has access to everything we need, including the ability to download apps from a github repository, whether public or private (upper right of the image below), generate an app from OpenAPI (bottom left) or create them from scratch. In our case, since Virustotal doesn’t have a public OpenAPI spec, we will be using the “Create from scratch” function.
PS: If you want to skip the creation process, either search “virustotal” in the apps searchbar in Shuffle and activate it or click “Create from OpenAPI” and use the virustotal.yaml file here.
First things first: we need to understand what kind of authentication Virustotal uses and what their API URL is. The most used API authentication options are Bearer auth (specific header), Basic auth (username & password), API-keys (custom header or query) and Oauth2 (special, multi-layer). To find the API documentation for a specific platform, start by Googling or going to their website. In Virustotal’s case, it’s in their header bar.
Here’s what Virustotal says about authentication.
This tells us we have to use a custom header, meaning we should use an API key. Next, we have to find the URL. To find it, we’ll scroll to the function we’re building, which has an example including the full URL. Here’s a highlight of the function we’ll be creating, which returns data about an IP address.
Going back to Shuffle, we now have the information necessary to create an app for Virustotal (BASE URL & Authentication scheme). I’ve taken the liberty to add an image and a name for Virustotal as well in this one.
Take note that the BASE URL is optional to make it possible to have on-premise hosting of custom apps. Authentication is a drop-down you can choose from, and in our case, we chose “API key”, before plotting in “x-apikey” as described in their documentation.
With the first step out of the way, let’s create our first ACTION for the APP. Scroll down a bit further and click the “NEW ACTION” button.
With the view of a new action open, the next step is to fill out specific information for the action “Get IP report”. Fill in the name and optionally a description, before copying the following:
curl --request GET \
--url https://www.virustotal.com/api/v3/ip_addresses/{ip} \
--header 'x-apikey: <your API key>'The command uses curl — a heavily used command line tool to make network requests (e.g. HTTP). I won’t be diving too deep into how curl or HTTP works, but the essence is that it’s a way for different software to talk to each other. Shuffle has support for CURL parsing, meaning we can use this command directly to make our action.
Creating an ACTION in short:
- Give it good name
- Paste the CURL command in the URL field
- Hit the SUBMIT button!
With the creation process of our first action finished, it’s time to save. By clicking the big red (orange) save button, Shuffle will create three things: An OpenAPI specification, Python code with the APP SDK and a finished Docker image. This might take up to a minute, but you can start the workflow modification right away.
With the creation process done, we should now test our app. Testing frequently is the best way to avoid inherited problems when introducing new parts to workflows. Having created a new workflow, we start by dragging the Virustotal app into the workflow view, selecting it, before we fill the “apikey” and “ip” arguments (as seen in the image below).
To handle api keys or re-usable data, the best way is to use workflow variables. To find your API key for Virustotal, sign in before selecting “API key” from the upper right corner.
Save and execute, before checking the results. Remember that the result might be large depending on the chosen IP. If there’s any issue, reach out to me and I can hopefully help.
2. Add the Virustotal “Get IP report” action to our Workflow
With testing out of the way, we can now move on to the real workflow, the one we created in part two: “My first workflow”.
Here, we start by adding the node, before connecting it to the last node — “Print IP”. The goal here is to use our own IP, passed by the “Get URL” node, and search for it in Virustotal. We do this by writing $Get URL.ip, as the data returned from the “Get URL” node is in the form of {“ip”: “your_ip”}.
To test, we manually execute the workflow with the execution argument {“url”: “https://api.ipify.org/?format=json”} (PS: I’ve changed the URL we get our IP from, as the other one failed regularly). The result is JSON formatted data from the Virustotal node about our IP.
Having created an app for and retrieved data from Virustotal, we’re now ready to use the data. The next steps and blog posts move our focus away from basic usage of Shuffle and towards general integrations, and real use-cases.
3. Set up TheHive (skip if you got a test instance)
In this step we’ll quickly set up TheHive, grant our user access to create alerts and grab our API-key for later usage.
TheHive is well documented, and can be installed in many ways. Copy the following into a file called “docker-compose.yml”, and then type “docker-compose up -d”. This will take a few minutes depending on your network speed.
version: "2"
services:
elasticsearch:
image: elasticsearch:6.8.8
environment:
- http.host=0.0.0.0
- discovery.type=single-node
ulimits:
nofile:
soft: 65536
hard: 65536
cortex:
image: thehiveproject/cortex:latest
depends_on:
- elasticsearch
ports:
- "0.0.0.0:9001:9001"
thehive:
image: thehiveproject/thehive:latest
depends_on:
- elasticsearch
- cortex
ports:
- "0.0.0.0:9000:9000"
command: --cortex-port 9001- With the docker-compose execution finished, go to http://localhost:9000 (or the host you chose to install it on).
- Click “Update database”
- Set up an admin user and log in
- Click Admin > Users in the upper right corner
- Choose your user (or create a new one), and give it access to create alerts.
6. Copy the API key and save it as a variable in Shuffle
PS: If you run into errors related to Elasticsearch and this is a test instance, run the following to remove common errors:
docker exec -u 0 -it thehive_elasticsearch_1 curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'; curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'4. Create alert in TheHive with Virustotal data
With our API key for TheHive, we’re now able to create an alert. As Shuffle already has an app for TheHive, the only thing missing is to add our “Create alert” action to the Workflow. Drag it in from the left side, before connecting it to the Virustotal node.
Note about TheHive: type, source and sourceref need to be a unique combination every time. In my case, my external IP changed frequently enough, so I decided to use my own IP as a reference (sourceref): whois-$Get URL.ip. This has a good side effect as well: If my IP changes, it will generate a new alert on a schedule.
The latter fields are optional, but are very useful if not vital to analysts. Title and Description are heavily used to help decide what to do with an alert. Here’s what I used in my optional fields, but you can put anything you want.
title: Info for $Get URL.ipdescription: * IP owner: $Virustotal v3_1.data.attributes.as_owner \n * Country: $Virustotal V3_1.data.country \n* Reputation: $Virustotal V3_1.data.attributes.reputation \n* Whois: $Virustotal V3_1.data.attributes.whois
Executing leads to a single new alert as can be seen below. To make sure it creates a new alert every time our external IP changes, we need to re-enable the scheduler. I set mine to run every 5 minutes (300 seconds), leading to an alert in TheHive daily, as that’s the duration of my IP lease. If you want to continue expanding this build, I’d suggest checking out the existing security definitions, including our complete OpenAPI specification of Virustotal.
As you might have noticed, the first step was way longer than the latter three as it was the only “new” step. The latter three were all about workflow edition and installation, while the first one tried to explain a new concept. By knowing how you can create an app from a curl command or manually in minutes, you can implement workflows for your resources in days rather than months or years of proprietary in-house development time.
Whenever OpenAPI or other standards are used, it makes things simpler, more understandable and last and most important; you’re not bound to the technology itself. OpenAPI from Shuffle can be imported, exported, generated etc. and will (together with JSON API) become the de-facto standard for Rest API’s in the security industry for the next technology generation. InfoSec/IT automation is often done individually by developer “unicorns”, and isn’t even close to the point we can get to if we work together.
Next up we’ll be continuing our work with TheHive, focusing on real-time data transfer (Webhook / Message queues) and real integrations rather than fictive examples. We’ll also take a quick look at Shuffle for MSSP’s, potential multi-tenancy and other triggers.
I welcome you to further explore Shuffle, try it out and build on it. It can’t survive without a community, and these posts are here to kickstart it. That’s why I’ve now released the first of many workflows to Github that can be downloaded:
If you didn’t do it already, please star the project on Github, follow me, clap (up to 50 times!) and share this blog series with anyone interested. It would help a lot with reach.
Please reach out in any way possible if you’re interested. Here’s a simple site as an overview of what I do:
