Real-time executions and IoC’s with Shuffle, TheHive and MISP — Open Source SOAR part 4

Automation is useful to everyone — especially in Cyber security. Leadership wants their KPIs and graphs. How do you produce them? Automation. Managers want to know how their team is performing. How can you make this easier for them? Automation. Analysts don’t want to see the same old alerts. How do you manage fatigue? Automation.

All these are solvable tasks with a few clicks in Shuffle. How? Read on.

This is the fourth part of a series about Shuffle and automation. Previous posts have focused on understanding and using Shuffle in a general sense of the word, with this iteration being about real use-cases for IT and infosec practitioners. New concepts introduced are conditions, loops and webhooks

1. Introducing Shuffle — an Open Source SOAR platform
2. Getting started with Shuffle
3. Integrating Shuffle with Virustotal and TheHive
4. Real-time executions with TheHive, Cortex and MISP
5. An advanced workflow walk-through
6. The (not so distant) future of Shuffle (Mitre Att&ck, dynamic dashboards, asset management, search-ability, KPI’s and more)

The workflows related to this blogpost can be found here:

frikky/shuffle-workflows

With our previous installment being an introduction to the scheduler, the app creator and passing data between nodes, we are ready to move into some real use-cases. A big reason for the existence of Shuffle is the speed to automate something new. Speed to develop isn’t the only factor however, but also the reaction speed of your automation software.

“Real-time” in security environments often means Pub/Sub, Kafka or Webhooks — not just pulling on a schedule. Keeping this in mind, we will focus on how webhooks work in Shuffle, again by using TheHive.

I wrote a post on the same subject a while back using WALKOFF (development is discontinued). This post will follow some of the same steps, including TheHive webhook setup. Here’s the use-case we’ll tackle:

1. Get IoC’s (Indicator of Compromise) from random text (execution argument)
2. Casting: Create a case in TheHive
3. Condition: Were any IoC’s found?
4. Trigger: connect TheHive webhook with Shuffle
5. Loops: Add the IoC’s as artifact to TheHive case
6. Condition: Check whether TheHive Webhook is artifact
7. Search for the indicator in MISP using Cortex

This is both an interesting and useful workflow, as you often have a file or bulk of text from a report or a list of indicators you want to look for in your environment. Let’s get into it.

1. Parse IOC’s from the execution argument

Before we start, we have to make sure to update Shuffle and our existing apps. With the latest from Shuffle, we can jump straight into workflow creation. With our goal being to parse IoC’s we’ll have to find a way of doing that.

As Shuffle becomes more developed, so will its apps. Right now, there are two apps that are frequently used for security-relevant use-cases: Shuffle Tools and Testing with the latter being used in all our previous posts.

The newly added Shuffle tools. Icon is temporary.

In the case of IoC parsing, we’ll make use of the Shuffle Tools app. Shuffle Tools was created to help with security related tasks and because of Shuffle’s modularity, will be updated frequently with new improvements. The Shuffle Tools has an action called “Parse ioc”, which takes two arguments. The first one, input_string, is the text to parse from. But where should we read it from? In the previous blogpost, you might remember that we used the “execution argument” as an option. It fits our purpose very well, as the use-case is to take any random text and have it auto-analyzed.

Lets drag the app into the workflow view before selecting the “Parse ioc” action and setting the “input_string” argument to “Execution argument” ($exec) by clicking “Data from previous action”. Save, then write something (domain, hash, url, ip etc.) and execute.

Using the execution argument to return {“data”: “1.2.3.4”, “data_type”: “ip”}

2. Create a case in TheHive

In the last post, we created an alert in TheHive. This time around, we’ll be creating a case. The only difference between the two is that a case doesn’t require unique information as alerts do. To differentiate between cases however, and to learn about casting, we’ll be making a title that contains the amount of IoC’s from the Shuffle Tools_1 node:

Analyzing 3 observables from Shuffle

To make this title we’ll need a way to count the amount of IoC’s found from the first node. How do we do that though? From previous blogposts we know that it’s possible to get a previous nodes’ value by writing $ followed by the nodes name. Shuffle has many “hidden” features (visualization will improve over time) similar to $, including the ability to cast values, meaning to change a value into another kind of value. In this case, we’ll be using length(). Length works by counting the length of text or a list. By example, length(hello) will return 5. In our case though, we have a list of indicators. Here’s a breakdown of how to make the previously mentioned title:

• The previous node is called Shuffle Tools_1: $shuffle tools 1
• The length function can count the length: length(hello)
• Put together: length($shuffle tools 1)
• And the full title:

Analyzing length($shuffle tools_1) observables from Shuffle

TheHive integration with Workflow Variables

For the API key and url fields I decided to use workflow variables, as we will be re-using these fields in another action for TheHive — part 5 (adding observables).

PS: if there is a text field anywhere in a workflow (except Execution Argument), you can use workflow variables, previous results and casting.

3. Condition: were any IoC’s found?

Any automation system needs to be able to compare data. What are we trying to compare though? We want to check whether the length of the results from the Shuffle Tools_1 is greater than 0, which means it didn’t find any IoC’s. How do we do that? In Shuffle’s case, this can be done between the nodes themselves. How? By clicking the line between the nodes (branch)!

The line (branch) is used to create conditions

By clicking this, we’re presented with the following (left):

Adding a new condition

Looks familiar doesn’t it? That’s because it’s similar to how we handled variables earlier. Click the “New condition” button and we’ll make our first one.

PS: Conditions are “AND” statements, meaning they ALL have to pass for an action to run, otherwise the node will execute with result “FAILURE”. If you need an “Else” statement or “OR”, please write it here.

The basic condition window

When the condition window opens, we are presented with three fields: source (left), condition (middle) and destination (right). During execution, all conditions in a branch will be verified. If source is 0, destination is 1 and the check is if 0 EQUALS 1, it will fail, because 0 does not equal 1. Our case will instead be using a LARGER THAN to see whether the first node is larger than 0.

Text fields in Conditions have the same capabilities as text fields in any node. This is a consistent method that will be improved over time in all areas. Do you remember how we got the length of the first node, Shuffle Tools_1 in the previous step? No? Here’s how:

• Use $ before the node name: $shuffle tools 1
• Use length(<data>) to get the length: length($shuffle tools 1)

With that mapped out in the left field, lets move on to the middle field. Click the dropdown and select “larger than”, before moving to the destination field and typing 0. This means that the action will only execute if the length of the first node is larger than 0.

Our IOC checking condition

With that filled in, we hit the “Submit” button before saving the workflow (CTRL+S), and hitting the big play-button. The expected result is that TheHive will throw an error as such (left):

Expected result without an execution argument

IP 192.168.1.2 as execution argument

If, however, we type in an IP, a domain, hash, URL, Mitre Att&ck technique or something else in the “Execution Argument”, TheHive will execute successfully and create a case because the condition worked out.

When we move on to Webhooks, we will expand upon the condition concept by splitting traffic based on what kind of webhook is triggered from TheHive.

Successful TheHive case creation

If any of these systems are confusing, please give us feedback.

Whenever you execute with an observable, the case is created! But where is the indicator from step 1? That will come in step 5.

Case created in TheHive with Shuffle

4. Webhook setup with TheHive

In this section, we’ll keep building on the workflow we just created by exploring webhooks. Webhooks are one of the technologies that makes Shuffle so powerful. Webhooks are a form of real-time data transfer, meaning when something happens in system X (sender, HTTP POST), system Y (receiver, webserver) gets information about it. I previously wrote about this concept with WALKOFF. Webhooks in TheHive are made to send information whenever ANYTHING is updated in TheHive (e.g. add an artifact to a case). Since webhooks in Shuffle work by executing with the information of the webhook, this means that the execution argument is set to the data from TheHive.

Shuffle webhook receiver

In our case, system X (sender, HTTP POST) will be TheHive and system Y will be Shuffle. Let’s start by adding a webhook to Shuffle. In the bottom left corner, we’ll find the “Triggers” button, where we previously found the schedule. This time around, drag the Webhook into the workflow. It should auto-connect to our “Shuffle Tools_1” action, but we’ll change this. Before moving on, add another App, “Testing” into the Workflow and set it to “Repeat back to me” with $exec as the call. Here’s what the workflow looks like at this point:

Adding webhook (purple) with a repeater action.

For simplicity’s sake, we have the Webhook in the same workflow (it works!), making it easier to visualize what’s happening. The red arrow indicates that whenever a case is created in TheHive, it should send a webhook to the webhook location, with the purple being the Webhook (trigger).

To proceed, we’ll click the Webhook node, before looking for the “Webhook URI” field on the right hand side. Keep this field in mind, as it will be used to configure TheHive. My URI is http://192.168.3.6:5001/api/v1/hooks/webhook_e6ca13a6-41cf-4996-9aa6-d629917115b8. Keep this URL in mind for the next step. To enable the webhook, we’ll click the “START” button, just as we started schedule triggers last week.

PS: Webhooks can be started and stopped at any time. If it says “http://localhost…”, change localhost to your ip (windows CMD: ipconfig, UNIX: hostname -I). If you want to test the webhook without TheHive, run this curl command against YOUR webhook url:

curl -XPOST http://<CHANGEME>/api/v1/hooks/<WEBHOOK_ID> -d 'hello this is a webhook'

TheHive webhook sender

TheHive is used as an example as it’s used by blue-teamers aplenty. One issue with TheHive in my opinion, is the barrier to entry. It takes a while to set up all the systems and get them to send information to TheHive. That’s one of the things we’ll touch more upon later, but for now, our focus is on webhooks. Their installation guide can be found here. For the purpose of this blogpost, I’ve created a working configuration here:

frikky/medium-examples

What you’ll need to make that work is to open the application.conf file and change the line containing “url =” to have your webhook URL instead of mine, TheHive’s official webhook setup can be found here. Once you’ve got TheHive changed, run the normal docker-compose command:

docker-compose up -d

Testing the integration

To test the integration, we need to do an action in TheHive. If you have an empty instance like I have, the easiest is to create a case, but any action is sufficient (edit alert, write a note, add user, add task etc)

With a case created, we’ll have to check our executions. Executions in Shuffle can either be seen for the workflow under /workflows OR in the Workflow itself by clicking the running person icon (bottom) before “Refresh executions”. On the left side of each execution, you’ll see what executed it. The last execution here was a webhook. If you don’t see a webhook image in the execution, it wasn’t started by a webhook.

PS: Old executions before the latest update will have the “play” button no matter what triggered them.

“All Executions” view for a workflow in Shuffle

Case creation webhook from TheHive

With a webhook triggered, we’re now able to analyze our results and move into the handling of the webhook itself.

To reiterate, TheHive triggers a webhook for almost any action. In our case (left) it was a case creation. But what other actions exist in TheHive? Artifact addition = case_artifact

In step 3, we made our first condition. To move on, we’ll need to make another condition. Why? Because we need to differentiate each type of TheHive webhook. Before we do that however, we need to be able to put our IoC’s from step 1 into TheHive itself. How? Another TheHive action!

5. Loops: Add IoC’s as “observables” to our case

In step one, two and three, we made sure to parse our indicators from text. One thing we skipped was adding the IoC’s to TheHive. As indicators are a huge part of information security in general, this step is important to get done well. For now, to increase understanding, we’ll go about it in a basic manner by introducing loops.

Loops in Shuffle are a less than ideal way of interacting with lists. If you’re not familiar with programming, loops are a way of interacting lists of data (think rows in excel) and choosing what to do with a row, one row at a time. In our case, those rows contains two tables, “data” (e.g. 1.2.3.4) and “data_type” (ipv4), which we need to find. Then FOR EACH row that has these points, we’ll be creating an observable in TheHive. Keep this in mind as you move forward. Here’s some quick backtracking and explanation:

• You can interact with previous actions by using the dollar ($) sign: $create_thehive_case and $shuffle_tools_1
• You can parse JSON by using its field name: $create_thehive_case.id (gets TheHive case ID)
• You can use hashtag (#) to interact with lists. Since the result of “shuffle tools_1" is a list containing JSON we’ll write “$shuffle_tools_1.#” to parse every item in the list, before adding “.data” and “.data_type” to parse the json items: $shuffle_tools_1.#.data

Here’s what the “Add observable” action looks like:

An overview of the add observable action

By typing e.g. “1.2.3.4 google.com” into the Execution Argument before executing the workflow, our first action “Shuffle Tools 1” will find two indicators as such: [{“data”: “1.2.3.4”, “data_type”: “ipv4”}, {“data”: “google.com”, “data_type”: “domain”}]. Because there are more than 0 items (2) in the result list, our workflow proceeds to make a case in TheHive. When the case is created, the observables will be added to the case one-by-one, which will trigger our webhook each time. All in all, this will trigger three webhook actions: one for the case creation and two for the observable addition. Execute the workflow and try it for yourself!

As we talked about at the end of step 4, the next step is to differentiate between case creation and observable addition. How does our workflow know whether to continue with artifacts or not?

6. Condition: Check the Webhook type

As we’ve created conditions before (step 3), here’s a brief reminder. A condition is a way to check whether we want to continue executing our workflow or not. It can do all necessary checks. So what are we checking? We are checking whether the webhook is an artifact. How do we know whether it’s an artifact? We look for “case_artifact”. How do we know that?

A case_artifact creation

If we check the results (click the running person on the bottom) from our last workflow execution, we’ll see at least 3 new executions: 1 normal and 2+ webhooks. Check the webhooks, and you’ll find that one of them looks like the example on the left. See the “objectType” field? That’s what we’ll be using!

Before we can add a condition however, we’ll need somewhere to write the condition. We add another action to our workflow, using the “repeat back to me” action, connect it to the Webhook repeater, and then clicking the branch itself (same as step 3). Click the “New condition” on the right hand side. When filling in the fields, remember that we found “objectType” in the Execution Argument, which translates to $exec.objectType.

An illustration of data source

Before moving to a real action for the artifact we’ll have to make sure that it works. Save the workflow, enter an e.g. an IP into the Execution argument field, and then execute. If your new node (“Case artifact” in mine) executes, you’re ready to move on.

7. Searching for the artifact with a Webhook

Since we’ve carved a path together to handle only artifacts, we can now do whatever we want. Do we want to analyze it manually in Virustotal as in the last blog? Do we want to push the observable to MISP? Do we want to post it on Twitter (there might be a real use-case for this)? Most of the time you’ll want to know whether:

1. The artifact is malicious
2. It’s been seen in your environment

Before you might take the actions of:

1. Creating a new task for manual exploration
2. Ban a hash in your EDR
3. Block traffic in your firewall
4. Add as new alert to your SIEM
5. Etc..

There are a wide array of available options, but in our case, since this post is long enough already, we’ll just run a cortex analyzer with MISP for now. If you haven’t set up Cortex already, follow their guide here. As always, we start by finding the app we’re looking for in the left hand view, and dragging it in. Since we already know what analyzer to run, MISP, and we’ve already checked out the Execution Argument, there’s not much to do except fill it in!

Blue line = IP moving from argument -> tools -> webhook -> data (right)

With that all set up, the only thing missing is execution. Save it, write something (e.g. IP 1.2.3.4) in the Execution Argument and run it. To recap what will happen:

• Shuffle Tools will find the IP 1.2.3.4
• TheHive will create a case (triggers “case” webhook)
• Webhook for case creation will fail, because it’s not “case_artifact”
• TheHive will add 1.2.3.4 to the case (triggers “case_artifact” webhook)
• Webhook will run all the way to cortex because it’s a “case_artifact” as our condition required

Here’s my challenge to you: Get the result from the Cortex execution (the action exists).

If I’ve done my job well, you will hopefully grasp a few more concepts of a Shuffle workflows by now. Shuffle is a tool that enables you to change your processes, procedures, platforms and everything software-related on the fly without having to re-code, develop and have a full CI/CD environment to avoid bugs. It’s also a way to visualize how your procedures work without having cronjobs and python/bash/ruby (or god forbid, perl) scripts everywhere.

If you want a complete webhook handler for TheHive (as in the header image), import it from here:

frikky/shuffle-workflows

Please share, follow me, clap (up to 50 times) and check out my other posts! I write about Infosec, automation, cloud development and startup things in Tokyo :)

Frikky - Medium