Getting started with Shuffle — an Open Source SOAR platform part 2
Getting started with any new and shiny tool is hard, but exciting. My job is to demystify and make it fun and easy. SOAR and Open Source tools can be complicated messes if done wrong, but we’re avoiding that like the plague with Shuffle. This post will bring some much needed light to installation, user creation and workflow editing. It’s aimed at analysts, developers and anyone wanting to learn about automation, and makes you proficient at creating and running basic workflows.
This post is the second part in a series of blog posts about Shuffle. We’ve previously delved into what SOAR is (and isn’t) and an introductory overview to apps, workflows, triggers, actions and more. This post will build upon that with examples of these terms, with next post focusing on on practical implementations with Virustotal and TheHive.
- 1. Introducing Shuffle - an Open Source SOAR platform
- 2. Getting started with Shuffle (This blogpost)
- 3. Integrating Shuffle with Virustotal and TheHive
- 4. Real-time executions with TheHive, Cortex and MISP
- 5. An advanced workflow walk-through
- 6. The (not so distant) future of Shuffle (Mitre Att&ck, dynamic dashboards, asset management, KPI’s etc.)
To jump straight into it, check out the documentation: https://shuffler.io/docs/features. Let’s get automating.
Deploying Shuffle
Want a video for how to install Shuffle? Check out this one from our good friends at OpenSecure!
With everything Shuffle-related being on Github and running in Docker, Docker, Docker-compose and git to get started. With that out of the way, let’s delve into the setup itself. The following commands will clone the repository, enter it, before running the docker containers specified in docker-compose.yml.
git clone https://github.com/frikky/Shuffle
cd Shuffle
docker-compose up -dThis might take a few minutes as we download the required images. When that’s all set and done, go to https://localhost:3443. This will throw an expected certificate warning, as it’s a self-signed certificate. If you want to use HTTP, go to http://localhost:3001. Please contact us or create an issue if there’s a problem in any of these steps.
We are now on the first and only step of the admin setup: creating the first admin user.
There’s no way to register users from outside the platform itself other than here. If you forget before your first login, delete the database (/etc/shuffle) and restart the backend.
After registering an admin account, we are redirected to the login screen, expecting the same credentials.
After logging in, we’re presented with a welcome screen. It has links to everything necessary to get cooking. In this blog, we’ll check out the Workflows only, but make sure to check the apps, and read the docs on other parts as well.
PS: If you have some issue, try updating:
git pull
docker-compose down
docker-compose pull
docker-compose upYour first workflow
As stated in the first part of the series, workflows are the part of Shuffle where everything comes together. To get started, click the “New workflow” button. A view will pop up asking for a name and (optional) description.
After submitting, we’re in the workflow view. This is an empty view with a bunch of apps on the left side. If the app list is empty, wait a few minutes and refresh the window. Some apps are loaded and built the first time Shuffle is ran, which might take some time. Meanwhile, let’s get a look at the nodes we’ll be making:
- A “hello world” node. This will be our start node, meaning the first action to execute.
- A node that repeats data from the hello world node. This is to show the basics of passing data between nodes.
- A node that makes a http GET request based on the execution argument, trying to parse the JSON value “url” (see node point 5).
- A node that attempts to read the JSON data from the third node, and print the IP it finds (searching virustotal in a later blogpost)
- A scheduler. This will make our workflow execute every X seconds with the JSON data {“url”: “https://ip.seeip.org/jsonip”}, parsed by node 3
1. The hello world node
The GIF above shows the entire process. We scroll through the app list and find the “test” app. This app has an action, “hello world” (pre-selected), which we proceed to give a name (optional), and that’s it!
Before testing the node, we need to save. Hit the save icon, and then the big red (orange) play button. From here on, wait a little bit, and the execution should run by itself.
If there aren’t any results within 15 seconds, something most likely went wrong. Contact us or make an issue so we can sort it out.
2. The first repeater node
The point of the repeater node is to show how changing action and passing data works. This is extensively covered in the documentation, but is valuable to show again and again nonetheless.
The picture on the left has two nodes: the initial node (Hello world) and a new one. When dragging the second node into the workflow, they should be automatically connected. Hovering one of them and dragging from the small blue circle at the top of the action can change these connections. They can also be deleted.
The difference between these nodes is that the “Action” drop-down has been changed from “Hello world” to “Repeat back to me”. This function is useful to debug what result a node gives, so keep it in mind when developing workflows later.
Next, we need to choose what data to repeat. “Repeat back to me” takes a single parameter, “call”, which is the value to be repeated. We can just write something in the field (e.g. hello), or we can use data from a previous node or variable. There are multiple ways to do this, but the easiest is to click the “circled plus” icon inside the field, which will show a dropdown. This dropdown contains all variables you have access to, based on what the previous nodes are. In our case, we want to choose the “Hello_world_node”.
PS: Notice how the color of the node changed as you hovered it in the dropdown?
3. Getting our IP
There are multiple ways to get our IP, but for this test, we want to get our public ones, meaning we should get it from an external source. I found the service https://ip.seeip.org/jsonip, which returns this in the form of JSON.
To make a GET request, we will try a new app; HTTP. This app is made to test connections to different services, and can connect any web services together. It supports all the “normal” functions of HTTP — GET, POST, PUT, DELETE.. — along with the ability to execute curl directly.
In the case below, the first node is connected to both “First repeater” and “Get URL”. They can be positioned any way you prefer. When the HTTP app is dragged into view, hover one of the other nodes before dragging a line from the small blue dot to it. Now, with the action connected — click it and “GET” from the Actions list.
GET takes three parameters, with only the first one being required, indicated by the orange rather than yellow mark. Paste any URL in there, before saving and executing.
We will make slight modifications to this node later to connect it with the scheduler.
4. Parsing the IP from our GET request
When the HTTP request (Get URL) is successfully executed with the URL https://ip.seeip.org/jsonip, it returns our IP as:
{“ip”: “YOUR IP”}To expand our use-case, we now want to use the IP we received and say “My IP is 1.2.3.4” if our IP indeed is 1.2.3.4.
To do this, we do the same as with node 2 - add the “test” app and choose the “repeat back to me” action for our new node.
This time however, we want to use text together with our variable. We need to use the normal text field. Here’s how to use data from a previous node:
- Start with $ and then the nodename: $Get_URL
- Now we want “ip” from the JSON data: $Get URL.ip
- All together: My IP is $Get URL.ip
Tip: Use the dropdown menu! It will autocomplete this for you.
Scheduling the workflow
Scheduling is and will always be a vital part of automation. The reason is simple: we need to look for periodic updates (especially if they services don’t support webhooks). This can be from our ticketing systems, mailboxes, threat intelligence feeds or anything else. In our case, we’ll use it as a way to demonstrate automated workflow execution.
Adding a trigger
To add a trigger, find the “Triggers” button below the apps on the left hand side. As seen in the photo on the left, there are currently five ways to automatically execute workflows:
- Webhook: Used for realtime executions from an external resource (e.g. TheHive)
- Shuffle Workflow: Execute another workflow.
- User Input: Waits for a user to click before continuing execution
- Schedule: Executes the workflow on a schedule
- Email: Executes when a mail is received in a specified mailbox
As with apps, triggers are draggable. Once inside the workflow editor, the schedule will auto-connect to the starting node, as that’s where the execution should start (later, this can be modified for sub-executions). With the schedule node clicked, it shows you that it needs two parameters: an interval in seconds (time between executions) and the execution argument.
We haven’t previously covered this, but you might have noticed the “Execution argument” field next to the big play button:
Anything that’s put in that field becomes a value available to all nodes in your workflow through the $exec variable. In our case, we decided previously that we want to provide a URL in this field as such: {“url”: “https://ip.seeip.org/jsonip”}. This URL is then used in the “Get_URL” node to define what URL we want to get. Here’s the schedule implementation in action, with an interval of 5, meaning the workflow will run every 5 seconds.
To follow up and track the executions, click the running person on the bottom or go back to /workflows. This is where information about previous executions are shown.
What’s next?
Having traversed installation, admin setup and the creation of a workflow with variables, you should now be ready to go deeper. We didn’t delve too far into specific apps, but the next post will start explaining integrations with TheHive, and continue building our workflow by creating a Virustotal App on the fly with the App creator.
You’re of course open to do these things yourself before then. It’s been a long week of bug fixing, feature improvements and documenting Shuffle, and I don’t expect it to stop soon. It should get to a stable, secure and empowering stage pretty quickly at this pace.
If you didn’t do it already, please star and share the project as well as this blog series (this is 2/6).
If you’re scared of Github (I know many of you are, as I used to be), get in touch and tell me about anything from excitement to issues.
Please clap (up to 50 times) and share the post. Follow me for updates, and don’t hesitate to get in touch.
