Introduction to stoQ

This pertains to stoQ v1, though the same capabilities exist in latest version of stoQ, these instructions will not work with the latest release.

Simply put: we were tired of wasting time. As network defenders, we have seen the market explode with countless security tools promising better awareness, functionality, and speed (we’ll make you taller AND better looking!!!). Unfortunately, most of these solutions have one thing in common — they don’t talk to each other, and in many cases, results are lost the moment our screen scrolls away or our browser window closes. As analysts, we have been frustrated by this inefficiency for some time now. It doesn’t have to be this difficult. We just want our data, and we want to be able to easily search it in our datastore of choice. So, we decided to do something about it.

As we developed stoq, our mantra was “make no assumptions.” We wanted to develop a tool for analysts that would allow them to organize and automate the data they received on a daily basis in an easy-to-use fashion using whatever tools were important to them. Analysts are really particular with their pet tools, so we didn’t assume stoQ had to be wedded to any one set. Consequently, we created more than 40 plugins for everything from input to output, to include any intermediary analytics such as carving and extracting. If you want to scan a file against a new API or tool, support a different database, extract content from a file, or just about anything else, stoQ can do it. You’ll notice popular workers for FireEye, yara, and OPSWAT, all of which have plug-and-play functionality within stoQ. We’re expanding these workers each week. Want one in particular? Send us an email.

Ready to go? Perhaps the easiest way to get started is to download stoQ here. Once you have it ready, we recommend scanning a single file using a tool like, exiftool, and displaying the results to the console. Let’s take a look at what the results look like for this example:

stoq@ubuntu-xenial:~stoq# ./stoq-cli.py exif -F 03.exe
{
    "results": [
        {
            "source_meta": {},
            "plugin": "exif",
            "sha1": "9e6414bf2802c98fbd13172817db80380c5eeb6a",
            "md5": "16d9f6e5491d99beb46d7ab1500c1799",
            "payload_id": 0,
            "path": "/vagrant/03.exe",
            "uuid": [
                "de2a7b50-c47a-409d-a837-af00187d2245"
            ],
            "scan": {
                "InitializedDataSize": 120832,
                "TimeStamp": "2013:04:20 14:50:10+00:00",
                "MIMEType": "application/octet-stream",
                "FilePermissions": "rw-r--r--",
                "FileTypeExtension": "exe",
                "PEType": "PE32",
                "OSVersion": 5.0,
                "Directory": "/usr/local/stoq/temp",
                "MachineType": "Intel 386 or later, and compatibles",
                "FileInodeChangeDate": "2018:04:21 00:18:17+00:00",
                "SubsystemVersion": 5.0,
                "ImageVersion": 0.0,
                "FileSize": "496 kB",
                "FileAccessDate": "2018:04:21 00:18:17+00:00",
                "Subsystem": "Windows GUI",
                "FileName": "7a5a84b2-aeea-42bb-897c-c5d42305ee68.stoq",
                "UninitializedDataSize": 0,
                "ExifToolVersion": 10.59,
                "LinkerVersion": 9.0,
                "FileType": "Win32 EXE",
                "FileModifyDate": "2018:04:21 00:18:17+00:00",
                "EntryPoint": "0x32dc0",
                "SourceFile": "/usr/local/stoq/temp/7a5a84b2-aeea-42bb-897c-c5d42305ee68.stoq",
                "CodeSize": 386048
            },
            "archive": "file",
            "size": 507904,
            "sha256": "47c6e9b102324ea6c54dd95ad3fdf4b48b18775053b105e241a371a3731488c0",
            "sha512": "d1cb5df161ed4f9266c59b3eab1c004f8496fad591ac869a6cd2558e29d0d39094fe4e47e87ddfab328a3761af9a81fcd7bc7ad9946ef5625c2fbe3aca1b5808",
            "filename": "03.exe"
        }
    ],
    "plugins": {
        "0": "exif"
    },
    "tlp": "white",
    "payloads": 1,
    "date": "2018-04-21T00:18:17.344234"
}

Here’s a more advanced example that uses Bro and/or Suricata to extract files from your network and analyze them with stoQ. Once ingested by stoQ it will ensure compressed, XOR’d, OLE streams, and base64 encoded content is decoded or extracted. stoQ will then scan those independent streams against exiftool, Fireeye, Yara, Virustotal, and #totalhash. Once stoQ receives the results from exiftool, Fireeye, and yara stoQ will insert them into ElasticSearch. For the results from Virustotal and #totalhash, stoQ will insert them into Mongodb instead. No matter how you want your data to be ingested, analyzed, and stored, stoQ can simplify and automate that process.

stoQ is an extremely agile and robust framework that helps to automate the majority of redundant tasks we as analysts do on a regular basis. Carving executables or SWF’s out of a file should not be a time consuming and cumbersome task. Scanning files with yara, exif, or clamav should not be difficult, nor should those results be lost as you work between tools. You and your network should be able to focus on what matters, not on jumping from tool to tool, and then manually connecting the dots.

You can browse and download the source code for stoQ here, and also check out more than 40 publicly available plugins here. Additionally, the most up-to-date documentation for stoQ can be found here.

In future posts, we will be focusing on blog posts that will outline the use cases for stoQ and how to quickly enhance your visibility of threats.