Know thy e-mail
This pertains to stoQ v1, though the same capabilities exist in latest version of stoQ, these instructions will not work with the latest release.
Have you ever needed to check if your users received a malicious file with a specific filename? Perhaps you know of a malicious e-mail address from a threat intel service, but have no way to know if they ever e-mailed your users. Would you like to be able to scan every attachment in every e-mail within in your enterprise with yara or any other tool? Today, we’re releasing our newest plugin for stoQ, the SMTP Parser that allows you to do just that. You can now parse every e-mail coming into your enterprise, insert it into any data store you wish to use, and scan every attachment with your application of choice. Let’s get started and show you how you can quickly deploy this in your environment today.
Getting Started
Before we get to the fun stuff, let’s make sure our environment is setup properly so you can play along. We will assume that you have a basic stoQ environment already setup and ready to go with the latest version (v0.9.18) and plugins. If you don’t, you can quickly get one setup using Docker, Vagrant, or using the installation script. Need more help? Feel free to take a look at the documentation or send us a note.
This post will also assume that you are doing everything from an Ubuntu box.
Postfix
Ok, now that we have stoQ all setup, let’s setup a basic postfix server. First, let’s install a barebones postfix server.
vagrant@vagrant$ sudo apt-get install postfixFor this use case, we will select “Internet Site” from the menu and set the domain name to your e-mail domain. In this case, we will just use ‘yourdomain’.
Next, we need to make sure that Postfix will use Maildir as it’s mailbox format.
vagrant@vagrant$ sudo postconf -e 'home_mailbox = Maildir/'Let’s go ahead and restart postfix now.
vagrant@vagrant$ sudo service postfix restartHandling E-mails
Now, we get to the fun part. There are several ways we can ingest e-mails. For now, we will keep it simple. We will handle any e-mail that is sent to stoq@yourdomain.
First, make sure you are logged in as the stoQ user.
vagrant@vagrant$ sudo su - stoqBecause we’ve not received any e-mail yet, let’s create the Maildir directory structure and set permissions so we can tell stoQ to monitor it.
stoq@vagrant$ mkdir ~/Maildir/new
stoq@vagrant$ chmod -R go-wrx ~/MaildirYou should now be the stoQ user and be in a python virtual environment. Let’s run the SMTP worker plugin and monitor our Maildir for new e-mails.
stoq@vagrant$ ./stoq-cli.py smtp -I dirmon -F ~/Maildir/new .d8888b. 888 .d88888b.
d88P Y88b 888 d88P" "Y88b
Y88b. 888 888 888
"Y888b. 888888 .d88b. 888 888
"Y88b. 888 d88""88b 888 888
"888 888 888 888 888 Y8b 888
Y88b d88P Y88b. Y88..88P Y88b.Y8b88P
"Y8888P" "Y888 "Y88P" "Y888888"
Y8b
Analysis. Simplified.
v0.9.18[*] root: Monitoring /usr/local/stoq/Maildir/new for new files...
What we’ve done here is started the smtp worker plugin, told stoQ to use the directory monitor plugin as the source, and that the ~/Maildir/new directory should be monitored for new files. Now, let’s open up a new terminal and send a quick test e-mail to the stoQ user so we can see what happens.
stoq@vagrant$ echo "This is a test e-mail" | mail -s "Test" stoq@localhostLooking back at the window we have stoQ running in, you should see something similar to this:
{
"results": [
{
"uuid": "776b4232-c8cc-4d65-9f5e-43d4167e9f54",
"path": "/usr/local/stoq/Maildir/new/1462987239.V801I637afM519163.vagrant-ubuntu-trusty-64",
"payload_id": 0,
"archive": "file",
"md5": "fa11ac74cdb54adfe8c2e30969e01839",
"sha256": "17c13ab194086fe602a957bbbf949525d3e15a89efc6ad92b9562039870e5399",
"plugin": "smtp",
"sha512": "3466bbd6f35449a16f69cf2a4b516dc2ffdbe7818e4208a927c2c0edfbb7652cbba7a629216496dcf6f9ab5b16769f92ff3d7012a09b00a6a641e3b5656411b4",
"scan": {
"uuid": "03150da1-6331-4c73-b8b4-43ad3244e860",
"date": "Wed, 11 May 2016 17:20:39 +0000 (UTC)",
"delivered-to": "stoq@localhost",
"x-mailer": "mail (GNU Mailutils 2.99.98)",
"body": "This is a test e-mail\n\r\n--69b1cbbb-0f6f-4a26-ac4a-08faa1f144a6--\r\n",
"message-id": "<20160511172039 .7d2b7637b0="" vagrant-ubuntu-trusty-64="">",
"subject": "Test",
"x-original-to": "stoq@localhost",
"from": "stoq@vagrant-ubuntu-trusty-64 (stoQ)",
"return-path": "",
"from_isnew": true,
"received": "by vagrant-ubuntu-trusty-64 (Postfix, from userid 999)\n\tid 7D2B7637B0; Wed, 11 May 2016 17:20:39 +0000 (UTC)",
"to": "",
"puuid": "776b4232-c8cc-4d65-9f5e-43d4167e9f54",
"body_html": "",
"att": []
},
"source_meta": {},
"sha1": "4fe7324d8ad495e0c1a42d819c29948caadcb8a6",
"filename": "1462987239.V801I637afM519163.vagrant-ubuntu-trusty-64",
"size": 472
}
],
"date": "2016-05-11T17:20:40.022749",
"payloads": 1
}What just happened here? You’ve successfully analyzed an e-mail with stoQ from the stoQ users inbox. Looking at the results, you can see all of the details about the e-mail itself within the scankey. It tells you who the e-mail went to, the subject, every other header in the raw session, along with the body of the e-mail itself. We’ve also used the iocextract plugins to extract all IP addresses and URLs in the e-mail and headers, making them easily available for analysis. In short, the entire email session was extracted and made parseable by stoQ.
Now, let’s get to some even cooler things. What if we want to extract every attachment and scan it using yara, exif, trid, and peinfo workers? It’s already configured to do just that. Just send an e-mail to your stoQ user with an attachment and each attachment will be extracted and scanned with each of the previously mentioned worker plugins. Done and done.
What if we want to scan each attachment with another worker plugin, such as clamav. Each plugin has it’s own configuration file that ends with .stoq To get the smtp worker scanning attachment with clamav, all we need to do is edit it’s stoq configuration file located at plugins/worker/smtp/smtp.stoq.
[options]
hashpayload = True
saveresults = True# Should a queuing plugin be used? True or False
# publisher worker plugin is required to be installed
use_queue = False# Default workers that all attachments will be scanned with
workers_list = yara, exif, trid, peinfo, clamav# What is the default attachment connector plugin to use?
attachment_connector = file# Should bloom filters be used to track specified SMTP headers?
# If True, the SMTP plugin may only be run once at a time
use_bloom = True# Where is the bloom filters file located?
bloom_directory = plugins/worker/smtp/email_filters# Expected maximum values in bloom filter
bloom_size = 50000000# Desired False Positive rate
bloom_fp_rate = 0.001# How often should the bloomfilter be written to disk?
bloom_update = 60# SMTP header to extract a value from and insert into bloom filter
bloomfield_list = from
We’ve added clamav to the workers_list option. Now, when we receive an e-mail each attachment will also be scanned with clamav.
Enterprise Deployment
Let’s get everything setup so we can use this in an enterprise environment. For this use case, we will use RabbitMQ as our queueing system of choice and saving all results and attachments to disk. We will also use supervisor to run stoQ as a daemon in the background.
If you installed stoQ using Vagrant or the installation script, then you have RabbitMQ installed and operational on your localhost. If not, you’ll need to install and setup a RabbitMQ server and configure the stoQ RabbitMQ plugin to use that server by editing plugins/source/rabbitmq/rabbitmq.stoq.
Let’s install supervisor so we can daemonize stoQ.
vagrant@vagrant$ sudo apt-get install supervisorNow, let’s add our stoQ workers so supervisor can start and monitor the processes.
vagrant@vagrant$ sudo vi /etc/supervisor/conf.d/stoq.confAnd then add the below to the newly created file.
[program:exif]
command=/usr/local/stoq/.stoq-pyenv/bin/python stoq-cli.py %(program_name)s -I rabbitmq -C file
process_name=%(program_name)s_%(process_num)02d
directory=/usr/local/stoq
autostart=true
autorestart=true
startretries=3
numprocs=1
user=stoq[program:yara]
command=/usr/local/stoq/.stoq-pyenv/bin/python stoq-cli.py %(program_name)s -I rabbitmq -C file
process_name=%(program_name)s_%(process_num)02d
directory=/usr/local/stoq
autostart=true
autorestart=true
startretries=3
numprocs=1
user=stoq[program:peinfo]
command=/usr/local/stoq/.stoq-pyenv/bin/python stoq-cli.py %(program_name)s -I rabbitmq -C file
process_name=%(program_name)s_%(process_num)02d
directory=/usr/local/stoq
autostart=true
autorestart=true
startretries=3
numprocs=1
user=stoq[program:trid]
command=/usr/local/stoq/.stoq-pyenv/bin/python stoq-cli.py %(program_name)s -I rabbitmq -C file
process_name=%(program_name)s_%(process_num)02d
directory=/usr/local/stoq
autostart=true
autorestart=true
startretries=3
numprocs=1
user=stoq[program:clamav]
command=/usr/local/stoq/.stoq-pyenv/bin/python stoq-cli.py %(program_name)s -I rabbitmq -C file
process_name=%(program_name)s_%(process_num)02d
directory=/usr/local/stoq
autostart=true
autorestart=true
startretries=3
numprocs=1
user=stoq[program:smtp]
command=/usr/local/stoq/.stoq-pyenv/bin/python stoq-cli.py %(program_name)s -I rabbitmq -C file
process_name=%(program_name)s_%(process_num)02d
directory=/usr/local/stoq
autostart=true
autorestart=true
startretries=3
numprocs=1
user=stoq[program:publisher]
command=/usr/local/stoq/.stoq-pyenv/bin/python stoq-cli.py %(program_name)s -I dirmon -F /usr/local/stoq/Maildir/new -C file -w smtp
process_name=%(program_name)s_%(process_num)02d
directory=/usr/local/stoq
autostart=true
autorestart=true
startretries=3
numprocs=1
user=stoq
Ok, that’s a lot. Let’s go over what we just did. We’ve told supervisor to start multiple stoQ workers (exif, yara, peinfo, trid, clamav, and smtp). These stoQ workers were all executed with the source plugin of rabbitmq and the connector plugin of file. What this means is these workers will monitor RabbitMQ for new messages, process each of those messages, then save the results to a file on disk. We could just as easily change the connector plugin to use mongodb, elasticsearch, or any of the other available plugins use for saving results.
The final program being run in the supervisor configuration file is the stoQ publisher worker. This worker has a very simple function, which is take events (such as a new file being created) and then send that event on to a queueing system, such as RabbitMQ. In this case, we are using the publisher worker to detect that a new e-mail has been received, and it will then send a message to the queue for the smtp worker.
Now, we need to make sure the smtp worker knows to use RabbitMQ to publish messages to all of the other workers when it extracts an attachment. Let’s update the smtp worker configuration file (plugins/worker/smtp/smtp.stoq) update it use_queue to True.
# Should a queuing plugin be used? True or False
# publisher worker plugin is required to be installed
use_queue = TrueWe are pretty much all set now. All we need to do is load the new configuration file and start the stoQ services.
vagrant@vagrant$ sudo supervisorctl reloadLet’s check the status to make sure everything started up ok.
vagrant@vagrant$ sudo supervisorctl status
clamav:clamav_00 RUNNING pid 6534, uptime 0:00:01
exif:exif_00 RUNNING pid 6533, uptime 0:00:01
peinfo:peinfo_00 RUNNING pid 6536, uptime 0:00:01
publisher:publisher_00 RUNNING pid 6538, uptime 0:00:01
smtp:smtp_00 RUNNING pid 6535, uptime 0:00:01
trid:trid_00 RUNNING pid 6539, uptime 0:00:01
yara:yara_00 RUNNING pid 6532, uptime 0:00:01You should see everything running. If not, check out logs/stoq.log for any errors that may have occurred.
Now, go ahead and try sending an email to your stoQ user and include an attachment with that e-mail. Once each plugin is done scanning, it will write the results to disk in their respective folder in results/. For instance, the yara worker will write its results to results/yara/ and the smtp worker will write to results/smtp/.
Once you’re ready, you can now forward any e-mail you would like scanned to your stoQ user account and review the results at your leisure.
Conclusion
We’ve shown you how to build a basic postfix server for use with stoQ, and what the parsed data looks like when sent through the smtp worker. You have also learned how to scan extracted attachments with other stoQ plugins and how to configure those plugins. We then went over how to bring it all together so you can scan all of your e-mail and their attachments in your enterprise environment.
Why would you want to do this? Imagine having complete visibility into all of your e-mail and their attachments. No longer will you have to shrug your shoulders when asked by leadership if you’ve ever received an e-mail from a malicious actor. You will never have to wonder if a malicious file has been sent to your users. You no longer have to pay tens of thousands of dollars annually to big name vendors just to have visibility into your own traffic. You can automate more tasks allowing your analysts to focus on what matters, not on trying to get information that should be simple to obtain.
Want to see how to use this with your enterprise Splunk deployment? We will be presenting stoQ’ing your Splunk this June at DFIR 2016 in Austin, TX. Come by, learn more about stoQ and how you can leverage it with the power of Splunk to simplify your security operations.
