Data is Everything
Whether hunting down hackers in your network, optimizing traffic flow, or just trying to build a network baseline, you’ll learn that data is everything. Data is our gateway to gaining deeper insights into the patterns or behaviors of our users and devices. The data we are hoping to gain insights into is network traffic.
Following the Data, Information, Knowledge, Wisdom (DIWK) model, we’re going to walk through building a pipeline for turning network traffic data into network traffic knowledge. Out two tools of choice will be Moloch and Elastic.
What is Moloch?
In short terms, Moloch is an open-source, scalable packet capture and indexing solution. Initially developed by AOL for handling their large volumes of traffic, the kind folks there released it under the Apache 2.0 license for use by the community. What makes Moloch a fantastic tool for Security Operations is that one of its core components is built on Elasticsearch already, allowing you to leverage your existing Elastic Stack if you choose to.
The main components of Moloch are as follows:
- MolochCapture: PCAP capture and analysis
- MolochViewer: Web-based GUI viewing data and transferring PCAPs
- Elasticsearch: Search the database
Ideally, Moloch would be deployed much like an IDS sensor. If your network is small, centrally located, or cloud-based, it may be possible to implement a single instance and capture all your organization’s traffic. However, if you have a network the sprawls out physically, such as a university campus, then it may make sense to deploy multiple “sensors” to capture your core routing and switching backbone. In the same vein, it would be up to you to decide if you want to host your Elastic Stack on the same machine as Moloch or elsewhere This flexibility is part of what makes Moloch such a unique tool for gaining insight into your network traffic.
Deploying Elastic Stack
Here we’ll be using an Elastic stack container in Photon OS. Photon is a lightweight Operating System built by VMware and designed to run in virtual environments. I’m going to assume you’ve already installed Photon OS on your hypervisor of choice and have installed all updates. If not, there are plenty of tutorials on building and deploying hypervisors, with just as many debates on which vendor or technology is better.
Note: This guide should work regardless of what OS you decide to use. The main requirement is Docker, and you aren’t constrained to running it in a VM. If you deploy Docker on a bare-metal OS, everything should still work correctly.
Before pulling down our Elastic Stack image, we have to verify the virtual memory mapping (mmap). The Elastic Stack requires a mmap count equal to or more than 262,144.
$ sysctl vm.max_map_count
vm.max_map_count = 262144
If you need to change the vm.max_map_count of your machine, enter the following:
sysctl -w vm.max_map_count=262144
Since we have Docker, already installed, and our mmap set correctly, let us go ahead and pull the sebp/elk image for our Elastic Stack and run it.
$ docker pull sebp/elk
$ docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it \
--name elk sebp/elk
Let’s verify our Elastic Stack is listening.
$ netstat -plnt
We can see our container is up and listening now. The sebp image uses the following port assignments:
- 5601 (Kibana web interface)
- 9200 (Elasticsearch JSON interface)
- 5044 (Logstash Beats interface)
Now that our Elastic Stack is up and running, it’s time to deploy Moloch.
For this guide, we’ll only be using one Moloch instance, running on a Centos 7 machine. Like with the Docker section, I’ll assume you’ve installed Centos 7 on your own and upgraded all it’s core packages.
First, we’ll pull down the rpm and install it.
$ wget https://files.molo.ch/builds/centos-7/moloch-2.2.1-1.x86_64.rpm
$ sudo yum install moloch-2.2.1-1.x86_64.rpm
Once installed, we’ll go ahead and configure Moloch. Be sure to have the IP address for your Photon VM. In this case, our Photon VM is 192.168.1.67.
$ cd /data/moloch/bin/
$ ls -l
-rwxr-xr-x. 1 root root 9851 Jan 21 05:19 Configure
-rwxr-xr-x. 1 root root 100 Jan 21 05:19 moloch_add_user.sh
-rwxr-xr-x. 1 root root 6767952 Jan 21 05:16 moloch-capture
-rwxr-xr-x. 1 root root 829 Jan 21 05:19 moloch_update_geo.sh
lrwxrwxrwx. 1 root root 35 Feb 3 17:57 node -> ../node-v10.18.1-linux-x64/bin/node
lrwxrwxrwx. 1 root root 34 Feb 3 17:57 npm -> ../node-v10.18.1-linux-x64/bin/npm
lrwxrwxrwx. 1 root root 34 Feb 3 17:57 npx -> ../node-v10.18.1-linux-x64/bin/npx
-rwxr-xr-x. 1 root root 2007 Jan 21 05:16 taggerUpload.pl
$ sudo ./Configure
Next, initialize the database.
$ sudo /data/moloch/db/db.pl http://ESHOST:9200 init
Let’s add a user account to be able to reach the web front end.
$ /data/moloch/bin/moloch_add_user.sh admin "Admin User" SECRETPASSWORD --admin
Now that Moloch is configured and initialized, we’ll go ahead and start the molochcapture and molochviewer services as well as set them to start after reboots.
$ sudo systemctl enable molochcapture.service
$ sudo systemctl enable molochviewer.service$ sudo systemctl start molocapture.service
$ sudo systemctl start molochviewer.service
View our services to ensure they started.
$ systemctl status molochcapture molochviewer
As we saw, our molochcapture failed to start. After viewing the logs at /data/moloch/logs/capture.log, we discovered that molochcapture was trying to capture on eth1, not eth0. We can fix this by opening /data/moloch/etc/config.ini.
$ sudo vim /data/moloch/etc/config.ini
Here at line 56, I changed the interface from eth1 to eth0, saved the config, and restarted my services. ALL GREEN! (I learned after I selected the wrong interface when I ran the ./Configure script.)
Time to add the finishing touches for Moloch. Depending on your setup, you may have to adjust your iptables or firewalld setting to be able to reach web GUI from another machine. In our case, firewalld is the Centos 7 firewall, so we’ll add entries to allow access.
$ sudo firewall-cmd --add-port=8005/tcp
success$ sudo firewall-cmd --runtime-to-permanent
Boom! Our Moloch sensor is deployed and configured.
Molochviewer is excellent and everything, but we started this to set up an architecture that can handle multiple Moloch sensors, and we wanted a way to analyze our data in aggregate. To that end, we’ll put the finishing touches on our Elastic Stack to handle the newly indexed Moloch data.
We’ll connect to our Kibana application on port 5601. If this was a fresh install of Elastic Stack, select “Explore on my own.”
Click the “Discover” tab on the right. From there, we’ll enter “sessions*” as our index pattern and click next.
After we select “timestamp” as our Time Filter field name, then click the “Create Index Pattern.”
When we go back to the “Discover” tab, we’ll see that our packet capture information has now populated our Elastic Stack!
Here we went over standing up a Moloch sensor and sending the network traffic to an Elastic Stack. Both tools are potent in their own right. Still, when leveraged together, it will allow the security analyst or network engineer to gain far greater insights into the traffic traversing their network without the need for expensive licenses or proprietary software solutions.
The next step would be setting up visualizations and dashboards in Kibana, but the content for those would depend entirely on your organization’s needs.
Before You Go…
Check out my latest article regarding how Adversaries can exploit BioCyber Security: