Protect the Box
March 2016 update: unifiedbeat was mentioned at Elastic{ON}16 in this presentation.
Unifiedbeat can help to protect your hosts and network. It is open source software that tails and indexes alert data (event, packet, and extradata records) from unified2 binary output files directly into ElasticSearch. Unified2 files can be created by intrusion detection software (IDS) such as Snort or Suricata.
Unifiedbeat can replace the existing set of software commonly in use for network security monitoring, which is usually based on Barnyard2 and MySQL — and that existing software is starting to show its age.
To infinity and beyond
Why is Unifiedbeat and ElasticSearch a better solution?
- MySQL’s search capabilities are extremely limited when compared to ElasticSearch/Lucene, and SQL databases can be slow with large amounts of data (especially when joining tables)
- ElasticSearch is designed for big data with very fast search responses
- ElasticSearch is easy to install and setup — especially a single node
- ElasticSearch makes it simple to add/delete indices — so one can index past/present unified2 files for analysis… there’s even a snapshot/restore API in ElasticSearch
- Unifiedbeat is written in Go and takes advantage of the concurrency built into libbeat — so it’s fast and indexes data in bulk using a harvester for each file being indexed
- Unifiedbeat indexes the rule/signature that triggered the alert — using the existing .rules files and gen-msg.map — so there’s no more rule lookups at search time … in other words, one gets a more complete picture of what happened and what triggered it
- Unifiedbeat indexes all of the records within a unified2 file — it does not attempt to aggregate/coalesce the records into an event, so no network data/packets are missed — everything is indexed as-is
- Unifiedbeat adds geolocation data for source/destination IPs — both version 4/6 addresses — allowing geographic visualizations
- Kibana may be used to search and visualize the data, or a custom app such as pakquery — which is designed for searching unified2 records
- While Sguil and ELSA are still very useful software, it’s 2016, and the time has arrived to use more modern software and analysis techniques. So replacing older software entirely on linux distributions like Security Onion and Kali with the combination of elasticsearch, unifiedbeat, kibana, or a custom app is now a reasonable possibility.
To give it a try, just do a git clone and use Go to compile a binary for any desired platform — even Windows (egads!). In the github repository you will find instructions, an overview diagram, and some sweet screenshots.
As of Feb. 29, 2016, a simple web app written in Go called pakquery is available to search the unified2 data that has been indexed by unifiedbeat into elasticsearch. It is definitely not as full featured as kibana, but it is tailored to searching unified2 records. I am hoping to expand pakquery to also search within osquery logs — osquery is a cool idea. The osquery logs created by osqueryd can be indexed into elasticsearch using filebeat. The combination of unifiedbeat, pakquery (or kibana), and osquery offers the equivalent of both HIDS and NIDS for servers and networks.
Also, there is a youtube video here.
Now is the time to gird one’s loins.
Beyond big data
Capturing and searching through a gigantic amount of data is almost impossible. Often security boils down to being concerned about inbound and outbound activity — you know, IP’s and ports. So there are other approaches to this problem which require less storage and may result in faster searches and detection:
- RITA — observe inbound/outbound communication (beacons) over time
- Flow indexer — extract/search only the IP addresses from log files
- Bro — instead or along with Snort and Suricata … however, this still creates a lot of log files but it’s pattern based
- ideal thoughts
- OpenNSM
