@OwlHnet: SoftwareTAP for GCLOUD and AWS.
I like to see traffic. sorry, I do. While logs are OK and Flows provide great statistics, I usually prefer to be able to have access to full traffic and allow network IDS solutions like Suricata and Bro to do the analysis, but in a cloud environment seems to be like a crazy thing.
With a bit of work, I’ve found the way to have the visibility I would like to have. As I don’t want to change cloud architecture, a ‘box-in-the-middle’ like a gateway, firewall, router, etc. is something, while easy, not always possible. Where can I find the info? Cloud Logs and flows are part of cloud offering but there is no full traffic. Where then? the traffic related to an instance can be found in the instance.
The instance must survive
The most important thing is that you shouldn’t kill the instance. among other things, you must take care of CPU, Memory, Disk Storage, Disk access performance, Network Traffic. But if you can do what you want without impact, then you are ready to go.
The listener
Maybe in the future, we should build something, but right now a lightweight tool like tcpdump is doing the work allowing us to do full traffic capture, export to a standard and portable traffic format as PCAP, save multiple files while working, … and all this is done without killing the instance.
Well, it will need some help. someone to start the listener, verify that resources like CPU, Disk, Memory, Traffic are ok… this is the OwlH Master job.
Full versus Targeted traffic capture
While I want to see the full detail of the traffic I want to analyze, maybe I don’t need to capture everything. Using the right filters will help saving resources as well as providing just the right traffic you want to analyze that is related to your instance.
Usually, the way to decide if you want/need to capture something is to ask your self, What would I do with this traffic? if your answer is ‘nothing’, maybe you don’t need to capture it. If I use something like Pareto principle and the 80/20 rule, for many events, roughly 80% of the effects come from 20% of the causes, so the 20% of the traffic can show me the 80% of the real significant issues on my network. But anyway, no one knows your network better than you, well, no one should.
The Cloud architecture
The Cloud cost is something to take into account. Please be sure to identify your right environment. Something that may help is to define an architecture that will provide an easy to audit environment. things, like deploy different VPCs or deploy administration networks, will for sure help to capture, transport and analyze your traffic without killing your wallet.
Suggestion: deploy a secondary interface in your instances, use this to build an administration network, and use this admin network to transport captured traffic.
The orchestration
Someone must be responsible for running all this together and be sure that nothing is killing the functionality. In the end, killing what you want to protect is not a good idea. This is what the OwlH Master instance does. In this OwlH Master instance, we will register each instance to be analyzed, will specify capture parameters globally and/or per instance, will do all monitoring to be sure capture works under right performance parameters, and will run the analysis of captured traffic.
The analysis
We rely on two old friends: Suricata and Bro as the Network IDS open source solutions to analyze the captured traffic. Both will run in the OwlH Master instance. A handicap is to analyze multiple PCAP files. Creating a Dummy interface and running both Network IDS listening to it is going to save our day. A solution as Tcpreplay will definitely do the trick injecting the PCAP file to the Dummy interface allowing a continuous traffic analysis that has been stored in different files.
OwlH Master will also provide analysis configuration capabilities like rules and policies management as well as network traffic specific parameters related to Interface and Network IDS configuration
The visualization
After all this is done, we will have a good set of alarms as the result of this work. This is what should be escalated to a central console to take the right action. Maybe action will be alerting, maybe report, maybe store it, being able to search and visualize or as usual, all of them.
We manage output to be transported to the right visualization and storage solution. this SoftwareTAP can be integrated directly to an ELK solution or what is helping more, it can be integrated with a well-known solution like WAZUH.
Closing this long tale
I found that there is an Open Source way, without too much impact, in which one you can do a full traffic analysis in a cloud environment like GCLOUD or AWS. There are more quarters to play, sure.
More info at www.owlh.net
http://documentation.owlh.net/en/latest/main/OwlHsoftwaretap.html
Send your comments, suggestions — contact@owlh.net
