What is the ELK Stack and how to apply it to information security
Written by Jad Hamdouch & Ismael Bouarfa.
Securing the IT infrastructure is not only about deploying hard defensive tools around the assets. It is more about preventing and detecting any anomaly. That requires a monitoring approach and many tools associated with this activity. Monitoring is all about the capacity of collecting security events and translating them into actions (raising alerts) and information (KPIs).
Not every company can afford massive deployments. However there is importance in tracing what is happening in the IT activity and in raising alarms when something is going wrong. Monitoring performance and anomalies is an essential step when securing a company’s infrastructure.
All IT equipment produces logs. These are compilations of every event that occurs (connection attempts, time, origin, destination…). Generally, in IT ecosystems we deal with operational and security logs. These different logs come from our hardware and software infrastructure ( routers, switches, applications, databases, servers…).The analysis of such logs can enable us to have a general view of the activity, investigate an incident, detect suspicious activities…
Given the previous context, security teams are looking for a solution to manage and visualize this activity. The ELK stack is a combination of three open source tools (ElasticSearch, Logstach and Kibana) that allows you to store a large amount of logs in order to visualize them easier.
ElasticSearch is a distributed RESTful search and analysis engine that allows us to store large amounts of information in JSON format in a powerful indexing engine. Indeed, ElasticSearch can be seen as a search engine that uses Lucence (Apache library) to index the content.
Logstash is a data collector and data processor based on a set of different plug-ins. These plug-ins allow you to easily configure the tool to collect, load and transfer data in a number of different architectures and send it to ElasticSearch.
Kibana provides the UI accessible by web browser to query ElasticSearch. This tool allows us to have several different visualisations like histograms, linear graphs, pie charts, sunbursts, and more. You can also use the Vega language to design your own visualizations. You will notice that you won’t need advanced knowledge to understand how to query ElasticSearch from Kibana.
Setting up the ELK stack requires some technical knowledge, especially for the configuration of Logstash, but the benefits are numerous:
- Adaptable architecture: Because each element of the stack can work independently and can be deployed on different devices or areas of the architecture.
- Fast searches, almost in real time: ElasticSearch, according to the CAP theorem, is in the AP (Availability-Partition Tolerance) pair, which seeks to provide a record response time while distributing the data.
- Open Source: New features and plugins to detect anomalies are under development or in test phase. The Machine Learning component has just been added to the ELK stack, so several options will be available in the near future.
Note that ElasticSearch is oriented towards BigData & Hadoop and that the entire solution already has different cloud implementation possibilities.
The implementation of ELK means the implementation of innovative and open source technologies oriented Big Data that provide an excellent tool for investigation, forensics and analysis of security events.
The limitations of the ELK Stack
- Complexity: In addition to the set up of the ELK Stack, it requires much time, work and effort to extract an added value from the stack.
- Security of the Stack itself : Almost all security settings are not enabled by default. When deploying the stack it is important to deploy all the security measures to protect the ElasticSearch cluster from the internet.
- No official courses or free certifications available : There are no official courses available for free as Splunk did in the first releases.
Benchmark ELK vs Splunk
Splunk is a set of products/solutions that offers a response to the same needs as the ELK stack. It works differently as the ELK Stack and it is also more complete. Note that Splunk is not open source and it is owned by Splunk Inc.
Roadmap to ELK success
Not all ELK deployments were success stories. It’s recommended to start deploying a proof of concept (POC) to understand how it works, the limitation and future needs. Some baselines are listed:
- Choose the equipments and applications to supervise.
- Choose the correct way to forward logs to the Logstash (we recommend using Beats).
- Choose the most adapted architecture (deploying and ELK server or putting every tool in a separated part of the architecture).
- Work with Grok in Logstash (some Regex are available in the Internet).
- Start shipping logs and creating dashboards to familiarize with Kibana.
ELK’s own security
It is important to secure the security tools. We are going to analyse ELK Security based in the CIA triad (Confidentiality — Integrity — Availability) :
Confidentiality: Protect the ElasticSearch database disabling access from any IP (don’t expose it to public networks or unsecure subnets).
Integrity: Only Kibana needs to access the ElasticSearch. Protect the integrity of your dashboards and log information enabling access control with the Nginx Proxy or with firewall rules.
Availability: Backup your logs if Logstash is not available sending them to another Linux server.
Final words
When it comes to absorbing large amounts of unstructured data and visualize it, think: “ELK Stack” can do the job. The ELK Stack is a good tool to collect, store and query data. The idea is to create dashboards with the power of Kibana, store the data in the noSQL database of ElasticSearch and absorb the data with Logstash.
The ELK Stack has also been deployed it other use cases as marketing analysis or finance. Find out more about marketing analysis here and finance here.
