Elastic{on} Frankfurt recap
A few days ago another event from Elastic{on} Tour series took place in Frankfurt welcoming around 200 developers/architects or whoever likes data engineering.
If you are not familiar with Elastic{on} Tour series short intro for you: It is a smaller, one-day version of original Elastic{on} which began with the idea to bring Elastic{on} closer to its fans all around the world. Understandably — the original Elastic{on} is located in San Francisco and usually takes 3 days.
Content
Even though Elastic{on} Tour in Frankfurt took just single day the content was very concentrated and every single minute was worth of visiting.
The topics of talks were sorted into three categories — the news from Elasticsearch and its ecosystem, the case studies and how-to/best practices. Apart from talks the Ask-me-anything booth with a dozen of Elastic tech. guys present was the right place to discuss problems or get insider info about new stuff.
The news
As the list of new things is quite long, here are just the most interesting ones:
- Canvas in Kibana
A feature which helps you create visually appealing infographics directly in Kibana — with live data feeds from Elasticsearch so your Infographics will never get outdated.
Apart from Infographics, it can be used as live billboards, presentations with most current data, reports and lot of other cases, where you need to visualize over current data. How cool is that?!
You can find more info and how to try it out on dedicated page http://canvas.elastic.co/
- Field aliases
The index aliases are well known for everyone working with Elasticsearch. Now the fields might have aliases in the same manner as indices. - Add data UI in Kibana
The UI for uploading CSV files in Kibana which stores the data from such file in Elasticsearch. This way we can start fiddling with Elasticsearch and Kibana without the need of external data pump like Logstash which fills data into Elasticsearch - Elastic Common Schema
ECS defines a set of fields which helps with the process of data ingestion in order to make a correlation between different data sources easier. It is still in development and authors are waiting for comments from users. - File integrity and Linux Kernel metrics
New metricsets for Auditbeat for watching over Linux kernel operations and files integrity — very useful for critical systems with high-security level. - Netflow and ArcSight modules in Logstash/Kibana
Two modules for ingesting/parsing and visualizing data from Netflow and ArcSight - Persistent Queue in Logstash and Spool Queue in Beats
Persistent queue in Logstash is queue synchronized into a disk which lies between inputs and filters/outputs. After the successful parsing and output, the raw events are erased from this queue. Thanks to its persistent nature — even in the case of Logstash crash, we won’t lose any data — when the Logstash is back up, the parsing is replayed from this queue.
In the same principle, Spool Queue in Beats stores events into a file on local disk and only after successful transport into the target system (usually Logstash) they are removed from such queue
Both these new functionalities prepare the foundations for End-to-End acknowledge of event transport - Data rollups in Elasticsearch
With this functionality, we can make a smaller copy of the original — raw index which holds only fields useful for long-term reporting and visualizations. The original index then might be stored on tapes or any other sort of cheap space. Right now the only API developed but UI for Kibana is in progress
You can find more info in this blog post. - Discover search field with suggestions in Kibana
Awesome functionality for not-so-technical people, the search field on Discover page in Kibana will have suggestions so even Business people will be able to search over data without the help of developers (yeehaw! :-) - Adaptive replica selection
Simplified — the cluster won’t route search requests to nodes that are not in a good condition. There is very good and almost science-paper like blog about Adaptive replica selection on elastic.co https://www.elastic.co/blog/improving-response-latency-in-elasticsearch-with-adaptive-replica-selection - Frozen indices
Ability to freeze certain indices in order to save resources. Unlike the Indices closure which makes them inaccessible for search, the documents in frozen indices will still be searchable but it won’t be possible to write new documents in such indices. - Index lifecycle management
An alternative to a Curator tool but will have UI for administering jobs in Kibana - UI for logs exploration in Kibana
Makes the transition from reading logs in the shell to Kibana little bit easier. It will include some cool features like live streaming (no more `tail -f`), infinite scrolling, a timeline with histogram and quick navigation and keyword highlighting.
Use cases
The most interesting use case was presented by guys from Mayr-Melnhof Karton company. MM Karton is world’s major cardboard producer with plants all over the world and as they are using modern machines full of various sensors the idea of gathering data into one single data store with real-time monitoring of production process comes somehow naturally. And, as their case was presented on Elastic[on}, it is clear which technology to store and visualize such data they chose. Of course — Elasticsearch and Kibana. Thanks to real-time monitoring and also long-term reporting, they were able to reduce manufacturing costs and improve the overall quality of produced cardboards.
It is clear, that Elasticsearch is very versatile (well the name suggest it :-) and apart from common use cases like log analysis, system monitoring, security auditing and of course full-text search, there are other cases which works with the material world. At the end of the day, this connection between the material world and IT world can help to produce a better product and save resources.
Best practices
This part deserves much more space because of the complexity of the topic so I will sum it up in a separate blog post which will be published in near future.
The end
This was just very brief summarization of content presented in one single day, and in reality, there were even much more things presented and discussed — so you can see the word “concentrated” from the post’s intro might be insufficient.
All-in-all I can recommend you to attend some Elastic{on} event — if you are interested in data engineering, you won’t be disappointed and I’m looking forward to next cool features from Elastic.
