ES Kibana Tutorial: From Zero to Hero
10 Steps to Big Data Mastery
ElasticSearch and other NoSQL storage is great because it makes for a flexible and forgiving storage platform. When combined with multidimensional data of various types, it can be a tool for testing hypotheses quickly (let’s put the SCIENCE back in DS). Perform the following numbered steps in order.
Requirements:
- X-Pack
- Permissions (call your Apache Ranger person)
- Crisp Data with lots of columns
1. Dev Tools
Slide to the left side and enter Kibana’s interface. Through tutorials you can PUT
everything from movie scripts to star maps and also query data before an index pattern is created (more on that in step 2). We’ll call our index uspto_Rocks.
Example Queries:
GET uspto_Rocks/_count
-
GET uspto_Rocks/_search
{
"query": {
"match": {"THISISANINTFIELD": 12345678}
}
}
-
GET _count
{
"query": {
"wildcard": {"THISISANINTFIELD": 12*}
}
}
-
GET _count
{
"query": {
"range": {
"lte": 12000000,
"gte": 13000000
}
}
}
}
2. Create Index Pattern
An annoying step which must be done to activate a loaded data index (e.g. from Zeppelin or DevTools).
Management >> Index Patterns >> Plus Button >> Type in Index Pattern
On the “Configure an index pattern” page you must type in the index name exactly (uspto_rocks won’t do) and it will automatically search. Same a ‘time-field name’ to get it going.
3. Discover
SET YOUR TIMEFRAME First. Next select your index from the dropdown and have fun clicking around in the columns. The “Quick Count” functionality is particularly helpful in building intuition for later plotting.
4. Visualize and Dashboard
You make lots of interactive plots to test out hypotheses (we are scientists after all). Best part? They can be easily imbedded into HTML. Big Data Online Tableau.
Click Plus Button >> Select Type >> Choose Index >> SET YOUR TIMEFRAME in the upper right >>Add Y-Axis: {“ Aggregation*” , “Field” }>> [Add X-Axis, Split Series, Split Chart] >> Play Button
SAVE IT
Click Dashboard and put a lot of them together for fun. Don’t forget to embed elsewhere.
5. TimeLion
If you have streaming or time-series data, this is going to get even better.
.wbi(country=de, indicator=NY.GDP.MKTP.CD),
.wbi(country=fr, indicator=NY.GDP.MKTP.CD)
-
.wbi(indicator=NY.GDP.MKTP.CD).trend(),
.wbi(indicator=NY.GDP.MKTP.CD).color(#AAA)
GDP of FR, GM, and World in one line:
API for Quandl means you can also compare big stocks for immediate visual comparison. Derivatives and other transforms are available for some FAST higher level math (next tutorial idea: Taylor Rule economic model in 3 lines of Kibana).
6. Anomaly Detection
An RNN, probably an LSTM; to TCB.
8. Graph
Start with Date datatypes.
You might get Set fielddata=true error requiring a command like this if your job is to heavy on the compute:
PUT application_txn_score/_mapping/bucketType { "properties": { "interests": { "type": "text", "fielddata": true } }
}
Well I actually guess it was only in eight steps. Go import some data and build a few models in an hour. Don’t forget to embed and share the links! Kibana provides excellent tools for creating reproducible sharable research from the browser.
Credit to:
Kailen Nordhausen, Lisa Krymova , and TheNounProject for the Vector Art!