ES Kibana Tutorial: From Zero to Hero

10 Steps to Big Data Mastery

Don’t Let All That Data Get you Down

ElasticSearch and other NoSQL storage is great because it makes for a flexible and forgiving storage platform. When combined with multidimensional data of various types, it can be a tool for testing hypotheses quickly (let’s put the SCIENCE back in DS). Perform the following numbered steps in order.

Requirements:

• X-Pack
• Permissions (call your Apache Ranger person)
• Crisp Data with lots of columns

1. Dev Tools

Slide to the left side and enter Kibana’s interface. Through tutorials you can PUT everything from movie scripts to star maps and also query data before an index pattern is created (more on that in step 2). We’ll call our index uspto_Rocks.

Example Queries:

GET uspto_Rocks/_count

-

GET uspto_Rocks/_search
{
     "query": {
          "match": {"THISISANINTFIELD":  12345678}
     }
}

-

GET _count
{
     "query": {
          "wildcard": {"THISISANINTFIELD":  12*}
     }
}

-

GET _count
{
     "query": {
          "range": {
               "lte":  12000000,
               "gte":  13000000
               }
          }
     }
}

2. Create Index Pattern

An annoying step which must be done to activate a loaded data index (e.g. from Zeppelin or DevTools).

Management >> Index Patterns >> Plus Button >> Type in Index Pattern

On the “Configure an index pattern” page you must type in the index name exactly (uspto_rocks won’t do) and it will automatically search. Same a ‘time-field name’ to get it going.

3. Discover

SET YOUR TIMEFRAME First. Next select your index from the dropdown and have fun clicking around in the columns. The “Quick Count” functionality is particularly helpful in building intuition for later plotting.

4. Visualize and Dashboard

You make lots of interactive plots to test out hypotheses (we are scientists after all). Best part? They can be easily imbedded into HTML. Big Data Online Tableau.

Click Plus Button >> Select Type >> Choose Index >> SET YOUR TIMEFRAME in the upper right >>Add Y-Axis: {“ Aggregation*” , “Field” }>> [Add X-Axis, Split Series, Split Chart] >> Play Button

SAVE IT

Click Dashboard and put a lot of them together for fun. Don’t forget to embed elsewhere.

5. TimeLion

If you have streaming or time-series data, this is going to get even better.

Getting Started with Time Series Analysis in Kibana

.wbi(country=de, indicator=NY.GDP.MKTP.CD),
     .wbi(country=fr, indicator=NY.GDP.MKTP.CD)

-

.wbi(indicator=NY.GDP.MKTP.CD).trend(),
     .wbi(indicator=NY.GDP.MKTP.CD).color(#AAA)

GDP of FR, GM, and World in one line:

API for Quandl means you can also compare big stocks for immediate visual comparison. Derivatives and other transforms are available for some FAST higher level math (next tutorial idea: Taylor Rule economic model in 3 lines of Kibana).

6. Anomaly Detection

An RNN, probably an LSTM; to TCB.

The two ‘warning’ triggers seam weak

Adam Gradient Descent makes that error rate drop; Learning

8. Graph

Start with Date datatypes.

You might get Set fielddata=true error requiring a command like this if your job is to heavy on the compute:

PUT application_txn_score/_mapping/bucketType
    {
     "properties": {
          "interests": {
               "type":     "text",
               "fielddata": true
               }
          }
     }

Well I actually guess it was only in eight steps. Go import some data and build a few models in an hour. Don’t forget to embed and share the links! Kibana provides excellent tools for creating reproducible sharable research from the browser.

Credit to:

Kailen Nordhausen, Lisa Krymova , and TheNounProject for the Vector Art!