Bitcoin Analytics with ELK (Elasticsearch, Logstash, Kibana)

In this (first) post, I will show you how to study the impact of news on bitcoin’s change price. We will use the ELK stack (Elasticsearch, Logstash and Kibana) to collect, store and visualize the bitcoin’s price changes and some of the most relevant world news, but don’t be scared it will be very easy!

I’m assuming that you have Elasticsearch, Kibana and Logstash (version >= 6.2 ) installed.

First of all we will use the free web service https://www.coindesk.com/ to fetch the bitcoin’s change value respect to the main currency (USD, GBP and EUR). Here there is an example of the raw json returned by the service:

We will also fetch news from rss sources, using the https://rss2json.com/ api. It would be possible to use directly the rss sources and parse the xml using Logstash, but the advantage of this approach is that not only the xml is translated into json (almost ready to be indexed in elastic) but also the schema of the json will be independent of the rss source: in this way we can easily add or remove sources without impact on parsing and indexing.

Here there is an example of the raw json returned by the news feed:

To integrate this two information sources, we will simply use Logstash: so don’t be scary! Here there is our logstash conf file:

We use two different input: one for bitcoin price and one for rss news feed (this input may have multiple urls, I provide only an example of bbc world news rss).

In the filter we clean the data and we prepare to be indexed on elastic (in particular each news article is returned into an array, but we need to split that array in separate documents).

Finally here are reported some visualize done using Kibana:

In particular we can see how is changing the bitcoin price, and see the relevant news of the selected time range. In this way you may find a pattern of certain news have on the bitcoin’s price and make better investment decision!