Real-time Coronavirus map with Elasticsearch
--
Hi there, My name is Sevban, I have created a Real Time coronavirus map with using Elasticsearch, Logstash, Kibana and Python Beautifulsoup package.
This map shows real-time Corona virus data from all over the world.
Map is updated dynamically every 5 seconds.
Since some countries no longer share their coronavirus case data, there may be data deficiencies in some fields.
This map can be edited and redesigned to reflect users’ specific needs and requirements with additional or customized graphics.
The preparation and implementation of the project took a total of 1.5 month.
I would like to express my thanks to Musab Doğan for their contribution in the construction, preparation, and implementation of this project.
For your questions, suggestions, and curiosities about the project, you can send an email to tumersevban@hotmail.com.
Project’s sample exist on Youtube.
Also Github.
My Linkedn profile here.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Let’s get started..
Initially, I have decided website which will be scraped.
It was; https://www.worldometers.info/coronavirus/
This website is sharing real-time Coronavirus data from all over the world. Due to countries not sharing their coronavirus data as they did at the beginning of the pandemic, there are inconsistencies in the data.
And I have written the code that will scrape this website. I have preferred to use the Python Beautiful Soup package. You can choose whatever you prefer.
This code block, scrapes data from website and writes to a .csv file properly. Also all of the codes about this project, available on my Github repository.
Here is my .csv file
The website contains values that aren’t compatible with Elasticsearch. Such as N/A, whitespaces and more. Don’t forget to convert those values to be compatible with Elasticsearch. You can separate a section in your Python code as follows:
We must create a mapping for our index. It’s crucial for every projects.
Now, we use logstash, that’s why we should write a .conf file.
I have removed some fields using filter plugins. I thought these were unnecessary.
Next step is to run this pipeline. There is something important here. We should add a setting which provides continuous data streaming.
To better understand you can read this article.
Good news! If you see this message on your screen, it means that your data has been successfully indexed.
Let’s take a look at the Elasticsearch side as well.
It appears that everything is fine in Elasticsearch.
Next step is create a data view. It exist on stack management part.
Our data view is successfully created.
Lets add some table to our data views
I generally prefer tag cloud in this project. these must be aggregation-based tables.
Set your tables whatever you want. And apply it to other tables. I did.
Here is how my first page looked. You can add the desired tables and templates. I have added a timestamp table. It shows last updates.
In Second page, I have added a map which shows world countries and their cases. Actually, this didn’t happen right away :) I encountered an issue with my index and received the following error.
And I have add to my python code which involves countries’ 2 digit codes and I repeated these steps. You can see here countries’ codes.
I have prefered choropleth map because I have only countries’ 2 digit codes. if you have geospatial fields you can choose other maps.
I added a bar that displays metric information about the country when you hover over the cursor.
I repeated the same thing for my other two pages as well.
Bar seems like that:
Consequently, I have a dashboard which consist of 3 page.
But there was a problem, how can i connected them with each other?
Markdown was solution for this. and I have created three icons that are connected to each other.
Markdown’s syntax same as github syntax. You can find detailed information here.
Here is how my second and third pages look like.
Every pages has different task.
Page 1: It provides information about the total number of cases worldwide.
Page 2: It provides information about the cases in different countries and includes a world map.
Page 3: It provides real-time updates from around the world on a map.
If I want the data in my CSV file to be continuously refreshed, my Python code should run continuously. I solved this by using a bash script.
Logstash is currently running and sending any changes in the CSV file to Elasticsearch. The Python code is also refreshing my CSV file every 5 seconds. There are no issues anywhere.
I shared the steps of my project with you. While working on this project, I learned a lot and enjoyed it. Please don’t hesisate reach out to provide feedback or suggestions. I have attached my accounts above :)
Have a good day!
