Build a Flask web app by scraping a website & display the data using Chart.js

How I created a web app in Python using Flask that scrapes the Ministry of Health & Family Welfare website and displays the state-wise stats of coronavirus in India.

Photo by visuals on Unsplash

Introduction

As the whole world is in the grip of coronavirus, it is rather important to keep ourselves informed of the latest stats. Keeping this in view, I decided to build a web-app using Python, Flask & Chart.js that will pull the data from the Ministry of Health & Family Welfare’s website & display that information in form of colorful and interactive charts. In this post, I will show you how I built this app.

You can grab the entire code from my GitHub repository.

shashank-ssriva/India-COVID-19-Stats

I have Dockerised & then deployed the app to Heroku as well. Please go to the below URL to see the live app.

https://india-covid-stats.herokuapp.com/

I will show you how you can Dockerise a Flask web-app & deploy that on Heroku in my upcoming post.

Tools/Technologies used

1. Python 3.7.6 — Web-app’s backend.
2. BeautifulSoup — Scrapes the information from the website.
3. Flask — Python’s web-development framework
4. Chart.js — Displays the information as beautiful, colorful, interactive charts.

Web-app screenshot

Below is what the finished web-app looks like (it is a fullscreen screenshot). As mentioned above, you can view the app live at https://india-covid-stats.herokuapp.com/. These charts are interactive & you can hover your cursor over the charts for additional information.

https://india-covid-stats.herokuapp.com/

Project structure

Since it is a Flask project, we need to maintain a specific directory structure for our app to work. Below is the project skeleton.

I am excluding the non-important files & directories.

root@shashank-mbp /U/a/D/CoronaVirus-Status-India# ls -lh
-rw-r--r--   1 root  staff   2.6K Mar 29 11:42 app.py
-rw-r--r--   1 root  staff   121B Mar 27 19:12 requirements.txt
drwxr-xr-x   5 root  staff   160B Mar 27 09:26 static
drwxr-xr-x   3 root  staff    96B Mar 28 22:52 templates
root@shashank-mbp /U/a/D/CoronaVirus-Status-India# tree static templates
static
├── css
│   ├── dashboard.css
│   ├── uikit-rtl.css
│   ├── uikit-rtl.min.css
│   ├── uikit.css
│   └── uikit.min.css
├── img
│   └── shashank-srivastava.jpeg
└── js
    ├── chartScripts.js
    ├── uikit-icons.js
    ├── uikit-icons.min.js
    ├── uikit.js
    └── uikit.min.js
templates
└── index.html

Steps performed

1. Installed required Python modules

For this, I created a requirements.txt file with the below content & executed it using pip.

pytest
flake8
python-dateutil
coverage
bs4
click
coverage
Flask
itsdangerous
Jinja2
MarkupSafe
requests
Werkzeug
selenium

pip in action.

root@shashank-mbp /U/a/D/CoronaVirus-Status-India# pip install -r requirements.txt

2. Scraped the Ministry of Health & Family Welfare’s website to get the data.

I then created varous Python lists to obtain the values I needed. For this, I created a file called app.py.

@app.route('/', methods=['POST', 'GET'])
def main():
    def extract_contents(row): return [x.text.replace('\n', '') for x in row]
    URL = 'https://www.mohfw.gov.in/'
    response = requests.get(URL).content
    soup = BeautifulSoup(response, 'html.parser')
    table = soup.find('div', {'id': 'cases'})
    state_wise_stats = []
    all_rows = table.find_all('tr')
    for row in all_rows:
        stat = extract_contents(row.find_all('td'))
        if stat:
            if len(stat) == 5:
                # last row
                stat = ['', *stat]
                state_wise_stats.append(stat)
            elif len(stat) == 6:
                state_wise_stats.append(stat)

Since this post is not to show you how the code works, I will skip this part. My aim here is to tell how I wrote the web-app. You can check the code to see how I got the values to be used in the charts.

3. Created charts using Chart.js

Chart.js is an awesome chart library that features beautiful charts and they can be created without much pain.

Once I extracted all the values, like the number of confirmed coronavirus cases in India, I drew the charts using these values inside my index.html page.

Below is an example of how to create a pie-chart using Chart.js.

<!-- panel -->
    <div class="uk-width-1-1">
    <div class="uk-card-body">
    <div class="chart-container">
     <canvas id="pie-chart" width="1200" height="600" style="display: block; height: 300px; width: 600px;"></canvas>
     <script>
     var chartData = {
      labels: ['Confirmed cases', 'Deaths', 'Cured/Discharged'],
       datasets: [{
        label: 'Pie chart',
        data: [{{ total_cases }}, {{ total_casualties }}, {{ num_people_cured }}],
         spanGaps: false,
         backgroundColor: ['orange', 'red', '#90EE90' ],
        }]
       }
       // get chart canvas
       var ctx = document.getElementById("pie-chart").getContext("2d");
       // create the chart using the chart canvas
       var myChart = new Chart(ctx, {
        type: 'pie',
        data: chartData,
        options: {
         title: {
          display: true,
          text: 'Data at a glance',
         }
        }
       });
       </script>
      </div>
     </div>
    
   </div>
<!-- /panel -->

4. Put it all together.

Once my code was ready, I simply had to execute the below command to start my web-app. Please go to my GitHub repository to get the entire code.

root@shashank-mbp /U/a/D/CoronaVirus-Status-India# python3 app.py
 * Running on http://0.0.0.0:5002/ (Press CTRL+C to quit)

This marks the end of this post. I hope it was informative.