Python for Beginners — scrape AFL data and display on a webpage

Creating a dashboard site to display Australian Football League (AFL) data can be a great project for beginners looking to learn web scraping and web development. In this tutorial, we will use Python and the popular web scraping library, Beautiful Soup, to scrape data from the AFL website, and then use the Flask framework to create a simple web app to display the data. In the next tutorial we will swap out flask for Dash by Plotly and will use the Pandas library to store the data scraped.

First you’ll need to install python and set up your environment, if you need to know how to install python please start at my first tutorial:

A Beginners tutorial to Python: Part 1

Next, we'll create a new directory for our project and navigate into it:

mkdir afl-dashboard
cd afl-dashboard

Now, let's create and activate a virtual environment using the following command:

python -m venv venv
source venv/bin/activate

With the virtual environment set up, we can proceed to install the necessary libraries. We will be using Beautiful Soup to scrape the data and Flask to create the web app. To install these libraries, type the following command:

pip install beautifulsoup4 flask

Next, we'll create a new file for our Python script and name it scraper.py. In the command line, type:

touch scraper.py

Open the file in your text editor and import the necessary libraries at the top:

from bs4 import BeautifulSoup
import requests

We will use the requests library to make a GET request to the AFL website, and then parse the HTML using Beautiful Soup. Then we can extract the data we need. The sample code to get the data from a specific web page can look like this:

url = 'https://afl.com.au/stats/player-ratings/afl/total-disposals'
html = requests.get(url).text
soup = BeautifulSoup(html, 'html.parser')

You can then use the soup.find() method to find specific elements and extract the data, like player name and disposal numbers.

Now that we have scraped the data, let's move on to creating the web app. Create a new file named app.py. In the command line, type:

touch app.py

Open the file in your text editor and import the necessary libraries at the top:

from flask import Flask, render_template

We will create a new Flask application and define routes to handle requests to the root URL and a page to display the scraped data. Here is an example of how you can set up the Flask app:

app = Flask(__name__)

@app.route('/')
def index():
    return 'Welcome to the AFL Dashboard'

@app.route('/data')
def data():
    player_data = get_player_data()
    return render_template('data.html', data=player_data)

if __name__ == '__main__':
    app.run()

We also need to create a template for the data page, create a new folder named templates and a new file named data.html in it.

In the data.html file, we will use the Jinja2 template engine to display the scraped data. Here is an example of how you can set up the template:

<table>
    <tr>
        <th>Player Name</th>
        <th>Total Disposals</th>
    </tr>
    {% for player in data %}
    <tr>
        <td>{{ player.name }}</td>
        <td>{{ player.disposals }}</td>
    </tr>
    {% endfor %}
</table>

In this template, we are using a for loop to iterate over the data list and displaying the player name and total disposals in a table.

Now that we have scraped the data and set up the web app, we can run the script to start the development server:

python app.py

You should be able to access the web app by going to http://localhost:5000 in your web browser and see the welcome message. To view the scraped data, go to http://localhost:5000/data.

Keep in mind that this is just a basic example, there are many more things you can do with the scraped data and the web app, such as adding more data points, creating charts, or adding a search feature.

In this tutorial, we've covered the basics of web scraping and creating a simple web app to display the data. With a little creativity and some hard work, you'll be able to create something truly amazing.

Please go to the next tutorial in this series to take this a bit further and introduce Dash by Plotly and Pandas.

Hope that helps, and happy coding!