Web Scraping with Python in Indonesian E-Commerce (Tokopedia) Part 3: Present the Data as a Web Page
Well, we can see the text data like name, city, and price directly. But we can’t see the data directly from the CSV, right? And there’s an easy way to see all the data clearly, that is by creating a web page. Of course in this article, we will keep the page quite simple.
So in this section, we will try to create a web page to see the data that we already scraped from Tokopedia. We will use Flask in this section and we can install Flask by following this link.
Then let’s create the web page. For the page itself, we will use datatables it because we can see the data easily and there are many features that we can get if we create a table using that library. It will be loaded from our HTML page directly. So let's create an HTML file named view.html and fill it with this code.
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdn.datatables.net/1.12.1/css/jquery.dataTables.min.css">
<title>Product Data</title>
</head>
<body>
<nav class="navbar navbar-expand-md navbar-light bg-light">
<a class="navbar-brand" href="{{url_for('index')}}">Product Data</a>
</nav>
<div class="container">
<table id="datas" class="display table"style="width:100%">
<thead>
<tr>
<th>Name</th>
<th>Price</th>
<th>City</th>
<th>Image</th>
</tr>
</thead>
<tbody>
{% for item in datas %}
<tr>
<td>{{item["name"]}}</td>
<td>{{item["price"]}}</td>
<td>{{item["city"]}}</td>
<td>
<img src="{{item['img']}}" alt="" srcset="">
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
<script src="https://cdn.datatables.net/1.12.1/js/jquery.dataTables.min.js"></script>
<script>
$(document).ready(function () {
$('#datas').DataTable({
pagingType: 'full_numbers',
});
});
</script>
</body>
</html>Well, there are so many things that happened here.
We import important CSS libraries like bootstrap and datatables in these two lines.
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdn.datatables.net/1.12.1/css/jquery.dataTables.min.css">And we import important JavaScript libraries like jQuery, bootstrap, and datatables in these lines.
script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
<script src="https://cdn.datatables.net/1.12.1/js/jquery.dataTables.min.js"></script>Then we create a table tag in the body in these lines. Aside from creating a table, we will draw the row of the table using templating syntax from Flask that uses brackets like this {% %} and this {{ }}.
<table id="datas" class="display table" style="width:100%">
<thead>
<tr>
<th>Name</th>
<th>Price</th>
<th>City</th>
<th>Image</th>
</tr>
</thead>
<tbody>
{% for item in datas %}
<tr>
<td>{{item["name"]}}</td>
<td>{{item["price"]}}</td>
<td>{{item["city"]}}</td>
<td>
<img src="{{item['img']}}" alt="" srcset="">
</td>
</tr>
{% endfor %}
</tbody>
</table>Then finally we will make the table into a datatable's a table with this code.
<script>
$(document).ready(function () {
$('#datas').DataTable({
pagingType:'full_numbers',
});
});
</script>After we create the HTML page, now we need to make the web page can be accessed after the scraping process is finished.
So we will open the main.py file again and modify it to be like this.
from scraper import Scraper
from csv_creator import CSVCreator
from flask import Flask, render_template
scraper = Scraper()
datas = scraper.get_data()
app = Flask(__name__, template_folder=".")
@app.route('/')
def index():
return render_template('view.html', datas=datas)
if __name__ == "__main__":
CSVCreator.create_csv(datas)
app.run()There are some changes that I made from the article before.
I move the codes of the scraper to the outside of the main section because I want the data can be accessed by the Flask's routing.
Then I add this line so we have a Flask app.
app = Flask(__name__, template_folder=".")It will be using the current directory as the directory to search for templates because of template_folder="." parameters. It must be done because we did not create any templates folder which is the default folder that Flask app will search for HTML files.
Then I define the routing and the function that will handle the routing with these lines.
@app.route('/')
def index():
return render_template('view.html', datas=datas)We can see that it will use view.html as the HTML file that will be rendered to the browser and passing the data with datas=datas the argument.
And the last is to run the Flask app with this line.
app.run()If we run the main.py file and the scraping process is done, we will get a CSV file and the Flask app can be seen running at the terminal.
And if we open the address that showed at the terminal, then we will see this page.
And that’s all from me. Of course, we can improve the app more by adding some features, grabbing more useful data, present with a better format, etc. And this is the link to the repository if you want to see the code directly.
If you want to discuss something just contact me on my LinkedIn.
You can check the overall step on my blog with this link.
Thank you very much and goodbye.
