Mastering Data Scraping using Python — Get the Latest Time an Earthquake happen in Indonesia!
Data scraping is an activity to retrieve data programmatically from the website as needed. This action can actually be done manually, but by programming it, you will save a lot of time. In addition, multiplying websites from which data is taken, sometimes does not require any coding work at all. Things like this will not happen if the scraping process is still done manually using mouse clicks, copy paste and so on.
There are several technology that let you scrape website, most prominent one is Python: by its simplicity and scripting nature, Python code can easily get the job done in smallest time compare to other.
What website to scrape?
Indonesia is country with earth quake: lots of it. To inform citizen, our gorverment setup this site: https://www.bmkg.go.id/ to let us see the latest earth quake.
Wouldn’t it be awesome if we can scrape this?
Use Google Chrome inspect element, and try to get the latest time the earth quake happens:
See? Have a look at that div=>ul=>li=>span. That’s our target! That what are going to scrape!
How to Scrape using Python?
In Python paradigm, there one prominent solution for data scraping: that is .. BeautifulSoup4 accompanied with Requests. Let’s see how it is done.
First, create a repository in Github, like mine here: https://github.com/swdevbali/Scrape_BMKG
And put your requirements.txt as below:
And these below is your first code in the world of Scraping! Yeiy!
If you run it, you will have this result:
This means, your code already able to get all the html text from bmkg. Congrats!
Getting the Correct Data using BeatifulSoup4
Request functions is just to, well, request the url that you specify. To extract data, you will need BeautifulSoup4. Have a look at this code that simply extract BMKG web page title
Run it! What do you see?
But we don’t want to scrape the title. We want to get the latest time the earthquake occur! How?
See code below!
The result? Let me screenshot it for you!
With those code, you can get the latest time an Eearthquake occur in Indonesia.
Now, your task: how to get all the relevant code such as location, deep and magnitude?
See you in the next article!