Disclaimer: This is not political. Just a data analysis for fun.
Introduction
In an ideal world, all the important information has been assembled and curated for use in a nice and orderly fashion. But they are not. Therefore, the skill to collect and present data is useful. Here, I present the method to answer some questions that were not readily available at the time. For this specific article, I was interested in the recent Malaysian General Election (PRU-15) results.
Main Objective: To illustrate the steps of doing web scraping and show cleaning of messy real-world data.
Question 1: Who are the elected Members of Parliament during PRU-15?
Question 2: How many votes do the winners get?
Strategy: 1. Get the data; 2. Data cleaning; 3. Visualisation
Tools: Python, Selenium, Pandas, Regex, & Matplotlib
Get the data
I found that a Berita Harian article provides the most complete results in one page. In this case, we need to use Selenium because the website is a javascript-based, therefore simple python “requests” do not work.
The content of the class ‘dable-content-wrapper’ was extracted and stored into a variable. Protip! Usually, we use the inspect feature in a web browser to know which specific name of a class or section of HTML.
Data cleaning
Initially, the text was split into a list of lines for further processing. Variable ‘parliaments’ is a list of tuples containing line numbers and patterns resembling parliaments numbers. Similarly, the variables ‘special’ & ‘members’ are for lines with winning candidates’ names. There were many variations because either the winners reported wrongly or had different text patterns.
The results were then combined into a sorted pandas data frame. Then split them into two data frames of parliaments and members. Then join them again to make them columns in the same data frame.
The results were dirty. We want the parliament number, parliament name, member’s name, and votes in different columns.
Tactics used here were:
- Split by space
- Extract matching patterns
- Replace unwanted text with an empty string
The newly extracted columns were combined into a single data frame with the appropriate columns name.
Next, generate the ‘coalition’ column based on knowledge (from other sources).
The final cleaned result can be seen below.
Quick Visualisation
Caveat: these absolute numbers can be misleading.
The top 50 members of parliament with the most votes are shown below. Some MPs are just too popular or their constituent is too populous.
The distribution of winners’ votes is shown below. Most winners get votes in the range of 0–60,000 votes. There are a small number of winners who get votes above 80,000.
Winners’ total votes by the coalition are shown below. However, this can be misleading because this is NOT total votes. This data excludes the votes of the losing candidates.
Conclusion
Question 1: Who are the elected Members of Parliament during PRU-15?
There are 221 winners and we got all their names, parties, and coalitions.
Question 2: How many votes do the winners get?
Many. Not all winners win the same number of votes.
Web scraping is a basic skill for a data professional. An understanding of the data and its context are essential in order to extract “correct” information. Without domain knowledge, we may pass the wrong and misleading information to the next analyst. They might use it to build whatever narrative based on their agenda. However, our task is to collect the most accurate data.
Challenges
- You need to know the context of the data.
- Too many unique patterns.
- Probably not the most efficient way.
GitHub
https://github.com/elvinado/MP-PRU15
Maybe Next…
Extract other information such as from the same page:
- Total voters
- Ethnic percentage of voters
- Spoil votes
- Votes majority count
- Redo the visualisation with a relative number (such as percentages)
