Web Scraping Tutorial + Project. Scraping Stack Overflow

Naveen Verma
Jun 14 · 10 min read
Photo by Chris Ried on Unsplash

In this blog post, I will show how to scrape websites using Python and Beautiful Soup.

We will be scraping StackOverflow website and:

  • Goal 1: List Most mentioned/tagged languages along with their tag counts
  • Goal 2: List Most voted questions along with with their attributes (votes, summary, tags, number of votes, answers and views)

We will divide our project into the above mentioned two goals. Before starting our project, we need to understand few basic concepts regarding Web Scraping.

You can get the code for this Blog Post here.


Web Scraping Basics

When we visit a page, our browser makes a request to a web server. Most of the times, this request is a GET Request. Our web browser then receives a bunch of files, typically (HTML, CSS, JavaScript). HTML contains the content, CSS & JavaScript tell browser how to render the webpage. So, we will be mainly interested in the HTML file.

HTML

HTML has elements called tags, which help in differentiating different parts of a HTML Document. Different types of tags are:

  • html - all content is inside this tag
  • head - contains title andd other related files
  • body - contains main cotent to be displayed on the webpage
  • div - division or area of a page
  • p - paragraph
  • a - links

We will get our content inside the body tag and use p and a tags for getting paragraphs and links.

HTML also has class and id properties. These properties give HTML elements names and makes it easier for us to refer to a particular element. can be shared among multiple elements and an element can have moer then one class. Whereas, needs to be unique for a given element and can be used just once in the document.

Requests

The requests module in python lets us easily download pages from the web.
We can request contents of a webpage by using requests.get(), passing in target link as a parameter. This will give us a response object.

Beautiful Soup

Beautiful Soup library helps us parse contents of the webpage in an easy to use manner. It provides us with some very useful methods and attributes like:

  • find(), select_one() - retuns first occurence of the tag object that matches our filter
  • find_all(), select() - retuns a list of the tag object that matches our filter
  • children - provides list of direct nested tags of the given paramter/tag

These methods help us in extracting specific portions from the webpage.

Tip: When Scraping, we try to find common properties shared among target objects. This helps us in extracting all of them in just one or two commands.

For e.g. We want to scrap points of teams on a league table. In such a scenario, we can go to each element and extract its value. Or else, we can find a common thread (like same class, same parent + same element type) between all the points. And then, pass that common thread as an argument to BeautifulSoup. BeautifulSoup will then extract and return the elements to us.


Goal 1: Listing most tagged Languages

Now that we know the basics of Web Scraping, we will move towards our first goal.

For Goal 1, we have to list most tagged Languages along with their Tag Count. First, lets make a list of steps to follow:

Download Webpage from stackoverflow.

Parse the document content into BeautifulSoup

Extract Top Languages

Extract their respective Tag Counts

Put all code together and join the two lists

Plot Data

Let’s import all the required libraries and packages

Downloading Tags page from StackOverflow

We will download the tags page from stackoverflow, where it has all the languages listed with their tag count.

200

Parsing the document into Beautiful Soup

bs4.element.Tag

Extract Top Languages

In order to acheive this, we need to understand HTML structure of the document that we have. And then, narrow down to our element of interest.

One way of doing this would be manually searching the webpage (hint: print body variable from above and search through it).
Second method, is to use the browser's Developr Tools.

We will use this second one. On Chrome, open tags page and right-click on the language name (shown in top left) and choose Inspect.

You will see that the Language name is inside a tag, which in turn is inside a lot of div tags. This seems, difficult to extract. Here, the class and id, we spoke about earlier comes to our rescue.

Figure 1. Stack Overflow Tags Page

However, if we look more closely, we can see that the a tag has a class of post-tag (refer Figure 1.). Using this class along with a tag, we can extract all the language links in a list.

Next, using list comprehension, we will extract all the language names.

Extract Tag Counts

To extract tag counts, we will follow the same process.

On Chrome, open tags page and right-click on the tag count, next to the top language (shown in top left) and choose Inspect.

Here, the tag counts are inside span tag, with a class of item-multiplier-count (refer Figure 1.). Using this class along with span tag, we will extract all the tag count spans in a list.

Next, using list comprehension, we will extract all the Tag Counts.

Put all code together and join the two lists

We will use Pandas.DataFrame to put the two lists together.
In order to make a DataFrame, we need to pass both the lists (in dictionary form) as argument to our function.

Plot Data

DataFrame: Top Languages and Tag Counts

Now, we will plot the Top Languages along with their Tag Counts.

Language vs Tag Count (Bar Graph)

Goal 2: Listing most voted Questions

Now that we have collected data using web scraping one time, it won’t be difficult the next time. For Goal 2, we have to list questions with most votes along with their attributes, like:

  • Tags
  • Number of Votes
  • Number of Answers
  • Number of Views

I would suggest giving it a try on your own, then come here to see my solution.

Similar to previous step, we will make a list of steps to act upon:

Download Webpage from stackoverflow

Parse the document content into BeautifulSoup

Extract Top Questions

Extract their respective Summary

Extract their respective Tags

Extract their respective no. of votes, answers and views

Put all code togther and join the lists

Plot Data

A different Scraping Function

In this section, we will use a different function to return BeautifulSoup objects Now, we will make use of select() and select_one(). While find_all uses HTML tags, select uses CSS Selectors in the filter. Personally, I tend to use the latter one more.

For example:

  • p a — finds all a tags inside of a p tag.
  • div.outer-text - finds all div tags with a class of outer-text.
  • div#first - finds all div tags with an id of first.
  • body p.outer-text - finds any p tags with a class of outer-text inside of a body tag.

Downloading Questions page from StackOverflow

We will download the questions page from stackoverflow, where it has all the top voted questions listed.

Here, I’ve appended ?sort=votes&pagesize=50 to the end of the defualt questions URL, to get a list of top 50 questions.

Parsing the document into Beautiful Soup

Extract Top Questions

On Chrome, open questions page and right-click on the top question and choose Inspect.

Figure 2. Question, Summary, Votes, Answers, Views

We can see that the question is inside a tag, which has a class of question-hyperlink(refer Figure 2.).

Taking cue from our previous Goal, we can use this class along with a tag, to extract all the question links in a list. However, there are more question hyperlinks in sidebar which will also be extracted in this case. To avoid this scenario, we can combine a tag, question-hyperlink class with their parent h3 tag. This will give us exactly 50 Tags.

List comprehension, to extract all the questions.

Extract Summary

On Chrome, open questions page and right-click on summary of the top question and choose Inspect.

We can see that the question is inside div tag, which has a class of excerpt(refer Figure 2.). Using this class along with div tag, we can extract all the question links in a list.

List comprehension, to extract all the questions.

Here, we will also use strip() method on each div’s text. This is to remove both leading and trailing unwanted characters from a string.

Extract Tags

On Chrome, open questions page and right-click on summary of the top question and choose Inspect.

Figure 3. Tags per question

Extracting tags per question is the most complex task in this post. Here, we cannot find unique class or id for each tag, and there are multiple tags per question that we need to store.

To extract tags per question, we will follow a multi-step process:

  1. As shown in Figure 3, individual tags are in a third layer, under two nested div tags. With the upper div tag, only having unique class (summary).
  • First, we will extract div with summaryclass.
  • Now notice our target div is third child overall and second div child of the above extracted object. Here, we can use nth-of-type() method to extract this 2nd div child. Usage of this method is very easy and few exmaples can be found here. This method will extract the 2nd div child directly, without extracting summary div first.

2. Now, we can use list comprehension to extract a tags in a list, grouped per question.

3. Now we will run a for loop for going through each question and use list comprehension inside it, to extract the tags names.

Extract Number of votes, answers and views

On Chrome, open questions page and inspect vote, answers and views for the topmost answer.

No. of Votes

  • They can be found by using span tag along with vote-count-post class and nested strong tags (refer Figure 2.).

List comprehension, to extract vote counts.

No. of Answers

  • They can be found by using div tag along with status class and nested strong tags. Here, we don't use answered-accepted because its not common among all questions, few of them (whose answer are not accepted) have the class - answered (refer Figure 2.).

List comprehension, to extract answer counts.

No. of Views

  • For views, we can see two options. One is short form in number of millions and other is full number of views. We will extract the full version.
  • They can be found by using div tag along with supernova class (refer Figure 2.). Then we need to clean the string and convert it into integer format.

List comprehension, to extract vote counts.

Putting all of them together in a dataframe

Plotting Votes v/s Views v/s Answers

Votes, Views and Answers for Top 50 Questions

Here, we may observe that there is no collinearity between the votes, views and answers related to a question.

I hope this tutorial helps you in learning a thing or two about Web Scraping. If you have any queries or suggestions, kindly comment below. You can also reach out to me via Whatsapp.

Visit my website to find similar projects.

Useful Resources:

Happy Learning !!!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade