Web Scraping in Python for Finding your Next Job

Web scraping is the process of extracting data from websites. One can Web scrape data in Python using the requests and beautiful soup package. Examples of data that can be scraped include lists of products on an e-commerce site or unstructured data like a list of articles on a news site or looking over job listings.

Web scraping is often used for for data mining, because for some machine learning project certain datasets may be limited or not even exist. There are many other reasons someone or a business might want to build Web scrapers. Some websites allow web scraping, while others may block web scrapers by using CAPTCHAs or other measures to prevent automated access.

In this article you will learn how to web scrape data related to job postings from LinkedIn. The program will take user inputs of job title and location and return all matches. The results are printed in order of 1) role, 2) company, and 3) location. Below is the complete code example of a simple web scraper using Python’s requests and BeautifulSoup libraries:

import requests
from bs4 import BeautifulSoup

# Prompt the user for a job title and location
title = input('Enter a job title to search for: ')
location = input('Enter a location to search for: ')
print('\n')

# Format the job title and location for use in the URL
title = title.replace('…