DATA COLLECTION

Web Scraping Using Scrapy and Python

A beginner-friendly guide to scraping web data

Source: unDraw.co

Introduction

Web data is one of the most readily accessible sources of data out there. For this reason, being able to extract and utilize the plethora of data that exists on the web is a necessary skill for every data scientist. And if this skill is not in your skillset just yet, needless to worry because this tutorial has got you covered. By the end of this tutorial, you’ll have learned the fundamentals of web scraping using Scrapy and will have a fully functional Python web scraper that extracts Covid-19 data from Worldometers.info.

The Web Scraping Process

Image by Author

In a typical web scraping project, you would:

• Define your project’s end goal and identify the web sources which you believe contain your target data.
• Acquire data using code and extract it into meaningful data structures.
• Transform the acquired data to suit your project needs and run the appropriate analysis. This is the most fluid part of your web scraping…