How to Make Money with Web Scraping Using Python
From the most obvious ways to the least.
--
Data is a very useful resource. Some even call data the oil of the 21st century.
This is why not only companies but regular people would pay to get clean data in a spreadsheet that they later will turn into something that can help them stay ahead of their competitors, or even start their own business!
There’s no doubt that there’s a high demand to get data from websites and you can use web scraping to automatically extract all this data.
Here are 3 ways to make money using web scraping. You can use web scraping libraries in Python such as Beautiful Soup, Selenium, or Scrapy to do this or use any other programming language that allows you to scrape websites.
In case you don’t feel like reading, watch my video below.
Selling Data
When it comes to making money selling the data you web scraped, the most common way is selling it to companies. That said, there are also other potential clients that will pay you to get data.
Selling data to companies (the most obvious way)
There are many companies out there that need data to power their business and do research to stay ahead of their competitors.
The good news is that there are data all over the internet. However, there’s a little problem — most of that public data is unstructured. That is, the data isn’t organized in a pre-defined manner, so instead of having the data in a nice-looking Excel spreadsheet, you’d find this unstructured data in an email, social media posts, videos, images, etc.
This is a big issue because working with unstructured data isn’t simple, so companies want to extract this messy data and put it into a clean spreadsheet like the one you see below.
Here’s when web scraping can help us. You only need to build a web scraper that extracts information from a website and once you have that data you can sell it to a client.
Of course, before scraping a website, you need to have a client who wants that data! Some sites where you can find a web scraping gig are Craiglist and Fiverr.
Keep in mind that, selling data that you web scraped is simple when you need to deliver data to a client only once. However, when the data needs to be extracted, say every week, you need to build a robust web scraper that can be easy to maintain, so if the code behind the website changes you won’t need to do major fixes.
But there’s more
Companies are not the only ones who want data. Regular people like you and I need to do repetitive tasks to download hundreds of files from a website or click hundreds of times to copy data from websites and organize it in a structured way using a spreadsheet.
You can make the life of these people easier using web scraping too. In this case, you’ll create a bot that repeats the steps a person would follow to download files or copy data from websites.
These gigs can be found on Fiverr too and, believe me, there are a lot of people who will pay you if you make their lives easier.
Trading
Trading is all about buying and selling things in order to make a profit. Traditionally people buy and sell stocks, bonds, currencies, and commodities, but you can profit from any buying and selling activity like with cryptos and sports betting.
Nowadays there are many apps that send notifications whenever prices go up and down. For example, whenever the price of Bitcoin drops or rises, Coinbase sends me a notification on my phone and also an email. That said, there are some small niches where such apps don’t exist (or are expensive!).
Here’s when web scraping comes into action.
You can build a web scraper that tracks the price of something valuable for you, so when the price changes you get a notification. But here’s the catch, you need to build an efficient and fast web scraper because in some cases the price is so volatile that you may lose your trading opportunity if your scraper is inefficient.
I’ve tried this doing sports arbitrage. This consists in taking advantage of the odds offered by two or more bookies. Say Team A and Team B are playing a soccer game. If bookie 1 offers odds of 2 for Over 2.5 Goals and bookie 2 offers odds of 2.1 for Under 2.5 Goals, you have a surebet. That is, you’ll make a profit regardless of the game outcome if you place one bet per outcome with different bookies.
If you’d like to know more about how to use web scraping to do sports arbitrage, check out this article.
Creating an Aggregator Website
Sometimes you don’t need to sell data to make money but use the data to build something valuable (and then profit from it). This is what aggregator websites are about — they bring valuable data from different sources to one place. This makes the life of people easier because they don’t have to navigate through many websites to get the information they want, but only need to go to one single place to get it.
Good examples of aggregator websites are Groupon (coupon aggregation), Kelkoo (price comparison), Booking.com (travel aggregation), etc.
How do these websites make money?
There are different ways to make money with an aggregator website. You can charge a commission for each deal made through your website, feature content (like those sponsored products on Amazon), charge for a subscription membership, or the most obvious, display ads on the website.
Of course, to get there, you need to go beyond web scraping.
You need to identify a good niche, build a robust web scraper, and build a good-looking website if you want to make money with your aggregator website.
Turn websites into datasets! Get my FREE Web Scraping Cheat Sheet by joining my email list with 10k+ people.
If you enjoy reading stories like these and want to support me as a writer, consider signing up to become a Medium member. It’s $5 a month, giving you unlimited access to thousands of Python guides and Data science articles. If you sign up using my link, I’ll earn a small commission with no extra cost to you.
