Web Scraping Project Ideas

Sagun Shrestha
Nov 14, 2019 · 3 min read
Photo by AbsolutVision on Unsplash

Web Scraping is basically a process of extracting data from website using some scripts or automation tool/software. There are a ton of websites in the internet with a lot of data. If you are wondering what might be a good way to start with, here are some of the cool web scraping project ideas that you can implement.

1. Price Monitoring

Price monitoring is a very common yet useful technique that you can use to automate the process of checking prices on various websites. For e.g. you want to buy a laptop but you are waiting for the price to drop that is within your budget. You can write a simple script, deploy it somewhere and automate sending you an email as soon as your criteria is matched.

Sellers on amazon often use price monitoring services to kep track of their competitors. Companies manufcturing products also use this technique to monitor if the retailers are selling it above Minimum Advertised Price (MAP) or not.

2. Email Mining

You can scrape emails from various web directories, websites or search engines based on certain criteria and use it for marketing purpose or simply sell it to someone else. Email mining is very common in marketing world and people often buy email lists.

Email mining can be fruitful to advertise about or product or services to your possible client. However you need to be careful not to spam them.

3. Job Aggregator

Job aggregation can be your next big scraping project. There are people actively looking for jobs and there are companies looking to hire suitable manpower. The problem is there are a ton of job boards with a lot of listings. What if you can scrape the job links and title, put it in a single place from where job seeker can get the details.

A job aggregator will help all 3 parties: Job board, Company and Job seeker. However when crawling, don’t crawl all the data and bypass the job board. I would suggest scraping just the title and link would be enough. Job seeker can be redirected to the original site where the job was posted.

4. Scraping Reviews

Reviews are important for businesses to know better about their customer. You can build a review collecting system by scraping reviews from sites like yelp, tripadvisor, trustpilot, etc.

Monetizing a review collection platform is a viable option as businesses are trying to acces reviews in the easiest way possible. You can build a platform with dashboard and analytics just for reviews and sell it to interested businesses.

5. Imdb Scraper

Imdb is probably the easiest site to scrape. You can scrape movie data from Imdb, analyze them based on reviews, ratings and votes. There is a ton of data available in Imdb regarding movies.

You can also scrape other movie sites and compare the data with what you got from Imdb.

6. Reddit Scraper

Reddit is popular for its content and community. There are a lot of subreddits that you might be interested in. Scraping these subreddits for popular posts can be fun project to work on.

Further you can also scrape comments, upvotes and build some viualization around it to show post engagement, popular subreddits, etc.

Conclusion

Web Scraping might seem fun but you must always respect a website’s policy and don’t overload the server too much. Chances are you might banned frequently. Scrape Responsibly.

If you are looking for a customized scraping solution for your business, contact us or email us hello@scraperhq.com. You can connect with me on Twitter.

Web scraping as a service (https://lambdascraper.com/)

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store