AutoScraper and Flask: Create an API From Any Website in Less Than 5 Minutes

And with fewer than 20 lines of Python

Alireza Mika
Sep 17 · 3 min read
Image for post
Image for post
Photo by Vipul Jha on Unsplash.

In this tutorial, we are going to create our own e-commerce search API with support for both eBay and Etsy without using any external APIs. With the power of AutoScraper and Flask, we are able to achieve this goal in fewer than 20 lines of Python code for each site. I recommend reading my last article about AutoScraper if you haven’t done so yet.


Install the required libraries using pip:

pip install -U autoscraper flask

Let’s Do It

First, we are going to create a smart scraper to fetch data from eBay’s search results page. Let’s say we want to get the title, price, and product link of each item. Using AutoScraper, it would be easily done by just providing some sample data:

Note that if you want to copy and run this code, you may need to update the wanted_list.

Now let’s get the results grouped by scraping rules:

scraper.get_result_similar(url, grouped=True)

From the output, we’ll know which rule corresponds to which data, so we can use it accordingly. Let’s set some aliases based on the output, remove redundant rules, and save the model so we can use it later:

Note that the rule IDs will be different for you if you run the code.

OK, we’ve got eBay covered. Let’s add support for Etsy search results too. We’ll start by building its scraper. This time, we will use wanted_dict instead of wanted_list. It will automatically set aliases for us:

As the links are generated with a unique ID on Etsy each time, we added one sample product ID to the wanted_dict so we can create the link from it. Also, we provided two samples for title and price, as the structure of items on Etsy search result pages is different and we want the scraper to learn them all.

After analyzing the output, let’s keep our desired rules, remove the rest, and save our model:

Now that we have our scrapers ready, we can create our fully functioning API for both sites in fewer than 40 lines:

Here, we are defining an API with the parameter q as our search query. We will get and join eBay and Etsy search results and return them as a response. Note that we are passing group_by_alias=True to the scraper to get the results grouped by our defined aliases.

By running this code, the API server will be up listening on port 8080. So let’s test our API by opening http://localhost:8080/?q=headphone in our browser:

Image for post
Image for post
Some results from eBay
Image for post
Image for post
Some results from Etsy

Voila! We have our e-commerce API ready. Just replace headphone in the URL with your desired query to get its search results.

Final Notes

The final code for this tutorial is available on GitHub.

This is a development setup suitable for developing and testing. Flask’s built-in server is not suitable for production. For production usage, please check Flask’s deployment options.

This tutorial is intended for personal and educational use. If you want to scrape websites, you can check their policies regarding the scraping bots.

I hope this article is useful and helps to bring your ideas into code faster than ever. Happy coding!

Better Programming

Advice for programmers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store