How to create a scraper for shopify

Note for the impatients: For a simple python script to download products from shopify you can use this project: https://github.com/lagenar/shopify-scraper

With more than 300,000 merchants using it, Shopify is the largest ecommerce platform available. A lot of people come to me asking if I can extract the all the products of a shopify site, usually this information is used to analize the products sold by the site and also pick revelant information about their prices. This information can be used for competitive analysis (are my products cheaper/more expensive?) but it’s also used to discover new products to sell.

To achieve extracting all products from a site I usually created a custom web scraper that would go through all the categories and product pages, parse the relevant product information. Once you do this for some shopify sites I tried to find a simple way to do this automatically instead creating a new scraper for each site.

The main question I tried to answer was: is there any way to create a scraper that to which I’ll pass the url of any shopify site and it will scrape all their products automatically?

The anwser is Yes

Shopify provides an API endpoint called products.json that will give you all products with their prices, variants and all other attributes.

The documentation can be found here https://help.shopify.com/api/reference/product#index

While in the documentation appears that the API endpoint is only for admin authenticated users if you point your browsers url to the path /products.json in any shopify site you can see that it’s accesible by any visitor.

Since most people need the data to be formatted in a table I created a small python script that will receive the website as the first only parameter and will output all products and variants in the site to a CSV file named products.csv. The export contains the product’s category, name, variant name, price and url but can be easily extended to add more information.

Paste the following code into a file named shopify_scraper.py

Now open a terminal in the same directory as the script and run python shopify_scraper.py <website url>

For example if the website is http://ecommercewebsite.com you can use

python shopify_scraper http://ecommercewebsite.com

If you prefer to do this without any programming just go to http://scrapefast.com/shopify_scraper and enter the details to receive the data into your email.

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store