Increase your scraping speed with Go and Colly! — The Basics

Let’s scrape Amazon to see how fast this can be. But first, let’s learn about the basics.

Jérôme Mottet
Jun 10, 2019 · 6 min read
Image for post
Image for post

Introduction

Let’s inspect Amazon to determine the CSS selectors

Image for post
Image for post
Here is how Amazon’s result page looks like
Image for post
Image for post
Name: span.a-size-medium.a-color-base.a-text-normal
Stars: span.a-icon-alt
Price: span.a-price > span.a-offscreen
https://www.amazon.com/s?k={search-term}&ref=nb_sb_noss_1
https://www.amazon.com/s?k=nintendo+switch&ref=nb_sb_noss_1

Go & Colly implementation

Let’s create our Collector !

go run main.go
Visiting https://www.amazon.com/s?k=nintendo+switch&ref=nb_sb_noss_1

Time to parse that HTML!

Product Name: Super Smash Bros. Ultimate 
Product Name: New Super Mario Bros. U Deluxe - Nintendo Switch
Product Name: Accessories kit for Nintendo Switch, VOKOO Steering
Product Name: AmazonBasics Car Charger for Nintendo Switch
...
Image for post
Image for post
The tests for those functions will be provided in the Github repository.
Product Name: Nintendo Switch - Gray Joy-Con 
Stars: 4.5
Price: Unknown
Product Name: Nintendo Switch Console w/ Mario Kart 8 Deluxe
Stars: 4.7
Price: 394.58
Product Name: Lego Star Wars Skywalker Saga - PlayStation 4 Standard Edition
Stars: Unknown
Price: 59.99

Conclusion

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

Jérôme Mottet

Written by

Software Engineer - React and React-Native Developer - Web Scraping Enthusiast

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

Jérôme Mottet

Written by

Software Engineer - React and React-Native Developer - Web Scraping Enthusiast

The Startup

Medium's largest active publication, followed by +752K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store