The Techlife
Published in

The Techlife

Designing a distributed web crawler part 2 — components and in-depth design

Let’s dive deep into the components of a web crawler

Photo by Arnold Francisca on Unsplash

In part 1, we have established the fundamentals and requirements for designing a distributed web crawler. In this part, we are going to actually dive deep into the design.

The design

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store