How to Make a Web Crawler in Swift 🕷

Federico Zanetello
Apr 4, 2017 · 3 min read

Hi! This post has moved to a new blog! Come to for the latest articles!

More often than not, my App Suite is thirsty for information that can be found around the Web.

Unless it’s something that takes a couple of minutes, for this kind of needs I like to automate as much as possible (it saves hours of work!).

My go-to languages have always been Python and Javascript however, the recent launch of John Sundell’s Marathon made me wonder:
is Swift ready for scripting? 🤔

Building A Web Crawler in Swift

Since I needed to extract some information from the Internet, this time I’ve decided to give Swift a try.

There are two main ways to do scripts in Swift:

  1. by following Hector Matos’s awesome guide here;
  2. by using John Sundell’s Marathon (the is all you need);

It turns out, coding a Swift script is not any different than coding a new Swift class, function etc. I’ve never felt out of home.

Also, building a Web Crawler in Swift is incredibly easy:

I believe the script is pretty self-explanatory:
you input a starting web page, the number of maximum webpages to crawl, and the word you’re interested in.

In my case, I didn’t need these inputs to change: but if you do want them to, Hector Matos explains how to write a script that takes arguments here.

Once launched, the script will start crawling the web from the given page and jump around all the links it can find.

If you want a deeper explanation, please jump to the comments section 👇🏻 where I analyze each component of the script in detail.

In conclusions, I’d say that, depending on what you want to do, and by using the right tools, Swift sure is one of the very viable options for scripting!

A Note About Semaphores

If you’re wondering why there is a DispatchSemaphore in the script above, it’s because any Swift Script exits as soon as its control flow reaches the end.

If you must handle asynchronicity (like in my case, due to the loading of webpages), you need to have something that makes your script stay alive.

Shameless plug: if want to know more about Semaphores, make sure to read my previous article here.

Scripts and Xcode Playgrounds

If your script allows it, I’d suggest you to use Xcode Playgrounds during the script development:

This way it’s much faster to test it without the need to keep switching between your terminal and your code editor.

Once ready, you can quickly turn it into a real script.

Code Snippets

For this post I’ve made a small repository, Selenops 🕷 (a spider that flies, yay! 😱), on GitHub:
in there you'll find the very same script written as an Xcode Playground, as a Marathon Script, and as a standard “Command Line Tool” Script.

Happy Scripting!

Federico is a Bangkok-based Software Engineer with a strong passion for Swift, Minimalism, Design, and iOS Development.

Swiftly Swift

Write code swiftly

Federico Zanetello

Written by

Swiftly Swift author, building apps @kimchi_media .

Swiftly Swift

Write code swiftly

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade