How to Bypass Cloudflare in 2024

At Lenspricer, we scrape many websites which protect themselves with Cloudflare. In this article, I’m going to showcase how we tackle this problem.

I’m going to let you in on a little secret: it can be difficult. We’ve tried running our own virtual machines with proxies and headed remote controlled browsers. When we finally got something to work, Cloudflare made some changes and we had to go back to the drawing board and redo some of the things. After spending a few months going back and forth, we thought to look elsewhere — could we pay someone for all of this hassle instead? Introducing: Scraping APIs.

Scraping APIs

A scraping API is essentially a SaaS which let’s you very easily scrape a website using advanced features like proxies, JavaScript rendering, and most importantly — it lets you bypass Cloudflare and captchas.

It’s as simple as: if you want to scrape https://example.com, and let’s say that site was protected by Cloudflare, you just send the site to the scraping API, and when you call the scraping API, you get the HTML back from the original site, with Cloudflare bypassed. Simple as that. Calling the API could look like:

https://scrapeapi.com?api_key=123?bypasscf=true?get=https://example.com

And this is exactly what we do. Instead of maintaining our own infrastructure and keeping up with all the anti-scraping changes, we just pay a small monthly fee, and we get access to a scraping API which handles everything for us.

There are many scraping API services, but the one we settled on is ZenRows. We found it to be very affordable at just $69 per month (previously we paid $500 in hardware + developer salary just to maintain our own infrastructure), and it just works. We’ve never experienced any site that they couldn’t handle.

The big 3 scraping APIs are ZenRows, ScraperAPI and ScrapingBee. If you’re interested in reading more about different web scraping services, I recently wrote a post on the top 5 best web scraping APIs, comparing each service using 10 different metrics.

ZenRows offers a free trial with plenty of credits, which we found very helpful when testing if it would work for us, and their online API request builder was just super helpful:

The Cheap Method to Bypass Cloudflare

Just be aware: while this may not be that expensive compared to a scraping API, you’re going to pay with your own labor, which might in the end be more valuable than money.

If you are up for it, you can set up your own scraping infrastructure to bypass Cloudflare. I don’t think there exists a single “this is the way” resource out there describing how to do this (the ones who are figuring it out have built companies around it and want to keep it secret).

To give you some starting hints, you are probably going to need a good undetectable chrome driver, some rotating residential proxies, and some trial and error. You can find some resources on Reddit, but things are constantly changing on how to skip/get around Cloudflare, so you should make a search on reddit to find the latest tips.

You will still have to pay for the proxies, and maybe also for some captcha solvers.

Conclusion

With the many scraping APIs available in 2024, it’s no longer a full-time job to bypass Cloudflare. It’s as simple as sign up for an account, pass the URL you want to scrape to the API, and the API returns the full HTML of the original URL, without any Cloudflare messing with you.

If you have any questions about how we scrape 50,000 products from 100+ stores each day at Lenspricer.com, or have any comments for the article, please feel free to reach out to me.

• Updated August 12, 2024: Added section on alternative ways to bypass Cloudflare.

Note: some links in this post is sponsored, but the content truly reflects my own opinion, and how we handle scraping at Lenspricer.