How to Bypass Captcha in 2024
tl;dr: the easiest way to bypass Cloudflare is to use a scraping API such as ZenRows or ScraperApi.
Running into Captcha/reCaptcha while scraping a site can be frustrating. Standard open source libraries cannot handle them, and you may feel like you’re running into a wall. In this post, I’m going to explain how I bypass captchas while scraping sites in 2024.
How I Bypass Captchas
The short answer is that I use a web scraping API (I use ZenRows, but other scraping APIs work as well). A web scraping API can automatically bypass/solve any captchas on the site you’re trying to scrape.
Say you wanted to scrape https://example.com, which was protected by a Captcha — instead of scraping the site directly, you would pass the URL to the scraping API, as below:
https://apiscrape.com?api_key=456?skipCaptcha=true?get=https://example.comThe scraping API then skips/solves the captcha automatically, and returns the HTML of the site, without you having to worry about anything.
Web scraping APIs usually cost around $49 per month, which is quite cheap compared to trying to maintain a scraping infrastructure that is able to bypass captchas, on your own. I would suggest you sign up for a free trial, and see if the scraping API can handle the captcha you’re trying to solve.
Why I use a web scraping API to bypass Captchas
When I was first starting out scraping sites sites that had Captcha protection on them, I would try to bypass the captchas using my own code. Sometimes I could scrape maybe like 5 pages on a site, but then their Captcha protection kicked in, and I was stuck.
I tried to integrate anti-captcha/2Captcha directly into my own scraper, but even when I used those captcha solvers, I was still sometimes having issues scraping the sites.
Instead of spending ages trying to fix my own scrapers and extending them with captcha solvers, I started using a web scraping API to solve the problem for me. I was already using a web scraping API to bypass Cloudflare, and I actually learned that the same scraping API that I’m using can also handle Captchas. The API is able to handle all Captcha types, e.g. Cloudflare Captcha, Google’s reCAPTCHA, LinkedIn’s captcha, etc.
Which web scraping API to use for captcha protected sites
The web scraping API I use for bypassing Captchas is ZenRows. I was already using them for other purposes, so it was an obvious choice. Many of the other scraping API providers also solve Captchas (e.g. ScrapingBee and Bright Data). If you’re interested in learning about the alternatives, check my post where I examine and compare the top 5 best web scraping APIs.
Conclusion
I believe the best option for skipping/bypassing/solving captchas when scraping sites in 2024 is to use a web scraping API. It’s very easy to use, and you can focus on the interesting parts of your code instead of having to maintain cumbersome captcha bypassing code.
If you have any questions on how I bypass captchas, please feel free to reach out. I use a web scraping API to bypass captcha for my website, Lenspricer, which is a price comparison site for contact lenses. We scrape 200+ sites in 30 countries each day, some of which are protected by captchas. Once I started using a scraping API, I very rarely run into websites that I have trouble scraping, so it’s really quite nice to have this option. Finally, I want to disclose that some of the links in this article are partner links, however, everything I write truly reflects my own opinion, and is how we handle scraping at Lenspricer.
