Scrape DuckDuckGo Ad Results with Python
Contents: intro, imports, what will be scraped, process, code, links.
Intro
This blog post is a continuation of DuckDuckGo web scraping series. Here you’ll see how to scrape Ad Results using Python with selenium library. An alternative API solution will be shown.
Imports
from selenium import webdriverWhat will be scraped
Process
Selecting title, link, snippet, source, sitelinks.
What I was doing is basically applying .results--ads selector before additional selector for the title, link, snippet, source, and sitelinks to get the most accurate selector.
A CSS selectors reference.
Code
Using DuckDuckGo Ad Results API
SerpApi is a paid API with a free plan.
The difference is that is much faster, you don’t need to render the page with selenium and you get an immediate structured JSON string.
Links
GitHub Gist • DuckDuckGo Ad Results API
Yours,
- D
Originally published at https://serpapi.com on August 11, 2021.
