Scraping Games Done Quick Data with Python, Scrapy, & BeautifulSoup
Published in
11 min readMay 11, 2023
This article will show you how to scrape bid information from the Games Done Quick website using Scrapy and BeautifulSoup in Python.
Specifically, it will cover the following topics:
- What is Games Done Quick (GDQ)? What are GDQ Bids?
- Is my spider is allowed to crawl “X”? How fast can my spider crawl?
- How do I create and run a scrapy + BeautifulSoup spider that crawls the GDQ site for bids data, parses the response, and saves it locally?
The goal is to provide a concrete example of how Scrapy and BeautifulSoup can be used together.
What is Games Done Quick (GDQ)?
From Games Done Quick:
Games Done Quick (GDQ) is a series of charity video game events. These events feature high-level gameplay by speedrunners raising money for charity. Games Done Quick has teamed up with several charities including Doctors Without Borders and the Prevent Cancer Foundation. We are currently the largest fundraising event globally for both charities!
