Scrapy: Crawl WhoScored For Football Stats

I have written code earlier to crawl Google Play, iTunes AppStore and websites. But every time I re-wrote the code to get content from website, parse it using BeautifulSoup while maintaining the list of crawled URLs to avoid crawling them again. This was a lot of work and I don’t know why I didn’t google for an existing crawler framework.

A while ago I discovered Scrapy. It’s a python based framework that makes it super easy to setup a crawler and let’s one focus on extracting the necessary data and processing it. In this post I will talk through installing Scrapy, writing a crawler to crawl WhoScored and extracting match information from it.

Installation is super easy. On a Mac, with pip installed, one just needs to fire the following command to install Scrapy along with all it’s dependencies:

>> sudo pip install Scrapy

Now we need to setup a Scrapy project. Let’s call our project soccerstats.

>> scrapy startproject soccerstats

This will create a directory soccerstats for us. Now to setup the crawler, we need to define an item which is a container of the crawled data and write a spider that actually extracts the data. Additionally we can define our custom LinkExtractor and a processing Pipeline.

The SoccerStats project can be found at — GitHub SoccerStats.

Let’s first configure the crawler.

# Automatically created by: scrapy startproject
# For more information about the [deploy] section see:

default = soccerstats.settings

#url = http://localhost:6800/
project = soccerstats

The above tells Scrapy that the settings file is present under the Python package soccerstats and is named Let’s now see the settings file.

# -*- coding: utf-8 -*-

# Scrapy settings for soccerstats project
# For simplicity, this file contains only the most important settings by
# default. All the other settings are documented here:

BOT_NAME = 'soccerstats'

SPIDER_MODULES = ['soccerstats.spiders']
NEWSPIDER_MODULE = 'soccerstats.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Anuvrat Singh (+'

# WhoScored settings
TOURNAMENTS = [9155, # England-Premier-League-2014-2015
11369, # Italy-Serie-A-2014-2015
11363, # Spain-La-Liga-2014-2015
9192, # Germany-Bundesliga-2014-2015
9105, # France-Ligue-1-2014-2015
9121, # Netherlands-Eredivisie-2014-2015
9145, # Russia-Premier-League-2014-2015
4185, # Fixtures/Brazil-Brasileiro-2014
8358, # USA-Major-League-Soccer-2014
11306, # Turkey-Super-Lig-2014-2015
9156, # England-Championship-2014-2015
9189, # Europe-UEFA-Champions-League-2014-2015
9187, # Europe-UEFA-Europa-League-2014-2015
10274 # International-FIFA-World-Cup-2014
TOURNAMENT_YEARS = [2014, 2015] # Years for the current season


LOG_FILE = 'application.log'

The SPIDER_MODULES settings tells Scrapy that the spider can be found in soccerstats.spiders module. We have also defined a user_agent so that website owners can know who has been crawling them. Then there are a few settings to configure concurrent requests count and logging.

One can add any configuration in this file and access it in the project by calling the method get_project_settings().get(CONFIG_KEY_NAME). I have added the match and feed URLs and the list of tournaments we are interested in. I prefer extracting application configurations into a separate file than hardcoding everything in code files.

Next we need to define the Item that we will extract from each page. Each Item object needs to extend the scrapy.Item class. Below code is pretty self-explanatory.

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
# See documentation in:

import scrapy

class WhoScoredRatingsItem(scrapy.Item):
Define the item that will store the
player ratings for a team in a match
match_id = scrapy.Field()
venue_name = scrapy.Field()
referee_name = scrapy.Field()
start_time = scrapy.Field()
man_of_the_match = scrapy.Field()
competition = scrapy.Field()

home_team_id = scrapy.Field()
home_team_name = scrapy.Field()
home_team_average_age = scrapy.Field()
home_team_manager = scrapy.Field()
home_team_rating = scrapy.Field()
home_team_formation = scrapy.Field()
home_team_players = scrapy.Field()
home_team_score_halftime = scrapy.Field()
home_team_score_fulltime = scrapy.Field()

away_team_id = scrapy.Field()
away_team_name = scrapy.Field()
away_team_average_age = scrapy.Field()
away_team_manager = scrapy.Field()
away_team_rating = scrapy.Field()
away_team_formation = scrapy.Field()
away_team_players = scrapy.Field()
away_team_score_halftime = scrapy.Field()
away_team_score_fulltime = scrapy.Field()

Now we need to define a CrawlSpider and a LinkExtractor. You provide a list of start_urls to the CrawlSpider. Scrapy then gets the content of this website and extracts all the URLs matching allowed_domains. In our particular case, we provide a list of tournamentfeed URLs to the CrawlSpider. The data is a list of list in which first element of each list is the matchId. Using this matchId we can get the link to the match page. To extract the matchIds and form the links to match pages we will write a custom LinkExtractor. Let’s first start with that.

# -*- coding: utf-8 -*-

Link extractors for WhoScored web pages.

import ast

from scrapy.contrib.linkextractors import LinkExtractor
from import Link
from scrapy.utils.project import get_project_settings

class WhoScoredLinkExtractor(LinkExtractor):
Extract links from WhoScored Tournament Feed pages

def extract_links(self, response):
Given the response from the tournament feed page, extract all the matches and return URLs to their page.
:param response: The contents of the tournament feed page
:return: The list of URL to match pages
match_page = get_project_settings().get('WHOSCORED_MATCH_FEED')

return [Link(match_page % match[0]) for match in ast.literal_eval(response.body.replace(',,', ',0,'))]

To implement a link extractor, one needs to extend the LinkExtractor class and define the extract_links(response) method. This method is called for every start URL. The return type of this method is a Link object that the CrawlSpider can crawl next.

Let’s look at the actual spider now.

# -*- coding: utf-8 -*-

Spider to crawl WhoScored web pages.

import re

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.utils.project import get_project_settings
from soccerstats.spiders.whoscored_linkextractor import WhoScoredLinkExtractor
from soccerstats.items import WhoScoredRatingsItem
import json

class WhoScoredSpider(CrawlSpider):
Define the crawler to crawwl WhoScored web pages and extract player ratings
name = 'WhoScored'
allowed_domains = ['']
rules = (
Rule(WhoScoredLinkExtractor(), callback='parse_item'),

def __init__(self, tournament=None, year=None, month=None, *args, **kwargs):
super(WhoScoredSpider, self).__init__(*args, **kwargs)
self.start_urls = self.__prepare_seed_list(tournament, year, month)

def parse_item(self, response):
Given the match feed page, extract ratings for both the teams
:param response: The contents of the match page
:return: The rating items for both the teams.

match_data_element = response.xpath('//script[contains(., "matchCentreData")]/text()').extract()
if len(match_data_element) == 0:

match_data = match_data_element[0]
match_center_data = json.loads("matchCentreData = (.+?);", match_data).group(1))

item = WhoScoredRatingsItem()

item['match_id'] ="matchId = (.+?);", match_data).group(1)
item['venue_name'] = match_center_data['venueName'] if 'venueName' in match_center_data else ''
item['referee_name'] = match_center_data['refereeName'] if 'refereeName' in match_center_data else ''
item['start_time'] = match_center_data['startTime']
item['competition'] = response.xpath('//div[@id="breadcrumb-nav"]/a/text()').extract()

for pos in ['home', 'away']:
team_data = match_center_data[pos]
item[pos + '_team_id'] = team_data['teamId']
item[pos + '_team_name'] = team_data['name']
item[pos + '_team_average_age'] = team_data['averageAge']
item[pos + '_team_manager'] = team_data['managerName']
item[pos + '_team_formation'] = team_data['formations'][0]['formationName']
item[pos + '_team_score_halftime'] = team_data['scores']['halftime']
item[pos + '_team_score_fulltime'] = team_data['scores']['fulltime']

players = {}
team_rating = 0.0
players_involved = 0
for player in team_data['players']:
player_id = player['playerId']

ratings_array = player['stats']['ratings'] if 'ratings' in player['stats'] else None
if ratings_array:
rating = ratings_array[max(ratings_array, key = int)]
team_rating += rating
players_involved += 1
rating = -1

players[player_id] = {'age': player['age'],
'height': player['height'],
'shirt': player['shirtNo'] if 'shirtNo' in player else -1,
'position': player['position'],
'name': player['name'],
'started': 'isFirstEleven' in player and player['isFirstEleven'],
'rating': rating

if player['isManOfTheMatch']:
item['man_of_the_match'] = {'id': player_id, 'name': player['name']}

item[pos + '_team_players'] = players
item[pos + '_team_rating'] = team_rating / players_involved

return item

def __prepare_seed_list(self, tournament, year, month):
whoscored_feed_url = get_project_settings().get('WHOSCORED_FEED_URL')

if tournament and year and month:
return [whoscored_feed_url % (tournament, year, month.zfill(2))]

tournaments = get_project_settings().get('TOURNAMENTS')
years = get_project_settings().get('TOURNAMENT_YEARS')

dates = [(years[0], month) for month in xrange(06, 12)]
#dates.extend([(years[1], month) for month in xrange(01, 06)])

return [whoscored_feed_url % (tournament, year, str(month).zfill(2)) for tournament in tournaments for (year, month) in dates]

The spider class needs to extend the CrawlSpider class. We have defined the list of allowed_domain, which in our case is just Scrapy will only extract links to crawl and ignore the rest. The method __prepare_seed_list(tournament, year, month) is responsible for generating the list of start_urls.

We have also defined the rule to use WhoScoredLinkExtractor() to extract links from the start_url pages. Notice that we have defined a callback for this rule. This means that the method parse_item will be called for every URL returned by the link extractor.

The method parse_item() does the main job of extracting data from the webpage by parsing appropriate xpaths. It extracts the data into the Item we created earlier and returns the item. A point to remember here is that if one wants to return multiple items from a page then all he needs to do is yield each item object.

To run the crawler we have a couple of options:

To write the output into ratings.json file.

>> scrapy crawl WhoScored -o ratings.json -t json -L INFO

To write the output in the console, and also provide inputs to the crawler.

>> scrapy crawl WhoScored -a tournament=9155 -a year=2014 -a month=7

Output from a run of the crawler:

2015-01-16 22:15:14+0530 [scrapy] INFO: Scrapy 0.24.4 started (bot: soccerstats)
2015-01-16 22:15:14+0530 [scrapy] INFO: Optional features available: ssl, http11, boto
2015-01-16 22:15:14+0530 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'soccerstats.spiders', 'FEED_URI': 'ratings.json', 'LOG_LEVEL': 'INFO', 'CONCURRENT_REQUESTS_PER_DOMAIN': 16, 'CONCURRENT_REQUESTS': 32, 'SPIDER_MODULES': ['soccerstats.spiders'], 'BOT_NAME': 'soccerstats', 'CONCURRENT_ITEMS': 200, 'ROBOTSTXT_OBEY': True, 'COOKIES_ENABLED': False, 'USER_AGENT': 'Anuvrat Singh (+', 'FEED_FORMAT': 'json', 'LOG_FILE': 'application.log'}
2015-01-16 22:15:14+0530 [scrapy] INFO: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2015-01-16 22:15:14+0530 [scrapy] INFO: Enabled downloader middlewares: RobotsTxtMiddleware, HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-01-16 22:15:14+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-01-16 22:15:14+0530 [scrapy] INFO: Enabled item pipelines:
2015-01-16 22:15:14+0530 [WhoScored] INFO: Spider opened
2015-01-16 22:15:14+0530 [WhoScored] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-01-16 22:16:14+0530 [WhoScored] INFO: Crawled 136 pages (at 136 pages/min), scraped 103 items (at 103 items/min)
2015-01-16 22:17:14+0530 [WhoScored] INFO: Crawled 258 pages (at 122 pages/min), scraped 224 items (at 121 items/min)
2015-01-16 22:18:14+0530 [WhoScored] INFO: Crawled 387 pages (at 129 pages/min), scraped 354 items (at 130 items/min)
2015-01-16 22:19:14+0530 [WhoScored] INFO: Crawled 513 pages (at 126 pages/min), scraped 478 items (at 124 items/min)
2015-01-16 22:20:14+0530 [WhoScored] INFO: Crawled 638 pages (at 125 pages/min), scraped 605 items (at 127 items/min)
2015-01-16 22:21:14+0530 [WhoScored] INFO: Crawled 783 pages (at 145 pages/min), scraped 730 items (at 125 items/min)
2015-01-16 22:22:14+0530 [WhoScored] INFO: Crawled 920 pages (at 137 pages/min), scraped 867 items (at 137 items/min)
2015-01-16 22:23:14+0530 [WhoScored] INFO: Crawled 1059 pages (at 139 pages/min), scraped 1006 items (at 139 items/min)
2015-01-16 22:24:14+0530 [WhoScored] INFO: Crawled 1280 pages (at 221 pages/min), scraped 1122 items (at 116 items/min)
2015-01-16 22:25:14+0530 [WhoScored] INFO: Crawled 1392 pages (at 112 pages/min), scraped 1234 items (at 112 items/min)
2015-01-16 22:26:14+0530 [WhoScored] INFO: Crawled 1537 pages (at 145 pages/min), scraped 1379 items (at 145 items/min)
2015-01-16 22:26:51+0530 [WhoScored] INFO: Closing spider (finished)
2015-01-16 22:26:51+0530 [WhoScored] INFO: Stored json feed (1432 items) in: ratings.json
2015-01-16 22:26:51+0530 [WhoScored] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 623577,
'downloader/request_count': 1880,
'downloader/request_method_count/GET': 1880,
'downloader/response_bytes': 310691128,
'downloader/response_count': 1880,
'downloader/response_status_count/200': 1702,
'downloader/response_status_count/403': 178,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2015, 1, 16, 16, 56, 51, 113574),
'item_scraped_count': 1432,
'log_count/INFO': 19,
'request_depth_max': 1,
'response_received_count': 1880,
'scheduler/dequeued': 1879,
'scheduler/dequeued/memory': 1879,
'scheduler/enqueued': 1879,
'scheduler/enqueued/memory': 1879,
'start_time': datetime.datetime(2015, 1, 16, 16, 45, 14, 402884)}
2015-01-16 22:26:51+0530 [WhoScored] INFO: Spider closed (finished)

I want to add here that if were to change their page and move things around then the only classes we would need to check would be the WhoScoredLinkExtractor and WhoScoredSpider. On the other hand, if we want to add a new crawler we just need to define the new link extractor and spider. Everything else is taken care of by Scrapy.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.