Python Web Scraping Part: 2

In my last article

https://medium.com/@tommarler/python-web-scraping-part-1-338d5c818641

I talked about beautifulsoup and thought scrapy might be the solution to all my problems. Scrapy is cool and does a lot of great things however I did run into a few problems. In this article I am going to take the time and explain scrapy because it is a great tool for the proper use case.

What is scrapy?

It’s a framework when used correctly makes getting data from websites easier. Since scrapy is a framework it has a certain way of doing things and follows a specific structure.

How do you create a project with scrapy?

Whenever I work in python I always create a virtual env:

virutalenv <project name>

I dont like using venv I think <project name> is naming the virtual environment and to me that makes sense. I could be wrong I dont know(Please leave a comment below).

cd into the project, source bin/activate and install my packages example: pip install scrapy

Next run: scrapy startproject <project name>

subl . or code . opens the project up in an IDE.

What are all these folders?