Python Web Scraping Part: 2
In my last article
https://medium.com/@tommarler/python-web-scraping-part-1-338d5c818641
I talked about beautifulsoup and thought scrapy might be the solution to all my problems. Scrapy is cool and does a lot of great things however I did run into a few problems. In this article I am going to take the time and explain scrapy because it is a great tool for the proper use case.
What is scrapy?
It’s a framework when used correctly makes getting data from websites easier. Since scrapy is a framework it has a certain way of doing things and follows a specific structure.
How do you create a project with scrapy?
Whenever I work in python I always create a virtual env:
virutalenv <project name>
I dont like using venv I think <project name> is naming the virtual environment and to me that makes sense. I could be wrong I dont know(Please leave a comment below).
cd into the project, source bin/activate and install my packages example: pip install scrapy
Next run: scrapy startproject <project name>
subl . or code . opens the project up in an IDE.
What are all these folders?