RabbitMQ Scrapy Item Publisher in Python

It’s a small note about the Scrapy Item Pipeline that publishes a Scrapy Item to the RabbitMQ.

Photo by rawpixel on Unsplash

If you are interested in parsing/scraping projects in Python, you should know about Scrapy. Let’s imagine that your project is something bigger than a one-time data extraction. This means that you need to do something with your data. One of the possible approaches is to use RabbitMQ to publish items to some queue. This method can help you to build a custom consumer that can process Scrapy items.

Code (repository can be found out here):

Calling Celery Tasks not from Python

Thanks for the attention to the topic, feel free to leave your questions in the comments for discussion.