How to crawl the web politely with Scrapy

The first rule of web crawling is you do not harm the website. The second rule of web crawling is you do NOT harm the website. We’re supporters of the democratization of web data, but not at the expense of the website’s owners.

Some of my thoughts as we launch Pepo today and I attempt to put the epic failure of my previous company behind me.

Does it matter if Facebook is a tech company or a media company?

Join the debate… What is your take?

Learning Style Space with a Human Touch

Motivation:

The Battle of the Ecosystems

To be able to make a product that just works you need to control the whole ecosystem. Only with this level of control over every part of the product can a company accomplish this. This was Steve Jobs thinking. This is why Apple is structured the way that it is. By the way it…