How to install Scrapy on debian using virtualenv

Posted febrero 15th, 2012 in Blog, Data and Metrics by admin

Scrapy allthough it’s in python is the best swiss army knife tool arround for scraping, forget about nokogiri :) or nasty perl regexp crawlers.. scrapy is serious bussines.

In this guide i will show you how to setup python virtualenvs and install the last version of scrapy into one. If you are not really familiar with virtualenvs i suggest you read http://pypi.python.org/pypi/virtualenv.

Before starting you need to install

Once all packages are installed, lets setup our first virtualenv

Now lets install scrapy from the nightly build version (if you want to install a stable version, just checkout a branch instead)

If you follow every steps and the all mighty gods of linux grant you their grace, hopefully everything should work and you can test it by just running:

Some helpfull links you may find cool:
* http://www.stereoplex.com/blog/understanding-imports-and-pythonpath | Everything you should know about python paths
* http://doc.scrapy.org/en/0.12/topics/architecture.html | Scrapy architecture overview