How to install Scrapy on debian using virtualenv

Posted febrero 15th, 2012 in Blog, Data and Metrics and tagged , , by admin

Scrapy allthough it’s in python is the best swiss army knife tool arround for scraping, forget about nokogiri :) or nasty perl regexp crawlers.. scrapy is serious bussines.

In this guide i will show you how to setup python virtualenvs and install the last version of scrapy into one. If you are not really familiar with virtualenvs i suggest you read http://pypi.python.org/pypi/virtualenv.

Before starting you need to install

Once all packages are installed, lets setup our first virtualenv

Now lets install scrapy from the nightly build version (if you want to install a stable version, just checkout a branch instead)

If you follow every steps and the all mighty gods of linux grant you their grace, hopefully everything should work and you can test it by just running:

Some helpfull links you may find cool:
* http://www.stereoplex.com/blog/understanding-imports-and-pythonpath | Everything you should know about python paths
* http://doc.scrapy.org/en/0.12/topics/architecture.html | Scrapy architecture overview

Leave a Reply