How can I make scrapy crawl break and exit when encountering the first exception?

Asked 2012-03-01T22:06:50.577

For development purposes, I would like to stop all scrapy crawling activity as soon a first exception (in a spider or a pipeline) occurs.

Any advice?

python exception scrapy

Edit Report

1 Answer

In spider, you can just throw CloseSpider exception.

def parse_page(self, response):
    if 'Bandwidth exceeded' in response.body:
        raise CloseSpider('bandwidth_exceeded')

For others (middlewares, pipeline, etc), you can manually call close_spider as akhter mentioned.

answered 2013-04-23T03:30:45.290