Alex Rivera | Logout

How to access scrapy settings from item Pipeline

Asked 2012-12-28T21:19:39.267
35

How do I access the scrapy settings in settings.py from the item pipeline. The documentation mentions it can be accessed through the crawler in extensions, but I don't see how to access the crawler in the pipelines.

Edit
Report

1 Answer

31

Ok, so the documentation at http://doc.scrapy.org/en/latest/topics/extensions.html says that

The main entry point for a Scrapy extension (this also includes middlewares and pipelines) is the from_crawler class method which receives a Crawler instance which is the main object controlling the Scrapy crawler. Through that object you can access settings, signals, stats, and also control the crawler behaviour, if your extension needs to such thing.

So then you can have a function to get the settings.

@classmethod
def from_crawler(cls, crawler):
    settings = crawler.settings
    my_setting = settings.get("MY_SETTING")
    return cls(my_setting)

The crawler engine then calls the pipeline's init function with my_setting, like so:

def __init__(self, my_setting):
    self.my_setting = my_setting

And other functions can access it with self.my_setting, as expected.

Alternatively, in the from_crawler() function you can pass the crawler.settings object to __init__(), and then access settings from the pipeline as needed instead of pulling them all out in the constructor.

answered 2012-12-28T21:19:39.267

Your Answer