Edit Question - KnowledgeHub

Title Body

I am trying to access the session cookie within a spider. I first login to a social network using in a spider: def parse(self, response): return [FormRequest.from_response(response, formname='login_form', formdata={'email': '...', 'pass':'...'}, callback=self.after_login)] In after_login , I would like to access the session cookies, in order to pass them to another module (selenium here) to further process the page with an authentificated session. I would like something like that: def after_login(self, response): # process response ..... # access the cookies of that session to access another URL in the # same domain with the autehnticated session. # Something like: session_cookies = XXX.get_session_cookies() data = another_function(url,cookies) Unfortunately, response.cookies does not return the session cookies. How can I get the session cookies ? I was looking at the cookies middleware: scrapy.contrib.downloadermiddleware.cookies and scrapy.http.cookies but there doesn't seem to be any straightforward way to access the session cookies. Some more details here bout my original question: Unfortunately, I used your idea but I dind't see the cookies, although I know for sure that they exists since the scrapy.contrib.downloadermiddleware.cookies middleware does print out the cookies! These are exactly the cookies that I want to grab. So here is what I am doing: The after_login(self,response) method receives the response variable after proper authentication, and then I access an URL with the session data: def after_login(self, response): # testing to see if I can get the session cookies cookieJar = response.meta.setdefault('coo

Tags (comma-separated) Cancel