Edit Question - KnowledgeHub

Title Body

I'm trying to get some content automatically from a site using requests (and bs4) I have a script that gets a cookie: def getCookies(self): username = 'username' password = 'password' URL = 'logonURL' r = requests.get(URL, auth=('username', 'password')) cookies = r.cookies dump of the cookies looks like: <<class 'requests.cookies.RequestsCookieJar'>[<Cookie ASP.NET_SessionId=yqokjr55ezarqbijyrwnov45 for URL.com/>, <Cookie BIGipServerPE_Journals.lww.com_80=1440336906.20480.0000 for URL.com/>, <Cookie JournalsLockCookie=id=a5720750-3f20-4207-a500-93ae4389213c&ip=IP address for URL.com/>]> But when I pass the cookie object to the next URL: soup = Soup(s.get(URL, cookies = cookies).content) its not working out - I can see by dumping the soup that I'm not giving the webserver my credentials properly I tried running a requests session: def getCookies(self): self.s = requests.session() username = 'username' password = 'password' URL = 'logURL' r = self.s.get(URL, auth=('username', 'password')) and I get the same no joy. I looked at the header via liveHttp in FF when I visit the 2nd page, and see a very different form: Cookie: WT_FPC=id=264b0aa85e0247eb4f11355304127862:lv=1355317068013:ss=1355314918680; UserInfo=Username=username; BIGipServerPE_Journals.lww.com_80=1423559690.20480.0000; PlatformAuthCookie=true; Institution=ReferrerUrl=http://logonURL.com/?wa=wsignin1.0&wtrealm=urn:adis&wctx=http://URL.com/_layouts/Authenticate.aspx?Source=%252fpecnews%252ftoc%252f2012%252f06440&token=method|ExpireAbsolute; counterSessionGuidId=6e2bd57f-b6da-4dd4-bcb0-742428e08b5e; MyListsRefresh=12/13/2012 12:59:04 AM; ASP.NET_SessionId=40a04p45zppozc45wbadah45; JournalsLockCookie=id=85d1f38f-dcbb-476a-bc2e-92f7ac1ae493&i

Tags (comma-separated) Cancel