I'm writing a script(multi-threaded) to retrieve contents from a website, and the site's not very stable so every now and then there's hanging http request which cannot even be time-outed by socket.setdefaulttimeout(). Since I have no control over that website, the only thing I can do is to improve my codes but I'm running out of ideas right now.
Sample codes:
socket.setdefaulttimeout(150)
MechBrowser = mechanize.Browser()
Header = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)'}
Url = "http://example.com"
Data = "Justatest=whatever&letstry=doit"
Request = urllib2.Request(Url, Data, Header)
Response = MechBrowser.open(Request)
Response.close()
What should I do to force the hanging request to quit? Actually I want to know why socket.setdefaulttimeout(150) is not working in the first place. Anybody can help me out?
Added:(and yes problem still not solved)
OK, I've followed tomasz's suggestion and changed codes to MechBrowser.open(Request, timeout = 60), but same thing happens. I still got hanging requests randomly till now, sometimes it's several hours and other times it could be several days. What do I do now? Is there a way to force these hanging requests to quit?