Edit Question - KnowledgeHub

Title Body

I have a script that fetches several web pages and parses the info. (An example can be seen at http://bluedevilbooks.com/search/?DEPT=MATH&CLASS=103&SEC=01 ) I ran cProfile on it, and as I assumed, urlopen takes up a lot of time. Is there a way to fetch the pages faster? Or a way to fetch several pages at once? I'll do whatever is simplest, as I'm new to python and web developing. Thanks in advance! :) UPDATE: I have a function called fetchURLs() , which I use to make an array of the URLs I need so something like urls = fetchURLS() .The URLS are all XML files from Amazon and eBay APIs (which confuses me as to why it takes so long to load, maybe my webhost is slow?) What I need to do is load each URL, read each page, and send that data to another part of the script which will parse and display the data. Note that I can't do the latter part until ALL of the pages have been fetched, that's what my issue is. Also, my host limits me to 25 processes at a time, I believe, so whatever is easiest on the server would be nice :) Here it is for time: Sun Aug 15 20:51:22 2010 prof 211352 function calls (209292 primitive calls) in 22.254 CPU seconds Ordered by: internal time List reduced from 404 to 10 due to restriction <10> ncalls tottime percall cumtime percall filename:lineno(function) 10 18.056 1.806 18.056 1.806 {_socket.getaddrinfo} 4991 2.730 0.001 2.730 0.001 {method 'recv' of '_socket.socket' objects} 10 0.490 0.049 0.490 0.049 {method 'connect' of '_socket.socket' objects} 2415 0.079 0.000 0.079 0.000 {method 'translate' of 'unicode' objects} 12 0.061 0.005 0.745 0.062 /usr/local/lib/python2.6/HTMLParser.py:132(goahead) 3428

Tags (comma-separated) Cancel