Edit Question - KnowledgeHub

Title Body

I've built a crawler that had to run on about 5M pages (by increasing the url ID) and then parses the pages which contain the info' I need. after using an algorithm which run on the urls (200K) and saved the good and bad results I found that the I'm wasting a lot of time. I could see that there are a a few returning subtrahends which I can use to check the next valid url. you can see the subtrahends quite fast (a little ex' of the few first "good IDs") - 510000011 # +8 510000029 # +18 510000037 # +8 510000045 # +8 510000052 # +7 510000060 # +8 510000078 # +18 510000086 # +8 510000094 # +8 510000102 # +8 510000110 # etc' 510000128 510000136 510000144 510000151 510000169 510000177 510000185 510000193 510000201 after crawling about 200K urls which gave me only 14K good results I knew I was wasting my time and need to optimize it, so I run some statistics and built a function that will check the urls while increasing the id with 8\18\17\8 (top returning subtrahends ) etc'. this is the function - def checkNextID(ID): global numOfRuns, curRes, lastResult while ID < lastResult: try: numOfRuns += 1 if numOfRuns % 10 == 0: time.sleep(3) # sleep every 10 iterations if isValid(ID + 8): parseHTML(curRes) checkNextID(ID + 8) return 0 if isValid(ID + 18): parseHTML(curRes) checkNextID(ID + 18) return 0 if isValid(ID + 7): parseHTML(curRes) checkNextID(ID + 7) return 0 if isValid(ID + 17): parseHTML(curRes) checkNextID(ID + 17) return 0 if isValid(ID+6): parseHTML(curRes) checkNextID(ID + 6) return 0 if isValid(ID + 16): par

Tags (comma-separated) Cancel