How do I iterate through sites with Scrapy? I'd like to extract the body of all sites that match http://www.saylor.org/site/syllabus.php?cid=NUMBER, where NUMBER is 1 through 400 or so.

I've written this spider:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from syllabi.items import SyllabiItem

class SyllabiSpider(CrawlSpider):

    name = 'saylor'
    allowed_domains = ['saylor.org']
    start_urls = ['http://www.saylor.org/site/syllabus.php?cid=']
    rules = [Rule(SgmlLinkExtractor(allow=['\d+']), 'parse_syllabi')]

    def parse_syllabi(self, response):
        x = HtmlXPathSelector(response)

        syllabi = SyllabiItem()
        syllabi['url'] = response.url
        syllabi['body'] = x.select("/html/body/text()").extract()
        return syllabi

But it doesn't work. I understand it's looking for links in that start_url, which is not really what I want it to do. I want to iterate through the sites. Make sense?

Thanks for the help.

python scrapy
Edit
Report