KnowledgeHub
Questions
Tags
Users
Search
Alex Rivera
|
Logout
Edit Question
Title
Body
I want to scrape a page of data (using the Python Scrapy library) without having to define each individual field on the page. Instead I want to dynamically generate fields using the id of the element as the field name. At first I was thinking the best way to do this would be to have a pipeline that collects all the data, and outputs it once it has it all. Then I realised that I need to pass the data to the pipeline in an item, but I can't define an item as I don't know what fields it will need! What's the best way for me to tackle this problem?
Tags (comma-separated)
Save Edits
Cancel