I want to scrape a page of data (using the Python Scrapy library) without having to define each individual field on the page. Instead I want to dynamically generate fields using the id of the element as the field name.

At first I was thinking the best way to do this would be to have a pipeline that collects all the data, and outputs it once it has it all.

Then I realised that I need to pass the data to the pipeline in an item, but I can't define an item as I don't know what fields it will need!

What's the best way for me to tackle this problem?

Edit
Report