I am looking for a design pattern that handles large data sets over the internet, and does periodic updating of these objects. I am developing an application that will display thousands of records in the UI at one time. Additionally, various properties on these objects are quite transient and need to be updated on the client to keep the user aware of the changing state of these records in the system. I have a few ideas how to approach this problem, but figured there might be a design pattern (or patterns) out there that handles this type of scenario.
Limitations:
- The client-side for this is being written in Silverlight.
- The objects themselves are not very big (about 15 value-type and string properties), but querying for all the data is expensive. The 15 or so properties contain data from various sources; no clever join statement or indexing is going to speed up the query. I am thinking of populating only a subset of the properties on initial load and then filling in the more expensive details as the user zooms in on a given grouping of objects. Think Google maps, but instead of streets and building it is showing the objects.
- I will be able to limit the portion of the thousands of objects that are being updated. However, I will need the user to be able to "zoom out" of an context that allows granular updating to one that shows all the thousands of objects. I imagine that updating will be disabled again for objects when they leave a sufficient zoom context.
Ideas on how to tackle all or part of this problem? Like I mentioned I am considering a few ideas already, but nothing I have put together so far gives me a good feeling about the success of this project.
Edit:
I think the difficult parts really boil down to two things for which I may need two distinct patterns/practices/strategies:
- Loading a large number of records over the internet (~5k).
- Kee