I recently did some performance testing and analysis of an ASP.NET application using out-of-process session state - this is necessary when using session state on a web farm so that state can be retrieved on any of the web servers, e.g. if subsequent HTTP requests arrive at a different server because the sessions aren't 'sticky' or the original server is down, etc.
What surprised me was that when I ran the web servers at full load and profiled the CPU usage something like 99% of CPU time was spent serializing and deserializing session state. Subsequently we implemented a customised 'caching' state server; this always serializes state but also keeps state in-memory so that if you use sticky sessions the state doesn't have to be deserialized most of the time. This improved server throughput by a factor of 2; However, serialization still makes up 98% or more of CPU time.
We obtained some further improvements in speed by 'trimming' unnecessary object references between objects in the session state prior to serialization - fixing up the references manually upon desrialization. This improved speed by another 10-20% or so. The reasoning here is that some of the performance loss is due to the built in serialization having to walk the graph of object pointers, which becomes a more complex task with more pointers.
Continuining the investigation we wrote customized serialization routines for some of our classes rather than relying on .Net's built-in serialization. What we found was that performance was greatly improved, by a factor of about 50x. It seems the bulk of the CPU load is caused by built in .Net serialization, which in turn is slow due to reliance on using Reflection to walk the object pointers/graph and extract field data.
It's very tempting to boost our performance by 50x, thus reducing the web server hardware requirements by a large factor (and power requirements by a lesser but still significant