For the context of a soft real time system that should not pause for more than 200ms, We're looking for a way to have an advance warning before a Full GC is imminent. We realize we might not be able to avoid it, but we'd like to fail over to another node before the system stalls.
We've been able to come up with a scheme that will provide us with an advance warning, ahead of imminent full GC that may cause the system to stall for several seconds (which we need to avoid).
What we've been able to come up with relies on CMS free list statistics: -XX:PrintFLSStatistics=1. This prints free list statistics into the GC log after every GC cycle, including young GC, so the information is available at short intervals, and will appear even more frequently during intervals of high memory allocation rate. It probably costs a little in terms of performance, but our working assumption is that we can afford it.
The output to the log looks like so:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 382153298
Max Chunk Size: 382064598
Number of Blocks: 28
Av. Block Size: 13648332
Tree Height: 8
In particular, the maximum free chunk size is 382064598 words. With 64-bit words this should amount to just below 2915MB. This number has been decreasing very slowly, at a rate of roughly 1MB per hour.
It is our understanding that so long as the maximum free chunk size is larger than the young generation (assuming no humungous object allocation), every object promotion should succeed.
Recently, we've run a several-days-long stress tests, and have been seeing that CMS was able to maintain maximum chunk sizes upward of 94% of total old region space. The maximum free chunk size appears to be decreasing at a rate of less than 1MB/hour, which should be fine -- according to this we won't be hitting full GC any time soon, and the servers will likely be down for ma