Running a .NET application on Windows Server 2008 x64 with 16GB of RAM. This application needs to fetch and analyze a very large amount of data (about 64GB), and keep it all in memory at one time.

What I expect to see: Process size expands past 16GB to 64GB. Windows uses virtual memory to page the extra data to/from disk as needed. This is the classic virtual memory use case.

What I actually see: Process size is limited to the amount of physical memory (16GB). Application spends 99.8% of its time in the garbage collector.

Why is our application failing to use virtual memory? Is this a problem in the configuration of the .NET garbage collector, or in the Windows x64 virtual memory manager itself? What can I do to get our application to use virtual memory rather than be limited to physical memory?

Thanks.

-- Brian

Update: I have written a very small program that exhibits the same behavior:

using System;

namespace GCTest
{
    class Program
    {
        static void Main()
        {
            byte[][] arrays = new byte[100000000][];
            for (int i = 0; i < arrays.Length; ++i)
            {
                arrays[i] = new byte[320];
                if (i % 100000 == 0)
                {
                    Console.WriteLine("{0} arrays allocated", i);
                    System.Threading.Thread.Sleep(100);
                }
            }
        }
    }
}

If you want to try it, make sure to build for x64. You may have to modify the constants a bit to stress your system. The behavior I see is that the process bogs down as it approaches a size of 16GB. There is no error message or exception thrown. Performance monitor reports that the % of CPU time in GC approaches 100%.

Isn't this unacceptable? Where's the virtual memory system?

Edit
Report