.NET Does NOT Have Reliable Asynchronouos Socket Communication?

I once wrote a Crawler in .NET. In order to improve its scalability, I tried to take advantage of asynchronous API of .NET.

The System.Net.HttpWebRequest has asynchronous API BeginGetResponse/EndGetResponse. However, this pair of API is just to get a HTTP response headers and a Stream instance from which we can extract HTTP response content. So, my strategy is to use BeginGetResponse/EndGetResponse to asynchronously get the response Stream, then use BeginRead/EndRead to asynchronously get bytes from the response Stream instance.

Everything seems perfect until the Crawler goes to stress test. Under stress test, the Crawler suffers from high memory usage. I checked the memory with WinDbg+SoS and found out that lots of byte arrays are pined by System.Threading.OverlappedData instances. After some searching in internet, I found this KB http://support.microsoft.com/kb/947862 from microsoft.

According to the KB, the number of asynchronous I/O should have a "upper bound", but it doesn't tell a "suggested" bound value. So, in my eye, this KB helps nothing. This is obviously a .NET bug. Finally, I have to drop the idea to do asynchronous extracting bytes from response Stream, and just do it in synchronous way.

The .NET library that allows Asynchronous IO with dot net sockets (Socket.BeginSend / Socket.BeginReceive / NetworkStream.BeginRead / NetworkStream.BeginWrite) must have an upper bound on the amount of buffers outstanding (either send or receive) with their asynchronous IO.

The network application should have an upper bound on the number of outstanding asynchronous IO that it posts.

Edit: Add some question marks.

Anybody has any experience to do asynchronous I/O on Socket & NetworkStream? Generally speaking, does crawler in production do I/O with internet with Synchronous or Asynch

Edit Report

This is not a .NET framework problem. The linked KB article could have been a bit more explanatory, every .NET program needs to run on an operating system and deal with its limitations. An operating system does not publish what it can do, or what resources are left, so no option of counting-down what you consumed. Necessarily so, its resources need to be shared by all programs that run on it.

And not limited to sockets, something as basic as memory is not endlessly available. As the name of the website helps us remember. If you use too much then you'll find out, the OS fails the request and you see that back as an exception in your program.

Resource management is still very much our job. A basic workaround in a stress test is to use a SemaphoreSlim. Call its Wait() method before you start the request, Release() when it completes. There is more than one OS limit. The TCP/IP design can't have more than 65535 active ports. The buffer used for the transfer needs to be pinned so the network device driver can write to it, that's the one you hit here. For a stress test, initializing the semaphore to 1000 is a decent and very high limit. Experiment to see how high you can go.

answered 2008-10-25T14:56:04.483

.NET Does NOT Have Reliable Asynchronouos Socket Communication?

1 Answer

Your Answer