Alex Rivera | Logout

Win32: Write to file without buffering?

Asked 2008-11-25T15:54:24.120
14

I need to create a new file handle so that any write operations to that handle get written to disk immediately.

Extra info: The handle will be the inherited STDOUT of a child process, so I need any output from that process to immediately be written to disk.

Studying the CreateFile documentation, the FILE_FLAG_WRITE_THROUGH flag looked like exactly what I need:

Write operations will not go through any intermediate cache, they will go directly to disk.

I wrote a very basic test program and, well, it's not working. I used the flag on CreateFile then used WriteFile(myHandle,...) in a long loop, writing about 100MB of data in about 15 seconds. (I added some Sleep()'s).

I then set up a professional monitoring environment consisting of continuously hitting 'F5' in explorer. The results: the file stays at 0kB then jumps to 100MB about the time the test program ends.

Next thing I tried was to manually flush the file after each write, with FlushFileBuffers(myHandle). This makes the observed file size grow nice and steady, as expected.

My question is, then, shouldn't the FILE_FLAG_WRITE_THROUGH have done this without manually flushing the file? Am I missing something? In the 'real world' program, I can't flush the file, 'cause I don't have any control over the child process that's using it.

There's also the FILE_FLAG_NO_BUFFERING flag, that I can't be used for the same reason - no control over the process that's using the handle, so I can't manually align the writes as required by this flag.

EDIT: I have made a separate project specifically for watching how the size of the file changes. It uses the .NET FileSystemWatcher class. I also write less d

winapi file-io buffering
Edit
Report

1 Answer

2

Perhaps you could be satisfied enough with FlushFileBuffers:

Flushes the buffers of a specified file and causes all buffered data to be written to a file.

Typically the WriteFile and WriteFileEx functions write data to an internal buffer that the operating system writes to a disk or communication pipe on a regular basis. The FlushFileBuffers function writes all the buffered information for a specified file to the device or pipe.

They do warn that calling flush, to flush the buffers a lot, is inefficient - and it's better to just disable caching (i.e. Tim's answer):

Due to disk caching interactions within the system, the FlushFileBuffers function can be inefficient when used after every write to a disk drive device when many writes are being performed separately. If an application is performing multiple writes to disk and also needs to ensure critical data is written to persistent media, the application should use unbuffered I/O instead of frequently calling FlushFileBuffers. To open a file for unbuffered I/O, call the CreateFile function with the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH flags. This prevents the file contents from being cached and flushes the metadata to disk with each write. For more information, see answered 2010-03-03T20:09:13.907

Your Answer