Alex Rivera | Logout

Is it possible to use threads to speed up file reading?

Asked 2010-06-16T14:54:59.650
23

I want to read a file as fast as possible (40k lines) [Edit : the rest is obsolete].

Edit: Andres Jaan Tack suggested a solution based on one thread per file, and I want to be sure I got this (thus this is the fastest way) :

  • One thread per entry file reads it whole and stocks its content in a container associated (-> as many containers as there are entry files)
  • One thread calculates the linear combination of every cell read by the input threads, and stocks the results in the exit container (associated to the output file).
  • One thread writes by block (every 4kB of data, so about 10 lines) the content of the output container.

Should I deduce that I must not use m-mapped files (because the program's on standby waiting for the data) ?

Thanks aforehand.

Sincerely,

Mister mystère.

c++ multithreading file operating-system filesystems
Edit
Report

2 Answers

4

Yes, it's a waste of time. At very best you'll end up with about the same performance. At worst, it might hurt performance from the disk seeking to different parts of the file instead of reading through it consecutively.

answered 2010-06-16T15:00:50.680
4

In contrast to other readers I believe that theoretically there can be some benifit, even if you're running on an SP (single-processor) system. However I'd never do this for as much as 40K lines (assuming you talk about normal-sized lines).

They key is Amardeep's answer, where he/she says that creating threads is useful when a thread becomes blocked for some reason.

Now, how do mapped files "work"? When you access a memory page in that region for the first time - the processor generates a page fault. The OS loads the contents of the file (this involves disk access) into the memory page. Then the execution returns to your thread.

I also believe upon page fault the OS fills a bunch of consecutive pages, not just single one.

Now, what's important is that during the page fault processing your thread is suspended. Also during this period the CPU isn't loaded (apart from what other processes may do).

So that if you look at the time scale you see a period of two sections: one where CPU is loaded (here you read the contents of the page and do some processing), and one where CPU is nearly idle and the I/O on the disk is performed.

On the other hand you may create several threads, each one is assigned to read a different portion of the file. You benefit from two effects:

  1. Other thread has a chance to load the CPU (or several CPUs if MP system) when one is blocked by I/O.

  2. Even in case where the processing is very short (hence the CPU is not the bottleneck) - still there's a benefit. It's related to the fact that if you issue several I/O on the same physical device - it has a chance to perform them more efficiently.

For instance, when reading many different sectors from the HD drive you can actually read them all within one disk rotation.

P.S.

And, of course, I'd never thought to do this for 40K lines. The overh

answered 2010-06-16T15:20:13.167

Your Answer