Alex Rivera | Logout

Fastest Matlab file reading?

Asked 2012-02-25T02:21:18.783
38

My MATLAB program is reading a file about 7m lines long and wasting far too much time on I/O. I know that each line is formatted as two integers, but I don't know exactly how many characters they take up. str2num is deathly slow, what matlab function should I be using instead?

Catch: I have to operate on each line one at a time without storing the whole file memory, so none of the commands that read entire matrices are on the table.

fid = fopen('file.txt');
tline = fgetl(fid);
while ischar(tline)
    nums = str2num(tline);    
    %do stuff with nums
    tline = fgetl(fid);
end
fclose(fid);
matlab file-io
Edit
Report

2 Answers

63

Problem statement

This is a common struggle, and there is nothing like a test to answer. Here are my assumptions:

  1. A well formatted ASCII file, containing two columns of numbers. No headers, no inconsistent lines etc.

  2. The method must scale to reading files that are too large to be contained in memory, (although my patience is limited, so my test file is only 500,000 lines).

  3. The actual operation (what the OP calls "do stuff with nums") must be performed one row at a time, cannot be vectorized.

Discussion

With that in mind, the answers and comments seem to be encouraging efficiency in three areas:

  • reading the file in larger batches
  • performing the string to number conversion more efficiently (either via batching, or using better functions)
  • making the actual processing more efficient (which I have ruled out via rule 3, above).

Results

I put together a quick script to test out the ingestion speed (and consistency of result) of 6 variations on these themes. The results are:

  • Initial code. 68.23 sec. 582582 check
  • Using sscanf, once per line. 27.20 sec. 582582 check
  • Using fscanf in large batches. 8.93 sec. 582582 check
  • Using textscan in large batches. 8.79 sec. 582582 check
  • Reading large batches into memory, then sscanf. 8.15 sec. 582582 check
  • Using java single line file reader and sscanf on single lines. 63.56 sec. 582582 check
  • Using java single item token scanner. 81.19 sec. 582582 check
  • Fully batched operations (non-compliant). 1.02 sec. 508680 check (violates rule 3)

Summary

More than half of the original time (68 -> 27 sec) was consumed with inefficiencies in the str2num call, which can be removed by switching the sscanf.

About a

answered 2012-02-25T06:23:50.297
4

I have had good results (speedwise) using memmapfile(). This minimises the amount of memory data copying, and makes use of the kernel's IO buffering. You need enough free address space (though not actual free memory) to map the entire file, and enough free memory to hold the output variable (obviously!)

The example code below reads a text file into a two-column matrix data of int32 type.

fname = 'file.txt';
fstats = dir(fname);
% Map the file as one long character string
m = memmapfile(fname, 'Format', {'uint8' [ 1 fstats.bytes] 'asUint8'});
textdata = char(m.Data(1).asUint8);
% Use textscan() to parse the string and convert to an int32 matrix
data = textscan(textdata, '%d %d', 'CollectOutput', 1);
data = data{:};
% Tidy up!
clear('m')

You may need to fiddle with the parameters to textscan() to get exactly what you want - see the online docs.

answered 2012-02-27T14:19:31.560

Your Answer