What's the most efficient way to process really large binary files in Haskell?
The standard answer is to read the entire file as a lazy ByteString and then use something like the Binary packet to write a parser over it. There are a couple of problems with that...
First, libraries like Binary don't really handle parse failure, and I'm explicitly expecting parsing to fail sometimes.
Second, I'm not parsing the entire file contents. I'm going to be skipping over large chunks of it. And reading gigabytes of data from disk into RAM only to have the garbage collector throw it away again seems rather unperformant.
Related to that, I need to be able to tell if the skip I want to perform will take me off the end of the file or not (and error out if it does).
I may also need to seek backwards, or maybe to a specific byte offset within the file, which does not appear to be well-supported by a lazy ByteString approach. (There's a severe danger of ending up holding the entire file in RAM.)
The alternative, of course, is to read individual bytes one by one, interleaved with hSeek commands. But now the problem is, how efficient is reading a file one byte at a time? That sounds like it could also be quite slow. I'm not sure is hSetBuffering has an effect on this. (?)
Then of course there's mmap. But that seems to freak out the virtual memory system if used on large files. (Which is odd, considering that's the entire purpose for it existing...)
What do we think, folks? What's the best way to approach this, in terms of I/O performance and code maintainability?