Alex Rivera | Logout

sorting lines of an enormous file.txt in java

Asked 2012-01-12T09:38:08.773
9

I'm working with a very big text file (755Mb). I need to sort the lines (about 1890000) and then write them back in another file.

I already noticed that discussion that has a starting file really similar to mine: Sorting Lines Based on words in them as keys

The problem is that i cannot store the lines in a collection in memory because I get a Java Heap Space Exception (even if i expanded it at maximum)..(already tried!)

I can't either open it with excel and use the sorting feature because the file is too large and it cannot be completely loaded..

I thought about using a DB ..but i think that writing all the lines then use the SELECT query it's too much long in terms of time executing..am I wrong?

Any hints appreciated Thanks in advance

java sorting text-files lines
Edit
Report

2 Answers

1

Algorithm:

How much memory do we have available? Let’s assume we have X MB of memory available.

  1. Divide the file into K chunks, where X * K = 2 GB. Bring each chunk into memory and sort the lines as usual using any O(n log n) algorithm. Save the lines back to the file.

  2. Now bring the next chunk into memory and sort.

  3. Once we’re done, merge them one by one.

The above algorithm is also known as external sort. Step 3 is known as N-way merge

answered 2012-01-12T09:44:38.890
0

Why don't you try multithreading and increasing heap size of the program you are running? (this also requires you to use merge sort kind of thing provided you have more memory than 755mb in your system.)

answered 2012-01-12T09:43:43.243

Your Answer