Edit Question - KnowledgeHub

Title Body

I have a tool to compare 2 csv files and then bucket each cell into one of the 6 buckets. Basically, it reads in the csv files (using fast csv reader, credit: http://www.codeproject.com/KB/database/CsvReader.aspx ) and then creates a dictionary pertaining to each file based on the keys provided by the user. I then iterate through th dictionaries comparing the values and writing a result csv file. While it is blazing fast, it is very inefficient in terms of memory usage. I cannot compare more than 150 MB files on my box with 3 GB physical memory. Here is a code snippet to read the expected file. At the end of this piece, the memory usage is close to 500 MB from the task manager. // Read Expected long rowNumExp; System.IO.StreamReader readerStreamExp = new System.IO.StreamReader(@expFile); SortedDictionary<string, string[]> dictExp = new SortedDictionary<string, string[]>(); List<string[]> listDupExp = new List<string[]>(); using (CsvReader readerCSVExp = new CsvReader(readerStreamExp, hasHeaders, 4096)) { readerCSVExp.SkipEmptyLines = false; readerCSVExp.DefaultParseErrorAction = ParseErrorAction.ThrowException; readerCSVExp.MissingFieldAction = MissingFieldAction.ParseError; fieldCountExp = readerCSVExp.FieldCount; string keyExp; string[] rowExp = null; while (readerCSVExp.ReadNextRecord()) { if (hasHeaders == true) { rowNumExp = readerCSVExp.CurrentRecordIndex + 2; } else { rowNumExp = readerCSVExp.CurrentRecordIndex + 1; } try { rowExp = new string[fieldCount + 1]; } catch (Exception exExpOutOfMemory) { MessageBox.Show(exExpOutOfMemory.Message); Environment.Exit(1); } key

Tags (comma-separated) Cancel