Somehow one of our batches failed last night, the file size that would have been generated is 500mb with around 500,000 records (rows) in it.
Just wonderingly, is there any file size hard limit for .csv files?
I understand that Excel application has 60k row hard limit, but thats opening.
Depending on the way you are handling the CSV file, you might run into memory and/or CPU bottlenecks with large files.
Also, on 32 bit Linux systems there exists a 2GB file size limit which will limit your maximum usable CSV size regardless of your CPU power and memory. Having said that, CSV files that big are a good sign that you should consider a more efficient and robust solution for handling your data such as a database system.