Alex Rivera | Logout

Storage of many log files

Asked 2009-06-24T08:11:33.710
12

I have a system which is receiving log files from different places through http (>10k producers, 10 logs per day, ~100 lines of text each).

I would like to store them to be able to compute misc. statistics over them nightly , export them (ordered by date of arrival or first line content) ...

My question is : what's the best way to store them ?

  • Flat text files (with proper locking), one file per uploaded file, one directory per day/producer
  • Flat text files, one (big) file per day for all producers (problem here will be indexing and locking)
  • Database Table with text (MySQL is preferred for internal reasons) (pb with DB purge as delete can be very long !)
  • Database Table with one record per line of text
  • Database with sharding (one table per day), allowing simple data purge. (this is partitioning. However the version of mysql I have access to (ie supported internally) does not support it)
  • Document based DB à la couchdb or mongodb (problem could be with indexing / maturity / speed of ingestion)

Any advice ?

database language-agnostic logging
Edit
Report

1 Answer

1

Since you would like to store them to be able to compute misc. statistics over them nightly , export them (ordered by date of arrival or first line content) ... You're expecting 100,000 files a day, at a total of 10,000,000 lines:

I'd suggest:

  1. Store all the files as regular textfiles using the following format : yyyymmdd/producerid/fileno.
  2. At the end of the day, clear the database, and load all the textfiles for the day.
  3. After loading the files, it would be easy to get the stats from the database, and post them in any format needed. (maybe even another "stats" database). You could also generate graphs.
  4. To save space ,you could compress the daily folder. Since they're textfiles, they would compress well.

So you would only be using the database to be able to easily aggregate the data. You could also reproduce the reports for an older day if the process didn't work, by going through the same steps.

answered 2009-06-26T10:25:36.687

Your Answer