SOLR - Best approach to import 20 million documents from csv file

Asked 2012-02-25T23:47:35.890

17

My current task on hand is to figure out the best approach to load millions of documents in solr. The data file is an export from DB in csv format.

Currently, I am thinking about splitting the file into smaller files and having a script while post this smaller ones using curl.

I have noticed that if u post high amount of data, most of the time the request times out.

I am looking into Data importer and it seems like a good option

Any others ideas highly appreciated

Thanks

csv import solr bulkinsert dataimporthandler

Edit Report

1 Answer

12

In SOLR 4.0 (currently in BETA), CSV's from a local directory can be imported directly using the UpdateHandler. Modifying the example from the SOLR Wiki

curl http://localhost:8983/solr/update?stream.file=exampledocs/books.csv&stream.contentType=text/csv;charset=utf-8

And this streams the file from the local location, so no need to chunk it up and POST it via HTTP.

answered 2012-10-11T14:26:38.077

Your Answer