I'm trying to insert a large amount of records (in the millions) into an SQLite database. The data is read in from a file stream (C++). If I begin a single transaction, perform all of the inserts and then commit the transaction, I only get a very small percentage of records actually inserted in my database.
The ones that are inserted seem random -- I can't really see any pattern for which ones do get inserted and which ones get left out. However, if I commit and then begin the transaction again after something like 2000 inserts, I don't have this problem and all of the records are inserted, even though the process is much slower. So...
Is there a strict limit to how many inserts can be done within one transaction? Is there a way to change this limit?