How to read through collection in chunks by 1000?

Asked 2012-03-20T12:27:13.083

I need to read whole collection from MongoDB ( collection name is "test" ) in Python code. I tried like

    self.__connection__ = Connection('localhost',27017)
    dbh = self.__connection__['test_db']            
    collection = dbh['test']

How to read through collection in chunks by 1000 ( to avoid memory overflow because collection can be very large ) ?

python mongodb pymongo

Edit Report

1 Answer

I agree with Remon, but you mention batches of 1000, which his answer doesn't really cover. You can set a batch size on the cursor:

cursor.batch_size(1000);

You can also skip records, e.g.:

cursor.skip(4000);

Is this what you're looking for? This is effectively a pagination pattern. However, if you're just trying to avoid memory exhaustion then you don't really need to set batch size or skip.

answered 2012-03-20T14:28:25.370

How to read through collection in chunks by 1000?

1 Answer

Your Answer