Reserve memory for list in Python?

Asked 2009-02-11T14:41:14.923

When programming in Python, is it possible to reserve memory for a list that will be populated with a known number of items, so that the list will not be reallocated several times while building it? I've looked through the docs for a Python list type, and have not found anything that seems to do this. However, this type of list building shows up in a few hotspots of my code, so I want to make it as efficient as possible.

Edit: Also, does it even make sense to do something like this in a language like Python? I'm a fairly experienced programmer, but new to Python and still getting a feel for its way of doing things. Does Python internally allocate all objects in separate heap spaces, defeating the purpose of trying to minimize allocations, or are primitives like ints, floats, etc. stored directly in lists?

Edit Report

2 Answers

Here's four variants:

an incremental list creation
"pre-allocated" list
array.array()
numpy.zeros()

python -mtimeit -s"N=10**6" "a = []; app = a.append;"\
    "for i in xrange(N):  app(i);"
10 loops, best of 3: 390 msec per loop

python -mtimeit -s"N=10**6" "a = [None]*N; app = a.append;"\
    "for i in xrange(N):  a[i] = i"
10 loops, best of 3: 245 msec per loop

python -mtimeit -s"from array import array; N=10**6" "a = array('i', [0]*N)"\
    "for i in xrange(N):" "  a[i] = i"
10 loops, best of 3: 541 msec per loop

python -mtimeit -s"from numpy import zeros; N=10**6" "a = zeros(N,dtype='i')"\
    "for i in xrange(N):" "  a[i] = i"
10 loops, best of 3: 353 msec per loop

It shows that [None]*N is the fastest and array.array is the slowest in this case.

answered 2009-02-11T15:47:29.947

In Python, all objects are allocated on the heap.
But Python uses a special memory allocator so malloc won't be called every time you need a new object.
There are also some optimizations for small integers (and the like) which are cached; however, which types, and how, is implementation dependent.

answered 2009-02-11T15:19:39.483

Reserve memory for list in Python?

2 Answers

Your Answer