Alex Rivera | Logout

Baffling non-object python memory leak

Asked 2013-01-09T08:07:32.483
10

I have a django webserver running under uwsgi that appears to leak memory.

Specifically, the RSS of the processes slowly grow until eventually I have to restart it.

I am aware of the other similar questions to this, however all the solutions/conclusions found so far don't appear to apply (that I can find) in this case.

So far, I have used meliae, Heapy, pympler and objgraph to inspect the python heap and they all report the same thing: A normal looking heap using about 40MB of memory (expected) with very little variance over time (as desired).

This is unfortunately entirely inconsistent with the process RSS, which will happily grow to 400MB+ with no reflection in the python heap size.

Some sample output to illustrate my point -

Pympler output comparing python heap/object memory vs process RSS:

Memory snapshot:
                                        types |   # objects |   total size
============================================= | =========== | ============
                                         dict |       20868 |     19852512
                                          str |      118598 |     11735239
                                      unicode |       19038 |     10200248
                                        tuple |       58718 |      5032528
                                         type |        1903 |      1720312
                                         code |       13225 |      1587000
                                         list |       11393 |      1289704
                            datetime.datetime |        6953 |       333744
                                          int |       12615 |       302760
  <clas
Edit
Report

1 Answer

2

What version of Python are you using? In Python 2.4 memory was not returned to the OS by the Python memory allocator.

Still in newer versions you can see a problem that's either related to Python's memory allocator that keeps lists of freed simple types, or in case you are running on Linux an issue intrinsic to how glibc's malloc implementation allocates memory from the OS. Take a look at http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm and http://pushingtheweb.com/2010/06/python-and-tcmalloc/.

answered 2013-01-09T09:01:46.067

Your Answer