Alex Rivera | Logout

How to free memory after opening a file in Python

Asked 2012-09-13T22:39:06.517
17

I'm opening a 3 GB file in Python to read strings. I then store this data in a dictionary. My next goal is to build a graph using this dictionary so I'm closely monitoring memory usage.

It seems to me that Python loads the whole 3 GB file into memory and I can't get rid of it. My code looks like that :

with open(filename) as data:

    accounts = dict()

    for line in data:
        username = line.split()[1]
        IP = line.split()[0]

        try:
            accounts[username].add(IP)
        except KeyError:
            accounts[username] = set()
            accounts[username].add(IP)

print "The accounts will be deleted from memory in 5 seconds"
time.sleep(5)
accounts.clear()

print "The accounts have been deleted from memory"
time.sleep(5)

print "End of script"

The last lines are there so that I could monitor memory usage. The script uses a bit more than 3 GB in memory. Clearing the dictionary frees around 300 MB. When the script ends, the rest of the memory is freed.

I'm using Ubuntu and I've monitored memory usage using both "System Monitor" and the "free" command in terminal.

What I don't understand is why does Python need so much memory after I've cleared the dictionary. Is the file still stored in memory ? If so, how can I get rid of it ? Is it a problem with my OS not seeing freed memory ?

EDIT : I've tried to force a gc.collect() after clearing the dictionary, to no avail.

EDIT2 : I'm running Python 2.7.3 on Ubuntu 12.04.LTS

EDIT3 : I realize I forgot to mention something quite important. My real problem is not that my OS does not "get back" the memory used by Python. It's that later on, Python does not seem to reuse that memory (it just asks for more memory to the OS).

Edit
Report

1 Answer

0

The gc module may be useful, particularly the collect function. I have never used it myself, but from the documentation, it looks like it may be useful. I would try running gc.collect() before you run accounts.clear().

answered 2012-09-13T22:43:29.563

Your Answer