Alex Rivera | Logout

multiprocessing and garbage collection

Asked 2012-03-31T20:51:02.050
14

In py2.6+, the multiprocessing module offers a Pool class, so one can do:

class Volatile(object):
    def do_stuff(self, ...):
        pool = multiprocessing.Pool()
        return pool.imap(...)

However, with the standard Python implementation at 2.7.2, this approach soon leads to "IOError: [Errno 24] Too many open files". Apparently the pool object never gets garbage collected, so its processes never terminate, accumulating whatever descriptors are opened internally. I think this because the following works:

class Volatile(object):
    def do_stuff(self, ...):
        pool = multiprocessing.Pool()
        result = pool.map(...)
        pool.terminate()
        return result

I would like to keep the "lazy" iterator approach of imap; how does the garbage collector work in that case? How to fix the code?

python unix garbage-collection multiprocessing
Edit
Report

1 Answer

8

In the end, I ended up passing the pool reference around and terminating it manually once the pool.imap iterator was finished:

class Volatile(object):
    def do_stuff(self, ...):
        pool = multiprocessing.Pool()
        return pool, pool.imap(...)

    def call_stuff(self):
        pool, results = self.do_stuff()
        for result in results:
            # lazy evaluation of the imap
        pool.terminate()

In case anyone stumbles upon this solution in the future: the chunksize parameter is very important in Pool.imap (as opposed to plain Pool.map, where it didn't matter). I manually set it so that each process receives 1 + len(input) / len(pool) jobs. Leaving it to the default chunksize=1 gave me the same performance as if I didn't use parallel processing at all... bad.

I guess there's no real benefit to using ordered imap vs. ordered map, I just personally like iterators better.

answered 2012-04-01T13:05:42.320

Your Answer