Edit Question - KnowledgeHub

Title Body

I'm processing very large amounts of data, stored in a dictionary, using multiprocessing. Basically all I'm doing is loading some signatures, stored in a dictionary, building a shared dict object out of it (getting the 'proxy' object returned by Manager.dict() ) and passing this proxy as argument to the function that has to be executed in multiprocessing. Just to clarify: signatures = dict() load_signatures(signatures) [...] manager = Manager() signaturesProxy = manager.dict(signatures) [...] result = pool.map ( myfunction , [ signaturesProxy ]*NUM_CORES ) Now, everything works perfectly if signatures is less than 2 million entries or so. Anyways, I have to process a dictionary with 5.8M keys (pickling signatures in binary format generates a 4.8 GB file). In this case, the process dies during the creation of the proxy object: Traceback (most recent call last): File "matrix.py", line 617, in <module> signaturesProxy = manager.dict(signatures) File "/usr/lib/python2.6/multiprocessing/managers.py", line 634, in temp token, exp = self._create(typeid, *args, **kwds) File "/usr/lib/python2.6/multiprocessing/managers.py", line 534, in _create id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds) File "/usr/lib/python2.6/multiprocessing/managers.py", line 79, in dispatch raise convert_to_error(kind, result) multiprocessing.managers.RemoteError: --------------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/multiprocessing/managers.py", line 173, in handle_request request = c.recv() EOFError --------------------------------------------------------------------------- I know the data structure is huge but I'm working on a machine equipped w/ 32GB of RAM, and running top I see that the process, after loading the signatures, occupies 7GB of RAM. It then starts building the proxy

Tags (comma-separated) Cancel