Alex Rivera | Logout

Is multiprocessing.Manager().dict().setdefault() broken?

Asked 2012-05-29T22:31:35.890
8

The its-late-and-im-probably-stupid department presents:

>>> import multiprocessing
>>> mgr = multiprocessing.Manager()
>>> d = mgr.dict()
>>> d.setdefault('foo', []).append({'bar': 'baz'})
>>> print d.items()
[('foo', [])]         <-- Where did the dict go?

Whereas:

>>> e = mgr.dict()
>>> e['foo'] = [{'bar': 'baz'}]
>>> print e.items()
[('foo', [{'bar': 'baz'}])]

Version:

>>> sys.version
'2.7.2+ (default, Jan 20 2012, 23:05:38) \n[GCC 4.6.2]'

Bug or wug?

EDIT: More of the same, on python 3.2:

>>> sys.version
'3.2.2rc1 (default, Aug 14 2011, 21:09:07) \n[GCC 4.6.1]'

>>> e['foo'] = [{'bar': 'baz'}]
>>> print(e.items())
[('foo', [{'bar': 'baz'}])]

>>> id(type(e['foo']))
137341152
>>> id(type([]))
137341152

>>> e['foo'].append({'asdf': 'fdsa'})
>>> print(e.items())
[('foo', [{'bar': 'baz'}])]

How can the list in the dict proxy not contain the additional element?

python multiprocessing
Edit
Report

2 Answers

9

This is some pretty interesting behavior, I am not exactly sure how it works but I'll take a crack at why the behavior is the way it is.

First, note that multiprocessing.Manager().dict() is not a dict, it is a DictProxy object:

>>> d = multiprocessing.Manager().dict()
>>> d
<DictProxy object, typeid 'dict' at 0x7fa2bbe8ea50>

The purpose of the DictProxy class is to give you a dict that is safe to share across processes, which means that it must implement some locking on top of the normal dict functions.

Apparently part of the implementation here is to not allow you to directly access mutable objects nested inside of a DictProxy, because if that was allowed you would be able to modify your shared object in a way that bypasses all of the locking that makes DictProxy safe to use.

Here is some evidence that you can't access mutable objects, which is similar to what is going on with setdefault():

>>> d['foo'] = []
>>> foo = d['foo']
>>> id(d['foo'])
140336914055536
>>> id(foo)
140336914056184

With a normal dictionary you would expect d['foo'] and foo to point to the same list object, and modifications to one would modify the other. As you have seen, this is not the case for the DictProxy class because of the additional process safety requirement imposed by the multiprocessing module.

edit: The following note from the multiprocessing documentation clarifies what I was trying to say above:

Note: Modifications to mutable values or items in dict and list proxies will not be prop

answered 2012-05-29T23:09:12.147
2

The Manager().dict() is a DictProxy object:

>>> mgr.dict()
<DictProxy object, typeid 'dict' at 0x1007bab50>
>>> type(mgr.dict())
<class 'multiprocessing.managers.DictProxy'>

DictProxy is a subclass of the BaseProxy type, which does not behave entirely like a normal dict: http://docs.python.org/library/multiprocessing.html?highlight=multiprocessing#multiprocessing.managers.BaseProxy

So, it seems you have to address the mgr.dict() differently than you would a base dict.

answered 2012-05-29T23:01:48.920

Your Answer