Update DynamoDB Atomic Counter with Python / Boto

Asked 2012-04-26T13:32:28.153

I am trying to update an atomic count counter with Python Boto 2.3.0, but can find no documentation for the operation.

It seems there is no direct interface, so I tried to go to "raw" updates using the layer1 interface, but I was unable to complete even a simple update.

I tried the following variations but all with no luck

dynoConn.update_item(INFLUENCER_DATA_TABLE, 
                     {'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"}, 
                     {'new': {'Value': {'N':"1"}, 'Action': "ADD"}})    

dynoConn.update_item('influencer_data', 
                     {'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"}, 
                     {'new': {'S' :'hello'}})                                 

dynoConn.update_item("influencer_data", 
                     {"HashKeyElement": "9f08b4f5-d25a-4950-a948-0381c34aed1c"},
                     {"AttributesToPut" : {"new": {"S" :"hello"}}})

They all produce the same error:

  File "/usr/local/lib/python2.6/dist-packages/boto-2.3.0-py2.6.egg/boto/dynamodb/layer1.py", line 164, in _retry_handler
    data)
boto.exception.DynamoDBResponseError: DynamoDBResponseError: 400 Bad Request
{u'Message': u'Expected null', u'__type': u'com.amazon.coral.service#SerializationException'}

I also investigated the API docs here but they were pretty spartan.

I have done a lot of searching and fiddling, and the only thing I have left is to use the PHP API and dive into the code to find where it "formats" the JSON body, but that is a bit of a pain. Please save me from that pain!

Edit Report

2 Answers

For those looking for the answer I have found it. First IMPORTANT NOTE, I am currently unaware of what is going on BUT for the moment, to get a layer1 instance I have had to do the following:

import boto
AWS_ACCESS_KEY=XXXXX
AWS_SECRET_KEY=YYYYY
dynoConn = boto.connect_dynamodb(AWS_ACCESS_KEY, AWS_SECRET_KEY)
dynoConnLayer1 = boto.dynamodb.layer1.Layer1(AWS_ACCESS_KEY, AWS_SECRET_KEY)

Essentially instantiating a layer2 FIRST and THEN a layer 1. Maybe Im doing something stupid but at this point Im just happy to have it working.... I'll sort the details later. THEN...to actually do the atomic update call:

dynoConnLayer1.update_item("influencer_data", 
                    {"HashKeyElement":{"S":"9f08b4f5-d25a-4950-a948-0381c34aed1c"}},
                    {"direct_influence":
                        {"Action":"ADD","Value":{"N":"20"}}
                    }
                );

Note in the example above Dynamo will ADD 20 to what ever the current value is and this operation will be atomic meaning other operations happening at the "same time" will be correctly "scheduled" to happen after the new value has been established as +20 OR before this operation is executed. Either way the desired effect will be accomplished.

Be certain to do this on the instance of the layer1 connection as the layer2 will throw errors given it expects a different set of parameter types.

Thats all there is to it!!!! Just so folks know, I figured this out using the PHP SDK. Takes a very short time to install and set up AND THEN when you do a call, the debug data will actually show you the format of the HTTP request body so you will be able to copy/model your layer1 parameters after the example. Here is the code I used to do the atomic update in PHP:

<?php 
    // Instantiate the class
    $dynamodb = new AmazonDynamoDB();

    $update_response = $dynamodb->update_item(array(
        'Ta


                        
                        
                            answered 2012-04-27T05:06:33.697


            
            
                
                    
                        
                            
                            
                        
                        0
                        
                            
                            
                        
                        
                    
                    
                        
                        There is no high-level function in DynamoDB for atomic counters.  However, you can implement an atomic counter using the conditional write feature.  For example, let's say you a table with an string hash key called like this.

>>> import boto
>>> c = boto.connect_dynamodb()
>>> schema = s.create_schema('id', 's')
>>> counter_table = c.create_table('counter', schema, 5, 5)


You now write an item to that table that includes an attribute called 'n' whose value is zero.

>>> n = 0
>>> item = counter_table.new_item('counter', {'n': n})
>>> item.put()


Now, if I want to update the value of my counter, I would perform a conditional write operation that will bump the value of 'n' to 1 iff it's current value agrees with my idea of it's current value.

>>> n += 1
>>> item['n'] = n
>>> item.put(expected_value={'n': n-1})


This will set the value of 'n' in the item to 1 but only if the current value in the DynamoDB is zero.  If the value was already incremented by someone else, the write would fail and I would then need to increment by local counter and try again.

This is kind of complicated but all of this could be wrapped up in some code to make it much simpler to use.  I did a similar thing for SimpleDB that you can find here:

http://www.elastician.com/2010/02/stupid-boto-tricks-2-reliable-counters.html

I should probably try to update that example to use DynamoDB

                        
                        
                            answered 2012-04-26T17:13:27.650



        
        
            Your Answer