Pagination In Amazon Dynamodb Using Boto

How do I paginate my results from DynamoDB using the Boto python library? From the Boto API documentation, I can't figure out if it even has support for pagination, although the Dy

Solution 1:

Boto does have support for "pagination" like behavior using a combination of "ExclusiveStartKey" and "Limit". For example, to paginate Scan.

Here is an example that should parse a whole table by chunks of 10

esk = NonewhileTrue:
    # load this batch
    scan_generator = MyTable.scan(max_results=10, exclusive_start_key=esk)

    # do something usefullfor item in scan_generator:
        pass# do something usefull# are we done yet ?else:
        break;

    # Load the last keys
    esk = scan_generator.kwargs['exclusive_start_key'].values()

EDIT:

As pointed out by @garnaat, it is possible that I misunderstood your actual goal. The above suggestion allows you to provide pagination like SO does for questions for example. No more than 15 per pages.

If you just need a way to load the whole result set produced by a given Scan, Boto is a great library and already abstracts this for you with no need for black magic like in my answer. In this case, you should follow what he (@garnaat) advises. Btw, he is the author of Boto and, as such, a great reference for Boto related questions :)

Solution 2:

Perhaps I'm misunderstanding the question but I think you are making it more difficult than it needs to be. If you are using the layer2 DynamoDB interface in boto (the default) it handles the pagination for you.

So, if you want to do a query operation, you simply do this:

import boto

c = boto.connect_dynamodb()
t = c.get_table('mytable')
for item in t.query(hash_key='foo'):
    print item

This will automatically handle the pagination of results from DynamoDB. The same would also work for a scan request.

Solution 3:

there is a good chance you want something like this:

qms = tms.query(hash_key=415772421368583351, max_results=2, exclusive_start_key=None)
for i in qms:
    print i
lek = qms.last_evaluated_key
qms = tms.query(hash_key=415772421368583351, max_results=2, exclusive_start_key=lek)
for i in qms:
  print i

of course this is a silly example for demonstration. The key here is to use last_evaluated_key not the exclusive_start_key