Pymongo - Message Length Is Larger Than Server Max Message Size

the line for doc in collection.find({'is_timeline_valid': True}): is giving the Message Length error. How can I get all the collection without the error? I know about the find().li

Solution 1:

Here is a simple paginator that splits the query execution into paginated queries.

from itertools import count

classPaginatedCursor(object):
    def__init__(self, cur, limit=100):
        self.cur = cur
        self.limit = limit
        self.count = cur.count()

    def__iter__(self):
        skipper = count(start=0, step=self.limit)

        for skip in skipper:
            if skip >= self.count:
                breakfor document in self.cur.skip(skip).limit(self.limit):
                yield document

            self.cur.rewind()

...
cur = collection.find({'is_timeline_valid': True})
...
for doc in PaginatedCursor(cur, limit=100):
   ...

Solution 2:

I ran into this problem today, and it turns out it has to do with the size of a particular document within the collection exceeding the max_bson_size limit. When adding documents to the collection, make sure the document size doesn't exceed max_bson_size size.

document_size_limit = client.max_bson_size
assertlen(json.dumps(data)) < document_size_limit

I'm currently investigating why the collection allowed a document larger than max_bson_size in the first place.

Solution 3:

we can add batch_size to find() to reduce the message size.

for doc in collection.find({'is_timeline_valid': True}):

becomes

for doc in collection.find({'is_timeline_valid': True}, batch_size=1):