Python Redis Subscribe Can Not Get All Datas?

I am using python to get datas from redis and then parser it to kafka. It works well in most situations. But when I use python to simulate build the data to redis, or there was a f

Solution 1:

I assume you are using redis-py.

The documentation refers to listen as older version of the lib... Maybe you should use another method for message reading. For example with a callback

p = r.pubsub()

defmy_handler(message):
    print'MY HANDLER: ', message['data']
    if item['type'] == 'message':
         num += 1
         a.parser(item['data'])
         print num

p.subscribe('bdwaf', my_handler)
# read the subscribe confirmation message
p.get_message()

Edit:

It is possible that your redis server is running out of memory when you publish 20000 messages at once. Try increasing redis memory in redis.conf file

maxmemory 500mb # or greater if needed

It is a memory problem, check out this question for more information on how to handle it.