Python: Why Different Threads Get Their Own Series Of Values From One Generator?

I'm learning multithreading in Python. I want to know how to provide data to multiple threads using generators. Here's what I wrote: import threading data = [i for i in xrange

Solution 1:

You instantiate a new generator in each thread in run function with this:

for i in generator():

each generator call returns a new instance of generator:

>>> data = [i for i in xrange(10)]
>>> a, b = generator(), generator()
>>> id(a), id(b)
(37528032, 37527952)

Here a and b have different ids, and generate identical results even without threading:

>>> list(a), list(b)
([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Note however, that generators aren't thread safe, it is tricky to use them in threaded application. You have to worry about locking, see example, or you'll get ValueError: generator already executing error once in a while. Alternatively you can use Queue.Queue for thread communication.

Solution 2:

Can you just use Python Queue class. I believe that is thread safe. Basically you could just use your generator to fill the Queue and then have each thread pull from it.

#!/usr/bin/python

import Queue
import threading

queueOfNumbers = Queue.Queue()

data = [i for i in xrange(100)]

def generator():
    for i in data:
        yield i

class CountThread(threading.Thread):
    def __init__(self, name, queue):
        threading.Thread.__init__(self)
        self.name = name
        self.queue = queue

    def run(self):
        i = self.queue.get()
        print '%s %s' % (self.name, i)


a = CountThread('a', queueOfNumbers)
b = CountThread('b', queueOfNumbers)
a.start()
b.start()

for num in generator():
    queueOfNumbers.put(num)

queueOfNumbers.join()

http://www.ibm.com/developerworks/aix/library/au-threadingpython/