Alex Rivera | Logout

python multi-threading slower than serial?

Asked 2012-05-28T18:35:18.617
19

I'm trying to figure out multi-threading programming in python. Here's the simple task with which I want to compare serial and parallel speeds.

import threading
import Queue
import time
import math

def sinFunc(offset, n):
  result = []
  for i in range(n):
    result.append(math.sin(offset + i * i))
  return result

def timeSerial(k, n):
  t1 = time.time()    
  answers = []
  for i in range(k):
    answers.append(sinFunc(i, n))
  t2 = time.time()
  print "Serial time elapsed: %f" % (t2-t1)

class Worker(threading.Thread):

  def __init__(self, queue, name):
    self.__queue = queue
    threading.Thread.__init__(self)
    self.name = name

  def process(self, item):
    offset, n = item
    self.__queue.put(sinFunc(offset, n))
    self.__queue.task_done()
    self.__queue.task_done()

  def run(self):
    while 1:
        item = self.__queue.get()
        if item is None:
            self.__queue.task_done()
            break
        self.process(item)

def timeParallel(k, n, numThreads):
  t1 = time.time()    
  queue = Queue.Queue(0)
  for i in range(k):
    queue.put((i, n))
  for i in range(numThreads):
    queue.put(None)    
  for i in range(numThreads):
    Worker(queue, i).start()
  queue.join()
  t2 = time.time()
  print "Serial time elapsed: %f" % (t2-t1)

if __name__ == '__main__':

  n = 100000
  k = 100
  numThreads = 10

  timeSerial(k, n)
  timeParallel(k, n, numThreads)

#Serial time elapsed: 2.350883
#Serial time elapsed: 2.843030

Can someone explain to me what's going on? I'm used to C++, and a similar version of this using the module sees the speed-up we would expect.

Edit
Report

1 Answer

8

Python has a severe threading problem. Basically, adding threads to a Python application almost always fails to make it faster, and sometimes makes it slower.

This is due to the Global Interpreter Lock, or GIL.

Here's blog post about it that includes a talk on the subject.

One way to bypass this limitation is to use processes instead of threads; this is made easier by the multiprocessing module.

answered 2012-05-28T18:39:52.107

Your Answer