Multiprocessing in Python

There are numerous ways to multiprocess in python. Pools give a very easy way.

Note: If you do not catch keyboard interrups as shown in the code then you will have to wait until all workers have finished even if you send Ctrl+C to the parent process. There are other methods such as map (blocks until each result is received and orders sequentially), apply and apply_async where you can provide a callback function that actually determines what must be done with the result. If the task items are not iterable then you must use apply or apply_async and handle the input in the callback.

from multiprocessing import Pool
from functools import partial

import time
import sys

def doWork(i, x):
    """
    Does the actual work.
    """
    print i
    time.sleep(2)
    return (x * i)
    pass


def multi():
    """
    Does the actual multiprocessing.
    """
    # Create the pool.
    pool = Pool()

    # We must convert the doWork to a function of one argument.
    # This argument will be filled in by an iterator.
    partialWorker = partial(doWork, x=5)

    tasks = range(100)

    p = pool.map_async(partialWorker, tasks)

    try:
        results = p.get(0xFFFF)
    except KeyboardInterrupt:
        print "Received Ctrl-c"
        sys.exit(1)

    pool.close()
    pool.join()          
    pass


if __name__ == "__main__":
    multi()