Thursday, June 3, 2010

Multiprocessing

In Python GIL (Global Interpreter Lock) prevents from running more than one thread at a time in a process. If you use Jython or IronPython this limitation is not there. But with CPython it is there. Use the multiprocess module (from Python version 2.6 afterwards) to overcome this limitation and have truly multiprocessing capabilities. If you encounter an error during execution and if your processes become zombies then you could either close the terminal (which will kill the zombies as well), or more elegantly use kill -9 on the "parent process id (PPID" of those zombies. To find the process ids use "ps -epw" command. You can also kill a process within the top window by typing "k" and then entering the corresponding process id.

multiprocessing module has similar API as in threading module. It works on Windows, Linux and OS/X.
The following code shows how to create and start several processes.

from multiprocessing import Process
import os

def info(title):
    print title
    print 'module name:', __name__
    print 'parent process:', os.getppid()
    print 'process id:', os.getpid()

def f(name):
    info('function f')
    print 'hello', name
    c = 0
    for i in range(1,100000):
        for j in range(1, 100000):
            if i == j:
                c += 1
    pass
    

    
if __name__ == "__main__":
    info('main line')
    p = Process(target=f, args=('bob',))
    q = Process(target=f, args=('sam',))
    r = Process(target=f, args=('dan',))
    s = Process(target=f, args=('david',))
    q.start()
    p.start()
    r.start()
    s.start()
    p.join()
    q.join()
    r.join()
    s.join()
    pass



No comments:

Post a Comment

Continuously monitor GPU usage

 For nvidia GPUs do the follwing: nvidia-smi -l 1