Testing Python multiprocessing

This piece of code is intended for those interested in running python on a cluster with embarrassingly parallel processes;

You can use 'htop' to check all cores are being used.

Let me know of any bugs/improvements in parallelization!

#README:

#cd to folder where you placed mp_hello.py

#then run 'python mp_hello.py &'

#to view output run 'less logs/mp_hello*date*.log'

#if that works then comment line 111

#and uncomment line 112 to run large job

Expected output:

--------------------------------START---------------------------------

This node has 4 cores

Perform heavy function in parallel using multiprocessing pool.map

time = 5.69208908081

integers: [30190929843 47298420295 28684640889 ..., 42092960003 91328802780 6992124617]

largest prime factors: [10063643281 42829 250979 ..., 14717 1522146713 739]

Perform heavy function in parallel using alternative Process and queue

........................................time = 5.72581887245

integers: [30190929843 47298420295 28684640889 ..., 42092960003 91328802780 6992124617]

largest prime factors: [10063643281 42829 250979 ..., 14717 1522146713 739]

Perform heavy function in serial mode

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . time = 11.6820590496

integers: [30190929843 47298420295 28684640889 ..., 42092960003 91328802780 6992124617]

largest prime factors: [10063643281 42829 250979 ..., 14717 1522146713 739]

TODO:lots of other parellisation methods in multiprocessing

---------------------------------END----------------------------------

Code references:

1. http://docs.python.org/2/library/multiprocessing.html

Download