This piece of code is intended for those interested in running python on a cluster with embarrassingly parallel processes;
You can use 'htop' to check all cores are being used.
Let me know of any bugs/improvements in parallelization!
#README:
#cd to folder where you placed mp_hello.py
#then run 'python mp_hello.py &'
#to view output run 'less logs/mp_hello*date*.log'
#
#if that works then comment line 111
#and uncomment line 112 to run large job
Expected output:
--------------------------------START---------------------------------
This node has 4 cores
Perform heavy function in parallel using multiprocessing pool.map
time = 5.69208908081
integers: [30190929843 47298420295 28684640889 ..., 42092960003 91328802780 6992124617]
largest prime factors: [10063643281 42829 250979 ..., 14717 1522146713 739]
Perform heavy function in parallel using alternative Process and queue
........................................time = 5.72581887245
integers: [30190929843 47298420295 28684640889 ..., 42092960003 91328802780 6992124617]
largest prime factors: [10063643281 42829 250979 ..., 14717 1522146713 739]
Perform heavy function in serial mode
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . time = 11.6820590496
integers: [30190929843 47298420295 28684640889 ..., 42092960003 91328802780 6992124617]
largest prime factors: [10063643281 42829 250979 ..., 14717 1522146713 739]
OK
TODO:lots of other parellisation methods in multiprocessing
---------------------------------END----------------------------------
Code references:
1. http://docs.python.org/2/library/multiprocessing.html
2. http://docs.python.org/2/tutorial/
3. http://www.stealthcopter.com/blog/2009/11/python-factors-of-a-number/