[1] https://mpi4py.readthedocs.io/en/stable/tutorial.html
[2] https://www.csc.fi/documents/200270/224572/basic_MPI.pdf/d9c2a1ec-0f6b-4ccf-a509-c06c214a420d
[3] https://www.microsoft.com/en-us/download/confirmation.aspx?id=100593
[4] https://mpitutorial.com/tutorials/mpi-broadcast-and-collective-communication/
[6] http://web.stanford.edu/class/cme194/cgi-bin/lecture5.pdf
[7] https://info.gwdg.de/wiki/doku.php?id=wiki:hpc:mpi4py - Nice MPI4PY tutorial
[8] https://mpi4py.readthedocs.io/en/stable/overview.html
[9] https://mpi4py.readthedocs.io/en/stable/tutorial.html?highlight=isend#tutorial
- The same program code and instructions are executed by all processes
- Execution code can reside in different nodes
- The way to launch a parallel program is implementation-dependent:
mpirun, mpiexec, srun, aprun, poe, etc.
- All variables and data structures are local to the process
- Processes can exchange data by sending and receiving messages between two processes
- Processing can communicate via sending and receiving messages between several processes
- Synchronization between processes
- MPI runtime assigns each process a rank – ranks start from 0 and extent to N-1
- Processes can perform different tasks and handle different databasing on their rank which is written in its logic
- "Communicator" is an object connecting a group of processes
- Initially, MPI_COMM_WORLD contains all the processes, but users can define their own communicators
Meta information
- Information about the communicator number of processes and rank of the process. For details [2]
Styles of communication
- Point-to-point communication
Messages are sent between two processes. One process sends a message to another process that receives it. Sends and receives in a program should match – one receive per send.
In another case, it leads to undefined behavior
- Collective communication
Involving a number of processes at the same time
- Broadcasting
The task root sends a buffer of data to all other tasks
Communication Modes
• Blocking (vs. non-blocking, non-blocking also known as immediate)
Blocking means that, when the function returns, it has completed the operation it was meant to do.
• Buffered (vs. unbuffered)
The implementation can copy the message data into a local buffer, before streaming it into the communication system.
This buffered communication mode allows the send call to return immediately after completion of the local copy operation, irrespective of the posting of a matching receive call.
Unbuffered implies blocking.
• Synchronous (vs. asynchronous)
It implies synchronisation among processes. The blocking send operation can be implemented by directly streaming the message data from memory into the communication system.
In this case, the send call will block until a corresponding receive call has been posted and copied the data stream into the memory of the receiving task.
• Ready mode
The send call can be started only if a matching receive call already has been posted, such that the overhead for establishing the availability of the receiver may be avoided.
In this ready communication mode, the send operation will return with an error, if a matching receive call has been not posted previously.
The standard "send" operation in MPI is a combination of
- Buffered mode for short messages
- And synchronous mode for long messages.
In MPI terms, Bcast is blocking operations
The border between the two modes is implementation-dependent, portable programs using the standard send should not depend on the properties of one of the modes. Some details are available at [7].
#!/usr/bin/env python3
# launch cmdline: mpiexec -n 4 python basic_example.py
from mpi4py import MPI mpi_rank = MPI.COMM_WORLD.Get_rank() mpi_size = MPI.COMM_WORLD.Get_size() print("rank of this process: ", mpi_rank)print("number of processes in the communicator: ", mpi_size)
#!/usr/bin/env python3
# mpiexec -n 4 python basic_bcast.py from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.Get_rank() if rank == 0: data = {'key1' : [7, 2.72, 2+3j], 'key2' : ( 'abc', 'xyz')}else: data = None if rank == 0: data = comm.bcast(data, root = 0) data = comm.bcast(data, root = 0)else: data = comm.bcast(data, root = 0) print("received data: ", data, " rank: ", rank)
#!/usr/bin/env python3
# mpiexec -n 4 python basic_send.py
from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.Get_rank() if rank == 0: data = {'a': 7, 'b': 3.14} print("sending data: ", data) req = comm.isend(data, dest=1, tag=11) req.wait()elif rank == 1: req = comm.irecv(source=0, tag=11) data = req.wait() print("received data: ", data)
Multiple hosts
/usr/bin/mpiexec.mpich --hosts 10.10.10.10,11.11.11.11 -n 4 python basic_example.py
Relevant link with install prerequisites: https://mpitutorial.com/tutorials/running-an-mpi-cluster-within-a-lan/
Also, it maybe usefull to make chmod 600 * for all files in ~/.ssh/ folder and make chmod 700 ~/.ssh for .ssh fodler by itself.
#!/usr/bin/env python3# launch cmdline: mpiexec -n 4 python basic_example.pyfrom mpi4py import MPI import socket mpi_rank = MPI.COMM_WORLD.Get_rank() mpi_size = MPI.COMM_WORLD.Get_size() comm = MPI.COMM_WORLD print("host:", socket.gethostname(), "rank of this process: ", mpi_rank, "/", mpi_size) if mpi_rank == 0: data = {'a': 7, 'b': 3.14} print("sending data [0 => 1]: ", data) req = comm.isend(data, dest=1, tag=11) req.wait()elif mpi_rank == 1: req = comm.irecv(source=0, tag=11) data = req.wait() print("received data [0 => 1]: ", data)