Subject: Re: MPI-2 thread safety and collectives

From: [email-masked]
Date: Mon, 16 Nov 1998 16:22:59 -0500

The MPI2 standards specifies:  "matching of collective calls on a
communicator, window or file handle is done according to the order in which
the call are issued on each process.  If concurrent threads issue such
calls on the same communicator, window or file handle, it is up to the user
to ensure that the call are correctly ordered, using interthread
communication. "   The text speaks of calls done on the same object
(communicator, file,, or window), not on calls done on distinct objects.
A code of the form

                  Process 0
Process 1
thread 0.0                          thread 0.1
thread 1.0                           thread 1.1
MPI_Barrier(comm1)  MPI_Barrier(comm2)              MPI_Barrier(comm1)
MPI_Barrier(comm2)

is perfectly legal, and should work.
(Suppose that the threads do not execute concurrently: thread 0.0 and 1.1
start.  At this point both threads are blocked and cannot continue.  But
"Blocking MPI calls will block the calling thread only".  Thus, 0.1, and
1.0 can start executing.  At this point, matching calls occurred at both
processes and the collective calls must complete, according to the usual
MPI progress rule.)

Similarly a collective call on a window can be concurrent with a collective
call with a communicator, and the code should work -- the argument is same
as in my simpler example.

 If Romio uses the user provided communicator to synchronize file accesses,
then the implementation is incorrect, since it may lead to illegal code,
where concurrent threads issue collective calls on THE SAME communicator in
the worng order.  Besides, such an implementation violates a frequently
reiterated rule of good MPI design: if one develops a collective library,
then such a library should use private communicators for its internal
communications, to avoid races.  The recommended approach would be to cache
this private communicator with the window object.

Marc Snir
Senior Manager, Scalable Parallel Systems
IBM T J Watson Research Center, PO Box 218, Yorktown Heights 10598
Tel: 914 945 3204 (862 3204) Fax: 914 945 4425 (862 44245)
URL: http://www.research.ibm.com/people/s/snir

......................
Consider scenario S1 below, which illustrates this thread-safety issue as
it affects the ROMIO MPI-2 I/O layered implementation.

ROMIO function MPI_File_open()  calls MPI_Allreduce() using the
communicator
passed by the user into MPI_File_open.    Suppose the user had another
thread
executing an MPI_Allreduce() collective operation on the same communicator.

If a race condition allows the threads to execute in different orders,
we would get an incorrect result:

          Scenario S1
          -----------

     |  Process 0                       |       Process 1
     |  ---------                       |       ---------
     |                                  |
Time |  thread A calls MPI_Allreduce    |       thread C calls
MPI_Allreduce
 |   |  via MPI_File_open               |
 |   |                                  |
 |   |  thread B calls MPI_Allreduce    |       thread D calls
MPI_Allreduce
 |   |                                  |       via MPI_File_open
 V   |                                  |

Under interpretation #1, you could say that the user needs to ensure
that all threads issuing any collective calls on the same communicator
must be properly ordered by the user.  With this interpretation, scenario
S1 would be erroneous and the implementation wouldn't need to deal with
the matching up of the collective calls on multiple threads.

Notice that under both interpretations, a scenario like S2 should
be erroneous because the same collective function is called by
the two threads.

                Scenario S2
                -----------

     |  Process 0                       |       Process 1
     |  ---------                       |       ---------
     |                                  |
Time |  thread A calls MPI_Allreduce    |       thread C calls
MPI_Allreduce
 |   |  for MPI_MAX operation           |       for MPI_SUM operation
 |   |                                  |
 |   |  thread B calls MPI_Allreduce    |       thread D calls
MPI_Allreduce
 |   |  for MPI_SUM operation           |       for MPI_MAX operation
 V   |                                  |

Thanks for any opinions or recollections of forum discussions on this
matter.

Karl Feind                                    E-Mail: [email-masked]
Silicon Graphics                              Phone:  612/683-5673
655F Lone Oak Drive                           Fax:    612/683-5276
Eagan, MN  55121

---

From: [email-masked]
Date: Mon, 16 Nov 1998 16:22:59 -0500

Process 0
Process 1
thread 0.0                          thread 0.1
thread 1.0                           thread 1.1

Marc Snir

......................

          Scenario S1
          -----------

     |  Process 0                       |       Process 1
     |  ---------                       |       ---------
     |                                  |

                Scenario S2
                -----------

     |  Process 0                       |       Process 1
     |  ---------                       |       ---------
     |                                  |