Subject: Comments on inter-communicators

From: "Nathan E. Doss" <[email-masked]>
Date: Wed, 5 Jan 1994 16:08:59 -0600 (CST)

>Date: Mon, 3 Jan 94 22:23:19 EST
>From: "Marc Snir" <[email-masked]>
>To: [email-masked]
>Status: RO
>
>Suggestion for changes in 5.5 (Intercommunicators)
>
>5.5.2.
>
>[deleted]
>
>I suggest the following changes:
>
>1.  Change the semantics of MPI_INTERCOMM_FINISH to be
>nonblocking. Thus, replace MPI_INTERCOMM_FINISH with
>MPI_INTERCOMM_TEST(inter_request, flag, newcomm).  Returns flag=true if the
>intercommunicator creation has completed, in which case newcomm is a handle to
>the new intercommunicator.    MPI_INTERCOMM_START is used, as before, to start
>the process.

 o When issued at the group leaders, MPI_INTERCOMM_START behaves similarly to 
   an MPI_ISEND call performed on the peer communicator.  MPI_INTERCOMM_FINISH 
   then behaves like MPI_RECV.  MPI_INTERCOMM_TEST parallels the MPI_TEST 
   function.  I would be in favor of keeping MPI_INTERCOMM_FINISH just as
   MPI_RECV was kept when MPI_TEST was added.

 o If the communicators are disjoint, then deadlock should not occur when
   using MPI_INTERCOMM_FINISH or MPI_INTERCOMM_TEST.  If they are not disjoint,
   then every process in the intersection must call the two MPI_INTERCOMM_START
   calls first, followed by the two MPI_INTERCOMM_FINISH functions (same 
   order). Since the MPI_INTERCOMM_TEST function implies a synchronization 
   within the local communicator, the following code would be illegal in the 
   intersection processes since the MPI_INTERCOMM_TEST call to complete first
   may not be the same for all processes in the intersection.

      MPI_INTERCOMM_START (for comm A);
      MPI_INTERCOMM_START (for comm B);

      status_A = status_B = false;
      while ( !status_A || !status_B ) {
        MPI_INTERCOMM_TEST  (for comm A);
        MPI_INTERCOMM_TEST  (for comm B);
      }

   A correct calling order would have to guarantee that the MPI_INTERCOMM_TEST
   calls finish in the same order in all intersecting processes.

 o MPI_INTERCOMM_TEST must always block when called by non-leaders
   since they will not know when the MPI_INTERCOMM_TEST called by the leader
   has completed.   This can be avoided by having the leader send a message
   to the non-leaders notifying them that the remote communicator has
   made contact.
  
>2.  Allow the "remote_leader" argument in the call MPI_INTERCOMM_START to be
>MPI_ANY_SOURCE, even at the local_leader, where it is significant. Thus, only
>one of the two pairing leaders need know the identity of the other (the call 
>is OK if both leaders provide the identity of the other, and erroneous if 
>neither do).

  o Assume we want to create an inter-communicator between a server
    comm A (MPI_ANY_SOURCE is given to MPI_INTERCOMM_START) and a 
    client comm B. If there is an intersection between A and B, then 
    the two starts must be called first followed by the finish on A,
    then the finish on B.  The finish on the server comm must always 
    be called before the finish on the client comm or deadlock may 
    (will) occur.

  o A new function should possibly be used to set up a server
    connection instead of allowing MPI_INTERCOMM_START to accept
    MPI_ANY_SOURCE since accepting MPI_ANY_SOURCE changes the
    communication patterns in both MPI_INTERCOMM_START and
    MPI_INTERCOMM_FINISH. 
    
  o If MPI_ANY_SOURCE is given, the most efficient implementation 
    may be for MPI_INTERCOMM_START to do nothing.  Instead of allowing 
    MPI_INTERCOMM_START to accept MPI_ANY_SOURCE, one alternative
    would be for there to be an MPI_INTERCOMM_SERVER_MAKE.  The client
    process would call MPI_INTERCOMM_START and MPI_INTERCOMM_FINISH
    as normal while the server process would simply call
    MPI_INTERCOMM_SERVER_MAKE.  To avoid deadlock in a process in the
    intersection the client MPI_INTERCOMM_START must be called first
    followed by the server's MPI_INTERCOMM_SERVER_MAKE and then the
    client's MPI_INTERCOMM_FINISH.

>3.  For good measure, we can add a blocking call:
>MPI_INTERCOMM_MAKE(local_comm, local_leader, peer_comm, remote_leader, tag,
>                   intercomm),
>which blocks until the intercommunicator is created.

  o This call will deadlock if there are processes in the intersection 
    of the two communicators to be connected.  In the intersecting 
    processes, the first MPI_INTERCOMM_MAKE cannot complete until the second
    MPI_INTERCOMM_MAKE is called.  The second MPI_INTERCOMM_MAKE is not
    called until the first has completed.

>[deleted]

-----------------------------------------------------------------------------
>Date: Tue, 4 Jan 94 10:28:12 MST
>From: [email-masked] (Tom Henderson)
>Message-Id: <[email-masked]>
>To: [email-masked]
>Subject: Re:  Suggestion for changes in 5.5 (Intercommunicators)
>Status: RO
>
>Marc,
>
>Your suggestions for changing section 5.5.2 (MPI_INTERCOMM_TEST(), etc.) make 
>sense to me.  I assume that a "high quality" implementation will be able to 
>detect the erroneous case where "remote_leader = MPI_ANY_SOURCE" for BOTH 
>leaders in calls to MPI_INTERCOMM_START() or MPI_INTERCOMM_MAKE() and return 
>an appropriate error status to the user...  Is there any reason why this would
>not be possible? 

 o The implementation probably could not return an error when two 
   MPI_INTERCOM_START's had used "remote_leader = MPI_ANY_SOURCE" since it 
   won't know if the calls are really meant for each other.
 
>[deleted]

-- 
Nathan Doss          [email-masked]

---

From: [email-masked] (Tom Henderson)
Date: Wed, 5 Jan 94 15:37:30 MST

> >...  I assume that a "high quality" implementation will be able to 

Nathan,

I think you're right here.  Unfortunately, the typical MPI response to this kind of 
situation has been "...the behavior of an erroneous program is undefined...".  I 
think we're giving the user more rope here (as usual).  

Tom

---

From: [email-masked] (Marc Snir)
Date: Thu, 06 Jan 94 10:52:39 EST

Date: Thu, 06 Jan 94 10:52:48 -0500

If two servers decide to start listening for requests and nobody
ever ask for their services, then they wiat forever (unless one has
a timeout).  This is the equivalent of two groups calling
MPI_INTERCOMM_START with a dontcare peer leader.