From: "Marc Snir" <[email-masked]>
Date: Mon, 3 Jan 94 22:23:19 EST

Suggestion for changes in 5.5 (Intercommunicators)

5.5.2.
The usual mechanism for binding client to server (e.g. via sockets) is that a
client opens a connection, without specifying the remote port (client), and
starts listening (receiving).  The client opens the connection by specifying
both local and remote ports, and starts sending messages.
This protocol is asymmetric: the server need not know the name/address of the
client, whereas the client provides full information on the server.  Also, the
server operation is nonblocking:  The server may continue serving other clients
while waiting for a new client to join.

This mechanism cannot be supported now with
intercommunicators: a receive can be posted only after the intercommunicator
creation has completed (with a call to MPI_INTERCOMM_FINISH).  This call is
blocking. Thus, if a server is not multithreaded, its binding to a client is
blocking.  Also, we offer a binding mechanism where both sides need know the
identity of the other (MPI_INTERCOMM_START ...FINISH) and a binding mechanism
where neither sides need know the identity of the other
(MPI_INTERCOMM_NAME_START ...FINISH); not a mechanism where client need not
know the name of the server, but server need know the name of the client.

I believe we should offer a nonblocking binding mechanism.  This in support of
current practice, and in order to reduce the chances for deadlock (In an
asymmetric scenario, there can be no deadlock, if clients and servers are
disjoint). I suggest the following changes:

1.  Change the semantics of MPI_INTERCOMM_FINISH to be
nonblocking. Thus, replace MPI_INTERCOMM_FINISH with
MPI_INTERCOMM_TEST(inter_request, flag, newcomm).  Returns flag=true if the
intercommunicator creation has completed, in which case newcomm is a handle to
the new intercommunicator.    MPI_INTERCOMM_START is used, as before, to start
the process.

2.  Allow the "remote_leader" argument in the call MPI_INTERCOMM_START to be
MPI_ANY_SOURCE, even at the local_leader, where it is significant.   Thus, only
one of the two pairing leaders need know the identity of the other (the call is
OK if both leaders provide the identity of the other, and erroneous if neither
do).

3.  For good measure, we can add a blocking call:
MPI_INTERCOMM_MAKE(local_comm, local_leader, peer_comm, remote_leader, tag,
                   intercomm),
which blocks until the intercommunicator is created.

With these changes the protocol for binding to a server is as follows:

The following need to be agreed beforehand:

1.  Who is the leader of the client group.
2.  Who is the leader of the server group.
3.  Which communicator (peer_comm) is used by a client leader to bind to the
server leader.
4.  The client leader need know  "peer_comm" and the rank of the server leader
in peer_comm.
5.  The server leader need only know "peer_comm"

When a client wants to bind with a server, it executes a call to
MPI_INTERCOMM_MAKE.  The client leader supplies in this call the "peer_comm"
and "remote_leader" parameters that allow to communicate with the server leader.
When the call returns, the client can start communicating with the server.

A server that can accept new clients executes a call to MPI_INTERCOMM_START.
The server leader provides the "peer_comm" parameter and passes
"remote_leader = MPI_ANY_SOURCE".
The server then periodically calls MPI_INTERCOMM_TEST to check if it has pending
requests to bind.   If it returns "flag=true" then the server starts
communicating with the new client.

Alternatively, in a multithreaded environment, the server spawns a thread that
blocks on a call to MPI_INTERCOMM_MAKE.  The call at the server leader is with
parameters (...,peer_comm, MPI_ANY_SOURCE).

Note that the functionality proposed here is, essentially, a superset of the
functionality in the draft, so that no power is lost.

Implementation:

MPI_INTERCOMM_START
The leader coordinates a new local context in the local group (this is a
  collective operation in the local group);
The leader posts a nonblocking receive with communicator, source and tag as
  specified by the peer_comm, remote_leader and tag parameters of the call
  (source might be MPI_SOURCE_ANY);
If remote_leader <> MPI_SOURCE_ANY,
  then the leader posts a nonblocking send with
  communicator, destination and tag as
  specified by the peer_comm, remote_leader, and tag parameters; the message
  contains the new context and the local group table.

MPI_INTERCOMM_TEST
The leader tests if the receive posted by _START has been satisfied (this is
  a call to MPI_TEST);
If not, then the leader broadcasts flag=false in the group;
If yes then {
    the leader broadcast flag=true and the received message in the  group;
    if it did not post a send in _START
        then the send is posted now (to the source of the message received);
    the leader waits for send completion
    }.

MPI_INTERCOMM_MAKE
is MPI_INTERCOMM_START, followed by MPI_INTERCOMM_TEST, with MPI_TEST replaced
by MPI_WAIT.

Note:
One should warn that the same communicator and tag should not be used for
communication between the two leaders while an intercommunicator is being
created, unless we want to preserve a separate tag space for intercommunicator
operations (this is true with the old version, as well).

5.5.3

I propose to change the name service constructs for the following reasons:

1.  It is an overkill to force each MPI implementation to have an (anonymous)
name server:  When I run with 8 processes, I may not want an additional name
server process, or have one of the eight processes perturbed by an additional
name serving function.
2.  I may want in a large system multiple name servers, and perhaps
hierarchical name servers.
3.  Name servers, as currently defined, may be difficult to extent to a dynamic
MPI, where processes come and go.  To do so, we should have name servers that
are local to a communication universe (e.g., MPI_COMM_WORLD).  The mechanism
provided to extend a communication universe will, ipso facto, also extend the
group of processes serviced by a name server.   Quite generally, we should want
our standard "communication context" mechanism, namely communicators, to extend
to name servers as well.
4.  Various communicators may correspond to various
communication protocols, various protection mechanisms, etc.  One more reason to
have the name server "local" to a communication context.
5.  It seems that the functions in 5.2.2 can be used to provide the functions of
a name server, as well, thus reducing the number of functions.   Basically, the
name server allows to bind two groups via communication to a third process.
A mechanism to do so, using the previous calls is to allow the leaders of each
group to communicate with a third server process, rather than with each
other.

I suggest the following changes:

An MPI implementation may provide one or more "intercom servers".
The predefined constant MPI_SERVER contains the rank in the group of
MPI_COMM_WORLD of an intercom server for the group, if such is
available; MPI_SERVER = MPI_UNDEFINED, if there is no such server.
Additional intercom servers may be provided.

In a call to MPI_INTERCOMM_MAKE or MPI_INTERCOMM_START, the parameter
"peer_comm" may be the rank of an "intercom server", rather than the rank
of a remote group leader (or MPI_ANY_SOURCE).   If two groups call
MPI_INTERCOMM_MAKE (or MPI_INTERCOMM_START followed by MPI_INTERCOMM_TEST),
the leaders in each group  provide the same "peer_comm", "remote_leader" and
"tag" parameters, and "remote_leader" is the rank of an "intercom server", then
a new intercommunicator will be created that binds at least one of these two
groups (The two groups will be bound together unless there is a race with
another matching call.)

Thus, a call to
MPI_INTERCOMM_NAME_START(comm, name, inter_request)
can be replaced by a call to
MPI_INTERCOMM_START(comm, 0, MPI_COMM_WORLD, MPI_SERVER, name, inter_request)
("name", which was a character string, becomes an integer tag).
One could also  use another local leader, or another peer communicator which is
known to have a name server.

Implementation:
NO CHANGES are needed in the code to MPI_INTERCOMM_MAKE, MPI_INERCOMM_START, and
MPI_INTERCOMM_TEST.
Each server needs to know which communicator it serves.
The server code is busy loop which does the following:

post a (blocking) receive with source = MPI_ANY_SOURCE and tag = MPI_ANY_TAG;
test tag of incoming message (messageA);
if previous message (messageB) with same tag is available in local store
  then {
     exchange messages (send messageA to sender of messageB and send messageB to
     sender of messageA);
     erase messageB from store
     }
  else put messageA in store

Of course, one can replace blocking communication with nonblocking
communication.