Subject: Routines for communicators etc

From: [email-masked] (David Walker)
Date: Tue, 29 Jun 1993 11:24:43 -0400

Here is what I think should form the core of the groups, contexts, and
communicators proposal. I believe that anything additional needs to be
justified with example codes. In fact, I think we should have example
codes illustrating all of these routines.

My basic idea of a communicator is that it provides for communication
between a local group and a remote group. A send context is used when
sending messages, and a receive context is used when receiving messages.
Thus, contexts provide unidirectional communication streams.
The send context for one of the 2 groups must be the receive context of the 
other group, and vice versa. Likewise, given 2 groups, A and B, in group
A, the local  and remote groups refer to A and B, respectively. In group
B they refer to groups B and A, respectively.

For intragroup communication, the local and remote groups are identical.
In general, the send and receive contexts will also be identical, but this
is not necessarily so.

Unless otherwise stated it is assumed that groups are undecorated.

Predefined Communicators
------------------------

MPI_COMM_ALL.   90% of users will probably only ever use this communicator.
		Thus, we don't need the mpi_init_group routine.

Group Inquiry Routines
----------------------
MPI_GROUP_SIZE (group, size)
IN  group (handle)
OUT size  (integer)

MPI_GROUP_RANK (group, rank)
IN  group (handle)
OUT rank  (integer)

Support for Transmission of Groups
----------------------------------
MPI_GROUP_FLATTEN (group, max_length, buffer, actual_length)
IN  group (handle)
IN  max_length (integer)
OUT buffer (integer array)
OUT actual_length (integer)

MPI_GROUP_UNFLATTEN (max_length, buffer, group)
IN  max_length (integer)
IN  buffer (integer array)
OUT group (handle)

Local Group Constructors
------------------------
MPI_GROUP_ENUMERATE (group, n, ranks, new_group)   (New name. Used to be
IN  group (handle)                                  MPI_LOCAL_SUBGROUP)
IN  n (integer)
IN  ranks (integer array)
OUT new_group (handle)

Collective Group Constructors
-----------------------------
MPI_GROUP_PARTITION (group, key, index, new_group)
IN  group (handle)
IN  key (integer)           (This routine replaces MPI_COLL_SUBGROUP and
IN  index (integer)          MPI_COLL_GROUP_PERMUTE)
OUT new_group (handle)

Local Context Operations
------------------------
MPI_CONTEXT_FREE (n, contexts)
IN  n (integer)
IN  contexts (integer array)

Collective Context Operations
-----------------------------
MPI_CONTEXT_ALLOC (group, n, contexts)     (Uses group instead of comm)
IN  group (handle)
IN  n (integer)
OUT contexts (integer array)

Communicator Inquiry Routines
-----------------------------
MPI_COMM_LOCAL_GROUP (comm, group)    (Returns local group of communicator)
IN  comm (handle)
OUT group (handle)

MPI_COMM_REMOTE_GROUP (comm, group)   (Returns remote group of communicator)
IN  comm (handle)
OUT group (handle)

MPI_COMM_SEND_CONTEXT (comm, context) (Returns send context of communicator)
IN  comm (handle)
OUT context (integer)

MPI_COMM_RECV_CONTEXT (comm, context) (Returns receive context of communicator)
IN  comm (handle)
OUT context (integer)

Local Communicator Constructors
-------------------------------
MPI_COMM_BIND (local_group, remote_group, send_context, recv_context, comm)
IN  local_group  (handle)
IN  remote_group (handle)      (This differs from original routine)
IN  send_context (integer)
IN  recv_context (integer)
OUT comm (handle)

MPI_COMM_MERGE (local_comm, remote_comm, comm)
IN  local_comm  (handle)
IN  remote_comm (handle)
OUT comm    (handle)

   mpi_comm_merge (local_comm, remote_comm, comm) is equivalent to:
      mpi_comm_local_group  (local_comm,  local_group)
      mpi_comm_remote_group (remote_comm, remote_group)
      mpi_comm_send_context (local_comm,  send_context)
      mpi_comm_recv_context (remote_comm, recv_context)
      mpi_comm_bind (local_group,remote_group,send_context,recv_context,comm)

MPI_COMM_DUP (comm, new_context, new_comm)
IN  comm (handle)
IN  new_context (integer)
OUT new_comm (handle)

MPI_COMM_UNBIND (comm)
IN  comm (handle)

Collective Communicator Constructors 
------------------------------------
MPI_COMM_MAKE (group, comm)       (This differs from original)
IN  group (handle)
OUT comm (handle)

   mpi_comm_make (group, comm) is equivalent to:
      mpi_context_alloc (group, 1, context)
      mpi_comm_bind (group, group, context, context, comm)

Publishing and Subscribing to Communicators
-------------------------------------------
MPI_COMM_PUBLISH (comm, label, persistence)
IN  comm (handle)
IN  label (character string)
IN  persistence (integer)

MPI_COMM_SUBSCRIBE (group, label, comm)
IN  group (handle)
IN  label (character string)
OUT comm (handle)

MPI_UNPUBLISH (label)
IN  label (character string)

---

From: James Cownie <[email-masked]>
Date: Tue, 29 Jun 1993 17:15:07 +0100

David Walker has stung me into putting out my ramblings on this
subject too, so here they are.

Before I start on the detail, some things I hope are not too
contentious.

What are Contexts for ?
=======================
The selling point of contexts is that they allow the development of
modular software by insulating library routines from the state of the
communications system when they are called.

I think this is the SOLE purpose of contexts.

Issues with the current proposal
================================
1) It doesn't supply the security we want.
   Users are allowed to arbitrarily bind context numbers which they
   invent to communicators in an entirely anarchic manner. There is
   therefore no possible security guarantee.

2) Because the user can locally construct communicators and bind
   contexts to them the receiver of a message where the receive has a
   wild-carded from field (given widespread current practice which
   doesn't support selection on from, this is likely to be a large
   proportion of receives !) has to perform a back translation from an
   absolute process identifier into a group rank. Unfortunately this
   is non-trivial to implement, and has to be done even if the message
   is really an inter-group communication, since there is no easy way
   (either at sender or receiver) of knowing this fact.

3) The intra group communicators have this funny property of being
   uni-directional, when inter group ones are not. This is most
   peculiar. (It also seems unnecessary. If the range of valid
   contexts really is large, then it should be possible to find one
   which is free at both ends. This may be expensive, but then the
   whole publish/subscribe mechanism already is !)

4) I have yet to see the example which implements (any of) the
   collective communications with the properties we desire that
	a) The communicator passed in need not be quiescent
	b) The operation is insulated from the non-quiescence
	c) There is low cost in operating on different communicators
 
   All of the library examples shown to date assume that the library
   can be statically bound to the group of processes on which it is to
   operate.  This is certainly not the case for the collective comms
   routines.  (Note that you need a context per group for the
   collective routines, otherwise overlapping groups will fail. One
   context for the library is insufficient).

Where do the problems come from ?
=================================

1) The insecurity arise from the ability of users to invent contexts
   and use them with no system intervention. This is like being
   allowed to access physical disk blocks while a file system is
   running. 

2) The need to back translate node ids comes from the lack of
   negotiation in setting up communicators. If all the members of a
   communicator understand it the same, then there's no need for a
   translation. We hope and expect that this will be the case for
   communicators used for inter-group communication. (I think many
   many user's will be very confused if this is not the case, just
   imagine the confusion from the debugging messages you can generate
   with crossed over groups, e.g.

	Process 0 		Process 1
	group {0, 1}            group {1, 0}
	       bind the same context
	send to 1		receive from 1
	receive from 1		send to 1	
	
    etc.)  

   The process names can no longer be guaranteed to be anything more
   than totally local.
	
   The perverse thing here is that actually we could assert stronger
   rules about the creation of communicators which ensured that the
   same context was only ever bound to the same group, which should be
   sufficient for the common (inter-group) cases, and then relax this
   somewhat for the intra-group cases, which have to undergo
   negotiation via publish/subscribe anyway. (Of course we would lose
   the ability to flatten a group and distribute it through a shared
   file...) 

   Anyhow I really dislike having to pay an extra cost everywhere for
   something I may never use. 

Please Please Please let's find a way to avoid the back translation on
   the common cases.

-- Jim
James Cownie 
Meiko Limited			Meiko Inc.
650 Aztec West			Reservoir Place
Bristol BS12 4SD		1601 Trapelo Road
England				Waltham
				MA 02154

Phone : +44 454 616171		+1 617 890 7676
FAX   : +44 454 618188		+1 617 890 5042
E-Mail: [email-masked]   or    [email-masked]

---

From: James Cownie <[email-masked]>
Date: Tue, 29 Jun 1993 17:22:05 +0100

> Collective Context Operations

I'm afraid it may need a communicator rather than a context, since
otherwise how can it communicate to reserve the contexts ? It also
probably needs the context in the communicator to be
quiescent. (Actually this routine is just as hard as all the other
collective routines to implement if you can't assume quiescence of the
context. Let's see how we're supposed to do it (Lyndon ? Tony ? Mark
?))

-- Jim
James Cownie 
Meiko Limited			Meiko Inc.