From: L J Clarke <[email-masked]>
Date: Tue, 20 Apr 93 18:23:45 BST

Subject: mpi-context; intercommunication etc (long)

Dear mpi-context colleagues

I  previously  wrote  regarding  context  management  and  binding  of
contexts for intracommunication in the  letter 
[Subject:  mpi-context: context management and  group binding (long)]  
and  sent  out a  short correction in the letter 
[Subject: mpi-context: CORRECTION to previous message] 
to which I draw your attention.

In this letter I wish to briefly revisit and recap the above subjects,
then move on to  briefly discuss and  make  a  concrete suggestion for
intercommunication.   This is a long letter. Probably best to print it
and read over a coffee.

I really must clarify  the  nature of the context which I am assuming.
In  this letter contexts are assumed to be global in the sense that if
a process P  creates a context C, it can send C to another  process Q,
and Q can both send and  receive  messages of context C.  This is  the
model  adopted  by Zipcode, which  I view as the  exemplar of existing
practice regarding message context.

Regarding intracommunication I hope  to slightly simplify the  content
of my suggestion compared to the letters referred to above.  Regarding
intercommunication the particular suggestion I  make is  motivated  by
conformity  with intracommunication both in the point-to-point  syntax
class, and in the content of the message envelope.

			o--------------------o

1. Communicator and Communication
=================================

Communicator   objects   provide    point-to-point    and   collective
communication in MPI.  A communicator object is a binding of a message
context   and   one  or   more  process  worlds.   Two  subclasses  of
communicator   object   are  defined  below,   intracommunicator   and
intercommunicator.   Communicator  objects are identified  by  process
local object identifiers.

1.1 Construction, Destruction and Information
---------------------------------------------

MPI  provides subclass specific  communicator  constructors  described
below. MPI provides a subclass generic communicator object  destructor
procedure.

     mpi_delete_communicator(id)

id           is identifier of a communicator

purpose      deletes the communicator object identified by id
             See Note 1) under intracommunicator construction
             and Note 1) under intercommunicator construction

Notes:

1) This procedure could be replaced with MPI_FREE if we wish to fit in
with the manipulation of communication handles and buffer descriptor
handles described in the point-to-point chapter.

MPI provides a subclass generic procedure which returns the context
identifier of a communicator object.

context = mpi_communicator_context(id)

context   is the context bound to the communicator
id        is the identifier of a communicator
          See Note 1) under intracommunicator construction
          and Note 1) under intercommunicator construction

purpose   informs the caller of the context bound to a communicator

1.2 Discussion
--------------

2. Intracommunicator and Intracommunication
===========================================

Intracommunicator objects provide point-to-point communication between
processes of the same process world in MPI.  Intracommunicator objects
also provide collective communication in MPI.

2.1 Construction and Information
--------------------------------

MPI provides a subclass intracommunicator constructor.

id = mpi_create_intracommunicator(context, world)

id           is identifier of created communicator
context      is message context for communications
world        is process world of receiver and sender in both send and recv

purpose      creates an intracommunicator object

Notes:

1) The context of an intracommunicator is either an actual context  or
the null  context (MPI_NULL). If the context is an actual context then
the call does not  synchronise processes in  the process world of  the
intracommunicator.  If  the context is the null  context then the call
synchronises  the process world of  the  communicator  and  creates  a
context for the communicator. In this case the context is deleted when
the communicator is  itself deleted  calling  mpi_delete_communicator,
and that call  will synchronise the  process world.  In this  case the
information procedure mpi_communicator_context will return MPI_NULL to
the caller --- the  caller is  not  allowed to  have knowledge  of the
context created.

2) The  process  world  of  an intracommunicator  object is either  an
actual process group or the null group (MPI_NULL). If the world is  an
actual  process  group then  the world  is  understood to contain  all
processes  composing the process  group  and  the  communicator object
identifies processes in  a relative  sense, i.e. as a  rank within the
process group.   If the world is the  null  group  then  the  world is
dunerstood to  contain  all  processes  composing the program  and the
communicator  object identifies  processes in an absolute  sense, i.e.
as a process identifier.

MPI  provides  a subclass  information  procedure  which  returns  the
identifier of the world of the intracommunicator.

world = mpi_intracommunicator_world(id)

world        is process world of the communicator
id           is identifier of created communicator

purpose      returns the world identifier of the intracommunicator, 
             either an actual group identifier or the null group
             identifier (MPI_NULL)

2.2 Point-to-point
------------------

I deal with generic "send" and "recv" seperately, and  can  ignore the
multiple flavours thereof.

send(id, process, label, ...)

id           is identifier of intracommunicator object
process      is identifier of receiver in world of object
label        is message tag in context of object        

recv(id, process, label, ...)

id           is identifier of intracommunicator object,
                and cannot be wildcard
process      is identifier of sender in world of object, 
                and can be wildcard
label        is message tag in context of object, 
                and can be wildcard

Notes:

1) The caller  must be  in the  world of  the intracommunicator,  i.e.
either  it is the  null process  group  or an  actual process group of
which the caller is a member.

2.3 Collective
--------------

I  deal  with a  generic collective "operation",  and  can ignore  the
multiple flavours thereof.

operation(id, ...)

id           is identifier of intracommunicator object

Notes:

1) The intracommunicator must have a world which  is an actual process
group of which the caller is a member.

2.4 Envelope
------------

The message envelope for intracommunication consists of:

* sender identifier within process world of communicator (pid or rank)
* receiver routing (implementation defined)
* message context of communicator
* message tag
* message length   (implementation defined)

The sender and  reciever must  bind  the  context to  the same process
world in an intracommunicator, thus the world is determinable.

2.5 Discussion
--------------

The facilities for intracommunication, coupled with the context model,
provide a  convenient and powerful interface  for communications which
are  closed  within  the   scope  of  a  group  and   for  the  serial
client-server model.

The ability to create an  intracommunicator without synchronisation of
processes  simplifies the construction  of  libraries  in highly  MIMD
programs, and  can be  used  to  advantage  in  conjunction  with  the
association and location facilities described below.

3. Association, Dissociation, Location, Passivation and Activation
==================================================================

3.1 Association, Dissociation and Location 
------------------------------------------

These facilities allow the user to bind names to process, group, and
context objects.

     mpi_associate(name, id)

name     is a string which is the name bound to the given object 
id       is the object identifier (process, group or context)

prupose  associates name with object identified by id

id = mpi_locate(name, wait)

id       is the object identifier (process, group or context)
name     is a string which is the name bound to the given object
wait     is a boolean value determining whether the caller waits for
         the name to become associated with an object of given class

purpose  creates a copy of the object associated with name

     mpi_dissociate(id)

id       is the object identifier (process, group or context)

purpose  removes the association of name with object id, and can only
         be performed by the process which previously associated name.

Notes:

1) These facilities are a name service. This could be implemented by a
name server process  which can run on  a host or login  node, and need
not consume expensive numerical computation resources.

3.2 Passivation and Activation 
------------------------------

These  facilities  allow  the  user  to  transmit a process, group and
context objects.   Passivation and  activation  produce  a  "portable"
description  of the object in  a  memory buffer  (conventionally these
operations produce  a description in a file, but a  memory  buffer  is
more convenient for transmission in a message :-).

     mpi_passivate(id, buf, len)

id       is the object identifier (process, group or context)
buf      is an array of character
len      is the length of the array buf

purpose  writes a portable description of object identified by id in
         the memory buffer buf

id = mpi_activate(buf, len)

id       is the object identifier (process, group or context)
buf      is an array of character
len      is the length of the array buf

purpose  reads a portable description of an object and creates a copy
         of the object

Notes:

1) The detailed type  of the  memory buffer is not of great importance
provided  that  we define that type.  I have used character above,  we
could choose integer, for example.

3.3 Discussion
--------------

I  have assumed  that  MPI  can  distinguish the class  of  the object
(process, group  or  context)  given the object  identifier.   If this
cannot be the case  then we can describe a different set of procedures
for each class or we can add a class argument to the above procedures.

The  name association and location  service is the most manageable way
of  describing  which   groups   communicate  with  one  another.  The
passivation activation facilities are potentially a building block  in
the implementation of the name association and location service.

Deletion  of objects created  by  activation or  location  should only
delete the process local copy of the object.  It should not delete the
original copy. 

When location and activation "create" an object and the object already
exists within the calling process, a  new object should not be created
and the id of the existing object should be returned.  This means that
such  object  have  multiple  references,  so  we  should  define  the
destructors in  terms of deleting  references to  objects, leaving the
implementation to delete the object when there are zero references.

4. Intercommunicator and Intercommunication
===========================================

Intercommunicator objects provide point-to-point communication between
processes  of  different  process  worlds  in  MPI.  Intercommunicator
objects do not provide collective communication in MPI (yet :-).

4.1 Construction
----------------

id = mpi_create_intercommunicator(context, local_world, remote_world)

id           is identifier of created communicator
context      is message context for communications
local_world  is process world of sender in send and receiver in recv
remote_world is process world of receiver in send and sender in recv

purpose      creates an intercommunicator object

Notes:

1)  The  context  can  be  an  actual  context  or  the  null  context
(MPI_NULL).  If  the context is an actual  context then the  call does
not  synchronise  processes  within  the  two  process  worlds of  the
communicator.  If  the  context  is the  null  context then  the  call
synchronises the two process worlds of  the communicator and creates a
context for the communicator. In this case the context is deleted when
the communicator  is  itself deleted calling  mpi_delete_communicator,
and that call  will synchronise the  process world.   In this case the
information procedure mpi_communicator_context will return MPI_NULL to
the caller ---  the caller is not  allowed to have  knowledge  of  the
context created.

2)  Each process world of  an  intercommunicator object  is  either an
actual process  group or the null group (MPI_NULL). If the world is an
actual process group  then  the world  is  understood to  contain  all
processes  composing  the  process  group and  the communicator object
identifies processes in that world in a relative sense, i.e. as a rank
within the process group.  If  the world is  the null  group  then the
world is understood to contain all processes composing the program and
the communicator  object  identifies  processes  in  that  world in an
absolute sense,  i.e.   as a  global process identifier.  

MPI  provides  subclass  information  procedures  which  return  the
identifier of the local_world and remote_world of the intercommunicator.

world = mpi_intercommunicator_local_world(id)

world        is local process world of the communicator
id           is identifier of created communicator

purpose      returns the local world identifier of the intercommunicator, 
             either an actual group identifier or the null group
             identifier (MPI_NULL)

world = mpi_intercommunicator_remote_world(id)

world        is remote process world of the communicator
id           is identifier of created communicator

purpose      returns the remote world identifier of the intercommunicator, 
             either an actual group identifier or the null group
             identifier (MPI_NULL)

4.2 Point-to-point
------------------

I deal with generic "send" and "recv" seperately, and  can  ignore the
multiple flavours thereof.

send(id, process, label, ...)

id           is identifier of intracommunicator object
process      is identifier of receiver in remote_world of object
label        is message tag in context of object        

recv(id, process, label, ...)

id           is identifier of intracommunicator object,
                and cannot be wildcard
process      is identifier of sender in remote_world of object, 
                and can be a wildcard
label        is message tag in context of object, 
                and can be a wildcard

1)  The  caller must be  in  the local_world of the intracommunicator,
i.e.  either it is the null process group or an  actual process  group
of which the caller is a member.

4.3 Envelope
------------

The message envelope for intercommunication consists of:

* sender identifier within process world of communicator (pid or rank)
* receiver routing (implementation defined)
* message context of communicator
* message tag
* message length   (implementation defined)

The  sender and reciever  must bind  the context to  the  same process
worlds  in  an  intercommunicator, thus both the  local_world and  the
remote_world are determinable.

This is identical to the envelope of intracommunication.

4.4 Discussion
--------------

The facilities for intercommunication, coupled with the context model,
and the name service, provide a convenient interface for  the parallel
client-server   model  and   parallel  modular-application   software,
provided    that   the   WAIT_ANY()   facilities   of   point-to-point
communication are fair.

The ability  to create an intercommunicator without synchronisation of
processes  simplifies  the   programming  of  parallel   client-server
software, and avoids a dependency graph problem when writing  parallel
modular-application software in which the module graph contains loops.

5. Discussion
=============

I find  it  a  wee  bit  amusing that an  intercommunicator  in  which
local_world  and  remote_world  are  the same  is  no different to  an
intracommunicator.  This  suggests to me that either  (a) there should
only  be  an   intercommunicator  class  or  (b)   we  think  of   the
intracommunicator   class   as  simply   syntactic  sugar  around  the
intercommunicator class. 

The  communicator  object  class  names   are  rather  long.   Perhaps
programmers would prefer shorter names  in programs. We could take the
approach  of  deriving  names  from  the  list  of  objects  which   a
communicator binds, for example: "intracommunicator"  becomes "CW"  as
it is a binding of a context and  a world; "intercommunicator" becomes
"CWW" as it is a binding of a context  and  a world and another world.
On the other hand we  could take  collections of letters from the long
names,  for  example: "intracommunicator"  becomes  "RACO"  or  "ACO";
"intercommunicator" becomes "ERCO" or "ECO".

			o--------------------o

Comments, questions, flames, please!

Best Wishes
Lyndon 

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/