Subject: mpi-context: context management and group binding (long)

From: L J Clarke <[email-masked]>
Date: Fri, 9 Apr 93 20:32:21 BST

Dear MPI Colleagues

I now discuss context management and group binding. As promised I omit
inter(group,context)communication  for  the  present.  This letter  is
further to letters of today and yesterday to mpi-comm and mpi-context.

Some of the people I talked with  about contexts at the recent meeting
wanted to be able  to  generate some contexts values  themselves, i.e.
not by calling context constructor procedures. This  is accomadated in
the recommendations of this letter.

In  my letter to mpi-comm  "Subject: various (long)"  I suggested that
the  question  of   secure/insecure   point-to-point  and   collective
communications  could  be described  as a property of  a context, with
some advantage. In this letter I will incorporate this feature.

I will also be discussing communicator objects as  in  Proposal X, but
with sensible names.  Tony Skjellum has  made the valued suggestion to
me,  privately,  that it  is better  to  attributise the  communicator
object with the secure/insecure  stuff,  rather  than  the context.  I
shall  adopt  this suggestion  in  this  letter  and  attributise  the
communicator object rather than the context.

			o--------------------o

Message Contexts
----------------

In this  proposal  a (message) context identifier  is (like a  message
tag) just an integer which is used in message selection and  (unlike a
message tag) may not be wildcard.

The interval of context identifiers (1, ...,  MPI_NUMUSR_CONTEXTS) are
reserved for  the  MPI user to manage as she sees  fit.  Use  of these
contexts  allows the user to  write programs  which do not make use of
the provided context creation and deletion facilities.  How big should
MPI_NUMUSR_CONTEXTS be? Say  1, 2, 4, 8, 16,  32, 64, 128, ...   Steve
and Tom, and friends, can you advise?

The  MPI system provides a procedure which  creates  a  unique context
outside  the  interval  of reserved user  context identifiers,  and  a
procedure  which deletes a created context  (it does  not delete  user
reserved contexts).  For example:

            context = mpi_create_context()
            mpi_delete_context(context)

There  may  be  advantage  in defining  the context  create and delete
functions such that they create and delete more than one context at  a
time, in order to amortise creation/deletion overhead. 

Please note that  these  context generation calls are made by a single
process and are  asynchronous.  They  can be implemented  as a process
local  operation  by  attatching  the  global  process identifier to a
process local context  allocator, at the  expense  of needing a lot of
bits in  the  context.   They can  also be implemented  via access  to
shared data (or a reactive server) in which case the bit length of the
context can be made smaller.  [I view this as an implementation detail
which we  should not dwell on in MPI, and should be the freedom of the
implementor  to  choose any  formally  correct method  which hopefully
optimises execution on the target platform.]

The  user program may make  use of the user reserved  contexts. ClassB
libraries (encapsulated  objects) are  expected  to use system created
contexts. These can  be created  as above or through  the Communicator
object constructors described below.

Communicator Objects
--------------------

The context  acquired by the  user in either of  the above ways is not
valid  for  communication.  Communication  is  effected by  use  of  a
Communicator  object,  which is a binding  of  context, zero  or  more
groups (just zero or one in this letter),  and communicator attributes
(just one in this letter).

Two classes of communicator are described in this letter:

* WorldCommunicator - an instance of a WorldCommunicator is a binding of
  context   to  nothing.   This   communicator   allows   the   user  to
  intracommunicate within  the world  of  processes comprising  the user
  application,  labelling  processes with their (process  local) process
  identifier.

* GroupCommunicator - an instance of a GroupCommunicator is a binding of
  context to  a process  group.  This  communicator  allows the user  to
  intracommunicate within the  group of  processes comprising the group,
  labelling processes with their (group global) rank within group.

Communicator   creation  defines   the   SECURITY  attribute   of  the
communicator to be created, which may be any of the following:

* MPI_DEFAULT_COMMUNICATOR - the default Security attribute
  specified in environmental management.

* MPI_REGULAR_COMMUNICATOR - the regular Security attribute
  which provides regular point-to-point and collective semantics

* MPI_SECURE_COMMUNICATOR  - the secure Security attribute
  which provides secure point-to-point and collective semantics

Communicator objects are opaque objects of undefined size referenced
by an object handle which is expressed as in integer in the host
language.

Communicator  creation will create a context  for the Communicator, or
will  accept and  bind  a  user  managed  context. MPI should  provide
procedures for creation of each class of Communicator objects, and for
deletion of any class of Communicator object.  
For example,
            handle = mpi_create_world_communicator(context, security)
            handle = mpi_create_group_communicator(group, context, security)
            mpi_delete_communicator(handle)

In  each  creation  procedure  "security" is  the  security  attribute
described above. It is  the responsibility  of the user to ensure that
all communicators with the same context also have the same security.

In each creation procedure "context"  may be a user managed context or
may take the value  MPI_NULL_CONTEXT  (or  something  like that :-) in
which  case  the creation  procedure also creates  a context  for  the
communicator.  If the creation procedure creates  a  context  then the
procedure  synchronises the  calling processes  (all  processes for  a
WorldCommunicator and the  group of processes for a GroupCommunicator)
and returns the same context to each copy of the  communicator object.
If a user managed context was supplied then the  procedure is  process
local  and it is the responsibility of the  user  to ensure that  each
user managed context is bound to no more than one  communicator at any
time.

In the GroupCommunicator  creation procedure "group"  is a handle to a
group description.

The  communicator deletion procedure deletes the bound context if that
context  was created in the communicator creation  procedure  but does
not delete a user managed context.

Short Examples
--------------

A user program which only makes use of  two user reserved contexts and
makes no  use  of process  groupings  can "enable" the  user  reserved
contexts by creating WorldCommunicator objects.
For example,
            c0 = mpi_create_world_communicator(0,MPI_DEFAULT_COMMUNICATOR)
            c1 = mpi_create_world_communicator(1,MPI_DEFAULT_COMMUNICATOR)

A ClassA library can accept a communicator object as argument.
For example,
            void class_a_procedure(int communicator, ...) 
            {
              /* do it */
            }

A ClassB library can accept a group as argument and create private
GroupCommunicator objects.
For example,
            void class_b_procedure(int group, ...)
            {
              static int communicator = MPI_NULL_COMMUNICATOR;

              if (communicator != MPI_NULL_COMMUNICATOR) 
              {
                  communicator = mpi_create_group_communicator(group,
                                                    MPI_NULL_CONTEXT, 
                                            MPI_SECURE_COMMUNICATOR);
              }

              /* do it */
            }
This example could  be generalised by adding a group "cache"  facility
as described by Rik Littlefield.

Point-to-point communication
----------------------------

The  point-to-point  (intra)communication  procedures  have a  generic
process     and     message     addressing     form     (communicator,
process_label,message_label).  I  shall  deal with  Send  and  Receive
separately.

Send(communicator, process-label, message-label)
----

* communicator  is   a WorldCommunicator or a GroupCommunicator

* process-label is { the (process local) identifier of the receiver when
                   {                 communicator is a WorldCommunicator
                   {
                   { the rank in communicator.group of the receiver when
                   {                 communicator is a GroupCommunicator

* message-label is   the message tag in communicator.context.

The point-to-point  communication is  REGULAR if communicator.security
has    the    value    MPI_REGULAR_COMMUNICATOR,    and    SECURE   if
communicator.security has the value MPI_SECURE_COMMUNICATOR.

Recv(communicator, process-label, message-label)
----

* communicator  is   a WorldCommunicator or a GroupCommunicator

* process-label is { the (process local) identifier of the receiver when
                   {                 communicator is a WorldCommunicator
                   {
                   { the rank in communicator.group of the receiver when
                   {                 communicator is a GroupCommunicator
                   {
                   { a wildcard value in either case

* message-label is   the message tag in communicator.context or a
  wildcard value

The point-to-point  communication is  REGULAR if communicator.security
has    the    value    MPI_REGULAR_COMMUNICATOR,    and    SECURE   if
communicator.security has the value MPI_SECURE_COMMUNICATOR.

Collective communication
------------------------

The WorldCommunicator is not valid for MPI collective communication.

The  GroupCommunicator  is  valid  for  MPI  collective  communication
procedures.     The   collective    communication   is   REGULAR    if
communicator.security  has  the  value  MPI_REGULAR_COMMUNICATOR,  and
SECURE if communicator.security has the value MPI_SECURE_COMMUNICATOR.

			o--------------------o

Comments, questions, (flames :-), please!

For your conveniene, my plan now is to go into a session of deep thought
regarding intercommunication, the work we have done at EPCC, and MPI.  I
will then discuss these thoughts with my colleagues here, and promise to
return quality discussion of intercommunication to you sometime next
week. 

[If anyone wants to discuss intercommunication with me, I prefer to do
so privately until I have really thought longer and harder than before.]

I have an oustanding reply to Paul Pierce's recent letter, which I shall
make now.  I'll be off-line for a while, probably come on-line again
Sunday, and will reply to letters which I hope you will write in a
reactive and less prolific fashion. 

Happy reading :-)

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/