Subject: mpi-comm: various (long)

From: L J Clarke <[email-masked]>
Date: Thu, 8 Apr 93 12:10:04 BST

Dear MPI Colleagues

I am sending this letter to mpi-comm as it seems to cover issues in many
subcommittees and in particular: point-to-point, collective, context,
environmental.  This is a long, but I believe important, letter. 

The letter is a discussion of some issues primarily in point-to-point,
collective  and context, with some strong and powerful recommendations
for each of these subcommittees.

                      o--------------------o

1)  In  the recent  meeting  we  discussed and  narrowly  accepted  an
amendment to the  point-to-point  chapter which  included  secure (aka
synchronous, aka unbuffered) communication. 

Paul Pierce  made  the point that  providing  different point-to-point
procedures does not seem to be  the right model for this capability as
it has  a  microscopic effect  on  MPI  user software.  This  point is
actually well taken (by myself at least). Paul went on to suggest that
there should be application global control over whether point-to-point
communications are all secure (aka synchronous, aka unbuffered) or all
insecure (aka regular, aka asynchronous, aka buffered). I opposed this
suggestion as it does not respect the requirements of library writers.

There appeared to be  a number of people who were  concerned about the
geometric  increase  in  the number  of  point-to-point  communication
procedures.

The  discussion of secure  send and secure receive became messy, and I
promised to return some  more sane discussion of the point.

In this letter I will  later (after consideration  of some  issues  in
collective  communications   and   communication   contexts)  make   a
recommendation which: respects the well taken point of  Paul; provides
a mechanism for the global control suggested  by Paul while respecting
the independence  of library writers from user program writers; avoids
geometric  growth  in  the number of  point-to-point  procedures;  and
completely side-steps the secure send and secure receive issue.

2) The question of  whether collective communication  procedures do or
do  not  barrier  synchronise the group  has  been  raised a number of
times.  In the recent meeting this lead to Steve  Otto suggesting that
MPI user software  should be  written as  though these  operations  do
barrier synchronise although the  MPI implementation is not obliged to
make  the  operations  so  do. 

This is exactly analagous to the point-to-point insecure (aka regular,
...) communications which may or  may  not synchronise the  sender and
receiver depending  on  the  implementation (read depending  on system
buffering resource availability if you prefer). In order to write safe
(i.e. guaranteed portable)  programs with insecure  (aka regular, ...)
communications the MPI user software should be written as though these
communications were secure (aka synchronous, ...).

In  this  letter  I will  later (after consideration of some issues in
communication contexts) make a recommendation which: provides for both
secure (i.e. barrier synchronous) and insecure (i.e. may or may not be
barrier   synchronous   depending    on   implementation)   collective
communications;  and  avoids  geometric  growth  in   the   number  of
collective communication procedures.

3) In the  context subcommittee Mark Sears,  in  Proposal  VIII and in
presentation at the recent meeting, has pointed out that there are two
classes of MPI user libraries.  I will  discuss this (from my point of
view) briefly here for concreteness.

There is a class of libraries which accept context(s) from the  caller
and  use  these context(s)  for communication.  Note  that  since  the
library  uses the contexts(s) given by  the caller then it is  clearly
the  responsibility  of  the  caller  to  ensure  that  there  are  no
"dangling"  communications, either  in buffers or  nonblocking,  which
will interfere  with the communications of the  library, and it is the
responsibility of  the  library to ensure  that there are no  dangling
communications between calls to the library which  will interfere with
communications of the user.  Let me label this class as ClassA. 

There is a  class of libraries  which  create  context(s) for  library
private use and use only these context(s) for communication. Note that
since the library uses  private context(s) then the user does not have
any concern regarding dangling communications which may interfere with
the library communications, and equally the library may leave dangling
communications between library calls since  they cannot interfere with
the  user communications.  Let me  label this class as  ClassB. 

ClassA libraries will  generally operate within one (or more) group(s)
and accept  one (or  more)  group(s) from  the caller,  in addition to
context(s).  ClassB libraries will  also generally operate within  one
(or more) group(s) and accept one  (or more) group(s) from the caller,
but do not accept  context(s). This observation  suggests that context
and group should be different entities within MPI. (Whether we have an
object which  binds a group and a context is another issue to  which I
wish to return in a separate message of narrower circulation.)

Observe  that  the  (blocking)  collective  communications  which  are
described in  the  MPI  collective  communication  chapter  draft  are
conformant  with the ClassA library described  and s single group. The
kind  of libraries  which I  am  writing  (and I  suspect  also  Tony,
although  I am  not  sure)  are  conformant  with  the  ClassB library
described.

4)  I  now  make  a  recommendation  which  ties  together the  points
discussed above and delivers the promises  made. First we observe that
context is the MPI mechanism for distinguishing the communications  of
the user from (ClassB) libraries.

The recommendation is that secure or insecure modes of operation are a
a  property of the communication context. This applies equally to both
point-to-point communications.

In point-to-point communications  a secure context provides the secure
point-to-point  communication which I described at the recent meeting,
and   an  insecure   context  provides  the   inseure   point-to-point
communications  which is as discussed (regular) in the current chapter
draft. (The semantics and desirability of ready-receiver communication
in secure and insecure contexts is a subject to which I wish to return
in a separate message of narrower circulation.)

In  collective  communications  a  secure   context  provides  barrier
synchronisation  across  the group, and an insecure context may or may
not  provide  barrier synchronisation which  is  as  discussed in  the
recent meeting.

There should be an application global default context property, either
secure or insecure, which is specified by the user in an environmental
management procedure (perhaps as  argument to mpi_init()?).   Creation
of  context  should  specify  whether the  created context is to have:
default property; secure property; insecure property. It is possibe to
provide these capabilities in  terms of buffering  in  different ways.
One  possibility  is that  the property of a  context is the buffering
associated with that context, in which case a  secure context  is  one
with zero  associated  buffering.   Another  possibility is  that  the
property  of a  context  is  whether or  not  it  should use  whatever
buffering is available,  in which  case  a secure context is one which
does not use any buffering.  My preference is  for the latter since it
does not prescribe the description and association of buffering.

This recommendation does deliver  the promises  made. The requirements
of the (ClassB) library writer are  respected  while  providing global
control for  the  user program  writer. The capabilities  required are
provided  without  geometric  growth  of the number  of point-to-point
and/or collective communications. 

                      o--------------------o

Free beer to all who can demonstrate that they have read and considered
this (long long letter).

Comments, questions, (flames :-) please?!

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

---

From: L J Clarke <[email-masked]>
Date: Fri, 9 Apr 93 16:24:14 BST

Dear MPI Colleagues

This message is short.  It is two little things which I forgot to say in
the "mpi-comm: various (long)" message.

We can loosely associate ClassA libraries as collections of procedures
for which one can use the substitution model of evaluation, and ClassB
procedures as encapsulated objects.

The argument lists of of ClassA and ClassB libraries differ primarily in
that ClassA libraries accept context(s) whereas ClassB libraries do not. 
This seems (to me) to convey just the right idea to the library user
regarding the responsibilities for management of contexts and messages
within those contexts. 

Best Wishes
Lyndon