Subject: Revised Multi-level MPI Suggestion

From: Rusty Lusk <[email-masked]>
Date: Tue, 02 Mar 1993 15:01:44 -0600

Before the last meeting we sent out a suggestion on how the potential
complexity of the evolving MPI library could be managed by dividing the
routines into layers.  The idea was to provide a very simple interface at the
highest layer, which we call Level 4, and provide all of the flexibility
necessary at lower levels, which we called 1 and 2.

     Since then the February meeting has taken place, and some of our thinking
has changed.  The basic idea still seems like a good one to us, so here is a
new proposal for multiple levels in MPI.  At the February meeting, Marc Snir's
detailed point-to-point proposal basically covered what we called level 1
(where there are a small number of routines, including "init" routines for
operations) and level 2 (where sets of the level 1 routines are bundled
together for efficiency, but almost all functionality is preserved).  There
also seemed to be general agreement that there was no need to have separate
routines for supporting communication among heterogeneous machines; rather,
contiguous buffers could be specified by (addr,datatype,numitems) instead of
(addr,len).  The concept of buffer-descriptor (details in another message)
would be used to describe noncontiguous buffers and buffers of mixed data
types.

     Therefore it seems unnecessary to have *two* higher levels, since one of
the differences between our level 3 and level 4 was the specification of data
types.  At the same time, it still seems useful to us to have a level of MPI
routines that are simpler than the full set specified in Marc's document.  We
therefore suggest here a level 3 for MPI that imposes a set of restrictions in
the interest of simplicity.  (So, for example, it will not use buffer
descriptors, but restrict users to contiguous buffers of data of the same
type.)  It corresponds roughly to "current practice", but adds thread safety
(no "last message" semantics).  It assumes the existence of only one default
"all" group and one "universal" context, so that these need not be part of the
calls.  It does include nonblocking sends and receives, but simplified
versions of wait, status, and probe.  Users requiring more flexibility can
use the routines of level 2 instead.

Level 3 MPI routines:  (syntax only specified for definiteness)

  numids()				returns number of processes
  myrank()      			returns caller's rank in group "all"

  bsend(dest,tag,buf,datatype,numitems)  blocking send (until buffer is empty)

  brecv(from,tag,buf,datatype,maxitems,  blocking receive, with separate 
        actual_from,actual_tag,		 output arguments to handle wild-card
        actual_numitems)		 input arguments

Note that lots of programs have been written with just these four calls.

  handle = nsend(dest,tag,buf,datatype,numitems)  non-blocking send

  handle = nrecv(from,tag,buf,datatype,maxitems,  non-blocking receive
        actual_from,actual_tag,		output arguments to handle wild-card
        actual_numitems)		input arguments

  status(handle)			check status of non-blocking operation
  wait(handle)				wait for a non-blocking operation

  probe(from,tag,actual_from,actual_tag,actual_length)  
 					look in queue for matching message,
					get length so can allocate buffer

This is a larger set than the "four-function" interface we proposed as level 4
earlier, but not much, and has the advantage that one can write correct
programs even on machines with no buffering.  It corresponds roughly to
current practice in that most existing programs can be easily converted to
this level.  It gains simplicity and ease of use by not offering access to
MPI's advanced features like noncontiguous buffers of various and mixed types,
explicit handle management, high-performance protocols, and elaborate status
and wait calls.  It has no explicit mention of groups or contexts.

The collective operations seem to be getting complicated too, and would
benefit from a restricted, simple level.  One way to do this would be to
add a (blocking) barrier and collective operations in their simplest forms,
without explicit mention of group-id's.

These Level 3 operations add nothing to MPI that cannot be done at lower
levels.  However, it seems to us that adopting these simplified versions into
the standard will make it easier for many people to use and hence pave the way
for its acceptance.

Any comments?

Bill Gropp & Rusty Lusk

---

From: [email-masked] (Tom Henderson)
Date: Tue, 2 Mar 93 16:06:22 MST

A Suggestion for probe() in the New Rusty and Bill Scheme

I'm a bit confused about probe() in Rusty and Bill's latest proposal:  

...
> 					get length so can allocate buffer
...
> Bill Gropp & Rusty Lusk

I believe that there was general agreement that probe() should provide the 
caller with a "handle" to avoid "hidden state" (the thread-safety issue).  
(The probe() in the current point-to-point proposal does this.)  Should the 
probe() proposed above return "handle"?:  

  handle = probe(from,tag,actual_from,actual_tag,actual_length)  

If so, we need to add a "recv()" that has "handle" as an argument at Level 3 
(basically precv() as I understand it).  (This "handle" was also occaisionally 
referred to as "lock" in the last meeting.)  Also, the current proposals all 
require that a receiving process know the data type(s) in an incoming message 
before that message is received.  This means that datatype can be included in 
probe() and "numitems" can be used instead of "length".  Syntax is then 
consistent with other routines in the new "Level 3".  

Here's a counter-proposal (in the spirit of Rusty and Bill's syntax):  

 handle = probe(from, tag, datatype, actual_from, actual_tag, actual_numitems) 

 precv(handle, buf)

The use of "handle" guarantees that input parameters to probe() do not need to 
be repeated in precv().  "maxitems" (as in brecv()) doesn't seem to be 
necessary since the caller will know how much memory is required before buf is 
specified.  

It might be a good idea to add "unlock(handle)" to Level 3.  (OR remove 
probe() from Level 3.)  

Of course my preference is to dump the whole mess, including buffer 
descriptor, into an opaque data structure (as in Leslie's and my "message 
capsule" proposal on Jan 15 in mpi-pt2pt  :-).  

Tom Henderson
NOAA Forecast Systems Lab
[email-masked]