Subject: A suggestion for a multi-level MPI

From: Rusty Lusk <[email-masked]>
Date: Wed, 03 Feb 93 00:40:59 CST

At the last meeting, it became apparent that many people were uncomfortable
with the large number of routines (1024 sends?) that we seemed to be going
towards.  There are good reasons for the standard not to be incomprehensibly
complex.  There are good reasons also for it not to be crippled by lack of
functionality.  Are we stuck?

There are two ways to deal with this problem.  This note is a proposal that we
adopt both of them.

The reason the problem can be solved is that the sets of options that we have
been considering (blocking or not, contiguous buffer or not, heterogeneous
machines or not, etc.) are orthogonal, and one of a large number of possible
send routines can thus be specified simply and without extra parameters.  In
addition, certain subsets of the set of all operations can be identified and
packaged for ease of use.

Here is a suggestion for how to organize the routines and end up with a
powerful, flexible, yet easy-to-use library of point-to-point functions.
(We assume that these will all be renamed to have an MPI prefix, to avoid
name-space pollution)

Level 4 (very restrictive, very simple):

    send(dest,tag,buf,len)
    recv(dest,tag,buf,len)

Both of these block until the operation is locally complete.  The buffer is a
contiguous sequence of bytes. No special protocols or translation for
heterogeneous machines.

This level is enough to express lots of parallel algorithms.  The main reason
we need more is for efficiency and control.  We now add these in steps.

Level 3 (still simple, but allows for overlap of computation and communication
         and better buffer management, and allows heterogeneous machines to
         communicate.) 

    bsend(tag,dest,bufadd,buflen)
    nsend(tag,dest,bufadd,buflen)
    bsendh(tag,dest,bufadd,datatype,numitems)
    nsendh(tag,dest,bufadd,datatype,numitems)
    brecv(tag,dest,bufadd,buflen)
    nrecv(tag,dest,bufadd,buflen)
    brecvh(tag,dest,bufadd,datatype,numitems)
    nrecvh(tag,dest,bufadd,datatype,numitems)

Here the n or b selects block or nonblocking operations, h specifies
translation for heterogeneous machines.  The restrictions (which allow the
parameter lists to be short and the number of routines to be small) are: no
noncontiguous buffers, data in message all the same type, no globally
synchronized, CSP-like send-receive, and no special protocol (like the
ready-receiver protocol discussed at the last meeting).

In order to emphasize the orthogonality, we might organize these routine names
in a diagram like this:

   [n][send][ ]
   [b][recv][h]

That is, one makes a choice in each column.

Level 2 (access to almost all MPI functionality via a large number of
orthogonally named routines):

   [c][n][send][ ][  ]
   [s][b][recv][h][rr]
   [g][s] 

the first column designates the buffer type (contiguous, strided, general
gather/scatter), the second is the termination type (nonblocking, blocking,
synchronized), then send or receive, "h" or not for heterogeneous
communication, "rr" or not for "ready-receiver" protocol.  This is a lot of
routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to
understand. 

The only restriction here is that there are no "channels":  the initialization
of a send operation is combined with initiating it.

Level 1 (full MPI functionality, in a small, flexible set of routines)

   handle = init_send(tag,dest)
            mod_send(handle,option,value)
            do_send(handle)
            free_send(handle)

along with the corresponding receive routines.

The idea is that the "init" routines will set up a usable set of defaults for
all the options, specifying only minimal parameters, and then the "mod"
routine can be repeatedly called to specify all the options with regard to 
buffer type, termination type, etc.

The point is that at this level we can offer not only all possible options,
with a small number of routines, and include the "setup" routines proposed by
Marc for increased efficiency, but we also allow room for growth in the
standard through the addition of more values for "option" in the "mod" calls.

So here's how it might look, this time from the bottom up:

Level 1:  full functionality, small number of routines, channel setup

Level 2:  sequences of Level 1 calls of the form

            init...
            mod...
            mod...
            mod...
            do...
      
are given specific names in an organized way.  This promotes ease of use and
also efficiency (fewer calls).

Level 3: several of the Level 2 routines are renamed and promoted to this
level to provide a useful subset of MPI functionality in a small number of
routines, each having the minimal number of parameters.

Level 4:  ultra-simple interface

All four levels would be part of the standard and it would be possible to mix
levels in the same program.

We have not considered carefully yet how this interacts with groups and
contexts, but similar ideas might be useful there, where there are similar
problems in providing both functionality and simplicity.

Comments?

Rusty Lusk and Bill Gropp

---

From: [email-masked] (James Cownie)
Date: Wed, 3 Feb 93 11:34:48 GMT

The fundamental approach seems fine to me (though Chuck makes some
good points...)

However there's at least one point to note :-

1) The Fortran binding issue suggests that in the heterogeneous cases
   there should be a  routine for each data type, rather than having the
   data type as an argument.

-- Jim
James Cownie 
Meiko Limited			Meiko Inc.
650 Aztec West			Reservoir Place
Bristol BS12 4SD		1601 Trapelo Road
England				Waltham
				MA 02154

Phone : +44 454 616171		+1 617 890 7676
FAX   : +44 454 618188		+1 617 890 5042
E-Mail: [email-masked]   or    [email-masked]