Subject: Multiple levels

From: Rusty Lusk <[email-masked]>
Date: Wed, 03 Feb 93 23:01:33 CST

Chuck --

Thanks for your comments.  They are of two kinds.  The first kind are about
the suggested multiple-level organization of the library.  There I think I
didn't explain some things well enough, and will try to do better here.  The
other kind are about what should be in or out of the library altogether, and
there I can only try to summarize what people talked about in Dallas;  the
things we incorporated are just the things that the straw votes indicated
people wanted in the standard.

| 
| I do view the interface as being layered, but I would organize the layers
| differently.  And I have a couple of arguments that I haven't tried out
| yet against some of the send/recv routines.
| 
| First, it sounds to me like you are suggesting that the 72 routines
| be implemented as 72 simple routines that each make a small number of
| subroutine calls to very simple routines.

This is not at all what we intended to suggest.  We used the word "levels"
rather than "layers".  The upper levels could be *defined* in terms of the
lower ones (That's what makes them upper-level) but the intention is that
they would *not* be implemented as calls to the lower-level routines.  In
fact what we called level 2 (large number of routines) exists partly to
avoid the multiple subroutine calls necessary in level 1 (where there are a
small number of routines.)  In fact, level 2 routines don't need to use the
data structures implied by the get_handle-mod_handle-use_handle-free_handle
at all.  They just have to do what they say they do, as efficiently as
possible. 

| 
| For heterogeneity, I would argue that it is wrong to put the datatype
| translation filter in the communications channel.
|

Our discussions seemed to indicate that people want it there.  The popularity
of systems like PVM, p4, Express, and others that handle this for the user
attests to this.  I think the reason is portability;  people like being able
to move code unchanged from a heterogeneous network to a multicomputer and
back.  The system can (relatively easily) handle all the translations,
including figuring out whether it is necessary at all or not.  Many users
hardly even know about the non-portability of data formats;  they think (in my
opinion rightly) that the communication system should handle that if they want
it to. 

|                                         In your model, it
| is relatively easy to do things like:
|
| 	bsendh (dest, tag, buf, FLOAT, 1);
| or even:
| 	bsendh (dest, tag, buf, ARRAY|FLOAT, nitems);
| but what about
| 	bsendh (dest, tag, buf, rpc_t, 1);
| where 'rpc_t' is some user defined data structure?  Thus, this model
| lacks generality. 

Not quite true, although not entirely false.  In the "draft implementation" of
mpi that you can get from here, we use an iovec-like structure which is a
vector of (addr,datatype,nitems) triples to describe a scattered, mixed type
buffer.  An arbitrary structure can be sent this way, provided the user builds
the structure descriptor (It's too bad compilers don't do this for us.)

| 
| It would be as efficient to implement the translation routines completely
| separate from the communications routines.
| 
....
| 
| The above is essentially the same argument that I'm going to use for
| strided messages.   Strided messages are only efficient if you have
| hardware that can DMA the strided vector, and then its not so efficient
| that it's obviously worth the effort.

Machines do have this kind of hardware, and we can expect to see more of it.
One thing that I will continue to argue in favor of is giving the user access
to as much of the speed the hardware can deliver as possible.

| 
| For ready-receiver, I'm going to argue that the communications subsystem
| should always assume the receiver is ready and then handle the case where
| this isn't true.
|

Some existing machines can perform *significantly* faster if the user can
relieve their system of some buffer-management chores.  If we don't give the
power users access to this efficiency, they simply won't use MPI.  And it will
come to pass that everyone will agree that to get real efficiency, you have to
use the vendor's unique routines.  Or you have to use "MPI with the XXX
extension package".  Portability will be destroyed.  This is a way the MPI
effort could fail.

| 
| Implementing the ready-receiver places strong semantics on the sender
| and receiver.  The sender has to know whether or not the receiver was
| implemented in a fashion that allows the sender to assume that the receiver
| is ready.
| 

Agreed.  The naive user can use what we call the Level 3 or Level 4 routines,
and never read the chapter in the manual that describes "ready-receiver"
semantics and requirements.  The power user can use them to write *portable*
fast code.

| 
| Finally, let's consider synchronous messages versus blocking messages.  For
| most applications, you want reliability.  Implementing blocking
| non-synchronous messages doesn't help you implement this.  You can't reuse
| the buffer being sent until you know that it has been received at the other
| side.  So you might as well either send the message synchronously, or send
| it non-blocking and go off and do something else while you're waiting to see
| if you'll need to retransmit the buffer.
|

Most people want to compute while the message is in transit, and even while
the buffer is being emptied. They don't want this capability taken from them.

| 
| This reduces the number of interface routines to 8.  I could probably
| be convinced that adding 4 more interface routines to support strided
| messages doesn't overly complicate the interface.
| 

The Level 4 routines are only 2.  More routines are needed only for more
efficiency and functionality.  In the last few weeks I have talked to lots of
people about the MPI effort, and they are mainly afraid that we will leave
something important out.  If we don't supply all the functionality and
efficiency that we possibly can, MPI will fail.  The problem is to do so
without making the package too complicated.  Our posting about multiple levels
was merely an attempt to demonstrate that it could be done.

Regards,
Rusty Lusk & Bill Gropp