Subject: Multiple levels of collective operations

From: Rusty Lusk <[email-masked]>
Date: Wed, 10 Feb 93 00:15:27 CST

A few days ago, we posted a suggestion for a multi-level MPI specification as
a way to manage the complexity inherent in providing a highly functional
interface, and to make MPI accessible to users who did not need or want all of
its capabilities.  The examples there were only for point-to-point
communication.  We thought it worth exploring whether the same ideas might be
useful in the context of the group operations.  Here are a few thoughts.

The idea here is not to propose specific syntax, but to suggest how
simultaneous adoption of multiple levels of collective operations might help
provide both a simple interface and a full-featured one.

Level 4: (very, very simple)

Here there would be only one group, the default group of all processes, so
there is no group id in the parameter lists.

MPI_barrier
MPI_combine(...,SUM,...)
                MAX
                ...
  (That is, only a fixed set of combining operations)

MPI_broadcast(....)

All operations would block until globally completed.

Message types would be just like level 4 of the point-to-point operations:
either contiguous strings of bytes, or arrays of specific data types specified
like  (...,datatype,numitems)

The main point is that using just this level of group operations one could
port many existing programs that use global operations, without introducing
any explicit management of groups.

Level 3: (still pretty simple, but more functionality)

All the utilities proposed by Al Geist at the last meeting for the management
of groups (creation, inquiry, etc.)

Operations at this level have group id as an explicit argument (Again as Al
suggested).

User-defined combining functions.

Gather operations.

Main restriction: for simplicity, still simple contiguous buffers made up of
either untyped byte strings or arrays of fixed types.

All operations still block.  In fact, we propose that there not be
non-blocking collective operations, even at lower levels.  The necessity of 
cooperating in the global operation beyond just making one's own explicit
contribution puts non-blocking global operations more in the "threads" domain.

Level 2:  (More options for buffer structure, as in the point-to-point routines
at this level)

Here buffers specified for broadcasting, combining, etc., could be completely
general, including mixed-type, non-contiguous buffers described by vectors of
(address,datatype,numitems) triples.

Level 1:  (Full functionality, plus separate "setup" parts of the operations,
like in Marc's point-to-point proposal.)

Here operations would have a separate "init" call.  Being separate has the 
same advantages as for the point-to-point operations, in that resources could
be reused in loops.  The "init" call would return a handle that could be
further modified (setting buffer type, address, etc.) before use.  The payoff
seems potentially greater here because some of the "init" calls may turn out
to be inherently non-scalable, and therefore reuse of their results is
important. 

Summary: Multiple levels might help deal with conflicting demands for both
simplicity and functionality in the case of collective operations as well as
for point-to-point operations.  What was generally discussed at the last
meeting is very much like Level 3 above.  An even simpler interface (so that
groups need not be explicit at all) is provided by Level 4, while the somewhat
grubby aspects of multiple and complicated buffer formats is pushed down into
Level 2.  Finally, Level 1 provides the opportunity to absorb some of the
difficult-to-scale parts of global operations into separate, reusable calls.

Comments?

Rusty Lusk and Bill Gropp