Subject: A few general comments

From: [email-masked] (Tom Henderson)
Date: Tue, 5 Jan 93 15:31:37 MST

Here are a few comments/questions regarding the MPI1 draft proposal from folks 
here (Forecast Systems Lab).  Apologies if you have seen them before!  

1)  Currently, communication between groups operating in different 
    communication contexts appears to be very difficult.  One way to alleviate 
    this problem might be to provide a way to "name" contexts.  Communicating 
    groups could then refer to a context by name to ensure that the same 
    message types were used.  Message registration would also accomplish this 
    and might be easier to implement (see our "Proposal 1" below).  

2)  We believe that MPI1 should include a specification of low level
    protocols that vendors should implement so that heterogeneous computing
    can be made available.  Each media type (standard ethernet, FDDI, HIPPI,
    etc.) should have a specification.  This would greatly aid in this 
    standard being useful in a workstation environment.  The protocol
    document would be a companion document to the syntactical standard for
    MPI1.

3)  On p. 9  "No reference may be made to any process or group outside the 
    current group context."  How do different groups that are simultaneously 
    working on different stages of a problem communicate?  MPI_PARTG()??  

4)  The use of stacks to manage contexts and groups may be a bad idea if MPI 
    ever intends to support MIMD programs (how do groups synchronize stacks of 
    message contexts?).  Message registration would do the job without 
    stacks.  Registration of blocks of messages at the beginning of a program 
    (or large module) would not affect performance very much.  For groups, we 
    like Al Geist's approach of identifying a process by one or more 
    (group ID, process ID) pairs.  We feel that communication between groups 
    is very useful, even for SPMD programs.  

5)  It is clear that some way of responding to asynchronous events must be 
    provided.  We believe that "interrupt" type routines (such as Intel's 
    hrecv() and hsend()) may not be the best approach.  We have a brief 
    alternative proposal for "Remote Services" below ("Proposal 2").  

6)  The consensus seems to be that "parallel I/O" should not be a part of 
    MPI1.  Should it be a part of MPI2 or MPI3?  If our intent is to create a 
    standard that supports the development of portable code then we believe 
    that I/O must be considered at some point.  We should make sure that the 
    specification of MPI1 includes the hooks necessary to include I/O at a 
    later date.  If our intent is to create only a low level library, then we 
    can leave all this up to us users.  We have some ideas about I/O and Data 
    Decomposition below ("Proposal 3").  Successful implementation of such a 
    proposal would require process topology information (yes, here's a plug 
    for the Process Topology subcommittee).  

The proposals follow.  

Regards, 

                      Leslie Hart ([email-masked])
                      Tom Henderson ([email-masked])
                      Bernardo Rodriguez ([email-masked])

               PROPOSALS

Proposal 1.  Message Type Registration.

Message types are managed using a global registry.  Routines are
provided that allow the user to "reserve" blocks of message types for
use by the application.  Message type registration is intended to
prevent message type "collisions" when user code and third-party library
routines are used together (historically, a serious problem with typed
message passing systems).  Initial concepts for message type
registration are due to Tony Skjellum of Lawrence Livermore National
Laboratory.

Two proposed routines to register and monitor message types are described 
below.  Arguments have been omitted for brevity.  

Message type registration routine:

MPI_ReserveMsgType()
Reserve one or more message types for use by the calling process.  This
routine is used to prevent message type "collisions" between user
applications and libraries.  Inputs include an ID string and the number
of message types to be reserved.  The routine first checks the locally
maintained message type registry and returns an error condition if the ID
string matches an existing entry.  The routine then goes to the global
message type registry to get the message types.  If the ID string matches an
existing entry in the global message type registry then the message types are
copied from that entry to the calling process.  Otherwise, new
message types are registered with the ID string, entered into the global
message type registry, and copied to the calling process.  The new ID string
and message types are entered into the locally maintained message type 
registry.

Message type registration inquiry routine:

MPI_InquireMsgTypeByName()
Ask the message type registry to obtain the message types reserved under
an ID string.  An error is returned if no message types have been reserved
under the ID string. 

Proposal 2.  Remote Services (brief concept only).

A remote service is similar in concept to a RPC (remote procedure call).
Remote services are most often used to respond to specified
asynchronous events.  The actions taken in response to asynchronous
events are performed by agent processes known as protocol handlers.

A remote service is requested of one process by the same or another
process.  The remote service request is accompanied by a small amount of
data to guide the action of the service.  Routines are provided to allow 
user-created routine to be attached to specified requests.  An attached 
user-created routine is called a protocol handler.  A protocol handler does 
not have access to the address space of the its associated process.  

The protocol handler has the status of a full process (ie. it may perform I/O 
and message-passing communication).  This is an advantage over some current 
interrupt-driven approaches.  Remote service requests are handled one at a 
time on a first-come first-served basis (ie. a new remote service request will 
not be handled until all pending remote service requests are handled).  This 
may be a drawback to this approach.  

Proposal 3.  Some ideas about Cartesian Data Decomposition and Parallel I/O.  

Due to length, we have just included a few ideas.  If there is any interest, 
we can provide (lots) more detail.  

1)  We propose Cartesian data decomposition routines to support the
    scalable distribution of multidimensional arrays among the processes in
    a group. These routines isolate the user from the details of
    implementing scalable data parallelism in a message passing environment.
    The routines also enhance performance by distributing data in a way 
    that is most advantageous for a particular architecture.  

2)  A Cartesian data decomposition defines the distribution of a
    multidimensional data array onto a "logical" array of processes with the
    same dimension.  The processes are members of a specified group. 
    A Cartesian data decomposition is defined by information provided by the
    user and by run-time parameters determined by the system.  The user
    specifies the group, array dimension, and number of elements in
    each dimension of the data array.  The user may also optionally specify
    re-ordering of processes in the group and boundary behavior.  The 
    Cartesian data decomposition routines then combine this information with 
    the number of processes in the group (provided by the system at run time) 
    to determine which portion of the data array will reside on each process 
    in the group.  All information about the decomposition is then stored in 
    a structure (or FORTRAN77 array).  

    The "logical" array of processors is created using the Processor Topology 
    routines (this idea has already been discussed in some detail in that 
    subcommittee).  

3)  Cartesian data decomposition and I/O should be intimately related.  Once a 
    decomposition structure has been created, I/O should be possible using 
    routines with syntax similar to this:  
    
        cget(file, array, decomp)
        cput(file, array, decomp)

    where decomp is the decomposition structure.  

4)  Data decomposition tools have been available for several years, most
    notably from Parasoft's Express software package.  These tools are
    extremely useful for almost all data parallel applications.  Use of these 
    tools in conjunction with I/O greatly simplifies program development on
    message passing systems.

5)  Few areas in parallel processing have been dealt with in less detail by
    standardization efforts than parallel I/O.  We would like to see 
    behavior defined for such operations as seek/read and seek/write when 
    several processes are accessing the same file.  The concept of "loose 
    synchronization" (Express and elsewhere) has proven to be very useful 
    and should be included in a standard.  Encapsulation of I/O in 
    standardized routines would also permit performance optimization by the 
    people who (should) know best how to do it (ie. the hardware vendors).