Subject: collcomm changes, summary

From: [email-masked]
Date: Sun, 14 Mar 93 11:59:13 PST

SUMMARY OF SUGGESTED CHANGES TO COLLECTIVE COMMUNICATION PROPOSAL

The draft proposal that Al Geist distributed several days ago
contains some features that would prevent it from being
implemented as a layer on top of MPI point-to-point facilities.

The purpose of this note is to propose changes to the group
control routines in order to permit layering, and to propose
other changes for better and more predictable performance.

A discussion of the rationale for these proposed changes 
will be distributed separately because of its length.

The main changes introduced in this note are:

. The concept of group identification is firmed up.  Most
  operations use a "group handle" that is local to the process.
  (Think of the group handle as being just the address of a
  potentially large and complex "group descriptor".)  There is
  still a "group ID" that is globally unique, but it has only a
  secondary role and can be ignored by most applications.  The
  "group name" is entirely removed from MPI-1.  (Group names are
  still anticipated in MPI-2, but upward-compatibility is
  maintained in a different way from the draft proposal.)

. A semantic restriction is introduced, that a process can access
  information about a group only if the process holds a group
  handle for it.  Group handles can be obtained in two ways: 1)
  they are produced by group formation routines, and 2) a process
  can explicitly distribute copies of its group handles to other
  processes, using new routines introduced specifically for that
  purpose.

. A cacheing mechanism is introduced, that allows modules to
  attach arbitrary information to a group descriptor in such a
  way that it can be quickly retrieved.  Cacheing facilitates the
  construction of collective communication routines that are
  "fast after the first execution in a group", no matter how the
  other group operations are implemented.

. A new group formation routine is introduced, that is less
  synchronous and more general than MPI_PARTGROUP.

Specifically, the following routines are proposed to be added or
modified:

1. Arbitrary group formation:

    newgrp_handle = MPI_FORMGROUP (grouptag,groupsize,knownmembers)

    where
     grouptag     is a user-provided integer tag, sufficiently unique
                  to disambiguate overlapping groups that might be
                  formed simultaneously (say by multiple threads).

     groupsize    is the number of members that will compose the group.

     knownmembers is a set of pid's of some or all members of the group.
                  Each member of the group must provide the same
                  set of knownmembers.

     newgrp_handle  is a group handle for the newly formed group

    This new routine must be called synchronously, but only by those
    processes forming the group.

2. Group partitioning:

    newgrp_handle = MPI_PARTGROUP (oldgrp_handle,grouptag)

    where the semantics are the same as the draft proposal except that
    the return value is now a new group handle instead of a rank.
    (The rank can be determined by a separate call to
    MPI_GETRANK(group_handle,pid) .)

3. Group disbanding:

    MPI_LVGROUP (group_handle)

    where the semantics are the same as the draft proposal except that
    MPI_LVGROUP now does not return any result.  (Since groups can now
    be formed arbitrarily, not just by partitioning, it is not obvious
    what MPI_LVGROUP could return in general.)  This routine can be
    called only by members of the group.

4. Distribution of group handles and disposition of distributed handles:

    MPI_SendGroupHandle (pid,context,tag,old_group_handle)

    new_group_handle = MPI_RecvGroupHandle (pid,context,tag)

    MPI_FreeGroupHandle (group_handle)

    (The latter routine is similar to MPI_LVGROUP except that
    it can be called only for distributed group handles.  This is
    solely for semantic clarity; a single interface routine would do.)

5. Cacheing group-specific process-local information:

    The following routines get and free keys for use with group
    cacheing.

      key = MPI_GetAttributeKey ()
      MPI_FreeAttributeKey ()

    The following routines cache and retrieve information.

      MPI_SetGroupAttribute  (grouphandle,key,value,destructor_routine)
      status = MPI_TestGroupAttribute (grouphandle,key,&value)

    where
      key         must be unique within the group
      value       is anything the size of a pointer
      destructor_routine   is an application-provided routine that
                           is called by MPI_LVGROUP, with arguments
                           being the group handle, cached key and value.

    Cached information is stripped from the new group handle
    returned by MPI_SendGroupHandle.

    In a conforming implementation, MPI_TestGroupAttribute must
    be no slower than a point-to-point communication call.

6. Retrieving global group ID:

    global_id = MPI_GetGlobalGroupID (grouphandle)

7. Other collective communications:

   Consistently substitute "grouphandle" in place of "group".

----------------------------------------------------------------------
[email-masked]               Rik Littlefield
Tel: 509-375-3927                    Pacific Northwest Lab, MS K1-87
                                     P.O.Box 999, Richland, WA  99352