Subject: My comments on Jim's coments, etc.

From: [email-masked] (Tom Henderson)
Date: Fri, 16 Jul 93 15:03:08 MDT

Tony,

Here are my comments on Jim's comments and a (hopefully constructive) 
proposal.  I am attempting to take the "user's" perspective.  I agree with Jim 
on some things, often not for the same reason...  I share Jim's desire to help 
in a constructive way.  If context doesn't happen at the next meeting, I fear 
for the success of MPI.  

Please let me know what you think of all this.  I know there are things in the 
details of the proposal which are not quite right.  Also, I realize that right 
now you're on the spot to produce.  If there is anything you can "farm out", 
I'll be happy to assist...   

As usual, feel free to reflect, forward, etc.  I've sent a copy to Jim also.  

Tom

1)  I agree with Jim that the current proposal is too complicated.  That's why 
    I asked you lots of questions about "which function should I use?" in my 
    comments on the draft.  The "security" issue also has some validity.  I 
    don't feel quite as strongly as Jim about this.  I do feel that 
    "non-expert" users should be able to have "security" without thinking 
    about it too much.  

    We have run into this problem in point-to-point.  We "solved" the 
    problem by introducing an EXPLICIT multi-level organization to handle
    "normal" vs. "expert" users.  I think this approach may solve some of our 
    problems in the context draft.  This means that the "easy" routines must 
    be able to handle everything that a "normal" user would want to do.  We 
    could then vote on whether to include the "expert" routines at all...  

    In my view, contexts and groups are "low-level" "expert-user" things.  The 
    communicator is a "high-level" thing that "normal" users should be able to 
    deal with.  "Normal" users should not have to worry about groups and 
    contexts.  I make a more detailed proposal below...  

2)  I confess that I don't completely follow the arguments about difficulty of 
    the "who sent me this" inquiry.  How does this work in CHIMP?  Can we look 
    at the source code (Lyndon)?  As for the unique one-to-one binding of 
    contexts (a context may be bound to one and only one group), this can be 
    enforced for "normal users" in the multilevel scheme (see proposal 
    below...).  

3)  Jim points out that inter-group communicators may be hard to use in the 
    current proposal.  I feel that some routines are "high-level" enough to 
    make them easy to use.  MPI_COMM_PUBLISH_SUBSCRIBE() and MPI_COMM_JOIN() 
    seem fairly straightforward.  Is Jim objecting to the use of a pair of 
    communicators (Send_to_A, Recv_from_A) rather than a single one 
    (communicate_with_A)?  

4)  I am less concerned than Jim with eliminating inter-group communication.  
    I don't see a huge difference between explicit inter-group communication 
    and Jim's "group fusion" (which is already supported in your current draft 
    using MPI_COMM_JOIN() as I understand it...).  

5)  I agree that we need "fast" creation of communicators to support 
    libraries.  Library writers can use the "expert" routines to do this...  

6)  If Jim puts together another proposal, we MUST discuss it in detail via 
    email well in advance of the next meeting.  

  PROPOSAL:  MULTI-LEVEL ORGANIZATION OF GROUP/CONTEXT/COMMUNICATOR ROUTINES

I propose that we use a "multi-level" organization of routines to solve the 
complexity problems in the current context draft.  This approach has been used 
successfully in the point-to-point draft to solve similar problems.  Routines 
in the current proposal are re-organized into two levels:  "basic", and 
"expert".  Routines are modified/added/deleted as necessary to preserve the 
current functionality at the appropriate level.  

The basic level allows manipulation of communicators only.  Contexts and 
groups are not visible.  This level provides the "security" of system-managed 
contexts at the expense of optimum performance.  This level is intended for 
use by the "typical" user.  It supports "fusion" of communicators (as 
suggested by Jim).  It does not support explicit publish/subscribe or 
flatten/unflatten.  

The expert level has all the low-level functionality required by library and 
server developers.  Explicit context and group manipulation is allowed as well 
as construction/destruction of communicators via bind/unbind.  This level 
provides the fastest possible performance at the expense of "security".  This 
level is intended for use by the "expert" user.  It supports explicit 
publish/subscribe and flatten/unflatten.  

I had a difficult time deciding what level the functionality of the local 
group operations (MPI_LOCAL_SUBGROUP(), MPI_LOCAL_SUBGROUP_RANGES(), 
MPI_LOCAL_GROUP_UNION(), MPI_LOCAL_GROUP_INTERSECT(), and 
MPI_LOCAL_GROUP_DIFFERENCE()) should be on.  After thrashing around for a 
while with versions that operate on communicators, I decided to leave them in 
the expert level.  The rationale here is that all the "local" operations 
exist primarily for performance reasons.  They give the "typical user" a lot 
of rope.  I can envision a situation where an incorrect sequence of calling 
various local communicator construction operations could result in two 
processes having the same context bound to a different group.  (Yeah, I know, 
"the behavior of an erroneous program is undefined..." :-)   I certainly could 
see having MPI_COMM_COLL_INTERSECT() or MPI_COMM_COLL_DIFFERENCE() at the 
basic level.  
Comments please?  

This organization will help us vote on the more advanced features as they will 
be isolated in the "expert" level.  It will help answer all the "which 
function should I use?" questions.  And, hopefully, "multiple layers" will be 
more palatable to the committee as a whole, since most of us have already 
bought the idea in point-to-point.  

In the current context draft, routines are organized as follows:  

>> GROUPS
>> 
>> Local Operations
>> MPI_GROUP_SIZE(group, size)
>> MPI_GROUP_RANK(group, rank)
>> MPI_TRANSLATE_RANKS (group_a, n, ranks_a, group_b, ranks_b)
>> MPI_GROUP_FLATTEN(group, max_length, buffer, actual_length)
>> 
>> Local Group Constructors
>> MPI_GROUP_UNFLATTEN(max_length, buffer, group)
>> MPI_LOCAL_SUBGROUP(group, n, ranks, new_group)
>> MPI_LOCAL_SUBGROUP_RANGES(group, n, ranges, new_group)
>> MPI_LOCAL_GROUP_UNION(group1, group2, group_out) 
>> MPI_LOCAL_GROUP_INTERSECT(group1, group2, group_out) 
>> MPI_LOCAL_GROUP_DIFFERENCE(group1, group2, group_out)
>> MPI_GROUP_FREE(group)
>> MPI_GROUP_DUP(group, new_group)
>> 
>> Collective Group Constructors
>> MPI_COLL_SUBGROUP(comm, membership_key, new_group)
>> MPI_COLL_GROUP_PERMUTE(comm, new_rank, new_group)
>> 
>> 
>> CONTEXTS
>> 
>> Local Operations
>> MPI_CONTEXTS_RESERVE(n, contexts)
>> MPI_CONTEXTS_FREE(n, contexts)
>> 
>> Collective Operations
>> MPI_CONTEXTS_ALLOC(group, n, contexts)
>> MPI_CONTEXTS_JOIN(local_comm, remote_comm_send, remote_comm_recv, n, 
>>                   contexts)
>> 
>> 
>> COMMUNICATORS
>> 
>> Local Communicator Operations
>> MPI_COMM_SIZE(comm, size)
>> MPI_COMM_RANK(comm, rank)
>> MPI_COMM_FLATTEN(comm, max_length, buffer, actual_length)
>> 
>> Local Constructors
>> MPI_COMM_UNFLATTEN(max_length, buffer, comm)
>> MPI_COMM_BIND(group, context, comm_new)
>> MPI_COMM_UNBIND(comm)
>> MPI_COMM_GROUP(comm, group)
>> MPI_COMM_CONTEXT(comm, context)
>> MPI_COMM_DUP(comm, new_context, new_comm)
>> MPI_COMM_MERGE(comm, comm_remote, comm_send, comm_recv)
>> 
>> Collective Communicator Constructors
>> MPI_COMM_MAKE(sync_group, comm_group, comm_new) 
>> 
>> 
>> INTER-COMMUNICATION
>> 
>> MPI_COMM_PUBLISH(comm, label, persistence_flag)
>> MPI_COMM_UNPUBLISH(label)
>> MPI_COMM_SUBSCRIBE(my_comm, label, comm)
>> MPI_COMM_PUBLISH_SUBSCRIBE(comm_A, label_A, label_B, send_to_B, recv_from_B)
>> MPI_COMM_JOIN(local_comm, remote_comm_send, remote_comm_recv, order, 
>>               joined_comm)

The "multi-level" organization follows.  Details are included for routines 
that are different from the current draft.  A few "discussion" points are also 
indicated.  The basic level contains 7 routines.  The expert level contains 28 
routines.  (The current draft has 34 so I haven't done too badly...  :-)

"BASIC" LEVEL

Local Communicator Operations

MPI_COMM_SIZE(comm, size)
MPI_COMM_RANK(comm, rank)

CHANGED ROUTINE (communicators instead of groups):  

MPI_COMM_LOCAL_TRANSLATE_RANKS (comm_a, n, ranks_a, comm_b, ranks_b)

Local Destructor

NEW ROUTINE:  

MPI_COMM_FREE(comm)
    IN    comm            communicator object handle previously defined

  This operation frees communicator handle comm.

  DISCUSS:  all communicator construction at the basic level is currently done 
            with collective operations.  Should this be MPI_COMM_COLL_FREE()??  

Collective Communicator Constructors

NEW ROUTINE (replaces MPI_COMM_MAKE()):  

MPI_COMM_COLL_DUP(sync_comm, comm_to_dup, n, new_comms) 
    IN    sync_comm       communicator specifying participants in this call
    IN    comm_to_dup     communicator object handle to be duplicated
    IN    n               number of new communicators to create
    OUT   new_comms       array of n new communicators

  MPI_COMM_COLL_DUP() is a collective version of MPI_COMM_LOCAL_DUP().  
  MPI_COMM_COLL_DUP() creates n new communicators and stores them in array 
  new_comms.  Each new communicator has a context unique among all processes 
  in sync_comm.  Each new communicator is otherwise identical to comm_to_dup.  
  comm_to_dup will often be a subset of sync_comm.  It is not erroneous if 
  comm_to_dup is not a subset of sync_comm.  (This functionality is primarily 
  for the "expert" user.)  

  Tony's discussion of statically initialized, single-invocation collective 
  operations (that lock out other threads) applies to this routine when 
  comm_to_dup is not a subset of sync_comm.  I have replaced "sync_group" with
  "sync_comm" assuming that the context associated with sync_comm can be 
  ignored (if the implementation requires it).  

  Tony:  will the next two routines also need to be single-invocation 
         collective operations (with an additional "sync_comm" argument)?  

NEW ROUTINE (replaces MPI_COLL_SUBGROUP()):  

MPI_COMM_COLL_SUBSET(comm, membership_key, new_comm)

  Same as MPI_COLL_SUBGROUP() in the current draft except that new 
  communicator new_comm is returned (instead of new group new_group).  
  new_comm has a new context.  

NEW ROUTINE (replaces MPI_COLL_GROUP_PERMUTE()):  

MPI_COMM_COLL_PERMUTE(comm, new_rank, new_comm)

  Same as MPI_COLL_GROUP_PERMUTE() in the current draft except that new 
  communicator new_comm is returned (instead of new group new_group).  
  new_comm has a new context.  

  DISCUSS:  is there enough functionality for communicator construction at the 
            basic level??  

"Inter-communication"

NEW ROUTINE (combines MPI_COMM_PUBLISH_SUBSCRIBE() and MPI_COMM_JOIN()):  

MPI_COMM_FUSE(comm_local, label_local, label_remote, order, comm_joined)
    IN    comm_local      local communicator
    IN    label_local     string label for local communicator
    IN    label_remote    string label for remote communicator
    IN    order           if comm_local is first, then MPI_TRUE, else MPI_FALSE
    OUT   comm_joined     merged communicator

  MPI_COMM_FUSE() (name thanks to Jim :-) is a combination of routines 
  MPI_COMM_PUBLISH_SUBSCRIBE() and MPI_COMM_JOIN() from the current draft.  
  This routine creates the merged communicator comm_joined given local 
  communicator comm_local, string label label_local, string label 
  label_remote, and int order.  String label_local is used as a unique name 
  for local_comm.  String label_remote is used as a unique name for the remote 
  communicator that will be merged with comm_local.  The routine synchronizes 
  over the processes in both communicators, and produces a new, merged 
  communicator (comm_joined), whose processes are the ordered union of the two 
  underlying process groups.  The processes in comm_local will be first in 
  comm_joined iff order == MPI_TRUE.  Otherwise, the processes in the remote 
  communicator will be first.    The new communicator has a unique context of 
  communication.  

  This routine is called in pairs by all of the processes belonging to each 
  communicator.  Labels and order must correspond.  For example:  

  /* comm A call */
  MPI_COMM_FUSE(comm_A, "COMM_A", "COMM_B", MPI_TRUE, comm_A_B);
  ...

  /* comm B call */
  MPI_COMM_FUSE(comm_B, "COMM_B", "COMM_A", MPI_FALSE, comm_A_B);
  ...

  After these calls have completed, comm_A_B will consist of the processes in 
  comm_A followed by the processes in comm_B and a new context unique across 
  all processors in comm_A and comm_B.  

  (I was going to call this MPI_COMM_COLL_FUSE() but MPI_COMM_FUSE() seems 
  much more appropriate!  :-)

"EXPERT" LEVEL

GROUPS

Local Operations
MPI_GROUP_SIZE(group, size)
MPI_GROUP_RANK(group, rank)
MPI_TRANSLATE_RANKS (group_a, n, ranks_a, group_b, ranks_b)
MPI_GROUP_FLATTEN(group, max_length, buffer, actual_length)

Local Group Constructors
MPI_GROUP_UNFLATTEN(max_length, buffer, group)
MPI_LOCAL_SUBGROUP(group, n, ranks, new_group)
MPI_LOCAL_SUBGROUP_RANGES(group, n, ranges, new_group)
MPI_LOCAL_GROUP_UNION(group1, group2, group_out) 
MPI_LOCAL_GROUP_INTERSECT(group1, group2, group_out) 
MPI_LOCAL_GROUP_DIFFERENCE(group1, group2, group_out)
MPI_GROUP_FREE(group)
MPI_GROUP_DUP(group, new_group)

CONTEXTS

Local Operations
MPI_CONTEXTS_RESERVE(n, contexts)
MPI_CONTEXTS_FREE(n, contexts)

Collective Operations
MPI_CONTEXTS_ALLOC(group, n, contexts)
MPI_CONTEXTS_JOIN(local_comm, remote_comm_send, remote_comm_recv, n, contexts)

COMMUNICATORS

Local Communicator Operations
MPI_COMM_FLATTEN(comm, max_length, buffer, actual_length)

Local Constructors
MPI_COMM_UNFLATTEN(max_length, buffer, comm)
MPI_COMM_BIND(group, context, comm_new)
MPI_COMM_UNBIND(comm)
MPI_COMM_GROUP(comm, group)
MPI_COMM_CONTEXT(comm, context)
MPI_COMM_MERGE(comm, comm_remote, comm_send, comm_recv)

INTER-COMMUNICATION

MPI_COMM_PUBLISH(comm, label, persistence_flag)
MPI_COMM_UNPUBLISH(label)
MPI_COMM_SUBSCRIBE(my_comm, label, comm)
MPI_COMM_PUBLISH_SUBSCRIBE(comm_A, label_A, label_B, send_to_B, recv_from_B)
MPI_COMM_JOIN(local_comm, remote_comm_send, remote_comm_recv, order, 
              joined_comm)

OK, blast away!!