Subject: Re: Context and safety

From: [email-masked] (Tom Henderson)
Date: Tue, 27 Jul 93 17:33:37 MDT

Jim, 

I think your proposal has the following main points (please correct whatever 
I've got wrong):  

  1)  "Context" is not directly visible to the user.  

  2)  Multiple contexts may be attached to a group (communicator).  

  3)  Contexts are indirectly accessed by a key pair consisting of a 
      function address and an integer "instance number".  

  4)  The number of contexts attached to a group can be increased using a 
      collective operation.  (I'm not absolutely clear on this.  At one point 
      you say we should be able to "locally pre-allocate a set of contexts for 
      a given group".  Later you say that the routine that does this "is a 
      group synchronization"-- which sounds "collective".)  

  5)  A context can be "accessed" using an operation that is local UNLESS 
      there are not enough contexts and OP_LOOKUPCREATE is specified (in 
      which case the operation is collective as in #4).  

  6)  Assumption:  the "collective" operation in #4 must be called with 
      matching parameters in all processes in the specified group.  

A few questions:  

  A)  Why use a function address for the first key in the key pair?  It seems 
      to me that you're using a function address only because it is something 
      that a user cannot modify that is guaranteed to be unique within a 
      process.  It is also globally visible (within a process).  Are there any 
      other reasons for using a function pointer here?  Why not use a local 
      GET_KEY() routine that returns some opaque thing (key_t? :-) ?  

  B)  You say that mpi_groupsReserve() is a "group synchronization".  I assume 
      that if I call mpi_lookupGroup(..., OP_LOOKUPCREATE) and there are no 
      more contexts, then mpi_groupsReserve() must be called inside 
      mpi_lookupGroup() to allocate new contexts.  This means that 
      mpi_lookupGroup(..., OP_LOOKUPCREATE) may or may not be a collective 
      operation.  As a user, I'd like to be able to find this out before I 
      make the call.  How about a local inquiry routine that returns the 
      number of available contexts attached to a group?  

  C)  In a heterogeneous implementation, comparison of the function pointer 
      key in a call to mpi_groupsReserve() won't be helpful.  This would also 
      be true in a MIMD program.  Specifically, the following error cannot be 
      detected and will not deadlock a machine:  

      MPI_COMM_RANK(MPI_COMM_ALL, &me);
      if (me == 0)
          mpi_groupsReserve(MPI_COMM_ALL, myFuncA(), 1);
      else
          mpi_groupsReserve(MPI_COMM_ALL, myFuncB(), 1);

      Is this worse/better than being unable to detect bindings of multiple 
      groups to a single integer context?  (By the way, bindings of multiple 
      groups to a single OPAQUE context could be detected.)  

  D)  (Just a dumb question :-)  What happens if I create a new group G from 
      MPI_GROUP_ALL (using mpi_lookupGroup()), reserve a bunch of contexts for 
      G, create a group H from G, and then "delete" G?  I assume it is still 
      safe to use group H?  

      mpi_groupsReserve(MPI_COMM_ALL, myFuncA(), 10);
      G = mpi_lookupGroup(MPI_COMM_ALL, myFuncA(), 5, OP_LOOKUP);
      mpi_groupsReserve(G, myFuncB(), 3);
      H = mpi_lookupGroup(G, myFuncB(), 1, OP_LOOKUP);
      (void)mpi_lookupGroup(MPI_COMM_ALL, myFuncA(), 5, OP_DELETE);
      /* OK to use group H... */

Typos?  

> void user_end_op(MPI_GROUP g, userLibHandle key)
> {
>     MPI_GROUP myGroup = mpi_lookupGroup(g, 
> 	                                (void(*)(void))init_user_lib,
> 	                                key,
> 		                        OP_LOOKUP);
>     
>     if (!barrierGroup)
           ^^^^^^^^^^^^ myGroup ??
>     {
>         /* 
>          * Fatal error, attempt to end user op without starting it
>          */
>     } else
>     {
> 	/* 
>          * Wait for all communications in myGroup to complete
>          * The proposed waitAll( GROUP ) would be useful here,
> 	 * otherwise we need the return tag from startBarrier
$$$                                              ^^^^^^^^^^^^ user_start_op() ?
>          * to contain the comms handles of all the non-blocking ops we
>          * started so that we can wait for them to complete.
>          */
>     }
> 
>     /* Free the group we used. */  
>     mpi_lookupGroup (void(*)(void))init_user_lib,
> 	                                key,
> 		                        OP_DELETE);
> }

> FEEDBACK PLEASE !!!!!

Hope this is useful...  

Tom

---

From: James Cownie <[email-masked]>
Date: Wed, 28 Jul 1993 10:53:24 +0100

Tom,

>   1)  "Context" is not directly visible to the user.  
Correct
>   2)  Multiple contexts may be attached to a group (communicator).  
Correct
>       function address and an integer "instance number".  
Correct
>       group synchronization"-- which sounds "collective".)  
Correct, you have detected the crucial typo, as you say it should read
COLLECTIVE not LOCAL. Score one beer !
>       which case the operation is collective as in #4).  
Correct
>       matching parameters in all processes in the specified group.  
Correct

>       GET_KEY() routine that returns some opaque thing (key_t? :-) ?  
You are right, it is guaranteed unique in a process, and visible
throughout the process. It is also cheap to access. I prefer this to
allocating unique keys because this is cheaper and I don't have to
remember the key in a data structure. Suppose I want to implement the
MPI collective operations, then I don't have any arguments passed into
them to give me the key, I have to remember it locally. If I use a
function pointer, then it is immediately visible without any effort on
my part.

>       number of available contexts attached to a group?  

You can already achieve approximately this effect by using
mpi_lookupGroup(..., OP_LOOKUPCREATENB) which will fail if the
LOOKUPCREATE would have blocked (cf EWOULDBLOCK). I have no strong
objection to being able to query the number of "free contexts" on the
group. The only problem arises in a multi-threaded environment where
some other thread could consume contexts between you inquiring and
using them. I really prefer the explicit allocation which is there so
that you can ensure that you won't block.

>           mpi_groupsReserve(MPI_COMM_ALL, myFuncA(), 1);
            /* probably don't want these ()        ^^ */
>           mpi_groupsReserve(MPI_COMM_ALL, myFuncB(), 1);
            /* probably don't want these ()        ^^ */
>       groups to a single OPAQUE context could be detected.)  

I could argue that this isn't an error, it all depends on how the
new group is used. (The allocation order is clearly specified, the
choice of key to use to find the context is a local decision, provided
the usage is consistent this code could work !). [Imagine for instance
a MIMD non SPMD code in which a group was constructed for some
collextive operation but not all of the processes in the group
contained the same set of functions. Then this would actually be
useful.] I agree that it will often be a problem, however. If we do
want to be able to check this, then my best suggestion is to add an
additional string argument which can be checked in a debugging version
of the library or ignored in a kamikaze speed first version. 

This is rather like the discussion about a tag in a barrier. That could
also be useful to find exactly analogous bugs. We voted it out on the
grounds that all it was doing was changing the behaviour of codes
which were already erroneous (and therefore had unspecified behaviour).
[I'd still quite like to see it there, solely for this debugging purpose, but I
think that argument is lost...]

>       /* OK to use group H... */
Yes, I think this is fine. However I'd disallow use of both G
and H if you deleted the MPI_COMM_ALL.

> Typos?  
Yes, I took this example from one which was explicitly implementing
the famous MPI -1 non-blocking barrier, hence barrierGroup. For
comparability I adopted Tony's names but then failed to make all the
changes. Sorry. (but not another beer !)

> Hope this is useful...  
Definitely, thanks.

> One other question I forgot to include...  You mention waitAll(),  
> 
> >          * The proposed waitAll( GROUP ) would be useful here
> 
> How would waitAll() work with nested groups (as in my "dumb question")?  
> 
>       mpi_groupsReserve(G, myFuncB(), 4);
>       H = mpi_lookupGroup(G, myFuncB(), 2, OP_LOOKUP);
>       waitAll(G);
> 
> Would waitAll() wait for all communication in H?  Or just communication in 
> the first context of H (which I assume is context "2" in G)?  

I don't think that waitAll(GROUP) exists, so this is a moot point. I'd
certainly expect it only to wait on the actual group it is passed, so
in this case (which I now find confusing, should the waitAll be on H ?) 
[attempt to get back to parity on the beers !], as written it would
wait for completion of all communications on the G group.
Communications in H or MPI_COMM_ALL would be irrelevant.

James Cownie 
Meiko Limited			Meiko Inc.
650 Aztec West			Reservoir Place
Bristol BS12 4SD		1601 Trapelo Road
England				Waltham
				MA 02154

Phone : +44 454 616171		+1 617 890 7676
FAX   : +44 454 618188		+1 617 890 5042
E-Mail: [email-masked]   or    [email-masked]