Subject: Another intergroup example

From: [email-masked] (David Walker)
Date: Fri, 18 Jun 1993 15:37:59 -0400

Below is another example MPI code concernec with intergroup 
communication. This is a routine that partitions a group into two
subgroups and returns a communicator for communicating between them.
Here I'm following the suggestion from Lyndon of viewing a communicator
as consisting of 4 fields, namely:

	local_group,
        local_context,
        remote_group,
        remote_context

For intragroup communication, remote_group=local_group and 
remote_context=local_context.

When communicating between groups, the sender gives the destination 
relative to the remote group, and the local context is used associated
with the message. The receiver uses a source relative to the remote group,
and assumes the message is associated with the remote context.
I invented a new convenience routine, mpi_merge_comm, to
form the intergroup communicator.

David

/* PROGRAM 4
   This subroutine partitions a group into 2 subgroups, and returns a
   communicator for communicating between them. key must equal either 
   0 or 1 on entry. */

intergroup (key, index, group, comm)
{  
   int     key, index;
   group_t group;
   comm_t  comm;
   group_t lgroup;
   int     errno, rank;
   comm_t  lcomm, rcomm;

   /* partition the group, and find rank in subgroup */
   errno = mpi_partition_group (group, key, index, lgroup);
   errno = mpi_rank (lgroup, rank);

   /* form a communicator for local group, get communicator for remote group */
   errno = mpi_safemake_comm (lgroup, lgroup, lcomm);
   if (key==0) {
      if (rank==0) errno = mpi_publish (lcomm, "G0");
      errno = mpi_subscribe (rcomm, "G1");
   }
   elseif (key==1) {
      if (rank==0) errno = mpi_publish (lcomm, "G1");
      errno = mpi_subscribe (rcomm, "G0");
   }
   errno = mpi_merge_comm (lcomm, rcomm, comm);

   /* free local and remote communicators */
   errno = mpi_free (lcomm);
   errno = mpi_free (rcomm);
}

---

From: L J Clarke <[email-masked]>
Date: Sat, 19 Jun 93 20:30:33 BST

Hi David

> 
> Below is another example MPI code concerned with intergroup 

I think a little explanation is due on my part.  In old proposal x there
was a communicator kind of object which had one context and a local
group and a remote group.  More recently I have suggested a communicator
object with the fields you describe above.  The reason for the change in
suggestion is simple and floows directly from difference between context
allocation which was intended in proposal x and that described in the
current subcommittee working document. 

In proposal x members of the local group could allocate a context which
would be safe for members of the local group - *and* members of the
remote group - to use for receive (and the other way around where the
remote group allocates the context).  Therefore you only needed one
context for the communicator object which either group can allocate and
both local and remote groups can use for receive (ergo used for send in
both local and remote groups).

In the working document members of the local group can allocate a
context would be safe for member of the local group - *but not* members
of the remote group - to use for receive.  Therefore the communicator
object needs to contain two contexts.  One of these is allocated by the
local group and is used for receive in the local group (ergo used for
send in the remote group).  The other is allocated by the remote group
and is used for receive in the remote group (ergo used for send in the
local group). 

> form the intergroup communicator.

I thought about seeing what its like to introduce such a routine, since
it allows us to make use of the publish/subscribe mechanism which
Tony/Mark suggest.  This is nicely demonstrated by part of your example. 

Lets look however at the last part of the example.

>    errno = mpi_free (rcomm);

We should not make use of "lcomm" and "rcomm" after mpi_merge_comm since
they must contain the same contexts as "comm" and this can lead to
messages getting received in the wrong peice of code.  Therefore we
should immediately free them as the example does.  In order to do this
we have to be careful in our definition of what mpi_free(comm) actually
frees, remembering that a communicator is a composite of a group and a
context.  

I expect we would agree that mpi_free(comm) does not free the group
associated with comm.  The question is whether it does free the context
associated with comm.  If the answer is YES, then the "comm"
communicator is no longer safe as the contexts could become used for
other purposes.  If the answer is NO then we need to provide a mechanism
for the user to explicitly free the context, but that would be ugly
anyway if the user never explicitly allocated the context in the first
place.  There is another possible answer, which is YES in general but NO
not in this case because MPI recognises that the context is being used
in the communicator "comm" also.  This means that MPI keeps count of
references to contexts, which is going to be quite expensive if context
is just an integer as opposed to a handle to an opaque object which can
contain a reference count. 

Perhaps "mpi_merge_comm(acomm, bcomm, comm)" should "free" acomm and
bcomm.  I put "free" in quotes because I mean to free the communicator
objects but not the contexts therein, the contexts being used in the new
communicator "comm".  Thus any use of acomm and bcomm after the call to
mpi_merge_comm would be an error.  This saves the user from having to do
the mpi_free, and avoids some nasty questions in the definition of
mpi_free. 

I include the example 4 amended in this fashion, and another example
which just shows how two groups can "hook-up" just knowing a name for
the local "group" and a name for the other "group". Then I give example
4 written to use example 5.

Best Wishes
Lyndon

/* PROGRAM 4
 * This subroutine partitions a group into 2 subgroups, and returns a
 *  communicator for communicating between them. key must equal either 
 *  0 or 1 on entry. 
 */

   comm_t  *comm;
{
   group_t lgroup;
   int     errno, rank;
   comm_t  lcomm, rcomm;

   errno = mpi_merge_comm (lcomm, rcomm, *comm);

}

/* PROGRAM 5
 *
 * This example routine shows how the user can generate a communicator
 * for intergroup communication given a group of which the callee is
 * a member, a name for the local side and a name for the remote side.
 */

make_inter_thing(comm_t *comm, group_t group, char *this, char *that)
{
  comm_t lcomm, rcomm;
  int    errno, rank;

  errno = mpi_safemake_comm(group, lcomm);

  errno = mpi_rank(group, &rank);
  if (rank == 0)
    errno = mpi_publish(lcomm, this);

  errno = mpi_subscribe(rcomm, that);

  errno = mpi_merge_comm(lcomm, rcomm, *comm);

}

/* PROGRAM 4B
 *
 * This is program 4 using the routine of example 5.
 */

intergroup (int key, int index, group_t group, comm_t *comm)
{  
   group_t lgroup;
   int     errno;

   /* partition the group */
   errno = mpi_partition_group (group, key, index, lgroup);

   /* call make_inter_thing with appropriate names */
   make_inter_thing(&comm, group, ((key == 0) ? "G0" : "G1"),
                                 ((key == 0) ? "G0" : "G1"));
   
}

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

---

From: David W. Walker <[email-masked]>
Date: Mon, 21 Jun 93 08:47:33 -0500

Lyndon,

Thanks for your comments on the latest intergroup example, and for pointing out the
potential problems of freeing a communicator. The view of contexts that we have
lately been discussing comes down to saying that they are unidirectional, rather than
bidirectional. I haven't had time to look in detail at your examples as I'm about to
leave. I'll look at them on the plane.

David