Subject: more "multi-level" "basic" examples...

From: [email-masked] (Tom Henderson)
Date: Mon, 19 Jul 93 18:34:38 MDT

Tony & Jim,

As promised, here are the (non-trivial) examples from the current context 
draft recoded using the "basic" level of my "multi-level" proposal.  The 
examples are identified by section in the current draft.  Examples in sections 
3.11.1 and 3.11.2 are unchanged.  

Some conclusions are:  

1)  The "basic" routines can accomplish everything in the examples (as I 
    understand them-- Tony??).  

2)  Some of the "expert" routines clearly are better for getting performance 
    out of libraries.  

3)  We could use a few more examples...  :-)   Ideas?  

Comments?

Enjoy,

Tom

-----------------------------------------------------------------------------

The "basic" level currently contains the following EIGHT routines (yeah, I 
can't count :-):  

MPI_COMM_SIZE(comm, size)
MPI_COMM_RANK(comm, rank)
MPI_COMM_LOCAL_TRANSLATE_RANKS (comm_a, n, ranks_a, comm_b, ranks_b)
MPI_COMM_FREE(comm)
MPI_COMM_COLL_DUP(sync_comm, comm_to_dup, n, new_comms) 
MPI_COMM_COLL_SUBSET(comm, membership_key, new_comm)
MPI_COMM_COLL_PERMUTE(comm, new_rank, new_comm)
MPI_COMM_FUSE(comm_local, label_local, label_remote, order, comm_joined)

(With the caveat that sync_comm might need to appear in a few more 
places...  We'll see when I understand "single-invocation library" better.)  

3.11.3  (Approximate) Current Practice #3

int me;
int membershipKey;    /* membership key for MPI_COMM_COLL_SUBSET() */
void *commslave;      /* communicator for "slave" sub-group */
...

/* Initialize, etc. */
MPI_INIT();
MPI_COMM_RANK(MPI_COMM_ALL, &me);    /* local */

/* Set up "slave" sub-group. */
if (me == 0)
    membershipKey = 0;    /* process 0 in MPI_COMM_ALL */
else
    membershipKey = 1;    /* "slave" processes */
MPI_COMM_COLL_SUBSET(MPI_COMM_ALL, membershipKey, &commslave);

if (me != 0)
    {
    /* compute on slave */
    ...
    MPI_REDUCE(commslave, ...);
    ...
    }
/* zero falls through immediately to this reduce, others do later... */
MPI_REDUCE(MPI_COMM_ALL, ...);

3.11.4  Example #4

#define TAG_ARBITRARY 12345
#define SOME_COUNT       50
...

int me;
int membershipKey;              /* membership key for MPI_COMM_COLL_SUBSET() */
void *pt2pt_comm, *coll_comm;   /* communicators for sub-group */
...

/* Initialize, etc. */
MPI_INIT();
MPI_COMM_RANK(MPI_COMM_ALL, &me);    /* local */

/* Make communicators for sub-group. */
if ((me == 2) || (me == 4) || (me == 6) || (me == 8))
    membershipKey = 0;    /* processes in sub-group */
else
    membershipKey = 1;    /* other processes */
MPI_COMM_COLL_SUBSET(MPI_COMM_ALL, membershipKey, &pt2pt_comm);

if (membershipKey == 0)
    {
    MPI_COMM_COLL_DUP(pt2pt_comm, pt2pt_comm, 1, coll_comm);

    /* asynchronous receive: */
    MPI_IRECV(..., MPI_SRC_ANY, TAG_ARBITRARY, pt2pt_comm);

    /* collective communication */
    for (i=0; i<SOME_COUNT; i++)
        {
        MPI_REDUCE(coll_comm, ...);
        }
    ...
    }

3.11.5  Library Example #1

The main program:  

int done = 0;
user_lib_t *libh_a, *libh_b;
void *dataset1, *dataset2;
...
MPI_INIT();
...
init_user_lib(MPI_COMM_ALL, &libh_a);
init_user_lib(MPI_COMM_ALL, &libh_b);
...
user_start_op(libh_a, dataset1);
user_start_op(libh_b, dataset2);
...
while (!done)
    {
    /* work */
    ...
    MPI_REDUCE(MPI_COMM_ALL, ...);
    ...
    /* see if done */
    ...
    }
user_end_op(libh_a);
user_end_op(libh_b);

The user library initialization code:  

void init_user_lib(void *comm, user_lib_t **handle)
    {
    user_lib_t *save;

    user_lib_initsave(&save);  /* local */
    MPI_COMM_COLL_DUP(comm, comm, 1, save -> comm);

    /* other inits */
    ...

    *handle = save;
    }

Notice that the communicator comm passed to the library IS needed to allocate 
new communicators.  

User start-up code:  

void user_start_op(user_lib_t *handle, void *data)
    {
    user_lib_state *state;

    state = handle -> state;
    MPI_IRECV(save -> comm, ..., data, ... &(state -> irecv_handle));
    MPI_ISEND(save -> comm, ..., data, ... &(state -> isend_handle));
    }

User clean-up code:  

void user_end_op(user_lib_t *handle, void *data)
    {
    MPI_WAIT(save -> state -> isend_handle);
    MPI_WAIT(save -> state -> irecv_handle);
    }

3.11.6  Library Example #2

The main program:  

int ma, mb, me;
int membershipKeyA, membershipKeyB;       /* membership keys */
void *comm_a, comm_b;                     /* communicators for sub-groups */
...
/* Initialize, etc. */
MPI_INIT();
MPI_COMM_RANK(MPI_COMM_ALL, &me);    /* local */
...

/* Make communicators for sub-groups. */
if ((me == 0) || (me == 1))
    membershipKeyA = 0;    /* processes in sub-group A */
else
    membershipKeyA = 1;    /* other processes */
MPI_COMM_COLL_SUBSET(MPI_COMM_ALL, membershipKey, &comm_a);
if ((me == 0) || (me == 2) || (me == 3))
    membershipKeyB = 0;    /* processes in sub-group B */
else
    membershipKeyB = 1;    /* other processes */
MPI_COMM_COLL_SUBSET(MPI_COMM_ALL, membershipKey, &comm_b);

if (membershipKeyA == 0)
    lib_call(comm_a);
if (membershipKeyB == 0)
    {
    lib_call(comm_b);
    lib_call(comm_b);
    }

The library:  

void lib_call(void *comm)
    {
    int me, done = 0;

    MPI_COMM_RANK(comm, &me);
    if (me == 0)
        {
        while (!done)
            {
            MPI_RECV(..., comm, MPI_SRC_ANY);
            ...
            }
        }
    else
        {
        /* work */
        MPI_SEND(..., comm, 0);
        ....
        }
    MPI_SYNC(comm);  /* include/no safety for safety/no safety */
    }