Subject: More on "multi-level" and "group fusion"

From: [email-masked] (Tom Henderson)
Date: Mon, 19 Jul 93 17:15:41 MDT

Tony & Jim, 

I've been thinking about "group fusion"...  As usual, I think best with 
examples.  I decided to re-code the extremely popular "ISAR" example using 
Jim's "group fusion" and see what happens...  I've restricted myself to using 
routines from the "basic" level of my "multi-level" proposal (which includes 
"fusion").  

Some sections of the original example have been deleted (since you've already 
got them).  

The biggest changes have been:  

1)    Creation of the communicators is much simpler (expected, since this is 
      basically a "publish-subscribe" mechanism).  

2)    The user functions that send data/parameters between the three process 
      groups are now more complicated.  Before, I could pass a single 
      communicator into these functions.  Each function could then find out 
      who to communicate with directly using the single communicator.  Now, I 
      am passing the "fused" communicator, the size of the local sub-group 
      communicator, and the membership key.  The functions use the size and 
      the key to determine which parts of the fused communicator correspond to 
      the "local" and "remote" sub-groups.  (Alternatively, the "local" 
      communicator could be passed in instead of the size and key.  The 
      MPI_COMM_TRANSLATE_RANKS() routine would then be used to identify the 
      "local" sub-group.  The "remote" sub-group would be the remaining 
      processes in the fused communicator.)  

My (not very well thought-out) conclusions after this exercise are:  

1)    The "basic" level provides enough flexibility to make this example 
      work OK.  I will attempt to re-do the examples in the current draft 
      using the "multi-level" proposal and see what happens there...  

2)    Not having explicit groups and contexts running around is a definite 
      plus.  Lack of local operations doesn't hurt in this example.  

3)    "Fused" groups are a bit clumsy.  It is intuitive (to me) to be able to 
      send to another communicator, directly addressing members of that 
      communicator as [0,...,(N-1)].  With a "fused" group, I need to keep 
      track of which "half" of the group I want to communicate with.  This 
      feels a bit clumsy, but it certainly allows me to do what I want.  

By the way, if this multi-level idea is no good, say so and I'll attempt to 
help out with some other problem.  I've had no feedback yet, positive or 
negative.  

Tom Henderson
NOAA Forecast Systems Laboratory

------------------------------ BEGIN "PSEUDO" CODE ---------------------------

/* #includes */
#include <mpi.h>       /* :-) */
#include <fftLib.h>    /* For FFT library. */

/* #defines */

/* Membership keys for sub-group creation. */
#define SEARCH_GROUP_KEY      0
#define COMPENSATE_GROUP_KEY  1
#define DISPLAY_GROUP_KEY     2

/* Publication strings. */
#define SEARCH_COMM_STRING     "SEARCH_COMM"
#define COMPENSATE_COMM_STRING "COMPENSATE_COMM"
#define DISPLAY_COMM_STRING    "DISPLAY_COMM"

/* typedefs */
...

main()
    {
    comm myComm, search_compensateComm, compensate_displayComm;
    float **radarData;
    char **image;
    isarParameters isarParams;
    decompParameters searchDecomp, compensateDecomp, displayDecomp;
    int membershipKey, myRankInAll;
    int allCommSize, searchCommSize, compensateCommSize, displayCommSize;

/* Generic setup. */
    MPI_INIT();
    MPI_COMM_SIZE(MPI_COMM_ALL, &allCommSize);
    MPI_COMM_RANK(MPI_COMM_ALL, &myRankInAll);

/* Initialize library called inside function compensateData(). */
    INIT_FFT_LIB(...);

/* Other setup stuff (malloc() 2D arrays, etc...) */
    ...

/* Build "sub-groups". */
    /* "User" function... */
    getCommSizes(myRankInAll, allCommSize, &searchCommSize, 
                 &compensateCommSize, &displayCommSize, &membershipKey, ...);
    MPI_COMM_COLL_SUBSET(MPI_COMM_ALL, membershipKey, myComm);

/* Build "fused" communicators search_compensateComm and */
/* compensate_displayComm.  Search group is first in */
/* search_compensateComm, compensate group is first in */
/* compensate_displayComm. */
    if (membershipKey == SEARCH_GROUP_KEY)
        {
        MPI_COMM_FUSE(myComm, SEARCH_COMM_STRING, COMPENSATE_COMM_STRING, 
                      MPI_TRUE, search_compensateComm);
        }
    else if (membershipKey == COMPENSATE_GROUP_KEY)
        {
        MPI_COMM_FUSE(myComm, COMPENSATE_COMM_STRING, SEARCH_COMM_STRING, 
                      MPI_FALSE, search_compensateComm);
        /* Be careful to make calls in the right order! */
        MPI_COMM_FUSE(myComm, COMPENSATE_COMM_STRING, DISPLAY_COMM_STRING, 
                      MPI_TRUE, compensate_displayComm);
        }
    else if (membershipKey == DISPLAY_GROUP_KEY)
        {
        MPI_COMM_FUSE(myComm, DISPLAY_COMM_STRING, COMPENSATE_COMM_STRING, 
                      MPI_FALSE, compensate_displayComm);
        }
    else
        {
        handleError(...);
        }

/* Data decomposition set-up code for all groups... */
    makeDecompParams(searchCommSize, searchDecomp, ...);
    makeDecompParams(compensateCommSize, compensateDecomp, ...);
    makeDecompParams(displayCommSize, displayDecomp, ...);

/* Other stuff... */
    ...

/* MAIN LOOPS for each sub-group... */
/* NOTE changes in arguments to user functions sendCurrentData(), */
/* sendMotion(), recvCurrentData(), and recvMotion.  These functions */
/* now require the "fused" communicator (search_compensateComm or */
/* compensate_displayComm), size of the "local" sub-group communicator, */
/* and the membership key. */
    /* MAIN LOOP for "search" group... */
    if (membershipKey == SEARCH_GROUP_KEY)
        {
        searchDone = 0;
        while (searchDone == 0)
            {
            ...
            addNewData(myComm, radarData, searchDecomp, ...);
            findNewMotion(myComm, radarData, isarParams, searchDecomp, ...);
            sendCurrentData(search_compensateComm, searchCommSize, 
                            membershipKey, radarData, 
                            searchDecomp, compensateDecomp);
            sendMotion(search_compensateComm, searchCommSize, 
                       membershipKey, isarParams);
            ...
            }  /* end of search group while */
        }  /* end of search group if */
    /* MAIN LOOP for "compensate" group... */
    else if (membershipKey == COMPENSATE_GROUP_KEY)
        {
        compensateDone = 0;
        while (compensateDone == 0)
            {
            ...
            recvCurrentData(search_compensateComm, compensateCommSize, 
                            membershipKey, radarData, 
                            compensateDecomp, searchDecomp);
            recvMotion(search_compensateComm, compensateCommSize, 
                       membershipKey, isarParams);
            compensateData(myComm, radarData, isarParams, 
                           compensateDecomp, ...);
            sendCurrentData(compensate_displayComm, compensateCommSize, 
                            membershipKey, radarData, 
                            compensateDecomp, displayDecomp);
            ...
            }  /* end of compensate group while */
        }  /* end of compensate group if */
    /* MAIN LOOP for "display" group... */
    else if (membershipKey == DISPLAY_GROUP_KEY)
        {
        displayDone = 0;
        while (displayDone == 0)
            {
            ...
            recvCurrentData(compensate_displayComm, displayCommSize, 
                            membershipKey, radarData, 
                            displayDecomp, compensateDecomp);
            scaleThresholdCenter(myComm, radarData, image, 
                                 displayDecomp, ...);
            displayImage(myComm, image, displayDecomp);
            ...
            }  /* end of display group while */
        }  /* end of display group if */
     else
        {
        handleError(...);
        }

/* Cleanup... */
    ...

    }  /* end of main() */