Subject: Next MPI meeting

From: [email-masked] (David Walker)
Date: Thu, 10 Jun 1993 08:35:55 -0400

Unfortunately I won't be at the next MPI meeting, which is why I'm keen to 
understand the current context ideas and proposal(s) now. I've not had much
feedback on the examples I sent out a couple of days ago. I'm particularly
keen to get a correct version of the convolution example. I think examples
are important if we are to make others understand a fairly complex proposal.
They also help us understand what is really needed.

Here's a question that I've asked myself, and which I'm sure others will
ask. Are naked contexts really necessary? Communicators provide the scope
of a communication operation. Why would an application writer use 
mpi_alloc_context and mpi_make_comm, rather than mpi_safemake_comm?

David

---

From: L J Clarke <[email-masked]>
Date: Thu, 10 Jun 93 17:03:00 BST

Dear David

>  I think examples

I agree. I fear that we may need complex examples, and people will have
time to study them.

> They also help us understand what is really needed.

I agree again, and have the same fear - i.e.  that whereas the examples
can be conceptually simple the example code cannot be.

> mpi_alloc_context and mpi_make_comm, rather than mpi_safemake_comm?

Naked groups allow the user to perform group manipulations on groups. 
This is conceptually cleaner than doing it on communicators, in my
humble opinion. 

The context-free :-) communicator constructors synchronise the processes
invloved in the communicator construction.  The context-oriented
constructors do not.  Naked contexts allow preallocation of contexts and
avoidance of synchronisation for communicator construction. 

I think that a modification of your example program in line with a
comment about destination rank addressing will reveal another use for
naked context.  I'll have to redefine publish/subscribe a little, but I
dont see this as a problem as the example redefines them relative to my
understanding of the intentions of Tony and Mark (Sears).  I'll send
that modification after this message. 

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Tue May 11 21:56:56 1993
Date: Tue, 11 May 93 20:52:44 MDT
From: [email-masked] (Mark P. Sears)
To: [email-masked]
Subject: Proposal VIII -- response to Riks probs.
Status: R

The following is my response to Rik Littlefields problem
sets. I will bring copies to the meeting.

Mark Sears
Sandia National Laboratories

Comments on Rik Littlefields problems:

1) (describe a circular shift routine)

Here I give a contiguous byte version only. Extensions to other
data types and to non contigous buffers is straightforward. Error handling
is left out, and I use an aggressive style of message passing (send before
receive). Note that tag must be provided in addition to context.

// MPI_CSHIFT_B -- shift a contiguous buffer around a group in circular order
//
//   limitations:
//      uses aggressive style of message passing.
//      no check for len being same on all processes.
//
//   assumptions:
//      MPI_MYNODE()  returns process id
//      MOD(a,b) = a modulo b
//      SEND(buf, len, dest, tag, context) sends buffer to destination.
//      RECV(buf, len, src, tag, context)  receives buffer from source.
//
void  MPI_CSHIFT_B ( char *in, char *out, int len, int shift,
     MPIGroup group, int context, int tag)
    {
     int myrank;       // rank of this process in the group
     int left, right;  // process ids for sending and receiving

// exit routine if this process is not a group member:

     if(!MPI_groupiselement(group, MPI_MYNODE())) return;

// get my rank in group and compute left and right neighbors

     myrank = MPI_grouprank(group, MPI_MYNODE());
     
     left = MPI_groupelement(group,
          MOD((myrank - shift), MPI_grouporder(group)));

     right = MPI_groupelement(group,
          MOD((myrank + shift), MPI_grouporder(group)));

// now execute the shift:

    SEND(in, len, right, tag, context);
    RECV(out, len, left, tag, context);

    }

2) (bflyexchange)

// MPI_BFLY_B -- butterfly exchange
//
//   limitations:
//      uses aggressive style of message passing.
//      no check for len being same on all processes.
//
//   assumptions:
//      MPI_MYNODE()  returns process id
//      MOD(a,b) = a modulo b
//      SEND(buf, len, dest, tag, context) sends buffer to destination.
//      RECV(buf, len, src, tag, context)  receives buffer from source.
//
void  MPI_BFLY_B ( char *in, char *out, int len, int shift,
     MPIGroup group, int context, int tag)
    {
     int myrank;       // rank of this process in the group
     int partner;      // process id for exchanging
     int j;

// exit routine if this process is not a group member:

     if(!MPI_groupiselement(group, MPI_MYNODE())) return;

// get my rank in group and compute partner for exchange:

     myrank = MPI_grouprank(group, MPI_MYNODE());
     j = (myrank % (shift<<1))/shift;

     partner = MPI_groupelement(group,
          myrank + shift(1-2*j));

// (the problem statement was a little unclear about what to do if
//  the exchange partner was not in the group. Here I just assume
//  that it is in the group.) 

// now execute the exchange:

    SEND(in, len, partner, tag, context);
    RECV(out, len, partner, tag, context);

    }

  Discussion:

    Safety:

    This routine (and other similarly implemented global operations)
    are safe if no process in the group has a send or receive using
    the same context and tag.

    The routine is unsafe if a process in the group
    issues another send outside the routine to its partner in the
    dual exchange (using the same tag),  or if a process posts a
    wildcard receive prior to calling this routine which would match
    the incoming message.

    Thus, the safety of this routine is the same as the safety associated
    with normal point to point message passing.

    Performance:

    Performance of this routine acheives the limit of one send/receive
    per process. The computational overhead of the group operations will
    depend on the type of group used, but with use of code inlining and
    for simple kinds of groups this overhead will be moderate.
    Note that group operations require no communications, since a group
    is defined algorithmically.

2) Guidelines for library developers.

There are two possible models for developing library or third party
software which uses MPI, but which is in turn intended for use by other
independently developed software. The first kind uses context and tags
provided to it by the calling software, and is typical of communication
services. An example of such software that would not be provided by
MPI would be a matrix transpose. Writing software of this type is similar
to current practice, but the author must avoid using tags and context for
which permission has not been granted. One possibility for the author
is to simply specify a context in the interface and make the explicitly
declared assumption that tags within that context are usable in any way
within the library module. An author of multiple library modules that
need to work together can develop an internal scheme for tag management
which allows the use of minimum numbers of contexts.

The second kind of library software establishes and manages its own context
and tag spaces. This kind of software is typical of higher level operations
which provide computational functions (which implicitly use communications).
There are a number of ways for a library to do this management. For
example, to begin with a library module might make use of static contexts
and assume it has only one 'instantiation' within a program. More advanced
libraries might bind context to data objects that are being manipulated and
allow multiple simultaneous use of the library code.

In both cases, a primary source of problems will be receives where the
tag filter is wild-carded. Such receives should be used with care, and
independent context values associated with each. A secondary
source of problems will be collective communication operations among
groups which intersect. Again, such operations can be made non-interfering
by choosing different contexts for each.

3) host-node programming

The description of this problem makes some assumptions that
I don't agree with. First is that the ALL group contains the host. My
opinion is that there might be two preexisting groups, the ALL group
containing all the nodes, and the ALLH group containing all the nodes
and the host process. Here I make the assumption that in the ALLH group
the host is the last process.

To make the problem interesting, the program uses a static context.

These groups can be built as follows, assuming constructors for
building linear groups and list groups:

// build the ALL group
MPIGroup ALL;     // all processes
MPIGroup ALLH;    // all processes plus host

...

// Build ALL, ALLH groups
//
// Assume
//    MPI_HOSTID()  -- returns id of host
//    MPI_NNODES()  -- returns number of node processes
//    MPI_makelineargroup(int order, int start, int stride)
//                  -- constructor for linear group
//    MPI_makelistgroup(int order, int *list)
//                  -- constructor for list group
//
void MPIBuildAlls()
{
   int *list; int n; int i;

   n = MPI_NNODES();
   ALL = MPI_makelineargroup(
         /* order: */ n, /* start */ 0, /* delta */ 1);

   list = (int *) malloc(sizeof(int) * (n + 1))
   for(i=0;i<n;i++)
     {
      list[i] = i; 
     }
   list[n] = MPI_HOST();

   ALLH = MPI_makelistgroup(n+1, list);
   free(list);
}

Now the following code should be ok:

// 
#include <mpi.h>

MPIGroup hgroup;     // group containing host only
MPIGroup ngroup;     // group containing all nodes

static int me;
static int hostid;
static int context = 100;   // a randomly chosen context
static int tags = 13367;    // a randomly chosen tag base
main()
{
   me = MPI_NODE();
   hostid = MPI_HOSTID();

   MPI_opencontext(context);  // use a static context

   if(me == hostid)
     host();
   else
     node();

   MPI_closecontext(context);  // context is no longer in use
}

host()
{
   int myrank;

   // build host, node groups:
   hgroup = MPI_makelistgroup(1, &hostid);
   ngroup = ALL;  // lets not do more work than necessary

   // compute rank of host in ALLH group
   myrank = MPI_grouprank(ALLH, hostid);
   
   // broadcast from host to all nodes (perfectly legitimate)
   MPI_BCAST( ..., myrank, ALLH, tags, context);

   // send a message to each node in turn
   for(i=0;i<MPI_grouporder(ALL);i++)
      {
// this is the same as
//     MPI_SEND(..., i,    tags+1, context)
       MPI_SEND(..., MPI_groupelement(ALL, i), tags+1, context);
      }

   // receive result from node 0
   MPI_RECV(..., 0, tags+10, context);

}

node()
{
   int myrank,hrank,nnodes,i;

   // build host, node groups:
   hgroup = MPI_makelistgroup(1, &hostid);
   ngroup = ALL;  // lets not do more work than necessary

   myrank = MPI_grouprank(ALL, me);
   nnodes = MPI_NNODES();

   // compute rank of host in ALLH group
   hrank = MPI_grouprank(ALLH, hostid);
   
   // broadcast from host to all nodes (perfectly legitimate)
   // in node() we are recipient!
   MPI_BCAST( ..., hrank, ALLH, tags, context);

   // recv message from host, thought of as member of host group
   MPI_RECV( ..., MPI_groupelement(hgroup, 0), tags+1, context);

   // inter node communications using group notation instead of
   // direct addressing:
   MPI_SEND(... , MPI_groupelement(ALL, (myrank+1)%nnodes), tags+2,
      context);
   MPI_RECV(... , MPI_groupelement(ALL, (myrank+nnodes-1)%nnodes), tags+2,
      context);

   // synchronize
   MPI_SYNC(ALL, tags+3, context);

   // send completion message to host
   if(me == 0) MPI_SEND(..., hostid, tags+4, context);
}

>From [email-masked] Wed Jun  9 14:53:36 1993
Date: Wed, 9 Jun 93 07:53:48 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: May 12 mpi mail #2
Status: R

LaTeX source (sender forgotten: I deleted the verbiage before the LaTeX stuff
so I could just print it.. I think it was sent by Steve Otto)
DW --------------------------- cut here ----------------------------------------
\documentstyle[twoside,11pt]{report}
\pagestyle{headings}
%\markright{ {\em Draft Document of the MPI Standard,\/ \today} }
\marginparwidth 0pt
\oddsidemargin=.25in
\evensidemargin  .25in
\marginparsep 0pt
\topmargin=-.5in
\textwidth=6.0in
\textheight=9.0in
\parindent=2em

%   ----------------------------------------------------------------------
%   mpi-macs.tex  --- man page macros,
%                discuss, missing, mpifunc macros
%
% ----------------------------------------------------------------------
% a couple of commands from Marc Snir, modified S. Otto

\newlength{\discussSpace}
\setlength{\discussSpace}{.7cm}

\newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace}
}

\newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace}
}

\newlength{\codeSpace}
\setlength{\codeSpace}{.3cm}

\newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} }

% -----------------------------------------------------------------------
%  A few commands to help in writing MPI man pages
%
\def\twoc#1#2{
\begin{list}
{\hbox to95pt{#1\hfil}}
{\setlength{\leftmargin}{120pt}
 \setlength{\labelwidth}{95pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#2}
\end{list}
}
\outer\long\def\onec#1{
\begin{list}
{}
{\setlength{\leftmargin}{25pt}
 \setlength{\labelwidth}{0pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#1}
\end{list}
}
\def\manhead#1{\noindent{\bf{#1}}}

\hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple}

\begin{document}

\setcounter{page}{1}
\pagenumbering{roman}

\title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface}

\author{Scott Berryman, {\em Yale Univ} \\
James Cownie, {\em Meiko Ltd} \\
Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\
Al Geist, {\em ORNL} \\
Bill Gropp, {\em ANL} \\
Rolf Hempel, {\em GMD} \\
Bob Knighten, {\em Intel} \\
Rusty Lusk, {\em ANL} \\
Steve Otto, {\em Oregon Graduate Inst} \\
Tony Skjellum, {\em Missisippi State Univ} \\
Marc Snir, {\em IBM T. J. Watson} \\
David Walker, {\em ORNL} \\
Steve Zenith, {\em Kuck \& Associates}    }

\date{May 10, 1993 \\
This work was supported by ARPA and NSF under contract number \#\#\#,
by the National Science Foundation Science and
Technology Center Cooperative Agreement No. CCR-8809615.
}

\maketitle
\hfuzz=5pt
%\tableofcontents

%\begin{abstract}
%We don't have an abstract yet.
%\end{abstract}

\setcounter{page}{1}
\pagenumbering{arabic}

\label{sec:context}

%======================================================================%

\chapter{Contexts -- Proposal X}
\begin{center}
{\em Lyndon J.~Clarke, Tony Skjellum, Rik Littlefield}\medskip
\end{center}

This chapter describes the definition of context, its relation to
group and tag, and describes communication safety, while indicating
how all this relates to point-to-point and global communication.  We
include examples that illustrate the features incorporated, including
the ``Rik Littlefield'' examples, plus the logical grid topology
context example of Skjellum.

We have utilized C-language semantics to describe the functionality,
whereas other authors have utilized Fortran semantics.  This should
not be construed to denigrate (or render impossible) a Fortran
interface.

%
%----------------------------------------------------------------------
% Context, Tag, Group and Process

\section{Context, Tag, Group and Process}

This section describes the view of context, tag, group and process
taken by MPI.  MPI views group and context as
different and unrelated entities.  Group is used for management of
processes and context is used for management of messages.

The purpose of these definitions are to promote the generation
of library modules, and to allow users safely to combine programs
written at different times, or by different programmers, without
globalizing the semantics of how messaging is done.  To do this,
we introduce scope to message passing.  This is accomplished
through contexts and groups, primarily, and then by tags, within
the user's perview.  Note that we take a {\em laissez faire\/} view of the
context resource (in the same spirit as Unix ports), in that
we provide an resource allocation mechanism, but also provide for
correct programs that define their own ``well-known'' contexts.
Depending on implementation, either ``well-known'' or system-allocated
contexts might be of higher performance, or the first allocated
might always be faster, depending on the implementation details.
(In doing so, we have removed a major incompatibility of this
MPI formulation with the alternate Proposal VIII (Mark Sears)).

We do not view groups, contexts, and tags as totally ``orthogonal,''
but we seek to make them orthogonal as possible, consistent with
providing a safe, optimizable environment for programming.  Since key
events in the formation of groups may require unique contexts for
their creation, and because contexts are often naturally associated
with groups, their is a tendency to link context heavily with group.
Actually, we have found that the group-to-context relationship is not
one to one, both based on experience in experimental systems, and in
recent discussions.  Except where needed, contexts and groups are
separate concepts, but we build up communicator objects in the
following, that intentionally bind them together for programming
convenience.  In the end, this formulation remains at odds with
Proposal I (Marc Snir), which uses the word ``context'' in a
semantically distinct sense.  Here, context always means a logical
partition of the tag space into a user-defined and system-defined
space.  The latter is subsequently allocated/deallocated under either
a system- or user-controlled safety mechanism, or a combination of
both.  Though we permit contexts to be integer values, and hence
transmittable like tags, the operating system may choose internal
optimizations by hashing such contexts to make reference to hardware
message tags, or other facilities that we cannot anticipate in a
portable standard.  These runtime optimizations are not excluded.

Furthermore, in what follows, we note at times that simplifications
occur for the SPMD model.  These simplifications imply a simpler
implementation, and less potential overhead for that programming
model, while including (rather than excluding) more general
programming in the MPMD model.  MPI libraries for SPMD and MPMD may
be appropriately optimized at compile-time and/or run-time by the
implementor.

\subsection{Tag and Context}

\subsubsection{Tag}
MPI defines a {\em tag\/} as the familiar non-negative integer label
of a message.  The tag is a field in message selection which may be
wildcard.  There is no concept of creation or deletion of tags. Any
non-negative integer value may be used as a tag. The MPI user is
responsible for management of tag usage.  It is correct for a process
{\cal P} to transmit a tag value T to another process {\cal Q}, and,
subsequently, for {\cal P} to then send messages of tag T, and for
{\cal Q} to then receive messages of tag T.  A tag is a 32-bit
integer.  Users assign non-negative tag values.

{\small
{\bf Discussion:} This definition appears compatible with the point-to-point
definition of tag, at present.  We will introduce an amendment to
tag matching to point-to-point, to allow bit selection (which
was narrowly defeated last time, and justify that better this time).
Other than that, tag definition is in no apparent conflict with
point-to-point.}

\subsubsection{Context}
MPI defines a {\em context\/} as the familiar integer label
of a distinct message space, or more formally as a distinct message
tag space.  The context is a field in message selection that may not
be wildcarded. There is no concept of creation or deletion of
contexts. Key features are as follows:
\begin{itemize} \item A context is
exactly analogous to a tag, except that it may not be wildcarded in
selection. \item In MPI, a context is stored in a 32-bit integer.
Implementations that facilitate context management need not span
the full 32 bits.  Users are constrained to use non-negative contexts.
System-provided contexts (see below), will be negative.
\end{itemize}

The MPI user is responsible for the management of context
usage.  Any non-negative integer value may be used as a context.  It
is correct for a process {\cal P} to transmit a context value {\bf C} to
another process {\cal Q}, and, subsequently, for {\cal
P}  to send and receive messages of context {\bf C}, and for {\cal Q} to 
send and receive messages of context C. 

{\small
{\bf Discussion:} A switch to 16-bit contexts would be a reasonable
implementation alternative; we are convinced that a compliant
implementation should permit at least 16 bits of context.  Because
these numbers are essentially a hash to hardware resource, in some
implementations, 16 bits may provide nominal performance improvements.
It is also thought that 16 bits would provide a sufficiently large
number of contexts for practical applications, as currently envisaged,
even those that have a number of objects.  We strongly oppose less
than 16 bits, because they could be insufficient for reasonable
applications. We suggest the natural symmetry of tag and context
sizes may simplify alignment in implementations, and reduce the number
of data types in MPI by one.  Again, we view the MPI contexts
are software contexts; the mapping to ``hardware contexts'' is
an implementation issue; there may only be an infinitesmal number of
such hardware contexts.  Still, in analogy to register variables in
a C program, a program continues to compile and run after the
fast register resource is exhausted\ldots one suggests to the implementor
that at least one context support hooks to a slower, but bigger 
internal context map.}

\subsubsection{Context Management}
MPI provides services that facilitate context management and
recommends, although does not mandate, use of these services. MPI
provides a context allocation and deallocation facility that allows
user processes to generate unique context values.

\begin{verbatim}
   int context_array[]; /* OUT */
   int number_to_alloc; /* IN  */
   mpi_alloc_contexts(context_array, number_to_alloc)
\end{verbatim}

This procedure allocates \verb|number_to_alloc| unique context values
and and stores them in \verb|context_array|.  None of the context
values allocated have been returned by a previous call to the
procedure in any process unless such values have also previously been
deallocated by calling \verb|mpi_free_contexts|.

\begin{verbatim}
   int context_array[]; /* IN  */
   int number_to_free;  /* IN  */
   mpi_free_contexts(context_array, number_to_free)
\end{verbatim}

This procedure deallocates \verb|number_to_free| unique context values
stored in the array \verb|context_array|.  Each context value must
have previously been allocated by calling \verb|mpi_alloc_contexts| or
else the program is erroneous.  The context values deallocated may be
allocated with future calls of \verb|mpi_alloc_contexts|.

{\small
{\bf Discussion:} Allocation and deallocation of contexts involves a global
synchronization in the SPMD model, and the possibility of a context
server in the full MPMD programming model.  The degree of
implementation difficulty depends on which model the programmer is
using.  SPMD programming involves no server.  Thus, extra overhead
(however minimal) is removed from the SPMD model.  An appropriate MPI
implementation could provide an SPMD and MPMD version to facilitate
this optimization, or via runtime checking.}

{\small
{\bf Discussion:} We allow the user to write programs with complicated
and possibly unpredictable behavior by allocating a context C in one
process P and sending the context to another process Q which frees the
context.  P continues to use context C while a third process R
allocates C for another purpose.  Therefore, the user is expected to
maintain safety in such cases, through systematic programming.  Programs
that do not maintain their own context coherence after \verb|mpi_free_contexts|
are said to be erroneous.}

MPI permits correct programs to include a mixture of code that do
their own allocation of contexts with code that utilizes the automatic
context registration mechanisms.  The ``fastest'' contexts are
implementation-defined, and implementations may provide hints as to
which contexts are faster, or undertake heuristics to utilize faster
contexts before slower contexts.

{\small
{\bf Discussion:} Selection of a range of system- and user-allocated
contexts, as opposed to positive/negative (equi-partition of bits)
would be viewed as a friendly amendment, if this further articulation
is seen as sufficiently important to merit the additional mechanisms
needed to implement it.  The functionality is equivalent, but the
latter form may be more flexible for implementors, and more useful
to users.  This issue could be remanded to the Environment Subcommittee.}

\subsubsection{Context Global Association}
MPI provides name services which allow the user to make global
associations of names with contexts and locate contexts by associated
names.

\begin{verbatim}
   char *name;  /* IN  */
   int context; /* IN  */
   mpi_associate_context(name, context)
\end{verbatim}

This procedure associates \verb|name| with context value
\verb|context|. The procedure fails if \verb|name| is 
associated with any context value.

\begin{verbatim}
   char *name;  /* IN  */
   int context; /* IN  */
   mpi_dissociate_context(name, context)
\end{verbatim}

This procedure dissociates \verb|name| from the context value
\verb|context|. The procedure fails  if \verb|name| is not
associated with the context value \verb|context|.

\begin{verbatim}
   char *name;  /* IN  */
   int context; /* OUT */
   int wait;    /* IN  */
   mpi_locate_context(name, &context, wait)
\end{verbatim}

This procedure  locates  the  context value \verb|context|  associated
with \verb|name|. If  \verb|name|  is not  associated  with  a context
value then the behavior is determined by the logical value of
\verb|wait|.  If \verb|wait| is \verb|true| then  the  procedure waits
until \verb|name| becomes associated with a context value. If
\verb|wait| is \verb|false| the procedure returns
\verb|MPI_NULL_CONTEXT| in \verb|context|.

\subsection{Group and Process}

\subsubsection{Process}
MPI views a process in the familiar sense, for example as a
Unix or Intel NX process.  MPI defines a {\em process
identifier\/} (pid for short) as a process-local integer entity that
refers to an opaque process description object of undefined size.
This means that if two processes {\cal P} and {\cal Q} both know the
identifier of a process {\cal R} then the relationship between the values of
the identifiers known to {\cal P} and {\cal Q} is undefined.

\begin{small}
{\bf  Discussion:}  This  pid   definition  leaves  freedom  for   the
implementor to  do whatever is efficient  for the machine(s)  at hand,
and does not exclude implementations in which the pid is global.
\end{small}

\subsubsection{Group Definition}
MPI defines a group as an ordered collection of processes,
or more formally as an ordered collection of pids. MPI
defines a {\em group identifier\/} (gid for short) as a process-local
integer entity that refers to an opaque group description object of
undefined size.  This means that if two processes {\cal P} and {\cal
Q} both know the identifier of a group {\bf G} then the relationship between
the values of the identifiers known to {\cal P} and {\cal Q} is undefined.

This MPI formulation defines a pid as formally the identifier of a
singleton group containing only the process.

\subsubsection{Group Services}
MPI provides services that allow the user to create and delete groups.

\begin{verbatim}
   int new_group;  /* OUT */
   int old_group;  /* IN  */
   int subset_key; /* IN  */
   mpi_subset_group(&new_group, old_group, subset_key)
\end{verbatim}

This procedure  creates  one  or more  new  groups  as subsets  of  an
existing group  according to a subset key.  The procedure synchronizes
the member processes of the existing group.

\begin{verbatim}
mpi_permute_group(&new_group, old_group, permute_rank)
\end{verbatim}

This procedure creates one new group by permuting the member  ranks of
an existing group. The procedure synchronizes the member  processes of
the existing group.

\begin{verbatim}
   int new_group;           /* OUT */
   int old_group_array[];   /* IN  */
   int number_of_old_group; /* IN  */
   mpi_union_group(&new_group, old_group_array, number_of_old_group)
\end{verbatim}

This  procedure  creates  one new  group by  a union  of  one  or more
distinct  existing  groups.   The  procedure  synchronizes the  member
processes of the union. Note in  particular that each  existing  group
can of course be the singleton group of a process.

\begin{verbatim}
   mpi_delete_group(group)
\end{verbatim}

This procedure deletes a group. The procedure synchronizes the  member
processes of the group.

\begin{small}
{\bf Discussion: } We have concluded that at least one default
context is needed, safely to implement the above group creation,
deletion, and union operations.  This context may be a hidden
context, per the implementation.  It is at this point that
group and context become non-orthogonal, because a safe messaging
mechanism must be supported, deterministically to accomplish
these operations.    The communicators described below reveal 
context and group merger explicitly to the user.  The above operations
do not.
\end{small}

\subsubsection{Group Transmission}
MPI provides services which allow the user to  write  a portable group
description into a memory buffer and read a portable group description
from  a  memory buffer. The  memory buffer  can  be  transmitted in  a
message, allowing transmission of group descriptions.

\begin{verbatim}
   char *buf_ptr; /* OUT */
   int   buf_len; /* IN  */
   int   group;   /* IN  */
   mpi_passivate_group(group, buf_ptr, buf_len)
\end{verbatim}

This  procedure writes a portable description of \verb|group| into the
buffer \verb|buf_ptr| of length \verb|buf_len| bytes.

\begin{verbatim}
   char *buf_ptr; /* OUT */
   int   buf_len; /* IN  */
   int   group;   /* OUT */
   mpi_activate_group(&group, buf_ptr, buf_len)
\end{verbatim}

This procedure reads a portable description of a group from the buffer
\verb|buf_ptr|  of  length \verb|buf_len|  bytes  and  returns a group
descriptor identifier for the read group in \verb|group|.

MPI  provides  name services  that  allow  the user  to  make  global
associations of names  with  groups and  locate  groups by  associated
names.

\begin{verbatim}
   char *name; /* IN  */
   int group;  /* IN  */
   mpi_associate_group(name, group)
\end{verbatim}

This procedure associates \verb|name| with group identifier by
\verb|group|. The procedure fails if \verb|name| is 
associated with any group.

\begin{verbatim}
   char *name; /* IN  */
   int group;  /* IN  */
   mpi_dissociate_group(name, group)
\end{verbatim}

This procedure dissociates \verb|name| from the group identified by
\verb|group|. The procedure fails  if \verb|name| is not
associated with the group identified by \verb|group|.

\begin{verbatim}
   char *name; /* IN  */
   int group;  /* OUT */
   int wait;   /* IN  */
   mpi_locate_group(name, &group, wait)
\end{verbatim}

This  procedure  locates  the group associated  with  \verb|name|  and
returns a group identifier for the located group in \verb|group|.  If
\verb|name| is not associated with  a context value then the behavior
is determined by the logical value of
\verb|wait|.  If \verb|wait| is \verb|true| then  the  procedure waits
until \verb|name| becomes associated with a group. If
\verb|wait| is \verb|false| the procedure returns
\verb|MPI_NULL_GROUP| in \verb|group|.

\begin{small}
{\bf Discussion:} When  a process  receives a group description either
by  means of  \verb|mpi_locate_group| or  \verb|mpi_activate_group|, it
may later delete that group using \verb|mpi_delete_group|. If so, then
the group deletion should simply delete the received copy of the group
description rather than the actual group.
\end{small}

\subsubsection{Default Groups}
MPI provides each process with two  predefined groups, which cannot be
deleted.  These are  the singleton group of which the  process  is the
only member and a single group of which all processes are a member.

\begin{verbatim}
   int pid;
   mpi_my_pid(&pid)
\end{verbatim}

This  process  returns the singleton  group identifier,  that is,  process
identifier, of the calling process.

\begin{verbatim}
   int gid;
   mpi_all_group(&gid)
\end{verbatim}

This process returns the group identifier of the group of all
processes.

\begin{small}
{\bf Discussion:}  The concept  of the ``all''  group  is  not  really
extensible to a dynamic process world.  We could  replace this with an
``initial'' group for example. It might be more general to replace the
all group by the following. Creation of the set of processes composing
the program  is  implementation specific.  Require an  implementation
specific method  of assigning each process to a {\em base\/}  process
group, with different and  distinct  groups of processes in  different
base groups.  Default is to  make  all  processes members of  the same
base  group.  Provide a  procedure  to  return the  base  group of the
process.
\end{small}

%----------------------------------------------------------------------
% Communicator and Communication

\section{Communicator and Communication}

Now that we have established the fundamental workings of tag, context,
and group, we can turn to the standard user-level communication
objects of MPI.  In particular, communication between processes is
provided by communicator objects that are a binding of a message
context and one or more process ``worlds.''  A process world is either
a process group or all processes composing the user program.

\subsection{Intra-communication}

Intra-communication is communication between processes within the same
process world.  This service is provided by an intra-communicator
object, which is a binding of a context, and a process world.  The
process world is either a group or all processes composing the user
program.

\begin{verbatim}
   int communicator; /* OUT */
   int context;      /* IN  */
   int world;        /* IN  */
mpi_create_intra_communicator(&communicator, context, world)
\end{verbatim}

This  procedure   creates   an  intra-communicator  and   returns  the
communicator identifier in \verb|communicator|. 

The intra-communicator world  supplied in \verb|world| may be an
actual group identifier or the null group \verb|MPI_NULL_GROUP|. If
\verb|world| is an actual group identifier, then the world is
the  identified group, and  processes  within the  world are identified
relatively by {\bf rank} within that group. Otherwise the world contains all
processes within the user program  and processes within the  world are
identified absolutely by process identifier.  

The communicator context supplied in  \verb|context| may  be an actual
context value or the null context \verb|MPI_NULL_CONTEXT|.  If
\verb|context| is an actual context then the procedure completes
asynchronously.  If  \verb|context|  is  the  null  context  then  the
procedure  allocates  a single  context using  the  context  allocator
described above, synchronizes  all  members  of  the 
process world,   and   uses   the   allocated   context   in   each
intra-communicator created.

\begin{small}
{\bf    Discussion:}  The name ``intra-communicator'' is long.
We propose ``C1'' as the nick-name for an intra-communicator object.

{\bf Discussion:} We conserve the Proposal I rank-mechanism for referring
to processes in groups, for most cases (see above), and under communication
primitives.   We see this as the most easily understood, intuitive way
to describe communication relative to the group; the level of virtualization
implied by this mapping is also desirable from the point of view of 
safety.
\end{small}

\begin{verbatim}
   int communicator; /* IN  */
   int world;        /* OUT */
   mpi_intra_communicator_world(communicator, &world)
\end{verbatim}

This procedure  returns in  \verb|world|  the  process  world  of  the
intra-communicator identified by \verb|communicator|.

\begin{verbatim}
   int communicator; /* IN  */
   int context;      /* OUT */
   mpi_communicator_context(communicator, &context)
\end{verbatim}

This procedure returns in \verb|context| the context of a communicator
object identified by \verb|communicator|.  If the context  supplied in
communicator creation  was the null context, then the procedure returns
the context allocated during communicator creation.

\begin{verbatim}
   int communicator; /* IN  */
   mpi_delete_communicator(communicator)
\end{verbatim}

This procedure deletes a  communicator object. If the context supplied
in  communicator creation  was  the null  context  then this procedure
synchronizes  the  process   members  of  process   world(s)   of  the
communicator  and   deallocates   the   context  allocated   for   the
communicator. Otherwise the procedure completes asynchronously.

\subsection{Inter-communication}

Intercommunication is communication between processes within two
different process worlds.  This service is provided by an
inter-communicator object, which is a binding of a context, a local
process world, and a remote process world. Each process world is
either a group or all processes composing the user program.

\begin{verbatim}
   int communicator; /* OUT */
   int context;      /* IN  */
   int local;        /* IN  */
   int remote;       /* IN  */
   mpi_create_intercommunicator(&communicator, context, local, remote)
\end{verbatim}

This  procedure   creates   an  inter-communicator  and   returns  the
communicator identifier in \verb|communicator|. 

The intercommunicator local world supplied in \verb|local| may be an
actual group identifier or the null group \verb|MPI_NULL_GROUP|. If
\verb|local| is an actual group identifier then the local world is
the  identified  group  and  processes  within  the  local  world  are
identified relatively  by rank within  that group. Otherwise the local
world  contains all  processes within  the  user program and processes
within   the   local  world  are  identified  absolutely   by  process
identifier.  

The inter-communicator remote world supplied in \verb|remote| may be
an actual group identifier or the null group \verb|MPI_NULL_GROUP|.  If
\verb|remote|  is an actual group identifier then the  remote world is
the  identified  group  and  processes  within  the  remote world  are
identified  relatively by rank within that group. Otherwise the remote
world  contains all  processes within the user program  and  processes
within  the  remote  world  are  identified   absolutely  by   process
identifier.

The communicator context supplied in \verb|context| may be an actual
context value or the null context \verb|MPI_NULL_CONTEXT|.  If
\verb|context| is an actual context then the procedure completes
asynchronously. If  \verb|context|  is  the  null  context  then  the
procedure  allocates  a single  context using  the  context  allocator
described above, synchronizes  all  members  of  the local  and remote
process   worlds,   and   uses   the   allocated   context   in   each
intercommunicator created.

Where the user explicitly manages context values then he or she must
ensure that the same process world(s) are bound to the same context
value in each process which uses that context value, otherwise the
program is erroneous.

\begin{small}
{\bf Discussion:} We propose the nickname ``C2'' for the
inter-communicator.
\end{small}

\begin{verbatim}
   int communicator; /* IN  */
   int local;        /* OUT */
   mpi_intercommunicator_local(communicator, &local)
\end{verbatim}

This  procedure  returns  in  \verb|local|  the  local  world  of the
inter-communicator identified by \verb|communicator|.

\begin{verbatim}
   int communicator; /* IN  */
   int remote;       /* OUT */
   mpi_intercommunicator_remote(communicator, &remote)
\end{verbatim}

This procedure  returns  in  \verb|remote|  the  remote  world  of the
intercommunicator identified by \verb|communicator|.

The context of an inter-communicator may be determined using the
procedure\linebreak[4]\verb|mpi_communicator_context| described above.  The
procedure will again return the context allocated dynamically when the
context supplied in communicator creation was the null context.

An inter-communicator may be deleted using the procedure
\verb|mpi_delete_communicator|  described  above.  The  procedure will
synchronize all members of  both the local  and  remote process worlds
when  the  context  supplied  in communicator  creation  was the  null
context, and will otherwise complete asynchronously.

\subsection{Point-to-point}
The generic point-to-point communication send \verb|send| accepts a
single communicator, a process label, and a message label as arguments
and send-specific arguments:

\begin{verbatim}
   send(communicator, process_label, message_label, ...)
\end{verbatim}

\verb|communicator| is the identifier of a communicator object. If the
communicator is an intra-communicator then the sender  must be a member
of  the   intra-communicator   world.   If   the   communicator  is  an
inter-communicator   then  the  sender   must  be   a  member  of   the
inter-communicator local world.

\verb|process_label| identifies the receiver process. If the
communicator is an intra-communicator and the world is an actual
group, or if the communicator is an inter-communicator and the remote
world is an actual group, then \verb|process-label| is a rank within
the group.  If the communicator is an intra-communicator and the world
is the null group, or if the communicator is an inter-communicator and
the remote world is the null group, then \verb|process-label| is a
process identifier.

\verb|message_label| is the message tag in the context+tag space of
the communicator.

The generic point-to-point communication receive \verb|recv| accepts a
single communicator, a process label, and a message label as arguments
and receive-specific arguments:

\begin{verbatim}
   recv(communicator, process_label, message_label, ...)
\end{verbatim}

\verb|communicator| is the identifier of a communicator object. If the
communicator is an intra-communicator then the sender  must be a member
of  the   intra-communicator   world.   If   the   communicator  is  an
intercommunicator   then  the  sender   must  be   a  member  of   the
intercommunicator local world.

\verb|process_label| identifies the sender process. If the
communicator is an intra-communicator and the world is an actual
group, or if the communicator is an intercommunicator and the remote
world is an actual group, then \verb|process-label| is a rank within
the group.  If the communicator is an intra-communicator and the world
is the null group, or if the communicator is an intercommunicator and
the remote world is the null group, then \verb|process-label| is a
process identifier.  This field may be wildcarded by supplying the value
\verb|MPI_PID_ANY|.

\verb|message_label| is the message tag in the tag space of
the communicator.  This field may be wildcard by supplying the value
\verb|MPI_TAG_ANY|.

\subsection{Collective}
The generic collective communication operation \verb|operation|
accepts a single communicator as argument and operation specific
arguments.

\begin{verbatim}
   operation(communicator, ...)
\end{verbatim}

Communicator must be an intra-communicator with a process world which
is a process group of which the caller is a member, or else the
program is erroneous.

\subsection{Message Envelope}
To provide the services and capabilities we envisage for MPI, the
prototypical message envelope is to consist of :
\begin{description}
\item[context], which is the identifier of the communicator
\item[sender], which is the label of the sender according to the world,
or remote world, of the communicator
\item[tag], which is the message tag
\item[receiver], which is implementation- specific information required
to route the message to the receiver
\end{description}

\begin{small}
{\bf Discussion:} The name of sender and receiver, has to be chosen
in a way that is useful to the MPI implementation, rather than the user.
The user works in ranks, but the system may actually transfer its own
internal name for the sender.  This remains an implementation detail,
with a standard interface based on rank.
\end{small}

\section{Safety}

The user of collective communications must be aware that the
operations have no mechanism for protecting the point-to-point
messages of which they are composed from those of the user program
within the same context. The user is responsible for ensuring that
there are no oustanding communications with the same communicator when
the collective communication procedure is called.  The writer of
collective communications cannot use wildcard on either tag or sender
in point-to-point receive calls, and strict sequencing of
point-to-point messages is assumed. 

This is more easily handled by utilizing a duplicate communicator: one
for point-to-point only, and one for deterministic global operations,
all of which have the strict ordering assumptions.  Each additional
global operation (non-deterministic) will in general need its own
communicator (new context, plus copy of the same group).  This issue
is now well understood, and the features included in this formulation
provide a straightforward means for user and library code to achieve
safety.

%----------------------------------------------------------------------
% Point-to-point Chapter

\section{Point-to-point Chapter}

{\bf Changes to draft to go here... we will work these out based
on newest Snir chapter, on Wednesday night, May 12}

%----------------------------------------------------------------------
% Collective Chapter

\section{Collective Chapter}

{\bf Changes to draft to go here... we will work these out based
on newest Geist chapter, on Wednesday night, May 12 \& 13}

%----------------------------------------------------------------------
% Worked Examples

\section{Worked Examples}

{\bf Worked examples to go here... we will add these for you ASAP}
{\bf Littlefield examples, Skjellum examples}

%----------------------------------------------------------------------
% Conclusion

\section{Conclusion}

This formulation of context, group, and tag, provide a powerful,
forward-looking standard within MPI, that ``closes the loops'' needed
to complete the standard.  It does so by providing tiers of services,
which keep context and group as separate entities as much as possible
at the low level, but which subsequently unite them for the user's
benefit in intra-communicator (C1) and inter-communicator (C2)
objects, that cope with group and context simultaneously.  The latter
provides a group union capability, especially important in MPMD and
distributed-computing situations.  Through careful choice of features,
we are able to support safe programs in which both system-provided and
user-defined contexts are permissible.  We are able to provide a tag
field that is totally at the user's disposal, and big enough to allow
sophisticated application-dependent selectivity (32 bits).

This formulation achieves the goal of enabling software module
formulation, and provides servers to allow name/context matching,
another feature of potential impact to both MPMD and
distributed-computing arenas.  It further has the clear potential to
drop servers whenever the SPMD model is elected, replacing same with
synchronizations over all processes.

We have left out some things compared to earlier versions.  The cacheing
facility can be implemented later, in MPI2, or as an implementation-specific
set of features, for now.  We have omitted discussion of virtual topologies,
because they can be layered on top of MPI1, using two features:
\begin{itemize}
\item contexts for each sub-topology,
\item layerability of tag selectivity (dont\_care bits, match bits).
\end{itemize}
Given these capabilities, we see no need to stipulate virtual
topologies in the base MPI is apparent, and so we have not gone into
this further here.

%======================================================================%
\end{document}

>From [email-masked] Sat May 15 22:36:57 1993
Date: Sat, 15 May 93 22:35:13 CDT
From: Tony Skjellum <[email-masked]>
To: [email-masked]
Subject: the gathering (part II)
Status: R

We have a unified proposal between Marc Snir, myself, Lyndon Clarke,
and Mark Sears, which we worked out at the last meeting, and which we will
flesh out better over the next weeks, and present to you here.  

This progress report is mainly for the benefit of those who did not attend
the meeting. 

One point that did not appear in the pre-meeting proposal rounds was the
cacheing facility.  It will definitely be needed to support virtual
topologies.

We will send you update soon,
Tony

>From [email-masked] Fri May 21 13:21:09 1993
Date: Fri, 21 May 93 14:07:42 EDT
From: "Marc Snir" <[email-masked]>
To: [email-masked]
Status: R

%!PS-Adobe-2.0
%%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software
%%Title: CON-V6.DVI.*
%%Pages: 11 1
%%BoundingBox: 0 0 612 792
%%EndComments
%%BeginProcSet: texc.pro
/TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch
load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{
isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale
Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get
round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10
N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{
/vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{
statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N
/FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin
/FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array
/BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2
array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}
B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont
setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup
length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{
ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B
/ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0
N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S
dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0
ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice
ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N
/rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub
/rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod
S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index
getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw
cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and
or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255
eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2
index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval
putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv
1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{
adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1
add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{
/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S
ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr
ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI
save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook
known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if
/VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1
string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N
/BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V
statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval
(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale
rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex
ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B
/delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}
B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B
/k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1
w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{
/SS save N}B /eos{clear SS restore}B end
%%EndProcSet
TeXDict begin 1000 300 300 @start /Fa 6 119 df<EA03CCEA07FCEA0E7CEA183CEA3838
12301270A2EAE070A3137113E3A212E1EA63E6EA7E7CEA3C3810127B9115>97
D<EA01F0EA07F8EA0E18EA1C3CEA3878EA3030EA7000A25AA513101338EA70F0EA3FC0EA1F000E
127B9113>99 D<EA01E0EA07F8EA0E18121C123812701370EA7FE0EAFF80EAE000A41310EA6038
EA70F0EA3FC0EA1F000D127B9113>101 D<13C0EA01E013C01380C7FCA6120E123FEA33801263
EAC700A21207120EA35A1340EA38C0A3EA3980EA3F00121E0B1C7D9B0D>105
D<13C01201A3EA0380A4EAFFE0A2EA0700A2120EA45AA31320EA3860A213C0A2EA1F80EA0F000B
1A7D990E>116 D<EA0F03EA1F871233EA6383A212C3EA0703A2EA0E06A31304130C121CEA0E18
1330EA07E0EA03C010127D9113>118 D E /Fb 1 27 df<1470EB01F0EB03E0EB0780EB0F0013
1E5B5BA25BB3AD485AA25B1203485A90C7FC120E123C12705A1270123C120E7E7F6C7E12017FA2
6C7EB3AD1378A27F7F7FEB0780EB03E0EB01F0EB007014637B811F>26 D
E /Fc 31 121 df<1238127C12FE12FFA2127F123B1203A212071206A2120C121C121812701220
08117CA210>39 D<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA21201
B3A7387FFFE0A213207C9F1C>49 D<EA03FCEA0FFF383C1FC0387007E0007C13F0EAFE0314F8A2
1301127CEA3803120014F0A2EB07E014C0EB0F80EB1F00133E13385BEBE018EA01C0EA0380EA07
00000E1338380FFFF05A5A5AB5FCA215207D9F1C>I<EA01FE3807FFC0380F07E0381E03F0123F
EB01F813811301EA1F03000C13F0120014E0EB07C0EB1F803801FE007F380007C0EB01F014F8EB
00FCA214FE127CA212FEA214FCEA7C01007813F8383C07F0380FFFC03803FE0017207E9F1C>I<
14E013011303A21307130F131FA21337137713E7EA01C71387EA03071207120E120C1218123812
7012E0B512FEA2380007E0A7EBFFFEA217207E9F1C>I<12601278387FFFFEA214FC14F8A214F0
38E0006014C038C00180EB0300A2EA00065B131C131813381378A25BA31201A31203A76C5A1722
7DA11C>55 D<D903FE138090381FFF819038FF01E33901F8003FD803E0131F4848130F48481307
121F48C71203A2481401127EA200FE91C7FCA8127EED0180127F7E15036C6C1400120F6C6C1306
D803F05B6C6C13386CB413F090381FFFC0D903FEC7FC21227DA128>67 D<D8FFF0EC0FFF6D5C00
0716E0D806FC1437A3017E1467A26D14C7A290391F800187A290390FC00307A3903807E006A290
3803F00CA2903801F818A3903800FC30A2EC7E60A2EC3FC0A2EC1F80A3EC0F00D8FFF091B5FC14
0630227EA135>77 D<D8FFF8EB1FFE7F0007EC00C07FEA06FF6D7E6D7E6D7E130F806D7E6D7E6D
7E130080EC7F80EC3FC0EC1FE0EC0FF0140715F8EC03FCEC01FEEC00FF157FA2153F151F150F15
071503A2D8FFF01301150027227EA12C>I<EB07FC90383FFF809038FC07E03903F001F848486C
7E4848137E48487FA248C7EA1F80A24815C0007E140FA200FE15E0A9007E15C0007F141FA26C15
806D133F001F15006C6C137E6C6C5B6C6C485A3900FC07E090383FFF80D907FCC7FC23227DA12A
>I<B6FC15E03907F007F0EC01FC1400157EA2157FA5157EA215FC1401EC07F090B512E0150001
F0C7FCADB57EA220227EA126>I<B53A0FFFF01FFEA2260FF00090C712E000076E14C0A26C6C91
38800180153F6D1503000103C01300A26C6C90387FE006156F7F6D9038C7F00CA20280EBF81C90
263F81831318A2D91FC36D5A150114E3903A0FE600FE60A202F6EBFFE0D907FC6D5AA201035D4A
133FA26D486DC7FCA20100141E4A130EA237227FA13A>87 D<EA07FC381FFF80383F0FC0EB07E0
130314F0121E1200A213FF1207EA1FC3EA3F03127E12FCA4EA7E07EB1DF8381FF8FF3807E07F18
167E951B>97 D<EBFF80000713E0380F83F0EA1F03123E127E387C01E090C7FC12FCA6127C127E
A2003E13306C1360380FC0E03807FF803800FE0014167E9519>99 D<EB03FEA2EB007EABEA01FC
EA07FF380F81FEEA1F00003E137E127E127C12FCA8127CA27E001E13FEEA0F833907FF7FC0EA01
FC1A237EA21F>I<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FC
C7FCA3127CA2127E003E13186C1330380FC0703803FFC0C6130015167E951A>I<3801FE1F0007
B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018
C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80
FC6CB45A000113C019217F951C>103 D<B47EA2121FABEB87E0EB9FF8EBB8FCEBE07CEBC07EA2
1380AE39FFF1FFC0A21A237EA21F>I<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2
EAFFF0A20C247FA30F>I<EAFF80A2121FB3ADEAFFF0A20C237FA20F>108
D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFF
F1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0
A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7007C137C
A26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEB
C03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A2
1A207E951F>I<EAFF1FEB3FC0381F67E013C7A3EB83C0EB8000ADEAFFF8A213167E9517>114
D<EA07F3EA1FFFEA780FEA7007EAF003A26CC7FCB4FC13F0EA7FFC6C7E6C7E120738003F80EAC0
0F130712E0A200F01300EAFC1EEAEFFCEAC7F011167E9516>I<13C0A41201A212031207120F12
1FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF
14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<3AFFE3FF87F8A23A1F807C00C0D80FC0EB01
80147E13E0000790387F030014DF01F05B00031486EBF18FD801F913CC13FB9038FF07DC6C14F8
EBFE03017E5BA2EB7C01013C5BEB380001185B25167F9528>119 D<39FFF07FC0A2390FC01C00
6C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D7E497E80EB67F013E33801C1F8380381FC
48C67E000E137E39FF81FFE0A21B167F951E>I E /Fd 30 122 df<EA01C01203EA0780EA0F00
121E5A123812781270A212F05AA77E1270A212781238123C7E7EEA0780EA03C012010A1D7A9914
>40 D<12E07E12787E7E7E7E13801203A213C01201A712031380A2120713005A121E5A5A5A5A0A
1D7D9914>I<1238127C127EA2123E121EA2127C12F81260070A7A8414>44
D<EA01C0487EA21360A2EA0770A4EA0630EA0E38A5EA1FFCA3EA1C1CEA3C1E38FE3F80A311177F
9614>65 D<EA03C6EA0FFE121FEA3E3EEA3C1EEA780E127012F0EAE000A7EAF00E12701278EA3C
1EEA3E3CEA1FF8EA0FF0EA03C00F177E9614>67 D<EAFFE013F87FEA383E131E7F7FA2EB0380A7
130714005B131E133EEAFFFC5B13E011177F9614>I<B51280A3EA1C03A490C7FC1338A2EA1FF8
A3EA1C38A290C7FCEB01C0A4B5FCA31217809614>I<B51280A3EA1C03A490C7FC1338A2EA1FF8
A3EA1C38A290C7FCA5B47EA311177F9614>I<38FE3F80A338380E00A7EA3FFEA3EA380EA738FE
3F80A311177F9614>72 D<EAFFFEA3EA0380B1EAFFFEA30F177E9614>I<EAFE7FA3EA383C1378
13F0A2EA39E0EA3BC01380EA3FC0A213E0123EEA3CF0EA387013781338133CA2EAFE3FA310177F
9614>75 D<EAFFC0A3001CC7FCADEB0380A4B5FCA311177F9614>I<38FE0FE0A3383B1B80A413
BBA2EA39B3A313F3EA38E3A21303A538FE0FE0A31317809614>I<38FE3F80A3383B0E00A4138E
1239A213CEA31238A213EE136EA4EAFE3EA311177F9614>I<EA1FF0EA7FFCA2EA701CEAF01EEA
E00EADEAF01EEA783CEA7FFCA2EA1FF00F177E9614>I<EAFFF813FE7FEA1C0FEB07801303A413
07EB0F00EA1FFF5B13F8001CC7FCA6B47EA31117809614>I<EAFFE07F7FEA383C7F130EA3131E
5BEA3FF85B7FEA383C131CA41480EB1DC0EAFE1F130FEB070012177F9614>82
D<EA0FCCEA3FFC127FEAF87CEAF03CEAE01CA2EAF000A2127EEA3FE0EA0FF8EA01FCEA003C131E
130E12E0A2EAF01EEAF83CEAFFFC13F8EAC7E00F177E9614>I<B51280A3EAE1C3A43801C000AD
EA0FF8A311177F9614>I<38FE3F80A338380E00A26C5AA56C5AA4EA0630EA0770A3EA0360A213
E0A26C5A11177F9614>86 D<EAFE7FA3EA1E38EA0E78EA0F70EA07F05B12035B120112037FA2EA
0770A2EA0E781338EA1C3C131C38FE3F80A311177F9614>88 D<38FE3F80A3383C1E00EA1C1CEA
1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>I<EA1FC0EA7FF07FEA783C
EA301CEA00FC120F123FEA7F1C12F012E0133CEAF07C387FFF80A2EA1F8F11107E8F14>97
D<EA03F0EA0FFC123FEA7C3CEA7818EAF0005AA4EAF00E1278EA7E1EEA3FFCEA0FF8EA03E00F10
7E8F14>99 D<133C13FE1201EA03DE138C1380A2EAFFFEA3EA0380AAEA7FFCA30F177F9614>
102 D<EAFB8EB51280A2EA3CF3A2EA38E3A838FEFBE0A31310808F14>109
D<EA07C0EA1FF0EA3FF8EA783CEA701CEAF01EEAE00EA4EAF01EEA701CEA7C7CEA3FF8EA1FF0EA
07C00F107E8F14>111 D<EAFCFCB47E7F381F0780EA1E03001C13C01301A4EA1E03EB0780EA1F
0FEBFF005BEA1CF890C7FCA5B47EA31218808F14>I<38FF1F80EB7FC013FF3807E080EBC0005B
A290C7FCA6EAFFFCA31210808F14>114 D<38FF3F80A3381C1C00A2120E5BA212071330A2EA03
70A26C5AA35BA3EA7B80127F90C7FC127E123C11187F8F14>121 D E /Fe
1 106 df<EA01C01203A2EA0180C7FCA6121E123F126712C7A212CE120EA25AA213C01238A2EA
3980EA3F00121E0A1A7F990D>105 D E /Ff 57 123 df<1218123C123E121E120EA3121E121C
123C127812F01260070D799816>39 D<13E01201EA07C013005A121E5A123812781270A312F05A
A77E1270A312781238123C7E7E7E13C0EA01E012000B217A9C16>I<12E07E127C121C121E7EEA
0780120313C01201A313E01200A7120113C0A3120313801207EA0F00121E121C127C12F05A0B21
7C9C16>I<EA01C0A4EA71C738F9CF80387FFF00EA1FFCEA07F0A2EA1FFCEA7FFF38F9CF803871
C700EA01C0A411127E9516>I<1238127C127EA2123E120E121E123C127C12F81260070B798416>
44 D<EB01801303130714005B130E131E131C133C13381378137013F05B12015BA212035B1207
90C7FC5A120E121E121C123C12381278127012F05AA211207E9C16>47 D<EA018012031207A212
1F127F12FF12731203AEEA7FF813FC13F80E197C9816>49 D<EA07E0EA1FF8EA7FFEEA783FEAF0
0FEB07801303A21200A2130714005B131E5B5B5BEA03E0EA078048C7FC381E0380123CEA7FFFB5
FC7E11197E9816>I<1238127CA312381200A81238127CA3123C121C123C123812F812F0126006
18799116>59 D<13E0487EA213B0A2EA03B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3
387F1FC000FF13E0007F13C013197F9816>65 D<EA7FF8EAFFFE6C7EEA1C0FEB07801303A31307
1400EA1FFF5BA2EA1C1FEB038014C01301A41303EB0780EA7FFFB51200EA7FFC12197F9816>I<
3801F180EA07FF5AEA1F0FEA3C0712781303127000F0C7FC5AA77E387003801278A2EA3C07381F
0F00EA0FFE6C5AEA01F011197E9816>I<EA7FF8EAFFFE6C7EEA1C0FEB0780EB03C01301A214E0
1300A8EB01C0A21303EB0780130F387FFF00485AEA7FF81319809816>I<387FFFC0B5FC7EEA1C
01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816>I<B512E0
A3EA1C00A41400A2131CA2EA1FFCA3EA1C1CA290C7FCA6B47E7F5B13197F9816>I<387F1FC038
FFBFE0387F1FC0381C0700A7EA1FFFA3EA1C07A9387F1FC038FFBFE0387F1FC013197F9816>72
D<EAFFFEA3EA0380B3EAFFFEA30F197D9816>I<387F0FE038FF8FF0387F0FE0381C0780EB0F00
131E131C133C5B5BEA1DE07F121F7F1338EA1E3C131CEA1C1E7F7F14801303387F07E038FF8FF0
387F07E01419809816>75 D<EAFFC0A3001CC7FCAE144014E0A4B5FCA313197F9816>I<38FC07
E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313197F9816
>I<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA213E7A31367A21377A21337A3
1317EA7F1FEAFF9FEA7F0F13197F9816>I<EA1FFC487E487EEA780F38F00780EAE003AEEAF007
A238780F00EA7FFF6C5A6C5A11197E9816>I<EA7FF8EAFFFE6C7E381C0F80130314C01301A313
031480130F381FFF005B13F8001CC7FCA7127F487E6CC7FC12197F9816>I<EA7FE0EAFFF86C7E
EA1C1E7F7FA45B131EEA1FFC5B7FEA1C3E130EA414201470A2387F0FF038FF87E0387F03C01419
7F9816>82 D<EA07E3EA1FFF127FEA781F487E487EA290C7FC7E1278EA7F80EA1FF0EA07FCC67E
130FEB07801303A212E0A2EAF00738F80F00EAFFFE5BEAC7F011197E9816>I<387FFFE0B5FCA2
EAE0E0A400001300AFEA07FC487E6C5A13197F9816>I<387F07F038FF8FF8387F07F0381C01C0
B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519809816>I<38FE0FE0EAFF1FEAFE0F3838
0380381C0700A4EA0E0EA4EA060CEA071CA4EA031813B8A3EA01B013F0A26C5A13197F9816>I<
387F1F80133F131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B8120713
1CA2EA0E0EA2487E387F1FC000FF13E0007F13C013197F9816>88 D<38FE0FE0EAFF1FEAFE0F38
1C0700A2EA0E0EA26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I<387FFF
80B5FCA238E007005B131E131CEA003C5B137013F0485A5B1203485A90C7FC5A381E0380121C12
3C12781270B5FCA311197E9816>I<B51280A311037E7E16>95 D<120C121E123C1278127012F0
12E0A312F012F812781230070D789B16>I<EA1FE0EA7FF87FEA783CEA301EEA000E133EEA07FE
123FEA7FCEEAF80E12E0A2EAF01EEAF83E387FFFE0EA3FF7EA0FC313127E9116>I<127E12FE12
7E120EA4133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E01381EB83C013FF
000E1300EA067C1419809816>I<EA03F8EA0FFE121FEA3C1EEA780CEA700012F05AA47EEA7007
1278EA3E0FEA1FFEEA0FFCEA03F010127D9116>I<133F5B7F1307A4EA03E7EA0FFF123FEA3C1F
487E1270EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E014197F9816>I<EA
07E0EA0FF8EA1FFCEA3C3EEA780EEA700FEAF007B5FCA3EAE0007EEA70071278EA3E1FEA1FFEEA
0FFCEA03F010127D9116>I<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FCA2EA01C0ACEA
3FFE487E6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3EA3C1E6C5A
EA1FFC485AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3EAF001387C
07C0383FFF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0FFF1480138713
03A2120EA9387FC7F038FFE7F8387FC7F01519809816>I<EA0180EA03C0A2EA0180C7FCA4EA7F
C0A31201ACEA7FFFB5FC7E101A7D9916>I<127E12FE127E120EA4EB7FE0A3EB0F00131E5B5B5B
EA0FF8A213BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816>107
D<EAFFC0A31201B3B51280A311197E9816>I<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870
AA38FE7CF8A31512809116>I<EA7E7CB5FC6C1380EA0F871303A2120EA9387FC7F038FFE7F838
7FC7F01512809116>I<EA03E0EA0FF8487EEA3C1E487EEA700738E00380A5EAF00700701300EA
780FEA3C1EEA1FFC6C5AEA03E011127E9116>I<EA7E3E38FEFF80007F13C0380F83E01301EB00
F0120E1470A4000F13F014E01381EB83C013FF000E1300137C90C7FCA6EA7FC0487E6C5A141B80
9116>I<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F9116>114
D<EA0FECEA3FFC127FEAF03CEAE01CA2EAF000EA7F80EA1FF0EA07FCEA003EEAE00EA212F0EAF8
1EEAFFFC13F8EAC7E00F127D9116>I<12035AA4EA7FFFB5FCA20007C7FCA75BEB0380A2130713
873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C
13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4
EA03B8A3EA01F0A26C5A13127F9116>I<38FF1FE013BF131F38380380A413E33819F300A213B3
EA1DB7A4EA0F1EA313127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813F012016C5A
12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>I<387F1FC038FF
9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35BA3485A1279
127BEA7F806CC7FC123C131B7F9116>I<383FFFC05AA238700780EB0F00131EC65A13F8485A48
5A485A48C7FC381E01C0123C1278B5FCA312127F9116>I E /Fg 58 124
df<EB7C3C3801FEFE38038FCF38070F8F000E1306EB0700A5B512F0A2380E0700AC387F0FF0A2
181A809916>11 D<137CEA01FEEA0387485A120E130690C7FCA4B5FCA2EA0E07AC387F0FE0A213
1A809915>I<EA60C0EAF1E0EAF9F01278EA1830A3EA3060EA70E0EAE1C0EA40800C0B7F9913>
34 D<126012F012F812781218A31230127012E01240050B7D990B>39 D<13C0EA0180EA030012
06120E120C5A1238A212301270A21260A212E0AA1260A21270A212301238A212187E120E12067E
EA0180EA00C00A267E9B0F>I<12C012607E7E121C120C7E1207A27E1380A21201A213C0AA1380
A21203A213005AA212065A121C12185A5A5A0A267E9B0F>I<126012F0A212701230A31260A212
C01240040B7D830B>44 D<EAFFC0A30A0380880D>I<126012F0A2126004047D830B>I<130CA213
1C1318A213381330A213701360A213E013C0A212011380A2120313005A1206A2120E120CA2121C
1218A212381230A212701260A212E05AA20E257E9B13>I<126012F0A212601200A8126012F0A2
126004107D8F0B>58 D<126012F0A212601200A8126012F0A212701230A31260A212C012400417
7D8F0B>I<EA1F80EA3FE0EA7070EAE03812F0A21260EA007013E0EA01C0EA03801300A21206A5
C7FCA41206120FA212060D1A7E9912>63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0
A348B47EA2130000061378A3487F121E39FF81FFC0A21A1B7F9A1D>65 D<EB3F023801FFC63803
E0EE3807003E000E131E5A003C130E123800781306127012F01400A61406127012781238003C13
0C121C6C13186C13303803E0E03801FFC038003F00171C7E9A1C>67 D<B57E14E0380F00F01438
8080A2140F801580A81500A25C140E5C143C14F8B512E01480191A7F991D>I<B512F8A2380F00
7814381418141C140C1303A214005B13FFA213077FA214061300140CA3141CA21478B512F8A217
1A7F991A>I<B512F0A2EA0F001470143014381418A213061400A2130E13FEA2130E1306A390C7
FCA6EAFFF8A2151A7F9919>I<EB3F023801FFC63803E0EE3807003E000E131E5A003C130E1238
00781306127012F091C7FCA5903803FFC0A23970001E0012781238123C121C7E3807803E3803E0
763801FFE2D8003FC7FC1A1C7E9A1E>I<39FFF3FFC0A2390F003C00A9EBFFFCA2EB003CAB39FF
F3FFC0A21A1A7F991D>I<EAFFF0A2EA0F00B3A4EAFFF0A20C1A7F990E>I<39FFF07FC0A2390F00
3E0014185C5C5C495A49C7FC13065B131E133F136FEBC780EB87C013036D7E8013001478A28014
3E39FFF0FFC0A21A1A7F991E>75 D<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3EB78
18A3EB3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>77
D<B4EBFFC07F000FEB1E00EBC00CEA0DE0A2EA0CF013F81378133C133E131E130FA2EB078CEB03
CCA2EB01ECEB00FCA2147CA2143C001E131CEAFFC0140C1A1A7F991D>I<137F3801FFC03807C1
F0380F0078001E7F001C131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C13
1C001E133C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>I<B5128014E0380F00F014781438
143CA51438147814F0EBFFE0148090C7FCA9EAFFF0A2161A7F991A>I<EAFFFEEBFFC0380F01E0
6D7E14701478A4147014F0495AEBFFC05CEB03C06D7E6D7EA615C0A238FFF078EC3F80C7EA0F00
1A1B7F991C>82 D<EA07C2EA1FF6EA383EEA701EEA600E12E01306A3EAF0001278EA7F80EA3FF0
EA1FF8EA0FFCEA01FEEA001E130F130712C0A312E01306EAF00EEAFC1CEACFF8EA83E0101C7E9A
15>I<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FCAF3803FFF0A2191A
7F991C>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC5DD807801306EB819E
A2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE13F0EBFC03017C5BA2
EB7801A201385BEB3000241B7F9927>87 D<EA1020EA3870EA70E0EA60C0EAC180A3EAF1E0EAF9
F01278EA30600C0B7B9913>92 D<EA3F80EAFFC0EAF0E0137012601200EA0FF0123FEA787012E0
A21373A2EA70F3EA7FFEEA3E3C10107E8F13>97 D<12FCA2121CA813F8EA1FFE130F381C0380A2
EB01C0A6EB0380A2381F0F00EA1BFEEA18F8121A7F9915>I<EA07F0EA0FFCEA3C3C1230EA7018
EAE000A61270EA380CEA3C18EA0FF8EA07E00E107F8F11>I<137EA2130EA8EA07CEEA1FFEEA3C
1EEA700EA212E0A61270131EEA3C3E381FFFC0EA07CF121A7F9915>I<EA07C0EA1FF0EA3838EA
7018131CEAFFFCA2EAE000A41270EA300CEA3C18EA0FF0EA07E00E107F8F11>I<13F8EA01FCEA
03BCEA073CEA0E181300A5EAFFC0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA
3877EA7038A5EA3870EA3FE0EA7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781E
EA1FF8EA07E010197F9013>I<12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A
7F9915>I<1218123CA21218C7FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00
C01300A6EA07E0A21200B0126012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F
80A2EB3C0013305B5B121DEA1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12
FCA2121CB3A4EAFF80A2091A80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301
AA39FF9FE7F8A21D107F8F20>I<EAFCF8B47EEA1F1EEA1E0E121CAA38FF9FC0A212107F8F15>I<
EA07E0EA1FF8EA381CEA700EEA6006EAE007A6EA700EA2EA3C3CEA1FF8EA07E010107F8F13>I<
EAFCF8EAFFFEEA1F0F381C07801303EB01C0A6EB03801307381F0F0013FEEA1CF890C7FCA5B47E
A212177F8F15>I<EA07C6EA1FF6EA3C3EEA781EEA700E12E0A61270EA781EEA3C3EEA1FEEEA07
CEEA000EA5EB7FC0A212177F8F14>I<EAFDE0EAFFF0EA1F70EA1E20EA1C00AAEAFFC0A20C107F
8F0F>I<EA1F20EA7FE01270EAE060A2EAF000127FEA3FC0EA1FE0EA01F0EAC070A212E0EAF0E0
EAFFC0EA8F800C107F8F0F>I<120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B
177F960F>I<EAFC7EA2EA1C0EAA131E133E380FFFC0EA07CF12107F8F15>I<38FF3F80A2381C0E
00130CA26C5AA21338EA0730A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00
381C3E0C13361337000E5B13631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1F
C0A2380E1E00EA071CEA03B813B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A213
10808F14>I<38FF3F80A2381C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA21261
00F3C7FC12C712FE127811177F8F14>I<EA7FF8A2EA7070EA60E0126113C0EA63801207EA0F18
120E121CEA3C38EA3830EA7070EAFFF0A20D107F8F11>I<B512C0A21202808A13>I
E /Fh 8 118 df<127812FCA412781200A5127812FCA4127806117D900C>58
D<B512F014FC380FC03FEC0F80EC07C0EC03E0A2EC01F0A215F8A815F0A2140315E0EC07C0EC0F
80EC3F00B512FE14F01D1A7E9922>68 D<EA03FCEA0FFEEA1E1F123C127CEA780E00F8C7FCA512
78127C383C0180381F0300EA0FFEEA03F811117F9014>99 D<121E123FA4121EC7FCA4127FA212
1FADEAFFC0A20A1B809A0C>105 D<EAFF1FEB7FC0EA1FC3EB83E0A21303AA38FFE7FCA216117F
9019>110 D<EA03F8EA0FFE383E0F80EA3C07387803C0A200F813E0A6007813C0EA7C07383E0F
80380FFE00EA03F813117F9016>I<EA1FB0EA3FF0EA7070EAE030A2EAF800EAFFC0EA7FF0EA3F
F8EA1FFC1201EAC03C131C12E0EAF038EAFFF0EACFC00E117F9011>115
D<38FF1FE0A2EA1F03AB1307130F380FFFFCEA03F316117F9019>117 D
E /Fi 3 34 df<B61280A219027D8A20>0 D<127012F8A3127005057D8C0C>I<1506A281168015
01ED00E0B712F816F0C912E0ED0180150316001506A2250E7E902A>33 D
E /Fj 11 123 df<127012F8A3127005057D840C>58 D<127012F012F8A212781218A31230A212
7012601240050D7D840C>I<EC0380EC0F00143C14F0EB03C0010FC7FC133C13F0EA03C0000FC8
FC123C12F0A2123C120FEA03C0EA00F0133C130FEB03C0EB00F0143C140FEC038019187D9520>
I<13F8EA03FEEA0F06121E123C1238EA781CEA7FF8EAFFE0EAF000A4EA7004130EEA383CEA1FF0
EA0FC00F127F9113>101 D<EB01E0EB07F0EB0E7814F0EB1C601400A3133C1338A23803FFC014
80380038005BA55BA5485AA5485AA30063C7FC12F712F612FC127815257E9C14>I<13E01201A2
EA00C01300A6120EEA1F8012331263124312C3EA0700A2120EA3EA1C201360A2EA38E013C0EA1F
80EA0F000B1C7F9B0E>105 D<1307130FA213061300A613F0EA01FCEA031C1206A2120C1200A2
1338A41370A413E0A4EA01C01261EAF380EAF70012FE127C1024809B11>I<391C0FC0F8393F1F
E3FC396730770EEB607C38C7C078138038070070A2000E495AA3ED3840261C01C013C0A2157116
803A3803803F00D81801131E22127F9124>109 D<381C0F80383F3FC0386770E013C0EAC780A2
EA0700A2380E01C0A3EB0384001C138CA2EB071C1418383803F0381801E016127F9119>I<EA01
F0EA03FCEA070CEA0E1EEA1C3C1318EA1E00EA1FE0EA0FF0EA07F8EA0078EA203CEA703812F013
30EAE0F0EA7FE0EA1F800F127E9113>115 D<3801E180EA07F1380FFF00EA0C1EEA180CC65A5B
5B5B485A48C7FCEA0601485A1218EA3F0EEA7FFCEA61F8EAC0F011127F9113>122
D E /Fk 42 121 df<1238127C12FCA2127C120CA21218A21230126012C01280060D799C0C>39
D<1230127812F81278127005057D840C>46 D<13181370EA03F0120FEA1C701200A213E0A6EA01
C0A6EA0380A61207EAFFF8A20D1C7C9B15>49 D<137E3801FF80380387C0EA0603EB01E0120FA2
121FA2120E1200EB03C0A2EB0780EB0F00130E5B5B13605BEA038038070180120E381C03001230
EA7FFF485AA2131C7E9B15>I<137C48B4FC38038780EA0603000F13C0A3380E07801200140013
0E133CEA01F8485AEA001C7F130FA4127012F8A2EAF01E12E05BEA7078EA3FE0EA0F80121D7D9B
15>I<EB01C0EB03801307130FA2131B1333EB770013E713C7EA0187EA03071207EA0E0E120C12
1812301260387FFFE0B512C038001C00A5133C3803FF80A2131C7E9B15>I<133E13FF3801C380
EA0383EA0707120E001C130090C7FC123C1238EA79F8EA7FFCEA7C1C487EA2EAF00FA4EAE01EA4
131C5BEA70705BEA3FC0EA0F80111D7C9B15>54 D<137C48B4FC38038780EA0703120EEB01C0A2
381E0380A2EB0700EA0F8613DCEA07F81203487EEA0E7E487EEA380F12707F12E0A31306130EEA
701CEA3878EA1FF0EA0FC0121D7D9B15>56 D<14301470A214F0801301A2EB0378A21306147CEB
0C3CA21318A21330A2497EA2EBFFFEA23801801EA23803001F801206120F397FC0FFF012FF1C1D
7F9C1F>65 D<903807F030EB1FFC90387C0E603901F003E0EA03C038078001EA0F00000E130000
1E14C05AA25A1500A25AA5EC0180EC03001270127814066C5B001C131C000F5B3807C0E06CB45A
D8007EC7FC1C1E7C9C1E>67 D<380FFFFCECFF803900F007C0EC01E0EC00F015701578485A1538
A3153C1538485A1578A3157015F0484813E0140115C0EC03801407EC0E00380F007CB512F014C0
1E1C7E9B20>I<000FB512E0A23800F003EC01C01400A33801E060A31500EBE1E013FF485B13C1
13C0A215C0ECC18038078001A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>I<000FB512
C0A23800F007EC03801401A3EA01E01461A2EC600014E013E148B45AA213C113C0A33807818001
80C7FCA5120FEAFFFC5B1A1C7E9B1B>I<903807F030EB1FFC90387C0E603901F003E0EA03C038
078001EA0F00000E1300001E14C05AA25A1500A25AA3ECFFF0A2EC0F80150012701278A27E001C
5B000F133E3807C0F63803FFC2C66CC7FC1C1E7C9C21>I<390FFF9FFE143F3900F003C0A53901
E00780A590B5FC481400EBC00FA53807801EA6000F133E39FFF3FFE015C01F1C7E9B1F>I<380F
FF80A23800F000A5485AA6485AA6485AA6120FEAFFF8A2111C7F9B0F>I<390FFF87FEA23900F0
03E0EC0180EC030014065C48485A14705CEBE1C013E313E73803CDE013D9EBF1F013E013C01478
EA078080A280A280000FEB1F8039FFF87FF013F01F1C7E9B20>75 D<380FFFC0A23800F000A548
5AA6485AA41406140CEA0780A2141C141814381478380F01F8B512F0A2171C7E9B1A>I<D80FFC
EB0FFCA20000EC1F80A2153713DE1567D8019EEB6F0015CFA2EC018F138FEC030FD8030F131E14
06A2EB078CA2149800065C14B0A2EB03E0A214C0001E147C3AFFE387FFC001C31480261C7E9B26
>I<390FF80FFEA23900FC01E0EC00C0A213DEA239018F0180A214811387A2EB83C1390303C300
EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38FFE00C13C01F1C7E9B1F>I<EB07F0EB
3FFCEBF81E3801E0073903C00380D8070013C0481301001E14E0121C003C13001238007814F0A3
48EB01E0A4EC03C0A30070EB0780007814005C6C131E5C6C5B380F83E03803FF80C648C7FC1C1E
7C9C20>I<380FFFFC14FF3900F00F80140315C01401A23801E003A315801407EC0F003803C03E
EBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B1C>I<380FFFF814FE3900F00F801403
15C01401A23801E003A3EC0780EC0F00143E3803FFF85CEBC07880A2143E3807803CA51518000F
1430EAFFF89038F01FE0C7EA07C01D1D7E9B1F>82 D<EB3F0CEBFFCC3801C1F838030078481338
120EA3001E13301400A26C7E13F86CB4FC6C13806C13C038003FE0130313011300A21260A3EB01
C0127038F8038038FE0F00EACFFEEA83F8161E7E9C17>I<001FB512F05A383C07810038EB8060
12701260A2EB0F0012C0A200001400A3131EA65BA6137C381FFFE0A21C1C7C9B1E>I<39FFF8FF
E0A2390F001E00140CA4001E5BA6485BA6485BA300385BA2383C0180D81C03C7FCEA0E0EEA07FC
EA01F01B1D7A9B1F>I<3BFFF0FFE1FF805B3B1F001F007C006C15384A13301670026F1360A202
CF5BA23A07818F018015819026838783C7FC140715861386158E01CC138C120301D813D8140301
F013F0A25D13E05DEA01C05DEB8001291D7B9B2B>87 D<3907FF8FFEA239007C03E0013C138001
3E1300EB1E06140EEB1F1CEB0F1814B0EB07E0A26D5A80A2497E130E130C497EEB3078EB707CEB
E03CEBC03E3801801E3803001FEA0F8039FFE07FF014FF1F1C7F9B1F>I<EA03F8EA0FFC130EEA
0E07120C12005BEA03FF121FEA3E0E127812F01460A2131E38F87EC0EA7FEF381F8F0013127E91
15>97 D<13FCEA03FEEA0F0FEA1E0EEA3C0CEA38001278A35AA31270EA7806EA380CEA3E18EA1F
F0EA07E010127E9112>99 D<EB01F81303EB0070A614E0A3EA01F8EA03FEEA0F87381E01C0123C
12381278A338F00380A312701307EA780FEA3C1F381FF7E0EA07C7151D7E9C17>I<EA01F8EA07
FCEA0F0EEA1C0FEA3C071238EA7FFFA2EA78005AA31270EA7806EA380CEA3C18EA1FF0EA07C010
127E9112>I<EA01C0EA03E0A213C0EA0180C7FCA6EA1F80A212071300A6120EA65AEAFF80A20B
1D7F9C0C>105 D<391F8FC0FC9038BFE3FE3907E0F60F9038C07C07EB8078EB0070A4000EEBE0
0EA64848485A3AFF8FF8FF80A221127F9124>109 D<381F8F80EBBFC03807E0E013C013801300
A4380E01C0A6381C038038FF8FF0139F14127F9117>I<13FCEA03FF380F0780381E03C0EA3C01
1238007813E0A338F003C0A300701380130738780F00EA3C1EEA1FF8EA07E013127E9115>I<38
0FC7E0EBFFF03803F078EBC03C1380A2141EA33807003CA31478A2EB80F0380FC3E0380EFF80EB
3E0090C7FCA35AA3B47EA2171A809117>I<EA1FBC13FEEA07CF138E138C1300A4120EA6121EEA
FFC0A210127F9110>114 D<EA03F2EA0FFEEA1E1EEA1C0C1238A2EA3E00EA1FE0EA0FF0EA07F8
EA007CEA601CA3EA7038EAF870EADFE0EA8FC00F127F9110>I<1206A3120EA2120C121C123CEA
FFE0A2EA1C005AA65A13C0A4EA7180A2EA7F00123C0B1A7C9910>I<38FC1F80A2EA3C07003813
00A6EA700EA4131E133E137E383FDF80EA1F1F11127C9117>I<381FE3FCEA3FE7380383C03801
C380EBC700EA00EE13EC13781370137813F8EA01DCEA039EEA070EEA0607121E38FF1FF0A21612
7F9116>120 D E /Fl 51 123 df<13301360EA01C0EA038013005A120E121E121C123CA21238
1278A312F85AA97E1278A31238123CA2121C121E120E7E7E1380EA01C0EA006013300C297D9E13
>40 D<12C0126012387E120C120E7E1380120313C0A2120113E0A313F01200A9120113E0A313C0
1203A2138012071300120E120C121C5A12605A0C297D9E13>I<1238127C12FE12FFA2127F123B
120312071206A2120C121812301220080F7D860D>44 D<1360EA01E0120F12FF12F31203B3A238
7FFF80A2111B7D9A18>49 D<EA07F8EA1FFEEA383F38781F8038FC0FC014E01307A21278EA000F
14C0A2EB1F801400133E13785B485A38038060130012064813E0381FFFC05A5AB5FCA2131B7E9A
18>I<EA0FF0EA3FFCEA703EEAF03F12F8A3EA203EEA007C13F813F0EA01C0A21380A5C7FCA4EA
0380EA07C0EA0FE0A3EA07C0EA0380101D7D9C17>63 D<EB0380A2497EA3497EA2497E131BA2EB
31F8A2EB61FC1360EBE0FEEBC07EA248487E90B5FC481480EB001FA20006EB0FC0A2000E14E000
0C130739FFC07FFEA21F1C7E9B24>65 D<B512F814FF390FC01F80EC0FC0A215E0A515C0EC1F80
90B512005C9038C03F80EC0FC0EC07E0A215F0A515E0140FEC1FC0B61280ECFC001C1C7E9B22>
I<90381FE0209038FFF8E03803F80F3807E007380FC001EA1F8048C7FCA2007E1460A212FE1500
A7007E1460A27E15C06C7E390FC001803907E003003803F80E3800FFFCEB1FE01B1C7D9B22>I<
B512F814FF390FC01FC0EC07E0EC03F0EC01F8A2EC00FCA315FEA815FCA3EC01F8A2EC03F0EC07
E0EC1FC0B6120014F81F1C7E9B25>I<B6FCA2380FC01F1407801580A2EBC181A3EC800013C313
FFA213C313C1A21560A29038C000E015C0A21401A21403EC0F80B6FCA21B1C7E9B1F>I<B6FCA2
380FC01F1407801580A21401EBC181A2EC8000A213C313FFA213C313C1A401C0C7FCA6B5FCA219
1C7E9B1E>I<B5FCA2EA07E0B3A6B5FCA2101C7F9B12>73 D<39FFFC07FFA2390FC000E0EC01C0
EC0380EC06005C14385C5CEBC1C013C3EBC7E013CFEBFBF0EBF3F813E1EBC0FC80147E80EC1F80
15C0140FEC07E015F039FFFC3FFFA2201C7E9B25>75 D<B5FCA2EA0FC0AF1403A31407A2140614
0E141E147EB512FEA2181C7E9B1D>I<D8FFC0EB03FF6D5B000F15F0D80DF0130DA3D80CF81319
A2017C1331A26D1361A26D13C1A390380F8181A2903807C301A2EB03E6A2EB01FCA3EB00F8A23A
FFC0701FFFA2281C7E9B2D>I<39FFE003FFA2390FF000307FEA0DFCEA0CFE137E7FEB1F8014C0
EB0FE0EB07F01303EB01F814FCEB00FE147F143FEC1FB015F0140F1407140314011400A2D8FFC0
13701530201C7E9B25>I<EB3FE03801FFFC3803F07E390FC01F80391F800FC0EB00074814E048
14F0007E1303A200FE14F8A8007E14F0A2007F13076C14E0EB800F001F14C0390FC01F803903F0
7E006CB45A38003FE01D1C7D9B24>I<B512F814FE390FC03F80141FEC0FC015E0A615C0EC1F80
143F9038FFFE0014F801C0C7FCAAEAFFFCA21B1C7E9B21>I<B512F014FE380FC03F6E7E6E7E81
A55D4A5A4AC7FCEBFFFE14F0EBC0F8147C80143FA381A3EDC180EC1FE33AFFFC0FFF00EC03FE21
1C7E9B24>82 D<3807F860381FFEE0EA3C07EA7801EA700012F01460A26C130012FEEAFFE0EA7F
FE6C7E1480000F13C06C13E0EA007FEB03F01301130012C0A214E07E38F001C0EAFC0338EFFF00
EAC3FC141C7D9B1B>I<007FB512E0A238781F81007013800060146000E0147000C01430A40000
1400B03807FFFEA21C1C7E9B21>I<39FFFC03FFA2390FC00030B3120715606C6C13E03901F001
C03900FC078090387FFE00EB0FF8201C7E9B25>I<3AFFFC01FF80A23A0FC00018006C6C5BA26D
1370000314606D13E000015C7F0000495AA2D97E03C7FCA2EB7F07EB3F06148EEB1F8C14CCEB0F
D8A2EB07F0A36D5AA26D5AA2211C7F9B24>I<397FFE1FFEA23907F001803803F8036C6C48C7FC
000013066D5AEB7F1C6D5A14B0EB1FE0130FA26D7E6D7E1307497EEB0CFEEB187EEB387F496C7E
EB601F01C07F00016D7E496C7EEA03003AFFF03FFF80A2211C7F9B24>88
D<3AFFFC01FF80A23A0FE00038006C6C13306C6C137015606C6C5B3800FE015DD97F03C7FCEB3F
871486EB1FCEEB0FFC5C13076D5AAAEB3FFFA2211C7F9B24>I<387FFFFCA2387E01F8EA780300
7013F038E007E0130F00C013C0131F148038003F005B137E13FE485A5B0003130613F0EA07E012
0FEBC00E121FEB801CEA3F0048133C007E13FCB5FCA2171C7D9B1D>I<EA0FF8EA1FFE383E1F80
130714C0121C1200EA03FF121FEA3F87EA7E0712FCA3130FEA7E1F383FFBF8EA0FE115127F9117
>97 D<B4FCA2121FA9EB1FC0EBFFE0EBC1F81300147CA2147EA6147CA2EB80F8EBC1F0381C7FE0
38181F80171D7F9C1B>I<EA03FCEA0FFEEA1F1F123E127C130E00FCC7FCA6127C387E0180EA3E
03381F0700EA0FFEEA03F811127E9115>I<EB0FF0A21301A9EA03F1EA0FFFEA1F07EA3E01127C
A212FCA6127CA2EA3E03EA1F0F380FFDFEEA03F1171D7E9C1B>I<EA01FCEA0FFF381F0F80383E
07C0EA7C0314E012FCB5FCA200FCC7FCA3127C007E1360003E13C0EA1F81380FFF00EA01FC1312
7F9116>I<133F3801FF803803E7C0EA07C7EA0F87EB8380EB8000A5EAFFF8A2EA0F80AEEA7FF8
A2121D809C0F>I<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E1FEA1FFC
EA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F0383E03E038
1FFFC03803FE00151B7F9118>I<B4FCA2121FA9EB1FC0EB7FE0EBE1F0EB80F8A21300AB38FFE7
FFA2181D7F9C1B>I<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>I<B4
FCA2121FA9EB0FF8A2EB0380EB0700130C5B137013F87F13BC133E7F7F1480EB07C0EB03E038FF
C7FCA2161D7F9C19>107 D<B4FCA2121FB3A7EAFFE0A20B1D7F9C0E>I<39FF1FC0FE90387FE3FF
3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1FC0EB7FE038
1FE1F0EB80F8A21300AB38FFE7FFA218127F911B>I<EA01FC380FFF80381F07C0383E03E0387C
01F0A200FC13F8A6007C13F0A2383E03E0381F07C0380FFF803801FC0015127F9118>I<38FF1F
C0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090C7FCA6EAFF
E0A2171A7F911B>I<EAFE3E137F381ECF80EA1F8FA2EB070090C7FCAAEAFFF0A211127F9114>
114 D<EA1FD8EA3FF8EA7038EAE018A2EAF000EAFF80EA7FE013F0EA1FF8EA07FCEA007CEAC01C
A212E0EAF038EAFFF0EACFC00E127E9113>I<1203A35AA25AA2123FEAFFFCA2EA1F00A9130CA4
EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA03F818127F
911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C
5AA21378A2133016127F9119>I<39FF8FF8FEA2391F03E030A201831370000FEBF0601386D807
C613C0EBCEF8EBEC790003EB7D80EBF83D0001EB3F00A2497E0000131EEBE00EA21F127F9122>
I<38FFC7FCA2381F8180EA0F833807C700EA03EEEA01FC5B1200137C13FEEA01DFEA039F38070F
80380607C0380C03E038FF07FCA216127F9119>I<38FFC1FCA2381F0060EB80E0000F13C013C0
3807C180A23803E300A2EA01F6A213FE6C5AA21378A21330A25B1270EAF8E0EAC0C0EAE380007F
C7FC123E161A7F9119>I<383FFF80A2383C1F00EA303EEA707EEA60FC5BEA61F012033807E180
13C1EA0F81EA1F83383F0300EA3E07485AB5FCA211127F9115>I E /Fm
70 124 df<EB3F0F9038FFBF803903C3F3C0380703E3ECC180390E01C000A6B512FCA2380E01C0
AE387F87FCA21A1D809C18>11 D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B512C0A2
EA0E01AE387F87F8A2151D809C17>I<90383F03F09038FFCFF83903C0FC1C390701F03CA2390E
00E01892C7FCA5B612FCA2390E00E01CAE3A7FC7FCFF80A2211D809C23>14
D<EA7070EAF8F8EAFCFCA2EA7C7CEA0C0CA3EA1818A2EA3030EA6060EA40400E0D7F9C15>34
D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C>39 D<13C0EA0180EA030012
06120E120C121C121812381230A21270A21260A212E0AC1260A21270A21230A212381218121C12
0C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C120C120E120612077EA21380
A21201A213C0AC1380A21203A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>I<1306
ADB612E0A2D80006C7FCAD1B1C7E9720>43 D<127012F012F8A212781218A31230A21270126012
40050D7D840C>I<EAFFE0A30B0380890E>I<127012F8A3127005057D840C>I<1303A213071306
A2130E130C131C1318A213381330A213701360A213E013C0A21201138012031300A25A1206A212
0E120CA2121C1218A21238123012701260A212E05AA210297E9E15>I<EA03C0EA0FF0EA1C38EA
381CA2EA700EA3EAF00FADEA700EA3EA381CA2EA1C38EA0FF0EA07E0101D7E9B15>I<12035A12
3FB4FC12C71207B3A3EAFFF8A20D1C7C9B15>I<EA07C0EA1FF0EA3878EA603C131E12F0EAF80F
A312701200130E131E131C133C1378137013E0EA01C0EA0380EA0700EA0E03120C1218EA3006EA
7FFE12FFA2101C7E9B15>I<EA07E0EA1FF0EA3838EA301CEA781EA312381200133C13381370EA
07E0A2EA0038131C131E130E130FA2127012F8A2130EEAF01EEA601CEA3838EA1FF0EA07C0101D
7E9B15>I<131CA2133C137CA213DC1201139C1203EA071C1206120E120C121812381230126012
E0B512C0A238001C00A63801FFC0A2121C7F9B15>I<EA300CEA3FFC13F813E0EA3000A6EA33E0
EA37F0EA3C38EA381CEA301E130EEA000FA4127012F0A2EAE00EEA601E131CEA3878EA1FF0EA07
C0101D7E9B15>I<13F0EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EA
FC1CEAF81E130E12F0130FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>I<1260
387FFF80A21400EA6003EAC0065BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B
15>I<EA03E0EA0FF0EA1C38EA381CEA300E1270A31278131C123EEA3FB8EA1FE0EA07F0EA0FF8
EA1CFCEA387EEA701E130F12E01307A4EA700E130CEA3C3CEA1FF0EA07E0101D7E9B15>I<EA03
C0EA0FF0EA1C38EA381C1278EA700E12F0A2130FA51270131F1278EA383FEA1FEFEA0FCFEA000E
A3EA301C12781338EA7030EA30F0EA3FC0EA0F80101D7E9B15>I<127012F8A312701200A81270
12F8A3127005127D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A21270
12601240051A7D910C>I<007FB512C0B612E0C9FCA8B612E06C14C01B0C7E8F20>61
D<1306130FA3497EA4EB33C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3
001F131F39FFC0FFF0A21C1D7F9C1F>65 D<B512C014F0380F00F8143C141C141EA4141C143C14
78EBFFF014E0EB00F8143C141EA2140FA5141E143E147CB512F814C0181C7E9B1D>I<90381F80
80EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091C7FCA70070
EB01801278A27E001CEB0300121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E9C1E>I<B5
12C014F0380F007C141E8080EC038015C0A2140115E0A815C0A2140315801407EC0F00141E147C
B512F014C01B1C7E9B20>I<B512FCA2380F007C141C140C140E14061303A314005B13FFA21307
7FA21403A213001406A3140E141E147CB512FCA2181C7E9B1C>I<B512F8A2380F007814381418
141C140CA21303A21400A25B13FFA213077FA490C7FCA6EAFFF8A2161C7E9B1B>I<39FFF3FFC0
A2390F003C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>72 D<EAFFF0A2EA0F00B3A6EA
FFF0A20C1C7F9B0F>I<EAFFF8A2000FC7FCAF1418A414381430147014F01301B5FCA2151C7E9B
1A>76 D<B46CEBFF806D5A000FECF800A2390DE00378A3380CF006A3EB780CA3EB3C18A3EB1E30
A3EB0F60A3EB07C0A3381E03803AFFC387FF80A2211C7E9B26>I<B4EBFFC07F000FEB1E00EBC0
0CA2EA0DE0EA0CF0A21378A2133C131EA2130FA2EB078C14CC1303EB01ECA2EB00FCA2147C143C
A2001E131CEAFFC0140C1A1C7E9B1F>I<EB3F80EBFFE03803E0F83807803C380E000E001E130F
48EB078000381303007814C0A20070130100F014E0A80078EB03C0A36CEB0780A26CEB0F006C13
1E6C6C5A3803E0F83800FFE0EB3F801B1E7E9C20>I<B5128014E0380F00F01438143C141EA614
3C143814F0EBFFE0148090C7FCAAEAFFF0A2171C7E9B1C>I<B5FC14E0380F00F01438143C80A5
5C143814F0EBFFE05CEB01E06D7E1478A71530143839FFF03C60EC1FE0C7EA07C01C1D7E9B1F>
82 D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA
07FFC61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F8121E7E9C
17>I<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B03803FFFC
A21B1C7F9B1E>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA20180147000079038678060A3
2603C0E713C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13F6017E137E
A3013C133CA3011C133801181318281D7F9B2B>87 D<EA0808EA1818EA3030EA6060A2EAC0C0A3
EAF8F8EAFCFCA2EA7C7CEA38380E0D7B9C15>92 D<EA0FE0EA1FF8EA3C3C7FEA180E1200131EEA
07FE121FEA3E0E127812F01460A2131EEA783E383FFFC0381F878013127F9115>97
D<12FCA2121CA9137EEA1DFF381F8780381E01C0001C13E0130014F0A614E01301001E13C0381F
07803819FF00EA187C141D7F9C17>I<EA03F0EA0FF8EA1E3C1238EA7818EA700012F0A6127813
06123CEA1E0CEA0FF8EA03E00F127F9112>I<EB1F80A21303A9EA03E3EA0FFBEA1E0FEA3807EA
7803127012F0A612701278EA3807EA1E1F380FFBF0EA07E3141D7F9C17>I<EA03E0EA0FF0EA1C
38EA381CEA781EEA700EEAFFFEA2EAF000A41270EA7806123CEA1E0CEA0FF8EA03E00F127F9112
>I<1378EA01FCEA039EEA071EEA0E0C1300A6EAFFE0A2EA0E00AEEA7FE0A20F1D809C0D>I<EB03
803807E7C0EA0FFDEA3C3D38381C00EA781EA4EA381CEA3C3CEA3FF0EA37E00070C7FCA21230EA
3FFC6CB4FC481380EA700738E001C0A438700380383C0F00EA1FFEEA07F8121C7F9215>I<12FC
A2121CA9137CEA1DFFEA1F07381E0380A2121CAB38FF9FF0A2141D7F9C17>I<1218123C127C12
3C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C0C>I<EA01C0EA03E0A3EA01C0C7FCA6EA0FE0
A21200B31260EAF1C0A2EA7F80EA3E000B25839C0D>I<12FCA2121CA9EB7FC0A2EB3E0013185B
5B5BEA1DE0121FEA1E70EA1C781338133C131C7F130F38FF9FE0A2131D7F9C16>I<12FCA2121C
B3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF9FF8391F83B838391E01E01CA2001C13C0AB
3AFF8FF8FF80A221127F9124>I<EAFC7CEAFDFFEA1F07381E0380A2121CAB38FF9FF0A214127F
9117>I<EA03F0EA0FFCEA1E1EEA380700781380EA700300F013C0A600701380EA780700381300
EA1E1EEA0FFCEA03F012127F9115>I<EAFC7EEAFDFF381F8780381E03C0381C01E0A2EB00F0A6
EB01E0A2381E03C0381F0780381DFF00EA1C7C90C7FCA6B47EA2141A7F9117>I<3803E180EA0F
F9EA1E1FEA3C071278130312F0A612781307123CEA1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A
7F9116>I<EAFDE0EAFFF0EA1F78121E1330EA1C00ABEAFFC0A20D127F9110>I<EA1F90EA3FF0EA
7070EAE030A3EAF800EA7F80EA3FE0EA0FF0EA00F8EAC038A212E0A2EAF070EADFE0EA8FC00D12
7F9110>I<120CA5121CA2123CEAFFE0A2EA1C00A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<
38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA03E314127F9117>I<38FF0FE0A2381C0780EB03
00EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA313127F9116>I<39FF3FCFE0A2391C0F07
80EC0300131F380E1B061486A2EB318E000713CCA213603803E0F8A33801C070A31B127F911E>
I<387F8FF0A2380F078038070600EA038EEA01DC13D8EA00F01370137813F8EA01DCEA038E130E
EA0607380F038038FF8FF8A21512809116>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA213
1CEA0398A213F86C5AA26C5AA35BA3EAF180A200C7C7FC127E123C131A7F9116>I<EA7FFCA2EA
7838EA7070EA60F013E0EA61C01263EA0380EA070C120F120EEA1C1CEA3C181238EA7078EAFFF8
A20E127F9112>I<B512F0A21402808B15>I E /Fn 35 121 df<121C127FEAFF80A213C0A3127F
121C1200A212011380A21203EA07001206120E5A5A12300A157BA913>39
D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FEA317277BA622>49
D<EBFF80000713F0001F13FC383F03FFD87C001380007FEB7FC0EAFF80EC3FE0A3141FEA7F0000
1C133FC7FC15C0A2EC7F80A2ECFF00495A5CEB03F0495A495A495A90383E00E05B13789038F001
C0EA01C0EA038048B5FC5A5A5A481480B6FCA31B277DA622>I<EB7F803801FFF0000713FC380F
81FE381F80FF487E9038E07F80A5381FC0FFD807001300C7FC495AEB03F8495AEBFFC014F0EB01
FC6DB4FCEC7F8015C0143F15E0121EEA7F80A2EAFFC0A315C0147FD87F801380387E00FF6C4813
00380FFFFC000313F0C613801B277DA622>I<14075C5C5C5C5CA25B5B497E130F130E131C1338
137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F13F8A31D277EA6
22>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC0380FFFF8EB80FC38
0E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300007E137F007814806CEBFF
00381F01FE380FFFF8000313E0C690C7FC1B277DA622>I<EB07F0EB3FFCEBFFFE3801FC0F3807
F01F390FE03F80EBC07F121FEA3F80A2007FEB3F00EB001E91C7FCA25AEB0FC0EB3FF8EB70FEEB
E03F01C013809038801FC0A3010013E0A47EA4D83F8013C0A2001FEB3F80EA0FC03907E0FF0038
03FFFCC613F0EB3FC01B277DA622>I<1238123E003FB512F0A315E04814C01580A21500387000
1E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213FEA41201A86C5A13781C29
7CA822>I<EB3FC03801FFF04813FC380FC07EEB003F001E7F1580003E130FA2123F1380EBE01F
01F81300EBFE3F381FFF7E14F86C5B6C13FC6C7F6C7F00071480D81F9F13C0EA3F07D87E0313E0
EA7C0000FC133F48131F1407A21403A26C14C0007C1307007E14806C130F391FC03F00380FFFFC
000313F038007FC01B277DA622>I<91393FF00180903903FFFE07010FEBFF8F90393FF007FF90
38FF80014848C7127FD807FC143F49141F4848140F485A003F15075B007F1503A3484891C7FCAB
6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007F001
0FB512C0010391C7FC9038003FF829297CA832>67 D<B712E0A33903FE001FED07F015011500A2
1670A3913801C0781638A302031300A2140F90B5FCA3EBFE0F1403A20201130EA3161C91C7FCA3
163C1638167816F815011503151FB712F0A327297DA82D>69 D<B512FEA300011300B3B1B512FE
A317297FA81A>73 D<B512FEA3D803FEC8FCB3A3ED01C0A415031680A21507A2150FA2151F157F
913801FF00B7FCA322297DA829>76 D<B592383FFFC0A26E5C0003EFF000A2D9BFC014EFA2D99F
E0EB01CFA2D98FF0EB038FA3D987F8EB070FA2D983FC130EA2D981FE131CA3D980FF1338A29138
7F8070A291383FC0E0A391381FE1C0A291380FF380A2913807FF00A36E5AA26E5AA26E5AD8FFFE
0203B512C0A215703A297DA841>I<B53CF87FFFF807FFF0A32703FE000190C7EA1C00A26C6C6F
5B816E16786C701370A26E6E13F0017F495D14E0013F496D485A169F02F01503011F9026070FF8
5BA2DAF80FEBFC07010FD90E0791C7FC14FC0107011EEBFE0EED1C0302FE151E010390393801FF
1CA2902601FF7814B8ED700015F06D16F04B137FA26E486D5AA2023F5D4B131FA2021F5D92C712
0FA2020E6EC8FC44297FA847>87 D<48B47E000F13F0381F81FC486C7E147FA2EC3F80A2EA0F00
C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F83DFFC38
0FFF0F3801FC031E1B7E9A21>97 D<EB1FF0EBFFFE3803F03F390FE07F80EA1FC0EA3F80A2127F
9038001E004890C7FCA97E7F003FEB01C013C0001F1303390FE007803903F01F003800FFFCEB1F
E01A1B7E9A1F>99 D<EC3FF8A31403ACEB1FE3EBFFFB3803F03F380FE00F381FC007383F8003A2
127F13005AA97EA2EA3F801407381FC00F380FE01F3A03F03FFF803800FFF3EB3FC3212A7EA926
>I<EB3FE03801FFF83803F07E380FE03F391FC01F80393F800FC0A2EA7F00EC07E05AA390B5FC
A290C8FCA47E7F003F14E01401D81FC013C0380FE0033903F81F803900FFFE00EB1FF01B1B7E9A
20>I<EB07F8EB3FFEEBFE3F3901FC7F80EA03F8A2EA07F0A2EC3F0091C7FCA6B512C0A3D807F0
C7FCB3A3387FFF80A3192A7EA915>I<9038FF81F00003EBE7FC390FC1FE7C391F80FCFC003FEB
FE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1F8381FFFE0001813800038C8FC123C
A2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007F8007C13015A1400A36C1301007EEB
03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A22>I<EAFFE0A3120FAC147F9038E1
FFC09038E787E09038EE07F09038FC03F813F813F0A313E0AF3AFFFE3FFF80A3212A7DA926>I<
1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14>I<
EAFFE0A3120FACEC1FFCA3EC07C0EC0F80EC1E00147C5CEBE1F0EBE3E0EBE7C0EBEFE0EBFFF0A2
80EBF3FCEBE1FE13C080EC7F80143F15C0EC1FE0EC0FF039FFFC3FFEA31F2A7EA924>107
D<EAFFE0A3120FB3B2EAFFFEA30F2A7DA914>I<3BFFC07F800FF0903AC1FFE03FFC903AC783F0
F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3FFF8
A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A313E0
AF3AFFFE3FFF80A3211B7D9A26>I<EB3FE03801FFFC3803F07E390FC01F80391F800FC0003F14
E0EB00074814F0A34814F8A86C14F0A2393F800FE0A2001F14C0390FC01F803907F07F003801FF
FC38003FE01D1B7E9A22>I<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F1400
81A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8FCA9
EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FCA318
1B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013FE38
7FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038F7FF
C000C11300161B7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD1438
A73803F870000113F03800FFE0EB1F8015267FA51B>I<39FFE03FF8A3000F1303B11407A2140F
0007131F3A03F03BFF803801FFF338003FC3211B7D9A26>I<3BFFFE7FFC0FFEA33B0FE007E000
E03B07F003F001C0A29039F807F80300031680A23B01FC0EFC0700A2D9FE1E5B000090381C7E0E
A29039FF383F1E017F141C0278133C90393FF01FB8A216F86D486C5AA26D486C5AA36D486C5AA2
2F1B7F9A32>119 D<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3F
F86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01F
FF80A3211B7F9A24>I E /Fo 24 122 df<127012F812FCA2127C120CA41218A21230A2126012
40060F7C840E>44 D<EA01801203120F12FF12F31203B3A8EAFFFEA20F217CA018>49
D<EA03F0EA0FFCEA1C1F383007801270007813C0A21303EA380712001480A2EB0F00130E133CEA
03F8A2EA001E7FEB078014C0130314E01220127012F8A200F013C01260EB07801230381C1F00EA
0FFCEA03F013227EA018>51 D<EA01F0EA07FCEA0E0E487E383803801278127038F001C0A314E0
A5127013031278EA3807EA1C0DEA0FF9EA07F1380081C0130113031480A2383007001278130EEA
701C6C5AEA1FF0EA0FC013227EA018>57 D<90380FE01090383FF8309038F81C703801E0063903
C003F03807800148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA200
3C1460123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>67
D<EAFFFEA2EA0780B3EC0180A41403A215005CA25C143FB6FCA219227EA11E>76
D<D8FFC0EB03FF6D5B000715E0A2D806F0130DA301781319A36D1331A36D1361A36D13C1A29038
078181A3903803C301A3EB01E6A3EB00FCA31478EA1F80D8FFF0EB3FFF143028227EA12D>I<38
03F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3FF0EA1FFE
6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF80138EF0380
38C7FF00EA81FC14247DA21B>83 D<007FB512F8A2387C07800070143800601418A200E0141C00
C0140CA500001400B3A20003B5FCA21E227EA123>I<EA0FE0EA1FF8EA3C1C7FEA18071200A25B
EA03FF120FEA3F07127C127812F01418A2130F1278387C3FB8383FF3F0380FC3C015157E9418>
97 D<EA01FEEA07FF380F0780121C383803000078C7FC127012F0A7127814C07E381E0180380F
0300EA07FEEA01F812157E9416>99 D<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA
38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>I<EA01
FCEA07FF380F0780381C03C0EA3801007813E0EA7000B5FCA200F0C7FCA5127814607E6C13C038
0F83803807FF00EA00FC13157F9416>I<121C121E123E121E121CC7FCA8120E12FEA2121E120E
AFEAFFC0A20A227FA10E>105 D<EA01C0EA03E0A3EA01C0C7FCA8EA01E0120FA212011200B3A4
EA60C012F11380EA7F00123E0B2C82A10F>I<120E12FEA2121E120EAAEB0FFCA2EB07E0EB0380
EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03C0130114E014F038FFE3FEA21723
7FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>I<390E1FC07F3AFE7FE1FF809039
C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>I<380E1F8038
FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>I<EA01FCEA07FF380F07
80381C01C0383800E0007813F00070137000F01378A700701370007813F0003813E0381C01C038
0F07803807FF00EA01FC15157F9418>I<EA0E3CEAFEFEEAFFCFEA1F8FEA0F061300120EADEAFF
F0A210157F9413>114 D<EA0F88EA3FF8EA7078EAE0381318A3EAF000127FEA3FE0EA1FF0EA01
F8EA003CEAC01CA212E0A2EAF018EAF878EADFF0EA8FC00E157E9413>I<000E137038FE07F0A2
EA1E00000E1370AC14F01301380703783803FE7FEA01F818157F941B>117
D<38FFC3FEA2381E00F8000E1360A26C13C0A338038180A213C300011300A2EA00E6A3137CA313
38A21330A213701360A2EAF0C012F1EAF380007FC7FC123E171F7F941A>121
D E /Fp 15 121 df<127812FCA212FEA2127E1206A5120CA31218A2123012701260124007147A
8512>44 D<91383FE001903901FFF803903807F01E90391F80070790393E00018F49EB00CF01F0
147F4848143F0003151F485A4848140FA248C8FC16075A123E007E1503A3127C00FC1500AB127C
007E1503A3123E123F6C1506A26C7E160C6C7E6C6C141812016C6C1430017C14606DEB01C09039
1F800380903907F01E00903801FFFC9038003FE028337CB130>67 D<DA1FE013809138FFFC0190
3807F00F90390F800383013EC712C749146749143F4848141F485A4848140F491407120F48C8FC
16035A123E1601127EA2127C00FC92C7FCAA92380FFFFC127C007E9138001FC0EE0F80123EA212
3F7EA26C7E6C7EA26C7E6C7E6C6C141F137C013F1473D90FC013E3903907F807C10100B51200DA
1FF013002E337CB134>71 D<D8FFF0ED7FF86D15FF0007170000035E017CEC01BEA36DEC033EA3
6D1406A36D6C130CA36D6C1318A36D6C1330A36D6C1360A216C06D7EA291387C0180A391383E03
00A3EC1F06A3EC0F8CA3EC07D8A3EC03F0A3486C6C5AD80FC0157FD8FFFC91380FFFF8EC00C035
317CB03D>77 D<EA01FE380FFFC0381C03E0383C00F0003E137880141C0008131EC7FCA4EB01FE
133F3801FF1EEA07F0EA0F80EA1F00123E5AA248140CA3143EA2007C137E6CEBDF1C391F038FB8
390FFF07F03903F803C01E1F7D9E21>97 D<EC01E0143FA214031401AFEB3F81EBFFE13803E079
3807800D380F0007001E130314015A127CA2127812F8A91278127CA2123C003E1303001E13076C
130F3807801D3903E071F03901FFE1FF38003F0120327DB125>100 D<EB3F80EBFFE03803E0F8
3807803C48487E121E805A127C15800078130712F8B6FCA200F8C8FCA61278127C123CEC01807E
6CEB0300EB80063807C00E3801F03C3800FFF0EB1FC0191F7E9E1D>I<380781FE39FF87FF8090
388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E3AFFFC1FFF80A2211F7E9E25>
110 D<EB1FC0EBFFF83801E03C3807800F390F000780001EEB03C0A248EB01E0A248EB00F0A300
F814F8A8007814F0007C1301003C14E0A26CEB03C0A26CEB07803907C01F003801F07C6CB45AEB
1FC01D1F7E9E21>I<380780FC39FF87FF8090389E07C0390FB803E03907E000F04913F8157C5B
153EA3151FA9153EA3157C6D137815F89038E001F09038B803E090389E0FC0903887FF00EB81FC
0180C7FCAB487EEAFFFCA2202D7E9E25>I<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE038EB
C000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813705A
1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2141C
7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A31203A2
1207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB19>
I<D8078013F000FF131FA2000F130100071300B31401A2140300031307EBC00E3901F03CF83A00
FFF0FF80EB3FC0211F7E9E25>I<39FFF807FFA2390FE003F83903C001E001E05B0001495A6C6C
48C7FCEB7806EB7C0CEB3C1C6D5AEB0F3014E013076D5A80801307EB0E78EB0C7C497EEB381E49
7E01607F496C7E0001130348486C7E000780391FC003F83AFFE007FFC0A2221F7F9E23>120
D E /Fq 5 85 df<1630167016F0A21501A21503A21507150FA2151B821531A2156115E115C1EC
0181A2EC0301A21406A2140C141C14181430A202607FA2ECC000A249B5FC5B91C7FC1306A25BA2
5BA25B1370136013E01201000381D80FF01301D8FFFE90383FFFF0A22C337CB235>65
D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E83147E188002FE140FA24A15C0
A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA2011F157E17FE4A5CA2013F4A
5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495AED07E00001EC3F80B600FE
C8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7121EA24A140CA2147EA214
FEA25CA20101151CEEC0184A1400A201031301A202F05B15030107130F91B5FC93C7FCECE00F01
0F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA2137EA213FEA25BA21201B512
FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092380003F0EE01F84AEB00FC
A2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804AEB3F00167E0107EB03F891
B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA2013F1301A21400A249495AA2
137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0C9EA1F802F327CB034>82
D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C130F121800385CA20030011F
1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FEA25CA21301A25CA21303A2
5CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D3174B033>84
D E end
%%EndProlog
%%BeginSetup
%%Feature: *Resolution 300
TeXDict begin
%%EndSetup
%%Page: 1 1
bop 795 220 a Fq(D)26 b(R)g(A)f(F)h(T)570 311 y Fp(Maps,)20
b(Groups)i(and)e(Con)n(texts)391 433 y Fo(Lyndon)d(Clark)o(e,)e(Mark)h
(Sears,)h(T)l(on)o(y)f(Skjellum,)d(Marc)j(Snir)832 530 y(Ma)o(y)g(13,)g(1993)
75 715 y Fn(1)69 b(In)n(tro)r(duction)75 807 y Fm(MPI)15 b(pro)o(vides)f
(supp)q(ort)h(for)f(the)h(execution)g(of)f Fl(parallel)f(pro)q(cedures)p
Fm(.)k(A)e(parallel)e(pro)q(cedure)j(is)e(executed)75 857 y(collectiv)o(ely)g
(b)o(y)g(a)f(set)j(of)d(comm)o(unicating)e(pro)q(cesses.)22
b(T)m(ransfer)14 b(of)g(con)o(trol)g(to)g(the)g(parallel)f(pro)q(cedure,)j
(and)75 907 y(bac)o(k,)d(is)g(ac)o(hiev)o(ed)g(b)o(y)g(ha)o(ving)e(eac)o(h)j
(executing)g(pro)q(cess)g(transfer)g(con)o(trol)f(to)g(the)h(lo)q(cal)e(pro)q
(cedure)j(co)q(de,)e(and)75 957 y(return)j(from)e(it.)22 b(Not)15
b(all)f(pro)q(cesses)k(need)e(transfer)g(con)o(trol)f(to)h(the)f(same)g
(parallel)f(pro)q(cedure;)j(the)f(parallel)75 1006 y(pro)q(cedure)k(ma)o(y)d
(in)o(v)o(olv)o(e)g(only)h(a)g(subset)i(of)d(pro)q(cesses,)22
b(or)d(di\013eren)o(t)g(parallel)e(calls)h(ma)o(y)f(b)q(e)i(executed)h(b)o(y)
75 1056 y(di\013eren)o(t)15 b(subsets)g(of)e(pro)q(cesses.)21
b(The)14 b(collectiv)o(e)f(comm)o(unication)e(calls)i(pro)o(vided)g(b)o(y)h
(MPI)g(are)g(examples)f(of)75 1106 y(suc)o(h)i(parallel)d(pro)q(cedures.)21
b(Libraries)14 b(suc)o(h)g(as)g(parallel)f(scien)o(ti\014c)i(libraries)e
(will)g(b)q(e)h(another)h(example.)158 1157 y(It)j(is)g(highly)e(desirable)j
(to)f(allo)o(w)e(the)i(pro)q(cesses)j(that)d(execute)i(a)d(parallel)g(pro)q
(cedure)j(use)e(a)g(\\virtual)75 1206 y(pro)q(cess)i(name)d(space")j(lo)q
(cal)d(to)i(the)g(in)o(v)o(ok)n(ation.)29 b(Th)o(us,)19 b(the)g(co)q(de)h(of)
d(the)i(parallel)f(pro)q(cedure)i(will)d(lo)q(ok)75 1256 y(iden)o(tical,)c
(irresp)q(ectiv)o(e)j(of)e(the)h(absolute)f(addresses)j(of)d(the)h(executing)
g(pro)q(cesses.)22 b(It)14 b(is)g(often)h(the)g(case)g(that)75
1306 y(parallel)10 b(application)g(co)q(de)j(is)e(built)g(b)o(y)g(comp)q
(osing)f(sev)o(eral)i(parallel)e(mo)q(dules)h(\(e.g.,)f(a)i(n)o(umerical)e
(solv)o(er,)h(and)75 1356 y(a)j(graphic)g(displa)o(y)f(mo)q(dule\).)18
b(Supp)q(ort)d(of)e(a)h(virtual)f(name)g(space)j(for)e(eac)o(h)g(mo)q(dule)f
(will)g(allo)o(w)f(to)i(comp)q(ose)75 1406 y(mo)q(dules)19
b(that)g(w)o(ere)i(dev)o(elop)q(ed)f(separately)g(without)f(c)o(hanging)g
(all)g(message)g(passing)g(calls)h(within)e(eac)o(h)75 1455
y(mo)q(dule.)e(The)e(set)g(of)e(pro)q(cesses)k(that)d(execute)i(a)e(parallel)
f(pro)q(cedure)j(ma)o(y)c(b)q(e)j(\014xed,)f(or)g(ma)o(y)e(b)q(e)j
(determined)75 1505 y(dynamically)9 b(b)q(efore)k(the)g(in)o(v)o(ok)n(ation.)
i(Th)o(us,)e(MPI)f(has)h(to)f(pro)o(vide)g(a)g(mec)o(hanism)d(for)j(creating)
h(dynamically)75 1555 y(sets)20 b(of)f(lo)q(cally)e(named)h(pro)q(cesses.)36
b(W)m(e)18 b(alw)o(a)o(ys)g(n)o(um)o(b)q(er)g(pro)q(cesses)k(that)d(execute)i
(a)d(parallel)g(pro)q(cedure)75 1605 y(consecutiv)o(ely)m(,)d(starting)g
(form)e(zero.)23 b(Th)o(us,)15 b(a)f Fl(group)g Fm(is)g(an)h(ordered)h(set)g
(of)f(pro)q(cesses.)24 b(Eac)o(h)15 b(pro)q(cess)i(in)d(a)75
1655 y(group)i(is)g(asso)q(ciated)h(with)f(a)f Fl(rank)p Fm(,)h(starting)g
(from)e(zero.)26 b(Pro)q(cesses)19 b(are)d(iden)o(ti\014ed)h(b)o(y)f(their)g
(ranks)g(when)75 1705 y(comm)o(unication)10 b(o)q(ccurs)16
b(within)d(the)h(group.)158 1755 y(Another)h(imp)q(ortan)o(t)d(goal)h(is)h
(the)h(supp)q(ort)g(of)f(a)g Fl(lo)q(osely)g(sync)o(hronous)e
Fm(transfer)j(of)e(con)o(trol:)19 b(no)14 b(syn-)75 1805 y(c)o(hronization)f
(o)q(ccurs)i(either)f(b)q(efore)g(or)f(after)h(the)g(call.)j(Th)o(us,)c(a)g
(pro)q(cess)i(ma)o(y)d(start)i(sending)f(messages)h(that)75
1855 y(p)q(ertain)g(to)g(the)h(execution)g(of)e(a)h(parallel)f(pro)q(cedure)j
(b)q(efore)f(all)e(other)i(participating)e(pro)q(cesses)k(ha)o(v)o(e)c
(joined)75 1904 y(the)j(execution;)h(a)f(pro)q(cess)i(ma)o(y)13
b(receiv)o(e)18 b(a)d(message)h(that)f(p)q(ertains)i(to)f(the)g(execution)h
(of)e(a)g(parallel)g(pro)q(ce-)75 1954 y(dure,)f(while)g(participating)f(in)g
(another)i(parallel)e(execution.)19 b(A)14 b Fl(comm)o(unicatio)o(n)f(con)o
(text)f Fm(mec)o(hanism)g(is)75 2004 y(needed)k(to)e(distinguish)g(comm)o
(unicatio)o(n)e(that)i(b)q(elongs)h(to)f(distinct)g(parallel)g(pro)q(cedure)i
(executions.)k(\(Ev)o(en)75 2054 y(if)11 b(parallel)h(transfer)h(of)f(con)o
(trol)g(is)g(executed)i(sync)o(hroneously)m(,)f(one)f(still)g(needs)h(a)f
(con)o(text)h(mec)o(hanism)d(for)i(the)75 2104 y(sync)o(hronization)i(calls)f
(themselv)o(es.\))158 2154 y(Normally)m(,)i(a)i(parallel)f(pro)q(cedure)k(is)
d(written)h(so)g(that)f(all)f(messages)i(pro)q(duced)h(during)e(its)g
(execution)75 2204 y(are)e(also)f(consumed)g(b)o(y)g(the)h(pro)q(cesses)i
(that)d(execute)j(the)e(pro)q(cedure.)21 b(Ho)o(w)o(ev)o(er,)15
b(if)e(one)i(parallel)e(pro)q(cedure)75 2254 y(calls)i(another,)h(then)h(it)e
(migh)o(t)f(b)q(e)i(desirable)g(to)g(allo)o(w)e(suc)o(h)i(call)f(to)h(pro)q
(ceed)h(while)e(messages)h(are)g(p)q(ending)75 2304 y(\(the)21
b(messages)f(will)f(b)q(e)i(consumed)f(b)o(y)g(the)h(pro)q(cedure)h(after)e
(the)h(call)e(returns\).)39 b(In)20 b(suc)o(h)h(case,)i(a)d(new)75
2354 y(comm)o(unication)11 b(con)o(text)k(is)f(needed)i(for)d(the)i(called)f
(parallel)f(pro)q(cedure,)j(ev)o(en)f(if)e(the)i(transfer)g(of)f(con)o(trol)g
(is)75 2403 y(sync)o(hronized.)158 2454 y(A)j Fl(con)o(text)f
Fm(is)h(the)h(MPI)f(mec)o(hanism)e(for)i(partitioning)f(comm)o(unication)e
(space.)29 b(A)17 b(send)h(made)e(in)h(a)75 2504 y(con)o(text)e(cannot)f(b)q
(e)h(receiv)o(ed)h(in)d(another)i(con)o(text.)k(Con)o(texts)c(are)g(iden)o
(ti\014ed)f(in)g(MPI)g(using)g(in)o(teger-v)n(alued)75 2553
y Fl(con)o(text)p 234 2553 15 2 v 16 w(id)p Fm('s.)158 2604
y(The)k(comm)o(unication)c(domain)h(used)k(b)o(y)e(a)h(parallel)e(pro)q
(cedure)k(is)d(iden)o(ti\014ed)h(b)o(y)f(a)g Fl(comm)o(unicator)p
Fm(.)75 2654 y(Comm)o(uni)o(cators)c(bring)h(together)i(the)f(concepts)i(of)d
(pro)q(cess)j(group)d(and)h(comm)o(unicatio)o(n)d(con)o(text.)22
b(A)14 b(com-)75 2704 y(m)o(unicator)g(is)i(an)g(explicit)g(parameter)g(in)f
(eac)o(h)i(p)q(oin)o(t-to-p)q(oin)o(t)d(comm)o(unication)f(op)q(eration.)24
b(The)17 b(comm)o(u-)965 2828 y(1)p eop
%%Page: 2 2
bop 75 -100 a Fm(2)1596 b Fk(2)42 b(MAPS)75 45 y Fm(nicator)16
b(iden)o(ti\014es)g(the)g(comm)o(unicatio)o(n)d(con)o(text)j(of)f(that)h(op)q
(eration;)g(it)f(iden)o(ti\014es)i(the)f(group)f(of)g(pro)q(cesses)75
95 y(that)j(can)f(b)q(e)h(in)o(v)o(olv)o(ed)f(in)g(this)g(comm)o(unication;)f
(and)h(it)g(pro)o(vides)h(the)g(translation)f(from)e(virtual)i(pro)q(cess)75
145 y(names,)e(whic)o(h)g(are)h(ranks)g(within)f(the)h(group,)f(in)o(to)g
(absolute)h(addresses.)25 b(Collectiv)o(e)15 b(comm)o(unication)d(calls)75
195 y(also)i(tak)o(e)g(a)g(comm)o(unicator)d(as)k(parameter;)e(it)h(is)h(exp)
q(ected)h(that)e(parallel)f(libraries)h(will)f(b)q(e)i(built)f(to)g(accept)75
244 y(a)g(comm)o(uni)o(cator)e(as)i(parameter.)j(Comm)o(unicators)11
b(are)k(represen)o(ted)h(b)o(y)e(opaque)g(MPI)g(ob)r(jects.)158
297 y(MPI)i(do)q(es)h(not)f(pro)o(vide)g(for)g(absolute)g(pro)q(cess)i
(names.)24 b(Rather,)17 b(pro)q(cesses)h(are)f(alw)o(a)o(ys)e(iden)o
(ti\014ed)h(b)o(y)75 347 y(their)d(rank)g(inside)g(a)f(group.)18
b(A)13 b(univ)o(ersal)g(group)f(that)h(includes)h(all)d(pro)q(cesses)16
b(a)o(v)n(ailable)10 b(when)k(computation)75 397 y(starts)21
b(is)g(the)g(nearest)h(equiv)n(alen)o(t)e(to)g(an)g(absolute)h(pro)q(cess)h
(name)d(space)j(in)e(MPI.)g(Relativ)o(e)g(naming)e(is)75 447
y(consisten)o(t)d(with)f(MPI)g(implem)o(en)o(tations)d(where)k(pro)q(cesses)i
(can)d(b)q(e)h(added)f(dynamically)l(.)158 500 y(New)g(pro)q(cess)h(groups)e
(are)h(built)e(b)o(y)h(subsetting)h(and)f(reordering)h(pro)q(cesses)i(within)
c(existing)h(groups)h(\(as)75 549 y(de\014ned)20 b(b)o(y)e(comm)o
(unicators\).)29 b(A)19 b(new)g(group)f(is)h(de\014ned)g(from)e(an)h(old)g
(group)g(b)o(y)h(sp)q(ecifying)f(the)h(corre-)75 599 y(sp)q(ondence)e(b)q(et)
o(w)o(een)f(the)g(rank)e(of)h(eac)o(h)g(pro)q(cess)i(in)d(the)i(new)f(group)g
(and)g(its)g(rank)f(in)h(the)g(old)g(group.)21 b(Suc)o(h)75
649 y(corresp)q(ondence)c(is)d(called)f(a)h Fl(map)p Fm(;)f(it)g(is)h
(represen)o(ted)j(in)c(MPI)h(b)o(y)g(an)g(opaque)f(ob)r(ject.)75
802 y Fn(2)69 b(Maps)75 899 y Fm(A)14 b Fl(map)g Fm(is)h(a)f(one-to)g
(one-mapping)e(from)h(0)p Fj(:::m)8 b Fi(\000)i Fm(1)k(in)o(to)g(the)h(set)g
(of)f(non)g(negativ)o(e)h(in)o(tegers;)g Fj(m)g Fm(is)f(the)h
Fl(size)75 949 y Fm(of)g(map)g Fj(f)t Fm(.)25 b(A)16 b(p)q(ossible)g
(represen)o(tation)i(for)d(suc)o(h)i(map)d(is)i(a)g(list)f(of)h
Fj(m)g Fm(in)o(tegers;)h(it)f(is)g(often)g(con)o(v)o(enien)o(t)g(to)75
999 y(think)d(of)g(a)g(map)e(as)j(b)q(eing)f(suc)o(h)h(list.)j(Another)e(p)q
(ossible)e(represen)o(tation)i(is)e(as)h(an)f(algorithmic)d(sp)q
(eci\014cation)75 1048 y(of)j(the)i(map)d(\(e.g.,)h(hash)h(function\).)k(A)c
(map)e(is)i(represen)o(ted)i(b)o(y)e(an)g(opaque)g(map)e(ob)r(ject.)158
1101 y(Maps)i(are)g(used)h(in)f(order)g(to)g(represen)o(t)i(groups.)i(If)c
(pro)q(cesses)i(are)f(n)o(um)o(b)q(ered)e(from)g(0)g(to)h Fj(n)9
b Fi(\000)h Fm(1,)j(then)h(a)75 1151 y(map)f Fj(f)18 b Fm(:)13
b(0)p Fj(:::m)c Fi(\000)h Fm(1)j Fi(!)f Fm(0)p Fj(:::n)d Fi(\000)h
Fm(1)k(represen)o(ts)k(a)c(subset)j(group)d(of)h Fj(m)g Fm(pro)q(cesses:)23
b(pro)q(cess)16 b Fj(f)t Fm(\()p Fj(i)p Fm(\))g(has)f(rank)g
Fj(i)g Fm(in)75 1201 y(the)e(new)f(group.)18 b(A)12 b(map,)e(b)o(y)i(itself,)
g(do)q(es)h(not)f(represen)o(t)i(a)e(group;)g(it)g(do)q(es)h(so)f(only)f
(relativ)o(e)h(to)g(a)g(pre-de\014ned)75 1251 y(con)o(taining)h(group,)g
(i.e.,)f(relativ)o(e)i(to)g(another)g(ranking)f(of)h(the)g(pro)q(cesses.)158
1382 y Fh(Discussion:)38 b Fg(Should)16 b(decide)f(if)f(opaque)h(ob)r(jects)f
(are)f(justi\014ed,)i(or)f(whether)g(an)g(arra)o(y)g(of)g(indices)i(is)e(OK.)
f(On)75 1428 y(the)i(\\opaque)h(ob)r(ject")f(side:)21 b(Ma)o(y)15
b(ha)o(v)o(e)g(algorithmic)i(map)e(de\014nitions)i(and)f(ma)o(y)e(spread)i
(the)f(de\014nition)i(o)o(v)o(er)e(more)75 1473 y(than)d(one)g(no)q(de,)h(on)
f(a)f(v)o(ery)h(large)h(system.)k(On)11 b(the)h(explicit)i(represen)o(tation)
g(side:)j(The)12 b(represen)o(tation)i(is)e(simple,)h(w)o(e)75
1519 y(a)o(v)o(oid)i(an)f(additional)i(ob)r(ject,)e(and)g(there)g(is)g(no)f
(m)o(uc)o(h)h(w)o(aste.)k(Also,)c(with)g(an)g(explicit)i(represen)o(tation,)f
(the)f(user)f(can)75 1565 y(easily)j(add)e(his/her)i(o)o(wn)e(map)g
(constructors,)h(and)g(building)i(comm)o(unicators)f(is)e(faster.)20
b(Need)14 b(to)g(understand)i(what)75 1610 y(an)d(opaque)i(map)e(w)o(ould)h
(b)q(e)f(in)h(F)m(ortran.)158 1663 y(If)e(maps)i(are)f(explictly)j(represen)o
(ted)e(as)f(arra)o(ys,)g(then)h(one)f(needs)h(map)f(size)h(to)f(b)q(e)g(an)h
(additional)i(parameter.)158 1799 y Fm(F)m(or)h(eac)o(h)g Ff(m)p
Fm(,)g(an)g(iden)o(tit)o(y)f(map)f Ff(MPI)p 764 1799 14 2 v
15 w(IDENT\(m\))h Fm(of)g(size)i Ff(m)e Fm(is)h(prede\014ned:)26
b Ff(MPI)p 1475 1799 V 15 w(IDENT)p Fm(\()p Ff(m)p Fm(\))o(\()p
Fj(i)p Fm(\))17 b(=)g Fj(i)g(;)7 b(i)17 b Fm(=)75 1849 y(0)p
Fj(;)7 b(:::;)g(m)g Fi(\000)i Fm(1.)158 1984 y Fh(Discussion:)35
b Fg(If)11 b(maps)i(are)g(non)g(opaque)h(then)f(the)f(iden)o(t)i(map)f(is)g
(just)f(a)h(long)g(arra)o(y)g(with)g Fe(i)g Fg(stored)g(at)f(the)h
Fe(i)p Fg(-th)75 2034 y(en)o(try)m(,)f(and)g(it)g(is)g(just)f
Fd(MPI)p 465 2034 12 2 v 13 w(IDENT)p Fg(.)e(If)i(maps)h(are)g(opaque,)h(w)o
(e)e(need)h(a)f(di\013eren)o(t)j(map)d(for)h(eac)o(h)g(domain)h(size)f(\(a)f
(function)75 2084 y(from)i(in)o(tegers)h(to)f(maps?\))18 b(Need)13
b(to)g(think)h(of)f(F)m(ortran)g(implemen)o(tation.)75 2299
y Fc(2.1)56 b(Op)r(erations)18 b(on)h(maps)75 2381 y Fm(The)14
b(follo)o(wing)e(op)q(erations)i(are)g(de\014ned)h(on)f(maps.)i(Eac)o(h)f
(function)e(is)h(in)o(v)o(ok)o(ed)f(lo)q(cally)f(b)o(y)i(a)g(pro)q(cess.)158
2469 y Fl(MPI)p 257 2469 15 2 v 17 w(MAP)p 388 2469 V 17 w(SIZE\(map,)i
(size\))75 2608 y(IN)g(map)21 b Fm(handle)13 b(to)h(map)e(ob)r(ject)75
2704 y Fl(OUT)k(size)k Fm(size)14 b(of)g(map)e(\(in)o(teger\))p
eop
%%Page: 3 3
bop 75 -100 a Fk(2.2)41 b(Map)13 b(constructors)1369 b Fm(3)158
45 y Fl(MPI)p 257 45 15 2 v 17 w(MAP)p 388 45 V 17 w(APPL)l(Y\()15
b(map,)h(argumen)o(t,)e(v)m(alue\))75 174 y(IN)i(map)21 b Fm(handle)13
b(to)h(map)e(ob)r(ject)75 259 y Fl(IN)k(argumen)o(t)j Fm(map)12
b(argumen)o(t)h(\(in)o(teger\))75 345 y Fl(OUT)j(v)m(alue)k
Fm(image)12 b(of)h Ff(argument)f Fm(under)j Ff(map)e Fm(\(in)o(teger\))158
474 y Fl(MPI)p 257 474 V 17 w(MAP)p 388 474 V 17 w(INVERSE\()k(map,)e
(argumen)o(t,)f(v)m(alue\))75 603 y(IN)i(map)21 b Fm(handle)13
b(to)h(map)e(ob)r(ject)75 688 y Fl(OUT)k(argumen)o(t)i Fm(in)o(v)o(erse)d
(image)d(of)h Ff(value)g Fm(under)h Ff(map)p Fm(;)f(-1)h(if)f
Ff(value)f Fm(is)i(not)g(in)f(the)i(range)f(of)f Ff(map)p Fm(.)75
773 y Fl(IN)j(v)m(alue)21 b Fm(map)12 b(v)n(alue)h(\(in)o(teger\))158
902 y Fl(MPI)p 257 902 V 17 w(MAP)p 388 902 V 17 w(LIST\(map,)i(len,)g(arra)o
(y)p 850 902 V 17 w(of)p 906 902 V 17 w(in)o(teger,)e(size\))158
988 y Fm(Creates)i(an)f(explicit)f(list)h(represen)o(tation)h(of)e(a)h(map.)
75 1073 y Fl(IN)i(map)21 b Fm(handle)13 b(to)h(map)e(ob)r(ject)75
1158 y Fl(IN)k(LEN)22 b Fm(length)13 b(of)h(arra)o(y)f(\(in)o(teger\))75
1243 y Fl(OUT)j(arra)o(y)p 310 1243 V 16 w(of)p 365 1243 V
17 w(in)o(teger)i Fm(list)c(of)f(v)n(alues)g(in)h(the)g(range)h(of)e
Ff(map)75 1329 y Fl(OUT)j(size)k Fm(length)13 b(of)h(returned)h(list)f({)f
(size)i(of)e(map)g(\(in)o(teger\).)158 1492 y Fh(Discussion:)158
1542 y Fg(I)g(c)o(hanged)i(terminology)h(from)e(\\elemen)o(t,)h(rank")f(to)g
(\\argumen)o(t,)g(v)n(alue")i(since)e(b)q(oth)h(argumen)o(t)f(and)h(v)n(alue)
g(are)75 1592 y(ranks.)75 1794 y Fc(2.2)56 b(Map)19 b(constructors)75
1872 y Fm(The)14 b(map)f(constructors)i(ma)o(y)d(b)q(e)j(either)g(lo)q(cal)e
(or)h(ma)o(y)d(require)k(comm)o(unication.)158 1957 y Fl(MPI)p
257 1957 V 17 w(MAP)p 388 1957 V 17 w(BUILD\(map,)g(arra)o(y)p
807 1957 V 17 w(of)p 863 1957 V 16 w(in)o(teger,)f(size\))158
2043 y Fm(Build)f(a)g(map)f(from)g(an)h(explicit)g(list)g(represen)o(tation.)
20 b(This)13 b(function)g(is)g(called)h(lo)q(cally)e(b)o(y)h(one)h(pro)q
(cess.)75 2128 y Fl(OUT)i(map)k Fm(handle)14 b(to)f(map)g(ob)r(ject)75
2213 y Fl(IN)j(arra)o(y)p 259 2213 V 17 w(of)p 315 2213 V 17
w(in)o(teger)i Fm(list)13 b(of)h(v)n(alues)f(in)h(the)g(range)g(of)f
Ff(map)75 2298 y Fl(IN)j(size)k Fm(n)o(um)o(b)q(er)13 b(of)h(en)o(tries)h(in)
e(arra)o(y)h({)f(map)g(size)h(\(in)o(teger\))158 2419 y Fl(MPI)p
257 2419 V 17 w(MAP)p 388 2419 V 17 w(SPLIT\(comm,)h(map,)h(k)o(ey)l(,)g
(index,)e(newmap\))158 2504 y Fm(This)g(is)f(a)h(collectiv)o(e)g(function)f
(that)h(is)g(called)f(b)o(y)h(all)e(pro)q(cesses)17 b(in)c(the)i(group)e
(asso)q(ciated)i(with)e Ff(\(comm,)75 2554 y(map\))p Fm(.)23
b(All)14 b(calling)h(pro)q(cesses)j(pro)o(vide)e(the)g(same)f(v)n(alues)g
(for)h(the)g(parameters)g Ff(comm)f Fm(and)g Ff(map)p Fm(.)23
b(A)16 b(separate)75 2604 y(group)e(of)f(pro)q(cesses)k(is)d(formed)f(for)g
(eac)o(h)i(distinct)f(v)n(alue)f(of)h Ff(key)p Fm(,)f(with)g(the)i(pro)q
(cesses)i(ordered)e(according)f(to)75 2654 y(the)g(v)n(alue)g(of)f
Ff(index)p Fm(.)k(Eac)o(h)d(pro)q(cess)i(in)d(suc)o(h)h(group)g(is)g
(returned)h(in)f Ff(newmap)e Fm(the)j(map)d(of)h(this)h(group)g(within)75
2704 y Ff(comm)p Fm(.)p eop
%%Page: 4 4
bop 75 -100 a Fm(4)1442 b Fk(3)42 b(CONTEXT)p 1815 -100 13
2 v 15 w(ID)75 45 y Fl(IN)16 b(comm)21 b Fm(comm)o(unicator)11
b(ob)r(ject)k(handle)75 149 y Fl(IN)h(map)21 b Fm(map)12 b(ob)r(ject)j
(handle)75 253 y Fl(IN)h(k)o(ey)21 b Fm(\(in)o(teger\))75 357
y Fl(IN)16 b(index)k Fm(\(in)o(teger\))75 460 y Fl(OUT)c(newmap)k
Fm(map)12 b(ob)r(ject)j(handle)158 572 y(Additional)d(map)h(creation)h
(functions)g(ma)o(y)e(b)q(e)i(pro)o(vided.)k(F)m(ollo)o(ws)12
b(a)i(p)q(ossible)g(list)g(constructors.)158 710 y Fh(Discussion:)k
Fg(Do)13 b(w)o(e)g(w)o(an)o(t)g(to)g(mandate)h(them)f(in)g(MPI?)158
883 y Fl(MPI)p 257 883 15 2 v 17 w(MAP)p 388 883 V 17 w(UNION\()j(map1,)g
(map2,)f(newmap\))158 974 y Ff(map1)i Fm(and)g Ff(map2)g Fm(are)h(maps)e
(with)h(disjoin)o(t)g(ranges.)30 b(The)18 b(resulting)f(concatenation)h(is)g
(a)f(map)f(with)h(a)75 1024 y(range)d(whic)o(h)h(is)f(the)h(union)e(of)h(the)
h(ranges)g(of)e Ff(map1)h Fm(and)g Ff(map2)p Fm(,)f(with)h(elemen)o(ts)g(in)g
(the)g(range)h(of)f Ff(map1)f Fm(listed)75 1073 y(\014rst,)18
b(and)g(original)d(order)j(preserv)o(ed)h(within)e(eac)o(h)h(range.)28
b(The)18 b(union)f(op)q(eration)g(is)g(asso)q(ciativ)o(e)h(but)f(not)75
1123 y(comm)o(utativ)o(e.)459 1245 y Ff(newmap)o Fm(\()p Fj(i)p
Fm(\))12 b(=)692 1186 y Fb(\032)744 1219 y Ff(map1)o Fm(\()p
Fj(i)p Fm(\))287 b(if)13 b Fj(i)e(<)h(siz)r(e)p Fm(\()p Ff(map1)q
Fm(\))744 1269 y Ff(map2)o Fm(\()p Fj(i)e Fi(\000)f Fj(siz)r(e)p
Fm(\()p Ff(map1)q Fm(\)\))42 b(otherwise)75 1369 y Fl(IN)16
b(map1)21 b Fm(map)12 b(ob)r(ject)i(handle)75 1473 y Fl(IN)i(map2)21
b Fm(map)12 b(ob)r(ject)i(handle)75 1577 y Fl(OUT)i(newmap)k
Fm(map)12 b(ob)r(ject)j(handle)158 1724 y Fl(MPI)p 257 1724
V 17 w(MAP)p 388 1724 V 17 w(PR)o(ODUCT\()g(map1,)h(map2,)f(range,)g
(newmap\))158 1815 y Ff(map1)c Fm(is)i(a)f(map)e(with)i(v)n(alues)g(in)g(the)
h(range)g(0)p Fj(;)7 b(:::;)g Ff(ran)o(ge)s Fi(\000)f Fm(1.)18
b Ff(newmap)11 b Fm(is)h(the)h(cartesian)g(pro)q(duct)g(of)f
Ff(map1)75 1864 y Fm(and)k Ff(map2)p Fm(,)f(whic)o(h)h(maps)f(the)i(pair)f
(\()p Fj(i;)7 b(j)r Fm(\))16 b(in)o(to)g(the)g(pair)g(\()p
Ff(map1)o Fm(\()p Fj(i)p Fm(\))p Fj(;)7 b Ff(map2)p Fm(\()p
Fj(j)r Fm(\)\).)25 b(P)o(airs)16 b(in)g(the)h(domain)c(and)j(in)75
1914 y(the)d(range)f(of)g(this)h(mapping)d(are)i(n)o(um)o(b)q(ered)g(in)g(ro)
o(w)g(ma)r(jor)f(order,)i(i.e.,)e(pair)h(\()p Fj(i;)7 b(j)r
Fm(\))13 b(has)f(n)o(um)o(b)q(er)g Fj(i)6 b Fi(\001)g Ff(range)f
Fm(+)h Fj(j)r Fm(.)75 1964 y(The)14 b(pro)q(duct)h(op)q(eration)f(is)g(asso)q
(ciativ)o(e)f(but)i(not)e(comm)o(utativ)o(e.)523 2079 y Ff(newmap)o
Fm(\()p Fj(i)c Fi(\001)g Ff(range)f Fm(+)i Fj(j)r Fm(\))i(=)g
Ff(map1)o Fm(\()p Fj(i)p Fm(\))e Fi(\001)e Ff(range)h Fm(+)g
Ff(map2)o Fm(\()p Fj(j)r Fm(\))75 2181 y Fl(IN)16 b(map1)21
b Fm(map)12 b(ob)r(ject)i(handle)75 2285 y Fl(IN)i(map2)21
b Fm(map)12 b(ob)r(ject)i(handle)75 2388 y Fl(OUT)i(newmap)k
Fm(map)12 b(ob)r(ject)j(handle)75 2553 y Fn(3)69 b(Con)n(text)p
424 2553 21 2 v 24 w(id)75 2654 y Fm(A)13 b Fl(con)o(text)p
278 2654 15 2 v 15 w(id)e Fm(is)i(an)f(in)o(teger.)18 b(The)13
b(range)g(of)f(v)n(alid)f(v)n(alues)h(for)g Ff(context)p 1274
2654 14 2 v 14 w(id)g Fm(is)h(implem)o(en)o(tation)d(dep)q(enden)o(t,)75
2704 y(and)k(can)g(b)q(e)g(found)g(b)o(y)f(calling)g(a)g(suitable)h(query)h
(function,)e(as)h(describ)q(ed)h(in)f Fl(??)p Fm(.)p eop
%%Page: 5 5
bop 75 -100 a Fk(3.1)41 b(Op)q(erations)14 b(on)g(con)o(text)p
576 -100 13 2 v 16 w(id's)1201 b Fm(5)75 45 y Fc(3.1)56 b(Op)r(erations)18
b(on)h(con)n(text)p 753 45 17 2 v 20 w(id's)75 168 y Fl(MPI)p
174 168 15 2 v 17 w(CONTEXT)p 431 168 V 19 w(ALLOCA)l(TE\(comm,)d(map,)g
(arra)o(y)p 1112 168 V 16 w(of)p 1167 168 V 17 w(con)o(textids,)d(len\))158
259 y Fm(Allo)q(cates)e(an)g(arra)o(y)g(of)g(con)o(text)p 676
259 13 2 v 15 w(id's.)17 b(This)11 b(is)g(a)g(collectiv)o(e)g(op)q(eration)g
(that)g(is)g(executed)i(b)o(y)e(all)f(pro)q(cesses)75 309 y(in)g(the)i(group)
f(de\014ned)h(b)o(y)e Ff(\(comm,)21 b(map\))p Fm(.)16 b(The)11
b(con)o(text)p 984 309 V 16 w(ids)g(that)g(are)g(returned)h(are)g(unique)f
(within)f(the)h(group)75 359 y(asso)q(ciated)i(with)f Ff(\(comm,)21
b(map\))p Fm(.)16 b(The)d(arra)o(y)f(returned)i(is)e(the)h(same)e(on)h(all)g
(pro)q(cesses)i(that)f(call)e(the)i(function)75 409 y(\(same)f(order,)i(same)
e(n)o(um)o(b)q(er)g(of)g(elemen)o(ts\).)18 b(The)13 b(call)f(ma)o(y)f(blo)q
(c)o(k)i(un)o(til)f(all)g(pro)q(cesses)j(within)e(the)g(executing)75
458 y(group)h(ha)o(v)o(e)f(in)o(v)o(ok)o(ed)g(the)i(call.)75
573 y Fl(IN)h(comm)21 b Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75
679 y Fl(IN)h(map)21 b Fm(map)12 b(ob)r(ject)j(handle)75 786
y Fl(OUT)h(arra)o(y)p 310 786 15 2 v 16 w(of)p 365 786 V 17
w(con)o(text)p 538 786 V 16 w(ids)j Fm(\(arra)o(y)14 b(of)f(in)o(tegers\))75
892 y Fl(IN)j(len)k Fm(n)o(um)o(b)q(er)13 b(of)g(con)o(text)p
562 892 13 2 v 16 w(id's)g(to)h(allo)q(cate)f(\(in)o(teger\))158
1042 y Fl(MPI)p 257 1042 15 2 v 17 w(CONTEXT)p 514 1042 V 19
w(RESER)-5 b(VE\(con)o(text)p 931 1042 V 15 w(id\))158 1133
y Fm(Reserv)o(e)21 b(a)e(con)o(text)p 492 1133 13 2 v 16 w(id)g(v)n(alue.)34
b(This)20 b(is)f(a)g(lo)q(cal)g(call)g(that)g(is)h(in)o(v)o(ok)o(ed)e(b)o(y)i
(a)f(single)g(pro)q(cess.)37 b(A)19 b(re-)75 1183 y(serv)o(ed)i(con)o(text)p
343 1183 V 15 w(id)e(v)n(alue)g(will)f(not)h(b)q(e)h(allo)q(cated)f(b)o(y)g
(a)g(subsequen)o(t)i(call)d(to)h Ff(MPI)p 1457 1183 14 2 v
15 w(CONTEXT)p 1626 1183 V 15 w(ALLOCATE)e Fm(b)o(y)75 1233
y(the)g(same)e(pro)q(cess.)27 b(It)16 b(is)g(erroneous)i(to)e(reserv)o(e)i(a)
e(con)o(text)p 1070 1233 13 2 v 16 w(id)f(that)i(has)f(b)q(een)h(already)f(b)
q(een)h(allo)q(cated)f(b)o(y)75 1282 y Ff(MPI)p 144 1282 14
2 v 15 w(CONTEXT)p 313 1282 V 14 w(ALLOCATE)p Fm(.)75 1397
y Fl(IN)g(con)o(text)p 305 1397 15 2 v 16 w(id)k Fm(\(in)o(teger\))158
1590 y Fh(Discussion:)158 1642 y Fg(Ma)o(y)13 b(w)o(an)o(t)g(to)g(reserv)o(e)
h(an)f(arra)o(y)g(of)g(v)n(alues,)h(rather)g(than)f(one)h(v)n(alue.)158
1697 y(Implemen)o(tations)i(ma)o(y)d(ha)o(v)o(e)g(there)h(o)o(wn)f(reserv)o
(ed)g(v)n(alues.)19 b(The)13 b(e\013ect)g(is)g(as)h(if)f(calls)h(to)f
Fd(MPI)p 1575 1697 12 2 v 13 w(CONTEXT)p 1728 1697 V 12 w(RESERVE)75
1747 y Fg(where)g(executed)h(in)g(a)f(pream)o(ble.)158 1921
y Fl(MPI)p 257 1921 15 2 v 17 w(CONTEXT)p 514 1921 V 19 w(FREE\(con)o(text)p
836 1921 V 16 w(id\))158 2012 y Fm(F)m(ree)j(a)e(reserv)o(ed)j(con)o(text)p
583 2012 13 2 v 16 w(id)d(v)n(alue.)21 b(The)15 b(v)n(alue)f(b)q(ecomes)h(a)o
(v)n(ailable)e(for)h(a)h(subsequen)o(t)i(reserv)n(ation)e(b)o(y)75
2062 y Ff(MPI)p 144 2062 14 2 v 15 w(CONTEXT)p 313 2062 V 14
w(RESERVE)e Fm(or)g(subsequen)o(t)j(allo)q(cation)c(b)o(y)i
Ff(MPI)p 1071 2062 V 15 w(CONTEXT)p 1240 2062 V 14 w(ALLOCATE)p
Fm(.)d(It)j(is)g(erroneous)h(to)f(free)g(a)75 2112 y(v)n(alue)f(that)h(is)g
(asso)q(ciated)h(with)e(an)h(activ)o(e)g(comm)o(unicator.)75
2226 y Fl(IN)i(con)o(text)p 305 2226 15 2 v 16 w(id)k Fm(\(in)o(teger\))158
2419 y Fh(Discussion:)158 2471 y Fg(Here,)12 b(to)q(o,)h(w)o(e)g(migh)o(t)h
(prefer)f(to)g(free)f(an)i(arra)o(y)f(of)g(v)n(alues,)h(rather)f(than)h(one.)
158 2522 y(Comm)o(unicators,)i(lik)o(e)h(all)f(opaque)g(ob)r(jects,)f(are)g
(freed)g(b)o(y)g(a)f(call)j(to)d Fd(MPI)p 1286 2522 12 2 v
13 w(FREE)p Fg(.)f(W)m(e)i(ma)o(y)g(prefer)g(to)f(ha)o(v)o(e)i(that)75
2568 y(same)c(call)g(free)g(also)g(the)g(asso)q(ciated)h(con)o(text)p
757 2568 V 14 w(id.)18 b(With)12 b(the)g(curren)o(t)g(design,)h(if)e(w)o(e)g
(w)o(an)o(t)h(to)f(free)g(b)q(oth)h(comm)o(unicator)75 2614
y(and)i(con)o(text)p 275 2614 V 14 w(id)g(w)o(e)f(need)g(to)g(store)g(the)g
(con)o(text)p 810 2614 V 15 w(id)h(v)n(alue,)g(free)e(the)i(comm)o(unicator,)
g(next)f(free)g(the)g(con)o(text)p 1730 2614 V 15 w(id.)p eop
%%Page: 6 6
bop 75 -100 a Fm(6)1328 b Fk(4)41 b(COMMUNICA)m(TORS)75 45
y Fn(4)69 b(Comm)n(unicators)75 144 y Fm(A)17 b Fl(comm)o(unicator)d
Fm(is)j(an)g(opaque)g(ob)r(ject)h(that)f(iden)o(ti\014es)g(a)g(group)g(of)f
(pro)q(cesses)k(and)d(a)f(comm)o(unication)75 194 y(con)o(text)i(for)f(that)g
(group.)28 b(Lik)o(e)17 b(other)h(opaque)g(ob)r(jects,)g(comm)o(unicators)d
(cannot)j(b)q(e)g(transfered)h(b)q(et)o(w)o(een)75 244 y(pro)q(cesses;)d(con)
o(text)p 401 244 13 2 v 16 w(id's)d(are)h(used)h(to)f(transfer)h(information)
c(on)i(con)o(text.)158 298 y(As)g(a)f(short-hand,)g(w)o(e)h(shall)e(iden)o
(ti\014y)h(the)h(group)f(of)g(pro)q(cesses)j(asso)q(ciated)e(with)f(a)g(comm)
o(unicator)e Ff(comm)75 348 y Fm(with)17 b(the)h(comm)o(unicator)d(itself.)28
b(Th)o(us,)18 b(\\the)g(pro)q(cess)h(with)f(rank)f Fj(i)h Fm(in)f
Ff(comm)p Fm(")f(should)h(b)q(e)h(understo)q(o)q(d)h(as)75
398 y(meaning)14 b(\\)h(the)h(pro)q(cess)h(with)e(rank)g Fj(i)h
Fm(in)f(the)h(group)f(asso)q(ciated)i(with)e Ff(comm)p Fm(".)21
b(In)16 b(the)g(same)e(manner,)h(\\the)75 447 y(comm)o(unication)7
b(con)o(text)k Ff(comm)p Fm(")e(should)h(b)q(e)h(understo)q(o)q(d)h(to)e
(mean)f(\\the)i(comm)o(unication)c(con)o(text)k(asso)q(ciated)75
497 y(with)j Ff(comm)p Fm(".)158 551 y(An)19 b(initial)e(comm)o(unicator)f
Ff(MPI)p 701 551 14 2 v 15 w(COMM)p 804 551 V 15 w(INIT)i Fm(is)h(de\014ned)h
(when)f(the)h(program)d(starts.)34 b(Its)19 b(asso)q(ciated)75
601 y(group)e(con)o(tains)h(all)e(pro)q(cesses)k(that)e(start)g(the)g
(computation.)28 b(Applications)16 b(that)i(do)f(not)h(need)g(m)o(ultiple)75
651 y(pro)q(cess)e(groups)e(or)g(m)o(ultiple)d(con)o(texts,)k(will)d(only)h
(use)i(this)f(comm)o(unicator.)158 705 y(Let)f Ff(comm)g Fm(b)q(e)g(a)g(comm)
o(unicator)d(that)j(is)g(asso)q(ciated)g(with)g(a)g(group)g(of)f(size)i
Fj(n)p Fm(,)e(and)h(let)g Ff(map)e Fm(:)g(0)p Fj(::m)c Fi(\000)g
Fm(1)k Fi(!)75 755 y Fm(0)p Fj(::n)t Fi(\000)t Fm(1)g(b)q(e)h(a)f(map.)16
b(Then)c(the)g(pair)f Ff(\(comm,)20 b(map\))11 b Fm(de\014nes)i(a)e
(subgroup,)h(namely)e(the)i(subgroup)f(of)g(pro)q(cesses)75
805 y(with)j(ranks)g Ff(map)o Fm(\(0\))p Fj(;)7 b(:::;)g Ff(map)m
Fm(\()p Fj(m)j Fi(\000)f Fm(1\))14 b(in)g Ff(comm)p Fm(.)75
944 y Fc(4.1)56 b(Op)r(erations)18 b(on)h(comm)n(unicators)75
1064 y Fl(MPI)p 174 1064 15 2 v 17 w(COMM)p 351 1064 V 18 w(MAP\(comm,)d(sub)
q(comm,)f(map\))158 1154 y Fm(Returns)d(a)e(map)f(suc)o(h)i(that)g
Ff(\(comm,)21 b(map\))9 b Fm(is)i(the)g(group)g(asso)q(ciated)g(with)f
Ff(subcomm)p Fm(;)g(i.e.,)g(if)g Ff(i)g Fm(is)h(the)g(rank)75
1204 y(of)i(a)h(pro)q(cess)h(in)e(the)i(group)e(asso)q(ciated)i(with)e
Ff(subcomm)p Fm(,)f(then)i Ff(map\(i\))f Fm(is)g(the)i(rank)e(of)g(that)h
(same)f(pro)q(cess)i(in)75 1254 y(the)g(group)e(asso)q(ciated)i(with)f
Ff(comm)p Fm(.)j(The)e(call)e(is)h(erroneous)h(if)f Ff(subcomm)e
Fm(has)i(a)g(pro)q(cess)i(that)e(is)g(not)g(mem)o(b)q(er)75
1304 y(of)f Ff(comm)p Fm(.)75 1412 y Fl(IN)j(comm)21 b Fm(comm)o(unicator)11
b(ob)r(ject)k(handle)75 1513 y Fl(IN)h(sub)q(comm)k Fm(comm)o(unicator)11
b(ob)r(ject)k(handle)75 1613 y Fl(OUT)h(map)k Fm(map)12 b(ob)r(ject)j(handle)
158 1722 y(The)h Ff(MPI)p 314 1722 14 2 v 15 w(COMM)p 417 1722
V 15 w(MAP)f Fm(function)g(retriev)o(es)i(the)f(group)f(of)g(pro)q(cesses)j
(that)e(is)f(asso)q(ciated)i(with)e(a)g(comm)o(u-)75 1772 y(nicator.)36
b(Ho)o(w)o(ev)o(er,)21 b(since)f(absolute)g(pro)q(cess)i(names)d(are)h(not)g
(visible)f(in)g(MPI,)g(the)i(group)e(can)h(only)f(b)q(e)75
1822 y(de\014ned)h(relativ)o(e)e(to)h(another)g(encompassing)f(group.)33
b(The)19 b(\\absolute")f(pro)q(cess)j(n)o(um)o(b)q(er)d(is)h(obtained)f(b)o
(y)75 1872 y(using)c Ff(MPI)p 253 1872 V 15 w(COMM)p 356 1872
V 15 w(INIT)f Fm(as)h(a)f(reference)j(comm)o(unicator.)158
1961 y Fl(MPI)p 257 1961 15 2 v 17 w(COMM)p 434 1961 V 18 w(SIZE\(comm,)g
(size\))158 2051 y Fm(Returns)f(the)f(size)h(of)e(the)i(group)e(asso)q
(ciated)i(with)f Ff(comm)p Fm(.)75 2160 y Fl(IN)i(comm)21 b
Fm(handle)14 b(to)f(comm)o(unicator)75 2260 y Fl(OUT)j(size)k
Fm(group)13 b(size)i(\(in)o(teger\))158 2405 y Fl(MPI)p 257
2405 V 17 w(COMM)p 434 2405 V 18 w(CONTEXTID\(comm,)i(con)o(text)p
1077 2405 V 16 w(id\))158 2494 y Fm(Returns)e(the)f(con)o(text)p
522 2494 13 2 v 16 w(id)f(asso)q(ciated)i(with)f(the)g(comm)o(unicator)d
Ff(comm)p Fm(.)75 2603 y Fl(IN)16 b(comm)21 b Fm(comm)o(unicator)11
b(ob)r(ject)k(handle)75 2704 y Fl(OUT)h(con)o(text)p 356 2704
15 2 v 15 w(id)k Fm(con)o(text)p 564 2704 13 2 v 16 w(id)p
eop
%%Page: 7 7
bop 75 -100 a Fk(4.2)41 b(Comm)n(unicator)11 b(constructors)1182
b Fm(7)75 45 y Fc(4.2)56 b(Comm)n(unicator)19 b(constructors)75
157 y Fl(MPI)p 174 157 15 2 v 17 w(COMM)p 351 157 V 18 w(MAKE\(comm,)e(map,)f
(con)o(text)p 967 157 V 15 w(id,)f(new)o(comm\))158 242 y Fm(Creates)20
b(a)f(new)g(comm)o(unicator)d(ob)r(ject,)k(whic)o(h)f(is)g(asso)q(ciated)g
(with)g(the)g(group)g(de\014ned)h(b)o(y)e Ff(\(comm,)75 292
y(map\))p Fm(,)e(and)g(the)h(con)o(text)p 483 292 13 2 v 15
w(id)f Ff(context)p 704 292 14 2 v 15 w(id)p Fm(.)25 b(This)16
b(is)g(a)g(lo)q(cal)g(call)f(executed)j(b)o(y)f(one)f(pro)q(cess.)27
b(Ho)o(w)o(ev)o(er,)17 b(the)75 342 y(new)c(comm)o(unicator)e(ob)r(ject)i
(should)g(not)g(b)q(e)g(used)h(for)f(comm)o(unicatio)o(n)d(b)q(et)o(w)o(een)
15 b(t)o(w)o(o)d(pro)q(cesses)k(unless)d(they)75 392 y(b)q(oth)h(ha)o(v)o(e)g
(called)f(the)i(function.)158 442 y(It)h(is)g(erroneous)h(to)f(create)h(on)f
(the)g(same)f(pro)q(cess)j(t)o(w)o(o)d(distinct)h(comm)o(unicators)e(with)h
(the)i(same)e(con-)75 491 y(text)p 149 491 13 2 v 16 w(id.)75
582 y Fl(IN)h(comm)21 b Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75
665 y Fl(IN)h(map)21 b Fm(map)12 b(ob)r(ject)j(handle)75 747
y Fl(IN)h(con)o(text)p 305 747 15 2 v 16 w(id)k Fm(con)o(text)p
514 747 13 2 v 15 w(id)75 830 y Fl(OUT)c(new)o(comm)k Fm(comm)o(unicator)11
b(ob)r(ject)j(handle)158 920 y(The)c(con)o(tect)p 370 920 V
16 w(id)f(used)h(in)f(this)g(call)g(ma)o(y)e(b)q(e)j(an)f(in)o(teger)h
(returned)h(b)o(y)e(a)g(previous)g(call)g(to)g Ff(MPI)p 1627
920 14 2 v 15 w(ALLOC)p 1752 920 V 15 w(CONTEXT)p Fm(,)75 970
y(executed)18 b(within)d(the)h(same)f(group.)23 b(In)16 b(suc)o(h)h(case,)f
(it)g(is)f(guaran)o(teed)i(that)e(the)i(con)o(text)p 1552 970
13 2 v 15 w(id)f(is)f(unique,)h(and)75 1020 y(has)i(not)f(b)q(een)i(used)f
(already)f(to)h(create)h(a)e(comm)o(unicator.)26 b(Ho)o(w)o(ev)o(er,)18
b(the)h(user)f(is)g(free)g(to)f(manage)f(on)h(it)75 1070 y(o)o(wn)e(the)h
(con)o(text)p 371 1070 V 16 w(id's,)f(and)g(use)i(other)f(mec)o(hanisms)e
(for)h(their)h(allo)q(cation.)21 b(F)m(or)16 b(example,)e(one)i(could)f(ha)o
(v)o(e)75 1119 y(a)f(single)g Fl(con)o(text)p 385 1119 15 2
v 16 w(id)h(serv)o(er)e Fm(generate)j(unique)e(con)o(text)p
1031 1119 13 2 v 16 w(id's)g(for)g(the)h(en)o(tire)g(system;)f(one)g(can)h
(preallo)q(cate)75 1169 y(statically)e(some)g(con)o(text)p
493 1169 V 16 w(id)g(v)n(alues,)g(for)h(the)g(use)h(of)e(libraries;)g(etc.)
158 1298 y Fh(Discussion:)158 1343 y Fg(Do)g(w)o(e)f(w)o(an)o(t)g(to)g
(prohibit)j(implemen)o(tations)g(that)e(sync)o(hronize)h(the)f(creation)g(of)
f(comm)o(unicators)j(\(i.e.,)c(where)i(a)75 1389 y(call)h(to)f
Fd(MPI)p 254 1389 12 2 v 13 w(MAKE)p 347 1389 V 13 w(COMM)e
Fg(blo)q(c)o(ks)k(un)o(til)f(all)h(mem)o(b)q(ers)e(of)g(the)g(group)h(ha)o(v)
o(e)f(made)h(the)f(call\)?)158 1435 y(Is)g(it)g(preferable)i(to)d(ha)o(v)o(e)
i(a)f Fd(array)p 686 1435 V 12 w(of)p 738 1435 V 14 w(comm)e
Fg(as)i(parameter,)g(rather)h(than)f(one)h(comm)o(unicator?)158
1484 y(W)m(e)g(don't)h(ha)o(v)o(e)g(no)o(w)f(the)h(abilit)o(y)h(to)f(ha)o(v)o
(e)g(t)o(w)o(o)e(di\013eren)o(t)j(groups)g(with)e(the)h(same)f(con)o(text;)i
(e.g.,)d(t)o(w)o(o)h(di\013eren)o(t)75 1534 y(n)o(um)o(b)q(erings)i(of)d(the)
h(pro)q(cesses,)g(used)g(with)g(the)g(same)g(comm)o(unication)i(con)o(text.)j
(The)14 b(alternativ)o(e)h(is)f(to)g(remo)o(v)o(e)g(the)75
1584 y(restriction)f(that)e(t)o(w)o(o)g(distinct)i(comm)o(unicators)g(cannot)
f(use)f(the)g(same)h(con)o(text)p 1282 1584 V 14 w(id.)17 b(It)11
b(is)h(no)o(w)f(the)g(user)h(resp)q(onsibili)q(t)o(y)75 1634
y(to)17 b(disam)o(biguate)i(these)f(t)o(w)o(o)e(comm)o(unicators;)k(the)d
(system)g(do)q(es)h(not)f(guaran)o(tee)h(that)f(a)g(message)g(sen)o(t)g(with)
g(one)75 1684 y(comm)o(unicator)d(can)f(b)q(e)g(receiv)o(ed)h(only)f(with)g
(this)h(same)e(comm)o(unicator,)i(if)f(they)g(use)f(the)h(same)g(con)o(text)p
1668 1684 V 14 w(id.)18 b(W)m(e)12 b(sa)o(v)o(e)75 1733 y(on)i(con)o(text)p
254 1733 V 15 w(id's)g(but)g(lo)q(ose)h(safet)o(y)f(\(the)g(sender)h(and)f
(receiv)o(er)h(ma)o(y)f(b)q(e)g(using)h(di\013eren)o(t)h(pro)q(cess)e(n)o(um)
o(b)q(erings,)i(if)e(t)o(w)o(o)75 1783 y(di\013eren)o(t)h(comm)o(unicators)f
(are)g(used)f({)g(the)g(receiv)o(er)h(ma)o(y)g(not)f(iden)o(tify)i(correctly)
f(the)f(sender\).)75 2003 y Fn(5)69 b(W)-6 b(orking)24 b(without)e(con)n
(text)p 940 2003 21 2 v 24 w(id's)75 2094 y Fm(It)9 b(is)h(p)q(ossible)f(to)g
(create)i(comm)o(unicators)c(directly)m(,)j(without)f(using)g(con)o(text)p
1296 2094 13 2 v 15 w(id's.)16 b(A)10 b(single)f(function)g(com)o(bines)75
2144 y(the)k(allo)q(cation)e(of)g(a)h(new)h(con)o(text)p 631
2144 V 16 w(id)e(and)h(the)h(generation)g(of)e(a)h(new)h(comm)o(unicator.)i
(This)d(allo)o(ws)f(a)h(\\naiv)o(e")75 2194 y(user)22 b(that)e(do)q(es)i(not)
e(need)i(to)f(customize)f(con)o(text)p 942 2194 V 16 w(id)g(allo)q(cation)f
(to)h(ignore)h(this)f(MPI)h(feature.)39 b(Direct)75 2243 y(comm)o(unicator)11
b(creation)j(is)g(safer,)g(since)h(uniqueness)g(of)e(con)o(text)p
1150 2243 V 16 w(id's)g(is)h(guaran)o(teed,)g(b)o(y)g(construction.)158
2329 y Fl(MPI)p 257 2329 15 2 v 17 w(COMM)p 434 2329 V 18 w(SAFEMAKE\(comm,)j
(map,)f(new)o(comm\))158 2414 y Fm(Creates)h(a)f(new)g(comm)o(unicator)d
(\(with)j(an)f(attac)o(hed)i(con)o(text)p 1186 2414 13 2 v
15 w(id\))f(that)g(is)g(asso)q(ciated)g(with)g(the)g(group)75
2464 y(de\014ned)h(b)o(y)g(the)f(pair)g Ff(\(comm,)21 b(map\))p
Fm(.)j(This)17 b(is)f(a)g(collectiv)o(e)g(call)g(that)g(has)g(to)h(b)q(e)g
(in)o(v)o(ok)o(ed)e(b)o(y)h(all)f(mem)o(b)q(ers)75 2514 y(in)h(this)g(group.)
24 b(All)15 b(of)h(them)f(are)i(pro)o(viding)d(the)j(same)e(input)h
(parameters)g Ff(comm)f Fm(and)h Ff(map)p Fm(.)24 b(The)17
b(call)e(ma)o(y)75 2563 y(blo)q(c)o(k)g(un)o(til)g(all)f(pro)q(cesses)k(in)d
(the)h(group)f(ha)o(v)o(e)h(in)o(v)o(ok)o(ed)e(it.)23 b(The)16
b(new)g(comm)o(unicator)c(ma)o(y)i(b)q(e)i(safely)f(used)75
2613 y(for)f(comm)o(uni)o(cation)d(with)j(an)o(y)f(mem)o(b)q(er)f(of)h(the)i
(new)f(group,)g(once)g(the)h(call)e(returns.)75 2704 y Fl(IN)j(comm)21
b Fm(comm)o(unicator)11 b(handle)p eop
%%Page: 8 8
bop 75 -100 a Fm(8)710 b Fk(6)42 b(W)o(ORKING)13 b(WITHOUT)h(MAPS)g(AND)g
(CONTEXT)p 1780 -100 13 2 v 16 w(ID'S)75 45 y Fl(IN)i(map)21
b Fm(map)12 b(handle)75 126 y Fl(OUT)k(new)o(comm)k Fm(comm)o(unicator)11
b(handle)158 213 y Ff(MPI)p 227 213 14 2 v 15 w(COMM)p 330
213 V 15 w(SAFEMAKE)h Fm(is)75 300 y Ff(MPI_CONTEXT_ALLOC)o(\(comm)o(,)19
b(map,)i(context_id,)f(1\);)75 350 y(MPI_COMM_MAKE\(com)o(m,)f(map,)i
(context_id,)e(newcomm\);)75 400 y(``MPI_SYNCH\(comm,)o(map\)')o(')g(/*)i(by)
h(which)f(we)g(mean)g(a)g(synchronization)533 450 y(operation)f(in)h(the)g
(subgroup)f(defined)h(by)g(\(comm,map\))f(*/)158 615 y Fh(Discussion:)158
661 y Fg(Do)g(w)o(e)e(prefer)i(to)f(create)g(an)h(arra)o(y)g(of)f(comm)o
(unicators,)j(rather)e(than)f(one?)37 b(\(T)m(o)19 b(b)q(e)g(consisten)o(t)i
(with)f(the)75 707 y Fd(MPI)p 137 707 12 2 v 13 w(CONTEXT)p
290 707 V 11 w(ALLOC)11 b Fg(function\).)158 752 y Fd(MPI)p
220 752 V 13 w(SYNCH\(comm)o(,m)o(ap\))e Fg(is)k(not)g(curren)o(tly)h
(de\014ned)f(in)h(the)f(collectiv)o(e)i(comm)o(unication)g(library;)f(it)f
(can)g(b)q(e)g(co)q(ded)75 798 y(using)h(p)q(oin)o(t-to-p)q(oin)o(t)i(calls,)
e(but)g(seems)f(imp)q(ortan)o(t)h(enough)g(to)f(b)q(e)h(included)h(there.)158
848 y(General)e(though)o(t)f({)g(if)f(w)o(e)g(supp)q(ort)i(collectiv)o(e)g
(comm)o(unication)i(calls)d(with)g(a)g(map)f(as)h(an)g(additional)i
(parameter,)75 898 y(then)h(w)o(e)f(ha)o(v)o(e)h(disasso)q(ciated)i(the)e
(\\con)o(text")g(of)f(a)h(collectiv)o(e)h(comm)o(unication,)h(from)e(its)g
(supp)q(orting)h(\\group".)23 b(W)m(e)75 948 y(need)14 b(to)f(do)g(it)g(when)
h(w)o(e)e(build)j(new)e(comm)o(unicators,)i(in)f(order)f(to)g(b)q(o)q
(otstrap)h({)f(if)h(w)o(e)e(w)o(an)o(t)h(the)g(call)i(that)e(generates)75
997 y(a)g(new)g(comm)o(unicator)i(to)e(in)o(v)o(olv)o(e)i(only)f(the)f(pro)q
(cesses)h(in)g(the)f(new)g(group.)75 1217 y Fn(6)69 b(W)-6
b(orking)24 b(without)e(maps)h(and)g(con)n(text)p 1251 1217
21 2 v 25 w(id's)75 1307 y Fm(It)14 b(is)g(p)q(ossible)g(to)f(a)o(v)o(oid)g
(b)q(oth)h(the)g(use)h(of)e(map)g(ob)r(jects)i(and)e(of)h(con)o(text)p
1264 1307 13 2 v 15 w(id's)g(altogether.)k(A)c(single)f(function)75
1357 y(com)o(bines)20 b(the)h(generation)g(of)f(a)h(map,)f(the)h(allo)q
(cation)e(of)i(a)f(con)o(text)p 1263 1357 V 16 w(id)g(and)h(the)g(generation)
g(of)f(a)h(new)75 1407 y(comm)o(unicator.)14 b(This)e(allo)o(ws)e(\\naiv)o
(e")g(users)j(to)e(use)h(MPI)g(without)f(b)q(eing)g(exp)q(osed)i(to)e(maps)f
(or)h(con)o(text)p 1787 1407 V 16 w(id's.)158 1492 y Fl(MPI)p
257 1492 15 2 v 17 w(COMM)p 434 1492 V 18 w(BUILD\(comm,)k(arra)o(y)p
889 1492 V 17 w(of)p 945 1492 V 16 w(ranks,)h(size,)f(new)o(comm\))158
1578 y Fm(Build)j(a)f(map)g(from)f(an)h(explicit)h(list)g(represen)o(tation.)
31 b(This)18 b(is)g(a)g(collectiv)o(e)g(function)f(that)h(is)g(called)75
1627 y(b)o(y)d(all)e(mem)o(b)q(ers)h(of)g(the)h(new)g(group.)21
b(All)14 b(pro)o(vide)g(the)i(same)e(parameters)g(for)h Ff(comm)p
Fm(,)f Ff(array)p 1614 1627 14 2 v 14 w(of)p 1672 1627 V 15
w(ranks)g Fm(and)75 1677 y Ff(size)p Fm(.)j(The)d(call)f(ma)o(y)e(blo)q(c)o
(k)j(un)o(til)e(it)i(w)o(as)f(in)o(v)o(ok)o(ed)g(on)g(all)f(the)j(group)e
(mem)o(b)q(ers.)k(When)c(it)h(returns,)g(the)h(new)75 1727
y(comm)o(unicator)c(ma)o(y)h(b)q(e)j(used)f(to)g(comm)o(unicate)e(with)h(an)o
(y)h(mem)o(b)q(er)e(of)h(the)i(new)f(group.)75 1807 y Fl(IN)i(comm)21
b Fm(handle)14 b(to)f(comm)o(unicator)75 1888 y Fl(IN)j(arra)o(y)p
259 1888 15 2 v 17 w(of)p 315 1888 V 17 w(ranks)k Fm(list)13
b(of)h(v)n(alues)f(in)h(the)g(range)g(of)g Ff(map)75 1969 y
Fl(IN)i(size)k Fm(n)o(um)o(b)q(er)13 b(of)h(en)o(tries)h(in)e(arra)o(y)h({)f
(map)g(size)h(\(in)o(teger\))75 2050 y Fl(OUT)i(new)o(comm)k
Fm(handle)13 b(to)h(new)g(comm)o(unicator)158 2130 y Ff(MPI)p
227 2130 14 2 v 15 w(COMM)p 330 2130 V 15 w(BUILD)f Fm(is)75
2217 y Ff(MPI_MAP_BUILD\(com)o(m,)19 b(array_of_ranks,)g(size,)h(newmap\);)75
2267 y(MPI_COMM_SAFEMAKE)o(\(comm)o(,)f(newmap,)h(newcomm\);)158
2389 y Fl(MPI)p 257 2389 15 2 v 17 w(COMM)p 434 2389 V 18 w(COPY\(comm,)c
(new)o(comm\))158 2475 y Fm(Creates)d(a)f(new)g(comm)o(unicator)d(with)j(the)
h(same)e(group)g(as)h(the)h(old)e(comm)o(unicator.)k(This)d(is)f(a)h
(collectiv)o(e)75 2524 y(function)17 b(that)g(is)h(called)f(b)o(y)g(all)f
(mem)o(b)q(ers)g(of)h(new)h(group.)28 b(All)16 b(pro)o(vide)h(the)h(same)f
(parameters)g(for)g Ff(comm)p Fm(.)75 2574 y(The)j(call)f(ma)o(y)f(blo)q(c)o
(k)h(un)o(til)g(it)g(w)o(as)h(in)o(v)o(ok)o(ed)f(on)g(all)g(the)h(group)g
(mem)o(b)q(ers.)34 b(When)20 b(it)f(returns,)j(the)f(new)75
2624 y(comm)o(unicator)11 b(ma)o(y)h(b)q(e)j(used)f(to)g(comm)o(unicate)e
(with)h(an)o(y)h(mem)o(b)q(er)e(of)h(the)i(group.)75 2704 y
Fl(IN)h(comm)21 b Fm(handle)14 b(to)f(comm)o(unicator)p eop
%%Page: 9 9
bop 1854 -100 a Fm(9)75 45 y Fl(OUT)16 b(new)o(comm)k Fm(handle)13
b(to)h(new)g(comm)o(unicator)158 127 y Ff(MPI)p 227 127 14
2 v 15 w(COMM)p 330 127 V 15 w(COPY)f Fm(is)75 217 y Ff(MPI_COMM_SIZE\(com)o
(m,)19 b(size\);)75 267 y(MPI_COMM_SAFEMAKE)o(\(comm)o(,)g
(MPI_IDENT\(size\),)g(newcomm\);)158 392 y Fl(MPI)p 257 392
15 2 v 17 w(COMM)p 434 392 V 18 w(SPLIT\(comm,)c(k)o(ey)l(,)h(index,)f(new)o
(comm\))158 477 y Fm(Split)g(the)h(group)g(asso)q(ciated)g(with)g
Ff(comm)p Fm(;)f(creates)j(a)d(new)h(group)g(for)f(eac)o(h)i(distinct)f(v)n
(alue)f(of)g Ff(key)g Fm(that)75 527 y(con)o(tains)d(the)h(pro)q(cesses)i
(that)d(supplied)g(that)g(k)o(ey)g(v)n(alue;)g(the)h(pro)q(cesses)h(are)f
(rank)o(ed)f(according)g(to)g(the)h Ff(index)75 577 y Fm(v)n(alues)f(they)h
(supplied.)18 b(A)13 b(new)g(comm)o(unicator)d(is)i(created)i(for)e(eac)o(h)h
(subgroup.)18 b(Eac)o(h)13 b(pro)q(cess)i(is)d(returned)i(a)75
627 y(handle)e(to)g(the)h(comm)o(unicator)c(for)j(the)h(new)g(subgroup)f(it)g
(b)q(elongs)g(to.)17 b(This)c(is)f(a)f(collectiv)o(e)i(call)e(executed)j(b)o
(y)75 676 y(all)c(pro)q(cesses)15 b(in)c(the)h(group)f(asso)q(ciated)i(with)e
Ff(comm)p Fm(.)16 b(They)c(call)f(ma)o(y)f(blo)q(c)o(k)h(un)o(til)g(all)f
(pro)q(cesses)k(ha)o(v)o(e)e(in)o(v)o(ok)o(ed)75 726 y(the)j(function.)21
b(When)15 b(the)g(call)f(returns)i(the)g(new)f(comm)o(unicator)d(ma)o(y)h(b)q
(e)i(safely)f(used)i(for)e(comm)o(unication)75 776 y(in)f(the)i(new)f(group.)
75 866 y Fl(IN)i(comm)21 b Fm(handle)14 b(to)f(comm)o(unicator)75
948 y Fl(IN)j(k)o(ey)21 b Fm(\(in)o(teger\))75 1031 y Fl(IN)16
b(index)k Fm(\(in)o(teger\))75 1113 y Fl(OUT)c(new)o(comm)k
Fm(handle)13 b(to)h(new)g(comm)o(unicator)158 1203 y Ff(MPI)p
227 1203 14 2 v 15 w(COMM)p 330 1203 V 15 w(SPLIT\(comm,)19
b(key,)i(index,)g(newcomm\))12 b Fm(is)75 1293 y Ff(MPI_COMM_SIZE\(com)o(m,)
19 b(size\);)75 1343 y(MPI_MAP_SPLIT\(com)o(m,)g(MPI_IDENT\(size\),)f(key,)j
(index,)g(newmap\);)75 1393 y(MPI_COMM_SAFEMAKE)o(\(comm)o(,)e(newmap,)h
(newcomm\);)158 1561 y Fh(Discussion:)158 1607 y Fg(Ma)o(y)13
b(w)o(an)o(t)f(a)g(sp)q(ecial)i(DONTCARE)e(k)o(ey)g(v)n(alue)h(that)g
(indicates)h(that)f(the)f(caller)h(need)g(not)g(ha)o(v)o(e)f(a)h(new)f(comm)o
(u-)75 1652 y(nicator)75 1872 y Fn(7)69 b(Examples)75 1963
y Fm(W)m(e)20 b(sa)o(y)g(that)h(a)f(parallel)f(pro)q(cedure)k(is)d
Fa(active)h Fm(at)f(a)g(pro)q(cess)i(if)e(the)h(pro)q(cess)h(b)q(elongs)f(to)
f(a)h(group)f(that)75 2013 y(ma)o(y)13 b(collectiv)o(ely)h(execute)i(the)f
(pro)q(cedure,)i(and)d(some)g(mem)o(b)q(er)f(of)h(that)g(group)h(is)f(curren)
o(tly)i(executing)f(the)75 2063 y(pro)q(cedure)j(co)q(de.)27
b(If)16 b(a)h(parallel)e(pro)q(cedure)j(is)f(activ)o(e)f(at)h(a)f(pro)q
(cess,)j(then)e(this)f(pro)q(cess)j(ma)o(y)14 b(b)q(e)k(receiving)75
2113 y(messages)10 b(p)q(ertaining)f(to)h(this)g(pro)q(cedure,)i(ev)o(en)e
(if)f(it)g(do)q(es)i(not)e(curren)o(tly)i(execute)h(the)e(co)q(de)h(of)e
(this)h(pro)q(cedure.)75 2228 y Fc(7.1)56 b(Nonreen)n(tran)n(t)18
b(parallel)j(pro)r(cedures)75 2305 y Fm(This)13 b(co)o(v)o(ers)h(the)f(case)h
(where,)g(at)f(an)o(y)g(p)q(oin)o(t)f(in)h(time,)e(at)i(most)f(one)h(in)o(v)o
(ok)n(ation)e(of)h(a)h(parallel)f(pro)q(cedure)j(can)75 2355
y(b)q(e)e(activ)o(e)g(at)g(an)o(y)f(pro)q(cess.)20 b(I.e.,)12
b(concurren)o(t)j(in)o(v)o(ok)n(ations)c(of)h(the)h(same)f(parallel)g(pro)q
(cedure)j(ma)o(y)c(o)q(ccur)j(only)75 2405 y(within)h(disjoin)o(t)h(groups)g
(of)g(pro)q(cesses.)27 b(F)m(or)16 b(example,)f(all)g(in)o(v)o(ok)n(ations)f
(of)i(parallel)f(pro)q(cedures)j(in)o(v)o(olv)o(e)d(all)75
2455 y(pro)q(cesses,)h(pro)q(cesses)h(are)d(single-threaded,)g(and)g(there)h
(are)f(no)g(recursiv)o(e)h(in)o(v)o(ok)n(ations.)158 2504 y(In)e(suc)o(h)h(a)
f(case,)h(a)f(con)o(text)p 604 2504 13 2 v 15 w(id)g(can)g(b)q(e)h
(statically)e(allo)q(cated)h(to)g(eac)o(h)h(pro)q(cedure.)19
b(The)14 b(static)g(allo)q(cation)75 2554 y(can)c(b)q(e)h(done)f(in)g(a)g
(pream)o(ble,)f(as)h(part)g(of)g(initialization)d(co)q(de.)17
b(Or,)11 b(it)f(can)g(b)q(e)h(done)f(a)g(compile/link)d(time,)i(if)g(the)75
2604 y(implemen)o(tatio)o(n)h(has)j(additional)f(mec)o(hanisms)e(to)j(reserv)
o(e)i(con)o(text)p 1190 2604 V 16 w(id)d(v)n(alues.)18 b(Comm)n(unicators)11
b(to)i(b)q(e)g(used)75 2654 y(b)o(y)f(the)g(di\013eren)o(t)h(pro)q(cedures)h
(can)e(b)q(e)g(build)f(in)h(a)f(pream)o(ble,)g(if)g(the)i(executing)f(groups)
g(are)g(statically)f(de\014ned;)75 2704 y(if)17 b(the)i(executing)f(groups)h
(c)o(hange)f(dynamically)m(,)d(then)k(a)e(new)i(comm)o(unicator)c(has)j(to)g
(b)q(e)g(built)g(whenev)o(er)p eop
%%Page: 10 10
bop 75 -100 a Fm(10)1584 b Fk(8)42 b(LEFT)75 45 y Fm(the)15
b(executing)h(group)e(c)o(hanges,)h(but)g(this)g(new)g(comm)o(uni)o(cator)d
(can)j(b)q(e)g(built)f(using)h(the)g(same)f(preallo)q(cated)75
95 y(con)o(text)p 210 95 13 2 v 16 w(id.)19 b(If)14 b(the)h(parallel)e(pro)q
(cedures)j(can)f(b)q(e)g(organized)f(in)o(to)g(libraries,)f(so)i(that)f(only)
g(one)g(pro)q(cedure)j(of)75 145 y(eac)o(h)c(library)e(can)i(b)q(e)g
(concurren)o(tly)g(activ)o(e)g(at)f(eac)o(h)h(pro)q(cessor,)h(then)f(it)f(is)
g(su\016cien)o(t)h(to)f(allo)q(cate)g(one)g(con)o(text)75 195
y(p)q(er)j(library)m(.)75 318 y Fc(7.2)56 b(P)n(arallel)14
b(pro)r(cedures)e(that)g(are)g(nonreen)n(tran)n(t)h(within)h(eac)n(h)f
(executing)f(group)75 397 y Fm(This)i(co)o(v)o(ers)i(the)f(case)h(where,)f
(at)g(an)o(y)f(p)q(oin)o(t)g(in)g(time,)f(for)h(eac)o(h)h(pro)q(cess)i
(group,)d(there)i(can)f(b)q(e)g(at)f(most)g(one)75 447 y(activ)o(e)h(in)o(v)o
(ok)n(ation)e(of)h(a)g(parallel)g(pro)q(cedure)j(b)o(y)d(a)h(pro)q(cess)i
(mem)o(b)q(er.)i(Ho)o(w)o(ev)o(er,)c(it)g(migh)o(t)d(b)q(e)k(p)q(ossible)f
(that)75 497 y(the)e(same)e(pro)q(cedure)j(is)e(concurren)o(tly)i(in)o(v)o
(ok)o(ed)d(in)h(t)o(w)o(o)g(partially)e(\(or)j(completely\))e(o)o(v)o
(erlapping)g(groups.)17 b(F)m(or)75 547 y(example,)12 b(the)h(same)g
(collectiv)o(e)g(comm)o(unicati)o(on)d(function)j(ma)o(y)e(b)q(e)j(concurren)
o(tly)h(in)o(v)o(ok)o(ed)d(on)h(t)o(w)o(o)g(partially)75 596
y(o)o(v)o(erlapping)g(groups.)158 648 y(In)e(suc)o(h)i(a)e(case,)h(a)f(con)o
(text)p 595 648 V 16 w(id)g(is)g(asso)q(ciated)h(with)f(eac)o(h)h(parallel)e
(pro)q(cedure)j(and)f(eac)o(h)f(executing)i(group,)75 697 y(so)k(that)h(o)o
(v)o(erlapping)e(execution)i(groups)f(ha)o(v)o(e)g(distinct)h(comm)o(unicati)
o(on)c(con)o(texts.)30 b(\(One)18 b(do)q(es)g(not)f(need)75
747 y(a)f(di\013eren)o(t)i(con)o(text)p 414 747 V 16 w(id)e(from)f(eac)o(h)i
(group;)h(one)f(merely)e(needs)j(a)f(\\coloring")e(of)h(the)i(groups,)f(so)f
(that)h(One)75 797 y(can)c(generate)i(the)f(comm)o(unicators)d(for)i(eac)o(h)
g(parallel)f(pro)q(cedure)k(when)d(the)h(execution)g(groups)g(are)f
(de\014ned.)75 847 y(Here,)i(again,)d(one)i(only)g(need)h(one)f(con)o(text)g
(for)g(eac)o(h)h(library)m(,)d(if)h(no)h(t)o(w)o(o)f(pro)q(cedures)k(from)12
b(the)j(same)e(library)75 897 y(can)h(b)q(e)h(concurren)o(tly)g(activ)o(e)f
(in)f(the)i(same)e(group.)158 948 y(Note)18 b(that,)f(for)g(collectiv)o(e)h
(comm)o(unicati)o(on)c(libraries,)k(w)o(e)f(do)g(allo)o(w)f(sev)o(eral)i
(concurren)o(t)h(in)o(v)o(ok)n(ations)75 998 y(within)e(the)i(same)e(group:)
25 b(a)18 b(broadcast)g(in)g(a)f(group)h(ma)o(y)e(b)q(e)j(started)g(at)e(a)h
(pro)q(cess)h(b)q(efore)g(the)g(previous)75 1047 y(broadcast)h(in)f(that)h
(group)g(ended)g(at)g(another)g(pro)q(cess.)37 b(In)20 b(suc)o(h)g(a)g(case,)
h(one)f(cannot)g(rely)g(on)f(con)o(text)75 1097 y(mec)o(hanisms)12
b(to)i(disam)o(biguate)e(successiv)o(e)17 b(in)o(v)o(ok)n(ations)12
b(of)h(the)i(same)f(parallel)f(pro)q(cedure)j(within)d(the)i(same)75
1147 y(group:)31 b(the)21 b(pro)q(cedure)i(need)e(b)q(e)h(implem)o(en)o(ted)d
(so)h(as)h(to)f(a)o(v)o(oid)g(confusion.)37 b(E.g.,)21 b(for)f(broadcast,)j
(one)75 1197 y(ma)o(y)13 b(need)k(to)e(carry)h(additional)d(information)f(in)
j(messages,)h(suc)o(h)g(as)f(the)h(broadcast)g(ro)q(ot,)f(to)g(help)g(in)g
(suc)o(h)75 1247 y(disam)o(biguation;)9 b(one)i(also)g(relies)h(on)f(preserv)
n(ation)h(of)f(message)g(order)i(b)o(y)e(MPI.)g(With)g(suc)o(h)h(an)f
(approac)o(h,)h(w)o(e)75 1297 y(ma)o(y)g(b)q(e)i(gaining)e(p)q(erformance,)h
(but)h(w)o(e)g(lo)q(ose)g(mo)q(dularit)o(y)m(.)h(It)f(is)f(not)h(su\016cien)o
(t)g(to)g(implemen)o(t)d(the)j(parallel)75 1346 y(pro)q(cedure)f(so)e(that)h
(it)e(w)o(orks)i(correctly)g(in)f(isolation,)e(when)j(in)o(v)o(ok)o(ed)e
(only)h(once;)h(it)f(needs)h(to)f(b)q(e)h(implemen)o(ted)75
1396 y(so)j(that)h(an)o(y)f(n)o(um)o(b)q(er)f(of)h(successiv)o(e)i(in)o(v)o
(ok)n(ations)d(will)g(execute)j(correctly)m(.)23 b(Of)15 b(course,)i(the)f
(same)e(approac)o(h)75 1446 y(can)g(b)q(e)h(used)f(for)g(other)g(parallel)f
(libraries.)75 1569 y Fc(7.3)56 b(W)-5 b(ell)19 b(nested)f(parallel)j(pro)r
(cedures)75 1649 y Fm(Calls)12 b(of)h(parallel)f(pro)q(cedures)j(are)f(w)o
(ell)f(nested)h(if)f(a)g(new)g(parallel)f(pro)q(cedure)j(is)e(alw)o(a)o(ys)g
(in)o(v)o(ok)o(ed)f(in)h(a)g(subset)75 1698 y(of)j(a)g(group)g(executing)h
(the)g(same)e(parallel)g(pro)q(cedure.)27 b(Th)o(us,)17 b(pro)q(cesses)i
(that)d(execute)i(the)f(same)e(parallel)75 1748 y(pro)q(cedure)h(ha)o(v)o(e)e
(the)g(same)f(execution)i(stac)o(k.)158 1799 y(In)i(suc)o(h)g(a)f(case,)i(a)e
(new)h(con)o(text)g(need)h(to)e(b)q(e)h(dynamically)d(allo)q(cated)i(for)g
(eac)o(h)h(new)g(in)o(v)o(ok)n(ation)d(of)i(a)75 1849 y(parallel)e(pro)q
(cedure.)24 b(Ho)o(w)o(ev)o(er,)16 b(a)f(stac)o(k)h(mec)o(hanism)d(can)i(b)q
(e)h(used)g(for)f(allo)q(cating)f(new)h(con)o(texts.)24 b(Th)o(us,)15
b(a)75 1899 y(p)q(ossible)e(mec)o(hanism)d(is)j(to)f(allo)q(cate)g(\014rst)i
(a)e(large)g(n)o(um)o(b)q(er)g(of)g(con)o(text)p 1233 1899
V 16 w(id's)g(\(up)h(to)f(the)i(upp)q(er)f(b)q(ound)g(on)f(the)75
1949 y(depth)f(of)f(nested)i(parallel)e(pro)q(cedure)i(calls\),)f(and)f(then)
h(use)h(a)e(lo)q(cal)g(stac)o(k)h(managemen)o(t)d(of)i(these)i(con)o(text)p
1799 1949 V 16 w(id's)75 1999 y(on)i(eac)o(h)g(pro)q(cess)i(to)e(create)h(a)e
(new)i(comm)o(unicator)c(\(using)j Ff(MPI)p 1129 1999 14 2
v 15 w(COMM)p 1232 1999 V 15 w(MAKE)p Fm(\))f(for)g(eac)o(h)i(new)f(in)o(v)o
(ok)n(ation.)158 2132 y Fh(Discussion:)k Fg(General)c(case)158
2266 y Fm(In)g(the)h(general)f(case,)h(there)g(ma)o(y)d(b)q(e)j(m)o(ultiple)c
(concurren)o(tly)16 b(activ)o(e)e(in)o(v)o(ok)n(ations)e(of)h(the)i(same)e
(parallel)75 2316 y(pro)q(cedure)18 b(within)d(the)i(same)e(group;)h(in)o(v)o
(ok)n(ations)e(ma)o(y)g(not)i(b)q(e)h(w)o(ell)e(nested.)26
b(A)16 b(new)g(con)o(text)h(need)g(to)e(b)q(e)75 2366 y(created)i(for)e(eac)o
(h)h(in)o(v)o(ok)n(ation.)21 b(It)16 b(is)f(the)h(user)h(resp)q(onsibilit)o
(y)e(to)g(mak)o(e)f(sure)j(that,)f(if)e(t)o(w)o(o)h(distinct)h(parallel)75
2416 y(pro)q(cedures)h(are)e(in)o(v)o(ok)o(ed)f(concurren)o(tly)i(on)f(o)o(v)
o(erlapping)f(sets)i(of)e(pro)q(cesses,)j(then)f(con)o(text)p
1584 2416 13 2 v 15 w(id)f(allo)q(cation)e(or)75 2466 y(comm)o(unicator)e
(creation)j(is)g(prop)q(erly)g(co)q(ordinated.)75 2610 y Fn(8)69
b(Left)75 2704 y Fm(Remote)13 b(pro)q(cedure)j(call)d(and)g(C2)h(ob)r(jects.)
p eop
%%Page: 11 11
bop 1833 -100 a Fm(11)158 45 y(Ho)o(w)14 b(to)f(handle)h(gro)o(wing)f(pro)q
(cess)j(set.)p eop
%%Trailer
end
userdict /end-hook known{end-hook}if
%%EOF

>From [email-masked] Wed Jun  9 15:37:19 1993
Date: Wed, 9 Jun 93 08:38:50 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: An inquiry you may not have received
Status: R

On May 31 there was a post in comp.parallel asking "what is MPI?"  -- I
responded to the guy, telling him to contact you if he wanted to get in one
or more of the email lists and/or to attend the meeting.  He is from exxon.
Now it occurs to me that he might have sent you mail at cs.utk and maybe it
didn't get forwarded to france or whatever, so just for reference here is the
guy's name and email: Denny Willen [email-masked]

>From [email-masked] Wed May 26 15:38:20 1993
Date: Wed, 26 May 1993 16:35:06 -0400
From: [email-masked] (David Walker)
To: [email-masked], [email-masked]
Subject: Group nakedness
Status: R

It has occurred to me, while reading the "Maps, Groups, and Contexts" (MGC)
draft, that groups no longer really exist in MPI in the sense that there is
no routine whose input or output is a group. 

It seems we have discarded the intuitively appealing concept of a group, and
replaced it with lesser obvious concept of a map. I would like to get rid
of maps, and restore groups to MPI. For most of the routines in MGC this can
be done by replacing the word "map" with "group", and clarifying the names
of some routines. I would like to see the MGC draft rewritten with the 
routines below. In most cases there is a clear correspondence between a "map"
routine and a "group" routine that does something very similar.

David

MPI_MAP_DOMAIN --> MPI_GROUP_SIZE
  size_of_group = MPI_GROUP_SIZE(group)
  IN  group         - handle to group object
  OUT size_of_group - number of processes in group (integer)

MPI_MAP_ELEMENT --> MPI_RANK_IN_PARENT
  old_rank = MPI_RANK_IN_PARENT(group,rank)
  IN  group    - handle to group object
  IN  rank     - rank in group (integer)
  OUT old_rank - rank of process (group,rank) in parent group

MPI_MAP_RANK --> MPI_RANK_IN_GROUP
  rank = MPI_RANK_IN_GROUP(group,old_rank)
  IN  group    - handle to group object
  IN  old_rank - rank in parent group (integer)
  OUT rank     - rank in group of process whose rank in the parent group is
		 (group,old_rank) (integer)

MPI_MAP_FLATTEN --> MPI_INQ_GROUP
  MPI_INQ_GROUP(group,array_of_integer,max_size)
  IN    group            - handle to group object
  OUT   array_of_integer - list of the ranks in the parent group of the
			   processes in group
  INOUT max_dim          - on entry the maximum number of integers that can be
			   put into array_of_integer.
			   on exit the size of the group, i.e. the actual number
			   of integers put into array_of_integer (integer).

MPI_MAP_BUILD --> MPI_GROUP_BUILD
  MPI_MAKE_GROUP(group,array_of_integer)
  OUT group            - handle to new group object
  IN  array_of_integer - ranks in parent group of processes in new group
  (Note: this is still a local group constructor since communicators will stiil
	 be used to all communication routines)

MPI_MAP_CONCAT --> MPI_GROUP_CONCAT
  MPI_GROUP_CONCAT(group1,group2,new_group)
  IN  group1    - handle to first group
  IN  group2    - handle to second group (does not overlap with group1)
  OUT new_group - handle to new group that is the concatentation of group1 and
		  group2

MPI_MAP_SPLIT --> MPI_GROUP_SPLIT
   MPI_GROUP_SPLIT(group,key,index,new_group)
   IN  group     - handle to group
   IN  key       - key used to partition group (integer)
   IN  index     - value whose relative order over all processes in new_group 
		   determines the rank within new_group (integer)
   OUT new_group - handle to new group
   (Note: this is a collective group constructor)

MPI_ALLOC_CONTEXT --> MPI_ALLOC_CONTEXT
   MPI_ALLOC_CONTEXT(comm,array_of_contextids,len)
   IN  comm                - handle to communicator
   IN  len                 - number of contexts to be created (integer)
   OUT array_of_contextids - array of len context IDs
   (Note: this is a collective operation involving all the process in the
	  group in comm)

MPI_MAKE_COMM --> MPI_MAKE_COMM
   MPI_MAKE_COMM(group,context_id,new_comm)
   IN  group      - handle to group
   IN  context_id - context ID
   OUT new_comm   - handle to communicator containing group and context_id

MPI_SAFEMAKE_COMM --> MPI_SAFEMAKE_COMM
   MPI_SAFEMAKE_COMM(group,new_comm)
   IN  group    - handle to group
   OUT new_comm - handle to communicator
   (Note: this is a collective operation)

MPI_MAP --> not needed

MPI_CONTEXT --> what is this for?

nothing --> MPI_PARENT (a new routine, may be useful)
   MPI_PARENT(group,parent_group)
   IN  group        - handle to group
   OUT parent_group - handle to parent of group

Date: Fri, 28 May 93 14:42:06 BST
Message-Id: <[email-masked]>
From: L J Clarke <[email-masked]>
Subject: Re: Group nakedness
To: [email-masked] (David Walker)
In-Reply-To: David Walker's message of Wed, 26 May 1993 16:35:06 -0400
Reply-To: [email-masked]
Cc: [email-masked]
Status: RO

David writes:

> It has occurred to me, while reading the "Maps, Groups, and Contexts" (MGC)
> draft, that groups no longer really exist in MPI in the sense that there is
> no routine whose input or output is a group. 
> 
> It seems we have discarded the intuitively appealing concept of a group, and
> replaced it with lesser obvious concept of a map. I would like to get rid
> of maps, and restore groups to MPI.

Me too! I think this makes a lot of sense.  I explained it to future
users here and they just said "Eh?".  I can't comment on the detailed
suggestions which was given below, I aint had time to read it yet. 

Best Wishes
Lyndon (the ex-prolific :-)

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Thu Jun  3 10:29:00 1993
Date: Thu, 3 Jun 93 16:14:37 BST
From: L J Clarke <[email-masked]>
Subject: mpi-context: group nakedness and deconvolution
To: [email-masked]
Reply-To: [email-masked]
Cc: [email-masked]
Status: R

Hi Tony

How's  the  context draft  coming  along  between  yourself  and  Mark
(Sears)?   I'll be  interested to hear what's happening on  the "group
nakedness" which  was mentioned over the net by David, yourself and I.
I found time to consider the concrete suggestion put forward by David,
at last, and have some input which I hope will be of use.

First  off I think that the deletion of MAP and exposure of GROUP is a
Good Thing. It helps to deconvolve the conceptual context of the draft
which I  now see as a major difficulty.  I explained it to  people and
the convolution was quite a problem.

I'll go through David's suggestion making the  detailed  comments, and
then make the overall comments after all that. Lines of David's  begin
with "> ", and I'll  include pointers  to the  overall comments  where
appropriate.

                       o--------------------o

Detailed Comments
- -----------------

> MPI_MAP_DOMAIN --> MPI_GROUP_SIZE
>   size_of_group = MPI_GROUP_SIZE(group)
>   IN  group         - handle to group object
>   OUT size_of_group - number of processes in group (integer)

Looks fine to me.

> MPI_MAP_ELEMENT --> MPI_RANK_IN_PARENT
>   old_rank = MPI_RANK_IN_PARENT(group,rank)
>   IN  group    - handle to group object
>   IN  rank     - rank in group (integer)
>   OUT old_rank - rank of process (group,rank) in parent group

This  introduces the idea of  ranking  within the  space  of a  parent
group,  presumably that within which the  (child)  group  was  created
(overall comment below). The group object must store the handle of the
parent group, which doesnt seem to be a problem.

> MPI_MAP_RANK --> MPI_RANK_IN_GROUP
>   rank = MPI_RANK_IN_GROUP(group,old_rank)
>   IN  group    - handle to group object
>   IN  old_rank - rank in parent group (integer)
>   OUT rank     - rank in group of process whose rank in the parent group is
>                  (group,old_rank) (integer)

Is consistent.

> MPI_MAP_FLATTEN --> MPI_INQ_GROUP
>   MPI_INQ_GROUP(group,array_of_integer,max_size)
>   IN    group            - handle to group object
>   OUT   array_of_integer - list of the ranks in the parent group of the
>                            processes in group
>   INOUT max_dim          - on entry the maximum number of integers that can be
>                            put into array_of_integer.
>                            on exit the size of the group, i.e. the actual number                           of integers put into array_of_integer (integer).

Is consistent.

> MPI_MAP_BUILD --> MPI_GROUP_BUILD
>   MPI_MAKE_GROUP(group,array_of_integer)
>   OUT group            - handle to new group object
>   IN  array_of_integer - ranks in parent group of processes in new group
>   (Note: this is still a local group constructor since communicators will stiil
>          be used to all communication routines)

Is not  consistent.  Needs the parent group as an argument in order to
record the  parent group  in the  created group (at least).  Needs the
size of the group to create as argument.

Its far  from  clear  how much  use MPI_MAKE_GROUP  and  MPI_INQ_GROUP
really are with the  business of  always referencing the  parent.   In
order to send a group in a message this way the user will have to send
the entire ancestry of the group in the message(s). Please see overall
comments below.

> MPI_MAP_CONCAT --> MPI_GROUP_CONCAT
>   MPI_GROUP_CONCAT(group1,group2,new_group)
>   IN  group1    - handle to first group
>   IN  group2    - handle to second group (does not overlap with group1)
>   OUT new_group - handle to new group that is the concatentation of group1 and
>                   group2

Within the  parent  reference framework the  groups must have the same
parent, or with more  complication a common ancestor which becomes the
parent of the created group. Decision should be made and documented.

> MPI_MAP_SPLIT --> MPI_GROUP_SPLIT
>    MPI_GROUP_SPLIT(group,key,index,new_group)
>    IN  group     - handle to group
>    IN  key       - key used to partition group (integer)
>    IN  index     - value whose relative order over all processes in new_group 
>                    determines the rank within new_group (integer)
>    OUT new_group - handle to new group
>    (Note: this is a collective group constructor)

Is  consistent.  Should  document  that  "group"  is  parent  of  each
"new_group" (obvious to us, I know, but ...).

> MPI_ALLOC_CONTEXT --> MPI_ALLOC_CONTEXT
>    MPI_ALLOC_CONTEXT(comm,array_of_contextids,len)
>    IN  comm                - handle to communicator
>    IN  len                 - number of contexts to be created (integer)
>    OUT array_of_contextids - array of len context IDs
>    (Note: this is a collective operation involving all the process in the
>           group in comm)

Please see overall comments below.

> MPI_MAKE_COMM --> MPI_MAKE_COMM
>    MPI_MAKE_COMM(group,context_id,new_comm)
>    IN  group      - handle to group
>    IN  context_id - context ID
>    OUT new_comm   - handle to communicator containing group and context_id

Is consistent (although see following message on intercommuncation).

> MPI_SAFEMAKE_COMM --> MPI_SAFEMAKE_COMM
>    MPI_SAFEMAKE_COMM(group,new_comm)
>    IN  group    - handle to group
>    OUT new_comm - handle to communicator
>    (Note: this is a collective operation)

Is  inconsistent.   There  is  no  communicator  with  which  to  call
MPI_ALLOC_CONTEXT.   Can   be  fixed,   for  example,   by  redefining
MPI_ALLOC_CONTEXT to accept a group rather than a communicator.

> MPI_MAP --> not needed

I assume this is a reference to MPI_COMM_MAP. Equivalent operator
still needed, returning group.

MPI_COMM_GROUP(comm, group)
  IN  comm  communicator handle
  OUT group handle of group bound in communicator

MPI_CONTEXT --> what is this for?

I assume this is a reference to MPI_COMM_CONTEXTID. Should keep it.

> nothing --> MPI_PARENT (a new routine, may be useful)
>    MPI_PARENT(group,parent_group)
>    IN  group        - handle to group
>    OUT parent_group - handle to parent of group

In the preant reference framework this is essential.

Overall Comments
- ----------------

1. Parent framework.

I mentioned that  MPI_INQ_GROUP and  MPI_BUILD_GROUP will be difficult
to use. 

Recommendation is  to  drop the  parent reference framework  and  have
absolute groups. This means absolute process identification  which can
be, for  example, indices within a group containing all processes, but
not mandated to be such.

2. Context allocation.

I  mentioned   that  MPI_ALLOC_CONTEXT   and   MPI_SAFEMAKE_COMM   are
inconsistent.

Recommendation   is   to   replace   the  communicator   argument   of
MPI_ALLOC_CONTEXT  with  a  group.  MPI  implementation  provides  the
context or  whatever  for the implied collective operation, as it does
for MPI_SAFEMAKE_COMM as described.

This recommendation effects deconvolution of the conceptual content of
the draft. It will make the whole thing much easier to understand.

                       o--------------------o

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Wed Jun  9 15:47:16 1993
Date: Wed, 9 Jun 93 08:48:42 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: June 3 mpi mail #1 of 1
Status: R

>From [email-masked] Thu Jun  3 12:29:51 1993
Date: Thu, 3 Jun 93 18:21:28 BST
From: L J Clarke <[email-masked]>
Subject: mpi-context: intercommunication etcetara
To: [email-masked]
Reply-To: [email-masked]
Cc: [email-masked]
Status: R

Hi Tony

Here are  thoughts on  how we can proceed with  intercommunication.  I
write  this letter as though maps  were deleted and groups exposed, in
anticipation :-)

In summary this letter says what we do not need two different kinds of
communicator objects - indeed we can have just one communicator object
which is described  in the letter.  We do  not really  need  different
constructors   to  build  communicators   for  intracommunication  and
intercommunication, but we should  provide them for the convenience of
the  user. The letter  adds at most nine procedures to this section of
MPI, and suggests deletion of a few others :-)

The "context" in the  draft from  Marc (Snir)  is weaker  than that in
Zipcode  and Proopsal X, as I am sure  you  are  aware.  The allocator
ensures only  that the context is  unique within the set  of processes
performing the  allocation. The context  (or context_id in the  draft)
can be sent in a message.

Presumably  this  means  the  following.   A group G could allocate  a
context C, send  it  to a group H. H can then send messages to G using
context  C which in the knowledge that G knows such messages come from
H.  However,  G  cannot send  messages to  H using  context C  in  the
knowledge that such messages have come from G.

I  have  no  complaint  with  this  context  concept.   It  does  have
consequences,   one   of   which  is   that  attempting   to  simulate
intercommunication by group union  intracommunication not particularly
palatable.  Never  mind,  I  hope  this  letter  can  largely  resolve
difficulties with intercommunication.

With this  kind  of context  in  mind, I  write  out  generic send and
receive in longhand, in two slightly different ways. Please bear  with
me on this.

      send(sender-group,                send(local-group,        
           receiver-group,   		     remote-group,       
           receiver-rank,       OR           remote-rank,        
           receiver-context, 		     remote-context,     
           tag,              		     tag,                
      ...)                   		...)                     
					                         
      receive(receiver-group,           receive(local-group,     
              sender-group,     OR              remote-group,    
              sender-rank,     	                remote-rank,     
              receiver-context,	                local-context,   
              tag,             	                tag,             
      ...)                     	        ...)                     

The receive operation  is  allowed  to wildcard  on "sender-rank"  (OR
"remote-rank") and "tag" (OR "tag" :-) only.

The "sender-group" (OR "local-group") argument to "send" is just there
so that  the rank of the sender can be placed in the message envelope.
In principle  this could just be some identifier  for the sender,  but
the natural identifier to use in MPI terms in the rank within a group,
and  putting  the  group  there is  good programming practice since it
documents the send.

The   two   group   arguments   to   receive,   "receiver-group"   and
"sender-group"  (OR "local-group" and "remote-group") appear at  first
to serve no  purpose since  "receive"  cannot select on  them. However
they may be useful in optimisations of interpretation of "sender-rank"
(OR "remote-rank") and the context, and are  good programming practice
since they document the receive.

In general  people at MPI  (excepting  myself) have been  thinking and
talking  about  the  (special) case  where it  just  so  happens  that
"sender-group  =  receiver-group"  (OR "remote-group =  local-group").
This point is key. I suggest that we can simply  let this be a special
case,  provide  convenience  functions  for   this  case,  and  retain
efficiency. Sounds groovy to me :-)

So what is a communicator then? No suprises to  learn that the idea is
something like

typedef struct 
{
  group_t   local_group;
  context_t local_context;
  group_t   remote_group;
  context_t remote_context;
} 
communicator_t;

where I have written "group_t" for the type of a group and "context_t"
for  the type  of a context  (and yes, I am talking a bit too close to
implementation  here,  but that is just  to demonstrate the  potential
for a single implementation).

If  the communicator is to  be used  for intracommunication, then  the
construction of the communicator sets "remote_group = local_group" and
"remote_context = local_context".

If the communicator is  to be  used for  intercommunication,  then the
construction of the communicator sets "remote_group" and "local_group"
differently.   It  may   or   may   not   set   "remote_context"   and
"local_context"  differently  depending  on  the  scope  within  which
contexts were allocated.

Now the interesting  case of the  two "group_t" being set the same and
the two "context_t" being set different. Yes, it even means something,
it  means two different  modules within  the same process group  which
communicate with  one  another,  and  need private  context  for  such
communications (it's intercommunication really). Now we're motoring :-)

How is  it used  in point-to-point?  For  pedagogical  purposes, I now
write out the generic send and receive  using the communicator  object
described and show how they correspond to the above.

send(communicator,
     rank,
     tag,
...)

sender-group     = local-group    = communicator.local_group
receiver-group   = remote-group   = communicator.remote_group
receiver-rank    = remote-rank    = rank
receiver-context = remote-context = communicator.remote_context
tag              = tag            = tag

receive(communicator,
        rank,
        tag,
...)
                         
receiver-group   = local-group    = communicator.local_group
sender-group     = remote-group   = communicator.remote_group
sender-rank      = remote-rank    = rank
receiver-context = local-context  = communicator.local_context
tag              = tag            = tag

So  how  is it  used  in collective?   Well  we  have  only collective
operations appliacble to the  case where the local and remote parts of
the  communicator  are  the  same,  so they  are  all  errors  if  the
communicator supplied is not of that nature. We could invent jargon to
describe a communicator with that nature, if we really wanted to.

I think that deals with what the object is and how it is  used. Better
lets think about the constructors and accessors and all the such now.

Okay, I said we keep functions for making the special case of remote and
local parts the same for user convenience.  So just two new procedures
for constructors.  Names for these? Well, as you know I'm not very good
at that part :-) I added the letter 'A' to 'COMM' (a for assymetric, and
use of the string "comma" is of course silly). 

MPI_COMMA_MAKE(local_group, local_context, remote_group, remote_context, comm)

  IN local_group local group for communicator which caller must be
     member of
  IN local_context for communicator which local_group processes must
     be able to use for receive
  IN remote_group for communicator which caller moy or may not be a
     member of
  IN remote_context for communicator which remote_group processes must
     be able to use for receive
  OUT comm communicator created

MPI_COMMA_SAFEMAKE(lcoal_group, remote_group, comm)
  IN local_group  as for MPI_COMMA_MAKE
  IN remote_group as for MPI_COMMA_MAKE
  OUT comm        as for MPI_COMMA_MAKE
  (Note that this is a collective operation synchronising all members
  of both process groups)

[Aside - I *much* prefer to drop the "safemake" stuff and be able to
pass a NULL or MPI_NULL_CONTEXT to the "make" stuff in which case it
allocates the contexts for the user.  It allows just one or other of the
two contexts in these cases to be allocated automatically.  I propose
this.]
  
Six new procedures for accessors.  They are pretty self explanantory so
I do not detail the argument lists here, it can be added trivially
later.  

MPI_COMM_LOCAL_GROUP(comm, local_group)

MPI_COMM_REMOTE_GROUP(comm, remote_group)

MPI_COMM_LOCAL_CONTEXTID(comm, local_context)

MPI_COMM_REMOTE_CONTEXTID(comm, remote_context)

MPI_COMM_LOCAL_SIZE(comm, local_size)

MPI_COMM_REMOTE_SIZE(comm, remote_size)

The  three accessors in the  draft should  probably  be documented  as
errors  if the remote  and local parts  of  the communicator  are  not
identical. On the other hand, they could just be deleted :-)

[Aside - why have  accessors for the group  size  when  the  user  can
easily  get at the group  and ask that object what the size is? If  we
have this function it should be in Section 6.]

Regarding Section 6, "Working without maps and context_id's". 

I guess we'd have a new procedure which is pretty obvious by now

MPI_COMMA_BUILD(comm, local_arrayofranks, local_size,
                      remote_arrayofranks, remote_size, newcomm)

[Aside - I'd  like to see MPI_COMM[A]_BUILD  dropped.  This  should be
done at the group level. I propose this.]

MPI_COMM_SPLIT  should  be documented  as  an error if  the local  and
remote parts of the existing  communicator are  not identical. 

Finally I cannot avoid the matter of providing mechanisms for allowing
users  to  get  information  about remote  groups  in  order to  build
communicators  in which the local  and remote  parts are not the  same
(otherwise  there is  no  point  being able  to do that in  the  first
place).  I'll just say this again. The BEST KNOWN WAY TO DO THIS IS BY
A NAME SERVER, now for contexts and groups.

We  rely on sequencing  for allocation of  contexts  and binding  with
groups into communicators when we are planning for intracommunication.
We justify this by virtue of our SPMD view of the process group - this
is the condition under  which we can rely on sequencing. When we think
about  activities  spanning  pairs  of  groups  we   just  cannot  use
sequencing - we certainly cannot invoke an SPMD view of the group pair
although we  can  invoke an  SPMD view of  each  group  separately and
individually. 

The NAME SERVICE is TRACTABLE.  It is EXISTING PRACTICE.  I'd like to
see it in MPI.  If not then I need the capabiliities which allow me to
implement one.  Minimum requirement: there are mechanisms for
transmission of contexts and groups in messages.  I already have given
them at the previous meeting.  Please don't make them too perverse :-)

Oh well, I hope all this typing helped.

Best Wishes
Lyndon (the now sporadically(sp?) prolific)

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Wed Jun  9 15:48:40 1993
Date: Wed, 9 Jun 93 08:50:15 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: June 4 mpi mail #1 of 1
Status: R

>From [email-masked] Fri Jun  4 12:42:16 1993
To: [email-masked]
Subject: Progress report on context draft.
Date: Fri, 04 Jun 93 11:31:01 MST
From: [email-masked]
Status: R

MPI colleagues,

We plan to rewrite the draft to include the following:

   1) Explanation of what a process knows when it is born.
     e.g. the default communicators.

     We think the requirements are going to be pretty minimal. For example, a
   communicator that allows the created process to talk to its
   parent, itself, and possibly a server. We are trying to avoid
   overspecifying what the process management external to MPI must provide.

   2) Explanation of what happens when an MPI send or receive
   call occurs, i.e. how the message envelope is built or interpreted.

    The purpose of this discussion is to reveal to implementors the
   information they need to know about rank->address translation and
   when. We do this without specifying a single correct implementation.

   3) An explanation of the proper/safe way to use communicators
   in the presence of subroutines of possibly arbitrary nesting and
   complexity. The rule we have definitely established is as follows:

     When a caller passes a communicator (which contains a context and
   group) to a callee, that communicator must be free of side effects on
   entry and exit to the subprogram. This provides the basic guarantee of
   safety.

     The callee has permission to do whatever communication it likes with
   the communicator, and under the above guarantee knows that no other
   communications will interfere. Since we permit the creation of new
   communicators without synchronization ( assuming preallocated context
   ids), this does not impose a significant overhead.

     This form of safety is analogous to other common computer science
   usages, such as passing a descriptor of an array to a library routine.
   The library routine has every right to expect such a descriptor to
   be valid and modifiable.

   4) Examples of how inter communication (communication between independent
   groups) is set up and used safely.

     We have a mechanism worked out for implementing such communication
   without a C2 object. C1 objects will provide a 'picture' of the remote
   group/context (including virtual topology info as appropriate). This
    communicator will only support pt to pt.

     A mechanism can also be created to allow construction of a union, but
   the details are unresolved at present.

   5) Demonstration of how a user server might be set up.
      In order to support servers we will allow receipt of messages
     in a context from a process outside a group. Since the process is
     outside the group, there is no valid rank for it, so the query
     of the rank will produce a MPI_ANYPROCESS (or something like that)
     return.

   6) A better (clearer) definition of what a communicator is.

     For instance, we are restricting communicators to a single context id,
   though we may cache virt top info and methods that
   implement communication inside a communicator. We do not support
   a hidden context id for collective communication (remember the user
   can always get another communicator).
   This strategy is perceived to be thread safe.

   7) Clarification of properties of groups and discussion of absolute
   versus relative groups. We currently favor the former, getting rid of
   maps a la Walker.

Mark Sears
Tony Skjellum

>From [email-masked] Wed Jun  9 15:51:34 1993
Date: Wed, 9 Jun 93 08:52:29 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: June 5-6 mpi intercommunication discussion thread
Status: R

>From [email-masked] Sat Jun  5 10:12:59 1993
Date: Sat, 5 Jun 93 09:23:10 CDT
From: Tony Skjellum <[email-masked]>
To: [email-masked]
Subject: intercommunication
Cc: [email-masked], [email-masked]
Status: R

Lyndon, Mark and I had a long discussion (well Barney McCabe was also
there for a while)...

Inter communicator would have these properties (following your
intentions, I believe, but without additional data structures):

A mechanism called publish will be created:

	mpi_publish(my_comm, "label-for-my_comm", [permissions],
		persistence_flag)

This registers a passivated C1 communicator my_comm with the server,
or server-equivalent mechanism.  "label-for-my_comm" uniquely
identifies it.  [permissions] are flags (eg, an array of contexts
that may subscribe, tbd).  persistence_flag is either MPI_EPHEMERAL,
or MPI_PERSISTENT, so that a server can persistently leave access
to a communicator available.

	mpi_subscribe("label-for_my-comm", new_comm)

gives the subscriber (in new_comm) a picture of my_comm of the publisher.
Both send and receive by the subscriber refers to ranks in the sender.
[NB Appropriate virtual topology info goes with a passivated C1.]

Now, what about a symmetric picture?  Well,

A group:
	mpi_publish(A_comm, "A_comm", [permissions], MPI_EPHEMERAL);
        mpi_subscribe(B_comm, "B_comm");

B group:
	mpi_publish(B_comm, "B_comm", [permissions], MPI_EPHEMERAL};
	mpi_subscribe(A_comm, "A_comm");

So, have we failed to deal with any problems?  Well, one will
have to accept messages as MPI_ANY to get them from outside the group
(but with in the context, still, of course).  We do not see this as a
problem.

Factoids:
	publish and subscribe do not create new contexts

	we still want something that can take A_comm and B_comm,
	make a "union" group, and give it new context (that is still
	in thought stage, because of ordering issues). Ideas?

- - Tony

>From [email-masked] Sat Jun  5 10:19:36 1993
Date: Sat, 5 Jun 93 09:33:32 CDT
From: Tony Skjellum <[email-masked]>
To: [email-masked]
Subject: Re:  intercommunication
Cc: [email-masked], [email-masked]
Status: R

Further factoids on intercommunication

i) mpi_publish is non-blocking
ii) mpi_subscribe is blocking
iii) cancelling publish or subscribe?  First to ask for this
     is in deep trouble :-)
iv) [IMPORTANT]  Passivated C1's shared during publish/subscribe
    process inherently update a processes' internal table that allows
    it to do address translation on group members acquired through
    a subscribe.  

    Mark Sears and I discussed the possibility that subscribe could
    be allowed to fail if subscription implied an unactivable C1.
    In other words, it might be impossible for certain processes to
    talk to other processes in a real system.  We will have to decide
    how conforming implementations are allowed to fail ...

MPI_EPHEMERAL allows a single publish to be subscribed to and removed
without race condition.

If MPI server is more powerful than user-space itself, Barney points
out that is can be used to connect to servers outside user space,
and hence must address permissions intelligently.

- - Tony

ps Lyndon, please comment further on need for relative groups, re Walker.
   Why do we need this ???

>From [email-masked] Sun Jun  6 10:48:27 1993
Date: Sun, 6 Jun 93 16:39:45 BST
From: L J Clarke <[email-masked]>
Subject: mpi-context; comments on "June 5 - intercommunication"
To: [email-masked]
Reply-To: [email-masked]
Status: R

Dear Tony

Here are (hopefully) better  comments on the  material you kindly sent
out regarding the discussion  between yourself and Mark  Sears  on the
subject of intercommunication.  I refer to your letter  to mpi-context
"intercommunication" of June 5.

You appear to suggest a mechanism whereby a process which can obtain a
picture  of a C1 object (i.e.,  binding  of one context and one group)
which it  has not  itself created. This works by  the creator using  a
publish operation to name the  C1  within a global name space, and the
non-creator using a subscribe operation  to  find out about the  C1 in
the global name space by name.   I really am happy with the acceptance
of a global name space!

There are  some general comments about the publish/subscribe mechanism
suggested which  follow. These raise an important question that I hope
can be answered fairly  quickly (latter  part of  next paragraph), and
leave a pointer to material not yet covered.

The subscriber  can use  the  C1 object  obtained by subscribe to send
messages to  the publisher using  the usual "rank  in group" notation.
The subscribe operation will make this possible. The material you sent
also says that the subscriber can also  receive using  the usual "rank
in group" notation for the publisher group.  I don't understand  this.
The reason is  that  with the context  allocation scheme of the extant
draft it  would not be safe for the subscriber to attempt  to  use the
communicator for  receive. The context  embedded  therein is  safe for
receive only at the publisher.  Is  this just a simple error preparing
the letter, or are you also changing the context allocation scheme? If
you are then we  really  do need to know that because it has important
ramifications for later material.

The publisher can receive  messages  from any subscriber  by receiving
with the  published communicator.   The  publisher  cannot  choose the
sender and will only be able to receive from subscribers  by executing
a receive from MPI_ANY sender rank. The  publisher cannot find out the
identity of  the subscriber sender  as the rank of the sender will  be
returned as MPI_ANYPROCESS rank (or equivalent). We recognise that the
mechamism suggested is not capable of providing the publisher with the
ability to choose the sending subscriber.  We also  recognise that the
machamism is not capable of  providing the  identity  of  the  sending
subscriber  (unless the  sending subscriber explicitly places identity
in the data part of the message). This is not intended as a criticism,
just a statement  about the mechanism on which we can agree. There are
situations in  which  not choosing and  not knowing the sender is fine
(although in my experience they are uncommon). We also need to provide
for the situations  in which  (a)  the receiver needs  receive from  a
particular sender  and  (b) the  receiver  needs to  receive from  any
sender and needs  to know the identity  of  the actual sender.  I come
back to this matter later in the letter.

There  are  two  detailed comments on  the publish/subscribe mechanism
suggested.   Some  of these may be the  result of  misunderstanding or
incomplete information.

* The "[permissions]" argument to the publish operation does  seem  to
have a  problem.  If this  is an  array  of contexts as suggested,  or
similar  thing, then how  can  the  publisher  know which contexts the
subscriber(s) will be using.  There does not seem to  be a  mechanism.
We could  also ask  the question the other way around, i.e.  how can a
subscriber determine a permitted context. Again there does not seem to
be  a mechanism. Should such a mechanism exist  the publish  operation
does not accept a context and therefore there is nothing to  check the
permitted contexts against.   Is this a small mistake in preparing the
letter  - should mpi_subscribe  accept  an  existing  communicator  to
provide a context for the subscribe?

The MPI server has power  which  the MPI user  does not  have. However
this does not seem to mean that the user can use the server to connect
outside user space since  the user is not  provided with  a  mechanism
with which to make the server to that.  So what purpose do permissions
really serve?

SUGGEST we remove the permissions argument.

*  The "persistence_flag" argument to the publish operation allows the
publisher to publish a name which the first subscribe  will atomically
subscribe and remove. Bearing in mind that the consuming  subscribe is
executed by a single process and that in general a  group of processes
will need to subscribe, I  find it difficult to see what the advantage
of this feature is. Is there a small mistake in preparing the letter -
should mpi_subscribe accept a communicator in order to provide a group
for the subscribe?

* There is no way to remove a persistent publish. We should have a way
of doing this. 

SUGGEST we add an mpi_unpublish("label") which removes "label" from the
global name space.  [Yes, I know, I'm in deep trouble, and probably
launched into the swimming pool in a couple of weeks :-)]

Now to sender-knowing intercommunication.  The letter suggests that a
"union" is the mechanism for communication when the receiver needs to
either choose or discover the sender.  I think that there are a couple
of basic problems with that approach. 

Observe that with the context allocator of the extant draft we do rely
on sequencing to correctly allocate contexts to library modules within
a group.  If process P  in  group G is allocating a context  for  some
module then  all other processes  in  G are  allocating  for  the same
module, otherwise things get all  twisted.  We do this because we have
a strong SPMD view of the behaviour of a process  group.  This fine so
far. Now  if we  use  a union to  do sender-knowing intercommunication
between two functionally distinct groups (e.g.  a client  group  and a
server  group, or two  groups in a modular application) then  we  have
created  a group which we cannot view  as SPMD.  It is not possible to
rely on sequencing for context allocation in this union group.

The suggestion is that we somehow have a union operator which accepts
two communicators, makes a "union" group of the groups within the
communicators, and gives a context for the "union" group.  Neither of
the contexts inside the original communicators can be used as the
context for the union, since each is allocated within a different group
and thus is not safe for receive in the other group.  With the extant
context allocator of the draft the union operator would have to
synchronise the "union" group, the two original groups, in order to
allocate a context.  Unfortunately this is only possible if we can rely
on sequencing across the two groups, which the previous paragraph argues
we cannot. 

This problem can be partially alleviated if we change the
context allocator of the extant draft to that intended in X and existing
in Zipcode - but that would not be working within the extant draft so I
have not worked through the nitty gritty details. 

The "union" would have to be a union operator of the two groups, which
would have to be distinct.  This just means that intercommunication
between co-located (in terms of mapped into same group) parallel modules
would have to be done with different notation between intercommunication
between non-co-colocated parallel modules.  This is not the biggest
point, but I do think it is valid nevertheless. 

These observations  led me to look at other ways  in which we  can
remove the multiplicity  of different  kinds of communicator  objects,
which resulted in my  previous  letter  to  mpi-context  "mpi-context:
intercommunication etcetara" of June 3.

I make the suggestion that I can  probably join  the publish/subscribe
mechanism  you suggest with suggestion of my previous letter,  without
much grief. If you like us to go that  way,  then please say so, and I
will work through the nitty gritty details of that line.

Well that's all for now folks. I hope some of the questions raised can
be answered. How would we like to move forward from here?

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Tue Jun  8 16:22:21 1993
Date: Tue, 8 Jun 1993 17:15:59 -0400
From: [email-masked] (David Walker)
To: [email-masked]
Subject: Example MPI programs
Status: R

I've been trying to get a better understanding of the use of communicators,
and have actually been trying to write MPI code. Below are two examples that
I would like feedback on. Both programs split the processes into
two groups that then compute in parallel. The first uses just intragroup
communicators. The second example merges results from the two groups, and
involves intergroup communication.

I think it generally useful to try to build up a small suite of example
programs like this. My second example merges results from two disjoint
groups. It would be nice to have an example in which they over lap.

I look forward to your comments.

David

/* PROGRAM 1.
    Just to get warmed up here's a simple MPI program that splits the
    initial group into subgroups. One subgroup contains the processes
    with even ranks in the initial group, and the other the odd ranks.
    Then, one subgroup does a parallel FFT (A->FFT(A)), and the other a
    parallel matrix multiplication (Z=XY). The global length of A is lenA.
    The global sizes of X, Y, and Z are M by L, L by N, and M by N.        */

main ()
{
   group_t   init_group, new_group;
   comm_t    comm;
   int       init_rank, key, ret, lenA, M, N, L;
   double    A[100], X[100][100], Y[100][100], Z[100][100];

   ret = get_input1 (lenA, M, N, L);      /* All processes get scalar input   */

   init_group = mpi_initial_group ();    /* Determine initial group           */
   init_rank  = mpi_rank (init_group);   /* Determine my rank in initial group*/

   key = init_rank&1;          /* Split initial group into odd/even subgroups */
   ret = mpi_split_group (init_group, key, init_rank, new_group);

   ret = mpi_safemake_comm (new_group, new_group, comm);

   if (key) {
      ret = initialize_fft (A, lenA, comm);
      ret = parallel_fft   (A, lenA, comm);
      ret = output_fft     (A, lenA, comm);
   }
   else {
      ret = initialize_matmul (X, Y, Z, M, N, L, comm);
      ret = parallel_matmul   (X, Y, Z, M, N, L, comm);
      ret = output_matmul     (X, Y, Z, M, N, L, comm);
   }

   exit (0);
}

/* PROGRAM 2.
    Now we'll try something a little harder. In the next program we find the
    convolution, C, of two vectors, A and B, by first finding the FFT of A
    and B, evaluating their elementwise product in Fourier space, and then
    finding the inverse transform, i.e.,
    
	C = INVFFT ( FFT(A) * FFT(B) )

    Of course, we can do this by having all processes first find FFT(A) and
    then FFT(B). The product in Fourier space can then be found with no
    communication being necessary. Finally, all processes cooperate to
    perform the inverse FFT. Program 2 doesn't do this. Instead, we split
    the initial group into two equally-sized subgroups, one of which
    evaluates FFT(A) and the other FFT(B) in parallel. Communication is
    then necessary between the initial group and the subgroups to 
    re-distribute the data before doing the Fourier space product in the
    initial group. Finally, the processes in the initial group evaluate
    the inverse FFT, giving the convolution, C. The point here is that
    communication between groups is required.                                 */

main ()
{
   group_t init_group, fft_group;
   comm_t  comm_fft, comm_merge_A, comm_merge_B, comm_inv;
   comm_handle_t handle1, handle2;
   rso_handle_t  rso;
   int     init_rank, init_size, key, fftlen, local_len, nbytes;
   int     source_rank, dest_rank1, dest_rank2;
   double  A[100], B[100], C[100];
   double tmpA[100], tmpB[100];

   ret = get_input2 (fftlen);

   init_group = mpi_initial_group ();
   init_rank  = mpi_rank (init_group);
   init_size  = mpi_group_size (init_group);

   local_len  = fftlen/init_size;
   nbytes     = local_len*sizeof(double);

   key = init_rank&1;
   ret = mpi_split_group (init_group, key, init_rank, fft_group);

   ret = mpi_safemake_comm (fft_group,  fft_group,  comm_fft);
   ret = mpi_safemake_comm (init_group, init_group, comm_inv);

   source_rank = mpi_rank (fft_group);
   dest_rank1  = 2*source_rank;
   dest_rank2  = dest_rank1 + 1;

   if (key) {
      ret = mpi_safemake_comm (fft_group,  init_group, comm_merge_A);
      ret = mpi_publish (comm_merge_A, "GROUP_A");
      ret = initialize_fft (A, fftlen, comm_fft);
      ret = parallel_fft   (A, fftlen, comm_fft);
      ret = mpi_isendc (handle1, &A[0], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank1, comm_merge_A);
      ret = mpi_isendc (handle2, &A[local_len], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank2, comm_merge_A);
      ret = mpi_subscribe ("GROUP_B", comm_merge_B);
   }
   else {
      ret = mpi_safemake_comm (fft_group,  init_group, comm_merge_B);
      ret = mpi_publish (comm_merge_B, "GROUP_B");
      ret = initialize_fft (B, fftlen, comm_fft);
      ret = parallel_fft   (B, fftlen, comm_fft);
      ret = mpi_isendc (handle1, &B[0], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank1, comm_merge_B);
      ret = mpi_isendc (handle2, &B[local_len], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank2, comm_merge_B);
      ret = mpi_subscribe ("GROUP_A", comm_merge_A);
   }

   ret = mpi_recvc (tmpA, nbytes, MPI_DOUBLE, MPI_DONTCARE, 
                       source_rank, comm_merge_A, rso);
   ret = mpi_recvc (tmpB, nbytes, MPI_DOUBLE, MPI_DONTCARE, 
                       source_rank, comm_merge_B, rso);
  
   ret = mpi_wait (handle1, rso);
   ret = mpi_wait (handle2, rso);

   ret = parallel_product (tmpA, tmpB, C, fftlen, comm_inv);
   ret = parallel_invfft  (C, fftlen, comm_inv);
   ret = output_result    (A, B, C, fftlen, comm_inv); 

   exit (0);
}

>From [email-masked] Tue Jun  8 20:42:16 1993
Date: Tue, 8 Jun 1993 18:07:19 -0400
From: [email-masked] (David Walker)
To: [email-masked]
Subject: Correction to second program
Status: R

Sorry! There was an error in the second example program I sent
out this afternoon. Below is a corrected version.

David
/* PROGRAM 1.
    Just to get warmed up here's a simple MPI program that splits the
    initial group into subgroups. One subgroup contains the processes
    with even ranks in the initial group, and the other the odd ranks.
    Then, one subgroup does a parallel FFT (A->FFT(A)), and the other a
    parallel matrix multiplication (Z=XY). The global length of A is lenA.
    The global sizes of X, Y, and Z are M by L, L by N, and M by N.        */

main ()
{
   group_t   init_group, new_group;
   comm_t    comm;
   int       init_rank, key, ret, lenA, M, N, L;
   double    A[100], X[100][100], Y[100][100], Z[100][100];

   ret = get_input1 (lenA, M, N, L);      /* All processes get scalar input   */

   init_group = mpi_initial_group ();    /* Determine initial group           */
   init_rank  = mpi_rank (init_group);   /* Determine my rank in initial group*/

   key = init_rank&1;          /* Split initial group into odd/even subgroups */
   ret = mpi_split_group (init_group, key, init_rank, new_group);

   ret = mpi_safemake_comm (new_group, new_group, comm);

   if (key) {
      ret = initialize_fft (A, lenA, comm);
      ret = parallel_fft   (A, lenA, comm);
      ret = output_fft     (A, lenA, comm);
   }
   else {
      ret = initialize_matmul (X, Y, Z, M, N, L, comm);
      ret = parallel_matmul   (X, Y, Z, M, N, L, comm);
      ret = output_matmul     (X, Y, Z, M, N, L, comm);
   }

   exit (0);
}

/* PROGRAM 2.
    Now we'll try something a little harder. In the next program we find the
    convolution, C, of two vectors, A and B, by first finding the FFT of A
    and B, evaluating their elementwise product in Fourier space, and then
    finding the inverse transform, i.e.,
    
	C = INVFFT ( FFT(A) * FFT(B) )

    Of course, we can do this by having all processes first find FFT(A) and
    then FFT(B). The product in Fourier space can then be found with no
    communication being necessary. Finally, all processes cooperate to
    perform the inverse FFT. Program 2 doesn't do this. Instead, we split
    the initial group into two equally-sized subgroups, one of which
    evaluates FFT(A) and the other FFT(B) in parallel. Communication is
    then necessary between the initial group and the subgroups to 
    re-distribute the data before doing the Fourier space product in the
    initial group. Finally, the processes in the initial group evaluate
    the inverse FFT, giving the convolution, C. The point here is that
    communication between groups is required.                                 */

main ()
{
   group_t init_group, fft_group;
   comm_t  comm_fft, comm_merge, comm_merge_A, comm_merge_B, comm_inv;
   comm_handle_t handle1, handle2;
   rso_handle_t  rso;
   int     init_rank, init_size, key, fftlen, local_len, nbytes;
   int     source_rank, dest_rank1, dest_rank2;
   double  A[100], B[100], C[100];
   double tmpA[100], tmpB[100];

   ret = get_input2 (fftlen);

   init_group = mpi_initial_group ();
   init_rank  = mpi_rank (init_group);
   init_size  = mpi_group_size (init_group);

   local_len  = fftlen/init_size;
   nbytes     = local_len*sizeof(double);

   key = init_rank&1;
   ret = mpi_split_group (init_group, key, init_rank, fft_group);

   ret = mpi_safemake_comm (fft_group,  fft_group,  comm_fft);
   ret = mpi_safemake_comm (fft_group,  init_group, comm_merge);
   ret = mpi_safemake_comm (init_group, init_group, comm_inv);

   source_rank = mpi_rank (fft_group);
   dest_rank1  = 2*source_rank;
   dest_rank2  = dest_rank1 + 1;

   if (key) {
      ret = mpi_copy_comm (comm_merge, comm_merge_A);
      ret = mpi_safemake_comm (fft_group,  init_group, comm_merge_A);
      ret = mpi_publish (comm_merge_A, "GROUP_A");
      ret = initialize_fft (A, fftlen, comm_fft);
      ret = parallel_fft   (A, fftlen, comm_fft);
      ret = mpi_isendc (handle1, &A[0], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank1, comm_merge_A);
      ret = mpi_isendc (handle2, &A[local_len], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank2, comm_merge_A);
      ret = mpi_subscribe ("GROUP_B", comm_merge_B);
   }
   else {
      ret = mpi_copy_comm (comm_merge, comm_merge_B);
      ret = mpi_publish (comm_merge_B, "GROUP_B");
      ret = initialize_fft (B, fftlen, comm_fft);
      ret = parallel_fft   (B, fftlen, comm_fft);
      ret = mpi_isendc (handle1, &B[0], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank1, comm_merge_B);
      ret = mpi_isendc (handle2, &B[local_len], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank2, comm_merge_B);
      ret = mpi_subscribe ("GROUP_A", comm_merge_A);
   }

   ret = mpi_recvc (tmpA, nbytes, MPI_DOUBLE, MPI_DONTCARE, 
                       source_rank, comm_merge_A, rso);
   ret = mpi_recvc (tmpB, nbytes, MPI_DOUBLE, MPI_DONTCARE, 
                       source_rank, comm_merge_B, rso);
  
   ret = mpi_wait (handle1, rso);
   ret = mpi_wait (handle2, rso);

   ret = parallel_product (tmpA, tmpB, C, fftlen, comm_inv);
   ret = parallel_invfft  (C, fftlen, comm_inv);
   ret = output_result    (A, B, C, fftlen, comm_inv); 

   exit (0);
}

>From [email-masked] Wed Jun  9 04:36:46 1993
Date: Wed, 9 Jun 93 10:30:08 BST
From: L J Clarke <[email-masked]>
Subject: Re: Correction to second program
To: [email-masked] (David Walker), [email-masked]
Reply-To: [email-masked]
Status: R

Hi y'all

I agree that it is useful to have examples. 

The first example you sent shows that MPI makes it quite easy to do the
kind of function driven parallelism where the two "functions" do not
communicate.  The second example starts to think about communication
between two different "functions".  The two functions communicate by
passing use of the common ancestor which was easy as the ancestor was a
parent and is in scope at the time of communication. 

I'm a little confused about the mpi_safemake_comm calls.  I'm more
confused by the receives and sends in the second example.  Here the
destination rank in the sends appears to be different to the intended
source rank in the receives.  Also the receives from the "other" group
appear to be choosing the source which the publish/subscribe example
described by Tony does not permit with the published/subscribed
communicator. [Perhaps this hardly matters in this simpl example since
each process receives just one message from the other group.]

Hopefully you can help unravel my confusion.

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Wed Jun  9 04:49:38 1993
Date: Wed, 9 Jun 93 10:44:15 BST
From: L J Clarke <[email-masked]>
Subject: Correction to [Re: Correction to second program]
To: [email-masked], [email-masked] (David Walker),
        [email-masked]
Reply-To: [email-masked]
Status: R

Oh dear, I'm not quite sure what happened at the end of the first para. 
in my previous message.  I've corrected (read "deleted the junk") and
sent again. 

I agree that it is useful to have examples. 

The first example you sent shows that MPI makes it quite easy to do the
kind of function driven parallelism where the two "functions" do not
communicate.  The second example starts to think about communication
between two different "functions". Perhaps I should devise a more
complicated yet simple example of this.

I'm a little confused about the mpi_safemake_comm calls.  I'm more
confused by the receives and sends in the second example.  Here the
destination rank in the sends appears to be different to the intended
source rank in the receives.  Also the receives from the "other" group
appear to be choosing the source which the publish/subscribe example
described by Tony does not permit with the published/subscribed
communicator. [Perhaps this hardly matters in this simpl example since
each process receives just one message from the other group.]

Hopefully you can help unravel my confusion.

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Wed Jun  9 07:56:55 1993
Date: Wed, 9 Jun 1993 08:50:17 -0400
From: [email-masked] (David Walker)
To: [email-masked]
Subject: Re: Correction to second program
Cc: [email-masked]
Status: R

> Date: Wed, 9 Jun 93 10:44:15 BST
> From: L J Clarke <[email-masked]>

> I agree that it is useful to have examples. 

	Good.

> The first example you sent shows that MPI makes it quite easy to do the
> kind of function driven parallelism where the two "functions" do not
> communicate.  The second example starts to think about communication
> between two different "functions". Perhaps I should devise a more
> complicated yet simple example of this.

	A more complicated example based on the convolution example could
	use overlapping groups. Assume for example that the parallel FFT
	requires the number of processes to be a power of two, and we have
	a total of 12 processes. Then we could put 0,1,2,..,7 in one group
	and 4,5,6,...,11 in the other group. I'm not sure how much this
	complicates things.

> I'm a little confused about the mpi_safemake_comm calls.

	Sorry, in my corrected version there was still an extraneous
	mpi_safemake_comm inside the first conditional branch. This has 
	been corrected in the revised version below. The arguments to
	my version of mpi_safemake_comm are:

		mpi_safemake_comm (local_group, remote_group, communicator)
		IN  local_group
		IN  remote_group
		OUT communicator

> I'm more confused by the receives and sends in the second example.  Here the
> destination rank in the sends appears to be different to the intended
> source rank in the receives.  Also the receives from the "other" group
> appear to be choosing the source which the publish/subscribe example
> described by Tony does not permit with the published/subscribed
> communicator. [Perhaps this hardly matters in this simpl example since
> each process receives just one message from the other group.]

	In the sends the rank of the destination process is relative to the
	remote group, i.e., the receiving group. In the receives the rank 
	of the source process is relative to the remote group, i.e., the
	sending group. This seems the natural way to do things. As you
	point out, the source rank in the receives can be replace by
	MPI_DONTCARE. It's not obvious to me why a C2 receive can't
	specify a particular source in the remote group. Please explain 
	this to me.
 
> Hopefully you can help unravel my confusion.

	Yes, hopefully.

Regards
David
/* PROGRAM 2.
    Now we'll try something a little harder. In the next program we find the
    convolution, C, of two vectors, A and B, by first finding the FFT of A
    and B, evaluating their elementwise product in Fourier space, and then
    finding the inverse transform, i.e.,
    
	C = INVFFT ( FFT(A) * FFT(B) )

    Of course, we can do this by having all processes first find FFT(A) and
    then FFT(B). The product in Fourier space can then be found with no
    communication being necessary. Finally, all processes cooperate to
    perform the inverse FFT. Program 2 doesn't do this. Instead, we split
    the initial group into two equally-sized subgroups, one of which
    evaluates FFT(A) and the other FFT(B) in parallel. Communication is
    then necessary between the initial group and the subgroups to 
    re-distribute the data before doing the Fourier space product in the
    initial group. Finally, the processes in the initial group evaluate
    the inverse FFT, giving the convolution, C. The point here is that
    communication between groups is required.                                 */

main ()
{
   group_t init_group, fft_group;
   comm_t  comm_fft, comm_merge, comm_merge_A, comm_merge_B, comm_inv;
   comm_handle_t handle1, handle2;
   rso_handle_t  rso;
   int     init_rank, init_size, key, fftlen, local_len, nbytes;
   int     source_rank, dest_rank1, dest_rank2;
   double  A[100], B[100], C[100];
   double tmpA[100], tmpB[100];

   ret = get_input2 (fftlen);

   init_group = mpi_initial_group ();
   init_rank  = mpi_rank (init_group);
   init_size  = mpi_group_size (init_group);

   local_len  = fftlen/init_size;
   nbytes     = local_len*sizeof(double);

   key = init_rank&1;
   ret = mpi_split_group (init_group, key, init_rank, fft_group);

   ret = mpi_safemake_comm (fft_group,  fft_group,  comm_fft);
   ret = mpi_safemake_comm (fft_group,  init_group, comm_merge);
   ret = mpi_safemake_comm (init_group, init_group, comm_inv);

   source_rank = mpi_rank (fft_group);
   dest_rank1  = 2*source_rank;
   dest_rank2  = dest_rank1 + 1;

   if (key) {
      ret = mpi_copy_comm (comm_merge, comm_merge_A);
      ret = mpi_publish (comm_merge_A, "GROUP_A");
      ret = initialize_fft (A, fftlen, comm_fft);
      ret = parallel_fft   (A, fftlen, comm_fft);
      ret = mpi_isendc (handle1, &A[0], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank1, comm_merge_A);
      ret = mpi_isendc (handle2, &A[local_len], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank2, comm_merge_A);
      ret = mpi_subscribe ("GROUP_B", comm_merge_B);
   }
   else {
      ret = mpi_copy_comm (comm_merge, comm_merge_B);
      ret = mpi_publish (comm_merge_B, "GROUP_B");
      ret = initialize_fft (B, fftlen, comm_fft);
      ret = parallel_fft   (B, fftlen, comm_fft);
      ret = mpi_isendc (handle1, &B[0], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank1, comm_merge_B);
      ret = mpi_isendc (handle2, &B[local_len], nbytes, MPI_DOUBLE,
                           MPI_DONTCARE, dest_rank2, comm_merge_B);
      ret = mpi_subscribe ("GROUP_A", comm_merge_A);
   }

   ret = mpi_recvc (tmpA, nbytes, MPI_DOUBLE, MPI_DONTCARE, 
                       MPI_DONTCARE, comm_merge_A, rso);
   ret = mpi_recvc (tmpB, nbytes, MPI_DOUBLE, MPI_DONTCARE, 
                       MPI_DONTCARE, comm_merge_B, rso);
  
   ret = mpi_wait (handle1, rso);
   ret = mpi_wait (handle2, rso);

   ret = parallel_product (tmpA, tmpB, C, fftlen, comm_inv);
   ret = parallel_invfft  (C, fftlen, comm_inv);
   ret = output_result    (A, B, C, fftlen, comm_inv); 

   exit (0);
}

>From [email-masked] Wed Jun  9 15:30:09 1993
Date: Wed, 9 Jun 93 08:31:47 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: May 26 mpi mail #1 of 1
Cc: [email-masked]
Status: R

>From [email-masked] Wed May 26 10:07:53 1993
To: [email-masked] (David Walker)
Cc: [email-masked]
Subject: Re: local map constructors 
Date: Wed, 26 May 93 09:06:00 MST
From: [email-masked]
Status: R

We did not have time at the last meeting to write up a large
set of useful constructors. We anticipated many of the operations
you suggest, and we also anticipate that map constructors might be
developed by third parties (i.e. there will be an interface that will
let you roll your own.)

Also, remember that maps are NOT sets, and therefore set operations
need something added to them to be meaningful. For example, suppose
I begin with the maps A: [0,1,2] -> [7, 2, 9] and B: [0,1] -> [7, 13].
What is their union? The union of the sets (2,7,9) and (7, 13) is well
defined: (2,7,9,13). But what ordering should be applied?

Mark Sears

>From [email-masked] Wed May 26 10:24:17 1993
Date: Wed, 26 May 1993 11:21:24 -0400
From: [email-masked] (David Walker)
To: [email-masked]
Subject: Re: local map constructors
Cc: [email-masked]
Status: R

Mark Sears writes:
> We did not have time at the last meeting to write up a large
> set of useful constructors. We anticipated many of the operations
> you suggest, and we also anticipate that map constructors might be
> developed by third parties (i.e. there will be an interface that will
> let you roll your own.)

	OK

> Also, remember that maps are NOT sets, and therefore set operations
> need something added to them to be meaningful. For example, suppose
> I begin with the maps A: [0,1,2] -> [7, 2, 9] and B: [0,1] -> [7, 13].
> What is their union? The union of the sets (2,7,9) and (7, 13) is well
> defined: (2,7,9,13). But what ordering should be applied?

	Well I thought maps were just sets. According to the MGC draft
	"A possible representation for such [a] map is a list of m integers."
	No mention of order there. A group, on the other hand, is an
	ordered set of processes, but the order is arbitary. I think a
	better definition of group is to say "A group is a set of N processes
	each of which is uniquely labeled by an integer in the range
	0 to N-1." So I would say that the union of the maps A and B is
	[2,7,9,13], or any other ordering of these 4 integers.

David

>From [email-masked] Thu Jun 10 14:39:10 1993
Date: Thu, 10 Jun 93 07:14:26 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: one more mpi email to forward
Status: R

>From [email-masked] Thu Jun 10 05:53:09 1993
Date: Thu, 10 Jun 93 11:48:07 BST
From: L J Clarke <[email-masked]>
Subject: Re: June 22 context meeting
To: Tony Skjellum <[email-masked]>, [email-masked]
Reply-To: [email-masked]
Status: R

I will arrive evening of June 21.

> Gentlemen, I will arrive on June 21, and hope to see as many of the context
> subcommittee members for pre-meeting discussion on June 22, at the Bristol Suites.
> I am writing this so everyone on the sub-committee will know that we are trying
> to roll up our sleeves again, this time, pre-meeting, to be sure we're on track
> by the time the meeting starts.  
> -Tony
> 
         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Thu Jun 10 17:42:46 1993
Date: Thu, 10 Jun 93 11:42:33 -0400
From: Mail Delivery Subsystem <[email-masked]>
Subject: Returned mail: Cannot send message for 3 days
To: [email-masked], [email-masked], [email-masked]
Status: R

   ----- Transcript of session follows -----
421 surfer.epm.ornl.gov (tcp)... Deferred: Connection timed out during user open with surfer.epm.ornl.gov

   ----- Unsent message follows -----
Date: Mon, 7 Jun 1993 11:12:32 -0400
From: [email-masked] (David Walker)
To: [email-masked]
Subject: Next MPI meeting

The next meeting of the Message Passing Interface (MPI) Forum will take
place in Dallas, June 23-25, 1993. Details are given below. 

The minutes of the last meeting in May are not yet available
electronically, but should be soon.

The ongoing email discussion on MPI standardization issues can be obtained
from netlib. To find out what is available via email send the
following message to [email-masked]:

	send index from mpi

or use xnetlib ("send index from xnetlib" for info).

The schedule for future meetings is as follows:

	August 11-13, 1993      Dallas
	September 22-24, 1993   Dallas
	October 27-29, 1993     Europe

MPI Forum Meeting, June 23-25, 1993
===================================

The next meeting of the MPI forum will take place at

      Bristol Suites Hotel
      7800 Alpha Road
      Dallas, Texas
      
The meeting will start at 1pm on Wednesday June 23, 1993, and finish at 
noon on Friday, June 25, 1993. 

Rooms are $89 per night and reservations may be made by calling (214) 233-7600 
(mention MPI meeting). 

The meeting registration fee will be $75. Please make checks and POs payable 
to University of Tennessee. The registration fee will be collected at the 
meeting. The registration fee will go for coffee breaks, meeting rooms, AV and 
printer rentals.

Certain organizations need to see a registration form before giving their
employees money for meetings like this. You can get such a form in PostScript
by sending the following message to [email-masked]:

	send mpi-form.ps from mpi

There is no need to send me the registration form or bring it to the meeting.

TBS Shuttle Service will be providing complimentary shuttle service to
and from the airports.  If you fly into DFW, use their courtesy telephone
and dial 03.  If you fly into Love Field, you'll have to use a pay phone.
They can be reached at 817-267-5150.  Upon boarding the shuttle refer to the 
MPI meeting.

We have NOT been able to make any special arrangements with airlines to get 
reduced fares. 

We have secured limited funding from ARPA/NSF for travel expenses of MPI
meeting participants who are from U.S. universities. If you would like
to apply for financial support to attend the June MPI meeting please
send email to me at [email-masked], with justification of why you
need support and an estimate of your travel expenses.

Please send comments and/or suggested changes to the agenda below to me at
[email-masked]

        Provisional Agenda for MPI Meeting, June 23-25, 1993

Wednesday
1:30-6:00   First Reading of the contexts draft.
            (Skjellum)
6:00-7:30   Unofficial dinner break 
7:30-10:30  Break up for subcommittee meetings
        
Thursday
9:00-12:00  First reading of the process topologies draft.
	    (Hempel)
12:00-1:30  Lunch (provided)
1:30-4:30   Second reading of the collective communication draft
	    (Geist)
4:30-6:00   Second reading of the profiling draft
	    (Cownie)
6:00-8:00   Dinner (attendees pay, but hotel provides transport to area
                   restaurant)
8:00-10:00  Continued informal subcommittee meetings if necessary
        
Friday 
9:00-10:30  First reading of the subset draft.
	    (Huss-Lederman)
10:30-12:00 First reading of the formal specifications draft.
	    (???)

------- End of Forwarded Message

---

From: L J Clarke <[email-masked]>
Date: Thu, 10 Jun 93 18:43:07 BST

Hi

As promised, here is David's convolution example modified.  It uses a
bare context in one place.  Notice how the relation of ranks of sender
and receiver is simpler.  No calculations needed in this case.  I
optimised as well by removing some redundant communications - hope I
understood correctly!

Actually I must say that in this particular example everything is so
simple that it really is easier to do the merge in the initial group. 
This is an artefact of the example.  We should imagine the situation
where the initial group is no longer in scope (because things are down
in libraries) or where there are other things going on in other toplevel
subgroups of the initial group at the same time (because there is
function and data/event driven parallelism going on at the same time). 
Then it gets messy to do the merge in the initial group. 

Discussion of example
---------------------

First I better describe the less familiar mpi style routines.

mpi_safemake_comm(group, comm)

IN  group - the local and remote groups for the communicator
OUT comm  - the communicator made

Note that this is an accelerator. It is defined as

mpi_alloc_contexts(group, 1, &context)
mpi_make_comm(group, context, comm)

although it can be implemented faster.  It is synchronous becasue
mpi_alloc_contexts is synchronous - ie it synchronises processes in
group. 

mpi_make_comm(group, context, comm)

IN  group   - the local and remote groups for the communicator
IN  context - the local and remote contexts for the communicator
OUT comm    - the communicator made

Note that this is an accelerator. It is defined as

mpi_make_comma(group, context, group, context, comm)

although it can be implemented faster. It is asynchronous -
ie does not synchronise processes.

mpi_make_comma(group1, context1, group2, context2, comm)

IN  group1   - the local group for the communicator
IN  context1 - the local context for the communicator
IN  group2   - the remote group for the communicator
IN  context2 - the remote context for the communicator
OUT comm     - the communicator made

This procedure is asynchronous - ie does not synchronise processes. 

mpi_publish(group, context, name)

IN  group   - group to associate with name, i.e. publish
IN  context - context to associate with name, i.e. publish
IN  name    - name to publish by

This procedure is asynchronous - ie does not synchronise processes. 

mpi_subscribe(name, group, context)

IN  name    - name to subscribe by
OUT group   - group associated with name
OUT context - context associated with name

This procedure is asynchronous - ie does not synchronise processes. 

Now I explain out why it is better to publish a (group, context) pair
rather than a communicator.  Its simple really.  If you publish by
communicator then when you add the two communicators together
(mpi_make_comma) you have to delete the old communicators, because the
contexts are already in use.  This means you just have to create this
intermdiate object in order to add two of them together and then delete
the intermediate.  This is messy.  So the code is simpler if you
publish/subscribe on a group and context pair instead of a communicator. 
The other problem is whether we define mpi_free of a communicator to
delete the group and/or context. 

----------------------------------------------------------------------
main ()
{
   group_t       init_group, fft_group, other_group;
   comm_t        comm_fft, comm_merge, comm_inv;
   context_t     context, other_context;
   comm_handle_t handle;
   rso_handle_t  rso;
   int           init_rank, init_size, key, 
                 fft_rank, fft_size,
                 fftlen, local_len, nbytes;
   double        A[100], B[100], C[100];

   key = init_rank&1;
   ret = mpi_split_group (init_group, key, 0, fft_group);
   fft_rank = mpi_rank (fft_group);

   mpi_alloc_context(fft_group, 1, &context);
   if (fft_rank == 0)
   {
     ret = mpi_publish (fft_group, context , (key ? "even" : "odd");
   }
   ret = mpi_subscribe ((key ? "odd" : "even") , &other_group, &other_context);
   ret = mpi_make_comma(fft_group, context, 
                        other_group, other_context, comm_merge);

   if (key) {
     ret = mpi_isendc (handle, &A[local_len], nbytes, MPI_DOUBLE,
                        0, fft_rank, merge_A);
     ret = mpi_recvc (B, nbytes, MPI_DOUBLE,
                      0, fft_rank, comm_merge, rso);
     ret = mpi_wait (handle, rso);

   }
   else {
     ret = mpi_isendc (handle, &A[0], nbytes, MPI_DOUBLE,
                        0, fft_rank, comm_merge);
     ret = mpi_recvc (B, nbytes, MPI_DOUBLE,
                      0, fft_rank, comm_merge, rso);
     ret = mpi_wait (handle, rso);
     bcopy((&A[local_len], &[0], nbytes);
   }

   ret = mpi_safemake_comm (init_group, comm_inv);
 
   ret = parallel_product (A, B, C, fftlen, comm_inv);
   ret = parallel_invfft  (C, fftlen, comm_inv);
   ret = output_result    (C, fftlen, comm_inv);

   exit (0);
}

----------------------------------------------------------------------

---

From: [email-masked] (David Walker)
Date: Thu, 10 Jun 1993 15:17:07 -0400

Lyndon,
	First I've corected a few typos in our code. These lines I've marked ">>"
main ()
{
   group_t       init_group, fft_group, other_group;
   comm_t        comm_fft, comm_merge, comm_inv;
   context_t     context, other_context;
   comm_handle_t handle;
   rso_handle_t  rso;
>> int           init_rank, init_size, key, ret
                 fft_rank, fft_size,

   key = init_rank&1;

>> ret = mpi_safemake_comm (fft_group, comm_fft);

   if (key) {
>>   ret = mpi_recvc (&B[0], nbytes, MPI_DOUBLE,   /* changed this for consistency */
>>   ret = mpi_recvc (&B[0], nbytes, MPI_DOUBLE,  /* changed this for consistency */
>>   bcopy((&A[local_len], &A[0], nbytes);
   }

>> ret = parallel_product (A, B, C, local_len);/* No communication -> no comm_inv needed */

   exit (0);
}

Do we now agree that the above is correct?

Your code differs from mine since you actually communicate between the "odd" and
"even" groups, whereas in my example communication is between the "even" group and
the initial group, and the "odd" group and the initial group.

Let N be the number of processors in the initial group. Then the 2 subgroups each
contain N/2 processes (we assume N to be even). The subgroups do not overlap. Any
method can be used to divide the initial group into equal sized, non-intersecting
subgroups. We form subgroups based on odd/even ranking in the initial group.

Immediately before the communication phase group 1 (G1) contains the first FFT in A,
and group 2 (G2) contains the other FFT in A.

On completion of the communication and buffer copy stuff, what do A and B contain
in each group? G1 contains even chunks of FFT1 in A and even chunks of FFT2 in B.
G2 contains odd chunks of FFT1 in A and odd chunks of FFT2 in B. Relative
to the initial group everything is in the correct place to form a correctly
distributed C.

My example code is correct regardless of how the subgroups are formed. Lyndon's code
is correct only if the odd/even method for forming subgroups is used. That's why I used
communication between each subgroup and the initial group to get the data correctly
distributed prior to the product and invfft steps. Anyway this is a minor point. Both
examples are valid and useful, though mine could do with a bit more tidying up.

Did we decide at the last meeting that all processes bound into a communicator had 
to sync before using it? If so the above code needs a sync after the mpi_make_comma call.

Well that all for now.

David

---

From: L J Clarke <[email-masked]>
Date: Fri, 11 Jun 93 11:53:51 BST

Hi David

Thanks for the typo corrections and further comments.  I realise that
there are further redundancies/optimisations, and just one typo, so I
enclose hopefully a final version of this line at the foot of the
message. 

> the initial group, and the "odd" group and the initial group.

Yes.  On reflection, in this example, you did it right and I did it
wrong.  Lets look at the timeline of this program. 

   -------------------------        -------------------------
  |  odd group - fft        |      | even group - fft        |
   -------------------------        -------------------------
          ||                             ||
          ||                             ||
          \/                             \/
   ----------------------------------------------------------
  |   init group - parallel product then inverse fft         |
   ----------------------------------------------------------

The arrows represent the merge communications.  These comms are clearly
happening between the two fft groups which I've labelled odd and even,
and the init group.  So its best programming practice to write the
communications that way.  I got us into this mess - I'll send out a
cleaned up version of your original example later. 

> to the initial group everything is in the correct place to form a correctly

Yup.

> is correct only if the odd/even method for forming subgroups is used. 

I disagree.  In the code you wrote the program calculates the
destination ranks dest_rank1 and dest_rank2.  This calculation uses
knowledge of how the subgroups were formed to relate ranks in the fft
groups to those in the init group.  Both codes rely on how the subgroups
were formed. 

> to sync before using it? If so the above code needs a sync after the mpi_make_comma call.

I recall Tony explaining that in the Friday morning meeting the opposite
agreement was made after a point raised by Paul Pierce.  So no, I do not
think that an mpi_sync is needed.  In this example it doesnt really
matter an iota whether or not there has to be a sync there, since the
program is strongly SPMD and its safe to rely on
sequencing/synchronisation. 

Best Wishes
Lyndon

----------------------------------------------------------------------
main ()
{
   group_t       init_group, fft_group, other_group;
   comm_t        comm_fft, comm_merge, comm_inv;
   context_t     context, other_context;
   comm_handle_t handle;
   rso_handle_t  rso;
   int           init_rank, init_size, key, ret,
                 fft_rank, fft_size, fftlen, 
                 local_len, nbytes, keep_part, give_part;
   double        A[100], B[100], C[100];

   key = init_rank&1;

   

   keep_part = (key != 0) ? 0 : local_len;
   give_part = (key == 0) ? 0 : local_len;
     
   ret = mpi_isendc (handle, &A[give_part], nbytes, MPI_DOUBLE,
                                      0, fft_rank, comm_merge);
   ret = mpi_recvc (&B[keep_part], nbytes, MPI_DOUBLE,   
                        0, fft_rank, comm_merge, rso);
   ret = mpi_wait (handle, rso);

   ret = parallel_product (&A[keep_part], &B[keep_part], C, local_len);

   exit (0);
}

---

From: [email-masked] (David Walker)
Date: Fri, 11 Jun 1993 09:37:36 -0400

Date: Fri, 11 Jun 93 11:53:51 BST
From: L J Clarke <[email-masked]>

> Hi David
> 
>    [stuff deleted]
> 
>> Your code differs from mine since you actually communicate between the "odd"
>> and "even" groups, whereas in my example communication is between the "even"
>> group and the initial group, and the "odd" group and the initial group.
> 

   I look forward to seeing the cleaned up version of my code.

>>   [stuff deleted]

>> My example code is correct regardless of how the subgroups are formed. 
>> Lyndon's code is correct only if the odd/even method for forming 
>> subgroups is used. 

> groups to those in the init group.  Both codes rely on how the subgroups

   I still think I am right. When sending, the rank of the destination
   process in the init_group is relative to the init_group. 
   Similarly when receiving, the rank of the source process in the 
   fft_group is relative to that group. In my code I can change the way
   key is calculated, and hence the composition of the subgroups, and
   still everything will work OK. For example, I could say,

		key = (init_rank<init_size/2) ? 0 : 1;

   More generally, my example will still work for 3 arbitrary
   groups, provided the two FFT groups are distinct and each is
   half the size of the third group (which does the product and inverse
   FFT). For example, if we had N processes, N/4 could work on FFT1,
   N/4 could work on FFT2, and N/2 could work on the product and
   inverse FFT. So I could input one array and find its FFT on a bunch
   of workstations at ORNL, input and do the second FFT on the Delta at
   Caltech, and each could send its results to EPCC where the product and
   inverse are done on a transputer array and displayed to a graphical
   device.
 
>> Did we decide at the last meeting that all processes bound into a 
>> communicator had to sync before using it? If so the above code needs a 
>> sync after the mpi_make_comma call.

> sequencing/synchronisation. 
 
   Well if we're not requiring synchronization (which would amount to
   another sort of ready mode related to communicators) this discussion
   is academic (so I'll continue). It seems to me that if one subgroup
   is running on a very faster bunch of processors and the other isn't 
   then the fast group could whip through its mpi_make_comma and
   mpi_isendc and get a message to the other group before that group
   could complete its mpi_make_comma

Best Wishes,
David

> Lyndon's code --------------------------------------------------------
> main ()
> {