Subject: Re: Meeting in Europe ?

From: [email-masked] (Dan Nessett)
Date: Wed, 21 Apr 1993 13:50:13 -0800

Because of budgetary cuts, foreign travel has been restricted recently for
us at LLNL. So, I couldn't make it to a meeting held outside the US.

Dan

---

From: [email-masked] (Tom Henderson)
Date: Wed, 21 Apr 93 15:24:44 MDT

> Heh.  I'm fully willing to go to Europe.  However, even if Europe is not the
> eventual venue, somewhere (anywhere?) other than a hotel isolated in North
> Dallas is to be desired.
> 
> moose

I'll second this.  If this is a subcommittee meeting, you've got two votes 
from this organization!  

Tom Henderson
NOAA Forecast Systems Laboratory

---

From: [email-masked] (Anne C. Elster)
Date: Wed, 21 Apr 93 23:10:31 -0400

James Cownie writes:

>At the last MPI meeting we agreed to have a meeting in August (and
>possibly even September as well).
>
>After this decision had been taken, various people approached me and
>asked whether we could have one of these meetings in Europe. (Not only
>Rolf and Lyndon !).
>

Hear, hear! If MPI is supposed to be a truely INTERNATIONAL forum,
I think it is indeed appropriate to schedule at least one of the
meetings outside the U.S.  Now, if the organizers (Jack/David) would
still sponsor at least some of the airfare for those of us (academics)
they now support, there is no way you can keep me away from the meeting! :-)

> eventual venue, somewhere (anywhere?) other than a hotel isolated in North

No kidding!  Hmmm ..  maybe the HPF peope picked the local to make sure
people didn't feel like taking some sessions off to explore less technical
things ... ;-) (devious smile)

							Anne

-------------------------------------------------------------------------------
Anne C. Elster			   |  (607) 254-8653 [off]   255-6236 [FAX]
Space Plasma Physics		   |
School of Electrical Engineering   |  (607) 272-2986 [home/answ.mach]
Cornell University, 351 Theory Ctr.|
Ithaca, NY 14853		   |   e-mail:  [email-masked]
USA				   |	        [email-masked]

---

From: [email-masked]
Date: Thu, 22 Apr 1993 07:44:49 GMT

It's an excellent idea to hold one of the meetings in Europe. If you
need an additional reason to accept it, here it is: in August and
probably also in September it's a lot cooler in Europe (especially in
England) than at Dallas.

Rolf Hempel

---

From: L J Clarke <[email-masked]>
Date: Thu, 22 Apr 93 09:51:46 BST

> asked whether we could have one of these meetings in Europe. (Not only

I solid proposal if ever I heard one indeed! My support can be assumed.

> I have since spoken to Professor Hey, who has kindly offered to host
> the meeting at Southampton and handle local arrangements if we were to
> decide to hold a meeting in Europe.

Southampton would be fine.  We could probably hold one here in Edinburgh
(much prettier town than Southampton, imho :-).  Of course I'd just love
that. 

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

>From [email-masked] Wed May 12 04:35:20 1993
From: [email-masked] (Anne C. Elster)
Date: Wed, 12 May 93 05:14:09 -0400
To: [email-masked]
Subject: MPI_LOAD_INFO sugestion
Cc: [email-masked]
Status: R

As much as I hate to expand the number of fuctions in MPI,
I think this function is a MUST the way I see parallel
computing in teh future.

Below is a latex-version of the write-up I will be bringing to
the meeting today. (Yes, yes, I know I should have discussed
this long time ago, but ...). I will bring hard-copies to the meeting,
so that people can start thinking about it.

	Hope you like the idea!

- -------------------------------------------------------------------------------
Anne C. Elster			   |  (607) 254-8653 [off]   255-6236 [FAX]
Xerox Design Research Institute	   |  (607) 272-2986 [home/answ.mach]
Cornell University, 502 Theory Ctr.|
Ithaca, NY 14853		   |   e-mail:  [email-masked]
USA				   |	        [email-masked]

\documentstyle[fullpage,12pt]{article}
\baselineskip=12pt
\oddsidemargin -0.6in

\begin{document}

\setlength{\parskip}{1ex}
% \maketitle

%\pagestyle{empty}

\begin{center}
{\large MPI\_LOAD\_INFO: Implementing load-balanced applications
		on distributed-memory systems}

Anne C. Elster\\
\small Cornell University, Ithaca NY 14853

\date{\today}

\end{center}

\footnotesize

\subsection*{Motivation}

Currently, most parallelized algorithms for distributed memory machines
assume that they will run on a homogeneous set of processors.
(I.e. that each node in the parallel system has approximately
the same computing power and I/O speed.)
The main focus in parallel computing has thus far been to minimize
communication over-head and achieve a load-balance for the
particular algorithm or application.

However, as each computational node a parallel system become more complex, 
their individual processor/node performance characteristics will tend to
deviate more as some nodes perform I/O while others concentrate on
system tasks, etc.
This behavior has so far been addressed through multi-tasking on
large shared-memory systems such as a network of IBM 3090s, however,
no simple, straight-forward solutions exist currently
for the distributed memory cases.

It is, however, this authors belief that load-balancing application
at the ``load-info'' level (i.e. based on each node's dynamic load at
run-time) will become very important in the future -- especially
if there were mechanisms built into MPI that would facilitate
such implementations.

\subsection*{Current options}

The current options for getting load-info read into
a Fortran or C application on Unix-based systems are
not wonderful:

System load is typically from the OS-side done by {\bf reading kernel
memory (-kmem)}. This will, however, usually require {\em setgid}
permissions and grubbing through kernel symbol tables.
This  brute-force approach might fork and exec something with the required
kmem permissions.

Another alternative is to use {\bf rstat} which  does not require
such permission, but unfortunately, is not available on all systems.
Rstat typically gives you statistics from a remote kernel
that should be current disregarding network delay.

The third approach I can think of is to use the output from 
{\bf uptime}. In C, this could possibly be done though
 fscanf on the return value of popen.

\subsection*{MPI suggestion}

Since I fond none of the previous methods very programmer-friendly,
my suggestion  is to add a {\bf MPI\_LOAD\_INFO} that would return
the desired information. In the point-to-point case, this would
be similar to what one could get from $rstat$, whereas in the
collective sense, one could be gathering information on a whole
process group (or nodes within a context?) . I see the latter
as VERY useful. 

Clearly the specific format of the info. to be received via
such an MPI call needs to be discussed.

If the feed-back on this suggestion is positive, this author
would gladly write up a more detailed proposal for such a feature.

Anne C. Elster
Cornell Univ.

\end{document}

>From [email-masked] Wed Jun  9 14:59:20 1993
Date: Wed, 9 Jun 93 07:59:43 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: May 12 mpi mail #4
Status: R

Another one from Steve Otto (initial verbosity deleted so I could print the
LaTeX file --DW):

\documentstyle[twoside,11pt]{report}
%\pagestyle{headings}
%%\markright{ {\em Draft Document of the MPI Standard,\/ \today} }

\marginparwidth 0pt
\oddsidemargin=.25in
\evensidemargin  .25in
\marginparsep 0pt
\topmargin=-.5in
\textwidth=6.0in
\textheight=9.0in
\parindent=2em

%   ----------------------------------------------------------------------
%   mpi-macs.tex  --- man page macros,
%                discuss, missing, mpifunc macros
%
% ----------------------------------------------------------------------
% a couple of commands from Marc Snir, modified S. Otto

\newlength{\discussSpace}
\setlength{\discussSpace}{.7cm}

\newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace}
}

\newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace}
}

\newlength{\codeSpace}
\setlength{\codeSpace}{.3cm}

\newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} }

% -----------------------------------------------------------------------
%  A few commands to help in writing MPI man pages
%
\def\twoc#1#2{
\begin{list}
{\hbox to95pt{#1\hfil}}
{\setlength{\leftmargin}{120pt}
 \setlength{\labelwidth}{95pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#2}
\end{list}
}
\outer\long\def\onec#1{
\begin{list}
{}
{\setlength{\leftmargin}{25pt}
 \setlength{\labelwidth}{0pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#1}
\end{list}
}
\def\manhead#1{\noindent{\bf{#1}}}

\hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple}

\begin{document}

\setcounter{page}{1}
\pagenumbering{roman}

\title{ {\em D R A F T} \\ Proposal VIII for contexts and groups}

\author{Mark P. Sears, {\em Sandia National Laboratories}
}

\maketitle
\hfuzz=5pt
%\tableofcontents

%\begin{abstract}
%We don't have an abstract yet.
%\end{abstract}

\setcounter{page}{1}
\pagenumbering{arabic}

\label{sec:context}

%======================================================================%
% BEGIN "Proposal VIII"
%
% written by Mark P. Sears
%            Sandia National Laboratories
%            March 30, 1993
%

% I defined these because the discuss macro wont let me
% use \verb or verbatim environment.

\def\dishdr{\vspace{\discussSpace} \begingroup\setlength{\parindent}{0pt}
\setlength{\parskip}{0pt}\leftskip=.6cm \small {\bf Discussion: }}

\def\disend{\par\vspace{\discussSpace} \endgroup\par }

% end of macro definitions

\chapter{Contexts and Groups -- Proposal VIII}
\begin{center}
{Mark P. Sears, Sandia National Laboratories}
\end{center}

\section{Introduction and motivation.}

\dishdr
March 30 draft -- handed out at MPI meeting

April 13 draft -- modifications to context allocation scheme, added
         discussion of limitations, interaction with pt2pt draft,
         some reorganization.

May 11 draft -- added getgroupcontext function which allows the
         allocation of a dynamic context within a group.

On reading this proposal, anything in the discussion sections is
considered to be commentary, is open to debate and is probably
not part of the proposal as it would go into the MPI document as a whole.
\disend

The MPI committee has determined that message passing common practice
needs to be extended to satisfy two objectives. The first of these is
to add facilities which allow development and use of independent software
modules (libraries and other third party software). The primary need
here is seen to be enabling the establishment of independent communication
environment, or context, for independent software components.
 The second objective is to
add capabilities, especially for collective communications, which allow
the software developer to use subsets, or groups, of the processes in the
parallel task in a convenient way. The purpose of this chapter is to
define these concepts more precisely. The development of these ideas
follows primarily from the principle that context and group concepts should
be as orthogonal as possible, in order to make them easy to understand
and implement.

The remainder of this chapter defines the terms {\bf process},
 {\bf parallel task}, {\bf process address},  {\bf context},
 {\bf group }, and {\bf topology}  as they are used in this document.
In addition this chapter describes the MPI functions needed to assign,
create, and manipulate contexts and groups, and describes the overall syntax
needed to define point to point and collective communications. Later
chapters in the document contain precise descriptions of 
point to point and collective communication functions syntax and
semantics.

In the following, a {\bf global quantity}
 is one which is defined and can be used
in the same way on every process of the parallel task. For example, it
can be sent as part of a message from one process to another and used
on the second process in the same way as on the first. A {\bf local quantity}
is defined only on the process which creates it and cannot be sent
(in a meaningful way) to another process. Process addresses, tag values,
and context values are global quantitites; groups are local quantities.
Similarly a {\bf global call} or {\bf group call} is a function call
executed (loosely) synchronously on all processes or all processes
which are members of a group. A {\bf local call} is a function call
executed on a single process independent of any execution on other
processes.

\dishdr
This proposal differs from the other context-group proposals in that
context and group are held to be orthogonal and distinct entities. It
is closest in flavor to Proposal III, where context is a low level
concept, but here the definition of group is unrelated to the definition
of context. In this proposal, groups have no context or tags associated
with them, default or otherwise.

\begin{enumerate}

\item  The purpose of contexts is to promote software modularity by
allowing the construction of independent tag spaces. A context is an
integer-valued extension to the tag field which must match exactly between
sender and receiver of point-to-point communications.

\item In this proposal, contexts are global and there is no concept of a
process ``belonging'' to a context. Contexts are also anticipated to
be a scarce resource. Context allocation is a rare event, typically limited
to program startup.

\item The purpose of groups is to provide tools for management of subsets
of processes in a parallel task. A group is defined abstractly as an
ordered set of unique integers (the elements). Group as a type is a pointer
to an object which defines the mapping between rank and the elements
of the group, and is a locally defined opaque type. Note that a group
can be considered as a ``collection of processes'' only insofar as the
elements of the group are process addresses.

\item As defined in this proposal, groups contain no context or tag
information, nor is a facility for caching user defined information
provided. This implies that context and tag must be specified in addition
to group for collective communication functions.
This accords with the goal of MPI as specifying a lower
level implementation that can be used to create different kinds of higher level
implementations.

\end{enumerate}

This proposal is distinguished from proposal III by a much more
complete orthogonalization of the concepts of group and context,
as well as specifying group at a lower, more fundamental level than
proposal III. The advantages of this lower level definition are
primarily ease of implementation and understanding and 
additional flexibility in creating higher level functionality. In addition,
because groups use no context or tag resources, they are limited only
by available memory and CPU.

\disend

\section{Parallel task and process models.}

An MPI program or {\bf parallel task} consists of a set of {\bf
processes}, each executing independent code in an independent address
space. The model of a process is similar to that of UNIX.
A process can be single threaded or multithreaded, and
in a multithreaded environment, MPI makes no distinction between the
several threads of a process.
Messages can be sent and received between
processes, but there is no mechanism to specify the receiving
thread or obtain the sending thread of a message.

The set of processes in a parallel task is fixed. Each process is
addressed by its rank in the task, which is fixed and assigned in an
implementation defined way when the parallel task is created. The
assignment of processes to processors is also implementation defined, and
MPI defines a method (described below) to return a description of
the process assignment, or {\bf global topology}.

The process address of a process is an integer defined globally
within the parallel task, which can be sent in messages to any
other process and used with the same meaning on any other process.

In addition to the processes in a parallel task, an MPI implementation
may also allow communication with a {\bf host process}, if it exists.
The host process
is not considered part of the parallel task, but point to point
communications with it are allowed. The host process may send to or
receive messages from any process in the parallel task. The process
address of the host process is an integer defined globally within
the parallel task, but is not a valid rank in the parallel task.
A routine, \verb+MPI_gethost+, returns the host address if available,
or returns the constant \verb+MPI_NOHOST+ if there is no host process.
An MPI implementation may support a limited subset of point-to-point
communications with a host process.

The creation and termination of processes and parallel tasks is not
defined within MPI. 

\dishdr
This section does not differ greatly from other proposals.
This section would modify section 1.3 of the draft Point to Point communication
document, mostly in regard to the question of how processes are
addressed.

I have expanded on the host process stuff a little from the discussion
at the last meeting. The committee agreed (I think) to include the
possibility of a host process.

I feel it is unfortunate but necessary to postpone any discussion of
communication between parallel tasks to MPI 2.0. Some of the possible
properties that such communication might have are:

\begin{itemize}
\item A process may belong to at most one parallel task.
\item Process addressing must be extended to allow for the inclusion
 of a parallel task identifier.
\item Inter-task communication requires a session layer which intra-task
communication does not.
\end{itemize}

but all this remains open to debate.

% i want to think about this a little:
%
% Inter-task communication can be added to the above proposal as follows.
% Each parallel task has a system-wide unique identifier. The definition
% of a process address is widened to include both a task id and the process
% rank within the task. In other words process-address = (task-id, rank).
% The result must be encodable in an integer. No changes would be made
% to the syntax or semantics of point-to-point communications, except that
%
%

The concept of dynamic parallel tasks, tasks where the number of processes
belonging to the task changes over time, is also full of difficulties
and should also be postponed to MPI 2.0, along with any discussion of the
means by which parallel tasks are created or destroyed. Note that it would
be quite useful to define these things. 

\disend

\section{Contexts.}

The purpose of contexts is to promote software modularity by
allowing the construction of independent tag spaces. A context is an
integer-valued extension to the tag field which must match exactly between
sender and receiver of point-to-point communications. By specifying 
a unique context in addition to tag and other features of a message,
the communication between sender and receiver is made independent of
other communications which use different contexts. In the following,
context values are assigned or allocated by various software components,
but they are not pushed or popped during program execution. This more
static model avoids the state associated with such a stack mechanism.
Contexts as described here do not provide a mechanism
for a software component to protect contexts that it has allocated
or assigned from misuse. The only protection is that software components
which use different contexts are free to use tags within those contexts
with a guarantee of no interference.

A {\bf context} or {\bf context value} is an integer in a range
\verb+[0..MPI_MAXCONTEXT]+ which is part of the message envelope.
The sender and receiver of the message must specify exactly matching context
values; no wildcard filtering on the context part of a message envelope is
allowed. Filtering of messages based on sender address and/or tag
is discussed further in the section on point to point communications.

The number of available context values, \verb+MPI_NUMCONTEXTS+, is an
implementation-defined quantity. This proposal recommends a minimum
number of 256, although implementations allowing a greater or lesser
number of context values are allowed.

\dishdr

There has been considerable discussion on the network about the
number of available context values. One proposal (by Rik Littlefield) is
that MPI should go ahead and specify that context and tag are both
32 bit qtys and that the implementor should provide for the possibility of
contexts and tags anywhere in this space. I think these limits
are too generous. The other side (e.g. Peter
Rigsbee) is to not specify any hard limits and let the marketplace decide.
I don't quite agree with this either, since other standards (e.g. Fortran
77) {\it do} specify limits for some things.

Perhaps the committee could decide that there is a lower limit (say
256, my original estimate of 16 was seen by many as too small and I
have altered it (see above)) for compliance,
and a recommended number of $2^{16}$ or $2^{32}$.
 Then we could identify compliant
implementations (satisfies the lower limit) and strongly compliant
implementations (satisfies the recommended limit). I propose that
this point be left up for grabs and discussed further by the environmental
subcommittee.

The above statement specifies that there exists an implementation defined
limit on the size of the context space. But it does not specify that there
is a limit on the number of contexts in the space that can be used at any
one time. Should we specify the existence of such a limit? I don't think
so.

\disend

There exists a predefined and allocated context value,
\verb+MPI_DEFAULTCONTEXT+ which does not need to be allocated or opened
and cannot be freed or closed. This context value is intended for
initialization purposes and is free for use by all software components.

Context values may either be assigned statically or allocated dynamically,
and so we refer to them as static contexts and dynamic contexts. Since
context values are integers, it is not appropriate to speak of contexts
being created or destroyed. 

 Static
context values are chosen by the programmer and should lie in the
range \verb+[0, MPI_STATICCONTEXTLIMIT]+, where \verb+MPI_STATICCONTEXTLIMIT+
is an implementation-defined constant. Dynamic context values will be
allocated outside this range, within the limit \verb+MPI_CONTEXTLIMIT+
mentioned previously.

\dishdr
A problem could arise when a static context value is used which would normally
be allocated by the dynamic allocation routines. Here we only specify
an administrative solution, because that might be the intended program
behavior.  Generally, we adhere to the notion of allowing
any reasonable use of context values rather than specifing strong limitations.
\disend

Static context values must be opened before use and can be closed when
no longer needed. The routine
\begin{verbatim}
    void MPI_opencontext(int context)
\end{verbatim}
is a local operation that specifies that the context can be used in sending
 and  receiving messages.

The routine
\begin{verbatim}
    void MPI_closecontext(int context)
\end{verbatim}
is a local operation that specifies that the context is no longer needed
and no further messages will be sent or received using the context value.

Dynamic context values can be allocated with the routine
\begin{verbatim}
    int MPI_getcontext()
\end{verbatim}
which returns a context value and
which must be called synchronously by all processes in the parallel
task before messages can be sent or received with the context.

Dynamic context values can be freed with the routine
\begin{verbatim}
    void MPI_freecontext(int context-value)
\end{verbatim}
which is a local operation and which signals to MPI that the
given context value will no longer be used.

In addition, dynamic context values can be allocated within a group
with the routine
\begin{verbatim}
    int MPI_getgroupcontext(Group g)
\end{verbatim}
This routine must be called synchronously by all processes which are
members of the group. It will allocate a context value which has not
been previously allocated on any of the processes in the group, so the
context value is guaranteed to be unique within the group.
Note that if there are two groups which do not intersect then the
same context value may be allocated on each. Note also that it is
possible for a context value to be allocated in a group and sent to
a process not in the group which then opens the context and uses it for
communications. There is no guarantee of uniqueness on the second
process. Finally, there is no coupling of the context with the
group. It is not possible to get the group that was used to allocate
the context, nor is the context associated with the group in any way.

\dishdr
The getgroupcontext function is a collective communication operation. Here
it has been specified without reference to a predecessor context or a set
of tags, which means that it must be implemented using context and
tags internal to the implementation. The semantics needs to be refined
to handle the possibility of multithreading (programmer must mutex other
threads).

Both the getcontext and getgroupcontext functions give the user a lot of
flexibility and a lot of room to hang himself or herself.

\disend

\dishdr
The original version of this proposal only allowed for dynamic global
context values. After discussion, I have modified the proposal to
allow for statically and dynamically allocated contexts, as well as
contexts which are allocated only within a group.

Internally, MPI implementations may need to set up private data structures
or other resources (queues, buffers, etc.) in order to send and receive
messages with a particular context value. The \verb+MPI_getcontext()+ routine
is the appropriate entry point for such setup to occur. Some implementations
will simply set up resources for all possible context values at program
initiation.

% Deleted:
% Obviously, there are many possible schemes for context allocation and
%  deallocation. The scheme proposed here avoids the need and overhead 
% of a complex server. Another option is to avoid the allocation problem
% altogether, have only static allocation
% (as for Fortran unit numbers or TCP/IP ports), and have a routine
% \verb+MPI_opencontext(int context-value)+ which initializes use of the
% context.
% Fancier context allocation could be built on top of such a proposal.

It may be necessary for some MPI implementations to preallocate other
context values for internal use. Such context values would not be accessible
by user code.

\disend

\section{Groups.}

The purpose of groups is to provide tools for management of subsets
of processes in a parallel task. A group is defined abstractly as an
ordered set of unique integers (the elements); this abstract definition
allows groups to be constructed in many different ways and allows the
definition of a group as an explicit list (e.g. 7,4,13) or a computational
rule (e.g. processes with odd addresses). Groups as program objects may
be defined by MPI or may be developed as third party software.

A {\bf group} is a one-to-one mapping from the integers
$(0, 1, ..., N-1)$ to another set of integers, the {\bf elements} of the group.
The number of elements $N$ is the {\bf order} of the group. The {\bf range}
of the group is an upper limit on the size of any element. Note that
a group is not an ``array of integers'', although that is one possible
implementation. 

An ordered collection of processes is said to be a group if each element of
the group defines a process address, or rank in the parallel task. A process
belongs to a group if its address is an element of the group.

In the following, the opaque type {\tt MPIGroup} is intended to be thought of
as a pointer to one of many possible group structures. This {\bf group identifier}
is local to a process and may not be legitimately passed from one process
to another.

\dishdr
It would not be too difficult to define functions \verb+MPI_getgroupdsc()+
and \linebreak[4]
\verb+MPI_bldgroupfromdsc()+ which would compute an ASCII description of a 
group or build a group from its ASCII description. Such functions would
ease the problem of sending groups in messages.

\disend

Groups are defined only by construction in a process, so the
correct method of passing a group from one process to another is to pass
the information needed to construct it. Group construction and destruction
is therefore local to a process, and is independent of whether the process
belongs to the group or not.

There may be many classes of groups, some defined by MPI and some defined
as user extensions. Any group class must define the following methods:
\begin{verbatim}
    int MPI_grouporder(MPIGroup g)               -- compute order of a group.
    int MPI_grouprange(MPIGroup g)               -- compute range of a group.
    int MPI_groupelement(MPIGroup g, int rank)   -- compute element given rank.
    int MPI_grouprank(MPIGroup g, int element)   -- compute rank of element.
    int MPI_groupiselement(MPIGroup g, int element)  -- membership predicate.
    MPIGroup MPI_copygroup(MPIGroup g)           -- make a copy of a group.
    void MPI_freegroup(MPIGroup g)               -- group destructor.
\end{verbatim}

\dishdr The public definition of a group would contain a class member
defining all of these operations, so, for example, \verb+MPI_grouporder+ would
simply execute the function \verb+group->class->order+. In order for the group
concept to be user extensible, some common definition of the basic
group object and class object needs to be established. This definition is
properly done by the binding committee. Thus, although {\bf MPIGroup} is an opaque
type it is not entirely hidden -- a sophisticated user can get at it and
create new group types.

\disend

In addition, each class of groups must define a group constructor. The
constructor takes as parameters the information needed to construct the
group and returns the opaque group identifier.
>From the point of view of this proposal, group construction and use
are inexpensive operations, limited only by memory and CPU resources.

\dishdr For example, let us define a simple class of groups, which we
call {\bf linear} groups. For such a group, we define the mapping from
rank to element to be the function $(start + delta * rank)$, so the
parameters needed to create the group are the order and the constants
$start, delta$. The group constructor would then be defined as
\begin{verbatim}
      MPIGroup MPI_makelineargroup(int order, int start, int delta).
\end{verbatim}

Another simple class would be a {\bf list} group, for which the
construction parameters are the order of the group and a list of the
elements. The constructor is
\begin{verbatim}
      MPIGroup MPI_makelistgroup(int order, int *elements).
\end{verbatim}

Note that in the first case, there is probably no need for any
communication to create the group. In the second case, the list
might have to be distributed using communications, but the group
construction is purely local to each process.

\disend

\section{Using groups in point-to-point communications.}

Groups can be used to simplify point-to-point communication. A process
in a group can be defined either by its process address (equivalent to
the rank in the task) or by its rank in the group. A typical call to
send a message using a (group,rank) address might have the form
\begin{verbatim}
  MPI_send(buffer, length, MPI_groupelement(group, rank), tag, context)
\end{verbatim}

Note that in order for a process to send a message to a
process in another group using the second process' group rank, it must
construct a local copy of the group.

Since a group defines an order for its elements, it is easy to
compute the preceding and following processes of a particular process in
a group. The
preceding and following processes (elements) of the current process
are given by
\begin{verbatim}
   my_rank = MPI_grouprank(group, my_pid)  
   pid_precede = MPI_groupelement(group, my_rank-1)
   pid_follow = MPI_groupelement(group, my_rank+1).
\end{verbatim}
modulo boundary conditions.

\dishdr Two and three dimensional generalizations are straightforward but
require additional functions specific to particular types of groups
(e.g. Cartesian groups).

\disend

\section{Groups and collective communication.}

Groups are the primary method of organizing collective communication
operations. All collective communication operations have the form
\begin{verbatim}
   void MPI_xxx(group, context, ...)
\end{verbatim}
Such functions must be called by all of the processes in the group
in order to complete successfully.

\dishdr 

Previously I proposed that collective communication routines be passed
both a context and one or more tags. But a much cleaner solution is to
pass the routine just a context under the assumption that the routine
may make any use of tags within the context that it likes.

Groups may contain private (internal to MPI) information about
methods and data to accelerate common collective communication operations.
For example, a group implementation may include a private copy of a
spanning tree to be used for fast broadcasting. Such information is
implementation dependent and not normally accessible to a user. The
definition of a group should contain a hook for the collective comm routines
to hang such data.

\disend

\section{Topology.}

There are several meanings for the term topology as used in MPI. The first
meaning is associated with the global topology or assignment of processes
to processors in a particular network. Such a topology might be classified
as a hypercube, mesh, or general network topology. Information about the
global topology is returned by the routine
\begin{verbatim}
  char *MPI_globaltopology()
\end{verbatim}
which returns a string containing a description of the global topology. The
form of this string is generally {\it topology-class parameter ...}.

\dishdr For example, various implementations might return strings like
\begin{verbatim}
       N 564         -- a random (Ethernet) network with 564 processes.
       H 5           -- a 5 dimensional hypercube.
       R 2 16 13     -- a two dimensional mesh, 16 by 13.
\end{verbatim}

\disend

The second type of topology is that internal to a group.
Since, in this proposal, groups are ordered sets and therefore have
an implicitly defined  (one dimensional) topology, there is no need
for independent topology routines.
A different ordering can be selected by building a different group. For
example, to select a Gray ordering instead of a natural ordering (useful
on a hypercube for some applications), one would build a permutation group
implementing the Gray ordering. Alternatively, one could impose a Gray ordering
on an existing group by a group composition operation.

Third, we define a {\bf virtual topology} as a method of referring to
elements of a group other than by straightforward rank. For example,
we might want to think of a group as a rectangle or higher order
Cartesian object, and reference elements by x-rank, y-rank, etc. Another
example might be a tree virtual topology, where we would refer to
elements in terms of parent, sibling and child. Virtual topologies
can be implemented by layering on top of MPI.

\section{Contexts, groups, and third party software.}

>From the point of view of this proposal, there are two kinds of third party
or library software, i.e. software written to use MPI but intended for use
in turn by other
applications. The first type is similar to the MPI collective communication
operations as described above. This type uses the contexts and tag space
provided by whatever software calls it. The second type, typically associated
with much more complex functions, establishes and manages its own context
and tag spaces. MPI must be careful to support both possibilities.

Library software of the first kind, including MPI collective communications,
might benefit from the specification of some sort of ``tag manager'' so that
only context would need to be provided. This proposal does not attempt to
answer this question.

\section{Cache facility.}

This proposal does not define a cache facility for adding information to
groups or contexts. Such a facility, although it has attractive features,
is more appropriately layered on top of MPI.

\section{Modifications to other documents.}

The discussions in the current point-to-point document (sections 1.3 and\
1.4 of the May 9 draft would be deleted).

All of the collective communication routines would be modified to
conform to the guideline that a collective communication routine is
passed a group and context value separately.

\section{Subsets of MPI.}

Because this proposal maintains a strong orthogonality between the
context and group concepts, it is easy to eliminate all collective
communication operations (indeed all group operations) from MPI in
order to generate a useful subset. The mechanisms for managing context
are simple and do not require a server concept.

\end{document}

>From [email-masked] Wed Jun  9 15:02:26 1993
Date: Wed, 9 Jun 93 08:02:23 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: May 12 mpi mail #5
Status: R

Another LaTeX source with initial verbosity deleted ...

\documentstyle[twoside]{report}
\pagestyle{headings}
%\markright{ {\em Draft Document of the MPI Standard,\/ \today} }
\marginparwidth 0pt
\oddsidemargin=.25in
\evensidemargin  .25in
\marginparsep 0pt
\topmargin=-.5in
\textwidth=6.0in
\textheight=9.0in
\parindent=2em

%   ----------------------------------------------------------------------
%   mpi-macs.tex  --- man page macros,
%                discuss, missing, mpifunc macros
%
% ----------------------------------------------------------------------
% a couple of commands from Marc Snir, modified S. Otto

\newlength{\discussSpace}
\setlength{\discussSpace}{.7cm}

\newcommand{\discuss}[1]{\vspace{\discussSpace}
{\small {\bf Discussion:} #1} \vspace{\discussSpace}
}

\newcommand{\missing}[1]{\vspace{\discussSpace}
{\small {\bf Missing:} #1} \vspace{\discussSpace}
}

\newcommand{\implement}[1]{\vspace{\discussSpace}
{\small {\bf Implementation note:} #1} \vspace{\discussSpace} }

\newlength{\codeSpace}
\setlength{\codeSpace}{.3cm}

\newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} }

% -----------------------------------------------------------------------
%  A few commands to help in writing MPI man pages
%
\def\twoc#1#2{
\begin{list}
{\hbox to95pt{#1\hfil}}
{\setlength{\leftmargin}{120pt}
 \setlength{\labelwidth}{95pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#2}
\end{list}
}
\outer\long\def\onec#1{
\begin{list}
{}
{\setlength{\leftmargin}{25pt}
 \setlength{\labelwidth}{0pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#1}
\end{list}
}
\def\manhead#1{\noindent{\bf{#1}}}

\hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple}

\begin{document}

\setcounter{page}{1}
\pagenumbering{roman}

\title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface}

\author{Scott Berryman, {\em Yale Univ} \\
James Cownie, {\em Meiko Ltd} \\
Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\
Al Geist, {\em ORNL} \\
Bill Gropp, {\em ANL} \\
Rolf Hempel, {\em GMD} \\
Bob Knighten, {\em Intel} \\
Rusty Lusk, {\em ANL} \\
Steve Otto, {\em Oregon Graduate Inst} \\
Tony Skjellum, {\em Missisippi State Univ} \\
Marc Snir, {\em IBM T. J. Watson} \\
David Walker, {\em ORNL} \\
Steve Zenith, {\em Kuck \& Associates}    }

\date{May 9, 1993 \\
This work was supported by ARPA and NSF under contract number \#\#\#,
by the National Science Foundation Science and
Technology Center Cooperative Agreement No. CCR-8809615.
}

\maketitle
\hfuzz=5pt
%\tableofcontents

%\begin{abstract}
%We don't have an abstract yet.
%\end{abstract}

\setcounter{page}{1}
\pagenumbering{arabic}

\chapter{Point to Point Communication}
\label{sec:pt2pt}

\begin{center}
Marc Snir \\ William Gropp and Ewing Lusk
\end{center}

\section{Introduction}

This section is a draft of the current proposal for point-to-point
communication.  It does not yet include a description of the Fortran 77
and C bindings.

I have tried to indicate, wherever appropriate, gaps and unresolved issues,
using small type.

\discuss{
The following sections of the introduction contain general information
on the design of MPI procedures.   The material should be moved to a general
introduction for the entire document.
The actual binding of these objects in Fortran and C will be discussed in
the language binding subcommittee.

}

\section{Data Types}

\subsection{Handle}

MPI procedures use at various places {\em handles}.  Handles
are used to access opaque objects.  Such object
can be created, updated and destroyed only by by calling suitable MPI
procedures, and providing the handle as parameter. Opaque
objects hide from the user the internal representation used for
various
MPI objects, thus allowing to have similar calls in C and Fortran, allowing to
overcome problems with the typing rules in these languages, and allowing for
future extension of their functionality.
The mechanism for opaque objects used here loosely follows the POSIX Fortran
binding standard.

An opaque object can be {\em persistent} or {\em ephemeral}.  A persistent
object persists until destroyed by an explicit operation.  An ephemeral object
is good for a single use; thus an ephemeral object associated with a
communication operation disappears once this operation is completed (or once
this object is not needed anymore for the completion of the operation).

Opaque object are created by calls that are specific to each object type.  They
are destroyed by a call to {\tt MPI\_FREE}.  Additional MPI functions are
available to access and update specific opaque objects.

\mpifunc{MPI\_FREE(handle)}
\begin{description}
\item[IN handle] handle to object
\end{description}

Mark the handle for destruction.  The object can destroyed as soon as there
there is no pending operation that is using this object.   This is equivalent to
changing the persistence of the handle from {\tt persistent} to {\tt ephemeral}.
For example, if a communication object is freed, after the communication started
but before it completed, then the pending operation is not affected; one can use
the handle in calls to {\tt MPI\_WAIT} or {\tt MPI\_STATUS}.  However, once the
operation completes, it is erroneous to reuse the freed handle for a new
communication.

\discuss{

An alternative design point was discussed, whereby it is the user responsibility
to free a handle only if there are no pending operations on this handle.
The current design was deemed more elegant.  Note
that it is not necessary to destroy the handle as soon as it is not needed
anymore.  A valid implementation may just mark freed handles and garbage collect
them periodically.

On the other hand, some objects, such as buffer descriptor objects,
can be used by several distinct concurrent
communication operations.  Thus, to find out whether such object can
be freed, one would need a reference count of pending operations.  This
complication was not observed at our last meeting, and may be a
good argument to reconsider the current definition of {\tt MPI\_FREE}.

We do not have yet a mechanism to control where from storage is allocated for
opaque objects.   Some believe it is desirable to allow the user to control
this allocation.

I now have a generic {\tt MPI\_FREE} function for all kinds of opaque objects.
Any reason to have specialized functions for each kind of object, as for {\tt
CREATE}?

}

\mpifunc{MPI\_ASSOCIATED(handle, type)} \begin{description}
\item[IN handle] handle to object
\item[OUT type] state
\end{description}

Returns the type of the object the handle is currently associated with, if
such exists.  Returns the special type {\tt MPI\_NULL} if the handle is
not currently associated with any object.

MPI may provide predefined opaque objects and predefined, static handles to
these objects.  Such objects may not be destroyed.

\subsection{Array of handles}
An MPI call may need a parameter that is an {\em array of handles}.
The array-of-handles object is not self-delimiting; it does not carry
information on the number of entries in the array.  This information has to be
provided by an additional {\tt len} parameter, whenever a list-of-handles
parameter is used.

\discuss{

The reason for not having self-delimiting arrays is that records are not
first-class citizens in Fortran and that having an array where one entry encodes
length, whereas the other entries encode handles was deemed to be too
error-prone.   This line of argumentation really implies that we do not view
array-of-handles as an opaque object, and that we expect the user to build these
objects explicitly, not via MPI calls.   The alternative is for array-of-handles
to be an opaque object, manipulated via s set of MPI calls.

}

\subsection{State}

MPI procedures use at various places arguments with {\em state} types.  The
values of such data type are all identified by names, and no operation is
defined on them. For example, the {\tt MPI\_APPEND} routine has a
state type parameter with values {\tt MPI\_INT, MPI\_REAL,} etc.

\subsection{Named constants}

MPI procedures sometimes assign a special meaning to a special value of a
basic type parameter; e.g. {\tt tag} is an integer valued parameter of
point-to-point communication operations, with a special {\tt DONTCARE} value.
Such parameters will have a range of regular values, which is a proper
subrange
of the range of values of the corresponding basic type; special values (such as
DONTCARE) will be outside the regular range.   The range of regular values can
be queried, and sometimes set, using environment inquiry or environment setting
functions (Section~\ref{sec:inquiry}).

\discuss{

Need to agree on a language mechanism for named constants.

Implementers should detect, whenever possible,
illegal uses of ``special values''.  Thus,
the use of the {\tt DONTCARE} value to tag a message sent will be flagged as
an error.

}

\subsection{Choice}

MPI functions sometimes use parameters with a {\em choice} (or union) data
type.  I.e., distinct calls to the same routine may pass by reference actual
parameters of different types. The mechanism for providing such
parameters will differ from language to language.

\discuss{

The Fortran 77 standard specifies that the type of actual arguments need to
agree with the type of dummy arguments; no construct equivalent to C pointers is
available.  Thus, it would seem that there is no standard conforming mechanism
to support choice parameters.
However, most Fortran compiler either don't check type
consistency of calls to external routines, or support a special mechanism to
link foreign (e.g., C) routines.  I suggest that we accept this nonconformity
with Fortran 77 standard.  I.e., we accept that the same routine may be passed
an actual parameter of a different type at distinct calls.

Generic routines can be used in Fortran 90 to provide a standard
conforming solution.  This solution will be consistent with our nonstandard
conforming Fortran 77 solution.

}

\section{Processes}

\discuss{This material belongs to an introduction or environment section}

An MPI program is executed by several autonomous processes that each execute
their own code, in a MIMD style. The codes executed by each process need not be
identical.  The processes communicate via calls to MPI communication primitives.
Typically, each processor executes in its own address space, although
shared-memory implementations of MPI are possible.
This document specifies the behavior of a parallel program assuming that only
MPI calls are used for communication.  The interaction of an MPI program with
other possible means of communication (e.g., shared memory) is not specified.
In particular, it is assumed that message buffers at distinct
processors are disjoint.

MPI does not specify the execution model for each process.  A process can be
sequential, or can be multithreaded, with threads possibly executing
concurrently.  Care has been taken to make MPI ``thread-safe'', by avoiding the
use of implicit global states.

The initial allocation of processes to an MPI computation and their
binding to physical processors is not specified by the
program itself.  It is expected that vendors will provide mechanisms to do so
either at load time or at run time.   Such mechanisms will allow the
specification of the initial number of required processes, the
code to be executed by each initial process, and the allocation of processes to
processors.  Also, the current proposal does not provide for dynamic creation or
deletion of processes during program execution (the total number of processes
is fixed),  although it is intended to be
consistent with such extension.   Finally, the current proposal does not specify
a naming scheme for processes.  We propose to always identify processes
according to their relative rank in a context (group), so that, effectively,
processes are identified by consecutive integers.

\subsection{Error Handling}

\discuss{ Material of this section should be moved to a general introduction
section.  This material has not yet been approved by MPIF}

MPI provides the user with reliable message transmission:
A message sent is always received
correctly, and the user does not need to check for transmission errors,
time-outs, or other error conditions.  In
other words, MPI does not provide mechanisms for
dealing with failures in the
communication system.
Where the MPI implementation is built on an unreliable underlying
mechanism, then it is the job of the implementer of the MPI subsystem
to insulate the user from this unreliability, or to reflect unrecoverable
errors as global
program failures.  Similarly MPI itself provides no mechanisms for
handling node failures.

Of course, MPI programs may still be erroneous.  A {\bf program error} can
occur when an MPI call is called with an incorrect parameter (non-existing
destination in a send operation, buffer too small in a receive operation, etc.)
This type of error would occur in any implementation.
In addition, a {\bf resource error} may occur when a program exceeds the amount
of available system resources (number of pending messages, system buffers,
etc.).   The occurrence of this type of error depends on the amount of
available resources in the system and the resource allocation mechanism used;
this may differ from system to system.   The recommended implementation profile
provides several mechanisms to alleviate the portability problem this
represents.  One can also write {\bf safe} programs, that are not subject to
resource errors.

All MPI procedure calls return an error parameter that indicates successful
completion of the operation, or the error condition that occurred, otherwise.

The recommended implementation profile in a POSIX environment is for any MPI
routine that encounters a recoverable error to generate an {\em MPI
error signal}, using a special signal value.  The default handler for
this signal terminates the execution of all involved processes, with a suitable
error message being returned to the user.  However, the user can provide his or
her own signal handling routine.  In particular, the user can
specify a ``noop'' signal handler, thus relegating all error handling to the
user code, using the error parameters returned by the MPI calls.

MPI calls may initiate operations that continue
asynchronously after the call returned.  Thus, the operation may return with a
code indicating successful completion, yet later cause an error exception to be
raised.  If there is a subsequent call that relates to the same operation (e.g.,
a call that verifies that an asynchronous operation has completed) then the
error parameter associated with this call will be used to indicate the nature
of the error.  In a few cases, the error may occur after all calls that
relate to the operation have completed, so that no error parameter can be used
to indicate the nature of the error (e.g., an error in
a send with the ready mode).  In such cases, an error will be undetected, if
the user disabled the MPI error signal.

\discuss{
The alternative choice is to have fatal and non-fatal signals.

One might want different signals for different modules.

The details of such proposal need be elaborated in an appropriate
``profile'' subcommittee.
}

\section{Messages}

A message consists of an {\em envelope} and {\em data}.

\subsection{Data}

The data part of a message consists of a sequence of values, each of a basic
datatype in the host language.  Thus, in Fortran 77, a message consists of
a sequence of values that are each of type {\tt INTEGER}, {\tt REAL}, {\tt
DOUBLE PRECISION}, {\tt COMPLEX}, {\tt DOUBLE PRECISION COMPLEX}, {\tt LOGICAL},
or (length 1) {\tt CHARACTER}.
A message may also contain a value of type {\tt BYTE}, which consists of 8
binary digits.   A byte is different from a character.
Different machines may
have different representations for the same character, or may use more than one byte
to represent a character.  On the other hand, a byte has the same binary value
on any machine.
A message may mix values of different types.

\missing{

Need to agree on the C types (including handling of signed/unsigned),
and on Fortran 90 types (including handling of kinds).

}

\subsection{Envelope}
\label{subsec:envelope}

The following information is associated with each message:
\begin{description}
\item[source] The rank of the sending process
\item[destination] The rank of the receiving process
\item[tag] User defined
\item[context]  handle
\end{description}

The range of valid values for the {\bf
source} and {\bf destination} fields is {\tt 0 ... n-1}, where
{\tt n} is the number of processes in the specified context.
{\tt source = destination} is allowed:  a process can send a message
to itself.

The ranges
of valid values for {\tt tag} is implementation dependent, and
can be found by calling a suitable query function, as described in
Section~\ref{sec:inquiry}.

The {\tt tag} field can be arbitrarily set by the application, and can be used
to distinguish different messages.

{\tt Context} should be a context shared by both
source and destination.

The actual mechanism used to associate an envelope with a message
is implementation dependent; some of the information (e.g., {\bf
sender} or {\bf receiver}) may be implicit,  and need not be
explicitly carried by a message.

\section{Communication Buffers}
\label{subsec:buffers}

The basic point to point communication operations are {\bf send} and {\bf
receive}. A {\bf send} operation creates a message; the message data is
taken from the {\bf send buffer}.  A {\bf receive} operation consumes a
message; the message data is put into the {\bf receive buffer}.  The
specification of send or receive buffers uses the same syntax.

\subsection{Communication buffer components}

A buffer consists of a sequence of {\bf buffer components}.  Each
component consists of a sequence of (not necessarily distinct)
variables of the same basic datatype.  The datatype of such a variable is
specified by a state parameter.  The possible values for Fortran are
{\tt MPI\_INT}, {\tt MPI\_REAL}, {\tt MPI\_DOUBLE},
{\tt MPI\_COMPLEX}, {\tt MPI\_DCOMPLEX}, {\tt MPI\_LOGICAL},
{\tt MPI\_CHAR} and {\tt MPI\_BYTE}, corresponding to datatypes
{\tt INTEGER}, {\tt REAL}, {\tt
DOUBLE PRECISION}, {\tt COMPLEX}, {\tt DOUBLE PRECISION COMPLEX}, {\tt LOGICAL},
{\tt CHARACTER} and to untyped {\tt BYTE}.  The possible values for C are
{\tt MPI\_CHAR}, {\tt MPI\_SHORT}, {\tt MPI\_INT}, {\tt MPI\_LONG}, {\tt
MPI\_FLOAT}, {\tt MPI\_DOUBLE} and {\tt MPI\_BYTE}.

\discuss{

May need to take care of kinds in Fortran 90 and of signed/unsigned in C.  More
elaborate definitions belong to the language binding.

}

There are five kinds of buffer components:

\subsubsection{Contiguous component}

A sequence of contiguous values of the same basic type,
specified by
\begin{description}
\item[start]
Initial element
\item[count]
Number of elements  ($\tt count \ge 0$)
\item[datatype]
Type of elements
\end{description}

Thus, if {\tt A} is an array of integers, then {\tt A(1), 4, MPI\_INT} is
the buffer component with entries {\tt A(1), A(2), A(3), A(4)}.

\subsubsection{Vector component}

A sequence of equally spaced and equally sized blocks of elements of the same
basic type, specified by
\begin{description}
\item[start]
Initial element
\item[count]
Number of elements ($\tt count \ge 0$)
\item[stride]
Number of elements between the start of each block
\item[lenblk]
Number of elements in each block  ($\tt lenblk > 0$)
\item[datatype]
Type of elements
\end{description}

Assume that an element of type {\tt datatype} occupies {\tt m} bytes.  Then, in
a byte addressable machine, the
buffer component described by these five parameters consists of the {\tt
count} variables at addresses
{\tt start, start+m, ... , start+m*(lenblk-1),
start+m*stride,} \\
{\tt start+m*(stride+1), ..., start+m*(stride+lenblk-1), ...}.
For example, if {\tt A} is an array of integers, then
\begin{itemize}
\item
{\tt A(1), 5, 3, 2, MPI\_INT} is the buffer component with entries
{\tt A(1), A(2), A(4), A(5), A(7)}.
\item
{\tt A(1), 5, 0, 2, MPI\_INT} is the buffer component with entries
{\tt A(1), A(2), A(1), A(2), A(1)}.
\item
{\tt A(10), 5, -1, 2, MPI\_INT} is the buffer component with entries
{\tt A(10), A(11), A(9), A(10), A(8)}.
\end{itemize}

{\tt Contiguous} is a special case of {\tt vector}.
A vector buffer component can be used to extract an arbitrary
submatrix of a two-dimensional matrix.

\subsubsection{Indexed component}

A sequence of elements of the same basic type, specified by
\begin{description}
\item[start]
initial element
\item[count]
Number of elements  ($\tt count \ge 0$)
\item[array\_of\_indices]
Array of displacements of the elements in the buffer components, relative to
the initial element.
\item[datatype]
Type of elements
\end{description}

For example, if {\tt A} is an array of integers, and {\tt I} is an array with
values {\tt  2, -1, 2, 1} then
{\tt A(3), 4, I, MPI\_INT} is the buffer component with entries
{\tt A(5), A(2), A(5), A(4)}.

{\tt Vector} can be thought of as a special case of {\tt indexed} in which the
indices are generated with a constant stride.

\subsubsection{Heterogeneous vector (hvector) component}

Same as {\tt vector}, except that {\tt stride} is given in multiple of bytes,
rather than multiples of elements:

\begin{description}
\item[start]
Initial element
\item[count]
Number of elements ($\tt count \ge 0$)
\item[stride]
Number of bytes between the start of each block
\item[lenblk]
Number of elements in each block  ($\tt lenblk > 0$)
\item[datatype]
Type of elements
\end{description}

Assume that an element of type {\tt datatype} occupies {\tt m} bytes.  Then, in
a byte addressable machine, the
buffer component described by these five parameters consists of the {\tt
count} variables at addresses {\tt start, start+m, ... , start+m*(lenblk-1),
start+stride, start+stride+m, ..., start+stride+m*(lenblk-1), ...}.

Consider a C array declared as

\begin{verbatim}
struct {
   char info[3];
   short number;
   double val;
       }   a[100]
\end{verbatim}

Assume that all structure components are byte aligned, a {\tt short}
occupies two bytes, and a {\tt double} occupies eight bytes.
Then the buffer component {\tt \&a, 100, 13, 2, MPI\_CHAR} extracts the first
two characters of each structure.  The buffer component
{\tt \&(a.number), 100, 13, 1, MPI\_SHORT} extracts the {\tt short} field
of each structure; and the buffer component {\tt \&(a.val), 100, 13, 1}
extracts the {\tt double} field of each structure.
Thus, heterogeneous vectors can be used to extract vectors of homogeneous values
from heterogeneous data.

{\tt Vector} is a special case of {\tt heterogeneous vector}.

\subsubsection{Heterogeneous indexed (hindexed) component}

Same as indexed, except that displacements are in bytes, rather than multiples
of element size.

\begin{description}
\item[start]
initial element
\item[count]
Number of elements  ($\tt count \ge 0$)
\item[array\_of\_indices]
Array of byte displacements of the elements in the buffer components, relative
to the initial element.
\item[datatype]
Type of elements
\end{description}

For example, if {\tt a} is the C array of the previous example, and
{\tt I} is an array with values {\tt  0, 13, 13, 27, 26} then
{\tt \&a, 5, I, MPI\_CHAR} is the buffer component with entries
{\tt a[0].info[0], a[1].info[0], a[1].info[0], a[1].info[1], a[1].info[0]}.

Like {\tt heterogeneous vector}, {\tt heterogeneous indexed} extends the {\tt
indexed} facility so as to allow the extraction of homogeneous data from a
heterogeneous layout.

\discuss{

The use of displacements in indexed buffer components has the same effect as
indirect addressing into an array with initial index zero.   I.e., if {\tt a} is
an array with initial index zero, then a buffer component defined by
parameters  {\tt a[0], N, i, type} extract the elements {\tt a[i[0]], a[i[1]],
..., a[i[n]]}. This fits well C usage but ill-fits Fortran usage (initial index
one).
We took a decision of principle to force ``start from zero'' index
arithmetic on Fortran users, rather than have different index arithmetic in the C
and Fortran binding of MPI.

The use of both buffer components with byte and element size displacements is
motivated by the observation that some applications require the more general
byte displacement, but that most applications would use element size
displacement, which is much more natural and convenient (and machine
independent).  Thus, rather than forcing all users to do byte arithmetics, we
decided to provide separately the more advanced function to ``expert
programmers''.

Leslie Hart proposes to replace ``heterogeneous'' with ``general''.

}

\subsection{Buffer operations}

A buffer is described by an opaque {\bf buffer descriptor}
object accessed via a {\bf buffer handle}.
Such an object is created by {\tt MPI\_CREATE\_BUFFER}. It is
associated with successive buffer components by calling in succession one of the
functions {\tt MPI\_APPEND\_xxx}, where {\tt xxx} identifies the type of
buffer component appended to the buffer.
It becomes available for use in communication operations once committed by the
function {\tt MPI\_COMMIT\_BUFFER}.   Once committed, the descriptor cannot be
modified any more.

Note that the commit operation does not create yet a message, and
does not bind the data to be sent.  It is a commit for the buffer descriptor,
not a commit for the buffer content.
A buffer descriptor is destroyed after the completion of the first communication
operation that uses it, if it is ephemeral, or after it is freed by a call to
{\tt MPI\_FREE} and any pending communication operation that uses it has
completed, if it is persistent.   After the buffer descriptor object has been
destroyed, the handle is undefined.

\mpifunc{MPI\_CREATE\_BUFFER( buffer, persistence )}

Create a new buffer descriptor object.
The parameters are:
\begin{description}
\item[OUT buffer]
buffer handle
\item[IN persistence]
buffer persistence (state type with possible values {\tt MPI\_PERSISTENT} or
{\tt MPI\_EPHEMERAL}).
\end{description}

\discuss{

May want to provide information on maximum object size, and perhaps user space
for the object.

Leslie Hart proposes {\tt MPI\_CREATE\_BUFFER\_DESC}, rather than {\tt
MPI\_CREATE\_BUFFER}.  More accurate, but longer.   Perhaps {\tt
MPI\_CREATE\_BUFDESC}?

}

\mpifunc{MPI\_APPEND\_CONTIGUOUS( buffer, start, count, datatype)}

Append a block component to buffer.
The parameters are:
\begin{description}
\item[INOUT buffer]
buffer handle
\item[IN start]
buffer component initial element (choice)
\item[IN count]
Number of elements (integer)
\item[IN datatype]
datatype identifier (status)
\end{description}

\mpifunc{MPI\_APPEND\_VEC( buffer, count, stride, lenblk, datatype )}

Append a vector buffer component to buffer.
The parameters are:
\begin{description}
\item[INOUT buffer]
buffer handle
\item[IN start]
buffer component initial element (choice)
\item[IN count]
Number of elements (integer)
\item[IN stride]
Number of elements between the start of each block (integer)
\item[IN lenblk]
Number of elements in each block (integer)
\item[IN datatype]
datatype identifier (status)
\end{description}

\mpifunc{MPI\_APPEND\_INDEXED( buffer, start, count, array\_of\_indices, datatype)}

Append an indexed buffer component to buffer.
The parameters are:
\begin{description}
\item[INOUT buffer]
buffer handle
\item[IN start]
initial position for indexing (choice)
\item[IN count]
Number of elements (integer)
\item[IN array\_of\_indices]
array of displacements of entries relative to start (array of integers)
\item[IN datatype]
datatype identifier (status)
\end{description}

The index values used to create this component are the values provided at
the call; subsequent updates of the array of indices will not affect the buffer
descriptor component.

\mpifunc{MPI\_APPEND\_HVEC( buffer, start, count, stride, lenblk, datatype )}

Append a heterogeneous vector buffer component to buffer.
The parameters are:
\begin{description}
\item[INOUT buffer]
buffer handle
\item[IN start]
buffer component initial element (choice)
\item[IN count]
Number of elements (integer)
\item[IN stride]
Number of bytes between the start of each block (integer)
\item[IN lenblk]
Number of elements in each block (integer)
\item[IN datatype]
datatype identifier (status)
\end{description}

\mpifunc{MPI\_APPEND\_HINDEXED( buffer, start, count, array\_of\_indices,
datatype)}

Append an heterogeneous indexed buffer component to buffer.
The parameters are:
\begin{description}
\item[INOUT buffer]
buffer handle
\item[IN start]
initial position for indexing (choice)
\item[IN count]
Number of elements (integer)
\item[IN array\_of\_indices]
array of byte displacements of entries relative to start (array of integers)
\item[IN datatype]
datatype identifier (status)
\end{description}

\mpifunc{MPI\_COMMIT\_BUFFER( bufferin )}

Commit a buffer descriptor object.  The object cannot be changed anymore after
it is committed.   A buffer descriptor object can be used for communication only
after it is committed.
The parameters are:
\begin{description}
\item[INOUT buffer]
buffer descriptor handle
\end{description}

\discuss{

The commit function may do preprocessing of the buffer descriptor to simplify
its use and commit resources in the communication subsystem for its
use.  The commit function may generate a new object and, hence, modify
the object handle.

The current proposal does not allow to create a buffer descriptor, use it in a
communication after committed, then modify it partially and use it for another
communication -- a new buffer descriptor has to be created.  Of course, the same
buffer descriptor can be reused many times in successive communications that
send data from the same buffers; e.g., a new send with a buffer descriptor
handle can be initiated once the previous send with this handle has completed.

}

Example 1: \\
Consider the following fragment of Fortran code

\begin{verbatim}
DOUBLE PRECISION A(10,20)
INTEGER B, C(5,10)
INTEGER BH
...
CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT)
CALL MPI_APPEND_BLOCK (BH, B, 1, MPI_INT)
CALL MPI_APPEND_VEC (BH, A(1,3), 11, 4, 2, MPI_DOUBLE)
CALL MPI_APPEND_INDEX(BH, C(3,7), (4,2,1), MPI_INT)
\end{verbatim}

Then the buffer associated with the handle {\tt BH} consists of the sequence of
variables

{\tt B, A(1,3), A(2,3), A(5,3), A(6,3), A(9,3), A(10,3),  A(3,4), A(4,4),
A(7,4), A(8,4), A(1,5), C(2,8), C(5,7), C(4,7)}.

A message created from this buffer will consist of a sequence of one integer,
followed by eleven double precision reals, followed by three integers.

Example 2: \\
Suppose one has an array of data declared in C as

\begin{verbatim}
struct {
   int type;
   double position[3];
   double velocity[3];
       } particle[1000]
\end{verbatim}

and one wants to transfer in one message the number of particles of type 1, the
number of particles of type 2, and the positions of all the particles of these
two types.  The following code fragment creates a suitable buffer descriptor.

\begin{verbatim}
int count1 = 0;
int count2 = 0;
...
mpi_create_buffer( &handle, MPI_PERSISTENT);
mpi_append_contiguous( &handle, &count1, MPI_INT);
mpi_append_contiguous( &handle, &count2, MPI_INT);
for (i=0; i<1000; i++) {
   if (particle[i].type == 1) {
      count1++;
      mpi_append_contiguous( &handle, &particle[i].position, 3, MPI_DOUBLE);
      }
   elseif (particle[i].type == 2) {
      count2++;
      mpi_append_contiguous( &handle, &particle[i].position, 3, MPI_DOUBLE);
      }
   }
mpi_commit_buffer(handle)
...
\end{verbatim}

The buffer descriptor handle can now be used to send (repeatedly) the particle
counts for type 1 and 2, and the positions of the particles with these two
types, as long as the types of the particles is not modified.

Suppose the incoming data is to be stored at the receiver into the two
count variables {\tt count1} and {\tt count2}
and into the {\tt position} components of the initial entries of the array of
particles.   The following code generates a suitable buffer descriptor for the
receive operation (we assume 4 bytes for integer and 8 bytes for double).

\begin{verbatim}
...
mpi_create_buffer( &handle, MPI_PERSISTENT);
mpi_append_contiguous( &handle, &count1, MPI_INT);
mpi_append_contiguous( &handle, &count2, MPI_INT);
mpi_append_hvec( &handle, &particle[0].position, 1000, 52, 3, MPI_DOUBLE);
mpi_commit_buffer( &handle)
...
\end{verbatim}

A buffer handle can be used for communication, even if it is not associated
with any  variables (i.e., even if it was not set by any {\tt MPI\_APPEND\_xxx}
call).
Such a handle is associated with an empty buffer, and a message created from it
contains no data.   The handle still need be committed before it is used.

The same entry may appear more than once in a buffer component, and distinct
buffer components may overlap.  If a buffer descriptor with replicated
entries is used to receive a message, then the final value of an entry
that occurs more than once in the buffer descriptor is undefined.

\discuss{

It is not clear to me that overlaps have any useful purpose, and they may
complicate implementation.  But we had an overwhelming vote in favor of this
feature.

}

\subsection{Type Matching and Data Conversion}

\subsubsection{Type matching}

One can think of message transmission as consisting of three phases:
\begin{enumerate}
\item
Data is pulled out of the send buffer and a message is assembled
\item
A message is transferred from sender to receiver
\item
Data is pulled from the incoming message and disassembled into the receive
buffer
\end{enumerate}

Type matching has to be observed at each of these three phases:  The type
of each variable in the sender buffer has to match
the type of the corresponding entry in the buffer descriptor; the type of an
entry in the buffer
descriptor used to send a message has to match the type of the corresponding
entry in the buffer descriptor used to receive the message; and the type of each
variable in the receive buffer has to match the corresponding entry in the
buffer descriptor for this buffer.   A program that fails to observe these
three rules is erroneous.

To define type matching more precisely, we need to deal with two issues:
matching of types of the host language with buffer descriptor types; and
matching of buffer descriptor types at sender and receiver.
We restrict ourselves to the case where the sending and receiving programs are
written in the same language.

The type of an entry in the sender buffer descriptor matches the type of an
entry in the receiver buffer descriptor if they have identical names:  {\tt
MPI\_INT} matches {\tt MPI\_INT}, {\tt MPI\_REAL} matches {\tt MPI\_REAL}, and
so on.

The type of a variable in a host program matches the type of the entry in a
buffer descriptor if the name of the buffer descriptor entry type corresponds to
the basic type of the host program variable:   an entry with type name {\tt
MPI\_INT} matches a Fortran variable of type {\tt INTEGER}, an entry with
type name  {\tt MPI\_REAL} matches a Fortran variable of type {\tt REAL}, and so
on.  There is one exception to this last rule:  An entry with type name {\tt
MPI\_BYTE} can be used to match
any byte of storage (on a byte-addressable machine),
irrespective of the datatype of the variable that contains this byte.  The value
of the message entry will be the binary value of the corresponding byte in
memory.

We thus have two cases:
\begin{itemize}
\item
Communication of typed values (e.g., with datatype different from {\tt
MPI\_BYTE}, where the datatypes of the corresponding entries in the sender
program, in the sender
buffer descriptor, in the receiver buffer descriptor and in the receiver program
should all match.
\item
Communication of untyped values (e.g., of datatype {\tt MPI\_BYTE}, where the
the corresponding entries in the sender and the receiver buffer
descriptors have both datatype {\tt MPI\_BYTE}.  In this case, there are no
requirements on the datatypes of the corresponding entries in the sender and the
receiver programs, nor is it required that they be the same.
\end{itemize}

\subsection{Data conversion}

We use the following terminology:
\begin{description}
\item[type conversion] changes the datatype of a value, e.g. by rounding a
{\tt REAL} to an {\tt INTEGER}.
\item[representation conversion] changes the binary representation of a value,
e.g. from Hex floating point to IEEE floating point.
\end{description}

Representation conversion
is needed when data is moved across machines that use different binary encodings
for the same datatype.  Representation conversion preserves, to the greatest
possible extend, the {\bf value}  represented.  However, rounding errors and
overflow and underflow exceptions may occur during floating point conversions;
conversion of integers may also lead to exceptions when a value that
can represented in one system cannot be represented in the other
system.
MPI does not specify rules for representation conversion.

The type matching rules imply that MPI communication never entails type
conversion.  On the other hand, MPI requires that a representation conversion be
performed when a typed value is transferred across environments that use
different representations for the datatype of this value.  An exception
occurring during representation conversion results in a failure of the
communication; an error occurs either in the send operation, or the receive
operation, or both.

If a value sent in a message is untyped (i.e., of type {\tt MPI\_BYTE}),
then the binary representation of the byte stored at the receiver is
identical to the binary representation of the byte loaded at the sender.  This
holds true, whether sender and receiver run in the same or in distinct
environments; no representation conversion is required.

Note that no conversions ever occur when an MPI program executes in
a homogeneous system, where all processes run in the same environment.
Also note the different behavior of {\tt MPI\_BYTE} and of {\tt MPI\_CHAR}.
A buffer descriptor entry with datatype of {\tt MPI\_CHAR} can only match
a Fortran variable of type {\tt CHAR}; and representation conversion may occur
when values of type {\tt MPI\_CHAR} are transferred., e.g., from an EBCDIC
encoding to an ASCII encoding.

Consider the following examples.

\begin{verbatim}
!  FIRST EXAMPLE
INTEGER A(100)
...
CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT)
CALL MPI_APPEND_CONTIGUOUS (BH, A(1), 1, MPI_CHAR)
\end{verbatim}

This program is erroneous, since the type of the buffer descriptor does not
match the type of the variable in the buffer.  The behavior or this program is
undefined.   A desirable behavior is for that error to be detected
(at compile time or run time) and result in an error condition.

\begin{verbatim}
!  SECOND EXAMPLE
INTEGER A(100)
...
CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT)
CALL MPI_APPEND_CONTIGUOUS (BH, A(1), 1, MPI_BYTE)
\end{verbatim}

This program is correct.  The byte that will be
transmitted in a message created with this buffer descriptor is
the first byte in the storage occupied by {\tt A(1)}.  The
value of this byte is implementation dependent, and will vary according to the
number of bytes used to store
Fortran variables of type {\tt INTEGER} and the byte ordering on the host
machine.

\begin{verbatim}
!  THIRD EXAMPLE
INTEGER A(100)
...
CALL MPI_CREATE_BUFFER(BH1, MPI_PERSISTENT)
CALL MPI_APPEND_CONTIGUOUS (BH1, A(1), 1, MPI_INTEGER)
...
CALL MPI_CREATE_BUFFER(BH2, MPI_PERSISTENT)
CALL MPI_APPEND_CONTIGUOUS (BH2, A(1), 4, MPI_BYTE)
\end{verbatim}

Consider the following three cases:
\begin{enumerate}
\item
A message is sent using buffer descriptor handle {\tt BH1} and received using
buffer descriptor handle {\tt BH2}.   The program is erroneous, since the sender
and receiver buffer descriptors don't match.  The behavior of the program is
undefined, and a possible (desirable but unlikely) behavior is for an error
condition to occur.
\item
Both sender and receiver use buffer descriptor handle {\tt BH1}.
The program is correct, and an integer value is transmitted from sender to
receiver.  The value assigned to {\tt A(1)} at the receiver is identical
to the value of {\tt A(1)} at the sender,
but the binary representation of this value may be different, if
sender and receiver run in different environments.
\item
Both sender and receiver use buffer descriptor handle {\tt BH2}.  The program is
correct.  Assume that both sender and receiver use four bytes to store
integers.  Then the
value stored in {\tt A(1)} at the receiver has the same binary representation as
the value of {\tt A(1} at the sender.  If the same byte ordering and
the same encoding of integer values is used at both ends then
{\tt A(1)} at the receiver is set to the value of {\tt A(1)} at the
sender; otherwise the same 32 bits may encode different values.
\end{enumerate}

\implement{

The current definition does not require messages to carry type information.
A message can be composed and sent using
only the information in the sender buffer descriptor, and can be
received and stored using only the information in the receiver buffer
descriptor.  If messages are sent between different machines then one can either
use a ``universal'' data encoding for messages, use knowledge of the receiver
environment in order to convert data at the sender, or use knowledge of the
sender environment in order to convert data at the receiver.   In either case
the local buffer descriptor can be used to derive the types of the values
transferred.  However, additional
type information carried by messages can provide better error detection.
It is possible to add always such additional type information, for better
error
checking, or add it when running in ``debug'' mode, where some performance is
sacrificed for added error checking, etc.

}

\discuss{

There is no agreement yet on the need for a definition and, if needed, on the
right definition of type matching and conversion between programs written in
different languages.   The natural interpretation of ``send from C and
receive in Fortran'' is either ``send from C, receive in C and convert to
Fortran'', or ``convert to Fortran, send from Fortran and receive in
Fortran''.  However, the correspondence between Fortran types and C types
is implementation dependent.  Thus, the diagram shown below is not commutative,
and the two interpretations of C to Fortran communication may lead to different
results.

\begin{tabular}{c c c}
machine 1 & & machine 2 \\
& send & \\
C program & $\rightarrow$ & C program \\
& & \\
$\downarrow$ & & $\downarrow$ \\
& send & \\
Fortran program & $\rightarrow$ & Fortran program
\end{tabular}

A good solution would make this diagram commutative.   Otherwise, one needs to
pick one of the two paths as the canonical definition of interlanguage
communication.  If the upper path is chosen (message carries sender datatypes),
then the receiver of a message needs
to know the language the message was sent from; if the lower path is chosen,
(message carries receiver datatypes), then
the sender needs to know the language the message is received into.

A possible solution (proposed by Danny Nessett) is to leave to the user the
choice of specifying whether to send ``Fortran datatypes'' or ``C datatypes'',
in the above situation.   One would still require exact matching of send buffer
descriptors and receive buffer descriptors.  However, a Fortran program could
use a buffer descriptor with ``C datatypes'', and vice versa.  The standard
would specify which matching of Fortran and C datatypes are legal.   For
example, it would be correct to send a Fortran variable of type {\tt REAL} using
a buffer descriptor of type {\tt MPI\_FLOAT}.  Data conversion might be required
if C float numbers and Fortran real numbers do not use the same presentation.

Another (non)solution is to ignore the issue and let the behavior
of C to Fortran communication be undefined.  The user can always match language
at sending and receiving end if he or she is willing
to struggle with the issue of calling MPI C functions from
Fortran or vice-versa.

}

\section{Receive Criteria}

The selection of a message by a receive operation is done uniquely
according to the value of the message envelope.  The receive operation specifies
an {\bf envelope pattern}; a message can be received by that receive operation
only if its envelope matches that pattern.
A pattern specifies
values for the {\tt source}, {\tt tag} and {\tt context} fields of
the message envelope.  In addition, the value for the {\tt dest} field is set,
implicitly, to be equal to the receiving process id.
The receiver may specify a {\tt DONTCARE} value for {\tt source}, or {\tt tag},
indicating that any source and/or tag are acceptable.   It cannot specify a
DONTCARE value for {\tt context} or {\tt dest}.
Thus, a message can be received by a receive
operation only if it is addressed
to the receiving task, has a matching context, has matching source unless
source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in
the pattern.   DONTCARE values for source or tag are specified via a named
constant for source dontcare and a named constant for tag dontcare.

The length of the received message must be less then or equal the length of the
receive buffer.  I.e., all incoming data must fit, without truncation, into the
receive buffer.  It is erroneous to receive a message which length exceed the
receive buffer, and the outcome of program where this occurs is undetermined.

\section{Communication Mode}

A send operation can occur in one of three modes:
\begin{description}
\item[STANDARD]
The send may start whether or not a matching receive has been posted.
\item[READY]
The send may start only if a matching receive has been posted.
\item[SECURE]
The send operation will not complete successfully until it is guaranteed that
the message sent will be received.
\end{description}

The start and completion of a {\bf standard} send does not depend on the status
of a matching receive.

A {\bf ready send} can start only if a matching receive is already posted;
otherwise the operation is erroneous and its outcome is undefined.  In some
systems, this allows the removal of a hand-shake
operation that is otherwise required, and results in improved
performance. Its completion does not depend on the
status of a matching receive.

A {\bf secure send}  will complete if a matching receive is posted, and the
receive operation is committed to receive the message sent by the secure send.
(I.e., the send and receive operations have already been matched, the
receive operation has already started and cannot be cancelled anymore.)
The start of a secure send does not depend on the status of a matching receive.

A receive operation can occur in one of two modes: {\bf STANDARD} and {\bf
SECURE}.   A secure send can be matched only by a {\bf secure receive}; a
program where a secure send can be matched by a standard receive is
erroneous and its
behavior is undefined.  The secure receive can be posted either before or
after the matching send.

A standard send and a ready send can both be matched only by a {\bf standard
receive}; a program where a secure receive can match a nonsecure send is
erroneous and its behavior is undefined.  The receive has to be posted ahead of
the send for a ready send and can be posted either before or after the send is
posted for a standard send.

\discuss{

I got criticisms about the previous notation (REGULAR) and the current one
(STANDARD). I can also try NORMAL, DEFAULT, ... We should probably vote, at some
point.   Also, Lyndon proposes to use SYNCHRONOUS, rather than secure, since the
other modes can also offer secure communication with the right use.
}

\implement{

The current definition assumes it is the user responsibility to
prevent matching of a regular send with a secure receive and
vice-versa.  Otherwise, messages need be tagged as standard or secure.

Assume an implementation of MPI communication that avoids retries.  What
protocols can be used?  Since a receive may have a wildcard source field, the
first step in the protocol must be executed by the sender.

For a ready send, this first step may involve sending the data, since the
receiver
is guaranteed to have space to store it.   The protocol is send, possibly
followed by an ack-send.

The protocol for secure send is likely to be the following:
The sender sends a request-to-send message.
The receiver stores this request (which contains a copy of the message
envelope).  When a matching receive is posted, the receiver sends back an
acknowledge, and the sender now sends the message.  The protocol is (req,
ack-req, send), possibly followed by an ack-send.

What protocol is used for standard sends?
Either the second protocol (req, ack-req, send) or a mix of the first and the
second (e.g. first used for short messages, second for long messages).

When the receiver does not wildcard the source field then the protocol
can be reversed:  The receiver sends a clear-to-send message, and the
sender can proceed with the short protocol if it has a clear-to-send
when the send is executed.

What do we gain from the ``secure receive''?  Some optimizations for
regular receives (e.g., in a regular receive of a short message the
receiver knows the short protocol will be used, and need not check for
a request-to-send).  Q: is that enough justification for the secure
receive?

A counter argument (Leslie Hart) is that one could conceive of an implementation
where a regular receive reserves buffer space, so that a program that use
regular receives is not safe, even if only secure sends are used.
I doubt this is a strong argument.  On the other hand, it indicates that, if the
expectation is that programs written using only secure sends are safe,
then this is an additional correctness condition on the implementation that need
be made explicit.

}

\section{Communication Objects}

An opaque communication object identifies various properties of a communication
operation, such as the buffer descriptor that is associated with it, its
context, the tag and destination parameters to be used for a send, or the tag
and source parameters to be used for a receive.  In addition, this object stores
information about the status of the last communication  operation that was
performed with this object.  This object is accessed using a communication
handle.

One can consider
communication operations to consist of the following suboperations:

\begin{description}
\item[INIT(operation, params, handle)]
Process provides all relevant parameters for its participation in the
communication operation (type of operation, data buffer, tag, participants,
etc.). An object is created that identifies the operation.
\item[START(handle)]
The communication operation is started
\item[COMPLETE(handle)]
The communication operation is completed.
\item[FREE(handle)]
The communication object, and associated resources are freed.
\end{description}
Correct invocation of these suboperations is a sequence of the form
\[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\]
I.e., an object needs be created before communication occurs; it can be reused
only after the previous use has completed; and it needs to be freed eventually
(of course, one can assume that all objects are freed at program termination,
by default).

The above scenario pertains to {\em persistent} objects.  One can also create
{\em ephemeral} objects.  Such an object persists only until the communication
operation is completed, at which point it is destroyed.  Thus the correct
invocation of suboperations with an ephemeral object is
{\bf INIT START COMPLETE}.

A user may directly invoke these suboperations.  This would allow the
amortization of
the overhead of setting up a communication over many successive uses of the same
handle, and allows the overlap of communication and computation.   Simpler
communication operations combine several of these suboperations into one
operation, thus simplifying the use of communication primitives.
Thus, one only needs to specify
precisely the semantics of these suboperations in order to specify the semantics
of MPI point to point communication operations.

We say that a communication operation (send or receive) is {\bf posted} once
a {\bf start} suboperation has been invoked; the operation is {\bf completed}
once the {\bf complete} suboperation has completed.  A send and a receive
operation
{\bf match} if the receive pattern specified by the receive matches the message
envelope created by the send.

\subsection{Communication Object Creation}

\discuss{  This section has not yet been approved}

An object for a send operation is created by a call to {\bf MPI\_INIT\_SEND}.
A call to {\bf MPI\_INIT\_RECV} is similarly used for creating an object for a
receive operation.   The creation of a communication object is a local operation
that need not involve communication with a remote process.

\mpifunc{MPI\_INIT\_SEND (handle, buffer\_handle, dest, tag, context, mode,
persistence)}

Creates a send communication object.  Parameters are
\begin{description}
\item[OUT handle] message handle.  The handle should not be associated with any
object before the call.
\item[IN buffer\_handle] handle to send buffer descriptor
\item[IN dest] rank in context of destination (integer)
\item[IN tag] user tag for messages sent with this handle (integer)
\item[IN context] context of messages sent with this handle
\item[IN mode] send mode (state type, with three values: {\tt MPI\_STANDARD},
{\tt MPI\_READY} and {\tt MPI\_SECURE})
\item[IN persistence] handle persistence (state type, with two values: {\tt
MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL})
\end{description}

\mpifunc{MPI\_INIT\_RECV (handle, buffer\_handle, source, tag, context, mode,
persistence)}

Create a receive handle.  Parameters are
\begin{description}
\item[OUT handle] message handle.  The handle should not be associated with any
object before the call.
\item[IN buffer\_handle] handle to receive buffer descriptor.
\item[IN source] rank in context of source, or {\tt MPI\_ANY\_SOURCE} (integer).
\item[IN tag] user tag for messages received with this handle, or {\tt
MPI\_ANY\_TAG} (integer).
\item[IN context] context of messages received with this handle.
\item[IN mode] receive mode (state type, with two values: {\tt MPI\_STANDARD},
and {\tt MPI\_SECURE})
\item[IN persistence] handle persistence (state type, with two values: {\tt
MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL})
\end{description}

See Section~\ref{subsec:envelope} for a discussion of source, tag and context.

\discuss{
I have not included proposals for partially specified message handles, that
some peoples seem to desire.

I have merged all handle setup into one call.

The functions {\tt MPI\_INIT\_SEND} and {\tt MPI\_INIT\_RECV} have the same
list of parameters.  Jim Cownie suggests to have only one function with an
extra argument.

Ephemeral ``persistent handles'' are not terribly useful as such.  But their
existence allow us to describe all the higher level send/receive
operations in terms of basic operations on handles.

}

\subsection{Communication Start}

\mpifunc{MPI\_START(handle)}
\begin{description}
\item[IN handle] communication handle
\end{description}

The {\tt MPI\_START} function starts the execution of a communication operation
(send or receive).  A sender should not update the send buffer after a
send operation has started until after it has completed.  A receiver should
not access the receive buffer after a receive operation was started until
after it has
completed.  A program that does not satisfy these conditions is erroneous and
its outcome is undetermined.

\subsection{Communication Completion}
\label{subsec:complete_ops}

\mpifunc{MPI\_WAIT ( handle, return\_status\_handle)}
\begin{description}
\item[IN handle]  communication handle
\item[IN return\_handle] handle that is associated with return status object.
the return status object is updated by call.
\end{description}

A call to {\bf MPI\_WAIT} returns when the send operation
identified by {\tt handle} is complete.
The completion of a send operation indicates that the sender is now free to
update the locations in the send buffer, or any other location that can be
referenced by the send operation.  However, it does not indicate that the
message has been received; rather it may have been buffered by the communication
subsystem.

The completion of a receive operation indicates that the receiver is now free to
access the locations in the receive buffer, which contain the received message,
or any other location that can be referenced by the receive operation.  It does
not indicate that the matching send operation has completed.

The call returns a handle to an opaque object that contains information on
the completed operation -- the {\bf return status} object.

\mpifunc{MPI\_STATUS (handle, flag, return\_handle)}
\begin{description}
\item[IN handle]  communication handle
\item[OUT flag] logical
\item[OUT return\_handle] handle that is associated with return status
object.
\end{description}

A call to {\bf MPI\_STATUS} returns {\tt flag=true} if the
operation identified by {\tt handle} is complete.  In such case, the return
handle points to an opaque object that contains information on the completed
operation.  It returns
{\tt flag=false}, otherwise. In such case, the value of the return handle is
undefined.

A call to {\tt MPI\_WAIT} blocks only the executing thread.  If the
executing process is multithreaded, then other threads within the process can be
scheduled for execution.

The use of a blocking receive operation ({\tt
MPI\_WAIT}) allows the operating system to deschedule the blocked thread
and schedule another thread for execution, if such is available.  The use of
a nonblocking receive operation ({\tt MPI\_STATUS}) allows the user to
schedule alternative activities within a single thread of execution.

The intended implementation of {\tt MPI\_STATUS} is for that operation
to return as soon as possible.  However, if repeatedly called for an operation
that is enabled, it must eventually succeed.

The return status object for a send operation carries no information.  The
return status object for a receive operation carries information on the source,
tag and length of the received message.  These fields are required because the
receive operation may have specified {\tt DONTCARE} in either source or tag
field, and the message may have been shorter than the receive buffer.

\mpifunc{MPI\_QUERY( handle, count, source, tag)}
\begin{description}
\item[IN handle] handle to return status object
\item[OUT count] number of elements received.
\item[OUT source] rank of message sender in message context (integer).
\item[OUT tag] tag of received message (integer).
\end{description}

\mpifunc{MPI\_CREATE\_STATUS( handle)}

Creates a new return status object.
\begin{description}
\item[OUT handle] handle to return status object.
\end{description}

\discuss{

The use of a return status object, rather than a list of parameters may simplify
the use of MPI routines, if the values stored in the object are seldom checked.
A predefined  return status object should be provided, to ease programming.
The main reason for the use of a return-status object is that one wants
to be able to use the same {\tt MPI\_WAIT} and {\tt MPI\_STATUS} calls
for checking on different types of operations, e.g. both sends and
receives, and perhaps new operations in the future.  These different
operations return different information (albeit we could decree that a
send returns the same information as a receive).

The issue of memory allocation for return\_status handles is not yet
solved.  There is a desire that the user will be able to allocate
space for them (e.g., on the stack).

We return number of elements received, rather than bytes received
because ``number of elements'' is closer to the application semantic
level (user does not need to be aware of the size of elements), and is
more invariant.
The return status object returns the number of {\bf elements}
received.  If there was no truncation, then this is equal to the
number of elements sent.  On the other hand, the number of bytes
collected from the sender memory, the number of bytes sent over the
wire, and the number of bytes stored in the receiver memory may all be
different, in a heterogeneous environment.
The number of elements sent can be computed
form the send buffer descriptor; the number of elements received can
be computed from the receive buffer descriptor and the length (in
bytes) of the received message.

{\tt MPI\_STATUS} and {\tt MPI\_QUERY} are not good names -- they can be easily
confused.  Leslie Hart suggests {\tt MPI\_TEST} or {\tt MPI\_CHECK} for the
first.  Perhaps we should replace {\tt MPI\_STATUS} with {\tt MPI\_TEST}
and {\tt MPI\_QUERY} with {\tt MPI\_STATUS}.

}

\subsection{Multiple Completions}

It is convenient to be able to wait for the completion of any or all the
operations in a set, rather than having to wait for a specific message.
A call to {\tt MPI\_WAITANY} or {\tt MPI\_STATUSANY} can be used to wait for the
completion of one out of several operations; a call to {\tt MPI\_WAITALL} can be
used to wait for all pending operations in a list.

\mpifunc{MPI\_WAITANY ( list\_of\_handles, count, index, return\_handle)}

Blocks until one of the operations associated with the communication
handles in the array has completed.  Returns the index of that handle in the
array, and returns the status of that operation in the object associated with
the return\_handle.  The parameters are:

\begin{description}
\item[IN list\_of\_handles] list of handles to communication objects.
\item[IN count] list length (integer)
\item[OUT index] index of handle for operation that completed (integer).
\item[OUT return\_handle] handle that is associated with return status
object.  Set to return status of operation that completed.
\end{description}

The successful execution of {\tt MPI\_WAITANY(list\_of\_handles, index,
return\_handle)} has the same effect as the successful execution of {\tt
MPI\_WAIT(handle[i], return\_handle)}, where {\tt i} is the value
returned by {\tt index} and {\tt handle[i]} is the {\tt i}-th handle in the
list, and the cancellation of all remaining wait operations.

If more then one operation is enabled and can terminate, one is arbitrarily
chosen (subject to the restrictions on operation termination order,
and fairness, see Section~\ref{subsec:correct}).

{\tt MPI\_WAITANY ( list\_of\_handles, count, index, return\_status\_handle)}
is
\begin{verbatim}
   {MPI_WAIT (handle[0], return_handle); index = 0} || ...
||
   {MPI_WAIT (handle[count-1], return_handle); index = count-1}
\end{verbatim}

(``$||$'' indicates choice; one of the alternatives is chosen,
nondeterministically.)

\mpifunc{MPI\_STATUSANY ( list\_of\_handles, count, index, return\_handle)}

Causes either one or none of the operations associated with the communication
handles to return.   In the former case, it has the same return semantics as a
call to {\tt MPI\_WAIT\_ANY}.  In the latter case, it returns a value of --1 in
{\tt index} and {\tt return\_handle} is undefined.
The parameters are:

\begin{description}
\item[IN list\_of\_handles] list of handles to communication objects.
\item[IN count] list length (integer)
\item[OUT index] index of handle for operation that completed, or -1 if none
completed (integer).
\item[OUT return\_handle] handle that is associated with return status
object.  Set to return status of operation that completed, if any; undefined
when {\tt index = -1}.
\end{description}

\mpifunc{MPI\_WAITALL(list\_of\_handles, count, list\_of\_return\_handles)}

Blocks until all communication operations associated with handles in the list
complete, and return the status of all these operations.
The parameters are:

\begin{description}
\item[IN list\_of\_handles] list of handles to communication objects.
\item[IN count] lists length (integer)
\item[OUT list\_of\_return\_handles]  Must have the same length as the
first list.  Each return status object is set to the return status of the
corresponding operation in the first list.
\end{description}

\discuss{

The fairness requirement given in Section~\ref{subsec:correct} implies
that the use of {\tt WAIT\_ANY} cannot lead to starvation:  If the
sending process has issued a send complete operation, and the receiving
process repeatedly post a receive for the message sent, then that
message must be eventually received.  Section~\ref{subsec:correct} has
not yet been discussed.  The fairness requirement can be attacked
either for being too weak (eventually is not good enough), or too
strong (hard to implement).  In the later case, if the requirement of
fairness is dropped from MPI implementations, then some mechanism need
be provided to the user to achieve fairness by his or her own devices.
One such proposal is to be able to specify a rotating priority order
on the operations posted by a {\tt MPI\_WAITANY}:  search the list
sequentially, starting from a user specified position.

}

\section{Blocking Communication}

Blocking send and receive operations combine all communication suboperations
into
one call.  The operation returns only when the communication completes and no
communication object persists after the call completed.  However, the buffer
descriptor object needs be created ahead of the call.

We use the following naming convention for such operations:

\[
\left[ \begin{array}{c} - \\ \bf R \\ \bf S \end{array} \right]
\left[ \begin{array}{c} \bf SEND \\ \bf RECV \end{array} \right]
\]

The first letter (void, {\bf R}  or {\bf S}) indicates the start mode (standard,
ready, or secure).  Only two of the combinations (standard and secure)
are meaningful for receives.

\mpifunc{MPI\_SEND (buffer\_handle, dest, tag, context)}
is
\begin{verbatim}
MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_STANDARD,
MPI_EPHEMERAL)
MPI_START(handle)
MPI_WAIT(handle, null)
\end{verbatim}

\mpifunc{MPI\_RSEND (buffer\_handle, dest, tag, context)}
is
\begin{verbatim}
MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_READY, MPI_EPHEMERAL)
MPI_START(handle)
MPI_WAIT(handle, null)
\end{verbatim}

\mpifunc{MPI\_SSEND (buffer\_handle, dest, tag, context)}
is
\begin{verbatim}
MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_SECURE, MPI_EPHEMERAL)
MPI_START(handle)
MPI_WAIT(handle, null)
\end{verbatim}

\mpifunc{MPI\_RECV(buffer\_handle, source, tag, context,
return\_handle)}
is
\begin{verbatim}
MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_STANDARD, MPI_EPHEMERAL)
MPI_START(handle)
MPI_WAIT(handle,return_handle)
\end{verbatim}

\mpifunc{MPI\_SRECV(buffer\_handle, source, tag, context,
return\_handle)}
is
\begin{verbatim}
MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_SECURE, MPI_EPHEMERAL)
MPI_START(handle)
MPI_WAIT(handle,return_handle)
\end{verbatim}

\implement{

While these functions can be implemented via calls to functions that
implement suboperations, as described in this subsection, an efficient
implementation may optimize away these multiple calls, provided it does not
change the behavior of correct programs.

}

\discuss{

We use a different function name, rather than an additional mode parameter, in
order to save on an additional parameter (performance and user convenience).
Lyndon and perhaps other prefer fewer functions and an additional parameter.

}

\section{Nonblocking Communication}

Nonblocking send and receive operations combine the first two
suboperations ({\tt INIT} and {\tt START}) into one call.
They use ephemeral communication objects, so that the operation is completed,
and the associated resources are freed, by using one of the functions {\tt
MPI\_WAIT, MPI\_STATUS, MPI\_WAITANY, MPI\_STATUSANY}, or {\tt MPI\_WAITALL}.
Here, too, a buffer object has to be created ahead of the communication
initiation operation.

We use the same naming convention as for blocking operations:  a prefix of
{\bf R} ({\bf S}) indicates the {\tt READY} ({\bf SECURE}) mode.
In addition, a prefix of {\bf I} is
used to indicate {\em immediate} (i.e., nonblocking) execution.

\mpifunc{MPI\_ISEND (handle, buffer\_handle, dest, tag, context)}
is
\begin{verbatim}
MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_STANDARD, MPI_EPHEMERAL)
MPI_START(handle)
\end{verbatim}

\mpifunc{MPI\_IRSEND (handle, buffer\_handle, dest, tag, context)}
is
\begin{verbatim}
MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_READY, MPI_EPHEMERAL)
MPI_START(handle)
\end{verbatim}

\mpifunc{MPI\_ISSEND (handle, buffer\_handle, dest, tag, context)}
is
\begin{verbatim}
MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_SECURE, MPI_EPHEMERAL)
MPI_START(handle)
\end{verbatim}

\mpifunc{MPI\_IRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)}
is
\begin{verbatim}
MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_STANDARD, MPI_EPHEMERAL)
MPI_START(handle)
\end{verbatim}

\mpifunc{MPI\_ISRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)}
is
\begin{verbatim}
MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_SECURE, MPI_EPHEMERAL)
MPI_START(handle)
\end{verbatim}

\section{Contiguous Block Communication Operations}

The most frequent type of buffer used is a buffer with one contiguous component.
We specialize the functions in
the two previous subsections to this case, thus avoiding the need for the
creation of a buffer descriptor object.  We use the same naming scheme used
in the previous subsections, and append a {\bf C} in the function name, for {\tt
CONTIGUOUS}.

\mpifunc{MPI\_SENDC (start, count, datatype, dest, tag, context)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_SEND (buffer_handle, count, dest, tag, context)
\end{verbatim}

\mpifunc{MPI\_RSENDC (handle, start, count, datatype, dest, tag, context)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_RSEND( buffer_handle, count, dest, tag, context)
\end{verbatim}

\mpifunc{MPI\_SSENDC (handle, start, count, datatype, dest, tag, context)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_SSEND( buffer_handle, count, dest, tag, context)
\end{verbatim}

\mpifunc{MPI\_RECVC (start, count, datatype, source, tag, context,
return\_status\_handle)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_RECV( buffer_handle, source, tag, context, return_status_handle)
\end{verbatim}

\mpifunc{MPI\_SRECVC (start, count, datatype, source, tag, context,
return\_status\_handle)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_SRECV( buffer_handle, source, tag, context, return_status_handle)
\end{verbatim}

\mpifunc{MPI\_ISENDC (handle, start, count, datatype, dest, tag, context)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_ISEND( handle, buffer_handle, dest, tag, context)
\end{verbatim}

\mpifunc{MPI\_IRSENDC (handle, start, count, datatype, dest, tag, context)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_IRSEND( handle, buffer_handle, dest, tag, context)
\end{verbatim}

\mpifunc{MPI\_ISSENDC (handle, start, count, datatype, dest, tag, context)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_ISSEND( handle, buffer_handle, dest, tag, context)
\end{verbatim}

\mpifunc{MPI\_IRECVC(handle, start, count, datatype, source, tag, context,
return\_status\_handle)}
is
\begin{verbatim}
MPI_CREATE_BUFFER (buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_IRECV( handle, buffer_handle, source, tag, context)
\end{verbatim}

\mpifunc{MPI\_ISRECVC(handle, start, count, datatype, source, tag, context,
return\_status\_handle)}
is
\begin{verbatim}
MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL)
MPI_APPEND( buffer_handle, start, count, datatype)
MPI_ISRECV( handle, buffer_handle, source, tag, context)
\end{verbatim}

\section{Probe and Cancel}

The {\tt MPI\_PROBE} operation allows incoming messages to be checked for,
without
actually receiving them.  The user can then decide where to receive them, based
on the information returned by the probe (basically, the information on the
message envelope).  An additional function, {\tt MPI\_GET\_LEN} allows
the amount of storage needed to receive the message to be computed, when this
length is not readily computed from the information returned by
{\tt MPI\_PROBE}.

The {\tt MPI\_CANCEL} operation allows pending communications to be cancelled.
This
is required for cleanup.   Posting a send or a receive ties user resources (send
or receive buffers), and a cancel may be needed to free these resources
gracefully.

\mpifunc{MPI\_PROBE( source, tag, context, flag, datatype, return\_status)}

\begin{description}
\item[IN source] rank in context of source, or {\tt MPI\_ANY\_SOURCE} (integer).
\item[IN tag] user tag for messages received with this handle, or {\tt
MPI\_ANY\_TAG} (integer).
\item[IN context] context of messages received with this handle.
\item[OUT flag] (logical)
\item[IN type] assumed type of data in message (status variable).
\item[OUT return\_handle] handle that is associated with return status
object.
\end{description}

{\tt MPI\_PROBE} returns {\tt flag = true} if there is a message
that can be received and that matches the pattern specified by the
parameters {\tt source}, {\tt tag}, and {\tt context}.
It returns {\tt flag = false}, otherwise. If {\tt MPI\_PROBE} returns
{\tt flag = true}, then the length, source and tag
of the message matched are returned in the return status object.  These are the
same values that would have been returned by a call to {\tt MPI\_RECV} or to
{\tt MPI\_SRECV} executed at the same point in the program (with a caveat
concerning length; see below).
These values can be decoded from the return status object using the {\tt
MPI\_RETURN\_STAT} function.  The value returned in the return status object is
undefined if {\tt flag=false}.

The length value returned by the return status object is (correctly) the number
of elements in the message, provided that all elements in the message are of the
type specified by the {\tt type} parameter; otherwise the length value returned
is undefined.

A subsequent
receive executed with the same context, and the source and tag
returned by the call to {\tt MPI\_PROBE} will receive the message
that was matched by the probe, if no other intervening receive occurred after
the probe.  If the receiving process is multithreaded, it is the user
responsibility to ensure that the last condition holds.

\discuss{

Do we want {\tt MPI\_PROBE} to return mode (standard or secure)?
If yes, we need to carry mode information with messages (it's deja vu all over
again).  If not, then it is up to the user to encode this information (in the
tag) so that receiver can decide whether to use a secure or standard send.

MPI guarantees that successive messages sent from a source to a destination
within the same context are received in the order they are sent.  Thus,  MPI
must support, either explicitly or implicitly, a FIFO structure to
manage messages between each pair of messages, for each context. {\tt
MPI\_PROBE} returns information on the first matching message in this FIFO; this
will also be the message received by the first subsequent receive with the same
source, tag and context as the message matched by {\tt MPI\_PROBE}.

Message passing in MPI can be implemented without appending
type information to messages.  A message is merely a string of bytes and the
interpretation of these bytes
into a sequence of typed elements is done using the information in
the buffer descriptors at each end.
The ability to use such implementation strategy is deemed to be an
important goal. In such implementation, when a message arrives,
it is not be known how many elements it contains, or even how much storage is
needed to receive that message (because of possible representation
conversion in a
heterogeneous environment).  The probe function cannot use a buffer descriptor;
this defeats the purpose of probing in order to decide where to receive a
message.  Therefore, probe cannot, in general, return correct
length information.   Still, it is often the case that probe is used to decide
how much storage to allocate in order to receive a message.  Encoding such
information in the message tag is deemed to be too awkward, and it is deemed
important for {\tt MPI\_PROBE} to return some useful size information.
The current definition of {\tt MPI\_PROBE} is a compromise between these two
goals.   For the most common case of messages where all entries have the
same type {\tt MPI\_PROBE} returns the correct length information; the more
esoteric case is handled by the {\tt MPI\_GET\_LEN} that is described below.
The current solution saves us the need for one additional word per message that
would otherwise be needed to transfer the message length (in elements) with the
message.

There was a request for a blocking probe function.

}

\mpifunc{MPI\_GET\_LEN( count, return\_status, buffer\_descriptor)}

Computes the number of elements that were to be received, if the message
that is associated with the return status handle would be received in the
buffer associated with the buffer descriptor handle.
\begin{description}
\item[OUT count] number of elements that were to be received (integer)
\item[IN return\_status] handle to return status descriptor
\item[IN buffer\_descriptor] handle to buffer descriptor
\end{description}

\discuss {

In order to support this function, an additional field is needed in the
return status object, i.e., number of bytes in the incoming message matched by
probe, or value of datatype parameter provided by probe (unless we add this
parameter to the {\tt MPI\_GET\_LEN} function, which is too ugly to bear).

}

\mpifunc{MPI\_CANCEL( handle, flag)}

\begin{description}
\item [IN handle] handle to communication object
\item[OUT flag] (logical)
\end{description}

A call to {\tt MPI\_CANCEL} cancels a pending communication operation
(send or receive).  The call returns {\tt flag = true} if the cancel
operation succeeded, {\tt flag = false} otherwise.  It must be the
case that either the cancel operation succeeds or that the pending
communication operation completes (but not both).
If a communication operation was cancelled successfully, the effect is
as if this operation was never executed.
If a send was cancelled successfully then the message sent will not be
received, the receive buffer of any posted receive for that message
will not be altered, and any such receive has to be satisfied
by another send; if a receive was cancelled successfully, then the
receive buffer posted will not be altered, and any send
matching this receive has to be satisfied by another receive.

\implement{

There is not expectation that a cancel operation will be fast.

}

\section{Correctness}
\label{subsec:correct}

\discuss{The material in this section has not yet been discussed by MPIF.
Some or all of it is likely to move to Section~\ref{sec:formal}.
It is incorporated here for completeness.}

\subsection{Order}

MPI preserves the order of messages within the same context, between any fixed
pair of
processes. In other words, if process A executes two successive send {\tt start}
suboperations within the same context,
process B executes two successive receive {\tt start} operations, and both
receives
match either sends, then the first receive will receive the message sent by the
first send, and the second receive will receive the message sent by the second
send.  Thus, if a two messages from the same source can satisfy a pending
receive, the first message sent is accepted; if a message can satisfy two
pending receives, the first receive posted is satisfied.

The last paragraph assumes that the send {\tt start} operations are ordered by
the
program order at process A, and the receive {\tt start} operations are ordered
by the
program order at process B.  If a process is multithreaded and the operations
are executed by distinct threads, then the semantics of the threaded system may
not define an order between the two operations, in which case the condition is
void.

\subsection{Progress and Fairness}

We can model the execution of MPI programs as an interaction between executing
processes that execute each their own program, and the {\bf communication
subsystem}.  The communication subsystem may have various constraints on the
amount of resources it can use.  E.g.:

Bounds on the number and total sizes of active communication objects.
Such bound can be global, per node, or per pair of communicating nodes.

Bounds on the number and total sizes of messages buffered in the system.  Such
bound can, again, be global, per node, or per pair of communicating node.  In
addition, a message may be buffered at the sender, at the receiver, at both, or
perhaps at another place altogether.

Thus, it will be difficult to set rules on resource management of the
communication subsystem.   However, it is generally expected that implementers
will provide information on the mechanism used for resource allocation, and that
query and set functions will allow to query and possibly control the amount
of available resources.

We provide in this section a set of minimal requirements on the communication
subsystem.  Programs that execute on any subsystem that fulfils these minimal
requirements are {\bf safe} and will port to any MPI implementation.
{\bf Unsafe} programs may execute on some MPI implementations, depending on the
amount of available resources and the implementation used for the MPI
communication subsystem.  Finally {\bf  erroneous} programs never execute
correctly.  (While it is desirable to detect erroneous programs, it is not
possible to do so at compile time, and often prohibitive to do so a run time.
Thus, the document does not specify a behavior for erroneous programs, although
the desired behavior is to return a useful error message.)

\begin{enumerate}
\item
If a process executes an {\tt INIT}  operation,
then the operation eventually succeeds, or a {\em resource exception} occurs.
The standard does not specify when a resource exception is allowed to occur.  It
is expected that an operational definition will be made available, in the form
of test programs that have to execute with no resource exceptions.
It is highly desirable to have generous bounds on the number of
concurrently active communication objects each process may have, so that,
in practice, {\tt INIT} operations will always be guaranteed to succeed.
\item
Each process can initiate a communication operation for each active
communication object. I.e. correct {\tt START} operations always succeed
(eventually).
\item
A send operation is {\bf enabled} if the sending process has issued a
{\tt COMPLETE} operation and the receiving process has issued a {\tt
START} operation for a matching receive.  Symmetrically,
a receive operation is {\bf enabled} if the receiving process has issued
a {\tt COMPLETE} operation and the sending process has issued a {\tt
START} operation for a matching send.
An enabled operation may become {\bf disabled} either because it completes
successfully or, in the case of a receive, because the matching message is
successfully received by another receive operation or, in the case of
a send, because the matching receive successfully receives another
message.

{\bf An enabled operation either completes successfully or becomes permanently
disabled.}

I.e., an enabled operation either eventually succeeds, or becomes
disabled (progress); and an operation cannot be enabled infinitely many
times without ever succeeding (fairness).

\item
A {\tt FREE} operation always succeeds (eventually).
\end{enumerate}

The four conditions guarantee progress in the communication subsystem.  The
third condition guarantee (weak) fairness among competing communication
requests.

Examples (involving two processors with ranks 0 and 1)

The following program is safe, and should always succeed.
\begin{verbatim}

CALL MPI_RANK(rank, context)
IF (rank.EQ.0) THEN
   CALL MPI_SENDC(sendbuf, count, 1, tag, context)
   CALL MPI_RECVC(recvbuf, count, 1, tag, context)
ELSE    ! rank.EQ.1
   CALL MPI_RECVC(recvbuf, count, 0, tag, context)
   CALL MPI_SENDC(sendbuf, count, 0, tag, context)
END IF
\end{verbatim}

The following program is erroneous, and should always fail.

\begin{verbatim}

CALL MPI_RANK(rank, context)
IF (rank.EQ.0) THEN
   CALL MPI_RECVC(recvbuf, count, 1, tag, context)
   CALL MPI_SENDC(sendbuf, count, 1, tag, context)
ELSE    ! rank.EQ.1
   CALL MPI_RECVC(recvbuf, count, 0, tag, context)
   CALL MPI_SENDC(sendbuf, count, 0, tag, context)
END IF
\end{verbatim}

The receive operation of the first process must complete before its send, and
can complete only if the matching send
of the second processor is executed; the receive operation of the second
process must complete before its send and
can complete only if the matching send of the first process is executed.
This program will deadlock.

The following
program is unsafe, and may succeed or fail, depending on implementation.

\begin{verbatim}

CALL MPI_RANK(rank, context)
IF (rank.EQ.0) THEN
   CALL MPI_SENDC(sendbuf, count, 1, tag, context)
   CALL MPI_RECVC(recvbuf, count, 1, tag, context)
ELSE    ! rank.EQ.1
   CALL MPI_SENDC(sendbuf, count, 0, tag, context)
   CALL MPI_RECVC(recvbuf, count, 0, tag, context)
END IF

\end{verbatim}

The message sent by each process has to be copied out before the send operation
returns and the receive operation starts.  For the program to complete, it is
necessary that at least one of the two messages sent is buffered out of
either processes' address space.  Thus, this program can
succeed only if the communication system has sufficient
buffer space to buffer {\tt count} words of data.

If additional requirements will become part of the standard (e.g., bounds
on the minimal number of concurrently active handles that need be supported,
then further programs become safe.

\section{Missing}
\paragraph*{send-receive}

a function of the form  (for blocking communication)

{\tt MPI\_SEND\_RECV( send\_buffer, destination, recv\_buffer, source,
tag, context)}

It may be superfluous because the collective communication shift has similar
functionality.  Yet it allows to generate more easily communication patterns
that would lead to deadlock if implemented naively, with blocking communication.

An important variant is the case where {\tt send\_buffer = recv\_buffer}  (this
can either be a subcase of the general send\_recv function, or a separate
function, if send buffer and receive buffer are required to be disjoint in
the general function).  This introduces new functionality that is not available,
otherwise, and which seems to be often required.

An {\tt exchange} is the special case where source = destination.
\end{document}

>From [email-masked] Wed Jun  9 15:10:24 1993
Date: Wed, 9 Jun 93 08:11:37 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: May 14 mpi mail #1 of 1
Status: R

>From [email-masked] Fri May 14 00:15:16 1993
Date: Fri, 14 May 93 01:14:38 EDT
From: "Marc Snir" <[email-masked]>
To: [email-masked], [email-masked]
Status: R

%     MPI Authors:
% This is MY version of YOUR chapter.  It has a wrapper so that you
% should be able to simply LaTeX this.
%
% Please work from this text so that we are in synch.
%
% --Steve Otto

\documentstyle[twoside]{article}
\pagestyle{headings}
%\markright{ {\em Draft Document of the MPI Standard,\/ \today} }
\marginparwidth 0pt
\oddsidemargin=.25in
\evensidemargin  .25in
\marginparsep 0pt
\topmargin=-.5in
\textwidth=6.0in
\textheight=9.0in
\parindent=2em

%   ----------------------------------------------------------------------
%   mpi-macs.tex  --- man page macros,
%                discuss, missing, mpifunc macros
%
% ----------------------------------------------------------------------
% a couple of commands from Marc Snir, modified S. Otto

\newlength{\discussSpace}
\setlength{\discussSpace}{.7cm}

\newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace}
}

\newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace}
}

\newcommand{\implement}[1]{\vspace{\discussSpace}
{\small {\bf Implementation note:} #1} \vspace{\discussSpace} }

\newlength{\codeSpace}
\setlength{\codeSpace}{.3cm}

\newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} }

% -----------------------------------------------------------------------
%  A few commands to help in writing MPI man pages
%
\def\twoc#1#2{
\begin{list}
{\hbox to95pt{#1\hfil}}
{\setlength{\leftmargin}{120pt}
 \setlength{\labelwidth}{95pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#2}
\end{list}
}
\outer\long\def\onec#1{
\begin{list}
{}
{\setlength{\leftmargin}{25pt}
 \setlength{\labelwidth}{0pt}
 \setlength{\labelsep}{0pt}
 \setlength{\partopsep}{0pt}
 \setlength{\parskip}{0pt}
 \setlength{\topsep}{0pt}
}
\item
{#1}
\end{list}
}
\def\manhead#1{\noindent{\bf{#1}}}

\hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple}

\begin{document}

\setcounter{page}{1}
\pagenumbering{roman}

\title{ {\em D R A F T} \\ Maps, Groups and Contexts }

\author{
Lyndon Clarke, Mark Sears, Tony Skjellum, Marc Snir}

\date{May 13, 1993}

\maketitle
\hfuzz=5pt
%\tableofcontents

%\begin{abstract}
%We don't have an abstract yet.
%\end{abstract}

\setcounter{page}{1}
\pagenumbering{arabic}
\section{Introduction}

Goals

Assumptions (closed universe -- to be extended later)

Main objects: Maps and communicators

\section{Maps}

A {\bf map} is a one-to one-mapping $f: 0...m-1 \rightarrow 0...n-1$.  A
possible representation for such map is a list of $m$ integers in the range
$0...n-1$; it is often convenient to think of a map as being such list.
Another possible representation is as an algorithmic specification of
the map (e.g., hash function).  We represent a map by an opaque map object.

A {\bf group} is an ordered set of processes (where
a process is a primitive of MPI discussed elsewhere).
Each process in a group is
associated with a {\bf rank}, starting from zero.
We shall use maps in order to represent groups.
If processes
are numbered from $0$ to $n-1$, then a map $f: 0...m-1 \rightarrow 0...n-1$
represents a subset group of $m$ processes.  Note that a map, by itself, does
not represent a group; it does so only relative to a pre-defined
containing group.

\section{Context}
A {\bf context} is the MPI mechanism for partitioning communication
space.  A send made in a context cannot be received in another context.
Contexts are identified in MPI using {\bf context\_id}'s.

\section{Context ID's}
A {\bf context\_id} is a integer-like type that can be shared between
processes in MPI.

\section{Communicators}
All MPI communication (point to point and collective)
functions use {\bf communicators} {this replaces the word {\bf context}
everywhere in
current pt2pt and collcomm drafts} to provide a context
for the communication.  Communicators
bring together the concepts of group and context.

The source and destination
of a message is identified by the rank of that process within the group, and
communication using this communicator is restricted to processes within this
group.
Thus, the communicator restricts the ``spatial'' scope of communication,
and provides local process addressing.

\discuss{
The current communicators correspond to C1 objects in the previous draft.
Future extensions are expected to include C2 objects, that encapsulate two
groups: the sender group and the receiver group.
}

Communicators are represented by opaque {\bf communicator objects}.  Such
objects are opaque, and cannot be transferred from one node to another.
However, communicator objects can be identified
using {\bf context\_id}'s; these can be
communicated across processes.

\discuss{ Is the datatype of context\_id's {\tt INTEGER} or MPI-defined?}

Let {\tt comm} be a communicator that is associated with a group $G$ of size
$n$, and let $f:0..m-1 \rightarrow 0..n-1$ be a map.  Then
the pair {\tt (comm, f)}
defines a subgroup {\tt G(comm,f)} of $G$, namely the subgroup of processes
with ranks $f(0),...,f(m-1)$.
We say that {\tt map} is the map of the subgroup
{\tt G(comm,f)} within {\tt comm}.

An initial communicator {\tt MPI\_COMM\_INIT}
is defined when the program starts.
its associated group contains all processes that start the computation.

\section{Operations on maps}

\mpifunc{domain = MPI\_MAP\_DOMAIN(map)}

\begin{description}
\item[IN map] handle to map object
\item[OUT domain] domain of map (integer) -- the domain is the set $0...{\tt
domain} -1$.
\end{description}

\mpifunc{element = MPI\_MAP\_ELEMENT( map, rank)}

\begin{description}
\item[IN map] handle to map object
\item[IN rank] map argument (integer)
\item[OUT element] value of map  on argument {\tt rank} (integer)
\end{description}

\mpifunc{rank = MPI\_MAP\_RANK( map, element)}

\begin{description}
\item[IN map] handle to map object
\item[IN element] element in range of map
\item[OUT rank] inverse image of element, MPI\_INVALID\_ELEMENT
results if element is not in
range
\end{description}

\mpifunc{MPI\_MAP\_FLATTEN(map, array\_of\_integer, domain)}

\begin{description}
\item[IN map] handle to map object
\item[OUT array\_of\_integer]  list of values in the range of {\tt map}
\item[OUT domain] length of returned list -1 (integer).
\end{description}

\discuss{Definition of ``mpi\_map\_flatten'' is not definitive.}

\section{Map constructors}
The map constructors may be either local or may require communication.

\subsection{Local map constructors}

The execution of the following operation does not require interprocess
communication.  Examples of local map constructors are as follows:

\mpifunc{MPI\_MAP\_BUILD(map, array\_of\_integer)}

\begin{description}
\item[OUT map] handle to map object
\item[IN array\_of\_integer]  list of values in the range of {\tt map}
\item[IN domain] length of returned list -1 (integer).
\end{description}

\discuss{Syntax may change, (see also ``flatten'')}

\mpifunc{MPI\_MAP\_CONCAT(map1, map2, mapout)}
\begin{description}
\item[IN map1] map object handle
\item[IN map2] map object handle
\item[OUT mapout] map object handle
\end{description}
If map1 and map2 are distinct, then is the concatenation of
the maps.  $f(i) = f_1(1) \rm{if} i \le {\rm dom}(f_1), {\rm else}
= f_2(i-{\rm dom}(f_1))$.

\subsection{Collective map constructors}

The execution of the following operations require collective communication
within a group.   Examples are as follows:

\mpifunc{MPI\_MAP\_SPLIT(comm, mapin, key, index, mapout)}

This is a collective function that is called by all processes in the group
associated with  {\tt (comm, mapin)}.
All calling processes provide the same
values for the parameters {\tt comm} and {\tt mapin}.   A separate group of
processes is formed for each distinct value of {\tt key}, with the processes
ordered according to the value of {\tt index}.
Each process in such group is
returned in {\tt mapout} the map of this group within {\tt comm}.

\begin{description}
\item[IN comm] communicator object handle
\item[IN mapin] map object handle
\item[IN key] (integer)
\item[IN index] (integer)
\item[OUT mapout] map object handle
\end{description}

\section{Operations on communicators}

\mpifunc{MPI\_ALLOC\_CONTEXT(comm, map, array\_of\_contextids, len)}

Allocates an array of contextids.  This is a collective operation that is
executed by all processes in the group defined by {\tt (comm, map)}.  The
context\_ids that are returned are unique within the group associated with
{\tt (comm, map)}.

The array returned is the same on all processes that call the function
(same order, same number of elements).

\begin{description}
\item[IN comm] communicator object handle
\item[IN map] map object handle
\item[OUT array\_of\_contextids]
\item[IN len] number of context\_id's to allocate
\end{description}

\mpifunc{MPI\_MAKE\_COMM(comm, map, context\_id, newcomm)}

Creates a new communicator object, which is associated with the group defined by
{\tt (comm, map)}, and the contextid {\tt contextid}.  The operation does not
require communication.
However, the new communicator object should not be used
for communication between two processes unless they both have called the
function.

It is erroneous to invoke this function twice in the same process
with the same context\_id (unless the context\_id has been freed).

\discuss{Mark Sears disagrees with this!}

\begin{description}
\item[IN comm] communicator object handle
\item[IN map] map object handle
\item[IN contextid] contextid
\item[OUT newcomm] communicator object handle
\end{description}

\mpifunc{MPI\_SAFEMAKE\_COMM(comm, map, newcomm)}

is

\begin{verbatim}
MPI_ALLOC_CONTEXT(comm, map, context_id, 1);
MPI_MAKE_COMM(comm, map, context_id, newcomm);
``MPI_SYNCH(comm,map)''  /* by which we mean a synchronization operation
                       in the subgroup defined by (comm,map) */
\end{verbatim}

\discuss{ {\tt MPI\_SYNCH(comm,map)} is important to bootstrap -- but can be
coded using point-to-point.}

{\tt MPI\_SAFEMAKE\_COMM} creates a new communicator
object and synchronizes all
processes in the group associated with this object.  Thus, when this call
returns, the new communicator can be safely used.

\mpifunc{MPI\_MAP(comm, subcomm, map)}

Returns a map such that {\tt (comm, map)} is the group associated with {\tt
subcomm} ({\em i.e.},
returns the indices in the group associated
with {\tt comm} for all the processes
in the group associated with {\tt subcomm}).

\begin{description}
\item[IN comm] communicator object handle
\item[IN subcomm] communicator object handle
\item[OUT map] map object handle
\end{description}

\mpifunc{MPI\_CONTEXTID(comm, contextid)}

Returns the context\_id associated with the communicator {\tt comm}.

\begin{description}
\item[IN comm] communicator object handle
\item[OUT contextid] contextid
\end{description}

\discuss{Possible addition of functions are as follows:
{\tt MPI\_PARENT(comm)} returns the communicator used to create
{\tt comm};
}

\end{document}

>From [email-masked] Wed Jun  9 15:10:57 1993
Date: Wed, 9 Jun 93 08:06:42 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: May 12 mpi mail #6
Status: R

A postscript file from snir or otto (I think this is different from the LaTeX
source I forwarded just before this one...)

%!PS-Adobe-2.0
%%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software
%%Title: PT2PT-V3.DVI.*
%%Pages: 29 1
%%BoundingBox: 0 0 612 792
%%EndComments
%%BeginProcSet: texc.pro
/TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch
load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{
isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale
Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get
round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10
N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{
/vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{
statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N
/FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin
/FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array
/BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2
array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}
B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont
setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup
length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{
ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B
/ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0
N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S
dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0
ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice
ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N
/rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub
/rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod
S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index
getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw
cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and
or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255
eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2
index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval
putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv
1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{
adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1
add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{
/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S
ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr
ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI
save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook
known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if
/VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1
string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N
/BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V
statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval
(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale
rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex
ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B
/delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}
B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B
/k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1
w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{
/SS save N}B /eos{clear SS restore}B end
%%EndProcSet
TeXDict begin 1000 300 300 @start /Fa 6 54 df<EAFFE0A3EAE000B3B3B3B3B3A3EAFFE0
A30B63768116>20 D<EAFFE0A31200B3B3B3B3B3A312FFA30B63808116>I<EAFFFCA3EAE000B3
B3B3B10E4A73801C>50 D<EAFFFCA3EA001CB3B3B3B10E4A80801C>I<12E0B3B3B3B1EAFFFCA3
0E4A73811C>I<131CB3B3B3B1EAFFFCA30E4A80811C>I E /Fb 1 4 df<12041206A2EAC660EA
F7E0EA3F80EA1F00EA3F80EAF7E0EAC660EA0600A212040B0D7E8D11>3
D E /Fc 2 36 df<1560A28181A2150EB71280A2C8EA0E001518A25D5DA2210E7E8F26>33
D<1203B3A7EA8304EAE31CEA7B78EA1FE0EA0FC0EA0780EA03007E0E217E9913>35
D E /Fd 2 63 df<127012F8A3127005057D840C>58 D<12E01278121EEA0780EA01E0EA007813
1EEB0780EB01E0EB0078141EEC0780A2EC1E001478EB01E0EB0780011EC7FC1378EA01E0EA0780
001EC8FC127812E019187D9520>62 D E /Fe 4 107 df<B61280A219027D8A20>0
D<EA07E0EA1FF8EA3FFCEA7FFEA2B5FCA6EA7FFEA2EA3FFCEA1FF8EA07E010107E9115>15
D<12C012F0123C120FEA03C0EA00F0133C130FEB03C0EB00F0143C140FEC0380EC0F00143C14F0
EB03C0010FC7FC133C13F0EA03C0000FC8FC123C12F012C0C9FCA7B61280A219227D9920>21
D<12C0B3B3A502297B9E0C>106 D E /Ff 36 122 df<1238127C127EA2123E121EA2127C12F8
1260070A7A8414>44 D<127012F8A312700505798414>46 D<EA07C0EA0FE0EA1FF0EA3C78EA38
38EA701CA2EAE00EA8EAF01EEA701CA2EA3838EA3C78EA1FF0EA0FE0EA07C00F177E9614>48
D<12035AA25A123FB4FC12F712471207ACEAFFF0A30C177C9614>I<EA01C0487EA21360A2EA07
70A4EA0630EA0E38A5EA1FFCA3EA1C1CEA3C1E38FE3F80A311177F9614>65
D<EAFFF813FE7FEA1C0FEB0380A4EB0700133FEA1FFEA27FEA1C07EB0380A41307EB0F00B5FC5B
5B1117809614>I<EA03C6EA0FFE121FEA3E3EEA3C1EEA780E127012F0EAE000A7EAF00E127012
78EA3C1EEA3E3CEA1FF8EA0FF0EA03C00F177E9614>I<EAFFE013F87FEA383E131E7F7FA2EB03
80A7130714005B131E133EEAFFFC5B13E011177F9614>I<B51280A3EA1C03A490C7FC1338A2EA
1FF8A3EA1C38A290C7FCEB01C0A4B5FCA31217809614>I<B51280A3EA1C03A490C7FC1338A2EA
1FF8A3EA1C38A290C7FCA5B47EA311177F9614>I<EA03C6EA0FFE121FEA3E3EEA3C1EEA780E12
7012F048C7FCA4EB7F80A338F00E001270EA781E123CEA3E3EEA1FFE120FEA03CE11177F9614>
I<38FE3F80A338380E00A7EA3FFEA3EA380EA738FE3F80A311177F9614>I<EAFFFEA3EA0380B1
EAFFFEA30F177E9614>I<EAFE7FA3EA383C137813F0A2EA39E0EA3BC01380EA3FC0A213E0123E
EA3CF0EA387013781338133CA2EAFE3FA310177F9614>75 D<EAFFC0A3001CC7FCADEB0380A4B5
FCA311177F9614>I<38FE0FE0A3383B1B80A413BBA2EA39B3A313F3EA38E3A21303A538FE0FE0
A31317809614>I<38FE3F80A3383B0E00A4138E1239A213CEA31238A213EE136EA4EAFE3EA311
177F9614>I<EA1FF0EA7FFCA2EA701CEAF01EEAE00EADEAF01EEA783CEA7FFCA2EA1FF00F177E
9614>I<EAFFF813FE7FEA1C0FEB07801303A41307EB0F00EA1FFF5B13F8001CC7FCA6B47EA311
17809614>I<EA1FF0EA7FFCA2EA783CEAF01EEAE00EACEAE3CEEAF3DEEA79FC127FA2EA1FF0EA
00781338133C131C131E0F1C7E9614>I<EAFFE07F7FEA383C7F130EA3131E5BEA3FF85B7FEA38
3C131CA41480EB1DC0EAFE1F130FEB070012177F9614>I<EA0FCCEA3FFC127FEAF87CEAF03CEA
E01CA2EAF000A2127EEA3FE0EA0FF8EA01FCEA003C131E130E12E0A2EAF01EEAF83CEAFFFC13F8
EAC7E00F177E9614>I<B51280A3EAE1C3A43801C000ADEA0FF8A311177F9614>I<38FF8FF8A338
1C01C0AEEA1E03000E1380EA0F073807FF006C5AEA00F81517819614>I<38FE3F80A338380E00
A26C5AA56C5AA4EA0630EA0770A3EA0360A213E0A26C5A11177F9614>I<38FE0FE0A338380380
A50018130013E3EA1DF7A213B7A5EA0DB6A21316EA0F1EA31317809614>I<38FE3F80A3383C1E
00EA1C1CEA1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>89
D<EAFFE0A3EAE000B3A5EAFFE0A30B1D799914>91 D<EAFFE0A31200B3A512FFA30B1D7F9914>
93 D<EA1FC0EA7FF07FEA783CEA301CEA00FC120F123FEA7F1C12F012E0133CEAF07C387FFF80
A2EA1F8F11107E8F14>97 D<EA07E0EA1FF8EA3FFCEA7C3CEA781EEAF00EEAFFFEA3EAE000EAF0
0E1278EA7E1EEA3FFCEA0FF8EA03E00F107E8F14>101 D<1203EA0780A2EA0300C7FCA4EAFF80
A31203AAEAFFFEA30F187D9714>105 D<EAFCF8EAFFFE7FEA1F0FEA1E07121CA838FF9FE0A313
10808F14>110 D<EAFCFCB47E7F381F0780EA1E03001C13C01301A4EA1E03EB0780EA1F0FEBFF
005BEA1CF890C7FCA5B47EA31218808F14>112 D<1207A5EAFFFCA3EA0700A6130EA3139E13FC
EA03F8EA00E00F157F9414>116 D<38FF3F80A3381C1C00A2120E5BA212071330A2EA0370A26C
5AA35BA3EA7B80127F90C7FC127E123C11187F8F14>121 D E /Fg 29 86
df<1230127812F81278127005057D840C>46 D<13181370EA03F0120FEA1C701200A213E0A6EA
01C0A6EA0380A61207EAFFF8A20D1C7C9B15>49 D<137E3801FF80380387C0EA0603EB01E0120F
A2121FA2120E1200EB03C0A2EB0780EB0F00130E5B5B13605BEA038038070180120E381C030012
30EA7FFF485AA2131C7E9B15>I<137C48B4FC38038780EA0603000F13C0A3380E078012001400
130E133CEA01F8485AEA001C7F130FA4127012F8A2EAF01E12E05BEA7078EA3FE0EA0F80121D7D
9B15>I<EB01C0EB03801307130FA2131B1333EB770013E713C7EA0187EA03071207EA0E0E120C
121812301260387FFFE0B512C038001C00A5133C3803FF80A2131C7E9B15>I<380601C03807FF
80140013FC0006C7FCA25AA413F8EA0FFCEA1F0EEA1C0F7F121800001380A3EB0F00127012F013
0EEAE01EEAC01CEA6038EA70F0EA3FE0EA0F80121D7D9B15>I<1218383FFFC0A21480EA700138
6003001306485AC65A5BA25B5B12015B1203A248C7FCA25A120EA2121EA35AA31218121D7B9B15
>55 D<137C48B4FC38038780EA0703120EEB01C0A2381E0380A2EB0700EA0F8613DCEA07F81203
487EEA0E7E487EEA380F12707F12E0A31306130EEA701CEA3878EA1FF0EA0FC0121D7D9B15>I<
13F8EA03FCEA070EEA0E07121CA2003C138012381278A4130F1400A2EA381F5BEA1FEFEA0FDEEA
001E131CA2EA603812F05B485AEAC3C0EA7F80003EC7FC111D7C9B15>I<14301470A214F08013
01A2EB0378A21306147CEB0C3CA21318A21330A2497EA2EBFFFEA23801801EA23803001F801206
120F397FC0FFF012FF1C1D7F9C1F>65 D<380FFFFC14FF3900F00780EC03C0A2EC01E0A2D801E0
13C01403A2EC0780EC1F00EBFFFC5AEBC03E140FEC0780A3EA0780A4EC0F00141E380F007CB55A
14E01B1C7E9B1D>I<903807F030EB1FFC90387C0E603901F003E0EA03C038078001EA0F00000E
1300001E14C05AA25A1500A25AA5EC0180EC03001270127814066C5B001C131C000F5B3807C0E0
6CB45AD8007EC7FC1C1E7C9C1E>I<380FFFFCECFF803900F007C0EC01E0EC00F015701578485A
1538A3153C1538485A1578A3157015F0484813E0140115C0EC03801407EC0E00380F007CB512F0
14C01E1C7E9B20>I<000FB512E0A23800F003EC01C01400A33801E060A31500EBE1E013FF485B
13C113C0A215C0ECC18038078001A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>I<000F
B512C0A23800F007EC03801401A3EA01E01461A2EC600014E013E148B45AA213C113C0A3380781
800180C7FCA5120FEAFFFC5B1A1C7E9B1B>I<903807F030EB1FFC90387C0E603901F003E0EA03
C038078001EA0F00000E1300001E14C05AA25A1500A25AA3ECFFF0A2EC0F80150012701278A27E
001C5B000F133E3807C0F63803FFC2C66CC7FC1C1E7C9C21>I<390FFF9FFE143F3900F003C0A5
3901E00780A590B5FC481400EBC00FA53807801EA6000F133E39FFF3FFE015C01F1C7E9B1F>I<
380FFF80A23800F000A5485AA6485AA6485AA6120FEAFFF8A2111C7F9B0F>I<3801FFF0A23800
0F00A5131EA65BA65BA2127012F8A2485A485AEAE1C0EA7F80001FC7FC141D7E9B15>I<390FFF
87FEA23900F003E0EC0180EC030014065C48485A14705CEBE1C013E313E73803CDE013D9EBF1F0
13E013C01478EA078080A280A280000FEB1F8039FFF87FF013F01F1C7E9B20>I<380FFFC0A238
00F000A5485AA6485AA41406140CEA0780A2141C141814381478380F01F8B512F0A2171C7E9B1A
>I<D80FFCEB0FFCA20000EC1F80A2153713DE1567D8019EEB6F0015CFA2EC018F138FEC030FD8
030F131E1406A2EB078CA2149800065C14B0A2EB03E0A214C0001E147C3AFFE387FFC001C31480
261C7E9B26>I<390FF80FFEA23900FC01E0EC00C0A213DEA239018F0180A214811387A2EB83C1
390303C300EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38FFE00C13C01F1C7E9B1F>
I<EB07F0EB3FFCEBF81E3801E0073903C00380D8070013C0481301001E14E0121C003C13001238
007814F0A348EB01E0A4EC03C0A30070EB0780007814005C6C131E5C6C5B380F83E03803FF80C6
48C7FC1C1E7C9C20>I<380FFFFC14FF3900F00F80140315C01401A23801E003A315801407EC0F
003803C03EEBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B1C>I<380FFFF814FE3900
F00F80140315C01401A23801E003A3EC0780EC0F00143E3803FFF85CEBC07880A2143E3807803C
A51518000F1430EAFFF89038F01FE0C7EA07C01D1D7E9B1F>82 D<EB3F0CEBFFCC3801C1F83803
0078481338120EA3001E13301400A26C7E13F86CB4FC6C13806C13C038003FE0130313011300A2
1260A3EB01C0127038F8038038FE0F00EACFFEEA83F8161E7E9C17>I<001FB512F05A383C0781
0038EB806012701260A2EB0F0012C0A200001400A3131EA65BA6137C381FFFE0A21C1C7C9B1E>
I<39FFF8FFE0A2390F001E00140CA4001E5BA6485BA6485BA300385BA2383C0180D81C03C7FCEA
0E0EEA07FCEA01F01B1D7A9B1F>I E /Fh 53 122 df<90381FE1F890B512FC3903F03F3E3807
C07EEB807C000F141C1500A5B612C0A2390F807C00AE397FE1FFC0A21F1D809C1C>11
D<EB1FFC13FF3803F07CEA07C01380120FA6B512FCA2380F807CAE397FF3FF80A2191D809C1B>
13 D<13301360EA01C0EA038013005A120E121E121C123CA212381278A312F85AA97E1278A312
38123CA2121C121E120E7E7E1380EA01C0EA006013300C297D9E13>40 D<12C0126012387E120C
120E7E1380120313C0A2120113E0A313F01200A9120113E0A313C01203A2138012071300120E12
0C121C5A12605A0C297D9E13>I<1238127C12FE12FFA2127F123B120312071206A2120C121812
301220080F7D860D>44 D<EAFFE0A40B047F8A10>I<1238127C12FEA3127C123807077D860D>I<
EA0FF0EA3FFCEA703EEAF03F12F8A3EA203EEA007C13F813F0EA01C0A21380A5C7FCA4EA0380EA
07C0EA0FE0A3EA07C0EA0380101D7D9C17>63 D<EB0380A2497EA3497EA2497E131BA2EB31F8A2
EB61FC1360EBE0FEEBC07EA248487E90B5FC481480EB001FA20006EB0FC0A2000E14E0000C1307
39FFC07FFEA21F1C7E9B24>65 D<B512F814FF390FC01F80EC0FC0A215E0A515C0EC1F8090B512
005C9038C03F80EC0FC0EC07E0A215F0A515E0140FEC1FC0B61280ECFC001C1C7E9B22>I<9038
1FE0209038FFF8E03803F80F3807E007380FC001EA1F8048C7FCA2007E1460A212FE1500A7007E
1460A27E15C06C7E390FC001803907E003003803F80E3800FFFCEB1FE01B1C7D9B22>I<B512F8
14FF390FC01FC0EC07E0EC03F0EC01F8A2EC00FCA315FEA815FCA3EC01F8A2EC03F0EC07E0EC1F
C0B6120014F81F1C7E9B25>I<B6FCA2380FC01F1407801580A2EBC181A3EC800013C313FFA213
C313C1A21560A29038C000E015C0A21401A21403EC0F80B6FCA21B1C7E9B1F>I<B6FCA2380FC0
1F1407801580A21401EBC181A2EC8000A213C313FFA213C313C1A401C0C7FCA6B5FCA2191C7E9B
1E>I<90380FF00890387FFE383901FC07783907E001F8380FC0004848137848C71238A2481418
127E12FE1500A5EC7FFFA2007EEB01F8127F7EA2EA1F80EA0FC0EA07F03801FC0739007FFE7890
380FF818201C7D9B26>I<39FFFC3FFFA2390FC003F0AA90B5FCA2EBC003AC39FFFC3FFFA2201C
7E9B25>I<B5FCA2EA07E0B3A6B5FCA2101C7F9B12>I<B5FCA2EA0FC0AF1403A31407A21406140E
141E147EB512FEA2181C7E9B1D>76 D<D8FFC0EB03FF6D5B000F15F0D80DF0130DA3D80CF81319
A2017C1331A26D1361A26D13C1A390380F8181A2903807C301A2EB03E6A2EB01FCA3EB00F8A23A
FFC0701FFFA2281C7E9B2D>I<39FFE003FFA2390FF000307FEA0DFCEA0CFE137E7FEB1F8014C0
EB0FE0EB07F01303EB01F814FCEB00FE147F143FEC1FB015F0140F1407140314011400A2D8FFC0
13701530201C7E9B25>I<EB3FE03801FFFC3803F07E390FC01F80391F800FC0EB00074814E048
14F0007E1303A200FE14F8A8007E14F0A2007F13076C14E0EB800F001F14C0390FC01F803903F0
7E006CB45A38003FE01D1C7D9B24>I<B512F814FE390FC03F80141FEC0FC015E0A615C0EC1F80
143F9038FFFE0014F801C0C7FCAAEAFFFCA21B1C7E9B21>I<EB3FE03801FFFC3803F07E390FC0
1F80391F800FC0A2393F0007E04814F0007E1303A200FE14F8A8007E14F0A2007F1307D83F0F13
E0EB9FCF391FB0EFC0390FF07F800003EB7E006CB45A39003FF808EB003CEC3FF8A2141F15F014
0FEC07E0EC03C01D247D9B24>I<B512F014FE380FC03F6E7E6E7E81A55D4A5A4AC7FCEBFFFE14
F0EBC0F8147C80143FA381A3EDC180EC1FE33AFFFC0FFF00EC03FE211C7E9B24>I<3807F86038
1FFEE0EA3C07EA7801EA700012F01460A26C130012FEEAFFE0EA7FFE6C7E1480000F13C06C13E0
EA007FEB03F01301130012C0A214E07E38F001C0EAFC0338EFFF00EAC3FC141C7D9B1B>I<007F
B512E0A238781F81007013800060146000E0147000C01430A400001400B03807FFFEA21C1C7E9B
21>I<39FFFC03FFA2390FC00030B3120715606C6C13E03901F001C03900FC078090387FFE00EB
0FF8201C7E9B25>I<3AFFFC01FF80A23A0FC00018006C6C5BA26D1370000314606D13E000015C
7F0000495AA2D97E03C7FCA2EB7F07EB3F06148EEB1F8C14CCEB0FD8A2EB07F0A36D5AA26D5AA2
211C7F9B24>I<3BFFFC7FFE0FFCA23B0FC007E000C081D9E003130100071680EC07F8D803F0EC
0300A29039F80CFC0700011506EC1CFE9039FC187E0E0000150CEC387F90397E303F18A290397F
601FB8013F14B002E013F0ECC00F011F5CA26D486C5AA2EC00036D5CA22E1C7F9B31>I<397FFE
1FFEA23907F001803803F8036C6C48C7FC000013066D5AEB7F1C6D5A14B0EB1FE0130FA26D7E6D
7E1307497EEB0CFEEB187EEB387F496C7EEB601F01C07F00016D7E496C7EEA03003AFFF03FFF80
A2211C7F9B24>I<3AFFFC01FF80A23A0FE00038006C6C13306C6C137015606C6C5B3800FE015D
D97F03C7FCEB3F871486EB1FCEEB0FFC5C13076D5AAAEB3FFFA2211C7F9B24>I<EA0FF8EA1FFE
383E1F80130714C0121C1200EA03FF121FEA3F87EA7E0712FCA3130FEA7E1F383FFBF8EA0FE115
127F9117>97 D<B4FCA2121FA9EB1FC0EBFFE0EBC1F81300147CA2147EA6147CA2EB80F8EBC1F0
381C7FE038181F80171D7F9C1B>I<EA03FCEA0FFEEA1F1F123E127C130E00FCC7FCA6127C387E
0180EA3E03381F0700EA0FFEEA03F811127E9115>I<EB0FF0A21301A9EA03F1EA0FFFEA1F07EA
3E01127CA212FCA6127CA2EA3E03EA1F0F380FFDFEEA03F1171D7E9C1B>I<EA01FCEA0FFF381F
0F80383E07C0EA7C0314E012FCB5FCA200FCC7FCA3127C007E1360003E13C0EA1F81380FFF00EA
01FC13127F9116>I<133F3801FF803803E7C0EA07C7EA0F87EB8380EB8000A5EAFFF8A2EA0F80
AEEA7FF8A2121D809C0F>I<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E
1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F038
3E03E0381FFFC03803FE00151B7F9118>I<B4FCA2121FA9EB1FC0EB7FE0EBE1F0EB80F8A21300
AB38FFE7FFA2181D7F9C1B>I<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F
9D0E>I<B4FCA2121FA9EB0FF8A2EB0380EB0700130C5B137013F87F13BC133E7F7F1480EB07C0
EB03E038FFC7FCA2161D7F9C19>107 D<B4FCA2121FB3A7EAFFE0A20B1D7F9C0E>I<39FF1FC0FE
90387FE3FF3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1F
C0EB7FE0381FE1F0EB80F8A21300AB38FFE7FFA218127F911B>I<EA01FC380FFF80381F07C038
3E03E0387C01F0A200FC13F8A6007C13F0A2383E03E0381F07C0380FFF803801FC0015127F9118
>I<38FF1FC0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090
C7FCA6EAFFE0A2171A7F911B>I<EAFE3E137F381ECF80EA1F8FA2EB070090C7FCAAEAFFF0A211
127F9114>114 D<EA1FD8EA3FF8EA7038EAE018A2EAF000EAFF80EA7FE013F0EA1FF8EA07FCEA
007CEAC01CA212E0EAF038EAFFF0EACFC00E127E9113>I<1203A35AA25AA2123FEAFFFCA2EA1F
00A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA
03F818127F911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01
F6A213FE6C5AA21378A2133016127F9119>I<38FFC7FCA2381F8180EA0F833807C700EA03EEEA
01FC5B1200137C13FEEA01DFEA039F38070F80380607C0380C03E038FF07FCA216127F9119>
120 D<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C5A
A21378A21330A25B1270EAF8E0EAC0C0EAE380007FC7FC123E161A7F9119>I
E /Fi 74 126 df<126012F0AF12601200A4126012F0A212600419779816>33
D<EA03C0487E120FEA1EF0EA1C70A3EBF7E013E7121D381FC7006C5A130E121FEA3F1EEA7F9CEA
F3BCEAE3F812E1EBF1C012E0EAF1F9EA7FFF383F9F80381F0F0013197F9816>38
D<13E01201EA07C013005A121E5A123812781270A312F05AA77E1270A312781238123C7E7E7E13
C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780120313C01201A313E01200A7
120113C0A3120313801207EA0F00121E121C127C12F05A0B217C9C16>I<EA01C0A4EA71C738F9
CF80387FFF00EA1FFCEA07F0A2EA1FFCEA7FFF38F9CF803871C700EA01C0A411127E9516>I<EA
01C0A8B51280A33801C000A811137E9516>I<1238127C127EA2123E120E121E123C127C12F812
60070B798416>I<B51280A311037E8D16>I<127012F8A312700505788416>I<EA03E0EA0FF848
7EEA1E3CEA380EEA780FEA7007A238E00380A8EAF00700701300A2EA780FEA3C1E6C5AEA1FFC6C
5AEA03E011197E9816>48 D<EA018012031207A2121F127F12FF12731203AEEA7FF813FC13F80E
197C9816>I<EA07E0EA1FF8EA7FFEEA783FEAF00FEB07801303A21200A2130714005B131E5B5B
5BEA03E0EA078048C7FC381E0380123CEA7FFFB5FC7E11197E9816>I<EA07E0EA1FFCEA7FFEEA
781F7F7F12005BA2133EEA07FC5B7FEA001E130714801303A212F013071400EAF81FEA7FFE6C5A
EA07E011197E9816>I<137C13FC13DC1201EA039CA2EA071C120F120E121E123C1238127812F0
B512E0A338001C00A53801FFC0A313197F9816>I<EA3FFEA30038C7FCA7EA3FF013FC7FEA3C0F
EA3007000013801303126012F01307EB0F006C5AEA7FFEEA1FF8EA07E011197E9816>I<13F8EA
03FEEA0FFFEA1F0F123E123CEA78060070C7FC12F0EAF7F8EAFFFEA2EAF80F38F00780A2EAE003
12F0A21270EA7807EB0F006C5AEA1FFEEA0FF8EA03E011197E9816>I<12E0B51280A338E00F00
131EC65A13381378137013F05B12015B12035BA3120790C7FCA7111A7E9916>I<EA07F0EA1FFC
487EEA7C1F38F00780EAE003A338700700EA7C1FEA1FFCEA07F0EA1FFCEA3C1EEA700700F01380
EAE003A3EAF00700701300EA7C1FEA3FFE6C5AEA07F011197E9816>I<EA03E0EA0FF8EA3FFCEA
7C1EEA780F487E12E01480A21303EAF007A2EA780FEA3FFFA2EA0FF3EA00071400EA300FEA780E
131E137CEA7FF8EA3FE0EA0F8011197E9816>I<1238127CA312381200A81238127CA3123C121C
123C123812F812F012600618799116>59 D<EB03801307131FEB3E0013FCEA03F0485AEA1F8000
3EC7FC12FC12F012FC123E6C7EEA07E06C7EEA00FC133EEB1F801307130311157E9616>I<EA7F
FFB51280A2C8FCA5B51280A26C1300110B7E9116>I<13E0487EA213B0A2EA03B8A31318EA071C
A5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C013197F9816>65
D<EA7FF8EAFFFE6C7EEA1C0FEB07801303A313071400EA1FFF5BA2EA1C1FEB038014C01301A413
03EB0780EA7FFFB51200EA7FFC12197F9816>I<3801F180EA07FF5AEA1F0FEA3C071278130312
7000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C5AEA01F011197E9816>I<EA7F
F8EAFFFE6C7EEA1C0FEB0780EB03C01301A214E01300A8EB01C0A21303EB0780130F387FFF0048
5AEA7FF81319809816>I<387FFFC0B5FC7EEA1C01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7
FC14E0A5EA7FFFB5FC7E13197F9816>I<B512E0A3EA1C00A41400A2131CA2EA1FFCA3EA1C1CA2
90C7FCA6B47E7F5B13197F9816>I<EA03E348B4FC121FEA3E1FEA3C0F12787F127000F0C7FC5A
A4EB3F80EB7FC0EB3F8038F007001270EA780FA2123CEA3E1F6CB4FC1207EA03E712197E9816>
I<387F1FC038FFBFE0387F1FC0381C0700A7EA1FFFA3EA1C07A9387F1FC038FFBFE0387F1FC013
197F9816>I<EAFFFEA3EA0380B3EAFFFEA30F197D9816>I<387F0FE038FF8FF0387F0FE0381C07
80EB0F00131E131C133C5B5BEA1DE07F121F7F1338EA1E3C131CEA1C1E7F7F14801303387F07E0
38FF8FF0387F07E01419809816>75 D<EAFFC0A3001CC7FCAE144014E0A4B5FCA313197F9816>
I<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313
197F9816>I<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA213E7A31367A21377
A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>I<EA1FFC487E487EEA780F38F00780EAE003
AEEAF007A238780F00EA7FFF6C5A6C5A11197E9816>I<EA7FF8EAFFFE6C7E381C0F80130314C0
1301A313031480130F381FFF005B13F8001CC7FCA7127F487E6CC7FC12197F9816>I<EA1FFC48
7E487EEA780F38F00780EAE003AD13E312E1EAF0F738787F00EA7FFF6C5A6C5AEA001C131E130E
130F7FA2111F7E9816>I<EA7FE0EAFFF86C7EEA1C1E7F7FA45B131EEA1FFC5B7FEA1C3E130EA4
14201470A2387F0FF038FF87E0387F03C014197F9816>I<EA07E3EA1FFF127FEA781F487E487E
A290C7FC7E1278EA7F80EA1FF0EA07FCC67E130FEB07801303A212E0A2EAF00738F80F00EAFFFE
5BEAC7F011197E9816>I<387FFFE0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816>
I<387F07F038FF8FF8387F07F0381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519
809816>I<38FE0FE0EAFF1FEAFE0F38380380381C0700A4EA0E0EA4EA060CEA071CA4EA031813
B8A3EA01B013F0A26C5A13197F9816>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803
A313E3EA39F3A213B300191300A61313EA1B1BEA0F1EA2EA0E0E13197F9816>I<387F1F80133F
131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B81207131CA2EA0E0EA2
487E387F1FC000FF13E0007F13C013197F9816>I<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0E
A26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I<EAFFF0A3EAE000B3A8EA
FFF0A30C20789C16>91 D<EAFFF0A3EA0070B3A8EAFFF0A30C207F9C16>93
D<B51280A311037E7E16>95 D<EA1FE0EA7FF87FEA783CEA301EEA000E133EEA07FE123FEA7FCE
EAF80E12E0A2EAF01EEAF83E387FFFE0EA3FF7EA0FC313127E9116>97 D<127E12FE127E120EA4
133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E01381EB83C013FF000E1300
EA067C1419809816>I<EA03F8EA0FFE121FEA3C1EEA780CEA700012F05AA47EEA70071278EA3E
0FEA1FFEEA0FFCEA03F010127D9116>I<133F5B7F1307A4EA03E7EA0FFF123FEA3C1F487E1270
EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E014197F9816>I<EA07E0EA0F
F8EA1FFCEA3C3EEA780EEA700FEAF007B5FCA3EAE0007EEA70071278EA3E1FEA1FFEEA0FFCEA03
F010127D9116>I<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FCA2EA01C0ACEA3FFE487E
6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3EA3C1E6C5AEA1FFC48
5AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3EAF001387C07C0383F
FF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0FFF148013871303A2120E
A9387FC7F038FFE7F8387FC7F01519809816>I<EA0180EA03C0A2EA0180C7FCA4EA7FC0A31201
ACEA7FFFB5FC7E101A7D9916>I<127E12FE127E120EA4EB7FE0A3EB0F00131E5B5B5BEA0FF8A2
13BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816>107
D<EAFFC0A31201B3B51280A311197E9816>I<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870
AA38FE7CF8A31512809116>I<EA7E7CB5FC6C1380EA0F871303A2120EA9387FC7F038FFE7F838
7FC7F01512809116>I<EA03E0EA0FF8487EEA3C1E487EEA700738E00380A5EAF00700701300EA
780FEA3C1EEA1FFC6C5AEA03E011127E9116>I<EA7E3E38FEFF80007F13C0380F83E01301EB00
F0120E1470A4000F13F014E01381EB83C013FF000E1300137C90C7FCA6EA7FC0487E6C5A141B80
9116>I<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F9116>114
D<EA0FECEA3FFC127FEAF03CEAE01CA2EAF000EA7F80EA1FF0EA07FCEA003EEAE00EA212F0EAF8
1EEAFFFC13F8EAC7E00F127D9116>I<12035AA4EA7FFFB5FCA20007C7FCA75BEB0380A2130713
873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C
13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4
EA03B8A3EA01F0A26C5A13127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813F01201
6C5A12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>120
D<387F1FC038FF9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0
A35BA3485A1279127BEA7F806CC7FC123C131B7F9116>I<EB1F80137FEA01FFEBE0005BA91203
EA7F8048C7FCA26C7EEA03C01201A97FEBFF80EA007F131F11207E9C16>123
D<12E0B3AE0320779C16>I<12FCB4FC13C012031201A97F6CB4FCEB7F80A2EBFF00EA01E05BA9
120312FF90C7FC12FC11207E9C16>I E /Fj 23 122 df<3801FFE0A238003C00A45BA45BA448
5AA4485AA4485AA4EAFFF0A2131C7E9B10>73 D<48B4EB03FE1507D8003F14C0150F151BA2016F
EB3780EB67801567A201C7EBCF00EC818FA2EC830FD80187131E1486A2148C390307983CA214B0
EB03F00006495AA214C0000E138039FFC38FFF140F271C7E9B25>77 D<3801FFFEECFF8039003C
03C0EC01E0A31378A315C0EBF0031580EC0700141E3801FFFC14F001E0C7FCA2485AA4485AA4EA
FFF0A21B1C7E9B1C>80 D<EA03CCEA07FCEA0E7CEA183CEA383812301270A2EAE070A3137113E3
A212E1EA63E6EA7E7CEA3C3810127B9115>97 D<EA01F0EA07F8EA0E18EA1C3CEA3878EA3030EA
7000A25AA513101338EA70F0EA3FC0EA1F000E127B9113>99 D<EB1F80133F13071400A4130EA4
EA03DCEA07FCEA0E7CEA183CEA383812301270A2485AA313715BA212E1EA63E6EA7E7CEA3C3811
1D7B9C15>I<EA01E0EA07F8EA0E18121C123812701370EA7FE0EAFF80EAE000A41310EA6038EA
70F0EA3FC0EA1F000D127B9113>I<EB03C0EB07E0EB0CF0EB1DE0EB1CC014005BA53803FF805A
38007000A45BA5485AA5485AA448C7FC126712F612FE5A12781425819C0D>I<EB798013FFEA01
CFEA0307000713001206120EA2EA1C0EA45BA2133CEA0C7CEA0FF8EA07B8EA0038A25B1260EAF0
E012F1EAFF806CC7FC111A7E9113>I<EA0FC0121F12035BA448C7FCA4EA0E78EA0FFC138E130E
121E121CA3485AA3EB3840007013C0A2EB7180A238E03F00EA601E121D7D9C15>I<13C0EA01E0
13C01380C7FCA6120E123FEA33801263EAC700A21207120EA35A1340EA38C0A3EA3980EA3F0012
1E0B1C7D9B0D>I<EA1F80123F12071300A4120EA45AA45AA45AA3127212E6A4127C1238091D7C
9C0B>108 D<391E1F07C0393F3F8FE0396761D8703863C1F038C781E0A2380701C0A2390E0380
E0A3EC81C2391C0701C6A2EC038CA239380E01F839180600F01F127D9122>I<EA1E1EEA3F7F38
67E380EA63C3EAC78313031207A2380E0700A3EB0E10001C1330A2EB1C60A238380FC038180780
14127D9117>I<EA01E0EA07F8EA0E1C121CEA380CEA300E1270A2EAE01CA31338A213701360EA
71C0EA3F80EA1E000F127B9115>I<EA0787380FDF803819F9C0EA18E0003113E013C01201A238
0381C0A31480EA070314001387138EEA0EFC137090C7FCA25AA4B47EA2131A7F9115>I<EA1E3C
EA3F7EEA67C3EA6387EAC78F130FEA07061300120EA45AA45A121810127D9112>114
D<EA01F0EA07F8EA0E18133CEA1C781330EA1E00EA1FC0EA0FE0EA07F01200EA2078EA707012F0
A2EAE0E0EA7FC0EA1F000E127D9111>I<13C01201A3EA0380A4EAFFE0A2EA0700A2120EA45AA3
1320EA3860A213C0A2EA1F80EA0F000B1A7D990E>I<380F0180EA1F8312331263EB870012C3EA
0707A2EA0E0EA31420380C1C60121C120C380E3CC0380FEF803803C70013127D9116>I<EA0F03
EA1F871233EA6383A212C3EA0703A2EA0E06A31304130C121CEA0E181330EA07E0EA03C010127D
9113>I<EA0F1EEA1FBFEA31E3EA61E713CF12C1EA01C613C0EA0380A3EA7381EAF703A2EAE706
EACF8EEA7DFCEA70F010127D9113>120 D<380F0180EA1F8312331263EB870012C3EA0707A2EA
0E0EA4EA0C1C121C120CEA0E3CEA0FF81203EA0038A2EA3070EA786013E0EA71C0EA3F80001EC7
FC111A7D9114>I E /Fk 42 122 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07
E0D80FC01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11
D<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA21201B3A7387FFFE0A2
13207C9F1C>49 D<EA03FCEA0FFF383C1FC0387007E0007C13F0EAFE0314F8A21301127CEA3803
120014F0A2EB07E014C0EB0F80EB1F00133E13385BEBE018EA01C0EA0380EA0700000E1338380F
FFF05A5A5AB5FCA215207D9F1C>I<EA01FE3807FFC0380F07E0381E03F0123FEB01F813811301
EA1F03000C13F0120014E0EB07C0EB1F803801FE007F380007C0EB01F014F8EB00FCA214FE127C
A212FEA214FCEA7C01007813F8383C07F0380FFFC03803FE0017207E9F1C>I<14E013011303A2
1307130F131FA21337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA2
380007E0A7EBFFFEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC000
18C7FCA4EA19FCEA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F8
386007E0003013C0381C1F80380FFF00EA03F815207D9F1C>I<13FE3803FFC0380703E0380E00
F05A1478123C123E123F1380EBE0F0381FF9E0EBFFC06C13806C13C06C13E04813F0381E7FF838
3C1FFCEA7807EB01FEEAF000143E141EA2141C7E007813387E381F01F0380FFFC0000113001720
7E9F1C>56 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB30
0FA201607F140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A2
25227EA12A>65 D<B67E15E03907F001F86E7E157EA2157FA5157E15FE5DEC03F890B55AA29038
F001FCEC007E811680151F16C0A6ED3F80A2ED7F00EC01FEB612F815C022227EA128>I<D903FE
138090381FFF819038FF01E33901F8003FD803E0131F4848130F48481307121F48C71203A24814
01127EA200FE91C7FCA8127EED0180127F7E15036C6C1400120F6C6C1306D803F05B6C6C13386C
B413F090381FFFC0D903FEC7FC21227DA128>I<B67E15F03907F003FCEC007E81ED1F80ED0FC0
ED07E0A216F01503A316F8A916F0A3ED07E0A2ED0FC0ED1F80ED3F00157EEC03FCB612F0158025
227EA12B>I<B612FCA23807F000153C151C150C150EA215061418A3150014381478EBFFF8A2EB
F07814381418A21503A214001506A3150EA2151E153EEC01FCB6FCA220227EA125>I<B612F8A2
3807F001EC007815381518151CA2150CA21418A21500A214381478EBFFF8A2EBF07814381418A4
91C7FCA8B512E0A21E227EA123>I<B53883FFFEA23A07F0001FC0AD90B6FCA29038F0001FAFB5
3883FFFEA227227EA12C>72 D<D8FFF0EC0FFF6D5C000716E0D806FC1437A3017E1467A26D14C7
A290391F800187A290390FC00307A3903807E006A2903803F00CA2903801F818A3903800FC30A2
EC7E60A2EC3FC0A2EC1F80A3EC0F00D8FFF091B5FC140630227EA135>77
D<D8FFF8EB1FFE7F0007EC00C07FEA06FF6D7E6D7E6D7E130F806D7E6D7E6D7E130080EC7F80EC
3FC0EC1FE0EC0FF0140715F8EC03FCEC01FEEC00FF157FA2153F151F150F15071503A2D8FFF013
01150027227EA12C>I<EB07FC90383FFF809038FC07E03903F001F848486C7E4848137E48487F
A248C7EA1F80A24815C0007E140FA200FE15E0A9007E15C0007F141FA26C15806D133F001F1500
6C6C137E6C6C5B6C6C485A3900FC07E090383FFF80D907FCC7FC23227DA12A>I<B6FC15E03907
F007F0EC01FC1400157EA2157FA5157EA215FC1401EC07F090B512E0150001F0C7FCADB57EA220
227EA126>I<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400
B4FC13E06CB4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA3
6C131EA26C133C12FCB413F838C7FFE00080138018227DA11F>83 D<007FB61280A2397E03F80F
00781407007014030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127
>I<EA07FC381FFF80383F0FC0EB07E0130314F0121E1200A213FF1207EA1FC3EA3F03127E12FC
A4EA7E07EB1DF8381FF8FF3807E07F18167E951B>97 D<B47EA2121FABEB8FE0EBBFF8EBF07CEB
C01EEB801FEC0F80A215C0A81580141F1500EBC03EEB607C381E3FF8381C0FC01A237EA21F>I<
EBFF80000713E0380F83F0EA1F03123E127E387C01E090C7FC12FCA6127C127EA2003E13306C13
60380FC0E03807FF803800FE0014167E9519>I<EB03FEA2EB007EABEA01FCEA07FF380F81FEEA
1F00003E137E127E127C12FCA8127CA27E001E13FEEA0F833907FF7FC0EA01FC1A237EA21F>I<
13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA2127E
003E13186C1330380FC0703803FFC0C6130015167E951A>I<EB3F80EBFFC03801F3E0EA03E7EA
07C7120FEBC3C0EBC000A6EAFFFCA2EA0FC0B2EA7FFCA213237FA211>I<3801FE1F0007B51280
380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC12
1CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6CB4
5A000113C019217F951C>I<B47EA2121FABEB87E0EB9FF8EBB8FCEBE07CEBC07EA21380AE39FF
F1FFC0A21A237EA21F>I<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A20C
247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC133E137EEA78
FCEA7FF0EA1FC0102E83A311>I<EAFF80A2121FB3ADEAFFF0A20C237FA20F>108
D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFF
F1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0
A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7007C137C
A26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEB
C03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A2
1A207E951F>I<EAFF1FEB3FC0381F67E013C7A3EB83C0EB8000ADEAFFF8A213167E9517>114
D<EA07F3EA1FFFEA780FEA7007EAF003A26CC7FCB4FC13F0EA7FFC6C7E6C7E120738003F80EAC0
0F130712E0A200F01300EAFC1EEAEFFCEAC7F011167E9516>I<13C0A41201A212031207120F12
1FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF
14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E000713
0CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E
>I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C6C5AA2EB
7EC0A2137F6D5AA26DC7FCA2130EA2130CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B20
7F951E>121 D E /Fl 71 124 df<EB7C3C3801FEFE38038FCF38070F8F000E1306EB0700A5B5
12F0A2380E0700AC387F0FF0A2181A809916>11 D<137CEA01FEEA0387485A120E130690C7FCA4
B5FCA2EA0E07AC387F0FE0A2131A809915>I<137F485AEA038FEA070FEA0E07A6B5FCA2EA0E07
AC387F9FE0A2131A809915>I<90383E0F803901FF7FC0390383F0E0380707C1000E1381903803
80C01500A4B612E0A2380E0380AC397F8FE3FCA21E1A809920>I<EA60C0EAF1E0EAF9F01278EA
1830A3EA3060EA70E0EAE1C0EA40800C0B7F9913>34 D<126012F012F812781218A31230127012
E01240050B7D990B>39 D<13C0EA0180EA03001206120E120C5A1238A212301270A21260A212E0
AA1260A21270A212301238A212187E120E12067EEA0180EA00C00A267E9B0F>I<12C012607E7E
121C120C7E1207A27E1380A21201A213C0AA1380A21203A213005AA212065A121C12185A5A5A0A
267E9B0F>I<126012F0A212701230A31260A212C01240040B7D830B>44
D<EAFFC0A30A0380880D>I<126012F0A2126004047D830B>I<130CA2131C1318A213381330A213
701360A213E013C0A212011380A2120313005A1206A2120E120CA2121C1218A212381230A21270
1260A212E05AA20E257E9B13>I<EA0780EA1FE0EA3870EA3030EA7038EA6018A2EAE01CACEA60
18A2EA7038EA3030EA3870EA1FE0EA07800E1A7E9813>I<12035AB4FCA21207B3EAFFF0A20C19
7D9813>I<EA0F80EA1FE0EA30F0EA6078EAF038EAF83CA2131C1270EA003CA213381370A213E0
EA01C0EA0380EA0700EA0E0C120C1218EA3018EA7FF812FFA20E197E9813>I<EA0F80EA1FE0EA
38F0EA7070EA7878133813781230EA0070A213E0EA0FC01380EA00E013701338133CA2127012F8
A2EAF038EA6078EA70F0EA3FE0EA0F800E1A7E9813>I<1260EA7FFEA213FCEA600CEAC0181330
A2EA006013C0A21201A2EA0380A41207A8EA03000F1A7E9813>55 D<EA0780EA1FE0EA3870EA30
30EA6038EAE018A2131CA5EA603C1270EA307CEA3FDCEA1F9CEA0018A21338EA3030EA78701360
EA71C0EA3F80EA1F000E1A7E9813>57 D<126012F0A212601200A8126012F0A2126004107D8F0B
>I<126012F0A212601200A8126012F0A212701230A31260A212C0124004177D8F0B>I<EA1F80EA
3FE0EA7070EAE03812F0A21260EA007013E0EA01C0EA03801300A21206A5C7FCA41206120FA212
060D1A7E9912>63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0A348B47EA213000006
1378A3487F121E39FF81FFC0A21A1B7F9A1D>65 D<B5128014E0380F00F01478143CA5147814F0
EBFFE0A2EB01F0EB0078143C141EA5143CA214F8B512F014C0171A7F991B>I<EB3F023801FFC6
3803E0EE3807003E000E131E5A003C130E123800781306127012F01400A6140612701278123800
3C130C121C6C13186C13303803E0E03801FFC038003F00171C7E9A1C>I<B57E14E0380F00F014
388080A2140F801580A81500A25C140E5C143C14F8B512E01480191A7F991D>I<B512F8A2380F
007814381418141C140C1303A214005B13FFA213077FA214061300140CA3141CA21478B512F8A2
171A7F991A>I<B512F0A2EA0F001470143014381418A213061400A2130E13FEA2130E1306A390
C7FCA6EAFFF8A2151A7F9919>I<EB3F023801FFC63803E0EE3807003E000E131E5A003C130E12
3800781306127012F091C7FCA5903803FFC0A23970001E0012781238123C121C7E3807803E3803
E0763801FFE2D8003FC7FC1A1C7E9A1E>I<39FFF3FFC0A2390F003C00A9EBFFFCA2EB003CAB39
FFF3FFC0A21A1A7F991D>I<EAFFF0A2EA0F00B3A4EAFFF0A20C1A7F990E>I<EA0FFFA2EA0078B1
127012F8A3EAF0F0EA70E0EA3FC0EA0F80101B7F9914>I<EAFFF8A2000FC7FCAE1430A31470A2
146014E01303B5FCA2141A7F9918>76 D<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3
EB7818A3EB3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>I<B4EBFF
C07F000FEB1E00EBC00CEA0DE0A2EA0CF013F81378133C133E131E130FA2EB078CEB03CCA2EB01
ECEB00FCA2147CA2143C001E131CEAFFC0140C1A1A7F991D>I<137F3801FFC03807C1F0380F00
78001E7F001C131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C131C001E13
3C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>I<B5128014E0380F00F014781438143CA514
38147814F0EBFFE0148090C7FCA9EAFFF0A2161A7F991A>I<137F3801FFC03807C1F0380F0078
001E7F001C131C003C131E487FA200707F00F01480A80070140000785BA2383C1C1E381C3E1C38
1E633C380F61F83807E1F03901FFC080EA007F130014E1ECFF0080143E141C19227E9A1E>I<EA
FFFEEBFFC0380F01E06D7E14701478A4147014F0495AEBFFC05CEB03C06D7E6D7EA615C0A238FF
F078EC3F80C7EA0F001A1B7F991C>I<EA07C2EA1FF6EA383EEA701EEA600E12E01306A3EAF000
1278EA7F80EA3FF0EA1FF8EA0FFCEA01FEEA001E130F130712C0A312E01306EAF00EEAFC1CEACF
F8EA83E0101C7E9A15>I<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FC
AF3803FFF0A2191A7F991C>I<39FFF0FFC0A2390F001E00140CB17E6D5A12036D5A3801E0E038
007FC0011FC7FC1A1B7F991D>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC
5DD807801306EB819EA2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE
13F0EBFC03017C5BA2EB7801A201385BEB3000241B7F9927>87 D<39FFE01FF0A2390F800F0000
0713066C6C5A13E000015BEBF038000013306D5A137CEB3CC0133F6D5AA26DC7FCA9EBFFF0A21C
1A80991D>89 D<EA1020EA3870EA70E0EA60C0EAC180A3EAF1E0EAF9F01278EA30600C0B7B9913
>92 D<EA3F80EAFFC0EAF0E0137012601200EA0FF0123FEA787012E0A21373A2EA70F3EA7FFEEA
3E3C10107E8F13>97 D<12FCA2121CA813F8EA1FFE130F381C0380A2EB01C0A6EB0380A2381F0F
00EA1BFEEA18F8121A7F9915>I<EA07F0EA0FFCEA3C3C1230EA7018EAE000A61270EA380CEA3C
18EA0FF8EA07E00E107F8F11>I<137EA2130EA8EA07CEEA1FFEEA3C1EEA700EA212E0A6127013
1EEA3C3E381FFFC0EA07CF121A7F9915>I<EA07C0EA1FF0EA3838EA7018131CEAFFFCA2EAE000
A41270EA300CEA3C18EA0FF0EA07E00E107F8F11>I<13F8EA01FCEA03BCEA073CEA0E181300A5
EAFFC0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA3877EA7038A5EA3870EA3F
E0EA7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781EEA1FF8EA07E010197F9013
>I<12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A7F9915>I<1218123CA21218
C7FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00C01300A6EA07E0A21200B0
126012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F80A2EB3C0013305B5B121D
EA1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12FCA2121CB3A4EAFF80A209
1A80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301AA39FF9FE7F8A21D107F8F
20>I<EAFCF8B47EEA1F1EEA1E0E121CAA38FF9FC0A212107F8F15>I<EA07E0EA1FF8EA381CEA70
0EEA6006EAE007A6EA700EA2EA3C3CEA1FF8EA07E010107F8F13>I<EAFCF8EAFFFEEA1F0F381C
07801303EB01C0A6EB03801307381F0F0013FEEA1CF890C7FCA5B47EA212177F8F15>I<EA07C6
EA1FF6EA3C3EEA781EEA700E12E0A61270EA781EEA3C3EEA1FEEEA07CEEA000EA5EB7FC0A21217
7F8F14>I<EAFDE0EAFFF0EA1F70EA1E20EA1C00AAEAFFC0A20C107F8F0F>I<EA1F20EA7FE01270
EAE060A2EAF000127FEA3FC0EA1FE0EA01F0EAC070A212E0EAF0E0EAFFC0EA8F800C107F8F0F>
I<120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B177F960F>I<EAFC7EA2EA1C
0EAA131E133E380FFFC0EA07CF12107F8F15>I<38FF3F80A2381C0E00130CA26C5AA21338EA07
30A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00381C3E0C13361337000E5B
13631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1FC0A2380E1E00EA071CEA03
B813B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A21310808F14>I<38FF3F80A2
381C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA2126100F3C7FC12C712FE127811
177F8F14>I<EA7FF8A2EA7070EA60E0126113C0EA63801207EA0F18120E121CEA3C38EA3830EA
7070EAFFF0A20D107F8F11>I<B512C0A21202808A13>I E /Fm 18 118
df<127812FCA412781200A5127812FCA4127806117D900C>58 D<EA0FF0EA3FFCEA703EEAF81F
A3EA703EEA007E13F8EA01E013C0EA03801300A4C7FCA4EA0780EA0FC0A4EA0780101A7E9915>
63 D<B512F014FC380FC03FEC0F80EC07C0EC03E0A2EC01F0A215F8A815F0A2140315E0EC07C0
EC0F80EC3F00B512FE14F01D1A7E9922>68 D<EAFFFCA2EA0FC0B3A4EAFFFCA20E1A7F9911>73
D<D8FFC0EB1FF86D133F000F1580A2D80DF0136FA2D80CF813CFA290387C018FA390383E030FA2
EB1F06A2EB0F8CA3EB07D8A2EB03F0A2EB01E0A2D8FFC1EBFFF8EBC0C0251A7E992A>77
D<EA0FF0EA1FFCEA3E3E131E131F121C1200EA07FF121FEA7E1F127C12F8A3EA7C3F383FEFE0EA
0F8713117F9015>97 D<EA03FCEA0FFEEA1E1F123C127CEA780E00F8C7FCA51278127C383C0180
381F0300EA0FFEEA03F811117F9014>99 D<EA03F0EA0FFCEA1E1E487E487E148012F8B5FCA200
F8C7FCA21278127C383C0180381F0300EA0FFEEA03F811117F9014>101
D<3807E3C0381FFFE0EA3C3C38381CC038781E00A4EA381CEA3C3CEA3FF8EA37E00070C7FCA2EA
3FFEEBFF806C13C0127F38F003E01301A3387C07C0383FFF803807FC0013197F9016>103
D<121E123FA4121EC7FCA4127FA2121FADEAFFC0A20A1B809A0C>105 D<B4FCA2121FB3A4EAFF
E0A20B1A80990C>108 D<39FF1F81F890387FE7FE391FE3EE3E903881F81FA2EB01F0AA3AFFE7
FE7FE0A223117F9026>I<EAFF1FEB7FC0EA1FC3EB83E0A21303AA38FFE7FCA216117F9019>I<EA
03F8EA0FFE383E0F80EA3C07387803C0A200F813E0A6007813C0EA7C07383E0F80380FFE00EA03
F813117F9016>I<EAFF3FEBFFC0381FC3F0130014F8147CA6147814F814F0EBC3E0EBFFC0EB3F
0090C7FCA5EAFFE0A216187F9019>I<EA1FB0EA3FF0EA7070EAE030A2EAF800EAFFC0EA7FF0EA
3FF8EA1FFC1201EAC03C131C12E0EAF038EAFFF0EACFC00E117F9011>115
D<1203A35AA25A123FEAFFF0A2EA1F00A81318A5EA0FF0EA03E00D187F9711>I<38FF1FE0A2EA
1F03AB1307130F380FFFFCEA03F316117F9019>I E /Fn 42 122 df<903901FF81FF011F01EF
13C0903A7F80FF87E0D9FE01EB0FF03903FC03FE13F8D807F013FCA2EE07E0020190C7FCA6B712
F8A32707F001FCC7FCB3A33A7FFF1FFFE0A32C2A7FA928>11 D<121C127FEAFF80A5EA7F00121C
09097B8813>46 D<EB3F803801FFF03807E0FC380FC07E48487EA2393F001F80A24814C0A44814
E0AF6C14C0A46C1480A2391F803F00A2380FC07E6C6C5A3801FFF038003F801B277DA622>48
D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FEA317277BA622>I<EBFF80000713F0001F
13FC383F03FFD87C001380007FEB7FC0EAFF80EC3FE0A3141FEA7F00001C133FC7FC15C0A2EC7F
80A2ECFF00495A5CEB03F0495A495A495A90383E00E05B13789038F001C0EA01C0EA038048B5FC
5A5A5A481480B6FCA31B277DA622>I<EB7F803801FFF0000713FC380F81FE381F80FF487E9038
E07F80A5381FC0FFD807001300C7FC495AEB03F8495AEBFFC014F0EB01FC6DB4FCEC7F8015C014
3F15E0121EEA7F80A2EAFFC0A315C0147FD87F801380387E00FF6C481300380FFFFC000313F0C6
13801B277DA622>I<14075C5C5C5C5CA25B5B497E130F130E131C1338137013F013E0EA01C0EA
0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F13F8A31D277EA622>I<000C1303380F80
3FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC0380FFFF8EB80FC380E007F000C1480C7123F
15C0A215E0A2123E127FEAFF80A315C01300007E137F007814806CEBFF00381F01FE380FFFF800
0313E0C690C7FC1B277DA622>I<EB07F0EB3FFCEBFFFE3801FC0F3807F01F390FE03F80EBC07F
121FEA3F80A2007FEB3F00EB001E91C7FCA25AEB0FC0EB3FF8EB70FEEBE03F01C013809038801F
C0A3010013E0A47EA4D83F8013C0A2001FEB3F80EA0FC03907E0FF003803FFFCC613F0EB3FC01B
277DA622>I<1238123E003FB512F0A315E04814C01580A215003870001E5C5C48137014F0495A
C6485A13075C130F91C7FC5B5BA3137EA213FEA41201A86C5A13781C297CA822>I<EB3FC03801
FFF04813FC380FC07EEB003F001E7F1580003E130FA2123F1380EBE01F01F81300EBFE3F381FFF
7E14F86C5B6C13FC6C7F6C7F00071480D81F9F13C0EA3F07D87E0313E0EA7C0000FC133F48131F
1407A21403A26C14C0007C1307007E14806C130F391FC03F00380FFFFC000313F038007FC01B27
7DA622>I<EB7F803801FFF000077F380FE0FE381FC07E48487ED87F001380A248EB1FC0A415E0
A46C133FA3383F807F001F13FF380FE1DF3803FF9F38007E1F130015C0A2120F381F803FD83FC0
1380A2EC7F00A2EB80FE381F01FC495A380FFFE06C1380D801FEC7FC1B277DA622>I<B612FCED
FF8016E03A03FC001FF0ED07F8821503A2821501A315035EA24B5A4B5A4B5AED7FC090B6C7FC16
E09039FC0007F0ED03FC6F7EA26F7EA21780A617005D4B5A15074B5AB712F016C04BC7FC29297D
A831>66 D<91393FF00180903903FFFE07010FEBFF8F90393FF007FF9038FF80014848C7127FD8
07FC143F49141F4848140F485A003F15075B007F1503A3484891C7FCAB6C7EEE0380A2123F7F00
1F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007F0010FB512C0010391C7FC90
38003FF829297CA832>I<B612F8EDFF8016E03A03FE001FF8ED07FCED01FE6F7EEE7F80EE3FC0
161F17E0A2EE0FF0A417F8AA17F0A3EE1FE0A217C0163FEE7F801700ED01FE4B5AED1FF8B712E0
168003FCC7FC2D297DA835>I<B512FEA300011300B3B1B512FEA317297FA81A>73
D<B592383FFFC0A26E5C0003EFF000A2D9BFC014EFA2D99FE0EB01CFA2D98FF0EB038FA3D987F8
EB070FA2D983FC130EA2D981FE131CA3D980FF1338A291387F8070A291383FC0E0A391381FE1C0
A291380FF380A2913807FF00A36E5AA26E5AA26E5AD8FFFE0203B512C0A215703A297DA841>77
D<B5ECFFFE80A200036DEB03808080EBBFF8EB9FFC138FEB87FE8001831380018113C01380EC7F
E015F0EC3FF8EC1FFCEC0FFE1407EC03FF16836E13C36E13E3157FED3FF3ED1FFB16FF81818181
A281167F163F161F160FEAFFFE160716032F297DA836>I<ECFFE0010713FC90393FC07F809039
7F001FC0D801FCEB07F048486D7E48486D7E000F8148486D7EA24848EC7F80A2007F16C049143F
A300FF16E0AA007F16C06D147FA2003F1680A26C6CECFF00A26C6C495A00075D6C6C495A6C6C49
5A6CB4EB1FE090393FC07F8090260FFFFEC7FC010013E02B297CA834>I<B612F815FF16C03A03
FE003FE0ED0FF0ED07F816FC150316FEA716FC150716F8ED0FF0ED3FE090B61280EDFE0049C8FC
B0B512F8A327297DA82F>I<B612E015FE6F7E3A03FE007FE0ED0FF06F7E82150382A65E4B5AA2
ED1FE0ED7FC090B500FEC7FC5D9038FE01FF9138007FC082153F82151FA81707A2ED0FF8170FB5
39F807FE1E923801FFFC9238003FF030297DA834>82 D<007FB712C0A39039807FC03FD87C0014
0700781503A20070150100F016E0A2481500A5C71500B3A490B612E0A32B287EA730>84
D<48B47E000F13F0381F81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA
1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97
D<EAFFE0A3120FACEBE1FE9038E7FF809038FE07E09038F803F8496C7E496C7EA2157FA21680A9
16005D5D7F4A5A6D485A90389E07E090380FFF80260E01FCC7FC212A7EA926>I<EB1FF0EBFFFE
3803F03F390FE07F80EA1FC0EA3F80A2127F9038001E004890C7FCA97E7F003FEB01C013C0001F
1303390FE007803903F01F003800FFFCEB1FE01A1B7E9A1F>I<EC3FF8A31403ACEB1FE3EBFFFB
3803F03F380FE00F381FC007383F8003A2127F13005AA97EA2EA3F801407381FC00F380FE01F3A
03F03FFF803800FFF3EB3FC3212A7EA926>I<EB3FE03801FFF83803F07E380FE03F391FC01F80
393F800FC0A2EA7F00EC07E05AA390B5FCA290C8FCA47E7F003F14E01401D81FC013C0380FE003
3903F81F803900FFFE00EB1FF01B1B7E9A20>I<9038FF81F00003EBE7FC390FC1FE7C391F80FC
FC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1F8381FFFE0001813800038
C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007F8007C13015A1400A36C13
01007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A22>103
D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14>
105 D<EB0380EB0FE0EB1FF0A5EB0FE0EB038090C7FCA7EBFFF0A31307B3A9127E12FFA2EB0FE0
A2EB1FC0387E3F80383FFE00EA07F0143785AA15>I<EAFFE0A3120FACEC1FFCA3EC07C0EC0F80
EC1E00147C5CEBE1F0EBE3E0EBE7C0EBEFE0EBFFF0A280EBF3FCEBE1FE13C080EC7F80143F15C0
EC1FE0EC0FF039FFFC3FFEA31F2A7EA924>I<EAFFE0A3120FB3B2EAFFFEA30F2A7DA914>I<3BFF
C07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF0013
8001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0
390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>I<EB3FE03801FFFC
3803F07E390FC01F80391F800FC0003F14E0EB00074814F0A34814F8A86C14F0A2393F800FE0A2
001F14C0390FC01F803907F07F003801FFFC38003FE01D1B7E9A22>I<38FFE1FE9038E7FF8090
38FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE
0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E38
0FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800
127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0
137C143C7EA26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203
A21207381FFFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B>
I<39FFE03FF8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A
26>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7
FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3AFFFE03FF80A3
3A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB
1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807007C90C8FCEAFE0F130E131E5BEA7C78EA
3FE0EA0FC021277F9A24>121 D E /Fo 74 124 df<EB3F0F9038FFBF803903C3F3C0380703E3
ECC180390E01C000A6B512FCA2380E01C0AE387F87FCA21A1D809C18>11
D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B512C0A2EA0E01AE387F87F8A2151D809C
17>I<EB3FC013FFEA03C3EA07031301120EA6B5FCA2EA0E01AE387FCFF8A2151D809C17>I<9038
3F03F09038FFCFF83903C0FC1C390701F03CA2390E00E01892C7FCA5B612FCA2390E00E01CAE3A
7FC7FCFF80A2211D809C23>I<EA7070EAF8F8EAFCFCA2EA7C7CEA0C0CA3EA1818A2EA3030EA60
60EA40400E0D7F9C15>34 D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C>
39 D<13C0EA0180EA03001206120E120C121C121812381230A21270A21260A212E0AC1260A212
70A21230A212381218121C120C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C
120C120E120612077EA21380A21201A213C0AC1380A21203A21300A25A1206120E120C121C1218
5A5A5A0A2A7E9E10>I<127012F012F8A212781218A31230A2127012601240050D7D840C>44
D<EAFFE0A30B0380890E>I<127012F8A3127005057D840C>I<1303A213071306A2130E130C131C
1318A213381330A213701360A213E013C0A21201138012031300A25A1206A2120E120CA2121C12
18A21238123012701260A212E05AA210297E9E15>I<EA03C0EA0FF0EA1C38EA381CA2EA700EA3
EAF00FADEA700EA3EA381CA2EA1C38EA0FF0EA07E0101D7E9B15>I<12035A123FB4FC12C71207
B3A3EAFFF8A20D1C7C9B15>I<EA07C0EA1FF0EA3878EA603C131E12F0EAF80FA312701200130E
131E131C133C1378137013E0EA01C0EA0380EA0700EA0E03120C1218EA3006EA7FFE12FFA2101C
7E9B15>I<EA07E0EA1FF0EA3838EA301CEA781EA312381200133C13381370EA07E0A2EA003813
1C131E130E130FA2127012F8A2130EEAF01EEA601CEA3838EA1FF0EA07C0101D7E9B15>I<131C
A2133C137CA213DC1201139C1203EA071C1206120E120C121812381230126012E0B512C0A23800
1C00A63801FFC0A2121C7F9B15>I<EA300CEA3FFC13F813E0EA3000A6EA33E0EA37F0EA3C38EA
381CEA301E130EEA000FA4127012F0A2EAE00EEA601E131CEA3878EA1FF0EA07C0101D7E9B15>
I<13F0EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EAFC1CEAF81E130E
12F0130FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>I<1260387FFF80A21400
EA6003EAC0065BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B15>I<EA03E0EA
0FF0EA1C38EA381CEA300E1270A31278131C123EEA3FB8EA1FE0EA07F0EA0FF8EA1CFCEA387EEA
701E130F12E01307A4EA700E130CEA3C3CEA1FF0EA07E0101D7E9B15>I<EA03C0EA0FF0EA1C38
EA381C1278EA700E12F0A2130FA51270131F1278EA383FEA1FEFEA0FCFEA000EA3EA301C127813
38EA7030EA30F0EA3FC0EA0F80101D7E9B15>I<127012F8A312701200A8127012F8A312700512
7D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A2127012601240051A7D
910C>I<007FB512C0B612E0C9FCA8B612E06C14C01B0C7E8F20>61 D<1306130FA3497EA4EB33
C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C
1D7F9C1F>65 D<B512C014F0380F00F8143C141C141EA4141C143C1478EBFFF014E0EB00F8143C
141EA2140FA5141E143E147CB512F814C0181C7E9B1D>I<90381F8080EBFFE13803F033380780
1B380F000F001E1307001C1303123C5A1401127012F091C7FCA70070EB01801278A27E001CEB03
00121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E9C1E>I<B512C014F0380F007C141E80
80EC038015C0A2140115E0A815C0A2140315801407EC0F00141E147CB512F014C01B1C7E9B20>
I<B512FCA2380F007C141C140C140E14061303A314005B13FFA213077FA21403A213001406A314
0E141E147CB512FCA2181C7E9B1C>I<B512F8A2380F007814381418141C140CA21303A21400A2
5B13FFA213077FA490C7FCA6EAFFF8A2161C7E9B1B>I<90381F8080EBFFE13803F0333807801B
380F000F001E1307001C1303123C5A1401127012F091C7FCA5ECFFF0A20070EB07801278A27E12
1C121E7E3807800F3803F0393800FFF090381FC0001C1E7E9C21>I<39FFF3FFC0A2390F003C00
AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>I<EAFFF0A2EA0F00B3A6EAFFF0A20C1C7F9B
0F>I<EAFFF8A2000FC7FCAF1418A414381430147014F01301B5FCA2151C7E9B1A>76
D<B46CEBFF806D5A000FECF800A2390DE00378A3380CF006A3EB780CA3EB3C18A3EB1E30A3EB0F
60A3EB07C0A3381E03803AFFC387FF80A2211C7E9B26>I<B4EBFFC07F000FEB1E00EBC00CA2EA
0DE0EA0CF0A21378A2133C131EA2130FA2EB078C14CC1303EB01ECA2EB00FCA2147C143CA2001E
131CEAFFC0140C1A1C7E9B1F>I<EB3F80EBFFE03803E0F83807803C380E000E001E130F48EB07
8000381303007814C0A20070130100F014E0A80078EB03C0A36CEB0780A26CEB0F006C131E6C6C
5A3803E0F83800FFE0EB3F801B1E7E9C20>I<B5128014E0380F00F01438143C141EA6143C1438
14F0EBFFE0148090C7FCAAEAFFF0A2171C7E9B1C>I<B5FC14E0380F00F01438143C80A55C1438
14F0EBFFE05CEB01E06D7E1478A71530143839FFF03C60EC1FE0C7EA07C01C1D7E9B1F>82
D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA07FF
C61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F8121E7E9C17>
I<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B03803FFFCA21B
1C7F9B1E>I<39FFF0FFC0A2390F001E00140CB36C5B138000035BEA01C03800E0E0EB7FC0011F
C7FC1A1D7E9B1F>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA20180147000079038678060
A32603C0E713C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13F6017E13
7EA3013C133CA3011C133801181318281D7F9B2B>87 D<397FF0FFC0A23907C03E0000031338EB
E03000011370EBF06000005B1379EB7D80013FC7FC7FA27F80131FEB3BC0EB33E01371EB61F0EB
C0F800011378EB807C3803003C487F380F801F39FFE0FFF0A21C1C7F9B1F>I<39FFF00FFCA239
078003C001C0138000031400EBE006EA01F000005BEBF81CEB7818EB7C38EB3C306D5A131F6D5A
A26D5AAAEB7FF8A21E1C809B1F>I<EA0808EA1818EA3030EA6060A2EAC0C0A3EAF8F8EAFCFCA2
EA7C7CEA38380E0D7B9C15>92 D<EA0FE0EA1FF8EA3C3C7FEA180E1200131EEA07FE121FEA3E0E
127812F01460A2131EEA783E383FFFC0381F878013127F9115>97 D<12FCA2121CA9137EEA1DFF
381F8780381E01C0001C13E0130014F0A614E01301001E13C0381F07803819FF00EA187C141D7F
9C17>I<EA03F0EA0FF8EA1E3C1238EA7818EA700012F0A612781306123CEA1E0CEA0FF8EA03E0
0F127F9112>I<EB1F80A21303A9EA03E3EA0FFBEA1E0FEA3807EA7803127012F0A612701278EA
3807EA1E1F380FFBF0EA07E3141D7F9C17>I<EA03E0EA0FF0EA1C38EA381CEA781EEA700EEAFF
FEA2EAF000A41270EA7806123CEA1E0CEA0FF8EA03E00F127F9112>I<1378EA01FCEA039EEA07
1EEA0E0C1300A6EAFFE0A2EA0E00AEEA7FE0A20F1D809C0D>I<EB03803807E7C0EA0FFDEA3C3D
38381C00EA781EA4EA381CEA3C3CEA3FF0EA37E00070C7FCA21230EA3FFC6CB4FC481380EA7007
38E001C0A438700380383C0F00EA1FFEEA07F8121C7F9215>I<12FCA2121CA9137CEA1DFFEA1F
07381E0380A2121CAB38FF9FF0A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121C
AEEAFF80A2091D7F9C0C>I<EA01C0EA03E0A3EA01C0C7FCA6EA0FE0A21200B31260EAF1C0A2EA
7F80EA3E000B25839C0D>I<12FCA2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA
1C781338133C131C7F130F38FF9FE0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>
I<39FC7E07E039FDFF9FF8391F83B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124
>I<EAFC7CEAFDFFEA1F07381E0380A2121CAB38FF9FF0A214127F9117>I<EA03F0EA0FFCEA1E1E
EA380700781380EA700300F013C0A600701380EA780700381300EA1E1EEA0FFCEA03F012127F91
15>I<EAFC7EEAFDFF381F8780381E03C0381C01E0A2EB00F0A6EB01E0A2381E03C0381F078038
1DFF00EA1C7C90C7FCA6B47EA2141A7F9117>I<3803E180EA0FF9EA1E1FEA3C071278130312F0
A612781307123CEA1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A7F9116>I<EAFDE0EAFFF0EA1F
78121E1330EA1C00ABEAFFC0A20D127F9110>I<EA1F90EA3FF0EA7070EAE030A3EAF800EA7F80
EA3FE0EA0FF0EA00F8EAC038A212E0A2EAF070EADFE0EA8FC00D127F9110>I<120CA5121CA212
3CEAFFE0A2EA1C00A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307
EA0C0F380FFBF0EA03E314127F9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA
0398A213F86C5AA26C5AA313127F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B0614
86A2EB318E000713CCA213603803E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038
070600EA038EEA01DC13D8EA00F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8
A21512809116>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C
5AA35BA3EAF180A200C7C7FC127E123C131A7F9116>I<EA7FFCA2EA7838EA7070EA60F013E0EA
61C01263EA0380EA070C120F120EEA1C1CEA3C181238EA7078EAFFF8A20E127F9112>I<B512F0
A21402808B15>I E /Fp 10 118 df<DC7FFE140E030FB500E0133E92B600F8137E020303FE13
FE021FEDFF81027FD9F80113E391B53980003FF7010301FCC7EA07FF4901F08049491400490180
157F4990C9123F4948161F4948160F485B4818074A1603485B481801A2485B1900A2485B1A7E5A
A391CCFCA2B5FCAD7EA2801A3EA27EA26C7FA21A7E6C6D177CA26C19FC6C6D17F86E16016CF003
F06C7F6D6CEE07E06D6CEE0FC06D6DED1F806D01E0ED7F006D6D15FE6D01FCEC03FC0100D9FF80
EB0FF86E01F8EBFFF0021F90B612C0020393C7FC020015FC030F14E09226007FFEC8FC474979C7
56>67 D<B97E18FC18FF19C019F0D8001F902680000F13FC05017F716C7E7213807213C0841AE0
A27213F0A31AF8A81AF0A34E13E0A21AC04E1380604E13004D485A050F13F892B75A19C04EC7FC
18F003C0CAFCB3A9B712F8A545477CC651>80 D<903807FFFE017FEBFFE048B612F84815FE4890
39001FFF8003077F48D980017F6F7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC
130F013FEBF03F3901FFFE004813F04813C04890C7FC485A485A485AA212FF5BA3167FA26D14FF
6C6CEB01EF003F140301FF90380FCFFF6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEB
F0010107903980007FF8362E7DAD3A>97 D<91381FFFC049B512FC010714FF011F158090267FFC
0113C0EBFFF048D9C00313E04813804813005A485AA248486D13C06F1380007FED7F0093C7FC5B
A212FFAA127F7FA3123F6D15F8121F16016C6C15F06C6D13036C6DEB07E06C6DEB0FC06C01F8EB
3F80903A7FFE01FF00011FB55A010714F8010114E0D9001F90C7FC2D2E7CAD35>99
D<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0B5FCA512
037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC033F01FE
011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026C7C00190
26E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5DB3A8B6D8
C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13FE92B6FC
9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25CA25CB3A8
B6D8C07FEBFFE0A53B2E7CAD42>I<EC0FFF91B512F0010714FE011F6E7E90263FFC037F903AFF
E0007FF0480180EB1FF84890C76C7E48486E7E000F824980001F1780003F17C04980A2007F17E0
A300FF17F0AA007F17E0A46C6C4A13C0A2001F17806D5C000F17006C6C4A5A6C6D495A6C6D495A
6C6D495A903A7FFC03FFE0011FB6128001074AC7FC010014F0020F90C8FC342E7DAD3B>I<EB01
F0A61303A31307A3130FA2131F133FA2137FEA01FF5A000F90B512C0B7FCA4C601F0C7FCB3A5ED
01F0A91503D97FF813E01507D93FFC13C090391FFE1F806DB5FC6D1400010113FC9038003FF024
427EC12E>116 D<D97FE0EC3FF0B5EC7FFFA5000315016C81B3A85EA35E5E6CED0F7F6E131F01
7F023E7F90273FFE01FCEBFFE06DB512F86D14F0010314C09039003FFE003B2E7CAD42>I
E /Fq 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879B7
32>49 D<DB1FFE14E00203B5EAC001021FECF803027FECFE07903B01FFFC00FF0F010701C0EB1F
DF4990C7EA07FFD93FFC804948804948804849157F4849153F4849151FA24890C9120FA2484816
07A2485AA2007F1703A34993C7FC12FFAD127F7FF001E0A2123FA26C7E18036C6C17C018076C7F
F00F806C7F6C6DED1F006C6D153E6D6C5D6D6C5DD90FFFEC03F06D01E0EB0FE0010101FCEB7FC0
6D6CB6C7FC021F14FC020314E09126001FFEC8FC3B3D7ABB48>67 D<EB1FFF48B512F0000714FC
390FF807FF001F01017F6D6C7F6F7EA282153F6C5A6C5AEA01C0C8FCA2EC07FF0103B5FC131F90
B5123F000313C0000F1300EA1FFC485A485A5B12FF5BA3157FA26C6C13FF6D5A6C6C4813FC3B1F
FC0FDFFFF00007B5120F0001EBFC0739003FF0012C267DA530>97 D<903801FFC0011F13F8017F
13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED03
FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C147C000F15786C6C14F86C6CEB01
F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E026267DA52D>101
D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C01027814804A7E4A14C0
5CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13
FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE3FF891C7121F17FC160F17FEA3EE07
FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E481380DAFC07130091383FFFFC020F
13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF80
91387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F80ED0E0091C8FCB3A3B512FEA423267D
A529>114 D<EB0780A5130FA4131FA2133FA2137F13FF5A1207001FEBFFF8B6FCA30001EB8000
B3153CA86C147814C0017F13F8ECE1F090381FFFE0010713C0010113001E377EB627>116
D E /Fr 36 119 df<EB078090391FE003809038387007EB703001E0130301C014000001137090
3880F0061203EC600C48C75A5D15E090383C01F09038FE03B89038C30618EB830C903803181C39
0F06300CEA1F86381DFC36393CF03F183838001F0078130E1400A200705C12F000705C12785D00
38495A003C49C7FC6C130E380F807C3803FFF038007F8021257BA325>38
D<127012F8A212F012E005057A840F>46 D<14035CA25C1580141FA214371477146714C7A2EB01
87A2EB0307A21306130E130C1318A2133090383FFFC05BEB600313C012011380EA0300A21206A2
121E39FFC03FFC13801E237DA224>65 D<90B512E015F890380F003C151E131E150E150FA24913
0E151EA2153C49137815F0EC01E090387FFFC090B5FC9038F003E0EC00F01578485A1538153CA2
48481378A315F039078001E01403EC07C0EC1F00B512FE14F020227DA122>I<90B512F015FC90
380F003E150F011EEB0780150316C015015B16E0A35BA449EB03C0A44848EB0780A216005D4848
130E5D153C5D48485B4A5AEC0780021FC7FCB512FC14F023227DA125>68
D<91387E0180903903FF810090380F80C390383E00670178133F49133ED801C0131E485A120748
C7121C120E121E5A15185A92C7FCA25AA4EC3FFC5AEC01E0A26C495AA312700078495A1238003C
130F6C131B260FC0F3C7FC3803FFC1C690C8FC212479A226>71 D<EBFFF8A2EB0F00A2131EA45B
A45BA45BA4485AA4485AA4485AA4EAFFF8A215227DA113>73 D<903807FFC04913809038003C00
A25CA45CA4495AA4495AA4495AA449C7FCA212381278EAF81EA2485AEA6078EA70F0EA3FE0EA1F
801A237CA11A>I<9039FFF80FFCA290390F0007C01600011E130E5D5D1560495B4A5A4AC7FC14
0E495A5C147814FCEBF1BCEBF33CEBFE1E13FC3801F01F497EA2813803C007A26E7EA2EA07806E
7EA28139FFF80FFEA226227DA125>I<EBFFFCA2010FC7FCA2131EA45BA45BA45BA4485A1560A2
15C0485AA2EC0180A238078003EC07005C147FB512FEA21B227DA11F>I<D9FFC0EB0FFCA2010F
EC1F801637011BEC3F00166FA216CF0133EB019EA2ED031EEB31E00161EB063CA2150C151801C1
5C1530A21560D80181495AA2ECE180EB80F13A0300F301E014F6A214FC00064A5A14F8A2001F13
F03AFFE0E07FFCA22E227DA12C>I<01FFEB1FFC1480010FEB03C01680D91BC01300A3EB19E001
311306A2EB30F0A201605B1478A3496C5AA3141ED801805BA2140FA2D803005BEC07E0A300065C
1403A2121F39FFE00180A226227DA124>I<14FE903807FF8090380F03E090383C00F001701378
4913384848133C4848131C48C7FC48141E121EA25AA25AA348143CA31578A34814F0A26CEB01E0
15C01403EC07800078EB0F00141E6C5B6C13F8380F83E03807FF80D801FCC7FC1F2479A225>I<
90B512C015F090380F0078153C011E131E150EA349131EA3153C491338157815F0EC03C090B512
005CEBF00FEC07803901E003C0A43903C00780A43907800F001503A2EC0706D8FFF8138EEC03FC
C7EA01F020237DA124>82 D<903801F06090380FFC4090381E0EC0EB3807EB700301E0138013C0
1201A2D803801300A26DC7FCA27FEA01F813FF6C13E06D7EEB1FF8EB03FCEB007C143C80A30030
131CA31418007013385C00781360387C01C038EF0380D8C7FFC7FCEA81FC1B247DA21B>I<001F
B512F8A2391E03C07800381438EB0780123000601430A2EB0F0012C0A3D8001E1300A45BA45BA4
5BA4485AA31203B57E91C7FC1D2277A123>I<393FFE07FFA23903C000F015E0484813C0A4390F
000180A4001EEB0300A4481306A4485BA4485BA25C12705C5C6C485A49C7FCEA1E0EEA0FFCEA03
F0202377A124>I<3BFFF03FF81FF8D9E07FEB3FF03B1F0007800780001E010FEB03001606141F
5E14375E14675E14C75E381F0187000F5DEB0307ED818013060383C7FC130C158613180138138C
0130139C0160139815B801C013B015E013805D13005D120E92C8FC120C2D2376A131>87
D<39FFF003FFA2000FC712F015E090388001C000071480EC0300EBC0060003130E5CEBE0180001
5B5CEBF0E000005BEBF18001FBC7FC13FF137E137C1378A45BA4485AA4EA3FFE5B202276A124>
89 D<EBF8C0EA01FDEA078F380F0780120E121CEA3C03383807001278A3EAF00EA214101418EB
1C30EA703C137C3838FC60383FCFC0380F078015157B9419>97 D<137E48B4FC3803C380EA0703
EA0E07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99
D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03
383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2
19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA
701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF
F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>I<EB1F
18EB3FB8EBF1F83801E0F013C0EA038000071370EB00E05AA3381E01C0A4EB0380EA0E07130FEA
071FEBFF00EA01E7EA0007A2130EA3EA701CEAF0385BEA7FE0EA3F80151F7E9416>I<136013F0
13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA
1F80EA0E000C217CA00F>105 D<13F0EA0FE0A21200A2485AA4485AA448C7FCEB01E0EB07F0EB
0E30380E1870EB30F01360EBC060381D8000121F13E0EA1CF0EA3838133CEB1C20143038703860
A21440EB18C038E01F8038600F0014237DA216>107 D<EA01E0EA1FC0A21201A2EA0380A4EA07
00A4120EA45AA45AA45AA212711380EAE300A312E6127E123C0B237CA20C>I<381E0780383F1F
E0EA63B8EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140C
EB0318003813F0381801E018157C941B>110 D<137E48B4FC3803C380380701C0120E001C13E0
123CA21278A338F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803
C1F03807E3F8380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EB
E3C038077F80EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863
B0E013E0EAC3C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114
D<13FC48B4FC38038380EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA70
07EAF00EA2485AEA7038EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2
EA0700120EA45AA45AA213101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F13
30381F8070EA31C0006113E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2
130FEB1F183807F3F03803E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC3
80A2000313C0EA0700A3380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I
E /Fs 53 122 df<ECC018A301011338EC8030A301031370EC0060A34913E001065BA3EB0E0101
0C5BB712C0A226001803C7FCA3EB3807EB3006A4B712C0A22600600CC7FCEBE01CEBC018A30001
1338EB8030A300031370EB0060A34813E000065BA2222D7DA229>35 D<127012F812FCA2127C12
0CA41218A21230A212601240060F7C840E>44 D<EAFFE0A30B037F8B10>I<127012F8A3127005
057C840E>I<EA01F0EA07FCEA0E0E487E38380380A2007813C0EA7001A300F013E0AE007013C0
A3EA780300381380A2381C0700EA0E0EEA07FCEA01F013227EA018>48 D<EA01801203120F12FF
12F31203B3A8EAFFFEA20F217CA018>I<EA03F0EA0FFCEA1C1F383007801270007813C0A21303
EA380712001480A2EB0F00130E133CEA03F8A2EA001E7FEB078014C0130314E01220127012F8A2
00F013C01260EB07801230381C1F00EA0FFCEA03F013227EA018>51 D<00101380EA1C07381FFF
005B5B13F00018C7FCA613F8EA1BFEEA1F0F381C0780EA180314C0EA000114E0A4126012F0A214
C0EAC0031260148038300700EA1C1EEA0FFCEA03F013227EA018>53 D<137E48B4FC3803C18038
0701C0EA0E03121CEB018048C7FCA2127812701320EAF1FCEAF3FEEAF60738FC038000F813C013
0112F014E0A51270A3003813C0130300181380381C0700EA0E0EEA07FCEA01F013227EA018>I<
EA01F0EA07FCEA0E0F38180780EA3803383001C01270A31278EB0380123E383F0700EA1FCEEA0F
FCEA03F87FEA0F7F381C3F80EA380F387007C0130338E001E01300A5387001C0A238380380381E
0F00EA0FFEEA03F013227EA018>56 D<EA01F0EA07FCEA0E0E487E383803801278127038F001C0
A314E0A5127013031278EA3807EA1C0DEA0FF9EA07F1380081C0130113031480A2383007001278
130EEA701C6C5AEA1FF0EA0FC013227EA018>I<497E497EA3497EA3497E130CA2EB1CF8EB1878
A2EB383C1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F000
1F130139FFC00FFFA220237EA225>65 D<B512F814FE3907800F80EC07C0EC03E0140115F0A515
E01403EC07C0EC0F8090B512005C9038801F80EC07C0EC03E0EC01F0140015F8A6EC01F0140315
E0EC0FC0B6120014FC1D227EA123>I<90380FE01090383FF8309038F81C703801E0063903C003
F03807800148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C14
60123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>I<B512
F014FE3807801FEC07C01403EC01E0EC00F015F81578157C153CA3153EA9153CA2157C1578A215
F0EC01E01403EC07C0EC1F00B512FE14F81F227EA125>I<B612C0A23807800F14031401140015
E0A21560A21460A21500A214E0138113FFA2138113801460A491C7FCA8EAFFFEA21B227EA120>
70 D<903807F00890383FFC189038FC0E383801E0033903C001F83807800048C71278121E1538
5AA2007C14181278A212F81500A6EC1FFF1278007CEB0078A2123CA27EA27E6C7E6C6C13F83801
F0013900FC079890383FFE08903807F80020247DA226>I<39FFFC3FFFA239078001E0AD90B5FC
A2EB8001AF39FFFC3FFFA220227EA125>I<3803FFF0A238000F00B3A6127012F8A3EAF01EEA60
1CEA3878EA1FF0EA07C014237EA119>74 D<39FFFC07FFA239078001F015C05D4AC7FC14065C5C
14385C5CEB81C0EB8380EB87C080138DEB98F013B0EBE078497E13808080A26E7E8114036E7EA2
6E7E4A7E3AFFFC07FF80A221227EA126>I<EAFFFEA2EA0780B3EC0180A41403A215005CA25C14
3FB6FCA219227EA11E>I<D8FFC0EB03FF6D5B000715E0A2D806F0130DA301781319A36D1331A3
6D1361A36D13C1A29038078181A3903803C301A3EB01E6A3EB00FCA31478EA1F80D8FFF0EB3FFF
143028227EA12D>I<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7FA2
EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000EA
FFF0156020227EA125>I<EB0FE0EB7FFCEBF83E3903E00F8039078003C0390F0001E0A2001EEB
00F0003E14F8003C1478007C147CA20078143CA200F8143EA9007C147CA3003C1478003E14F800
1E14F06CEB01E0EB80033907C007C03903E00F803900F83E00EB7FFCEB0FE01F247DA226>I<B5
12F014FC3807803FEC0F801407EC03C0A215E0A515C0A2EC0780140FEC3F00EBFFFC14F00180C7
FCADEAFFFCA21B227EA121>I<B512E014F83807803E140F6E7E816E7EA64A5A5D4AC7FC143EEB
FFF85CEB80788080140E140FA481A3ED818015C114073AFFFC03E300EC01FEC8127C21237EA124
>82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3F
F0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF801
38EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200E0
141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<3BFFF03FFC07FEA23B0F0007C001
F00203EB00E01760D807806D13C0A33B03C007F001801406A216032701E00C781300A33A00F018
3C06A3903978383E0CEC301EA2161C90393C600F18A390391EC007B0A3010F14E0EC8003A36D48
6C5AA32F237FA132>87 D<387FFFFEA2EB003C007C137C0070137814F84813F0EB01E0EAC00314
C01307148038000F005B131E133E133C5B13F85B0001130313E0EA03C012071380000F13071300
001E1306003E130E003C131E007C133E007813FEB5FCA218227DA11E>90
D<EA0FE0EA1FF8EA3C1C7FEA18071200A25BEA03FF120FEA3F07127C127812F01418A2130F1278
387C3FB8383FF3F0380FC3C015157E9418>97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0
F0EB0078000E1338143C141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237F
A21B>I<EA01FEEA07FF380F0780121C383803000078C7FC127012F0A7127814C07E381E018038
0F0300EA07FEEA01F812157E9416>I<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA
38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>I<EA01
FCEA07FF380F0780381C03C0EA3801007813E0EA7000B5FCA200F0C7FCA5127814607E6C13C038
0F83803807FF00EA00FC13157F9416>I<133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700
B1EA7FF8A2102380A20F>I<14F03801F1F83807FFB8380F1F38381E0F00EA1C07003C1380A500
1C1300EA1E0FEA0F1EEA1FFCEA19F00018C7FCA2121CEA1FFF6C13C04813E0383801F038700070
481338A400701370007813F0381E03C0380FFF803801FC0015217F9518>I<120E12FEA2121E12
0EAAEB1F80EB7FC0380FC1E0EB80F0EB0070120EAE38FFE7FFA218237FA21B>I<121C121E123E
121E121CC7FCA8120E12FEA2121E120EAFEAFFC0A20A227FA10E>I<EA01C0EA03E0A3EA01C0C7
FCA8EA01E0120FA212011200B3A4EA60C012F11380EA7F00123E0B2C82A10F>I<120E12FEA212
1E120EAAEB0FFCA2EB07E0EB0380EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03
C0130114E014F038FFE3FEA217237FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>
I<390E1FC07F3AFE7FE1FF809039C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8F
FEA227157F942A>I<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA21815
7F941B>I<EA01FCEA07FF380F0780381C01C0383800E0007813F00070137000F01378A7007013
70007813F0003813E0381C01C0380F07803807FF00EA01FC15157F9418>I<380E1F8038FE7FE0
38FFC1F0380F0078120E143CA2141EA7143CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FC
A8EAFFE0A2171F7F941B>I<EA0E3CEAFEFEEAFFCFEA1F8FEA0F061300120EADEAFFF0A210157F
9413>114 D<EA0F88EA3FF8EA7078EAE0381318A3EAF000127FEA3FE0EA1FF0EA01F8EA003CEA
C01CA212E0A2EAF018EAF878EADFF0EA8FC00E157E9413>I<1206A5120EA3121E123EEAFFF8A2
EA0E00AA130CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E
1370AC14F01301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A2
6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2
391E01C07CD81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D8
1980141DA2EBF00F00001400A2497EEB600620157F9423>I<38FFC3FEA2381E00F8000E1360A2
6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A21330A213701360A2EAF0C012F1
EAF380007FC7FC123E171F7F941A>121 D E /Ft 20 118 df<B51280A311037F9016>45
D<B612E015FC3907E0007F0003EC0F806F7EED01E06F7E1678167C82A282A2EE0F80A217C0A216
07A217E0AB17C0A3160F1780A3EE1F00A2163E5E167816F84B5AED07E0ED0F800007027FC7FCB6
12FC15E02B317CB033>68 D<B51280A23807F0006C5AB3B3A7487EB51280A211317DB017>73
D<D8FFF0ED7FF86D15FF0007170000035E017CEC01BEA36DEC033EA36D1406A36D6C130CA36D6C
1318A36D6C1330A36D6C1360A216C06D7EA291387C0180A391383E0300A3EC1F06A3EC0F8CA3EC
07D8A3EC03F0A3486C6C5AD80FC0157FD8FFFC91380FFFF8EC00C035317CB03D>77
D<B612C015F83907E000FE0003141FED0F80ED07C0ED03E016F01501A216F8A616F0A2150316E0
ED07C0ED0F80ED1F0015FE90B512F815C001E0C8FCB3A2487EB57EA225317CB02D>80
D<90387F80203901FFE0603807C0F8390F001CE0001E130F481307003813030078130112701400
12F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013C0EC
1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039E780
070038C1F01E38C07FFC38800FF01D337CB125>83 D<EA01FE380FFFC0381C03E0383C00F0003E
137880141C0008131EC7FCA4EB01FE133F3801FF1EEA07F0EA0F80EA1F00123E5AA248140CA314
3EA2007C137E6CEBDF1C391F038FB8390FFF07F03903F803C01E1F7D9E21>97
D<EB3FC0EBFFF83803E01C3807801E380F003E121EA2481308007C1300A2127812F8A9127CA36C
1303121E001F1306380F800E3807C01C3803F0383800FFE0EB3F80181F7D9E1D>99
D<EC01E0143FA214031401AFEB3F81EBFFE13803E0793807800D380F0007001E130314015A127C
A2127812F8A91278127CA2123C003E1303001E13076C130F3807801D3903E071F03901FFE1FF38
003F0120327DB125>I<EB3F80EBFFE03803E0F83807803C48487E121E805A127C158000781307
12F8B6FCA200F8C8FCA61278127C123CEC01807E6CEB0300EB80063807C00E3801F03C3800FFF0
EB1FC0191F7E9E1D>I<EB03E0EB1FF8EB3C38EB707C13F0EA01E014383803C000ACB512C0A238
03C000B3A8487EEA7FFFA216327FB114>I<15F090387F03F83901FFCF1C3803C1FC390780F818
390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0
D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB
00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F
7E9F21>I<1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8
A20D307EAF12>105 D<260781FEEB3FC03BFF87FF80FFF0903A8E07C1C0F83B0F9803E3007C27
07B001E6133C9026E000FC7F495BA3495BB3486C486C133F3CFFFC1FFF83FFF0A2341F7E9E38>
109 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E
3AFFFC1FFF80A2211F7E9E25>I<EB1FC0EBFFF83801E03C3807800F390F000780001EEB03C0A2
48EB01E0A248EB00F0A300F814F8A8007814F0007C1301003C14E0A26CEB03C0A26CEB07803907
C01F003801F07C6CB45AEB1FC01D1F7E9E21>I<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE0
38EBC000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813
705A1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2
141C7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A312
03A21207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB
19>I<D8078013F000FF131FA2000F130100071300B31401A2140300031307EBC00E3901F03CF8
3A00FFF0FF80EB3FC0211F7E9E25>I E /Fu 5 85 df<1630167016F0A21501A21503A2150715
0FA2151B821531A2156115E115C1EC0181A2EC0301A21406A2140C141C14181430A202607FA2EC
C000A249B5FC5B91C7FC1306A25BA25BA25B1370136013E01201000381D80FF01301D8FFFE9038
3FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E8314
7E188002FE140FA24A15C0A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA201
1F157E17FE4A5CA2013F4A5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495A
ED07E00001EC3F80B600FEC8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7
121EA24A140CA2147EA214FEA25CA20101151CEEC0184A1400A201031301A202F05B1503010713
0F91B5FC93C7FCECE00F010F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA213
7EA213FEA25BA21201B512FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092
380003F0EE01F84AEB00FCA2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804A
EB3F00167E0107EB03F891B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA201
3F1301A21400A249495AA2137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0
C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C13
0F121800385CA20030011F1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FE
A25CA21301A25CA21303A25CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D31
74B033>84 D E end
%%EndProlog
%%BeginSetup
%%Feature: *Resolution 300
TeXDict begin
%%EndSetup
%%Page: 0 1
bop 795 707 a Fu(D)26 b(R)g(A)f(F)h(T)225 798 y Ft(Do)r(cumen)n(t)20
b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)685
981 y Fs(Scott)c(Berryman,)e Fr(Y)l(ale)19 b(Univ)700 1039
y Fs(James)c(Co)o(wnie,)h Fr(Meiko)i(Ltd)474 1097 y Fs(Jac)o(k)e(Dongarra,)i
Fr(Univ.)23 b(of)17 b(T)l(ennesse)n(e)j(and)d(ORNL)801 1155
y Fs(Al)f(Geist,)f Fr(ORNL)795 1213 y Fs(Bill)f(Gropp,)j Fr(ANL)767
1271 y Fs(Rolf)f(Hemp)q(el,)e Fr(GMD)762 1329 y Fs(Bob)i(Knigh)o(ten,)g
Fr(Intel)786 1387 y Fs(Rust)o(y)g(Lusk,)h Fr(ANL)614 1445 y
Fs(Stev)o(e)e(Otto,)h Fr(Or)n(e)n(gon)h(Gr)n(aduate)g(Inst)578
1504 y Fs(T)l(on)o(y)g(Skjellum,)c Fr(Missisippi)j(State)j(Univ)653
1562 y Fs(Marc)d(Snir,)g Fr(IBM)h(T.)g(J.)g(Watson)744 1620
y Fs(Da)o(vid)f(W)l(alk)o(er,)f Fr(ORNL)626 1678 y Fs(Stev)o(e)g(Zenith,)g
Fr(Kuck)k(&)f(Asso)n(ciates)844 1796 y Fs(Ma)o(y)e(9,)g(1993)87
1854 y(This)g(w)o(ork)g(w)o(as)h(supp)q(orted)g(b)o(y)f(ARP)l(A)g(and)g(NSF)g
(under)g(con)o(tract)g(n)o(um)o(b)q(er)f(###,)g(b)o(y)g(the)192
1912 y(National)h(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o(hnology)
f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)650 1970 y(Agreemen)o(t)f(No.)21
b(CCR-8809615.)p eop
%%Page: 1 2
bop 75 359 a Fq(Chapter)32 b(1)75 570 y Fp(P)m(oin)m(t)41 b(to)f(P)m(oin)m(t)
h(Comm)m(unication)884 789 y Fo(Marc)15 b(Snir)684 838 y(William)10
b(Gropp)j(and)h(Ewing)f(Lusk)75 991 y Fn(1.1)70 b(In)n(tro)r(duction)75
1088 y Fo(This)17 b(section)h(is)e(a)h(draft)g(of)f(the)i(curren)o(t)g(prop)q
(osal)f(for)g(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(unication.)24
b(It)17 b(do)q(es)h(not)f(y)o(et)75 1137 y(include)d(a)g(description)g(of)f
(the)i(F)m(ortran)f(77)f(and)h(C)f(bindings.)158 1190 y(I)h(ha)o(v)o(e)g
(tried)g(to)g(indicate,)f(wherev)o(er)i(appropriate,)f(gaps)g(and)f(unresolv)
o(ed)i(issues,)g(using)e(small)f(t)o(yp)q(e.)158 1321 y Fm(Discussion:)38
b Fl(The)13 b(follo)o(wing)j(sections)f(of)f(the)g(in)o(tro)q(duction)i(con)o
(tain)f(general)g(information)h(on)e(the)g(design)h(of)75 1367
y(MPI)h(pro)q(cedures.)27 b(The)16 b(material)i(should)g(b)q(e)e(mo)o(v)o(ed)
g(to)g(a)g(general)h(in)o(tro)q(duction)i(for)d(the)g(en)o(tire)h(do)q(cumen)
o(t.)27 b(The)75 1413 y(actual)14 b(binding)i(of)d(these)g(ob)r(jects)g(in)h
(F)m(ortran)g(and)f(C)g(will)h(b)q(e)g(discussed)h(in)e(the)h(language)h
(binding)g(sub)q(committee.)75 1648 y Fn(1.2)70 b(Data)23 b(T)n(yp)r(es)75
1753 y Fk(1.2.1)55 b(Handle)75 1835 y Fo(MPI)17 b(pro)q(cedures)j(use)e(at)f
(v)n(arious)g(places)h Fj(hand)r(les)p Fo(.)28 b(Handles)18
b(are)g(used)g(to)f(access)i(opaque)e(ob)r(jects.)30 b(Suc)o(h)75
1885 y(ob)r(ject)20 b(can)g(b)q(e)g(created,)i(up)q(dated)e(and)f(destro)o(y)
o(ed)i(only)d(b)o(y)h(b)o(y)h(calling)e(suitable)h(MPI)h(pro)q(cedures,)i
(and)75 1934 y(pro)o(viding)14 b(the)i(handle)f(as)g(parameter.)21
b(Opaque)16 b(ob)r(jects)g(hide)g(from)d(the)j(user)g(the)g(in)o(ternal)e
(represen)o(tation)75 1984 y(used)20 b(for)f(v)n(arious)f(MPI)i(ob)r(jects,)h
(th)o(us)e(allo)o(wing)e(to)i(ha)o(v)o(e)g(similar)e(calls)i(in)f(C)h(and)g
(F)m(ortran,)h(allo)o(wing)d(to)75 2034 y(o)o(v)o(ercome)f(problems)g(with)h
(the)g(t)o(yping)f(rules)i(in)e(these)j(languages,)d(and)h(allo)o(wing)e(for)
h(future)i(extension)g(of)75 2084 y(their)13 b(functionalit)o(y)m(.)j(The)e
(mec)o(hanism)c(for)j(opaque)g(ob)r(jects)h(used)g(here)g(lo)q(osely)e(follo)
o(ws)g(the)h(POSIX)h(F)m(ortran)75 2134 y(binding)f(standard.)158
2186 y(An)i(opaque)h(ob)r(ject)g(can)f(b)q(e)h Fj(p)n(ersistent)f
Fo(or)g Fj(ephemer)n(al)p Fo(.)22 b(A)15 b(p)q(ersisten)o(t)i(ob)r(ject)f(p)q
(ersists)h(un)o(til)e(destro)o(y)o(ed)75 2236 y(b)o(y)h(an)g(explicit)g(op)q
(eration.)25 b(An)17 b(ephemeral)e(ob)r(ject)j(is)e(go)q(o)q(d)g(for)g(a)g
(single)g(use;)h(th)o(us)g(an)f(ephemeral)g(ob)r(ject)75 2286
y(asso)q(ciated)i(with)g(a)f(comm)o(unicatio)o(n)e(op)q(eration)i(disapp)q
(ears)i(once)f(this)g(op)q(eration)f(is)h(completed)f(\(or)g(once)75
2336 y(this)d(ob)r(ject)h(is)e(not)h(needed)i(an)o(ymore)c(for)i(the)g
(completion)e(of)h(the)i(op)q(eration\).)158 2389 y(Opaque)i(ob)r(ject)g(are)
g(created)h(b)o(y)e(calls)g(that)g(are)h(sp)q(eci\014c)g(to)g(eac)o(h)f(ob)r
(ject)i(t)o(yp)q(e.)25 b(They)17 b(are)g(destro)o(y)o(ed)75
2438 y(b)o(y)d(a)f(call)h(to)f Fi(MPI)p 363 2438 14 2 v 15
w(FREE)p Fo(.)g(Additional)f(MPI)j(functions)f(are)g(a)o(v)n(ailable)e(to)h
(access)j(and)e(up)q(date)h(sp)q(eci\014c)g(opaque)75 2488
y(ob)r(jects.)158 2576 y Fh(MPI)p 257 2576 15 2 v 17 w(FREE\(handle\))75
2704 y(IN)h(handle)j Fo(handle)14 b(to)g(ob)r(ject)965 2828
y(1)p eop
%%Page: 2 3
bop 75 -100 a Fo(2)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g
(COMMUNICA)m(TION)158 45 y Fo(Mark)e(the)h(handle)g(for)f(destruction.)18
b(The)13 b(ob)r(ject)g(can)g(destro)o(y)o(ed)h(as)e(so)q(on)h(as)f(there)i
(there)f(is)g(no)f(p)q(ending)75 95 y(op)q(eration)g(that)g(is)g(using)g
(this)g(ob)r(ject.)18 b(This)12 b(is)g(equiv)n(alen)o(t)g(to)g(c)o(hanging)f
(the)i(p)q(ersistence)i(of)c(the)i(handle)f(from)75 145 y Fi(persistent)c
Fo(to)i Fi(ephemeral)p Fo(.)15 b(F)m(or)9 b(example,)g(if)g(a)h(comm)o
(unication)d(ob)r(ject)j(is)g(freed,)i(after)e(the)g(comm)o(unication)75
195 y(started)15 b(but)f(b)q(efore)h(it)f(completed,)f(then)h(the)h(p)q
(ending)f(op)q(eration)g(is)f(not)h(a\013ected;)h(one)g(can)f(use)h(the)f
(handle)75 244 y(in)f(calls)g(to)g Fi(MPI)p 334 244 14 2 v
15 w(WAIT)f Fo(or)h Fi(MPI)p 565 244 V 15 w(STATUS)p Fo(.)f(Ho)o(w)o(ev)o
(er,)h(once)h(the)g(op)q(eration)f(completes,)g(it)g(is)g(erroneous)h(to)f
(reuse)75 294 y(the)h(freed)h(handle)f(for)g(a)f(new)h(comm)o(unication.)158
426 y Fm(Discussion:)158 475 y Fl(An)f(alternativ)o(e)i(design)f(p)q(oin)o(t)
g(w)o(as)f(discussed,)h(whereb)o(y)g(it)f(is)h(the)e(user)i(resp)q(onsibili)q
(t)o(y)i(to)d(free)f(a)h(handle)i(only)f(if)75 520 y(there)h(are)g(no)g(p)q
(ending)i(op)q(erations)g(on)e(this)h(handle.)24 b(The)15 b(curren)o(t)g
(design)h(w)o(as)f(deemed)g(more)g(elegan)o(t.)24 b(Note)15
b(that)75 566 y(it)f(is)g(not)f(necessary)i(to)e(destro)o(y)h(the)g(handle)h
(as)e(so)q(on)h(as)g(it)f(is)h(not)g(needed)g(an)o(ymore.)19
b(A)13 b(v)n(alid)i(implemen)o(tation)i(ma)o(y)75 612 y(just)c(mark)g(freed)g
(handles)i(and)f(garbage)g(collect)g(them)f(p)q(erio)q(dical)q(ly)n(.)158
660 y(On)k(the)f(other)h(hand,)h(some)f(ob)r(jects,)h(suc)o(h)f(as)g
(bu\013er)g(descriptor)i(ob)r(jects,)e(can)g(b)q(e)g(used)g(b)o(y)g(sev)o
(eral)h(distinct)75 706 y(concurren)o(t)c(comm)o(unication)i(op)q(erations.)j
(Th)o(us,)13 b(to)g(\014nd)h(out)f(whether)h(suc)o(h)g(ob)r(ject)f(can)g(b)q
(e)h(freed,)f(one)g(w)o(ould)h(need)75 752 y(a)f(reference)h(coun)o(t)g(of)f
(p)q(ending)j(op)q(erations.)j(This)c(complication)h(w)o(as)d(not)h(observ)o
(ed)g(at)g(our)f(last)h(meeting,)h(and)f(ma)o(y)75 797 y(b)q(e)f(a)g(go)q(o)q
(d)h(argumen)o(t)g(to)f(reconsider)i(the)e(curren)o(t)g(de\014nition)j(of)d
Ff(MPI)p 1130 797 12 2 v 13 w(FREE)p Fl(.)158 846 y(W)m(e)h(do)f(not)h(ha)o
(v)o(e)g(y)o(et)g(a)f(mec)o(hanism)i(to)f(con)o(trol)g(where)g(from)f
(storage)h(is)g(allo)q(cated)i(for)d(opaque)i(ob)r(jects.)j(Some)75
892 y(b)q(eliev)o(e)d(it)e(is)h(desirable)h(to)e(allo)o(w)h(the)g(user)f(to)g
(con)o(trol)h(this)g(allo)q(cation.)158 941 y(I)e(no)o(w)h(ha)o(v)o(e)g(a)g
(generic)h Ff(MPI)p 576 941 V 13 w(FREE)d Fl(function)j(for)e(all)i(kinds)g
(of)f(opaque)g(ob)r(jects.)18 b(An)o(y)12 b(reason)i(to)e(ha)o(v)o(e)i(sp)q
(ecialized)75 987 y(functions)h(for)d(eac)o(h)i(kind)g(of)f(ob)r(ject,)g(as)g
(for)g Ff(CREATE)p Fl(?)158 1158 y Fh(MPI)p 257 1158 15 2 v
17 w(ASSOCIA)l(TED\(handle,)h(t)o(yp)q(e\))75 1286 y(IN)i(handle)j
Fo(handle)14 b(to)g(ob)r(ject)75 1382 y Fh(OUT)i(t)o(yp)q(e)k
Fo(state)158 1474 y(Returns)c(the)f(t)o(yp)q(e)g(of)f(the)i(ob)r(ject)f(the)h
(handle)e(is)h(curren)o(tly)h(asso)q(ciated)f(with,)f(if)g(suc)o(h)i(exists.)
21 b(Returns)75 1524 y(the)14 b(sp)q(ecial)h(t)o(yp)q(e)f Fi(MPI)p
444 1524 14 2 v 15 w(NULL)f Fo(if)g(the)i(handle)e(is)h(not)g(curren)o(tly)h
(asso)q(ciated)f(with)g(an)o(y)f(ob)r(ject.)158 1577 y(MPI)18
b(ma)o(y)e(pro)o(vide)h(prede\014ned)j(opaque)d(ob)r(jects)i(and)f
(prede\014ned,)i(static)e(handles)g(to)f(these)i(ob)r(jects.)75
1627 y(Suc)o(h)14 b(ob)r(jects)h(ma)o(y)d(not)i(b)q(e)h(destro)o(y)o(ed.)75
1760 y Fk(1.2.2)55 b(Arra)n(y)19 b(of)g(handles)75 1843 y Fo(An)14
b(MPI)h(call)f(ma)o(y)e(need)j(a)f(parameter)g(that)g(is)h(an)f
Fj(arr)n(ay)g(of)i(hand)r(les)p Fo(.)k(The)14 b(arra)o(y-of-handles)g(ob)r
(ject)h(is)f(not)75 1893 y(self-delimiting;)9 b(it)i(do)q(es)i(not)f(carry)g
(information)d(on)i(the)i(n)o(um)o(b)q(er)e(of)g(en)o(tries)i(in)e(the)i
(arra)o(y)m(.)j(This)c(information)75 1942 y(has)i(to)g(b)q(e)g(pro)o(vided)g
(b)o(y)g(an)f(additional)f Fi(len)i Fo(parameter,)f(whenev)o(er)i(a)f
(list-of-handles)e(parameter)i(is)g(used.)158 2074 y Fm(Discussion:)158
2123 y Fl(The)d(reason)g(for)g(not)g(ha)o(ving)h(self-delimiti)q(ng)i(arra)o
(ys)d(is)h(that)f(records)g(are)g(not)g(\014rst-class)h(citizens)g(in)g(F)m
(ortran)f(and)75 2168 y(that)h(ha)o(ving)i(an)f(arra)o(y)f(where)h(one)f(en)o
(try)h(enco)q(des)g(length,)g(whereas)g(the)f(other)h(en)o(tries)g(enco)q(de)
g(handles)h(w)o(as)e(deemed)75 2214 y(to)k(b)q(e)g(to)q(o)g(error-prone.)27
b(This)17 b(line)g(of)f(argumen)o(tation)i(really)g(implies)g(that)e(w)o(e)g
(do)g(not)g(view)h(arra)o(y-of-handles)h(as)75 2260 y(an)f(opaque)g(ob)r
(ject,)h(and)f(that)f(w)o(e)g(exp)q(ect)h(the)g(user)g(to)f(build)j(these)e
(ob)r(jects)f(explicitl)q(y)m(,)k(not)c(via)i(MPI)e(calls.)29
b(The)75 2305 y(alternativ)o(e)15 b(is)f(for)e(arra)o(y-of-handles)k(to)c(b)q
(e)i(an)f(opaque)h(ob)r(ject,)f(manipulated)j(via)e(s)f(set)g(of)g(MPI)g
(calls.)75 2521 y Fk(1.2.3)55 b(State)75 2604 y Fo(MPI)14 b(pro)q(cedures)i
(use)e(at)f(v)n(arious)g(places)h(argumen)o(ts)f(with)g Fj(state)g
Fo(t)o(yp)q(es.)19 b(The)14 b(v)n(alues)f(of)g(suc)o(h)h(data)g(t)o(yp)q(e)g
(are)75 2654 y(all)f(iden)o(ti\014ed)h(b)o(y)g(names,)e(and)i(no)g(op)q
(eration)g(is)f(de\014ned)i(on)f(them.)k(F)m(or)13 b(example,)f(the)j
Fi(MPI)p 1589 2654 V 15 w(APPEND)e Fo(routine)75 2704 y(has)h(a)g(state)g(t)o
(yp)q(e)h(parameter)e(with)h(v)n(alues)f Fi(MPI)p 864 2704
V 16 w(INT,)20 b(MPI)p 1054 2704 V 16 w(REAL,)12 b Fo(etc.)p
eop
%%Page: 3 4
bop 75 -100 a Fg(1.3.)31 b(PR)o(OCESSES)1437 b Fo(3)75 45 y
Fk(1.2.4)55 b(Named)18 b(constan)n(ts)75 124 y Fo(MPI)e(pro)q(cedures)j
(sometimes)14 b(assign)i(a)g(sp)q(ecial)g(meaning)e(to)i(a)g(sp)q(ecial)g(v)n
(alue)g(of)f(a)h(basic)g(t)o(yp)q(e)h(parameter;)75 173 y(e.g.)i
Fi(tag)14 b Fo(is)g(an)g(in)o(teger)g(v)n(alued)g(parameter)g(of)g(p)q(oin)o
(t-to-p)q(oin)o(t)f(comm)o(unicatio)o(n)f(op)q(erations,)i(with)g(a)g(sp)q
(ecial)75 223 y Fi(DONTCARE)g Fo(v)n(alue.)24 b(Suc)o(h)17
b(parameters)f(will)f(ha)o(v)o(e)h(a)g(range)g(of)f(regular)h(v)n(alues,)g
(whic)o(h)g(is)g(a)g(prop)q(er)h(subrange)75 273 y(of)f(the)h(range)g(of)f(v)
n(alues)h(of)f(the)h(corresp)q(onding)h(basic)f(t)o(yp)q(e;)h(sp)q(ecial)f(v)
n(alues)f(\(suc)o(h)i(as)f(DONTCARE\))f(will)75 323 y(b)q(e)f(outside)f(the)h
(regular)f(range.)19 b(The)c(range)f(of)g(regular)g(v)n(alues)g(can)g(b)q(e)h
(queried,)f(and)g(sometimes)f(set,)h(using)75 373 y(en)o(vironmen)o(t)f
(inquiry)g(or)h(en)o(vironmen)o(t)f(setting)h(functions)g(\(Section)g
Fh(??)p Fo(\).)158 502 y Fm(Discussion:)158 549 y Fl(Need)f(to)g(agree)g(on)h
(a)f(language)i(mec)o(hanism)f(for)f(named)h(constan)o(ts.)158
595 y(Implemen)o(ters)j(should)g(detect,)f(whenev)o(er)g(p)q(ossible,)i
(illegal)g(uses)e(of)f(\\sp)q(ecial)j(v)n(alues".)25 b(Th)o(us,)16
b(the)g(use)g(of)f(the)75 641 y Ff(DONTCARE)10 b Fl(v)n(alue)k(to)f(tag)g(a)g
(message)h(sen)o(t)f(will)h(b)q(e)g(\015agged)g(as)f(an)g(error.)75
845 y Fk(1.2.5)55 b(Choice)75 924 y Fo(MPI)15 b(functions)h(sometimes)d(use)j
(parameters)f(with)g(a)f Fj(choic)n(e)i Fo(\(or)f(union\))g(data)f(t)o(yp)q
(e.)23 b(I.e.,)14 b(distinct)h(calls)g(to)75 973 y(the)i(same)f(routine)h(ma)
o(y)e(pass)i(b)o(y)f(reference)j(actual)d(parameters)h(of)f(di\013eren)o(t)i
(t)o(yp)q(es.)27 b(The)17 b(mec)o(hanism)d(for)75 1023 y(pro)o(viding)f(suc)o
(h)h(parameters)g(will)f(di\013er)h(from)e(language)h(to)h(language.)158
1153 y Fm(Discussion:)158 1199 y Fl(The)i(F)m(ortran)g(77)g(standard)h(sp)q
(eci\014es)h(that)e(the)g(t)o(yp)q(e)g(of)g(actual)h(argumen)o(ts)g(need)f
(to)g(agree)g(with)h(the)f(t)o(yp)q(e)g(of)75 1245 y(dumm)o(y)g(argumen)o
(ts;)i(no)f(construct)f(equiv)n(alen)o(t)i(to)e(C)g(p)q(oin)o(ters)h(is)f(a)o
(v)n(ailable.)29 b(Th)o(us,)16 b(it)g(w)o(ould)h(seem)f(that)g(there)g(is)75
1291 y(no)f(standard)h(conforming)g(mec)o(hanism)g(to)e(supp)q(ort)i(c)o
(hoice)g(parameters.)22 b(Ho)o(w)o(ev)o(er,)14 b(most)h(F)m(ortran)g
(compiler)h(either)75 1336 y(don't)g(c)o(hec)o(k)g(t)o(yp)q(e)g(consistency)i
(of)e(calls)h(to)f(external)h(routines,)g(or)f(supp)q(ort)h(a)f(sp)q(ecial)h
(mec)o(hanism)h(to)d(link)j(foreign)75 1382 y(\(e.g.,)d(C\))f(routines.)25
b(I)15 b(suggest)h(that)f(w)o(e)g(accept)g(this)h(nonconformit)o(y)h(with)f
(F)m(ortran)f(77)h(standard.)24 b(I.e.,)15 b(w)o(e)f(accept)75
1428 y(that)f(the)g(same)h(routine)g(ma)o(y)f(b)q(e)g(passed)h(an)g(actual)g
(parameter)g(of)e(a)h(di\013eren)o(t)i(t)o(yp)q(e)e(at)g(distinct)i(calls.)
158 1474 y(Generic)h(routines)g(can)f(b)q(e)g(used)g(in)g(F)m(ortran)g(90)g
(to)g(pro)o(vide)h(a)e(standard)i(conforming)g(solution.)24
b(This)16 b(solution)75 1520 y(will)f(b)q(e)e(consisten)o(t)h(with)g(our)f
(nonstandard)i(conforming)g(F)m(ortran)e(77)g(solution.)75
1745 y Fn(1.3)70 b(Pro)r(cesses)75 1921 y Fm(Discussion:)18
b Fl(This)c(material)g(b)q(elongs)h(to)e(an)g(in)o(tro)q(duction)j(or)d(en)o
(vironmen)o(t)i(section)158 2054 y Fo(An)k(MPI)g(program)e(is)i(executed)h(b)
o(y)f(sev)o(eral)g(autonomous)e(pro)q(cesses)22 b(that)c(eac)o(h)i(execute)g
(their)f(o)o(wn)75 2104 y(co)q(de,)d(in)f(a)h(MIMD)f(st)o(yle.)23
b(The)16 b(co)q(des)h(executed)g(b)o(y)f(eac)o(h)g(pro)q(cess)h(need)f(not)g
(b)q(e)g(iden)o(tical.)22 b(The)16 b(pro)q(cesses)75 2154 y(comm)o(unicate)d
(via)h(calls)h(to)g(MPI)g(comm)o(unication)d(primitiv)o(es.)19
b(T)o(ypically)m(,)13 b(eac)o(h)j(pro)q(cessor)h(executes)g(in)e(its)75
2203 y(o)o(wn)f(address)i(space,)f(although)f(shared-memory)f(implem)o(en)o
(tations)f(of)i(MPI)h(are)g(p)q(ossible.)20 b(This)14 b(do)q(cumen)o(t)75
2253 y(sp)q(eci\014es)i(the)f(b)q(eha)o(vior)f(of)g(a)g(parallel)g(program)e
(assuming)h(that)i(only)e(MPI)i(calls)f(are)h(used)g(for)f(comm)o(unica-)75
2303 y(tion.)j(The)d(in)o(teraction)f(of)g(an)g(MPI)h(program)e(with)h(other)
h(p)q(ossible)f(means)g(of)g(comm)o(unicatio)o(n)e(\(e.g.,)h(shared)75
2353 y(memory\))e(is)i(not)h(sp)q(eci\014ed.)19 b(In)13 b(particular,)f(it)g
(is)g(assumed)g(that)h(message)f(bu\013ers)i(at)e(distinct)h(pro)q(cessors)i
(are)75 2403 y(disjoin)o(t.)158 2454 y(MPI)e(do)q(es)g(not)f(sp)q(ecify)h
(the)g(execution)g(mo)q(del)e(for)h(eac)o(h)h(pro)q(cess.)20
b(A)12 b(pro)q(cess)i(can)f(b)q(e)g(sequen)o(tial,)f(or)h(can)75
2503 y(b)q(e)h(m)o(ultithreaded,)f(with)g(threads)i(p)q(ossibly)e(executing)i
(concurren)o(tly)m(.)j(Care)c(has)g(b)q(een)h(tak)o(en)f(to)f(mak)o(e)g(MPI)
75 2553 y(\\thread-safe",)h(b)o(y)f(a)o(v)o(oiding)f(the)j(use)f(of)g
(implicit)d(global)h(states.)158 2604 y(The)h(initial)e(allo)q(cation)f(of)i
(pro)q(cesses)j(to)e(an)f(MPI)h(computation)e(and)h(their)h(binding)e(to)i
(ph)o(ysical)f(pro)q(ces-)75 2654 y(sors)k(is)f(not)h(sp)q(eci\014ed)g(b)o(y)
g(the)f(program)f(itself.)22 b(It)16 b(is)f(exp)q(ected)i(that)f(v)o(endors)g
(will)e(pro)o(vide)h(mec)o(hanisms)e(to)75 2704 y(do)h(so)g(either)h(at)e
(load)g(time)g(or)h(at)g(run)g(time.)j(Suc)o(h)d(mec)o(hanisms)e(will)h(allo)
o(w)f(the)i(sp)q(eci\014cation)h(of)e(the)i(initial)p eop
%%Page: 4 5
bop 75 -100 a Fo(4)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g
(COMMUNICA)m(TION)75 45 y Fo(n)o(um)o(b)q(er)i(of)g(required)i(pro)q(cesses,)
i(the)d(co)q(de)h(to)f(b)q(e)g(executed)i(b)o(y)d(eac)o(h)i(initial)d(pro)q
(cess,)j(and)f(the)h(allo)q(cation)75 95 y(of)g(pro)q(cesses)j(to)e(pro)q
(cessors.)34 b(Also,)20 b(the)f(curren)o(t)h(prop)q(osal)e(do)q(es)i(not)e
(pro)o(vide)h(for)f(dynamic)f(creation)i(or)75 145 y(deletion)14
b(of)f(pro)q(cesses)j(during)e(program)e(execution)i(\(the)h(total)e(n)o(um)o
(b)q(er)g(of)g(pro)q(cesses)j(is)e(\014xed\),)g(although)e(it)75
195 y(is)k(in)o(tended)h(to)f(b)q(e)h(consisten)o(t)h(with)e(suc)o(h)h
(extension.)26 b(Finally)m(,)14 b(the)j(curren)o(t)h(prop)q(osal)e(do)q(es)h
(not)f(sp)q(ecify)h(a)75 244 y(naming)d(sc)o(heme)j(for)f(pro)q(cesses.)29
b(W)m(e)16 b(prop)q(ose)i(to)e(alw)o(a)o(ys)g(iden)o(tify)g(pro)q(cesses)j
(according)e(to)f(their)h(relativ)o(e)75 294 y(rank)d(in)f(a)h(con)o(text)g
(\(group\),)g(so)g(that,)f(e\013ectiv)o(ely)m(,)h(pro)q(cesses)j(are)d(iden)o
(ti\014ed)g(b)o(y)g(consecutiv)o(e)h(in)o(tegers.)75 443 y
Fk(1.3.1)55 b(Error)18 b(Handling)75 614 y Fm(Discussion:)34
b Fl(Material)14 b(of)e(this)h(section)g(should)h(b)q(e)e(mo)o(v)o(ed)h(to)f
(a)g(general)h(in)o(tro)q(duction)i(section.)j(This)13 b(material)g(has)75
664 y(not)g(y)o(et)g(b)q(een)h(appro)o(v)o(ed)g(b)o(y)g(MPIF)158
802 y Fo(MPI)i(pro)o(vides)h(the)f(user)h(with)f(reliable)g(message)g
(transmission:)21 b(A)16 b(message)g(sen)o(t)h(is)f(alw)o(a)o(ys)f(receiv)o
(ed)75 852 y(correctly)m(,)20 b(and)f(the)g(user)h(do)q(es)f(not)g(need)g(to)
g(c)o(hec)o(k)g(for)g(transmission)e(errors,)k(time-outs,)d(or)h(other)g
(error)75 902 y(conditions.)f(In)c(other)g(w)o(ords,)g(MPI)g(do)q(es)h(not)f
(pro)o(vide)g(mec)o(hanisms)e(for)h(dealing)g(with)h(failures)f(in)h(the)g
(com-)75 952 y(m)o(unication)9 b(system.)17 b(Where)12 b(the)g(MPI)f
(implemen)o(tation)d(is)j(built)f(on)h(an)g(unreliable)g(underlying)g(mec)o
(hanism,)75 1002 y(then)16 b(it)f(is)g(the)h(job)f(of)g(the)h(implem)o(en)o
(ter)e(of)h(the)h(MPI)f(subsystem)h(to)f(insulate)g(the)h(user)h(from)c(this)
j(unrelia-)75 1051 y(bilit)o(y)m(,)11 b(or)j(to)f(re\015ect)j(unreco)o(v)o
(erable)f(errors)g(as)e(global)f(program)g(failures.)18 b(Similarly)10
b(MPI)k(itself)f(pro)o(vides)h(no)75 1101 y(mec)o(hanisms)e(for)h(handling)g
(no)q(de)h(failures.)158 1157 y(Of)c(course,)i(MPI)e(programs)f(ma)o(y)f
(still)i(b)q(e)h(erroneous.)18 b(A)10 b Fh(program)g(error)f
Fo(can)h(o)q(ccur)i(when)e(an)g(MPI)h(call)75 1207 y(is)i(called)g(with)g(an)
g(incorrect)h(parameter)f(\(non-existing)g(destination)g(in)f(a)h(send)h(op)q
(eration,)f(bu\013er)h(to)q(o)f(small)75 1257 y(in)k(a)f(receiv)o(e)j(op)q
(eration,)e(etc.\))29 b(This)17 b(t)o(yp)q(e)g(of)g(error)h(w)o(ould)e(o)q
(ccur)i(in)f(an)o(y)f(implemen)o(tation.)24 b(In)18 b(addition,)75
1307 y(a)f Fh(resource)i(error)e Fo(ma)o(y)f(o)q(ccur)i(when)h(a)e(program)f
(exceeds)k(the)e(amoun)o(t)e(of)i(a)o(v)n(ailable)d(system)j(resources)75
1357 y(\(n)o(um)o(b)q(er)d(of)h(p)q(ending)g(messages,)g(system)g(bu\013ers,)
h(etc.\).)26 b(The)16 b(o)q(ccurrence)j(of)c(this)h(t)o(yp)q(e)h(of)e(error)i
(dep)q(ends)75 1406 y(on)h(the)g(amoun)o(t)e(of)h(a)o(v)n(ailable)f
(resources)k(in)e(the)g(system)g(and)f(the)i(resource)h(allo)q(cation)c(mec)o
(hanism)g(used;)75 1456 y(this)g(ma)o(y)d(di\013er)j(from)e(system)h(to)h
(system.)22 b(The)16 b(recommended)f(implemen)o(tation)d(pro\014le)k(pro)o
(vides)f(sev)o(eral)75 1506 y(mec)o(hanisms)d(to)h(alleviate)g(the)h(p)q
(ortabilit)o(y)e(problem)g(this)i(represen)o(ts.)21 b(One)14
b(can)g(also)f(write)h Fh(safe)f Fo(programs,)75 1556 y(that)h(are)g(not)g
(sub)r(ject)h(to)f(resource)i(errors.)158 1612 y(All)e(MPI)h(pro)q(cedure)i
(calls)e(return)h(an)f(error)h(parameter)f(that)g(indicates)g(successful)i
(completion)c(of)h(the)75 1662 y(op)q(eration,)f(or)h(the)h(error)f
(condition)g(that)g(o)q(ccurred,)h(otherwise.)158 1718 y(The)e(recommended)f
(implem)o(en)o(tation)e(pro\014le)i(in)g(a)g(POSIX)i(en)o(vironmen)o(t)d(is)h
(for)g(an)o(y)g(MPI)h(routine)g(that)75 1767 y(encoun)o(ters)18
b(a)f(reco)o(v)o(erable)g(error)g(to)g(generate)g(an)g Fj(MPI)g(err)n(or)f
(signal)p Fo(,)h(using)f(a)g(sp)q(ecial)h(signal)e(v)n(alue.)26
b(The)75 1817 y(default)10 b(handler)h(for)g(this)g(signal)f(terminates)g
(the)i(execution)f(of)f(all)g(in)o(v)o(olv)o(ed)g(pro)q(cesses,)j(with)e(a)f
(suitable)h(error)75 1867 y(message)k(b)q(eing)g(returned)i(to)e(the)h(user.)
22 b(Ho)o(w)o(ev)o(er,)16 b(the)g(user)g(can)f(pro)o(vide)g(his)g(or)g(her)h
(o)o(wn)f(signal)f(handling)75 1917 y(routine.)j(In)11 b(particular,)f(the)i
(user)g(can)e(sp)q(ecify)i(a)e(\\no)q(op")g(signal)g(handler,)h(th)o(us)g
(relegating)f(all)g(error)h(handling)75 1967 y(to)j(the)g(user)h(co)q(de,)f
(using)g(the)h(error)f(parameters)g(returned)i(b)o(y)e(the)g(MPI)g(calls.)158
2023 y(MPI)h(calls)g(ma)o(y)e(initiate)h(op)q(erations)h(that)g(con)o(tin)o
(ue)g(async)o(hronously)g(after)g(the)h(call)e(returned.)23
b(Th)o(us,)75 2072 y(the)d(op)q(eration)e(ma)o(y)f(return)k(with)d(a)h(co)q
(de)h(indicating)d(successful)k(completion,)d(y)o(et)h(later)g(cause)h(an)f
(error)75 2122 y(exception)d(to)f(b)q(e)g(raised.)22 b(If)15
b(there)h(is)f(a)g(subsequen)o(t)i(call)d(that)h(relates)h(to)f(the)h(same)e
(op)q(eration)h(\(e.g.,)f(a)h(call)75 2172 y(that)i(v)o(eri\014es)i(that)e
(an)g(async)o(hronous)i(op)q(eration)e(has)h(completed\))f(then)h(the)g
(error)g(parameter)f(asso)q(ciated)75 2222 y(with)e(this)g(call)g(will)f(b)q
(e)i(used)g(to)f(indicate)h(the)g(nature)g(of)e(the)i(error.)24
b(In)15 b(a)g(few)h(cases,)g(the)g(error)g(ma)o(y)e(o)q(ccur)75
2272 y(after)h(all)f(calls)h(that)g(relate)h(to)e(the)i(op)q(eration)f(ha)o
(v)o(e)g(completed,)f(so)h(that)h(no)e(error)i(parameter)f(can)g(b)q(e)h
(used)75 2321 y(to)e(indicate)f(the)i(nature)f(of)f(the)i(error)g(\(e.g.,)d
(an)i(error)h(in)e(a)g(send)i(with)e(the)i(ready)f(mo)q(de\).)j(In)d(suc)o(h)
g(cases,)h(an)75 2371 y(error)g(will)d(b)q(e)j(undetected,)g(if)e(the)i(user)
g(disabled)f(the)g(MPI)g(error)h(signal.)158 2506 y Fm(Discussion:)35
b Fl(The)13 b(alternativ)o(e)i(c)o(hoice)f(is)g(to)f(ha)o(v)o(e)g(fatal)h
(and)g(non-fatal)g(signals.)158 2558 y(One)f(migh)o(t)h(w)o(an)o(t)f
(di\013eren)o(t)h(signals)i(for)c(di\013eren)o(t)j(mo)q(dules.)158
2614 y(The)e(details)i(of)e(suc)o(h)g(prop)q(osal)i(need)f(b)q(e)f(elab)q
(orated)i(in)f(an)f(appropriate)i(\\pro\014le")g(sub)q(committee.)p
eop
%%Page: 5 6
bop 75 -100 a Fg(1.4.)31 b(MESSA)o(GES)1456 b Fo(5)75 45 y
Fn(1.4)70 b(Messages)75 136 y Fo(A)14 b(message)g(consists)g(of)g(an)f
Fj(envelop)n(e)i Fo(and)f Fj(data)p Fo(.)75 252 y Fk(1.4.1)55
b(Data)75 329 y Fo(The)17 b(data)f(part)h(of)f(a)g(message)g(consists)h(of)f
(a)g(sequence)j(of)d(v)n(alues,)g(eac)o(h)h(of)f(a)g(basic)h(datat)o(yp)q(e)g
(in)f(the)h(host)75 379 y(language.)29 b(Th)o(us,)19 b(in)e(F)m(ortran)h(77,)
f(a)h(message)f(consists)i(of)e(a)h(sequence)i(of)d(v)n(alues)h(that)g(are)g
(eac)o(h)g(of)f(t)o(yp)q(e)75 428 y Fi(INTEGER)p Fo(,)12 b
Fi(REAL)p Fo(,)h Fi(DOUBLE)20 b(PRECISION)p Fo(,)12 b Fi(COMPLEX)p
Fo(,)g Fi(DOUBLE)20 b(PRECISION)g(COMPLEX)p Fo(,)12 b Fi(LOGICAL)p
Fo(,)g(or)i(\(length)g(1\))75 478 y Fi(CHARACTER)p Fo(.)i(A)i(message)f(ma)o
(y)f(also)h(con)o(tain)h(a)f(v)n(alue)h(of)f(t)o(yp)q(e)h Fi(BYTE)p
Fo(,)f(whic)o(h)h(consists)h(of)e(8)h(binary)f(digits.)75 528
y(A)e(b)o(yte)g(is)g(di\013eren)o(t)h(from)d(a)i(c)o(haracter.)23
b(Di\013eren)o(t)15 b(mac)o(hines)f(ma)o(y)f(ha)o(v)o(e)i(di\013eren)o(t)h
(represen)o(tations)h(for)d(the)75 578 y(same)e(c)o(haracter,)i(or)f(ma)o(y)e
(use)i(more)f(than)h(one)g(b)o(yte)g(to)g(represen)o(t)i(a)e(c)o(haracter.)19
b(On)13 b(the)g(other)h(hand,)e(a)h(b)o(yte)75 628 y(has)h(the)h(same)e
(binary)g(v)n(alue)g(on)h(an)o(y)f(mac)o(hine.)k(A)d(message)g(ma)o(y)d(mix)h
(v)n(alues)i(of)f(di\013eren)o(t)i(t)o(yp)q(es.)158 756 y Fm(Missing:)158
802 y Fl(Need)g(to)h(agree)g(on)f(the)h(C)f(t)o(yp)q(es)h(\(including)i
(handling)g(of)e(signed/unsign)q(ed\),)i(and)e(on)g(F)m(ortran)g(90)f(t)o(yp)
q(es)h(\(in-)75 847 y(cluding)f(handling)h(of)d(kinds\).)75
1046 y Fk(1.4.2)55 b(En)n(v)n(elop)r(e)75 1123 y Fo(The)14
b(follo)o(wing)e(information)f(is)i(asso)q(ciated)i(with)e(eac)o(h)i
(message:)75 1205 y Fh(source)20 b Fo(The)14 b(rank)g(of)f(the)i(sending)f
(pro)q(cess)75 1288 y Fh(destinatio)o(n)k Fo(The)c(rank)g(of)f(the)i
(receiving)f(pro)q(cess)75 1370 y Fh(tag)20 b Fo(User)15 b(de\014ned)75
1453 y Fh(con)o(text)k Fo(handle)158 1536 y(The)14 b(range)g(of)f(v)n(alid)f
(v)n(alues)h(for)g(the)h Fh(source)f Fo(and)g Fh(destination)d
Fo(\014elds)k(is)f Fi(0)22 b(...)43 b(n-1)p Fo(,)12 b(where)j
Fi(n)e Fo(is)g(the)75 1585 y(n)o(um)o(b)q(er)e(of)h(pro)q(cesses)j(in)c(the)i
(sp)q(eci\014ed)h(con)o(text.)k Fi(source)j(=)g(destination)10
b Fo(is)i(allo)o(w)o(ed:)k(a)c(pro)q(cess)i(can)e(send)75 1635
y(a)i(message)f(to)h(itself.)158 1685 y(The)g(ranges)h(of)e(v)n(alid)f(v)n
(alues)i(for)f Fi(tag)g Fo(is)h(implem)o(en)o(tation)d(dep)q(enden)o(t,)k
(and)f(can)g(b)q(e)g(found)f(b)o(y)h(calling)e(a)75 1735 y(suitable)i(query)g
(function,)f(as)h(describ)q(ed)i(in)d(Section)i Fh(??)p Fo(.)158
1785 y(The)f Fi(tag)e Fo(\014eld)i(can)f(b)q(e)h(arbitrarily)e(set)j(b)o(y)e
(the)h(application,)d(and)i(can)h(b)q(e)g(used)g(to)f(distinguish)g
(di\013eren)o(t)75 1835 y(messages.)158 1884 y Fi(Context)f
Fo(should)i(b)q(e)h(a)e(con)o(text)i(shared)g(b)o(y)e(b)q(oth)h(source)h(and)
f(destination.)158 1934 y(The)g(actual)g(mec)o(hanism)d(used)k(to)f(asso)q
(ciate)g(an)g(en)o(v)o(elop)q(e)g(with)g(a)f(message)h(is)g(implem)o(en)o
(tation)d(dep)q(en-)75 1984 y(den)o(t;)h(some)f(of)g(the)i(information)8
b(\(e.g.,)j Fh(sender)f Fo(or)i Fh(receiv)o(er)p Fo(\))e(ma)o(y)g(b)q(e)i
(implicit,)d(and)i(need)i(not)f(b)q(e)g(explicitly)75 2034
y(carried)j(b)o(y)e(a)h(message.)75 2171 y Fn(1.5)70 b(Comm)n(unication)20
b(Bu\013ers)75 2262 y Fo(The)h(basic)f(p)q(oin)o(t)g(to)g(p)q(oin)o(t)g(comm)
o(unicatio)o(n)e(op)q(erations)i(are)h Fh(send)e Fo(and)h Fh(receiv)o(e)p
Fo(.)36 b(A)20 b Fh(send)f Fo(op)q(eration)75 2312 y(creates)c(a)d(message;)g
(the)i(message)e(data)h(is)f(tak)o(en)h(from)e(the)j Fh(send)f(bu\013er)p
Fo(.)j(A)d Fh(receiv)o(e)e Fo(op)q(eration)i(consumes)75 2361
y(a)j(message;)h(the)g(message)f(data)g(is)g(put)h(in)o(to)e(the)i
Fh(receiv)o(e)h(bu\013er)p Fo(.)23 b(The)17 b(sp)q(eci\014cation)g(of)f(send)
h(or)g(receiv)o(e)75 2411 y(bu\013ers)e(uses)g(the)g(same)e(syn)o(tax.)75
2527 y Fk(1.5.1)55 b(Comm)n(unication)21 b(bu\013er)d(comp)r(onen)n(ts)75
2604 y Fo(A)d(bu\013er)g(consists)h(of)d(a)i(sequence)h(of)e
Fh(bu\013er)h(comp)q(onen)o(ts)p Fo(.)j(Eac)o(h)d(comp)q(onen)o(t)e(consists)
j(of)e(a)g(sequence)j(of)75 2654 y(\(not)f(necessarily)g(distinct\))g(v)n
(ariables)f(of)g(the)h(same)f(basic)h(datat)o(yp)q(e.)23 b(The)16
b(datat)o(yp)q(e)g(of)f(suc)o(h)h(a)g(v)n(ariable)e(is)75 2704
y(sp)q(eci\014ed)g(b)o(y)f(a)f(state)i(parameter.)j(The)d(p)q(ossible)f(v)n
(alues)f(for)h(F)m(ortran)f(are)h Fi(MPI)p 1357 2704 14 2 v
15 w(INT)p Fo(,)f Fi(MPI)p 1528 2704 V 15 w(REAL)p Fo(,)g Fi(MPI)p
1721 2704 V 15 w(DOUBLE)p Fo(,)p eop
%%Page: 6 7
bop 75 -100 a Fo(6)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g
(COMMUNICA)m(TION)75 45 y Fi(MPI)p 144 45 14 2 v 15 w(COMPLEX)p
Fo(,)k Fi(MPI)p 409 45 V 15 w(DCOMPLEX)p Fo(,)g Fi(MPI)p 696
45 V 15 w(LOGICAL)p Fo(,)h Fi(MPI)p 962 45 V 15 w(CHAR)g Fo(and)h
Fi(MPI)p 1237 45 V 15 w(BYTE)p Fo(,)f(corresp)q(onding)i(to)f(datat)o(yp)q
(es)75 95 y Fi(INTEGER)p Fo(,)e Fi(REAL)p Fo(,)g Fi(DOUBLE)j(PRECISION)p
Fo(,)c Fi(COMPLEX)p Fo(,)h Fi(DOUBLE)i(PRECISION)g(COMPLEX)p
Fo(,)e Fi(LOGICAL)p Fo(,)g Fi(CHARACTER)75 145 y Fo(and)j(to)g(un)o(t)o(yp)q
(ed)g Fi(BYTE)p Fo(.)f(The)h(p)q(ossible)g(v)n(alues)g(for)g(C)g(are)g
Fi(MPI)p 1162 145 V 15 w(CHAR)p Fo(,)f Fi(MPI)p 1363 145 V
15 w(SHORT)p Fo(,)f Fi(MPI)p 1585 145 V 15 w(INT)p Fo(,)h Fi(MPI)p
1764 145 V 15 w(LONG)p Fo(,)75 195 y Fi(MPI)p 144 195 V 15
w(FLOAT)p Fo(,)12 b Fi(MPI)p 359 195 V 15 w(DOUBLE)h Fo(and)h
Fi(MPI)p 666 195 V 15 w(BYTE)p Fo(.)158 325 y Fm(Discussion:)158
372 y Fl(Ma)o(y)g(need)g(to)g(tak)o(e)g(care)f(of)h(kinds)h(in)f(F)m(ortran)g
(90)g(and)g(of)g(signed/unsign)q(ed)j(in)d(C.)f(More)h(elab)q(orate)h
(de\014nitions)75 418 y(b)q(elong)g(to)e(the)g(language)i(binding.)158
552 y Fo(There)g(are)f(\014v)o(e)g(kinds)g(of)g(bu\013er)g(comp)q(onen)o(ts:)
75 670 y Fh(Con)o(tiguous)f(comp)q(onen)o(t)75 750 y Fo(A)h(sequence)i(of)d
(con)o(tiguous)h(v)n(alues)g(of)f(the)h(same)f(basic)h(t)o(yp)q(e,)g(sp)q
(eci\014ed)i(b)o(y)75 839 y Fh(start)k Fo(Initial)12 b(elemen)o(t)75
930 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p
Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 1020 y Fh(datat)o(yp)q(e)19
b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1109 y(Th)o(us,)h(if)e
Fi(A)i Fo(is)f(an)h(arra)o(y)f(of)g(in)o(tegers,)h(then)h Fi(A\(1\),)21
b(4,)g(MPI)p 1127 1109 V 15 w(INT)13 b Fo(is)h(the)g(bu\013er)h(comp)q(onen)o
(t)e(with)g(en)o(tries)75 1159 y Fi(A\(1\),)21 b(A\(2\),)g(A\(3\),)f(A\(4\))p
Fo(.)75 1277 y Fh(V)l(ector)15 b(comp)q(onen)o(t)75 1357 y
Fo(A)f(sequence)j(of)c(equally)h(spaced)h(and)f(equally)f(sized)i(blo)q(c)o
(ks)f(of)g(elemen)o(ts)g(of)g(the)g(same)g(basic)g(t)o(yp)q(e,)g(sp)q
(eci\014ed)75 1407 y(b)o(y)75 1495 y Fh(start)20 b Fo(Initial)12
b(elemen)o(t)75 1586 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14
b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75
1677 y Fh(stride)18 b Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(b)q(et)o(w)o(een)h
(the)g(start)f(of)f(eac)o(h)i(blo)q(c)o(k)75 1767 y Fh(len)o(blk)j
Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\()p
Fi(lenblk)c Fd(>)i Fi(0)p Fo(\))75 1858 y Fh(datat)o(yp)q(e)19
b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1947 y(Assume)20 b(that)h(an)f(elemen)
o(t)g(of)f(t)o(yp)q(e)i Fi(datatype)e Fo(o)q(ccupies)i Fi(m)f
Fo(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75
1996 y(mac)o(hine,)11 b(the)i(bu\013er)h(comp)q(onen)o(t)d(describ)q(ed)k(b)o
(y)d(these)i(\014v)o(e)f(parameters)f(consists)i(of)e(the)h
Fi(count)e Fo(v)n(ariables)h(at)75 2046 y(addresses)k Fi(start,)21
b(start+m,)f(...)43 b(,)21 b(start+m*\(lenblk-1\),)d(start+m*stride,)75
2096 y(start+m*\(stride+1)o(\),)h(...,)i(start+m*\(stride+l)o(enbl)o(k-1\),)d
(...)p Fo(.)i(F)m(or)14 b(example,)f(if)h Fi(A)g Fo(is)h(an)f(arra)o(y)h(of)
75 2146 y(in)o(tegers,)f(then)137 2234 y Fe(\017)21 b Fi(A\(1\),)g(5,)g(3,)g
(2,)h(MPI)p 575 2234 V 15 w(INT)c Fo(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e
(with)h(en)o(tries)h Fi(A\(1\),)h(A\(2\),)f(A\(4\),)h(A\(5\),)179
2284 y(A\(7\))p Fo(.)137 2375 y Fe(\017)g Fi(A\(1\),)g(5,)g(0,)g(2,)h(MPI)p
575 2375 V 15 w(INT)c Fo(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e(with)h(en)o
(tries)h Fi(A\(1\),)h(A\(2\),)f(A\(1\),)h(A\(2\),)179 2425
y(A\(1\))p Fo(.)137 2515 y Fe(\017)g Fi(A\(10\),)f(5,)i(-1,)f(2,)g(MPI)p
618 2515 V 15 w(INT)9 b Fo(is)g(the)h(bu\013er)g(comp)q(onen)o(t)f(with)g(en)
o(tries)h Fi(A\(10\),)21 b(A\(11\),)f(A\(9\),)h(A\(10\),)179
2565 y(A\(8\))p Fo(.)158 2654 y Fi(Contiguous)13 b Fo(is)h(a)g(sp)q(ecial)h
(case)h(of)e Fi(vector)p Fo(.)19 b(A)c(v)o(ector)g(bu\013er)h(comp)q(onen)o
(t)d(can)i(b)q(e)h(used)f(to)f(extract)i(an)75 2704 y(arbitrary)e(submatrix)e
(of)i(a)f(t)o(w)o(o-dimensional)e(matrix.)p eop
%%Page: 7 8
bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1091
b Fo(7)75 45 y Fh(Indexed)15 b(comp)q(onen)o(t)75 122 y Fo(A)f(sequence)i(of)
d(elemen)o(ts)h(of)f(the)i(same)e(basic)h(t)o(yp)q(e,)g(sp)q(eci\014ed)h(b)o
(y)75 207 y Fh(start)20 b Fo(initial)12 b(elemen)o(t)75 291
y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p
Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 376 y Fh(arra)o(y)p 188
376 15 2 v 17 w(of)p 244 376 V 16 w(indices)19 b Fo(Arra)o(y)14
b(of)f(displacemen)o(ts)h(of)f(the)i(elemen)o(ts)e(in)h(the)h(bu\013er)g
(comp)q(onen)o(ts,)e(relativ)o(e)h(to)f(the)179 425 y(initial)f(elemen)o(t.)
75 510 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158
594 y(F)m(or)f(example,)f(if)h Fi(A)g Fo(is)g(an)g(arra)o(y)g(of)g(in)o
(tegers,)h(and)g Fi(I)f Fo(is)g(an)g(arra)o(y)h(with)f(v)n(alues)g
Fi(2,)21 b(-1,)g(2,)h(1)12 b Fo(then)h Fi(A\(3\),)75 644 y(4,)21
b(I,)h(MPI)p 275 644 14 2 v 15 w(INT)13 b Fo(is)h(the)g(bu\013er)h(comp)q
(onen)o(t)e(with)h(en)o(tries)h Fi(A\(5\),)21 b(A\(2\),)f(A\(5\),)h(A\(4\))p
Fo(.)158 694 y Fi(Vector)13 b Fo(can)h(b)q(e)g(though)o(t)g(of)f(as)h(a)g(sp)
q(ecial)g(case)h(of)e Fi(indexed)g Fo(in)g(whic)o(h)h(the)g(indices)h(are)f
(generated)h(with)75 744 y(a)f(constan)o(t)g(stride.)75 854
y Fh(Heterogeneous)f(v)o(ector)i(\(h)o(v)o(ector\))f(comp)q(onen)o(t)75
931 y Fo(Same)d(as)h Fi(vector)p Fo(,)f(except)j(that)e Fi(stride)f
Fo(is)h(giv)o(en)g(in)f(m)o(ultiple)f(of)i(b)o(ytes,)h(rather)g(than)f(m)o
(ultiples)e(of)i(elemen)o(ts:)75 1024 y Fh(start)20 b Fo(Initial)12
b(elemen)o(t)75 1108 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14
b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75
1193 y Fh(stride)18 b Fo(Num)o(b)q(er)c(of)f(b)o(ytes)i(b)q(et)o(w)o(een)g
(the)g(start)f(of)f(eac)o(h)i(blo)q(c)o(k)75 1278 y Fh(len)o(blk)j
Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\()p
Fi(lenblk)c Fd(>)i Fi(0)p Fo(\))75 1362 y Fh(datat)o(yp)q(e)19
b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1455 y(Assume)20 b(that)h(an)f(elemen)
o(t)g(of)f(t)o(yp)q(e)i Fi(datatype)e Fo(o)q(ccupies)i Fi(m)f
Fo(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75
1505 y(mac)o(hine,)15 b(the)i(bu\013er)g(comp)q(onen)o(t)f(describ)q(ed)i(b)o
(y)e(these)i(\014v)o(e)e(parameters)h(consists)g(of)f(the)g
Fi(count)g Fo(v)n(ariables)75 1554 y(at)c(addresses)j Fi(start,)20
b(start+m,)h(...)43 b(,)21 b(start+m*\(lenblk-1\))o(,)e(start+stride,)g
(start+stride+m,)75 1604 y(...,)i(start+stride+m*\(l)o(enblk)o(-1\),)d(...)p
Fo(.)158 1654 y(Consider)c(a)g(C)g(arra)o(y)f(declared)i(as)75
1748 y Fi(struct)21 b({)140 1798 y(char)g(info[3];)140 1847
y(short)g(number;)140 1897 y(double)g(val;)228 1947 y(})65
b(a[100])158 2040 y Fo(Assume)19 b(that)f(all)g(structure)j(comp)q(onen)o(ts)
d(are)h(b)o(yte)h(aligned,)e(a)h Fi(short)e Fo(o)q(ccupies)j(t)o(w)o(o)e(b)o
(ytes,)j(and)d(a)75 2090 y Fi(double)i Fo(o)q(ccupies)i(eigh)o(t)e(b)o(ytes.)
40 b(Then)21 b(the)h(bu\013er)g(comp)q(onen)o(t)e Fi(&a,)h(100,)g(13,)g(2,)g
(MPI)p 1610 2090 V 16 w(CHAR)f Fo(extracts)75 2139 y(the)i(\014rst)h(t)o(w)o
(o)e(c)o(haracters)i(of)e(eac)o(h)h(structure.)43 b(The)22
b(bu\013er)h(comp)q(onen)o(t)e Fi(&\(a.number\),)e(100,)i(13,)g(1,)75
2189 y(MPI)p 144 2189 V 15 w(SHORT)16 b Fo(extracts)i(the)f
Fi(short)f Fo(\014eld)g(of)g(eac)o(h)i(structure;)h(and)e(the)g(bu\013er)h
(comp)q(onen)o(t)e Fi(&\(a.val\),)k(100,)75 2239 y(13,)h(1)e
Fo(extracts)i(the)f Fi(double)e Fo(\014eld)h(of)g(eac)o(h)h(structure.)36
b(Th)o(us,)21 b(heterogeneous)g(v)o(ectors)f(can)g(b)q(e)g(used)g(to)75
2289 y(extract)15 b(v)o(ectors)g(of)e(homogeneous)g(v)n(alues)h(from)e
(heterogeneous)k(data.)158 2339 y Fi(Vector)d Fo(is)g(a)h(sp)q(ecial)g(case)h
(of)e Fi(heterogeneous)19 b(vector)p Fo(.)75 2449 y Fh(Heterogeneous)13
b(indexed)h(\(hindexed\))f(comp)q(onen)o(t)75 2526 y Fo(Same)g(as)h(indexed,)
g(except)h(that)f(displacemen)o(ts)f(are)i(in)e(b)o(ytes,)h(rather)h(than)f
(m)o(ultiples)e(of)h(elemen)o(t)g(size.)75 2619 y Fh(start)20
b Fo(initial)12 b(elemen)o(t)75 2704 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14
b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))p
eop
%%Page: 8 9
bop 75 -100 a Fo(8)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g
(COMMUNICA)m(TION)75 45 y Fh(arra)o(y)p 188 45 15 2 v 17 w(of)p
244 45 V 16 w(indices)19 b Fo(Arra)o(y)12 b(of)g(b)o(yte)g(displacemen)o(ts)g
(of)g(the)g(elemen)o(ts)h(in)e(the)i(bu\013er)g(comp)q(onen)o(ts,)f(relativ)o
(e)g(to)179 95 y(the)i(initial)e(elemen)o(t.)75 189 y Fh(datat)o(yp)q(e)19
b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 292 y(F)m(or)18 b(example,)f(if)h
Fi(a)g Fo(is)g(the)h(C)f(arra)o(y)g(of)f(the)i(previous)g(example,)e(and)i
Fi(I)e Fo(is)i(an)f(arra)o(y)g(with)f(v)n(alues)h Fi(0,)75
342 y(13,)j(13,)g(27,)h(26)16 b Fo(then)i Fi(&a,)k(5,)f(I,)g(MPI)p
782 342 14 2 v 15 w(CHAR)c Fo(is)g(the)h(bu\013er)h(comp)q(onen)o(t)d(with)h
(en)o(tries)i Fi(a[0].info[0],)75 392 y(a[1].info[0],)g(a[1].info[0],)g
(a[1].info[1],)g(a[1].info[0])p Fo(.)158 444 y(Lik)o(e)e Fi(heterogeneous)i
(vector)p Fo(,)d Fi(heterogeneous)j(indexed)c Fo(extends)k(the)e
Fi(indexed)f Fo(facilit)o(y)f(so)i(as)g(to)75 494 y(allo)o(w)12
b(the)j(extraction)f(of)f(homogeneous)g(data)h(from)e(a)h(heterogeneous)j(la)
o(y)o(out.)158 625 y Fm(Discussion:)158 674 y Fl(The)d(use)h(of)e
(displacemen)o(ts)k(in)e(indexed)h(bu\013er)f(comp)q(onen)o(ts)h(has)e(the)h
(same)f(e\013ect)g(as)h(indirect)h(addressing)g(in)o(to)75
720 y(an)d(arra)o(y)g(with)g(initial)j(index)e(zero.)j(I.e.,)11
b(if)h Ff(a)f Fl(is)h(an)g(arra)o(y)g(with)g(initial)j(index)e(zero,)e(then)h
(a)g(bu\013er)h(comp)q(onen)o(t)g(de\014ned)75 765 y(b)o(y)g(parameters)g
Ff(a[0],)k(N,)i(i,)g(type)11 b Fl(extract)h(the)h(elemen)o(ts)g
Ff(a[i[0]],)j(a[i[1]],)h(...,)h(a[i[n]])p Fl(.)c(This)f(\014ts)f(w)o(ell)h(C)
75 811 y(usage)g(but)f(ill-\014ts)i(F)m(ortran)e(usage)h(\(initial)h(index)g
(one\).)i(W)m(e)c(to)q(ok)h(a)f(decision)i(of)d(principle)k(to)d(force)g
(\\start)g(from)f(zero")75 857 y(index)j(arithmetic)f(on)g(F)m(ortran)f
(users,)h(rather)f(than)h(ha)o(v)o(e)f(di\013eren)o(t)i(index)g(arithmetic)f
(in)g(the)f(C)g(and)h(F)m(ortran)f(binding)75 902 y(of)h(MPI.)158
951 y(The)19 b(use)f(of)h(b)q(oth)g(bu\013er)h(comp)q(onen)o(ts)f(with)h(b)o
(yte)e(and)i(elemen)o(t)f(size)g(displacemen)o(ts)j(is)d(motiv)n(ated)h(b)o
(y)f(the)75 996 y(observ)n(ation)c(that)f(some)g(applications)j(require)d
(the)g(more)f(general)i(b)o(yte)f(displacemen)o(t,)h(but)f(that)g(most)f
(applicatio)q(ns)75 1042 y(w)o(ould)d(use)g(elemen)o(t)g(size)h(displacemen)o
(t,)h(whic)o(h)e(is)g(m)o(uc)o(h)g(more)g(natural)g(and)g(con)o(v)o(enien)o
(t)i(\(and)e(mac)o(hine)h(indep)q(enden)o(t\).)75 1088 y(Th)o(us,)f(rather)f
(than)h(forcing)g(all)g(users)g(to)f(do)g(b)o(yte)h(arithmetics,)h(w)o(e)e
(decided)h(to)f(pro)o(vide)i(separately)g(the)e(more)g(adv)n(anced)75
1133 y(function)14 b(to)f(\\exp)q(ert)h(programmers".)158 1182
y(Leslie)g(Hart)f(prop)q(oses)i(to)d(replace)j(\\heterogeneous")g(with)e
(\\general".)75 1396 y Fk(1.5.2)55 b(Bu\013er)18 b(op)r(erations)75
1478 y Fo(A)c(bu\013er)i(is)e(describ)q(ed)i(b)o(y)e(an)g(opaque)g
Fh(bu\013er)h(descriptor)c Fo(ob)r(ject)k(accessed)h(via)e(a)g
Fh(bu\013er)h(handle)p Fo(.)i(Suc)o(h)75 1528 y(an)f(ob)r(ject)g(is)g
(created)h(b)o(y)f Fi(MPI)p 582 1528 V 15 w(CREATE)p 729 1528
V 15 w(BUFFER)p Fo(.)e(It)i(is)f(asso)q(ciated)i(with)f(successiv)o(e)i
(bu\013er)f(comp)q(onen)o(ts)e(b)o(y)75 1577 y(calling)g(in)i(succession)h
(one)f(of)f(the)i(functions)e Fi(MPI)p 914 1577 V 15 w(APPEND)p
1061 1577 V 15 w(xxx)p Fo(,)g(where)i Fi(xxx)e Fo(iden)o(ti\014es)h(the)h(t)o
(yp)q(e)f(of)f(bu\013er)75 1627 y(comp)q(onen)o(t)e(app)q(ended)h(to)f(the)h
(bu\013er.)20 b(It)14 b(b)q(ecomes)h(a)o(v)n(ailable)d(for)i(use)h(in)f(comm)
o(unicatio)o(n)e(op)q(erations)i(once)75 1677 y(committed)c(b)o(y)h(the)i
(function)e Fi(MPI)p 635 1677 V 15 w(COMMIT)p 782 1677 V 15
w(BUFFER)p Fo(.)f(Once)j(committed,)c(the)k(descriptor)g(cannot)f(b)q(e)g(mo)
q(di\014ed)75 1727 y(an)o(y)h(more.)158 1780 y(Note)18 b(that)g(the)g(commit)
c(op)q(eration)k(do)q(es)g(not)g(create)h(y)o(et)f(a)f(message,)h(and)f(do)q
(es)h(not)g(bind)f(the)h(data)75 1829 y(to)d(b)q(e)h(sen)o(t.)23
b(It)16 b(is)f(a)g(commit)d(for)j(the)h(bu\013er)g(descriptor,)h(not)e(a)g
(commit)d(for)j(the)h(bu\013er)h(con)o(ten)o(t.)23 b(A)15 b(bu\013er)75
1879 y(descriptor)f(is)e(destro)o(y)o(ed)i(after)f(the)g(completion)d(of)i
(the)h(\014rst)h(comm)o(unicati)o(on)c(op)q(eration)i(that)h(uses)g(it,)f(if)
g(it)g(is)75 1929 y(ephemeral,)g(or)h(after)g(it)g(is)g(freed)h(b)o(y)e(a)h
(call)f(to)h Fi(MPI)p 899 1929 V 15 w(FREE)f Fo(and)h(an)o(y)f(p)q(ending)i
(comm)o(uni)o(cation)c(op)q(eration)j(that)75 1979 y(uses)j(it)e(has)h
(completed,)f(if)f(it)i(is)f(p)q(ersisten)o(t.)22 b(After)15
b(the)h(bu\013er)f(descriptor)h(ob)r(ject)f(has)g(b)q(een)h(destro)o(y)o(ed,)
f(the)75 2029 y(handle)f(is)f(unde\014ned.)158 2117 y Fh(MPI)p
257 2117 15 2 v 17 w(CREA)l(TE)p 471 2117 V 19 w(BUFFER\()i(bu\013er,)g(p)q
(ersistence)e(\))158 2205 y Fo(Create)i(a)e(new)i(bu\013er)g(descriptor)g(ob)
r(ject.)k(The)14 b(parameters)g(are:)75 2296 y Fh(OUT)i(bu\013er)j
Fo(bu\013er)c(handle)75 2391 y Fh(IN)h(p)q(ersistence)i Fo(bu\013er)11
b(p)q(ersistence)h(\(state)e(t)o(yp)q(e)g(with)f(p)q(ossible)g(v)n(alues)g
Fi(MPI)p 1350 2391 14 2 v 15 w(PERSISTENT)e Fo(or)j Fi(MPI)p
1705 2391 V 15 w(EPHEMERAL)p Fo(\).)158 2561 y Fm(Discussion:)158
2609 y Fl(Ma)o(y)j(w)o(an)o(t)g(to)g(pro)o(vide)i(information)g(on)e(maxim)o
(um)h(ob)r(ject)g(size,)f(and)h(p)q(erhaps)g(user)f(space)h(for)f(the)g(ob)r
(ject.)158 2658 y(Leslie)20 b(Hart)e(prop)q(oses)i Ff(MPI)p
592 2658 12 2 v 13 w(CREATE)p 725 2658 V 12 w(BUFFER)p 857
2658 V 12 w(DESC)p Fl(,)c(rather)j(than)g Ff(MPI)p 1252 2658
V 13 w(CREATE)p 1385 2658 V 12 w(BUFFER)p Fl(.)c(More)k(accurate,)h(but)75
2704 y(longer.)e(P)o(erhaps)c Ff(MPI)p 414 2704 V 13 w(CREATE)p
547 2704 V 12 w(BUFDESC)p Fl(?)p eop
%%Page: 9 10
bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1091
b Fo(9)158 45 y Fh(MPI)p 257 45 15 2 v 17 w(APPEND)p 481 45
V 17 w(CONTIGUOUS\()15 b(bu\013er,)g(start,)g(coun)o(t,)f(datat)o(yp)q(e\))
158 130 y Fo(App)q(end)h(a)e(blo)q(c)o(k)h(comp)q(onen)o(t)f(to)h(bu\013er.)
19 b(The)14 b(parameters)g(are:)75 208 y Fh(INOUT)i(bu\013er)j
Fo(bu\013er)c(handle)75 289 y Fh(IN)h(start)k Fo(bu\013er)15
b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))75
369 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o
(teger\))75 450 y Fh(IN)i(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14
b(iden)o(ti\014er)g(\(status\))158 563 y Fh(MPI)p 257 563 V
17 w(APPEND)p 481 563 V 17 w(VEC\()j(bu\013er,)d(coun)o(t,)h(stride,)f(len)o
(blk,)f(datat)o(yp)q(e)i(\))158 648 y Fo(App)q(end)g(a)e(v)o(ector)i
(bu\013er)g(comp)q(onen)o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)
75 726 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 807
y Fh(IN)h(start)k Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h
(\(c)o(hoice\))75 887 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14
b(of)f(elemen)o(ts)h(\(in)o(teger\))75 967 y Fh(IN)i(stride)j
Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(b)q(et)o(w)o(een)j(the)e(start)h(of)e
(eac)o(h)h(blo)q(c)o(k)g(\(in)o(teger\))75 1048 y Fh(IN)i(len)o(blk)i
Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h)i(blo)q(c)o(k)e(\(in)o
(teger\))75 1128 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14
b(iden)o(ti\014er)g(\(status\))158 1242 y Fh(MPI)p 257 1242
V 17 w(APPEND)p 481 1242 V 17 w(INDEXED\()i(bu\013er,)f(start,)g(coun)o(t,)f
(arra)o(y)p 1294 1242 V 17 w(of)p 1350 1242 V 17 w(indices,)f(datat)o(yp)q
(e\))158 1327 y Fo(App)q(end)i(an)e(indexed)i(bu\013er)g(comp)q(onen)o(t)e
(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 1405 y Fh(INOUT)i
(bu\013er)j Fo(bu\013er)c(handle)75 1485 y Fh(IN)h(start)k
Fo(initial)12 b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75
1566 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o
(teger\))75 1646 y Fh(IN)i(arra)o(y)p 259 1646 V 17 w(of)p
315 1646 V 17 w(indices)i Fo(arra)o(y)c(of)f(displacemen)o(ts)g(of)h(en)o
(tries)h(relativ)o(e)e(to)h(start)h(\(arra)o(y)e(of)h(in)o(tegers\))75
1727 y Fh(IN)i(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g
(\(status\))158 1805 y(The)g(index)f(v)n(alues)g(used)h(to)f(create)i(this)f
(comp)q(onen)o(t)e(are)i(the)g(v)n(alues)f(pro)o(vided)g(at)g(the)h(call;)e
(subsequen)o(t)75 1854 y(up)q(dates)j(of)e(the)i(arra)o(y)e(of)h(indices)g
(will)e(not)i(a\013ect)h(the)f(bu\013er)h(descriptor)g(comp)q(onen)o(t.)158
1940 y Fh(MPI)p 257 1940 V 17 w(APPEND)p 481 1940 V 17 w(HVEC\()i(bu\013er,)e
(start,)g(coun)o(t,)f(stride,)g(len)o(blk,)f(datat)o(yp)q(e)i(\))158
2025 y Fo(App)q(end)g(a)e(heterogeneous)j(v)o(ector)f(bu\013er)g(comp)q(onen)
o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 2103
y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 2183 y Fh(IN)h(start)k
Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))75
2264 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o
(teger\))75 2344 y Fh(IN)i(stride)j Fo(Num)o(b)q(er)13 b(of)h(b)o(ytes)g(b)q
(et)o(w)o(een)h(the)g(start)f(of)g(eac)o(h)g(blo)q(c)o(k)g(\(in)o(teger\))75
2425 y Fh(IN)i(len)o(blk)i Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h)
i(blo)q(c)o(k)e(\(in)o(teger\))75 2505 y Fh(IN)j(datat)o(yp)q(e)k
Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 2618
y Fh(MPI)p 257 2618 V 17 w(APPEND)p 481 2618 V 17 w(HINDEXED\()i(bu\013er,)f
(start,)g(coun)o(t,)g(arra)o(y)p 1332 2618 V 16 w(of)p 1387
2618 V 17 w(indices,)f(datat)o(yp)q(e\))158 2704 y Fo(App)q(end)h(an)e
(heterogeneous)j(indexed)f(bu\013er)g(comp)q(onen)o(t)e(to)h(bu\013er.)19
b(The)14 b(parameters)g(are:)p eop
%%Page: 10 11
bop 75 -100 a Fo(10)716 b Fg(CHAPTER)14 b(1.)27 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fh(INOUT)i(bu\013er)j
Fo(bu\013er)c(handle)75 124 y Fh(IN)h(start)k Fo(initial)12
b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75 203 y Fh(IN)j(coun)o(t)j
Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 282
y Fh(IN)i(arra)o(y)p 259 282 15 2 v 17 w(of)p 315 282 V 17
w(indices)i Fo(arra)o(y)c(of)f(b)o(yte)h(displacemen)o(ts)g(of)f(en)o(tries)i
(relativ)o(e)f(to)f(start)i(\(arra)o(y)f(of)f(in)o(tegers\))75
361 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g
(\(status\))158 471 y Fh(MPI)p 257 471 V 17 w(COMMIT)p 485
471 V 19 w(BUFFER\()h(bu\013erin)e(\))158 557 y Fo(Commi)o(t)7
b(a)j(bu\013er)h(descriptor)g(ob)r(ject.)18 b(The)10 b(ob)r(ject)h(cannot)f
(b)q(e)g(c)o(hanged)h(an)o(ymore)d(after)i(it)g(is)g(committed.)75
606 y(A)h(bu\013er)g(descriptor)h(ob)r(ject)g(can)f(b)q(e)g(used)g(for)g
(comm)o(uni)o(cation)d(only)i(after)g(it)h(is)f(committed.)15
b(The)c(parameters)75 656 y(are:)75 731 y Fh(INOUT)16 b(bu\013er)j
Fo(bu\013er)c(descriptor)g(handle)158 885 y Fm(Discussion:)158
930 y Fl(The)i(commit)h(function)g(ma)o(y)g(do)f(prepro)q(cessing)j(of)d(the)
g(bu\013er)h(descriptor)h(to)e(simplify)i(its)f(use)f(and)h(commit)75
976 y(resources)12 b(in)g(the)g(comm)o(unication)i(subsystem)e(for)f(its)h
(use.)k(The)c(commit)f(function)i(ma)o(y)f(generate)g(a)f(new)g(ob)r(ject)g
(and,)75 1022 y(hence,)i(mo)q(dify)h(the)g(ob)r(ject)f(handle.)158
1067 y(The)18 b(curren)o(t)g(prop)q(osal)h(do)q(es)g(not)f(allo)o(w)h(to)e
(create)h(a)g(bu\013er)g(descriptor,)i(use)e(it)h(in)f(a)g(comm)o(unication)i
(after)75 1113 y(committed,)14 b(then)g(mo)q(dify)g(it)g(partially)i(and)e
(use)g(it)g(for)f(another)h(comm)o(unication)i({)d(a)h(new)f(bu\013er)h
(descriptor)h(has)f(to)75 1159 y(b)q(e)h(created.)21 b(Of)13
b(course,)i(the)g(same)f(bu\013er)h(descriptor)h(can)f(b)q(e)f(reused)h(man)o
(y)g(times)g(in)g(successiv)o(e)h(comm)o(unications)75 1204
y(that)d(send)g(data)g(from)f(the)g(same)h(bu\013ers;)g(e.g.,)f(a)g(new)h
(send)g(with)g(a)f(bu\013er)i(descriptor)g(handle)g(can)f(b)q(e)f(initiated)j
(once)75 1250 y(the)e(previous)i(send)f(with)f(this)h(handle)h(has)e
(completed.)158 1382 y Fo(Example)f(1:)75 1432 y(Consider)i(the)h(follo)o
(wing)c(fragmen)o(t)h(of)i(F)m(ortran)f(co)q(de)75 1513 y Fi(DOUBLE)21
b(PRECISION)e(A\(10,20\))75 1563 y(INTEGER)h(B,)i(C\(5,10\))75
1613 y(INTEGER)e(BH)75 1663 y(...)75 1713 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)
d(MPI_PERSISTENT\))75 1763 y(CALL)j(MPI_APPEND_BLOCK)d(\(BH,)j(B,)h(1,)f
(MPI_INT\))75 1812 y(CALL)g(MPI_APPEND_VEC)e(\(BH,)i(A\(1,3\),)f(11,)h(4,)h
(2,)f(MPI_DOUBLE\))75 1862 y(CALL)g(MPI_APPEND_INDEX\()o(BH,)e(C\(3,7\),)h
(\(4,2,1\),)g(MPI_INT\))158 1943 y Fo(Then)14 b(the)h(bu\013er)g(asso)q
(ciated)f(with)g(the)g(handle)g Fi(BH)g Fo(consists)h(of)e(the)h(sequence)i
(of)e(v)n(ariables)158 1993 y Fi(B,)21 b(A\(1,3\),)g(A\(2,3\),)f(A\(5,3\),)g
(A\(6,3\),)h(A\(9,3\),)f(A\(10,3\),)g(A\(3,4\),)h(A\(4,4\),)f(A\(7,4\),)g
(A\(8,4\),)75 2043 y(A\(1,5\),)g(C\(2,8\),)h(C\(5,7\),)f(C\(4,7\))p
Fo(.)158 2093 y(A)14 b(message)f(created)j(from)c(this)i(bu\013er)h(will)d
(consist)j(of)e(a)g(sequence)j(of)e(one)g(in)o(teger,)f(follo)o(w)o(ed)g(b)o
(y)g(elev)o(en)75 2143 y(double)h(precision)g(reals,)g(follo)o(w)o(ed)e(b)o
(y)i(three)h(in)o(tegers.)158 2192 y(Example)d(2:)75 2242 y(Supp)q(ose)j(one)
f(has)g(an)g(arra)o(y)f(of)g(data)h(declared)h(in)e(C)h(as)75
2323 y Fi(struct)21 b({)140 2373 y(int)h(type;)140 2423 y(double)f
(position[3];)140 2473 y(double)g(velocity[3];)228 2523 y(})g(particle[1000])
158 2604 y Fo(and)d(one)g(w)o(an)o(ts)h(to)f(transfer)h(in)f(one)g(message)g
(the)h(n)o(um)o(b)q(er)e(of)h(particles)g(of)g(t)o(yp)q(e)h(1,)f(the)h(n)o
(um)o(b)q(er)f(of)75 2654 y(particles)g(of)g(t)o(yp)q(e)g(2,)g(and)g(the)g(p)
q(ositions)g(of)f(all)g(the)h(particles)h(of)e(these)i(t)o(w)o(o)f(t)o(yp)q
(es.)31 b(The)18 b(follo)o(wing)d(co)q(de)75 2704 y(fragmen)o(t)d(creates)k
(a)e(suitable)f(bu\013er)i(descriptor.)p eop
%%Page: 11 12
bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1070
b Fo(11)75 45 y Fi(int)21 b(count1)g(=)g(0;)75 95 y(int)g(count2)g(=)g(0;)75
145 y(...)75 195 y(mpi_create_buffer)o(\()e(&handle,)h(MPI_PERSISTENT\);)75
244 y(mpi_append_contig)o(uous\()e(&handle,)i(&count1,)h(MPI_INT\);)75
294 y(mpi_append_contig)o(uous\()d(&handle,)i(&count2,)h(MPI_INT\);)75
344 y(for)g(\(i=0;)g(i<1000;)f(i++\))h({)140 394 y(if)h(\(particle[i].typ)o
(e)d(==)i(1\))h({)206 444 y(count1++;)206 493 y(mpi_append_conti)o(guous)o
(\()d(&handle,)h(&particle[i].posit)o(ion,)e(3,)k(MPI_DOUBLE\);)206
543 y(})140 593 y(elseif)f(\(particle[i].type)d(==)k(2\))f({)206
643 y(count2++;)206 693 y(mpi_append_conti)o(guous)o(\()e(&handle,)h
(&particle[i].posit)o(ion,)e(3,)k(MPI_DOUBLE\);)206 742 y(})140
792 y(})75 842 y(mpi_commit_buffer)o(\(hand)o(le\))75 892 y(...)158
980 y Fo(The)14 b(bu\013er)h(descriptor)f(handle)g(can)g(no)o(w)f(b)q(e)h
(used)g(to)g(send)g(\(rep)q(eatedly\))h(the)f(particle)g(coun)o(ts)g(for)f(t)
o(yp)q(e)75 1030 y(1)g(and)f(2,)h(and)g(the)g(p)q(ositions)g(of)f(the)i
(particles)f(with)g(these)h(t)o(w)o(o)f(t)o(yp)q(es,)g(as)g(long)f(as)h(the)h
(t)o(yp)q(es)g(of)e(the)h(particles)75 1079 y(is)h(not)g(mo)q(di\014ed.)158
1129 y(Supp)q(ose)i(the)f(incoming)d(data)i(is)h(to)f(b)q(e)h(stored)h(at)e
(the)h(receiv)o(er)h(in)o(to)e(the)h(t)o(w)o(o)g(coun)o(t)f(v)n(ariables)g
Fi(count1)75 1179 y Fo(and)h Fi(count2)f Fo(and)h(in)o(to)f(the)i
Fi(position)d Fo(comp)q(onen)o(ts)i(of)f(the)i(initial)d(en)o(tries)j(of)f
(the)g(arra)o(y)g(of)f(particles.)23 b(The)75 1229 y(follo)o(wing)10
b(co)q(de)i(generates)i(a)e(suitable)g(bu\013er)h(descriptor)h(for)d(the)i
(receiv)o(e)g(op)q(eration)f(\(w)o(e)h(assume)f(4)f(b)o(ytes)i(for)75
1279 y(in)o(teger)h(and)g(8)g(b)o(ytes)g(for)g(double\).)75
1367 y Fi(...)75 1416 y(mpi_create_buffer)o(\()19 b(&handle,)h
(MPI_PERSISTENT\);)75 1466 y(mpi_append_contig)o(uous\()e(&handle,)i
(&count1,)h(MPI_INT\);)75 1516 y(mpi_append_contig)o(uous\()d(&handle,)i
(&count2,)h(MPI_INT\);)75 1566 y(mpi_append_hvec\()d(&handle,)j
(&particle[0].pos)o(itio)o(n,)e(1000,)i(52,)g(3,)g(MPI_DOUBLE\);)75
1616 y(mpi_commit_buffer)o(\()e(&handle\))75 1665 y(...)158
1753 y Fo(A)c(bu\013er)g(handle)f(can)h(b)q(e)g(used)g(for)f(comm)o
(unication,)d(ev)o(en)k(if)f(it)g(is)g(not)g(asso)q(ciated)i(with)e(an)o(y)g
(v)n(ariables)75 1803 y(\(i.e.,)e(ev)o(en)i(if)e(it)h(w)o(as)g(not)g(set)h(b)
o(y)f(an)o(y)g Fi(MPI)p 761 1803 14 2 v 15 w(APPEND)p 908 1803
V 15 w(xxx)f Fo(call\).)17 b(Suc)o(h)d(a)f(handle)g(is)g(asso)q(ciated)h
(with)f(an)g(empt)o(y)75 1853 y(bu\013er,)i(and)f(a)g(message)g(created)i
(from)c(it)i(con)o(tains)g(no)g(data.)19 b(The)c(handle)f(still)f(need)i(b)q
(e)g(committed)d(b)q(efore)75 1903 y(it)i(is)f(used.)158 1953
y(The)h(same)e(en)o(try)i(ma)o(y)d(app)q(ear)i(more)f(than)i(once)f(in)g(a)g
(bu\013er)h(comp)q(onen)o(t,)e(and)h(distinct)g(bu\013er)h(comp)q(o-)75
2002 y(nen)o(ts)i(ma)o(y)d(o)o(v)o(erlap.)21 b(If)15 b(a)g(bu\013er)h
(descriptor)g(with)f(replicated)h(en)o(tries)g(is)f(used)h(to)f(receiv)o(e)h
(a)f(message,)f(then)75 2052 y(the)g(\014nal)g(v)n(alue)f(of)g(an)h(en)o(try)
g(that)g(o)q(ccurs)i(more)d(than)g(once)i(in)e(the)i(bu\013er)g(descriptor)g
(is)f(unde\014ned.)158 2180 y Fm(Discussion:)158 2226 y Fl(It)f(is)g(not)h
(clear)g(to)f(me)g(that)g(o)o(v)o(erlaps)i(ha)o(v)o(e)e(an)o(y)h(useful)g
(purp)q(ose,)g(and)g(they)g(ma)o(y)f(complicate)i(implemen)o(tation.)75
2272 y(But)e(w)o(e)g(had)g(an)h(o)o(v)o(erwhelming)h(v)o(ote)e(in)h(fa)o(v)o
(or)f(of)g(this)h(feature.)75 2470 y Fk(1.5.3)55 b(T)n(yp)r(e)18
b(Matc)n(hing)i(and)f(Data)g(Con)n(v)n(ersion)75 2547 y Fh(T)o(yp)q(e)d(matc)
o(hing)75 2623 y Fo(One)f(can)f(think)f(of)h(message)f(transmission)g(as)h
(consisting)g(of)f(three)i(phases:)126 2704 y(1.)20 b(Data)13
b(is)h(pulled)f(out)h(of)f(the)i(send)g(bu\013er)f(and)g(a)g(message)f(is)h
(assem)o(bled)p eop
%%Page: 12 13
bop 75 -100 a Fo(12)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)126 45 y Fo(2.)20 b(A)14 b(message)f(is)h
(transferred)i(from)c(sender)j(to)f(receiv)o(er)126 127 y(3.)20
b(Data)13 b(is)h(pulled)f(from)f(the)j(incoming)c(message)j(and)g(disassem)o
(bled)f(in)o(to)g(the)i(receiv)o(e)g(bu\013er)158 209 y(T)o(yp)q(e)c(matc)o
(hing)e(has)i(to)f(b)q(e)h(observ)o(ed)h(at)f(eac)o(h)g(of)f(these)i(three)g
(phases:)17 b(The)11 b(t)o(yp)q(e)g(of)f(eac)o(h)i(v)n(ariable)d(in)h(the)75
259 y(sender)15 b(bu\013er)g(has)f(to)g(matc)o(h)f(the)h(t)o(yp)q(e)g(of)g
(the)g(corresp)q(onding)h(en)o(try)f(in)g(the)g(bu\013er)h(descriptor;)g(the)
f(t)o(yp)q(e)h(of)75 309 y(an)e(en)o(try)g(in)f(the)h(bu\013er)h(descriptor)g
(used)g(to)e(send)i(a)f(message)f(has)h(to)f(matc)o(h)g(the)h(t)o(yp)q(e)h
(of)e(the)h(corresp)q(onding)75 358 y(en)o(try)f(in)e(the)i(bu\013er)g
(descriptor)g(used)g(to)f(receiv)o(e)h(the)g(message;)f(and)g(the)h(t)o(yp)q
(e)f(of)g(eac)o(h)g(v)n(ariable)f(in)h(the)h(receiv)o(e)75
408 y(bu\013er)i(has)f(to)g(matc)o(h)e(the)j(corresp)q(onding)f(en)o(try)h
(in)e(the)i(bu\013er)g(descriptor)g(for)e(this)h(bu\013er.)19
b(A)13 b(program)e(that)75 458 y(fails)i(to)g(observ)o(e)i(these)h(three)f
(rules)f(is)g(erroneous.)158 508 y(T)m(o)g(de\014ne)h(t)o(yp)q(e)g(matc)o
(hing)d(more)h(precisely)m(,)i(w)o(e)f(need)h(to)g(deal)f(with)g(t)o(w)o(o)f
(issues:)20 b(matc)o(hing)13 b(of)g(t)o(yp)q(es)j(of)75 558
y(the)h(host)g(language)f(with)g(bu\013er)h(descriptor)h(t)o(yp)q(es;)g(and)f
(matc)o(hing)d(of)i(bu\013er)i(descriptor)g(t)o(yp)q(es)f(at)f(sender)75
607 y(and)c(receiv)o(er.)18 b(W)m(e)12 b(restrict)h(ourselv)o(es)g(to)f(the)g
(case)h(where)g(the)f(sending)g(and)g(receiving)g(programs)e(are)i(written)75
657 y(in)h(the)i(same)e(language.)158 707 y(The)21 b(t)o(yp)q(e)h(of)e(an)g
(en)o(try)i(in)e(the)h(sender)i(bu\013er)f(descriptor)g(matc)o(hes)e(the)i(t)
o(yp)q(e)f(of)f(an)h(en)o(try)g(in)f(the)75 757 y(receiv)o(er)13
b(bu\013er)g(descriptor)g(if)e(they)h(ha)o(v)o(e)g(iden)o(tical)f(names:)16
b Fi(MPI)p 1131 757 14 2 v 15 w(INT)11 b Fo(matc)o(hes)h Fi(MPI)p
1448 757 V 15 w(INT)p Fo(,)e Fi(MPI)p 1617 757 V 15 w(REAL)h
Fo(matc)o(hes)75 807 y Fi(MPI)p 144 807 V 15 w(REAL)p Fo(,)i(and)g(so)h(on.)
158 857 y(The)i(t)o(yp)q(e)g(of)e(a)h(v)n(ariable)g(in)g(a)g(host)g(program)f
(matc)o(hes)h(the)h(t)o(yp)q(e)g(of)e(the)i(en)o(try)g(in)f(a)g(bu\013er)i
(descriptor)75 906 y(if)e(the)i(name)e(of)g(the)i(bu\013er)g(descriptor)g(en)
o(try)g(t)o(yp)q(e)g(corresp)q(onds)h(to)e(the)g(basic)g(t)o(yp)q(e)h(of)f
(the)g(host)h(program)75 956 y(v)n(ariable:)k(an)15 b(en)o(try)i(with)e(t)o
(yp)q(e)h(name)f Fi(MPI)p 794 956 V 15 w(INT)g Fo(matc)o(hes)h(a)f(F)m
(ortran)h(v)n(ariable)e(of)i(t)o(yp)q(e)g Fi(INTEGER)p Fo(,)e(an)h(en)o(try)
75 1006 y(with)10 b(t)o(yp)q(e)i(name)d Fi(MPI)p 433 1006 V
15 w(REAL)h Fo(matc)o(hes)h(a)f(F)m(ortran)h(v)n(ariable)e(of)i(t)o(yp)q(e)g
Fi(REAL)p Fo(,)e(and)i(so)g(on.)17 b(There)12 b(is)e(one)h(exception)75
1056 y(to)h(this)h(last)f(rule:)18 b(An)13 b(en)o(try)g(with)f(t)o(yp)q(e)h
(name)e Fi(MPI)p 917 1056 V 15 w(BYTE)h Fo(can)h(b)q(e)g(used)g(to)g(matc)o
(h)e(an)o(y)h(b)o(yte)h(of)f(storage)h(\(on)f(a)75 1106 y(b)o
(yte-addressable)j(mac)o(hine\),)d(irresp)q(ectiv)o(e)j(of)e(the)g(datat)o
(yp)q(e)h(of)f(the)h(v)n(ariable)e(that)i(con)o(tains)f(this)g(b)o(yte.)19
b(The)75 1155 y(v)n(alue)13 b(of)h(the)g(message)g(en)o(try)g(will)f(b)q(e)h
(the)g(binary)g(v)n(alue)f(of)g(the)i(corresp)q(onding)g(b)o(yte)f(in)f
(memory)m(.)158 1205 y(W)m(e)h(th)o(us)g(ha)o(v)o(e)g(t)o(w)o(o)f(cases:)137
1287 y Fe(\017)21 b Fo(Comm)o(uni)o(cation)6 b(of)j(t)o(yp)q(ed)h(v)n(alues)f
(\(e.g.,)g(with)g(datat)o(yp)q(e)g(di\013eren)o(t)h(from)e
Fi(MPI)p 1430 1287 V 15 w(BYTE)p Fo(,)g(where)i(the)g(datat)o(yp)q(es)179
1337 y(of)k(the)h(corresp)q(onding)g(en)o(tries)g(in)f(the)h(sender)h
(program,)d(in)h(the)h(sender)h(bu\013er)f(descriptor,)h(in)d(the)i(re-)179
1386 y(ceiv)o(er)g(bu\013er)f(descriptor)i(and)d(in)h(the)g(receiv)o(er)i
(program)c(should)i(all)e(matc)o(h.)137 1469 y Fe(\017)21 b
Fo(Comm)o(uni)o(cation)13 b(of)j(un)o(t)o(yp)q(ed)h(v)n(alues)f(\(e.g.,)g(of)
g(datat)o(yp)q(e)g Fi(MPI)p 1224 1469 V 15 w(BYTE)p Fo(,)f(where)j(the)f(the)
g(corresp)q(onding)179 1518 y(en)o(tries)e(in)e(the)i(sender)g(and)f(the)g
(receiv)o(er)h(bu\013er)g(descriptors)h(ha)o(v)o(e)d(b)q(oth)h(datat)o(yp)q
(e)g Fi(MPI)p 1631 1518 V 16 w(BYTE)p Fo(.)e(In)i(this)179
1568 y(case,)k(there)g(are)f(no)g(requiremen)o(ts)g(on)g(the)g(datat)o(yp)q
(es)h(of)e(the)i(corresp)q(onding)f(en)o(tries)h(in)f(the)g(sender)179
1618 y(and)c(the)i(receiv)o(er)g(programs,)e(nor)h(is)f(it)h(required)h(that)
f(they)g(b)q(e)g(the)h(same.)75 1734 y Fk(1.5.4)55 b(Data)19
b(con)n(v)n(ersion)75 1811 y Fo(W)m(e)13 b(use)i(the)g(follo)o(wing)c
(terminology:)75 1892 y Fh(t)o(yp)q(e)k(con)o(v)o(ersion)j
Fo(c)o(hanges)d(the)f(datat)o(yp)q(e)g(of)f(a)h(v)n(alue,)f(e.g.)18
b(b)o(y)13 b(rounding)h(a)f Fi(REAL)g Fo(to)h(an)g Fi(INTEGER)p
Fo(.)75 1974 y Fh(represen)o(tati)o(on)e(con)o(v)o(ersion)18
b Fo(c)o(hanges)e(the)g(binary)f(represen)o(tation)i(of)e(a)g(v)n(alue,)f
(e.g.)23 b(from)13 b(Hex)j(\015oating)179 2024 y(p)q(oin)o(t)d(to)h(IEEE)g
(\015oating)f(p)q(oin)o(t.)158 2106 y(Represen)o(tation)20
b(con)o(v)o(ersion)f(is)g(needed)h(when)g(data)e(is)h(mo)o(v)o(ed)e(across)j
(mac)o(hines)e(that)h(use)h(di\013eren)o(t)75 2156 y(binary)9
b(enco)q(dings)h(for)f(the)h(same)f(datat)o(yp)q(e.)16 b(Represen)o(tation)11
b(con)o(v)o(ersion)e(preserv)o(es,)k(to)c(the)h(greatest)h(p)q(ossible)75
2205 y(extend,)h(the)e Fh(v)m(alue)g Fo(represen)o(ted.)20
b(Ho)o(w)o(ev)o(er,)11 b(rounding)e(errors)j(and)e(o)o(v)o(er\015o)o(w)g(and)
g(under\015o)o(w)h(exceptions)g(ma)o(y)75 2255 y(o)q(ccur)17
b(during)e(\015oating)f(p)q(oin)o(t)h(con)o(v)o(ersions;)h(con)o(v)o(ersion)g
(of)f(in)o(tegers)h(ma)o(y)e(also)g(lead)h(to)h(exceptions)g(when)g(a)75
2305 y(v)n(alue)d(that)h(can)g(represen)o(ted)j(in)c(one)h(system)g(cannot)g
(b)q(e)h(represen)o(ted)h(in)e(the)g(other)h(system.)i(MPI)d(do)q(es)h(not)75
2355 y(sp)q(ecify)f(rules)h(for)e(represen)o(tation)j(con)o(v)o(ersion.)158
2405 y(The)j(t)o(yp)q(e)f(matc)o(hing)e(rules)j(imply)d(that)i(MPI)g(comm)o
(unication)d(nev)o(er)k(en)o(tails)e(t)o(yp)q(e)i(con)o(v)o(ersion.)31
b(On)75 2455 y(the)15 b(other)g(hand,)f(MPI)g(requires)i(that)e(a)g(represen)
o(tation)i(con)o(v)o(ersion)e(b)q(e)h(p)q(erformed)f(when)h(a)f(t)o(yp)q(ed)g
(v)n(alue)g(is)75 2504 y(transferred)g(across)g(en)o(vironmen)o(ts)d(that)i
(use)g(di\013eren)o(t)g(represen)o(tations)i(for)d(the)h(datat)o(yp)q(e)g(of)
e(this)i(v)n(alue.)k(An)75 2554 y(exception)f(o)q(ccurring)g(during)f
(represen)o(tation)i(con)o(v)o(ersion)f(results)g(in)f(a)h(failure)e(of)h
(the)h(comm)o(unicatio)o(n;)d(an)75 2604 y(error)i(o)q(ccurs)g(either)g(in)e
(the)i(send)g(op)q(eration,)e(or)h(the)g(receiv)o(e)h(op)q(eration,)f(or)g(b)
q(oth.)158 2654 y(If)g(a)f(v)n(alue)h(sen)o(t)h(in)e(a)h(message)g(is)f(un)o
(t)o(yp)q(ed)i(\(i.e.,)e(of)g(t)o(yp)q(e)i Fi(MPI)p 1180 2654
V 15 w(BYTE)p Fo(\),)e(then)h(the)h(binary)e(represen)o(tation)75
2704 y(of)h(the)g(b)o(yte)h(stored)g(at)f(the)h(receiv)o(er)h(is)e(iden)o
(tical)f(to)h(the)h(binary)f(represen)o(tation)i(of)d(the)i(b)o(yte)g(loaded)
e(at)h(the)p eop
%%Page: 13 14
bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1070
b Fo(13)75 45 y(sender.)24 b(This)16 b(holds)f(true,)h(whether)h(sender)g
(and)e(receiv)o(er)i(run)f(in)f(the)h(same)f(or)g(in)g(distinct)h(en)o
(vironmen)o(ts;)75 95 y(no)e(represen)o(tation)h(con)o(v)o(ersion)f(is)g
(required.)158 145 y(Note)f(that)g(no)g(con)o(v)o(ersions)g(ev)o(er)h(o)q
(ccur)g(when)g(an)e(MPI)h(program)f(executes)j(in)d(a)h(homogeneous)f
(system,)75 195 y(where)g(all)e(pro)q(cesses)j(run)e(in)g(the)g(same)f(en)o
(vironmen)o(t.)16 b(Also)11 b(note)g(the)g(di\013eren)o(t)h(b)q(eha)o(vior)f
(of)f Fi(MPI)p 1654 195 14 2 v 15 w(BYTE)g Fo(and)h(of)75 244
y Fi(MPI)p 144 244 V 15 w(CHAR)p Fo(.)g(A)h(bu\013er)h(descriptor)h(en)o(try)
f(with)e(datat)o(yp)q(e)i(of)e Fi(MPI)p 1106 244 V 16 w(CHAR)g
Fo(can)h(only)g(matc)o(h)f(a)g(F)m(ortran)h(v)n(ariable)g(of)75
294 y(t)o(yp)q(e)i Fi(CHAR)p Fo(;)e(and)h(represen)o(tation)i(con)o(v)o
(ersion)f(ma)o(y)d(o)q(ccur)j(when)g(v)n(alues)f(of)g(t)o(yp)q(e)h
Fi(MPI)p 1473 294 V 15 w(CHAR)e Fo(are)i(transferred.,)75 344
y(e.g.,)f(from)f(an)h(EBCDIC)i(enco)q(ding)f(to)f(an)h(ASCI)q(I)g(enco)q
(ding.)158 394 y(Consider)g(the)h(follo)o(wing)c(examples.)75
481 y Fi(!)43 b(FIRST)21 b(EXAMPLE)75 531 y(INTEGER)f(A\(100\))75
581 y(...)75 630 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75
680 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h(MPI_CHAR\))158
767 y Fo(This)15 b(program)f(is)i(erroneous,)g(since)h(the)f(t)o(yp)q(e)g(of)
e(the)j(bu\013er)f(descriptor)h(do)q(es)f(not)f(matc)o(h)g(the)h(t)o(yp)q(e)g
(of)75 817 y(the)g(v)n(ariable)f(in)g(the)i(bu\013er.)25 b(The)16
b(b)q(eha)o(vior)f(or)h(this)g(program)e(is)i(unde\014ned.)25
b(A)16 b(desirable)g(b)q(eha)o(vior)g(is)f(for)75 867 y(that)f(error)h(to)e
(b)q(e)i(detected)h(\(at)e(compile)e(time)h(or)h(run)g(time\))e(and)i(result)
h(in)e(an)h(error)h(condition.)75 954 y Fi(!)43 b(SECOND)21
b(EXAMPLE)75 1004 y(INTEGER)f(A\(100\))75 1054 y(...)75 1104
y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 1153
y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h(MPI_BYTE\))158
1241 y Fo(This)11 b(program)f(is)h(correct.)19 b(The)12 b(b)o(yte)g(that)f
(will)f(b)q(e)i(transmitted)f(in)g(a)g(message)g(created)i(with)e(this)h
(bu\013er)75 1290 y(descriptor)h(is)f(the)g(\014rst)h(b)o(yte)f(in)g(the)g
(storage)h(o)q(ccupied)f(b)o(y)g Fi(A\(1\))p Fo(.)17 b(The)12
b(v)n(alue)f(of)h(this)f(b)o(yte)i(is)e(implemen)o(tation)75
1340 y(dep)q(enden)o(t,)17 b(and)e(will)f(v)n(ary)h(according)g(to)g(the)h(n)
o(um)o(b)q(er)f(of)f(b)o(ytes)i(used)h(to)e(store)h(F)m(ortran)f(v)n
(ariables)g(of)f(t)o(yp)q(e)75 1390 y Fi(INTEGER)e Fo(and)i(the)h(b)o(yte)f
(ordering)g(on)f(the)i(host)f(mac)o(hine.)75 1477 y Fi(!)43
b(THIRD)21 b(EXAMPLE)75 1527 y(INTEGER)f(A\(100\))75 1577 y(...)75
1627 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH1,)d(MPI_PERSISTENT\))75
1677 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH1,)j(A\(1\),)g(1,)g
(MPI_INTEGER\))75 1726 y(...)75 1776 y(CALL)g(MPI_CREATE_BUFFER)o(\(BH2,)d
(MPI_PERSISTENT\))75 1826 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH2,)j
(A\(1\),)g(4,)g(MPI_BYTE\))158 1913 y Fo(Consider)14 b(the)h(follo)o(wing)c
(three)k(cases:)126 1993 y(1.)20 b(A)10 b(message)g(is)g(sen)o(t)h(using)e
(bu\013er)j(descriptor)f(handle)f Fi(BH1)f Fo(and)h(receiv)o(ed)i(using)e
(bu\013er)h(descriptor)g(handle)179 2043 y Fi(BH2)p Fo(.)19
b(The)d(program)d(is)h(erroneous,)i(since)f(the)h(sender)g(and)e(receiv)o(er)
i(bu\013er)g(descriptors)g(don't)f(matc)o(h.)179 2093 y(The)g(b)q(eha)o(vior)
f(of)g(the)h(program)d(is)j(unde\014ned,)g(and)f(a)g(p)q(ossible)h
(\(desirable)g(but)g(unlik)o(ely\))e(b)q(eha)o(vior)h(is)179
2142 y(for)f(an)h(error)h(condition)e(to)h(o)q(ccur.)126 2224
y(2.)20 b(Both)d(sender)h(and)f(receiv)o(er)i(use)e(bu\013er)h(descriptor)g
(handle)f Fi(BH1)p Fo(.)26 b(The)18 b(program)d(is)i(correct,)h(and)f(an)179
2274 y(in)o(teger)12 b(v)n(alue)f(is)g(transmitted)g(from)f(sender)j(to)f
(receiv)o(er.)19 b(The)12 b(v)n(alue)f(assigned)h(to)f Fi(A\(1\))g
Fo(at)g(the)h(receiv)o(er)179 2323 y(is)h(iden)o(tical)f(to)h(the)g(v)n(alue)
g(of)f Fi(A\(1\))g Fo(at)h(the)h(sender,)g(but)f(the)h(binary)f(represen)o
(tation)h(of)e(this)i(v)n(alue)e(ma)o(y)179 2373 y(b)q(e)i(di\013eren)o(t,)h
(if)e(sender)i(and)f(receiv)o(er)h(run)g(in)e(di\013eren)o(t)i(en)o(vironmen)
o(ts.)126 2455 y(3.)20 b(Both)c(sender)h(and)e(receiv)o(er)i(use)g(bu\013er)f
(descriptor)h(handle)f Fi(BH2)p Fo(.)22 b(The)16 b(program)e(is)h(correct.)25
b(Assume)179 2504 y(that)14 b(b)q(oth)h(sender)h(and)e(receiv)o(er)i(use)f
(four)f(b)o(ytes)h(to)f(store)i(in)o(tegers.)k(Then)15 b(the)g(v)n(alue)f
(stored)h(in)f Fi(A\(1\))179 2554 y Fo(at)i(the)h(receiv)o(er)h(has)e(the)h
(same)f(binary)g(represen)o(tation)i(as)e(the)h(v)n(alue)f(of)f
Fi(A\(1)h Fo(at)g(the)h(sender.)27 b(If)16 b(the)179 2604 y(same)c(b)o(yte)h
(ordering)g(and)g(the)h(same)e(enco)q(ding)h(of)g(in)o(teger)g(v)n(alues)g
(is)g(used)g(at)g(b)q(oth)g(ends)h(then)g Fi(A\(1\))e Fo(at)179
2654 y(the)j(receiv)o(er)h(is)f(set)g(to)g(the)g(v)n(alue)f(of)g
Fi(A\(1\))g Fo(at)h(the)g(sender;)h(otherwise)g(the)f(same)f(32)g(bits)h(ma)o
(y)e(enco)q(de)179 2704 y(di\013eren)o(t)i(v)n(alues.)p eop
%%Page: 14 15
bop 75 -100 a Fo(14)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)158 45 y Fm(Implemen)o(tation)f(note:)158
94 y Fl(The)d(curren)o(t)g(de\014nition)j(do)q(es)d(not)h(require)g(messages)
g(to)e(carry)i(t)o(yp)q(e)f(information.)18 b(A)9 b(message)i(can)f(b)q(e)h
(comp)q(osed)75 140 y(and)g(sen)o(t)f(using)i(only)f(the)f(information)i(in)f
(the)g(sender)g(bu\013er)g(descriptor,)h(and)e(can)h(b)q(e)f(receiv)o(ed)i
(and)e(stored)h(using)h(only)75 186 y(the)i(information)i(in)e(the)g(receiv)o
(er)h(bu\013er)g(descriptor.)21 b(If)13 b(messages)h(are)g(sen)o(t)g(b)q(et)o
(w)o(een)g(di\013eren)o(t)i(mac)o(hines)f(then)f(one)75 231
y(can)e(either)h(use)g(a)f(\\univ)o(ersal")j(data)d(enco)q(ding)i(for)e
(messages,)h(use)f(kno)o(wledge)i(of)d(the)i(receiv)o(er)g(en)o(vironmen)o(t)
h(in)e(order)75 277 y(to)i(con)o(v)o(ert)h(data)f(at)g(the)h(sender,)f(or)h
(use)f(kno)o(wledge)i(of)e(the)g(sender)h(en)o(vironmen)o(t)h(in)f(order)f
(to)g(con)o(v)o(ert)h(data)g(at)f(the)75 323 y(receiv)o(er.)k(In)c(either)g
(case)f(the)h(lo)q(cal)g(bu\013er)g(descriptor)h(can)f(b)q(e)f(used)h(to)f
(deriv)o(e)i(the)e(t)o(yp)q(es)h(of)f(the)g(v)n(alues)h(transferred.)75
368 y(Ho)o(w)o(ev)o(er,)f(additional)k(t)o(yp)q(e)c(information)j(carried)e
(b)o(y)g(messages)g(can)g(pro)o(vide)h(b)q(etter)f(error)f(detection.)19
b(It)13 b(is)h(p)q(ossible)75 414 y(to)d(add)g(alw)o(a)o(ys)h(suc)o(h)f
(additional)j(t)o(yp)q(e)d(information,)i(for)e(b)q(etter)g(error)g(c)o(hec)o
(king,)h(or)f(add)h(it)f(when)g(running)i(in)e(\\debug")75
460 y(mo)q(de,)i(where)g(some)g(p)q(erformance)h(is)g(sacri\014ced)h(for)d
(added)i(error)g(c)o(hec)o(king,)g(etc.)158 675 y Fm(Discussion:)158
724 y Fl(There)h(is)g(no)g(agreemen)o(t)h(y)o(et)f(on)g(the)g(need)g(for)g(a)
g(de\014nition)i(and,)f(if)f(needed,)h(on)f(the)g(righ)o(t)g(de\014nition)j
(of)c(t)o(yp)q(e)75 770 y(matc)o(hing)k(and)e(con)o(v)o(ersion)j(b)q(et)o(w)o
(een)d(programs)h(written)g(in)g(di\013eren)o(t)g(languages.)29
b(The)16 b(natural)h(in)o(terpretation)i(of)75 815 y(\\send)11
b(from)e(C)g(and)i(receiv)o(e)f(in)h(F)m(ortran")f(is)g(either)h(\\send)g
(from)e(C,)g(receiv)o(e)i(in)f(C)f(and)i(con)o(v)o(ert)f(to)g(F)m(ortran",)g
(or)g(\\con)o(v)o(ert)75 861 y(to)i(F)m(ortran,)h(send)g(from)f(F)m(ortran)g
(and)h(receiv)o(e)g(in)g(F)m(ortran".)18 b(Ho)o(w)o(ev)o(er,)11
b(the)i(corresp)q(ondence)h(b)q(et)o(w)o(een)f(F)m(ortran)f(t)o(yp)q(es)75
907 y(and)g(C)f(t)o(yp)q(es)g(is)h(implemen)o(tation)j(dep)q(enden)o(t.)j(Th)
o(us,)11 b(the)h(diagram)g(sho)o(wn)g(b)q(elo)o(w)g(is)g(not)f(comm)o(utativ)
o(e,)i(and)f(the)f(t)o(w)o(o)75 952 y(in)o(terpretations)16
b(of)c(C)h(to)g(F)m(ortran)g(comm)o(unication)j(ma)o(y)d(lead)h(to)f
(di\013eren)o(t)i(results.)238 999 y(mac)o(hine)g(1)285 b(mac)o(hine)15
b(2)513 1045 y(send)232 1091 y(C)e(program)118 b Fc(!)g Fl(C)13
b(program)313 1182 y Fc(#)436 b(#)513 1228 y Fl(send)183 1273
y(F)m(ortran)13 b(program)69 b Fc(!)g Fl(F)m(ortran)13 b(program)158
1321 y(A)h(go)q(o)q(d)h(solution)h(w)o(ould)f(mak)o(e)f(this)h(diagram)h
(comm)o(utativ)o(e.)21 b(Otherwise,)15 b(one)g(needs)g(to)f(pic)o(k)h(one)f
(of)g(the)g(t)o(w)o(o)75 1366 y(paths)k(as)f(the)g(canonical)i(de\014nition)h
(of)d(in)o(terlanguage)j(comm)o(unication.)31 b(If)16 b(the)h(upp)q(er)h
(path)g(is)f(c)o(hosen)h(\(message)75 1412 y(carries)c(sender)g(datat)o(yp)q
(es\),)h(then)e(the)h(receiv)o(er)g(of)f(a)g(message)h(needs)g(to)g(kno)o(w)f
(the)h(language)h(the)f(message)f(w)o(as)h(sen)o(t)75 1458
y(from;)h(if)g(the)f(lo)o(w)o(er)h(path)h(is)f(c)o(hosen,)g(\(message)h
(carries)f(receiv)o(er)h(datat)o(yp)q(es\),)g(then)f(the)f(sender)i(needs)f
(to)g(kno)o(w)g(the)75 1503 y(language)g(the)e(message)h(is)f(receiv)o(ed)i
(in)o(to.)158 1553 y(A)c(p)q(ossible)i(solution)h(\(prop)q(osed)e(b)o(y)g
(Dann)o(y)g(Nessett\))f(is)h(to)f(lea)o(v)o(e)h(to)f(the)g(user)h(the)f(c)o
(hoice)h(of)f(sp)q(ecifying)i(whether)75 1599 y(to)d(send)g(\\F)m(ortran)g
(datat)o(yp)q(es")i(or)d(\\C)h(datat)o(yp)q(es",)h(in)g(the)f(ab)q(o)o(v)o(e)
g(situation.)18 b(One)10 b(w)o(ould)h(still)g(require)g(exact)f(matc)o(hing)
75 1644 y(of)16 b(send)h(bu\013er)g(descriptors)h(and)f(receiv)o(e)g
(bu\013er)g(descriptors.)28 b(Ho)o(w)o(ev)o(er,)17 b(a)f(F)m(ortran)h
(program)g(could)g(use)g(a)f(bu\013er)75 1690 y(descriptor)f(with)e(\\C)g
(datat)o(yp)q(es",)h(and)g(vice)g(v)o(ersa.)j(The)c(standard)h(w)o(ould)g(sp)
q(ecify)h(whic)o(h)e(matc)o(hing)i(of)e(F)m(ortran)g(and)75
1735 y(C)i(datat)o(yp)q(es)j(are)e(legal.)27 b(F)m(or)16 b(example,)i(it)e(w)
o(ould)h(b)q(e)f(correct)g(to)g(send)h(a)f(F)m(ortran)g(v)n(ariable)i(of)e(t)
o(yp)q(e)g Ff(REAL)e Fl(using)k(a)75 1781 y(bu\013er)c(descriptor)g(of)f(t)o
(yp)q(e)g Ff(MPI)p 556 1781 12 2 v 13 w(FLOAT)p Fl(.)d(Data)j(con)o(v)o
(ersion)i(migh)o(t)f(b)q(e)f(required)h(if)f(C)f(\015oat)i(n)o(um)o(b)q(ers)f
(and)h(F)m(ortran)f(real)75 1827 y(n)o(um)o(b)q(ers)h(do)f(not)h(use)f(the)g
(same)g(presen)o(tation.)158 1876 y(Another)j(\(non\)solution)i(is)f(to)e
(ignore)i(the)e(issue)i(and)f(let)g(the)g(b)q(eha)o(vior)h(of)e(C)g(to)h(F)m
(ortran)f(comm)o(unication)k(b)q(e)75 1922 y(unde\014ned.)32
b(The)18 b(user)g(can)g(alw)o(a)o(ys)g(matc)o(h)g(language)h(at)f(sending)h
(and)f(receiving)i(end)e(if)f(he)h(or)g(she)g(is)g(willing)i(to)75
1968 y(struggle)14 b(with)g(the)f(issue)h(of)f(calling)i(MPI)e(C)g(functions)
i(from)d(F)m(ortran)i(or)f(vice-v)o(ersa.)75 2207 y Fn(1.6)70
b(Receiv)n(e)20 b(Criteria)75 2305 y Fo(The)e(selection)g(of)f(a)g(message)g
(b)o(y)g(a)g(receiv)o(e)i(op)q(eration)e(is)h(done)f(uniquely)g(according)h
(to)f(the)h(v)n(alue)f(of)f(the)75 2355 y(message)c(en)o(v)o(elop)q(e.)18
b(The)13 b(receiv)o(e)h(op)q(eration)e(sp)q(eci\014es)i(an)e
Fh(en)o(v)o(elop)q(e)g(pattern)p Fo(;)f(a)h(message)g(can)h(b)q(e)g(receiv)o
(ed)75 2405 y(b)o(y)20 b(that)g(receiv)o(e)h(op)q(eration)f(only)f(if)g(its)h
(en)o(v)o(elop)q(e)h(matc)o(hes)e(that)h(pattern.)38 b(A)20
b(pattern)g(sp)q(eci\014es)i(v)n(alues)75 2455 y(for)c(the)h
Fi(source)p Fo(,)f Fi(tag)g Fo(and)h Fi(context)e Fo(\014elds)i(of)f(the)h
(message)f(en)o(v)o(elop)q(e.)32 b(In)19 b(addition,)f(the)h(v)n(alue)f(for)g
(the)75 2504 y Fi(dest)g Fo(\014eld)g(is)g(set,)i(implicitly)m(,)c(to)i(b)q
(e)h(equal)f(to)h(the)g(receiving)f(pro)q(cess)j(id.)31 b(The)19
b(receiv)o(er)h(ma)o(y)c(sp)q(ecify)j(a)75 2554 y Fi(DONTCARE)11
b Fo(v)n(alue)i(for)f Fi(source)p Fo(,)g(or)h Fi(tag)p Fo(,)f(indicating)g
(that)h(an)o(y)g(source)h(and/or)f(tag)f(are)i(acceptable.)19
b(It)13 b(cannot)75 2604 y(sp)q(ecify)k(a)f(DONTCARE)h(v)n(alue)e(for)h
Fi(context)f Fo(or)i Fi(dest)p Fo(.)25 b(Th)o(us,)17 b(a)f(message)g(can)g(b)
q(e)i(receiv)o(ed)f(b)o(y)g(a)f(receiv)o(e)75 2654 y(op)q(eration)e(only)f
(if)g(it)g(is)h(addressed)h(to)f(the)g(receiving)g(task,)g(has)f(a)h(matc)o
(hing)e(con)o(text,)i(has)g(matc)o(hing)e(source)75 2704 y(unless)18
b(source=DONTCARE)i(in)d(the)h(pattern,)h(and)e(has)h(a)f(matc)o(hing)f(tag)h
(unless)i(tag=DONTCARE)e(in)p eop
%%Page: 15 16
bop 75 -100 a Fg(1.7.)31 b(COMMUNICA)m(TION)14 b(MODE)1136
b Fo(15)75 45 y(the)18 b(pattern.)28 b(DONTCARE)16 b(v)n(alues)h(for)g
(source)h(or)f(tag)f(are)i(sp)q(eci\014ed)g(via)e(a)h(named)f(constan)o(t)h
(for)g(source)75 95 y(don)o(tcare)e(and)e(a)h(named)f(constan)o(t)h(for)g
(tag)f(don)o(tcare.)158 145 y(The)i(length)f(of)g(the)g(receiv)o(ed)i
(message)e(m)o(ust)f(b)q(e)i(less)g(then)g(or)f(equal)g(the)h(length)f(of)g
(the)h(receiv)o(e)g(bu\013er.)75 195 y(I.e.,)d(all)f(incoming)f(data)i(m)o
(ust)f(\014t,)i(without)f(truncation,)g(in)o(to)g(the)h(receiv)o(e)g
(bu\013er.)19 b(It)12 b(is)h(erroneous)g(to)g(receiv)o(e)75
244 y(a)h(message)f(whic)o(h)h(length)g(exceed)i(the)e(receiv)o(e)h
(bu\013er,)g(and)e(the)i(outcome)e(of)g(program)g(where)i(this)f(o)q(ccurs)h
(is)75 294 y(undetermined.)75 432 y Fn(1.7)70 b(Comm)n(unication)20
b(Mo)r(de)75 523 y Fo(A)14 b(send)h(op)q(eration)f(can)g(o)q(ccur)h(in)e(one)
h(of)f(three)j(mo)q(des:)75 606 y Fh(ST)l(AND)o(ARD)k Fo(The)14
b(send)h(ma)o(y)d(start)i(whether)i(or)d(not)h(a)g(matc)o(hing)e(receiv)o(e)j
(has)f(b)q(een)h(p)q(osted.)75 690 y Fh(READ)o(Y)20 b Fo(The)15
b(send)g(ma)o(y)c(start)k(only)e(if)g(a)h(matc)o(hing)e(receiv)o(e)j(has)f(b)
q(een)h(p)q(osted.)75 773 y Fh(SECURE)21 b Fo(The)12 b(send)g(op)q(eration)f
(will)f(not)h(complete)g(successfully)h(un)o(til)e(it)h(is)g(guaran)o(teed)h
(that)f(the)h(message)179 823 y(sen)o(t)i(will)f(b)q(e)h(receiv)o(ed.)158
906 y(The)k(start)g(and)f(completion)e(of)i(a)g Fh(standard)e
Fo(send)j(do)q(es)g(not)f(dep)q(end)h(on)f(the)h(status)g(of)f(a)g(matc)o
(hing)75 956 y(receiv)o(e.)158 1006 y(A)e Fh(ready)h(send)e
Fo(can)h(start)g(only)f(if)g(a)h(matc)o(hing)e(receiv)o(e)j(is)f(already)f(p)
q(osted;)i(otherwise)g(the)f(op)q(eration)75 1056 y(is)e(erroneous)h(and)e
(its)h(outcome)f(is)g(unde\014ned.)19 b(In)13 b(some)f(systems,)g(this)h
(allo)o(ws)f(the)h(remo)o(v)n(al)d(of)i(a)h(hand-shak)o(e)75
1106 y(op)q(eration)h(that)g(is)f(otherwise)i(required,)g(and)e(results)i(in)
f(impro)o(v)o(ed)e(p)q(erformance.)18 b(Its)c(completion)e(do)q(es)j(not)75
1155 y(dep)q(end)g(on)f(the)g(status)h(of)e(a)h(matc)o(hing)e(receiv)o(e.)158
1205 y(A)20 b Fh(secure)h(send)d Fo(will)g(complete)h(if)g(a)g(matc)o(hing)f
(receiv)o(e)j(is)e(p)q(osted,)i(and)f(the)g(receiv)o(e)h(op)q(eration)e(is)75
1255 y(committed)c(to)h(receiv)o(e)j(the)e(message)g(sen)o(t)h(b)o(y)f(the)g
(secure)i(send.)28 b(\(I.e.,)17 b(the)h(send)g(and)e(receiv)o(e)j(op)q
(erations)75 1305 y(ha)o(v)o(e)h(already)h(b)q(een)g(matc)o(hed,)h(the)f
(receiv)o(e)h(op)q(eration)e(has)h(already)f(started)i(and)e(cannot)h(b)q(e)g
(cancelled)75 1355 y(an)o(ymore.\))c(The)d(start)h(of)e(a)g(secure)j(send)f
(do)q(es)g(not)f(dep)q(end)h(on)e(the)i(status)g(of)e(a)g(matc)o(hing)f
(receiv)o(e.)158 1405 y(A)j(receiv)o(e)i(op)q(eration)e(can)h(o)q(ccur)g(in)f
(one)h(of)f(t)o(w)o(o)g(mo)q(des:)20 b Fh(ST)l(AND)o(ARD)14
b Fo(and)h Fh(SECURE)p Fo(.)h(A)f(secure)75 1454 y(send)i(can)g(b)q(e)g(matc)
o(hed)f(only)g(b)o(y)g(a)g Fh(secure)i(receiv)o(e)p Fo(;)e(a)g(program)f
(where)j(a)e(secure)i(send)g(can)f(b)q(e)g(matc)o(hed)75 1504
y(b)o(y)e(a)f(standard)h(receiv)o(e)h(is)f(erroneous)h(and)f(its)f(b)q(eha)o
(vior)h(is)g(unde\014ned.)22 b(The)15 b(secure)i(receiv)o(e)f(can)f(b)q(e)g
(p)q(osted)75 1554 y(either)g(b)q(efore)f(or)g(after)g(the)h(matc)o(hing)d
(send.)158 1604 y(A)f(standard)g(send)g(and)f(a)h(ready)f(send)i(can)f(b)q
(oth)f(b)q(e)h(matc)o(hed)f(only)g(b)o(y)g(a)g Fh(standard)h(receiv)o(e)p
Fo(;)e(a)i(program)75 1654 y(where)k(a)f(secure)i(receiv)o(e)g(can)e(matc)o
(h)f(a)h(nonsecure)i(send)f(is)g(erroneous)g(and)f(its)g(b)q(eha)o(vior)g(is)
g(unde\014ned.)21 b(The)75 1704 y(receiv)o(e)14 b(has)g(to)f(b)q(e)h(p)q
(osted)g(ahead)f(of)f(the)i(send)g(for)f(a)g(ready)g(send)h(and)f(can)h(b)q
(e)g(p)q(osted)g(either)g(b)q(efore)g(or)f(after)75 1753 y(the)h(send)h(is)f
(p)q(osted)h(for)e(a)h(standard)g(send.)158 1882 y Fm(Discussion:)158
1932 y Fl(I)h(got)h(criticisms)h(ab)q(out)f(the)g(previous)h(notation)g
(\(REGULAR\))e(and)h(the)g(curren)o(t)g(one)g(\(ST)m(AND)o(ARD\).)e(I)h(can)
75 1982 y(also)g(try)e(NORMAL,)g(DEF)l(A)o(UL)m(T,)f(...)18
b(W)m(e)c(should)h(probably)h(v)o(ote,)d(at)h(some)g(p)q(oin)o(t.)19
b(Also,)14 b(Lyndon)h(prop)q(oses)g(to)e(use)75 2031 y(SYNCHR)o(ONOUS,)f
(rather)i(than)g(secure,)g(since)h(the)e(other)h(mo)q(des)g(can)g(also)h
(o\013er)e(secure)i(comm)o(unication)h(with)e(the)75 2081 y(righ)o(t)g(use.)
158 2292 y Fm(Implemen)o(tation)f(note:)158 2338 y Fl(The)g(curren)o(t)h
(de\014nition)i(assumes)e(it)g(is)g(the)f(user)h(resp)q(onsibil)q(it)o(y)i
(to)e(prev)o(en)o(t)g(matc)o(hing)g(of)f(a)h(regular)g(send)g(with)75
2384 y(a)f(secure)h(receiv)o(e)g(and)f(vice-v)o(ersa.)18 b(Otherwise,)c
(messages)g(need)f(b)q(e)g(tagged)h(as)f(standard)i(or)e(secure.)158
2429 y(Assume)i(an)g(implemen)o(tation)j(of)d(MPI)f(comm)o(unication)k(that)d
(a)o(v)o(oids)h(retries.)23 b(What)15 b(proto)q(cols)i(can)e(b)q(e)g(used?)75
2475 y(Since)j(a)e(receiv)o(e)i(ma)o(y)f(ha)o(v)o(e)g(a)g(wildcard)h(source)f
(\014eld,)i(the)d(\014rst)h(step)g(in)h(the)e(proto)q(col)i(m)o(ust)f(b)q(e)g
(executed)h(b)o(y)f(the)75 2521 y(sender.)158 2567 y(F)m(or)c(a)g(ready)h
(send,)f(this)h(\014rst)f(step)h(ma)o(y)f(in)o(v)o(olv)o(e)i(sending)g(the)e
(data,)h(since)g(the)f(receiv)o(er)h(is)g(guaran)o(teed)g(to)f(ha)o(v)o(e)75
2612 y(space)h(to)f(store)g(it.)k(The)c(proto)q(col)h(is)g(send,)f(p)q
(ossibly)j(follo)o(w)o(ed)e(b)o(y)g(an)f(ac)o(k-send.)158 2658
y(The)g(proto)q(col)h(for)e(secure)i(send)f(is)g(lik)o(ely)i(to)e(b)q(e)g
(the)g(follo)o(wing:)19 b(The)13 b(sender)g(sends)h(a)e(request-to-send)i
(message.)75 2704 y(The)f(receiv)o(er)h(stores)f(this)g(request)h(\(whic)o(h)
g(con)o(tains)g(a)f(cop)o(y)g(of)g(the)g(message)g(en)o(v)o(elop)q(e\).)19
b(When)14 b(a)f(matc)o(hing)h(receiv)o(e)p eop
%%Page: 16 17
bop 75 -100 a Fo(16)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fl(is)g(p)q(osted,)h(the)f(receiv)o
(er)h(sends)f(bac)o(k)h(an)f(ac)o(kno)o(wledge,)i(and)e(the)g(sender)h(no)o
(w)e(sends)i(the)f(message.)20 b(The)14 b(proto)q(col)h(is)75
91 y(\(req,)e(ac)o(k-req,)g(send\),)g(p)q(ossibly)j(follo)o(w)o(ed)e(b)o(y)f
(an)h(ac)o(k-send.)158 136 y(What)g(proto)q(col)h(is)f(used)g(for)f(standard)
i(sends?)k(Either)14 b(the)g(second)g(proto)q(col)h(\(req,)e(ac)o(k-req,)g
(send\))h(or)g(a)f(mix)h(of)75 182 y(the)f(\014rst)g(and)h(the)f(second)h
(\(e.g.)j(\014rst)c(used)h(for)e(short)i(messages,)f(second)h(for)f(long)h
(messages\).)158 228 y(When)h(the)g(receiv)o(er)g(do)q(es)g(not)f(wildcard)i
(the)f(source)g(\014eld)g(then)g(the)f(proto)q(col)i(can)f(b)q(e)f(rev)o
(ersed:)21 b(The)14 b(receiv)o(er)75 273 y(sends)h(a)f(clear-to-send)h
(message,)g(and)f(the)h(sender)f(can)h(pro)q(ceed)g(with)f(the)g(short)h
(proto)q(col)g(if)f(it)h(has)f(a)g(clear-to-send)75 319 y(when)f(the)g(send)h
(is)g(executed.)158 365 y(What)f(do)g(w)o(e)g(gain)g(from)g(the)g(\\secure)g
(receiv)o(e"?)19 b(Some)13 b(optimizations)i(for)e(regular)h(receiv)o(es)g
(\(e.g.,)d(in)j(a)e(regular)75 410 y(receiv)o(e)18 b(of)f(a)g(short)g
(message)h(the)f(receiv)o(er)h(kno)o(ws)g(the)f(short)g(proto)q(col)i(will)f
(b)q(e)g(used,)g(and)g(need)f(not)h(c)o(hec)o(k)f(for)g(a)75
456 y(request-to-send\).)h(Q:)12 b(is)i(that)f(enough)i(justi\014cation)g
(for)e(the)g(secure)g(receiv)o(e?)158 502 y(A)k(coun)o(ter)g(argumen)o(t)h
(\(Leslie)g(Hart\))f(is)h(that)f(one)g(could)i(conceiv)o(e)f(of)f(an)g
(implemen)o(tation)j(where)d(a)g(regular)75 547 y(receiv)o(e)12
b(reserv)o(es)g(bu\013er)g(space,)f(so)h(that)f(a)g(program)h(that)f(use)h
(regular)g(receiv)o(es)g(is)g(not)f(safe,)g(ev)o(en)h(if)f(only)h(secure)g
(sends)75 593 y(are)h(used.)18 b(I)12 b(doubt)i(this)g(is)f(a)g(strong)h
(argumen)o(t.)k(On)13 b(the)g(other)g(hand,)h(it)f(indicates)i(that,)e(if)g
(the)g(exp)q(ectation)i(is)f(that)75 639 y(programs)i(written)g(using)g(only)
h(secure)e(sends)h(are)g(safe,)f(then)h(this)g(is)f(an)h(additional)i
(correctness)e(condition)i(on)e(the)75 684 y(implemen)o(tation)g(that)d(need)
h(b)q(e)f(made)h(explicit.)75 903 y Fn(1.8)70 b(Comm)n(unication)20
b(Ob)t(jects)75 994 y Fo(An)15 b(opaque)g(comm)o(unication)d(ob)r(ject)k
(iden)o(ti\014es)g(v)n(arious)e(prop)q(erties)j(of)e(a)g(comm)o(uni)o(cation)
d(op)q(eration,)j(suc)o(h)75 1043 y(as)h(the)g(bu\013er)h(descriptor)h(that)d
(is)h(asso)q(ciated)h(with)e(it,)h(its)g(con)o(text,)g(the)h(tag)e(and)h
(destination)g(parameters)75 1093 y(to)f(b)q(e)h(used)g(for)f(a)g(send,)h(or)
f(the)h(tag)f(and)g(source)h(parameters)g(to)f(b)q(e)g(used)i(for)d(a)h
(receiv)o(e.)24 b(In)15 b(addition,)f(this)75 1143 y(ob)r(ject)g(stores)h
(information)10 b(ab)q(out)j(the)h(status)g(of)e(the)i(last)f(comm)o
(unication)d(op)q(eration)j(that)g(w)o(as)g(p)q(erformed)75
1193 y(with)h(this)f(ob)r(ject.)19 b(This)14 b(ob)r(ject)h(is)f(accessed)i
(using)d(a)h(comm)o(unicatio)o(n)d(handle.)158 1243 y(One)k(can)f(consider)h
(comm)o(uni)o(cation)c(op)q(erations)j(to)g(consist)g(of)g(the)g(follo)o
(wing)d(sub)q(op)q(erations:)75 1324 y Fh(INIT\(op)q(eration,)j(params,)i
(handle\))i Fo(Pro)q(cess)c(pro)o(vides)f(all)f(relev)n(an)o(t)g(parameters)h
(for)f(its)h(participation)179 1374 y(in)i(the)i(comm)o(unication)c(op)q
(eration)j(\(t)o(yp)q(e)g(of)g(op)q(eration,)g(data)g(bu\013er,)h(tag,)e
(participan)o(ts,)i(etc.\).)25 b(An)179 1424 y(ob)r(ject)14
b(is)g(created)i(that)e(iden)o(ti\014es)g(the)g(op)q(eration.)75
1503 y Fh(ST)l(AR)l(T\(handle\))k Fo(The)c(comm)o(unication)c(op)q(eration)k
(is)g(started)75 1582 y Fh(COMPLETE\(handle\))19 b Fo(The)14
b(comm)o(unication)c(op)q(eration)k(is)g(completed.)75 1661
y Fh(FREE\(handle\))k Fo(The)d(comm)o(unicati)o(on)c(ob)r(ject,)j(and)g(asso)
q(ciated)h(resources)h(are)e(freed.)75 1743 y(Correct)h(in)o(v)o(o)q(cation)e
(of)g(these)i(sub)q(op)q(erations)g(is)f(a)f(sequence)k(of)c(the)h(form)587
1825 y Fh(INIT)j Fo(\()p Fh(ST)l(AR)l(T)f(COMPLETE)p Fo(\))1187
1808 y Fb(\003)1223 1825 y Fh(FREE)p Fd(:)75 1907 y Fo(I.e.,)10
b(an)h(ob)r(ject)g(needs)h(b)q(e)f(created)i(b)q(efore)e(comm)o(unication)c
(o)q(ccurs;)13 b(it)d(can)h(b)q(e)g(reused)i(only)c(after)i(the)h(previous)75
1956 y(use)k(has)g(completed;)f(and)g(it)g(needs)i(to)e(b)q(e)h(freed)h(ev)o
(en)o(tually)d(\(of)h(course,)i(one)f(can)f(assume)g(that)h(all)e(ob)r(jects)
75 2006 y(are)g(freed)h(at)f(program)e(termination,)g(b)o(y)h(default\).)158
2056 y(The)21 b(ab)q(o)o(v)o(e)g(scenario)g(p)q(ertains)g(to)g
Fj(p)n(ersistent)f Fo(ob)r(jects.)39 b(One)22 b(can)f(also)f(create)i
Fj(ephemer)n(al)e Fo(ob)r(jects.)75 2106 y(Suc)o(h)c(an)g(ob)r(ject)h(p)q
(ersists)h(only)d(un)o(til)g(the)i(comm)o(uni)o(cation)c(op)q(eration)j(is)g
(completed,)f(at)h(whic)o(h)g(p)q(oin)o(t)f(it)h(is)75 2156
y(destro)o(y)o(ed.)i(Th)o(us)12 b(the)g(correct)i(in)o(v)o(o)q(cation)c(of)h
(sub)q(op)q(erations)i(with)e(an)g(ephemeral)g(ob)r(ject)i(is)e
Fh(INIT)j(ST)l(AR)l(T)75 2205 y(COMPLETE)p Fo(.)158 2255 y(A)19
b(user)i(ma)o(y)c(directly)j(in)o(v)o(ok)o(e)f(these)h(sub)q(op)q(erations.)
36 b(This)19 b(w)o(ould)g(allo)o(w)e(the)j(amortization)e(of)g(the)75
2305 y(o)o(v)o(erhead)d(of)g(setting)g(up)g(a)g(comm)o(unication)d(o)o(v)o
(er)j(man)o(y)e(successiv)o(e)k(uses)f(of)f(the)g(same)g(handle,)f(and)h
(allo)o(ws)75 2355 y(the)d(o)o(v)o(erlap)f(of)g(comm)o(unication)e(and)i
(computation.)16 b(Simpler)10 b(comm)o(unicatio)o(n)f(op)q(erations)j(com)o
(bine)e(sev)o(eral)75 2405 y(of)17 b(these)i(sub)q(op)q(erations)g(in)o(to)e
(one)h(op)q(eration,)g(th)o(us)g(simplifying)d(the)j(use)h(of)e(comm)o
(unicatio)o(n)e(primitiv)o(es.)75 2455 y(Th)o(us,)h(one)h(only)e(needs)j(to)e
(sp)q(ecify)g(precisely)h(the)g(seman)o(tics)f(of)f(these)j(sub)q(op)q
(erations)f(in)f(order)g(to)g(sp)q(ecify)75 2504 y(the)e(seman)o(tics)g(of)f
(MPI)h(p)q(oin)o(t)g(to)f(p)q(oin)o(t)h(comm)o(unicatio)o(n)d(op)q(erations.)
158 2554 y(W)m(e)i(sa)o(y)g(that)g(a)g(comm)o(unicatio)o(n)d(op)q(eration)j
(\(send)i(or)e(receiv)o(e\))h(is)f Fh(p)q(osted)f Fo(once)i(a)f
Fh(start)e Fo(sub)q(op)q(eration)75 2604 y(has)18 b(b)q(een)h(in)o(v)o(ok)o
(ed;)g(the)g(op)q(eration)f(is)g Fh(completed)e Fo(once)i(the)h
Fh(complete)d Fo(sub)q(op)q(eration)j(has)f(completed.)75 2654
y(A)e(send)h(and)f(a)g(receiv)o(e)i(op)q(eration)e Fh(matc)o(h)f
Fo(if)g(the)i(receiv)o(e)h(pattern)f(sp)q(eci\014ed)g(b)o(y)f(the)h(receiv)o
(e)h(matc)o(hes)d(the)75 2704 y(message)f(en)o(v)o(elop)q(e)g(created)h(b)o
(y)f(the)g(send.)p eop
%%Page: 17 18
bop 75 -100 a Fg(1.8.)31 b(COMMUNICA)m(TION)14 b(OBJECTS)1073
b Fo(17)75 45 y Fk(1.8.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)75
206 y Fm(Discussion:)35 b Fl(This)14 b(section)g(has)f(not)h(y)o(et)f(b)q
(een)g(appro)o(v)o(ed)158 339 y Fo(An)c(ob)r(ject)i(for)d(a)h(send)i(op)q
(eration)e(is)g(created)i(b)o(y)e(a)g(call)f(to)h Fh(MPI)p
1180 339 15 2 v 18 w(INIT)p 1304 339 V 17 w(SEND)p Fo(.)g(A)g(call)g(to)g
Fh(MPI)p 1728 339 V 17 w(INIT)p 1851 339 V 18 w(RECV)75 389
y Fo(is)17 b(similarly)c(used)18 b(for)f(creating)g(an)f(ob)r(ject)i(for)e(a)
h(receiv)o(e)h(op)q(eration.)26 b(The)18 b(creation)f(of)f(a)g(comm)o
(unication)75 439 y(ob)r(ject)f(is)e(a)h(lo)q(cal)f(op)q(eration)h(that)g
(need)h(not)e(in)o(v)o(olv)o(e)g(comm)o(unicatio)o(n)e(with)j(a)f(remote)h
(pro)q(cess.)158 525 y Fh(MPI)p 257 525 V 17 w(INIT)p 380 525
V 18 w(SEND)i(\(handle,)e(bu\013er)p 856 525 V 15 w(handle,)g(dest,)h(tag,)h
(con)o(text,)f(mo)q(de,)g(p)q(ersistence\))158 611 y Fo(Creates)g(a)f(send)h
(comm)o(unicati)o(on)c(ob)r(ject.)19 b(P)o(arameters)14 b(are)75
696 y Fh(OUT)i(handle)i Fo(message)g(handle.)28 b(The)18 b(handle)g(should)f
(not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g(ob)r(ject)h(b)q(efore)g(the)
179 746 y(call.)75 832 y Fh(IN)e(bu\013er)p 273 832 V 16 w(handle)j
Fo(handle)13 b(to)h(send)h(bu\013er)g(descriptor)75 918 y Fh(IN)h(dest)k
Fo(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o(teger\))75
1004 y Fh(IN)i(tag)21 b Fo(user)15 b(tag)e(for)h(messages)g(sen)o(t)g(with)g
(this)g(handle)g(\(in)o(teger\))75 1090 y Fh(IN)i(con)o(text)k
Fo(con)o(text)14 b(of)f(messages)h(sen)o(t)h(with)f(this)g(handle)75
1176 y Fh(IN)i(mo)q(de)21 b Fo(send)14 b(mo)q(de)f(\(state)i(t)o(yp)q(e,)f
(with)f(three)j(v)n(alues:)h Fi(MPI)p 1119 1176 14 2 v 15 w(STANDARD)p
Fo(,)12 b Fi(MPI)p 1400 1176 V 15 w(READY)h Fo(and)h Fi(MPI)p
1685 1176 V 15 w(SECURE)p Fo(\))75 1262 y Fh(IN)i(p)q(ersistence)i
Fo(handle)10 b(p)q(ersistence)i(\(state)e(t)o(yp)q(e,)g(with)f(t)o(w)o(o)g(v)
n(alues:)15 b Fi(MPI)p 1318 1262 V 15 w(PERSISTENT)8 b Fo(and)h
Fi(MPI)p 1703 1262 V 15 w(EPHEMERAL)p Fo(\))158 1383 y Fh(MPI)p
257 1383 15 2 v 17 w(INIT)p 380 1383 V 18 w(RECV)15 b(\(handle,)e(bu\013er)p
859 1383 V 15 w(handle,)g(source,)h(tag,)g(con)o(text,)f(mo)q(de,)i(p)q
(ersistence\))158 1469 y Fo(Create)g(a)e(receiv)o(e)j(handle.)i(P)o
(arameters)c(are)75 1555 y Fh(OUT)i(handle)i Fo(message)g(handle.)28
b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g
(ob)r(ject)h(b)q(efore)g(the)179 1604 y(call.)75 1690 y Fh(IN)e(bu\013er)p
273 1690 V 16 w(handle)j Fo(handle)13 b(to)h(receiv)o(e)h(bu\013er)g
(descriptor.)75 1777 y Fh(IN)h(source)k Fo(rank)14 b(in)f(con)o(text)i(of)e
(source,)i(or)f Fi(MPI)p 897 1777 14 2 v 15 w(ANY)p 978 1777
V 15 w(SOURCE)f Fo(\(in)o(teger\).)75 1863 y Fh(IN)j(tag)21
b Fo(user)15 b(tag)e(for)h(messages)g(receiv)o(ed)h(with)f(this)f(handle,)h
(or)g Fi(MPI)p 1232 1863 V 15 w(ANY)p 1313 1863 V 15 w(TAG)f
Fo(\(in)o(teger\).)75 1949 y Fh(IN)j(con)o(text)k Fo(con)o(text)14
b(of)f(messages)h(receiv)o(ed)i(with)d(this)h(handle.)75 2035
y Fh(IN)i(mo)q(de)21 b Fo(receiv)o(e)15 b(mo)q(de)e(\(state)h(t)o(yp)q(e,)g
(with)g(t)o(w)o(o)f(v)n(alues:)18 b Fi(MPI)p 1133 2035 V 15
w(STANDARD)p Fo(,)12 b(and)i Fi(MPI)p 1495 2035 V 15 w(SECURE)p
Fo(\))75 2121 y Fh(IN)i(p)q(ersistence)i Fo(handle)10 b(p)q(ersistence)i
(\(state)e(t)o(yp)q(e,)g(with)f(t)o(w)o(o)g(v)n(alues:)15 b
Fi(MPI)p 1318 2121 V 15 w(PERSISTENT)8 b Fo(and)h Fi(MPI)p
1703 2121 V 15 w(EPHEMERAL)p Fo(\))158 2206 y(See)15 b(Section)f(1.4.2)e(for)
i(a)f(discussion)i(of)e(source,)i(tag)e(and)h(con)o(text.)158
2335 y Fm(Discussion:)35 b Fl(I)12 b(ha)o(v)o(e)h(not)g(included)i(prop)q
(osals)f(for)f(partially)i(sp)q(eci\014ed)f(message)f(handles,)i(that)d(some)
h(p)q(eoples)75 2381 y(seem)g(to)g(desire.)158 2427 y(I)g(ha)o(v)o(e)g
(merged)h(all)g(handle)h(setup)e(in)o(to)h(one)g(call.)158
2474 y(The)c(functions)i Ff(MPI)p 459 2474 12 2 v 13 w(INIT)p
552 2474 V 12 w(SEND)c Fl(and)j Ff(MPI)p 784 2474 V 13 w(INIT)p
877 2474 V 13 w(RECV)d Fl(ha)o(v)o(e)j(the)f(same)g(list)h(of)f(parameters.)
17 b(Jim)10 b(Co)o(wnie)h(suggests)75 2519 y(to)i(ha)o(v)o(e)g(only)i(one)e
(function)i(with)e(an)g(extra)h(argumen)o(t.)158 2566 y(Ephemeral)f(\\p)q
(ersisten)o(t)h(handles")f(are)f(not)g(terribly)h(useful)g(as)f(suc)o(h.)17
b(But)12 b(their)g(existence)h(allo)o(w)g(us)f(to)f(describ)q(e)75
2612 y(all)j(the)f(higher)i(lev)o(el)f(send/receiv)o(e)h(op)q(erations)g(in)f
(terms)f(of)g(basic)h(op)q(erations)h(on)e(handles.)p eop
%%Page: 18 19
bop 75 -100 a Fo(18)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fk(1.8.2)55 b(Comm)n(unication)21
b(Start)75 157 y Fh(MPI)p 174 157 15 2 v 17 w(ST)l(AR)l(T\(handle\))75
274 y(IN)16 b(handle)j Fo(comm)o(unication)10 b(handle)158
355 y(The)16 b Fi(MPI)p 314 355 14 2 v 15 w(START)f Fo(function)h(starts)g
(the)h(execution)f(of)g(a)f(comm)o(unication)e(op)q(eration)i(\(send)i(or)f
(receiv)o(e\).)75 404 y(A)i(sender)i(should)d(not)h(up)q(date)h(the)f(send)h
(bu\013er)g(after)f(a)g(send)h(op)q(eration)f(has)g(started)h(un)o(til)e
(after)h(it)g(has)75 454 y(completed.)31 b(A)18 b(receiv)o(er)i(should)e(not)
h(access)h(the)f(receiv)o(e)h(bu\013er)f(after)g(a)f(receiv)o(e)h(op)q
(eration)g(w)o(as)f(started)75 504 y(un)o(til)c(after)h(it)f(has)h
(completed.)21 b(A)15 b(program)e(that)i(do)q(es)g(not)g(satisfy)g(these)h
(conditions)e(is)h(erroneous)h(and)f(its)75 554 y(outcome)e(is)h
(undetermined.)75 670 y Fk(1.8.3)55 b(Comm)n(unication)21 b(Completion)75
782 y Fh(MPI)p 174 782 15 2 v 17 w(W)-5 b(AIT)17 b(\()e(handle,)f(return)p
675 782 V 15 w(status)p 816 782 V 16 w(handle\))75 898 y(IN)i(handle)j
Fo(comm)o(unication)10 b(handle)75 980 y Fh(IN)16 b(return)p
284 980 V 15 w(handle)j Fo(handle)13 b(that)h(is)f(asso)q(ciated)i(with)e
(return)i(status)f(ob)r(ject.)19 b(the)14 b(return)g(status)h(ob)r(ject)f(is)
179 1030 y(up)q(dated)g(b)o(y)g(call.)158 1111 y(A)g(call)e(to)i
Fh(MPI)p 428 1111 V 17 w(W)-5 b(AIT)14 b Fo(returns)h(when)f(the)g(send)g(op)
q(eration)g(iden)o(ti\014ed)f(b)o(y)h Fi(handle)e Fo(is)h(complete.)18
b(The)75 1161 y(completion)13 b(of)h(a)g(send)h(op)q(eration)g(indicates)g
(that)f(the)h(sender)i(is)d(no)o(w)g(free)i(to)e(up)q(date)h(the)g(lo)q
(cations)f(in)g(the)75 1211 y(send)e(bu\013er,)h(or)e(an)o(y)g(other)h(lo)q
(cation)e(that)i(can)f(b)q(e)h(referenced)i(b)o(y)d(the)h(send)g(op)q
(eration.)17 b(Ho)o(w)o(ev)o(er,)12 b(it)f(do)q(es)h(not)75
1261 y(indicate)f(that)h(the)g(message)f(has)h(b)q(een)g(receiv)o(ed;)h
(rather)g(it)e(ma)o(y)e(ha)o(v)o(e)j(b)q(een)g(bu\013ered)h(b)o(y)e(the)h
(comm)o(unication)75 1310 y(subsystem.)158 1360 y(The)20 b(completion)e(of)i
(a)f(receiv)o(e)i(op)q(eration)f(indicates)g(that)g(the)h(receiv)o(er)g(is)f
(no)o(w)f(free)i(to)f(access)h(the)75 1410 y(lo)q(cations)15
b(in)g(the)h(receiv)o(e)h(bu\013er,)f(whic)o(h)f(con)o(tain)g(the)h(receiv)o
(ed)h(message,)e(or)h(an)o(y)f(other)h(lo)q(cation)e(that)i(can)75
1460 y(b)q(e)g(referenced)i(b)o(y)d(the)h(receiv)o(e)h(op)q(eration.)23
b(It)15 b(do)q(es)h(not)g(indicate)f(that)h(the)g(matc)o(hing)d(send)j(op)q
(eration)g(has)75 1510 y(completed.)158 1559 y(The)k(call)e(returns)j(a)e
(handle)h(to)f(an)g(opaque)g(ob)r(ject)h(that)g(con)o(tains)f(information)d
(on)k(the)g(completed)75 1609 y(op)q(eration)14 b({)f(the)i
Fh(return)f(status)e Fo(ob)r(ject.)158 1694 y Fh(MPI)p 257
1694 V 17 w(ST)l(A)l(TUS)j(\(handle,)f(\015ag,)i(return)p 898
1694 V 15 w(handle\))75 1811 y(IN)g(handle)j Fo(comm)o(unication)10
b(handle)75 1893 y Fh(OUT)16 b(\015ag)k Fo(logical)75 1975
y Fh(OUT)c(return)p 335 1975 V 15 w(handle)i Fo(handle)c(that)g(is)g(asso)q
(ciated)g(with)g(return)h(status)g(ob)r(ject.)158 2056 y(A)e(call)f(to)h
Fh(MPI)p 426 2056 V 17 w(ST)l(A)l(TUS)f Fo(returns)j Fi(flag=true)c
Fo(if)h(the)h(op)q(eration)g(iden)o(ti\014ed)g(b)o(y)g Fi(handle)e
Fo(is)i(complete.)75 2106 y(In)j(suc)o(h)g(case,)g(the)h(return)f(handle)g(p)
q(oin)o(ts)f(to)h(an)f(opaque)h(ob)r(ject)g(that)g(con)o(tains)f(information)
e(on)i(the)h(com-)75 2156 y(pleted)f(op)q(eration.)k(It)c(returns)h
Fi(flag=false)p Fo(,)c(otherwise.)20 b(In)15 b(suc)o(h)g(case,)g(the)g(v)n
(alue)f(of)f(the)i(return)h(handle)e(is)75 2205 y(unde\014ned.)158
2255 y(A)f(call)g(to)g Fi(MPI)p 397 2255 14 2 v 15 w(WAIT)f
Fo(blo)q(c)o(ks)h(only)f(the)i(executing)g(thread.)k(If)13
b(the)h(executing)g(pro)q(cess)h(is)e(m)o(ultithreaded,)75
2305 y(then)i(other)f(threads)h(within)e(the)i(pro)q(cess)g(can)f(b)q(e)h(sc)
o(heduled)g(for)f(execution.)158 2355 y(The)h(use)h(of)f(a)f(blo)q(c)o(king)h
(receiv)o(e)h(op)q(eration)f(\()p Fi(MPI)p 977 2355 V 15 w(WAIT)p
Fo(\))f(allo)o(ws)g(the)i(op)q(erating)f(system)f(to)h(desc)o(hedule)75
2405 y(the)j(blo)q(c)o(k)o(ed)g(thread)h(and)e(sc)o(hedule)i(another)g
(thread)f(for)f(execution,)j(if)c(suc)o(h)j(is)f(a)o(v)n(ailable.)27
b(The)18 b(use)h(of)e(a)75 2455 y(non)o(blo)q(c)o(king)12 b(receiv)o(e)i(op)q
(eration)f(\()p Fi(MPI)p 710 2455 V 15 w(STATUS)p Fo(\))f(allo)o(ws)g(the)i
(user)g(to)f(sc)o(hedule)h(alternativ)o(e)f(activities)g(within)75
2504 y(a)h(single)f(thread)i(of)e(execution.)158 2554 y(The)g(in)o(tended)g
(implemen)o(tation)c(of)j Fi(MPI)p 820 2554 V 16 w(STATUS)f
Fo(is)h(for)h(that)f(op)q(eration)h(to)f(return)i(as)f(so)q(on)f(as)h(p)q
(ossible.)75 2604 y(Ho)o(w)o(ev)o(er,)h(if)f(rep)q(eatedly)i(called)f(for)f
(an)h(op)q(eration)g(that)f(is)h(enabled,)g(it)f(m)o(ust)g(ev)o(en)o(tually)h
(succeed.)158 2654 y(The)g(return)h(status)g(ob)r(ject)g(for)e(a)h(send)h(op)
q(eration)e(carries)i(no)f(information.)h(The)f(return)h(status)g(ob)r(ject)
75 2704 y(for)h(a)g(receiv)o(e)i(op)q(eration)f(carries)g(information)d(on)i
(the)i(source,)g(tag)e(and)g(length)h(of)f(the)h(receiv)o(ed)h(message.)p
eop
%%Page: 19 20
bop 75 -100 a Fg(1.8.)31 b(COMMUNICA)m(TION)14 b(OBJECTS)1073
b Fo(19)75 45 y(These)13 b(\014elds)g(are)f(required)h(b)q(ecause)h(the)e
(receiv)o(e)i(op)q(eration)e(ma)o(y)e(ha)o(v)o(e)h(sp)q(eci\014ed)j
Fi(DONTCARE)c Fo(in)i(either)h(source)75 95 y(or)h(tag)f(\014eld,)h(and)g
(the)g(message)g(ma)o(y)e(ha)o(v)o(e)h(b)q(een)i(shorter)g(than)f(the)h
(receiv)o(e)g(bu\013er.)158 180 y Fh(MPI)p 257 180 15 2 v 17
w(QUER)l(Y\()h(handle,)e(coun)o(t,)g(source,)h(tag\))75 295
y(IN)h(handle)j Fo(handle)14 b(to)g(return)h(status)f(ob)r(ject)75
376 y Fh(OUT)i(coun)o(t)j Fo(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(receiv)o
(ed.)75 457 y Fh(OUT)i(source)j Fo(rank)14 b(of)g(message)f(sender)j(in)d
(message)h(con)o(text)g(\(in)o(teger\).)75 538 y Fh(OUT)i(tag)k
Fo(tag)14 b(of)f(receiv)o(ed)i(message)f(\(in)o(teger\).)158
652 y Fh(MPI)p 257 652 V 17 w(CREA)l(TE)p 471 652 V 19 w(ST)l(A)l(TUS\()h
(handle\))158 738 y Fo(Creates)g(a)f(new)g(return)h(status)g(ob)r(ject.)75
817 y Fh(OUT)h(handle)i Fo(handle)c(to)g(return)h(status)g(ob)r(ject.)158
974 y Fm(Discussion:)158 1020 y Fl(The)c(use)g(of)f(a)h(return)g(status)g(ob)
r(ject,)g(rather)g(than)g(a)g(list)h(of)e(parameters)i(ma)o(y)f(simplify)i
(the)e(use)g(of)f(MPI)h(routines,)75 1066 y(if)i(the)f(v)n(alues)i(stored)f
(in)g(the)f(ob)r(ject)h(are)f(seldom)i(c)o(hec)o(k)o(ed.)j(A)12
b(prede\014ned)i(return)f(status)g(ob)r(ject)f(should)j(b)q(e)d(pro)o(vided,)
75 1111 y(to)h(ease)g(programming.)19 b(The)12 b(main)i(reason)g(for)e(the)h
(use)g(of)g(a)f(return-status)i(ob)r(ject)f(is)g(that)g(one)h(w)o(an)o(ts)e
(to)h(b)q(e)g(able)h(to)75 1157 y(use)c(the)h(same)f Ff(MPI)p
357 1157 12 2 v 13 w(WAIT)e Fl(and)j Ff(MPI)p 590 1157 V 13
w(STATUS)c Fl(calls)12 b(for)e(c)o(hec)o(king)h(on)g(di\013eren)o(t)g(t)o(yp)
q(es)g(of)f(op)q(erations,)i(e.g.)k(b)q(oth)10 b(sends)h(and)75
1203 y(receiv)o(es,)k(and)f(p)q(erhaps)h(new)e(op)q(erations)j(in)e(the)g
(future.)19 b(These)14 b(di\013eren)o(t)h(op)q(erations)g(return)g
(di\013eren)o(t)g(information)75 1248 y(\(alb)q(eit)g(w)o(e)d(could)j(decree)
e(that)g(a)g(send)h(returns)g(the)f(same)g(information)i(as)e(a)g(receiv)o
(e\).)158 1294 y(The)h(issue)h(of)f(memory)g(allo)q(cation)j(for)d(return)p
868 1294 V 15 w(status)g(handles)i(is)f(not)f(y)o(et)g(solv)o(ed.)22
b(There)14 b(is)g(a)g(desire)h(that)g(the)75 1340 y(user)e(will)i(b)q(e)e
(able)h(to)f(allo)q(cate)i(space)f(for)e(them)i(\(e.g.,)e(on)h(the)g(stac)o
(k\).)158 1385 y(W)m(e)j(return)g(n)o(um)o(b)q(er)g(of)f(elemen)o(ts)i
(receiv)o(ed,)g(rather)e(than)i(b)o(ytes)e(receiv)o(ed)i(b)q(ecause)g(\\n)o
(um)o(b)q(er)f(of)f(elemen)o(ts")i(is)75 1431 y(closer)c(to)f(the)g
(application)j(seman)o(tic)f(lev)o(el)f(\(user)f(do)q(es)h(not)f(need)h(to)f
(b)q(e)g(a)o(w)o(are)g(of)g(the)g(size)h(of)f(elemen)o(ts\),)g(and)h(is)g
(more)75 1477 y(in)o(v)n(arian)o(t.)23 b(The)14 b(return)h(status)f(ob)r
(ject)h(returns)g(the)f(n)o(um)o(b)q(er)h(of)f Fm(elemen)o(ts)g
Fl(receiv)o(ed.)22 b(If)13 b(there)i(w)o(as)f(no)g(truncation,)75
1522 y(then)h(this)g(is)g(equal)g(to)f(the)h(n)o(um)o(b)q(er)g(of)f(elemen)o
(ts)h(sen)o(t.)21 b(On)14 b(the)g(other)h(hand,)g(the)f(n)o(um)o(b)q(er)h(of)
f(b)o(ytes)h(collected)h(from)75 1568 y(the)d(sender)i(memory)m(,)e(the)g(n)o
(um)o(b)q(er)h(of)f(b)o(ytes)h(sen)o(t)g(o)o(v)o(er)f(the)h(wire,)f(and)h
(the)f(n)o(um)o(b)q(er)i(of)e(b)o(ytes)g(stored)h(in)g(the)g(receiv)o(er)75
1614 y(memory)c(ma)o(y)g(all)i(b)q(e)e(di\013eren)o(t,)i(in)e(a)g
(heterogeneous)i(en)o(vironmen)o(t.)18 b(The)10 b(n)o(um)o(b)q(er)g(of)g
(elemen)o(ts)h(sen)o(t)f(can)h(b)q(e)f(computed)75 1659 y(form)k(the)g(send)h
(bu\013er)h(descriptor;)g(the)e(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)i(receiv)o
(ed)f(can)g(b)q(e)g(computed)g(from)f(the)h(receiv)o(e)g(bu\013er)75
1705 y(descriptor)g(and)e(the)g(length)i(\(in)e(b)o(ytes\))h(of)f(the)g
(receiv)o(ed)h(message.)158 1751 y Ff(MPI)p 220 1751 V 13 w(STATUS)g
Fl(and)j Ff(MPI)p 505 1751 V 13 w(QUERY)d Fl(are)i(not)h(go)q(o)q(d)g(names)g
({)g(they)f(can)h(b)q(e)g(easily)h(confused.)28 b(Leslie)18
b(Hart)e(suggests)75 1796 y Ff(MPI)p 137 1796 V 13 w(TEST)d
Fl(or)i Ff(MPI)p 352 1796 V 13 w(CHECK)e Fl(for)h(the)h(\014rst.)23
b(P)o(erhaps)16 b(w)o(e)e(should)j(replace)f Ff(MPI)p 1229
1796 V 13 w(STATUS)c Fl(with)j Ff(MPI)p 1523 1796 V 13 w(TEST)e
Fl(and)j Ff(MPI)p 1766 1796 V 13 w(QUERY)75 1842 y Fl(with)d
Ff(MPI)p 224 1842 V 13 w(STATUS)p Fl(.)75 2040 y Fk(1.8.4)55
b(Multiple)20 b(Completions)75 2117 y Fo(It)c(is)g(con)o(v)o(enien)o(t)h(to)f
(b)q(e)h(able)e(to)h(w)o(ait)g(for)g(the)g(completion)f(of)g(an)o(y)h(or)g
(all)f(the)i(op)q(erations)f(in)g(a)g(set,)h(rather)75 2166
y(than)c(ha)o(ving)f(to)g(w)o(ait)g(for)h(a)g(sp)q(eci\014c)h(message.)j(A)c
(call)g(to)f Fi(MPI)p 1096 2166 14 2 v 15 w(WAITANY)g Fo(or)h
Fi(MPI)p 1393 2166 V 15 w(STATUSANY)e Fo(can)i(b)q(e)g(used)h(to)75
2216 y(w)o(ait)f(for)h(the)h(completion)d(of)h(one)i(out)f(of)f(sev)o(eral)i
(op)q(erations;)e(a)h(call)f(to)h Fi(MPI)p 1340 2216 V 15 w(WAITALL)f
Fo(can)h(b)q(e)h(used)g(to)f(w)o(ait)75 2266 y(for)g(all)e(p)q(ending)i(op)q
(erations)g(in)g(a)f(list.)158 2351 y Fh(MPI)p 257 2351 15
2 v 17 w(W)-5 b(AIT)l(ANY)17 b(\()f(list)p 628 2351 V 15 w(of)p
682 2351 V 16 w(handles,)e(coun)o(t,)h(index,)f(return)p 1306
2351 V 15 w(handle\))158 2436 y Fo(Blo)q(c)o(ks)g(un)o(til)f(one)h(of)f(the)h
(op)q(erations)g(asso)q(ciated)h(with)e(the)i(comm)o(uni)o(cation)c(handles)j
(in)f(the)h(arra)o(y)g(has)75 2486 y(completed.)19 b(Returns)c(the)g(index)g
(of)e(that)i(handle)f(in)g(the)h(arra)o(y)m(,)e(and)i(returns)h(the)f(status)
g(of)f(that)g(op)q(eration)75 2536 y(in)f(the)i(ob)r(ject)g(asso)q(ciated)f
(with)g(the)g(return)p 798 2536 13 2 v 16 w(handle.)k(The)d(parameters)f
(are:)75 2623 y Fh(IN)i(list)p 215 2623 15 2 v 15 w(of)p 269
2623 V 17 w(handles)j Fo(list)13 b(of)g(handles)h(to)g(comm)o(unication)d(ob)
r(jects.)75 2704 y Fh(IN)16 b(coun)o(t)j Fo(list)14 b(length)g(\(in)o
(teger\))p eop
%%Page: 20 21
bop 75 -100 a Fo(20)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fh(OUT)i(index)j Fo(index)14
b(of)f(handle)h(for)f(op)q(eration)h(that)g(completed)f(\(in)o(teger\).)75
129 y Fh(OUT)j(return)p 335 129 15 2 v 15 w(handle)i Fo(handle)d(that)g(is)g
(asso)q(ciated)h(with)e(return)i(status)g(ob)r(ject.)22 b(Set)16
b(to)f(return)h(status)g(of)179 179 y(op)q(eration)e(that)f(completed.)158
271 y(The)h(successful)i(execution)e(of)f Fi(MPI)p 733 271
14 2 v 16 w(WAITANY\(list)p 1013 271 V 13 w(of)p 1070 271 V
15 w(handles,)20 b(index,)h(return)p 1566 271 V 14 w(handle\))12
b Fo(has)i(the)75 321 y(same)i(e\013ect)i(as)f(the)h(successful)g(execution)g
(of)e Fi(MPI)p 922 321 V 15 w(WAIT\(handle[i],)j(return)p 1418
321 V 14 w(handle\))p Fo(,)d(where)i Fi(i)f Fo(is)f(the)75
371 y(v)n(alue)g(returned)i(b)o(y)e Fi(index)g Fo(and)g Fi(handle[i])f
Fo(is)h(the)h Fi(i)p Fo(-th)f(handle)h(in)f(the)h(list,)f(and)g(the)h
(cancellation)f(of)g(all)75 421 y(remaining)c(w)o(ait)h(op)q(erations.)158
471 y(If)i(more)e(then)j(one)f(op)q(eration)g(is)g(enabled)g(and)g(can)g
(terminate,)f(one)h(is)g(arbitrarily)f(c)o(hosen)i(\(sub)r(ject)g(to)75
521 y(the)e(restrictions)i(on)d(op)q(eration)h(termination)e(order,)i(and)g
(fairness,)g(see)h(Section)f(1.13\).)158 571 y Fi(MPI)p 227
571 V 15 w(WAITANY)20 b(\()i(list)p 548 571 V 15 w(of)p 607
571 V 15 w(handles,)e(count,)h(index,)f(return)p 1255 571 V
15 w(status)p 1402 571 V 14 w(handle\))13 b Fo(is)140 655 y
Fi({MPI_WAIT)20 b(\(handle[0],)g(return_handle\);)f(index)h(=)i(0})f(||)h
(...)75 705 y(||)140 755 y({MPI_WAIT)e(\(handle[count-1],)f(return_handle\);)
f(index)j(=)h(count-1})158 838 y Fo(\(\\)p Fe(jj)p Fo(")13
b(indicates)h(c)o(hoice;)g(one)g(of)f(the)i(alternativ)o(es)f(is)f(c)o
(hosen,)i(nondeterministically)m(.\))158 924 y Fh(MPI)p 257
924 15 2 v 17 w(ST)l(A)l(TUSANY)h(\()g(list)p 685 924 V 15
w(of)p 739 924 V 16 w(handles,)e(coun)o(t,)h(index,)f(return)p
1363 924 V 15 w(handle\))158 1009 y Fo(Causes)20 b(either)h(one)f(or)f(none)h
(of)f(the)h(op)q(erations)g(asso)q(ciated)g(with)f(the)h(comm)o(unication)c
(handles)k(to)75 1059 y(return.)f(In)13 b(the)h(former)d(case,)j(it)f(has)g
(the)h(same)e(return)i(seman)o(tics)e(as)i(a)e(call)g(to)h
Fi(MPI)p 1449 1059 14 2 v 15 w(WAIT)p 1552 1059 V 15 w(ANY)p
Fo(.)f(In)h(the)h(latter)75 1109 y(case,)g(it)g(returns)h(a)f(v)n(alue)f(of)g
({1)h(in)f Fi(index)g Fo(and)h Fi(return)p 991 1109 V 14 w(handle)f
Fo(is)h(unde\014ned.)19 b(The)14 b(parameters)g(are:)75 1201
y Fh(IN)i(list)p 215 1201 15 2 v 15 w(of)p 269 1201 V 17 w(handles)j
Fo(list)13 b(of)g(handles)h(to)g(comm)o(unication)d(ob)r(jects.)75
1286 y Fh(IN)16 b(coun)o(t)j Fo(list)14 b(length)g(\(in)o(teger\))75
1370 y Fh(OUT)i(index)j Fo(index)14 b(of)f(handle)h(for)f(op)q(eration)h
(that)g(completed,)f(or)h(-1)f(if)g(none)h(completed)g(\(in)o(teger\).)75
1454 y Fh(OUT)i(return)p 335 1454 V 15 w(handle)i Fo(handle)d(that)g(is)g
(asso)q(ciated)h(with)e(return)i(status)g(ob)r(ject.)22 b(Set)16
b(to)f(return)h(status)g(of)179 1504 y(op)q(eration)e(that)f(completed,)g(if)
g(an)o(y;)g(unde\014ned)i(when)g Fi(index)21 b(=)g(-1)p Fo(.)158
1631 y Fh(MPI)p 257 1631 V 17 w(W)-5 b(AIT)l(ALL\(list)p 580
1631 V 15 w(of)p 634 1631 V 17 w(handles,)13 b(coun)o(t,)i(list)p
1047 1631 V 15 w(of)p 1101 1631 V 17 w(return)p 1253 1631 V
15 w(handles\))158 1717 y Fo(Blo)q(c)o(ks)k(un)o(til)e(all)g(comm)o(unicati)o
(on)e(op)q(erations)k(asso)q(ciated)g(with)f(handles)g(in)g(the)g(list)g
(complete,)g(and)75 1767 y(return)d(the)g(status)f(of)g(all)e(these)j(op)q
(erations.)k(The)14 b(parameters)g(are:)75 1859 y Fh(IN)i(list)p
215 1859 V 15 w(of)p 269 1859 V 17 w(handles)j Fo(list)13 b(of)g(handles)h
(to)g(comm)o(unication)d(ob)r(jects.)75 1943 y Fh(IN)16 b(coun)o(t)j
Fo(lists)14 b(length)g(\(in)o(teger\))75 2027 y Fh(OUT)i(list)p
266 2027 V 15 w(of)p 320 2027 V 16 w(return)p 471 2027 V 15
w(handles)j Fo(Must)h(ha)o(v)o(e)g(the)g(same)f(length)h(as)f(the)i(\014rst)f
(list.)35 b(Eac)o(h)20 b(return)h(status)179 2077 y(ob)r(ject)14
b(is)g(set)h(to)f(the)g(return)h(status)g(of)e(the)i(corresp)q(onding)g(op)q
(eration)e(in)h(the)g(\014rst)h(list.)158 2248 y Fm(Discussion:)158
2294 y Fl(The)9 b(fairness)i(requiremen)o(t)f(giv)o(en)h(in)f(Section)g(1.13)
g(implies)h(that)f(the)f(use)h(of)f Ff(WAIT)p 1376 2294 12
2 v 12 w(ANY)f Fl(cannot)i(lead)g(to)g(starv)n(ation:)75 2340
y(If)15 b(the)i(sending)g(pro)q(cess)g(has)g(issued)g(a)f(send)h(complete)g
(op)q(eration,)i(and)d(the)g(receiving)j(pro)q(cess)d(rep)q(eatedly)i(p)q
(ost)f(a)75 2385 y(receiv)o(e)i(for)f(the)h(message)g(sen)o(t,)g(then)g(that)
f(message)h(m)o(ust)f(b)q(e)h(ev)o(en)o(tually)i(receiv)o(ed.)34
b(Section)19 b(1.13)g(has)f(not)h(y)o(et)75 2431 y(b)q(een)14
b(discussed.)19 b(The)12 b(fairness)i(requiremen)o(t)h(can)e(b)q(e)g(attac)o
(k)o(ed)h(either)g(for)e(b)q(eing)j(to)q(o)e(w)o(eak)g(\(ev)o(en)o(tually)i
(is)f(not)f(go)q(o)q(d)75 2477 y(enough\),)g(or)f(to)q(o)g(strong)g(\(hard)h
(to)f(implemen)o(t\).)18 b(In)12 b(the)g(later)g(case,)g(if)g(the)g
(requiremen)o(t)i(of)d(fairness)i(is)g(dropp)q(ed)g(from)75
2522 y(MPI)g(implemen)o(tation)q(s,)j(then)d(some)h(mec)o(hanism)h(need)e(b)q
(e)h(pro)o(vided)h(to)e(the)h(user)f(to)h(ac)o(hiev)o(e)g(fairness)h(b)o(y)e
(his)h(or)g(her)75 2568 y(o)o(wn)g(devices.)22 b(One)14 b(suc)o(h)h(prop)q
(osal)h(is)f(to)f(b)q(e)g(able)i(to)e(sp)q(ecify)h(a)f(rotating)h(priorit)o
(y)h(order)f(on)g(the)f(op)q(erations)i(p)q(osted)75 2614 y(b)o(y)d(a)g
Ff(MPI)p 222 2614 V 13 w(WAITANY)p Fl(:)d(searc)o(h)k(the)f(list)h(sequen)o
(tially)n(,)h(starting)f(from)f(a)g(user)g(sp)q(eci\014ed)i(p)q(osition.)p
eop
%%Page: 21 22
bop 75 -100 a Fg(1.9.)31 b(BLOCKING)14 b(COMMUNICA)m(TION)1038
b Fo(21)75 45 y Fn(1.9)70 b(Blo)r(c)n(king)21 b(Comm)n(unication)75
140 y Fo(Blo)q(c)o(king)14 b(send)i(and)f(receiv)o(e)h(op)q(erations)g(com)o
(bine)d(all)h(comm)o(unication)e(sub)q(op)q(erations)j(in)o(to)g(one)g(call.)
21 b(The)75 190 y(op)q(eration)16 b(returns)j(only)c(when)i(the)h(comm)o
(unicati)o(on)c(completes)i(and)g(no)h(comm)o(unicatio)o(n)d(ob)r(ject)j(p)q
(ersists)75 239 y(after)d(the)h(call)e(completed.)k(Ho)o(w)o(ev)o(er,)d(the)h
(bu\013er)f(descriptor)i(ob)r(ject)e(needs)i(b)q(e)e(created)h(ahead)f(of)g
(the)g(call.)158 291 y(W)m(e)g(use)g(the)h(follo)o(wing)c(naming)h(con)o(v)o
(en)o(tion)h(for)h(suc)o(h)g(op)q(erations:)793 352 y Fa(2)793
427 y(4)844 386 y Fe(\000)842 435 y Fh(R)846 485 y(S)898 352
y Fa(3)898 427 y(5)933 377 y(\024)979 411 y Fh(SEND)976 460
y(RECV)1135 377 y Fa(\025)158 583 y Fo(The)h(\014rst)g(letter)g(\(v)o(oid,)e
Fh(R)h Fo(or)g Fh(S)p Fo(\))g(indicates)g(the)h(start)g(mo)q(de)e
(\(standard,)h(ready)m(,)g(or)g(secure\).)21 b(Only)14 b(t)o(w)o(o)75
633 y(of)f(the)i(com)o(binations)d(\(standard)i(and)g(secure\))i(are)e
(meaningful)d(for)j(receiv)o(es.)158 720 y Fh(MPI)p 257 720
15 2 v 17 w(SEND)i(\(bu\013er)p 565 720 V 15 w(handle,)e(dest,)h(tag,)h(con)o
(text\))c Fo(is)75 846 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h
(dest,)i(tag,)g(context,)f(MPI_STANDARD,)75 896 y(MPI_EPHEMERAL\))75
946 y(MPI_START\(handle\))75 996 y(MPI_WAIT\(handle,)e(null\))158
1120 y Fh(MPI)p 257 1120 V 17 w(RSEND)e(\(bu\013er)p 601 1120
V 15 w(handle,)e(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1246
y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,)
f(MPI_READY,)g(MPI_EPHEMERAL\))75 1296 y(MPI_START\(handle\))75
1346 y(MPI_WAIT\(handle,)e(null\))158 1470 y Fh(MPI)p 257 1470
V 17 w(SSEND)e(\(bu\013er)p 592 1470 V 15 w(handle,)e(dest,)h(tag,)h(con)o
(text\))11 b Fo(is)75 1597 y Fi(MPI_INIT_SEND\(han)o(dle,)18
b(buffer_handle,)h(dest,)i(tag,)g(context,)f(MPI_SECURE,)g(MPI_EPHEMERAL\))75
1647 y(MPI_START\(handle\))75 1696 y(MPI_WAIT\(handle,)e(null\))158
1821 y Fh(MPI)p 257 1821 V 17 w(RECV\(bu\013er)p 554 1821 V
16 w(handle,)c(source,)h(tag,)h(con)o(text,)f(return)p 1312
1821 V 15 w(handle\))c Fo(is)75 1947 y Fi(MPI_INIT_RECV\(han)o(dle,)18
b(buffer_handle,)h(source,)i(tag,)g(context,)f(MPI_STANDARD,)f
(MPI_EPHEMERAL\))75 1997 y(MPI_START\(handle\))75 2047 y(MPI_WAIT\(handle,r)o
(eturn)o(_hand)o(le\))158 2171 y Fh(MPI)p 257 2171 V 17 w(SRECV\(bu\013er)p
581 2171 V 16 w(handle,)14 b(source,)h(tag,)h(con)o(text,)e(return)p
1338 2171 V 15 w(handle\))e Fo(is)75 2298 y Fi(MPI_INIT_RECV\(han)o(dle,)18
b(buffer_handle,)h(source,)i(tag,)g(context,)f(MPI_SECURE,)f(MPI_EPHEMERAL\))
75 2347 y(MPI_START\(handle\))75 2397 y(MPI_WAIT\(handle,r)o(eturn)o(_hand)o
(le\))158 2565 y Fm(Implemen)o(tation)13 b(note:)158 2612 y
Fl(While)20 b(these)f(functions)g(can)g(b)q(e)f(implemen)o(ted)i(via)f(calls)
h(to)e(functions)i(that)e(implemen)o(t)i(sub)q(op)q(erations,)i(as)75
2658 y(describ)q(ed)16 b(in)g(this)f(subsection,)h(an)f(e\016cien)o(t)g
(implemen)o(tation)j(ma)o(y)d(optimize)h(a)o(w)o(a)o(y)e(these)h(m)o(ultiple)
i(calls,)f(pro)o(vided)75 2704 y(it)d(do)q(es)h(not)f(c)o(hange)h(the)f(b)q
(eha)o(vior)i(of)e(correct)g(programs.)p eop
%%Page: 22 23
bop 75 -100 a Fo(22)716 b Fg(CHAPTER)14 b(1.)27 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)158 45 y Fm(Discussion:)158
91 y Fl(W)m(e)h(use)g(a)g(di\013eren)o(t)h(function)h(name,)e(rather)g(than)g
(an)h(additional)i(mo)q(de)d(parameter,)h(in)f(order)g(to)g(sa)o(v)o(e)g(on)g
(an)75 136 y(additional)g(parameter)d(\(p)q(erformance)g(and)g(user)g(con)o
(v)o(enience\).)18 b(Lyndon)13 b(and)f(p)q(erhaps)g(other)g(prefer)g(few)o
(er)e(functions)75 182 y(and)k(an)f(additional)j(parameter.)75
401 y Fn(1.10)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)75
492 y Fo(Non)o(blo)q(c)o(king)11 b(send)i(and)f(receiv)o(e)h(op)q(erations)f
(com)o(bine)f(the)i(\014rst)g(t)o(w)o(o)e(sub)q(op)q(erations)i(\()p
Fi(INIT)f Fo(and)g Fi(START)p Fo(\))f(in)o(to)75 542 y(one)16
b(call.)21 b(They)16 b(use)g(ephemeral)f(comm)o(uni)o(cation)d(ob)r(jects,)17
b(so)e(that)g(the)h(op)q(eration)f(is)h(completed,)e(and)h(the)75
592 y(asso)q(ciated)d(resources)i(are)e(freed,)h(b)o(y)e(using)h(one)f(of)g
(the)i(functions)e Fi(MPI)p 1221 592 14 2 v 15 w(WAIT,)21 b(MPI)p
1433 592 V 15 w(STATUS,)g(MPI)p 1689 592 V 15 w(WAITANY,)75
641 y(MPI)p 144 641 V 15 w(STATUSANY)p Fo(,)12 b(or)i Fi(MPI)p
498 641 V 15 w(WAITALL)p Fo(.)e(Here,)j(to)q(o,)f(a)g(bu\013er)h(ob)r(ject)g
(has)g(to)f(b)q(e)g(created)i(ahead)e(of)g(the)h(comm)o(u-)75
691 y(nication)e(initiation)f(op)q(eration.)158 741 y(W)m(e)h(use)h(the)h
(same)d(naming)f(con)o(v)o(en)o(tion)j(as)f(for)g(blo)q(c)o(king)g(op)q
(erations:)18 b(a)13 b(pre\014x)h(of)f Fh(R)g Fo(\()p Fh(S)p
Fo(\))h(indicates)f(the)75 791 y Fi(READY)e Fo(\()p Fh(SECURE)p
Fo(\))i(mo)q(de.)k(In)12 b(addition,)f(a)h(pre\014x)h(of)f
Fh(I)g Fo(is)g(used)h(to)g(indicate)f Fj(imme)n(diate)g Fo(\(i.e.,)f(non)o
(blo)q(c)o(king\))75 841 y(execution.)158 926 y Fh(MPI)p 257
926 15 2 v 17 w(ISEND)16 b(\(handle,)e(bu\013er)p 750 926 V
15 w(handle,)g(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1039
y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,)
f(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1089 y(MPI_START\(handle\))158
1203 y Fh(MPI)p 257 1203 V 17 w(IRSEND)d(\(handle,)e(bu\013er)p
786 1203 V 15 w(handle,)g(dest,)h(tag,)h(con)o(text\))c Fo(is)75
1316 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g
(context,)f(MPI_READY,)g(MPI_EPHEMERAL\))75 1366 y(MPI_START\(handle\))158
1479 y Fh(MPI)p 257 1479 V 17 w(ISSEND)c(\(handle,)d(bu\013er)p
776 1479 V 16 w(handle,)h(dest,)h(tag,)h(con)o(text\))c Fo(is)75
1593 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g
(context,)f(MPI_SECURE,)g(MPI_EPHEMERAL\))75 1643 y(MPI_START\(handle\))158
1756 y Fh(MPI)p 257 1756 V 17 w(IRECV\(handle,)15 b(bu\013er)p
740 1756 V 15 w(handle,)f(source,)i(tag,)f(con)o(text,)g(return)p
1497 1756 V 15 w(status)p 1638 1756 V 16 w(handle\))c Fo(is)75
1869 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g
(context,)f(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1919 y(MPI_START\(handle\))158
2033 y Fh(MPI)p 257 2033 V 17 w(ISRECV\(handle,)14 b(bu\013er)p
766 2033 V 16 w(handle,)g(source,)h(tag,)h(con)o(text,)e(return)p
1523 2033 V 15 w(status)p 1664 2033 V 16 w(handle\))d Fo(is)75
2146 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g
(context,)f(MPI_SECURE,)f(MPI_EPHEMERAL\))75 2196 y(MPI_START\(handle\))75
2332 y Fn(1.11)70 b(Con)n(tiguous)23 b(Blo)r(c)n(k)f(Comm)n(unication)f(Op)r
(erations)75 2423 y Fo(The)c(most)f(frequen)o(t)i(t)o(yp)q(e)g(of)e(bu\013er)
i(used)g(is)f(a)g(bu\013er)h(with)f(one)g(con)o(tiguous)g(comp)q(onen)o(t.)27
b(W)m(e)16 b(sp)q(ecialize)75 2473 y(the)e(functions)f(in)f(the)i(t)o(w)o(o)e
(previous)i(subsections)g(to)f(this)g(case,)h(th)o(us)f(a)o(v)o(oiding)e(the)
j(need)g(for)f(the)g(creation)h(of)75 2523 y(a)h(bu\013er)i(descriptor)f(ob)r
(ject.)24 b(W)m(e)15 b(use)h(the)g(same)e(naming)g(sc)o(heme)h(used)h(in)f
(the)h(previous)g(subsections,)h(and)75 2573 y(app)q(end)d(a)g
Fh(C)g Fo(in)g(the)g(function)g(name,)e(for)i Fi(CONTIGUOUS)p
Fo(.)158 2658 y Fh(MPI)p 257 2658 V 17 w(SENDC)j(\(start,)d(coun)o(t,)h
(datat)o(yp)q(e,)f(dest,)h(tag,)h(con)o(text\))c Fo(is)p eop
%%Page: 23 24
bop 75 -100 a Fg(1.11.)26 b(CONTIGUOUS)14 b(BLOCK)h(COMMUNICA)m(TION)g(OPERA)
m(TIONS)505 b Fo(23)75 45 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g
(MPI_EPHEMERAL\))75 95 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g
(datatype\))75 145 y(MPI_SEND)f(\(buffer_handle,)f(count,)h(dest,)h(tag,)g
(context\))158 257 y Fh(MPI)p 257 257 15 2 v 17 w(RSENDC)16
b(\(handle,)e(start,)h(coun)o(t,)f(datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o
(text\))c Fo(is)75 368 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g
(MPI_EPHEMERAL\))75 418 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g
(datatype\))75 468 y(MPI_RSEND\()f(buffer_handle,)f(count,)h(dest,)h(tag,)g
(context\))158 580 y Fh(MPI)p 257 580 V 17 w(SSENDC)16 b(\(handle,)e(start,)h
(coun)o(t,)f(datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))c
Fo(is)75 691 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g
(MPI_EPHEMERAL\))75 741 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g
(datatype\))75 791 y(MPI_SSEND\()f(buffer_handle,)f(count,)h(dest,)h(tag,)g
(context\))158 903 y Fh(MPI)p 257 903 V 17 w(RECV)o(C)f(\(start,)e(coun)o(t,)
h(datat)o(yp)q(e,)f(source,)h(tag,)g(con)o(text,)g(return)p
1565 903 V 15 w(status)p 1706 903 V 16 w(handle\))75 953 y
Fo(is)75 1064 y Fi(MPI_CREATE_BUFFER)o(\()g(buffer_handle,)g(MPI_EPHEMERAL\))
75 1114 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75
1164 y(MPI_RECV\()f(buffer_handle,)f(source,)h(tag,)h(context,)f
(return_status_handl)o(e\))158 1276 y Fh(MPI)p 257 1276 V 17
w(SRECV)o(C)c(\(start,)f(coun)o(t,)f(datat)o(yp)q(e,)h(source,)g(tag,)g(con)o
(text,)g(return)p 1565 1276 V 15 w(status)p 1706 1276 V 16
w(handle\))75 1326 y Fo(is)75 1437 y Fi(MPI_CREATE_BUFFER)o(\()k
(buffer_handle,)g(MPI_EPHEMERAL\))75 1487 y(MPI_APPEND\()h(buffer_handle,)e
(start,)j(count,)g(datatype\))75 1537 y(MPI_SRECV\()f(buffer_handle,)f
(source,)h(tag,)h(context,)f(return_status_hand)o(le\))158
1649 y Fh(MPI)p 257 1649 V 17 w(ISENDC)d(\(handle,)c(start,)i(coun)o(t,)g
(datat)o(yp)q(e,)g(dest,)g(tag,)g(con)o(text\))d Fo(is)75 1761
y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75
1810 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75
1860 y(MPI_ISEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)g(context\))158
1972 y Fh(MPI)p 257 1972 V 17 w(IRSENDC)16 b(\(handle,)e(start,)h(coun)o(t,)g
(datat)o(yp)q(e,)f(dest,)h(tag,)h(con)o(text\))c Fo(is)75 2084
y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75
2134 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75
2183 y(MPI_IRSEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)f(context\))158
2295 y Fh(MPI)p 257 2295 V 17 w(ISSENDC)c(\(handle,)e(start,)h(coun)o(t,)f
(datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))c Fo(is)75 2407
y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75
2457 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75
2507 y(MPI_ISSEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)f(context\))158
2618 y Fh(MPI)p 257 2618 V 17 w(IRECV)o(C\(handle,)11 b(start,)f(coun)o(t,)h
(datat)o(yp)q(e,)f(source,)h(tag,)h(con)o(text,)e(return)p
1679 2618 V 15 w(status)p 1820 2618 V 16 w(handle\))75 2668
y Fo(is)p eop
%%Page: 24 25
bop 75 -100 a Fo(24)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fi(MPI_CREATE_BUFFER)k
(\(buffer_handle,)h(MPI_EPHEMERAL\))75 95 y(MPI_APPEND\()h(buffer_handle,)e
(start,)j(count,)g(datatype\))75 145 y(MPI_IRECV\()f(handle,)g
(buffer_handle,)f(source,)h(tag,)h(context\))158 266 y Fh(MPI)p
257 266 15 2 v 17 w(ISRECV)o(C\(handle,)10 b(start,)h(coun)o(t,)f(datat)o(yp)
q(e,)h(source,)g(tag,)g(con)o(text,)g(return)p 1706 266 V 15
w(status)p 1847 266 V 15 w(handle\))75 316 y Fo(is)75 439 y
Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75
489 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75
539 y(MPI_ISRECV\()f(handle,)g(buffer_handle,)f(source,)h(tag,)h(context\))75
682 y Fn(1.12)70 b(Prob)r(e)23 b(and)h(Cancel)75 775 y Fo(The)18
b Fi(MPI)p 233 775 14 2 v 15 w(PROBE)e Fo(op)q(eration)h(allo)o(ws)f
(incoming)f(messages)i(to)g(b)q(e)h(c)o(hec)o(k)o(ed)g(for,)g(without)e
(actually)h(receiving)75 825 y(them.)22 b(The)16 b(user)g(can)g(then)g
(decide)h(where)f(to)g(receiv)o(e)g(them,)f(based)h(on)f(the)h(information)d
(returned)k(b)o(y)e(the)75 874 y(prob)q(e)i(\(basically)m(,)f(the)h
(information)d(on)i(the)i(message)e(en)o(v)o(elop)q(e\).)27
b(An)17 b(additional)e(function,)h Fi(MPI)p 1716 874 V 16 w(GET)p
1798 874 V 15 w(LEN)75 924 y Fo(allo)o(ws)11 b(the)j(amoun)o(t)c(of)i
(storage)h(needed)i(to)d(receiv)o(e)i(the)f(message)g(to)f(b)q(e)h(computed,)
f(when)h(this)g(length)g(is)f(not)75 974 y(readily)h(computed)h(from)e(the)i
(information)d(returned)16 b(b)o(y)e Fi(MPI)p 1094 974 V 15
w(PROBE)p Fo(.)158 1025 y(The)f Fi(MPI)p 311 1025 V 15 w(CANCEL)e
Fo(op)q(eration)h(allo)o(ws)f(p)q(ending)i(comm)o(unicatio)o(ns)d(to)i(b)q(e)
h(cancelled.)19 b(This)12 b(is)g(required)h(for)75 1075 y(clean)o(up.)20
b(P)o(osting)14 b(a)h(send)g(or)g(a)f(receiv)o(e)i(ties)f(user)g(resources)i
(\(send)f(or)e(receiv)o(e)i(bu\013ers\),)g(and)e(a)h(cancel)g(ma)o(y)75
1125 y(b)q(e)g(needed)g(to)f(free)g(these)i(resources)g(gracefully)m(.)158
1211 y Fh(MPI)p 257 1211 15 2 v 17 w(PR)o(OBE\()f(source,)g(tag,)h(con)o
(text,)f(\015ag,)g(datat)o(yp)q(e,)g(return)p 1367 1211 V 15
w(status\))75 1342 y(IN)h(source)k Fo(rank)14 b(in)f(con)o(text)i(of)e
(source,)i(or)f Fi(MPI)p 897 1342 14 2 v 15 w(ANY)p 978 1342
V 15 w(SOURCE)f Fo(\(in)o(teger\).)75 1429 y Fh(IN)j(tag)21
b Fo(user)15 b(tag)e(for)h(messages)g(receiv)o(ed)h(with)f(this)f(handle,)h
(or)g Fi(MPI)p 1232 1429 V 15 w(ANY)p 1313 1429 V 15 w(TAG)f
Fo(\(in)o(teger\).)75 1517 y Fh(IN)j(con)o(text)k Fo(con)o(text)14
b(of)f(messages)h(receiv)o(ed)i(with)d(this)h(handle.)75 1604
y Fh(OUT)i(\015ag)k Fo(\(logical\))75 1692 y Fh(IN)c(t)o(yp)q(e)k
Fo(assumed)14 b(t)o(yp)q(e)g(of)f(data)h(in)f(message)h(\(status)h(v)n
(ariable\).)75 1779 y Fh(OUT)h(return)p 335 1779 15 2 v 15
w(handle)i Fo(handle)c(that)g(is)g(asso)q(ciated)g(with)g(return)h(status)g
(ob)r(ject.)158 1875 y Fi(MPI)p 227 1875 14 2 v 15 w(PROBE)h
Fo(returns)i Fi(flag)j(=)g(true)16 b Fo(if)f(there)j(is)f(a)f(message)g(that)
h(can)f(b)q(e)i(receiv)o(ed)f(and)g(that)f(matc)o(hes)75 1924
y(the)21 b(pattern)h(sp)q(eci\014ed)g(b)o(y)e(the)h(parameters)g
Fi(source)p Fo(,)g Fi(tag)p Fo(,)g(and)f Fi(context)p Fo(.)37
b(It)21 b(returns)h Fi(flag)f(=)h(false)p Fo(,)75 1974 y(otherwise.)38
b(If)19 b Fi(MPI)p 411 1974 V 16 w(PROBE)g Fo(returns)i Fi(flag)g(=)h(true)p
Fo(,)e(then)h(the)g(length,)g(source)g(and)f(tag)g(of)g(the)g(message)75
2024 y(matc)o(hed)13 b(are)i(returned)h(in)d(the)i(return)g(status)g(ob)r
(ject.)20 b(These)15 b(are)g(the)f(same)g(v)n(alues)f(that)i(w)o(ould)e(ha)o
(v)o(e)h(b)q(een)75 2074 y(returned)k(b)o(y)d(a)h(call)f(to)h
Fi(MPI)p 542 2074 V 15 w(RECV)f Fo(or)h(to)g Fi(MPI)p 832 2074
V 15 w(SRECV)f Fo(executed)j(at)e(the)g(same)f(p)q(oin)o(t)h(in)f(the)i
(program)d(\(with)75 2124 y(a)g(ca)o(v)o(eat)g(concerning)h(length;)f(see)i
(b)q(elo)o(w\).)j(These)c(v)n(alues)f(can)h(b)q(e)g(deco)q(ded)g(from)e(the)i
(return)g(status)g(ob)r(ject)75 2173 y(using)f(the)g Fi(MPI)p
324 2173 V 15 w(RETURN)p 471 2173 V 15 w(STAT)f Fo(function.)18
b(The)c(v)n(alue)f(returned)j(in)e(the)g(return)h(status)g(ob)r(ject)g(is)f
(unde\014ned)h(if)75 2223 y Fi(flag=false)p Fo(.)158 2274 y(The)j(length)g(v)
n(alue)f(returned)i(b)o(y)e(the)i(return)f(status)h(ob)r(ject)f(is)g
(\(correctly\))h(the)f(n)o(um)o(b)q(er)f(of)g(elemen)o(ts)75
2324 y(in)h(the)h(message,)f(pro)o(vided)h(that)f(all)f(elemen)o(ts)h(in)g
(the)h(message)f(are)h(of)e(the)i(t)o(yp)q(e)g(sp)q(eci\014ed)h(b)o(y)e(the)h
Fi(type)75 2374 y Fo(parameter;)13 b(otherwise)i(the)f(length)g(v)n(alue)f
(returned)j(is)e(unde\014ned.)158 2425 y(A)g(subsequen)o(t)j(receiv)o(e)e
(executed)h(with)e(the)h(same)f(con)o(text,)g(and)h(the)g(source)g(and)f(tag)
g(returned)i(b)o(y)e(the)75 2475 y(call)h(to)g Fi(MPI)p 274
2475 V 15 w(PROBE)f Fo(will)g(receiv)o(e)i(the)g(message)f(that)h(w)o(as)f
(matc)o(hed)g(b)o(y)g(the)h(prob)q(e,)g(if)e(no)h(other)h(in)o(terv)o(ening)
75 2524 y(receiv)o(e)c(o)q(ccurred)g(after)f(the)g(prob)q(e.)18
b(If)10 b(the)h(receiving)g(pro)q(cess)i(is)d(m)o(ultithreaded,)g(it)g(is)h
(the)g(user)g(resp)q(onsibilit)o(y)75 2574 y(to)j(ensure)h(that)f(the)h(last)
e(condition)g(holds.)158 2704 y Fm(Discussion:)p eop
%%Page: 25 26
bop 75 -100 a Fg(1.12.)31 b(PR)o(OBE)14 b(AND)g(CANCEL)1195
b Fo(25)158 45 y Fl(Do)12 b(w)o(e)f(w)o(an)o(t)g Ff(MPI)p 427
45 12 2 v 13 w(PROBE)e Fl(to)i(return)h(mo)q(de)g(\(standard)h(or)e
(secure\)?)18 b(If)10 b(y)o(es,)i(w)o(e)f(need)h(to)f(carry)h(mo)q(de)g
(information)75 91 y(with)j(messages)g(\(it's)f(deja)h(vu)g(all)g(o)o(v)o(er)
g(again\).)22 b(If)14 b(not,)g(then)h(it)g(is)g(up)g(to)f(the)g(user)h(to)f
(enco)q(de)h(this)h(information)g(\(in)75 136 y(the)d(tag\))g(so)g(that)h
(receiv)o(er)g(can)f(decide)h(whether)g(to)f(use)g(a)g(secure)g(or)g
(standard)i(send.)158 182 y(MPI)10 b(guaran)o(tees)h(that)e(successiv)o(e)j
(messages)e(sen)o(t)g(from)f(a)h(source)g(to)g(a)g(destination)i(within)f
(the)f(same)g(con)o(text)g(are)75 228 y(receiv)o(ed)i(in)f(the)g(order)g
(they)g(are)f(sen)o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)h(supp)q(ort,)h
(either)f(explicitly)j(or)c(implicitl)q(y)m(,)k(a)c(FIF)o(O)g(structure)75
273 y(to)i(manage)h(messages)h(b)q(et)o(w)o(een)e(eac)o(h)h(pair)g(of)f
(messages,)h(for)f(eac)o(h)h(con)o(text.)k Ff(MPI)p 1308 273
V 13 w(PROBE)11 b Fl(returns)i(information)h(on)f(the)75 319
y(\014rst)i(matc)o(hing)g(message)g(in)h(this)f(FIF)o(O;)e(this)j(will)g
(also)f(b)q(e)g(the)f(message)h(receiv)o(ed)h(b)o(y)f(the)f(\014rst)h
(subsequen)o(t)h(receiv)o(e)75 365 y(with)d(the)h(same)f(source,)g(tag)g(and)
h(con)o(text)f(as)h(the)f(message)g(matc)o(hed)h(b)o(y)f Ff(MPI)p
1261 365 V 13 w(PROBE)p Fl(.)158 410 y(Message)19 b(passing)g(in)g(MPI)e(can)
i(b)q(e)f(implemen)o(ted)i(without)e(app)q(ending)j(t)o(yp)q(e)d(information)
i(to)d(messages.)32 b(A)75 456 y(message)14 b(is)g(merely)h(a)f(string)g(of)g
(b)o(ytes)g(and)g(the)g(in)o(terpretation)i(of)d(these)h(b)o(ytes)g(in)o(to)h
(a)e(sequence)i(of)e(t)o(yp)q(ed)i(elemen)o(ts)75 502 y(is)f(done)f(using)i
(the)e(information)i(in)e(the)g(bu\013er)h(descriptors)h(at)e(eac)o(h)g(end.)
18 b(The)13 b(abilit)o(y)i(to)e(use)g(suc)o(h)h(implemen)o(tation)75
547 y(strategy)k(is)f(deemed)h(to)f(b)q(e)g(an)g(imp)q(ortan)o(t)i(goal.)30
b(In)17 b(suc)o(h)g(implemen)o(tation)q(,)j(when)d(a)g(message)h(arriv)o(es,)
h(it)e(is)h(not)75 593 y(b)q(e)g(kno)o(wn)g(ho)o(w)f(man)o(y)h(elemen)o(ts)h
(it)f(con)o(tains,)i(or)d(ev)o(en)h(ho)o(w)g(m)o(uc)o(h)g(storage)g(is)g
(needed)h(to)e(receiv)o(e)h(that)g(message)75 639 y(\(b)q(ecause)13
b(of)f(p)q(ossible)j(represen)o(tation)f(con)o(v)o(ersion)g(in)f(a)g
(heterogeneous)h(en)o(vironmen)o(t\).)k(The)12 b(prob)q(e)h(function)h
(cannot)75 684 y(use)h(a)g(bu\013er)g(descriptor;)i(this)f(defeats)f(the)g
(purp)q(ose)h(of)e(probing)j(in)e(order)g(to)g(decide)h(where)f(to)f(receiv)o
(e)i(a)e(message.)75 730 y(Therefore,)d(prob)q(e)h(cannot,)g(in)f(general,)i
(return)e(correct)g(length)h(information.)19 b(Still,)13 b(it)e(is)g(often)g
(the)g(case)g(that)g(prob)q(e)h(is)75 776 y(used)g(to)f(decide)i(ho)o(w)e(m)o
(uc)o(h)g(storage)h(to)f(allo)q(cate)i(in)f(order)g(to)f(receiv)o(e)h(a)f
(message.)17 b(Enco)q(ding)d(suc)o(h)d(information)j(in)e(the)75
821 y(message)i(tag)f(is)g(deemed)h(to)f(b)q(e)g(to)q(o)g(a)o(wkw)o(ard,)g
(and)g(it)g(is)h(deemed)g(imp)q(ortan)o(t)g(for)f Ff(MPI)p
1396 821 V 13 w(PROBE)d Fl(to)j(return)h(some)f(useful)75 867
y(size)k(information.)28 b(The)16 b(curren)o(t)h(de\014nition)i(of)c
Ff(MPI)p 884 867 V 13 w(PROBE)f Fl(is)j(a)f(compromise)i(b)q(et)o(w)o(een)e
(these)h(t)o(w)o(o)e(goals.)28 b(F)m(or)16 b(the)75 913 y(most)f(common)g
(case)g(of)f(messages)h(where)g(all)h(en)o(tries)f(ha)o(v)o(e)g(the)g(same)g
(t)o(yp)q(e)g Ff(MPI)p 1316 913 V 12 w(PROBE)e Fl(returns)i(the)g(correct)f
(length)75 958 y(information;)20 b(the)d(more)g(esoteric)g(case)g(is)g
(handled)i(b)o(y)e(the)g Ff(MPI)p 1076 958 V 13 w(GET)p 1149
958 V 12 w(LEN)f Fl(that)h(is)g(describ)q(ed)h(b)q(elo)o(w.)29
b(The)16 b(curren)o(t)75 1004 y(solution)g(sa)o(v)o(es)e(us)g(the)f(need)h
(for)g(one)g(additional)i(w)o(ord)e(p)q(er)g(message)g(that)g(w)o(ould)g
(otherwise)h(b)q(e)f(needed)g(to)g(transfer)75 1050 y(the)f(message)h(length)
g(\(in)g(elemen)o(ts\))g(with)f(the)g(message.)158 1095 y(There)g(w)o(as)g(a)
g(request)h(for)e(a)h(blo)q(c)o(king)j(prob)q(e)e(function.)158
1263 y Fh(MPI)p 257 1263 15 2 v 17 w(GET)p 376 1263 V 18 w(LEN\()h(coun)o(t,)
g(return)p 805 1263 V 15 w(status,)g(bu\013er)p 1098 1263 V
15 w(descriptor\))158 1349 y Fo(Computes)g(the)i(n)o(um)o(b)q(er)e(of)h
(elemen)o(ts)g(that)g(w)o(ere)h(to)f(b)q(e)g(receiv)o(ed,)i(if)d(the)i
(message)e(that)h(is)g(asso)q(ciated)75 1398 y(with)e(the)i(return)f(status)h
(handle)f(w)o(ould)e(b)q(e)j(receiv)o(ed)g(in)e(the)h(bu\013er)h(asso)q
(ciated)f(with)g(the)g(bu\013er)h(descriptor)75 1448 y(handle.)75
1529 y Fh(OUT)g(coun)o(t)j Fo(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(that)g
(w)o(ere)h(to)f(b)q(e)g(receiv)o(ed)h(\(in)o(teger\))75 1612
y Fh(IN)h(return)p 284 1612 V 15 w(status)k Fo(handle)13 b(to)h(return)h
(status)g(descriptor)75 1694 y Fh(IN)h(bu\013er)p 273 1694
V 16 w(descriptor)i Fo(handle)13 b(to)h(bu\013er)h(descriptor)158
1854 y Fm(Discussion:)158 1899 y Fl(In)d(order)h(to)g(supp)q(ort)g(this)h
(function,)f(an)g(additional)j(\014eld)d(is)g(needed)h(in)f(the)g(return)g
(status)f(ob)r(ject,)h(i.e.,)f(n)o(um)o(b)q(er)75 1945 y(of)17
b(b)o(ytes)g(in)h(the)f(incoming)j(message)d(matc)o(hed)h(b)o(y)f(prob)q(e,)i
(or)e(v)n(alue)h(of)f(datat)o(yp)q(e)h(parameter)g(pro)o(vided)h(b)o(y)e
(prob)q(e)75 1991 y(\(unless)d(w)o(e)f(add)h(this)g(parameter)f(to)g(the)g
Ff(MPI)p 768 1991 12 2 v 13 w(GET)p 841 1991 V 13 w(LEN)f Fl(function,)i
(whic)o(h)g(is)f(to)q(o)h(ugly)g(to)f(b)q(ear\).)158 2159 y
Fh(MPI)p 257 2159 15 2 v 17 w(CANCEL\()k(handle,)d(\015ag\))75
2283 y(IN)i(handle)j Fo(handle)14 b(to)g(comm)o(uni)o(cation)d(ob)r(ject)75
2365 y Fh(OUT)16 b(\015ag)k Fo(\(logical\))158 2455 y(A)c(call)f(to)h
Fi(MPI)p 405 2455 14 2 v 16 w(CANCEL)e Fo(cancels)j(a)f(p)q(ending)g(comm)o
(unication)d(op)q(eration)j(\(send)h(or)f(receiv)o(e\).)26
b(The)17 b(call)75 2504 y(returns)12 b Fi(flag)21 b(=)g(true)9
b Fo(if)h(the)g(cancel)h(op)q(eration)f(succeeded,)j Fi(flag)21
b(=)h(false)9 b Fo(otherwise.)17 b(It)11 b(m)o(ust)e(b)q(e)h(the)h(case)75
2554 y(that)16 b(either)g(the)g(cancel)h(op)q(eration)e(succeeds)j(or)e(that)
g(the)g(p)q(ending)g(comm)o(uni)o(cation)d(op)q(eration)i(completes)75
2604 y(\(but)k(not)g(b)q(oth\).)33 b(If)19 b(a)f(comm)o(unication)e(op)q
(eration)i(w)o(as)h(cancelled)h(successfully)m(,)g(the)g(e\013ect)g(is)f(as)g
(if)f(this)75 2654 y(op)q(eration)12 b(w)o(as)h(nev)o(er)g(executed.)20
b(If)12 b(a)g(send)h(w)o(as)g(cancelled)g(successfully)g(then)g(the)h
(message)e(sen)o(t)h(will)e(not)h(b)q(e)75 2704 y(receiv)o(ed,)j(the)f
(receiv)o(e)h(bu\013er)g(of)e(an)o(y)g(p)q(osted)i(receiv)o(e)g(for)e(that)h
(message)f(will)g(not)g(b)q(e)i(altered,)e(and)h(an)o(y)f(suc)o(h)p
eop
%%Page: 26 27
bop 75 -100 a Fo(26)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fo(receiv)o(e)i(has)g(to)f(b)q(e)h
(satis\014ed)g(b)o(y)f(another)g(send;)h(if)f(a)g(receiv)o(e)h(w)o(as)f
(cancelled)h(successfully)m(,)g(then)g(the)g(receiv)o(e)75
95 y(bu\013er)f(p)q(osted)g(will)d(not)i(b)q(e)h(altered,)f(and)f(an)o(y)h
(send)h(matc)o(hing)d(this)i(receiv)o(e)h(has)f(to)g(b)q(e)g(satis\014ed)h(b)
o(y)f(another)75 145 y(receiv)o(e.)158 276 y Fm(Implemen)o(tation)f(note:)158
324 y Fl(There)g(is)h(not)f(exp)q(ectation)i(that)e(a)g(cancel)h(op)q
(eration)h(will)g(b)q(e)e(fast.)75 559 y Fn(1.13)70 b(Correctness)75
737 y Fm(Discussion:)18 b Fl(The)13 b(material)h(in)g(this)f(section)h(has)g
(not)f(y)o(et)f(b)q(een)i(discussed)h(b)o(y)e(MPIF.)f(Some)i(or)e(all)j(of)d
(it)h(is)h(lik)o(ely)h(to)75 787 y(mo)o(v)o(e)e(to)g(Section)i
Fm(??)p Fl(.)h(It)d(is)g(incorp)q(orated)i(here)f(for)f(completeness.)75
1001 y Fk(1.13.1)55 b(Order)75 1083 y Fo(MPI)15 b(preserv)o(es)i(the)e(order)
g(of)f(messages)h(within)f(the)h(same)f(con)o(text,)h(b)q(et)o(w)o(een)h(an)o
(y)e(\014xed)h(pair)f(of)g(pro)q(cesses.)75 1132 y(In)19 b(other)g(w)o(ords,)
h(if)e(pro)q(cess)j(A)e(executes)i(t)o(w)o(o)e(successiv)o(e)i(send)f
Fi(start)d Fo(sub)q(op)q(erations)j(within)e(the)i(same)75
1182 y(con)o(text,)13 b(pro)q(cess)i(B)d(executes)j(t)o(w)o(o)d(successiv)o
(e)j(receiv)o(e)f Fi(start)e Fo(op)q(erations,)g(and)h(b)q(oth)f(receiv)o(es)
j(matc)o(h)c(either)75 1232 y(sends,)17 b(then)f(the)g(\014rst)g(receiv)o(e)h
(will)d(receiv)o(e)j(the)f(message)f(sen)o(t)h(b)o(y)f(the)h(\014rst)h(send,)
f(and)f(the)h(second)h(receiv)o(e)75 1282 y(will)12 b(receiv)o(e)i(the)g
(message)f(sen)o(t)h(b)o(y)e(the)i(second)g(send.)19 b(Th)o(us,)13
b(if)f(a)h(t)o(w)o(o)g(messages)g(from)e(the)j(same)e(source)j(can)75
1332 y(satisfy)h(a)g(p)q(ending)h(receiv)o(e,)h(the)f(\014rst)g(message)f
(sen)o(t)h(is)f(accepted;)j(if)d(a)g(message)g(can)h(satisfy)f(t)o(w)o(o)g(p)
q(ending)75 1381 y(receiv)o(es,)f(the)g(\014rst)f(receiv)o(e)h(p)q(osted)g
(is)f(satis\014ed.)158 1434 y(The)h(last)f(paragraph)f(assumes)h(that)h(the)f
(send)i Fi(start)d Fo(op)q(erations)h(are)h(ordered)g(b)o(y)f(the)h(program)d
(order)75 1484 y(at)i(pro)q(cess)h(A,)f(and)f(the)i(receiv)o(e)g
Fi(start)e Fo(op)q(erations)h(are)g(ordered)h(b)o(y)f(the)g(program)e(order)j
(at)e(pro)q(cess)j(B.)e(If)f(a)75 1534 y(pro)q(cess)i(is)f(m)o(ultithreaded)e
(and)h(the)h(op)q(erations)g(are)g(executed)h(b)o(y)f(distinct)f(threads,)h
(then)h(the)f(seman)o(tics)f(of)75 1583 y(the)e(threaded)h(system)f(ma)o(y)d
(not)j(de\014ne)h(an)e(order)h(b)q(et)o(w)o(een)h(the)g(t)o(w)o(o)e(op)q
(erations,)h(in)f(whic)o(h)h(case)g(the)h(condition)75 1633
y(is)i(v)o(oid.)75 1764 y Fk(1.13.2)55 b(Progress)19 b(and)g(F)-5
b(airness)75 1846 y Fo(W)m(e)15 b(can)i(mo)q(del)d(the)i(execution)h(of)e
(MPI)h(programs)f(as)h(an)g(in)o(teraction)f(b)q(et)o(w)o(een)j(executing)e
(pro)q(cesses)j(that)75 1896 y(execute)e(eac)o(h)e(their)h(o)o(wn)e(program,)
f(and)i(the)g Fh(comm)o(unication)f(subsystem)p Fo(.)19 b(The)c(comm)o
(unication)d(sub-)75 1945 y(system)i(ma)o(y)e(ha)o(v)o(e)h(v)n(arious)h
(constrain)o(ts)g(on)g(the)g(amoun)o(t)e(of)i(resources)i(it)d(can)h(use.)19
b(E.g.:)158 1998 y(Bounds)f(on)f(the)g(n)o(um)o(b)q(er)f(and)h(total)g(sizes)
h(of)e(activ)o(e)h(comm)o(unication)d(ob)r(jects.)28 b(Suc)o(h)18
b(b)q(ound)f(can)g(b)q(e)75 2048 y(global,)12 b(p)q(er)j(no)q(de,)e(or)h(p)q
(er)h(pair)e(of)h(comm)o(unicati)o(ng)d(no)q(des.)158 2100
y(Bounds)18 b(on)e(the)i(n)o(um)o(b)q(er)e(and)h(total)f(sizes)i(of)f
(messages)g(bu\013ered)h(in)f(the)h(system.)27 b(Suc)o(h)17
b(b)q(ound)g(can,)75 2150 y(again,)h(b)q(e)h(global,)e(p)q(er)j(no)q(de,)f
(or)g(p)q(er)g(pair)f(of)g(comm)o(unicating)d(no)q(de.)32 b(In)19
b(addition,)f(a)g(message)g(ma)o(y)e(b)q(e)75 2200 y(bu\013ered)f(at)f(the)h
(sender,)g(at)e(the)i(receiv)o(er,)g(at)f(b)q(oth,)f(or)h(p)q(erhaps)h(at)f
(another)g(place)g(altogether.)158 2253 y(Th)o(us,)f(it)g(will)e(b)q(e)j
(di\016cult)e(to)h(set)h(rules)f(on)g(resource)i(managemen)o(t)c(of)h(the)i
(comm)o(uni)o(cation)c(subsystem.)75 2302 y(Ho)o(w)o(ev)o(er,)i(it)f(is)g
(generally)g(exp)q(ected)j(that)d(implemen)o(ters)f(will)g(pro)o(vide)h
(information)e(on)i(the)h(mec)o(hanism)d(used)75 2352 y(for)j(resource)i
(allo)q(cation,)c(and)i(that)h(query)f(and)g(set)h(functions)g(will)d(allo)o
(w)h(to)h(query)h(and)f(p)q(ossibly)f(con)o(trol)h(the)75 2402
y(amoun)o(t)g(of)h(a)o(v)n(ailable)f(resources.)158 2455 y(W)m(e)18
b(pro)o(vide)g(in)g(this)g(section)i(a)e(set)h(of)f(minim)o(al)c(requiremen)o
(ts)19 b(on)f(the)h(comm)o(unication)c(subsystem.)75 2504 y(Programs)f(that)h
(execute)h(on)f(an)o(y)f(subsystem)h(that)g(ful\014ls)f(these)j(minim)o(al)11
b(requiremen)o(ts)k(are)g Fh(safe)g Fo(and)f(will)75 2554 y(p)q(ort)i(to)g
(an)o(y)f(MPI)h(implemen)o(tation.)21 b Fh(Unsafe)15 b Fo(programs)f(ma)o(y)g
(execute)k(on)d(some)g(MPI)i(implem)o(en)o(tations,)75 2604
y(dep)q(ending)e(on)f(the)h(amoun)o(t)e(of)g(a)o(v)n(ailable)f(resources)17
b(and)d(the)h(implemen)o(tation)c(used)k(for)f(the)h(MPI)g(comm)o(u-)75
2654 y(nication)f(subsystem.)20 b(Finally)13 b Fh(erroneous)f
Fo(programs)h(nev)o(er)i(execute)i(correctly)m(.)j(\(While)14
b(it)g(is)g(desirable)h(to)75 2704 y(detect)i(erroneous)f(programs,)e(it)h
(is)g(not)g(p)q(ossible)g(to)g(do)g(so)h(at)f(compile)e(time,)h(and)h(often)g
(prohibitiv)o(e)f(to)h(do)p eop
%%Page: 27 28
bop 75 -100 a Fg(1.13.)26 b(CORRECTNESS)1328 b Fo(27)75 45
y(so)16 b(a)g(run)g(time.)22 b(Th)o(us,)17 b(the)f(do)q(cumen)o(t)f(do)q(es)i
(not)f(sp)q(ecify)g(a)g(b)q(eha)o(vior)f(for)h(erroneous)h(programs,)e
(although)75 95 y(the)f(desired)i(b)q(eha)o(vior)d(is)h(to)g(return)h(a)e
(useful)h(error)h(message.\))126 186 y(1.)20 b(If)13 b(a)g(pro)q(cess)i
(executes)h(an)d Fi(INIT)g Fo(op)q(eration,)g(then)h(the)g(op)q(eration)f(ev)
o(en)o(tually)g(succeeds,)j(or)d(a)h Fj(r)n(esour)n(c)n(e)179
236 y(exc)n(eption)20 b Fo(o)q(ccurs.)38 b(The)20 b(standard)h(do)q(es)g(not)
e(sp)q(ecify)i(when)f(a)g(resource)i(exception)f(is)e(allo)o(w)o(ed)g(to)179
286 y(o)q(ccur.)33 b(It)18 b(is)h(exp)q(ected)h(that)f(an)f(op)q(erational)f
(de\014nition)i(will)d(b)q(e)k(made)d(a)o(v)n(ailable,)g(in)h(the)h(form)e
(of)179 336 y(test)e(programs)d(that)i(ha)o(v)o(e)f(to)h(execute)h(with)f(no)
f(resource)j(exceptions.)j(It)14 b(is)f(highly)g(desirable)h(to)f(ha)o(v)o(e)
179 386 y(generous)18 b(b)q(ounds)g(on)f(the)h(n)o(um)o(b)q(er)f(of)f
(concurren)o(tly)j(activ)o(e)e(comm)o(unication)d(ob)r(jects)k(eac)o(h)g(pro)
q(cess)179 435 y(ma)o(y)12 b(ha)o(v)o(e,)h(so)h(that,)f(in)h(practice,)g
Fi(INIT)f Fo(op)q(erations)h(will)f(alw)o(a)o(ys)g(b)q(e)h(guaran)o(teed)h
(to)e(succeed.)126 519 y(2.)20 b(Eac)o(h)d(pro)q(cess)h(can)f(initiate)f(a)g
(comm)o(unicatio)o(n)e(op)q(eration)j(for)f(eac)o(h)h(activ)o(e)f(comm)o
(unication)e(ob)r(ject.)179 568 y(I.e.)k(correct)d Fi(START)e
Fo(op)q(erations)h(alw)o(a)o(ys)f(succeed)j(\(ev)o(en)o(tually\).)126
652 y(3.)k(A)f(send)i(op)q(eration)e(is)h Fh(enabled)d Fo(if)h(the)j(sending)
e(pro)q(cess)j(has)d(issued)i(a)e Fi(COMPLETE)f Fo(op)q(eration)h(and)179
701 y(the)d(receiving)h(pro)q(cess)g(has)f(issued)h(a)f Fi(START)f
Fo(op)q(eration)g(for)h(a)g(matc)o(hing)e(receiv)o(e.)25 b(Symmetrically)l(,)
13 b(a)179 751 y(receiv)o(e)h(op)q(eration)f(is)g Fh(enabled)d
Fo(if)i(the)i(receiving)f(pro)q(cess)i(has)e(issued)h(a)f Fi(COMPLETE)e
Fo(op)q(eration)i(and)f(the)179 801 y(sending)h(pro)q(cess)i(has)e(issued)h
(a)f Fi(START)f Fo(op)q(eration)h(for)g(a)f(matc)o(hing)g(send.)18
b(An)c(enabled)f(op)q(eration)g(ma)o(y)179 851 y(b)q(ecome)g
Fh(disabled)e Fo(either)k(b)q(ecause)g(it)e(completes)h(successfully)h(or,)e
(in)g(the)h(case)h(of)e(a)h(receiv)o(e,)g(b)q(ecause)179 901
y(the)g(matc)o(hing)d(message)i(is)g(successfully)i(receiv)o(ed)f(b)o(y)f
(another)h(receiv)o(e)h(op)q(eration)e(or,)g(in)f(the)i(case)h(of)d(a)179
951 y(send,)i(b)q(ecause)i(the)e(matc)o(hing)e(receiv)o(e)j(successfully)g
(receiv)o(es)h(another)e(message.)179 1017 y Fh(An)i(enabled)f(op)q(eration)f
(either)h(completes)f(successfully)g(or)j(b)q(ecomes)f(p)q(ermanen)o(tl)o(y)e
(dis-)179 1067 y(abled.)179 1133 y Fo(I.e.,)g(an)h(enabled)h(op)q(eration)f
(either)h(ev)o(en)o(tually)e(succeeds,)k(or)d(b)q(ecomes)g(disabled)g
(\(progress\);)i(and)e(an)179 1183 y(op)q(eration)f(cannot)g(b)q(e)g(enabled)
g(in\014nitely)f(man)o(y)f(times)h(without)h(ev)o(er)h(succeeding)g
(\(fairness\).)126 1266 y(4.)20 b(A)14 b Fi(FREE)f Fo(op)q(eration)h(alw)o(a)
o(ys)e(succeeds)17 b(\(ev)o(en)o(tually\).)158 1358 y(The)e(four)g
(conditions)f(guaran)o(tee)h(progress)h(in)e(the)i(comm)o(unicati)o(on)c
(subsystem.)21 b(The)15 b(third)g(condition)75 1408 y(guaran)o(tee)f(\(w)o
(eak\))g(fairness)h(among)d(comp)q(eting)g(comm)o(unication)f(requests.)158
1458 y(Examples)i(\(in)o(v)o(olving)f(t)o(w)o(o)h(pro)q(cessors)j(with)e
(ranks)g(0)f(and)h(1\))158 1507 y(The)g(follo)o(wing)e(program)g(is)i(safe,)f
(and)h(should)g(alw)o(a)o(ys)f(succeed.)75 1640 y Fi(CALL)21
b(MPI_RANK\(rank,)e(context\))75 1690 y(IF)i(\(rank.EQ.0\))f(THEN)140
1740 y(CALL)h(MPI_SENDC\(sendbuf,)d(count,)j(1,)g(tag,)g(context\))140
1790 y(CALL)g(MPI_RECVC\(recvbuf,)d(count,)j(1,)g(tag,)g(context\))75
1840 y(ELSE)86 b(!)22 b(rank.EQ.1)140 1889 y(CALL)f(MPI_RECVC\(recvbuf,)d
(count,)j(0,)g(tag,)g(context\))140 1939 y(CALL)g(MPI_SENDC\(sendbuf,)d
(count,)j(0,)g(tag,)g(context\))75 1989 y(END)g(IF)158 2072
y Fo(The)14 b(follo)o(wing)e(program)g(is)i(erroneous,)g(and)g(should)g(alw)o
(a)o(ys)f(fail.)75 2214 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75
2263 y(IF)i(\(rank.EQ.0\))f(THEN)140 2313 y(CALL)h(MPI_RECVC\(recvbuf,)d
(count,)j(1,)g(tag,)g(context\))140 2363 y(CALL)g(MPI_SENDC\(sendbuf,)d
(count,)j(1,)g(tag,)g(context\))75 2413 y(ELSE)86 b(!)22 b(rank.EQ.1)140
2463 y(CALL)f(MPI_RECVC\(recvbuf,)d(count,)j(0,)g(tag,)g(context\))140
2512 y(CALL)g(MPI_SENDC\(sendbuf,)d(count,)j(0,)g(tag,)g(context\))75
2562 y(END)g(IF)158 2654 y Fo(The)13 b(receiv)o(e)h(op)q(eration)f(of)f(the)i
(\014rst)f(pro)q(cess)i(m)o(ust)d(complete)g(b)q(efore)i(its)e(send,)i(and)f
(can)g(complete)f(only)75 2704 y(if)g(the)h(matc)o(hing)d(send)k(of)e(the)h
(second)g(pro)q(cessor)i(is)d(executed;)i(the)f(receiv)o(e)h(op)q(eration)f
(of)e(the)j(second)f(pro)q(cess)p eop
%%Page: 28 29
bop 75 -100 a Fo(28)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15
b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fo(m)o(ust)i(complete)h(b)q(efore)h
(its)g(send)g(and)f(can)h(complete)e(only)h(if)f(the)i(matc)o(hing)e(send)i
(of)f(the)h(\014rst)g(pro)q(cess)h(is)75 95 y(executed.)h(This)14
b(program)e(will)g(deadlo)q(c)o(k.)158 145 y(The)i(follo)o(wing)e(program)g
(is)i(unsafe,)f(and)h(ma)o(y)e(succeed)k(or)e(fail,)e(dep)q(ending)i(on)g
(implemen)o(tatio)o(n.)75 286 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75
336 y(IF)i(\(rank.EQ.0\))f(THEN)140 385 y(CALL)h(MPI_SENDC\(sendbuf,)d
(count,)j(1,)g(tag,)g(context\))140 435 y(CALL)g(MPI_RECVC\(recvbuf,)d
(count,)j(1,)g(tag,)g(context\))75 485 y(ELSE)86 b(!)22 b(rank.EQ.1)140
535 y(CALL)f(MPI_SENDC\(sendbuf,)d(count,)j(0,)g(tag,)g(context\))140
585 y(CALL)g(MPI_RECVC\(recvbuf,)d(count,)j(0,)g(tag,)g(context\))75
635 y(END)g(IF)158 776 y Fo(The)15 b(message)g(sen)o(t)h(b)o(y)f(eac)o(h)g
(pro)q(cess)i(has)e(to)g(b)q(e)h(copied)f(out)g(b)q(efore)h(the)g(send)g(op)q
(eration)f(returns)h(and)75 826 y(the)h(receiv)o(e)g(op)q(eration)e(starts.)
26 b(F)m(or)15 b(the)i(program)d(to)i(complete,)f(it)g(is)h(necessary)i(that)
e(at)g(least)g(one)g(of)f(the)75 875 y(t)o(w)o(o)d(messages)g(sen)o(t)h(is)f
(bu\013ered)i(out)e(of)f(either)i(pro)q(cesses')h(address)g(space.)k(Th)o
(us,)13 b(this)f(program)e(can)j(succeed)75 925 y(only)g(if)g(the)i(comm)o
(unicati)o(on)c(system)j(has)g(su\016cien)o(t)g(bu\013er)h(space)g(to)f
(bu\013er)h Fi(count)d Fo(w)o(ords)i(of)g(data.)158 975 y(If)c(additional)e
(requiremen)o(ts)i(will)f(b)q(ecome)h(part)g(of)f(the)i(standard)f(\(e.g.,)g
(b)q(ounds)h(on)e(the)i(minim)o(al)c(n)o(um)o(b)q(er)75 1025
y(of)13 b(concurren)o(tly)i(activ)o(e)f(handles)g(that)g(need)h(b)q(e)g(supp)
q(orted,)f(then)h(further)g(programs)d(b)q(ecome)i(safe.)75
1162 y Fn(1.14)70 b(Missing)75 1253 y Fh(send-receiv)o(e)39
b Fo(a)13 b(function)h(of)f(the)i(form)d(\(for)i(blo)q(c)o(king)e(comm)o
(unication\))158 1303 y Fi(MPI)p 227 1303 14 2 v 15 w(SEND)p
330 1303 V 15 w(RECV\()21 b(send)p 564 1303 V 15 w(buffer,)f(destination,)f
(recv)p 1124 1303 V 15 w(buffer,)h(source,)h(tag,)g(context\))158
1353 y Fo(It)13 b(ma)o(y)e(b)q(e)j(sup)q(er\015uous)h(b)q(ecause)f(the)g
(collectiv)o(e)f(comm)o(unicatio)o(n)e(shift)h(has)h(similar)e(functionalit)o
(y)m(.)16 b(Y)m(et)75 1402 y(it)10 b(allo)o(ws)g(to)g(generate)i(more)e
(easily)g(comm)o(unication)d(patterns)12 b(that)f(w)o(ould)f(lead)g(to)h
(deadlo)q(c)o(k)f(if)g(implemen)o(ted)75 1452 y(naiv)o(ely)m(,)i(with)h(blo)q
(c)o(king)g(comm)o(unication.)158 1502 y(An)20 b(imp)q(ortan)o(t)f(v)n(arian)
o(t)g(is)h(the)h(case)g(where)h Fi(send)p 1020 1502 V 14 w(buffer)f(=)h(recv)
p 1319 1502 V 14 w(buffer)d Fo(\(this)i(can)f(either)h(b)q(e)g(a)75
1552 y(sub)q(case)c(of)f(the)g(general)g(send)p 578 1552 13
2 v 16 w(recv)h(function,)f(or)f(a)h(separate)h(function,)f(if)f(send)h
(bu\013er)h(and)f(receiv)o(e)h(bu\013er)75 1602 y(are)g(required)g(to)f(b)q
(e)h(disjoin)o(t)e(in)g(the)i(general)f(function\).)25 b(This)16
b(in)o(tro)q(duces)i(new)e(functionalit)o(y)f(that)h(is)g(not)75
1651 y(a)o(v)n(ailable,)11 b(otherwise,)k(and)e(whic)o(h)h(seems)g(to)g(b)q
(e)g(often)g(required.)158 1701 y(An)g Fi(exchange)e Fo(is)i(the)g(sp)q
(ecial)h(case)f(where)i(source)f(=)f(destination.)p eop
%%Trailer
end
userdict /end-hook known{end-hook}if
%%EOF

>From [email-masked] Wed Jun  9 15:10:59 1993
Date: Wed, 9 Jun 93 08:10:32 -0500
From: [email-masked] (Dennis Weeks)
To: [email-masked]
Subject: May 13 mpi mail #1 of 1
Status: R

(received from walker, as I recall; I believe he posted it to mpi but
I didn't receive it, so eventually he mailed it directly to me - DW)

                Minutes of the Message Passing Interface Forum
                                Dallas, Texas
                           March 30 - April 2, 1993

The MPI Forum met March 30 - April 2, 1993, at the Bristol Suites Hotel in
North Dallas.

This was the fifth meeting of the MPIF and the third of the now regular
meetings in Dallas.  There were both general meetings of the committee as a
whole and meetings of several of the subcommittees.

For the first time a number of formal votes were taken at this meeting.
All of these are recorded in these minutes (and can be found by searching
for VOTE) and have also been published (to the mpi-core mailing list) in a
summary of all the formal votes and all of the straw votes for the
committee as a whole.

The notes for these minutes were taken by Bob Knighten
([email-masked]) and Rusty Lusk ([email-masked]).

These minutes are quite long.  If you want to see the important topics you
can search for --- and this will quickly the lead to each topic (and a few
other things.)

Attendees:
- ---------

Joe Baron               IBM Austin               [email-masked]
Eric Barszcz		NASA Ames                [email-masked]
Harry Scott Berryman    Yale Univ.               [email-masked]
Rob Bjornson		SCA                      [email-masked]
Lyndon Clarke	        EPCC, U. Edinburgh       [email-masked]
James Cownie	        Meiko			 [email-masked]
Jack Dongarra	        UT/ORNL			 [email-masked]
Anne C. Elster          Cornell U.               [email-masked]
Sam Fineberg		NASA Ames                [email-masked]
Jon Flower	        ParaSoft		 [email-masked]
Ian Glendinning         U. of Southampton        [email-masked]
Adam Greenberg          TMC			 [email-masked]
Bill Gropp              ANL                      [email-masked]
Leslie Hart	        NOAA/FSL		 [email-masked]
Tom Haupt	        Syracuse U.		 [email-masked]
Rolf Hempel	        GMD			 [email-masked]
Tom Henderson	        NOAA/FSL		 [email-masked]
C. T. Howard Ho         IBM Almaden              [email-masked]
Steven Huss-Lederman    SRC			 [email-masked]
Rusty Lusk              ANL                      [email-masked]
John Kapenga            Western Michigan U.      [email-masked]
Bob Knighten	        Intel SSD		 [email-masked]
Rik Littlefield         PNL			 [email-masked]
Peter Madams            nCube			 [email-masked]
Arthur B. Maccabe       U. of New Mexico         [email-masked]
Oliver McBryan          U. Colorado              [email-masked]
Dan Nessett	        LLNL			 [email-masked]
Steve Otto	        Oregon Graduate Instiute [email-masked]
Peter Pacheco           U. of San Francisco      [email-masked]
Paul Pierce	        Intel			 [email-masked]
Sanjay Ranka            Syracuse U.              [email-masked]
Arch Robison            Shell Development        [email-masked]
Mark Sears              Sandia			 [email-masked]
Anthony Skjellum        Mississippi State U.	 [email-masked]
Marc Snir               IBM, T.J. Watson         [email-masked]
Alan Sussman            U. of Maryland           [email-masked]
David Walker            ORNL                     [email-masked]
Dennis Weeks	        Convex			 [email-masked]
Stephen Wheat           Sandia NL                [email-masked]
Wednesday, March 30
- ---------  --------

- -------------------------------------------------------------------------------
			       General Meeting
- -------------------------------------------------------------------------------

Jack Dongarra called the meeting to order at 1:30.  The first topic for
discussion was the agenda.  David Walker had mailed out the following

        Provisional Agenda for MPI Meeting, March 31-April 2, 1993

Wednesday
1:30-6:00   Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone)
            (Snir)
6:00-7:30   Unofficial dinner break 
7:30-10:30  Break up for subcommittee meetings
        
Thursday
9:00-12:00  Discussion of Snir & Geist collection communication 
	    proposal (everyone)
	    (Otto?)
12:00-1:30  Lunch (provided)
1:30-3:00   Full group meeting for presentation of alternate approaches to
	    groups and contexts, dynamic vs. static process models, and other
	    issues
	    (Volunteer?)
3:00-6:00   Full group meeting for presentation of process topology
	    subcommittee ideas and proposals.
	    (Hempel)
6:00-8:00   Dinner (attendees pay, but hotel provides transport to area
                   restaurant)
8:00-10:00  Continued informal subcommittee meetings if necessary
        
Friday 
9:00-11:00  Full group meeting with the intent of taking binding votes
	    on point-to-point and collective communication proposals, or
	    sending proposals back to subcommittees for revision.
	    (Snir?)
11:00-12:00 Full group meeting for defining timetable for producing MPI (or
	    subset) by deadline in July.
	    (Dongarra)

Following the discussion on the mpi-core mailing list the question was moving
the discussion of the timetable for producding MPI to the beginning of the
meeting.  After a brief discussion it was decided to proceed first with
reports from the Communication Context and Point-to-Point Communication subcommittees in
order to have a basis for discussing the schedule.  The schedule was discussed
on Thursday afternoon, following the completion of the Point-to-Point
subcommittee report.

The Context subcommittee was alloted two hours, from 1:30 to 3:00, with the
Point-to-Point Communications subcommittee scheduled from 3:00 to 6:00 and on
Thursday morning.

- -------------------------------------------------------------------------------
              Report From the Communication Context Subcommittee
- -------------------------------------------------------------------------------

Tony Skjellum presided.  There was a large volume of activity on the
mpi-context mailing list before this meeting and so there were five
proposals available for consideration, labeled:

I     (Marc Snir)
III   (Tony Skjellum)
VII   (Lyndon Clarke/Rik Littlefield)
VIII  (Mark Sears)
X     (Tony Skjellum/Lyndon Clarke)

25 minutes was alloted for presentation of each of these in the order I
(Marc Snir), VII+X (Lyndon Clarke), VIII (Mark Sears) , III (Tony
Skjellum), X followed by general discussion.  Tonight there will be a
subcommittee meeting to produce a single proposal.

Proposal I (Marc Snir)
- ---------- -----------

{Marc used overhead projector slides and these notes are largely a
transcription of those slides.}

Group=Context Proposal

Goals:
  + Keep it simple (and keep MPI small)
  + Keep it efficient

Minimal needs:
  + Protection mechanism
  + Local name space

Group = Context = Ordered set of processes.
- --
Method for protecting communication between e.g. libraries.
- --
 + All point-to-point communication is WITHIN a group and uses 
   (group,rank) address.

 + All collective communications by a group (which is a context)

OPERATIONS:
  + Group copy
  + Group partition
  + Group creation by list
  + Group deletion
  + ALL group prexists

Group handle has only local (i.e. within group) use and meaning.
- --
There is no reason to pass the handle of a group outside the group - it
has no use.
- --
1.  Impact on "current practice":

  Need additional argument ALL in all p-p calls.

2.  Overhead for p-p
  send - no impact when ALL use. One lookup for other groups.
  receive - context id match

    Overhead at creation
  Loosely synch collective communication within group affected
    
    Storage
  Member table (good protection)
- --

3.  Compatibility with dynamic process creation and deletion

  Process creation/deletion requires same for group
{What is ALL group after process creation or deletion?}

4.  Interaction with topology
  Group has no topology information (but it can be used as a peg for such
information.)

5. Inter group communication (e.g. client-server models)

       +-------------------------------------+

          +---+ ---> (---)
          | * |    
          | * | 
          | * | 
          +---+
       +--------------------------------------+

Do the communication within an encompassing group.
- -
 + Encompassing group needed for protection
 + May not be convenient for naming (e.g. send(server[5], ...))
 + Inconvenience does not warrant change in p-p layer
 + Can be handled by creating and explicityl passing arrays of ranks:

MPI_LIST_RANKS(list, subgroup, group)
  returns the list of the rank of each subgroup member within group
- --
Discussion:

How have available both subgroup and group?  There must be an encompassing
group which has full knowledge, e.g. server that is member of both the
group and the subgroup.

This proposal is orthogonal to question of attaching additional
information (e.g. topology information, caching, etc.)

Can't deal with situation of contacting an independent pre-existing
server.  Marc's approach is that dynamically adding processes requires
dynamically creating a group containing all of the processes.

Opacity vs. accesibility to mechanism.

 =============================================================================
Lyndon   VII and X

VII:

Context is a higher level mechanism than a group.  It is basically a
unique identifier together with a reference to a group.  This means that
as a group changes, all contents that reference that group change as well.  

Same ability to hang on facilities (e.g. topology, caching) as others.

Relation to p-p:  Three forms: "closed form", "null form", "open form"
Open form is to allow communication between different groups.  Experience
is that creating encompassing groups is difficult. Disagree with Snir's claim.

Addressing is via (context, rank).

Need a "context allocation" mechanism - this implies global communication.

Relation to c-c:  Works very cleanly for all-all using closed form.
Two group communcations - for MPI-2
Discussion:
How to establish communication between groups?  Can send context.

What is opaque? Lyndon - not important.  Startup/bootstrap - everyone
starts in the ALL group.  Can use common ancestor or name registry.  Power
relative to I? Lyndon claims that this is more convenient once
communication is setup.

Basic idea is to be able to communicate via (my_context, remote_context, rank)

ORTHOGONAL ISSUES - caching, tag selection, transfer of ????

X: Attempt to synthesize III & VII

A CONTEX is a space of tags.  A GROUP is a set of process references.
[What does this mean?]

Idea is to give method for combining groups and context for purpose of
communication.  

COMMUNICATORS (see pp. 3-4)
  Silly names, but a serious proposal.

  Floopy - arbitrary communication between processes allowing wild card
on tag.

  Bongo - basically like Marc's proposal - communication within a group
using rank naming  ("closed")

  Bingo - communication between groups.

Question:  Why do we want ANY of these proposals?
Performance, build large scale software safely

We need examples for all of these proposals!
Collective -> group; express collective in terms of p-p implies need to
discriminate message -> context.

But there are reasons to have groups and contexts that have nothing to do
with collective communication.  Argument that in Marc's proposal the need
for a context means have to have an additional group - but is this a
problem? 

Lyndon argues that there are good reasons to separate group and context.
Static vs. dynamic groups.  Ability to move context.

Proposal VIII - Mark Sears
{{{
MPI-1

char * MPI_ALLOCATE Group and Context Proposal
   				Number();
}}}
- --
Context and Groups are orthogonal
  + Orthogonal purpose
  + Orthogonal functionality
  + Orthongonal implementation
- --
                               Contexts

Purpose - promote software modularity by allowing construction of
independent tag spaces

Definition: A CONTEXT is an integer-valued extension to the tag
component of the meggage envelope, and must match exactly between
sender and reciver
- --
Model:
 + Contexts are global.
 + No concept of process belonging to a context.
 + Contexts are scarce resources (16).
 + Context allocation is a rare event.
 + MPI p-p requries no reference to groups.
- --
Context allocation/deallocation
ALLOCATION:

 int mpi_getcontext()

   + called synchronously by all (EVERY SINGLE ONE) processes
   + signals to mpi that use of context is now allowed

DEALLOCATON

  void MPI_free_context(value)

MPI_DEFAULT CONTEXT
  + Preallocated; can't be freed.
  + Solves initialization
  + Free-for-all

But believes that allocation/deallocation not truly needed - could have
entirely static system.

  + Contexts are global
  + No concept of process belonging to a context
  + Context are scarce resources (e.g. 16)
  + Context allocation is a rare event
  + MPI p-p requires no reference to groups
- --
                                GROUPS

Purpose - provide tools for organizing subsets of processes in a
parallel task (i.e. MPI program.)

Definition:  A groups is a 1-1 mapping from (0..n-1) to another set of integer.
A group is a collection of processes only in so far as the elements are process
addresses.

Groups have no associated have no associated context or tags, default or otherwise.
- --
Group Implementation
 + local to each process based on information needed to construct the mapping
 + Group type is local and opaque
 + Groups can be sent in message only by sending the information needed
   to construct the group.
 + Groups are objects in the OOP sense
- --
Usage:

MPI_SEND(n,buf,process,tag,context)

MPI_BROADCAST(n,buf,group,tag,context)

Group identifier is a local opaque type, thought of as a pointer to
one of many possible group structures.

  MPI_SEND(..., element(group,rank),tag,context)
                            GROUP FUNCTIONS

int order (group)
int range(group)
int element(group, int rank)
int iselement(group, int element)
int rank(group, int element)

                                CLASSES
identity, permutation, linear, list, bilinear, composition, cartesian

                             CONSTRUCTORS
group makelineargroup(order,start,delta)
- --
Two kinds of 3rd party code
1. Code that inherits context and tag space from caller.  Example:
MPI collective communication

2. Code which allocates and manges own context and tag space.

MPI should allow both of these.
- --
                               Topology

Global topology - mapping of processes to processors.
provide inquiry function returning a string describing this mapping:

  char * MPI_global_topology()

Examples of output:
  "N 564"	-  random network of 564 processes
  "H 5"         - 5 dimensional hypercube
  "R 2 16 13"   - 2D mesh, 16x13

Local topology - implicity within group; no additional functions needed.
- --
ADVANTAGES
 + Ease of implementation
 + Close to hardware
 + Good use of resources
 + Flexibility in implementation of higher level concepts
 + MPI p-p requires no reference to groups
 + MPI c-c can be layered on top of MPI p-p
- --
Discussion - Serious problem with global communication.  This destroys
the software modularity.

How to do global operations using groups?  Responsibility on code to
insure there is disambiguation, synchronization, etc.

Tony - III

 + Tag/context partionting message space for "safe" software

 + Groups encapsulate scope of operations for
    1) notation
    2) optimization
    3) performance
                              ...........
     ...............     /     . GROUPS .
     .            .    /        .........
     . CONTEXTS    . /  lower,higher?
      .............

Relation between groups and contexts?  Groups can be orthogonal except
for group creation.
- --
Forms of communiations

 + collective communication
   ON groups
   (compatible with I)

 + p-p
   A] (group,rank,tag) - analagous to I
   B] (context,pid,tag)

Models
1) Contexts created/destroyed
2) contexts can be published
    dynamic server implied or shared address area

Contexts & groups interelate
when creating new groups not necessarily from LCA
                                              ^^^{{{WHAT DOES THIS MEAN?}}}
- --

contexts & groups interrelate when creating new groups, not necessarily
from LCA (what does this mean?)

Dispute/reply regarding optimization.  Argument that group can be used to
provide to information about special situations (e.g. shared memory) that
can be used for optimization.

Dynamic groups more feasible using III or VII.

Can contexts be sent in III?  Yes.

VII more complex than III because it offers more layers.

Tony believes that dynamic groups are essential for the heterogeneous
case and so believes that I is inappropriate.

John K. notes that other proposals can be built on top of VIII.

Proposal to defer straw vote until tomorrow to give people time to ponder.

Is global synchronization an essential part of VIII?  Sears - no, there
are various possibilities.  
Why is it important that contexts are global?  Because a context is not
associated with a group of processes.  This looses much of the safety.
It also looses the local addressing within a group.  Sears argues that
this complicates p-p, but Snir says that something like his proposal has
been implemented and is not complicated or expensive.

What level of protection is needed/desirable?  
Picture of using context for safety - at startup send a context to each library
used which it then uses for internal safety.  Problem - libraries using libraries
using libraries, etc.  Importance of receiving on wild cards and relation.

Host/Node model?  Must support this and all do.  (What about loading
program - not part of THIS discussion.)

Subcommittee will meet tonight and present a more unified proposal tomorrow morning.

Rik asked for example showing how to implement safe barrier using p-p, group, context.

Adam - How can we evaluate these proposals if we have not agreed on what
we want from the context concept?
- -----
break 4-4:20 pm
- -----
- -------------------------------------------------------------------------------
           Report From the Point-to-Point Communication Subcommittee
- -------------------------------------------------------------------------------

First reading of p-p proposal - Marc Snir presiding

Presentation with some minimal assumptions about use of context.
No language binding included.

Use of handles and opaque objects.
[Ignore text comments on implementation - issues for language experts.]

Discussion of ephemeral/persistant.  Greenberg asked about "error to
free a handle with pending operation" - change to "handle becomes
ephemeral" or some such.

How deal with lists - included length, separate length, EOL marker.

States  [Implementation again - not part of current proposal]

If have only one send/receive, what is type of buffer and what to do
about this?  Marc's proposal is to accept breaking F77 in this case.

Skipping 2.4 (Contexts) for now (including error handling.)

2.5.1: What about C/Fortran compatibility for messages?  Also skipped
for now.

2.6: Greenberg wants at least escape hatch to allow functions
(MPI_ADD_?) that add, e.g., other F90 objects.

Discussion of len in MPI_ADD_BLOCK as "Number of elements" rather than
as bytes.  There are language and portability issues.

Marc mentioned issues in middle of p. 14 (negative displacements, ...)
and notes that these must be settled.

Note that delete/commit functions discussed last time are not in this
proposal.

2.6.1 (Data Conversion)  It just happens - this needs to be stated clearly.

- -- back to 2.2 - vote on each section

[[NOTE:  A proposal, in the form of a proposed chapter is offered.
Votes are on amendments and on accepting particular parts of chapter.]

How are handles allocated?  User or system.  There are efficiency
advantages to user allocation, e.g. on stack.

PRP argues for a create for each data type rather than a generic
MPI_CREATE.  At the moment there is an admixture in the proposal.

======
Discussion of voting rules.  Organizations voting: 24
=====

VOTE: Separate mpi_create for each type
- ----  Yes: 20 No: 0 Abstain: 4

Possible meanings of free:

                                 free
1.1 can always be done				1.2 can be done only if
user will not use handle                        no pending operation
after free
2 if no pending operation, deallocate otherwise free when done

1.2 = current proposal; 2 is Greenberg proposal

Greenberg explantion - common thing done in their system is to have handlers with free
only taking effect after handler completes.  PRP - Has primitive that does essentially
2 (which is not "mark ephemeral") It does not imply cancel.  Fire and forget.
Cownie: Two arguments - Paul's fire and forget; Adam's handler for messages.  
Snir: Is buffer available after free?  PRP/Adam: No.
Why not just have another kind of free?  Multiplicity of functions.

VOTE: mpi_free is valid even if handle is in use, but effect is to
- ----  free the object when the operation completes. 
      Yes: 7 No: 8 Abstain: 

List of handles - change name to array of handles

VOTE: list_length explicit (rather than included in array or EOL)?
- ----  Yes: 22 No: 0

Cownie wants a method to provide cheap allocation, e.g. on user's
stack rather than in system heap.  Need a concrete proposal.

RLK wants explict statement of what is erroneous, checked, etc.

VOTE: Accept 2.2
- ----  Yes: 23 No: 1

2.3 & 2.4 skipped -- they will be considered elsewhere.

2.5  Cownie - proposes that there be both CHAR and BYTE data types
rather than just the basic data types of the host language.

Another proposal is to have an MPI_STRING data type.  But what length? Null terminated?

VOTE: BYTE data type.
- ----  Yes: 24 No: 0

Pending action of Context Subcommitte is the content of the envelope.

VOTE: Accept 2.5 (minus 2.5.2)
- ----  Yes: unanimous

Proposals for units of bytes and for consistency.

1. Units are elements
2. bytes in C and elements in F77  (No one favors this.  Against: 14)
3. bytes everywhere (and provide some way of getting size of basic types)
    a) index stays as is, but bytes elsewhere     Pro: 9  Con: 13 (VOTE 1)
    b) truly bytes everywhere, including indices  Pro: 10 Con: 8  (VOTE 2)

Greenberg will bring forward additional proposal for 2.6 at next meeting.

VOTE:  In vector, stride may be negative?
- ----   Yes: 10 No: 2

VOTE:  Allow repetition?
- ----   Yes: 13 No: 1

VOTE:  Can multiple components overlap?
- ----   vote not taken

PRP proposes that in vector len is a count of the number of blocks.

VOTE: total length is an integer multiple of the block size
- ----  Yes: 4 No: 10
Tony proposes adding a COMMIT operation.
Note: Current proposal does not clearly state it, but a handle cannot
be modified after it is in use.  There is nothing that explicitly
specifies that the modifications are complete.

VOTE: Commit?
- ----  Yes: 11 No: 6

VOTE: Accept 2.6
- ----  Yes: 12 No: 4

- -----
Break for dinner - 6:15 p.m.
- -----

Subcommittees tonight:  Here: Vendor caucus
                        Rm 1: Process topology
                        Rm 2: Collective Communications Subcommittee
                        Rm 3: Context Subcommittee
   			BAR:  formal, language binding, profile

meet at 8 p.m.

 -----------------------------------------------------------------------------

Revised agenda

Thursday 
  9 p-p (cont)
  12 lunch
   1:30 (registration)
  1:30 collective communication
  3    process topology
  6-8 dinner

Future:  May 12-14; June 23-25; August ?

 =============================================================================

                      Thursday, April 1, 1993
                              9:10 a.m. - 

- -------------------------------------------------------------------------------
     Report From the Point-to-Point Communication Subcommittee (continued)
- -------------------------------------------------------------------------------

Point-to-Point Communication - First Reading (continued):
- -------------- -------------   ----- -------

Marc Snir presiding

2.7 (Receive Criteria) & 2.8 (Communication Mode)
- -------------------------------------------------

Suggestion that we need more than one DONTCARE, a SOURCE_DONTCARE and
a TAG_DONTCARE.

Lyndon Clarke proposed a "secure" communication mode where the send
that will return once the system can guarantee that the receive will
actually complete.  

(During the discussion of this there was a suggestion that the word REGULAR
on p. 16 of Marc Snir's draft be changed to STANDARD so first can always be
used.  Marc remarked that he welcomes all manner of stylistic improvements,
but asked that they be sent to him, not voted on here.)

Why such a secure communication mode?  To provide a portable manner to
write programs that are guaraneed to be safe, even without buffering.
This is similar to but weaker than the synchronous functions that were
rejected in a straw vote last time.  The unease with a proliferation of
functions was again mentioned.

Adam Greenberg suggested that one could be against this and still specify
equivalent function by requiring no buffering throughout the system.

Rik Littlefield (as a pseudo Tony Skjellum) proposed receive criteria based
on an intag and mask.  Variations on this have been discussed in the past.

VOTE: Typed DONTCARE?
- ----  Yes: 19 No: 1

After the vote, a count showed 25 organizations present.

Proposal: Receive selection based on (tag & mask) = (intag & mask)

Why do this if have context?

Discussion of efficiency.  PRP proposes sending tag (exact match by
system) and extra-info (for recognition of message category by
application) in envelope.  Cownie unhappy because of effect on latency
of small messages.

Alternatives being considered:

(1) RECEIVE(..., tag, info, ...)  no DONTCARE for tag
(2) RECEIVE(..., tag, mask, ...)  no DONTCARE for tag
(3) RECEIVE(..., tag, ...)        DONTCARE for tag

Does this imply small tags?  PRP: reasonable for implementation to
limit size of tag, category, and context.
Greenberg argues againt PRP propsal because of existing practice and
because it could force user to duplicate some system function.

Eric argues in favor of wildcarding because it often allows reuse of
buffer and so fewer resources.

VOTE: (1)  fails for lack of second.
- ----  (2)  Yes: 6  No: 12
so (3) remains.

Proposal: Secure communication mode.
Rusty - P4 experience argues against this.  Cownie - this is useful
for reasoning about program correctness.  
An alternative is to have a "secure mode".  But what happens if using
secure mode, but some library may have been written without secure mode?

VOTE: Do we want a "secure" communication mode?
- ----  Yes: 7 No: 6

VOTE: Delete send/ready_receive?
- ----  Yes: 10 No: 10
[Amendment tabled]

VOTE: Accept amended versions of 2.7 & 2.8
- ----  Yes: 25 No: 1

Discussion of 2.9 (Communication Objects):
- -----------------------------------------

Marc Snir began with an overview of the four subsections.

He asked if there should be handles for user space objects?  This is not in
the current proposal.  He noted that the use of a STATUS_ALL compared to
using STATUS in a loop is one of convenience.

Jim Cownie argued: WAIT_ANY should be told where to start scan because of
fairness concerns. 

[Discussion - this is a strong implementation requirement.  MPI requires
"fairness", though the requirement is so weak as to be untestable.
WAIT_ANY is small part of the fairness problem.  Easier for user to pass in
this information than for the system to maintain it.  The user can always
guarantee fairness.  But at some cost.  Postpone until we discuss
correctness and fairness in general.]

The current draft has MPI_RETURN_STAT providing the free space in the
buffer.  The current practice appears to return the number of bytes
received.  

Issues: Byte count of data received; Where do handles come from; Partially
specified handles; Make explicit that can a message can be sent to oneself.

VOTE: The return status for a receive operation should be the number
- ----  of bytes received
      Yes: 17 No: 3 Abstain: 7

The return status handle should be allocated by the user:
- --------------------------------------------------------
Typically the user knows exactly the what needs to be done and the
lifetime is often short.  Thread safety says can't use global storage,
so ...

Suggestion that there be an overall proposal to deal with handles in
user space.  But this is not a general handle - this is a special
situation.  Postpone for general consideration.

Partial handles
- ---------------

return_status_handle
 + part of communication handle
 + separate user space object
 + separate system object

VOTE: Accept 2.9 (with amendments) excluding 2.9.1 and WAIT_ANY
- ----  Yes: 19 No: 2 Abstain: 3

- -----
break 10:48 - 11:08
- -----

Discussion of sections 2.10-2.12
- --------------------------------
Marc presented the following table for discussion:

              |general|contig|vector|contig|
              |buffer |byte  |byte  |type  |
- --------------+-------+------+------+------+--------------------------------
blocking      |       |      |      |      |
send          |  *    |      |      |      |
receive       |       |      |      |      |
- --------------+-------+------+------+------+
blocking      |       |      |      |      |
ready-receive |  *    |      |      |      |
send          |       |      |      |      |
- --------------+-------+------+------+------+
immediate     |       |      |      |      |
send          |  *    |      |      |      |
receive       |       |      |      |      |
- --------------+-------+------+------+------+
immediate     |       |      |      |      |
ready-receive |  *    |      |      |      |
send          |       |      |      |      |
- --------------+-------+------+------+------+
secure        |       |      |      |      |
send          |       |      |      |      |
receive       |       |      |      |      |
- --------------+-------+------+------+------+
Same buffer types are used in Collective Communication.

Discuss probe

Issues: Counting units?  Vector type?  Contig type includes contig
byte because have byte type?

Lyndon - There should be a secure-receive (for optimization of the
protocol.)  Marc - Is system or user responsible for insuring this works?
Lyndon - User.  Various - What is value of secure-receive.  Gropp -
Because these are different protocols loose performance if don't have
different functions as general must always deal with worst case.

Marc:
                   error?
ssend(2)	----------------------------      recv(1)

Proposal is that it is erroneous to attempt to receive a message sent
by a secure-send by other than a secure-receive.  Rusty argues that
this should not be erroneous because ???

Possibilities:
  1) secure-send {can, cannot} be received by receive
  2) Enforced by MPI or user responsibility

Adam argues that this is confusing the secure-send features with
pre-acknowledgement and these are independent issues.

The purpose of secure-receive is entirely performance - it does not
have any semantic content.

secure-send/secure-receive can be implemented on top of regular p-p.

Note that the proposal is to add 2 secure-sends, 2 secure-receive

VOTE: Add both blocking and immediate secure-send/secure-receive
- ----  with failure of program to match being erroneous.
      Yes: 10 No: 8 Abstain: 9

VOTE: 2.10 as amended
- ----  Yes: 20 No: 4 Abstain: 3

VOTE: 2.11
- ----  Yes: 26 No: 0 Abstain: 1

2.12
- ----

Are blocks just blocks of bytes or are they typed?
How do count the size of blocks?

Lusk - Want to have blocks of typed data for use in a heterogeneous
environment.

PRP - Offers precise propsal that use the same parameters as in
MPI_ADD_BLOCK.

VOTE: Use exactly the same parameters in MPI_ADD_BLOCK as amended
- ----  Yes: 26 No: 0 Abstain: 1

Adam - Proposal to have functions to have strided messages using blocks.
Rusty - Argues against becasue of problem of proliferation at this
level.  [Arguement was clearly on matter of taste. This is syntactic
sugar as the most general low level routines certainly can be used
for this.]

VOTE: Have strided block message functions.
- ----  Yes: 5 No: 9 Abstain: 12

- -----
lunch 12:00 - 1:35
- -----

Continuation of discussion of Chapter 2.  Proposal that more time
discussing 2.10-2.14 needed, so p-p subcommittee will meet after
dinner tonight.

Discussion of Schedule
- ----------------------

Future meetings
  4. May 12-14       set
  5. June 23-25      set
  6. August 11-13    tent.
  7. September 22-24 tent.

Draft to be available: November 15-19  SC '93 Portland

Reading Schedule
- ----------------

P-P			Snir		April & May
Collective		Otto/Guist      April & May
Profiling		Cownie		April & May

Process Topolgy         Hempel          May & June
Environmental Mgmt      Gropp		May & June
Lang. Bind. gen.        Berryman	May & June
Context			Skjellum	May & June

Formal Spec		Zenith		June & August  ???

Specific language bindings will follow general material by one meeting.
Where is the language binding material?  There will be a general
principles chapter and also the actual bindings.  These will be
separate votes.

General language	Berryman	June
Specific lang. bind.	Berryman	

Is anything coming in the Formal Spec?  Does anyone care.  Rusty -
one participant and two observers.  Zenith told Rusty that he was
working on something, but nothing has appeared and he is not here.

What about public comment?  Discussion of the HPFF model of two
opportunities for comment.  Proposal to have only one round of public
comment with draft released to public at Supercomputing '93.

Reference implementation - Gropp/Lusk effort
Test suite - Greenberg and Haup will lead an effort
Subset - 

"Implementation Order Recommendation"

Huss

                                 Goals
 + Define a reasonable subset of MPI that is recommended for initial
   implementation

 + Only a minimum - welcome to implement more

 + Allow MPI to begin to show up in a timely fashion while still
   consistent across vendors

 + Consistent with complete standard
- --
                                Method
 + Create new "subset committee"
 + Write an Annex (like HPF)
 + Present flushed out proposal/annex at next meeting
 + Hope "other" committee will create initial test suite for minimal
implementation :-)  Might motivate implementors
- --
First shot list
  + want subcommitte members
  + Begin discussion via e-mail
- -----
NOT IN SUBSET
1  No persistent handles
2  No multicomponent buffer descriptors {only one item described}
3  No indexed component
4  No waitany or waitall
5  No name server model

If interested send mail to Huss

- -------------------------------------------------------------------------------
             Report From the Collective Communication Subcommittee
- -------------------------------------------------------------------------------

Collective Communications - First Reading
- -----------------------------------------

Steve Otto

Subcommitte of three met last night (Otto, Ho, El...)

Propose a discussion about safety, semantics and function of collect c
  NOT:  Contexts/groups
  NOT:  detailed questions of data types, lan. bind, p-p

Semantic Warm-up
- ----------------
  Barriere implies a time synchronization of processes, but NOT an
emptying of all message buffers.  I.e. p-p messages may span (in
time) a barrier:

   1					2
post-send(2)			post-receive(1)
barrier				barrier
complete-send(2)

barrier does not imply that message queues were emptied.  Object to
statement on p. 8. {{{QUOTE}}}
- --
If we want collective function that does ensure all messages queues
were emptied, let's invent it.

                          wait-for-all-global
Example 3. p. 46
Is this safe?

   0					1
bcast                /---------------   receive(0)
send(1)	------------/                   bcast

Does this deadlock?

Depends on implementation of bcast
 If bcast implemented with buffered contenxt-unique messages, probably won't deadlock
 If bcast synchronizes strongly, probably will deadlock

Otto unhappy about defining sematics of collective communication
routines in terms of operational p-p routines because this depends on
side effects such as emptying message queues.
- --
Cownie/Snir argue that this is not true because ... {{{MISSED}}}
- --
                         Conservative Proposal

Instead of mandating that example above is "safe", we propose that no
messages are allowed to be "in the air" upon entry to collective
communication call.  So:

We require that c-c routines be used AS IF they implied barrier synch
                                  but
                                  === 
the user cannot assume that they actually provide barrier synch 
- -- 

Confusion about what is actually intended.  Unhappiness about phrase
"in the air". The example above is unsafe, but is it erroneous?  Claim
is that this is unsafe, but an implementation that allows it is
compliant.

Now what is the situation of the example on p. 46?  It is there
because Marc want people to be aware that the behavior of a valid (if
unsafe) program may be surprising.
- --
Related point: May want to have a "barrier mode" for c-c so that they
do behave as barriers when this is on.  Useful - can detect many
erroneous programs.
- --
Keeping multiple c-c's separated
- --------------------------------
   /group 1                           /collective operation 2
[                          (    ]                               )
   \collective operation 1   ^         \group 2
                             |
                         can messages in here to go to wrong destination?

Intersecting spanning trees ... can they catch each other's messages

It seems that good implementations can be constructed so that they don't.

 ==> NOT dependent on user-tags

TAGS ARE SUPERFLUOUS FOR C-C
 + keep for consistency with p-p?
 + but, what is the implemntor supposed to do with them?
 + ignore them?
 + what do we tell users about tags in c-c?

We propose - NO TAGS in c-c!

- --
Why have them - for debugging.  Anything else?
Does lack of tags affect non-blocking c-c?  Yes.
Steve Huss: Want to make sure that system must insure that successive
broadcasts will never result in confusion of messages.
Marc: What is the exact semantics of c-c, in particular when is the
burden on the user of MPI to insure no ambiguity vs.l when this is
guaranteed by the system?  This is particularly important if
generalize to allow non-blocking c-c.
PRP: Might want to guarantee that ordering is guaranteed between c-c.
Snir: Certainly want to guarantee that if parameters in c-c are
different, then there is no ambiguity.  Further may guarantee even if have
same parameters.  Proposal - no tags and no ambiguity (so matching is
via parameters and sequencing.) [System responsiblity vs. user
responsiblity]
Steve: Wants MPI to guarantee preservation of order
Kapenga/Pierce: Order may not be meaningful
Gropp: Already have similar issue for p-p.
Snir: So what is behavior when order is not meaningful?
Cownie: Kapenga made important point that multithreaded libraries
cannot use c-c without context information.
Various: MIght be able to do this by using duplicated groups.  Again,
issues is what responsibility falls on system and what on user.
One possible solution is to require user to provide ordering of c-c.
- --

Functions
- ---------
  bcast
  barrier
  gather
  global reductions
  cshift/eoshift
  scans
  all-to-all bcast
  index
- --

3.1 Introduction
- ----------------
  Note that non-blocking c-c is not included

VOTE: Include non-blocking c-c?
- ----   Yes: 0 No: uncounted Abstain: 9

  Note that groups carry no topology information.

Weeks proposes adding a perfect shuffle c-c function.  Ranka proposes
adding a permutation c-c function.  Mention of variations on this.

Rik observes that the arbitrary buffer descriptor versions will be
extremely complex in implementation.  He also objected that this
cannot be implemented using p-p.  Marc responed that it is possible
using p-p because can send message to oneself.

Discuss this in detail when we get to the reduction section.

3.2 (Group Functions) & 3.3 (
- -----------------------------
LATER

3.4 Synchronization
- -------------------

Tag is removed, so examples are now wrong and must be replaced.

Jon wants the examples removed as they contain semantics that are not
guaranteed.  Marc suggests moving to appendix with the hope that this
would eventually contain a full specification of c-c in terms of p-p.
Agreed.

VOTE:  Remove tag in all c-c?
- ----   Yes: 15 No: 5 Abstain: 5

VOTE:  Accept just semantics of 3.4
- ----   Yes: 23 No: 0 Abstain: 2

- ---
break 3:15 p.m. - 3:35 p.m.
- ---

Ho has paper on collective-communication library.  He will make it
generally available.

3.5 (Data move functions)
- -------------------------

Otto: max size to shift?  NO

Issues:
  + What if inbuf is outbuf? 
  + Periodicity in topology?  [Hempel]
  + Tie topology to chsift    [Hemple]
  + cshift as sendreceive(source, destination, ...)  [Flower]

Allowing inbuf and outbuf to be the same violates F77. Rik -
user-level double buffering a pain.
various: back and forth on responsibility of user vs. system.
Steve Huss: Proposes a cshift with only 1 buffer.  
Disallowing partially overlapping buffers.  Note that this new cshift
will have an INOUT argument.
Does this same argument apply to other reduction functions?  Have to
check one by one.

VOTE:  Allow 2nd cshift with only one buffer. No partial overlap on
- ----   orig cshift
       Yes: 21 No: 0 Abstain: 6

Note that earlier encompassing vote imples that we will have types in
cshift.

Marc remarks that changes to measuring size in bytes affects
statement in text that buffers have same number of units.

eoshift
- -------

Second single buffer form?
zero filling - another argument?

VOTE:  Accept cshift, eoshift proposals with amendment of 2nd form of each.
- ----   Yes: 20 No: 2 Abstain: 4

bcast
- -----

VOTE:  Accept bcast
- ----   Yes: 22 No: 0 Abstain: 2

gather
- ------
 Note - len is number of bytes, not what is written.

 Long general, rambling discussion of gather.

Proposal - separate IN argument to gather of sizes; separate
functions to find sizes.

Proposal - version of gather with list of outbufs

Proposal [Flower] - all-to-all gather(???)

Cownie moved to direct gather and scatter back to subcommittee for
further consideration.  Accepted.

3.6 (Global Compute Operations)
- -------------------------------

Issues:  
  + inbuf=outbuf problem?
      2nd version
  + have types, so don't need (R,I)MAX, etc.
  + return value to all?
      3rd version
  + maxloc (etc.) return location and value
  + restriction on user defined functions
  + vectorized user function

Returned to committee for further work.

scan - nothing to say for today
correctness - nothing to say for today

Finished with collective communication for today.

- -------------------------------------------------------------------------------
                 Report From the Process Topology Subcommittee
- -------------------------------------------------------------------------------

Rolf Hempel presided.

No presentation, but need to discuss the direction to go.  First
question - is topology going to be part of group management at all?

Rolf remarks that vast majority of applications have a natural topology.
There are implementation efficiencies - e.g. avoiding tables for
mapping to processors.

Marc: User visible mapping of processes to processors is not likely
to be valuable.  Trend in hardware is to hide hardware topology.

What is the advantage of topology?  Convenient in writing programs
when topology is natural to problem.  This information may be useful
for implementation on particular systems.

What do vendors have to say about the utility of such information.
Cownie: The point is to be as flat as possible.  

Snir: Topology can certainly be built as a superstructure.  What is
value of integration?  Convenience, safety.  Safety does not appear
important and he does not see sufficient value to convenience.  Major
issue is relation of topologies and groups.  Can store topology as
part of group (current proposal) or as a superstructure (Marc's
preference.)

Certainly convenient to have standardized method to build e.g. row
group, column group, etc.  This is what is in current draft, but the
issue is one of integration.  But is it more efficient or even
substantially more convenient.

Three possibilies: Not in MPI, in MPI but not integrated, integrated
into group mechanism in MPI.

Various repetitive arguments for each of these positions.

Straw vote:  Topologies in MPI?
- ----------   If yes
                integrated with groups (e.g. eoshift), OR
                separate library, environmental inquiry
             In: 25 Out: 4 Abstain: 5
                integrated: 4  Separate: 23  Abstain: 7

Back to discussion in the subcommittee.

Meeting tonight:  p-p  room 1
                  c-c  room 2
 =============================================================================

                           Friday, April 2, 1993

- -------------------------------------------------------------------------------
                    Report From the Profiling Subcommittee
- -------------------------------------------------------------------------------

Profiling - Jim Cownie
- ----------------------

Based on draft section ?? of document.  This is a single level approach
and is basically static, i.e. based on selecting actual functions at link
time.

Questions:

1)  Is one level OK?
     - other option is chain of function pointers

2)  Debugging support?
     - Dump all message envelopes
     - Status of active handles

Single level has no extra cost, but limits as might want to use the
single level for another purpose e.g. as a network intermediary.

Yes, that limit could be significant.
Provide multi-level interface in the same manner (i.e. publishing
alternative names for each level)?  [Problem is setting the multiple
interfaces.] 
Provide environmental facilities for exporting a multi-level dynamic
approach?

General agreement that single level is better than extra cost for
everyone.  Note that single level actually can support full dynamic
approach as the MPI routines can be replace by functions that do function
pointer swizzling.

Debugging support?
- ------------------

A small amount of discussion last night.  What is needed and what can be
provided? Besides items mentioned above, need to be able to decode the
opaque objects in MPI.  Useful to have a recording mechanism.

- -------------------------------------------------------------------------------
                 Report From the Language Binding Subcommittee
- -------------------------------------------------------------------------------

Scott Berryman

Gross assumptions:
   No Fortran 90 binding
   C++ binding = C binding
   Specification says nothing about language interoperability
   {The confusion bomb went off!} 

[sender deals with native message; XDR in general; underspecified buffer
descriptors - incl. lengths or incl.  lang. spec.; general vs. limited
translation; know transl in hetero - don't know in homo; this is not just
a language issue - it is a language implementation issue][Need concrete
proposals]

- -------------------------------------------------------------------------------
     Report From the Point-to-Point Communication Subcommittee (continued)
- -------------------------------------------------------------------------------
Marc Snir

Byte vs. Element Count

 + Need ADD_VEC, ADD_INDEX with byte displacement

 + Most usage will be element displacement
- --
[R] [III] [R] [III] [R] [III] [R] [III]  (array of records) so odd displacement
- --
Have two different components

 + Block
    start
    len   -  number of elements
    data type

 + VEC
    start
    len  - total number of elements
    stride - number of elements between blocks
    lenblk - number of elements per block
    data type

 + HVEC
    start
    len - total number of elements
    stride - number of BYTES between blocks
    lenblk - number of elements per block
    data type

[///] [////] [ ][///] [////] [ ][///] [////] [ ]

  VEC(..., 5, 3, 2, REAL)
 HVEC(..., 5, 3*size_of_real, 2, REAL)

INDEX
  start
  array_of_indices - element index (start has index zero)
  type

HINDEX
  start
  array_of_indices - element displacement in bytes
  type

HVEC, HINDEX more general but less convenient and more error prone

Returned length of received messages
- ------------------------------------

Number of elements IS always meaningful (even for message containing
multiple type.)  And number of elements sent is always the same as the
number of elements received, while the number of bytes sent/received need
not be the same.  Moreover the number of elements is not too hard to compute.

Proposal: Use the element count except for displacements in Hxx buffer
components. 

Organizations present: 22

VOTE: proposal as above
- ----  Yes: 19  No: 0  Abstain: 3
Probe
- -----

1) Use to decide where to receive message (allocate memory)
2) Use to debug.

Propose to support 1.

Do we need probe?  Why not receive in a system buffer and return a pointer?
Problems - system buffer is untyped; memory management

Proposal:  MPI_PROBE(source, tag, context, flag, return_status_handle)
MPI_RETURN_STATUS(handle, len, source, tag)

Assuming no other concurrent receive (single thread) MPI_RECV executed
with source/tag returned by PROBE and same context will return message
found by probe.  Multithread programs need suitable critical region.

What is returned in the len field?  Should be the number of elements, but
unfortunately don't know this without buffer descriptor.  So return
number of bytes?  This may not be useful for deciding size of buffer in a
hetero env.  So possibilities:
  1)  len = -1
     and provide  DECODE(buff_desc, msg_status_handle)
  2)  Number of bytes ("on the wire")
  3)  Number of elements (cost of including this info in envelope)

Could have 2) and also provide DECODE.  The only inconvenience is the
difference in units between DECODE and RECEIVE.

Another alternative is to provide a data type as part of the PROBE rather
than a buffer_descriptor - this would cover the most common case of
uniform buffer_descriptors.

Cost of rebuilding buffer_descriptor for receive because have to add in
pointer for actual storage space. {{???Is this right}}

Oliver object proposal again

  4)  PROBE(source, tag, context, type, return_status_handle) returning
      number of elements.

Straw Vote:  Probe?
- ----------   Yes: 23  No: 2

Straw Vote:  1) probe - no length              8
- ----------   2) probe - simple type length    17
             3) probe - byte count            11
             4) probe - element count          6
             5) decode function               22

Cancel
- ------

MPI_CANCEL(handle)
  Either communication succeeds or CANCEL succeeds, but not both nor neither.
  (This is to cancel a non-blocking send.)

  + Need CANCEL to recover committed resource
  + Implementation is not trivial

Is it valid for CANCEL to always fail?  No, if there is a send with no
corresponding posted receive, then CANCEL must succeed.

Recognition that CANCEL may be a very expensive operation.  For example
it may require an interrupt drive mechanism.

What is the effect (cost) on normal communication?

Straw Vote:  CANCEL?
- ----------   Yes: 14  No: 6

Type Mismatch
- -------------

Suppose send 4 bytes and receive integer or conversely.  Is this unsafe
or erroneous?  Various proposals: type mismatch is always erroneous; type
mismatch is never erroneous; BYTE type is an escape hatch.  Another
proposal is to allow type conversion as well.

Straw Vote:
- ----------
             1.  type mismatch always erroneous             2
             2.  type mismatch erroneous except BYTE       12
             3.  never erroneous                            9

- -------------------------------------------------------------------------------
                              Report on MPI -1.2
- -------------------------------------------------------------------------------

Jim Cownie presented.

Procedure clarification:  All official votes require majority of ABSTENTIONS.

Added Features

  Insecure send for users who lack confidence

  Tags are opaque objects

  Message data is opaque

  All lengths are in bits, measure as floats (for sufficient precision)

  Messages are not passed in envelopes (they are too small), but packing
    crates. 

  Context proposal number server
    - returns 64 bit integer (expected to last at least 1 week)

  Group simplification: maximum number of elements in a group is 1

  All communications occur in a group