Subject: Environmental Inquiry Draft

From: William Gropp <[email-masked]>
Date: Thu, 17 Feb 1994 09:09:14 -0600

Here is the updated Environment chapter.  It contains a number of points for
discussion at next week's meeting.
Bill

%
% Updated to include comments from MPI goes to Europe...
%
% Here it is.  I think I've handled the typos and problems.  Let me know
% if anything else needs to be done.
% Bill
%
%
\raggedbottom
\newenvironment{discussion}{\ifvmode\else\par\fi\vspace{\discussSpace}%
\begin{small}\narrower{\bf Discussion:}}{\ifvmode\else\par\fi\end{small}%
\vspace{\discussSpace}}

%\def\comm{comm}
%\def\Comm{Communicator}

%\begin{document}
\chapter{MPI Environmental Management}
\label{sec:environment}
\label{chap:environment}

%\footnotetext[1]{Version as of October 14, 1993}
% changed significantly buffer setting and suggesting
% added HOST inquiry
% deleted type decode (because it is still incomplete)

This chapter discusses routines for getting and, where appropriate, setting
various parameters that relate to the MPI implementation and the execution environment.
(such as error handling and communication buffering).
The procedures for entering and leaving the MPI execution environment are also
dexcribed here.

\section{Implementation information}
\label{sec:inquiry-impl}

\subsection{Environmental Inquiries}
\label{subsec:inquiry-inquiry}

The following routines allow to inquire on the execution environment of an MPI program.

\begin{funcdef}{MPI\_GET\_VALID\_TAG\_RANGE ( low, high )}
\funcarg{\OUT}{low}{Smallest valid tag value (integer)}
\funcarg{\OUT}{high}{Largest valid tag value (integer)}
\end{funcdef}

\mpibind{MPI\_Get\_valid\_tag\_range(int~*low, int~*high)}

\mpifbind{MPI\_GET\_VALID\_TAG\_RANGE(LOW, HIGH, IERROR)\fargs INTEGER LOW, HIGH, IERROR}

\noindent
This routine gets the range of valid tags; that is, the user may use any
tag value from \code{low} to \code{high} inclusive.
These values are guaranteed to be unchanging during the execution of an MPI
program. 

\change
\begin{discussion}
After MPI in Europe, there was considerable demand for the two features:
\begin{enumerate}
\item zero is guaranteed to be valid (low \(\le 0 \le\) high).
\item high is guaranteed to be at least 127 (gives a minimum range that users
can count on; virtually everyone expects a larger range).
\end{enumerate}
An alternate proposal is to eliminate \code{low}, requiring that \code{0} be
the minimum value.
\end{discussion}

\begin{funcdef}{MPI\_GET\_PROCESSOR\_NAME( len, name )}
\funcarg{\IN}{len}{Length (in characters) of the space in \code{name}}
\funcarg{\OUT}{name}{A unique specifier for the actual (as
opposed to virtual) node.}
\end{funcdef}

\mpibind{MPI\_Get\_processor\_name(int~len, char~*name)}

\mpifbind{MPI\_GET\_PROCESSOR\_NAME(NAME, IERROR)\fargs CHARACTER*(*) NAME\\INTEGER IERROR}

The name is a character string for maximum flexibility.  From 
this value it must be possible to identify a specific piece of hardware;
possible values include ``processor 9 in rack 4 of mpp.cs.org'' and ``231''
(where 231 is the actual processor number in the running homogeneous system).
A error code of \const{MPI\_UNDEFINED} is allowed but strongly discouraged.
The value of \const{MPI\_MAX\_PROCESSOR\_NAME} is the maximum length required
by \mpifunc{MPI\_GET\_PROCESSOR\_NAME} (including, in the C binding, the
terminating null).

\change
This routine returns the name of the processor on which it was called at the
moment of the call.  This is to allow MPI implementation that do process
migration to return the current processor.  Note that nothing in MPI {\em
requires} or defines process migration; this definition of
\mpifunc{MPI\_GET\_PROCESSOR\_NAME} simply allows such an implementation.

\begin{funcdef}{MPI\_GET\_HOST( comm, rank)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{rank}{rank of {\tt HOST} in group of {\tt comm}; \const{MPI\_PROCNULL} if there is no {\tt HOST} (integer)}
\end{funcdef}

\mpibind{MPI\_Get\_host(MPI\_Comm~comm, int~*rank)}

\mpifbind{MPI\_GET\_HOST(COMM, RANK, IERROR)\fargs INTEGER COMM, RANK, IERROR}

This routine gets the rank of the {\tt HOST} process in the group associated
with communicator {\tt comm}, if there is such.  MPI does not specify what it
means for a process to be a {\tt HOST}, nor does it requires that a {\tt HOST}
exists. 

\subsubsection{Alternative}
\change

A
set of predefined integer-valued attributes is associated with the
communicator
\const{MPI\_COMM\_WORLD} at initialization time.  These define the execution
environment of MPI programs.  The values of these attributes can be
retrieved by
calling \mpifunc{MPI\_GET\_ATTR}.  These attributes cannot be deleted.
The list of predefined attributes keys include
\begin{description}
\item[\const{MPI\_TAG\_UB}]  Upper bound for tag value
\item[\const{MPI\_TAG\_LB}]  Lower bound for tag value (if it is not set to be
zero)
\item[\const{MPI\_HOST}] Host process rank, if such exists,
\const{MPI\_UNDEFINED}, otherwise.
\item[\const{MPI\_SERVER}] Server rank, or 0/1 flag, indicating whether a
server exists (depending on final decision on intercommunicator name servers)
\item[\const{MPI\_IO}] rank of a node that has regular I/O facilities (possibly
myrank).  Nodes in the same communicator may return different values for this
attribute. 
\item[\const{MPI\_ERROR}] error handler type: fatal, noerror, or user defined.
\end{description}

Vendors may add implementation specific attributes (such as node number,
real memory size, virtual memory size, etc.)

\begin{rationale}
(1) Why have a slew of inquiry functions if we already have an attribute
mechanism?

(2) The open ended form of this mechanism allows to easily add attributes.
\end{rationale}

We may still want the \mpifunc{MPI\_GET\_PROCESSOR\_NAME} call, which returns a
string.

\subsection{Buffering}
\label{subsec:inquiry-buffer}
While it is possible to design algorithms that do not require that any
buffering be provided for sending and receiving ``blocking'' messages, many
practitioners expect that the message-passing system will provide some
buffering.  That is, there is often an expectation that the program
fragment, executed by all tasks, 
\begin{verbatim}
MPI_send( buf, size, datatype, destination, tag, comm )
MPI_recv( buf, size, datatype, destination, tag, comm )
\end{verbatim}
\noindent
will execute for messages whose \const{size} is not too large.  
In fact, this code may deadlock at the first \code{MPI\_send}.
The purpose of
this section is to give the user a way to determine what ``too large''
is and to help the implementation provide enough buffering for the user's
need. 

Note that nothing in this section {\em requires\/} an implementation to
provide any buffering; that is, an implementation may define the above program
fragment to be erroneous for any positive value of \const{size}. 

\subsubsection{General buffering parameters}

\begin{funcdef}{MPI\_GET\_BUFFER\_PARAMS( numsend, sizesend, numrecv, sizerecv, totalnumsend, totalsizesend, totalnumrecv, totalsizerecv)}
\funcarg{\OUT}{numsend}{Bound on the number of pending messages with
source at calling process and same destination}
\funcarg{\OUT}{sizesend}{Bound on the total size in bytes of pending
messages with source at calling process and same destination}
\funcarg{\OUT}{numrecv}{Bound on the number of pending messages with
destination at calling process and same source}
\funcarg{\OUT}{sizerecv}{Bound on the total size in bytes of pending
messages with destination at calling process and same source}
\funcarg{\OUT}{totalnumsend}{Bound on the number of pending messages
with source at the calling process and any destination}
\funcarg{\OUT}{totalsizesend}{Bound on the total size in bytes of pending
messages with source at the calling process and any destination}
\funcarg{\OUT}{totalnumrecv}{Bound on the number of pending messages
 from any source with the calling process as destination}
\funcarg{\OUT}{totalsizerecv}{Bound on the total size in bytes of
pending messages from any source with destination at the calling process}
\end{funcdef}

\mpibind{MPI\_Get\_buffer\_params(int~*numsend, int~*sizesend, int~*numrecv, int~*sizerecv, int~*totalnumsend, int~*totalsizesend, int~*totalnumrecv, int~*totalsizerecv)}

\mpifbind{MPI\_GET\_BUFFER\_PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR)\fargs INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEDND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR}

A program where at any point in time no process has more then \mpiarg{numsend} pending 
messages sent by it to the same destination, 
no more than \mpiarg{sizesend} bytes of data in pending  
messages sent by it to the same destination, no more than \mpiarg{totalnumsend} 
pending messages sent by it to any destination, 
no more than \mpiarg{totalsizesend} bytes of 
data in pending messages sent by it to any destination, no more than \mpiarg{numrecv}  
pending messages sent to it from the same source, no more than
\mpiarg{sizerecv} bytes in pending messages sent to it from the same
source, no more than \mpiarg{totalnumrecv} pending messages sent to it
 from any source, and no more than \mpiarg{totalsizerecv} bytes in
pending messages sent to it from any source is guaranteed not to
deadlock because of the lack of buffer space for pending messages.

Examples:  

1. Suppose that pending messages are always buffered at the sender, all
buffers are shared, total buffer size at the process is {\tt B}, 
each pending message takes {\tt h} bytes of
storage, in addition to the storage for its data, and no storage is
lost due to fragmentation. Then possible return
values for \mpifunc{MPI\_GET\_BUFFER\_PARAMS} are
{\tt numsend = totalnumsend = k,  sizesend = totalsizesend = B-kh},
and {\tt numrecv = totalnumrecv = sizerecv = totalsizerecv = INT\_MAX},
for some $\tt 0 < k < B/h$.

2. Suppose that messages are always buffered at the receiver, and that a
separate buffer with {\tt C} bytes is allocated for each pair of
communicating processes; entries for pending messages are kept in a
seeparate table, with room for total of {\tt m} pending messages at
each receiving process; the total number of processes is {\tt p}, and
no storage is lost due to fragmentation.
Then possible return values for \mpifunc{MPI\_GET\_BUFFER\_PARAMS} are
{\tt sendcount = totalsendcount = sendsize = totalsendsize =
INT\_MAX, recvcount = totalrecvcount = m, recsize = C} and
{\tt totalrecvsize = (p-1)$\times$C}.

The call may return the value \const{MPI\_UNDEFINED} in any argument.
If this is the case, then the MPI implementation makes no guarantee
about buffering.

Some implementations may be able to adjust their buffering.  In
addition, since the parameters returned by
\mpifunc{MPI\_GET\_BUFFER\_PARAMS} represent a guarantee about a particular
choice of number of buffers, it may not guarantee buffering that it can
in fact perform.  Thus, in example 1 above, the implementation picks
an arbitrary value for {\tt k}, trading off number of messages for
total message size, rather than letting the user make this trade-off.
In order to allow the MPI user to inquire about a
specific buffering need, the routine \mpifunc{MPI\_SUGGEST\_BUFFER\_PARAMS}
is provided.

\begin{funcdef}{MPI\_SUGGEST\_BUFFER\_PARAMS( numsend, sizesend, numrecv, sizerecv, totalnumsend, totalsizesend, totalnumrecv, totalsizerecv, flag)}
\funcarg{\IN}{numsend}{Bound on number of  pending messages with source at calling process and same destination}
\funcarg{\IN}{sizesend}{Bound on  total size in bytes of pending messages with source at calling process and same destination}
\funcarg{\IN}{numrecv}{Bound on the number of pending messages with destination at calling process and same source}
\funcarg{\IN}{sizerecv}{Bound on total size in bytes of pending messages with destination at calling process and same source}
\funcarg{\IN}{totalnumsend}{Bound on the number of pending messages with source at the calling process and any destination}
\funcarg{\IN}{totalsizesend}{Bound on total size in bytes of pending messages with source at the calling process and any destination}
\funcarg{\IN}{totalnumrecv}{Bound on the number of pending messages from any source with the calling process as destination}
\funcarg{\IN}{totalsizerecv}{Bound on the total size in bytes of pending messages from any source with destination at the calling process}
\funcarg{\OUT}{flag}{\const{TRUE} if the suggestion was accepted,
\const{FALSE}, otherwise}
\end{funcdef}

\mpibind{MPI\_Suggest\_buffer\_params(int~numsend, int~sizesend, int~numrecv, int~sizerecv, int~totalnumsend, int~totalsizesen, int~totalnumrecv, int~totalsizerecv, int~*flag)}

\mpifbind{MPI\_SUGGEST\_BUFFER\_PARAMS(NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, FLAG, IERROR)\fargs INTEGER NUMSEND, SIZESEND, NUMRECV, SIZERECV, TOTALNUMSEND, TOTALSIZESEN, TOTALNUMRECV, TOTALSIZERECV, IERROR \\ LOGICAL FLAG}

If a value of true is returned in \mpiarg{flag} then it is as if the
values provided by the user in \mpifunc{MPI\_SUGGEST\_BUFFER\_PARAMS}
were returned by a call to \mpifunc{MPI\_GET\_BUFFER\_PARAMS}: the
user is guaranteed that no deadlock will occur due to lack of buffers,
provided it never exceeds the limitations on number and size of
pending messages implied by these parameters.
  
This routines allows the user to tell the MPI implementation that a
certain amount of buffering would be desireable,  The MPI
implementation is not required to provide such buffering.

It is expected that a common use of these routines will be
\begin{verbatim}
MPI_Get_buffer_params( &sendnum, &sendsize, ... );
if (sendsize < buffer_needed) {
    fprintf( stderr, "Program requires more buffering than is available\n" );
    MPI_Abort( 1 );
    }
else
    exploit buffering
\end{verbatim}
This allows a program to fail rather than deadlock.  It is also possible
for a program to switch to a different algorithm that requires less
or no buffering.

Another use is to inform the MPI implementation of the buffer space desires of
the application:
\begin{verbatim}
MPI_Suggest_buffer_params( 1, buffer_needed, 1, buffer_needed, 0, 0, 0, 
                                                               0, &flag );
if (flag) {
    exploit buffering
    }
else {
    fprintf( stderr, "Program requires more buffering than is available\n" );
    MPI_abort( 1 );
    }
\end{verbatim}

\discuss{An implementation is free to accept suggestions only after
\mpifunc{MPI\_INIT} and before any MPI call that performs any
communication.  It is always correct to call
\mpifunc{MPI\_SUGGEST\_BUFFER\_PARAMS} after \mpifunc{MPI\_INIT}.}

\discuss{
These functions are complex (eight parameters), and still do not provide an
accurate information on an implementation (do nonblocking messages share buffer
space with blocking ones?  How many messages are generated by collective
communications?  Or communicator generating calls?)   Are these
functions really
useful, as a ``noop'' implementation is legitimate? especially if we
have packing/unpacking?

The generic attribute mechanism can be used for these communicator parameters,
if we wish to do so.  I.e., add eight buffering attributes to each
communicator,
that can be queried and sometimes modified.
Problems with this approach are that the eight-parameter model allows the
values to be coupled (messages traded against buffer space).

User comments at MPI in Europe were extremely negative about MPI not mandating
large amounts of buffering.  

Many users also read this as mandating no buffering; this caused a great deal
of confusion.
}

\subsubsection{Application provided buffering}

An application program can optionally provide space for 
MPI's use in buffering message data via the following call:

%
%\begin{funcdef}{MPI\_USER\_SPECIFIES\_BUFFER( comm, b, t, flag )}
%\funcarg{\IN}{comm}{ is a communicator with which a 
                         %buffer is to be associated}
%\funcarg{\IN}{b}{the length of the buffer, in bytes}
%\funcarg{\IN}{t}{the number of blocking sends that can be outstanding
%while sharing this buffer}
%\funcarg{\OUT}{flag}{indicates whether the request was successful.  If
%the call is not successful, flag indicates why by using
%\code{MPI\_TOO\_LARGE} for \code{b} to large and \code{MPI\_TOO\_MANY}
%for \code{t} too large.  If both \code{b} and \code{t} are too large, an
%implemtation is free to return either \code{MPI\_TOO\_LARGE} or
%\code{MPI\_TOO\_MANY}.}
%\end{funcdef}
%
%\mpibind{MPI\_User\_specifies\_buffer(MPI\_Comm~comm, int b, int t, int *flag )}
%
%\mpifbind{MPI\_USER\_SPECIFIES\_BUFFER(COMM, B, T, FLAG)\fargs INTEGER COMM, B, T, FLAG}

\begin{funcdef}{MPI\_USER\_SPECIFIES\_BUFFER( comm, buffer, size)}
\funcarg{\IN}{comm}{ is a communicator with which the buffer is to be
associated (handle)}
\funcarg{\IN}{buffer}{initial buffer address (choice)}
\funcarg{\IN}{size}{buffer size, in bytes (integer)} 
\end{funcdef}

\mpibind{MPI\_User\_specifies\_buffer(MPI\_Comm~comm, void*~buffer, int~size)}

\mpifbind{MPI\_USER\_SPECIFIES\_BUFFER(COMM, BUFFER, SIZE, IERROR)\fargs <type> BUFFER(*) \\ INTEGER COMM, SIZE, IERROR}

If \mpifunc{MPI\_USER\_SPECIFIES\_BUFFER} is called and the value returned in
\code{flag} is \code{MPI\_OK}, then for subsequent
regular sends using the specified communicator, MPI must
provide as much safety {\em as if\/} outgoing message data were
buffered by the sending process, in the specified buffer space,
using a circular contiguous-space allocation policy.
%Following the call to \mpifunc{MPI\_USER\_SPECIFIES\_BUFFER}, the
%application is not permitted to access the buffer until the
%corresponding communicator has been freed.
The buffer is managed by MPI.

Access to the provided buffer space is not inherited by
communicators derived from the initially specified one.
 
\implement{      
One approach is for MPI to simply transform all blocking sends
on the specified communicator into 
\begin{itemize}
\item copy data to buffer space
\item issue non-blocking send from the buffer copy
\item return to application
\item check completion on subsequent MPI call(s)
\end{itemize}

No doubt many optimizations within MPI are possible---the
proposal just requires that MPI provide at least this much
safety.

%The reason that MPI provides the buffer rather than the user is that once the
%user frees a communicator, only MPI knows when all data-structures associated
%with that communicator can be freed.
}

Note that collective communications and other nonlocal operations will
share suggested or user-supplied communication buffers with point to point communication.

\discuss{
When a communicator associated with a user buffer is freed, then
\mpifunc{MPI\_COMM\_FREE} should not return until all
pending communications have terminated.  Thus, after it returns, the
buffer can be freed.
}

\section{Error handling}
\label{sec:inquiry-error}

This section contains the suggested POSIX binding for error handling in
MPI.  MPI implementations that do not run in a POSIX-compliant
environment may need to use alternative mechanisms.

An MPI implementation cannot or may choose not to handle some errors
that occur during MPI calls.  These can include errors that generate
exceptions or traps, such as floating point errors or access
violations.
The set of errors that are handled by MPI is implementation-dependent.
Each such error generates an {\bf MPI exception}.

A user can associate an error handler with a communicator.  The
specified error handling routine will be used for any MPI exception
that occurs during a call to MPI for a communication with this communicator.
MPI calls that are not related to any communicator are considered to
be attached to the communicator \const{MPI\_COMM\_WORLD}.
The attachment of error handlers to communicators is purely local:
different processes may attach different error handlers
to the same communicator.

A newly created communicator inherits the error
handler that is associated with the ``parent'' communicator.
In particular, the user can specify a ``global'' error handler for
all communicators by
associating this handler with the communicator \const{MPI\_COMM\_WORLD}
immediately after initialization.

Several predefined error handlers are available in MPI:
\begin{description}
\item[\const{MPI\_ERRORSFATAL}] The handler, when called, causes the
program to abort on all executing processes.  This has the same effect as if
\mpifunc{MPI\_ABORT} was called by the process that invoked the handler.
\item[\const{MPI\_ERRORSRETURN}] The handler has no effect.
\end{description}

Implementations may provide additional predefined error handlers and
programmers can code their own error handlers.

The error handler
\const{MPI\_ERRORSFATAL} is associated by default with \const{MPI\_COMM\_WORLD}
after initialization.  Thus, if the user chooses not to control error handling,
every error that MPI handles is treated as fatal.
Since (almost) all MPI calls return an error code, a user may choose to handle
errors in its main code, by testing the return code of MPI calls and
executing a
suitable recovery code when the call was not successful.  In such case, the
error handler \const{MPI\_ERRORSRETURN} will be used.   Usually it is more
convenient and more efficient not to test for errors after each MPI call, and
have such error handled by a non trivial MPI error handler.

\discuss{
Do we specify that \const{MPI\_ERRORSFATAL} should
generate a \const{SIGABRT} signal at each executing process?  Is that the
desired effect?  (e.g., so as to allow a debugger to take over.)   Same remark
applies to \mpifunc{MPI\_ABORT}.
}

An MPI error handler is an opaque object, which is accessed by a handle.
MPI calls are provided to create new error handlers, to associate error
handlers with communicators, and to test which error handler is associated with
a communicator.

\begin{funcdef}{MPI\_CREATE\_ERRHANDLER( function, errhandler )}
\funcarg{\IN}{function}{user defined error handling procedure}
\funcarg{\OUT}{errhandler}{MPI error handler (handle)}
\end{funcdef}

\mpibind{MPI\_Create\_errhandler(MPI\_Handlerfunction~*function,
MPI\_Errhandler~*errhandler)}

\mpifbind{MPI\_CREATE\_ERRHANDLER(FUNCTION, HANDLER, IERROR)\fargs
EXTERNAL FUNCTION \\ INTEGER ERRHANDLER, IERROR}
Register the user routine \mpiarg{handler\_function} for use as an MPI
exception handler.  Returns in \mpiarg{errhandler} a handle to the registered
exception handler.

\implement{
The handle returned may contain the address of the error handling routine.
Such call is superfluous in C, which has a referencing operator, but is
necessary in Fortran.}

The user routine should be a C function of type \const{MPI\_Handler\_function},
which is defined as

\begin{verbatim}
typedef void (MPI_Handler_function)(MPI_Comm, int , ...);
\end{verbatim}
The first argument is the communicator in use.  The second is
the error code to be returned by the MPI routine.
The remaining arguments are ``\code{varargs}'' arguments
whose number and meaning is implementation-dependent.  An implementation
should clearly document these arguments.

\begin{rationale}
The variable argument list is provided because it provides an
ANSI-standard hook for providing additional information to the error
handler; without this hook, ANSI C prohibits additional arguments.
\end{rationale}

\begin{funcdef}{MPI\_SET\_ERRHANDLER( comm, errhandler )}
\funcarg{\IN}{comm}{communicator to set the error handler for (handle)}
\funcarg{\IN}{errhandler}{new MPI error handler for communicator (handle)}
\end{funcdef}

\mpibind{MPI\_Set\_errhandler(MPI\_Comm~comm, MPI\_Errhandler~errhandler)}

\mpifbind{MPI\_SET\_ERRHANDLER(COMM, ERRHANDLER, IERROR)\fargs INTEGER
COMM, ERRHANDLER, IERROR}

Associates the new error handler \mpiarg{errorhandler}
with communicator \mpiarg{comm} at the calling process.

\begin{funcdef}{MPI\_GET\_ERRHANDLER( comm, errhandler )}
\funcarg{\IN}{comm}{communicator to get the error handler from (handle)}
\funcarg{\OUT}{errhandler}{MPI error handler
currently associated with communicator (handle)}
\end{funcdef}

\mpibind{MPI\_Get\_errhandler(MPI\_Comm~comm, MPI\_Errhandler~*errhandler)}

\mpifbind{MPI\_GET\_ERRHANDLER(COMM, ERRHANDLER, IERROR)\fargs INTEGER
COMM, ERRHANDLER, IERROR}

Returns in \mpiarg{errhandler} (a handle to) the error handler that is
currently
associated with communicator \mpiarg{comm}.

Example:
A library function may register at its entry point the current error
handler for a
communicator, set its own private error handler for this communicator, and
restore before exiting the previous error handler.

\discuss{
One may have the function \mpifunc{MPI\_SET\_ERRHANDLER} both set a new error
handler and return the previous error handler (the function would have two
errhandler arguments), thus saving one call when both \_GET and \_SET are
needed.  But, in any case, one needs both the old and new error-handlers.
}

\begin{funcdef}{MPI\_ERROR\_STRING( errorcode, len, string )}
\funcarg{\IN}{errorcode}{Error code returned by an MPI routine}
\funcarg{\IN}{len}{Length of \const{string}}
\funcarg{\OUT}{string}{Text that corresponds to the \const{errorcode}}
\end{funcdef}

\mpibind{MPI\_Error\_string(int~errorcode, int~len, char~*string)}

\mpifbind{MPI\_ERROR\_STRING(ERRORCODE, STRING, IERROR)\fargs
INTEGER ERRORCODE, IERROR \\ CHARACTER(*) STRING}

Returns the error string associated with an error code.

\begin{rationale}
The form of this proposal was chosen to make the Fortran and C
bindings similar.  A version that returns a pointer to a string has two
difficulties.  First, the return string must be statically allocated and
different for each error message (allowing the pointers returned by
successive calls to \const{MPI\_ERROR\_STRING} to point to the correct
message.  Second, in Fortran, a function declared as returning
\const{CHARACTER*(*)} can not be referenced in, for example, a
\const{PRINT} statement.
\end{rationale}

\section{Error codes}
MPI specifies some of the error return values that an MPI implementation may
return.  This is not an exhaustive list, but MPI implementations must use
these where appropriate. \change
\begin{center}
\begin{tabular}{ll}
MPI\_ERR\_BUFFER            &Invalid buffer pointer                     \\
MPI\_ERR\_COUNT             &Invalid count argument                     \\
MPI\_ERR\_TYPE              &Invalid datatype argument                  \\
MPI\_ERR\_TAG 	          &Invalid tag argument \\
MPI\_ERR\_COMM              &Invalid communicator\\
MPI\_ERR\_RANK              &Invalid rank \\
MPI\_ERR\_REQUEST           &Invalid request (handle)\\
MPI\_ERR\_ROOT              &Invalid root \\
MPI\_ERR\_GROUP             &Invalid group\\
MPI\_ERR\_OP                &Invalid operation \\
MPI\_ERR\_TOPOLOGY          &Invalid topology \\
MPI\_ERR\_DIMS              &Invalid dimension argument \\
MPI\_ERR\_ARG               &Invalid argument of some other kind \\
MPI\_ERR\_UNKNOWN           &Unknown error \\
MPI\_ERR\_TRUNCATE          &Message truncated on receive\\
MPI\_ERR\_INTERN            &Internal MPI error\\
MPI\_ERR\_LASTCODE          &Last standard error code\\
\end{tabular}
\end{center}

\section{Timers}
MPI defines two timers.  These timers are specified even though they are not
``message-passing'' because timing parallel programs is important in
``performance debugging'' and because existing timers (both in POSIX
1003.1-1988 and 1003.4D 14.1 and in Fortran 90) are either inconvenient or do
not provide adequate access to high-resolution timers.  

\begin{funcdefna}{MPI\_WTIME()}
\end{funcdefna}
\mpibind{double MPI\_Wtime()}
\mpifbind{DOUBLE PRECISION MPI\_WTIME()}
returns a floating-point number of seconds,
representing elapsed wall-clock time since some time in
the past.

\begin{funcdefna}{MPI\_PTIME()}
\end{funcdefna}
\mpibind{double MPI\_Ptime()}
\mpifbind{DOUBLE PRECISION MPI\_PTIME()}
returns a floating-point number of seconds,
representing elapsed process time since some time in
the past.

The ``time in the past'' is guaranteed not to change during the life of the
process.  The user is responsible for converting large numbers of seconds to
other units if they are preferred.  

These functions are portable (return seconds, not "ticks"), allow
high-resolution, and carry no unnecessary baggage.  One would use them like
this:

\begin{verbatim}
{
   double starttime, endtime;
   starttime = double MPI_Wtime();
    ....  stuff to be timed  ...
   endtime   = double MPI_Wtime();
   printf("That took %f seconds\n",endtime-starttime);
}
\end{verbatim}

The times returned by these routines are local to the node that called them.
There is no requirement that different nodes return ``the same time.''  

\begin{discussion}
User's at MPI in Europe were strongly in favor of having timing functions.
\end{discussion}

\section{Startup}
\label{sec:inquiry-startup}
One goal of MPI is to achieve {\em source code portability}.  By this we mean
that a program written using MPI and complying with the relevant language
standards is portable as written, and must not require any source code changes
when moved from one system to another.  This explicitly does {\em not\/} say
anything about how an MPI program is started or launched from the command
line, nor what the user must do to set up the environment in which an MPI
program will run.  However, an implementation may require some setup to be
performed before other MPI routines may be called.  To provide for this, MPI
includes an initialization routine \mpifunc{MPI\_INIT}.

\begin{funcdefna}{MPI\_INIT()}
\end{funcdefna}

\mpibind{MPI\_Init(int~*argc, char~**argv)}

\mpifbind{MPI\_INIT(IERROR)\fargs INTEGER IERROR}

This routine must be called before any other MPI routine.  It must be called
at most once; subsequent calls are erroneous (see \mpifunc{MPI\_INITIALIZED}).

All MPI programs must contain a call to \mpifunc{MPI\_init}; this routine must be
called before any other MPI routine (apart from \code{MPI\_INITIALIZED}) is
called. 
The version for ANSI C accepts the \const{argc} and \const{argv} that are
provided by the arguments to \code{main}:
\begin{verbatim}
MPI_init( argc, argv );
\end{verbatim}
The Fortran version takes no arguments.  

\discuss{Note that POSIX includes a {\em requirement\/} that access to the
command line be provided to Fortran programs (see 8.9 in Std 1003.9--1992).

MPI implementations that read command-line arguments should insure
the the MPI command-line arguments are clearly identifed, for example, by
prefixing them with \code{-mpi}.

Note the use of the {\em address} of \code{argc} in the C binding of
\mpifunc{MPI\_INIT}.  This is to allow \mpifunc{MPI\_INIT} to remove those
arguments that it uses from the argument list.
}

\begin{funcdefna}{MPI\_FINALIZE()}
\end{funcdefna}

\mpibind{MPI\_Finalize()}

\mpifbind{MPI\_FINALIZE(IERROR)\fargs INTEGER IERROR}

This routines cleans all MPI state.  Once this routine is called, no MPI
routine (even \mpifunc{MPI\_INIT}) may be called.
The user must ensure that all pending communications
involving a process complete before the process calls \mpifunc{MPI\_FINALIZE}.

\begin{funcdef}{MPI\_INITIALIZED( flag )}
\funcarg{\OUT}{flag}Flag is true if \mpifunc{MPI\_INIT} has been called and false
otherwise. 
\end{funcdef}

\mpibind{MPI\_Initialized(int~*flag)}

\mpifbind{MPI\_INITIALIZED(FLAG, IERROR)\fargs LOGICAL FLAG \\ INTEGER IERROR}

This routine may be used to determine whether \mpifunc{MPI\_INIT} has been
called.  It is the {\em only} routine that may be called before
\mpifunc{MPI\_INIT} is called. 

\begin{funcdef}{MPI\_ABORT( comm, errorcode )}
\funcarg{\IN}{comm}{communicator of tasks to abort}  
\funcarg{\IN}{errorcode}{error code to return to invoking environment}  
\end{funcdef}

\mpibind{MPI\_Abort(MPI\_Comm~comm, int~errorcode)}

\mpifbind{MPI\_ABORT(COMM, ERRORCODE, IERROR)\fargs INTEGER COMM, ERRORCODE, IERROR}

This routine makes a ``best attempt'' to abort all tasks in the group
of \mpiarg{comm}.
This function does not require that the invoking environment take any action
with the error code.  However, a Unix or POSIX environment should handle this
as a \code{return errorcode} from the main program or an
\code{abort(errorcode)}. 

MPI implementations are required to define the behavior of
\mpifunc{MPI\_ABORT} at least for 
a \code{comm} of \const{MPI\_COMM\_WORLD}.

---

From: William Gropp <[email-masked]>
Date: Thu, 17 Feb 1994 13:01:48 -0600

I've placed a compressed postscript version of the inquiry chapter in
pub/mpi/inquiry.ps.Z on info.mcs.anl.gov .
Bill