Subject: Timers for MPI

From: Rusty Lusk <[email-masked]>
Date: Fri, 11 Feb 1994 23:07:57 -0600

Here are some thoughts on timer functions.  Comments are welcome.

Rusty Lusk

Timers for MPI
==============

A number of users have suggested that MPI provide a mechanism for timing
sections of programs.  While this is not strictly speaking part of message
passing, parallel programmers, perhaps more than others, do a lot of timing,
and have expressed the hope that a portable library like MPI could provide
another portable function that they need.  At the recent European meeting,
users expressed particularly strongly the need for a timing mechanism that was
portable, easy to use, and capable of expressing high resolution.  Both
wall-clock time and process time are needed.

There has been a reluctance on the part of the MPI Forum to define a timing
mechanism, on the grounds that other standards should do so.  A tentative
decision made at the European meeting was that we should mandate that every
MPI implementation provide an appropriate subset of POSIX timer functions.  I
agreed to look into what that might mean precisely.  In this note I survey the
current Unix and POSIX mechanisms from the point of view of portability, ease
of use, and high resolution, and I conclude that none of the existing POSIX
mechanisms really meet these requirements.  I conclude, somewhat reluctantly,
that MPI should define its own, and make a concrete proposal.

POSIX 1003.1-1988  
-----------------

The "time" function for wall-clock time and the "times" function for process
time are what are currently commonly available.

time_t time ( time_t *tloc ) 

fills in the time_t structure with seconds "since the Epoch".  Its resolution
is too low.

clock_t times ( tms *buffer )

fills in at least

    clock_t  tms_utime     (user time)
    clock_t  tms_stime     (system time)

for the process, in the tms structure pointed to by buffer.  It returns the
elapsed time since an arbitrary time in the past.  The type clock_t is defined
to be a long integer, at least on some systems.  The integer represents time
in CLK_TCK-ths of a second, where this number is implementation-dependent, but
is typically a 60th of a second.  Therefore this function is not truly
portable, since the numbers it returns may differ widely on different systems.
It is also not typically high-resolution.

Berkeley
--------

Another common timer is gettimeofday, which is referred to as a "Berkeleyism"
by the POSIX documents.  It is not a standard, but is widely used.

int gettimeofday ( tp, tzp )

where tp points to a structure containing at least

  long seconds;
  long microseconds;

since midnight January 1, 1970, GMT.  The variable tzp points to a time-zone
structure, or is NULL.

The resolution is allowed to vary per implementation, but the values do
represent fixed time intervals.

This is not a POSIX standard, allows for only medium-high resolution, and adds
the unneeded cumbersomeness of the time-zone structure.  (It also becomes
obsolete in the year 2038.)

POSIX P1003.4D 14.1
-------------------

This is the current POSIX draft proposal.

Several calls use the timespec structure:

  time_t  seconds
  long    nanoseconds

where time_t is the current structure used by time(), containing years,
months, days, etc., down to seconds.

Paragraph 14.2.1  - Clocks
--------------------------

int clock_settime(clockid_t, clock_id, struct timespec *tp)
int clock_gettime(clockid_t, clock_id, struct timespec *tp)
int clock_getres(clockid_t, clock_id, struct timespec *tp)

These functions are for setting and getting the value of assorted clocks.  It
is required that all POSIX implementations support at least one clock_id,
called CLOCK_REALTIME.  For this clock, time is measured "since the Epoch".
So this is the POSIX version of gettimeofday() for getting wall-clock time,
and its resolution can be retrieved explicitly.

Paragraph 14.2.2  - Timers
--------------------------

These functions manage timers for measuring process time, but primarily for
managing alarm-type timers and signals.

int timer_create ( clockid_t clock_id, struct sigevent *evp, timer_t *timerid )
int timer_delete ( timer_t timerid )

int timer_settime(timer_t timer_id, int flags, struct itimerspec *value,
                  struct itimerspec *ovalue )
int timer_gettime(timer_t timerid, struct itimerspec *value)
int timer_getoverrun(timer_t timerid )

For example, timer_gettime returns, in the itimerspec structure, the time left
until the specified timer expires.  This is not the kind of timer that the MPI
users are looking for.

For all of these, only the C binding is given in the POSIX documents.  I will
try to find out what Fortran-90 proposes, for example.

Criticism
---------

All of these are cumbersome to use, even in C.  They do provide what is needed
to implement the simple timers that people want, but they don't provide these
functions themselves.

Proposal
--------

   double MPI_Wtime()  returns a floating-point number of seconds,
                       representing elapsed wall-clock time since some time in
                       the past.

   double MPI_Ptime()  returns a floating-point number of seconds,
                       representing elapsed process time since some time in
                       the past.

The "time in the past" is guaranteed not to change during the life of the
process.  The user is responsible for converting large numbers of seconds to
other units if they are preferred.  (Some things are *really* outside the
scope of MPI.)

The Fortran bindings are the same;  the functions return double-precision
floating point numbers.

These functions are portable (return seconds, not "ticks"), allow
high-resolution, and carry no unnecessary baggage.  One would use them like
this:

{
   double starttime, endtime;
   starttime = double MPI_Wtime();
    ....  stuff to be timed  ...
   endtime   = double MPI_Wtime();
   printf("That took %f seconds\n",endtime-starttime);
}

which is what I think the users had in mind.

I suggest these be added to the environmental-inquiry chapter.  Comments and
improvements are welcome.

---

From: [email-masked] (James Cownie)
Date: Mon, 14 Feb 1994 13:22:50 +0000

We should make clear that the time values returned are process local,
and that even the epoch for the timers need not be the same on each
node, also that the resolution may differ on different nodes !
This is implicit in the proposal, but should be mentioned explicitly
as a warning point to users.

In other words that it makes no sense to do something like

	read timer on node 0
	broadcast value
	use value as base on all nodes

instead you must do

	gsync
	remember start time on each node
	use local base value

(This is NOT what users want, but it'll be hard to implement better on
a network of workstations ...)

-- Jim 
James Cownie 
Meiko Limited			Meiko Inc.
650 Aztec West			Reservoir Place
Bristol BS12 4SD		1601 Trapelo Road
England				Waltham
				MA 02154

Phone : +44 454 616171		+1 617 890 7676
FAX   : +44 454 618188		+1 617 890 5042
E-Mail: [email-masked]   or    [email-masked]

---

From: "R.J. Allan" <[email-masked]>
Date: Mon, 14 Feb 94 13:36:57 GMT

I like Rust Lusk's suggestions for MPI_Wtime and MPI_Ptime.

I beleive it is very important to have a fully portable implementation of
such timers for both benchmarking and optimisation purposes and to 
inferface to high level tools for analysis and load balancing...

Robert J. Allan

Advanced Research Computing Group,
Science and Engineering Research Council,
Daresbury Laboratory,
Daresbury,
Warrington, WA4 4AD,
U.K.

'phone +44 925 603207
FAX +44 925 603634

e-mail [email-masked]

---

From: "R.J. Allan" <[email-masked]>
Date: Mon, 14 Feb 94 13:38:06 GMT

I agree with James Cownies comments on the timers - all our software is
programmed to work in this way using local clocks...

Robert J. Allan

'phone +44 925 603207
FAX +44 925 603634