Subject: MPI_LOAD

From: [email-masked] (Anne C. Elster)
Date: Thu, 16 Sep 93 03:44:56 -0400

Well, folks,
I promised a nice write-up on my MPI_LOAD suggestion,
but am afraid you will have to do with this patch-up job I'm
throwing together in this wee hr in the morning. Sorry.

I am off to Norway later to day for my Dad's 60th b-day,
which also means I will not be able to make next week's meeting
in Dallas.

Have a nice and productive meeting!

Maybe somebody could get a t-shirt sent to me?

				Take care!
						Anne

-------------------------------------------------------------------------------
Anne C. Elster			   |  (607) 254-8653 [off]   255-6236 [FAX]
Xerox Design Research Institute	   |  (607) 272-2986 [home/answ.mach]
Cornell University, 502 Theory Ctr.|
Ithaca, NY 14853		   |   e-mail:  [email-masked]
USA				   |	        [email-masked]

		MPI_(CPU_)LOAD_INFO

MOTIVATION

Currently, most parallelized algorithms for distributed memory machines
assume that they will run on a homogeneous set of processors.
(I.e. that each node in the parallel system has approximately
the same computing power and I/O speed.)
The main focus in parallel computing has thus far been to minimize
communication over-head and achieve a load-balance for the
particular algorithm or application.

However, as each computational node a parallel system become more complex, 
their individual processor/node performance characteristics will tend to
deviate more as some nodes perform I/O while others concentrate on
system tasks, etc.

This behavior has so far been addressed through multi-tasking on
large shared-memory systems such as a network of IBM 3090s, however,
no simple, straight-forward solutions exist currently
for the distributed memory cases.

It is, however, this authors belief that load-balancing application
at the ``load-info'' level (i.e. based on each node's dynamic load at
run-time) will become very important in the future -- especially
if there were mechanisms built into MPI that would facilitate
such implementations.

CURRRENT OPTIONS

Options for getting load-info read into
a Fortran or C application on Unix-based systems are
not wonderful:

System load is typically from the OS-side done by {\bf reading kernel
memory (-kmem)}. This will, however, usually require {\em setgid}
permissions and grubbing through kernel symbol tables.
This  brute-force approach might fork and exec something with the required
kmem permissions.

Another alternative is to use {\bf rstat} which  does not require
such permission, but unfortunately, is not available on all systems.
Rstat typically gives you statistics from a remote kernel
that should be current disregarding network delay.

The third approach I can think of is to use the output from 
{\bf uptime}. In C, this could possibly be done though
 fscanf on the return value of popen.

MPI SUGGESTION

Since I fond none of the previous methods very programmer-friendly,
my suggestion  is to add a {\bf MPI\_LOAD\_INFO} that would return
the desired information. As I mentioned at the last meeting, I am now
convinced that the way ISIS/Horus does "load-info" for CPU load, i.e.
to return the no. of jobs the scheduler has outstanding, is a good start.

	MPI_(CPU_)LOAD_INFO(n)

	OUT	n  : the no. of jobs the scheduler has outstanding.

This way there is no timing issues associated with the local MPI
call (MPI_CPU_LOAD_INFO, if you wish).  The notion of needing some
kind of time information was the largest objection I heard re.
to my suggestion to include an MPI_LOAD_INFO back when I presented it
in March.

It is up to the applications programmers whether they'd want to do several
calls for a statistical average.  Global info. can be obtained via an MPI
collective comm. call.  Users will also be required to know scheduler queue
sizes and what they mean for the individual sytstem/node, but at least they
will not be stuck with non-standard system-supplied fucntions and "grubbing"
through "kmem" tables under super-user priveleges in order to get the 
information they need.

There are of course several other factors that influence the "load" of
the processing element, e.g.  memory traffic, communication traffic, I/O
traffice, etc, all which could be very useful information for applications
programmers to get their hands one -- especially in non-homogeneous or
multi-user environments. 

Last time I guess some people objected to the notion of adding another
MPI functions, some others felt it was not really an MPI call per se,
and hence should be avoided. I can see the arguments for this, but feel
the benefits of including such a functions too great to let it pas up.

---

From: [email-masked]
Date: Thu, 16 Sep 1993 08:00:57 -0500 (CDT)

Anne C. Elster writes:
[text deleted]
> particular algorithm or application.
[text deleted]
> through "kmem" tables under super-user priveleges in order to get the 
[text deleted]
> the benefits of including such a functions too great to let it pas up.

I'm merely a lurker on this mailing list, so I'll let others discuss
what is and isn't appropriate for MPI.

However, I have written a generalized network resource monitoring library
for the product I'm working on (NetShare SDK, [email-masked] if you
are curious).  In a real heterogeneous network environment you need to
track more than just the load average.  For example, a SPARCstation 10
with a load average of 0.6 will give a better response than an idle
diskless SPARCstation ELC.  Other applications may care more about memory
than load -- and then physical memory versus virtual memory.  Software
properties like licenses and OS type and version may be important.  

(As an example from NetShare, which does remote execution, you may want
to select hosts for a large simulation based on memory, SPECfloat rating,
simulator software license availablity, and having an idle keyboard)

Resource monitoring is important in a heterogeneous environment, however
the simple case of monitoring a single property is not useful in
practice.  

-- 
Ted Stockwell, Aggregate Computing                [email-masked]

---

From: "R.J. Allan" <[email-masked]>
Date: Wed, 22 Sep 93 19:22:20 +0100

A quick comment on load balancing and message-passing harnesses.

We have done load balancing using our own Fortnet harness on, for instance,
a domain-decomposed CFD code.

This is easy to do (given an algorithm!) providing you have access to
CPU time and WALL time, and can time over computation and communication
parts of the code.

I am pleased to see that MPI has this facility already, more general
load tools are very debatable... and probably program dependent in
some way

			Cheers,

Robert J. Allan

Advanced Research Computing Group,
Science and Engineering Research Council,
Daresbury Laboratory,
Daresbury,
Warrington, WA4 4AD,
U.K.

'phone +44 925 603207
FAX +44 925 603634

e-mail [email-masked]