Subject: MPI Job startup

From: Rusty Lusk <[email-masked]>
Date: Tue, 02 Mar 93 15:31:58 CST

We feel that the MPI standard should provide enough information to
specify an entire, source-code portable program.  To do this, it is
necessary to specify how parallel programs start and end.  This is the
analogue of PROGRAM MAIN ... END in Fortran and main(argc,argv)
{...exit(rc); }.  This proposal does not deal with methods for
spawning tasks during the execution of an MPI program, or for handling
explicitly MIMD models (beyond the usual way of starting a Turing
machine on all nodes and sending the appropriate tape).  This is not
meant to suggest that MPI not have such interfaces, just that we are
only proposing an SPMD model at this time.

Issues

It should be possible to write a program that is source-language
portable to any system, particularly if it requires only a simple
interface to the operating system.  An example is an environment that
starts up a program with a fixed number of processors.  Without
something like this, it is not possible to write a portable MPI
program.

However, no standard should restrict programs to a particular
interface.  For example, some applications may want to dynamically
added and remove tasks during the execution of an application.  We
leave it to others on the committee to propse a mechanism for this
case.

Below, we discuss three different approaches; we note that all of
these have been tried in some existing system or systems.

The seemingly simplest approach is to use an explicit initialization
routine, MPI_Init.  There are several problems with this approach.
The major one is that the parallel part of the program is the code
that "follows" the MPI_Init.  This is potentially error prone;
further, it can be awkward to implement on some systems.  This is
the way p4 does it.

Another approach is to insist that the entire program be run in
parallel; a simple way to do this is to replace the "main" program
with a routine.  In C, this could be MPI_Main( argc, argv ).  The
linking step then provides a "main" that handles all appropriate
startup (including the processing of command line arguments where
appropriate for information on the number of tasks etc.) and then
calls the user's MPI_main routine.

Yet another approach is to provide a routine that runs a user-defined
routine in parallel.  For example, MPI_Call( argc, argv, routine ) can
be used to run routine( argc, argv ) in parallel.  A generalization of
this would allow the running of a particular routine on a defined
group: MPI_Call( groupid, argc, argv, routine ).  This is more
flexible than the MPI_Main approach but retains the simplicity of
running a well-defined program unit (the routine) in parallel.

All three of these approaches are currently in use.

These are not the most general methods.  For example, these provide no
mechanism for dynamically creating (remote spawn) tasks on other
processors.  Also, there is no direct support for MIMD programming.
Finally, there is no support for adding an already-running process to
a group of running processes.

Proposal

We propose the MPI_Call form.  A slight variation would allow the use
of a general argument descriptor instead of argc/argv.  This would
allow the more convenient passing arguments to the routine.

We have used this mechanism (in Bill's Chameleon system) to write
programs for nX, the CM-5, the nCube, and networks of workstations
using both p4 and PVM, and it has worked well for us.  The point is
that all source code is portable.  The only thing that is different
is the shell command by which you start your program.  

Note that in this case, program termination is handled by having (a)
the routine exit and (b) the calling routine exit.  Thus an MPI
program would look something like this:

   main(argc,argv)
   int argc; char **argv;
   {
       ret_code = MPI_Call( argc, argv, worker );
       exit( irc );
   }

   int worker( argc, argv )
   int argc; char **argv;
   {

       ...

       if (error condition) return 1;
       ...
       return 0;
   }

In Fortran it would like something like:

      program main
      external worker

      irc = MPI_call(worker)

      stop
      end

      integer function worker

      ...

      worker = 0
      return
      end

It may be useful to have a more drastic exit.  We suggest
MPI_exitall(code); the code here is an exit code that may be ignored
by the invoking environment (but UNIX-like environments should treat
it like exit(code)).

In the case of heterogeneous environments, it is necessary to specify the set
of workstations to be used.  Our only requirement is that this be done, in the
C/POSIX case, through command-line arguments (thus maintaining source-code
portability).  We could specify an extensible, common format for specifying
the machines, executables, and resource limits, but, to avoid additional
debate, we suggest only that the command line option -mpi be reserved
for future extensions.

Bill & Rusty

---

From: [email-masked] (James Cownie)
Date: Mon, 8 Mar 93 12:59:09 GMT

> specify an entire, source-code portable program.  
I agree.

MPI_INIT style
==============
> further, it can be awkward to implement on some systems.  This is
I agree that this is simplest. I also prefer it to the others. I don't
see the "major problem" outlined above. The whole of the code is
parallel even before the MPI_INIT, it's just that it can't communicate
yet. 

I'd expect code like this (given an I/O everywhere model) to execute
fine

int main(int argc,char ** argv)
{
    printf("Hello world\n");
}

and print as many "Hello world" messages as the number of nodes on
which I ran it.

It also seems to me to be quite an easy way to allow the dynamicness
everyone seems to want. The newly created process starts up, and then
attaches to the existing MPI environment at the time it calls MPI_INIT.
	
In my view, then, what happens is

1) By some unspecified means a number of processes start to execute
2) They decide to collaborate by executing MPI_INIT

[If dynamic process creation is permitted
2a) New processes are forked/execed
2b) The new processes join the existing collaboration by executing
    MPI_INIT
]

3) They each exit (by calling exit), or you can have a drastic exit if
   you like.

MPI_CALL style
==============
I don't think I understand what this is intended to mean, which may be
why I don't like it ! Therefore please can you explain it to me.

1) Is the idea that there is one user process which then executes
   MPI_CALL at which point all of the other processes are created,and
   magically arrive in a user subroutine, or do all the processes
   execute main ?

2) If all the processes execute main, what can they do before calling
   MPI_CALL ? 

3) What global state is accessible to the MPI_CALLed routine ?
e.g. (your example with minor change...)

   extern int foo;   /* Of course this could be an arbitrarily complex
                        data structure ! */
	
   main(argc,argv)
   int argc; char **argv;
   {
       foo = time();  /* Something which may differ on each node ! */

   int worker( argc, argv )
   int argc; char **argv;
   {
	
       printf("%d: %d\n", MPI_MYTID(), foo);
       ...

At the moment I definitely prefer MPI_INIT !

-- Jim
James Cownie 
Meiko Limited			Meiko Inc.
650 Aztec West			Reservoir Place
Bristol BS12 4SD		1601 Trapelo Road
England				Waltham
				MA 02154

Phone : +44 454 616171		+1 617 890 7676
FAX   : +44 454 618188		+1 617 890 5042
E-Mail: [email-masked]   or    [email-masked]