From: [email-masked] (John Kapenga)
Date: Tue, 16 Feb 93 14:53:15 EST

Hi;

A late pre-meeting post. I hope we can talk about the possibility
of adding an io specification to MPI in Dallas.

A couple of us (Leslie Hart, ... and myself) have been talking about
possible MPI io specifications.

Here is a brief outline of IO specifications for one option, a loosely
synchronized version of multiple IO based on the ANSI-C librtary. 
Placing "mp_" before all names is fine (It is my preference to do that
in all libraries).

Leslie and others have a seek and read/write primative whuich has merit
if processes were going to read from differeent "random" positions in a
file. Assuming a message is sent for every call this could be high overhead,
but it keeps life simple. This also allows IO request to be asynchronous
(assuming the seeks are absolute), although we may wear out the disk head
actuator ;-).

Perhaps we should set basic goals for an IO system: 
1) interaction with a terminal
2) writing and reading buffers to files, which should correspond
   closely to sending and recieving messages.
3) three modes of operation - single_io(), ordered_io() and unordered_io()
4) some specification of conditions for deadlock free use.

Some questions:
Should all IO be loosely synchronized?
Should all IO "require" a message be sent with that request
	(implying no buffering / queueing)?
If not "require" then, should the user have any control over
	the buffering / queueing?

I'm not sure about how to address hetrogenious access to a file. I
have seen a couple options. I would tend to not want to force a "binary"
file compatable with all architectures. Text files should not pose as large
a problem. It does appear binary conversions by the message system in
pt-2-pt has supporters, if that works out then soemthing could here too.

This is still a thought exercise, not a proposal. 

john
----------------------------------------------------------------
The Three MPI IO Modes

The three levels of routines below and the three IO-modes are orthogonal.

All IO routines must be executed by all members of a group in
a loosely synchronized manner. All processes must invoke the same
primative on the same stream.

The following IO routines operate in three modes. Initializing IO
and switching between modes using:
	single_io(group)
	ordered_io(group)
	unordered_io(group)
Terminating IO from a group is done by:
	end_io(group)
Each of these calls must be executed by all members of a group.

In single_io() mode the IO routines should function from a process
as described in the relevant standard. On output or in setting the
file position the results from only one of the processes will be
done. On input all processes receive the same values.

In ordered_io() and unordered_io() the results of setting file
positions are done for only one process. Input and Output from a
group of size p results in p separate items being read or written.

The ordered_io() mode requires the p IO operations to occur in process order.
(If we have a group order this should be group order) The order of the p IO
operations in unordered_io() mode is unspecified.

****************************************************************

The POSIX IEEE Std 1003.1-1988 C functions
From Chapter 8, 4.9  (Which is a pointer to the ANSI-C standard
X3.159-1989 which is essentially the same as ISO 9899-1990)
Those functions not mentioned in POSIX which were later included
in the ANSI-C are still be be considered required by POSIX.

Below:
	the functions marked with a "*" are non-ANSI-C
	the functions marked with a 1 provide a complete set,
		without access to buffer cantrol.

Level-1 IO --------------------------------------------------------
1	fopen(), 
		FILE * fopen(const char *name, const char *mode)
	freopen(), 
		FILE * fopen(const char *name, const char *mode, FILE *stream)
1	fclose(), 
		int fclose(FILE *stream)
1	clearerr(), 
		int clearerr(FILE *stream)
1	feof(), 
		int feof(FILE *stream)
1	ferror(), 
		int ferror(FILE *stream)
	setvbuf(),     (not mentioned in POSIX, but it is in ANSI-C)
		int setvbuf(FILE *stream, char *buf, int smode, size_t size)
	fflush(),
		int fflush(FILE *stream)
	rewind(),
		int rewind(FILE *stream)
	fsetpos(),  (Not in POSIX, but in ANSI-C)
		int fsetpos(FILE *stream, const f_pos *pos)
1	fseek(), 
		int fseek(FILE *stream, long offset, int ptrnane)
	ftell(), 
		long ftell(FILE *stream)
	fgetpos()      (Not in POSIX, but in ANSI-C),
		int fgetpos(FILE *stream, fpos_t *pos)
1	fread(), 
		int fread(char *ptr, int size, int nitems, FILE *stream);
1*	sfread(),     (a possible non-ANSI-C stride version)
1*	ifread(),     (a possible non-ANSI_C iovec version)
	fwrite(),  
		int fwrite(char *ptr, int size, int nitems, FILE *stream);
1*	sfwrite(),    (a possible non-ANSI-C stride version)
1*	ifwrite(),    (a possible non-ANSI_C iovec version)

Level-2 IO --------------------------------------------------------
	setbuf(),
		void setbuf(FILE *stream, char *buff)
	perror(), 
		int perror(const  char *err)
1	printf(), fprintf(), sprintf(),
1	scanf(), fscanf(), sscanf().

Other ANSI-C IO --------------------------------------------------------
	putc(), putchar(), puts(),
	fputc(), fputs(), 
	getc(), getchar(), gets(), 
	fgetc(), fgets(), 

	vprintf(), vfprintf(), vsprintf(), (Not in POSIX, but in ANSI-C)
	remove(), rename+(), 
	tmpfile(), tmpname(), 
	ungetc(),

******************************************************************
Remarks: Deadlock ------------------------------------------------
Something must be said about deadlock, this needs to be addressed
as part of the collective communications and pt 2 tp too. Some
options are No deadlock will occur if:
  no messages are "active" while any IO mode is enabled (too strong but safe)
  no messages are "active" in the IO group.
  no messages are "active" in the IO group, during the execution of each
	IO opeation.
  all processes in the IO group have free receive/transmit buffers during the
	execution of each IO operation.
Some type of inquiry function might help.
  All processes in the group may execute a probe to find out if it is safe
  to call the IO operations (humm....)

Remarks: Sizes of objects ----------------------------------------
Restriction on the sizes of objects generated by the different processes
could be made. Possible examples:
   In fread()/fwrite() size and nitems must be identical in all processes.
   In printf()/scanf() the same format string appears in all processes.
These would make implementation faster and easier (and should be true in
single_io() mode IO.) It does make slightly irregular data distribution
harder.  (Why can't P always divide N :-) ). 

Remarks: Buffering  -----------------------------------------------
There are two types of buffering to consider, queueing multiple output 
requests before sending them and buffering the response to input requests
(as the normal ANSI-C buffered IO rouitines do.)  Buffering works
in single_io() mode, but it may not make sense in ordered_io() or
unordered_io() modes. Queueing input requests could make sense,
for non-blocking requests.

In our distributed graphics system we have some asynchronious commands,
which are not loosely synchronized. For these commnads each process gives
an independent stream to the server, which it must de-mux into a single
valid stream.  Any calls which change "state" in the graphics server are
loosely synchronized.  So the server dose not have to keep state for each
process, just insure a stream can be interupted from one process at a
valid place before switching to another processes stream.  I don't
think that type of IO is needed most.

Since the most heavy use of IO would be the ordered_io() / unordered_io(),
and that would seem to be large size chuncks of data, buffering multiple
IO requests may not be needed.  For terminal IO, the usual line buffering
should be OK.

What I'm really trying to figure out is, if setting up things with
no buffering/queueing specified has a negitive enough impact on
any applications to worry about. That boils down to how applications
would use IO routines.

Remarks: Higher level IO -----------------------------------------
Mapping r-dimensional data onto a logical s-dimensional process array is
one fundimental high level IO operation. The ordered_io() and unordered_io()
calls above allow a fixed dimension data file to be operated on by
a fixed dimension array.  I'm not sure how often dimensions above 3
are needed. That is, how often problems of higher dimension can't 
be well handled by projecting into 1-3 dimensions first (for the 
data/process distribution).

If the size of the array was changed, a single_io() read of the file
could be done, disgarding items not needed. For non-parallel disks
(from a host or other single source) on an  MIMD machine, I don't think
this posses a serious problem. 

For high preformance parallel disks, or disks distributed on a network,
that could be unacceptable. The only way to support that type of
IO would be to define the process array and the data file and let
the system sort it out - as well as choose the files internal distribution.
(As in Hart's system propoasal).

The EXPRESS model is successful and should be considered.

Remarks: Multiple process IO -------------------------------------
What if anything to say about multiple processes operating on files?

limit the total number of open files? (eg System Constant  12)
limit any file to be open by only one group for writing, for defined
	behavior?
allow any number of groups to open the same file for reading?
	(requires the server or group to keep state for each group).

Remarks: The three buffering modes in ANSI-C. ---------------------
The three buffering modes are:
	unbuffered
	block buffered
	line buffered
by default:
	stdout is line buffered
	stderr is unbuffered
	all others are block buffered
The 
	setbuf(FILE *stream, char *buf) 
call can set buffering on a newly opened stream to unbuffered
(buf == NULL) or block buffered, using a user allocated buffer. 
he buffer buf must be of a fixed system supplied size.
Useful buffering calls, not mentioned in POSIX:
	setbuffer(FILE *stream, char *buf, int size), allows user
		specified buffer sizes to be used.
	setlinebuf(FILE* stream)
		puts a stream into line buffered mode.
	setvbuf(FILE *stream, char *buf, int type, int size)
		sets any of the three modes (using type) and
		allows a user sized buffer to be used.
setvbuf is in ANSI-C and I suggest it be included.

The tie in of the buffer and sending a message could be made,
implying a message is sent when the buffer is filled. Giving
the user some control. I'm not sure I would like to state
something like this as a requirement.

Remarks: non-C Language IO -----------------------------------------
The functions above are clearly C oriented. Compatioble calls
could be available in other language bindings. No attempt to
specify native Fortran IO in MPI is suggested.

Remarks: The varargs forms -----------------------------------------
The varags form of the buffered output family (vprintf(), vfprintf() and
vsprintf() ) is in ANSI-C (it was not mentioned in POSIX). They could
be useful in avoiding extra memory copies. 

There are not corresponding varagr buffered input versions defined
in ANSI-C.	

I would suggest that they would not be as useful as a stride and iovec
version of the fread() and fwrite() calls.  I think iovev/stride versions
of fread() and fwrite() should might be the first extension to the ANSI-C
standard to consider including in Level-1 IO. The format for these would
be compatible with whatever the pt-2-pt calls become.

Remarks: Blocking, Non-Blocking and Synchronized modes ----------
Some of these calls could be used in all 3 forms.
The setvbuf() call already could give a lot of control over buffering.
It might be wise to only consider "extra" modes for the fread() and
fwrite(), which would correspond the modes finally in pt2pt.
(though if extra modes are implemented here they should be easy for
the implementer to add to calls such as fprintf() and fscanf() )

Remarks: The modes in the fopen call ----------------------------
Limiting the fopen modes to read only and write only would simplify
some things.

Remarks: The unbuffered IO calls. -------------------------------
Although the POSIX IEEE Std 1003.1-1988 C functions do include the
usual UNIX unbuffered IO functions, I think we could take the same
approach ANSI-C does and only specify the buffered IO functions.
Because of the fread() and fwrite() calls no functionality is lost.

If the unbuffered calls were included I would suggest the calls
readv() and writev() be included as well.
     int readv(fd, iov, iovcnt)
     int fd;
     struct iovec *iov;	/* pointer to an array of iovec structures */
     int iovcnt;	/* number of iovec structures */
where,
     struct iovec {
        caddr_t   iov_base;
        int       iov_len;
     };