Subject: Packing/unpacking routines

From: [email-masked] (Rolf Hempel)
Date: Thu, 27 Jan 1994 09:31:47 GMT

I like the proposal by Rusty and Bill for packing and unpackine routines
in MPI, and I strongly support the inclusion of those or similar
functions in the standard.

Some proponents of this functionality wanted to have those functions in
situations where they send a single buffer to many different locations.
Thus, the buffer has to be packed one time only. However, with the
proposed functions the buffer has to be packed for each destination
process separately, because the conversion happens in the packing
routine (arguments dest,comm). The application programmer cannot reuse
the once packed buffer, because he/she does not know whether the
target processors have the same data representation or not.

Personally I have no problem with this, but we should at least mention
that the functions cannot be used for reusing the packed buffer for
many sends.

The main reason for including the functions in the standard will be
to ease code migration from systems like PARMACS or PVM to MPI, and I
think that's a good enough reason.

- Rolf

---

From: Rusty Lusk <[email-masked]>
Date: Thu, 27 Jan 1994 10:07:12 -0600

| Some proponents of this functionality wanted to have those functions in
| situations where they send a single buffer to many different locations.
| Thus, the buffer has to be packed one time only. However, with the
| proposed functions the buffer has to be packed for each destination
| process separately, because the conversion happens in the packing
| routine (arguments dest,comm). The application programmer cannot reuse
| the once packed buffer, because he/she does not know whether the
| target processors have the same data representation or not.
| 
| Personally I have no problem with this, but we should at least mention
| that the functions cannot be used for reusing the packed buffer for
| many sends.
| 

The idea of making packing process-specific is to be able to avoid
translations in the cases were you know that none will be needed because MPI
knows something about the other process, i.e., it knows precisely what has to
be done (byte-swapping instead of full xdr conversion or whatever).

Maybe we can solve this problem.  Suppose that if you used MPI_ANY_PROC as the
destination or source, MPI_PACK and MPI_UNPACK would then convert to a
canonical form (like XDR) that could be used at any other process.  That is,
if you wanted to re-use the packed buffer, you would say so and potentially
pay the price of a more general packing and unpacking procedure, but
everything would be safe, and maximally efficient when you specified the
process. 

Rusty & Bill

---

From: <[email-masked]>
Date: Thu, 27 Jan 94 10:02:22 -0800

Both Rolf and Tony make good points. I think Tony's comment that the
communicator can make it all work is perhaps a little too restrictive of the
implementation space. I propose we consider the following two restrictions:

1. The first call to pack or unpack for a new buffer area must have
size_so_far equal to zero. That way, the implementation can be guaranteed a
place to put a single generic representation descriptor if it wants to.

2. The calls to unpack for a particular buffer must be made with the same
datatypes and counts as the calls to pack that built the buffer. For instance,
if you pack an int and a bunch of floats this would be correct:

	pack the int
	pack the floats

	unpack the int
	unpack the floats

while this would be erroneous:

	pack the int and the floats

	unpack the int
	unpack the floats.

This way the implementation could put a representation descriptor in for each
pack call but wouldn't have to put in a marker for every element.

I like the first restriction because it doesn't constrict the application
programmer in any meaningful way. I don't know that the second restriction is
so good, but it does place a pretty powerful constraint on the use of pack and
unpack that might keep some implementations out of trouble.

As a separate issue, I would avoid the use of the <inout> size_so_far by
splitting it into an <in> start and <out> finish. Doesn't change the code
much, just adds one extra parameter to the list. A reasonable restriction on
the implementation would be that the programmer be allowed to put the same
variable in both positions. Not an issue in the C binding, but must be
mentioned for the FORTRAN binding.

	MPI_PACK( ... , 0, &size_so_far);
	MPI_PACK( ... , size_so_far, &size_so_far);
	MPI_PACK( ... , size_so_far, &size_so_far);

Paul

---

From: Rusty Lusk <[email-masked]>
Date: Thu, 27 Jan 1994 17:00:29 -0600

| 1. The first call to pack or unpack for a new buffer area must have
| size_so_far equal to zero. That way, the implementation can be guaranteed a
| place to put a single generic representation descriptor if it wants to.
| 

Agreed.

| 2. The calls to unpack for a particular buffer must be made with the same
| datatypes and counts as the calls to pack that built the buffer. 

I am not too keen on this one.  It seems like a restriction on the programmer,
who might find that he wants normal packing but incremental unpacking, and
the benefits are not clear to me.

| As a separate issue, I would avoid the use of the <inout> size_so_far by
| splitting it into an <in> start and <out> finish. Doesn't change the code
| much, just adds one extra parameter to the list. A reasonable restriction on
| the implementation would be that the programmer be allowed to put the same
| variable in both positions. Not an issue in the C binding, but must be
| mentioned for the FORTRAN binding.
| 

This seems to me to be going too far in order to avoid inouts.  Here, the
inout variable doesn't come from trying to overload some parameter but because
the caller is holding state for the callee, which makes an inout parameter the
natural thing.  I don't care very much, however.

- Rusty