Subject: MPI_PACK, MPI_UNPACK, MPI_PACK_SIZE, MPI_UNPACK_SIZE

From: Rusty Lusk <[email-masked]>
Date: Wed, 26 Jan 1994 13:22:16 -0600

(Apologies if this is your second copy; a glitch may have attacked the
previous posting)

Here is a concrete proposal for explicit and incremental packing and unpacking
of buffers, that some of us talked about at the European meeting.  It was
generated in response to a strong request from the users.  If it looks
reasonable to you, I will try to flesh it out more for the draft to be
presented at the Knoxville meeting.  It probably would be added to the pt2pt
chapter.  Comments explicitly solicited.

Rusty & Bill

Proposal:  Explicit and incremental packing and unpacking of MPI buffers

At the recent European Workshop, several attendees expressed the need for
incremental packing and unpacking of MPI buffers, a feature currently
available in PVM.  The idea is to provide for transferring data to/from memory
described by an MPI datatype from/to a contiguous area that is to be sent/has
been received with type MPI_BYTE. In our original discussions, we recommended
essentially that users always allow MPI to do packing and unpacking
automatically in in the course of moving data to and from user space.  This
allows for optimization by aggressive implementations and special hardware.
Nonetheless, there are reasons for adding explicit packing and unpacking
routines to MPI:

1.  MPI currently does not allow *incremental* packing and unpacking.  The
incremental pack is for simplicity (to avoid creating several complex MPI
datatypes, when efficiency is not important) or for situations where the
contiguous buffer will be reused.  The incremental unpack definitely adds
functionality.  For example when receiving an integer n together with n
floats, one might want to know n before allocating precisely the right amount
of space for the floats.  

2.  The users want it, and it will make translation of existing PVM programs
particularly straightforward.

3.  As pointed out by Nathan Doss et al., there are times when one wants to be
able to reuse the packed version of the buffer.  This will make implementation
of the collective operations, for example, easier to do on top of pt2pt.

4.  The essential functionality is likely to be present in an implementation
anyway, and might as well be made visible to the user.

The parameters of MPI_PACK and MPI_UNPACK arise out of the need to identify
the memory area described by the datatype (addr,datatype,count), the
contiguous memory to packed into or unpacked from (buffer, maxsize), the other
process involved (communicator, rank) in order to be able to do translations if
necessary, and an inout variable (size_so_far) needed to hold state across
calls, to support incrementality.  On the unpack we also need to be told how
many items were there (actualcount).

________________________________________________________________________

MPI_PACK(addr,datatype,count,dest,comm,buffer,maxsize,size_so_far)

The inout variable size_so_far represents a displacement into the buffer that
has been used so far.  MPI_PACK takes care of supplying pad bytes where
needed.  Other parameters are as described above.

________________________________________________________________________

Example of use  (in C):  Puts in 

{
    int size_so_far;
    int numfloats_a;
    int numfloats_b;
    float a[100];
    float b[100];
    char buffer[100000];  /* better solution below, using MPI_PACK_SIZE */

    dest	= 2;
    numfloats_a	= 25;
    numfloats_b	= 17;

    size_so_far	= 0;
    MPI_PACK(numfloats_a, MPI_INT, 1, dest, MPI_COMM_WORLD,
	     buffer, 100000, &size_so_far);
    MPI_PACK(a, MPI_FLOAT, numfloats_a, dest, MPI_COMM_WORLD,
	     buffer, 100000, &size_so_far);
    MPI_PACK(numfloats_b, MPI_INT, 1, dest, MPI_COMM_WORLD,
	     buffer, 100000, &size_so_far);
    MPI_PACK(b, MPI_FLOAT, numfloats_b, dest, MPI_COMM_WORLD,
	     buffer, 100000, &size_so_far);
    MPI_SEND(buffer, size_so_far, MPI_BYTE, dest, tag, MPI_COMM_WORLD);
}

________________________________________________________________________

MPI_UNPACK(addr,datatype,count,source,comm,buffer,maxsize,size_so_far,actcount)

moves data from buffer into the area designated by (addr,datatype,count).  
Actcount has the number of items actually moved.

________________________________________________________________________

 Example:

(unpack above message)

Instead of using a "real big" scratch area as above, one should be able to
find out how much space a message described as MPI_BYTE would take up if it
were to unpacked, or how much space a message described by an MPI buffer would
take up if it were to be packed.  These routines are very similar to the
above, but don't move data.  Therefore they don't need the two memory location
parameters.  The size_so_far parameter is inout as above, so they can be
called incrementally.

________________________________________________________________________

MPI_PACK_SIZE(datatype,count,dest,comm,size_so_far)

MPI_UNPACK_SIZE(datatype,count,source,comm,size_so_far)

________________________________________________________________________

 Example: pack and unpack into dynamically allocated memory.

This still needs a little more fleshing out, but I hope the idea is clear.