Subject: Again: data representation

From: [email-masked] (Rolf Hempel)
Date: Thu, 23 Jun 1994 08:36:29 +0200

I really don't want to bother you all with my problem of data
representation, but Tony's reply doesn't solve it. Therefore, I try
to explain the issue again.

Of course I know that MPI takes care of changing the data representation
between processors, if necessary. For this to work, however, MPI needs
a detailed specification of the the message contents (e.g. 4 integers,
followed by 10 doubles). In case the representations match, it is save
to send the message as a string of bytes with type MPI_BYTES (on a
machine with 4 bytes integers the above example would result in 96
bytes).

In a PARMACS application, the user creates the buffer himself.
Additionally, he specifies the contents of the buffer, as given in the
above example. If I translate this into MPI, I can save a lot of extra
work if I know that the message is going to a processor with the same
data representation. I just send the message as MPI_BYTES with the given
length. If the representations don't match, I either have to repack the
buffer segments using the pack/unpack routines, specifying for each
segment the corresponding MPI basic data type, or I have to build a
derived datatype for the whole message and then send the original
buffer with this type. I expect that both ways create a substantial
overhead, which should be avoided if the representations match. In the
first case, the message transmission requires two additional copies of
the whole message. I don't know how much time is spent in creating
the derived datatype for the second solution, but I hesitate to create
a derived datatype for every single message.

To summarize, it's not for reasons of functionality that I need to
know whether the data representations match, it's just for performance.
I hope that I could convince you that in terms of performance this
knowledge really does make a difference.

So, does anybody have an idea how I can get this piece of information?

- Rolf

---

From: Steve Otto <[email-masked]>
Date: Thu, 23 Jun 94 15:19:16 -0700

I'll have a go; Rolf raises some good issues.

>a detailed specification of the the message contents (e.g. 4 integers,

	yes, you should specify the type signature

>machine with 4 bytes integers the above example would result in 96

	yes, it is safe to do this, but you are circumventing
	the recommended procedure.  If the machine architectures are
	"matching" then the MPI implementation should be smart enough
	to know that representation conversion is not necessary
	and it won't do it.  But, you don't specify BYTE as the type!

	and, there are a few other issues...

>segment the corresponding MPI basic data type, or I have to build a

	OK, you want to do a direct translation from PARMACS, which has
	packing/unpacking.  And yes, if you use MPI's pack/unpack
	facilities, it is true that rep. conversion may "needlessly"
	occur since at pack time MPI does not necessarily know whether
	the architectures are matching or not.  But note that, if
	one is on a homogeneous system, MPI could easily avoid
	rep conversion, even for pack and unpack.  This could
	also be done on a group-by-group basis, since one 
	specifies a communicator
	with these calls, so if the process group is homogeneous, we
	could again make this optimization.

>a derived datatype for every single message.

	Yes, creating a derived datatype does entail some overhead,
	but note that it need not make additional copies of the data
	(that is, it need not assemble the data into a contiguous
	buffer of memory).

	So now, what about the overhead of constructing the derived
	datatype description itself (internal to MPI)?  OK, well,
	all you need to do is to use the datatype for multiple sends
	and recvs, and then you will have amortized the cost.  This
	should make the majority of applications happy.

>knowledge really does make a difference.

	yes, but the way in which one gets performance in MPI may
	be different than in PARMACS --- in MPI, the recommended
	procedure is to create derived datatypes one time, then use
	them for lots of communications.  Another optimization
	with a similar spirit is to use "persistent communication
	requests" (that is, a channel-like construct)

>So, does anybody have an idea how I can get this piece of information?

	--Steve Otto