Subject: Restricted contexts

From: [email-masked] (James Cownie)
Date: Tue, 2 Feb 93 15:58:57 GMT

A view on contexts
==================

At the last meeting in Dallas there was a strong feeling that some
form of "context" was required in MPI. The major reason for this is
the desire to separate communications on behalf of (independently
compiled) library routines from those performed by the user's
application. 

The use of specific message tags for this is discounted because it
would require tag registration both by the library and the user code.
This intrusion into the user's constitutional right to use whatever
tags she sees fit is viewed as unacceptable, hence the requirement to
add what is essentially just another tag field. [Of course here in
Britain we don't have a written constitution, and the government just
does what it likes...]

In fact adding a further tag field (called context) does not remove
the registration requirement, though it does mean that registration is
only required by library routines. (In effect a default context is
reserved by the allocator for the user, and she need not concern
herself about it.)

I should like to propose a different view of what a context is. 

Suppose we think of each context as a separate communications end
point in the process (and intend to implement it this way too !).

This has various implications :--

1) A context should be created before it can be used. (This has the
   effect of allocating a new message queue inside the MPI
   implementation). 

2) The number of contexts will be limited (you might expect < 256)
   [ I don't see this as a problem, after all you still have all of
   the tags available in each context, and how many nested libraries
   do you expect to use ? ]

3) A context should be deleted when use of it is complete.
   (So that the system can release the queue. This also gives a good
   point at which to report errors such as unreceived messages still
   on the queue).

4) Creation of a context can be a local operation. (Since the context
   is probably just an index into a local table. cf. unix file
   descriptors) [Though this may not be true when collective
   operations filtered by context are taking place, in this case one 
   will want the same context in all members of the collective group,
   perhaps some contexts need to be reserved for allocation in group.]

5) Context values can be passed in messages without needing
   translation or registration.
   (This is important for maintaining scalability. The context
   information need only be passed to those processes which require
   it. [Though there's an unaddressed bootstrap problem here...])

6) It is still possible to have a context lookup/allocation scheme if
   necessary. 

7) It is possible for the user to avoid most of the tag matching cost
   (as has been requested by Chuck Simmons), all she (the user, not
   Chuck!) has  to do is allocate contexts and target messages at
   them. Within a particular context all receives can have tag
   MATCH_ANY, and no queue searching is required.
   Since the context selection is fast (because we have restricted the
   number of contexts so that it is sensible to implement the
   selection as an array access) this improves the achieved
   performance. 

Given this style, then the implementation is extremely easy (and will
be fast). One simply has an array of message lists, the context number
being the index into the array. All of the operations (such as tag
matching) can then use the existing code. This will (in general)
produce an implemetation which runs faster than having a single queue
of messages and matching on both tag and context, since the message
queues will be shorter, and only contain relevant buffers. [Contrast
with the case of additional code to match contexts on one message list
which would slow down all code, whether or not contexts are being used !]

The model of communication is that the context is a part of the TARGET
address, both for collective and diadic communications. In effect the
end point for communication is only relevant at the receiver.

[Is this what we want ? The only place it seems to make a difference
is in the tests for completion of non-blocking sends. If the user has
queued many of these on the single outgoing queue, then when the
library is searching for completed ones it has to skip them.  If we
have many outgoing queues this is not necessary. It rather depends on
the way we test for completion of non-blocking sends, which has yet to
be determined...]

Functions might look like

typedef ... MPI_CONTEXT;

MPI_CONTEXT MPI_NEWCONTEXT(MPI_GROUP group);
Allocate a new context for use within the given group (may be
MPI_NO_GROUP, in which case the allocation is entirely local). Returns
MPI_NO_CONTEXTS if there are no suitable free contexts left. 
If the group is not MPI_NO_GROUP, then this is a barrier
synchronisation of all the processes in the group, and the result of
the function is the same in all of the participating processes.

int MPI_FREECONTEXT(const MPI_CONTEXT context);
Free the given context, returns 
MPI_SUCCESS     if the context has been freed
MPI_NOTCONTEXT  if the context is not valid
MPI_BUSY        if the context still has outstanding messages

(I'm not sure if we need this last one, but the model of unix file
descriptors suggests it might be useful...)

int MPI_MOVECONTEXT(const MPI_CONTEXT to, const MPI_CONTEXT from);
Rename the context from to be the context to. (cf dup2). This is a
simple renaming operation. 
MPI_SUCCESS     if the context has been renamed
MPI_NOTCONTEXT  if the from context is not valid
MPI_BUSY        if the to context is already allocated

One of the arguments to the send routines (both diadic and collective)
would then be an MPI_CONTEXT to which the message will be sent.

Feedback please.

Although this is a long message I'm not offering beer. [The pound is
now down to $1.45 and still falling !]

-- Jim
James Cownie 
Meiko Limited			Meiko Inc.
650 Aztec West			Reservoir Place
Bristol BS12 4SD		1601 Trapelo Road
England				Waltham
				MA 02154

Phone : +44 454 616171		+1 617 890 7676
FAX   : +44 454 618188		+1 617 890 5042
E-Mail: [email-masked]   or    [email-masked]

---

From: Tony Skjellum <[email-masked]>
Date: Tue, 2 Feb 93 12:44:09 CST

Addendum to my previous comments.  I do not think it is a bad idea for
vendors to provide limited, extremely fast registry mechanisms, like
the CM-5's 8 hardware contexts, or whatever Meiko might have in mind
along the same lines.  These should be viewed as low-level services that
can be incorporated into standard interfaces, increasing their performance,
when they can be used.  Active messages, for instance, while they are
lower-level than the MPI ideas, should not be discarded by vendors.  They
may be a very good way for strong, MPI-upward-compatible, message systems
to do a good job with hardware.  We must provide for a way for vendors
to extract performance from their hardware, within the context of MPI,
or else they will continue to gain commercial advantage from ignoring MPI,
rather than supporting it fully.

- Tony