Subject: my comments on Lyndon's comments on PropV

From: Tony Skjellum <[email-masked]>
Date: Sun, 21 Mar 93 13:00:37 CST

%**** Step 2.  My comments next to Lyndon's comments, critique of Rik's
%**** proposal V.
%****
%**** I have added my further remarks, with %****
%**** - Tony

General points
--------------

1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am
offering if you wish) should attempt to construct a more transparent
presentation before circulation to whole committee, for the
convenience of committee members.
%****
%**** I felt that things appear twice, because of summary (good)
%**** and because of implementation notes at end (confusing)
%****
 
2) I'm not a fan of much of this proposal, although I do indeed like
some of the ideas which it introduces. [On the other hand, I'm not a
great fan of all of the proposal which I wrote. I shall mail self
criticism of my proposal, and may have to write amended or alternative
proposal :-)]
%****
%**** Please be more specific.  I am having a hard time understanding
%**** why you really don't like it, Lyndon.  If the process model
%**** were a little less static, and servers were permitted (though
%**** hopefully bounded in cost), I think we would have an excellent
%**** proposal.
%****
3) I really like the way in which groups are something like ``frames''
in which contexts are created. This is conceptually much neater than
duplication of groups.
%****
%**** In practice, group subsetting will require groups to be copied,
%**** otherwise, subgroups will unfairly be penalized by the size
%**** of their ancestor.
%****
4) I like the idea of pushing information into the group structure. I
have a few qualms with the proposed details --- see specific points.
%****
%**** I have more confidence about this idea, and could demonstrate
%**** by June/July time-frame in Zipcode.
%****
5) See ``Writing a server in the point-to-point layer of MPI in four
easy steps'' at the foot of the message.
%****
%**** This seems like a nice thing.
%****
Specific points
---------------

Dealt with as LaTeX comments to body of text, appearing in the form

%[Lyndon]
% text of point

for your navigational convenience. These are quite detailed.

----------------------------------------------------------------------

\documentstyle{report}
\begin{document}
\title{``Proposal V'' for MPI Communication Context Subcommittee}
\author{Rik~Littlefield}
\date{March 1993}
\maketitle
%======================================================================%
% BEGIN "Proposal V"
% Rik Littlefield
% March 1993
%
\chapter{Proposal V}

\section{Summary}

\begin{itemize}

\item
    Group ID's have local scope, not global.  Groups are
    represented by descriptors of indefinite size and opaque
    structure, managed by MPI.  Group descriptors can be
    passed between processes by MPI routines that translate
    them as necessary.  

    (This satisfies the same needs as global group ID's, but
    its performance implications are easier to understand and
    control.)

%[Lyndon] 
% I support the approach whereby group descriptors are local
% objects. They could be pointers to structures, or indices
% into tables thereof. We let the implementation consider that.
%
% One difficulty arises as group descriptors can only be passed
% from process P to process Q if both P and Q members of some
% group G since the communication presumably must use a context
% known to both P and Q. Imagine that P is member of F and Q is not
% member of F; that Q is member of H and P is not member of H; that
% both P and Q are member of G. Let M be abritrary message data.
% 
% Initially -
% P can send F to Q, and Q can receive F from P, in a context of G.
% Q can send H to P, and P can receive H from Q, in a context of G.
% Thereafter -
% P can allocate a context C in F.
% P can send C to Q, and Q can receive C in the default context of H.
% Q can allocate a context D in H.
% Q can send D to P, and P can receive D, in the default context of F.
% Thereafter -
% P as member of F, and Q as member of H, can communicate using
% wildcard pid and tag by use of contexts C and D.
%
% Okay, this is possible, but it is messy :-)
%****
%**** Alternatives, Lyndon?
%****
\item
    Arbitrary process-local info can be cached in group
    descriptors.

    (This allows collective operations to run quickly after
    the first execution in each group, without complicating
    the calling sequences.)

%[Lyndon]
% I rather like this idea.
%**** Me too.
\item
    There are restrictions that permit groups to be layered
    on top of pt-pt.

\item
    Pt-pt communications use only TID, context, and tag, and
    are specified to be fast.

\item
    Group control operations (forming, disbanding, and
    translating between rank and TID) are NOT specified to be
    fast.

\end{itemize}

\section{Detailed Proposal}

\begin{itemize}

\item
  Pt-pt uses only ``TID'', ``context'', and ``message tag''.  TID's, 
  contexts, and message tags have global scope; they can be
  copied between processes as integers.  TID's and contexts are 
  opaque; their values are managed by MPI.  Message tags are
  controlled entirely by the application.

%[Lyndon]
% You probably ought to say that context and TID are integer with 
% opaque values.
%**** 
%**** 1) It is not obvious that TIDs should be restricted to 32 bits.
%**** 2) It is not obvious that contexts will be 32 bits (eg, 16 bits).
%****      I favor a whole word for a context, despite other limits,
%*****      just to make things simpler.
%****
%**** Internet addresses are going to get augmented from 32 to ??? bits
%**** is it reasonable to assume that certain MPI implementations might
%**** incorporate such internet addresses as TIDs (in future),
%****
%**** Opacity is partially violated if we say how big the data type is???
\item
  Pt-pt communication is specified to be fast in all cases.
  (E.g., MPI must initialize all processes such that any
  required translation of the TID is faster than the fastest
  pt-pt communication call.)

%[Lyndon]
% How do you imagine this to be acheived, considering that TIDs
% are global entities?
% I guess that you are thinking a TID is a (processor_number,
% process_number) pair of bit fields, a bit like one sees in NX and RK,
% and that network interface hardware will route based on the
% processor_number. 
%
% In another approach a TID is a process local entity just like the
% group descriptor. This satisfies efficiency when the above scheme
% is not applicable, for example in a workstation network.
%****
%**** where does this get us???
%**** Remember, we have to choose on some things, so we can have something
%**** to present in Dallas.  Is there an important difference here?
%****
%**** TIDs are global entities.  Is structure assumed to be global;
%**** in a truly opaque system, some TID component would have to be
%**** fixed, but the rest could vary structurally...
%****
\item
  A group is represented by a ``group descriptor'', of indefinite
  size and opaque structure, managed by MPI.  The group
  descriptor can be referenced via a ``process-local group ID'',
  which is typically just a pointer to the group descriptor.
  (Group ID's with global scope do not exist.)

%[Lyndon]
% I think I see, it is the context identifier which has global scope.
% Now, this really is just getting on the way toward the proposal
% that I really wish I had written for the subcommittee. I will flame
% myself!
%****
%**** Yes, contexts are global; group identifiers are just pointers
%**** typically, to data structures, describing
%****
%****		1) a groups context
%****		2) group members and their ranks (mappings, inverses,
%****				cached, hashed, unscalably stored, etc)
%****		3) TID-to-rank map and inverse (see possibilities in 2)
%****		4) A set of fixed global operations, accepted as standard,
%****			an accessible in O(1) time.  Possibly, each
%****			such operation should be a method, so that
%****			a parameter block can be passed with it.  Zipcode
%****			supports the Method type to do this.
%****              This is effectively a cache for some parts of item #5
%****              ...
%****		5) An AVL or similar tree of extensible operations.
%****		   New operations are registerable by the user.  These
%****		   tags are unique within a group, a specify an operation
%****		   i) pre-defined by MPI (in which case it can be cached
%****			in 4
%****		   ii) alternative operations (even if they do something
%****			standard, that are wanted to be accessed by
%****				name)  This name is group unique.
%****
%****		    A mechanism for DO_METHOD_FROM_GROUP(name,....)
%****               or GET_METHOD_FROM_GROUP(name,...)
%****		    and SET_METHOD_IN_GROUP(name,...) are clearly needed.
%****
  Group descriptors can be copied between arbitrary
  processes via MPI-provided routines that translate them
  to and from machine- and process-independent form.  A
  process receiving a group descriptor does not become a member
  of the group, but does become able to obtain information about
  the group (e.g., to translate between rank and TID).

%[Lyndon]
% Also crucially, to obtain and use the default context identifier
% of the received group descriptor.
%**** Yes, that is included, I believe, in concept.
%****
\item
  Arbitrary information can be cached in a group descriptor.
  This is, MPI routines are provided that allow any routine to
  augment a group descriptor with arbitrary information,
  identified by a key value, that subsequently can be retrieved
  by any routine having access to the descriptor.  Cached
  information is not visible to other processes and is stripped
  when a group descriptor is sent to another process.  Retrieval
  of cached information is specified to be fast (significantly 
  cheaper than a pt-pt communication call).
%[Lyndon]
% I like the general idea, but I'm nervous about two things:
% (a) implied associativity of group descriptor cache - this
%     will potentially be time expensive in implementation.
% (b) there is no method proposed for abritration of keys
%     between independently written modules, so we are 
%     in the same problem regime as just having message tag 
%     and no message context.
%     However, key's are local, so presumably you would find 
%     it acceptable to add a key registration service?
%****
%**** Stripping is extremely controversial aspect, and arbitrary.
%**** If the recipient has the methods with the same name, then
%**** a new rendezvous could be accomplished at the far end

\item  
  Group descriptors contain enough information to
  asynchronously translate between (group,rank) and TID for
  any member of the group, by any process holding a descriptor
  for the group.  MPI routines are provided to do these
  translations.  Translation is not specified to be fast.

%[Lyndon]
% How do you imagine that this will be done? 
% (a) Perhaps an array of TIDs which is just indexed on rank? Then
%     where is the case for not using directly rank.
% (b) Perhaps a hashing function? Then the case for not using rank
%     directly is marginal.
% (c) Perhaps generating a request to a service process? In which
%     case you admit here that a service process exists, which must
%     be propogated throughout the proposal and changes one of your
%     fundamental objectives.
% (d) Something else? Do tell!
%****
%**** Yes, these are all options.   Fastness seems to be an important
%**** issue.  If translation is very expensive, none of the "good"
%**** features will be used.
\item
  At creation, each group is assigned a globally unique
  ``default group context'' which is kept in the group
  descriptor and can be quickly extracted.  This extraction
  is specified to be fast (comparable to a procedure call).

\item
  The default group context can be used for pt-pt
  communication by any module operating within the group, but
  only for exactly paired send/receive with exact match on
  TID, context, and message tag.  Call this the
  ``paired-exact-match constraint''.  This constraint allows
  independent modules to use the same context without
  coordinating their tag values.  (Assumes: tight sequencing
  constraints for both blocking and non-blocking comms in
  pt-pt MPI.)

%[Lyndon]
% I understand what you want the paired-exact-match for. This
% might appear as pragmatics and advice to module writers. I
% think you should be firmer about sequencing constraints
% for point-to-point in MPI that this requires, to be
% sure that the constraint is not too large. 
%**** Again, I think this should be eliminated, and all references
%**** to this idea should be expunged.  It denies the context's
%**** ability to manage messages.
\item
  A modules that does not meet the paired-exact-match constraint
  must use a different globally unique context value.  An MPI
  routine will exist to generate such a value and broadcast it to
  the group.  The cost of this operation is specified to be
  comparable to that of an efficient broadcast in the group,
  i.e. log(G) pt-pt startup costs for a group of G processes.
  [See Implementation note 1.]

%[Lyndon]
% Perhaps I am missing something here. Please help. This
% is what my mind is thinking.
% The synchronisation requirement means that all context
% allocations in a group G must be performed in an identical 
% order by all members of G. Then the  sequence number of the 
% allocation is unique among all allocations within G. 
% Therefore the duplet 
% (default context of G, allocation sequence number)
% is a globally unique identification of the allocated
% context. The sequence number can be replaced by any one-to-one 
% map of the sequence number, of course. So, according to your
% synchronisation constraint, context generation can be ``free''.
%**** I agree that context allocation has to be done in sequence.
%**** That is why I am in favor of providing calls that allow
%**** groups to get numerous contexts at creation, and then
%****cooperatively, but potentially without further communication
%**** divide them(as they build subgroups, for instance).
%****
%**** I see these as services to be used in building virtual topology
%**** features, which will then be more widely used by users of MPI.
%****  
\item
  Context values are a plentiful but exhaustible resource.  If
  further use is anticipated, a context may be cached;
  otherwise it should be returned.  (Note: the number of
  available contexts should be several times the number of
  groups that can be formed.)

\item
  When a group disbands, its group descriptor is closed and
  any cached information is released.  (The MPI
  group-disbanding routine does this by calling destructor
  routines that the application provided when the
  information was cached.)  Copies of the group descriptor
  must be explicitly released.  It is an error to make any
  reference to a descriptor of a disbanded group, except to
  release that descriptor.

\item
  All processes that are initially allocated belong to the
  INITIAL group.  (In a static process model, this means all
  processes.)  An MPI routine exists for obtaining a reference to
  the INITIAL group descriptor (i.e., a local group ID for
  INITIAL).

\item
  Groups are formed and disbanded by synchronous calls on all
  participating processes.  In the case of overlapping groups,
  all processes must form and disband groups in the same
  order.  [See Implementation Note 2.]

\item
  All group formation calls are treated as a partitioning of some
  group, INITIAL if none other.  The calls include a reference to
  the descriptor of the group being partitioned, the TID of the
  new group's ``root process'', the number of processes in the
  group, and an integer ``group tag'' provided by the application.
  The group tag must discriminate between overlapping groups that
  may be formed concurrently, say by multiple threads within
  a process.  [See Implementation Note 2.]

%[Lyndon]
% The group partition you propose is essentially no different to
% the partition by key which has already been discussed, except
% that the key can encapsulate both (root process, group tag).
% So perhaps partition by key was better in the first place?
%****
%**** Do we get anything by having the root process?
%****
\item
  Collective communication routines are called by all members of
  a group in the same order.

\item
  Blocking collective communication routines are passed only a
  reference to the group descriptor.  To communicate, they can
  either use the default group context or internally allocate a
  new context, with or without cacheing.

\item
  Non-blocking collective communication routines are passed a
  reference to the group descriptor, plus a ``data tag'' to
  discriminate between concurrent operations in the same
  group.  (Question: is sequencing enough to do this?)  An
  implementation is allowed but not required to impose
  synchronization of all cooperating processes upon entry to a
  non-blocking collective communication routine.

%[Lyndon]
% If the requirement that collective operations within a group G are
% done in the identical order by all members of G even when such
% operations are non-blocking, then the sequence number of the operation 
% is unique and sufficient for disambiguation.
%
% The permission to force synchronisation - i.e., blocking - in the
% implementation of a non-blocking routine seems to make the routine
% less than useful. I can see whay you are asking for this, in order
% that you can generate a context for the routine call. In fact Rik
% I don't think you need the constraint, as I pointed out cheaper
% context generation exists above, unless of course I am missing 
% something.
%****
%**** I think that non-blocking collcomm is moribund in MPI1 or
%**** else MPI1 is moribund. :-)
%****
\end{itemize}

\section{Advantages \& Disadvantages}

\subsection*{Advantages}

\begin{itemize}

\item
  This proposal is implementable with no servers and can be
  layered easily on existing systems.

%[Lyndon]
% I am not of the opinion that the absence of services is such a big
% deal. I do think that programs which can conveniently not use
% services should not be forced to, but programs which cannot
% conveniently not use services should be allowed to.
%**** Too many negatives here for me to parse :-)

\item
  Cacheing auxiliary information in group descriptors allows 
  collective operations to run at maximum speed after the first
  execution in each group.  (Assumes that sufficient context
  values are available to permit cacheing.)

%[Lyndon]
% If you agree that context allocation is ``free'', then you can
% delete the bracketed qualifier.
%****
%**** Context allocation need not be free provided it can be made cheap,
%**** or cheap enough.
%****
%**** If one knows one will need several, then a single call could
%**** provide such contexts, amortizing overhead.  This is likely when
%**** bulding grids (ie, virtual topologies) in Zipcode, so it is 
%**** true in existing practice.
%****
%**** One should recognize the need for layering virtual top. calls
%**** on top of these calls, then these calls may appear painful,
%**** but perhaps they would be less used. Some users will use the
%**** provided virtual topology calls, others will prefer their own.
%**** Both will have equal power (see also,separate note on layerability).
%****
%**** If getting N contexts is a send-and-receive, plus a reactive server,
%**** then this is reasonably light weight,provided that hundreds of
%**** messages, or global operations ensue thereafter.  We can know in
%**** advance how heavy weight the context server will be.
%****
%**** if an implemention can use some locations of remote memory, with
%**** fetch and add, or locks, to achieve contexts, then this is even
%**** cheaper, in principle.
%****
%**** Despite Jim's earlier insistence that context numbers be kept to
%**** 256 or so, I think that this number should be much larger, so that
%**** much less efort goes into returning contexts, and so on, except
%**** occasionally, by processes.  Otherwise, a new kind of overhead,
%**** get-rid-of-context-because-I-am-out ensues, or programs block
%**** until contexts become available, offering the possibility of 
%**** deadlocks.
%****
\item
  Speed of cached collective operations does not depend on speed
  of group operations such as formation or translation between
  rank and TID.

%[Lyndon]
% True, but the cache is going to get big as user's are going to store
% arrays of TIDs in it.
%****
%**** Unscalability (of a limited form) should be permitted/selectable
%**** by user, to use as much per-node memory as the user wants, to reduce
%**** communication.
%****
\item
  Requires explicit translation between (group,rank) and TID,
  which makes pt-pt performance predictable no matter how
  the functionality of groups gets extended.

%[Lyndon]
% This is only true because you have asserted that implementations
% must have the property that:
% `` Pt-pt communication is specified to be fast in all cases.
%  (E.g., MPI must initialize all processes such that any
%  required translation of the TID is faster than the fastest
%  pt-pt communication call.)''
% So the advantage is not that which you have quoted, it is that
% you have made this assertion.
%**** I see,but what he means here is that there is no unpredictable
%**** translation cost because we do not write (group,rank) in pt2pt
%**** calls.  So, there is some validity to the statement.
\item
  Communication both within and between groups seems conceptually
  straightforward.

%[Lyndon]
% This is a conjecture. I believe that conjecture to be false.
% I especially believe this in the case of communication between
% groups. The methods which are available for ``hooking up'' 
% allows are at least perverse. I guess that the user could make
% use of a service process, to make life easier in this hooking up,
% so whay not provide one.
%**** Yes, that is why I have one in Zipcode.  I wish Zipcode were
%**** on netlib today, so you could try it.  Well, we are writingthe
%**** manual, and working at it as fast as we can.
%
% A further point. It seems to me that ``seems'' means that it seems 
% to you. This is not the point. It is how it seems to a lesser
% wizard than yourself which is of importance here. I conjecture
% that the reverse statment is true when the person doing the seeming
% is changed to a lesser wizard.
%****
%**** I lost something here, but I agree with the sense.  The word
%**** seems is subjective,and should disappear from our discussions,
%**** as much as seems prudent, anyway :-)
\end{itemize}

\subsection*{Disadvantages}

\begin{itemize}

\item
  Requires explicit translation between (group,rank) and TID,
  which may be considered awkward.

%[Lyndon]
% It is true that (group,rank) must be translated to TID. I can
% assure you that this is considered both awkward and redundant.
%****
%**** Yes,awkward, because it is nice to escape the TID realm and
%**** work within the (albeit simple) abstraction of group,rank.
%**** When layering virtual topologies on this, it would be so nice
%**** to write them to a group,rank syntax, not enforcing TID mappings
%**** everywhere.
\item
  Communication between different groups may be considered
  awkward.

%[Lyndon]
% You bet! Please see below.
%**** Indeed.

\item
  No free-for-all group formation.  A process must know
  something about who its collaborators are going to be
  (minimally, the root of the group).

%[Lyndon]
% Please see comments above on group creation.

\item
  Requires explicit coherency of group descriptors, which is not
  convenient -- application must keep track of which processes
  have copies and what to do with them.

%[Lyndon]
% I think all of the proposals will have this problem.
%**** Yes, and I think that loosely synchronous operations can maintain
%**** coherency, in practice.  That is, no operations that modify the
%**** group descriptors (other than cached lookup info) are permitted,
%**** without loose synchronization.
%**** This is nasty in that is would prohibit sending descriptors to
%**** processes not part of the group, so it is a clear trade-off.
%**** Perhaps such send-to-non-group-member operations could stipulate
%**** that this group information is somehow ephemeral, and that they
%**** need to join a new group to keep useful information over time???
%****
\item 
  Static group model.  Processes can join and leave
  collaborations only with the cooperation of the other group
  members in forming larger or smaller groups.

\item
  No direct support for identifying the group from which a
  message came.  The sender cannot embed its group ID in its
  message, since group ID's are not globally meaningful.  One
  method to address this problem is to have the sender embed its
  group context as data in the message, then have the receiver
  use that value to select among group descriptors that it had
  already been sent.

%[Lyndon]
% Yup, the user can ``do it manually with a search''. If you want
% to invoke this argument then I can dispose of almost everything
% in MPI in a period of a few minutes - in fact Steven Zenith will
% do it faster - so I refute the validity of the argument and claim
% that the MPI interfce should transmit said information.
%****
%**** Yes, that is exactly what Zipcode was written to avoid.  The
%**** user wants help managing things like this!!!!
%****
%**** The search, if any, must be MPI-supported, and as efficient as
%**** possible (eg, AVL trees, hash, partial hash with exceptions).
%****
%
% Further, the receiver is likely to want to be able to ask which
% rank in the sender group the sender was. Oh dear, well I suppose
% you think that's okay because the sender can put its rank into
% the message. This is just being inconvenient to the user who
% wants to send an array of something (double complex?) and has
% to pack a rank in by copying or sending a pre-message or the
% buffer descriptor kind of thing.
%****
%**** This is why I remain a strong advocate of (group,rank)
%**** addresssing in pt2pt.
%****
\end{itemize}

\section{Comments \& Alternatives}

\begin{itemize}

\item
  The performance of group control functions is explicitly
  unspecified in order to provide performance insurance.  The
  intent is to encourage program designs whose performance won't
  be killed if the functionality of groups is expanded so as to
  be unscalable or require expensive services.

%[Lyndon]
% I don't think that the intent expressed in the second sentence
% is satisfied. For example - group control is allowed to become the
% dominant feature of application time complexity. 
%****
%**** I addressed this in my Step-1 remarks.  Please see that.
%****
\item
  Adding global group ID's would seriously interfere with
  layering this proposal on pt-pt.  The problem is not generating
  a unique ID, but using it.  If just holding a global group ID
  were to allow a process to asynchronously ask questions about
  the group (like translating between rank and TID), then a group
  information server of some sort would be required, and MPI
  pt-pt does not provide enough capability to construct such a
  beast.

%[Lyndon]
% It is not the global uniqueness of group identifiers which creates
% the problem. There are globally unique labels of groups in your
% proposal anyway - the value of the default context identifier.
% The problem is that of allowing query of group information when
% that information cannot be recorded in the local process/processor
% memory.
%
% You claim that point-to-point does not have enough capability to 
% construct an information server. Firstly I should ask you whether
% this is an artefact of the manner in which you have defined the
% point-to-point communication. Secondly I assert that your claim
% is false. I shall append a description of server implementation
% to the foot of this message.
%****
%**** Thank you. These points are both well taken (ie these two paragraphs)
\item
  The restriction that only paired-exact-match modules can run in
  the default group context could be relaxed to something like ``a
  module that violates the paired-exact-match constraint can run
  in the default group context if and only if it is the only
  module to run in that context''.  This seems too error-prone to
  be worth the trouble, since even the standard collective ops
  might be excluded.  (It would also conflict with Implementation
  Note 1.]

\item
  Group formation can be faster when more information is provided
  than just the group tag and root process.  E.g. if all members
  of the group can identify all other members of a P-member
  group, in rank order, then O(log P) time is sufficient; if only
  the root is known, O(P) time may be required.  This aspect
  should be considered by the groups subcommittee in evaluating
  scalability.

%[Lyndon]
% Yes, partition does appear to be O(P) whereas definition by ordererd
% list appears to be O(log(P)).
%**** Also,see what I wrote in my Step-1 comments.  
%**** I believe O(log(P)) is still possible.
%****
\end{itemize}

\section{Implementation Notes}

\begin{enumerate}

\item
   To generate and broadcast a new context value: Generate the
   context value without communication by concatenating a
   locally maintained unique value with the process number in
   some 0..P-1 global ordering scheme.  This assumes that
   context values can be significantly longer than log(P) bits
   for an application involving P processes.  If not, then a
   server may be required, in which case by specification it
   has to be fast (comparable to a pt-pt call).  There is no
   need to generate a new context for the broadcast since it
   can be coded to use the default group context by meeting the
   paired-exact-match constraint.)

%[Lyndon]
% Please see notes above on the subject of context generation.
%**** Please see my Step-1 comments.
\item
   The following is intended only as a sanity check on the claim
   that this proposal can be layered on MPI.  Improvements to
   improve efficiency or to use fewer contexts will be greatly
   appreciated.

   In addition to a default group context, each group descriptor
   contains a ``group partitioning context'' and a ``group
   disbanding context'' that are obtained and broadcast at the
   time the group is created.  In the case of the INITIAL group,
   this is done using paired-exact-match code and any context,
   immediately after initialization of the MPI pt-pt level.  (At
   that time, no application code will have had a chance to issue
   wildcard receives to mess things up.)

   Group partitioning is accomplished using pt-pt messages in the
   partitioning context of the current group (i.e., the one being
   partitioned), with message tag equal to the group tag provided
   by the application.  In the worst (least scalable) case, the
   root of the new group must wildcard the source TID.  (This
   violates the paired-exact-match constraint, which is why group
   formation must happen in a special context.)

   Group disbanding is done with pt-pt messages in the group
   disbanding context.  This keeps group control from being
   messed up no matter how the application uses wildcards.

\end{enumerate}

\section{Examples}

{\bf (To be provided)}

%
% END "Proposal V"
%======================================================================%
\end{document}

----------------------------------------------------------------------

Writing a server in the point-to-point layer of MPI in four easy steps
----------------------------------------------------------------------

1) Partition the INITIAL group into two groups. A singleton group,
SERVER, and a group CLIENT which contains all of the other processes.

2) The single process in SERVER group records its TID. 

3) The processes in INITIAL group allocate a context SERVICE which
they remember either in the group cache or static data or something.

4) Use a broadcast in INITIAL group with ``sender'' as the one process
which is also in SERVER group, and the ``receivers'' as the (many)
processes which are also in CLIENT group, in the SERVICE context, in
order to disseminate the TID of the server process.

[Fanfare] a server process is in place as is a dedicated context for
the purposes of messages required to implement the service.

[Observation] the mpi point-to-point initialisation can do this
automatically.

%**** Zipcode's postmaster general works in this way, more or less.
%**** - Tony
----------------------------------------------------------------------

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/