From wade@cs.utk.edu Mon Jan 18 10:04:48 1993 Received: from THUD.CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15560; Mon, 18 Jan 93 10:04:48 -0500 Received: from LOCALHOST.cs.utk.edu by thud.cs.utk.edu with SMTP (5.61++/2.7c-UTK) id AA03584; Mon, 18 Jan 93 10:04:42 -0500 Message-Id: <9301181504.AA03584@thud.cs.utk.edu> To: mpi-context-archive@surfer.EPM.ORNL.GOV Cc: wade@cs.utk.edu Subject: Date: Mon, 18 Jan 93 10:04:42 EST From: Reed Wade test From walker@rios2.epm.ornl.gov Mon Jan 18 13:23:40 1993 Received: from rios2.EPM.ORNL.GOV by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19959; Mon, 18 Jan 93 13:23:40 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17128; Mon, 18 Jan 1993 13:23:36 -0500 Date: Mon, 18 Jan 1993 13:23:36 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181823.AA17128@rios2.epm.ornl.gov> To: mpi-context-archive@surfer.EPM.ORNL.GOV From root Fri Jan 15 15:33:35 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12164; Fri, 15 Jan 1993 15:33:34 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA07918; Fri, 15 Jan 93 15:33:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Fri, 15 Jan 93 15:29:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:29:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13112; Fri, 15 Jan 93 15:29:50 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13694; Fri, 15 Jan 1993 15:29:49 -0500 Date: Fri, 15 Jan 1993 15:29:49 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301152029.AA13694@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Status: RO I thought the discussion of communication contexts at the last Dallas meeting was lacking in focus, and I'm afraid the existing subcommittees may not deal with this area. If there are enough people interested perhaps there should be a separate communication context subcommittee. If you have any strong opinions on this please let me know. Also let me know if you would like to be on the communication context subcommittee if one is established. On the topic of communication contexts I think (at least) 4 approaches have been proposed. 1) Implicit communication contexts as in MPI1 controlled by push/pop. 2) Explicit registration as in Zipcode 3) Explicit communication contexts should be merged with explicit group contexts. Since groups cannot communicate with each other a new communication context can be created by creating a new group with the same members as the current group. 4) Chuck Simmons of Oracle has suggested that communication contexts are really a particular type of thread, and so can be handled using existing threads packages. If you have any ideas, or are strongly for or against any of the above approaches let mpi-comm know so we can initiate a discussion of this topic. David Walker From root Fri Jan 15 15:47:37 1993 Received: from antares.mcs.anl.gov by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16031; Fri, 15 Jan 1993 15:47:36 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03636; Fri, 15 Jan 93 14:47:35 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00709; Fri, 15 Jan 93 14:47:32 CST Message-Id: <9301152047.AA00709@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: walker@rios2.epm.ornl.gov Date: Fri, 15 Jan 93 14:47:31 CST From: Rusty Lusk Status: RO | | On the topic of communication contexts I think (at least) 4 approaches have | been proposed. | 1) Implicit communication contexts as in MPI1 controlled by push/pop. | 2) Explicit registration as in Zipcode | 3) Explicit communication contexts should be merged with explicit | group contexts. Since groups cannot communicate with each other | a new communication context can be created by creating a new group | with the same members as the current group. | 4) Chuck Simmons of Oracle has suggested that communication contexts | are really a particular type of thread, and so can be handled | using existing threads packages. | If you have any ideas, or are strongly for or against any of the above | approaches let mpi-comm know so we can initiate a discussion of this topic. I think that 1) contradicts our genral agreement to avoid state as much as possible. 4) is a problem on systems that don't support threads, and we have more-or-less agreed to be consistent with thread packages but not depend upon them. 3) should be discussed along with the converse: that contexts rather than groups should be the primary concept and groups defined in terms of contexts. David, perhaps you could create a mailing list on which this discussion can take place, parallel to the subcommittee mailing lists, so it doesn't have to go to the whole committee. Thanks. Rusty From root Fri Jan 15 16:14:41 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13265; Fri, 15 Jan 1993 16:14:39 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA08064; Fri, 15 Jan 93 16:14:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16784; Fri, 15 Jan 93 16:12:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 16:12:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16768; Fri, 15 Jan 93 16:12:01 -0500 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA21882; Fri, 15 Jan 93 15:11:54 -0600 Received: by mozart.convex.com (5.64/1.28) id AA01950; Fri, 15 Jan 93 15:11:17 -0600 Date: Fri, 15 Jan 93 15:11:17 -0600 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9301152111.AA01950@mozart.convex.com> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: lusk@antares.mcs.anl.gov, walker@rios2.epm.ornl.gov Status: RO Rusty Lusk writes: > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. I believe the default inclusion of everyone is a good idea, although creating a separate mailing list might be appropriate if some mpi-comm members wish to "unsubscribe". In the meantime, I will add one brief opinion: I felt that the discussions of contexts in the topology subgroup were quite sufficient, and it would probably be better to have the single subgroup which covers both groups and contexts, so that both will appear in the standard in a style which does not make them conflict with each other, and/or to avoid a proposal where groups and contexts would be totally redundant. I definitely agree that it is an important topic, and that the discussions last week in Dallas were far away from getting a proposal for groups and contexts that would be unanimously acceptable, so much work remains to be done. From root Fri Jan 15 17:03:11 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13294; Fri, 15 Jan 1993 17:03:06 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA08266; Fri, 15 Jan 93 17:03:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21190; Fri, 15 Jan 93 17:00:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 17:00:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21182; Fri, 15 Jan 93 17:00:54 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA20492; Fri, 15 Jan 93 22:00:50 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14823; Fri, 15 Jan 93 14:59:49 MST Date: Fri, 15 Jan 93 14:59:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Communication contexts Status: RO > | > | On the topic of communication contexts I think (at least) 4 approaches have > | been proposed. > | 1) Implicit communication contexts as in MPI1 controlled by push/pop. > | 2) Explicit registration as in Zipcode > | 3) Explicit communication contexts should be merged with explicit > | group contexts. Since groups cannot communicate with each other > | a new communication context can be created by creating a new group > | with the same members as the current group. > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > | If you have any ideas, or are strongly for or against any of the above > | approaches let mpi-comm know so we can initiate a discussion of this topic. > > I think that 1) contradicts our genral agreement to avoid state as much as > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather > than groups should be the primary concept and groups defined in terms of > contexts. > > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. > > Rusty I'm not sure, but I believe that Zipcode uses push/pop to control contexts (Tony, please correct me if I'm wrong). I'd like to add a 5th approach, basically the one proposed by Paul Pierce: 5) Contexts are used exclusively to insure that message collisions will not occur if independently developed sub-programs are combined. Contexts and groups are orthogonal. Contexts and threads are orthogonal. Each message has an associated context and tag. Message context is managed by library routines and is completely out of a user's control. Message tag is selected by the user. The method of managing message contexts is a separate issue (assuming we want contexts). Existing proposals are: 5a) Stack-based management (objected to due to hidden states). 5b) Explicit registration with user-defined "names" (probably requires some communication). 5c) Explicit registration by a central authority ("dollar bill" registration mentioned by Jim Cownie.) There is enough confusion about contexts and groups that we may want to keep this discussion open to everyone. If the concensus is to make a separate mailing list, please both of us to it. Thanks, Tom Henderson hender@fsl.noaa.gov Leslie Hart hart@fsl.noaa.gov From root Fri Jan 15 19:07:57 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13061; Fri, 15 Jan 1993 19:07:55 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA08466; Fri, 15 Jan 93 19:07:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27214; Fri, 15 Jan 93 19:05:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 19:05:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27206; Fri, 15 Jan 93 19:05:40 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00450; Fri, 15 Jan 93 18:05:36 CST Date: Fri, 15 Jan 93 18:05:36 CST From: Tony Skjellum Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu Subject: Re: Communication contexts Status: RO In Zipcode, contexts have a group-wide scope. Groups of processes can involve more than one context. For instance, if 10 processes are involved in several stages of a calculation, each stage might have a different notion of how to do messaging, but each would use all the processes. Groups are important, because they allow control over the scope of global operations. Contexts are important because they provide guarantees to sub-programs about restricted scope of messages. Contexts are added typing-like information, but which is controlled by the system, not ad hoc by each user program. When they determine to start messaging, sub-programs request a context from a "postmaster general" in Zipcode. Currently, this is a group-oriented request (loose synchronization) resulting in each group member getting the context, and other information. In a lower-level system, simpler tactics might be possible. The way we did it was supposed to avoid possibility of races or other problems like that. To build large-scale software, without globalization of the local use of messaging resources, and to provide scalable algorithms with efficient global operations, both groups and contexts are needed. The push/pop state is an implementation detail of little importance. -Tony From root Sat Jan 16 02:08:07 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA15898; Sat, 16 Jan 1993 02:08:06 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA09478; Sat, 16 Jan 93 02:08:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11696; Sat, 16 Jan 93 02:06:16 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 16 Jan 1993 02:06:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11688; Sat, 16 Jan 93 02:06:13 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA06036; Sat, 16 Jan 93 02:01:08 EST Date: Sat, 16 Jan 93 02:01:08 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301160701.AA06036@cs.wmich.edu> To: mpi-comm@cs.utk.edu Status: RO hi; It was my feeling that the concepts of group and context, that were clearly understood differently by different people at the start of the discussions, became somewhat better agreed on as the pt2pt discussion took place. Groups - used to define groups of processors for collective communication primitives and to allow mapping topologies onto processors. This allows a group of processors to be used in isolation to compute together. Contexts - As Tom (and Tony) express below, > From: hender@macaw.fsl.noaa.gov (Tom Henderson) > Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> > To: mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > I'd like to add a 5th approach, basically the one proposed by Paul Pierce: > > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. The only expression stated so far about the difference between tags and contexts I had gleaned was that context should now be wildcardable (IE no -1 for All contexts) while a receive ALL or some other MASK variant would be allowed on tags. > > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) Add: 5d) A context is not strictly managed, but use should follow suggested guidelines. This could allow static allocation of contexts when reasonable, but also support a context server (as the Postmaster General in Zipcode). Think of the static contexts or non-registrar provided contexts as reserved with respect to any registration system. This could be part of the MPI initialization call, if a registrar is included. The drawback here of course is the user could violate whatever guidelines were given. It seems to me most types of name registration or stack allocation schemes, that are not too restrictive and allows user code to interact, are likely to allow errant management of contexts as well. I expect that allowing 5d) would let strictly controlled context systems to be built over MPI. > From: Tony Skjellum > Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> > To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > In Zipcode, contexts have a group-wide scope. Groups of processes can Of course if we eliminate groups from the pt2pt call then contexts may not be explicitly limited to groups in MPI. (though a context server could be asked to provide a context relative to a group). > involve more than one context. For instance, if 10 processes are involved > in several stages of a calculation, each stage might have a different notion > of how to do messaging, but each would use all the processes. Groups > are important, because they allow control over the scope of global operations. > Contexts are important because they provide guarantees to sub-programs > about restricted scope of messages. Contexts are added typing-like > information, but which is controlled by the system, not ad hoc by each > user program. When they determine to start messaging, sub-programs > request a context from a "postmaster general" in Zipcode. Currently, this ... > To build large-scale software, without globalization of the local use of > messaging resources, and to provide scalable algorithms with efficient > global operations, both groups and contexts are needed. The push/pop > state is an implementation detail of little importance. I expect the use push/pop for context state would make implementation in some environments hard and I have some concern about requiring registration on very large systems. I do think a strictly controlled stack context system could be built over 5d). The question about context management relates to how high a level MPI should be at. I see it at a low level, that higher level systems can be built on. That is one reason for keeping context management "advisory". > > -Tony > I liked the zipcode documentation. I think it would be a good idea to make it available on netlib. john From root Sat Jan 16 06:08:50 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13121; Sat, 16 Jan 1993 06:08:44 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA09829; Sat, 16 Jan 93 06:07:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00620; Sat, 16 Jan 93 06:03:55 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 06:03:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00612; Sat, 16 Jan 93 06:03:37 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21711; Sat, 16 Jan 93 03:03:35 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01231; Sat, 16 Jan 93 03:08:56 PST Message-Id: <9301161108.AA01231@jewel.us.oracle.com> Date: Sat, 16 Jan 93 03:08:56 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: RE: Communication contexts Status: RO > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather Allow me to clarify my sentiments. The example of context usage given in the MPI draft documentation appears to explicitly assume the existance of threads, and here contexts appear to be being used as threads. > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. I could strongly support almost all of the above. [The part I wouldn't support is a "tag" as I will explain below.] We do our work on the nCUBE, which, as you may know, has a message passing interface essentially equivalent to that proposed by MPI. We ran into the exact problem quoted above: it was difficult to write libraries that we're guaranteed not to conflict in their usage of message types. The usage of the word "contexts" in the above is equivalent to the usage of the word "ports" amongst operating systems programmers. Because "port" is slightly less overloaded than "context", I prefer the word "port". Note that ports are sufficiently powerful that they subsume the concepts of "contexts", "pids", and "message types" or "tags" as used by MPI. >The only expression stated so far about the difference between tags and >contexts I had gleaned was that context should now be wildcardable (IE no >-1 for All contexts) while a receive ALL or some other MASK variant would >be allowed on tags. I agree that there is little difference between tags and contexts. This helps explain why ports are so good at subsuming both contexts and tags. > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) I don't particularly understand the above, probably because I haven't been listening in on enough of the discussion. I think I understand what you mean by stack-based management, and we do use the word "registration"... When using ports, stack-based management isn't an issue for the same reasons you don't manage pids and message types using stack based management. For example, when sending a message, instead of using the MPI push_context (contextp); send (dest, type, buf, buflen); pop_context (contextp); you use send (port, buf, buflen); In our usage, a "port" is simply a queue to which messages can be sent. The receiver can remove a message from the head of the queue. Wildcards are neither needed nor implemented, making ports more efficient. The efficiency arises from the fact that a process can have multiple ports. In MPI, you essentially implement one receive queue for each pid. If you want to use a wildcard to access messages out-of-order, you need to waste time scanning the queue. With multiple ports, however, you can set up multiple queues so that you never need to access anything other than the head of a queue. Sometimes ports do need to be "registered". In our terminology, registration is the process of publishing in a well-known location sufficient information about a "port" created by one process that another process can send messages to that port. This is a very high level action that is almost outside the scope of MPI. We use a nameserver for this purpose. (The nameserver is accessed via a well-known port, i.e. a port whose bit-pattern is a well-known constant.) [Do note that we require that the bit-pattern representing a port be allowed to be transmitted between to processes without the operating system needing to know that the bit-pattern represents a port. This means that accessing the nameserver is not required in order to use a port.] For groups, we have thought about extending the concept of a port to an array-port. The simple implementation of this allows each process in a group to create a port. These ports are all sent to a central location which creates an array of ports and broadcasts the array to all ports in the array. The complex memory efficient implementation may require processes of a group to be created in a special fashion. Basically, it allows the address of a port for a specific process in a group to be easily computed by another process given the port of the first process in the group and the index of the target process in the group. -- Chuck From owner-mpi-context@CS.UTK.EDU Tue Jan 19 08:54:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14013; Tue, 19 Jan 93 08:54:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13129; Tue, 19 Jan 93 08:53:41 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 19 Jan 1993 08:53:40 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Tue, 19 Jan 93 08:53:39 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16410; Tue, 19 Jan 1993 08:53:38 -0500 Date: Tue, 19 Jan 1993 08:53:38 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301191353.AA16410@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Chair Tony Skjellum of Mississippi State University has kindly agreed to chair the MPI subcommittee on contexts. Regards, David From owner-mpi-context@CS.UTK.EDU Tue Jan 19 11:37:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17976; Tue, 19 Jan 93 11:37:59 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21893; Tue, 19 Jan 93 11:37:36 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 19 Jan 1993 11:37:30 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21879; Tue, 19 Jan 93 11:37:21 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13277 (5.65c/IDA-1.4.4); Tue, 19 Jan 1993 11:36:49 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA27411; Tue, 19 Jan 93 16:36:38 GMT Date: Tue, 19 Jan 93 16:36:37 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301191636.AA27411@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05440; Tue, 19 Jan 93 16:35:18 GMT To: csimmons@us.oracle.com Cc: mpi-pt2pt@cs.utk.edu, mpi-context@cs.utk.edu In-Reply-To: Charles Simmons's message of Sat, 16 Jan 93 02:21:14 PST <9301161021.AA01226@jewel.us.oracle.com> Subject: mpi Content-Length: 3447 To: mpi-pt2pt@meiko.co.uk, mpi-context.@meiko.co.uk (Apologies to those who get it twice) Gentlepeople, Chuck Simmons has raised some interesting issues, with which I have a great deal of sympathy. The original messaging model which we implemented on our machines (and which we still support, and expect to continue to support) is in exactly the vein which Chuck is asking for (and has been used for an Oracle port !). In particular CSN supports 1) multiple end points for communication in a single process (know as "transports") 2) No tagging of messages 3) No system buffering, but the ability for users to queue multiple non-blocking receives on a transport, thus providing the buffering they require. 4) Send and receive by struct iovec. (As Chuck observes, the implementation ends up building an iovec (on the stack) for the simpler forms) 5) Both blocking and non-blocking tests for I/O completion. (our test actually has a timeout value). 6) The ability to pass transport addresses around the machine without the system being involved. 7) A standard name server service to associate textual names with transport addresses. If people are really interested then I can probably send the man pages. (Though this is not the main point of this note). HOWEVER (and this is one of the points) although this system had clean, specified semantics and a fast implementation (on our hardware), it hasn't helped to sell machines, and we have now produced an NX style interface as an alternative. I think that the problem with popularising such an interface among users is that it appears to do less for them than a model with implicit buffering, and is therefore harder to start to use. The fact that it allows them greater control and can ultimately produce higher performance is not their immediate concern and does not therefore count for much. Many FORTRAN programmers in particular to do not wish to be concerned with "system programming issues" like buffer management. I would certainly like MPI to be able to support such an interface (so that we can achieve the higher performance it offers, and also make MPI applicable in non-scientific application areas). (Though those who were present in Dallas will have noted that I wasn't trying to push such an approach there, mainly because I think it's a lost cause. NX style is what we have to live with...) (Second point coming up...) However it crosses my mind that there may be some potential for embedding such an interface within the MPI model. (This is a vague thought, but I'm throwing it out so others can think about it as well). Observe that 1) Marc's persistent communication descriptors seem to remove the tag matching requirement (though they're still rather vague !) 2) One way of viewing contexts would be to implement a context as a particular queue of messages (or in other words an entirely separate communication end point) within a process. (The implications of this for contexts would be a) contexts must be declared before use b) their number may be limited c) they should be freed after use ) The combination of the two things could then come somewhere near to what Chuck is asking for... -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 11:00:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21520; Tue, 2 Feb 93 11:00:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22663; Tue, 2 Feb 93 10:59:18 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 10:59:17 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22655; Tue, 2 Feb 93 10:59:11 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA28603 (5.65c/IDA-1.4.4 for ); Tue, 2 Feb 1993 10:59:07 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA22488; Tue, 2 Feb 93 15:58:57 GMT Date: Tue, 2 Feb 93 15:58:57 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302021558.AA22488@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03309; Tue, 2 Feb 93 15:57:21 GMT To: mpi-context@cs.utk.edu Subject: Restricted contexts Content-Length: 6135 A view on contexts ================== At the last meeting in Dallas there was a strong feeling that some form of "context" was required in MPI. The major reason for this is the desire to separate communications on behalf of (independently compiled) library routines from those performed by the user's application. The use of specific message tags for this is discounted because it would require tag registration both by the library and the user code. This intrusion into the user's constitutional right to use whatever tags she sees fit is viewed as unacceptable, hence the requirement to add what is essentially just another tag field. [Of course here in Britain we don't have a written constitution, and the government just does what it likes...] In fact adding a further tag field (called context) does not remove the registration requirement, though it does mean that registration is only required by library routines. (In effect a default context is reserved by the allocator for the user, and she need not concern herself about it.) I should like to propose a different view of what a context is. Suppose we think of each context as a separate communications end point in the process (and intend to implement it this way too !). This has various implications :-- 1) A context should be created before it can be used. (This has the effect of allocating a new message queue inside the MPI implementation). 2) The number of contexts will be limited (you might expect < 256) [ I don't see this as a problem, after all you still have all of the tags available in each context, and how many nested libraries do you expect to use ? ] 3) A context should be deleted when use of it is complete. (So that the system can release the queue. This also gives a good point at which to report errors such as unreceived messages still on the queue). 4) Creation of a context can be a local operation. (Since the context is probably just an index into a local table. cf. unix file descriptors) [Though this may not be true when collective operations filtered by context are taking place, in this case one will want the same context in all members of the collective group, perhaps some contexts need to be reserved for allocation in group.] 5) Context values can be passed in messages without needing translation or registration. (This is important for maintaining scalability. The context information need only be passed to those processes which require it. [Though there's an unaddressed bootstrap problem here...]) 6) It is still possible to have a context lookup/allocation scheme if necessary. 7) It is possible for the user to avoid most of the tag matching cost (as has been requested by Chuck Simmons), all she (the user, not Chuck!) has to do is allocate contexts and target messages at them. Within a particular context all receives can have tag MATCH_ANY, and no queue searching is required. Since the context selection is fast (because we have restricted the number of contexts so that it is sensible to implement the selection as an array access) this improves the achieved performance. Given this style, then the implementation is extremely easy (and will be fast). One simply has an array of message lists, the context number being the index into the array. All of the operations (such as tag matching) can then use the existing code. This will (in general) produce an implemetation which runs faster than having a single queue of messages and matching on both tag and context, since the message queues will be shorter, and only contain relevant buffers. [Contrast with the case of additional code to match contexts on one message list which would slow down all code, whether or not contexts are being used !] The model of communication is that the context is a part of the TARGET address, both for collective and diadic communications. In effect the end point for communication is only relevant at the receiver. [Is this what we want ? The only place it seems to make a difference is in the tests for completion of non-blocking sends. If the user has queued many of these on the single outgoing queue, then when the library is searching for completed ones it has to skip them. If we have many outgoing queues this is not necessary. It rather depends on the way we test for completion of non-blocking sends, which has yet to be determined...] Functions might look like typedef ... MPI_CONTEXT; MPI_CONTEXT MPI_NEWCONTEXT(MPI_GROUP group); Allocate a new context for use within the given group (may be MPI_NO_GROUP, in which case the allocation is entirely local). Returns MPI_NO_CONTEXTS if there are no suitable free contexts left. If the group is not MPI_NO_GROUP, then this is a barrier synchronisation of all the processes in the group, and the result of the function is the same in all of the participating processes. int MPI_FREECONTEXT(const MPI_CONTEXT context); Free the given context, returns MPI_SUCCESS if the context has been freed MPI_NOTCONTEXT if the context is not valid MPI_BUSY if the context still has outstanding messages (I'm not sure if we need this last one, but the model of unix file descriptors suggests it might be useful...) int MPI_MOVECONTEXT(const MPI_CONTEXT to, const MPI_CONTEXT from); Rename the context from to be the context to. (cf dup2). This is a simple renaming operation. MPI_SUCCESS if the context has been renamed MPI_NOTCONTEXT if the from context is not valid MPI_BUSY if the to context is already allocated One of the arguments to the send routines (both diadic and collective) would then be an MPI_CONTEXT to which the message will be sent. Feedback please. Although this is a long message I'm not offering beer. [The pound is now down to $1.45 and still falling !] -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 11:49:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22912; Tue, 2 Feb 93 11:49:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24957; Tue, 2 Feb 93 11:49:03 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 11:49:02 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24949; Tue, 2 Feb 93 11:48:58 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA28146; Tue, 2 Feb 93 16:48:54 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01526; Tue, 2 Feb 93 09:47:49 MST Date: Tue, 2 Feb 93 09:47:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302021647.AA01526@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: Re: Restricted contexts A few (probably dumb) questions about Jim's context proposal: > MPI_CONTEXT MPI_NEWCONTEXT(MPI_GROUP group); > Allocate a new context for use within the given group (may be > MPI_NO_GROUP, in which case the allocation is entirely local). Returns ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > MPI_NO_CONTEXTS if there are no suitable free contexts left. > If the group is not MPI_NO_GROUP, then this is a barrier > synchronisation of all the processes in the group, and the result of > the function is the same in all of the participating processes. In what situations would I want a context that is entirely local? Would I need to send this context to a process I intend to communicate with? I like the idea of being able to create a context within a group for scalability reasons. How do I know that two groups will not get the "same" new context returned from separate calls to MPI_NEWCONTEXT()? Is this even a problem? > One of the arguments to the send routines (both diadic and collective) > would then be an MPI_CONTEXT to which the message will be sent. Does this mean that MPI_CONTEXT is NOT a required argument to a receive routine? (Maybe I'm missing the whole point of this proposal! If so, please educate me. A pseudo-code examle might be very helpful.) > Although this is a long message I'm not offering beer. [The pound is > now down to $1.45 and still falling !] This whole beer thing is getting pretty scary. I think I'll keep all my messages short! :) Tom Henderson hender@fsl.noaa.gov From owner-mpi-context@CS.UTK.EDU Tue Feb 2 12:15:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23454; Tue, 2 Feb 93 12:15:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26210; Tue, 2 Feb 93 12:15:20 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 12:15:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26192; Tue, 2 Feb 93 12:15:16 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03830; Tue, 2 Feb 93 11:14:40 CST Date: Tue, 2 Feb 93 11:14:40 CST From: Tony Skjellum Message-Id: <9302021714.AA03830@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, jim@meiko.co.uk Subject: Re: Restricted contexts Ladies/Gentlemen: The notion of context implementation for efficiency should not drive it to be a highly limited field. It is true than if there are only a few contexts, then they could be moved to hardware support. I think that large software will have many hundreds of contexts, and that this resource should be less limited. I dislike the notion that hardware-specific details creep into the standard. For instance, CM-5 could/does have 8 hardware contexts of messages, at a low level, but these are not really published to the user at all. The issue of whether one or many queues are supported is completely separate, and can be addressed separately. Separate queues weaken the partial ordering guaranteee of a message passing system; ie, that messages are received in pairwise-preserved order. Hence, they have been avoided thus far in the work I have done. However, there is nothing in principle wrong with letting an implementation disperse messages as they come in based on context. The best partial screening for messages is highly application dependent. Perhaps some type of adaptive hashing could be used internal to strong implementations, but it should not be required. For instance, independent of context support totally, adaptive hashing on type/sender could be incorporated to existing systems. I do not think it too inefficient to test 64 bits instead of 32 bits for typing. In practice, there are much more serious overheads in a system. The notion that we have full push-down message stacks, is an example of this. Active messages systems are highly context oriented. They make the context bigger (eg, an address on the destination machine) rather than smaller. One can see that there is a basic unity between message selection and context support... contexts help to route messages to the right execution part of a destination process. Registry is very important, because it guarantees that code (be it user or other library code) will cooperate reasonably well, given the message resource. This type of registry is accepted for file systems, for instance, where we accept that FORTRAN unit number is a bad way to program. Unix file descriptors provide insulation between different parts of a program. Same idea for contexts. Is a context associated with a group? Well, I hope so, but enforcing this is less important. Groups are very important in and of themselves, and to have guarantee that a context is available in a group is very useful. It also makes sense to me that a loosely synchronous registry by a group of processes to get a context is a good idea. For instance, a big group of processes initially has a context. It can ask for more contexts, and pinch off subgroups, establishing an MPMD processing environment. Overlap is permitted. Smaller groups of processes, starting with their own contexts would need a way to union into bigger groups. A mechanism for this is needed. This direction is important for scalability, for fault tolerance, and for dynamicism in processing on large collections of distributed or parallel processors. For instance, homogeneous clusters could union into a heterogeneous collection, and the resulting communication structure would provide a tree-like (at least two-level) access to messaging, which could also be useful for optimizing performance of global communication in a heterogeneous setting. Other ideas? - Tony Skjellum From owner-mpi-context@CS.UTK.EDU Tue Feb 2 12:52:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24044; Tue, 2 Feb 93 12:52:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27903; Tue, 2 Feb 93 12:51:29 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 12:51:28 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27887; Tue, 2 Feb 93 12:51:22 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00407 (5.65c/IDA-1.4.4 for ); Tue, 2 Feb 1993 12:51:19 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA22983; Tue, 2 Feb 93 17:51:16 GMT Date: Tue, 2 Feb 93 17:51:15 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302021751.AA22983@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03315; Tue, 2 Feb 93 17:49:35 GMT To: hender@macaw.fsl.noaa.gov Cc: mpi-context@cs.utk.edu In-Reply-To: Tom Henderson's message of Tue, 2 Feb 93 09:47:49 MST <9302021647.AA01526@macaw.fsl.noaa.gov> Subject: Restricted contexts Content-Length: 2607 > A few (probably dumb) questions about Jim's context proposal: Not dumb. > In what situations would I want a context that is entirely local? Would I > need to send this context to a process I intend to communicate with? The basic model I have is that contexts are very similar to Unix file descriptors (maybe I should have said this more explicitly). Therefore you can create them locally, however if someone is going to send a message to one of your contexts the they need to know its identity. So the answer to the second question is YES, you'll have to send it to them [hence the bootstrap problem...]. > I like the idea of being able to create a context within a group for > scalability reasons. How do I know that two groups will not get the "same" > new context returned from separate calls to MPI_NEWCONTEXT()? Is this even > a problem? Since contexts are allocated within a group and context creation is a barrier synchronisation there are two cases to consider :- 1) the two groups creating contexts are disjoint. In this case the contexts are independently created, and may indeed be given the same number. (Just as opening files in two processes can return the same file descriptor). I don't think this is a problem. 2) The two groups share some members. In this case we only have a problem if we have threads (since otherwise a single thread needs to be at two barriers simultaneously, which it can't.) If we do have threads, then there will have to be a lock in the library to ensure that different results are given to the two groups. > > One of the arguments to the send routines (both diadic and collective) > > would then be an MPI_CONTEXT to which the message will be sent. > > Does this mean that MPI_CONTEXT is NOT a required argument to a receive > routine? (Maybe I'm missing the whole point of this proposal! If so, please > educate me. A pseudo-code examle might be very helpful.) No you do need a context in the receive (though it might be optional in the "simple" versions with some specific default value). The issue I was addressing was whether a send needed to specify one context (that at the target), or two (that at the target and that at the sender). I was tending towards the former solution. I'll try to write some examples tomorrow. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 13:12:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24463; Tue, 2 Feb 93 13:12:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28864; Tue, 2 Feb 93 13:10:58 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 13:10:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28850; Tue, 2 Feb 93 13:10:51 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00681 (5.65c/IDA-1.4.4 for ); Tue, 2 Feb 1993 13:10:42 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA23047; Tue, 2 Feb 93 18:10:37 GMT Date: Tue, 2 Feb 93 18:10:37 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302021810.AA23047@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03318; Tue, 2 Feb 93 18:09:00 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Tue, 2 Feb 93 11:14:40 CST <9302021714.AA03830@Aurora.CS.MsState.Edu> Subject: Restricted contexts Content-Length: 4249 > The notion of context implementation for efficiency should not drive > it to be a highly limited field. I'd like the additional cost of contexts to be small. I'd also like it NOT to impact people who don't use contexts. Both of these drive me in the direction of the sort of implementation I was proposing. This is easy provided there are relatively few contexts, hard if they're as anarchic as tags. > It is true that if there are only a > few contexts, then they could be moved to hardware support. Yes, but not the reason for my proposal. I have no desire (or need) to do this. > I think that > large software will have many hundreds of contexts, and that this resource > should be less limited. I can't imagine why you need another 10 bits of context when you already have 32 of tag. What are you going to use the context for ? > I dislike the notion that hardware-specific > details creep into the standard. For instance, CM-5 could/does have 8 > hardware contexts of messages, at a low level, but these are not really > published to the user at all. I agree entirely. My proposal is not based on any specific hardware implementation. (My hardware has a small processor in it which can execute C code and run comms processor threads in a remote user space without requiring any main processor intervention, but I'm not asking for that in the standard !) > The issue of whether one or many queues are supported is completely > separate, and can be addressed separately. Separate queues weaken > the partial ordering guaranteee of a message passing system; ie, that messages > are received in pairwise-preserved order. Hence, they have been avoided > thus far in the work I have done. True, I didn't see this as a problem, since I thought that messages in different contexts could reasonably be unordered with respect to each other. (This was based on my view of what contexts are for, as explained in the intro to my proposal, maybe you have a different use for them in mind ?) > However, there is nothing in principle > wrong with letting an implementation disperse messages as they come in > based on context. The best partial screening for messages is highly > application dependent. Perhaps some type of adaptive hashing could be > used internal to strong implementations, but it should not be required. > For instance, independent of context support totally, adaptive hashing > on type/sender could be incorporated to existing systems. Of course all of these things are possible, and all add to the baseline latency. I was hoping to reduce this, not increase it. > I do not think it too inefficient to test 64 bits instead of 32 bits for > typing. In practice, there are much more serious overheads in a system. > The notion that we have full push-down message stacks, is an example of > this. I haven't seen any proposal in the MPI context which doesn't take this approach, and since I'm not trying to rewrite the whole thing... > Registry is very important, because it guarantees that code (be it user > or other library code) will cooperate reasonably well, given the message > resource. This type of registry is accepted for file systems, for instance, > where we accept that FORTRAN unit number is a bad way to program. Unix > file descriptors provide insulation between different parts of a program. > Same idea for contexts. Exactly. That's why my context proposal is based on the file descriptor model. > Is a context associated with a group? Well, I hope so, but enforcing this > is less important. Groups are very important in and of themselves, and to > have guarantee that a context is available in a group is very useful. It > also makes sense to me that a loosely synchronous registry by a group of > processes to get a context is a good idea. In my model a group can request a context. It does not have one a priori. (Maybe it doesn't need one since it can use one which the relevant processes already have). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 13:45:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25590; Tue, 2 Feb 93 13:45:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01587; Tue, 2 Feb 93 13:44:48 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 13:44:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01579; Tue, 2 Feb 93 13:44:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03874; Tue, 2 Feb 93 12:44:09 CST Date: Tue, 2 Feb 93 12:44:09 CST From: Tony Skjellum Message-Id: <9302021844.AA03874@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, jim@meiko.co.uk Subject: Re: Restricted contexts Addendum to my previous comments. I do not think it is a bad idea for vendors to provide limited, extremely fast registry mechanisms, like the CM-5's 8 hardware contexts, or whatever Meiko might have in mind along the same lines. These should be viewed as low-level services that can be incorporated into standard interfaces, increasing their performance, when they can be used. Active messages, for instance, while they are lower-level than the MPI ideas, should not be discarded by vendors. They may be a very good way for strong, MPI-upward-compatible, message systems to do a good job with hardware. We must provide for a way for vendors to extract performance from their hardware, within the context of MPI, or else they will continue to gain commercial advantage from ignoring MPI, rather than supporting it fully. - Tony From owner-mpi-context@CS.UTK.EDU Tue Feb 2 13:52:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25863; Tue, 2 Feb 93 13:52:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02049; Tue, 2 Feb 93 13:52:10 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 13:52:09 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02040; Tue, 2 Feb 93 13:52:08 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03880; Tue, 2 Feb 93 12:51:25 CST Date: Tue, 2 Feb 93 12:51:25 CST From: Tony Skjellum Message-Id: <9302021851.AA03880@Aurora.CS.MsState.Edu> To: jim@meiko.co.uk, mpi-context@CS.UTK.EDU Subject: Re: Restricted contexts Jim, I do not think it is better to keep it a 8 or 10 bits, rather than 32, for the following reasons 1) alignment 2) comparison cost (8 v. 32) on modern architectures Yes, we are going to use more than 256. Libraries that use processes in multiple configurations, and with multiple sub-groupings generate a number of contexts. I will send you the Zipcode 1.0 tech. doc. later (when finished) and you can see for yourself that many contexts can reasonably be used. For each different library in a system, one could imagine using about twenty of these message contexts. It could be up to the implementor to use more/less, depending on how services are being provided, and how service priorities are set. Tags are an unstructured field. 32 bits for the tag may be too much, but when it is all the programmer has to control selectivity, 32 is a good quantity to choose. Context suggests structuring part of the selectivity. In this case, 32 bits may be more than enough. Real software will use contexts. People who don't use them are the people who will be writing their own codes, using no one else's libraries or codes, and basically not doing the same type of top-down design we expect from good software development. I would argue that people who want to save these incremental test costs will also be unwilling to do anything but drive the hardware from the most primitive system calls. - Tony From owner-mpi-context@CS.UTK.EDU Wed Feb 3 05:35:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18460; Wed, 3 Feb 93 05:35:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21954; Wed, 3 Feb 93 05:34:51 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 3 Feb 1993 05:34:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21946; Wed, 3 Feb 93 05:34:47 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13139 (5.65c/IDA-1.4.4 for ); Wed, 3 Feb 1993 05:34:43 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26644; Wed, 3 Feb 93 10:34:39 GMT Date: Wed, 3 Feb 93 10:34:38 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302031034.AA26644@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03416; Wed, 3 Feb 93 10:32:56 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-context@CS.UTK.EDU In-Reply-To: Tony Skjellum's message of Tue, 2 Feb 93 12:51:25 CST <9302021851.AA03880@Aurora.CS.MsState.Edu> Subject: Restricted contexts Content-Length: 1848 Tony, > I do not think it is better to keep it a 8 or 10 bits, rather than 32, > for the following reasons > > 1) alignment > 2) comparison cost (8 v. 32) on modern architectures Alignment may or may not be an issue. I have no objection to you sending the context as an aligned quadword in the message header if that is faster on your hardware. I'm only interested in the restricting range of valid values. My point is that if I have only a few (8 or 10 bits) then I can use a qualitatively different implementation [an array] than if I have to deal with random 32 bit values. Of course the cost of comparison is the same, but if I only have a few contexts I can remove the comparisons from the loop entirely. > Yes, we are going to use more than 256. Libraries that use processes > in multiple configurations, and with multiple sub-groupings generate a > number of contexts. I will send you the Zipcode 1.0 tech. doc. later > (when finished) and you can see for yourself that many contexts can > reasonably be used. > For each different library in a system, one could imagine using about > twenty of these message contexts. It could be up to the implementor to > use more/less, depending on how services are being provided, and how > service priorities are set. I agree with this, BUT this doesn't mean you need that many contexts all at once. The analogy with unix file descriptors is very close. Each library might use 20 open files, but you can still get by with a limit of 64 or 128 on the number of files open at one time. Similarly with contexts. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Wed Mar 3 16:57:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12038; Wed, 3 Mar 93 16:57:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08269; Wed, 3 Mar 93 16:56:38 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 3 Mar 1993 16:56:37 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08261; Wed, 3 Mar 93 16:56:36 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05229; Wed, 3 Mar 93 15:53:56 CST Date: Wed, 3 Mar 93 15:53:56 CST From: Tony Skjellum Message-Id: <9303032153.AA05229@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: Status Dear MPI contexts people, Per our agreement at the last MPI meeting, a sub-sub-committee consisting of Cownie, Clarke, and Skjellum are developing a multi- alternative "closure" proposal for MPI. This will be presented to the entire contexts sub-committee within two weeks. After further discussion, at least two viable scenarios, required to resolve group, context, tag, group ID conflicts/overlaps, will be presented at next MPI meeting. You will receive our proposal for review as soon as possible. - Tony From owner-mpi-context@CS.UTK.EDU Mon Mar 8 17:14:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21363; Mon, 8 Mar 93 17:14:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23364; Mon, 8 Mar 93 17:14:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 8 Mar 1993 17:14:15 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23356; Mon, 8 Mar 93 17:14:14 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08120; Mon, 8 Mar 93 16:10:45 CST Date: Mon, 8 Mar 93 16:10:45 CST From: Tony Skjellum Message-Id: <9303082210.AA08120@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: PRECIS for closure proposal Please comment. - Tony Skjellum ---------------------------------------------------------------------------- Here is the precis of the post-MPI meeting... Input was from Skjellum, Littlefield, Pierce, Ranka [and others I don't recall specifically]. There shall be three self-consistent "closures" for MPI, that completely remove inconsistency between the usage of the following concepts (while fully defining the following concepts): 1) tag 2) context 3) group 4) process ID It is our task to assemble such proposals and to place them before the context sub-committee prior to the next meeting, and provide a refined set of proposals (with recommendations) before the full committee at the next MPI meeting. The first reading of this section of the standard is needed to make the rest of the standard go forward. --------------------------------------------------------------------- Proposal I. GROUP ID = CONTEXT = OBJECT, aka "Snir" model groups, contexts, objects are local and opaque. naming of processes is group-relative. this option was discussed, in most part, by Snir at meeting. Proposal II. GROUP ID != CONTEXT, aka "Littlefield" model GROUP corresponds to "data/instance" in this model CONTEXT is statically defined at link time, corresponding to code or class In this model, group ID's appear explicitly, and separately from CONTEXTS in global operations. It was noted by Littlefield that he would write global operations, and product bits of the GROUP ID and static CONTEXT to guarantee that his code produced unique, deterministic, non-interfering results. Proposal III. GROUP ID null, GROUP scope == CONTEXT scope, aka Zipcode model CONTEXT is a dynamic concept, and global (or as global as necessary) Whenever a group is created, a context is assigned to it. CONTEXTS are used to disambiguate global operations. GROUP ID is the CONTEXT for the group. GROUP is an enumeration of PROCESS ID's OR ONE OF SEVERAL possible hashing formulas, with exceptions (more scalable) Proposal IV. GROUP ID != CONTEXT, aka fully "object-oriented" model CONTEXT is a dynamic concept, and global (or as global as necessary) GROUP ID is emasculated, in the sense that there are process groups, but CONTEXT is still needed in global operations The general case of "code U data" (the object-oriented model) is taken, with context+group forming a scope in the system. The "Server model," where CONTEXTS that have no group association made to them, is to be supported. GROUP is an enumeration of PROCESS ID's OR ONE OF SEVERAL possible hashing formulas, with exceptions (more scalable) ---------------------------------------------------------------------- Requirements for all proposals to meet closure 1) All global operations must be describable in terms of point-to-point operations without loss of generality 2) Global operations, including non-blocking barriers, and so on, must work correctly with overlapping process groups 3) There must be reasonable/scalable mechanism for assigning contexts 4) Tag usage seems to be non-controversial according to above discussion, but we need to watch out for the following. 1) desire to use part of tag for context, group ID or other field, and make that part of the standard 5) Provision of enough contexts to support all describable or reasonable scenarios that we can justify, but not more than are foreseeable, because there could be a performance penalty by requiring the number of contexts to be too great. 6) Justification is perceived to be required for dynamic contexts (why?) 7) A fully consistent definition for how process IDs are to work. 8) An explanation of how to extend to dynamic groups later, if possible. From owner-mpi-context@CS.UTK.EDU Tue Mar 9 18:14:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16448; Tue, 9 Mar 93 18:14:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03977; Tue, 9 Mar 93 18:13:41 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 9 Mar 1993 18:13:39 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03969; Tue, 9 Mar 93 18:13:37 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA15144 (5.65c/IDA-1.4.4 for ); Tue, 9 Mar 1993 17:13:34 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA12285; Tue, 9 Mar 93 17:13:30 CST Date: Tue, 9 Mar 93 17:13:30 CST Message-Id: <9303092313.AA12285@godzilla.mcs.anl.gov> To: tony@aurora.cs.msstate.edu Cc: mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Mon, 8 Mar 93 16:10:45 CST <9303082210.AA08120@Aurora.CS.MsState.Edu> Subject: PRECIS for closure proposal The one point in Tony's excellent summary that needs more elaboration is needed on whether process id's, when being used in point-to-point operations, are group-relative (ranks) or absolute (and possibly opaque). Only Proposal I specifies this (ids = ranks); the others leave this unspecified. In the interests of keeping the latency down, I'd like them to be absolute, at least at the lower levels. In any event, this choice is a critical one but one that is mostly orthogonal to the other issues. In fact, I'd have chosen absolute process id's as the "Snir" model. Bill From owner-mpi-context@CS.UTK.EDU Tue Mar 9 18:40:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16750; Tue, 9 Mar 93 18:40:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05349; Tue, 9 Mar 93 18:40:04 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 9 Mar 1993 18:40:01 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05321; Tue, 9 Mar 93 18:39:58 -0500 Date: Tue, 9 Mar 93 23:39:42 GMT Message-Id: <2472.9303092339@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: PRECIS for closure proposal To: William Gropp In-Reply-To: William Gropp's message of Tue, 9 Mar 93 17:13:30 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Bill writes: > In the > interests of keeping the latency down, I'd like them to be absolute, at least > at the lower levels. In any event, this choice is a critical one but one > that is mostly orthogonal to the other issues. In fact, I'd have chosen > absolute process id's as the "Snir" model. I think we all realise what your choice would be Bill :-) There are arguments both ways, perhaps we should have both. On the other hand, the speed demons probably should be advised to use the communication handle layer at which any overhead of mapping (group,rank) to absolute-implementation-specific-best-you-can-get, and other per initialisation overheads, should be paid exactly once in the handle initialisation. Isn't that exactly why we have this layer? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 9 18:42:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16757; Tue, 9 Mar 93 18:42:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05578; Tue, 9 Mar 93 18:42:14 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 9 Mar 1993 18:42:13 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05570; Tue, 9 Mar 93 18:42:10 -0500 Date: Tue, 9 Mar 93 23:42:06 GMT Message-Id: <2480.9303092342@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: [d39135@sodium.pnl.gov: Re: group descriptors] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Rik writes: > > PS. Tony, I'm a little surprised that your reply to me was not > broadcast, especially considering your reaction to being left > out of a loop earlier. > > I am also concerned that this group is not leaving a publicly > accessible trail. These discussions have a lot of content > that should be available to others even if it does not appear > in the final proposal. > > Should we not be using an mpi-contexts reflector and archiver at > Oak Ridge? > I'm beginning to think this is correct. We seem to be talking things which seem to go beyond the "Four Proposals" agreement. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 00:36:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21055; Wed, 10 Mar 93 00:36:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18558; Wed, 10 Mar 93 00:36:04 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 00:36:03 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18550; Wed, 10 Mar 93 00:36:02 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09605; Tue, 9 Mar 93 23:30:48 CST Date: Tue, 9 Mar 93 23:30:48 CST From: Tony Skjellum Message-Id: <9303100530.AA09605@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu Subject: Re: [d39135@sodium.pnl.gov: Re: group descriptors] Cc: mpi-context@cs.utk.edu I am not sure to which comment I incorrectly misresponded. Rik, please remind me... I think there are a total of two letters which are improperly sent only to you, and I do want them shared with everyone. I was quite rushed today, and hit the wrong R key, no doubt. To my recollection, both deal with Methods within group/contexts. If you want to elaborate mailing to entire mpi-contexts subcommittee, I concur. Let's make sure all of you are on that list, so we don't lose track of the main discussion people, however. So, my basic request to Rik is that he echo the two letters sent to him alone, which was my error. Mea culpa. :-) Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 01:10:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21883; Wed, 10 Mar 93 01:10:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19605; Wed, 10 Mar 93 01:09:48 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 01:09:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19597; Wed, 10 Mar 93 01:09:46 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09722; Wed, 10 Mar 93 00:05:40 CST Date: Wed, 10 Mar 93 00:05:40 CST From: Tony Skjellum Message-Id: <9303100605.AA09722@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov Subject: Re: group descriptors Cc: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu This corrects the second message, accidently sent only to Rik. - Tony From tony Tue Mar 9 23:23:17 1993 To: d39135@sodium.pnl.gov Subject: Re: group descriptors Content-Length: 897 Rik, I was foolish not to mention that we have implemented a registration mechanism with trees, in Zipcode 1.0. The user can add available methods for combine, and then,when a mailer is created, specific choices can be taken from the list. Currently, we implement with numbers, but this could be done with names, to allow better sharing. I will check on choices we made. It has been some time since I worked on that internal code. Anyway, I concur on the need to have registerable choices, that allow sharing, per your letter. As regards new functionality, they could go in the tree too, but there is no structural mechanism in Zipcode, other than defining a class' method list, for adding new instantiations to a mailer at creation. All of this was very elaborate coding, which I will elucidate in my article for Rolf. I am happy to share details with you as well, beforehand. - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 06:16:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13775; Wed, 10 Mar 93 06:16:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06917; Wed, 10 Mar 93 06:15:36 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 06:15:33 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06900; Wed, 10 Mar 93 06:15:29 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09848; Wed, 10 Mar 93 05:10:50 CST Date: Wed, 10 Mar 93 05:10:50 CST From: Tony Skjellum Message-Id: <9303101110.AA09848@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: re: static contexts writeup [My comments prepended with **** - Tony] **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with **** the first-generation PRECIS, to generate a more useful one, **** Please correct me if I am wrong, but I cannot find this! **** (Again, I apologize for calling this a PRECIS, not a SKETCH.) **** I want to generate the next version of this SKETCH/PRECIS, **** with specifics garnered from our lengthy bantering and **** discussions. Particularly, I will discuss the sparse matrix **** of proposals X closure options. Lyndon, you indicated that **** one of my comments challenged you to make a new added entry **** to one of the proposal categories. May I have that now/soon? **** **** In interest of refocusing discussion, I will work on such a **** new PRECIS/SKETCH, reviewing all the mail flurry from last days, **** and then suggest writing assignments. Lyndon will be off-line, but **** Lyndon will still get some brain work to do off-line. :-) Lyndon **** will you phone me periodically, to stay abreast of things? **** Perhaps we will need the manpower of Rusty/Bill to help us, given **** the amount of work on filling the sparse matrix of issues. Willing? **** **** Our PRECIS/SKETCH and work from it will not address the following things **** (related to discussion in our group, but not central to it): **** - the fully portable MPI program **** - I/O **** - the semantics and syntax of process management **** - the time-table for MPI-1,MPI-2,MPI-3,... **** In a truly global sense, these could be construed to be closure as **** well, but I want to exclude them for sanity reasons. I do believe **** that parts of these issues could be addressed within MPI-1, but **** it is not now this subcommittee's role to do that for MPI at-large. **** Strong objections? If so, we are in deep trouble:-) **** **** We will assume that discussion in this 4-proposal X closure effort is **** all essential to MPI-1. Anything you really don't think is essential **** for MPI-1... once we agree to same .... should probably get deferred. **** Is that controversial? **** **** I thank everyone for their sincere efforts. **** - Tony **** **** PS Per constructive advice from Rik, all of my mail is getting copied to **** mpi-context now. All should do so. I did not hear any dissent **** about this suggestion, and it adds potential help to **** discussion. As we need the help, and few people are writing **** anyway, this will not likely cause problems. **** Continue to include the five names jim, lyndon, tony, sanjay & **** Rik, incase any of us are not on the mpi-context mailing list!!! **** **** PPS Lyndon, where are you. I got up at 4am to read the next round **** of mail from you, but none to be found :-) **** **** Rik writes... >Before it gets TOO stale, here's a copy of my writeup on static >contexts et.al. as it last went to Lyndon and Tony. (You two can >stop reading at the line of dashes.) > >I have opted not to hack on static contexts any more because I >would rather kill them. **** Is this a proposal to include static contexts as in within **** the 4-proposal model, and that we stop working on same? **** If so, I can make it so... >I now suspect that I would trade static contexts for something >like the MPI_set_group_attribute cacheing facility that I have >outlined in recent email. I will think more about this and get >back to you. > >In the meantime, I would very much like more feedback on the >cacheing facility, especially how it breaks. > >Thanks, >--Rik > >---------------------------------------------------------------------- > >This note discusses message context ID's -- what they're good for, >how to generate them, how to use them. > >Credits.. At the last Dallas meeting, Mark Snir asked me to write >up some thoughts on this topic and to send them around to the >whole committee. Paul Pierce, Tony Skjellum, several other >people (whose names I can't remember) and I discussed some of the >ideas over lunch on the last day. Then Lyndon Clarke of the >contexts subcommittee emailed me asking for some clarification of >static context generation, which I guess I'm the main proponent >of. He reviewed an earlier (and even longer) draft of this note, >and it is now much improved. (Thanks, Lyndon!) However, I have >substantially rewritten it since talking with any of those folks, >so I have to take full blame for the fallacies of this version. > >1. Notation and assumptions > > In this note, I assume that > > . "context ID" means a message selection field > that cannot be wildcarded, > > . "tag" means a message selection field > that *can* be wildcarded, and > > . context ID, tag, and process ID are the *only* fields > that can be used for pt-pt message selection. > > In particular, I assume that "group ID" cannot be used for > selection unless it is somehow propagated into the tag or > context ID fields. > > I will speak of "modules" that operate within "groups". By > "module", I mean a procedure (subroutine) or collection of > procedures that can exploit knowledge of each other's internal > workings in order to coordinate their message passing. By > "group", I mean a collection of processors identified by a > globally unique "group ID". By "operating within a group", I > mean that the module is called simultaneously by all processes > belonging to the group and exchanges messages primarily with > those copies of itself. (I assume that arbitrary communication > is always possible. The focus of this discussion is how to > make restricted communication easy.) I assume that the group > ID is passed into the module. > > I will also speak of an "instance of a module". By this, I > mean a module, plus the group that it is operating in, plus the > data it is working on. (The latter becomes important if the > module supports nonblocking operation. In that case, I assume > that some data ID is passed into the module. An example is the > 'tag' argument of the nonblocking collective routines.) > > My notation requires that independently written modules can be > called sequentially within the same group. Specifically, if a > a group must be "duplicated" in order to guarantee that two > modules do not conflict, then I will say that those modules are > operating in different groups. > > By definition, independent modules do not coordinate their use > of tags. (A module can of course coordinate with itself. > That's what tags are for.) > >2. Why do we want context ID's? > > The most important purpose of context ID's is to avoid message > collisions between independent modules. (This is the only way > to avoid such collisions, since coordination of tag values is > excluded by definition.) > > Context ID's are sometimes also be the best way to avoid > collisions between instances of the same module. (This depends > on the definition of "best", as discussed below.) The tag > field can also be used for such avoidance, since the module has > full control over how the tags are used. > >3. Desirable features of contexts and groups: > > I think there will be little debate about these: > > A. It should easy to write modules that don't conflict with the > same or other modules, even when the modules are called > within overlapping groups. (Or, save us all, with multiple > outstanding non-blocking operations.) > > B. It should be easy to call these modules. > > There may be more debate about these, but I like them as sanity > checks: > > C. The MPI routines that manage groups and contexts should > be definable in terms of MPI point-to-point communication > and process-local operations. > > (For example, I get nervous about group and context > management semantics that require asynchronous servers or > their equivalent, like an asynchronous atomic > fetch-and-increment. We already decided to eliminate > interrupt receive at the user level, on the argument that > it was too hard to implement correctly. Asynchronous > servers are a bit easier, since they can run on > resources other than the processes calling MPI, but > personally I would rather avoid servers if possible.) > > D. The "closed-system constraint": standard global operations > (e.g., broadcasting within a group) should be definable in > terms of MPI point-to-point and group/context operations. > > (For example, we must be careful to avoid bootstrap > problems like needing a context specific to a group, in > order to create the group.) > >3. How do modules get and use context ID's ? > > Assume that we have already created a group and distributed its > group ID, and that we have a mechanism for broadcasting within > the group. (More on this later.) > > Then we have at least the following options for managing > context ID's: > > Option 1. "Context ID = group ID" (and no group-dup'ing). > > By my definition of "same group", i.e. dup'ing gets you > a different group, this option means that two modules can > get called back-to-back with the same context. > > I think this is OK, courtesy sequencing constraints, if and > only if both modules use exact match on tag and source. > Since the group ID is already available (passed in), this > provides a simple and efficient mechanism for many useful > algorithms, such as blocking global ops. > > This option fails, however, with modules that use wildcard > tag or source. (I won't bother to provide an example -- > having the same context is the same as having no context, > and we know what havoc that creates.) > > Option 2. One process obtains from MPI a unique context ID > for each instance of a module, and distributes it > for that instance's use. > > I believe that this is the strategy proposed by Tony Skjellum > in his 3-page handout at Dallas. The distribution might be > done either within a module or by its caller. Presumably > there are more callers than modules, so doing it within the > module is preferred. As noted above, distribution of the new > context can be done with a broadcast using exact match on > tag and source, with context=group, presuming of course that > the module follows the rule of doing wildcard matches only > in a different context. > > In the Dallas discussions, it seemed to be assumed that this > option requires an asynchronous server, since the context > generation calls have to be coordinated to ensure uniqueness > but can occur at any time. This bothered me. > > However, I realized while writing this note that this option > can also be done without a server by having a process-local > pool of preassigned context ID's. For example, let the > bottom bits of a context ID be the ordinal (in some global > numbering scheme) of the process that assigned it, and manage > the upper bits however you would in a server. For P > processes, this would require at most log(P) extra bits in > the context field, compared to having a server. > > So now my main concerns about this approach are the extra > communication to distribute the context ID's, plus a nagging > feeling that there's still a bootstrap problem if this is > the only option. > > Option 3. "Static context ID's" > > This is the option that I pushed at the Dallas meeting. > > The idea is that each module gets a context ID that is based > solely on a source code key, then manages its tag values to > keep the groups and instances separate. > > To be definite, assume that a module can obtain a context ID > via a call like > > error = MPI_get_context_for_name (name, &contextID) > > where is the character string unique to the module, and > the generated context ID depends only on the supplied name. > This call is made independently on all processes wishing to > share the context. > > The simplest convention would be to require that be > the name of a procedure within the module. That way the > linker guarantees uniqueness. There are other ways to get > unique strings, such as the "dollar bill algorithm" and > whatever strategies people use to reserve names for expansion > of libraries, but these do not affect the basic idea. > However, you cannot use the address of a routine in place of > its name, since this would not work in heterogeneous systems. > > The most efficient strategy is to require that all names be > "registered offline". By this, I mean that MPI be given a > list, prior to execution, of all the names that might be > passed to MPI_get_context_for_name by this particular > program. There are several ways that this could be > accomplished. For example, if we require that the names > always be those of modules, then a (conservative) list can be > generated at link time using Unix 'nm' or equivalent. > > With offline registration, MPI_get_context_for_name can > be implemented so as to require no communication at all. > > We can go farther and require that be a constant > string, in which case the call to MPI_get_context_for_name > could be implemented as a compile-time macro substitution. > (I hate citing it, but as precedent I note that this strategy > is used by the optimizer for some versions of 'Linda'.) > > Without offline registration, it seems that MPI_get_context_for_name > would require an asynchronous server. I will not discuss this > further because in my view static contexts are intended to > make for simple implementations and fast execution. > > OK, now we can quickly and efficiently (zero execution > cost) get a context ID that is unique to the module, but all > instances of that module have to share the context. (Please, > no kneejerk reactions -- hear me out before labeling this as > a stupid idea in complete violation of the spirit of contexts.) > > How does the module keep multiple instances straight? > It uses the message tag field. > > The idea is to generate an MPI tag value by smushing together > the group ID, data ID (if needed), and the nominal tag value > that the module would have used if it didn't have to worry > about multiple instances. No other module cares how the > smushing is done -- the context is unique and we have agreed > that modules can use the tag bits however they want. This is > just an example of a module coordinating with itself > regarding how to handle the tag bits. > > Yes, this scheme does seem like jumping through a hoop, but > it does have some advantages. If the group and data ID's are > short enough to be smushed into a message tag without losing > any bits, then the MPI tag can be generated with absolutely > no communication. This means that: > > 1. It provides a method to implement other group > and context management functions that is bulletproof > and free of bootstrap problems. > > 2. It executes fast. > > To understand the first advantage, consider the problem of > how to implement MPI_close_group(gid) using only MPI pt-pt > operations. Remember that the module executing within that > group on other nodes can legitimately still have a wildcard > receive posted. If MPI_close_group tries to use the group > context, its messages can get eaten. Sync the group first? > That just means you have the same problem implementing sync. > Having static contexts for MPI_close_group and/or sync solves > this problem. Are there other solutions for which MPI pt-pt > operations would be sufficient? > > The second advantage is perhaps weaker, but I for one keep > finding myself writing modules that are sensitive to > communication performance, particularly latency. In many > cases, I would prefer to write a few extra lines of code than > to use functionality that would cost extra messages. I > have lots of experience with tag-smushing due to working > with low-functionality communication systems (like select > on message tag only, and no useful wildcarding). I find > that extra coding effort is very small. Typically it's > embedded in a macro call or cover routine, so that I still > have to write only one line of code to handle a message. > > Note that I do NOT propose static contexts as a solution for > everyone. However, they are useful for speed, and may be > required under the closed-system constraint. > >4. Summary > > It looks to me like all three options presented here are > complementary. They might all might be provided, in the > expectation that they will be used by different audiences. > >Questions, comments? > >--Rik Littlefield > > From owner-mpi-context@CS.UTK.EDU Wed Mar 10 06:34:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17635; Wed, 10 Mar 93 06:34:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07441; Wed, 10 Mar 93 06:33:22 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 06:33:21 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07433; Wed, 10 Mar 93 06:33:19 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09870; Wed, 10 Mar 93 05:28:59 CST Date: Wed, 10 Mar 93 05:28:59 CST From: Tony Skjellum Message-Id: <9303101128.AA09870@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@CS.UTK.EDU Subject: latest lyndon-mail **** My latest comments start lines with ****. **** - Tony >From lyndon@epcc.ed.ac.uk Wed Mar 10 05:12:18 1993 >Received: from epcc.ed.ac.uk (daedalus.epcc.ed.ac.uk) by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); > id AA09849; Wed, 10 Mar 93 05:11:49 CST >Date: Wed, 10 Mar 93 11:14:44 GMT >Message-Id: <2772.9303101114@subnode.epcc.ed.ac.uk> >From: L J Clarke >Subject: [Tony Skjellum: Re: [Tony Skjellum: Re: Intra/inter group communications, UBONBOUND, PIDs]To: Tony Skjellum , Jim Cownie , > Rik Littlefield , > Sanjay Ranka >Reply-To: lyndon@epcc.ed.ac.uk >Apparently-To: ranka@top.cis.syr.edu >Apparently-To: d39135@sodium.pnl.gov >Apparently-To: jim@meiko.co.uk >Apparently-To: tony@aurora.cs.msstate.edu >Status: R > >Good morning > >Lyndon replying to Tony replying to Lyndon replying to Tony replying to ... > > >> >(again). Please help. >> > >> >First way, "context" is being used to indicate is a binding of group >> >plus more, and therefore I understand it easily. >> > >> >Second way, "context" is being used in a way which I understand. >> > >> ***** I think my comment made more sense to me when I wrote it before. >> ***** I withdraw it. > >Arrrghhhh! I realise that I have badly typed the "Second way", which >should say that "context" is being used in a whay which I do not >understand. This is very badly typed indeed, apologies. > >I think this was a request for information/sharing of understanding, >which I expect I still need. **** I was viewing the whole context-thing as a "port" specification. **** How about a domain-wide name for a piece of mail to be delivered? **** Such as mail from UK to US? **** I am still not sure this was cogent... **** >> >The point is that you do not actually want to produce a new group. You >> >want to keep thinking of the two groups as seperate entities, each >> >acting individually in a coherent fashion, and communicating with one >> >another perhaps according to some rules based on ranks in each >> >indicidual group. >> ***** I understand that, but why do you want to keep thinking of it that way, >> ***** except for some possible performance benefit? >> **** > >I tell you why, it's because we find it easy to think of it and discuss >it in that way. This really is the whole point, expressive power. **** **** I believe in the expressive-power argument, no problem. I am **** also willing to believe that we should keep the minimum cost of **** inter-group down to the point where it remains useful. It is **** dynamicism of groups that motivates inter-group communication. **** Else, the one-shot cost of merging groups, and staying intra-group **** would not be unacceptable. There must be some ineffable quality **** of groups (dynamic need to change, for instance) that you hope for! **** I agree with this abstractly, but please help me more... **** >> > >> >Same "I can read this two ways" as above. >> ***** OK, I withdraw this remark too. > >Look, I found it vague. I'd rather you helped me to understand, please. > **** Related thoughts... **** I am trying to understand my original comment, myself. I think **** of context/group/rank_id as a MPI_TID, if you wish. It all made more **** sense to me structured that way, rather than having a pair of **** contexts as a mechanism for inter-group. In short, I felt we were **** inventing our own TID called context/group/rank_id, and that this **** was a nice structure, and superceded the PVM notion of a TID. **** If I am foggy, sorry, but if we worked in those terms, with accessors **** on such a TID, we would know how to handle some of the problems **** described elsewhere. For instance, we could have the meaning of **** rank_id be group-scope/dependent. Rank would be a nice, fast hash on **** static groups, and a slower, more complex thing to map for **** intergroup [are people still liking NUL/UNBOUND???]. **** >> > >> >If this is an invitation to add a further proposal to the set we are >> >doing, then I accept such invitation. In fact, I can frame it as an >> >extension of proposal I. >> ***** YES. > >Will do. > >> > >> >> Of course, point-to-point could provide another set of procedures which >> >> handle inter-group communication cleanly. This is my preferred >> >> approach, but again I don't suppose that this will be acceptable, >> >> because so may people are already so concerned about the number of >> >> procedures in the point-to-point section. >> >> **** >> >> **** Again, why not try. We need to fill in our matrix of proposals X >> >> **** closures :-) >> >> **** >> > >> >Because I perceive that it would be like trying to drag a dead whale >> >along a beach :-) (You never tried that? I can tell you, its hard work >> >and slow going!) >> ***** Acknowledged. > >Ah, so you tried pulling a dead whale as well. Funny old world :-) > >> ***** I think I was interested in implementation with p2p, because >> ***** of the degrees of freedom inside an MPI envelope dictated by >> ***** some kinds of transcations. These transactions might, for >> ***** instance, motivate >> ***** 1) support of more than K contexts (where K is currently like 256) >> ***** 2) require dynamic contexts which are separate from group ID's >> ***** >> ***** Hence, by asking how the functions are to be done, I am asking >> ***** for possible justifications for the more general context model. >> ***** If hidden contexts are needed to do some of these things, >> ***** perhaps they could be justifiably be accomplished with an open >> ***** context mechanism. >> ***** >> ***** That was my motivation. - Tony >> ***** >> > >I'll think about these comments, but I suspect there is a language >barrier in operation here. > >Best Wishes >Lyndon > **** Lyndon, we know that these functions will have to be implemented some how. **** MPI itself might provide such mechanisms as published functionality or **** hidden functionality. If published functionality is denied in the **** standard (eg, general context management) because we cannot justify it **** properly, but must be implemented internally for most/all complying **** versions of MPI, then I think we lose out. My vague way of stating **** the requirement was to reveal the possibility of this, and thereby to **** promote such internal degrees of freedom into the public part of MPI, **** so applications can use them... Does that help???? **** - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 07:01:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23631; Wed, 10 Mar 93 07:01:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08660; Wed, 10 Mar 93 07:01:08 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 07:01:07 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08652; Wed, 10 Mar 93 07:00:58 -0500 Date: Wed, 10 Mar 93 12:00:52 GMT Message-Id: <2831.9303101200@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: [Tony Skjellum: re: static contexts writeup] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Tony writes: > **** PPS Lyndon, where are you. I got up at 4am to read the next round > **** of mail from you, but none to be found :-) > **** First off, many thanks to Tony for getting to his terminal at 4am. I really appreciate the effort to be UK compatible. I fell asleep at mine about 2am and then went home unable to stir mental activity. From 9am to 11am I was in meetings. I had a fair bit of mail to read, but more replies to my copious mail of last night yet expected. You should have got two messages beiefly describing some fairly basic inter-group communications kind of things that are important to me. I'll make as much time today as I can, but as I said last night, I have a hell of a lot of other things to get through as well today. I will come in additionally tomorrow morning from 9am to 10:30 am, to catch up on any further discussion developments. Then I really have to get on the train. > **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with Hey, I got about uncountable copies of Rik's note saved away, now another one :-) > **** Lyndon, you indicated that > **** one of my comments challenged you to make a new added entry > **** to one of the proposal categories. May I have that now/soon? challenged?, invited! Soon, Tony, soon. > **** Lyndon will be off-line, but > **** Lyndon will still get some brain work to do off-line. :-) :-) I can write up Proposal I, and Ib (Lyndon extension amendment) while away. > **** Lyndon > **** will you phone me periodically, to stay abreast of things? Okay, but this will be after 8pm my time, as the calls will come out of my pocket and appear on my parents 'phone bill, and its cheaper then, and they'll have to be real short. > **** Our PRECIS/SKETCH and work from it will not address the following things > **** (related to discussion in our group, but not central to it): > **** - the fully portable MPI program I vote we exclude from our discussion. > **** - I/O I vote we exclude from our discussion. > **** - the semantics and syntax of process management Perhaps something of this should be thought about. Quick thoughts/suggestions: * Create new process group (blob of processes :-) * Resize existing process group (shrink/grow blob, synchronously in blob) > **** - the time-table for MPI-1,MPI-2,MPI-3,... I vote we exclude from our discussion. > **** In a truly global sense, these could be construed to be closure as > **** well, but I want to exclude them for sanity reasons. I do believe > **** that parts of these issues could be addressed within MPI-1, but > **** it is not now this subcommittee's role to do that for MPI at-large. > **** Strong objections? If so, we are in deep trouble:-) No objections. > **** > **** We will assume that discussion in this 4-proposal X closure effort is > **** all essential to MPI-1. Anything you really don't think is essential > **** for MPI-1... once we agree to same .... should probably get deferred. > **** Is that controversial? Not controversial. > **** I thank everyone for their sincere efforts. - ditto - > **** PS Per constructive advice from Rik, all of my mail is getting copied to > **** mpi-context now. All should do so. I did not hear any dissent > **** about this suggestion, and it adds potential help to > **** discussion. As we need the help, and few people are writing > **** anyway, this will not likely cause problems. > **** Continue to include the five names jim, lyndon, tony, sanjay & > **** Rik, incase any of us are not on the mpi-context mailing list!!! Concur, being done. > > > **** Rik writes... > > >Before it gets TOO stale, here's a copy of my writeup on static > >contexts et.al. as it last went to Lyndon and Tony. (You two can > >stop reading at the line of dashes.) > > > >I have opted not to hack on static contexts any more because I > >would rather kill them. > **** Is this a proposal to include static contexts as in within > **** the 4-proposal model, and that we stop working on same? > **** If so, I can make it so... I'd like us to now ask Rik to write Proposal II. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 08:37:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28930; Wed, 10 Mar 93 08:37:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12056; Wed, 10 Mar 93 08:36:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 08:36:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12047; Wed, 10 Mar 93 08:36:43 -0500 Date: Wed, 10 Mar 93 13:36:37 GMT Message-Id: <2981.9303101336@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: [Tony Skjellum: re: static contexts writeup] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Proposal Ib Proposal Ib extends Proposal I (nee aka "snir" model) to provide group identification and inter-group communication for static (constant membership) groups. Group identification is effected by: facilities to send group in message; group registry service Inter-group communication needs to extend the basic (group,rank) notation. Two suggestions will be offered: procedures to do this; syntactic trickery (see previous emails). Regards Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 11:12:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03467; Wed, 10 Mar 93 11:12:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20567; Wed, 10 Mar 93 11:12:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 11:12:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20557; Wed, 10 Mar 93 11:12:11 -0500 Date: Wed, 10 Mar 93 16:12:01 GMT Message-Id: <3127.9303101612@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: private message To: Tony Skjellum In-Reply-To: Tony Skjellum's message of Wed, 10 Mar 93 10:00:55 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu No problem. This would be just fine. I perfer not to have to administrate it, especially as I will be off-line and therefore unable to service administration requests. Can I suggest that the address be precis@aurora.cs.mssatte.edu :-) /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 11:32:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03864; Wed, 10 Mar 93 11:32:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21943; Wed, 10 Mar 93 11:31:32 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 11:31:29 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21932; Wed, 10 Mar 93 11:31:27 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA10032; Wed, 10 Mar 93 10:25:42 CST Date: Wed, 10 Mar 93 10:25:42 CST From: Tony Skjellum Message-Id: <9303101625.AA10032@Aurora.CS.MsState.Edu> To: tony@aurora.cs.msstate.edu, jim@meiko.co.uk, d39135@sodium.pnl.gov, ranka@top.cis.syr.edu, lyndon@epcc.ed.ac.uk Subject: Re: [Tony Skjellum: re: static contexts writeup] Cc: mpi-context@cs.utk.edu **** My latest comments appear prepended with for ****'s. **** -Tony ----- Begin Included Message ----- From owner-mpi-context@CS.UTK.EDU Wed Mar 10 05:58:24 1993 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 07:01:07 EST Date: Wed, 10 Mar 93 12:00:52 GMT From: L J Clarke Subject: Re: [Tony Skjellum: re: static contexts writeup] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Content-Length: 4438 Tony writes: > **** PPS Lyndon, where are you. I got up at 4am to read the next round > **** of mail from you, but none to be found :-) > **** [comments on interactions over next day omitted, :-)] **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with Hey, I got about uncountable copies of Rik's note saved away, now another one :-) > **** Lyndon, you indicated that > **** one of my comments challenged you to make a new added entry > **** to one of the proposal categories. May I have that now/soon? challenged?, invited! Soon, Tony, soon. > **** Lyndon will be off-line, but > **** Lyndon will still get some brain work to do off-line. :-) :-) I can write up Proposal I, and Ib (Lyndon extension amendment) while away. > **** Lyndon > **** will you phone me periodically, to stay abreast of things? Okay, but this will be after 8pm my time, as the calls will come out of my pocket and appear on my parents 'phone bill, and its cheaper then, and they'll have to be real short. ***** Fine, I will phone you, if you provide time windows and phone number(s). ***** I will address costs. > **** Our PRECIS/SKETCH and work from it will not address the following things > **** (related to discussion in our group, but not central to it): > **** - the fully portable MPI program I vote we exclude from our discussion. > **** - I/O I vote we exclude from our discussion. > **** - the semantics and syntax of process management Perhaps something of this should be thought about. Quick thoughts/suggestions: * Create new process group (blob of processes :-) * Resize existing process group (shrink/grow blob, synchronously in blob) **** OK, I will think about the proposal X closure matrix placement of this. > **** - the time-table for MPI-1,MPI-2,MPI-3,... I vote we exclude from our discussion. > **** In a truly global sense, these could be construed to be closure as > **** well, but I want to exclude them for sanity reasons. I do believe > **** that parts of these issues could be addressed within MPI-1, but > **** it is not now this subcommittee's role to do that for MPI at-large. > **** Strong objections? If so, we are in deep trouble:-) No objections. > **** > **** We will assume that discussion in this 4-proposal X closure effort is > **** all essential to MPI-1. Anything you really don't think is essential > **** for MPI-1... once we agree to same .... should probably get deferred. > **** Is that controversial? Not controversial. > **** I thank everyone for their sincere efforts. - ditto - > **** PS Per constructive advice from Rik, all of my mail is getting copied to > **** mpi-context now. All should do so. I did not hear any dissent > **** about this suggestion, and it adds potential help to > **** discussion. As we need the help, and few people are writing > **** anyway, this will not likely cause problems. > **** Continue to include the five names jim, lyndon, tony, sanjay & > **** Rik, incase any of us are not on the mpi-context mailing list!!! Concur, being done. > > > **** Rik writes... > > >Before it gets TOO stale, here's a copy of my writeup on static > >contexts et.al. as it last went to Lyndon and Tony. (You two can > >stop reading at the line of dashes.) > > > >I have opted not to hack on static contexts any more because I > >would rather kill them. > **** Is this a proposal to include static contexts as in within > **** the 4-proposal model, and that we stop working on same? > **** If so, I can make it so... I'd like us to now ask Rik to write Proposal II. **** Yes, Rik please do proposal II!!! Best Wishes Lyndon ----- End Included Message ----- **** Best wishes, **** Tony Summary of what went on above: We agreed that Lyndon would do I and Ib (his extension) We requested that Rik would do II (please agree/decline :-) I am still not sure if Rik did/plans to do a precis/sketch merger??? By implication, I am left with III/IV, expecting to get help from Jim, Sanjay, et al on it. I have promised to do the proposal X closure precis/sketch update. - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 12:19:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05037; Wed, 10 Mar 93 12:19:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24634; Wed, 10 Mar 93 12:19:24 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 12:19:22 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24626; Wed, 10 Mar 93 12:19:20 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 09:14 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA21499; Wed, 10 Mar 93 09:12:13 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20633; Wed, 10 Mar 93 09:12:10 PST Date: Wed, 10 Mar 93 09:12:10 PST From: d39135@sodium.pnl.gov Subject: Problem: group descriptors & coherency To: d39135@sodium.pnl.gov, tony@Aurora.CS.MsState.Edu Cc: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Message-Id: <9303101712.AA20633@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Here's a problem that applies to all proposals. Groups are dynamic. Group ID's must be reusable unless they are very long or we bound the number of groups that can ever get created. Every reference to a group ID must be interpreted in terms of the current definition. This introduces a coherency problem analogous to shared memory. There are at least a half-dozen solutions to the coherency problem, with wide differences in overhead costs and convenience for the user. Explicit coherency control by the application tends to be cheaper but less convenient than implicit control. ==> I suggest that each proposal include some mechanism for solving the group ID coherency problem, including a discussion of its probable costs. Aside: Coherency of tid's seems like a smaller problem, based on the assumption that processes will come and go much less often than groups will be re-formed. But I'm not sure that we can ignore it. --Rik From owner-mpi-context@CS.UTK.EDU Wed Mar 10 13:18:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06332; Wed, 10 Mar 93 13:18:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28066; Wed, 10 Mar 93 13:18:05 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 13:18:04 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28057; Wed, 10 Mar 93 13:18:02 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:10 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA22798; Wed, 10 Mar 93 10:08:19 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20772; Wed, 10 Mar 93 10:08:13 PST Date: Wed, 10 Mar 93 10:08:13 PST From: d39135@sodium.pnl.gov Subject: implementation assumptions To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303101808.AA20772@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-comm@cs.utk.edu In a discussion we are having within the contexts subcommittee, Tony Skjellum commented that: > ... I want to make sure that we do > not provide illusory improvements to the standard, while letting other > parts of it exceed ours in the unscalability department, or let > implementations qualify as compliant, whose unscalability disables the > implied scalability of our syntax/semantics... I agree. This is why (I think) proposals should explicitly address performance. If a proposal assumes that the cost of a particular function is irrelevant, it should say that. If a proposal assumes that a particular function is going to be cheap, it should justify that assumption by naming a time/memory price and outlining an implementation that can achieve it. Sometimes the suggested implementation will reveal further assumptions that have wide implications. Ideally, each proposal will eventually evolve a set of bottom-line assumptions that are consistent and believable. These comments are particularly addressed to those of us beating on contexts and groups, where there seem to be lots of subtle interactions. But I am cross-posting them to the whole committee as food for thought... --Rik ("The Skeptic") Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 13:29:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06699; Wed, 10 Mar 93 13:29:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28812; Wed, 10 Mar 93 13:28:44 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 13:28:43 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28804; Wed, 10 Mar 93 13:28:41 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:21 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23065; Wed, 10 Mar 93 10:19:47 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20875; Wed, 10 Mar 93 10:19:42 PST Date: Wed, 10 Mar 93 10:19:42 PST From: d39135@sodium.pnl.gov Subject: Re: descriptors vs ID's To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303101819.AA20875@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu (This is a resend. Apparently several addresses got fouled up yesterday.) Rik said: > But I argue that the global ID's should NOT be the things that get > passed to the collective comm routines, and possibly not even to the > pt-pt routines. > > Passing a global ID implies some kind of table searching, unless we > put tight bounds on the value ranges so that we can do a simple > indirection. Of course, I suppose I could equally well argue that the cost of such a lookup would be small compared to the overall cost of the collective comm routine. Maybe this is a don't-care. --Rik (Good grief, Lyndon -- replying to one's own mail seems to be catching!) From owner-mpi-context@CS.UTK.EDU Wed Mar 10 13:59:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07404; Wed, 10 Mar 93 13:59:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00653; Wed, 10 Mar 93 13:58:49 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 13:58:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00645; Wed, 10 Mar 93 13:58:45 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:49 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23233; Wed, 10 Mar 93 10:47:14 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21230; Wed, 10 Mar 93 10:47:12 PST Date: Wed, 10 Mar 93 10:47:12 PST From: d39135@sodium.pnl.gov Subject: Re: descriptors vs ID's To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Message-Id: <9303101847.AA21230@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu (This is a resend with modifications [flagged with **]. The original, yesterday, had fouled up addresses.) Jim raises the issue: > Are the tid's of local or global scope. i.e. Can I pass a tid to > someone else, and it still work as an address for the person I > passed it to ? > For global scope : It's simpler for the user > Against : The implementor might want the "opaque identifier" > to be a pointer to a big data structure containing > route tables or something. The same problem arises with groups and I'm not sure where else. I agree that we need global scope ID's for tids and probably for groups. But I argue that the global ID's should NOT be the things that get passed to the collective comm routines, and possibly not even to the pt-pt routines. Passing a global ID implies some kind of table searching, unless we put tight bounds on the value ranges so that we can do a simple indirection. For tid's, that's OK because there are only as many tid's as there are processes, and I'm willing to believe that's a small bound. But in theory there can be lots more groups than processes, and even in practice I'm not sure that we can keep the global ID's both unique and small without excessive overhead. (E.g., recall my trick of generating unique context values without communication by cramming the creator's process number into the bottom bits.) I propose that most routines be passed a local descriptor reference, rather than the global ID value. ** The above may be don't-care. For collective comms, the cost ** of finding a (local) group descriptor, given the group ID, ** is arguably small compared to the cost of the comm itself. ** For tid's, perhaps the tid values can be kept small enough to ** permit simple indirection. Then we need either 1. routines to retrieve the global ID from a local descriptor (easy) and to construct a local descriptor from the global ID (harder) or 2. a mechanism to translate local descriptors to and from some machine-independent form. I prefer the second option because of my bias against servers. ** In another message, I raised the problem of maintaining ** coherence of group descriptors if the group ID's can be reused. ** The comments above also relate to methods of explicitly ** restoring coherency. Scheme 1 would be used if a process was ** told only "group XX has been redefined, please invalidate your ** descriptor"; scheme 2 would be used if a process was told ** "group XX has been redefined and here's the new definition". --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 14:20:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08095; Wed, 10 Mar 93 14:20:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01852; Wed, 10 Mar 93 14:19:58 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 14:19:57 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01844; Wed, 10 Mar 93 14:19:56 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 11:14 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23523; Wed, 10 Mar 93 11:13:08 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21416; Wed, 10 Mar 93 11:13:06 PST Date: Wed, 10 Mar 93 11:13:06 PST From: d39135@sodium.pnl.gov Subject: Re: [d39135@sodium.pnl.gov: Re: group descriptors] To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: mpi-context@cs.utk.edu Message-Id: <9303101913.AA21416@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu > So, my basic request to Rik is that he echo the two letters sent to him > alone, which was my error. Mea culpa. I think there was only one, and I included the whole thing in my note referencing it, so it has been echoed already. No problem. --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 14:21:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08148; Wed, 10 Mar 93 14:21:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01996; Wed, 10 Mar 93 14:21:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 14:21:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01988; Wed, 10 Mar 93 14:21:16 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 11:10 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23493; Wed, 10 Mar 93 11:08:57 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21333; Wed, 10 Mar 93 11:08:53 PST Date: Wed, 10 Mar 93 11:08:53 PST From: d39135@sodium.pnl.gov Subject: Re: group descriptors To: d39135@sodium.pnl.gov, tony@Aurora.CS.MsState.Edu Cc: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Message-Id: <9303101908.AA21333@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony writes > Anyway, I concur on the need to have registerable choices, that allow > sharing, per your letter. As regards new functionality, they could go > in the tree too, but there is no structural mechanism in Zipcode, > other than defining a class' method list, for adding new instantiations > to a mailer at creation. All of this was very elaborate coding, which > I will elucidate in my article for Rolf. I am happy to share details > with you as well, beforehand. Maybe I haven't thought about this enough, but I do not understand why elaborate coding is required if we do everything at runtime and do not try to provide type-safety. No one has yet pointed out a deficiency with my proposed interface: MPI_set_group_attribute (gid,key,value,destructor_routine) and MPI_test_group_attribute (gid,key,&value) This can be implemented at a data cost of adding a single slot to a group descriptor (pointing to the attribute dictionary), plus a loop in the group closing routine to call the destructors (if any), plus the routines to manage the dictionary. I do not believe that there would be much resistance from the committee to adding this capability, since it seems to be easy to implement and (almost) "free if you don't use it". What have I missed? --Rik Littlefield PS. OK, since nobody else has pointed out a deficiency, I will. The definition provided above does not support transferring attributes between processes. The uses that I had in mind involve information that is so local to the process that transferring it doesn't even make sense (e.g., lists of partners for collective comms). If one wanted to define attributes that could be transferred, the interface would have to be extended. From owner-mpi-context@CS.UTK.EDU Wed Mar 10 14:51:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08797; Wed, 10 Mar 93 14:51:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03683; Wed, 10 Mar 93 14:51:06 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 14:51:04 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03673; Wed, 10 Mar 93 14:51:02 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 11:48 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24208; Wed, 10 Mar 93 11:46:25 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21672; Wed, 10 Mar 93 11:46:20 PST Date: Wed, 10 Mar 93 11:46:20 PST From: d39135@sodium.pnl.gov Subject: re: static contexts writeup To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303101946.AA21672@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony, You write > **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with > **** the first-generation PRECIS, to generate a more useful one, > **** Please correct me if I am wrong, but I cannot find this! I think I'm clean. The exchange was > > > (Tony writes) > > > So, my first action item is to get a merger of what you wrote and our > > > precis, and redistribute that to group. OK? > > > > (Rik replies) > > I'm not sure that I've seen your precis yet. The stuff you've sent me > > so far seems to be quite a ways from precise (pardon the pun), so I'm > > reluctant to guess what's what. Can you send me a labeled copy? > > > > It would be faster and perhaps less confusing if I incorporated the > > last round of your and Lyndon's comments into my words and distributed > > that to the subcommittee as a separate document. Then we would have > > a collective opinion (well, five separate ones anyway) of how the > > two documents fit together, instead of just the one opinion of whoever > > did the merger. > > (Tony replies) > Yes, I concur. Go ahead. Several copies of my document were subsequently distributed, including ones by me and by you! As I noted when I sent it out, I did not incorporate your and Lyndon's comments because I decided my time might better go into exploring cacheing mechanisms. By the way, I'm sorry, but I'm still not sure that I have correctly identified your precis or sketch or whatever. You sent me some stuff at a level of detail like > Proposal II. GROUP ID != CONTEXT > > GROUP corresponds to "data/instance" in this model > CONTEXT is statically defined at link time, corresponding to > code or class > > In this model, group ID's appear explicitly, and separately > from CONTEXTS in global operations. It was noted by Littlefield > that he would write global operations, and product bits of the > GROUP ID and static CONTEXT to guarantee that his code produced > unique, deterministic, non-interfering results. > > Proposal III. GROUP ID null, GROUP scope == CONTEXT scope > > CONTEXT is a dynamic concept, and global (or as global as necessary) > Whenever a group is created, a context is assigned to it. CONTEXTS > are used to disambiguate global operations. > > GROUP ID is the CONTEXT for the group. > > GROUP is an enumeration of PROCESS ID's OR ONE OF SEVERAL > possible hashing formulas, with exceptions (more scalable) with a bunch of comments interspersed with it. Is this what you are talking about? In other mail, you have mentioned a "sparse matrix of proposals X closure options", but I haven't the foggiest idea what that refers to. Help? --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 15:13:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09008; Wed, 10 Mar 93 15:13:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06326; Wed, 10 Mar 93 15:12:24 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 15:12:22 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06312; Wed, 10 Mar 93 15:12:19 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 12:09 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24295; Wed, 10 Mar 93 12:07:30 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21860; Wed, 10 Mar 93 12:07:27 PST Date: Wed, 10 Mar 93 12:07:27 PST From: d39135@sodium.pnl.gov Subject: writing assignments To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Cc: mpi-context@cs.utk.edu Message-Id: <9303102007.AA21860@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu > I'd like us to now ask Rik to write Proposal II. > > Best Wishes > Lyndon I will be happy to write "Proposal II", but I'm not sure that we agree on what that means. I would definitely like to include a proposal for a cacheing facility, but this seems orthogonal to all other issues about groups, contexts, tid/pid format etc. I will also discuss the idea of static contexts, but will specifically not propose them, in favor of cacheing, if I can convince myself that cacheing combined with other capabilities meets all my requirements. (Cacheing is clearly superior in some ways, but I'm not sure that it covers all the bases.) --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 17:06:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12404; Wed, 10 Mar 93 17:06:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13359; Wed, 10 Mar 93 17:05:52 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 17:05:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13347; Wed, 10 Mar 93 17:05:49 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13032; Wed, 10 Mar 93 16:00:56 CST Date: Wed, 10 Mar 93 16:00:56 CST From: Tony Skjellum Message-Id: <9303102200.AA13032@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Subject: fixing misunderstanding about nomenclature In your last letter, Rik, you clearly identify what I have called the PRECIS/SKETCH. I have been talking loosely about a set of proposals (so marked), and Lyndon has understood these all along too. They are called I...IV, with some sub-proposals also. Then, for each of these, there are certain options on what things like tags do, and so on, so we imagine a (hopefully sparse) matrix of alternatives to be looked at each of which is a valid closure of MPI. Of the ones that make sense (eg, Proposal II with tags being blah, and TIDS being blah', and the following other side conditions defined, we have one possible closure of MPI. The sub-committee assigns a rank to this choice; we will hopefully have more than one. We vote on all of them, in rank order. If we disagree violently on rank, we present all of them in a coherent order, and vote on them in a reasonable order.) I want to move this SKETCH to have more precise information in it. I will look at doing same. Hopefully, you are doing proposal II, per Lyndon's suggestion, and the place to find what proposal II is is still in that SKETCH. - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 17:08:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12413; Wed, 10 Mar 93 17:08:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13470; Wed, 10 Mar 93 17:07:55 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 17:07:54 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13455; Wed, 10 Mar 93 17:07:53 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13048; Wed, 10 Mar 93 16:03:26 CST Date: Wed, 10 Mar 93 16:03:26 CST From: Tony Skjellum Message-Id: <9303102203.AA13048@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, ranka@top.cis.syr.edu Subject: Re: writing assignments Cc: mpi-context@cs.utk.edu Rik writes... >From d39135@sodium.pnl.gov Wed Mar 10 14:08:49 1993 >Received: from pnlg.pnl.gov by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); > id AA11263; Wed, 10 Mar 93 14:08:46 CST >Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 > 12:09 PST >Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24295; Wed, > 10 Mar 93 12:07:30 PST >Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21860; Wed, 10 Mar 93 12:07:27 > PST >Date: Wed, 10 Mar 93 12:07:27 PST >From: d39135@sodium.pnl.gov >Subject: writing assignments >To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, > ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu >Cc: mpi-context@cs.utk.edu >Message-Id: <9303102007.AA21860@sodium.pnl.gov> >X-Envelope-To: tony@aurora.cs.msstate.edu >Status: R > >> I'd like us to now ask Rik to write Proposal II. >> >> Best Wishes >> Lyndon > >I will be happy to write "Proposal II", but I'm not sure that >we agree on what that means. > >I would definitely like to include a proposal for a cacheing >facility, but this seems orthogonal to all other issues about >groups, contexts, tid/pid format etc. > >I will also discuss the idea of static contexts, but will >specifically not propose them, in favor of cacheing, if I can >convince myself that cacheing combined with other capabilities >meets all my requirements. (Cacheing is clearly superior in some >ways, but I'm not sure that it covers all the bases.) > >--Rik Littlefield > OK. The cacheing thing seems to add another item to our list of proposal things. Go ahead... think of it as another column in our sparse matrix, for which caching is the same for each major Proposal... - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 18:48:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13684; Wed, 10 Mar 93 18:48:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17757; Wed, 10 Mar 93 18:47:50 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 18:47:43 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17747; Wed, 10 Mar 93 18:47:40 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 15:45 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24514; Wed, 10 Mar 93 15:43:28 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA23252; Wed, 10 Mar 93 15:43:24 PST Date: Wed, 10 Mar 93 15:43:24 PST From: d39135@sodium.pnl.gov Subject: Re: fixing misunderstanding about nomenclature To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303102343.AA23252@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu > ... Of the ones that make sense > (eg, Proposal II with tags being blah, and TIDS being blah', and the > following other side conditions defined, we have one possible closure > of MPI. The sub-committee assigns a rank to this choice; we will hopefully > have more than one. We vote on all of them, in rank order. If we > disagree violently on rank, we present all of them in a coherent order, > and vote on them in a reasonable order.) > > I want to move this SKETCH to have more precise information in it. I will > look at doing same. Hopefully, you are doing proposal II, per Lyndon's > suggestion, and the place to find what proposal II is is still in that > SKETCH. > > - Tony > I assume that the goal is to get several proposals that are self-consistent but different. Since Proposal II was at one point represented as "Littlefield's model", I will try to write my personal favorite proposal into bullet-item form. I don't know close it will be to your current Proposal II, but it'll probably be different from anybody else's. I will try to get you this by tomorrow (Thursday). Once we have several different bullet-item proposals in hand, what is the best way to proceed? My inclination is to have several rounds of critique, with the author of each proposal responding to each criticism by one or more of: . modifying or extending the proposal while retaining its character . modifying or extending the statement of assumptions . modifying or specifying an implemention strategy to overcome the criticism . acknowledging a deficiency. Hopefully there will not be too much convergence between proposals. --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 23:22:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17588; Wed, 10 Mar 93 23:22:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27668; Wed, 10 Mar 93 23:21:36 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 23:21:34 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27659; Wed, 10 Mar 93 23:21:33 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13324; Wed, 10 Mar 93 22:17:22 CST Date: Wed, 10 Mar 93 22:17:22 CST From: Tony Skjellum Message-Id: <9303110417.AA13324@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Subject: Re: fixing misunderstanding about nomenclature Rik writes... (my responses prepended with four *'s - Tony)... > ... Of the ones that make sense > (eg, Proposal II with tags being blah, and TIDS being blah', and the > following other side conditions defined, we have one possible closure > of MPI. The sub-committee assigns a rank to this choice; we will hopefully > have more than one. We vote on all of them, in rank order. If we > disagree violently on rank, we present all of them in a coherent order, > and vote on them in a reasonable order.) > > I want to move this SKETCH to have more precise information in it. I will > look at doing same. Hopefully, you are doing proposal II, per Lyndon's > suggestion, and the place to find what proposal II is is still in that > SKETCH. > > - Tony > I assume that the goal is to get several proposals that are self-consistent but different. **** Yes. Since Proposal II was at one point represented as "Littlefield's model", I will try to write my personal favorite proposal into bullet-item form. I don't know close it will be to your current Proposal II, but it'll probably be different from anybody else's. I will try to get you this by tomorrow (Thursday). **** Thank you. **** Rik suggests the following strategy for handling proposal review. **** I would like to use it... **** Once we have several different bullet-item proposals in hand, what is the best way to proceed? My inclination is to have several rounds of critique, with the author of each proposal responding to each criticism by one or more of: . modifying or extending the proposal while retaining its character . modifying or extending the statement of assumptions . modifying or specifying an implemention strategy to overcome the criticism . acknowledging a deficiency. **** Fine, and we will use this as the criterion for all proposals. **** It is a sound strategy. **** **** Thanks on both counts. - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 11 19:08:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11521; Thu, 11 Mar 93 19:08:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25952; Thu, 11 Mar 93 19:08:20 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 11 Mar 1993 19:08:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25936; Thu, 11 Mar 93 19:08:14 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 11 Mar 93 15:59 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA26618; Thu, 11 Mar 93 15:58:00 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA25111; Thu, 11 Mar 93 15:57:57 PST Date: Thu, 11 Mar 93 15:57:57 PST From: d39135@sodium.pnl.gov Subject: contexts Proposal V To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: gropp@mcs.anl.gov, lusk@mcs.anl.gov Message-Id: <9303112357.AA25111@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony, OK, here's a detailed proposal regarding groups and contexts. It's much different from the earlier sketch called "Proposal II", so I renamed it. This proposal seems to meet my needs. Barring any new difficulties, I will be happy to drop static contexts. The proposal is long but hierarchical. I think there's enough in the summary to characterise it. I am also sending this description to Bill Gropp and Rusty Lusk. I hope they will wring it out, particularly my assertion that this proposal can be implemented as a layer on top of MPI pt-pt. --Rik Littlefield ---------------------------------------------------------------------- PROPOSAL V. Summary: . Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) . Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) . There are restrictions that permit groups to be layered on top of pt-pt. . Pt-pt communications use only TID, context, and tag, and are specified to be fast. . Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. Detailed Proposal: . Pt-pt uses only "TID", "context", and "message tag". TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. . Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) . A group is represented by a "group descriptor", of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a "process-local group ID", which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) . Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). . Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). . Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. . At creation, each group is assigned a globally unique "default group context" which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). . The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the "paired-exact-match constraint". This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) . A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] . Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) . When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. . All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). . Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] . All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's "root process", the number of processes in the group, and an integer "group tag" provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] . Collective communication routines are called by all members of a group in the same order. . Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. . Non-blocking collective communication routines are passed a reference to the group descriptor, plus a "data tag" to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. Advantages: . This proposal is implementable with no servers and can be layered easily on existing systems. . Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) . Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. . Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. . Communication both within and between groups seems conceptually straightforward. Disadvantages: . Requires explicit translation between (group,rank) and TID, which may be considered awkward. . Communication between different groups may be considered awkward. . No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). . Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. . Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. . No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. Comments / Alternatives: . The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. . Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. . The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like "a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context". This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] . Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. Implementation Notes: 1. To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) 2. The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a "group partitioning context" and a "group disbanding context" that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. Examples: (To be provided) From owner-mpi-context@CS.UTK.EDU Sun Mar 14 13:58:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05196; Sun, 14 Mar 93 13:58:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16166; Sun, 14 Mar 93 13:58:14 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 14 Mar 1993 13:58:10 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16158; Sun, 14 Mar 93 13:58:04 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 14 Mar 93 10:57 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA29374; Sun, 14 Mar 93 10:55:26 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA05149; Sun, 14 Mar 93 10:55:22 PST Date: Sun, 14 Mar 93 10:55:22 PST From: rj_littlefield@pnlg.pnl.gov Subject: proposal to mpi-collcomm To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, gropp@mcs.anl.gov, jim@meiko.co.uk, lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303141855.AA05149@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Al & Tony, et.al.: I am about to send to mpi-collcomm, two notes regarding changes I propose to the collective communication specification. (One note summarizes the changes; the other discusses the reasons for them.) I am also sending these notes to mpi-context and friends because they relate to other discussions going on there. Thought you'd like to know... --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Sun Mar 14 15:04:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06324; Sun, 14 Mar 93 15:04:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18133; Sun, 14 Mar 93 15:03:51 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 14 Mar 1993 15:03:49 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18125; Sun, 14 Mar 93 15:03:46 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 14 Mar 93 12:01 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA29382; Sun, 14 Mar 93 11:59:17 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA05208; Sun, 14 Mar 93 11:59:13 PST Date: Sun, 14 Mar 93 11:59:13 PST From: rj_littlefield@pnlg.pnl.gov Subject: collcomm changes, summary To: geist@gstws.epm.ornl.gov, gropp@mcs.anl.gov, jim@meiko.co.uk, lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303141959.AA05208@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu SUMMARY OF SUGGESTED CHANGES TO COLLECTIVE COMMUNICATION PROPOSAL The draft proposal that Al Geist distributed several days ago contains some features that would prevent it from being implemented as a layer on top of MPI point-to-point facilities. The purpose of this note is to propose changes to the group control routines in order to permit layering, and to propose other changes for better and more predictable performance. A discussion of the rationale for these proposed changes will be distributed separately because of its length. The main changes introduced in this note are: . The concept of group identification is firmed up. Most operations use a "group handle" that is local to the process. (Think of the group handle as being just the address of a potentially large and complex "group descriptor".) There is still a "group ID" that is globally unique, but it has only a secondary role and can be ignored by most applications. The "group name" is entirely removed from MPI-1. (Group names are still anticipated in MPI-2, but upward-compatibility is maintained in a different way from the draft proposal.) . A semantic restriction is introduced, that a process can access information about a group only if the process holds a group handle for it. Group handles can be obtained in two ways: 1) they are produced by group formation routines, and 2) a process can explicitly distribute copies of its group handles to other processes, using new routines introduced specifically for that purpose. . A cacheing mechanism is introduced, that allows modules to attach arbitrary information to a group descriptor in such a way that it can be quickly retrieved. Cacheing facilitates the construction of collective communication routines that are "fast after the first execution in a group", no matter how the other group operations are implemented. . A new group formation routine is introduced, that is less synchronous and more general than MPI_PARTGROUP. Specifically, the following routines are proposed to be added or modified: 1. Arbitrary group formation: newgrp_handle = MPI_FORMGROUP (grouptag,groupsize,knownmembers) where grouptag is a user-provided integer tag, sufficiently unique to disambiguate overlapping groups that might be formed simultaneously (say by multiple threads). groupsize is the number of members that will compose the group. knownmembers is a set of pid's of some or all members of the group. Each member of the group must provide the same set of knownmembers. newgrp_handle is a group handle for the newly formed group This new routine must be called synchronously, but only by those processes forming the group. 2. Group partitioning: newgrp_handle = MPI_PARTGROUP (oldgrp_handle,grouptag) where the semantics are the same as the draft proposal except that the return value is now a new group handle instead of a rank. (The rank can be determined by a separate call to MPI_GETRANK(group_handle,pid) .) 3. Group disbanding: MPI_LVGROUP (group_handle) where the semantics are the same as the draft proposal except that MPI_LVGROUP now does not return any result. (Since groups can now be formed arbitrarily, not just by partitioning, it is not obvious what MPI_LVGROUP could return in general.) This routine can be called only by members of the group. 4. Distribution of group handles and disposition of distributed handles: MPI_SendGroupHandle (pid,context,tag,old_group_handle) new_group_handle = MPI_RecvGroupHandle (pid,context,tag) MPI_FreeGroupHandle (group_handle) (The latter routine is similar to MPI_LVGROUP except that it can be called only for distributed group handles. This is solely for semantic clarity; a single interface routine would do.) 5. Cacheing group-specific process-local information: The following routines get and free keys for use with group cacheing. key = MPI_GetAttributeKey () MPI_FreeAttributeKey () The following routines cache and retrieve information. MPI_SetGroupAttribute (grouphandle,key,value,destructor_routine) status = MPI_TestGroupAttribute (grouphandle,key,&value) where key must be unique within the group value is anything the size of a pointer destructor_routine is an application-provided routine that is called by MPI_LVGROUP, with arguments being the group handle, cached key and value. Cached information is stripped from the new group handle returned by MPI_SendGroupHandle. In a conforming implementation, MPI_TestGroupAttribute must be no slower than a point-to-point communication call. 6. Retrieving global group ID: global_id = MPI_GetGlobalGroupID (grouphandle) 7. Other collective communications: Consistently substitute "grouphandle" in place of "group". ---------------------------------------------------------------------- rj_littlefield@pnl.gov Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Sun Mar 14 15:50:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06915; Sun, 14 Mar 93 15:50:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19420; Sun, 14 Mar 93 15:49:43 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 14 Mar 1993 15:49:41 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19411; Sun, 14 Mar 93 15:49:35 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 14 Mar 93 12:48 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA29389; Sun, 14 Mar 93 12:46:59 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA05301; Sun, 14 Mar 93 12:46:57 PST Date: Sun, 14 Mar 93 12:46:57 PST From: rj_littlefield@pnlg.pnl.gov Subject: collcomm changes, rationale To: geist@gstws.epm.ornl.gov, gropp@mcs.anl.gov, jim@meiko.co.uk, lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303142046.AA05301@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu RATIONALE FOR SUGGESTED CHANGES TO COLLECTIVE COMMUNICATION PROPOSAL In a related summary, I outlined a set of suggested changes to the concepts and routines in the collective communication proposal. The purpose of this note is to present the rationale for those suggestions and to discuss possible alternatives. The discussion is organized into 5 areas, flagged with "----- Topic #". Entries flagged with > are from my summary of suggested changes. Entries flagged with >>> are from the draft proposal sent out by Al Geist. ----- Topic #1: Group Identification ----- > . The concept of group identification has been firmed up. Most > operations use a "group handle" that is local to the process. > (Think of the group handle as being just the address of a > potentially large and complex "group descriptor".) > ... > . A semantic restriction is introduced, that a process can access > information about a group only if the process holds a group > handle for it. Group handles can be obtained in two ways: 1) > they are produced by group formation routines, and 2) a process > can explicitly distribute copies of its group handles to other > processes, using new routines introduced specifically for that > purpose. There are two issues here: one of being able to layer collective communications on top of point-to-point at all, and a secondary one of efficiency. The more fundamental issue is layering. Given only MPI point- to-point functionality, how can a group identifier (whatever it is) be transmitted between processes so as to be useful to the receiver? Presumably we want to allow group identifiers to be passed around so that any process holding the group identifier can use it for purposes like translating between (group,rank) and pid. We also want to allow this translation to be done asynchronously, i.e., without requiring the explicit cooperation of any other MPI process at the time of translation. Since MPI pt-pt does not support asynchronous servers or an interrupt receive capability, this implies that the group identifier must come complete with enough information to resolve all translations without communication. This prompts the concept that the group identifier must be associated with a "group descriptor" that is large and complex enough to fully describe the group. How is the association done? This is a question of efficiency. If the identifier is allowed to be process-local, the group descriptor can be located very quickly -- just make the identifier be a pointer to the group descriptor. Requiring the identifier to have global scope would not be so good. In that case, either the identifier has to be carefully constructed or the association has to be done with some sort of table search. These issues also arise with global pid's. However, groups can be formed much more often and in greater numbers than processes. I doubt that careful construction tricks could be assured to be adequate, and if not, then a table search would be required on each collective communication call. The conclusion is that, for most purposes, a process-local identifier generated by the system is preferred. Such things are typically called "handles", hence the term "group handle". > 4. Distribution of group handles and disposition of distributed handles: > > MPI_SendGroupHandle (pid,context,tag,old_group_handle) > > new_group_handle = MPI_RecvGroupHandle (pid,context,tag) > > MPI_FreeGroupHandle (group_handle) The next question is how group handles should be distributed. Implicit distribution is out because MPI pt-pt doesn't support a server capability, and presumably we aren't willing to synchronize all of the processes whenever somebody creates a group handle. So, explicit distribution is required. How do we handle it? Two ideas that I do not like are the following. MPI might provide routines to translate to and from some machine- and process- independent format, so that the translated information could be sent using normal point-point primitives. This strategy requires that the user program manage the storage of indefinite-length objects, which makes for an ugly Fortran interface. Or, group descriptors (and their translation routines) might be built into point-point MPI as another data type. This violates the spirit of layering collective communication on point-to-point, and has the same storage management problem. The three routines proposed above were the cleanest interface I could think of. ----- Topic #2: Global Group ID ----- > There is > still a "group ID" that is globally unique, but it has only a > secondary role and can be ignored by most applications. > ... > global_id = MPI_GetGlobalGroupID (grouphandle) Given that we are now able (and required) to pass around copies of group handles, it is not clear to me that MPI really needs special support for the concept of a global group ID. On the other hand, it's easy to provide, since we have to construct one or more globally unique context values for each group anyway. So just use the first such context value as the global ID. This gives something unique that all processes can agree on. But note that knowing just the global group ID does not let you get other information about the group -- you have to hold a group handle for that. (We could add a routine that would accept the global group ID and return a handle for that group, presuming that the process held one. This would be cheap to do, since group handles are managed by MPI anyway, and I can vaguely imagine that it might help some applications. On the other hand, there are no similar "handle lookup" facilities provided elsewhere in MPI, and I'm reluctant to set that kind of precedent without clear need.) ----- Topic #3: Group Formation ----- > . A new group formation routine is introduced, that is less > synchronous and more general than MPI_PARTGROUP. > ... > 1. Arbitrary group formation: > > newgrp_handle = MPI_FORMGROUP (grouptag,groupsize,knownmembers) > > where > grouptag is a user-provided integer tag, sufficiently unique > to disambiguate overlapping groups that might be > formed simultaneously, say by multiple threads. > > groupsize is the number of members that will compose the group. > > knownmembers is a set of pid's of some or all members of the group. > Each member of the group must provide the same > set of knownmembers. > > newgrp_handle is a group handle for the newly formed group > > This new routine must be called synchronously, but only by those > processes forming the group. The draft proposal distributed by Al Geist says that >>> A group is identified by a group name that is supplied by the user. A group name by itself is not enough to allow implementing groups as a layer on top of point-to-point, unless we impose restrictions that I think would be not acceptable. The problem is: how does a group-forming routine know whom it should send messages to, in order to form the group? MPI_PARTGROUP does not have a problem with this, because it has to be called synchronously by all members of the group. Since each current member of the group holds a handle (descriptor) for that group, it is easy for each member to figure out who talks to whom. Unfortunately, there are some important application designs that I do not see how to implement with just MPI_PARTGROUP. For example, I am now doing an application that uses a master-slaves strategy to asynchronously parcel out chunks of work, with each chunk being done by several processes working collaboratively. Collective communication between those processes is required, so it seems natural to organize them into MPI groups. Using a synchronous group partitioning routine would introduce a risk of load imbalance, because the varying chunk size implies that groups can finish their work at different times, and synchronous partitioning would delay their reassignment. Applications like this could benefit from a group formation routine that is called synchronously, but only by those processes forming the group -- hence MPI_FORMGROUP. This type of routine does have the problem of identifying its collaborators, and the only solution I can think of is to tell it. That's what the knownmembers argument is for. I have specified knownmembers in terms of pid's because I assume that point-to-point communication based on pid's is always fast and unrestricted. If knownmembers were based on (group,rank) pairs, then per the discussion above, all processes making this call would have to hold handles (descriptors) for the referenced groups. This seems to me to be more trouble than it's worth, but others may disagree. Another comment about efficiency... The size of the knownmembers set affects the efficiency of group formation. At one extreme, only one member is required to be known. This is scalable in a memory sense, but not in a time sense, because it implies O(P) group formation time for a group of P processes. At the other extreme, all members can be specified. This is not scalable in a memory sense, but allows guaranteed O(log P) formation time. Other tradeoffs are possible, such as O(sqrt P) knownmembers and O(sqrt P) formation time. The interface as specified allows each application to choose the type of scalability it wants. ----- Topic #4: Group Names ----- > ... The > "group name" is entirely removed from MPI-1. (Group names are > still anticipated in MPI-2, but upward-compatibility is > maintained in a different way from the draft proposal.) The draft distributed by Al Geist states: >>> To allow for future extensibility of the group concept >>> the present draft specifies that groups be named. Requiring names has the drawback that 1) it burdens the user with at least the appearance of having to create unique names, in order to be upward-compatible with dynamic groups, even though 2) in a layered MPI-1, there is no way in general to check global uniqueness, and thus programs can work fine with non-unique names. This combination strikes me as actually impeding upward- compatibility. The tendency will be for programmers to use non-unique names because it works and it's easy. But such programs would break when MPI-2 came along and started actually using the names for something. I don't like encouraging people to write programs that are going to break. I do support upward compatibility. However, rather than requiring names in MPI-1, I propose that they be deferred entirely to MPI-2, at which point they can be supported either just through MPI_JOINGROUP (as an alternative to MPI_FORMGROUP) or via additional routines to attach globally unique names to groups that have already been formed via MPI_JOINGROUP. ----- Topic #5: Cacheing ----- > 5. Cacheing group-specific process-local information: > > The following routines get and free keys for use with group > cacheing. > > key = MPI_GetAttributeKey () > MPI_FreeAttributeKey () > > The following routines cache and retrieve information. > > MPI_SetGroupAttribute (grouphandle,key,value,destructor_routine) > MPI_TestGroupAttribute (grouphandle,key,&value) > > where > key must be unique within the group > value is anything the size of a pointer > destructor_routine is an application-provided routine that > is called by MPI_LVGROUP, with arguments > being the group handle, cached key and value. > > Cached information is stripped from the new group handle > returned by MPI_SendGroupHandle. > > In a conforming implementation, MPI_TestGroupAttribute must > be no slower than a point-to-point communication call. This feature is purely for efficiency, but I think it's so valuable, cheap, and clean that something like it has to go in. One feature of collective communication is that the fastest algorithm for any particular job usually depends on the machine topology, which processes belong to the group, and the amount of data being manipulated. For example, global combine of L data elements across P = RC processes on a 2-D RxC mesh can be done in O(L log(P)) time using a fanin/fanout algorithm, or in O(L + sqrt(P)) time using a nested rings algorithm. The former is better for small L, the latter for big L, and using the wrong one can easily cost a factor of 3 in execution time. So, there is strong motivation to write collective communication routines that are adaptive in the sense of figuring out which algorithm is best. The problem is that it can take quite a lot of time to make the decision, starting from a scratch position of not even knowing which processes belong to the group. It's going to take lots of calls to the inquiry routines to get that information, and then some more cycles to make the proper decisions. Obviously it would be profitable to cache the information and/or decisions. The question is, where? It is tempting to say that the collective communication routine could or should keep its own cache, indexed by group handle and/or global group ID. The problem is, groups are dynamic in the sense of being formed and disbanded, so that unless group IDs can get very large, eventually they will have to be reused. Now, it wouldn't do to have a collective communication routine use stale cached information, so if the collective communication routine is keeping its own cache, then it needs to be notified of the reuse so that it can release the cached stuff. Alternatively, perhaps the cached information could be automatically released. (Either strategy guarantees immediate release of cached info when the group handle/descriptor is released. I presume we want to do that, to avoid getting into the morass of garbage collection.) The method proposed here can be thought of as implementing both strategies. The idea is that the routines that free group handles (and the associated descriptors) loop through the cached information, calling an application-provided destructor routine for each piece of cached information. Typically, the cached information will be a pointer to a hunk of memory managed by the collective communication, which the destructor will free in whatever way it has to. Upon return from the destructor, the group-freeing routine will release the little piece of memory holding the pointer, and everything will be cleaned up. If that group handle/descriptor is ever reused, it will be reinitialized to indicate no cached information, and MPI_TestGroupAttribute will return "not found". An efficient-after-first-call group-global operation using cacheing might look like this: static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_key_type gop_key; /* key for this module's stuff */ efficient_global_op (grphandle, ...) struct group_descriptor_type *grphandle; { struct gop_stuff_type *gop_stuff; /* whatever we need */ if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetAttributeKey()) ) { MPI_abort ("Insufficient keys available"); } } if (MPI_TestGroupAttribute (grphandle,gop_key,&gop_stuff)) { /* This module has executed in this group before. We will use the cached information */ } else { /* This is a group that we have not yet cached anything in. We will now do so. */ gop_stuff = /* malloc a gop_stuff_type */ /* ... fill in *gop_stuff with whatever we want ... */ MPI_SetGroupAttribute (grphandle, gop_key, gop_stuff, gop_stuff_destructor); } /* ... use contents of *gop_stuff to do the global op ... */ } gop_stuff_destructor (gop_stuff) /* called by MPI on group close */ struct gop_stuff_type *gop_stuff; { /* ... free storage pointed to by gop_stuff ... */ } ---------------------------------------------------------------------- rj_littlefield@pnl.gov Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Thu Mar 18 13:18:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16171; Thu, 18 Mar 93 13:18:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18320; Thu, 18 Mar 93 13:17:37 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 13:17:35 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18296; Thu, 18 Mar 93 13:17:07 -0500 Date: Thu, 18 Mar 93 18:17:01 GMT Message-Id: <9471.9303181817@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal I To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Here is Proposal I, basic and extended proposals, as promised. I have not added detailed justifications or implementation complexity discussions in this document. This should be transparent. I have crafted the proposal using LaTeX, in the form of a "chapter" in the "report" document style. You should be able to run this through LaTeX and get it printed off no problem, please ask me for postscript if you do have some problem. The stuff before and including, \begin{document}, and the stuff after and including \end{document} could be removed for inclusion into a report document style. Comments? Best Wishes Lyndon >------------------------------ Cut Here ------------------------------< \documentstyle{report} \begin{document} \title{``Proposal'' I for MPI Communication Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Lyndon J. Clarke % March 1993 % \chapter{Proposal I} \section{Introduction} This chapter proposes that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The proposal reflects the observation that an instance of a module in a parallel program typically operates within a group of processes, and as such any communication contexts associated with an instance of a module also bind the semantics of process groups. This chapter makes a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The proposals say nothing about use of message tags. It is assumed that these will be a bit string, expressed as an integer in the host langauge (i.e. ANSI C or Fortran~77, in the first instacnce. These proposals should be viewed as a collection of recommendations to other subcommittees within MPI, primarily the collective communications subcommittee, the point-to-point communications subcommittee and the process topologies subcommittee. Concrete syntax is given in the style of the C language for purposes of discussion only, and should be viewed as an example of possible syntax as detailed syntax of described operations is the responsibility of the language bindings subcommittee %----------------------------------------------------------------------% % BEGIN "Basic Proposal" % \subsection{Basic Proposal} The main features of the basic proposal are: \begin{itemize} \item Process identifiers and group identifiers have opaque values expressed as an integer type in the base language. Semantics for passing process identifiers in a message are defined, whereas semantics for passing group identifiers in a message are not defined. \item Group creation and destruction are concerted, synchronous, actions on the part of the membership of the group concerned. \item Groups are {\it static\/} in that no provision is made for modification of the membership of a group over the lifetime of the group. Dynamic group operations such as {\it resize\/} can be effected by destruction of the existing group and creation of a new resized group. \item Group creation is effected by four means: explicit definition of the membership of a group; partition of an existing group into one or more distinct subgroups; identical duplication of an existing group; topological duplication of an existing group. \item Point-to-point communication provides for intra-group communication, and makes no provision for inter-group communication. \item Collective communication provides for operations which are concerted actions on the part of members of one group, and makes no provision for operations which are concerted actions on the part of members of multiple groups. \end{itemize} \subsubsection{Process and Group Identifiers} A {\it process identifier\/} is an opaque reference to an object which is a single process. A process identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on process identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). Each process has exactly one process identifier. MPI should provide a procedure which allows the user to determine the process identifier of the calling process. For example, {\tt mypid = mpi\_mypid()}. The identifier of the {\it null process\/}, defined to be a process which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_PID\_NULL} in this proposal. With the single exception of the identifier of the null process, the value of a process identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a process P then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a process identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the received process identifier is {\it invalid\/}. MPI will provide a mechanism which allows a process identifier to be passed in a message in such a manner that the received identifier is valid. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a process identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_pid(bd, \&pid)}. Transmission of a process identifier using this mechanism returns to the recipient a process identifier which is valid for use in the MPI operations described below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a process identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_pid\_invalidate(pid)}. This is an error if {\tt pid} is {\tt MPI\_PID\_NULL}, or if {\tt pid} is the identifier of the calling process. It is further proposed that MPI provide a process identifier registry service. This service allows any process to register its own process identifer by name, and deregister its process identifier. The service allows any process to determine whether a name has been registered without blocking the calling process, and to map that name into a valid process identifier with the possibility of blocking the calling process. Use of this service is not mandated, and components of programs which do not require this service are not expected to make use thereof. A {\it group identifier\/} is an opaque reference to an object which is a group of processes. A group identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on group identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). The identifier of the {\it null group\/}, defined to be a group which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_GID\_NULL} in this proposal. With the single exception of the identifier of the null group, the value of a group identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a group G then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a group identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the identifier is {\tt invalid\/}. MPI will not provide a mechanism which allows a group identifier to be passed in a message in such a manner that the received identifier is valid. The canonical representation of a group is an array of distinct process identifiers, although it may be possible to use hashing functions with lower space complexity and marginally higher time complexity. A group is {\it static\/}, in that it's membership may not change over the lifetime of the group. There is a well defined {\it size\/} of a group, and MPI will provide a procedure which allows the user to determine the size of a group. For example, {\tt size = mpi\_grp\_size(gid)}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. There is a well defined {\it rank\/} of a process within the group, i.e. the position in the representational array at which the process identifier is held. MPI will provide a procedure which allows the user to determine the rank of the calling process within a group. For example, {\tt mpi\_grp\_myrank(gid)} returns the rank of the calling process within the group referred to by {\tt gid}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. MPI should provide a procedure to determine the member rank of a process within a group given the group identifier and a valid process identifier, e.g. {\tt rank = mpi\_grp\_rank(gid,pid)}. This procedure may ``fail'' if the process identified by {\tt pid} is not a member of the group identifier by {\tt gid}, and can be used to determine whether a given process is a member of a given group. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. MPI should provide a procedure to determine the process identifier of a group member given the group identifier and member rank, e.g. {\tt pid = mpi\_pid(gid,rank)}. This procedure should validate the returned process identifier. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. \subsubsection{Group Creation and Destruction} MPI will provide four methods for creation of groups, and a procedure to destroy an existing group. A group may be created by explicit definition of the pids of the members of the group. For example, {\tt gid = mpi\_grp\_definition(npids,pids)} where {\tt gid} is the identifier of the newly created group, {\tt npids} is the number of processes in the new group, and {\tt pids} is an array containing the (valid) process identifiers of the processes in the group. This procedure must be called my all members of the group, and does not return until all members have made the call. A group may be created by identical duplication of an existing group. For example, {\tt gidb = mpi\_grp\_duplication(gida)} where {\tt gidb} is the group identifier of the newly created group and {\tt gida} is the identifier of an existing group. The created group inherits all properties of the source group, including any topological properties. This operation has the same synchronisation properties as creation of group by definition. A group may be created by topological duplication of an existing group. Details of topological groups are under consideration within the process topologies subcommittee and will not be further discussed here. A group created by topological duplication inherits the size of the source group, and also inherits the membership list of the source group although the list may be ordered differently in the created group. These operations have the same synchronisation properties as creation of group by identical duplication. Where groups have additional topological attributes MPI should also provide procedures which allow the user to determine such attributes. One or more groups may be created by partition of an existing group into distinct subgroups by key. For example, {\tt gidb = mpi\_grp\_partition(gida, key)} where {\tt gidb} is the group identifier of the newly created group corresponding to the given {\tt key} and {\tt gida} is the group identifier of the group which is being partitioned according to the {\tt key} values supplied. MPI should define a named constant which is a {\it null\/} {\tt key} value, for example {\tt MPI\_KEY\_NULL}, in order that members of the parent group can choose not to become members of any child group, and in which case the procedure should return {\tt MPI\_GID\_NULL}. Groups created by partition share the same ordering of process member ranks as the parent group. This operation synchronises the members of the parent group, and therefore implicitly synchronises the members of the created group(s). A group may be destroyed, e.g. {\tt mpi\_grp\_deletion(gid)}, which destroys the group identified by {\tt gid}. This operation synchronises the group members, and invalidates the group identifier. \subsubsection{Point-to-point communication} There are arguments either way for point-to-point routines which accept a {\tt (gid, rank)} pair, and for routines which accept a {\tt pid}. This proposal supports and seperates both approaches to addressing and selection within the same syntax, avoiding the introduction of sets of procedures to handle each case. In order to provide expressive power to intra-group communication, point-to-point communication should accept a {\tt (gid, rank)} pair, where {\tt gid} is a valid group identifier and {\tt rank} is a member rank within the group identified by {\tt gid}. In message addressing the sender specifies the {\tt rank} and {\tt gid} of the receiver. In message selection the receiver specifies the {\tt rank} and {\tt gid} of the receiver. The sender and receiver must both specify the same {\tt gid} in order for a match to occur. The {\tt gid} field is not allowed to take a {\it wildcard\/} value in message selection. The {\tt rank} field is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_RANK\_WILD}, which will match with any rank. One is encouraged to visualise a seperate message queue or port at each process for each group of which that process is a member (and indeed this may be an advantageous implementation feature). In order to accommodate process identifier based addressing and selection into the same syntax, this proposal advocates that point-to-point communication should also accept the null group ({\tt gid = MPI\_GID\_NULL}), in which case the {\tt rank} is interpreted as a valid {\tt pid}. The {\tt pid} is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_PID\_WILD}. The point-to-point section should provide a procedure which allows a recipient to recover the process identifier of the sender. The discussion of matching above extends to this case, and one is also encouraged to visualise a separate message queue or port for messages referred to by {\tt MPI\_GID\_NULL}. In the case of process identified wildcard receive, the process identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the process identifier must be performed by the MPI implementation, in order that the recipient is returned a valid process identifier, else the returned identifier is of little or no use to the recipient. \subsubsection{Collective communication} Collective communication operations within MPI should be restricted to the scope of a single group. It will be sufficient for these procedures to accept a group identifier, and possibly a message tag in order to distinguish multiple outstanding operations within the same group. These procedures must not accept {\tt MPI\_GID\_NULL}. It is not possible to determine whether this is strategy allows all of the MPI collective communication routines to be written in terms of MPI point-to-point routines without loss of generality, since the set of collective communication routines is not yet determined. This proposal takes the view that it is the responsibility of the collective communications subcommittee to determine whether such a goal is desirable, and if so to describe procedures which comply with this goal. % % END "Basic Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Extended Proposal" % \subsection{Extended Proposal} The main additional features of the extended proposal are: \begin{itemize} \item Semantics for passing a group identifier in a message are defined, and a group registry is introduced. \item Point-to-point communication is extended to allow inter-group communication without syntactic intrusion on intra-group communication. \item Collective communication operations involving a concerted action on the part of members of two or more groups is suggested. \end{itemize} \subsubsection{Process and Group Identifiers} The extended proposal says nothing more about process identifiers than the basic proposal. The extended proposal defines a mechanism by which a group identifier may be passed in a message in such a manner that the received identifier is valid, and the mechanism should be analagous to that which allows process identifiers to be transmitted. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a group identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_gid(bd, \&gid)}. Transmission of a group identifier using this mechanism returns to the recipient a group identifier which is valid for use in the MPI operations described above and below, and as further qualified below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a group identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_invalidate\_gid(gid)}. This is an error if {\tt gid} is {\tt MPI\_GID\_NULL}, or if {\tt gid} is the identifier of a group of which the calling process is a member. It is further proposed that MPI provide a group identifier registry service. This service allows any group to register its own group identifer by name, and deregister its group identifier. The service allows any group to determine whether a name has been registered without blocking the calling group, and to map that name into a validated group identifier with the possibility of blocking the calling group. The group registry procedures should synchronise the calling group, permitting more efficient implementations than asynchronous operations. The definition of a valid group identifier is extended to include all groups of which the process is a member and all those which are validated by one of the above mechanisms. The small suite of procedures which map between process identifiers, group identifiers and member ranks are defined to work with valid group and process identifiers. \subsubsection{Point-to-point communication} The point-to-point communication syntax and semantics described in the basic proposal do not extend directly to the case of inter-group communication. The reason is quite simple --- the sender and receiver do not supply the same group since the group of the sender and the group of the receiver are different. The basic approach is to express point-to-point communication in terms of a triplet {\tt (localGroup, remoteGroup, remoteRank)}. In this notation the sender specifies the sender group identifier, then the receiver group identifier and finally the receiver rank. The receiver specifies the receiver group identifier, then the sender group identifier, and finally the sender rank. The {\tt localGroup} field may not take a wildcard value, corresponding directly to the rule that the group in the basic proposal may not take a wildcard value. The {\tt remoteGroup} field may take a wildcard value, e.g. {\tt MPI\_GID\_WILD}, which matches with any group. The {\tt remoteRank} may also take a wildcard value, as in the basic proposal. The point-to-point section should provide procedures which allow a message recipient to recover the group and rank of the sender. In the case of {\tt remoteGroup} wildcard receive, the group identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the group identifier must be performed by the MPI implementation, in order that the recipient is returned a valid group identifier, else the returned identifier is of little or no use to the recipient. In order to accomodate inter-group addressing and selection into the framework of the basic proposal, the extended proposal suggests a careful redefinition of the {\tt gid} discussed in the basic proposal. With careful presentation this redefinition need not intrude conceptually on the basic proposal, although I shall give a less careful description here, suggestive of two different flavours of presentation. In the basic proposal, the {\tt gid} was formally a reference to a group. In the extended proposal the {\tt gid} is formally composed of references to two groups, and can be thought of as a shorthand notation for {\tt (localGroup, remoteGroup)}. The identifier of the null group, a {\it null\/} identifier, is a valid identifier which is formally composed of a pair of references to the null group, and may be used in the fashion described in the basic proposal. The group creation functions provide a symmetric, or {\it unary\/}, identifier formally composed of two references to the same group. This group is {\it local\/} since the process in question is a member of the group. An identifier which composes a pair of references to a local group is logically identical to a group identifier as implied in the basic proposal, and may be used for point-to-point and collective communications in an identical fashion. The group identifier transmission and registry lookup procedures also provide a symmetric, or {\it unary\/}, identifier which again is composed of a pair of references to the same group. This group is {\it remote\/} when the process is not a member of the group, or is local when the process is a member of the group. An identifier which composes a pair of references to a remote group is logically identical to the identifier a remote group as implied above, and is not valid for either point-to-point or collective communications. Inter-group communication is effected by use of assymetric, or {\it binary} identifiers, composed of references to two different groups, the first of which is local and the second of which is either local or remote. Such identifiers are constructed by the use of a ``glob'' operation, which returns an identifier referencing the first operand as the local field and the second operand as the remote field. The glob is defined to be an error unless: both operands are symmetric (unary); the first operand refers to a local group. The identifier returned by a glob is valid for point-to-point communciation and invalid for collective communication. The point-to-point communication section should provide a procedure to determine the group identifiers object of a completed receive. Procedures which ``unglob'' an assymetric (binary) group identifier should be provided, returning the local and remote fields as valid identifiers. This can be viewed as a continuation of a generalised approach to point-to-point communication began in the basic proposal, in which operation are specified in terms of object identifiers and instance identifiers where objects are composed of multiple instances. The nature of the instance identified, and the semantics of the operation, are dependent of the nature of the object identified, exploiting some genericity. \begin{center} \begin{tabular}{lll} object identifier & instance identifier & action \\ \hline null group identifier & basic process identifier & basic communication \\ unary group identifier & local process rank & intra-group communication \\ binary group identifier & remote process rank & inter-group communication \\ \end{tabular} \end{center} \subsubsection{Collective communication} There are a number of collective communication operations which logically extend over two or more groups. Some examples of two-group collective communications which are common in the simple host-node programming model are: \begin{itemize} \item {\it Broadcast\/} is an implicitly assymetric operation. There is exactly one sender and there are many receivers, each of which receives an identical message. The sender is a member of a singleton group G and the receivers are members of a group H. \item {\it Scatter\/} is an implicitly assymetric operation. There is exactly one sender and there are may receivers, each of which receives a different message. The sender is a member of a singleton group G and the receivers are members of a group H. \item There is a variant of {\it gather\/}, which returns the gathered data to a single process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \item There is a variant of {\it reduce}, which returns the reduced data to a sinle process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \end{itemize} Other patterns arise in ``process'' graphs where each ``process'' is allowed to be parallel. For example: \begin{itemize} \item {\it all-to-all\/} communciation in which the senders and receivers are distinct processes. The senders are members of a group G and the receivers are members of a group H. The two groups G and H need not be of the same size. \end{itemize} The assymetric (binary) group identifier objects described for point-to-point communications in this extended proposal are immediately suitable for each of the two-group collective communication operations described. % % END "Extended Proposal" %----------------------------------------------------------------------% \section{Conclusion} This chapter has propose that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The chapter made a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The basic proposal provides expressive semantics for the case of intra-group communication such as arises in program which compose data driven parallelism, and is closely related to that which was discussed by Marc Snir at the February meeting in Dallas, and builds on discussions which have taken place in various subcommittees. The key additional features are: point-to-point communication can also be expressed in terms of process identifiers; a process registry service was added, use of which is optional. The extended proposal adds expressive semantics for the case of inter-group communication such as arises in programs which compose combinations of data and function driven parallelism. This functionality has been constructed in such a manner that there is no syntactic or performance intrusion on the content of the basic proposal, and the additional conceptual content can be presented seperately from a presentation of intra-group communication. % % END "Proposal I" %======================================================================% \end{document} >------------------------------ Cut Here ------------------------------< /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Mar 18 13:58:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16568; Thu, 18 Mar 93 13:58:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20541; Thu, 18 Mar 93 13:53:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 13:53:15 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20533; Thu, 18 Mar 93 13:53:14 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17730; Thu, 18 Mar 1993 13:53:13 -0500 Date: Thu, 18 Mar 1993 13:53:13 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303181853.AA17730@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Agenda Can the context subcommittee please get together and tell me when it wants to present its idea to the full group. Also is a separate context subcommittee session required at the meeting prior to the full group meeting on contexts. What would be the length of the session(s). Just let me know and I'll change the agenda accordingly. David From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:14:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17038; Thu, 18 Mar 93 14:14:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21495; Thu, 18 Mar 93 14:11:59 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:11:58 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21487; Thu, 18 Mar 93 14:11:57 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29366; Thu, 18 Mar 93 13:07:04 CST Date: Thu, 18 Mar 93 13:07:04 CST From: Tony Skjellum Message-Id: <9303181907.AA29366@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: Agenda We need to meet as a sub-committee, to vote on the priority of our internal proposals, which will also be presented beforehand to everyone, starting next Monday (night). We need about 2 hours for our own meeting time. For the meeting before the full committee, I think one hour is appropriate. Thank you, Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:14:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17042; Thu, 18 Mar 93 14:14:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21341; Thu, 18 Mar 93 14:10:06 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:10:03 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21325; Thu, 18 Mar 93 14:10:02 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29337; Thu, 18 Mar 93 13:03:12 CST Date: Thu, 18 Mar 93 13:03:12 CST From: Tony Skjellum Message-Id: <9303181903.AA29337@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu Subject: Proposals I & V Cc: mpi-context@cs.utk.edu Dear Context sub-committee.... we are alive and well, despite rumors of our demise :-), By 7pm EST on Monday, I would like this committee's accepted forms of Proposals I and V to be ready for repromulgation to the entire MPI list. This is necessary because we need to replace Marc Snir's straw CONTEXT proposals in pt2pt and collcomm; they are only supposed to be placeholders from what I understand, awaiting our efforts. From what I also have understood by speaking with Lyndon on telephone just now, Proposal I is our substantive replacement for these placeholders (but only one of the options for closure as we see it). For those of you who don't know, Rik sent out V a few days ago, but no commentary has been seen; Lyndon just sent out I (or I++ as he calls it ). This is to be followed by my presentation of III/IV by March 24, which we will discuss forthwith. There is no proposal 2 anymore. Please interact on I & V (I will comment as well), and help me to get these in shape by Monday, as stipulated. Please urge the organizers to include CONTEXT subcommittee in agenda as originally discussed in Dallas. We are no longer figuring as significantly as originally discussed/agreed, as far as I can tell, because of the chosen ordering of presentations (I am not sure we are even on the first day at all). Are we on the agenda now? Thanks, - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:16:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17146; Thu, 18 Mar 93 14:16:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21578; Thu, 18 Mar 93 14:14:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:14:00 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21570; Thu, 18 Mar 93 14:13:59 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29392; Thu, 18 Mar 93 13:09:05 CST Date: Thu, 18 Mar 93 13:09:05 CST From: Tony Skjellum Message-Id: <9303181909.AA29392@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: Agenda When is before pt2pt and collcomm does there's. - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:19:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17303; Thu, 18 Mar 93 14:19:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21794; Thu, 18 Mar 93 14:17:09 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:17:08 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21786; Thu, 18 Mar 93 14:17:07 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29426; Thu, 18 Mar 93 13:12:13 CST Date: Thu, 18 Mar 93 13:12:13 CST From: Tony Skjellum Message-Id: <9303181912.AA29426@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: Agenda Sorry for fragments... if "less controversial" topics could be discussed at beginning, context subcommittee could miss that general session to vote, followed by our presentation to the whole. If we do not present first, I am afraid our value to the whole will be lost. I know Lyndon agrees on this; I have not heard opinions other than fear that the "straw" context proposals now in pt2pt and collcomm would be used as the de facto choice. We have two of our three proposals in solid form. By Monday, they will go to everyone. We will be prioritizing these choices at our meeting at Dallas. I will not arrive till noon [arriving from Portland with Jack]. So, just making the opening bell, I see no hope for us to vote amongst ourselves, unless you discuss "other parts of MPI" at opening. Can this be done? Thanks, - Tony From owner-mpi-context@CS.UTK.EDU Fri Mar 19 06:37:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23767; Fri, 19 Mar 93 06:37:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09670; Fri, 19 Mar 93 06:36:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 06:36:51 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09648; Fri, 19 Mar 93 06:36:45 -0500 Date: Fri, 19 Mar 93 11:36:36 GMT Message-Id: <10215.9303191136@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposals V and I To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Dear Colleagues (1) Please do send comments on Proposal I++ :-) negative or positive as you think fit, as quickly as you can. I will receive and reply to email over the weekend for brief periods on both Saturday and Sunday, probably afternoon GMT. (2) I intend to give comments on Proposal V, thanks to Rik, this afternoon, most probably around 4pm GMT. (3) I distributed Proposal I++ as a chapter of LaTeX report document style --- I will place Proposal V into this form and post to the addressee list. This will happen before (2), real soon. Rik --- I will do this in such a way that you can read it as plain text without particular grief. Tony --- I am happy to do likewise with your proposal if you like. Hopefully this could make it easier to distribute to the whole committee as a single document, and at a later stage progress toward a draft for the MPI working document. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 07:38:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02144; Fri, 19 Mar 93 07:38:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12096; Fri, 19 Mar 93 07:37:50 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 07:37:49 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12088; Fri, 19 Mar 93 07:37:34 -0500 Date: Fri, 19 Mar 93 12:37:29 GMT Message-Id: <10292.9303191237@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal V, LaTeX To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu LaTeX-ified Proposal V of Rik. PostScript to follow. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. \item Communication both within and between groups seems conceptually straightforward. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. \item Communication between different groups may be considered awkward. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 07:40:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02169; Fri, 19 Mar 93 07:40:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12140; Fri, 19 Mar 93 07:39:47 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 07:39:46 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12112; Fri, 19 Mar 93 07:38:58 -0500 Date: Fri, 19 Mar 93 12:38:51 GMT Message-Id: <10301.9303191238@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal V To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu PostScript of LaTeX-ified Proposal V of Rik ---------------------------------------------------------------------- %!PS-Adobe-2.0 %%Creator: dvips 5.495 Copyright 1986, 1992 Radical Eye Software %%Title: context-rik.dvi %%CreationDate: Fri Mar 19 12:37:44 1993 %%Pages: 7 %%PageOrder: Ascend %%BoundingBox: 0 0 596 842 %%EndComments %DVIPSCommandLine: dvips context-rik %DVIPSSource: TeX output 1993.03.19:1234 %%BeginProcSet: tex.pro %! /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N} B /TR{translate}N /isls false N /vsize 11 72 mul N /@rigin{isls{[0 1 -1 0 0 0] concat}if 72 Resolution div 72 VResolution div neg scale isls{0 Resolution vsize 72 div mul TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{ CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N} B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook} if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255 {IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V{}B /RV statusdict begin /product where{ pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{ gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M} B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{ p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 39158280 55380996 1000 300 300 (/a/obsidian/disk/home/u36/lyndon/mpi/context-rik.dvi) @start /Fa 11 119 df<0020004001800380030006000E001C001C003C0038003800780078007800F800 F000F000F000F000F000F000F000F000F000F800780078007800380038003C001C001C000E0006 00030003800180004000200B297C9E13>40 D<800040003000380018000C000E00070007000780 0380038003C003C003C003E001E001E001E001E001E001E001E001E001E003E003C003C003C003 8003800780070007000E000C00180038003000400080000B297D9E13>I<7FFFFFE07FFFFFE078 1F81E0701F80E0601F8060E01F8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000 001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F80 00001F8000001F8000001F8000001F800007FFFE0007FFFE001C1C7E9B21>84 D98 D<000FF0000FF00001F00001F00001F00001F00001F0 0001F00001F00001F00001F001F9F00F07F01C03F03C01F07801F07801F0F801F0F801F0F801F0 F801F0F801F0F801F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C1B>100 D<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F800007800 007C00603C00601E00C00F038001FC0013127F9116>I<1E003F003F003F003F001E0000000000 0000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F 00FFE0FFE00B1E7F9D0E>105 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F8 00F8F800F8F800F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>111 DI114 D118 D E /Fb 11 119 df<000070000000007000000000F800000000F800000000F800000001FC00000001FC00000003 FE00000003FE00000003FE00000006FF000000067F0000000E7F8000000C3F8000000C3F800000 183FC00000181FC00000381FE00000300FE00000300FE00000600FF000006007F00000E007F800 00FFFFF80000FFFFF800018001FC00018001FC00038001FE00030000FE00030000FE000600007F 000600007F00FFE00FFFF8FFE00FFFF825227EA12A>65 D68 D<07FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F0003FF003FD F01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18167E951B >97 D<0001FE000001FE0000003E0000003E0000003E0000003E0000003E0000003E0000003E00 00003E0000003E0000003E0000003E0001FC3E0007FFBE000F81FE001F007E003E003E007E003E 007C003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E007C00 3E007C003E003E007E001E00FE000F83BE0007FF3FC001FC3FC01A237EA21F>100 D<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC00 00FC00007C00007C00007E00003E00181F00300FC07003FFC000FF0015167E951A>I<03FC1E0F FF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03C03E07C01F0F801FFF0013 FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC70003EF0001EF0001EF0001EF0 001E78003C7C007C3F01F80FFFE001FF0018217E951C>103 D<1C003F007F007F007F003F001C 000000000000000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F00FFE0FFE00B247EA310>105 D110 D<0FF3003FFF00781F00600700E00300E00300F00300FC00007FE0007FF800 3FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00C7F00011167E 9516>115 D<0180000180000180000180000380000380000780000780000F80003F8000FFFF00 FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8180 0F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>I118 D E /Fc 69 124 df<007E1F0001C1B1800303E3C00703C3C00E03C180 0E01C0000E01C0000E01C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0 000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01 C0000E01C0000E01C0000E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E 03C00E01800E00000E00000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C 17>I<007FC001C1C00303C00703C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C07FCFF8151D809C17>I<003F07E00001C09C18000380F018000701F03C00 0E01E03C000E00E018000E00E000000E00E000000E00E000000E00E000000E00E00000FFFFFFFC 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00 E01C007FC7FCFF80211D809C23>I<6060F0F0F8F8686808080808080810101010202040408080 0D0C7F9C15>34 D<60F0F8680808081010204080050C7C9C0C>39 D<004000800100020006000C 000C0018001800300030007000600060006000E000E000E000E000E000E000E000E000E000E000 E000E000600060006000700030003000180018000C000C00060002000100008000400A2A7D9E10 >I<800040002000100018000C000C000600060003000300038001800180018001C001C001C001 C001C001C001C001C001C001C001C001C0018001800180038003000300060006000C000C001800 10002000400080000A2A7E9E10>I<60F0F0701010101020204080040C7C830C>44 DI<60F0F06004047C830C>I<00010003000600060006000C000C000C00 18001800180030003000300060006000C000C000C0018001800180030003000300060006000C00 0C000C00180018001800300030003000600060006000C000C00010297E9E15>I<03C00C301818 300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E007600660 06700E300C300C18180C3007E0101D7E9B15>I<030007003F00C7000700070007000700070007 0007000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C 7C9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C00380070 006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<07E01830201C201C781E 780E781E381E001C001C00180030006007E00030001C001C000E000F000F700FF80FF80FF80FF0 0E401C201C183007E0101D7E9B15>I<000C00000C00001C00003C00003C00005C0000DC00009C 00011C00031C00021C00041C000C1C00081C00101C00301C00201C00401C00C01C00FFFFC0001C 00001C00001C00001C00001C00001C00001C0001FFC0121C7F9B15>I<300C3FF83FF03FC02000 2000200020002000200023E024302818301C200E000E000F000F000F600FF00FF00FF00F800E40 1E401C2038187007C0101D7E9B15>I<00F0030C06040C0E181E301E300C700070006000E3E0E4 30E818F00CF00EE006E007E007E007E007E007600760077006300E300C18180C3003E0101D7E9B 15>I<60F0F0600000000000000000000060F0F06004127C910C>58 D<60F0F060000000000000 0000000060F0F0701010101020204080041A7C910C>I<0FE03038401CE00EF00EF00EF00E000C 001C0030006000C000800180010001000100010001000100000000000000000000000300078007 8003000F1D7E9C14>63 D<000600000006000000060000000F0000000F0000000F000000178000 00178000001780000023C0000023C0000023C0000041E0000041E0000041E0000080F0000080F0 000180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E0004001E000C00 1F001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E06180 01801980070007800E0003801C0003801C00018038000180780000807800008070000080F00000 00F0000000F0000000F0000000F0000000F0000000F0000000F000000070000080780000807800 0080380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E >III<001F808000E0618001801980070007800E0003801C00 03801C00018038000180780000807800008070000080F0000000F0000000F0000000F0000000F0 000000F0000000F000FFF0F0000F80700007807800078078000780380007801C0007801C000780 0E00078007000B800180118000E06080001F80001C1E7E9C21>71 D73 D76 DII<003F800000E0E0000380380007001C000E00 0E001C0007003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F0 0001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C0780003C0380003803C000780 1C0007000E000E0007001C000380380000E0E000003F80001B1E7E9C20>II<003F800000E0E0000380380007001C000E000E001C0007003C000780380003807800 03C0780003C0700001C0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F0 0001E0700001C0780003C0780003C0380003803C0E07801C1107000E208E000720DC0003A0F800 00F0E020003FE0200000602000007060000078E000003FC000003FC000001F8000000F001B257E 9C20>II<07E0801C1980300580700380600180E00180E00080E00080E00080F000 00F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001C08001C08001 C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>I<7FFFFFC0700F01C0600F 00C0400F0040400F0040C00F0020800F0020800F0020800F0020000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 000F0000000F0000000F0000000F0000001F800003FFFC001B1C7F9B1E>I87 D91 D<08081010202040404040808080808080B0B0F8F8787830300D0C7A9C15>II<1FC000307000783800781C00301C00001C00001C0001FC000F1C00381C00701C00601C00E0 1C40E01C40E01C40603C40304E801F870012127E9115>97 DI<07E00C301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E 127E9112>I<003F00000700000700000700000700000700000700000700000700000700000700 03E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E00700600700 700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E6006E006FFFEE0 00E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C071E061E0E0C0E000E 000E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C00381C00381C00381C 001818001C38000C300013C0001000003000001800001FF8001FFF001FFF803003806001C0C000 C0C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C001C001C001C001C00 1C001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C00000 0000000000000000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>IIIII<03F0 000E1C00180600300300700380600180E001C0E001C0E001C0E001C0E001C0E001C06001807003 803003001806000E1C0003F00012127F9115>II<03C1000C3300180B00300F00 700700700700E00700E00700E00700E00700E00700E00700600700700700300F00180F000C3700 07C700000700000700000700000700000700000700000700003FE0131A7E9116>II<1F 9030704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608FC00D127F91 10>I<04000400040004000C000C001C003C00FFE01C001C001C001C001C001C001C001C001C00 1C101C101C101C101C100C100E2003C00C1A7F9910>IIII<7F8FF00F03800F030007 020003840001C80001D80000F00000700000780000F800009C00010E00020E000607000403801E 07C0FF0FF81512809116>II<7FFC70386038407040F040E041C003C003800700 0F040E041C043C0C380870087038FFF80E127F9112>II E /Fd 1 16 df<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF80FF003C0 10107E9115>15 D E /Fe 33 122 df<0003E0000000000FF0000000003E380000000078180000 0000780C00000000F80C00000000F00C00000001F00C00000001F00C00000001F01C00000001F8 1800000001F83000000001F87000000001F86000000001F8C001FFE000FD8001FFE000FF00001C 0000FE0000180000FE00003000007E00003000007F0000600000FF0000600001FF8000C00003BF 80018000071FC00180000F0FE00300001E0FE00600003E07F00600007E03F80C0000FE03FC1800 00FE01FE300000FE00FE600000FE007FC00000FE003F8000C07F001FC000C07F000FF001C03F80 1FF803801FC0F0FE0F0007FFC03FFE0001FE0007F8002B287DA732>38 D<3C7EFFFFFFFF7E3C08 087B8712>46 D<000C00001C0000FC000FFC00FFFC00F0FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC00FFFFFCFFFF FC16257BA420>49 D<00FF000007FFE0001E07F8003801FC007800FE007E00FF00FF007F00FF00 7F80FF007F80FF003F807E003F803C003F8000007F8000007F0000007F000000FE000000FE0000 01FC000001F8000003F0000007C00000078000000F0000001E0000003800000070018000E00180 01C001800380030007000300060003000FFFFF001FFFFF003FFFFF007FFFFE00FFFFFE00FFFFFE 0019257DA420>I<00FF800007FFF0000F03F8001800FC003E00FE007F00FF007F007F007F807F 007F007F003F00FF001E00FE000000FE000000FC000001F8000003F0000007E00000FF800000FF 00000003E0000001F8000000FC000000FE0000007F0000007F0000007F8018007F807E007F80FF 007F80FF007F80FF007F80FF007F00FE00FF007C00FE003801FC001E03F8000FFFE00001FF0000 19257DA420>I<0000380000007800000078000000F8000001F8000003F8000003F8000007F800 000FF800001DF8000019F8000031F8000071F8000061F80000C1F80001C1F8000381F8000301F8 000601F8000E01F8001C01F8001801F8003001F8007001F800E001F800FFFFFFE0FFFFFFE00001 F8000001F8000001F8000001F8000001F8000001F8000001F8000001F800007FFFE0007FFFE01B 257EA420>I<08000C000F007C000FFFF8000FFFF0000FFFE0000FFFC0000FFF80000FFE00000C 0000000C0000000C0000000C0000000C0000000C0000000C7F80000DFFE0000F81F8000E007C00 0C007E0000003E0000003F0000003F0000003F8000003F8038003F807C003F80FE003F80FE003F 80FE003F80FE003F007C003F0060007E0030007C001800F8000F03F00007FFC00001FF00001925 7DA420>I<000FE000007FF80001F81C0003E00E0007C03F000F807F001F807F001F007F003F00 7F003F003E007E0000007E0000007E000000FE040000FE3FE000FE7FF000FEC0F800FE807C00FF 003E00FF003F00FF003F00FE003F80FE003F80FE003F80FE003F807E003F807E003F807E003F80 7E003F803E003F003E003F001F003E001F007E000F807C0007C1F80001FFE000007F800019257D A420>I<00000600000000000F00000000000F00000000001F80000000001F80000000001F8000 0000003FC0000000003FC0000000007FE0000000006FE0000000006FE000000000CFF000000000 C7F000000001C7F80000000183F80000000183F80000000303FC0000000301FC0000000701FE00 00000600FE0000000600FE0000000C007F0000000C007F0000001C007F80000018003F80000018 003F8000003FFFFFC000003FFFFFC0000070001FE0000060000FE0000060000FE00000C00007F0 0000C00007F00001800007F80001800003F80001800003F80003000001FC0007000001FC00FFF8 003FFFF0FFF8003FFFF02C287EA731>65 D<0000FF80080007FFF018003FC03C38007E000E7801 FC0003F803F00001F807E00000F80FE00000781FC00000781FC00000383F800000383F80000038 7F800000187F000000187F00000018FF00000000FF00000000FF00000000FF00000000FF000000 00FF00000000FF00000000FF00000000FF00000000FF000000007F000000007F000000187F8000 00183F800000183F800000181FC00000301FC00000300FE000006007E000006003F00000C001FC 000180007E000700003FC03C000007FFF8000000FFC00025287CA72E>67 DII73 D78 D80 D<00FF004007FFE0C00F80F9 C01E001FC03C000FC07C0003C0780003C0780001C0F80001C0F80000C0F80000C0FC0000C0FC00 0000FE0000007F8000007FF800003FFFC0003FFFF8001FFFFC000FFFFE0003FFFF0000FFFF8000 0FFFC00000FFC000001FC000000FE0000007E0000007E0C00003E0C00003E0C00003E0C00003E0 E00003C0E00003C0F0000780F8000780FE000F00E7C03E00C1FFF800803FE0001B287CA724>83 D<03FF00000FFFE0001F03F0003F80F8003F80FC003F807C001F007E001F007E0000007E000000 7E0000007E00000FFE0001FFFE0007F07E001FC07E003F807E007F007E00FE007E00FE007E18FE 007E18FE007E18FE00BE187F01BE183F873FF01FFE1FE003F80F801D1A7E9920>97 D<00007FE000007FE0000007E0000007E0000007E0000007E0000007E0000007E0000007E00000 07E0000007E0000007E0000007E0000007E0007F87E001FFE7E007E077E00FC01FE01F800FE03F 0007E03F0007E07E0007E07E0007E0FE0007E0FE0007E0FE0007E0FE0007E0FE0007E0FE0007E0 FE0007E0FE0007E07E0007E07E0007E07F0007E03F0007E01F000FE00F801FE007E077E003FFC7 FE007F07FE1F287EA724>100 D<007F8003FFE007E1F00F80F81F007C3F007E7E003E7E003E7E 003FFE003FFE003FFFFFFFFFFFFFFE0000FE0000FE0000FE00007E00007E00003F00003F00031F 80060FC00607F01C01FFF0003FC0181A7E991D>I<00FE03E007FFCFF00F83F8F01F01F0F03E00 F8E03E00F8007E00FC007E00FC007E00FC007E00FC007E00FC003E00F8003E00F8001F01F0000F 83E0000FFFC00010FE00001000000038000000380000003C0000003FFFF0001FFFFE001FFFFF00 0FFFFF801FFFFFC03C001FC07C0007E0F80003E0F80003E0F80003E0F80003E07C0007C07C0007 C03E000F801FC07F0007FFFC0000FFE0001C267E9920>103 D<0F001F801FC03FC03FC01FC01F 800F000000000000000000000000000000FFC0FFC00FC00FC00FC00FC00FC00FC00FC00FC00FC0 0FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC0FFFCFFFC0E297FA811>105 D108 DII< 007FC00001FFF00007E0FC000F803E001F001F003F001F803E000F807E000FC07E000FC0FE000F E0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE07E000FC07E000FC07F00 1FC03F001F801F001F000F803E0007E0FC0001FFF000007FC0001B1A7E9920>II 114 D<03F8C01FFFC03C07C07801C07001C0F000C0F000C0F800C0FC0000FFC0007FFC003FFF00 1FFF800FFFC001FFE0000FF00003F0C001F0C000F0E000F0E000F0F000E0F801E0FE03C0E7FF80 C1FC00141A7E9919>I<00600000600000600000600000E00000E00001E00001E00003E00007E0 001FE000FFFFC0FFFFC007E00007E00007E00007E00007E00007E00007E00007E00007E00007E0 0007E00007E00007E00007E00007E06007E06007E06007E06007E06007E06003F0C001F0C000FF 80003E0013257FA419>III120 DI E /Ff 8 116 df80 D86 D<0007FFC0000000003FFFFC00 000000FFFFFF00000003F801FFC0000007F0003FE0000007F8001FF000000FFC000FF800000FFC 000FFC00000FFC0007FC00000FFC0007FE00000FFC0003FE000007F80003FF000003F00003FF00 0000000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000000000 03FF000000000003FF0000000007FFFF00000000FFFFFF0000000FFFE3FF0000007FF803FF0000 01FFC003FF000003FF0003FF000007FC0003FF00000FF80003FF00001FF00003FF00003FF00003 FF00007FE00003FF00007FE00003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FF C00003FF0380FFC00007FF0380FFC00007FF03807FE0000DFF03807FE0001DFF03803FF00039FF 87001FF80070FFCF000FFE03E07FFE0007FFFF807FFC0000FFFE001FF800001FF00007E000312E 7CAD37>97 D<00FF80FFFF80FFFF80FFFF8003FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF80FFFFFFFFFFFFFFFFFF18487DC71F>108 D<00003FFC00000001FFFF8000000FFFFFF000003FF00FFC00007F8001FE0001FE00007F8003FC 00003FC007F800001FE007F800001FE00FF000000FF01FE0000007F81FE0000007F83FE0000007 FC3FE0000007FC7FC0000003FE7FC0000003FE7FC0000003FE7FC0000003FEFFC0000003FFFFC0 000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003 FFFFC0000003FFFFC0000003FF7FC0000003FE7FC0000003FE7FC0000003FE7FE0000007FE3FE0 000007FC3FE0000007FC1FE0000007F81FF000000FF80FF000000FF007F800001FE007FC00003F E003FE00007FC001FF0000FF80007F8001FE00003FF00FFC00000FFFFFF0000003FFFFC0000000 3FFC0000302E7DAD37>111 D<00FF801FF80000FFFF80FFFF0000FFFF83FFFFC000FFFF8FC07F F00003FF9E000FF80001FFF80007FC0001FFF00003FE0001FFE00001FF0001FFC00000FF8001FF 800000FFC001FF8000007FC001FF8000007FE001FF8000007FE001FF8000003FE001FF8000003F F001FF8000003FF001FF8000003FF001FF8000001FF801FF8000001FF801FF8000001FF801FF80 00001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF8 01FF8000001FF801FF8000001FF801FF8000001FF001FF8000003FF001FF8000003FF001FF8000 003FF001FF8000003FE001FF8000007FE001FF8000007FC001FF800000FFC001FF800000FF8001 FFC00001FF8001FFE00001FF0001FFF00003FE0001FFF8000FFC0001FF9E001FF80001FF8F80FF E00001FF87FFFFC00001FF81FFFE000001FF803FF0000001FF800000000001FF800000000001FF 800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80000000 0001FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80 0000000001FF800000000001FF800000000001FF8000000000FFFFFF00000000FFFFFF00000000 FFFFFF0000000035427DAD3D>I<00FF007F00FFFF01FFC0FFFF03FFE0FFFF0787F003FF0E0FF0 01FF1C1FF801FF381FF801FF301FF801FF701FF801FF600FF001FF600FF001FFE003C001FFC000 0001FFC0000001FFC0000001FFC0000001FF80000001FF80000001FF80000001FF80000001FF80 000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF 80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001 FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF800000FFFFFFC000 FFFFFFC000FFFFFFC000252E7DAD2C>114 D<001FFC030000FFFF870003FFFFCF000FE007FF00 1F8000FF003E00003F003E00001F007C00001F007C00000F00FC00000F00FC00000700FC000007 00FE00000700FF00000700FF80000000FFE00000007FFE0000007FFFF800003FFFFF00003FFFFF C0001FFFFFF0000FFFFFF80007FFFFFC0001FFFFFE00007FFFFF00001FFFFF000000FFFF800000 07FF80000000FFC0E000007FC0E000003FC0E000001FC0F000000FC0F000000FC0F800000FC0F8 00000FC0F800000F80FC00000F80FE00001F80FF00001F00FF80003E00FFC0007C00F9F803F800 F0FFFFF000E03FFFC000C007FE0000222E7CAD2B>I E /Fg 8 117 df<00003000000070000001 F0000007F000007FF000FFFFF000FFFFF000FF8FF000000FF000000FF000000FF000000FF00000 0FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000 000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF0 00000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000F F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00000 0FF0007FFFFFFE7FFFFFFE7FFFFFFE1F3779B62D>49 D<000000FFE0001800000FFFFC00180000 7FFFFF00380001FFE00FC0780007FE0001F0F8000FF8000079F8003FE000003FF8007FC000000F F800FF80000007F801FF00000007F803FE00000003F807FC00000001F807F800000001F80FF800 000000F80FF000000000F81FF000000000781FF000000000783FE000000000783FE00000000038 7FE000000000387FE000000000387FE000000000387FC000000000007FC00000000000FFC00000 000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FF C00000000000FFC00000000000FFC00000000000FFC00000000000FFC000000000007FC0000000 00007FC000000000007FE000000000007FE000000000387FE000000000383FE000000000383FE0 00000000381FF000000000381FF000000000700FF000000000700FF8000000007007F800000000 E007FC00000000E003FE00000001C001FF000000038000FF8000000780007FC000000F00003FE0 00001E00000FF800003C000007FE0000F0000001FFE007E00000007FFFFF800000000FFFFE0000 000000FFE00000353B7BB940>67 D<003FFC00000001FFFF80000007E00FE000000FC003F80000 0FE001FC00001FF001FE00001FF000FF00001FF000FF00001FF0007F00000FE0007F800007C000 7F80000000007F80000000007F80000000007F80000000007F80000000007F800000000FFF8000 0007FFFF8000003FE07F800001FF007F800007FC007F80000FF0007F80001FE0007F80003FE000 7F80007FC0007F80007FC0007F8380FF80007F8380FF80007F8380FF80007F8380FF8000BF8380 FF8000BF83807FC0013F83807FC0033F83803FE0061FC7001FF81C0FFE0007FFF007FC00007FC0 03F00029257DA42D>97 D<0003FF0000001FFFE000007F07F00001FC01FC0003F000FE0007E000 7F000FE0003F001FC0003F801FC0003F803FC0001FC03F80001FC07F80001FC07F80001FE07F80 001FE0FF80001FE0FF80001FE0FFFFFFFFE0FFFFFFFFE0FF80000000FF80000000FF80000000FF 80000000FF800000007F800000007F800000007F800000003FC00000003FC00000001FC00000E0 1FE00000E00FE00001C007F000038003F800070000FE000E00007FC07C00001FFFF0000001FF80 0023257EA428>101 D<01FC00000000FFFC00000000FFFC00000000FFFC0000000007FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC01FF000003FC07FFC00003FC1C0FF00003FC 3007F80003FC6003F80003FCC003FC0003FC8001FC0003FD0001FE0003FF0001FE0003FE0001FE 0003FE0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE00FFFFF07FFFF8FFFFF07FFF F8FFFFF07FFFF82D3A7EB932>104 D<01FC03FE0000FFFC1FFFC000FFFC780FF000FFFDE003F8 0007FF8001FC0003FF0000FE0003FE0000FF0003FC00007F8003FC00007FC003FC00003FC003FC 00003FE003FC00003FE003FC00001FE003FC00001FE003FC00001FF003FC00001FF003FC00001F F003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC 00001FE003FC00003FE003FC00003FE003FC00003FC003FC00003FC003FC00007F8003FC00007F 8003FE0000FF0003FF0001FE0003FF8003FC0003FDC007F80003FCF81FE00003FC3FFF800003FC 07FC000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 00000000FFFFF0000000FFFFF0000000FFFFF00000002C357EA432>112 D<01F80FC0FFF83FF0FFF870F8FFF8C1FC07F883FE03F983FE03F903FE03FB03FE03FA01FC03FA 00F803FA007003FE000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC0000 03FC000003FC000003FC000003FC000003FC0000FFFFF800FFFFF800FFFFF8001F257EA424> 114 D<001C0000001C0000001C0000001C0000001C0000003C0000003C0000003C0000003C0000 007C0000007C000000FC000000FC000001FC000003FC000007FC00001FFFFFC0FFFFFFC0FFFFFF C003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC 000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E001FC00C001FC01C0 01FE01C000FE0380007F8700001FFE000007F8001B357EB423>116 D E /Fh 17 117 df<001FC0000070200000C010000180380003807800070078000700300007000000 070000000700000007000000070000000700000007000000FFFFF8000700780007003800070038 000700380007003800070038000700380007003800070038000700380007003800070038000700 38000700380007003800070038000700380007003800070038007FE1FF80192380A21B>12 D<008003800F80F380038003800380038003800380038003800380038003800380038003800380 03800380038003800380038003800380038003800380038007C0FFFE0F217CA018>49 D<03F8000C1E001007002007804007C07807C07803C07807C03807C0000780000780000700000F 00000E0000380003F000001C00000F000007800007800003C00003C00003E02003E07003E0F803 E0F803E0F003C04003C0400780200780100F000C1C0003F00013227EA018>51 D<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0F001E0F001E0F001 E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001C00001C00001C00003 80000380300300780700780600700C002018001030000FC00013227EA018>57 D76 DI82 D<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7001E07003C07007807 00700700F00708F00708F00708F00F087817083C23900FC1E015157E9418>97 D<01FE000703000C07801C0780380300780000700000F00000F00000F00000F00000F00000F000 00F000007000007800403800401C00800C010007060001F80012157E9416>99 D<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000 E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I<01FC0007 07000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000F0000070 00007800203800201C00400E008007030000FC0013157F9416>I<0E0000FE00001E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E0070FFE7FF18237FA21B>104 D<1C001E003E001E001C000000000000 00000000000000000000000E00FE001E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E00FFC00A227FA10E>I<0E0000FE00001E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E01800E0200 0E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E0 0E00F00E00F8FFE3FE17237FA21A>107 D<0E00FE001E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E00FFE00B237FA20E>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E00 0E000E000E000E000E000E000E000E000F00FFF010157F9413>114 D<02000200020002000600 060006000E001E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E040E040E 040E040E040E040708030801F00E1F7F9E13>116 D E /Fi 24 121 df<7803C0FC07E0FC07E0 FE07F0FE07F07A03D0020010020010020010020010020010040020040020040020080040080040 10008020010020010040020014147EB020>34 D<00003FC0020001FFF8060007E01C06001F0007 0E003C00018E00780000DE00F000005E01E000003E03C000001E078000001E0F0000000E0F0000 000E1E000000061E000000063E000000063C000000027C000000027C000000027C000000027800 000000F800000000F800000000F800000000F800000000F800000000F800000000F800000000F8 00000000F800000000F80000000078000000007C000000007C000000027C000000023C00000002 3E000000021E000000021E000000040F000000040F0000000C078000000803C000001801E00000 1000F00000200078000040003C000180001F0003000007E01E000001FFF80000003FC00027327C B02F>67 D73 D77 D80 D<007F004001FFC0400780F0C00F0018C01C000DC03C0007C0 380003C0780001C0700001C0F00000C0F00000C0F00000C0F0000040F0000040F0000040F80000 40F80000007C0000007E0000003F0000003FE000001FFE00000FFFE00007FFF80001FFFE00007F FF000007FF8000007F8000000FC0000007E0000003E0000003E0000001F0000001F0800000F080 0000F0800000F0800000F0800000F0C00000F0C00000E0E00001E0E00001E0F00001C0F8000380 EC000780C7000F00C1E03E0080FFF800801FE0001C327CB024>83 D86 D<040020080040080040100080200100200100400200 400200400200800400800400800400800400800400BC05E0FE07F0FE07F07E03F07E03F03C01E0 141476B020>92 D<00FE0000070380000801E0001000F0003C0078003E0078003E003C003E003C 001C003C0000003C0000003C0000003C000007FC00007C3C0003E03C0007803C001F003C003E00 3C003C003C007C003C0078003C08F8003C08F8003C08F8003C08F8007C08F8007C087C00BC083E 011E100F060FE003F807C01D1E7D9D20>97 D<018000003F800000FF800000FF8000000F800000 078000000780000007800000078000000780000007800000078000000780000007800000078000 00078000000780000007800000078000000781F80007860700079803C007A000E007C000F007C0 0078078000380780003C0780003E0780001E0780001E0780001F0780001F0780001F0780001F07 80001F0780001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C00070 074000F0072001C006100380060C0E000403F80020317EB024>I<001FC00000F0380001C00400 078002000F000F001E001F001E001F003C001F007C000E007C00000078000000F8000000F80000 00F8000000F8000000F8000000F8000000F8000000F8000000780000007C0000007C0000003C00 00801E0000801E0001000F0001000780020001C00C0000F03000001FC000191E7E9D1D>I<003F 800000E0F0000380380007001C000E001E001E000E003C000F003C000F007C000F807800078078 000780F8000780FFFFFF80F8000000F8000000F8000000F8000000F8000000F800000078000000 780000007C0000003C0000801C0000801E0001000F0002000700020001C00C0000F03000001FC0 00191E7E9D1D>101 D<0007E0001C1000383800707C00E07C01E07C01C03803C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E0007FFF007FFF 0016317FB014>I<07000F801F801F800F80070000000000000000000000000000000000000000 00000001801F80FF80FF800F800780078007800780078007800780078007800780078007800780 0780078007800780078007800780078007800FC0FFF8FFF80D2F7EAE12>105 D<01803F80FF80FF800F8007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 8007800780078007800780078007800FC0FFFCFFFC0E317EB012>108 D<0181FE003FC0003F86 0780C0F000FF8801C1003800FF9001E2003C000FA000E4001C0007A000F4001E0007C000F8001E 0007C000F8001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E00078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FF FC1FFF83FFF0341E7E9D38>I<0181FC003F860F00FF880380FF9003C00FA001C007C001E007C0 01E007C001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E007 8001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0 078001E00FC003F0FFFC3FFFFFFC3FFF201E7E9D24>I<003FC00000E0700003801C0007000E00 0E0007001E0007803C0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001 F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E0780001E07C0003E03C0003C03C00 03C01E0007800E00070007000E0003801C0000E07000003FC0001C1E7E9D20>I<0181F8003F86 0F00FF9803C0FFA001E007C000F007C00078078000780780003C0780003E0780003E0780001E07 80001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001E0780003E 0780003C0780003C0780007807C000F007C000F007A001C007900380078C0E000783F800078000 000780000007800000078000000780000007800000078000000780000007800000078000000780 00000FC00000FFFC0000FFFC0000202C7E9D24>I<0183E03F8C18FF907CFF907C0FA07C07C038 07C00007C00007C000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000780000780000FC000FFFE00FFFE00161E7E9D19> 114 D<03FC200E02603801E03000E0600060E00060E00020E00020F00020F000207C00007F8000 3FFC001FFF0007FF8001FFE0000FE00001F08000F8800078800038C00038C00038C00038E00030 F00070F00060C800C0C6038081FE00151E7E9D19>I<00400000400000400000400000400000C0 0000C00000C00001C00001C00003C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C01003C0 1003C01003C01003C01003C01003C01003C01001C02001E02000E0400078C0001F00142B7FAA19 >I<018000603F800FE0FF803FE0FF803FE00F8003E0078001E0078001E0078001E0078001E007 8001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0 078001E0078001E0078001E0078003E0078003E0078003E0038005E003C009E001C019F000F021 FF001FC1FF201E7E9D24>I120 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%PaperSize: A4 %%EndSetup %%Page: 0 1 0 0 bop 302 1086 a Fi(\\Prop)r(osal)24 b(V")e(for)f(MPI)i(Comm)n(unication)d (Con)n(text)774 1178 y(Sub)r(committee)829 1360 y Fh(Rik)15 b(Little\014eld)853 1481 y(Marc)o(h)h(1993)p eop %%Page: 1 2 1 1 bop 262 619 a Fg(Chapter)28 b(1)262 826 y Ff(Prop)s(osal)36 b(V)262 1067 y Fe(1.1)64 b(Summary)324 1158 y Fd(\017)20 b Fc(Group)i(ID's)g(ha)o(v)o(e)g(lo)q(cal)f(scop)q(e,)k(not)d(global.)41 b(Groups)22 b(are)h(represen)o(ted)i(b)o(y)365 1208 y(descriptors)f(of)d (inde\014nite)h(size)g(and)g(opaque)f(structure,)k(managed)20 b(b)o(y)i(MPI.)365 1257 y(Group)13 b(descriptors)j(can)d(b)q(e)h(passed)h(b)q (et)o(w)o(een)g(pro)q(cesses)h(b)o(y)d(MPI)h(routines)g(that)365 1307 y(translate)h(them)e(as)h(necessary)m(.)365 1374 y(\(This)j(satis\014es) g(the)g(same)e(needs)j(as)e(global)f(group)h(ID's,)g(but)g(its)h(p)q (erformance)365 1423 y(implications)11 b(are)k(easier)f(to)g(understand)h (and)f(con)o(trol.\))324 1506 y Fd(\017)20 b Fc(Arbitrary)14 b(pro)q(cess-lo)q(cal)h(info)e(can)h(b)q(e)h(cac)o(hed)g(in)e(group)h (descriptors.)365 1573 y(\(This)i(allo)o(ws)e(collectiv)o(e)h(op)q(erations)h (to)f(run)h(quic)o(kly)e(after)i(the)g(\014rst)g(execution)365 1623 y(in)e(eac)o(h)g(group,)f(without)h(complicating)d(the)k(calling)d (sequences.\))324 1706 y Fd(\017)20 b Fc(There)15 b(are)g(restrictions)g (that)f(p)q(ermit)f(groups)h(to)g(b)q(e)g(la)o(y)o(ered)g(on)g(top)g(of)f (pt-pt.)324 1789 y Fd(\017)20 b Fc(Pt-pt)c(comm)o(unications)d(use)k(only)e (TID,)g(con)o(text,)h(and)g(tag,)f(and)h(are)g(sp)q(eci\014ed)365 1839 y(to)e(b)q(e)h(fast.)324 1922 y Fd(\017)20 b Fc(Group)c(con)o(trol)g(op) q(erations)g(\(forming,)e(disbanding,)i(and)g(translating)f(b)q(et)o(w)o(een) 365 1971 y(rank)f(and)g(TID\))f(are)i(NOT)f(sp)q(eci\014ed)h(to)f(b)q(e)h (fast.)262 2109 y Fe(1.2)64 b(Detailed)24 b(Prop)r(osal)324 2200 y Fd(\017)c Fc(Pt-pt)d(uses)h(only)e(\\TID",)g(\\con)o(text",)h(and)f (\\message)h(tag".)25 b(TID's,)17 b(con)o(texts,)365 2249 y(and)12 b(message)f(tags)h(ha)o(v)o(e)g(global)e(scop)q(e;)j(they)f(can)g(b)q(e)h (copied)f(b)q(et)o(w)o(een)h(pro)q(cesses)365 2299 y(as)i(in)o(tegers.)21 b(TID's)14 b(and)h(con)o(texts)h(are)f(opaque;)f(their)h(v)n(alues)g(are)g (managed)e(b)o(y)365 2349 y(MPI.)h(Message)h(tags)f(are)g(con)o(trolled)g(en) o(tirely)g(b)o(y)g(the)g(application.)967 2574 y(1)p eop %%Page: 2 3 2 2 bop 324 307 a Fd(\017)20 b Fc(Pt-pt)15 b(comm)o(unication)c(is)j(sp)q (eci\014ed)i(to)e(b)q(e)h(fast)g(in)f(all)f(cases.)21 b(\(E.g.,)13 b(MPI)i(m)o(ust)365 357 y(initialize)i(all)h(pro)q(cesses)j(suc)o(h)e(that)f (an)o(y)g(required)i(translation)e(of)g(the)h(TID)f(is)365 407 y(faster)d(than)f(the)g(fastest)h(pt-pt)f(comm)o(unication)c(call.\))324 490 y Fd(\017)20 b Fc(A)h(group)f(is)g(represen)o(ted)j(b)o(y)d(a)g(\\group)f (descriptor",)k(of)c(inde\014nite)i(size)g(and)365 540 y(opaque)c(structure,) j(managed)15 b(b)o(y)i(MPI.)f(The)i(group)f(descriptor)h(can)f(b)q(e)h (refer-)365 589 y(enced)e(via)e(a)h(\\pro)q(cess-lo)q(cal)g(group)f(ID",)g (whic)o(h)h(is)f(t)o(ypically)g(just)g(a)h(p)q(oin)o(ter)g(to)365 639 y(the)g(group)f(descriptor.)19 b(\(Group)14 b(ID's)f(with)h(global)e (scop)q(e)j(do)e(not)h(exist.\))324 722 y Fd(\017)20 b Fc(Group)f (descriptors)i(can)e(b)q(e)h(copied)f(b)q(et)o(w)o(een)i(arbitrary)e(pro)q (cesses)i(via)e(MPI-)365 772 y(pro)o(vided)14 b(routines)g(that)g(translate)g (them)f(to)g(and)g(from)f(mac)o(hine-)g(and)i(pro)q(cess-)365 822 y(indep)q(enden)o(t)e(form.)j(A)10 b(pro)q(cess)i(receiving)f(a)f(group)g (descriptor)i(do)q(es)f(not)f(b)q(ecome)365 872 y(a)h(mem)o(b)q(er)f(of)h (the)h(group,)f(but)h(do)q(es)g(b)q(ecome)f(able)g(to)h(obtain)e(information) f(ab)q(out)365 922 y(the)15 b(group)f(\(e.g.,)e(to)i(translate)g(b)q(et)o(w)o (een)i(rank)d(and)h(TID\).)324 1005 y Fd(\017)20 b Fc(Arbitrary)15 b(information)d(can)i(b)q(e)i(cac)o(hed)f(in)f(a)g(group)h(descriptor.)21 b(This)14 b(is,)g(MPI)365 1054 y(routines)d(are)f(pro)o(vided)g(that)g(allo)o (w)f(an)o(y)g(routine)h(to)g(augmen)o(t)f(a)g(group)h(descriptor)365 1104 y(with)17 b(arbitrary)g(information,)e(iden)o(ti\014ed)i(b)o(y)g(a)g(k)o (ey)g(v)n(alue,)g(that)g(subsequen)o(tly)365 1154 y(can)f(b)q(e)g(retriev)o (ed)h(b)o(y)e(an)o(y)h(routine)f(ha)o(ving)g(access)i(to)f(the)g(descriptor.) 24 b(Cac)o(hed)365 1204 y(information)10 b(is)j(not)g(visible)f(to)h(other)h (pro)q(cesses)h(and)e(is)g(stripp)q(ed)h(when)f(a)g(group)365 1254 y(descriptor)18 b(is)e(sen)o(t)h(to)g(another)f(pro)q(cess.)28 b(Retriev)n(al)15 b(of)h(cac)o(hed)h(information)d(is)365 1303 y(sp)q(eci\014ed)d(to)f(b)q(e)g(fast)g(\(signi\014can)o(tly)f(c)o(heap)q(er)i (than)e(a)h(pt-pt)g(comm)o(uni)o(cation)d(call\).)324 1386 y Fd(\017)20 b Fc(Group)e(descriptors)h(con)o(tain)e(enough)h(information)d (to)i(async)o(hronously)h(trans-)365 1436 y(late)c(b)q(et)o(w)o(een)h (\(group,rank\))f(and)g(TID)g(for)f(an)o(y)h(mem)o(b)q(er)e(of)h(the)i (group,)e(b)o(y)h(an)o(y)365 1486 y(pro)q(cess)j(holding)c(a)h(descriptor)h (for)g(the)g(group.)k(MPI)c(routines)g(are)f(pro)o(vided)h(to)365 1536 y(do)f(these)h(translations.)j(T)m(ranslation)13 b(is)g(not)h(sp)q (eci\014ed)i(to)d(b)q(e)i(fast.)324 1619 y Fd(\017)20 b Fc(A)o(t)14 b(creation,)g(eac)o(h)h(group)f(is)f(assigned)i(a)f(globally)d(unique)j (\\default)g(group)g(con-)365 1669 y(text")g(whic)o(h)g(is)g(k)o(ept)g(in)f (the)i(group)e(descriptor)j(and)d(can)h(b)q(e)h(quic)o(kly)d(extracted.)365 1719 y(This)i(extraction)g(is)g(sp)q(eci\014ed)i(to)d(b)q(e)i(fast)f (\(comparable)e(to)i(a)g(pro)q(cedure)h(call\).)324 1802 y Fd(\017)20 b Fc(The)d(default)g(group)f(con)o(text)h(can)g(b)q(e)g(used)h (for)e(pt-pt)h(comm)o(unicati)o(on)d(b)o(y)i(an)o(y)365 1851 y(mo)q(dule)8 b(op)q(erating)h(within)g(the)h(group,)f(but)h(only)e(for)h (exactly)h(paired)f(send/receiv)o(e)365 1901 y(with)i(exact)g(matc)o(h)e(on)i (TID,)f(con)o(text,)h(and)g(message)f(tag.)17 b(Call)9 b(this)i(the)g (\\paired-)365 1951 y(exact-matc)o(h)k(constrain)o(t".)22 b(This)15 b(constrain)o(t)h(allo)o(ws)e(indep)q(enden)o(t)j(mo)q(dules)d(to)365 2001 y(use)19 b(the)f(same)f(con)o(text)i(without)e(co)q(ordinating)g(their)i (tag)e(v)n(alues.)30 b(\(Assumes:)365 2051 y(tigh)o(t)12 b(sequencing)g (constrain)o(ts)h(for)e(b)q(oth)h(blo)q(c)o(king)f(and)h(non-blo)q(c)o(king)f (comms)e(in)365 2100 y(pt-pt)14 b(MPI.\))324 2183 y Fd(\017)20 b Fc(A)f(mo)q(dules)f(that)h(do)q(es)g(not)g(meet)f(the)h(paired-exact-matc)o (h)g(constrain)o(t)g(m)o(ust)365 2233 y(use)e(a)f(di\013eren)o(t)h(globally)d (unique)i(con)o(text)g(v)n(alue.)24 b(An)16 b(MPI)g(routine)h(will)d(exist) 365 2283 y(to)h(generate)i(suc)o(h)f(a)f(v)n(alue)g(and)g(broadcast)h(it)f (to)g(the)h(group.)22 b(The)16 b(cost)g(of)f(this)365 2333 y(op)q(eration)e(is)f(sp)q(eci\014ed)i(to)e(b)q(e)h(comparable)e(to)h(that)g (of)g(an)g(e\016cien)o(t)h(broadcast)g(in)365 2383 y(the)h(group,)f(i.e.)k (log\(G\))c(pt-pt)g(startup)h(costs)h(for)e(a)g(group)g(of)g(G)g(pro)q (cesses.)20 b([See)365 2433 y(Implemen)o(tation)11 b(note)j(1.])967 2574 y(2)p eop %%Page: 3 4 3 3 bop 324 307 a Fd(\017)20 b Fc(Con)o(text)c(v)n(alues)g(are)g(a)g(plen)o (tiful)e(but)i(exhaustible)g(resource.)26 b(If)15 b(further)i(use)g(is)365 357 y(an)o(ticipated,)i(a)f(con)o(text)g(ma)o(y)e(b)q(e)j(cac)o(hed;)i (otherwise)e(it)f(should)g(b)q(e)g(returned.)365 407 y(\(Note:)g(the)c(n)o (um)o(b)q(er)e(of)h(a)o(v)n(ailable)e(con)o(texts)j(should)e(b)q(e)i(sev)o (eral)g(times)e(the)h(n)o(um-)365 457 y(b)q(er)i(of)e(groups)h(that)g(can)g (b)q(e)h(formed.\))324 540 y Fd(\017)20 b Fc(When)g(a)e(group)h(disbands,)h (its)f(group)g(descriptor)i(is)d(closed)i(and)f(an)o(y)g(cac)o(hed)365 589 y(information)e(is)j(released.)37 b(\(The)21 b(MPI)f(group-disbanding)f (routine)h(do)q(es)h(this)365 639 y(b)o(y)15 b(calling)e(destructor)j (routines)f(that)g(the)g(application)e(pro)o(vided)h(when)h(the)g(in-)365 689 y(formation)d(w)o(as)i(cac)o(hed.\))19 b(Copies)14 b(of)f(the)i(group)f (descriptor)h(m)o(ust)e(b)q(e)i(explicitly)365 739 y(released.)j(It)11 b(is)f(an)g(error)i(to)e(mak)o(e)f(an)o(y)h(reference)i(to)f(a)f(descriptor)h (of)f(a)g(disbanded)365 789 y(group,)k(except)h(to)f(release)h(that)f (descriptor.)324 872 y Fd(\017)20 b Fc(All)11 b(pro)q(cesses)i(that)f(are)f (initially)e(allo)q(cated)i(b)q(elong)f(to)h(the)h(INITIAL)f(group.)17 b(\(In)365 922 y(a)e(static)h(pro)q(cess)h(mo)q(del,)c(this)i(means)f(all)g (pro)q(cesses.\))25 b(An)15 b(MPI)h(routine)f(exists)365 971 y(for)j(obtaining)f(a)g(reference)k(to)d(the)h(INITIAL)f(group)g(descriptor)h (\(i.e.,)f(a)f(lo)q(cal)365 1021 y(group)d(ID)g(for)f(INITIAL\).)324 1104 y Fd(\017)20 b Fc(Groups)10 b(are)h(formed)e(and)h(disbanded)g(b)o(y)g (sync)o(hronous)h(calls)f(on)g(all)f(participating)365 1154 y(pro)q(cesses.)21 b(In)14 b(the)h(case)g(of)e(o)o(v)o(erlapping)g(groups,)g (all)g(pro)q(cesses)k(m)o(ust)c(form)f(and)365 1204 y(disband)i(groups)g(in)g (the)g(same)f(order.)19 b([See)14 b(Implemen)o(tation)d(Note)j(2.])324 1287 y Fd(\017)20 b Fc(All)f(group)h(formation)e(calls)h(are)i(treated)g(as)f (a)f(partitioning)g(of)g(some)g(group,)365 1337 y(INITIAL)c(if)f(none)h (other.)20 b(The)15 b(calls)g(include)f(a)h(reference)i(to)d(the)h (descriptor)h(of)365 1386 y(the)g(group)f(b)q(eing)h(partitioned,)f(the)h (TID)f(of)f(the)i(new)g(group's)f(\\ro)q(ot)g(pro)q(cess",)365 1436 y(the)d(n)o(um)o(b)q(er)e(of)h(pro)q(cesses)i(in)e(the)h(group,)f(and)f (an)h(in)o(teger)h(\\group)e(tag")h(pro)o(vided)365 1486 y(b)o(y)h(the)h (application.)j(The)d(group)f(tag)g(m)o(ust)f(discriminate)g(b)q(et)o(w)o (een)j(o)o(v)o(erlapping)365 1536 y(groups)i(that)f(ma)o(y)f(b)q(e)i(formed)e (concurren)o(tly)m(,)i(sa)o(y)f(b)o(y)g(m)o(ultiple)e(threads)j(within)365 1586 y(a)e(pro)q(cess.)20 b([See)14 b(Implemen)o(tation)d(Note)j(2.])324 1669 y Fd(\017)20 b Fc(Collectiv)o(e)c(comm)o(unication)d(routines)k(are)g (called)f(b)o(y)g(all)f(mem)o(b)q(ers)g(of)h(a)g(group)365 1719 y(in)e(the)g(same)f(order.)324 1802 y Fd(\017)20 b Fc(Blo)q(c)o(king)f (collectiv)o(e)g(comm)o(unicati)o(on)d(routines)k(are)f(passed)h(only)e(a)h (reference)365 1851 y(to)g(the)h(group)e(descriptor.)35 b(T)m(o)18 b(comm)o(unicate,)g(they)h(can)g(either)h(use)g(the)f(de-)365 1901 y(fault)14 b(group)h(con)o(text)g(or)g(in)o(ternally)e(allo)q(cate)h(a)h (new)g(con)o(text,)g(with)f(or)h(without)365 1951 y(cac)o(heing.)324 2034 y Fd(\017)20 b Fc(Non-blo)q(c)o(king)13 b(collectiv)o(e)h(comm)o(uni)o (cation)d(routines)j(are)g(passed)h(a)f(reference)i(to)365 2084 y(the)h(group)f(descriptor,)i(plus)e(a)f(\\data)h(tag")f(to)h (discriminate)f(b)q(et)o(w)o(een)j(concur-)365 2134 y(ren)o(t)13 b(op)q(erations)g(in)f(the)h(same)f(group.)17 b(\(Question:)h(is)13 b(sequencing)g(enough)g(to)f(do)365 2183 y(this?\))18 b(An)11 b(implemen)o(tatio)o(n)e(is)i(allo)o(w)o(ed)e(but)j(not)f(required)h(to)f (imp)q(ose)f(sync)o(hron-)365 2233 y(ization)g(of)h(all)f(co)q(op)q(erating)h (pro)q(cesses)j(up)q(on)d(en)o(try)g(to)g(a)g(non-blo)q(c)o(king)f(collectiv) o(e)365 2283 y(comm)o(unication)h(routine.)967 2574 y(3)p eop %%Page: 4 5 4 4 bop 262 307 a Fe(1.3)64 b(Adv)l(an)n(tages)22 b(&)g(Disadv)l(an)n(tages) 262 406 y Fb(Adv)m(an)n(tages)324 483 y Fd(\017)e Fc(This)14 b(prop)q(osal)g(is)g(implemen)o(tabl)o(e)e(with)i(no)f(serv)o(ers)j(and)e (can)g(b)q(e)h(la)o(y)o(ered)f(easily)365 533 y(on)g(existing)g(systems.)324 616 y Fd(\017)20 b Fc(Cac)o(heing)c(auxiliary)e(information)e(in)j(group)h (descriptors)h(allo)o(ws)d(collectiv)o(e)i(op-)365 666 y(erations)d(to)g(run) g(at)g(maxim)n(um)c(sp)q(eed)14 b(after)f(the)g(\014rst)h(execution)g(in)e (eac)o(h)h(group.)365 715 y(\(Assumes)i(that)f(su\016cien)o(t)g(con)o(text)g (v)n(alues)g(are)g(a)o(v)n(ailable)e(to)i(p)q(ermit)f(cac)o(heing.\))324 799 y Fd(\017)20 b Fc(Sp)q(eed)c(of)e(cac)o(hed)i(collectiv)o(e)e(op)q (erations)h(do)q(es)h(not)e(dep)q(end)i(on)f(sp)q(eed)h(of)e(group)365 848 y(op)q(erations)g(suc)o(h)h(as)f(formation)d(or)j(translation)f(b)q(et)o (w)o(een)j(rank)e(and)f(TID.)324 931 y Fd(\017)20 b Fc(Requires)13 b(explicit)e(translation)h(b)q(et)o(w)o(een)h(\(group,rank\))f(and)g(TID,)f (whic)o(h)h(mak)o(es)365 981 y(pt-pt)j(p)q(erformance)g(predictable)g(no)g (matter)f(ho)o(w)g(the)h(functionalit)o(y)e(of)h(groups)365 1031 y(gets)h(extended.)324 1114 y Fd(\017)20 b Fc(Comm)o(unicatio)o(n)7 b(b)q(oth)i(within)g(and)g(b)q(et)o(w)o(een)h(groups)g(seems)f(conceptually)h (straigh)o(t-)365 1164 y(forw)o(ard.)262 1280 y Fb(Disadv)m(an)n(tages)324 1357 y Fd(\017)20 b Fc(Requires)d(explicit)f(translation)g(b)q(et)o(w)o(een)i (\(group,rank\))e(and)g(TID,)f(whic)o(h)i(ma)o(y)365 1406 y(b)q(e)e (considered)g(a)o(wkw)o(ard.)324 1489 y Fd(\017)20 b Fc(Comm)o(unicatio)o(n) 11 b(b)q(et)o(w)o(een)16 b(di\013eren)o(t)e(groups)g(ma)o(y)e(b)q(e)j (considered)g(a)o(wkw)o(ard.)324 1572 y Fd(\017)20 b Fc(No)d(free-for-all)f (group)h(formation.)24 b(A)17 b(pro)q(cess)i(m)o(ust)d(kno)o(w)g(something)g (ab)q(out)365 1622 y(who)e(its)g(collab)q(orators)f(are)h(going)f(to)h(b)q(e) g(\(minimal)o(ly)l(,)d(the)j(ro)q(ot)g(of)f(the)i(group\).)324 1705 y Fd(\017)20 b Fc(Requires)15 b(explicit)f(coherency)j(of)d(group)g (descriptors,)i(whic)o(h)e(is)h(not)f(con)o(v)o(enien)o(t)365 1755 y({)f(application)f(m)o(ust)g(k)o(eep)i(trac)o(k)f(of)g(whic)o(h)f(pro)q (cesses)k(ha)o(v)o(e)d(copies)h(and)f(what)g(to)365 1805 y(do)h(with)g(them.) 324 1888 y Fd(\017)20 b Fc(Static)13 b(group)f(mo)q(del.)k(Pro)q(cesses)f (can)e(join)f(and)g(lea)o(v)o(e)g(collab)q(orations)f(only)h(with)365 1938 y(the)k(co)q(op)q(eration)f(of)f(the)h(other)h(group)e(mem)o(b)q(ers)g (in)g(forming)f(larger)h(or)h(smaller)365 1988 y(groups.)324 2071 y Fd(\017)20 b Fc(No)c(direct)g(supp)q(ort)g(for)f(iden)o(tifying)f(the) i(group)g(from)d(whic)o(h)j(a)f(message)g(came.)365 2120 y(The)j(sender)g (cannot)g(em)o(b)q(ed)e(its)h(group)g(ID)g(in)f(its)h(message,)g(since)h (group)f(ID's)365 2170 y(are)h(not)f(globally)e(meaningful.)25 b(One)17 b(metho)q(d)g(to)g(address)h(this)f(problem)f(is)h(to)365 2220 y(ha)o(v)o(e)11 b(the)h(sender)h(em)o(b)q(ed)d(its)h(group)g(con)o(text) h(as)f(data)g(in)f(the)i(message,)f(then)h(ha)o(v)o(e)365 2270 y(the)k(receiv)o(er)g(use)g(that)f(v)n(alue)f(to)g(select)j(among)12 b(group)j(descriptors)h(that)f(it)g(had)365 2320 y(already)f(b)q(een)h(sen)o (t.)967 2574 y(4)p eop %%Page: 5 6 5 5 bop 262 307 a Fe(1.4)64 b(Commen)n(ts)20 b(&)i(Alternativ)n(es)324 398 y Fd(\017)e Fc(The)13 b(p)q(erformance)g(of)f(group)g(con)o(trol)h (functions)f(is)h(explicitly)e(unsp)q(eci\014ed)k(in)d(or-)365 448 y(der)g(to)f(pro)o(vide)g(p)q(erformance)h(insurance.)18 b(The)11 b(in)o(ten)o(t)h(is)f(to)g(encourage)h(program)365 498 y(designs)17 b(whose)g(p)q(erformance)f(w)o(on't)f(b)q(e)i(killed)e(if)g (the)h(functionalit)o(y)f(of)g(groups)365 548 y(is)f(expanded)h(so)f(as)g(to) f(b)q(e)i(unscalable)f(or)g(require)g(exp)q(ensiv)o(e)h(services.)324 631 y Fd(\017)20 b Fc(Adding)13 b(global)f(group)h(ID's)f(w)o(ould)h (seriously)g(in)o(terfere)h(with)f(la)o(y)o(ering)f(this)i(pro-)365 680 y(p)q(osal)d(on)g(pt-pt.)17 b(The)11 b(problem)f(is)h(not)g(generating)g (a)g(unique)g(ID,)f(but)h(using)g(it.)17 b(If)365 730 y(just)e(holding)e(a)g (global)g(group)h(ID)g(w)o(ere)h(to)f(allo)o(w)f(a)g(pro)q(cess)j(to)e(async) o(hronously)365 780 y(ask)i(questions)h(ab)q(out)f(the)h(group)f(\(lik)o(e)f (translating)h(b)q(et)o(w)o(een)h(rank)f(and)g(TID\),)365 830 y(then)e(a)e(group)h(information)d(serv)o(er)k(of)e(some)g(sort)i(w)o(ould)e (b)q(e)h(required,)h(and)e(MPI)365 880 y(pt-pt)i(do)q(es)h(not)f(pro)o(vide)g (enough)g(capabilit)o(y)e(to)i(construct)h(suc)o(h)g(a)e(b)q(east.)324 963 y Fd(\017)20 b Fc(The)c(restriction)g(that)f(only)f(paired-exact-matc)o (h)g(mo)q(dules)g(can)h(run)g(in)g(the)g(de-)365 1012 y(fault)j(group)g(con)o (text)h(could)g(b)q(e)g(relaxed)g(to)f(something)f(lik)o(e)h(\\a)g(mo)q(dule) f(that)365 1062 y(violates)g(the)h(paired-exact-matc)o(h)f(constrain)o(t)h (can)g(run)f(in)h(the)g(default)f(group)365 1112 y(con)o(text)e(if)f(and)g (only)g(if)g(it)g(is)g(the)h(only)f(mo)q(dule)f(to)h(run)h(in)f(that)g(con)o (text".)21 b(This)365 1162 y(seems)11 b(to)q(o)g(error-prone)h(to)e(b)q(e)i (w)o(orth)e(the)i(trouble,)f(since)g(ev)o(en)h(the)f(standard)g(col-)365 1212 y(lectiv)o(e)g(ops)g(migh)o(t)e(b)q(e)i(excluded.)18 b(\(It)11 b(w)o(ould)f(also)g(con\015ict)h(with)g(Implemen)o(tation)365 1262 y(Note)k(1.])324 1345 y Fd(\017)20 b Fc(Group)d(formation)d(can)j(b)q(e) g(faster)h(when)f(more)f(information)d(is)k(pro)o(vided)g(than)365 1394 y(just)i(the)f(group)g(tag)g(and)g(ro)q(ot)g(pro)q(cess.)32 b(E.g.)e(if)17 b(all)g(mem)o(b)q(ers)g(of)g(the)i(group)365 1444 y(can)c(iden)o(tify)e(all)g(other)h(mem)o(b)q(ers)f(of)g(a)h(P-mem)o(b)q (er)f(group,)g(in)h(rank)g(order,)g(then)365 1494 y(O\(log)h(P\))h(time)e(is) i(su\016cien)o(t;)g(if)f(only)g(the)h(ro)q(ot)f(is)h(kno)o(wn,)f(O\(P\))h (time)e(ma)o(y)g(b)q(e)365 1544 y(required.)24 b(This)16 b(asp)q(ect)h (should)e(b)q(e)h(considered)h(b)o(y)f(the)g(groups)g(sub)q(committee)365 1594 y(in)e(ev)n(aluating)e(scalabilit)o(y)m(.)262 1731 y Fe(1.5)64 b(Implemen)n(tation)21 b(Notes)312 1822 y Fc(1.)f(T)m(o)f(generate)h(and)f (broadcast)g(a)g(new)h(con)o(text)f(v)n(alue:)28 b(Generate)20 b(the)g(con)o(text)365 1872 y(v)n(alue)9 b(without)g(comm)o(unicatio)o(n)e(b) o(y)i(concatenating)g(a)g(lo)q(cally)f(main)o(tained)f(unique)365 1921 y(v)n(alue)j(with)g(the)g(pro)q(cess)i(n)o(um)o(b)q(er)e(in)f(some)g (0..P-1)g(global)g(ordering)h(sc)o(heme.)17 b(This)365 1971 y(assumes)g(that)g(con)o(text)g(v)n(alues)f(can)h(b)q(e)h(signi\014can)o(tly) d(longer)i(than)f(log\(P\))h(bits)365 2021 y(for)i(an)f(application)f(in)o(v) o(olving)g(P)i(pro)q(cesses.)35 b(If)18 b(not,)h(then)h(a)e(serv)o(er)i(ma)o (y)d(b)q(e)365 2071 y(required,)d(in)f(whic)o(h)g(case)i(b)o(y)e(sp)q (eci\014cation)h(it)f(has)g(to)g(b)q(e)h(fast)g(\(comparable)e(to)h(a)365 2121 y(pt-pt)i(call\).)i(There)f(is)e(no)f(need)i(to)f(generate)i(a)d(new)i (con)o(text)g(for)e(the)i(broadcast)365 2171 y(since)20 b(it)e(can)h(b)q(e)g (co)q(ded)h(to)f(use)g(the)h(default)e(group)g(con)o(text)i(b)o(y)e(meeting)g (the)365 2220 y(paired-exact-matc)o(h)13 b(constrain)o(t.\))312 2303 y(2.)20 b(The)e(follo)o(wing)d(is)j(in)o(tended)g(only)f(as)h(a)f(sanit) o(y)g(c)o(hec)o(k)i(on)e(the)h(claim)e(that)i(this)365 2353 y(prop)q(osal)f(can)g(b)q(e)g(la)o(y)o(ered)g(on)f(MPI.)h(Impro)o(v)o(emen)o (ts)e(to)h(impro)o(v)o(e)f(e\016ciency)j(or)365 2403 y(to)c(use)h(few)o(er)f (con)o(texts)h(will)e(b)q(e)h(greatly)g(appreciated.)967 2574 y(5)p eop %%Page: 6 7 6 6 bop 365 307 a Fc(In)16 b(addition)f(to)h(a)f(default)h(group)f(con)o (text,)i(eac)o(h)f(group)g(descriptor)h(con)o(tains)f(a)365 357 y(\\group)f(partitioning)g(con)o(text")h(and)f(a)g(\\group)h(disbanding)e (con)o(text")j(that)e(are)365 407 y(obtained)h(and)g(broadcast)h(at)f(the)h (time)d(the)j(group)f(is)g(created.)26 b(In)16 b(the)h(case)g(of)365 457 y(the)f(INITIAL)g(group,)f(this)h(is)f(done)h(using)g(paired-exact-matc)o (h)e(co)q(de)j(and)e(an)o(y)365 506 y(con)o(text,)g(immediately)d(after)j (initialization)c(of)k(the)g(MPI)g(pt-pt)g(lev)o(el.)20 b(\(A)o(t)15 b(that)365 556 y(time,)c(no)g(application)f(co)q(de)i(will)e(ha)o(v)o(e)i (had)f(a)g(c)o(hance)i(to)e(issue)i(wildcard)e(receiv)o(es)365 606 y(to)j(mess)g(things)f(up.\))365 672 y(Group)g(partitioning)e(is)i (accomplished)f(using)h(pt-pt)g(messages)g(in)f(the)i(partition-)365 722 y(ing)k(con)o(text)h(of)f(the)i(curren)o(t)g(group)e(\(i.e.,)g(the)i(one) e(b)q(eing)h(partitioned\),)g(with)365 772 y(message)d(tag)h(equal)f(to)g (the)h(group)f(tag)g(pro)o(vided)h(b)o(y)f(the)h(application.)25 b(In)16 b(the)365 822 y(w)o(orst)g(\(least)g(scalable\))g(case,)g(the)h(ro)q (ot)e(of)g(the)i(new)f(group)f(m)o(ust)g(wildcard)g(the)365 872 y(source)22 b(TID.)d(\(This)h(violates)f(the)h(paired-exact-matc)o(h)g (constrain)o(t,)h(whic)o(h)f(is)365 922 y(wh)o(y)14 b(group)g(formation)d(m)o (ust)i(happ)q(en)h(in)g(a)f(sp)q(ecial)h(con)o(text.\))365 988 y(Group)k(disbanding)e(is)i(done)g(with)f(pt-pt)h(messages)g(in)f(the)h (group)f(disbanding)365 1038 y(con)o(text.)25 b(This)15 b(k)o(eeps)i(group)f (con)o(trol)f(from)f(b)q(eing)i(messed)g(up)f(no)h(matter)f(ho)o(w)365 1088 y(the)g(application)d(uses)j(wildcards.)262 1225 y Fe(1.6)64 b(Examples)262 1316 y Fa(\(T)l(o)15 b(b)q(e)h(pro)o(vided\))967 2574 y Fc(6)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 11:53:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07957; Fri, 19 Mar 93 11:53:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26990; Fri, 19 Mar 93 11:53:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 11:53:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26973; Fri, 19 Mar 93 11:53:02 -0500 Date: Fri, 19 Mar 93 16:52:24 GMT Message-Id: <10550.9303191652@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: delayed comments To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all I'm afraid that comments to Rik's Proposal V will be delayed until approx 6pm GMT, two hours later than promised. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 12:35:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08995; Fri, 19 Mar 93 12:35:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29197; Fri, 19 Mar 93 12:35:21 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 12:35:19 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29127; Fri, 19 Mar 93 12:34:35 -0500 Date: Fri, 19 Mar 93 17:34:23 GMT Message-Id: <10600.9303191734@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: comments arrive To: d39135@sodium.pnl.gov Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Rik Here are my comments of this afternoon. I hope they make some sense to you! Regards Lyndon General points -------------- 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am offering if you wish) should attempt to construct a more transparent presentation before circulation to whole committee, for the convenience of committee members. 2) I'm not a fan of much of this proposal, although I do indeed like some of the ideas which it introduces. [On the other hand, I'm not a great fan of all of the proposal which I wrote. I shall mail self criticism of my proposal, and may have to write amended or alternative proposal :-)] 3) I really like the way in which groups are something like ``frames'' in which contexts are created. This is conceptually much neater than duplication of groups. 4) I like the idea of pushing information into the group structure. I have a few qualms with the proposed details --- see specific points. 5) See ``Writing a server in the point-to-point layer of MPI in four easy steps'' at the foot of the message. Specific points --------------- Dealt with as LaTeX comments to body of text, appearing in the form %[Lyndon] % text of point for your navigational convenience. These are quite detailed. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- Writing a server in the point-to-point layer of MPI in four easy steps ---------------------------------------------------------------------- 1) Partition the INITIAL group into two groups. A singleton group, SERVER, and a group CLIENT which contains all of the other processes. 2) The single process in SERVER group records its TID. 3) The processes in INITIAL group allocate a context SERVICE which they remember either in the group cache or static data or something. 4) Use a broadcast in INITIAL group with ``sender'' as the one process which is also in SERVER group, and the ``receivers'' as the (many) processes which are also in CLIENT group, in the SERVICE context, in order to disseminate the TID of the server process. [Fanfare] a server process is in place as is a dedicated context for the purposes of messages required to implement the service. [Observation] the mpi point-to-point initialisation can do this automatically. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 14:04:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10876; Fri, 19 Mar 93 14:04:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04631; Fri, 19 Mar 93 14:03:37 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 14:03:35 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04623; Fri, 19 Mar 93 14:03:32 -0500 Date: Fri, 19 Mar 93 19:03:21 GMT Message-Id: <10723.9303191903@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: gone To: d39135@sodium.pnl.gov Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Rik I away home with the laptop, and off-line for the evening. If you want to discuss my comments, or make comments to me, or whatever, today then feel free to phone me at home if you are able. My homephone is + 44 31 557 0480. I am going to work on proposal I, to contract it to essentially the "Snir" model as originally discussed --- i.e. I'm going to delete the extended part at least. I never really did like retrofitting my requirements onto that. I am then going to write a proposal II' which separates contexts and groups, and meets requirements, and which I all along preferred in shape. I hope you don't mind me borrowing the number, I have annotated to indicate that this is not the original II :-) There will be much room for the group identifier cache in II'. I will mention this but not deal with it in any detail at this point. This is work I will do with the optimistic assumption that the "gang of three (four, five?)" will find this acceptable. I hope to mail out tomorrow afternoon GMT. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 12:08:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29146; Sun, 21 Mar 93 12:08:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20050; Sun, 21 Mar 93 12:08:12 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 12:08:09 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20042; Sun, 21 Mar 93 12:08:08 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03376; Sun, 21 Mar 93 11:02:21 CST Date: Sun, 21 Mar 93 11:02:21 CST From: Tony Skjellum Message-Id: <9303211702.AA03376@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu Subject: Re: Proposals V and I Cc: mpi-context@cs.utk.edu I will put my own doc in Latex, but I agree this is good. We will make one jointly authored "document", ala Rusty & Bill. More to follow [lots] - Tony From owner-mpi-context@CS.UTK.EDU Sun Mar 21 13:14:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29712; Sun, 21 Mar 93 13:14:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21849; Sun, 21 Mar 93 13:14:13 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 13:14:11 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21841; Sun, 21 Mar 93 13:14:10 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03573; Sun, 21 Mar 93 12:08:23 CST Date: Sun, 21 Mar 93 12:08:23 CST From: Tony Skjellum Message-Id: <9303211808.AA03573@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: My comments on Proposal V [tex format] Step 1. I am commenting directly on Proposal V. Step 2. I am comment on Lyndon's comments on Proposal V. This will come in a separate e-mail, to reduce [my] confusion, and hopefully yours. - Tony 1. My annotations to proposal V by Rik; %**** is my prefix \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %**** %**** Seems doable. %**** \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %**** %**** Seems doable. %**** \item There are restrictions that permit groups to be layered on top of pt-pt. %**** %**** I don't understand the word 'restriction' here. %**** Restriction of what. %**** \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. %**** %**** What does "fast" mean. %**** \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. %**** %**** OK, the above two items are identical to what Zipcode %**** provides in practice, but people have argued that groups %**** might be created/deleted more often in some apps, and %**** that these apps ought to be supportable %**** \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %**** %**** Yes, and I want at least 32-bits of message tag. %**** \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %**** %**** This could be difficult, in practice, if one mails a %**** message to one's own process, and MPI is smart enough %**** to optimize. %**** \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %**** %**** Sounds good. %**** \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %**** %**** In Zipcode 1.0, we allow multiple global operations %**** to be provided on a message-class (eg, grid-oriented messages) %**** The identifiers for these possible operations are user-specified %**** presently, but the "names" of the global operations are fixed %**** at compile-time. %**** %**** That means that there is O(1) time to find combine, fanout, send, %**** etc, on a group-wide scope. However, other operations cannot %**** be accessed in O(1) time (they are not in the opaque structure). %**** %**** The same mechanism used by Zipcode to allow multiple methods for %**** combine to be registered by the user, could also allow extensibility %**** just like Rik describes, with little effort. We use AVL trees. %**** %**** In fact, I will add this to Zipcode 1.x. Why say this? It is %**** not far from existing practice, and I have a lot of the machinery %**** in place already, and I am confident that it is useful. %**** \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %**** %**** This seems to be a serious flaw. It will have to be cached %**** on an LRU basis, with system/user/both specifying how much %**** caching is allowed (ie, how much unscalable memory use). %**** If the first time is expensive, OK, but not the Nth time. %**** \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). %**** %**** OK, I see no problem with this (so far). %**** \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %**** %**** NO. This violates the concept of context entirely. %**** (ie, an oxymoron ... contexts same, but still no need for %**** tag disambiguation...) %**** %**** Use the default group context to establish (cooperatively) %**** other contexts, and then use these. This is a seriously %**** bad feature, in my mind. %**** \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %**** %**** I do not think we should support the paired-exact-match thing. %**** \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) %**** %**** Concur. This suggests many more than "256" %**** \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] %**** %**** This is the Zipcode model. It could say loosely synchronous. %**** \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %**** %**** I don't understand the thread issue here. %**** \item Collective communication routines are called by all members of a group in the same order. %**** %**** Yes. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. %**** What does caching really imply here ??? Help. %**** \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %**** I think that contexts are really important in this case, %**** to keep things straight, but that non-blocking collcomm should %**** be omitted from MPI1 (cf, Geist). Sequencing supports %**** a sufficient disambiguation, as long as the entire group %**** is always the participant in operations. That is, you have %**** to form subgroups, with new contexts, to do global ops on %**** subsets. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %**** Why aren't servers needed to create contexts. Where do they %**** come from? If you rely on the fact that INITIAL will do %**** a loosely synchonous cooperative operation each time a new %**** context is needed, then a simple (easily implementable server, %**** or fetch-and-add remote access) is replaced by a more rigid %**** computation model. %**** %**** If we can get rid of this disagreement, me might be able to %**** reduce our total proposal space by one whole proposal. %**** \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %**** %**** If contexts are being used very dynamically, how are they being %**** assigned, kept, released, reissued without a server? Sorry if %**** I missed something, but I don't see it, without a restrictive %**** SPMD model of computation (Zipcode obviates its server for the %**** SPMD model, for instance). \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %**** %**** Can you clarify this with examples. %**** \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %**** I like this, of course. \item Communication both within and between groups seems conceptually straightforward. %**** Well, is point-to-point group oriented. Not. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %**** I think it is awkward. \item Communication between different groups may be considered awkward. %**** OK, but one can form a new group, as I have argued before. %**** Use the "awkward" pt2pt to get the right info shared between %**** group leaders, make the new group, use unawkward collective %**** operations on new group (with new context). \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %**** %**** This again is in practice, in Zipcode. %**** \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %**** %**** Sounds dangerous. What must application do to maintain %**** coherency, since group descriptors are opaque. %**** \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. %**** No, loosely synchronous process model, unless you mean %**** cooperation of INITIAL at all such join/leave steps. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %**** %**** No, you can't know the group or rank in group of sender. %**** If there were one context per group (isn't that so here?), %**** then all you need is the rank. With TID_TO_RANK_IN_GROUP %**** operation, this could be provided, but no wildcarding %**** or receipt selectivity could be done at this level. %**** \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %**** %**** No, it just does not provide guarantees that certain kinds %**** of applications will run OK. (ie, those that do group %**** creation/deletion relatively often). Zipcode has assumed %**** that such operations would be relatively seldom. Thus, I do %**** not quibble that this is a reasonable choice,but a fairer %**** way to say this is that it may be difficult to support such %**** applications. That reveals an issue to be studied more. %**** \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %**** %**** Perhaps they should do. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] %**** Dump this. \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %**** %**** No, a non-deterministic broadcast can be used, with a token. %**** This requires a token server. Again, implementable with fetch+ %**** add on most systems, or a light reactive server. %**** %**** Once the non-deterministic broadcast has finished, a fanin/collapse %**** is done to the original root, which then frees the token. %**** \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %**** %**** Why not just given in and allow the server. %**** I don't like the paired-exact-match constraint AT ALL. %**** \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) %**** %**** Seems OK, but why need the paired-exact-match thing again. %**** Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) %**** %**** Again, OK, but I want to see this work without the paired-exact- %**** match, if possible. Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. %**** %**** So, now, you have concurred with my (previously flamed) idea %**** that group construction/destruction should be realizable using %**** pt2pt, just like global operations should do. I like this %**** because 1) it is explicable to the implementor, 2) it allows %**** simple intitial implemtations, 3) it sets some ideas for how %**** much these things will cost [upper bound]. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} From owner-mpi-context@CS.UTK.EDU Sun Mar 21 13:22:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29866; Sun, 21 Mar 93 13:22:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22128; Sun, 21 Mar 93 13:22:00 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 13:21:59 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22120; Sun, 21 Mar 93 13:21:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03587; Sun, 21 Mar 93 12:16:22 CST Date: Sun, 21 Mar 93 12:16:22 CST From: Tony Skjellum Message-Id: <9303211816.AA03587@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: mini-proposal on layerability Dear Context subcommittee, This is something we should share with pt2pt, but I put it before you first.... (and to topologies, for their interest). To build effective layers in MPI, a reasonable receipt selectivity must be provided. As suggested by Jim, this can be accomplished by two integers as selectors for message wildcarding... dont_care care So, each receipt function uses the algorithm (received_rag & ~dont_care)& care With such a feature, layerability (eg, virtual topologies) could be much more easily added into MPI. Without this, (ie, accept or reject tag as ANY or only one specific), it is very difficult to layer. It is useful that wildcarding of sender is provided, and all-or-nothing wildcarding is fine for sender, since sender is specified by opaque TID. Example, wildcard receipt in a 3D grid usage of tags to implement a virtual topology. If a topology is embedded in a tag (p,q,r), then it is conveniently possible to do wildcarding on any dimension, given the care/dont_care bits. Another option would always be to multiple out the (p,q,r) triplet to give the rank-in-group. This could be stored in the tag instead, but severe multiplications/division needs would occur to determine a match (provided a general matching function were also contemplated). In short, a lot of flexiblity occurs with the care/dont_care bits, and I will ask for this feature, so that layerability can reasonably be claimed in MPI1. From owner-mpi-context@CS.UTK.EDU Sun Mar 21 13:44:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00158; Sun, 21 Mar 93 13:44:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22724; Sun, 21 Mar 93 13:43:47 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 13:43:46 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22716; Sun, 21 Mar 93 13:43:44 -0500 Date: Sun, 21 Mar 93 18:43:27 GMT Message-Id: <12975.9303211843@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Proposals V and I To: Tony Skjellum , d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu In-Reply-To: Tony Skjellum's message of Sun, 21 Mar 93 11:02:21 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu > I will put my own doc in Latex, but I agree this is good. We will make > one jointly authored "document", ala Rusty & Bill. Super. I thought about this and thought that until a proposal is chosen that goes through a report documenststyle would probably be easiest, so each proposal appears as a \chapter. I think we should probably attribute authorship and credits to the individual proposal chapters at least for now. > More to follow [lots] Goodie Goodie :-) Me too, lost more to follow. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:07:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00224; Sun, 21 Mar 93 14:07:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23503; Sun, 21 Mar 93 14:06:52 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:06:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23495; Sun, 21 Mar 93 14:06:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03668; Sun, 21 Mar 93 13:00:37 CST Date: Sun, 21 Mar 93 13:00:37 CST From: Tony Skjellum Message-Id: <9303211900.AA03668@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: my comments on Lyndon's comments on PropV %**** Step 2. My comments next to Lyndon's comments, critique of Rik's %**** proposal V. %**** %**** I have added my further remarks, with %**** %**** - Tony General points -------------- 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am offering if you wish) should attempt to construct a more transparent presentation before circulation to whole committee, for the convenience of committee members. %**** %**** I felt that things appear twice, because of summary (good) %**** and because of implementation notes at end (confusing) %**** 2) I'm not a fan of much of this proposal, although I do indeed like some of the ideas which it introduces. [On the other hand, I'm not a great fan of all of the proposal which I wrote. I shall mail self criticism of my proposal, and may have to write amended or alternative proposal :-)] %**** %**** Please be more specific. I am having a hard time understanding %**** why you really don't like it, Lyndon. If the process model %**** were a little less static, and servers were permitted (though %**** hopefully bounded in cost), I think we would have an excellent %**** proposal. %**** 3) I really like the way in which groups are something like ``frames'' in which contexts are created. This is conceptually much neater than duplication of groups. %**** %**** In practice, group subsetting will require groups to be copied, %**** otherwise, subgroups will unfairly be penalized by the size %**** of their ancestor. %**** 4) I like the idea of pushing information into the group structure. I have a few qualms with the proposed details --- see specific points. %**** %**** I have more confidence about this idea, and could demonstrate %**** by June/July time-frame in Zipcode. %**** 5) See ``Writing a server in the point-to-point layer of MPI in four easy steps'' at the foot of the message. %**** %**** This seems like a nice thing. %**** Specific points --------------- Dealt with as LaTeX comments to body of text, appearing in the form %[Lyndon] % text of point for your navigational convenience. These are quite detailed. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %**** %**** Alternatives, Lyndon? %**** \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %**** Me too. \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %**** %**** 1) It is not obvious that TIDs should be restricted to 32 bits. %**** 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %**** I favor a whole word for a context, despite other limits, %***** just to make things simpler. %**** %**** Internet addresses are going to get augmented from 32 to ??? bits %**** is it reasonable to assume that certain MPI implementations might %**** incorporate such internet addresses as TIDs (in future), %**** %**** Opacity is partially violated if we say how big the data type is??? \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %**** %**** where does this get us??? %**** Remember, we have to choose on some things, so we can have something %**** to present in Dallas. Is there an important difference here? %**** %**** TIDs are global entities. Is structure assumed to be global; %**** in a truly opaque system, some TID component would have to be %**** fixed, but the rest could vary structurally... %**** \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %**** %**** Yes, contexts are global; group identifiers are just pointers %**** typically, to data structures, describing %**** %**** 1) a groups context %**** 2) group members and their ranks (mappings, inverses, %**** cached, hashed, unscalably stored, etc) %**** 3) TID-to-rank map and inverse (see possibilities in 2) %**** 4) A set of fixed global operations, accepted as standard, %**** an accessible in O(1) time. Possibly, each %**** such operation should be a method, so that %**** a parameter block can be passed with it. Zipcode %**** supports the Method type to do this. %**** This is effectively a cache for some parts of item #5 %**** ... %**** 5) An AVL or similar tree of extensible operations. %**** New operations are registerable by the user. These %**** tags are unique within a group, a specify an operation %**** i) pre-defined by MPI (in which case it can be cached %**** in 4 %**** ii) alternative operations (even if they do something %**** standard, that are wanted to be accessed by %**** name) This name is group unique. %**** %**** A mechanism for DO_METHOD_FROM_GROUP(name,....) %**** or GET_METHOD_FROM_GROUP(name,...) %**** and SET_METHOD_IN_GROUP(name,...) are clearly needed. %**** Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %**** Yes, that is included, I believe, in concept. %**** \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %**** %**** Stripping is extremely controversial aspect, and arbitrary. %**** If the recipient has the methods with the same name, then %**** a new rendezvous could be accomplished at the far end \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %**** %**** Yes, these are all options. Fastness seems to be an important %**** issue. If translation is very expensive, none of the "good" %**** features will be used. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %**** Again, I think this should be eliminated, and all references %**** to this idea should be expunged. It denies the context's %**** ability to manage messages. \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. %**** I agree that context allocation has to be done in sequence. %**** That is why I am in favor of providing calls that allow %**** groups to get numerous contexts at creation, and then %****cooperatively, but potentially without further communication %**** divide them(as they build subgroups, for instance). %**** %**** I see these as services to be used in building virtual topology %**** features, which will then be more widely used by users of MPI. %**** \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %**** %**** Do we get anything by having the root process? %**** \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %**** %**** I think that non-blocking collcomm is moribund in MPI1 or %**** else MPI1 is moribund. :-) %**** \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %**** Too many negatives here for me to parse :-) \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %**** %**** Context allocation need not be free provided it can be made cheap, %**** or cheap enough. %**** %**** If one knows one will need several, then a single call could %**** provide such contexts, amortizing overhead. This is likely when %**** bulding grids (ie, virtual topologies) in Zipcode, so it is %**** true in existing practice. %**** %**** One should recognize the need for layering virtual top. calls %**** on top of these calls, then these calls may appear painful, %**** but perhaps they would be less used. Some users will use the %**** provided virtual topology calls, others will prefer their own. %**** Both will have equal power (see also,separate note on layerability). %**** %**** If getting N contexts is a send-and-receive, plus a reactive server, %**** then this is reasonably light weight,provided that hundreds of %**** messages, or global operations ensue thereafter. We can know in %**** advance how heavy weight the context server will be. %**** %**** if an implemention can use some locations of remote memory, with %**** fetch and add, or locks, to achieve contexts, then this is even %**** cheaper, in principle. %**** %**** Despite Jim's earlier insistence that context numbers be kept to %**** 256 or so, I think that this number should be much larger, so that %**** much less efort goes into returning contexts, and so on, except %**** occasionally, by processes. Otherwise, a new kind of overhead, %**** get-rid-of-context-because-I-am-out ensues, or programs block %**** until contexts become available, offering the possibility of %**** deadlocks. %**** \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %**** %**** Unscalability (of a limited form) should be permitted/selectable %**** by user, to use as much per-node memory as the user wants, to reduce %**** communication. %**** \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. %**** I see,but what he means here is that there is no unpredictable %**** translation cost because we do not write (group,rank) in pt2pt %**** calls. So, there is some validity to the statement. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %**** Yes, that is why I have one in Zipcode. I wish Zipcode were %**** on netlib today, so you could try it. Well, we are writingthe %**** manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %**** %**** I lost something here, but I agree with the sense. The word %**** seems is subjective,and should disappear from our discussions, %**** as much as seems prudent, anyway :-) \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %**** %**** Yes,awkward, because it is nice to escape the TID realm and %**** work within the (albeit simple) abstraction of group,rank. %**** When layering virtual topologies on this, it would be so nice %**** to write them to a group,rank syntax, not enforcing TID mappings %**** everywhere. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %**** Indeed. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %**** Yes, and I think that loosely synchronous operations can maintain %**** coherency, in practice. That is, no operations that modify the %**** group descriptors (other than cached lookup info) are permitted, %**** without loose synchronization. %**** This is nasty in that is would prohibit sending descriptors to %**** processes not part of the group, so it is a clear trade-off. %**** Perhaps such send-to-non-group-member operations could stipulate %**** that this group information is somehow ephemeral, and that they %**** need to join a new group to keep useful information over time??? %**** \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %**** %**** Yes, that is exactly what Zipcode was written to avoid. The %**** user wants help managing things like this!!!! %**** %**** The search, if any, must be MPI-supported, and as efficient as %**** possible (eg, AVL trees, hash, partial hash with exceptions). %**** % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %**** %**** This is why I remain a strong advocate of (group,rank) %**** addresssing in pt2pt. %**** \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %**** %**** I addressed this in my Step-1 remarks. Please see that. %**** \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %**** %**** Thank you. These points are both well taken (ie these two paragraphs) \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %**** Also,see what I wrote in my Step-1 comments. %**** I believe O(log(P)) is still possible. %**** \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %**** Please see my Step-1 comments. \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- Writing a server in the point-to-point layer of MPI in four easy steps ---------------------------------------------------------------------- 1) Partition the INITIAL group into two groups. A singleton group, SERVER, and a group CLIENT which contains all of the other processes. 2) The single process in SERVER group records its TID. 3) The processes in INITIAL group allocate a context SERVICE which they remember either in the group cache or static data or something. 4) Use a broadcast in INITIAL group with ``sender'' as the one process which is also in SERVER group, and the ``receivers'' as the (many) processes which are also in CLIENT group, in the SERVICE context, in order to disseminate the TID of the server process. [Fanfare] a server process is in place as is a dedicated context for the purposes of messages required to implement the service. [Observation] the mpi point-to-point initialisation can do this automatically. %**** Zipcode's postmaster general works in this way, more or less. %**** - Tony ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:37:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00298; Sun, 21 Mar 93 14:37:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24385; Sun, 21 Mar 93 14:37:22 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:37:21 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24377; Sun, 21 Mar 93 14:37:19 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03706; Sun, 21 Mar 93 13:31:14 CST Date: Sun, 21 Mar 93 13:31:14 CST From: Tony Skjellum Message-Id: <9303211931.AA03706@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Subject: my comments on Lyndon's Proposal I %**** %**** Below are my comments on Lyndon's Proposal I. In the next paragraph %**** I note that we are actually converging to less than N proposals, though %**** I have not seen Lyndon's new II proposal yet. Does it exist now? %**** %**** To achieve a reasonable presentation at Dallas, we can have multiple %**** proposals still on table (I think this a fair, well-thought-out %**** approach, but if we can condense some, let's do it). %**** %**** %**** After reading V, making my comments, and making my comments %**** in addition to Lyndon's comments, I am convinced we can %**** advance proposal V into something that is acceptable in the %**** III/IV mold, without a further III/IV proposal. Rik's %**** dropping of the static context concept has simplified our %**** group's efforts considerably, and I cannot disambiguate my %**** III/IV proposal from Proposal V, given the Lyndon and Tony %**** provisos and suggested improvements. This is not a wimp out %**** on my part. I do not see benefit of advancing something that %**** will look 90% like Proposal V at this point, and 97% like it %**** if the Lyndon/Tony comments obtain in it (which they would if %**** I wrote it now). %**** I would prefer that we hone Proposal V. If Rik wants to keep %**** his style, then I propose that Proposal III/IV become exactly %**** what I just said, a reworking of V + comments. %**** However, I think this will cause an unnecessary delay and %**** digression just to achieve details. Instead, we might pull %**** such choices into Proposal V at this time (server vs. no server). %**** to make what is common in our "approaches" more obvious. %**** - Tony %**** %**** (PS, Lyndon: Rik/Tony/Lyndon are authors of whole paper, because %**** we are all working on this, and because one of us (ie, me) will %**** make the final document cohesive, create a unified style, format, %**** set of meanings. This is the nature of collaboration. I do %**** not propose to include our other co-conspirators on such a document's %**** list of authors, as there has been minimal input from them. I %**** Think that Rusty/Bill and Rik/Tony/Lyndon are operating equivalently.) \documentstyle{report} \begin{document} \title{``Proposal'' I for MPI Communication Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Lyndon J. Clarke % March 1993 % \chapter{Proposal I} \section{Introduction} This chapter proposes that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The proposal reflects the observation that an instance of a module in a parallel program typically operates within a group of processes, and as such any communication contexts associated with an instance of a module also bind the semantics of process groups. This chapter makes a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The proposals say nothing about use of message tags. It is assumed that these will be a bit string, expressed as an integer in the host langauge (i.e. ANSI C or Fortran~77, in the first instacnce. These proposals should be viewed as a collection of recommendations to other subcommittees within MPI, primarily the collective communications subcommittee, the point-to-point communications subcommittee and the process topologies subcommittee. Concrete syntax is given in the style of the C language for purposes of discussion only, and should be viewed as an example of possible syntax as detailed syntax of described operations is the responsibility of the language bindings subcommittee %----------------------------------------------------------------------% % BEGIN "Basic Proposal" % \subsection{Basic Proposal} The main features of the basic proposal are: \begin{itemize} \item Process identifiers and group identifiers have opaque values expressed as an integer type in the base language. Semantics for passing process identifiers in a message are defined, whereas semantics for passing group identifiers in a message are not defined. \item Group creation and destruction are concerted, synchronous, actions on the part of the membership of the group concerned. %**** %**** loosely synchronous %**** \item Groups are {\it static\/} in that no provision is made for modification of the membership of a group over the lifetime of the group. Dynamic group operations such as {\it resize\/} can be effected by destruction of the existing group and creation of a new resized group. %**** Why not by making an off-spring, instead of destructively!!! \item Group creation is effected by four means: explicit definition of the membership of a group; partition of an existing group into one or more distinct subgroups; identical duplication of an existing group; topological duplication of an existing group. \item Point-to-point communication provides for intra-group communication, and makes no provision for inter-group communication. \item Collective communication provides for operations which are concerted actions on the part of members of one group, and makes no provision for operations which are concerted actions on the part of members of multiple groups. \end{itemize} \subsubsection{Process and Group Identifiers} A {\it process identifier\/} is an opaque reference to an object which is a single process. A process identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on process identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). Each process has exactly one process identifier. MPI should provide a procedure which allows the user to determine the process identifier of the calling process. For example, {\tt mypid = mpi\_mypid()}. The identifier of the {\it null process\/}, defined to be a process which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_PID\_NULL} in this proposal. With the single exception of the identifier of the null process, the value of a process identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a process P then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a process identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the received process identifier is {\it invalid\/}. MPI will provide a mechanism which allows a process identifier to be passed in a message in such a manner that the received identifier is valid. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a process identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_pid(bd, \&pid)}. Transmission of a process identifier using this mechanism returns to the recipient a process identifier which is valid for use in the MPI operations described below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a process identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_pid\_invalidate(pid)}. This is an error if {\tt pid} is {\tt MPI\_PID\_NULL}, or if {\tt pid} is the identifier of the calling process. It is further proposed that MPI provide a process identifier registry service. This service allows any process to register its own process identifer by name, and deregister its process identifier. The service allows any process to determine whether a name has been registered without blocking the calling process, and to map that name into a valid process identifier with the possibility of blocking the calling process. Use of this service is not mandated, and components of programs which do not require this service are not expected to make use thereof. %**** I don't get all of this. Why? %**** A {\it group identifier\/} is an opaque reference to an object which is a group of processes. A group identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on group identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). The identifier of the {\it null group\/}, defined to be a group which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_GID\_NULL} in this proposal. With the single exception of the identifier of the null group, the value of a group identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a group G then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a group identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the identifier is {\tt invalid\/}. MPI will not provide a mechanism which allows a group identifier to be passed in a message in such a manner that the received identifier is valid. %**** Then why allow it to be passed? %**** The canonical representation of a group is an array of distinct process identifiers, although it may be possible to use hashing functions with lower space complexity and marginally higher time complexity. A group is {\it static\/}, in that it's membership may not change over the lifetime of the group. There is a well defined {\it size\/} of a group, and MPI will provide a procedure which allows the user to determine the size of a group. For example, {\tt size = mpi\_grp\_size(gid)}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. There is a well defined {\it rank\/} of a process within the group, i.e. the position in the representational array at which the process identifier is held. MPI will provide a procedure which allows the user to determine the rank of the calling process within a group. For example, {\tt mpi\_grp\_myrank(gid)} returns the rank of the calling process within the group referred to by {\tt gid}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. MPI should provide a procedure to determine the member rank of a process within a group given the group identifier and a valid process identifier, e.g. {\tt rank = mpi\_grp\_rank(gid,pid)}. This procedure may ``fail'' if the process identified by {\tt pid} is not a member of the group identifier by {\tt gid}, and can be used to determine whether a given process is a member of a given group. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. %**** %**** Why an error, why not just fail? %**** MPI should provide a procedure to determine the process identifier of a group member given the group identifier and member rank, e.g. {\tt pid = mpi\_pid(gid,rank)}. This procedure should validate the returned process identifier. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. \subsubsection{Group Creation and Destruction} MPI will provide four methods for creation of groups, and a procedure to destroy an existing group. A group may be created by explicit definition of the pids of the members of the group. For example, {\tt gid = mpi\_grp\_definition(npids,pids)} where {\tt gid} is the identifier of the newly created group, {\tt npids} is the number of processes in the new group, and {\tt pids} is an array containing the (valid) process identifiers of the processes in the group. This procedure must be called my all members of the group, and does not return until all members have made the call. A group may be created by identical duplication of an existing group. For example, {\tt gidb = mpi\_grp\_duplication(gida)} where {\tt gidb} is the group identifier of the newly created group and {\tt gida} is the identifier of an existing group. The created group inherits all properties of the source group, including any topological properties. This operation has the same synchronisation properties as creation of group by definition. %**** It is not obvious to me that we want to enforce topology at this %**** juncture. However, we could register topology information in %**** the extensible structure strategy of Proposal V. %**** A group may be created by topological duplication of an existing group. Details of topological groups are under consideration within the process topologies subcommittee and will not be further discussed here. A group created by topological duplication inherits the size of the source group, and also inherits the membership list of the source group although the list may be ordered differently in the created group. These operations have the same synchronisation properties as creation of group by identical duplication. Where groups have additional topological attributes MPI should also provide procedures which allow the user to determine such attributes. One or more groups may be created by partition of an existing group into distinct subgroups by key. For example, {\tt gidb = mpi\_grp\_partition(gida, key)} where {\tt gidb} is the group identifier of the newly created group corresponding to the given {\tt key} and {\tt gida} is the group identifier of the group which is being partitioned according to the {\tt key} values supplied. MPI should define a named constant which is a {\it null\/} {\tt key} value, for example {\tt MPI\_KEY\_NULL}, in order that members of the parent group can choose not to become members of any child group, and in which case the procedure should return {\tt MPI\_GID\_NULL}. Groups created by partition share the same ordering of process member ranks as the parent group. This operation synchronises the members of the parent group, and therefore implicitly synchronises the members of the created group(s). A group may be destroyed, e.g. {\tt mpi\_grp\_deletion(gid)}, which destroys the group identified by {\tt gid}. This operation synchronises the group members, and invalidates the group identifier. \subsubsection{Point-to-point communication} There are arguments either way for point-to-point routines which accept a {\tt (gid, rank)} pair, and for routines which accept a {\tt pid}. This proposal supports and seperates both approaches to addressing and selection within the same syntax, avoiding the introduction of sets of procedures to handle each case. In order to provide expressive power to intra-group communication, point-to-point communication should accept a {\tt (gid, rank)} pair, where {\tt gid} is a valid group identifier and {\tt rank} is a member rank within the group identified by {\tt gid}. In message addressing the sender specifies the {\tt rank} and {\tt gid} of the receiver. In message selection the receiver specifies the {\tt rank} and {\tt gid} of the receiver. The sender and receiver must both specify the same {\tt gid} in order for a match to occur. The {\tt gid} field is not allowed to take a {\it wildcard\/} value in message selection. The {\tt rank} field is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_RANK\_WILD}, which will match with any rank. One is encouraged to visualise a seperate message queue or port at each process for each group of which that process is a member (and indeed this may be an advantageous implementation feature). %**** But not a required feature of an implementation! In order to accommodate process identifier based addressing and selection into the same syntax, this proposal advocates that point-to-point communication should also accept the null group ({\tt gid = MPI\_GID\_NULL}), in which case the {\tt rank} is interpreted as a valid {\tt pid}. The {\tt pid} is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_PID\_WILD}. The point-to-point section should provide a procedure which allows a recipient to recover the process identifier of the sender. The discussion of matching above extends to this case, and one is also encouraged to visualise a separate message queue or port for messages referred to by {\tt MPI\_GID\_NULL}. In the case of process identified wildcard receive, the process identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the process identifier must be performed by the MPI implementation, in order that the recipient is returned a valid process identifier, else the returned identifier is of little or no use to the recipient. %**** %**** I do not understand the usefulness or formal need for all this %**** validation and invalidation of process identifiers. Why, where %**** did it come from, what does it get us? How can this be related %**** to anything I have seen before? %**** \subsubsection{Collective communication} Collective communication operations within MPI should be restricted to the scope of a single group. It will be sufficient for these procedures to accept a group identifier, and possibly a message tag in order to distinguish multiple outstanding operations within the same group. These procedures must not accept {\tt MPI\_GID\_NULL}. %**** Why not, but do nothing... It is not possible to determine whether this is strategy allows all of the MPI collective communication routines to be written in terms of MPI point-to-point routines without loss of generality, since the set of collective communication routines is not yet determined. This proposal takes the view that it is the responsibility of the collective communications subcommittee to determine whether such a goal is desirable, and if so to describe procedures which comply with this goal. %**** Check, but it is desirable that they be so writable, so we will %**** have to watch. % % END "Basic Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Extended Proposal" % \subsection{Extended Proposal} The main additional features of the extended proposal are: \begin{itemize} \item Semantics for passing a group identifier in a message are defined, and a group registry is introduced. \item Point-to-point communication is extended to allow inter-group communication without syntactic intrusion on intra-group communication. \item Collective communication operations involving a concerted action on the part of members of two or more groups is suggested. \end{itemize} \subsubsection{Process and Group Identifiers} The extended proposal says nothing more about process identifiers than the basic proposal. The extended proposal defines a mechanism by which a group identifier may be passed in a message in such a manner that the received identifier is valid, and the mechanism should be analagous to that which allows process identifiers to be transmitted. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a group identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_gid(bd, \&gid)}. Transmission of a group identifier using this mechanism returns to the recipient a group identifier which is valid for use in the MPI operations described above and below, and as further qualified below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a group identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_invalidate\_gid(gid)}. This is an error if {\tt gid} is {\tt MPI\_GID\_NULL}, or if {\tt gid} is the identifier of a group of which the calling process is a member. %**** I understand the idea here, but not all the details. Can this %**** be justified/exemplified/simplified? %**** It is further proposed that MPI provide a group identifier registry service. This service allows any group to register its own group identifer by name, and deregister its group identifier. The service allows any group to determine whether a name has been registered without blocking the calling group, and to map that name into a validated group identifier with the possibility of blocking the calling group. The group registry procedures should synchronise the calling group, permitting more efficient implementations than asynchronous operations. The definition of a valid group identifier is extended to include all groups of which the process is a member and all those which are validated by one of the above mechanisms. The small suite of procedures which map between process identifiers, group identifiers and member ranks are defined to work with valid group and process identifiers. \subsubsection{Point-to-point communication} The point-to-point communication syntax and semantics described in the basic proposal do not extend directly to the case of inter-group communication. The reason is quite simple --- the sender and receiver do not supply the same group since the group of the sender and the group of the receiver are different. The basic approach is to express point-to-point communication in terms of a triplet {\tt (localGroup, remoteGroup, remoteRank)}. In this notation the sender specifies the sender group identifier, then the receiver group identifier and finally the receiver rank. The receiver specifies the receiver group identifier, then the sender group identifier, and finally the sender rank. The {\tt localGroup} field may not take a wildcard value, corresponding directly to the rule that the group in the basic proposal may not take a wildcard value. The {\tt remoteGroup} field may take a wildcard value, e.g. {\tt MPI\_GID\_WILD}, which matches with any group. The {\tt remoteRank} may also take a wildcard value, as in the basic proposal. The point-to-point section should provide procedures which allow a message recipient to recover the group and rank of the sender. In the case of {\tt remoteGroup} wildcard receive, the group identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the group identifier must be performed by the MPI implementation, in order that the recipient is returned a valid group identifier, else the returned identifier is of little or no use to the recipient. In order to accomodate inter-group addressing and selection into the framework of the basic proposal, the extended proposal suggests a careful redefinition of the {\tt gid} discussed in the basic proposal. With careful presentation this redefinition need not intrude conceptually on the basic proposal, although I shall give a less careful description here, suggestive of two different flavours of presentation. In the basic proposal, the {\tt gid} was formally a reference to a group. In the extended proposal the {\tt gid} is formally composed of references to two groups, and can be thought of as a shorthand notation for {\tt (localGroup, remoteGroup)}. %**** I dislike this intensely. There should be a group-pair data %**** structure. Group is never a pair of sub-groups. It is a %**** bad idea. This is all to get around changing syntax, no? The identifier of the null group, a {\it null\/} identifier, is a valid identifier which is formally composed of a pair of references to the null group, and may be used in the fashion described in the basic proposal. The group creation functions provide a symmetric, or {\it unary\/}, identifier formally composed of two references to the same group. This group is {\it local\/} since the process in question is a member of the group. An identifier which composes a pair of references to a local group is logically identical to a group identifier as implied in the basic proposal, and may be used for point-to-point and collective communications in an identical fashion. The group identifier transmission and registry lookup procedures also provide a symmetric, or {\it unary\/}, identifier which again is composed of a pair of references to the same group. This group is {\it remote\/} when the process is not a member of the group, or is local when the process is a member of the group. An identifier which composes a pair of references to a remote group is logically identical to the identifier a remote group as implied above, and is not valid for either point-to-point or collective communications. Inter-group communication is effected by use of assymetric, or {\it binary} identifiers, composed of references to two different groups, the first of which is local and the second of which is either local or remote. Such identifiers are constructed by the use of a ``glob'' operation, which returns an identifier referencing the first operand as the local field and the second operand as the remote field. The glob is defined to be an error unless: both operands are symmetric (unary); the first operand refers to a local group. The identifier returned by a glob is valid for point-to-point communciation and invalid for collective communication. The point-to-point communication section should provide a procedure to determine the group identifiers object of a completed receive. Procedures which ``unglob'' an assymetric (binary) group identifier should be provided, returning the local and remote fields as valid identifiers. This can be viewed as a continuation of a generalised approach to point-to-point communication began in the basic proposal, in which operation are specified in terms of object identifiers and instance identifiers where objects are composed of multiple instances. The nature of the instance identified, and the semantics of the operation, are dependent of the nature of the object identified, exploiting some genericity. \begin{center} \begin{tabular}{lll} object identifier & instance identifier & action \\ \hline null group identifier & basic process identifier & basic communication \\ unary group identifier & local process rank & intra-group communication \\ binary group identifier & remote process rank & inter-group communication \\ \end{tabular} \end{center} \subsubsection{Collective communication} There are a number of collective communication operations which logically extend over two or more groups. Some examples of two-group collective communications which are common in the simple host-node programming model are: \begin{itemize} \item {\it Broadcast\/} is an implicitly assymetric operation. There is exactly one sender and there are many receivers, each of which receives an identical message. The sender is a member of a singleton group G and the receivers are members of a group H. \item {\it Scatter\/} is an implicitly assymetric operation. There is exactly one sender and there are may receivers, each of which receives a different message. The sender is a member of a singleton group G and the receivers are members of a group H. \item There is a variant of {\it gather\/}, which returns the gathered data to a single process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \item There is a variant of {\it reduce}, which returns the reduced data to a sinle process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \end{itemize} Other patterns arise in ``process'' graphs where each ``process'' is allowed to be parallel. For example: \begin{itemize} \item {\it all-to-all\/} communciation in which the senders and receivers are distinct processes. The senders are members of a group G and the receivers are members of a group H. The two groups G and H need not be of the same size. \end{itemize} The assymetric (binary) group identifier objects described for point-to-point communications in this extended proposal are immediately suitable for each of the two-group collective communication operations described. % % END "Extended Proposal" %----------------------------------------------------------------------% %**** So, I gather that a set of groups is passable to a collcomm, %**** and a pair is passable to a pt2pt. That is neat, but it should %**** still be a separate data structure, with separate calls than %**** the intra-group version (at least for the pt2pt calls). %**** \section{Conclusion} This chapter has propose that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The chapter made a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The basic proposal provides expressive semantics for the case of intra-group communication such as arises in program which compose data driven parallelism, and is closely related to that which was discussed by Marc Snir at the February meeting in Dallas, and builds on discussions which have taken place in various subcommittees. The key additional features are: point-to-point communication can also be expressed in terms of process identifiers; a process registry service was added, use of which is optional. The extended proposal adds expressive semantics for the case of inter-group communication such as arises in programs which compose combinations of data and function driven parallelism. This functionality has been constructed in such a manner that there is no syntactic or performance intrusion on the content of the basic proposal, and the additional conceptual content can be presented seperately from a presentation of intra-group communication. % % END "Proposal I" %======================================================================% \end{document} >------------------------------ Cut Here ------------------------------< /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:47:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00323; Sun, 21 Mar 93 14:47:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24683; Sun, 21 Mar 93 14:46:49 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:46:48 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24675; Sun, 21 Mar 93 14:46:45 -0500 Date: Sun, 21 Mar 93 19:46:42 GMT Message-Id: <13090.9303211946@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal I - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is LaTeX of Proposal I, credits entirely to Marc Snir, 15 minutes earlier than previously advertised. PostScript to follow. My sincere apology to everyone who has prepared comments on Proposal I++. Some of these comments will carry through to Proposal II, which will be sent out shortly. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal I\\MPI Context Subcommittee} \author{Marc~Snir} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Written by Marc Snir % Edited by Lyndon J. Clarke % March 1993 % \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \chapter{Proposal I} {\it Editorial Note: This chapter is the proposal of Marc~Snir which also appears in the working documents of the point-to-point subcommittee (March 15) and collective communication subcommittee (March 16). This is a minor edit of the \LaTeX\ source of the point-to-point document provided by Marc. The edit was performed by Lyndon~Clarke.} \section{Contexts} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \subsection{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \subsection{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \subsection{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \subsection{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \section{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \subsection{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } % % END "Proposal I" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:49:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00329; Sun, 21 Mar 93 14:49:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24720; Sun, 21 Mar 93 14:48:46 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:48:45 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24711; Sun, 21 Mar 93 14:48:36 -0500 Date: Sun, 21 Mar 93 19:48:30 GMT Message-Id: <13097.9303211948@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal I - PostScript To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is PostScript of Proposal I, credits entirely to Marc Snir, 15 minutes earlier than previously advertised. LaTeX preceeded. ---------------------------------------------------------------------- %!PS-Adobe-2.0 %%Creator: dvips 5.495 Copyright 1986, 1992 Radical Eye Software %%Title: context-i.dvi %%CreationDate: Sun Mar 21 18:30:21 1993 %%Pages: 7 %%PageOrder: Ascend %%BoundingBox: 0 0 596 842 %%EndComments %DVIPSCommandLine: dvips context-i %DVIPSSource: TeX output 1993.03.21:1810 %%BeginProcSet: tex.pro %! /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N} B /TR{translate}N /isls false N /vsize 11 72 mul N /@rigin{isls{[0 1 -1 0 0 0] concat}if 72 Resolution div 72 VResolution div neg scale isls{0 Resolution vsize 72 div mul TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{ CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N} B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook} if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255 {IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V{}B /RV statusdict begin /product where{ pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{ gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M} B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{ p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 39158280 55380996 1000 300 300 (/a/obsidian/disk/home/u36/lyndon/mpi/context-i.dvi) @start /Fa 9 122 df<00E00001F00001F00001B00001B00003B80003B80003B800031800071C00071C 00071C00071C00071C000E0E000E0E000FFE000FFE001FFF001C07001C07001C07007F1FC0FF1F E07F1FC013197F9816>65 D76 D<003F00007F00003F0000070000070000070000070003 C7000FF7001FFF003C1F00780F00700700E00700E00700E00700E00700E00700E00700700F0070 0F003C1F001FFFE00FE7F007C7E014197F9816>100 D<03E00FF81FFC3C1E780E7007E007FFFF FFFFFFFFE000E000700778073C0F1FFE0FFC03F010127D9116>I<018003C003C0018000000000 000000007FC07FC07FC001C001C001C001C001C001C001C001C001C001C001C001C07FFFFFFF7F FF101A7D9916>105 D<7E0000FE00007E00000E00000E00000E00000E00000E7FE00E7FE00E7F E00E0F000E1E000E3C000E78000EF0000FF0000FF8000FBC000F1E000E0E000E07000E07807F87 F0FFCFF07F87F01419809816>107 D<7E3C00FEFE007FFF000F87800F03800E03800E03800E03 800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01512809116>110 D<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001F00000E00001E00001F00003B800073C 00071C000E0E007F1FC0FF3FE07F1FC013127F9116>120 D<7F1FC0FF9FE07F1FC01C07000E07 000E0E000E0E00070E00071C00071C00039C00039C0003980001B80001B80000F00000F00000F0 0000E00000E00000E00001C00079C0007BC0007F80003F00003C0000131B7F9116>I E /Fb 11 121 df<01C00003E00003E0000360000360000770000770000770000770000630000E 38000E38000E38000E38000E38001FFC001FFC001C1C001C1C003C1E00380E00FE3F80FE3F8011 177F9614>65 D76 D<1FC0007FF000707800201800001C00001C 0007FC001FFC003C1C00701C00E01C00E01C00E01C00707C003FFF800F8F8011107E8F14>97 D<03F80FFC1C1C380870006000E000E000E000E00060007000380E1C1E0FFC03F00F107E8F14> 99 D<07E00FF01C38301C700CE00EE00EFFFEFFFEE00060007000380E1C1E0FFC03F00F107E8F 14>101 D107 D110 D<07C01FF03C78701C701CE00EE00EE00E E00EE00EE00E701C783C3C781FF007C00F107E8F14>I 114 D<030007000700070007007FFCFFFC07000700070007000700070007000700070E070E070E 070C03FC00F00F157F9414>116 D<7E3F007E3F001E38000E780007700007E00003E00001C000 03C00003E0000770000E78000E38001C1C00FE3F80FE3F8011107F8F14>120 D E /Fc 1 16 df<07801FE03FF07FF87FF8FFFCFFFCFFFCFFFCFFFCFFFC7FF87FF83FF01FE007 800E107E9013>15 D E /Fd 48 123 df<00FC7C0183C607078E0607040E07000E07000E07000E 07000E07000E0700FFFFF00E07000E07000E07000E07000E07000E07000E07000E07000E07000E 07000E07000E07000E07000E07007F0FF0171A809916>11 D<00FC000182000703000607000E02 000E00000E00000E00000E00000E0000FFFF000E07000E07000E07000E07000E07000E07000E07 000E07000E07000E07000E07000E07000E07000E07007F0FE0131A809915>I<007E1F8001C170 400703C060060380E00E0380400E0380000E0380000E0380000E0380000E038000FFFFFFE00E03 80E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E 0380E00E0380E00E0380E00E0380E07F8FE3FC1E1A809920>14 D<00800100020004000C000800 18003000300030006000600060006000E000E000E000E000E000E000E000E000E000E000600060 0060006000300030003000180008000C00040002000100008009267D9B0F>40 D<8000400020001000180008000C00060006000600030003000300030003800380038003800380 0380038003800380038003000300030003000600060006000C0008001800100020004000800009 267E9B0F>I<60F0F07010101020204080040B7D830B>44 DI<60F0F060 04047D830B>I<60F0F060000000000000000060F0F06004107D8F0B>58 D<60F0F060000000000000000060F0F0701010102020408004177D8F0B>I<000C0000000C0000 000C0000001E0000001E0000003F000000270000002700000043800000438000004380000081C0 000081C0000081C0000100E0000100E00001FFE000020070000200700006007800040038000400 380008001C0008001C001C001E00FF00FFC01A1A7F991D>65 DI<003F0201C0C6 03002E0E001E1C000E1C0006380006780002700002700002F00000F00000F00000F00000F00000 F000007000027000027800023800041C00041C00080E000803003001C0C0003F00171A7E991C> I69 D72 DI77 DI<007F000001C1C000070070000E0038001C001C003C001E0038000E0078000F 0070000700F0000780F0000780F0000780F0000780F0000780F0000780F0000780F00007807800 0F0078000F0038000E003C001E001C001C000E0038000700700001C1C000007F0000191A7E991E >II<0FC21836200E6006C006C002C002C002E00070007E003FE01FF807FC003E 000E00070003800380038003C002C006E004D81887E0101A7E9915>83 D<7FFFFF00701C070040 1C0100401C0100C01C0180801C0080801C0080801C0080001C0000001C0000001C0000001C0000 001C0000001C0000001C0000001C0000001C0000001C0000001C0000001C0000001C0000001C00 00001C0000001C0000001C000003FFE000191A7F991C>I<3F8070C070E020700070007007F01C 7030707070E070E071E071E0F171FB1E3C10107E8F13>97 DI<07F80C1C381C30 087000E000E000E000E000E000E0007000300438080C1807E00E107F8F11>I<007E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E0003CE000C3E00380E00300E00700E00E0 0E00E00E00E00E00E00E00E00E00E00E00600E00700E00381E001C2E0007CFC0121A7F9915>I< 07C01C3030187018600CE00CFFFCE000E000E000E0006000300438080C1807E00E107F8F11>I< 01F0031807380E100E000E000E000E000E000E00FFC00E000E000E000E000E000E000E000E000E 000E000E000E000E000E007FE00D1A80990C>I<0FCE187330307038703870387038303018602F C02000600070003FF03FFC1FFE600FC003C003C003C0036006381C07E010187F8F13>II<18003C003C001800000000000000000000000000FC001C001C001C001C001C001C001C 001C001C001C001C001C001C001C00FF80091A80990A>I<018003C003C0018000000000000000 00000000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C0 01C001C041C0E180E3007E000A2182990C>IIII< FCF8001D0C001E0E001E0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E00 1C0E001C0E00FF9FC012107F8F15>I<07E01C38300C700E6006E007E007E007E007E007E00760 06700E381C1C3807E010107F8F13>II<03C2000C2600381E00300E00700E00E00E00E00E00E00E00E0 0E00E00E00E00E00700E00700E00381E001C2E0007CE00000E00000E00000E00000E00000E0000 0E00007FC012177F8F14>II<1F2060E04020C020C020F0007F003FC01FE000F080708030C030C0 20F0408F800C107F8F0F>I<0400040004000C000C001C003C00FFC01C001C001C001C001C001C 001C001C001C201C201C201C201C200E4003800B177F960F>IIIIII<7FF86070407040E041C041C003 80070007000E081C081C08381070107030FFF00D107F8F11>I E /Fe 36 121 df<004000800300020006000C001C001800380038007000700070007000F000F000F000F0 00F000F000F000F000F00070007000700070003800380018001C000C0006000200030000800040 0A257C9B11>40 D<800040003000100018000C000E00060007000700038003800380038003C003 C003C003C003C003C003C003C003C003800380038003800700070006000E000C00180010003000 400080000A257E9B11>I<70F8FCFCFC7C04040808102040060D7D850C>44 D<78FCFCFCFC78000000000078FCFCFCFC7806117D900C>58 D<00030000000780000007800000 078000000FC000000FC000001BE000001BE000001BE0000031F0000031F0000060F8000060F800 00E0FC0000C07C0000C07C0001803E0001FFFE0003FFFF0003001F0003001F0006000F8006000F 800E000FC0FFC07FFCFFC07FFC1E1A7F9921>65 D<001FE02000FFFCE003F80FE007C003E01F80 01E01F0000E03E0000E07E0000607C000060FC000000FC000000FC000000FC000000FC000000FC 000000FC000000FC0000007C0000607E0000603E0000601F0000C01F8000C007C0038003F80F00 00FFFC00001FF0001B1A7E9920>67 DII73 D77 D I<003FC00001E0780007801E000F000F001F000F803E0007C03E0007C07C0003E07C0003E0FC00 03F0FC0003F0FC0003F0FC0003F0FC0003F0FC0003F0FC0003F0FC0003F07C0003E07E0007E03E 0007C03E0007C01F000F800F801F0007C03E0001E07800003FC0001C1A7E9921>II82 D<07F0401FFDC03C0FC07803C07001C0F0 01C0F000C0F000C0F80000FF00007FF8003FFF001FFF800FFFC001FFE0000FE00003F00001F0C0 00F0C000F0C000F0E000E0F001E0FC03C0EFFF8083FE00141A7E9919>I<7FFFFF807FFFFF8078 1F0780701F0380601F0180E01F01C0C01F00C0C01F00C0C01F00C0001F0000001F0000001F0000 001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F00 00001F0000001F000007FFFC0007FFFC001A1A7E991F>I<7FF87FF07FF87FF007E0060003E00C 0003F01C0001F8380000F83000007C6000007EE000003FC000001F8000001F8000000FC000000F C000000FE000001BF0000039F8000030F8000060FC0000E07E0001C03E0001801F0003001F8007 000FC0FFE07FFCFFE07FFC1E1A7F9921>88 D<0FF0001C3C003E1E003E0E003E0F001C0F00000F 0000FF000FCF003E0F007C0F00F80F00F80F00F80F00F817007C27E01FC3E013117F9015>97 D<03FC000F0E001C1F003C1F00781F00780E00F80000F80000F80000F80000F800007800007800 003C01801C03000F060003FC0011117F9014>99 D<000FE0000FE00001E00001E00001E00001E0 0001E00001E00001E003F9E00F07E01C03E03C01E07801E07801E0F801E0F801E0F801E0F801E0 F801E07801E07801E03C01E01C03E00F0DFC03F9FC161A7F9919>I<03F0000E1C001C0E003C07 00780700780780F80780F80780FFFF80F80000F800007800007800003C01801C03000E060003FC 0011117F9014>I<00FE0003C700078F800F0F800F0F800F07000F00000F00000F0000FFF000FF F0000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00007FE0007FE000111A80990E>I104 D<1C003E003E003E003E001C00000000 00000000007E007E001E001E001E001E001E001E001E001E001E001E001E001E001E00FF80FF80 091B7F9A0D>I107 DIII<03F8000E0E003C0780 3803807803C07803C0F803E0F803E0F803E0F803E0F803E0F803E07803C07C07C03C07800E0E00 03F80013117F9016>II114 D<1FB020704030C030C030F000FF807FE03FF807F8003CC00C C00CE00CE008F830CFE00E117F9011>I<06000600060006000E000E001E003FF0FFF01E001E00 1E001E001E001E001E001E001E181E181E181E181E180F3003E00D187F9711>II119 DI E /Ff 31 122 df45 D<387CFEFEFE7C3807077C8610>I<00 180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC07007E07C07 F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C0000F80001F 00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07FFFF0FFFF F0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F81F03F81F 03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC3C00FE7E 00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C>I<0000E0 0001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187E00307E0 0707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007E00007E0 0007E00007E00007E000FFFE00FFFE17207E9F1C>I<0003FE0080001FFF818000FF01E38001F8 003F8003E0001F8007C0000F800F800007801F800007803F000003803F000003807F000001807E 000001807E00000180FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000 FE00000000FE000000007E000000007E000001807F000001803F000001803F000003801F800003 000F8000030007C000060003F0000C0001F800380000FF00F000001FFFC0000003FE000021227D A128>67 D70 D76 D85 D<07FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F000 3FF003FDF01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18 167E951B>97 DI<00FF 8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00 007C00007E00007E00003E00301F00600FC0E007FF8000FE0014167E9519>I<0001FE000001FE 0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E000000 3E0000003E0001FC3E0007FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC 003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E00 1E00FE000F83BE0007FF3FC001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C 00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F 00300FC07003FFC000FF0015167E951A>I<003F8000FFC001E3E003C7E007C7E00F87E00F83C0 0F80000F80000F80000F80000F80000F8000FFFC00FFFC000F80000F80000F80000F80000F8000 0F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8000 7FF8007FF80013237FA211>I<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03 E07C03E03C03C03E07C01F0F801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFF FC3800FC70003EF0001EF0001EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C >II<1C003F007F007F 007F003F001C000000000000000000000000000000FF00FF001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B247EA310>I108 DI I<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC007EFC00 7EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE0017167E951C>II114 D<0FF3003FFF00781F00600700E00300E00300F00300FC00007FE0007F F8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00C7F00011 167E9516>I<0180000180000180000180000380000380000780000780000F80003F8000FFFF00 FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8180 0F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>IIII121 D E /Fg 1 111 df<381F004E61804681C04701C08F01C08E01C00E01C00E01C01C 03801C03801C03801C0700380710380710380E10380E2070064030038014127E9119>110 D E /Fh 2 16 df0 D<03C00FF01FF83FFC7FFE7FFEFFFFFF FFFFFFFFFF7FFE7FFE3FFC1FF80FF003C010107E9115>15 D E /Fi 37 123 df<0020004001800380030006000E001C001C003C0038003800780078007800F800F000F0 00F000F000F000F000F000F000F000F800780078007800380038003C001C001C000E0006000300 03800180004000200B297C9E13>40 D<800040003000380018000C000E00070007000780038003 8003C003C003C003E001E001E001E001E001E001E001E001E001E003E003C003C003C003800380 0780070007000E000C00180038003000400080000B297D9E13>I<78FCFCFEFE7A020204040808 3040070E7D850D>44 D<00038000000380000007C0000007C0000007C000000FE000000FE00000 1FF000001BF000001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000C07E00 01803F0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0FFC07F FEFFC07FFE1F1C7E9B24>65 D<001FE02000FFF8E003F80FE007C003E00F8001E01F0000E03E00 00E03E0000607E0000607C000060FC000000FC000000FC000000FC000000FC000000FC000000FC 000000FC0000007C0000607E0000603E0000603E0000C01F0000C00F80018007C0030003F80E00 00FFFC00001FE0001B1C7D9B22>67 D69 D73 D75 DIII<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E 0003E07E0003F07C0001F07C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8 FC0001F8FC0001F87C0001F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E 0001F07C00003FE0001D1C7D9B24>II82 D<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE00 00FFE0007FFE003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000 E0E000E0F001C0FC03C0EFFF0083FC00141C7D9B1B>I<7FFFFFE07FFFFFE0781F81E0701F80E0 601F8060E01F8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F80 00001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F 8000001F8000001F800007FFFE0007FFFE001C1C7E9B21>II87 D<7FFE1FFE007FFE1FFE0007F001800003F803800001FC07000000FC06000000 FE0C0000007F1C0000003F380000003FB00000001FE00000000FE00000000FE000000007F00000 0003F800000007F80000000FFC0000000CFE000000187E000000387F000000703F800000601F80 0000C01FC00001C00FE000018007F000030007F000FFF03FFF80FFF03FFF80211C7F9B24>II<7FFFFC7FFFFC7E01F878 03F87003F0E007E0E007E0C00FC0C01FC0C01F80003F00007F00007E0000FC0000FC0001F80003 F80603F00607E0060FE0060FC00E1F800E1F801C3F001C7F003C7E00FCFFFFFCFFFFFC171C7D9B 1D>I<0FF8001C1E003E0F803E07803E07C01C07C00007C0007FC007E7C01F07C03C07C07C07C0 F807C0F807C0F807C0780BC03E13F80FE1F815127F9117>97 D<03FC000E0E001C1F003C1F0078 1F00780E00F80000F80000F80000F80000F80000F800007800007801803C01801C03000E0E0003 F80011127E9115>99 D<000FF0000FF00001F00001F00001F00001F00001F00001F00001F00001 F00001F001F9F00F07F01C03F03C01F07801F07801F0F801F0F801F0F801F0F801F0F801F0F801 F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C1B>I<01FC000F07001C03803C01C078 01C07801E0F801E0F801E0FFFFE0F80000F80000F800007800007C00603C00601E00C00F038001 FC0013127F9116>I<03F8F00E0F381E0F381C07303C07803C07803C07803C07801C07001E0F00 0E0E001BF8001000001800001800001FFF001FFFC00FFFE01FFFF07801F8F00078F00078F00078 7000707800F01E03C007FF00151B7F9118>103 D<1E003F003F003F003F001E00000000000000 000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FF E0FFE00B1E7F9D0E>105 D107 D110 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8 F800F8F800F8F800F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>I114 D<1FD830786018E018E018F000FF807FE07F F01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007000F 000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08079803F0 0E1A7F9913>I119 DII< 3FFF803C1F00303F00303E00607C0060FC0060F80001F00003F00007E00007C1800F81801F8180 1F03803E03007E07007C0F00FFFF0011127F9115>I E /Fj 15 121 df<3C7EFFFFFFFF7E3C08 087B8712>46 D<000C00001C0000FC000FFC00FFFC00F0FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC00FFFFFCFFFF FC16257BA420>49 D<00FF000007FFE0001E07F8003801FC007800FE007E00FF00FF007F00FF00 7F80FF007F80FF003F807E003F803C003F8000007F8000007F0000007F000000FE000000FE0000 01FC000001F8000003F0000007C00000078000000F0000001E0000003800000070018000E00180 01C001800380030007000300060003000FFFFF001FFFFF003FFFFF007FFFFE00FFFFFE00FFFFFE 0019257DA420>I<0000FF80080007FFF018003FC03C38007E000E7801FC0003F803F00001F807 E00000F80FE00000781FC00000781FC00000383F800000383F800000387F800000187F00000018 7F00000018FF00000000FF00000000FF00000000FF00000000FF00000000FF00000000FF000000 00FF00000000FF00000000FF000000007F000000007F000000187F800000183F800000183F8000 00181FC00000301FC00000300FE000006007E000006003F00000C001FC000180007E000700003F C03C000007FFF8000000FFC00025287CA72E>67 D<0001FF0000001FFFF000007F01FC0000FC00 7E0003F8003F8007F0001FC00FE0000FE00FC00007E01FC00007F03F800003F83F800003F83F80 0003F87F000001FC7F000001FC7F000001FCFF000001FEFF000001FEFF000001FEFF000001FEFF 000001FEFF000001FEFF000001FEFF000001FEFF000001FEFF000001FE7F000001FC7F000001FC 7F800003FC7F800003FC3F800003F83FC00007F81FC00007F00FC00007E00FE0000FE007F0001F C003F8003F8000FC007E00007F01FC00001FFFF0000001FF000027287CA730>79 D<03FF00000FFFE0001F03F0003F80F8003F80FC003F807C001F007E001F007E0000007E000000 7E0000007E00000FFE0001FFFE0007F07E001FC07E003F807E007F007E00FE007E00FE007E18FE 007E18FE007E18FE00BE187F01BE183F873FF01FFE1FE003F80F801D1A7E9920>97 D<007F8003FFE007E1F00F80F81F007C3F007E7E003E7E003E7E003FFE003FFE003FFFFFFFFFFF FFFE0000FE0000FE0000FE00007E00007E00003F00003F00031F80060FC00607F01C01FFF0003F C0181A7E991D>101 D<0F001F801FC03FC03FC01FC01F800F0000000000000000000000000000 00FFC0FFC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC0 0FC00FC00FC00FC00FC0FFFCFFFC0E297FA811>105 D110 D<007FC00001FFF00007E0FC000F803E001F001F00 3F001F803E000F807E000FC07E000FC0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000F E0FE000FE0FE000FE07E000FC07E000FC07F001FC03F001F801F001F000F803E0007E0FC0001FF F000007FC0001B1A7E9920>II114 D<03F8C01FFFC03C07C07801C07001C0F000C0 F000C0F800C0FC0000FFC0007FFC003FFF001FFF800FFFC001FFE0000FF00003F0C001F0C000F0 E000F0E000F0F000E0F801E0FE03C0E7FF80C1FC00141A7E9919>I<0060000060000060000060 0000E00000E00001E00001E00003E00007E0001FE000FFFFC0FFFFC007E00007E00007E00007E0 0007E00007E00007E00007E00007E00007E00007E00007E00007E00007E00007E06007E06007E0 6007E06007E06007E06003F0C001F0C000FF80003E0013257FA419>I120 D E /Fk 1 98 df<00200000700000700000700000B80000B80000B800011C00011C00011C00020E00020E0004 070004070007FF000803800803800803801801C03803C0FE0FF815157F9419>97 D E /Fl 62 123 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 0E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00 000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<003F07E00001C0 9C18000380F018000701F03C000E01E03C000E00E018000E00E000000E00E000000E00E000000E 00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C00 0E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>14 D<6060F0F0F8F868680808 08080808101010102020404080800D0C7F9C15>34 D<60F0F8680808081010204080050C7C9C0C >39 D<004000800100020006000C000C0018001800300030007000600060006000E000E000E000 E000E000E000E000E000E000E000E000E000600060006000700030003000180018000C000C0006 0002000100008000400A2A7D9E10>I<800040002000100018000C000C00060006000300030003 8001800180018001C001C001C001C001C001C001C001C001C001C001C001C00180018001800380 03000300060006000C000C00180010002000400080000A2A7E9E10>I<60F0F070101010102020 4080040C7C830C>44 DI<60F0F06004047C830C>I<03C00C301818300C 300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E0076006600670 0E300C300C18180C3007E0101D7E9B15>48 D<030007003F00C700070007000700070007000700 07000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C7C 9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C0038007000 6000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<07E01830201C201C781E78 0E781E381E001C001C00180030006007E00030001C001C000E000F000F700FF80FF80FF80FF00E 401C201C183007E0101D7E9B15>I<000C00000C00001C00003C00003C00005C0000DC00009C00 011C00031C00021C00041C000C1C00081C00101C00301C00201C00401C00C01C00FFFFC0001C00 001C00001C00001C00001C00001C00001C0001FFC0121C7F9B15>I<300C3FF83FF03FC0200020 00200020002000200023E024302818301C200E000E000F000F000F600FF00FF00FF00F800E401E 401C2038187007C0101D7E9B15>I<00F0030C06040C0E181E301E300C700070006000E3E0E430 E818F00CF00EE006E007E007E007E007E007600760077006300E300C18180C3003E0101D7E9B15 >I<60F0F0600000000000000000000060F0F06004127C910C>58 D<60F0F06000000000000000 00000060F0F0701010101020204080041A7C910C>I<000600000006000000060000000F000000 0F0000000F00000017800000178000001780000023C0000023C0000023C0000041E0000041E000 0041E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C0006003E 0004001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 D<001F808000E061800180 1980070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F0 000000F0000000F0000000F0000000F0000000F0000000F0000000700000807800008078000080 380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>67 D69 DI73 D76 DII<003F800000E0E0000380380007001C000E00 0E001C0007003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F0 0001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C0780003C0380003803C000780 1C0007000E000E0007001C000380380000E0E000003F80001B1E7E9C20>II82 D<07E0801C1980300580700380600180E00180E00080E00080E00080 F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001C08001C0 8001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>I<7FFFFFC0700F01C0 600F00C0400F0040400F0040C00F0020800F0020800F0020800F0020000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000001F800003FFFC001B1C7F9B1E>II< FFE0FFE0FF1F001F003C1E001E00180F001F00100F001F00100F001F001007801F002007802780 20078027802003C027804003C043C04003C043C04003E043C04001E081E08001E081E08001E081 E08000F100F10000F100F10000F100F100007900FA00007A007A00007A007A00003E007C00003C 003C00003C003C00003C003C00001800180000180018000018001800281D7F9B2B>87 D<7FF0FFC00FC03E000780180003C0180003E0100001E0200001F0600000F0400000788000007D 8000003D0000001E0000001F0000000F0000000F8000000F80000013C0000023E0000021E00000 41F00000C0F8000080780001007C0003003C0002001E0006001F001F003F80FFC0FFF01C1C7F9B 1F>I<08081010202040404040808080808080B0B0F8F8787830300D0C7A9C15>92 D<1FC000307000783800781C00301C00001C00001C0001FC000F1C00381C00701C00601C00E01C 40E01C40E01C40603C40304E801F870012127E9115>97 D I<07E00C301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E12 7E9112>I<003F0000070000070000070000070000070000070000070000070000070000070003 E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E0070060070070 0700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E6006E006FFFEE000 E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C071E061E0E0C0E000E00 0E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C00381C00381C00381C00 1818001C38000C300013C0001000003000001800001FF8001FFF001FFF803003806001C0C000C0 C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C001C001C001C001C001C 001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C0000000 00000000000000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>IIIII<03F000 0E1C00180600300300700380600180E001C0E001C0E001C0E001C0E001C0E001C0600180700380 3003001806000E1C0003F00012127F9115>II<03C1000C3300180B00300F0070 0700700700E00700E00700E00700E00700E00700E00700600700700700300F00180F000C370007 C700000700000700000700000700000700000700000700003FE0131A7E9116>II<1F90 30704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608FC00D127F9110 >I<04000400040004000C000C001C003C00FFE01C001C001C001C001C001C001C001C001C001C 101C101C101C101C100C100E2003C00C1A7F9910>IIII<7F8FF00F03800F03000702 0003840001C80001D80000F00000700000780000F800009C00010E00020E000607000403801E07 C0FF0FF81512809116>II<7FFC70386038407040F040E041C003C0038007000F 040E041C043C0C380870087038FFF80E127F9112>I E /Fm 38 122 df<000300060008001800 30006000C000C0018003000300060006000C000C001C0018001800380030003000700070006000 600060006000E000E000E000E000E0006000600060006000600020003000100008000800102A7B 9E11>40 D<001000100008000C0004000600060006000600060007000700070007000700060006 00060006000E000E000C000C001C001800180038003000300060006000C000C001800300030006 000C00180010006000C000102A809E11>I45 D<3078F06005047C830D>I<00020006000C001C007C039C003800380038003800700070007000 7000E000E000E000E001C001C001C001C003800380038003800780FFF00F1C7C9B15>49 D<00C06000FFC001FF8001FE00010000010000020000020000020000020000047800058C000606 00040600080600000700000700000600000E00000E00700E00700C00E01C008018008038004030 0040600021C0001F0000131D7C9B15>53 D<000F0000308000C080018380038380030000060000 0E00000C00001C00001CF0003B18003C0C00380C00780C00700E00700E00700E00601C00E01C00 E01C00E01C00E03800E03800E0700060600060C0002180001E0000111D7B9B15>I<060F0F0600 0000000000000000003078F06008127C910D>58 D<0003F020001E0C60003002E000E003C001C0 01C0038001C0070000C00E0000801E0000801C0000803C0000803C000000780000007800000078 000000F0000000F0000000F0000000F0000000F0000400F0000400F0000400F000080070000800 7000100038002000180040000C0180000706000001F800001B1E7A9C1E>67 D<01FFFFE0003C00E0003800600038004000380040003800400070004000700040007020400070 200000E0400000E0400000E0C00000FFC00001C0800001C0800001C0800001C080000381010003 8001000380020003800200070004000700040007000C00070018000E007800FFFFF0001B1C7D9B 1C>69 D<01FFE0003C0000380000380000380000380000700000700000700000700000E00000E0 0000E00000E00001C00001C00001C00001C0000380080380080380080380100700100700300700 600700E00E03C0FFFFC0151C7D9B1A>76 D<01FE0007F8003E000780002E000F00002E00170000 2E001700002E002700004E002E00004E004E00004E004E00004E008E00008E011C00008E011C00 008E021C00008E021C000107043800010704380001070838000107103800020710700002072070 0002072070000207407000040740E000040780E000040700E0000C0700E0001C0601E000FF861F FC00251C7D9B25>I<01FC03FE001C0070003C0060002E0040002E0040002E0040004700800047 008000470080004380800083810000838100008181000081C1000101C2000101C2000100E20001 00E2000200E4000200740002007400020074000400380004003800040038000C0018001C001000 FF8010001F1C7D9B1F>I<000F8400304C00403C00801801001803001803001806001006001006 000007000007000003E00003FC0001FF00007F800007C00001C00001C00000C00000C02000C020 00C0600180600180600300600200F00400CC180083E000161E7D9C17>83 D<1FFFFFC01C0701C0300E00C0200E0080600E0080400E0080401C0080801C0080801C0080001C 0000003800000038000000380000003800000070000000700000007000000070000000E0000000 E0000000E0000000E0000001C0000001C0000001C0000001C0000003C000007FFE00001A1C799B 1E>I<03CC063C0C3C181C3838303870387038E070E070E070E070E0E2C0E2C0E261E462643C38 0F127B9115>97 D<3F00070007000E000E000E000E001C001C001C001C0039C03E603830383070 38703870387038E070E070E070E060E0E0C0C0C1C0618063003C000D1D7B9C13>I<01F007080C 08181C3838300070007000E000E000E000E000E000E008E010602030C01F000E127B9113>I<00 1F80000380000380000700000700000700000700000E00000E00000E00000E0003DC00063C000C 3C00181C00383800303800703800703800E07000E07000E07000E07000E0E200C0E200C0E20061 E4006264003C3800111D7B9C15>I<01E007100C1018083810701070607F80E000E000E000E000 E000E0086010602030C01F000D127B9113>I<0003C0000670000C70001C60001C00001C000038 0000380000380000380000380003FF8000700000700000700000700000700000E00000E00000E0 0000E00000E00001C00001C00001C00001C00001C0000380000380000380000300000300000700 00C60000E60000CC00007800001425819C0D>I<00F3018F030F06070E0E0C0E1C0E1C0E381C38 1C381C381C383830383038187818F00F700070007000E000E0C0C0E1C0C3007E00101A7D9113> I<0FC00001C00001C0000380000380000380000380000700000700000700000700000E78000E8C 000F0E000E0E001C0E001C0E001C0E001C0E00381C00381C00381C003838007038807038807070 80707100E03200601C00111D7D9C15>I<01800380010000000000000000000000000000001C00 2600470047008E008E000E001C001C001C0038003800710071007100720072003C00091C7C9B0D >I<0FC00001C00001C0000380000380000380000380000700000700000700000700000E0F000E 11000E23800E43801C83001C80001D00001E00003F800039C00038E00038E00070E20070E20070 E20070E400E06400603800111D7D9C13>107 D<1F800380038007000700070007000E000E000E 000E001C001C001C001C0038003800380038007000700070007000E400E400E400E40068003800 091D7C9C0B>I<3C1E0780266318C04683A0E04703C0E08E0380E08E0380E00E0380E00E0380E0 1C0701C01C0701C01C0701C01C070380380E0388380E0388380E0708380E0710701C0320300C01 C01D127C9122>I<3C3C002646004687004707008E07008E07000E07000E07001C0E001C0E001C 0E001C1C00381C40381C40383840383880701900300E0012127C9117>I<01E007180C0C180C38 0C300E700E700EE01CE01CE01CE018E038E030E06060C031801E000F127B9115>I<07870004D9 8008E0C008E0C011C0E011C0E001C0E001C0E00381C00381C00381C00381800703800703000707 000706000E8C000E70000E00000E00001C00001C00001C00001C00003C0000FF8000131A7F9115 >I<3C3C26C2468747078E068E000E000E001C001C001C001C0038003800380038007000300010 127C9112>114 D<01F006080C080C1C18181C001F001FC00FF007F0007800386030E030C03080 6060C01F000E127D9111>I<00C001C001C001C00380038003800380FFE00700070007000E000E 000E000E001C001C001C001C00384038403840388019000E000B1A7D990E>I<1E030027070047 0700470700870E00870E000E0E000E0E001C1C001C1C001C1C001C1C0038388038388018388018 39001C5900078E0011127C9116>I<1E06270E470E4706870287020E020E021C041C041C041C08 18083808181018200C4007800F127C9113>I<1E01832703874703874703838707018707010E07 010E07011C0E021C0E021C0E021C0E04180C04181C04181C081C1C100C263007C3C018127C911C >I<070E0019910010E38020E38041C30041C00001C00001C00003800003800003800003800007 0200670200E70400CB04008B080070F00011127D9113>I<1E03270747074707870E870E0E0E0E 0E1C1C1C1C1C1C1C1C38383838183818381C7007F00070007000E0E0C0E1C0818047003C00101A 7C9114>I E /Fn 8 116 df73 D80 D<0007FFC0000000003FFFFC00000000FFFFFF00000003F801 FFC0000007F0003FE0000007F8001FF000000FFC000FF800000FFC000FFC00000FFC0007FC0000 0FFC0007FE00000FFC0003FE000007F80003FF000003F00003FF000000000003FF000000000003 FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000 0007FFFF00000000FFFFFF0000000FFFE3FF0000007FF803FF000001FFC003FF000003FF0003FF 000007FC0003FF00000FF80003FF00001FF00003FF00003FF00003FF00007FE00003FF00007FE0 0003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00007FF03 80FFC00007FF03807FE0000DFF03807FE0001DFF03803FF00039FF87001FF80070FFCF000FFE03 E07FFE0007FFFF807FFC0000FFFE001FF800001FF00007E000312E7CAD37>97 D<00FF80FFFF80FFFF80FFFF8003FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF80FFFFFFFFFFFFFFFFFF18487DC71F>108 D<00003FFC00000001FFFF8000000FFFFFF000003FF00FFC00007F8001FE0001FE00007F8003FC 00003FC007F800001FE007F800001FE00FF000000FF01FE0000007F81FE0000007F83FE0000007 FC3FE0000007FC7FC0000003FE7FC0000003FE7FC0000003FE7FC0000003FEFFC0000003FFFFC0 000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003 FFFFC0000003FFFFC0000003FF7FC0000003FE7FC0000003FE7FC0000003FE7FE0000007FE3FE0 000007FC3FE0000007FC1FE0000007F81FF000000FF80FF000000FF007F800001FE007FC00003F E003FE00007FC001FF0000FF80007F8001FE00003FF00FFC00000FFFFFF0000003FFFFC0000000 3FFC0000302E7DAD37>111 D<00FF801FF80000FFFF80FFFF0000FFFF83FFFFC000FFFF8FC07F F00003FF9E000FF80001FFF80007FC0001FFF00003FE0001FFE00001FF0001FFC00000FF8001FF 800000FFC001FF8000007FC001FF8000007FE001FF8000007FE001FF8000003FE001FF8000003F F001FF8000003FF001FF8000003FF001FF8000001FF801FF8000001FF801FF8000001FF801FF80 00001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF8 01FF8000001FF801FF8000001FF801FF8000001FF001FF8000003FF001FF8000003FF001FF8000 003FF001FF8000003FE001FF8000007FE001FF8000007FC001FF800000FFC001FF800000FF8001 FFC00001FF8001FFE00001FF0001FFF00003FE0001FFF8000FFC0001FF9E001FF80001FF8F80FF E00001FF87FFFFC00001FF81FFFE000001FF803FF0000001FF800000000001FF800000000001FF 800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80000000 0001FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80 0000000001FF800000000001FF800000000001FF8000000000FFFFFF00000000FFFFFF00000000 FFFFFF0000000035427DAD3D>I<00FF007F00FFFF01FFC0FFFF03FFE0FFFF0787F003FF0E0FF0 01FF1C1FF801FF381FF801FF301FF801FF701FF801FF600FF001FF600FF001FFE003C001FFC000 0001FFC0000001FFC0000001FFC0000001FF80000001FF80000001FF80000001FF80000001FF80 000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF 80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001 FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF800000FFFFFFC000 FFFFFFC000FFFFFFC000252E7DAD2C>114 D<001FFC030000FFFF870003FFFFCF000FE007FF00 1F8000FF003E00003F003E00001F007C00001F007C00000F00FC00000F00FC00000700FC000007 00FE00000700FF00000700FF80000000FFE00000007FFE0000007FFFF800003FFFFF00003FFFFF C0001FFFFFF0000FFFFFF80007FFFFFC0001FFFFFE00007FFFFF00001FFFFF000000FFFF800000 07FF80000000FFC0E000007FC0E000003FC0E000001FC0F000000FC0F000000FC0F800000FC0F8 00000FC0F800000F80FC00000F80FE00001F80FF00001F00FF80003E00FFC0007C00F9F803F800 F0FFFFF000E03FFFC000C007FE0000222E7CAD2B>I E /Fo 8 117 df<00003000000070000001 F0000007F000007FF000FFFFF000FFFFF000FF8FF000000FF000000FF000000FF000000FF00000 0FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000 000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF0 00000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000F F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00000 0FF0007FFFFFFE7FFFFFFE7FFFFFFE1F3779B62D>49 D<000000FFE0001800000FFFFC00180000 7FFFFF00380001FFE00FC0780007FE0001F0F8000FF8000079F8003FE000003FF8007FC000000F F800FF80000007F801FF00000007F803FE00000003F807FC00000001F807F800000001F80FF800 000000F80FF000000000F81FF000000000781FF000000000783FE000000000783FE00000000038 7FE000000000387FE000000000387FE000000000387FC000000000007FC00000000000FFC00000 000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FF C00000000000FFC00000000000FFC00000000000FFC00000000000FFC000000000007FC0000000 00007FC000000000007FE000000000007FE000000000387FE000000000383FE000000000383FE0 00000000381FF000000000381FF000000000700FF000000000700FF8000000007007F800000000 E007FC00000000E003FE00000001C001FF000000038000FF8000000780007FC000000F00003FE0 00001E00000FF800003C000007FE0000F0000001FFE007E00000007FFFFF800000000FFFFE0000 000000FFE00000353B7BB940>67 D<003FFC00000001FFFF80000007E00FE000000FC003F80000 0FE001FC00001FF001FE00001FF000FF00001FF000FF00001FF0007F00000FE0007F800007C000 7F80000000007F80000000007F80000000007F80000000007F80000000007F800000000FFF8000 0007FFFF8000003FE07F800001FF007F800007FC007F80000FF0007F80001FE0007F80003FE000 7F80007FC0007F80007FC0007F8380FF80007F8380FF80007F8380FF80007F8380FF8000BF8380 FF8000BF83807FC0013F83807FC0033F83803FE0061FC7001FF81C0FFE0007FFF007FC00007FC0 03F00029257DA42D>97 D<0003FF0000001FFFE000007F07F00001FC01FC0003F000FE0007E000 7F000FE0003F001FC0003F801FC0003F803FC0001FC03F80001FC07F80001FC07F80001FE07F80 001FE0FF80001FE0FF80001FE0FFFFFFFFE0FFFFFFFFE0FF80000000FF80000000FF80000000FF 80000000FF800000007F800000007F800000007F800000003FC00000003FC00000001FC00000E0 1FE00000E00FE00001C007F000038003F800070000FE000E00007FC07C00001FFFF0000001FF80 0023257EA428>101 D<01FC00000000FFFC00000000FFFC00000000FFFC0000000007FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC01FF000003FC07FFC00003FC1C0FF00003FC 3007F80003FC6003F80003FCC003FC0003FC8001FC0003FD0001FE0003FF0001FE0003FE0001FE 0003FE0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE00FFFFF07FFFF8FFFFF07FFF F8FFFFF07FFFF82D3A7EB932>104 D<01FC03FE0000FFFC1FFFC000FFFC780FF000FFFDE003F8 0007FF8001FC0003FF0000FE0003FE0000FF0003FC00007F8003FC00007FC003FC00003FC003FC 00003FE003FC00003FE003FC00001FE003FC00001FE003FC00001FF003FC00001FF003FC00001F F003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC 00001FE003FC00003FE003FC00003FE003FC00003FC003FC00003FC003FC00007F8003FC00007F 8003FE0000FF0003FF0001FE0003FF8003FC0003FDC007F80003FCF81FE00003FC3FFF800003FC 07FC000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 00000000FFFFF0000000FFFFF0000000FFFFF00000002C357EA432>112 D<01F80FC0FFF83FF0FFF870F8FFF8C1FC07F883FE03F983FE03F903FE03FB03FE03FA01FC03FA 00F803FA007003FE000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC0000 03FC000003FC000003FC000003FC000003FC0000FFFFF800FFFFF800FFFFF8001F257EA424> 114 D<001C0000001C0000001C0000001C0000001C0000003C0000003C0000003C0000003C0000 007C0000007C000000FC000000FC000001FC000003FC000007FC00001FFFFFC0FFFFFFC0FFFFFF C003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC 000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E001FC00C001FC01C0 01FE01C000FE0380007F8700001FFE000007F8001B357EB423>116 D E /Fp 11 115 df<008003800F80F380038003800380038003800380038003800380038003800380 03800380038003800380038003800380038003800380038003800380038007C0FFFE0F217CA018 >49 D<03F8000C1E001007002007804007C07807C07803C07807C03807C0000780000780000700 000F00000E0000380003F000001C00000F000007800007800003C00003C00003E02003E07003E0 F803E0F803E0F003C04003C0400780200780100F000C1C0003F00013227EA018>51 D<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0F001E0F001E0F001 E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001C00001C00001C00003 80000380300300780700780600700C002018001030000FC00013227EA018>57 D77 D<03F0200C0C601802603001E07000E060 0060E00060E00060E00020E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003 FF80003FC00007E00001E00000F00000F0000070800070800070800070800070C00060C00060E0 00C0F000C0C80180C6070081FC0014247DA21B>83 D<0FE0001838003C0C003C0E001807000007 0000070000070000FF0007C7001E07003C0700780700700700F00708F00708F00708F00F087817 083C23900FC1E015157E9418>97 D<01FE000703000C07801C0780380300780000700000F00000 F00000F00000F00000F00000F00000F000007000007800403800401C00800C010007060001F800 12157E9416>99 D<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7FF18237FA2 1B>104 D<1C001E003E001E001C00000000000000000000000000000000000E00FE001E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC00A227FA10E >I<0E1F80FE60C01E80E00F00700F00700E00700E00700E00700E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E0070FFE7FF18157F941B>110 D<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E000E000E00 0F00FFF010157F9413>114 D E /Fq 20 121 df<00003FC0020001FFF8060007E01C06001F00 070E003C00018E00780000DE00F000005E01E000003E03C000001E078000001E0F0000000E0F00 00000E1E000000061E000000063E000000063C000000027C000000027C000000027C0000000278 00000000F800000000F800000000F800000000F800000000F800000000F800000000F800000000 F800000000F800000000F80000000078000000007C000000007C000000027C000000023C000000 023E000000021E000000021E000000040F000000040F0000000C078000000803C000001801E000 001000F00000200078000040003C000180001F0003000007E01E000001FFF80000003FC0002732 7CB02F>67 D 73 D77 D80 D<007F004001FFC0400780F0C00F0018C01C000DC03C0007C0 380003C0780001C0700001C0F00000C0F00000C0F00000C0F0000040F0000040F0000040F80000 40F80000007C0000007E0000003F0000003FE000001FFE00000FFFE00007FFF80001FFFE00007F FF000007FF8000007F8000000FC0000007E0000003E0000003E0000001F0000001F0800000F080 0000F0800000F0800000F0800000F0C00000F0C00000E0E00001E0E00001E0F00001C0F8000380 EC000780C7000F00C1E03E0080FFF800801FE0001C327CB024>83 D<00FE0000070380000801E0 001000F0003C0078003E0078003E003C003E003C001C003C0000003C0000003C0000003C000007 FC00007C3C0003E03C0007803C001F003C003E003C003C003C007C003C0078003C08F8003C08F8 003C08F8003C08F8007C08F8007C087C00BC083E011E100F060FE003F807C01D1E7D9D20>97 D<018000003F800000FF800000FF8000000F800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 81F80007860700079803C007A000E007C000F007C00078078000380780003C0780003E0780001E 0780001E0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F078000 1E0780003E0780003C0780003C0780007807C00070074000F0072001C006100380060C0E000403 F80020317EB024>I<001FC00000F0380001C00400078002000F000F001E001F001E001F003C00 1F007C000E007C00000078000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000007C0000003C0000801E0000801E0001000F00010007800200 01C00C0000F03000001FC000191E7E9D1D>I<003F800000E0F0000380380007001C000E001E00 1E000E003C000F003C000F007C000F807800078078000780F8000780FFFFFF80F8000000F80000 00F8000000F8000000F8000000F800000078000000780000007C0000003C0000801C0000801E00 01000F0002000700020001C00C0000F03000001FC000191E7E9D1D>101 D<07000F801F801F800F8007000000000000000000000000000000000000000000000001801F80 FF80FF800F80078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800FC0FFF8FFF80D2F7EAE12>105 D<01803F80FF80FF800F80078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800FC0FFFCFFFC0E317EB012>108 D<0181FE003FC0003F860780C0F000FF8801C1003800FF90 01E2003C000FA000E4001C0007A000F4001E0007C000F8001E0007C000F8001E00078000F0001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341E7E9D38>I<01 81FC003F860F00FF880380FF9003C00FA001C007C001E007C001E007C001E0078001E0078001E0 078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00FC003F0FFFC3FFFFFFC 3FFF201E7E9D24>I<003FC00000E0700003801C0007000E000E0007001E0007803C0003C03C00 03C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F8 0001F0F80001F0780001E0780001E07C0003E03C0003C03C0003C01E0007800E00070007000E00 03801C0000E07000003FC0001C1E7E9D20>I<0181F8003F860F00FF9803C0FFA001E007C000F0 07C00078078000780780003C0780003E0780003E0780001E0780001F0780001F0780001F078000 1F0780001F0780001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C0 00F007C000F007A001C007900380078C0E000783F8000780000007800000078000000780000007 8000000780000007800000078000000780000007800000078000000FC00000FFFC0000FFFC0000 202C7E9D24>I<0183E03F8C18FF907CFF907C0FA07C07C03807C00007C00007C0000780000780 000780000780000780000780000780000780000780000780000780000780000780000780000780 000780000780000780000FC000FFFE00FFFE00161E7E9D19>114 D<03FC200E02603801E03000 E0600060E00060E00020E00020F00020F000207C00007F80003FFC001FFF0007FF8001FFE0000F E00001F08000F8800078800038C00038C00038C00038E00030F00070F00060C800C0C6038081FE 00151E7E9D19>I<00400000400000400000400000400000C00000C00000C00001C00001C00003 C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C01003C01003C01003C01003C01003C01003 C01003C01001C02001E02000E0400078C0001F00142B7FAA19>I<018000603F800FE0FF803FE0 FF803FE00F8003E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00780 03E0078003E0078003E0038005E003C009E001C019F000F021FF001FC1FF201E7E9D24>I120 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%PaperSize: A4 %%EndSetup %%Page: 0 1 0 0 bop 831 1086 a Fq(Prop)r(osal)24 b(I)575 1178 y(MPI)f(Con)n(text)f(Sub)r (committee)871 1360 y Fp(Marc)16 b(Snir)853 1481 y(Marc)o(h)g(1993)p eop %%Page: 1 2 1 1 bop 262 619 a Fo(Chapter)28 b(1)262 826 y Fn(Prop)s(osal)36 b(I)262 1042 y Fm(Editorial)17 b(Note:)24 b(This)18 b(chapter)g(is)f(the)h (pr)n(op)n(osal)g(of)g(Mar)n(c)f(Snir)h(which)g(also)f(app)n(e)n(ars)262 1092 y(in)h(the)h(working)f(do)n(cuments)h(of)f(the)h(p)n(oint-to-p)n(oint)f (sub)n(c)n(ommitte)n(e)g(\(Mar)n(ch)h(15\))g(and)262 1142 y(c)n(ol)r(le)n (ctive)13 b(c)n(ommunic)n(ation)i(sub)n(c)n(ommitte)n(e)f(\(Mar)n(ch)g(16\).) 19 b(This)14 b(is)g(a)g(minor)g(e)n(dit)g(of)h(the)262 1191 y Fl(L)273 1186 y Fk(a)292 1191 y Fl(T)315 1204 y(E)338 1191 y(X)j Fm(sour)n(c)n(e)h(of)f(the)g(p)n(oint-to-p)n(oint)h(do)n(cument)g(pr)n (ovide)n(d)g(by)f(Mar)n(c.)29 b(The)19 b(e)n(dit)f(was)262 1241 y(p)n(erforme)n(d)c(by)h(Lyndon)h(Clarke.)262 1378 y Fj(1.1)64 b(Con)n(texts)262 1469 y Fl(A)13 b Fi(con)o(text)g Fl(consists)i(of:)324 1552 y Fh(\017)20 b Fl(A)15 b(set)h(of)f(pro)q(cesses)j(that)d(curren)o(tly)h (b)q(elong)f(to)f(the)i(con)o(text)g(\(p)q(ossibly)f(all)f(pro-)365 1602 y(cesses,)i(or)e(a)g(prop)q(er)g(subset\).)324 1685 y Fh(\017)20 b Fl(A)15 b Fi(ranking)e Fl(of)h(the)h(pro)q(cesses)j(within)c (that)h(con)o(text,)g(i.e.,)e(a)i(n)o(um)o(b)q(ering)e(of)h(the)365 1735 y(pro)q(cesses)e(in)d(that)h(con)o(text)g(from)e(0)h(to)g Fg(n)p Fh(\000)p Fl(1,)i(where)g Fg(n)e Fl(is)g(the)h(n)o(um)o(b)q(er)f(of)g (pro)q(cesses)365 1785 y(in)14 b(that)g(con)o(text.)324 1868 y(A)g(pro)q(cess)h(ma)o(y)d(b)q(elong)i(to)g(sev)o(eral)g(con)o(texts)h(at)f (the)g(same)f(time.)324 1918 y(An)o(y)f(in)o(terpro)q(cess)h(comm)o (unication)c(o)q(ccurs)k(within)e(a)h(con)o(text,)g(and)g(messages)g(sen)o(t) 262 1968 y(within)h(one)i(con)o(text)g(can)f(b)q(e)h(receiv)o(ed)h(only)e (within)f(the)i(same)f(con)o(text.)20 b(A)15 b(con)o(text)g(is)262 2017 y(sp)q(eci\014ed)d(using)e(a)g Fm(c)n(ontext)i(hand)r(le)g Fl(\(i.e.,)d(a)i(handle)f(to)g(an)h(opaque)f(ob)r(ject)i(that)e(iden)o (ti\014es)262 2067 y(a)15 b(con)o(text\).)23 b(Con)o(text)15 b(handles)h(cannot)g(b)q(e)g(transferred)h(for)e(one)g(pro)q(cess)i(to)f (another;)262 2117 y(they)e(can)g(b)q(e)h(used)f(only)f(on)h(the)h(pro)q (cess)g(where)g(they)g(where)g(created.)324 2167 y(F)m(ollo)o(ws)d(examples)h (of)g(p)q(ossible)h(uses)h(for)f(con)o(texts.)262 2283 y Ff(1.1.1)55 b(Lo)r(osely)17 b(sync)n(hronous)i(library)e(call)h(in)n(terface)262 2360 y Fl(Consider)g(the)g(case)h(where)g(a)e(parallel)g(application)f (executes)k(a)d(\\parallel)g(call")f(to)i(a)262 2409 y(library)c(routine,)h (i.e.,)f(where)j(all)d(pro)q(cesses)j(transfer)g(con)o(trol)e(to)g(the)g (library)g(routine.)967 2574 y(1)p eop %%Page: 2 3 2 2 bop 262 307 a Fl(If)14 b(the)i(library)f(w)o(as)g(dev)o(elop)q(ed)h (separately)m(,)g(then)g(one)f(should)h(b)q(ew)o(are)g(of)f(the)h(p)q(ossib-) 262 357 y(ilit)o(y)d(that)j(the)g(library)f(co)q(de)h(ma)o(y)e(receiv)o(e)j (b)o(y)e(mistak)o(e)f(messages)i(send)g(b)o(y)g(the)g(caller)262 407 y(co)q(de,)f(and)g(vice-v)o(ersa.)22 b(T)m(o)14 b(prev)o(en)o(t)i(suc)o (h)f(o)q(ccurrence)j(one)d(migh)o(t)e(use)i(a)g(barrier)g(syn-)262 457 y(c)o(hronization)e(b)q(efore)h(and)f(after)h(the)g(parallel)e(library)h (call.)k(Instead,)d(one)f(can)h(allo)q(cate)262 506 y(a)g(di\013eren)o(t)h (con)o(text)g(to)f(the)h(library)m(,)d(th)o(us)j(prev)o(en)o(ting)g(un)o(w)o (an)o(ted)f(in)o(terference.)21 b(No)o(w,)262 556 y(the)14 b(transfer)h(of)e(con)o(trol)h(to)f(the)i(library)e(need)i(not)f(b)q(e)g (sync)o(hronized.)262 672 y Ff(1.1.2)55 b(F)-5 b(unctional)20 b(decomp)r(osition)e(and)j(mo)r(dular)e(co)r(de)h(dev)n(el-)433 731 y(opmen)n(t)262 807 y Fl(Often,)d(a)f(parallel)f(application)g(is)i(dev)o (elop)q(ed)g(b)o(y)f(in)o(tegrating)g(sev)o(eral)h(distinct)g(func-)262 857 y(tional)f(mo)q(dules,)h(that)h(is)g(eac)o(h)g(dev)o(elop)q(ed)h (separately)m(.)30 b(Eac)o(h)18 b(mo)q(dule)f(is)g(a)h(parallel)262 907 y(program)c(that)j(runs)g(on)f(a)g(dedicated)h(set)h(of)d(pro)q(cesses,) 20 b(and)c(the)h(computation)e(con-)262 957 y(sists)10 b(of)g(phases)h(where) g(mo)q(dules)e(compute)h(separately)m(,)g(in)o(termixed)f(with)h(global)f (phases)262 1006 y(where)h(all)f(pro)q(cesses)k(comm)o(uni)o(cate.)i(It)10 b(is)f(con)o(v)o(enien)o(t)i(to)e(allo)o(w)g(eac)o(h)h(mo)q(dule)e(to)i(use)h (its)262 1056 y(o)o(wn)h(priv)n(ate)h(pro)q(cess)i(n)o(um)o(b)q(ering)d(sc)o (heme,)h(for)g(the)g(in)o(tramo)q(dule)f(computation.)k(This)262 1106 y(is)11 b(ac)o(hiev)o(ed)g(b)o(y)g(using)g(a)g(priv)n(ate)g(mo)q(dule)f (con)o(text)i(for)f(in)o(tramo)q(dule)e(computation,)h(and)262 1156 y(a)j(global)f(con)o(text)j(for)e(in)o(termo)q(dule)g(comm)o(unication.) 262 1272 y Ff(1.1.3)55 b(Collectiv)n(e)16 b(comm)n(unication)262 1349 y Fl(MPI)g(supp)q(orts)i(collectiv)o(e)f(comm)o(unication)c(within)j (dynamically)e(created)k(groups)f(of)262 1399 y(pro)q(cesses.)37 b(Eac)o(h)19 b(suc)o(h)h(group)f(can)h(b)q(e)g(represen)o(ted)i(b)o(y)d(a)g (distinct)h(con)o(text.)35 b(This)262 1448 y(pro)o(vides)18 b(a)g(simple)f(mec)o(hanism)f(to)i(ensure)h(that)g(comm)o(unicati)o(on)c (that)k(p)q(ertains)g(to)262 1498 y(collectiv)o(e)13 b(comm)o(unicati)o(on)d (within)j(one)g(group)g(is)g(not)g(confused)h(with)f(collectiv)o(e)g(com-)262 1548 y(m)o(unication)e(within)i(another)i(group.)262 1664 y Ff(1.1.4)55 b(Ligh)n(t)n(w)n(eigh)n(t)19 b(gang)g(sc)n(heduling)262 1741 y Fl(Consider)14 b(an)f(en)o(vironmen)o(t)g(where)i(pro)q(cesses)h(are)f (m)o(ultith)o(treaded.)i(Con)o(texts)d(can)g(b)q(e)262 1791 y(used)19 b(to)g(pro)o(vide)f(a)g(mec)o(hanism)f(whereb)o(y)i(all)f(pro)q (cesses)j(are)e(time-shared)f(b)q(et)o(w)o(een)262 1840 y(sev)o(eral)d (parallel)e(executions,)j(and)e(can)h(con)o(text)h(switc)o(h)f(from)d(one)j (parallel)f(execution)262 1890 y(to)k(another,)h(in)f(a)g(lo)q(osely)g(sync)o (hronous)h(manner.)31 b(A)19 b(thread)g(is)f(allo)q(cated)g(on)g(eac)o(h)262 1940 y(pro)q(cess)h(to)f(eac)o(h)g(parallel)e(execution,)j(and)f(a)f (di\013eren)o(t)i(con)o(text)g(is)e(used)i(to)e(iden)o(tify)262 1990 y(eac)o(h)h(parallel)e(execution.)29 b(Th)o(us,)19 b(tra\016c)e(from)f (one)i(execution)g(cannot)g(b)q(e)g(confused)262 2040 y(with)f(tra\016c)g (from)f(another)i(execution.)30 b(The)17 b(blo)q(c)o(king)g(and)g(un)o(blo)q (c)o(king)g(of)g(threads)262 2089 y(due)h(to)f(comm)o(unication)e(ev)o(en)o (ts)k(pro)o(vide)e(a)h(\\lazy")e(con)o(text)j(switc)o(hing)e(mec)o(hanism.) 262 2139 y(This)g(can)h(b)q(e)g(extended)i(to)d(the)i(case)f(where)h(the)f (parallel)f(executions)i(are)f(spanning)262 2189 y(distinct)c(pro)q(cess)h (subsets.)20 b(\(MPI)14 b(do)q(es)h(not)f(require)h(m)o(ultithreaded)d(pro)q (cesses.\))262 2326 y Fe(Discussion:)46 b Fd(A)16 b(con)o(text)h(handle)h (migh)o(t)f(b)q(e)g(implemen)o(ted)i(as)d(a)g(p)q(oin)o(ter)i(to)e(a)h (structure)262 2372 y(that)f(consists)h(of)f(con)o(text)h(lab)q(el)h(\(that)e (is)h(carried)g(b)o(y)f(messages)h(sen)o(t)f(within)i(this)f(con)o(text\))262 2417 y(and)10 b(a)h(con)o(text)f(mem)o(b)q(er)h(table,)h(that)e(translates)i (pro)q(cess)f(ranks)g(within)h(a)e(con)o(text)h(to)f(absolute)967 2574 y Fl(2)p eop %%Page: 3 4 3 3 bop 262 307 a Fd(addresses)16 b(or)g(to)g(routing)h(information.)27 b(Of)15 b(course,)i(other)f(implemen)o(tations)j(are)c(p)q(ossible,)262 353 y(including)f(implemen)o(tation)q(s)g(that)e(do)g(not)f(require)i(eac)o (h)f(con)o(text)g(mem)o(b)q(er)g(to)f(store)h(a)g(full)g(list)262 399 y(of)g(the)h(con)o(text)h(mem)o(b)q(ers.)324 444 y(Con)o(texts)k(can)g(b) q(e)g(used)g(only)h(on)f(the)g(pro)q(cess)h(where)e(they)h(w)o(ere)g (created.)31 b(Since)19 b(the)262 490 y(con)o(text)d(carries)g(information)i (on)e(the)f(group)i(of)e(pro)q(cesses)h(that)g(b)q(elong)h(to)f(this)g(con)o (text,)g(a)262 535 y(pro)q(cess)g(can)g(send)g(a)f(message)i(within)g(a)e (con)o(text)h(only)h(to)e(other)h(pro)q(cesses)h(that)e(b)q(elong)j(to)262 581 y(that)13 b(con)o(text.)18 b(Th)o(us,)c(eac)o(h)f(pro)q(cess)h(needs)g (to)g(k)o(eep)f(trac)o(k)h(only)h(of)e(the)g(con)o(texts)h(that)f(where)262 627 y(created)f(at)h(that)f(pro)q(cess;)h(the)g(total)g(n)o(um)o(b)q(er)g(of) g(con)o(texts)g(p)q(er)f(pro)q(cess)i(is)f(lik)o(ely)h(to)f(b)q(e)f(small.) 324 672 y(The)d(only)i(di\013erence)h(I)d(see)g(b)q(et)o(w)o(een)h(this)h (curren)o(t)f(de\014nition)i(of)d(con)o(text,)i(whic)o(h)f(subsumes)262 718 y(the)16 b(group)h(concept,)g(and)g(a)f(pared)h(do)o(wn)f(de\014nition,)j (if)d(that)h(I)e(assume)i(here)f(that)h(pro)q(cess)262 764 y(n)o(um)o(b)q(ering)12 b(is)g(relativ)o(e)g(to)f(the)g(con)o(text,)g(rather) g(then)h(b)q(eing)g(global,)h(th)o(us)e(requiring)i(a)e(con)o(text)262 809 y(mem)o(b)q(er)e(table.)17 b(I)9 b(argue)h(that)f(this)h(is)g(not)f(m)o (uc)o(h)h(added)g(o)o(v)o(erhead,)h(and)f(giv)o(es)g(m)o(uc)o(h)g(additional) 262 855 y(needed)k(functionalit)o(y)m(.)325 917 y Fc(\017)21 b Fd(If)16 b(a)g(new)g(con)o(text)g(is)h(created)f(b)o(y)h(cop)o(ying)h(a)e (previous)i(con)o(text,)f(then)f(one)h(do)q(es)f(not)365 963 y(need)d(a)f(new)f(mem)o(b)q(er)i(table;)g(rather,)f(one)g(needs)h(just)e(a)h (new)g(con)o(text)g(lab)q(el)i(and)f(a)f(new)365 1009 y(p)q(oin)o(ter)17 b(to)e(the)g(same)g(old)h(con)o(text)f(mem)o(b)q(er)h(table.)23 b(This)16 b(holds)h(true,)e(in)h(particular,)365 1054 y(for)d(con)o(texts)h (that)f(include)i(all)g(pro)q(cesses.)325 1117 y Fc(\017)21 b Fd(A)10 b(con)o(text)h(mem)o(b)q(er)g(table)g(mak)o(es)g(sure)g(that)g(a)f (message)h(is)g(sen)o(t)g(only)g(to)g(a)f(pro)q(cess)h(that)365 1162 y(can)16 b(execute)f(in)h(the)f(con)o(text)g(of)f(the)h(message.)23 b(The)15 b(alternativ)o(e)i(mec)o(hanism,)g(whic)o(h)365 1208 y(is)c(c)o(hec)o(king)h(at)e(reception,)i(is)f(less)g(e\016cien)o(t,)g(and)g (requires)h(that)e(eac)o(h)h(con)o(text)g(lab)q(el)h(b)q(e)365 1254 y(system-wide)h(unique.)21 b(This)15 b(requires)g(that,)f(to)g(the)g (least,)g(all)i(pro)q(cesses)e(in)h(a)f(con)o(text)365 1299 y(execute)g(a)f(collectiv)o(e)i(agreemen)o(t)f(algorithm)h(at)e(the)g (creation)h(of)f(this)h(con)o(text.)325 1362 y Fc(\017)21 b Fd(The)c(use)h(of)e(relativ)o(e)j(addressing)g(within)g(eac)o(h)e(con)o(text) h(is)f(needed)h(to)f(supp)q(ort)h(true)365 1407 y(mo)q(dular)e(dev)o(elopmen) o(t)f(of)f(sub)q(computations)j(that)d(execute)g(on)g(a)g(subset)h(of)e(the)h (pro-)365 1453 y(cesses.)26 b(There)16 b(is)g(also)h(a)e(big)i(adv)n(an)o (tage)g(in)g(using)g(the)f(same)g(con)o(text)g(construct)h(for)365 1499 y(collectiv)o(e)f(comm)o(unications)g(as)d(w)o(ell.)262 1802 y Fj(1.2)64 b(Con)n(text)22 b(Op)r(erations)262 1893 y Fl(A)13 b(global)f(con)o(text)j Fi(ALL)e Fl(is)g(prede\014ned.)20 b(All)13 b(pro)q(cesses)j(b)q(elong)e(to)f(this)h(con)o(text)g(when)262 1943 y(computation)i(starts.)30 b(MPI)18 b(do)q(es)h(not)f(sp)q(ecify)g(ho)o (w)f(pro)q(cesses)k(are)d(initially)d(rank)o(ed)262 1992 y(within)k(the)h (con)o(text)g(ALL.)g(It)f(is)h(exp)q(ected)i(that)e(the)g(start-up)g(pro)q (cedure)i(used)f(to)262 2042 y(initiate)c(an)h(MPI)g(program)f(\(at)h (load-time)e(or)i(run-time\))g(will)e(pro)o(vide)j(information)262 2092 y(or)c(con)o(trol)g(on)g(this)g(initial)e(ranking)i(\(e.g.,)f(b)o(y)h (sp)q(ecifying)g(that)h(pro)q(cesses)i(are)d(rank)o(ed)262 2142 y(according)e(to)g(their)g(pid's,)g(or)g(according)g(to)g(the)h(ph)o (ysical)e(addresses)k(of)c(the)i(executing)262 2192 y(pro)q(cessors,)h(or)f (according)g(to)g(a)f(n)o(um)o(b)q(ering)g(sc)o(heme)h(sp)q(eci\014ed)h(at)f (load)f(time\).)262 2341 y Fe(Discussion:)18 b Fd(If)c(w)o(e)g(think)i(of)e (adding)j(new)d(pro)q(cesses)i(at)e(run-time,)i(then)f Fb(ALL)e Fd(con)o(v)o(eys)i(the)262 2391 y(wrong)e(impression,)i(since)f(it)f(is)h (just)f(the)g(initial)j(set)d(of)g(pro)q(cesses.)967 2574 y Fl(3)p eop %%Page: 4 5 4 4 bop 324 407 a Fl(The)14 b(follo)o(wing)d(op)q(erations)k(are)f(a)o(v)n (ailable)d(for)j(creating)g(new)h(con)o(texts.)262 506 y Fi(MPI)p 361 506 15 2 v 17 w(COPY)p 517 506 V 17 w(CONTEXT\(new)o(con)o(text,)g(con)o (text\))324 556 y Fl(Create)d(a)e(new)h(con)o(text)h(that)f(includes)g(all)e (pro)q(cesses)14 b(in)c(the)i(old)e(con)o(text.)17 b(The)12 b(rank)262 606 y(of)f(the)i(pro)q(cesses)i(in)d(the)g(previous)h(con)o(text)g (is)f(preserv)o(ed.)20 b(The)12 b(call)g(m)o(ust)f(b)q(e)i(executed)262 656 y(b)o(y)f(all)f(pro)q(cesses)k(in)d(the)h(old)f(con)o(text.)18 b(It)13 b(is)f(a)g(blo)q(c)o(king)g(call:)k(No)d(call)e(returns)j(un)o(til)e (all)262 706 y(pro)q(cesses)k(ha)o(v)o(e)e(called)f(the)i(function.)j(The)c (parameters)g(are)262 797 y Fi(OUT)h(new)o(con)o(text)k Fl(handle)e(to)g (newly)f(created)j(con)o(text.)28 b(The)17 b(handle)g(should)g(not)365 847 y(b)q(e)e(asso)q(ciated)f(with)g(an)g(ob)r(ject)g(b)q(efore)h(the)f (call.)262 930 y Fi(IN)i(con)o(text)j Fl(handle)14 b(to)g(old)f(con)o(text) 262 1108 y Fe(Discussion:)49 b Fd(I)17 b(considered)j(adding)f(a)e(string)h (parameter,)h(to)e(pro)o(vide)i(a)e(unique)i(iden)o(ti-)262 1154 y(\014er)13 b(to)g(the)g(next)h(con)o(text.)k(But,)13 b(in)h(an)g(en)o(vironmen)o(t)h(where)e(pro)q(cesses)h(are)g(single)h (threaded,)262 1200 y(this)c(is)g(not)g(m)o(uc)o(h)g(help:)17 b(Either)11 b(all)h(pro)q(cesses)g(agree)f(on)f(the)h(order)g(they)g(create)g (new)f(con)o(texts,)262 1245 y(or)j(the)h(application)i(deadlo)q(c)o(ks.)21 b(A)13 b(k)o(ey)h(ma)o(y)f(help)i(in)f(an)g(en)o(vironmen)o(t)h(where)f(pro)q (cesses)g(are)262 1291 y(m)o(ultithreaded,)20 b(to)c(distinguis)q(h)k(call)e (from)e(distinct)j(threads)e(of)g(the)g(same)g(pro)q(cess;)i(but)e(it)262 1337 y(migh)o(t)c(b)q(e)h(simpler)g(to)f(use)h(a)f(m)o(utex)g(algorithm)i(at) e(eac)o(h)g(pro)q(cess.)324 1386 y Fe(Implemen)o(tation)j(note:)24 b Fd(No)16 b(comm)o(unication)j(is)d(needed)i(to)e(create)g(a)g(new)g(con)o (text,)262 1436 y(b)q(ey)o(ond)j(a)f(barrier)i(sync)o(hronization;)k(all)19 b(pro)q(cesses)g(can)g(agree)f(to)h(use)f(the)h(same)f(naming)262 1486 y(sc)o(heme)c(for)g(successiv)o(e)h(copies)g(of)f(the)g(same)g(con)o (text.)20 b(Also,)15 b(no)f(new)g(rank)g(table)h(is)g(needed,)262 1536 y(just)d(a)h(new)g(con)o(text)h(lab)q(el)h(and)e(a)g(new)g(p)q(oin)o (ter)i(to)e(the)g(same)g(old)h(table.)262 1735 y Fi(MPI)p 361 1735 V 17 w(NEW)p 495 1735 V 18 w(CONTEXT\(new)o(con)o(text,)h(con)o(text,)g (k)o(ey)l(,)h(index\))262 1826 y(OUT)f(new)o(con)o(text)k Fl(handle)e(to)g (newly)g(created)i(con)o(text)f(at)f(calling)f(pro)q(cess.)30 b(This)365 1876 y(handle)14 b(should)g(not)g(b)q(e)g(asso)q(ciated)h(with)e (an)h(ob)r(ject)h(b)q(efore)f(the)h(call.)262 1959 y Fi(IN)h(con)o(text)j Fl(handle)14 b(to)g(old)f(con)o(text)262 2042 y Fi(IN)j(k)o(ey)21 b Fl(in)o(teger)262 2125 y Fi(IN)16 b(index)j Fl(in)o(teger)324 2217 y(A)11 b(new)h(con)o(text)g(is)f(created)i(for)e(eac)o(h)g(distinct)h(v) n(alue)f(of)f Fa(key)p Fl(;)h(this)h(con)o(text)g(is)f(shared)262 2267 y(b)o(y)j(all)g(pro)q(cesses)k(that)d(made)f(the)h(call)g(with)f(this)i (k)o(ey)f(v)n(alue.)21 b(Within)13 b(eac)o(h)j(new)g(con-)262 2316 y(text)g(the)h(pro)q(cesses)h(are)f(rank)o(ed)f(according)g(to)g(the)g (order)h(of)e(the)i Fa(index)d Fl(v)n(alues)i(they)262 2366 y(pro)o(vided;)c(in)g(case)i(of)e(ties,)h(pro)q(cesses)j(are)d(rank)o(ed)g (according)g(to)g(their)g(rank)g(in)f(the)h(old)262 2416 y(con)o(text.)967 2574 y(4)p eop %%Page: 5 6 5 5 bop 324 307 a Fl(This)16 b(call)f(is)h(blo)q(c)o(king:)22 b(No)16 b(call)g(returns)i(un)o(til)d(all)g(pro)q(cesses)k(in)d(the)g(old)g (con)o(text)262 357 y(executed)f(the)g(call.)324 407 y(P)o(articular)e(uses)j (of)d(this)h(function)f(are:)324 457 y(\(i\))19 b(Reordering)h(pro)q(cesses:) 33 b(All)19 b(pro)q(cesses)k(pro)o(vide)d(the)g(same)f Fa(key)g Fl(v)n(alue,)i(and)262 506 y(pro)o(vide)13 b(their)i(index)e(in)h(the)g(new)h (order.)324 556 y(\(ii\))i(Splitting)g(a)g(con)o(text)i(in)o(to)e(sub)q(con)o (texts,)k(while)c(preserving)i(the)g(old)e(relativ)o(e)262 606 y(order)11 b(among)e(pro)q(cesses:)19 b(All)9 b(pro)q(cesses)14 b(pro)o(vide)c(the)i(same)d Fa(index)h Fl(v)n(alue,)g(and)h(pro)o(vide)262 656 y(a)i(k)o(ey)h(iden)o(tifying)e(their)j(new)f(sub)q(con)o(text.)262 756 y Fi(MPI)p 361 756 15 2 v 17 w(RANK\(rank,)i(con)o(text\))262 836 y(OUT)f(rank)21 b Fl(in)o(teger)262 914 y Fi(IN)16 b(con)o(text)j Fl(con)o(text)c(handle)324 994 y(Return)f(the)h(rank)e(of)h(the)g(calling)f (pro)q(cess)i(within)f(the)g(sp)q(eci\014ed)h(con)o(text.)262 1094 y Fi(MPI)p 361 1094 V 17 w(SIZE\(size,)g(con)o(text\))262 1174 y(OUT)g(size)20 b Fl(in)o(teger)262 1252 y Fi(IN)c(con)o(text)j Fl(con)o(text)c(handle)324 1333 y(Return)f(the)h(n)o(um)o(b)q(er)e(of)g(pro)q (cesses)k(that)d(b)q(elong)f(to)h(the)g(sp)q(eci\014ed)i(con)o(text.)262 1447 y Ff(1.2.1)55 b(Usage)18 b(note)262 1523 y Fl(Use)c(of)f(con)o(texts)i (for)e(libraries:)k(Eac)o(h)d(library)f(ma)o(y)f(pro)o(vide)h(an)h (initialization)c(routine)262 1573 y(that)k(is)g(to)h(b)q(e)g(called)f(b)o(y) g(all)g(pro)q(cesses,)j(and)d(that)g(generate)i(a)e(con)o(text)i(for)e(the)h (use)g(of)262 1623 y(that)e(library)m(.)324 1673 y(Use)j(of)g(con)o(texts)g (for)g(functional)f(decomp)q(osition:)20 b(A)c(harness)h(program,)d(running) 262 1723 y(in)f(the)i(con)o(text)g Fa(ALL)f Fl(generates)i(a)e(sub)q(con)o (text)h(for)f(eac)o(h)h(mo)q(dule)e(and)h(then)h(starts)g(the)262 1773 y(submo)q(dule)d(within)i(the)g(corresp)q(onding)h(con)o(text.)324 1822 y(Use)g(of)f(con)o(texts)i(for)e(collectiv)o(e)g(comm)o(unication:)i(A)e (con)o(text)i(is)e(created)i(for)e(eac)o(h)262 1872 y(group)f(of)h(pro)q (cesses)i(where)f(collectiv)o(e)f(comm)o(unication)c(is)k(to)g(o)q(ccur.)324 1922 y(Use)i(of)f(con)o(texts)i(for)e(con)o(text-switc)o(hing)h(among)d(sev)o (eral)k(parallel)d(executions:)23 b(A)262 1972 y(pream)o(ble)15 b(co)q(de)j(is)f(used)g(to)g(generate)h(a)f(di\013eren)o(t)h(con)o(text)f (for)g(eac)o(h)g(execution;)i(this)262 2022 y(pream)o(ble)8 b(co)q(de)j(needs)g(to)e(use)i(a)e(m)o(utual)f(exclusion)i(proto)q(col)f(to)h (mak)o(e)e(sure)j(eac)o(h)f(thread)262 2071 y(claims)i(the)i(righ)o(t)g(con)o (text.)262 2208 y Fe(Discussion:)30 b Fd(If)10 b(pro)q(cess)h(handles)h(are)e (made)h(explicit)i(in)e(MPI,)f(then)h(an)f(additional)k(function)262 2254 y(needed)d(is)g Fe(MPI)p 515 2254 14 2 v 15 w(PR)o(OCESS\(pro)q(cess,)i (con)o(text,)g(rank\))p Fd(,)e(whic)o(h)g(returns)g(a)f(handle)i(to)e(the)262 2300 y(pro)q(cess)j(iden)o(ti\014ed)j(b)o(y)d(the)g Fb(rank)f Fd(and)h Fb(context)d Fd(parameters.)324 2350 y(A)e(p)q(ossible)j(addition)g (is)f(a)f(function)h(of)e(the)h(form)f Fe(MPI)p 1136 2350 V 16 w(CREA)l(TE)p 1335 2350 V 17 w(CONTEXT\(new)o(con)o(text,)262 2399 y(list)p 325 2399 V 15 w(of)p 376 2399 V 15 w(pro)q(cess)p 531 2399 V 17 w(handles\))j Fd(whic)o(h)i(creates)f(a)f(new)h(con)o(text)g (out)g(of)g(an)g(explicit)i(list)f(of)f(mem-)262 2449 y(b)q(ers)k(\(and)g (rank)h(them)f(in)h(their)g(order)f(of)g(o)q(ccurrence)h(in)f(the)g(list\).) 27 b(This,)17 b(coupled)h(with)e(a)967 2574 y Fl(5)p eop %%Page: 6 7 6 6 bop 262 307 a Fd(mec)o(hanism)13 b(for)e(requiring)j(the)e(spa)o(wning)h (of)e(new)h(pro)q(cesses)g(to)g(the)g(computation,)h(will)g(allo)o(w)262 357 y(to)g(create)g(a)g(new)h(all)g(inclusiv)o(e)i(con)o(text)e(that)g (includes)h(the)f(additional)i(pro)q(cesses.)j(Ho)o(w)o(ev)o(er,)262 407 y(I)12 b(opp)q(ose)j(the)e(idea)h(of)f(requiring)j(dynamic)f(pro)q(cess)f (creation)g(as)g(part)f(of)g(MPI.)g(Man)o(y)h(imple-)262 457 y(men)o(ters)g(w)o(an)o(t)g(to)g(run)g(MPI)g(in)h(an)f(en)o(vironmen)o(t)i (where)e(pro)q(cesses)i(are)e(statically)i(allo)q(cated)262 506 y(at)c(load-time.)967 2574 y Fl(6)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:50:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00341; Sun, 21 Mar 93 14:50:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24784; Sun, 21 Mar 93 14:50:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:50:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24770; Sun, 21 Mar 93 14:50:17 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03760; Sun, 21 Mar 93 13:44:53 CST Date: Sun, 21 Mar 93 13:44:53 CST From: Tony Skjellum Message-Id: <9303211944.AA03760@Aurora.CS.MsState.Edu> To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu Subject: Re: Proposal I - LaTeX Lyndon, what about your extensive proposal (that I just scribbled about endlessly). Are you shooting that? Anyway, tell me specifically, and call whatever new thing Proposal VI, not proposal II' or II. - Tony From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:52:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00353; Sun, 21 Mar 93 14:52:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24896; Sun, 21 Mar 93 14:52:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:52:00 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24888; Sun, 21 Mar 93 14:51:55 -0500 Date: Sun, 21 Mar 93 19:51:46 GMT Message-Id: <13105.9303211951@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal II' - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is the (all new) Proposal II', credits to myself, which was written over the last two days, intensely. PostScript follws. I have now received comments from Tony on Proposal I++ (now defunct, I truly hope). I will propogate these into this the LaTeX of Proposal II' where appropriate in the style of my reply to Proposal V, shortly. Best Wishes Lyndon ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal II$^\prime$\\MPI Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal II'" % Written by Lyndon J. Clarke % March 1993 % \newcommand{\LabelNote}[1]{\label{ii:note:#1}} \newcommand{\SeeReferNote}[1]{{\bf See~Note~\ref{ii:note:#1}.}} \newcommand{\LabelSection}[1]{\label{ii:sect:#1}} \newcommand{\ReferSection}[1]{{\bf Section~\ref{ii:sect:#1}.}} \chapter{Proposal II$^\prime$} {\it Editorial Note: This is not Proposal II as identified at the post-meet lunch after the February meeting in Dallas. Rik~Littlefied came on board the proposal writing process, and decided to write a different proposal which appears as Proposal V. There is no longer a proposal which contains static contexts as discussed at the post-meet lunch.} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction} This chapter proposes that communication contexts and process groupings within MPI appear as related concepts. In particular process groupings appear as ``frames'' which are used in the creation of communication contexts. Communications contexts retain a reference to, and inherit properties of, their process grouping frames. This reflects the observation that an invocation of a module in a parallel program typically operates within one or more groups of processes, and as such any communication contexts associated with invocations of modules also bind certain semantics of process groupings. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of MPI, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language for only purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The notes assumes knowledge of the proposal text and are therefore best examined after familiarisation with that text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the text of the proposal. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal} \LabelSection{proposal} This section presents the detailed proposal, discussing in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes} \LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and MPI does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{process:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of MPI services. MPI provides a procedure which returns the own descriptor of the calling process. For example, {\tt pd = mpi\_own\_pd()}. \subsubsection*{Process Creation and Destruction} This proposal makes no statements regarding creation and destruction of processes. \SeeReferNote{dynamic:processes} \subsubsection*{Descriptor Transmission} The value of a process descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. MPI provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_PD\_TYPE}. MPI provides a process descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a process descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_pd(pd)}. The user is not allowed to invalidate the process descriptor of the calling process. \SeeReferNote{process:coherency} \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding processor descriptor attributes. \subsection{Process Groupings} \LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to?) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. \SeeReferNote{dynamic:groups} The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to the descriptor of the $N$ processes composing the grouping. The structure of a process grouping is defined by a process grouping topology and this proposal makes no further statements regarding such structures. \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{group:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the process grouping until the termination of MPI services. \SeeReferNote{group:blobs} MPI provides a procedure which returns the descriptor of the home group of the calling process. For example, {\tt gd = mpi\_home\_gd()}. \SeeReferNote{own:group} \subsubsection*{Group Creation and Deletion} MPI provides a procedure which allows users to dynamically create one or more groups which are subsets of existing groups. For example, {\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} partition an existing group {\tt gda} into one or more distinct subsets. This procedure is called by and synchronises all members of {\tt gda}. MPI provides a procedure which allows users to dynamically create one group by explicit definition of its membership as a list of process descriptors. For example, {\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. MPI provides a procedure which allows users to delete a created group. This procedure accepts the descriptor of a group which was created by the calling process and destroys the identified group. For example, {\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. MPI provides additional procedure which allow users to construct process groupings which have a process grouping topology. \subsubsection*{Descriptor Transmission} The value of a group descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. MPI provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_GD\_TYPE}. MPI provides a group descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a group descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_gd(gd)}. The user is not allowed to invalidate the own group descriptor of the process or the group descriptor of any group created by the calling process. \SeeReferNote{group:coherency} \subsubsection*{Descriptor Attributes} MPI provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identifier group. For example, {\tt rank = mpi\_group\_rank(gd)}. MPI provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, {\tt size = mpi\_group\_size(gd)}. MPI provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, {\tt pd = mpi\_group\_pd(gd, rank)}. \SeeReferNote{pd:to:rank} MPI provides additional procedures which allow users to determine the process grouping topology attributes. \subsection{Communication Contexts} \LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping is used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{context:identifiers} \SeeReferNote{descriptor:cache} The creation of MPI process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains its own context until MPI process grouping deletion. MPI provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, {\tt cd = mpi\_own\_cd(gd)}. \SeeReferNote{own:context} \subsubsection*{Context Creation and Deletion} MPI provides a procedure which allows users to dynamically create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, {\tt cd = mpi\_context\_create(gd)}. This procedure is called by and synchronises all members of {\tt gd}. \SeeReferNote{dynamic:contexts} MPI provides a procedure which allows users to destroy created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes that context identifier. For example, {\tt mpi\_context\_delete(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. \subsubsection*{Descriptor Transmission} The value of a context descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_CD\_TYPE}. MPI provides a context descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a context descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_cd(cd)}. The user is not allowed to invalidate the own context descriptor of a group or the context descriptor of any context created by the calling process. \SeeReferNote{context:coherency} \subsubsection*{Descriptor Attributes} MPI provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, {\tt gd = mpi\_context\_gd(cd)}. \subsection{Point-to-Point Communication} \LabelSection{point2point} This proposal recommends three forms for MPI point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a Send operation of form X cannot match with a receive operation of form Y, which requires that the form is embedded into the message envelope. The three forms are described followed by considerations of uniform integration of these forms in the point-to-point communication section of MPI. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. \SeeReferNote{null:context} Message selection and addressing are expressed by {\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. Send supplies the {\tt pd} of the receiver. Receive supplies the {\tt pd} of the sender. Receive can wildcard on {\tt pd} by supplying the value of a named constant process descirptor, e.g. {\tt MPI\_PD\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. \SeeReferNote{closed:context} Message selection and addressing are expressed by {\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. Send supplies the {\tt cd} of the receiver and sender, and the {\tt rank} of the receiver. Receive supplies the {\tt cd} of the sender and receiver, and the rank of the sender. The {\tt (cd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt cd}. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. \SeeReferNote{open:context} Message selection and addressing are expressed by {\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. Send supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. Receive supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The {\tt (rcd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt lcd}. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD}, in which case Receive {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point section of MPI by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with the form. This is at the expense of multiplying the number of Send and Receive procedures by a factor of three, and some difficulty in details of the current point-to-point working document which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options each have advantages and disadvantages. \paragraph*{Option i:} The framework of the open context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii:} The framework of the closed context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as {\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the {\tt (lcd, rcd)} pair at the expense of introducing some trickery. We can define a mechanism which ``globs'' together these two fields onto in an identifier which Send and Receive can distinguish from a context identifier and treat as the shorthand notation. Then we should also define a mechanism by which a receiver which has completed a Receive with wildcard on {\tt rcd} is able to determine the valid context descriptor of the sender. This is a inconvenient. \paragraph*{Option iii:} The framework of the null context form is adopted and extended. The null context form is expressed as {\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is clumsy. \subsection{Collective Communication} \LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context and frame in which they are to operate. MPI does plan to describe symmetric collective communication operations. It is not possible to determine whether the proposal is sufficient to allow implementation of the collective communication section of MPI in terms of the point-to-point section of MPI without loss of generality, since the collective operations are not yet defined. Assymetric collective communication operations, especially those in which the sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a ``glob'' form) which identify the contexts and frames in which they are to operate. MPI does not plan to describe assymetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model and comprise communicative funtionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of MPI. % % END "Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion} \LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents arguments for and against the MIMD model annex, and to some extent explores the consquences of these arguments for MPI in a wider sense. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model, in which process execute in a stricly SIMD fashion. The material describing processes in \ReferSection{processes} is simplified: \begin{itemize} \item processes have identical instruction blocks and different data blocks; \item process descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query and group descriptor attribute query is capable of providing access to all process descriptors); \item dynamic process models are not considered. \end{itemize} The material describing process groupings in \ReferSection{groupings} is simplified: \begin{itemize} \item group descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query capable of providing access to all group descriptors.); \item the own process grouping explicitly becomes a group containing all processes. \end{itemize} The material describing communication contexts in \ReferSection{contexts} is simplified: \begin{itemize} \item context descriptor transmission and registry become unnecessary. \end{itemize} The material describing point-to-point communication in \ReferSection{point2point} is simplified: \begin{itemize} \item the open context form becomes redundant; \item uniform integration ``Option i'' is deleted, and ``Option ii'' loses ``globs'' thus becoming simple enough that ``Option iii'' need not be further considered. \end{itemize} The material describing collective communication in \ReferSection{collective} is simplified: \begin{itemize} \item there is no possibility of collective communication operations spanning more than context. \end{itemize} \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as MPI are not suited to fine grained function driven or data driven programming. The annex is intended to provide expresive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in EXPRESS and PARMACS, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or vice versa!). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computating. The debate is whether or not message passing approaches such as MPI (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD parallel prgrams. The positive argument is that communications between SPMD modules requires high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development and standardisation procedures for distributed operating systems. The latter argument is comparable with the argument for MPI versus parallel compilation. \subsection{Notes} \LabelSection{notes} \begin{enumerate} \item\LabelNote{integer:descriptors} Descriptors are a plentiful but bounded resource. They are expressed as integers for two reasons. Firstly, this expression facilitates uniform binding to ANSI~C and Fortran~77. Secondly, there is a later convenience in ensuring that process descriptors in particular are expressed in integers, and I made all the descriptors are integers for consistency. In practice the descriptor could be an index into a table of objects description structures or pointers to such structures. \item\LabelNote{descriptor:cache} It is possible to allow the user to save user defined attributes in descriptors, as Littlefield has suggested. Such attributes should not be communicated in either descriptor transmission or the descriptor registry. \item\LabelNote{process:identifiers} The process descriptor is not a global unique process identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional process information in implementations. For example, the process identifier of a receiver may not be sufficient to route a message to the receiver, and this information can be referenced from the process descriptor. \item\LabelNote{group:identifiers} The group descriptor is not a global unique group identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependeent, and enables more efficient access to additional group information in implementations. For example, the size of the group and the map from member ranks to process descriptors can be referenced from the group descriptor. \item\LabelNote{context:identifiers} The context descriptor is not a global unique context identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional context information in implementations. For example, the group descriptor of the frame can be referenced from the context descriptor. \item\LabelNote{dynamic:processes} The proposal does not prevent a process model which allows dynamic creation and termination of processes however it does not favour an asynchronous process creation model in which singleton processes are created and terminated in an unstructured fashion. The proposal does favour a model in which blobs of processes are created (and terminated) in a concerted fashion, and in which each group so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying worloads. The author suggests that concerted blob expand and contract operations are most suitable for this purpose than asynchronous spawn/kill operations. \item\LabelNote{registry:check} There should probably also be a registry operation which checks whether a name has been registered without the possibility of blocking the calling process indefinitely. The same registry could be used for process, group and context descriptors. \item\LabelNote{process:coherency} In the static process model it is assumed that processes are created in concert, and that termination of one process implies termination of all processes. In this case there is no coherency problem associated with processes and process descriptors. In a dynamic process model there is a coherency problem which processes and process descriptors since a process can retain the descriptor of a process which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{dynamic:groups} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The author suggests concerted group expand and contract operations are more generally useful than asynchronous join/leave operations. \item\LabelNote{group:blobs} MPI has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is intended to be a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and different blobs have different own process groupings. An SPMD program is a single blob. A host-node program composes two blobs, the node blob and the host blob (which is a singleton). There is a sense in which a blob has a locally SPMD view. \item\LabelNote{own:group} This procedure looks like a process descriptor attribute query. \item\LabelNote{group:coherency} Dynamic processes introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Dynamic processes potentially introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Group transmission introduces a group and group descriptor coherenncy problem since a process can retain a group descriptor of a group which has been identified group. The proposal expects the user to ensure coherent usage. \item\LabelNote{pd:to:rank} The proposal did not include a function to convert a {\tt (gd, pd)} pair into a rank. It is suggested that this inverse map is allowed to be arbitrarily slow. \item\LabelNote{dynamic:contexts} Marc Snir has described a method by which global unique group identifiers can be generated without use of shared global data. The description of context creation anticipates a similar method for global unique context identifier generation. However the synchronisation requirement of this method makes it unnecessary for context creation. Synchronisation at creation of context within a process grouping frame implies that all processes within the frame perform the same context creations in the same sequence. Therefore the combination of global unique frame identifier and context creation sequence number provides a global unique context identifier without communication. \item\LabelNote{own:context} This procedure looks like a group descriptor attribute query. \item\LabelNote{context:coherency} The retention of a reference to a group descriptor by a context introduces the potential for a context and context descriptor coherency problem, as a context could contain a reference to the group descriptor of a group which has been deleted. This could be circumvented by demanding that all such contexts are deleted before a group is deleted. Context descriptor transmission introduces a coherency problem since a process can retain the descriptor of a context which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{null:context} This is somewhat like and PARMACS and PVM~3. It is general purpose, but not particularly expressive. It does not provide facilities for writers of parallel libraries. \item\LabelNote{open:context} This is somewhat like ZIPCODE and VENUS. It is expressive in certain SPMD programs where noncommunicative data driven parallel computations can be performed concurrently. It provides facilities for writers of SPMD like parallel libraries. \item\LabelNote{closed:context} This is somewhat like CHIMP and PVM~2. It is expressive in certain MIMD programs where communicative data driven parallel computations can be performed concurrently. It provides facilities for MSPMD like parallel libraries. \end{enumerate} % % END "Discussion & Notes" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Conclusion" % \section{Conclusion} This chapter has presented and discussed a proposal for communication contexts within MPI. In the proposal process groupings appeared as frames for the creation of communication contexts, and communication contexts retained certain properties of the frames used in their creation. The author strongly recommends this proposal to the committee. % % END "Conclusion" %----------------------------------------------------------------------% % % END "Proposal II'" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:53:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00376; Sun, 21 Mar 93 14:53:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24914; Sun, 21 Mar 93 14:53:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:52:59 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24905; Sun, 21 Mar 93 14:52:46 -0500 Date: Sun, 21 Mar 93 19:52:39 GMT Message-Id: <13111.9303211952@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal II' - PostScript To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is the PostScript of the (all new) Proposal II'. LateX preceeded. Best Wishes Lyndon ---------------------------------------------------------------------- %!PS-Adobe-2.0 %%Creator: dvips 5.495 Copyright 1986, 1992 Radical Eye Software %%Title: context-ii.dvi %%CreationDate: Sun Mar 21 18:35:09 1993 %%Pages: 15 %%PageOrder: Ascend %%BoundingBox: 0 0 596 842 %%EndComments %DVIPSCommandLine: dvips context-ii %DVIPSSource: TeX output 1993.03.21:1832 %%BeginProcSet: tex.pro %! /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N} B /TR{translate}N /isls false N /vsize 11 72 mul N /@rigin{isls{[0 1 -1 0 0 0] concat}if 72 Resolution div 72 VResolution div neg scale isls{0 Resolution vsize 72 div mul TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{ CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N} B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook} if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255 {IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V{}B /RV statusdict begin /product where{ pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{ gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M} B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{ p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 39158280 55380996 1000 300 300 (/a/obsidian/disk/home/u36/lyndon/mpi/context-ii.dvi) @start /Fa 1 16 df<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF80FF003C010 107E9115>15 D E /Fb 1 79 df<07E01FC000E006000170040001700400013804000138040002 1C0800021C0800020E0800020E0800040710000407100004039000040390000801E0000801E000 0800E0000800E00018004000FE0040001A147F931A>78 D E /Fc 3 111 df<01FC00FF80001C001C00002E001800002E001000002E001000002700100000470020000043 002000004380200000438020000081C040000081C040000081C040000080E040000100E0800001 007080000100708000010070800002003900000200390000020039000002001D000004001E0000 04000E000004000E00000C000E00001C00040000FF80040000211C7E9B21>78 D<00FFFFE000F001C001C003800180070001000E0001001E0002001C0002003800020070000000 E0000001C0000003800000070000000F0000001E0000001C0000003800000070020000E0040001 C0040003800400070008000F0008000E0018001C003000380070007001E000FFFFE0001B1C7E9B 1C>90 D<381F004E61804681C04701C08F01C08E01C00E01C00E01C01C03801C03801C03801C07 00380710380710380E10380E2070064030038014127E9119>110 D E /Fd 44 123 df<00E001E0038007000E001C001C0038003800700070007000E000E000E000E000E000 E000E000E000E000700070007000380038001C001C000E000700038001E000E00B217A9C16>40 DI<387C7E7E 3E0E1E1C78F060070B798416>44 D<7FFF00FFFF80FFFF80000000000000000000000000000000 FFFF80FFFF807FFF00110B7E9116>61 D<00E00001F00001F00001B00001B00003B80003B80003 B800031800071C00071C00071C00071C00071C000E0E000E0E000FFE000FFE001FFF001C07001C 07001C07007F1FC0FF1FE07F1FC013197F9816>65 D<01F18007FB800FFF801F0F803C07803803 80700380700380F00000E00000E00000E00000E00000E00000E00000E00000F000007003807003 803803803C07001F0F000FFE0007FC0001F00011197E9816>67 D<7FF800FFFE007FFF001C0F00 1C07801C03C01C01C01C01C01C01E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E0 1C01C01C01C01C03C01C07801C0F807FFF00FFFE007FF8001319809816>I<7FFFC0FFFFC07FFF C01C01C01C01C01C01C01C01C01C00001C00001C1C001C1C001FFC001FFC001FFC001C1C001C1C 001C00001C00E01C00E01C00E01C00E01C00E07FFFE0FFFFE07FFFE013197F9816>I<03E30007 FF000FFF001E1F003C0F00380700700700700700F00000E00000E00000E00000E00000E03F80E0 7FC0E03F80F00700700700700700380F003C0F001E1F000FFF0007F70003E70012197E9816>71 D73 D<7F0FE0FF8FF07F0FE01C07801C0F001C0E001C 1C001C3C001C78001CF0001CE0001DF0001FF0001FF8001F38001E1C001C1C001C0E001C0E001C 07001C07001C03807F07E0FF8FF07F07E01419809816>75 DII<7E1FC0FF3FE0 7F1FC01D07001D87001D87001D87001DC7001DC7001CC7001CC7001CE7001CE7001CE7001C6700 1C67001C77001C77001C37001C37001C37001C17007F1F00FF9F007F0F0013197F9816>I<7FF8 00FFFE007FFF001C0F801C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE 001FF8001C00001C00001C00001C00001C00001C00001C00007F0000FF80007F000012197F9816 >80 D<7FE000FFF8007FFC001C1E001C0F001C07001C07001C07001C07001C0F001C1E001FFC00 1FF8001FFC001C1C001C0E001C0E001C0E001C0E001C0E201C0E701C0E707F07E0FF87E07F03C0 14197F9816>82 D<7FFFE0FFFFE0FFFFE0E0E0E0E0E0E0E0E0E0E0E0E000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00007FC000F FE0007FC0013197F9816>84 D<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C00E03800E03800707 0007FF0003FE0000F8001519809816>I87 D89 D<1FE0003FF0007FF800783C00 300E00000E00000E0003FE001FFE003E0E00700E00E00E00E00E00E00E00783E007FFFE03FE7E0 0F83E013127E9116>97 D<7E0000FE00007E00000E00000E00000E00000E00000E3E000EFF000F FF800F83C00F00E00E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00F FF800EFF00063C001419809816>I<03F80FFC1FFE3C1E780C7000E000E000E000E000E000F000 700778073E0E1FFC0FF803F010127D9116>I<003F00007F00003F000007000007000007000007 0003C7000FF7001FFF003C1F00780F00700700E00700E00700E00700E00700E00700E00700700F 00700F003C1F001FFFE00FE7F007C7E014197F9816>I<03E00FF81FFC3C1E780E7007E007FFFF FFFFFFFFE000E000700778073C0F1FFE0FFC03F010127D9116>I<001F00007F8000FF8001E780 01C30001C00001C0007FFF00FFFF00FFFF0001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C00001C0003FFE007FFF003FFE0011197F9816>I<03E3C007F7E00FFF E01C1CC0380E00380E00380E00380E00380E001C1C000FF8001FF0001BE0003800001800001FFC 001FFF003FFF807803C0E000E0E000E0E000E0E000E07001C07C07C03FFF800FFE0003F800131C 7F9116>I<7E0000FE00007E00000E00000E00000E00000E00000E3C000EFE000FFF000F87800F 03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87F C7F01519809816>I<018003C003C0018000000000000000007FC07FC07FC001C001C001C001C0 01C001C001C001C001C001C001C001C07FFFFFFF7FFF101A7D9916>I<7E0000FE00007E00000E 00000E00000E00000E00000E7FE00E7FE00E7FE00E0F000E1E000E3C000E78000EF0000FF0000F F8000FBC000F1E000E0E000E07000E07807F87F0FFCFF07F87F01419809816>107 DII<7E3C00FEFE007FFF000F8780 0F03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F8 7FC7F01512809116>I<03E0000FF8001FFC003C1E00780F00700700E00380E00380E00380E003 80E00380F00780700700780F003C1E001FFC000FF80003E00011127E9116>I<7E3E00FEFF007F FF800F83C00F00E00E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00F FF800EFF000E3C000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000141B8091 16>I114 D<0FEC3FFC7FFCF03CE01CE01C 70007F801FF007F8003C600EE00EF00EF81EFFFCFFF8C7E00F127D9116>I<0300000700000700 000700000700007FFF00FFFF00FFFF000700000700000700000700000700000700000700000701 0007038007038007038007870003FE0001FC0000F80011177F9616>I<7E1F80FE3F807E1F800E 03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E0F800FFFF007 FBF803E3F01512809116>I<7F1FC0FF1FE07F1FC01C07001E0F000E0E000E0E000E0E00071C00 071C00071C00071C0003B80003B80003B80001F00001F00000E00013127F9116>II<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001 F00000E00001E00001F00003B800073C00071C000E0E007F1FC0FF3FE07F1FC013127F9116>I< 7F1FC0FF9FE07F1FC01C07000E07000E0E000E0E00070E00071C00071C00039C00039C00039800 01B80001B80000F00000F00000F00000E00000E00000E00001C00079C0007BC0007F80003F0000 3C0000131B7F9116>I<3FFFC07FFFC07FFFC0700780700F00701E00003C0000780001F00003E0 000780000F00001E01C03C01C07801C0FFFFC0FFFFC0FFFFC012127F9116>I E /Fe 28 121 df45 D<387CFEFEFE7C3807077C8610>I<00 180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC07007E07C07 F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C0000F80001F 00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07FFFF0FFFF F0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F81F03F81F 03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC3C00FE7E 00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C>I<0000E0 0001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187E00307E0 0707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007E00007E0 0007E00007E00007E000FFFE00FFFE17207E9F1C>I<1000201E01E01FFFC01FFF801FFF001FFE 001FF8001BC00018000018000018000018000019FC001FFF001E0FC01807E01803E00003F00003 F00003F80003F83803F87C03F8FE03F8FE03F8FC03F0FC03F07007E03007C01C1F800FFF0003F8 0015207D9F1C>I<0003FE0080001FFF818000FF01E38001F8003F8003E0001F8007C0000F800F 800007801F800007803F000003803F000003807F000001807E000001807E00000180FE00000000 FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE000000007E000000 007E000001807F000001803F000001803F000003801F800003000F8000030007C000060003F000 0C0001F800380000FF00F000001FFFC0000003FE000021227DA128>67 DI<0003FE0040001FFFC0C0007F00F1C001F8003FC003F0000FC007C0 0007C00FC00003C01F800003C03F000001C03F000001C07F000000C07E000000C07E000000C0FE 00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE000FFFFC 7E000FFFFC7F00001FC07F00001FC03F00001FC03F00001FC01F80001FC00FC0001FC007E0001F C003F0001FC001FC003FC0007F80E7C0001FFFC3C00003FF00C026227DA12C>71 D78 D80 D<07FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F000 3FF003FDF01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18 167E951B>97 D<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC00 00FC0000FC0000FC00007C00007E00007E00003E00301F00600FC0E007FF8000FE0014167E9519 >99 D<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000 FC0000FC00007C00007C00007E00003E00181F00300FC07003FFC000FF0015167E951A>101 D<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03C03E07C01F0F 801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC70003EF0001EF000 1EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C>103 D<1C003F007F007F007F003F001C000000000000000000000000000000FF00FF001F001F001F00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B247EA310 >105 D108 DII<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC 007EFC007EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE0017167E951C>I< FF0FE000FF3FF8001FF07C001F803E001F001F001F001F801F001F801F000FC01F000FC01F000F C01F000FC01F000FC01F000FC01F000FC01F000FC01F001F801F001F801F803F001FC03E001FE0 FC001F3FF8001F0FC0001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F 000000FFE00000FFE000001A207E951F>I114 D<0FF3003FFF00781F00600700E00300E00300F00300FC0000 7FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00 C7F00011167E9516>I<0180000180000180000180000380000380000780000780000F80003F80 00FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>III120 D E /Ff 43 121 df<78FCFCFCFC7806067D850D>46 D<03F8000F1E001C07003C07803803807803C07803C07803C0F803E0F803E0F803E0F803E0F803 E0F803E0F803E0F803E0F803E0F803E0F803E0F803E07803C07803C03803803C07801C07000F1E 0003F800131B7E9A18>48 D<00600001E0000FE000FFE000F3E00003E00003E00003E00003E000 03E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000 03E00003E00003E0007FFF807FFF80111B7D9A18>I<07F8001FFE00383F80780FC0FC07C0FC07 E0FC03E0FC03E07803E00007E00007C00007C0000F80001F00001E0000380000700000E0000180 600300600600600800E01FFFC03FFFC07FFFC0FFFFC0FFFFC0131B7E9A18>I<03F8001FFE003C 1F003C0F807C07C07E07C07C07C03807C0000F80000F80001E00003C0003F800001E00000F8000 07C00007C00007E03007E07807E0FC07E0FC07E0FC07C0780F80781F001FFE0007F800131B7E9A 18>I<000180000380000780000F80001F80003F80006F8000CF80008F80018F80030F80060F80 0C0F80180F80300F80600F80C00F80FFFFF8FFFFF8000F80000F80000F80000F80000F80000F80 01FFF801FFF8151B7F9A18>I<1801801FFF001FFE001FFC001FF8001FC0001800001800001800 0018000019F8001E0E00180F801007800007C00007E00007E00007E07807E0F807E0F807E0F807 C0F007C0600F80381F001FFE0007F000131B7E9A18>I<007E0003FF000781800F03C01E07C03C 07C03C0380780000780000F80000F8F800FB0E00FA0780FC0380FC03C0F803E0F803E0F803E0F8 03E07803E07803E07803C03C03C03C07801E0F0007FE0003F800131B7E9A18>I<6000007FFFE0 7FFFE07FFFC07FFF807FFF80E00300C00600C00C00C0180000300000300000600000E00000E000 01E00001C00003C00003C00003C00003C00007C00007C00007C00007C00007C00007C000038000 131C7D9B18>I<03F8000FFE001E0F803807803803C07803C07803C07E03C07F83807FC7003FFE 001FFC000FFE0007FF801DFF80387FC0781FE0F007E0F003E0F001E0F001E0F001E07801C07803 803E07801FFE0003F800131B7E9A18>I<03F8000FFE001E0F003C07807807807803C0F803C0F8 03C0F803E0F803E0F803E0F803E07807E03807E03C0BE00E1BE003E3E00003E00003C00003C038 07C07C07807C0700780F00383C001FF8000FE000131B7E9A18>I<78FCFCFCFC78000000000000 78FCFCFCFC7806127D910D>I<00038000000380000007C0000007C0000007C000000FE000000F E000001FF000001BF000001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000 C07E0001803F0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0 FFC07FFEFFC07FFE1F1C7E9B24>65 D<001FE02000FFF8E003F80FE007C003E00F8001E01F0000 E03E0000E03E0000607E0000607C000060FC000000FC000000FC000000FC000000FC000000FC00 0000FC000000FC0000007C0000607E0000603E0000603E0000C01F0000C00F80018007C0030003 F80E0000FFFC00001FE0001B1C7D9B22>67 DI70 D<000FF008007FFE3801FC07F807E001 F80F8000781F0000783F0000383E0000387E0000187C000018FC000000FC000000FC000000FC00 0000FC000000FC000000FC007FFFFC007FFF7C0001F87E0001F83E0001F83F0001F81F0001F80F 8001F807E001F801FC07F8007FFE78000FF818201C7D9B26>I73 D77 DI<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E0003E07E0003F07C 0001F07C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8 7C0001F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E0001F07C00003FE0 001D1C7D9B24>II<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE00 00FFE0007FFE003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000 E0E000E0F001C0FC03C0EFFF0083FC00141C7D9B1B>83 D<7FFFFFE07FFFFFE0781F81E0701F80 E0601F8060E01F8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F 8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F800000 1F8000001F8000001F800007FFFE0007FFFE001C1C7E9B21>II<0FF8001C1E003E0F803E07 803E07C01C07C00007C0007FC007E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13 F80FE1F815127F9117>97 DI<03FC000E0E001C1F003C1F 00781F00780E00F80000F80000F80000F80000F80000F800007800007801803C01801C03000E0E 0003F80011127E9115>I<000FF0000FF00001F00001F00001F00001F00001F00001F00001F000 01F00001F001F9F00F07F01C03F03C01F07801F07801F0F801F0F801F0F801F0F801F0F801F0F8 01F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C1B>I<01FC000F07001C03803C01C0 7801C07801E0F801E0F801E0FFFFE0F80000F80000F800007800007C00603C00601E00C00F0380 01FC0013127F9116>I<007F0001E38003C7C00787C00F87C00F83800F80000F80000F80000F80 000F8000FFF800FFF8000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F80000F80000F80000F80007FF8007FF800121D809C0F>I<03F8F00E0F381E0F381C07303C 07803C07803C07803C07801C07001E0F000E0E001BF8001000001800001800001FFF001FFFC00F FFE01FFFF07801F8F00078F00078F000787000707800F01E03C007FF00151B7F9118>I<1E003F 003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D108 DII<01FC000F07801C01C03C01E07800F07800F0F800F8 F800F8F800F8F800F8F800F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118> II114 D<1FD830786018E018E018F000FF807FE07FF01FF807FC007CC01CC01CE01CE018F830CFC00E12 7E9113>I<0300030003000300070007000F000F003FFCFFFC1F001F001F001F001F001F001F00 1F001F001F0C1F0C1F0C1F0C0F08079803F00E1A7F9913>II120 D E /Fg 77 125 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 0E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00 000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<007FC001C1C003 03C00703C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F CFF8151D809C17>I<003F07E00001C09C18000380F018000701F03C000E01E03C000E00E01800 0E00E000000E00E000000E00E000000E00E000000E00E00000FFFFFFFC000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF80211D 809C23>I<60F0F0F0F0F0F0F060606060606060606060606060000000000060F0F060041E7C9D 0C>33 D<6060F0F0F8F86868080808080808101010102020404080800D0C7F9C15>I<00E00000 019000000308000003080000070800000708000007080000070800000710000007100000072000 000740000003C03FE003800F00038006000380040005C0040009C0080010E0100030E010006070 200060702000E0384000E03C4000E01C8000E00F0020E0070020700780403009C0401830E18007 C03E001B1F7E9D20>38 D<004000800100020006000C000C001800180030003000700060006000 6000E000E000E000E000E000E000E000E000E000E000E000E00060006000600070003000300018 0018000C000C00060002000100008000400A2A7D9E10>40 D<800040002000100018000C000C00 0600060003000300038001800180018001C001C001C001C001C001C001C001C001C001C001C001 C0018001800180038003000300060006000C000C00180010002000400080000A2A7E9E10>I<00 060000000600000006000000060000000600000006000000060000000600000006000000060000 000600000006000000060000FFFFFFE0FFFFFFE000060000000600000006000000060000000600 0000060000000600000006000000060000000600000006000000060000000600001B1C7E9720> 43 D<60F0F0701010101020204080040C7C830C>II<60F0F06004047C 830C>I<00010003000600060006000C000C000C0018001800180030003000300060006000C000 C000C0018001800180030003000300060006000C000C000C001800180018003000300030006000 60006000C000C00010297E9E15>I<03C00C301818300C300C700E60066006E007E007E007E007 E007E007E007E007E007E007E007E007E00760066006700E300C300C18180C3007E0101D7E9B15 >I<030007003F00C7000700070007000700070007000700070007000700070007000700070007 0007000700070007000700070007000F80FFF80D1C7C9B15>I<07C01830201C400C400EF00FF8 0FF807F8077007000F000E000E001C001C00380070006000C00180030006010C01180110023FFE 7FFEFFFE101C7E9B15>I<07E01830201C201C781E780E781E381E001C001C00180030006007E0 0030001C001C000E000F000F700FF80FF80FF80FF00E401C201C183007E0101D7E9B15>I<000C 00000C00001C00003C00003C00005C0000DC00009C00011C00031C00021C00041C000C1C00081C 00101C00301C00201C00401C00C01C00FFFFC0001C00001C00001C00001C00001C00001C00001C 0001FFC0121C7F9B15>I<300C3FF83FF03FC020002000200020002000200023E024302818301C 200E000E000F000F000F600FF00FF00FF00F800E401E401C2038187007C0101D7E9B15>I<00F0 030C06040C0E181E301E300C700070006000E3E0E430E818F00CF00EE006E007E007E007E007E0 07600760077006300E300C18180C3003E0101D7E9B15>I<4000007FFF807FFF007FFF00400200 80040080040080080000100000100000200000600000400000C00000C00001C000018000018000 038000038000038000038000078000078000078000078000078000078000030000111D7E9B15> I<03E00C301008200C20066006600660067006780C3E083FB01FE007F007F818FC307E601E600F C007C003C003C003C00360026004300C1C1007E0101D7E9B15>I<03C00C301818300C700C600E E006E006E007E007E007E007E0076007700F300F18170C2707C700060006000E300C780C781870 10203030C00F80101D7E9B15>I<60F0F0600000000000000000000060F0F06004127C910C>I<60 F0F0600000000000000000000060F0F0701010101020204080041A7C910C>I<0FE03038401CE0 0EF00EF00EF00E000C001C0030006000C000800180010001000100010001000100000000000000 0000000003000780078003000F1D7E9C14>63 D<000600000006000000060000000F0000000F00 00000F00000017800000178000001780000023C0000023C0000023C0000041E0000041E0000041 E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C0006003E0004 001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 D<001F808000E0618001801980 070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F00000007000008078000080780000803800 00801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>67 DIII<001F808000E061800180 1980070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F0 000000F0000000F0000000F0000000F0000000F000FFF0F0000F80700007807800078078000780 380007801C0007801C0007800E00078007000B800180118000E06080001F80001C1E7E9C21>I< FFF3FFC00F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C 000F003C000F003C000F003C000FFFFC000F003C000F003C000F003C000F003C000F003C000F00 3C000F003C000F003C000F003C000F003C000F003C000F003C000F003C00FFF3FFC01A1C7E9B1F >II76 DII<003F800000E0 E0000380380007001C000E000E001C0007003C00078038000380780003C0780003C0700001C0F0 0001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C0 780003C0380003803C0007801C0007000E000E0007001C000380380000E0E000003F80001B1E7E 9C20>II82 D<07E0801C1980300580700380600180 E00180E00080E00080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F80 0007C00003C00001C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E 9C17>I<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F00 20000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003FFFC001B 1C7F9B1E>IIII<7FF0FFC00FC03E000780180003C0180003E0100001E0200001 F0600000F0400000788000007D8000003D0000001E0000001F0000000F0000000F8000000F8000 0013C0000023E0000021E0000041F00000C0F8000080780001007C0003003C0002001E0006001F 001F003F80FFC0FFF01C1C7F9B1F>II<7FFFF07C01F07001E06003C06003C0400780400F80 400F00401E00001E00003C00007C0000780000F00000F00001E00003E00003C010078010078010 0F00101F00301E00203C00203C00607800E0F803E0FFFFE0141C7E9B19>I<0808101020204040 4040808080808080B0B0F8F8787830300D0C7A9C15>92 D<1FC000307000783800781C00301C00 001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F8700 12127E9115>97 DI<07E00C301878307870306000E000E0 00E000E000E000E00060007004300418080C3007C00E127E9112>I<003F000007000007000007 0000070000070000070000070000070000070000070003E7000C1700180F003007007007006007 00E00700E00700E00700E00700E00700E00700600700700700300700180F000C370007C7E0131D 7E9C17>I<03E00C301818300C700E6006E006FFFEE000E000E000E00060007002300218040C18 03E00F127F9112>I<00F8018C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003 C4C00C38C01C3880181800381C00381C00381C00381C001818001C38000C300013C00010000030 00001800001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007 F800121C7F9215>II<18003C003C001800000000000000 0000000000000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C 001C00FF80091D7F9C0C>I<00C001E001E000C000000000000000000000000000000FE000E000 E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0 F0C0F1C061803E000B25839C0D>IIIII<03F0000E1C00180600300300700380600180E0 01C0E001C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F91 15>II<03C1000C3300180B00300F00700700700700E00700E00700E00700E007 00E00700E00700600700700700300F00180F000C370007C7000007000007000007000007000007 00000700000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE0 0FF000F880388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C00 3C00FFE01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C 1A7F9910>IIII<7F8FF00F03800F030007020003840001C80001D80000F000007000 00780000F800009C00010E00020E000607000403801E07C0FF0FF81512809116>II<7FFC70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E 127F9112>I124 D E /Fh 24 118 df<0003E0000000000FF00000 00003E3800000000781800000000780C00000000F80C00000000F00C00000001F00C00000001F0 0C00000001F01C00000001F81800000001F83000000001F87000000001F86000000001F8C001FF E000FD8001FFE000FF00001C0000FE0000180000FE00003000007E00003000007F0000600000FF 0000600001FF8000C00003BF80018000071FC00180000F0FE00300001E0FE00600003E07F00600 007E03F80C0000FE03FC180000FE01FE300000FE00FE600000FE007FC00000FE003F8000C07F00 1FC000C07F000FF001C03F801FF803801FC0F0FE0F0007FFC03FFE0001FE0007F8002B287DA732 >38 D<3C7EFFFFFFFF7E3C08087B8712>46 D<000C00001C0000FC000FFC00FFFC00F0FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC00FFFFFCFFFFFC16257BA420>49 D<00FF000007FFE0001E07F8003801FC007800 FE007E00FF00FF007F00FF007F80FF007F80FF003F807E003F803C003F8000007F8000007F0000 007F000000FE000000FE000001FC000001F8000003F0000007C00000078000000F0000001E0000 003800000070018000E0018001C001800380030007000300060003000FFFFF001FFFFF003FFFFF 007FFFFE00FFFFFE00FFFFFE0019257DA420>I<00FF800007FFF0000F03F8001800FC003E00FE 007F00FF007F007F007F807F007F007F003F00FF001E00FE000000FE000000FC000001F8000003 F0000007E00000FF800000FF00000003E0000001F8000000FC000000FE0000007F0000007F0000 007F8018007F807E007F80FF007F80FF007F80FF007F80FF007F00FE00FF007C00FE003801FC00 1E03F8000FFFE00001FF000019257DA420>I<0000380000007800000078000000F8000001F800 0003F8000003F8000007F800000FF800001DF8000019F8000031F8000071F8000061F80000C1F8 0001C1F8000381F8000301F8000601F8000E01F8001C01F8001801F8003001F8007001F800E001 F800FFFFFFE0FFFFFFE00001F8000001F8000001F8000001F8000001F8000001F8000001F80000 01F800007FFFE0007FFFE01B257EA420>I<0000FF80080007FFF018003FC03C38007E000E7801 FC0003F803F00001F807E00000F80FE00000781FC00000781FC00000383F800000383F80000038 7F800000187F000000187F00000018FF00000000FF00000000FF00000000FF00000000FF000000 00FF00000000FF00000000FF00000000FF00000000FF000000007F000000007F000000187F8000 00183F800000183F800000181FC00000301FC00000300FE000006007E000006003F00000C001FC 000180007E000700003FC03C000007FFF8000000FFC00025287CA72E>67 DI73 D78 D80 D<03FF00000FFFE0001F03F0003F80F8003F80FC003F807C001F 007E001F007E0000007E0000007E0000007E00000FFE0001FFFE0007F07E001FC07E003F807E00 7F007E00FE007E00FE007E18FE007E18FE007E18FE00BE187F01BE183F873FF01FFE1FE003F80F 801D1A7E9920>97 D<003FE001FFF807E07C0FC0FE1F80FE3F00FE3F007C7E007C7E0000FE0000 FE0000FE0000FE0000FE0000FE0000FE0000FE00007E00007F00003F00003F80031F80070FC006 07F01C01FFF0003FC0181A7E991D>99 D<00007FE000007FE0000007E0000007E0000007E00000 07E0000007E0000007E0000007E0000007E0000007E0000007E0000007E0000007E0007F87E001 FFE7E007E077E00FC01FE01F800FE03F0007E03F0007E07E0007E07E0007E0FE0007E0FE0007E0 FE0007E0FE0007E0FE0007E0FE0007E0FE0007E0FE0007E07E0007E07E0007E07F0007E03F0007 E01F000FE00F801FE007E077E003FFC7FE007F07FE1F287EA724>I<007F8003FFE007E1F00F80 F81F007C3F007E7E003E7E003E7E003FFE003FFE003FFFFFFFFFFFFFFE0000FE0000FE0000FE00 007E00007E00003F00003F00031F80060FC00607F01C01FFF0003FC0181A7E991D>I<0F001F80 1FC03FC03FC01FC01F800F000000000000000000000000000000FFC0FFC00FC00FC00FC00FC00F C00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC0FFFCFFFC 0E297FA811>105 D108 D110 D<007FC00001FFF00007E0FC000F803E001F001F003F001F 803E000F807E000FC07E000FC0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE00 0FE0FE000FE07E000FC07E000FC07F001FC03F001F801F001F000F803E0007E0FC0001FFF00000 7FC0001B1A7E9920>II114 D<03F8C01FFFC03C07C07801C07001C0F000C0F000C0 F800C0FC0000FFC0007FFC003FFF001FFF800FFFC001FFE0000FF00003F0C001F0C000F0E000F0 E000F0F000E0F801E0FE03C0E7FF80C1FC00141A7E9919>I<00600000600000600000600000E0 0000E00001E00001E00003E00007E0001FE000FFFFC0FFFFC007E00007E00007E00007E00007E0 0007E00007E00007E00007E00007E00007E00007E00007E00007E00007E06007E06007E06007E0 6007E06007E06003F0C001F0C000FF80003E0013257FA419>II E /Fi 40 123 df<0001FC3C00060E67000C0E C7001C0DC6001C01C0003801C0003803800038038000380380003803800070038007FFFFF80070 0700007007000070070000E0070000E00E0000E00E0000E00E0000E00E0001C00E0001C01C0001 C01C0001C01C0001C01C0003801C00038038000380380003803800030038000700300007007000 06006000C6606000E470C000C8618000703E00002025819C19>11 D<0001FC000703000C03001C 07001C0300180000380000380000380000380000700007FFFC00701C00701C00701C00E03800E0 3800E03800E03800E07001C07001C07001C07001C0E201C0E201C0E20380E40380640380380380 00030000070000060000C60000E40000CC00007000001825819C17>I<18387838080810102040 4080050C7D830D>44 DI<3078F06005047C830D>I<060F0F060000 00000000000000003078F06008127C910D>58 D<01FFFE00003C0780003801C0003801C0003800 E0003800E0007000F00070007000700070007000F000E000F000E000F000E000F000E000F001C0 01E001C001E001C001E001C001C0038003C003800380038007800380070007000E0007001C0007 003800070070000E01C000FFFF00001C1C7D9B1F>68 D<01FFFFE0003C00E00038006000380040 00380040003800400070004000700040007020400070200000E0400000E0400000E0C00000FFC0 0001C0800001C0800001C0800001C0800003810100038001000380020003800200070004000700 040007000C00070018000E007800FFFFF0001B1C7D9B1C>I<01FFFFC0003C01C0003800C00038 008000380080003800800070008000700080007020800070200000E0400000E0400000E0C00000 FFC00001C0800001C0800001C0800001C080000381000003800000038000000380000007000000 0700000007000000070000000F000000FFF000001A1C7D9B1B>I<01FFC0003C00003800003800 00380000380000700000700000700000700000E00000E00000E00000E00001C00001C00001C000 01C0000380000380000380000380000700000700000700000700000F0000FFE000121C7E9B10> 73 D<01FFE0003C0000380000380000380000380000700000700000700000700000E00000E000 00E00000E00001C00001C00001C00001C000038008038008038008038010070010070030070060 0700E00E03C0FFFFC0151C7D9B1A>76 D<01FC03FE001C0070003C0060002E0040002E0040002E 0040004700800047008000470080004380800083810000838100008181000081C1000101C20001 01C2000100E2000100E2000200E400020074000200740002007400040038000400380004003800 0C0018001C001000FF8010001F1C7D9B1F>78 D<01FFFC00003C070000380380003801C0003801 C0003801C0007003C0007003C0007003C00070038000E0078000E0070000E00E0000E0380001FF E00001C0000001C0000001C0000003800000038000000380000003800000070000000700000007 000000070000000F000000FFE000001A1C7D9B1C>80 D<01FFF800003C0E000038070000380380 003803800038038000700780007007800070078000700F0000E00E0000E01C0000E0700000FFC0 0001C0C00001C0600001C0700001C07000038070000380700003807000038070000700F0000700 F0400700F0400700F0800F007880FFE0790000001E001A1D7D9B1E>82 D<1FFFFFC01C0701C030 0E00C0200E0080600E0080400E0080401C0080801C0080801C0080001C00000038000000380000 00380000003800000070000000700000007000000070000000E0000000E0000000E0000000E000 0001C0000001C0000001C0000001C0000003C000007FFE00001A1C799B1E>84 D86 D<03CC063C0C3C181C3838303870387038E070E070E070E070E0E2C0E2C0E261 E462643C380F127B9115>97 D<3F00070007000E000E000E000E001C001C001C001C0039C03E60 383038307038703870387038E070E070E070E060E0E0C0C0C1C0618063003C000D1D7B9C13>I< 01F007080C08181C3838300070007000E000E000E000E000E000E008E010602030C01F000E127B 9113>I<001F80000380000380000700000700000700000700000E00000E00000E00000E0003DC 00063C000C3C00181C00383800303800703800703800E07000E07000E07000E07000E0E200C0E2 00C0E20061E4006264003C3800111D7B9C15>I<01E007100C1018083810701070607F80E000E0 00E000E000E000E0086010602030C01F000D127B9113>I<0003C0000670000C70001C60001C00 001C0000380000380000380000380000380003FF8000700000700000700000700000700000E000 00E00000E00000E00000E00001C00001C00001C00001C00001C000038000038000038000030000 030000070000C60000E60000CC00007800001425819C0D>I<00F3018F030F06070E0E0C0E1C0E 1C0E381C381C381C381C383830383038187818F00F700070007000E000E0C0C0E1C0C3007E0010 1A7D9113>I<0FC00001C00001C000038000038000038000038000070000070000070000070000 0E78000E8C000F0E000E0E001C0E001C0E001C0E001C0E00381C00381C00381C00383800703880 703880707080707100E03200601C00111D7D9C15>I<0180038001000000000000000000000000 0000001C002600470047008E008E000E001C001C001C0038003800710071007100720072003C00 091C7C9B0D>I<0FC00001C00001C0000380000380000380000380000700000700000700000700 000E0F000E11000E23800E43801C83001C80001D00001E00003F800039C00038E00038E00070E2 0070E20070E20070E400E06400603800111D7D9C13>107 D<1F80038003800700070007000700 0E000E000E000E001C001C001C001C0038003800380038007000700070007000E400E400E400E4 0068003800091D7C9C0B>I<3C1E0780266318C04683A0E04703C0E08E0380E08E0380E00E0380 E00E0380E01C0701C01C0701C01C0701C01C070380380E0388380E0388380E0708380E0710701C 0320300C01C01D127C9122>I<3C3C002646004687004707008E07008E07000E07000E07001C0E 001C0E001C0E001C1C00381C40381C40383840383880701900300E0012127C9117>I<01E00718 0C0C180C380C300E700E700EE01CE01CE01CE018E038E030E06060C031801E000F127B9115>I< 07870004D98008E0C008E0C011C0E011C0E001C0E001C0E00381C00381C00381C0038180070380 0703000707000706000E8C000E70000E00000E00001C00001C00001C00001C00003C0000FF8000 131A7F9115>I<3C3C26C2468747078E068E000E000E001C001C001C001C003800380038003800 7000300010127C9112>114 D<01F006080C080C1C18181C001F001FC00FF007F0007800386030 E030C030806060C01F000E127D9111>I<00C001C001C001C00380038003800380FFE007000700 07000E000E000E000E001C001C001C001C00384038403840388019000E000B1A7D990E>I<1E03 00270700470700470700870E00870E000E0E000E0E001C1C001C1C001C1C001C1C003838803838 801838801839001C5900078E0011127C9116>I<1E06270E470E4706870287020E020E021C041C 041C041C0818083808181018200C4007800F127C9113>I<1E0183270387470387470383870701 8707010E07010E07011C0E021C0E021C0E021C0E04180C04181C04181C081C1C100C263007C3C0 18127C911C>I<070E0019910010E38020E38041C30041C00001C00001C0000380000380000380 00038000070200670200E70400CB04008B080070F00011127D9113>I<1E03270747074707870E 870E0E0E0E0E1C1C1C1C1C1C1C1C38383838183818381C7007F00070007000E0E0C0E1C0818047 003C00101A7C9114>I<038207C20FEC0838100800100020004000800100020004000808100838 3067F043E081C00F127D9111>I E /Fj 1 49 df<001E003F003F003F003F007E007E007E00FC 00FC00FC00F801F801F801F001F003F003E003E003C007C007C0078007800F800F000F000F001E 001E001E003C003C003C0038007800780070007000F000E0006000102A7EAD14>48 D E /Fk 8 116 df73 D80 D<0007FFC0000000003FFFFC00000000FFFFFF00000003F801FFC000 0007F0003FE0000007F8001FF000000FFC000FF800000FFC000FFC00000FFC0007FC00000FFC00 07FE00000FFC0003FE000007F80003FF000003F00003FF000000000003FF000000000003FF0000 00000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000000007FF FF00000000FFFFFF0000000FFFE3FF0000007FF803FF000001FFC003FF000003FF0003FF000007 FC0003FF00000FF80003FF00001FF00003FF00003FF00003FF00007FE00003FF00007FE00003FF 0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00007FF0380FFC0 0007FF03807FE0000DFF03807FE0001DFF03803FF00039FF87001FF80070FFCF000FFE03E07FFE 0007FFFF807FFC0000FFFE001FF800001FF00007E000312E7CAD37>97 D<00FF80FFFF80FFFF80 FFFF8003FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF80FFFFFFFFFFFFFFFFFF18487DC71F>108 D<00003FFC00000001FFFF8000000FFFFFF000 003FF00FFC00007F8001FE0001FE00007F8003FC00003FC007F800001FE007F800001FE00FF000 000FF01FE0000007F81FE0000007F83FE0000007FC3FE0000007FC7FC0000003FE7FC0000003FE 7FC0000003FE7FC0000003FEFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC000 0003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FF7FC0000003FE 7FC0000003FE7FC0000003FE7FE0000007FE3FE0000007FC3FE0000007FC1FE0000007F81FF000 000FF80FF000000FF007F800001FE007FC00003FE003FE00007FC001FF0000FF80007F8001FE00 003FF00FFC00000FFFFFF0000003FFFFC00000003FFC0000302E7DAD37>111 D<00FF801FF80000FFFF80FFFF0000FFFF83FFFFC000FFFF8FC07FF00003FF9E000FF80001FFF8 0007FC0001FFF00003FE0001FFE00001FF0001FFC00000FF8001FF800000FFC001FF8000007FC0 01FF8000007FE001FF8000007FE001FF8000003FE001FF8000003FF001FF8000003FF001FF8000 003FF001FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801 FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF800000 1FF801FF8000001FF001FF8000003FF001FF8000003FF001FF8000003FF001FF8000003FE001FF 8000007FE001FF8000007FC001FF800000FFC001FF800000FF8001FFC00001FF8001FFE00001FF 0001FFF00003FE0001FFF8000FFC0001FF9E001FF80001FF8F80FFE00001FF87FFFFC00001FF81 FFFE000001FF803FF0000001FF800000000001FF800000000001FF800000000001FF8000000000 01FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF8000 00000001FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001 FF800000000001FF8000000000FFFFFF00000000FFFFFF00000000FFFFFF0000000035427DAD3D >I<00FF007F00FFFF01FFC0FFFF03FFE0FFFF0787F003FF0E0FF001FF1C1FF801FF381FF801FF 301FF801FF701FF801FF600FF001FF600FF001FFE003C001FFC0000001FFC0000001FFC0000001 FFC0000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF800000 01FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF8000 0001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80 000001FF80000001FF80000001FF80000001FF800000FFFFFFC000FFFFFFC000FFFFFFC000252E 7DAD2C>114 D<001FFC030000FFFF870003FFFFCF000FE007FF001F8000FF003E00003F003E00 001F007C00001F007C00000F00FC00000F00FC00000700FC00000700FE00000700FF00000700FF 80000000FFE00000007FFE0000007FFFF800003FFFFF00003FFFFFC0001FFFFFF0000FFFFFF800 07FFFFFC0001FFFFFE00007FFFFF00001FFFFF000000FFFF80000007FF80000000FFC0E000007F C0E000003FC0E000001FC0F000000FC0F000000FC0F800000FC0F800000FC0F800000F80FC0000 0F80FE00001F80FF00001F00FF80003E00FFC0007C00F9F803F800F0FFFFF000E03FFFC000C007 FE0000222E7CAD2B>I E /Fl 8 117 df<00003000000070000001F0000007F000007FF000FFFF F000FFFFF000FF8FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00000 0FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000 000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF0 00000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000F F000000FF000000FF000000FF000000FF000000FF000000FF000000FF0007FFFFFFE7FFFFFFE7F FFFFFE1F3779B62D>49 D<000000FFE0001800000FFFFC001800007FFFFF00380001FFE00FC078 0007FE0001F0F8000FF8000079F8003FE000003FF8007FC000000FF800FF80000007F801FF0000 0007F803FE00000003F807FC00000001F807F800000001F80FF800000000F80FF000000000F81F F000000000781FF000000000783FE000000000783FE000000000387FE000000000387FE0000000 00387FE000000000387FC000000000007FC00000000000FFC00000000000FFC00000000000FFC0 0000000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FFC000000000 00FFC00000000000FFC00000000000FFC000000000007FC000000000007FC000000000007FE000 000000007FE000000000387FE000000000383FE000000000383FE000000000381FF00000000038 1FF000000000700FF000000000700FF8000000007007F800000000E007FC00000000E003FE0000 0001C001FF000000038000FF8000000780007FC000000F00003FE000001E00000FF800003C0000 07FE0000F0000001FFE007E00000007FFFFF800000000FFFFE0000000000FFE00000353B7BB940 >67 D<003FFC00000001FFFF80000007E00FE000000FC003F800000FE001FC00001FF001FE0000 1FF000FF00001FF000FF00001FF0007F00000FE0007F800007C0007F80000000007F8000000000 7F80000000007F80000000007F80000000007F800000000FFF80000007FFFF8000003FE07F8000 01FF007F800007FC007F80000FF0007F80001FE0007F80003FE0007F80007FC0007F80007FC000 7F8380FF80007F8380FF80007F8380FF80007F8380FF8000BF8380FF8000BF83807FC0013F8380 7FC0033F83803FE0061FC7001FF81C0FFE0007FFF007FC00007FC003F00029257DA42D>97 D<0003FF0000001FFFE000007F07F00001FC01FC0003F000FE0007E0007F000FE0003F001FC000 3F801FC0003F803FC0001FC03F80001FC07F80001FC07F80001FE07F80001FE0FF80001FE0FF80 001FE0FFFFFFFFE0FFFFFFFFE0FF80000000FF80000000FF80000000FF80000000FF800000007F 800000007F800000007F800000003FC00000003FC00000001FC00000E01FE00000E00FE00001C0 07F000038003F800070000FE000E00007FC07C00001FFFF0000001FF800023257EA428>101 D<01FC00000000FFFC00000000FFFC00000000FFFC0000000007FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC01FF000003FC07FFC00003FC1C0FF00003FC3007F80003FC6003F8 0003FCC003FC0003FC8001FC0003FD0001FE0003FF0001FE0003FE0001FE0003FE0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE00FFFFF07FFFF8FFFFF07FFFF8FFFFF07FFFF82D3A 7EB932>104 D<01FC03FE0000FFFC1FFFC000FFFC780FF000FFFDE003F80007FF8001FC0003FF 0000FE0003FE0000FF0003FC00007F8003FC00007FC003FC00003FC003FC00003FE003FC00003F E003FC00001FE003FC00001FE003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC 00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FE003FC00003F E003FC00003FE003FC00003FC003FC00003FC003FC00007F8003FC00007F8003FE0000FF0003FF 0001FE0003FF8003FC0003FDC007F80003FCF81FE00003FC3FFF800003FC07FC000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000000FFFFF00000 00FFFFF0000000FFFFF00000002C357EA432>112 D<01F80FC0FFF83FF0FFF870F8FFF8C1FC07 F883FE03F983FE03F903FE03FB03FE03FA01FC03FA00F803FA007003FE000003FC000003FC0000 03FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC00 0003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC 0000FFFFF800FFFFF800FFFFF8001F257EA424>114 D<001C0000001C0000001C0000001C0000 001C0000003C0000003C0000003C0000003C0000007C0000007C000000FC000000FC000001FC00 0003FC000007FC00001FFFFFC0FFFFFFC0FFFFFFC003FC000003FC000003FC000003FC000003FC 000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC000003FC000003FC000003FC000003FC000003FC00E003FC00E003FC00E003FC00E003FC00E0 03FC00E003FC00E003FC00E001FC00C001FC01C001FE01C000FE0380007F8700001FFE000007F8 001B357EB423>116 D E /Fm 18 122 df<008003800F80F38003800380038003800380038003 800380038003800380038003800380038003800380038003800380038003800380038003800380 038007C0FFFE0F217CA018>49 D<03F8000C1E001007002007804007C07807C07803C07807C038 07C0000780000780000700000F00000E0000380003F000001C00000F000007800007800003C000 03C00003E02003E07003E0F803E0F803E0F003C04003C0400780200780100F000C1C0003F00013 227EA018>51 D<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0F001 E0F001E0F001E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001C00001 C00001C0000380000380300300780700780600700C002018001030000FC00013227EA018>57 D<0007E0100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C00 00303C0000307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F800000078000000780000107C0000103C0000103C0000101C0000201E000020 0E000040070000400380008001C0010000E0020000381C000007E0001C247DA223>67 D<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00700F00F80F00F80F00F80E00F01E00401C0020380018700007C00014237EA119>74 D76 DI<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7 001E07003C0700780700700700F00708F00708F00708F00F087817083C23900FC1E015157E9418 >97 D<01FE000703000C07801C0780380300780000700000F00000F00000F00000F00000F00000 F00000F000007000007800403800401C00800C010007060001F80012157E9416>99 D<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000 E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I<01FC0007 07000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000F0000070 00007800203800201C00400E008007030000FC0013157F9416>I<0E0000FE00001E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E0070FFE7FF18237FA21B>104 D<0E0000FE00001E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E0180 0E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C0 0E01E00E00F00E00F8FFE3FE17237FA21A>107 D<0E00FE001E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E00FFE00B237FA20E>I<0E1F80FE60C01E80E00F00700F00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E 0070FFE7FF18157F941B>110 D<01FC000707000C01801800C03800E0700070700070F00078F0 0078F00078F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC0015 157F9418>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E 000E000E000F00FFF010157F9413>114 D 121 D E /Fn 1 49 df<038007C007C007C007800F800F800F000F001F001E001E001E001C003C 0038003800380070007000700060006000E000C00040000A1A7F9B0D>48 D E /Fo 20 121 df<00003FC0020001FFF8060007E01C06001F00070E003C00018E00780000DE 00F000005E01E000003E03C000001E078000001E0F0000000E0F0000000E1E000000061E000000 063E000000063C000000027C000000027C000000027C000000027800000000F800000000F80000 0000F800000000F800000000F800000000F800000000F800000000F800000000F800000000F800 00000078000000007C000000007C000000027C000000023C000000023E000000021E000000021E 000000040F000000040F0000000C078000000803C000001801E000001000F00000200078000040 003C000180001F0003000007E01E000001FFF80000003FC00027327CB02F>67 D73 D77 D80 D<007F004001FFC0400780F0C00F0018C01C000DC03C0007C0 380003C0780001C0700001C0F00000C0F00000C0F00000C0F0000040F0000040F0000040F80000 40F80000007C0000007E0000003F0000003FE000001FFE00000FFFE00007FFF80001FFFE00007F FF000007FF8000007F8000000FC0000007E0000003E0000003E0000001F0000001F0800000F080 0000F0800000F0800000F0800000F0C00000F0C00000E0E00001E0E00001E0F00001C0F8000380 EC000780C7000F00C1E03E0080FFF800801FE0001C327CB024>83 D<00FE0000070380000801E0 001000F0003C0078003E0078003E003C003E003C001C003C0000003C0000003C0000003C000007 FC00007C3C0003E03C0007803C001F003C003E003C003C003C007C003C0078003C08F8003C08F8 003C08F8003C08F8007C08F8007C087C00BC083E011E100F060FE003F807C01D1E7D9D20>97 D<018000003F800000FF800000FF8000000F800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 81F80007860700079803C007A000E007C000F007C00078078000380780003C0780003E0780001E 0780001E0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F078000 1E0780003E0780003C0780003C0780007807C00070074000F0072001C006100380060C0E000403 F80020317EB024>I<001FC00000F0380001C00400078002000F000F001E001F001E001F003C00 1F007C000E007C00000078000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000007C0000003C0000801E0000801E0001000F00010007800200 01C00C0000F03000001FC000191E7E9D1D>I<003F800000E0F0000380380007001C000E001E00 1E000E003C000F003C000F007C000F807800078078000780F8000780FFFFFF80F8000000F80000 00F8000000F8000000F8000000F800000078000000780000007C0000003C0000801C0000801E00 01000F0002000700020001C00C0000F03000001FC000191E7E9D1D>101 D<07000F801F801F800F8007000000000000000000000000000000000000000000000001801F80 FF80FF800F80078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800FC0FFF8FFF80D2F7EAE12>105 D<01803F80FF80FF800F80078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800FC0FFFCFFFC0E317EB012>108 D<0181FE003FC0003F860780C0F000FF8801C1003800FF90 01E2003C000FA000E4001C0007A000F4001E0007C000F8001E0007C000F8001E00078000F0001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341E7E9D38>I<01 81FC003F860F00FF880380FF9003C00FA001C007C001E007C001E007C001E0078001E0078001E0 078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00FC003F0FFFC3FFFFFFC 3FFF201E7E9D24>I<003FC00000E0700003801C0007000E000E0007001E0007803C0003C03C00 03C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F8 0001F0F80001F0780001E0780001E07C0003E03C0003C03C0003C01E0007800E00070007000E00 03801C0000E07000003FC0001C1E7E9D20>I<0181F8003F860F00FF9803C0FFA001E007C000F0 07C00078078000780780003C0780003E0780003E0780001E0780001F0780001F0780001F078000 1F0780001F0780001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C0 00F007C000F007A001C007900380078C0E000783F8000780000007800000078000000780000007 8000000780000007800000078000000780000007800000078000000FC00000FFFC0000FFFC0000 202C7E9D24>I<0183E03F8C18FF907CFF907C0FA07C07C03807C00007C00007C0000780000780 000780000780000780000780000780000780000780000780000780000780000780000780000780 000780000780000780000FC000FFFE00FFFE00161E7E9D19>114 D<03FC200E02603801E03000 E0600060E00060E00020E00020F00020F000207C00007F80003FFC001FFF0007FF8001FFE0000F E00001F08000F8800078800038C00038C00038C00038E00030F00070F00060C800C0C6038081FE 00151E7E9D19>I<00400000400000400000400000400000C00000C00000C00001C00001C00003 C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C01003C01003C01003C01003C01003C01003 C01003C01001C02001E02000E0400078C0001F00142B7FAA19>I<018000603F800FE0FF803FE0 FF803FE00F8003E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00780 03E0078003E0078003E0038005E003C009E001C019F000F021FF001FC1FF201E7E9D24>I120 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%PaperSize: A4 %%EndSetup %%Page: 0 1 0 0 bop 811 1086 a Fo(Prop)r(osal)24 b(I)r(I)1129 1048 y Fn(0)575 1178 y Fo(MPI)f(Con)n(text)f(Sub)r(committee)799 1360 y Fm(Lyndon)17 b(J)f(Clark)o(e)853 1481 y(Marc)o(h)g(1993)p eop %%Page: 1 2 1 1 bop 262 619 a Fl(Chapter)28 b(1)262 826 y Fk(Prop)s(osal)36 b(I)s(I)803 765 y Fj(0)262 1042 y Fi(Editorial)12 b(Note:)18 b(This)13 b(is)g(not)h(Pr)n(op)n(osal)f(II)g(as)h(identi\014e)n(d)g(at)f(the) g(p)n(ost-me)n(et)g(lunch)h(after)262 1092 y(the)d(F)m(ebruary)f(me)n(eting)h (in)h(Dal)r(las.)17 b(R)o(ik)11 b(Little\014e)n(d)g(c)n(ame)g(on)h(b)n(o)n (ar)n(d)f(the)g(pr)n(op)n(osal)g(writing)262 1142 y(pr)n(o)n(c)n(ess,)j(and)i (de)n(cide)n(d)g(to)f(write)f(a)i(di\013er)n(ent)f(pr)n(op)n(osal)g(which)g (app)n(e)n(ars)h(as)f(Pr)n(op)n(osal)g(V.)262 1191 y(Ther)n(e)f(is)g(no)h (longer)f(a)h(pr)n(op)n(osal)g(which)f(c)n(ontains)h(static)f(c)n(ontexts)h (as)g(discusse)n(d)g(at)g(the)262 1241 y(p)n(ost-me)n(et)f(lunch.)262 1378 y Fh(1.1)64 b(In)n(tro)r(duction)262 1469 y Fg(This)9 b(c)o(hapter)h(prop)q(oses)g(that)g(comm)o(unicatio)o(n)d(con)o(texts)j(and)f (pro)q(cess)i(groupings)e(within)262 1519 y(MPI)20 b(app)q(ear)g(as)g (related)h(concepts.)38 b(In)20 b(particular)g(pro)q(cess)i(groupings)d(app)q (ear)i(as)262 1569 y(\\frames")14 b(whic)o(h)i(are)g(used)h(in)e(the)h (creation)h(of)e(comm)o(unicati)o(on)e(con)o(texts.)25 b(Comm)o(u-)262 1619 y(nications)13 b(con)o(texts)i(retain)g(a)e(reference)k(to,)d(and)g (inherit)g(prop)q(erties)h(of,)e(their)i(pro)q(cess)262 1669 y(grouping)f(frames.)22 b(This)15 b(re\015ects)i(the)f(observ)n(ation)f(that) h(an)f(in)o(v)o(o)q(cation)f(of)h(a)g(mo)q(dule)262 1718 y(in)c(a)i(parallel) e(program)f(t)o(ypically)h(op)q(erates)j(within)e(one)g(or)h(more)e(groups)i (of)e(pro)q(cesses,)262 1768 y(and)f(as)h(suc)o(h)h(an)o(y)e(comm)o (unication)d(con)o(texts)12 b(asso)q(ciated)g(with)e(in)o(v)o(o)q(cations)g (of)h(mo)q(dules)262 1818 y(also)i(bind)g(certain)i(seman)o(tics)e(of)h(pro)q (cess)h(groupings.)324 1868 y(The)20 b(prop)q(osal)f(pro)o(vides)g(pro)q (cess)i(iden)o(ti\014ed)f(comm)o(unicatio)o(n,)e(comm)o(unicati)o(ons)262 1918 y(whic)o(h)d(are)g(limited)e(in)i(scop)q(e)h(to)f(single)g(con)o(texts,) h(and)f(comm)o(unicatio)o(ns)e(whic)o(h)i(ha)o(v)o(e)262 1968 y(scop)q(e)g(spanning)e(pairs)h(of)f(con)o(texts.)19 b(The)c(prop)q(osal)e (mak)o(es)g(no)h(statemen)o(ts)g(regarding)262 2017 y(message)c(tags.)16 b(It)11 b(is)f(assumed)g(that)g(these)i(will)d(b)q(e)i(a)f(bit)g(string)g (expressed)j(as)d(an)g(in)o(teger)262 2067 y(in)j(the)h(host)h(language.)324 2117 y(Muc)o(h)d(of)e(this)i(prop)q(osal)f(m)o(ust)f(b)q(e)i(view)o(ed)f(as)h (recommendations)d(to)i(other)h(sub)q(com-)262 2167 y(mittees)g(of)g(MPI,)g (primarily)e(the)j(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(unication)e(sub)q (committee)j(and)262 2217 y(the)17 b(collectiv)o(e)f(comm)o(unicatio)o(ns)e (sub)q(committee.)25 b(Concrete)17 b(syn)o(tax)g(is)f(giv)o(en)g(in)g(the)262 2266 y(st)o(yle)e(of)f(the)h(ANSI)h(C)e(host)i(language)d(for)i(only)f(purp)q (oses)i(of)f(discussion.)324 2316 y(The)g(detailed)g(prop)q(osal)g(is)g (presen)o(ted)i(in)d Ff(Section)h(1.2.)p Fg(,)g(whic)o(h)f(refers)j(the)e (reader)262 2366 y(to)i(a)g(set)h(of)e(discussion)i(notes)g(in)f Ff(Section)g(1.3.2.)p Fg(.)26 b(The)17 b(notes)g(assumes)f(kno)o(wledge)262 2416 y(of)e(the)h(prop)q(osal)f(text)h(and)g(are)g(therefore)h(b)q(est)f (examined)f(after)h(famili)o(arisatio)o(n)d(with)967 2574 y(1)p eop %%Page: 2 3 2 2 bop 262 307 a Fg(that)13 b(text.)18 b(Asp)q(ects)d(of)d(the)i(prop)q (osal)f(are)g(discussed)i(in)d(section)i Ff(Section)f(1.3.1.)p Fg(,)g(and)262 357 y(it)g(is)g(also)g(recommended)f(that)i(this)f(material)f (b)q(e)i(read)g(after)g(famili)o(arisatio)o(n)d(with)i(the)262 407 y(text)h(of)f(the)i(prop)q(osal.)262 544 y Fh(1.2)64 b(Detailed)24 b(Prop)r(osal)262 635 y Fg(This)15 b(section)h(presen)o(ts)h(the)g(detailed)e (prop)q(osal,)g(discussing)h(in)f(order)h(of)f(app)q(earance:)262 685 y(pro)q(cesses;)k(pro)q(cess)g(groupings;)d(comm)o(unication)d(con)o (texts;)18 b(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(u-)262 735 y(nication;)c(collectiv)o(e)i(comm)o(unication.)262 851 y Fe(1.2.1)55 b(Pro)r(cesses)262 927 y Fg(This)13 b(prop)q(osal)h(views)f(pro)q(cesses)k (in)c(the)i(famili)o(ar)c(w)o(a)o(y)m(,)h(as)i(one)g(thinks)g(of)f(pro)q (cesses)j(in)262 977 y(Unix)f(or)i(NX)f(for)g(example.)24 b(Eac)o(h)16 b(pro)q(cess)i(is)e(a)g(distinct)h(space)g(of)f(instructions)h(and)262 1027 y(data.)g(Eac)o(h)c(pro)q(cess)h(is)f(allo)o(w)o(ed)e(to)h(comp)q(ose)h (m)o(ultiple)d(concurren)o(t)k(threads)g(and)e(MPI)262 1077 y(do)q(es)i(not)g(distinguish)f(suc)o(h)i(threads.)262 1185 y Ff(Pro)q(cess)g(Descriptor)262 1261 y Fg(Eac)o(h)h(pro)q(cess)h(is)f (describ)q(ed)h(b)o(y)f(a)f Fi(pr)n(o)n(c)n(ess)h(descriptor)k Fg(whic)o(h)15 b(is)h(expressed)i(as)e(an)f(in-)262 1311 y(teger)f(t)o(yp)q (e)f(in)g(the)h(host)f(language)f(and)h(has)h(an)e(opaque)i(v)n(alue)e(whic)o (h)h(is)g(pro)q(cess)i(lo)q(cal.)262 1361 y Ff(See)g(Note)g(1.)k(See)c(Note)h (3.)i(See)d(Note)h(2.)324 1411 y Fg(The)d(initialisation)c(of)j(MPI)h (services)h(will)d(assign)h(to)h(eac)o(h)g(pro)q(cess)h(an)e Fi(own)k Fg(pro)q(cess)262 1461 y(descriptor.)i(Eac)o(h)11 b(pro)q(cess)i(retains)e(its)g(o)o(wn)g(pro)q(cess)i(descriptor)f(un)o(til)e (the)h(termination)262 1510 y(of)h(MPI)h(services.)19 b(MPI)13 b(pro)o(vides)g(a)g(pro)q(cedure)h(whic)o(h)f(returns)h(the)g(o)o(wn)e (descriptor)i(of)262 1560 y(the)g(calling)e(pro)q(cess.)20 b(F)m(or)14 b(example,)e Fd(pd)21 b(=)h(mpi)p 1052 1560 14 2 v 15 w(own)p 1133 1560 V 15 w(pd\(\))p Fg(.)262 1668 y Ff(Pro)q(cess)15 b(Creation)f(and)h(Destruction)262 1745 y Fg(This)g(prop)q(osal)g(mak)o(es)f (no)h(statemen)o(ts)h(regarding)f(creation)h(and)f(destruction)i(of)e(pro-) 262 1795 y(cesses.)20 b Ff(See)15 b(Note)h(6.)262 1903 y(Descriptor)d(T)l (ransmission)262 1979 y Fg(The)18 b(v)n(alue)g(of)f(a)h(pro)q(cess)i (descriptor)g(can)e(b)q(e)h(transmitted)e(in)h(a)g(message)g(as)g(an)g(in-) 262 2029 y(teger)d(since)h(it)e(is)h(an)f(in)o(teger)i(t)o(yp)q(e)f(in)f(the) h(host)g(language.)20 b(Ho)o(w)o(ev)o(er)15 b(the)h(recipien)o(t)f(of)262 2079 y(the)h(descriptor)i(can)e(mak)o(e)f(no)h(de\014ned)h(use)h(of)d(the)i (v)n(alue)f(of)f(in)h(the)h(MPI)f(op)q(erations)262 2129 y(describ)q(ed)f(in) f(this)g(prop)q(osal)f(|)g(the)i(descriptor)g(is)f Fi(invalid)p Fg(.)324 2178 y(MPI)j(pro)o(vides)f(a)g(mec)o(hanism)f(whereb)o(y)i(the)g (user)h(can)e(transmit)g(a)g(v)n(alid)f(pro)q(cess)262 2228 y(descriptor)j(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 2278 y(tegrated)e(with)g(the)g(capabilit)o(y)e(to)i(transmit)e(t)o(yp)q(ed)j (messages.)j(It)13 b(is)h(suggested)h(that)f(a)262 2328 y(notional)e(data)i (t)o(yp)q(e)g(should)g(b)q(e)g(in)o(tro)q(duced)h(for)e(this)h(purp)q(ose,)h (e.g.)i Fd(MPI)p 1468 2328 V 16 w(PD)p 1528 2328 V 15 w(TYPE)p Fg(.)324 2378 y(MPI)10 b(pro)o(vides)g(a)f(pro)q(cess)j(descriptor)f (registry)f(service.)18 b(The)10 b(op)q(erations)g(whic)o(h)g(this)262 2427 y(service)16 b(upp)q(orts)h(are:)22 b(register)16 b(descriptor)h(b)o(y)e (name;)g(deregister)i(descriptor;)g(lo)q(okup)967 2574 y(2)p eop %%Page: 3 4 3 3 bop 262 307 a Fg(descriptor)11 b(iden)o(ti\014er)g(b)o(y)f(name)g(and)g (v)n(alidate,)f(blo)q(c)o(king)h(the)h(caller)f(un)o(til)g(the)h(name)e(has) 262 357 y(b)q(een)16 b(registered.)24 b Ff(See)17 b(Note)g(7.)23 b Fg(Use)16 b(of)f(this)g(service)i(is)e(not)g(mandated.)22 b(Programs)262 407 y(whic)o(h)d(can)g(con)o(v)o(enien)o(tly)g(b)q(e)h (expressed)h(without)e(using)g(the)g(service)i(can)e(ignore)g(it)262 457 y(without)13 b(p)q(enalt)o(y)m(.)324 506 y(Note)19 b(that)g(receipt)i(of) d(a)h(pro)q(cess)i(descriptor)f(in)f(the)g(fashions)g(describ)q(ed)i(ab)q(o)o (v)o(e)262 556 y(ma)o(y)11 b(ha)o(v)o(e)i(a)g(p)q(ersisten)o(t)i(e\013ect)g (on)e(the)h(implem)o(en)o(tation)c(of)j(MPI)g(at)g(the)h(receiv)o(er,)h(and) 262 606 y(in)h(particular)h(ma)o(y)f(reserv)o(e)j(state.)29 b(MPI)18 b(will)d(pro)o(vide)j(a)f(pro)q(cedure)i(whic)o(h)e(in)o(v)n(alid-) 262 656 y(ates)d(a)g(v)n(alid)f(descriptor,)i(allo)o(wing)d(the)j(implemen)o (tatio)o(n)d(to)i(free)h(reserv)o(ed)h(state.)k(F)m(or)262 706 y(example,)d Fd(mpi)p 510 706 14 2 v 15 w(invalidate)p 745 706 V 14 w(pd\(pd\))p Fg(.)29 b(The)19 b(user)g(is)f(not)h(allo)o(w)o(ed) e(to)h(in)o(v)n(alidate)e(the)262 756 y(pro)q(cess)f(descriptor)g(of)f(the)g (calling)f(pro)q(cess.)19 b Ff(See)d(Note)f(8.)262 863 y(Descriptor)e(A)o (ttribu)o(tes)262 940 y Fg(This)g(prop)q(osal)h(mak)o(es)f(no)g(statemen)o (ts)h(regarding)g(pro)q(cessor)i(descriptor)f(attributes.)262 1056 y Fe(1.2.2)55 b(Pro)r(cess)18 b(Groupings)262 1133 y Fg(This)d(prop)q (osal)h(views)g(a)f(pro)q(cess)j(grouping)d(as)h(an)g(ordered)h(collection)e (of)h(\(references)262 1183 y(to?\))h(distinct)d(pro)q(cesses,)h(the)f(mem)o (b)q(ership)e(and)h(ordering)g(of)g(whic)o(h)g(do)q(es)h(not)f(c)o(hange)262 1232 y(o)o(v)o(er)i(the)i(lifetime)d(of)h(the)h(grouping.)24 b Ff(See)17 b(Note)h(9.)25 b Fg(The)16 b(canonical)f(represen)o(tation)262 1282 y(of)d(a)i(grouping)e(re\015ects)k(the)e(pro)q(cess)i(ordering)d(and)h (is)f(a)g(one-to-one)h(map)e(from)f Fc(Z)1611 1288 y Fb(N)1657 1282 y Fg(to)262 1332 y(the)j(descriptor)h(of)e(the)i Fc(N)k Fg(pro)q(cesses)d(comp)q(osing)d(the)h(grouping.)324 1382 y(The)c(structure)j (of)c(a)h(pro)q(cess)i(grouping)d(is)h(de\014ned)i(b)o(y)e(a)f(pro)q(cess)j (grouping)e(top)q(ology)262 1432 y(and)j(this)h(prop)q(osal)g(mak)o(es)e(no)i (further)h(statemen)o(ts)f(regarding)g(suc)o(h)g(structures.)262 1540 y Ff(Group)g(Descriptor)262 1616 y Fg(Eac)o(h)20 b(group)g(is)g(iden)o (ti\014ed)g(b)o(y)g(a)f Fi(gr)n(oup)i(descriptor)j Fg(whic)o(h)c(is)g (expressed)i(as)e(an)g(in-)262 1666 y(teger)14 b(t)o(yp)q(e)f(in)g(the)h (host)f(language)f(and)h(has)h(an)e(opaque)i(v)n(alue)e(whic)o(h)h(is)g(pro)q (cess)i(lo)q(cal.)262 1716 y Ff(See)g(Note)g(1.)k(See)c(Note)h(4.)i(See)d (Note)h(2.)324 1766 y Fg(The)f(initialisation)d(of)j(MPI)g(services)i(will)c (assign)i(to)g(eac)o(h)g(pro)q(cess)i(an)e Fi(own)g Fg(group)262 1816 y(descriptor)c(for)f(a)g(pro)q(cess)i(grouping)d(of)h(whic)o(h)g(the)g (pro)q(cess)i(is)e(a)g(mem)o(b)q(er.)15 b(Eac)o(h)c(pro)q(cess)262 1865 y(retains)j(its)h(o)o(wn)f(group)g(descriptor)h(and)f(mem)o(b)q(ership)f (of)h(the)h(pro)q(cess)h(grouping)d(un)o(til)262 1915 y(the)18 b(termination)f(of)h(MPI)g(services.)34 b Ff(See)20 b(Note)h(10.)31 b Fg(MPI)19 b(pro)o(vides)g(a)f(pro)q(cedure)262 1965 y(whic)o(h)h(returns)j (the)f(descriptor)g(of)e(the)i(home)e(group)g(of)h(the)g(calling)f(pro)q (cess.)38 b(F)m(or)262 2015 y(example,)12 b Fd(gd)21 b(=)h(mpi)p 614 2015 V 15 w(home)p 717 2015 V 15 w(gd\(\))p Fg(.)17 b Ff(See)e(Note)g (11.)262 2123 y(Group)f(Creation)g(and)h(Deletion)262 2199 y Fg(MPI)d(pro)o(vides)g(a)f(pro)q(cedure)j(whic)o(h)e(allo)o(ws)e(users)k (to)d(dynamically)e(create)14 b(one)e(or)g(more)262 2249 y(groups)d(whic)o(h) g(are)h(subsets)h(of)e(existing)g(groups.)16 b(F)m(or)9 b(example,)g Fd(gdb)21 b(=)g(mpi)p 1489 2249 V 15 w(group)p 1614 2249 V 15 w(partition\(gda,)262 2299 y(key\))13 b Fg(creates)k(one)e(or)g(more)e (new)j(groups)f Fd(gdb)f Fg(partition)g(an)g(existing)h(group)f Fd(gda)g Fg(in)o(to)262 2349 y(one)h(or)h(more)e(distinct)i(subsets.)25 b(This)16 b(pro)q(cedure)h(is)f(called)f(b)o(y)g(and)h(sync)o(hronises)h(all) 262 2399 y(mem)o(b)q(ers)12 b(of)h Fd(gda)p Fg(.)967 2574 y(3)p eop %%Page: 4 5 4 4 bop 324 307 a Fg(MPI)9 b(pro)o(vides)h(a)f(pro)q(cedure)i(whic)o(h)f (allo)o(ws)e(users)j(to)e(dynamically)d(create)11 b(one)f(group)262 357 y(b)o(y)j(explicit)h(de\014nition)g(of)f(its)h(mem)o(b)q(ership)f(as)h(a) g(list)f(of)h(pro)q(cess)i(descriptors.)k(F)m(or)13 b(ex-)262 407 y(ample,)c Fd(gd)21 b(=)h(mpi)p 571 407 14 2 v 15 w(group)p 696 407 V 14 w(definition\(listofpd)o(\))8 b Fg(creates)k(one)f(new)g(group)g Fd(gd)f Fg(with)262 457 y(mem)o(b)q(ership)17 b(and)i(ordering)g(describ)q (ed)i(b)o(y)e(the)h(pro)q(cess)h(descriptor)f(list)f Fd(listofpd)p Fg(.)262 506 y(This)9 b(pro)q(cedure)j(is)e(called)f(b)o(y)h(and)g(sync)o (hronises)h(all)e(pro)q(cesses)j(iden)o(ti\014ed)e(in)g Fd(listofpd)p Fg(.)324 556 y(MPI)h(pro)o(vides)h(a)f(pro)q(cedure)i(whic)o(h)e(allo)o(ws)f (users)j(to)e(delete)i(a)e(created)i(group.)k(This)262 606 y(pro)q(cedure)11 b(accepts)h(the)f(descriptor)g(of)f(a)f(group)h(whic)o(h)g (w)o(as)g(created)i(b)o(y)d(the)i(calling)e(pro-)262 656 y(cess)15 b(and)f(destro)o(ys)i(the)f(iden)o(ti\014ed)f(group.)19 b(F)m(or)14 b(example,)e Fd(mpi)p 1295 656 V 15 w(group)p 1420 656 V 15 w(deletion\(gd\))262 706 y Fg(deletes)17 b(an)e(existing)h(group)f Fd(gd)p Fg(.)23 b(This)16 b(pro)q(cedure)i(is)d(called)h(b)o(y)f(and)h(sync)o (hronises)h(all)262 756 y(mem)o(b)q(ers)12 b(of)h Fd(gd)p Fg(.)324 805 y(MPI)k(pro)o(vides)g(additional)e(pro)q(cedure)k(whic)o(h)e(allo)o(w)e (users)j(to)f(construct)i(pro)q(cess)262 855 y(groupings)13 b(whic)o(h)h(ha)o(v)o(e)f(a)h(pro)q(cess)i(grouping)d(top)q(ology)m(.)262 963 y Ff(Descriptor)g(T)l(ransmission)262 1040 y Fg(The)i(v)n(alue)f(of)g(a)g (group)h(descriptor)h(can)f(b)q(e)g(transmitted)g(in)f(a)g(message)h(as)g(an) f(in)o(teger)262 1089 y(since)j(it)g(is)f(an)h(in)o(teger)g(t)o(yp)q(e)g(in)g (the)g(host)g(language.)26 b(Ho)o(w)o(ev)o(er)17 b(the)g(recipien)o(t)h(of)e (the)262 1139 y(descriptor)h(can)g(mak)o(e)e(no)h(de\014ned)h(use)h(of)d(the) i(v)n(alue)f(of)g(in)g(the)h(MPI)g(op)q(erations)f(de-)262 1189 y(scrib)q(ed)f(in)e(this)h(prop)q(osal)g(|)f(the)h(descriptor)i(is)d Fi(invalid)p Fg(.)324 1239 y(MPI)19 b(pro)o(vides)g(a)g(mec)o(hanism)d (whereb)o(y)k(the)g(user)g(can)f(transmit)f(a)g(v)n(alid)g(group)262 1289 y(descriptor)g(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 1339 y(tegrated)e(with)g(the)g(capabilit)o(y)e(to)i(transmit)e(t)o(yp)q(ed)j (messages.)j(It)13 b(is)h(suggested)h(that)f(a)262 1388 y(notional)e(data)i (t)o(yp)q(e)g(should)g(b)q(e)g(in)o(tro)q(duced)h(for)e(this)h(purp)q(ose,)h (e.g.)i Fd(MPI)p 1468 1388 V 16 w(GD)p 1528 1388 V 15 w(TYPE)p Fg(.)324 1438 y(MPI)c(pro)o(vides)f(a)h(group)f(descriptor)i(registry)f (service.)19 b(The)13 b(op)q(erations)g(whic)o(h)g(this)262 1488 y(service)j(upp)q(orts)h(are:)22 b(register)16 b(descriptor)h(b)o(y)e (name;)g(deregister)i(descriptor;)g(lo)q(okup)262 1538 y(descriptor)11 b(iden)o(ti\014er)g(b)o(y)f(name)g(and)g(v)n(alidate,)f(blo)q(c)o(king)h(the) h(caller)f(un)o(til)g(the)h(name)e(has)262 1588 y(b)q(een)16 b(registered.)24 b Ff(See)17 b(Note)g(7.)23 b Fg(Use)16 b(of)f(this)g (service)i(is)e(not)g(mandated.)22 b(Programs)262 1637 y(whic)o(h)d(can)g (con)o(v)o(enien)o(tly)g(b)q(e)h(expressed)h(without)e(using)g(the)g(service) i(can)e(ignore)g(it)262 1687 y(without)13 b(p)q(enalt)o(y)m(.)324 1737 y(Note)h(that)f(receipt)i(of)d(a)i(group)f(descriptor)h(in)f(the)h (fashions)f(describ)q(ed)i(ab)q(o)o(v)o(e)f(ma)o(y)262 1787 y(ha)o(v)o(e)i(a)g(p)q(ersisten)o(t)i(e\013ect)g(on)e(the)h(implemen)o (tation)c(of)j(MPI)g(at)h(the)g(receiv)o(er,)h(and)e(in)262 1837 y(particular)g(ma)o(y)f(reserv)o(e)k(state.)27 b(MPI)17 b(will)f(pro)o(vide)g(a)h(pro)q(cedure)h(whic)o(h)f(in)o(v)n(alidates)262 1886 y(a)e(v)n(alid)f(descriptor,)j(allo)o(wing)c(the)j(implemen)o(tatio)o(n) d(to)i(free)i(reserv)o(ed)g(state.)24 b(F)m(or)15 b(ex-)262 1936 y(ample,)d Fd(mpi)p 465 1936 V 15 w(invalidate)p 700 1936 V 13 w(gd\(gd\))p Fg(.)18 b(The)c(user)h(is)f(not)g(allo)o(w)o(ed)f(to)h(in)o (v)n(alidate)e(the)j(o)o(wn)262 1986 y(group)d(descriptor)j(of)d(the)i(pro)q (cess)h(or)e(the)h(group)f(descriptor)h(of)f(an)o(y)f(group)h(created)i(b)o (y)262 2036 y(the)f(calling)e(pro)q(cess.)20 b Ff(See)c(Note)f(12.)262 2144 y(Descriptor)e(A)o(ttribu)o(tes)262 2220 y Fg(MPI)j(pro)o(vides)g(a)g (pro)q(cedure)i(whic)o(h)e(accepts)i(a)d(v)n(alid)g(group)h(descriptor)h(and) f(returns)262 2270 y(the)g(rank)f(of)g(the)i(calling)d(pro)q(cess)j(within)e (the)i(iden)o(ti\014er)f(group.)23 b(F)m(or)15 b(example,)f Fd(rank)262 2320 y(=)21 b(mpi)p 374 2320 V 15 w(group)p 499 2320 V 15 w(rank\(gd\))p Fg(.)324 2370 y(MPI)10 b(pro)o(vides)h(a)f(pro)q (cedure)j(whic)o(h)d(accepts)i(a)e(v)n(alid)f(group)h(descriptor)i(and)e (returns)262 2420 y(the)15 b(n)o(um)o(b)q(er)g(of)g(mem)o(b)q(ers,)e(or)j Fi(size)p Fg(,)e(of)h(the)h(iden)o(ti\014ed)f(group.)22 b(F)m(or)15 b(example,)f Fd(size)21 b(=)967 2574 y Fg(4)p eop %%Page: 5 6 5 5 bop 262 307 a Fd(mpi)p 331 307 14 2 v 15 w(group)p 456 307 V 14 w(size\(gd\))p Fg(.)324 357 y(MPI)16 b(pro)o(vides)f(a)g(pro)q (cedure)j(whic)o(h)d(accepts)i(a)e(v)n(alid)f(group)i(descriptor)g(and)g (pro-)262 407 y(cess)h(order)g(n)o(um)o(b)q(er,)f(or)g Fi(r)n(ank)p Fg(,)g(and)g(returns)i(the)f(v)n(alid)e(descriptor)i(of)f(the)h(pro)q(cess)h (to)262 457 y(whic)o(h)d(the)h(supplied)g(rank)f(maps)g(within)f(the)j(iden)o (ti\014ed)e(group.)23 b(F)m(or)15 b(example,)f Fd(pd)22 b(=)262 506 y(mpi)p 331 506 V 15 w(group)p 456 506 V 14 w(pd\(gd,)f(rank\))p Fg(.)324 556 y Ff(See)15 b(Note)h(13.)324 606 y Fg(MPI)c(pro)o(vides)h (additional)e(pro)q(cedures)j(whic)o(h)f(allo)o(w)d(users)k(to)e(determine)h (the)g(pro-)262 656 y(cess)i(grouping)e(top)q(ology)g(attributes.)262 772 y Fe(1.2.3)55 b(Comm)n(unication)16 b(Con)n(texts)262 849 y Fg(This)c(prop)q(osal)h(views)g(a)g(comm)o(unicatio)o(n)d(con)o(text)k(as)f (a)g(uniquely)f(iden)o(ti\014ed)h(reference)262 899 y(to)f(exactly)g(one)h (pro)q(cess)h(grouping,)e(whic)o(h)g(is)g(a)h(\014eld)f(in)g(a)g(message)g (en)o(v)o(elop)q(e)h(and)g(ma)o(y)262 948 y(therefore)j(b)q(e)g(used)g(to)f (distinguish)g(messages.)21 b(The)16 b(con)o(text)g(inherits)f(the)h (referenced)262 998 y(pro)q(cess)i(grouping)e(as)h(a)g(\\frame".)25 b(Eac)o(h)17 b(pro)q(cess)i(grouping)d(is)h(used)h(as)f(a)f(frame)g(for)262 1048 y(m)o(ultiple)11 b(con)o(texts.)262 1156 y Ff(Con)o(text)j(Descriptor) 262 1232 y Fg(Eac)o(h)h(con)o(text)i(is)e(iden)o(ti\014ed)h(b)o(y)f(a)g Fi(c)n(ontext)i(descriptor)j Fg(whic)o(h)15 b(is)g(expressed)j(as)e(an)f(in-) 262 1282 y(teger)f(t)o(yp)q(e)f(in)g(the)h(host)f(language)f(and)h(has)h(an)e (opaque)i(v)n(alue)e(whic)o(h)h(is)g(pro)q(cess)i(lo)q(cal.)262 1332 y Ff(See)g(Note)g(1.)k(See)c(Note)h(5.)i(See)d(Note)h(2.)324 1382 y Fg(The)g(creation)g(of)e(MPI)i(pro)q(cess)h(groupings)e(allo)q(cates)h (an)f Fi(own)k Fg(con)o(text)d(whic)o(h)f(in-)262 1432 y(herits)k(the)g (created)h(grouping)e(as)h(a)f(frame)g(and)g(can)h(b)q(e)g(though)o(t)g(of)f (as)g(a)h(prop)q(ert)o(y)262 1482 y(of)c(the)h(created)h(grouping.)23 b(The)16 b(grouping)f(retains)h(its)f(o)o(wn)h(con)o(text)g(un)o(til)f(MPI)h (pro-)262 1531 y(cess)g(grouping)e(deletion.)21 b(MPI)15 b(pro)o(vides)g(a)g (pro)q(cedure)i(whic)o(h)d(accepts)j(a)d(v)n(alid)g(group)262 1581 y(descriptor)e(and)e(returns)i(the)g(con)o(text)f(descriptor)h(of)e(the) h(o)o(wn)g(con)o(text)g(of)f(the)h(iden)o(ti\014ed)262 1631 y(group.)17 b(F)m(or)d(example,)e Fd(cd)21 b(=)h(mpi)p 822 1631 V 15 w(own)p 903 1631 V 15 w(cd\(gd\))p Fg(.)17 b Ff(See)e(Note)h(15.) 262 1739 y(Con)o(text)e(Creation)h(and)g(Deletion)262 1816 y Fg(MPI)i(pro)o(vides)g(a)g(pro)q(cedure)i(whic)o(h)d(allo)o(ws)g(users)j (to)d(dynamically)f(create)j(con)o(texts.)262 1865 y(This)11 b(pro)q(cedure)j(accepts)f(a)e(v)n(alid)f(descriptor)j(of)e(a)g(group)h(of)f (whic)o(h)g(the)i(calling)d(pro)q(cess)262 1915 y(is)16 b(a)h(mem)o(b)q(er,)f (and)h(returns)i(a)e(con)o(text)g(descriptor)i(whic)o(h)e(references)j(the)d (iden)o(ti\014ed)262 1965 y(group.)j(F)m(or)15 b(example,)e Fd(cd)21 b(=)h(mpi)p 827 1965 V 15 w(context)p 996 1965 V 14 w(create\(gd\))p Fg(.)d(This)14 b(pro)q(cedure)j(is)e(called)262 2015 y(b)o(y)e(and)h(sync)o(hronises)h(all)e(mem)o(b)q(ers)g(of)g Fd(gd)p Fg(.)k Ff(See)f(Note)f(14.)324 2065 y Fg(MPI)i(pro)o(vides)h(a)f(pro) q(cedure)i(whic)o(h)e(allo)o(ws)f(users)i(to)f(destro)o(y)h(created)h(con)o (texts.)262 2114 y(This)11 b(pro)q(cedure)j(accepts)f(a)e(v)n(alid)f(con)o (text)j(descriptor)g(whic)o(h)e(w)o(as)h(created)h(b)o(y)e(the)i(call-)262 2164 y(ing)8 b(pro)q(cess)j(and)e(deletes)i(that)e(con)o(text)h(iden)o (ti\014er.)17 b(F)m(or)9 b(example,)f Fd(mpi)p 1400 2164 V 15 w(context)p 1569 2164 V 15 w(delete\(cd\))p Fg(.)262 2214 y(This)13 b(pro)q(cedure)j(is)e(called)f(b)o(y)h(and)g(sync)o(hronises)h(all) e(mem)o(b)q(ers)f(of)i(the)g(frame)f(of)g Fd(cd)p Fg(.)262 2322 y Ff(Descriptor)g(T)l(ransmission)262 2399 y Fg(The)18 b(v)n(alue)f(of)h(a)f(con)o(text)i(descriptor)g(can)f(b)q(e)h(transmitted)f (in)f(a)h(message)f(as)h(an)g(in-)262 2448 y(teger)d(since)h(it)e(is)h(an)f (in)o(teger)i(t)o(yp)q(e)f(in)f(the)h(host)g(language.)20 b(Ho)o(w)o(ev)o(er) 15 b(the)h(recipien)o(t)f(of)967 2574 y(5)p eop %%Page: 6 7 6 6 bop 262 307 a Fg(the)16 b(descriptor)i(can)e(mak)o(e)f(no)h(de\014ned)h (use)h(of)d(the)i(v)n(alue)f(of)f(in)h(the)h(MPI)f(op)q(erations)262 357 y(describ)q(ed)f(in)f(this)g(prop)q(osal)f(|)g(the)i(descriptor)g(is)f Fi(invalid)p Fg(.)324 407 y(MPI)i(pro)o(vides)h(a)f(mec)o(hanism)d(whereb)o (y)18 b(the)f(user)g(can)f(transmit)f(a)h(v)n(alid)f(con)o(text)262 457 y(descriptor)j(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 506 y(tegrated)e(with)g(the)g(capabilit)o(y)e(to)i(transmit)e(t)o(yp)q(ed)j (messages.)j(It)13 b(is)h(suggested)h(that)f(a)262 556 y(notional)e(data)i(t) o(yp)q(e)g(should)g(b)q(e)g(in)o(tro)q(duced)h(for)e(this)h(purp)q(ose,)h (e.g.)i Fd(MPI)p 1468 556 14 2 v 16 w(CD)p 1528 556 V 15 w(TYPE)p Fg(.)324 606 y(MPI)9 b(pro)o(vides)h(a)f(con)o(text)h(descriptor)h(registry)f (service.)18 b(The)10 b(op)q(erations)f(whic)o(h)h(this)262 656 y(service)16 b(upp)q(orts)h(are:)22 b(register)16 b(descriptor)h(b)o(y)e (name;)g(deregister)i(descriptor;)g(lo)q(okup)262 706 y(descriptor)11 b(iden)o(ti\014er)g(b)o(y)f(name)g(and)g(v)n(alidate,)f(blo)q(c)o(king)h(the) h(caller)f(un)o(til)g(the)h(name)e(has)262 756 y(b)q(een)16 b(registered.)24 b Ff(See)17 b(Note)g(7.)23 b Fg(Use)16 b(of)f(this)g (service)i(is)e(not)g(mandated.)22 b(Programs)262 805 y(whic)o(h)d(can)g(con) o(v)o(enien)o(tly)g(b)q(e)h(expressed)h(without)e(using)g(the)g(service)i (can)e(ignore)g(it)262 855 y(without)13 b(p)q(enalt)o(y)m(.)324 905 y(Note)e(that)g(receipt)h(of)f(a)f(con)o(text)i(descriptor)g(in)e(the)i (fashions)e(describ)q(ed)j(ab)q(o)o(v)o(e)e(ma)o(y)262 955 y(ha)o(v)o(e)16 b(a)g(p)q(ersisten)o(t)i(e\013ect)g(on)e(the)h(implemen)o (tation)c(of)j(MPI)g(at)h(the)g(receiv)o(er,)h(and)e(in)262 1005 y(particular)g(ma)o(y)f(reserv)o(e)k(state.)27 b(MPI)17 b(will)f(pro)o(vide)g(a)h(pro)q(cedure)h(whic)o(h)f(in)o(v)n(alidates)262 1054 y(a)e(v)n(alid)f(descriptor,)j(allo)o(wing)c(the)j(implemen)o(tatio)o(n) d(to)i(free)i(reserv)o(ed)g(state.)24 b(F)m(or)15 b(ex-)262 1104 y(ample,)d Fd(mpi)p 465 1104 V 15 w(invalidate)p 700 1104 V 13 w(cd\(cd\))p Fg(.)18 b(The)c(user)h(is)f(not)g(allo)o(w)o(ed)f(to)h(in)o (v)n(alidate)e(the)j(o)o(wn)262 1154 y(con)o(text)h(descriptor)h(of)e(a)g (group)h(or)g(the)g(con)o(text)g(descriptor)h(of)e(an)o(y)h(con)o(text)g (created)262 1204 y(b)o(y)d(the)i(calling)d(pro)q(cess.)20 b Ff(See)15 b(Note)h(16.)262 1312 y(Descriptor)d(A)o(ttribu)o(tes)262 1388 y Fg(MPI)f(pro)o(vides)g(a)g(pro)q(cedure)h(whic)o(h)f(allo)o(ws)f (users)i(to)f(determine)g(the)h(pro)q(cess)g(grouping)262 1438 y(whic)o(h)g(is)h(the)h(frame)d(of)h(a)h(con)o(text.)19 b(F)m(or)13 b(example,)f Fd(gd)22 b(=)f(mpi)p 1282 1438 V 15 w(context)p 1451 1438 V 15 w(gd\(cd\))p Fg(.)262 1554 y Fe(1.2.4)55 b(P)n(oin)n(t-to-P)n (oin)n(t)19 b(Comm)n(unication)262 1631 y Fg(This)11 b(prop)q(osal)g (recommends)f(three)j(forms)d(for)h(MPI)g(p)q(oin)o(t-to-p)q(oin)o(t)f (message)h(address-)262 1681 y(ing)k(and)h(selection:)23 b(n)o(ull)15 b(con)o(text;)j(closed)e(con)o(text;)i(op)q(en)e(con)o(text.)26 b(It)16 b(is)g(further)h(re-)262 1731 y(commended)d(that)j(messages)f(comm)o (unicated)e(in)i(eac)o(h)g(form)f(are)i(distinguished)f(suc)o(h)262 1780 y(that)c(a)h(Send)g(op)q(eration)g(of)f(form)g(X)g(cannot)h(matc)o(h)f (with)h(a)f(receiv)o(e)i(op)q(eration)f(of)f(form)262 1830 y(Y,)h(whic)o(h)h(requires)h(that)f(the)g(form)e(is)i(em)o(b)q(edded)g(in)o (to)f(the)i(message)e(en)o(v)o(elop)q(e.)324 1880 y(The)j(three)h(forms)d (are)i(describ)q(ed)h(follo)o(w)o(ed)d(b)o(y)i(considerations)g(of)f(uniform) e(in)o(teg-)262 1930 y(ration)g(of)g(these)i(forms)e(in)g(the)i(p)q(oin)o (t-to-p)q(oin)o(t)d(comm)o(unication)f(section)j(of)g(MPI.)262 2038 y Ff(Null)g(Con)o(text)g(F)l(orm)262 2114 y Fg(The)g Fi(nul)r(l)h(c)n (ontext)j Fg(form)12 b(con)o(tains)i(no)f(message)h(con)o(text.)19 b Ff(See)c(Note)h(17.)324 2164 y Fg(Message)g(selection)f(and)g(addressing)g (are)g(expressed)i(b)o(y)e Fd(\(pd,)21 b(tag\))14 b Fg(where:)20 b Fd(pd)15 b Fg(is)262 2214 y(a)e(pro)q(cess)j(descriptor;)f Fd(tag)e Fg(is)g(a)h(message)g(tag.)324 2264 y(Send)g(supplies)h(the)f Fd(pd)f Fg(of)h(the)g(receiv)o(er.)20 b(Receiv)o(e)14 b(supplies)h(the)f Fd(pd)f Fg(of)h(the)g(sender.)324 2314 y(Receiv)o(e)20 b(can)f(wildcard)g(on) g Fd(pd)g Fg(b)o(y)h(supplying)e(the)i(v)n(alue)f(of)g(a)g(named)f(constan)o (t)262 2363 y(pro)q(cess)f(descirptor,)h(e.g.)24 b Fd(MPI)p 773 2363 V 15 w(PD)p 832 2363 V 15 w(WILD)p Fg(.)15 b(This)h(prop)q(osal)g (mak)o(es)e(no)i(statmen)o(t)g(ab)q(out)262 2413 y(the)e(pro)o(vision)f(for)g (wildcard)h(on)g Fd(tag)p Fg(.)967 2574 y(6)p eop %%Page: 7 8 7 7 bop 262 307 a Ff(Closed)14 b(Con)o(text)h(F)l(orm)262 384 y Fg(The)f Fi(close)n(d)g(c)n(ontext)k Fg(form)12 b(p)q(ermits)h(comm)o (unication)d(b)q(et)o(w)o(een)15 b(mem)o(b)q(ers)d(of)h(the)h(same)262 434 y(con)o(text.)k Ff(See)d(Note)h(19.)324 483 y Fg(Message)d(selection)f (and)g(addressing)g(are)g(expressed)i(b)o(y)d Fd(\(cd,)21 b(rank,)g(tag\))11 b Fg(where:)262 533 y Fd(cd)k Fg(is)g(a)h(con)o(text)g(descriptor;)i Fd(rank)c Fg(is)i(a)f(pro)q(cess)j(rank)e(in)f(the)h(frame)f(of)g Fd(cd)p Fg(;)g Fd(tag)g Fg(is)h(a)262 583 y(message)d(tag.)324 633 y(Send)f(supplies)g(the)g Fd(cd)f Fg(of)g(the)h(receiv)o(er)h(and)e (sender,)i(and)f(the)g Fd(rank)f Fg(of)f(the)j(receiv)o(er.)262 683 y(Receiv)o(e)i(supplies)f(the)i Fd(cd)e Fg(of)f(the)j(sender)g(and)e (receiv)o(er,)i(and)e(the)h(rank)f(of)g(the)h(sender.)262 732 y(The)i Fd(\(cd,)k(rank\))16 b Fg(pair)g(in)h(Send)g(\(Receiv)o(e\))h(is)f (su\016cien)o(t)g(to)g(determine)g(the)h(pro)q(cess)262 782 y(descriptor)d(of)e(the)h(receiv)o(er)i(\(sender\).)324 832 y(Receiv)o(e)e(cannot)g(wildcard)f(on)h Fd(cd)p Fg(.)j(Receiv)o(e)d(can)g (wildcard)g(on)f Fd(rank)g Fg(b)o(y)g(supplying)262 882 y(the)e(v)n(alue)g (of)f(a)h(named)f(constan)o(t)h(in)o(teger,)h(e.g.)k Fd(MPI)p 1101 882 14 2 v 16 w(RANK)p 1205 882 V 14 w(WILD)p Fg(.)10 b(This)h(prop)q(osal)g(mak)o(es)262 932 y(no)i(statmen)o(t)h(ab)q(out)f(the)i (pro)o(vision)e(for)g(wildcard)h(on)f Fd(tag)p Fg(.)262 1040 y Ff(Op)q(en)i(Con)o(text)f(F)l(orm)262 1116 y Fg(The)k Fi(op)n(en)h(c)n (ontext)j Fg(form)16 b(p)q(ermits)h(comm)o(unication)e(b)q(et)o(w)o(een)k (mem)o(b)q(ers)e(of)g(an)o(y)g(t)o(w)o(o)262 1166 y(con)o(texts.)i Ff(See)c(Note)g(18.)324 1216 y Fg(Message)d(selection)h(and)e(addressing)h (are)g(expressed)i(b)o(y)d Fd(\(lcd,)21 b(rcd,)g(rank,)f(tag\))262 1266 y Fg(where:)d Fd(lcd)11 b Fg(is)h(a)f(\\lo)q(cal")f(con)o(text)i (descriptor;)h Fd(rcd)e Fg(is)g(a)g(\\remote")g(con)o(text)h(descriptor;)262 1316 y Fd(rank)h Fg(is)g(a)h(pro)q(cess)i(rank)d(in)h(the)g(frame)f(of)g Fd(rcd)p Fg(;)g Fd(tag)g Fg(is)h(a)f(message)h(tag.)324 1365 y(Send)22 b(supplies)g(the)h(con)o(text)f(descriptor)h(for)f(the)g(sender)i (in)d Fd(lcd)p Fg(,)h(the)h(con)o(text)262 1415 y(descriptor)c(for)f(the)h (receiv)o(er)h(in)e Fd(rcd)p Fg(,)h(and)f(the)h Fd(rank)e Fg(of)h(the)h (receiv)o(er)h(in)e(the)h(frame)262 1465 y(of)12 b Fd(rcd)p Fg(.)18 b(Receiv)o(e)c(supplies)f(the)h(con)o(text)g(descriptor)h(for)e(the)h (receiv)o(er)h(in)e Fd(lcd)p Fg(,)g(the)h(con-)262 1515 y(text)f(descriptor)i (for)d(the)i(sender)h(in)d Fd(rcd)p Fg(,)g(and)h(the)h Fd(rank)e Fg(of)g(the)i(sender)h(receiv)o(er)f(in)f(the)262 1565 y(frame)c(of)i Fd(rcd)p Fg(.)16 b(The)c Fd(\(rcd,)21 b(rank\))10 b Fg(pair)h(in)f(Send)i (\(Receiv)o(e\))g(is)f(su\016cien)o(t)h(to)f(determine)262 1614 y(the)j(pro)q(cess)i(descriptor)f(of)e(the)i(receiv)o(er)g(\(sender\).) 324 1664 y(Receiv)o(e)f(cannot)g(wildcard)f(on)h Fd(lcd)p Fg(.)j(Receiv)o(e)d (can)g(wildcard)f(on)h Fd(rcd)f Fg(b)o(y)g(supplying)262 1714 y(the)j(v)n(alue)g(of)f(a)h(named)f(constan)o(t)i(con)o(text)g(descriptor,)g (e.g.)25 b Fd(MPI)p 1352 1714 V 15 w(CD)p 1411 1714 V 15 w(WILD)p Fg(,)15 b(in)h(whic)o(h)262 1764 y(case)f(Receiv)o(e)g Fi(must)k Fg(also)14 b(wildcard)g(on)g Fd(rank)g Fg(as)h(there)h(is)e(insu\016cien)o(t) h(information)d(to)262 1814 y(determine)j(the)h(pro)q(cess)i(descriptor)f(of) e(the)h(sender.)26 b(Receiv)o(e)16 b(can)g(wildcard)f(on)g Fd(rank)262 1863 y Fg(b)o(y)f(supplying)h(the)h(v)n(alue)e(of)h(a)g(named)f (constan)o(t)i(in)o(teger,)f(e.g.)22 b Fd(MPI)p 1384 1863 V 15 w(RANK)p 1487 1863 V 15 w(WILD)p Fg(.)14 b(This)262 1913 y(prop)q(osal)f(mak)o(es)g(no)g(statmen)o(t)h(ab)q(out)g(the)g(pro)o(vision)f (for)h(wildcard)f(on)h Fd(tag)p Fg(.)262 2021 y Ff(Uniform)f(In)o(tegration) 262 2098 y Fg(The)j(three)h(forms)e(of)g(addressing)i(and)f(selection)g (describ)q(ed)i(ha)o(v)o(e)e(di\013eren)o(t)h(syn)o(tactic)262 2148 y(framew)o(orks.)29 b(W)m(e)17 b(can)h(consider)h(in)o(tegrating)e (these)j(forms)c(in)o(to)h(the)i(p)q(oin)o(t-to-p)q(oin)o(t)262 2197 y(section)c(of)e(MPI)i(b)o(y)f(de\014ning)g(a)g(further)h(orthogonal)e (axis)h(\(as)h(in)e(the)i(m)o(ulti-lev)o(el)d(pro-)262 2247 y(p)q(osal)j(of)g(Gropp)h(&)g(Lusk\))g(whic)o(h)g(deals)g(with)g(the)g(form.) 23 b(This)15 b(is)h(at)g(the)g(exp)q(ense)i(of)262 2297 y(m)o(ultiplyi)o(ng)c (the)k(n)o(um)o(b)q(er)e(of)g(Send)i(and)e(Receiv)o(e)i(pro)q(cedures)h(b)o (y)e(a)f(factor)h(of)g(three,)262 2347 y(and)f(some)g(di\016cult)o(y)g(in)g (details)h(of)f(the)i(curren)o(t)g(p)q(oin)o(t-to-p)q(oin)o(t)d(w)o(orking)h (do)q(cumen)o(t)262 2397 y(whic)o(h)d(uniformly)f(assumes)h(a)h(single)f (addressing)i(and)f(selection)g(form.)967 2574 y(7)p eop %%Page: 8 9 8 8 bop 324 307 a Fg(There)22 b(are)g(v)n(arious)f(approac)o(hes)h(to)f (uni\014cation)g(of)g(the)h(syn)o(tactic)g(framew)o(orks)262 357 y(whic)o(h)16 b(ma)o(y)e(simplify)f(in)o(tegration.)24 b(Three)17 b(options)f(are)h(no)o(w)e(describ)q(ed,)j(eac)o(h)f(based)262 407 y(on)e(reten)o(tion)i(and)e(extension)i(of)e(the)i(framew)o(ork)d(of)h (one)h(form.)22 b(These)c(options)d(eac)o(h)262 457 y(ha)o(v)o(e)e(adv)n(an)o (tages)h(and)f(disadv)n(an)o(tages.)262 565 y Ff(Option)f(i:)41 b Fg(The)14 b(framew)o(ork)e(of)h(the)g(op)q(en)h(con)o(text)g(form)e(is)h (adopted)g(and)h(extended.)324 614 y(W)m(e)g(in)o(tro)q(duce)h(the)g Fi(nul)r(l)k Fg(descriptor,)c(the)g(v)n(alue)f(of)g(whic)o(h)g(is)h (de\014ned)g(b)o(y)g(a)f(named)262 664 y(constan)o(t,)f(e.g.)18 b Fd(MPI)p 590 664 14 2 v 15 w(NULL)p Fg(.)324 714 y(The)j(n)o(ull)e(con)o (text)j(form)c(is)j(expressed)i(as)d Fd(\(MPI)p 1153 714 V 15 w(NULL,)h(MPI)p 1365 714 V 15 w(NULL,)g(pd,)g(tag\))p Fg(,)262 764 y(whic)o(h)16 b(is)h(a)f(little)h(clumsy)m(.)25 b(The)17 b(closed)g(con)o(text)h(form)d(is)i(expressed)i(as)e Fd(\(MPI)p 1573 764 V 15 w(NULL,)262 814 y(cd,)k(rank,)f(tag\))p Fg(,)15 b(whic)o(h)h(is)f(marginally)d(incon)o(v)o(enien)o(t.)23 b(The)17 b(op)q(en)f(con)o(text)g(form)e(is)262 863 y(expressed)i(as)e Fd(\(lcd,)20 b(rcd,)h(rank,)g(tag)p Fg(\),)13 b(whic)o(h)h(is)g(of)f(course)i (natural.)262 971 y Ff(Option)9 b(ii:)40 b Fg(The)11 b(framew)o(ork)d(of)i (the)g(closed)h(con)o(text)g(form)d(is)i(adopted)h(and)f(extended.)324 1021 y(W)m(e)k(in)o(tro)q(duce)h(the)g Fi(nul)r(l)k Fg(descriptor,)c(the)g(v) n(alue)f(of)g(whic)o(h)g(is)h(de\014ned)g(b)o(y)g(a)f(named)262 1071 y(constan)o(t,)f(e.g.)18 b Fd(MPI)p 590 1071 V 15 w(NULL)p Fg(.)324 1121 y(The)c(n)o(ull)f(con)o(text)h(form)e(is)i(expressed)i(as)e Fd(\(MPI)p 1106 1121 V 15 w(NULL,)20 b(pd,)i(tag\))p Fg(,)12 b(whic)o(h)i(is)f(mar-)262 1171 y(ginally)8 b(incon)o(v)o(enien)o(t.)17 b(The)10 b(closed)h(con)o(text)g(form)e(is)h(expressed)j(as)d Fd(\(cd,)21 b(rank,)g(tag\))p Fg(,)262 1220 y(whic)o(h)12 b(is)h(of)f(course) i(natural.)j(Expression)c(of)f(the)i(op)q(en)f(con)o(text)g(form)e(requires)j (a)e(little)262 1270 y(more)g(w)o(ork.)324 1320 y(W)m(e)k(can)i(use)g(the)f Fd(cd)g Fg(\014eld)g(as)g(\\shorthand)h(notation")e(for)g(the)i Fd(\(lcd,)j(rcd\))16 b Fg(pair)262 1370 y(at)d(the)h(exp)q(ense)i(of)d(in)o (tro)q(ducing)g(some)g(tric)o(k)o(ery)m(.)18 b(W)m(e)13 b(can)h(de\014ne)g(a) f(mec)o(hanism)f(whic)o(h)262 1420 y(\\globs")c(together)i(these)h(t)o(w)o(o) e(\014elds)h(on)o(to)f(in)g(an)g(iden)o(ti\014er)h(whic)o(h)f(Send)h(and)f (Receiv)o(e)h(can)262 1469 y(distinguish)i(from)f(a)h(con)o(text)i(iden)o (ti\014er)f(and)g(treat)h(as)e(the)i(shorthand)f(notation.)k(Then)262 1519 y(w)o(e)e(should)f(also)g(de\014ne)i(a)f(mec)o(hanism)d(b)o(y)j(whic)o (h)g(a)f(receiv)o(er)j(whic)o(h)e(has)g(completed)f(a)262 1569 y(Receiv)o(e)h(with)f(wildcard)g(on)g Fd(rcd)g Fg(is)h(able)f(to)g(determine) h(the)g(v)n(alid)e(con)o(text)i(descriptor)262 1619 y(of)e(the)h(sender.)20 b(This)14 b(is)f(a)h(incon)o(v)o(enien)o(t.)262 1727 y Ff(Option)e(iii:)39 b Fg(The)13 b(framew)o(ork)e(of)h(the)h(n)o(ull)f(con)o(text)h(form)e(is)h (adopted)h(and)g(extended.)324 1777 y(The)f(n)o(ull)f(con)o(text)i(form)d(is) i(expressed)j(as)d Fd(\(pd,)21 b(tag\))p Fg(,)11 b(whic)o(h)h(is)g(of)f (course)j(natural.)262 1826 y(Expression)g(of)g(the)g(op)q(en)g(and)g(closed) h(con)o(text)f(forms)f(requires)i(a)e(little)h(more)f(w)o(ork.)324 1876 y(W)m(e)g(can)g(use)h(the)g Fd(pd)f Fg(\014eld)g(as)g(\\shorthand)h (notation")e(for)h Fd(\(cd,)21 b(rank\))12 b Fg(and)h Fd(\(lcd,)262 1926 y(rcd,)20 b(rank\))15 b Fg(b)o(y)h(con)o(tin)o(uation)f(of)g(the)h(tric) o(k)o(ery)h(used)f(in)g(the)g(previous)h(option.)23 b(This)262 1976 y(is)13 b(clumsy)m(.)262 2092 y Fe(1.2.5)55 b(Collectiv)n(e)16 b(Comm)n(unication)262 2169 y Fg(Symmetric)d(collectiv)o(e)j(comm)o (unication)c(op)q(erations)k(are)h(complian)o(t)c(with)j(the)g(closed)262 2218 y(con)o(text)e(form)d(describ)q(ed)16 b(ab)q(o)o(v)o(e.)h(This)d(prop)q (osal)f(recommends)g(that)g(suc)o(h)h(op)q(erations)262 2268 y(accept)e(a)f(con)o(text)h(descriptor)h(whic)o(h)e(iden)o(ti\014es)h(the)g (con)o(text)g(and)g(frame)e(in)h(whic)o(h)g(they)262 2318 y(are)j(to)g(op)q (erate.)324 2368 y(MPI)h(do)q(es)g(plan)f(to)h(describ)q(e)h(symmetric)d (collectiv)o(e)i(comm)o(unicati)o(on)d(op)q(erations.)262 2418 y(It)k(is)h(not)g(p)q(ossible)g(to)g(determine)g(whether)h(the)f(prop)q(osal) g(is)f(su\016cien)o(t)i(to)e(allo)o(w)g(im-)967 2574 y(8)p eop %%Page: 9 10 9 9 bop 262 307 a Fg(plemen)o(tation)16 b(of)h(the)h(collectiv)o(e)g(comm)o (unication)d(section)j(of)g(MPI)g(in)f(terms)h(of)f(the)262 357 y(p)q(oin)o(t-to-p)q(oin)o(t)h(section)j(of)f(MPI)g(without)g(loss)g(of)f (generalit)o(y)m(,)i(since)g(the)g(collectiv)o(e)262 407 y(op)q(erations)14 b(are)g(not)g(y)o(et)g(de\014ned.)324 457 y(Assymetric)j(collectiv)o(e)g (comm)o(unication)d(op)q(erations,)j(esp)q(ecially)g(those)h(in)f(whic)o(h) 262 506 y(the)12 b(sender\(s\))i(and)d(receiv)o(er\(s\))j(are)e(distinct)g (pro)q(cesses,)i(are)e(complian)o(t)e(with)h(the)h(op)q(en)262 556 y(con)o(text)i(form)d(describ)q(ed)16 b(ab)q(o)o(v)o(e.)h(This)d(prop)q (osal)f(recommends)g(that)g(suc)o(h)h(op)q(erations)262 606 y(accept)i(a)g(pair)f(of)g(con)o(text)h(descriptors)h(\(p)q(erhaps)g(in)e(a)h (\\glob")e(form\))g(whic)o(h)h(iden)o(tify)262 656 y(the)f(con)o(texts)h(and) f(frames)f(in)g(whic)o(h)h(they)g(are)h(to)e(op)q(erate.)324 706 y(MPI)j(do)q(es)g(not)f(plan)g(to)h(describ)q(e)h(assymetric)f(collectiv) o(e)f(comm)o(unication)d(op)q(era-)262 756 y(tions.)17 b(Suc)o(h)11 b(op)q(erations)h(are)g(expressiv)o(e)h(when)e(writing)g(programs)f(b)q(ey)o (ond)i(the)g(SPMD)262 805 y(mo)q(del)d(and)i(comprise)g(comm)o(unicati)o(v)o (e)e(fun)o(tionally)g(distinct)i(pro)q(cess)i(groupings.)k(This)262 855 y(prop)q(osal)d(recommends)h(that)g(suc)o(h)h(op)q(erations)g(should)f(b) q(e)h(considered)g(in)f(some)g(rein-)262 905 y(carnation)e(of)g(MPI.)262 1042 y Fh(1.3)64 b(Discussion)24 b(&)d(Notes)262 1133 y Fg(This)14 b(section)h(comprises)g(a)f(discussion)h(of)f(certain)h(asp)q(ects)h(of)e (this)h(prop)q(osal)f(follo)o(w)o(ed)262 1183 y(b)o(y)f(the)i(notes)f (referenced)j(in)c(the)i(detailed)e(prop)q(osal.)262 1299 y Fe(1.3.1)55 b(Discussion)262 1376 y Fg(W)m(e)15 b(can)h(dissect)i(the)e(prop) q(osal)g(in)o(to)f(t)o(w)o(o)h(parts:)22 b(an)16 b(SPMD)g(mo)q(del)f(core;)i (an)f(MIMD)262 1426 y(mo)q(del)f(annex.)27 b(In)17 b(this)g(discussion)g(the) h(dissection)f(is)g(exp)q(osed)h(and)e(the)i(conceptual)262 1475 y(foundation)d(of)i(eac)o(h)h(part)f(is)g(describ)q(ed.)30 b(The)17 b(discussion)h(also)e(presen)o(ts)j(argumen)o(ts)262 1525 y(for)14 b(and)g(against)g(the)i(MIMD)e(mo)q(del)f(annex,)i(and)f(to)h (some)e(exten)o(t)j(explores)f(the)h(con-)262 1575 y(squences)f(of)f(these)h (argumen)o(ts)e(for)h(MPI)g(in)f(a)h(wider)g(sense.)262 1683 y Ff(SPMD)h(mo)q(del)f(core)262 1760 y Fg(The)e(SPMD)g(mo)q(del)e(core)j(pro) o(vides)f(noncomm)o(unicativ)o(e)d(pro)q(cess)14 b(groupings)d(and)h(com-)262 1809 y(m)o(unication)j(con)o(texts)k(for)f(writers)g(of)g(SPMD)f(parallel)g (libraries.)30 b(It)18 b(is)f(in)o(tended)i(to)262 1859 y(pro)o(vide)11 b(expressiv)o(e)i(p)q(o)o(w)o(er)f(b)q(ey)o(ond)g(the)h(\\SPIMD")e(mo)q(del,) f(in)i(whic)o(h)f(pro)q(cess)j(execute)262 1909 y(in)f(a)h(stricly)g(SIMD)f (fashion.)324 1959 y(The)h(material)e(describing)i(pro)q(cesses)j(in)c Ff(Section)h(1.2.1.)19 b Fg(is)14 b(simpli\014ed:)324 2050 y Fa(\017)20 b Fg(pro)q(cesses)d(ha)o(v)o(e)d(iden)o(tical)f(instruction)h (blo)q(c)o(ks)g(and)g(di\013eren)o(t)g(data)g(blo)q(c)o(ks;)324 2133 y Fa(\017)20 b Fg(pro)q(cess)k(descriptor)f(transmission)d(and)h (registry)i(b)q(ecome)e(redundan)o(t)h(\(Note,)365 2183 y(ho)o(w)o(ev)o(er,)g (that)e(they)h(are)f(an)o(yw)o(a)o(y)f(redundan)o(t)i(as)f(con)o(text)h (descriptor)g(trans-)365 2233 y(mission)13 b(and)g(registry)i(coupled)g(with) e(con)o(text)i(descriptor)h(attribute)e(query)h(and)365 2283 y(group)f(descriptor)h(attribute)f(query)g(is)f(capable)h(of)f(pro)o(viding)f (access)k(to)d(all)g(pro-)365 2332 y(cess)j(descriptors\);)324 2415 y Fa(\017)k Fg(dynamic)12 b(pro)q(cess)k(mo)q(dels)d(are)h(not)g (considered.)967 2574 y(9)p eop %%Page: 10 11 10 10 bop 324 307 a Fg(The)14 b(material)e(describing)i(pro)q(cess)i (groupings)e(in)f Ff(Section)h(1.2.2.)19 b Fg(is)13 b(simpli\014ed:)324 399 y Fa(\017)20 b Fg(group)9 b(descriptor)i(transmission)d(and)h(registry)h (b)q(ecome)f(redundan)o(t)h(\(Note,)g(ho)o(w)o(ev)o(er,)365 448 y(that)17 b(they)h(are)f(an)o(yw)o(a)o(y)f(redundan)o(t)h(as)g(con)o (text)h(descriptor)g(transmission)e(and)365 498 y(registry)c(coupled)f(with)f (con)o(text)h(descriptor)h(attribute)f(query)h(capable)e(of)g(pro)o(vid-)365 548 y(ing)j(access)j(to)e(all)f(group)g(descriptors.\);)324 631 y Fa(\017)20 b Fg(the)d(o)o(wn)f(pro)q(cess)i(grouping)d(explicitly)g(b)q (ecomes)h(a)g(group)g(con)o(taining)f(all)g(pro-)365 681 y(cesses.)324 772 y(The)g(material)d(describing)j(comm)o(unication)c(con)o(texts)16 b(in)e Ff(Section)g(1.2.3.)21 b Fg(is)14 b(sim-)262 822 y(pli\014ed:)324 913 y Fa(\017)20 b Fg(con)o(text)15 b(descriptor)g(transmission)e(and)g (registry)i(b)q(ecome)f(unnecessary)m(.)324 1005 y(The)f(material)e (describing)j(p)q(oin)o(t-to-p)q(oin)o(t)d(comm)o(unication)f(in)i Ff(Section)h(1.2.4.)19 b Fg(is)262 1054 y(simpli\014ed:)324 1146 y Fa(\017)h Fg(the)15 b(op)q(en)f(con)o(text)h(form)d(b)q(ecomes)i (redundan)o(t;)324 1229 y Fa(\017)20 b Fg(uniform)15 b(in)o(tegration)g (\\Option)h(i")g(is)g(deleted,)i(and)e(\\Option)h(ii")e(loses)i(\\globs")365 1279 y(th)o(us)f(b)q(ecoming)e(simple)g(enough)h(that)h(\\Option)e(iii")g (need)i(not)f(b)q(e)h(further)h(con-)365 1328 y(sidered.)324 1420 y(The)c(material)e(describing)i(collectiv)o(e)g(comm)o(unicatio)o(n)d (in)j Ff(Section)f(1.2.5.)19 b Fg(is)12 b(sim-)262 1469 y(pli\014ed:)324 1561 y Fa(\017)20 b Fg(there)h(is)e(no)h(p)q(ossibilit)o(y)e(of)h(collectiv)o (e)g(comm)o(unication)d(op)q(erations)k(spanning)365 1611 y(more)13 b(than)h(con)o(text.)262 1719 y Ff(MIMD)i(mo)q(del)e(annex)262 1795 y Fg(The)d(MIMD)g(mo)q(del)f(annex)i(extends)g(and)f(mo)q(di\014es)g (the)g(SPMD)h(mo)q(del)d(core)k(to)e(pro)o(vide)262 1845 y(expressiv)o(e)g(p) q(o)o(w)o(er)f(for)f(MIMD)h(programs)e(whic)o(h)i(com)o(bine)e(\(coarse)j (grain\))e(function)h(and)262 1895 y(data)17 b(driv)o(en)g(parallelism.)27 b(The)18 b(MIMD)f(mo)q(del)f(annex)i(is)g(not)f(in)o(tended)h(to)g(pro)o (vide)262 1945 y(expressiv)o(e)d(p)q(o)o(w)o(er)g(to)f(\014ne)h(grained)f (function)g(driv)o(en)g(parallel)f(programs)g(|)h(it)f(is)i(con-)262 1994 y(jectured)j(that)g(message)f(passing)g(approac)o(hes)h(suc)o(h)h(as)e (MPI)h(are)g(not)f(suited)h(to)f(\014ne)262 2044 y(grained)c(function)g(driv) o(en)g(or)h(data)f(driv)o(en)h(programmi)o(ng.)h(The)f(annex)f(is)h(in)o (tended)g(to)262 2094 y(pro)o(vide)f(expresiv)o(e)i(p)q(o)o(w)o(er)f(for)g (the)h(\\MSPMD")e(mo)q(del,)f(whic)o(h)i(is)f(no)o(w)h(describ)q(ed.)324 2144 y(One)19 b(of)g(the)g(simplest)f(MIMD)h(mo)q(dels)f(is)h(the)g (\\host-no)q(de")g(mo)q(del,)g(famili)o(ar)d(in)262 2194 y(EXPRESS)f(and)f(P) m(ARMA)o(CS,)f(con)o(taining)h(t)o(w)o(o)g(functional)g(groups:)19 b(one)c(no)q(de)g(group)262 2243 y(\(SPMD)f(lik)o(e\))f(;)g(one)h(host)g (group)g(\(a)g(singleton\).)324 2293 y(The)d(\\parallel)e(clien)o(t-serv)o (er")j(mo)q(del,)e(in)g(whic)o(h)h(eac)o(h)g(of)f(the)i Fc(n)e Fg(clien)o(ts)h(is)g(comp)q(osed)262 2343 y(of)i(parallel)g(pro)q(cesses,)k (and)d(in)g(whic)o(h)g(the)h(serv)o(er)g(ma)o(y)e(also)g(b)q(e)i(comp)q(osed) f(of)g(parallel)262 2393 y(pro)q(cesses,)k(con)o(tains)d(1)10 b(+)h Fc(n)k Fg(functional)g(groups:)21 b Fc(n)15 b Fg(clien)o(t)h(groups)g (\(SPMD)f(lik)o(e\);)g(one)262 2443 y(serv)o(er)i(group)f(\(singleton,)f (SPMD)h(lik)o(e\).)23 b(The)17 b(\\host-no)q(de")f(mo)q(del)e(is)i(a)f(case)i (of)f(this)957 2574 y(10)p eop %%Page: 11 12 11 11 bop 262 307 a Fg(mo)q(del)12 b(in)i(whic)o(h)h(the)f(host)h(can)g(b)q (e)g(view)o(ed)f(as)h(a)f(singleton)g(clien)o(t)g(and)g(the)h(no)q(des)g(can) 262 357 y(b)q(e)f(view)o(ed)g(as)g(an)g(SPMD)g(lik)o(e)f(serv)o(er)i(\(or)f (vice)g(v)o(ersa!\).)324 407 y(The)f(\\parallel)e(mo)q(dule)g(graph")h(mo)q (del,)f(in)h(whic)o(h)g(eac)o(h)h(mo)q(dule)e(within)h(the)h(graph)262 457 y(ma)o(y)j(b)q(e)k(comp)q(osed)e(of)g(parallel)g(pro)q(cesses)j (\(singleton,)f(SPMD)e(lik)o(e\),)h(con)o(tains)g(an)o(y)262 506 y(n)o(um)o(b)q(er)c(of)g(functional)g(groups)h(with)g(arbitrarily)f (complex)g(relations.)24 b(The)16 b(\\parallel)262 556 y(clien)o(t-serv)o(er) e(mo)q(del")d(is)i(a)g(case)h(of)e(this)h(mo)q(del)e(in)i(whic)o(h)g(the)g (mo)q(dule)f(graph)h(con)o(tains)262 606 y(arcs)h(joining)e(the)j(serv)o(er)g (to)f(eac)o(h)g(clien)o(t.)324 656 y(The)19 b(MIMD)g(mo)q(del)e(annex)j(is)e (in)o(tended)i(to)f(pro)o(vide)g(expressiv)o(e)h(p)q(o)o(w)o(er)f(for)g(the) 262 706 y(\\parallel)14 b(mo)q(dule)h(graph")h(mo)q(del,)f(whic)o(h)h(I)g (refer)h(to)g(as)f(the)h(MPSMD)f(mo)q(del.)24 b(This)262 756 y(mo)q(del)14 b(requires)i(supp)q(ort)h(at)e(some)g(lev)o(el)g(as)h (commercial)d(and)i(mo)q(dular)f(applications)262 805 y(are)i(increasingly)g (mo)o(ving)d(in)j(to)f(parallel)g(computating.)23 b(The)16 b(debate)h(is)f(whether)i(or)262 855 y(not)c(message)g(passing)g(approac)o (hes)i(suc)o(h)f(as)g(MPI)f(\(whic)o(h)h(I)f(simply)f(refer)i(to)g(as)f (MPI\))262 905 y(should)f(pro)o(vide)h(for)f(this)h(mo)q(del.)324 955 y(The)d(negativ)o(e)f(argumen)o(t)g(is)g(that)h(suc)o(h)g(SPMD)g(mo)q (dules)f(should)g(b)q(e)h(con)o(trolled)g(and)262 1005 y(comm)o(uni)o(cate)i (with)h(one)h(another)g(as)g(\\parallel)f(pro)q(cesses")j(at)d(the)i (distributed)f(op)q(er-)262 1054 y(ating)f(system)g(lev)o(el.)21 b(The)15 b(argumen)o(t)f(has)h(some)f(app)q(eal)g(as)h(the)h(w)o(orld)e(of)g (distributed)262 1104 y(op)q(erating)j(systems)h(m)o(ust)f(deal)h(with)f (di\016cult)g(issues)i(suc)o(h)g(as)f(pro)q(cess)h(con)o(trol)f(and)262 1154 y(coherency)m(.)23 b(Av)o(oidance)16 b(of)f(duplication)f(in)h(MPI)g (allo)o(ws)g(MPI)g(to)h(fo)q(cus)f(on)h(pro)o(vision)262 1204 y(of)11 b(a)h(smaller)f(set)i(of)e(facilities)g(with)h(greater)i(emphasis)d (on)h(maxim)n(um)c(p)q(erformance)k(for)262 1254 y(data)h(driv)o(en)h(SPMD)g (parallel)f(prgrams.)324 1303 y(The)h(p)q(ositiv)o(e)g(argumen)o(t)f(is)h (that)g(comm)o(unicatio)o(ns)e(b)q(et)o(w)o(een)j(SPMD)f(mo)q(dules)f(re-)262 1353 y(quires)19 b(high)f(p)q(erformance)h(and)g(MPI)g(can)h(pro)o(vide)f (suc)o(h)g(p)q(erformance)g(with)g(tuned)262 1403 y(seman)o(tics)11 b(whic)o(h)g(exp)q(ect)j(the)e(user)h(to)f(deal)f(with)g(coherency)j(issues.) k(There)13 b(is)f(also)f(the)262 1453 y(argumen)o(t)g(that)h(MPI)g(is)g(able) g(to)g(deal)g(with)g(this)g(in)f(a)h(shorter)i(time)c(than)i(dev)o(elopmen)o (t)262 1503 y(and)k(standardisation)h(pro)q(cedures)j(for)c(distributed)i(op) q(erating)f(systems.)28 b(The)17 b(latter)262 1553 y(argumen)o(t)10 b(is)h(comparable)f(with)i(the)g(argumen)o(t)e(for)h(MPI)h(v)o(ersus)h (parallel)d(compilation.)262 1669 y Fe(1.3.2)55 b(Notes)312 1745 y Fg(1.)20 b(Descriptors)i(are)e(a)g(plen)o(tiful)f(but)i(b)q(ounded)f (resource.)39 b(They)21 b(are)f(expressed)365 1795 y(as)g(in)o(tegers)h(for)e (t)o(w)o(o)g(reasons.)37 b(Firstly)m(,)20 b(this)g(expression)h(facilitates)e (uniform)365 1845 y(binding)12 b(to)h(ANSI)h(C)f(and)g(F)m(ortran)g(77.)k (Secondly)m(,)c(there)h(is)f(a)g(later)g(con)o(v)o(enience)365 1895 y(in)e(ensuring)h(that)f(pro)q(cess)i(descriptors)g(in)d(particular)h (are)h(expressed)i(in)c(in)o(tegers,)365 1945 y(and)j(I)f(made)f(all)h(the)h (descriptors)h(are)f(in)o(tegers)g(for)f(consistency)m(.)19 b(In)12 b(practice)i(the)365 1994 y(descriptor)h(could)e(b)q(e)h(an)f(index)g (in)o(to)f(a)h(table)g(of)g(ob)r(jects)h(description)g(structures)365 2044 y(or)g(p)q(oin)o(ters)h(to)e(suc)o(h)i(structures.)312 2127 y(2.)20 b(It)10 b(is)g(p)q(ossible)g(to)g(allo)o(w)e(the)j(user)g(to)f (sa)o(v)o(e)g(user)h(de\014ned)g(attributes)g(in)e(descriptors,)365 2177 y(as)14 b(Little\014eld)f(has)g(suggested.)19 b(Suc)o(h)14 b(attributes)g(should)f(not)g(b)q(e)h(comm)o(unicated)365 2227 y(in)g(either)g(descriptor)i(transmission)c(or)i(the)h(descriptor)g(registry) m(.)312 2310 y(3.)20 b(The)c(pro)q(cess)h(descriptor)f(is)f(not)g(a)g(global) e(unique)i(pro)q(cess)i(iden)o(ti\014er)e(but)h(m)o(ust)365 2360 y(reference)i(suc)o(h)f(an)e(iden)o(ti\014er.)24 b(The)16 b(use)g(of)f(descriptors)i(hides)g(suc)o(h)f(iden)o(ti\014ers)365 2410 y(in)f(order)i(that)e(they)h(ma)o(y)e(b)q(e)i(implemen)o(tatio)o(n)d (dep)q(enden)o(t,)k(and)e(enables)i(more)957 2574 y(11)p eop %%Page: 12 13 12 12 bop 365 307 a Fg(e\016cien)o(t)14 b(access)i(to)d(additional)f(pro)q (cess)j(information)c(in)i(implemen)o(tations.)i(F)m(or)365 357 y(example,)d(the)h(pro)q(cess)i(iden)o(ti\014er)e(of)g(a)g(receiv)o(er)h (ma)o(y)d(not)i(b)q(e)h(su\016cien)o(t)f(to)g(route)365 407 y(a)f(message)f(to)h(the)g(receiv)o(er,)i(and)d(this)h(information)d(can)j(b) q(e)h(referenced)h(from)c(the)365 457 y(pro)q(cess)16 b(descriptor.)312 540 y(4.)k(The)h(group)e(descriptor)j(is)e(not)f(a)h(global)e(unique)i(group) g(iden)o(ti\014er)g(but)h(m)o(ust)365 589 y(reference)d(suc)o(h)f(an)e(iden)o (ti\014er.)24 b(The)16 b(use)g(of)f(descriptors)i(hides)g(suc)o(h)f(iden)o (ti\014ers)365 639 y(in)e(order)g(that)g(they)g(ma)o(y)e(b)q(e)i(implemen)o (tation)c(dep)q(endeen)o(t,)16 b(and)d(enables)i(more)365 689 y(e\016cien)o(t)i(access)h(to)f(additional)d(group)i(information)e(in)i (implem)o(en)o(tations.)23 b(F)m(or)365 739 y(example,)11 b(the)h(size)g(of)g (the)g(group)f(and)h(the)g(map)e(from)g(mem)o(b)q(er)g(ranks)i(to)g(pro)q (cess)365 789 y(descriptors)k(can)e(b)q(e)g(referenced)j(from)12 b(the)i(group)g(descriptor.)312 872 y(5.)20 b(The)15 b(con)o(text)g (descriptor)h(is)f(not)f(a)g(global)f(unique)i(con)o(text)g(iden)o(ti\014er)g (but)g(m)o(ust)365 922 y(reference)j(suc)o(h)f(an)e(iden)o(ti\014er.)24 b(The)16 b(use)g(of)f(descriptors)i(hides)g(suc)o(h)f(iden)o(ti\014ers)365 971 y(in)f(order)i(that)e(they)h(ma)o(y)e(b)q(e)i(implemen)o(tatio)o(n)d(dep) q(enden)o(t,)k(and)e(enables)i(more)365 1021 y(e\016cien)o(t)d(access)h(to)e (additional)e(con)o(text)j(information)d(in)h(implemen)o(tations.)j(F)m(or) 365 1071 y(example,)j(the)h(group)f(descriptor)i(of)d(the)i(frame)e(can)i(b)q (e)g(referenced)h(from)d(the)365 1121 y(con)o(text)e(descriptor.)312 1204 y(6.)20 b(The)h(prop)q(osal)e(do)q(es)i(not)f(prev)o(en)o(t)h(a)f(pro)q (cess)h(mo)q(del)e(whic)o(h)h(allo)o(ws)e(dynamic)365 1254 y(creation)13 b(and)g(termination)e(of)h(pro)q(cesses)j(ho)o(w)o(ev)o(er)e (it)f(do)q(es)i(not)f(fa)o(v)o(our)e(an)i(asyn-)365 1303 y(c)o(hronous)i(pro) q(cess)h(creation)f(mo)q(del)e(in)g(whic)o(h)i(singleton)e(pro)q(cesses)k (are)e(created)365 1353 y(and)h(terminated)g(in)g(an)g(unstructured)j (fashion.)25 b(The)16 b(prop)q(osal)g(do)q(es)h(fa)o(v)o(our)f(a)365 1403 y(mo)q(del)d(in)h(whic)o(h)g(blobs)g(of)f(pro)q(cesses)k(are)d(created)i (\(and)e(terminated\))g(in)g(a)g(con-)365 1453 y(certed)i(fashion,)d(and)h (in)g(whic)o(h)g(eac)o(h)h(group)f(so)g(created)i(is)e(assigned)h(a)f (di\013eren)o(t)365 1503 y(o)o(wn)g(pro)q(cess)j(grouping.)h(This)d(mo)q(del) e(do)q(es)i(not)f(tak)o(e)h(in)o(to)e(accoun)o(t)i(the)g(p)q(oten-)365 1553 y(tial)e(desire)j(to)e(expand)g(or)g(con)o(tract)h(an)f(existing)g(blob) g(of)f(pro)q(cesses)k(in)d(order)h(to)365 1602 y(tak)o(e)d(in)o(to)g(accoun)o (t)g(\(presumably)g(slo)o(wly\))f(time)f(v)n(arying)h(w)o(orloads.)17 b(The)c(author)365 1652 y(suggests)21 b(that)f(concerted)h(blob)e(expand)h (and)f(con)o(tract)h(op)q(erations)g(are)g(most)365 1702 y(suitable)14 b(for)g(this)g(purp)q(ose)h(than)e(async)o(hronous)i(spa)o(wn/kill)d(op)q (erations.)312 1785 y(7.)20 b(There)15 b(should)f(probably)f(also)g(b)q(e)h (a)g(registry)g(op)q(eration)g(whic)o(h)f(c)o(hec)o(ks)i(whether)365 1835 y(a)f(name)e(has)i(b)q(een)g(registered)i(without)d(the)h(p)q(ossibilit) o(y)e(of)h(blo)q(c)o(king)g(the)h(calling)365 1885 y(pro)q(cess)k (inde\014nitely)m(.)25 b(The)16 b(same)g(registry)h(could)f(b)q(e)h(used)g (for)f(pro)q(cess,)i(group)365 1934 y(and)c(con)o(text)h(descriptors.)312 2017 y(8.)20 b(In)f(the)g(static)g(pro)q(cess)h(mo)q(del)d(it)h(is)g(assumed) h(that)f(pro)q(cesses)j(are)e(created)h(in)365 2067 y(concert,)h(and)e(that)f (termination)f(of)h(one)h(pro)q(cess)h(implies)d(termination)g(of)h(all)365 2117 y(pro)q(cesses.)39 b(In)20 b(this)h(case)g(there)g(is)f(no)g(coherency)h (problem)e(asso)q(ciated)i(with)365 2167 y(pro)q(cesses)c(and)e(pro)q(cess)h (descriptors.)21 b(In)14 b(a)g(dynamic)f(pro)q(cess)j(mo)q(del)d(there)j(is)e (a)365 2217 y(coherency)f(problem)c(whic)o(h)i(pro)q(cesses)j(and)c(pro)q (cess)j(descriptors)g(since)e(a)g(pro)q(cess)365 2267 y(can)f(retain)g(the)g (descriptor)h(of)e(a)g(pro)q(cess)i(whic)o(h)f(has)g(b)q(een)g(deleted.)18 b(The)10 b(prop)q(osal)365 2316 y(exp)q(ects)16 b(the)f(user)g(to)e(ensure)j (coheren)o(t)f(usage.)312 2399 y(9.)20 b(Pro)q(cess)f(groupings)c(are)i (dynamic)d(in)i(the)h(sense)h(that)e(they)h(can)f(b)q(e)h(created)h(at)365 2449 y(an)o(y)12 b(time,)e(and)i(static)g(in)f(the)h(sense)i(that)e(the)g (mem)o(b)q(ership)e(is)i(constan)o(t)g(o)o(v)o(er)g(the)957 2574 y(12)p eop %%Page: 13 14 13 13 bop 365 307 a Fg(lifetime)11 b(of)g(the)j(pro)q(cess)g(grouping.)j(The) 12 b(author)h(suggests)h(concerted)g(group)e(ex-)365 357 y(pand)f(and)f(con)o (tract)i(op)q(erations)f(are)g(more)e(generally)i(useful)f(than)h(async)o (hronous)365 407 y(join/lea)o(v)o(e)i(op)q(erations.)291 490 y(10.)20 b(MPI)15 b(has)f(discussed)j(the)e(concept)g(of)f(the)h(\\all")e (group)h(whic)o(h)g(con)o(tains)h(all)e(pro-)365 540 y(cesses.)35 b(The)19 b(\\o)o(wn")e(group)i(concept)h(is)e(in)o(tended)h(to)g(b)q(e)g(a)f (generalisation)g(of)365 589 y(the)h(\\all")d(group)h(concept)i(whic)o(h)f (is)g(expressiv)o(e)h(for)e(programs)g(including)g(and)365 639 y(b)q(ey)o(ond)g(the)g(SPMD)f(mo)q(del.)24 b(Pro)q(cesses)19 b(are)d(created)i(in)e(\\blobs",)g(where)h(eac)o(h)365 689 y(mem)o(b)q(er)g(of)g(a)h(blob)f(is)h(a)f(mem)o(b)q(er)g(of)g(the)i(same)e(o) o(wn)g(pro)q(cess)j(grouping,)e(and)365 739 y(di\013eren)o(t)e(blobs)e(ha)o (v)o(e)g(di\013eren)o(t)h(o)o(wn)f(pro)q(cess)i(groupings.)j(An)14 b(SPMD)g(program)365 789 y(is)f(a)h(single)e(blob.)18 b(A)13 b(host-no)q(de)h(program)e(comp)q(oses)h(t)o(w)o(o)g(blobs,)g(the)g(no)q(de)h (blob)365 839 y(and)f(the)h(host)g(blob)f(\(whic)o(h)g(is)g(a)g(singleton\).) 18 b(There)c(is)f(a)g(sense)i(in)e(whic)o(h)g(a)g(blob)365 888 y(has)h(a)g(lo)q(cally)e(SPMD)i(view.)291 971 y(11.)20 b(This)14 b(pro)q(cedure)i(lo)q(oks)d(lik)o(e)g(a)h(pro)q(cess)i(descriptor)f (attribute)f(query)m(.)291 1054 y(12.)20 b(Dynamic)8 b(pro)q(cesses)j(in)o (tro)q(duce)f(a)f(group)h(coherency)h(problem)d(as)h(a)g(group)g(descriptor) 365 1104 y(can)18 b(con)o(tain)f(the)h(pro)q(cess)h(descriptor)f(of)f(a)g (pro)q(cess)i(whic)o(h)e(has)h(b)q(een)g(deleted.)365 1154 y(Dynamic)12 b(pro)q(cesses)17 b(p)q(oten)o(tially)c(in)o(tro)q(duce)i(a)e (group)h(coherency)i(problem)d(as)h(a)365 1204 y(group)e(descriptor)h(can)f (con)o(tain)g(the)g(pro)q(cess)i(descriptor)f(of)e(a)h(pro)q(cess)i(whic)o(h) d(has)365 1254 y(b)q(een)h(deleted.)17 b(Group)10 b(transmission)f(in)o(tro)q (duces)i(a)f(group)g(and)g(group)g(descriptor)365 1303 y(coherenncy)21 b(problem)d(since)i(a)f(pro)q(cess)j(can)d(retain)h(a)f(group)g(descriptor)h (of)f(a)365 1353 y(group)c(whic)o(h)f(has)h(b)q(een)g(iden)o(ti\014ed)g (group.)20 b(The)14 b(prop)q(osal)h(exp)q(ects)h(the)f(user)h(to)365 1403 y(ensure)g(coheren)o(t)f(usage.)291 1486 y(13.)20 b(The)c(prop)q(osal)f (did)f(not)h(include)g(a)g(function)g(to)g(con)o(v)o(ert)h(a)e Fd(\(gd,)21 b(pd\))15 b Fg(pair)f(in)o(to)365 1536 y(a)i(rank.)22 b(It)16 b(is)f(suggested)i(that)f(this)f(in)o(v)o(erse)i(map)d(is)h(allo)o(w) o(ed)f(to)i(b)q(e)g(arbitrarily)365 1586 y(slo)o(w.)291 1669 y(14.)k(Marc)15 b(Snir)e(has)h(describ)q(ed)i(a)d(metho)q(d)g(b)o(y)h(whic)o (h)g(global)e(unique)h(group)h(iden)o(ti\014-)365 1719 y(ers)j(can)e(b)q(e)h (generated)h(without)e(use)h(of)f(shared)h(global)e(data.)22 b(The)16 b(description)365 1768 y(of)i(con)o(text)h(creation)g(an)o (ticipates)g(a)f(similar)e(metho)q(d)i(for)g(global)f(unique)i(con-)365 1818 y(text)h(iden)o(ti\014er)g(generation.)35 b(Ho)o(w)o(ev)o(er)20 b(the)g(sync)o(hronisation)f(requiremen)o(t)g(of)365 1868 y(this)e(metho)q(d) e(mak)o(es)h(it)g(unnecessary)i(for)e(con)o(text)h(creation.)26 b(Sync)o(hronisation)365 1918 y(at)11 b(creation)g(of)f(con)o(text)i(within)e (a)g(pro)q(cess)j(grouping)d(frame)f(implies)g(that)i(all)f(pro-)365 1968 y(cesses)19 b(within)d(the)h(frame)e(p)q(erform)g(the)i(same)f(con)o (text)h(creations)g(in)f(the)g(same)365 2017 y(sequence.)39 b(Therefore)22 b(the)f(com)o(bination)c(of)j(global)e(unique)i(frame)f(iden)o (ti\014er)365 2067 y(and)e(con)o(text)h(creation)f(sequence)i(n)o(um)o(b)q (er)e(pro)o(vides)g(a)g(global)e(unique)i(con)o(text)365 2117 y(iden)o(ti\014er)e(without)e(comm)o(unication.)291 2200 y(15.)20 b(This)14 b(pro)q(cedure)i(lo)q(oks)d(lik)o(e)g(a)h(group)g(descriptor)h (attribute)f(query)m(.)291 2283 y(16.)20 b(The)13 b(reten)o(tion)h(of)e(a)g (reference)k(to)c(a)h(group)f(descriptor)i(b)o(y)f(a)f(con)o(text)i(in)o(tro) q(duces)365 2333 y(the)i(p)q(oten)o(tial)e(for)h(a)f(con)o(text)i(and)f(con)o (text)g(descriptor)h(coherency)h(problem,)d(as)365 2383 y(a)19 b(con)o(text)h(could)e(con)o(tain)h(a)f(reference)k(to)c(the)i(group)f (descriptor)h(of)e(a)h(group)365 2433 y(whic)o(h)c(has)h(b)q(een)g(deleted.) 24 b(This)15 b(could)g(b)q(e)h(circum)o(v)o(en)o(ted)f(b)o(y)g(demanding)f (that)957 2574 y(13)p eop %%Page: 14 15 14 14 bop 365 307 a Fg(all)11 b(suc)o(h)i(con)o(texts)h(are)e(deleted)i(b)q (efore)f(a)f(group)g(is)g(deleted.)19 b(Con)o(text)12 b(descriptor)365 357 y(transmission)17 b(in)o(tro)q(duces)i(a)e(coherency)j(problem)c(since)j (a)e(pro)q(cess)j(can)d(retain)365 407 y(the)d(descriptor)h(of)e(a)g(con)o (text)i(whic)o(h)e(has)h(b)q(een)g(deleted.)19 b(The)14 b(prop)q(osal)g(exp)q (ects)365 457 y(the)h(user)g(to)f(ensure)h(coheren)o(t)g(usage.)291 540 y(17.)20 b(This)14 b(is)f(somewhat)f(lik)o(e)h(and)g(P)m(ARMA)o(CS)g(and) g(PVM)g(3.)18 b(It)13 b(is)g(general)h(purp)q(ose,)365 589 y(but)f(not)f(particularly)f(expressiv)o(e.)19 b(It)13 b(do)q(es)g(not)f(pro) o(vide)g(facilities)f(for)h(writers)h(of)365 639 y(parallel)g(libraries.)291 722 y(18.)20 b(This)15 b(is)g(somewhat)f(lik)o(e)g(ZIPCODE)i(and)e(VENUS.)h (It)g(is)g(expressiv)o(e)i(in)d(certain)365 772 y(SPMD)g(programs)f(where)i (noncomm)o(uni)o(cativ)o(e)c(data)j(driv)o(en)f(parallel)g(computa-)365 822 y(tions)k(can)g(b)q(e)g(p)q(erformed)g(concurren)o(tly)m(.)27 b(It)17 b(pro)o(vides)g(facilities)f(for)g(writers)i(of)365 872 y(SPMD)c(lik)o(e)f(parallel)g(libraries.)291 955 y(19.)20 b(This)f(is)f(somewhat)g(lik)o(e)g(CHIMP)h(and)g(PVM)g(2.)32 b(It)19 b(is)f(expressiv)o(e)j(in)d(certain)365 1005 y(MIMD)13 b(programs)e(where)i(comm)o(unicativ)o(e)d(data)i(driv)o(en)h(parallel)e (computations)365 1054 y(can)21 b(b)q(e)g(p)q(erformed)g(concurren)o(tly)m(.) 38 b(It)21 b(pro)o(vides)g(facilities)e(for)i(MSPMD)f(lik)o(e)365 1104 y(parallel)13 b(libraries.)262 1241 y Fh(1.4)64 b(Conclusion)262 1332 y Fg(This)17 b(c)o(hapter)h(has)g(presen)o(ted)i(and)d(discussed)i(a)e (prop)q(osal)g(for)h(comm)o(uni)o(cation)d(con-)262 1382 y(texts)k(within)g (MPI.)f(In)h(the)h(prop)q(osal)f(pro)q(cess)h(groupings)f(app)q(eared)h(as)f (frames)f(for)262 1432 y(the)d(creation)g(of)f(comm)o(unication)e(con)o (texts,)j(and)g(comm)o(unicatio)o(n)d(con)o(texts)k(retained)262 1482 y(certain)e(prop)q(erties)h(of)f(the)g(frames)f(used)i(in)e(their)i (creation.)324 1532 y(The)f(author)g(strongly)g(recommends)e(this)i(prop)q (osal)g(to)g(the)g(committee.)957 2574 y(14)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:58:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00429; Sun, 21 Mar 93 14:58:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25091; Sun, 21 Mar 93 14:58:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:58:00 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25083; Sun, 21 Mar 93 14:57:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03816; Sun, 21 Mar 93 13:52:35 CST Date: Sun, 21 Mar 93 13:52:35 CST From: Tony Skjellum Message-Id: <9303211952.AA03816@Aurora.CS.MsState.Edu> To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu Subject: Re: Proposal II' - LaTeX Your subcommittee chair has been telling you to call this Proposal VI for the last half hour. Please listen. Drop the Editorial note, forthwith, after naming it proposal VI. Please don't be write-only. - Tony From owner-mpi-context@CS.UTK.EDU Sun Mar 21 15:07:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00520; Sun, 21 Mar 93 15:07:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25403; Sun, 21 Mar 93 15:07:05 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 15:07:04 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25395; Sun, 21 Mar 93 15:07:02 -0500 Date: Sun, 21 Mar 93 20:07:00 GMT Message-Id: <13161.9303212007@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: clarification, i hope :-) To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Dear Tony I am trying to send you a "clean table" as requested. Proposal Number Notes on Proposal --------------- ----------------- I++ Defunct number, defunct proposal. Was circulated as Proposal I. II' Defunct number, live proposal. To be circulated as Proposal VI. I Marc Snir's proposal which appeared in point-to-point working document. I think we would do ourselves harm to just ignore this. VI My proposal which I am personally happy with and am also happy to move along towards the other proposals prepared or being prepared by Rik and Tony. Was called Proposal II'. III/IV The proposal which Tony is working on. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 17:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01141; Sun, 21 Mar 93 17:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28876; Sun, 21 Mar 93 17:14:29 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 17:14:28 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28868; Sun, 21 Mar 93 17:14:22 -0500 Date: Sun, 21 Mar 93 22:14:17 GMT Message-Id: <13284.9303212214@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal VI (was Proposal II') - LaTeX To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Tony Thanks for the phone call. I have stopped calling this thing II' and it is now Proposal VI, as you ask, and of course you are right that we should avoid confusion. Here is the LaTeX for Proposal VI. I have propogated the comments that you gave me for Proposal I++ (now defunct name and defunct proposal) into this document as I believe appropriate. Sometimes this has required replicating your comment at several points in the document, and I have tried to give the same second order comments in each case. Perhaps it will be as well to read the whole thing before making further comments, rather than "real time" comments (which I often do, bad practice). PostScript will follow. Best Wishes Lyndon ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal VI\\MPI Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal VI" % Written by Lyndon J. Clarke % March 1993 % \newcommand{\LabelNote}[1]{\label{vi:note:#1}} \newcommand{\SeeReferNote}[1]{{\bf See~Note~\ref{vi:note:#1}.}} \newcommand{\LabelSection}[1]{\label{vi:sect:#1}} \newcommand{\ReferSection}[1]{{\bf Section~\ref{vi:sect:#1}.}} \chapter{Proposal VI} {\it Editorial Note: This is not Proposal II as identified at the post-meet lunch after the February meeting in Dallas. Rik~Littlefied came on board the proposal writing process, and decided to write a different proposal which appears as Proposal V. There is no longer a proposal which contains static contexts as discussed at the post-meet lunch.} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction} This chapter proposes that communication contexts and process groupings within MPI appear as related concepts. In particular process groupings appear as ``frames'' which are used in the creation of communication contexts. Communications contexts retain a reference to, and inherit properties of, their process grouping frames. This reflects the observation that an invocation of a module in a parallel program typically operates within one or more groups of processes, and as such any communication contexts associated with invocations of modules also bind certain semantics of process groupings. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of MPI, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language for only purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The notes assumes knowledge of the proposal text and are therefore best examined after familiarisation with that text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the text of the proposal. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal} \LabelSection{proposal} This section presents the detailed proposal, discussing in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes} \LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and MPI does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{process:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of MPI services. MPI provides a procedure which returns the own descriptor of the calling process. For example, {\tt pd = mpi\_own\_pd()}. \subsubsection*{Process Creation and Destruction} This proposal makes no statements regarding creation and destruction of processes. \SeeReferNote{dynamic:processes} \subsubsection*{Descriptor Transmission} The value of a process descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %[Lyndon] % Since it is an integer one cannot prevent the user from passing it. % The discussion notes should help answer why is is an integer. More % here. % We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. % Because rank will be an integer, then pd must also be an integer, % so I write that all descriptors are integers for consistency beteen % the different descriptors. MPI provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_PD\_TYPE}. MPI provides a process descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %[Lyndon] % I don't understand what you don't get. A registry service is just % an easier (nonscalable, okay) way for concurrent entities to hook up % with one another, rather than having some common ancestor send descriptors % around in messages. % In fact I don't really need a group or process registry service, as mentioned % in the discussion section (yes, I know, not well presented), but % I do need a context registry service, and I'm being consistent % between different kinds of descriptors again. % Suggestive syntax for a registry service is pretty boring, so % I skipped it. Note that receipt of a process descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_pd(pd)}. The user is not allowed to invalidate the process descriptor of the calling process. \SeeReferNote{process:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %[Lyndon] % Acquisition of a descriptor requires an allocator, and consumes % resource. In such cases it is good practice to provide a % deallocator which frees resource. % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %[Lyndon] % Unless I am missing the additional point in this comment, I can't % see the problem. Probably poor presentation of Proposal I++ :-) \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding processor descriptor attributes. \subsection{Process Groupings} \LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to?) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. \SeeReferNote{dynamic:groups} The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to the descriptor of the $N$ processes composing the grouping. The structure of a process grouping is defined by a process grouping topology and this proposal makes no further statements regarding such structures. %[Tony] % It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % %[Lyndon] % I am deliberately making weak statements about topology while % acknolwedging the the process topology subcommittee. \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{group:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the process grouping until the termination of MPI services. \SeeReferNote{group:blobs} MPI provides a procedure which returns the descriptor of the home group of the calling process. For example, {\tt gd = mpi\_home\_gd()}. \SeeReferNote{own:group} \subsubsection*{Group Creation and Deletion} MPI provides a procedure which allows users to dynamically create one or more groups which are subsets of existing groups. For example, {\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} partition an existing group {\tt gda} into one or more distinct subsets. This procedure is called by and synchronises all members of {\tt gda}. MPI provides a procedure which allows users to dynamically create one group by explicit definition of its membership as a list of process descriptors. For example, {\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. %[Tony] % loosely synchronous % %[Lyndon] % Do we mean the same thing? the constructors require each member % if the object under construction to participate in the construction, % and return a descriptor for the constructed operation. Therefore % no member can terminate construction before all other members have % commenced, at least. MPI provides a procedure which allows users to delete a created group. This procedure accepts the descriptor of a group which was created by the calling process and destroys the identified group. For example, {\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. MPI provides additional procedure which allow users to construct process groupings which have a process grouping topology. \subsubsection*{Descriptor Transmission} The value of a group descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %[Lyndon] % Since it is an integer one cannot prevent the user from passing it. % The discussion notes should help answer why is is an integer. More % here. % We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. % Because rank will be an integer, then pd must also be an integer, % so I write that all descriptors are integers for consistency beteen % the different descriptors. MPI provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_GD\_TYPE}. MPI provides a group descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %[Lyndon] % I don't understand what you don't get. A registry service is just % an easier (nonscalable, okay) way for concurrent entities to hook up % with one another, rather than having some common ancestor send descriptors % around in messages. % In fact I don't really need a group or process registry service, as mentioned % in the discussion section (yes, I know, not well presented), but % I do need a context registry service, and I'm being consistent % between different kinds of descriptors again. % Suggestive syntax for a registry service is pretty boring, so % I skipped it. Note that receipt of a group descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_gd(gd)}. The user is not allowed to invalidate the own group descriptor of the process or the group descriptor of any group created by the calling process. \SeeReferNote{group:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %[Lyndon] % Acquisition of a descriptor requires an allocator, and consumes % resource. In such cases it is good practice to provide a % deallocator which frees resource. % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %[Lyndon] % Unless I am missing the additional point in this comment, I can't % see the problem. Probably poor presentation of Proposal I++ :-) % \subsubsection*{Descriptor Attributes} MPI provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identifier group. For example, {\tt rank = mpi\_group\_rank(gd)}. MPI provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, {\tt size = mpi\_group\_size(gd)}. MPI provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, {\tt pd = mpi\_group\_pd(gd, rank)}. \SeeReferNote{pd:to:rank} MPI provides additional procedures which allow users to determine the process grouping topology attributes. %[Tony] It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % % %[Lyndon] % I am deliberately making weak statements about topology while % acknolwedging the the process topology subcommittee. \subsection{Communication Contexts} \LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping is used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{context:identifiers} \SeeReferNote{descriptor:cache} The creation of MPI process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains its own context until MPI process grouping deletion. MPI provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, {\tt cd = mpi\_own\_cd(gd)}. \SeeReferNote{own:context} \subsubsection*{Context Creation and Deletion} MPI provides a procedure which allows users to dynamically create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, {\tt cd = mpi\_context\_create(gd)}. This procedure is called by and synchronises all members of {\tt gd}. \SeeReferNote{dynamic:contexts} %[Tony] % loosely synchronous % %[Lyndon] % Do we mean the same thing? the constructors require each member % if the object under construction to participate in the construction, % and return a descriptor for the constructed operation. Therefore % no member can terminate construction before all other members have % commenced, at least MPI provides a procedure which allows users to destroy created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes that context identifier. For example, {\tt mpi\_context\_delete(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. \subsubsection*{Descriptor Transmission} The value of a context descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %[Lyndon] % Since it is an integer one cannot prevent the user from passing it. % The discussion notes should help answer why is is an integer. More % here. % We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. % Because rank will be an integer, then pd must also be an integer, % so I write that all descriptors are integers for consistency beteen % the different descriptors. MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_CD\_TYPE}. %[Tony] % I don't get all of this. Why? % %[Lyndon] % I don't understand what you don't get. A registry service is just % an easier (nonscalable, okay) way for concurrent entities to hook up % with one another, rather than having some common ancestor send descriptors % around in messages. % In fact I don't really need a group or process registry service, as mentioned % in the discussion section (yes, I know, not well presented), but % I do need a context registry service, and I'm being consistent % between different kinds of descriptors again. % Suggestive syntax for a registry service is pretty boring, so % I skipped it. MPI provides a context descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a context descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_cd(cd)}. The user is not allowed to invalidate the own context descriptor of a group or the context descriptor of any context created by the calling process. \SeeReferNote{context:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %[Lyndon] % Acquisition of a descriptor requires an allocator, and consumes % resource. In such cases it is good practice to provide a % deallocator which frees resource. % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %[Lyndon] % Unless I am missing the additional point in this comment, I can't % see the problem. Probably poor presentation of Proposal I++ :-) % \subsubsection*{Descriptor Attributes} MPI provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, {\tt gd = mpi\_context\_gd(cd)}. \subsection{Point-to-Point Communication} \LabelSection{point2point} This proposal recommends three forms for MPI point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a Send operation of form X cannot match with a receive operation of form Y, which requires that the form is embedded into the message envelope. The three forms are described followed by considerations of uniform integration of these forms in the point-to-point communication section of MPI. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. \SeeReferNote{null:context} Message selection and addressing are expressed by {\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. Send supplies the {\tt pd} of the receiver. Receive supplies the {\tt pd} of the sender. Receive can wildcard on {\tt pd} by supplying the value of a named constant process descirptor, e.g. {\tt MPI\_PD\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. \SeeReferNote{closed:context} Message selection and addressing are expressed by {\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. Send supplies the {\tt cd} of the receiver and sender, and the {\tt rank} of the receiver. Receive supplies the {\tt cd} of the sender and receiver, and the rank of the sender. The {\tt (cd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt cd}. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. \SeeReferNote{open:context} Message selection and addressing are expressed by {\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. Send supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. Receive supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The {\tt (rcd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt lcd}. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD}, in which case Receive {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point section of MPI by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with the form. This is at the expense of multiplying the number of Send and Receive procedures by a factor of three, and some difficulty in details of the current point-to-point working document which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options each have advantages and disadvantages. \paragraph*{Option i:} The framework of the open context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii:} The framework of the closed context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as {\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the {\tt (lcd, rcd)} pair at the expense of introducing some trickery. We can define a mechanism which ``globs'' together these two fields onto in an identifier which Send and Receive can distinguish from a context identifier and treat as the shorthand notation. Then we should also define a mechanism by which a receiver which has completed a Receive with wildcard on {\tt rcd} is able to determine the valid context descriptor of the sender. This is a inconvenient. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % % [Lyndon] % I dislike this also. Of course it is all to get around extending % the syntax, it stinks of that. \paragraph*{Option iii:} The framework of the null context form is adopted and extended. The null context form is expressed as {\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is clumsy. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % % % [Lyndon] % I dislike this also. Of course it is all to get around extending % the syntax, it stinks of that. \subsection{Collective Communication} \LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context and frame in which they are to operate. MPI does plan to describe symmetric collective communication operations. It is not possible to determine whether the proposal is sufficient to allow implementation of the collective communication section of MPI in terms of the point-to-point section of MPI without loss of generality, since the collective operations are not yet defined. %[Tony] Check, but it is desirable that they be so writable, so we will % have to watch. % Assymetric collective communication operations, especially those in which the sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a ``glob'' form) which identify the contexts and frames in which they are to operate. MPI does not plan to describe assymetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model and comprise communicative funtionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of MPI. % % END "Proposal" %----------------------------------------------------------------------% % [Tony] % So, I gather that a set of groups is passable to a collcomm, % and a pair is passable to a pt2pt. That is neat, but it should % still be a separate data structure, with separate calls than % the intra-group version (at least for the pt2pt calls). % % [Lyndon] % Currently, I am only thinking of passing either singlets or duplets % to point-to-point and collective. % I was trying to avoid separate - extra - calls to point-to-point % because that is already very large and there is a swell of concern % about the size thereof. %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion} \LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents arguments for and against the MIMD model annex, and to some extent explores the consquences of these arguments for MPI in a wider sense. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model, in which process execute in a stricly SIMD fashion. The material describing processes in \ReferSection{processes} is simplified: \begin{itemize} \item processes have identical instruction blocks and different data blocks; \item process descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query and group descriptor attribute query is capable of providing access to all process descriptors); \item dynamic process models are not considered. \end{itemize} The material describing process groupings in \ReferSection{groupings} is simplified: \begin{itemize} \item group descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query capable of providing access to all group descriptors.); \item the own process grouping explicitly becomes a group containing all processes. \end{itemize} The material describing communication contexts in \ReferSection{contexts} is simplified: \begin{itemize} \item context descriptor transmission and registry become unnecessary. \end{itemize} The material describing point-to-point communication in \ReferSection{point2point} is simplified: \begin{itemize} \item the open context form becomes redundant; \item uniform integration ``Option i'' is deleted, and ``Option ii'' loses ``globs'' thus becoming simple enough that ``Option iii'' need not be further considered. \end{itemize} The material describing collective communication in \ReferSection{collective} is simplified: \begin{itemize} \item there is no possibility of collective communication operations spanning more than context. \end{itemize} \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as MPI are not suited to fine grained function driven or data driven programming. The annex is intended to provide expresive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in EXPRESS and PARMACS, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or vice versa!). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computating. The debate is whether or not message passing approaches such as MPI (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD parallel prgrams. The positive argument is that communications between SPMD modules requires high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development and standardisation procedures for distributed operating systems. The latter argument is comparable with the argument for MPI versus parallel compilation. \subsection{Notes} \LabelSection{notes} \begin{enumerate} \item\LabelNote{integer:descriptors} Descriptors are a plentiful but bounded resource. They are expressed as integers for two reasons. Firstly, this expression facilitates uniform binding to ANSI~C and Fortran~77. Secondly, there is a later convenience in ensuring that process descriptors in particular are expressed in integers, and I made all the descriptors are integers for consistency. In practice the descriptor could be an index into a table of objects description structures or pointers to such structures. \item\LabelNote{descriptor:cache} It is possible to allow the user to save user defined attributes in descriptors, as Littlefield has suggested. Such attributes should not be communicated in either descriptor transmission or the descriptor registry. \item\LabelNote{process:identifiers} The process descriptor is not a global unique process identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional process information in implementations. For example, the process identifier of a receiver may not be sufficient to route a message to the receiver, and this information can be referenced from the process descriptor. \item\LabelNote{group:identifiers} The group descriptor is not a global unique group identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependeent, and enables more efficient access to additional group information in implementations. For example, the size of the group and the map from member ranks to process descriptors can be referenced from the group descriptor. \item\LabelNote{context:identifiers} The context descriptor is not a global unique context identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional context information in implementations. For example, the group descriptor of the frame can be referenced from the context descriptor. \item\LabelNote{dynamic:processes} The proposal does not prevent a process model which allows dynamic creation and termination of processes however it does not favour an asynchronous process creation model in which singleton processes are created and terminated in an unstructured fashion. The proposal does favour a model in which blobs of processes are created (and terminated) in a concerted fashion, and in which each group so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying worloads. The author suggests that concerted blob expand and contract operations are most suitable for this purpose than asynchronous spawn/kill operations. \item\LabelNote{registry:check} There should probably also be a registry operation which checks whether a name has been registered without the possibility of blocking the calling process indefinitely. The same registry could be used for process, group and context descriptors. \item\LabelNote{process:coherency} In the static process model it is assumed that processes are created in concert, and that termination of one process implies termination of all processes. In this case there is no coherency problem associated with processes and process descriptors. In a dynamic process model there is a coherency problem which processes and process descriptors since a process can retain the descriptor of a process which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{dynamic:groups} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The author suggests concerted group expand and contract operations are more generally useful than asynchronous join/leave operations. \item\LabelNote{group:blobs} MPI has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is intended to be a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and different blobs have different own process groupings. An SPMD program is a single blob. A host-node program composes two blobs, the node blob and the host blob (which is a singleton). There is a sense in which a blob has a locally SPMD view. \item\LabelNote{own:group} This procedure looks like a process descriptor attribute query. \item\LabelNote{group:coherency} Dynamic processes introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Dynamic processes potentially introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Group transmission introduces a group and group descriptor coherenncy problem since a process can retain a group descriptor of a group which has been identified group. The proposal expects the user to ensure coherent usage. \item\LabelNote{pd:to:rank} The proposal did not include a function to convert a {\tt (gd, pd)} pair into a rank. It is suggested that this inverse map is allowed to be arbitrarily slow. \item\LabelNote{dynamic:contexts} Marc Snir has described a method by which global unique group identifiers can be generated without use of shared global data. The description of context creation anticipates a similar method for global unique context identifier generation. However the synchronisation requirement of this method makes it unnecessary for context creation. Synchronisation at creation of context within a process grouping frame implies that all processes within the frame perform the same context creations in the same sequence. Therefore the combination of global unique frame identifier and context creation sequence number provides a global unique context identifier without communication. \item\LabelNote{own:context} This procedure looks like a group descriptor attribute query. \item\LabelNote{context:coherency} The retention of a reference to a group descriptor by a context introduces the potential for a context and context descriptor coherency problem, as a context could contain a reference to the group descriptor of a group which has been deleted. This could be circumvented by demanding that all such contexts are deleted before a group is deleted. Context descriptor transmission introduces a coherency problem since a process can retain the descriptor of a context which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{null:context} This is somewhat like and PARMACS and PVM~3. It is general purpose, but not particularly expressive. It does not provide facilities for writers of parallel libraries. \item\LabelNote{open:context} This is somewhat like ZIPCODE and VENUS. It is expressive in certain SPMD programs where noncommunicative data driven parallel computations can be performed concurrently. It provides facilities for writers of SPMD like parallel libraries. \item\LabelNote{closed:context} This is somewhat like CHIMP and PVM~2. It is expressive in certain MIMD programs where communicative data driven parallel computations can be performed concurrently. It provides facilities for MSPMD like parallel libraries. \end{enumerate} % % END "Discussion & Notes" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Conclusion" % \section{Conclusion} This chapter has presented and discussed a proposal for communication contexts within MPI. In the proposal process groupings appeared as frames for the creation of communication contexts, and communication contexts retained certain properties of the frames used in their creation. The author strongly recommends this proposal to the committee. % % END "Conclusion" %----------------------------------------------------------------------% % % END "Proposal VI" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 18:51:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01899; Sun, 21 Mar 93 18:51:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01788; Sun, 21 Mar 93 18:50:17 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 18:50:14 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01768; Sun, 21 Mar 93 18:50:09 -0500 Date: Sun, 21 Mar 93 23:50:04 GMT Message-Id: <13403.9303212350@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: V commenting To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Tony I ran over schedule before finishing. Comments are marked in the Lyndon favoured style % %[name] % text % Comments to comments are nested %% %[name] % text %%[name2] %%text2 %% % There are comments to comments to comments down there as well %%% Good luck! Lyndon ---------------------------------------------------------------------- % [Lyndon] General points % -------------- % % 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am % offering if you wish) should attempt to construct a more transparent % presentation before circulation to whole committee, for the % convenience of committee members. %%[Tony] %% I felt that things appear twice, because of summary (good) %% and because of implementation notes at end (confusing) %% % 2) I'm not a fan of much of this proposal, although I do indeed like % some of the ideas which it introduces. [On the other hand, I'm not a % great fan of all of the proposal which I wrote. I shall mail self % criticism of my proposal, and may have to write amended or alternative % proposal :-)] %%[Tony] %% Please be more specific. I am having a hard time understanding %% why you really don't like it, Lyndon. If the process model %% were a little less static, and servers were permitted (though %% hopefully bounded in cost), I think we would have an excellent %% proposal. %% %%%[Lyndon] %%% I would have thought that the points below make my major %%% objections perfectly clear. Perhaps not. Here they are: %%% a) Paired-exact-match stuff %%% b) Translation of (group,rank) into TID all over the place % % 3) I really like the way in which groups are something like ``frames'' % in which contexts are created. This is conceptually much neater than % duplication of groups. %%[Tony] %% In practice, group subsetting will require groups to be copied, %% otherwise, subgroups will unfairly be penalized by the size %% of their ancestor. %% %%%[Lyndon] %%% I am anticipating that when one or more groups are created by %%% subsetting that, for example if the parent were described by %%% a proces list, then the children will be described by process %%% lists which are distinct sublists of the parent. So each element %%% of the parent list gets copied, exactly once. %%% The difficulty I have is that if a group were to be expanded %%% or contracted, then the ``duplicates'' thereof would no longer %%% be duplicates. Saying that duplicate creates a group bu retains %%% the process list of the old group is conceptually muddy since %%% the new group is a reference to a group, whereas the old group %%% or an even older group must be an actual group. Yuk! Now, if %%% we introduce the concept of a reference to an actual group, %%% and say this reference is unqieuly identified, then is is %%% conceptually sound and this object we describe really is a context. % % 4) I like the idea of pushing information into the group structure. I % have a few qualms with the proposed details --- see specific points. %%[Tony] %% I have more confidence about this idea, and could demonstrate %% by June/July time-frame in Zipcode. %% % % 5) See ``Writing a server in the point-to-point layer of MPI in four % easy steps'' at the foot of the message. %%[Tony] %% This seems like a nice thing. %% %%%[Lyndon] %%% You are too kind :-) \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %%[Tony] %% Alternatives, Lyndon? %% %%%[Lyndon] %%% I don't suppose for one minute that you will like this, but I really %%% would suggest that in this case a group descriptor registry may %%% be appropriate. %[Tony] % Seems doable. % %%[Lyndon] %% But usable with some grief, as above. \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %% [Tony] %% Me too. %[Tony] % Seems doable. % \item There are restrictions that permit groups to be layered on top of pt-pt. %[Tony] % I don't understand the word 'restriction' here. % Restriction of what. % %%[Lyndon] %% Rik is speaking of the pair-exact-match stuff you see later on. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. %[Tony] % What does "fast" mean. % %%[Lyndon] %% Fair question! \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. %[Tony] % OK, the above two items are identical to what Zipcode % provides in practice, but people have argued that groups % might be created/deleted more often in some apps, and % that these apps ought to be supportable % %%[Lyndon] %% In our work group creation/deletion is an infrequent operation %% and we are happy to live with reasonable cost for this operation. %% I think Marc Snir is thinking abour a different group model in %% group created/deletion is frequent. %% Perhaps we should provide both or neither (even handedness principle). \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %% [Tony] %% 1) It is not obvious that TIDs should be restricted to 32 bits. %%%[Lyndon] %%% I did not imply that they were. %% 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %% I favor a whole word for a context, despite other limits, %% just to make things simpler. %% %%%[Lyndon] %%% ditto %% Internet addresses are going to get augmented from 32 to ??? bits %% is it reasonable to assume that certain MPI implementations might %% incorporate such internet addresses as TIDs (in future), %%%[Lyndon] %%% No, it is not reasonable, in the least. And both you and Rik appear %%% to be ignoring the possibility that the process descriptor could %%% be required to store routing data of significant length. Therefore %%% the sensible thing to do is use a process descriptor which in %%% practice might be a table index --- fits into integer for sure --- %%% the value of which is process local for sure, and the table %%% contains the real process identifier of implementation defined size. %% Opacity is partially violated if we say how big the data type is??? %%%[Lyndon] %%% I understand the point you make, but it gets blown away by the %%% point I have just made in reply to you. %[Tony] % Yes, and I want at least 32-bits of message tag. % %%[Lyndon] %% Yes, and I want exactly zero bits of message tag. %% I'll just keep quiet about message tags. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %% %%[Tony] %% where does this get us??? %% Remember, we have to choose on some things, so we can have something %% to present in Dallas. Is there an important difference here? %%%[Lyndon] %%% For sure their is an important difference. See comment above. %% %% TIDs are global entities. Is structure assumed to be global; %% in a truly opaque system, some TID component would have to be %% fixed, but the rest could vary structurally... %% %[Tony] % This could be difficult, in practice, if one mails a % message to one's own process, and MPI is smart enough % to optimize. % \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %% %% Yes, contexts are global; group identifiers are just pointers %% typically, to data structures, describing %% %% 1) a groups context %% 2) group members and their ranks (mappings, inverses, %% cached, hashed, unscalably stored, etc) %% 3) TID-to-rank map and inverse (see possibilities in 2) %% 4) A set of fixed global operations, accepted as standard, %% an accessible in O(1) time. Possibly, each %% such operation should be a method, so that %% a parameter block can be passed with it. Zipcode %% supports the Method type to do this. %% This is effectively a cache for some parts of item #5 %% ... %% 5) An AVL or similar tree of extensible operations. %% New operations are registerable by the user. These %% tags are unique within a group, a specify an operation %% i) pre-defined by MPI (in which case it can be cached %% in 4 %% ii) alternative operations (even if they do something %% standard, that are wanted to be accessed by %% name) This name is group unique. %% %% A mechanism for DO_METHOD_FROM_GROUP(name,....) %% or GET_METHOD_FROM_GROUP(name,...) %% and SET_METHOD_IN_GROUP(name,...) are clearly needed. %% %[Tony] % Sounds good. % \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %%[Tony] %% Yes, that is included, I believe, in concept. %% \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %%[Tony] %% Stripping is extremely controversial aspect, and arbitrary. %% If the recipient has the methods with the same name, then %% a new rendezvous could be accomplished at the far end %[Tony] % In Zipcode 1.0, we allow multiple global operations % to be provided on a message-class (eg, grid-oriented messages) % The identifiers for these possible operations are user-specified % presently, but the "names" of the global operations are fixed % at compile-time. % % That means that there is O(1) time to find combine, fanout, send, % etc, on a group-wide scope. However, other operations cannot % be accessed in O(1) time (they are not in the opaque structure). % % The same mechanism used by Zipcode to allow multiple methods for % combine to be registered by the user, could also allow extensibility % just like Rik describes, with little effort. We use AVL trees. % % In fact, I will add this to Zipcode 1.x. Why say this? It is % not far from existing practice, and I have a lot of the machinery % in place already, and I am confident that it is useful. % \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %%[Tony] %% Yes, these are all options. Fastness seems to be an important %% issue. If translation is very expensive, none of the "good" %% features will be used. %[Tony] % This seems to be a serious flaw. It will have to be cached % on an LRU basis, with system/user/both specifying how much % caching is allowed (ie, how much unscalable memory use). % If the first time is expensive, OK, but not the Nth time. % %%[Lyndon] %% Check. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). %[Tony] % OK, I see no problem with this (so far). % \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %%[Tony] %% Again, I think this should be eliminated, and all references %% to this idea should be expunged. It denies the context's %% ability to manage messages. %%%[Lyndon] %%% Check. %[Tony] % NO. This violates the concept of context entirely. % (ie, an oxymoron ... contexts same, but still no need for % tag disambiguation...) % % Use the default group context to establish (cooperatively) % other contexts, and then use these. This is a seriously % bad feature, in my mind. % \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. %%[Tony] %% I agree that context allocation has to be done in sequence. %% That is why I am in favor of providing calls that allow %% groups to get numerous contexts at creation, and then %% cooperatively, but potentially without further communication %% divide them(as they build subgroups, for instance). %% %% I see these as services to be used in building virtual topology %% features, which will then be more widely used by users of MPI. %% %%%[Lyndon] %%% If the context allocations are done in sequence, then I have %%% indicated how they can be done for free. I am getting confused. %[Tony] % I do not think we should support the paired-exact-match thing. % \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) %[Tony] % Concur. This suggests many more than "256" % %%[Lyndon] %% The number of contexts in the whole program? Sure 256 is too small! %% The number of contexts in each process? Maybe something like 256 is okay? \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] %[Tony] % This is the Zipcode model. It could say loosely synchronous. % \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %%[Tony] %% Do we get anything by having the root process? %% %%%[Lyndon] %%% No. %[Tony] % I don't understand the thread issue here. % %%[Lyndon] %% If two threads are concurrently partitioning the same group, you %% need to disambiguate the partition operations. Analagous to %% concurrent collective operations or nonblocking. \item Collective communication routines are called by all members of a group in the same order. %[Tony] % Yes. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. %[Tony] % What does caching really imply here ??? Help. % %%[Lyndon] %% Dunno. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %%[Tony] %% I think that non-blocking collcomm is moribund in MPI1 or %% else MPI1 is moribund. :-) %% %%%[Lyndon] %%% Check. %[Tony] % I think that contexts are really important in this case, % to keep things straight, but that non-blocking collcomm should % be omitted from MPI1 (cf, Geist). Sequencing supports % a sufficient disambiguation, as long as the entire group % is always the participant in operations. That is, you have % to form subgroups, with new contexts, to do global ops on % subsets. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %%[Tony] %% Too many negatives here for me to parse :-) %[Tony] % Why aren't servers needed to create contexts. Where do they % come from? If you rely on the fact that INITIAL will do % a loosely synchonous cooperative operation each time a new % context is needed, then a simple (easily implementable server, % or fetch-and-add remote access) is replaced by a more rigid % computation model. % % If we can get rid of this disagreement, me might be able to % reduce our total proposal space by one whole proposal. % \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %%[Tony] %% Context allocation need not be free provided it can be made cheap, %% or cheap enough. %% %% If one knows one will need several, then a single call could %% provide such contexts, amortizing overhead. This is likely when %% bulding grids (ie, virtual topologies) in Zipcode, so it is %% true in existing practice. %% %% One should recognize the need for layering virtual top. calls %% on top of these calls, then these calls may appear painful, %% but perhaps they would be less used. Some users will use the %% provided virtual topology calls, others will prefer their own. %% Both will have equal power (see also,separate note on layerability). %% %% If getting N contexts is a send-and-receive, plus a reactive server, %% then this is reasonably light weight,provided that hundreds of %% messages, or global operations ensue thereafter. We can know in %% advance how heavy weight the context server will be. %% %% if an implemention can use some locations of remote memory, with %% fetch and add, or locks, to achieve contexts, then this is even %% cheaper, in principle. %% %% Despite Jim's earlier insistence that context numbers be kept to %% 256 or so, I think that this number should be much larger, so that %% much less efort goes into returning contexts, and so on, except %% occasionally, by processes. Otherwise, a new kind of overhead, %% get-rid-of-context-because-I-am-out ensues, or programs block %% until contexts become available, offering the possibility of %% deadlocks. %% %[Tony] % If contexts are being used very dynamically, how are they being % assigned, kept, released, reissued without a server? Sorry if % I missed something, but I don't see it, without a restrictive % SPMD model of computation (Zipcode obviates its server for the % SPMD model, for instance). %%[Lyndon] %% MPI stinks of SPMD. I wouldn't mind if MPI would just say SPMD. \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %%[Tony] %% Unscalability (of a limited form) should be permitted/selectable %% by user, to use as much per-node memory as the user wants, to reduce %% communication. %% %[Tony] % Can you clarify this with examples. % \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. % %%[Tony] %% I see,but what he means here is that there is no unpredictable %% translation cost because we do not write (group,rank) in pt2pt %% calls. So, there is some validity to the statement. %%%[Lyndon] %%% However he ignores that the TID might require translation the %%% cost of which might be unpredictable. This because the TID has %%% a global value and cannot therefore hold process local information %%% such as ``how do I route to that process''. %[Tony] % I like this, of course. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %%[Tony] %% Yes, that is why I have one in Zipcode. I wish Zipcode were %% on netlib today, so you could try it. Well, we are writingthe %% manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %%[Tony] %% I lost something here, but I agree with the sense. The word %% seems is subjective,and should disappear from our discussions, %% as much as seems prudent, anyway :-) %[Tony] % Well, is point-to-point group oriented. Not. %%[Lyndon] %% Check. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %%[Tony] %% Yes,awkward, because it is nice to escape the TID realm and %% work within the (albeit simple) abstraction of group,rank. %% When layering virtual topologies on this, it would be so nice %% to write them to a group,rank syntax, not enforcing TID mappings %% everywhere. %[Tony] % I think it is awkward. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %%[Tony] %% Indeed. %%%[Lyndon] %%% More so than you think, I think! %[Tony] % OK, but one can form a new group, as I have argued before. % Use the "awkward" pt2pt to get the right info shared between % group leaders, make the new group, use unawkward collective % operations on new group (with new context). %%[Lyndon] %% This is only one model of group-group interaction, which in my %% experience and understanding really is still steeped in SPMD. %% Please consider the examples of non SPMD group usage which I %% mailed out. You can say to me - oh, you shouldn't do this kind %% of thing with MPI, Lyndon - if you like, if you believe that. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. %[Tony] % This again is in practice, in Zipcode. % \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %%[Tony] %% Yes, and I think that loosely synchronous operations can maintain %% coherency, in practice. That is, no operations that modify the %% group descriptors (other than cached lookup info) are permitted, %% without loose synchronization. %% This is nasty in that is would prohibit sending descriptors to %% processes not part of the group, so it is a clear trade-off. %% Perhaps such send-to-non-group-member operations could stipulate %% that this group information is somehow ephemeral, and that they %% need to join a new group to keep useful information over time??? %% %%%[Lyndon] %%% I am over schedule. I have to stop here. I will come back to this %%% tomorrow, if applicable (ie, you may have overtaken me). %[Tony] % Sounds dangerous. What must application do to maintain % coherency, since group descriptors are opaque. % \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. %[Tony] % No, loosely synchronous process model, unless you mean % cooperation of INITIAL at all such join/leave steps. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %%[Tony] %% Yes, that is exactly what Zipcode was written to avoid. The %% user wants help managing things like this!!!! %% %% The search, if any, must be MPI-supported, and as efficient as %% possible (eg, AVL trees, hash, partial hash with exceptions). %% % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %%[Tony] %% This is why I remain a strong advocate of (group,rank) %% addresssing in pt2pt. %% %[Tony] % No, you can't know the group or rank in group of sender. % If there were one context per group (isn't that so here?), % then all you need is the rank. With TID_TO_RANK_IN_GROUP % operation, this could be provided, but no wildcarding % or receipt selectivity could be done at this level. % \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %%[Tony] %% I addressed this in my Step-1 remarks. Please see that. BELOW %[Tony] % No, it just does not provide guarantees that certain kinds % of applications will run OK. (ie, those that do group % creation/deletion relatively often). Zipcode has assumed % that such operations would be relatively seldom. Thus, I do % not quibble that this is a reasonable choice,but a fairer % way to say this is that it may be difficult to support such % applications. That reveals an issue to be studied more. % \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %%[Tony] %% Thank you. These points are both well taken (ie these two paragraphs) %[Tony] % Perhaps they should do. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] %[Tony] % Dump this. \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %%[Tony] %% Also,see what I wrote in my Step-1 comments. BELOW. %% I believe O(log(P)) is still possible. %% %[Tony] % No, a non-deterministic broadcast can be used, with a token. % This requires a token server. Again, implementable with fetch+ % add on most systems, or a light reactive server. % % Once the non-deterministic broadcast has finished, a fanin/collapse % is done to the original root, which then frees the token. % \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %%[Tony] %% Please see my Step-1 comments. %[Tony] % Why not just given in and allow the server. % I don't like the paired-exact-match constraint AT ALL. % \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) %[Tony] % Seems OK, but why need the paired-exact-match thing again. % Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) %[Tony] % Again, OK, but I want to see this work without the paired-exact- % match, if possible. Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. %[Tony] % So, now, you have concurred with my (previously flamed) idea % that group construction/destruction should be realizable using % pt2pt, just like global operations should do. I like this % because 1) it is explicable to the implementor, 2) it allows % simple intitial implemtations, 3) it sets some ideas for how % much these things will cost [upper bound]. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} %[Lyndon] % Writing a server in the point-to-point layer of MPI in four easy steps % ---------------------------------------------------------------------- % % 1) Partition the INITIAL group into two groups. A singleton group, % SERVER, and a group CLIENT which contains all of the other processes. % % 2) The single process in SERVER group records its TID. % % 3) The processes in INITIAL group allocate a context SERVICE which % they remember either in the group cache or static data or something. % % 4) Use a broadcast in INITIAL group with ``sender'' as the one process % which is also in SERVER group, and the ``receivers'' as the (many) % processes which are also in CLIENT group, in the SERVICE context, in % order to disseminate the TID of the server process. % % [Fanfare] a server process is in place as is a dedicated context for % the purposes of messages required to implement the service. % % [Observation] the mpi point-to-point initialisation can do this % automatically. %%[Tony] %% Zipcode's postmaster general works in this way, more or less. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 01:52:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07571; Mon, 22 Mar 93 01:52:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15011; Mon, 22 Mar 93 01:51:24 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 01:51:23 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15000; Mon, 22 Mar 93 01:51:21 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 21 Mar 93 22:47 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06109; Sun, 21 Mar 93 22:45:30 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15778; Sun, 21 Mar 93 22:45:28 PST Date: Sun, 21 Mar 93 22:45:28 PST From: rj_littlefield@pnlg.pnl.gov Subject: Re: mini-proposal on layerability To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9303220645.AA15778@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony writes: > So, each receipt function uses the algorithm (received_rag & ~dont_care)& > care Shouldn't this be (received_tag & ~dont_care) == care i.e. ignore some bits but check the others for equality? If so then yes, I will strongly support this feature. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 03:30:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11999; Mon, 22 Mar 93 03:30:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20829; Mon, 22 Mar 93 03:29:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 03:29:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20810; Mon, 22 Mar 93 03:29:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 00:13 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06118; Mon, 22 Mar 93 00:11:53 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15810; Mon, 22 Mar 93 00:11:50 PST Date: Mon, 22 Mar 93 00:11:50 PST From: rj_littlefield@pnlg.pnl.gov Subject: new proposal To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220811.AA15810@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony and Lyndon, I will respond in a separate message to some of your detailed comments on Proposal V. But maybe we can move faster by popping up a level. It seems to me that you like the idea of cacheing arbitrary info in group descriptors, you like the idea of groups as things within which contexts get formed, you like performance guarantees, and you don't like having to use opaque id's in point-to-point communications. I'm not quite sure whether static groups and synchronous group control are OK, but I'll presume here that they are. Well, this is pretty neat. If this keeps up maybe we can converge on just one or two proposals. Most of my gripes with other proposals have been based on performance and/or a need for asynchronous servers (with attendant performance and non-portability gripes). But I notice that explicit performance expectations have been gradually appearing and the need for servers disappearing from other people's proposals. Perhaps it is time for a synthesis. Here's a sketch of a new proposal (VI ?). . The functionality of point-to-point communications is per Snir, Lusk, and Gropp, augmented by my proposed MPI_FORM_CONTEXT to allow assembling arbitrary collections of processes. (Marc has already accepted this in a private email to me -- don't know why he didn't post it.) . Performance expectations of point-to-point communications are explicitly stated as follows: - MPI_COPY_CONTEXT does not synchronize the participating processes and costs significantly less than a point-to-point fanout among them (e.g., it uses a communication-free counting strategy); - all other context formation routines cost no more than if they were implemented using a single fanin/fanout among the participating processes; - translation of (context,rank) to absolute processor ID costs no more than if it were implemented via the lookup table that Snir suggests. . Groups and contexts are not equal. A group consists of a base context (from which other contexts can be created quickly by MPI_COPY_CONTEXT), plus topology information, plus my cacheing facility. Conceptually, I like this better than proposal V. Do we already have a proposal like this? Should we have one? In general, do you guys buy off on the concept of including performance expectations in the specification? A couple of discussion points... 1. The separation of group and context is defensive. I think I understand what it means to copy a context. I am not sure of either the functional or performance implications of copying a group. E.g., does cached info get copied? 2. I will respond here to two criticisms that have been raised against the cacheing facility. Lyndon notes > % I like the general idea, but I'm nervous about two things: > % (a) implied associativity of group descriptor cache - this > % will potentially be time expensive in implementation. > % (b) there is no method proposed for abritration of keys > % between independently written modules, so we are > % in the same problem regime as just having message tag > % and no message context. > % However, key's are local, so presumably you would find > % it acceptable to add a key registration service? I implicitly proposed a key assignment service in my long-ago example. It said in part: static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_key_type gop_key; /* key for this module's stuff */ ... if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetAttributeKey()) ) { MPI_abort ("Insufficient keys available"); } } This is not really "registration" because nothing goes into the key assignment routine except the number of times it's called. But assuming that each call site is protected by a separate variable, the effect is to register the call site. As Lyndon notes, this is highly local. But it also allows the returned keys to have their values restricted so as to permit rapid testing and/or retrieval. Tony notes > %**** Stripping is extremely controversial aspect, and arbitrary. > %**** If the recipient has the methods with the same name, then > %**** a new rendezvous could be accomplished at the far end Yes, stripping is arbitrary. My motivation is that this greatly simplifies the design and satisfies what I view as the most critical need: to make collective comms run fast without complicating the calling sequence. I have no objection in principle to extending the facility to include classes of information that do not get stripped. But I sure didn't want to try creating and selling a spec that would handle, e.g., heterogeneous systems. Enough for here -- what do you think of "proposal VI" ? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:42:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25028; Mon, 22 Mar 93 05:42:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00467; Mon, 22 Mar 93 05:42:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 05:41:55 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00422; Mon, 22 Mar 93 05:41:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 01:48 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06136; Mon, 22 Mar 93 01:47:08 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15840; Mon, 22 Mar 93 01:47:05 PST Date: Mon, 22 Mar 93 01:47:05 PST From: rj_littlefield@pnlg.pnl.gov Subject: Rik's comments on Lyndon's Proposal I To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220947.AA15840@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu >>>>> I have added here a few additions to Tony's comments. >>>>> I have also deleted the bulk of the proposal regarding which >>>>> I have no comments at 1:45 AM. >>>>> -- Rik %**** %**** Below are my comments on Lyndon's Proposal I. In the next paragraph %**** I note that we are actually converging to less than N proposals, though %**** I have not seen Lyndon's new II proposal yet. Does it exist now? %**** %**** To achieve a reasonable presentation at Dallas, we can have multiple %**** proposals still on table (I think this a fair, well-thought-out %**** approach, but if we can condense some, let's do it). %**** %**** %**** After reading V, making my comments, and making my comments %**** in addition to Lyndon's comments, I am convinced we can %**** advance proposal V into something that is acceptable in the %**** III/IV mold, without a further III/IV proposal. Rik's %**** dropping of the static context concept has simplified our %**** group's efforts considerably, and I cannot disambiguate my %**** III/IV proposal from Proposal V, given the Lyndon and Tony %**** provisos and suggested improvements. This is not a wimp out %**** on my part. I do not see benefit of advancing something that %**** will look 90% like Proposal V at this point, and 97% like it %**** if the Lyndon/Tony comments obtain in it (which they would if %**** I wrote it now). %**** I would prefer that we hone Proposal V. If Rik wants to keep %**** his style, then I propose that Proposal III/IV become exactly %**** what I just said, a reworking of V + comments. %**** However, I think this will cause an unnecessary delay and %**** digression just to achieve details. Instead, we might pull %**** such choices into Proposal V at this time (server vs. no server). %**** to make what is common in our "approaches" more obvious. %**** - Tony %**** %**** (PS, Lyndon: Rik/Tony/Lyndon are authors of whole paper, because %**** we are all working on this, and because one of us (ie, me) will %**** make the final document cohesive, create a unified style, format, %**** set of meanings. This is the nature of collaboration. I do %**** not propose to include our other co-conspirators on such a document's %**** list of authors, as there has been minimal input from them. I %**** Think that Rusty/Bill and Rik/Tony/Lyndon are operating equivalently.) >>>>> >>>>> I happens to think that the style of my draft proposal V stinks. >>>>> I had much too much trouble becoming (fairly) sure that the content >>>>> was right, to also make it readable. >>>>> --Rik .... A group may be created by identical duplication of an existing group. For example, {\tt gidb = mpi\_grp\_duplication(gida)} where {\tt gidb} is the group identifier of the newly created group and {\tt gida} is the identifier of an existing group. The created group inherits all properties of the source group, including any topological properties. This operation has the same synchronisation properties as creation of group by definition. %**** It is not obvious to me that we want to enforce topology at this %**** juncture. However, we could register topology information in %**** the extensible structure strategy of Proposal V. %**** >>>>> Why such strong synchronization?! >>>>> If this has the same synchronization properties as creation, >>>>> then it won't return until all members have made the call. >>>>> But that means you actually have to sync everybody, which >>>>> implies 2 log(P) messages. Isn't it enough to just require >>>>> a loosely synchronous call, and use a counting strategy> ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:42:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25031; Mon, 22 Mar 93 05:42:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00499; Mon, 22 Mar 93 05:42:07 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 05:42:06 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00443; Mon, 22 Mar 93 05:41:56 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 01:30 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06131; Mon, 22 Mar 93 01:29:00 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15820; Mon, 22 Mar 93 01:28:55 PST Date: Mon, 22 Mar 93 01:28:55 PST From: rj_littlefield@pnlg.pnl.gov Subject: Rik on Tony on Lyndon on PropV To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220928.AA15820@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu This is the Proprosal V first draft, critiqued by Lyndon, then by Tony, and now with responses by Rik. Comments are flagged as % Lyndon's %**** Tony's >>>>> Rik's responses [Lyndon's leadin] General points -------------- 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am offering if you wish) should attempt to construct a more transparent presentation before circulation to whole committee, for the convenience of committee members. %**** %**** I felt that things appear twice, because of summary (good) %**** and because of implementation notes at end (confusing) %**** >>>>> I agree with both of the above. But let's decide whether >>>>> Proposal V will be replaced by VI or its equivalent before >>>>> we do any rewriting. 2) I'm not a fan of much of this proposal, although I do indeed like some of the ideas which it introduces. [On the other hand, I'm not a great fan of all of the proposal which I wrote. I shall mail self criticism of my proposal, and may have to write amended or alternative proposal :-)] %**** %**** Please be more specific. I am having a hard time understanding %**** why you really don't like it, Lyndon. If the process model %**** were a little less static, and servers were permitted (though %**** hopefully bounded in cost), I think we would have an excellent %**** proposal. %**** >>>>> See Proposal VI. I'd be happy to see the good ideas adopted >>>>> and the crap die quietly. 3) I really like the way in which groups are something like ``frames'' in which contexts are created. This is conceptually much neater than duplication of groups. %**** %**** In practice, group subsetting will require groups to be copied, %**** otherwise, subgroups will unfairly be penalized by the size %**** of their ancestor. %**** >>>>> Right, but I think of that as creating a new group. After all, >>>>> even the ranking structure is different. 4) I like the idea of pushing information into the group structure. I have a few qualms with the proposed details --- see specific points. %**** %**** I have more confidence about this idea, and could demonstrate %**** by June/July time-frame in Zipcode. %**** 5) See ``Writing a server in the point-to-point layer of MPI in four easy steps'' at the foot of the message. %**** %**** This seems like a nice thing. %**** >>>>> The implementation that Lyndon suggests consumes an entire >>>>> process for the server. There are times when this is OK, >>>>> but also times when it isn't. E.g., if you have a divide-and- >>>>> conquer algorithm that really wants 2^i non-server processes, >>>>> and you're working on a machine with 2^n processors that >>>>> doesn't support multiple processes per processor, then >>>>> some of the users will get upset that they can only use >>>>> half the machine. A year and a half ago, I got flamed for >>>>> making a suggestion like this in connection with the Delta. >>>>> (And now I suppose I get flamed for using the Delta as an >>>>> example again... Maybe if I use the CM5...) Specific points --------------- Dealt with as LaTeX comments to body of text, appearing in the form %[Lyndon] % text of point for your navigational convenience. These are quite detailed. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %**** %**** Alternatives, Lyndon? %**** \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %**** Me too. >>>>> Progress!! \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %**** %**** 1) It is not obvious that TIDs should be restricted to 32 bits. %**** 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %**** I favor a whole word for a context, despite other limits, %***** just to make things simpler. %**** %**** Internet addresses are going to get augmented from 32 to ??? bits %**** is it reasonable to assume that certain MPI implementations might %**** incorporate such internet addresses as TIDs (in future), %**** %**** Opacity is partially violated if we say how big the data type is??? \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %**** %**** where does this get us??? %**** Remember, we have to choose on some things, so we can have something %**** to present in Dallas. Is there an important difference here? %**** %**** TIDs are global entities. Is structure assumed to be global; %**** in a truly opaque system, some TID component would have to be %**** fixed, but the rest could vary structurally... %**** \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %**** %**** Yes, contexts are global; group identifiers are just pointers %**** typically, to data structures, describing %**** %**** 1) a groups context %**** 2) group members and their ranks (mappings, inverses, %**** cached, hashed, unscalably stored, etc) %**** 3) TID-to-rank map and inverse (see possibilities in 2) %**** 4) A set of fixed global operations, accepted as standard, %**** an accessible in O(1) time. Possibly, each %**** such operation should be a method, so that %**** a parameter block can be passed with it. Zipcode %**** supports the Method type to do this. %**** This is effectively a cache for some parts of item #5 %**** ... %**** 5) An AVL or similar tree of extensible operations. %**** New operations are registerable by the user. These %**** tags are unique within a group, a specify an operation %**** i) pre-defined by MPI (in which case it can be cached %**** in 4 %**** ii) alternative operations (even if they do something %**** standard, that are wanted to be accessed by %**** name) This name is group unique. %**** %**** A mechanism for DO_METHOD_FROM_GROUP(name,....) %**** or GET_METHOD_FROM_GROUP(name,...) %**** and SET_METHOD_IN_GROUP(name,...) are clearly needed. %**** >>>>> The model of contexts used in this proposal is that the >>>>> context value has global scope. It doesn't have to be that way. >>>>> The Snir, Lusk, and Gropp proposal could be implemented with each >>>>> processor contributing a different context value to the context >>>>> descriptor. E.g., process 3 says "if you want to talk to me >>>>> in this context, you have to use context value 7", while process >>>>> 4 says to use context value 2. I don't know whether there are >>>>> advantages to that extra flexibility. Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %**** Yes, that is included, I believe, in concept. %**** >>>>> Right. \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %**** %**** Stripping is extremely controversial aspect, and arbitrary. %**** If the recipient has the methods with the same name, then %**** a new rendezvous could be accomplished at the far end >>>>> >>>>> See my "Proposal VI" note for responses to this. \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %**** %**** Yes, these are all options. Fastness seems to be an important %**** issue. If translation is very expensive, none of the "good" %**** features will be used. >>>>> I was actually trying to be nice to the service process >>>>> people, giving them a designated hook where they could be >>>>> slow and I could work around them via the cacheing mechanism. >>>>> I'd really rather just specify it as being fast. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %**** Again, I think this should be eliminated, and all references %**** to this idea should be expunged. It denies the context's %**** ability to manage messages. >>>>> >>>>> When I wrote this, I was presuming that contexts could >>>>> not be generated "for free" and that therefore it was >>>>> a good idea to specify a method by which codes could >>>>> be written so as to not require them. By the way, >>>>> you may be interested to note that the sample collective >>>>> comms defined by Snir and Geist in their recent proposal >>>>> implicitly rely on exactly the paired-exact-match >>>>> constraint. As written, those routines would break >>>>> if preceded by a call to a module that issued a wildcard >>>>> receive in the same group. (And there's an endnote >>>>> that suggests they know that.) \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. >>>>> I think this is right. This counting strategy probably >>>>> places some constraints on how big the context value field >>>>> has to be, but I've incorporated it into Proposal VI. %**** I agree that context allocation has to be done in sequence. %**** That is why I am in favor of providing calls that allow %**** groups to get numerous contexts at creation, and then %****cooperatively, but potentially without further communication %**** divide them(as they build subgroups, for instance). %**** %**** I see these as services to be used in building virtual topology %**** features, which will then be more widely used by users of MPI. %**** \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %**** %**** Do we get anything by having the root process? %**** >>>>> The group partitioning concept has been refined in several >>>>> other postings to mpi-context, mpi-collcom, and mpi-pt2pt, >>>>> in which "root" was replaced by a "knownmember" set. >>>>> The idea is that if you have lots of processes joining a >>>>> group, they don't have to know who all their compatriots >>>>> are, but they should know at least one that they all have >>>>> in common, who can serve to organize the communications. \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %**** %**** I think that non-blocking collcomm is moribund in MPI1 or %**** else MPI1 is moribund. :-) %**** >>>>> Nicely put. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %**** Too many negatives here for me to parse :-) \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %**** %**** Context allocation need not be free provided it can be made cheap, %**** or cheap enough. %**** %**** If one knows one will need several, then a single call could %**** provide such contexts, amortizing overhead. This is likely when %**** bulding grids (ie, virtual topologies) in Zipcode, so it is %**** true in existing practice. %**** %**** One should recognize the need for layering virtual top. calls %**** on top of these calls, then these calls may appear painful, %**** but perhaps they would be less used. Some users will use the %**** provided virtual topology calls, others will prefer their own. %**** Both will have equal power (see also,separate note on layerability). %**** %**** If getting N contexts is a send-and-receive, plus a reactive server, %**** then this is reasonably light weight,provided that hundreds of %**** messages, or global operations ensue thereafter. We can know in %**** advance how heavy weight the context server will be. %**** %**** if an implemention can use some locations of remote memory, with %**** fetch and add, or locks, to achieve contexts, then this is even %**** cheaper, in principle. %**** %**** Despite Jim's earlier insistence that context numbers be kept to %**** 256 or so, I think that this number should be much larger, so that %**** much less efort goes into returning contexts, and so on, except %**** occasionally, by processes. Otherwise, a new kind of overhead, %**** get-rid-of-context-because-I-am-out ensues, or programs block %**** until contexts become available, offering the possibility of %**** deadlocks. %**** \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %**** %**** Unscalability (of a limited form) should be permitted/selectable %**** by user, to use as much per-node memory as the user wants, to reduce %**** communication. %**** >>>>> Besides which, the system will do this anyway if (group,rank) >>>>> translations are required to be fast. All I'm doing is moving >>>>> it to the user level. \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. %**** I see,but what he means here is that there is no unpredictable %**** translation cost because we do not write (group,rank) in pt2pt %**** calls. So, there is some validity to the statement. >>>>> Tony has it right. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %**** Yes, that is why I have one in Zipcode. I wish Zipcode were %**** on netlib today, so you could try it. Well, we are writingthe %**** manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %**** %**** I lost something here, but I agree with the sense. The word %**** seems is subjective,and should disappear from our discussions, %**** as much as seems prudent, anyway :-) >>>>> Silly me -- trying to call out an opinion that might be >>>>> disagreed with. It occurred to me when I wrote the "s" word >>>>> that I'd probably be better off to just follow standard practice >>>>> and state my opinions as fact. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %**** %**** Yes,awkward, because it is nice to escape the TID realm and %**** work within the (albeit simple) abstraction of group,rank. %**** When layering virtual topologies on this, it would be so nice %**** to write them to a group,rank syntax, not enforcing TID mappings %**** everywhere. >>>>> >>>>> I happen to agree with this. But even though it's late, I can't >>>>> resist pointing out that there's not a shred of scientific >>>>> evidence that these opinions are in fact true. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %**** Indeed. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %**** Yes, and I think that loosely synchronous operations can maintain %**** coherency, in practice. That is, no operations that modify the %**** group descriptors (other than cached lookup info) are permitted, %**** without loose synchronization. %**** This is nasty in that is would prohibit sending descriptors to %**** processes not part of the group, so it is a clear trade-off. %**** Perhaps such send-to-non-group-member operations could stipulate %**** that this group information is somehow ephemeral, and that they %**** need to join a new group to keep useful information over time??? %**** \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %**** %**** Yes, that is exactly what Zipcode was written to avoid. The %**** user wants help managing things like this!!!! %**** %**** The search, if any, must be MPI-supported, and as efficient as %**** possible (eg, AVL trees, hash, partial hash with exceptions). %**** % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %**** %**** This is why I remain a strong advocate of (group,rank) %**** addresssing in pt2pt. %**** \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %**** %**** I addressed this in my Step-1 remarks. Please see that. %**** \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. >>>>> I thought I said that. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %**** %**** Thank you. These points are both well taken (ie these two paragraphs) >>>>> >>>>> See my comments near the start of this message, regarding >>>>> the proposed server implementation. What I should have said >>>>> was that MPI did not provide enough capability to construct >>>>> a server without consuming a processor, since it neither >>>>> provides an interrupt-receive function nor specifies that >>>>> processors be able to time-share between multiple processes. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %**** Also,see what I wrote in my Step-1 comments. %**** I believe O(log(P)) is still possible. %**** >>>>> I'd be interested in seeing the O(log(P)) algorithm, >>>>> sometime after MPI quiets down. \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %**** Please see my Step-1 comments. \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- Writing a server in the point-to-point layer of MPI in four easy steps >>>>> by potentially consuming an entire processor ---------------------------------------------------------------------- 1) Partition the INITIAL group into two groups. A singleton group, SERVER, and a group CLIENT which contains all of the other processes. 2) The single process in SERVER group records its TID. 3) The processes in INITIAL group allocate a context SERVICE which they remember either in the group cache or static data or something. 4) Use a broadcast in INITIAL group with ``sender'' as the one process which is also in SERVER group, and the ``receivers'' as the (many) processes which are also in CLIENT group, in the SERVICE context, in order to disseminate the TID of the server process. [Fanfare] a server process is in place as is a dedicated context for the purposes of messages required to implement the service. [Observation] the mpi point-to-point initialisation can do this automatically. %**** Zipcode's postmaster general works in this way, more or less. %**** - Tony ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:50:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26950; Mon, 22 Mar 93 05:50:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20829; Mon, 22 Mar 93 03:29:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 03:29:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20810; Mon, 22 Mar 93 03:29:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 00:13 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06118; Mon, 22 Mar 93 00:11:53 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15810; Mon, 22 Mar 93 00:11:50 PST Date: Mon, 22 Mar 93 00:11:50 PST From: rj_littlefield@pnlg.pnl.gov Subject: new proposal To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220811.AA15810@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony and Lyndon, I will respond in a separate message to some of your detailed comments on Proposal V. But maybe we can move faster by popping up a level. It seems to me that you like the idea of cacheing arbitrary info in group descriptors, you like the idea of groups as things within which contexts get formed, you like performance guarantees, and you don't like having to use opaque id's in point-to-point communications. I'm not quite sure whether static groups and synchronous group control are OK, but I'll presume here that they are. Well, this is pretty neat. If this keeps up maybe we can converge on just one or two proposals. Most of my gripes with other proposals have been based on performance and/or a need for asynchronous servers (with attendant performance and non-portability gripes). But I notice that explicit performance expectations have been gradually appearing and the need for servers disappearing from other people's proposals. Perhaps it is time for a synthesis. Here's a sketch of a new proposal (VI ?). . The functionality of point-to-point communications is per Snir, Lusk, and Gropp, augmented by my proposed MPI_FORM_CONTEXT to allow assembling arbitrary collections of processes. (Marc has already accepted this in a private email to me -- don't know why he didn't post it.) . Performance expectations of point-to-point communications are explicitly stated as follows: - MPI_COPY_CONTEXT does not synchronize the participating processes and costs significantly less than a point-to-point fanout among them (e.g., it uses a communication-free counting strategy); - all other context formation routines cost no more than if they were implemented using a single fanin/fanout among the participating processes; - translation of (context,rank) to absolute processor ID costs no more than if it were implemented via the lookup table that Snir suggests. . Groups and contexts are not equal. A group consists of a base context (from which other contexts can be created quickly by MPI_COPY_CONTEXT), plus topology information, plus my cacheing facility. Conceptually, I like this better than proposal V. Do we already have a proposal like this? Should we have one? In general, do you guys buy off on the concept of including performance expectations in the specification? A couple of discussion points... 1. The separation of group and context is defensive. I think I understand what it means to copy a context. I am not sure of either the functional or performance implications of copying a group. E.g., does cached info get copied? 2. I will respond here to two criticisms that have been raised against the cacheing facility. Lyndon notes > % I like the general idea, but I'm nervous about two things: > % (a) implied associativity of group descriptor cache - this > % will potentially be time expensive in implementation. > % (b) there is no method proposed for abritration of keys > % between independently written modules, so we are > % in the same problem regime as just having message tag > % and no message context. > % However, key's are local, so presumably you would find > % it acceptable to add a key registration service? I implicitly proposed a key assignment service in my long-ago example. It said in part: static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_key_type gop_key; /* key for this module's stuff */ ... if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetAttributeKey()) ) { MPI_abort ("Insufficient keys available"); } } This is not really "registration" because nothing goes into the key assignment routine except the number of times it's called. But assuming that each call site is protected by a separate variable, the effect is to register the call site. As Lyndon notes, this is highly local. But it also allows the returned keys to have their values restricted so as to permit rapid testing and/or retrieval. Tony notes > %**** Stripping is extremely controversial aspect, and arbitrary. > %**** If the recipient has the methods with the same name, then > %**** a new rendezvous could be accomplished at the far end Yes, stripping is arbitrary. My motivation is that this greatly simplifies the design and satisfies what I view as the most critical need: to make collective comms run fast without complicating the calling sequence. I have no objection in principle to extending the facility to include classes of information that do not get stripped. But I sure didn't want to try creating and selling a spec that would handle, e.g., heterogeneous systems. Enough for here -- what do you think of "proposal VI" ? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:53:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27630; Mon, 22 Mar 93 05:53:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01772; Mon, 22 Mar 93 05:53:21 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 05:53:19 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01761; Mon, 22 Mar 93 05:53:16 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA18140 (5.65c/IDA-1.4.4 for ); Mon, 22 Mar 1993 04:48:07 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA24191; Mon, 22 Mar 93 09:48:03 GMT Date: Mon, 22 Mar 93 09:48:03 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303220948.AA24191@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00311; Mon, 22 Mar 93 09:44:36 GMT To: tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu In-Reply-To: Tony Skjellum's message of Sun, 21 Mar 93 12:16:22 CST <9303211816.AA03587@Aurora.CS.MsState.Edu> Subject: mini-proposal on layerability Content-Length: 695 There is a typo in Tony's message : > So, each receipt function uses the algorithm > (received_rag & ~dont_care) & care This should read (received_tag & ~dontcare) == care i.e. don't care specifies a set of bits to be ignored in the comparison, care specifies the precise value of the other bits Note that you can do complete wild-carding like this by setting dont_care = -1; care = 0; -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Mon Mar 22 06:21:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03967; Mon, 22 Mar 93 06:21:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02909; Mon, 22 Mar 93 06:20:50 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 06:20:49 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02898; Mon, 22 Mar 93 06:20:46 -0500 Date: Mon, 22 Mar 93 10:01:09 GMT Message-Id: <13945.9303221001@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Rik's comments on Lyndon's Proposal I To: rj_littlefield@pnlg.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu In-Reply-To: rj_littlefield@pnlg.pnl.gov's message of Mon, 22 Mar 93 01:47:05 PST Reply-To: lyndon@epcc.ed.ac.uk Cc: d39135@sodium.pnl.gov Hi Rik Thanks for the comments. The proposal which I had sent out is dead. It has been replaced with a cut-and-paste of the section on contexts which Marc wrote into the point-to-point document. There is no proposal II. There is however a proposal VI. This naming/numbering is Tony's suggestion with which I concur. I will propogate the relevant comments you have made into the LaTeX of proposal VI (as identified LaTeX comments) and repost. I really am very sorry to have messed you about. I have a lot of other email, incl. from yourself, to read. (I am cancelling meetings left, right and centre here.) I will post later today. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 06:32:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06460; Mon, 22 Mar 93 06:32:56 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03270; Mon, 22 Mar 93 06:32:22 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 06:32:21 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03262; Mon, 22 Mar 93 06:32:15 -0500 Date: Mon, 22 Mar 93 11:32:10 GMT Message-Id: <14066.9303221132@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: Reference Point To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear Colleagues This message is a reference point summary of the current state of evolution of proposals. This is intended to help reduce confusion, as there has been a lot of development in the last 48 hours. Here is a complete summary of proposals to date, with proposal identifiers, contacts, status and brief description. In summary of the table: There are currently FOUR live proposals, one of which will shortly be circulated. These are labelled: I (in circulation); III/IV (not circulated); V (in circulated); VI (in circulated). There are currently THREE dead proposals. These are labelled: I++ (circulated as I); II (circulated?); II' (circulated as II). Identifier Status Contact Brief ---------- ------ ------- ----- I++ dead lyndon The "Proposal I" agreed at the post meet lunch, augmented by lyndon, and now defunct. This was circulated as Proposal I last week. Please kill this one. I live marc The proposal of marc which also appeared in the point-to-point document. It was cut-and-paste'd out of that document. This is currently in circulation as Proposal I. II dead rik The "Proposal II" agreed at the post meet lunch, now defunct. Please kill this one. II' dead lyndon A proposal unrelated to II. This has been named VI as of yesterday, to help avoid confusion. This was circulated as Proposal II' yestreday. Please kill this one. III/IV live tony The proposals III and IV agreed at the post lunch meet, to be merged. These have not yet appeared and should do within < 24 hours. This will be circulated as Proposal III/IV. V live Rik Proposal of Rik after abandonment of Proposal II. This was circulated as Proposal V in plain text and LaTeX source formats, last week. VI live lyndon This was, for about 4 hours, called II', yesterday. It has some common ground with I++ and other proposals. This was circulated as Proposal VI around 11:30pm GMT yesterday as LaTeX. There is currently a sketch which Rik is also naming as Proposal VI. Please Rik, either call this Proposal VII or call it a sketch, just to save confusion. The sketch suggests that we may have sufficient convergence to offer a single proposal. Tony and I discussed this yesterday evening and we concur. Tony and I suggest that the following occur: 1) Tony completes Proposal III/IV 2) Tony/Rik/Lyndon/??? discuss and hopefully agree a merger of VI, III/IV and V. There is no current suggestion to attempt to merge in I. I will call this Proposal X, for now. 3) if [ merger agreed in 2) ] then Draft contains Proposal I Proposal X else Draft contains Proposal I Proposal III/IV Proposal V Proposal VI fi 4) The false branch of 3) can be modified by a pair merger if we find that two of the three extant proposals can be merged but not all. There is currently also one mini-proposal from Tony dealing with layerability and tag selection. I understand that there will be one further mini-proposal from Tony dealing with threads. I sincerely apologise for having been the prime creator of confusion. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:10:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15088; Mon, 22 Mar 93 07:10:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04660; Mon, 22 Mar 93 07:09:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:09:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04652; Mon, 22 Mar 93 07:09:52 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00212 (5.65c/IDA-1.4.4 for ); Mon, 22 Mar 1993 07:09:47 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA24691; Mon, 22 Mar 93 12:09:44 GMT Date: Mon, 22 Mar 93 12:09:44 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303221209.AA24691@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00429; Mon, 22 Mar 93 12:06:23 GMT To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 22 Mar 93 11:32:10 GMT <14066.9303221132@subnode.epcc.ed.ac.uk> Subject: MPI-CONTEXT: Reference Point Content-Length: 599 Lyndon, since I've been rather busy with other things (which are of more immediate import to Meiko), I haven't been actively contributing to the debate. (Though I have been skimming, and filing all of the mail). Is it possible to pin down (or send out once again) the definitive versions of each proposal. (A copy of the table in your previous mail with a mail date and source for the top copy of each would do, alternatively a clean current copy of each...) Your last mail gives the references, but without following all of the history it's hard to get to the top copy of each. Thanks -- Jim From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:18:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16848; Mon, 22 Mar 93 07:18:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04947; Mon, 22 Mar 93 07:17:51 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:17:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04939; Mon, 22 Mar 93 07:17:45 -0500 Date: Mon, 22 Mar 93 12:17:35 GMT Message-Id: <14115.9303221217@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal Comments To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear Colleagues In the previous email "Reference Point" I sent out a summary of where the proposals are. I have been religously keeping "clean" copies of proposals in LaTeX form, and collating/embedding comments as LaTeX comments. I will now circulate the LaTeX source of each live proposal, with collated and up-to-date comments streams embedded therein. I will circulate PostScript (in which our comments do not appear) on request. Please, delete all previous circulations from your memory :-) First, I explain the comment notation I used. Reader comment lines begin with '%'. First line of block is %[reader-name] Reader comment to comment lines are considered literally as comments to comments and therefore begin with %%[reader-name]. This is a comment tree, as is the '%' indent tree of the LaTeX source. This has been time consuming and I do not propose to keep receiving comments and embedding in this fashion - things are more fluid than I had expected. Three LaTeX files to follow, in turn. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:18:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16981; Mon, 22 Mar 93 07:18:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04963; Mon, 22 Mar 93 07:18:29 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:18:28 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04954; Mon, 22 Mar 93 07:18:18 -0500 Date: Mon, 22 Mar 93 12:18:13 GMT Message-Id: <14121.9303221218@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal I - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal I\\MPI Context Subcommittee} \author{Marc~Snir} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Written by Marc Snir % Edited by Lyndon J. Clarke % March 1993 % \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \chapter{Proposal I} \section{Contexts} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \subsection{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \subsection{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \subsection{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \subsection{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \section{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \subsection{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } % % END "Proposal I" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:20:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17043; Mon, 22 Mar 93 07:20:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05007; Mon, 22 Mar 93 07:19:37 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:19:35 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04984; Mon, 22 Mar 93 07:19:13 -0500 Date: Mon, 22 Mar 93 12:18:58 GMT Message-Id: <14127.9303221218@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal V - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk ---------------------------------------------------------------------- % [Lyndon] General points % -------------- % % 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am % offering if you wish) should attempt to construct a more transparent % presentation before circulation to whole committee, for the % convenience of committee members. %%[Tony] %% I felt that things appear twice, because of summary (good) %% and because of implementation notes at end (confusing) %% %%%[Rik] %%% I agree with both of the above. But let's decide whether %%% Proposal V will be replaced by VI or its equivalent before %%% we do any rewriting. %%%%[Lyndon] %%%% Proposal VI as referred to is the sketch which Rik sent out %%%% suggesting a merger. There is an actual Proposal VI circulated. % 2) I'm not a fan of much of this proposal, although I do indeed like % some of the ideas which it introduces. [On the other hand, I'm not a % great fan of all of the proposal which I wrote. I shall mail self % criticism of my proposal, and may have to write amended or alternative % proposal :-)] %%[Tony] %% Please be more specific. I am having a hard time understanding %% why you really don't like it, Lyndon. If the process model %% were a little less static, and servers were permitted (though %% hopefully bounded in cost), I think we would have an excellent %% proposal. %%%[Rik] %%% See Proposal VI. I'd be happy to see the good ideas adopted %%% and the crap die quietly. %% %%%[Lyndon] %%% I would have thought that the points below make my major %%% objections perfectly clear. Perhaps not. Here they are: %%% a) Paired-exact-match stuff %%% b) Translation of (group,rank) into TID all over the place % % 3) I really like the way in which groups are something like ``frames'' % in which contexts are created. This is conceptually much neater than % duplication of groups. %%[Tony] %% In practice, group subsetting will require groups to be copied, %% otherwise, subgroups will unfairly be penalized by the size %% of their ancestor. %%%[Tony] %%% Right, but I think of that as creating a new group. After all, %%% even the ranking structure is different. %% %%%[Lyndon] %%% I am anticipating that when one or more groups are created by %%% subsetting that, for example if the parent were described by %%% a proces list, then the children will be described by process %%% lists which are distinct sublists of the parent. So each element %%% of the parent list gets copied, exactly once. %%% The difficulty I have is that if a group were to be expanded %%% or contracted, then the ``duplicates'' thereof would no longer %%% be duplicates. Saying that duplicate creates a group bu retains %%% the process list of the old group is conceptually muddy since %%% the new group is a reference to a group, whereas the old group %%% or an even older group must be an actual group. Yuk! Now, if %%% we introduce the concept of a reference to an actual group, %%% and say this reference is unqieuly identified, then is is %%% conceptually sound and this object we describe really is a context. % % 4) I like the idea of pushing information into the group structure. I % have a few qualms with the proposed details --- see specific points. %%[Tony] %% I have more confidence about this idea, and could demonstrate %% by June/July time-frame in Zipcode. %% % % 5) See ``Writing a server in the point-to-point layer of MPI in four % easy steps'' at the foot of the message. %%[Tony] %% This seems like a nice thing. %%%[Tony] %%% The implementation that Lyndon suggests consumes an entire %%% process for the server. There are times when this is OK, %%% but also times when it isn't. E.g., if you have a divide-and- %%% conquer algorithm that really wants 2^i non-server processes, %%% and you're working on a machine with 2^n processors that %%% doesn't support multiple processes per processor, then %%% some of the users will get upset that they can only use %%% half the machine. A year and a half ago, I got flamed for %%% making a suggestion like this in connection with the Delta. %%% (And now I suppose I get flamed for using the Delta as an %%% example again... Maybe if I use the CM5...) %% %%%[Lyndon] %%% You are too kind :-) %% \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %%[Tony] %% Alternatives, Lyndon? %% %%%[Lyndon] %%% I don't suppose for one minute that you will like this, but I really %%% would suggest that in this case a group descriptor registry may %%% be appropriate. %[Tony] % Seems doable. % %%[Lyndon] %% But usable with some grief, as above. \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %% [Tony] %% Me too. %%%[Rik] %%% Progress!! %[Tony] % Seems doable. % \item There are restrictions that permit groups to be layered on top of pt-pt. %[Tony] % I don't understand the word 'restriction' here. % Restriction of what. % %%[Lyndon] %% Rik is speaking of the pair-exact-match stuff you see later on. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. %[Tony] % What does "fast" mean. % %%[Lyndon] %% Fair question! \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. %[Tony] % OK, the above two items are identical to what Zipcode % provides in practice, but people have argued that groups % might be created/deleted more often in some apps, and % that these apps ought to be supportable % %%[Lyndon] %% In our work group creation/deletion is an infrequent operation %% and we are happy to live with reasonable cost for this operation. %% I think Marc Snir is thinking abour a different group model in %% group created/deletion is frequent. %% Perhaps we should provide both or neither (even handedness principle). \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %% [Tony] %% 1) It is not obvious that TIDs should be restricted to 32 bits. %%%[Lyndon] %%% I did not imply that they were. %% 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %% I favor a whole word for a context, despite other limits, %% just to make things simpler. %% %%%[Lyndon] %%% ditto %% Internet addresses are going to get augmented from 32 to ??? bits %% is it reasonable to assume that certain MPI implementations might %% incorporate such internet addresses as TIDs (in future), %%%[Lyndon] %%% No, it is not reasonable, in the least. And both you and Rik appear %%% to be ignoring the possibility that the process descriptor could %%% be required to store routing data of significant length. Therefore %%% the sensible thing to do is use a process descriptor which in %%% practice might be a table index --- fits into integer for sure --- %%% the value of which is process local for sure, and the table %%% contains the real process identifier of implementation defined size. %% Opacity is partially violated if we say how big the data type is??? %%%[Lyndon] %%% I understand the point you make, but it gets blown away by the %%% point I have just made in reply to you. %[Tony] % Yes, and I want at least 32-bits of message tag. % %%[Lyndon] %% Yes, and I want exactly zero bits of message tag. %% I'll just keep quiet about message tags. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %% %%[Tony] %% where does this get us??? %% Remember, we have to choose on some things, so we can have something %% to present in Dallas. Is there an important difference here? %%%[Lyndon] %%% For sure their is an important difference. See comment above. %% %% TIDs are global entities. Is structure assumed to be global; %% in a truly opaque system, some TID component would have to be %% fixed, but the rest could vary structurally... %% %[Tony] % This could be difficult, in practice, if one mails a % message to one's own process, and MPI is smart enough % to optimize. % \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %% %% Yes, contexts are global; group identifiers are just pointers %% typically, to data structures, describing %% %% 1) a groups context %% 2) group members and their ranks (mappings, inverses, %% cached, hashed, unscalably stored, etc) %% 3) TID-to-rank map and inverse (see possibilities in 2) %% 4) A set of fixed global operations, accepted as standard, %% an accessible in O(1) time. Possibly, each %% such operation should be a method, so that %% a parameter block can be passed with it. Zipcode %% supports the Method type to do this. %% This is effectively a cache for some parts of item #5 %% ... %% 5) An AVL or similar tree of extensible operations. %% New operations are registerable by the user. These %% tags are unique within a group, a specify an operation %% i) pre-defined by MPI (in which case it can be cached %% in 4 %% ii) alternative operations (even if they do something %% standard, that are wanted to be accessed by %% name) This name is group unique. %% %% A mechanism for DO_METHOD_FROM_GROUP(name,....) %% or GET_METHOD_FROM_GROUP(name,...) %% and SET_METHOD_IN_GROUP(name,...) are clearly needed. %% %%%[Rik] %%% The model of contexts used in this proposal is that the %%% context value has global scope. It doesn't have to be that way. %%% The Snir, Lusk, and Gropp proposal could be implemented with each %%% processor contributing a different context value to the context %%% descriptor. E.g., process 3 says "if you want to talk to me %%% in this context, you have to use context value 7", while process %%% 4 says to use context value 2. I don't know whether there are %%% advantages to that extra flexibility. %[Tony] % Sounds good. % \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %%[Tony] %% Yes, that is included, I believe, in concept. %%[Rik] %% Right. \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %%[Tony] %% Stripping is extremely controversial aspect, and arbitrary. %% If the recipient has the methods with the same name, then %% a new rendezvous could be accomplished at the far end %%%[Rik] %%% See my "Proposal VI" note for responses to this. %[Tony] % In Zipcode 1.0, we allow multiple global operations % to be provided on a message-class (eg, grid-oriented messages) % The identifiers for these possible operations are user-specified % presently, but the "names" of the global operations are fixed % at compile-time. % % That means that there is O(1) time to find combine, fanout, send, % etc, on a group-wide scope. However, other operations cannot % be accessed in O(1) time (they are not in the opaque structure). % % The same mechanism used by Zipcode to allow multiple methods for % combine to be registered by the user, could also allow extensibility % just like Rik describes, with little effort. We use AVL trees. % % In fact, I will add this to Zipcode 1.x. Why say this? It is % not far from existing practice, and I have a lot of the machinery % in place already, and I am confident that it is useful. % \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %%[Tony] %% Yes, these are all options. Fastness seems to be an important %% issue. If translation is very expensive, none of the "good" %% features will be used. %%%[Rik] %%% I was actually trying to be nice to the service process %%% people, giving them a designated hook where they could be %%% slow and I could work around them via the cacheing mechanism. %%% I'd really rather just specify it as being fast. %[Tony] % This seems to be a serious flaw. It will have to be cached % on an LRU basis, with system/user/both specifying how much % caching is allowed (ie, how much unscalable memory use). % If the first time is expensive, OK, but not the Nth time. % %%[Lyndon] %% Check. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). %[Tony] % OK, I see no problem with this (so far). % \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %%[Tony] %% Again, I think this should be eliminated, and all references %% to this idea should be expunged. It denies the context's %% ability to manage messages. %%%[Lyndon] %%% Check. %%%[Rik] %%% When I wrote this, I was presuming that contexts could %%% not be generated "for free" and that therefore it was %%% a good idea to specify a method by which codes could %%% be written so as to not require them. By the way, %%% you may be interested to note that the sample collective %%% comms defined by Snir and Geist in their recent proposal %%% implicitly rely on exactly the paired-exact-match %%% constraint. As written, those routines would break %%% if preceded by a call to a module that issued a wildcard %%% receive in the same group. (And there's an endnote %%% that suggests they know that.) %[Tony] % NO. This violates the concept of context entirely. % (ie, an oxymoron ... contexts same, but still no need for % tag disambiguation...) % % Use the default group context to establish (cooperatively) % other contexts, and then use these. This is a seriously % bad feature, in my mind. % \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. %%[Tony] %% I agree that context allocation has to be done in sequence. %% That is why I am in favor of providing calls that allow %% groups to get numerous contexts at creation, and then %% cooperatively, but potentially without further communication %% divide them(as they build subgroups, for instance). %% %% I see these as services to be used in building virtual topology %% features, which will then be more widely used by users of MPI. %% %%%[Lyndon] %%% If the context allocations are done in sequence, then I have %%% indicated how they can be done for free. I am getting confused. %%[Rik] %% I think this is right. This counting strategy probably %% places some constraints on how big the context value field %% has to be, but I've incorporated it into Proposal VI. %[Tony] % I do not think we should support the paired-exact-match thing. % \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) %[Tony] % Concur. This suggests many more than "256" % %%[Lyndon] %% The number of contexts in the whole program? Sure 256 is too small! %% The number of contexts in each process? Maybe something like 256 is okay? \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] %[Tony] % This is the Zipcode model. It could say loosely synchronous. % \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %%[Tony] %% Do we get anything by having the root process? %% %%%[Lyndon] %%% No. %%%[Rik] %%% The group partitioning concept has been refined in several %%% other postings to mpi-context, mpi-collcom, and mpi-pt2pt, %%% in which "root" was replaced by a "knownmember" set. %%% The idea is that if you have lots of processes joining a %%% group, they don't have to know who all their compatriots %%% are, but they should know at least one that they all have %%% in common, who can serve to organize the communications. %[Tony] % I don't understand the thread issue here. % %%[Lyndon] %% If two threads are concurrently partitioning the same group, you %% need to disambiguate the partition operations. Analagous to %% concurrent collective operations or nonblocking. \item Collective communication routines are called by all members of a group in the same order. %[Tony] % Yes. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. %[Tony] % What does caching really imply here ??? Help. % %%[Lyndon] %% Dunno. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %%[Tony] %% I think that non-blocking collcomm is moribund in MPI1 or %% else MPI1 is moribund. :-) %% %%%[Lyndon] %%% Check. %%%[Rik] %%% Nicely put. %[Tony] % I think that contexts are really important in this case, % to keep things straight, but that non-blocking collcomm should % be omitted from MPI1 (cf, Geist). Sequencing supports % a sufficient disambiguation, as long as the entire group % is always the participant in operations. That is, you have % to form subgroups, with new contexts, to do global ops on % subsets. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %%[Tony] %% Too many negatives here for me to parse :-) %[Tony] % Why aren't servers needed to create contexts. Where do they % come from? If you rely on the fact that INITIAL will do % a loosely synchonous cooperative operation each time a new % context is needed, then a simple (easily implementable server, % or fetch-and-add remote access) is replaced by a more rigid % computation model. % % If we can get rid of this disagreement, me might be able to % reduce our total proposal space by one whole proposal. % \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %%[Tony] %% Context allocation need not be free provided it can be made cheap, %% or cheap enough. %% %% If one knows one will need several, then a single call could %% provide such contexts, amortizing overhead. This is likely when %% bulding grids (ie, virtual topologies) in Zipcode, so it is %% true in existing practice. %% %% One should recognize the need for layering virtual top. calls %% on top of these calls, then these calls may appear painful, %% but perhaps they would be less used. Some users will use the %% provided virtual topology calls, others will prefer their own. %% Both will have equal power (see also,separate note on layerability). %% %% If getting N contexts is a send-and-receive, plus a reactive server, %% then this is reasonably light weight,provided that hundreds of %% messages, or global operations ensue thereafter. We can know in %% advance how heavy weight the context server will be. %% %% if an implemention can use some locations of remote memory, with %% fetch and add, or locks, to achieve contexts, then this is even %% cheaper, in principle. %% %% Despite Jim's earlier insistence that context numbers be kept to %% 256 or so, I think that this number should be much larger, so that %% much less efort goes into returning contexts, and so on, except %% occasionally, by processes. Otherwise, a new kind of overhead, %% get-rid-of-context-because-I-am-out ensues, or programs block %% until contexts become available, offering the possibility of %% deadlocks. %% %[Tony] % If contexts are being used very dynamically, how are they being % assigned, kept, released, reissued without a server? Sorry if % I missed something, but I don't see it, without a restrictive % SPMD model of computation (Zipcode obviates its server for the % SPMD model, for instance). %%[Lyndon] %% MPI stinks of SPMD. I wouldn't mind if MPI would just say SPMD. \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %%[Tony] %% Unscalability (of a limited form) should be permitted/selectable %% by user, to use as much per-node memory as the user wants, to reduce %% communication. %% %%%[Rik] %%% Besides which, the system will do this anyway if (group,rank) %%% translations are required to be fast. All I'm doing is moving %%% it to the user level. %[Tony] % Can you clarify this with examples. % \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. % %%[Tony] %% I see,but what he means here is that there is no unpredictable %% translation cost because we do not write (group,rank) in pt2pt %% calls. So, there is some validity to the statement. %%%[Lyndon] %%% However he ignores that the TID might require translation the %%% cost of which might be unpredictable. This because the TID has %%% a global value and cannot therefore hold process local information %%% such as ``how do I route to that process''. %%%[Rik] %%% Tony has it right. %[Tony] % I like this, of course. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %%[Tony] %% Yes, that is why I have one in Zipcode. I wish Zipcode were %% on netlib today, so you could try it. Well, we are writingthe %% manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %%[Tony] %% I lost something here, but I agree with the sense. The word %% seems is subjective,and should disappear from our discussions, %% as much as seems prudent, anyway :-) %%%[Rik] %%% Silly me -- trying to call out an opinion that might be %%% disagreed with. It occurred to me when I wrote the "s" word %%% that I'd probably be better off to just follow standard practice %%% and state my opinions as fact. %[Tony] % Well, is point-to-point group oriented. Not. %%[Lyndon] %% Check. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %%[Tony] %% Yes,awkward, because it is nice to escape the TID realm and %% work within the (albeit simple) abstraction of group,rank. %% When layering virtual topologies on this, it would be so nice %% to write them to a group,rank syntax, not enforcing TID mappings %% everywhere. %%%[Rik] %%% I happen to agree with this. But even though it's late, I can't %%% resist pointing out that there's not a shred of scientific %%% evidence that these opinions are in fact true. %[Tony] % I think it is awkward. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %%[Tony] %% Indeed. %%%[Lyndon] %%% More so than you think, I think! %[Tony] % OK, but one can form a new group, as I have argued before. % Use the "awkward" pt2pt to get the right info shared between % group leaders, make the new group, use unawkward collective % operations on new group (with new context). %%[Lyndon] %% This is only one model of group-group interaction, which in my %% experience and understanding really is still steeped in SPMD. %% Please consider the examples of non SPMD group usage which I %% mailed out. You can say to me - oh, you shouldn't do this kind %% of thing with MPI, Lyndon - if you like, if you believe that. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. %[Tony] % This again is in practice, in Zipcode. % \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %%[Tony] %% Yes, and I think that loosely synchronous operations can maintain %% coherency, in practice. That is, no operations that modify the %% group descriptors (other than cached lookup info) are permitted, %% without loose synchronization. %% This is nasty in that is would prohibit sending descriptors to %% processes not part of the group, so it is a clear trade-off. %% Perhaps such send-to-non-group-member operations could stipulate %% that this group information is somehow ephemeral, and that they %% need to join a new group to keep useful information over time??? %% %%%[Lyndon] %%% I agree that the (loosely) synchronous semantics remove coherency %%% until processes find out about groups of which they are not %%% members. I cannot advocate prohibition of this. I know there is a %%% coherency issue and I think we should let the user deal with it. %%% In practice, the user implements protocols such that if group A %%% communicates with group B, then before group B is destroyed it %%% must have let group A know of its impending doom, unless group A %%% can already have knowledge that this will be the case, which it %%% often can anyway. %[Tony] % Sounds dangerous. What must application do to maintain % coherency, since group descriptors are opaque. % \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. %[Tony] % No, loosely synchronous process model, unless you mean % cooperation of INITIAL at all such join/leave steps. %%[Lyndon] %% Tony, I have difficulty understanding what you mean. If you are %% thinking of group creation by giving a list of group members then %% I can understand. If you are thinking of the partition operation %% then I cannot understand as this operation seems (to me:-) to be %% a synchronisation anyway. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %%[Tony] %% Yes, that is exactly what Zipcode was written to avoid. The %% user wants help managing things like this!!!! %% %% The search, if any, must be MPI-supported, and as efficient as %% possible (eg, AVL trees, hash, partial hash with exceptions). %% % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %%[Tony] %% This is why I remain a strong advocate of (group,rank) %% addresssing in pt2pt. %% %%%[Lyndon] %%% Et moi! %[Tony] % No, you can't know the group or rank in group of sender. % If there were one context per group (isn't that so here?), % then all you need is the rank. With TID_TO_RANK_IN_GROUP % operation, this could be provided, but no wildcarding % or receipt selectivity could be done at this level. % %%[Lyndon] %% Where a member of group A sends to a member of group B then the %% receiver in group B really does often want to know the rank of the %% in sender in group A. This is not hypothetical, we program this way. \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %%[Tony] %% I addressed this in my Step-1 remarks. Please see that. BELOW %[Tony] % No, it just does not provide guarantees that certain kinds % of applications will run OK. (ie, those that do group % creation/deletion relatively often). Zipcode has assumed % that such operations would be relatively seldom. Thus, I do % not quibble that this is a reasonable choice,but a fairer % way to say this is that it may be difficult to support such % applications. That reveals an issue to be studied more. % %%[Lyndon] %% Check. On performance specification this belongs in the %% implementation profile and the specification actually %% cannot say anything about performance. %% \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. %% [Rik] %% I thought I said that. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %%[Tony] %% Thank you. These points are both well taken (ie these two paragraphs) %%%[Tony] %%% See my comments near the start of this message, regarding %%% the proposed server implementation. What I should have said %%% was that MPI did not provide enough capability to construct %%% a server without consuming a processor, since it neither %%% provides an interrupt-receive function nor specifies that %%% processors be able to time-share between multiple processes. %[Tony] % Perhaps they should do. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] %[Tony] % Dump this. \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %%[Tony] %% Also,see what I wrote in my Step-1 comments. BELOW. %% I believe O(log(P)) is still possible. %% %%%[Rik] %%% I'd be interested in seeing the O(log(P)) algorithm, %%% sometime after MPI quiets down. %%%%[Lyndon] %%%% Me too! %[Tony] % No, a non-deterministic broadcast can be used, with a token. % This requires a token server. Again, implementable with fetch+ % add on most systems, or a light reactive server. % % Once the non-deterministic broadcast has finished, a fanin/collapse % is done to the original root, which then frees the token. % \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %%[Tony] %% Please see my Step-1 comments. %[Tony] % Why not just given in and allow the server. % I don't like the paired-exact-match constraint AT ALL. % %%[Lyndon] %% Ni moi! \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) %[Tony] % Seems OK, but why need the paired-exact-match thing again. % Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) %[Tony] % Again, OK, but I want to see this work without the paired-exact- % match, if possible. Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. %[Tony] % So, now, you have concurred with my (previously flamed) idea % that group construction/destruction should be realizable using % pt2pt, just like global operations should do. I like this % because 1) it is explicable to the implementor, 2) it allows % simple intitial implemtations, 3) it sets some ideas for how % much these things will cost [upper bound]. % %%[Lyndon] %% I still do not concur. It is restrictive on what we can do. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} %[Lyndon] % Writing a server in the point-to-point layer of MPI in four easy steps %%[Rik] %% by potentially consuming an entire processor %%%[Lyndon] %%% Oh dear, how sad. I know what a pain it can be when you have one %%% thumping great number cruncher which cant multitask, and its sits %%% there as a do nothing server all day long. But look, it's a do %%% nothing server, so it *can* run in the user environment like on %%% the ``host'' or ``login'' processor. Look again, this is because %%% of the ``cannot multitask'' - let us hope that is history :-) % ---------------------------------------------------------------------- % % 1) Partition the INITIAL group into two groups. A singleton group, % SERVER, and a group CLIENT which contains all of the other processes. % % 2) The single process in SERVER group records its TID. % % 3) The processes in INITIAL group allocate a context SERVICE which % they remember either in the group cache or static data or something. % % 4) Use a broadcast in INITIAL group with ``sender'' as the one process % which is also in SERVER group, and the ``receivers'' as the (many) % processes which are also in CLIENT group, in the SERVICE context, in % order to disseminate the TID of the server process. % % [Fanfare] a server process is in place as is a dedicated context for % the purposes of messages required to implement the service. % % [Observation] the mpi point-to-point initialisation can do this % automatically. %%[Tony] %% Zipcode's postmaster general works in this way, more or less. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:20:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17057; Mon, 22 Mar 93 07:20:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05062; Mon, 22 Mar 93 07:20:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:20:15 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05009; Mon, 22 Mar 93 07:19:57 -0500 Date: Mon, 22 Mar 93 12:19:43 GMT Message-Id: <14133.9303221219@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal VI - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal VI\\MPI Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal VI" % Written by Lyndon J. Clarke % March 1993 % \newcommand{\LabelNote}[1]{\label{vi:note:#1}} \newcommand{\SeeReferNote}[1]{{\bf See~Note~\ref{vi:note:#1}.}} \newcommand{\LabelSection}[1]{\label{vi:sect:#1}} \newcommand{\ReferSection}[1]{{\bf Section~\ref{vi:sect:#1}.}} \chapter{Proposal VI} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction} This chapter proposes that communication contexts and process groupings within MPI appear as related concepts. In particular process groupings appear as ``frames'' which are used in the creation of communication contexts. Communications contexts retain a reference to, and inherit properties of, their process grouping frames. This reflects the observation that an invocation of a module in a parallel program typically operates within one or more groups of processes, and as such any communication contexts associated with invocations of modules also bind certain semantics of process groupings. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of MPI, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language for only purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The notes assumes knowledge of the proposal text and are therefore best examined after familiarisation with that text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the text of the proposal. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal} \LabelSection{proposal} This section presents the detailed proposal, discussing in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes} \LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and MPI does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{process:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of MPI services. MPI provides a procedure which returns the own descriptor of the calling process. For example, {\tt pd = mpi\_own\_pd()}. \subsubsection*{Process Creation and Destruction} This proposal makes no statements regarding creation and destruction of processes. \SeeReferNote{dynamic:processes} \subsubsection*{Descriptor Transmission} The value of a process descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %%[Lyndon] %% Since it is an integer one cannot prevent the user from passing it. %% The discussion notes should help answer why is is an integer. More %% here. %% We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. %% Because rank will be an integer, then pd must also be an integer, %% so I write that all descriptors are integers for consistency beteen %% the different descriptors. %% MPI provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_PD\_TYPE}. MPI provides a process descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %%[Lyndon] %% I don't understand what you don't get. A registry service is just %% an easier (nonscalable, okay) way for concurrent entities to hook up %% with one another, rather than having some common ancestor send descriptors %% around in messages. %% In fact I don't really need a group or process registry service, as mentioned %% in the discussion section (yes, I know, not well presented), but %% I do need a context registry service, and I'm being consistent %% between different kinds of descriptors again. %% Suggestive syntax for a registry service is pretty boring, so %% I skipped it. %% Note that receipt of a process descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_pd(pd)}. The user is not allowed to invalidate the process descriptor of the calling process. \SeeReferNote{process:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %%[Lyndon] %% Acquisition of a descriptor requires an allocator, and consumes %% resource. In such cases it is good practice to provide a %% deallocator which frees resource. %% % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %%[Lyndon] %% Unless I am missing the additional point in this comment, I can't %% see the problem. Probably poor presentation of Proposal I++ :-) %% \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding processor descriptor attributes. \subsection{Process Groupings} \LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to?) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. \SeeReferNote{dynamic:groups} The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to the descriptor of the $N$ processes composing the grouping. The structure of a process grouping is defined by a process grouping topology and this proposal makes no further statements regarding such structures. %[Tony] % It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % %%[Lyndon] %% I am deliberately making weak statements about topology while %% acknolwedging the the process topology subcommittee. %% \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{group:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the process grouping until the termination of MPI services. \SeeReferNote{group:blobs} MPI provides a procedure which returns the descriptor of the home group of the calling process. For example, {\tt gd = mpi\_home\_gd()}. \SeeReferNote{own:group} \subsubsection*{Group Creation and Deletion} MPI provides a procedure which allows users to dynamically create one or more groups which are subsets of existing groups. For example, {\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} partition an existing group {\tt gda} into one or more distinct subsets. This procedure is called by and synchronises all members of {\tt gda}. MPI provides a procedure which allows users to dynamically create one group by explicit definition of its membership as a list of process descriptors. For example, {\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. %[Tony] % loosely synchronous % %%[Lyndon] %% Do we mean the same thing? the constructors require each member %% if the object under construction to participate in the construction, %% and return a descriptor for the constructed operation. Therefore %% no member can terminate construction before all other members have %% commenced, at least. %% %[Rik] % Why such strong synchronization?! % If this has the same synchronization properties as creation, % then it won't return until all members have made the call. % But that means you actually have to sync everybody, which % implies 2 log(P) messages. Isn't it enough to just require % a loosely synchronous call, and use a counting strategy> % MPI provides a procedure which allows users to delete a created group. This procedure accepts the descriptor of a group which was created by the calling process and destroys the identified group. For example, {\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. MPI provides additional procedure which allow users to construct process groupings which have a process grouping topology. \subsubsection*{Descriptor Transmission} The value of a group descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %%[Lyndon] %% Since it is an integer one cannot prevent the user from passing it. %% The discussion notes should help answer why is is an integer. More %% here. %% We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. %% Because rank will be an integer, then pd must also be an integer, %% so I write that all descriptors are integers for consistency beteen %% the different descriptors. %% MPI provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_GD\_TYPE}. MPI provides a group descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %%[Lyndon] %% I don't understand what you don't get. A registry service is just %% an easier (nonscalable, okay) way for concurrent entities to hook up %% with one another, rather than having some common ancestor send descriptors %% around in messages. %% In fact I don't really need a group or process registry service, as mentioned %% in the discussion section (yes, I know, not well presented), but %% I do need a context registry service, and I'm being consistent %% between different kinds of descriptors again. %% Suggestive syntax for a registry service is pretty boring, so %% I skipped it. %% Note that receipt of a group descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_gd(gd)}. The user is not allowed to invalidate the own group descriptor of the process or the group descriptor of any group created by the calling process. \SeeReferNote{group:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %%[Lyndon] %% Acquisition of a descriptor requires an allocator, and consumes %% resource. In such cases it is good practice to provide a %% deallocator which frees resource. %% % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %%[Lyndon] %% Unless I am missing the additional point in this comment, I can't %% see the problem. Probably poor presentation of Proposal I++ :-) %% \subsubsection*{Descriptor Attributes} MPI provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identifier group. For example, {\tt rank = mpi\_group\_rank(gd)}. MPI provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, {\tt size = mpi\_group\_size(gd)}. MPI provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, {\tt pd = mpi\_group\_pd(gd, rank)}. \SeeReferNote{pd:to:rank} MPI provides additional procedures which allow users to determine the process grouping topology attributes. %[Tony] It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % %%[Lyndon] %% I am deliberately making weak statements about topology while %% acknolwedging the the process topology subcommittee. %% \subsection{Communication Contexts} \LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping is used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{context:identifiers} \SeeReferNote{descriptor:cache} The creation of MPI process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains its own context until MPI process grouping deletion. MPI provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, {\tt cd = mpi\_own\_cd(gd)}. \SeeReferNote{own:context} \subsubsection*{Context Creation and Deletion} MPI provides a procedure which allows users to dynamically create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, {\tt cd = mpi\_context\_create(gd)}. This procedure is called by and synchronises all members of {\tt gd}. \SeeReferNote{dynamic:contexts} %[Tony] % loosely synchronous % %%[Lyndon] %% Do we mean the same thing? the constructors require each member %% if the object under construction to participate in the construction, %% and return a descriptor for the constructed operation. Therefore %% no member can terminate construction before all other members have %% commenced, at least %% MPI provides a procedure which allows users to destroy created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes that context identifier. For example, {\tt mpi\_context\_delete(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. \subsubsection*{Descriptor Transmission} The value of a context descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %%[Lyndon] %% Since it is an integer one cannot prevent the user from passing it. %% The discussion notes should help answer why is is an integer. More %% here. %% We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. %% Because rank will be an integer, then pd must also be an integer, %% so I write that all descriptors are integers for consistency beteen %% the different descriptors. %% MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_CD\_TYPE}. %[Tony] % I don't get all of this. Why? % %%[Lyndon] %% I don't understand what you don't get. A registry service is just %% an easier (nonscalable, okay) way for concurrent entities to hook up %% with one another, rather than having some common ancestor send descriptors %% around in messages. %% In fact I don't really need a group or process registry service, as mentioned %% in the discussion section (yes, I know, not well presented), but %% I do need a context registry service, and I'm being consistent %% between different kinds of descriptors again. %% Suggestive syntax for a registry service is pretty boring, so %% I skipped it. %% MPI provides a context descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a context descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_cd(cd)}. The user is not allowed to invalidate the own context descriptor of a group or the context descriptor of any context created by the calling process. \SeeReferNote{context:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %%[Lyndon] %% Acquisition of a descriptor requires an allocator, and consumes %% resource. In such cases it is good practice to provide a %% deallocator which frees resource. %% % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %%[Lyndon] %% Unless I am missing the additional point in this comment, I can't %% see the problem. Probably poor presentation of Proposal I++ :-) %% \subsubsection*{Descriptor Attributes} MPI provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, {\tt gd = mpi\_context\_gd(cd)}. \subsection{Point-to-Point Communication} \LabelSection{point2point} This proposal recommends three forms for MPI point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a Send operation of form X cannot match with a receive operation of form Y, which requires that the form is embedded into the message envelope. The three forms are described followed by considerations of uniform integration of these forms in the point-to-point communication section of MPI. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. \SeeReferNote{null:context} Message selection and addressing are expressed by {\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. Send supplies the {\tt pd} of the receiver. Receive supplies the {\tt pd} of the sender. Receive can wildcard on {\tt pd} by supplying the value of a named constant process descirptor, e.g. {\tt MPI\_PD\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. \SeeReferNote{closed:context} Message selection and addressing are expressed by {\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. Send supplies the {\tt cd} of the receiver and sender, and the {\tt rank} of the receiver. Receive supplies the {\tt cd} of the sender and receiver, and the rank of the sender. The {\tt (cd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt cd}. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. \SeeReferNote{open:context} Message selection and addressing are expressed by {\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. Send supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. Receive supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The {\tt (rcd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt lcd}. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD}, in which case Receive {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point section of MPI by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with the form. This is at the expense of multiplying the number of Send and Receive procedures by a factor of three, and some difficulty in details of the current point-to-point working document which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options each have advantages and disadvantages. \paragraph*{Option i:} The framework of the open context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii:} The framework of the closed context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as {\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the {\tt (lcd, rcd)} pair at the expense of introducing some trickery. We can define a mechanism which ``globs'' together these two fields onto in an identifier which Send and Receive can distinguish from a context identifier and treat as the shorthand notation. Then we should also define a mechanism by which a receiver which has completed a Receive with wildcard on {\tt rcd} is able to determine the valid context descriptor of the sender. This is a inconvenient. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % %% [Lyndon] %% I dislike this also. Of course it is all to get around extending %% the syntax, it stinks of that. %% \paragraph*{Option iii:} The framework of the null context form is adopted and extended. The null context form is expressed as {\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is clumsy. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % %% [Lyndon] %% I dislike this also. Of course it is all to get around extending %% the syntax, it stinks of that. %% \subsection{Collective Communication} \LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context and frame in which they are to operate. MPI does plan to describe symmetric collective communication operations. It is not possible to determine whether the proposal is sufficient to allow implementation of the collective communication section of MPI in terms of the point-to-point section of MPI without loss of generality, since the collective operations are not yet defined. %[Tony] Check, but it is desirable that they be so writable, so we will % have to watch. % Assymetric collective communication operations, especially those in which the sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a ``glob'' form) which identify the contexts and frames in which they are to operate. MPI does not plan to describe assymetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model and comprise communicative funtionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of MPI. % % END "Proposal" %----------------------------------------------------------------------% % [Tony] % So, I gather that a set of groups is passable to a collcomm, % and a pair is passable to a pt2pt. That is neat, but it should % still be a separate data structure, with separate calls than % the intra-group version (at least for the pt2pt calls). % %% [Lyndon] %% Currently, I am only thinking of passing either singlets or duplets %% to point-to-point and collective. %% I was trying to avoid separate - extra - calls to point-to-point %% because that is already very large and there is a swell of concern %% about the size thereof. %% %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion} \LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents arguments for and against the MIMD model annex, and to some extent explores the consquences of these arguments for MPI in a wider sense. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model, in which process execute in a stricly SIMD fashion. The material describing processes in \ReferSection{processes} is simplified: \begin{itemize} \item processes have identical instruction blocks and different data blocks; \item process descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query and group descriptor attribute query is capable of providing access to all process descriptors); \item dynamic process models are not considered. \end{itemize} The material describing process groupings in \ReferSection{groupings} is simplified: \begin{itemize} \item group descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query capable of providing access to all group descriptors.); \item the own process grouping explicitly becomes a group containing all processes. \end{itemize} The material describing communication contexts in \ReferSection{contexts} is simplified: \begin{itemize} \item context descriptor transmission and registry become unnecessary. \end{itemize} The material describing point-to-point communication in \ReferSection{point2point} is simplified: \begin{itemize} \item the open context form becomes redundant; \item uniform integration ``Option i'' is deleted, and ``Option ii'' loses ``globs'' thus becoming simple enough that ``Option iii'' need not be further considered. \end{itemize} The material describing collective communication in \ReferSection{collective} is simplified: \begin{itemize} \item there is no possibility of collective communication operations spanning more than context. \end{itemize} \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as MPI are not suited to fine grained function driven or data driven programming. The annex is intended to provide expresive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in EXPRESS and PARMACS, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or vice versa!). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computating. The debate is whether or not message passing approaches such as MPI (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD parallel prgrams. The positive argument is that communications between SPMD modules requires high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development and standardisation procedures for distributed operating systems. The latter argument is comparable with the argument for MPI versus parallel compilation. \subsection{Notes} \LabelSection{notes} \begin{enumerate} \item\LabelNote{integer:descriptors} Descriptors are a plentiful but bounded resource. They are expressed as integers for two reasons. Firstly, this expression facilitates uniform binding to ANSI~C and Fortran~77. Secondly, there is a later convenience in ensuring that process descriptors in particular are expressed in integers, and I made all the descriptors are integers for consistency. In practice the descriptor could be an index into a table of objects description structures or pointers to such structures. \item\LabelNote{descriptor:cache} It is possible to allow the user to save user defined attributes in descriptors, as Littlefield has suggested. Such attributes should not be communicated in either descriptor transmission or the descriptor registry. \item\LabelNote{process:identifiers} The process descriptor is not a global unique process identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional process information in implementations. For example, the process identifier of a receiver may not be sufficient to route a message to the receiver, and this information can be referenced from the process descriptor. \item\LabelNote{group:identifiers} The group descriptor is not a global unique group identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependeent, and enables more efficient access to additional group information in implementations. For example, the size of the group and the map from member ranks to process descriptors can be referenced from the group descriptor. \item\LabelNote{context:identifiers} The context descriptor is not a global unique context identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional context information in implementations. For example, the group descriptor of the frame can be referenced from the context descriptor. \item\LabelNote{dynamic:processes} The proposal does not prevent a process model which allows dynamic creation and termination of processes however it does not favour an asynchronous process creation model in which singleton processes are created and terminated in an unstructured fashion. The proposal does favour a model in which blobs of processes are created (and terminated) in a concerted fashion, and in which each group so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying worloads. The author suggests that concerted blob expand and contract operations are most suitable for this purpose than asynchronous spawn/kill operations. \item\LabelNote{registry:check} There should probably also be a registry operation which checks whether a name has been registered without the possibility of blocking the calling process indefinitely. The same registry could be used for process, group and context descriptors. \item\LabelNote{process:coherency} In the static process model it is assumed that processes are created in concert, and that termination of one process implies termination of all processes. In this case there is no coherency problem associated with processes and process descriptors. In a dynamic process model there is a coherency problem which processes and process descriptors since a process can retain the descriptor of a process which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{dynamic:groups} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The author suggests concerted group expand and contract operations are more generally useful than asynchronous join/leave operations. \item\LabelNote{group:blobs} MPI has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is intended to be a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and different blobs have different own process groupings. An SPMD program is a single blob. A host-node program composes two blobs, the node blob and the host blob (which is a singleton). There is a sense in which a blob has a locally SPMD view. \item\LabelNote{own:group} This procedure looks like a process descriptor attribute query. \item\LabelNote{group:coherency} Dynamic processes introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Dynamic processes potentially introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Group transmission introduces a group and group descriptor coherenncy problem since a process can retain a group descriptor of a group which has been identified group. The proposal expects the user to ensure coherent usage. \item\LabelNote{pd:to:rank} The proposal did not include a function to convert a {\tt (gd, pd)} pair into a rank. It is suggested that this inverse map is allowed to be arbitrarily slow. \item\LabelNote{dynamic:contexts} Marc Snir has described a method by which global unique group identifiers can be generated without use of shared global data. The description of context creation anticipates a similar method for global unique context identifier generation. However the synchronisation requirement of this method makes it unnecessary for context creation. Synchronisation at creation of context within a process grouping frame implies that all processes within the frame perform the same context creations in the same sequence. Therefore the combination of global unique frame identifier and context creation sequence number provides a global unique context identifier without communication. \item\LabelNote{own:context} This procedure looks like a group descriptor attribute query. \item\LabelNote{context:coherency} The retention of a reference to a group descriptor by a context introduces the potential for a context and context descriptor coherency problem, as a context could contain a reference to the group descriptor of a group which has been deleted. This could be circumvented by demanding that all such contexts are deleted before a group is deleted. Context descriptor transmission introduces a coherency problem since a process can retain the descriptor of a context which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{null:context} This is somewhat like and PARMACS and PVM~3. It is general purpose, but not particularly expressive. It does not provide facilities for writers of parallel libraries. \item\LabelNote{open:context} This is somewhat like ZIPCODE and VENUS. It is expressive in certain SPMD programs where noncommunicative data driven parallel computations can be performed concurrently. It provides facilities for writers of SPMD like parallel libraries. \item\LabelNote{closed:context} This is somewhat like CHIMP and PVM~2. It is expressive in certain MIMD programs where communicative data driven parallel computations can be performed concurrently. It provides facilities for MSPMD like parallel libraries. \end{enumerate} % % END "Discussion & Notes" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Conclusion" % \section{Conclusion} This chapter has presented and discussed a proposal for communication contexts within MPI. In the proposal process groupings appeared as frames for the creation of communication contexts, and communication contexts retained certain properties of the frames used in their creation. The author strongly recommends this proposal to the committee. % % END "Conclusion" %----------------------------------------------------------------------% % % END "Proposal VI" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 11:39:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25825; Mon, 22 Mar 93 11:39:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14730; Mon, 22 Mar 93 11:38:27 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 11:38:25 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14722; Mon, 22 Mar 93 11:38:23 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 08:34 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06464; Mon, 22 Mar 93 08:32:36 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA16539; Mon, 22 Mar 93 08:32:32 PST Date: Mon, 22 Mar 93 08:32:32 PST From: rj_littlefield@pnlg.pnl.gov Subject: RE: Rik's comments on Lyndon's Proposal I To: jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, rj_littlefield@pnlg.pnl.gov, tony@aurora.cs.msstate.edu Cc: d39135@carbon.pnl.gov Message-Id: <9303221632.AA16539@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Lyndon says > There is no proposal II. There is however a proposal VI. This > naming/numbering is Tony's suggestion with which I concur. Is this the same or different from the "proposal VI" I sketched last night? For reference, that was (Rik said) > Here's a sketch of a new proposal (VI ?). > > . The functionality of point-to-point communications is per Snir, > Lusk, and Gropp, augmented by my proposed MPI_FORM_CONTEXT to allow > assembling arbitrary collections of processes. (Marc has already > accepted this in a private email to me -- don't know why he > didn't post it.) > > . Performance expectations of point-to-point communications are > explicitly stated as follows: > > - MPI_COPY_CONTEXT does not synchronize the participating processes > and costs significantly less than a point-to-point fanout among > them (e.g., it uses a communication-free counting strategy); > > - all other context formation routines cost no more than if they > were implemented using a single fanin/fanout among the > participating processes; > > - translation of (context,rank) to absolute processor ID costs no > more than if it were implemented via the lookup table that Snir > suggests. > > . Groups and contexts are not equal. A group consists of a base > context (from which other contexts can be created quickly by > MPI_COPY_CONTEXT), plus topology information, plus my cacheing > facility. Lyndon says > I really am very sorry to have messed you about. I have no idea what this refers to. It would bother me to have MPI produce a bad standard. Not much else does. I would be delighted to have all of my specific suggestions and/or words replaced by something better. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 12:30:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27511; Mon, 22 Mar 93 12:30:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17027; Mon, 22 Mar 93 12:29:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 12:29:14 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17019; Mon, 22 Mar 93 12:29:12 -0500 Received: from newton.sandia.gov (newton.cs.sandia.gov) by cs.sandia.gov (4.1/SMI-4.1) id AA17003; Mon, 22 Mar 93 10:29:10 MST Received: by newton.sandia.gov (5.57/Ultrix3.0-C) id AA00503; Mon, 22 Mar 93 10:30:02 -0700 Date: Mon, 22 Mar 93 10:30:02 -0700 From: mpsears@newton.cs.sandia.gov (Mark P. Sears) Message-Id: <9303221730.AA00503@newton.sandia.gov> To: mpi-context@cs.utk.edu Subject: A new proposal for context and groups. Proposal VI Mark P. Sears Sandia National Laboratories The following proposal for context and group definitions is intended to fill a gap in Tony Skjellum's list. It is most closely related to Rik Littlefield's proposal V. The main difference is that in my proposal, context and groups are completely orthogonal, both in purpose and function. There are a number of advantages to this proposal: 1) The meanings of context and group are simple and it is easy to see how they can be implemented quickly and efficiently. The proposal is also free of chicken vs. egg problems. 2) MPI 1.0 can be closed at the point-to-point level without any reference to groups or group operations. We might actually finish on time. 3) Other proposals which combine aspects of context and group as defined here could be layered on top of this proposal, but not vice versa. Indeed, some of these proposals are very different from each other and it is not clear which is the best. 4) Elsewhere in MPI we have agreed to define low level layers on which higher level stuff can be built. This proposal follows that line of reasoning. 5) This proposal requires no server code and most of the group code is not even parallel. I did a test implementation of groups as defined here and was able to build code for identity groups, permutation groups, linear and bilinear groups, general set groups (Tony's favorite), composition groups, Cartesian products and cartesian subsets (my favorite), all in about 700 lines of code. The group definition really lends itself to object-oriented user extensibility, like X widgets but easier. 6) Groups are easily constructed and destroyed, since no global communication is required. Dynamic groups are not excluded, although they must be used carefully. Since groups have no associated context, there are no resources limiting their construction other than memory and CPU time. To summarize, in this proposal: 1) The purpose of context is to allow different software modules to maintain independence by allowing independence of tag spaces. A context is an integer-valued extension to the tag field which must match exactly for both sender and receiver. Allocation and deallocation of contexts must occur globally and synchronously. Two predefined contexts exist, one for internal use by MPI and one a free-for-all context for application programs. 2) Processes are addressed by their rank within the parallel task. This global rank is fixed and assigned in an implementation-defined way. There must exist environment functions which obtain the topology of the global process assignment (hypercube, mesh, random network, ...) [ This is an area where MPI 2.0 could really break some new ground: define interaction between different parallel tasks, define creation and deletion of parallel tasks, ...] 3) The purpose of groups is to allow the management of subsets of processes in a parallel task. A group is defined simply as an ordered set of unique integers (the elements). Two groups are functionally the same if they define the same ordered set. Groups have no associated context, default or otherwise. They are also free of state. Groups are defined locally by construction, and have no global meaning. A group can be passed from one process to another by passing the information needed to construct it. A process is a member of a group if its global rank is an element of the group. 4) Groups have an implicit topology, defined by the ordering of the elements. Any other ordering can be defined by constructing a new group with the same elements in the new order. There is no need for any other topology function. 5) Any group implementation must define at least the following functions: order(group) -- Number of elements in the group. range(group) -- Upper limit on elements. Lower limit is always zero. iselement(group, element) -- Predicate to determine if element is in a group. rank(group, element) -- Compute rank of element. element(group, rank) -- Compute element given rank. Some other proposals have scamped on the need for being able to compute the rank of an element or the predicate defining whether something is even an element of the group. Using these functions, it is easy to construct a minimum spanning tree or set of binary exchanges for all of the interesting collective communication functions, for any group. This information could be cached with the local group definition. There is no reason to disallow user-defined groups (e.g. dynamic groups). Note that since groups are ordered, the previous and next elements are easily computed using the rank and element functions. 6) MPI collective communication functions which operate on subsets of processors use groups to define the subset of interest, and must be called with the same context, tags(s) and group, by at least all processes which are members of the group. Note that with this definition of group it is the responsibilty of the caller to provide context and tags. Two (or more) such communication functions are guaranteed to complete if they are called in the same order on all processes which are members of both groups (this condition is less stringent for multithreaded environments). Otherwise the program is erroneous. From owner-mpi-context@CS.UTK.EDU Mon Mar 22 19:01:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04642; Mon, 22 Mar 93 19:01:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03825; Mon, 22 Mar 93 19:00:39 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 19:00:38 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03811; Mon, 22 Mar 93 19:00:35 -0500 Date: Tue, 23 Mar 93 00:00:31 GMT Message-Id: <15033.9303230000@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: tum-tee-tum To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear Colleagues A colleage of mine (not of MPI) reviewed Proposal VI and made a number of very sensible suggestions regarding technical details and presentational details, although not technical guts. The outcome is that I must now revise this proposal. I will repost to this group if there is any interest --- can not happen for approx. 18 hours. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 23 10:42:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19318; Tue, 23 Mar 93 10:42:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14088; Tue, 23 Mar 93 10:41:12 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 10:41:12 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14077; Tue, 23 Mar 93 10:41:02 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA01482 (5.65c/IDA-1.4.4 for ); Tue, 23 Mar 1993 10:39:47 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA11198; Tue, 23 Mar 93 15:39:42 GMT Date: Tue, 23 Mar 93 15:39:41 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303231539.AA11198@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05333; Tue, 23 Mar 93 15:36:15 GMT To: mpi-context@cs.utk.edu Cc: snir@watson.ibm.com, rj_littlefield@pnlg.pnl.gov, lyndon@epcc.ed.ac.uk Subject: Context proposals overview Content-Length: 5160 Jim's initial summary/comments on the Contexts proposals Gentlemen, here is my summary and initial comclusions on the three extant Contexts proposals received via Lyndon. Since I have probably misunderstood or misrepresented some of the proposals, please correct me as appropriate. Proposal I (Marc Snir) ---------------------- Key points : A group and a context are identical. All communication (including point to point) is addressed by group and rank. The receiver MUST be a member of the same group as the sender, and explicitly give the group as an argument to the receive function. Group descriptors are not passable to other processors. Creation of a new context involves creation of a new group which is a group synchronising operation. For: Simplicity, it's easy to explain and understand (and probably implement). Insisting on a group for all addressing gives the 0..n-1 rank model for subgroups which aids in importing libraries. Against: The receiver of a message has to both know the group of the sender, and be a part of it. This feels like it makes servers hard, but may be ok. If you keep having to go into the ALL group, then the security which was the whole point of a context is lost. The argument that contexts can be checked at the sender is incorrect because a single process can be in many different groups/contexts, and therefore the group/context must be transferred to ensure correct matching. (Maybe this wasn't the intent of the comments anyway...) I don't understand how to build a group/context using only point to point messages. I still seem to have the bootstrap problem that I need the new context to safely receive the message which will tell me what the new context is. Proposal V (Rik Littlefield) ---------------------------- Key points : A desire to have caching of arbitrary library (or even user) defined entities on the groups. Point to point communication is expected to be addressed by TID and Context. TIDs have global scope, they can be passed around arbitrarily. There is explicit translation from (group,rank) to TID. Group descriptors are local objects, but can be passed around through special mechanisms. Contexts are globally unique, (and expected to be allocated locally through one of the normal bit space partitioned schemes. i.e. processor number // identifier. There appears to be an expectation that the identifiers are managed closely), and can be passed transparently in messages (? is this true ?). Contexts are created inside groups, creating a context is a barrier synchronisation (? is this true ?). For: The caching is good idea, but orthogonal to all the other issues, it could be added just as easily to any of the other proposals The use of TIDs in the point to point removes any issues of group membership from these functions. There is thus a way to achieve any inter group communication which is required (even if it is unpleasant). Against: The complexity of the "paired exact match criterion" is non trivial to explain. Proposal VI (Lyndon Clarke) --------------------------- Key points: ALL descriptors (process, group, context) are process local, but may be sent elsewhere through special mechanisms. Contexts are created inside groups, and are bound to them. The context can be used to refer (indirectly) to the owning group. Creating a context is a barrier synchronisation of the group. Three different forms of process address are discussed for point to point messaging. (Ouch !) For: Can be made to address complex inter group communications Against: The complexity of the whole proposal. The passing of opaque entities with translation is rather unpleasant for people with homogeneous machines. Before this we could ignore all of the type info, now we have to check some of it and take special actions. Jim's initial conclusions... ---------------------------- 1) Marc's approach of insisting that all communications occur inside a group has some simplifying effects. In particular it easily gives the base shift for sub-groups, so that they "think" their processors are numbered 0..n-1. It has the disadvantage that there is an extra translation required on ALL communications. This will add to the latency at the most fundamental level (though not much if we expect to consume space linear in the group size). 2) Creating a new group for each context may be expensive, though Marc suggests an implementation which makes this relatively cheap. Adding Rik's caching ideas here could also defer the cost until it was required. (e.g. there is no need to build a broadcast tree in a group until the first broadcast is performed. [Fundamental optimisation 0: "Don't do it until you need to, you might never have to do it at all"]) At the moment I favour Marc's with added caching. Can anyone explain where the big gain is from the added complexity of the others ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Mar 23 12:22:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23133; Tue, 23 Mar 93 12:22:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19811; Tue, 23 Mar 93 12:21:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 12:21:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19803; Tue, 23 Mar 93 12:21:49 -0500 Date: Tue, 23 Mar 93 17:21:18 GMT Message-Id: <15889.9303231721@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Context proposals overview To: jim@meiko.co.uk (James Cownie), mpi-context@cs.utk.edu In-Reply-To: James Cownie's message of Tue, 23 Mar 93 15:39:41 GMT Reply-To: lyndon@epcc.ed.ac.uk Cc: snir@watson.ibm.com, rj_littlefield@pnlg.pnl.gov, lyndon@epcc.ed.ac.uk Jim writes: > Gentlemen, here is my summary and initial comclusions on the three > extant Contexts proposals received via Lyndon. Since I have probably > misunderstood or misrepresented some of the proposals, please correct > me as appropriate. > > Proposal I (Marc Snir) > ---------------------- > Creation of a new context involves creation of a new group which is a > group synchronising operation. More accurately it involves duplication of an existing group. One problem which I have with this is that if we anticipate future extension to groups which are dynamic in membership (they expand and contract) then the duplication concept breaks because the identical nature of the grouping is lost. This is why I much prefer to have contexts as references to groups and groups as frames of contexts - the contexts shrink and stretch according to the expansion and contraction of the underlying group. > For: > Simplicity, it's easy to explain and understand (and probably > implement). Indeed but please see further comments attatched to Proposal VI below. > Insisting on a group for all addressing gives the 0..n-1 rank model > for subgroups which aids in importing libraries. Indeed and Proposal VI gives all (and more) of this aid. > Against: > The receiver of a message has to both know the group of the sender, > and be a part of it. This feels like it makes servers hard, but may be > ok. If you keep having to go into the ALL group, then the security > which was the whole point of a context is lost. The restriction in Proposal I really does make programming of communicative functionally distinct groups hard. I am thinking well beyond singleton service processes and I have explained some of the things I am thinking previous messages. > The argument that contexts can be checked at the sender is incorrect > because a single process can be in many different groups/contexts, and > therefore the group/context must be transferred to ensure correct > matching. (Maybe this wasn't the intent of the comments anyway...) Our interpretations concur. > Proposal V (Rik Littlefield) > ---------------------------- > Key points : > A desire to have caching of arbitrary library (or even user) defined > entities on the groups. > > Point to point communication is expected to be addressed by TID and > Context. TIDs have global scope, they can be passed around > arbitrarily. > > There is explicit translation from (group,rank) to TID. ... which the poor user *must* do. > Group descriptors are local objects, but can be passed around through > special mechanisms. Check. If the feature of Proposal VI is a disdavantage of that proposal then it is also a disadvantage of Proposal V! Just being consist :-) > Contexts are globally unique, (and expected to be allocated locally > through one of the normal bit space partitioned schemes. i.e. > processor number // identifier. There appears to be an expectation > that the identifiers are managed closely), and can be passed > transparently in messages (? is this true ?). Our interpretations concur. > Contexts are created inside groups, creating a context is a barrier > synchronisation (? is this true ?). Our interpretations concur. > For: > The caching is good idea, but orthogonal to all the other issues, it > could be added just as easily to any of the other proposals Precisely and this is recognised in the revision of Proposal VI which will be circluated within the hour. > The use of TIDs in the point to point removes any issues of group > membership from these functions. There is thus a way to achieve any > inter group communication which is required (even if it is unpleasant). It is unpleasant indeed. I appreciate that I am stating a value jugement here which may be interpreted as "my personal opinion". In fact at Edinburgh we have done plenty of intergroup communication programming and the opinion is shared. > Against: > The complexity of the "paired exact match criterion" is non trivial to > explain. I concur. > Proposal VI (Lyndon Clarke) > --------------------------- > > Key points: > ALL descriptors (process, group, context) are process local, but may > be sent elsewhere through special mechanisms. The revision, particularly in the improved discussion notes, explores the possibility that processes are expressed as identifiers but groups and contexts are process local opaque references to objects of undefined size and structure (implemenation dependent). > Contexts are created inside groups, and are bound to them. The context > can be used to refer (indirectly) to the owning group. Exactly. Please recall the point I made above about group (context) duplication in Proposal I. > Creating a context is a barrier synchronisation of the group. The proposal says this, the discusion explains that it may not be necessary to actually synchronise. My mind is still working on this one. > Three different forms of process address are discussed for point to > point messaging. (Ouch !) From the point of view of the MPI provider this is a just small amount of additional work. The three forms link to a generic send/recv so there is not at all three times as much implementation. The size of a test suite increases of course. Form the point of view of the MPI user there could be an issue here, however the facilities can be described in a stepwise manner such that users do not need to be explored to more complexity then they need to use. The revision tries to clean up the business of presenting these forms in a uniform syntactic framework which is one way to address the "point-to-point contains too many procedures" concern. > For: > Can be made to address complex inter group communications Does address such communications directly and with expressive power. I'd like to add "For: VI is functional superset of Proposal I; group/context user cache of V is easy to add to VI." > Against: > The complexity of the whole proposal. Proposal VI is larger than Proposal I which is at a similarly mature stage of development, because it is more powerful, as I say it is functionally a superset of Proposal I (and this is drawn out better in the discussion in the revision). The facilities can be presented to users in a friendly fashion which does not require substantially different entry level user sophistication than Proposal I. The more "advanced" facilities can be introduced in a user manual after the more "basic" facilities. I conjecture that there is no real problem here. > The passing of opaque entities with translation is rather unpleasant > for people with homogeneous machines. Before this we could ignore all > of the type info, now we have to check some of it and take special > actions. > I mentioned above that if this is a disadvantage of VI then it must also be a disadvantage of V. This was one suggestion as to how we might give the user an interface to transmission of these opaque objects. I'm sure we can think of other approaches which avoid the implementor having to check the data type description in homogeneous kit. For example we could provide a set of procedures which do the send and receive, although this does again add more procedures to point-to-point which I am trying to avoid, honest guv' :-) I do not believe that this is any big deal for the user, since the kind of users who are currently doing this kind of programming (and they are certainly not high priests) are understand and think about data translation issues anyway. If these are the only real objections you have to passing around descriptors then I am confident we can do something about that. Would you like to make a suggestion as to how you would prefer to see transmission done? > Jim's initial conclusions... > ---------------------------- > > 1) Marc's approach of insisting that all communications occur inside a > group has some simplifying effects. In particular it easily gives the > base shift for sub-groups, so that they "think" their processors are > numbered 0..n-1. It has the disadvantage that there is an extra > translation required on ALL communications. This will add to the > latency at the most fundamental level (though not much if we expect to > consume space linear in the group size). Proposal VI has all of the facilities, which has been implemented and used in systems for a couple years now, and a whole lot more. For speed demons Proposal VI lets them use process descriptots (or identifiers) directly. > At the moment I favour Marc's with added caching. Can anyone explain > where the big gain is from the added complexity of the others ? I hope I did. If you need more explanation then please do ask. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 23 13:28:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24476; Tue, 23 Mar 93 13:28:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22791; Tue, 23 Mar 93 13:27:53 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 13:27:52 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22783; Tue, 23 Mar 93 13:27:50 -0500 Message-Id: <9303231827.AA22783@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6757; Tue, 23 Mar 93 13:27:47 EST Date: Tue, 23 Mar 93 13:10:01 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-context@cs.utk.edu Reply-To: SNIR@watson.ibm.com Subject: Context proposals overview Reference: Attached note from jim@meiko.co.uk Reply *************** Forwarded Note *************** Received: from marge.meiko.com by watson.ibm.com (IBM VM SMTP V2R3) with TCP; Tue, 23 Mar 93 10:39:58 EST Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA01482 (5.65c/IDA-1.4.4 for ); Tue, 23 Mar 1993 10:39:47 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA11198; Tue, 23 Mar 93 15:39:42 GMT Date: Tue, 23 Mar 93 15:39:41 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303231539.AA11198@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05333; Tue, 23 Mar 93 15:36:15 GMT To: mpi-context@cs.utk.edu Cc: snir@watson.ibm.com, rj_littlefield@pnlg.pnl.gov, lyndon@epcc.ed.ac.uk Subject: Context proposals overview Content-Length: 5160 Jim's initial summary/comments on the Contexts proposals Gentlemen, here is my summary and initial comclusions on the three extant Contexts proposals received via Lyndon. Since I have probably misunderstood or misrepresented some of the proposals, please correct me as appropriate. Proposal I (Marc Snir) ---------------------- Key points : A group and a context are identical. All communication (including point to point) is addressed by group and rank. The receiver MUST be a member of the same group as the sender, and explicitly give the group as an argument to the receive function. Group descriptors are not passable to other processors. Creation of a new context involves creation of a new group which is a group synchronising operation. For: Simplicity, it's easy to explain and understand (and probably implement). Insisting on a group for all addressing gives the 0..n-1 rank model for subgroups which aids in importing libraries. Against: The receiver of a message has to both know the group of the sender, and be a part of it. This feels like it makes servers hard, but may be ok. If you keep having to go into the ALL group, then the security which was the whole point of a context is lost. >>> One can, of course, have several distinct contexts that include >>> all processes. The argument that contexts can be checked at the sender is incorrect because a single process can be in many different groups/contexts, and therefore the group/context must be transferred to ensure correct matching. (Maybe this wasn't the intent of the comments anyway...) >>> group/context need be transfered in the message. The point I was >>> trying to make is that one can make sure that a message with a >>> given context tag will be sent only to a process that "knows" about >>> this context. This allows some relaxations: e.g., context names >>> need not be system-wide unique, only locally unique at each process; >>> handling of "illegal context" errors is simplified. I don't understand how to build a group/context using only point to point messages. I still seem to have the bootstrap problem that I need the new context to safely receive the message which will tell me what the new context is. >>> Well, we have an existential proof, since we support dynamic group >>> creation in our system. A new context is created within an old, >>> preexisting context; so ALL need to be there from start. Proposal V (Rik Littlefield) ---------------------------- Key points : A desire to have caching of arbitrary library (or even user) defined entities on the groups. Point to point communication is expected to be addressed by TID and Context. TIDs have global scope, they can be passed around arbitrarily. There is explicit translation from (group,rank) to TID. Group descriptors are local objects, but can be passed around through special mechanisms. Contexts are globally unique, (and expected to be allocated locally through one of the normal bit space partitioned schemes. i.e. processor number // identifier. There appears to be an expectation that the identifiers are managed closely), and can be passed transparently in messages (? is this true ?). Contexts are created inside groups, creating a context is a barrier synchronisation (? is this true ?). For: The caching is good idea, but orthogonal to all the other issues, it could be added just as easily to any of the other proposals The use of TIDs in the point to point removes any issues of group membership from these functions. There is thus a way to achieve any inter group communication which is required (even if it is unpleasant). Against: The complexity of the "paired exact match criterion" is non trivial to explain. Proposal VI (Lyndon Clarke) --------------------------- Key points: ALL descriptors (process, group, context) are process local, but may be sent elsewhere through special mechanisms. Contexts are created inside groups, and are bound to them. The context can be used to refer (indirectly) to the owning group. Creating a context is a barrier synchronisation of the group. Three different forms of process address are discussed for point to point messaging. (Ouch !) For: Can be made to address complex inter group communications Against: The complexity of the whole proposal. The passing of opaque entities with translation is rather unpleasant for people with homogeneous machines. Before this we could ignore all of the type info, now we have to check some of it and take special actions. Jim's initial conclusions... ---------------------------- 1) Marc's approach of insisting that all communications occur inside a group has some simplifying effects. In particular it easily gives the base shift for sub-groups, so that they "think" their processors are numbered 0..n-1. It has the disadvantage that there is an extra translation required on ALL communications. This will add to the latency at the most fundamental level (though not much if we expect to consume space linear in the group size). 2) Creating a new group for each context may be expensive, though Marc suggests an implementation which makes this relatively cheap. Adding Rik's caching ideas here could also defer the cost until it was required. (e.g. there is no need to build a broadcast tree in a group until the first broadcast is performed. [Fundamental optimisation 0: "Don't do it until you need to, you might never have to do it at all"]) At the moment I favour Marc's with added caching. Can anyone explain where the big gain is from the added complexity of the others ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Mar 23 13:44:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24968; Tue, 23 Mar 93 13:44:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23491; Tue, 23 Mar 93 13:43:28 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 13:43:27 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23478; Tue, 23 Mar 93 13:43:11 -0500 Date: Tue, 23 Mar 93 18:43:04 GMT Message-Id: <15951.9303231843@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal VI, First Revision, LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Here is LaTeX for the revision of Proposal VI. Please see my reply to Jim of today for an idea of what is changed. I apologise that this is one hour later then I had previously announced. PostScript follows. ---------------------------------------------------------------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % LYNDON % % REMBER TO MERGE IN THE COMMENTS FROM THE PREVIOUS REVISION OF THIS % % DOCUMENT. OH, AND TO GET BETTER SET UP FOR TRANSFER TO LAPTOP! % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentstyle{report} \begin{document} \title{MPI Context Subcommittee\\Proposal VI\\First Revision} \author{Lyndon~J~Clarke} \date{March 23, 1993} \maketitle %======================================================================% % BEGIN "Proposal VI" % Lyndon J. Clarke % March 1993 % \chapter{Proposal VI} \newcommand{\LabelNote}[1]{\label{vi:note:#1}} \newcommand{\ReferNote}[1]{Note~\ref{vi:note:#1}} \newcommand{\LabelSection}[1]{\label{vi:sect:#1}} \newcommand{\ReferSection}[1]{Section~\ref{vi:sect:#1}} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction}\LabelSection{introduction} This chapter proposes that communication contexts and process groupings within {\sc mpi} appear as loosely coupled concepts. One or more communication contexts may be associated with each process grouping, and each communication context inherits properties of the associated process grouping. This reflects the observations that invocations of modules in a parallel program typically operate within process groupings, and that there may be multiple modules operating within each process grouping. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of {\sc mpi}, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language, only for purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The note assumes forward knowledge of the proposal text and are therefore best examined after familiarisation with the proposal text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the proposal text. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal}\LabelSection{proposal} This section presents the detailed proposal, describing, in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes}\LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and {\sc mpi} does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} (or {\it handle}) which is expressed as an integer type in the host language and has an opaque value which is process local. See \ReferNote{descriptor}. The initialisation of {\sc mpi} services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of {\sc mpi} services. {\sc mpi} provides a procedure which returns the own descriptor of the calling process. For example, \mbox{\tt pd = mpi\_own\_process()}. \subsubsection*{Process Creation \& Destruction} This proposal makes no statements regarding creation and destruction of processes. See \ReferNote{processes:dynamic}. \subsubsection*{Descriptor Transmission} Since a descriptor is an integer type in the host language the value may be transmitted in a message as an integer. The recipient of the descriptor value can make no defined use of the value in the {\sc mpi} operations described in this proposal --- the descriptor is {\it invalid}. {\sc mpi} provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. \mbox{\tt MPI\_PD\_TYPE}. See \ReferNote{descriptor:transmission}. {\sc mpi} provides a process descriptor registry service. Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. See \ReferNote{descriptor:registry}. Note that receipt of a valid process descriptor may have a persistent effect on the implementation of {\sc mpi} at the receiver, and in particular may reserve state. {\sc mpi} will provide a procedure which frees (or invalidates) a valid descriptor, allowing the implementation to free reserved state. For example, \mbox{\tt mpi\_free\_pd(pd)}. The user is not allowed to free the process descriptor of the calling process. See \ReferNote{descriptor:deallocation}. See \ReferNote{coherency}. \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding process descriptor attributes. \subsection{Process Groupings}\LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. See \ReferNote{groupings:dynamic}. The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to descriptors of the $N$ processes composing the grouping. There may be structure associated with a process grouping defined by a process topology. This proposal makes no further statements regarding such structures. \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} (or {\it handle\/}) which is expressed as an integer type in the host language and has an opaque value which is process local. See \ReferNote{descriptor}. The initialisation of {\sc mpi} services will assign to each process an {\it own\/} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the own process grouping until the termination of {\sc mpi} services. See \ReferNote{grouping:blobs}. {\sc mpi} provides a procedure which accepts a valid process descriptor and returns the own group descriptor of the identified process. For example, \mbox{\tt gd = mpi\_own\_group(pd)}. See \ReferNote{grouping:own}. \subsubsection*{Group Creation and Deletion} {\sc mpi} provides facilities which allow users to dynamically create and delete process groupings in addition to the own grouping(s). The procedures described here generate groups which are static in membership. {\sc mpi} provides a procedure which allows users to create one or more groups which are subsets of existing groups. For example, \mbox{\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} which are distinct subsets of an existing group {\tt gda} according to the supplied values of {\tt key}. This procedure is called by and synchronises all members of {\tt gda}. {\sc mpi} provides a procedure which allows users to create a group by permutation of an existing group. For example, \mbox{\tt gdb = mpi\_group\_permutation(gda, rank)} creates one new group with the same membership as {\tt gda} with a permutation of process ranking, and returns the created group descriptor in {\tt gdb}. This procedure is called by and synchronises all members of {\tt gda}. {\sc mpi} provides a procedure which allows users to create a group by explicit definition of its membership as a list of process descriptors. For example, \mbox{\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. {\sc mpi} provides a procedure which allows users to delete user created groups. This procedure accepts the descriptor of a group which was created by the calling process and deletes the identified group. For example, \mbox{\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. {\sc mpi} may provide additional procedures which allow users to construct process groupings with a process grouping topology. \subsubsection*{Descriptor Transmission} Since a descriptor is an integer type in the host language the value may be transmitted in a message as an integer. The recipient of the descriptor can make no defined use of the value in the {\sc mpi} operations described in this proposal --- the descriptor is {\it invalid}. See \ReferNote{descriptor}. {\sc mpi} provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. \mbox{\tt MPI\_GD\_TYPE}. See \ReferNote{descriptor:transmission}. {\sc mpi} provides a group descriptor registry service. Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. See \ReferNote{descriptor:registry}. Note that receipt of a valid group descriptor may have a persistent effect on the implementation of {\sc mpi} at the receiver, and in particular may reserve state. {\sc mpi} will provide a procedure which frees (or invalidates) a valid descriptor, allowing the implementation to free reserved state. For example, \mbox{\tt mpi\_free\_gd(gd)}. The user is not allowed to free the own group descriptor of the calling process or the group descriptor of any group created by the calling process. See \ReferNote{descriptor:deallocation}. See \ReferNote{coherency}. \subsubsection*{Descriptor Attributes} {\sc mpi} provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identified group. For example, \mbox{\tt rank = mpi\_group\_rank(gd)}. {\sc mpi} provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, \mbox{\tt size = mpi\_group\_size(gd)}. {\sc mpi} provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, \mbox{\tt pd = mpi\_group\_pd(gd, rank)}. See \ReferNote{inverse:map}. {\sc mpi} may provide additional procedures which allow users to determine the process grouping topology attributes. \subsection{Communication Contexts}\LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping may be used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} (or {\it handle}) which is expressed as an integer type in the host language and has an opaque value which is process local. See \ReferNote{descriptor}. The creation of {\sc mpi} process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains the descriptor of its own context until {\sc mpi} process grouping deletion. {\sc mpi} provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, \mbox{\tt cd = mpi\_own\_context(gd)}. See \ReferNote{context:own}. \subsubsection*{Context Creation and Deletion} {\sc mpi} provides facilities which allows user to dynamically create and delete contexts in addition to the own contexts associated with process groupings. See \ReferNote{grouping:deletion}. {\sc mpi} provides a procedure which allows users to create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, \mbox{\tt cd = mpi\_context\_creation(gd)}. This procedure is called by and synchronises all members of {\tt gd}. {\sc mpi} provides a procedure which allows users to delete user created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes the identified context. For example, \mbox{\tt mpi\_context\_deletion(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. See \ReferNote{context:creation:deletion} \subsubsection*{Descriptor Transmission} Since a descriptor is an integer type in the host language the value may be transmitted in a message as an integer. The recipient of the descriptor can make no defined use of the value in the {\sc mpi} operations described in this proposal --- the descriptor is {\it invalid}. See \ReferNote{descriptor}. MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. \mbox{\tt MPI\_CD\_TYPE}. See \ReferNote{descriptor:transmission}. {\sc mpi} provides a context descriptor registry service. Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. See \ReferNote{descriptor:registry}. Note that receipt of a valid context descriptor may have a persistent effect on the implementation of {\sc mpi} at the receiver, and in particular may reserve state. {\sc mpi} will provide a procedure which frees (or invalidates) invalidates a valid descriptor, allowing the implementation to free reserved state. For example, \mbox{\tt mpi\_free\_cd(cd)}. The user is not allowed to free the own context descriptor of any group or the context descriptor of any context created by the calling process. See \ReferNote{coherency}. \subsubsection*{Descriptor Attributes} {\sc mpi} provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, \mbox{\tt gd = mpi\_context\_group(cd)}. \subsection{Point-to-Point Communication}\LabelSection{point2point} This proposal recommends three forms for {\sc mpi} point-to-point message addressing and selection: null context; closed context; open context. (See \ReferNote{context:form}). It is further recommended that messages communicated in each form are distinguished such that a {\tt Send} operation of form X cannot match with a {\tt Receive} operation of form Y, requiring that form is embedded into the message envelope. The three forms are described, followed by considerations of uniform integration of these forms in the point-to-point communication chapter of {\sc mpi}. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. Message selection and addressing are expressed by \mbox{\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. {\tt Send} supplies the {\tt pd} of the receiver. {\tt Receive} supplies the {\tt pd} of the sender. {\tt Receive} can wildcard on {\tt pd} by supplying the value of a named constant process descriptor, e.g. {\tt MPI\_PD\_WILD}. See \ReferNote{descriptor:wildcard}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. Message selection and addressing are expressed by \mbox{\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. {\tt Send} supplies the {\tt cd} of the receiver (and sender), and the {\tt rank} of the receiver. {\tt Receive} supplies the {\tt cd} of the sender (and receiver), and the rank of the sender. The \mbox{\tt (cd, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process descriptor of the receiver (sender). {\tt Receive} cannot wildcard on {\tt cd}. {\tt Receive} can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. See \ReferNote{rank:wildcard}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. Message selection and addressing are expressed by \mbox{\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. {\tt Send} supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. {\tt Receive} supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The \mbox{\tt (rcd, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process descriptor of the receiver (sender). {\tt Receive} cannot wildcard on {\tt lcd} which is the open context form analog of there being no wildcard on {\tt cd} in the closed context form. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD} (See \ReferNote{descriptor:wildcard}), in which case {\tt Receive} {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. {\tt Receive} can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. See \ReferNote{rank:wildcard}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point chapter of {\sc mpi} by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with form. This is at the expense of multiplying the number of {\tt Send} and {\tt Receive} procedures by a factor of three, and some further but trivial work with details of the current point-to-point chapter which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options represent different compromises between the three forms. \paragraph*{Option i: Open Context Framework} The framework of the open context form is adopted and extended. Introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. See \ReferNote{descriptor:null}. The null context form is expressed as \mbox{\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as \mbox{\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as \mbox{\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii: Closed Context Framework} The framework of the closed context form is adopted and extended. Introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. See \ReferNote{descriptor:null}. The null context form is expressed as \mbox{\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as \mbox{\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the \mbox{\tt (lcd, rcd)} pair at the expense of introducing some trickery. We define a ``context duplet descriptor'' which is formally composed of two references to contexts, and provide a procedure which constructs such a descriptor given two context descriptors. Both {\tt Send} and {\tt Receive} will accept a duplet descriptor in {\tt cd}, are able to distinguish the duplet descriptor from a singlet descriptor, and treat the duplet as shorthand notation. We should also define a mechanism by which a receiver which has completed a {\tt Receive} with wildcard on {\tt rcd} is able to determine the valid singlet descriptor of the sender, which just adds one further enquiry procedure to the point-to-point chapter(?). This option is a little inconvenient but does have some useful properties for collective communications. It is conjectured that this option is the best choice for {\sc mpi}. \paragraph*{Option iii: Null Context Framework} The framework of the null context form is adopted and extended. The null context form is expressed as \mbox{\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is rather clumsy. \subsection{Collective Communication}\LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context (thus frame) in which they are to operate. {\sc mpi} does plan to describe symmetric collective communication operations. It is not possible to determine whether this proposal is sufficient to allow implementation of the collective communication chapter of {\sc mpi} in terms of the point-to-point chapter of {\sc mpi} without loss of generality, since the collective operations are not yet defined. Asymmetric collective communication operations, especially those in which sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a duplet descriptor form) which identify the contexts (thus frames) in which they are to operate. {\sc mpi} does not plan to describe asymmetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model, which are composed of communicative functionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of {\sc mpi}. % % END "Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion}\LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents brief arguments for and against the MIMD model annex. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model (in which process execute in an SIMD fashion). The material describing processes in \ReferSection{processes} is simplified: processes have identical instruction blocks and different data blocks; process descriptor transmission and registry become redundant; dynamic process models are not considered. The material describing process groupings in \ReferSection{groupings} is simplified: group descriptor transmission and registry become redundant; the own process grouping explicitly becomes a single group containing all processes. The material describing communication contexts in \ReferSection{contexts} is simplified: context descriptor transmission and registry become unnecessary. The material describing point-to-point communication in \ReferSection{point2point} is simplified: the open context form becomes redundant; uniform integration ``Option i'' is deleted, and ``Option ii'' loses duplet descriptors, becoming simple enough that ``Option iii'' need not be further considered. The material describing collective communication in \ReferSection{collective} is simplified: there is no possibility of collective communication operations spanning more than one context. \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as {\sc mpi} are not suited to fine grained parallel programming. The annex is intended to provide expressive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in {\sc express} and {\sc parmacs}, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or the host as a singleton server and the nodes as an SPMD like client). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph only contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computing. The debate is whether or not message passing approaches such as {\sc mpi} (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD like modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD like parallel programs. The positive argument is that communications between such SPMD like modules require high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development (and standardisation) procedures for distributed operating systems. The latter argument is somewhat comparable with the argument for message passing versus parallel compilation. \subsection{Notes}\LabelSection{notes} \begin{enumerate} \item\LabelNote{descriptor} {\bf Descriptors:} Descriptors are assumed to be a plentiful but bounded resource. They are opaque references to objects of undefined size and structure. They are not global unique identifiers however they must reference such identifiers, and they protect the user from the form of such identifiers allowing them to be implementation dependent. The proposal expresses descriptors as integer types in the host language (in practice we might expect descriptors to be indices into tables of structures, or tables of pointers to structures, or indeed pointers to structures themselves). This expression facilitates uniform binding to ANSI~C and Fortran~77. The context descriptor must at least reference: the global unique context identifier; the group descriptor of the frame. The group descriptor must at least reference: the global unique group identifier; the own context descriptor; the rank space to process descriptor map of the group (including the size of the group); the process rank. The process descriptor must at least reference; the global unique process identifier; the group descriptor of the own group. The proposal text distinguishes process, group and context identifiers more strongly than is strictly necessary. There is potential advantage in the concept of a unified descriptor which can be used to reference either kind of actual descriptor. For example descriptor unification goes some way toward cleaning up the duplet, descriptors described in ``Option ii'' of section \ReferSection{point2point}. Definition in this fashion requires the referenced object to contain a class identifier (which in this proposal could be as little as 3 bits wide) This suggestion is explored in some of the notes below. Rik Littlefield has suggested descriptors could be used to ``cache'' per object user information, as appears in {\sc zipcode}. This descriptor capability could be seriously useful for example in context or group specific implementations of collective communications. The suggestion both requires and deserves more work. There were additional motivations for expressing process descriptors as integers in the proposal. Firstly, the author anticipated ``Option ii'' of \ReferSection{point2point}. Secondly, it is observed that definition of process descriptors can be weakened such that they may be implemented as global unique process identifiers, expressed as integers, should this infer advantage. {[\it I wish I had realised the second point before the first!]} The consequence of allowing process descriptors to be implemented as process identifiers is explored in some of the notes below. Please also note that this would exclude process descriptors from the ``caching'' capability mentioned above. These ideas suggest an alternate proposal, containing global unique process identifiers expressed as integers, and context and group descriptors as opaque references of yet unspecified language binding (i.e., perhaps not an integer). This suggestion is also explored in some of the notes below. \item\LabelNote{processes:dynamic} {\bf Dynamic Processes:} The proposal does not prevent a process model which allows dynamic creation and deletion of processes however it does not favour an asynchronous process model in which singleton processes are created and deleted in an arbitrary fashion. The proposal does favour a model in which blobs of processes are created (and deleted) in a concerted fashion, and in which each blob so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying workloads. It is conjectured that concerted blob expand and contract operations are more suitable for this purpose than asynchronous singleton spawn and kill operations. \item\LabelNote{descriptor:transmission} {\bf Descriptor transmission:} In the spirit of descriptor unification (See \ReferNote{descriptor}) the three {\tt MPI\_?D\_TYPE} names can be collapsed into something like {\tt MPI\_DESC\_TYPE}. If process descriptors are replaced by global unique process identifiers (See \ReferNote{descriptor}) then no special measures are required for transmission thereof. If group and context descriptors are expressed as opaque objects of yet unspecified type then, in ANSI~C at least, it will be possible to prevent nonsemantic transmission thereof. \item\LabelNote{descriptor:registry} {\bf Descriptor registry:} The registry service is just a simpler way for concurrent objects to identify and establish communications with one another. The operations of interest, expressed in the spirit of descriptor unification (See \ReferNote{descriptor}), are: registration of descriptor by name, e.g. \mbox{\tt mpi\_register(name,desc)}; deregistration, e.g \mbox{\tt mpi\_deregister(desc)}; and lookup by name, e.g. \mbox{\tt mpi\_lookup(name, \&desc, wait)} where {\tt wait} controls whether the lookup waits for the name to be registered. If process descriptors are replaced by global unique process identifiers (See \ReferNote{descriptor}) then perhaps process identifier registry is not so important. The MIMD model does not actually required process descriptor or group descriptor registry to be visible to the user since context descriptor registry and context descriptor attribute determination gives access to all groups and thus group descriptor attribute determination gives access to all processes. The proposal was written to handle descriptors consistently. \item\LabelNote{descriptor:deallocation} {\bf Descriptor deallocation:} In the spirit of descriptor unification (See \ReferNote{descriptor}) the three \mbox{\tt mpi\_?d\_free(?d)} can be collapsed into something like \mbox{\tt mpi\_desc\_free(desc)}. The receipt of a descriptor in descriptor transmission and registry is an allocator, hence provision of the deallocator. Perhaps there should be an explicit allocator which the user must call in order to receive a descriptor, and can deallocate when no longer required. If process descriptors are replaced by global unique process identifiers (See \ReferNote{descriptor}) then process identifier deallocation is moot. \item\LabelNote{coherency} {\bf Coherency:} The proposal admits incoherency as descriptors may be received in transmission or registry. The SPMD core contains no incoherency. The inclusion of dynamic process creation and deletion admits incoherency since processes can retain descriptors of processes which have been deleted. The inclusion of grouping descriptor transmission and registry admits incoherency since processes can retain descriptors of groupings which have been deleted. The inclusion of dynamic groupings admits incoherency since processes can retain descriptors of groupings of which the rank to process map has changed. The inclusion of context descriptor transmission and registry admits incoherency since processes can retain descriptors of contexts which have been deleted. The proposal expects the user to ensure coherent usage. It is conjectured that this is acceptable provided that the user is not also expected to implement process fault tolerance. \item\LabelNote{groupings:dynamic} {\bf Dynamic groupings:} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The proposal specifies static groupings however the loose separation of communication contexts from process groupings simplifies extension to dynamic groupings as contexts stretch or shrink according to the changes in their frames. It is conjectured that concerted grouping expand and contract operations are more suitable than asynchronous singleton join and leave operations. \item\LabelNote{grouping:blobs} {\bf Process blobs:} {\sc mpi} has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and