From wade@cs.utk.edu Mon Jan 18 10:04:48 1993 Received: from THUD.CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15560; Mon, 18 Jan 93 10:04:48 -0500 Received: from LOCALHOST.cs.utk.edu by thud.cs.utk.edu with SMTP (5.61++/2.7c-UTK) id AA03584; Mon, 18 Jan 93 10:04:42 -0500 Message-Id: <9301181504.AA03584@thud.cs.utk.edu> To: mpi-context-archive@surfer.EPM.ORNL.GOV Cc: wade@cs.utk.edu Subject: Date: Mon, 18 Jan 93 10:04:42 EST From: Reed Wade test From walker@rios2.epm.ornl.gov Mon Jan 18 13:23:40 1993 Received: from rios2.EPM.ORNL.GOV by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19959; Mon, 18 Jan 93 13:23:40 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17128; Mon, 18 Jan 1993 13:23:36 -0500 Date: Mon, 18 Jan 1993 13:23:36 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181823.AA17128@rios2.epm.ornl.gov> To: mpi-context-archive@surfer.EPM.ORNL.GOV From root Fri Jan 15 15:33:35 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12164; Fri, 15 Jan 1993 15:33:34 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA07918; Fri, 15 Jan 93 15:33:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Fri, 15 Jan 93 15:29:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:29:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13112; Fri, 15 Jan 93 15:29:50 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13694; Fri, 15 Jan 1993 15:29:49 -0500 Date: Fri, 15 Jan 1993 15:29:49 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301152029.AA13694@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Status: RO I thought the discussion of communication contexts at the last Dallas meeting was lacking in focus, and I'm afraid the existing subcommittees may not deal with this area. If there are enough people interested perhaps there should be a separate communication context subcommittee. If you have any strong opinions on this please let me know. Also let me know if you would like to be on the communication context subcommittee if one is established. On the topic of communication contexts I think (at least) 4 approaches have been proposed. 1) Implicit communication contexts as in MPI1 controlled by push/pop. 2) Explicit registration as in Zipcode 3) Explicit communication contexts should be merged with explicit group contexts. Since groups cannot communicate with each other a new communication context can be created by creating a new group with the same members as the current group. 4) Chuck Simmons of Oracle has suggested that communication contexts are really a particular type of thread, and so can be handled using existing threads packages. If you have any ideas, or are strongly for or against any of the above approaches let mpi-comm know so we can initiate a discussion of this topic. David Walker From root Fri Jan 15 15:47:37 1993 Received: from antares.mcs.anl.gov by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16031; Fri, 15 Jan 1993 15:47:36 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03636; Fri, 15 Jan 93 14:47:35 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00709; Fri, 15 Jan 93 14:47:32 CST Message-Id: <9301152047.AA00709@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: walker@rios2.epm.ornl.gov Date: Fri, 15 Jan 93 14:47:31 CST From: Rusty Lusk Status: RO | | On the topic of communication contexts I think (at least) 4 approaches have | been proposed. | 1) Implicit communication contexts as in MPI1 controlled by push/pop. | 2) Explicit registration as in Zipcode | 3) Explicit communication contexts should be merged with explicit | group contexts. Since groups cannot communicate with each other | a new communication context can be created by creating a new group | with the same members as the current group. | 4) Chuck Simmons of Oracle has suggested that communication contexts | are really a particular type of thread, and so can be handled | using existing threads packages. | If you have any ideas, or are strongly for or against any of the above | approaches let mpi-comm know so we can initiate a discussion of this topic. I think that 1) contradicts our genral agreement to avoid state as much as possible. 4) is a problem on systems that don't support threads, and we have more-or-less agreed to be consistent with thread packages but not depend upon them. 3) should be discussed along with the converse: that contexts rather than groups should be the primary concept and groups defined in terms of contexts. David, perhaps you could create a mailing list on which this discussion can take place, parallel to the subcommittee mailing lists, so it doesn't have to go to the whole committee. Thanks. Rusty From root Fri Jan 15 16:14:41 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13265; Fri, 15 Jan 1993 16:14:39 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA08064; Fri, 15 Jan 93 16:14:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16784; Fri, 15 Jan 93 16:12:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 16:12:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16768; Fri, 15 Jan 93 16:12:01 -0500 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA21882; Fri, 15 Jan 93 15:11:54 -0600 Received: by mozart.convex.com (5.64/1.28) id AA01950; Fri, 15 Jan 93 15:11:17 -0600 Date: Fri, 15 Jan 93 15:11:17 -0600 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9301152111.AA01950@mozart.convex.com> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: lusk@antares.mcs.anl.gov, walker@rios2.epm.ornl.gov Status: RO Rusty Lusk writes: > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. I believe the default inclusion of everyone is a good idea, although creating a separate mailing list might be appropriate if some mpi-comm members wish to "unsubscribe". In the meantime, I will add one brief opinion: I felt that the discussions of contexts in the topology subgroup were quite sufficient, and it would probably be better to have the single subgroup which covers both groups and contexts, so that both will appear in the standard in a style which does not make them conflict with each other, and/or to avoid a proposal where groups and contexts would be totally redundant. I definitely agree that it is an important topic, and that the discussions last week in Dallas were far away from getting a proposal for groups and contexts that would be unanimously acceptable, so much work remains to be done. From root Fri Jan 15 17:03:11 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13294; Fri, 15 Jan 1993 17:03:06 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA08266; Fri, 15 Jan 93 17:03:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21190; Fri, 15 Jan 93 17:00:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 17:00:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21182; Fri, 15 Jan 93 17:00:54 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA20492; Fri, 15 Jan 93 22:00:50 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14823; Fri, 15 Jan 93 14:59:49 MST Date: Fri, 15 Jan 93 14:59:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Communication contexts Status: RO > | > | On the topic of communication contexts I think (at least) 4 approaches have > | been proposed. > | 1) Implicit communication contexts as in MPI1 controlled by push/pop. > | 2) Explicit registration as in Zipcode > | 3) Explicit communication contexts should be merged with explicit > | group contexts. Since groups cannot communicate with each other > | a new communication context can be created by creating a new group > | with the same members as the current group. > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > | If you have any ideas, or are strongly for or against any of the above > | approaches let mpi-comm know so we can initiate a discussion of this topic. > > I think that 1) contradicts our genral agreement to avoid state as much as > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather > than groups should be the primary concept and groups defined in terms of > contexts. > > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. > > Rusty I'm not sure, but I believe that Zipcode uses push/pop to control contexts (Tony, please correct me if I'm wrong). I'd like to add a 5th approach, basically the one proposed by Paul Pierce: 5) Contexts are used exclusively to insure that message collisions will not occur if independently developed sub-programs are combined. Contexts and groups are orthogonal. Contexts and threads are orthogonal. Each message has an associated context and tag. Message context is managed by library routines and is completely out of a user's control. Message tag is selected by the user. The method of managing message contexts is a separate issue (assuming we want contexts). Existing proposals are: 5a) Stack-based management (objected to due to hidden states). 5b) Explicit registration with user-defined "names" (probably requires some communication). 5c) Explicit registration by a central authority ("dollar bill" registration mentioned by Jim Cownie.) There is enough confusion about contexts and groups that we may want to keep this discussion open to everyone. If the concensus is to make a separate mailing list, please both of us to it. Thanks, Tom Henderson hender@fsl.noaa.gov Leslie Hart hart@fsl.noaa.gov From root Fri Jan 15 19:07:57 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13061; Fri, 15 Jan 1993 19:07:55 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA08466; Fri, 15 Jan 93 19:07:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27214; Fri, 15 Jan 93 19:05:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 19:05:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27206; Fri, 15 Jan 93 19:05:40 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00450; Fri, 15 Jan 93 18:05:36 CST Date: Fri, 15 Jan 93 18:05:36 CST From: Tony Skjellum Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu Subject: Re: Communication contexts Status: RO In Zipcode, contexts have a group-wide scope. Groups of processes can involve more than one context. For instance, if 10 processes are involved in several stages of a calculation, each stage might have a different notion of how to do messaging, but each would use all the processes. Groups are important, because they allow control over the scope of global operations. Contexts are important because they provide guarantees to sub-programs about restricted scope of messages. Contexts are added typing-like information, but which is controlled by the system, not ad hoc by each user program. When they determine to start messaging, sub-programs request a context from a "postmaster general" in Zipcode. Currently, this is a group-oriented request (loose synchronization) resulting in each group member getting the context, and other information. In a lower-level system, simpler tactics might be possible. The way we did it was supposed to avoid possibility of races or other problems like that. To build large-scale software, without globalization of the local use of messaging resources, and to provide scalable algorithms with efficient global operations, both groups and contexts are needed. The push/pop state is an implementation detail of little importance. -Tony From root Sat Jan 16 02:08:07 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA15898; Sat, 16 Jan 1993 02:08:06 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA09478; Sat, 16 Jan 93 02:08:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11696; Sat, 16 Jan 93 02:06:16 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 16 Jan 1993 02:06:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11688; Sat, 16 Jan 93 02:06:13 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA06036; Sat, 16 Jan 93 02:01:08 EST Date: Sat, 16 Jan 93 02:01:08 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301160701.AA06036@cs.wmich.edu> To: mpi-comm@cs.utk.edu Status: RO hi; It was my feeling that the concepts of group and context, that were clearly understood differently by different people at the start of the discussions, became somewhat better agreed on as the pt2pt discussion took place. Groups - used to define groups of processors for collective communication primitives and to allow mapping topologies onto processors. This allows a group of processors to be used in isolation to compute together. Contexts - As Tom (and Tony) express below, > From: hender@macaw.fsl.noaa.gov (Tom Henderson) > Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> > To: mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > I'd like to add a 5th approach, basically the one proposed by Paul Pierce: > > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. The only expression stated so far about the difference between tags and contexts I had gleaned was that context should now be wildcardable (IE no -1 for All contexts) while a receive ALL or some other MASK variant would be allowed on tags. > > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) Add: 5d) A context is not strictly managed, but use should follow suggested guidelines. This could allow static allocation of contexts when reasonable, but also support a context server (as the Postmaster General in Zipcode). Think of the static contexts or non-registrar provided contexts as reserved with respect to any registration system. This could be part of the MPI initialization call, if a registrar is included. The drawback here of course is the user could violate whatever guidelines were given. It seems to me most types of name registration or stack allocation schemes, that are not too restrictive and allows user code to interact, are likely to allow errant management of contexts as well. I expect that allowing 5d) would let strictly controlled context systems to be built over MPI. > From: Tony Skjellum > Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> > To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > In Zipcode, contexts have a group-wide scope. Groups of processes can Of course if we eliminate groups from the pt2pt call then contexts may not be explicitly limited to groups in MPI. (though a context server could be asked to provide a context relative to a group). > involve more than one context. For instance, if 10 processes are involved > in several stages of a calculation, each stage might have a different notion > of how to do messaging, but each would use all the processes. Groups > are important, because they allow control over the scope of global operations. > Contexts are important because they provide guarantees to sub-programs > about restricted scope of messages. Contexts are added typing-like > information, but which is controlled by the system, not ad hoc by each > user program. When they determine to start messaging, sub-programs > request a context from a "postmaster general" in Zipcode. Currently, this ... > To build large-scale software, without globalization of the local use of > messaging resources, and to provide scalable algorithms with efficient > global operations, both groups and contexts are needed. The push/pop > state is an implementation detail of little importance. I expect the use push/pop for context state would make implementation in some environments hard and I have some concern about requiring registration on very large systems. I do think a strictly controlled stack context system could be built over 5d). The question about context management relates to how high a level MPI should be at. I see it at a low level, that higher level systems can be built on. That is one reason for keeping context management "advisory". > > -Tony > I liked the zipcode documentation. I think it would be a good idea to make it available on netlib. john From root Sat Jan 16 06:08:50 1993 Received: from msr.EPM.ORNL.GOV by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13121; Sat, 16 Jan 1993 06:08:44 -0500 Received: from CS.UTK.EDU by msr.epm.ornl.gov (5.67/1.34) id AA09829; Sat, 16 Jan 93 06:07:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00620; Sat, 16 Jan 93 06:03:55 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 06:03:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00612; Sat, 16 Jan 93 06:03:37 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21711; Sat, 16 Jan 93 03:03:35 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01231; Sat, 16 Jan 93 03:08:56 PST Message-Id: <9301161108.AA01231@jewel.us.oracle.com> Date: Sat, 16 Jan 93 03:08:56 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: RE: Communication contexts Status: RO > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather Allow me to clarify my sentiments. The example of context usage given in the MPI draft documentation appears to explicitly assume the existance of threads, and here contexts appear to be being used as threads. > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. I could strongly support almost all of the above. [The part I wouldn't support is a "tag" as I will explain below.] We do our work on the nCUBE, which, as you may know, has a message passing interface essentially equivalent to that proposed by MPI. We ran into the exact problem quoted above: it was difficult to write libraries that we're guaranteed not to conflict in their usage of message types. The usage of the word "contexts" in the above is equivalent to the usage of the word "ports" amongst operating systems programmers. Because "port" is slightly less overloaded than "context", I prefer the word "port". Note that ports are sufficiently powerful that they subsume the concepts of "contexts", "pids", and "message types" or "tags" as used by MPI. >The only expression stated so far about the difference between tags and >contexts I had gleaned was that context should now be wildcardable (IE no >-1 for All contexts) while a receive ALL or some other MASK variant would >be allowed on tags. I agree that there is little difference between tags and contexts. This helps explain why ports are so good at subsuming both contexts and tags. > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) I don't particularly understand the above, probably because I haven't been listening in on enough of the discussion. I think I understand what you mean by stack-based management, and we do use the word "registration"... When using ports, stack-based management isn't an issue for the same reasons you don't manage pids and message types using stack based management. For example, when sending a message, instead of using the MPI push_context (contextp); send (dest, type, buf, buflen); pop_context (contextp); you use send (port, buf, buflen); In our usage, a "port" is simply a queue to which messages can be sent. The receiver can remove a message from the head of the queue. Wildcards are neither needed nor implemented, making ports more efficient. The efficiency arises from the fact that a process can have multiple ports. In MPI, you essentially implement one receive queue for each pid. If you want to use a wildcard to access messages out-of-order, you need to waste time scanning the queue. With multiple ports, however, you can set up multiple queues so that you never need to access anything other than the head of a queue. Sometimes ports do need to be "registered". In our terminology, registration is the process of publishing in a well-known location sufficient information about a "port" created by one process that another process can send messages to that port. This is a very high level action that is almost outside the scope of MPI. We use a nameserver for this purpose. (The nameserver is accessed via a well-known port, i.e. a port whose bit-pattern is a well-known constant.) [Do note that we require that the bit-pattern representing a port be allowed to be transmitted between to processes without the operating system needing to know that the bit-pattern represents a port. This means that accessing the nameserver is not required in order to use a port.] For groups, we have thought about extending the concept of a port to an array-port. The simple implementation of this allows each process in a group to create a port. These ports are all sent to a central location which creates an array of ports and broadcasts the array to all ports in the array. The complex memory efficient implementation may require processes of a group to be created in a special fashion. Basically, it allows the address of a port for a specific process in a group to be easily computed by another process given the port of the first process in the group and the index of the target process in the group. -- Chuck From owner-mpi-context@CS.UTK.EDU Tue Jan 19 08:54:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14013; Tue, 19 Jan 93 08:54:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13129; Tue, 19 Jan 93 08:53:41 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 19 Jan 1993 08:53:40 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Tue, 19 Jan 93 08:53:39 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16410; Tue, 19 Jan 1993 08:53:38 -0500 Date: Tue, 19 Jan 1993 08:53:38 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301191353.AA16410@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Chair Tony Skjellum of Mississippi State University has kindly agreed to chair the MPI subcommittee on contexts. Regards, David From owner-mpi-context@CS.UTK.EDU Tue Jan 19 11:37:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17976; Tue, 19 Jan 93 11:37:59 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21893; Tue, 19 Jan 93 11:37:36 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 19 Jan 1993 11:37:30 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21879; Tue, 19 Jan 93 11:37:21 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13277 (5.65c/IDA-1.4.4); Tue, 19 Jan 1993 11:36:49 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA27411; Tue, 19 Jan 93 16:36:38 GMT Date: Tue, 19 Jan 93 16:36:37 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301191636.AA27411@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05440; Tue, 19 Jan 93 16:35:18 GMT To: csimmons@us.oracle.com Cc: mpi-pt2pt@cs.utk.edu, mpi-context@cs.utk.edu In-Reply-To: Charles Simmons's message of Sat, 16 Jan 93 02:21:14 PST <9301161021.AA01226@jewel.us.oracle.com> Subject: mpi Content-Length: 3447 To: mpi-pt2pt@meiko.co.uk, mpi-context.@meiko.co.uk (Apologies to those who get it twice) Gentlepeople, Chuck Simmons has raised some interesting issues, with which I have a great deal of sympathy. The original messaging model which we implemented on our machines (and which we still support, and expect to continue to support) is in exactly the vein which Chuck is asking for (and has been used for an Oracle port !). In particular CSN supports 1) multiple end points for communication in a single process (know as "transports") 2) No tagging of messages 3) No system buffering, but the ability for users to queue multiple non-blocking receives on a transport, thus providing the buffering they require. 4) Send and receive by struct iovec. (As Chuck observes, the implementation ends up building an iovec (on the stack) for the simpler forms) 5) Both blocking and non-blocking tests for I/O completion. (our test actually has a timeout value). 6) The ability to pass transport addresses around the machine without the system being involved. 7) A standard name server service to associate textual names with transport addresses. If people are really interested then I can probably send the man pages. (Though this is not the main point of this note). HOWEVER (and this is one of the points) although this system had clean, specified semantics and a fast implementation (on our hardware), it hasn't helped to sell machines, and we have now produced an NX style interface as an alternative. I think that the problem with popularising such an interface among users is that it appears to do less for them than a model with implicit buffering, and is therefore harder to start to use. The fact that it allows them greater control and can ultimately produce higher performance is not their immediate concern and does not therefore count for much. Many FORTRAN programmers in particular to do not wish to be concerned with "system programming issues" like buffer management. I would certainly like MPI to be able to support such an interface (so that we can achieve the higher performance it offers, and also make MPI applicable in non-scientific application areas). (Though those who were present in Dallas will have noted that I wasn't trying to push such an approach there, mainly because I think it's a lost cause. NX style is what we have to live with...) (Second point coming up...) However it crosses my mind that there may be some potential for embedding such an interface within the MPI model. (This is a vague thought, but I'm throwing it out so others can think about it as well). Observe that 1) Marc's persistent communication descriptors seem to remove the tag matching requirement (though they're still rather vague !) 2) One way of viewing contexts would be to implement a context as a particular queue of messages (or in other words an entirely separate communication end point) within a process. (The implications of this for contexts would be a) contexts must be declared before use b) their number may be limited c) they should be freed after use ) The combination of the two things could then come somewhere near to what Chuck is asking for... -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 11:00:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21520; Tue, 2 Feb 93 11:00:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22663; Tue, 2 Feb 93 10:59:18 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 10:59:17 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22655; Tue, 2 Feb 93 10:59:11 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA28603 (5.65c/IDA-1.4.4 for ); Tue, 2 Feb 1993 10:59:07 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA22488; Tue, 2 Feb 93 15:58:57 GMT Date: Tue, 2 Feb 93 15:58:57 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302021558.AA22488@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03309; Tue, 2 Feb 93 15:57:21 GMT To: mpi-context@cs.utk.edu Subject: Restricted contexts Content-Length: 6135 A view on contexts ================== At the last meeting in Dallas there was a strong feeling that some form of "context" was required in MPI. The major reason for this is the desire to separate communications on behalf of (independently compiled) library routines from those performed by the user's application. The use of specific message tags for this is discounted because it would require tag registration both by the library and the user code. This intrusion into the user's constitutional right to use whatever tags she sees fit is viewed as unacceptable, hence the requirement to add what is essentially just another tag field. [Of course here in Britain we don't have a written constitution, and the government just does what it likes...] In fact adding a further tag field (called context) does not remove the registration requirement, though it does mean that registration is only required by library routines. (In effect a default context is reserved by the allocator for the user, and she need not concern herself about it.) I should like to propose a different view of what a context is. Suppose we think of each context as a separate communications end point in the process (and intend to implement it this way too !). This has various implications :-- 1) A context should be created before it can be used. (This has the effect of allocating a new message queue inside the MPI implementation). 2) The number of contexts will be limited (you might expect < 256) [ I don't see this as a problem, after all you still have all of the tags available in each context, and how many nested libraries do you expect to use ? ] 3) A context should be deleted when use of it is complete. (So that the system can release the queue. This also gives a good point at which to report errors such as unreceived messages still on the queue). 4) Creation of a context can be a local operation. (Since the context is probably just an index into a local table. cf. unix file descriptors) [Though this may not be true when collective operations filtered by context are taking place, in this case one will want the same context in all members of the collective group, perhaps some contexts need to be reserved for allocation in group.] 5) Context values can be passed in messages without needing translation or registration. (This is important for maintaining scalability. The context information need only be passed to those processes which require it. [Though there's an unaddressed bootstrap problem here...]) 6) It is still possible to have a context lookup/allocation scheme if necessary. 7) It is possible for the user to avoid most of the tag matching cost (as has been requested by Chuck Simmons), all she (the user, not Chuck!) has to do is allocate contexts and target messages at them. Within a particular context all receives can have tag MATCH_ANY, and no queue searching is required. Since the context selection is fast (because we have restricted the number of contexts so that it is sensible to implement the selection as an array access) this improves the achieved performance. Given this style, then the implementation is extremely easy (and will be fast). One simply has an array of message lists, the context number being the index into the array. All of the operations (such as tag matching) can then use the existing code. This will (in general) produce an implemetation which runs faster than having a single queue of messages and matching on both tag and context, since the message queues will be shorter, and only contain relevant buffers. [Contrast with the case of additional code to match contexts on one message list which would slow down all code, whether or not contexts are being used !] The model of communication is that the context is a part of the TARGET address, both for collective and diadic communications. In effect the end point for communication is only relevant at the receiver. [Is this what we want ? The only place it seems to make a difference is in the tests for completion of non-blocking sends. If the user has queued many of these on the single outgoing queue, then when the library is searching for completed ones it has to skip them. If we have many outgoing queues this is not necessary. It rather depends on the way we test for completion of non-blocking sends, which has yet to be determined...] Functions might look like typedef ... MPI_CONTEXT; MPI_CONTEXT MPI_NEWCONTEXT(MPI_GROUP group); Allocate a new context for use within the given group (may be MPI_NO_GROUP, in which case the allocation is entirely local). Returns MPI_NO_CONTEXTS if there are no suitable free contexts left. If the group is not MPI_NO_GROUP, then this is a barrier synchronisation of all the processes in the group, and the result of the function is the same in all of the participating processes. int MPI_FREECONTEXT(const MPI_CONTEXT context); Free the given context, returns MPI_SUCCESS if the context has been freed MPI_NOTCONTEXT if the context is not valid MPI_BUSY if the context still has outstanding messages (I'm not sure if we need this last one, but the model of unix file descriptors suggests it might be useful...) int MPI_MOVECONTEXT(const MPI_CONTEXT to, const MPI_CONTEXT from); Rename the context from to be the context to. (cf dup2). This is a simple renaming operation. MPI_SUCCESS if the context has been renamed MPI_NOTCONTEXT if the from context is not valid MPI_BUSY if the to context is already allocated One of the arguments to the send routines (both diadic and collective) would then be an MPI_CONTEXT to which the message will be sent. Feedback please. Although this is a long message I'm not offering beer. [The pound is now down to $1.45 and still falling !] -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 11:49:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22912; Tue, 2 Feb 93 11:49:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24957; Tue, 2 Feb 93 11:49:03 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 11:49:02 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24949; Tue, 2 Feb 93 11:48:58 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA28146; Tue, 2 Feb 93 16:48:54 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01526; Tue, 2 Feb 93 09:47:49 MST Date: Tue, 2 Feb 93 09:47:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302021647.AA01526@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: Re: Restricted contexts A few (probably dumb) questions about Jim's context proposal: > MPI_CONTEXT MPI_NEWCONTEXT(MPI_GROUP group); > Allocate a new context for use within the given group (may be > MPI_NO_GROUP, in which case the allocation is entirely local). Returns ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > MPI_NO_CONTEXTS if there are no suitable free contexts left. > If the group is not MPI_NO_GROUP, then this is a barrier > synchronisation of all the processes in the group, and the result of > the function is the same in all of the participating processes. In what situations would I want a context that is entirely local? Would I need to send this context to a process I intend to communicate with? I like the idea of being able to create a context within a group for scalability reasons. How do I know that two groups will not get the "same" new context returned from separate calls to MPI_NEWCONTEXT()? Is this even a problem? > One of the arguments to the send routines (both diadic and collective) > would then be an MPI_CONTEXT to which the message will be sent. Does this mean that MPI_CONTEXT is NOT a required argument to a receive routine? (Maybe I'm missing the whole point of this proposal! If so, please educate me. A pseudo-code examle might be very helpful.) > Although this is a long message I'm not offering beer. [The pound is > now down to $1.45 and still falling !] This whole beer thing is getting pretty scary. I think I'll keep all my messages short! :) Tom Henderson hender@fsl.noaa.gov From owner-mpi-context@CS.UTK.EDU Tue Feb 2 12:15:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23454; Tue, 2 Feb 93 12:15:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26210; Tue, 2 Feb 93 12:15:20 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 12:15:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26192; Tue, 2 Feb 93 12:15:16 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03830; Tue, 2 Feb 93 11:14:40 CST Date: Tue, 2 Feb 93 11:14:40 CST From: Tony Skjellum Message-Id: <9302021714.AA03830@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, jim@meiko.co.uk Subject: Re: Restricted contexts Ladies/Gentlemen: The notion of context implementation for efficiency should not drive it to be a highly limited field. It is true than if there are only a few contexts, then they could be moved to hardware support. I think that large software will have many hundreds of contexts, and that this resource should be less limited. I dislike the notion that hardware-specific details creep into the standard. For instance, CM-5 could/does have 8 hardware contexts of messages, at a low level, but these are not really published to the user at all. The issue of whether one or many queues are supported is completely separate, and can be addressed separately. Separate queues weaken the partial ordering guaranteee of a message passing system; ie, that messages are received in pairwise-preserved order. Hence, they have been avoided thus far in the work I have done. However, there is nothing in principle wrong with letting an implementation disperse messages as they come in based on context. The best partial screening for messages is highly application dependent. Perhaps some type of adaptive hashing could be used internal to strong implementations, but it should not be required. For instance, independent of context support totally, adaptive hashing on type/sender could be incorporated to existing systems. I do not think it too inefficient to test 64 bits instead of 32 bits for typing. In practice, there are much more serious overheads in a system. The notion that we have full push-down message stacks, is an example of this. Active messages systems are highly context oriented. They make the context bigger (eg, an address on the destination machine) rather than smaller. One can see that there is a basic unity between message selection and context support... contexts help to route messages to the right execution part of a destination process. Registry is very important, because it guarantees that code (be it user or other library code) will cooperate reasonably well, given the message resource. This type of registry is accepted for file systems, for instance, where we accept that FORTRAN unit number is a bad way to program. Unix file descriptors provide insulation between different parts of a program. Same idea for contexts. Is a context associated with a group? Well, I hope so, but enforcing this is less important. Groups are very important in and of themselves, and to have guarantee that a context is available in a group is very useful. It also makes sense to me that a loosely synchronous registry by a group of processes to get a context is a good idea. For instance, a big group of processes initially has a context. It can ask for more contexts, and pinch off subgroups, establishing an MPMD processing environment. Overlap is permitted. Smaller groups of processes, starting with their own contexts would need a way to union into bigger groups. A mechanism for this is needed. This direction is important for scalability, for fault tolerance, and for dynamicism in processing on large collections of distributed or parallel processors. For instance, homogeneous clusters could union into a heterogeneous collection, and the resulting communication structure would provide a tree-like (at least two-level) access to messaging, which could also be useful for optimizing performance of global communication in a heterogeneous setting. Other ideas? - Tony Skjellum From owner-mpi-context@CS.UTK.EDU Tue Feb 2 12:52:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24044; Tue, 2 Feb 93 12:52:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27903; Tue, 2 Feb 93 12:51:29 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 12:51:28 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27887; Tue, 2 Feb 93 12:51:22 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00407 (5.65c/IDA-1.4.4 for ); Tue, 2 Feb 1993 12:51:19 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA22983; Tue, 2 Feb 93 17:51:16 GMT Date: Tue, 2 Feb 93 17:51:15 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302021751.AA22983@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03315; Tue, 2 Feb 93 17:49:35 GMT To: hender@macaw.fsl.noaa.gov Cc: mpi-context@cs.utk.edu In-Reply-To: Tom Henderson's message of Tue, 2 Feb 93 09:47:49 MST <9302021647.AA01526@macaw.fsl.noaa.gov> Subject: Restricted contexts Content-Length: 2607 > A few (probably dumb) questions about Jim's context proposal: Not dumb. > In what situations would I want a context that is entirely local? Would I > need to send this context to a process I intend to communicate with? The basic model I have is that contexts are very similar to Unix file descriptors (maybe I should have said this more explicitly). Therefore you can create them locally, however if someone is going to send a message to one of your contexts the they need to know its identity. So the answer to the second question is YES, you'll have to send it to them [hence the bootstrap problem...]. > I like the idea of being able to create a context within a group for > scalability reasons. How do I know that two groups will not get the "same" > new context returned from separate calls to MPI_NEWCONTEXT()? Is this even > a problem? Since contexts are allocated within a group and context creation is a barrier synchronisation there are two cases to consider :- 1) the two groups creating contexts are disjoint. In this case the contexts are independently created, and may indeed be given the same number. (Just as opening files in two processes can return the same file descriptor). I don't think this is a problem. 2) The two groups share some members. In this case we only have a problem if we have threads (since otherwise a single thread needs to be at two barriers simultaneously, which it can't.) If we do have threads, then there will have to be a lock in the library to ensure that different results are given to the two groups. > > One of the arguments to the send routines (both diadic and collective) > > would then be an MPI_CONTEXT to which the message will be sent. > > Does this mean that MPI_CONTEXT is NOT a required argument to a receive > routine? (Maybe I'm missing the whole point of this proposal! If so, please > educate me. A pseudo-code examle might be very helpful.) No you do need a context in the receive (though it might be optional in the "simple" versions with some specific default value). The issue I was addressing was whether a send needed to specify one context (that at the target), or two (that at the target and that at the sender). I was tending towards the former solution. I'll try to write some examples tomorrow. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 13:12:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24463; Tue, 2 Feb 93 13:12:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28864; Tue, 2 Feb 93 13:10:58 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 13:10:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28850; Tue, 2 Feb 93 13:10:51 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00681 (5.65c/IDA-1.4.4 for ); Tue, 2 Feb 1993 13:10:42 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA23047; Tue, 2 Feb 93 18:10:37 GMT Date: Tue, 2 Feb 93 18:10:37 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302021810.AA23047@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03318; Tue, 2 Feb 93 18:09:00 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Tue, 2 Feb 93 11:14:40 CST <9302021714.AA03830@Aurora.CS.MsState.Edu> Subject: Restricted contexts Content-Length: 4249 > The notion of context implementation for efficiency should not drive > it to be a highly limited field. I'd like the additional cost of contexts to be small. I'd also like it NOT to impact people who don't use contexts. Both of these drive me in the direction of the sort of implementation I was proposing. This is easy provided there are relatively few contexts, hard if they're as anarchic as tags. > It is true that if there are only a > few contexts, then they could be moved to hardware support. Yes, but not the reason for my proposal. I have no desire (or need) to do this. > I think that > large software will have many hundreds of contexts, and that this resource > should be less limited. I can't imagine why you need another 10 bits of context when you already have 32 of tag. What are you going to use the context for ? > I dislike the notion that hardware-specific > details creep into the standard. For instance, CM-5 could/does have 8 > hardware contexts of messages, at a low level, but these are not really > published to the user at all. I agree entirely. My proposal is not based on any specific hardware implementation. (My hardware has a small processor in it which can execute C code and run comms processor threads in a remote user space without requiring any main processor intervention, but I'm not asking for that in the standard !) > The issue of whether one or many queues are supported is completely > separate, and can be addressed separately. Separate queues weaken > the partial ordering guaranteee of a message passing system; ie, that messages > are received in pairwise-preserved order. Hence, they have been avoided > thus far in the work I have done. True, I didn't see this as a problem, since I thought that messages in different contexts could reasonably be unordered with respect to each other. (This was based on my view of what contexts are for, as explained in the intro to my proposal, maybe you have a different use for them in mind ?) > However, there is nothing in principle > wrong with letting an implementation disperse messages as they come in > based on context. The best partial screening for messages is highly > application dependent. Perhaps some type of adaptive hashing could be > used internal to strong implementations, but it should not be required. > For instance, independent of context support totally, adaptive hashing > on type/sender could be incorporated to existing systems. Of course all of these things are possible, and all add to the baseline latency. I was hoping to reduce this, not increase it. > I do not think it too inefficient to test 64 bits instead of 32 bits for > typing. In practice, there are much more serious overheads in a system. > The notion that we have full push-down message stacks, is an example of > this. I haven't seen any proposal in the MPI context which doesn't take this approach, and since I'm not trying to rewrite the whole thing... > Registry is very important, because it guarantees that code (be it user > or other library code) will cooperate reasonably well, given the message > resource. This type of registry is accepted for file systems, for instance, > where we accept that FORTRAN unit number is a bad way to program. Unix > file descriptors provide insulation between different parts of a program. > Same idea for contexts. Exactly. That's why my context proposal is based on the file descriptor model. > Is a context associated with a group? Well, I hope so, but enforcing this > is less important. Groups are very important in and of themselves, and to > have guarantee that a context is available in a group is very useful. It > also makes sense to me that a loosely synchronous registry by a group of > processes to get a context is a good idea. In my model a group can request a context. It does not have one a priori. (Maybe it doesn't need one since it can use one which the relevant processes already have). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Feb 2 13:45:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25590; Tue, 2 Feb 93 13:45:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01587; Tue, 2 Feb 93 13:44:48 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 13:44:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01579; Tue, 2 Feb 93 13:44:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03874; Tue, 2 Feb 93 12:44:09 CST Date: Tue, 2 Feb 93 12:44:09 CST From: Tony Skjellum Message-Id: <9302021844.AA03874@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, jim@meiko.co.uk Subject: Re: Restricted contexts Addendum to my previous comments. I do not think it is a bad idea for vendors to provide limited, extremely fast registry mechanisms, like the CM-5's 8 hardware contexts, or whatever Meiko might have in mind along the same lines. These should be viewed as low-level services that can be incorporated into standard interfaces, increasing their performance, when they can be used. Active messages, for instance, while they are lower-level than the MPI ideas, should not be discarded by vendors. They may be a very good way for strong, MPI-upward-compatible, message systems to do a good job with hardware. We must provide for a way for vendors to extract performance from their hardware, within the context of MPI, or else they will continue to gain commercial advantage from ignoring MPI, rather than supporting it fully. - Tony From owner-mpi-context@CS.UTK.EDU Tue Feb 2 13:52:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25863; Tue, 2 Feb 93 13:52:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02049; Tue, 2 Feb 93 13:52:10 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 2 Feb 1993 13:52:09 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02040; Tue, 2 Feb 93 13:52:08 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03880; Tue, 2 Feb 93 12:51:25 CST Date: Tue, 2 Feb 93 12:51:25 CST From: Tony Skjellum Message-Id: <9302021851.AA03880@Aurora.CS.MsState.Edu> To: jim@meiko.co.uk, mpi-context@CS.UTK.EDU Subject: Re: Restricted contexts Jim, I do not think it is better to keep it a 8 or 10 bits, rather than 32, for the following reasons 1) alignment 2) comparison cost (8 v. 32) on modern architectures Yes, we are going to use more than 256. Libraries that use processes in multiple configurations, and with multiple sub-groupings generate a number of contexts. I will send you the Zipcode 1.0 tech. doc. later (when finished) and you can see for yourself that many contexts can reasonably be used. For each different library in a system, one could imagine using about twenty of these message contexts. It could be up to the implementor to use more/less, depending on how services are being provided, and how service priorities are set. Tags are an unstructured field. 32 bits for the tag may be too much, but when it is all the programmer has to control selectivity, 32 is a good quantity to choose. Context suggests structuring part of the selectivity. In this case, 32 bits may be more than enough. Real software will use contexts. People who don't use them are the people who will be writing their own codes, using no one else's libraries or codes, and basically not doing the same type of top-down design we expect from good software development. I would argue that people who want to save these incremental test costs will also be unwilling to do anything but drive the hardware from the most primitive system calls. - Tony From owner-mpi-context@CS.UTK.EDU Wed Feb 3 05:35:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18460; Wed, 3 Feb 93 05:35:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21954; Wed, 3 Feb 93 05:34:51 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 3 Feb 1993 05:34:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21946; Wed, 3 Feb 93 05:34:47 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13139 (5.65c/IDA-1.4.4 for ); Wed, 3 Feb 1993 05:34:43 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26644; Wed, 3 Feb 93 10:34:39 GMT Date: Wed, 3 Feb 93 10:34:38 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302031034.AA26644@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03416; Wed, 3 Feb 93 10:32:56 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-context@CS.UTK.EDU In-Reply-To: Tony Skjellum's message of Tue, 2 Feb 93 12:51:25 CST <9302021851.AA03880@Aurora.CS.MsState.Edu> Subject: Restricted contexts Content-Length: 1848 Tony, > I do not think it is better to keep it a 8 or 10 bits, rather than 32, > for the following reasons > > 1) alignment > 2) comparison cost (8 v. 32) on modern architectures Alignment may or may not be an issue. I have no objection to you sending the context as an aligned quadword in the message header if that is faster on your hardware. I'm only interested in the restricting range of valid values. My point is that if I have only a few (8 or 10 bits) then I can use a qualitatively different implementation [an array] than if I have to deal with random 32 bit values. Of course the cost of comparison is the same, but if I only have a few contexts I can remove the comparisons from the loop entirely. > Yes, we are going to use more than 256. Libraries that use processes > in multiple configurations, and with multiple sub-groupings generate a > number of contexts. I will send you the Zipcode 1.0 tech. doc. later > (when finished) and you can see for yourself that many contexts can > reasonably be used. > For each different library in a system, one could imagine using about > twenty of these message contexts. It could be up to the implementor to > use more/less, depending on how services are being provided, and how > service priorities are set. I agree with this, BUT this doesn't mean you need that many contexts all at once. The analogy with unix file descriptors is very close. Each library might use 20 open files, but you can still get by with a limit of 64 or 128 on the number of files open at one time. Similarly with contexts. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Wed Mar 3 16:57:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12038; Wed, 3 Mar 93 16:57:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08269; Wed, 3 Mar 93 16:56:38 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 3 Mar 1993 16:56:37 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08261; Wed, 3 Mar 93 16:56:36 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05229; Wed, 3 Mar 93 15:53:56 CST Date: Wed, 3 Mar 93 15:53:56 CST From: Tony Skjellum Message-Id: <9303032153.AA05229@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: Status Dear MPI contexts people, Per our agreement at the last MPI meeting, a sub-sub-committee consisting of Cownie, Clarke, and Skjellum are developing a multi- alternative "closure" proposal for MPI. This will be presented to the entire contexts sub-committee within two weeks. After further discussion, at least two viable scenarios, required to resolve group, context, tag, group ID conflicts/overlaps, will be presented at next MPI meeting. You will receive our proposal for review as soon as possible. - Tony From owner-mpi-context@CS.UTK.EDU Mon Mar 8 17:14:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21363; Mon, 8 Mar 93 17:14:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23364; Mon, 8 Mar 93 17:14:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 8 Mar 1993 17:14:15 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23356; Mon, 8 Mar 93 17:14:14 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08120; Mon, 8 Mar 93 16:10:45 CST Date: Mon, 8 Mar 93 16:10:45 CST From: Tony Skjellum Message-Id: <9303082210.AA08120@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: PRECIS for closure proposal Please comment. - Tony Skjellum ---------------------------------------------------------------------------- Here is the precis of the post-MPI meeting... Input was from Skjellum, Littlefield, Pierce, Ranka [and others I don't recall specifically]. There shall be three self-consistent "closures" for MPI, that completely remove inconsistency between the usage of the following concepts (while fully defining the following concepts): 1) tag 2) context 3) group 4) process ID It is our task to assemble such proposals and to place them before the context sub-committee prior to the next meeting, and provide a refined set of proposals (with recommendations) before the full committee at the next MPI meeting. The first reading of this section of the standard is needed to make the rest of the standard go forward. --------------------------------------------------------------------- Proposal I. GROUP ID = CONTEXT = OBJECT, aka "Snir" model groups, contexts, objects are local and opaque. naming of processes is group-relative. this option was discussed, in most part, by Snir at meeting. Proposal II. GROUP ID != CONTEXT, aka "Littlefield" model GROUP corresponds to "data/instance" in this model CONTEXT is statically defined at link time, corresponding to code or class In this model, group ID's appear explicitly, and separately from CONTEXTS in global operations. It was noted by Littlefield that he would write global operations, and product bits of the GROUP ID and static CONTEXT to guarantee that his code produced unique, deterministic, non-interfering results. Proposal III. GROUP ID null, GROUP scope == CONTEXT scope, aka Zipcode model CONTEXT is a dynamic concept, and global (or as global as necessary) Whenever a group is created, a context is assigned to it. CONTEXTS are used to disambiguate global operations. GROUP ID is the CONTEXT for the group. GROUP is an enumeration of PROCESS ID's OR ONE OF SEVERAL possible hashing formulas, with exceptions (more scalable) Proposal IV. GROUP ID != CONTEXT, aka fully "object-oriented" model CONTEXT is a dynamic concept, and global (or as global as necessary) GROUP ID is emasculated, in the sense that there are process groups, but CONTEXT is still needed in global operations The general case of "code U data" (the object-oriented model) is taken, with context+group forming a scope in the system. The "Server model," where CONTEXTS that have no group association made to them, is to be supported. GROUP is an enumeration of PROCESS ID's OR ONE OF SEVERAL possible hashing formulas, with exceptions (more scalable) ---------------------------------------------------------------------- Requirements for all proposals to meet closure 1) All global operations must be describable in terms of point-to-point operations without loss of generality 2) Global operations, including non-blocking barriers, and so on, must work correctly with overlapping process groups 3) There must be reasonable/scalable mechanism for assigning contexts 4) Tag usage seems to be non-controversial according to above discussion, but we need to watch out for the following. 1) desire to use part of tag for context, group ID or other field, and make that part of the standard 5) Provision of enough contexts to support all describable or reasonable scenarios that we can justify, but not more than are foreseeable, because there could be a performance penalty by requiring the number of contexts to be too great. 6) Justification is perceived to be required for dynamic contexts (why?) 7) A fully consistent definition for how process IDs are to work. 8) An explanation of how to extend to dynamic groups later, if possible. From owner-mpi-context@CS.UTK.EDU Tue Mar 9 18:14:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16448; Tue, 9 Mar 93 18:14:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03977; Tue, 9 Mar 93 18:13:41 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 9 Mar 1993 18:13:39 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03969; Tue, 9 Mar 93 18:13:37 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA15144 (5.65c/IDA-1.4.4 for ); Tue, 9 Mar 1993 17:13:34 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA12285; Tue, 9 Mar 93 17:13:30 CST Date: Tue, 9 Mar 93 17:13:30 CST Message-Id: <9303092313.AA12285@godzilla.mcs.anl.gov> To: tony@aurora.cs.msstate.edu Cc: mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Mon, 8 Mar 93 16:10:45 CST <9303082210.AA08120@Aurora.CS.MsState.Edu> Subject: PRECIS for closure proposal The one point in Tony's excellent summary that needs more elaboration is needed on whether process id's, when being used in point-to-point operations, are group-relative (ranks) or absolute (and possibly opaque). Only Proposal I specifies this (ids = ranks); the others leave this unspecified. In the interests of keeping the latency down, I'd like them to be absolute, at least at the lower levels. In any event, this choice is a critical one but one that is mostly orthogonal to the other issues. In fact, I'd have chosen absolute process id's as the "Snir" model. Bill From owner-mpi-context@CS.UTK.EDU Tue Mar 9 18:40:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16750; Tue, 9 Mar 93 18:40:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05349; Tue, 9 Mar 93 18:40:04 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 9 Mar 1993 18:40:01 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05321; Tue, 9 Mar 93 18:39:58 -0500 Date: Tue, 9 Mar 93 23:39:42 GMT Message-Id: <2472.9303092339@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: PRECIS for closure proposal To: William Gropp In-Reply-To: William Gropp's message of Tue, 9 Mar 93 17:13:30 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Bill writes: > In the > interests of keeping the latency down, I'd like them to be absolute, at least > at the lower levels. In any event, this choice is a critical one but one > that is mostly orthogonal to the other issues. In fact, I'd have chosen > absolute process id's as the "Snir" model. I think we all realise what your choice would be Bill :-) There are arguments both ways, perhaps we should have both. On the other hand, the speed demons probably should be advised to use the communication handle layer at which any overhead of mapping (group,rank) to absolute-implementation-specific-best-you-can-get, and other per initialisation overheads, should be paid exactly once in the handle initialisation. Isn't that exactly why we have this layer? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 9 18:42:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16757; Tue, 9 Mar 93 18:42:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05578; Tue, 9 Mar 93 18:42:14 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 9 Mar 1993 18:42:13 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05570; Tue, 9 Mar 93 18:42:10 -0500 Date: Tue, 9 Mar 93 23:42:06 GMT Message-Id: <2480.9303092342@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: [d39135@sodium.pnl.gov: Re: group descriptors] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Rik writes: > > PS. Tony, I'm a little surprised that your reply to me was not > broadcast, especially considering your reaction to being left > out of a loop earlier. > > I am also concerned that this group is not leaving a publicly > accessible trail. These discussions have a lot of content > that should be available to others even if it does not appear > in the final proposal. > > Should we not be using an mpi-contexts reflector and archiver at > Oak Ridge? > I'm beginning to think this is correct. We seem to be talking things which seem to go beyond the "Four Proposals" agreement. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 00:36:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21055; Wed, 10 Mar 93 00:36:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18558; Wed, 10 Mar 93 00:36:04 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 00:36:03 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18550; Wed, 10 Mar 93 00:36:02 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09605; Tue, 9 Mar 93 23:30:48 CST Date: Tue, 9 Mar 93 23:30:48 CST From: Tony Skjellum Message-Id: <9303100530.AA09605@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu Subject: Re: [d39135@sodium.pnl.gov: Re: group descriptors] Cc: mpi-context@cs.utk.edu I am not sure to which comment I incorrectly misresponded. Rik, please remind me... I think there are a total of two letters which are improperly sent only to you, and I do want them shared with everyone. I was quite rushed today, and hit the wrong R key, no doubt. To my recollection, both deal with Methods within group/contexts. If you want to elaborate mailing to entire mpi-contexts subcommittee, I concur. Let's make sure all of you are on that list, so we don't lose track of the main discussion people, however. So, my basic request to Rik is that he echo the two letters sent to him alone, which was my error. Mea culpa. :-) Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 01:10:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21883; Wed, 10 Mar 93 01:10:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19605; Wed, 10 Mar 93 01:09:48 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 01:09:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19597; Wed, 10 Mar 93 01:09:46 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09722; Wed, 10 Mar 93 00:05:40 CST Date: Wed, 10 Mar 93 00:05:40 CST From: Tony Skjellum Message-Id: <9303100605.AA09722@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov Subject: Re: group descriptors Cc: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu This corrects the second message, accidently sent only to Rik. - Tony From tony Tue Mar 9 23:23:17 1993 To: d39135@sodium.pnl.gov Subject: Re: group descriptors Content-Length: 897 Rik, I was foolish not to mention that we have implemented a registration mechanism with trees, in Zipcode 1.0. The user can add available methods for combine, and then,when a mailer is created, specific choices can be taken from the list. Currently, we implement with numbers, but this could be done with names, to allow better sharing. I will check on choices we made. It has been some time since I worked on that internal code. Anyway, I concur on the need to have registerable choices, that allow sharing, per your letter. As regards new functionality, they could go in the tree too, but there is no structural mechanism in Zipcode, other than defining a class' method list, for adding new instantiations to a mailer at creation. All of this was very elaborate coding, which I will elucidate in my article for Rolf. I am happy to share details with you as well, beforehand. - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 06:16:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13775; Wed, 10 Mar 93 06:16:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06917; Wed, 10 Mar 93 06:15:36 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 06:15:33 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06900; Wed, 10 Mar 93 06:15:29 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09848; Wed, 10 Mar 93 05:10:50 CST Date: Wed, 10 Mar 93 05:10:50 CST From: Tony Skjellum Message-Id: <9303101110.AA09848@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: re: static contexts writeup [My comments prepended with **** - Tony] **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with **** the first-generation PRECIS, to generate a more useful one, **** Please correct me if I am wrong, but I cannot find this! **** (Again, I apologize for calling this a PRECIS, not a SKETCH.) **** I want to generate the next version of this SKETCH/PRECIS, **** with specifics garnered from our lengthy bantering and **** discussions. Particularly, I will discuss the sparse matrix **** of proposals X closure options. Lyndon, you indicated that **** one of my comments challenged you to make a new added entry **** to one of the proposal categories. May I have that now/soon? **** **** In interest of refocusing discussion, I will work on such a **** new PRECIS/SKETCH, reviewing all the mail flurry from last days, **** and then suggest writing assignments. Lyndon will be off-line, but **** Lyndon will still get some brain work to do off-line. :-) Lyndon **** will you phone me periodically, to stay abreast of things? **** Perhaps we will need the manpower of Rusty/Bill to help us, given **** the amount of work on filling the sparse matrix of issues. Willing? **** **** Our PRECIS/SKETCH and work from it will not address the following things **** (related to discussion in our group, but not central to it): **** - the fully portable MPI program **** - I/O **** - the semantics and syntax of process management **** - the time-table for MPI-1,MPI-2,MPI-3,... **** In a truly global sense, these could be construed to be closure as **** well, but I want to exclude them for sanity reasons. I do believe **** that parts of these issues could be addressed within MPI-1, but **** it is not now this subcommittee's role to do that for MPI at-large. **** Strong objections? If so, we are in deep trouble:-) **** **** We will assume that discussion in this 4-proposal X closure effort is **** all essential to MPI-1. Anything you really don't think is essential **** for MPI-1... once we agree to same .... should probably get deferred. **** Is that controversial? **** **** I thank everyone for their sincere efforts. **** - Tony **** **** PS Per constructive advice from Rik, all of my mail is getting copied to **** mpi-context now. All should do so. I did not hear any dissent **** about this suggestion, and it adds potential help to **** discussion. As we need the help, and few people are writing **** anyway, this will not likely cause problems. **** Continue to include the five names jim, lyndon, tony, sanjay & **** Rik, incase any of us are not on the mpi-context mailing list!!! **** **** PPS Lyndon, where are you. I got up at 4am to read the next round **** of mail from you, but none to be found :-) **** **** Rik writes... >Before it gets TOO stale, here's a copy of my writeup on static >contexts et.al. as it last went to Lyndon and Tony. (You two can >stop reading at the line of dashes.) > >I have opted not to hack on static contexts any more because I >would rather kill them. **** Is this a proposal to include static contexts as in within **** the 4-proposal model, and that we stop working on same? **** If so, I can make it so... >I now suspect that I would trade static contexts for something >like the MPI_set_group_attribute cacheing facility that I have >outlined in recent email. I will think more about this and get >back to you. > >In the meantime, I would very much like more feedback on the >cacheing facility, especially how it breaks. > >Thanks, >--Rik > >---------------------------------------------------------------------- > >This note discusses message context ID's -- what they're good for, >how to generate them, how to use them. > >Credits.. At the last Dallas meeting, Mark Snir asked me to write >up some thoughts on this topic and to send them around to the >whole committee. Paul Pierce, Tony Skjellum, several other >people (whose names I can't remember) and I discussed some of the >ideas over lunch on the last day. Then Lyndon Clarke of the >contexts subcommittee emailed me asking for some clarification of >static context generation, which I guess I'm the main proponent >of. He reviewed an earlier (and even longer) draft of this note, >and it is now much improved. (Thanks, Lyndon!) However, I have >substantially rewritten it since talking with any of those folks, >so I have to take full blame for the fallacies of this version. > >1. Notation and assumptions > > In this note, I assume that > > . "context ID" means a message selection field > that cannot be wildcarded, > > . "tag" means a message selection field > that *can* be wildcarded, and > > . context ID, tag, and process ID are the *only* fields > that can be used for pt-pt message selection. > > In particular, I assume that "group ID" cannot be used for > selection unless it is somehow propagated into the tag or > context ID fields. > > I will speak of "modules" that operate within "groups". By > "module", I mean a procedure (subroutine) or collection of > procedures that can exploit knowledge of each other's internal > workings in order to coordinate their message passing. By > "group", I mean a collection of processors identified by a > globally unique "group ID". By "operating within a group", I > mean that the module is called simultaneously by all processes > belonging to the group and exchanges messages primarily with > those copies of itself. (I assume that arbitrary communication > is always possible. The focus of this discussion is how to > make restricted communication easy.) I assume that the group > ID is passed into the module. > > I will also speak of an "instance of a module". By this, I > mean a module, plus the group that it is operating in, plus the > data it is working on. (The latter becomes important if the > module supports nonblocking operation. In that case, I assume > that some data ID is passed into the module. An example is the > 'tag' argument of the nonblocking collective routines.) > > My notation requires that independently written modules can be > called sequentially within the same group. Specifically, if a > a group must be "duplicated" in order to guarantee that two > modules do not conflict, then I will say that those modules are > operating in different groups. > > By definition, independent modules do not coordinate their use > of tags. (A module can of course coordinate with itself. > That's what tags are for.) > >2. Why do we want context ID's? > > The most important purpose of context ID's is to avoid message > collisions between independent modules. (This is the only way > to avoid such collisions, since coordination of tag values is > excluded by definition.) > > Context ID's are sometimes also be the best way to avoid > collisions between instances of the same module. (This depends > on the definition of "best", as discussed below.) The tag > field can also be used for such avoidance, since the module has > full control over how the tags are used. > >3. Desirable features of contexts and groups: > > I think there will be little debate about these: > > A. It should easy to write modules that don't conflict with the > same or other modules, even when the modules are called > within overlapping groups. (Or, save us all, with multiple > outstanding non-blocking operations.) > > B. It should be easy to call these modules. > > There may be more debate about these, but I like them as sanity > checks: > > C. The MPI routines that manage groups and contexts should > be definable in terms of MPI point-to-point communication > and process-local operations. > > (For example, I get nervous about group and context > management semantics that require asynchronous servers or > their equivalent, like an asynchronous atomic > fetch-and-increment. We already decided to eliminate > interrupt receive at the user level, on the argument that > it was too hard to implement correctly. Asynchronous > servers are a bit easier, since they can run on > resources other than the processes calling MPI, but > personally I would rather avoid servers if possible.) > > D. The "closed-system constraint": standard global operations > (e.g., broadcasting within a group) should be definable in > terms of MPI point-to-point and group/context operations. > > (For example, we must be careful to avoid bootstrap > problems like needing a context specific to a group, in > order to create the group.) > >3. How do modules get and use context ID's ? > > Assume that we have already created a group and distributed its > group ID, and that we have a mechanism for broadcasting within > the group. (More on this later.) > > Then we have at least the following options for managing > context ID's: > > Option 1. "Context ID = group ID" (and no group-dup'ing). > > By my definition of "same group", i.e. dup'ing gets you > a different group, this option means that two modules can > get called back-to-back with the same context. > > I think this is OK, courtesy sequencing constraints, if and > only if both modules use exact match on tag and source. > Since the group ID is already available (passed in), this > provides a simple and efficient mechanism for many useful > algorithms, such as blocking global ops. > > This option fails, however, with modules that use wildcard > tag or source. (I won't bother to provide an example -- > having the same context is the same as having no context, > and we know what havoc that creates.) > > Option 2. One process obtains from MPI a unique context ID > for each instance of a module, and distributes it > for that instance's use. > > I believe that this is the strategy proposed by Tony Skjellum > in his 3-page handout at Dallas. The distribution might be > done either within a module or by its caller. Presumably > there are more callers than modules, so doing it within the > module is preferred. As noted above, distribution of the new > context can be done with a broadcast using exact match on > tag and source, with context=group, presuming of course that > the module follows the rule of doing wildcard matches only > in a different context. > > In the Dallas discussions, it seemed to be assumed that this > option requires an asynchronous server, since the context > generation calls have to be coordinated to ensure uniqueness > but can occur at any time. This bothered me. > > However, I realized while writing this note that this option > can also be done without a server by having a process-local > pool of preassigned context ID's. For example, let the > bottom bits of a context ID be the ordinal (in some global > numbering scheme) of the process that assigned it, and manage > the upper bits however you would in a server. For P > processes, this would require at most log(P) extra bits in > the context field, compared to having a server. > > So now my main concerns about this approach are the extra > communication to distribute the context ID's, plus a nagging > feeling that there's still a bootstrap problem if this is > the only option. > > Option 3. "Static context ID's" > > This is the option that I pushed at the Dallas meeting. > > The idea is that each module gets a context ID that is based > solely on a source code key, then manages its tag values to > keep the groups and instances separate. > > To be definite, assume that a module can obtain a context ID > via a call like > > error = MPI_get_context_for_name (name, &contextID) > > where is the character string unique to the module, and > the generated context ID depends only on the supplied name. > This call is made independently on all processes wishing to > share the context. > > The simplest convention would be to require that be > the name of a procedure within the module. That way the > linker guarantees uniqueness. There are other ways to get > unique strings, such as the "dollar bill algorithm" and > whatever strategies people use to reserve names for expansion > of libraries, but these do not affect the basic idea. > However, you cannot use the address of a routine in place of > its name, since this would not work in heterogeneous systems. > > The most efficient strategy is to require that all names be > "registered offline". By this, I mean that MPI be given a > list, prior to execution, of all the names that might be > passed to MPI_get_context_for_name by this particular > program. There are several ways that this could be > accomplished. For example, if we require that the names > always be those of modules, then a (conservative) list can be > generated at link time using Unix 'nm' or equivalent. > > With offline registration, MPI_get_context_for_name can > be implemented so as to require no communication at all. > > We can go farther and require that be a constant > string, in which case the call to MPI_get_context_for_name > could be implemented as a compile-time macro substitution. > (I hate citing it, but as precedent I note that this strategy > is used by the optimizer for some versions of 'Linda'.) > > Without offline registration, it seems that MPI_get_context_for_name > would require an asynchronous server. I will not discuss this > further because in my view static contexts are intended to > make for simple implementations and fast execution. > > OK, now we can quickly and efficiently (zero execution > cost) get a context ID that is unique to the module, but all > instances of that module have to share the context. (Please, > no kneejerk reactions -- hear me out before labeling this as > a stupid idea in complete violation of the spirit of contexts.) > > How does the module keep multiple instances straight? > It uses the message tag field. > > The idea is to generate an MPI tag value by smushing together > the group ID, data ID (if needed), and the nominal tag value > that the module would have used if it didn't have to worry > about multiple instances. No other module cares how the > smushing is done -- the context is unique and we have agreed > that modules can use the tag bits however they want. This is > just an example of a module coordinating with itself > regarding how to handle the tag bits. > > Yes, this scheme does seem like jumping through a hoop, but > it does have some advantages. If the group and data ID's are > short enough to be smushed into a message tag without losing > any bits, then the MPI tag can be generated with absolutely > no communication. This means that: > > 1. It provides a method to implement other group > and context management functions that is bulletproof > and free of bootstrap problems. > > 2. It executes fast. > > To understand the first advantage, consider the problem of > how to implement MPI_close_group(gid) using only MPI pt-pt > operations. Remember that the module executing within that > group on other nodes can legitimately still have a wildcard > receive posted. If MPI_close_group tries to use the group > context, its messages can get eaten. Sync the group first? > That just means you have the same problem implementing sync. > Having static contexts for MPI_close_group and/or sync solves > this problem. Are there other solutions for which MPI pt-pt > operations would be sufficient? > > The second advantage is perhaps weaker, but I for one keep > finding myself writing modules that are sensitive to > communication performance, particularly latency. In many > cases, I would prefer to write a few extra lines of code than > to use functionality that would cost extra messages. I > have lots of experience with tag-smushing due to working > with low-functionality communication systems (like select > on message tag only, and no useful wildcarding). I find > that extra coding effort is very small. Typically it's > embedded in a macro call or cover routine, so that I still > have to write only one line of code to handle a message. > > Note that I do NOT propose static contexts as a solution for > everyone. However, they are useful for speed, and may be > required under the closed-system constraint. > >4. Summary > > It looks to me like all three options presented here are > complementary. They might all might be provided, in the > expectation that they will be used by different audiences. > >Questions, comments? > >--Rik Littlefield > > From owner-mpi-context@CS.UTK.EDU Wed Mar 10 06:34:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17635; Wed, 10 Mar 93 06:34:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07441; Wed, 10 Mar 93 06:33:22 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 06:33:21 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07433; Wed, 10 Mar 93 06:33:19 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA09870; Wed, 10 Mar 93 05:28:59 CST Date: Wed, 10 Mar 93 05:28:59 CST From: Tony Skjellum Message-Id: <9303101128.AA09870@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@CS.UTK.EDU Subject: latest lyndon-mail **** My latest comments start lines with ****. **** - Tony >From lyndon@epcc.ed.ac.uk Wed Mar 10 05:12:18 1993 >Received: from epcc.ed.ac.uk (daedalus.epcc.ed.ac.uk) by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); > id AA09849; Wed, 10 Mar 93 05:11:49 CST >Date: Wed, 10 Mar 93 11:14:44 GMT >Message-Id: <2772.9303101114@subnode.epcc.ed.ac.uk> >From: L J Clarke >Subject: [Tony Skjellum: Re: [Tony Skjellum: Re: Intra/inter group communications, UBONBOUND, PIDs]To: Tony Skjellum , Jim Cownie , > Rik Littlefield , > Sanjay Ranka >Reply-To: lyndon@epcc.ed.ac.uk >Apparently-To: ranka@top.cis.syr.edu >Apparently-To: d39135@sodium.pnl.gov >Apparently-To: jim@meiko.co.uk >Apparently-To: tony@aurora.cs.msstate.edu >Status: R > >Good morning > >Lyndon replying to Tony replying to Lyndon replying to Tony replying to ... > > >> >(again). Please help. >> > >> >First way, "context" is being used to indicate is a binding of group >> >plus more, and therefore I understand it easily. >> > >> >Second way, "context" is being used in a way which I understand. >> > >> ***** I think my comment made more sense to me when I wrote it before. >> ***** I withdraw it. > >Arrrghhhh! I realise that I have badly typed the "Second way", which >should say that "context" is being used in a whay which I do not >understand. This is very badly typed indeed, apologies. > >I think this was a request for information/sharing of understanding, >which I expect I still need. **** I was viewing the whole context-thing as a "port" specification. **** How about a domain-wide name for a piece of mail to be delivered? **** Such as mail from UK to US? **** I am still not sure this was cogent... **** >> >The point is that you do not actually want to produce a new group. You >> >want to keep thinking of the two groups as seperate entities, each >> >acting individually in a coherent fashion, and communicating with one >> >another perhaps according to some rules based on ranks in each >> >indicidual group. >> ***** I understand that, but why do you want to keep thinking of it that way, >> ***** except for some possible performance benefit? >> **** > >I tell you why, it's because we find it easy to think of it and discuss >it in that way. This really is the whole point, expressive power. **** **** I believe in the expressive-power argument, no problem. I am **** also willing to believe that we should keep the minimum cost of **** inter-group down to the point where it remains useful. It is **** dynamicism of groups that motivates inter-group communication. **** Else, the one-shot cost of merging groups, and staying intra-group **** would not be unacceptable. There must be some ineffable quality **** of groups (dynamic need to change, for instance) that you hope for! **** I agree with this abstractly, but please help me more... **** >> > >> >Same "I can read this two ways" as above. >> ***** OK, I withdraw this remark too. > >Look, I found it vague. I'd rather you helped me to understand, please. > **** Related thoughts... **** I am trying to understand my original comment, myself. I think **** of context/group/rank_id as a MPI_TID, if you wish. It all made more **** sense to me structured that way, rather than having a pair of **** contexts as a mechanism for inter-group. In short, I felt we were **** inventing our own TID called context/group/rank_id, and that this **** was a nice structure, and superceded the PVM notion of a TID. **** If I am foggy, sorry, but if we worked in those terms, with accessors **** on such a TID, we would know how to handle some of the problems **** described elsewhere. For instance, we could have the meaning of **** rank_id be group-scope/dependent. Rank would be a nice, fast hash on **** static groups, and a slower, more complex thing to map for **** intergroup [are people still liking NUL/UNBOUND???]. **** >> > >> >If this is an invitation to add a further proposal to the set we are >> >doing, then I accept such invitation. In fact, I can frame it as an >> >extension of proposal I. >> ***** YES. > >Will do. > >> > >> >> Of course, point-to-point could provide another set of procedures which >> >> handle inter-group communication cleanly. This is my preferred >> >> approach, but again I don't suppose that this will be acceptable, >> >> because so may people are already so concerned about the number of >> >> procedures in the point-to-point section. >> >> **** >> >> **** Again, why not try. We need to fill in our matrix of proposals X >> >> **** closures :-) >> >> **** >> > >> >Because I perceive that it would be like trying to drag a dead whale >> >along a beach :-) (You never tried that? I can tell you, its hard work >> >and slow going!) >> ***** Acknowledged. > >Ah, so you tried pulling a dead whale as well. Funny old world :-) > >> ***** I think I was interested in implementation with p2p, because >> ***** of the degrees of freedom inside an MPI envelope dictated by >> ***** some kinds of transcations. These transactions might, for >> ***** instance, motivate >> ***** 1) support of more than K contexts (where K is currently like 256) >> ***** 2) require dynamic contexts which are separate from group ID's >> ***** >> ***** Hence, by asking how the functions are to be done, I am asking >> ***** for possible justifications for the more general context model. >> ***** If hidden contexts are needed to do some of these things, >> ***** perhaps they could be justifiably be accomplished with an open >> ***** context mechanism. >> ***** >> ***** That was my motivation. - Tony >> ***** >> > >I'll think about these comments, but I suspect there is a language >barrier in operation here. > >Best Wishes >Lyndon > **** Lyndon, we know that these functions will have to be implemented some how. **** MPI itself might provide such mechanisms as published functionality or **** hidden functionality. If published functionality is denied in the **** standard (eg, general context management) because we cannot justify it **** properly, but must be implemented internally for most/all complying **** versions of MPI, then I think we lose out. My vague way of stating **** the requirement was to reveal the possibility of this, and thereby to **** promote such internal degrees of freedom into the public part of MPI, **** so applications can use them... Does that help???? **** - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 07:01:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23631; Wed, 10 Mar 93 07:01:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08660; Wed, 10 Mar 93 07:01:08 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 07:01:07 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08652; Wed, 10 Mar 93 07:00:58 -0500 Date: Wed, 10 Mar 93 12:00:52 GMT Message-Id: <2831.9303101200@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: [Tony Skjellum: re: static contexts writeup] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Tony writes: > **** PPS Lyndon, where are you. I got up at 4am to read the next round > **** of mail from you, but none to be found :-) > **** First off, many thanks to Tony for getting to his terminal at 4am. I really appreciate the effort to be UK compatible. I fell asleep at mine about 2am and then went home unable to stir mental activity. From 9am to 11am I was in meetings. I had a fair bit of mail to read, but more replies to my copious mail of last night yet expected. You should have got two messages beiefly describing some fairly basic inter-group communications kind of things that are important to me. I'll make as much time today as I can, but as I said last night, I have a hell of a lot of other things to get through as well today. I will come in additionally tomorrow morning from 9am to 10:30 am, to catch up on any further discussion developments. Then I really have to get on the train. > **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with Hey, I got about uncountable copies of Rik's note saved away, now another one :-) > **** Lyndon, you indicated that > **** one of my comments challenged you to make a new added entry > **** to one of the proposal categories. May I have that now/soon? challenged?, invited! Soon, Tony, soon. > **** Lyndon will be off-line, but > **** Lyndon will still get some brain work to do off-line. :-) :-) I can write up Proposal I, and Ib (Lyndon extension amendment) while away. > **** Lyndon > **** will you phone me periodically, to stay abreast of things? Okay, but this will be after 8pm my time, as the calls will come out of my pocket and appear on my parents 'phone bill, and its cheaper then, and they'll have to be real short. > **** Our PRECIS/SKETCH and work from it will not address the following things > **** (related to discussion in our group, but not central to it): > **** - the fully portable MPI program I vote we exclude from our discussion. > **** - I/O I vote we exclude from our discussion. > **** - the semantics and syntax of process management Perhaps something of this should be thought about. Quick thoughts/suggestions: * Create new process group (blob of processes :-) * Resize existing process group (shrink/grow blob, synchronously in blob) > **** - the time-table for MPI-1,MPI-2,MPI-3,... I vote we exclude from our discussion. > **** In a truly global sense, these could be construed to be closure as > **** well, but I want to exclude them for sanity reasons. I do believe > **** that parts of these issues could be addressed within MPI-1, but > **** it is not now this subcommittee's role to do that for MPI at-large. > **** Strong objections? If so, we are in deep trouble:-) No objections. > **** > **** We will assume that discussion in this 4-proposal X closure effort is > **** all essential to MPI-1. Anything you really don't think is essential > **** for MPI-1... once we agree to same .... should probably get deferred. > **** Is that controversial? Not controversial. > **** I thank everyone for their sincere efforts. - ditto - > **** PS Per constructive advice from Rik, all of my mail is getting copied to > **** mpi-context now. All should do so. I did not hear any dissent > **** about this suggestion, and it adds potential help to > **** discussion. As we need the help, and few people are writing > **** anyway, this will not likely cause problems. > **** Continue to include the five names jim, lyndon, tony, sanjay & > **** Rik, incase any of us are not on the mpi-context mailing list!!! Concur, being done. > > > **** Rik writes... > > >Before it gets TOO stale, here's a copy of my writeup on static > >contexts et.al. as it last went to Lyndon and Tony. (You two can > >stop reading at the line of dashes.) > > > >I have opted not to hack on static contexts any more because I > >would rather kill them. > **** Is this a proposal to include static contexts as in within > **** the 4-proposal model, and that we stop working on same? > **** If so, I can make it so... I'd like us to now ask Rik to write Proposal II. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 08:37:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28930; Wed, 10 Mar 93 08:37:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12056; Wed, 10 Mar 93 08:36:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 08:36:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12047; Wed, 10 Mar 93 08:36:43 -0500 Date: Wed, 10 Mar 93 13:36:37 GMT Message-Id: <2981.9303101336@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: [Tony Skjellum: re: static contexts writeup] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Proposal Ib Proposal Ib extends Proposal I (nee aka "snir" model) to provide group identification and inter-group communication for static (constant membership) groups. Group identification is effected by: facilities to send group in message; group registry service Inter-group communication needs to extend the basic (group,rank) notation. Two suggestions will be offered: procedures to do this; syntactic trickery (see previous emails). Regards Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 11:12:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03467; Wed, 10 Mar 93 11:12:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20567; Wed, 10 Mar 93 11:12:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 11:12:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20557; Wed, 10 Mar 93 11:12:11 -0500 Date: Wed, 10 Mar 93 16:12:01 GMT Message-Id: <3127.9303101612@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: private message To: Tony Skjellum In-Reply-To: Tony Skjellum's message of Wed, 10 Mar 93 10:00:55 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu No problem. This would be just fine. I perfer not to have to administrate it, especially as I will be off-line and therefore unable to service administration requests. Can I suggest that the address be precis@aurora.cs.mssatte.edu :-) /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 10 11:32:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03864; Wed, 10 Mar 93 11:32:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21943; Wed, 10 Mar 93 11:31:32 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 11:31:29 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21932; Wed, 10 Mar 93 11:31:27 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA10032; Wed, 10 Mar 93 10:25:42 CST Date: Wed, 10 Mar 93 10:25:42 CST From: Tony Skjellum Message-Id: <9303101625.AA10032@Aurora.CS.MsState.Edu> To: tony@aurora.cs.msstate.edu, jim@meiko.co.uk, d39135@sodium.pnl.gov, ranka@top.cis.syr.edu, lyndon@epcc.ed.ac.uk Subject: Re: [Tony Skjellum: re: static contexts writeup] Cc: mpi-context@cs.utk.edu **** My latest comments appear prepended with for ****'s. **** -Tony ----- Begin Included Message ----- From owner-mpi-context@CS.UTK.EDU Wed Mar 10 05:58:24 1993 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 07:01:07 EST Date: Wed, 10 Mar 93 12:00:52 GMT From: L J Clarke Subject: Re: [Tony Skjellum: re: static contexts writeup] To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Content-Length: 4438 Tony writes: > **** PPS Lyndon, where are you. I got up at 4am to read the next round > **** of mail from you, but none to be found :-) > **** [comments on interactions over next day omitted, :-)] **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with Hey, I got about uncountable copies of Rik's note saved away, now another one :-) > **** Lyndon, you indicated that > **** one of my comments challenged you to make a new added entry > **** to one of the proposal categories. May I have that now/soon? challenged?, invited! Soon, Tony, soon. > **** Lyndon will be off-line, but > **** Lyndon will still get some brain work to do off-line. :-) :-) I can write up Proposal I, and Ib (Lyndon extension amendment) while away. > **** Lyndon > **** will you phone me periodically, to stay abreast of things? Okay, but this will be after 8pm my time, as the calls will come out of my pocket and appear on my parents 'phone bill, and its cheaper then, and they'll have to be real short. ***** Fine, I will phone you, if you provide time windows and phone number(s). ***** I will address costs. > **** Our PRECIS/SKETCH and work from it will not address the following things > **** (related to discussion in our group, but not central to it): > **** - the fully portable MPI program I vote we exclude from our discussion. > **** - I/O I vote we exclude from our discussion. > **** - the semantics and syntax of process management Perhaps something of this should be thought about. Quick thoughts/suggestions: * Create new process group (blob of processes :-) * Resize existing process group (shrink/grow blob, synchronously in blob) **** OK, I will think about the proposal X closure matrix placement of this. > **** - the time-table for MPI-1,MPI-2,MPI-3,... I vote we exclude from our discussion. > **** In a truly global sense, these could be construed to be closure as > **** well, but I want to exclude them for sanity reasons. I do believe > **** that parts of these issues could be addressed within MPI-1, but > **** it is not now this subcommittee's role to do that for MPI at-large. > **** Strong objections? If so, we are in deep trouble:-) No objections. > **** > **** We will assume that discussion in this 4-proposal X closure effort is > **** all essential to MPI-1. Anything you really don't think is essential > **** for MPI-1... once we agree to same .... should probably get deferred. > **** Is that controversial? Not controversial. > **** I thank everyone for their sincere efforts. - ditto - > **** PS Per constructive advice from Rik, all of my mail is getting copied to > **** mpi-context now. All should do so. I did not hear any dissent > **** about this suggestion, and it adds potential help to > **** discussion. As we need the help, and few people are writing > **** anyway, this will not likely cause problems. > **** Continue to include the five names jim, lyndon, tony, sanjay & > **** Rik, incase any of us are not on the mpi-context mailing list!!! Concur, being done. > > > **** Rik writes... > > >Before it gets TOO stale, here's a copy of my writeup on static > >contexts et.al. as it last went to Lyndon and Tony. (You two can > >stop reading at the line of dashes.) > > > >I have opted not to hack on static contexts any more because I > >would rather kill them. > **** Is this a proposal to include static contexts as in within > **** the 4-proposal model, and that we stop working on same? > **** If so, I can make it so... I'd like us to now ask Rik to write Proposal II. **** Yes, Rik please do proposal II!!! Best Wishes Lyndon ----- End Included Message ----- **** Best wishes, **** Tony Summary of what went on above: We agreed that Lyndon would do I and Ib (his extension) We requested that Rik would do II (please agree/decline :-) I am still not sure if Rik did/plans to do a precis/sketch merger??? By implication, I am left with III/IV, expecting to get help from Jim, Sanjay, et al on it. I have promised to do the proposal X closure precis/sketch update. - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 12:19:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05037; Wed, 10 Mar 93 12:19:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24634; Wed, 10 Mar 93 12:19:24 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 12:19:22 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24626; Wed, 10 Mar 93 12:19:20 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 09:14 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA21499; Wed, 10 Mar 93 09:12:13 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20633; Wed, 10 Mar 93 09:12:10 PST Date: Wed, 10 Mar 93 09:12:10 PST From: d39135@sodium.pnl.gov Subject: Problem: group descriptors & coherency To: d39135@sodium.pnl.gov, tony@Aurora.CS.MsState.Edu Cc: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Message-Id: <9303101712.AA20633@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Here's a problem that applies to all proposals. Groups are dynamic. Group ID's must be reusable unless they are very long or we bound the number of groups that can ever get created. Every reference to a group ID must be interpreted in terms of the current definition. This introduces a coherency problem analogous to shared memory. There are at least a half-dozen solutions to the coherency problem, with wide differences in overhead costs and convenience for the user. Explicit coherency control by the application tends to be cheaper but less convenient than implicit control. ==> I suggest that each proposal include some mechanism for solving the group ID coherency problem, including a discussion of its probable costs. Aside: Coherency of tid's seems like a smaller problem, based on the assumption that processes will come and go much less often than groups will be re-formed. But I'm not sure that we can ignore it. --Rik From owner-mpi-context@CS.UTK.EDU Wed Mar 10 13:18:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06332; Wed, 10 Mar 93 13:18:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28066; Wed, 10 Mar 93 13:18:05 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 13:18:04 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28057; Wed, 10 Mar 93 13:18:02 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:10 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA22798; Wed, 10 Mar 93 10:08:19 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20772; Wed, 10 Mar 93 10:08:13 PST Date: Wed, 10 Mar 93 10:08:13 PST From: d39135@sodium.pnl.gov Subject: implementation assumptions To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303101808.AA20772@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-comm@cs.utk.edu In a discussion we are having within the contexts subcommittee, Tony Skjellum commented that: > ... I want to make sure that we do > not provide illusory improvements to the standard, while letting other > parts of it exceed ours in the unscalability department, or let > implementations qualify as compliant, whose unscalability disables the > implied scalability of our syntax/semantics... I agree. This is why (I think) proposals should explicitly address performance. If a proposal assumes that the cost of a particular function is irrelevant, it should say that. If a proposal assumes that a particular function is going to be cheap, it should justify that assumption by naming a time/memory price and outlining an implementation that can achieve it. Sometimes the suggested implementation will reveal further assumptions that have wide implications. Ideally, each proposal will eventually evolve a set of bottom-line assumptions that are consistent and believable. These comments are particularly addressed to those of us beating on contexts and groups, where there seem to be lots of subtle interactions. But I am cross-posting them to the whole committee as food for thought... --Rik ("The Skeptic") Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 13:29:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06699; Wed, 10 Mar 93 13:29:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28812; Wed, 10 Mar 93 13:28:44 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 13:28:43 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28804; Wed, 10 Mar 93 13:28:41 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:21 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23065; Wed, 10 Mar 93 10:19:47 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20875; Wed, 10 Mar 93 10:19:42 PST Date: Wed, 10 Mar 93 10:19:42 PST From: d39135@sodium.pnl.gov Subject: Re: descriptors vs ID's To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303101819.AA20875@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu (This is a resend. Apparently several addresses got fouled up yesterday.) Rik said: > But I argue that the global ID's should NOT be the things that get > passed to the collective comm routines, and possibly not even to the > pt-pt routines. > > Passing a global ID implies some kind of table searching, unless we > put tight bounds on the value ranges so that we can do a simple > indirection. Of course, I suppose I could equally well argue that the cost of such a lookup would be small compared to the overall cost of the collective comm routine. Maybe this is a don't-care. --Rik (Good grief, Lyndon -- replying to one's own mail seems to be catching!) From owner-mpi-context@CS.UTK.EDU Wed Mar 10 13:59:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07404; Wed, 10 Mar 93 13:59:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00653; Wed, 10 Mar 93 13:58:49 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 13:58:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00645; Wed, 10 Mar 93 13:58:45 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:49 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23233; Wed, 10 Mar 93 10:47:14 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21230; Wed, 10 Mar 93 10:47:12 PST Date: Wed, 10 Mar 93 10:47:12 PST From: d39135@sodium.pnl.gov Subject: Re: descriptors vs ID's To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Message-Id: <9303101847.AA21230@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu (This is a resend with modifications [flagged with **]. The original, yesterday, had fouled up addresses.) Jim raises the issue: > Are the tid's of local or global scope. i.e. Can I pass a tid to > someone else, and it still work as an address for the person I > passed it to ? > For global scope : It's simpler for the user > Against : The implementor might want the "opaque identifier" > to be a pointer to a big data structure containing > route tables or something. The same problem arises with groups and I'm not sure where else. I agree that we need global scope ID's for tids and probably for groups. But I argue that the global ID's should NOT be the things that get passed to the collective comm routines, and possibly not even to the pt-pt routines. Passing a global ID implies some kind of table searching, unless we put tight bounds on the value ranges so that we can do a simple indirection. For tid's, that's OK because there are only as many tid's as there are processes, and I'm willing to believe that's a small bound. But in theory there can be lots more groups than processes, and even in practice I'm not sure that we can keep the global ID's both unique and small without excessive overhead. (E.g., recall my trick of generating unique context values without communication by cramming the creator's process number into the bottom bits.) I propose that most routines be passed a local descriptor reference, rather than the global ID value. ** The above may be don't-care. For collective comms, the cost ** of finding a (local) group descriptor, given the group ID, ** is arguably small compared to the cost of the comm itself. ** For tid's, perhaps the tid values can be kept small enough to ** permit simple indirection. Then we need either 1. routines to retrieve the global ID from a local descriptor (easy) and to construct a local descriptor from the global ID (harder) or 2. a mechanism to translate local descriptors to and from some machine-independent form. I prefer the second option because of my bias against servers. ** In another message, I raised the problem of maintaining ** coherence of group descriptors if the group ID's can be reused. ** The comments above also relate to methods of explicitly ** restoring coherency. Scheme 1 would be used if a process was ** told only "group XX has been redefined, please invalidate your ** descriptor"; scheme 2 would be used if a process was told ** "group XX has been redefined and here's the new definition". --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 14:20:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08095; Wed, 10 Mar 93 14:20:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01852; Wed, 10 Mar 93 14:19:58 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 14:19:57 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01844; Wed, 10 Mar 93 14:19:56 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 11:14 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23523; Wed, 10 Mar 93 11:13:08 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21416; Wed, 10 Mar 93 11:13:06 PST Date: Wed, 10 Mar 93 11:13:06 PST From: d39135@sodium.pnl.gov Subject: Re: [d39135@sodium.pnl.gov: Re: group descriptors] To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: mpi-context@cs.utk.edu Message-Id: <9303101913.AA21416@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu > So, my basic request to Rik is that he echo the two letters sent to him > alone, which was my error. Mea culpa. I think there was only one, and I included the whole thing in my note referencing it, so it has been echoed already. No problem. --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 14:21:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08148; Wed, 10 Mar 93 14:21:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01996; Wed, 10 Mar 93 14:21:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 14:21:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01988; Wed, 10 Mar 93 14:21:16 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 11:10 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23493; Wed, 10 Mar 93 11:08:57 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21333; Wed, 10 Mar 93 11:08:53 PST Date: Wed, 10 Mar 93 11:08:53 PST From: d39135@sodium.pnl.gov Subject: Re: group descriptors To: d39135@sodium.pnl.gov, tony@Aurora.CS.MsState.Edu Cc: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Message-Id: <9303101908.AA21333@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony writes > Anyway, I concur on the need to have registerable choices, that allow > sharing, per your letter. As regards new functionality, they could go > in the tree too, but there is no structural mechanism in Zipcode, > other than defining a class' method list, for adding new instantiations > to a mailer at creation. All of this was very elaborate coding, which > I will elucidate in my article for Rolf. I am happy to share details > with you as well, beforehand. Maybe I haven't thought about this enough, but I do not understand why elaborate coding is required if we do everything at runtime and do not try to provide type-safety. No one has yet pointed out a deficiency with my proposed interface: MPI_set_group_attribute (gid,key,value,destructor_routine) and MPI_test_group_attribute (gid,key,&value) This can be implemented at a data cost of adding a single slot to a group descriptor (pointing to the attribute dictionary), plus a loop in the group closing routine to call the destructors (if any), plus the routines to manage the dictionary. I do not believe that there would be much resistance from the committee to adding this capability, since it seems to be easy to implement and (almost) "free if you don't use it". What have I missed? --Rik Littlefield PS. OK, since nobody else has pointed out a deficiency, I will. The definition provided above does not support transferring attributes between processes. The uses that I had in mind involve information that is so local to the process that transferring it doesn't even make sense (e.g., lists of partners for collective comms). If one wanted to define attributes that could be transferred, the interface would have to be extended. From owner-mpi-context@CS.UTK.EDU Wed Mar 10 14:51:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08797; Wed, 10 Mar 93 14:51:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03683; Wed, 10 Mar 93 14:51:06 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 14:51:04 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03673; Wed, 10 Mar 93 14:51:02 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 11:48 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24208; Wed, 10 Mar 93 11:46:25 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21672; Wed, 10 Mar 93 11:46:20 PST Date: Wed, 10 Mar 93 11:46:20 PST From: d39135@sodium.pnl.gov Subject: re: static contexts writeup To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303101946.AA21672@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony, You write > **** In mail immemorial, Rik agreed to merge his (enclosed) write-up with > **** the first-generation PRECIS, to generate a more useful one, > **** Please correct me if I am wrong, but I cannot find this! I think I'm clean. The exchange was > > > (Tony writes) > > > So, my first action item is to get a merger of what you wrote and our > > > precis, and redistribute that to group. OK? > > > > (Rik replies) > > I'm not sure that I've seen your precis yet. The stuff you've sent me > > so far seems to be quite a ways from precise (pardon the pun), so I'm > > reluctant to guess what's what. Can you send me a labeled copy? > > > > It would be faster and perhaps less confusing if I incorporated the > > last round of your and Lyndon's comments into my words and distributed > > that to the subcommittee as a separate document. Then we would have > > a collective opinion (well, five separate ones anyway) of how the > > two documents fit together, instead of just the one opinion of whoever > > did the merger. > > (Tony replies) > Yes, I concur. Go ahead. Several copies of my document were subsequently distributed, including ones by me and by you! As I noted when I sent it out, I did not incorporate your and Lyndon's comments because I decided my time might better go into exploring cacheing mechanisms. By the way, I'm sorry, but I'm still not sure that I have correctly identified your precis or sketch or whatever. You sent me some stuff at a level of detail like > Proposal II. GROUP ID != CONTEXT > > GROUP corresponds to "data/instance" in this model > CONTEXT is statically defined at link time, corresponding to > code or class > > In this model, group ID's appear explicitly, and separately > from CONTEXTS in global operations. It was noted by Littlefield > that he would write global operations, and product bits of the > GROUP ID and static CONTEXT to guarantee that his code produced > unique, deterministic, non-interfering results. > > Proposal III. GROUP ID null, GROUP scope == CONTEXT scope > > CONTEXT is a dynamic concept, and global (or as global as necessary) > Whenever a group is created, a context is assigned to it. CONTEXTS > are used to disambiguate global operations. > > GROUP ID is the CONTEXT for the group. > > GROUP is an enumeration of PROCESS ID's OR ONE OF SEVERAL > possible hashing formulas, with exceptions (more scalable) with a bunch of comments interspersed with it. Is this what you are talking about? In other mail, you have mentioned a "sparse matrix of proposals X closure options", but I haven't the foggiest idea what that refers to. Help? --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 15:13:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09008; Wed, 10 Mar 93 15:13:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06326; Wed, 10 Mar 93 15:12:24 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 15:12:22 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06312; Wed, 10 Mar 93 15:12:19 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 12:09 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24295; Wed, 10 Mar 93 12:07:30 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21860; Wed, 10 Mar 93 12:07:27 PST Date: Wed, 10 Mar 93 12:07:27 PST From: d39135@sodium.pnl.gov Subject: writing assignments To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Cc: mpi-context@cs.utk.edu Message-Id: <9303102007.AA21860@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu > I'd like us to now ask Rik to write Proposal II. > > Best Wishes > Lyndon I will be happy to write "Proposal II", but I'm not sure that we agree on what that means. I would definitely like to include a proposal for a cacheing facility, but this seems orthogonal to all other issues about groups, contexts, tid/pid format etc. I will also discuss the idea of static contexts, but will specifically not propose them, in favor of cacheing, if I can convince myself that cacheing combined with other capabilities meets all my requirements. (Cacheing is clearly superior in some ways, but I'm not sure that it covers all the bases.) --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 17:06:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12404; Wed, 10 Mar 93 17:06:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13359; Wed, 10 Mar 93 17:05:52 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 17:05:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13347; Wed, 10 Mar 93 17:05:49 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13032; Wed, 10 Mar 93 16:00:56 CST Date: Wed, 10 Mar 93 16:00:56 CST From: Tony Skjellum Message-Id: <9303102200.AA13032@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Subject: fixing misunderstanding about nomenclature In your last letter, Rik, you clearly identify what I have called the PRECIS/SKETCH. I have been talking loosely about a set of proposals (so marked), and Lyndon has understood these all along too. They are called I...IV, with some sub-proposals also. Then, for each of these, there are certain options on what things like tags do, and so on, so we imagine a (hopefully sparse) matrix of alternatives to be looked at each of which is a valid closure of MPI. Of the ones that make sense (eg, Proposal II with tags being blah, and TIDS being blah', and the following other side conditions defined, we have one possible closure of MPI. The sub-committee assigns a rank to this choice; we will hopefully have more than one. We vote on all of them, in rank order. If we disagree violently on rank, we present all of them in a coherent order, and vote on them in a reasonable order.) I want to move this SKETCH to have more precise information in it. I will look at doing same. Hopefully, you are doing proposal II, per Lyndon's suggestion, and the place to find what proposal II is is still in that SKETCH. - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 17:08:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12413; Wed, 10 Mar 93 17:08:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13470; Wed, 10 Mar 93 17:07:55 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 17:07:54 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13455; Wed, 10 Mar 93 17:07:53 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13048; Wed, 10 Mar 93 16:03:26 CST Date: Wed, 10 Mar 93 16:03:26 CST From: Tony Skjellum Message-Id: <9303102203.AA13048@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, ranka@top.cis.syr.edu Subject: Re: writing assignments Cc: mpi-context@cs.utk.edu Rik writes... >From d39135@sodium.pnl.gov Wed Mar 10 14:08:49 1993 >Received: from pnlg.pnl.gov by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); > id AA11263; Wed, 10 Mar 93 14:08:46 CST >Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 > 12:09 PST >Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24295; Wed, > 10 Mar 93 12:07:30 PST >Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21860; Wed, 10 Mar 93 12:07:27 > PST >Date: Wed, 10 Mar 93 12:07:27 PST >From: d39135@sodium.pnl.gov >Subject: writing assignments >To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, > ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu >Cc: mpi-context@cs.utk.edu >Message-Id: <9303102007.AA21860@sodium.pnl.gov> >X-Envelope-To: tony@aurora.cs.msstate.edu >Status: R > >> I'd like us to now ask Rik to write Proposal II. >> >> Best Wishes >> Lyndon > >I will be happy to write "Proposal II", but I'm not sure that >we agree on what that means. > >I would definitely like to include a proposal for a cacheing >facility, but this seems orthogonal to all other issues about >groups, contexts, tid/pid format etc. > >I will also discuss the idea of static contexts, but will >specifically not propose them, in favor of cacheing, if I can >convince myself that cacheing combined with other capabilities >meets all my requirements. (Cacheing is clearly superior in some >ways, but I'm not sure that it covers all the bases.) > >--Rik Littlefield > OK. The cacheing thing seems to add another item to our list of proposal things. Go ahead... think of it as another column in our sparse matrix, for which caching is the same for each major Proposal... - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 10 18:48:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13684; Wed, 10 Mar 93 18:48:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17757; Wed, 10 Mar 93 18:47:50 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 18:47:43 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17747; Wed, 10 Mar 93 18:47:40 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 15:45 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24514; Wed, 10 Mar 93 15:43:28 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA23252; Wed, 10 Mar 93 15:43:24 PST Date: Wed, 10 Mar 93 15:43:24 PST From: d39135@sodium.pnl.gov Subject: Re: fixing misunderstanding about nomenclature To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303102343.AA23252@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu > ... Of the ones that make sense > (eg, Proposal II with tags being blah, and TIDS being blah', and the > following other side conditions defined, we have one possible closure > of MPI. The sub-committee assigns a rank to this choice; we will hopefully > have more than one. We vote on all of them, in rank order. If we > disagree violently on rank, we present all of them in a coherent order, > and vote on them in a reasonable order.) > > I want to move this SKETCH to have more precise information in it. I will > look at doing same. Hopefully, you are doing proposal II, per Lyndon's > suggestion, and the place to find what proposal II is is still in that > SKETCH. > > - Tony > I assume that the goal is to get several proposals that are self-consistent but different. Since Proposal II was at one point represented as "Littlefield's model", I will try to write my personal favorite proposal into bullet-item form. I don't know close it will be to your current Proposal II, but it'll probably be different from anybody else's. I will try to get you this by tomorrow (Thursday). Once we have several different bullet-item proposals in hand, what is the best way to proceed? My inclination is to have several rounds of critique, with the author of each proposal responding to each criticism by one or more of: . modifying or extending the proposal while retaining its character . modifying or extending the statement of assumptions . modifying or specifying an implemention strategy to overcome the criticism . acknowledging a deficiency. Hopefully there will not be too much convergence between proposals. --Rik Littlefield From owner-mpi-context@CS.UTK.EDU Wed Mar 10 23:22:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17588; Wed, 10 Mar 93 23:22:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27668; Wed, 10 Mar 93 23:21:36 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 10 Mar 1993 23:21:34 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27659; Wed, 10 Mar 93 23:21:33 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13324; Wed, 10 Mar 93 22:17:22 CST Date: Wed, 10 Mar 93 22:17:22 CST From: Tony Skjellum Message-Id: <9303110417.AA13324@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Subject: Re: fixing misunderstanding about nomenclature Rik writes... (my responses prepended with four *'s - Tony)... > ... Of the ones that make sense > (eg, Proposal II with tags being blah, and TIDS being blah', and the > following other side conditions defined, we have one possible closure > of MPI. The sub-committee assigns a rank to this choice; we will hopefully > have more than one. We vote on all of them, in rank order. If we > disagree violently on rank, we present all of them in a coherent order, > and vote on them in a reasonable order.) > > I want to move this SKETCH to have more precise information in it. I will > look at doing same. Hopefully, you are doing proposal II, per Lyndon's > suggestion, and the place to find what proposal II is is still in that > SKETCH. > > - Tony > I assume that the goal is to get several proposals that are self-consistent but different. **** Yes. Since Proposal II was at one point represented as "Littlefield's model", I will try to write my personal favorite proposal into bullet-item form. I don't know close it will be to your current Proposal II, but it'll probably be different from anybody else's. I will try to get you this by tomorrow (Thursday). **** Thank you. **** Rik suggests the following strategy for handling proposal review. **** I would like to use it... **** Once we have several different bullet-item proposals in hand, what is the best way to proceed? My inclination is to have several rounds of critique, with the author of each proposal responding to each criticism by one or more of: . modifying or extending the proposal while retaining its character . modifying or extending the statement of assumptions . modifying or specifying an implemention strategy to overcome the criticism . acknowledging a deficiency. **** Fine, and we will use this as the criterion for all proposals. **** It is a sound strategy. **** **** Thanks on both counts. - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 11 19:08:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11521; Thu, 11 Mar 93 19:08:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25952; Thu, 11 Mar 93 19:08:20 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 11 Mar 1993 19:08:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25936; Thu, 11 Mar 93 19:08:14 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 11 Mar 93 15:59 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA26618; Thu, 11 Mar 93 15:58:00 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA25111; Thu, 11 Mar 93 15:57:57 PST Date: Thu, 11 Mar 93 15:57:57 PST From: d39135@sodium.pnl.gov Subject: contexts Proposal V To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: gropp@mcs.anl.gov, lusk@mcs.anl.gov Message-Id: <9303112357.AA25111@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony, OK, here's a detailed proposal regarding groups and contexts. It's much different from the earlier sketch called "Proposal II", so I renamed it. This proposal seems to meet my needs. Barring any new difficulties, I will be happy to drop static contexts. The proposal is long but hierarchical. I think there's enough in the summary to characterise it. I am also sending this description to Bill Gropp and Rusty Lusk. I hope they will wring it out, particularly my assertion that this proposal can be implemented as a layer on top of MPI pt-pt. --Rik Littlefield ---------------------------------------------------------------------- PROPOSAL V. Summary: . Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) . Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) . There are restrictions that permit groups to be layered on top of pt-pt. . Pt-pt communications use only TID, context, and tag, and are specified to be fast. . Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. Detailed Proposal: . Pt-pt uses only "TID", "context", and "message tag". TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. . Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) . A group is represented by a "group descriptor", of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a "process-local group ID", which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) . Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). . Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). . Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. . At creation, each group is assigned a globally unique "default group context" which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). . The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the "paired-exact-match constraint". This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) . A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] . Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) . When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. . All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). . Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] . All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's "root process", the number of processes in the group, and an integer "group tag" provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] . Collective communication routines are called by all members of a group in the same order. . Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. . Non-blocking collective communication routines are passed a reference to the group descriptor, plus a "data tag" to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. Advantages: . This proposal is implementable with no servers and can be layered easily on existing systems. . Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) . Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. . Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. . Communication both within and between groups seems conceptually straightforward. Disadvantages: . Requires explicit translation between (group,rank) and TID, which may be considered awkward. . Communication between different groups may be considered awkward. . No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). . Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. . Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. . No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. Comments / Alternatives: . The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. . Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. . The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like "a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context". This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] . Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. Implementation Notes: 1. To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) 2. The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a "group partitioning context" and a "group disbanding context" that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. Examples: (To be provided) From owner-mpi-context@CS.UTK.EDU Sun Mar 14 13:58:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05196; Sun, 14 Mar 93 13:58:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16166; Sun, 14 Mar 93 13:58:14 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 14 Mar 1993 13:58:10 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16158; Sun, 14 Mar 93 13:58:04 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 14 Mar 93 10:57 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA29374; Sun, 14 Mar 93 10:55:26 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA05149; Sun, 14 Mar 93 10:55:22 PST Date: Sun, 14 Mar 93 10:55:22 PST From: rj_littlefield@pnlg.pnl.gov Subject: proposal to mpi-collcomm To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, gropp@mcs.anl.gov, jim@meiko.co.uk, lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303141855.AA05149@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Al & Tony, et.al.: I am about to send to mpi-collcomm, two notes regarding changes I propose to the collective communication specification. (One note summarizes the changes; the other discusses the reasons for them.) I am also sending these notes to mpi-context and friends because they relate to other discussions going on there. Thought you'd like to know... --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Sun Mar 14 15:04:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06324; Sun, 14 Mar 93 15:04:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18133; Sun, 14 Mar 93 15:03:51 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 14 Mar 1993 15:03:49 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18125; Sun, 14 Mar 93 15:03:46 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 14 Mar 93 12:01 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA29382; Sun, 14 Mar 93 11:59:17 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA05208; Sun, 14 Mar 93 11:59:13 PST Date: Sun, 14 Mar 93 11:59:13 PST From: rj_littlefield@pnlg.pnl.gov Subject: collcomm changes, summary To: geist@gstws.epm.ornl.gov, gropp@mcs.anl.gov, jim@meiko.co.uk, lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303141959.AA05208@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu SUMMARY OF SUGGESTED CHANGES TO COLLECTIVE COMMUNICATION PROPOSAL The draft proposal that Al Geist distributed several days ago contains some features that would prevent it from being implemented as a layer on top of MPI point-to-point facilities. The purpose of this note is to propose changes to the group control routines in order to permit layering, and to propose other changes for better and more predictable performance. A discussion of the rationale for these proposed changes will be distributed separately because of its length. The main changes introduced in this note are: . The concept of group identification is firmed up. Most operations use a "group handle" that is local to the process. (Think of the group handle as being just the address of a potentially large and complex "group descriptor".) There is still a "group ID" that is globally unique, but it has only a secondary role and can be ignored by most applications. The "group name" is entirely removed from MPI-1. (Group names are still anticipated in MPI-2, but upward-compatibility is maintained in a different way from the draft proposal.) . A semantic restriction is introduced, that a process can access information about a group only if the process holds a group handle for it. Group handles can be obtained in two ways: 1) they are produced by group formation routines, and 2) a process can explicitly distribute copies of its group handles to other processes, using new routines introduced specifically for that purpose. . A cacheing mechanism is introduced, that allows modules to attach arbitrary information to a group descriptor in such a way that it can be quickly retrieved. Cacheing facilitates the construction of collective communication routines that are "fast after the first execution in a group", no matter how the other group operations are implemented. . A new group formation routine is introduced, that is less synchronous and more general than MPI_PARTGROUP. Specifically, the following routines are proposed to be added or modified: 1. Arbitrary group formation: newgrp_handle = MPI_FORMGROUP (grouptag,groupsize,knownmembers) where grouptag is a user-provided integer tag, sufficiently unique to disambiguate overlapping groups that might be formed simultaneously (say by multiple threads). groupsize is the number of members that will compose the group. knownmembers is a set of pid's of some or all members of the group. Each member of the group must provide the same set of knownmembers. newgrp_handle is a group handle for the newly formed group This new routine must be called synchronously, but only by those processes forming the group. 2. Group partitioning: newgrp_handle = MPI_PARTGROUP (oldgrp_handle,grouptag) where the semantics are the same as the draft proposal except that the return value is now a new group handle instead of a rank. (The rank can be determined by a separate call to MPI_GETRANK(group_handle,pid) .) 3. Group disbanding: MPI_LVGROUP (group_handle) where the semantics are the same as the draft proposal except that MPI_LVGROUP now does not return any result. (Since groups can now be formed arbitrarily, not just by partitioning, it is not obvious what MPI_LVGROUP could return in general.) This routine can be called only by members of the group. 4. Distribution of group handles and disposition of distributed handles: MPI_SendGroupHandle (pid,context,tag,old_group_handle) new_group_handle = MPI_RecvGroupHandle (pid,context,tag) MPI_FreeGroupHandle (group_handle) (The latter routine is similar to MPI_LVGROUP except that it can be called only for distributed group handles. This is solely for semantic clarity; a single interface routine would do.) 5. Cacheing group-specific process-local information: The following routines get and free keys for use with group cacheing. key = MPI_GetAttributeKey () MPI_FreeAttributeKey () The following routines cache and retrieve information. MPI_SetGroupAttribute (grouphandle,key,value,destructor_routine) status = MPI_TestGroupAttribute (grouphandle,key,&value) where key must be unique within the group value is anything the size of a pointer destructor_routine is an application-provided routine that is called by MPI_LVGROUP, with arguments being the group handle, cached key and value. Cached information is stripped from the new group handle returned by MPI_SendGroupHandle. In a conforming implementation, MPI_TestGroupAttribute must be no slower than a point-to-point communication call. 6. Retrieving global group ID: global_id = MPI_GetGlobalGroupID (grouphandle) 7. Other collective communications: Consistently substitute "grouphandle" in place of "group". ---------------------------------------------------------------------- rj_littlefield@pnl.gov Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Sun Mar 14 15:50:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06915; Sun, 14 Mar 93 15:50:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19420; Sun, 14 Mar 93 15:49:43 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 14 Mar 1993 15:49:41 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19411; Sun, 14 Mar 93 15:49:35 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 14 Mar 93 12:48 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA29389; Sun, 14 Mar 93 12:46:59 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA05301; Sun, 14 Mar 93 12:46:57 PST Date: Sun, 14 Mar 93 12:46:57 PST From: rj_littlefield@pnlg.pnl.gov Subject: collcomm changes, rationale To: geist@gstws.epm.ornl.gov, gropp@mcs.anl.gov, jim@meiko.co.uk, lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303142046.AA05301@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu RATIONALE FOR SUGGESTED CHANGES TO COLLECTIVE COMMUNICATION PROPOSAL In a related summary, I outlined a set of suggested changes to the concepts and routines in the collective communication proposal. The purpose of this note is to present the rationale for those suggestions and to discuss possible alternatives. The discussion is organized into 5 areas, flagged with "----- Topic #". Entries flagged with > are from my summary of suggested changes. Entries flagged with >>> are from the draft proposal sent out by Al Geist. ----- Topic #1: Group Identification ----- > . The concept of group identification has been firmed up. Most > operations use a "group handle" that is local to the process. > (Think of the group handle as being just the address of a > potentially large and complex "group descriptor".) > ... > . A semantic restriction is introduced, that a process can access > information about a group only if the process holds a group > handle for it. Group handles can be obtained in two ways: 1) > they are produced by group formation routines, and 2) a process > can explicitly distribute copies of its group handles to other > processes, using new routines introduced specifically for that > purpose. There are two issues here: one of being able to layer collective communications on top of point-to-point at all, and a secondary one of efficiency. The more fundamental issue is layering. Given only MPI point- to-point functionality, how can a group identifier (whatever it is) be transmitted between processes so as to be useful to the receiver? Presumably we want to allow group identifiers to be passed around so that any process holding the group identifier can use it for purposes like translating between (group,rank) and pid. We also want to allow this translation to be done asynchronously, i.e., without requiring the explicit cooperation of any other MPI process at the time of translation. Since MPI pt-pt does not support asynchronous servers or an interrupt receive capability, this implies that the group identifier must come complete with enough information to resolve all translations without communication. This prompts the concept that the group identifier must be associated with a "group descriptor" that is large and complex enough to fully describe the group. How is the association done? This is a question of efficiency. If the identifier is allowed to be process-local, the group descriptor can be located very quickly -- just make the identifier be a pointer to the group descriptor. Requiring the identifier to have global scope would not be so good. In that case, either the identifier has to be carefully constructed or the association has to be done with some sort of table search. These issues also arise with global pid's. However, groups can be formed much more often and in greater numbers than processes. I doubt that careful construction tricks could be assured to be adequate, and if not, then a table search would be required on each collective communication call. The conclusion is that, for most purposes, a process-local identifier generated by the system is preferred. Such things are typically called "handles", hence the term "group handle". > 4. Distribution of group handles and disposition of distributed handles: > > MPI_SendGroupHandle (pid,context,tag,old_group_handle) > > new_group_handle = MPI_RecvGroupHandle (pid,context,tag) > > MPI_FreeGroupHandle (group_handle) The next question is how group handles should be distributed. Implicit distribution is out because MPI pt-pt doesn't support a server capability, and presumably we aren't willing to synchronize all of the processes whenever somebody creates a group handle. So, explicit distribution is required. How do we handle it? Two ideas that I do not like are the following. MPI might provide routines to translate to and from some machine- and process- independent format, so that the translated information could be sent using normal point-point primitives. This strategy requires that the user program manage the storage of indefinite-length objects, which makes for an ugly Fortran interface. Or, group descriptors (and their translation routines) might be built into point-point MPI as another data type. This violates the spirit of layering collective communication on point-to-point, and has the same storage management problem. The three routines proposed above were the cleanest interface I could think of. ----- Topic #2: Global Group ID ----- > There is > still a "group ID" that is globally unique, but it has only a > secondary role and can be ignored by most applications. > ... > global_id = MPI_GetGlobalGroupID (grouphandle) Given that we are now able (and required) to pass around copies of group handles, it is not clear to me that MPI really needs special support for the concept of a global group ID. On the other hand, it's easy to provide, since we have to construct one or more globally unique context values for each group anyway. So just use the first such context value as the global ID. This gives something unique that all processes can agree on. But note that knowing just the global group ID does not let you get other information about the group -- you have to hold a group handle for that. (We could add a routine that would accept the global group ID and return a handle for that group, presuming that the process held one. This would be cheap to do, since group handles are managed by MPI anyway, and I can vaguely imagine that it might help some applications. On the other hand, there are no similar "handle lookup" facilities provided elsewhere in MPI, and I'm reluctant to set that kind of precedent without clear need.) ----- Topic #3: Group Formation ----- > . A new group formation routine is introduced, that is less > synchronous and more general than MPI_PARTGROUP. > ... > 1. Arbitrary group formation: > > newgrp_handle = MPI_FORMGROUP (grouptag,groupsize,knownmembers) > > where > grouptag is a user-provided integer tag, sufficiently unique > to disambiguate overlapping groups that might be > formed simultaneously, say by multiple threads. > > groupsize is the number of members that will compose the group. > > knownmembers is a set of pid's of some or all members of the group. > Each member of the group must provide the same > set of knownmembers. > > newgrp_handle is a group handle for the newly formed group > > This new routine must be called synchronously, but only by those > processes forming the group. The draft proposal distributed by Al Geist says that >>> A group is identified by a group name that is supplied by the user. A group name by itself is not enough to allow implementing groups as a layer on top of point-to-point, unless we impose restrictions that I think would be not acceptable. The problem is: how does a group-forming routine know whom it should send messages to, in order to form the group? MPI_PARTGROUP does not have a problem with this, because it has to be called synchronously by all members of the group. Since each current member of the group holds a handle (descriptor) for that group, it is easy for each member to figure out who talks to whom. Unfortunately, there are some important application designs that I do not see how to implement with just MPI_PARTGROUP. For example, I am now doing an application that uses a master-slaves strategy to asynchronously parcel out chunks of work, with each chunk being done by several processes working collaboratively. Collective communication between those processes is required, so it seems natural to organize them into MPI groups. Using a synchronous group partitioning routine would introduce a risk of load imbalance, because the varying chunk size implies that groups can finish their work at different times, and synchronous partitioning would delay their reassignment. Applications like this could benefit from a group formation routine that is called synchronously, but only by those processes forming the group -- hence MPI_FORMGROUP. This type of routine does have the problem of identifying its collaborators, and the only solution I can think of is to tell it. That's what the knownmembers argument is for. I have specified knownmembers in terms of pid's because I assume that point-to-point communication based on pid's is always fast and unrestricted. If knownmembers were based on (group,rank) pairs, then per the discussion above, all processes making this call would have to hold handles (descriptors) for the referenced groups. This seems to me to be more trouble than it's worth, but others may disagree. Another comment about efficiency... The size of the knownmembers set affects the efficiency of group formation. At one extreme, only one member is required to be known. This is scalable in a memory sense, but not in a time sense, because it implies O(P) group formation time for a group of P processes. At the other extreme, all members can be specified. This is not scalable in a memory sense, but allows guaranteed O(log P) formation time. Other tradeoffs are possible, such as O(sqrt P) knownmembers and O(sqrt P) formation time. The interface as specified allows each application to choose the type of scalability it wants. ----- Topic #4: Group Names ----- > ... The > "group name" is entirely removed from MPI-1. (Group names are > still anticipated in MPI-2, but upward-compatibility is > maintained in a different way from the draft proposal.) The draft distributed by Al Geist states: >>> To allow for future extensibility of the group concept >>> the present draft specifies that groups be named. Requiring names has the drawback that 1) it burdens the user with at least the appearance of having to create unique names, in order to be upward-compatible with dynamic groups, even though 2) in a layered MPI-1, there is no way in general to check global uniqueness, and thus programs can work fine with non-unique names. This combination strikes me as actually impeding upward- compatibility. The tendency will be for programmers to use non-unique names because it works and it's easy. But such programs would break when MPI-2 came along and started actually using the names for something. I don't like encouraging people to write programs that are going to break. I do support upward compatibility. However, rather than requiring names in MPI-1, I propose that they be deferred entirely to MPI-2, at which point they can be supported either just through MPI_JOINGROUP (as an alternative to MPI_FORMGROUP) or via additional routines to attach globally unique names to groups that have already been formed via MPI_JOINGROUP. ----- Topic #5: Cacheing ----- > 5. Cacheing group-specific process-local information: > > The following routines get and free keys for use with group > cacheing. > > key = MPI_GetAttributeKey () > MPI_FreeAttributeKey () > > The following routines cache and retrieve information. > > MPI_SetGroupAttribute (grouphandle,key,value,destructor_routine) > MPI_TestGroupAttribute (grouphandle,key,&value) > > where > key must be unique within the group > value is anything the size of a pointer > destructor_routine is an application-provided routine that > is called by MPI_LVGROUP, with arguments > being the group handle, cached key and value. > > Cached information is stripped from the new group handle > returned by MPI_SendGroupHandle. > > In a conforming implementation, MPI_TestGroupAttribute must > be no slower than a point-to-point communication call. This feature is purely for efficiency, but I think it's so valuable, cheap, and clean that something like it has to go in. One feature of collective communication is that the fastest algorithm for any particular job usually depends on the machine topology, which processes belong to the group, and the amount of data being manipulated. For example, global combine of L data elements across P = RC processes on a 2-D RxC mesh can be done in O(L log(P)) time using a fanin/fanout algorithm, or in O(L + sqrt(P)) time using a nested rings algorithm. The former is better for small L, the latter for big L, and using the wrong one can easily cost a factor of 3 in execution time. So, there is strong motivation to write collective communication routines that are adaptive in the sense of figuring out which algorithm is best. The problem is that it can take quite a lot of time to make the decision, starting from a scratch position of not even knowing which processes belong to the group. It's going to take lots of calls to the inquiry routines to get that information, and then some more cycles to make the proper decisions. Obviously it would be profitable to cache the information and/or decisions. The question is, where? It is tempting to say that the collective communication routine could or should keep its own cache, indexed by group handle and/or global group ID. The problem is, groups are dynamic in the sense of being formed and disbanded, so that unless group IDs can get very large, eventually they will have to be reused. Now, it wouldn't do to have a collective communication routine use stale cached information, so if the collective communication routine is keeping its own cache, then it needs to be notified of the reuse so that it can release the cached stuff. Alternatively, perhaps the cached information could be automatically released. (Either strategy guarantees immediate release of cached info when the group handle/descriptor is released. I presume we want to do that, to avoid getting into the morass of garbage collection.) The method proposed here can be thought of as implementing both strategies. The idea is that the routines that free group handles (and the associated descriptors) loop through the cached information, calling an application-provided destructor routine for each piece of cached information. Typically, the cached information will be a pointer to a hunk of memory managed by the collective communication, which the destructor will free in whatever way it has to. Upon return from the destructor, the group-freeing routine will release the little piece of memory holding the pointer, and everything will be cleaned up. If that group handle/descriptor is ever reused, it will be reinitialized to indicate no cached information, and MPI_TestGroupAttribute will return "not found". An efficient-after-first-call group-global operation using cacheing might look like this: static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_key_type gop_key; /* key for this module's stuff */ efficient_global_op (grphandle, ...) struct group_descriptor_type *grphandle; { struct gop_stuff_type *gop_stuff; /* whatever we need */ if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetAttributeKey()) ) { MPI_abort ("Insufficient keys available"); } } if (MPI_TestGroupAttribute (grphandle,gop_key,&gop_stuff)) { /* This module has executed in this group before. We will use the cached information */ } else { /* This is a group that we have not yet cached anything in. We will now do so. */ gop_stuff = /* malloc a gop_stuff_type */ /* ... fill in *gop_stuff with whatever we want ... */ MPI_SetGroupAttribute (grphandle, gop_key, gop_stuff, gop_stuff_destructor); } /* ... use contents of *gop_stuff to do the global op ... */ } gop_stuff_destructor (gop_stuff) /* called by MPI on group close */ struct gop_stuff_type *gop_stuff; { /* ... free storage pointed to by gop_stuff ... */ } ---------------------------------------------------------------------- rj_littlefield@pnl.gov Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Thu Mar 18 13:18:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16171; Thu, 18 Mar 93 13:18:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18320; Thu, 18 Mar 93 13:17:37 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 13:17:35 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18296; Thu, 18 Mar 93 13:17:07 -0500 Date: Thu, 18 Mar 93 18:17:01 GMT Message-Id: <9471.9303181817@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal I To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Here is Proposal I, basic and extended proposals, as promised. I have not added detailed justifications or implementation complexity discussions in this document. This should be transparent. I have crafted the proposal using LaTeX, in the form of a "chapter" in the "report" document style. You should be able to run this through LaTeX and get it printed off no problem, please ask me for postscript if you do have some problem. The stuff before and including, \begin{document}, and the stuff after and including \end{document} could be removed for inclusion into a report document style. Comments? Best Wishes Lyndon >------------------------------ Cut Here ------------------------------< \documentstyle{report} \begin{document} \title{``Proposal'' I for MPI Communication Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Lyndon J. Clarke % March 1993 % \chapter{Proposal I} \section{Introduction} This chapter proposes that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The proposal reflects the observation that an instance of a module in a parallel program typically operates within a group of processes, and as such any communication contexts associated with an instance of a module also bind the semantics of process groups. This chapter makes a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The proposals say nothing about use of message tags. It is assumed that these will be a bit string, expressed as an integer in the host langauge (i.e. ANSI C or Fortran~77, in the first instacnce. These proposals should be viewed as a collection of recommendations to other subcommittees within MPI, primarily the collective communications subcommittee, the point-to-point communications subcommittee and the process topologies subcommittee. Concrete syntax is given in the style of the C language for purposes of discussion only, and should be viewed as an example of possible syntax as detailed syntax of described operations is the responsibility of the language bindings subcommittee %----------------------------------------------------------------------% % BEGIN "Basic Proposal" % \subsection{Basic Proposal} The main features of the basic proposal are: \begin{itemize} \item Process identifiers and group identifiers have opaque values expressed as an integer type in the base language. Semantics for passing process identifiers in a message are defined, whereas semantics for passing group identifiers in a message are not defined. \item Group creation and destruction are concerted, synchronous, actions on the part of the membership of the group concerned. \item Groups are {\it static\/} in that no provision is made for modification of the membership of a group over the lifetime of the group. Dynamic group operations such as {\it resize\/} can be effected by destruction of the existing group and creation of a new resized group. \item Group creation is effected by four means: explicit definition of the membership of a group; partition of an existing group into one or more distinct subgroups; identical duplication of an existing group; topological duplication of an existing group. \item Point-to-point communication provides for intra-group communication, and makes no provision for inter-group communication. \item Collective communication provides for operations which are concerted actions on the part of members of one group, and makes no provision for operations which are concerted actions on the part of members of multiple groups. \end{itemize} \subsubsection{Process and Group Identifiers} A {\it process identifier\/} is an opaque reference to an object which is a single process. A process identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on process identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). Each process has exactly one process identifier. MPI should provide a procedure which allows the user to determine the process identifier of the calling process. For example, {\tt mypid = mpi\_mypid()}. The identifier of the {\it null process\/}, defined to be a process which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_PID\_NULL} in this proposal. With the single exception of the identifier of the null process, the value of a process identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a process P then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a process identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the received process identifier is {\it invalid\/}. MPI will provide a mechanism which allows a process identifier to be passed in a message in such a manner that the received identifier is valid. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a process identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_pid(bd, \&pid)}. Transmission of a process identifier using this mechanism returns to the recipient a process identifier which is valid for use in the MPI operations described below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a process identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_pid\_invalidate(pid)}. This is an error if {\tt pid} is {\tt MPI\_PID\_NULL}, or if {\tt pid} is the identifier of the calling process. It is further proposed that MPI provide a process identifier registry service. This service allows any process to register its own process identifer by name, and deregister its process identifier. The service allows any process to determine whether a name has been registered without blocking the calling process, and to map that name into a valid process identifier with the possibility of blocking the calling process. Use of this service is not mandated, and components of programs which do not require this service are not expected to make use thereof. A {\it group identifier\/} is an opaque reference to an object which is a group of processes. A group identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on group identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). The identifier of the {\it null group\/}, defined to be a group which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_GID\_NULL} in this proposal. With the single exception of the identifier of the null group, the value of a group identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a group G then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a group identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the identifier is {\tt invalid\/}. MPI will not provide a mechanism which allows a group identifier to be passed in a message in such a manner that the received identifier is valid. The canonical representation of a group is an array of distinct process identifiers, although it may be possible to use hashing functions with lower space complexity and marginally higher time complexity. A group is {\it static\/}, in that it's membership may not change over the lifetime of the group. There is a well defined {\it size\/} of a group, and MPI will provide a procedure which allows the user to determine the size of a group. For example, {\tt size = mpi\_grp\_size(gid)}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. There is a well defined {\it rank\/} of a process within the group, i.e. the position in the representational array at which the process identifier is held. MPI will provide a procedure which allows the user to determine the rank of the calling process within a group. For example, {\tt mpi\_grp\_myrank(gid)} returns the rank of the calling process within the group referred to by {\tt gid}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. MPI should provide a procedure to determine the member rank of a process within a group given the group identifier and a valid process identifier, e.g. {\tt rank = mpi\_grp\_rank(gid,pid)}. This procedure may ``fail'' if the process identified by {\tt pid} is not a member of the group identifier by {\tt gid}, and can be used to determine whether a given process is a member of a given group. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. MPI should provide a procedure to determine the process identifier of a group member given the group identifier and member rank, e.g. {\tt pid = mpi\_pid(gid,rank)}. This procedure should validate the returned process identifier. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. \subsubsection{Group Creation and Destruction} MPI will provide four methods for creation of groups, and a procedure to destroy an existing group. A group may be created by explicit definition of the pids of the members of the group. For example, {\tt gid = mpi\_grp\_definition(npids,pids)} where {\tt gid} is the identifier of the newly created group, {\tt npids} is the number of processes in the new group, and {\tt pids} is an array containing the (valid) process identifiers of the processes in the group. This procedure must be called my all members of the group, and does not return until all members have made the call. A group may be created by identical duplication of an existing group. For example, {\tt gidb = mpi\_grp\_duplication(gida)} where {\tt gidb} is the group identifier of the newly created group and {\tt gida} is the identifier of an existing group. The created group inherits all properties of the source group, including any topological properties. This operation has the same synchronisation properties as creation of group by definition. A group may be created by topological duplication of an existing group. Details of topological groups are under consideration within the process topologies subcommittee and will not be further discussed here. A group created by topological duplication inherits the size of the source group, and also inherits the membership list of the source group although the list may be ordered differently in the created group. These operations have the same synchronisation properties as creation of group by identical duplication. Where groups have additional topological attributes MPI should also provide procedures which allow the user to determine such attributes. One or more groups may be created by partition of an existing group into distinct subgroups by key. For example, {\tt gidb = mpi\_grp\_partition(gida, key)} where {\tt gidb} is the group identifier of the newly created group corresponding to the given {\tt key} and {\tt gida} is the group identifier of the group which is being partitioned according to the {\tt key} values supplied. MPI should define a named constant which is a {\it null\/} {\tt key} value, for example {\tt MPI\_KEY\_NULL}, in order that members of the parent group can choose not to become members of any child group, and in which case the procedure should return {\tt MPI\_GID\_NULL}. Groups created by partition share the same ordering of process member ranks as the parent group. This operation synchronises the members of the parent group, and therefore implicitly synchronises the members of the created group(s). A group may be destroyed, e.g. {\tt mpi\_grp\_deletion(gid)}, which destroys the group identified by {\tt gid}. This operation synchronises the group members, and invalidates the group identifier. \subsubsection{Point-to-point communication} There are arguments either way for point-to-point routines which accept a {\tt (gid, rank)} pair, and for routines which accept a {\tt pid}. This proposal supports and seperates both approaches to addressing and selection within the same syntax, avoiding the introduction of sets of procedures to handle each case. In order to provide expressive power to intra-group communication, point-to-point communication should accept a {\tt (gid, rank)} pair, where {\tt gid} is a valid group identifier and {\tt rank} is a member rank within the group identified by {\tt gid}. In message addressing the sender specifies the {\tt rank} and {\tt gid} of the receiver. In message selection the receiver specifies the {\tt rank} and {\tt gid} of the receiver. The sender and receiver must both specify the same {\tt gid} in order for a match to occur. The {\tt gid} field is not allowed to take a {\it wildcard\/} value in message selection. The {\tt rank} field is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_RANK\_WILD}, which will match with any rank. One is encouraged to visualise a seperate message queue or port at each process for each group of which that process is a member (and indeed this may be an advantageous implementation feature). In order to accommodate process identifier based addressing and selection into the same syntax, this proposal advocates that point-to-point communication should also accept the null group ({\tt gid = MPI\_GID\_NULL}), in which case the {\tt rank} is interpreted as a valid {\tt pid}. The {\tt pid} is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_PID\_WILD}. The point-to-point section should provide a procedure which allows a recipient to recover the process identifier of the sender. The discussion of matching above extends to this case, and one is also encouraged to visualise a separate message queue or port for messages referred to by {\tt MPI\_GID\_NULL}. In the case of process identified wildcard receive, the process identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the process identifier must be performed by the MPI implementation, in order that the recipient is returned a valid process identifier, else the returned identifier is of little or no use to the recipient. \subsubsection{Collective communication} Collective communication operations within MPI should be restricted to the scope of a single group. It will be sufficient for these procedures to accept a group identifier, and possibly a message tag in order to distinguish multiple outstanding operations within the same group. These procedures must not accept {\tt MPI\_GID\_NULL}. It is not possible to determine whether this is strategy allows all of the MPI collective communication routines to be written in terms of MPI point-to-point routines without loss of generality, since the set of collective communication routines is not yet determined. This proposal takes the view that it is the responsibility of the collective communications subcommittee to determine whether such a goal is desirable, and if so to describe procedures which comply with this goal. % % END "Basic Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Extended Proposal" % \subsection{Extended Proposal} The main additional features of the extended proposal are: \begin{itemize} \item Semantics for passing a group identifier in a message are defined, and a group registry is introduced. \item Point-to-point communication is extended to allow inter-group communication without syntactic intrusion on intra-group communication. \item Collective communication operations involving a concerted action on the part of members of two or more groups is suggested. \end{itemize} \subsubsection{Process and Group Identifiers} The extended proposal says nothing more about process identifiers than the basic proposal. The extended proposal defines a mechanism by which a group identifier may be passed in a message in such a manner that the received identifier is valid, and the mechanism should be analagous to that which allows process identifiers to be transmitted. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a group identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_gid(bd, \&gid)}. Transmission of a group identifier using this mechanism returns to the recipient a group identifier which is valid for use in the MPI operations described above and below, and as further qualified below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a group identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_invalidate\_gid(gid)}. This is an error if {\tt gid} is {\tt MPI\_GID\_NULL}, or if {\tt gid} is the identifier of a group of which the calling process is a member. It is further proposed that MPI provide a group identifier registry service. This service allows any group to register its own group identifer by name, and deregister its group identifier. The service allows any group to determine whether a name has been registered without blocking the calling group, and to map that name into a validated group identifier with the possibility of blocking the calling group. The group registry procedures should synchronise the calling group, permitting more efficient implementations than asynchronous operations. The definition of a valid group identifier is extended to include all groups of which the process is a member and all those which are validated by one of the above mechanisms. The small suite of procedures which map between process identifiers, group identifiers and member ranks are defined to work with valid group and process identifiers. \subsubsection{Point-to-point communication} The point-to-point communication syntax and semantics described in the basic proposal do not extend directly to the case of inter-group communication. The reason is quite simple --- the sender and receiver do not supply the same group since the group of the sender and the group of the receiver are different. The basic approach is to express point-to-point communication in terms of a triplet {\tt (localGroup, remoteGroup, remoteRank)}. In this notation the sender specifies the sender group identifier, then the receiver group identifier and finally the receiver rank. The receiver specifies the receiver group identifier, then the sender group identifier, and finally the sender rank. The {\tt localGroup} field may not take a wildcard value, corresponding directly to the rule that the group in the basic proposal may not take a wildcard value. The {\tt remoteGroup} field may take a wildcard value, e.g. {\tt MPI\_GID\_WILD}, which matches with any group. The {\tt remoteRank} may also take a wildcard value, as in the basic proposal. The point-to-point section should provide procedures which allow a message recipient to recover the group and rank of the sender. In the case of {\tt remoteGroup} wildcard receive, the group identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the group identifier must be performed by the MPI implementation, in order that the recipient is returned a valid group identifier, else the returned identifier is of little or no use to the recipient. In order to accomodate inter-group addressing and selection into the framework of the basic proposal, the extended proposal suggests a careful redefinition of the {\tt gid} discussed in the basic proposal. With careful presentation this redefinition need not intrude conceptually on the basic proposal, although I shall give a less careful description here, suggestive of two different flavours of presentation. In the basic proposal, the {\tt gid} was formally a reference to a group. In the extended proposal the {\tt gid} is formally composed of references to two groups, and can be thought of as a shorthand notation for {\tt (localGroup, remoteGroup)}. The identifier of the null group, a {\it null\/} identifier, is a valid identifier which is formally composed of a pair of references to the null group, and may be used in the fashion described in the basic proposal. The group creation functions provide a symmetric, or {\it unary\/}, identifier formally composed of two references to the same group. This group is {\it local\/} since the process in question is a member of the group. An identifier which composes a pair of references to a local group is logically identical to a group identifier as implied in the basic proposal, and may be used for point-to-point and collective communications in an identical fashion. The group identifier transmission and registry lookup procedures also provide a symmetric, or {\it unary\/}, identifier which again is composed of a pair of references to the same group. This group is {\it remote\/} when the process is not a member of the group, or is local when the process is a member of the group. An identifier which composes a pair of references to a remote group is logically identical to the identifier a remote group as implied above, and is not valid for either point-to-point or collective communications. Inter-group communication is effected by use of assymetric, or {\it binary} identifiers, composed of references to two different groups, the first of which is local and the second of which is either local or remote. Such identifiers are constructed by the use of a ``glob'' operation, which returns an identifier referencing the first operand as the local field and the second operand as the remote field. The glob is defined to be an error unless: both operands are symmetric (unary); the first operand refers to a local group. The identifier returned by a glob is valid for point-to-point communciation and invalid for collective communication. The point-to-point communication section should provide a procedure to determine the group identifiers object of a completed receive. Procedures which ``unglob'' an assymetric (binary) group identifier should be provided, returning the local and remote fields as valid identifiers. This can be viewed as a continuation of a generalised approach to point-to-point communication began in the basic proposal, in which operation are specified in terms of object identifiers and instance identifiers where objects are composed of multiple instances. The nature of the instance identified, and the semantics of the operation, are dependent of the nature of the object identified, exploiting some genericity. \begin{center} \begin{tabular}{lll} object identifier & instance identifier & action \\ \hline null group identifier & basic process identifier & basic communication \\ unary group identifier & local process rank & intra-group communication \\ binary group identifier & remote process rank & inter-group communication \\ \end{tabular} \end{center} \subsubsection{Collective communication} There are a number of collective communication operations which logically extend over two or more groups. Some examples of two-group collective communications which are common in the simple host-node programming model are: \begin{itemize} \item {\it Broadcast\/} is an implicitly assymetric operation. There is exactly one sender and there are many receivers, each of which receives an identical message. The sender is a member of a singleton group G and the receivers are members of a group H. \item {\it Scatter\/} is an implicitly assymetric operation. There is exactly one sender and there are may receivers, each of which receives a different message. The sender is a member of a singleton group G and the receivers are members of a group H. \item There is a variant of {\it gather\/}, which returns the gathered data to a single process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \item There is a variant of {\it reduce}, which returns the reduced data to a sinle process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \end{itemize} Other patterns arise in ``process'' graphs where each ``process'' is allowed to be parallel. For example: \begin{itemize} \item {\it all-to-all\/} communciation in which the senders and receivers are distinct processes. The senders are members of a group G and the receivers are members of a group H. The two groups G and H need not be of the same size. \end{itemize} The assymetric (binary) group identifier objects described for point-to-point communications in this extended proposal are immediately suitable for each of the two-group collective communication operations described. % % END "Extended Proposal" %----------------------------------------------------------------------% \section{Conclusion} This chapter has propose that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The chapter made a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The basic proposal provides expressive semantics for the case of intra-group communication such as arises in program which compose data driven parallelism, and is closely related to that which was discussed by Marc Snir at the February meeting in Dallas, and builds on discussions which have taken place in various subcommittees. The key additional features are: point-to-point communication can also be expressed in terms of process identifiers; a process registry service was added, use of which is optional. The extended proposal adds expressive semantics for the case of inter-group communication such as arises in programs which compose combinations of data and function driven parallelism. This functionality has been constructed in such a manner that there is no syntactic or performance intrusion on the content of the basic proposal, and the additional conceptual content can be presented seperately from a presentation of intra-group communication. % % END "Proposal I" %======================================================================% \end{document} >------------------------------ Cut Here ------------------------------< /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Mar 18 13:58:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16568; Thu, 18 Mar 93 13:58:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20541; Thu, 18 Mar 93 13:53:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 13:53:15 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20533; Thu, 18 Mar 93 13:53:14 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17730; Thu, 18 Mar 1993 13:53:13 -0500 Date: Thu, 18 Mar 1993 13:53:13 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303181853.AA17730@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Agenda Can the context subcommittee please get together and tell me when it wants to present its idea to the full group. Also is a separate context subcommittee session required at the meeting prior to the full group meeting on contexts. What would be the length of the session(s). Just let me know and I'll change the agenda accordingly. David From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:14:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17038; Thu, 18 Mar 93 14:14:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21495; Thu, 18 Mar 93 14:11:59 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:11:58 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21487; Thu, 18 Mar 93 14:11:57 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29366; Thu, 18 Mar 93 13:07:04 CST Date: Thu, 18 Mar 93 13:07:04 CST From: Tony Skjellum Message-Id: <9303181907.AA29366@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: Agenda We need to meet as a sub-committee, to vote on the priority of our internal proposals, which will also be presented beforehand to everyone, starting next Monday (night). We need about 2 hours for our own meeting time. For the meeting before the full committee, I think one hour is appropriate. Thank you, Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:14:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17042; Thu, 18 Mar 93 14:14:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21341; Thu, 18 Mar 93 14:10:06 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:10:03 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21325; Thu, 18 Mar 93 14:10:02 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29337; Thu, 18 Mar 93 13:03:12 CST Date: Thu, 18 Mar 93 13:03:12 CST From: Tony Skjellum Message-Id: <9303181903.AA29337@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu Subject: Proposals I & V Cc: mpi-context@cs.utk.edu Dear Context sub-committee.... we are alive and well, despite rumors of our demise :-), By 7pm EST on Monday, I would like this committee's accepted forms of Proposals I and V to be ready for repromulgation to the entire MPI list. This is necessary because we need to replace Marc Snir's straw CONTEXT proposals in pt2pt and collcomm; they are only supposed to be placeholders from what I understand, awaiting our efforts. From what I also have understood by speaking with Lyndon on telephone just now, Proposal I is our substantive replacement for these placeholders (but only one of the options for closure as we see it). For those of you who don't know, Rik sent out V a few days ago, but no commentary has been seen; Lyndon just sent out I (or I++ as he calls it ). This is to be followed by my presentation of III/IV by March 24, which we will discuss forthwith. There is no proposal 2 anymore. Please interact on I & V (I will comment as well), and help me to get these in shape by Monday, as stipulated. Please urge the organizers to include CONTEXT subcommittee in agenda as originally discussed in Dallas. We are no longer figuring as significantly as originally discussed/agreed, as far as I can tell, because of the chosen ordering of presentations (I am not sure we are even on the first day at all). Are we on the agenda now? Thanks, - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:16:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17146; Thu, 18 Mar 93 14:16:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21578; Thu, 18 Mar 93 14:14:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:14:00 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21570; Thu, 18 Mar 93 14:13:59 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29392; Thu, 18 Mar 93 13:09:05 CST Date: Thu, 18 Mar 93 13:09:05 CST From: Tony Skjellum Message-Id: <9303181909.AA29392@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: Agenda When is before pt2pt and collcomm does there's. - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 18 14:19:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17303; Thu, 18 Mar 93 14:19:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21794; Thu, 18 Mar 93 14:17:09 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 18 Mar 1993 14:17:08 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21786; Thu, 18 Mar 93 14:17:07 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29426; Thu, 18 Mar 93 13:12:13 CST Date: Thu, 18 Mar 93 13:12:13 CST From: Tony Skjellum Message-Id: <9303181912.AA29426@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: Agenda Sorry for fragments... if "less controversial" topics could be discussed at beginning, context subcommittee could miss that general session to vote, followed by our presentation to the whole. If we do not present first, I am afraid our value to the whole will be lost. I know Lyndon agrees on this; I have not heard opinions other than fear that the "straw" context proposals now in pt2pt and collcomm would be used as the de facto choice. We have two of our three proposals in solid form. By Monday, they will go to everyone. We will be prioritizing these choices at our meeting at Dallas. I will not arrive till noon [arriving from Portland with Jack]. So, just making the opening bell, I see no hope for us to vote amongst ourselves, unless you discuss "other parts of MPI" at opening. Can this be done? Thanks, - Tony From owner-mpi-context@CS.UTK.EDU Fri Mar 19 06:37:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23767; Fri, 19 Mar 93 06:37:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09670; Fri, 19 Mar 93 06:36:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 06:36:51 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09648; Fri, 19 Mar 93 06:36:45 -0500 Date: Fri, 19 Mar 93 11:36:36 GMT Message-Id: <10215.9303191136@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposals V and I To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Dear Colleagues (1) Please do send comments on Proposal I++ :-) negative or positive as you think fit, as quickly as you can. I will receive and reply to email over the weekend for brief periods on both Saturday and Sunday, probably afternoon GMT. (2) I intend to give comments on Proposal V, thanks to Rik, this afternoon, most probably around 4pm GMT. (3) I distributed Proposal I++ as a chapter of LaTeX report document style --- I will place Proposal V into this form and post to the addressee list. This will happen before (2), real soon. Rik --- I will do this in such a way that you can read it as plain text without particular grief. Tony --- I am happy to do likewise with your proposal if you like. Hopefully this could make it easier to distribute to the whole committee as a single document, and at a later stage progress toward a draft for the MPI working document. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 07:38:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02144; Fri, 19 Mar 93 07:38:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12096; Fri, 19 Mar 93 07:37:50 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 07:37:49 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12088; Fri, 19 Mar 93 07:37:34 -0500 Date: Fri, 19 Mar 93 12:37:29 GMT Message-Id: <10292.9303191237@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal V, LaTeX To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu LaTeX-ified Proposal V of Rik. PostScript to follow. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. \item Communication both within and between groups seems conceptually straightforward. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. \item Communication between different groups may be considered awkward. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 07:40:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02169; Fri, 19 Mar 93 07:40:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12140; Fri, 19 Mar 93 07:39:47 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 07:39:46 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12112; Fri, 19 Mar 93 07:38:58 -0500 Date: Fri, 19 Mar 93 12:38:51 GMT Message-Id: <10301.9303191238@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal V To: Tony Skjellum , Jim Cownie , Rik Littlefield , Sanjay Ranka Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu PostScript of LaTeX-ified Proposal V of Rik ---------------------------------------------------------------------- %!PS-Adobe-2.0 %%Creator: dvips 5.495 Copyright 1986, 1992 Radical Eye Software %%Title: context-rik.dvi %%CreationDate: Fri Mar 19 12:37:44 1993 %%Pages: 7 %%PageOrder: Ascend %%BoundingBox: 0 0 596 842 %%EndComments %DVIPSCommandLine: dvips context-rik %DVIPSSource: TeX output 1993.03.19:1234 %%BeginProcSet: tex.pro %! /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N} B /TR{translate}N /isls false N /vsize 11 72 mul N /@rigin{isls{[0 1 -1 0 0 0] concat}if 72 Resolution div 72 VResolution div neg scale isls{0 Resolution vsize 72 div mul TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{ CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N} B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook} if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255 {IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V{}B /RV statusdict begin /product where{ pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{ gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M} B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{ p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 39158280 55380996 1000 300 300 (/a/obsidian/disk/home/u36/lyndon/mpi/context-rik.dvi) @start /Fa 11 119 df<0020004001800380030006000E001C001C003C0038003800780078007800F800 F000F000F000F000F000F000F000F000F000F800780078007800380038003C001C001C000E0006 00030003800180004000200B297C9E13>40 D<800040003000380018000C000E00070007000780 0380038003C003C003C003E001E001E001E001E001E001E001E001E001E003E003C003C003C003 8003800780070007000E000C00180038003000400080000B297D9E13>I<7FFFFFE07FFFFFE078 1F81E0701F80E0601F8060E01F8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000 001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F80 00001F8000001F8000001F8000001F800007FFFE0007FFFE001C1C7E9B21>84 D98 D<000FF0000FF00001F00001F00001F00001F00001F0 0001F00001F00001F00001F001F9F00F07F01C03F03C01F07801F07801F0F801F0F801F0F801F0 F801F0F801F0F801F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C1B>100 D<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F800007800 007C00603C00601E00C00F038001FC0013127F9116>I<1E003F003F003F003F001E0000000000 0000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F 00FFE0FFE00B1E7F9D0E>105 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F8 00F8F800F8F800F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>111 DI114 D118 D E /Fb 11 119 df<000070000000007000000000F800000000F800000000F800000001FC00000001FC00000003 FE00000003FE00000003FE00000006FF000000067F0000000E7F8000000C3F8000000C3F800000 183FC00000181FC00000381FE00000300FE00000300FE00000600FF000006007F00000E007F800 00FFFFF80000FFFFF800018001FC00018001FC00038001FE00030000FE00030000FE000600007F 000600007F00FFE00FFFF8FFE00FFFF825227EA12A>65 D68 D<07FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F0003FF003FD F01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18167E951B >97 D<0001FE000001FE0000003E0000003E0000003E0000003E0000003E0000003E0000003E00 00003E0000003E0000003E0000003E0001FC3E0007FFBE000F81FE001F007E003E003E007E003E 007C003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E007C00 3E007C003E003E007E001E00FE000F83BE0007FF3FC001FC3FC01A237EA21F>100 D<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC00 00FC00007C00007C00007E00003E00181F00300FC07003FFC000FF0015167E951A>I<03FC1E0F FF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03C03E07C01F0F801FFF0013 FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC70003EF0001EF0001EF0001EF0 001E78003C7C007C3F01F80FFFE001FF0018217E951C>103 D<1C003F007F007F007F003F001C 000000000000000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F00FFE0FFE00B247EA310>105 D110 D<0FF3003FFF00781F00600700E00300E00300F00300FC00007FE0007FF800 3FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00C7F00011167E 9516>115 D<0180000180000180000180000380000380000780000780000F80003F8000FFFF00 FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8180 0F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>I118 D E /Fc 69 124 df<007E1F0001C1B1800303E3C00703C3C00E03C180 0E01C0000E01C0000E01C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0 000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01 C0000E01C0000E01C0000E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E 03C00E01800E00000E00000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C 17>I<007FC001C1C00303C00703C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C07FCFF8151D809C17>I<003F07E00001C09C18000380F018000701F03C00 0E01E03C000E00E018000E00E000000E00E000000E00E000000E00E000000E00E00000FFFFFFFC 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00 E01C007FC7FCFF80211D809C23>I<6060F0F0F8F8686808080808080810101010202040408080 0D0C7F9C15>34 D<60F0F8680808081010204080050C7C9C0C>39 D<004000800100020006000C 000C0018001800300030007000600060006000E000E000E000E000E000E000E000E000E000E000 E000E000600060006000700030003000180018000C000C00060002000100008000400A2A7D9E10 >I<800040002000100018000C000C000600060003000300038001800180018001C001C001C001 C001C001C001C001C001C001C001C001C0018001800180038003000300060006000C000C001800 10002000400080000A2A7E9E10>I<60F0F0701010101020204080040C7C830C>44 DI<60F0F06004047C830C>I<00010003000600060006000C000C000C00 18001800180030003000300060006000C000C000C0018001800180030003000300060006000C00 0C000C00180018001800300030003000600060006000C000C00010297E9E15>I<03C00C301818 300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E007600660 06700E300C300C18180C3007E0101D7E9B15>I<030007003F00C7000700070007000700070007 0007000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C 7C9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C00380070 006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<07E01830201C201C781E 780E781E381E001C001C00180030006007E00030001C001C000E000F000F700FF80FF80FF80FF0 0E401C201C183007E0101D7E9B15>I<000C00000C00001C00003C00003C00005C0000DC00009C 00011C00031C00021C00041C000C1C00081C00101C00301C00201C00401C00C01C00FFFFC0001C 00001C00001C00001C00001C00001C00001C0001FFC0121C7F9B15>I<300C3FF83FF03FC02000 2000200020002000200023E024302818301C200E000E000F000F000F600FF00FF00FF00F800E40 1E401C2038187007C0101D7E9B15>I<00F0030C06040C0E181E301E300C700070006000E3E0E4 30E818F00CF00EE006E007E007E007E007E007600760077006300E300C18180C3003E0101D7E9B 15>I<60F0F0600000000000000000000060F0F06004127C910C>58 D<60F0F060000000000000 0000000060F0F0701010101020204080041A7C910C>I<0FE03038401CE00EF00EF00EF00E000C 001C0030006000C000800180010001000100010001000100000000000000000000000300078007 8003000F1D7E9C14>63 D<000600000006000000060000000F0000000F0000000F000000178000 00178000001780000023C0000023C0000023C0000041E0000041E0000041E0000080F0000080F0 000180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E0004001E000C00 1F001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E06180 01801980070007800E0003801C0003801C00018038000180780000807800008070000080F00000 00F0000000F0000000F0000000F0000000F0000000F0000000F000000070000080780000807800 0080380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E >III<001F808000E0618001801980070007800E0003801C00 03801C00018038000180780000807800008070000080F0000000F0000000F0000000F0000000F0 000000F0000000F000FFF0F0000F80700007807800078078000780380007801C0007801C000780 0E00078007000B800180118000E06080001F80001C1E7E9C21>71 D73 D76 DII<003F800000E0E0000380380007001C000E00 0E001C0007003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F0 0001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C0780003C0380003803C000780 1C0007000E000E0007001C000380380000E0E000003F80001B1E7E9C20>II<003F800000E0E0000380380007001C000E000E001C0007003C000780380003807800 03C0780003C0700001C0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F0 0001E0700001C0780003C0780003C0380003803C0E07801C1107000E208E000720DC0003A0F800 00F0E020003FE0200000602000007060000078E000003FC000003FC000001F8000000F001B257E 9C20>II<07E0801C1980300580700380600180E00180E00080E00080E00080F000 00F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001C08001C08001 C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>I<7FFFFFC0700F01C0600F 00C0400F0040400F0040C00F0020800F0020800F0020800F0020000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000 000F0000000F0000000F0000000F0000001F800003FFFC001B1C7F9B1E>I87 D91 D<08081010202040404040808080808080B0B0F8F8787830300D0C7A9C15>II<1FC000307000783800781C00301C00001C00001C0001FC000F1C00381C00701C00601C00E0 1C40E01C40E01C40603C40304E801F870012127E9115>97 DI<07E00C301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E 127E9112>I<003F00000700000700000700000700000700000700000700000700000700000700 03E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E00700600700 700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E6006E006FFFEE0 00E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C071E061E0E0C0E000E 000E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C00381C00381C00381C 001818001C38000C300013C0001000003000001800001FF8001FFF001FFF803003806001C0C000 C0C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C001C001C001C001C00 1C001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C00000 0000000000000000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>IIIII<03F0 000E1C00180600300300700380600180E001C0E001C0E001C0E001C0E001C0E001C06001807003 803003001806000E1C0003F00012127F9115>II<03C1000C3300180B00300F00 700700700700E00700E00700E00700E00700E00700E00700600700700700300F00180F000C3700 07C700000700000700000700000700000700000700000700003FE0131A7E9116>II<1F 9030704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608FC00D127F91 10>I<04000400040004000C000C001C003C00FFE01C001C001C001C001C001C001C001C001C00 1C101C101C101C101C100C100E2003C00C1A7F9910>IIII<7F8FF00F03800F030007 020003840001C80001D80000F00000700000780000F800009C00010E00020E000607000403801E 07C0FF0FF81512809116>II<7FFC70386038407040F040E041C003C003800700 0F040E041C043C0C380870087038FFF80E127F9112>II E /Fd 1 16 df<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF80FF003C0 10107E9115>15 D E /Fe 33 122 df<0003E0000000000FF0000000003E380000000078180000 0000780C00000000F80C00000000F00C00000001F00C00000001F00C00000001F01C00000001F8 1800000001F83000000001F87000000001F86000000001F8C001FFE000FD8001FFE000FF00001C 0000FE0000180000FE00003000007E00003000007F0000600000FF0000600001FF8000C00003BF 80018000071FC00180000F0FE00300001E0FE00600003E07F00600007E03F80C0000FE03FC1800 00FE01FE300000FE00FE600000FE007FC00000FE003F8000C07F001FC000C07F000FF001C03F80 1FF803801FC0F0FE0F0007FFC03FFE0001FE0007F8002B287DA732>38 D<3C7EFFFFFFFF7E3C08 087B8712>46 D<000C00001C0000FC000FFC00FFFC00F0FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC00FFFFFCFFFF FC16257BA420>49 D<00FF000007FFE0001E07F8003801FC007800FE007E00FF00FF007F00FF00 7F80FF007F80FF003F807E003F803C003F8000007F8000007F0000007F000000FE000000FE0000 01FC000001F8000003F0000007C00000078000000F0000001E0000003800000070018000E00180 01C001800380030007000300060003000FFFFF001FFFFF003FFFFF007FFFFE00FFFFFE00FFFFFE 0019257DA420>I<00FF800007FFF0000F03F8001800FC003E00FE007F00FF007F007F007F807F 007F007F003F00FF001E00FE000000FE000000FC000001F8000003F0000007E00000FF800000FF 00000003E0000001F8000000FC000000FE0000007F0000007F0000007F8018007F807E007F80FF 007F80FF007F80FF007F80FF007F00FE00FF007C00FE003801FC001E03F8000FFFE00001FF0000 19257DA420>I<0000380000007800000078000000F8000001F8000003F8000003F8000007F800 000FF800001DF8000019F8000031F8000071F8000061F80000C1F80001C1F8000381F8000301F8 000601F8000E01F8001C01F8001801F8003001F8007001F800E001F800FFFFFFE0FFFFFFE00001 F8000001F8000001F8000001F8000001F8000001F8000001F8000001F800007FFFE0007FFFE01B 257EA420>I<08000C000F007C000FFFF8000FFFF0000FFFE0000FFFC0000FFF80000FFE00000C 0000000C0000000C0000000C0000000C0000000C0000000C7F80000DFFE0000F81F8000E007C00 0C007E0000003E0000003F0000003F0000003F8000003F8038003F807C003F80FE003F80FE003F 80FE003F80FE003F007C003F0060007E0030007C001800F8000F03F00007FFC00001FF00001925 7DA420>I<000FE000007FF80001F81C0003E00E0007C03F000F807F001F807F001F007F003F00 7F003F003E007E0000007E0000007E000000FE040000FE3FE000FE7FF000FEC0F800FE807C00FF 003E00FF003F00FF003F00FE003F80FE003F80FE003F80FE003F807E003F807E003F807E003F80 7E003F803E003F003E003F001F003E001F007E000F807C0007C1F80001FFE000007F800019257D A420>I<00000600000000000F00000000000F00000000001F80000000001F80000000001F8000 0000003FC0000000003FC0000000007FE0000000006FE0000000006FE000000000CFF000000000 C7F000000001C7F80000000183F80000000183F80000000303FC0000000301FC0000000701FE00 00000600FE0000000600FE0000000C007F0000000C007F0000001C007F80000018003F80000018 003F8000003FFFFFC000003FFFFFC0000070001FE0000060000FE0000060000FE00000C00007F0 0000C00007F00001800007F80001800003F80001800003F80003000001FC0007000001FC00FFF8 003FFFF0FFF8003FFFF02C287EA731>65 D<0000FF80080007FFF018003FC03C38007E000E7801 FC0003F803F00001F807E00000F80FE00000781FC00000781FC00000383F800000383F80000038 7F800000187F000000187F00000018FF00000000FF00000000FF00000000FF00000000FF000000 00FF00000000FF00000000FF00000000FF00000000FF000000007F000000007F000000187F8000 00183F800000183F800000181FC00000301FC00000300FE000006007E000006003F00000C001FC 000180007E000700003FC03C000007FFF8000000FFC00025287CA72E>67 DII73 D78 D80 D<00FF004007FFE0C00F80F9 C01E001FC03C000FC07C0003C0780003C0780001C0F80001C0F80000C0F80000C0FC0000C0FC00 0000FE0000007F8000007FF800003FFFC0003FFFF8001FFFFC000FFFFE0003FFFF0000FFFF8000 0FFFC00000FFC000001FC000000FE0000007E0000007E0C00003E0C00003E0C00003E0C00003E0 E00003C0E00003C0F0000780F8000780FE000F00E7C03E00C1FFF800803FE0001B287CA724>83 D<03FF00000FFFE0001F03F0003F80F8003F80FC003F807C001F007E001F007E0000007E000000 7E0000007E00000FFE0001FFFE0007F07E001FC07E003F807E007F007E00FE007E00FE007E18FE 007E18FE007E18FE00BE187F01BE183F873FF01FFE1FE003F80F801D1A7E9920>97 D<00007FE000007FE0000007E0000007E0000007E0000007E0000007E0000007E0000007E00000 07E0000007E0000007E0000007E0000007E0007F87E001FFE7E007E077E00FC01FE01F800FE03F 0007E03F0007E07E0007E07E0007E0FE0007E0FE0007E0FE0007E0FE0007E0FE0007E0FE0007E0 FE0007E0FE0007E07E0007E07E0007E07F0007E03F0007E01F000FE00F801FE007E077E003FFC7 FE007F07FE1F287EA724>100 D<007F8003FFE007E1F00F80F81F007C3F007E7E003E7E003E7E 003FFE003FFE003FFFFFFFFFFFFFFE0000FE0000FE0000FE00007E00007E00003F00003F00031F 80060FC00607F01C01FFF0003FC0181A7E991D>I<00FE03E007FFCFF00F83F8F01F01F0F03E00 F8E03E00F8007E00FC007E00FC007E00FC007E00FC007E00FC003E00F8003E00F8001F01F0000F 83E0000FFFC00010FE00001000000038000000380000003C0000003FFFF0001FFFFE001FFFFF00 0FFFFF801FFFFFC03C001FC07C0007E0F80003E0F80003E0F80003E0F80003E07C0007C07C0007 C03E000F801FC07F0007FFFC0000FFE0001C267E9920>103 D<0F001F801FC03FC03FC01FC01F 800F000000000000000000000000000000FFC0FFC00FC00FC00FC00FC00FC00FC00FC00FC00FC0 0FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC0FFFCFFFC0E297FA811>105 D108 DII< 007FC00001FFF00007E0FC000F803E001F001F003F001F803E000F807E000FC07E000FC0FE000F E0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE07E000FC07E000FC07F00 1FC03F001F801F001F000F803E0007E0FC0001FFF000007FC0001B1A7E9920>II 114 D<03F8C01FFFC03C07C07801C07001C0F000C0F000C0F800C0FC0000FFC0007FFC003FFF00 1FFF800FFFC001FFE0000FF00003F0C001F0C000F0E000F0E000F0F000E0F801E0FE03C0E7FF80 C1FC00141A7E9919>I<00600000600000600000600000E00000E00001E00001E00003E00007E0 001FE000FFFFC0FFFFC007E00007E00007E00007E00007E00007E00007E00007E00007E00007E0 0007E00007E00007E00007E00007E06007E06007E06007E06007E06007E06003F0C001F0C000FF 80003E0013257FA419>III120 DI E /Ff 8 116 df80 D86 D<0007FFC0000000003FFFFC00 000000FFFFFF00000003F801FFC0000007F0003FE0000007F8001FF000000FFC000FF800000FFC 000FFC00000FFC0007FC00000FFC0007FE00000FFC0003FE000007F80003FF000003F00003FF00 0000000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000000000 03FF000000000003FF0000000007FFFF00000000FFFFFF0000000FFFE3FF0000007FF803FF0000 01FFC003FF000003FF0003FF000007FC0003FF00000FF80003FF00001FF00003FF00003FF00003 FF00007FE00003FF00007FE00003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FF C00003FF0380FFC00007FF0380FFC00007FF03807FE0000DFF03807FE0001DFF03803FF00039FF 87001FF80070FFCF000FFE03E07FFE0007FFFF807FFC0000FFFE001FF800001FF00007E000312E 7CAD37>97 D<00FF80FFFF80FFFF80FFFF8003FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF80FFFFFFFFFFFFFFFFFF18487DC71F>108 D<00003FFC00000001FFFF8000000FFFFFF000003FF00FFC00007F8001FE0001FE00007F8003FC 00003FC007F800001FE007F800001FE00FF000000FF01FE0000007F81FE0000007F83FE0000007 FC3FE0000007FC7FC0000003FE7FC0000003FE7FC0000003FE7FC0000003FEFFC0000003FFFFC0 000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003 FFFFC0000003FFFFC0000003FF7FC0000003FE7FC0000003FE7FC0000003FE7FE0000007FE3FE0 000007FC3FE0000007FC1FE0000007F81FF000000FF80FF000000FF007F800001FE007FC00003F E003FE00007FC001FF0000FF80007F8001FE00003FF00FFC00000FFFFFF0000003FFFFC0000000 3FFC0000302E7DAD37>111 D<00FF801FF80000FFFF80FFFF0000FFFF83FFFFC000FFFF8FC07F F00003FF9E000FF80001FFF80007FC0001FFF00003FE0001FFE00001FF0001FFC00000FF8001FF 800000FFC001FF8000007FC001FF8000007FE001FF8000007FE001FF8000003FE001FF8000003F F001FF8000003FF001FF8000003FF001FF8000001FF801FF8000001FF801FF8000001FF801FF80 00001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF8 01FF8000001FF801FF8000001FF801FF8000001FF001FF8000003FF001FF8000003FF001FF8000 003FF001FF8000003FE001FF8000007FE001FF8000007FC001FF800000FFC001FF800000FF8001 FFC00001FF8001FFE00001FF0001FFF00003FE0001FFF8000FFC0001FF9E001FF80001FF8F80FF E00001FF87FFFFC00001FF81FFFE000001FF803FF0000001FF800000000001FF800000000001FF 800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80000000 0001FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80 0000000001FF800000000001FF800000000001FF8000000000FFFFFF00000000FFFFFF00000000 FFFFFF0000000035427DAD3D>I<00FF007F00FFFF01FFC0FFFF03FFE0FFFF0787F003FF0E0FF0 01FF1C1FF801FF381FF801FF301FF801FF701FF801FF600FF001FF600FF001FFE003C001FFC000 0001FFC0000001FFC0000001FFC0000001FF80000001FF80000001FF80000001FF80000001FF80 000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF 80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001 FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF800000FFFFFFC000 FFFFFFC000FFFFFFC000252E7DAD2C>114 D<001FFC030000FFFF870003FFFFCF000FE007FF00 1F8000FF003E00003F003E00001F007C00001F007C00000F00FC00000F00FC00000700FC000007 00FE00000700FF00000700FF80000000FFE00000007FFE0000007FFFF800003FFFFF00003FFFFF C0001FFFFFF0000FFFFFF80007FFFFFC0001FFFFFE00007FFFFF00001FFFFF000000FFFF800000 07FF80000000FFC0E000007FC0E000003FC0E000001FC0F000000FC0F000000FC0F800000FC0F8 00000FC0F800000F80FC00000F80FE00001F80FF00001F00FF80003E00FFC0007C00F9F803F800 F0FFFFF000E03FFFC000C007FE0000222E7CAD2B>I E /Fg 8 117 df<00003000000070000001 F0000007F000007FF000FFFFF000FFFFF000FF8FF000000FF000000FF000000FF000000FF00000 0FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000 000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF0 00000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000F F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00000 0FF0007FFFFFFE7FFFFFFE7FFFFFFE1F3779B62D>49 D<000000FFE0001800000FFFFC00180000 7FFFFF00380001FFE00FC0780007FE0001F0F8000FF8000079F8003FE000003FF8007FC000000F F800FF80000007F801FF00000007F803FE00000003F807FC00000001F807F800000001F80FF800 000000F80FF000000000F81FF000000000781FF000000000783FE000000000783FE00000000038 7FE000000000387FE000000000387FE000000000387FC000000000007FC00000000000FFC00000 000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FF C00000000000FFC00000000000FFC00000000000FFC00000000000FFC000000000007FC0000000 00007FC000000000007FE000000000007FE000000000387FE000000000383FE000000000383FE0 00000000381FF000000000381FF000000000700FF000000000700FF8000000007007F800000000 E007FC00000000E003FE00000001C001FF000000038000FF8000000780007FC000000F00003FE0 00001E00000FF800003C000007FE0000F0000001FFE007E00000007FFFFF800000000FFFFE0000 000000FFE00000353B7BB940>67 D<003FFC00000001FFFF80000007E00FE000000FC003F80000 0FE001FC00001FF001FE00001FF000FF00001FF000FF00001FF0007F00000FE0007F800007C000 7F80000000007F80000000007F80000000007F80000000007F80000000007F800000000FFF8000 0007FFFF8000003FE07F800001FF007F800007FC007F80000FF0007F80001FE0007F80003FE000 7F80007FC0007F80007FC0007F8380FF80007F8380FF80007F8380FF80007F8380FF8000BF8380 FF8000BF83807FC0013F83807FC0033F83803FE0061FC7001FF81C0FFE0007FFF007FC00007FC0 03F00029257DA42D>97 D<0003FF0000001FFFE000007F07F00001FC01FC0003F000FE0007E000 7F000FE0003F001FC0003F801FC0003F803FC0001FC03F80001FC07F80001FC07F80001FE07F80 001FE0FF80001FE0FF80001FE0FFFFFFFFE0FFFFFFFFE0FF80000000FF80000000FF80000000FF 80000000FF800000007F800000007F800000007F800000003FC00000003FC00000001FC00000E0 1FE00000E00FE00001C007F000038003F800070000FE000E00007FC07C00001FFFF0000001FF80 0023257EA428>101 D<01FC00000000FFFC00000000FFFC00000000FFFC0000000007FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC01FF000003FC07FFC00003FC1C0FF00003FC 3007F80003FC6003F80003FCC003FC0003FC8001FC0003FD0001FE0003FF0001FE0003FE0001FE 0003FE0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE00FFFFF07FFFF8FFFFF07FFF F8FFFFF07FFFF82D3A7EB932>104 D<01FC03FE0000FFFC1FFFC000FFFC780FF000FFFDE003F8 0007FF8001FC0003FF0000FE0003FE0000FF0003FC00007F8003FC00007FC003FC00003FC003FC 00003FE003FC00003FE003FC00001FE003FC00001FE003FC00001FF003FC00001FF003FC00001F F003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC 00001FE003FC00003FE003FC00003FE003FC00003FC003FC00003FC003FC00007F8003FC00007F 8003FE0000FF0003FF0001FE0003FF8003FC0003FDC007F80003FCF81FE00003FC3FFF800003FC 07FC000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 00000000FFFFF0000000FFFFF0000000FFFFF00000002C357EA432>112 D<01F80FC0FFF83FF0FFF870F8FFF8C1FC07F883FE03F983FE03F903FE03FB03FE03FA01FC03FA 00F803FA007003FE000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC0000 03FC000003FC000003FC000003FC000003FC0000FFFFF800FFFFF800FFFFF8001F257EA424> 114 D<001C0000001C0000001C0000001C0000001C0000003C0000003C0000003C0000003C0000 007C0000007C000000FC000000FC000001FC000003FC000007FC00001FFFFFC0FFFFFFC0FFFFFF C003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC 000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E001FC00C001FC01C0 01FE01C000FE0380007F8700001FFE000007F8001B357EB423>116 D E /Fh 17 117 df<001FC0000070200000C010000180380003807800070078000700300007000000 070000000700000007000000070000000700000007000000FFFFF8000700780007003800070038 000700380007003800070038000700380007003800070038000700380007003800070038000700 38000700380007003800070038000700380007003800070038007FE1FF80192380A21B>12 D<008003800F80F380038003800380038003800380038003800380038003800380038003800380 03800380038003800380038003800380038003800380038007C0FFFE0F217CA018>49 D<03F8000C1E001007002007804007C07807C07803C07807C03807C0000780000780000700000F 00000E0000380003F000001C00000F000007800007800003C00003C00003E02003E07003E0F803 E0F803E0F003C04003C0400780200780100F000C1C0003F00013227EA018>51 D<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0F001E0F001E0F001 E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001C00001C00001C00003 80000380300300780700780600700C002018001030000FC00013227EA018>57 D76 DI82 D<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7001E07003C07007807 00700700F00708F00708F00708F00F087817083C23900FC1E015157E9418>97 D<01FE000703000C07801C0780380300780000700000F00000F00000F00000F00000F00000F000 00F000007000007800403800401C00800C010007060001F80012157E9416>99 D<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000 E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I<01FC0007 07000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000F0000070 00007800203800201C00400E008007030000FC0013157F9416>I<0E0000FE00001E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E0070FFE7FF18237FA21B>104 D<1C001E003E001E001C000000000000 00000000000000000000000E00FE001E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E00FFC00A227FA10E>I<0E0000FE00001E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E01800E0200 0E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E0 0E00F00E00F8FFE3FE17237FA21A>107 D<0E00FE001E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E00FFE00B237FA20E>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E00 0E000E000E000E000E000E000E000E000F00FFF010157F9413>114 D<02000200020002000600 060006000E001E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E040E040E 040E040E040E040708030801F00E1F7F9E13>116 D E /Fi 24 121 df<7803C0FC07E0FC07E0 FE07F0FE07F07A03D0020010020010020010020010020010040020040020040020080040080040 10008020010020010040020014147EB020>34 D<00003FC0020001FFF8060007E01C06001F0007 0E003C00018E00780000DE00F000005E01E000003E03C000001E078000001E0F0000000E0F0000 000E1E000000061E000000063E000000063C000000027C000000027C000000027C000000027800 000000F800000000F800000000F800000000F800000000F800000000F800000000F800000000F8 00000000F800000000F80000000078000000007C000000007C000000027C000000023C00000002 3E000000021E000000021E000000040F000000040F0000000C078000000803C000001801E00000 1000F00000200078000040003C000180001F0003000007E01E000001FFF80000003FC00027327C B02F>67 D73 D77 D80 D<007F004001FFC0400780F0C00F0018C01C000DC03C0007C0 380003C0780001C0700001C0F00000C0F00000C0F00000C0F0000040F0000040F0000040F80000 40F80000007C0000007E0000003F0000003FE000001FFE00000FFFE00007FFF80001FFFE00007F FF000007FF8000007F8000000FC0000007E0000003E0000003E0000001F0000001F0800000F080 0000F0800000F0800000F0800000F0C00000F0C00000E0E00001E0E00001E0F00001C0F8000380 EC000780C7000F00C1E03E0080FFF800801FE0001C327CB024>83 D86 D<040020080040080040100080200100200100400200 400200400200800400800400800400800400800400BC05E0FE07F0FE07F07E03F07E03F03C01E0 141476B020>92 D<00FE0000070380000801E0001000F0003C0078003E0078003E003C003E003C 001C003C0000003C0000003C0000003C000007FC00007C3C0003E03C0007803C001F003C003E00 3C003C003C007C003C0078003C08F8003C08F8003C08F8003C08F8007C08F8007C087C00BC083E 011E100F060FE003F807C01D1E7D9D20>97 D<018000003F800000FF800000FF8000000F800000 078000000780000007800000078000000780000007800000078000000780000007800000078000 00078000000780000007800000078000000781F80007860700079803C007A000E007C000F007C0 0078078000380780003C0780003E0780001E0780001E0780001F0780001F0780001F0780001F07 80001F0780001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C00070 074000F0072001C006100380060C0E000403F80020317EB024>I<001FC00000F0380001C00400 078002000F000F001E001F001E001F003C001F007C000E007C00000078000000F8000000F80000 00F8000000F8000000F8000000F8000000F8000000F8000000780000007C0000007C0000003C00 00801E0000801E0001000F0001000780020001C00C0000F03000001FC000191E7E9D1D>I<003F 800000E0F0000380380007001C000E001E001E000E003C000F003C000F007C000F807800078078 000780F8000780FFFFFF80F8000000F8000000F8000000F8000000F8000000F800000078000000 780000007C0000003C0000801C0000801E0001000F0002000700020001C00C0000F03000001FC0 00191E7E9D1D>101 D<0007E0001C1000383800707C00E07C01E07C01C03803C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E0007FFF007FFF 0016317FB014>I<07000F801F801F800F80070000000000000000000000000000000000000000 00000001801F80FF80FF800F800780078007800780078007800780078007800780078007800780 0780078007800780078007800780078007800FC0FFF8FFF80D2F7EAE12>105 D<01803F80FF80FF800F8007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 8007800780078007800780078007800FC0FFFCFFFC0E317EB012>108 D<0181FE003FC0003F86 0780C0F000FF8801C1003800FF9001E2003C000FA000E4001C0007A000F4001E0007C000F8001E 0007C000F8001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E00078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FF FC1FFF83FFF0341E7E9D38>I<0181FC003F860F00FF880380FF9003C00FA001C007C001E007C0 01E007C001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E007 8001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0 078001E00FC003F0FFFC3FFFFFFC3FFF201E7E9D24>I<003FC00000E0700003801C0007000E00 0E0007001E0007803C0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001 F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E0780001E07C0003E03C0003C03C00 03C01E0007800E00070007000E0003801C0000E07000003FC0001C1E7E9D20>I<0181F8003F86 0F00FF9803C0FFA001E007C000F007C00078078000780780003C0780003E0780003E0780001E07 80001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001E0780003E 0780003C0780003C0780007807C000F007C000F007A001C007900380078C0E000783F800078000 000780000007800000078000000780000007800000078000000780000007800000078000000780 00000FC00000FFFC0000FFFC0000202C7E9D24>I<0183E03F8C18FF907CFF907C0FA07C07C038 07C00007C00007C000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000780000780000FC000FFFE00FFFE00161E7E9D19> 114 D<03FC200E02603801E03000E0600060E00060E00020E00020F00020F000207C00007F8000 3FFC001FFF0007FF8001FFE0000FE00001F08000F8800078800038C00038C00038C00038E00030 F00070F00060C800C0C6038081FE00151E7E9D19>I<00400000400000400000400000400000C0 0000C00000C00001C00001C00003C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C01003C0 1003C01003C01003C01003C01003C01003C01001C02001E02000E0400078C0001F00142B7FAA19 >I<018000603F800FE0FF803FE0FF803FE00F8003E0078001E0078001E0078001E0078001E007 8001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0 078001E0078001E0078001E0078003E0078003E0078003E0038005E003C009E001C019F000F021 FF001FC1FF201E7E9D24>I120 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%PaperSize: A4 %%EndSetup %%Page: 0 1 0 0 bop 302 1086 a Fi(\\Prop)r(osal)24 b(V")e(for)f(MPI)i(Comm)n(unication)d (Con)n(text)774 1178 y(Sub)r(committee)829 1360 y Fh(Rik)15 b(Little\014eld)853 1481 y(Marc)o(h)h(1993)p eop %%Page: 1 2 1 1 bop 262 619 a Fg(Chapter)28 b(1)262 826 y Ff(Prop)s(osal)36 b(V)262 1067 y Fe(1.1)64 b(Summary)324 1158 y Fd(\017)20 b Fc(Group)i(ID's)g(ha)o(v)o(e)g(lo)q(cal)f(scop)q(e,)k(not)d(global.)41 b(Groups)22 b(are)h(represen)o(ted)i(b)o(y)365 1208 y(descriptors)f(of)d (inde\014nite)h(size)g(and)g(opaque)f(structure,)k(managed)20 b(b)o(y)i(MPI.)365 1257 y(Group)13 b(descriptors)j(can)d(b)q(e)h(passed)h(b)q (et)o(w)o(een)g(pro)q(cesses)h(b)o(y)d(MPI)h(routines)g(that)365 1307 y(translate)h(them)e(as)h(necessary)m(.)365 1374 y(\(This)j(satis\014es) g(the)g(same)e(needs)j(as)e(global)f(group)h(ID's,)g(but)g(its)h(p)q (erformance)365 1423 y(implications)11 b(are)k(easier)f(to)g(understand)h (and)f(con)o(trol.\))324 1506 y Fd(\017)20 b Fc(Arbitrary)14 b(pro)q(cess-lo)q(cal)h(info)e(can)h(b)q(e)h(cac)o(hed)g(in)e(group)h (descriptors.)365 1573 y(\(This)i(allo)o(ws)e(collectiv)o(e)h(op)q(erations)h (to)f(run)h(quic)o(kly)e(after)i(the)g(\014rst)g(execution)365 1623 y(in)e(eac)o(h)g(group,)f(without)h(complicating)d(the)k(calling)d (sequences.\))324 1706 y Fd(\017)20 b Fc(There)15 b(are)g(restrictions)g (that)f(p)q(ermit)f(groups)h(to)g(b)q(e)g(la)o(y)o(ered)g(on)g(top)g(of)f (pt-pt.)324 1789 y Fd(\017)20 b Fc(Pt-pt)c(comm)o(unications)d(use)k(only)e (TID,)g(con)o(text,)h(and)g(tag,)f(and)h(are)g(sp)q(eci\014ed)365 1839 y(to)e(b)q(e)h(fast.)324 1922 y Fd(\017)20 b Fc(Group)c(con)o(trol)g(op) q(erations)g(\(forming,)e(disbanding,)i(and)g(translating)f(b)q(et)o(w)o(een) 365 1971 y(rank)f(and)g(TID\))f(are)i(NOT)f(sp)q(eci\014ed)h(to)f(b)q(e)h (fast.)262 2109 y Fe(1.2)64 b(Detailed)24 b(Prop)r(osal)324 2200 y Fd(\017)c Fc(Pt-pt)d(uses)h(only)e(\\TID",)g(\\con)o(text",)h(and)f (\\message)h(tag".)25 b(TID's,)17 b(con)o(texts,)365 2249 y(and)12 b(message)f(tags)h(ha)o(v)o(e)g(global)e(scop)q(e;)j(they)f(can)g(b)q(e)h (copied)f(b)q(et)o(w)o(een)h(pro)q(cesses)365 2299 y(as)i(in)o(tegers.)21 b(TID's)14 b(and)h(con)o(texts)h(are)f(opaque;)f(their)h(v)n(alues)g(are)g (managed)e(b)o(y)365 2349 y(MPI.)h(Message)h(tags)f(are)g(con)o(trolled)g(en) o(tirely)g(b)o(y)g(the)g(application.)967 2574 y(1)p eop %%Page: 2 3 2 2 bop 324 307 a Fd(\017)20 b Fc(Pt-pt)15 b(comm)o(unication)c(is)j(sp)q (eci\014ed)i(to)e(b)q(e)h(fast)g(in)f(all)f(cases.)21 b(\(E.g.,)13 b(MPI)i(m)o(ust)365 357 y(initialize)i(all)h(pro)q(cesses)j(suc)o(h)e(that)f (an)o(y)g(required)i(translation)e(of)g(the)h(TID)f(is)365 407 y(faster)d(than)f(the)g(fastest)h(pt-pt)f(comm)o(unication)c(call.\))324 490 y Fd(\017)20 b Fc(A)h(group)f(is)g(represen)o(ted)j(b)o(y)d(a)g(\\group)f (descriptor",)k(of)c(inde\014nite)i(size)g(and)365 540 y(opaque)c(structure,) j(managed)15 b(b)o(y)i(MPI.)f(The)i(group)f(descriptor)h(can)f(b)q(e)h (refer-)365 589 y(enced)e(via)e(a)h(\\pro)q(cess-lo)q(cal)g(group)f(ID",)g (whic)o(h)h(is)f(t)o(ypically)g(just)g(a)h(p)q(oin)o(ter)g(to)365 639 y(the)g(group)f(descriptor.)19 b(\(Group)14 b(ID's)f(with)h(global)e (scop)q(e)j(do)e(not)h(exist.\))324 722 y Fd(\017)20 b Fc(Group)f (descriptors)i(can)e(b)q(e)h(copied)f(b)q(et)o(w)o(een)i(arbitrary)e(pro)q (cesses)i(via)e(MPI-)365 772 y(pro)o(vided)14 b(routines)g(that)g(translate)g (them)f(to)g(and)g(from)f(mac)o(hine-)g(and)i(pro)q(cess-)365 822 y(indep)q(enden)o(t)e(form.)j(A)10 b(pro)q(cess)i(receiving)f(a)f(group)g (descriptor)i(do)q(es)f(not)f(b)q(ecome)365 872 y(a)h(mem)o(b)q(er)f(of)h (the)h(group,)f(but)h(do)q(es)g(b)q(ecome)f(able)g(to)h(obtain)e(information) f(ab)q(out)365 922 y(the)15 b(group)f(\(e.g.,)e(to)i(translate)g(b)q(et)o(w)o (een)i(rank)d(and)h(TID\).)324 1005 y Fd(\017)20 b Fc(Arbitrary)15 b(information)d(can)i(b)q(e)i(cac)o(hed)f(in)f(a)g(group)h(descriptor.)21 b(This)14 b(is,)g(MPI)365 1054 y(routines)d(are)f(pro)o(vided)g(that)g(allo)o (w)f(an)o(y)g(routine)h(to)g(augmen)o(t)f(a)g(group)h(descriptor)365 1104 y(with)17 b(arbitrary)g(information,)e(iden)o(ti\014ed)i(b)o(y)g(a)g(k)o (ey)g(v)n(alue,)g(that)g(subsequen)o(tly)365 1154 y(can)f(b)q(e)g(retriev)o (ed)h(b)o(y)e(an)o(y)h(routine)f(ha)o(ving)g(access)i(to)f(the)g(descriptor.) 24 b(Cac)o(hed)365 1204 y(information)10 b(is)j(not)g(visible)f(to)h(other)h (pro)q(cesses)h(and)e(is)g(stripp)q(ed)h(when)f(a)g(group)365 1254 y(descriptor)18 b(is)e(sen)o(t)h(to)g(another)f(pro)q(cess.)28 b(Retriev)n(al)15 b(of)h(cac)o(hed)h(information)d(is)365 1303 y(sp)q(eci\014ed)d(to)f(b)q(e)g(fast)g(\(signi\014can)o(tly)f(c)o(heap)q(er)i (than)e(a)h(pt-pt)g(comm)o(uni)o(cation)d(call\).)324 1386 y Fd(\017)20 b Fc(Group)e(descriptors)h(con)o(tain)e(enough)h(information)d (to)i(async)o(hronously)h(trans-)365 1436 y(late)c(b)q(et)o(w)o(een)h (\(group,rank\))f(and)g(TID)g(for)f(an)o(y)h(mem)o(b)q(er)e(of)h(the)i (group,)e(b)o(y)h(an)o(y)365 1486 y(pro)q(cess)j(holding)c(a)h(descriptor)h (for)g(the)g(group.)k(MPI)c(routines)g(are)f(pro)o(vided)h(to)365 1536 y(do)f(these)h(translations.)j(T)m(ranslation)13 b(is)g(not)h(sp)q (eci\014ed)i(to)d(b)q(e)i(fast.)324 1619 y Fd(\017)20 b Fc(A)o(t)14 b(creation,)g(eac)o(h)h(group)f(is)f(assigned)i(a)f(globally)d(unique)j (\\default)g(group)g(con-)365 1669 y(text")g(whic)o(h)g(is)g(k)o(ept)g(in)f (the)i(group)e(descriptor)j(and)d(can)h(b)q(e)h(quic)o(kly)d(extracted.)365 1719 y(This)i(extraction)g(is)g(sp)q(eci\014ed)i(to)d(b)q(e)i(fast)f (\(comparable)e(to)i(a)g(pro)q(cedure)h(call\).)324 1802 y Fd(\017)20 b Fc(The)d(default)g(group)f(con)o(text)h(can)g(b)q(e)g(used)h (for)e(pt-pt)h(comm)o(unicati)o(on)d(b)o(y)i(an)o(y)365 1851 y(mo)q(dule)8 b(op)q(erating)h(within)g(the)h(group,)f(but)h(only)e(for)h (exactly)h(paired)f(send/receiv)o(e)365 1901 y(with)i(exact)g(matc)o(h)e(on)i (TID,)f(con)o(text,)h(and)g(message)f(tag.)17 b(Call)9 b(this)i(the)g (\\paired-)365 1951 y(exact-matc)o(h)k(constrain)o(t".)22 b(This)15 b(constrain)o(t)h(allo)o(ws)e(indep)q(enden)o(t)j(mo)q(dules)d(to)365 2001 y(use)19 b(the)f(same)f(con)o(text)i(without)e(co)q(ordinating)g(their)i (tag)e(v)n(alues.)30 b(\(Assumes:)365 2051 y(tigh)o(t)12 b(sequencing)g (constrain)o(ts)h(for)e(b)q(oth)h(blo)q(c)o(king)f(and)h(non-blo)q(c)o(king)f (comms)e(in)365 2100 y(pt-pt)14 b(MPI.\))324 2183 y Fd(\017)20 b Fc(A)f(mo)q(dules)f(that)h(do)q(es)g(not)g(meet)f(the)h(paired-exact-matc)o (h)g(constrain)o(t)g(m)o(ust)365 2233 y(use)e(a)f(di\013eren)o(t)h(globally)d (unique)i(con)o(text)g(v)n(alue.)24 b(An)16 b(MPI)g(routine)h(will)d(exist) 365 2283 y(to)h(generate)i(suc)o(h)f(a)f(v)n(alue)g(and)g(broadcast)h(it)f (to)g(the)h(group.)22 b(The)16 b(cost)g(of)f(this)365 2333 y(op)q(eration)e(is)f(sp)q(eci\014ed)i(to)e(b)q(e)h(comparable)e(to)h(that)g (of)g(an)g(e\016cien)o(t)h(broadcast)g(in)365 2383 y(the)h(group,)f(i.e.)k (log\(G\))c(pt-pt)g(startup)h(costs)h(for)e(a)g(group)g(of)g(G)g(pro)q (cesses.)20 b([See)365 2433 y(Implemen)o(tation)11 b(note)j(1.])967 2574 y(2)p eop %%Page: 3 4 3 3 bop 324 307 a Fd(\017)20 b Fc(Con)o(text)c(v)n(alues)g(are)g(a)g(plen)o (tiful)e(but)i(exhaustible)g(resource.)26 b(If)15 b(further)i(use)g(is)365 357 y(an)o(ticipated,)i(a)f(con)o(text)g(ma)o(y)e(b)q(e)j(cac)o(hed;)i (otherwise)e(it)f(should)g(b)q(e)g(returned.)365 407 y(\(Note:)g(the)c(n)o (um)o(b)q(er)e(of)h(a)o(v)n(ailable)e(con)o(texts)j(should)e(b)q(e)i(sev)o (eral)g(times)e(the)h(n)o(um-)365 457 y(b)q(er)i(of)e(groups)h(that)g(can)g (b)q(e)h(formed.\))324 540 y Fd(\017)20 b Fc(When)g(a)e(group)h(disbands,)h (its)f(group)g(descriptor)i(is)d(closed)i(and)f(an)o(y)g(cac)o(hed)365 589 y(information)e(is)j(released.)37 b(\(The)21 b(MPI)f(group-disbanding)f (routine)h(do)q(es)h(this)365 639 y(b)o(y)15 b(calling)e(destructor)j (routines)f(that)g(the)g(application)e(pro)o(vided)h(when)h(the)g(in-)365 689 y(formation)d(w)o(as)i(cac)o(hed.\))19 b(Copies)14 b(of)f(the)i(group)f (descriptor)h(m)o(ust)e(b)q(e)i(explicitly)365 739 y(released.)j(It)11 b(is)f(an)g(error)i(to)e(mak)o(e)f(an)o(y)h(reference)i(to)f(a)f(descriptor)h (of)f(a)g(disbanded)365 789 y(group,)k(except)h(to)f(release)h(that)f (descriptor.)324 872 y Fd(\017)20 b Fc(All)11 b(pro)q(cesses)i(that)f(are)f (initially)e(allo)q(cated)i(b)q(elong)f(to)h(the)h(INITIAL)f(group.)17 b(\(In)365 922 y(a)e(static)h(pro)q(cess)h(mo)q(del,)c(this)i(means)f(all)g (pro)q(cesses.\))25 b(An)15 b(MPI)h(routine)f(exists)365 971 y(for)j(obtaining)f(a)g(reference)k(to)d(the)h(INITIAL)f(group)g(descriptor)h (\(i.e.,)f(a)f(lo)q(cal)365 1021 y(group)d(ID)g(for)f(INITIAL\).)324 1104 y Fd(\017)20 b Fc(Groups)10 b(are)h(formed)e(and)h(disbanded)g(b)o(y)g (sync)o(hronous)h(calls)f(on)g(all)f(participating)365 1154 y(pro)q(cesses.)21 b(In)14 b(the)h(case)g(of)e(o)o(v)o(erlapping)g(groups,)g (all)g(pro)q(cesses)k(m)o(ust)c(form)f(and)365 1204 y(disband)i(groups)g(in)g (the)g(same)f(order.)19 b([See)14 b(Implemen)o(tation)d(Note)j(2.])324 1287 y Fd(\017)20 b Fc(All)f(group)h(formation)e(calls)h(are)i(treated)g(as)f (a)f(partitioning)g(of)g(some)g(group,)365 1337 y(INITIAL)c(if)f(none)h (other.)20 b(The)15 b(calls)g(include)f(a)h(reference)i(to)d(the)h (descriptor)h(of)365 1386 y(the)g(group)f(b)q(eing)h(partitioned,)f(the)h (TID)f(of)f(the)i(new)g(group's)f(\\ro)q(ot)g(pro)q(cess",)365 1436 y(the)d(n)o(um)o(b)q(er)e(of)h(pro)q(cesses)i(in)e(the)h(group,)f(and)f (an)h(in)o(teger)h(\\group)e(tag")h(pro)o(vided)365 1486 y(b)o(y)h(the)h (application.)j(The)d(group)f(tag)g(m)o(ust)f(discriminate)g(b)q(et)o(w)o (een)j(o)o(v)o(erlapping)365 1536 y(groups)i(that)f(ma)o(y)f(b)q(e)i(formed)e (concurren)o(tly)m(,)i(sa)o(y)f(b)o(y)g(m)o(ultiple)e(threads)j(within)365 1586 y(a)e(pro)q(cess.)20 b([See)14 b(Implemen)o(tation)d(Note)j(2.])324 1669 y Fd(\017)20 b Fc(Collectiv)o(e)c(comm)o(unication)d(routines)k(are)g (called)f(b)o(y)g(all)f(mem)o(b)q(ers)g(of)h(a)g(group)365 1719 y(in)e(the)g(same)f(order.)324 1802 y Fd(\017)20 b Fc(Blo)q(c)o(king)f (collectiv)o(e)g(comm)o(unicati)o(on)d(routines)k(are)f(passed)h(only)e(a)h (reference)365 1851 y(to)g(the)h(group)e(descriptor.)35 b(T)m(o)18 b(comm)o(unicate,)g(they)h(can)g(either)h(use)g(the)f(de-)365 1901 y(fault)14 b(group)h(con)o(text)g(or)g(in)o(ternally)e(allo)q(cate)h(a)h (new)g(con)o(text,)g(with)f(or)h(without)365 1951 y(cac)o(heing.)324 2034 y Fd(\017)20 b Fc(Non-blo)q(c)o(king)13 b(collectiv)o(e)h(comm)o(uni)o (cation)d(routines)j(are)g(passed)h(a)f(reference)i(to)365 2084 y(the)h(group)f(descriptor,)i(plus)e(a)f(\\data)h(tag")f(to)h (discriminate)f(b)q(et)o(w)o(een)j(concur-)365 2134 y(ren)o(t)13 b(op)q(erations)g(in)f(the)h(same)f(group.)17 b(\(Question:)h(is)13 b(sequencing)g(enough)g(to)f(do)365 2183 y(this?\))18 b(An)11 b(implemen)o(tatio)o(n)e(is)i(allo)o(w)o(ed)e(but)j(not)f(required)h(to)f (imp)q(ose)f(sync)o(hron-)365 2233 y(ization)g(of)h(all)f(co)q(op)q(erating)h (pro)q(cesses)j(up)q(on)d(en)o(try)g(to)g(a)g(non-blo)q(c)o(king)f(collectiv) o(e)365 2283 y(comm)o(unication)h(routine.)967 2574 y(3)p eop %%Page: 4 5 4 4 bop 262 307 a Fe(1.3)64 b(Adv)l(an)n(tages)22 b(&)g(Disadv)l(an)n(tages) 262 406 y Fb(Adv)m(an)n(tages)324 483 y Fd(\017)e Fc(This)14 b(prop)q(osal)g(is)g(implemen)o(tabl)o(e)e(with)i(no)f(serv)o(ers)j(and)e (can)g(b)q(e)h(la)o(y)o(ered)f(easily)365 533 y(on)g(existing)g(systems.)324 616 y Fd(\017)20 b Fc(Cac)o(heing)c(auxiliary)e(information)e(in)j(group)h (descriptors)h(allo)o(ws)d(collectiv)o(e)i(op-)365 666 y(erations)d(to)g(run) g(at)g(maxim)n(um)c(sp)q(eed)14 b(after)f(the)g(\014rst)h(execution)g(in)e (eac)o(h)h(group.)365 715 y(\(Assumes)i(that)f(su\016cien)o(t)g(con)o(text)g (v)n(alues)g(are)g(a)o(v)n(ailable)e(to)i(p)q(ermit)f(cac)o(heing.\))324 799 y Fd(\017)20 b Fc(Sp)q(eed)c(of)e(cac)o(hed)i(collectiv)o(e)e(op)q (erations)h(do)q(es)h(not)e(dep)q(end)i(on)f(sp)q(eed)h(of)e(group)365 848 y(op)q(erations)g(suc)o(h)h(as)f(formation)d(or)j(translation)f(b)q(et)o (w)o(een)j(rank)e(and)f(TID.)324 931 y Fd(\017)20 b Fc(Requires)13 b(explicit)e(translation)h(b)q(et)o(w)o(een)h(\(group,rank\))f(and)g(TID,)f (whic)o(h)h(mak)o(es)365 981 y(pt-pt)j(p)q(erformance)g(predictable)g(no)g (matter)f(ho)o(w)g(the)h(functionalit)o(y)e(of)h(groups)365 1031 y(gets)h(extended.)324 1114 y Fd(\017)20 b Fc(Comm)o(unicatio)o(n)7 b(b)q(oth)i(within)g(and)g(b)q(et)o(w)o(een)h(groups)g(seems)f(conceptually)h (straigh)o(t-)365 1164 y(forw)o(ard.)262 1280 y Fb(Disadv)m(an)n(tages)324 1357 y Fd(\017)20 b Fc(Requires)d(explicit)f(translation)g(b)q(et)o(w)o(een)i (\(group,rank\))e(and)g(TID,)f(whic)o(h)i(ma)o(y)365 1406 y(b)q(e)e (considered)g(a)o(wkw)o(ard.)324 1489 y Fd(\017)20 b Fc(Comm)o(unicatio)o(n) 11 b(b)q(et)o(w)o(een)16 b(di\013eren)o(t)e(groups)g(ma)o(y)e(b)q(e)j (considered)g(a)o(wkw)o(ard.)324 1572 y Fd(\017)20 b Fc(No)d(free-for-all)f (group)h(formation.)24 b(A)17 b(pro)q(cess)i(m)o(ust)d(kno)o(w)g(something)g (ab)q(out)365 1622 y(who)e(its)g(collab)q(orators)f(are)h(going)f(to)h(b)q(e) g(\(minimal)o(ly)l(,)d(the)j(ro)q(ot)g(of)f(the)i(group\).)324 1705 y Fd(\017)20 b Fc(Requires)15 b(explicit)f(coherency)j(of)d(group)g (descriptors,)i(whic)o(h)e(is)h(not)f(con)o(v)o(enien)o(t)365 1755 y({)f(application)f(m)o(ust)g(k)o(eep)i(trac)o(k)f(of)g(whic)o(h)f(pro)q (cesses)k(ha)o(v)o(e)d(copies)h(and)f(what)g(to)365 1805 y(do)h(with)g(them.) 324 1888 y Fd(\017)20 b Fc(Static)13 b(group)f(mo)q(del.)k(Pro)q(cesses)f (can)e(join)f(and)g(lea)o(v)o(e)g(collab)q(orations)f(only)h(with)365 1938 y(the)k(co)q(op)q(eration)f(of)f(the)h(other)h(group)e(mem)o(b)q(ers)g (in)g(forming)f(larger)h(or)h(smaller)365 1988 y(groups.)324 2071 y Fd(\017)20 b Fc(No)c(direct)g(supp)q(ort)g(for)f(iden)o(tifying)f(the) i(group)g(from)d(whic)o(h)j(a)f(message)g(came.)365 2120 y(The)j(sender)g (cannot)g(em)o(b)q(ed)e(its)h(group)g(ID)g(in)f(its)h(message,)g(since)h (group)f(ID's)365 2170 y(are)h(not)f(globally)e(meaningful.)25 b(One)17 b(metho)q(d)g(to)g(address)h(this)f(problem)f(is)h(to)365 2220 y(ha)o(v)o(e)11 b(the)h(sender)h(em)o(b)q(ed)d(its)h(group)g(con)o(text) h(as)f(data)g(in)f(the)i(message,)f(then)h(ha)o(v)o(e)365 2270 y(the)k(receiv)o(er)g(use)g(that)f(v)n(alue)f(to)g(select)j(among)12 b(group)j(descriptors)h(that)f(it)g(had)365 2320 y(already)f(b)q(een)h(sen)o (t.)967 2574 y(4)p eop %%Page: 5 6 5 5 bop 262 307 a Fe(1.4)64 b(Commen)n(ts)20 b(&)i(Alternativ)n(es)324 398 y Fd(\017)e Fc(The)13 b(p)q(erformance)g(of)f(group)g(con)o(trol)h (functions)f(is)h(explicitly)e(unsp)q(eci\014ed)k(in)d(or-)365 448 y(der)g(to)f(pro)o(vide)g(p)q(erformance)h(insurance.)18 b(The)11 b(in)o(ten)o(t)h(is)f(to)g(encourage)h(program)365 498 y(designs)17 b(whose)g(p)q(erformance)f(w)o(on't)f(b)q(e)i(killed)e(if)g (the)h(functionalit)o(y)f(of)g(groups)365 548 y(is)f(expanded)h(so)f(as)g(to) f(b)q(e)i(unscalable)f(or)g(require)g(exp)q(ensiv)o(e)h(services.)324 631 y Fd(\017)20 b Fc(Adding)13 b(global)f(group)h(ID's)f(w)o(ould)h (seriously)g(in)o(terfere)h(with)f(la)o(y)o(ering)f(this)i(pro-)365 680 y(p)q(osal)d(on)g(pt-pt.)17 b(The)11 b(problem)f(is)h(not)g(generating)g (a)g(unique)g(ID,)f(but)h(using)g(it.)17 b(If)365 730 y(just)e(holding)e(a)g (global)g(group)h(ID)g(w)o(ere)h(to)f(allo)o(w)f(a)g(pro)q(cess)j(to)e(async) o(hronously)365 780 y(ask)i(questions)h(ab)q(out)f(the)h(group)f(\(lik)o(e)f (translating)h(b)q(et)o(w)o(een)h(rank)f(and)g(TID\),)365 830 y(then)e(a)e(group)h(information)d(serv)o(er)k(of)e(some)g(sort)i(w)o(ould)e (b)q(e)h(required,)h(and)e(MPI)365 880 y(pt-pt)i(do)q(es)h(not)f(pro)o(vide)g (enough)g(capabilit)o(y)e(to)i(construct)h(suc)o(h)g(a)e(b)q(east.)324 963 y Fd(\017)20 b Fc(The)c(restriction)g(that)f(only)f(paired-exact-matc)o (h)g(mo)q(dules)g(can)h(run)g(in)g(the)g(de-)365 1012 y(fault)j(group)g(con)o (text)h(could)g(b)q(e)g(relaxed)g(to)f(something)f(lik)o(e)h(\\a)g(mo)q(dule) f(that)365 1062 y(violates)g(the)h(paired-exact-matc)o(h)f(constrain)o(t)h (can)g(run)f(in)h(the)g(default)f(group)365 1112 y(con)o(text)e(if)f(and)g (only)g(if)g(it)g(is)g(the)h(only)f(mo)q(dule)f(to)h(run)h(in)f(that)g(con)o (text".)21 b(This)365 1162 y(seems)11 b(to)q(o)g(error-prone)h(to)e(b)q(e)i (w)o(orth)e(the)i(trouble,)f(since)g(ev)o(en)h(the)f(standard)g(col-)365 1212 y(lectiv)o(e)g(ops)g(migh)o(t)e(b)q(e)i(excluded.)18 b(\(It)11 b(w)o(ould)f(also)g(con\015ict)h(with)g(Implemen)o(tation)365 1262 y(Note)k(1.])324 1345 y Fd(\017)20 b Fc(Group)d(formation)d(can)j(b)q(e) g(faster)h(when)f(more)f(information)d(is)k(pro)o(vided)g(than)365 1394 y(just)i(the)f(group)g(tag)g(and)g(ro)q(ot)g(pro)q(cess.)32 b(E.g.)e(if)17 b(all)g(mem)o(b)q(ers)g(of)g(the)i(group)365 1444 y(can)c(iden)o(tify)e(all)g(other)h(mem)o(b)q(ers)f(of)g(a)h(P-mem)o(b)q (er)f(group,)g(in)h(rank)g(order,)g(then)365 1494 y(O\(log)h(P\))h(time)e(is) i(su\016cien)o(t;)g(if)f(only)g(the)h(ro)q(ot)f(is)h(kno)o(wn,)f(O\(P\))h (time)e(ma)o(y)g(b)q(e)365 1544 y(required.)24 b(This)16 b(asp)q(ect)h (should)e(b)q(e)h(considered)h(b)o(y)f(the)g(groups)g(sub)q(committee)365 1594 y(in)e(ev)n(aluating)e(scalabilit)o(y)m(.)262 1731 y Fe(1.5)64 b(Implemen)n(tation)21 b(Notes)312 1822 y Fc(1.)f(T)m(o)f(generate)h(and)f (broadcast)g(a)g(new)h(con)o(text)f(v)n(alue:)28 b(Generate)20 b(the)g(con)o(text)365 1872 y(v)n(alue)9 b(without)g(comm)o(unicatio)o(n)e(b) o(y)i(concatenating)g(a)g(lo)q(cally)f(main)o(tained)f(unique)365 1921 y(v)n(alue)j(with)g(the)g(pro)q(cess)i(n)o(um)o(b)q(er)e(in)f(some)g (0..P-1)g(global)g(ordering)h(sc)o(heme.)17 b(This)365 1971 y(assumes)g(that)g(con)o(text)g(v)n(alues)f(can)h(b)q(e)h(signi\014can)o(tly) d(longer)i(than)f(log\(P\))h(bits)365 2021 y(for)i(an)f(application)f(in)o(v) o(olving)g(P)i(pro)q(cesses.)35 b(If)18 b(not,)h(then)h(a)e(serv)o(er)i(ma)o (y)d(b)q(e)365 2071 y(required,)d(in)f(whic)o(h)g(case)i(b)o(y)e(sp)q (eci\014cation)h(it)f(has)g(to)g(b)q(e)h(fast)g(\(comparable)e(to)h(a)365 2121 y(pt-pt)i(call\).)i(There)f(is)e(no)f(need)i(to)f(generate)i(a)d(new)i (con)o(text)g(for)e(the)i(broadcast)365 2171 y(since)20 b(it)e(can)h(b)q(e)g (co)q(ded)h(to)f(use)g(the)h(default)e(group)g(con)o(text)i(b)o(y)e(meeting)g (the)365 2220 y(paired-exact-matc)o(h)13 b(constrain)o(t.\))312 2303 y(2.)20 b(The)e(follo)o(wing)d(is)j(in)o(tended)g(only)f(as)h(a)f(sanit) o(y)g(c)o(hec)o(k)i(on)e(the)h(claim)e(that)i(this)365 2353 y(prop)q(osal)f(can)g(b)q(e)g(la)o(y)o(ered)g(on)f(MPI.)h(Impro)o(v)o(emen)o (ts)e(to)h(impro)o(v)o(e)f(e\016ciency)j(or)365 2403 y(to)c(use)h(few)o(er)f (con)o(texts)h(will)e(b)q(e)h(greatly)g(appreciated.)967 2574 y(5)p eop %%Page: 6 7 6 6 bop 365 307 a Fc(In)16 b(addition)f(to)h(a)f(default)h(group)f(con)o (text,)i(eac)o(h)f(group)g(descriptor)h(con)o(tains)f(a)365 357 y(\\group)f(partitioning)g(con)o(text")h(and)f(a)g(\\group)h(disbanding)e (con)o(text")j(that)e(are)365 407 y(obtained)h(and)g(broadcast)h(at)f(the)h (time)d(the)j(group)f(is)g(created.)26 b(In)16 b(the)h(case)g(of)365 457 y(the)f(INITIAL)g(group,)f(this)h(is)f(done)h(using)g(paired-exact-matc)o (h)e(co)q(de)j(and)e(an)o(y)365 506 y(con)o(text,)g(immediately)d(after)j (initialization)c(of)k(the)g(MPI)g(pt-pt)g(lev)o(el.)20 b(\(A)o(t)15 b(that)365 556 y(time,)c(no)g(application)f(co)q(de)i(will)e(ha)o(v)o(e)i (had)f(a)g(c)o(hance)i(to)e(issue)i(wildcard)e(receiv)o(es)365 606 y(to)j(mess)g(things)f(up.\))365 672 y(Group)g(partitioning)e(is)i (accomplished)f(using)h(pt-pt)g(messages)g(in)f(the)i(partition-)365 722 y(ing)k(con)o(text)h(of)f(the)i(curren)o(t)g(group)e(\(i.e.,)g(the)i(one) e(b)q(eing)h(partitioned\),)g(with)365 772 y(message)d(tag)h(equal)f(to)g (the)h(group)f(tag)g(pro)o(vided)h(b)o(y)f(the)h(application.)25 b(In)16 b(the)365 822 y(w)o(orst)g(\(least)g(scalable\))g(case,)g(the)h(ro)q (ot)e(of)g(the)i(new)f(group)f(m)o(ust)g(wildcard)g(the)365 872 y(source)22 b(TID.)d(\(This)h(violates)f(the)h(paired-exact-matc)o(h)g (constrain)o(t,)h(whic)o(h)f(is)365 922 y(wh)o(y)14 b(group)g(formation)d(m)o (ust)i(happ)q(en)h(in)g(a)f(sp)q(ecial)h(con)o(text.\))365 988 y(Group)k(disbanding)e(is)i(done)g(with)f(pt-pt)h(messages)g(in)f(the)h (group)f(disbanding)365 1038 y(con)o(text.)25 b(This)15 b(k)o(eeps)i(group)f (con)o(trol)f(from)f(b)q(eing)i(messed)g(up)f(no)h(matter)f(ho)o(w)365 1088 y(the)g(application)d(uses)j(wildcards.)262 1225 y Fe(1.6)64 b(Examples)262 1316 y Fa(\(T)l(o)15 b(b)q(e)h(pro)o(vided\))967 2574 y Fc(6)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 11:53:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07957; Fri, 19 Mar 93 11:53:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26990; Fri, 19 Mar 93 11:53:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 11:53:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26973; Fri, 19 Mar 93 11:53:02 -0500 Date: Fri, 19 Mar 93 16:52:24 GMT Message-Id: <10550.9303191652@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: delayed comments To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all I'm afraid that comments to Rik's Proposal V will be delayed until approx 6pm GMT, two hours later than promised. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 12:35:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08995; Fri, 19 Mar 93 12:35:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29197; Fri, 19 Mar 93 12:35:21 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 12:35:19 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29127; Fri, 19 Mar 93 12:34:35 -0500 Date: Fri, 19 Mar 93 17:34:23 GMT Message-Id: <10600.9303191734@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: comments arrive To: d39135@sodium.pnl.gov Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Rik Here are my comments of this afternoon. I hope they make some sense to you! Regards Lyndon General points -------------- 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am offering if you wish) should attempt to construct a more transparent presentation before circulation to whole committee, for the convenience of committee members. 2) I'm not a fan of much of this proposal, although I do indeed like some of the ideas which it introduces. [On the other hand, I'm not a great fan of all of the proposal which I wrote. I shall mail self criticism of my proposal, and may have to write amended or alternative proposal :-)] 3) I really like the way in which groups are something like ``frames'' in which contexts are created. This is conceptually much neater than duplication of groups. 4) I like the idea of pushing information into the group structure. I have a few qualms with the proposed details --- see specific points. 5) See ``Writing a server in the point-to-point layer of MPI in four easy steps'' at the foot of the message. Specific points --------------- Dealt with as LaTeX comments to body of text, appearing in the form %[Lyndon] % text of point for your navigational convenience. These are quite detailed. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- Writing a server in the point-to-point layer of MPI in four easy steps ---------------------------------------------------------------------- 1) Partition the INITIAL group into two groups. A singleton group, SERVER, and a group CLIENT which contains all of the other processes. 2) The single process in SERVER group records its TID. 3) The processes in INITIAL group allocate a context SERVICE which they remember either in the group cache or static data or something. 4) Use a broadcast in INITIAL group with ``sender'' as the one process which is also in SERVER group, and the ``receivers'' as the (many) processes which are also in CLIENT group, in the SERVICE context, in order to disseminate the TID of the server process. [Fanfare] a server process is in place as is a dedicated context for the purposes of messages required to implement the service. [Observation] the mpi point-to-point initialisation can do this automatically. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Mar 19 14:04:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10876; Fri, 19 Mar 93 14:04:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04631; Fri, 19 Mar 93 14:03:37 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 19 Mar 1993 14:03:35 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04623; Fri, 19 Mar 93 14:03:32 -0500 Date: Fri, 19 Mar 93 19:03:21 GMT Message-Id: <10723.9303191903@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: gone To: d39135@sodium.pnl.gov Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Rik I away home with the laptop, and off-line for the evening. If you want to discuss my comments, or make comments to me, or whatever, today then feel free to phone me at home if you are able. My homephone is + 44 31 557 0480. I am going to work on proposal I, to contract it to essentially the "Snir" model as originally discussed --- i.e. I'm going to delete the extended part at least. I never really did like retrofitting my requirements onto that. I am then going to write a proposal II' which separates contexts and groups, and meets requirements, and which I all along preferred in shape. I hope you don't mind me borrowing the number, I have annotated to indicate that this is not the original II :-) There will be much room for the group identifier cache in II'. I will mention this but not deal with it in any detail at this point. This is work I will do with the optimistic assumption that the "gang of three (four, five?)" will find this acceptable. I hope to mail out tomorrow afternoon GMT. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 12:08:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29146; Sun, 21 Mar 93 12:08:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20050; Sun, 21 Mar 93 12:08:12 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 12:08:09 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20042; Sun, 21 Mar 93 12:08:08 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03376; Sun, 21 Mar 93 11:02:21 CST Date: Sun, 21 Mar 93 11:02:21 CST From: Tony Skjellum Message-Id: <9303211702.AA03376@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu Subject: Re: Proposals V and I Cc: mpi-context@cs.utk.edu I will put my own doc in Latex, but I agree this is good. We will make one jointly authored "document", ala Rusty & Bill. More to follow [lots] - Tony From owner-mpi-context@CS.UTK.EDU Sun Mar 21 13:14:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29712; Sun, 21 Mar 93 13:14:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21849; Sun, 21 Mar 93 13:14:13 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 13:14:11 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21841; Sun, 21 Mar 93 13:14:10 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03573; Sun, 21 Mar 93 12:08:23 CST Date: Sun, 21 Mar 93 12:08:23 CST From: Tony Skjellum Message-Id: <9303211808.AA03573@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: My comments on Proposal V [tex format] Step 1. I am commenting directly on Proposal V. Step 2. I am comment on Lyndon's comments on Proposal V. This will come in a separate e-mail, to reduce [my] confusion, and hopefully yours. - Tony 1. My annotations to proposal V by Rik; %**** is my prefix \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %**** %**** Seems doable. %**** \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %**** %**** Seems doable. %**** \item There are restrictions that permit groups to be layered on top of pt-pt. %**** %**** I don't understand the word 'restriction' here. %**** Restriction of what. %**** \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. %**** %**** What does "fast" mean. %**** \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. %**** %**** OK, the above two items are identical to what Zipcode %**** provides in practice, but people have argued that groups %**** might be created/deleted more often in some apps, and %**** that these apps ought to be supportable %**** \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %**** %**** Yes, and I want at least 32-bits of message tag. %**** \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %**** %**** This could be difficult, in practice, if one mails a %**** message to one's own process, and MPI is smart enough %**** to optimize. %**** \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %**** %**** Sounds good. %**** \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %**** %**** In Zipcode 1.0, we allow multiple global operations %**** to be provided on a message-class (eg, grid-oriented messages) %**** The identifiers for these possible operations are user-specified %**** presently, but the "names" of the global operations are fixed %**** at compile-time. %**** %**** That means that there is O(1) time to find combine, fanout, send, %**** etc, on a group-wide scope. However, other operations cannot %**** be accessed in O(1) time (they are not in the opaque structure). %**** %**** The same mechanism used by Zipcode to allow multiple methods for %**** combine to be registered by the user, could also allow extensibility %**** just like Rik describes, with little effort. We use AVL trees. %**** %**** In fact, I will add this to Zipcode 1.x. Why say this? It is %**** not far from existing practice, and I have a lot of the machinery %**** in place already, and I am confident that it is useful. %**** \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %**** %**** This seems to be a serious flaw. It will have to be cached %**** on an LRU basis, with system/user/both specifying how much %**** caching is allowed (ie, how much unscalable memory use). %**** If the first time is expensive, OK, but not the Nth time. %**** \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). %**** %**** OK, I see no problem with this (so far). %**** \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %**** %**** NO. This violates the concept of context entirely. %**** (ie, an oxymoron ... contexts same, but still no need for %**** tag disambiguation...) %**** %**** Use the default group context to establish (cooperatively) %**** other contexts, and then use these. This is a seriously %**** bad feature, in my mind. %**** \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %**** %**** I do not think we should support the paired-exact-match thing. %**** \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) %**** %**** Concur. This suggests many more than "256" %**** \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] %**** %**** This is the Zipcode model. It could say loosely synchronous. %**** \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %**** %**** I don't understand the thread issue here. %**** \item Collective communication routines are called by all members of a group in the same order. %**** %**** Yes. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. %**** What does caching really imply here ??? Help. %**** \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %**** I think that contexts are really important in this case, %**** to keep things straight, but that non-blocking collcomm should %**** be omitted from MPI1 (cf, Geist). Sequencing supports %**** a sufficient disambiguation, as long as the entire group %**** is always the participant in operations. That is, you have %**** to form subgroups, with new contexts, to do global ops on %**** subsets. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %**** Why aren't servers needed to create contexts. Where do they %**** come from? If you rely on the fact that INITIAL will do %**** a loosely synchonous cooperative operation each time a new %**** context is needed, then a simple (easily implementable server, %**** or fetch-and-add remote access) is replaced by a more rigid %**** computation model. %**** %**** If we can get rid of this disagreement, me might be able to %**** reduce our total proposal space by one whole proposal. %**** \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %**** %**** If contexts are being used very dynamically, how are they being %**** assigned, kept, released, reissued without a server? Sorry if %**** I missed something, but I don't see it, without a restrictive %**** SPMD model of computation (Zipcode obviates its server for the %**** SPMD model, for instance). \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %**** %**** Can you clarify this with examples. %**** \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %**** I like this, of course. \item Communication both within and between groups seems conceptually straightforward. %**** Well, is point-to-point group oriented. Not. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %**** I think it is awkward. \item Communication between different groups may be considered awkward. %**** OK, but one can form a new group, as I have argued before. %**** Use the "awkward" pt2pt to get the right info shared between %**** group leaders, make the new group, use unawkward collective %**** operations on new group (with new context). \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %**** %**** This again is in practice, in Zipcode. %**** \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %**** %**** Sounds dangerous. What must application do to maintain %**** coherency, since group descriptors are opaque. %**** \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. %**** No, loosely synchronous process model, unless you mean %**** cooperation of INITIAL at all such join/leave steps. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %**** %**** No, you can't know the group or rank in group of sender. %**** If there were one context per group (isn't that so here?), %**** then all you need is the rank. With TID_TO_RANK_IN_GROUP %**** operation, this could be provided, but no wildcarding %**** or receipt selectivity could be done at this level. %**** \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %**** %**** No, it just does not provide guarantees that certain kinds %**** of applications will run OK. (ie, those that do group %**** creation/deletion relatively often). Zipcode has assumed %**** that such operations would be relatively seldom. Thus, I do %**** not quibble that this is a reasonable choice,but a fairer %**** way to say this is that it may be difficult to support such %**** applications. That reveals an issue to be studied more. %**** \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %**** %**** Perhaps they should do. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] %**** Dump this. \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %**** %**** No, a non-deterministic broadcast can be used, with a token. %**** This requires a token server. Again, implementable with fetch+ %**** add on most systems, or a light reactive server. %**** %**** Once the non-deterministic broadcast has finished, a fanin/collapse %**** is done to the original root, which then frees the token. %**** \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %**** %**** Why not just given in and allow the server. %**** I don't like the paired-exact-match constraint AT ALL. %**** \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) %**** %**** Seems OK, but why need the paired-exact-match thing again. %**** Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) %**** %**** Again, OK, but I want to see this work without the paired-exact- %**** match, if possible. Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. %**** %**** So, now, you have concurred with my (previously flamed) idea %**** that group construction/destruction should be realizable using %**** pt2pt, just like global operations should do. I like this %**** because 1) it is explicable to the implementor, 2) it allows %**** simple intitial implemtations, 3) it sets some ideas for how %**** much these things will cost [upper bound]. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} From owner-mpi-context@CS.UTK.EDU Sun Mar 21 13:22:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29866; Sun, 21 Mar 93 13:22:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22128; Sun, 21 Mar 93 13:22:00 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 13:21:59 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22120; Sun, 21 Mar 93 13:21:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03587; Sun, 21 Mar 93 12:16:22 CST Date: Sun, 21 Mar 93 12:16:22 CST From: Tony Skjellum Message-Id: <9303211816.AA03587@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: mini-proposal on layerability Dear Context subcommittee, This is something we should share with pt2pt, but I put it before you first.... (and to topologies, for their interest). To build effective layers in MPI, a reasonable receipt selectivity must be provided. As suggested by Jim, this can be accomplished by two integers as selectors for message wildcarding... dont_care care So, each receipt function uses the algorithm (received_rag & ~dont_care)& care With such a feature, layerability (eg, virtual topologies) could be much more easily added into MPI. Without this, (ie, accept or reject tag as ANY or only one specific), it is very difficult to layer. It is useful that wildcarding of sender is provided, and all-or-nothing wildcarding is fine for sender, since sender is specified by opaque TID. Example, wildcard receipt in a 3D grid usage of tags to implement a virtual topology. If a topology is embedded in a tag (p,q,r), then it is conveniently possible to do wildcarding on any dimension, given the care/dont_care bits. Another option would always be to multiple out the (p,q,r) triplet to give the rank-in-group. This could be stored in the tag instead, but severe multiplications/division needs would occur to determine a match (provided a general matching function were also contemplated). In short, a lot of flexiblity occurs with the care/dont_care bits, and I will ask for this feature, so that layerability can reasonably be claimed in MPI1. From owner-mpi-context@CS.UTK.EDU Sun Mar 21 13:44:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00158; Sun, 21 Mar 93 13:44:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22724; Sun, 21 Mar 93 13:43:47 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 13:43:46 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22716; Sun, 21 Mar 93 13:43:44 -0500 Date: Sun, 21 Mar 93 18:43:27 GMT Message-Id: <12975.9303211843@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Proposals V and I To: Tony Skjellum , d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, ranka@top.cis.syr.edu In-Reply-To: Tony Skjellum's message of Sun, 21 Mar 93 11:02:21 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu > I will put my own doc in Latex, but I agree this is good. We will make > one jointly authored "document", ala Rusty & Bill. Super. I thought about this and thought that until a proposal is chosen that goes through a report documenststyle would probably be easiest, so each proposal appears as a \chapter. I think we should probably attribute authorship and credits to the individual proposal chapters at least for now. > More to follow [lots] Goodie Goodie :-) Me too, lost more to follow. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:07:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00224; Sun, 21 Mar 93 14:07:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23503; Sun, 21 Mar 93 14:06:52 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:06:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23495; Sun, 21 Mar 93 14:06:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03668; Sun, 21 Mar 93 13:00:37 CST Date: Sun, 21 Mar 93 13:00:37 CST From: Tony Skjellum Message-Id: <9303211900.AA03668@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: my comments on Lyndon's comments on PropV %**** Step 2. My comments next to Lyndon's comments, critique of Rik's %**** proposal V. %**** %**** I have added my further remarks, with %**** %**** - Tony General points -------------- 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am offering if you wish) should attempt to construct a more transparent presentation before circulation to whole committee, for the convenience of committee members. %**** %**** I felt that things appear twice, because of summary (good) %**** and because of implementation notes at end (confusing) %**** 2) I'm not a fan of much of this proposal, although I do indeed like some of the ideas which it introduces. [On the other hand, I'm not a great fan of all of the proposal which I wrote. I shall mail self criticism of my proposal, and may have to write amended or alternative proposal :-)] %**** %**** Please be more specific. I am having a hard time understanding %**** why you really don't like it, Lyndon. If the process model %**** were a little less static, and servers were permitted (though %**** hopefully bounded in cost), I think we would have an excellent %**** proposal. %**** 3) I really like the way in which groups are something like ``frames'' in which contexts are created. This is conceptually much neater than duplication of groups. %**** %**** In practice, group subsetting will require groups to be copied, %**** otherwise, subgroups will unfairly be penalized by the size %**** of their ancestor. %**** 4) I like the idea of pushing information into the group structure. I have a few qualms with the proposed details --- see specific points. %**** %**** I have more confidence about this idea, and could demonstrate %**** by June/July time-frame in Zipcode. %**** 5) See ``Writing a server in the point-to-point layer of MPI in four easy steps'' at the foot of the message. %**** %**** This seems like a nice thing. %**** Specific points --------------- Dealt with as LaTeX comments to body of text, appearing in the form %[Lyndon] % text of point for your navigational convenience. These are quite detailed. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %**** %**** Alternatives, Lyndon? %**** \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %**** Me too. \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %**** %**** 1) It is not obvious that TIDs should be restricted to 32 bits. %**** 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %**** I favor a whole word for a context, despite other limits, %***** just to make things simpler. %**** %**** Internet addresses are going to get augmented from 32 to ??? bits %**** is it reasonable to assume that certain MPI implementations might %**** incorporate such internet addresses as TIDs (in future), %**** %**** Opacity is partially violated if we say how big the data type is??? \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %**** %**** where does this get us??? %**** Remember, we have to choose on some things, so we can have something %**** to present in Dallas. Is there an important difference here? %**** %**** TIDs are global entities. Is structure assumed to be global; %**** in a truly opaque system, some TID component would have to be %**** fixed, but the rest could vary structurally... %**** \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %**** %**** Yes, contexts are global; group identifiers are just pointers %**** typically, to data structures, describing %**** %**** 1) a groups context %**** 2) group members and their ranks (mappings, inverses, %**** cached, hashed, unscalably stored, etc) %**** 3) TID-to-rank map and inverse (see possibilities in 2) %**** 4) A set of fixed global operations, accepted as standard, %**** an accessible in O(1) time. Possibly, each %**** such operation should be a method, so that %**** a parameter block can be passed with it. Zipcode %**** supports the Method type to do this. %**** This is effectively a cache for some parts of item #5 %**** ... %**** 5) An AVL or similar tree of extensible operations. %**** New operations are registerable by the user. These %**** tags are unique within a group, a specify an operation %**** i) pre-defined by MPI (in which case it can be cached %**** in 4 %**** ii) alternative operations (even if they do something %**** standard, that are wanted to be accessed by %**** name) This name is group unique. %**** %**** A mechanism for DO_METHOD_FROM_GROUP(name,....) %**** or GET_METHOD_FROM_GROUP(name,...) %**** and SET_METHOD_IN_GROUP(name,...) are clearly needed. %**** Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %**** Yes, that is included, I believe, in concept. %**** \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %**** %**** Stripping is extremely controversial aspect, and arbitrary. %**** If the recipient has the methods with the same name, then %**** a new rendezvous could be accomplished at the far end \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %**** %**** Yes, these are all options. Fastness seems to be an important %**** issue. If translation is very expensive, none of the "good" %**** features will be used. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %**** Again, I think this should be eliminated, and all references %**** to this idea should be expunged. It denies the context's %**** ability to manage messages. \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. %**** I agree that context allocation has to be done in sequence. %**** That is why I am in favor of providing calls that allow %**** groups to get numerous contexts at creation, and then %****cooperatively, but potentially without further communication %**** divide them(as they build subgroups, for instance). %**** %**** I see these as services to be used in building virtual topology %**** features, which will then be more widely used by users of MPI. %**** \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %**** %**** Do we get anything by having the root process? %**** \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %**** %**** I think that non-blocking collcomm is moribund in MPI1 or %**** else MPI1 is moribund. :-) %**** \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %**** Too many negatives here for me to parse :-) \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %**** %**** Context allocation need not be free provided it can be made cheap, %**** or cheap enough. %**** %**** If one knows one will need several, then a single call could %**** provide such contexts, amortizing overhead. This is likely when %**** bulding grids (ie, virtual topologies) in Zipcode, so it is %**** true in existing practice. %**** %**** One should recognize the need for layering virtual top. calls %**** on top of these calls, then these calls may appear painful, %**** but perhaps they would be less used. Some users will use the %**** provided virtual topology calls, others will prefer their own. %**** Both will have equal power (see also,separate note on layerability). %**** %**** If getting N contexts is a send-and-receive, plus a reactive server, %**** then this is reasonably light weight,provided that hundreds of %**** messages, or global operations ensue thereafter. We can know in %**** advance how heavy weight the context server will be. %**** %**** if an implemention can use some locations of remote memory, with %**** fetch and add, or locks, to achieve contexts, then this is even %**** cheaper, in principle. %**** %**** Despite Jim's earlier insistence that context numbers be kept to %**** 256 or so, I think that this number should be much larger, so that %**** much less efort goes into returning contexts, and so on, except %**** occasionally, by processes. Otherwise, a new kind of overhead, %**** get-rid-of-context-because-I-am-out ensues, or programs block %**** until contexts become available, offering the possibility of %**** deadlocks. %**** \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %**** %**** Unscalability (of a limited form) should be permitted/selectable %**** by user, to use as much per-node memory as the user wants, to reduce %**** communication. %**** \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. %**** I see,but what he means here is that there is no unpredictable %**** translation cost because we do not write (group,rank) in pt2pt %**** calls. So, there is some validity to the statement. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %**** Yes, that is why I have one in Zipcode. I wish Zipcode were %**** on netlib today, so you could try it. Well, we are writingthe %**** manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %**** %**** I lost something here, but I agree with the sense. The word %**** seems is subjective,and should disappear from our discussions, %**** as much as seems prudent, anyway :-) \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %**** %**** Yes,awkward, because it is nice to escape the TID realm and %**** work within the (albeit simple) abstraction of group,rank. %**** When layering virtual topologies on this, it would be so nice %**** to write them to a group,rank syntax, not enforcing TID mappings %**** everywhere. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %**** Indeed. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %**** Yes, and I think that loosely synchronous operations can maintain %**** coherency, in practice. That is, no operations that modify the %**** group descriptors (other than cached lookup info) are permitted, %**** without loose synchronization. %**** This is nasty in that is would prohibit sending descriptors to %**** processes not part of the group, so it is a clear trade-off. %**** Perhaps such send-to-non-group-member operations could stipulate %**** that this group information is somehow ephemeral, and that they %**** need to join a new group to keep useful information over time??? %**** \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %**** %**** Yes, that is exactly what Zipcode was written to avoid. The %**** user wants help managing things like this!!!! %**** %**** The search, if any, must be MPI-supported, and as efficient as %**** possible (eg, AVL trees, hash, partial hash with exceptions). %**** % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %**** %**** This is why I remain a strong advocate of (group,rank) %**** addresssing in pt2pt. %**** \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %**** %**** I addressed this in my Step-1 remarks. Please see that. %**** \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %**** %**** Thank you. These points are both well taken (ie these two paragraphs) \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %**** Also,see what I wrote in my Step-1 comments. %**** I believe O(log(P)) is still possible. %**** \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %**** Please see my Step-1 comments. \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- Writing a server in the point-to-point layer of MPI in four easy steps ---------------------------------------------------------------------- 1) Partition the INITIAL group into two groups. A singleton group, SERVER, and a group CLIENT which contains all of the other processes. 2) The single process in SERVER group records its TID. 3) The processes in INITIAL group allocate a context SERVICE which they remember either in the group cache or static data or something. 4) Use a broadcast in INITIAL group with ``sender'' as the one process which is also in SERVER group, and the ``receivers'' as the (many) processes which are also in CLIENT group, in the SERVICE context, in order to disseminate the TID of the server process. [Fanfare] a server process is in place as is a dedicated context for the purposes of messages required to implement the service. [Observation] the mpi point-to-point initialisation can do this automatically. %**** Zipcode's postmaster general works in this way, more or less. %**** - Tony ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:37:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00298; Sun, 21 Mar 93 14:37:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24385; Sun, 21 Mar 93 14:37:22 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:37:21 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24377; Sun, 21 Mar 93 14:37:19 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03706; Sun, 21 Mar 93 13:31:14 CST Date: Sun, 21 Mar 93 13:31:14 CST From: Tony Skjellum Message-Id: <9303211931.AA03706@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu Subject: my comments on Lyndon's Proposal I %**** %**** Below are my comments on Lyndon's Proposal I. In the next paragraph %**** I note that we are actually converging to less than N proposals, though %**** I have not seen Lyndon's new II proposal yet. Does it exist now? %**** %**** To achieve a reasonable presentation at Dallas, we can have multiple %**** proposals still on table (I think this a fair, well-thought-out %**** approach, but if we can condense some, let's do it). %**** %**** %**** After reading V, making my comments, and making my comments %**** in addition to Lyndon's comments, I am convinced we can %**** advance proposal V into something that is acceptable in the %**** III/IV mold, without a further III/IV proposal. Rik's %**** dropping of the static context concept has simplified our %**** group's efforts considerably, and I cannot disambiguate my %**** III/IV proposal from Proposal V, given the Lyndon and Tony %**** provisos and suggested improvements. This is not a wimp out %**** on my part. I do not see benefit of advancing something that %**** will look 90% like Proposal V at this point, and 97% like it %**** if the Lyndon/Tony comments obtain in it (which they would if %**** I wrote it now). %**** I would prefer that we hone Proposal V. If Rik wants to keep %**** his style, then I propose that Proposal III/IV become exactly %**** what I just said, a reworking of V + comments. %**** However, I think this will cause an unnecessary delay and %**** digression just to achieve details. Instead, we might pull %**** such choices into Proposal V at this time (server vs. no server). %**** to make what is common in our "approaches" more obvious. %**** - Tony %**** %**** (PS, Lyndon: Rik/Tony/Lyndon are authors of whole paper, because %**** we are all working on this, and because one of us (ie, me) will %**** make the final document cohesive, create a unified style, format, %**** set of meanings. This is the nature of collaboration. I do %**** not propose to include our other co-conspirators on such a document's %**** list of authors, as there has been minimal input from them. I %**** Think that Rusty/Bill and Rik/Tony/Lyndon are operating equivalently.) \documentstyle{report} \begin{document} \title{``Proposal'' I for MPI Communication Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Lyndon J. Clarke % March 1993 % \chapter{Proposal I} \section{Introduction} This chapter proposes that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The proposal reflects the observation that an instance of a module in a parallel program typically operates within a group of processes, and as such any communication contexts associated with an instance of a module also bind the semantics of process groups. This chapter makes a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The proposals say nothing about use of message tags. It is assumed that these will be a bit string, expressed as an integer in the host langauge (i.e. ANSI C or Fortran~77, in the first instacnce. These proposals should be viewed as a collection of recommendations to other subcommittees within MPI, primarily the collective communications subcommittee, the point-to-point communications subcommittee and the process topologies subcommittee. Concrete syntax is given in the style of the C language for purposes of discussion only, and should be viewed as an example of possible syntax as detailed syntax of described operations is the responsibility of the language bindings subcommittee %----------------------------------------------------------------------% % BEGIN "Basic Proposal" % \subsection{Basic Proposal} The main features of the basic proposal are: \begin{itemize} \item Process identifiers and group identifiers have opaque values expressed as an integer type in the base language. Semantics for passing process identifiers in a message are defined, whereas semantics for passing group identifiers in a message are not defined. \item Group creation and destruction are concerted, synchronous, actions on the part of the membership of the group concerned. %**** %**** loosely synchronous %**** \item Groups are {\it static\/} in that no provision is made for modification of the membership of a group over the lifetime of the group. Dynamic group operations such as {\it resize\/} can be effected by destruction of the existing group and creation of a new resized group. %**** Why not by making an off-spring, instead of destructively!!! \item Group creation is effected by four means: explicit definition of the membership of a group; partition of an existing group into one or more distinct subgroups; identical duplication of an existing group; topological duplication of an existing group. \item Point-to-point communication provides for intra-group communication, and makes no provision for inter-group communication. \item Collective communication provides for operations which are concerted actions on the part of members of one group, and makes no provision for operations which are concerted actions on the part of members of multiple groups. \end{itemize} \subsubsection{Process and Group Identifiers} A {\it process identifier\/} is an opaque reference to an object which is a single process. A process identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on process identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). Each process has exactly one process identifier. MPI should provide a procedure which allows the user to determine the process identifier of the calling process. For example, {\tt mypid = mpi\_mypid()}. The identifier of the {\it null process\/}, defined to be a process which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_PID\_NULL} in this proposal. With the single exception of the identifier of the null process, the value of a process identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a process P then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a process identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the received process identifier is {\it invalid\/}. MPI will provide a mechanism which allows a process identifier to be passed in a message in such a manner that the received identifier is valid. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a process identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_pid(bd, \&pid)}. Transmission of a process identifier using this mechanism returns to the recipient a process identifier which is valid for use in the MPI operations described below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a process identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_pid\_invalidate(pid)}. This is an error if {\tt pid} is {\tt MPI\_PID\_NULL}, or if {\tt pid} is the identifier of the calling process. It is further proposed that MPI provide a process identifier registry service. This service allows any process to register its own process identifer by name, and deregister its process identifier. The service allows any process to determine whether a name has been registered without blocking the calling process, and to map that name into a valid process identifier with the possibility of blocking the calling process. Use of this service is not mandated, and components of programs which do not require this service are not expected to make use thereof. %**** I don't get all of this. Why? %**** A {\it group identifier\/} is an opaque reference to an object which is a group of processes. A group identifier is expressed as an integer in the host language and has a value defined by the system. The only meaningful host language operations on group identifiers are assignment ({\tt =}), equality ({\tt ==}) and inequality ({\tt !=}). The identifier of the {\it null group\/}, defined to be a group which cannot exist, is defined as a named constant and shall be referred to as {\tt MPI\_GID\_NULL} in this proposal. With the single exception of the identifier of the null group, the value of a group identifier is {\it process local\/}, meaning that if two processes A and B know the identifier of a group G then the relationship between the values of the identifiers known to A and B is undefined. The user can pass the value of a group identifier in a message, since it is an integer type in the host language, however the recipient of the value cannot make defined use of that value in the MPI operations described below --- the identifier is {\tt invalid\/}. MPI will not provide a mechanism which allows a group identifier to be passed in a message in such a manner that the received identifier is valid. %**** Then why allow it to be passed? %**** The canonical representation of a group is an array of distinct process identifiers, although it may be possible to use hashing functions with lower space complexity and marginally higher time complexity. A group is {\it static\/}, in that it's membership may not change over the lifetime of the group. There is a well defined {\it size\/} of a group, and MPI will provide a procedure which allows the user to determine the size of a group. For example, {\tt size = mpi\_grp\_size(gid)}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. There is a well defined {\it rank\/} of a process within the group, i.e. the position in the representational array at which the process identifier is held. MPI will provide a procedure which allows the user to determine the rank of the calling process within a group. For example, {\tt mpi\_grp\_myrank(gid)} returns the rank of the calling process within the group referred to by {\tt gid}. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. MPI should provide a procedure to determine the member rank of a process within a group given the group identifier and a valid process identifier, e.g. {\tt rank = mpi\_grp\_rank(gid,pid)}. This procedure may ``fail'' if the process identified by {\tt pid} is not a member of the group identifier by {\tt gid}, and can be used to determine whether a given process is a member of a given group. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. %**** %**** Why an error, why not just fail? %**** MPI should provide a procedure to determine the process identifier of a group member given the group identifier and member rank, e.g. {\tt pid = mpi\_pid(gid,rank)}. This procedure should validate the returned process identifier. This procedure is an error if {\tt gid} is {\tt MPI\_GID\_NULL}. \subsubsection{Group Creation and Destruction} MPI will provide four methods for creation of groups, and a procedure to destroy an existing group. A group may be created by explicit definition of the pids of the members of the group. For example, {\tt gid = mpi\_grp\_definition(npids,pids)} where {\tt gid} is the identifier of the newly created group, {\tt npids} is the number of processes in the new group, and {\tt pids} is an array containing the (valid) process identifiers of the processes in the group. This procedure must be called my all members of the group, and does not return until all members have made the call. A group may be created by identical duplication of an existing group. For example, {\tt gidb = mpi\_grp\_duplication(gida)} where {\tt gidb} is the group identifier of the newly created group and {\tt gida} is the identifier of an existing group. The created group inherits all properties of the source group, including any topological properties. This operation has the same synchronisation properties as creation of group by definition. %**** It is not obvious to me that we want to enforce topology at this %**** juncture. However, we could register topology information in %**** the extensible structure strategy of Proposal V. %**** A group may be created by topological duplication of an existing group. Details of topological groups are under consideration within the process topologies subcommittee and will not be further discussed here. A group created by topological duplication inherits the size of the source group, and also inherits the membership list of the source group although the list may be ordered differently in the created group. These operations have the same synchronisation properties as creation of group by identical duplication. Where groups have additional topological attributes MPI should also provide procedures which allow the user to determine such attributes. One or more groups may be created by partition of an existing group into distinct subgroups by key. For example, {\tt gidb = mpi\_grp\_partition(gida, key)} where {\tt gidb} is the group identifier of the newly created group corresponding to the given {\tt key} and {\tt gida} is the group identifier of the group which is being partitioned according to the {\tt key} values supplied. MPI should define a named constant which is a {\it null\/} {\tt key} value, for example {\tt MPI\_KEY\_NULL}, in order that members of the parent group can choose not to become members of any child group, and in which case the procedure should return {\tt MPI\_GID\_NULL}. Groups created by partition share the same ordering of process member ranks as the parent group. This operation synchronises the members of the parent group, and therefore implicitly synchronises the members of the created group(s). A group may be destroyed, e.g. {\tt mpi\_grp\_deletion(gid)}, which destroys the group identified by {\tt gid}. This operation synchronises the group members, and invalidates the group identifier. \subsubsection{Point-to-point communication} There are arguments either way for point-to-point routines which accept a {\tt (gid, rank)} pair, and for routines which accept a {\tt pid}. This proposal supports and seperates both approaches to addressing and selection within the same syntax, avoiding the introduction of sets of procedures to handle each case. In order to provide expressive power to intra-group communication, point-to-point communication should accept a {\tt (gid, rank)} pair, where {\tt gid} is a valid group identifier and {\tt rank} is a member rank within the group identified by {\tt gid}. In message addressing the sender specifies the {\tt rank} and {\tt gid} of the receiver. In message selection the receiver specifies the {\tt rank} and {\tt gid} of the receiver. The sender and receiver must both specify the same {\tt gid} in order for a match to occur. The {\tt gid} field is not allowed to take a {\it wildcard\/} value in message selection. The {\tt rank} field is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_RANK\_WILD}, which will match with any rank. One is encouraged to visualise a seperate message queue or port at each process for each group of which that process is a member (and indeed this may be an advantageous implementation feature). %**** But not a required feature of an implementation! In order to accommodate process identifier based addressing and selection into the same syntax, this proposal advocates that point-to-point communication should also accept the null group ({\tt gid = MPI\_GID\_NULL}), in which case the {\tt rank} is interpreted as a valid {\tt pid}. The {\tt pid} is allowed to take a {\it wildcard\/} value in message selection, e.g. {\tt MPI\_PID\_WILD}. The point-to-point section should provide a procedure which allows a recipient to recover the process identifier of the sender. The discussion of matching above extends to this case, and one is also encouraged to visualise a separate message queue or port for messages referred to by {\tt MPI\_GID\_NULL}. In the case of process identified wildcard receive, the process identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the process identifier must be performed by the MPI implementation, in order that the recipient is returned a valid process identifier, else the returned identifier is of little or no use to the recipient. %**** %**** I do not understand the usefulness or formal need for all this %**** validation and invalidation of process identifiers. Why, where %**** did it come from, what does it get us? How can this be related %**** to anything I have seen before? %**** \subsubsection{Collective communication} Collective communication operations within MPI should be restricted to the scope of a single group. It will be sufficient for these procedures to accept a group identifier, and possibly a message tag in order to distinguish multiple outstanding operations within the same group. These procedures must not accept {\tt MPI\_GID\_NULL}. %**** Why not, but do nothing... It is not possible to determine whether this is strategy allows all of the MPI collective communication routines to be written in terms of MPI point-to-point routines without loss of generality, since the set of collective communication routines is not yet determined. This proposal takes the view that it is the responsibility of the collective communications subcommittee to determine whether such a goal is desirable, and if so to describe procedures which comply with this goal. %**** Check, but it is desirable that they be so writable, so we will %**** have to watch. % % END "Basic Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Extended Proposal" % \subsection{Extended Proposal} The main additional features of the extended proposal are: \begin{itemize} \item Semantics for passing a group identifier in a message are defined, and a group registry is introduced. \item Point-to-point communication is extended to allow inter-group communication without syntactic intrusion on intra-group communication. \item Collective communication operations involving a concerted action on the part of members of two or more groups is suggested. \end{itemize} \subsubsection{Process and Group Identifiers} The extended proposal says nothing more about process identifiers than the basic proposal. The extended proposal defines a mechanism by which a group identifier may be passed in a message in such a manner that the received identifier is valid, and the mechanism should be analagous to that which allows process identifiers to be transmitted. It is proposed that this shall be integrated with the buffer descriptor mechanism (proposed by Bill Gropp and Rusty Lusk), by addition of a procedure which places a logical reference to a group identifier into the buffer descriptor, e.g. {\tt mpi\_bd\_gid(bd, \&gid)}. Transmission of a group identifier using this mechanism returns to the recipient a group identifier which is valid for use in the MPI operations described above and below, and as further qualified below. This transmission may side effect state in the implementation of MPI at the recipient, and in particular may reserve state at the recipient. MPI will provide a procedure which invalidates a group identifier, allowing the implementation of MPI to recover reserved state, e.g. {\tt mpi\_invalidate\_gid(gid)}. This is an error if {\tt gid} is {\tt MPI\_GID\_NULL}, or if {\tt gid} is the identifier of a group of which the calling process is a member. %**** I understand the idea here, but not all the details. Can this %**** be justified/exemplified/simplified? %**** It is further proposed that MPI provide a group identifier registry service. This service allows any group to register its own group identifer by name, and deregister its group identifier. The service allows any group to determine whether a name has been registered without blocking the calling group, and to map that name into a validated group identifier with the possibility of blocking the calling group. The group registry procedures should synchronise the calling group, permitting more efficient implementations than asynchronous operations. The definition of a valid group identifier is extended to include all groups of which the process is a member and all those which are validated by one of the above mechanisms. The small suite of procedures which map between process identifiers, group identifiers and member ranks are defined to work with valid group and process identifiers. \subsubsection{Point-to-point communication} The point-to-point communication syntax and semantics described in the basic proposal do not extend directly to the case of inter-group communication. The reason is quite simple --- the sender and receiver do not supply the same group since the group of the sender and the group of the receiver are different. The basic approach is to express point-to-point communication in terms of a triplet {\tt (localGroup, remoteGroup, remoteRank)}. In this notation the sender specifies the sender group identifier, then the receiver group identifier and finally the receiver rank. The receiver specifies the receiver group identifier, then the sender group identifier, and finally the sender rank. The {\tt localGroup} field may not take a wildcard value, corresponding directly to the rule that the group in the basic proposal may not take a wildcard value. The {\tt remoteGroup} field may take a wildcard value, e.g. {\tt MPI\_GID\_WILD}, which matches with any group. The {\tt remoteRank} may also take a wildcard value, as in the basic proposal. The point-to-point section should provide procedures which allow a message recipient to recover the group and rank of the sender. In the case of {\tt remoteGroup} wildcard receive, the group identifier recovered by the receiver may be unknown to the receiver. It is proposed that an implicit validation of the group identifier must be performed by the MPI implementation, in order that the recipient is returned a valid group identifier, else the returned identifier is of little or no use to the recipient. In order to accomodate inter-group addressing and selection into the framework of the basic proposal, the extended proposal suggests a careful redefinition of the {\tt gid} discussed in the basic proposal. With careful presentation this redefinition need not intrude conceptually on the basic proposal, although I shall give a less careful description here, suggestive of two different flavours of presentation. In the basic proposal, the {\tt gid} was formally a reference to a group. In the extended proposal the {\tt gid} is formally composed of references to two groups, and can be thought of as a shorthand notation for {\tt (localGroup, remoteGroup)}. %**** I dislike this intensely. There should be a group-pair data %**** structure. Group is never a pair of sub-groups. It is a %**** bad idea. This is all to get around changing syntax, no? The identifier of the null group, a {\it null\/} identifier, is a valid identifier which is formally composed of a pair of references to the null group, and may be used in the fashion described in the basic proposal. The group creation functions provide a symmetric, or {\it unary\/}, identifier formally composed of two references to the same group. This group is {\it local\/} since the process in question is a member of the group. An identifier which composes a pair of references to a local group is logically identical to a group identifier as implied in the basic proposal, and may be used for point-to-point and collective communications in an identical fashion. The group identifier transmission and registry lookup procedures also provide a symmetric, or {\it unary\/}, identifier which again is composed of a pair of references to the same group. This group is {\it remote\/} when the process is not a member of the group, or is local when the process is a member of the group. An identifier which composes a pair of references to a remote group is logically identical to the identifier a remote group as implied above, and is not valid for either point-to-point or collective communications. Inter-group communication is effected by use of assymetric, or {\it binary} identifiers, composed of references to two different groups, the first of which is local and the second of which is either local or remote. Such identifiers are constructed by the use of a ``glob'' operation, which returns an identifier referencing the first operand as the local field and the second operand as the remote field. The glob is defined to be an error unless: both operands are symmetric (unary); the first operand refers to a local group. The identifier returned by a glob is valid for point-to-point communciation and invalid for collective communication. The point-to-point communication section should provide a procedure to determine the group identifiers object of a completed receive. Procedures which ``unglob'' an assymetric (binary) group identifier should be provided, returning the local and remote fields as valid identifiers. This can be viewed as a continuation of a generalised approach to point-to-point communication began in the basic proposal, in which operation are specified in terms of object identifiers and instance identifiers where objects are composed of multiple instances. The nature of the instance identified, and the semantics of the operation, are dependent of the nature of the object identified, exploiting some genericity. \begin{center} \begin{tabular}{lll} object identifier & instance identifier & action \\ \hline null group identifier & basic process identifier & basic communication \\ unary group identifier & local process rank & intra-group communication \\ binary group identifier & remote process rank & inter-group communication \\ \end{tabular} \end{center} \subsubsection{Collective communication} There are a number of collective communication operations which logically extend over two or more groups. Some examples of two-group collective communications which are common in the simple host-node programming model are: \begin{itemize} \item {\it Broadcast\/} is an implicitly assymetric operation. There is exactly one sender and there are many receivers, each of which receives an identical message. The sender is a member of a singleton group G and the receivers are members of a group H. \item {\it Scatter\/} is an implicitly assymetric operation. There is exactly one sender and there are may receivers, each of which receives a different message. The sender is a member of a singleton group G and the receivers are members of a group H. \item There is a variant of {\it gather\/}, which returns the gathered data to a single process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \item There is a variant of {\it reduce}, which returns the reduced data to a sinle process, which is an implicitly assymetric operation. There is exactly one receiver and there are many senders. The receiver is a member of a singleton group G and the senders are members of a group H. \end{itemize} Other patterns arise in ``process'' graphs where each ``process'' is allowed to be parallel. For example: \begin{itemize} \item {\it all-to-all\/} communciation in which the senders and receivers are distinct processes. The senders are members of a group G and the receivers are members of a group H. The two groups G and H need not be of the same size. \end{itemize} The assymetric (binary) group identifier objects described for point-to-point communications in this extended proposal are immediately suitable for each of the two-group collective communication operations described. % % END "Extended Proposal" %----------------------------------------------------------------------% %**** So, I gather that a set of groups is passable to a collcomm, %**** and a pair is passable to a pt2pt. That is neat, but it should %**** still be a separate data structure, with separate calls than %**** the intra-group version (at least for the pt2pt calls). %**** \section{Conclusion} This chapter has propose that ordered groups are used to provide communication contexts, and that communication contexts do not appear independently of process groupings. The chapter made a basic proposal which provides intra-group communication but does not provide inter-group communication, and an extended proposal which also provides inter-group communication. The basic proposal provides expressive semantics for the case of intra-group communication such as arises in program which compose data driven parallelism, and is closely related to that which was discussed by Marc Snir at the February meeting in Dallas, and builds on discussions which have taken place in various subcommittees. The key additional features are: point-to-point communication can also be expressed in terms of process identifiers; a process registry service was added, use of which is optional. The extended proposal adds expressive semantics for the case of inter-group communication such as arises in programs which compose combinations of data and function driven parallelism. This functionality has been constructed in such a manner that there is no syntactic or performance intrusion on the content of the basic proposal, and the additional conceptual content can be presented seperately from a presentation of intra-group communication. % % END "Proposal I" %======================================================================% \end{document} >------------------------------ Cut Here ------------------------------< /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:47:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00323; Sun, 21 Mar 93 14:47:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24683; Sun, 21 Mar 93 14:46:49 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:46:48 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24675; Sun, 21 Mar 93 14:46:45 -0500 Date: Sun, 21 Mar 93 19:46:42 GMT Message-Id: <13090.9303211946@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal I - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is LaTeX of Proposal I, credits entirely to Marc Snir, 15 minutes earlier than previously advertised. PostScript to follow. My sincere apology to everyone who has prepared comments on Proposal I++. Some of these comments will carry through to Proposal II, which will be sent out shortly. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal I\\MPI Context Subcommittee} \author{Marc~Snir} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Written by Marc Snir % Edited by Lyndon J. Clarke % March 1993 % \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \chapter{Proposal I} {\it Editorial Note: This chapter is the proposal of Marc~Snir which also appears in the working documents of the point-to-point subcommittee (March 15) and collective communication subcommittee (March 16). This is a minor edit of the \LaTeX\ source of the point-to-point document provided by Marc. The edit was performed by Lyndon~Clarke.} \section{Contexts} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \subsection{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \subsection{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \subsection{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \subsection{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \section{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \subsection{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } % % END "Proposal I" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:49:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00329; Sun, 21 Mar 93 14:49:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24720; Sun, 21 Mar 93 14:48:46 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:48:45 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24711; Sun, 21 Mar 93 14:48:36 -0500 Date: Sun, 21 Mar 93 19:48:30 GMT Message-Id: <13097.9303211948@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal I - PostScript To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is PostScript of Proposal I, credits entirely to Marc Snir, 15 minutes earlier than previously advertised. LaTeX preceeded. ---------------------------------------------------------------------- %!PS-Adobe-2.0 %%Creator: dvips 5.495 Copyright 1986, 1992 Radical Eye Software %%Title: context-i.dvi %%CreationDate: Sun Mar 21 18:30:21 1993 %%Pages: 7 %%PageOrder: Ascend %%BoundingBox: 0 0 596 842 %%EndComments %DVIPSCommandLine: dvips context-i %DVIPSSource: TeX output 1993.03.21:1810 %%BeginProcSet: tex.pro %! /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N} B /TR{translate}N /isls false N /vsize 11 72 mul N /@rigin{isls{[0 1 -1 0 0 0] concat}if 72 Resolution div 72 VResolution div neg scale isls{0 Resolution vsize 72 div mul TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{ CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N} B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook} if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255 {IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V{}B /RV statusdict begin /product where{ pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{ gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M} B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{ p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 39158280 55380996 1000 300 300 (/a/obsidian/disk/home/u36/lyndon/mpi/context-i.dvi) @start /Fa 9 122 df<00E00001F00001F00001B00001B00003B80003B80003B800031800071C00071C 00071C00071C00071C000E0E000E0E000FFE000FFE001FFF001C07001C07001C07007F1FC0FF1F E07F1FC013197F9816>65 D76 D<003F00007F00003F0000070000070000070000070003 C7000FF7001FFF003C1F00780F00700700E00700E00700E00700E00700E00700E00700700F0070 0F003C1F001FFFE00FE7F007C7E014197F9816>100 D<03E00FF81FFC3C1E780E7007E007FFFF FFFFFFFFE000E000700778073C0F1FFE0FFC03F010127D9116>I<018003C003C0018000000000 000000007FC07FC07FC001C001C001C001C001C001C001C001C001C001C001C001C07FFFFFFF7F FF101A7D9916>105 D<7E0000FE00007E00000E00000E00000E00000E00000E7FE00E7FE00E7F E00E0F000E1E000E3C000E78000EF0000FF0000FF8000FBC000F1E000E0E000E07000E07807F87 F0FFCFF07F87F01419809816>107 D<7E3C00FEFE007FFF000F87800F03800E03800E03800E03 800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01512809116>110 D<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001F00000E00001E00001F00003B800073C 00071C000E0E007F1FC0FF3FE07F1FC013127F9116>120 D<7F1FC0FF9FE07F1FC01C07000E07 000E0E000E0E00070E00071C00071C00039C00039C0003980001B80001B80000F00000F00000F0 0000E00000E00000E00001C00079C0007BC0007F80003F00003C0000131B7F9116>I E /Fb 11 121 df<01C00003E00003E0000360000360000770000770000770000770000630000E 38000E38000E38000E38000E38001FFC001FFC001C1C001C1C003C1E00380E00FE3F80FE3F8011 177F9614>65 D76 D<1FC0007FF000707800201800001C00001C 0007FC001FFC003C1C00701C00E01C00E01C00E01C00707C003FFF800F8F8011107E8F14>97 D<03F80FFC1C1C380870006000E000E000E000E00060007000380E1C1E0FFC03F00F107E8F14> 99 D<07E00FF01C38301C700CE00EE00EFFFEFFFEE00060007000380E1C1E0FFC03F00F107E8F 14>101 D107 D110 D<07C01FF03C78701C701CE00EE00EE00E E00EE00EE00E701C783C3C781FF007C00F107E8F14>I 114 D<030007000700070007007FFCFFFC07000700070007000700070007000700070E070E070E 070C03FC00F00F157F9414>116 D<7E3F007E3F001E38000E780007700007E00003E00001C000 03C00003E0000770000E78000E38001C1C00FE3F80FE3F8011107F8F14>120 D E /Fc 1 16 df<07801FE03FF07FF87FF8FFFCFFFCFFFCFFFCFFFCFFFC7FF87FF83FF01FE007 800E107E9013>15 D E /Fd 48 123 df<00FC7C0183C607078E0607040E07000E07000E07000E 07000E07000E0700FFFFF00E07000E07000E07000E07000E07000E07000E07000E07000E07000E 07000E07000E07000E07000E07007F0FF0171A809916>11 D<00FC000182000703000607000E02 000E00000E00000E00000E00000E0000FFFF000E07000E07000E07000E07000E07000E07000E07 000E07000E07000E07000E07000E07000E07000E07007F0FE0131A809915>I<007E1F8001C170 400703C060060380E00E0380400E0380000E0380000E0380000E0380000E038000FFFFFFE00E03 80E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E0380E00E 0380E00E0380E00E0380E00E0380E07F8FE3FC1E1A809920>14 D<00800100020004000C000800 18003000300030006000600060006000E000E000E000E000E000E000E000E000E000E000600060 0060006000300030003000180008000C00040002000100008009267D9B0F>40 D<8000400020001000180008000C00060006000600030003000300030003800380038003800380 0380038003800380038003000300030003000600060006000C0008001800100020004000800009 267E9B0F>I<60F0F07010101020204080040B7D830B>44 DI<60F0F060 04047D830B>I<60F0F060000000000000000060F0F06004107D8F0B>58 D<60F0F060000000000000000060F0F0701010102020408004177D8F0B>I<000C0000000C0000 000C0000001E0000001E0000003F000000270000002700000043800000438000004380000081C0 000081C0000081C0000100E0000100E00001FFE000020070000200700006007800040038000400 380008001C0008001C001C001E00FF00FFC01A1A7F991D>65 DI<003F0201C0C6 03002E0E001E1C000E1C0006380006780002700002700002F00000F00000F00000F00000F00000 F000007000027000027800023800041C00041C00080E000803003001C0C0003F00171A7E991C> I69 D72 DI77 DI<007F000001C1C000070070000E0038001C001C003C001E0038000E0078000F 0070000700F0000780F0000780F0000780F0000780F0000780F0000780F0000780F00007807800 0F0078000F0038000E003C001E001C001C000E0038000700700001C1C000007F0000191A7E991E >II<0FC21836200E6006C006C002C002C002E00070007E003FE01FF807FC003E 000E00070003800380038003C002C006E004D81887E0101A7E9915>83 D<7FFFFF00701C070040 1C0100401C0100C01C0180801C0080801C0080801C0080001C0000001C0000001C0000001C0000 001C0000001C0000001C0000001C0000001C0000001C0000001C0000001C0000001C0000001C00 00001C0000001C0000001C000003FFE000191A7F991C>I<3F8070C070E020700070007007F01C 7030707070E070E071E071E0F171FB1E3C10107E8F13>97 DI<07F80C1C381C30 087000E000E000E000E000E000E0007000300438080C1807E00E107F8F11>I<007E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E0003CE000C3E00380E00300E00700E00E0 0E00E00E00E00E00E00E00E00E00E00E00600E00700E00381E001C2E0007CFC0121A7F9915>I< 07C01C3030187018600CE00CFFFCE000E000E000E0006000300438080C1807E00E107F8F11>I< 01F0031807380E100E000E000E000E000E000E00FFC00E000E000E000E000E000E000E000E000E 000E000E000E000E000E007FE00D1A80990C>I<0FCE187330307038703870387038303018602F C02000600070003FF03FFC1FFE600FC003C003C003C0036006381C07E010187F8F13>II<18003C003C001800000000000000000000000000FC001C001C001C001C001C001C001C 001C001C001C001C001C001C001C00FF80091A80990A>I<018003C003C0018000000000000000 00000000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C0 01C001C041C0E180E3007E000A2182990C>IIII< FCF8001D0C001E0E001E0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E001C0E00 1C0E001C0E00FF9FC012107F8F15>I<07E01C38300C700E6006E007E007E007E007E007E00760 06700E381C1C3807E010107F8F13>II<03C2000C2600381E00300E00700E00E00E00E00E00E00E00E0 0E00E00E00E00E00700E00700E00381E001C2E0007CE00000E00000E00000E00000E00000E0000 0E00007FC012177F8F14>II<1F2060E04020C020C020F0007F003FC01FE000F080708030C030C0 20F0408F800C107F8F0F>I<0400040004000C000C001C003C00FFC01C001C001C001C001C001C 001C001C001C201C201C201C201C200E4003800B177F960F>IIIIII<7FF86070407040E041C041C003 80070007000E081C081C08381070107030FFF00D107F8F11>I E /Fe 36 121 df<004000800300020006000C001C001800380038007000700070007000F000F000F000F0 00F000F000F000F000F00070007000700070003800380018001C000C0006000200030000800040 0A257C9B11>40 D<800040003000100018000C000E00060007000700038003800380038003C003 C003C003C003C003C003C003C003C003800380038003800700070006000E000C00180010003000 400080000A257E9B11>I<70F8FCFCFC7C04040808102040060D7D850C>44 D<78FCFCFCFC78000000000078FCFCFCFC7806117D900C>58 D<00030000000780000007800000 078000000FC000000FC000001BE000001BE000001BE0000031F0000031F0000060F8000060F800 00E0FC0000C07C0000C07C0001803E0001FFFE0003FFFF0003001F0003001F0006000F8006000F 800E000FC0FFC07FFCFFC07FFC1E1A7F9921>65 D<001FE02000FFFCE003F80FE007C003E01F80 01E01F0000E03E0000E07E0000607C000060FC000000FC000000FC000000FC000000FC000000FC 000000FC000000FC0000007C0000607E0000603E0000601F0000C01F8000C007C0038003F80F00 00FFFC00001FF0001B1A7E9920>67 DII73 D77 D I<003FC00001E0780007801E000F000F001F000F803E0007C03E0007C07C0003E07C0003E0FC00 03F0FC0003F0FC0003F0FC0003F0FC0003F0FC0003F0FC0003F0FC0003F07C0003E07E0007E03E 0007C03E0007C01F000F800F801F0007C03E0001E07800003FC0001C1A7E9921>II82 D<07F0401FFDC03C0FC07803C07001C0F0 01C0F000C0F000C0F80000FF00007FF8003FFF001FFF800FFFC001FFE0000FE00003F00001F0C0 00F0C000F0C000F0E000E0F001E0FC03C0EFFF8083FE00141A7E9919>I<7FFFFF807FFFFF8078 1F0780701F0380601F0180E01F01C0C01F00C0C01F00C0C01F00C0001F0000001F0000001F0000 001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F00 00001F0000001F000007FFFC0007FFFC001A1A7E991F>I<7FF87FF07FF87FF007E0060003E00C 0003F01C0001F8380000F83000007C6000007EE000003FC000001F8000001F8000000FC000000F C000000FE000001BF0000039F8000030F8000060FC0000E07E0001C03E0001801F0003001F8007 000FC0FFE07FFCFFE07FFC1E1A7F9921>88 D<0FF0001C3C003E1E003E0E003E0F001C0F00000F 0000FF000FCF003E0F007C0F00F80F00F80F00F80F00F817007C27E01FC3E013117F9015>97 D<03FC000F0E001C1F003C1F00781F00780E00F80000F80000F80000F80000F800007800007800 003C01801C03000F060003FC0011117F9014>99 D<000FE0000FE00001E00001E00001E00001E0 0001E00001E00001E003F9E00F07E01C03E03C01E07801E07801E0F801E0F801E0F801E0F801E0 F801E07801E07801E03C01E01C03E00F0DFC03F9FC161A7F9919>I<03F0000E1C001C0E003C07 00780700780780F80780F80780FFFF80F80000F800007800007800003C01801C03000E060003FC 0011117F9014>I<00FE0003C700078F800F0F800F0F800F07000F00000F00000F0000FFF000FF F0000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00007FE0007FE000111A80990E>I104 D<1C003E003E003E003E001C00000000 00000000007E007E001E001E001E001E001E001E001E001E001E001E001E001E001E00FF80FF80 091B7F9A0D>I107 DIII<03F8000E0E003C0780 3803807803C07803C0F803E0F803E0F803E0F803E0F803E0F803E07803C07C07C03C07800E0E00 03F80013117F9016>II114 D<1FB020704030C030C030F000FF807FE03FF807F8003CC00C C00CE00CE008F830CFE00E117F9011>I<06000600060006000E000E001E003FF0FFF01E001E00 1E001E001E001E001E001E001E181E181E181E181E180F3003E00D187F9711>II119 DI E /Ff 31 122 df45 D<387CFEFEFE7C3807077C8610>I<00 180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC07007E07C07 F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C0000F80001F 00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07FFFF0FFFF F0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F81F03F81F 03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC3C00FE7E 00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C>I<0000E0 0001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187E00307E0 0707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007E00007E0 0007E00007E00007E000FFFE00FFFE17207E9F1C>I<0003FE0080001FFF818000FF01E38001F8 003F8003E0001F8007C0000F800F800007801F800007803F000003803F000003807F000001807E 000001807E00000180FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000 FE00000000FE000000007E000000007E000001807F000001803F000001803F000003801F800003 000F8000030007C000060003F0000C0001F800380000FF00F000001FFFC0000003FE000021227D A128>67 D70 D76 D85 D<07FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F000 3FF003FDF01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18 167E951B>97 DI<00FF 8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00 007C00007E00007E00003E00301F00600FC0E007FF8000FE0014167E9519>I<0001FE000001FE 0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E000000 3E0000003E0001FC3E0007FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC 003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E00 1E00FE000F83BE0007FF3FC001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C 00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F 00300FC07003FFC000FF0015167E951A>I<003F8000FFC001E3E003C7E007C7E00F87E00F83C0 0F80000F80000F80000F80000F80000F8000FFFC00FFFC000F80000F80000F80000F80000F8000 0F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8000 7FF8007FF80013237FA211>I<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03 E07C03E03C03C03E07C01F0F801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFF FC3800FC70003EF0001EF0001EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C >II<1C003F007F007F 007F003F001C000000000000000000000000000000FF00FF001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B247EA310>I108 DI I<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC007EFC00 7EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE0017167E951C>II114 D<0FF3003FFF00781F00600700E00300E00300F00300FC00007FE0007F F8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00C7F00011 167E9516>I<0180000180000180000180000380000380000780000780000F80003F8000FFFF00 FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8180 0F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>IIII121 D E /Fg 1 111 df<381F004E61804681C04701C08F01C08E01C00E01C00E01C01C 03801C03801C03801C0700380710380710380E10380E2070064030038014127E9119>110 D E /Fh 2 16 df0 D<03C00FF01FF83FFC7FFE7FFEFFFFFF FFFFFFFFFF7FFE7FFE3FFC1FF80FF003C010107E9115>15 D E /Fi 37 123 df<0020004001800380030006000E001C001C003C0038003800780078007800F800F000F0 00F000F000F000F000F000F000F000F800780078007800380038003C001C001C000E0006000300 03800180004000200B297C9E13>40 D<800040003000380018000C000E00070007000780038003 8003C003C003C003E001E001E001E001E001E001E001E001E001E003E003C003C003C003800380 0780070007000E000C00180038003000400080000B297D9E13>I<78FCFCFEFE7A020204040808 3040070E7D850D>44 D<00038000000380000007C0000007C0000007C000000FE000000FE00000 1FF000001BF000001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000C07E00 01803F0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0FFC07F FEFFC07FFE1F1C7E9B24>65 D<001FE02000FFF8E003F80FE007C003E00F8001E01F0000E03E00 00E03E0000607E0000607C000060FC000000FC000000FC000000FC000000FC000000FC000000FC 000000FC0000007C0000607E0000603E0000603E0000C01F0000C00F80018007C0030003F80E00 00FFFC00001FE0001B1C7D9B22>67 D69 D73 D75 DIII<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E 0003E07E0003F07C0001F07C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8 FC0001F8FC0001F87C0001F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E 0001F07C00003FE0001D1C7D9B24>II82 D<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE00 00FFE0007FFE003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000 E0E000E0F001C0FC03C0EFFF0083FC00141C7D9B1B>I<7FFFFFE07FFFFFE0781F81E0701F80E0 601F8060E01F8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F80 00001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F 8000001F8000001F800007FFFE0007FFFE001C1C7E9B21>II87 D<7FFE1FFE007FFE1FFE0007F001800003F803800001FC07000000FC06000000 FE0C0000007F1C0000003F380000003FB00000001FE00000000FE00000000FE000000007F00000 0003F800000007F80000000FFC0000000CFE000000187E000000387F000000703F800000601F80 0000C01FC00001C00FE000018007F000030007F000FFF03FFF80FFF03FFF80211C7F9B24>II<7FFFFC7FFFFC7E01F878 03F87003F0E007E0E007E0C00FC0C01FC0C01F80003F00007F00007E0000FC0000FC0001F80003 F80603F00607E0060FE0060FC00E1F800E1F801C3F001C7F003C7E00FCFFFFFCFFFFFC171C7D9B 1D>I<0FF8001C1E003E0F803E07803E07C01C07C00007C0007FC007E7C01F07C03C07C07C07C0 F807C0F807C0F807C0780BC03E13F80FE1F815127F9117>97 D<03FC000E0E001C1F003C1F0078 1F00780E00F80000F80000F80000F80000F80000F800007800007801803C01801C03000E0E0003 F80011127E9115>99 D<000FF0000FF00001F00001F00001F00001F00001F00001F00001F00001 F00001F001F9F00F07F01C03F03C01F07801F07801F0F801F0F801F0F801F0F801F0F801F0F801 F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C1B>I<01FC000F07001C03803C01C078 01C07801E0F801E0F801E0FFFFE0F80000F80000F800007800007C00603C00601E00C00F038001 FC0013127F9116>I<03F8F00E0F381E0F381C07303C07803C07803C07803C07801C07001E0F00 0E0E001BF8001000001800001800001FFF001FFFC00FFFE01FFFF07801F8F00078F00078F00078 7000707800F01E03C007FF00151B7F9118>103 D<1E003F003F003F003F001E00000000000000 000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FF E0FFE00B1E7F9D0E>105 D107 D110 D<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8 F800F8F800F8F800F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>I114 D<1FD830786018E018E018F000FF807FE07F F01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007000F 000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08079803F0 0E1A7F9913>I119 DII< 3FFF803C1F00303F00303E00607C0060FC0060F80001F00003F00007E00007C1800F81801F8180 1F03803E03007E07007C0F00FFFF0011127F9115>I E /Fj 15 121 df<3C7EFFFFFFFF7E3C08 087B8712>46 D<000C00001C0000FC000FFC00FFFC00F0FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC00FFFFFCFFFF FC16257BA420>49 D<00FF000007FFE0001E07F8003801FC007800FE007E00FF00FF007F00FF00 7F80FF007F80FF003F807E003F803C003F8000007F8000007F0000007F000000FE000000FE0000 01FC000001F8000003F0000007C00000078000000F0000001E0000003800000070018000E00180 01C001800380030007000300060003000FFFFF001FFFFF003FFFFF007FFFFE00FFFFFE00FFFFFE 0019257DA420>I<0000FF80080007FFF018003FC03C38007E000E7801FC0003F803F00001F807 E00000F80FE00000781FC00000781FC00000383F800000383F800000387F800000187F00000018 7F00000018FF00000000FF00000000FF00000000FF00000000FF00000000FF00000000FF000000 00FF00000000FF00000000FF000000007F000000007F000000187F800000183F800000183F8000 00181FC00000301FC00000300FE000006007E000006003F00000C001FC000180007E000700003F C03C000007FFF8000000FFC00025287CA72E>67 D<0001FF0000001FFFF000007F01FC0000FC00 7E0003F8003F8007F0001FC00FE0000FE00FC00007E01FC00007F03F800003F83F800003F83F80 0003F87F000001FC7F000001FC7F000001FCFF000001FEFF000001FEFF000001FEFF000001FEFF 000001FEFF000001FEFF000001FEFF000001FEFF000001FEFF000001FE7F000001FC7F000001FC 7F800003FC7F800003FC3F800003F83FC00007F81FC00007F00FC00007E00FE0000FE007F0001F C003F8003F8000FC007E00007F01FC00001FFFF0000001FF000027287CA730>79 D<03FF00000FFFE0001F03F0003F80F8003F80FC003F807C001F007E001F007E0000007E000000 7E0000007E00000FFE0001FFFE0007F07E001FC07E003F807E007F007E00FE007E00FE007E18FE 007E18FE007E18FE00BE187F01BE183F873FF01FFE1FE003F80F801D1A7E9920>97 D<007F8003FFE007E1F00F80F81F007C3F007E7E003E7E003E7E003FFE003FFE003FFFFFFFFFFF FFFE0000FE0000FE0000FE00007E00007E00003F00003F00031F80060FC00607F01C01FFF0003F C0181A7E991D>101 D<0F001F801FC03FC03FC01FC01F800F0000000000000000000000000000 00FFC0FFC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC0 0FC00FC00FC00FC00FC0FFFCFFFC0E297FA811>105 D110 D<007FC00001FFF00007E0FC000F803E001F001F00 3F001F803E000F807E000FC07E000FC0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000F E0FE000FE0FE000FE07E000FC07E000FC07F001FC03F001F801F001F000F803E0007E0FC0001FF F000007FC0001B1A7E9920>II114 D<03F8C01FFFC03C07C07801C07001C0F000C0 F000C0F800C0FC0000FFC0007FFC003FFF001FFF800FFFC001FFE0000FF00003F0C001F0C000F0 E000F0E000F0F000E0F801E0FE03C0E7FF80C1FC00141A7E9919>I<0060000060000060000060 0000E00000E00001E00001E00003E00007E0001FE000FFFFC0FFFFC007E00007E00007E00007E0 0007E00007E00007E00007E00007E00007E00007E00007E00007E00007E00007E06007E06007E0 6007E06007E06007E06003F0C001F0C000FF80003E0013257FA419>I120 D E /Fk 1 98 df<00200000700000700000700000B80000B80000B800011C00011C00011C00020E00020E0004 070004070007FF000803800803800803801801C03803C0FE0FF815157F9419>97 D E /Fl 62 123 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 0E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00 000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<003F07E00001C0 9C18000380F018000701F03C000E01E03C000E00E018000E00E000000E00E000000E00E000000E 00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C00 0E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>14 D<6060F0F0F8F868680808 08080808101010102020404080800D0C7F9C15>34 D<60F0F8680808081010204080050C7C9C0C >39 D<004000800100020006000C000C0018001800300030007000600060006000E000E000E000 E000E000E000E000E000E000E000E000E000600060006000700030003000180018000C000C0006 0002000100008000400A2A7D9E10>I<800040002000100018000C000C00060006000300030003 8001800180018001C001C001C001C001C001C001C001C001C001C001C001C00180018001800380 03000300060006000C000C00180010002000400080000A2A7E9E10>I<60F0F070101010102020 4080040C7C830C>44 DI<60F0F06004047C830C>I<03C00C301818300C 300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E0076006600670 0E300C300C18180C3007E0101D7E9B15>48 D<030007003F00C700070007000700070007000700 07000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C7C 9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C0038007000 6000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<07E01830201C201C781E78 0E781E381E001C001C00180030006007E00030001C001C000E000F000F700FF80FF80FF80FF00E 401C201C183007E0101D7E9B15>I<000C00000C00001C00003C00003C00005C0000DC00009C00 011C00031C00021C00041C000C1C00081C00101C00301C00201C00401C00C01C00FFFFC0001C00 001C00001C00001C00001C00001C00001C0001FFC0121C7F9B15>I<300C3FF83FF03FC0200020 00200020002000200023E024302818301C200E000E000F000F000F600FF00FF00FF00F800E401E 401C2038187007C0101D7E9B15>I<00F0030C06040C0E181E301E300C700070006000E3E0E430 E818F00CF00EE006E007E007E007E007E007600760077006300E300C18180C3003E0101D7E9B15 >I<60F0F0600000000000000000000060F0F06004127C910C>58 D<60F0F06000000000000000 00000060F0F0701010101020204080041A7C910C>I<000600000006000000060000000F000000 0F0000000F00000017800000178000001780000023C0000023C0000023C0000041E0000041E000 0041E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C0006003E 0004001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 D<001F808000E061800180 1980070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F0 000000F0000000F0000000F0000000F0000000F0000000F0000000700000807800008078000080 380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>67 D69 DI73 D76 DII<003F800000E0E0000380380007001C000E00 0E001C0007003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F0 0001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C0780003C0380003803C000780 1C0007000E000E0007001C000380380000E0E000003F80001B1E7E9C20>II82 D<07E0801C1980300580700380600180E00180E00080E00080E00080 F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001C08001C0 8001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>I<7FFFFFC0700F01C0 600F00C0400F0040400F0040C00F0020800F0020800F0020800F0020000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000001F800003FFFC001B1C7F9B1E>II< FFE0FFE0FF1F001F003C1E001E00180F001F00100F001F00100F001F001007801F002007802780 20078027802003C027804003C043C04003C043C04003E043C04001E081E08001E081E08001E081 E08000F100F10000F100F10000F100F100007900FA00007A007A00007A007A00003E007C00003C 003C00003C003C00003C003C00001800180000180018000018001800281D7F9B2B>87 D<7FF0FFC00FC03E000780180003C0180003E0100001E0200001F0600000F0400000788000007D 8000003D0000001E0000001F0000000F0000000F8000000F80000013C0000023E0000021E00000 41F00000C0F8000080780001007C0003003C0002001E0006001F001F003F80FFC0FFF01C1C7F9B 1F>I<08081010202040404040808080808080B0B0F8F8787830300D0C7A9C15>92 D<1FC000307000783800781C00301C00001C00001C0001FC000F1C00381C00701C00601C00E01C 40E01C40E01C40603C40304E801F870012127E9115>97 D I<07E00C301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E12 7E9112>I<003F0000070000070000070000070000070000070000070000070000070000070003 E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E0070060070070 0700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E6006E006FFFEE000 E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C071E061E0E0C0E000E00 0E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C00381C00381C00381C00 1818001C38000C300013C0001000003000001800001FF8001FFF001FFF803003806001C0C000C0 C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C001C001C001C001C001C 001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I<00C001E001E000C0000000 00000000000000000000000FE000E000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E000E000E060E0F0C0F1C061803E000B25839C0D>IIIII<03F000 0E1C00180600300300700380600180E001C0E001C0E001C0E001C0E001C0E001C0600180700380 3003001806000E1C0003F00012127F9115>II<03C1000C3300180B00300F0070 0700700700E00700E00700E00700E00700E00700E00700600700700700300F00180F000C370007 C700000700000700000700000700000700000700000700003FE0131A7E9116>II<1F90 30704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608FC00D127F9110 >I<04000400040004000C000C001C003C00FFE01C001C001C001C001C001C001C001C001C001C 101C101C101C101C100C100E2003C00C1A7F9910>IIII<7F8FF00F03800F03000702 0003840001C80001D80000F00000700000780000F800009C00010E00020E000607000403801E07 C0FF0FF81512809116>II<7FFC70386038407040F040E041C003C0038007000F 040E041C043C0C380870087038FFF80E127F9112>I E /Fm 38 122 df<000300060008001800 30006000C000C0018003000300060006000C000C001C0018001800380030003000700070006000 600060006000E000E000E000E000E0006000600060006000600020003000100008000800102A7B 9E11>40 D<001000100008000C0004000600060006000600060007000700070007000700060006 00060006000E000E000C000C001C001800180038003000300060006000C000C001800300030006 000C00180010006000C000102A809E11>I45 D<3078F06005047C830D>I<00020006000C001C007C039C003800380038003800700070007000 7000E000E000E000E001C001C001C001C003800380038003800780FFF00F1C7C9B15>49 D<00C06000FFC001FF8001FE00010000010000020000020000020000020000047800058C000606 00040600080600000700000700000600000E00000E00700E00700C00E01C008018008038004030 0040600021C0001F0000131D7C9B15>53 D<000F0000308000C080018380038380030000060000 0E00000C00001C00001CF0003B18003C0C00380C00780C00700E00700E00700E00601C00E01C00 E01C00E01C00E03800E03800E0700060600060C0002180001E0000111D7B9B15>I<060F0F0600 0000000000000000003078F06008127C910D>58 D<0003F020001E0C60003002E000E003C001C0 01C0038001C0070000C00E0000801E0000801C0000803C0000803C000000780000007800000078 000000F0000000F0000000F0000000F0000000F0000400F0000400F0000400F000080070000800 7000100038002000180040000C0180000706000001F800001B1E7A9C1E>67 D<01FFFFE0003C00E0003800600038004000380040003800400070004000700040007020400070 200000E0400000E0400000E0C00000FFC00001C0800001C0800001C0800001C080000381010003 8001000380020003800200070004000700040007000C00070018000E007800FFFFF0001B1C7D9B 1C>69 D<01FFE0003C0000380000380000380000380000700000700000700000700000E00000E0 0000E00000E00001C00001C00001C00001C0000380080380080380080380100700100700300700 600700E00E03C0FFFFC0151C7D9B1A>76 D<01FE0007F8003E000780002E000F00002E00170000 2E001700002E002700004E002E00004E004E00004E004E00004E008E00008E011C00008E011C00 008E021C00008E021C000107043800010704380001070838000107103800020710700002072070 0002072070000207407000040740E000040780E000040700E0000C0700E0001C0601E000FF861F FC00251C7D9B25>I<01FC03FE001C0070003C0060002E0040002E0040002E0040004700800047 008000470080004380800083810000838100008181000081C1000101C2000101C2000100E20001 00E2000200E4000200740002007400020074000400380004003800040038000C0018001C001000 FF8010001F1C7D9B1F>I<000F8400304C00403C00801801001803001803001806001006001006 000007000007000003E00003FC0001FF00007F800007C00001C00001C00000C00000C02000C020 00C0600180600180600300600200F00400CC180083E000161E7D9C17>83 D<1FFFFFC01C0701C0300E00C0200E0080600E0080400E0080401C0080801C0080801C0080001C 0000003800000038000000380000003800000070000000700000007000000070000000E0000000 E0000000E0000000E0000001C0000001C0000001C0000001C0000003C000007FFE00001A1C799B 1E>I<03CC063C0C3C181C3838303870387038E070E070E070E070E0E2C0E2C0E261E462643C38 0F127B9115>97 D<3F00070007000E000E000E000E001C001C001C001C0039C03E603830383070 38703870387038E070E070E070E060E0E0C0C0C1C0618063003C000D1D7B9C13>I<01F007080C 08181C3838300070007000E000E000E000E000E000E008E010602030C01F000E127B9113>I<00 1F80000380000380000700000700000700000700000E00000E00000E00000E0003DC00063C000C 3C00181C00383800303800703800703800E07000E07000E07000E07000E0E200C0E200C0E20061 E4006264003C3800111D7B9C15>I<01E007100C1018083810701070607F80E000E000E000E000 E000E0086010602030C01F000D127B9113>I<0003C0000670000C70001C60001C00001C000038 0000380000380000380000380003FF8000700000700000700000700000700000E00000E00000E0 0000E00000E00001C00001C00001C00001C00001C0000380000380000380000300000300000700 00C60000E60000CC00007800001425819C0D>I<00F3018F030F06070E0E0C0E1C0E1C0E381C38 1C381C381C383830383038187818F00F700070007000E000E0C0C0E1C0C3007E00101A7D9113> I<0FC00001C00001C0000380000380000380000380000700000700000700000700000E78000E8C 000F0E000E0E001C0E001C0E001C0E001C0E00381C00381C00381C003838007038807038807070 80707100E03200601C00111D7D9C15>I<01800380010000000000000000000000000000001C00 2600470047008E008E000E001C001C001C0038003800710071007100720072003C00091C7C9B0D >I<0FC00001C00001C0000380000380000380000380000700000700000700000700000E0F000E 11000E23800E43801C83001C80001D00001E00003F800039C00038E00038E00070E20070E20070 E20070E400E06400603800111D7D9C13>107 D<1F800380038007000700070007000E000E000E 000E001C001C001C001C0038003800380038007000700070007000E400E400E400E40068003800 091D7C9C0B>I<3C1E0780266318C04683A0E04703C0E08E0380E08E0380E00E0380E00E0380E0 1C0701C01C0701C01C0701C01C070380380E0388380E0388380E0708380E0710701C0320300C01 C01D127C9122>I<3C3C002646004687004707008E07008E07000E07000E07001C0E001C0E001C 0E001C1C00381C40381C40383840383880701900300E0012127C9117>I<01E007180C0C180C38 0C300E700E700EE01CE01CE01CE018E038E030E06060C031801E000F127B9115>I<07870004D9 8008E0C008E0C011C0E011C0E001C0E001C0E00381C00381C00381C00381800703800703000707 000706000E8C000E70000E00000E00001C00001C00001C00001C00003C0000FF8000131A7F9115 >I<3C3C26C2468747078E068E000E000E001C001C001C001C0038003800380038007000300010 127C9112>114 D<01F006080C080C1C18181C001F001FC00FF007F0007800386030E030C03080 6060C01F000E127D9111>I<00C001C001C001C00380038003800380FFE00700070007000E000E 000E000E001C001C001C001C00384038403840388019000E000B1A7D990E>I<1E030027070047 0700470700870E00870E000E0E000E0E001C1C001C1C001C1C001C1C0038388038388018388018 39001C5900078E0011127C9116>I<1E06270E470E4706870287020E020E021C041C041C041C08 18083808181018200C4007800F127C9113>I<1E01832703874703874703838707018707010E07 010E07011C0E021C0E021C0E021C0E04180C04181C04181C081C1C100C263007C3C018127C911C >I<070E0019910010E38020E38041C30041C00001C00001C00003800003800003800003800007 0200670200E70400CB04008B080070F00011127D9113>I<1E03270747074707870E870E0E0E0E 0E1C1C1C1C1C1C1C1C38383838183818381C7007F00070007000E0E0C0E1C0818047003C00101A 7C9114>I E /Fn 8 116 df73 D80 D<0007FFC0000000003FFFFC00000000FFFFFF00000003F801 FFC0000007F0003FE0000007F8001FF000000FFC000FF800000FFC000FFC00000FFC0007FC0000 0FFC0007FE00000FFC0003FE000007F80003FF000003F00003FF000000000003FF000000000003 FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000 0007FFFF00000000FFFFFF0000000FFFE3FF0000007FF803FF000001FFC003FF000003FF0003FF 000007FC0003FF00000FF80003FF00001FF00003FF00003FF00003FF00007FE00003FF00007FE0 0003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00007FF03 80FFC00007FF03807FE0000DFF03807FE0001DFF03803FF00039FF87001FF80070FFCF000FFE03 E07FFE0007FFFF807FFC0000FFFE001FF800001FF00007E000312E7CAD37>97 D<00FF80FFFF80FFFF80FFFF8003FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF 8001FF8001FF8001FF8001FF80FFFFFFFFFFFFFFFFFF18487DC71F>108 D<00003FFC00000001FFFF8000000FFFFFF000003FF00FFC00007F8001FE0001FE00007F8003FC 00003FC007F800001FE007F800001FE00FF000000FF01FE0000007F81FE0000007F83FE0000007 FC3FE0000007FC7FC0000003FE7FC0000003FE7FC0000003FE7FC0000003FEFFC0000003FFFFC0 000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003 FFFFC0000003FFFFC0000003FF7FC0000003FE7FC0000003FE7FC0000003FE7FE0000007FE3FE0 000007FC3FE0000007FC1FE0000007F81FF000000FF80FF000000FF007F800001FE007FC00003F E003FE00007FC001FF0000FF80007F8001FE00003FF00FFC00000FFFFFF0000003FFFFC0000000 3FFC0000302E7DAD37>111 D<00FF801FF80000FFFF80FFFF0000FFFF83FFFFC000FFFF8FC07F F00003FF9E000FF80001FFF80007FC0001FFF00003FE0001FFE00001FF0001FFC00000FF8001FF 800000FFC001FF8000007FC001FF8000007FE001FF8000007FE001FF8000003FE001FF8000003F F001FF8000003FF001FF8000003FF001FF8000001FF801FF8000001FF801FF8000001FF801FF80 00001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF8 01FF8000001FF801FF8000001FF801FF8000001FF001FF8000003FF001FF8000003FF001FF8000 003FF001FF8000003FE001FF8000007FE001FF8000007FC001FF800000FFC001FF800000FF8001 FFC00001FF8001FFE00001FF0001FFF00003FE0001FFF8000FFC0001FF9E001FF80001FF8F80FF E00001FF87FFFFC00001FF81FFFE000001FF803FF0000001FF800000000001FF800000000001FF 800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80000000 0001FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80 0000000001FF800000000001FF800000000001FF8000000000FFFFFF00000000FFFFFF00000000 FFFFFF0000000035427DAD3D>I<00FF007F00FFFF01FFC0FFFF03FFE0FFFF0787F003FF0E0FF0 01FF1C1FF801FF381FF801FF301FF801FF701FF801FF600FF001FF600FF001FFE003C001FFC000 0001FFC0000001FFC0000001FFC0000001FF80000001FF80000001FF80000001FF80000001FF80 000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF 80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001 FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF800000FFFFFFC000 FFFFFFC000FFFFFFC000252E7DAD2C>114 D<001FFC030000FFFF870003FFFFCF000FE007FF00 1F8000FF003E00003F003E00001F007C00001F007C00000F00FC00000F00FC00000700FC000007 00FE00000700FF00000700FF80000000FFE00000007FFE0000007FFFF800003FFFFF00003FFFFF C0001FFFFFF0000FFFFFF80007FFFFFC0001FFFFFE00007FFFFF00001FFFFF000000FFFF800000 07FF80000000FFC0E000007FC0E000003FC0E000001FC0F000000FC0F000000FC0F800000FC0F8 00000FC0F800000F80FC00000F80FE00001F80FF00001F00FF80003E00FFC0007C00F9F803F800 F0FFFFF000E03FFFC000C007FE0000222E7CAD2B>I E /Fo 8 117 df<00003000000070000001 F0000007F000007FF000FFFFF000FFFFF000FF8FF000000FF000000FF000000FF000000FF00000 0FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000 000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF0 00000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000F F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00000 0FF0007FFFFFFE7FFFFFFE7FFFFFFE1F3779B62D>49 D<000000FFE0001800000FFFFC00180000 7FFFFF00380001FFE00FC0780007FE0001F0F8000FF8000079F8003FE000003FF8007FC000000F F800FF80000007F801FF00000007F803FE00000003F807FC00000001F807F800000001F80FF800 000000F80FF000000000F81FF000000000781FF000000000783FE000000000783FE00000000038 7FE000000000387FE000000000387FE000000000387FC000000000007FC00000000000FFC00000 000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FF C00000000000FFC00000000000FFC00000000000FFC00000000000FFC000000000007FC0000000 00007FC000000000007FE000000000007FE000000000387FE000000000383FE000000000383FE0 00000000381FF000000000381FF000000000700FF000000000700FF8000000007007F800000000 E007FC00000000E003FE00000001C001FF000000038000FF8000000780007FC000000F00003FE0 00001E00000FF800003C000007FE0000F0000001FFE007E00000007FFFFF800000000FFFFE0000 000000FFE00000353B7BB940>67 D<003FFC00000001FFFF80000007E00FE000000FC003F80000 0FE001FC00001FF001FE00001FF000FF00001FF000FF00001FF0007F00000FE0007F800007C000 7F80000000007F80000000007F80000000007F80000000007F80000000007F800000000FFF8000 0007FFFF8000003FE07F800001FF007F800007FC007F80000FF0007F80001FE0007F80003FE000 7F80007FC0007F80007FC0007F8380FF80007F8380FF80007F8380FF80007F8380FF8000BF8380 FF8000BF83807FC0013F83807FC0033F83803FE0061FC7001FF81C0FFE0007FFF007FC00007FC0 03F00029257DA42D>97 D<0003FF0000001FFFE000007F07F00001FC01FC0003F000FE0007E000 7F000FE0003F001FC0003F801FC0003F803FC0001FC03F80001FC07F80001FC07F80001FE07F80 001FE0FF80001FE0FF80001FE0FFFFFFFFE0FFFFFFFFE0FF80000000FF80000000FF80000000FF 80000000FF800000007F800000007F800000007F800000003FC00000003FC00000001FC00000E0 1FE00000E00FE00001C007F000038003F800070000FE000E00007FC07C00001FFFF0000001FF80 0023257EA428>101 D<01FC00000000FFFC00000000FFFC00000000FFFC0000000007FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC01FF000003FC07FFC00003FC1C0FF00003FC 3007F80003FC6003F80003FCC003FC0003FC8001FC0003FD0001FE0003FF0001FE0003FE0001FE 0003FE0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE00FFFFF07FFFF8FFFFF07FFF F8FFFFF07FFFF82D3A7EB932>104 D<01FC03FE0000FFFC1FFFC000FFFC780FF000FFFDE003F8 0007FF8001FC0003FF0000FE0003FE0000FF0003FC00007F8003FC00007FC003FC00003FC003FC 00003FE003FC00003FE003FC00001FE003FC00001FE003FC00001FF003FC00001FF003FC00001F F003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC 00001FE003FC00003FE003FC00003FE003FC00003FC003FC00003FC003FC00007F8003FC00007F 8003FE0000FF0003FF0001FE0003FF8003FC0003FDC007F80003FCF81FE00003FC3FFF800003FC 07FC000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 00000000FFFFF0000000FFFFF0000000FFFFF00000002C357EA432>112 D<01F80FC0FFF83FF0FFF870F8FFF8C1FC07F883FE03F983FE03F903FE03FB03FE03FA01FC03FA 00F803FA007003FE000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC0000 03FC000003FC000003FC000003FC000003FC0000FFFFF800FFFFF800FFFFF8001F257EA424> 114 D<001C0000001C0000001C0000001C0000001C0000003C0000003C0000003C0000003C0000 007C0000007C000000FC000000FC000001FC000003FC000007FC00001FFFFFC0FFFFFFC0FFFFFF C003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC 000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E001FC00C001FC01C0 01FE01C000FE0380007F8700001FFE000007F8001B357EB423>116 D E /Fp 11 115 df<008003800F80F380038003800380038003800380038003800380038003800380 03800380038003800380038003800380038003800380038003800380038007C0FFFE0F217CA018 >49 D<03F8000C1E001007002007804007C07807C07803C07807C03807C0000780000780000700 000F00000E0000380003F000001C00000F000007800007800003C00003C00003E02003E07003E0 F803E0F803E0F003C04003C0400780200780100F000C1C0003F00013227EA018>51 D<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0F001E0F001E0F001 E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001C00001C00001C00003 80000380300300780700780600700C002018001030000FC00013227EA018>57 D77 D<03F0200C0C601802603001E07000E060 0060E00060E00060E00020E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003 FF80003FC00007E00001E00000F00000F0000070800070800070800070800070C00060C00060E0 00C0F000C0C80180C6070081FC0014247DA21B>83 D<0FE0001838003C0C003C0E001807000007 0000070000070000FF0007C7001E07003C0700780700700700F00708F00708F00708F00F087817 083C23900FC1E015157E9418>97 D<01FE000703000C07801C0780380300780000700000F00000 F00000F00000F00000F00000F00000F000007000007800403800401C00800C010007060001F800 12157E9416>99 D<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7FF18237FA2 1B>104 D<1C001E003E001E001C00000000000000000000000000000000000E00FE001E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC00A227FA10E >I<0E1F80FE60C01E80E00F00700F00700E00700E00700E00700E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E0070FFE7FF18157F941B>110 D<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E000E000E00 0F00FFF010157F9413>114 D E /Fq 20 121 df<00003FC0020001FFF8060007E01C06001F00 070E003C00018E00780000DE00F000005E01E000003E03C000001E078000001E0F0000000E0F00 00000E1E000000061E000000063E000000063C000000027C000000027C000000027C0000000278 00000000F800000000F800000000F800000000F800000000F800000000F800000000F800000000 F800000000F800000000F80000000078000000007C000000007C000000027C000000023C000000 023E000000021E000000021E000000040F000000040F0000000C078000000803C000001801E000 001000F00000200078000040003C000180001F0003000007E01E000001FFF80000003FC0002732 7CB02F>67 D 73 D77 D80 D<007F004001FFC0400780F0C00F0018C01C000DC03C0007C0 380003C0780001C0700001C0F00000C0F00000C0F00000C0F0000040F0000040F0000040F80000 40F80000007C0000007E0000003F0000003FE000001FFE00000FFFE00007FFF80001FFFE00007F FF000007FF8000007F8000000FC0000007E0000003E0000003E0000001F0000001F0800000F080 0000F0800000F0800000F0800000F0C00000F0C00000E0E00001E0E00001E0F00001C0F8000380 EC000780C7000F00C1E03E0080FFF800801FE0001C327CB024>83 D<00FE0000070380000801E0 001000F0003C0078003E0078003E003C003E003C001C003C0000003C0000003C0000003C000007 FC00007C3C0003E03C0007803C001F003C003E003C003C003C007C003C0078003C08F8003C08F8 003C08F8003C08F8007C08F8007C087C00BC083E011E100F060FE003F807C01D1E7D9D20>97 D<018000003F800000FF800000FF8000000F800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 81F80007860700079803C007A000E007C000F007C00078078000380780003C0780003E0780001E 0780001E0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F078000 1E0780003E0780003C0780003C0780007807C00070074000F0072001C006100380060C0E000403 F80020317EB024>I<001FC00000F0380001C00400078002000F000F001E001F001E001F003C00 1F007C000E007C00000078000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000007C0000003C0000801E0000801E0001000F00010007800200 01C00C0000F03000001FC000191E7E9D1D>I<003F800000E0F0000380380007001C000E001E00 1E000E003C000F003C000F007C000F807800078078000780F8000780FFFFFF80F8000000F80000 00F8000000F8000000F8000000F800000078000000780000007C0000003C0000801C0000801E00 01000F0002000700020001C00C0000F03000001FC000191E7E9D1D>101 D<07000F801F801F800F8007000000000000000000000000000000000000000000000001801F80 FF80FF800F80078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800FC0FFF8FFF80D2F7EAE12>105 D<01803F80FF80FF800F80078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800FC0FFFCFFFC0E317EB012>108 D<0181FE003FC0003F860780C0F000FF8801C1003800FF90 01E2003C000FA000E4001C0007A000F4001E0007C000F8001E0007C000F8001E00078000F0001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341E7E9D38>I<01 81FC003F860F00FF880380FF9003C00FA001C007C001E007C001E007C001E0078001E0078001E0 078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00FC003F0FFFC3FFFFFFC 3FFF201E7E9D24>I<003FC00000E0700003801C0007000E000E0007001E0007803C0003C03C00 03C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F8 0001F0F80001F0780001E0780001E07C0003E03C0003C03C0003C01E0007800E00070007000E00 03801C0000E07000003FC0001C1E7E9D20>I<0181F8003F860F00FF9803C0FFA001E007C000F0 07C00078078000780780003C0780003E0780003E0780001E0780001F0780001F0780001F078000 1F0780001F0780001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C0 00F007C000F007A001C007900380078C0E000783F8000780000007800000078000000780000007 8000000780000007800000078000000780000007800000078000000FC00000FFFC0000FFFC0000 202C7E9D24>I<0183E03F8C18FF907CFF907C0FA07C07C03807C00007C00007C0000780000780 000780000780000780000780000780000780000780000780000780000780000780000780000780 000780000780000780000FC000FFFE00FFFE00161E7E9D19>114 D<03FC200E02603801E03000 E0600060E00060E00020E00020F00020F000207C00007F80003FFC001FFF0007FF8001FFE0000F E00001F08000F8800078800038C00038C00038C00038E00030F00070F00060C800C0C6038081FE 00151E7E9D19>I<00400000400000400000400000400000C00000C00000C00001C00001C00003 C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C01003C01003C01003C01003C01003C01003 C01003C01001C02001E02000E0400078C0001F00142B7FAA19>I<018000603F800FE0FF803FE0 FF803FE00F8003E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00780 03E0078003E0078003E0038005E003C009E001C019F000F021FF001FC1FF201E7E9D24>I120 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%PaperSize: A4 %%EndSetup %%Page: 0 1 0 0 bop 831 1086 a Fq(Prop)r(osal)24 b(I)575 1178 y(MPI)f(Con)n(text)f(Sub)r (committee)871 1360 y Fp(Marc)16 b(Snir)853 1481 y(Marc)o(h)g(1993)p eop %%Page: 1 2 1 1 bop 262 619 a Fo(Chapter)28 b(1)262 826 y Fn(Prop)s(osal)36 b(I)262 1042 y Fm(Editorial)17 b(Note:)24 b(This)18 b(chapter)g(is)f(the)h (pr)n(op)n(osal)g(of)g(Mar)n(c)f(Snir)h(which)g(also)f(app)n(e)n(ars)262 1092 y(in)h(the)h(working)f(do)n(cuments)h(of)f(the)h(p)n(oint-to-p)n(oint)f (sub)n(c)n(ommitte)n(e)g(\(Mar)n(ch)h(15\))g(and)262 1142 y(c)n(ol)r(le)n (ctive)13 b(c)n(ommunic)n(ation)i(sub)n(c)n(ommitte)n(e)f(\(Mar)n(ch)g(16\).) 19 b(This)14 b(is)g(a)g(minor)g(e)n(dit)g(of)h(the)262 1191 y Fl(L)273 1186 y Fk(a)292 1191 y Fl(T)315 1204 y(E)338 1191 y(X)j Fm(sour)n(c)n(e)h(of)f(the)g(p)n(oint-to-p)n(oint)h(do)n(cument)g(pr)n (ovide)n(d)g(by)f(Mar)n(c.)29 b(The)19 b(e)n(dit)f(was)262 1241 y(p)n(erforme)n(d)c(by)h(Lyndon)h(Clarke.)262 1378 y Fj(1.1)64 b(Con)n(texts)262 1469 y Fl(A)13 b Fi(con)o(text)g Fl(consists)i(of:)324 1552 y Fh(\017)20 b Fl(A)15 b(set)h(of)f(pro)q(cesses)j(that)d(curren)o(tly)h (b)q(elong)f(to)f(the)i(con)o(text)g(\(p)q(ossibly)f(all)f(pro-)365 1602 y(cesses,)i(or)e(a)g(prop)q(er)g(subset\).)324 1685 y Fh(\017)20 b Fl(A)15 b Fi(ranking)e Fl(of)h(the)h(pro)q(cesses)j(within)c (that)h(con)o(text,)g(i.e.,)e(a)i(n)o(um)o(b)q(ering)e(of)h(the)365 1735 y(pro)q(cesses)e(in)d(that)h(con)o(text)g(from)e(0)h(to)g Fg(n)p Fh(\000)p Fl(1,)i(where)g Fg(n)e Fl(is)g(the)h(n)o(um)o(b)q(er)f(of)g (pro)q(cesses)365 1785 y(in)14 b(that)g(con)o(text.)324 1868 y(A)g(pro)q(cess)h(ma)o(y)d(b)q(elong)i(to)g(sev)o(eral)g(con)o(texts)h(at)f (the)g(same)f(time.)324 1918 y(An)o(y)f(in)o(terpro)q(cess)h(comm)o (unication)c(o)q(ccurs)k(within)e(a)h(con)o(text,)g(and)g(messages)g(sen)o(t) 262 1968 y(within)h(one)i(con)o(text)g(can)f(b)q(e)h(receiv)o(ed)h(only)e (within)f(the)i(same)f(con)o(text.)20 b(A)15 b(con)o(text)g(is)262 2017 y(sp)q(eci\014ed)d(using)e(a)g Fm(c)n(ontext)i(hand)r(le)g Fl(\(i.e.,)d(a)i(handle)f(to)g(an)h(opaque)f(ob)r(ject)i(that)e(iden)o (ti\014es)262 2067 y(a)15 b(con)o(text\).)23 b(Con)o(text)15 b(handles)h(cannot)g(b)q(e)g(transferred)h(for)e(one)g(pro)q(cess)i(to)f (another;)262 2117 y(they)e(can)g(b)q(e)h(used)f(only)f(on)h(the)h(pro)q (cess)g(where)g(they)g(where)g(created.)324 2167 y(F)m(ollo)o(ws)d(examples)h (of)g(p)q(ossible)h(uses)h(for)f(con)o(texts.)262 2283 y Ff(1.1.1)55 b(Lo)r(osely)17 b(sync)n(hronous)i(library)e(call)h(in)n(terface)262 2360 y Fl(Consider)g(the)g(case)h(where)g(a)e(parallel)g(application)f (executes)k(a)d(\\parallel)g(call")f(to)i(a)262 2409 y(library)c(routine,)h (i.e.,)f(where)j(all)d(pro)q(cesses)j(transfer)g(con)o(trol)e(to)g(the)g (library)g(routine.)967 2574 y(1)p eop %%Page: 2 3 2 2 bop 262 307 a Fl(If)14 b(the)i(library)f(w)o(as)g(dev)o(elop)q(ed)h (separately)m(,)g(then)g(one)f(should)h(b)q(ew)o(are)g(of)f(the)h(p)q(ossib-) 262 357 y(ilit)o(y)d(that)j(the)g(library)f(co)q(de)h(ma)o(y)e(receiv)o(e)j (b)o(y)e(mistak)o(e)f(messages)i(send)g(b)o(y)g(the)g(caller)262 407 y(co)q(de,)f(and)g(vice-v)o(ersa.)22 b(T)m(o)14 b(prev)o(en)o(t)i(suc)o (h)f(o)q(ccurrence)j(one)d(migh)o(t)e(use)i(a)g(barrier)g(syn-)262 457 y(c)o(hronization)e(b)q(efore)h(and)f(after)h(the)g(parallel)e(library)h (call.)k(Instead,)d(one)f(can)h(allo)q(cate)262 506 y(a)g(di\013eren)o(t)h (con)o(text)g(to)f(the)h(library)m(,)d(th)o(us)j(prev)o(en)o(ting)g(un)o(w)o (an)o(ted)f(in)o(terference.)21 b(No)o(w,)262 556 y(the)14 b(transfer)h(of)e(con)o(trol)h(to)f(the)i(library)e(need)i(not)f(b)q(e)g (sync)o(hronized.)262 672 y Ff(1.1.2)55 b(F)-5 b(unctional)20 b(decomp)r(osition)e(and)j(mo)r(dular)e(co)r(de)h(dev)n(el-)433 731 y(opmen)n(t)262 807 y Fl(Often,)d(a)f(parallel)f(application)g(is)i(dev)o (elop)q(ed)g(b)o(y)f(in)o(tegrating)g(sev)o(eral)h(distinct)g(func-)262 857 y(tional)f(mo)q(dules,)h(that)h(is)g(eac)o(h)g(dev)o(elop)q(ed)h (separately)m(.)30 b(Eac)o(h)18 b(mo)q(dule)f(is)g(a)h(parallel)262 907 y(program)c(that)j(runs)g(on)f(a)g(dedicated)h(set)h(of)d(pro)q(cesses,) 20 b(and)c(the)h(computation)e(con-)262 957 y(sists)10 b(of)g(phases)h(where) g(mo)q(dules)e(compute)h(separately)m(,)g(in)o(termixed)f(with)h(global)f (phases)262 1006 y(where)h(all)f(pro)q(cesses)k(comm)o(uni)o(cate.)i(It)10 b(is)f(con)o(v)o(enien)o(t)i(to)e(allo)o(w)g(eac)o(h)h(mo)q(dule)e(to)i(use)h (its)262 1056 y(o)o(wn)h(priv)n(ate)h(pro)q(cess)i(n)o(um)o(b)q(ering)d(sc)o (heme,)h(for)g(the)g(in)o(tramo)q(dule)f(computation.)k(This)262 1106 y(is)11 b(ac)o(hiev)o(ed)g(b)o(y)g(using)g(a)g(priv)n(ate)g(mo)q(dule)f (con)o(text)i(for)f(in)o(tramo)q(dule)e(computation,)h(and)262 1156 y(a)j(global)f(con)o(text)j(for)e(in)o(termo)q(dule)g(comm)o(unication.) 262 1272 y Ff(1.1.3)55 b(Collectiv)n(e)16 b(comm)n(unication)262 1349 y Fl(MPI)g(supp)q(orts)i(collectiv)o(e)f(comm)o(unication)c(within)j (dynamically)e(created)k(groups)f(of)262 1399 y(pro)q(cesses.)37 b(Eac)o(h)19 b(suc)o(h)h(group)f(can)h(b)q(e)g(represen)o(ted)i(b)o(y)d(a)g (distinct)h(con)o(text.)35 b(This)262 1448 y(pro)o(vides)18 b(a)g(simple)f(mec)o(hanism)f(to)i(ensure)h(that)g(comm)o(unicati)o(on)c (that)k(p)q(ertains)g(to)262 1498 y(collectiv)o(e)13 b(comm)o(unicati)o(on)d (within)j(one)g(group)g(is)g(not)g(confused)h(with)f(collectiv)o(e)g(com-)262 1548 y(m)o(unication)e(within)i(another)i(group.)262 1664 y Ff(1.1.4)55 b(Ligh)n(t)n(w)n(eigh)n(t)19 b(gang)g(sc)n(heduling)262 1741 y Fl(Consider)14 b(an)f(en)o(vironmen)o(t)g(where)i(pro)q(cesses)h(are)f (m)o(ultith)o(treaded.)i(Con)o(texts)d(can)g(b)q(e)262 1791 y(used)19 b(to)g(pro)o(vide)f(a)g(mec)o(hanism)f(whereb)o(y)i(all)f(pro)q (cesses)j(are)e(time-shared)f(b)q(et)o(w)o(een)262 1840 y(sev)o(eral)d (parallel)e(executions,)j(and)e(can)h(con)o(text)h(switc)o(h)f(from)d(one)j (parallel)f(execution)262 1890 y(to)k(another,)h(in)f(a)g(lo)q(osely)g(sync)o (hronous)h(manner.)31 b(A)19 b(thread)g(is)f(allo)q(cated)g(on)g(eac)o(h)262 1940 y(pro)q(cess)h(to)f(eac)o(h)g(parallel)e(execution,)j(and)f(a)f (di\013eren)o(t)i(con)o(text)g(is)e(used)i(to)e(iden)o(tify)262 1990 y(eac)o(h)h(parallel)e(execution.)29 b(Th)o(us,)19 b(tra\016c)e(from)f (one)i(execution)g(cannot)g(b)q(e)g(confused)262 2040 y(with)f(tra\016c)g (from)f(another)i(execution.)30 b(The)17 b(blo)q(c)o(king)g(and)g(un)o(blo)q (c)o(king)g(of)g(threads)262 2089 y(due)h(to)f(comm)o(unication)e(ev)o(en)o (ts)k(pro)o(vide)e(a)h(\\lazy")e(con)o(text)j(switc)o(hing)e(mec)o(hanism.) 262 2139 y(This)g(can)h(b)q(e)g(extended)i(to)d(the)i(case)f(where)h(the)f (parallel)f(executions)i(are)f(spanning)262 2189 y(distinct)c(pro)q(cess)h (subsets.)20 b(\(MPI)14 b(do)q(es)h(not)f(require)h(m)o(ultithreaded)d(pro)q (cesses.\))262 2326 y Fe(Discussion:)46 b Fd(A)16 b(con)o(text)h(handle)h (migh)o(t)f(b)q(e)g(implemen)o(ted)i(as)d(a)g(p)q(oin)o(ter)i(to)e(a)h (structure)262 2372 y(that)f(consists)h(of)f(con)o(text)h(lab)q(el)h(\(that)e (is)h(carried)g(b)o(y)f(messages)h(sen)o(t)f(within)i(this)f(con)o(text\))262 2417 y(and)10 b(a)h(con)o(text)f(mem)o(b)q(er)h(table,)h(that)e(translates)i (pro)q(cess)f(ranks)g(within)h(a)e(con)o(text)h(to)f(absolute)967 2574 y Fl(2)p eop %%Page: 3 4 3 3 bop 262 307 a Fd(addresses)16 b(or)g(to)g(routing)h(information.)27 b(Of)15 b(course,)i(other)f(implemen)o(tations)j(are)c(p)q(ossible,)262 353 y(including)f(implemen)o(tation)q(s)g(that)e(do)g(not)f(require)i(eac)o (h)f(con)o(text)g(mem)o(b)q(er)g(to)f(store)h(a)g(full)g(list)262 399 y(of)g(the)h(con)o(text)h(mem)o(b)q(ers.)324 444 y(Con)o(texts)k(can)g(b) q(e)g(used)g(only)h(on)f(the)g(pro)q(cess)h(where)e(they)h(w)o(ere)g (created.)31 b(Since)19 b(the)262 490 y(con)o(text)d(carries)g(information)i (on)e(the)f(group)i(of)e(pro)q(cesses)h(that)g(b)q(elong)h(to)f(this)g(con)o (text,)g(a)262 535 y(pro)q(cess)g(can)g(send)g(a)f(message)i(within)g(a)e (con)o(text)h(only)h(to)e(other)h(pro)q(cesses)h(that)e(b)q(elong)j(to)262 581 y(that)13 b(con)o(text.)18 b(Th)o(us,)c(eac)o(h)f(pro)q(cess)h(needs)g (to)g(k)o(eep)f(trac)o(k)h(only)h(of)e(the)g(con)o(texts)h(that)f(where)262 627 y(created)f(at)h(that)f(pro)q(cess;)h(the)g(total)g(n)o(um)o(b)q(er)g(of) g(con)o(texts)g(p)q(er)f(pro)q(cess)i(is)f(lik)o(ely)h(to)f(b)q(e)f(small.) 324 672 y(The)d(only)i(di\013erence)h(I)d(see)g(b)q(et)o(w)o(een)h(this)h (curren)o(t)f(de\014nition)i(of)d(con)o(text,)i(whic)o(h)f(subsumes)262 718 y(the)16 b(group)h(concept,)g(and)g(a)f(pared)h(do)o(wn)f(de\014nition,)j (if)d(that)h(I)e(assume)i(here)f(that)h(pro)q(cess)262 764 y(n)o(um)o(b)q(ering)12 b(is)g(relativ)o(e)g(to)f(the)g(con)o(text,)g(rather) g(then)h(b)q(eing)g(global,)h(th)o(us)e(requiring)i(a)e(con)o(text)262 809 y(mem)o(b)q(er)e(table.)17 b(I)9 b(argue)h(that)f(this)h(is)g(not)f(m)o (uc)o(h)h(added)g(o)o(v)o(erhead,)h(and)f(giv)o(es)g(m)o(uc)o(h)g(additional) 262 855 y(needed)k(functionalit)o(y)m(.)325 917 y Fc(\017)21 b Fd(If)16 b(a)g(new)g(con)o(text)g(is)h(created)f(b)o(y)h(cop)o(ying)h(a)e (previous)i(con)o(text,)f(then)f(one)h(do)q(es)f(not)365 963 y(need)d(a)f(new)f(mem)o(b)q(er)i(table;)g(rather,)f(one)g(needs)h(just)e(a)h (new)g(con)o(text)g(lab)q(el)i(and)f(a)f(new)365 1009 y(p)q(oin)o(ter)17 b(to)e(the)g(same)g(old)h(con)o(text)f(mem)o(b)q(er)h(table.)23 b(This)16 b(holds)h(true,)e(in)h(particular,)365 1054 y(for)d(con)o(texts)h (that)f(include)i(all)g(pro)q(cesses.)325 1117 y Fc(\017)21 b Fd(A)10 b(con)o(text)h(mem)o(b)q(er)g(table)g(mak)o(es)g(sure)g(that)g(a)f (message)h(is)g(sen)o(t)g(only)g(to)g(a)f(pro)q(cess)h(that)365 1162 y(can)16 b(execute)f(in)h(the)f(con)o(text)g(of)f(the)h(message.)23 b(The)15 b(alternativ)o(e)i(mec)o(hanism,)g(whic)o(h)365 1208 y(is)c(c)o(hec)o(king)h(at)e(reception,)i(is)f(less)g(e\016cien)o(t,)g(and)g (requires)h(that)e(eac)o(h)h(con)o(text)g(lab)q(el)h(b)q(e)365 1254 y(system-wide)h(unique.)21 b(This)15 b(requires)g(that,)f(to)g(the)g (least,)g(all)i(pro)q(cesses)e(in)h(a)f(con)o(text)365 1299 y(execute)g(a)f(collectiv)o(e)i(agreemen)o(t)f(algorithm)h(at)e(the)g (creation)h(of)f(this)h(con)o(text.)325 1362 y Fc(\017)21 b Fd(The)c(use)h(of)e(relativ)o(e)j(addressing)g(within)g(eac)o(h)e(con)o(text) h(is)f(needed)h(to)f(supp)q(ort)h(true)365 1407 y(mo)q(dular)e(dev)o(elopmen) o(t)f(of)f(sub)q(computations)j(that)d(execute)g(on)g(a)g(subset)h(of)e(the)h (pro-)365 1453 y(cesses.)26 b(There)16 b(is)g(also)h(a)e(big)i(adv)n(an)o (tage)g(in)g(using)g(the)f(same)g(con)o(text)g(construct)h(for)365 1499 y(collectiv)o(e)f(comm)o(unications)g(as)d(w)o(ell.)262 1802 y Fj(1.2)64 b(Con)n(text)22 b(Op)r(erations)262 1893 y Fl(A)13 b(global)f(con)o(text)j Fi(ALL)e Fl(is)g(prede\014ned.)20 b(All)13 b(pro)q(cesses)j(b)q(elong)e(to)f(this)h(con)o(text)g(when)262 1943 y(computation)i(starts.)30 b(MPI)18 b(do)q(es)h(not)f(sp)q(ecify)g(ho)o (w)f(pro)q(cesses)k(are)d(initially)d(rank)o(ed)262 1992 y(within)k(the)h (con)o(text)g(ALL.)g(It)f(is)h(exp)q(ected)i(that)e(the)g(start-up)g(pro)q (cedure)i(used)f(to)262 2042 y(initiate)c(an)h(MPI)g(program)f(\(at)h (load-time)e(or)i(run-time\))g(will)e(pro)o(vide)j(information)262 2092 y(or)c(con)o(trol)g(on)g(this)g(initial)e(ranking)i(\(e.g.,)f(b)o(y)h (sp)q(ecifying)g(that)h(pro)q(cesses)i(are)d(rank)o(ed)262 2142 y(according)e(to)g(their)g(pid's,)g(or)g(according)g(to)g(the)h(ph)o (ysical)e(addresses)k(of)c(the)i(executing)262 2192 y(pro)q(cessors,)h(or)f (according)g(to)g(a)f(n)o(um)o(b)q(ering)g(sc)o(heme)h(sp)q(eci\014ed)h(at)f (load)f(time\).)262 2341 y Fe(Discussion:)18 b Fd(If)c(w)o(e)g(think)i(of)e (adding)j(new)d(pro)q(cesses)i(at)e(run-time,)i(then)f Fb(ALL)e Fd(con)o(v)o(eys)i(the)262 2391 y(wrong)e(impression,)i(since)f(it)f(is)h (just)f(the)g(initial)j(set)d(of)g(pro)q(cesses.)967 2574 y Fl(3)p eop %%Page: 4 5 4 4 bop 324 407 a Fl(The)14 b(follo)o(wing)d(op)q(erations)k(are)f(a)o(v)n (ailable)d(for)j(creating)g(new)h(con)o(texts.)262 506 y Fi(MPI)p 361 506 15 2 v 17 w(COPY)p 517 506 V 17 w(CONTEXT\(new)o(con)o(text,)g(con)o (text\))324 556 y Fl(Create)d(a)e(new)h(con)o(text)h(that)f(includes)g(all)e (pro)q(cesses)14 b(in)c(the)i(old)e(con)o(text.)17 b(The)12 b(rank)262 606 y(of)f(the)i(pro)q(cesses)i(in)d(the)g(previous)h(con)o(text)g (is)f(preserv)o(ed.)20 b(The)12 b(call)g(m)o(ust)f(b)q(e)i(executed)262 656 y(b)o(y)f(all)f(pro)q(cesses)k(in)d(the)h(old)f(con)o(text.)18 b(It)13 b(is)f(a)g(blo)q(c)o(king)g(call:)k(No)d(call)e(returns)j(un)o(til)e (all)262 706 y(pro)q(cesses)k(ha)o(v)o(e)e(called)f(the)i(function.)j(The)c (parameters)g(are)262 797 y Fi(OUT)h(new)o(con)o(text)k Fl(handle)e(to)g (newly)f(created)j(con)o(text.)28 b(The)17 b(handle)g(should)g(not)365 847 y(b)q(e)e(asso)q(ciated)f(with)g(an)g(ob)r(ject)g(b)q(efore)h(the)f (call.)262 930 y Fi(IN)i(con)o(text)j Fl(handle)14 b(to)g(old)f(con)o(text) 262 1108 y Fe(Discussion:)49 b Fd(I)17 b(considered)j(adding)f(a)e(string)h (parameter,)h(to)e(pro)o(vide)i(a)e(unique)i(iden)o(ti-)262 1154 y(\014er)13 b(to)g(the)g(next)h(con)o(text.)k(But,)13 b(in)h(an)g(en)o(vironmen)o(t)h(where)e(pro)q(cesses)h(are)g(single)h (threaded,)262 1200 y(this)c(is)g(not)g(m)o(uc)o(h)g(help:)17 b(Either)11 b(all)h(pro)q(cesses)g(agree)f(on)f(the)h(order)g(they)g(create)g (new)f(con)o(texts,)262 1245 y(or)j(the)h(application)i(deadlo)q(c)o(ks.)21 b(A)13 b(k)o(ey)h(ma)o(y)f(help)i(in)f(an)g(en)o(vironmen)o(t)h(where)f(pro)q (cesses)g(are)262 1291 y(m)o(ultithreaded,)20 b(to)c(distinguis)q(h)k(call)e (from)e(distinct)j(threads)e(of)g(the)g(same)g(pro)q(cess;)i(but)e(it)262 1337 y(migh)o(t)c(b)q(e)h(simpler)g(to)f(use)h(a)f(m)o(utex)g(algorithm)i(at) e(eac)o(h)g(pro)q(cess.)324 1386 y Fe(Implemen)o(tation)j(note:)24 b Fd(No)16 b(comm)o(unication)j(is)d(needed)i(to)e(create)g(a)g(new)g(con)o (text,)262 1436 y(b)q(ey)o(ond)j(a)f(barrier)i(sync)o(hronization;)k(all)19 b(pro)q(cesses)g(can)g(agree)f(to)h(use)f(the)h(same)f(naming)262 1486 y(sc)o(heme)c(for)g(successiv)o(e)h(copies)g(of)f(the)g(same)g(con)o (text.)20 b(Also,)15 b(no)f(new)g(rank)g(table)h(is)g(needed,)262 1536 y(just)d(a)h(new)g(con)o(text)h(lab)q(el)h(and)e(a)g(new)g(p)q(oin)o (ter)i(to)e(the)g(same)g(old)h(table.)262 1735 y Fi(MPI)p 361 1735 V 17 w(NEW)p 495 1735 V 18 w(CONTEXT\(new)o(con)o(text,)h(con)o(text,)g (k)o(ey)l(,)h(index\))262 1826 y(OUT)f(new)o(con)o(text)k Fl(handle)e(to)g (newly)g(created)i(con)o(text)f(at)f(calling)f(pro)q(cess.)30 b(This)365 1876 y(handle)14 b(should)g(not)g(b)q(e)g(asso)q(ciated)h(with)e (an)h(ob)r(ject)h(b)q(efore)f(the)h(call.)262 1959 y Fi(IN)h(con)o(text)j Fl(handle)14 b(to)g(old)f(con)o(text)262 2042 y Fi(IN)j(k)o(ey)21 b Fl(in)o(teger)262 2125 y Fi(IN)16 b(index)j Fl(in)o(teger)324 2217 y(A)11 b(new)h(con)o(text)g(is)f(created)i(for)e(eac)o(h)g(distinct)h(v) n(alue)f(of)f Fa(key)p Fl(;)h(this)h(con)o(text)g(is)f(shared)262 2267 y(b)o(y)j(all)g(pro)q(cesses)k(that)d(made)f(the)h(call)g(with)f(this)i (k)o(ey)f(v)n(alue.)21 b(Within)13 b(eac)o(h)j(new)g(con-)262 2316 y(text)g(the)h(pro)q(cesses)h(are)f(rank)o(ed)f(according)g(to)g(the)g (order)h(of)e(the)i Fa(index)d Fl(v)n(alues)i(they)262 2366 y(pro)o(vided;)c(in)g(case)i(of)e(ties,)h(pro)q(cesses)j(are)d(rank)o(ed)g (according)g(to)g(their)g(rank)g(in)f(the)h(old)262 2416 y(con)o(text.)967 2574 y(4)p eop %%Page: 5 6 5 5 bop 324 307 a Fl(This)16 b(call)f(is)h(blo)q(c)o(king:)22 b(No)16 b(call)g(returns)i(un)o(til)d(all)g(pro)q(cesses)k(in)d(the)g(old)g (con)o(text)262 357 y(executed)f(the)g(call.)324 407 y(P)o(articular)e(uses)j (of)d(this)h(function)f(are:)324 457 y(\(i\))19 b(Reordering)h(pro)q(cesses:) 33 b(All)19 b(pro)q(cesses)k(pro)o(vide)d(the)g(same)f Fa(key)g Fl(v)n(alue,)i(and)262 506 y(pro)o(vide)13 b(their)i(index)e(in)h(the)g(new)h (order.)324 556 y(\(ii\))i(Splitting)g(a)g(con)o(text)i(in)o(to)e(sub)q(con)o (texts,)k(while)c(preserving)i(the)g(old)e(relativ)o(e)262 606 y(order)11 b(among)e(pro)q(cesses:)19 b(All)9 b(pro)q(cesses)14 b(pro)o(vide)c(the)i(same)d Fa(index)h Fl(v)n(alue,)g(and)h(pro)o(vide)262 656 y(a)i(k)o(ey)h(iden)o(tifying)e(their)j(new)f(sub)q(con)o(text.)262 756 y Fi(MPI)p 361 756 15 2 v 17 w(RANK\(rank,)i(con)o(text\))262 836 y(OUT)f(rank)21 b Fl(in)o(teger)262 914 y Fi(IN)16 b(con)o(text)j Fl(con)o(text)c(handle)324 994 y(Return)f(the)h(rank)e(of)h(the)g(calling)f (pro)q(cess)i(within)f(the)g(sp)q(eci\014ed)h(con)o(text.)262 1094 y Fi(MPI)p 361 1094 V 17 w(SIZE\(size,)g(con)o(text\))262 1174 y(OUT)g(size)20 b Fl(in)o(teger)262 1252 y Fi(IN)c(con)o(text)j Fl(con)o(text)c(handle)324 1333 y(Return)f(the)h(n)o(um)o(b)q(er)e(of)g(pro)q (cesses)k(that)d(b)q(elong)f(to)h(the)g(sp)q(eci\014ed)i(con)o(text.)262 1447 y Ff(1.2.1)55 b(Usage)18 b(note)262 1523 y Fl(Use)c(of)f(con)o(texts)i (for)e(libraries:)k(Eac)o(h)d(library)f(ma)o(y)f(pro)o(vide)h(an)h (initialization)c(routine)262 1573 y(that)k(is)g(to)h(b)q(e)g(called)f(b)o(y) g(all)g(pro)q(cesses,)j(and)d(that)g(generate)i(a)e(con)o(text)i(for)e(the)h (use)g(of)262 1623 y(that)e(library)m(.)324 1673 y(Use)j(of)g(con)o(texts)g (for)g(functional)f(decomp)q(osition:)20 b(A)c(harness)h(program,)d(running) 262 1723 y(in)f(the)i(con)o(text)g Fa(ALL)f Fl(generates)i(a)e(sub)q(con)o (text)h(for)f(eac)o(h)h(mo)q(dule)e(and)h(then)h(starts)g(the)262 1773 y(submo)q(dule)d(within)i(the)g(corresp)q(onding)h(con)o(text.)324 1822 y(Use)g(of)f(con)o(texts)i(for)e(collectiv)o(e)g(comm)o(unication:)i(A)e (con)o(text)i(is)e(created)i(for)e(eac)o(h)262 1872 y(group)f(of)h(pro)q (cesses)i(where)f(collectiv)o(e)f(comm)o(unication)c(is)k(to)g(o)q(ccur.)324 1922 y(Use)i(of)f(con)o(texts)i(for)e(con)o(text-switc)o(hing)h(among)d(sev)o (eral)k(parallel)d(executions:)23 b(A)262 1972 y(pream)o(ble)15 b(co)q(de)j(is)f(used)g(to)g(generate)h(a)f(di\013eren)o(t)h(con)o(text)f (for)g(eac)o(h)g(execution;)i(this)262 2022 y(pream)o(ble)8 b(co)q(de)j(needs)g(to)e(use)i(a)e(m)o(utual)f(exclusion)i(proto)q(col)f(to)h (mak)o(e)e(sure)j(eac)o(h)f(thread)262 2071 y(claims)i(the)i(righ)o(t)g(con)o (text.)262 2208 y Fe(Discussion:)30 b Fd(If)10 b(pro)q(cess)h(handles)h(are)e (made)h(explicit)i(in)e(MPI,)f(then)h(an)f(additional)k(function)262 2254 y(needed)d(is)g Fe(MPI)p 515 2254 14 2 v 15 w(PR)o(OCESS\(pro)q(cess,)i (con)o(text,)g(rank\))p Fd(,)e(whic)o(h)g(returns)g(a)f(handle)i(to)e(the)262 2300 y(pro)q(cess)j(iden)o(ti\014ed)j(b)o(y)d(the)g Fb(rank)f Fd(and)h Fb(context)d Fd(parameters.)324 2350 y(A)e(p)q(ossible)j(addition)g (is)f(a)f(function)h(of)e(the)h(form)f Fe(MPI)p 1136 2350 V 16 w(CREA)l(TE)p 1335 2350 V 17 w(CONTEXT\(new)o(con)o(text,)262 2399 y(list)p 325 2399 V 15 w(of)p 376 2399 V 15 w(pro)q(cess)p 531 2399 V 17 w(handles\))j Fd(whic)o(h)i(creates)f(a)f(new)h(con)o(text)g (out)g(of)g(an)g(explicit)i(list)f(of)f(mem-)262 2449 y(b)q(ers)k(\(and)g (rank)h(them)f(in)h(their)g(order)f(of)g(o)q(ccurrence)h(in)f(the)g(list\).) 27 b(This,)17 b(coupled)h(with)e(a)967 2574 y Fl(5)p eop %%Page: 6 7 6 6 bop 262 307 a Fd(mec)o(hanism)13 b(for)e(requiring)j(the)e(spa)o(wning)h (of)e(new)h(pro)q(cesses)g(to)g(the)g(computation,)h(will)g(allo)o(w)262 357 y(to)g(create)g(a)g(new)h(all)g(inclusiv)o(e)i(con)o(text)e(that)g (includes)h(the)f(additional)i(pro)q(cesses.)j(Ho)o(w)o(ev)o(er,)262 407 y(I)12 b(opp)q(ose)j(the)e(idea)h(of)f(requiring)j(dynamic)f(pro)q(cess)f (creation)g(as)g(part)f(of)g(MPI.)g(Man)o(y)h(imple-)262 457 y(men)o(ters)g(w)o(an)o(t)g(to)g(run)g(MPI)g(in)h(an)f(en)o(vironmen)o(t)i (where)e(pro)q(cesses)i(are)e(statically)i(allo)q(cated)262 506 y(at)c(load-time.)967 2574 y Fl(6)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:50:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00341; Sun, 21 Mar 93 14:50:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24784; Sun, 21 Mar 93 14:50:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:50:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24770; Sun, 21 Mar 93 14:50:17 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03760; Sun, 21 Mar 93 13:44:53 CST Date: Sun, 21 Mar 93 13:44:53 CST From: Tony Skjellum Message-Id: <9303211944.AA03760@Aurora.CS.MsState.Edu> To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu Subject: Re: Proposal I - LaTeX Lyndon, what about your extensive proposal (that I just scribbled about endlessly). Are you shooting that? Anyway, tell me specifically, and call whatever new thing Proposal VI, not proposal II' or II. - Tony From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:52:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00353; Sun, 21 Mar 93 14:52:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24896; Sun, 21 Mar 93 14:52:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:52:00 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24888; Sun, 21 Mar 93 14:51:55 -0500 Date: Sun, 21 Mar 93 19:51:46 GMT Message-Id: <13105.9303211951@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal II' - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is the (all new) Proposal II', credits to myself, which was written over the last two days, intensely. PostScript follws. I have now received comments from Tony on Proposal I++ (now defunct, I truly hope). I will propogate these into this the LaTeX of Proposal II' where appropriate in the style of my reply to Proposal V, shortly. Best Wishes Lyndon ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal II$^\prime$\\MPI Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal II'" % Written by Lyndon J. Clarke % March 1993 % \newcommand{\LabelNote}[1]{\label{ii:note:#1}} \newcommand{\SeeReferNote}[1]{{\bf See~Note~\ref{ii:note:#1}.}} \newcommand{\LabelSection}[1]{\label{ii:sect:#1}} \newcommand{\ReferSection}[1]{{\bf Section~\ref{ii:sect:#1}.}} \chapter{Proposal II$^\prime$} {\it Editorial Note: This is not Proposal II as identified at the post-meet lunch after the February meeting in Dallas. Rik~Littlefied came on board the proposal writing process, and decided to write a different proposal which appears as Proposal V. There is no longer a proposal which contains static contexts as discussed at the post-meet lunch.} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction} This chapter proposes that communication contexts and process groupings within MPI appear as related concepts. In particular process groupings appear as ``frames'' which are used in the creation of communication contexts. Communications contexts retain a reference to, and inherit properties of, their process grouping frames. This reflects the observation that an invocation of a module in a parallel program typically operates within one or more groups of processes, and as such any communication contexts associated with invocations of modules also bind certain semantics of process groupings. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of MPI, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language for only purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The notes assumes knowledge of the proposal text and are therefore best examined after familiarisation with that text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the text of the proposal. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal} \LabelSection{proposal} This section presents the detailed proposal, discussing in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes} \LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and MPI does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{process:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of MPI services. MPI provides a procedure which returns the own descriptor of the calling process. For example, {\tt pd = mpi\_own\_pd()}. \subsubsection*{Process Creation and Destruction} This proposal makes no statements regarding creation and destruction of processes. \SeeReferNote{dynamic:processes} \subsubsection*{Descriptor Transmission} The value of a process descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. MPI provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_PD\_TYPE}. MPI provides a process descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a process descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_pd(pd)}. The user is not allowed to invalidate the process descriptor of the calling process. \SeeReferNote{process:coherency} \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding processor descriptor attributes. \subsection{Process Groupings} \LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to?) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. \SeeReferNote{dynamic:groups} The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to the descriptor of the $N$ processes composing the grouping. The structure of a process grouping is defined by a process grouping topology and this proposal makes no further statements regarding such structures. \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{group:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the process grouping until the termination of MPI services. \SeeReferNote{group:blobs} MPI provides a procedure which returns the descriptor of the home group of the calling process. For example, {\tt gd = mpi\_home\_gd()}. \SeeReferNote{own:group} \subsubsection*{Group Creation and Deletion} MPI provides a procedure which allows users to dynamically create one or more groups which are subsets of existing groups. For example, {\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} partition an existing group {\tt gda} into one or more distinct subsets. This procedure is called by and synchronises all members of {\tt gda}. MPI provides a procedure which allows users to dynamically create one group by explicit definition of its membership as a list of process descriptors. For example, {\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. MPI provides a procedure which allows users to delete a created group. This procedure accepts the descriptor of a group which was created by the calling process and destroys the identified group. For example, {\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. MPI provides additional procedure which allow users to construct process groupings which have a process grouping topology. \subsubsection*{Descriptor Transmission} The value of a group descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. MPI provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_GD\_TYPE}. MPI provides a group descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a group descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_gd(gd)}. The user is not allowed to invalidate the own group descriptor of the process or the group descriptor of any group created by the calling process. \SeeReferNote{group:coherency} \subsubsection*{Descriptor Attributes} MPI provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identifier group. For example, {\tt rank = mpi\_group\_rank(gd)}. MPI provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, {\tt size = mpi\_group\_size(gd)}. MPI provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, {\tt pd = mpi\_group\_pd(gd, rank)}. \SeeReferNote{pd:to:rank} MPI provides additional procedures which allow users to determine the process grouping topology attributes. \subsection{Communication Contexts} \LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping is used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{context:identifiers} \SeeReferNote{descriptor:cache} The creation of MPI process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains its own context until MPI process grouping deletion. MPI provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, {\tt cd = mpi\_own\_cd(gd)}. \SeeReferNote{own:context} \subsubsection*{Context Creation and Deletion} MPI provides a procedure which allows users to dynamically create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, {\tt cd = mpi\_context\_create(gd)}. This procedure is called by and synchronises all members of {\tt gd}. \SeeReferNote{dynamic:contexts} MPI provides a procedure which allows users to destroy created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes that context identifier. For example, {\tt mpi\_context\_delete(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. \subsubsection*{Descriptor Transmission} The value of a context descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_CD\_TYPE}. MPI provides a context descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a context descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_cd(cd)}. The user is not allowed to invalidate the own context descriptor of a group or the context descriptor of any context created by the calling process. \SeeReferNote{context:coherency} \subsubsection*{Descriptor Attributes} MPI provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, {\tt gd = mpi\_context\_gd(cd)}. \subsection{Point-to-Point Communication} \LabelSection{point2point} This proposal recommends three forms for MPI point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a Send operation of form X cannot match with a receive operation of form Y, which requires that the form is embedded into the message envelope. The three forms are described followed by considerations of uniform integration of these forms in the point-to-point communication section of MPI. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. \SeeReferNote{null:context} Message selection and addressing are expressed by {\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. Send supplies the {\tt pd} of the receiver. Receive supplies the {\tt pd} of the sender. Receive can wildcard on {\tt pd} by supplying the value of a named constant process descirptor, e.g. {\tt MPI\_PD\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. \SeeReferNote{closed:context} Message selection and addressing are expressed by {\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. Send supplies the {\tt cd} of the receiver and sender, and the {\tt rank} of the receiver. Receive supplies the {\tt cd} of the sender and receiver, and the rank of the sender. The {\tt (cd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt cd}. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. \SeeReferNote{open:context} Message selection and addressing are expressed by {\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. Send supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. Receive supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The {\tt (rcd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt lcd}. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD}, in which case Receive {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point section of MPI by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with the form. This is at the expense of multiplying the number of Send and Receive procedures by a factor of three, and some difficulty in details of the current point-to-point working document which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options each have advantages and disadvantages. \paragraph*{Option i:} The framework of the open context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii:} The framework of the closed context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as {\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the {\tt (lcd, rcd)} pair at the expense of introducing some trickery. We can define a mechanism which ``globs'' together these two fields onto in an identifier which Send and Receive can distinguish from a context identifier and treat as the shorthand notation. Then we should also define a mechanism by which a receiver which has completed a Receive with wildcard on {\tt rcd} is able to determine the valid context descriptor of the sender. This is a inconvenient. \paragraph*{Option iii:} The framework of the null context form is adopted and extended. The null context form is expressed as {\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is clumsy. \subsection{Collective Communication} \LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context and frame in which they are to operate. MPI does plan to describe symmetric collective communication operations. It is not possible to determine whether the proposal is sufficient to allow implementation of the collective communication section of MPI in terms of the point-to-point section of MPI without loss of generality, since the collective operations are not yet defined. Assymetric collective communication operations, especially those in which the sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a ``glob'' form) which identify the contexts and frames in which they are to operate. MPI does not plan to describe assymetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model and comprise communicative funtionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of MPI. % % END "Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion} \LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents arguments for and against the MIMD model annex, and to some extent explores the consquences of these arguments for MPI in a wider sense. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model, in which process execute in a stricly SIMD fashion. The material describing processes in \ReferSection{processes} is simplified: \begin{itemize} \item processes have identical instruction blocks and different data blocks; \item process descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query and group descriptor attribute query is capable of providing access to all process descriptors); \item dynamic process models are not considered. \end{itemize} The material describing process groupings in \ReferSection{groupings} is simplified: \begin{itemize} \item group descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query capable of providing access to all group descriptors.); \item the own process grouping explicitly becomes a group containing all processes. \end{itemize} The material describing communication contexts in \ReferSection{contexts} is simplified: \begin{itemize} \item context descriptor transmission and registry become unnecessary. \end{itemize} The material describing point-to-point communication in \ReferSection{point2point} is simplified: \begin{itemize} \item the open context form becomes redundant; \item uniform integration ``Option i'' is deleted, and ``Option ii'' loses ``globs'' thus becoming simple enough that ``Option iii'' need not be further considered. \end{itemize} The material describing collective communication in \ReferSection{collective} is simplified: \begin{itemize} \item there is no possibility of collective communication operations spanning more than context. \end{itemize} \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as MPI are not suited to fine grained function driven or data driven programming. The annex is intended to provide expresive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in EXPRESS and PARMACS, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or vice versa!). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computating. The debate is whether or not message passing approaches such as MPI (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD parallel prgrams. The positive argument is that communications between SPMD modules requires high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development and standardisation procedures for distributed operating systems. The latter argument is comparable with the argument for MPI versus parallel compilation. \subsection{Notes} \LabelSection{notes} \begin{enumerate} \item\LabelNote{integer:descriptors} Descriptors are a plentiful but bounded resource. They are expressed as integers for two reasons. Firstly, this expression facilitates uniform binding to ANSI~C and Fortran~77. Secondly, there is a later convenience in ensuring that process descriptors in particular are expressed in integers, and I made all the descriptors are integers for consistency. In practice the descriptor could be an index into a table of objects description structures or pointers to such structures. \item\LabelNote{descriptor:cache} It is possible to allow the user to save user defined attributes in descriptors, as Littlefield has suggested. Such attributes should not be communicated in either descriptor transmission or the descriptor registry. \item\LabelNote{process:identifiers} The process descriptor is not a global unique process identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional process information in implementations. For example, the process identifier of a receiver may not be sufficient to route a message to the receiver, and this information can be referenced from the process descriptor. \item\LabelNote{group:identifiers} The group descriptor is not a global unique group identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependeent, and enables more efficient access to additional group information in implementations. For example, the size of the group and the map from member ranks to process descriptors can be referenced from the group descriptor. \item\LabelNote{context:identifiers} The context descriptor is not a global unique context identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional context information in implementations. For example, the group descriptor of the frame can be referenced from the context descriptor. \item\LabelNote{dynamic:processes} The proposal does not prevent a process model which allows dynamic creation and termination of processes however it does not favour an asynchronous process creation model in which singleton processes are created and terminated in an unstructured fashion. The proposal does favour a model in which blobs of processes are created (and terminated) in a concerted fashion, and in which each group so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying worloads. The author suggests that concerted blob expand and contract operations are most suitable for this purpose than asynchronous spawn/kill operations. \item\LabelNote{registry:check} There should probably also be a registry operation which checks whether a name has been registered without the possibility of blocking the calling process indefinitely. The same registry could be used for process, group and context descriptors. \item\LabelNote{process:coherency} In the static process model it is assumed that processes are created in concert, and that termination of one process implies termination of all processes. In this case there is no coherency problem associated with processes and process descriptors. In a dynamic process model there is a coherency problem which processes and process descriptors since a process can retain the descriptor of a process which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{dynamic:groups} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The author suggests concerted group expand and contract operations are more generally useful than asynchronous join/leave operations. \item\LabelNote{group:blobs} MPI has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is intended to be a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and different blobs have different own process groupings. An SPMD program is a single blob. A host-node program composes two blobs, the node blob and the host blob (which is a singleton). There is a sense in which a blob has a locally SPMD view. \item\LabelNote{own:group} This procedure looks like a process descriptor attribute query. \item\LabelNote{group:coherency} Dynamic processes introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Dynamic processes potentially introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Group transmission introduces a group and group descriptor coherenncy problem since a process can retain a group descriptor of a group which has been identified group. The proposal expects the user to ensure coherent usage. \item\LabelNote{pd:to:rank} The proposal did not include a function to convert a {\tt (gd, pd)} pair into a rank. It is suggested that this inverse map is allowed to be arbitrarily slow. \item\LabelNote{dynamic:contexts} Marc Snir has described a method by which global unique group identifiers can be generated without use of shared global data. The description of context creation anticipates a similar method for global unique context identifier generation. However the synchronisation requirement of this method makes it unnecessary for context creation. Synchronisation at creation of context within a process grouping frame implies that all processes within the frame perform the same context creations in the same sequence. Therefore the combination of global unique frame identifier and context creation sequence number provides a global unique context identifier without communication. \item\LabelNote{own:context} This procedure looks like a group descriptor attribute query. \item\LabelNote{context:coherency} The retention of a reference to a group descriptor by a context introduces the potential for a context and context descriptor coherency problem, as a context could contain a reference to the group descriptor of a group which has been deleted. This could be circumvented by demanding that all such contexts are deleted before a group is deleted. Context descriptor transmission introduces a coherency problem since a process can retain the descriptor of a context which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{null:context} This is somewhat like and PARMACS and PVM~3. It is general purpose, but not particularly expressive. It does not provide facilities for writers of parallel libraries. \item\LabelNote{open:context} This is somewhat like ZIPCODE and VENUS. It is expressive in certain SPMD programs where noncommunicative data driven parallel computations can be performed concurrently. It provides facilities for writers of SPMD like parallel libraries. \item\LabelNote{closed:context} This is somewhat like CHIMP and PVM~2. It is expressive in certain MIMD programs where communicative data driven parallel computations can be performed concurrently. It provides facilities for MSPMD like parallel libraries. \end{enumerate} % % END "Discussion & Notes" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Conclusion" % \section{Conclusion} This chapter has presented and discussed a proposal for communication contexts within MPI. In the proposal process groupings appeared as frames for the creation of communication contexts, and communication contexts retained certain properties of the frames used in their creation. The author strongly recommends this proposal to the committee. % % END "Conclusion" %----------------------------------------------------------------------% % % END "Proposal II'" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:53:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00376; Sun, 21 Mar 93 14:53:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24914; Sun, 21 Mar 93 14:53:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:52:59 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24905; Sun, 21 Mar 93 14:52:46 -0500 Date: Sun, 21 Mar 93 19:52:39 GMT Message-Id: <13111.9303211952@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal II' - PostScript To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk This is the PostScript of the (all new) Proposal II'. LateX preceeded. Best Wishes Lyndon ---------------------------------------------------------------------- %!PS-Adobe-2.0 %%Creator: dvips 5.495 Copyright 1986, 1992 Radical Eye Software %%Title: context-ii.dvi %%CreationDate: Sun Mar 21 18:35:09 1993 %%Pages: 15 %%PageOrder: Ascend %%BoundingBox: 0 0 596 842 %%EndComments %DVIPSCommandLine: dvips context-ii %DVIPSSource: TeX output 1993.03.21:1832 %%BeginProcSet: tex.pro %! /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N} B /TR{translate}N /isls false N /vsize 11 72 mul N /@rigin{isls{[0 1 -1 0 0 0] concat}if 72 Resolution div 72 VResolution div neg scale isls{0 Resolution vsize 72 div mul TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{ CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N} B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook} if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255 {IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V{}B /RV statusdict begin /product where{ pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{ gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M} B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{ p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 39158280 55380996 1000 300 300 (/a/obsidian/disk/home/u36/lyndon/mpi/context-ii.dvi) @start /Fa 1 16 df<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF80FF003C010 107E9115>15 D E /Fb 1 79 df<07E01FC000E006000170040001700400013804000138040002 1C0800021C0800020E0800020E0800040710000407100004039000040390000801E0000801E000 0800E0000800E00018004000FE0040001A147F931A>78 D E /Fc 3 111 df<01FC00FF80001C001C00002E001800002E001000002E001000002700100000470020000043 002000004380200000438020000081C040000081C040000081C040000080E040000100E0800001 007080000100708000010070800002003900000200390000020039000002001D000004001E0000 04000E000004000E00000C000E00001C00040000FF80040000211C7E9B21>78 D<00FFFFE000F001C001C003800180070001000E0001001E0002001C0002003800020070000000 E0000001C0000003800000070000000F0000001E0000001C0000003800000070020000E0040001 C0040003800400070008000F0008000E0018001C003000380070007001E000FFFFE0001B1C7E9B 1C>90 D<381F004E61804681C04701C08F01C08E01C00E01C00E01C01C03801C03801C03801C07 00380710380710380E10380E2070064030038014127E9119>110 D E /Fd 44 123 df<00E001E0038007000E001C001C0038003800700070007000E000E000E000E000E000 E000E000E000E000700070007000380038001C001C000E000700038001E000E00B217A9C16>40 DI<387C7E7E 3E0E1E1C78F060070B798416>44 D<7FFF00FFFF80FFFF80000000000000000000000000000000 FFFF80FFFF807FFF00110B7E9116>61 D<00E00001F00001F00001B00001B00003B80003B80003 B800031800071C00071C00071C00071C00071C000E0E000E0E000FFE000FFE001FFF001C07001C 07001C07007F1FC0FF1FE07F1FC013197F9816>65 D<01F18007FB800FFF801F0F803C07803803 80700380700380F00000E00000E00000E00000E00000E00000E00000E00000F000007003807003 803803803C07001F0F000FFE0007FC0001F00011197E9816>67 D<7FF800FFFE007FFF001C0F00 1C07801C03C01C01C01C01C01C01E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E0 1C01C01C01C01C03C01C07801C0F807FFF00FFFE007FF8001319809816>I<7FFFC0FFFFC07FFF C01C01C01C01C01C01C01C01C01C00001C00001C1C001C1C001FFC001FFC001FFC001C1C001C1C 001C00001C00E01C00E01C00E01C00E01C00E07FFFE0FFFFE07FFFE013197F9816>I<03E30007 FF000FFF001E1F003C0F00380700700700700700F00000E00000E00000E00000E00000E03F80E0 7FC0E03F80F00700700700700700380F003C0F001E1F000FFF0007F70003E70012197E9816>71 D73 D<7F0FE0FF8FF07F0FE01C07801C0F001C0E001C 1C001C3C001C78001CF0001CE0001DF0001FF0001FF8001F38001E1C001C1C001C0E001C0E001C 07001C07001C03807F07E0FF8FF07F07E01419809816>75 DII<7E1FC0FF3FE0 7F1FC01D07001D87001D87001D87001DC7001DC7001CC7001CC7001CE7001CE7001CE7001C6700 1C67001C77001C77001C37001C37001C37001C17007F1F00FF9F007F0F0013197F9816>I<7FF8 00FFFE007FFF001C0F801C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE 001FF8001C00001C00001C00001C00001C00001C00001C00007F0000FF80007F000012197F9816 >80 D<7FE000FFF8007FFC001C1E001C0F001C07001C07001C07001C07001C0F001C1E001FFC00 1FF8001FFC001C1C001C0E001C0E001C0E001C0E001C0E201C0E701C0E707F07E0FF87E07F03C0 14197F9816>82 D<7FFFE0FFFFE0FFFFE0E0E0E0E0E0E0E0E0E0E0E0E000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00007FC000F FE0007FC0013197F9816>84 D<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C00E03800E03800707 0007FF0003FE0000F8001519809816>I87 D89 D<1FE0003FF0007FF800783C00 300E00000E00000E0003FE001FFE003E0E00700E00E00E00E00E00E00E00783E007FFFE03FE7E0 0F83E013127E9116>97 D<7E0000FE00007E00000E00000E00000E00000E00000E3E000EFF000F FF800F83C00F00E00E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00F FF800EFF00063C001419809816>I<03F80FFC1FFE3C1E780C7000E000E000E000E000E000F000 700778073E0E1FFC0FF803F010127D9116>I<003F00007F00003F000007000007000007000007 0003C7000FF7001FFF003C1F00780F00700700E00700E00700E00700E00700E00700E00700700F 00700F003C1F001FFFE00FE7F007C7E014197F9816>I<03E00FF81FFC3C1E780E7007E007FFFF FFFFFFFFE000E000700778073C0F1FFE0FFC03F010127D9116>I<001F00007F8000FF8001E780 01C30001C00001C0007FFF00FFFF00FFFF0001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C00001C0003FFE007FFF003FFE0011197F9816>I<03E3C007F7E00FFF E01C1CC0380E00380E00380E00380E00380E001C1C000FF8001FF0001BE0003800001800001FFC 001FFF003FFF807803C0E000E0E000E0E000E0E000E07001C07C07C03FFF800FFE0003F800131C 7F9116>I<7E0000FE00007E00000E00000E00000E00000E00000E3C000EFE000FFF000F87800F 03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87F C7F01519809816>I<018003C003C0018000000000000000007FC07FC07FC001C001C001C001C0 01C001C001C001C001C001C001C001C07FFFFFFF7FFF101A7D9916>I<7E0000FE00007E00000E 00000E00000E00000E00000E7FE00E7FE00E7FE00E0F000E1E000E3C000E78000EF0000FF0000F F8000FBC000F1E000E0E000E07000E07807F87F0FFCFF07F87F01419809816>107 DII<7E3C00FEFE007FFF000F8780 0F03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F8 7FC7F01512809116>I<03E0000FF8001FFC003C1E00780F00700700E00380E00380E00380E003 80E00380F00780700700780F003C1E001FFC000FF80003E00011127E9116>I<7E3E00FEFF007F FF800F83C00F00E00E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00F FF800EFF000E3C000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000141B8091 16>I114 D<0FEC3FFC7FFCF03CE01CE01C 70007F801FF007F8003C600EE00EF00EF81EFFFCFFF8C7E00F127D9116>I<0300000700000700 000700000700007FFF00FFFF00FFFF000700000700000700000700000700000700000700000701 0007038007038007038007870003FE0001FC0000F80011177F9616>I<7E1F80FE3F807E1F800E 03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E0F800FFFF007 FBF803E3F01512809116>I<7F1FC0FF1FE07F1FC01C07001E0F000E0E000E0E000E0E00071C00 071C00071C00071C0003B80003B80003B80001F00001F00000E00013127F9116>II<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001 F00000E00001E00001F00003B800073C00071C000E0E007F1FC0FF3FE07F1FC013127F9116>I< 7F1FC0FF9FE07F1FC01C07000E07000E0E000E0E00070E00071C00071C00039C00039C00039800 01B80001B80000F00000F00000F00000E00000E00000E00001C00079C0007BC0007F80003F0000 3C0000131B7F9116>I<3FFFC07FFFC07FFFC0700780700F00701E00003C0000780001F00003E0 000780000F00001E01C03C01C07801C0FFFFC0FFFFC0FFFFC012127F9116>I E /Fe 28 121 df45 D<387CFEFEFE7C3807077C8610>I<00 180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC07007E07C07 F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C0000F80001F 00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07FFFF0FFFF F0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F81F03F81F 03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC3C00FE7E 00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C>I<0000E0 0001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187E00307E0 0707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007E00007E0 0007E00007E00007E000FFFE00FFFE17207E9F1C>I<1000201E01E01FFFC01FFF801FFF001FFE 001FF8001BC00018000018000018000018000019FC001FFF001E0FC01807E01803E00003F00003 F00003F80003F83803F87C03F8FE03F8FE03F8FC03F0FC03F07007E03007C01C1F800FFF0003F8 0015207D9F1C>I<0003FE0080001FFF818000FF01E38001F8003F8003E0001F8007C0000F800F 800007801F800007803F000003803F000003807F000001807E000001807E00000180FE00000000 FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE000000007E000000 007E000001807F000001803F000001803F000003801F800003000F8000030007C000060003F000 0C0001F800380000FF00F000001FFFC0000003FE000021227DA128>67 DI<0003FE0040001FFFC0C0007F00F1C001F8003FC003F0000FC007C0 0007C00FC00003C01F800003C03F000001C03F000001C07F000000C07E000000C07E000000C0FE 00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE000FFFFC 7E000FFFFC7F00001FC07F00001FC03F00001FC03F00001FC01F80001FC00FC0001FC007E0001F C003F0001FC001FC003FC0007F80E7C0001FFFC3C00003FF00C026227DA12C>71 D78 D80 D<07FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F000 3FF003FDF01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18 167E951B>97 D<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC00 00FC0000FC0000FC00007C00007E00007E00003E00301F00600FC0E007FF8000FE0014167E9519 >99 D<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000 FC0000FC00007C00007C00007E00003E00181F00300FC07003FFC000FF0015167E951A>101 D<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03C03E07C01F0F 801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC70003EF0001EF000 1EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C>103 D<1C003F007F007F007F003F001C000000000000000000000000000000FF00FF001F001F001F00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B247EA310 >105 D108 DII<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC 007EFC007EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE0017167E951C>I< FF0FE000FF3FF8001FF07C001F803E001F001F001F001F801F001F801F000FC01F000FC01F000F C01F000FC01F000FC01F000FC01F000FC01F000FC01F001F801F001F801F803F001FC03E001FE0 FC001F3FF8001F0FC0001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F 000000FFE00000FFE000001A207E951F>I114 D<0FF3003FFF00781F00600700E00300E00300F00300FC0000 7FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00 C7F00011167E9516>I<0180000180000180000180000380000380000780000780000F80003F80 00FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>III120 D E /Ff 43 121 df<78FCFCFCFC7806067D850D>46 D<03F8000F1E001C07003C07803803807803C07803C07803C0F803E0F803E0F803E0F803E0F803 E0F803E0F803E0F803E0F803E0F803E0F803E0F803E07803C07803C03803803C07801C07000F1E 0003F800131B7E9A18>48 D<00600001E0000FE000FFE000F3E00003E00003E00003E00003E000 03E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000 03E00003E00003E0007FFF807FFF80111B7D9A18>I<07F8001FFE00383F80780FC0FC07C0FC07 E0FC03E0FC03E07803E00007E00007C00007C0000F80001F00001E0000380000700000E0000180 600300600600600800E01FFFC03FFFC07FFFC0FFFFC0FFFFC0131B7E9A18>I<03F8001FFE003C 1F003C0F807C07C07E07C07C07C03807C0000F80000F80001E00003C0003F800001E00000F8000 07C00007C00007E03007E07807E0FC07E0FC07E0FC07C0780F80781F001FFE0007F800131B7E9A 18>I<000180000380000780000F80001F80003F80006F8000CF80008F80018F80030F80060F80 0C0F80180F80300F80600F80C00F80FFFFF8FFFFF8000F80000F80000F80000F80000F80000F80 01FFF801FFF8151B7F9A18>I<1801801FFF001FFE001FFC001FF8001FC0001800001800001800 0018000019F8001E0E00180F801007800007C00007E00007E00007E07807E0F807E0F807E0F807 C0F007C0600F80381F001FFE0007F000131B7E9A18>I<007E0003FF000781800F03C01E07C03C 07C03C0380780000780000F80000F8F800FB0E00FA0780FC0380FC03C0F803E0F803E0F803E0F8 03E07803E07803E07803C03C03C03C07801E0F0007FE0003F800131B7E9A18>I<6000007FFFE0 7FFFE07FFFC07FFF807FFF80E00300C00600C00C00C0180000300000300000600000E00000E000 01E00001C00003C00003C00003C00003C00007C00007C00007C00007C00007C00007C000038000 131C7D9B18>I<03F8000FFE001E0F803807803803C07803C07803C07E03C07F83807FC7003FFE 001FFC000FFE0007FF801DFF80387FC0781FE0F007E0F003E0F001E0F001E0F001E07801C07803 803E07801FFE0003F800131B7E9A18>I<03F8000FFE001E0F003C07807807807803C0F803C0F8 03C0F803E0F803E0F803E0F803E07807E03807E03C0BE00E1BE003E3E00003E00003C00003C038 07C07C07807C0700780F00383C001FF8000FE000131B7E9A18>I<78FCFCFCFC78000000000000 78FCFCFCFC7806127D910D>I<00038000000380000007C0000007C0000007C000000FE000000F E000001FF000001BF000001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000 C07E0001803F0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0 FFC07FFEFFC07FFE1F1C7E9B24>65 D<001FE02000FFF8E003F80FE007C003E00F8001E01F0000 E03E0000E03E0000607E0000607C000060FC000000FC000000FC000000FC000000FC000000FC00 0000FC000000FC0000007C0000607E0000603E0000603E0000C01F0000C00F80018007C0030003 F80E0000FFFC00001FE0001B1C7D9B22>67 DI70 D<000FF008007FFE3801FC07F807E001 F80F8000781F0000783F0000383E0000387E0000187C000018FC000000FC000000FC000000FC00 0000FC000000FC000000FC007FFFFC007FFF7C0001F87E0001F83E0001F83F0001F81F0001F80F 8001F807E001F801FC07F8007FFE78000FF818201C7D9B26>I73 D77 DI<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E0003E07E0003F07C 0001F07C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8 7C0001F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E0001F07C00003FE0 001D1C7D9B24>II<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE00 00FFE0007FFE003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000 E0E000E0F001C0FC03C0EFFF0083FC00141C7D9B1B>83 D<7FFFFFE07FFFFFE0781F81E0701F80 E0601F8060E01F8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F 8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F800000 1F8000001F8000001F800007FFFE0007FFFE001C1C7E9B21>II<0FF8001C1E003E0F803E07 803E07C01C07C00007C0007FC007E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13 F80FE1F815127F9117>97 DI<03FC000E0E001C1F003C1F 00781F00780E00F80000F80000F80000F80000F80000F800007800007801803C01801C03000E0E 0003F80011127E9115>I<000FF0000FF00001F00001F00001F00001F00001F00001F00001F000 01F00001F001F9F00F07F01C03F03C01F07801F07801F0F801F0F801F0F801F0F801F0F801F0F8 01F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C1B>I<01FC000F07001C03803C01C0 7801C07801E0F801E0F801E0FFFFE0F80000F80000F800007800007C00603C00601E00C00F0380 01FC0013127F9116>I<007F0001E38003C7C00787C00F87C00F83800F80000F80000F80000F80 000F8000FFF800FFF8000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F80000F80000F80000F80007FF8007FF800121D809C0F>I<03F8F00E0F381E0F381C07303C 07803C07803C07803C07801C07001E0F000E0E001BF8001000001800001800001FFF001FFFC00F FFE01FFFF07801F8F00078F00078F000787000707800F01E03C007FF00151B7F9118>I<1E003F 003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D108 DII<01FC000F07801C01C03C01E07800F07800F0F800F8 F800F8F800F8F800F8F800F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118> II114 D<1FD830786018E018E018F000FF807FE07FF01FF807FC007CC01CC01CE01CE018F830CFC00E12 7E9113>I<0300030003000300070007000F000F003FFCFFFC1F001F001F001F001F001F001F00 1F001F001F0C1F0C1F0C1F0C0F08079803F00E1A7F9913>II120 D E /Fg 77 125 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 0E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00 000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<007FC001C1C003 03C00703C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F CFF8151D809C17>I<003F07E00001C09C18000380F018000701F03C000E01E03C000E00E01800 0E00E000000E00E000000E00E000000E00E000000E00E00000FFFFFFFC000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF80211D 809C23>I<60F0F0F0F0F0F0F060606060606060606060606060000000000060F0F060041E7C9D 0C>33 D<6060F0F0F8F86868080808080808101010102020404080800D0C7F9C15>I<00E00000 019000000308000003080000070800000708000007080000070800000710000007100000072000 000740000003C03FE003800F00038006000380040005C0040009C0080010E0100030E010006070 200060702000E0384000E03C4000E01C8000E00F0020E0070020700780403009C0401830E18007 C03E001B1F7E9D20>38 D<004000800100020006000C000C001800180030003000700060006000 6000E000E000E000E000E000E000E000E000E000E000E000E00060006000600070003000300018 0018000C000C00060002000100008000400A2A7D9E10>40 D<800040002000100018000C000C00 0600060003000300038001800180018001C001C001C001C001C001C001C001C001C001C001C001 C0018001800180038003000300060006000C000C00180010002000400080000A2A7E9E10>I<00 060000000600000006000000060000000600000006000000060000000600000006000000060000 000600000006000000060000FFFFFFE0FFFFFFE000060000000600000006000000060000000600 0000060000000600000006000000060000000600000006000000060000000600001B1C7E9720> 43 D<60F0F0701010101020204080040C7C830C>II<60F0F06004047C 830C>I<00010003000600060006000C000C000C0018001800180030003000300060006000C000 C000C0018001800180030003000300060006000C000C000C001800180018003000300030006000 60006000C000C00010297E9E15>I<03C00C301818300C300C700E60066006E007E007E007E007 E007E007E007E007E007E007E007E007E00760066006700E300C300C18180C3007E0101D7E9B15 >I<030007003F00C7000700070007000700070007000700070007000700070007000700070007 0007000700070007000700070007000F80FFF80D1C7C9B15>I<07C01830201C400C400EF00FF8 0FF807F8077007000F000E000E001C001C00380070006000C00180030006010C01180110023FFE 7FFEFFFE101C7E9B15>I<07E01830201C201C781E780E781E381E001C001C00180030006007E0 0030001C001C000E000F000F700FF80FF80FF80FF00E401C201C183007E0101D7E9B15>I<000C 00000C00001C00003C00003C00005C0000DC00009C00011C00031C00021C00041C000C1C00081C 00101C00301C00201C00401C00C01C00FFFFC0001C00001C00001C00001C00001C00001C00001C 0001FFC0121C7F9B15>I<300C3FF83FF03FC020002000200020002000200023E024302818301C 200E000E000F000F000F600FF00FF00FF00F800E401E401C2038187007C0101D7E9B15>I<00F0 030C06040C0E181E301E300C700070006000E3E0E430E818F00CF00EE006E007E007E007E007E0 07600760077006300E300C18180C3003E0101D7E9B15>I<4000007FFF807FFF007FFF00400200 80040080040080080000100000100000200000600000400000C00000C00001C000018000018000 038000038000038000038000078000078000078000078000078000078000030000111D7E9B15> I<03E00C301008200C20066006600660067006780C3E083FB01FE007F007F818FC307E601E600F C007C003C003C003C00360026004300C1C1007E0101D7E9B15>I<03C00C301818300C700C600E E006E006E007E007E007E007E0076007700F300F18170C2707C700060006000E300C780C781870 10203030C00F80101D7E9B15>I<60F0F0600000000000000000000060F0F06004127C910C>I<60 F0F0600000000000000000000060F0F0701010101020204080041A7C910C>I<0FE03038401CE0 0EF00EF00EF00E000C001C0030006000C000800180010001000100010001000100000000000000 0000000003000780078003000F1D7E9C14>63 D<000600000006000000060000000F0000000F00 00000F00000017800000178000001780000023C0000023C0000023C0000041E0000041E0000041 E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C0006003E0004 001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 D<001F808000E0618001801980 070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F00000007000008078000080780000803800 00801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>67 DIII<001F808000E061800180 1980070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F0 000000F0000000F0000000F0000000F0000000F000FFF0F0000F80700007807800078078000780 380007801C0007801C0007800E00078007000B800180118000E06080001F80001C1E7E9C21>I< FFF3FFC00F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C 000F003C000F003C000F003C000FFFFC000F003C000F003C000F003C000F003C000F003C000F00 3C000F003C000F003C000F003C000F003C000F003C000F003C000F003C00FFF3FFC01A1C7E9B1F >II76 DII<003F800000E0 E0000380380007001C000E000E001C0007003C00078038000380780003C0780003C0700001C0F0 0001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C0 780003C0380003803C0007801C0007000E000E0007001C000380380000E0E000003F80001B1E7E 9C20>II82 D<07E0801C1980300580700380600180 E00180E00080E00080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F80 0007C00003C00001C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E 9C17>I<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F00 20000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003FFFC001B 1C7F9B1E>IIII<7FF0FFC00FC03E000780180003C0180003E0100001E0200001 F0600000F0400000788000007D8000003D0000001E0000001F0000000F0000000F8000000F8000 0013C0000023E0000021E0000041F00000C0F8000080780001007C0003003C0002001E0006001F 001F003F80FFC0FFF01C1C7F9B1F>II<7FFFF07C01F07001E06003C06003C0400780400F80 400F00401E00001E00003C00007C0000780000F00000F00001E00003E00003C010078010078010 0F00101F00301E00203C00203C00607800E0F803E0FFFFE0141C7E9B19>I<0808101020204040 4040808080808080B0B0F8F8787830300D0C7A9C15>92 D<1FC000307000783800781C00301C00 001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F8700 12127E9115>97 DI<07E00C301878307870306000E000E0 00E000E000E000E00060007004300418080C3007C00E127E9112>I<003F000007000007000007 0000070000070000070000070000070000070000070003E7000C1700180F003007007007006007 00E00700E00700E00700E00700E00700E00700600700700700300700180F000C370007C7E0131D 7E9C17>I<03E00C301818300C700E6006E006FFFEE000E000E000E00060007002300218040C18 03E00F127F9112>I<00F8018C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003 C4C00C38C01C3880181800381C00381C00381C00381C001818001C38000C300013C00010000030 00001800001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007 F800121C7F9215>II<18003C003C001800000000000000 0000000000000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C 001C00FF80091D7F9C0C>I<00C001E001E000C000000000000000000000000000000FE000E000 E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0 F0C0F1C061803E000B25839C0D>IIIII<03F0000E1C00180600300300700380600180E0 01C0E001C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F91 15>II<03C1000C3300180B00300F00700700700700E00700E00700E00700E007 00E00700E00700600700700700300F00180F000C370007C7000007000007000007000007000007 00000700000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE0 0FF000F880388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C00 3C00FFE01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C 1A7F9910>IIII<7F8FF00F03800F030007020003840001C80001D80000F000007000 00780000F800009C00010E00020E000607000403801E07C0FF0FF81512809116>II<7FFC70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E 127F9112>I124 D E /Fh 24 118 df<0003E0000000000FF00000 00003E3800000000781800000000780C00000000F80C00000000F00C00000001F00C00000001F0 0C00000001F01C00000001F81800000001F83000000001F87000000001F86000000001F8C001FF E000FD8001FFE000FF00001C0000FE0000180000FE00003000007E00003000007F0000600000FF 0000600001FF8000C00003BF80018000071FC00180000F0FE00300001E0FE00600003E07F00600 007E03F80C0000FE03FC180000FE01FE300000FE00FE600000FE007FC00000FE003F8000C07F00 1FC000C07F000FF001C03F801FF803801FC0F0FE0F0007FFC03FFE0001FE0007F8002B287DA732 >38 D<3C7EFFFFFFFF7E3C08087B8712>46 D<000C00001C0000FC000FFC00FFFC00F0FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC 0000FC0000FC00FFFFFCFFFFFC16257BA420>49 D<00FF000007FFE0001E07F8003801FC007800 FE007E00FF00FF007F00FF007F80FF007F80FF003F807E003F803C003F8000007F8000007F0000 007F000000FE000000FE000001FC000001F8000003F0000007C00000078000000F0000001E0000 003800000070018000E0018001C001800380030007000300060003000FFFFF001FFFFF003FFFFF 007FFFFE00FFFFFE00FFFFFE0019257DA420>I<00FF800007FFF0000F03F8001800FC003E00FE 007F00FF007F007F007F807F007F007F003F00FF001E00FE000000FE000000FC000001F8000003 F0000007E00000FF800000FF00000003E0000001F8000000FC000000FE0000007F0000007F0000 007F8018007F807E007F80FF007F80FF007F80FF007F80FF007F00FE00FF007C00FE003801FC00 1E03F8000FFFE00001FF000019257DA420>I<0000380000007800000078000000F8000001F800 0003F8000003F8000007F800000FF800001DF8000019F8000031F8000071F8000061F80000C1F8 0001C1F8000381F8000301F8000601F8000E01F8001C01F8001801F8003001F8007001F800E001 F800FFFFFFE0FFFFFFE00001F8000001F8000001F8000001F8000001F8000001F8000001F80000 01F800007FFFE0007FFFE01B257EA420>I<0000FF80080007FFF018003FC03C38007E000E7801 FC0003F803F00001F807E00000F80FE00000781FC00000781FC00000383F800000383F80000038 7F800000187F000000187F00000018FF00000000FF00000000FF00000000FF00000000FF000000 00FF00000000FF00000000FF00000000FF00000000FF000000007F000000007F000000187F8000 00183F800000183F800000181FC00000301FC00000300FE000006007E000006003F00000C001FC 000180007E000700003FC03C000007FFF8000000FFC00025287CA72E>67 DI73 D78 D80 D<03FF00000FFFE0001F03F0003F80F8003F80FC003F807C001F 007E001F007E0000007E0000007E0000007E00000FFE0001FFFE0007F07E001FC07E003F807E00 7F007E00FE007E00FE007E18FE007E18FE007E18FE00BE187F01BE183F873FF01FFE1FE003F80F 801D1A7E9920>97 D<003FE001FFF807E07C0FC0FE1F80FE3F00FE3F007C7E007C7E0000FE0000 FE0000FE0000FE0000FE0000FE0000FE0000FE00007E00007F00003F00003F80031F80070FC006 07F01C01FFF0003FC0181A7E991D>99 D<00007FE000007FE0000007E0000007E0000007E00000 07E0000007E0000007E0000007E0000007E0000007E0000007E0000007E0000007E0007F87E001 FFE7E007E077E00FC01FE01F800FE03F0007E03F0007E07E0007E07E0007E0FE0007E0FE0007E0 FE0007E0FE0007E0FE0007E0FE0007E0FE0007E0FE0007E07E0007E07E0007E07F0007E03F0007 E01F000FE00F801FE007E077E003FFC7FE007F07FE1F287EA724>I<007F8003FFE007E1F00F80 F81F007C3F007E7E003E7E003E7E003FFE003FFE003FFFFFFFFFFFFFFE0000FE0000FE0000FE00 007E00007E00003F00003F00031F80060FC00607F01C01FFF0003FC0181A7E991D>I<0F001F80 1FC03FC03FC01FC01F800F000000000000000000000000000000FFC0FFC00FC00FC00FC00FC00F C00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC0FFFCFFFC 0E297FA811>105 D108 D110 D<007FC00001FFF00007E0FC000F803E001F001F003F001F 803E000F807E000FC07E000FC0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE00 0FE0FE000FE07E000FC07E000FC07F001FC03F001F801F001F000F803E0007E0FC0001FFF00000 7FC0001B1A7E9920>II114 D<03F8C01FFFC03C07C07801C07001C0F000C0F000C0 F800C0FC0000FFC0007FFC003FFF001FFF800FFFC001FFE0000FF00003F0C001F0C000F0E000F0 E000F0F000E0F801E0FE03C0E7FF80C1FC00141A7E9919>I<00600000600000600000600000E0 0000E00001E00001E00003E00007E0001FE000FFFFC0FFFFC007E00007E00007E00007E00007E0 0007E00007E00007E00007E00007E00007E00007E00007E00007E00007E06007E06007E06007E0 6007E06007E06003F0C001F0C000FF80003E0013257FA419>II E /Fi 40 123 df<0001FC3C00060E67000C0E C7001C0DC6001C01C0003801C0003803800038038000380380003803800070038007FFFFF80070 0700007007000070070000E0070000E00E0000E00E0000E00E0000E00E0001C00E0001C01C0001 C01C0001C01C0001C01C0003801C00038038000380380003803800030038000700300007007000 06006000C6606000E470C000C8618000703E00002025819C19>11 D<0001FC000703000C03001C 07001C0300180000380000380000380000380000700007FFFC00701C00701C00701C00E03800E0 3800E03800E03800E07001C07001C07001C07001C0E201C0E201C0E20380E40380640380380380 00030000070000060000C60000E40000CC00007000001825819C17>I<18387838080810102040 4080050C7D830D>44 DI<3078F06005047C830D>I<060F0F060000 00000000000000003078F06008127C910D>58 D<01FFFE00003C0780003801C0003801C0003800 E0003800E0007000F00070007000700070007000F000E000F000E000F000E000F000E000F001C0 01E001C001E001C001E001C001C0038003C003800380038007800380070007000E0007001C0007 003800070070000E01C000FFFF00001C1C7D9B1F>68 D<01FFFFE0003C00E00038006000380040 00380040003800400070004000700040007020400070200000E0400000E0400000E0C00000FFC0 0001C0800001C0800001C0800001C0800003810100038001000380020003800200070004000700 040007000C00070018000E007800FFFFF0001B1C7D9B1C>I<01FFFFC0003C01C0003800C00038 008000380080003800800070008000700080007020800070200000E0400000E0400000E0C00000 FFC00001C0800001C0800001C0800001C080000381000003800000038000000380000007000000 0700000007000000070000000F000000FFF000001A1C7D9B1B>I<01FFC0003C00003800003800 00380000380000700000700000700000700000E00000E00000E00000E00001C00001C00001C000 01C0000380000380000380000380000700000700000700000700000F0000FFE000121C7E9B10> 73 D<01FFE0003C0000380000380000380000380000700000700000700000700000E00000E000 00E00000E00001C00001C00001C00001C000038008038008038008038010070010070030070060 0700E00E03C0FFFFC0151C7D9B1A>76 D<01FC03FE001C0070003C0060002E0040002E0040002E 0040004700800047008000470080004380800083810000838100008181000081C1000101C20001 01C2000100E2000100E2000200E400020074000200740002007400040038000400380004003800 0C0018001C001000FF8010001F1C7D9B1F>78 D<01FFFC00003C070000380380003801C0003801 C0003801C0007003C0007003C0007003C00070038000E0078000E0070000E00E0000E0380001FF E00001C0000001C0000001C0000003800000038000000380000003800000070000000700000007 000000070000000F000000FFE000001A1C7D9B1C>80 D<01FFF800003C0E000038070000380380 003803800038038000700780007007800070078000700F0000E00E0000E01C0000E0700000FFC0 0001C0C00001C0600001C0700001C07000038070000380700003807000038070000700F0000700 F0400700F0400700F0800F007880FFE0790000001E001A1D7D9B1E>82 D<1FFFFFC01C0701C030 0E00C0200E0080600E0080400E0080401C0080801C0080801C0080001C00000038000000380000 00380000003800000070000000700000007000000070000000E0000000E0000000E0000000E000 0001C0000001C0000001C0000001C0000003C000007FFE00001A1C799B1E>84 D86 D<03CC063C0C3C181C3838303870387038E070E070E070E070E0E2C0E2C0E261 E462643C380F127B9115>97 D<3F00070007000E000E000E000E001C001C001C001C0039C03E60 383038307038703870387038E070E070E070E060E0E0C0C0C1C0618063003C000D1D7B9C13>I< 01F007080C08181C3838300070007000E000E000E000E000E000E008E010602030C01F000E127B 9113>I<001F80000380000380000700000700000700000700000E00000E00000E00000E0003DC 00063C000C3C00181C00383800303800703800703800E07000E07000E07000E07000E0E200C0E2 00C0E20061E4006264003C3800111D7B9C15>I<01E007100C1018083810701070607F80E000E0 00E000E000E000E0086010602030C01F000D127B9113>I<0003C0000670000C70001C60001C00 001C0000380000380000380000380000380003FF8000700000700000700000700000700000E000 00E00000E00000E00000E00001C00001C00001C00001C00001C000038000038000038000030000 030000070000C60000E60000CC00007800001425819C0D>I<00F3018F030F06070E0E0C0E1C0E 1C0E381C381C381C381C383830383038187818F00F700070007000E000E0C0C0E1C0C3007E0010 1A7D9113>I<0FC00001C00001C000038000038000038000038000070000070000070000070000 0E78000E8C000F0E000E0E001C0E001C0E001C0E001C0E00381C00381C00381C00383800703880 703880707080707100E03200601C00111D7D9C15>I<0180038001000000000000000000000000 0000001C002600470047008E008E000E001C001C001C0038003800710071007100720072003C00 091C7C9B0D>I<0FC00001C00001C0000380000380000380000380000700000700000700000700 000E0F000E11000E23800E43801C83001C80001D00001E00003F800039C00038E00038E00070E2 0070E20070E20070E400E06400603800111D7D9C13>107 D<1F80038003800700070007000700 0E000E000E000E001C001C001C001C0038003800380038007000700070007000E400E400E400E4 0068003800091D7C9C0B>I<3C1E0780266318C04683A0E04703C0E08E0380E08E0380E00E0380 E00E0380E01C0701C01C0701C01C0701C01C070380380E0388380E0388380E0708380E0710701C 0320300C01C01D127C9122>I<3C3C002646004687004707008E07008E07000E07000E07001C0E 001C0E001C0E001C1C00381C40381C40383840383880701900300E0012127C9117>I<01E00718 0C0C180C380C300E700E700EE01CE01CE01CE018E038E030E06060C031801E000F127B9115>I< 07870004D98008E0C008E0C011C0E011C0E001C0E001C0E00381C00381C00381C0038180070380 0703000707000706000E8C000E70000E00000E00001C00001C00001C00001C00003C0000FF8000 131A7F9115>I<3C3C26C2468747078E068E000E000E001C001C001C001C003800380038003800 7000300010127C9112>114 D<01F006080C080C1C18181C001F001FC00FF007F0007800386030 E030C030806060C01F000E127D9111>I<00C001C001C001C00380038003800380FFE007000700 07000E000E000E000E001C001C001C001C00384038403840388019000E000B1A7D990E>I<1E03 00270700470700470700870E00870E000E0E000E0E001C1C001C1C001C1C001C1C003838803838 801838801839001C5900078E0011127C9116>I<1E06270E470E4706870287020E020E021C041C 041C041C0818083808181018200C4007800F127C9113>I<1E0183270387470387470383870701 8707010E07010E07011C0E021C0E021C0E021C0E04180C04181C04181C081C1C100C263007C3C0 18127C911C>I<070E0019910010E38020E38041C30041C00001C00001C0000380000380000380 00038000070200670200E70400CB04008B080070F00011127D9113>I<1E03270747074707870E 870E0E0E0E0E1C1C1C1C1C1C1C1C38383838183818381C7007F00070007000E0E0C0E1C0818047 003C00101A7C9114>I<038207C20FEC0838100800100020004000800100020004000808100838 3067F043E081C00F127D9111>I E /Fj 1 49 df<001E003F003F003F003F007E007E007E00FC 00FC00FC00F801F801F801F001F003F003E003E003C007C007C0078007800F800F000F000F001E 001E001E003C003C003C0038007800780070007000F000E0006000102A7EAD14>48 D E /Fk 8 116 df73 D80 D<0007FFC0000000003FFFFC00000000FFFFFF00000003F801FFC000 0007F0003FE0000007F8001FF000000FFC000FF800000FFC000FFC00000FFC0007FC00000FFC00 07FE00000FFC0003FE000007F80003FF000003F00003FF000000000003FF000000000003FF0000 00000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000000007FF FF00000000FFFFFF0000000FFFE3FF0000007FF803FF000001FFC003FF000003FF0003FF000007 FC0003FF00000FF80003FF00001FF00003FF00003FF00003FF00007FE00003FF00007FE00003FF 0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FFC00007FF0380FFC0 0007FF03807FE0000DFF03807FE0001DFF03803FF00039FF87001FF80070FFCF000FFE03E07FFE 0007FFFF807FFC0000FFFE001FF800001FF00007E000312E7CAD37>97 D<00FF80FFFF80FFFF80 FFFF8003FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF80FFFFFFFFFFFFFFFFFF18487DC71F>108 D<00003FFC00000001FFFF8000000FFFFFF000 003FF00FFC00007F8001FE0001FE00007F8003FC00003FC007F800001FE007F800001FE00FF000 000FF01FE0000007F81FE0000007F83FE0000007FC3FE0000007FC7FC0000003FE7FC0000003FE 7FC0000003FE7FC0000003FEFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC000 0003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FF7FC0000003FE 7FC0000003FE7FC0000003FE7FE0000007FE3FE0000007FC3FE0000007FC1FE0000007F81FF000 000FF80FF000000FF007F800001FE007FC00003FE003FE00007FC001FF0000FF80007F8001FE00 003FF00FFC00000FFFFFF0000003FFFFC00000003FFC0000302E7DAD37>111 D<00FF801FF80000FFFF80FFFF0000FFFF83FFFFC000FFFF8FC07FF00003FF9E000FF80001FFF8 0007FC0001FFF00003FE0001FFE00001FF0001FFC00000FF8001FF800000FFC001FF8000007FC0 01FF8000007FE001FF8000007FE001FF8000003FE001FF8000003FF001FF8000003FF001FF8000 003FF001FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801 FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF800000 1FF801FF8000001FF001FF8000003FF001FF8000003FF001FF8000003FF001FF8000003FE001FF 8000007FE001FF8000007FC001FF800000FFC001FF800000FF8001FFC00001FF8001FFE00001FF 0001FFF00003FE0001FFF8000FFC0001FF9E001FF80001FF8F80FFE00001FF87FFFFC00001FF81 FFFE000001FF803FF0000001FF800000000001FF800000000001FF800000000001FF8000000000 01FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF8000 00000001FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001 FF800000000001FF8000000000FFFFFF00000000FFFFFF00000000FFFFFF0000000035427DAD3D >I<00FF007F00FFFF01FFC0FFFF03FFE0FFFF0787F003FF0E0FF001FF1C1FF801FF381FF801FF 301FF801FF701FF801FF600FF001FF600FF001FFE003C001FFC0000001FFC0000001FFC0000001 FFC0000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF800000 01FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF8000 0001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80 000001FF80000001FF80000001FF80000001FF800000FFFFFFC000FFFFFFC000FFFFFFC000252E 7DAD2C>114 D<001FFC030000FFFF870003FFFFCF000FE007FF001F8000FF003E00003F003E00 001F007C00001F007C00000F00FC00000F00FC00000700FC00000700FE00000700FF00000700FF 80000000FFE00000007FFE0000007FFFF800003FFFFF00003FFFFFC0001FFFFFF0000FFFFFF800 07FFFFFC0001FFFFFE00007FFFFF00001FFFFF000000FFFF80000007FF80000000FFC0E000007F C0E000003FC0E000001FC0F000000FC0F000000FC0F800000FC0F800000FC0F800000F80FC0000 0F80FE00001F80FF00001F00FF80003E00FFC0007C00F9F803F800F0FFFFF000E03FFFC000C007 FE0000222E7CAD2B>I E /Fl 8 117 df<00003000000070000001F0000007F000007FF000FFFF F000FFFFF000FF8FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00000 0FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000 000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF0 00000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000F F000000FF000000FF000000FF000000FF000000FF000000FF000000FF0007FFFFFFE7FFFFFFE7F FFFFFE1F3779B62D>49 D<000000FFE0001800000FFFFC001800007FFFFF00380001FFE00FC078 0007FE0001F0F8000FF8000079F8003FE000003FF8007FC000000FF800FF80000007F801FF0000 0007F803FE00000003F807FC00000001F807F800000001F80FF800000000F80FF000000000F81F F000000000781FF000000000783FE000000000783FE000000000387FE000000000387FE0000000 00387FE000000000387FC000000000007FC00000000000FFC00000000000FFC00000000000FFC0 0000000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FFC000000000 00FFC00000000000FFC00000000000FFC000000000007FC000000000007FC000000000007FE000 000000007FE000000000387FE000000000383FE000000000383FE000000000381FF00000000038 1FF000000000700FF000000000700FF8000000007007F800000000E007FC00000000E003FE0000 0001C001FF000000038000FF8000000780007FC000000F00003FE000001E00000FF800003C0000 07FE0000F0000001FFE007E00000007FFFFF800000000FFFFE0000000000FFE00000353B7BB940 >67 D<003FFC00000001FFFF80000007E00FE000000FC003F800000FE001FC00001FF001FE0000 1FF000FF00001FF000FF00001FF0007F00000FE0007F800007C0007F80000000007F8000000000 7F80000000007F80000000007F80000000007F800000000FFF80000007FFFF8000003FE07F8000 01FF007F800007FC007F80000FF0007F80001FE0007F80003FE0007F80007FC0007F80007FC000 7F8380FF80007F8380FF80007F8380FF80007F8380FF8000BF8380FF8000BF83807FC0013F8380 7FC0033F83803FE0061FC7001FF81C0FFE0007FFF007FC00007FC003F00029257DA42D>97 D<0003FF0000001FFFE000007F07F00001FC01FC0003F000FE0007E0007F000FE0003F001FC000 3F801FC0003F803FC0001FC03F80001FC07F80001FC07F80001FE07F80001FE0FF80001FE0FF80 001FE0FFFFFFFFE0FFFFFFFFE0FF80000000FF80000000FF80000000FF80000000FF800000007F 800000007F800000007F800000003FC00000003FC00000001FC00000E01FE00000E00FE00001C0 07F000038003F800070000FE000E00007FC07C00001FFFF0000001FF800023257EA428>101 D<01FC00000000FFFC00000000FFFC00000000FFFC0000000007FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC01FF000003FC07FFC00003FC1C0FF00003FC3007F80003FC6003F8 0003FCC003FC0003FC8001FC0003FD0001FE0003FF0001FE0003FE0001FE0003FE0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE00FFFFF07FFFF8FFFFF07FFFF8FFFFF07FFFF82D3A 7EB932>104 D<01FC03FE0000FFFC1FFFC000FFFC780FF000FFFDE003F80007FF8001FC0003FF 0000FE0003FE0000FF0003FC00007F8003FC00007FC003FC00003FC003FC00003FE003FC00003F E003FC00001FE003FC00001FE003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC 00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FE003FC00003F E003FC00003FE003FC00003FC003FC00003FC003FC00007F8003FC00007F8003FE0000FF0003FF 0001FE0003FF8003FC0003FDC007F80003FCF81FE00003FC3FFF800003FC07FC000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000000FFFFF00000 00FFFFF0000000FFFFF00000002C357EA432>112 D<01F80FC0FFF83FF0FFF870F8FFF8C1FC07 F883FE03F983FE03F903FE03FB03FE03FA01FC03FA00F803FA007003FE000003FC000003FC0000 03FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC00 0003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC 0000FFFFF800FFFFF800FFFFF8001F257EA424>114 D<001C0000001C0000001C0000001C0000 001C0000003C0000003C0000003C0000003C0000007C0000007C000000FC000000FC000001FC00 0003FC000007FC00001FFFFFC0FFFFFFC0FFFFFFC003FC000003FC000003FC000003FC000003FC 000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC000003FC000003FC000003FC000003FC000003FC00E003FC00E003FC00E003FC00E003FC00E0 03FC00E003FC00E003FC00E001FC00C001FC01C001FE01C000FE0380007F8700001FFE000007F8 001B357EB423>116 D E /Fm 18 122 df<008003800F80F38003800380038003800380038003 800380038003800380038003800380038003800380038003800380038003800380038003800380 038007C0FFFE0F217CA018>49 D<03F8000C1E001007002007804007C07807C07803C07807C038 07C0000780000780000700000F00000E0000380003F000001C00000F000007800007800003C000 03C00003E02003E07003E0F803E0F803E0F003C04003C0400780200780100F000C1C0003F00013 227EA018>51 D<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0F001 E0F001E0F001E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001C00001 C00001C0000380000380300300780700780600700C002018001030000FC00013227EA018>57 D<0007E0100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C00 00303C0000307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F800000078000000780000107C0000103C0000103C0000101C0000201E000020 0E000040070000400380008001C0010000E0020000381C000007E0001C247DA223>67 D<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00700F00F80F00F80F00F80E00F01E00401C0020380018700007C00014237EA119>74 D76 DI<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7 001E07003C0700780700700700F00708F00708F00708F00F087817083C23900FC1E015157E9418 >97 D<01FE000703000C07801C0780380300780000700000F00000F00000F00000F00000F00000 F00000F000007000007800403800401C00800C010007060001F80012157E9416>99 D<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000 E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I<01FC0007 07000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000F0000070 00007800203800201C00400E008007030000FC0013157F9416>I<0E0000FE00001E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E0070FFE7FF18237FA21B>104 D<0E0000FE00001E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E0180 0E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C0 0E01E00E00F00E00F8FFE3FE17237FA21A>107 D<0E00FE001E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E00FFE00B237FA20E>I<0E1F80FE60C01E80E00F00700F00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E 0070FFE7FF18157F941B>110 D<01FC000707000C01801800C03800E0700070700070F00078F0 0078F00078F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC0015 157F9418>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E 000E000E000F00FFF010157F9413>114 D 121 D E /Fn 1 49 df<038007C007C007C007800F800F800F000F001F001E001E001E001C003C 0038003800380070007000700060006000E000C00040000A1A7F9B0D>48 D E /Fo 20 121 df<00003FC0020001FFF8060007E01C06001F00070E003C00018E00780000DE 00F000005E01E000003E03C000001E078000001E0F0000000E0F0000000E1E000000061E000000 063E000000063C000000027C000000027C000000027C000000027800000000F800000000F80000 0000F800000000F800000000F800000000F800000000F800000000F800000000F800000000F800 00000078000000007C000000007C000000027C000000023C000000023E000000021E000000021E 000000040F000000040F0000000C078000000803C000001801E000001000F00000200078000040 003C000180001F0003000007E01E000001FFF80000003FC00027327CB02F>67 D73 D77 D80 D<007F004001FFC0400780F0C00F0018C01C000DC03C0007C0 380003C0780001C0700001C0F00000C0F00000C0F00000C0F0000040F0000040F0000040F80000 40F80000007C0000007E0000003F0000003FE000001FFE00000FFFE00007FFF80001FFFE00007F FF000007FF8000007F8000000FC0000007E0000003E0000003E0000001F0000001F0800000F080 0000F0800000F0800000F0800000F0C00000F0C00000E0E00001E0E00001E0F00001C0F8000380 EC000780C7000F00C1E03E0080FFF800801FE0001C327CB024>83 D<00FE0000070380000801E0 001000F0003C0078003E0078003E003C003E003C001C003C0000003C0000003C0000003C000007 FC00007C3C0003E03C0007803C001F003C003E003C003C003C007C003C0078003C08F8003C08F8 003C08F8003C08F8007C08F8007C087C00BC083E011E100F060FE003F807C01D1E7D9D20>97 D<018000003F800000FF800000FF8000000F800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 81F80007860700079803C007A000E007C000F007C00078078000380780003C0780003E0780001E 0780001E0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F078000 1E0780003E0780003C0780003C0780007807C00070074000F0072001C006100380060C0E000403 F80020317EB024>I<001FC00000F0380001C00400078002000F000F001E001F001E001F003C00 1F007C000E007C00000078000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000007C0000003C0000801E0000801E0001000F00010007800200 01C00C0000F03000001FC000191E7E9D1D>I<003F800000E0F0000380380007001C000E001E00 1E000E003C000F003C000F007C000F807800078078000780F8000780FFFFFF80F8000000F80000 00F8000000F8000000F8000000F800000078000000780000007C0000003C0000801C0000801E00 01000F0002000700020001C00C0000F03000001FC000191E7E9D1D>101 D<07000F801F801F800F8007000000000000000000000000000000000000000000000001801F80 FF80FF800F80078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800FC0FFF8FFF80D2F7EAE12>105 D<01803F80FF80FF800F80078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800FC0FFFCFFFC0E317EB012>108 D<0181FE003FC0003F860780C0F000FF8801C1003800FF90 01E2003C000FA000E4001C0007A000F4001E0007C000F8001E0007C000F8001E00078000F0001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341E7E9D38>I<01 81FC003F860F00FF880380FF9003C00FA001C007C001E007C001E007C001E0078001E0078001E0 078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00FC003F0FFFC3FFFFFFC 3FFF201E7E9D24>I<003FC00000E0700003801C0007000E000E0007001E0007803C0003C03C00 03C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F8 0001F0F80001F0780001E0780001E07C0003E03C0003C03C0003C01E0007800E00070007000E00 03801C0000E07000003FC0001C1E7E9D20>I<0181F8003F860F00FF9803C0FFA001E007C000F0 07C00078078000780780003C0780003E0780003E0780001E0780001F0780001F0780001F078000 1F0780001F0780001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C0 00F007C000F007A001C007900380078C0E000783F8000780000007800000078000000780000007 8000000780000007800000078000000780000007800000078000000FC00000FFFC0000FFFC0000 202C7E9D24>I<0183E03F8C18FF907CFF907C0FA07C07C03807C00007C00007C0000780000780 000780000780000780000780000780000780000780000780000780000780000780000780000780 000780000780000780000FC000FFFE00FFFE00161E7E9D19>114 D<03FC200E02603801E03000 E0600060E00060E00020E00020F00020F000207C00007F80003FFC001FFF0007FF8001FFE0000F E00001F08000F8800078800038C00038C00038C00038E00030F00070F00060C800C0C6038081FE 00151E7E9D19>I<00400000400000400000400000400000C00000C00000C00001C00001C00003 C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C01003C01003C01003C01003C01003C01003 C01003C01001C02001E02000E0400078C0001F00142B7FAA19>I<018000603F800FE0FF803FE0 FF803FE00F8003E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001 E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E00780 03E0078003E0078003E0038005E003C009E001C019F000F021FF001FC1FF201E7E9D24>I120 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%PaperSize: A4 %%EndSetup %%Page: 0 1 0 0 bop 811 1086 a Fo(Prop)r(osal)24 b(I)r(I)1129 1048 y Fn(0)575 1178 y Fo(MPI)f(Con)n(text)f(Sub)r(committee)799 1360 y Fm(Lyndon)17 b(J)f(Clark)o(e)853 1481 y(Marc)o(h)g(1993)p eop %%Page: 1 2 1 1 bop 262 619 a Fl(Chapter)28 b(1)262 826 y Fk(Prop)s(osal)36 b(I)s(I)803 765 y Fj(0)262 1042 y Fi(Editorial)12 b(Note:)18 b(This)13 b(is)g(not)h(Pr)n(op)n(osal)f(II)g(as)h(identi\014e)n(d)g(at)f(the) g(p)n(ost-me)n(et)g(lunch)h(after)262 1092 y(the)d(F)m(ebruary)f(me)n(eting)h (in)h(Dal)r(las.)17 b(R)o(ik)11 b(Little\014e)n(d)g(c)n(ame)g(on)h(b)n(o)n (ar)n(d)f(the)g(pr)n(op)n(osal)g(writing)262 1142 y(pr)n(o)n(c)n(ess,)j(and)i (de)n(cide)n(d)g(to)f(write)f(a)i(di\013er)n(ent)f(pr)n(op)n(osal)g(which)g (app)n(e)n(ars)h(as)f(Pr)n(op)n(osal)g(V.)262 1191 y(Ther)n(e)f(is)g(no)h (longer)f(a)h(pr)n(op)n(osal)g(which)f(c)n(ontains)h(static)f(c)n(ontexts)h (as)g(discusse)n(d)g(at)g(the)262 1241 y(p)n(ost-me)n(et)f(lunch.)262 1378 y Fh(1.1)64 b(In)n(tro)r(duction)262 1469 y Fg(This)9 b(c)o(hapter)h(prop)q(oses)g(that)g(comm)o(unicatio)o(n)d(con)o(texts)j(and)f (pro)q(cess)i(groupings)e(within)262 1519 y(MPI)20 b(app)q(ear)g(as)g (related)h(concepts.)38 b(In)20 b(particular)g(pro)q(cess)i(groupings)d(app)q (ear)i(as)262 1569 y(\\frames")14 b(whic)o(h)i(are)g(used)h(in)e(the)h (creation)h(of)e(comm)o(unicati)o(on)e(con)o(texts.)25 b(Comm)o(u-)262 1619 y(nications)13 b(con)o(texts)i(retain)g(a)e(reference)k(to,)d(and)g (inherit)g(prop)q(erties)h(of,)e(their)i(pro)q(cess)262 1669 y(grouping)f(frames.)22 b(This)15 b(re\015ects)i(the)f(observ)n(ation)f(that) h(an)f(in)o(v)o(o)q(cation)f(of)h(a)g(mo)q(dule)262 1718 y(in)c(a)i(parallel) e(program)f(t)o(ypically)h(op)q(erates)j(within)e(one)g(or)h(more)e(groups)i (of)e(pro)q(cesses,)262 1768 y(and)f(as)h(suc)o(h)h(an)o(y)e(comm)o (unication)d(con)o(texts)12 b(asso)q(ciated)g(with)e(in)o(v)o(o)q(cations)g (of)h(mo)q(dules)262 1818 y(also)i(bind)g(certain)i(seman)o(tics)e(of)h(pro)q (cess)h(groupings.)324 1868 y(The)20 b(prop)q(osal)f(pro)o(vides)g(pro)q (cess)i(iden)o(ti\014ed)f(comm)o(unicatio)o(n,)e(comm)o(unicati)o(ons)262 1918 y(whic)o(h)d(are)g(limited)e(in)i(scop)q(e)h(to)f(single)g(con)o(texts,) h(and)f(comm)o(unicatio)o(ns)e(whic)o(h)i(ha)o(v)o(e)262 1968 y(scop)q(e)g(spanning)e(pairs)h(of)f(con)o(texts.)19 b(The)c(prop)q(osal)e (mak)o(es)g(no)h(statemen)o(ts)g(regarding)262 2017 y(message)c(tags.)16 b(It)11 b(is)f(assumed)g(that)g(these)i(will)d(b)q(e)i(a)f(bit)g(string)g (expressed)j(as)d(an)g(in)o(teger)262 2067 y(in)j(the)h(host)h(language.)324 2117 y(Muc)o(h)d(of)e(this)i(prop)q(osal)f(m)o(ust)f(b)q(e)i(view)o(ed)f(as)h (recommendations)d(to)i(other)h(sub)q(com-)262 2167 y(mittees)g(of)g(MPI,)g (primarily)e(the)j(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(unication)e(sub)q (committee)j(and)262 2217 y(the)17 b(collectiv)o(e)f(comm)o(unicatio)o(ns)e (sub)q(committee.)25 b(Concrete)17 b(syn)o(tax)g(is)f(giv)o(en)g(in)g(the)262 2266 y(st)o(yle)e(of)f(the)h(ANSI)h(C)e(host)i(language)d(for)i(only)f(purp)q (oses)i(of)f(discussion.)324 2316 y(The)g(detailed)g(prop)q(osal)g(is)g (presen)o(ted)i(in)d Ff(Section)h(1.2.)p Fg(,)g(whic)o(h)f(refers)j(the)e (reader)262 2366 y(to)i(a)g(set)h(of)e(discussion)i(notes)g(in)f Ff(Section)g(1.3.2.)p Fg(.)26 b(The)17 b(notes)g(assumes)f(kno)o(wledge)262 2416 y(of)e(the)h(prop)q(osal)f(text)h(and)g(are)g(therefore)h(b)q(est)f (examined)f(after)h(famili)o(arisatio)o(n)d(with)967 2574 y(1)p eop %%Page: 2 3 2 2 bop 262 307 a Fg(that)13 b(text.)18 b(Asp)q(ects)d(of)d(the)i(prop)q (osal)f(are)g(discussed)i(in)d(section)i Ff(Section)f(1.3.1.)p Fg(,)g(and)262 357 y(it)g(is)g(also)g(recommended)f(that)i(this)f(material)f (b)q(e)i(read)g(after)g(famili)o(arisatio)o(n)d(with)i(the)262 407 y(text)h(of)f(the)i(prop)q(osal.)262 544 y Fh(1.2)64 b(Detailed)24 b(Prop)r(osal)262 635 y Fg(This)15 b(section)h(presen)o(ts)h(the)g(detailed)e (prop)q(osal,)g(discussing)h(in)f(order)h(of)f(app)q(earance:)262 685 y(pro)q(cesses;)k(pro)q(cess)g(groupings;)d(comm)o(unication)d(con)o (texts;)18 b(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(u-)262 735 y(nication;)c(collectiv)o(e)i(comm)o(unication.)262 851 y Fe(1.2.1)55 b(Pro)r(cesses)262 927 y Fg(This)13 b(prop)q(osal)h(views)f(pro)q(cesses)k (in)c(the)i(famili)o(ar)c(w)o(a)o(y)m(,)h(as)i(one)g(thinks)g(of)f(pro)q (cesses)j(in)262 977 y(Unix)f(or)i(NX)f(for)g(example.)24 b(Eac)o(h)16 b(pro)q(cess)i(is)e(a)g(distinct)h(space)g(of)f(instructions)h(and)262 1027 y(data.)g(Eac)o(h)c(pro)q(cess)h(is)f(allo)o(w)o(ed)e(to)h(comp)q(ose)h (m)o(ultiple)d(concurren)o(t)k(threads)g(and)e(MPI)262 1077 y(do)q(es)i(not)g(distinguish)f(suc)o(h)i(threads.)262 1185 y Ff(Pro)q(cess)g(Descriptor)262 1261 y Fg(Eac)o(h)h(pro)q(cess)h(is)f (describ)q(ed)h(b)o(y)f(a)f Fi(pr)n(o)n(c)n(ess)h(descriptor)k Fg(whic)o(h)15 b(is)h(expressed)i(as)e(an)f(in-)262 1311 y(teger)f(t)o(yp)q (e)f(in)g(the)h(host)f(language)f(and)h(has)h(an)e(opaque)i(v)n(alue)e(whic)o (h)h(is)g(pro)q(cess)i(lo)q(cal.)262 1361 y Ff(See)g(Note)g(1.)k(See)c(Note)h (3.)i(See)d(Note)h(2.)324 1411 y Fg(The)d(initialisation)c(of)j(MPI)h (services)h(will)d(assign)h(to)h(eac)o(h)g(pro)q(cess)h(an)e Fi(own)k Fg(pro)q(cess)262 1461 y(descriptor.)i(Eac)o(h)11 b(pro)q(cess)i(retains)e(its)g(o)o(wn)g(pro)q(cess)i(descriptor)f(un)o(til)e (the)h(termination)262 1510 y(of)h(MPI)h(services.)19 b(MPI)13 b(pro)o(vides)g(a)g(pro)q(cedure)h(whic)o(h)f(returns)h(the)g(o)o(wn)e (descriptor)i(of)262 1560 y(the)g(calling)e(pro)q(cess.)20 b(F)m(or)14 b(example,)e Fd(pd)21 b(=)h(mpi)p 1052 1560 14 2 v 15 w(own)p 1133 1560 V 15 w(pd\(\))p Fg(.)262 1668 y Ff(Pro)q(cess)15 b(Creation)f(and)h(Destruction)262 1745 y Fg(This)g(prop)q(osal)g(mak)o(es)f (no)h(statemen)o(ts)h(regarding)f(creation)h(and)f(destruction)i(of)e(pro-) 262 1795 y(cesses.)20 b Ff(See)15 b(Note)h(6.)262 1903 y(Descriptor)d(T)l (ransmission)262 1979 y Fg(The)18 b(v)n(alue)g(of)f(a)h(pro)q(cess)i (descriptor)g(can)e(b)q(e)h(transmitted)e(in)h(a)g(message)g(as)g(an)g(in-) 262 2029 y(teger)d(since)h(it)e(is)h(an)f(in)o(teger)i(t)o(yp)q(e)f(in)f(the) h(host)g(language.)20 b(Ho)o(w)o(ev)o(er)15 b(the)h(recipien)o(t)f(of)262 2079 y(the)h(descriptor)i(can)e(mak)o(e)f(no)h(de\014ned)h(use)h(of)d(the)i (v)n(alue)f(of)f(in)h(the)h(MPI)f(op)q(erations)262 2129 y(describ)q(ed)f(in) f(this)g(prop)q(osal)f(|)g(the)i(descriptor)g(is)f Fi(invalid)p Fg(.)324 2178 y(MPI)j(pro)o(vides)f(a)g(mec)o(hanism)f(whereb)o(y)i(the)g (user)h(can)e(transmit)g(a)g(v)n(alid)f(pro)q(cess)262 2228 y(descriptor)j(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 2278 y(tegrated)e(with)g(the)g(capabilit)o(y)e(to)i(transmit)e(t)o(yp)q(ed)j (messages.)j(It)13 b(is)h(suggested)h(that)f(a)262 2328 y(notional)e(data)i (t)o(yp)q(e)g(should)g(b)q(e)g(in)o(tro)q(duced)h(for)e(this)h(purp)q(ose,)h (e.g.)i Fd(MPI)p 1468 2328 V 16 w(PD)p 1528 2328 V 15 w(TYPE)p Fg(.)324 2378 y(MPI)10 b(pro)o(vides)g(a)f(pro)q(cess)j(descriptor)f (registry)f(service.)18 b(The)10 b(op)q(erations)g(whic)o(h)g(this)262 2427 y(service)16 b(upp)q(orts)h(are:)22 b(register)16 b(descriptor)h(b)o(y)e (name;)g(deregister)i(descriptor;)g(lo)q(okup)967 2574 y(2)p eop %%Page: 3 4 3 3 bop 262 307 a Fg(descriptor)11 b(iden)o(ti\014er)g(b)o(y)f(name)g(and)g (v)n(alidate,)f(blo)q(c)o(king)h(the)h(caller)f(un)o(til)g(the)h(name)e(has) 262 357 y(b)q(een)16 b(registered.)24 b Ff(See)17 b(Note)g(7.)23 b Fg(Use)16 b(of)f(this)g(service)i(is)e(not)g(mandated.)22 b(Programs)262 407 y(whic)o(h)d(can)g(con)o(v)o(enien)o(tly)g(b)q(e)h (expressed)h(without)e(using)g(the)g(service)i(can)e(ignore)g(it)262 457 y(without)13 b(p)q(enalt)o(y)m(.)324 506 y(Note)19 b(that)g(receipt)i(of) d(a)h(pro)q(cess)i(descriptor)f(in)f(the)g(fashions)g(describ)q(ed)i(ab)q(o)o (v)o(e)262 556 y(ma)o(y)11 b(ha)o(v)o(e)i(a)g(p)q(ersisten)o(t)i(e\013ect)g (on)e(the)h(implem)o(en)o(tation)c(of)j(MPI)g(at)g(the)h(receiv)o(er,)h(and) 262 606 y(in)h(particular)h(ma)o(y)f(reserv)o(e)j(state.)29 b(MPI)18 b(will)d(pro)o(vide)j(a)f(pro)q(cedure)i(whic)o(h)e(in)o(v)n(alid-) 262 656 y(ates)d(a)g(v)n(alid)f(descriptor,)i(allo)o(wing)d(the)j(implemen)o (tatio)o(n)d(to)i(free)h(reserv)o(ed)h(state.)k(F)m(or)262 706 y(example,)d Fd(mpi)p 510 706 14 2 v 15 w(invalidate)p 745 706 V 14 w(pd\(pd\))p Fg(.)29 b(The)19 b(user)g(is)f(not)h(allo)o(w)o(ed) e(to)h(in)o(v)n(alidate)e(the)262 756 y(pro)q(cess)f(descriptor)g(of)f(the)g (calling)f(pro)q(cess.)19 b Ff(See)d(Note)f(8.)262 863 y(Descriptor)e(A)o (ttribu)o(tes)262 940 y Fg(This)g(prop)q(osal)h(mak)o(es)f(no)g(statemen)o (ts)h(regarding)g(pro)q(cessor)i(descriptor)f(attributes.)262 1056 y Fe(1.2.2)55 b(Pro)r(cess)18 b(Groupings)262 1133 y Fg(This)d(prop)q (osal)h(views)g(a)f(pro)q(cess)j(grouping)d(as)h(an)g(ordered)h(collection)e (of)h(\(references)262 1183 y(to?\))h(distinct)d(pro)q(cesses,)h(the)f(mem)o (b)q(ership)e(and)h(ordering)g(of)g(whic)o(h)g(do)q(es)h(not)f(c)o(hange)262 1232 y(o)o(v)o(er)i(the)i(lifetime)d(of)h(the)h(grouping.)24 b Ff(See)17 b(Note)h(9.)25 b Fg(The)16 b(canonical)f(represen)o(tation)262 1282 y(of)d(a)i(grouping)e(re\015ects)k(the)e(pro)q(cess)i(ordering)d(and)h (is)f(a)g(one-to-one)h(map)e(from)f Fc(Z)1611 1288 y Fb(N)1657 1282 y Fg(to)262 1332 y(the)j(descriptor)h(of)e(the)i Fc(N)k Fg(pro)q(cesses)d(comp)q(osing)d(the)h(grouping.)324 1382 y(The)c(structure)j (of)c(a)h(pro)q(cess)i(grouping)d(is)h(de\014ned)i(b)o(y)e(a)f(pro)q(cess)j (grouping)e(top)q(ology)262 1432 y(and)j(this)h(prop)q(osal)g(mak)o(es)e(no)i (further)h(statemen)o(ts)f(regarding)g(suc)o(h)g(structures.)262 1540 y Ff(Group)g(Descriptor)262 1616 y Fg(Eac)o(h)20 b(group)g(is)g(iden)o (ti\014ed)g(b)o(y)g(a)f Fi(gr)n(oup)i(descriptor)j Fg(whic)o(h)c(is)g (expressed)i(as)e(an)g(in-)262 1666 y(teger)14 b(t)o(yp)q(e)f(in)g(the)h (host)f(language)f(and)h(has)h(an)e(opaque)i(v)n(alue)e(whic)o(h)h(is)g(pro)q (cess)i(lo)q(cal.)262 1716 y Ff(See)g(Note)g(1.)k(See)c(Note)h(4.)i(See)d (Note)h(2.)324 1766 y Fg(The)f(initialisation)d(of)j(MPI)g(services)i(will)c (assign)i(to)g(eac)o(h)g(pro)q(cess)i(an)e Fi(own)g Fg(group)262 1816 y(descriptor)c(for)f(a)g(pro)q(cess)i(grouping)d(of)h(whic)o(h)g(the)g (pro)q(cess)i(is)e(a)g(mem)o(b)q(er.)15 b(Eac)o(h)c(pro)q(cess)262 1865 y(retains)j(its)h(o)o(wn)f(group)g(descriptor)h(and)f(mem)o(b)q(ership)f (of)h(the)h(pro)q(cess)h(grouping)d(un)o(til)262 1915 y(the)18 b(termination)f(of)h(MPI)g(services.)34 b Ff(See)20 b(Note)h(10.)31 b Fg(MPI)19 b(pro)o(vides)g(a)f(pro)q(cedure)262 1965 y(whic)o(h)h(returns)j (the)f(descriptor)g(of)e(the)i(home)e(group)g(of)h(the)g(calling)f(pro)q (cess.)38 b(F)m(or)262 2015 y(example,)12 b Fd(gd)21 b(=)h(mpi)p 614 2015 V 15 w(home)p 717 2015 V 15 w(gd\(\))p Fg(.)17 b Ff(See)e(Note)g (11.)262 2123 y(Group)f(Creation)g(and)h(Deletion)262 2199 y Fg(MPI)d(pro)o(vides)g(a)f(pro)q(cedure)j(whic)o(h)e(allo)o(ws)e(users)k (to)d(dynamically)e(create)14 b(one)e(or)g(more)262 2249 y(groups)d(whic)o(h) g(are)h(subsets)h(of)e(existing)g(groups.)16 b(F)m(or)9 b(example,)g Fd(gdb)21 b(=)g(mpi)p 1489 2249 V 15 w(group)p 1614 2249 V 15 w(partition\(gda,)262 2299 y(key\))13 b Fg(creates)k(one)e(or)g(more)e (new)j(groups)f Fd(gdb)f Fg(partition)g(an)g(existing)h(group)f Fd(gda)g Fg(in)o(to)262 2349 y(one)h(or)h(more)e(distinct)i(subsets.)25 b(This)16 b(pro)q(cedure)h(is)f(called)f(b)o(y)g(and)h(sync)o(hronises)h(all) 262 2399 y(mem)o(b)q(ers)12 b(of)h Fd(gda)p Fg(.)967 2574 y(3)p eop %%Page: 4 5 4 4 bop 324 307 a Fg(MPI)9 b(pro)o(vides)h(a)f(pro)q(cedure)i(whic)o(h)f (allo)o(ws)e(users)j(to)e(dynamically)d(create)11 b(one)f(group)262 357 y(b)o(y)j(explicit)h(de\014nition)g(of)f(its)h(mem)o(b)q(ership)f(as)h(a) g(list)f(of)h(pro)q(cess)i(descriptors.)k(F)m(or)13 b(ex-)262 407 y(ample,)c Fd(gd)21 b(=)h(mpi)p 571 407 14 2 v 15 w(group)p 696 407 V 14 w(definition\(listofpd)o(\))8 b Fg(creates)k(one)f(new)g(group)g Fd(gd)f Fg(with)262 457 y(mem)o(b)q(ership)17 b(and)i(ordering)g(describ)q (ed)i(b)o(y)e(the)h(pro)q(cess)h(descriptor)f(list)f Fd(listofpd)p Fg(.)262 506 y(This)9 b(pro)q(cedure)j(is)e(called)f(b)o(y)h(and)g(sync)o (hronises)h(all)e(pro)q(cesses)j(iden)o(ti\014ed)e(in)g Fd(listofpd)p Fg(.)324 556 y(MPI)h(pro)o(vides)h(a)f(pro)q(cedure)i(whic)o(h)e(allo)o(ws)f (users)j(to)e(delete)i(a)e(created)i(group.)k(This)262 606 y(pro)q(cedure)11 b(accepts)h(the)f(descriptor)g(of)f(a)f(group)h(whic)o(h)g (w)o(as)g(created)i(b)o(y)d(the)i(calling)e(pro-)262 656 y(cess)15 b(and)f(destro)o(ys)i(the)f(iden)o(ti\014ed)f(group.)19 b(F)m(or)14 b(example,)e Fd(mpi)p 1295 656 V 15 w(group)p 1420 656 V 15 w(deletion\(gd\))262 706 y Fg(deletes)17 b(an)e(existing)h(group)f Fd(gd)p Fg(.)23 b(This)16 b(pro)q(cedure)i(is)d(called)h(b)o(y)f(and)h(sync)o (hronises)h(all)262 756 y(mem)o(b)q(ers)12 b(of)h Fd(gd)p Fg(.)324 805 y(MPI)k(pro)o(vides)g(additional)e(pro)q(cedure)k(whic)o(h)e(allo)o(w)e (users)j(to)f(construct)i(pro)q(cess)262 855 y(groupings)13 b(whic)o(h)h(ha)o(v)o(e)f(a)h(pro)q(cess)i(grouping)d(top)q(ology)m(.)262 963 y Ff(Descriptor)g(T)l(ransmission)262 1040 y Fg(The)i(v)n(alue)f(of)g(a)g (group)h(descriptor)h(can)f(b)q(e)g(transmitted)g(in)f(a)g(message)h(as)g(an) f(in)o(teger)262 1089 y(since)j(it)g(is)f(an)h(in)o(teger)g(t)o(yp)q(e)g(in)g (the)g(host)g(language.)26 b(Ho)o(w)o(ev)o(er)17 b(the)g(recipien)o(t)h(of)e (the)262 1139 y(descriptor)h(can)g(mak)o(e)e(no)h(de\014ned)h(use)h(of)d(the) i(v)n(alue)f(of)g(in)g(the)h(MPI)g(op)q(erations)f(de-)262 1189 y(scrib)q(ed)f(in)e(this)h(prop)q(osal)g(|)f(the)h(descriptor)i(is)d Fi(invalid)p Fg(.)324 1239 y(MPI)19 b(pro)o(vides)g(a)g(mec)o(hanism)d (whereb)o(y)k(the)g(user)g(can)f(transmit)f(a)g(v)n(alid)g(group)262 1289 y(descriptor)g(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 1339 y(tegrated)e(with)g(the)g(capabilit)o(y)e(to)i(transmit)e(t)o(yp)q(ed)j (messages.)j(It)13 b(is)h(suggested)h(that)f(a)262 1388 y(notional)e(data)i (t)o(yp)q(e)g(should)g(b)q(e)g(in)o(tro)q(duced)h(for)e(this)h(purp)q(ose,)h (e.g.)i Fd(MPI)p 1468 1388 V 16 w(GD)p 1528 1388 V 15 w(TYPE)p Fg(.)324 1438 y(MPI)c(pro)o(vides)f(a)h(group)f(descriptor)i(registry)f (service.)19 b(The)13 b(op)q(erations)g(whic)o(h)g(this)262 1488 y(service)j(upp)q(orts)h(are:)22 b(register)16 b(descriptor)h(b)o(y)e (name;)g(deregister)i(descriptor;)g(lo)q(okup)262 1538 y(descriptor)11 b(iden)o(ti\014er)g(b)o(y)f(name)g(and)g(v)n(alidate,)f(blo)q(c)o(king)h(the) h(caller)f(un)o(til)g(the)h(name)e(has)262 1588 y(b)q(een)16 b(registered.)24 b Ff(See)17 b(Note)g(7.)23 b Fg(Use)16 b(of)f(this)g (service)i(is)e(not)g(mandated.)22 b(Programs)262 1637 y(whic)o(h)d(can)g (con)o(v)o(enien)o(tly)g(b)q(e)h(expressed)h(without)e(using)g(the)g(service) i(can)e(ignore)g(it)262 1687 y(without)13 b(p)q(enalt)o(y)m(.)324 1737 y(Note)h(that)f(receipt)i(of)d(a)i(group)f(descriptor)h(in)f(the)h (fashions)f(describ)q(ed)i(ab)q(o)o(v)o(e)f(ma)o(y)262 1787 y(ha)o(v)o(e)i(a)g(p)q(ersisten)o(t)i(e\013ect)g(on)e(the)h(implemen)o (tation)c(of)j(MPI)g(at)h(the)g(receiv)o(er,)h(and)e(in)262 1837 y(particular)g(ma)o(y)f(reserv)o(e)k(state.)27 b(MPI)17 b(will)f(pro)o(vide)g(a)h(pro)q(cedure)h(whic)o(h)f(in)o(v)n(alidates)262 1886 y(a)e(v)n(alid)f(descriptor,)j(allo)o(wing)c(the)j(implemen)o(tatio)o(n) d(to)i(free)i(reserv)o(ed)g(state.)24 b(F)m(or)15 b(ex-)262 1936 y(ample,)d Fd(mpi)p 465 1936 V 15 w(invalidate)p 700 1936 V 13 w(gd\(gd\))p Fg(.)18 b(The)c(user)h(is)f(not)g(allo)o(w)o(ed)f(to)h(in)o (v)n(alidate)e(the)j(o)o(wn)262 1986 y(group)d(descriptor)j(of)d(the)i(pro)q (cess)h(or)e(the)h(group)f(descriptor)h(of)f(an)o(y)f(group)h(created)i(b)o (y)262 2036 y(the)f(calling)e(pro)q(cess.)20 b Ff(See)c(Note)f(12.)262 2144 y(Descriptor)e(A)o(ttribu)o(tes)262 2220 y Fg(MPI)j(pro)o(vides)g(a)g (pro)q(cedure)i(whic)o(h)e(accepts)i(a)d(v)n(alid)g(group)h(descriptor)h(and) f(returns)262 2270 y(the)g(rank)f(of)g(the)i(calling)d(pro)q(cess)j(within)e (the)i(iden)o(ti\014er)f(group.)23 b(F)m(or)15 b(example,)f Fd(rank)262 2320 y(=)21 b(mpi)p 374 2320 V 15 w(group)p 499 2320 V 15 w(rank\(gd\))p Fg(.)324 2370 y(MPI)10 b(pro)o(vides)h(a)f(pro)q (cedure)j(whic)o(h)d(accepts)i(a)e(v)n(alid)f(group)h(descriptor)i(and)e (returns)262 2420 y(the)15 b(n)o(um)o(b)q(er)g(of)g(mem)o(b)q(ers,)e(or)j Fi(size)p Fg(,)e(of)h(the)h(iden)o(ti\014ed)f(group.)22 b(F)m(or)15 b(example,)f Fd(size)21 b(=)967 2574 y Fg(4)p eop %%Page: 5 6 5 5 bop 262 307 a Fd(mpi)p 331 307 14 2 v 15 w(group)p 456 307 V 14 w(size\(gd\))p Fg(.)324 357 y(MPI)16 b(pro)o(vides)f(a)g(pro)q (cedure)j(whic)o(h)d(accepts)i(a)e(v)n(alid)f(group)i(descriptor)g(and)g (pro-)262 407 y(cess)h(order)g(n)o(um)o(b)q(er,)f(or)g Fi(r)n(ank)p Fg(,)g(and)g(returns)i(the)f(v)n(alid)e(descriptor)i(of)f(the)h(pro)q(cess)h (to)262 457 y(whic)o(h)d(the)h(supplied)g(rank)f(maps)g(within)f(the)j(iden)o (ti\014ed)e(group.)23 b(F)m(or)15 b(example,)f Fd(pd)22 b(=)262 506 y(mpi)p 331 506 V 15 w(group)p 456 506 V 14 w(pd\(gd,)f(rank\))p Fg(.)324 556 y Ff(See)15 b(Note)h(13.)324 606 y Fg(MPI)c(pro)o(vides)h (additional)e(pro)q(cedures)j(whic)o(h)f(allo)o(w)d(users)k(to)e(determine)h (the)g(pro-)262 656 y(cess)i(grouping)e(top)q(ology)g(attributes.)262 772 y Fe(1.2.3)55 b(Comm)n(unication)16 b(Con)n(texts)262 849 y Fg(This)c(prop)q(osal)h(views)g(a)g(comm)o(unicatio)o(n)d(con)o(text)k(as)f (a)g(uniquely)f(iden)o(ti\014ed)h(reference)262 899 y(to)f(exactly)g(one)h (pro)q(cess)h(grouping,)e(whic)o(h)g(is)g(a)h(\014eld)f(in)g(a)g(message)g (en)o(v)o(elop)q(e)h(and)g(ma)o(y)262 948 y(therefore)j(b)q(e)g(used)g(to)f (distinguish)g(messages.)21 b(The)16 b(con)o(text)g(inherits)f(the)h (referenced)262 998 y(pro)q(cess)i(grouping)e(as)h(a)g(\\frame".)25 b(Eac)o(h)17 b(pro)q(cess)i(grouping)d(is)h(used)h(as)f(a)f(frame)g(for)262 1048 y(m)o(ultiple)11 b(con)o(texts.)262 1156 y Ff(Con)o(text)j(Descriptor) 262 1232 y Fg(Eac)o(h)h(con)o(text)i(is)e(iden)o(ti\014ed)h(b)o(y)f(a)g Fi(c)n(ontext)i(descriptor)j Fg(whic)o(h)15 b(is)g(expressed)j(as)e(an)f(in-) 262 1282 y(teger)f(t)o(yp)q(e)f(in)g(the)h(host)f(language)f(and)h(has)h(an)e (opaque)i(v)n(alue)e(whic)o(h)h(is)g(pro)q(cess)i(lo)q(cal.)262 1332 y Ff(See)g(Note)g(1.)k(See)c(Note)h(5.)i(See)d(Note)h(2.)324 1382 y Fg(The)g(creation)g(of)e(MPI)i(pro)q(cess)h(groupings)e(allo)q(cates)h (an)f Fi(own)k Fg(con)o(text)d(whic)o(h)f(in-)262 1432 y(herits)k(the)g (created)h(grouping)e(as)h(a)f(frame)g(and)g(can)h(b)q(e)g(though)o(t)g(of)f (as)g(a)h(prop)q(ert)o(y)262 1482 y(of)c(the)h(created)h(grouping.)23 b(The)16 b(grouping)f(retains)h(its)f(o)o(wn)h(con)o(text)g(un)o(til)f(MPI)h (pro-)262 1531 y(cess)g(grouping)e(deletion.)21 b(MPI)15 b(pro)o(vides)g(a)g (pro)q(cedure)i(whic)o(h)d(accepts)j(a)d(v)n(alid)g(group)262 1581 y(descriptor)e(and)e(returns)i(the)g(con)o(text)f(descriptor)h(of)e(the) h(o)o(wn)g(con)o(text)g(of)f(the)h(iden)o(ti\014ed)262 1631 y(group.)17 b(F)m(or)d(example,)e Fd(cd)21 b(=)h(mpi)p 822 1631 V 15 w(own)p 903 1631 V 15 w(cd\(gd\))p Fg(.)17 b Ff(See)e(Note)h(15.) 262 1739 y(Con)o(text)e(Creation)h(and)g(Deletion)262 1816 y Fg(MPI)i(pro)o(vides)g(a)g(pro)q(cedure)i(whic)o(h)d(allo)o(ws)g(users)j (to)d(dynamically)f(create)j(con)o(texts.)262 1865 y(This)11 b(pro)q(cedure)j(accepts)f(a)e(v)n(alid)f(descriptor)j(of)e(a)g(group)h(of)f (whic)o(h)g(the)i(calling)d(pro)q(cess)262 1915 y(is)16 b(a)h(mem)o(b)q(er,)f (and)h(returns)i(a)e(con)o(text)g(descriptor)i(whic)o(h)e(references)j(the)d (iden)o(ti\014ed)262 1965 y(group.)j(F)m(or)15 b(example,)e Fd(cd)21 b(=)h(mpi)p 827 1965 V 15 w(context)p 996 1965 V 14 w(create\(gd\))p Fg(.)d(This)14 b(pro)q(cedure)j(is)e(called)262 2015 y(b)o(y)e(and)h(sync)o(hronises)h(all)e(mem)o(b)q(ers)g(of)g Fd(gd)p Fg(.)k Ff(See)f(Note)f(14.)324 2065 y Fg(MPI)i(pro)o(vides)h(a)f(pro) q(cedure)i(whic)o(h)e(allo)o(ws)f(users)i(to)f(destro)o(y)h(created)h(con)o (texts.)262 2114 y(This)11 b(pro)q(cedure)j(accepts)f(a)e(v)n(alid)f(con)o (text)j(descriptor)g(whic)o(h)e(w)o(as)h(created)h(b)o(y)e(the)i(call-)262 2164 y(ing)8 b(pro)q(cess)j(and)e(deletes)i(that)e(con)o(text)h(iden)o (ti\014er.)17 b(F)m(or)9 b(example,)f Fd(mpi)p 1400 2164 V 15 w(context)p 1569 2164 V 15 w(delete\(cd\))p Fg(.)262 2214 y(This)13 b(pro)q(cedure)j(is)e(called)f(b)o(y)h(and)g(sync)o(hronises)h(all) e(mem)o(b)q(ers)f(of)i(the)g(frame)f(of)g Fd(cd)p Fg(.)262 2322 y Ff(Descriptor)g(T)l(ransmission)262 2399 y Fg(The)18 b(v)n(alue)f(of)h(a)f(con)o(text)i(descriptor)g(can)f(b)q(e)h(transmitted)f (in)f(a)h(message)f(as)h(an)g(in-)262 2448 y(teger)d(since)h(it)e(is)h(an)f (in)o(teger)i(t)o(yp)q(e)f(in)f(the)h(host)g(language.)20 b(Ho)o(w)o(ev)o(er) 15 b(the)h(recipien)o(t)f(of)967 2574 y(5)p eop %%Page: 6 7 6 6 bop 262 307 a Fg(the)16 b(descriptor)i(can)e(mak)o(e)f(no)h(de\014ned)h (use)h(of)d(the)i(v)n(alue)f(of)f(in)h(the)h(MPI)f(op)q(erations)262 357 y(describ)q(ed)f(in)f(this)g(prop)q(osal)f(|)g(the)i(descriptor)g(is)f Fi(invalid)p Fg(.)324 407 y(MPI)i(pro)o(vides)h(a)f(mec)o(hanism)d(whereb)o (y)18 b(the)f(user)g(can)f(transmit)f(a)h(v)n(alid)f(con)o(text)262 457 y(descriptor)j(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 506 y(tegrated)e(with)g(the)g(capabilit)o(y)e(to)i(transmit)e(t)o(yp)q(ed)j (messages.)j(It)13 b(is)h(suggested)h(that)f(a)262 556 y(notional)e(data)i(t) o(yp)q(e)g(should)g(b)q(e)g(in)o(tro)q(duced)h(for)e(this)h(purp)q(ose,)h (e.g.)i Fd(MPI)p 1468 556 14 2 v 16 w(CD)p 1528 556 V 15 w(TYPE)p Fg(.)324 606 y(MPI)9 b(pro)o(vides)h(a)f(con)o(text)h(descriptor)h(registry)f (service.)18 b(The)10 b(op)q(erations)f(whic)o(h)h(this)262 656 y(service)16 b(upp)q(orts)h(are:)22 b(register)16 b(descriptor)h(b)o(y)e (name;)g(deregister)i(descriptor;)g(lo)q(okup)262 706 y(descriptor)11 b(iden)o(ti\014er)g(b)o(y)f(name)g(and)g(v)n(alidate,)f(blo)q(c)o(king)h(the) h(caller)f(un)o(til)g(the)h(name)e(has)262 756 y(b)q(een)16 b(registered.)24 b Ff(See)17 b(Note)g(7.)23 b Fg(Use)16 b(of)f(this)g (service)i(is)e(not)g(mandated.)22 b(Programs)262 805 y(whic)o(h)d(can)g(con) o(v)o(enien)o(tly)g(b)q(e)h(expressed)h(without)e(using)g(the)g(service)i (can)e(ignore)g(it)262 855 y(without)13 b(p)q(enalt)o(y)m(.)324 905 y(Note)e(that)g(receipt)h(of)f(a)f(con)o(text)i(descriptor)g(in)e(the)i (fashions)e(describ)q(ed)j(ab)q(o)o(v)o(e)e(ma)o(y)262 955 y(ha)o(v)o(e)16 b(a)g(p)q(ersisten)o(t)i(e\013ect)g(on)e(the)h(implemen)o (tation)c(of)j(MPI)g(at)h(the)g(receiv)o(er,)h(and)e(in)262 1005 y(particular)g(ma)o(y)f(reserv)o(e)k(state.)27 b(MPI)17 b(will)f(pro)o(vide)g(a)h(pro)q(cedure)h(whic)o(h)f(in)o(v)n(alidates)262 1054 y(a)e(v)n(alid)f(descriptor,)j(allo)o(wing)c(the)j(implemen)o(tatio)o(n) d(to)i(free)i(reserv)o(ed)g(state.)24 b(F)m(or)15 b(ex-)262 1104 y(ample,)d Fd(mpi)p 465 1104 V 15 w(invalidate)p 700 1104 V 13 w(cd\(cd\))p Fg(.)18 b(The)c(user)h(is)f(not)g(allo)o(w)o(ed)f(to)h(in)o (v)n(alidate)e(the)j(o)o(wn)262 1154 y(con)o(text)h(descriptor)h(of)e(a)g (group)h(or)g(the)g(con)o(text)g(descriptor)h(of)e(an)o(y)h(con)o(text)g (created)262 1204 y(b)o(y)d(the)i(calling)d(pro)q(cess.)20 b Ff(See)15 b(Note)h(16.)262 1312 y(Descriptor)d(A)o(ttribu)o(tes)262 1388 y Fg(MPI)f(pro)o(vides)g(a)g(pro)q(cedure)h(whic)o(h)f(allo)o(ws)f (users)i(to)f(determine)g(the)h(pro)q(cess)g(grouping)262 1438 y(whic)o(h)g(is)h(the)h(frame)d(of)h(a)h(con)o(text.)19 b(F)m(or)13 b(example,)f Fd(gd)22 b(=)f(mpi)p 1282 1438 V 15 w(context)p 1451 1438 V 15 w(gd\(cd\))p Fg(.)262 1554 y Fe(1.2.4)55 b(P)n(oin)n(t-to-P)n (oin)n(t)19 b(Comm)n(unication)262 1631 y Fg(This)11 b(prop)q(osal)g (recommends)f(three)j(forms)d(for)h(MPI)g(p)q(oin)o(t-to-p)q(oin)o(t)f (message)h(address-)262 1681 y(ing)k(and)h(selection:)23 b(n)o(ull)15 b(con)o(text;)j(closed)e(con)o(text;)i(op)q(en)e(con)o(text.)26 b(It)16 b(is)g(further)h(re-)262 1731 y(commended)d(that)j(messages)f(comm)o (unicated)e(in)i(eac)o(h)g(form)f(are)i(distinguished)f(suc)o(h)262 1780 y(that)c(a)h(Send)g(op)q(eration)g(of)f(form)g(X)g(cannot)h(matc)o(h)f (with)h(a)f(receiv)o(e)i(op)q(eration)f(of)f(form)262 1830 y(Y,)h(whic)o(h)h(requires)h(that)f(the)g(form)e(is)i(em)o(b)q(edded)g(in)o (to)f(the)i(message)e(en)o(v)o(elop)q(e.)324 1880 y(The)j(three)h(forms)d (are)i(describ)q(ed)h(follo)o(w)o(ed)d(b)o(y)i(considerations)g(of)f(uniform) e(in)o(teg-)262 1930 y(ration)g(of)g(these)i(forms)e(in)g(the)i(p)q(oin)o (t-to-p)q(oin)o(t)d(comm)o(unication)f(section)j(of)g(MPI.)262 2038 y Ff(Null)g(Con)o(text)g(F)l(orm)262 2114 y Fg(The)g Fi(nul)r(l)h(c)n (ontext)j Fg(form)12 b(con)o(tains)i(no)f(message)h(con)o(text.)19 b Ff(See)c(Note)h(17.)324 2164 y Fg(Message)g(selection)f(and)g(addressing)g (are)g(expressed)i(b)o(y)e Fd(\(pd,)21 b(tag\))14 b Fg(where:)20 b Fd(pd)15 b Fg(is)262 2214 y(a)e(pro)q(cess)j(descriptor;)f Fd(tag)e Fg(is)g(a)h(message)g(tag.)324 2264 y(Send)g(supplies)h(the)f Fd(pd)f Fg(of)h(the)g(receiv)o(er.)20 b(Receiv)o(e)14 b(supplies)h(the)f Fd(pd)f Fg(of)h(the)g(sender.)324 2314 y(Receiv)o(e)20 b(can)f(wildcard)g(on) g Fd(pd)g Fg(b)o(y)h(supplying)e(the)i(v)n(alue)f(of)g(a)g(named)f(constan)o (t)262 2363 y(pro)q(cess)f(descirptor,)h(e.g.)24 b Fd(MPI)p 773 2363 V 15 w(PD)p 832 2363 V 15 w(WILD)p Fg(.)15 b(This)h(prop)q(osal)g (mak)o(es)e(no)i(statmen)o(t)g(ab)q(out)262 2413 y(the)e(pro)o(vision)f(for)g (wildcard)h(on)g Fd(tag)p Fg(.)967 2574 y(6)p eop %%Page: 7 8 7 7 bop 262 307 a Ff(Closed)14 b(Con)o(text)h(F)l(orm)262 384 y Fg(The)f Fi(close)n(d)g(c)n(ontext)k Fg(form)12 b(p)q(ermits)h(comm)o (unication)d(b)q(et)o(w)o(een)15 b(mem)o(b)q(ers)d(of)h(the)h(same)262 434 y(con)o(text.)k Ff(See)d(Note)h(19.)324 483 y Fg(Message)d(selection)f (and)g(addressing)g(are)g(expressed)i(b)o(y)d Fd(\(cd,)21 b(rank,)g(tag\))11 b Fg(where:)262 533 y Fd(cd)k Fg(is)g(a)h(con)o(text)g(descriptor;)i Fd(rank)c Fg(is)i(a)f(pro)q(cess)j(rank)e(in)f(the)h(frame)f(of)g Fd(cd)p Fg(;)g Fd(tag)g Fg(is)h(a)262 583 y(message)d(tag.)324 633 y(Send)f(supplies)g(the)g Fd(cd)f Fg(of)g(the)h(receiv)o(er)h(and)e (sender,)i(and)f(the)g Fd(rank)f Fg(of)f(the)j(receiv)o(er.)262 683 y(Receiv)o(e)i(supplies)f(the)i Fd(cd)e Fg(of)f(the)j(sender)g(and)e (receiv)o(er,)i(and)e(the)h(rank)f(of)g(the)h(sender.)262 732 y(The)i Fd(\(cd,)k(rank\))16 b Fg(pair)g(in)h(Send)g(\(Receiv)o(e\))h(is)f (su\016cien)o(t)g(to)g(determine)g(the)h(pro)q(cess)262 782 y(descriptor)d(of)e(the)h(receiv)o(er)i(\(sender\).)324 832 y(Receiv)o(e)e(cannot)g(wildcard)f(on)h Fd(cd)p Fg(.)j(Receiv)o(e)d(can)g (wildcard)g(on)f Fd(rank)g Fg(b)o(y)g(supplying)262 882 y(the)e(v)n(alue)g (of)f(a)h(named)f(constan)o(t)h(in)o(teger,)h(e.g.)k Fd(MPI)p 1101 882 14 2 v 16 w(RANK)p 1205 882 V 14 w(WILD)p Fg(.)10 b(This)h(prop)q(osal)g(mak)o(es)262 932 y(no)i(statmen)o(t)h(ab)q(out)f(the)i (pro)o(vision)e(for)g(wildcard)h(on)f Fd(tag)p Fg(.)262 1040 y Ff(Op)q(en)i(Con)o(text)f(F)l(orm)262 1116 y Fg(The)k Fi(op)n(en)h(c)n (ontext)j Fg(form)16 b(p)q(ermits)h(comm)o(unication)e(b)q(et)o(w)o(een)k (mem)o(b)q(ers)e(of)g(an)o(y)g(t)o(w)o(o)262 1166 y(con)o(texts.)i Ff(See)c(Note)g(18.)324 1216 y Fg(Message)d(selection)h(and)e(addressing)h (are)g(expressed)i(b)o(y)d Fd(\(lcd,)21 b(rcd,)g(rank,)f(tag\))262 1266 y Fg(where:)d Fd(lcd)11 b Fg(is)h(a)f(\\lo)q(cal")f(con)o(text)i (descriptor;)h Fd(rcd)e Fg(is)g(a)g(\\remote")g(con)o(text)h(descriptor;)262 1316 y Fd(rank)h Fg(is)g(a)h(pro)q(cess)i(rank)d(in)h(the)g(frame)f(of)g Fd(rcd)p Fg(;)g Fd(tag)g Fg(is)h(a)f(message)h(tag.)324 1365 y(Send)22 b(supplies)g(the)h(con)o(text)f(descriptor)h(for)f(the)g(sender)i (in)d Fd(lcd)p Fg(,)h(the)h(con)o(text)262 1415 y(descriptor)c(for)f(the)h (receiv)o(er)h(in)e Fd(rcd)p Fg(,)h(and)f(the)h Fd(rank)e Fg(of)h(the)h (receiv)o(er)h(in)e(the)h(frame)262 1465 y(of)12 b Fd(rcd)p Fg(.)18 b(Receiv)o(e)c(supplies)f(the)h(con)o(text)g(descriptor)h(for)e(the)h (receiv)o(er)h(in)e Fd(lcd)p Fg(,)g(the)h(con-)262 1515 y(text)f(descriptor)i (for)d(the)i(sender)h(in)d Fd(rcd)p Fg(,)g(and)h(the)h Fd(rank)e Fg(of)g(the)i(sender)h(receiv)o(er)f(in)f(the)262 1565 y(frame)c(of)i Fd(rcd)p Fg(.)16 b(The)c Fd(\(rcd,)21 b(rank\))10 b Fg(pair)h(in)f(Send)i (\(Receiv)o(e\))g(is)f(su\016cien)o(t)h(to)f(determine)262 1614 y(the)j(pro)q(cess)i(descriptor)f(of)e(the)i(receiv)o(er)g(\(sender\).) 324 1664 y(Receiv)o(e)f(cannot)g(wildcard)f(on)h Fd(lcd)p Fg(.)j(Receiv)o(e)d (can)g(wildcard)f(on)h Fd(rcd)f Fg(b)o(y)g(supplying)262 1714 y(the)j(v)n(alue)g(of)f(a)h(named)f(constan)o(t)i(con)o(text)g(descriptor,)g (e.g.)25 b Fd(MPI)p 1352 1714 V 15 w(CD)p 1411 1714 V 15 w(WILD)p Fg(,)15 b(in)h(whic)o(h)262 1764 y(case)f(Receiv)o(e)g Fi(must)k Fg(also)14 b(wildcard)g(on)g Fd(rank)g Fg(as)h(there)h(is)e(insu\016cien)o(t) h(information)d(to)262 1814 y(determine)j(the)h(pro)q(cess)i(descriptor)f(of) e(the)h(sender.)26 b(Receiv)o(e)16 b(can)g(wildcard)f(on)g Fd(rank)262 1863 y Fg(b)o(y)f(supplying)h(the)h(v)n(alue)e(of)h(a)g(named)f (constan)o(t)i(in)o(teger,)f(e.g.)22 b Fd(MPI)p 1384 1863 V 15 w(RANK)p 1487 1863 V 15 w(WILD)p Fg(.)14 b(This)262 1913 y(prop)q(osal)f(mak)o(es)g(no)g(statmen)o(t)h(ab)q(out)g(the)g(pro)o(vision)f (for)h(wildcard)f(on)h Fd(tag)p Fg(.)262 2021 y Ff(Uniform)f(In)o(tegration) 262 2098 y Fg(The)j(three)h(forms)e(of)g(addressing)i(and)f(selection)g (describ)q(ed)i(ha)o(v)o(e)e(di\013eren)o(t)h(syn)o(tactic)262 2148 y(framew)o(orks.)29 b(W)m(e)17 b(can)h(consider)h(in)o(tegrating)e (these)j(forms)c(in)o(to)h(the)i(p)q(oin)o(t-to-p)q(oin)o(t)262 2197 y(section)c(of)e(MPI)i(b)o(y)f(de\014ning)g(a)g(further)h(orthogonal)e (axis)h(\(as)h(in)e(the)i(m)o(ulti-lev)o(el)d(pro-)262 2247 y(p)q(osal)j(of)g(Gropp)h(&)g(Lusk\))g(whic)o(h)g(deals)g(with)g(the)g(form.) 23 b(This)15 b(is)h(at)g(the)g(exp)q(ense)i(of)262 2297 y(m)o(ultiplyi)o(ng)c (the)k(n)o(um)o(b)q(er)e(of)g(Send)i(and)e(Receiv)o(e)i(pro)q(cedures)h(b)o (y)e(a)f(factor)h(of)g(three,)262 2347 y(and)f(some)g(di\016cult)o(y)g(in)g (details)h(of)f(the)i(curren)o(t)g(p)q(oin)o(t-to-p)q(oin)o(t)d(w)o(orking)h (do)q(cumen)o(t)262 2397 y(whic)o(h)d(uniformly)f(assumes)h(a)h(single)f (addressing)i(and)f(selection)g(form.)967 2574 y(7)p eop %%Page: 8 9 8 8 bop 324 307 a Fg(There)22 b(are)g(v)n(arious)f(approac)o(hes)h(to)f (uni\014cation)g(of)g(the)h(syn)o(tactic)g(framew)o(orks)262 357 y(whic)o(h)16 b(ma)o(y)e(simplify)f(in)o(tegration.)24 b(Three)17 b(options)f(are)h(no)o(w)e(describ)q(ed,)j(eac)o(h)f(based)262 407 y(on)e(reten)o(tion)i(and)e(extension)i(of)e(the)i(framew)o(ork)d(of)h (one)h(form.)22 b(These)c(options)d(eac)o(h)262 457 y(ha)o(v)o(e)e(adv)n(an)o (tages)h(and)f(disadv)n(an)o(tages.)262 565 y Ff(Option)f(i:)41 b Fg(The)14 b(framew)o(ork)e(of)h(the)g(op)q(en)h(con)o(text)g(form)e(is)h (adopted)g(and)h(extended.)324 614 y(W)m(e)g(in)o(tro)q(duce)h(the)g Fi(nul)r(l)k Fg(descriptor,)c(the)g(v)n(alue)f(of)g(whic)o(h)g(is)h (de\014ned)g(b)o(y)g(a)f(named)262 664 y(constan)o(t,)f(e.g.)18 b Fd(MPI)p 590 664 14 2 v 15 w(NULL)p Fg(.)324 714 y(The)j(n)o(ull)e(con)o (text)j(form)c(is)j(expressed)i(as)d Fd(\(MPI)p 1153 714 V 15 w(NULL,)h(MPI)p 1365 714 V 15 w(NULL,)g(pd,)g(tag\))p Fg(,)262 764 y(whic)o(h)16 b(is)h(a)f(little)h(clumsy)m(.)25 b(The)17 b(closed)g(con)o(text)h(form)d(is)i(expressed)i(as)e Fd(\(MPI)p 1573 764 V 15 w(NULL,)262 814 y(cd,)k(rank,)f(tag\))p Fg(,)15 b(whic)o(h)h(is)f(marginally)d(incon)o(v)o(enien)o(t.)23 b(The)17 b(op)q(en)f(con)o(text)g(form)e(is)262 863 y(expressed)i(as)e Fd(\(lcd,)20 b(rcd,)h(rank,)g(tag)p Fg(\),)13 b(whic)o(h)h(is)g(of)f(course)i (natural.)262 971 y Ff(Option)9 b(ii:)40 b Fg(The)11 b(framew)o(ork)d(of)i (the)g(closed)h(con)o(text)g(form)d(is)i(adopted)h(and)f(extended.)324 1021 y(W)m(e)k(in)o(tro)q(duce)h(the)g Fi(nul)r(l)k Fg(descriptor,)c(the)g(v) n(alue)f(of)g(whic)o(h)g(is)h(de\014ned)g(b)o(y)g(a)f(named)262 1071 y(constan)o(t,)f(e.g.)18 b Fd(MPI)p 590 1071 V 15 w(NULL)p Fg(.)324 1121 y(The)c(n)o(ull)f(con)o(text)h(form)e(is)i(expressed)i(as)e Fd(\(MPI)p 1106 1121 V 15 w(NULL,)20 b(pd,)i(tag\))p Fg(,)12 b(whic)o(h)i(is)f(mar-)262 1171 y(ginally)8 b(incon)o(v)o(enien)o(t.)17 b(The)10 b(closed)h(con)o(text)g(form)e(is)h(expressed)j(as)d Fd(\(cd,)21 b(rank,)g(tag\))p Fg(,)262 1220 y(whic)o(h)12 b(is)h(of)f(course) i(natural.)j(Expression)c(of)f(the)i(op)q(en)f(con)o(text)g(form)e(requires)j (a)e(little)262 1270 y(more)g(w)o(ork.)324 1320 y(W)m(e)k(can)i(use)g(the)f Fd(cd)g Fg(\014eld)g(as)g(\\shorthand)h(notation")e(for)g(the)i Fd(\(lcd,)j(rcd\))16 b Fg(pair)262 1370 y(at)d(the)h(exp)q(ense)i(of)d(in)o (tro)q(ducing)g(some)g(tric)o(k)o(ery)m(.)18 b(W)m(e)13 b(can)h(de\014ne)g(a) f(mec)o(hanism)f(whic)o(h)262 1420 y(\\globs")c(together)i(these)h(t)o(w)o(o) e(\014elds)h(on)o(to)f(in)g(an)g(iden)o(ti\014er)h(whic)o(h)f(Send)h(and)f (Receiv)o(e)h(can)262 1469 y(distinguish)i(from)f(a)h(con)o(text)i(iden)o (ti\014er)f(and)g(treat)h(as)e(the)i(shorthand)f(notation.)k(Then)262 1519 y(w)o(e)e(should)f(also)g(de\014ne)i(a)f(mec)o(hanism)d(b)o(y)j(whic)o (h)g(a)f(receiv)o(er)j(whic)o(h)e(has)g(completed)f(a)262 1569 y(Receiv)o(e)h(with)f(wildcard)g(on)g Fd(rcd)g Fg(is)h(able)f(to)g(determine) h(the)g(v)n(alid)e(con)o(text)i(descriptor)262 1619 y(of)e(the)h(sender.)20 b(This)14 b(is)f(a)h(incon)o(v)o(enien)o(t.)262 1727 y Ff(Option)e(iii:)39 b Fg(The)13 b(framew)o(ork)e(of)h(the)h(n)o(ull)f(con)o(text)h(form)e(is)h (adopted)h(and)g(extended.)324 1777 y(The)f(n)o(ull)f(con)o(text)i(form)d(is) i(expressed)j(as)d Fd(\(pd,)21 b(tag\))p Fg(,)11 b(whic)o(h)h(is)g(of)f (course)j(natural.)262 1826 y(Expression)g(of)g(the)g(op)q(en)g(and)g(closed) h(con)o(text)f(forms)f(requires)i(a)e(little)h(more)f(w)o(ork.)324 1876 y(W)m(e)g(can)g(use)h(the)g Fd(pd)f Fg(\014eld)g(as)g(\\shorthand)h (notation")e(for)h Fd(\(cd,)21 b(rank\))12 b Fg(and)h Fd(\(lcd,)262 1926 y(rcd,)20 b(rank\))15 b Fg(b)o(y)h(con)o(tin)o(uation)f(of)g(the)h(tric) o(k)o(ery)h(used)f(in)g(the)g(previous)h(option.)23 b(This)262 1976 y(is)13 b(clumsy)m(.)262 2092 y Fe(1.2.5)55 b(Collectiv)n(e)16 b(Comm)n(unication)262 2169 y Fg(Symmetric)d(collectiv)o(e)j(comm)o (unication)c(op)q(erations)k(are)h(complian)o(t)c(with)j(the)g(closed)262 2218 y(con)o(text)e(form)d(describ)q(ed)16 b(ab)q(o)o(v)o(e.)h(This)d(prop)q (osal)f(recommends)g(that)g(suc)o(h)h(op)q(erations)262 2268 y(accept)e(a)f(con)o(text)h(descriptor)h(whic)o(h)e(iden)o(ti\014es)h(the)g (con)o(text)g(and)g(frame)e(in)h(whic)o(h)g(they)262 2318 y(are)j(to)g(op)q (erate.)324 2368 y(MPI)h(do)q(es)g(plan)f(to)h(describ)q(e)h(symmetric)d (collectiv)o(e)i(comm)o(unicati)o(on)d(op)q(erations.)262 2418 y(It)k(is)h(not)g(p)q(ossible)g(to)g(determine)g(whether)h(the)f(prop)q(osal) g(is)f(su\016cien)o(t)i(to)e(allo)o(w)g(im-)967 2574 y(8)p eop %%Page: 9 10 9 9 bop 262 307 a Fg(plemen)o(tation)16 b(of)h(the)h(collectiv)o(e)g(comm)o (unication)d(section)j(of)g(MPI)g(in)f(terms)h(of)f(the)262 357 y(p)q(oin)o(t-to-p)q(oin)o(t)h(section)j(of)f(MPI)g(without)g(loss)g(of)f (generalit)o(y)m(,)i(since)g(the)g(collectiv)o(e)262 407 y(op)q(erations)14 b(are)g(not)g(y)o(et)g(de\014ned.)324 457 y(Assymetric)j(collectiv)o(e)g (comm)o(unication)d(op)q(erations,)j(esp)q(ecially)g(those)h(in)f(whic)o(h) 262 506 y(the)12 b(sender\(s\))i(and)d(receiv)o(er\(s\))j(are)e(distinct)g (pro)q(cesses,)i(are)e(complian)o(t)e(with)h(the)h(op)q(en)262 556 y(con)o(text)i(form)d(describ)q(ed)16 b(ab)q(o)o(v)o(e.)h(This)d(prop)q (osal)f(recommends)g(that)g(suc)o(h)h(op)q(erations)262 606 y(accept)i(a)g(pair)f(of)g(con)o(text)h(descriptors)h(\(p)q(erhaps)g(in)e(a)h (\\glob")e(form\))g(whic)o(h)h(iden)o(tify)262 656 y(the)f(con)o(texts)h(and) f(frames)f(in)g(whic)o(h)h(they)g(are)h(to)e(op)q(erate.)324 706 y(MPI)j(do)q(es)g(not)f(plan)g(to)h(describ)q(e)h(assymetric)f(collectiv) o(e)f(comm)o(unication)d(op)q(era-)262 756 y(tions.)17 b(Suc)o(h)11 b(op)q(erations)h(are)g(expressiv)o(e)h(when)e(writing)g(programs)f(b)q(ey)o (ond)i(the)g(SPMD)262 805 y(mo)q(del)d(and)i(comprise)g(comm)o(unicati)o(v)o (e)e(fun)o(tionally)g(distinct)i(pro)q(cess)i(groupings.)k(This)262 855 y(prop)q(osal)d(recommends)h(that)g(suc)o(h)h(op)q(erations)g(should)f(b) q(e)h(considered)g(in)f(some)g(rein-)262 905 y(carnation)e(of)g(MPI.)262 1042 y Fh(1.3)64 b(Discussion)24 b(&)d(Notes)262 1133 y Fg(This)14 b(section)h(comprises)g(a)f(discussion)h(of)f(certain)h(asp)q(ects)h(of)e (this)h(prop)q(osal)f(follo)o(w)o(ed)262 1183 y(b)o(y)f(the)i(notes)f (referenced)j(in)c(the)i(detailed)e(prop)q(osal.)262 1299 y Fe(1.3.1)55 b(Discussion)262 1376 y Fg(W)m(e)15 b(can)h(dissect)i(the)e(prop) q(osal)g(in)o(to)f(t)o(w)o(o)h(parts:)22 b(an)16 b(SPMD)g(mo)q(del)f(core;)i (an)f(MIMD)262 1426 y(mo)q(del)f(annex.)27 b(In)17 b(this)g(discussion)g(the) h(dissection)f(is)g(exp)q(osed)h(and)e(the)i(conceptual)262 1475 y(foundation)d(of)i(eac)o(h)h(part)f(is)g(describ)q(ed.)30 b(The)17 b(discussion)h(also)e(presen)o(ts)j(argumen)o(ts)262 1525 y(for)14 b(and)g(against)g(the)i(MIMD)e(mo)q(del)f(annex,)i(and)f(to)h (some)e(exten)o(t)j(explores)f(the)h(con-)262 1575 y(squences)f(of)f(these)h (argumen)o(ts)e(for)h(MPI)g(in)f(a)h(wider)g(sense.)262 1683 y Ff(SPMD)h(mo)q(del)f(core)262 1760 y Fg(The)e(SPMD)g(mo)q(del)e(core)j(pro) o(vides)f(noncomm)o(unicativ)o(e)d(pro)q(cess)14 b(groupings)d(and)h(com-)262 1809 y(m)o(unication)j(con)o(texts)k(for)f(writers)g(of)g(SPMD)f(parallel)g (libraries.)30 b(It)18 b(is)f(in)o(tended)i(to)262 1859 y(pro)o(vide)11 b(expressiv)o(e)i(p)q(o)o(w)o(er)f(b)q(ey)o(ond)g(the)h(\\SPIMD")e(mo)q(del,) f(in)i(whic)o(h)f(pro)q(cess)j(execute)262 1909 y(in)f(a)h(stricly)g(SIMD)f (fashion.)324 1959 y(The)h(material)e(describing)i(pro)q(cesses)j(in)c Ff(Section)h(1.2.1.)19 b Fg(is)14 b(simpli\014ed:)324 2050 y Fa(\017)20 b Fg(pro)q(cesses)d(ha)o(v)o(e)d(iden)o(tical)f(instruction)h (blo)q(c)o(ks)g(and)g(di\013eren)o(t)g(data)g(blo)q(c)o(ks;)324 2133 y Fa(\017)20 b Fg(pro)q(cess)k(descriptor)f(transmission)d(and)h (registry)i(b)q(ecome)e(redundan)o(t)h(\(Note,)365 2183 y(ho)o(w)o(ev)o(er,)g (that)e(they)h(are)f(an)o(yw)o(a)o(y)f(redundan)o(t)i(as)f(con)o(text)h (descriptor)g(trans-)365 2233 y(mission)13 b(and)g(registry)i(coupled)g(with) e(con)o(text)i(descriptor)h(attribute)e(query)h(and)365 2283 y(group)f(descriptor)h(attribute)f(query)g(is)f(capable)h(of)f(pro)o(viding)f (access)k(to)d(all)g(pro-)365 2332 y(cess)j(descriptors\);)324 2415 y Fa(\017)k Fg(dynamic)12 b(pro)q(cess)k(mo)q(dels)d(are)h(not)g (considered.)967 2574 y(9)p eop %%Page: 10 11 10 10 bop 324 307 a Fg(The)14 b(material)e(describing)i(pro)q(cess)i (groupings)e(in)f Ff(Section)h(1.2.2.)19 b Fg(is)13 b(simpli\014ed:)324 399 y Fa(\017)20 b Fg(group)9 b(descriptor)i(transmission)d(and)h(registry)h (b)q(ecome)f(redundan)o(t)h(\(Note,)g(ho)o(w)o(ev)o(er,)365 448 y(that)17 b(they)h(are)f(an)o(yw)o(a)o(y)f(redundan)o(t)h(as)g(con)o (text)h(descriptor)g(transmission)e(and)365 498 y(registry)c(coupled)f(with)f (con)o(text)h(descriptor)h(attribute)f(query)h(capable)e(of)g(pro)o(vid-)365 548 y(ing)j(access)j(to)e(all)f(group)g(descriptors.\);)324 631 y Fa(\017)20 b Fg(the)d(o)o(wn)f(pro)q(cess)i(grouping)d(explicitly)g(b)q (ecomes)h(a)g(group)g(con)o(taining)f(all)g(pro-)365 681 y(cesses.)324 772 y(The)g(material)d(describing)j(comm)o(unication)c(con)o(texts)16 b(in)e Ff(Section)g(1.2.3.)21 b Fg(is)14 b(sim-)262 822 y(pli\014ed:)324 913 y Fa(\017)20 b Fg(con)o(text)15 b(descriptor)g(transmission)e(and)g (registry)i(b)q(ecome)f(unnecessary)m(.)324 1005 y(The)f(material)e (describing)j(p)q(oin)o(t-to-p)q(oin)o(t)d(comm)o(unication)f(in)i Ff(Section)h(1.2.4.)19 b Fg(is)262 1054 y(simpli\014ed:)324 1146 y Fa(\017)h Fg(the)15 b(op)q(en)f(con)o(text)h(form)d(b)q(ecomes)i (redundan)o(t;)324 1229 y Fa(\017)20 b Fg(uniform)15 b(in)o(tegration)g (\\Option)h(i")g(is)g(deleted,)i(and)e(\\Option)h(ii")e(loses)i(\\globs")365 1279 y(th)o(us)f(b)q(ecoming)e(simple)g(enough)h(that)h(\\Option)e(iii")g (need)i(not)f(b)q(e)h(further)h(con-)365 1328 y(sidered.)324 1420 y(The)c(material)e(describing)i(collectiv)o(e)g(comm)o(unicatio)o(n)d (in)j Ff(Section)f(1.2.5.)19 b Fg(is)12 b(sim-)262 1469 y(pli\014ed:)324 1561 y Fa(\017)20 b Fg(there)h(is)e(no)h(p)q(ossibilit)o(y)e(of)h(collectiv)o (e)g(comm)o(unication)d(op)q(erations)k(spanning)365 1611 y(more)13 b(than)h(con)o(text.)262 1719 y Ff(MIMD)i(mo)q(del)e(annex)262 1795 y Fg(The)d(MIMD)g(mo)q(del)f(annex)i(extends)g(and)f(mo)q(di\014es)g (the)g(SPMD)h(mo)q(del)d(core)k(to)e(pro)o(vide)262 1845 y(expressiv)o(e)g(p) q(o)o(w)o(er)f(for)f(MIMD)h(programs)e(whic)o(h)i(com)o(bine)e(\(coarse)j (grain\))e(function)h(and)262 1895 y(data)17 b(driv)o(en)g(parallelism.)27 b(The)18 b(MIMD)f(mo)q(del)f(annex)i(is)g(not)f(in)o(tended)h(to)g(pro)o (vide)262 1945 y(expressiv)o(e)d(p)q(o)o(w)o(er)g(to)f(\014ne)h(grained)f (function)g(driv)o(en)g(parallel)f(programs)g(|)h(it)f(is)i(con-)262 1994 y(jectured)j(that)g(message)f(passing)g(approac)o(hes)h(suc)o(h)h(as)e (MPI)h(are)g(not)f(suited)h(to)f(\014ne)262 2044 y(grained)c(function)g(driv) o(en)g(or)h(data)f(driv)o(en)h(programmi)o(ng.)h(The)f(annex)f(is)h(in)o (tended)g(to)262 2094 y(pro)o(vide)f(expresiv)o(e)i(p)q(o)o(w)o(er)f(for)g (the)h(\\MSPMD")e(mo)q(del,)f(whic)o(h)i(is)f(no)o(w)h(describ)q(ed.)324 2144 y(One)19 b(of)g(the)g(simplest)f(MIMD)h(mo)q(dels)f(is)h(the)g (\\host-no)q(de")g(mo)q(del,)g(famili)o(ar)d(in)262 2194 y(EXPRESS)f(and)f(P) m(ARMA)o(CS,)f(con)o(taining)h(t)o(w)o(o)g(functional)g(groups:)19 b(one)c(no)q(de)g(group)262 2243 y(\(SPMD)f(lik)o(e\))f(;)g(one)h(host)g (group)g(\(a)g(singleton\).)324 2293 y(The)d(\\parallel)e(clien)o(t-serv)o (er")j(mo)q(del,)e(in)g(whic)o(h)h(eac)o(h)g(of)f(the)i Fc(n)e Fg(clien)o(ts)h(is)g(comp)q(osed)262 2343 y(of)i(parallel)g(pro)q(cesses,)k (and)d(in)g(whic)o(h)g(the)h(serv)o(er)g(ma)o(y)e(also)g(b)q(e)i(comp)q(osed) f(of)g(parallel)262 2393 y(pro)q(cesses,)k(con)o(tains)d(1)10 b(+)h Fc(n)k Fg(functional)g(groups:)21 b Fc(n)15 b Fg(clien)o(t)h(groups)g (\(SPMD)f(lik)o(e\);)g(one)262 2443 y(serv)o(er)i(group)f(\(singleton,)f (SPMD)h(lik)o(e\).)23 b(The)17 b(\\host-no)q(de")f(mo)q(del)e(is)i(a)f(case)i (of)f(this)957 2574 y(10)p eop %%Page: 11 12 11 11 bop 262 307 a Fg(mo)q(del)12 b(in)i(whic)o(h)h(the)f(host)h(can)g(b)q (e)g(view)o(ed)f(as)h(a)f(singleton)g(clien)o(t)g(and)g(the)h(no)q(des)g(can) 262 357 y(b)q(e)f(view)o(ed)g(as)g(an)g(SPMD)g(lik)o(e)f(serv)o(er)i(\(or)f (vice)g(v)o(ersa!\).)324 407 y(The)f(\\parallel)e(mo)q(dule)g(graph")h(mo)q (del,)f(in)h(whic)o(h)g(eac)o(h)h(mo)q(dule)e(within)h(the)h(graph)262 457 y(ma)o(y)j(b)q(e)k(comp)q(osed)e(of)g(parallel)g(pro)q(cesses)j (\(singleton,)f(SPMD)e(lik)o(e\),)h(con)o(tains)g(an)o(y)262 506 y(n)o(um)o(b)q(er)c(of)g(functional)g(groups)h(with)g(arbitrarily)f (complex)g(relations.)24 b(The)16 b(\\parallel)262 556 y(clien)o(t-serv)o(er) e(mo)q(del")d(is)i(a)g(case)h(of)e(this)h(mo)q(del)e(in)i(whic)o(h)g(the)g (mo)q(dule)f(graph)h(con)o(tains)262 606 y(arcs)h(joining)e(the)j(serv)o(er)g (to)f(eac)o(h)g(clien)o(t.)324 656 y(The)19 b(MIMD)g(mo)q(del)e(annex)j(is)e (in)o(tended)i(to)f(pro)o(vide)g(expressiv)o(e)h(p)q(o)o(w)o(er)f(for)g(the) 262 706 y(\\parallel)14 b(mo)q(dule)h(graph")h(mo)q(del,)f(whic)o(h)h(I)g (refer)h(to)g(as)f(the)h(MPSMD)f(mo)q(del.)24 b(This)262 756 y(mo)q(del)14 b(requires)i(supp)q(ort)h(at)e(some)g(lev)o(el)g(as)h (commercial)d(and)i(mo)q(dular)f(applications)262 805 y(are)i(increasingly)g (mo)o(ving)d(in)j(to)f(parallel)g(computating.)23 b(The)16 b(debate)h(is)f(whether)i(or)262 855 y(not)c(message)g(passing)g(approac)o (hes)i(suc)o(h)f(as)g(MPI)f(\(whic)o(h)h(I)f(simply)f(refer)i(to)g(as)f (MPI\))262 905 y(should)f(pro)o(vide)h(for)f(this)h(mo)q(del.)324 955 y(The)d(negativ)o(e)f(argumen)o(t)g(is)g(that)h(suc)o(h)g(SPMD)g(mo)q (dules)f(should)g(b)q(e)h(con)o(trolled)g(and)262 1005 y(comm)o(uni)o(cate)i (with)h(one)h(another)g(as)g(\\parallel)f(pro)q(cesses")j(at)d(the)i (distributed)f(op)q(er-)262 1054 y(ating)f(system)g(lev)o(el.)21 b(The)15 b(argumen)o(t)f(has)h(some)f(app)q(eal)g(as)h(the)h(w)o(orld)e(of)g (distributed)262 1104 y(op)q(erating)j(systems)h(m)o(ust)f(deal)h(with)f (di\016cult)g(issues)i(suc)o(h)g(as)f(pro)q(cess)h(con)o(trol)f(and)262 1154 y(coherency)m(.)23 b(Av)o(oidance)16 b(of)f(duplication)f(in)h(MPI)g (allo)o(ws)g(MPI)g(to)h(fo)q(cus)f(on)h(pro)o(vision)262 1204 y(of)11 b(a)h(smaller)f(set)i(of)e(facilities)g(with)h(greater)i(emphasis)d (on)h(maxim)n(um)c(p)q(erformance)k(for)262 1254 y(data)h(driv)o(en)h(SPMD)g (parallel)f(prgrams.)324 1303 y(The)h(p)q(ositiv)o(e)g(argumen)o(t)f(is)h (that)g(comm)o(unicatio)o(ns)e(b)q(et)o(w)o(een)j(SPMD)f(mo)q(dules)f(re-)262 1353 y(quires)19 b(high)f(p)q(erformance)h(and)g(MPI)g(can)h(pro)o(vide)f (suc)o(h)g(p)q(erformance)g(with)g(tuned)262 1403 y(seman)o(tics)11 b(whic)o(h)g(exp)q(ect)j(the)e(user)h(to)f(deal)f(with)g(coherency)j(issues.) k(There)13 b(is)f(also)f(the)262 1453 y(argumen)o(t)g(that)h(MPI)g(is)g(able) g(to)g(deal)g(with)g(this)g(in)f(a)h(shorter)i(time)c(than)i(dev)o(elopmen)o (t)262 1503 y(and)k(standardisation)h(pro)q(cedures)j(for)c(distributed)i(op) q(erating)f(systems.)28 b(The)17 b(latter)262 1553 y(argumen)o(t)10 b(is)h(comparable)f(with)i(the)g(argumen)o(t)e(for)h(MPI)h(v)o(ersus)h (parallel)d(compilation.)262 1669 y Fe(1.3.2)55 b(Notes)312 1745 y Fg(1.)20 b(Descriptors)i(are)e(a)g(plen)o(tiful)f(but)i(b)q(ounded)f (resource.)39 b(They)21 b(are)f(expressed)365 1795 y(as)g(in)o(tegers)h(for)e (t)o(w)o(o)g(reasons.)37 b(Firstly)m(,)20 b(this)g(expression)h(facilitates)e (uniform)365 1845 y(binding)12 b(to)h(ANSI)h(C)f(and)g(F)m(ortran)g(77.)k (Secondly)m(,)c(there)h(is)f(a)g(later)g(con)o(v)o(enience)365 1895 y(in)e(ensuring)h(that)f(pro)q(cess)i(descriptors)g(in)d(particular)h (are)h(expressed)i(in)c(in)o(tegers,)365 1945 y(and)j(I)f(made)f(all)h(the)h (descriptors)h(are)f(in)o(tegers)g(for)f(consistency)m(.)19 b(In)12 b(practice)i(the)365 1994 y(descriptor)h(could)e(b)q(e)h(an)f(index)g (in)o(to)f(a)h(table)g(of)g(ob)r(jects)h(description)g(structures)365 2044 y(or)g(p)q(oin)o(ters)h(to)e(suc)o(h)i(structures.)312 2127 y(2.)20 b(It)10 b(is)g(p)q(ossible)g(to)g(allo)o(w)e(the)j(user)g(to)f (sa)o(v)o(e)g(user)h(de\014ned)g(attributes)g(in)e(descriptors,)365 2177 y(as)14 b(Little\014eld)f(has)g(suggested.)19 b(Suc)o(h)14 b(attributes)g(should)f(not)g(b)q(e)h(comm)o(unicated)365 2227 y(in)g(either)g(descriptor)i(transmission)c(or)i(the)h(descriptor)g(registry) m(.)312 2310 y(3.)20 b(The)c(pro)q(cess)h(descriptor)f(is)f(not)g(a)g(global) e(unique)i(pro)q(cess)i(iden)o(ti\014er)e(but)h(m)o(ust)365 2360 y(reference)i(suc)o(h)f(an)e(iden)o(ti\014er.)24 b(The)16 b(use)g(of)f(descriptors)i(hides)g(suc)o(h)f(iden)o(ti\014ers)365 2410 y(in)f(order)i(that)e(they)h(ma)o(y)e(b)q(e)i(implemen)o(tatio)o(n)d (dep)q(enden)o(t,)k(and)e(enables)i(more)957 2574 y(11)p eop %%Page: 12 13 12 12 bop 365 307 a Fg(e\016cien)o(t)14 b(access)i(to)d(additional)f(pro)q (cess)j(information)c(in)i(implemen)o(tations.)i(F)m(or)365 357 y(example,)d(the)h(pro)q(cess)i(iden)o(ti\014er)e(of)g(a)g(receiv)o(er)h (ma)o(y)d(not)i(b)q(e)h(su\016cien)o(t)f(to)g(route)365 407 y(a)f(message)f(to)h(the)g(receiv)o(er,)i(and)d(this)h(information)d(can)j(b) q(e)h(referenced)h(from)c(the)365 457 y(pro)q(cess)16 b(descriptor.)312 540 y(4.)k(The)h(group)e(descriptor)j(is)e(not)f(a)h(global)e(unique)i(group) g(iden)o(ti\014er)g(but)h(m)o(ust)365 589 y(reference)d(suc)o(h)f(an)e(iden)o (ti\014er.)24 b(The)16 b(use)g(of)f(descriptors)i(hides)g(suc)o(h)f(iden)o (ti\014ers)365 639 y(in)e(order)g(that)g(they)g(ma)o(y)e(b)q(e)i(implemen)o (tation)c(dep)q(endeen)o(t,)16 b(and)d(enables)i(more)365 689 y(e\016cien)o(t)i(access)h(to)f(additional)d(group)i(information)e(in)i (implem)o(en)o(tations.)23 b(F)m(or)365 739 y(example,)11 b(the)h(size)g(of)g (the)g(group)f(and)h(the)g(map)e(from)g(mem)o(b)q(er)g(ranks)i(to)g(pro)q (cess)365 789 y(descriptors)k(can)e(b)q(e)g(referenced)j(from)12 b(the)i(group)g(descriptor.)312 872 y(5.)20 b(The)15 b(con)o(text)g (descriptor)h(is)f(not)f(a)g(global)f(unique)i(con)o(text)g(iden)o(ti\014er)g (but)g(m)o(ust)365 922 y(reference)j(suc)o(h)f(an)e(iden)o(ti\014er.)24 b(The)16 b(use)g(of)f(descriptors)i(hides)g(suc)o(h)f(iden)o(ti\014ers)365 971 y(in)f(order)i(that)e(they)h(ma)o(y)e(b)q(e)i(implemen)o(tatio)o(n)d(dep) q(enden)o(t,)k(and)e(enables)i(more)365 1021 y(e\016cien)o(t)d(access)h(to)e (additional)e(con)o(text)j(information)d(in)h(implemen)o(tations.)j(F)m(or) 365 1071 y(example,)j(the)h(group)f(descriptor)i(of)d(the)i(frame)e(can)i(b)q (e)g(referenced)h(from)d(the)365 1121 y(con)o(text)e(descriptor.)312 1204 y(6.)20 b(The)h(prop)q(osal)e(do)q(es)i(not)f(prev)o(en)o(t)h(a)f(pro)q (cess)h(mo)q(del)e(whic)o(h)h(allo)o(ws)e(dynamic)365 1254 y(creation)13 b(and)g(termination)e(of)h(pro)q(cesses)j(ho)o(w)o(ev)o(er)e (it)f(do)q(es)i(not)f(fa)o(v)o(our)e(an)i(asyn-)365 1303 y(c)o(hronous)i(pro) q(cess)h(creation)f(mo)q(del)e(in)g(whic)o(h)i(singleton)e(pro)q(cesses)k (are)e(created)365 1353 y(and)h(terminated)g(in)g(an)g(unstructured)j (fashion.)25 b(The)16 b(prop)q(osal)g(do)q(es)h(fa)o(v)o(our)f(a)365 1403 y(mo)q(del)d(in)h(whic)o(h)g(blobs)g(of)f(pro)q(cesses)k(are)d(created)i (\(and)e(terminated\))g(in)g(a)g(con-)365 1453 y(certed)i(fashion,)d(and)h (in)g(whic)o(h)g(eac)o(h)h(group)f(so)g(created)i(is)e(assigned)h(a)f (di\013eren)o(t)365 1503 y(o)o(wn)g(pro)q(cess)j(grouping.)h(This)d(mo)q(del) e(do)q(es)i(not)f(tak)o(e)h(in)o(to)e(accoun)o(t)i(the)g(p)q(oten-)365 1553 y(tial)e(desire)j(to)e(expand)g(or)g(con)o(tract)h(an)f(existing)g(blob) g(of)f(pro)q(cesses)k(in)d(order)h(to)365 1602 y(tak)o(e)d(in)o(to)g(accoun)o (t)g(\(presumably)g(slo)o(wly\))f(time)f(v)n(arying)h(w)o(orloads.)17 b(The)c(author)365 1652 y(suggests)21 b(that)f(concerted)h(blob)e(expand)h (and)f(con)o(tract)h(op)q(erations)g(are)g(most)365 1702 y(suitable)14 b(for)g(this)g(purp)q(ose)h(than)e(async)o(hronous)i(spa)o(wn/kill)d(op)q (erations.)312 1785 y(7.)20 b(There)15 b(should)f(probably)f(also)g(b)q(e)h (a)g(registry)g(op)q(eration)g(whic)o(h)f(c)o(hec)o(ks)i(whether)365 1835 y(a)f(name)e(has)i(b)q(een)g(registered)i(without)d(the)h(p)q(ossibilit) o(y)e(of)h(blo)q(c)o(king)g(the)h(calling)365 1885 y(pro)q(cess)k (inde\014nitely)m(.)25 b(The)16 b(same)g(registry)h(could)f(b)q(e)h(used)g (for)f(pro)q(cess,)i(group)365 1934 y(and)c(con)o(text)h(descriptors.)312 2017 y(8.)20 b(In)f(the)g(static)g(pro)q(cess)h(mo)q(del)d(it)h(is)g(assumed) h(that)f(pro)q(cesses)j(are)e(created)h(in)365 2067 y(concert,)h(and)e(that)f (termination)f(of)h(one)h(pro)q(cess)h(implies)d(termination)g(of)h(all)365 2117 y(pro)q(cesses.)39 b(In)20 b(this)h(case)g(there)g(is)f(no)g(coherency)h (problem)e(asso)q(ciated)i(with)365 2167 y(pro)q(cesses)c(and)e(pro)q(cess)h (descriptors.)21 b(In)14 b(a)g(dynamic)f(pro)q(cess)j(mo)q(del)d(there)j(is)e (a)365 2217 y(coherency)f(problem)c(whic)o(h)i(pro)q(cesses)j(and)c(pro)q (cess)j(descriptors)g(since)e(a)g(pro)q(cess)365 2267 y(can)f(retain)g(the)g (descriptor)h(of)e(a)g(pro)q(cess)i(whic)o(h)f(has)g(b)q(een)g(deleted.)18 b(The)10 b(prop)q(osal)365 2316 y(exp)q(ects)16 b(the)f(user)g(to)e(ensure)j (coheren)o(t)f(usage.)312 2399 y(9.)20 b(Pro)q(cess)f(groupings)c(are)i (dynamic)d(in)i(the)h(sense)h(that)e(they)h(can)f(b)q(e)h(created)h(at)365 2449 y(an)o(y)12 b(time,)e(and)i(static)g(in)f(the)h(sense)i(that)e(the)g (mem)o(b)q(ership)e(is)i(constan)o(t)g(o)o(v)o(er)g(the)957 2574 y(12)p eop %%Page: 13 14 13 13 bop 365 307 a Fg(lifetime)11 b(of)g(the)j(pro)q(cess)g(grouping.)j(The) 12 b(author)h(suggests)h(concerted)g(group)e(ex-)365 357 y(pand)f(and)f(con)o (tract)i(op)q(erations)f(are)g(more)e(generally)i(useful)f(than)h(async)o (hronous)365 407 y(join/lea)o(v)o(e)i(op)q(erations.)291 490 y(10.)20 b(MPI)15 b(has)f(discussed)j(the)e(concept)g(of)f(the)h(\\all")e (group)h(whic)o(h)g(con)o(tains)h(all)e(pro-)365 540 y(cesses.)35 b(The)19 b(\\o)o(wn")e(group)i(concept)h(is)e(in)o(tended)h(to)g(b)q(e)g(a)f (generalisation)g(of)365 589 y(the)h(\\all")d(group)h(concept)i(whic)o(h)f (is)g(expressiv)o(e)h(for)e(programs)g(including)g(and)365 639 y(b)q(ey)o(ond)g(the)g(SPMD)f(mo)q(del.)24 b(Pro)q(cesses)19 b(are)d(created)i(in)e(\\blobs",)g(where)h(eac)o(h)365 689 y(mem)o(b)q(er)g(of)g(a)h(blob)f(is)h(a)f(mem)o(b)q(er)g(of)g(the)i(same)e(o) o(wn)g(pro)q(cess)j(grouping,)e(and)365 739 y(di\013eren)o(t)e(blobs)e(ha)o (v)o(e)g(di\013eren)o(t)h(o)o(wn)f(pro)q(cess)i(groupings.)j(An)14 b(SPMD)g(program)365 789 y(is)f(a)h(single)e(blob.)18 b(A)13 b(host-no)q(de)h(program)e(comp)q(oses)h(t)o(w)o(o)g(blobs,)g(the)g(no)q(de)h (blob)365 839 y(and)f(the)h(host)g(blob)f(\(whic)o(h)g(is)g(a)g(singleton\).) 18 b(There)c(is)f(a)g(sense)i(in)e(whic)o(h)g(a)g(blob)365 888 y(has)h(a)g(lo)q(cally)e(SPMD)i(view.)291 971 y(11.)20 b(This)14 b(pro)q(cedure)i(lo)q(oks)d(lik)o(e)g(a)h(pro)q(cess)i(descriptor)f (attribute)f(query)m(.)291 1054 y(12.)20 b(Dynamic)8 b(pro)q(cesses)j(in)o (tro)q(duce)f(a)f(group)h(coherency)h(problem)d(as)h(a)g(group)g(descriptor) 365 1104 y(can)18 b(con)o(tain)f(the)h(pro)q(cess)h(descriptor)f(of)f(a)g (pro)q(cess)i(whic)o(h)e(has)h(b)q(een)g(deleted.)365 1154 y(Dynamic)12 b(pro)q(cesses)17 b(p)q(oten)o(tially)c(in)o(tro)q(duce)i(a)e (group)h(coherency)i(problem)d(as)h(a)365 1204 y(group)e(descriptor)h(can)f (con)o(tain)g(the)g(pro)q(cess)i(descriptor)f(of)e(a)h(pro)q(cess)i(whic)o(h) d(has)365 1254 y(b)q(een)h(deleted.)17 b(Group)10 b(transmission)f(in)o(tro)q (duces)i(a)f(group)g(and)g(group)g(descriptor)365 1303 y(coherenncy)21 b(problem)d(since)i(a)f(pro)q(cess)j(can)d(retain)h(a)f(group)g(descriptor)h (of)f(a)365 1353 y(group)c(whic)o(h)f(has)h(b)q(een)g(iden)o(ti\014ed)g (group.)20 b(The)14 b(prop)q(osal)h(exp)q(ects)h(the)f(user)h(to)365 1403 y(ensure)g(coheren)o(t)f(usage.)291 1486 y(13.)20 b(The)c(prop)q(osal)f (did)f(not)h(include)g(a)g(function)g(to)g(con)o(v)o(ert)h(a)e Fd(\(gd,)21 b(pd\))15 b Fg(pair)f(in)o(to)365 1536 y(a)i(rank.)22 b(It)16 b(is)f(suggested)i(that)f(this)f(in)o(v)o(erse)i(map)d(is)h(allo)o(w) o(ed)f(to)i(b)q(e)g(arbitrarily)365 1586 y(slo)o(w.)291 1669 y(14.)k(Marc)15 b(Snir)e(has)h(describ)q(ed)i(a)d(metho)q(d)g(b)o(y)h(whic)o (h)g(global)e(unique)h(group)h(iden)o(ti\014-)365 1719 y(ers)j(can)e(b)q(e)h (generated)h(without)e(use)h(of)f(shared)h(global)e(data.)22 b(The)16 b(description)365 1768 y(of)i(con)o(text)h(creation)g(an)o (ticipates)g(a)f(similar)e(metho)q(d)i(for)g(global)f(unique)i(con-)365 1818 y(text)h(iden)o(ti\014er)g(generation.)35 b(Ho)o(w)o(ev)o(er)20 b(the)g(sync)o(hronisation)f(requiremen)o(t)g(of)365 1868 y(this)e(metho)q(d) e(mak)o(es)h(it)g(unnecessary)i(for)e(con)o(text)h(creation.)26 b(Sync)o(hronisation)365 1918 y(at)11 b(creation)g(of)f(con)o(text)i(within)e (a)g(pro)q(cess)j(grouping)d(frame)f(implies)g(that)i(all)f(pro-)365 1968 y(cesses)19 b(within)d(the)h(frame)e(p)q(erform)g(the)i(same)f(con)o (text)h(creations)g(in)f(the)g(same)365 2017 y(sequence.)39 b(Therefore)22 b(the)f(com)o(bination)c(of)j(global)e(unique)i(frame)f(iden)o (ti\014er)365 2067 y(and)e(con)o(text)h(creation)f(sequence)i(n)o(um)o(b)q (er)e(pro)o(vides)g(a)g(global)e(unique)i(con)o(text)365 2117 y(iden)o(ti\014er)e(without)e(comm)o(unication.)291 2200 y(15.)20 b(This)14 b(pro)q(cedure)i(lo)q(oks)d(lik)o(e)g(a)h(group)g(descriptor)h (attribute)f(query)m(.)291 2283 y(16.)20 b(The)13 b(reten)o(tion)h(of)e(a)g (reference)k(to)c(a)h(group)f(descriptor)i(b)o(y)f(a)f(con)o(text)i(in)o(tro) q(duces)365 2333 y(the)i(p)q(oten)o(tial)e(for)h(a)f(con)o(text)i(and)f(con)o (text)g(descriptor)h(coherency)h(problem,)d(as)365 2383 y(a)19 b(con)o(text)h(could)e(con)o(tain)h(a)f(reference)k(to)c(the)i(group)f (descriptor)h(of)e(a)h(group)365 2433 y(whic)o(h)c(has)h(b)q(een)g(deleted.) 24 b(This)15 b(could)g(b)q(e)h(circum)o(v)o(en)o(ted)f(b)o(y)g(demanding)f (that)957 2574 y(13)p eop %%Page: 14 15 14 14 bop 365 307 a Fg(all)11 b(suc)o(h)i(con)o(texts)h(are)e(deleted)i(b)q (efore)f(a)f(group)g(is)g(deleted.)19 b(Con)o(text)12 b(descriptor)365 357 y(transmission)17 b(in)o(tro)q(duces)i(a)e(coherency)j(problem)c(since)j (a)e(pro)q(cess)j(can)d(retain)365 407 y(the)d(descriptor)h(of)e(a)g(con)o (text)i(whic)o(h)e(has)h(b)q(een)g(deleted.)19 b(The)14 b(prop)q(osal)g(exp)q (ects)365 457 y(the)h(user)g(to)f(ensure)h(coheren)o(t)g(usage.)291 540 y(17.)20 b(This)14 b(is)f(somewhat)f(lik)o(e)h(and)g(P)m(ARMA)o(CS)g(and) g(PVM)g(3.)18 b(It)13 b(is)g(general)h(purp)q(ose,)365 589 y(but)f(not)f(particularly)f(expressiv)o(e.)19 b(It)13 b(do)q(es)g(not)f(pro) o(vide)g(facilities)f(for)h(writers)h(of)365 639 y(parallel)g(libraries.)291 722 y(18.)20 b(This)15 b(is)g(somewhat)f(lik)o(e)g(ZIPCODE)i(and)e(VENUS.)h (It)g(is)g(expressiv)o(e)i(in)d(certain)365 772 y(SPMD)g(programs)f(where)i (noncomm)o(uni)o(cativ)o(e)c(data)j(driv)o(en)f(parallel)g(computa-)365 822 y(tions)k(can)g(b)q(e)g(p)q(erformed)g(concurren)o(tly)m(.)27 b(It)17 b(pro)o(vides)g(facilities)f(for)g(writers)i(of)365 872 y(SPMD)c(lik)o(e)f(parallel)g(libraries.)291 955 y(19.)20 b(This)f(is)f(somewhat)g(lik)o(e)g(CHIMP)h(and)g(PVM)g(2.)32 b(It)19 b(is)f(expressiv)o(e)j(in)d(certain)365 1005 y(MIMD)13 b(programs)e(where)i(comm)o(unicativ)o(e)d(data)i(driv)o(en)h(parallel)e (computations)365 1054 y(can)21 b(b)q(e)g(p)q(erformed)g(concurren)o(tly)m(.) 38 b(It)21 b(pro)o(vides)g(facilities)e(for)i(MSPMD)f(lik)o(e)365 1104 y(parallel)13 b(libraries.)262 1241 y Fh(1.4)64 b(Conclusion)262 1332 y Fg(This)17 b(c)o(hapter)h(has)g(presen)o(ted)i(and)d(discussed)i(a)e (prop)q(osal)g(for)h(comm)o(uni)o(cation)d(con-)262 1382 y(texts)k(within)g (MPI.)f(In)h(the)h(prop)q(osal)f(pro)q(cess)h(groupings)f(app)q(eared)h(as)f (frames)f(for)262 1432 y(the)d(creation)g(of)f(comm)o(unication)e(con)o (texts,)j(and)g(comm)o(unicatio)o(n)d(con)o(texts)k(retained)262 1482 y(certain)e(prop)q(erties)h(of)f(the)g(frames)f(used)i(in)e(their)i (creation.)324 1532 y(The)f(author)g(strongly)g(recommends)e(this)i(prop)q (osal)g(to)g(the)g(committee.)957 2574 y(14)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 14:58:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00429; Sun, 21 Mar 93 14:58:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25091; Sun, 21 Mar 93 14:58:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 14:58:00 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25083; Sun, 21 Mar 93 14:57:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA03816; Sun, 21 Mar 93 13:52:35 CST Date: Sun, 21 Mar 93 13:52:35 CST From: Tony Skjellum Message-Id: <9303211952.AA03816@Aurora.CS.MsState.Edu> To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu Subject: Re: Proposal II' - LaTeX Your subcommittee chair has been telling you to call this Proposal VI for the last half hour. Please listen. Drop the Editorial note, forthwith, after naming it proposal VI. Please don't be write-only. - Tony From owner-mpi-context@CS.UTK.EDU Sun Mar 21 15:07:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00520; Sun, 21 Mar 93 15:07:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25403; Sun, 21 Mar 93 15:07:05 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 15:07:04 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25395; Sun, 21 Mar 93 15:07:02 -0500 Date: Sun, 21 Mar 93 20:07:00 GMT Message-Id: <13161.9303212007@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: clarification, i hope :-) To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Dear Tony I am trying to send you a "clean table" as requested. Proposal Number Notes on Proposal --------------- ----------------- I++ Defunct number, defunct proposal. Was circulated as Proposal I. II' Defunct number, live proposal. To be circulated as Proposal VI. I Marc Snir's proposal which appeared in point-to-point working document. I think we would do ourselves harm to just ignore this. VI My proposal which I am personally happy with and am also happy to move along towards the other proposals prepared or being prepared by Rik and Tony. Was called Proposal II'. III/IV The proposal which Tony is working on. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 17:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01141; Sun, 21 Mar 93 17:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28876; Sun, 21 Mar 93 17:14:29 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 17:14:28 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28868; Sun, 21 Mar 93 17:14:22 -0500 Date: Sun, 21 Mar 93 22:14:17 GMT Message-Id: <13284.9303212214@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal VI (was Proposal II') - LaTeX To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Tony Thanks for the phone call. I have stopped calling this thing II' and it is now Proposal VI, as you ask, and of course you are right that we should avoid confusion. Here is the LaTeX for Proposal VI. I have propogated the comments that you gave me for Proposal I++ (now defunct name and defunct proposal) into this document as I believe appropriate. Sometimes this has required replicating your comment at several points in the document, and I have tried to give the same second order comments in each case. Perhaps it will be as well to read the whole thing before making further comments, rather than "real time" comments (which I often do, bad practice). PostScript will follow. Best Wishes Lyndon ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal VI\\MPI Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal VI" % Written by Lyndon J. Clarke % March 1993 % \newcommand{\LabelNote}[1]{\label{vi:note:#1}} \newcommand{\SeeReferNote}[1]{{\bf See~Note~\ref{vi:note:#1}.}} \newcommand{\LabelSection}[1]{\label{vi:sect:#1}} \newcommand{\ReferSection}[1]{{\bf Section~\ref{vi:sect:#1}.}} \chapter{Proposal VI} {\it Editorial Note: This is not Proposal II as identified at the post-meet lunch after the February meeting in Dallas. Rik~Littlefied came on board the proposal writing process, and decided to write a different proposal which appears as Proposal V. There is no longer a proposal which contains static contexts as discussed at the post-meet lunch.} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction} This chapter proposes that communication contexts and process groupings within MPI appear as related concepts. In particular process groupings appear as ``frames'' which are used in the creation of communication contexts. Communications contexts retain a reference to, and inherit properties of, their process grouping frames. This reflects the observation that an invocation of a module in a parallel program typically operates within one or more groups of processes, and as such any communication contexts associated with invocations of modules also bind certain semantics of process groupings. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of MPI, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language for only purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The notes assumes knowledge of the proposal text and are therefore best examined after familiarisation with that text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the text of the proposal. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal} \LabelSection{proposal} This section presents the detailed proposal, discussing in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes} \LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and MPI does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{process:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of MPI services. MPI provides a procedure which returns the own descriptor of the calling process. For example, {\tt pd = mpi\_own\_pd()}. \subsubsection*{Process Creation and Destruction} This proposal makes no statements regarding creation and destruction of processes. \SeeReferNote{dynamic:processes} \subsubsection*{Descriptor Transmission} The value of a process descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %[Lyndon] % Since it is an integer one cannot prevent the user from passing it. % The discussion notes should help answer why is is an integer. More % here. % We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. % Because rank will be an integer, then pd must also be an integer, % so I write that all descriptors are integers for consistency beteen % the different descriptors. MPI provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_PD\_TYPE}. MPI provides a process descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %[Lyndon] % I don't understand what you don't get. A registry service is just % an easier (nonscalable, okay) way for concurrent entities to hook up % with one another, rather than having some common ancestor send descriptors % around in messages. % In fact I don't really need a group or process registry service, as mentioned % in the discussion section (yes, I know, not well presented), but % I do need a context registry service, and I'm being consistent % between different kinds of descriptors again. % Suggestive syntax for a registry service is pretty boring, so % I skipped it. Note that receipt of a process descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_pd(pd)}. The user is not allowed to invalidate the process descriptor of the calling process. \SeeReferNote{process:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %[Lyndon] % Acquisition of a descriptor requires an allocator, and consumes % resource. In such cases it is good practice to provide a % deallocator which frees resource. % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %[Lyndon] % Unless I am missing the additional point in this comment, I can't % see the problem. Probably poor presentation of Proposal I++ :-) \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding processor descriptor attributes. \subsection{Process Groupings} \LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to?) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. \SeeReferNote{dynamic:groups} The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to the descriptor of the $N$ processes composing the grouping. The structure of a process grouping is defined by a process grouping topology and this proposal makes no further statements regarding such structures. %[Tony] % It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % %[Lyndon] % I am deliberately making weak statements about topology while % acknolwedging the the process topology subcommittee. \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{group:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the process grouping until the termination of MPI services. \SeeReferNote{group:blobs} MPI provides a procedure which returns the descriptor of the home group of the calling process. For example, {\tt gd = mpi\_home\_gd()}. \SeeReferNote{own:group} \subsubsection*{Group Creation and Deletion} MPI provides a procedure which allows users to dynamically create one or more groups which are subsets of existing groups. For example, {\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} partition an existing group {\tt gda} into one or more distinct subsets. This procedure is called by and synchronises all members of {\tt gda}. MPI provides a procedure which allows users to dynamically create one group by explicit definition of its membership as a list of process descriptors. For example, {\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. %[Tony] % loosely synchronous % %[Lyndon] % Do we mean the same thing? the constructors require each member % if the object under construction to participate in the construction, % and return a descriptor for the constructed operation. Therefore % no member can terminate construction before all other members have % commenced, at least. MPI provides a procedure which allows users to delete a created group. This procedure accepts the descriptor of a group which was created by the calling process and destroys the identified group. For example, {\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. MPI provides additional procedure which allow users to construct process groupings which have a process grouping topology. \subsubsection*{Descriptor Transmission} The value of a group descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %[Lyndon] % Since it is an integer one cannot prevent the user from passing it. % The discussion notes should help answer why is is an integer. More % here. % We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. % Because rank will be an integer, then pd must also be an integer, % so I write that all descriptors are integers for consistency beteen % the different descriptors. MPI provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_GD\_TYPE}. MPI provides a group descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %[Lyndon] % I don't understand what you don't get. A registry service is just % an easier (nonscalable, okay) way for concurrent entities to hook up % with one another, rather than having some common ancestor send descriptors % around in messages. % In fact I don't really need a group or process registry service, as mentioned % in the discussion section (yes, I know, not well presented), but % I do need a context registry service, and I'm being consistent % between different kinds of descriptors again. % Suggestive syntax for a registry service is pretty boring, so % I skipped it. Note that receipt of a group descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_gd(gd)}. The user is not allowed to invalidate the own group descriptor of the process or the group descriptor of any group created by the calling process. \SeeReferNote{group:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %[Lyndon] % Acquisition of a descriptor requires an allocator, and consumes % resource. In such cases it is good practice to provide a % deallocator which frees resource. % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %[Lyndon] % Unless I am missing the additional point in this comment, I can't % see the problem. Probably poor presentation of Proposal I++ :-) % \subsubsection*{Descriptor Attributes} MPI provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identifier group. For example, {\tt rank = mpi\_group\_rank(gd)}. MPI provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, {\tt size = mpi\_group\_size(gd)}. MPI provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, {\tt pd = mpi\_group\_pd(gd, rank)}. \SeeReferNote{pd:to:rank} MPI provides additional procedures which allow users to determine the process grouping topology attributes. %[Tony] It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % % %[Lyndon] % I am deliberately making weak statements about topology while % acknolwedging the the process topology subcommittee. \subsection{Communication Contexts} \LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping is used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{context:identifiers} \SeeReferNote{descriptor:cache} The creation of MPI process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains its own context until MPI process grouping deletion. MPI provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, {\tt cd = mpi\_own\_cd(gd)}. \SeeReferNote{own:context} \subsubsection*{Context Creation and Deletion} MPI provides a procedure which allows users to dynamically create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, {\tt cd = mpi\_context\_create(gd)}. This procedure is called by and synchronises all members of {\tt gd}. \SeeReferNote{dynamic:contexts} %[Tony] % loosely synchronous % %[Lyndon] % Do we mean the same thing? the constructors require each member % if the object under construction to participate in the construction, % and return a descriptor for the constructed operation. Therefore % no member can terminate construction before all other members have % commenced, at least MPI provides a procedure which allows users to destroy created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes that context identifier. For example, {\tt mpi\_context\_delete(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. \subsubsection*{Descriptor Transmission} The value of a context descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %[Lyndon] % Since it is an integer one cannot prevent the user from passing it. % The discussion notes should help answer why is is an integer. More % here. % We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. % Because rank will be an integer, then pd must also be an integer, % so I write that all descriptors are integers for consistency beteen % the different descriptors. MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_CD\_TYPE}. %[Tony] % I don't get all of this. Why? % %[Lyndon] % I don't understand what you don't get. A registry service is just % an easier (nonscalable, okay) way for concurrent entities to hook up % with one another, rather than having some common ancestor send descriptors % around in messages. % In fact I don't really need a group or process registry service, as mentioned % in the discussion section (yes, I know, not well presented), but % I do need a context registry service, and I'm being consistent % between different kinds of descriptors again. % Suggestive syntax for a registry service is pretty boring, so % I skipped it. MPI provides a context descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a context descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_cd(cd)}. The user is not allowed to invalidate the own context descriptor of a group or the context descriptor of any context created by the calling process. \SeeReferNote{context:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %[Lyndon] % Acquisition of a descriptor requires an allocator, and consumes % resource. In such cases it is good practice to provide a % deallocator which frees resource. % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %[Lyndon] % Unless I am missing the additional point in this comment, I can't % see the problem. Probably poor presentation of Proposal I++ :-) % \subsubsection*{Descriptor Attributes} MPI provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, {\tt gd = mpi\_context\_gd(cd)}. \subsection{Point-to-Point Communication} \LabelSection{point2point} This proposal recommends three forms for MPI point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a Send operation of form X cannot match with a receive operation of form Y, which requires that the form is embedded into the message envelope. The three forms are described followed by considerations of uniform integration of these forms in the point-to-point communication section of MPI. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. \SeeReferNote{null:context} Message selection and addressing are expressed by {\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. Send supplies the {\tt pd} of the receiver. Receive supplies the {\tt pd} of the sender. Receive can wildcard on {\tt pd} by supplying the value of a named constant process descirptor, e.g. {\tt MPI\_PD\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. \SeeReferNote{closed:context} Message selection and addressing are expressed by {\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. Send supplies the {\tt cd} of the receiver and sender, and the {\tt rank} of the receiver. Receive supplies the {\tt cd} of the sender and receiver, and the rank of the sender. The {\tt (cd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt cd}. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. \SeeReferNote{open:context} Message selection and addressing are expressed by {\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. Send supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. Receive supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The {\tt (rcd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt lcd}. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD}, in which case Receive {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point section of MPI by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with the form. This is at the expense of multiplying the number of Send and Receive procedures by a factor of three, and some difficulty in details of the current point-to-point working document which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options each have advantages and disadvantages. \paragraph*{Option i:} The framework of the open context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii:} The framework of the closed context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as {\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the {\tt (lcd, rcd)} pair at the expense of introducing some trickery. We can define a mechanism which ``globs'' together these two fields onto in an identifier which Send and Receive can distinguish from a context identifier and treat as the shorthand notation. Then we should also define a mechanism by which a receiver which has completed a Receive with wildcard on {\tt rcd} is able to determine the valid context descriptor of the sender. This is a inconvenient. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % % [Lyndon] % I dislike this also. Of course it is all to get around extending % the syntax, it stinks of that. \paragraph*{Option iii:} The framework of the null context form is adopted and extended. The null context form is expressed as {\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is clumsy. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % % % [Lyndon] % I dislike this also. Of course it is all to get around extending % the syntax, it stinks of that. \subsection{Collective Communication} \LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context and frame in which they are to operate. MPI does plan to describe symmetric collective communication operations. It is not possible to determine whether the proposal is sufficient to allow implementation of the collective communication section of MPI in terms of the point-to-point section of MPI without loss of generality, since the collective operations are not yet defined. %[Tony] Check, but it is desirable that they be so writable, so we will % have to watch. % Assymetric collective communication operations, especially those in which the sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a ``glob'' form) which identify the contexts and frames in which they are to operate. MPI does not plan to describe assymetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model and comprise communicative funtionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of MPI. % % END "Proposal" %----------------------------------------------------------------------% % [Tony] % So, I gather that a set of groups is passable to a collcomm, % and a pair is passable to a pt2pt. That is neat, but it should % still be a separate data structure, with separate calls than % the intra-group version (at least for the pt2pt calls). % % [Lyndon] % Currently, I am only thinking of passing either singlets or duplets % to point-to-point and collective. % I was trying to avoid separate - extra - calls to point-to-point % because that is already very large and there is a swell of concern % about the size thereof. %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion} \LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents arguments for and against the MIMD model annex, and to some extent explores the consquences of these arguments for MPI in a wider sense. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model, in which process execute in a stricly SIMD fashion. The material describing processes in \ReferSection{processes} is simplified: \begin{itemize} \item processes have identical instruction blocks and different data blocks; \item process descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query and group descriptor attribute query is capable of providing access to all process descriptors); \item dynamic process models are not considered. \end{itemize} The material describing process groupings in \ReferSection{groupings} is simplified: \begin{itemize} \item group descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query capable of providing access to all group descriptors.); \item the own process grouping explicitly becomes a group containing all processes. \end{itemize} The material describing communication contexts in \ReferSection{contexts} is simplified: \begin{itemize} \item context descriptor transmission and registry become unnecessary. \end{itemize} The material describing point-to-point communication in \ReferSection{point2point} is simplified: \begin{itemize} \item the open context form becomes redundant; \item uniform integration ``Option i'' is deleted, and ``Option ii'' loses ``globs'' thus becoming simple enough that ``Option iii'' need not be further considered. \end{itemize} The material describing collective communication in \ReferSection{collective} is simplified: \begin{itemize} \item there is no possibility of collective communication operations spanning more than context. \end{itemize} \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as MPI are not suited to fine grained function driven or data driven programming. The annex is intended to provide expresive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in EXPRESS and PARMACS, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or vice versa!). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computating. The debate is whether or not message passing approaches such as MPI (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD parallel prgrams. The positive argument is that communications between SPMD modules requires high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development and standardisation procedures for distributed operating systems. The latter argument is comparable with the argument for MPI versus parallel compilation. \subsection{Notes} \LabelSection{notes} \begin{enumerate} \item\LabelNote{integer:descriptors} Descriptors are a plentiful but bounded resource. They are expressed as integers for two reasons. Firstly, this expression facilitates uniform binding to ANSI~C and Fortran~77. Secondly, there is a later convenience in ensuring that process descriptors in particular are expressed in integers, and I made all the descriptors are integers for consistency. In practice the descriptor could be an index into a table of objects description structures or pointers to such structures. \item\LabelNote{descriptor:cache} It is possible to allow the user to save user defined attributes in descriptors, as Littlefield has suggested. Such attributes should not be communicated in either descriptor transmission or the descriptor registry. \item\LabelNote{process:identifiers} The process descriptor is not a global unique process identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional process information in implementations. For example, the process identifier of a receiver may not be sufficient to route a message to the receiver, and this information can be referenced from the process descriptor. \item\LabelNote{group:identifiers} The group descriptor is not a global unique group identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependeent, and enables more efficient access to additional group information in implementations. For example, the size of the group and the map from member ranks to process descriptors can be referenced from the group descriptor. \item\LabelNote{context:identifiers} The context descriptor is not a global unique context identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional context information in implementations. For example, the group descriptor of the frame can be referenced from the context descriptor. \item\LabelNote{dynamic:processes} The proposal does not prevent a process model which allows dynamic creation and termination of processes however it does not favour an asynchronous process creation model in which singleton processes are created and terminated in an unstructured fashion. The proposal does favour a model in which blobs of processes are created (and terminated) in a concerted fashion, and in which each group so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying worloads. The author suggests that concerted blob expand and contract operations are most suitable for this purpose than asynchronous spawn/kill operations. \item\LabelNote{registry:check} There should probably also be a registry operation which checks whether a name has been registered without the possibility of blocking the calling process indefinitely. The same registry could be used for process, group and context descriptors. \item\LabelNote{process:coherency} In the static process model it is assumed that processes are created in concert, and that termination of one process implies termination of all processes. In this case there is no coherency problem associated with processes and process descriptors. In a dynamic process model there is a coherency problem which processes and process descriptors since a process can retain the descriptor of a process which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{dynamic:groups} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The author suggests concerted group expand and contract operations are more generally useful than asynchronous join/leave operations. \item\LabelNote{group:blobs} MPI has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is intended to be a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and different blobs have different own process groupings. An SPMD program is a single blob. A host-node program composes two blobs, the node blob and the host blob (which is a singleton). There is a sense in which a blob has a locally SPMD view. \item\LabelNote{own:group} This procedure looks like a process descriptor attribute query. \item\LabelNote{group:coherency} Dynamic processes introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Dynamic processes potentially introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Group transmission introduces a group and group descriptor coherenncy problem since a process can retain a group descriptor of a group which has been identified group. The proposal expects the user to ensure coherent usage. \item\LabelNote{pd:to:rank} The proposal did not include a function to convert a {\tt (gd, pd)} pair into a rank. It is suggested that this inverse map is allowed to be arbitrarily slow. \item\LabelNote{dynamic:contexts} Marc Snir has described a method by which global unique group identifiers can be generated without use of shared global data. The description of context creation anticipates a similar method for global unique context identifier generation. However the synchronisation requirement of this method makes it unnecessary for context creation. Synchronisation at creation of context within a process grouping frame implies that all processes within the frame perform the same context creations in the same sequence. Therefore the combination of global unique frame identifier and context creation sequence number provides a global unique context identifier without communication. \item\LabelNote{own:context} This procedure looks like a group descriptor attribute query. \item\LabelNote{context:coherency} The retention of a reference to a group descriptor by a context introduces the potential for a context and context descriptor coherency problem, as a context could contain a reference to the group descriptor of a group which has been deleted. This could be circumvented by demanding that all such contexts are deleted before a group is deleted. Context descriptor transmission introduces a coherency problem since a process can retain the descriptor of a context which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{null:context} This is somewhat like and PARMACS and PVM~3. It is general purpose, but not particularly expressive. It does not provide facilities for writers of parallel libraries. \item\LabelNote{open:context} This is somewhat like ZIPCODE and VENUS. It is expressive in certain SPMD programs where noncommunicative data driven parallel computations can be performed concurrently. It provides facilities for writers of SPMD like parallel libraries. \item\LabelNote{closed:context} This is somewhat like CHIMP and PVM~2. It is expressive in certain MIMD programs where communicative data driven parallel computations can be performed concurrently. It provides facilities for MSPMD like parallel libraries. \end{enumerate} % % END "Discussion & Notes" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Conclusion" % \section{Conclusion} This chapter has presented and discussed a proposal for communication contexts within MPI. In the proposal process groupings appeared as frames for the creation of communication contexts, and communication contexts retained certain properties of the frames used in their creation. The author strongly recommends this proposal to the committee. % % END "Conclusion" %----------------------------------------------------------------------% % % END "Proposal VI" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Sun Mar 21 18:51:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01899; Sun, 21 Mar 93 18:51:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01788; Sun, 21 Mar 93 18:50:17 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 21 Mar 1993 18:50:14 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01768; Sun, 21 Mar 93 18:50:09 -0500 Date: Sun, 21 Mar 93 23:50:04 GMT Message-Id: <13403.9303212350@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: V commenting To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Tony I ran over schedule before finishing. Comments are marked in the Lyndon favoured style % %[name] % text % Comments to comments are nested %% %[name] % text %%[name2] %%text2 %% % There are comments to comments to comments down there as well %%% Good luck! Lyndon ---------------------------------------------------------------------- % [Lyndon] General points % -------------- % % 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am % offering if you wish) should attempt to construct a more transparent % presentation before circulation to whole committee, for the % convenience of committee members. %%[Tony] %% I felt that things appear twice, because of summary (good) %% and because of implementation notes at end (confusing) %% % 2) I'm not a fan of much of this proposal, although I do indeed like % some of the ideas which it introduces. [On the other hand, I'm not a % great fan of all of the proposal which I wrote. I shall mail self % criticism of my proposal, and may have to write amended or alternative % proposal :-)] %%[Tony] %% Please be more specific. I am having a hard time understanding %% why you really don't like it, Lyndon. If the process model %% were a little less static, and servers were permitted (though %% hopefully bounded in cost), I think we would have an excellent %% proposal. %% %%%[Lyndon] %%% I would have thought that the points below make my major %%% objections perfectly clear. Perhaps not. Here they are: %%% a) Paired-exact-match stuff %%% b) Translation of (group,rank) into TID all over the place % % 3) I really like the way in which groups are something like ``frames'' % in which contexts are created. This is conceptually much neater than % duplication of groups. %%[Tony] %% In practice, group subsetting will require groups to be copied, %% otherwise, subgroups will unfairly be penalized by the size %% of their ancestor. %% %%%[Lyndon] %%% I am anticipating that when one or more groups are created by %%% subsetting that, for example if the parent were described by %%% a proces list, then the children will be described by process %%% lists which are distinct sublists of the parent. So each element %%% of the parent list gets copied, exactly once. %%% The difficulty I have is that if a group were to be expanded %%% or contracted, then the ``duplicates'' thereof would no longer %%% be duplicates. Saying that duplicate creates a group bu retains %%% the process list of the old group is conceptually muddy since %%% the new group is a reference to a group, whereas the old group %%% or an even older group must be an actual group. Yuk! Now, if %%% we introduce the concept of a reference to an actual group, %%% and say this reference is unqieuly identified, then is is %%% conceptually sound and this object we describe really is a context. % % 4) I like the idea of pushing information into the group structure. I % have a few qualms with the proposed details --- see specific points. %%[Tony] %% I have more confidence about this idea, and could demonstrate %% by June/July time-frame in Zipcode. %% % % 5) See ``Writing a server in the point-to-point layer of MPI in four % easy steps'' at the foot of the message. %%[Tony] %% This seems like a nice thing. %% %%%[Lyndon] %%% You are too kind :-) \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %%[Tony] %% Alternatives, Lyndon? %% %%%[Lyndon] %%% I don't suppose for one minute that you will like this, but I really %%% would suggest that in this case a group descriptor registry may %%% be appropriate. %[Tony] % Seems doable. % %%[Lyndon] %% But usable with some grief, as above. \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %% [Tony] %% Me too. %[Tony] % Seems doable. % \item There are restrictions that permit groups to be layered on top of pt-pt. %[Tony] % I don't understand the word 'restriction' here. % Restriction of what. % %%[Lyndon] %% Rik is speaking of the pair-exact-match stuff you see later on. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. %[Tony] % What does "fast" mean. % %%[Lyndon] %% Fair question! \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. %[Tony] % OK, the above two items are identical to what Zipcode % provides in practice, but people have argued that groups % might be created/deleted more often in some apps, and % that these apps ought to be supportable % %%[Lyndon] %% In our work group creation/deletion is an infrequent operation %% and we are happy to live with reasonable cost for this operation. %% I think Marc Snir is thinking abour a different group model in %% group created/deletion is frequent. %% Perhaps we should provide both or neither (even handedness principle). \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %% [Tony] %% 1) It is not obvious that TIDs should be restricted to 32 bits. %%%[Lyndon] %%% I did not imply that they were. %% 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %% I favor a whole word for a context, despite other limits, %% just to make things simpler. %% %%%[Lyndon] %%% ditto %% Internet addresses are going to get augmented from 32 to ??? bits %% is it reasonable to assume that certain MPI implementations might %% incorporate such internet addresses as TIDs (in future), %%%[Lyndon] %%% No, it is not reasonable, in the least. And both you and Rik appear %%% to be ignoring the possibility that the process descriptor could %%% be required to store routing data of significant length. Therefore %%% the sensible thing to do is use a process descriptor which in %%% practice might be a table index --- fits into integer for sure --- %%% the value of which is process local for sure, and the table %%% contains the real process identifier of implementation defined size. %% Opacity is partially violated if we say how big the data type is??? %%%[Lyndon] %%% I understand the point you make, but it gets blown away by the %%% point I have just made in reply to you. %[Tony] % Yes, and I want at least 32-bits of message tag. % %%[Lyndon] %% Yes, and I want exactly zero bits of message tag. %% I'll just keep quiet about message tags. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %% %%[Tony] %% where does this get us??? %% Remember, we have to choose on some things, so we can have something %% to present in Dallas. Is there an important difference here? %%%[Lyndon] %%% For sure their is an important difference. See comment above. %% %% TIDs are global entities. Is structure assumed to be global; %% in a truly opaque system, some TID component would have to be %% fixed, but the rest could vary structurally... %% %[Tony] % This could be difficult, in practice, if one mails a % message to one's own process, and MPI is smart enough % to optimize. % \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %% %% Yes, contexts are global; group identifiers are just pointers %% typically, to data structures, describing %% %% 1) a groups context %% 2) group members and their ranks (mappings, inverses, %% cached, hashed, unscalably stored, etc) %% 3) TID-to-rank map and inverse (see possibilities in 2) %% 4) A set of fixed global operations, accepted as standard, %% an accessible in O(1) time. Possibly, each %% such operation should be a method, so that %% a parameter block can be passed with it. Zipcode %% supports the Method type to do this. %% This is effectively a cache for some parts of item #5 %% ... %% 5) An AVL or similar tree of extensible operations. %% New operations are registerable by the user. These %% tags are unique within a group, a specify an operation %% i) pre-defined by MPI (in which case it can be cached %% in 4 %% ii) alternative operations (even if they do something %% standard, that are wanted to be accessed by %% name) This name is group unique. %% %% A mechanism for DO_METHOD_FROM_GROUP(name,....) %% or GET_METHOD_FROM_GROUP(name,...) %% and SET_METHOD_IN_GROUP(name,...) are clearly needed. %% %[Tony] % Sounds good. % \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %%[Tony] %% Yes, that is included, I believe, in concept. %% \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %%[Tony] %% Stripping is extremely controversial aspect, and arbitrary. %% If the recipient has the methods with the same name, then %% a new rendezvous could be accomplished at the far end %[Tony] % In Zipcode 1.0, we allow multiple global operations % to be provided on a message-class (eg, grid-oriented messages) % The identifiers for these possible operations are user-specified % presently, but the "names" of the global operations are fixed % at compile-time. % % That means that there is O(1) time to find combine, fanout, send, % etc, on a group-wide scope. However, other operations cannot % be accessed in O(1) time (they are not in the opaque structure). % % The same mechanism used by Zipcode to allow multiple methods for % combine to be registered by the user, could also allow extensibility % just like Rik describes, with little effort. We use AVL trees. % % In fact, I will add this to Zipcode 1.x. Why say this? It is % not far from existing practice, and I have a lot of the machinery % in place already, and I am confident that it is useful. % \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %%[Tony] %% Yes, these are all options. Fastness seems to be an important %% issue. If translation is very expensive, none of the "good" %% features will be used. %[Tony] % This seems to be a serious flaw. It will have to be cached % on an LRU basis, with system/user/both specifying how much % caching is allowed (ie, how much unscalable memory use). % If the first time is expensive, OK, but not the Nth time. % %%[Lyndon] %% Check. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). %[Tony] % OK, I see no problem with this (so far). % \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %%[Tony] %% Again, I think this should be eliminated, and all references %% to this idea should be expunged. It denies the context's %% ability to manage messages. %%%[Lyndon] %%% Check. %[Tony] % NO. This violates the concept of context entirely. % (ie, an oxymoron ... contexts same, but still no need for % tag disambiguation...) % % Use the default group context to establish (cooperatively) % other contexts, and then use these. This is a seriously % bad feature, in my mind. % \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. %%[Tony] %% I agree that context allocation has to be done in sequence. %% That is why I am in favor of providing calls that allow %% groups to get numerous contexts at creation, and then %% cooperatively, but potentially without further communication %% divide them(as they build subgroups, for instance). %% %% I see these as services to be used in building virtual topology %% features, which will then be more widely used by users of MPI. %% %%%[Lyndon] %%% If the context allocations are done in sequence, then I have %%% indicated how they can be done for free. I am getting confused. %[Tony] % I do not think we should support the paired-exact-match thing. % \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) %[Tony] % Concur. This suggests many more than "256" % %%[Lyndon] %% The number of contexts in the whole program? Sure 256 is too small! %% The number of contexts in each process? Maybe something like 256 is okay? \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] %[Tony] % This is the Zipcode model. It could say loosely synchronous. % \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %%[Tony] %% Do we get anything by having the root process? %% %%%[Lyndon] %%% No. %[Tony] % I don't understand the thread issue here. % %%[Lyndon] %% If two threads are concurrently partitioning the same group, you %% need to disambiguate the partition operations. Analagous to %% concurrent collective operations or nonblocking. \item Collective communication routines are called by all members of a group in the same order. %[Tony] % Yes. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. %[Tony] % What does caching really imply here ??? Help. % %%[Lyndon] %% Dunno. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %%[Tony] %% I think that non-blocking collcomm is moribund in MPI1 or %% else MPI1 is moribund. :-) %% %%%[Lyndon] %%% Check. %[Tony] % I think that contexts are really important in this case, % to keep things straight, but that non-blocking collcomm should % be omitted from MPI1 (cf, Geist). Sequencing supports % a sufficient disambiguation, as long as the entire group % is always the participant in operations. That is, you have % to form subgroups, with new contexts, to do global ops on % subsets. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %%[Tony] %% Too many negatives here for me to parse :-) %[Tony] % Why aren't servers needed to create contexts. Where do they % come from? If you rely on the fact that INITIAL will do % a loosely synchonous cooperative operation each time a new % context is needed, then a simple (easily implementable server, % or fetch-and-add remote access) is replaced by a more rigid % computation model. % % If we can get rid of this disagreement, me might be able to % reduce our total proposal space by one whole proposal. % \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %%[Tony] %% Context allocation need not be free provided it can be made cheap, %% or cheap enough. %% %% If one knows one will need several, then a single call could %% provide such contexts, amortizing overhead. This is likely when %% bulding grids (ie, virtual topologies) in Zipcode, so it is %% true in existing practice. %% %% One should recognize the need for layering virtual top. calls %% on top of these calls, then these calls may appear painful, %% but perhaps they would be less used. Some users will use the %% provided virtual topology calls, others will prefer their own. %% Both will have equal power (see also,separate note on layerability). %% %% If getting N contexts is a send-and-receive, plus a reactive server, %% then this is reasonably light weight,provided that hundreds of %% messages, or global operations ensue thereafter. We can know in %% advance how heavy weight the context server will be. %% %% if an implemention can use some locations of remote memory, with %% fetch and add, or locks, to achieve contexts, then this is even %% cheaper, in principle. %% %% Despite Jim's earlier insistence that context numbers be kept to %% 256 or so, I think that this number should be much larger, so that %% much less efort goes into returning contexts, and so on, except %% occasionally, by processes. Otherwise, a new kind of overhead, %% get-rid-of-context-because-I-am-out ensues, or programs block %% until contexts become available, offering the possibility of %% deadlocks. %% %[Tony] % If contexts are being used very dynamically, how are they being % assigned, kept, released, reissued without a server? Sorry if % I missed something, but I don't see it, without a restrictive % SPMD model of computation (Zipcode obviates its server for the % SPMD model, for instance). %%[Lyndon] %% MPI stinks of SPMD. I wouldn't mind if MPI would just say SPMD. \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %%[Tony] %% Unscalability (of a limited form) should be permitted/selectable %% by user, to use as much per-node memory as the user wants, to reduce %% communication. %% %[Tony] % Can you clarify this with examples. % \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. % %%[Tony] %% I see,but what he means here is that there is no unpredictable %% translation cost because we do not write (group,rank) in pt2pt %% calls. So, there is some validity to the statement. %%%[Lyndon] %%% However he ignores that the TID might require translation the %%% cost of which might be unpredictable. This because the TID has %%% a global value and cannot therefore hold process local information %%% such as ``how do I route to that process''. %[Tony] % I like this, of course. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %%[Tony] %% Yes, that is why I have one in Zipcode. I wish Zipcode were %% on netlib today, so you could try it. Well, we are writingthe %% manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %%[Tony] %% I lost something here, but I agree with the sense. The word %% seems is subjective,and should disappear from our discussions, %% as much as seems prudent, anyway :-) %[Tony] % Well, is point-to-point group oriented. Not. %%[Lyndon] %% Check. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %%[Tony] %% Yes,awkward, because it is nice to escape the TID realm and %% work within the (albeit simple) abstraction of group,rank. %% When layering virtual topologies on this, it would be so nice %% to write them to a group,rank syntax, not enforcing TID mappings %% everywhere. %[Tony] % I think it is awkward. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %%[Tony] %% Indeed. %%%[Lyndon] %%% More so than you think, I think! %[Tony] % OK, but one can form a new group, as I have argued before. % Use the "awkward" pt2pt to get the right info shared between % group leaders, make the new group, use unawkward collective % operations on new group (with new context). %%[Lyndon] %% This is only one model of group-group interaction, which in my %% experience and understanding really is still steeped in SPMD. %% Please consider the examples of non SPMD group usage which I %% mailed out. You can say to me - oh, you shouldn't do this kind %% of thing with MPI, Lyndon - if you like, if you believe that. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. %[Tony] % This again is in practice, in Zipcode. % \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %%[Tony] %% Yes, and I think that loosely synchronous operations can maintain %% coherency, in practice. That is, no operations that modify the %% group descriptors (other than cached lookup info) are permitted, %% without loose synchronization. %% This is nasty in that is would prohibit sending descriptors to %% processes not part of the group, so it is a clear trade-off. %% Perhaps such send-to-non-group-member operations could stipulate %% that this group information is somehow ephemeral, and that they %% need to join a new group to keep useful information over time??? %% %%%[Lyndon] %%% I am over schedule. I have to stop here. I will come back to this %%% tomorrow, if applicable (ie, you may have overtaken me). %[Tony] % Sounds dangerous. What must application do to maintain % coherency, since group descriptors are opaque. % \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. %[Tony] % No, loosely synchronous process model, unless you mean % cooperation of INITIAL at all such join/leave steps. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %%[Tony] %% Yes, that is exactly what Zipcode was written to avoid. The %% user wants help managing things like this!!!! %% %% The search, if any, must be MPI-supported, and as efficient as %% possible (eg, AVL trees, hash, partial hash with exceptions). %% % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %%[Tony] %% This is why I remain a strong advocate of (group,rank) %% addresssing in pt2pt. %% %[Tony] % No, you can't know the group or rank in group of sender. % If there were one context per group (isn't that so here?), % then all you need is the rank. With TID_TO_RANK_IN_GROUP % operation, this could be provided, but no wildcarding % or receipt selectivity could be done at this level. % \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %%[Tony] %% I addressed this in my Step-1 remarks. Please see that. BELOW %[Tony] % No, it just does not provide guarantees that certain kinds % of applications will run OK. (ie, those that do group % creation/deletion relatively often). Zipcode has assumed % that such operations would be relatively seldom. Thus, I do % not quibble that this is a reasonable choice,but a fairer % way to say this is that it may be difficult to support such % applications. That reveals an issue to be studied more. % \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %%[Tony] %% Thank you. These points are both well taken (ie these two paragraphs) %[Tony] % Perhaps they should do. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] %[Tony] % Dump this. \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %%[Tony] %% Also,see what I wrote in my Step-1 comments. BELOW. %% I believe O(log(P)) is still possible. %% %[Tony] % No, a non-deterministic broadcast can be used, with a token. % This requires a token server. Again, implementable with fetch+ % add on most systems, or a light reactive server. % % Once the non-deterministic broadcast has finished, a fanin/collapse % is done to the original root, which then frees the token. % \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %%[Tony] %% Please see my Step-1 comments. %[Tony] % Why not just given in and allow the server. % I don't like the paired-exact-match constraint AT ALL. % \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) %[Tony] % Seems OK, but why need the paired-exact-match thing again. % Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) %[Tony] % Again, OK, but I want to see this work without the paired-exact- % match, if possible. Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. %[Tony] % So, now, you have concurred with my (previously flamed) idea % that group construction/destruction should be realizable using % pt2pt, just like global operations should do. I like this % because 1) it is explicable to the implementor, 2) it allows % simple intitial implemtations, 3) it sets some ideas for how % much these things will cost [upper bound]. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} %[Lyndon] % Writing a server in the point-to-point layer of MPI in four easy steps % ---------------------------------------------------------------------- % % 1) Partition the INITIAL group into two groups. A singleton group, % SERVER, and a group CLIENT which contains all of the other processes. % % 2) The single process in SERVER group records its TID. % % 3) The processes in INITIAL group allocate a context SERVICE which % they remember either in the group cache or static data or something. % % 4) Use a broadcast in INITIAL group with ``sender'' as the one process % which is also in SERVER group, and the ``receivers'' as the (many) % processes which are also in CLIENT group, in the SERVICE context, in % order to disseminate the TID of the server process. % % [Fanfare] a server process is in place as is a dedicated context for % the purposes of messages required to implement the service. % % [Observation] the mpi point-to-point initialisation can do this % automatically. %%[Tony] %% Zipcode's postmaster general works in this way, more or less. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 01:52:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07571; Mon, 22 Mar 93 01:52:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15011; Mon, 22 Mar 93 01:51:24 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 01:51:23 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15000; Mon, 22 Mar 93 01:51:21 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Sun, 21 Mar 93 22:47 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06109; Sun, 21 Mar 93 22:45:30 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15778; Sun, 21 Mar 93 22:45:28 PST Date: Sun, 21 Mar 93 22:45:28 PST From: rj_littlefield@pnlg.pnl.gov Subject: Re: mini-proposal on layerability To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9303220645.AA15778@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony writes: > So, each receipt function uses the algorithm (received_rag & ~dont_care)& > care Shouldn't this be (received_tag & ~dont_care) == care i.e. ignore some bits but check the others for equality? If so then yes, I will strongly support this feature. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 03:30:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11999; Mon, 22 Mar 93 03:30:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20829; Mon, 22 Mar 93 03:29:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 03:29:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20810; Mon, 22 Mar 93 03:29:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 00:13 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06118; Mon, 22 Mar 93 00:11:53 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15810; Mon, 22 Mar 93 00:11:50 PST Date: Mon, 22 Mar 93 00:11:50 PST From: rj_littlefield@pnlg.pnl.gov Subject: new proposal To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220811.AA15810@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony and Lyndon, I will respond in a separate message to some of your detailed comments on Proposal V. But maybe we can move faster by popping up a level. It seems to me that you like the idea of cacheing arbitrary info in group descriptors, you like the idea of groups as things within which contexts get formed, you like performance guarantees, and you don't like having to use opaque id's in point-to-point communications. I'm not quite sure whether static groups and synchronous group control are OK, but I'll presume here that they are. Well, this is pretty neat. If this keeps up maybe we can converge on just one or two proposals. Most of my gripes with other proposals have been based on performance and/or a need for asynchronous servers (with attendant performance and non-portability gripes). But I notice that explicit performance expectations have been gradually appearing and the need for servers disappearing from other people's proposals. Perhaps it is time for a synthesis. Here's a sketch of a new proposal (VI ?). . The functionality of point-to-point communications is per Snir, Lusk, and Gropp, augmented by my proposed MPI_FORM_CONTEXT to allow assembling arbitrary collections of processes. (Marc has already accepted this in a private email to me -- don't know why he didn't post it.) . Performance expectations of point-to-point communications are explicitly stated as follows: - MPI_COPY_CONTEXT does not synchronize the participating processes and costs significantly less than a point-to-point fanout among them (e.g., it uses a communication-free counting strategy); - all other context formation routines cost no more than if they were implemented using a single fanin/fanout among the participating processes; - translation of (context,rank) to absolute processor ID costs no more than if it were implemented via the lookup table that Snir suggests. . Groups and contexts are not equal. A group consists of a base context (from which other contexts can be created quickly by MPI_COPY_CONTEXT), plus topology information, plus my cacheing facility. Conceptually, I like this better than proposal V. Do we already have a proposal like this? Should we have one? In general, do you guys buy off on the concept of including performance expectations in the specification? A couple of discussion points... 1. The separation of group and context is defensive. I think I understand what it means to copy a context. I am not sure of either the functional or performance implications of copying a group. E.g., does cached info get copied? 2. I will respond here to two criticisms that have been raised against the cacheing facility. Lyndon notes > % I like the general idea, but I'm nervous about two things: > % (a) implied associativity of group descriptor cache - this > % will potentially be time expensive in implementation. > % (b) there is no method proposed for abritration of keys > % between independently written modules, so we are > % in the same problem regime as just having message tag > % and no message context. > % However, key's are local, so presumably you would find > % it acceptable to add a key registration service? I implicitly proposed a key assignment service in my long-ago example. It said in part: static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_key_type gop_key; /* key for this module's stuff */ ... if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetAttributeKey()) ) { MPI_abort ("Insufficient keys available"); } } This is not really "registration" because nothing goes into the key assignment routine except the number of times it's called. But assuming that each call site is protected by a separate variable, the effect is to register the call site. As Lyndon notes, this is highly local. But it also allows the returned keys to have their values restricted so as to permit rapid testing and/or retrieval. Tony notes > %**** Stripping is extremely controversial aspect, and arbitrary. > %**** If the recipient has the methods with the same name, then > %**** a new rendezvous could be accomplished at the far end Yes, stripping is arbitrary. My motivation is that this greatly simplifies the design and satisfies what I view as the most critical need: to make collective comms run fast without complicating the calling sequence. I have no objection in principle to extending the facility to include classes of information that do not get stripped. But I sure didn't want to try creating and selling a spec that would handle, e.g., heterogeneous systems. Enough for here -- what do you think of "proposal VI" ? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:42:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25028; Mon, 22 Mar 93 05:42:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00467; Mon, 22 Mar 93 05:42:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 05:41:55 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00422; Mon, 22 Mar 93 05:41:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 01:48 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06136; Mon, 22 Mar 93 01:47:08 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15840; Mon, 22 Mar 93 01:47:05 PST Date: Mon, 22 Mar 93 01:47:05 PST From: rj_littlefield@pnlg.pnl.gov Subject: Rik's comments on Lyndon's Proposal I To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220947.AA15840@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu >>>>> I have added here a few additions to Tony's comments. >>>>> I have also deleted the bulk of the proposal regarding which >>>>> I have no comments at 1:45 AM. >>>>> -- Rik %**** %**** Below are my comments on Lyndon's Proposal I. In the next paragraph %**** I note that we are actually converging to less than N proposals, though %**** I have not seen Lyndon's new II proposal yet. Does it exist now? %**** %**** To achieve a reasonable presentation at Dallas, we can have multiple %**** proposals still on table (I think this a fair, well-thought-out %**** approach, but if we can condense some, let's do it). %**** %**** %**** After reading V, making my comments, and making my comments %**** in addition to Lyndon's comments, I am convinced we can %**** advance proposal V into something that is acceptable in the %**** III/IV mold, without a further III/IV proposal. Rik's %**** dropping of the static context concept has simplified our %**** group's efforts considerably, and I cannot disambiguate my %**** III/IV proposal from Proposal V, given the Lyndon and Tony %**** provisos and suggested improvements. This is not a wimp out %**** on my part. I do not see benefit of advancing something that %**** will look 90% like Proposal V at this point, and 97% like it %**** if the Lyndon/Tony comments obtain in it (which they would if %**** I wrote it now). %**** I would prefer that we hone Proposal V. If Rik wants to keep %**** his style, then I propose that Proposal III/IV become exactly %**** what I just said, a reworking of V + comments. %**** However, I think this will cause an unnecessary delay and %**** digression just to achieve details. Instead, we might pull %**** such choices into Proposal V at this time (server vs. no server). %**** to make what is common in our "approaches" more obvious. %**** - Tony %**** %**** (PS, Lyndon: Rik/Tony/Lyndon are authors of whole paper, because %**** we are all working on this, and because one of us (ie, me) will %**** make the final document cohesive, create a unified style, format, %**** set of meanings. This is the nature of collaboration. I do %**** not propose to include our other co-conspirators on such a document's %**** list of authors, as there has been minimal input from them. I %**** Think that Rusty/Bill and Rik/Tony/Lyndon are operating equivalently.) >>>>> >>>>> I happens to think that the style of my draft proposal V stinks. >>>>> I had much too much trouble becoming (fairly) sure that the content >>>>> was right, to also make it readable. >>>>> --Rik .... A group may be created by identical duplication of an existing group. For example, {\tt gidb = mpi\_grp\_duplication(gida)} where {\tt gidb} is the group identifier of the newly created group and {\tt gida} is the identifier of an existing group. The created group inherits all properties of the source group, including any topological properties. This operation has the same synchronisation properties as creation of group by definition. %**** It is not obvious to me that we want to enforce topology at this %**** juncture. However, we could register topology information in %**** the extensible structure strategy of Proposal V. %**** >>>>> Why such strong synchronization?! >>>>> If this has the same synchronization properties as creation, >>>>> then it won't return until all members have made the call. >>>>> But that means you actually have to sync everybody, which >>>>> implies 2 log(P) messages. Isn't it enough to just require >>>>> a loosely synchronous call, and use a counting strategy> ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:42:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25031; Mon, 22 Mar 93 05:42:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00499; Mon, 22 Mar 93 05:42:07 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 05:42:06 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00443; Mon, 22 Mar 93 05:41:56 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 01:30 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06131; Mon, 22 Mar 93 01:29:00 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15820; Mon, 22 Mar 93 01:28:55 PST Date: Mon, 22 Mar 93 01:28:55 PST From: rj_littlefield@pnlg.pnl.gov Subject: Rik on Tony on Lyndon on PropV To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220928.AA15820@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu This is the Proprosal V first draft, critiqued by Lyndon, then by Tony, and now with responses by Rik. Comments are flagged as % Lyndon's %**** Tony's >>>>> Rik's responses [Lyndon's leadin] General points -------------- 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am offering if you wish) should attempt to construct a more transparent presentation before circulation to whole committee, for the convenience of committee members. %**** %**** I felt that things appear twice, because of summary (good) %**** and because of implementation notes at end (confusing) %**** >>>>> I agree with both of the above. But let's decide whether >>>>> Proposal V will be replaced by VI or its equivalent before >>>>> we do any rewriting. 2) I'm not a fan of much of this proposal, although I do indeed like some of the ideas which it introduces. [On the other hand, I'm not a great fan of all of the proposal which I wrote. I shall mail self criticism of my proposal, and may have to write amended or alternative proposal :-)] %**** %**** Please be more specific. I am having a hard time understanding %**** why you really don't like it, Lyndon. If the process model %**** were a little less static, and servers were permitted (though %**** hopefully bounded in cost), I think we would have an excellent %**** proposal. %**** >>>>> See Proposal VI. I'd be happy to see the good ideas adopted >>>>> and the crap die quietly. 3) I really like the way in which groups are something like ``frames'' in which contexts are created. This is conceptually much neater than duplication of groups. %**** %**** In practice, group subsetting will require groups to be copied, %**** otherwise, subgroups will unfairly be penalized by the size %**** of their ancestor. %**** >>>>> Right, but I think of that as creating a new group. After all, >>>>> even the ranking structure is different. 4) I like the idea of pushing information into the group structure. I have a few qualms with the proposed details --- see specific points. %**** %**** I have more confidence about this idea, and could demonstrate %**** by June/July time-frame in Zipcode. %**** 5) See ``Writing a server in the point-to-point layer of MPI in four easy steps'' at the foot of the message. %**** %**** This seems like a nice thing. %**** >>>>> The implementation that Lyndon suggests consumes an entire >>>>> process for the server. There are times when this is OK, >>>>> but also times when it isn't. E.g., if you have a divide-and- >>>>> conquer algorithm that really wants 2^i non-server processes, >>>>> and you're working on a machine with 2^n processors that >>>>> doesn't support multiple processes per processor, then >>>>> some of the users will get upset that they can only use >>>>> half the machine. A year and a half ago, I got flamed for >>>>> making a suggestion like this in connection with the Delta. >>>>> (And now I suppose I get flamed for using the Delta as an >>>>> example again... Maybe if I use the CM5...) Specific points --------------- Dealt with as LaTeX comments to body of text, appearing in the form %[Lyndon] % text of point for your navigational convenience. These are quite detailed. ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %**** %**** Alternatives, Lyndon? %**** \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %**** Me too. >>>>> Progress!! \item There are restrictions that permit groups to be layered on top of pt-pt. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %**** %**** 1) It is not obvious that TIDs should be restricted to 32 bits. %**** 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %**** I favor a whole word for a context, despite other limits, %***** just to make things simpler. %**** %**** Internet addresses are going to get augmented from 32 to ??? bits %**** is it reasonable to assume that certain MPI implementations might %**** incorporate such internet addresses as TIDs (in future), %**** %**** Opacity is partially violated if we say how big the data type is??? \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %**** %**** where does this get us??? %**** Remember, we have to choose on some things, so we can have something %**** to present in Dallas. Is there an important difference here? %**** %**** TIDs are global entities. Is structure assumed to be global; %**** in a truly opaque system, some TID component would have to be %**** fixed, but the rest could vary structurally... %**** \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %**** %**** Yes, contexts are global; group identifiers are just pointers %**** typically, to data structures, describing %**** %**** 1) a groups context %**** 2) group members and their ranks (mappings, inverses, %**** cached, hashed, unscalably stored, etc) %**** 3) TID-to-rank map and inverse (see possibilities in 2) %**** 4) A set of fixed global operations, accepted as standard, %**** an accessible in O(1) time. Possibly, each %**** such operation should be a method, so that %**** a parameter block can be passed with it. Zipcode %**** supports the Method type to do this. %**** This is effectively a cache for some parts of item #5 %**** ... %**** 5) An AVL or similar tree of extensible operations. %**** New operations are registerable by the user. These %**** tags are unique within a group, a specify an operation %**** i) pre-defined by MPI (in which case it can be cached %**** in 4 %**** ii) alternative operations (even if they do something %**** standard, that are wanted to be accessed by %**** name) This name is group unique. %**** %**** A mechanism for DO_METHOD_FROM_GROUP(name,....) %**** or GET_METHOD_FROM_GROUP(name,...) %**** and SET_METHOD_IN_GROUP(name,...) are clearly needed. %**** >>>>> The model of contexts used in this proposal is that the >>>>> context value has global scope. It doesn't have to be that way. >>>>> The Snir, Lusk, and Gropp proposal could be implemented with each >>>>> processor contributing a different context value to the context >>>>> descriptor. E.g., process 3 says "if you want to talk to me >>>>> in this context, you have to use context value 7", while process >>>>> 4 says to use context value 2. I don't know whether there are >>>>> advantages to that extra flexibility. Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %**** Yes, that is included, I believe, in concept. %**** >>>>> Right. \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %**** %**** Stripping is extremely controversial aspect, and arbitrary. %**** If the recipient has the methods with the same name, then %**** a new rendezvous could be accomplished at the far end >>>>> >>>>> See my "Proposal VI" note for responses to this. \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %**** %**** Yes, these are all options. Fastness seems to be an important %**** issue. If translation is very expensive, none of the "good" %**** features will be used. >>>>> I was actually trying to be nice to the service process >>>>> people, giving them a designated hook where they could be >>>>> slow and I could work around them via the cacheing mechanism. >>>>> I'd really rather just specify it as being fast. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %**** Again, I think this should be eliminated, and all references %**** to this idea should be expunged. It denies the context's %**** ability to manage messages. >>>>> >>>>> When I wrote this, I was presuming that contexts could >>>>> not be generated "for free" and that therefore it was >>>>> a good idea to specify a method by which codes could >>>>> be written so as to not require them. By the way, >>>>> you may be interested to note that the sample collective >>>>> comms defined by Snir and Geist in their recent proposal >>>>> implicitly rely on exactly the paired-exact-match >>>>> constraint. As written, those routines would break >>>>> if preceded by a call to a module that issued a wildcard >>>>> receive in the same group. (And there's an endnote >>>>> that suggests they know that.) \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. >>>>> I think this is right. This counting strategy probably >>>>> places some constraints on how big the context value field >>>>> has to be, but I've incorporated it into Proposal VI. %**** I agree that context allocation has to be done in sequence. %**** That is why I am in favor of providing calls that allow %**** groups to get numerous contexts at creation, and then %****cooperatively, but potentially without further communication %**** divide them(as they build subgroups, for instance). %**** %**** I see these as services to be used in building virtual topology %**** features, which will then be more widely used by users of MPI. %**** \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %**** %**** Do we get anything by having the root process? %**** >>>>> The group partitioning concept has been refined in several >>>>> other postings to mpi-context, mpi-collcom, and mpi-pt2pt, >>>>> in which "root" was replaced by a "knownmember" set. >>>>> The idea is that if you have lots of processes joining a >>>>> group, they don't have to know who all their compatriots >>>>> are, but they should know at least one that they all have >>>>> in common, who can serve to organize the communications. \item Collective communication routines are called by all members of a group in the same order. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %**** %**** I think that non-blocking collcomm is moribund in MPI1 or %**** else MPI1 is moribund. :-) %**** >>>>> Nicely put. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %**** Too many negatives here for me to parse :-) \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %**** %**** Context allocation need not be free provided it can be made cheap, %**** or cheap enough. %**** %**** If one knows one will need several, then a single call could %**** provide such contexts, amortizing overhead. This is likely when %**** bulding grids (ie, virtual topologies) in Zipcode, so it is %**** true in existing practice. %**** %**** One should recognize the need for layering virtual top. calls %**** on top of these calls, then these calls may appear painful, %**** but perhaps they would be less used. Some users will use the %**** provided virtual topology calls, others will prefer their own. %**** Both will have equal power (see also,separate note on layerability). %**** %**** If getting N contexts is a send-and-receive, plus a reactive server, %**** then this is reasonably light weight,provided that hundreds of %**** messages, or global operations ensue thereafter. We can know in %**** advance how heavy weight the context server will be. %**** %**** if an implemention can use some locations of remote memory, with %**** fetch and add, or locks, to achieve contexts, then this is even %**** cheaper, in principle. %**** %**** Despite Jim's earlier insistence that context numbers be kept to %**** 256 or so, I think that this number should be much larger, so that %**** much less efort goes into returning contexts, and so on, except %**** occasionally, by processes. Otherwise, a new kind of overhead, %**** get-rid-of-context-because-I-am-out ensues, or programs block %**** until contexts become available, offering the possibility of %**** deadlocks. %**** \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %**** %**** Unscalability (of a limited form) should be permitted/selectable %**** by user, to use as much per-node memory as the user wants, to reduce %**** communication. %**** >>>>> Besides which, the system will do this anyway if (group,rank) >>>>> translations are required to be fast. All I'm doing is moving >>>>> it to the user level. \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. %**** I see,but what he means here is that there is no unpredictable %**** translation cost because we do not write (group,rank) in pt2pt %**** calls. So, there is some validity to the statement. >>>>> Tony has it right. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %**** Yes, that is why I have one in Zipcode. I wish Zipcode were %**** on netlib today, so you could try it. Well, we are writingthe %**** manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %**** %**** I lost something here, but I agree with the sense. The word %**** seems is subjective,and should disappear from our discussions, %**** as much as seems prudent, anyway :-) >>>>> Silly me -- trying to call out an opinion that might be >>>>> disagreed with. It occurred to me when I wrote the "s" word >>>>> that I'd probably be better off to just follow standard practice >>>>> and state my opinions as fact. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %**** %**** Yes,awkward, because it is nice to escape the TID realm and %**** work within the (albeit simple) abstraction of group,rank. %**** When layering virtual topologies on this, it would be so nice %**** to write them to a group,rank syntax, not enforcing TID mappings %**** everywhere. >>>>> >>>>> I happen to agree with this. But even though it's late, I can't >>>>> resist pointing out that there's not a shred of scientific >>>>> evidence that these opinions are in fact true. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %**** Indeed. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %**** Yes, and I think that loosely synchronous operations can maintain %**** coherency, in practice. That is, no operations that modify the %**** group descriptors (other than cached lookup info) are permitted, %**** without loose synchronization. %**** This is nasty in that is would prohibit sending descriptors to %**** processes not part of the group, so it is a clear trade-off. %**** Perhaps such send-to-non-group-member operations could stipulate %**** that this group information is somehow ephemeral, and that they %**** need to join a new group to keep useful information over time??? %**** \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %**** %**** Yes, that is exactly what Zipcode was written to avoid. The %**** user wants help managing things like this!!!! %**** %**** The search, if any, must be MPI-supported, and as efficient as %**** possible (eg, AVL trees, hash, partial hash with exceptions). %**** % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %**** %**** This is why I remain a strong advocate of (group,rank) %**** addresssing in pt2pt. %**** \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %**** %**** I addressed this in my Step-1 remarks. Please see that. %**** \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. >>>>> I thought I said that. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %**** %**** Thank you. These points are both well taken (ie these two paragraphs) >>>>> >>>>> See my comments near the start of this message, regarding >>>>> the proposed server implementation. What I should have said >>>>> was that MPI did not provide enough capability to construct >>>>> a server without consuming a processor, since it neither >>>>> provides an interrupt-receive function nor specifies that >>>>> processors be able to time-share between multiple processes. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %**** Also,see what I wrote in my Step-1 comments. %**** I believe O(log(P)) is still possible. %**** >>>>> I'd be interested in seeing the O(log(P)) algorithm, >>>>> sometime after MPI quiets down. \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %**** Please see my Step-1 comments. \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} ---------------------------------------------------------------------- Writing a server in the point-to-point layer of MPI in four easy steps >>>>> by potentially consuming an entire processor ---------------------------------------------------------------------- 1) Partition the INITIAL group into two groups. A singleton group, SERVER, and a group CLIENT which contains all of the other processes. 2) The single process in SERVER group records its TID. 3) The processes in INITIAL group allocate a context SERVICE which they remember either in the group cache or static data or something. 4) Use a broadcast in INITIAL group with ``sender'' as the one process which is also in SERVER group, and the ``receivers'' as the (many) processes which are also in CLIENT group, in the SERVICE context, in order to disseminate the TID of the server process. [Fanfare] a server process is in place as is a dedicated context for the purposes of messages required to implement the service. [Observation] the mpi point-to-point initialisation can do this automatically. %**** Zipcode's postmaster general works in this way, more or less. %**** - Tony ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:50:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26950; Mon, 22 Mar 93 05:50:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20829; Mon, 22 Mar 93 03:29:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 03:29:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20810; Mon, 22 Mar 93 03:29:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 00:13 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06118; Mon, 22 Mar 93 00:11:53 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15810; Mon, 22 Mar 93 00:11:50 PST Date: Mon, 22 Mar 93 00:11:50 PST From: rj_littlefield@pnlg.pnl.gov Subject: new proposal To: jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov Message-Id: <9303220811.AA15810@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Tony and Lyndon, I will respond in a separate message to some of your detailed comments on Proposal V. But maybe we can move faster by popping up a level. It seems to me that you like the idea of cacheing arbitrary info in group descriptors, you like the idea of groups as things within which contexts get formed, you like performance guarantees, and you don't like having to use opaque id's in point-to-point communications. I'm not quite sure whether static groups and synchronous group control are OK, but I'll presume here that they are. Well, this is pretty neat. If this keeps up maybe we can converge on just one or two proposals. Most of my gripes with other proposals have been based on performance and/or a need for asynchronous servers (with attendant performance and non-portability gripes). But I notice that explicit performance expectations have been gradually appearing and the need for servers disappearing from other people's proposals. Perhaps it is time for a synthesis. Here's a sketch of a new proposal (VI ?). . The functionality of point-to-point communications is per Snir, Lusk, and Gropp, augmented by my proposed MPI_FORM_CONTEXT to allow assembling arbitrary collections of processes. (Marc has already accepted this in a private email to me -- don't know why he didn't post it.) . Performance expectations of point-to-point communications are explicitly stated as follows: - MPI_COPY_CONTEXT does not synchronize the participating processes and costs significantly less than a point-to-point fanout among them (e.g., it uses a communication-free counting strategy); - all other context formation routines cost no more than if they were implemented using a single fanin/fanout among the participating processes; - translation of (context,rank) to absolute processor ID costs no more than if it were implemented via the lookup table that Snir suggests. . Groups and contexts are not equal. A group consists of a base context (from which other contexts can be created quickly by MPI_COPY_CONTEXT), plus topology information, plus my cacheing facility. Conceptually, I like this better than proposal V. Do we already have a proposal like this? Should we have one? In general, do you guys buy off on the concept of including performance expectations in the specification? A couple of discussion points... 1. The separation of group and context is defensive. I think I understand what it means to copy a context. I am not sure of either the functional or performance implications of copying a group. E.g., does cached info get copied? 2. I will respond here to two criticisms that have been raised against the cacheing facility. Lyndon notes > % I like the general idea, but I'm nervous about two things: > % (a) implied associativity of group descriptor cache - this > % will potentially be time expensive in implementation. > % (b) there is no method proposed for abritration of keys > % between independently written modules, so we are > % in the same problem regime as just having message tag > % and no message context. > % However, key's are local, so presumably you would find > % it acceptable to add a key registration service? I implicitly proposed a key assignment service in my long-ago example. It said in part: static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_key_type gop_key; /* key for this module's stuff */ ... if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetAttributeKey()) ) { MPI_abort ("Insufficient keys available"); } } This is not really "registration" because nothing goes into the key assignment routine except the number of times it's called. But assuming that each call site is protected by a separate variable, the effect is to register the call site. As Lyndon notes, this is highly local. But it also allows the returned keys to have their values restricted so as to permit rapid testing and/or retrieval. Tony notes > %**** Stripping is extremely controversial aspect, and arbitrary. > %**** If the recipient has the methods with the same name, then > %**** a new rendezvous could be accomplished at the far end Yes, stripping is arbitrary. My motivation is that this greatly simplifies the design and satisfies what I view as the most critical need: to make collective comms run fast without complicating the calling sequence. I have no objection in principle to extending the facility to include classes of information that do not get stripped. But I sure didn't want to try creating and selling a spec that would handle, e.g., heterogeneous systems. Enough for here -- what do you think of "proposal VI" ? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 05:53:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27630; Mon, 22 Mar 93 05:53:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01772; Mon, 22 Mar 93 05:53:21 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 05:53:19 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01761; Mon, 22 Mar 93 05:53:16 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA18140 (5.65c/IDA-1.4.4 for ); Mon, 22 Mar 1993 04:48:07 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA24191; Mon, 22 Mar 93 09:48:03 GMT Date: Mon, 22 Mar 93 09:48:03 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303220948.AA24191@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00311; Mon, 22 Mar 93 09:44:36 GMT To: tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu In-Reply-To: Tony Skjellum's message of Sun, 21 Mar 93 12:16:22 CST <9303211816.AA03587@Aurora.CS.MsState.Edu> Subject: mini-proposal on layerability Content-Length: 695 There is a typo in Tony's message : > So, each receipt function uses the algorithm > (received_rag & ~dont_care) & care This should read (received_tag & ~dontcare) == care i.e. don't care specifies a set of bits to be ignored in the comparison, care specifies the precise value of the other bits Note that you can do complete wild-carding like this by setting dont_care = -1; care = 0; -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Mon Mar 22 06:21:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03967; Mon, 22 Mar 93 06:21:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02909; Mon, 22 Mar 93 06:20:50 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 06:20:49 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02898; Mon, 22 Mar 93 06:20:46 -0500 Date: Mon, 22 Mar 93 10:01:09 GMT Message-Id: <13945.9303221001@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Rik's comments on Lyndon's Proposal I To: rj_littlefield@pnlg.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu In-Reply-To: rj_littlefield@pnlg.pnl.gov's message of Mon, 22 Mar 93 01:47:05 PST Reply-To: lyndon@epcc.ed.ac.uk Cc: d39135@sodium.pnl.gov Hi Rik Thanks for the comments. The proposal which I had sent out is dead. It has been replaced with a cut-and-paste of the section on contexts which Marc wrote into the point-to-point document. There is no proposal II. There is however a proposal VI. This naming/numbering is Tony's suggestion with which I concur. I will propogate the relevant comments you have made into the LaTeX of proposal VI (as identified LaTeX comments) and repost. I really am very sorry to have messed you about. I have a lot of other email, incl. from yourself, to read. (I am cancelling meetings left, right and centre here.) I will post later today. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 06:32:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06460; Mon, 22 Mar 93 06:32:56 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03270; Mon, 22 Mar 93 06:32:22 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 06:32:21 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03262; Mon, 22 Mar 93 06:32:15 -0500 Date: Mon, 22 Mar 93 11:32:10 GMT Message-Id: <14066.9303221132@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: Reference Point To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear Colleagues This message is a reference point summary of the current state of evolution of proposals. This is intended to help reduce confusion, as there has been a lot of development in the last 48 hours. Here is a complete summary of proposals to date, with proposal identifiers, contacts, status and brief description. In summary of the table: There are currently FOUR live proposals, one of which will shortly be circulated. These are labelled: I (in circulation); III/IV (not circulated); V (in circulated); VI (in circulated). There are currently THREE dead proposals. These are labelled: I++ (circulated as I); II (circulated?); II' (circulated as II). Identifier Status Contact Brief ---------- ------ ------- ----- I++ dead lyndon The "Proposal I" agreed at the post meet lunch, augmented by lyndon, and now defunct. This was circulated as Proposal I last week. Please kill this one. I live marc The proposal of marc which also appeared in the point-to-point document. It was cut-and-paste'd out of that document. This is currently in circulation as Proposal I. II dead rik The "Proposal II" agreed at the post meet lunch, now defunct. Please kill this one. II' dead lyndon A proposal unrelated to II. This has been named VI as of yesterday, to help avoid confusion. This was circulated as Proposal II' yestreday. Please kill this one. III/IV live tony The proposals III and IV agreed at the post lunch meet, to be merged. These have not yet appeared and should do within < 24 hours. This will be circulated as Proposal III/IV. V live Rik Proposal of Rik after abandonment of Proposal II. This was circulated as Proposal V in plain text and LaTeX source formats, last week. VI live lyndon This was, for about 4 hours, called II', yesterday. It has some common ground with I++ and other proposals. This was circulated as Proposal VI around 11:30pm GMT yesterday as LaTeX. There is currently a sketch which Rik is also naming as Proposal VI. Please Rik, either call this Proposal VII or call it a sketch, just to save confusion. The sketch suggests that we may have sufficient convergence to offer a single proposal. Tony and I discussed this yesterday evening and we concur. Tony and I suggest that the following occur: 1) Tony completes Proposal III/IV 2) Tony/Rik/Lyndon/??? discuss and hopefully agree a merger of VI, III/IV and V. There is no current suggestion to attempt to merge in I. I will call this Proposal X, for now. 3) if [ merger agreed in 2) ] then Draft contains Proposal I Proposal X else Draft contains Proposal I Proposal III/IV Proposal V Proposal VI fi 4) The false branch of 3) can be modified by a pair merger if we find that two of the three extant proposals can be merged but not all. There is currently also one mini-proposal from Tony dealing with layerability and tag selection. I understand that there will be one further mini-proposal from Tony dealing with threads. I sincerely apologise for having been the prime creator of confusion. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:10:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15088; Mon, 22 Mar 93 07:10:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04660; Mon, 22 Mar 93 07:09:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:09:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04652; Mon, 22 Mar 93 07:09:52 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00212 (5.65c/IDA-1.4.4 for ); Mon, 22 Mar 1993 07:09:47 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA24691; Mon, 22 Mar 93 12:09:44 GMT Date: Mon, 22 Mar 93 12:09:44 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303221209.AA24691@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00429; Mon, 22 Mar 93 12:06:23 GMT To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 22 Mar 93 11:32:10 GMT <14066.9303221132@subnode.epcc.ed.ac.uk> Subject: MPI-CONTEXT: Reference Point Content-Length: 599 Lyndon, since I've been rather busy with other things (which are of more immediate import to Meiko), I haven't been actively contributing to the debate. (Though I have been skimming, and filing all of the mail). Is it possible to pin down (or send out once again) the definitive versions of each proposal. (A copy of the table in your previous mail with a mail date and source for the top copy of each would do, alternatively a clean current copy of each...) Your last mail gives the references, but without following all of the history it's hard to get to the top copy of each. Thanks -- Jim From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:18:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16848; Mon, 22 Mar 93 07:18:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04947; Mon, 22 Mar 93 07:17:51 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:17:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04939; Mon, 22 Mar 93 07:17:45 -0500 Date: Mon, 22 Mar 93 12:17:35 GMT Message-Id: <14115.9303221217@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal Comments To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear Colleagues In the previous email "Reference Point" I sent out a summary of where the proposals are. I have been religously keeping "clean" copies of proposals in LaTeX form, and collating/embedding comments as LaTeX comments. I will now circulate the LaTeX source of each live proposal, with collated and up-to-date comments streams embedded therein. I will circulate PostScript (in which our comments do not appear) on request. Please, delete all previous circulations from your memory :-) First, I explain the comment notation I used. Reader comment lines begin with '%'. First line of block is %[reader-name] Reader comment to comment lines are considered literally as comments to comments and therefore begin with %%[reader-name]. This is a comment tree, as is the '%' indent tree of the LaTeX source. This has been time consuming and I do not propose to keep receiving comments and embedding in this fashion - things are more fluid than I had expected. Three LaTeX files to follow, in turn. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:18:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16981; Mon, 22 Mar 93 07:18:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04963; Mon, 22 Mar 93 07:18:29 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:18:28 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04954; Mon, 22 Mar 93 07:18:18 -0500 Date: Mon, 22 Mar 93 12:18:13 GMT Message-Id: <14121.9303221218@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal I - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal I\\MPI Context Subcommittee} \author{Marc~Snir} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal I" % Written by Marc Snir % Edited by Lyndon J. Clarke % March 1993 % \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \chapter{Proposal I} \section{Contexts} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \subsection{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \subsection{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \subsection{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \subsection{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \section{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \subsection{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } % % END "Proposal I" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:20:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17043; Mon, 22 Mar 93 07:20:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05007; Mon, 22 Mar 93 07:19:37 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:19:35 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04984; Mon, 22 Mar 93 07:19:13 -0500 Date: Mon, 22 Mar 93 12:18:58 GMT Message-Id: <14127.9303221218@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal V - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk ---------------------------------------------------------------------- % [Lyndon] General points % -------------- % % 1) I had to ``mine'' the text :-) Perhaps one of us (i.e., I am % offering if you wish) should attempt to construct a more transparent % presentation before circulation to whole committee, for the % convenience of committee members. %%[Tony] %% I felt that things appear twice, because of summary (good) %% and because of implementation notes at end (confusing) %% %%%[Rik] %%% I agree with both of the above. But let's decide whether %%% Proposal V will be replaced by VI or its equivalent before %%% we do any rewriting. %%%%[Lyndon] %%%% Proposal VI as referred to is the sketch which Rik sent out %%%% suggesting a merger. There is an actual Proposal VI circulated. % 2) I'm not a fan of much of this proposal, although I do indeed like % some of the ideas which it introduces. [On the other hand, I'm not a % great fan of all of the proposal which I wrote. I shall mail self % criticism of my proposal, and may have to write amended or alternative % proposal :-)] %%[Tony] %% Please be more specific. I am having a hard time understanding %% why you really don't like it, Lyndon. If the process model %% were a little less static, and servers were permitted (though %% hopefully bounded in cost), I think we would have an excellent %% proposal. %%%[Rik] %%% See Proposal VI. I'd be happy to see the good ideas adopted %%% and the crap die quietly. %% %%%[Lyndon] %%% I would have thought that the points below make my major %%% objections perfectly clear. Perhaps not. Here they are: %%% a) Paired-exact-match stuff %%% b) Translation of (group,rank) into TID all over the place % % 3) I really like the way in which groups are something like ``frames'' % in which contexts are created. This is conceptually much neater than % duplication of groups. %%[Tony] %% In practice, group subsetting will require groups to be copied, %% otherwise, subgroups will unfairly be penalized by the size %% of their ancestor. %%%[Tony] %%% Right, but I think of that as creating a new group. After all, %%% even the ranking structure is different. %% %%%[Lyndon] %%% I am anticipating that when one or more groups are created by %%% subsetting that, for example if the parent were described by %%% a proces list, then the children will be described by process %%% lists which are distinct sublists of the parent. So each element %%% of the parent list gets copied, exactly once. %%% The difficulty I have is that if a group were to be expanded %%% or contracted, then the ``duplicates'' thereof would no longer %%% be duplicates. Saying that duplicate creates a group bu retains %%% the process list of the old group is conceptually muddy since %%% the new group is a reference to a group, whereas the old group %%% or an even older group must be an actual group. Yuk! Now, if %%% we introduce the concept of a reference to an actual group, %%% and say this reference is unqieuly identified, then is is %%% conceptually sound and this object we describe really is a context. % % 4) I like the idea of pushing information into the group structure. I % have a few qualms with the proposed details --- see specific points. %%[Tony] %% I have more confidence about this idea, and could demonstrate %% by June/July time-frame in Zipcode. %% % % 5) See ``Writing a server in the point-to-point layer of MPI in four % easy steps'' at the foot of the message. %%[Tony] %% This seems like a nice thing. %%%[Tony] %%% The implementation that Lyndon suggests consumes an entire %%% process for the server. There are times when this is OK, %%% but also times when it isn't. E.g., if you have a divide-and- %%% conquer algorithm that really wants 2^i non-server processes, %%% and you're working on a machine with 2^n processors that %%% doesn't support multiple processes per processor, then %%% some of the users will get upset that they can only use %%% half the machine. A year and a half ago, I got flamed for %%% making a suggestion like this in connection with the Delta. %%% (And now I suppose I get flamed for using the Delta as an %%% example again... Maybe if I use the CM5...) %% %%%[Lyndon] %%% You are too kind :-) %% \documentstyle{report} \begin{document} \title{``Proposal V'' for MPI Communication Context Subcommittee} \author{Rik~Littlefield} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal V" % Rik Littlefield % March 1993 % \chapter{Proposal V} \section{Summary} \begin{itemize} \item Group ID's have local scope, not global. Groups are represented by descriptors of indefinite size and opaque structure, managed by MPI. Group descriptors can be passed between processes by MPI routines that translate them as necessary. (This satisfies the same needs as global group ID's, but its performance implications are easier to understand and control.) %[Lyndon] % I support the approach whereby group descriptors are local % objects. They could be pointers to structures, or indices % into tables thereof. We let the implementation consider that. % % One difficulty arises as group descriptors can only be passed % from process P to process Q if both P and Q members of some % group G since the communication presumably must use a context % known to both P and Q. Imagine that P is member of F and Q is not % member of F; that Q is member of H and P is not member of H; that % both P and Q are member of G. Let M be abritrary message data. % % Initially - % P can send F to Q, and Q can receive F from P, in a context of G. % Q can send H to P, and P can receive H from Q, in a context of G. % Thereafter - % P can allocate a context C in F. % P can send C to Q, and Q can receive C in the default context of H. % Q can allocate a context D in H. % Q can send D to P, and P can receive D, in the default context of F. % Thereafter - % P as member of F, and Q as member of H, can communicate using % wildcard pid and tag by use of contexts C and D. % % Okay, this is possible, but it is messy :-) %%[Tony] %% Alternatives, Lyndon? %% %%%[Lyndon] %%% I don't suppose for one minute that you will like this, but I really %%% would suggest that in this case a group descriptor registry may %%% be appropriate. %[Tony] % Seems doable. % %%[Lyndon] %% But usable with some grief, as above. \item Arbitrary process-local info can be cached in group descriptors. (This allows collective operations to run quickly after the first execution in each group, without complicating the calling sequences.) %[Lyndon] % I rather like this idea. %% [Tony] %% Me too. %%%[Rik] %%% Progress!! %[Tony] % Seems doable. % \item There are restrictions that permit groups to be layered on top of pt-pt. %[Tony] % I don't understand the word 'restriction' here. % Restriction of what. % %%[Lyndon] %% Rik is speaking of the pair-exact-match stuff you see later on. \item Pt-pt communications use only TID, context, and tag, and are specified to be fast. %[Tony] % What does "fast" mean. % %%[Lyndon] %% Fair question! \item Group control operations (forming, disbanding, and translating between rank and TID) are NOT specified to be fast. %[Tony] % OK, the above two items are identical to what Zipcode % provides in practice, but people have argued that groups % might be created/deleted more often in some apps, and % that these apps ought to be supportable % %%[Lyndon] %% In our work group creation/deletion is an infrequent operation %% and we are happy to live with reasonable cost for this operation. %% I think Marc Snir is thinking abour a different group model in %% group created/deletion is frequent. %% Perhaps we should provide both or neither (even handedness principle). \end{itemize} \section{Detailed Proposal} \begin{itemize} \item Pt-pt uses only ``TID'', ``context'', and ``message tag''. TID's, contexts, and message tags have global scope; they can be copied between processes as integers. TID's and contexts are opaque; their values are managed by MPI. Message tags are controlled entirely by the application. %[Lyndon] % You probably ought to say that context and TID are integer with % opaque values. %% [Tony] %% 1) It is not obvious that TIDs should be restricted to 32 bits. %%%[Lyndon] %%% I did not imply that they were. %% 2) It is not obvious that contexts will be 32 bits (eg, 16 bits). %% I favor a whole word for a context, despite other limits, %% just to make things simpler. %% %%%[Lyndon] %%% ditto %% Internet addresses are going to get augmented from 32 to ??? bits %% is it reasonable to assume that certain MPI implementations might %% incorporate such internet addresses as TIDs (in future), %%%[Lyndon] %%% No, it is not reasonable, in the least. And both you and Rik appear %%% to be ignoring the possibility that the process descriptor could %%% be required to store routing data of significant length. Therefore %%% the sensible thing to do is use a process descriptor which in %%% practice might be a table index --- fits into integer for sure --- %%% the value of which is process local for sure, and the table %%% contains the real process identifier of implementation defined size. %% Opacity is partially violated if we say how big the data type is??? %%%[Lyndon] %%% I understand the point you make, but it gets blown away by the %%% point I have just made in reply to you. %[Tony] % Yes, and I want at least 32-bits of message tag. % %%[Lyndon] %% Yes, and I want exactly zero bits of message tag. %% I'll just keep quiet about message tags. \item Pt-pt communication is specified to be fast in all cases. (E.g., MPI must initialize all processes such that any required translation of the TID is faster than the fastest pt-pt communication call.) %[Lyndon] % How do you imagine this to be acheived, considering that TIDs % are global entities? % I guess that you are thinking a TID is a (processor_number, % process_number) pair of bit fields, a bit like one sees in NX and RK, % and that network interface hardware will route based on the % processor_number. % % In another approach a TID is a process local entity just like the % group descriptor. This satisfies efficiency when the above scheme % is not applicable, for example in a workstation network. %% %%[Tony] %% where does this get us??? %% Remember, we have to choose on some things, so we can have something %% to present in Dallas. Is there an important difference here? %%%[Lyndon] %%% For sure their is an important difference. See comment above. %% %% TIDs are global entities. Is structure assumed to be global; %% in a truly opaque system, some TID component would have to be %% fixed, but the rest could vary structurally... %% %[Tony] % This could be difficult, in practice, if one mails a % message to one's own process, and MPI is smart enough % to optimize. % \item A group is represented by a ``group descriptor'', of indefinite size and opaque structure, managed by MPI. The group descriptor can be referenced via a ``process-local group ID'', which is typically just a pointer to the group descriptor. (Group ID's with global scope do not exist.) %[Lyndon] % I think I see, it is the context identifier which has global scope. % Now, this really is just getting on the way toward the proposal % that I really wish I had written for the subcommittee. I will flame % myself! %% %% Yes, contexts are global; group identifiers are just pointers %% typically, to data structures, describing %% %% 1) a groups context %% 2) group members and their ranks (mappings, inverses, %% cached, hashed, unscalably stored, etc) %% 3) TID-to-rank map and inverse (see possibilities in 2) %% 4) A set of fixed global operations, accepted as standard, %% an accessible in O(1) time. Possibly, each %% such operation should be a method, so that %% a parameter block can be passed with it. Zipcode %% supports the Method type to do this. %% This is effectively a cache for some parts of item #5 %% ... %% 5) An AVL or similar tree of extensible operations. %% New operations are registerable by the user. These %% tags are unique within a group, a specify an operation %% i) pre-defined by MPI (in which case it can be cached %% in 4 %% ii) alternative operations (even if they do something %% standard, that are wanted to be accessed by %% name) This name is group unique. %% %% A mechanism for DO_METHOD_FROM_GROUP(name,....) %% or GET_METHOD_FROM_GROUP(name,...) %% and SET_METHOD_IN_GROUP(name,...) are clearly needed. %% %%%[Rik] %%% The model of contexts used in this proposal is that the %%% context value has global scope. It doesn't have to be that way. %%% The Snir, Lusk, and Gropp proposal could be implemented with each %%% processor contributing a different context value to the context %%% descriptor. E.g., process 3 says "if you want to talk to me %%% in this context, you have to use context value 7", while process %%% 4 says to use context value 2. I don't know whether there are %%% advantages to that extra flexibility. %[Tony] % Sounds good. % \item Group descriptors can be copied between arbitrary processes via MPI-provided routines that translate them to and from machine- and process-independent form. A process receiving a group descriptor does not become a member of the group, but does become able to obtain information about the group (e.g., to translate between rank and TID). %[Lyndon] % Also crucially, to obtain and use the default context identifier % of the received group descriptor. %%[Tony] %% Yes, that is included, I believe, in concept. %%[Rik] %% Right. \item Arbitrary information can be cached in a group descriptor. This is, MPI routines are provided that allow any routine to augment a group descriptor with arbitrary information, identified by a key value, that subsequently can be retrieved by any routine having access to the descriptor. Cached information is not visible to other processes and is stripped when a group descriptor is sent to another process. Retrieval of cached information is specified to be fast (significantly cheaper than a pt-pt communication call). %[Lyndon] % I like the general idea, but I'm nervous about two things: % (a) implied associativity of group descriptor cache - this % will potentially be time expensive in implementation. % (b) there is no method proposed for abritration of keys % between independently written modules, so we are % in the same problem regime as just having message tag % and no message context. % However, key's are local, so presumably you would find % it acceptable to add a key registration service? %%[Tony] %% Stripping is extremely controversial aspect, and arbitrary. %% If the recipient has the methods with the same name, then %% a new rendezvous could be accomplished at the far end %%%[Rik] %%% See my "Proposal VI" note for responses to this. %[Tony] % In Zipcode 1.0, we allow multiple global operations % to be provided on a message-class (eg, grid-oriented messages) % The identifiers for these possible operations are user-specified % presently, but the "names" of the global operations are fixed % at compile-time. % % That means that there is O(1) time to find combine, fanout, send, % etc, on a group-wide scope. However, other operations cannot % be accessed in O(1) time (they are not in the opaque structure). % % The same mechanism used by Zipcode to allow multiple methods for % combine to be registered by the user, could also allow extensibility % just like Rik describes, with little effort. We use AVL trees. % % In fact, I will add this to Zipcode 1.x. Why say this? It is % not far from existing practice, and I have a lot of the machinery % in place already, and I am confident that it is useful. % \item Group descriptors contain enough information to asynchronously translate between (group,rank) and TID for any member of the group, by any process holding a descriptor for the group. MPI routines are provided to do these translations. Translation is not specified to be fast. %[Lyndon] % How do you imagine that this will be done? % (a) Perhaps an array of TIDs which is just indexed on rank? Then % where is the case for not using directly rank. % (b) Perhaps a hashing function? Then the case for not using rank % directly is marginal. % (c) Perhaps generating a request to a service process? In which % case you admit here that a service process exists, which must % be propogated throughout the proposal and changes one of your % fundamental objectives. % (d) Something else? Do tell! %%[Tony] %% Yes, these are all options. Fastness seems to be an important %% issue. If translation is very expensive, none of the "good" %% features will be used. %%%[Rik] %%% I was actually trying to be nice to the service process %%% people, giving them a designated hook where they could be %%% slow and I could work around them via the cacheing mechanism. %%% I'd really rather just specify it as being fast. %[Tony] % This seems to be a serious flaw. It will have to be cached % on an LRU basis, with system/user/both specifying how much % caching is allowed (ie, how much unscalable memory use). % If the first time is expensive, OK, but not the Nth time. % %%[Lyndon] %% Check. \item At creation, each group is assigned a globally unique ``default group context'' which is kept in the group descriptor and can be quickly extracted. This extraction is specified to be fast (comparable to a procedure call). %[Tony] % OK, I see no problem with this (so far). % \item The default group context can be used for pt-pt communication by any module operating within the group, but only for exactly paired send/receive with exact match on TID, context, and message tag. Call this the ``paired-exact-match constraint''. This constraint allows independent modules to use the same context without coordinating their tag values. (Assumes: tight sequencing constraints for both blocking and non-blocking comms in pt-pt MPI.) %[Lyndon] % I understand what you want the paired-exact-match for. This % might appear as pragmatics and advice to module writers. I % think you should be firmer about sequencing constraints % for point-to-point in MPI that this requires, to be % sure that the constraint is not too large. %%[Tony] %% Again, I think this should be eliminated, and all references %% to this idea should be expunged. It denies the context's %% ability to manage messages. %%%[Lyndon] %%% Check. %%%[Rik] %%% When I wrote this, I was presuming that contexts could %%% not be generated "for free" and that therefore it was %%% a good idea to specify a method by which codes could %%% be written so as to not require them. By the way, %%% you may be interested to note that the sample collective %%% comms defined by Snir and Geist in their recent proposal %%% implicitly rely on exactly the paired-exact-match %%% constraint. As written, those routines would break %%% if preceded by a call to a module that issued a wildcard %%% receive in the same group. (And there's an endnote %%% that suggests they know that.) %[Tony] % NO. This violates the concept of context entirely. % (ie, an oxymoron ... contexts same, but still no need for % tag disambiguation...) % % Use the default group context to establish (cooperatively) % other contexts, and then use these. This is a seriously % bad feature, in my mind. % \item A modules that does not meet the paired-exact-match constraint must use a different globally unique context value. An MPI routine will exist to generate such a value and broadcast it to the group. The cost of this operation is specified to be comparable to that of an efficient broadcast in the group, i.e. log(G) pt-pt startup costs for a group of G processes. [See Implementation note 1.] %[Lyndon] % Perhaps I am missing something here. Please help. This % is what my mind is thinking. % The synchronisation requirement means that all context % allocations in a group G must be performed in an identical % order by all members of G. Then the sequence number of the % allocation is unique among all allocations within G. % Therefore the duplet % (default context of G, allocation sequence number) % is a globally unique identification of the allocated % context. The sequence number can be replaced by any one-to-one % map of the sequence number, of course. So, according to your % synchronisation constraint, context generation can be ``free''. %%[Tony] %% I agree that context allocation has to be done in sequence. %% That is why I am in favor of providing calls that allow %% groups to get numerous contexts at creation, and then %% cooperatively, but potentially without further communication %% divide them(as they build subgroups, for instance). %% %% I see these as services to be used in building virtual topology %% features, which will then be more widely used by users of MPI. %% %%%[Lyndon] %%% If the context allocations are done in sequence, then I have %%% indicated how they can be done for free. I am getting confused. %%[Rik] %% I think this is right. This counting strategy probably %% places some constraints on how big the context value field %% has to be, but I've incorporated it into Proposal VI. %[Tony] % I do not think we should support the paired-exact-match thing. % \item Context values are a plentiful but exhaustible resource. If further use is anticipated, a context may be cached; otherwise it should be returned. (Note: the number of available contexts should be several times the number of groups that can be formed.) %[Tony] % Concur. This suggests many more than "256" % %%[Lyndon] %% The number of contexts in the whole program? Sure 256 is too small! %% The number of contexts in each process? Maybe something like 256 is okay? \item When a group disbands, its group descriptor is closed and any cached information is released. (The MPI group-disbanding routine does this by calling destructor routines that the application provided when the information was cached.) Copies of the group descriptor must be explicitly released. It is an error to make any reference to a descriptor of a disbanded group, except to release that descriptor. \item All processes that are initially allocated belong to the INITIAL group. (In a static process model, this means all processes.) An MPI routine exists for obtaining a reference to the INITIAL group descriptor (i.e., a local group ID for INITIAL). \item Groups are formed and disbanded by synchronous calls on all participating processes. In the case of overlapping groups, all processes must form and disband groups in the same order. [See Implementation Note 2.] %[Tony] % This is the Zipcode model. It could say loosely synchronous. % \item All group formation calls are treated as a partitioning of some group, INITIAL if none other. The calls include a reference to the descriptor of the group being partitioned, the TID of the new group's ``root process'', the number of processes in the group, and an integer ``group tag'' provided by the application. The group tag must discriminate between overlapping groups that may be formed concurrently, say by multiple threads within a process. [See Implementation Note 2.] %[Lyndon] % The group partition you propose is essentially no different to % the partition by key which has already been discussed, except % that the key can encapsulate both (root process, group tag). % So perhaps partition by key was better in the first place? %%[Tony] %% Do we get anything by having the root process? %% %%%[Lyndon] %%% No. %%%[Rik] %%% The group partitioning concept has been refined in several %%% other postings to mpi-context, mpi-collcom, and mpi-pt2pt, %%% in which "root" was replaced by a "knownmember" set. %%% The idea is that if you have lots of processes joining a %%% group, they don't have to know who all their compatriots %%% are, but they should know at least one that they all have %%% in common, who can serve to organize the communications. %[Tony] % I don't understand the thread issue here. % %%[Lyndon] %% If two threads are concurrently partitioning the same group, you %% need to disambiguate the partition operations. Analagous to %% concurrent collective operations or nonblocking. \item Collective communication routines are called by all members of a group in the same order. %[Tony] % Yes. \item Blocking collective communication routines are passed only a reference to the group descriptor. To communicate, they can either use the default group context or internally allocate a new context, with or without cacheing. %[Tony] % What does caching really imply here ??? Help. % %%[Lyndon] %% Dunno. \item Non-blocking collective communication routines are passed a reference to the group descriptor, plus a ``data tag'' to discriminate between concurrent operations in the same group. (Question: is sequencing enough to do this?) An implementation is allowed but not required to impose synchronization of all cooperating processes upon entry to a non-blocking collective communication routine. %[Lyndon] % If the requirement that collective operations within a group G are % done in the identical order by all members of G even when such % operations are non-blocking, then the sequence number of the operation % is unique and sufficient for disambiguation. % % The permission to force synchronisation - i.e., blocking - in the % implementation of a non-blocking routine seems to make the routine % less than useful. I can see whay you are asking for this, in order % that you can generate a context for the routine call. In fact Rik % I don't think you need the constraint, as I pointed out cheaper % context generation exists above, unless of course I am missing % something. %%[Tony] %% I think that non-blocking collcomm is moribund in MPI1 or %% else MPI1 is moribund. :-) %% %%%[Lyndon] %%% Check. %%%[Rik] %%% Nicely put. %[Tony] % I think that contexts are really important in this case, % to keep things straight, but that non-blocking collcomm should % be omitted from MPI1 (cf, Geist). Sequencing supports % a sufficient disambiguation, as long as the entire group % is always the participant in operations. That is, you have % to form subgroups, with new contexts, to do global ops on % subsets. \end{itemize} \section{Advantages \& Disadvantages} \subsection*{Advantages} \begin{itemize} \item This proposal is implementable with no servers and can be layered easily on existing systems. %[Lyndon] % I am not of the opinion that the absence of services is such a big % deal. I do think that programs which can conveniently not use % services should not be forced to, but programs which cannot % conveniently not use services should be allowed to. %%[Tony] %% Too many negatives here for me to parse :-) %[Tony] % Why aren't servers needed to create contexts. Where do they % come from? If you rely on the fact that INITIAL will do % a loosely synchonous cooperative operation each time a new % context is needed, then a simple (easily implementable server, % or fetch-and-add remote access) is replaced by a more rigid % computation model. % % If we can get rid of this disagreement, me might be able to % reduce our total proposal space by one whole proposal. % \item Cacheing auxiliary information in group descriptors allows collective operations to run at maximum speed after the first execution in each group. (Assumes that sufficient context values are available to permit cacheing.) %[Lyndon] % If you agree that context allocation is ``free'', then you can % delete the bracketed qualifier. %%[Tony] %% Context allocation need not be free provided it can be made cheap, %% or cheap enough. %% %% If one knows one will need several, then a single call could %% provide such contexts, amortizing overhead. This is likely when %% bulding grids (ie, virtual topologies) in Zipcode, so it is %% true in existing practice. %% %% One should recognize the need for layering virtual top. calls %% on top of these calls, then these calls may appear painful, %% but perhaps they would be less used. Some users will use the %% provided virtual topology calls, others will prefer their own. %% Both will have equal power (see also,separate note on layerability). %% %% If getting N contexts is a send-and-receive, plus a reactive server, %% then this is reasonably light weight,provided that hundreds of %% messages, or global operations ensue thereafter. We can know in %% advance how heavy weight the context server will be. %% %% if an implemention can use some locations of remote memory, with %% fetch and add, or locks, to achieve contexts, then this is even %% cheaper, in principle. %% %% Despite Jim's earlier insistence that context numbers be kept to %% 256 or so, I think that this number should be much larger, so that %% much less efort goes into returning contexts, and so on, except %% occasionally, by processes. Otherwise, a new kind of overhead, %% get-rid-of-context-because-I-am-out ensues, or programs block %% until contexts become available, offering the possibility of %% deadlocks. %% %[Tony] % If contexts are being used very dynamically, how are they being % assigned, kept, released, reissued without a server? Sorry if % I missed something, but I don't see it, without a restrictive % SPMD model of computation (Zipcode obviates its server for the % SPMD model, for instance). %%[Lyndon] %% MPI stinks of SPMD. I wouldn't mind if MPI would just say SPMD. \item Speed of cached collective operations does not depend on speed of group operations such as formation or translation between rank and TID. %[Lyndon] % True, but the cache is going to get big as user's are going to store % arrays of TIDs in it. %%[Tony] %% Unscalability (of a limited form) should be permitted/selectable %% by user, to use as much per-node memory as the user wants, to reduce %% communication. %% %%%[Rik] %%% Besides which, the system will do this anyway if (group,rank) %%% translations are required to be fast. All I'm doing is moving %%% it to the user level. %[Tony] % Can you clarify this with examples. % \item Requires explicit translation between (group,rank) and TID, which makes pt-pt performance predictable no matter how the functionality of groups gets extended. %[Lyndon] % This is only true because you have asserted that implementations % must have the property that: % `` Pt-pt communication is specified to be fast in all cases. % (E.g., MPI must initialize all processes such that any % required translation of the TID is faster than the fastest % pt-pt communication call.)'' % So the advantage is not that which you have quoted, it is that % you have made this assertion. % %%[Tony] %% I see,but what he means here is that there is no unpredictable %% translation cost because we do not write (group,rank) in pt2pt %% calls. So, there is some validity to the statement. %%%[Lyndon] %%% However he ignores that the TID might require translation the %%% cost of which might be unpredictable. This because the TID has %%% a global value and cannot therefore hold process local information %%% such as ``how do I route to that process''. %%%[Rik] %%% Tony has it right. %[Tony] % I like this, of course. \item Communication both within and between groups seems conceptually straightforward. %[Lyndon] % This is a conjecture. I believe that conjecture to be false. % I especially believe this in the case of communication between % groups. The methods which are available for ``hooking up'' % allows are at least perverse. I guess that the user could make % use of a service process, to make life easier in this hooking up, % so whay not provide one. %%[Tony] %% Yes, that is why I have one in Zipcode. I wish Zipcode were %% on netlib today, so you could try it. Well, we are writingthe %% manual, and working at it as fast as we can. % % A further point. It seems to me that ``seems'' means that it seems % to you. This is not the point. It is how it seems to a lesser % wizard than yourself which is of importance here. I conjecture % that the reverse statment is true when the person doing the seeming % is changed to a lesser wizard. %%[Tony] %% I lost something here, but I agree with the sense. The word %% seems is subjective,and should disappear from our discussions, %% as much as seems prudent, anyway :-) %%%[Rik] %%% Silly me -- trying to call out an opinion that might be %%% disagreed with. It occurred to me when I wrote the "s" word %%% that I'd probably be better off to just follow standard practice %%% and state my opinions as fact. %[Tony] % Well, is point-to-point group oriented. Not. %%[Lyndon] %% Check. \end{itemize} \subsection*{Disadvantages} \begin{itemize} \item Requires explicit translation between (group,rank) and TID, which may be considered awkward. %[Lyndon] % It is true that (group,rank) must be translated to TID. I can % assure you that this is considered both awkward and redundant. %%[Tony] %% Yes,awkward, because it is nice to escape the TID realm and %% work within the (albeit simple) abstraction of group,rank. %% When layering virtual topologies on this, it would be so nice %% to write them to a group,rank syntax, not enforcing TID mappings %% everywhere. %%%[Rik] %%% I happen to agree with this. But even though it's late, I can't %%% resist pointing out that there's not a shred of scientific %%% evidence that these opinions are in fact true. %[Tony] % I think it is awkward. \item Communication between different groups may be considered awkward. %[Lyndon] % You bet! Please see below. %%[Tony] %% Indeed. %%%[Lyndon] %%% More so than you think, I think! %[Tony] % OK, but one can form a new group, as I have argued before. % Use the "awkward" pt2pt to get the right info shared between % group leaders, make the new group, use unawkward collective % operations on new group (with new context). %%[Lyndon] %% This is only one model of group-group interaction, which in my %% experience and understanding really is still steeped in SPMD. %% Please consider the examples of non SPMD group usage which I %% mailed out. You can say to me - oh, you shouldn't do this kind %% of thing with MPI, Lyndon - if you like, if you believe that. \item No free-for-all group formation. A process must know something about who its collaborators are going to be (minimally, the root of the group). %[Lyndon] % Please see comments above on group creation. %[Tony] % This again is in practice, in Zipcode. % \item Requires explicit coherency of group descriptors, which is not convenient -- application must keep track of which processes have copies and what to do with them. %[Lyndon] % I think all of the proposals will have this problem. %%[Tony] %% Yes, and I think that loosely synchronous operations can maintain %% coherency, in practice. That is, no operations that modify the %% group descriptors (other than cached lookup info) are permitted, %% without loose synchronization. %% This is nasty in that is would prohibit sending descriptors to %% processes not part of the group, so it is a clear trade-off. %% Perhaps such send-to-non-group-member operations could stipulate %% that this group information is somehow ephemeral, and that they %% need to join a new group to keep useful information over time??? %% %%%[Lyndon] %%% I agree that the (loosely) synchronous semantics remove coherency %%% until processes find out about groups of which they are not %%% members. I cannot advocate prohibition of this. I know there is a %%% coherency issue and I think we should let the user deal with it. %%% In practice, the user implements protocols such that if group A %%% communicates with group B, then before group B is destroyed it %%% must have let group A know of its impending doom, unless group A %%% can already have knowledge that this will be the case, which it %%% often can anyway. %[Tony] % Sounds dangerous. What must application do to maintain % coherency, since group descriptors are opaque. % \item Static group model. Processes can join and leave collaborations only with the cooperation of the other group members in forming larger or smaller groups. %[Tony] % No, loosely synchronous process model, unless you mean % cooperation of INITIAL at all such join/leave steps. %%[Lyndon] %% Tony, I have difficulty understanding what you mean. If you are %% thinking of group creation by giving a list of group members then %% I can understand. If you are thinking of the partition operation %% then I cannot understand as this operation seems (to me:-) to be %% a synchronisation anyway. \item No direct support for identifying the group from which a message came. The sender cannot embed its group ID in its message, since group ID's are not globally meaningful. One method to address this problem is to have the sender embed its group context as data in the message, then have the receiver use that value to select among group descriptors that it had already been sent. %[Lyndon] % Yup, the user can ``do it manually with a search''. If you want % to invoke this argument then I can dispose of almost everything % in MPI in a period of a few minutes - in fact Steven Zenith will % do it faster - so I refute the validity of the argument and claim % that the MPI interfce should transmit said information. %%[Tony] %% Yes, that is exactly what Zipcode was written to avoid. The %% user wants help managing things like this!!!! %% %% The search, if any, must be MPI-supported, and as efficient as %% possible (eg, AVL trees, hash, partial hash with exceptions). %% % % Further, the receiver is likely to want to be able to ask which % rank in the sender group the sender was. Oh dear, well I suppose % you think that's okay because the sender can put its rank into % the message. This is just being inconvenient to the user who % wants to send an array of something (double complex?) and has % to pack a rank in by copying or sending a pre-message or the % buffer descriptor kind of thing. %%[Tony] %% This is why I remain a strong advocate of (group,rank) %% addresssing in pt2pt. %% %%%[Lyndon] %%% Et moi! %[Tony] % No, you can't know the group or rank in group of sender. % If there were one context per group (isn't that so here?), % then all you need is the rank. With TID_TO_RANK_IN_GROUP % operation, this could be provided, but no wildcarding % or receipt selectivity could be done at this level. % %%[Lyndon] %% Where a member of group A sends to a member of group B then the %% receiver in group B really does often want to know the rank of the %% in sender in group A. This is not hypothetical, we program this way. \end{itemize} \section{Comments \& Alternatives} \begin{itemize} \item The performance of group control functions is explicitly unspecified in order to provide performance insurance. The intent is to encourage program designs whose performance won't be killed if the functionality of groups is expanded so as to be unscalable or require expensive services. %[Lyndon] % I don't think that the intent expressed in the second sentence % is satisfied. For example - group control is allowed to become the % dominant feature of application time complexity. %%[Tony] %% I addressed this in my Step-1 remarks. Please see that. BELOW %[Tony] % No, it just does not provide guarantees that certain kinds % of applications will run OK. (ie, those that do group % creation/deletion relatively often). Zipcode has assumed % that such operations would be relatively seldom. Thus, I do % not quibble that this is a reasonable choice,but a fairer % way to say this is that it may be difficult to support such % applications. That reveals an issue to be studied more. % %%[Lyndon] %% Check. On performance specification this belongs in the %% implementation profile and the specification actually %% cannot say anything about performance. %% \item Adding global group ID's would seriously interfere with layering this proposal on pt-pt. The problem is not generating a unique ID, but using it. If just holding a global group ID were to allow a process to asynchronously ask questions about the group (like translating between rank and TID), then a group information server of some sort would be required, and MPI pt-pt does not provide enough capability to construct such a beast. %[Lyndon] % It is not the global uniqueness of group identifiers which creates % the problem. There are globally unique labels of groups in your % proposal anyway - the value of the default context identifier. % The problem is that of allowing query of group information when % that information cannot be recorded in the local process/processor % memory. %% [Rik] %% I thought I said that. % % You claim that point-to-point does not have enough capability to % construct an information server. Firstly I should ask you whether % this is an artefact of the manner in which you have defined the % point-to-point communication. Secondly I assert that your claim % is false. I shall append a description of server implementation % to the foot of this message. %%[Tony] %% Thank you. These points are both well taken (ie these two paragraphs) %%%[Tony] %%% See my comments near the start of this message, regarding %%% the proposed server implementation. What I should have said %%% was that MPI did not provide enough capability to construct %%% a server without consuming a processor, since it neither %%% provides an interrupt-receive function nor specifies that %%% processors be able to time-share between multiple processes. %[Tony] % Perhaps they should do. \item The restriction that only paired-exact-match modules can run in the default group context could be relaxed to something like ``a module that violates the paired-exact-match constraint can run in the default group context if and only if it is the only module to run in that context''. This seems too error-prone to be worth the trouble, since even the standard collective ops might be excluded. (It would also conflict with Implementation Note 1.] %[Tony] % Dump this. \item Group formation can be faster when more information is provided than just the group tag and root process. E.g. if all members of the group can identify all other members of a P-member group, in rank order, then O(log P) time is sufficient; if only the root is known, O(P) time may be required. This aspect should be considered by the groups subcommittee in evaluating scalability. %[Lyndon] % Yes, partition does appear to be O(P) whereas definition by ordererd % list appears to be O(log(P)). %%[Tony] %% Also,see what I wrote in my Step-1 comments. BELOW. %% I believe O(log(P)) is still possible. %% %%%[Rik] %%% I'd be interested in seeing the O(log(P)) algorithm, %%% sometime after MPI quiets down. %%%%[Lyndon] %%%% Me too! %[Tony] % No, a non-deterministic broadcast can be used, with a token. % This requires a token server. Again, implementable with fetch+ % add on most systems, or a light reactive server. % % Once the non-deterministic broadcast has finished, a fanin/collapse % is done to the original root, which then frees the token. % \end{itemize} \section{Implementation Notes} \begin{enumerate} \item To generate and broadcast a new context value: Generate the context value without communication by concatenating a locally maintained unique value with the process number in some 0..P-1 global ordering scheme. This assumes that context values can be significantly longer than log(P) bits for an application involving P processes. If not, then a server may be required, in which case by specification it has to be fast (comparable to a pt-pt call). There is no need to generate a new context for the broadcast since it can be coded to use the default group context by meeting the paired-exact-match constraint.) %[Lyndon] % Please see notes above on the subject of context generation. %%[Tony] %% Please see my Step-1 comments. %[Tony] % Why not just given in and allow the server. % I don't like the paired-exact-match constraint AT ALL. % %%[Lyndon] %% Ni moi! \item The following is intended only as a sanity check on the claim that this proposal can be layered on MPI. Improvements to improve efficiency or to use fewer contexts will be greatly appreciated. In addition to a default group context, each group descriptor contains a ``group partitioning context'' and a ``group disbanding context'' that are obtained and broadcast at the time the group is created. In the case of the INITIAL group, this is done using paired-exact-match code and any context, immediately after initialization of the MPI pt-pt level. (At that time, no application code will have had a chance to issue wildcard receives to mess things up.) %[Tony] % Seems OK, but why need the paired-exact-match thing again. % Group partitioning is accomplished using pt-pt messages in the partitioning context of the current group (i.e., the one being partitioned), with message tag equal to the group tag provided by the application. In the worst (least scalable) case, the root of the new group must wildcard the source TID. (This violates the paired-exact-match constraint, which is why group formation must happen in a special context.) %[Tony] % Again, OK, but I want to see this work without the paired-exact- % match, if possible. Group disbanding is done with pt-pt messages in the group disbanding context. This keeps group control from being messed up no matter how the application uses wildcards. %[Tony] % So, now, you have concurred with my (previously flamed) idea % that group construction/destruction should be realizable using % pt2pt, just like global operations should do. I like this % because 1) it is explicable to the implementor, 2) it allows % simple intitial implemtations, 3) it sets some ideas for how % much these things will cost [upper bound]. % %%[Lyndon] %% I still do not concur. It is restrictive on what we can do. \end{enumerate} \section{Examples} {\bf (To be provided)} % % END "Proposal V" %======================================================================% \end{document} %[Lyndon] % Writing a server in the point-to-point layer of MPI in four easy steps %%[Rik] %% by potentially consuming an entire processor %%%[Lyndon] %%% Oh dear, how sad. I know what a pain it can be when you have one %%% thumping great number cruncher which cant multitask, and its sits %%% there as a do nothing server all day long. But look, it's a do %%% nothing server, so it *can* run in the user environment like on %%% the ``host'' or ``login'' processor. Look again, this is because %%% of the ``cannot multitask'' - let us hope that is history :-) % ---------------------------------------------------------------------- % % 1) Partition the INITIAL group into two groups. A singleton group, % SERVER, and a group CLIENT which contains all of the other processes. % % 2) The single process in SERVER group records its TID. % % 3) The processes in INITIAL group allocate a context SERVICE which % they remember either in the group cache or static data or something. % % 4) Use a broadcast in INITIAL group with ``sender'' as the one process % which is also in SERVER group, and the ``receivers'' as the (many) % processes which are also in CLIENT group, in the SERVICE context, in % order to disseminate the TID of the server process. % % [Fanfare] a server process is in place as is a dedicated context for % the purposes of messages required to implement the service. % % [Observation] the mpi point-to-point initialisation can do this % automatically. %%[Tony] %% Zipcode's postmaster general works in this way, more or less. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 07:20:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17057; Mon, 22 Mar 93 07:20:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05062; Mon, 22 Mar 93 07:20:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 07:20:15 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05009; Mon, 22 Mar 93 07:19:57 -0500 Date: Mon, 22 Mar 93 12:19:43 GMT Message-Id: <14133.9303221219@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MPI-CONTEXT: PLEASE README - Proposal VI - LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk ---------------------------------------------------------------------- \documentstyle{report} \begin{document} \title{Proposal VI\\MPI Context Subcommittee} \author{Lyndon~J~Clarke} \date{March 1993} \maketitle %======================================================================% % BEGIN "Proposal VI" % Written by Lyndon J. Clarke % March 1993 % \newcommand{\LabelNote}[1]{\label{vi:note:#1}} \newcommand{\SeeReferNote}[1]{{\bf See~Note~\ref{vi:note:#1}.}} \newcommand{\LabelSection}[1]{\label{vi:sect:#1}} \newcommand{\ReferSection}[1]{{\bf Section~\ref{vi:sect:#1}.}} \chapter{Proposal VI} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction} This chapter proposes that communication contexts and process groupings within MPI appear as related concepts. In particular process groupings appear as ``frames'' which are used in the creation of communication contexts. Communications contexts retain a reference to, and inherit properties of, their process grouping frames. This reflects the observation that an invocation of a module in a parallel program typically operates within one or more groups of processes, and as such any communication contexts associated with invocations of modules also bind certain semantics of process groupings. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of MPI, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language for only purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The notes assumes knowledge of the proposal text and are therefore best examined after familiarisation with that text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the text of the proposal. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal} \LabelSection{proposal} This section presents the detailed proposal, discussing in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes} \LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and MPI does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{process:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of MPI services. MPI provides a procedure which returns the own descriptor of the calling process. For example, {\tt pd = mpi\_own\_pd()}. \subsubsection*{Process Creation and Destruction} This proposal makes no statements regarding creation and destruction of processes. \SeeReferNote{dynamic:processes} \subsubsection*{Descriptor Transmission} The value of a process descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %%[Lyndon] %% Since it is an integer one cannot prevent the user from passing it. %% The discussion notes should help answer why is is an integer. More %% here. %% We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. %% Because rank will be an integer, then pd must also be an integer, %% so I write that all descriptors are integers for consistency beteen %% the different descriptors. %% MPI provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_PD\_TYPE}. MPI provides a process descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %%[Lyndon] %% I don't understand what you don't get. A registry service is just %% an easier (nonscalable, okay) way for concurrent entities to hook up %% with one another, rather than having some common ancestor send descriptors %% around in messages. %% In fact I don't really need a group or process registry service, as mentioned %% in the discussion section (yes, I know, not well presented), but %% I do need a context registry service, and I'm being consistent %% between different kinds of descriptors again. %% Suggestive syntax for a registry service is pretty boring, so %% I skipped it. %% Note that receipt of a process descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_pd(pd)}. The user is not allowed to invalidate the process descriptor of the calling process. \SeeReferNote{process:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %%[Lyndon] %% Acquisition of a descriptor requires an allocator, and consumes %% resource. In such cases it is good practice to provide a %% deallocator which frees resource. %% % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %%[Lyndon] %% Unless I am missing the additional point in this comment, I can't %% see the problem. Probably poor presentation of Proposal I++ :-) %% \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding processor descriptor attributes. \subsection{Process Groupings} \LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to?) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. \SeeReferNote{dynamic:groups} The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to the descriptor of the $N$ processes composing the grouping. The structure of a process grouping is defined by a process grouping topology and this proposal makes no further statements regarding such structures. %[Tony] % It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % %%[Lyndon] %% I am deliberately making weak statements about topology while %% acknolwedging the the process topology subcommittee. %% \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{group:identifiers} \SeeReferNote{descriptor:cache} The initialisation of MPI services will assign to each process an {\it own} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the process grouping until the termination of MPI services. \SeeReferNote{group:blobs} MPI provides a procedure which returns the descriptor of the home group of the calling process. For example, {\tt gd = mpi\_home\_gd()}. \SeeReferNote{own:group} \subsubsection*{Group Creation and Deletion} MPI provides a procedure which allows users to dynamically create one or more groups which are subsets of existing groups. For example, {\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} partition an existing group {\tt gda} into one or more distinct subsets. This procedure is called by and synchronises all members of {\tt gda}. MPI provides a procedure which allows users to dynamically create one group by explicit definition of its membership as a list of process descriptors. For example, {\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. %[Tony] % loosely synchronous % %%[Lyndon] %% Do we mean the same thing? the constructors require each member %% if the object under construction to participate in the construction, %% and return a descriptor for the constructed operation. Therefore %% no member can terminate construction before all other members have %% commenced, at least. %% %[Rik] % Why such strong synchronization?! % If this has the same synchronization properties as creation, % then it won't return until all members have made the call. % But that means you actually have to sync everybody, which % implies 2 log(P) messages. Isn't it enough to just require % a loosely synchronous call, and use a counting strategy> % MPI provides a procedure which allows users to delete a created group. This procedure accepts the descriptor of a group which was created by the calling process and destroys the identified group. For example, {\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. MPI provides additional procedure which allow users to construct process groupings which have a process grouping topology. \subsubsection*{Descriptor Transmission} The value of a group descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %%[Lyndon] %% Since it is an integer one cannot prevent the user from passing it. %% The discussion notes should help answer why is is an integer. More %% here. %% We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. %% Because rank will be an integer, then pd must also be an integer, %% so I write that all descriptors are integers for consistency beteen %% the different descriptors. %% MPI provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_GD\_TYPE}. MPI provides a group descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. %[Tony] % I don't get all of this. Why? % %%[Lyndon] %% I don't understand what you don't get. A registry service is just %% an easier (nonscalable, okay) way for concurrent entities to hook up %% with one another, rather than having some common ancestor send descriptors %% around in messages. %% In fact I don't really need a group or process registry service, as mentioned %% in the discussion section (yes, I know, not well presented), but %% I do need a context registry service, and I'm being consistent %% between different kinds of descriptors again. %% Suggestive syntax for a registry service is pretty boring, so %% I skipped it. %% Note that receipt of a group descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_gd(gd)}. The user is not allowed to invalidate the own group descriptor of the process or the group descriptor of any group created by the calling process. \SeeReferNote{group:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %%[Lyndon] %% Acquisition of a descriptor requires an allocator, and consumes %% resource. In such cases it is good practice to provide a %% deallocator which frees resource. %% % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %%[Lyndon] %% Unless I am missing the additional point in this comment, I can't %% see the problem. Probably poor presentation of Proposal I++ :-) %% \subsubsection*{Descriptor Attributes} MPI provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identifier group. For example, {\tt rank = mpi\_group\_rank(gd)}. MPI provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, {\tt size = mpi\_group\_size(gd)}. MPI provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, {\tt pd = mpi\_group\_pd(gd, rank)}. \SeeReferNote{pd:to:rank} MPI provides additional procedures which allow users to determine the process grouping topology attributes. %[Tony] It is not obvious to me that we want to enforce topology at this % juncture. However, we could register topology information in % the extensible structure strategy of Proposal V. % %%[Lyndon] %% I am deliberately making weak statements about topology while %% acknolwedging the the process topology subcommittee. %% \subsection{Communication Contexts} \LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping is used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} which is expressed as an integer type in the host language and has an opaque value which is process local. \SeeReferNote{integer:descriptors} \SeeReferNote{context:identifiers} \SeeReferNote{descriptor:cache} The creation of MPI process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains its own context until MPI process grouping deletion. MPI provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, {\tt cd = mpi\_own\_cd(gd)}. \SeeReferNote{own:context} \subsubsection*{Context Creation and Deletion} MPI provides a procedure which allows users to dynamically create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, {\tt cd = mpi\_context\_create(gd)}. This procedure is called by and synchronises all members of {\tt gd}. \SeeReferNote{dynamic:contexts} %[Tony] % loosely synchronous % %%[Lyndon] %% Do we mean the same thing? the constructors require each member %% if the object under construction to participate in the construction, %% and return a descriptor for the constructed operation. Therefore %% no member can terminate construction before all other members have %% commenced, at least %% MPI provides a procedure which allows users to destroy created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes that context identifier. For example, {\tt mpi\_context\_delete(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. \subsubsection*{Descriptor Transmission} The value of a context descriptor can be transmitted in a message as an integer since it is an integer type in the host language. However the recipient of the descriptor can make no defined use of the value of in the MPI operations described in this proposal --- the descriptor is {\it invalid}. %[Tony] % Then why allow it to be passed? % %%[Lyndon] %% Since it is an integer one cannot prevent the user from passing it. %% The discussion notes should help answer why is is an integer. More %% here. %% We'd like (gd,rank) and (NULL,pd) to be suppliable to point-to-point. %% Because rank will be an integer, then pd must also be an integer, %% so I write that all descriptors are integers for consistency beteen %% the different descriptors. %% MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_CD\_TYPE}. %[Tony] % I don't get all of this. Why? % %%[Lyndon] %% I don't understand what you don't get. A registry service is just %% an easier (nonscalable, okay) way for concurrent entities to hook up %% with one another, rather than having some common ancestor send descriptors %% around in messages. %% In fact I don't really need a group or process registry service, as mentioned %% in the discussion section (yes, I know, not well presented), but %% I do need a context registry service, and I'm being consistent %% between different kinds of descriptors again. %% Suggestive syntax for a registry service is pretty boring, so %% I skipped it. %% MPI provides a context descriptor registry service. The operations which this service upports are: register descriptor by name; deregister descriptor; lookup descriptor identifier by name and validate, blocking the caller until the name has been registered. \SeeReferNote{registry:check} Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. Note that receipt of a context descriptor in the fashions described above may have a persistent effect on the implementation of MPI at the receiver, and in particular may reserve state. MPI will provide a procedure which invalidates a valid descriptor, allowing the implementation to free reserved state. For example, {\tt mpi\_invalidate\_cd(cd)}. The user is not allowed to invalidate the own context descriptor of a group or the context descriptor of any context created by the calling process. \SeeReferNote{context:coherency} %[Tony] % I do not understand the usefulness or formal need for all this % validation and invalidation of process identifiers. Why, where % did it come from, what does it get us? How can this be related % to anything I have seen before? % %%[Lyndon] %% Acquisition of a descriptor requires an allocator, and consumes %% resource. In such cases it is good practice to provide a %% deallocator which frees resource. %% % [Tony] % I understand the idea here, but not all the details. Can this % be justified/exemplified/simplified? % %%[Lyndon] %% Unless I am missing the additional point in this comment, I can't %% see the problem. Probably poor presentation of Proposal I++ :-) %% \subsubsection*{Descriptor Attributes} MPI provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, {\tt gd = mpi\_context\_gd(cd)}. \subsection{Point-to-Point Communication} \LabelSection{point2point} This proposal recommends three forms for MPI point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a Send operation of form X cannot match with a receive operation of form Y, which requires that the form is embedded into the message envelope. The three forms are described followed by considerations of uniform integration of these forms in the point-to-point communication section of MPI. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. \SeeReferNote{null:context} Message selection and addressing are expressed by {\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. Send supplies the {\tt pd} of the receiver. Receive supplies the {\tt pd} of the sender. Receive can wildcard on {\tt pd} by supplying the value of a named constant process descirptor, e.g. {\tt MPI\_PD\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. \SeeReferNote{closed:context} Message selection and addressing are expressed by {\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. Send supplies the {\tt cd} of the receiver and sender, and the {\tt rank} of the receiver. Receive supplies the {\tt cd} of the sender and receiver, and the rank of the sender. The {\tt (cd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt cd}. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. \SeeReferNote{open:context} Message selection and addressing are expressed by {\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. Send supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. Receive supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The {\tt (rcd, rank)} pair in Send (Receive) is sufficient to determine the process descriptor of the receiver (sender). Receive cannot wildcard on {\tt lcd}. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD}, in which case Receive {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. Receive can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. This proposal makes no statment about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point section of MPI by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with the form. This is at the expense of multiplying the number of Send and Receive procedures by a factor of three, and some difficulty in details of the current point-to-point working document which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options each have advantages and disadvantages. \paragraph*{Option i:} The framework of the open context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii:} The framework of the closed context form is adopted and extended. We introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. The null context form is expressed as {\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as {\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the {\tt (lcd, rcd)} pair at the expense of introducing some trickery. We can define a mechanism which ``globs'' together these two fields onto in an identifier which Send and Receive can distinguish from a context identifier and treat as the shorthand notation. Then we should also define a mechanism by which a receiver which has completed a Receive with wildcard on {\tt rcd} is able to determine the valid context descriptor of the sender. This is a inconvenient. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % %% [Lyndon] %% I dislike this also. Of course it is all to get around extending %% the syntax, it stinks of that. %% \paragraph*{Option iii:} The framework of the null context form is adopted and extended. The null context form is expressed as {\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is clumsy. %[Tony] % I dislike this intensely. There should be a group-pair data % structure. Group is never a pair of sub-groups. It is a % bad idea. This is all to get around changing syntax, no? % %% [Lyndon] %% I dislike this also. Of course it is all to get around extending %% the syntax, it stinks of that. %% \subsection{Collective Communication} \LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context and frame in which they are to operate. MPI does plan to describe symmetric collective communication operations. It is not possible to determine whether the proposal is sufficient to allow implementation of the collective communication section of MPI in terms of the point-to-point section of MPI without loss of generality, since the collective operations are not yet defined. %[Tony] Check, but it is desirable that they be so writable, so we will % have to watch. % Assymetric collective communication operations, especially those in which the sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a ``glob'' form) which identify the contexts and frames in which they are to operate. MPI does not plan to describe assymetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model and comprise communicative funtionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of MPI. % % END "Proposal" %----------------------------------------------------------------------% % [Tony] % So, I gather that a set of groups is passable to a collcomm, % and a pair is passable to a pt2pt. That is neat, but it should % still be a separate data structure, with separate calls than % the intra-group version (at least for the pt2pt calls). % %% [Lyndon] %% Currently, I am only thinking of passing either singlets or duplets %% to point-to-point and collective. %% I was trying to avoid separate - extra - calls to point-to-point %% because that is already very large and there is a swell of concern %% about the size thereof. %% %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion} \LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents arguments for and against the MIMD model annex, and to some extent explores the consquences of these arguments for MPI in a wider sense. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model, in which process execute in a stricly SIMD fashion. The material describing processes in \ReferSection{processes} is simplified: \begin{itemize} \item processes have identical instruction blocks and different data blocks; \item process descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query and group descriptor attribute query is capable of providing access to all process descriptors); \item dynamic process models are not considered. \end{itemize} The material describing process groupings in \ReferSection{groupings} is simplified: \begin{itemize} \item group descriptor transmission and registry become redundant (Note, however, that they are anyway redundant as context descriptor transmission and registry coupled with context descriptor attribute query capable of providing access to all group descriptors.); \item the own process grouping explicitly becomes a group containing all processes. \end{itemize} The material describing communication contexts in \ReferSection{contexts} is simplified: \begin{itemize} \item context descriptor transmission and registry become unnecessary. \end{itemize} The material describing point-to-point communication in \ReferSection{point2point} is simplified: \begin{itemize} \item the open context form becomes redundant; \item uniform integration ``Option i'' is deleted, and ``Option ii'' loses ``globs'' thus becoming simple enough that ``Option iii'' need not be further considered. \end{itemize} The material describing collective communication in \ReferSection{collective} is simplified: \begin{itemize} \item there is no possibility of collective communication operations spanning more than context. \end{itemize} \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as MPI are not suited to fine grained function driven or data driven programming. The annex is intended to provide expresive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in EXPRESS and PARMACS, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or vice versa!). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computating. The debate is whether or not message passing approaches such as MPI (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD parallel prgrams. The positive argument is that communications between SPMD modules requires high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development and standardisation procedures for distributed operating systems. The latter argument is comparable with the argument for MPI versus parallel compilation. \subsection{Notes} \LabelSection{notes} \begin{enumerate} \item\LabelNote{integer:descriptors} Descriptors are a plentiful but bounded resource. They are expressed as integers for two reasons. Firstly, this expression facilitates uniform binding to ANSI~C and Fortran~77. Secondly, there is a later convenience in ensuring that process descriptors in particular are expressed in integers, and I made all the descriptors are integers for consistency. In practice the descriptor could be an index into a table of objects description structures or pointers to such structures. \item\LabelNote{descriptor:cache} It is possible to allow the user to save user defined attributes in descriptors, as Littlefield has suggested. Such attributes should not be communicated in either descriptor transmission or the descriptor registry. \item\LabelNote{process:identifiers} The process descriptor is not a global unique process identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional process information in implementations. For example, the process identifier of a receiver may not be sufficient to route a message to the receiver, and this information can be referenced from the process descriptor. \item\LabelNote{group:identifiers} The group descriptor is not a global unique group identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependeent, and enables more efficient access to additional group information in implementations. For example, the size of the group and the map from member ranks to process descriptors can be referenced from the group descriptor. \item\LabelNote{context:identifiers} The context descriptor is not a global unique context identifier but must reference such an identifier. The use of descriptors hides such identifiers in order that they may be implementation dependent, and enables more efficient access to additional context information in implementations. For example, the group descriptor of the frame can be referenced from the context descriptor. \item\LabelNote{dynamic:processes} The proposal does not prevent a process model which allows dynamic creation and termination of processes however it does not favour an asynchronous process creation model in which singleton processes are created and terminated in an unstructured fashion. The proposal does favour a model in which blobs of processes are created (and terminated) in a concerted fashion, and in which each group so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying worloads. The author suggests that concerted blob expand and contract operations are most suitable for this purpose than asynchronous spawn/kill operations. \item\LabelNote{registry:check} There should probably also be a registry operation which checks whether a name has been registered without the possibility of blocking the calling process indefinitely. The same registry could be used for process, group and context descriptors. \item\LabelNote{process:coherency} In the static process model it is assumed that processes are created in concert, and that termination of one process implies termination of all processes. In this case there is no coherency problem associated with processes and process descriptors. In a dynamic process model there is a coherency problem which processes and process descriptors since a process can retain the descriptor of a process which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{dynamic:groups} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The author suggests concerted group expand and contract operations are more generally useful than asynchronous join/leave operations. \item\LabelNote{group:blobs} MPI has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is intended to be a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and different blobs have different own process groupings. An SPMD program is a single blob. A host-node program composes two blobs, the node blob and the host blob (which is a singleton). There is a sense in which a blob has a locally SPMD view. \item\LabelNote{own:group} This procedure looks like a process descriptor attribute query. \item\LabelNote{group:coherency} Dynamic processes introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Dynamic processes potentially introduce a group coherency problem as a group descriptor can contain the process descriptor of a process which has been deleted. Group transmission introduces a group and group descriptor coherenncy problem since a process can retain a group descriptor of a group which has been identified group. The proposal expects the user to ensure coherent usage. \item\LabelNote{pd:to:rank} The proposal did not include a function to convert a {\tt (gd, pd)} pair into a rank. It is suggested that this inverse map is allowed to be arbitrarily slow. \item\LabelNote{dynamic:contexts} Marc Snir has described a method by which global unique group identifiers can be generated without use of shared global data. The description of context creation anticipates a similar method for global unique context identifier generation. However the synchronisation requirement of this method makes it unnecessary for context creation. Synchronisation at creation of context within a process grouping frame implies that all processes within the frame perform the same context creations in the same sequence. Therefore the combination of global unique frame identifier and context creation sequence number provides a global unique context identifier without communication. \item\LabelNote{own:context} This procedure looks like a group descriptor attribute query. \item\LabelNote{context:coherency} The retention of a reference to a group descriptor by a context introduces the potential for a context and context descriptor coherency problem, as a context could contain a reference to the group descriptor of a group which has been deleted. This could be circumvented by demanding that all such contexts are deleted before a group is deleted. Context descriptor transmission introduces a coherency problem since a process can retain the descriptor of a context which has been deleted. The proposal expects the user to ensure coherent usage. \item\LabelNote{null:context} This is somewhat like and PARMACS and PVM~3. It is general purpose, but not particularly expressive. It does not provide facilities for writers of parallel libraries. \item\LabelNote{open:context} This is somewhat like ZIPCODE and VENUS. It is expressive in certain SPMD programs where noncommunicative data driven parallel computations can be performed concurrently. It provides facilities for writers of SPMD like parallel libraries. \item\LabelNote{closed:context} This is somewhat like CHIMP and PVM~2. It is expressive in certain MIMD programs where communicative data driven parallel computations can be performed concurrently. It provides facilities for MSPMD like parallel libraries. \end{enumerate} % % END "Discussion & Notes" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Conclusion" % \section{Conclusion} This chapter has presented and discussed a proposal for communication contexts within MPI. In the proposal process groupings appeared as frames for the creation of communication contexts, and communication contexts retained certain properties of the frames used in their creation. The author strongly recommends this proposal to the committee. % % END "Conclusion" %----------------------------------------------------------------------% % % END "Proposal VI" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Mar 22 11:39:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25825; Mon, 22 Mar 93 11:39:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14730; Mon, 22 Mar 93 11:38:27 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 11:38:25 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14722; Mon, 22 Mar 93 11:38:23 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 22 Mar 93 08:34 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06464; Mon, 22 Mar 93 08:32:36 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA16539; Mon, 22 Mar 93 08:32:32 PST Date: Mon, 22 Mar 93 08:32:32 PST From: rj_littlefield@pnlg.pnl.gov Subject: RE: Rik's comments on Lyndon's Proposal I To: jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, rj_littlefield@pnlg.pnl.gov, tony@aurora.cs.msstate.edu Cc: d39135@carbon.pnl.gov Message-Id: <9303221632.AA16539@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Lyndon says > There is no proposal II. There is however a proposal VI. This > naming/numbering is Tony's suggestion with which I concur. Is this the same or different from the "proposal VI" I sketched last night? For reference, that was (Rik said) > Here's a sketch of a new proposal (VI ?). > > . The functionality of point-to-point communications is per Snir, > Lusk, and Gropp, augmented by my proposed MPI_FORM_CONTEXT to allow > assembling arbitrary collections of processes. (Marc has already > accepted this in a private email to me -- don't know why he > didn't post it.) > > . Performance expectations of point-to-point communications are > explicitly stated as follows: > > - MPI_COPY_CONTEXT does not synchronize the participating processes > and costs significantly less than a point-to-point fanout among > them (e.g., it uses a communication-free counting strategy); > > - all other context formation routines cost no more than if they > were implemented using a single fanin/fanout among the > participating processes; > > - translation of (context,rank) to absolute processor ID costs no > more than if it were implemented via the lookup table that Snir > suggests. > > . Groups and contexts are not equal. A group consists of a base > context (from which other contexts can be created quickly by > MPI_COPY_CONTEXT), plus topology information, plus my cacheing > facility. Lyndon says > I really am very sorry to have messed you about. I have no idea what this refers to. It would bother me to have MPI produce a bad standard. Not much else does. I would be delighted to have all of my specific suggestions and/or words replaced by something better. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Mar 22 12:30:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27511; Mon, 22 Mar 93 12:30:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17027; Mon, 22 Mar 93 12:29:16 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 12:29:14 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17019; Mon, 22 Mar 93 12:29:12 -0500 Received: from newton.sandia.gov (newton.cs.sandia.gov) by cs.sandia.gov (4.1/SMI-4.1) id AA17003; Mon, 22 Mar 93 10:29:10 MST Received: by newton.sandia.gov (5.57/Ultrix3.0-C) id AA00503; Mon, 22 Mar 93 10:30:02 -0700 Date: Mon, 22 Mar 93 10:30:02 -0700 From: mpsears@newton.cs.sandia.gov (Mark P. Sears) Message-Id: <9303221730.AA00503@newton.sandia.gov> To: mpi-context@cs.utk.edu Subject: A new proposal for context and groups. Proposal VI Mark P. Sears Sandia National Laboratories The following proposal for context and group definitions is intended to fill a gap in Tony Skjellum's list. It is most closely related to Rik Littlefield's proposal V. The main difference is that in my proposal, context and groups are completely orthogonal, both in purpose and function. There are a number of advantages to this proposal: 1) The meanings of context and group are simple and it is easy to see how they can be implemented quickly and efficiently. The proposal is also free of chicken vs. egg problems. 2) MPI 1.0 can be closed at the point-to-point level without any reference to groups or group operations. We might actually finish on time. 3) Other proposals which combine aspects of context and group as defined here could be layered on top of this proposal, but not vice versa. Indeed, some of these proposals are very different from each other and it is not clear which is the best. 4) Elsewhere in MPI we have agreed to define low level layers on which higher level stuff can be built. This proposal follows that line of reasoning. 5) This proposal requires no server code and most of the group code is not even parallel. I did a test implementation of groups as defined here and was able to build code for identity groups, permutation groups, linear and bilinear groups, general set groups (Tony's favorite), composition groups, Cartesian products and cartesian subsets (my favorite), all in about 700 lines of code. The group definition really lends itself to object-oriented user extensibility, like X widgets but easier. 6) Groups are easily constructed and destroyed, since no global communication is required. Dynamic groups are not excluded, although they must be used carefully. Since groups have no associated context, there are no resources limiting their construction other than memory and CPU time. To summarize, in this proposal: 1) The purpose of context is to allow different software modules to maintain independence by allowing independence of tag spaces. A context is an integer-valued extension to the tag field which must match exactly for both sender and receiver. Allocation and deallocation of contexts must occur globally and synchronously. Two predefined contexts exist, one for internal use by MPI and one a free-for-all context for application programs. 2) Processes are addressed by their rank within the parallel task. This global rank is fixed and assigned in an implementation-defined way. There must exist environment functions which obtain the topology of the global process assignment (hypercube, mesh, random network, ...) [ This is an area where MPI 2.0 could really break some new ground: define interaction between different parallel tasks, define creation and deletion of parallel tasks, ...] 3) The purpose of groups is to allow the management of subsets of processes in a parallel task. A group is defined simply as an ordered set of unique integers (the elements). Two groups are functionally the same if they define the same ordered set. Groups have no associated context, default or otherwise. They are also free of state. Groups are defined locally by construction, and have no global meaning. A group can be passed from one process to another by passing the information needed to construct it. A process is a member of a group if its global rank is an element of the group. 4) Groups have an implicit topology, defined by the ordering of the elements. Any other ordering can be defined by constructing a new group with the same elements in the new order. There is no need for any other topology function. 5) Any group implementation must define at least the following functions: order(group) -- Number of elements in the group. range(group) -- Upper limit on elements. Lower limit is always zero. iselement(group, element) -- Predicate to determine if element is in a group. rank(group, element) -- Compute rank of element. element(group, rank) -- Compute element given rank. Some other proposals have scamped on the need for being able to compute the rank of an element or the predicate defining whether something is even an element of the group. Using these functions, it is easy to construct a minimum spanning tree or set of binary exchanges for all of the interesting collective communication functions, for any group. This information could be cached with the local group definition. There is no reason to disallow user-defined groups (e.g. dynamic groups). Note that since groups are ordered, the previous and next elements are easily computed using the rank and element functions. 6) MPI collective communication functions which operate on subsets of processors use groups to define the subset of interest, and must be called with the same context, tags(s) and group, by at least all processes which are members of the group. Note that with this definition of group it is the responsibilty of the caller to provide context and tags. Two (or more) such communication functions are guaranteed to complete if they are called in the same order on all processes which are members of both groups (this condition is less stringent for multithreaded environments). Otherwise the program is erroneous. From owner-mpi-context@CS.UTK.EDU Mon Mar 22 19:01:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04642; Mon, 22 Mar 93 19:01:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03825; Mon, 22 Mar 93 19:00:39 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 22 Mar 1993 19:00:38 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03811; Mon, 22 Mar 93 19:00:35 -0500 Date: Tue, 23 Mar 93 00:00:31 GMT Message-Id: <15033.9303230000@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: tum-tee-tum To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear Colleagues A colleage of mine (not of MPI) reviewed Proposal VI and made a number of very sensible suggestions regarding technical details and presentational details, although not technical guts. The outcome is that I must now revise this proposal. I will repost to this group if there is any interest --- can not happen for approx. 18 hours. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 23 10:42:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19318; Tue, 23 Mar 93 10:42:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14088; Tue, 23 Mar 93 10:41:12 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 10:41:12 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14077; Tue, 23 Mar 93 10:41:02 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA01482 (5.65c/IDA-1.4.4 for ); Tue, 23 Mar 1993 10:39:47 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA11198; Tue, 23 Mar 93 15:39:42 GMT Date: Tue, 23 Mar 93 15:39:41 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303231539.AA11198@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05333; Tue, 23 Mar 93 15:36:15 GMT To: mpi-context@cs.utk.edu Cc: snir@watson.ibm.com, rj_littlefield@pnlg.pnl.gov, lyndon@epcc.ed.ac.uk Subject: Context proposals overview Content-Length: 5160 Jim's initial summary/comments on the Contexts proposals Gentlemen, here is my summary and initial comclusions on the three extant Contexts proposals received via Lyndon. Since I have probably misunderstood or misrepresented some of the proposals, please correct me as appropriate. Proposal I (Marc Snir) ---------------------- Key points : A group and a context are identical. All communication (including point to point) is addressed by group and rank. The receiver MUST be a member of the same group as the sender, and explicitly give the group as an argument to the receive function. Group descriptors are not passable to other processors. Creation of a new context involves creation of a new group which is a group synchronising operation. For: Simplicity, it's easy to explain and understand (and probably implement). Insisting on a group for all addressing gives the 0..n-1 rank model for subgroups which aids in importing libraries. Against: The receiver of a message has to both know the group of the sender, and be a part of it. This feels like it makes servers hard, but may be ok. If you keep having to go into the ALL group, then the security which was the whole point of a context is lost. The argument that contexts can be checked at the sender is incorrect because a single process can be in many different groups/contexts, and therefore the group/context must be transferred to ensure correct matching. (Maybe this wasn't the intent of the comments anyway...) I don't understand how to build a group/context using only point to point messages. I still seem to have the bootstrap problem that I need the new context to safely receive the message which will tell me what the new context is. Proposal V (Rik Littlefield) ---------------------------- Key points : A desire to have caching of arbitrary library (or even user) defined entities on the groups. Point to point communication is expected to be addressed by TID and Context. TIDs have global scope, they can be passed around arbitrarily. There is explicit translation from (group,rank) to TID. Group descriptors are local objects, but can be passed around through special mechanisms. Contexts are globally unique, (and expected to be allocated locally through one of the normal bit space partitioned schemes. i.e. processor number // identifier. There appears to be an expectation that the identifiers are managed closely), and can be passed transparently in messages (? is this true ?). Contexts are created inside groups, creating a context is a barrier synchronisation (? is this true ?). For: The caching is good idea, but orthogonal to all the other issues, it could be added just as easily to any of the other proposals The use of TIDs in the point to point removes any issues of group membership from these functions. There is thus a way to achieve any inter group communication which is required (even if it is unpleasant). Against: The complexity of the "paired exact match criterion" is non trivial to explain. Proposal VI (Lyndon Clarke) --------------------------- Key points: ALL descriptors (process, group, context) are process local, but may be sent elsewhere through special mechanisms. Contexts are created inside groups, and are bound to them. The context can be used to refer (indirectly) to the owning group. Creating a context is a barrier synchronisation of the group. Three different forms of process address are discussed for point to point messaging. (Ouch !) For: Can be made to address complex inter group communications Against: The complexity of the whole proposal. The passing of opaque entities with translation is rather unpleasant for people with homogeneous machines. Before this we could ignore all of the type info, now we have to check some of it and take special actions. Jim's initial conclusions... ---------------------------- 1) Marc's approach of insisting that all communications occur inside a group has some simplifying effects. In particular it easily gives the base shift for sub-groups, so that they "think" their processors are numbered 0..n-1. It has the disadvantage that there is an extra translation required on ALL communications. This will add to the latency at the most fundamental level (though not much if we expect to consume space linear in the group size). 2) Creating a new group for each context may be expensive, though Marc suggests an implementation which makes this relatively cheap. Adding Rik's caching ideas here could also defer the cost until it was required. (e.g. there is no need to build a broadcast tree in a group until the first broadcast is performed. [Fundamental optimisation 0: "Don't do it until you need to, you might never have to do it at all"]) At the moment I favour Marc's with added caching. Can anyone explain where the big gain is from the added complexity of the others ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Mar 23 12:22:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23133; Tue, 23 Mar 93 12:22:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19811; Tue, 23 Mar 93 12:21:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 12:21:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19803; Tue, 23 Mar 93 12:21:49 -0500 Date: Tue, 23 Mar 93 17:21:18 GMT Message-Id: <15889.9303231721@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Context proposals overview To: jim@meiko.co.uk (James Cownie), mpi-context@cs.utk.edu In-Reply-To: James Cownie's message of Tue, 23 Mar 93 15:39:41 GMT Reply-To: lyndon@epcc.ed.ac.uk Cc: snir@watson.ibm.com, rj_littlefield@pnlg.pnl.gov, lyndon@epcc.ed.ac.uk Jim writes: > Gentlemen, here is my summary and initial comclusions on the three > extant Contexts proposals received via Lyndon. Since I have probably > misunderstood or misrepresented some of the proposals, please correct > me as appropriate. > > Proposal I (Marc Snir) > ---------------------- > Creation of a new context involves creation of a new group which is a > group synchronising operation. More accurately it involves duplication of an existing group. One problem which I have with this is that if we anticipate future extension to groups which are dynamic in membership (they expand and contract) then the duplication concept breaks because the identical nature of the grouping is lost. This is why I much prefer to have contexts as references to groups and groups as frames of contexts - the contexts shrink and stretch according to the expansion and contraction of the underlying group. > For: > Simplicity, it's easy to explain and understand (and probably > implement). Indeed but please see further comments attatched to Proposal VI below. > Insisting on a group for all addressing gives the 0..n-1 rank model > for subgroups which aids in importing libraries. Indeed and Proposal VI gives all (and more) of this aid. > Against: > The receiver of a message has to both know the group of the sender, > and be a part of it. This feels like it makes servers hard, but may be > ok. If you keep having to go into the ALL group, then the security > which was the whole point of a context is lost. The restriction in Proposal I really does make programming of communicative functionally distinct groups hard. I am thinking well beyond singleton service processes and I have explained some of the things I am thinking previous messages. > The argument that contexts can be checked at the sender is incorrect > because a single process can be in many different groups/contexts, and > therefore the group/context must be transferred to ensure correct > matching. (Maybe this wasn't the intent of the comments anyway...) Our interpretations concur. > Proposal V (Rik Littlefield) > ---------------------------- > Key points : > A desire to have caching of arbitrary library (or even user) defined > entities on the groups. > > Point to point communication is expected to be addressed by TID and > Context. TIDs have global scope, they can be passed around > arbitrarily. > > There is explicit translation from (group,rank) to TID. ... which the poor user *must* do. > Group descriptors are local objects, but can be passed around through > special mechanisms. Check. If the feature of Proposal VI is a disdavantage of that proposal then it is also a disadvantage of Proposal V! Just being consist :-) > Contexts are globally unique, (and expected to be allocated locally > through one of the normal bit space partitioned schemes. i.e. > processor number // identifier. There appears to be an expectation > that the identifiers are managed closely), and can be passed > transparently in messages (? is this true ?). Our interpretations concur. > Contexts are created inside groups, creating a context is a barrier > synchronisation (? is this true ?). Our interpretations concur. > For: > The caching is good idea, but orthogonal to all the other issues, it > could be added just as easily to any of the other proposals Precisely and this is recognised in the revision of Proposal VI which will be circluated within the hour. > The use of TIDs in the point to point removes any issues of group > membership from these functions. There is thus a way to achieve any > inter group communication which is required (even if it is unpleasant). It is unpleasant indeed. I appreciate that I am stating a value jugement here which may be interpreted as "my personal opinion". In fact at Edinburgh we have done plenty of intergroup communication programming and the opinion is shared. > Against: > The complexity of the "paired exact match criterion" is non trivial to > explain. I concur. > Proposal VI (Lyndon Clarke) > --------------------------- > > Key points: > ALL descriptors (process, group, context) are process local, but may > be sent elsewhere through special mechanisms. The revision, particularly in the improved discussion notes, explores the possibility that processes are expressed as identifiers but groups and contexts are process local opaque references to objects of undefined size and structure (implemenation dependent). > Contexts are created inside groups, and are bound to them. The context > can be used to refer (indirectly) to the owning group. Exactly. Please recall the point I made above about group (context) duplication in Proposal I. > Creating a context is a barrier synchronisation of the group. The proposal says this, the discusion explains that it may not be necessary to actually synchronise. My mind is still working on this one. > Three different forms of process address are discussed for point to > point messaging. (Ouch !) From the point of view of the MPI provider this is a just small amount of additional work. The three forms link to a generic send/recv so there is not at all three times as much implementation. The size of a test suite increases of course. Form the point of view of the MPI user there could be an issue here, however the facilities can be described in a stepwise manner such that users do not need to be explored to more complexity then they need to use. The revision tries to clean up the business of presenting these forms in a uniform syntactic framework which is one way to address the "point-to-point contains too many procedures" concern. > For: > Can be made to address complex inter group communications Does address such communications directly and with expressive power. I'd like to add "For: VI is functional superset of Proposal I; group/context user cache of V is easy to add to VI." > Against: > The complexity of the whole proposal. Proposal VI is larger than Proposal I which is at a similarly mature stage of development, because it is more powerful, as I say it is functionally a superset of Proposal I (and this is drawn out better in the discussion in the revision). The facilities can be presented to users in a friendly fashion which does not require substantially different entry level user sophistication than Proposal I. The more "advanced" facilities can be introduced in a user manual after the more "basic" facilities. I conjecture that there is no real problem here. > The passing of opaque entities with translation is rather unpleasant > for people with homogeneous machines. Before this we could ignore all > of the type info, now we have to check some of it and take special > actions. > I mentioned above that if this is a disadvantage of VI then it must also be a disadvantage of V. This was one suggestion as to how we might give the user an interface to transmission of these opaque objects. I'm sure we can think of other approaches which avoid the implementor having to check the data type description in homogeneous kit. For example we could provide a set of procedures which do the send and receive, although this does again add more procedures to point-to-point which I am trying to avoid, honest guv' :-) I do not believe that this is any big deal for the user, since the kind of users who are currently doing this kind of programming (and they are certainly not high priests) are understand and think about data translation issues anyway. If these are the only real objections you have to passing around descriptors then I am confident we can do something about that. Would you like to make a suggestion as to how you would prefer to see transmission done? > Jim's initial conclusions... > ---------------------------- > > 1) Marc's approach of insisting that all communications occur inside a > group has some simplifying effects. In particular it easily gives the > base shift for sub-groups, so that they "think" their processors are > numbered 0..n-1. It has the disadvantage that there is an extra > translation required on ALL communications. This will add to the > latency at the most fundamental level (though not much if we expect to > consume space linear in the group size). Proposal VI has all of the facilities, which has been implemented and used in systems for a couple years now, and a whole lot more. For speed demons Proposal VI lets them use process descriptots (or identifiers) directly. > At the moment I favour Marc's with added caching. Can anyone explain > where the big gain is from the added complexity of the others ? I hope I did. If you need more explanation then please do ask. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 23 13:28:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24476; Tue, 23 Mar 93 13:28:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22791; Tue, 23 Mar 93 13:27:53 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 13:27:52 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22783; Tue, 23 Mar 93 13:27:50 -0500 Message-Id: <9303231827.AA22783@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6757; Tue, 23 Mar 93 13:27:47 EST Date: Tue, 23 Mar 93 13:10:01 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-context@cs.utk.edu Reply-To: SNIR@watson.ibm.com Subject: Context proposals overview Reference: Attached note from jim@meiko.co.uk Reply *************** Forwarded Note *************** Received: from marge.meiko.com by watson.ibm.com (IBM VM SMTP V2R3) with TCP; Tue, 23 Mar 93 10:39:58 EST Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA01482 (5.65c/IDA-1.4.4 for ); Tue, 23 Mar 1993 10:39:47 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA11198; Tue, 23 Mar 93 15:39:42 GMT Date: Tue, 23 Mar 93 15:39:41 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303231539.AA11198@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05333; Tue, 23 Mar 93 15:36:15 GMT To: mpi-context@cs.utk.edu Cc: snir@watson.ibm.com, rj_littlefield@pnlg.pnl.gov, lyndon@epcc.ed.ac.uk Subject: Context proposals overview Content-Length: 5160 Jim's initial summary/comments on the Contexts proposals Gentlemen, here is my summary and initial comclusions on the three extant Contexts proposals received via Lyndon. Since I have probably misunderstood or misrepresented some of the proposals, please correct me as appropriate. Proposal I (Marc Snir) ---------------------- Key points : A group and a context are identical. All communication (including point to point) is addressed by group and rank. The receiver MUST be a member of the same group as the sender, and explicitly give the group as an argument to the receive function. Group descriptors are not passable to other processors. Creation of a new context involves creation of a new group which is a group synchronising operation. For: Simplicity, it's easy to explain and understand (and probably implement). Insisting on a group for all addressing gives the 0..n-1 rank model for subgroups which aids in importing libraries. Against: The receiver of a message has to both know the group of the sender, and be a part of it. This feels like it makes servers hard, but may be ok. If you keep having to go into the ALL group, then the security which was the whole point of a context is lost. >>> One can, of course, have several distinct contexts that include >>> all processes. The argument that contexts can be checked at the sender is incorrect because a single process can be in many different groups/contexts, and therefore the group/context must be transferred to ensure correct matching. (Maybe this wasn't the intent of the comments anyway...) >>> group/context need be transfered in the message. The point I was >>> trying to make is that one can make sure that a message with a >>> given context tag will be sent only to a process that "knows" about >>> this context. This allows some relaxations: e.g., context names >>> need not be system-wide unique, only locally unique at each process; >>> handling of "illegal context" errors is simplified. I don't understand how to build a group/context using only point to point messages. I still seem to have the bootstrap problem that I need the new context to safely receive the message which will tell me what the new context is. >>> Well, we have an existential proof, since we support dynamic group >>> creation in our system. A new context is created within an old, >>> preexisting context; so ALL need to be there from start. Proposal V (Rik Littlefield) ---------------------------- Key points : A desire to have caching of arbitrary library (or even user) defined entities on the groups. Point to point communication is expected to be addressed by TID and Context. TIDs have global scope, they can be passed around arbitrarily. There is explicit translation from (group,rank) to TID. Group descriptors are local objects, but can be passed around through special mechanisms. Contexts are globally unique, (and expected to be allocated locally through one of the normal bit space partitioned schemes. i.e. processor number // identifier. There appears to be an expectation that the identifiers are managed closely), and can be passed transparently in messages (? is this true ?). Contexts are created inside groups, creating a context is a barrier synchronisation (? is this true ?). For: The caching is good idea, but orthogonal to all the other issues, it could be added just as easily to any of the other proposals The use of TIDs in the point to point removes any issues of group membership from these functions. There is thus a way to achieve any inter group communication which is required (even if it is unpleasant). Against: The complexity of the "paired exact match criterion" is non trivial to explain. Proposal VI (Lyndon Clarke) --------------------------- Key points: ALL descriptors (process, group, context) are process local, but may be sent elsewhere through special mechanisms. Contexts are created inside groups, and are bound to them. The context can be used to refer (indirectly) to the owning group. Creating a context is a barrier synchronisation of the group. Three different forms of process address are discussed for point to point messaging. (Ouch !) For: Can be made to address complex inter group communications Against: The complexity of the whole proposal. The passing of opaque entities with translation is rather unpleasant for people with homogeneous machines. Before this we could ignore all of the type info, now we have to check some of it and take special actions. Jim's initial conclusions... ---------------------------- 1) Marc's approach of insisting that all communications occur inside a group has some simplifying effects. In particular it easily gives the base shift for sub-groups, so that they "think" their processors are numbered 0..n-1. It has the disadvantage that there is an extra translation required on ALL communications. This will add to the latency at the most fundamental level (though not much if we expect to consume space linear in the group size). 2) Creating a new group for each context may be expensive, though Marc suggests an implementation which makes this relatively cheap. Adding Rik's caching ideas here could also defer the cost until it was required. (e.g. there is no need to build a broadcast tree in a group until the first broadcast is performed. [Fundamental optimisation 0: "Don't do it until you need to, you might never have to do it at all"]) At the moment I favour Marc's with added caching. Can anyone explain where the big gain is from the added complexity of the others ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Tue Mar 23 13:44:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24968; Tue, 23 Mar 93 13:44:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23491; Tue, 23 Mar 93 13:43:28 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 13:43:27 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23478; Tue, 23 Mar 93 13:43:11 -0500 Date: Tue, 23 Mar 93 18:43:04 GMT Message-Id: <15951.9303231843@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal VI, First Revision, LaTeX To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Here is LaTeX for the revision of Proposal VI. Please see my reply to Jim of today for an idea of what is changed. I apologise that this is one hour later then I had previously announced. PostScript follows. ---------------------------------------------------------------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % LYNDON % % REMBER TO MERGE IN THE COMMENTS FROM THE PREVIOUS REVISION OF THIS % % DOCUMENT. OH, AND TO GET BETTER SET UP FOR TRANSFER TO LAPTOP! % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentstyle{report} \begin{document} \title{MPI Context Subcommittee\\Proposal VI\\First Revision} \author{Lyndon~J~Clarke} \date{March 23, 1993} \maketitle %======================================================================% % BEGIN "Proposal VI" % Lyndon J. Clarke % March 1993 % \chapter{Proposal VI} \newcommand{\LabelNote}[1]{\label{vi:note:#1}} \newcommand{\ReferNote}[1]{Note~\ref{vi:note:#1}} \newcommand{\LabelSection}[1]{\label{vi:sect:#1}} \newcommand{\ReferSection}[1]{Section~\ref{vi:sect:#1}} %----------------------------------------------------------------------% % BEGIN "Introduction" % \section{Introduction}\LabelSection{introduction} This chapter proposes that communication contexts and process groupings within {\sc mpi} appear as loosely coupled concepts. One or more communication contexts may be associated with each process grouping, and each communication context inherits properties of the associated process grouping. This reflects the observations that invocations of modules in a parallel program typically operate within process groupings, and that there may be multiple modules operating within each process grouping. The proposal provides process identified communication, communications which are limited in scope to single contexts, and communications which have scope spanning pairs of contexts. The proposal makes no statements regarding message tags. It is assumed that these will be a bit string expressed as an integer in the host language. Much of this proposal must be viewed as recommendations to other subcommittees of {\sc mpi}, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language, only for purposes of discussion. The detailed proposal is presented in \ReferSection{proposal}, which refers the reader to a set of discussion notes in \ReferSection{notes}. The note assumes forward knowledge of the proposal text and are therefore best examined after familiarisation with the proposal text. Aspects of the proposal are discussed in section \ReferSection{discussion}, and it is also recommended that this material be read after familiarisation with the proposal text. % % END "Introduction" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Detailed Proposal" % \section{Detailed Proposal}\LabelSection{proposal} This section presents the detailed proposal, describing, in order of appearance: processes; process groupings; communication contexts; point-to-point communication; collective communication. \subsection{Processes}\LabelSection{processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and {\sc mpi} does not distinguish such threads. \subsubsection*{Process Descriptor} Each process is described by a {\it process descriptor\/} (or {\it handle}) which is expressed as an integer type in the host language and has an opaque value which is process local. See \ReferNote{descriptor}. The initialisation of {\sc mpi} services will assign to each process an {\it own\/} process descriptor. Each process retains its own process descriptor until the termination of {\sc mpi} services. {\sc mpi} provides a procedure which returns the own descriptor of the calling process. For example, \mbox{\tt pd = mpi\_own\_process()}. \subsubsection*{Process Creation \& Destruction} This proposal makes no statements regarding creation and destruction of processes. See \ReferNote{processes:dynamic}. \subsubsection*{Descriptor Transmission} Since a descriptor is an integer type in the host language the value may be transmitted in a message as an integer. The recipient of the descriptor value can make no defined use of the value in the {\sc mpi} operations described in this proposal --- the descriptor is {\it invalid}. {\sc mpi} provides a mechanism whereby the user can transmit a valid process descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. \mbox{\tt MPI\_PD\_TYPE}. See \ReferNote{descriptor:transmission}. {\sc mpi} provides a process descriptor registry service. Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. See \ReferNote{descriptor:registry}. Note that receipt of a valid process descriptor may have a persistent effect on the implementation of {\sc mpi} at the receiver, and in particular may reserve state. {\sc mpi} will provide a procedure which frees (or invalidates) a valid descriptor, allowing the implementation to free reserved state. For example, \mbox{\tt mpi\_free\_pd(pd)}. The user is not allowed to free the process descriptor of the calling process. See \ReferNote{descriptor:deallocation}. See \ReferNote{coherency}. \subsubsection*{Descriptor Attributes} This proposal makes no statements regarding process descriptor attributes. \subsection{Process Groupings}\LabelSection{groupings} This proposal views a process grouping as an ordered collection of (references to) distinct processes, the membership and ordering of which does not change over the lifetime of the grouping. See \ReferNote{groupings:dynamic}. The canonical representation of a grouping reflects the process ordering and is a one-to-one map from $Z_N$ to descriptors of the $N$ processes composing the grouping. There may be structure associated with a process grouping defined by a process topology. This proposal makes no further statements regarding such structures. \subsubsection*{Group Descriptor} Each group is identified by a {\it group descriptor\/} (or {\it handle\/}) which is expressed as an integer type in the host language and has an opaque value which is process local. See \ReferNote{descriptor}. The initialisation of {\sc mpi} services will assign to each process an {\it own\/} group descriptor for a process grouping of which the process is a member. Each process retains its own group descriptor and membership of the own process grouping until the termination of {\sc mpi} services. See \ReferNote{grouping:blobs}. {\sc mpi} provides a procedure which accepts a valid process descriptor and returns the own group descriptor of the identified process. For example, \mbox{\tt gd = mpi\_own\_group(pd)}. See \ReferNote{grouping:own}. \subsubsection*{Group Creation and Deletion} {\sc mpi} provides facilities which allow users to dynamically create and delete process groupings in addition to the own grouping(s). The procedures described here generate groups which are static in membership. {\sc mpi} provides a procedure which allows users to create one or more groups which are subsets of existing groups. For example, \mbox{\tt gdb = mpi\_group\_partition(gda, key)} creates one or more new groups {\tt gdb} which are distinct subsets of an existing group {\tt gda} according to the supplied values of {\tt key}. This procedure is called by and synchronises all members of {\tt gda}. {\sc mpi} provides a procedure which allows users to create a group by permutation of an existing group. For example, \mbox{\tt gdb = mpi\_group\_permutation(gda, rank)} creates one new group with the same membership as {\tt gda} with a permutation of process ranking, and returns the created group descriptor in {\tt gdb}. This procedure is called by and synchronises all members of {\tt gda}. {\sc mpi} provides a procedure which allows users to create a group by explicit definition of its membership as a list of process descriptors. For example, \mbox{\tt gd = mpi\_group\_definition(listofpd)} creates one new group {\tt gd} with membership and ordering described by the process descriptor list {\tt listofpd}. This procedure is called by and synchronises all processes identified in {\tt listofpd}. {\sc mpi} provides a procedure which allows users to delete user created groups. This procedure accepts the descriptor of a group which was created by the calling process and deletes the identified group. For example, \mbox{\tt mpi\_group\_deletion(gd)} deletes an existing group {\tt gd}. This procedure is called by and synchronises all members of {\tt gd}. {\sc mpi} may provide additional procedures which allow users to construct process groupings with a process grouping topology. \subsubsection*{Descriptor Transmission} Since a descriptor is an integer type in the host language the value may be transmitted in a message as an integer. The recipient of the descriptor can make no defined use of the value in the {\sc mpi} operations described in this proposal --- the descriptor is {\it invalid}. See \ReferNote{descriptor}. {\sc mpi} provides a mechanism whereby the user can transmit a valid group descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. \mbox{\tt MPI\_GD\_TYPE}. See \ReferNote{descriptor:transmission}. {\sc mpi} provides a group descriptor registry service. Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. See \ReferNote{descriptor:registry}. Note that receipt of a valid group descriptor may have a persistent effect on the implementation of {\sc mpi} at the receiver, and in particular may reserve state. {\sc mpi} will provide a procedure which frees (or invalidates) a valid descriptor, allowing the implementation to free reserved state. For example, \mbox{\tt mpi\_free\_gd(gd)}. The user is not allowed to free the own group descriptor of the calling process or the group descriptor of any group created by the calling process. See \ReferNote{descriptor:deallocation}. See \ReferNote{coherency}. \subsubsection*{Descriptor Attributes} {\sc mpi} provides a procedure which accepts a valid group descriptor and returns the rank of the calling process within the identified group. For example, \mbox{\tt rank = mpi\_group\_rank(gd)}. {\sc mpi} provides a procedure which accepts a valid group descriptor and returns the number of members, or {\it size}, of the identified group. For example, \mbox{\tt size = mpi\_group\_size(gd)}. {\sc mpi} provides a procedure which accepts a valid group descriptor and process order number, or {\it rank}, and returns the valid descriptor of the process to which the supplied rank maps within the identified group. For example, \mbox{\tt pd = mpi\_group\_pd(gd, rank)}. See \ReferNote{inverse:map}. {\sc mpi} may provide additional procedures which allow users to determine the process grouping topology attributes. \subsection{Communication Contexts}\LabelSection{contexts} This proposal views a communication context as a uniquely identified reference to exactly one process grouping, which is a field in a message envelope and may therefore be used to distinguish messages. The context inherits the referenced process grouping as a ``frame''. Each process grouping may be used as a frame for multiple contexts. \subsubsection*{Context Descriptor} Each context is identified by a {\it context descriptor\/} (or {\it handle}) which is expressed as an integer type in the host language and has an opaque value which is process local. See \ReferNote{descriptor}. The creation of {\sc mpi} process groupings allocates an {\it own\/} context which inherits the created grouping as a frame and can be thought of as a property of the created grouping. The grouping retains the descriptor of its own context until {\sc mpi} process grouping deletion. {\sc mpi} provides a procedure which accepts a valid group descriptor and returns the context descriptor of the own context of the identified group. For example, \mbox{\tt cd = mpi\_own\_context(gd)}. See \ReferNote{context:own}. \subsubsection*{Context Creation and Deletion} {\sc mpi} provides facilities which allows user to dynamically create and delete contexts in addition to the own contexts associated with process groupings. See \ReferNote{grouping:deletion}. {\sc mpi} provides a procedure which allows users to create contexts. This procedure accepts a valid descriptor of a group of which the calling process is a member, and returns a context descriptor which references the identified group. For example, \mbox{\tt cd = mpi\_context\_creation(gd)}. This procedure is called by and synchronises all members of {\tt gd}. {\sc mpi} provides a procedure which allows users to delete user created contexts. This procedure accepts a valid context descriptor which was created by the calling process and deletes the identified context. For example, \mbox{\tt mpi\_context\_deletion(cd)}. This procedure is called by and synchronises all members of the frame of {\tt cd}. See \ReferNote{context:creation:deletion} \subsubsection*{Descriptor Transmission} Since a descriptor is an integer type in the host language the value may be transmitted in a message as an integer. The recipient of the descriptor can make no defined use of the value in the {\sc mpi} operations described in this proposal --- the descriptor is {\it invalid}. See \ReferNote{descriptor}. MPI provides a mechanism whereby the user can transmit a valid context descriptor in a message such that the received descriptor is valid. This is integrated with the capability to transmit typed messages. It is suggested that a notional data type should be introduced for this purpose, e.g. \mbox{\tt MPI\_CD\_TYPE}. See \ReferNote{descriptor:transmission}. {\sc mpi} provides a context descriptor registry service. Use of this service is not mandated. Programs which can conveniently be expressed without using the service can ignore it without penalty. See \ReferNote{descriptor:registry}. Note that receipt of a valid context descriptor may have a persistent effect on the implementation of {\sc mpi} at the receiver, and in particular may reserve state. {\sc mpi} will provide a procedure which frees (or invalidates) invalidates a valid descriptor, allowing the implementation to free reserved state. For example, \mbox{\tt mpi\_free\_cd(cd)}. The user is not allowed to free the own context descriptor of any group or the context descriptor of any context created by the calling process. See \ReferNote{coherency}. \subsubsection*{Descriptor Attributes} {\sc mpi} provides a procedure which allows users to determine the process grouping which is the frame of a context. For example, \mbox{\tt gd = mpi\_context\_group(cd)}. \subsection{Point-to-Point Communication}\LabelSection{point2point} This proposal recommends three forms for {\sc mpi} point-to-point message addressing and selection: null context; closed context; open context. (See \ReferNote{context:form}). It is further recommended that messages communicated in each form are distinguished such that a {\tt Send} operation of form X cannot match with a {\tt Receive} operation of form Y, requiring that form is embedded into the message envelope. The three forms are described, followed by considerations of uniform integration of these forms in the point-to-point communication chapter of {\sc mpi}. \subsubsection*{Null Context Form} The {\it null context\/} form contains no message context. Message selection and addressing are expressed by \mbox{\tt (pd, tag)} where: {\tt pd} is a process descriptor; {\tt tag} is a message tag. {\tt Send} supplies the {\tt pd} of the receiver. {\tt Receive} supplies the {\tt pd} of the sender. {\tt Receive} can wildcard on {\tt pd} by supplying the value of a named constant process descriptor, e.g. {\tt MPI\_PD\_WILD}. See \ReferNote{descriptor:wildcard}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsubsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. Message selection and addressing are expressed by \mbox{\tt (cd, rank, tag)} where: {\tt cd} is a context descriptor; {\tt rank} is a process rank in the frame of {\tt cd}; {\tt tag} is a message tag. {\tt Send} supplies the {\tt cd} of the receiver (and sender), and the {\tt rank} of the receiver. {\tt Receive} supplies the {\tt cd} of the sender (and receiver), and the rank of the sender. The \mbox{\tt (cd, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process descriptor of the receiver (sender). {\tt Receive} cannot wildcard on {\tt cd}. {\tt Receive} can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. See \ReferNote{rank:wildcard}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsubsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. Message selection and addressing are expressed by \mbox{\tt (lcd, rcd, rank, tag)} where: {\tt lcd} is a ``local'' context descriptor; {\tt rcd} is a ``remote'' context descriptor; {\tt rank} is a process rank in the frame of {\tt rcd}; {\tt tag} is a message tag. {\tt Send} supplies the context descriptor for the sender in {\tt lcd}, the context descriptor for the receiver in {\tt rcd}, and the {\tt rank} of the receiver in the frame of {\tt rcd}. {\tt Receive} supplies the context descriptor for the receiver in {\tt lcd}, the context descriptor for the sender in {\tt rcd}, and the {\tt rank} of the sender receiver in the frame of {\tt rcd}. The \mbox{\tt (rcd, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process descriptor of the receiver (sender). {\tt Receive} cannot wildcard on {\tt lcd} which is the open context form analog of there being no wildcard on {\tt cd} in the closed context form. Receive can wildcard on {\tt rcd} by supplying the value of a named constant context descriptor, e.g. {\tt MPI\_CD\_WILD} (See \ReferNote{descriptor:wildcard}), in which case {\tt Receive} {\it must\/} also wildcard on {\tt rank} as there is insufficient information to determine the process descriptor of the sender. {\tt Receive} can wildcard on {\tt rank} by supplying the value of a named constant integer, e.g. {\tt MPI\_RANK\_WILD}. See \ReferNote{rank:wildcard}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsubsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point chapter of {\sc mpi} by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with form. This is at the expense of multiplying the number of {\tt Send} and {\tt Receive} procedures by a factor of three, and some further but trivial work with details of the current point-to-point chapter which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Three options are now described, each based on retention and extension of the framework of one form. These options represent different compromises between the three forms. \paragraph*{Option i: Open Context Framework} The framework of the open context form is adopted and extended. Introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. See \ReferNote{descriptor:null}. The null context form is expressed as \mbox{\tt (MPI\_NULL, MPI\_NULL, pd, tag)}, which is a little clumsy. The closed context form is expressed as \mbox{\tt (MPI\_NULL, cd, rank, tag)}, which is marginally inconvenient. The open context form is expressed as \mbox{\tt (lcd, rcd, rank, tag}), which is of course natural. \paragraph*{Option ii: Closed Context Framework} The framework of the closed context form is adopted and extended. Introduce the {\it null\/} descriptor, the value of which is defined by a named constant, e.g. {\tt MPI\_NULL}. See \ReferNote{descriptor:null}. The null context form is expressed as \mbox{\tt (MPI\_NULL, pd, tag)}, which is marginally inconvenient. The closed context form is expressed as \mbox{\tt (cd, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt cd} field as ``shorthand notation'' for the \mbox{\tt (lcd, rcd)} pair at the expense of introducing some trickery. We define a ``context duplet descriptor'' which is formally composed of two references to contexts, and provide a procedure which constructs such a descriptor given two context descriptors. Both {\tt Send} and {\tt Receive} will accept a duplet descriptor in {\tt cd}, are able to distinguish the duplet descriptor from a singlet descriptor, and treat the duplet as shorthand notation. We should also define a mechanism by which a receiver which has completed a {\tt Receive} with wildcard on {\tt rcd} is able to determine the valid singlet descriptor of the sender, which just adds one further enquiry procedure to the point-to-point chapter(?). This option is a little inconvenient but does have some useful properties for collective communications. It is conjectured that this option is the best choice for {\sc mpi}. \paragraph*{Option iii: Null Context Framework} The framework of the null context form is adopted and extended. The null context form is expressed as \mbox{\tt (pd, tag)}, which is of course natural. Expression of the open and closed context forms requires a little more work. We can use the {\tt pd} field as ``shorthand notation'' for {\tt (cd, rank)} and {\tt (lcd, rcd, rank)} by continuation of the trickery used in the previous option. This is rather clumsy. \subsection{Collective Communication}\LabelSection{collective} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context (thus frame) in which they are to operate. {\sc mpi} does plan to describe symmetric collective communication operations. It is not possible to determine whether this proposal is sufficient to allow implementation of the collective communication chapter of {\sc mpi} in terms of the point-to-point chapter of {\sc mpi} without loss of generality, since the collective operations are not yet defined. Asymmetric collective communication operations, especially those in which sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (perhaps in a duplet descriptor form) which identify the contexts (thus frames) in which they are to operate. {\sc mpi} does not plan to describe asymmetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model, which are composed of communicative functionally distinct process groupings. This proposal recommends that such operations should be considered in some reincarnation of {\sc mpi}. % % END "Proposal" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Discussion & Notes" % \section{Discussion \& Notes} This section comprises a discussion of certain aspects of this proposal followed by the notes referenced in the detailed proposal. \subsection{Discussion}\LabelSection{discussion} We can dissect the proposal into two parts: an SPMD model core; an MIMD model annex. In this discussion the dissection is exposed and the conceptual foundation of each part is described. The discussion also presents brief arguments for and against the MIMD model annex. \subsubsection*{SPMD model core} The SPMD model core provides noncommunicative process groupings and communication contexts for writers of SPMD parallel libraries. It is intended to provide expressive power beyond the ``SPIMD'' model (in which process execute in an SIMD fashion). The material describing processes in \ReferSection{processes} is simplified: processes have identical instruction blocks and different data blocks; process descriptor transmission and registry become redundant; dynamic process models are not considered. The material describing process groupings in \ReferSection{groupings} is simplified: group descriptor transmission and registry become redundant; the own process grouping explicitly becomes a single group containing all processes. The material describing communication contexts in \ReferSection{contexts} is simplified: context descriptor transmission and registry become unnecessary. The material describing point-to-point communication in \ReferSection{point2point} is simplified: the open context form becomes redundant; uniform integration ``Option i'' is deleted, and ``Option ii'' loses duplet descriptors, becoming simple enough that ``Option iii'' need not be further considered. The material describing collective communication in \ReferSection{collective} is simplified: there is no possibility of collective communication operations spanning more than one context. \subsubsection*{MIMD model annex} The MIMD model annex extends and modifies the SPMD model core to provide expressive power for MIMD programs which combine (coarse grain) function and data driven parallelism. The MIMD model annex is not intended to provide expressive power to fine grained function driven parallel programs --- it is conjectured that message passing approaches such as {\sc mpi} are not suited to fine grained parallel programming. The annex is intended to provide expressive power for the ``MSPMD'' model, which is now described. One of the simplest MIMD models is the ``host-node'' model, familiar in {\sc express} and {\sc parmacs}, containing two functional groups: one node group (SPMD like) ; one host group (a singleton). The ``parallel client-server'' model, in which each of the $n$ clients is composed of parallel processes, and in which the server may also be composed of parallel processes, contains $1+n$ functional groups: $n$ client groups (SPMD like); one server group (singleton, SPMD like). The ``host-node'' model is a case of this model in which the host can be viewed as a singleton client and the nodes can be viewed as an SPMD like server (or the host as a singleton server and the nodes as an SPMD like client). The ``parallel module graph'' model, in which each module within the graph may be composed of parallel processes (singleton, SPMD like), contains any number of functional groups with arbitrarily complex relations. The ``parallel client-server model'' is a case of this model in which the module graph only contains arcs joining the server to each client. The MIMD model annex is intended to provide expressive power for the ``parallel module graph'' model, which I refer to as the MPSMD model. This model requires support at some level as commercial and modular applications are increasingly moving in to parallel computing. The debate is whether or not message passing approaches such as {\sc mpi} (which I simply refer to as MPI) should provide for this model. The negative argument is that such SPMD like modules should be controlled and communicate with one another as ``parallel processes'' at the distributed operating system level. The argument has some appeal as the world of distributed operating systems must deal with difficult issues such as process control and coherency. Avoidance of duplication in MPI allows MPI to focus on provision of a smaller set of facilities with greater emphasis on maximum performance for data driven SPMD like parallel programs. The positive argument is that communications between such SPMD like modules require high performance and MPI can provide such performance with tuned semantics which expect the user to deal with coherency issues. There is also the argument that MPI is able to deal with this in a shorter time than development (and standardisation) procedures for distributed operating systems. The latter argument is somewhat comparable with the argument for message passing versus parallel compilation. \subsection{Notes}\LabelSection{notes} \begin{enumerate} \item\LabelNote{descriptor} {\bf Descriptors:} Descriptors are assumed to be a plentiful but bounded resource. They are opaque references to objects of undefined size and structure. They are not global unique identifiers however they must reference such identifiers, and they protect the user from the form of such identifiers allowing them to be implementation dependent. The proposal expresses descriptors as integer types in the host language (in practice we might expect descriptors to be indices into tables of structures, or tables of pointers to structures, or indeed pointers to structures themselves). This expression facilitates uniform binding to ANSI~C and Fortran~77. The context descriptor must at least reference: the global unique context identifier; the group descriptor of the frame. The group descriptor must at least reference: the global unique group identifier; the own context descriptor; the rank space to process descriptor map of the group (including the size of the group); the process rank. The process descriptor must at least reference; the global unique process identifier; the group descriptor of the own group. The proposal text distinguishes process, group and context identifiers more strongly than is strictly necessary. There is potential advantage in the concept of a unified descriptor which can be used to reference either kind of actual descriptor. For example descriptor unification goes some way toward cleaning up the duplet, descriptors described in ``Option ii'' of section \ReferSection{point2point}. Definition in this fashion requires the referenced object to contain a class identifier (which in this proposal could be as little as 3 bits wide) This suggestion is explored in some of the notes below. Rik Littlefield has suggested descriptors could be used to ``cache'' per object user information, as appears in {\sc zipcode}. This descriptor capability could be seriously useful for example in context or group specific implementations of collective communications. The suggestion both requires and deserves more work. There were additional motivations for expressing process descriptors as integers in the proposal. Firstly, the author anticipated ``Option ii'' of \ReferSection{point2point}. Secondly, it is observed that definition of process descriptors can be weakened such that they may be implemented as global unique process identifiers, expressed as integers, should this infer advantage. {[\it I wish I had realised the second point before the first!]} The consequence of allowing process descriptors to be implemented as process identifiers is explored in some of the notes below. Please also note that this would exclude process descriptors from the ``caching'' capability mentioned above. These ideas suggest an alternate proposal, containing global unique process identifiers expressed as integers, and context and group descriptors as opaque references of yet unspecified language binding (i.e., perhaps not an integer). This suggestion is also explored in some of the notes below. \item\LabelNote{processes:dynamic} {\bf Dynamic Processes:} The proposal does not prevent a process model which allows dynamic creation and deletion of processes however it does not favour an asynchronous process model in which singleton processes are created and deleted in an arbitrary fashion. The proposal does favour a model in which blobs of processes are created (and deleted) in a concerted fashion, and in which each blob so created is assigned a different own process grouping. This model does not take into account the potential desire to expand or contract an existing blob of processes in order to take into account (presumably slowly) time varying workloads. It is conjectured that concerted blob expand and contract operations are more suitable for this purpose than asynchronous singleton spawn and kill operations. \item\LabelNote{descriptor:transmission} {\bf Descriptor transmission:} In the spirit of descriptor unification (See \ReferNote{descriptor}) the three {\tt MPI\_?D\_TYPE} names can be collapsed into something like {\tt MPI\_DESC\_TYPE}. If process descriptors are replaced by global unique process identifiers (See \ReferNote{descriptor}) then no special measures are required for transmission thereof. If group and context descriptors are expressed as opaque objects of yet unspecified type then, in ANSI~C at least, it will be possible to prevent nonsemantic transmission thereof. \item\LabelNote{descriptor:registry} {\bf Descriptor registry:} The registry service is just a simpler way for concurrent objects to identify and establish communications with one another. The operations of interest, expressed in the spirit of descriptor unification (See \ReferNote{descriptor}), are: registration of descriptor by name, e.g. \mbox{\tt mpi\_register(name,desc)}; deregistration, e.g \mbox{\tt mpi\_deregister(desc)}; and lookup by name, e.g. \mbox{\tt mpi\_lookup(name, \&desc, wait)} where {\tt wait} controls whether the lookup waits for the name to be registered. If process descriptors are replaced by global unique process identifiers (See \ReferNote{descriptor}) then perhaps process identifier registry is not so important. The MIMD model does not actually required process descriptor or group descriptor registry to be visible to the user since context descriptor registry and context descriptor attribute determination gives access to all groups and thus group descriptor attribute determination gives access to all processes. The proposal was written to handle descriptors consistently. \item\LabelNote{descriptor:deallocation} {\bf Descriptor deallocation:} In the spirit of descriptor unification (See \ReferNote{descriptor}) the three \mbox{\tt mpi\_?d\_free(?d)} can be collapsed into something like \mbox{\tt mpi\_desc\_free(desc)}. The receipt of a descriptor in descriptor transmission and registry is an allocator, hence provision of the deallocator. Perhaps there should be an explicit allocator which the user must call in order to receive a descriptor, and can deallocate when no longer required. If process descriptors are replaced by global unique process identifiers (See \ReferNote{descriptor}) then process identifier deallocation is moot. \item\LabelNote{coherency} {\bf Coherency:} The proposal admits incoherency as descriptors may be received in transmission or registry. The SPMD core contains no incoherency. The inclusion of dynamic process creation and deletion admits incoherency since processes can retain descriptors of processes which have been deleted. The inclusion of grouping descriptor transmission and registry admits incoherency since processes can retain descriptors of groupings which have been deleted. The inclusion of dynamic groupings admits incoherency since processes can retain descriptors of groupings of which the rank to process map has changed. The inclusion of context descriptor transmission and registry admits incoherency since processes can retain descriptors of contexts which have been deleted. The proposal expects the user to ensure coherent usage. It is conjectured that this is acceptable provided that the user is not also expected to implement process fault tolerance. \item\LabelNote{groupings:dynamic} {\bf Dynamic groupings:} Process groupings are dynamic in the sense that they can be created at any time, and static in the sense that the membership is constant over the lifetime of the process grouping. The proposal specifies static groupings however the loose separation of communication contexts from process groupings simplifies extension to dynamic groupings as contexts stretch or shrink according to the changes in their frames. It is conjectured that concerted grouping expand and contract operations are more suitable than asynchronous singleton join and leave operations. \item\LabelNote{grouping:blobs} {\bf Process blobs:} {\sc mpi} has discussed the concept of the ``all'' group which contains all processes. The ``own'' group concept is a generalisation of the ``all'' group concept which is expressive for programs including and beyond the SPMD model. Processes are created in ``blobs'', where each member of a blob is a member of the same own process grouping, and different blobs have different own process groupings. An SPMD program is a single blob. A host-node program composes two blobs, the node blob and the host blob (a singleton). There is a sense in which a blob is SPMD like. \item\LabelNote{grouping:own} {\bf \mbox{\tt mpi\_own\_group(pd)}:} This procedure looks like a case of process descriptor attribute determination. If process descriptors are allowed to be implemented as global unique process identifiers, or are replaced, this procedure should accept no arguments and return the own group descriptor of the calling process. \item\LabelNote{inverse:map} {\bf Inverse map:} The proposal did not include a function to convert \mbox{\tt (gd, pd)} pair into a rank. It is suggested that this inverse map is allowed to be ``slow'', i.e. could be a linear search over members of the group, but probably should be included for completeness. It can be used as a membership predicate. \item\LabelNote{context:own} {\bf \mbox{\tt mpi\_own\_context(gd)}:} This procedure looks like a case of group descriptor attribute determination. \item\LabelNote{grouping:deletion} {\bf Grouping Deletion:} The process grouping deletion operation should probably be defined to fail when there are user created contexts with that frame which have not themselves been deleted. This just requires a reference count in the group descriptor static attribute store. \item\LabelNote{context:creation:deletion} {\bf Context Creation \& Deletion:} Marc Snir has described a method by which global unique group identifiers can be generated without use of shared global data. The proposal states that context creation and deletion operations synchronise the processes within the frame of the context, anticipating use of this method for generation of context identifiers. However, the synchronisation requires that context creation and deletion calls within a frame are performed in the identical sequence by all members of the frame. The global unique group identifier and context creator reference count are then sufficient to generate a global unique context identifier without communication or synchronisation. Should context creation and deletion therefore not synchronise the frame? There may be advantage in defining context creation and deletion such that a number of contexts are created or deleted simultaneously, depending on how heavy we expect context management to be in implementations of {\sc mpi}. \item\LabelNote{descriptor:wildcard} {\bf Descriptor wildcard:} In the spirit of descriptor unification (See \ReferNote{descriptor}) the three named constants {\tt MPI\_?D\_WILD} can be collapsed into something like {\tt MPI\_DESC\_WILD}. If process descriptors are replaced with global unique process identifiers then perhaps the wildcard process identifier value can be the same as the wildcard tag value, and the same named constant. \item\LabelNote{context:form} {\bf Context forms:} The null form is like PVM~3. It is general purpose, but not particularly expressive. It does not provide facilities for writers of parallel libraries. It has the potential to provide maximum performance. The closed form is like {\sc zipcode}. It is expressive in SPMD programs where noncommunicative distinct data driven parallel computations can be performed concurrently. It provides facilities for writers of SPMD like parallel libraries. The open form is like {\sc chimp}. It is expressive in MIMD programs where communicative data driven parallel computations can be performed concurrently. It provides facilities for MIMD like parallel libraries. \item\LabelNote{rank:wildcard} {\bf Rank wildcard:} Since rank is an integer like message tag, perhaps they should have the same wildcard value, and the same named constant. \item\LabelNote{descriptor:null} {\bf {\tt MPI\_NULL}:} I am following the spirit of context unification (See \ReferNote{descriptor}) in the proposal text here. There may be advantage in defining the value of the null descriptor to be the ANSI~C constant {\tt NULL}, or even defining the value to be exactly zero (every rule having a useful exception). \end{enumerate} % % END "Discussion & Notes" %----------------------------------------------------------------------% %----------------------------------------------------------------------% % BEGIN "Conclusion" % \section{Conclusion}\LabelSection{conclusion} This chapter has presented and discussed a proposal for communication contexts within {\sc mpi}. In the proposal process groupings appeared as frames (or templates) for the construction of communication contexts, and communication contexts retained certain properties of the frames used in their construction. % % END "Conclusion" %----------------------------------------------------------------------% % % END "Proposal VI" %======================================================================% \end{document} ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 23 13:47:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25054; Tue, 23 Mar 93 13:47:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23695; Tue, 23 Mar 93 13:46:26 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 13:46:23 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23555; Tue, 23 Mar 93 13:45:19 -0500 Date: Tue, 23 Mar 93 18:45:08 GMT Message-Id: <15959.9303231845@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Proposal VI, First Revision, PostScript To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Here is PostScript for the revision of Proposal VI. Please see my reply to Jim of today for an idea of what is changed. I apologise that this is one hour later then I had previously announced. LaTeX previously sent. Best Wishes Lyndon ---------------------------------------------------------------------- %!PS-Adobe-2.0 %%Creator: dvips 5.495 Copyright 1986, 1992 Radical Eye Software %%Title: context-vi.dvi %%CreationDate: Tue Mar 23 18:43:31 1993 %%Pages: 16 %%PageOrder: Ascend %%BoundingBox: 0 0 596 842 %%EndComments %DVIPSCommandLine: dvips context-vi %DVIPSSource: TeX output 1993.03.23:1840 %%BeginProcSet: tex.pro %! /TeXDict 250 dict def TeXDict begin /N{def}def /B{bind def}N /S{exch}N /X{S N} B /TR{translate}N /isls false N /vsize 11 72 mul N /@rigin{isls{[0 1 -1 0 0 0] concat}if 72 Resolution div 72 VResolution div neg scale isls{0 Resolution vsize 72 div mul TR}if Resolution VResolution vsize -72 div 1 add mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@landscape{/isls true N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{ CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N} B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto /V matrix currentmatrix dup 1 get dup mul exch 0 get dup mul add .99 lt{/QV}{/RV}ifelse load def pop pop}N /eop{SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook} if pop /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255 {IE S 1 string dup 0 3 index put cvn put}for 65781.76 div /vsize X 65781.76 div /hsize X}N /p{show}N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V{}B /RV statusdict begin /product where{ pop product dup length 7 ge{0 7 getinterval dup(Display)eq exch 0 4 getinterval(NeXT)eq or}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /QV{ gsave transform round exch round exch itransform moveto rulex 0 rlineto 0 ruley neg rlineto rulex neg 0 rlineto fill grestore}B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M} B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{ p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{SS restore}B end %%EndProcSet TeXDict begin 39158280 55380996 1000 300 300 (/a/obsidian/disk/home/u36/lyndon/mpi/context-vi.dvi) @start /Fa 1 79 df<07E01FC000E0060001700400017004000138040001380400021C0800021C080002 0E0800020E0800040710000407100004039000040390000801E0000801E0000800E0000800E000 18004000FE0040001A147F931A>78 D E /Fb 3 111 df<01FC00FF80001C001C00002E001800 002E001000002E001000002700100000470020000043002000004380200000438020000081C040 000081C040000081C040000080E040000100E08000010070800001007080000100708000020039 00000200390000020039000002001D000004001E000004000E000004000E00000C000E00001C00 040000FF80040000211C7E9B21>78 D<00FFFFE000F001C001C003800180070001000E0001001E 0002001C0002003800020070000000E0000001C0000003800000070000000F0000001E0000001C 0000003800000070020000E0040001C0040003800400070008000F0008000E0018001C00300038 0070007001E000FFFFE0001B1C7E9B1C>90 D<381F004E61804681C04701C08F01C08E01C00E01 C00E01C01C03801C03801C03801C0700380710380710380E10380E2070064030038014127E9119 >110 D E /Fc 46 123 df<03800007E0000FE0001E70001C70001C70001C70001C77E01CE7E0 1DE7E00FC7000F8E000F0E001E0E003F1C007F1C00739C00E3F800E1F800E0F1C0E0F1C071F9C0 7FFFC03F9F801E070013197F9816>38 D<00E001E0038007000E001C001C003800380070007000 7000E000E000E000E000E000E000E000E000E000700070007000380038001C001C000E00070003 8001E000E00B217A9C16>40 DI<387C7E7E3E0E1E1C78F060070B798416>44 D<7FFF00FFFF80FFFF80000000 000000000000000000000000FFFF80FFFF807FFF00110B7E9116>61 D<0FE03FF87FFCF01EF00E F00E601E003C007800F001C0038003800380038003800300000000000000000003000780078003 000F197D9816>63 D<00E00001F00001F00001B00001B00003B80003B80003B800031800071C00 071C00071C00071C00071C000E0E000E0E000FFE000FFE001FFF001C07001C07001C07007F1FC0 FF1FE07F1FC013197F9816>65 D<01F18007FB800FFF801F0F803C0780380380700380700380F0 0000E00000E00000E00000E00000E00000E00000E00000F000007003807003803803803C07001F 0F000FFE0007FC0001F00011197E9816>67 D<7FF800FFFE007FFF001C0F001C07801C03C01C01 C01C01C01C01E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C01C01C01C01C03 C01C07801C0F807FFF00FFFE007FF8001319809816>I<7FFFC0FFFFC07FFFC01C01C01C01C01C 01C01C01C01C00001C00001C1C001C1C001FFC001FFC001FFC001C1C001C1C001C00001C00E01C 00E01C00E01C00E01C00E07FFFE0FFFFE07FFFE013197F9816>I<03E30007FF000FFF001E1F00 3C0F00380700700700700700F00000E00000E00000E00000E00000E03F80E07FC0E03F80F00700 700700700700380F003C0F001E1F000FFF0007F70003E70012197E9816>71 D73 D<7F0FE0FF8FF07F0FE01C07801C0F001C0E001C 1C001C3C001C78001CF0001CE0001DF0001FF0001FF8001F38001E1C001C1C001C0E001C0E001C 07001C07001C03807F07E0FF8FF07F07E01419809816>75 DII<7E1FC0FF3FE0 7F1FC01D07001D87001D87001D87001DC7001DC7001CC7001CC7001CE7001CE7001CE7001C6700 1C67001C77001C77001C37001C37001C37001C17007F1F00FF9F007F0F0013197F9816>I<7FF8 00FFFE007FFF001C0F801C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE 001FF8001C00001C00001C00001C00001C00001C00001C00007F0000FF80007F000012197F9816 >80 D<7FE000FFF8007FFC001C1E001C0F001C07001C07001C07001C07001C0F001C1E001FFC00 1FF8001FFC001C1C001C0E001C0E001C0E001C0E001C0E201C0E701C0E707F07E0FF87E07F03C0 14197F9816>82 D<07E3001FFF003FFF00781F00F00700E00700E00700E00000F000007800003F 80001FF00007FC0000FE00000F00000700000380000380600380E00380E00700F80F00FFFE00FF FC00C7F00011197E9816>I<7FFFE0FFFFE0FFFFE0E0E0E0E0E0E0E0E0E0E0E0E000E00000E000 00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 07FC000FFE0007FC0013197F9816>I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C00E03800E03 8007070007FF0003FE0000F8001519809816>I87 D89 D<1FE0003FF0007FF800 783C00300E00000E00000E0003FE001FFE003E0E00700E00E00E00E00E00E00E00783E007FFFE0 3FE7E00F83E013127E9116>97 D<7E0000FE00007E00000E00000E00000E00000E00000E3E000E FF000FFF800F83C00F00E00E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F 83C00FFF800EFF00063C001419809816>I<03F80FFC1FFE3C1E780C7000E000E000E000E000E0 00F000700778073E0E1FFC0FF803F010127D9116>I<003F00007F00003F000007000007000007 0000070003C7000FF7001FFF003C1F00780F00700700E00700E00700E00700E00700E00700E007 00700F00700F003C1F001FFFE00FE7F007C7E014197F9816>I<03E00FF81FFC3C1E780E7007E0 07FFFFFFFFFFFFE000E000700778073C0F1FFE0FFC03F010127D9116>I<001F00007F8000FF80 01E78001C30001C00001C0007FFF00FFFF00FFFF0001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C00001C00001C0003FFE007FFF003FFE0011197F9816>I<03E3C007F7 E00FFFE01C1CC0380E00380E00380E00380E00380E001C1C000FF8001FF0001BE0003800001800 001FFC001FFF003FFF807803C0E000E0E000E0E000E0E000E07001C07C07C03FFF800FFE0003F8 00131C7F9116>I<018003C003C0018000000000000000007FC07FC07FC001C001C001C001C001 C001C001C001C001C001C001C001C07FFFFFFF7FFF101A7D9916>105 D<7E0000FE00007E0000 0E00000E00000E00000E00000E7FE00E7FE00E7FE00E0F000E1E000E3C000E78000EF0000FF000 0FF8000FBC000F1E000E0E000E07000E07807F87F0FFCFF07F87F01419809816>107 DII<7E3C00FEFE007FFF000F8780 0F03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F8 7FC7F01512809116>I<03E0000FF8001FFC003C1E00780F00700700E00380E00380E00380E003 80E00380F00780700700780F003C1E001FFC000FF80003E00011127E9116>I<7E3E00FEFF007F FF800F83C00F00E00E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00F FF800EFF000E3C000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000141B8091 16>I114 D<0FEC3FFC7FFCF03CE01CE01C 70007F801FF007F8003C600EE00EF00EF81EFFFCFFF8C7E00F127D9116>I<0300000700000700 000700000700007FFF00FFFF00FFFF000700000700000700000700000700000700000700000701 0007038007038007038007870003FE0001FC0000F80011177F9616>I<7E1F80FE3F807E1F800E 03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E03800E0F800FFFF007 FBF803E3F01512809116>I<7F1FC0FF1FE07F1FC01C07001E0F000E0E000E0E000E0E00071C00 071C00071C00071C0003B80003B80003B80001F00001F00000E00013127F9116>II<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001 F00000E00001E00001F00003B800073C00071C000E0E007F1FC0FF3FE07F1FC013127F9116>I< 7F1FC0FF9FE07F1FC01C07000E07000E0E000E0E00070E00071C00071C00039C00039C00039800 01B80001B80000F00000F00000F00000E00000E00000E00001C00079C0007BC0007F80003F0000 3C0000131B7F9116>I<3FFFC07FFFC07FFFC0700780700F00701E00003C0000780001F00003E0 000780000F00001E01C03C01C07801C0FFFFC0FFFFC0FFFFC012127F9116>I E /Fd 27 123 df<0001FC000703000C03001C07001C0300180000380000380000380000380000 700007FFFC00701C00701C00701C00E03800E03800E03800E03800E07001C07001C07001C07001 C0E201C0E201C0E20380E4038064038038038000030000070000060000C60000E40000CC000070 00001825819C17>12 D<00C001E001E001E001C003C003C0038003800380030007000700060006 00060004000C000C00080008000000000000000000000030007800F00060000B1E7C9D0D>33 D<01FFC0003C0000380000380000380000380000700000700000700000700000E00000E00000E0 0000E00001C00001C00001C00001C0000380000380000380000380000700000700000700000700 000F0000FFE000121C7E9B10>73 D<003F80007F00000300000300000300000600000600000600 000600000C00000C00000C00000C00001800001800001800001800003000003000003000003000 00600000600000600000600000C00000C00000C00000C000018000018000018000018000030000 030000030000030000060000060000FE0000FE00001129819E0D>93 D<03CC063C0C3C181C3838 303870387038E070E070E070E070E0E2C0E2C0E261E462643C380F127B9115>97 D<3F00070007000E000E000E000E001C001C001C001C0039C03E60383038307038703870387038 E070E070E070E060E0E0C0C0C1C0618063003C000D1D7B9C13>I<01F007080C08181C38383000 70007000E000E000E000E000E000E008E010602030C01F000E127B9113>I<001F800003800003 80000700000700000700000700000E00000E00000E00000E0003DC00063C000C3C00181C003838 00303800703800703800E07000E07000E07000E07000E0E200C0E200C0E20061E4006264003C38 00111D7B9C15>I<01E007100C1018083810701070607F80E000E000E000E000E000E008601060 2030C01F000D127B9113>I<0003C0000670000C70001C60001C00001C00003800003800003800 00380000380003FF8000700000700000700000700000700000E00000E00000E00000E00000E000 01C00001C00001C00001C00001C000038000038000038000030000030000070000C60000E60000 CC00007800001425819C0D>I<00F3018F030F06070E0E0C0E1C0E1C0E381C381C381C381C3838 30383038187818F00F700070007000E000E0C0C0E1C0C3007E00101A7D9113>I<0FC00001C000 01C0000380000380000380000380000700000700000700000700000E78000E8C000F0E000E0E00 1C0E001C0E001C0E001C0E00381C00381C00381C00383800703880703880707080707100E03200 601C00111D7D9C15>I<01800380010000000000000000000000000000001C002600470047008E 008E000E001C001C001C0038003800710071007100720072003C00091C7C9B0D>I<0FC00001C0 0001C0000380000380000380000380000700000700000700000700000E0F000E11000E23800E43 801C83001C80001D00001E00003F800039C00038E00038E00070E20070E20070E20070E400E064 00603800111D7D9C13>107 D<1F800380038007000700070007000E000E000E000E001C001C00 1C001C0038003800380038007000700070007000E400E400E400E40068003800091D7C9C0B>I< 3C1E0780266318C04683A0E04703C0E08E0380E08E0380E00E0380E00E0380E01C0701C01C0701 C01C0701C01C070380380E0388380E0388380E0708380E0710701C0320300C01C01D127C9122> I<3C3C002646004687004707008E07008E07000E07000E07001C0E001C0E001C0E001C1C00381C 40381C40383840383880701900300E0012127C9117>I<01E007180C0C180C380C300E700E700E E01CE01CE01CE018E038E030E06060C031801E000F127B9115>I<07870004D98008E0C008E0C0 11C0E011C0E001C0E001C0E00381C00381C00381C00381800703800703000707000706000E8C00 0E70000E00000E00001C00001C00001C00001C00003C0000FF8000131A7F9115>I<3C3C26C246 8747078E068E000E000E001C001C001C001C0038003800380038007000300010127C9112>114 D<01F006080C080C1C18181C001F001FC00FF007F0007800386030E030C030806060C01F000E12 7D9111>I<00C001C001C001C00380038003800380FFE00700070007000E000E000E000E001C00 1C001C001C00384038403840388019000E000B1A7D990E>I<1E0300270700470700470700870E 00870E000E0E000E0E001C1C001C1C001C1C001C1C003838803838801838801839001C5900078E 0011127C9116>I<1E06270E470E4706870287020E020E021C041C041C041C0818083808181018 200C4007800F127C9113>I<1E01832703874703874703838707018707010E07010E07011C0E02 1C0E021C0E021C0E04180C04181C04181C081C1C100C263007C3C018127C911C>I<070E001991 0010E38020E38041C30041C00001C00001C000038000038000038000038000070200670200E704 00CB04008B080070F00011127D9113>I<038207C20FEC08381008001000200040008001000200 040008081008383067F043E081C00F127D9111>122 D E /Fe 39 122 df<003C000000E30000 01C1000003C18000038180000781800007C1800007C3000007C2000007C4000007C8000003F001 FF03E001FF03E0003003F0006001F000C003F800C004FC01801CFC0300387E0300783F0600781F 8C00F81FD800F80FF000FC07F0067C01F8067E07FE1C1FFE3FF807F007F0201D7E9C25>38 D<78FCFCFCFC7800000000000078FCFCFCFC7806127D910D>58 D<00038000000380000007C000 0007C0000007C000000FE000000FE000001FF000001BF000001BF0000031F8000031F8000061FC 000060FC0000E0FE0000C07E0000C07E0001803F0001FFFF0003FFFF8003001F8003001F800600 0FC006000FC00E000FE00C0007E0FFC07FFEFFC07FFE1F1C7E9B24>65 D<001FE02000FFF8E003 F80FE007C003E00F8001E01F0000E03E0000E03E0000607E0000607C000060FC000000FC000000 FC000000FC000000FC000000FC000000FC000000FC0000007C0000607E0000603E0000603E0000 C01F0000C00F80018007C0030003F80E0000FFFC00001FE0001B1C7D9B22>67 DI70 D<000FF008007FFE3801FC07F807E001F80F8000781F0000783F0000383E0000387E 0000187C000018FC000000FC000000FC000000FC000000FC000000FC000000FC007FFFFC007FFF 7C0001F87E0001F83E0001F83F0001F81F0001F80F8001F807E001F801FC07F8007FFE78000FF8 18201C7D9B26>I73 D77 DI<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E0003E07E0003F07C0001F0 7C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F87C0001 F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E0001F07C00003FE0001D1C 7D9B24>II82 D<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE0000FFE0007FFE003FFF 003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000E0E000E0F001C0FC03 C0EFFF0083FC00141C7D9B1B>I<7FFFFFE07FFFFFE0781F81E0701F80E0601F8060E01F8070C0 1F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F8000001F8000001F8000 001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F80 0007FFFE0007FFFE001C1C7E9B21>II<0FF8001C1E003E0F803E07803E07C01C07C00007C0 007FC007E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13F80FE1F815127F9117> 97 DI<03FC000E0E001C1F003C1F00781F00780E00F800 00F80000F80000F80000F80000F800007800007801803C01801C03000E0E0003F80011127E9115 >I<000FF0000FF00001F00001F00001F00001F00001F00001F00001F00001F00001F001F9F00F 07F01C03F03C01F07801F07801F0F801F0F801F0F801F0F801F0F801F0F801F07801F07801F03C 01F01C03F00F0FFE03F9FE171D7E9C1B>I<01FC000F07001C03803C01C07801C07801E0F801E0 F801E0FFFFE0F80000F80000F800007800007C00603C00601E00C00F038001FC0013127F9116> I<007F0001E38003C7C00787C00F87C00F83800F80000F80000F80000F80000F8000FFF800FFF8 000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F80007FF8007FF800121D809C0F>I<03F8F00E0F381E0F381C07303C07803C07803C07803C 07801C07001E0F000E0E001BF8001000001800001800001FFF001FFFC00FFFE01FFFF07801F8F0 0078F00078F000787000707800F01E03C007FF00151B7F9118>II<1E003F003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F 001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>I107 DIII< 01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800F8F800F8F800F87800F0 7800F03C01E01E03C00F078001FC0015127F9118>II114 D<1FD830786018E018E018F000FF807FE07FF01FF807FC007CC0 1CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007000F000F003FFCFFFC1F 001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08079803F00E1A7F9913>IIIIII E /Ff 28 121 df45 D<387CFEFEFE7C3807077C8610>I<00 180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001 F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC07007E07C07 F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C0000F80001F 00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07FFFF0FFFF F0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F81F03F81F 03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC3C00FE7E 00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C>I<0000E0 0001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187E00307E0 0707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007E00007E0 0007E00007E00007E000FFFE00FFFE17207E9F1C>I<1000201E01E01FFFC01FFF801FFF001FFE 001FF8001BC00018000018000018000018000019FC001FFF001E0FC01807E01803E00003F00003 F00003F80003F83803F87C03F8FE03F8FE03F8FC03F0FC03F07007E03007C01C1F800FFF0003F8 0015207D9F1C>I<0003FE0080001FFF818000FF01E38001F8003F8003E0001F8007C0000F800F 800007801F800007803F000003803F000003807F000001807E000001807E00000180FE00000000 FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE000000007E000000 007E000001807F000001803F000001803F000003801F800003000F8000030007C000060003F000 0C0001F800380000FF00F000001FFFC0000003FE000021227DA128>67 DI<0003FE0040001FFFC0C0007F00F1C001F8003FC003F0000FC007C0 0007C00FC00003C01F800003C03F000001C03F000001C07F000000C07E000000C07E000000C0FE 00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE000FFFFC 7E000FFFFC7F00001FC07F00001FC03F00001FC03F00001FC01F80001FC00FC0001FC007E0001F C003F0001FC001FC003FC0007F80E7C0001FFFC3C00003FF00C026227DA12C>71 D78 D80 D<07FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F000 3FF003FDF01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18 167E951B>97 D<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC00 00FC0000FC0000FC00007C00007E00007E00003E00301F00600FC0E007FF8000FE0014167E9519 >99 D<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000 FC0000FC00007C00007C00007E00003E00181F00300FC07003FFC000FF0015167E951A>101 D<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03C03E07C01F0F 801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC70003EF0001EF000 1EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C>103 D<1C003F007F007F007F003F001C000000000000000000000000000000FF00FF001F001F001F00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B247EA310 >105 D108 DII<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC 007EFC007EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE0017167E951C>I< FF0FE000FF3FF8001FF07C001F803E001F001F001F001F801F001F801F000FC01F000FC01F000F C01F000FC01F000FC01F000FC01F000FC01F000FC01F001F801F001F801F803F001FC03E001FE0 FC001F3FF8001F0FC0001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F 000000FFE00000FFE000001A207E951F>I114 D<0FF3003FFF00781F00600700E00300E00300F00300FC0000 7FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00 C7F00011167E9516>I<0180000180000180000180000380000380000780000780000F80003F80 00FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>III120 D E /Fg 14 123 df45 D<002000007000007000 00700000B80000B80000B800011C00011C00011C00020E00020E0004070004070007FF00080380 0803800803801801C03803C0FE0FF815157F9419>97 D<00FC200782600E01E01C00E038006078 0020700020F00020F00000F00000F00000F00000F00000F000207000207800203800401C00400E 008007830000FC0013157E9419>99 DII< FF8FF81C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01FFFC01C01C01C01C0 1C01C01C01C01C01C01C01C01C01C01C01C01C01C0FF8FF815157F9419>104 DI109 D<01F800070E000C03001C038038 01C07801E07000E0F000F0F000F0F000F0F000F0F000F0F000F0F000F07000E07801E03801C01C 03801E0780070E0001F80014157E941A>111 DI114 D<1F1030F06030C030C010C010C000E0007E003FC01FE003F0007800380018801880 188010C030F0608FC00D157E9413>I120 D122 D E /Fh 76 125 df<007E1F0001C1B18003 03E3C00703C3C00E03C1800E01C0000E01C0000E01C0000E01C0000E01C0000E01C000FFFFFC00 0E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0 000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00000E00000E00000E0000FFFFC00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C07F87F8151D809C17>I<007FC001C1C00303C00703C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCFF8151D809C17>I<003F07E00001 C09C18000380F018000701F03C000E01E03C000E00E018000E00E000000E00E000000E00E00000 0E00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>I<6060F0F0F8F868680808 08080808101010102020404080800D0C7F9C15>34 D<00E0000001900000030800000308000007 0800000708000007080000070800000710000007100000072000000740000003C03FE003800F00 038006000380040005C0040009C0080010E0100030E010006070200060702000E0384000E03C40 00E01C8000E00F0020E0070020700780403009C0401830E18007C03E001B1F7E9D20>38 D<004000800100020006000C000C0018001800300030007000600060006000E000E000E000E000 E000E000E000E000E000E000E000E000600060006000700030003000180018000C000C00060002 000100008000400A2A7D9E10>40 D<800040002000100018000C000C0006000600030003000380 01800180018001C001C001C001C001C001C001C001C001C001C001C001C0018001800180038003 000300060006000C000C00180010002000400080000A2A7E9E10>I<0006000000060000000600 000006000000060000000600000006000000060000000600000006000000060000000600000006 0000FFFFFFE0FFFFFFE00006000000060000000600000006000000060000000600000006000000 06000000060000000600000006000000060000000600001B1C7E9720>43 D<60F0F0701010101020204080040C7C830C>II<60F0F06004047C830C >I<03C00C301818300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E0 07E007E00760066006700E300C300C18180C3007E0101D7E9B15>48 D<030007003F00C7000700 070007000700070007000700070007000700070007000700070007000700070007000700070007 0007000F80FFF80D1C7C9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E00 0E001C001C00380070006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<07 E01830201C201C781E780E781E381E001C001C00180030006007E00030001C001C000E000F000F 700FF80FF80FF80FF00E401C201C183007E0101D7E9B15>I<000C00000C00001C00003C00003C 00005C0000DC00009C00011C00031C00021C00041C000C1C00081C00101C00301C00201C00401C 00C01C00FFFFC0001C00001C00001C00001C00001C00001C00001C0001FFC0121C7F9B15>I<30 0C3FF83FF03FC020002000200020002000200023E024302818301C200E000E000F000F000F600F F00FF00FF00F800E401E401C2038187007C0101D7E9B15>I<00F0030C06040C0E181E301E300C 700070006000E3E0E430E818F00CF00EE006E007E007E007E007E007600760077006300E300C18 180C3003E0101D7E9B15>I<4000007FFF807FFF007FFF00400200800400800400800800001000 00100000200000600000400000C00000C00001C000018000018000038000038000038000038000 078000078000078000078000078000078000030000111D7E9B15>I<03E00C301008200C200660 06600660067006780C3E083FB01FE007F007F818FC307E601E600FC007C003C003C003C0036002 6004300C1C1007E0101D7E9B15>I<03C00C301818300C700C600EE006E006E007E007E007E007 E0076007700F300F18170C2707C700060006000E300C780C78187010203030C00F80101D7E9B15 >I<60F0F0600000000000000000000060F0F06004127C910C>I<60F0F060000000000000000000 0060F0F0701010101020204080041A7C910C>I<0FE03038401CE00EF00EF00EF00E000C001C00 30006000C000800180010001000100010001000100000000000000000000000300078007800300 0F1D7E9C14>63 D<000600000006000000060000000F0000000F0000000F000000178000001780 00001780000023C0000023C0000023C0000041E0000041E0000041E0000080F0000080F0000180 F8000100780001FFF80003007C0002003C0002003C0006003E0004001E0004001E000C001F001E 001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E06180018019 80070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F000 0000F0000000F0000000F0000000F0000000F0000000F000000070000080780000807800008038 0000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>I III<001F808000E0618001801980 070007800E0003801C0003801C00018038000180780000807800008070000080F0000000F00000 00F0000000F0000000F0000000F0000000F000FFF0F0000F807000078078000780780007803800 07801C0007801C0007800E00078007000B800180118000E06080001F80001C1E7E9C21>II< FFF00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F 000F000F000F000F000F000F000F00FFF00C1C7F9B0F>I76 DII<003F800000E0 E0000380380007001C000E000E001C0007003C00078038000380780003C0780003C0700001C0F0 0001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C0 780003C0380003803C0007801C0007000E000E0007001C000380380000E0E000003F80001B1E7E 9C20>II82 D<07E0801C1980300580700380600180 E00180E00080E00080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F80 0007C00003C00001C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E 9C17>I<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F00 20000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003FFFC001B 1C7F9B1E>IIII<7FF0FFC00FC03E000780180003C0180003E0100001E0200001 F0600000F0400000788000007D8000003D0000001E0000001F0000000F0000000F8000000F8000 0013C0000023E0000021E0000041F00000C0F8000080780001007C0003003C0002001E0006001F 001F003F80FFC0FFF01C1C7F9B1F>II91 D<0808101020204040404080 8080808080B0B0F8F8787830300D0C7A9C15>I<1FC000307000783800781C00301C00001C0000 1C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F870012127E91 15>97 DI<07E00C301878307870306000E000E000E000E0 00E000E00060007004300418080C3007C00E127E9112>I<003F00000700000700000700000700 00070000070000070000070000070000070003E7000C1700180F00300700700700600700E00700 E00700E00700E00700E00700E00700600700700700300700180F000C370007C7E0131D7E9C17> I<03E00C301818300C700E6006E006FFFEE000E000E000E00060007002300218040C1803E00F12 7F9112>I<00F8018C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38 C01C3880181800381C00381C00381C00381C001818001C38000C300013C0001000003000001800 001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007F800121C 7F9215>II<18003C003C00180000000000000000000000 00000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF 80091D7F9C0C>I<00C001E001E000C000000000000000000000000000000FE000E000E000E000 E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0F0C0F1C0 61803E000B25839C0D>IIIII<03F0000E1C00180600300300700380600180E001C0E001 C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F9115>II<03C1000C3300180B00300F00700700700700E00700E00700E00700E00700E00700 E00700600700700700300F00180F000C370007C700000700000700000700000700000700000700 000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE00FF000F8 80388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C00FFE0 1C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C1A7F9910 >IIII<7F8FF00F03800F030007020003840001C80001D80000F00000700000780000 F800009C00010E00020E000607000403801E07C0FF0FF81512809116>II<7FFC 70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E127F9112 >I124 D E /Fi 24 118 df<0003E0000000000FF0000000003E38 00000000781800000000780C00000000F80C00000000F00C00000001F00C00000001F00C000000 01F01C00000001F81800000001F83000000001F87000000001F86000000001F8C001FFE000FD80 01FFE000FF00001C0000FE0000180000FE00003000007E00003000007F0000600000FF00006000 01FF8000C00003BF80018000071FC00180000F0FE00300001E0FE00600003E07F00600007E03F8 0C0000FE03FC180000FE01FE300000FE00FE600000FE007FC00000FE003F8000C07F001FC000C0 7F000FF001C03F801FF803801FC0F0FE0F0007FFC03FFE0001FE0007F8002B287DA732>38 D<3C7EFFFFFFFF7E3C08087B8712>46 D<000C00001C0000FC000FFC00FFFC00F0FC0000FC0000 FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000 FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000 FC0000FC00FFFFFCFFFFFC16257BA420>49 D<00FF000007FFE0001E07F8003801FC007800FE00 7E00FF00FF007F00FF007F80FF007F80FF003F807E003F803C003F8000007F8000007F0000007F 000000FE000000FE000001FC000001F8000003F0000007C00000078000000F0000001E00000038 00000070018000E0018001C001800380030007000300060003000FFFFF001FFFFF003FFFFF007F FFFE00FFFFFE00FFFFFE0019257DA420>I<00FF800007FFF0000F03F8001800FC003E00FE007F 00FF007F007F007F807F007F007F003F00FF001E00FE000000FE000000FC000001F8000003F000 0007E00000FF800000FF00000003E0000001F8000000FC000000FE0000007F0000007F0000007F 8018007F807E007F80FF007F80FF007F80FF007F80FF007F00FE00FF007C00FE003801FC001E03 F8000FFFE00001FF000019257DA420>I<0000380000007800000078000000F8000001F8000003 F8000003F8000007F800000FF800001DF8000019F8000031F8000071F8000061F80000C1F80001 C1F8000381F8000301F8000601F8000E01F8001C01F8001801F8003001F8007001F800E001F800 FFFFFFE0FFFFFFE00001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8 00007FFFE0007FFFE01B257EA420>I<0000FF80080007FFF018003FC03C38007E000E7801FC00 03F803F00001F807E00000F80FE00000781FC00000781FC00000383F800000383F800000387F80 0000187F000000187F00000018FF00000000FF00000000FF00000000FF00000000FF00000000FF 00000000FF00000000FF00000000FF00000000FF000000007F000000007F000000187F80000018 3F800000183F800000181FC00000301FC00000300FE000006007E000006003F00000C001FC0001 80007E000700003FC03C000007FFF8000000FFC00025287CA72E>67 DI73 D78 D 80 D<03FF00000FFFE0001F03F0003F80F8003F80FC003F807C001F007E001F007E0000007E00 00007E0000007E00000FFE0001FFFE0007F07E001FC07E003F807E007F007E00FE007E00FE007E 18FE007E18FE007E18FE00BE187F01BE183F873FF01FFE1FE003F80F801D1A7E9920>97 D<003FE001FFF807E07C0FC0FE1F80FE3F00FE3F007C7E007C7E0000FE0000FE0000FE0000FE00 00FE0000FE0000FE0000FE00007E00007F00003F00003F80031F80070FC00607F01C01FFF0003F C0181A7E991D>99 D<00007FE000007FE0000007E0000007E0000007E0000007E0000007E00000 07E0000007E0000007E0000007E0000007E0000007E0000007E0007F87E001FFE7E007E077E00F C01FE01F800FE03F0007E03F0007E07E0007E07E0007E0FE0007E0FE0007E0FE0007E0FE0007E0 FE0007E0FE0007E0FE0007E0FE0007E07E0007E07E0007E07F0007E03F0007E01F000FE00F801F E007E077E003FFC7FE007F07FE1F287EA724>I<007F8003FFE007E1F00F80F81F007C3F007E7E 003E7E003E7E003FFE003FFE003FFFFFFFFFFFFFFE0000FE0000FE0000FE00007E00007E00003F 00003F00031F80060FC00607F01C01FFF0003FC0181A7E991D>I<0F001F801FC03FC03FC01FC0 1F800F000000000000000000000000000000FFC0FFC00FC00FC00FC00FC00FC00FC00FC00FC00F C00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC00FC0FFFCFFFC0E297FA811>105 D108 D110 D<007FC00001FFF00007E0FC000F803E001F001F003F001F803E000F807E000F C07E000FC0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE0FE000FE07E00 0FC07E000FC07F001FC03F001F801F001F000F803E0007E0FC0001FFF000007FC0001B1A7E9920 >I I114 D<03F8C01FFFC03C07C07801C07001C0F000C0F000C0F800C0FC0000FFC0 007FFC003FFF001FFF800FFFC001FFE0000FF00003F0C001F0C000F0E000F0E000F0F000E0F801 E0FE03C0E7FF80C1FC00141A7E9919>I<00600000600000600000600000E00000E00001E00001 E00003E00007E0001FE000FFFFC0FFFFC007E00007E00007E00007E00007E00007E00007E00007 E00007E00007E00007E00007E00007E00007E00007E06007E06007E06007E06007E06007E06003 F0C001F0C000FF80003E0013257FA419>II E /Fj 9 116 df73 D80 D86 D<0007FFC0000000003FFFFC00 000000FFFFFF00000003F801FFC0000007F0003FE0000007F8001FF000000FFC000FF800000FFC 000FFC00000FFC0007FC00000FFC0007FE00000FFC0003FE000007F80003FF000003F00003FF00 0000000003FF000000000003FF000000000003FF000000000003FF000000000003FF0000000000 03FF000000000003FF0000000007FFFF00000000FFFFFF0000000FFFE3FF0000007FF803FF0000 01FFC003FF000003FF0003FF000007FC0003FF00000FF80003FF00001FF00003FF00003FF00003 FF00007FE00003FF00007FE00003FF0380FFC00003FF0380FFC00003FF0380FFC00003FF0380FF C00003FF0380FFC00007FF0380FFC00007FF03807FE0000DFF03807FE0001DFF03803FF00039FF 87001FF80070FFCF000FFE03E07FFE0007FFFF807FFC0000FFFE001FF800001FF00007E000312E 7CAD37>97 D<00FF80FFFF80FFFF80FFFF8003FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF80 01FF8001FF8001FF8001FF8001FF8001FF80FFFFFFFFFFFFFFFFFF18487DC71F>108 D<00003FFC00000001FFFF8000000FFFFFF000003FF00FFC00007F8001FE0001FE00007F8003FC 00003FC007F800001FE007F800001FE00FF000000FF01FE0000007F81FE0000007F83FE0000007 FC3FE0000007FC7FC0000003FE7FC0000003FE7FC0000003FE7FC0000003FEFFC0000003FFFFC0 000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003FFFFC0000003 FFFFC0000003FFFFC0000003FF7FC0000003FE7FC0000003FE7FC0000003FE7FE0000007FE3FE0 000007FC3FE0000007FC1FE0000007F81FF000000FF80FF000000FF007F800001FE007FC00003F E003FE00007FC001FF0000FF80007F8001FE00003FF00FFC00000FFFFFF0000003FFFFC0000000 3FFC0000302E7DAD37>111 D<00FF801FF80000FFFF80FFFF0000FFFF83FFFFC000FFFF8FC07F F00003FF9E000FF80001FFF80007FC0001FFF00003FE0001FFE00001FF0001FFC00000FF8001FF 800000FFC001FF8000007FC001FF8000007FE001FF8000007FE001FF8000003FE001FF8000003F F001FF8000003FF001FF8000003FF001FF8000001FF801FF8000001FF801FF8000001FF801FF80 00001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF801FF8000001FF8 01FF8000001FF801FF8000001FF801FF8000001FF001FF8000003FF001FF8000003FF001FF8000 003FF001FF8000003FE001FF8000007FE001FF8000007FC001FF800000FFC001FF800000FF8001 FFC00001FF8001FFE00001FF0001FFF00003FE0001FFF8000FFC0001FF9E001FF80001FF8F80FF E00001FF87FFFFC00001FF81FFFE000001FF803FF0000001FF800000000001FF800000000001FF 800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80000000 0001FF800000000001FF800000000001FF800000000001FF800000000001FF800000000001FF80 0000000001FF800000000001FF800000000001FF8000000000FFFFFF00000000FFFFFF00000000 FFFFFF0000000035427DAD3D>I<00FF007F00FFFF01FFC0FFFF03FFE0FFFF0787F003FF0E0FF0 01FF1C1FF801FF381FF801FF301FF801FF701FF801FF600FF001FF600FF001FFE003C001FFC000 0001FFC0000001FFC0000001FFC0000001FF80000001FF80000001FF80000001FF80000001FF80 000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF 80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001 FF80000001FF80000001FF80000001FF80000001FF80000001FF80000001FF800000FFFFFFC000 FFFFFFC000FFFFFFC000252E7DAD2C>114 D<001FFC030000FFFF870003FFFFCF000FE007FF00 1F8000FF003E00003F003E00001F007C00001F007C00000F00FC00000F00FC00000700FC000007 00FE00000700FF00000700FF80000000FFE00000007FFE0000007FFFF800003FFFFF00003FFFFF C0001FFFFFF0000FFFFFF80007FFFFFC0001FFFFFE00007FFFFF00001FFFFF000000FFFF800000 07FF80000000FFC0E000007FC0E000003FC0E000001FC0F000000FC0F000000FC0F800000FC0F8 00000FC0F800000F80FC00000F80FE00001F80FF00001F00FF80003E00FFC0007C00F9F803F800 F0FFFFF000E03FFFC000C007FE0000222E7CAD2B>I E /Fk 8 117 df<00003000000070000001 F0000007F000007FF000FFFFF000FFFFF000FF8FF000000FF000000FF000000FF000000FF00000 0FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000 000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF0 00000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000F F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00000 0FF0007FFFFFFE7FFFFFFE7FFFFFFE1F3779B62D>49 D<000000FFE0001800000FFFFC00180000 7FFFFF00380001FFE00FC0780007FE0001F0F8000FF8000079F8003FE000003FF8007FC000000F F800FF80000007F801FF00000007F803FE00000003F807FC00000001F807F800000001F80FF800 000000F80FF000000000F81FF000000000781FF000000000783FE000000000783FE00000000038 7FE000000000387FE000000000387FE000000000387FC000000000007FC00000000000FFC00000 000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FFC00000000000FF C00000000000FFC00000000000FFC00000000000FFC00000000000FFC000000000007FC0000000 00007FC000000000007FE000000000007FE000000000387FE000000000383FE000000000383FE0 00000000381FF000000000381FF000000000700FF000000000700FF8000000007007F800000000 E007FC00000000E003FE00000001C001FF000000038000FF8000000780007FC000000F00003FE0 00001E00000FF800003C000007FE0000F0000001FFE007E00000007FFFFF800000000FFFFE0000 000000FFE00000353B7BB940>67 D<003FFC00000001FFFF80000007E00FE000000FC003F80000 0FE001FC00001FF001FE00001FF000FF00001FF000FF00001FF0007F00000FE0007F800007C000 7F80000000007F80000000007F80000000007F80000000007F80000000007F800000000FFF8000 0007FFFF8000003FE07F800001FF007F800007FC007F80000FF0007F80001FE0007F80003FE000 7F80007FC0007F80007FC0007F8380FF80007F8380FF80007F8380FF80007F8380FF8000BF8380 FF8000BF83807FC0013F83807FC0033F83803FE0061FC7001FF81C0FFE0007FFF007FC00007FC0 03F00029257DA42D>97 D<0003FF0000001FFFE000007F07F00001FC01FC0003F000FE0007E000 7F000FE0003F001FC0003F801FC0003F803FC0001FC03F80001FC07F80001FC07F80001FE07F80 001FE0FF80001FE0FF80001FE0FFFFFFFFE0FFFFFFFFE0FF80000000FF80000000FF80000000FF 80000000FF800000007F800000007F800000007F800000003FC00000003FC00000001FC00000E0 1FE00000E00FE00001C007F000038003F800070000FE000E00007FC07C00001FFFF0000001FF80 0023257EA428>101 D<01FC00000000FFFC00000000FFFC00000000FFFC0000000007FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC01FF000003FC07FFC00003FC1C0FF00003FC 3007F80003FC6003F80003FCC003FC0003FC8001FC0003FD0001FE0003FF0001FE0003FE0001FE 0003FE0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE 0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC 0001FE0003FC0001FE0003FC0001FE0003FC0001FE0003FC0001FE00FFFFF07FFFF8FFFFF07FFF F8FFFFF07FFFF82D3A7EB932>104 D<01FC03FE0000FFFC1FFFC000FFFC780FF000FFFDE003F8 0007FF8001FC0003FF0000FE0003FE0000FF0003FC00007F8003FC00007FC003FC00003FC003FC 00003FE003FC00003FE003FC00001FE003FC00001FE003FC00001FF003FC00001FF003FC00001F F003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC00001FF003FC 00001FE003FC00003FE003FC00003FE003FC00003FC003FC00003FC003FC00007F8003FC00007F 8003FE0000FF0003FF0001FE0003FF8003FC0003FDC007F80003FCF81FE00003FC3FFF800003FC 07FC000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC000000 0003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC 00000000FFFFF0000000FFFFF0000000FFFFF00000002C357EA432>112 D<01F80FC0FFF83FF0FFF870F8FFF8C1FC07F883FE03F983FE03F903FE03FB03FE03FA01FC03FA 00F803FA007003FE000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC0000 03FC000003FC000003FC000003FC000003FC0000FFFFF800FFFFF800FFFFF8001F257EA424> 114 D<001C0000001C0000001C0000001C0000001C0000003C0000003C0000003C0000003C0000 007C0000007C000000FC000000FC000001FC000003FC000007FC00001FFFFFC0FFFFFFC0FFFFFF C003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC 000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003FC000003 FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E003FC00E001FC00C001FC01C0 01FE01C000FE0380007F8700001FFE000007F8001B357EB423>116 D E /Fl 20 122 df<70F8FCFC7404040404080810102040060F7C840E>44 D<008003800F80F38003 800380038003800380038003800380038003800380038003800380038003800380038003800380 038003800380038003800380038007C0FFFE0F217CA018>49 D<03F0000C1C0010070020078040 03C04003C08003E0F003E0F801E0F801E0F801E02003E00003E00003C00003C000078000070000 0E00001C0000180000300000600000C0000180000100000200200400200800201800603000403F FFC07FFFC0FFFFC013217EA018>I<03F8000C1E001007002007804007C07807C07803C07807C0 3807C0000780000780000700000F00000E0000380003F000001C00000F000007800007800003C0 0003C00003E02003E07003E0F803E0F803E0F003C04003C0400780200780100F000C1C0003F000 13227EA018>I<01F000060C000C0600180700380380700380700380F001C0F001C0F001C0F001 E0F001E0F001E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001C00001 C00001C0000380000380300300780700780600700C002018001030000FC00013227EA018>57 D<0007E0100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C00 00303C0000307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F800000078000000780000107C0000103C0000103C0000101C0000201E000020 0E000040070000400380008001C0010000E0020000381C000007E0001C247DA223>67 D<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00700F00F80F00F80F00F80E00F01E00401C0020380018700007C00014237EA119>74 D76 DI<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7 001E07003C0700780700700700F00708F00708F00708F00F087817083C23900FC1E015157E9418 >97 D<01FE000703000C07801C0780380300780000700000F00000F00000F00000F00000F00000 F00000F000007000007800403800401C00800C010007060001F80012157E9416>99 D<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000 E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I<01FC0007 07000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000F0000070 00007800203800201C00400E008007030000FC0013157F9416>I<0E0000FE00001E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E0 0F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E0070FFE7FF18237FA21B>104 D<0E0000FE00001E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01C00E0180 0E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C0 0E01E00E00F00E00F8FFE3FE17237FA21A>107 D<0E00FE001E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E00FFE00B237FA20E>I<0E1F80FE60C01E80E00F00700F00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E 0070FFE7FF18157F941B>110 D<01FC000707000C01801800C03800E0700070700070F00078F0 0078F00078F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC0015 157F9418>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E 000E000E000F00FFF010157F9413>114 D 121 D E /Fm 24 121 df<00003FC0020001FFF8060007E01C06001F00070E003C00018E007800 00DE00F000005E01E000003E03C000001E078000001E0F0000000E0F0000000E1E000000061E00 0000063E000000063C000000027C000000027C000000027C000000027800000000F800000000F8 00000000F800000000F800000000F800000000F800000000F800000000F800000000F800000000 F80000000078000000007C000000007C000000027C000000023C000000023E000000021E000000 021E000000040F000000040F0000000C078000000803C000001801E000001000F0000020007800 0040003C000180001F0003000007E01E000001FFF80000003FC00027327CB02F>67 D70 D73 D77 D80 D82 D<007F004001FFC0400780F0C00F0018C01C000DC03C0007C0380003C0780001 C0700001C0F00000C0F00000C0F00000C0F0000040F0000040F0000040F8000040F80000007C00 00007E0000003F0000003FE000001FFE00000FFFE00007FFF80001FFFE00007FFF000007FF8000 007F8000000FC0000007E0000003E0000003E0000001F0000001F0800000F0800000F0800000F0 800000F0800000F0C00000F0C00000E0E00001E0E00001E0F00001C0F8000380EC000780C7000F 00C1E03E0080FFF800801FE0001C327CB024>I86 D<00FE0000070380000801E0001000F0003C0078003E0078003E003C003E 003C001C003C0000003C0000003C0000003C000007FC00007C3C0003E03C0007803C001F003C00 3E003C003C003C007C003C0078003C08F8003C08F8003C08F8003C08F8007C08F8007C087C00BC 083E011E100F060FE003F807C01D1E7D9D20>97 D<018000003F800000FF800000FF8000000F80 000007800000078000000780000007800000078000000780000007800000078000000780000007 800000078000000780000007800000078000000781F80007860700079803C007A000E007C000F0 07C00078078000380780003C0780003E0780001E0780001E0780001F0780001F0780001F078000 1F0780001F0780001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C0 0070074000F0072001C006100380060C0E000403F80020317EB024>I<001FC00000F0380001C0 0400078002000F000F001E001F001E001F003C001F007C000E007C00000078000000F8000000F8 000000F8000000F8000000F8000000F8000000F8000000F8000000780000007C0000007C000000 3C0000801E0000801E0001000F0001000780020001C00C0000F03000001FC000191E7E9D1D>I< 003F800000E0F0000380380007001C000E001E001E000E003C000F003C000F007C000F80780007 8078000780F8000780FFFFFF80F8000000F8000000F8000000F8000000F8000000F80000007800 0000780000007C0000003C0000801C0000801E0001000F0002000700020001C00C0000F0300000 1FC000191E7E9D1D>101 D<07000F801F801F800F800700000000000000000000000000000000 0000000000000001801F80FF80FF800F8007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800FC0FFF8FFF80D2F7EAE12>105 D<01803F80FF80FF800F8007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 8007800780078007800780078007800FC0FFFCFFFC0E317EB012>108 D<0181FE003FC0003F86 0780C0F000FF8801C1003800FF9001E2003C000FA000E4001C0007A000F4001E0007C000F8001E 0007C000F8001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E00078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FF FC1FFF83FFF0341E7E9D38>I<0181FC003F860F00FF880380FF9003C00FA001C007C001E007C0 01E007C001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E007 8001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0 078001E00FC003F0FFFC3FFFFFFC3FFF201E7E9D24>I<003FC00000E0700003801C0007000E00 0E0007001E0007803C0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001 F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E0780001E07C0003E03C0003C03C00 03C01E0007800E00070007000E0003801C0000E07000003FC0001C1E7E9D20>I<0181F8003F86 0F00FF9803C0FFA001E007C000F007C00078078000780780003C0780003E0780003E0780001E07 80001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001E0780003E 0780003C0780003C0780007807C000F007C000F007A001C007900380078C0E000783F800078000 000780000007800000078000000780000007800000078000000780000007800000078000000780 00000FC00000FFFC0000FFFC0000202C7E9D24>I<0183E03F8C18FF907CFF907C0FA07C07C038 07C00007C00007C000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000780000780000FC000FFFE00FFFE00161E7E9D19> 114 D<03FC200E02603801E03000E0600060E00060E00020E00020F00020F000207C00007F8000 3FFC001FFF0007FF8001FFE0000FE00001F08000F8800078800038C00038C00038C00038E00030 F00070F00060C800C0C6038081FE00151E7E9D19>I<00400000400000400000400000400000C0 0000C00000C00001C00001C00003C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C01003C0 1003C01003C01003C01003C01003C01003C01001C02001E02000E0400078C0001F00142B7FAA19 >I<018000603F800FE0FF803FE0FF803FE00F8003E0078001E0078001E0078001E0078001E007 8001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0078001E0 078001E0078001E0078001E0078003E0078003E0078003E0038005E003C009E001C019F000F021 FF001FC1FF201E7E9D24>II120 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300dpi TeXDict begin %%PaperSize: A4 %%EndSetup %%Page: 0 1 0 0 bop 575 1056 a Fm(MPI)23 b(Con)n(text)f(Sub)r(committee)807 1147 y(Prop)r(osal)h(VI)778 1238 y(First)f(Revision)799 1421 y Fl(Lyndon)17 b(J)f(Clark)o(e)814 1541 y(Marc)o(h)f(23,)i(1993)p eop %%Page: 1 2 1 1 bop 262 619 a Fk(Chapter)28 b(1)262 826 y Fj(Prop)s(osal)36 b(VI)262 1067 y Fi(1.1)64 b(In)n(tro)r(duction)262 1158 y Fh(This)9 b(c)o(hapter)h(prop)q(oses)g(that)g(comm)o(unicatio)o(n)d(con)o(texts)j(and)f (pro)q(cess)i(groupings)e(within)262 1208 y Fg(mpi)14 b Fh(app)q(ear)h(as)f (lo)q(osely)g(coupled)h(concepts.)22 b(One)15 b(or)g(more)e(comm)o(unication) f(con)o(texts)262 1257 y(ma)o(y)d(b)q(e)j(asso)q(ciated)f(with)g(eac)o(h)h (pro)q(cess)h(grouping,)d(and)h(eac)o(h)h(comm)o(unication)c(con)o(text)262 1307 y(inherits)k(prop)q(erties)h(of)e(the)h(asso)q(ciated)h(pro)q(cess)g (grouping.)k(This)11 b(re\015ects)j(the)f(observ)n(a-)262 1357 y(tions)f(that)g(in)o(v)o(o)q(cations)f(of)h(mo)q(dules)f(in)h(a)g(parallel)f (program)g(t)o(ypically)g(op)q(erate)i(within)262 1407 y(pro)q(cess)19 b(groupings,)e(and)h(that)f(there)i(ma)o(y)c(b)q(e)j(m)o(ultiple)e(mo)q (dules)g(op)q(erating)h(within)262 1457 y(eac)o(h)d(pro)q(cess)i(grouping.) 324 1506 y(The)k(prop)q(osal)f(pro)o(vides)g(pro)q(cess)i(iden)o(ti\014ed)f (comm)o(unicatio)o(n,)e(comm)o(unicati)o(ons)262 1556 y(whic)o(h)d(are)g (limited)e(in)i(scop)q(e)h(to)f(single)g(con)o(texts,)h(and)f(comm)o (unicatio)o(ns)e(whic)o(h)i(ha)o(v)o(e)262 1606 y(scop)q(e)g(spanning)e (pairs)h(of)f(con)o(texts.)19 b(The)c(prop)q(osal)e(mak)o(es)g(no)h(statemen) o(ts)g(regarding)262 1656 y(message)c(tags.)16 b(It)11 b(is)f(assumed)g(that) g(these)i(will)d(b)q(e)i(a)f(bit)g(string)g(expressed)j(as)d(an)g(in)o(teger) 262 1706 y(in)j(the)h(host)h(language.)324 1756 y(Muc)o(h)d(of)e(this)i(prop) q(osal)f(m)o(ust)f(b)q(e)i(view)o(ed)f(as)h(recommendations)d(to)i(other)h (sub)q(com-)262 1805 y(mittees)i(of)f Fg(mpi)p Fh(,)h(primarily)e(the)j(p)q (oin)o(t-to-p)q(oin)o(t)e(comm)o(unicatio)o(n)f(sub)q(committee)h(and)262 1855 y(the)k(collectiv)o(e)f(comm)o(unicatio)o(ns)e(sub)q(committee.)25 b(Concrete)17 b(syn)o(tax)g(is)f(giv)o(en)g(in)g(the)262 1905 y(st)o(yle)e(of)f(the)h(ANSI)h(C)e(host)i(language,)d(only)h(for)h(purp)q (oses)h(of)e(discussion.)324 1955 y(The)h(detailed)f(prop)q(osal)g(is)g (presen)o(ted)j(in)c(Section)i(1.2,)e(whic)o(h)h(refers)i(the)f(reader)h(to) 262 2005 y(a)e(set)h(of)f(discussion)h(notes)h(in)e(Section)h(1.3.2.)i(The)e (note)g(assumes)g(forw)o(ard)f(kno)o(wledge)262 2054 y(of)h(the)h(prop)q (osal)f(text)h(and)g(are)g(therefore)h(b)q(est)f(examined)f(after)h(famili)o (arisatio)o(n)d(with)262 2104 y(the)f(prop)q(osal)g(text.)18 b(Asp)q(ects)13 b(of)e(the)h(prop)q(osal)f(are)g(discussed)i(in)e(section)h (Section)g(1.3.1,)262 2154 y(and)g(it)g(is)h(also)f(recommended)g(that)h (this)f(material)f(b)q(e)i(read)h(after)e(familiarisati)o(on)e(with)262 2204 y(the)k(prop)q(osal)g(text.)967 2574 y(1)p eop %%Page: 2 3 2 2 bop 262 307 a Fi(1.2)64 b(Detailed)24 b(Prop)r(osal)262 398 y Fh(This)14 b(section)g(presen)o(ts)i(the)f(detailed)f(prop)q(osal,)g (describing,)g(in)g(order)g(of)g(app)q(earance:)262 448 y(pro)q(cesses;)19 b(pro)q(cess)g(groupings;)d(comm)o(unication)d(con)o(texts;)18 b(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(u-)262 498 y(nication;)c(collectiv)o(e) i(comm)o(unication.)262 614 y Ff(1.2.1)55 b(Pro)r(cesses)262 691 y Fh(This)13 b(prop)q(osal)h(views)f(pro)q(cesses)k(in)c(the)i(famili)o (ar)c(w)o(a)o(y)m(,)h(as)i(one)g(thinks)g(of)f(pro)q(cesses)j(in)262 740 y(Unix)f(or)i(NX)f(for)g(example.)24 b(Eac)o(h)16 b(pro)q(cess)i(is)e(a)g (distinct)h(space)g(of)f(instructions)h(and)262 790 y(data.)h(Eac)o(h)c(pro)q (cess)i(is)e(allo)o(w)o(ed)e(to)i(comp)q(ose)g(m)o(ultiple)d(concurren)o(t)16 b(threads)f(and)f Fg(mpi)262 840 y Fh(do)q(es)g(not)g(distinguish)f(suc)o(h)i (threads.)262 948 y Fe(Pro)q(cess)g(Descriptor)262 1025 y Fh(Eac)o(h)d(pro)q (cess)i(is)e(describ)q(ed)i(b)o(y)e(a)f Fd(pr)n(o)n(c)n(ess)i(descriptor)j Fh(\(or)c Fd(hand)r(le)p Fh(\))h(whic)o(h)f(is)g(expressed)262 1074 y(as)f(an)f(in)o(teger)i(t)o(yp)q(e)f(in)g(the)h(host)f(language)f(and)h (has)g(an)g(opaque)g(v)n(alue)f(whic)o(h)h(is)g(pro)q(cess)262 1124 y(lo)q(cal.)17 b(See)d(Note)h(1.)324 1174 y(The)f(initialisation)d(of)i Fg(mpi)g Fh(services)j(will)c(assign)h(to)h(eac)o(h)g(pro)q(cess)h(an)f Fd(own)j Fh(pro)q(cess)262 1224 y(descriptor.)h(Eac)o(h)11 b(pro)q(cess)i(retains)e(its)g(o)o(wn)g(pro)q(cess)i(descriptor)f(un)o(til)e (the)h(termination)262 1274 y(of)j Fg(mpi)h Fh(services.)23 b Fg(mpi)15 b Fh(pro)o(vides)g(a)g(pro)q(cedure)i(whic)o(h)e(returns)h(the)g (o)o(wn)f(descriptor)h(of)262 1323 y(the)e(calling)e(pro)q(cess.)20 b(F)m(or)14 b(example,)e Fc(pd)21 b(=)h(mpi)p 1052 1323 14 2 v 15 w(own)p 1133 1323 V 15 w(process\(\))n Fh(.)262 1431 y Fe(Pro)q(cess)15 b(Creation)f(&)i(Destruction)262 1508 y Fh(This)f(prop)q(osal)g(mak)o(es)f(no)h(statemen)o(ts)h(regarding)f(creation) h(and)f(destruction)i(of)e(pro-)262 1558 y(cesses.)20 b(See)15 b(Note)f(2.)262 1666 y Fe(Descriptor)f(T)l(ransmission)262 1742 y Fh(Since)19 b(a)g(descriptor)h(is)f(an)f(in)o(teger)i(t)o(yp)q(e)f(in) f(the)i(host)f(language)f(the)i(v)n(alue)e(ma)o(y)f(b)q(e)262 1792 y(transmitted)f(in)g(a)g(message)h(as)g(an)f(in)o(teger.)27 b(The)17 b(recipien)o(t)h(of)e(the)h(descriptor)h(v)n(alue)262 1842 y(can)f(mak)o(e)f(no)h(de\014ned)h(use)g(of)f(the)h(v)n(alue)e(in)h(the) h Fg(mpi)f Fh(op)q(erations)h(describ)q(ed)h(in)e(this)262 1892 y(prop)q(osal)c(|)g(the)i(descriptor)g(is)f Fd(invalid)p Fh(.)324 1942 y Fg(mpi)j Fh(pro)o(vides)h(a)g(mec)o(hanism)d(whereb)o(y)k (the)f(user)h(can)f(transmit)f(a)h(v)n(alid)e(pro)q(cess)262 1991 y(descriptor)i(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 2041 y(tegrated)h(with)f(the)h(capabilit)o(y)e(to)h(transmit)f(t)o(yp)q(ed)i (messages.)26 b(It)17 b(is)f(suggested)i(that)262 2091 y(a)d(notional)f(data) h(t)o(yp)q(e)h(should)f(b)q(e)h(in)o(tro)q(duced)g(for)f(this)h(purp)q(ose,)g (e.g.)23 b Fc(MPI)p 1524 2091 V 15 w(PD)p 1583 2091 V 15 w(TYPE)o Fh(.)262 2141 y(See)14 b(Note)h(3.)324 2191 y Fg(mpi)f Fh(pro)o(vides)g(a)g (pro)q(cess)i(descriptor)f(registry)g(service.)20 b(Use)15 b(of)f(this)g(service)i(is)e(not)262 2240 y(mandated.)27 b(Programs)16 b(whic)o(h)h(can)g(con)o(v)o(enien)o(tly)h(b)q(e)f(expressed)j(without)d (using)g(the)262 2290 y(service)e(can)f(ignore)g(it)f(without)h(p)q(enalt)o (y)m(.)j(See)e(Note)f(4.)324 2340 y(Note)d(that)g(receipt)h(of)e(a)g(v)n (alid)g(pro)q(cess)i(descriptor)g(ma)o(y)d(ha)o(v)o(e)h(a)h(p)q(ersisten)o(t) h(e\013ect)h(on)262 2390 y(the)h(implemen)o(tation)d(of)i Fg(mpi)h Fh(at)f(the)i(receiv)o(er,)g(and)f(in)g(particular)f(ma)o(y)f(reserv)o(e)k (state.)967 2574 y(2)p eop %%Page: 3 4 3 3 bop 262 307 a Fg(mpi)9 b Fh(will)e(pro)o(vide)j(a)f(pro)q(cedure)i(whic)o (h)e(frees)i(\(or)e(in)o(v)n(alidates\))f(a)h(v)n(alid)f(descriptor,)j(allo)o (w-)262 357 y(ing)j(the)h(implemen)o(tatio)o(n)d(to)j(free)g(reserv)o(ed)i (state.)22 b(F)m(or)14 b(example,)f Fc(mpi)p 1436 357 14 2 v 15 w(free)p 1539 357 V 15 w(pd\(pd\))o Fh(.)262 407 y(The)g(user)g(is)g (not)g(allo)o(w)o(ed)e(to)h(free)i(the)f(pro)q(cess)i(descriptor)f(of)e(the)h (calling)e(pro)q(cess.)20 b(See)262 457 y(Note)14 b(5.)324 506 y(See)h(Note)f(6.)262 614 y Fe(Descriptor)f(A)o(ttribu)o(tes)262 691 y Fh(This)g(prop)q(osal)h(mak)o(es)f(no)g(statemen)o(ts)h(regarding)g (pro)q(cess)i(descriptor)f(attributes.)262 807 y Ff(1.2.2)55 b(Pro)r(cess)18 b(Groupings)262 884 y Fh(This)d(prop)q(osal)h(views)g(a)f (pro)q(cess)j(grouping)d(as)h(an)g(ordered)h(collection)e(of)h(\(references) 262 934 y(to\))f(distinct)h(pro)q(cesses,)i(the)e(mem)o(b)q(ership)e(and)h (ordering)g(of)g(whic)o(h)h(do)q(es)g(not)f(c)o(hange)262 983 y(o)o(v)o(er)g(the)g(lifetime)e(of)h(the)i(grouping.)21 b(See)15 b(Note)h(7.)21 b(The)15 b(canonical)g(represen)o(tation)h(of)262 1033 y(a)g(grouping)h(re\015ects)i(the)f(pro)q(cess)h(ordering)e(and)g(is)g (a)g(one-to-one)g(map)e(from)h Fb(Z)1608 1039 y Fa(N)1657 1033 y Fh(to)262 1083 y(descriptors)f(of)e(the)i Fb(N)k Fh(pro)q(cesses)d(comp)q (osing)c(the)j(grouping.)324 1133 y(There)20 b(ma)o(y)e(b)q(e)h(structure)j (asso)q(ciated)e(with)f(a)g(pro)q(cess)i(grouping)d(de\014ned)j(b)o(y)e(a)262 1183 y(pro)q(cess)g(top)q(ology)m(.)29 b(This)18 b(prop)q(osal)g(mak)o(es)f (no)h(further)h(statemen)o(ts)f(regarding)g(suc)o(h)262 1232 y(structures.)262 1340 y Fe(Group)c(Descriptor)262 1417 y Fh(Eac)o(h)e(group) g(is)h(iden)o(ti\014ed)f(b)o(y)g(a)h Fd(gr)n(oup)g(descriptor)j Fh(\(or)d Fd(hand)r(le)s Fh(\))g(whic)o(h)g(is)f(expressed)j(as)262 1467 y(an)f(in)o(teger)h(t)o(yp)q(e)g(in)f(the)h(host)f(language)g(and)g(has) h(an)f(opaque)g(v)n(alue)g(whic)o(h)g(is)h(pro)q(cess)262 1517 y(lo)q(cal.)i(See)d(Note)h(1.)324 1566 y(The)h(initialisation)d(of)j Fg(mpi)f Fh(services)j(will)c(assign)i(to)g(eac)o(h)g(pro)q(cess)i(an)e Fd(own)j Fh(group)262 1616 y(descriptor)11 b(for)f(a)g(pro)q(cess)i(grouping) d(of)h(whic)o(h)g(the)g(pro)q(cess)i(is)e(a)g(mem)o(b)q(er.)15 b(Eac)o(h)c(pro)q(cess)262 1666 y(retains)k(its)h(o)o(wn)f(group)g (descriptor)i(and)e(mem)o(b)q(ership)f(of)g(the)i(o)o(wn)f(pro)q(cess)i (grouping)262 1716 y(un)o(til)f(the)i(termination)e(of)h Fg(mpi)g Fh(services.)31 b(See)18 b(Note)g(8.)29 b Fg(mpi)17 b Fh(pro)o(vides)g(a)h (pro)q(cedure)262 1766 y(whic)o(h)d(accepts)h(a)f(v)n(alid)f(pro)q(cess)j (descriptor)f(and)f(returns)i(the)f(o)o(wn)e(group)h(descriptor)262 1816 y(of)e(the)h(iden)o(ti\014ed)g(pro)q(cess.)20 b(F)m(or)14 b(example,)e Fc(gd)21 b(=)h(mpi)p 1149 1816 V 15 w(own)p 1230 1816 V 15 w(group\(pd\))n Fh(.)c(See)d(Note)f(9.)262 1923 y Fe(Group)g(Creation)g(and)h(Deletion)262 2000 y Fg(mpi)9 b Fh(pro)o(vides)h(facilities)f(whic)o(h)h(allo)o(w)e(users)k(to)e(dynamically) d(create)k(and)f(delete)h(pro)q(cess)262 2050 y(groupings)k(in)h(addition)g (to)g(the)h(o)o(wn)f(grouping\(s\).)25 b(The)17 b(pro)q(cedures)h(describ)q (ed)g(here)262 2100 y(generate)d(groups)f(whic)o(h)g(are)g(static)g(in)g(mem) o(b)q(ership.)324 2149 y Fg(mpi)i Fh(pro)o(vides)g(a)g(pro)q(cedure)i(whic)o (h)e(allo)o(ws)f(users)i(to)f(create)i(one)e(or)g(more)f(groups)262 2199 y(whic)o(h)9 b(are)g(subsets)i(of)e(existing)g(groups.)17 b(F)m(or)9 b(example,)f Fc(gdb)21 b(=)h(mpi)p 1360 2199 V 15 w(group)p 1485 2199 V 15 w(partition\(gda,)c(key\))262 2249 y Fh(creates)g(one)f(or)g(more)f(new)h(groups)g Fc(gdb)f Fh(whic)o(h)g(are)i (distinct)f(subsets)h(of)e(an)h(existing)262 2299 y(group)d Fc(gda)f Fh(according)h(to)g(the)h(supplied)g(v)n(alues)f(of)f Fc(key)p Fh(.)19 b(This)14 b(pro)q(cedure)i(is)e(called)g(b)o(y)262 2349 y(and)f(sync)o(hronises)j(all)c(mem)o(b)q(ers)h(of)g Fc(gda)p Fh(.)324 2399 y Fg(mpi)g Fh(pro)o(vides)i(a)e(pro)q(cedure)j(whic)o(h)e(allo) o(ws)f(users)j(to)e(create)h(a)f(group)g(b)o(y)f(p)q(erm)o(uta-)262 2448 y(tion)8 b(of)h(an)g(existing)g(group.)16 b(F)m(or)9 b(example,)g Fc(gdb)21 b(=)g(mpi)p 1159 2448 V 15 w(group)p 1284 2448 V 15 w(permutation\(gda,)e(rank\))967 2574 y Fh(3)p eop %%Page: 4 5 4 4 bop 262 307 a Fh(creates)14 b(one)e(new)h(group)f(with)g(the)h(same)f (mem)o(b)q(ership)f(as)h Fc(gda)g Fh(with)g(a)g(p)q(erm)o(utation)f(of)262 357 y(pro)q(cess)k(ranking,)e(and)g(returns)i(the)g(created)g(group)e (descriptor)i(in)f Fc(gdb)p Fh(.)j(This)d(pro)q(ced-)262 407 y(ure)g(is)g(called)f(b)o(y)h(and)g(sync)o(hronises)h(all)e(mem)o(b)q(ers)f (of)i Fc(gda)p Fh(.)324 457 y Fg(mpi)19 b Fh(pro)o(vides)h(a)f(pro)q(cedure)i (whic)o(h)f(allo)o(ws)e(users)j(to)e(create)i(a)f(group)f(b)o(y)g(expli-)262 506 y(cit)d(de\014nition)h(of)f(its)h(mem)o(b)q(ership)e(as)i(a)f(list)h(of)f (pro)q(cess)j(descriptors.)28 b(F)m(or)16 b(example,)262 556 y Fc(gd)21 b(=)g(mpi)p 439 556 14 2 v 16 w(group)p 565 556 V 14 w(definition\(listofp)o(d\))10 b Fh(creates)16 b(one)d(new)h(group)f Fc(gd)g Fh(with)g(mem-)262 606 y(b)q(ership)k(and)g(ordering)g(describ)q(ed)h (b)o(y)f(the)g(pro)q(cess)i(descriptor)f(list)e Fc(listofpd)p Fh(.)25 b(This)262 656 y(pro)q(cedure)15 b(is)f(called)g(b)o(y)f(and)h(sync)o (hronises)h(all)e(pro)q(cesses)k(iden)o(ti\014ed)d(in)f Fc(listofpd)p Fh(.)324 706 y Fg(mpi)i Fh(pro)o(vides)g(a)g(pro)q(cedure)i(whic)o(h)f(allo)o (ws)e(users)i(to)f(delete)i(user)f(created)h(groups.)262 756 y(This)9 b(pro)q(cedure)i(accepts)g(the)g(descriptor)f(of)f(a)g(group)h(whic) o(h)f(w)o(as)h(created)h(b)o(y)e(the)h(calling)262 805 y(pro)q(cess)h(and)e (deletes)i(the)f(iden)o(ti\014ed)f(group.)17 b(F)m(or)9 b(example,)f Fc(mpi)p 1295 805 V 15 w(group)p 1420 805 V 15 w(deletion\(gd\))262 855 y Fh(deletes)17 b(an)e(existing)h(group)f Fc(gd)p Fh(.)23 b(This)16 b(pro)q(cedure)i(is)d(called)h(b)o(y)f(and)h(sync)o(hronises)h(all) 262 905 y(mem)o(b)q(ers)12 b(of)h Fc(gd)p Fh(.)324 955 y Fg(mpi)h Fh(ma)o(y)f(pro)o(vide)i(additional)e(pro)q(cedures)k(whic)o(h)d(allo)o(w)f (users)k(to)d(construct)j(pro-)262 1005 y(cess)e(groupings)e(with)h(a)g(pro)q (cess)h(grouping)e(top)q(ology)m(.)262 1112 y Fe(Descriptor)g(T)l (ransmission)262 1189 y Fh(Since)19 b(a)g(descriptor)h(is)f(an)f(in)o(teger)i (t)o(yp)q(e)f(in)f(the)i(host)f(language)f(the)i(v)n(alue)e(ma)o(y)f(b)q(e) 262 1239 y(transmitted)10 b(in)g(a)h(message)f(as)h(an)g(in)o(teger.)17 b(The)12 b(recipien)o(t)f(of)f(the)i(descriptor)g(can)f(mak)o(e)262 1289 y(no)j(de\014ned)h(use)h(of)e(the)h(v)n(alue)f(in)g(the)h Fg(mpi)f Fh(op)q(erations)h(describ)q(ed)h(in)e(this)h(prop)q(osal)f(|)262 1339 y(the)g(descriptor)h(is)f Fd(invalid)p Fh(.)k(See)d(Note)f(1.)324 1388 y Fg(mpi)20 b Fh(pro)o(vides)g(a)g(mec)o(hanism)e(whereb)o(y)j(the)g (user)h(can)e(transmit)f(a)h(v)n(alid)f(group)262 1438 y(descriptor)f(in)e(a) g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g(descriptor)g(is)f(v)n(alid.) 25 b(This)17 b(is)f(in-)262 1488 y(tegrated)h(with)f(the)h(capabilit)o(y)e (to)h(transmit)f(t)o(yp)q(ed)i(messages.)26 b(It)17 b(is)f(suggested)i(that) 262 1538 y(a)d(notional)f(data)h(t)o(yp)q(e)h(should)f(b)q(e)h(in)o(tro)q (duced)g(for)f(this)h(purp)q(ose,)g(e.g.)23 b Fc(MPI)p 1524 1538 V 15 w(GD)p 1583 1538 V 15 w(TYPE)o Fh(.)262 1588 y(See)14 b(Note)h(3.)324 1637 y Fg(mpi)h Fh(pro)o(vides)g(a)g(group)g(descriptor)h (registry)g(service.)26 b(Use)17 b(of)e(this)h(service)i(is)e(not)262 1687 y(mandated.)27 b(Programs)16 b(whic)o(h)h(can)g(con)o(v)o(enien)o(tly)h (b)q(e)f(expressed)j(without)d(using)g(the)262 1737 y(service)e(can)f(ignore) g(it)f(without)h(p)q(enalt)o(y)m(.)j(See)e(Note)f(4.)324 1787 y(Note)f(that)g(receipt)h(of)e(a)h(v)n(alid)e(group)i(descriptor)h(ma)o(y)d (ha)o(v)o(e)h(a)h(p)q(ersisten)o(t)h(e\013ect)h(on)262 1837 y(the)f(implemen)o(tation)d(of)i Fg(mpi)h Fh(at)f(the)i(receiv)o(er,)g(and)f (in)g(particular)f(ma)o(y)f(reserv)o(e)k(state.)262 1886 y Fg(mpi)9 b Fh(will)e(pro)o(vide)j(a)f(pro)q(cedure)i(whic)o(h)e(frees)i(\(or) e(in)o(v)n(alidates\))f(a)h(v)n(alid)f(descriptor,)j(allo)o(w-)262 1936 y(ing)j(the)h(implemen)o(tatio)o(n)d(to)j(free)g(reserv)o(ed)i(state.)22 b(F)m(or)14 b(example,)f Fc(mpi)p 1436 1936 V 15 w(free)p 1539 1936 V 15 w(gd\(gd\))o Fh(.)262 1986 y(The)i(user)h(is)f(not)f(allo)o(w)o(ed) g(to)h(free)h(the)f(o)o(wn)g(group)f(descriptor)i(of)f(the)g(calling)f(pro)q (cess)262 2036 y(or)e(the)h(group)f(descriptor)i(of)e(an)o(y)g(group)g (created)i(b)o(y)f(the)g(calling)e(pro)q(cess.)19 b(See)13 b(Note)g(5.)324 2086 y(See)i(Note)f(6.)262 2194 y Fe(Descriptor)f(A)o(ttribu) o(tes)262 2270 y Fg(mpi)d Fh(pro)o(vides)i(a)f(pro)q(cedure)i(whic)o(h)e (accepts)h(a)f(v)n(alid)f(group)h(descriptor)h(and)f(returns)i(the)262 2320 y(rank)c(of)f(the)i(calling)e(pro)q(cess)j(within)e(the)h(iden)o (ti\014ed)f(group.)16 b(F)m(or)9 b(example,)g Fc(rank)21 b(=)g(mpi)p 1691 2320 V 15 w(group)p 1816 2320 V 15 w(rank\(gd\))n Fh(.)324 2370 y Fg(mpi)e Fh(pro)o(vides)g(a)g(pro)q(cedure)i(whic)o(h)e(accepts)h(a)f (v)n(alid)f(group)h(descriptor)h(and)f(re-)262 2420 y(turns)f(the)g(n)o(um)o (b)q(er)f(of)h(mem)o(b)q(ers,)f(or)g Fd(size)p Fh(,)h(of)f(the)i(iden)o (ti\014ed)f(group.)29 b(F)m(or)17 b(example,)967 2574 y(4)p eop %%Page: 5 6 5 5 bop 262 307 a Fc(size)20 b(=)i(mpi)p 483 307 14 2 v 15 w(group)p 608 307 V 15 w(size\(gd\))n Fh(.)324 357 y Fg(mpi)16 b Fh(pro)o(vides)h(a)g(pro)q(cedure)h(whic)o(h)f(accepts)i(a)d(v)n(alid)f (group)i(descriptor)h(and)f(pro-)262 407 y(cess)22 b(order)f(n)o(um)o(b)q (er,)g(or)f Fd(r)n(ank)p Fh(,)i(and)e(returns)i(the)f(v)n(alid)e(descriptor)j (of)e(the)h(pro)q(cess)262 457 y(to)e(whic)o(h)g(the)h(supplied)g(rank)f (maps)f(within)h(the)h(iden)o(ti\014ed)g(group.)34 b(F)m(or)19 b(example,)262 506 y Fc(pd)i(=)g(mpi)p 439 506 V 16 w(group)p 565 506 V 14 w(pd\(gd,)g(rank\))o Fh(.)324 556 y(See)15 b(Note)f(10.)324 606 y Fg(mpi)h Fh(ma)o(y)e(pro)o(vide)i(additional)f(pro)q(cedures)j(whic)o (h)e(allo)o(w)f(users)j(to)e(determine)g(the)262 656 y(pro)q(cess)g(grouping) e(top)q(ology)g(attributes.)262 772 y Ff(1.2.3)55 b(Comm)n(unication)16 b(Con)n(texts)262 849 y Fh(This)c(prop)q(osal)h(views)g(a)g(comm)o(unicatio)o (n)d(con)o(text)k(as)f(a)g(uniquely)f(iden)o(ti\014ed)h(reference)262 899 y(to)f(exactly)g(one)h(pro)q(cess)h(grouping,)e(whic)o(h)g(is)g(a)h (\014eld)f(in)g(a)g(message)g(en)o(v)o(elop)q(e)h(and)g(ma)o(y)262 948 y(therefore)j(b)q(e)g(used)g(to)f(distinguish)g(messages.)21 b(The)16 b(con)o(text)g(inherits)f(the)h(referenced)262 998 y(pro)q(cess)g(grouping)d(as)h(a)g(\\frame".)j(Eac)o(h)d(pro)q(cess)i (grouping)e(ma)o(y)e(b)q(e)j(used)g(as)f(a)g(frame)262 1048 y(for)f(m)o(ultiple)f(con)o(texts.)262 1156 y Fe(Con)o(text)i(Descriptor)262 1232 y Fh(Eac)o(h)e(con)o(text)g(is)g(iden)o(ti\014ed)g(b)o(y)g(a)g Fd(c)n(ontext)h(descriptor)j Fh(\(or)c Fd(hand)r(le)p Fh(\))h(whic)o(h)e(is)h (expressed)262 1282 y(as)f(an)f(in)o(teger)i(t)o(yp)q(e)f(in)g(the)h(host)f (language)f(and)h(has)g(an)g(opaque)g(v)n(alue)f(whic)o(h)h(is)g(pro)q(cess) 262 1332 y(lo)q(cal.)17 b(See)d(Note)h(1.)324 1382 y(The)i(creation)g(of)f Fg(mpi)h Fh(pro)q(cess)h(groupings)f(allo)q(cates)f(an)h Fd(own)j Fh(con)o(text)e(whic)o(h)e(in-)262 1432 y(herits)f(the)h(created)h(grouping)d (as)i(a)e(frame)g(and)h(can)h(b)q(e)g(though)o(t)e(of)h(as)g(a)g(prop)q(ert)o (y)h(of)262 1482 y(the)h(created)i(grouping.)26 b(The)17 b(grouping)g (retains)g(the)h(descriptor)g(of)e(its)h(o)o(wn)g(con)o(text)262 1531 y(un)o(til)c Fg(mpi)h Fh(pro)q(cess)i(grouping)e(deletion.)19 b Fg(mpi)14 b Fh(pro)o(vides)g(a)g(pro)q(cedure)j(whic)o(h)d(accepts)i(a)262 1581 y(v)n(alid)c(group)h(descriptor)i(and)e(returns)i(the)f(con)o(text)h (descriptor)g(of)e(the)h(o)o(wn)f(con)o(text)h(of)262 1631 y(the)g(iden)o(ti\014ed)g(group.)k(F)m(or)13 b(example,)g Fc(cd)21 b(=)g(mpi)p 1074 1631 V 16 w(own)p 1156 1631 V 15 w(context\(gd\))m Fh(.)d(See)d(Note)f(11.)262 1739 y Fe(Con)o(text)g(Creation)h(and)g(Deletion) 262 1816 y Fg(mpi)f Fh(pro)o(vides)h(facilities)f(whic)o(h)h(allo)o(ws)f (user)i(to)e(dynamically)e(create)17 b(and)e(delete)h(con-)262 1865 y(texts)i(in)e(addition)g(to)h(the)h(o)o(wn)f(con)o(texts)h(asso)q (ciated)g(with)e(pro)q(cess)j(groupings.)28 b(See)262 1915 y(Note)14 b(12.)324 1965 y Fg(mpi)i Fh(pro)o(vides)g(a)g(pro)q(cedure)i(whic) o(h)e(allo)o(ws)f(users)i(to)f(create)i(con)o(texts.)26 b(This)16 b(pro-)262 2015 y(cedure)j(accepts)g(a)e(v)n(alid)f(descriptor)j(of)d(a)i (group)f(of)g(whic)o(h)g(the)h(calling)e(pro)q(cess)k(is)d(a)262 2065 y(mem)o(b)q(er,)10 b(and)i(returns)i(a)e(con)o(text)h(descriptor)g(whic) o(h)f(references)j(the)e(iden)o(ti\014ed)f(group.)262 2114 y(F)m(or)18 b(example,)g Fc(cd)j(=)h(mpi)p 699 2114 V 15 w(context)p 868 2114 V 14 w(creation\(gd\))m Fh(.)32 b(This)19 b(pro)q(cedure)h(is)f (called)f(b)o(y)262 2164 y(and)13 b(sync)o(hronises)j(all)c(mem)o(b)q(ers)h (of)g Fc(gd)p Fh(.)324 2214 y Fg(mpi)f Fh(pro)o(vides)h(a)f(pro)q(cedure)i (whic)o(h)f(allo)o(ws)e(users)j(to)f(delete)g(user)h(created)g(con)o(texts.) 262 2264 y(This)d(pro)q(cedure)j(accepts)f(a)e(v)n(alid)f(con)o(text)j (descriptor)g(whic)o(h)e(w)o(as)h(created)h(b)o(y)e(the)i(call-)262 2314 y(ing)8 b(pro)q(cess)j(and)e(deletes)i(the)f(iden)o(ti\014ed)f(con)o (text.)17 b(F)m(or)9 b(example,)g Fc(mpi)p 1389 2314 V 15 w(context)p 1558 2314 V 14 w(deletion\(cd\))n Fh(.)262 2363 y(This)k(pro)q(cedure)j(is)e (called)f(b)o(y)h(and)g(sync)o(hronises)h(all)e(mem)o(b)q(ers)f(of)i(the)g (frame)f(of)g Fc(cd)p Fh(.)324 2413 y(See)i(Note)f(13)967 2574 y(5)p eop %%Page: 6 7 6 6 bop 262 307 a Fe(Descriptor)13 b(T)l(ransmission)262 384 y Fh(Since)19 b(a)g(descriptor)h(is)f(an)f(in)o(teger)i(t)o(yp)q(e)f(in)f (the)i(host)f(language)f(the)i(v)n(alue)e(ma)o(y)f(b)q(e)262 434 y(transmitted)10 b(in)g(a)h(message)f(as)h(an)g(in)o(teger.)17 b(The)12 b(recipien)o(t)f(of)f(the)i(descriptor)g(can)f(mak)o(e)262 483 y(no)j(de\014ned)h(use)h(of)e(the)h(v)n(alue)f(in)g(the)h Fg(mpi)f Fh(op)q(erations)h(describ)q(ed)h(in)e(this)h(prop)q(osal)f(|)262 533 y(the)g(descriptor)h(is)f Fd(invalid)p Fh(.)k(See)d(Note)f(1.)324 583 y(MPI)i(pro)o(vides)h(a)f(mec)o(hanism)d(whereb)o(y)18 b(the)f(user)g(can)f(transmit)f(a)h(v)n(alid)f(con)o(text)262 633 y(descriptor)j(in)e(a)g(message)h(suc)o(h)h(that)e(the)i(receiv)o(ed)g (descriptor)g(is)f(v)n(alid.)25 b(This)17 b(is)f(in-)262 683 y(tegrated)h(with)f(the)h(capabilit)o(y)e(to)h(transmit)f(t)o(yp)q(ed)i (messages.)26 b(It)17 b(is)f(suggested)i(that)262 732 y(a)d(notional)f(data)h (t)o(yp)q(e)h(should)f(b)q(e)h(in)o(tro)q(duced)g(for)f(this)h(purp)q(ose,)g (e.g.)23 b Fc(MPI)p 1524 732 14 2 v 15 w(CD)p 1583 732 V 15 w(TYPE)o Fh(.)262 782 y(See)14 b(Note)h(3.)324 832 y Fg(mpi)e Fh(pro)o(vides)i(a)e(con)o(text)i(descriptor)g(registry)g(service.)k(Use)c (of)f(this)g(service)h(is)f(not)262 882 y(mandated.)27 b(Programs)16 b(whic)o(h)h(can)g(con)o(v)o(enien)o(tly)h(b)q(e)f(expressed)j(without)d (using)g(the)262 932 y(service)e(can)f(ignore)g(it)f(without)h(p)q(enalt)o(y) m(.)j(See)e(Note)f(4.)324 982 y(Note)d(that)f(receipt)i(of)e(a)g(v)n(alid)f (con)o(text)i(descriptor)h(ma)o(y)d(ha)o(v)o(e)h(a)g(p)q(ersisten)o(t)j (e\013ect)f(on)262 1031 y(the)i(implemen)o(tation)d(of)i Fg(mpi)h Fh(at)f(the)i(receiv)o(er,)g(and)f(in)g(particular)f(ma)o(y)f(reserv)o(e)k (state.)262 1081 y Fg(mpi)j Fh(will)g(pro)o(vide)h(a)g(pro)q(cedure)i(whic)o (h)d(frees)j(\(or)e(in)o(v)n(alidates\))f(in)o(v)n(alidates)f(a)i(v)n(alid) 262 1131 y(descriptor,)g(allo)o(wing)c(the)j(implemen)o(tation)c(to)j(free)h (reserv)o(ed)i(state.)32 b(F)m(or)18 b(example,)262 1181 y Fc(mpi)p 331 1181 V 15 w(free)p 434 1181 V 14 w(cd\(cd\))o Fh(.)26 b(The)17 b(user)h(is)e(not)h(allo)o(w)o(ed)e(to)h(free)i(the)f(o)o (wn)f(con)o(text)h(descriptor)262 1231 y(of)g(an)o(y)h(group)h(or)f(the)h (con)o(text)g(descriptor)h(of)e(an)o(y)g(con)o(text)h(created)h(b)o(y)e(the)h (calling)262 1280 y(pro)q(cess.)324 1330 y(See)c(Note)f(6.)262 1438 y Fe(Descriptor)f(A)o(ttribu)o(tes)262 1515 y Fg(mpi)f Fh(pro)o(vides)i(a)f(pro)q(cedure)i(whic)o(h)e(allo)o(ws)f(users)j(to)e (determine)g(the)h(pro)q(cess)h(grouping)262 1565 y(whic)o(h)e(is)h(the)h (frame)d(of)h(a)h(con)o(text.)19 b(F)m(or)13 b(example,)f Fc(gd)22 b(=)f(mpi)p 1282 1565 V 15 w(context)p 1451 1565 V 15 w(group\(cd\))n Fh(.)262 1681 y Ff(1.2.4)55 b(P)n(oin)n(t-to-P)n(oin)n(t)19 b(Comm)n(unication)262 1757 y Fh(This)12 b(prop)q(osal)h(recommends)f(three)i (forms)e(for)h Fg(mpi)f Fh(p)q(oin)o(t-to-p)q(oin)o(t)g(message)g(address-) 262 1807 y(ing)h(and)h(selection:)19 b(n)o(ull)13 b(con)o(text;)h(closed)g (con)o(text;)h(op)q(en)f(con)o(text.)19 b(\(See)c(Note)g(15\).)j(It)262 1857 y(is)d(further)i(recommended)e(that)i(messages)f(comm)o(uni)o(cated)e (in)i(eac)o(h)g(form)f(are)h(distin-)262 1907 y(guished)h(suc)o(h)h(that)f(a) g Fc(Send)f Fh(op)q(eration)i(of)e(form)g(X)h(cannot)g(matc)o(h)f(with)h(a)g Fc(Receive)262 1957 y Fh(op)q(eration)10 b(of)g(form)f(Y,)h(requiring)g(that) h(form)e(is)h(em)o(b)q(edded)h(in)o(to)f(the)h(message)g(en)o(v)o(elop)q(e.) 324 2006 y(The)k(three)g(forms)e(are)i(describ)q(ed,)h(follo)o(w)o(ed)d(b)o (y)h(considerations)h(of)f(uniform)e(in)o(teg-)262 2056 y(ration)h(of)g (these)i(forms)e(in)g(the)i(p)q(oin)o(t-to-p)q(oin)o(t)d(comm)o(unication)f (c)o(hapter)k(of)e Fg(mpi)p Fh(.)262 2164 y Fe(Null)h(Con)o(text)g(F)l(orm) 262 2241 y Fh(The)20 b Fd(nul)r(l)h(c)n(ontext)k Fh(form)18 b(con)o(tains)j(no)f(message)g(con)o(text.)38 b(Message)22 b(selection)f(and)262 2291 y(addressing)16 b(are)f(expressed)j(b)o(y)e Fc(\(pd,)k(tag\))15 b Fh(where:)22 b Fc(pd)15 b Fh(is)h(a)f(pro)q(cess)i (descriptor;)g Fc(tag)262 2340 y Fh(is)c(a)h(message)f(tag.)324 2390 y Fc(Send)g Fh(supplies)h(the)g Fc(pd)f Fh(of)g(the)h(receiv)o(er.)20 b Fc(Receive)12 b Fh(supplies)i(the)g Fc(pd)f Fh(of)g(the)h(sender.)967 2574 y(6)p eop %%Page: 7 8 7 7 bop 324 307 a Fc(Receive)16 b Fh(can)j(wildcard)e(on)h Fc(pd)g Fh(b)o(y)f(supplying)h(the)h(v)n(alue)e(of)g(a)h(named)f(constan)o(t) 262 357 y(pro)q(cess)j(descriptor,)h(e.g.)33 b Fc(MPI)p 788 357 14 2 v 15 w(PD)p 847 357 V 15 w(WILD)p Fh(.)18 b(See)i(Note)f(14.)33 b(This)18 b(prop)q(osal)h(mak)o(es)f(no)262 407 y(statemen)o(t)13 b(ab)q(out)h(the)h(pro)o(vision)d(for)i(wildcard)f(on)h Fc(tag)p Fh(.)262 515 y Fe(Closed)g(Con)o(text)h(F)l(orm)262 591 y Fh(The)f Fd(close)n(d)g(c)n(ontext)k Fh(form)12 b(p)q(ermits)h(comm)o(unication)d(b)q (et)o(w)o(een)15 b(mem)o(b)q(ers)d(of)h(the)h(same)262 641 y(con)o(text.)22 b(Message)16 b(selection)g(and)f(addressing)h(are)f (expressed)j(b)o(y)d Fc(\(cd,)21 b(rank,)f(tag\))262 691 y Fh(where:)e Fc(cd)11 b Fh(is)h(a)g(con)o(text)h(descriptor;)g Fc(rank)f Fh(is)f(a)h(pro)q(cess)i(rank)e(in)g(the)g(frame)f(of)h Fc(cd)p Fh(;)f Fc(tag)262 741 y Fh(is)i(a)h(message)f(tag.)324 791 y Fc(Send)i Fh(supplies)i(the)g Fc(cd)f Fh(of)g(the)h(receiv)o(er)h (\(and)f(sender\),)h(and)e(the)i Fc(rank)d Fh(of)h(the)h(re-)262 840 y(ceiv)o(er.)26 b Fc(Receive)15 b Fh(supplies)i(the)g Fc(cd)f Fh(of)g(the)h(sender)h(\(and)e(receiv)o(er\),)j(and)d(the)h(rank)f(of)262 890 y(the)f(sender.)24 b(The)16 b Fc(\(cd,)21 b(rank\))14 b Fh(pair)h(in)f Fc(Send)h Fh(\()p Fc(Receive)p Fh(\))f(is)h(su\016cien)o(t)h (to)f(determine)262 940 y(the)f(pro)q(cess)i(descriptor)f(of)e(the)i(receiv)o (er)g(\(sender\).)324 990 y Fc(Receive)9 b Fh(cannot)i(wildcard)f(on)g Fc(cd)p Fh(.)17 b Fc(Receive)9 b Fh(can)i(wildcard)f(on)g Fc(rank)g Fh(b)o(y)g(supplying)262 1040 y(the)15 b(v)n(alue)f(of)h(a)f(named)g(constan) o(t)i(in)o(teger,)f(e.g.)21 b Fc(MPI)p 1133 1040 V 15 w(RANK)p 1236 1040 V 15 w(WILD)p Fh(.)14 b(See)i(Note)f(16.)21 b(This)262 1089 y(prop)q(osal)13 b(mak)o(es)g(no)g(statemen)o(t)h(ab)q(out)g(the)h(pro)o (vision)d(for)i(wildcard)f(on)h Fc(tag)p Fh(.)262 1197 y Fe(Op)q(en)h(Con)o (text)f(F)l(orm)262 1274 y Fh(The)k Fd(op)n(en)h(c)n(ontext)j Fh(form)16 b(p)q(ermits)h(comm)o(unication)e(b)q(et)o(w)o(een)k(mem)o(b)q (ers)e(of)g(an)o(y)g(t)o(w)o(o)262 1324 y(con)o(texts.)g(Message)11 b(selection)e(and)h(addressing)g(are)f(expressed)j(b)o(y)d Fc(\(lcd,)21 b(rcd,)g(rank,)f(tag\))262 1374 y Fh(where:)d Fc(lcd)11 b Fh(is)h(a)f(\\lo)q(cal")f(con)o(text)i(descriptor;)h Fc(rcd)e Fh(is)g(a)g(\\remote")g(con)o(text)h(descriptor;)262 1423 y Fc(rank)h Fh(is)g(a)h(pro)q(cess)i(rank)d(in)h(the)g(frame)f(of)g Fc(rcd)p Fh(;)g Fc(tag)g Fh(is)h(a)f(message)h(tag.)324 1473 y Fc(Send)21 b Fh(supplies)h(the)g(con)o(text)h(descriptor)g(for)f(the)g (sender)h(in)f Fc(lcd)p Fh(,)g(the)h(con)o(text)262 1523 y(descriptor)c(for)f (the)h(receiv)o(er)h(in)e Fc(rcd)p Fh(,)h(and)f(the)h Fc(rank)e Fh(of)h(the)h(receiv)o(er)h(in)e(the)h(frame)262 1573 y(of)f Fc(rcd)p Fh(.)31 b Fc(Receive)17 b Fh(supplies)j(the)f(con)o(text)g (descriptor)h(for)f(the)g(receiv)o(er)h(in)e Fc(lcd)p Fh(,)h(the)262 1623 y(con)o(text)f(descriptor)g(for)f(the)h(sender)h(in)e Fc(rcd)p Fh(,)g(and)h(the)g Fc(rank)e Fh(of)h(the)h(sender)h(receiv)o(er)262 1672 y(in)c(the)h(frame)f(of)g Fc(rcd)p Fh(.)23 b(The)16 b Fc(\(rcd,)k(rank\))15 b Fh(pair)g(in)h Fc(Send)e Fh(\()p Fc(Receive)p Fh(\))h(is)h(su\016cien)o(t)g(to)262 1722 y(determine)d(the)i(pro)q(cess)h (descriptor)f(of)e(the)i(receiv)o(er)g(\(sender\).)324 1772 y Fc(Receive)f Fh(cannot)h(wildcard)g(on)h Fc(lcd)f Fh(whic)o(h)g(is)g(the)h (op)q(en)g(con)o(text)g(form)e(analog)g(of)262 1822 y(there)g(b)q(eing)f(no)f (wildcard)h(on)f Fc(cd)h Fh(in)f(the)i(closed)f(con)o(text)h(form.)h(Receiv)o (e)f(can)f(wildcard)262 1872 y(on)19 b Fc(rcd)g Fh(b)o(y)h(supplying)f(the)h (v)n(alue)f(of)g(a)h(named)f(constan)o(t)h(con)o(text)g(descriptor,)i(e.g.) 262 1922 y Fc(MPI)p 331 1922 V 15 w(CD)p 390 1922 V 15 w(WILD)12 b Fh(\(See)j(Note)f(14\),)e(in)h(whic)o(h)h(case)g Fc(Receive)e Fd(must)17 b Fh(also)c(wildcard)g(on)g Fc(rank)262 1971 y Fh(as)k(there)i(is) e(insu\016cien)o(t)h(information)c(to)k(determine)f(the)h(pro)q(cess)h (descriptor)g(of)e(the)262 2021 y(sender.)39 b Fc(Receive)18 b Fh(can)j(wildcard)f(on)g Fc(rank)f Fh(b)o(y)i(supplying)e(the)i(v)n(alue)f (of)g(a)g(named)262 2071 y(constan)o(t)e(in)o(teger,)i(e.g.)30 b Fc(MPI)p 750 2071 V 15 w(RANK)p 853 2071 V 15 w(WILD)p Fh(.)17 b(See)i(Note)g(16.)31 b(This)18 b(prop)q(osal)g(mak)o(es)f(no)262 2121 y(statemen)o(t)c(ab)q(out)h(the)h(pro)o(vision)d(for)i(wildcard)f(on)h Fc(tag)p Fh(.)262 2229 y Fe(Uniform)f(In)o(tegration)262 2305 y Fh(The)j(three)h(forms)e(of)g(addressing)i(and)f(selection)g(describ)q(ed)i (ha)o(v)o(e)e(di\013eren)o(t)h(syn)o(tactic)262 2355 y(framew)o(orks.)29 b(W)m(e)17 b(can)h(consider)h(in)o(tegrating)e(these)j(forms)c(in)o(to)h(the) i(p)q(oin)o(t-to-p)q(oin)o(t)967 2574 y(7)p eop %%Page: 8 9 8 8 bop 262 307 a Fh(c)o(hapter)15 b(of)f Fg(mpi)g Fh(b)o(y)h(de\014ning)f(a) h(further)g(orthogonal)f(axis)g(\(as)h(in)f(the)h(m)o(ulti-lev)o(el)d(pro-) 262 357 y(p)q(osal)i(of)f(Gropp)i(&)f(Lusk\))h(whic)o(h)g(deals)f(with)g (form.)k(This)d(is)f(at)g(the)i(exp)q(ense)g(of)e(m)o(ul-)262 407 y(tiplying)f(the)i(n)o(um)o(b)q(er)e(of)h Fc(Send)g Fh(and)g Fc(Receive)f Fh(pro)q(cedures)k(b)o(y)e(a)f(factor)g(of)g(three,)i(and)262 457 y(some)c(further)i(but)g(trivial)e(w)o(ork)h(with)g(details)h(of)f(the)h (curren)o(t)h(p)q(oin)o(t-to-p)q(oin)o(t)d(c)o(hapter)262 506 y(whic)o(h)h(uniformly)f(assumes)h(a)h(single)f(addressing)i(and)f(selection) g(form.)324 556 y(There)22 b(are)g(v)n(arious)f(approac)o(hes)h(to)f (uni\014cation)g(of)g(the)h(syn)o(tactic)g(framew)o(orks)262 606 y(whic)o(h)10 b(ma)o(y)f(simplify)f(in)o(tegration.)17 b(Three)12 b(options)e(are)i(no)o(w)e(describ)q(ed,)j(eac)o(h)e(based)h(on) 262 656 y(reten)o(tion)i(and)g(extension)g(of)f(the)i(framew)o(ork)d(of)h (one)h(form.)j(These)e(options)e(represen)o(t)262 706 y(di\013eren)o(t)h (compromises)e(b)q(et)o(w)o(een)k(the)e(three)h(forms.)262 814 y Fe(Option)d(i:)21 b(Op)q(en)14 b(Con)o(text)h(F)l(ramew)o(ork)41 b Fh(The)13 b(framew)o(ork)f(of)h(the)h(op)q(en)g(con)o(text)262 863 y(form)e(is)h(adopted)i(and)e(extended.)324 913 y(In)o(tro)q(duce)21 b(the)g Fd(nul)r(l)j Fh(descriptor,)e(the)f(v)n(alue)f(of)f(whic)o(h)h(is)g (de\014ned)h(b)o(y)f(a)g(named)262 963 y(constan)o(t,)e(e.g.)28 b Fc(MPI)p 605 963 14 2 v 15 w(NULL)p Fh(.)16 b(See)i(Note)g(17.)27 b(The)18 b(n)o(ull)e(con)o(text)i(form)e(is)h(expressed)j(as)262 1013 y Fc(\(MPI)p 353 1013 V 14 w(NULL,)h(MPI)p 564 1013 V 15 w(NULL,)g(pd,)g(tag\))o Fh(,)14 b(whic)o(h)f(is)h(a)f(little)g(clumsy)m(.) j(The)f(closed)f(con)o(text)262 1063 y(form)f(is)j(expressed)h(as)f Fc(\(MPI)p 736 1063 V 15 w(NULL,)21 b(cd,)g(rank,)g(tag\))o Fh(,)15 b(whic)o(h)g(is)h(marginally)c(incon-)262 1112 y(v)o(enien)o(t.)17 b(The)12 b(op)q(en)f(con)o(text)i(form)c(is)i(expressed)j(as)d Fc(\(lcd,)21 b(rcd,)g(rank,)g(tag)o Fh(\),)12 b(whic)o(h)262 1162 y(is)h(of)h(course)h(natural.)262 1270 y Fe(Option)d(ii:)19 b(Closed)14 b(Con)o(text)g(F)l(ramew)o(ork)41 b Fh(The)13 b(framew)o(ork)e (of)i(the)g(closed)h(con-)262 1320 y(text)g(form)e(is)i(adopted)g(and)g (extended.)324 1370 y(In)o(tro)q(duce)21 b(the)g Fd(nul)r(l)j Fh(descriptor,)e(the)f(v)n(alue)f(of)f(whic)o(h)h(is)g(de\014ned)h(b)o(y)f(a) g(named)262 1420 y(constan)o(t,)e(e.g.)28 b Fc(MPI)p 605 1420 V 15 w(NULL)p Fh(.)16 b(See)i(Note)g(17.)27 b(The)18 b(n)o(ull)e(con)o(text)i (form)e(is)h(expressed)j(as)262 1469 y Fc(\(MPI)p 353 1469 V 14 w(NULL,)h(pd,)g(tag\))o Fh(,)d(whic)o(h)g(is)f(marginally)e(incon)o(v)o (enien)o(t.)29 b(The)18 b(closed)g(con)o(text)262 1519 y(form)12 b(is)j(expressed)i(as)e Fc(\(cd,)21 b(rank,)g(tag\))o Fh(,)14 b(whic)o(h)h(is)f(of)h(course)h(natural.)k(Expression)262 1569 y(of)13 b(the)h(op)q(en)h(con)o(text)f(form)e(requires)j(a)f(little)f(more)g (w)o(ork.)324 1619 y(W)m(e)j(can)i(use)g(the)f Fc(cd)g Fh(\014eld)g(as)g (\\shorthand)h(notation")e(for)g(the)i Fc(\(lcd,)j(rcd\))16 b Fh(pair)262 1669 y(at)22 b(the)g(exp)q(ense)i(of)e(in)o(tro)q(ducing)g (some)f(tric)o(k)o(ery)m(.)43 b(W)m(e)22 b(de\014ne)h(a)f(\\con)o(text)g (duplet)262 1718 y(descriptor")10 b(whic)o(h)f(is)g(formally)d(comp)q(osed)j (of)g(t)o(w)o(o)g(references)j(to)d(con)o(texts,)i(and)e(pro)o(vide)262 1768 y(a)15 b(pro)q(cedure)j(whic)o(h)e(constructs)i(suc)o(h)f(a)f (descriptor)h(giv)o(en)f(t)o(w)o(o)g(con)o(text)g(descriptors.)262 1818 y(Both)g Fc(Send)e Fh(and)i Fc(Receive)e Fh(will)g(accept)j(a)f(duplet)f (descriptor)i(in)f Fc(cd)p Fh(,)f(are)h(able)f(to)h(dis-)262 1868 y(tinguish)g(the)i(duplet)g(descriptor)g(from)e(a)h(singlet)g (descriptor,)i(and)e(treat)h(the)g(duplet)262 1918 y(as)13 b(shorthand)i(notation.)i(W)m(e)c(should)h(also)f(de\014ne)i(a)e(mec)o (hanism)e(b)o(y)j(whic)o(h)g(a)f(receiv)o(er)262 1968 y(whic)o(h)h(has)h (completed)f(a)g Fc(Receive)f Fh(with)h(wildcard)g(on)h Fc(rcd)e Fh(is)i(able)f(to)h(determine)f(the)262 2017 y(v)n(alid)f(singlet)h (descriptor)i(of)e(the)h(sender,)h(whic)o(h)e(just)h(adds)g(one)g(further)g (enquiry)g(pro-)262 2067 y(cedure)f(to)f(the)h(p)q(oin)o(t-to-p)q(oin)o(t)e (c)o(hapter\(?\).)19 b(This)13 b(option)f(is)h(a)g(little)f(incon)o(v)o (enien)o(t)h(but)262 2117 y(do)q(es)d(ha)o(v)o(e)g(some)f(useful)h(prop)q (erties)i(for)d(collectiv)o(e)h(comm)o(unications.)k(It)c(is)g(conjectured) 262 2167 y(that)j(this)h(option)g(is)f(the)i(b)q(est)g(c)o(hoice)f(for)g Fg(mpi)p Fh(.)262 2275 y Fe(Option)g(iii:)21 b(Null)16 b(Con)o(text)f(F)l (ramew)o(ork)41 b Fh(The)15 b(framew)o(ork)f(of)g(the)h(n)o(ull)f(con)o(text) 262 2325 y(form)e(is)h(adopted)i(and)e(extended.)324 2374 y(The)f(n)o(ull)f (con)o(text)i(form)d(is)i(expressed)j(as)d Fc(\(pd,)21 b(tag\))o Fh(,)12 b(whic)o(h)g(is)g(of)f(course)j(natural.)262 2424 y(Expression)g(of)g (the)g(op)q(en)g(and)g(closed)h(con)o(text)f(forms)f(requires)i(a)e(little)h (more)f(w)o(ork.)967 2574 y(8)p eop %%Page: 9 10 9 9 bop 324 307 a Fh(W)m(e)13 b(can)g(use)h(the)g Fc(pd)f Fh(\014eld)g(as)g (\\shorthand)h(notation")e(for)h Fc(\(cd,)21 b(rank\))12 b Fh(and)h Fc(\(lcd,)262 357 y(rcd,)20 b(rank\))15 b Fh(b)o(y)h(con)o(tin)o (uation)f(of)g(the)h(tric)o(k)o(ery)h(used)f(in)g(the)g(previous)h(option.)23 b(This)262 407 y(is)13 b(rather)i(clumsy)m(.)262 523 y Ff(1.2.5)55 b(Collectiv)n(e)16 b(Comm)n(unication)262 599 y Fh(Symmetric)d(collectiv)o(e) j(comm)o(unication)c(op)q(erations)k(are)h(complian)o(t)c(with)j(the)g (closed)262 649 y(con)o(text)e(form)d(describ)q(ed)16 b(ab)q(o)o(v)o(e.)h (This)d(prop)q(osal)f(recommends)g(that)g(suc)o(h)h(op)q(erations)262 699 y(accept)j(a)e(con)o(text)i(descriptor)g(whic)o(h)e(iden)o(ti\014es)i (the)f(con)o(text)g(\(th)o(us)h(frame\))d(in)i(whic)o(h)262 749 y(they)e(are)g(to)g(op)q(erate.)324 799 y Fg(mpi)i Fh(do)q(es)h(plan)f (to)g(describ)q(e)j(symmetric)14 b(collectiv)o(e)j(comm)o(unicatio)o(n)d(op)q (erations.)262 848 y(It)i(is)g(not)g(p)q(ossible)g(to)g(determine)g(whether)i (this)e(prop)q(osal)g(is)g(su\016cien)o(t)g(to)g(allo)o(w)f(im-)262 898 y(plemen)o(tation)h(of)i(the)h(collectiv)o(e)f(comm)o(unicatio)o(n)e(c)o (hapter)j(of)f Fg(mpi)g Fh(in)f(terms)h(of)g(the)262 948 y(p)q(oin)o(t-to-p)q (oin)o(t)12 b(c)o(hapter)i(of)f Fg(mpi)h Fh(without)f(loss)g(of)g(generalit)o (y)m(,)g(since)h(the)g(collectiv)o(e)g(op-)262 998 y(erations)g(are)g(not)g (y)o(et)g(de\014ned.)324 1048 y(Asymmetric)f(collectiv)o(e)h(comm)o (unication)d(op)q(erations,)k(esp)q(ecially)f(those)i(in)e(whic)o(h)262 1097 y(sender\(s\))20 b(and)f(receiv)o(er\(s\))h(are)f(distinct)g(pro)q (cesses,)j(are)d(complian)o(t)d(with)i(the)h(op)q(en)262 1147 y(con)o(text)14 b(form)d(describ)q(ed)16 b(ab)q(o)o(v)o(e.)h(This)d(prop)q (osal)f(recommends)g(that)g(suc)o(h)h(op)q(erations)262 1197 y(accept)f(a)f(pair)g(of)f(con)o(text)i(descriptors)h(\(p)q(erhaps)g(in)d(a)h (duplet)h(descriptor)g(form\))e(whic)o(h)262 1247 y(iden)o(tify)i(the)h(con)o (texts)h(\(th)o(us)g(frames\))e(in)g(whic)o(h)h(they)g(are)g(to)g(op)q (erate.)324 1297 y Fg(mpi)21 b Fh(do)q(es)h(not)g(plan)e(to)i(describ)q(e)h (asymmetric)c(collectiv)o(e)j(comm)o(unicatio)o(n)d(op-)262 1346 y(erations.)33 b(Suc)o(h)19 b(op)q(erations)h(are)f(expressiv)o(e)h (when)g(writing)e(programs)g(b)q(ey)o(ond)h(the)262 1396 y(SPMD)13 b(mo)q(del,)f(whic)o(h)i(are)g(comp)q(osed)g(of)f(comm)o(unicati)o(v)o(e)e (functionally)i(distinct)h(pro-)262 1446 y(cess)g(groupings.)k(This)13 b(prop)q(osal)f(recommends)g(that)i(suc)o(h)g(op)q(erations)f(should)g(b)q(e) h(con-)262 1496 y(sidered)h(in)e(some)g(reincarnation)h(of)f Fg(mpi)p Fh(.)262 1633 y Fi(1.3)64 b(Discussion)24 b(&)d(Notes)262 1724 y Fh(This)14 b(section)h(comprises)g(a)f(discussion)h(of)f(certain)h (asp)q(ects)h(of)e(this)h(prop)q(osal)f(follo)o(w)o(ed)262 1774 y(b)o(y)f(the)i(notes)f(referenced)j(in)c(the)i(detailed)e(prop)q(osal.) 262 1889 y Ff(1.3.1)55 b(Discussion)262 1966 y Fh(W)m(e)15 b(can)h(dissect)i(the)e(prop)q(osal)g(in)o(to)f(t)o(w)o(o)h(parts:)22 b(an)16 b(SPMD)g(mo)q(del)f(core;)i(an)f(MIMD)262 2016 y(mo)q(del)f(annex.)27 b(In)17 b(this)g(discussion)g(the)h(dissection)f(is)g(exp)q(osed)h(and)e(the) i(conceptual)262 2066 y(foundation)d(of)h(eac)o(h)h(part)g(is)g(describ)q (ed.)28 b(The)17 b(discussion)h(also)e(presen)o(ts)i(brief)f(argu-)262 2116 y(men)o(ts)c(for)g(and)h(against)f(the)i(MIMD)e(mo)q(del)g(annex.)262 2223 y Fe(SPMD)i(mo)q(del)f(core)262 2300 y Fh(The)e(SPMD)g(mo)q(del)e(core)j (pro)o(vides)f(noncomm)o(unicativ)o(e)d(pro)q(cess)14 b(groupings)d(and)h (com-)262 2350 y(m)o(unication)j(con)o(texts)k(for)f(writers)g(of)g(SPMD)f (parallel)g(libraries.)30 b(It)18 b(is)f(in)o(tended)i(to)262 2399 y(pro)o(vide)11 b(expressiv)o(e)i(p)q(o)o(w)o(er)e(b)q(ey)o(ond)h(the)g (\\SPIMD")f(mo)q(del)e(\(in)i(whic)o(h)h(pro)q(cess)h(execute)262 2449 y(in)g(an)h(SIMD)f(fashion\).)967 2574 y(9)p eop %%Page: 10 11 10 10 bop 324 307 a Fh(The)18 b(material)e(describing)j(pro)q(cesses)h(in)e (Section)g(1.2.1)f(is)h(simpli\014ed:)24 b(pro)q(cesses)262 357 y(ha)o(v)o(e)17 b(iden)o(tical)f(instruction)i(blo)q(c)o(ks)f(and)g (di\013eren)o(t)h(data)f(blo)q(c)o(ks;)h(pro)q(cess)h(descriptor)262 407 y(transmission)14 b(and)i(registry)g(b)q(ecome)g(redundan)o(t;)h(dynamic) d(pro)q(cess)k(mo)q(dels)c(are)j(not)262 457 y(considered.)324 506 y(The)22 b(material)e(describing)i(pro)q(cess)i(groupings)d(in)g(Section) h(1.2.2)e(is)i(simpli\014ed:)262 556 y(group)13 b(descriptor)i(transmission)e (and)g(registry)i(b)q(ecome)e(redundan)o(t;)h(the)g(o)o(wn)g(pro)q(cess)262 606 y(grouping)f(explicitly)f(b)q(ecomes)j(a)e(single)h(group)f(con)o (taining)g(all)g(pro)q(cesses.)324 656 y(The)j(material)d(describing)j(comm)o (unicatio)o(n)d(con)o(texts)j(in)f(Section)h(1.2.3)e(is)h(simpli-)262 706 y(\014ed:)j(con)o(text)d(descriptor)g(transmission)d(and)i(registry)h(b)q (ecome)e(unnecessary)m(.)324 756 y(The)20 b(material)e(describing)i(p)q(oin)o (t-to-p)q(oin)o(t)e(comm)o(unication)e(in)k(Section)g(1.2.4)e(is)262 805 y(simpli\014ed:)c(the)e(op)q(en)g(con)o(text)g(form)d(b)q(ecomes)j (redundan)o(t;)g(uniform)d(in)o(tegration)i(\\Op-)262 855 y(tion)18 b(i")g(is)g(deleted,)j(and)d(\\Option)g(ii")g(loses)h(duplet)g(descriptors,)i (b)q(ecoming)c(simple)262 905 y(enough)d(that)f(\\Option)h(iii")e(need)j(not) f(b)q(e)g(further)h(considered.)324 955 y(The)k(material)e(describing)i (collectiv)o(e)g(comm)o(unicatio)o(n)d(in)i(Section)i(1.2.5)d(is)h(sim-)262 1005 y(pli\014ed:)h(there)d(is)e(no)h(p)q(ossibilit)o(y)e(of)h(collectiv)o(e) h(comm)o(unicatio)o(n)d(op)q(erations)j(spanning)262 1054 y(more)d(than)i (one)g(con)o(text.)262 1162 y Fe(MIMD)i(mo)q(del)e(annex)262 1239 y Fh(The)d(MIMD)g(mo)q(del)f(annex)i(extends)g(and)f(mo)q(di\014es)g (the)g(SPMD)h(mo)q(del)d(core)k(to)e(pro)o(vide)262 1289 y(expressiv)o(e)g(p) q(o)o(w)o(er)f(for)f(MIMD)h(programs)e(whic)o(h)i(com)o(bine)e(\(coarse)j (grain\))e(function)h(and)262 1339 y(data)h(driv)o(en)i(parallelism.)h(The)f (MIMD)f(mo)q(del)e(annex)j(is)f(not)g(in)o(tended)h(to)e(pro)o(vide)h(ex-)262 1388 y(pressiv)o(e)i(p)q(o)o(w)o(er)g(to)f(\014ne)h(grained)f(function)h (driv)o(en)f(parallel)f(programs)h(|)g(it)g(is)g(conjec-)262 1438 y(tured)e(that)g(message)f(passing)h(approac)o(hes)g(suc)o(h)h(as)f Fg(mpi)f Fh(are)h(not)g(suited)g(to)g(\014ne)g(grained)262 1488 y(parallel)k(programmi)o(ng.)22 b(The)17 b(annex)g(is)f(in)o(tended)h (to)f(pro)o(vide)g(expressiv)o(e)i(p)q(o)o(w)o(er)f(for)262 1538 y(the)d(\\MSPMD")f(mo)q(del,)f(whic)o(h)i(is)g(no)o(w)f(describ)q(ed.) 324 1588 y(One)g(of)e(the)i(simplest)e(MIMD)h(mo)q(dels)f(is)h(the)g (\\host-no)q(de")h(mo)q(del,)d(familia)o(r)g(in)h Fg(ex-)262 1637 y(press)g Fh(and)g Fg(p)m(arma)o(cs)p Fh(,)i(con)o(taining)d(t)o(w)o(o)h (functional)f(groups:)17 b(one)11 b(no)q(de)h(group)f(\(SPMD)262 1687 y(lik)o(e\))i(;)g(one)h(host)g(group)g(\(a)g(singleton\).)324 1737 y(The)d(\\parallel)e(clien)o(t-serv)o(er")j(mo)q(del,)e(in)g(whic)o(h)h (eac)o(h)g(of)f(the)i Fb(n)e Fh(clien)o(ts)h(is)g(comp)q(osed)262 1787 y(of)i(parallel)g(pro)q(cesses,)k(and)d(in)g(whic)o(h)g(the)h(serv)o(er) g(ma)o(y)e(also)g(b)q(e)i(comp)q(osed)f(of)g(parallel)262 1837 y(pro)q(cesses,)k(con)o(tains)d(1)10 b(+)h Fb(n)k Fh(functional)g(groups:)21 b Fb(n)15 b Fh(clien)o(t)h(groups)g(\(SPMD)f(lik)o(e\);)g(one)262 1886 y(serv)o(er)i(group)f(\(singleton,)f(SPMD)h(lik)o(e\).)23 b(The)17 b(\\host-no)q(de")f(mo)q(del)e(is)i(a)f(case)i(of)f(this)262 1936 y(mo)q(del)c(in)i(whic)o(h)h(the)f(host)h(can)g(b)q(e)g(view)o(ed)f(as)h (a)f(singleton)g(clien)o(t)g(and)g(the)h(no)q(des)g(can)262 1986 y(b)q(e)i(view)o(ed)h(as)f(an)g(SPMD)h(lik)o(e)e(serv)o(er)j(\(or)e(the) h(host)g(as)f(a)g(singleton)g(serv)o(er)i(and)e(the)262 2036 y(no)q(des)d(as)g(an)g(SPMD)g(lik)o(e)f(clien)o(t\).)324 2086 y(The)g(\\parallel)e(mo)q(dule)g(graph")h(mo)q(del,)f(in)h(whic)o(h)g(eac)o (h)h(mo)q(dule)e(within)h(the)h(graph)262 2136 y(ma)o(y)j(b)q(e)k(comp)q (osed)e(of)g(parallel)g(pro)q(cesses)j(\(singleton,)f(SPMD)e(lik)o(e\),)h (con)o(tains)g(an)o(y)262 2185 y(n)o(um)o(b)q(er)c(of)g(functional)g(groups)h (with)g(arbitrarily)f(complex)g(relations.)24 b(The)16 b(\\parallel)262 2235 y(clien)o(t-serv)o(er)j(mo)q(del")e(is)h(a)h(case)g(of)f(this)h(mo)q (del)e(in)h(whic)o(h)g(the)h(mo)q(dule)e(graph)h(only)262 2285 y(con)o(tains)13 b(arcs)i(joining)d(the)j(serv)o(er)g(to)f(eac)o(h)g(clien)o (t.)324 2335 y(The)19 b(MIMD)g(mo)q(del)e(annex)j(is)e(in)o(tended)i(to)f (pro)o(vide)g(expressiv)o(e)h(p)q(o)o(w)o(er)f(for)g(the)262 2385 y(\\parallel)14 b(mo)q(dule)h(graph")h(mo)q(del,)f(whic)o(h)h(I)g(refer) h(to)g(as)f(the)h(MPSMD)f(mo)q(del.)24 b(This)262 2434 y(mo)q(del)14 b(requires)i(supp)q(ort)h(at)e(some)g(lev)o(el)g(as)h(commercial)d(and)i(mo)q (dular)f(applications)957 2574 y(10)p eop %%Page: 11 12 11 11 bop 262 307 a Fh(are)13 b(increasingly)g(mo)o(ving)d(in)j(to)g (parallel)f(computing.)k(The)d(debate)h(is)f(whether)i(or)e(not)262 357 y(message)d(passing)h(approac)o(hes)h(suc)o(h)g(as)g Fg(mpi)e Fh(\(whic)o(h)i(I)f(simply)e(refer)j(to)f(as)g(MPI\))h(should)262 407 y(pro)o(vide)h(for)h(this)g(mo)q(del.)324 457 y(The)d(negativ)o(e)g (argumen)o(t)f(is)h(that)g(suc)o(h)h(SPMD)f(lik)o(e)f(mo)q(dules)g(should)h (b)q(e)h(con)o(trolled)262 506 y(and)e(comm)o(unicate)e(with)i(one)h(another) g(as)g(\\parallel)e(pro)q(cesses")k(at)e(the)g(distributed)g(op-)262 556 y(erating)g(system)h(lev)o(el.)17 b(The)c(argumen)o(t)e(has)h(some)f(app) q(eal)h(as)g(the)g(w)o(orld)g(of)f(distributed)262 606 y(op)q(erating)17 b(systems)h(m)o(ust)f(deal)h(with)f(di\016cult)g(issues)i(suc)o(h)g(as)f(pro) q(cess)h(con)o(trol)f(and)262 656 y(coherency)m(.)23 b(Av)o(oidance)16 b(of)f(duplication)f(in)h(MPI)g(allo)o(ws)g(MPI)g(to)h(fo)q(cus)f(on)h(pro)o (vision)262 706 y(of)11 b(a)h(smaller)f(set)i(of)e(facilities)g(with)h (greater)i(emphasis)d(on)h(maxim)n(um)c(p)q(erformance)k(for)262 756 y(data)h(driv)o(en)h(SPMD)g(lik)o(e)f(parallel)g(programs.)324 805 y(The)20 b(p)q(ositiv)o(e)g(argumen)o(t)f(is)h(that)h(comm)o(uni)o (cations)d(b)q(et)o(w)o(een)j(suc)o(h)g(SPMD)f(lik)o(e)262 855 y(mo)q(dules)11 b(require)j(high)e(p)q(erformance)g(and)h(MPI)g(can)f (pro)o(vide)h(suc)o(h)g(p)q(erformance)g(with)262 905 y(tuned)i(seman)o(tics) f(whic)o(h)h(exp)q(ect)h(the)f(user)h(to)e(deal)h(with)f(coherency)j(issues.) k(There)16 b(is)262 955 y(also)c(the)i(argumen)o(t)f(that)g(MPI)h(is)f(able)g (to)g(deal)h(with)f(this)g(in)g(a)g(shorter)i(time)d(than)h(de-)262 1005 y(v)o(elopmen)o(t)f(\(and)j(standardisation\))f(pro)q(cedures)j(for)d (distributed)h(op)q(erating)f(systems.)262 1054 y(The)j(latter)g(argumen)o(t) f(is)g(somewhat)g(comparable)g(with)g(the)i(argumen)o(t)d(for)i(message)262 1104 y(passing)c(v)o(ersus)i(parallel)e(compilation.)262 1220 y Ff(1.3.2)55 b(Notes)312 1297 y Fh(1.)20 b Fe(Descriptors:)29 b Fh(Descriptors)22 b(are)f(assumed)f(to)g(b)q(e)h(a)f(plen)o(tiful)f(but)i (b)q(ounded)365 1347 y(resource.)37 b(They)20 b(are)g(opaque)f(references)j (to)e(ob)r(jects)g(of)f(unde\014ned)i(size)f(and)365 1397 y(structure.)25 b(They)16 b(are)g(not)f(global)f(unique)i(iden)o(ti\014ers)g(ho)o(w)o(ev)o (er)g(they)g(m)o(ust)e(ref-)365 1446 y(erence)19 b(suc)o(h)e(iden)o (ti\014ers,)h(and)f(they)g(protect)h(the)f(user)h(from)d(the)i(form)e(of)h (suc)o(h)365 1496 y(iden)o(ti\014ers)g(allo)o(wing)c(them)i(to)h(b)q(e)h (implem)o(en)o(tation)c(dep)q(enden)o(t.)23 b(The)15 b(prop)q(osal)365 1546 y(expresses)h(descriptors)e(as)f(in)o(teger)g(t)o(yp)q(es)h(in)f(the)g (host)g(language)f(\(in)g(practice)i(w)o(e)365 1596 y(migh)o(t)9 b(exp)q(ect)j(descriptors)g(to)f(b)q(e)g(indices)g(in)o(to)f(tables)h(of)f (structures,)k(or)c(tables)h(of)365 1646 y(p)q(oin)o(ters)k(to)f(structures,) j(or)d(indeed)h(p)q(oin)o(ters)f(to)h(structures)h(themselv)o(es\).)k(This) 365 1696 y(expression)15 b(facilitates)f(uniform)d(binding)i(to)h(ANSI)g(C)g (and)g(F)m(ortran)f(77.)365 1762 y(The)h(con)o(text)h(descriptor)g(m)o(ust)d (at)i(least)g(reference:)20 b(the)14 b(global)e(unique)i(con)o(text)365 1812 y(iden)o(ti\014er;)j(the)f(group)f(descriptor)i(of)e(the)h(frame.)22 b(The)16 b(group)g(descriptor)h(m)o(ust)365 1862 y(at)j(least)g(reference:)32 b(the)20 b(global)e(unique)i(group)f(iden)o(ti\014er;)j(the)f(o)o(wn)e(con)o (text)365 1911 y(descriptor;)14 b(the)f(rank)g(space)g(to)g(pro)q(cess)h (descriptor)g(map)d(of)g(the)j(group)e(\(includ-)365 1961 y(ing)i(the)g(size) h(of)f(the)h(group\);)e(the)i(pro)q(cess)h(rank.)j(The)14 b(pro)q(cess)i (descriptor)g(m)o(ust)365 2011 y(at)10 b(least)h(reference;)i(the)e(global)e (unique)h(pro)q(cess)i(iden)o(ti\014er;)f(the)g(group)f(descriptor)365 2061 y(of)k(the)g(o)o(wn)g(group.)365 2127 y(The)c(prop)q(osal)f(text)h (distinguishes)g(pro)q(cess,)i(group)d(and)g(con)o(text)h(iden)o(ti\014ers)h (more)365 2177 y(strongly)18 b(than)g(is)g(strictly)h(necessary)m(.)32 b(There)19 b(is)f(p)q(oten)o(tial)g(adv)n(an)o(tage)f(in)h(the)365 2227 y(concept)d(of)e(a)g(uni\014ed)h(descriptor)h(whic)o(h)e(can)h(b)q(e)g (used)g(to)g(reference)i(either)e(kind)365 2277 y(of)k(actual)f(descriptor.) 31 b(F)m(or)18 b(example)f(descriptor)i(uni\014cation)e(go)q(es)h(some)f(w)o (a)o(y)365 2326 y(to)o(w)o(ard)j(cleaning)f(up)h(the)h(duplet,)g(descriptors) g(describ)q(ed)h(in)d(\\Option)h(ii")e(of)365 2376 y(section)i(Section)f (1.2.4.)31 b(De\014nition)18 b(in)g(this)h(fashion)f(requires)i(the)f (referenced)365 2426 y(ob)r(ject)h(to)f(con)o(tain)g(a)g(class)g(iden)o (ti\014er)h(\(whic)o(h)f(in)g(this)g(prop)q(osal)g(could)g(b)q(e)h(as)957 2574 y(11)p eop %%Page: 12 13 12 12 bop 365 307 a Fh(little)20 b(as)g(3)f(bits)h(wide\))g(This)g (suggestion)g(is)g(explored)g(in)g(some)f(of)g(the)i(notes)365 357 y(b)q(elo)o(w.)365 423 y(Rik)12 b(Little\014eld)h(has)h(suggested)g (descriptors)h(could)e(b)q(e)h(used)g(to)f(\\cac)o(he")h(p)q(er)g(ob-)365 473 y(ject)i(user)h(information,)c(as)i(app)q(ears)h(in)g Fg(zipcode)p Fh(.)22 b(This)15 b(descriptor)i(capabilit)o(y)365 523 y(could)f(b)q(e)h (seriously)f(useful)g(for)g(example)f(in)g(con)o(text)i(or)f(group)g(sp)q (eci\014c)i(imple-)365 573 y(men)o(tations)f(of)h(collectiv)o(e)g(comm)o (unications.)28 b(The)19 b(suggestion)f(b)q(oth)g(requires)365 623 y(and)c(deserv)o(es)i(more)d(w)o(ork.)365 689 y(There)20 b(w)o(ere)f(additional)d(motiv)n(ations)g(for)i(expressing)h(pro)q(cess)i (descriptors)f(as)365 739 y(in)o(tegers)f(in)f(the)h(prop)q(osal.)31 b(Firstly)m(,)19 b(the)g(author)f(an)o(ticipated)g(\\Option)g(ii")f(of)365 789 y(Section)11 b(1.2.4.)k(Secondly)m(,)c(it)f(is)g(observ)o(ed)h(that)g (de\014nition)f(of)g(pro)q(cess)i(descriptors)365 839 y(can)19 b(b)q(e)f(w)o(eak)o(ened)h(suc)o(h)g(that)f(they)h(ma)o(y)d(b)q(e)j(implemen) o(ted)d(as)i(global)f(unique)365 888 y(pro)q(cess)i(iden)o(ti\014ers,)e (expressed)i(as)e(in)o(tegers,)g(should)g(this)f(infer)h(adv)n(an)o(tage.)25 b([)p Fd(I)365 938 y(wish)15 b(I)g(had)g(r)n(e)n(alise)n(d)f(the)h(se)n(c)n (ond)g(p)n(oint)g(b)n(efor)n(e)g(the)g(\014rst!])365 1005 y Fh(The)21 b(consequence)h(of)d(allo)o(wing)f(pro)q(cess)j(descriptors)h(to)d (b)q(e)i(implemen)o(ted)d(as)365 1054 y(pro)q(cess)c(iden)o(ti\014ers)f(is)f (explored)h(in)f(some)f(of)h(the)h(notes)f(b)q(elo)o(w.)18 b(Please)13 b(also)e(note)365 1104 y(that)j(this)f(w)o(ould)g(exclude)h(pro)q (cess)h(descriptors)g(from)c(the)j(\\cac)o(hing")f(capabilit)o(y)365 1154 y(men)o(tioned)i(ab)q(o)o(v)o(e.)25 b(These)17 b(ideas)f(suggest)h(an)f (alternate)h(prop)q(osal,)f(con)o(taining)365 1204 y(global)h(unique)h(pro)q (cess)i(iden)o(ti\014ers)e(expressed)j(as)d(in)o(tegers,)h(and)f(con)o(text)h (and)365 1254 y(group)12 b(descriptors)h(as)e(opaque)g(references)k(of)10 b(y)o(et)i(unsp)q(eci\014ed)h(language)e(binding)365 1303 y(\(i.e.,)i(p)q (erhaps)i(not)f(an)g(in)o(teger\).)19 b(This)14 b(suggestion)h(is)e(also)h (explored)g(in)g(some)f(of)365 1353 y(the)i(notes)f(b)q(elo)o(w.)312 1436 y(2.)20 b Fe(Dynamic)f(Pro)q(cesses:)25 b Fh(The)19 b(prop)q(osal)e(do)q (es)h(not)g(prev)o(en)o(t)g(a)g(pro)q(cess)h(mo)q(del)365 1486 y(whic)o(h)d(allo)o(ws)e(dynamic)f(creation)j(and)f(deletion)g(of)g(pro)q (cesses)j(ho)o(w)o(ev)o(er)e(it)f(do)q(es)365 1536 y(not)c(fa)o(v)o(our)f(an) g(async)o(hronous)h(pro)q(cess)i(mo)q(del)c(in)h(whic)o(h)h(singleton)f(pro)q (cesses)j(are)365 1586 y(created)18 b(and)d(deleted)i(in)e(an)h(arbitrary)f (fashion.)23 b(The)16 b(prop)q(osal)g(do)q(es)g(fa)o(v)o(our)f(a)365 1636 y(mo)q(del)c(in)g(whic)o(h)h(blobs)f(of)h(pro)q(cesses)i(are)e(created)i (\(and)e(deleted\))h(in)e(a)h(concerted)365 1685 y(fashion,)20 b(and)f(in)g(whic)o(h)g(eac)o(h)h(blob)f(so)h(created)g(is)g(assigned)f(a)h (di\013eren)o(t)g(o)o(wn)365 1735 y(pro)q(cess)g(grouping.)29 b(This)18 b(mo)q(del)f(do)q(es)h(not)g(tak)o(e)g(in)o(to)f(accoun)o(t)i(the)f (p)q(oten)o(tial)365 1785 y(desire)c(to)f(expand)g(or)g(con)o(tract)h(an)f (existing)f(blob)h(of)f(pro)q(cesses)k(in)c(order)i(to)f(tak)o(e)365 1835 y(in)o(to)c(accoun)o(t)h(\(presumably)e(slo)o(wly\))h(time)f(v)n(arying) g(w)o(orkloads.)16 b(It)10 b(is)f(conjectured)365 1885 y(that)k(concerted)j (blob)c(expand)h(and)g(con)o(tract)h(op)q(erations)f(are)h(more)e(suitable)h (for)365 1934 y(this)h(purp)q(ose)h(than)f(async)o(hronous)h(singleton)e(spa) o(wn)h(and)g(kill)e(op)q(erations.)312 2017 y(3.)20 b Fe(Descriptor)f (transmissio)o(n:)25 b Fh(In)19 b(the)h(spirit)e(of)g(descriptor)i (uni\014cation)f(\(See)365 2067 y(Note)12 b(1\))g(the)g(three)h Fc(MPI)p 754 2067 14 2 v 15 w(?D)p 813 2067 V 16 w(TYPE)d Fh(names)h(can)h(b) q(e)g(collapsed)g(in)o(to)f(something)f(lik)o(e)365 2117 y Fc(MPI)p 434 2117 V 15 w(DESC)p 537 2117 V 15 w(TYPE)p Fh(.)365 2183 y(If)h(pro)q(cess)i(descriptors)g(are)e(replaced)h(b)o(y)f(global)f (unique)h(pro)q(cess)i(iden)o(ti\014ers)f(\(See)365 2233 y(Note)j(1\))e(then) i(no)f(sp)q(ecial)g(measures)g(are)g(required)h(for)f(transmission)e (thereof.)365 2300 y(If)17 b(group)g(and)g(con)o(text)h(descriptors)h(are)f (expressed)i(as)d(opaque)g(ob)r(jects)i(of)d(y)o(et)365 2350 y(unsp)q(eci\014ed)i(t)o(yp)q(e)f(then,)g(in)f(ANSI)g(C)g(at)g(least,)h(it)f (will)e(b)q(e)j(p)q(ossible)g(to)f(prev)o(en)o(t)365 2399 y(nonseman)o(tic)d (transmission)g(thereof.)957 2574 y(12)p eop %%Page: 13 14 13 13 bop 312 307 a Fh(4.)20 b Fe(Descriptor)13 b(registry:)k Fh(The)d(registry)h(service)g(is)f(just)g(a)f(simpler)g(w)o(a)o(y)g(for)h (con-)365 357 y(curren)o(t)22 b(ob)r(jects)f(to)f(iden)o(tify)f(and)g (establish)h(comm)o(unications)d(with)j(one)g(an-)365 407 y(other.)35 b(The)20 b(op)q(erations)f(of)g(in)o(terest,)i(expressed)h(in)d(the)h(spirit) f(of)f(descriptor)365 457 y(uni\014cation)i(\(See)i(Note)g(1\),)g(are:)32 b(registration)21 b(of)f(descriptor)i(b)o(y)e(name,)h(e.g.)365 506 y Fc(mpi)p 434 506 14 2 v 15 w(register\(name,desc\))l Fh(;)k(deregistration,)e(e.g)d Fc(mpi)p 1321 506 V 15 w(deregister\(desc\))m Fh(;)365 556 y(and)14 b(lo)q(okup)g(b)o(y)g(name,)f(e.g.)19 b Fc(mpi)p 915 556 V 15 w(lookup\(name,)g(&desc,)i(wait\))13 b Fh(where)i Fc(wait)365 606 y Fh(con)o(trols)f(whether)i(the)e(lo)q(okup)f (w)o(aits)h(for)f(the)i(name)d(to)i(b)q(e)h(registered.)365 671 y(If)c(pro)q(cess)i(descriptors)g(are)e(replaced)h(b)o(y)f(global)f (unique)h(pro)q(cess)i(iden)o(ti\014ers)f(\(See)365 721 y(Note)j(1\))e(then)i (p)q(erhaps)g(pro)q(cess)h(iden)o(ti\014er)e(registry)g(is)g(not)g(so)g(imp)q (ortan)o(t.)365 785 y(The)h(MIMD)e(mo)q(del)g(do)q(es)i(not)f(actually)f (required)i(pro)q(cess)g(descriptor)h(or)e(group)365 835 y(descriptor)20 b(registry)e(to)g(b)q(e)h(visible)e(to)h(the)g(user)h(since)g(con)o(text)g (descriptor)g(re-)365 885 y(gistry)e(and)g(con)o(text)g(descriptor)i (attribute)e(determination)f(giv)o(es)g(access)j(to)e(all)365 935 y(groups)e(and)f(th)o(us)h(group)g(descriptor)h(attribute)f (determination)e(giv)o(es)h(access)j(to)365 985 y(all)11 b(pro)q(cesses.)20 b(The)12 b(prop)q(osal)f(w)o(as)h(written)g(to)f(handle)h(descriptors)h (consisten)o(tly)m(.)312 1064 y(5.)20 b Fe(Descriptor)g(deallo)q(cation:)28 b Fh(In)20 b(the)h(spirit)f(of)f(descriptor)i(uni\014cation)f(\(See)365 1114 y(Note)c(1\))f(the)i(three)f Fc(mpi)p 769 1114 V 15 w(?d)p 828 1114 V 16 w(free\(?d\))d Fh(can)j(b)q(e)g(collapsed)g(in)o(to)e (something)g(lik)o(e)365 1164 y Fc(mpi)p 434 1164 V 15 w(desc)p 537 1164 V 15 w(free\(desc\))n Fh(.)365 1229 y(The)j(receipt)h(of)e(a)g (descriptor)i(in)f(descriptor)g(transmission)f(and)g(registry)h(is)g(an)365 1279 y(allo)q(cator,)c(hence)j(pro)o(vision)e(of)f(the)i(deallo)q(cator.)20 b(P)o(erhaps)15 b(there)h(should)e(b)q(e)h(an)365 1328 y(explicit)d(allo)q (cator)g(whic)o(h)g(the)h(user)h(m)o(ust)d(call)h(in)g(order)h(to)f(receiv)o (e)i(a)e(descriptor,)365 1378 y(and)i(can)g(deallo)q(cate)g(when)g(no)g (longer)g(required.)365 1443 y(If)d(pro)q(cess)i(descriptors)g(are)e (replaced)h(b)o(y)f(global)f(unique)h(pro)q(cess)i(iden)o(ti\014ers)f(\(See) 365 1493 y(Note)j(1\))e(then)i(pro)q(cess)h(iden)o(ti\014er)e(deallo)q (cation)f(is)g(mo)q(ot.)312 1572 y(6.)20 b Fe(Coherency:)f Fh(The)14 b(prop)q(osal)g(admits)f(incoherency)i(as)f(descriptors)i(ma)o(y)c (b)q(e)j(re-)365 1622 y(ceiv)o(ed)i(in)f(transmission)e(or)i(registry)m(.)25 b(The)17 b(SPMD)f(core)g(con)o(tains)g(no)g(incoher-)365 1672 y(ency)m(.)24 b(The)16 b(inclusion)f(of)g(dynamic)f(pro)q(cess)j(creation)f (and)g(deletion)f(admits)f(in-)365 1722 y(coherency)k(since)f(pro)q(cesses)i (can)d(retain)h(descriptors)g(of)f(pro)q(cesses)j(whic)o(h)d(ha)o(v)o(e)365 1772 y(b)q(een)i(deleted.)27 b(The)17 b(inclusion)e(of)h(grouping)g (descriptor)i(transmission)d(and)h(re-)365 1822 y(gistry)e(admits)d (incoherency)k(since)f(pro)q(cesses)i(can)e(retain)f(descriptors)i(of)e (group-)365 1871 y(ings)f(whic)o(h)h(ha)o(v)o(e)f(b)q(een)h(deleted.)19 b(The)13 b(inclusion)f(of)f(dynamic)g(groupings)h(admits)365 1921 y(incoherency)18 b(since)g(pro)q(cesses)h(can)e(retain)f(descriptors)j (of)d(groupings)g(of)g(whic)o(h)365 1971 y(the)e(rank)f(to)f(pro)q(cess)j (map)c(has)i(c)o(hanged.)18 b(The)13 b(inclusion)g(of)f(con)o(text)h (descriptor)365 2021 y(transmission)j(and)i(registry)g(admits)e(incoherency)i (since)h(pro)q(cesses)h(can)d(retain)365 2071 y(descriptors)j(of)e(con)o (texts)h(whic)o(h)f(ha)o(v)o(e)g(b)q(een)i(deleted.)32 b(The)19 b(prop)q(osal)f(exp)q(ects)365 2120 y(the)13 b(user)g(to)f(ensure)i(coheren)o (t)g(usage.)k(It)12 b(is)g(conjectured)i(that)e(this)g(is)h(acceptable)365 2170 y(pro)o(vided)19 b(that)g(the)g(user)h(is)e(not)h(also)f(exp)q(ected)j (to)d(implemen)o(t)e(pro)q(cess)k(fault)365 2220 y(tolerance.)312 2300 y(7.)g Fe(Dynamic)d(groupings)o(:)i Fh(Pro)q(cess)e(groupings)e(are)h (dynamic)d(in)i(the)h(sense)h(that)365 2350 y(they)c(can)g(b)q(e)g(created)h (at)e(an)o(y)g(time,)f(and)i(static)g(in)f(the)h(sense)h(that)e(the)h(mem)o (b)q(er-)365 2399 y(ship)k(is)f(constan)o(t)g(o)o(v)o(er)h(the)f(lifetime)f (of)g(the)i(pro)q(cess)h(grouping.)24 b(The)17 b(prop)q(osal)365 2449 y(sp)q(eci\014es)g(static)e(groupings)g(ho)o(w)o(ev)o(er)g(the)g(lo)q (ose)g(separation)g(of)f(comm)o(unication)957 2574 y(13)p eop %%Page: 14 15 14 14 bop 365 307 a Fh(con)o(texts)12 b(from)d(pro)q(cess)k(groupings)d (simpli\014es)f(extension)i(to)g(dynamic)e(groupings)365 357 y(as)j(con)o(texts)g(stretc)o(h)h(or)e(shrink)g(according)g(to)g(the)h(c)o (hanges)g(in)f(their)g(frames.)16 b(It)c(is)365 407 y(conjectured)17 b(that)f(concerted)h(grouping)d(expand)i(and)f(con)o(tract)h(op)q(erations)f (are)365 457 y(more)e(suitable)h(than)g(async)o(hronous)g(singleton)g(join)f (and)g(lea)o(v)o(e)h(op)q(erations.)312 540 y(8.)20 b Fe(Pro)q(cess)d(blobs:) i Fg(mpi)14 b Fh(has)i(discussed)g(the)g(concept)g(of)f(the)h(\\all")d(group) i(whic)o(h)365 589 y(con)o(tains)k(all)e(pro)q(cesses.)35 b(The)19 b(\\o)o(wn")e(group)i(concept)h(is)e(a)g(generalisation)g(of)365 639 y(the)h(\\all")d(group)h(concept)i(whic)o(h)f(is)g(expressiv)o(e)h(for)e (programs)g(including)g(and)365 689 y(b)q(ey)o(ond)g(the)g(SPMD)f(mo)q(del.) 24 b(Pro)q(cesses)19 b(are)d(created)i(in)e(\\blobs",)g(where)h(eac)o(h)365 739 y(mem)o(b)q(er)g(of)g(a)h(blob)f(is)h(a)f(mem)o(b)q(er)g(of)g(the)i(same) e(o)o(wn)g(pro)q(cess)j(grouping,)e(and)365 789 y(di\013eren)o(t)e(blobs)e (ha)o(v)o(e)g(di\013eren)o(t)h(o)o(wn)f(pro)q(cess)i(groupings.)j(An)14 b(SPMD)g(program)365 839 y(is)f(a)h(single)e(blob.)18 b(A)13 b(host-no)q(de)h(program)e(comp)q(oses)h(t)o(w)o(o)g(blobs,)g(the)g(no)q(de)h (blob)365 888 y(and)e(the)h(host)g(blob)e(\(a)h(singleton\).)17 b(There)d(is)e(a)g(sense)h(in)f(whic)o(h)g(a)g(blob)g(is)g(SPMD)365 938 y(lik)o(e.)312 1021 y(9.)20 b Fc(mpi)p 434 1021 14 2 v 15 w(own)p 515 1021 V 15 w(group\(pd\))o Fe(:)d Fh(This)10 b(pro)q(cedure)j(lo)q(oks)d(lik)o(e)g(a)h(case)h(of)e(pro)q(cess)i (descriptor)365 1071 y(attribute)18 b(determination.)26 b(If)17 b(pro)q(cess)i(descriptors)g(are)e(allo)o(w)o(ed)f(to)h(b)q(e)h(imple-)365 1121 y(men)o(ted)11 b(as)g(global)e(unique)i(pro)q(cess)i(iden)o(ti\014ers,)e (or)g(are)h(replaced,)g(this)f(pro)q(cedure)365 1171 y(should)16 b(accept)i(no)e(argumen)o(ts)f(and)h(return)h(the)g(o)o(wn)f(group)g (descriptor)h(of)f(the)365 1220 y(calling)d(pro)q(cess.)291 1303 y(10.)20 b Fe(In)o(v)o(erse)9 b(map:)16 b Fh(The)10 b(prop)q(osal)f(did) g(not)g(include)g(a)g(function)g(to)g(con)o(v)o(ert)h Fc(\(gd,)21 b(pd\))365 1353 y Fh(pair)c(in)o(to)g(a)g(rank.)28 b(It)17 b(is)h(suggested)g(that)g(this)f(in)o(v)o(erse)h(map)e(is)h(allo)o(w)o(ed)f (to)h(b)q(e)365 1403 y(\\slo)o(w",)12 b(i.e.)17 b(could)12 b(b)q(e)h(a)f(linear)g(searc)o(h)i(o)o(v)o(er)f(mem)o(b)q(ers)e(of)h(the)h (group,)f(but)h(prob-)365 1453 y(ably)e(should)h(b)q(e)g(included)g(for)g (completeness.)18 b(It)12 b(can)g(b)q(e)g(used)h(as)f(a)f(mem)o(b)q(ership) 365 1503 y(predicate.)291 1586 y(11.)20 b Fc(mpi)p 434 1586 V 15 w(own)p 515 1586 V 15 w(context\(gd\))n Fe(:)d Fh(This)9 b(pro)q(cedure)i(lo)q(oks)e(lik)o(e)f(a)h(case)h(of)f(group)g(descriptor)365 1636 y(attribute)15 b(determination.)291 1719 y(12.)20 b Fe(Grouping)f (Deletion:)26 b Fh(The)20 b(pro)q(cess)h(grouping)e(deletion)g(op)q(eration)g (should)365 1768 y(probably)h(b)q(e)h(de\014ned)h(to)e(fail)f(when)i(there)g (are)g(user)h(created)f(con)o(texts)h(with)365 1818 y(that)15 b(frame)f(whic)o(h)h(ha)o(v)o(e)g(not)g(themselv)o(es)g(b)q(een)h(deleted.)23 b(This)14 b(just)i(requires)g(a)365 1868 y(reference)h(coun)o(t)d(in)f(the)i (group)e(descriptor)j(static)e(attribute)g(store.)291 1951 y(13.)20 b Fe(Con)o(text)i(Creation)f(&)i(Deletion:)k Fh(Marc)21 b(Snir)e(has)h(describ)q(ed)i(a)d(metho)q(d)365 2001 y(b)o(y)g(whic)o(h)g (global)f(unique)h(group)g(iden)o(ti\014ers)h(can)f(b)q(e)h(generated)g (without)f(use)365 2051 y(of)h(shared)i(global)d(data.)38 b(The)21 b(prop)q(osal)g(states)h(that)e(con)o(text)i(creation)f(and)365 2100 y(deletion)13 b(op)q(erations)f(sync)o(hronise)i(the)e(pro)q(cesses)j (within)d(the)h(frame)e(of)h(the)h(con-)365 2150 y(text,)h(an)o(ticipating)f (use)i(of)e(this)h(metho)q(d)f(for)g(generation)h(of)g(con)o(text)g(iden)o (ti\014ers.)365 2200 y(Ho)o(w)o(ev)o(er,)h(the)g(sync)o(hronisation)g (requires)g(that)g(con)o(text)g(creation)g(and)g(deletion)365 2250 y(calls)10 b(within)f(a)g(frame)g(are)h(p)q(erformed)g(in)f(the)i(iden)o (tical)e(sequence)j(b)o(y)d(all)g(mem)o(b)q(ers)365 2300 y(of)i(the)i(frame.) j(The)c(global)e(unique)i(group)f(iden)o(ti\014er)h(and)g(con)o(text)g (creator)h(refer-)365 2350 y(ence)g(coun)o(t)e(are)g(then)h(su\016cien)o(t)f (to)g(generate)h(a)e(global)g(unique)h(con)o(text)g(iden)o(ti\014er)365 2399 y(without)k(comm)o(unicatio)o(n)d(or)j(sync)o(hronisation.)21 b(Should)15 b(con)o(text)g(creation)h(and)365 2449 y(deletion)e(therefore)h (not)f(sync)o(hronise)h(the)g(frame?)957 2574 y(14)p eop %%Page: 15 16 15 15 bop 365 307 a Fh(There)19 b(ma)o(y)d(b)q(e)i(adv)n(an)o(tage)e(in)h (de\014ning)h(con)o(text)g(creation)g(and)f(deletion)h(suc)o(h)365 357 y(that)10 b(a)f(n)o(um)o(b)q(er)g(of)g(con)o(texts)h(are)g(created)h(or)f (deleted)h(sim)o(ultaneously)m(,)c(dep)q(ending)365 407 y(on)12 b(ho)o(w)g(hea)o(vy)g(w)o(e)g(exp)q(ect)i(con)o(text)f(managemen)o(t)c(to)j (b)q(e)h(in)e(implemen)o(tations)e(of)365 457 y Fg(mpi)p Fh(.)291 540 y(14.)20 b Fe(Descriptor)10 b(wildcard:)16 b Fh(In)c(the)g(spirit)f(of)g (descriptor)i(uni\014cation)d(\(See)j(Note)f(1\))365 589 y(the)k(three)h (named)d(constan)o(ts)i Fc(MPI)p 935 589 14 2 v 15 w(?D)p 994 589 V 15 w(WILD)f Fh(can)g(b)q(e)h(collapsed)f(in)o(to)g(something)365 639 y(lik)o(e)e Fc(MPI)p 510 639 V 15 w(DESC)p 613 639 V 15 w(WILD)p Fh(.)365 706 y(If)k(pro)q(cess)h(descriptors)g(are)g(replaced)f (with)g(global)e(unique)h(pro)q(cess)j(iden)o(ti\014ers)365 756 y(then)14 b(p)q(erhaps)h(the)f(wildcard)f(pro)q(cess)i(iden)o(ti\014er)f (v)n(alue)f(can)h(b)q(e)g(the)g(same)e(as)i(the)365 805 y(wildcard)g(tag)f(v) n(alue,)g(and)h(the)g(same)f(named)g(constan)o(t.)291 888 y(15.)20 b Fe(Con)o(text)g(forms:)26 b Fh(The)19 b(n)o(ull)f(form)e(is)i(lik)o(e)g (PVM)h(3.)31 b(It)18 b(is)g(general)h(purp)q(ose,)365 938 y(but)13 b(not)f(particularly)f(expressiv)o(e.)19 b(It)13 b(do)q(es)g(not)f(pro)o (vide)g(facilities)f(for)h(writers)h(of)365 988 y(parallel)g(libraries.)18 b(It)c(has)g(the)g(p)q(oten)o(tial)f(to)h(pro)o(vide)g(maxim)n(um)c(p)q (erformance.)365 1054 y(The)21 b(closed)g(form)e(is)h(lik)o(e)f Fg(zipcode)p Fh(.)37 b(It)21 b(is)f(expressiv)o(e)i(in)d(SPMD)i(programs)365 1104 y(where)d(noncomm)o(uni)o(cativ)o(e)c(distinct)i(data)g(driv)o(en)g (parallel)g(computations)e(can)365 1154 y(b)q(e)h(p)q(erformed)f(concurren)o (tly)m(.)19 b(It)14 b(pro)o(vides)g(facilities)f(for)h(writers)h(of)e(SPMD)h (lik)o(e)365 1204 y(parallel)f(libraries.)365 1270 y(The)j(op)q(en)g(form)e (is)i(lik)o(e)e Fg(chimp)p Fh(.)23 b(It)16 b(is)f(expressiv)o(e)j(in)d(MIMD)g (programs)f(where)365 1320 y(comm)o(unicativ)o(e)e(data)j(driv)o(en)g (parallel)f(computations)f(can)i(b)q(e)h(p)q(erformed)f(con-)365 1370 y(curren)o(tly)m(.)k(It)14 b(pro)o(vides)g(facilities)f(for)g(MIMD)h (lik)o(e)f(parallel)g(libraries.)291 1453 y(16.)20 b Fe(Rank)c(wildcard:)g Fh(Since)e(rank)g(is)f(an)h(in)o(teger)g(lik)o(e)e(message)i(tag,)e(p)q (erhaps)j(they)365 1503 y(should)f(ha)o(v)o(e)g(the)g(same)f(wildcard)h(v)n (alue,)e(and)i(the)h(same)e(named)f(constan)o(t.)291 1586 y(17.)20 b Fc(MPI)p 434 1586 V 15 w(NULL)p Fe(:)14 b Fh(I)g(am)f(follo)o(wing)e(the)k (spirit)f(of)g(con)o(text)h(uni\014cation)e(\(See)i(Note)g(1\))f(in)365 1636 y(the)i(prop)q(osal)f(text)g(here.)23 b(There)16 b(ma)o(y)e(b)q(e)h(adv) n(an)o(tage)f(in)h(de\014ning)g(the)h(v)n(alue)e(of)365 1685 y(the)h(n)o(ull)e(descriptor)j(to)e(b)q(e)h(the)g(ANSI)f(C)g(constan)o(t)h Fc(NULL)p Fh(,)e(or)h(ev)o(en)h(de\014ning)f(the)365 1735 y(v)n(alue)g(to)f (b)q(e)i(exactly)f(zero)h(\(ev)o(ery)f(rule)g(ha)o(ving)f(a)h(useful)g (exception\).)262 1872 y Fi(1.4)64 b(Conclusion)262 1963 y Fh(This)9 b(c)o(hapter)j(has)e(presen)o(ted)i(and)e(discussed)i(a)d(prop)q (osal)h(for)g(comm)o(unicatio)o(n)e(con)o(texts)262 2013 y(within)h Fg(mpi)p Fh(.)16 b(In)10 b(the)h(prop)q(osal)e(pro)q(cess)j(groupings)e(app)q (eared)h(as)f(frames)f(\(or)h(templates\))262 2063 y(for)16 b(the)i(construction)g(of)f(comm)o(unicati)o(on)d(con)o(texts,)19 b(and)e(comm)o(unicatio)o(n)e(con)o(texts)262 2113 y(retained)f(certain)h (prop)q(erties)g(of)e(the)i(frames)e(used)h(in)g(their)g(construction.)957 2574 y(15)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 23 14:34:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25818; Tue, 23 Mar 93 14:34:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25950; Tue, 23 Mar 93 14:33:51 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 14:33:50 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25936; Tue, 23 Mar 93 14:33:47 -0500 Date: Tue, 23 Mar 93 19:33:41 GMT Message-Id: <16020.9303231933@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Context proposals overview To: SNIR@watson.ibm.com, mpi-context@cs.utk.edu In-Reply-To: Marc Snir's message of Tue, 23 Mar 93 13:10:01 EST Reply-To: lyndon@epcc.ed.ac.uk Marc writes (in reply to Jim): > Proposal I (Marc Snir) > ---------------------- > Against: > The receiver of a message has to both know the group of the sender, > and be a part of it. This feels like it makes servers hard, but may be > ok. If you keep having to go into the ALL group, then the security > which was the whole point of a context is lost. > > >>> One can, of course, have several distinct contexts that include > >>> all processes. Sure, I see, if there are M distinct services then you create M contexts which are replicates of the ALL context and messages for each service are protected from one another. Of course this is ignoring the nagging realisation that the client has to remember which rank within the all group the server has, when the expressive way of doing this is actually to describe a SERVER context and a CLIENT context, forget about the ALL context, and let the clients send messages to the server context and the server send messages to the client context. Well, actually this is only part of the story, because we should think about the CLIENT as being a concerted entity (a parallel client process) and allow there to me multiple distinct clients. Don't things look rather bad for this proposal now. Well, again we left something out of the story, because the SERVER itself could compose multiple processes acting as a concerted entity (a parallel server process). Now it's just getting a bit messy for the client to remember the set of server indices? It's worse for the server remembering the set of client indices. You bet the programmer ends up implementing inter-context communication by hand. Finally, are we in good shape when the parallel server for service X is the parallel client for service Z, which has a parallel server? I suggest not. Okay, well this is beginning to get a wee bit hypothetical, I don't know of anyone programming at this final level of complexity quite yet. But, I do know of people programming at the parallel servers - parallel clients level. > I don't understand how to build a group/context using only point to > point messages. I still seem to have the bootstrap problem that I need > the new context to safely receive the message which will tell me what > the new context is. > > >>> Well, we have an existential proof, since we support dynamic group > >>> creation in our system. A new context is created within an old, > >>> preexisting context; so ALL need to be there from start. > There is an implementation of this kind of group creation in your system - existence proof that this is implementable - but if your system does not use the point-to-point facilities of MPI then this may not be an existence proof that this is implementable with the point-to-point facilities of MPI. This doesn't bother me, personally. I cannot bring myself to being an advocate that the group and context constructors and destructors have to be implementable using the point-to-point section of MPI. This just seems to have the effect of restricting the contructors and desctructors. The argument is not the same as the similar argument for collective communications -- there should be few constructors and destructors whereas there may be many collective communications. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Mar 23 18:24:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01132; Tue, 23 Mar 93 18:24:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06265; Tue, 23 Mar 93 18:23:42 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 23 Mar 1993 18:23:40 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06257; Tue, 23 Mar 93 18:23:37 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Tue, 23 Mar 93 15:19 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA08184; Tue, 23 Mar 93 15:17:47 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA18045; Tue, 23 Mar 93 15:17:43 PST Date: Tue, 23 Mar 93 15:17:43 PST From: rj_littlefield@pnlg.pnl.gov Subject: overview and "sketch VII" To: jim@meiko.co.uk, mpi-context@cs.utk.edu Cc: d39135@sodium.pnl.gov, lyndon@epcc.ed.ac.uk, snir@watson.ibm.com Message-Id: <9303232317.AA18045@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu Many thanks to Jim for his summary of various proposals. The earlier proposals have become rather Talmudic, with layer upon layer of comments so detailed that it is difficult to tell what are the fundamental differences. Even after integration of the comments, proposals are now so long that they require much study. (Proposal VI is gonna take a while, Lyndon.) Rather than comment further on Jim's summary or any other specific proposal, I would like to sketch anew a possible synthesis of key ideas selected from all the proposals. This is a somewhat expanded version of the sketch that I distributed a couple of days ago with a confusing choice of title. Following both of Lyndon's suggestions, I will now call it "Sketch VII". My hope here is to provide a BRIEF description that may help clarify our thinking and contribute to a unified proposal. BASE FEATURES Functionality . All descriptors/identifiers are local and opaque. . A "context" consists of a collection of processes that have agreed amongst themselves regarding context-relative process names ("ranks"), and have also negotiated a protection mechanism to keep contexts separate. . There is a predefined context holding all the initial processes. . Contexts are formed and destroyed by loosely synchronous calls in only those processes belonging to the context. . Point-to-point messages can be sent and received within a context, in which case both sender and receiver are identified by rank. . Groups are not equal to contexts. From the user's standpoint, a group consists of a base context (from which other contexts can be created cheaply by copying), plus topology information, plus a facility for attaching arbitrary information to the group descriptor ("cacheing"). Performance/Implementation Expectations . Assembling a new collection of processes requires communication and imposes actual synchronization. The implementation is expected to be scalable and to cost no more than a point-to-point fanin/fanout on the participating processes. . "Copying" a context does not require communication and does not impose actual synchronization. (Use a counting strategy.) . There is no concept of a globally unique context value. If desired by the MPI implementor, each process can choose its own message selector value and broadcast it to the other processes upon context formation. EXTENDED FEATURES Functionality . Process, group, and context descriptors can be passed around by special mechanisms. This allows a process to obtain the descriptor for a group or context of which it is not a member. If a process receives a descriptor by one of these mechanisms, then that process assumes responsibility for explicitly releasing it when the descriptor is no longer needed or becomes stale. . Point-to-point communications can be extended to allow a process to send a message to any context for which it holds a descriptor. The receiver needs an easy way to identify and/or select on sender. (Lyndon's proposal VI introduces an "open context" form that works well if sender and receiver hold descriptors for each other's contexts, but is not clear on how the sender is identified if the receiver does not know the sender's context.) . Registration facilities can be added to allow global naming of processes, contexts, and groups. The registration facility translates between globally unique names and process-local descriptors. (I.e., to register a name, one specifies the name and matching descriptor; to lookup, one specifies the name and receives a descriptor in reply.) This capability, and none of the capabilities before it, requires an asynchronous server. Happy thinking... --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Wed Mar 24 09:39:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14644; Wed, 24 Mar 93 09:39:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16981; Wed, 24 Mar 93 09:38:32 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 24 Mar 1993 09:38:31 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16973; Wed, 24 Mar 93 09:38:30 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06527; Wed, 24 Mar 93 08:32:44 CST Date: Wed, 24 Mar 93 08:32:44 CST From: Tony Skjellum Message-Id: <9303241432.AA06527@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: status of this subcommittee Hi, with the strong efforts of Lyndon, Rik, and the full-proposal contributed by Marc, we have two proposals, and I am writing the last one now. These will be put to your straw poll by next Monday. Goal of straw poll is to rank proposals for order of presentation and precedence by committee. I will keep you informed. Current proposals: I - by Snir VI/VII - by Littlefield & Clarke - Tony From owner-mpi-context@CS.UTK.EDU Wed Mar 24 09:49:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14858; Wed, 24 Mar 93 09:49:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17436; Wed, 24 Mar 93 09:49:15 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 24 Mar 1993 09:49:14 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17425; Wed, 24 Mar 93 09:49:08 -0500 Date: Wed, 24 Mar 93 14:48:58 GMT Message-Id: <16984.9303241448@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: status of this subcommittee To: Tony Skjellum , mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Wed, 24 Mar 93 08:32:44 CST Reply-To: lyndon@epcc.ed.ac.uk Hi Tony > Goal of straw poll is to rank proposals for order of presentation and > precedence by committee. I will keep you informed. > > Current proposals: > I - by Snir > VI/VII - by Littlefield & Clarke Correction: VI/VII should be replaced by VII. Cheers Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 24 09:52:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14948; Wed, 24 Mar 93 09:52:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17626; Wed, 24 Mar 93 09:51:53 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 24 Mar 1993 09:51:52 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17618; Wed, 24 Mar 93 09:51:49 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA07283 (5.65c/IDA-1.4.4 for ); Wed, 24 Mar 1993 09:51:44 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15569; Wed, 24 Mar 93 14:51:40 GMT Date: Wed, 24 Mar 93 14:51:40 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303241451.AA15569@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA06620; Wed, 24 Mar 93 14:48:11 GMT To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Subject: Proposal VI' Content-Length: 1687 Lyndon, Please help me, I am having trouble understanding proposal VI'. It seems to me that there is a bootstrap problem with (at least) the NULL Context form of point to point. (I'd hoped that this would be the simplest and easiest to understand and use, so I started with it). First problem ------------- ALL descriptors are local (including process descriptors). Therefore before I can send a message to someone I have to find out (or construct) their (local to me) process descriptor. How do I do this ? They can send it to me, except that the problem just moves to them, as they don't know my address either ! (We had this exact problem in CSN, which is why we introduced the CSN nameserver at a known, fixed, address. Having used it I don't actually much like that solution !) Second Problem (more of a feature). ----------------------------------- Since the descriptors are ALL local, it will be extremely hard to make sense of trace/debug information. Consider code like this MPI_PD from; MPI_CRECV(... ,&from, ...); /* Wildcard receive but tell me where it came from */ printf("[%08X] Received from %8X\n",MPI_MYPD(), from); Coupled with code like this printf("[%08X]Sending to %08X\n", MPI_MYPD(), to); The output will be TOTALLY meaningless and confusing ! e.g. It could very easily look like this [00000001] Sending to 00000002 [00000001] Received from 00000002 etc. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Wed Mar 24 10:43:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17289; Wed, 24 Mar 93 10:43:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19426; Wed, 24 Mar 93 10:18:30 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 24 Mar 1993 10:18:29 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19407; Wed, 24 Mar 93 10:18:23 -0500 Date: Wed, 24 Mar 93 15:18:19 GMT Message-Id: <17017.9303241518@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Proposal VI' To: jim@meiko.co.uk (James Cownie) In-Reply-To: James Cownie's message of Wed, 24 Mar 93 14:51:40 GMT Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi Jim > Please help me, I am having trouble understanding proposal VI'. Sure. First thing, Rik and I have done a "merger" of Proposals V and VI, to produce a Proposal VII. This is a cut down of VI with inclusion of group/context user cache capability. I think your queries propagate into Proposal VII, and I shall reply. > > It seems to me that there is a bootstrap problem with (at least) the > NULL Context form of point to point. (I'd hoped that this would be the > simplest and easiest to understand and use, so I started with it). > There kind of is, indeed, and I know about that. Hence I explore in the discussion having global process identifiers instead of local process descriptor handles. Proposal VII went to identifiers and then back to descriptors as I and Rik played tug-of-war with the thing. > First problem > ------------- > ALL descriptors are local (including process descriptors). > > Therefore before I can send a message to someone I have to find out > (or construct) their (local to me) process descriptor. Yes. > How do I do this ? > They can send it to me, except that the problem just moves to them, as > they don't know my address either ! Indeed. For processes in which you share the initial group, then they can be sent in a message using closed context communication, or more easily you can use the (group, rank) -> process mapping routine. Of course this doesnt let you find out about any processes in a different initial group (either becaus either because the program was kicked off that way or because the other processes were created later, although that kind of process creation is not *yet* in MPI). > (We had this exact problem in CSN, which is why we introduced the > CSN nameserver at a known, fixed, address. Having used it I don't > actually much like that solution !) Well as you know we used that CS~Tools software a lot for a lot of different things. For data driven parallel programming we never liked it either. However for modular applications --- distinct data driven (SPMD like) modules composed as module graphs --- we find the name server approach for register and lookup of group very convenient and expressive indeed! > Second Problem (more of a feature). > ----------------------------------- > Since the descriptors are ALL local, it will be extremely hard to make > sense of trace/debug information. > > Consider code like this > > MPI_PD from; > > MPI_CRECV(... ,&from, ...); /* Wildcard receive but tell me > where it came from */ > printf("[%08X] Received from %8X\n",MPI_MYPD(), from); > > Coupled with code like this > > printf("[%08X]Sending to %08X\n", MPI_MYPD(), to); > > The output will be TOTALLY meaningless and confusing ! > > e.g. It could very easily look like this > [00000001] Sending to 00000002 > [00000001] Received from 00000002 > Yes, I should hope that it would! This is a good point. So I suggest that for this kind of thing one needs to either use one of the context oriented forms, or we need just to add a (group, process) -> rank mapping (which might be slow, but its only debugging after all), or a descriptor attribute query procedure to return a global unqiue process identifer which will be embedded into the descriptor anyway. So for me, your helpful questions translate into a requirement to add one or two additional procedures into Proposal VII as outlined in the above paragraph. This will have to happen at or after the meeting, since Proposal I and VII are already en route to Steve Otto our Draft Editor. By the way, Jim, I'd be real grateful if you would explain to me what the Elan/Elite "contexts" that I have heard mentioned do. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Mar 24 12:06:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20034; Wed, 24 Mar 93 12:06:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24341; Wed, 24 Mar 93 12:05:49 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 24 Mar 1993 12:05:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24327; Wed, 24 Mar 93 12:05:42 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06808; Wed, 24 Mar 93 10:57:27 CST Date: Wed, 24 Mar 93 10:57:27 CST From: Tony Skjellum Message-Id: <9303241657.AA06808@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: draft of MPI Context proposals (Latex) \documentstyle{report} \begin{document} \title{Context Subcommittee Proposals} \author{Lyndon~Clarke \and Anthony Skjellum \and Rik~Littlefield \and Marc~Snir} \date{March 24, 1993} \maketitle \newpage %======================================================================% % BEGIN "Proposal I" %======================================================================% % BEGIN "Proposal I" % Written by Marc Snir % Edited by Lyndon J. Clarke % March 1993 % \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \chapter{Proposal I --- Marc~Snir} \section{Contexts} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \subsection{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \subsection{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \subsection{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \subsection{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \section{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \subsection{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } % % END "Proposal I" %======================================================================% %\end{document} %\documentstyle{report} % \chapter{Proposal VII --- Lyndon~J~Clarke \& Rik~J~Littlefield} %---------------------------------------------------------------------- % Introduction \section{Introduction} This chapter is similar in basic principles to Proposal~I and includes all of the functionality of that proposal as a subset --- it extends in several ways and differs in some details. Certain features of other, now defunct, proposals discussed in the context subcommittee are included. In particular, this chapter proposes that: \begin{enumerate} \item Contexts and groups are not identical. A context is always associated with one group, but a group may have several contexts. Properties of groups are inherited by all of the associated contexts, for example process rank. \item Context and group descriptors can be explicitly transferred to processes that are not members of the context or group. \item In point-to-point messages, processes can be identified in any of three ways: by process, by rank in a shared context, or by ranks in separate sender and receiver contexts. \item A ``cache'' facility is provided that allows modules to attach arbitrary information to both contexts and groups. \end{enumerate} These extensions are somewhat independent of each other. The first reflects the observation that multiple modules often operate within each process group, so that context formation should be lighter weight than group formation. The second and third together provide expressive support for communication between modules within different groups of processes. The fourth allows modules to be significantly faster in common cases, without complicating their interface to the application. Much of this proposal must be viewed as recommendations to other subcommittees of {\sc mpi}, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language, only for purposes of discussion. %---------------------------------------------------------------------- % Processes \section{Processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and {\sc mpi} does not distinguish such threads. \subsection*{Process Identifier} Each process is identified by a process-local {\it process handle}, which is a reference to a {\it process descriptor} of undefined size and opaque structure. In a static process model process handles can be obtained by mapping from a group (or context) and rank. In a future extension for dynamic processes, handles may be returned by process creation functions. {\sc mpi} provides a procedure which returns a handle for the calling process. \begin{verbatim} process = mpi_my_process() \end{verbatim} \subsection*{Process Creation \& Destruction} This proposal makes no statements regarding creation and destruction of processes. {\sc mpi} provides facilities for descriptor transmission allowing the user to explicitly transfer a process decriptor from one process to another. These facilities are described below. %---------------------------------------------------------------------- % Groups \section{Process Groups} This proposal views a process group as an ordered collection of (references to) distinct processes, the membership and ordering of which does not change over the lifetime of the group. The canonical representation of a group is a one-to-one map from the integers $(0, 1, \ldots, N-1)$ to handles of the $N$ processes composing the group. There may be structure associated with a process group defined by a process topology. This proposal makes no further statements regarding such structures. \subsection*{Group Identifier} Each group is identified by a process-local {\it group handle}, which is a reference to a {\it group descriptor} of undefined size and opaque structure. The initialization of {\sc mpi} makes each process a member of the ``initial'' group. {\sc mpi} provides a procedure that returns a handle to this group. \begin{verbatim} group = mpi_initial_group() \end{verbatim} {\sc mpi} provides facilities for descriptor transmission allowing the user to explicitly transfer a group descriptor from one process to another. \subsection*{Group Creation and Deletion} {\sc mpi} provides facilities which allow users to dynamically create and delete process groups. The procedures described here generate groups which are static in membership. {\sc mpi} provides a procedure which allows users to create one or more groups which are subsets of existing groups. \begin{verbatim} groupb = mpi_group_partition(groupa, key) \end{verbatim} This procedure creates one or more new groups {\tt groupb} which are distinct subsets of an existing group {\tt groupa} according to the supplied values of {\tt key}. This procedure is called by and synchronises all members of {\tt groupa}. {\sc mpi} provides a procedure which allows users to create a group by permutation of an existing group. \begin{verbatim} groupb = mpi_group_permutation(groupa, rank) \end{verbatim} This procedure creates one new group with the same membership as {\tt groupa} with a permutation of process ranking, and returns the created group descriptor in {\tt groupb}. It is called by and synchronises all members of {\tt groupa}. {\sc mpi} provides a procedure which allows users to create a group by explicit definition of its membership as a list of process handles. \begin{verbatim} group = mpi_group_definition(listofprocess) \end{verbatim} This procedure creates one new group {\tt group} with membership and ordering described by the process handle list {\tt listofprocess}. It is called by and synchronises all processes identified in {\tt listofprocess}. {\sc mpi} provides a procedure which allows users to delete user created groups. \begin{verbatim} mpi_group_deletion(group) \end{verbatim} This procedure deletes an existing group {\tt group}. It is called by and synchronises all members of {\tt group}. {\sc mpi} may provide additional procedures which allow users to construct process groups with a process group topology. \subsection*{Group Attributes} {\sc mpi} provides a procedure which accepts a valid group handle and returns the rank of the calling process within the identified group. \begin{verbatim} rank = mpi_group_rank(group) \end{verbatim} {\sc mpi} provides a procedure which accepts a valid group handle and returns the number of members, or {\it size}, of the identified group. \begin{verbatim} size = mpi_group_size(group) \end{verbatim} {\sc mpi} provides a procedure which accepts a valid group handle and process order number, or {\it rank}, and returns the valid process handle to which the supplied rank maps within the identified group. \begin{verbatim} process = mpi_group_process(group, rank) \end{verbatim} {\sc mpi} may provide additional procedures which allow users to determine the process group topology attributes. {\sc mpi} provides a group descriptor cache facility which allows the user to attach attributes to group descriptors. %---------------------------------------------------------------------- % Contexts \section{Communication Contexts} This proposal views a communication context as the combination of a process group and a protection mechanism that avoids collision between messages sent to different contexts. The context inherits process ranking from its associated group, referred to as a {\it frame}. Each process group may be used as a frame for multiple contexts. \subsubsection*{Context Identifier} Each context is identified by a process-local {\it context handle}, which is a reference to a {\it context descriptor} of undefined size and opaque structure. The creation of a process group allocates a {\it base context} which inherits the created group as a frame and can be thought of as an attribute of the created group. {\sc mpi} provides a procedure which accepts a valid group handle and returns a handle to the base context within the identified group. \begin{verbatim} context = mpi_base_context(group) \end{verbatim} {\sc mpi} provides facilities for descriptor transmission allowing the user to explicitly transfer a context descriptor from one process to another. \subsubsection*{Context Creation and Deletion} {\sc mpi} provides facilities which allows user to dynamically create and delete contexts in addition to the base context associated with a process group. Contexts created in this fashion can be thought of as copies of the base context of the process group. {\sc mpi} provides a procedure which allows users to create contexts. This procedure accepts the handle of a group of which the calling process is a member, and returns a handle to the new context. \begin{verbatim} context = mpi_context_creation(group). \end{verbatim} This procedure must be called loosely synchronously by all members of {\tt group}. The procedure may not actually synchronize the member processes --- it is suggested that this is a lightweight procedure that can be implemented so as to not require interprocess communication. {\sc mpi} provides a procedure which allows users to delete user created contexts. The procedure accepts a context handle that was created by the calling process and deletes the identified context. \begin{verbatim} mpi_context_deletion(context) \end{verbatim} This procedure has the same synchronization behavior as context creation. \subsubsection*{Context Attributes} {\sc mpi} provides a procedure which allows users to determine the process group that is the frame of a context. \begin{verbatim} group = mpi_context_frame(context) \end{verbatim} {\sc mpi} provides a group descriptor cache facility which allows user to attach attributes to group descriptors. %---------------------------------------------------------------------- % Descriptors \section{Descriptor Facilities} This section describes the descriptor transmission and user cache facilities. \subsection*{Transmission Facility} {\sc mpi} provides a mechanism whereby the user can transmit a valid descriptor in a message such that the received descriptor handle is valid. This can be integrated with the capability to transmit typed messages, and it is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_DSCR\_TYPE}. There are other reasonable approaches to providing this facility. The descriptor is translated as necessary to be meaningful on the destination process, storage is allocated for it, and a handle to that storage is returned. Decorations are not transmitted. Handles are guaranteed to be unique within each process --- if processes A and B independently send to process C a descriptor for an object D, then process C will get two copies of the same handle. As with all transfers of descriptors, the receiving process is responsible for releasing the descriptor and its handle when it is no longer needed or becomes stale. MPI provides a procedure which frees a descriptor. \begin{verbatim} mpi_free_dscr(handle) \end{verbatim} A descriptor registry service which allows descriptors to be identified by name would be a useful additional feature. This service can be implemented at the user level using the point-to-point chapter of {\sc mpi} and the descriptor transmission facilities. These services can be deferred in this session of {\sc mpi}. \subsection*{Cache Facility} {\sc mpi} provides a ``cache'' facility that allows an application to attach arbitrary pieces of information, called {\em decorations}, to context and group descriptors. Decorations are local to the process and are not included if the descriptor is sent to another process. This facility is intended to support optimizations such as saving persistent communication handles and recording topology-based decisions by adaptive algorithms. {\sc mpi} provides the following services related to cacheing: \begin{description} \item [Generate key:] Generate cache key. \begin{verbatim} keyval = mpi_GetDecorationKey() \end{verbatim} \item [Store decoration:] Store decoration in cache by key. \begin{verbatim} mpi_SetDecoration(handle, keyval, decoration_val, decoration_destructor_routine) \end{verbatim} \item [Retrieve decoration:] Retrieve decoration from cache by key. \begin{verbatim} mpi_TestDecoration(handle,keyval,decoration) \end{verbatim} \item [Delete decoration:] Delete decoration from cache by key. \begin{verbatim} mpi_DeleteDecoration(handle,keyval) \end{verbatim} \end{description} Each decoration consists of a pointer or a value of the same size as a pointer, and would typically be a reference to a larger block of storage managed by the module. As an example, a global operation using cacheing to be more efficient for aall contexts of a group after the first call might look like this: {\small \begin{verbatim} static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_KEY_TYPE gop_key; /* key for this module's stuff */ efficient_global_op (context, ...) int context_handle; { struct gop_stuff_type *gop_stuff; /* whatever we need */ int group_handle = mpi_context_frame(context_handle); if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetDecorationKey()) ) { MPI_abort ("Insufficient keys available"); } } if (MPI_TestDecoration (group_handle,gop_key,&gop_stuff)) { /* This module has executed in this group before. We will use the cached information */ } else { /* This is a group that we have not yet cached anything in. We will now do so. */ gop_stuff = /* malloc a gop_stuff_type */ /* ... fill in *gop_stuff with whatever we want ... */ MPI_SetDecoration (group_handle, gop_key, gop_stuff, gop_stuff_destructor); } /* ... use contents of *gop_stuff to do the global op ... */ } gop_stuff_destructor (gop_stuff) /* called by MPI on group delete */ struct gop_stuff_type *gop_stuff; { /* ... free storage pointed to by gop_stuff ... */ } \end{verbatim} } The cache facility could also be provided for process descriptors, but it is less clear how such provision would be useful. It is suggested that the cache store, retrieve and delete decoration procedures should fail when applied to a process descriptor handle. %---------------------------------------------------------------------- % Point-to-point \section{Point-to-Point Communication} This proposal recommends three forms for {\sc mpi} point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a {\tt Send} operation of form X cannot match with a {\tt Receive} operation of form Y, requiring that form is embedded into the message envelope. The three forms are described, followed by considerations of uniform integration of these forms in the point-to-point communication chapter of {\sc mpi}. \subsection*{Null Context Form} The {\it null context\/} form contains no message context. Message selection and addressing are expressed by \begin{verbatim} (process, tag) \end{verbatim} where: {\tt process} is a process handle; {\tt tag} is a message tag. {\tt Send} supplies the {\tt process} of the receiver. {\tt Receive} supplies the {\tt process} of the sender. {\tt Receive} can wildcard on {\tt process} by supplying the wildcard descriptor handle value {\tt MPI\_WILDCARD}. In this case the receiver may have obtained the process descriptor of the sender, and the null descriptor handle {\tt MPI\_NULL} is returned in the relevant point-to-point enquiry procedure. \subsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. Message selection and addressing are expressed by \begin{verbatim} (context, rank, tag) \end{verbatim} where: {\tt context} is a context handle; {\tt rank} is a process rank in the frame of {\tt context}; {\tt tag} is a message tag. The calling process must be a member of the frame of {\tt context}. {\tt Send} supplies the {\tt context} of the receiver (and sender), and the {\tt rank} of the receiver. {\tt Receive} supplies the {\tt context} of the sender (and receiver), and the rank of the sender. The {\tt (context, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process identifier of the receiver (sender). {\tt Receive} cannot wildcard on {\tt context}. {\tt Receive} can wildcard on {\tt rank} by supplying the wildcard integer {\tt MPI\_DONTCARE}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. Message selection and addressing are expressed by \begin{verbatim} (lcontext, rcontext, rank, tag) \end{verbatim} where: {\tt lcontext} is a context handle; {\tt rcontext} is a context handle; {\tt rank} is a process rank in the frame of {\tt rcontext}; {\tt tag} is a message tag. The calling process must be a member of the frame of {\tt lcontext} and need not be a member of the frame of {\tt rcontext}. {\tt Send} supplies the context of the sender in {\tt lcontext}, the context of the receiver in {\tt rcontext}, and the {\tt rank} of the receiver in the frame of {\tt rcontext}. {\tt Receive} supplies the context of the receiver in {\tt lcontext}, the context of the sender in {\tt rcontext}, and the {\tt rank} of the sender in the frame of {\tt rcontext}. The {\tt (rcontext, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process identifier of the receiver (sender). {\tt Receive} cannot wildcard on {\tt lcontext}. {\tt Receive} can wildcard on {\tt rcontext} by supplying the wildcard descriptor handle value {\tt MPI\_WILDCARD}, in which case it must also wildcard on {\tt rank} since the process descriptor of the sender cannot be determined. In this case the receiver may not have obtained the context descriptor of the sender, and the null descriptor handle {\tt MPI\_NULL} is returned in the relevant point-to-point enquiry procedure. {\tt Receive} can wildcard on {\tt rank} by supplying the wildard intger value {\tt MPI\_DONTCARE}. \subsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point chapter of {\sc mpi} by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with form. This is at the expense of multiplying the number of {\tt Send} and {\tt Receive} procedures by a factor of three, and some further but trivial work with details of the current point-to-point chapter which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Two options are now described, each based on retention and extension of the framework of the closed and open contexts forms. The framework of the open context form could be adopted and extended. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, process, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, context, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcontext, rcontext, rank, tag}), which is of course natural. The framework of the closed context form could be adopted and extended. The null context form is expressed as {\tt (MPI\_NULL, process, tag)}, which is marginally inconvenient, and requires that descriptor handles are expressed as intgers. The closed context form is expressed as {\tt (context, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt context} field as ``shorthand notation'' for the {\tt (lcontext, rcontext)} pair at the expense of introducing some trickery. We define a ``duplet descriptor'' which is formally composed of two references to contexts, and provide a procedure which constructs such a descriptor given two context descriptors. Both {\tt Send} and {\tt Receive} accept a duplet descriptor in {\tt context}, are able to distinguish the duplet descriptor from a singlet descriptor, and treat the duplet as shorthand notation. It is conjectured that using this framework is the best choice for {\sc mpi}. %---------------------------------------------------------------------- % Point-to-point \section{Collective Communication} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context (thus frame) in which they are to operate. {\sc mpi} does plan to describe symmetric collective communication operations. It is not possible to determine whether this proposal is sufficient to allow implementation of the collective communication chapter of {\sc mpi} in terms of the point-to-point chapter of {\sc mpi} without loss of generality, since the collective operations are not yet defined. Asymmetric collective communication operations, especially those in which sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (a duplet descriptor) which identify the contexts (thus frames) in which they are to operate. {\sc mpi} does not plan to describe asymmetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model, which are composed of communicative functionally distinct process groups. These services can be deferred in this session of {\tt mpi}. %---------------------------------------------------------------------- % Conclusion \section{Conclusion} This chapter presented a proposal for communication contexts and process groups with {\sc mpi}. In the proposal process groups are created dynamically and are static in membership. Associated with each process group are one or more communication contexts which inherit process ranking. The recommendations for point-to-point communication are powerful. The proposal provides process addressed communication which occurs within an extended context. The proposal also contains closed context communication addressed in terms of context and rank which protects messages belonging to one context from those belonging to other contexts. The proposal also contains open context communications adressed in terms of sender context, receiver context, and process rank which provides expressive power for intercommunication between modules within different groups. The proposal is extensible to a number of features which might be included in future sessions of {\sc MPI}, for example: dynamic processes; dynamic groups; multiple group collective communications. % % END "Proposal VII" %======================================================================% % % BEGIN "Proposal III" % Anthony Skjellum % March 1993 % \chapter{Proposal III - A.~Skjellum {\em et al.}} %---------------------------------------------------------------------- % Introduction \section{Introduction} This chapter takes a slightly different approach to contexts and groups, than does Proposal~VII. It is of roughly equal conceptual ``power'' as Proposal~VII, with some differences. As appropriate, this chapter borrows directly from Proposal~VII, by Clarke and Littlefield. \begin{enumerate} \item Contexts are supported to discriminate between messages in the system. A context is a conceptual extension of the tag space into a system-defined part (not wildcardable), and a totally user-defined part (the traditional 32-bit tag). \item A context is a lower-level concept than a group, so that contexts not associated with groups are permitted. This permits the user to develop codes that build on the server model, or which build up groups dynamically (not otherwise supported by MPI1). \item Groups are used to describe cooperative communication in the system. Groups have one or more context of communication associated with them. When created, a group is given a context of communication. \item Context and group descriptors can be explicitly transferred to processes that are not members of the context or group. \item In point-to-point messages, processes can be identified in either of two ways: by opaque process identifier, or by rank in a group; in either case, communication scope is within a given context. \item The cache facility, allowing groups to add additional information (described in Proposal~VII) is embraced by this Proposal, with reservations as noted. The possible need to omit this cacheing feature from MPI1 should not invalidate the remainder of Proposal~VII from further consideration (severability). \end{enumerate} %---------------------------------------------------------------------- % Processes \section{Processes} This Proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and {\sc MPI\/} does not distinguish such threads. {\sc MPI\/} shall be thread-aware, but not thread supporting, so we make every attempt to make thread safe programs possible in defining what follows. \subsection*{Process Identifier} Each process is identified by an opaque process identifier, which is associable to a {\it process descriptor} of undefined size and opaque structure, through {\sc MPI} accessor calls. In a static process model process identifiers can be obtained by mapping from a group and rank. In a future extension for dynamic processes, identifiers may be returned by process creation functions. {\sc MPI} provides a procedure that returns an identifier for the calling process. \begin{verbatim} my_process = mpi_my_process() \end{verbatim} This identifier can be converted to a transmittable form by {\sc MPI} converter functions, though opaque, it is conceptually a pointer. \subsection*{Process Creation \& Destruction} This proposal makes no statements regarding creation and destruction of processes. {\sc MPI} provides facilities for identifier transmission allowing the user explicitly to transfer a process identifier from one process to another. These facilities are described below. MPI also provides means to transfer underlying information about the opaque process descriptor underlying. %---------------------------------------------------------------------- % Groups \section{Process Groups} This proposal views a process group as an ordered collection of distinct processes (via process identifiers), the membership and ordering of which does not change over the lifetime of the group. The canonical representation of a group is a one-to-one map from the integers $(0, 1, \ldots, N-1)$ to identifiers of the $N$ processes composing the group. There may be structure associated with a process group defined by a process topology. This proposal makes no further statements regarding such structures. There may be non-enumerative ways to construct and manipulate special groups, or for special machine architectures ({\em e.g.}, cohorts). This proposal makes no further statements about such special groups, other than the desirability of avoiding group-name enumeration, when possible. \subsection*{Group Identifier} Each group is identified by an opaque {\it group identifier}, which is a associable to a {\it group descriptor} of undefined size and opaque structure, through {\sc MPI} accessor functions. The initialization of {\sc MPI} makes each process a member of the ``initial'' group. {\sc MPI} provides a procedure that returns an identifier to this group. \begin{verbatim} group_ident = mpi_initial_group() \end{verbatim} {\sc MPI} provides facilities for descriptor transmission allowing the user explicitly transfer a group descriptor from one process to another. \subsection*{Group Creation and Deletion} {\sc MPI} provides facilities which allow users dynamically to create and delete process groups. The procedures described here generate groups which are static in membership. {\sc mpi} provides a procedure that allows users to create one or more groups which are subsets of existing groups. \begin{verbatim} new_group = mpi_group_partition(old_group, key) \end{verbatim} This procedure creates one or more new groups {\tt new\_group} which are distinct subsets of an existing group {\tt old\_group} according to the supplied values of {\tt key}. This procedure is called by and synchronises all members of {\tt new\_group}. No overlapping is permitted, so that exactly one {\tt new\_group} is achieved in each {\tt old\_group} process. The new groups have new contexts of communication. The number of new contexts depends on the number of different key values asserted. {\sc MPI} provides a procedure which allows users to create a group by permutation of an existing group. \begin{verbatim} new_group = mpi_group_permutation(old_group, rank) \end{verbatim} This procedure creates one new group with the same membership as {\tt old\_group} with a permutation of process ranking, and returns the created group descriptor in {\tt new\_group}. It is called by and synchronises all members of {\tt gha}. The new group has a new context of communication. {\sc MPI} provides a procedure that allows users to create a group by explicit definition of its membership as a list of process identifiers. \begin{verbatim} new_group = mpi_group_definition(array_of_process_ids,length, context_to_use) \end{verbatim} This procedure creates one new group {\tt new\_group} with membership and ordering described by the process handle list {\tt array\_of\_process\_ids}, of length {\tt length}. If {\tt context\_to\_use} is specified as the special context {\tt MPI\_GET\_CONTEXT}, then the system allocates the new context. Else, the system trusts the user to have allocated the context legally (see below). This procedure must be called by and synchronises all processes identified in the list. A further approach to new group definition is as follows \begin{verbatim} new_group = mpi_group_def_by_leader(leader_id, in_length, in_array_of_process_ids, context_to_use, out_array_of_process_ids, out_length) \end{verbatim} This weaker form requires all future group members to have identified only a leader (specified by {\tt leader\_id}). The leader knows the length a names of all participants. This is a synchronization of all participants. Same semantics for {\tt context\_to\_use} as above. {\sl MPI} provides a way formally to duplicate a group, in order to obtain a separate context of communication. This can be achieved by using other operations, but this procedure allows optimization in some implementations (no explicit group copy, for instance). \begin{verbatim} new_group = mpi_group_duplicate(old_group) \end{verbatim} The only difference between the old and new groups is that there is a new context of communication. {\sc MPI} provides a procedure which allows users to delete user created groups. \begin{verbatim} mpi_group_deletion(group) \end{verbatim} This procedure deletes an existing group {\tt group}. It is called by and synchronises all members of {\tt group}. {\sc MPI} may provide additional procedures which allow users to construct process groups with a process group topology. \subsection*{Group Attributes/Accessors} {\sc MPI} provides a procedure that accepts a valid group identifier and returns the rank of the calling process within the identified group. \begin{verbatim} rank = mpi_group_rank(group) \end{verbatim} {\sc MPI} provides a procedure that accepts a valid group identifier and returns the number of members, or {\it size}, of the identified group. \begin{verbatim} size = mpi_group_size(group) \end{verbatim} {\sc MPI} provides a procedure that accepts a valid group identifier {\it rank-in-group}, and returns the valid process identifier to which the supplied rank maps within the identified group. \begin{verbatim} process_id = mpi_group_process(group, rank) \end{verbatim} {\sc MPI} may provide additional procedures which allow users to determine the process group topology attributes. {\sc mpi} provides a group descriptor cache facility thath allows the user to attach attributes to group descriptors. See Proposal~VII for details. %---------------------------------------------------------------------- % Contexts \section{Communication Contexts} This proposal views a communication context as a partition of the tag space, which is a protection mechanism that avoids collision between messages sent between processes. Process groups have one or more contexts in {\sl MPI}. Unlike Proposal~VII, more contexts are obtained for a group using the above-discussed group creation and replication functions. Replication may only be formal for good implementations. This approach is viewed as simpler than what Proposal~VII describes. \subsubsection*{Context Identifier} Each context is identified by an opaque process identifier. It is conceptually an integer assigned by the system to partition a large tag space into a user-defined and system-controlled subspaces. This stategy provides the minimal level of isolation needed to build large libraries, and is close to practice. \subsubsection*{Context Creation and Deletion} {\sc MPI} provides facilities that allow user dynamically to allocate and free contexts. When contexts are used with groups, these calls are not needed. For more advanced users (such as building your own dynamic groups), these calls will be used. Above, where {\tt context\_to\_use} appears as an argument, the following call would have been used to secure such a context in advance. \begin{verbatim} mpi_context_creation(number_of_contexts_wanted,array_of_contexts, number_of_contexts_provided) \end{verbatim} This call is called by any process, with no synchronization to other processes. {\sc MPI} provides a procedure that allows users to delete user- created contexts. The procedure accepts a context identifier array, containing zero or more contexts created previously in the system. \begin{verbatim} mpi_context_deletion(context_array,length) \end{verbatim} No synchronization occurs here. The user can do erroneous things by freeing contexts that are still in use. For general applications, it may be nice to have a name service for contexts (necessary for building dynamic groups and servers, for yourself). Herewith: \begin{verbatim} mpi_associate_contexts_with_name(string_name,context_array,length) mpi_disassociate_contexts_with_name(string_name) mpi_get_contexts_by_name(string_name,max_length,out_length, context_array \end{verbatim} As with context generation, the above calls assume a simple reactive, global server, or shared name space mechanism (both achieveable easily in practice). %---------------------------------------------------------------------- % Descriptors \section{Descriptor Facilities} This section describes the descriptor transmission and user cache facilities. \subsection*{Conversion Facility} {\sc MPI} provides a mechanism whereby the user can convert a valid descriptor ({\em e.g.}, a group descriptor) idenfied through an identifier ({\em e.g.}, a group identifier) for use in a message such that the received descriptor can be reconstructed on the remote end. This can be integrated with message transmission as the user sees fit, without additional complication to the send/receive semantics of {\sc MPI}. An example follows: \begin{verbatim} error = mpi_group_group_transmit(group, group_buffer, max_length, act_length) \end{verbatim} If the buffer is not long enough to hold the information, an error occurs. A network independent format can be assumed in the {\tt group\_buffer}. Cached ``attributes'' are not transmitted (see below). \subsection*{Cache Facility} {\sc MPI} provides a ``cache'' facility that allows an application to attach arbitrary pieces of information, called {\em attributes}, to context and group descriptors. Attributes are local to the process and are not included if the descriptor is sent to another process. This facility is intended to support optimizations such as saving persistent communication handles and recording topology-based decisions by adaptive algorithms. {\sc MPI} provides the following services related to cacheing. We call our attributes `attributes'; Proposal~VII calls them (equivalently) decorations (no big difference, except naming, is anticipated). \begin{description} \item [Generate key:] Generate cache key. \begin{verbatim} keyval = mpi_get_attribute_key() \end{verbatim} \item [Store attribute:] Store attribute in cache by key. \begin{verbatim} mpi_set_attribute(handle, keyval, attribute_val, attribute_destructor_routine) \end{verbatim} \item [Retrieve attribute:] Retrieve attribute from cache by key. \begin{verbatim} mpi_Test_Attribute(handle,keyval,attribute) \end{verbatim} \item [Delete attribute:] Delete attribute from cache by key. \begin{verbatim} mpi_delete_attribute(handle,keyval) \end{verbatim} \end{description} Each attribute consists of a pointer or a value of the same size as a pointer, and would typically be a reference to a larger block of storage managed by the module. Our example will appear in a later draft, because we have semantic differences from some of the ancillary aspects of the example of Proposal~VII. The cache facility could also be provided for process identifiers, but it is less clear how such provision would be useful. It is suggested that the cache store, retrieve and delete attribute procedures should fail when applied to a process identifiers. Implementations should use AVL trees, or similar efficient data structures to provide relatively efficient access to attributes. %---------------------------------------------------------------------- % Point-to-point \section{Point-to-Point Communication} This proposal recommends two forms for {\sc MPI} point-to-point message addressing and selection: by-group notation, by process-ID notation; always, one is working in a context, as this is the fundamental management tactic of {\sc MPI} for messages. As a group always has a context at creation, and an ALL group is anticipated, this should prove fine for a static process model. We disagree significantly from Proposal~VII in what follows. The two forms are described, followed by considerations of uniform integration of these forms in the point-to-point communication chapter of {\sc MPI}. \subsection*{Group-Rank Form} The {\it group-rank\/} form permits communication between members of the same context and group. Message selection and addressing are expressed by \begin{verbatim} (group, rank, tag) \end{verbatim} where: {\tt group} is a group identifier; {\tt rank} is a process rank in that group; {\tt tag} is a message tag. The calling process must be a member of the frame of {\tt context}. {\tt Send} determines the context using information in the group identifier. It does all necessary mappings to the process identifier space. {\tt Receive} cannot wildcard on context, so a valid matching receive must refer to the same group information. {\tt Receive} can wildcard on {\tt rank} by supplying the wildcard integer {\tt MPI\_DONTCARE}. This proposal makes the following statement about the provision for wildcard on {\tt tag}. Two integer-form wildcard is needed for layering: {\tt care\_bits}, {\tt dont\_care\_bits}. Tags are matched if and only if \begin{equation} (received\_tag AND NOT dont\_care_bits) XOR care\_bits== 0 \end{equation} This general format can be used to partition the tag space for virtual topologies or other user-defined needs, and is quite important to the standard's flexibility. \subsection*{Process-Identifier Form} Communication takes place using the following parameters: \begin{verbatim} (context, process_identifier, tag) \end{verbatim} where: {\tt context} is a context identifier, {\tt process\_identifier} is a process identifier, {\tt tag} is a message tag. The calling process must be a member of the same context as the recipient. There is no reference to groups here. Contexts can have been shared by using a least common ancestor prior to this call, or by the above-mentioned context naming service. There is never wildcarding on context. Wildcarding on {\tt process\_identifier} is through {\tt MPI\_DONTCARE}. Tag wildcarding is through the integer pair described above. \subsection*{Uniform Integration} The two forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point chapter of {\sc MPI} by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with form. This is at the expense of multiplying the number of {\tt Send} and {\tt Receive} procedures by a factor of two, and some further but trivial work with details of the current point-to-point chapter which uniformly assumes a single addressing and selection form. No further details, other than naming that disambiguates the rank-group form from the process-id-context form is really needed, and the naming would seem uncontroversial. %---------------------------------------------------------------------- % Collective \section{Collective Communication} Symmetric collective communication operations are compliant with the group-rank form described above. This proposal recommends that such operations accept a group-identifier (which contains context and other information) needed to operate correctly. We recommend that the tag argument be included in collective calls where this could help with debugging. {\sc MPI} does plan to describe symmetric collective communication operations. It is impossible to determine whether this proposal is sufficient to allow implementation of the collective communication chapter of {\sc MPI} in terms of the point-to-point chapter of {\sc mpi} without loss of generality, since the collective operations are not yet defined. Asymmetric collective communication operations, especially those in which sender(s) and receiver(s) are distinct processes, should be made compliant with the group-rank form described above. {\sc MPI1} should forego non-blocking collective operations, but ask vendors to support thread models in lieu of such operations. %---------------------------------------------------------------------- % Conclusion \section{Conclusion} This proposal is substantially simpler than Proposal~VII, while incorporating many of its finer elements, specifically the ability to cache attributes, which would let users extend the usefulness of groups, in their applications. It places groups as conceptually higher objects than contexts, and removes a lot of the framework needed by Proposal~VII to accomplish inter-group transfers, by providing context-process-identifier communication and group-rank communication. Through a simple context management server, and name registry server, dynamic groups support becomes simple, and synchronizations via least-common ancestors becomes avoidable. Compared to Proposal~I, we have offered a more convenient process model, because contexts are seen clearly to partition and manage the tag space, whereas groups are seen as ways to describe collective operations, and attributes of cooperative communication. This offers a much more open interface an either VII or I, including a legitimate ability to do server-like computation. However, a simple, reactive server is needed to accomplish these services. Arguably, this server would also be needed in Proposal~VII, though not in Proposal~I. % % END "Proposal VI" %======================================================================% \end{document} From owner-mpi-context@CS.UTK.EDU Wed Mar 24 12:16:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20277; Wed, 24 Mar 93 12:16:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24774; Wed, 24 Mar 93 12:15:33 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 24 Mar 1993 12:15:31 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24747; Wed, 24 Mar 93 12:15:23 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06846; Wed, 24 Mar 93 11:04:43 CST Date: Wed, 24 Mar 93 11:04:43 CST From: Tony Skjellum Message-Id: <9303241704.AA06846@Aurora.CS.MsState.Edu> To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@aurora.cs.msstate.edu Subject: postscript of draft (uuencoded) begin 600 drafts.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0N %B MR!LX>>2D.8.&#H@8.7#8:)$C!@@I8+E4R)LZ;G2Z.6,63T88(.#:B"$# MQ(T<,I8626NR39LR;NC,*5OF3!HW4.2\&3.E#)V,=,K@<0%'L8(75,I@(;+3 MI0P8<'6.<4FFC!D0F3=W!I'0L!L0+YS +GV:S9N<&DW#%K)'S&$RNXP[Z2,$-A4IL.E0=3.'346AS($33]/]HYF> 2!Z+2]1X%Y-G,V;,%KLT6-L$E8!=CO.$&3W3TD<9I-]0EQ5=OL%$' M'6D8J%$:=MA5EQ4/SA'AA!6^IE.&;A0&PAP\0:5 AQ]2:"&'$$KHXFOLN2=4 MB6> T$8=/EVG8T4JX0'"&'7(,5%@;0"9AI!DU &'1DZ"0 ,(7M&A@&)H 3$4#[9Y),U4.D8"%BF!1((7'H)9GIT)*G=DL/!!@14=$0FQQXOU/C> M2PY-5]T+=X:1%HIAP%$&G_E]I%T=ZL'FIU @B"HG6'0P.>D@+I@ UXUS3 # M#C7@8"EUF,YP7Y_M_1E#2#74,(--EQ(*%1E>[N=A?6T::OPS7#J MH$ DZ48=/9E11AEDW#<''175,8=H+KV@++-L. LM"(\*]66"R!8(1QI?\3F" MN>A^A 6J+QC11$5;!!@7")4.Z,1E1@@A1'_WZ@N;&Z_)!UL21=Q+W&AR*#R; M&2U0FP8;?!(,@JG86@J"Q:T=QJ^!=%"1AZ(@&$N<$2#/.Z>09@0FAY G@^SO M&T+V6YULTZJT%FQBA)$>"%B D!15>5PFA%/S>@3T;G5,3,80:"2U!]1)"=$T M&Z7)4><+>Q6H5HX(:PP8<&/R:<8;;YSHV-F!]9'7T&'D,>1)O MQ>/"QK#"6\!+AAE\SG$:K;"UK%W-*FL\>,1A3-R' H,.OM^(L!F^] N*O[R% MY@*>>!J. 78-V!G!KSE''6*8J?L+Q*-1QDHM#<^[\=2&N=SRS4_Y?/15PDL\ M'D413I>IQ'./_)C*K]6\L>-+?R+TYO.>1_KWM7]\\E"17Y?J4K\JO40&)!E? M_HJ'!^T5[W_P"R#SNF23 CJ&.IB. M2#4Y6,UI9=C3',)@!Z$8RR:*88-/IJ ^+V@9S\S8%U^(QG[&><%1Z-#TCYB M0..PT'_="YKKGM3"AAFL?4)RR/G2-R3]\;%ZUV.)2^JW1S._,::E MC&X\D^&ZB"$W)6A!]O2;"_\6PU<>ZV## V(_@4,$>(GA)'NP5A0S]L63M - M:%N# M;@AC?B8J3HLY+#C+(-;ZBB1;96!H2. 2I) M&0I()S*MBV1R#A"] QQH"0*%RH&A+87#0R,* HI:%*-!'2I':;@U($PK*71( M:'I^NIID0E6I$C7J1??P5#G0 :L>A9&'9!2BS;&(K!8*6GQ ,JP]GTQIQ MZ(A3B^;M-N*YC!14)LK0Y2LNIML-$=[@&1O I80[TR&/RDA$.:Q6;D2T Y]< M"Q6Y8>$RM"UCT*P +RN:1I(,.N:YC#J,G._0$!9R9 QR\DX<4E"$.?7CND\ZCGUHNJ SM32][TT-# M^/YL;'O8PQFF6$748*<%K01P#!3@(YM4,$5"R:V0T\. MEB&7NSP#37U9R_3JE[]"\9& %2!@_Z)IM0M^[8D0_)(6._@F*I,PA15D80Q3 M:<,V[4-]VPN"087A/BA5Z1L$!]_N$3%U>_!B:=@0QWO)(A_C/&G+@QK#'EHPI28(;LS&,O.@%K40$*@Y<6.VR9M?<(8]P&7. MPI,SO-*PA[JH^3)JV$.:X;6&/9097G>P,YJ@O-)!4>Q)9%XNO-K0.GP9ZP[P MPBBDZX+I0;VATI6Z@P+@%8=*VZ33L-G3DS@-K_T\Z=+PBNJ3IH3J%S00+L9Y MDE+ M8A;3&"NEAC/*)2Z@V#H#W]X"3;G2D[WP&3.\XO@<^=:D_O>HSJ#J]=WZ# MG,.@ZUSW^@T4(':OX]S>-[=YS?D=@Q@,X09&R($09NZ#&QB+"#*G^=3U/G,C M[-WO/,ZYOSN.@Y[,J^JA03W1[E_SA,6#ZZ^1%&!7 M>N?^VL7B]AN,7 ABJ3M)\-YWOO/\YD7_.>)_[G&>4QWM_)ZZ H+N]L/?W_K_ M%WL!Z'C_)H"VUW0_UW'7)WMIYWK:9V^&!WU7YW-6YW/5%W84N'@8:(%>QW,= M.',Y9WCM9W8TYW,Q ')O=WYNYP/[!@(JES#Q D1ETG8D$7.MUW%G!W?@9P1' M]X ^R(,.\8 ZB(/[=H,P,'="< -=!W)$X ,T,!;L!QK=1X%2>'-1=WCU!G<= M2'Q$AV\B>(!2>(6 -W#QUW<.F'-$1WECF&]KQW-,EX:$]X8U]X5 >'CS!W=* MUX9?%W\E:&]XZ(=1]X=4N(>#6(4.48B"2(6)^'0W1X7P5X9[B'.+. -KMX@T M]WV&F(=%EXA#F(GOEW."F(9\6(8EJ(M#%XRWB(9"8(NM.'#'^(JGZ(>#-RJ4R'1% M,'@'MV]]V']^5W!["'[66(=IV(?-:(<%>'8R4 0R8 3T%@1$, ,T<')C%X7O M5W55J(=L.'&EB(-D"'1)-WA8B';X1HEQZ(F&]X>)2(ENR(@O=X\ V87#. 3A MB(4/68^K*)%G]X5$5P3)B'88&8LS5P3TV)$?R8./"'\>.9+:]Y'O1W:.R)&? MN)(?V'7,Z"7<*P'0(=(XW0 3J6(X^$!\VP7X7^7YU:(5F MR8!R&'7WV)91MW]L&9?Z.)<:29<'69=X>9=ZV9$VZ99VR9=Y"9A[Z9>!29A[ M"9=_:9B"N9B*V9AR&9B(69B/.9B3R9B5Z9A]>9F:F9@,>':/6))6^)DQ< ,T M@ -=^78KJ #Q82KPJ(T6J(BPF8&QZ7.0F(!"!X%,29L4:(1Y2($&Z7._Z8H6 M"(FT:(? -X_#^(BWJ)PFB6^+^)3"&9VUF('>1YW6.9S7*9V>.)W8V9TV^9S@ M"8GAF9W<&9U))XCG671%H(KJZ8GI^9/N&9_0^9Z@09\PX'C\AY_[IIK_>';] MQX8)MW?U5G]\Z9J$&']I>(4 R88T,)I%<(PR@ -A&0-X]8+K!APDT7;&$G, M-W!%Z)EGMWT!-Z(>6J(=1Z(G:J+UYW$1B*(KZJ(PJJ+[UGVP"'C3F(1#D ,@ M=P-.F ,IQP.-V(@)6'1$FIM%:G](.J1&NJ1)&G2&=Z1*VJ1,&:1].'%O1P1R MUW9U=W= .G_1V'=3!W=)MX8WR71K:'[V9GDPH*9LRG1M:F^;QV^@!QIQFF]S M>F]@-RM,]W8X$($&B*<)]Z<.N7""BGRT!Z>&*J@"N'MTNJB".G/Z)JA/&JE[ M"JDO5ZG+!X!T:GR:ZG"9^JB_!ZI'9X "V'\&N']^YW8)"J:J^H R,"M7*GIRV:9:T69QV6GY8*@1$QX)P@7=KMW;_F:Z"MZ[YUZX. MH:[N"J^O&*_L.J\P>:\P4!,U '(U >(,(RW_\)P0) MIXTQP+ =Y[ 02W!RNG .FP,6ZV]']WT.BP-']W8:BZ?^=@/ZUK$D:YK^-@-S MU[$J&WP U[(=*X#:F'4RFY*Q]Z4EZ) '*Z[[&G<-:G+Q 6_FZG[PUZY@UW\> MA[3UIK3SBGQ(BXE(6X19^*$&.X;_Z:Y8:Z]9>[5:BZ_W*H)T\2K\VJ \VG9U M8:'Q(A0T &]TL:'R5I\(6'V+MZS05W^+YYH%IXJY*8WP";=^>Y]]FX!XZX?$ MJJ*"9Z:%.Y"'V[?H-YI!@+$4ZH3ER@.(*X7_N+A)NG@K6J_RNG2!R[=_"[J MJZF=>[DF.K<("'W9!X%1U[CT!KDVX -)P /!+4D&Z;\Z8QA>K5'Q[=F M2K#>**3]Y[!9:+3O5Y8T9\#>5Y8Q,"LYP*\O%[LV4";L1XGRVWUJYY\]=W;8 M-Z,-[/6Z'-,)X)76,(B?,([)X:OF:)YVL$>M\'F2Z0\"*QK&)QM!Q/F M%Q/BNWZW>L%C"L#@9[12:W6FNK1&_*Y+.ZKXIL1P^Y],#+A.;*]//,7S6L15 M/*!'#';W"Z)]B)8CR&\,O&\Z'+NS"W\[R'=F?,5J;,6F"K55Q[6JNJ[IZGYI M*'AT;*7Y)Z_N2L7_F952_,[:@0WB'Q! MUW5/:JPNJXW:6(8FG,(H#)]#>'2?O(T_A\F^:WN5?("[&L0XF+"T**0*Z+ R M@,,CA\@D:ZNA>9(/^*(#IY(IVLNZ[,LQ"LPR.LS"?,LBJ7V3VL!82LM<6G,E MZ,P9C+RV>(#8F,?\YY#[^';KNGBSJ'1$L'^:J(GCJ(9"8,?0%WNDRJC5/*_7 M3,U3"Y//['VG%\WSG&\-&A/Y2LL2W*5_U\]*]\8 /70!_<_'"<<#O;4'W;4& MC7%_',4.[86#_(,2?; ,[,"16\99V(,:S9= . -!<(0))ZX3*](A_; AK9HC M[2\AK=(=)WM,[G7 ^W:(]S=$;7; ;7<@- M?,@7#:0].<@X:],'2P0N6WG**]7:J)I$,-5$ ++YEM4"K-5NY]5#@)'Y)]:O M2-9N9P,"C-;YIX1K+< 4B\1O[79Q/7&9''?4:]>?R'\5C=1DS -U3(9UV+VH MC*1CMV\$.+H>=]B*7:*+G:*-G=B,'=F.O7+2O^*1#G+18O,9'S,:K';7L3+4*Z]D%"-!?J- )O=!H-W^Z.G]A M>]1CG,@/:);_AJI6J\ZJ', $/*\"'-7)K\\6';LXD!?L1W,"]XWS>-DW2+U^JXWG6;[;JH$SZ8"\ M77A_>=]>1Z"3'> >=]D@&-_##:WW)MV@7:;!7<9H&WK_-H6'^*5H5X U !-BC,AE/-1,7J#+K=Q-ZV]!_<(J MR=.CHM,SD ,V'=-;/M-)Z])R2M-AWMY@;J=B;N9DOL#\)M)T^*>0S+]@;,C! M;9K2Z]0S:N=[N*#W^:?3&*82:'?%W7'PBX^]J70<_J?FZX!/-^C+!XKTJ[^C M:W@2SMP%7+"0;.D*O-<-WJ4(W.D'.\"4#K?K"9],5[_X>^KLZWJ#SIN\.>(? M+N(;&^*O+LI5&*96J^E*?JM=6KYH M2N* G9YA[8&CFW2C/NWV9X X1WS3>WV$['>A.BL(-')M%[L-;,N;*>N2[N&N M?I_-_L7MAV_9.*::*M^C&W3U/;KT[;OI[K+T;N\>B.W:;);PWH; SF_WG.1) MC8,[")0\"'6Q=Y,6%ZR?._&9*X5;9W= N:I\9Y!BB4!8.NZ)O)TW&7^+:* N M2[A+N/W(AI2) ;:<]RGNL# M.:KDK7?T!G5=)XIF&(DFZ<51JH36:'Y$U]F#/*"DEV\>KP#AV]?^5XN$1WD8 M6=N,K-NY:I9W38L+;9&YW?8"_?8$'?=8*]AT?W-@"]HI/KNO// '2\BZN^,& M"-D"7ME21ZJZ>WC[UY^(I[*R]]]O/-R=G8/0"G9;UZMP+NKPF8H .-Y8CZZN MW(F%#(M=+]SG#NL>ON\_V>ZN3-I\:G_UAWK_=GVQW\NR#[?U5_N RZ*Y/_NW MGZ+:IX6_+\T;^7)'CL^;7H%U:[K9>XBUF(6 !_J<&_WT.OWV"J]/#W_QH9I! MT)7O-MJE/^OK7L)B6I_CS^YU")3 MWW2>($>&?93&?9[5\YHQ^=&V(.SK]E^ M%\^W&(3Y3WH'G\^1*S[P,%Q% 'V2V<-M< ]K&9X#R #=GMQ#@ _0[!DW7I>K M>ITLVW3]HO6=I#/6T3!2M,LZ74?V%1P1N/OR%@D\@2=J!*; $L@"42 /HC?( MRP/&0!A(_-9'O@)Y2ZY,H3VT9W-:G\YY?;0O"/:^($7 M_.;9\'L5-G#TZ3WBTY+8T!BB6><)LY$RA_#8")_@^X):<$:5,LU&=4A5 6Q? M;\ALF1^NAP.56K.K7KD*WJ6@:R?O?DX=O'=XL)3=.R=%[\;4I0H\V8=H$;QV MQ^'"G[IC2T!(! VW):AT&A3$DH*<#O#0G/$7R/:0 4)3SDGKE!VN<]M.7BE# M>:#P$XI"!$<*Z1BXXU<@K^W0@!]UD716-M([J"?:N1TEAGB&4&>K=0.G"% [ M:F?OLAUH6DZWIXO1A9OT\1*>R3N&*RP4FK9E^/F2H2C\.IT0&::\E+=6@H_= MZF4%YVX!.<#F@-A2 :)!_ JO>+\%U? \4T\R@DHP'>*^=2@$T6'M2W>]#W4A M'C@XO1*.G8."$>H&CK9+-\@\'1-S6#WH"TVZ"T<03=W\RE\(4=6]KX58Z+X> M? );IS#NM$&:!)YCIM[B?LH(D)"N-.,:.+S1?%6I?\4<"Y9M9D82<8E^;7LZ/[^!%?B<& M=YY#H()?I^KXMK@8B 0C&IIFH2H+';$TQABAU3"D"ZCPHJV<>'$:9,4!&0LQ MQ[1D,)VS&3LC9_R,/JX0:4:=PW'(%L3+ 7.A'8D*6^5\=$[7\7&GQ_; QMEH M>[)A\+&-N)'Q_"XL1;+:D=[30$0'$_6C@:.FKISK\5C'4?9D'@TV8328$2 " M&(PM5KU_I-.^3G7$2G.+K9$_]40%<]> 6V1WKA*AHUV5N]C.: J'(<<'C(H? MQ7U,TZ"*8W*-3R&OT<5P[@)J3(]Z[S^IKF(5^/Y1D MRA$L14KP1)[I$5[+Z MA)HKX& NP:4@VY7A84(WX''5!-](N7*3YEJ0!Q(,_;)S)OVJ7T[TD/_GE]TM MIM0?+5D"BCH0$DM-R)"W(?>3#1@5FR?+@::>9 X/4F>;9R))1K[(VR/,>ADD MLXYX 3_2+O=H?FY7[E)-'HEWD;+M%P2<5SLB7M(+!"4O;50#F%=,:%!/$@K) MG-"%@$ID!:)D)O)(&;X0Z1]-).I"4F,2^3FG_RBE2)B&@V1$((>MR-\([XQ> M+$QVPP=EE9^R5I]V89_\D_?)3THZ7FA]OAO(DFMY$@0F).IUGN#=>1PYJ9%% M@D*)]Y2Z6'4260AR4FK*3,DI3UY=BV:@LE$IH1P5*17 [*I=]";IO23@1"2U M&)Q'0/U(E,8@:J.]^E::LMQ9L>LEOA+09T=!_L*3GN$ M1HER"%2CK%,.ZV9/$DOITF36G7U6 /U9QXR PUL.<"_23@1X,0\G*EM5ZHD M.!@LB:;>7')"+7)V-)(FZ$Q:Y1QIEE-EL32513DFEDS+@A=BK%(V8+6VQ!PA35*2<:IK*/$:&&1RT1+T?-=+K EC?SD M?2(][8]NPL*:LWG&U*/4F\.G76ZO]1FB+N8T^CZ)S7U>S.))/QM@ JR?PK%P M'L_=]@W/8]&DD*TS@&ZT?;F?"&A0ZS?M#:>A1'2T0%5B2ZP\#W0E;K4(VA+7 ME!#@.(;')#)+(D"AYI^9@HEQ3G:F1SIGKLKEN#27&@RYI:OU)(&P);:,EL_R MZ3A+99D6B2<#S)]=BQ.1/,[C.)EGKIR:#RYB ;OT..S8XYFB11;,6%6?V1/XM'S;<,;9AU/#Q$=@((QC-Y/^PF!9E&W%)X* M:XO6/J:5XVP< '):H.Q#;4/5!,!4YN[B-R1S>0K)F53'.INKY%.E9T :2/+7 MI]B==]0[<*A2"L BX$5/IG7D.H&H$$I2?U3M5J;5LUS:[@_*3&O'2:F=WY)W MEU !6:YF%_F@%BF\HV$K;\Y.F8.FX) ?37J U"/5)R"J?2K7V#.D2-*/)M(W MN$@;:<@+C >I+,V R ,CBQDN4YI0DY=%S6#VP9">,Y6$_O-QLM(Z]8WL826" M1:IH^#@G>V4HC97J"G@XJ!8BT0N$WV[DLWMO,4\E94'$Y@7#8&#<>77TB"H= MO(>\A*1/NJ=R$3<;5.^''.(Y]N:?:-#(TYL"L%UVHZ&J43,J MU#2JNF^9MI\9^>#.YRJE.="1!_Y*NOG"8N3I9%&F\T2ASJV:T[KJ5267_4*C MA560) 1&U:R ?O&EV(T_K94=V ,:SH%]:"".KGJW.IJ05VH?FRA1D6*MIK0 MJJT,>5M'%-W(9'\<_5BD?U35JMK)&P&PVA-&8) M,^$&,I&BD.9-PU#H"3UA*2R8="R6H56_FAY5(2NL.18L[< ^V;>)G".(HHZ( MU1=:I;2S_R 594V/II*5;DI/*?"FJU[.FN@VE7Y&E<)*GVU9HZ,8*:G5GI67ZL]'8^R%42UHCAF< CL*])? MV_(G$:AJZ8>@I8-].LDRPLY3 2C/8.LD%+ 8=B-V0<165AU/AU5&B570V8!] M=$$%3XD%6R7V\%2?FW6XSIFP))8 =/AUI%;$-6.?!H*A[:[C^+$'*!MY8VS%-]=21KL&8BH-V7%%]LST21-VE5H94E2E2 M[4]&[SSUH,HC)XOKJ[)E Z\*)D@596?A[(KZF6,JSXK2HVIH4906VUY-"5+% MR9&S(NF"!)N,1B!'&!C,^+8 CS M018LN8[/_M?_$I]W?862Z.B1GK.TQ!9I M#IB+8>G2PB CT"5*TP')FAS*.K;2H@4#X]3QBHV><;S"K:MX\H0M>FIH M/&9"N!-*Z0VV>Z>>M+ 9(-QW!#D9@&.X(RFB>;':YTD%'\6M3T"MXA*^BZL$ M-:ZUL[@!CN->7.#7D^0FR3U%Z(S;OIO/$):*E\S1CP#R$VHW>R4BMRNSC'4$ M4I#J7"[YMYAE>#UYOTQ>H;Z?:S'Q31 0"SEL[I@BJE1LAX .@E\L5F3O(Z*8H'+!OFE 9 XT,;23(W2,DW !O1;4_X&PSPH & M]76N6L^ANZ7"EAG>PYMV-R/;+3<%Q^TZG,5['AWO$5( OW3S.AT3=(6673X" M3I H.#78#H36.I ":+T^!Z?N'-F+J-Z8)O4W0?'V=IPD%4[CHBLS/!Q' P7? MG#-\^]8Y?V"MYA&\+E4#(32I]L+P:/,7L M:#I!?78(J#CERCB_[X"3O /(VO!H['X MB. $I,KNS2E;9^OL-5:O1_1RY"^_J@E!P =\"O7I1]M/$=Z^3 GZG#)>!96T M+[/:.TTX"E>@.+C;""#5H4-2F E/82>,A)]P$H;"6U@*'QR^5X1O3A$^01@8 M^E:>NG.V;E4CBU)P&$K)82PLI>IP'%963(I*!9V),W]?;1%0<;SL&UW!9]:? MFM)OLCP%>$TEG42,B/>P(Z93CSC!0>)#!$AC(Z *I)?8$@^JUW-OL@XGWL0_ M!_?$8%$'(\--^/D,:7KDS0/U0WWAD M40LFCVM*>T?SI"E5/(E%+^\RQEMGZF@SPZ.,?X\R;CW/N."Z3\O"3?^4_N1_ XWQ^\ANWO(AHPN.J=\B$3 MZ26/KXG"=X1'%&8R#$3R]E,_MDWE]_L*Y,%6N#00@6-64W8 <4\I?&/M47*2 M1XD()F!@2AM[-O"NJX/T#N+1'\S6?=NO1^[((-F=]KOP.P^E%1T^BPA8 :?D M*LR2^Q+N)%3*29BYIOGZT!+(A+3:\^-)7A6]J-. )(X[S1)C;J/ Z/HS@ ""R7 M*C14>A&=4S)TNZDLEUXDF9;;,NEURZ,W+J/EM\R(:!%Z57M F!UKX-E59/>L M9U0^^$>P@EJ4%1K'*RC[/4ZMZHV\A^6-M%P!BF4P&2K_1IS31P<58?UNR)'GYER#DR'! M#C9K53F(VE%8+QH#_FTDI9GIMH1UPDGJAH0G\LVDH;23?M+EK)SODYN\>?BX M/W*A4^II43,[5LVLU$Y*G9ML4UOJ^$Q\!=)"%J[;;(]R+;/.Q?? :'Q)^GX;;>]-S(9.GXC[L/6N,\*-- ( M.D^ZS?W\_":EJJU1ZVG5)MHPE(G0$38KPUFO9-737(S1@,X^%6BWK3+CMH!K MQ\:D=CO1[0I%G[;MQJ)5='9+T>]'R^T@&:WU;K&_/97*+_E1U,,[K>2@UNV0 M6S=(>Z&+"AJ <+^% 4$X/^ZC)7V6>70&TK,3[SG/YBE]GZFTSM6\;V[F2#+) M9V^$@ *(9F%COK0U-IZDWE MD!SR5/'4'=2,PA9,!J ^&NL"Z4%-_2ZSD/[1B+H7_QLC_9^3-)#:7>X5F7D^ M5 Q'=5X64V/=5+E9L4P]P#8UIKYB-,]3B^H976/UJA:J/.B9Y:[5"U:BF70V MVWEZY:.X&HV5UK8[5L7JADNJE)5R!GVK"SH_4E;XX^B8KH;11-M;-3CN2 M-=U%LRK1J'+,OGJ!"3=^ZL5:#T#EG9SU,WMFIS6@]A,L9D.L;!]!Q]IVI@_@ M_FRMZ=*+0N7L&A@Y3AKB.% '.>?!3>I)FW.]OH-ZT%CU3!!HG"DI<#9]LFX5 M#:%4BF3+V#>+:&SHNW6WF7NH!;6[JL_W!CV?7=KE?6QFVI5PFC<&WZN!_;\< M3L\QS,&G,P*A<=VH9E(&"WG?V!A?XZL+?>ISPQ;2^'<(Q.R9_:Y"[!/]-^)* ME4[L> <(9W6+OM6TVE;+:L'YLX6VJEK25!#!F9_JC)Y?19)ELBETNS'C=@5E MY=>>F[)5%LM:V3%+0\MUG!XP31N .C73W'>?9;EYY5K!@;($ MQVH7@1"X0'R];4(!2(7#?1Y-RA%KF"H-%XJ%[DPMF/+O%242I] MXXG!9KG9/X0YG@4=#EJXU3.F30,E8RR$K7BS)66 MPWBX2=F<[BU1U23 7)"52T/6J]&%?O95N-)23!?O.%T3:8__EI,"0^5;1R,_ MQ\.D25?_3I.!CW^3R5VV)KEWDNK?UYO8=FF[DG2U%%\^/;LK_P4\*T6\\^1- MBC@1)TY>M2JI@:&D\8+@$CP;K4IC%<)W%]B9"Z,)2]$I'U J?A3OR4 ?"+_A MUC+MH4W04+KA"BIK%B75^W#TG.3A1S\)IGKLJ@-^XC2YU$2E*/*I'H"I>DXO M!@?B[AKQ_,R\^^Y$9JJ:1GTXE@EA@16%;F$4USOI*2%-Z&"]^+@=6@:,H"SJ MI'$"#(Y%D&;423XITM+P.2['Y;C^9>,YUOE4.]M'4P'4PUZG?)04Q>-;N.8F M\LJ% 4+X'>DZ&8;@5G%P*I@]4+*"8^)S!<>4'6M*\>U#\B-O:8WL6"<_/)_< MCW-R<+S)/3DIAZ]?S_L4XDENR24Y)$\\D;JTF;:M9LCQ@@RPR+;3N)FI ;7+ M[3>^29G;U!#[([8;>(@YI:9-LSO; B-OE8V0.98]YAZ:B49RJWL%R_0+_D=T M*)B#XYZ\:#\3VBO-M;P=WZKE5,2;4L$4CK[8G ?J=+[-U[D[5^?PO!7K'>/& MW6RTEK([[3).0[/.=,3)D,UA0/\FJDH>Q"/0+P\O.\8)[M\@](4>6+,5B]YZ-*-"I\86':1G=)'^T;N2-2KI5RVDN]>/ MKL.EGN>M9U_(U"\NYN1'RM/-P::5V/[Z*@3AJC^2G'>N-5!TH5=-2 M#4?F/3JWX_;=?MN!#@ZMXXW\W0V=O$R1_S#M7$/O=Z0B][[.HKYVXUNS\IG45X[)%97^[B6!)^]_&NW,E[.;]$N@C5XLCR;'W> M>F].;^0+JUM!M%G.\]1SPYTUF!N)(/"#"$O=<*N'O5>U__>@69PBLC3V/U 8 MYW[LS)=^01D'M#A$+N2$ F*PBG M=.SAT]5S*V_A]1U3:^FNN(W6XGR7?-&B(ZST^/=OG?4:_A U\)4K>;@X5L-,6=? .5&Y@+#T?N;95-LK_=MZ6='P0Z9B?20WHH M7WU4T:BRAV!SONF *V?7M?+^)*/>>!'%;VS M''3245GUK;Z2OK)+<@J-0&&CIM])=X\&#?H]"IW,J75,[?/0H4LB:>^]7?H; M'O-\*-J;^W+/[F.1NU?W[1[>JU;D1._S3=M1N7Y8Q4VE\_43Y3R_=XCO*G*? M2X$;4W?M17]&1D^4MIY0A7H2/L-7^ W??/T>.@KQ*SYBYIJW:"--'&&OAFE5 M?'"Y@>^1G=[?9):G_"#R\U\O8:5\A;7R6WZB4_E@*N8_2,NIH\9TE]KW_3YR M__M$!_#]?3#*8."L]:P\[ ;)S-DX$V=VJ%N:LY^9L/B1M;[K8R?O0K-B! 5A MG^M_?;%/]E.YI4LZX@K?VWS,/7$2W_L14^7O M(D4CDI^JZCXH'SPJO_7D_57K?_0^RX?'7=_O]WV^;[X$?^$G_()_[R>ZX+>1 M!EE6@OM Z%5I^U>;?AR<3N78!:\+,?R!*?$=ON?O_&PH GW^A3_Z(S[I7X(Z M*/AM_,EOIUCD?A/@%0F&V7?ZG&#@ M:C-]7,WY:UR=U.''S$(M5=A$/IT_\]]GMWM-5 P@=\6LI#V-#,P'_XU]85^D M LDD)!:@!0BQ4&5)!P>%9C4P#EPDU \E0J=?"6CZG8 67^F7 H)^)B 9QY*!C.,#!)K_@' ): MD.-X152.U_H"_*@>/HZ0U^YD97&:F]5TR$502P'"0=& +!=H(U3%,_6,T%?D MZ'5A2DKGS_U$WILB=!CQ1)%=_8.PU'[FWS/#\ @]-.!H(KMD)>N&&B ES" R M0%V0:WDM/UX>PJC@(Y>=&6H)HAW/W"5:"FF IR E.(4)(/#.JH!\$AQ%P=(TK MPQ \PI*T68Y(M#.M*"_GRS,G\?DF^A]PHO]U0HZ=C%=Z&23#8-&7X@$XW) 5 MI-H)3ZI=RC;'\$3C"$^4A\QOX53@DBZM;#,'FV9TI$U!CMPD3/%U94FPLH., M.#<16^0)\C8"'EU"B\Q!^8T-YN==>J<(U#&8O3N7%"V8WX@%KXKY<73) !K8 MJ* ^D7.G%CGGG#$O2*"T1@/L,S"01Z(0BB0>T&U$;\@$\4'HU MW39>EES68V$=FU#APELE8',9 D*TE5D3B$0A3",HUA5#A4R@5LG9(851(%1Z%6*%2:!5FA4[AMN/TF%6?P6.EY:P@ M:POY@N@U(,T:J%.P<'D=C:\'D+2%IX?1XQX!*"\,SG'J>2D[%"0%Y0U&JQM? M-^GU?4,, E6$%%(="SLEB/5>_I\>IKPMAJ1'#? 9<#]%7\C#] 2"0M\ :,89 M) )QY/M+']\'EU(E50V/ISJE8@<3 &7)A(-)4N#T61FF60?"J$;7BW@QA-QH\AB.')&A_044D7J G5^7E\QI=)3L\=@10LU.^;&/ MF":.AU%&9JUQGLJE]TA],,(6FA+KC#W^A#:>3U",4"'I5>MF#J-=\.'G4)FA-;2)CQ5Y.GE,W M?X M?)X\8@'*(P%34")P&(A["0;GK01?,N%,%58\A0-=GE/GL64Q-DE M6%8IXBR-/,F'<$*2"3\\D%W1!^J#=$$0QBZU)@<+7!1^N#^,E3+U1(E346%; M\J$81E2A!Q.+-%.L$G0RY-PY)D@_1R$&)5%>D)@@"HF 2:\$) Z)2**.V*&\ M8SQBZQ:5E"C!T7_SY+%@:EQYTA5=020(SP?8-()"CL$1RXR(7LGZD,BI3Z*= M22*"P4&@ M@8@8[H <,D#]M'K( 8J9C>K!)_8;H0B79[3$'ZKBII*+-'6OHJ<2 M]($B'YXA4F59>^\-K4C7X8J$R*UH=.)=S)F;2?(HI((G7Q_ M0!/4$WN,B$4 IYB+%1SA5C4H>D4^459+\IX-1^&(J#@AVCD^3R)S M4X8$,.=BJWB-S3 4"'^G'LZ'BTE_0R0>B?$B!.(NY3^AW2O4D.B(#(F!A_6L M@5#Z<(%%XB$XU^$BB>@1CC MDQ@:XH5P4<=X,>I8'./'N#%ZC!GCR"@RAHPEHZ%8,J*,&F/M5.91-".B*]AV M0&5V!_75+"Y"RF*QJ)WDC.))SVB )8LZX['(,Y(G0*-XXB8*C:'(J:@T0B>H MHJG8[?&)F$RR,I <*7\(/&6(F(EF8N4!+7**M]P-H"4Y*PD*@K.JC!WWB$_& M*@YT\\#GN=L[27](B'"/LZ/[6/B437>1)G(5):>?#X^ MHP5R-!:-1./0:"P2D ?DU*B][#=ZR&>P-=(%Q5\E9"FB(HMB&@5\""+R5'R/$@X&^>98>*36NY3>- BF(T]7=\%WQHM) M$AZ%BMQ.LY>G,'OWD@;3X9V!_8;%^**A5T-,?_+1>9&;"A@I<6",6,B"GV@0\D M L&"W%UHX:CE7$6/O$Q$=NPXB?SB(DDA-I+;206#H#B2AX@DR4@JDI$D)6EU M89*09"=9/VZ2DV0F*4IZDJ$D*U.)MZ(Q02 0 M(XG8*0)6GHUS! KV>KTB5?)&*8H^4:)G]U E+@\>DN&U>JV>;$<*\H^6CB6D M:J%49L@!\\#D,$>.E^:."%7=2@=R%$Z%W&2-Z$UND]XD':5-+AVX^(\N=K1DZS=:V=/PG85B66'PB%%6 M';2==_CF$8HQU244W? ?P@LU.?[YAF\C&5)ZD79ZH06R4KZ43UI,Z5+*E"WE M=[*6J(DJTWLUE $FV@ZRUBAMAT]4*T=FL55OD',B5!:50*51^2;^E,?:=DC) M#67%CCCY-MXC.!(EE>SB@_$WMY7K:7ZV5\J5[.ES^32D)?FB5_S!%I[B1 ^4_Z<0DF@[E@\B/098,I87(Q;]@0( .,!*Y@R:'BM([D M9(?I>FV7V64W"6)BER2F5?@47B+RI!8%AFQR3>&?\HSH<[@@@)(6C55(7')R M*3DRAQ&-!LJ1ALQA4QB*;(;C2%MY8IIRN=RE$Y09F5=E0OD*_F%!U=!"5$V8 M"F:4Z6!^*/#D@VEE2IE0YI2I96:9#6:$26$**19F*WAT[6RI5C>RVWB7WM[Y M9[N EU0>)53=?%K,'3OI7L*'"XIZ62?"EW?F4YAGXHYU'IXYR?B9VR$_LF<* MFG^FGEEH\IG8(Z$9:.J2\>6>J5]J'QB:D4DCD8X@!UWI&#*4R)6;N% Y46RF M*")>0H?](GEY7_IQHV:4*%_"EZ6F>49JHIJLYGOI:O(_2&;[L5]2*";E5BF8 M2):^T+S3S=H.WZ$([2,D MBZ5IN'4I<>%NDTYV2X ).VEH8IE6)B^C3[9V]V2XR4^.F_MDN9EE0I=TGZHX M4*)1:HA4)&)ZF"$F50BVP)LE)K<22YH@8V:MJ>)L<@0?T6$U%7P#(,_6A>Q: MVE3 Z!0:G%NA5C@50I@+I\+I[H"-LX*T6;:$'&I>O[ E6BX[CQA$?PA("Z"[ MQXIUU-.?,>0!E M35K'9 93L5O7XV.I4?ZJ:6JFFG.FU9EU=IU:9]C9SVUR;^;RM\M-E1=6?P*+ M7)6TYLAQY&":/Q_"5NP\464:!@68E#K"V+"(V/V*1%Y3B&4](\A=9U+1B%!AU%(--)5.6:&"3C&K-$P0 YS!+1$1/X'?H5IC"<-&/",F, MR2,!'68@TGACOG@F)M[8#?D<^:$(DSO:(^W+]8?RE"(PT]2I\2Q_.TOQ&7EN M8&R6S\-J?4*;XUU2ZNQ8)>25R'>R9DOAWYGM!)[\)R0%D14!^D=8QT:ND8OG MXKGNV#H*DD17C!H9!TYW_2304?40P1U'"I0_.5035782+F MB*QFBW$WGT%BXZ]X'#; #ZHZO7"\E;,DLZ4P_@J_,@4]3SUO3APT]J%LB1H M()L*Y(9../S@?1*'FB.TQQ,B%F0<7Q.D@D ( >%+H:?(!2<_5Y7&L'5(7I$" MXGR$H="(-;2&+J)N4Y_2AI)71X?F]7;9=/T'[-;#024<5)S$ATX<'%I#Z;LT M=2\'&IHH02"!:'X&NB!'\8%IT8.^=.H)9,2':BB_U#8)X'!XZ&%O)0T!;+CH M,-5\0&#GS(OCH:Q'$]4U)(S>/]&6NT7J? 8.1PZ:C"*C=:CD<8=2'<"HO<=G M;:).FY(VOY1L_4:>) !!:S_DS82?\5R]2RKJ$*RBJEV:XHJ&+YTH29A\&"@U MU5ECIJ6@$V$O S#]-RHB*.(,V:+W*'&&BWY6K!N/18^BH U@0.H)':$_R89( MAIJ$]HP3"A/4&W[7E@02!5*S1_X'N-1S%&FV)(HB2ACIU[0>J:%J:,D2BE(O M/Y<"-Y$FH_T'!+I..F7%4#IJ 6@7,LL^H_"H[?H,V3R"&",5()'ZA1801NN MEI,&;0&3?P1SP5S#J!B$<-Q&PU4VM%9<'\CHZU7]M!T:'.0"K,DO%I.3YBO] M)V(!QJ&-C 2@@<TXU9([C$7HIYJ!QY*$X7\HE0Y@)RI8EJ?+*:@ MJ>2!HMQ <5*.4F]XBF/8MR86T&T\TA,R?'PT1P=6"KHA1Y -;;IQA5R3C:9V MJ0T8N^FEQM/(/GG2[A.QC2(MJ^A+M[1P4 A$CD(7W'+9E2>&;]RA MI='"=)7:1FB5L(4\ETPZ*NU .XC7^+K M0H-(AN<)6%9TYJ$FZ;)&WPR LE!^>NG)/OI?)!IQ#"C]QDNTA_:A@!4N2 ,L M,UGIVY6=.ARJR8O6=NRD15M/ZJ'\I#XI@"2,$BZ#R\TU?4HZ!LJA\Z&IHU70 M7'2+S8R)C'0J%EB83PC909ID0\C1PH29$FJ,%C9$A'8\[BG]9UR1)4&':0&* M1FP51Z1RR2&,5)FPXJ&T1OH?"7I%0AQN8%[ _=PPOQ2,JE3]=]N5(.JN&&KU MBGEZH@BA0F@1HIZ..&D?]Y/OS2Y96::(G6*EUZG+@K)4.V?J38KY/%EYC!SC MIOZBZ"G<-!@.I<2;>NIVX$K$2B6E!CFGQA!VT_WP&Q;IPQBH_H0"QT]XIL&E MK@ Q$-!1BEI6!(63C#I36#T&? @]88V MFI4"JIFBO9&$#E>.6:G*T(RJ+T=6FHAVGQG'T>&+ZJ'$4/C2FSVG+@BFY1/T M@2\!3$"]_5I3$FA0);EOP0N %GD9+U(2D+/TA*(5'.;%!6%P59(&5ZR>A[/7 M;J*3R$50B5\X4_H<:@H$TJW:*0J*6::G<)]FV:$R>ZV$N1=-N.P<'%JB8(-. MK6RF1Y)R!!*A&1+X)LHT(JB*Z!C'\"OBBLEQ(BHRF*(2)WLD?/_'A33 <5-> MB M:@ !RE]#"ZM(=7 @(YU$'A66JQ^]9J9UOG-?*A7PO$SUID6B(9'@&T= MG2-:TW 08*";M300.74**IPB@/QV]HA!D[+:7I:*DWC(.(EAUM,5M3* DXC! MP<.%EKC([?C$72'NVHQYE@TW_1W-&>'%I M.&Q)S29TR$6QD-"!W_"*KE%9A*#0I>L76A8782)Q40U&D(U!^=B*B7R%2!/K MZ8="NFYL59N:=*EJ .LBI,2A0:LE]"%[P#OJZ":"N1HK'">R4KBX.4,,0C:E MQ)+Z'*XBL[PAH\H;8AF6>G]*I79_M*Y[G605V7&LM%^_D=>M'O]?"6*R\B6H M6@HBJR!R3*1?4S[VD1%?H_?!B$&8S)%JR9@6'$=%HGG]'! ']+IYY:AHH.EE MO99\@%&V>H7Y)E?)51+#?:^W(+8BONY\D^4=8A1:*?RJOC%(DB\)2B+53WE[ MBU_K>("Q8IT0_@G#Z8+MFAMI"P(G64T>^9(8H #('@G K!J9$V2^KPDX%3K MAJWN7ML;][E:7F-FF:E2E-QUD]Y1F K*, A;I@BVB3[HQRV7YN&H\$L2!:EH M5A+8.W/Q6#+)J_R!V>1,]HZ Q%CI'"]L+#0:T1O*1US([1AU ,K1TJ T1:-/ M9$:Q>2)0RAX'NEPW+%M4=2,M,N\1('4H+6QNDRQ%/4D=.XZ#%T=18&D7QB/* M/')\UL@Q<=QRZQG68_]D:>_9=,3N=&\E69*B:3(E:LT8ZXF=*\\:9.04<;%; MDHMGB:AD)HQ%DA&^HP%(!F3LN$0D;-1&^8 Z.FOY92CV04@**"76@$_'B]0Q MPQZAKU%B18G@L(.'#LMGQ1T;GV3(WU!"'HL XKL]'%O'V*/TC$]TJ;T*O8(= M'M :6R/IH9%1UO1+]0L!AT-)MG%&18AH%0Q2'^Q7Y?,RW;(Z3Q;$EF%X:Q+( M.H@A=X-.//H2#BL2#Y&2M!U1(=J]%QFU([Y.N/6_S5Q2:05** *A+(I.XP/] M7.A6.M-8*;):5Y\ +%P<[0*4V$-?BA)"LD2BUB&[0ZNU/O,: M^'3M5%$Q*(]SY61E9$>4I7L$,/&1\5%P8H:Q;+Y!LO"PH,V$,KGH)80<"-)9 M6AU.BW.7Q02TGV4)BX1I7XO>*+/"ZFN-TA*EJ^!<3-$R6Z*J/WHE.G*[^*Q% M$@3W&H5L<-@0^[]5J=]8W5* :#VI;/W'ODH^X(< DKO<+#?MWS9W2C[=8 \: MIU0C8I-LY!K!A2Y1N(..2H:$:([F:%DO;]C8X:NHM%T22^LUI:$'V@?TRJTI M_IF8QE""?5-DI@?HG'#MD47JE>%)ODL%XR,M@+D/?,+(5%LW%^I3M&Z+(2H5 M=,,0M,65OI&KSEIM +!!H5!O(\FNA*[X* ^C23=F^@ '[:ZU&:):6Q-HN&Z_!0?B!YEIS&G%SAMQ8%V5Y=STUKI[) MWM'!K'NH+7NHG'QSINW,2K)>3X_5]72!O+;2SFP[H[2VMJUK>]O*MKCMWF%? M]K95",-SYPBWWTI*![G0771!N*7 A4MA$HGT9AZN7U+(6KZ%>W:8*"N'C6_< M6S8ER>EO:1(<9K!J+?.;A:G,9!X&H4,@OX%"FXM3 J7D;SG:,]LE<7U*2N=J MW7I][NVBI;]-M\YMB%1(?5UTRJS"_4PCG6A"^/08DVW7GN.:A:'#%3IRU9@6 M30CQ\J,(N NA0GB!6"#E$T/($%X\F&I-\*\-]RY MN(R@+T6)++I-QHTI6\3ADRB/T6&U.@5Z>8X,%.BLM#>+CU1H=\:?X)B9DG<^ M4GH1+I,793R^*_JA,.TSQDG*!,;M,G7?&&?PI98.J^J5/PY;\0=T*(-%'9K1 M6@8D;KDD;I<[XM*4W2@>%W]X>#:)7 C!GE: B>9B]U5ACP_G(:[$*AUL(F2=XD'X*$(87<<33 6WH6V!PER1^@&NL6CH O>J7*B M;9]+9#ZZIU=0:4]R0)82V%CT)5U?&G96/D:"(U5L!W_^(];<-/?Z]*A'9RCZ MK)"Z1F'NJ/-*!%YZM[1:)B_892*HX)E=/K-$;*!6KHF/+SRO6U MD6 1\N8R./UJ+/K_^3!UX?+UE)!#INU1F.S<(=SE\-21K*D2+3K&0:90!S9>5X6E[1@,F*%<#/'[I=F4X4KZZ]J^_ZN]CA2"*6<+#T7\V(-.5%DP\AUK=<07?N M-&?:]7*\"JB3>.Q8Q5P8->'BB]1G<3++E6V\#5 BH.R.I2-8\]Z\:>4BB%@"G<)<->?S;E-T MY^8XAH2V;)GK^)F()L.:"M*0#G(29"2RF/PA!=V:LKQI/HS'U+MS"'0('4S@ MB!4@6@Z=LD3=(.!4ZD&T_G![&/$1-](I\-_?IO:"O6RO*H9_@KT%3I[GJ4"@ M<&_2D^CPG#%,@R:(&;[?DLN;],HJ2V\= MTD=R0VW(@9CC2KV@;H+3'"8XNXK7&_OYAS9KQ(8V376]()D8@?0)7'GI#\'_5:;TN_S"]!%)D0(\QL% M*CKR8_(;E?0]#F\]B8],@5;);?,V]J[++FB]W=SP@)_K^-8 ,QPW NAV4A^X)1O'8PZ*R-BZXW (Y M/%)W[V\\=^H.)G72;05G#9:\DKRFH"=:O M[J\F^("6BY5@K%L)ZC16B3 9Y"8=EZ7)5;:V(\'K3)+WYC(;340+$&%O18=' M)X.9'8A0T;&8_8_.%6HWX)4B72*_J/-!83:I_<&DY80"B*OW*^DL^\A/F/X^ M+(E,5=;_H25+#VLV7O$C_BD_8EH$G&)0/6F""9JLJ,VJ!F>?(R8A_ +5P@'=H9P(\P( M/\*+<"2P)8SRM#O!%]^"7L4KDI"NN,S]&[^OK*+I/E-7 MCU7R4,9\DNY^]R;.8#QE.T-FD7Q!3AC'=6TVG2^ATXHP,K5A@B/Q\B-W5.O) MFI5/DV,74K(,K:E*CQE&B1\SY^VB" J%SO#J.8KDO<$L)4(-UX=4'B,2:.V+ M\2(=50'3)MU@^#JSYB8+[.D9PZ6@Y4CY$7=X:;,""\+#/)0K$SQ,X$DY@2* MTR_F=ITC7\)15B&R)0[BX5(B>MLT^"X.Q(I(;1*51$.S8#P&IP")Q[!O(NZ* MH!Q)WMN!J'8X%#H&6R**Y4DCXCTV,AMQ# M&SDY(HSRR*>C[=:ZKH@N")"8/$1(/MR%S(<*<8@ZBMR1_J_6BGW">#->4%SC M#<5/QP#Y$U^+R8 S[\F$M@RRHB4>R:@C&B9R MVF!W17J+R@?U6H?$C;K9T-MD-5DQ)#&,BC$G9LP%!N>*<[_*+TL>"9JAK6.7 MQ>2<@(V =Q/-A?2CNUCY*$:HF.*"Z/(\W$Z_8 [A?4<.7T\Z$]VT1D#7&$(9RP:_YG-7VE[&;\X/D^ M*-!.LLN*X&S//:QH#( L0CUH53C>1*>:A!E&WPV_1H M?8^Q9$SU:HF3<7&,@:Y:&O&U55$F?(%O=%R.NGW66CDJ'<VB1QK<>XL1PWX38[2NEO@ORU;)&O-R(3 MM4'+\:\W,/F;2R$LTV/0:+^Q?]48!=7XH, M\_T@:!]C%7= -4QFP+*+S;[<7XVL'-_(R#$]8ZTA;PO?7:>PF$'G##HRSG!+ M%1@;,ITU?>E5>A7U@7[.U"VR=E(HK/'$X?%A?4.+UH?&;LBDSU.)KNA"&,PQ-GC\DT-?C1P9V\@YLG%\;]J;TBA=R\R&',=5JC7N MA4 QZ4GPS"U%HPYA>R0H4\L 3B+@$W4%0*HI%*VZ2@O3(1BLTE8VD/.>)WG_US*GV)AW0TB$=<[)2?@D0E5,+%NO#;JE3W[AS L#+,E MJN$P^W0H'7,!74OLN3C'3(L#E(A^N]?IQT3EFIE,9]B\R@0]1_ ELQUX@5'[ M8U2^'6R1-F;2^7/#U'[B&?J=JB5O(L_V+54BT3@6CB1F+7+QO2$&7Y91;_X;RD=7E5?[?_J.#0,G^E!3;VK$1 MK[$V*T%"9F5Y#@&+ZS;BA9!4TAF@)L-TH9R M.%-1C3,.-3E?SI)SYJPX:\Z',^> MA)7K4T59KMJ*/EJ^A$Q#;T-#S][.C#/E#,4(1WP,[JP[]\Z,\^A<9IYJ1XYQ M2]?$=59+O@J-KDDM6P4"//_.MC/C3#O[SJ/.0Y,[/\_8<_0,/5O/E#-64HRD M?JQ9\6PZVX%WY6:8G*A4HY[F%Z:TE4S8PARXM% ):K\&M%,C\##([P(8E MCE?M+,Z>5-3%/^?.UH[_#-X"T/VS%Q) 1\ZZ\P%]/PO0#+1/9C+3M."S2G)9 M&L]P'5\6L@$Y)5PV,L&-D10-;CDIRO=PSR'414_]1"Z:WA$;V1]>1 M,D&Q+AAR1Q5Z)L6'9D.K)3WF7^=S _YS?]'"DL5B+"R7/XBL(B\FG%1'P2EH MI%'Q 9G:?#27H_63?"&<2S 3DH)"4NHW:J5M./G9&OT)!5WDVEE'#09=1A=/ MUY5P/=,((H=V;5=%%S;(=/A-&][?HNNE* ?)+ M?T+,DB-]UL5I#V-H0]X":V]0T*>1#FN7"GU#W]PU6I^EE$UCTR0R-ZU-9X,8 MES5]J?2D%Q,'Z)@]+#D,R &LQ=)Y[AXF1W.F7M<[+4^#5_0TI\2!G1T<%(ZB MH[P*H)?>$[.^1D>H/$BG+4I$%0,5I2 L MR7Q!]9;XFD6HNTO##:R'VH-%DA M8DD2,G*HTS)98:R@\2E*#^ZQ](P]ITH]1O@"/1YA_1:'U34%M1B,R+I'LC)$ MF6]8U/MT\'MSXL7:*N,1I_6O T=@^NA4;2F;D^6C3JB@[G\2L^7.TS,>BZ:% M:*6KTE>C:<@8]:G$C*R\FV$OB-;ZRK"TZW&4';&;V#C=J5BWWC0V_<(J/2PU M&WM.P]1--0\ (=^KF:QWA'L$3" UD"7Q3#WEW^YR*\<>(1U Q)\A6/"=!6EG+JESYS)TSE>(<13C_6/GGHW M845H6>O+3_-->,,4N8$U&6<72J=LM<)F31-2[JR+;,I&5F9('V74W61B#2'W M&N*[A\@?TE9+I[M4&TM6=]:&K3;5'I4?ETHO-$O=U68(4I9:"\E5#Y\7^5PD M64\VVU8O/1PUGW*3Q4;94Q'R4@/6ZYH%G;R,>SP:L*JL"JL9;GP %&K7V/7A M=<%=UV!3I+))AZ&=-'>]_7P\:Y\W%W&-R"8??"V5OM?O]<."IS!2+B\B-Q>X M<)&C3HB2I=#T9O89D70!A)9A2G"XMV/:'C=VW'AQ7&0QT5IYCBZ^XTI#N)+:T?MG*^K/F M)I$2T,;6M.L' PU6A,?KC^BT9BO^]?RJ[6='K'%/'Y> V-EG;7-;A^VRHPS?/1N4&INWB[EZ:=A'=DIU[1/0)YOJX8!C=[I4U#=H)B,:F:@/!+;:/?6HA M4D'VM,F#P1^#K >&V=BNE':#/=7M7B.8QX;6AIZNT:=M]Z!@I/:H;6R+VBL9 M2I82\EX&DUVH-7/6GB*1_3=#I3X._*)/GG9+-NTGM?)$]F7X.11B-B!8EFUE M5ROC:Y!B+\Z#*"PHLMM:,G].MKW0ADF$[X87<+&R$$@T$J"BKV7K9V>&67%' M&!?F;V=A8!BO@K%Z801W%V9PXW@(-\#]A0$?_W88YG![K %WQ+UP$]S[-J-; M<;.=G#7<&2K=8=5M^!:'G63S;<<#7-+*[AVA\)_W-H<&4=&V.0\?-_[-0$UA@MP3.8U8I/:(J>)R:Z'E M\7BD*8Z'1L9X.!X:2)W"@$EBG&S.(75'W;89UAWQ6-U9M]6==:@FF9Y35W%$ MW31+V!VQ*-JNB$D4 MWSU^/*B/*UZA[['"S]4\V2>2M&V=A!ZG!H8\!'N"CXZKIMX\Q:E(EUF LZUGQ*CNJ82^OS1!AE$W">T#K38O_>?2OPO6+[K4['W@J&1"#OS/U! MH%15GVOZ)KK,?H#=8.2M^' /"]GZNUIDQQ_/32]Y9,-W]WV5>=^;V1+&?5VT M;=7#R7(KV^9>D0!L[=8 7+%5O M4O=RQC*<-8_B%#N;5=FR1\O"L#^K*H:O;:[1*_4*A$^O_?7]5F5O*^.VN(V$ M+^'D-FVR1S;A1S@4'N#I-U9+H[T.5V2T$__S81=,^S?BC0W"PC18;@*&>>&, M(!HMAI_A2*$R8+8J'CF+00U;"X(BIX$T4SPM'%<[GJWE+U MYG.^")XN]%T D56?/C'8)\VJ9*BM"NT6N1[9ZSI+KI8AM1^_@8D 1^OV OCG MJ+<4K$03_Q$CVSCL[8:!*12C4*?\+%$-TE2]B!4ZSI*)5^!H0M"PJWN^/7.F MF V7#PO ,%Q'SH%XY!A(- 2^?N0D^6VUQRZN1H?;(<88Y.;UUUV\&6B95\0A MX7C1'AL&'<%UT:H29*.).J4GAWKCU^C6*(OF)=5QT1HT$/I):[$ &FIDJ\QI MWIYT6@:!X,)97(9#_W>3CC=JE=3LON'0#)T+1S;<+ZD%"$M"H\BJI7Y M2TH3R3;\E^YHPK).K!-ELY&-YKM*SDV:UTL=S'^L@+B3AG$8PO'\Y63R:-)/ M9V<:4T?-7'_4X!,0Y7%+5(PY!>.84R^J262^RGJQO09SFKIL3G\"H"Z:"Z+T6[GVGEW_E$OSX!-P/'8W:,*\FL< M^>2*F0WW!:%J-Q3M;JN:JZTS.#HM$?UJTQJ4=Y;WHY296FS5&"Q;E+67E6%K M%%.WMCL-Z)U9@&Z@LR@%^H!CUT0S-Z#0 Y@?SSB:).=K:J].VH/(ZRK2&'JB M)J\0:?G*F$S)DM'Z44+-A;R'T>QK[493O&WT&ZVBI^@7&Z32@VQIZ273T;Z% M:0QI?3['W>=&]0%TF:TUS>? =M$Q(]M,;>-2B7K9<)'.6-4VE5X_>V#YP>QY MC,90\54/^FASH6OH"6N&[K!=Z8DTE?[]$56D*9F\]L$BT,ESM&D&F:/S)8<9 M+BYN>;O"FX)(5HR,XZ;3.&]ZC!.G\Z/P2IL.IP/,($J+0XZH01)14_RHG270 MY&A595(SF]6!#J C3@2Z@)Z@'^J+TX(^%04C.VQD9(,#K'AX2G[8I+ <^(Q- MT0XYZOE$>W_82Y3/1@00N3F%76M%UVJZ^GFPM):_46UY6SZ7P^6P^ELNJ\^S M85GJ*KVOST^HWFU^IN0MCFI?FI[GO MLWW3HE9D9@,@=^:D50B++;VEE#FV#I &1X(6CJ57 MZ0];4(9O,*>1T:-]_CF]FQ'(]GJ4',>.\Z.=>2!3BW-GF\Y&P%;.,H:PZT[1 MHZVR+=,KFV8Z\)325"J5_J,Q471OP\ZP,U%D36 #U<%DCW;B4\HY?8IZQFZH M)^H;^Z+.L2/H[PR$W=ZYC%7(:SXR#^MR8#3Y#PTQQ)8C+4P#+H1TAIT0Y=3M MBUCT)";',U.]$VY^C9V>%44ZKS4P7EDI&8_@2:;?%,+#=I<:>5Q4Y=% M<6FTMP#AU-WL1.5T#7WXCY[.XN!#8LQ$Y.NE[+Y3 '-ON>SP2],:O,$IT%*R],;,"O-7&T2CT"@0#PAD>_!NFJPFF\#Q M2-*8H]+GE-W+ZFP$*AHV)SL<^W-?>(OMOC&K/(QI%?H1A-$%Q0NF]6I<$;$, M7_OLL4_="*JR*^WN]AXGN^WQ*)HMWM%L%MTI-5$50S(=KU8#P["\'*JL;X@2 M1B="#L_I?GV<.!]Q/!Q?[ZA*]7Z]-\;:._=.(UOOWKOUGKU_[]L[]4Z^A^_E M.\Z')Y,D6 E%W2 H(4) $& #]($]2K@5CP%XO]Y01LB]08#B;$73CGGVT" 3 MP-\K3&'I\8L9FGO7.64N$8ZCY"@"CT_5WH;J_T6KUAO M=;0*R=K^>I5@&E.5*4[L4PGOGZ ?2HA7,K]+YK0.4OS>?+S_BYF9,([OZ:QS M9#,-\+LP@X;#9\KX2ET-P7>VZ-7AV:VTQ!])9VN]4Z"C" ./#>^R-?P./\!O M/)8.<0[%7ST)8Y!+^N8WC[EJM9!4+;NO:!?SK3D(1%+W2 MBLL)--I\0 CBTC0\\\4)'T*)[(5ZXA3+KHHM1Y(&*1\4C4:C"<*>X(@ MY4R54E_K1#J1$/SU?L0W/0\B<>+(CR+.RE1V[(CK1$ 8'RVN\"@;*$3RPH#Y M8K[(9H<]\C<"W""J.^7+?$;*'Z#FCRB/#LYI]HE/MH5P))>@$L^(*/#V+8.F M4A+PELX3;\/C\CL($\?<#2%W2]36].R GCP+FDVJ+V&\"D_&;TE3XW\3CSF+ M<2#8<]I%D?HC#Y_#WSF87GXS\M"X56.*\P)-K3,-],MI\*RR 3ZMF0![PM'_#5\]2\G-GW!IB>X"*YYC5RKF4M\H9X M:!/1$4F5SV>FIS2T>U*#SW:1-834U ML/*!L"20R.+9XHWTX@E,J>W^C-CN2=\TK_3S_# 2?.)P-ASS&,'7<9:UV^'? M#@%-TC(O@.KJ+[QR5[E(BJ3+LD(77T])&XTK@;URMQ"A>2U;/72?+E?CLMF0 M_&#G&[??:GDZN4SJ187QQA(($;J8[#Q_AU3M0.[5?K4OXC.K..Z'5#E*-GPH M1!)X](CIBL*+\9A\&9]$0<-@L*)Z^#IT7:6[@Z^XKRDE@NR3@'FQ"%'H_V(P M).X _Y$/V*7BM7KG*/8M?*+*I;LDA\^KA>OBU5/8CG&2OV5OVJV/S MXYO8EXU@LNL0"[]0(I XMX@G#[R%U+AFK H) NQ;0O*#(@A_E6R\L/UQ,JO< M]$W2#,"R#D# ;H1X7#X]J*'P!!?Y=P;T^0O](D+;YP.?4H8C*>52TMP[OG7( MO)G =RNW/,Z74C8]UWUN*8;L\N^/?'C,9_?<_6H%WAOVXOWPR-W[(N1]2P+> M5_]\.]O50()'^-CXD1 X=Y3)*D$\/*(_C/8R5 MO#*3VY_"+&\D*#*=[O0]WCA53O?!X?D+X6LJ*>6$/_,(OQ;^(U/[S"P=C<:I M-GGX'?XL_Y;N9PX\+7@C'I>(H3/OQ>,LSKQHM7W.HAV-'OSAB_@P/L-%VMOW MR3TVG.+)]#A^CW_CW_C:X3=7!..,%^:79N"G-W07M0B/.+V2^%?X8#\B72\* M22KC*PEM--+(>_89\8E/OF]]+WV/_\C <>S)5;+3CT%+I M&^F^J)_JC_JA_JAOZI/ZK?ZJO^H?O=DA2:(;'C5- MDM"=-2WY'C +>JWNQ=V)$-=#JL1/XB>Y_'4/LWT>'I_*C[ MVB<%.F!K?]O]/'_E6^]SVNZX,@*,R*+!LL!K\PLDO.@!=XJX_?#1CMP%9&*S MG=P]/0W^)3CA:_=BUMMH"T&_Q/DM?^AS.VI)FW474OPO_G%/'ZI-$#]]^/#[ M=^0].!*9$#G$:Z(/]5SW/C[*_^.C_-[]CI]'SHEU<\@3U:"&/ MW VOUE?]<#\VC\6[XKU((@_.3RN/O+)O]P>0HX@>'/R9OFHZPWN7!/.=H2-3 M1G(>(,=M;7*\&_QU>)[BL23P*[$O^0^@:&2BBD>F>'=DYN^_;OZ8/R."-$O? MAG''"+_:3I%R,J+GG/XPZ['3RAX[=Z':<3'^\ZX_Q@C[$\=ZSNP_'.LYIG[M M#P52DH#]Q8CZK_Y(O.+6\6I,R^_3;QCCA9#5PU@BZH-X@O\_QOG_S9W_GP!PSC( +.=- M:&@I!8M.D1@/Y,#,DI&1T@9R!4 H &0 $@!G !: !,A%\ (H :P B@![ #^ M_W!F$S4?A![-*U%I0:OD<]I!*ICR3&%LX.$JR6#TF (/U0FSS93K.:+:.9NL M5I@>NK^67@YPW+(#A._->_X78B0@H.R/O/3](@+6_HR \Y!]A'DN"<@M8@(B MC(Z 34 JVA/0">C.!*&ARK'>_3H@ZIE M.ZI)W(\@@$>C9%:^@Y[L-IQK^+(IU\PAJ\%%*N^Y/YY$N:4[H \'NA>0P -: MC * &2,^(%YB#_@'C(7U 0F!@< _X.LD$<@(] /*_BA/ZQ,:"1=#$JBZ:SO$ M77047QKGWSWB"\C]ZJ$MF/IYAR#W40=)?L0ANAHU5^H9;Q$DBC0#[*$*1 6& M[5B!I\ T$3BC^\4$A +BD\!-BZ0?$],K9H9.NZC1[Y)/2XVIQ"\D0M0&Q /Z MD+2 ! L?5U["U.>_R$NHNT!^>HEGX ./">8$1 'B7*2!2#QJH!1P.-8,U$MT M ^8DQ0O"P3G(F$H&1"(4B9\.WD MB^1F (R7PVU- Z,*<5EU-XH6SP?_7*A-\8+2&J)9JHI@Q$"GAR#,3A*DHG,= M4<(H>AV<"]VDS%*QJH6PF70PA2^^PQR(^-,+W*=%V_!1NQ6]1EX%2!=74#7)%.?^H\W$K)P=H[]"H"#P+H$(= 2:!0$2= BQH%JP$5@6; L. M_\Z"P:6QH"%P$/@6S$AP_?J =!T[2^U^/CB#&70 ;%^#H0<#-7!$T,^('Y*P9] MJ]@0;+L\1&@P%)00*PW2) QBSPD-FR]H$"%;&EO)YW04?CIHA9!"-ECI$AUA M3\I($C:U7NLH((+W*_G01'Y]+(DVWI9O.)@LDD($ERIU-AR&&&SHZ@">X//( MKX1]BA:E \&H)+36T]6YG]8GNCK!(+_BZ,+,VE1A6N( )8.Z@*4%!(7WL^R! M\+ 7";X\TDC"P.0804G4@]I])[DOGC!0$_'2JV'XX9)B*B(FCQ_I"M$$YP>2Q[S$A1$14 M;'PE%<*3#_2+JK?8\NT]/78;I[LZ%V-KT7;]&$XL$/4($24(LH9102X@C'"!5 M":F$4\(LX9*P2T2)F( 4QM 2#CV&WIDP,8@F].:D"=^$WAR^8&G"_/!VH '< M$KB7R0Z'8T@HR*"AR"@C>(N(6T>#2$!*M!1",OWI:]* J- M<) ^_Z_=!!+E/31'TEPM/Q):$ZY"X:100K@I!!3*"#.%!3%+8?:B2V0>K/?\ MARR$\"T*88F,4SCK817.F?)_>ZQ&4*A0R'0I/'I45%XI.)?LA1\E*%&5T6_@ ML*)Z_SPOGM.#TW>)H$[,OS)D@!@M+UP$M$!I6+MP_P%=UP\IL X#@6/0?, HZB$ M@_B$0IPKEE'(N13"0YLQ")&%XI9Y4^U$&5B7.5G1<418 !YPD@C+3TCHN!+^ MS=1%GBU63[VH3/1H>A::KR@BSC;MA8J(3\CF0N(B>U823 D8>MA_H)^D% $[VASKA14V(B08K0UA#\\E'0I?C6QA.WA] (BV/Q M@J)9"J027:DPS>>DF6[$A[ 7;L/#2]SP5MCKJQ5BOPXO_L&*2MX00T@Q]#FP M!Y>%C4+ 82%"$/,GI$((8GX[V0O"(8]F97AX81P6(A!BX4(*X=;$(/8NPM_Y M2AQ;$R8FQ!M%/LD*_H3^GMD0W7)30BV8\[3]5H5L%5FCOZAL2 MQ"R'Z$&ST=HP"&'G>Q0V 3V'U25\82$"KP3UR/^ISNJ&0['484&,9R@AS!7N M#;==3@_;H<=0LE=#FL+%UZ9$!,/!86#/9RCY(QWN#K]_0;Y088D05;@]U!!V M"$6''L+>7KBA&/.G$0M#$-^7!&59G;7?LO@6C""T30 MVA@GV4,&XJX#@1C@"UB$'$8E,IL:@(FHG.4]O&,YP 96]L-(GGY0.W2/\.\Y MD3I;DKQ^RH3PN.0XC!ZRH/P[=ADJ'!JE+(,; ARNP@"$ZD']RN<0,=$^I ]> MYRI[9\(13G3.=2C0BT08Q3A%+!5B M#FV'90K8H2)BV'%15:6QCQB[A &B-^"U<2LJ2_X1D1C'@Y)",2 M#[,7S"+*81FQC;A&G"/N"8=M?,,Z8E3/=HB>PUYT#&%YO"#_X>Z0$J@KC"(6 M$IN(-B5$HO9OB;BV8B0^C_2(W[@C(E1(D(@^_"%N_A2):[V:GUTB?RB8<"'6 M][B%U FQ@^NP6%B703S- W=^WZ-FF^'P^D$I9.B1$9L3@4/A8:BP]#(26%NH MXVX =P70"UJCX%?K@QKZ$J.&P+TJS"GQ"J',&2:6!XV)\L!CHC(QF:BM$!XZ M$^F!T,0M5S31F%B_FB8N$V-)R$1LXC6Q"E- ++]I$YF)W\1PHC!QFR@.E":: M$[.)XT1P8CI1G/BLL":J$VT[[T1OXCIQG=@!0R?"$]F)\L1[XCS1ME--/">2 M$[V)UX^D82\1F/A 9!J2)! ((X'3Q/RN"8'/05L%$ZEPV4/B%7<64A"KP7J0Q!"L*=^Z)HIT;8D O$#3?,7_5QS:!ER(B7A Q62@\/'S- M@MIW2\2GHJA@3@B >I$TOCZ$^+-PX=/#U!<^[!M^#R6%N$,DCA3Q&X$X3 Z& M#A6+B4/JH8UIL9CXHBPF%@>(<$/)(F01>V%"M"P^%C&+TL/+HB51IZA(!"U^ M#D6+I<718GPMM)"C':DFM1M8A8_"S2%AV+ 2C8HFVQM8A:G"UR M F&%O$59X;SM/31<=$3\S1J+Q<5W7 _1"+%<%.0T@IB+E45)X681*-9#_!V& M]IR+L[TRBL]0%Q@ LRT)B$A#"K&#TL<0 ZA:3!"V#Z>*P9WWH.X0FM?%$[\H M^&")74-*%ROQO9C@&Q$JSX8I"<7G21"*1S%T0YK($LF'AT7NX?5#D@@LO'XH M#[>&!\;D#F/1BNC%4QP.#D$3LK?+(EV-'G'^^99\%,4GT\-TTH41&#*,N ') M$74PTT,M!V$1AM-@+/]D]JJ%FS+-'HI1$J$\,QS2=*Q+DJ6V3(D1S1(ZK()1 M&+,=)D7NU8J1ST,O ATQKDR')Q_KQ8]QKH,:%#)BE_B'"<,3DTPB[?%CS"X$ M$ DF3<88(Y[MQA@>T4E@.ZB,530@$7UL>FA-&0;Q/9J,9C9!');1A.B1FS$F M&9L46,9JX9!BS,A65&^E&:6,I#HNHX*P"<-F'! V*^2,X13'716FR7CM4<"Q M&<.,!IDTHU[O_\9G'#3&DIJ,6)0%R8X1A:1H-#,J".E6;,8\8[;*T7CMN5L) M&A,]ED8YH[[0 +=HW#,B8 2-7XP3(:*1T6B6H33^>$J-H$:F'A%/TZC?.'P< M&NM-WP@9CJDQULAA8C4Z,F2-H$:SHJAHJDQG.(P'#7J M-XJ-]:99DL0PV4A5W#42"*N):$3V%_I.@G@SM-Y1&[&'-:5A(6,KVWAMK'-Q M]"2("Y/\XOI M_8IX +UE8;X1)<%>/!7>$^>-]$: (PZ1IQ?50^*D$E5& D?\7<+1^)=*G+SCBF"2.!D>3C[MJX4@L_.BA%_F-'T>0WB6B=-5ANC=: M^TR.F$%X88E1WICXTOE%&+N'-2Y8HHWIHNA8+"C6)E*+K\73HFTQY[A:G"WR M'&N+4L2?HPVGQ1A9%"T*'2>+4ZE&#G#1YWA;!#HF'1.+(1JF(\ZQZ8A;3#H2 M'3.+1D>J(])Q^8$O7#J*%@V,\4/Q86 1)1%'A!"%$%%\.X>#4'XQ@_C>F5PP MM@J'7$.8X]@Q>\@S3-VA$1E;-,6+8XUK53)O1"0"'/.'W+J3XRP(X"AP)#B6 M%6U,'L<+86TQ\3@?XCC>')>%QT/&HL71W4AQ7#Q6'BF/R<+]%N+Q\KAYS#Q: M'CV/E$?(87 0\N@Z=#SJ%AF//4>0XY%(X$@])#P"#M>*3)PS(L#QOSAOI"1V M'G\]!::88WI0MTASU#DB\D*+)HF[X>0A\/(PL$>>-G$03HHJO?+AR9#[> M&->'PD**$;QH]@@\_#M6'X6%@,>68[)0YYARE#UB'XF(UD=2XO"1WHAY-.:= M'D&/CD6_GQ91[\B"^E]]'Z>%[\9K=ID: /'"/9)!I%7DP)R-*,)58<(:7P\/4GS!B6$C^*3LBO8@X1UP-;!.LE M)^R0\\'UD ]B54()T4FP)PY= <+0H9M)0QCR&DZ,@U82O;R81)X.LA= Q XF MQ2$_$U$ M'?.*U(J\(D[LBW@DE0&PDA7#_*"K$H':8_H2?1FU1*S,.-<)Z(U$A0HM#HG4@H?",2 MB1AN%,,D(O'Q$81^-!^2#V%,K$,7HAIIZX@NU"WZ=WPEVD+>4=CQ'>F.E$?V M$7JU0Q;20U']9':%XG,-44$32GD='W'Z!#^^.^D/O8:Y.;WAST#'U M"2E&KZR/9!H2^*A'K#E2#@585<@[HB/1?OA1U!XV%A.204?R(1]1)@E']!;. M)&U,4JDPXX$INN>D64D6Q&*2U F@Y!SI)OF.&T3("A**68T(1A!&)J Z^B]6 M9;J)8,A77B:R13AOA/K=4]Z*;(F(8Y]QT-C;>!B"\G1%?S*P))"P3#C8L2CJ M+P2'T$FR(R32V;ALC#9Z#X^3N$/*W5H2@%C84UL5 M.722_*:JA)'1\B>9;"D-"-TEK2Q;1]\#]">)-#OJXU:1S<),427/ Z*.F $4 M>D ]%D(IHANR6W%UO#OJ)V>+^\FDF!1Q5+1SG!O"%H..!DK$XI\0T(>@-#HF M*).##6E6X#2.0=1\NT2-B;FA77,H, M%1%#JT9?7QIIJN>30$[J#I&3;<;$9&72 T91+%WI!Q$[B">I7AXN9[BC5$\2 MB(B.,$K%I)I1+2DL)%+:&\.2%!4CY404JUI 6C4NBC%%!.*?620DHPY7!22,FM(QDBP+9[5\G(37'2C[0\5$\B M)-56JT@H2@+1"\E3\>)])B97I<3#%HH)/YCB:VG,"9DR=D*)D'?28KA2[DDOR&(EX;L DGI1Q&Q&DU$%T M*N$2G+7<.GA[[!EO@\Z521LZI%W20"87FPMNCW"T".%Q5BX\B(Y<,RCA-X)$"* M'*60J\?9X;>2=0BN[%A*I6J$\$8#R(#/P7B!M!@"!ZV, 3@&X81R9:FR;%EJ M(E"6S\61HX)R9LFR?%G2+%V6C,JA0W]R-HFSY%G:+&N6.$::FTG% N)823GHV\".1(+6=Q\YJM M3ZPI1L;H71;%L[4$FTU.(9IRWS_;TII2EK0B[ /5 *X:#R$_W9B/9]B&C"Z& M'>N6B<4 )9@R0^DI]%DZ+?^60,O 97+B)1FU%%Q"+0V7A4L#S(,R<7FX[%DZ M+FV6^;_%)<\RA>@VI%RV%ST?.5S.RX]H26G2VWH5P'5['-4UFV M\?)*%,N0$P%R&\8@1/G-=:1]#,)O81W#B)?T61$F?9:%:0AR8>')-7F#I#OB M#$-ZIQ:+)8WQN_@MD32V+CU@ ""*X71CY6@V:G,I/G*%J$M)9;=$37FEW%RB MK!9&'T=$6/:RQA2T?/J)\N2+$SU$WE,2V-5M#%^2+Q\1YBD.I-W!UK2*0+A! M*^*(;+;XY:PR:!DWK%\FP.Z7O2_Z)?[2>>FRM%^*+/^7;XRKH_^2?_FT'&#V M+_>70TL$9@%3@:F_!& >,/U<"TP'9@23@-G I&!:02IB#$R?I0%S@OG #&!F M,#>8%4P/H5]K@DC"'(D%2_H5ZDOY'ZUC;N@!HZTFGLF6T5)W*7@ISV!XA030F *_"8 M'$5Y#D30WZH+%>';V%T='EV4\^8_90_9K0RD'F&^V8@\PYIHZ0C(G&!&0J,E&$C$PWYB%3D%F%D&.V M,!>9:LP^YB1S5);(Y&/**#>9DDSI&R+S))?@^T2R"15#$+TV85!PM>4%E_+$+J)*69)T*<)#13;<4LND-2 M,A^9WLQMYB5339DKM&:.,\.9)CAP9B;3G*EX3&>>,]>9TLQ1ICH37RG/?&?B M,=F9]4PX9@)3GPG//&+:,PF9[DQ]9MGQ&_?@8&6ZUPR:$ Z$9D%3A#=,FIPP M*W%Z1(! 91DO,>D TTCV^F*3GC]?9;<"AIB<,#ZJB(9X1<0%H;,#DN>+V%%& M+6.Q5#/B M-/&-/$W'8K600 DJZ6D&*<=&0,V9IC?SJ&G.>XWQ)5.:)+$<);#2C&E*FQ&N M""V,^@WC8R72\4<.!!.&\.J#R4IHB._RFX,>E"6Q(YB5J4F5Q2Y3P'-*+%<6 M$,T8?:1WXU12P4=P=!DRO88;=YF0X?72G[GS _U1-0&:Y99K9!96+3O:C-!&SR] :5[@[(YM/OE#B,BR:.,RB;]\S(IF4SL]F (%!B M-AV;FLT2%S.SL:G8W%66-CE]AN0[F2A&PB%EN2H,WCIBURG5F'U% B M-XV;R<&\(SHSM/C<+&X6-SV99T>\I6:QMXD_6U!R-Y\._8KFE#I"HNF08L]$ MB(",/)#H8?6"#R;1Y= M 9.;R4W/YFF11/>@+'!"'VN,*4O#I8$3PR]FQ7%G#%',6.LD4H\1O8:*3QXE&;'0N M+'>QIYC,Y'YZ33TAGI7%B.BEHRIQT19!T%U,G(P>N( M.D^=5#!&CE$OB1+9\:M9%3>$'B_7Y+&R.$'4! [:&RY,=D(Z(5U S::PU&]$ MB&"(BS9@)]5+@N0@0WL(.W^=+"C?BZ22G,F?9':*,_F9]\QXYC?3 T;-C';R M-?.9V,YI9RFBF]G/='8&-+.=S\YRIK0SW*GM!'<&--^:U\YRI[J3W+G-Y&/2 M-J^0WLZM)!!SW.G"$W":'[N:7D52F;\)DZEKT6+.AJ87%*^2'Q_,4N3D@R@B MO%*;3CZ5Q4-SO F=NL$5,VV844J(Y\,S?/G)TS;^'7*;AD4T9>MHU;CQA$R" MPSR>VD4K)9S3S@B\5$C0$ L\PT@C'*APT/B6Y&VG<]%8R-Q&1;<28I %+0QB"*'K2#;^>?4+D)O9(0@B41'"^ M]O)(IIR"F&9&C>@3:\EDKT9E[4@)'-XR] /G^PQ9".6 U(F,0Q+1S*ES)##* M)B.+R,TL1ZXS6K2. %J,K?R+ 4MT8^)S:QAILR\R/A>?(\*W@VFA-)&OB)-$ MH49IF):&09^,!D$>W.BI!M.!*+_S9-/KYW9 1(1D?("&J$^\WS!R]=E1K&K. M!U6?J<^-9NSS];G1?,)=#&^?K$_8I^Y3]CFM$#Z.-'F?P$_:I_ 3WQG\A$?F M/HN?O\_A9_*3^"G\]$45 MA4J(6N8\.!('!)G'QA74R8OM M)ITI)+@$(=A0172RC ^M-/68@TPBX?)S &J@7&G>9=2>:#Y!GZ10"/F6C S(LE M"&-EI$F%'TO&I;D$"UA%$-&.R,D(Q?9S"-#]5 !->0"-5$@HA>YQAU98=/%@ MB^(1,421XK'0B^>N&A#V&8.4H+9OX13"WOF(#%.*^B"A6A\JZ#:-5H4*X([$15E@*C>P9+]5#GE"_S^DPOC%B'+UH M0A(SY)6Z+T1,HSK(DSA!#& M-;&.\D>I@\IB^1"_&PF,B=)^*(G#E\11(3:V1/HD4O2%+,"?I7DQY.2YY&_: M)C].W,KI1.RQ+^3&JX2J!P>2 ,:JA,_-8MCGO*3T.5D>@AS8%U<34Q@L+"(. M1'^?"2WLI0P#1]F";(*V%SNB3J_O3_N3WF-F0VFV# L6Y$))4+(E'#IP)$L" M3_A!2!>OA*P D<.:C%K>-H&%GDN^'@XRN,7$ >H8)ZB*W;&P!X]Q)]8?5)5I M*K^AKY"#8)7S0'CZQ'(B_%B)4%'<9X&I;D@5+49:1>=#5- ^ISK46Y&:*AE" MMD!@U\OAED\$4UCBT#Z=0[F)TD*MQ/P.K2$3E?0!+,82Y4V$*!MQ+CKEB1SV M"/V< ,3K(>50*UJ7\1VN$?FBOLZ\J&#T('J1!(PV01]_>M'$J&!T0DA19!WY M.B5(G8T=D(V+X6/CZBS%^5YE+96$YM&&'GK4QNGT2H\" M9*HS"WE# $;%"_(1SS%Y)?!"J=X M\ ;AR%?IU:4+PJ-5(P21A^DB-'!Z.//+[H7_8M:2"ND>T@E;7J%C]BB(=)91?VG\2?#L:S-<+ZD7E+VD)@4I#=.:EA: M^\I_C4$V'Z2PBUD^PAHEOE"950O9FYRTHTDG79.^(%^0!,HYJ9VTV<@G?3[J M23,F&!- WYP45Q>UX8^*Q(@87K-#*:&4]V,H34GLW9X?]3KO39[,QM7$Y//Y M7ZPF%Y3TP[@CY8!I^0C,*@X(?J"W18+HJFEV^]9\ZBJ@-Y: E->E%F7&ZZVT M2FEBKE((D=DQA64*:ZT5YHK]97J M2D\7<2#)!W$+)O"JV#>\@I!;/CE@51>-@Z:J.*^I0!YR%IIBU1_&@_-R05$L M(KL_5 M>NZEEIU83\WB@:4LH:E->]H]3WIN8^H7"G&9Y/A"63/^7F!/A6&2FW!U M;590M+>4*;YTUO-3PF0L>ZAUQ$^\DFYO4]0N9=?,.H(BP*$'UH@EA1%UV%\1 M@$(6;CO"Q1[G>440R4D1/K)(#*!Y#/>P H* M>CX:ZJN):4PE7138,[L!',HI!2?FQW%T7R87P;]@N98=EE/$W=G(-K&YTH_5 MN (QW1M>6.4+^:&.:CT]Y! 0IM/9&<_'6@/!?)S='!Q(626W)8FMQ(G6HE%< MS O6(?BZ3B,7>5CT9EJB1*%TZL.7IE$%^:A071L*HN7^8<+DQA/^_D.O544 MP?B#(SF3')BT2ZIR*LDAI-:GZ=/W:9B4?>IZF9_&3^&G]#!N1O>F@F%,H]^$ M3A-)N#0SB4>H$*>8:Z?QG*X1MAP2447&:T1].?Q1$"V2S[CM4"+FZ3;0N7KQ M9"RH#9CB1NK&&:&6 M8K"EC8J'W++*A3I#E:$RJQYRGAA-A?.![K6(0;Y![(H/&3D7:A 58:KD<:&V M=DZGGA8DZB^F^'*TTAT%M#R3/$:,8.\E0A%(^BO",3\2$:V0G'_C)!0.*ZG5 M;E2#FHLOZ@,KC,H)"3B(4=]CW1M#&QV.9,B1V!MJ-KP. PG:E27#+'.T0)\> M8 YS:M0P24=(_OCCV$"\S,8W_--4%_XF#S7YB-A!\0QBD8DO458)=L>"()'B M(9I7/ C+J3#%O32SZ$YA17(N@[01AU(J"!:"$Q=))Q):<,JWHJU#Z^!!W&.Z MW"!F+[2_G=!TE9ICRWP\417,HOD MAE@>NI43EE0B8<;OF%]88DX6U(N)3LHJK,:PZ&QHAG J4)!]Q[]2:G^Q,GIJSFZ=F2Z0X!X[RB(",_+)S@\[, M:V(F.%6;QTU5I[H8W*G>/((?ZRYI4F7IEOKQ69U@EC!SKHP]9K>APN.DZ):K7:T M5DT4J=79QVHUMLJ<>:T&06:KAXC#!FO5GUHW<8) IT@" 2QA<]"B:3W,*&4 M)$YQ9B[-A\Q#YN%3;:[F:W0FS]6J4>,/"(5\U5SF4YIY=5_\4;%3.X=P9P$L>8_Y2[5U1"B!U I*EQ8Y0J4:@W@.OJ*LV6H,P]J@>-T= M/C$M+@&[UO3F#V0/*GL,PJ83^U(@#G=FM=3-< 1Y),YJTPE'WP-K_/5=K."Y M *<5U$*GQMMP?:F(>'L^2["BA,)?AQMIY8G8D^-8Z)R%@-;Q80&,MT-HO>T< M(T\^QTF87YXUSVIG%15((V!% M,KQ6MM96H9Y)^I1KM;#N6A67NM9$!V#DU@KC0?7X6&2M+AVYUA*GU7H615$2 M1%>#;B1XD9UUS4KEE+9:6PN%U]9H*[9UVPHHG+9Z6[.M9T-MJ[@U.X'II!=^ M).P*#A4^*(&/7*'<.CO :QH1#C,01!VD5F;BL )-AZAV4#877_!MWPJXD GY M6QU84DH7E]=ATNI)/9PFBP9L_LLTI9+*_ITX_J:2, <)=ZLZ95M%?4;?XI/XPB@TOZ:/A6%+KK&68V61&4-DKPV@Q'_K[,?#\)*& M))BG[E/ MQ^Q%,XI.&: #&EH%NQLY*_)>TZF.0[BCY1K\BO"@A=!C@3*T:6]6\I[4ZU](#>0\1<$57/^E5!!10:G#"#%I=? T+32"X-.DA 7HOP47*Q>% M&>$\J-* 6WUK.F.U L/DMHZOKSUH5[!K^0IV=9ZR7C^>T-?7:TUM^JIA0Z"8 MQ$A+L"U/R?PL,R3*<^;%:6H"*$[R5A-"+*'F>0IEF7X.'$N/I2*B>$>((]^0 M;P"H!AR15?RU?$-_=;_.7^VOVYT11 -B'6$="0) Y@Z6[(>X70_/_BI_++>"O"U@'+ -V @MOJ^+(!'5[90IQA8W*!!B %>.@ M4V$215 9!NC!VP%^0CM,+,@:Y)3\A&O5^<0MPCN5:8*NK:U.16Y+!ENXDL$J MJO)L"KB04 +.*;&#I<'F)H*N-5A$6P\6"/N#S>#IV4Q".-B*4!)6BW&$S<'R M8&NP1-@FK!)6!PN%W>,,80]OE"EA637E/!<#2SNJBGY;7=CU27EF3Q>&S3QDNHQ=',\S3N[-NY5M.@UT5 M4 =EJJ78%G523L-!";^:(Q1 8;"I4Q&TZW66F+ONCR);M:VT3D4([RI*V;LJ M7O6M@-?>!.(U\-H^-5\L8Y6QS%C;D3$VWSJ,M;UB3 44AM-A*MTJ_JT(C1V3;N:7= "H^9(Z!IW"4\73FC\L&PN'CP M@!XI9KWIJY'5)519PZR)4S-K)=-^[$T( 0N0+$S+LEF8X]Q M]&I"B2J=[$]6* N4S58598>R)2'>&X;-'T>"8F@IWYZR3MD=G":KZ6"F, =A M+PT>7QI81!! 7$'L$<1B+F^ &B_>Z1_6$*N!.!KE80NQ9UE"K%H6#XN6=>#= M80$R;=FU+%Q6+ON6)7D$8NZPO EVUP9"&PK(^S_(4GH3=)-5UYS/EW>ON"S9 M8F->M)/PB&>+5%@()*60>N0B1ZL/E\P# Y$/2\9PK_IA-XN8!/Y*?YB-!4[4 M*J6M>,Y!A !F]@(J)-YH$MEJZ2%(Q,+L<#?A8FZIH[RG ,A!S'YK"48CA:+T M@?:=F0>X#EV@7+%RP#9 !/8"9(!M@VW@#$ 8< UT&YP$2P$C $4@N# 1R BH M ,X*( (+@<"!P* 9 #>X!,0-V08RP'&6V 1X IX!=X"9(&#PJQ!*1>T( D@ M#XP = 4@$G #9 "< .@ "(#> Z0 J 5R4&0 %, : '*0 Y J@0. 7< I$ M!LH *0 S I@VP 'R,^B (X-Z($4P+)$')(/ 0'(#8P

.@ "T-PS6 H@",">]= &#RRT'MKRPEZK/CL%6 .H M > +$H(V0 H@:*$ J,]* =( '-H5K836/PL5X +8,^ +9( 4@&G"P" J,!7D M 5 \X(?0@K@#&"?=0.D >0 0EJ*4\@A\G*D3=*. 3JT* T0( 6!7!A8@&D M +0"* "8 ,DA!? D""IL!6@)&0&;0%Y /.M7F-^! .8"\(;SK'Z6:@ 'T!,H M:*<$]=D8P'S639O>& L\:., UK%0($V0HN@90.D !H4-5H40!+ O( "X .D M - 2-JDP)560#L%<-)":?%!@()=XH,6#A"F-26D (8P]=GU;'OV/2L9$ SL M:?,",R,C+0@ 93"BG=$:"Z('1@#W;(&@1$NJC<\N:5>U'MI9;7M@6C '^-(> M:,T .@ %K0<*=C=@>-"V 5 /MK1Q'P6+@"K]= & 6:T98+Z+)S@4?NKW=(> M%U( @%I,PAP@/="K50.\9Z,&\EDRP'ZV2' DP-(^!6:I>TR%H40($64'ML: ^( =@ Y04!;4_ 4"N@/2ZT M !2TTQ>Z0$W 0;N?_0IP:[VT:8 #+92V#( "" /@:X\+^]H/+92623L^@!-P M 5( +@!A[91@@4&:<-#N'XZUR=H;P+*6-8"K;=5:MF"U[MGMP!K A&!>:-D6 M!:BUP0, +:3V6INM==A^!5( WEK?0?#@,'"A9=<&#RH"U-IZK8FV5.NE9=*F M 5P 90 7@)>68FNQC=C2:NL ,EH/K;H64!M%H-E2;&^VD-J<+<)68!%R^+,< M:7FV#=MM[<^68DNT%=<>;>6SD%JE;:U6/LND-0,H!F2T%%L8 ,[V#;"DW3^P M 5 [-DG@1$ 96MO(-.V;&, $%L4 ',AB@"@S1$8 ?ZV[%FA *N6/("O%=M" M:J^V65L4P-968[NEO=D29_6UB@%L;=J66SNV?0F,!(P%1UJB+<6I/ONVQ=?* M;>&S=("0K6YO9!LKP"L<:9^U3%JTK;:V5PNI31)@:678L6V E4!$($%]H#K>AVF% DZ-6N M 12W3@$T -&6:HN^I=7";INV%=NT +[V+["MI27T:I&W_ML40"90/!C!\-/B M;X>V[%E(K8$ 0!NQ?=W>:YFT/ 'VK(!6>$NMG0A@$C $RML@K037#0"P-2^P M:X6V^EL-[K>6@HL"F")@;P6T%MSX;,CV,R"B?=:^;/>SSEO8+;Z6ZW5UKW+,C@7ANQC1H( M-I&;;VTXMJJ;9;64$ &R.(*:(^W 5QJ[4D@#! '@!1T M;M6U^MD7[6C > NN9=3^#@ #Z]OW;!'W1]NKQ=2.)NX*#%N+[0TWADL'^-C. M<%^UZMD<[KV6GU:?M>*6<7^V<5M#046!;JNY%>)6;K<#AH,HPD1 A(L"B"PP M:B>X%]O0;<]V6VN\Y=M.;P.XI(7FP@[@5/L2V#>4"8ZT.=OR L4V@XNOY>!" M:A4*05H!K8&@A*O+C>"N<*FU/%MV[>X67[NXG0A4;JT'Y04F;3&W<;N?G0A\ M!\@ LMMA+:G@E8L", *D (ZUP@:^K8?V#M"K%==*!JZWXP5(+KO63"-!Z:V, I@1/;:P@4)F:14#%05K ;66#8#.I0K( 9*Y^]F!+;66:.N>U1.< M!S )IUIWEZWE0>NA7<^V![(&KEH1K=BV@CM-V-(NV+MU9[L!6!%"YY=OJ M=+>T#-V#+:;V,X!R.-** 1RZ$%TL 86@1 # ]>)*;4VZS046KD17I2O*'=W^ M;.^WV@%#029W8HO#+=$:;;,(\%J^+9/6Y>5V*[4BWG,NOW=:B MU$=N_+ER745L@$-V6!GRUOEL%0(HVB2O7'>)> M#PR['EH K;=67;O;I=BV <@#U (.;7!W@.NP->!2:],# ES-;OAVK5N]%>8" M=3>V.%S;;AG 2[O4%9\&<@6XWMI+@D( N*NQG2+,<.$-]5DJP#57?DNA2-16 M!I*WP=T$;L3V>7"E]=!J:9FT?-ML+9' 2 8P.A&;$&YQEWL@;F6MZM0 -HV M<4^Y/0,C ;H 2KO8+>AB:=$ B@&EKINV#U3N.-(Z:?4 [UL+;?!6B!M9$.X& M<(.T5%NC0!0!9UO=U=A&!EZZ"MRE+CN")'"DU0DD:V< C%OY;EG77FNK7>N: M2. M;TFU<@!GP9$ DR"[=0C49YT VES%+7=WPIOE(/+^W[UE,KII7C1O:7?*: M:-&Z8=LG[TB772OEC>N6")"X5EO2PI:6@\NQ=?#:=Q&Z-@)H;@H -4(28.!, M"8ZT>EIB@YFV.XNF!39D3=BTF%K,B]!@57O/S>?* %( L0(1K36W!1"M7><> M"&8$S%OZ+ J@-,"?+=#^<]L#,UV!;72778M2,- V"32TS5M?+X!60%O8'=7" M%PZV&0=CP;*$FGL2^ NP9RNZK%SB15Z HXL"> *(>-T 8-P#KXF7I0L52/'" M=!FZ" +XK=.V5QNQ+?8&=Q&[0MS,;B5WMRN@)=KJ=DFU&H;W[;L6>4O>7=X: M:B&U.H%IP6%@CENW10N, 22\=8&U15T EALB8-X*:'N]@-I?[U? 2YOE;=U" M:BFW3%J*0'TW2XNO-?>V:T\"Z=[G;AF@L1O07>^V>1.U18 PP,172VO710$@ M?&>]JP7Q[0&WB;NS/?&V= VUF%H-Y, 7=/L&B/?*:#&TK5MOK6N!.Z# Q>^J M<"VV3-K2@&C@F:OQA=M6:\6UYENU[1R N\NL_?#&;]FZM=X)08:7/:L F.4: M" "^((&%[9%V5VNJ[9/59\FY<("H >D64GO29=*>?!6^O5JK;QM@7/#9;>%N M?$^\CMWR F1WA^N_C2(<=_$ 2%P!+0J76GL&L WT#%*\8=]I0A]W?O=Z:5 P M;$^ZD%J);L&VJ[N[_>KB<-6W'MKV+9860;"]C=@F >2SV8_Z+.7VG4NKK?>F!VBZ75Q3+:;V4U#/E=]N M;ZVV5][,K?!69U"\%=>B"/*XUH._ ,.WDWL1..8R M<\^\85W5KQGW\3LNF/J&;.^W5 #][>!7267X3=WV:L$M]5F1[X,775#O5?<2 M=U$ $-TE[KRWB9N[50D8=G>_/UZS;X]7X1N@%>;"<^6S$5MW;A?7?TL5D/X" M:-6^U=\9P?*7C)O1O0O4?BVVWMJ^+T+ X.OGG=L:>#VY;-NZKZ[!MIORQ>"F M;S.W[=MPKX$@9+O !0&\6HB\I-Y.+3YW!H#J;?6:!%JZHP'=[X=V1HO>??S* M:-VS&>"I;T;7]^"@;=DV : BUKCKY,@8WL1,-56<(4-0UP*0;UW2\N?Q0 [ M:37 [%D,+0J7:*L&8->2 2BTUUOWK18AKCL&<.9&!H*T]UL-PUG (]"YS>OZ M<0EL#-ND+[=6=CMVJ,^&?$>^+& 1+7T7]0NIU0(["8C .MYTK26723L1>.]" M=^VU0=J([?D7>,O^=0(C=_>]1E]"KNRVU9O]Y>&Z:5<3J%N:+Y:6(U :^/[V M*<*_^-KVP-IW^OL7&/D:"L@#,MHS;NL7,/ \,.:&;>NX%F#(KZM7IIL&P-): M:,< U%ND[?TVL>M5B!SP?)V\\]P%1B*X0( "%@(K;T.Z]=DB\)7W"/R^-06S M9\D )]S\+1/X0%LBP->Z@0NTXUY\[22W>>L&, /@]]W;BT8A+O<#04# M?!M4Z9,CK1&8%&RAI=@N@=FS.MU);G/!5WL,?O&*!V<5\(:B@1&@U(O/I0&4 M@$6T3 !+[37/KZ:L7 D-R*K3'WE(L!9C+D M;\VUSERA;\@V8BNJ7=K&RA900/ M>]F_)M\A[H/W$CP'R 0K;K.Z>V!?K>)7$1P'$.;Z;+NU&N'_KQ[W+] "P"EX M=G.YE5RNK7QVFZN\Q>/Z<4<"U-ST0+V7I9O$9=?^=%^^C%J;+I%@1K#T-? 2 M@(6ZOEQ"[G-7:(L&%M#:;=.WO%U0;A-XVXOBW=**A:.S[%D_KF.(FANOO=\: M"E*Y483E+]%68(NOM0V8<],#)=Q&[X$ ^8O0/0M ;Y4%)0(Y@.S6)E"?M>&V M@(,'(>$=;J]68*M%Z/%2?M6]%MJ(K_C7OIO1%5M0<\7 >UN[\&B8XIN[=?EV M>\/">-^Q<(C 2^NM)0 3AD^Y1MZY2))WP/L#KOLV>;NX:5_7+Z+W/A6>Z.5Z"@:)7);P"M@ W@Y_!?ERTQJ7W@:NE/>-2 M!;S#5]K/K7CXI2L.'O&:AZG#,X(V[W5X7FOCK<_^:SW#6-J9[?J7 ,RD!?-J M;0W#1>'];(*89HO?S>M2=T/"V]^70(-@/MPD4.?&:]^YHN 8\$R78KOE!?T: M;>VW-&&(\!UW]$NM[>EZ!_0 >0"BKM77-RR?]=;. =C"X]]WK7%7)VS?Y0FW M 3C ?EPEQ'RX$+SRI>7*@5?"$=_X[-A@.!RO==O"=B.ZKU^]KT7X$SSL]0J[ MAEVZ#F$ <8B W)L4WM**%R2Y!.%#<'J#0_P'?O7&;U'#6=Z.K6- GDO#K<]R M<6_ BUH\[T% B N]708+:"<";=PGK3JW(TPAP Q3!(2^O-U@KE?W*O"QG>?> M1VK 1H"(;7B!2&#:-1 $:VTML%I(K0T7QML UN%":AVY6=PL[W&7)3#@W>#. M@;NXD^ +PT/X"BP4[N;BA'G T%__,'&819PF?A2,!HH$ -JEKN2AM5O'Y?QR M?:&W-UO.+Y/7Z.L=$-[>>_.UKEUV+>4VDJO@1>)*=]F].>$"KE< >GL'#OWJ M?87!X.!O[>:W4ERJ+13W<#\#CJ'O;I#6Z\ORI1;;:@^^%P8L[? 79QL&2.@R M=8VT%=M7 DV!=*NC;?["@NG 9X&I;O'VX=L,O@C("4($,UR20'W6V@NM3=06 M".JWVUU?+6 82MON#15/A$VT4U^DL&?W3'P8]M="!>:\>8&];B)XWRLA^!J( M;SW&HV(9,*!VY8LNGA,S:>W$O=T\L3'7/#P:9M?6A[W%E-R_@/!7B(O=_1"W M<#D,]N)^KQ:A,(S<]1?K&]+!U5KJKCI7+=R[#1=K;8G&\MS9K5_A54'-+0D_ M;U, 4%P=[^>6T2L!B(+#V,++9BV&_S+%= > M@S7&?5Z,,/2V='MZ^:P.]\EJ(KY_7:,NTW?'J=75[#ES4<'HX$CPV\!R# MBY7%+-QZ,9,VI6O_I1&K<<_&J>&O;)&V%%M?,?'6P!NOO183 M'&:^K=L%AHA6$#RC'>BB +##D=^R,(;6.4SQ7?QZAL^WE]MG 1G@@&NTG0DK M=S^T&U]X+55@XLNA'>&6<"6X?5ZQK7MX5SPYKA6WC8VYUV*R!<.V;DST%=@: M?5NWSN/YK<-VE4LE+NI2"PRU-^/@K[B8:-RWC1H#C5>^K&!K[;N82SPM9NA> MCSN^F&-/\4KXE-L3=O%.>G>)EEY$L8?VU%NF/0F<:;^S)0,I08WV=: U%O6> M9R&UV5\ ;0MXEUM>*/?N<1<825Z,+J0V!^SV9>8*;Z?"BMNJ< 27>CLL'M?R M=Q.XQ-XR@!'723O3_>_F=0?']]_$+7-A)X#ZM>Q"#YX'!-P@KYO6/ID(%MOJ M:!_'<5L# 2:A3@O 91!K>U>ZBMU9;YSWHE#_G?7JD.T9/.39L(%W0<#;=?@J M:D>_VUJL+0O7(BS91=KN; ?(&=TL!]08<[L_YMH2?V>T=0'-[\IW(K NI@:W MAU?%DEUF+B&W:B[6TR6-&+9ZX:7!'/NK. MB'?%2N2 +Y1$&VPUC@IS;D6[Y6*H0%U78GQ&;@$G!HS$O=\Z@U^]QN34[F[W6!O-+1-H(,<.#UHF;]R M!!S#]-H@LMI8A0O*Q??V>'^V MIE\+)7 8 KF?D>>H.T8=PM;8F LPLN'AK; MD8VWH>0RP"J7#EP1:"Y4>7&\XV1H ?16#6 MD,]F$KK%"V5F,":92=LK%N(: M:L/!4.&=[CB9L[OL?=/Z@\ZVYMQ4\4*X#+PJ?NTR5#;J*62#RCI11#%+ &DV/70L8X @S,A2KL!'@$20$O;4MY89((]N0" MD6G$O-M4L/1V__M2J /<=G6Z2^/V[_-8Z5M-M@LP7;+)R%J!%C>Y1NM-=MDV M?:G&.6&',N=VK1M1'N"& 0[(*]].,#.7'6TY^&;,-&V?JPK/B<3EO.Z@>34[U]Y>_NJ&"H3 MB%N]/0$]@1O@?3L$CMA6@G?"3MJ>, !WB(P&;BD/'XJUP.'X[P8KXENND;>-REBG#QV/M+[161#M)UA/3 MC,&U55XTKOB8T$L1F!94D9?&$5O)L1W7EZM25@NWE/$"69,C;63YL$P+A@JD M@$O+%5L-P[/@8_R>?>FR M1_N72C:W \%OP<>88>DL\UNV5 M"4B1B&)MLEI9'[Q_ "'/:S&T!5X),\66CTQ:O@77A@7&#MM%LNKX0[L:UM)> MDBW+"H"'[V8XHPPH3B"W<'? YERO LY6J5RZC0_8%1BVTM^D */6XCN\A0-, MA-^]6^5<\8UY71U'@H_(C@$^[]MX7IOE-=CV MF(U3Z^.C\(MW0.PVUM%JCM$#?%MZ>>>D&UB=QGKP/XSS(QMO5J;$/&\]PJ">IV_^"LQ04S MDOV^#&"UL!XH5CNB90(P 6JVDUL \B;WBQQR*!%(>L-"E>&EL?"X/BM?'NZ: ME'?'K-\)<_]VI-PVS@L3?E\%EI' ^F&P0# N+Z+)RX>"RBG1-S M;>O$BE[+;F*7B)L!HK) UL^L,PVA#O/544= M?J'%Y&;+;>N8V!P$<#6';)FTEU^5+[56,K!M)A5';-'%XN.C\57A#2Q67MT7%PW,J$7LQ QT-Y^ MOZ M?(^YFN-S,X6 >3MBEA?3;".V7-RH[1G@JLM.1O_"?H.X7%MNLSQY_6M&EB_K M=(VYXMIU#A.+!U5W8> !\D6@+"S!Y1BS M:\G.6>=%,AI7I4Q'OAK;D1?)QU_[KJCXN;MM/B$?<3FYJ^2-TMR!MD: MBH\!Z[W69CALKGO#R M&PZ_"%U_,*C8#MQYQO;Z:ML#!P+)\E9Y6,RNY2A/"]3&=%]TLZ'61IQE%M>" M?6//66,/5"RC2F*R9=6&D*FU9X/.<)B79NM]9@6#FOW"&^$F;DWXS3PQJ"N+ M:SVYZQ NHL7SY?ALU6 3?@H6Z+5VH\&CWF9P"W@ ; M>,_/J&"Z\%] 3[!H-N92>OT*^XKY, FX@LR=+0%X9Z,"4@*0P 9YU;!$QOJ< M9]VS3X J@'KWA@N"AD'O$EBUF&*E,!IWH:NH(.TWEKB<82""%T@'N.F M!;*X'%N( F97&:SH'2C;)^V\'%RJ[;;6,YP&:!_3AW>\5>8#+1P7!2#'I>0* M;SN\)-XJ<417B_!_)EXHDU>U[MDD@!,@G^PZ7C7WB)_1(-K1,QI7#@V#[@## M:3//X(7-L[6@^1RLQ3R#GN/$I&1=K4(8B7MB=AD3AO_!A078[T]W%!T8CAVG MB%?.EN11L'?Y"7T<-M>F :[!&X?YL!%WQ+MK7A7?F6_&]]K^\SGY^HL"L!K0 M 6J\>6CZ4Y-.RV+-S#Q24Z<)N_WEJV[K>7&QPM+@WD M!"#$$F(/[;QXA@M>MN$B<@_(:&/?+;O79PQ?*-#VEG^^,^G.[Y/60$!@-OJ> M=./'.^D:],BXAUN$ #)?>3W"?N*Q7&38->+:!62QLVOD%W M7J@2<$=ESY+*8V>(%--OD* M>$._&%I;;@MW@Q#.+1H_>4/,VEX,,'QVQWM@3C '#V2T.EV/\$S76TNY%=<> ME)G$Q>%FKMW8V*&,/L7UC7[*'C-3NC1L5=Z(G 3CM?>1R;/W,NUM?,OKV7%"#5C('G&73YF3.L0R7,QT$T ^\ M 6J\K=Z*@D*9G(RO73?3E=.Y@65E,#.9#$"4_O@X<*W/X5J0,H]Z!&TI;NL^ ME[FVO^*S,$XXI)R[C2F;==&\HE^;@AP!- \V!]D"O ME">_^F25M(SZ)8VO!4RG /ZV@N%W A[ 6GP(7KDR;'/1ZMV ,H\Z!.TZEB@ MH\FX6=RM\THY,2R?)OKVAC')[F,XL!;!RRMNWM9^?-.[XN66\@*#*7FD#4;/ M:$?/EUTO]CH,H?7IH!]CN2R MHUG"0Q@!M*)V'JV2=@ W@2_5^.A7M4AW)LR/5@OKA9/-9&IYM('74#VV:M;N M?^VUN8)V\[($41R/GD>;F1750-N6;:HX7#V[]>$B$+ZW,UKP\@M:+6R%[E3[ MFYF_9%WEPJ ':_$@*P-&9Y54NQ70.4 7I/75KNSLYW9).U@'N?^%$:\(N9?+O1&.;O_+0W@ 9*X+=OWKPT:4L EAAO[ M<4T3MMN+]!V7#+#*34J'HWD"K]_4LVBW7;#\[367F^6Z1^BI<_\8\"RT-BQS MG;&[/NIOM5CZ0.NE]N,V_PC-0FB1K6JB@>' )2HGHX?"-60^=+;6#WT&"-;* M>LW3E&) =(_:,5 DX/E&;!W$1MN) 6\7#)VQ=OS"H:NW=&MUK[6:\.M ;)W$U.=.KK\WUYL4#M9J MC;]$U%PN;A$W!9 &MM&V%([6=^2<=60:+*"@_?2.:!W4(MI)M-;ZMNLDOBEO MJ$\#3&N#=3*7U(RRGEK?GIV^JUU<8MY9Y;PB+DGG;+O.AVFG]7PY'2VDYNS& MF"6V68/1-2 I=]W%W5W[:J< ,=T\P2)9G*P6/DR;:^/3XX,<]/VW+'R_'2)G M<3&T6&@Y $?@78M[SBE[J4?,\(42\W+W6HQ 8/8"K^/5(MKK+;"Y?R/-%R8/E+*: !V]4=/6 M4D$ %V@W-0I "A $< (L ;BX,MO$+2&W4$SIS725;*F][F!6] 49%ETFP"N M>FO1YEDR-:1ZO%R?73?K@]75==]P=:CWU5)^+E<7E8?'^&)[- P:0^NJUAJ7 M)F:^#P)$],=ZW5SUK=;6CW.\H^IW+61Z=.NKK1=GECFVC>>1]$X913NH_C^7 M"NJY9&HB]K+$)O#(?@FLJ=O4RP7[;!) "U $X.(2"P"X4VS>=0\7K3'S'6/[ MI-\#GV\7UO8M*!8!QUOYECKC@79J^(_=@LWD?VCS0,#C2/-J!'4;0C8TJS/;?56 M :8(@.62[R2W^"P>A%2EH*_9 %H,K9*9D$NRA@"K>-^Z!&KB]?WV#(SZ%=<6 M>K>T@FFC<_R8A7W;3?S.D/>_VF<%]4P70SO7+09_@;/3M&)5<&P9%GPD_B,# MHO.^ZUZCLP)@=8RQEAS3AIV^3F*D+9/6*Q 9YOUFI.O7^.HM<]NWP%LAC@5G M.0+:#>9W-C69>WMF@1J#LUW7L&ER=O_81"V[AB8;"!C M-YE\V&XF.QY;A.W M:FO997,_3:0TO17MZR MBX7"C-H>]GVZ2:W_U5?W;N>_JU^G+Y78E]M5,-7>F0O"$0JH\?C _\O;K1<7 ML)NW_%VD,S'89=RLAMU>IU43H@*3]NNWHZV6+M5RB4@L(HZ%)QHGOH& M:R.VS]J/,D:;[;^V0PTL9!7#23CW7M4_"^=_F[XMY;ML4QA'; M=^?)%5N4PKO6^ O?K>ERA5?!5V)#K939#+TE-EPC(R M^UIUR6MKXZ2;1]GOW>BLNSN+:H6N[R6T#[T&99AR?+O!FI)73W *[,!Z@J]Q\O@@K M!N*U/NP4KX6:UWM7OD3?F(VYWEI]M60W,ZP7%F[O9[T#&X2J]9.X/@!)+C#' MJ.G0\]P&P9@Z41R01A$,I$,$MNL\SCW;LU^ZK&=8KZ'6F3R[AM_"G*E%R^1* D)YH+V=+E-+ MH"U;YQ'_P)1 6W)SF1# :0 6-HG0%ACVL#%%0-#;.?-YV0 KLRV4"RXA=1. M;7_"1&2G]"&Z<;UHWM*ZJO_-\6FQ;4'X@;&]UEF/D6G8>EQ%MH$W*-R\=N+V MK#FTTUO@,P&X2V $X/SJ<$78T.$\,REZ=*UOV#C JU6Z)6-T;A:W(4W3KC(+ MBV?,*F5.]$PW:6HI0+3H4'T\+@(8 M;F7!,PA*-Q);0YS)UE/SJ3O0'^A)-="X2.UM8=@R=*_/KX8_BT2FTDWJ;D#? M;%\-0[=9]ZC;LM6 1@T3&V0%!H96=Q([DPWB+A3_HG'266Z*;]QV!8WHWE%S M=J_;3=YQ+FZZH[TZGDGC 4[>U_^"=0I@7J'U;OMFZ MH'G4PFO;<7,[X-QDGOHR?(/!&&O2+U585@SN)EJ?I>G3?^K$+\::9;S$!3#7 M:Y',]]K7KD2;JTR#+A_?A(G),F[[]EAY#&S@10)LD96WP%W;=G[ZLD3-C1/G MD0^T< >\.OWX:T4ON-:;"7,=^+S B6Y]3SKA0(?=R>U[N*=-Q$8PAP:KM@" MD*$*76!\+9SXWARII=6BC8G34(%K<"RCY-V8CB+0BSN["=SF]LY7[$UQ=MK* MI.W"BFF;])4ZQ7W2U4G;CI_.2N&G H4 $%U"MD'CI3_#_N*];B!7/NO7Y3A3 M"/X"(=M3= 29VEL#,-QN>EW1G=ZQ0!GX=3#[]57):=6S48,Z[8B7'2&B/?7N MIL4"B&) +8$V!4"HG2(8:A&U]=FX NO[C"R@==1B:NM*#-L);84V'*T$P-<. M 32T'-IC[4W8!##/_3JXA)?8.%IJ;>^;21L/SA/T:'^TAMJ$[SPWXD1IYM)Z M:C'-20!-LW; GXOR[40;FU&U_@H0L/E9O(SGM02+OD'.#^OA]'%W=?VM3M?Z M#(C(3>A];T< *C!CCM'RSQ3W/O#&VAV$&\& W(ZTFKA$W:G/37>12[=@ *0R%U@K+ MC9FWD^W.[N\6T1W=G?6>D)&\^MP)C=["+"W05_'0.1 /["%%M$. M :(($@/V[ W765 MH"EHF76YLV, KHX:GAQ#SB$KNA.W$'#]MWY@2YOOY3QG MD8FVUVJ;];PV&;UG9M=*M8O _MEGP4O7$TSS!H!+>G5[(MKL=^]ZVSOXABK+ M 0"X0]LB,?(W46"G%2LKO7^V\6$6^+>V(F"[3FY1*"J\G5IJK==X+8U'3M1* MJ8_!(>ZY,> VV]S%17,/OGL"0^C*<(<;CNTZ[BT'AWL"YNLQ[PA\Q[N-QE43 M?7?5P8,>\XI*YEP,?B)C"> %PQP+1;V_LX/E_'L^.ZE>CY\W)WL5LBY@I# MNZ'-\]E6[^K;CDPA^-G*YKZ^?5Y N"8<#SXRMGLW%YP"".V@,$,;Y5RM/6NO MCBOA$6(B[B7\?USEU6[C <"YF>J,-;E99CL91M6BIQBV# %R\Q[\7AO&Y24S MDC_A3/"O0!6X(^!+0 K3LL36&/P)KX:3KA<8"H C[0R@!A[CO=A: MJ^.F^==4;X',ZN%4[?<;N\:NYV;SB:S"O/]A_[ MB6_9[)+/P(LD(4YZEC^C?XG7/V8#WD7G.';Z/.1VB[-\K;*2RQ!M,. MDCNWV-TU--KZ]&PX#DTJP.VXLN8.=A;9C"SR5MR: MBA&^3N#50M7ZR5L1H!:@@9_.OH&P[I2WI=P'TFPYG!'',N"V0Z(V7NMKYM=F MPIG4=NU=,E08$*Z^=E0?:4/(<^WI<&2 ._!D/N7VPENXZ5MRLOF8C)#8?22W>+&99\H%V=ER']BR#:!&Y1MP4."5W(I"LK0$$@3/$S5(I>$YY7YR%?C/3 MKLW(65OVK4NXKW!\+6!\1-S"E49[J0O:;5^&+J>8?$O,5=[^I@O,;61?;F,;7YTX=@.P MI,_/5>S)-PK !F#YMB!S>C'()(%',.<;C#WJ9=6F!V:UVV)3<@E\:/M3*.I6 M"<;!!6=Z-C.7WSPP!N&^>]6YSX.D=XZ9DHO7O@!GMC7 6-K([2\;ZWN-9HG# MD!W-SP()N6_P'?#H=\$\"=\QJQD'F:+F;G)S.C1\]F@2) )9T-O MD5>^&_(>;OB[M3L3%DZW!\X 3EHC+BW71,O+[>3Z#.RT#&,B\ 58PPLEQH@? MMU&X">-[\228%*PC5G?KL+_5Y-N++C=[#V3G-1H/>BNV26]PK_67>ULE<> V M ?S,[>QJ[6N7[D%H/[@5X; !J=MNV M?C?&SUW\K ^[#7 -KMT"F;/@LN=<\:I67+OV+04GK%$ 96N50)) )> ._^5Z M:RGB$'&+P$3X HM0K11Y(O#C;48;J,VU88"G=;W:,.#YG3-_ MM=VT#5RFN (\&NZBYC*/:)T 4X!%+:1V","HA2AV18)][3B7)PT:;X)?N#W1K6PV]J;6^GUI;@%O&PRTQ&S*],LA!4U4+DDG MP0^TKF]J;9N M!N;%E^5ZN4:< _[]YGK,"ABV50 G[=-:EXO^=@)@ ;:TSG!H.&=Y],S.)@WK M:%'#CG 6\>?V,PT(9CR+?'/*F.7];Y_9^QWB%N#>O@''L][C0<@64MLQI_@" ML$6YD>TG]"-<_4P55EL'FW_ :]_#,#.73SS_9D&7=?>[U>T"0::+.XA_+E+VL9]7&P%$$*H$ZK<&8>JUDSA+0 MFFD MN;GLXV@_TU8WBELI#FVKF+F[/:VY"L&S_SBA 7?1.2"\TYZ^NVKT@;/ M?UVZCW(X,B7:EAT<-A)WQ#_%"($<.!(7'=Y.[E@GO9FX^V%ZN,4664R&WG+GM)&X5V&8 $L\*)SQ7IO;J^&WX>$:MX$@_#RR?4*D M@Z/FA?+Z+ZMVA:TB;@0GC<7-".?4L?X6#8S 1D1;SAOGMFR,^;6<>BM('MB2 MJL7 86H1*:(XY3S,M6GW &"_A?(O0!L@#_ %."[L;KFXC&H&;EG(GHO_1I>? M;\6U$H'E;(09!4#\%M 2 7H)K@7O]\0:]4WM)2K#GO/?\/+]=\<:KZQ03C,+ MG",#C6GH[42 ']RV/N7JGRVT]FRH<",X=]YL%I/3?+&^YQ$'K@XZM-VG1103 MRJ?F0(0)>NA60+[RO>B:Q=L%(G3:<2-X'*XZ9_">=<6ZQ]U&L82\+>X[0&:+ M;1714P7#X>XJ7-? J-V/]S+5><& MSB?'=/'^L&1X/FL\AHR#:(W'=G2&;BT\0\PLI>:.C#W3XG"5P-:VINO,UAV' M>?_/V.('NA& 4\L0( %CFM/FA&Q.,PK@"# B=QH'H%NV[?+1\[N\4"LO1Q?0 MRPG#!?3?[S=\"SRY;IEWD;/.C%MG=EY;B<[MYNGV=C/GT6<4L44@A[XU9BL4 M>?G5KEKW+OCYH2VVW1'+E=W>^ML"+=56@#N6[MI>CI?06&XT\,C\GXQMIN0: MO '+[VY5..>VNQN [NNJ>)T%CV^!M?-7@1NQ+0?/:/7=!.*W\E^8%QRRK@/3 M%&+&/>2C;[?X<9P,SO*JE">X$?$6 "A7R&M.MYY+QU'3=V;[]=(V5[#RY>+J M;8D-CUL4P"I78GRX-=S&U"?**(!@+9-6IYY33P&D9W'J\^A5,^$6-(#J9=FR M:F, A6+!N$R86IL_IVM#>5>U?UMS-4U=CFXO-Q77>F?:'FK.=1):5$X0WP(W MQM/0C_&<=,%R\_TLD 14]H&Q%=P9[ M%<;!K>+O0!9]2[M%3PJXC$_4HO&5N>Y0IY5+XJEI&3 MOF'84.LG[C=<=/X&OJ2CSLGAJFV;[6/;U6M7!R-GD8D%M///[1KW=GXT1JO# MBD?7I8E$\/G9W/QP9NAB>&>ZWN>\KE; "/!&+P4[;CVTBG7E[=#VUW'!+5L*NDW8@4](YQ*+R:O;3/:WG4^NB9]K8O)#:37 MMUWDX&8F[B%=.$P[CM>2B/$ 6%^[@R2:'7ZN+4*S;B?I)?87.S@\R[M'SZ1? MRT_3<&'O9XWA;G2I>2D8O4N]UK#?<=NY97%5.AY=2ISL7BX;OG_>K.NI@C@]/CT_CP(3 MOA6X@^,5-+G9-NQ%QA-XTN'(+W,E-&-<00M1CQ4/&#C8\N,,.I>XS([.S?@V MSMW&CN/GMB%9G:L+]P8[UY_#76(&LI/Y2ALD_P)7S3G)[_04>#R=SSU<";1' MT)?F07:U>)>[PPXM:)]/MF_M%%^J+<_<+;ZE]2G,Q;O()^__;C'X:XO=12G$ MGW7A?EPUQ^$7VNYB[A+/=[',7NTE=$T7#\ UW]*.VOVXCZ$5^C3=2:"NC;&; MEF?LQ^ :NE,9AAPBX.(6;XG@X/ Q;[97>DU.7Q],N#VT[7*\N>_WPYZ,9G9W M;HO!%',%[L&76%V?YOL6TQVW8O8ML+I6C=+G@)',S\8E:#$"S%A5D MA=?$/_ T+J16,M!N=QQSSY':Q^"=;LN6;>TR!G-WD4>["?*8[O2\>MT3H%HG MS#'NM>6#M4W=0\L0#R\;@E&UHX+3\,*]?7[#G18?M'GKR.(Y=PO7T7O035#O MK&_0?N5@^S_<0@QS_X1;; _E?.Z,PX3[PDZ]?1ZT];A+ MVPW#V]P9L[6==UP<-D;/I^G)+G=J^K&X),X*5DZ' =#HI8GY,'K8?QP'7K>W MW.?M<( 4NIO6Q8YXK[\,UTBN+]=R.XD" -X M:">Q#J'3W^IMQFL)?N">>S-*ZX& P,3GDCWHFY'^&YM6S:G>Y3AJ=W M!"CA7]K]0\:]\PYKWN9^A./#XO(F[@VW[W[E=3*'I?W8!?9"LL]Y+7#EMB*/ MV'/:"NZ;\%F:D_LXKJO'TL'KZ&\H^+&9-$%ZW[W# =2U8/I" MW4OB4G<#]<@7TCOV=403PX7&M?<4^,^]Z[X%_KJW;/G<70=,PUY%[1/ MWK? 7X#9 0\92]MO;^O2X&L*6%HS@ U=P#X'")\?@M\-!W>C>Z_V9OX\%L#; MJUO4$/<_]40;'![G7L$OM;_156.M>SR==O[?!5[;G36ZU%Q6N_ 6*BP4#SB# MS)'K65QW=M.X[$ZQM7<7!6SHY]OPNP?JKHOY)25KCI_NB][P;1C=2TRA57OG M*^;#ZW>K.T#9\!TH=D(WH_?F-6PU-WU=@EZ&1^Z>X14#P]SP^^TZY'#IA:!/ MS?_KR^L >'7:*&R$GYR3AM_N+W)&[=R]L\W;73(X!EBX/@77K6$[R_MNG_ " MDN3M,?1\NR>>WVX@\+=C:3_A.?0O#0JZB4[^!B\'X)WMG?9/_*)9A_A=1YMO3DK/F.'PPE<_?[@9;_#P;7J!?6L= ?>\%S>U3(# MW1_'BO@3-.//PKY_IW,#V/?G?'?_^XI8IQOC-B;7DU'OCE]+/,6W]4Y(K[8G MX#GQ5V7-NH'7RPR)WJ$3LKWO,^:W^M]983X\%Y%3TUF_@?6:[V ]\@U!3D4S M;'$ [?%6]"LZ(\"VC."43A3^U0KVR=&Q]!!P:; MW>GH4.VX>K,]X:R.+[>3G3>^/N F+L56:MT3^!V8<%_O5=P0>K5;W?M\=US_ M(GE2[F0;WJRE[BT;PG?K[O:">$9W BUO-V,_ MWF7LDO=\^Y1[FG[EA0,4BGGLYO3]PX]]JNZA-;1;T*W,S/9*O-O]ZCQ%3LFG M=%'6CFB7?-[]#7PIELD[?>G C&OH.RLW"/57G]>ZI_7'U/@1K[L[Z^Z!!^ : MC WGPVE4-G,=ZIVA!L2[UD,.M62'>@?8[C#P16JCU^WM+OA\.[%@\Q[G=1)< MY3U0#XR$N".^% S^3;:?W2WH#?A;/$E^+'^2K].*C'F[_FJ M5K^]=Z6CU;/ MY'N_6N^^,]97)I"%U\L#;O'R46&A-V%Y8PN8+ZX?H9GKK_FHMP<=RGP39LDS M='?&#T3+LDOU!>[.V"&+@#]ST[P'HIS:,V^W@38.P2W?SS:-$U_Z/>7.W>/2E/9/_).!=K!0#[<_ MS(?;;?;Q.&#=G"M8[R+7Q67I.%T&>J]J7*)=FFQT;C1[W,'*<'=J^]PZ%(\YIMKF":C30FW)S^+[(I^ACS?/ MA#GJ7N3O>A^=3=RH7IAHYG?IUF]W,*:Y+_ "'@4C@5'DLG+M=V+\*HPK*3\C MW,G?K=YE>KQ<0#LOYQ+7EQWDJ7((N8=60H[1'NR>KSG><&&M[6% !BS.SNO6 MAJO3P?1#<)4$:CP>MR+O=+/>D0$ENFF9MFS%Q027D>NX1UV!>J7<-2T-?G6+ M>W_9%_(\;H;O6GKPW=+^NO=N N.C^MR3!+F%8/]S_O0E[ M,PVR=.KW:;E)O M?H7%WOIFMK ZZ@ZGW]T"U^>WS%O0N818BAL82$DCU_.S[.6AN\"X6@ELSCY"U^G5S\GP[\#S6['[?U67;O[#5O/ MZXG!.MPGO'->) ZJMWF7AEFBZP7;:?IC?4\W1MNMCXETWB/0ML MQ\'CH-V7KC5]=9PGP- 3QP'B@F3#MJS83OBJ]ZK3=._SQ_9N\D=^R-[E!M"S MW?/IE'.R/$K^@XZ6MT&[Y%?P,?EH-<6^@QN7Y_G*B@?VV/6T_2,W-7R2[HB7 M=*/CS^+M/!,ZMPX25]>OY1/O3F/M->-]5JO#1<^[X,$(98 O .&^6DR5C_Q: MY7?L13ZW)7G^PAY'!]$KV]'/*_I5^B:]^9MD-[-O;/_HW $G._P6RNY[;Y)/ MV6GT\W#) -;W58%688ECN[OL7NAT\5@WS Z#[MQCWT_J:79B, OWDYZE?K,W M%PCRE5Z#_)$V!Y"0WV*G:?T#I_:'/&J=U;RMQ_R.GO'LB^ZW/'@!OK!=L-#V M<,U3QO:I^1/>*T]DS]POVL'VN'9"NMP]D0ZFK=57VJ_>Z6;:KJ:]&2VE7L5/ MC:^^T5YVA(4\,3Z(7QV7XQG'\5ICN.;V;@^#[MFOU?>SJWK8[]Z7?QM-WVDC MWEG2S?L,<7P]9V\W5O?N;\';6=[SP ?^^&XD)MI3=XWV#>XD\WA=68_#95'_ MQL/'QG+$??\8K@P:)X#7!60%B6#4\-]^=%TJR"]OYDO86_O_.U@>@XZ+'] _ MVL.\XWNW]XG>:)QMM];O:0?$E-N/>81W+E]A/SBWSX'697D.O-@6A*ZVQZ<_ MO5_OU&[P/6O8)V]"1V:CAN?"A&V<,/P]+W\&G\MG@YV^PGD2^YM>7'MB)UJO MEUWORF@W+;9>WKZXM]4:[F/H;7PZ@.(>9)_'IZ@[A35"CLI79D/QQL=I]W+R&OX G MI=')*G 4)A"W#G &6!:3BO&\\&/P=KN7PZ[B/=HGZH.[W_$&LXF;OZXS.YW[]7 M[B/Q9'O1O := $^ZI[9_E _T;G;W/9TYD;[L;N"+E*GB8>I\!DO\_.RUYX*7 MYDVU].2K,B:Y-3^WI^[.O!W-RGJ[.]'7U[S;;7,;JQFZ5GQ0^R$XY PEKJ[ M[Y'"8ML^MG<>*?\]C]UNSA_T2&TX?L)>CK]D4-YFWAGL95T8L[F61__C]="RTM<6W?P(^K/:H+C;C%8F,\M,T>T-RL[PH7P[6]:7P&+D:?:0UC?\I' MWA7W&_TO@-UV0ZUY#^DK;\'GD/N1;6\D>\^9A_%ZYD'R&6DHLA.88:]))^FF MZ-W"F?N;N!1?2PNCS\1_>6_\XT>#1PD#D+MZ)WV1W8T+I"^1%\.1\%[ M>JK?#('*-Z;YB"ZDUZ3+>E>]E&+O/8#6J'^SA]OO\8/XB-S[>$9="T\G)E]; MWTO[G'O9O>C>0JM)B*MH_PK@TE<39_/C<0J U6N$#MV3 M!^K;N.9J[G(?M$S<3^VZ9Y7#$UY]P]A]_[ZZ_>MS[3W'56.#MY^^BW ^+@,$ MA9G>ZWRD=417A1O6=>XWF)_XQOG5/&/[M^:C=[N\U5O!]XC7?=7>$8KY<+R75*P$_@CC MK".Z.P7?@"I]/Y^_U^VG=D6[O]O$;A[8KXPZ+@NO[9VVL/P\>2V_E[ 6K^,* M>:_K35P,[237'$_L/1XH<,.V(?YT>'7\+""KMN!#R?.\6&T/NW3<8*S+U]=F MRX7F3^X/K0M@Z PV1FI'NBF??H6]HND6V+" +A,TH(D 4X A )+[2? J2)^\ M*DC?%VB!10.:"I %@ )XNM7P=-Z-_*;>F$NUC3_CR>'R/H.7-I Z\9OT5@RT MF '*U>/9-#V;5)T&Q]Q7]\_B(^UA1_#^BAVF?:E7?=WCF&\,,KAE&,*F[2!' MY,??(%KP\GR?=CPM;N]W!XB^QW A]>YVMFMS[V.3C?\"E^"U@#.8& Y&-VJ/ MT=?T,]U0+^ 40J^S9OHF:FL*5 ' \BY_]&NR]_%FD5G:3H$4-LK;5;WQ!@5; MP7T)$7TWCO9B?8ZT1_2R,Z@YXBP(G^9;^H^[>8)BOPEZ2F 9]@B M 9CG'W=:.[-@.V"OK24;CQG[TG!(K:/6W#Q>-UL# MM]VT77)M\:0W42L$>$*_$SKCAG!.=5-D9;UTM8'D<' MJR&U1( K=\XV6HP:COD"#FTW!0G(//N28I;_8?5XO MYP^Y-=SO+Z4X&LR'/^5:^M6\2.%N.6%_[YR?#I4?I=6Y'MS?+FP[;[X9+B1_ M!7C E'$@==VV^1OUABE GV_+2WR O^=>*"S -=5#;UW5H^R'?[&9MYO:G?7K MPPO#Q%O;_'O:F]G$\O1_NO_.7OG/BR^A]YHJ_FWKMC]\7S?2"6./T\G-T2Y_'S MULWJVEZ%PJA_QWN_'RVG>'G@NO/SKFH>*E#O3?'6F;'/\>6'N(DVI XI#_T.CM?@YV^4-JR9[#YSIZ=' MH??^7^"Q%2RWR?[/_@?^&O M:/OOEWMZLC;Y*W[G'XO[R./_!7W%?(U].'S\8;!?II@*%V3?<)D&F%Y M<8!=\':6?6=;Z +:6GUPN'#B6RUGY7%7;G!X2VOW6[==I'-27+YY9&'Z>FAP M5G1*6X-JX&B+<;=UDW[R7-MAR7YH=+,"\V$7 :!FC'Y=8E%>P77$%F(D=(1=AW]!:O]@@V4V=^)SB6!_88=D07EL8FY> M0GC2:O59T'_[>:UD=5_O!)Q<1F,Y=PMR&F.[9<==,WT+= I<&%I@<')R;6N!7@%Y M9P!Y "T B'MB=$-EAQ M9&I' +)VO%],6H!IAFM:>QY:K8 5;;5RFGJE@"-JU%J&:B-MJFJ? @=AT6L' M:]X%,%_@92]T9E\S=JV = #)?C* ,P)+ $)W=7T%=W9,I^@W=$6SE>*7J.?X5J)FL>7SUFV6DN81E],W;= M!LN AF0U>'-_T6H>6C5>+6#2:\. RU\Q;?& !@)? /F *0!*"#X*, %^<>" MV8#]@&-T:'6F8'( 7P"<=U-=JWW(6B1]=0&? 9MD^P?N67EL!&I#8"5I^X!^ M;=N K%H7@>1?MV#7@.& AWK[88)M=WXV<UV->$QK,729 =," M\(!37UQ6AY:^X"=79EW$(%Z71XEH;@;1K)8'!?8IG*(%&9P5KZ("]@-,7KPKB>HQW M,71/@64 ,X&96L!Z1V]2@:-[95VW8AM_>G6D70=:,F4W?MYSVE[&>1=[ M)G'&IIS\'A]94YJ<@#A:NE:AV\)8@ESEEW ?H!EIF6& K0$.VY!86=M,G+'?B9? M6&U:?_=;00"F8@UATV$:<\UW!EZ-9,Q;[6=$8]]IKE]>7M9=LV6"9U9TGW[$ M:B1BP660>LADJWX9;:MC0V$F@%]CWEHB>TEYU'2)7R)>16/&EO>8W%=\7-/6M=G$5\N Q!= M*@ O ,):#@'["M5ENW9T;TD 7P!+ $4 60!? .I='EIE?EA:GFZ.8_F!TFNK M7#PA(EH(@F-P.H$[?QQCX&V?6R@%"E]@9UYBP06&!/A:%((6@F]:+2OZ988$ M$GB 7)\6%H*8.)BS%L>6E\ <5[(8"!]<7Q57BX 3X*7>7 !>WT.9Q9P<7Q? M /. 05H>6EE<"FL6@C<)I@4,6WL "%MP :Y<0%RY5U<@KE$]5DM@MMAS 7? 54$4%J-9AY>!V5R M@A6";!L7 YAG5()==4X"5X($8%J"2F$O='-A&GWU>D]K)EIR?:]XX'@D9R@ M@&IU7UA:\5E? #MA$7(W@<%='UNK7&P;7 ]4>ZE;*7-<7"$ =(+[@>!XI%P> M6BL!M%I(7'5_DUZM7GN"+6!5 D!>$5LZ@5];5%^ 9@^"6V6*:Q%??X(Z@H6" M7X(Q 0!C88)M8"B"!VO\@5A:IEU$@K.".VZ0@F%:JUP'!VEHO@[S861DW7/X M6B$ ^%K^=LV"<5TM>'!B7P#)@$=T@&I06IAW-'C.6\Z _FD@?85BM8(46X5B M$5MA@AD'I@("$%!CY8+E8CU@O%U<7"( V6_/ T""$U_"911A07?#9V!:5 0S M=/EV-GT"7(Z"/8&,9!A?2(* :2N#)@#_=DAOGX)&?<$%"EP> M6FV"$(%7>5^"$"0S7V&"$5LM@DQ:1&43=_1?('306Y]K%X!T7.=Z4'R:6[4# M$P$A6P6DH!.?R-Z='5:985L-V%1 U4$ MU&A&7T$!PH)<@\EZ;@!Y !!' 5"#+1PA!\%W58.%;!-_^G#A9G-BBUUO M %"#6E:R @Y<8X/9@J4",U]T@FN".8+B6I""[%IR@TD"G6'@84U<"6#Y@9R# M.7A? IP<8(86I>#; MH#8E@+8+>6D^"(5MN81-_.G'G:W."IX-V@F5M'V'/ M=GZ"P&R*6QY>2 &! 5M:M(/46JZ#&F)!*"QQY8+O@ ^!96= @>R"OV#Y6B># MX'C/:,I; %RM@@=KRU^;@[R#1'!>6C$!&0JM*HQMNX-L@F8 -(-!>B!AL60P M9!#68-"6C9C1( 3;:AFI'Z&:81U3X*$@G=:_@']"G9>:(-,%STH &/= M@^:#6%HQ> ^!.0-Y7&!R)EH*;^&" H0X@CEXA6*I7GJ""8+;87!X&UP07=-F M(UR*@O)T@WV>=LB#A8(6A&@5\V'R>">")(3F@^B#QP2'9@"$&82&!*6"_@I? M&N)V1UIA@@0'\ 8D$]1IIAGV,%=?1O%&8P!3&"WV9489AU;&!::T]V M,63O7F^$:G*@>.D!HW[686QX#W4B6C5DSP,K?MAEZEY\91QO)X%]=6J#6U\< M!E5>M'(8@;=@/("Y>#]>"WLC@5UOXGPD:[%J.V[G 9)K4F)X7#I=67+H IIH M,V$66_5XDFAP9#ED W[]8\%HD6^ 4^IQ= %T@8E^L&C"8QMLRGN/?KQTG@*? M6N9RHG,47>]9(76G:U)O]] M7 9[%W;;>[I_T822]YO=7 B6E=;CVAO9?=?#&H';@!=.VXO;^=DJUS;@9UQ M*&$_; I:QF/Y9C9M-62.=8=D,G+1?$=:WHE?D)R(6+J M:3MW1%L\8[)D[%J.;CV A%U<6)_)WQ@?75<)5]W>DQKC0'T=#9; MPW& @3UY+84H !=S)%WJ84%C%0']>GM?^G_!> \"47'_7SV%QX$9=3=;,5]X M<(YNL7M9 ,M?Y67!:\5\^F$.8VAQOEH(=EQ;M&3=?_=K'VN->O6$#V&U?U!T MVUO3?F]:(P$=>U-P8W/,7(ACCW )9[1]886)6^EG,7H8-K*764@=!?O'_OA.%: M]E\FA9IM,%P[8<%_VX%<=Y: AG*6?+MI"'$\=$X / *&6MMAHEH571=@'G+ H9:6%Q)6Z)SDX+R6;]H.(-=A25?KUNN8$AA M.FJG8Q=Q[WR-;!.%(EI- )MEOGIJ8 "%M&L$8+=O9V7VA 5R_(2:>I=Q!'(^ M6P%J06')?Y6 \X(T7[M;B5O4 F< L7C"$L" '7>,6SIU2&WL>5YVZFI>>OQX MA&S>6L%HJUP>>:!D"V/OA4!>56/Z?71E3'Y:8S,C$VX46SN%(%XM6SEWTGEC M?<%X[84.>U![OEHQ?[YDHW0Q84AM3(6_72!::UH%AM%Y>'3U8W1N+'0_=.)U MFW,!"UQUW&?O!*-TVWQ_!?)V$X88 N%L+5L5?%>#9 "$8V9C3EPT7"+@8U;P7A$;%42_@'E!C,( M0VYL:D5N#6A7 $D 3 !9#4$ 4@!$ /IZ4(1B<+AG3G+V$C&Y/8DUQ%6G!>5%H!(4I7GM_ MF&U.SQTOV\*A6-P MJH4.8$Q:JUZM7EV%!6&X$N)OLX53@(!Y+P4 7:-:FFN[A3I6OW6\R% M^GVT;?YE@W><=C($UH5#:[UPZW;,9_)PW86'8%86\G9<7$V"#8;R8N.%70'F MA:F1R:'WX8OQ\4 'T8\%XZV8.>S5D3WUF;B%G M1V@B<1]K2%B(:Q9?I_](2R?N]970$S M7)IZ76B):U5\RF?)>=)ZKE^!?#I^XF3'A@]E!8<-7WYC/WAK@#][V%OY=3!\ M,V(Z7-EX(UO1;/)V X82=$9]&(:R9TM\[X8;?8]N_VFP::-=7X9R8RUB3W%6 M<$MB<@ ?6XE;!&!M;DMK/6<^:PN&RH8FAQ*!6@\,6S^'P5VC=!$*;&5,:RB& M>X,O6Q>&M(0WAUQF.8>\A=EU/(?D:ZQP(X8G>N1N(%[_?;]=%H9=I$#V%WC@$AM M,X=E=SEVFH*"6Q1;68?+=2U;)6>X=<\#TH$)7*M=R2VP7$]>6'V\AEM]1( % M?(9CZ7J(>=A=)7S& M0WD[AI9U#WBN!RVBP6JV'385O;-YJM%RTARZ&MH=1KWX82@5\BLWQ9VI4 #@]FFY7AHQ:1&5::8AUJ5_];<%=:5_L6CIJSF:2==.!S8>+=Q5;FUM) M6'@GZH:(1^"PS$A MY4WZ,7,F$ M86SK8-5>GF;%AL9CIH8W:ZV%JH:PA1IY 6'U6Z]=;]\\EO6=7AX M^VLC84A:S86>@-6&@(%V>F]G/W\[7IY@W(8#8UEDS(<@ 39>C'=<7&P Y(8Z M7^2&<7+FAN1BY841@;8NZX997:F 1G5E>/"&*UKYA4==IVU&7_2%5WLC9O"& M'WH0A]UB]&K+>?1C2&WZAM1K1W=MA5!C(WMP9#I?1VA,92=I*8<*=61X(?&8S]Y675U 9\"NH1:;QA::'(?AYA_ M*F4RB"->?6P-YN;HA^B-1KHVOV$3*'I6,1AO9Q-HSI]<(9M7W6']G[9=5V( M.H? >1EJ)'P@?-QL9 ?V>*2'MGG[6["(PH@M6U5>_H0.9FZ( (>LN MAV"&WVH&?7J(86] =S\$PHBRB)=\NG2_AQ2&L(&^'1716YZ(?HC/B-AG5'"N8]UT-WF$7[)\6U]$A^-K>X9]B,Z(K%JZ?AJ&TXA, MA]6(9(>GAR!=07)::F1]+GU@6\V(88>[AR%;WEKG9[=O&X5(;5!L4(C6>4EU MS%J"AUUG07)\B#.'3W@U@8N'2VS8B(Z'6UR\:QUU-V:3AP->-F:9APMZ!7.< MA])P7V!\78Q_KW4;?WM[[HC]8YYL1(=X?/^%^0PGAB:)%UHKAK*'%G#^A_UQ MO5R$>&U=\HBL6HEP$H9"B9UYK%JT7$%;3@/5AVEC!GMFB#J'X78;732&X5M& M9-"'@F=/8D>):8=[?V!:KFI<>B=L;ES(ASYN7'ZX3KTJMA84)T1NO&XH %"& M4H;CAU:&NX=57NENSGF17*]?X79X<*E=86@>AP9F<83X6[6'_7$W9WR(/FH% MAGMCY&O,=D)Y0G^W>P6'LG?\B%E>&5S+8/!VSH9$>P>$"GI:?<5^8' -<25? MWW]%!XEVWB:&B&SWDP:&!M>'![?WA>HH>.EOBX8!:HV& M<&V:B6Q=?&3)6[M?E(;RA%\%\826A6%AFX:49)V&.'-&?$^#/WR^AR6)P(?= M?5*).E[$ASR&QX<$B1Q:!6%U7BEH>77GB.,%.(:2A2V&MH?6A^]9J8$\=P%K M2HF1;@\C(UOOB<>)X(<::..'9WZBA7)Z:X A8!%S"P+ A0]JZ'.C6I$!BX76 M>W)Z &VT>$9S)W'H9^F$KVF$IET$;&!\0X= B%UO=H4?6LYL9(;^;=-M M*' W:2]]N'(I8%=:RV$5>'J(CF87 7MI\UDM8')JPF_QANM]_5L?7>I]ZVKQ M@#IB)'1+;^1Q0W@_?I-ZHUW7B9>&AH&4A\=F/7;K8YR%=EIM>H]NQ'^L6@1T M7F0[9*1J3WP8A0!R1FLG6@AFGG\O7>QV+76V>D%R_==Y''77^Q:3G3I;S-O MHUW\9D)WN^;*I\?FR&AZ5CH8B[7EB'0HG; M>^AX*&+QB8EN(E]+=HV!%&I'ARYSQV9A&D!9=<& (/=HQ: M,XH;6@!OAW%>9*D#N&(I<_)F-F1)Z-M(63H8T=O MEW$L>2H!^&0"6W5^MP&S9:!T763M6SIR,F!1?,N&%%HMBH-O)W5Y9K&*"',% M8)&(Q7I1;6AO-FQS9BMU2X'19L=C,XK%8XE:)P$B@*MC7'!#<#%D MMWWC<%MX7EKW6_IJN6/3!*QZ*6,G"IQPO%VTO**R88[?*%KM@'I 9!GWG!JA6EA^VMZ76@5 &-SA6]ML0P$!V^%SGHABPJ! M<&Z-A5Z AHK[@_M\N89 9--H8%JU:@1W_V'=ALYL@UV26^"%_HE0$F :QHG?AX6&KF;<7:=V;(E[,V,! M6XN[;E9^* !>BTP %SL>>EEZ(:\:\QU#&PSBT=:SGG&>/9B EJJ=$\!4V,Q M6M9EZ8 Q<7B'-&6RAB>+ZHA4;5V%YF9<8MJ%:EM5BVE?%XELB;\:^!S:3UR+ M_1U7P6#)EJR M6Q>'?F--7RJ+@XOW6B]^=EZ.;G6+ 6[7;DB(BHM=87J'2XE\APV&Y(849:=G MYV(#B(=R;8@1X,F;=B/9?&&.?8W0.%EX]<(APMH*#?O-A(0<)B)U>0&Y> M;'F!4P)[@7B'K5Z;B"*+[FY&;8&+"&U/BSZ+.%Q BSY^.0$%BUEDG'9'B\5^ MJGYV7N=G]UN]B71C]8:YAN1W*W>TB]N%9F'=7(R+_HDO5YB!<8E-AI*+R8E( M<(AX#EMMBU:((X!87<2+L66W@79>H(MK>DMCNUVQ>_M>SENEBZ-B#VW.=U)O MYF4Q:%UD>W_VAI)I\6>(BX*+9X X>=6!T%\L=5"$76AWA]^+BVSVB/.+:8@@ MC/)P5HN,B^2&T7V8BT!E!HS,=>!?BWVL7/%^WE\;?Q9@R(NN9\R+D&:P7T4 M]HOA?P6'17E(;SN+8H@9BK"%46<8C#YP;WJ%>,YO %JF<,1J#G\.7/**/(@X MBBM@W&(I8<)AQ&>/;BV'W'#4:R-I7FIG@#Q=7 "5!8]UHHBD!5-?,F(9;0.) MN(N?B+R%1&DN?_^(O(7T@AV)M&.Q?P%GA8I<6X>*@X%&BEU=R07*@9AQ,'!Y M7LEN!V137B-MLEY%I=&&=< $T"(&E*?^%<6V63?D1=87_/==-B M@'%38$IR;W !:BYJ%W__:?]ZTW63=K%L6&6CBA.+=@%R<-Q[*&_76@H /5MB M>01X,V*:=4AA98 F>;Z) 7^T8[QRRGSP6J:,\EN[B%YD679*?P-^;&*M7F8! MF7#(=CJ%I6-/>'-N3XG APY[]WPE=L1>3@)X CV&OHP371I%US. 4YABA?F8RP7WM_=5NH:0EE\UW/BI11KE&EMANV,VHP]>Y%@$ 5LC,9_DUQ;J5R_X%2:[2%-G .6E, >0 >A7,"]8&( M7-MTIG(K8'* F'+P6^)LVG*?9B* \&?O>TMR)'&,;]]HZ'T87":+TWGF?R]7 MSX:89/R'3HR1B5MXP8?UA^9,?'M\V67C9V)#'H588X!XV!:AL%A27#;A5.*^W>0>/1N M=E^WB@UZ!7#P'3!EOXJ1^9O=EJF;2]\RXE(+:&GK8?%D M'GU)@1%[?&Z5;+=;FHVC:)R-0G+.C:MPH8U_C3.-TX.+C2%C#HM8Q;[5S- M>CUXO&MXB#J,5&)_<_%G^VC#<=M;VWRO97Q]4'HM6U=S&6'3?.,%V'^K8K9[ MPFD^ SF-#XY$9*%H/HTJ:D&-HHUD@;=Y[6=H=-=P^F5+@5ECQFOW66A@VV$8 MCB* B5\?C$:,[0&.@$M_RER1@%Y;66\V7RM?PEO^:6MM_(P_>+]V]EE0 $T M5H8E:6=?\ 557LYY"8."<%=UI5VU9.-L8'HKB"V-_FT(<"]MH8PF9+YX?V99 M;VAT^ QA9@9\HW83B=8#KU^T;0)LUG])6XR.PFG;?[-Z_6/T;A.-;E_A?]1N M[8"AA4V)28M,B"B-UF\(>9\"B5KK6?1M?G;-AR9 )WU]7.!EZ8VJ8EAHK7]J M9C1?18W,="&-M&/0C:1@OF 1A5AEFW;>< U_G&M.*%5[3@+?7ZU>284O?&IX MB%R]:6>"6F*-LVGEC8EEB'>(;L..HWA5;_1S4F E7!%GI(H2@5.#L']< M<,!SO'9R:]-XD7P?>(![O('(< !\1%^-9K%WH7%L8%EO_H:O=V%<7'"570-Q M5VWCC"F/T7H%CY-?;'I0;E]C_6N>BY]Z75NNC3N UGDF!#&+,W)A=:E^Y<V%: ME6^:7HA^"HA[+;=P#8CU<99[Z8>Q>UY_L'EF;K-D'8J"B_&+F85/>DI@+WKR M6_I=]5N,@]MY\W3N<]=@F(&P7!^%Y&;*:(A<&(?):QJ/2G"$B3-APH6' =1C M"86"B]%Q?&V8CD.- 6-87U6/LUZUBRQG;V/;C3U?_F-E>D5_UGG-=0F,QP5Y M>OA\$H='=K-T%U]K H-@1&@V;>&$WEXV:H.+PX5+>7)PK(OHC?*+!8^*9O%;.0&GC]^ #WQ9 M?5MW7F+\B/6(NV*AA-B(0(?,97UR2G__6[1S3X &7G*'?([@?.YSS&?48#5= MP%QGCRUQMGFX?A)@V8\O9\!R#P<#=$)<-XB+>J9ENH%'=BMF*8J6>BENHW;M M:@U<2XO.8^F'=(,EUP M@&*M; "0+7#>6LY@:77@?\UG'WS9=>):1&P!B4-^M&/?7*%BAFN:>N=[1&0T M<>!GJURR7CB0]8&48H!P/)"$7ZYLVF"7:10"O'$YC(]C5' 2C@9MR([:A+Y@ MG'5O@<*$['&A=<6$:5KB@[-H;5J-B)YOJUIS6J%O=EJD;_59)&"G;\" JF\: MCZUOS%NP;V=D$5J+6FR06UH'6[QG7XTM6B]:6@186ZY_B(*K6LMCBV6C9\V+ M!6$2&=5O"W%Z6MAOVF^A6_I9T@7UC@<$WXVO?2)_%F-0BJIBR5T"=H=EW0'! M?(UR"9"[BF9AXXK4:29_F8\LBW-X?WM\9W-PWV^ AF!P15Z3C>):LW!.COUC MLF^ 6DD 9&_Y9L6+]XA8 BP!WV,?6FU<4G2S6F5Y\6V06CU=/&##AAI;1%V6 M<2M?J)#,=$Q:JY"*6OEIJVHS7P-\AXSJ7):0\FYL8F&+O8%3@%AR&%LJ =\! M:'#ECON.M&/VAT8"F7A;O>3-\%@'W8SUR/@7% Y*, 6HK7=)IP&+&+Z #0(, M6[H]@8"I7=YK;HL&<7]<66O9<#9MSUJ?CQB1)&4_;U2':(JU?_N#97E09MA_ M^W@[>_!YSV=A;J!@U<:B)FF+'H1. ]5V#81PD2./EG;I>X)FKQ;L:\M@'HBS M8[X#KU_0>0IR )&D8N: EIP8DAUEF?@#GMK!FX+<9QI=79<8 );@G#,;>E\ M18Z67#]@YEP^A4-@Y)#]C_MLWVKH?0U>V&78BGQQ%8M'6K.1WU]WAR>*IW*Z M>VI<,8+>=GIFKF.$>4>/L85P!E[G<$1[]8NO>^YPH'MC71)K5GVK?'!G?'%SCY9RQH4&7(ER577^ MD3I>&UY(8GB-@7CJ>O=OQG%6:>AO)7R^D'Q;?'%,9(=? ',O<7]M?G.O:@2+ M%GN?=EIO1GFZ==]Z5VY5?U:*<6M%;4!?[%K'8 MK,9*'9"1=Z0%292EQ/RB$ M9H*/@7%ABR@":H7D9M-BAFPC>KYD*XMY:P6(EEXL<<@$&*'X]T=QR*'(J%;CQG^U]?@\=W7%RE M 0=;+645;VB=7R=L MOP.\7VUNRGD3 4D"9 2 @LAWC)+BA,:0BUKBD+F2LFN/=:IB"FE9 S)@ V4\ MC/E:'G!4A^5B9G&_6WA\LU^;9.Z."W'EA+!]ZI*:DL20 M*F+D9&EQ"W.HC]N(^G*H7?YR)UPZDO>-I6*O8QYB*5P(<]V+9FO/B5T'\V$+ MBA-S W$16Q9S>73NEE\2Y(NCB5SBUTBD8B*(G&:=4]F M8**!G@_:^>,Q([P3>)X79Z,W5SFI#I M?H=NXEN?AX]GM5R]=4-[ZW6,@7ESE)-SAB=<38^(;MJ1=67P=DQTYXQ 7X!Y ME0&B'M_38?46A![YG7.;!-[TXJ9DPARN7.P8DE^F&$8?"Z0/F[P M=D,WU%XG>=MH,8* :$>3H8FQ+8]\C\-S!9#:789:1'AF;F]HG)/R=RU> MWHP^?M%SKF"U9$J(#U,.7$D"UW.S;6AVT&D;=4QX6FW*=Z=G,UQ';Q>/$6S2 M<>QS7' 6D)M! <;@]*3CV'YBJ&!.E]M>]F37HG2<&)S M7W02TI,RD.R,BXB" MB#&.HUNHBVP+#1T[87;8=6",70S=#5T XS;6CIVI6$%9>^2#9$4E)%< M*ES]?C=ZK)*YAN: =&Y)DXM:D'@QE%2'=EK-AK5?!)3: M88A<)UJT?(YVM9-ZB49?3G=29"AYW *TD"-[E'(R9==Z5%HA8(>3I%QW=G1\ MD%ISBJ1]9'03BM9PUW6+D'UX4'3SDFZ36'-5=,]=P 6L?^9:6G3! EUT GW1 M>YQV"(%^9E&%D6-#?,2*>'@G:GN/^5^J?>J$1'XVB:"' M4)/,?^%_A']\?G"%H7A)8@*2\W](#G-VYXPG?ZIB9WE?>(>41UPQ@E2!H%SY MA\-S8G--8KY[401U72U@TW\E:H24DVM$CZ1K(Y.YD6&%#W6)6K9TVUP2D])F M2I.D9&AF\I.9@)IE8ET+9V%;8'997JYS<@ 6Z1JAV,.8_Z/4(=??$]JQG5) M:J-=M9/QA8EE@WV]A UB(X;#E&L!N'1CA.-R"W&^=.9RA9#!=,9KRI&?8Y:' M8Y2G=V64\7*06EUZMWK2 M.GS=5/G4/@TQ:YFJO6JM:0W5Z6VB(K)$^ M>CM<-&R1:]]ZW(UB:4"5\VBXAOB4R&2X@2V3QP8T6S2 HUT.=79LU8';BDIA MG7RA@6=U=93ME>*L9.^6IV*[7KG6SUOQ:%7HC;265[6D CQN/V'F+=8-PZW/7DGYT5W1LE"9:BH9OE)=U-V:9=;>3 M- 4L?+E$5H_&7-MAG7PKDFMR@W!N "T #V%J9/& KXD ?9-P"62M8_)XM6&N MC\H"'0%095"!RVQ"7,IPQV2-B')P@'"?=P-Q X^,9PI<26>OBMAY(U\,>FU^ MP96SCQR0Y&8CB8!Y0F'SZ!J-_WU\>;O*2YI13=)&0LU]* M?SIJ*) LGA<;W/?>AE=V6FZ='R!U&E)D&-VN'6:E,%P\1U MW'4%6C-L0H)P=:Y_7(G:=1H&'I:^6JV3'H;3 ME"F1L'/R<.AU07O]8\Z4N'/N=563=%RRD"YF")1U;S-W7(["DZY_35TM;2QS MQ'9\712)35\U=O9U,F %ATMK?Y1Q;M]X5'2B8?Q];&-G>:B(W8A*:PQV7 !/ MED1=7V ;A0=\0XGW>=-F:(A[CS-\*7TEAZMD_G,V>2J2+9*'>9N4O7NCB665 M:PO88/9YWG# ME']^>'1;CZ=SW(RL:8J4]6T19W>20VNG>XETD)3@7B^ J#*I6UUV;EM@=F]Z M?WO;C$-_9'VY>S:57WA-8G]GHG1A:AD'I'?>6K.1'(@J:$)T&HN<=D^!]8XK M<," ?Y1>:-Y]_W>.EG=T!Y >=RIO>H6:> EX$P6:D>A[H8S<8H1Q1WA$?V2) MU'NGA&N*47! CK]E'X^%=91_JX0QEJZ4)8I>;K!V\XW0;;!?M7;QCSQK5W1$ M9+UVK7IR< .5PW9Z:Q0*#W\O?%R5(7J2>/5\+%RY?.)[QY8Z?,")/'P,>,^1 MW':I7]"!>9)!8G)P=(OF=DMF< *]6O=O2F:]> ICZ)$> 6U>,PAK?XQW"V6# ME@5:D'FI>.MY^W9Y7#5XIF!8 S=VKWA;8+F5!'<1@:U<1Y'XCKF4M8RX>*IW M.I9)>%5_TI'E:S&$O%M(;2%E;8GO:B4+XD%,AG8 OQJ9;-5W+5O!CBV.]I2Z M:3]X!Y8'ERAW2(PJ=X9WZ8PQA&*,F6N&6\$+,PBV'&:+/)P;>IE M\UKW;P:&M83X6D2&X8]KB3J!)8WT7&^%07QV:&!_L668:/]NYH VD2]]>7BI M90%PIFG'AN6-/963BDQK-)>F6ZDC8Y<"BI5\ZXY5?%O78LH_,F=)CE$$ MW@'>D\5;]H'06R>(T7Y$:L]=SW%NEM]#2&U :+,[:EI@9IR7JW4:;F9WUFDL8*9Q MTY!I:7F2;7F$E\YNK8LG8=618'IT@*IR8WH\B!2)V7VAAK=YD5_!=O%;Y%SS M:IV0@WWX?JMW+'WY8\67\7=A;_A>'W*7D)!:AFIHB6X CF@^=_Y[.'"?B>)G M%QB!=M-F\I:7>VN6?GQ@6I21^):1=CI\-GD:G>62X1K%T=Z%V]W !67+'BI7^IY,'B* D\&4E^L>&X M7%QQ7AZ7+Y;O>:!P%XOR>;QKG82L:F.4>H_FET=X_W#;=JYWXW'Z>!EJCX$0 MDEAC4W@A>*MF7UM>:EQ^%)%1?DQ)EFYSB5Y]$H'-<>N&'V&?<_*7?G>T:SUY M('/ED[9ES7Y\/R1S5^"?G"!#%LP 1.5L&@T7-UJ=H_G M6;&".G!^?7:7EYR+#X\%6K5L'';NAS)KQ8B7 MH54?1!]3UXG;\-D2Y/./ M7';17M6!-988>/ARY%PHB".8$6*V=_)TMGI'72-XY($YF%&/%I8I,5DOF 99^J":<='%>YX,K;)%MZ(._ TM? MAEKQ71TI&Q'>E=Z07S4E5!]J7<.=SY?-V&T ME\!X5)B<75Q^]@QO6D>1%VB8;@YX/I$%MK6Q$EYB- MVV*D9&QN1FTF>-%JA7@0@L%X>@6?;HL7Q@)()%*7MFYY75Q^/F8: MZ'A9^EJ3&ORB$\( M90)*" *99XOQ6F0"'B@XF&D6#,;(AYMHL+B;*5OI6I9O6&V8=F M:J.0FF_>F+T"F6Q' $4 ZEUX]]ZGU& M8\V1*V"4:169I%_96C>/Q(@.:U !_&"AC."%Z0$>7K*4BEMX?'-A[7+@A'Y\ MWEJ2B/%;,)3F=Y:3RG@3BUP/1)=FF*5\:)C,:<1P!I.91WM %? %J$7P#!F"QF(%V)AS %68(:6WR66%I7;H8,I;ZX<_ M92V,684_6Q=K:HHID+AXXW$BA[9MSWQ/:?:5I6<6EIQI] MGFPHF963>F!>B1&9FFDCECR9!@:5BPH "9F9;"YYP8N69[V'B8@B6OJ9V%U& M!02 FD1<()IME"MK+%H;:GA>Z0'* AYI8P#M7,U?]9,7F!B9SHCW M#("&+9A3?@UH'YE2 GHSJVJT;OPT#6@%7M6,(6Y(CAN587?C:@]:M92+6@Y[ M:I;:7F)>^WB[7=>*Z7L_<_^,,EW1;0Z7B6I <0F)1&O7DX-A7Y%N7SL%@'>L M:FAKTY9KB,>.5Y(!8I]HXF4F@2^4NI%79Z"7Y5]Q>,ABMW\27(=D_HULC0MT M16J5@VIHN&G;87U^87ENE3=\<7>?7%^ ^FR,A<-@+W#J:*M_F5L8=4]FX(7; ME29NE)@%;@YIEENEQ>:FM[T6/E9$R2%W>X=9J$4V(8D9)JG%U7:P\C>&N89-^ <)I"C7^8 MTEQXD&>60Y:,EE]Y#)1;:>")XWP<=(N:,GI4=HV80V;2EHZ$MGG3=^.6(W(; ME"\!?91L?[":,'BK>&"##)CN><9UCWO^8@(M*Y1G8AJ8#WW@:/)MAW?8F"MX MPWB48L5XZ6_(>#V2)WYME\UX>G@J<"IQ=E[2> .5UGCT=V]@+X=^!3X5W'@) ME"U;G7SS;$-V*%V=?PQ@VY9,CP1X['@JDUR957UN=KI:9H):::M[M";5CM!; MNG!\BFMKSY3G6^N4GYB E71K GE]BZ9XV6 2>=1TO('M649;1'U? 2\ 00!% MEAX"\G="7E^"'YM*DXN63'V%F..:.6@C>1Y>K5MA7%R.IGEI=_N)JWE09'*6 MBY.(E)Z0-G_#>;:!$UZ,%U*'=XC(=YN$@6P8E%]X]6(?9=UYA9D.E\"$SXO8 M<#^5EV\*B+4[D7^-?F4ARWO)AYI\QH"FF3!X4WEN8(66=G]9#J1WO9:!A^&: MDFE\E49X^'!=FV9YC%JI>69[:8=*D&.;(I;I=6^6"GS4A0%G%Y"=F#&"Y%P# ME7YYO'G:=>AU@WGLB,9R])FKC_V7( %F6XYYLW/Z8"@ A9L8ES!XDWGS=HJ; M>FLN?X*8Z%V_EFAWH7ETF<5\5%N5FY9S*WE'FYF;2Y=LCD22-XGR>KYYIVK&8:8+M MA8F'B9M=>?%YH7=Y?=N'CXT^;HQ^(9OYB@%R))M^CWV2@YF;BN1F-GSM>.=\ M[WCS<"^)GHG=C9!_YY#">A)Z9905>I9Q37OH7!EZHVL@((6.59KSECY[FWU& M@#YQ)X&(?0" 6($A=HYV^)9[>X^4;G9U<=1I!6&"ES!L))P8>>B91&LMG,F, MI91Y8UIP&%K0DBE>JY#L6CV4UH&UBC-P1'KF?U:42'H+<4IZ+6#8A#*-FFU0 MF12(4WIRQZ:@W<>D1!E M&)1?8I@$16KH8/:*HG&$B&)ELX^'>J=E]VA^>C)O*6[RF8:!A'KV7Z-JB80( MD1AR"I&UCX]Z3(IW@7B6X6X?6@.0>%Q/@,9]I'3QDEIO)5_DEE^918;/D7UG M\&S$:F*(!&0:7*M:4YKEATQ:$7,&9+=@[HF8?$F<\5O2D@!<+Y>AG&9A;GYG MAMB34&$L;19;]V] >K=_%W[*E"IGREO% P)IB8LL8?9XFYL"E"!V4FIU8%IR MT7M$E!%C(EH/ B=N@7&48B*;Q&"TD!Q@>=%Y9:FBIW2&]L7&J)_7ZFF!*)7YBA:$V4T6=U6E"4?W.BD*,)7WSK:K9Y MMFJR@OZ3R&)796'1LDTUEJ6O1A85AIV0?D7*<')._B"&$K&)VCE$$/)$1 M>_-'5L%EDRY#IE05LD7'$DR]Y^7 QD4)J.FX2D>Y_V%R76DU?+Y>K77YN M,&QN9M6)0GZ?FNJ:OEJ>;""$S6.V>II=BI%,7+2.9P&T8UMDX6CQDY^0;8=G M>MF.+%R?G=Y@$W]/D]U=>RTO82>+=6@U7$Z:MV3!735>=X7E6.RA IGL&"673=="6=@GIE1?JM^Z8YP?;R%B#Q(-%?;->+ !# YU=N9AW@B@ $9X5GHEQX8R0:+B!RP)M D! M"X%% 1&>(Y[Q7#5T()>,8"%<;H%C!Y-O+5NQ#$N880$ICUF0H5X9E5* )6G# M@FF(<'BID6E=K)'_%A1&;BGR/FJ5^^VA_FA1L;VV.CX*8MXA]G?]W MQ)O2F%J;B)'HFH5_.9TNFT9CZGQ9@W( <)!'>-IT\'LX>;B!:VDA6QA:76C& M:VR68GM>F_E\,&@@EGU[B)0'BU0!X5MS7X)M.5RN8+!M[P&A6YZ<\E^KE,&2 M)J"1'"+ M>YV"+^9>0 K@]4%#U[,F+.;K%J$:/-=I'PR?3UZ:7!-<^&1:AFL!,V9K:"R%%&IV9]V8,U\J;8"<18BV MDQ-_>XX*!6]M^YA'6M9.[WDI5T16M"5+6!( (Q;*%RV:V4A@Y%?@3-VMFJ+ 6UU M<'T1'EJ>@D)F(9]Q M ZJ>X .LGA68HT@>8(^&,73Y7A*?<94=GB #'Y[ZF^MG?HTJGOR5B864APP[47*EXC@$VGBV>*IZJF;1:>6:>;"!A9UY"G_1SV(-) M I2"[%KLN!PGUSDK&?49RF6XJ0 M#XBFABZ-*7 V;DU?J89^DXQ;?H5! M?[Z>0WL_E!I;6V7B9.&;NY^W>T)MKGK[E(J'R95+?LN5\XBQ>R^5%):3?I&2 M?'LA<=1;0)8%FQ.4$)VT8\B5VG7*E0>,T'0.>"V25I6O=:)WT7J>7+]]\VXC M.OUQ/C"@% M/UX.86D!,9I(G@Q^J7^?8C^*F'[?:/=O9H43A\]^S7_'=Q-PLG^)=$%BT7IL MF<%A1ER$:J=<[HV47#AOYG@ 8,N/$XXUFG!F>EPXFAN* MT&M_E&*7,PK?6]%B-)"\I\HA;Y< MZV5L?>!P?H;4(H\A5P3=G7.)HG"&!"->BG@F8'=^?)F*5 Q;/5T7>B6<[&)$ M70QAS7Y+:3&@C)<89RUD>I6_9F%>J8?=715X(6#J?P9=\UKW6ZF&\'_BDM*? M0E[+G5V>Y&G"FR)ZXF0]7;.2_G]LA"UZ H!EGB1P!H#=GC)@*I;16W9Q#(!) M 0Z LUU0FZ%C$X"!(7H!YO:! G R9\XC-;7!EMF'-FGIW M19B5H V:SUJ[?WI=#0I\89A]WU^(7*&=A'.-:&9\3V*B8].*EW$_@/]O@&). M6[B*QG('+N<@-D;Y)\?"): MX:!Q7;%KJ6)9@/EB\HE=@/]=CIIA@#Q?9("UG5I:9X#66VF <60DC!^*K&!< M7W& %(YCCSIP=X#8?GF N5RFB8%G]W0^FQ$" GF!@!*+X7!$7"R0QG[8:*J) M 6-L<"IJ'G7-:9& : !M !IZP9*"F..7NI3*;'F%6V3L9T.'ZIZ.7Z* "&T\ M?'=^4X_%:RQKZUVG=,-C0V&\G4)A7P74CS9K#*&F9F ^Z 9>B< .'6MG(Z2 MK)#E=UH$YV7A8J=<06=Y8L9YHV( 6Y"'?X%B@+N0BI6:76QF6(PL<<&:=)Q? MA?-M78XW8QUJ9P"YE91;K)4WFAT!SURHFGD!+Q(N@>Y9M8#BEK" )FNR@'R= MPW2N@+> >Z$I>M:76H$K@8QE!Y3!@'D PX"$FP-<^'9+GX6AD0'Z@0^!>0 H ME*.%/@K*:MV FY5\D#^<.%V":-R 2&,19]6ATY8/?+.2R*"#:25KP5T#>:5; M-0&LARYX< +\@=NA.F047V->]( "<_R;^( \@0J 29O^@/*A 8$>8/2'R:%O M?TAT=91Q7@N!6 (>:24"YH4!"["A)8ES B5BIF;ZH5-IUI\>6A^!"*+:H4F; MZ&?CD(-C)X$$?3.<*H$2D3,(%T3VFB&"O01? $$ ZJ$7GF!X.($Z@35A\J&8 M=W,"XF(X7=Z9JIBL9F&!\Z%.>\I;38&L6B^B85SJH1ASI7IQDG"%0G$8HH=: MCB%P;:20$)[N>^FA$:*)AXYS*H. :<2AM8+;?"UAXWIYBOJA^X."@6)KIV*6 MA2->8F=VB^Q:1(;RC@8B= MC%HBD&=YHX'"FJ6!-90:6OJ)@FMY:25K3G6P?G)FL8&C95X!M('A6VF;J%I[ M84!F!I"PE>#W)Z)74)F98K=8NMB5J).8%0!1@&A<+.5 M80#N;?F/S7*"G#=A2Z!=7B^*QV;["EEXU)?/D+A[-U_,D-9X1FV& L1]&EI\ M8#X#_68H8J]K1(#_81IS(XN06JVK(HH?@9^5IWG>:UZA'[0!D)F\X4_ M;T9D69:A6^]>' '[A_J=/7[]. :.$$XYSE.V:,**^H)R%+Q=DF#U9S&"M&;?8::+!6;E7C9^KH$]EL]XUG"9H*MB3W1-?6Q[?62U M?R=K)J/ C2"A8GN0B59VDGU)F\:?MABD=YQ:"W'L<;]TSX3:;]*$1Z,0BKYZ M\HXQC4JA28MGA&"V#6FQ-Y=F@&C8Z@:Y<9:RT M20!6AF:1"VK,::>5*5IE8;Z"\VX)E]U=)77RBDQ[:6SF=S>.V=26T6=]1T)'5*:/.&3*"#;ZNA5V#8 MD+V-OI)SA*N.=W4/DS-\R9.0;81L5VW-DW:1=(EYH3J*-)]?:7Z$2F@WFNO"IA\YV=8HRZ(3UZ0 MA;61+%SMF,M?X:-HCT%:P6EK9_Z1"W2 A0:+^'3M6T&%*Z"K>Y%HK5UE=)Y; M0HKRG4\ 'A:$B*78]CHU^/OG])H<"&U9/:D'%GH5V"F.%W(J&$?J\*):'B M9[%R*:$/B+.1+0 X 3YK)5_3H*^%,EH9B-Z$M(57AQV6V)M"!=*;'%JXAE6% MQF.+EPUK<9KZFFRB!F=99FB/BIP8FRN'&H8::MA;.'DP?D&.V(4E<"25/8C3 MAM2*>FTSH"5PV(8I9O=TYI8OC"-^/9(Q7^V.N(O+<=-P5(@VC,R84:/IAMMS Z&FF!-E[(DTAU^D_7[ZAB^/9F1-DDR7#H%N6S84VC-F6A6APF^*;;B-*78((U(9IV;QE\)JVZ18%9T9WGAF]AYD@0Y8AB4RH@*7'69@'RY M97M>[)I,.'AHG%AXB):6-KB8N)58H8E@:& MCXJTHMFD5HDDG:F(#Y;R=I)L#W/HI*@R*5E%PZ@-QK6%_ZB,-II7[KA"!OVJ0PAFN) M#H>*>)./ HO;8>:D+0!!A6.))WJN9$-J38QCG>%;5(>H@$>6Y'E"7<97']H#&0NV9(;08+4,JPGP!@**A&.67*DH61-!"Q-N80"57'T5@ (5BD"5O88+FDZE0H6S M;V6E@%I2 &^E*V'$!@]3&0IH%6VE-9DII;V&OI:L6CT A9MZ6P-[,)&K:$"5 MP5J'Q(\W8<]@-F8_D9F9)G]"D0.(B&R1 MG]9R9%T18<$%4G13H<-9>'3P=EJ.T)DSDEF>.Y&7F/)PZVT]F%!AFF-Y8L1Q MQ&D+N.ABFW.AE>4W93N>@:=SUX% MH(B4TJ#O?R&D+&&NGAU=3J,,I("4%(=4=,=MBXX@C1&-&)2;:/.=^%O]I*)H M"GHP<7)Z)I RC+J+VV'L>7R6(99F M>!8P%=\:85GKE?7]MA:2+7%ZDN*,\ MBY9E0Y)^GIN44UL)AOF;GVX+#-(;_J59F-^''W$4 OJEP&SYA:26ZI0/G:&5 MXFI_=2*6GW<"B'9F^86&F&:C9UP5AVN/=GF#? A]&X?*7!T!**,6?0V::)6QEWIDC%HZ:G!Y 'M:FYYS M>9)/:;F>"EI-7_R0MW [BEF.4&/FGC"F+XC/71%K*)TE<;R<.&$^?W!;F(6R M;8^@VEJ:=,%Z%0)D75B7;W@#6ZB;>IE&9[]R"Z-,I76?&Y4M (UI(0+@BW%L M@%NEH?%;IG\,GH..,8I!IO5J)0)2<;MJAXE:9[VDQ&-0IBUK@Z8T6R-D56;X MFXL!:7BF6Y9]=2HCF12F"P(+7@:'PFT8ID%INY- =T>&J&^N%:8" HKPI 2* M5(9G?N&(%%OJ7<=: XCIB99UQ)3E8_F?%W[FI!=;8(ZX8=V648WTFJ-M<*;1 MG4J.LF+/F0J*A(4-BHE:8G"KA1**HI RB_9YKX;GHY5_>71*6B2EZW+;&(WBLRCS6@5 M?-F2/HII;$"*_F !I ]@&HI>:V2%.I5AHF*CXXW:B7&+M5\.I&)K@IC+B?&7 MEYCS:^N3C)6P7SU@$ 5?BGZ2N7(U9!EG17'LC&>*DH;G7HEO,'-4HU**3%JJ M=0IR3%K$;Y!:3 !UBO6?-GQV:4MPXWXK;0YCM)1(GFYXBV82H'1GUVZ&BI%Q M4H5T7L<;19#3>D>0PY1!GT):59G4=:.;G%NOB,^,O&34:_Z$TXSEB"J<8IY( M87N59'&C 55_ W!9HP=:NGY"7C!SDIJ+E5"2>H0#I>Y?KXKBF?]ILHH?IR-S M,G 3 UB5Q(@TDRV)\Z;PA/6F8:,%9L&*8US!?WUUL)9WG*A?)5_YHPYCC7++ MBDEDH9C#:6L!0&E)I!Z*VH8"7.20B&%&I+Y0Z(/H%8I_H!_&XW7DZG\9Q$IGBFSYX'A.0!Y&U$<+X!J&0H M>RQ]_X;3F^-QCF[4DC):;:8E ]6!8F#4FV0 +0"KHSI<5HT(;[%E(6<4 :&, M*&DZ;OB,:8>C722/WY-GHIR!;)DR6O=O_EXY7'B47I5V7EMA[0%]D/!^YI)4 MG"X *HU7G"R-EFCV8$%>3J;AI1:.1'Q!>#B-FXTYD$=OE8Z@C6><8X^DC3R. M1HU/9C2.>F9*C<"*38U1:/J5H'*D FR;/FOLGK"%YY]9C7YAT)WU7GAC=Y'? M7PZ*5*-EC>2$A'K%?#.%AX2ECFV-2BKNDC!LE9NZABIE^I6FF.""\8^:CSQ? MNF(P;PI[M)/KE=.D%IZ\:S)^&J6M;&I^\5K\FZ6.Q9T7 >"0CUV_H:1DCXR< M:5B/KH\BA1EJ#H>6<:B!:F0)7/N3(Y (;8V&%P,^I/R-RUVQG9^=.&EN8Y]@ M3933:$NF46D<C:HZY5]?BEGZ8>2DKZ-% HYHQ6>='MEGG9&J#B)(XT1=.V-^5M)FVKCO"1?:A7CMP!#HLU76IMA( =CM]:'XXJ;CJ,X79??"6.+5SUGTB':H<& M9<+6E361PA?J5 M<6M!A>V89F&WC[^<7HTD;7I0X9>[DCN'%J.<@1U>"&;4:*&5=:+<8X:,V:>D MA-ZGKZC%J+5N=U77'9HZ-SX<\8#IF<)-_8@"7S9,$I$QC M90"$EAUM_*=:;OEOMFVGC2QA.&ZHG_".RZ?_;]!]/Q8LC M=Z::'FG*J'V?4 'A@1)GB:$H $N@;)[*'_(D..#_&D0C'>'7GF+9(1<$W&_ M=0>HI'(QI!9:OG/=FXAW+0!Q:[^&G(Y5>A9>=&FEG:AJ^9_1?2IE-*G3D9AD M>Z*5HX-EGH#^GHF?\UD.H0)<[FG/:S9_?7_: 9*4R3M 7(B?%6FQ>_&3^'C/ M9SAFIY&6DYV85W]Y9:M_OJ7/7;*>*9JFCBIB*F5=IC1KPP['F$-R(W'A=/V1 M*WNSE9EQS6Z">L>2!9;M:L@H4&-] 5ADMH346CBI20"K9:JHYI[X78"@PV&Y MG,9=H)S!9\I\/J89<<-SR*/);9&1+8NX$@&026>9HJ]?.8PJ;:>.@6\$D =N MT0:-D.J@@J$=;:M=1H^&?+V1(8$LJ<*H M$7(FH M^-UZ H/"%QEU-:;:=+Y=6CC2DOEPR95)R!9L17S*20X$?H/R<_Y8" M6@JJW8Q=982FN9RQ784!Q61R DIA5ZEG<01M6F0:7]*I.IVIG*5G'6]RFMY: M/F_#C*UPE(O1>A9;QE_KJ;EE@)\F9FY?>ZJ4?H*@&J6AE'1QI:DQ7>N,S6NO M7_=;>J:5G+)>00 IJ*%D_Y*QG7U^)'4 8X!_0(>:@<9U!VPV:&&31'M66QJE M.G$"6K-PNXU*G.J0.ZF?8)NFHIT!A[BBU952=*V0SHM(F$=:=@7$A%0 5 $3 C U$$55]I 0H /@5 !6D!6 5%!14%6P5C V %"P)Z 4\ W@=: end From owner-mpi-context@CS.UTK.EDU Thu Mar 25 00:08:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07003; Thu, 25 Mar 93 00:08:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21855; Thu, 25 Mar 93 00:08:25 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 00:08:24 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21847; Thu, 25 Mar 93 00:08:23 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07335; Wed, 24 Mar 93 23:02:31 CST Date: Wed, 24 Mar 93 23:02:31 CST From: Tony Skjellum Message-Id: <9303250502.AA07335@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: Silence? I have no mail in my inbox on contexts. Hasn't anyone anything to say about the tripartite proposal? I am hoping for input from others than the co-authors, of course :-) - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 25 05:10:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14110; Thu, 25 Mar 93 05:10:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05401; Thu, 25 Mar 93 05:09:19 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 05:09:18 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05393; Thu, 25 Mar 93 05:09:14 -0500 Date: Thu, 25 Mar 93 10:09:07 GMT Message-Id: <17744.9303251009@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Silence? To: Tony Skjellum , mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Wed, 24 Mar 93 23:02:31 CST Reply-To: lyndon@epcc.ed.ac.uk Hi Tony. Until your mail of yesterday, I hadn't realised that you intended the suggested straw poll of proposals to be held by email. I could be wrong, but I don't think this will work --- just too few people doing the email stuff. > I have no mail in my inbox on contexts. Hasn't anyone anything to say > about the tripartite proposal? Well there to date have been really only Rik, Jim, Marc (one message, respone to Jim), Tom Henserson (one message, clarification) and myself doing anything in this subcommittee mail list. Perhaps its just too early yet for any responses. > I am hoping for input from others than the co-authors, of course :-) However one will accept input from other authors, of course :-) I certainly do have comments to make on each of the proposals. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Mar 25 09:54:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15684; Thu, 25 Mar 93 09:54:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19738; Thu, 25 Mar 93 09:53:23 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 09:53:22 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19730; Thu, 25 Mar 93 09:53:20 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA28293; Thu, 25 Mar 93 14:53:16 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA02709; Thu, 25 Mar 93 07:51:59 MST Date: Thu, 25 Mar 93 07:51:59 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303251451.AA02709@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: Re: Silence? > I have no mail in my inbox on contexts. Hasn't anyone anything to say > about the tripartite proposal? > > I am hoping for input from others than the co-authors, of course :-) > - Tony Hey, I haven't even finished reading it! I only got it yesterday... :-) Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-context@CS.UTK.EDU Thu Mar 25 10:37:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16999; Thu, 25 Mar 93 10:37:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21399; Thu, 25 Mar 93 10:37:13 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 10:37:12 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21391; Thu, 25 Mar 93 10:37:11 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA03293; Thu, 25 Mar 93 08:37:09 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nbtzQ-0016bbC; Thu, 25 Mar 93 08:37 MST Message-Id: Date: Thu, 25 Mar 93 08:37 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-context@cs.utk.edu Subject: Re: Silence? I'm still trying to dig through a pile of documents to find which one I'm supposed to "be non-silent about" and which ones are too old. When you get 3 or so of these a day out of context (pun not intended), i.e., not an author, it's hard to keep up with it. By the time one doc is printed it is already history. Stephen From owner-mpi-context@CS.UTK.EDU Thu Mar 25 10:48:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17260; Thu, 25 Mar 93 10:48:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21849; Thu, 25 Mar 93 10:47:57 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 10:47:56 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21766; Thu, 25 Mar 93 10:45:43 -0500 Date: Thu, 25 Mar 93 15:44:59 GMT Message-Id: <18119.9303251544@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:15:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20646; Thu, 25 Mar 93 12:15:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25731; Thu, 25 Mar 93 12:15:00 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:14:59 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25723; Thu, 25 Mar 93 12:14:55 -0500 Date: Thu, 25 Mar 93 17:14:46 GMT Message-Id: <18221.9303251714@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Silence? To: srwheat@cs.sandia.gov (Stephen R. Wheat), mpi-context@cs.utk.edu In-Reply-To: Stephen R. Wheat's message of Thu, 25 Mar 93 08:37 MST Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Context Colleagues Stephen writes: > I'm still trying to dig through a pile of documents to find > which one I'm supposed to "be non-silent about" and which > ones are too old. I guess plenty of subcommittee members will have this difficulty. I believe that I can be of service by providing clarification. The chapter(s) from this subcommittee which (to my knowledge) Tony has sent for inclusion in the MPI draft were circulated by Tony to the subcomittee mail list. The messages of interest are: Date: Wed, 24 Mar 93 10:57:27 CST From: Tony Skjellum Subject: draft of MPI Context proposals (Latex) Date: Wed, 24 Mar 93 11:04:43 CST From: Tony Skjellum Subject: postscript of draft (uuencoded) I also refer you to a helpful summary sent out by myself :-) The message of interest is: Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals > When you get 3 or so of these a day > out of context (pun not intended), i.e., not an author, > it's hard to keep up with it. By the time one doc > is printed it is already history. Yes, this has been a busy mail list recently, and I do apologise for having worked so hard on the MPI communication contexts :-) If you have time and enthuiasm I expect that you would find tracing through the historical evolution of these documents over the past week both interesting and revealing. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:33:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21280; Thu, 25 Mar 93 12:33:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26541; Thu, 25 Mar 93 12:33:01 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:33:00 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26533; Thu, 25 Mar 93 12:32:59 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07552; Thu, 25 Mar 93 11:27:02 CST Date: Thu, 25 Mar 93 11:27:02 CST From: Tony Skjellum Message-Id: <9303251727.AA07552@Aurora.CS.MsState.Edu> To: tony@Aurora.CS.MsState.Edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Silence? Jack told me that it is essential that we have our act together as far as possible by meeting. A straw poll by e-mail might be meaningless, as you say, because of small interactions (many readers, few writers). Educationally speaking, I would like to present all three proposals to whole committee, to explain differences and similarities. Since our act is together now, it would be educational/beneficial not to exclude any one of the three. My straw poll was following Jack's advice. If there is no strong sentiment for it, we will not have it. COmments from everyone, please? I had assumed, at the moment of the conception of the straw poll, that we might have 5 rather than 3 proposals! Three is a managable presentation. They are sufficiently different that I do not see a way to merge them, and preserve their intended properties. - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:35:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21339; Thu, 25 Mar 93 12:35:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26701; Thu, 25 Mar 93 12:35:25 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:35:23 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26684; Thu, 25 Mar 93 12:35:22 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07560; Thu, 25 Mar 93 11:29:14 CST Date: Thu, 25 Mar 93 11:29:14 CST From: Tony Skjellum Message-Id: <9303251729.AA07560@Aurora.CS.MsState.Edu> To: tony@aurora.cs.msstate.edu, lyndon@epcc.ed.ac.uk Subject: Re: proposal iii Cc: mpi-context@cs.utk.edu ----- Begin Included Message ----- From lyndon@epcc.ed.ac.uk Thu Mar 25 06:16:18 1993 Date: Thu, 25 Mar 93 12:21:59 GMT From: L J Clarke Subject: proposal iii To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Content-Length: 950 Private message --------------- Dear Tony Regarding Proposals I, III and VII I will be prepared to make value judgement oriented comments over the next couple of days. Whereas the comparisions of Proposal III versus I and VII which you have included in III are useful, I believe that the value judgments which you also included are inappropriate to the document, and unhelpful to the process of the subcommittee (not to mention unfair on the authors of the other proposals especially Rik and myself). I hope that you will endeavour to edit such material out of your proposal and resubmit to Steve Otto for circulation in the MPI draft. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- Lyndon, I disagree and I will not remove them. Feel free to add your own conclusion comments to your and ask Marc to do so for his. I find them completely appropriate. - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:40:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21397; Thu, 25 Mar 93 12:40:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26868; Thu, 25 Mar 93 12:39:39 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:39:37 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26850; Thu, 25 Mar 93 12:39:35 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07566; Thu, 25 Mar 93 11:33:39 CST Date: Thu, 25 Mar 93 11:33:39 CST From: Tony Skjellum Message-Id: <9303251733.AA07566@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Thank you for this additional work, Lyndon. Why don't we include this as the preface to our proposal? It looks good, but I will have to read it in detail, before rendering my complete view on this matter. Who are these other, new cc people... - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:42:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21429; Thu, 25 Mar 93 12:42:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27079; Thu, 25 Mar 93 12:41:54 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:41:53 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27031; Thu, 25 Mar 93 12:41:07 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA02886 (5.65c/IDA-1.4.4); Thu, 25 Mar 1993 12:41:03 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA18733; Thu, 25 Mar 93 17:40:59 GMT Date: Thu, 25 Mar 93 17:40:59 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303251740.AA18733@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA07372; Thu, 25 Mar 93 17:37:30 GMT To: snir@watson.ibm.com Cc: mpi-pt2pt@cs.utk.edu Cc: mpi-context@cs.utk.edu Subject: PT2PT draft (MARCH 23) Content-Length: 5131 Marc, [This is a reconstruction from memory of some mail I sent this morning, which seems to have disappeared... If anyone got the original they can test how good my memory is by comparing the two mails :-)] Here are some comments and queries on the current (23 March) point to point proposal. To avoid duplication of lots of text I will reference the proposal bye page number. Apologies to those of you who haven't printed it ! Page 3 : Discussion of options for Lists of Handles My preference is for option 3 (A separate length parameter). I prefer this over 1 (length as first element) because this leads to off by one errors, or a funny structure in C with an arbitrary sized array. I prefer it over 2 (a delimited list) because it avoids a whole pass over the list to work out its length when this is required. (e.g. if a copy needs to be made. cf the horrible C code char * copy = strcpy(malloc(strlen(string)+1), string); which accesses every byte of the string twice) Page 4: Discussion of named constants in FORTRAN. The F90 discussion should mention MODULEs (e.g. "... can be made available via an INCLUDE file, or MODULE.") PLEASE PLEASE PLEASE don't use character strings. This is guaranteed to be slow. I'd rather have literal values in place than strings ! Page 8 et seq: Contexts. I don't understand how to implement your context model in the way you seem to be describing. On page 8 you suggest that you can check the context at transmission, and avoid a collective agreement alogrothm at context creation. Similarly on page 10 in the implementation note you say "No communication is needed to create a new context beyond a barrier synchronisation". Surely you MUST check the context at the receiver (you seemed to agree with this in previous mail in MPI-context), therefore the sender and receiver must agree on the context name. (Or at least the sender must know the value by which the context is known at the receiver, which could be different for each receiver if you like [I don't !]) To achive this surely you need a group co-operation on context creation. Consider Process number i.e. Rank in ALL 0 1 Group 0 0 1 Group 1 1 0 The number in the table is the rank of the process in the group. Certainly when process 1 receives a message from zero he must know the group/context from which it was sent, since the result of the sender inquiry will be different depending on the group/context. Even if you only allow partitioning, I don't think you can avoid needing a cooperation, consider a scenario like this Process number Time 0 0 1 2 Time 1 0 | 1 2 Partition of all Time 2 1 | 2 Partition of subgroup Time 3 0 1 | 2 Partition of all Where the | denotes a partition of the group. Now with only a sequnce number process 0 has has been involved in 2 partitions, but process 1 (which should be in the same group) has been involved in 3. Ah, you can get the right answer by using a count of the partitions of the the parent group, and its depth in the tree. (Somehow...) Anyway it all falls apart when you let in the group construction by list (I believe) ! Page 12: I agree that we shouldn't REQUIRE dynamic process creation. Page 13: A global errno is the CLASSIC example of global data which gives threads a problem. Do we REALLY want to do it like this ? Page 14: Data types I think we should have both MPI_CHAR and MPI_BYTE. The difference being that in a heterogeneous environment MPI_CHAR would be translated according to the local character sets (e.g. ASCII -> EBCDIC), MPI_BYTE would be an 8 bit integer. What should we do about Fortran 90 KINDs ? Page 15: Data buffers. Vector buffer : Do we allow a zero or negative inter block stride ? Zero is a neat way to perform a fill operation... Page 17: Last line on the page... What is the (4,2,1) ? It's not Fortran 77. Looks a bit like an F90 array constructor ... Page 18: Data conversion You state "No data conversion occurs when data is moved from a sender buffer into a message". In a heterogeneous environment surely this is an implementation issue. The implementation should be free to translate at source, at dest, or on the moon if it likes. Page 36: You should delete "out of either process' address space." The example only requires buffering, it doesn't matter where the implementation chooses to do it. General question ================ Where is the store which is referenced by handles allocated ? At present it appears that all of this (even for ephemeral objects) is MPI system managed. I would very much like it to be possible to have the user manage this store, as she can often do it cheaper. (e.g. allocating ephemeral objects on the stack). Hmmm... came out more or less the same I think. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:52:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21694; Thu, 25 Mar 93 12:52:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27592; Thu, 25 Mar 93 12:51:53 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:51:52 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27577; Thu, 25 Mar 93 12:51:50 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA28820; Thu, 25 Mar 93 17:51:46 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA03192; Thu, 25 Mar 93 10:50:29 MST Date: Thu, 25 Mar 93 10:50:29 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303251750.AA03192@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: Re: Silence? > Jack told me that it is essential that we have our act together as far > as possible by meeting. A straw poll by e-mail might be meaningless, as you > say, because of small interactions (many readers, few writers). > > My straw poll was following Jack's advice. If there is no strong sentiment > for it, we will not have it. COmments from everyone, please? > > I had assumed, at the moment of the conception of the straw poll, that we > might have 5 rather than 3 proposals! Three is a managable presentation. > They are sufficiently different that I do not see a way to merge them, > and preserve their intended properties. > > - Tony I think an email straw poll is a good idea. At least you'll get some information. Tom From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:53:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21701; Thu, 25 Mar 93 12:53:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27647; Thu, 25 Mar 93 12:53:38 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:53:37 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27639; Thu, 25 Mar 93 12:53:34 -0500 Date: Thu, 25 Mar 93 17:53:29 GMT Message-Id: <18278.9303251753@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: proposal iii To: Tony Skjellum In-Reply-To: Tony Skjellum's message of Thu, 25 Mar 93 11:29:14 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Tony I am dissapointed that you find it necessary to circulate the private message publically. I shall reply to you personally. I believe that the point I made is valid. Since you advise to add comments into the conclusion section of the chapters, will you also advise as to the mechanism considering that the chapters have gone to the MPI draft editor for inclusion into the draft. Best Wishes Lyndon > > > ----- Begin Included Message ----- > > >From lyndon@epcc.ed.ac.uk Thu Mar 25 06:16:18 1993 > Date: Thu, 25 Mar 93 12:21:59 GMT > From: L J Clarke > Subject: proposal iii > To: tony@aurora.cs.msstate.edu > Reply-To: lyndon@epcc.ed.ac.uk > Content-Length: 950 > > Private message > --------------- > > Dear Tony > > Regarding Proposals I, III and VII I will be prepared to make value > judgement oriented comments over the next couple of days. > > Whereas the comparisions of Proposal III versus I and VII which you have > included in III are useful, I believe that the value judgments which you > also included are inappropriate to the document, and unhelpful to the > process of the subcommittee (not to mention unfair on the authors of the > other proposals especially Rik and myself). > > I hope that you will endeavour to edit such material out of your > proposal and resubmit to Steve Otto for circulation in the MPI draft. > > > Best Wishes > Lyndon > > /--------------------------------------------------------\ > e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) > c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c > \--------------------------------------------------------/ > > > > > ----- End Included Message ----- > Lyndon, > > I disagree and I will not remove them. Feel free to add your own > conclusion comments to your and ask Marc to do so for his. I find them > completely appropriate. > > - Tony > > /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Mar 25 12:58:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21783; Thu, 25 Mar 93 12:58:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27841; Thu, 25 Mar 93 12:58:28 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 12:58:27 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27833; Thu, 25 Mar 93 12:58:23 -0500 Date: Thu, 25 Mar 93 17:58:13 GMT Message-Id: <18293.9303251758@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Silence? To: hender@macaw.fsl.noaa.gov (Tom Henderson), mpi-context@cs.utk.edu In-Reply-To: Tom Henderson's message of Thu, 25 Mar 93 10:50:29 MST Reply-To: lyndon@epcc.ed.ac.uk I will of course participate in an email straw poll. Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Mar 25 13:43:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22825; Thu, 25 Mar 93 13:43:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29945; Thu, 25 Mar 93 13:42:40 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 13:42:38 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29937; Thu, 25 Mar 93 13:42:37 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA01033; Thu, 25 Mar 93 13:42:34 -0500 Date: Thu, 25 Mar 93 13:42:34 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303251842.AA01033@msr.EPM.ORNL.GOV> To: mpi-context@cs.utk.edu Subject: Re: Silence? > I have no mail in my inbox... I'm with Tom. I haven't finished reading it yet. (-: ----------------------------- __o /\ Al Geist _`\<,_ /\/ \ Oak Ridge National Laboratory (_)/ (_) / \ (615) 574-3153 gst@ornl.gov * * * * * * * * * * ----------------------------- From owner-mpi-context@CS.UTK.EDU Thu Mar 25 13:59:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23332; Thu, 25 Mar 93 13:59:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00756; Thu, 25 Mar 93 13:58:38 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 13:58:36 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00747; Thu, 25 Mar 93 13:58:34 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07654; Thu, 25 Mar 93 12:52:38 CST Date: Thu, 25 Mar 93 12:52:38 CST From: Tony Skjellum Message-Id: <9303251852.AA07654@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Lyndon, as you know I was immensely under the gun yesterday, and my 9-hour trip was quite an ordeal, but I did get the sub-committee's proposal in on time, and it is basically quite good. I do feel that it is appropriate to include comparisons and even criticisms in the conclusions, but it was not done evenly, since mine was last. To this extent, I see why you did not like that. In same light, I can see why we should drop your name and Rik's name from proposal III, as this might unfairly indicate your support for proposal III (which is admittedly, an extended version of the programming model I support most, and is what Zipcode does, plus some additions). Please restrain yourself from completely dominating this subcommittee, because of the remarkable amount of time you are able to devote to it. To be fair, I cannot devote more than 2hr/day, and you are able to devote something like 12hr/day to it. People at the SIAM meeting were overwhelmed by the volume of mail you generate on different topics. You are a star performer, but you are also very demanding, and I must say that I have had some aggravations in this last weeks, but not only from you. I am trying not to push my personal viewpoint too hard. Please work from a viewpoint of cooperation with me, as we move towards the meeting next week. In this vein, see next paragraph. I will read your summary, but I already agree (note: agree) to drop all conclusion comments from all three subproposals that are "judgemental" (a loaded word). Since we are making a report out of this, the comparison should go in as latex. I will give this to Otto today (I will give it to him). Do you want me to latex your contribution, or will you (it is not too late there). Tell me. Ala Frankie & Johnny, I have to hang up now, since there is more mail from you to read :-)... I will not respond to that which is outdated by this letter. Please work on a latex version of your excellent new contribution, which also makes our joint report much better. I thank you. PS For rest of committee, the new contribution by Lyndon does not invalidate yesterday's tripartite proposal, it will merely be included in it. ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-context@CS.UTK.EDU Thu Mar 25 14:04:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23478; Thu, 25 Mar 93 14:04:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00948; Thu, 25 Mar 93 14:03:21 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 14:03:20 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00940; Thu, 25 Mar 93 14:03:19 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07664; Thu, 25 Mar 93 12:57:17 CST Date: Thu, 25 Mar 93 12:57:17 CST From: Tony Skjellum Message-Id: <9303251857.AA07664@Aurora.CS.MsState.Edu> To: tony@aurora.cs.msstate.edu, lyndon@epcc.ed.ac.uk, tony@Aurora.CS.MsState.Edu Subject: Re: proposal iii Cc: mpi-context@cs.utk.edu ----- Begin Included Message ----- From tony Thu Mar 25 11:29:14 1993 Date: Thu, 25 Mar 93 11:29:14 CST From: Tony Skjellum To: tony@aurora.cs.msstate.edu, lyndon@epcc.ed.ac.uk Subject: Re: proposal iii Cc: mpi-context@cs.utk.edu Content-Length: 1434 ----- Begin Included Message ----- >From lyndon@epcc.ed.ac.uk Thu Mar 25 06:16:18 1993 Date: Thu, 25 Mar 93 12:21:59 GMT From: L J Clarke Subject: proposal iii To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Content-Length: 950 Private message --------------- Dear Tony Regarding Proposals I, III and VII I will be prepared to make value judgement oriented comments over the next couple of days. Whereas the comparisions of Proposal III versus I and VII which you have included in III are useful, I believe that the value judgments which you also included are inappropriate to the document, and unhelpful to the process of the subcommittee (not to mention unfair on the authors of the other proposals especially Rik and myself). I hope that you will endeavour to edit such material out of your proposal and resubmit to Steve Otto for circulation in the MPI draft. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- Lyndon, I disagree and I will not remove them. Feel free to add your own conclusion comments to your and ask Marc to do so for his. I find them completely appropriate. - Tony ----- End Included Message ----- Please disregard this message. I was annoyed with Lyndon, and I had not seen his helpful summary then. I have proposed a more constructive resolution to our tiff. - Tony From owner-mpi-context@CS.UTK.EDU Thu Mar 25 20:01:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03122; Thu, 25 Mar 93 20:01:56 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15845; Thu, 25 Mar 93 20:00:59 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 25 Mar 1993 20:00:58 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15835; Thu, 25 Mar 93 20:00:54 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA11159; Thu, 25 Mar 93 18:54:43 CST Date: Thu, 25 Mar 93 18:54:43 CST From: Tony Skjellum Message-Id: <9303260054.AA11159@Aurora.CS.MsState.Edu> To: otto@iliamna.cse.ogi.edu Subject: Re: To MPI Authors: Status, Macros, etc Cc: mpi-context@cs.utk.edu Steve, Here is our updated version of the context subcommittee document, adhering to your canonical format. - Tony ---------------------------------------------------------------------------- \documentstyle[twoside,11pt]{report} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface} \author{Scott Berryman, {\em Yale Univ} \\ James Cownie, {\em Meiko Ltd} \\ Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\ Al Geist, {\em ORNL} \\ Bill Gropp, {\em ANL} \\ Rolf Hempel, {\em GMD} \\ Bob Knighten, {\em Intel} \\ Rusty Lusk, {\em ANL} \\ Steve Otto, {\em Oregon Graduate Inst} \\ Tony Skjellum, {\em Missisippi State Univ} \\ Marc Snir, {\em IBM T. J. Watson} \\ David Walker, {\em ORNL} \\ Steve Zenith, {\em Kuck \& Associates} } \date{March 25, 1993 \\ This work was supported by ARPA and NSF under contract number \#\#\#, by the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615. } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \label{sec:context} %======================================================================% % BEGIN "Proposal I" %======================================================================% % BEGIN "Proposal I" % Written by Marc Snir % Edited by Lyndon J. Clarke % March 1993 % \chapter{Contexts -- Proposal I} \begin{center} Marc~Snir \end{center} \section{Contexts} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \subsection{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \subsection{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \subsection{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \subsection{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \section{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \subsection{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } % % END "Proposal I" %======================================================================% %\end{document} %\documentstyle{report} % \chapter{Contexts -- Proposal VII} \begin{center} Lyndon~J~Clarke \& Rik~J~Littlefield \end{center} %---------------------------------------------------------------------- % Introduction \section{Introduction} This chapter is similar in basic principles to Proposal~I and includes all of the functionality of that proposal as a subset --- it extends in several ways and differs in some details. Certain features of other, now defunct, proposals discussed in the context subcommittee are included. In particular, this chapter proposes that: \begin{enumerate} \item Contexts and groups are not identical. A context is always associated with one group, but a group may have several contexts. Properties of groups are inherited by all of the associated contexts, for example process rank. \item Context and group descriptors can be explicitly transferred to processes that are not members of the context or group. \item In point-to-point messages, processes can be identified in any of three ways: by process, by rank in a shared context, or by ranks in separate sender and receiver contexts. \item A ``cache'' facility is provided that allows modules to attach arbitrary information to both contexts and groups. \end{enumerate} These extensions are somewhat independent of each other. The first reflects the observation that multiple modules often operate within each process group, so that context formation should be lighter weight than group formation. The second and third together provide expressive support for communication between modules within different groups of processes. The fourth allows modules to be significantly faster in common cases, without complicating their interface to the application. Much of this proposal must be viewed as recommendations to other subcommittees of {\sc mpi}, primarily the point-to-point communication subcommittee and the collective communications subcommittee. Concrete syntax is given in the style of the ANSI C host language, only for purposes of discussion. %---------------------------------------------------------------------- % Processes \section{Processes} This proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and {\sc mpi} does not distinguish such threads. \subsection*{Process Identifier} Each process is identified by a process-local {\it process handle}, which is a reference to a {\it process descriptor} of undefined size and opaque structure. In a static process model process handles can be obtained by mapping from a group (or context) and rank. In a future extension for dynamic processes, handles may be returned by process creation functions. {\sc mpi} provides a procedure which returns a handle for the calling process. \begin{verbatim} process = mpi_my_process() \end{verbatim} \subsection*{Process Creation \& Destruction} This proposal makes no statements regarding creation and destruction of processes. {\sc mpi} provides facilities for descriptor transmission allowing the user to explicitly transfer a process decriptor from one process to another. These facilities are described below. %---------------------------------------------------------------------- % Groups \section{Process Groups} This proposal views a process group as an ordered collection of (references to) distinct processes, the membership and ordering of which does not change over the lifetime of the group. The canonical representation of a group is a one-to-one map from the integers $(0, 1, \ldots, N-1)$ to handles of the $N$ processes composing the group. There may be structure associated with a process group defined by a process topology. This proposal makes no further statements regarding such structures. \subsection*{Group Identifier} Each group is identified by a process-local {\it group handle}, which is a reference to a {\it group descriptor} of undefined size and opaque structure. The initialization of {\sc mpi} makes each process a member of the ``initial'' group. {\sc mpi} provides a procedure that returns a handle to this group. \begin{verbatim} group = mpi_initial_group() \end{verbatim} {\sc mpi} provides facilities for descriptor transmission allowing the user to explicitly transfer a group descriptor from one process to another. \subsection*{Group Creation and Deletion} {\sc mpi} provides facilities which allow users to dynamically create and delete process groups. The procedures described here generate groups which are static in membership. {\sc mpi} provides a procedure which allows users to create one or more groups which are subsets of existing groups. \begin{verbatim} groupb = mpi_group_partition(groupa, key) \end{verbatim} This procedure creates one or more new groups {\tt groupb} which are distinct subsets of an existing group {\tt groupa} according to the supplied values of {\tt key}. This procedure is called by and synchronises all members of {\tt groupa}. {\sc mpi} provides a procedure which allows users to create a group by permutation of an existing group. \begin{verbatim} groupb = mpi_group_permutation(groupa, rank) \end{verbatim} This procedure creates one new group with the same membership as {\tt groupa} with a permutation of process ranking, and returns the created group descriptor in {\tt groupb}. It is called by and synchronises all members of {\tt groupa}. {\sc mpi} provides a procedure which allows users to create a group by explicit definition of its membership as a list of process handles. \begin{verbatim} group = mpi_group_definition(listofprocess) \end{verbatim} This procedure creates one new group {\tt group} with membership and ordering described by the process handle list {\tt listofprocess}. It is called by and synchronises all processes identified in {\tt listofprocess}. {\sc mpi} provides a procedure which allows users to delete user created groups. \begin{verbatim} mpi_group_deletion(group) \end{verbatim} This procedure deletes an existing group {\tt group}. It is called by and synchronises all members of {\tt group}. {\sc mpi} may provide additional procedures which allow users to construct process groups with a process group topology. \subsection*{Group Attributes} {\sc mpi} provides a procedure which accepts a valid group handle and returns the rank of the calling process within the identified group. \begin{verbatim} rank = mpi_group_rank(group) \end{verbatim} {\sc mpi} provides a procedure which accepts a valid group handle and returns the number of members, or {\it size}, of the identified group. \begin{verbatim} size = mpi_group_size(group) \end{verbatim} {\sc mpi} provides a procedure which accepts a valid group handle and process order number, or {\it rank}, and returns the valid process handle to which the supplied rank maps within the identified group. \begin{verbatim} process = mpi_group_process(group, rank) \end{verbatim} {\sc mpi} may provide additional procedures which allow users to determine the process group topology attributes. {\sc mpi} provides a group descriptor cache facility which allows the user to attach attributes to group descriptors. %---------------------------------------------------------------------- % Contexts \section{Communication Contexts} This proposal views a communication context as the combination of a process group and a protection mechanism that avoids collision between messages sent to different contexts. The context inherits process ranking from its associated group, referred to as a {\it frame}. Each process group may be used as a frame for multiple contexts. \subsubsection*{Context Identifier} Each context is identified by a process-local {\it context handle}, which is a reference to a {\it context descriptor} of undefined size and opaque structure. The creation of a process group allocates a {\it base context} which inherits the created group as a frame and can be thought of as an attribute of the created group. {\sc mpi} provides a procedure which accepts a valid group handle and returns a handle to the base context within the identified group. \begin{verbatim} context = mpi_base_context(group) \end{verbatim} {\sc mpi} provides facilities for descriptor transmission allowing the user to explicitly transfer a context descriptor from one process to another. \subsubsection*{Context Creation and Deletion} {\sc mpi} provides facilities which allows user to dynamically create and delete contexts in addition to the base context associated with a process group. Contexts created in this fashion can be thought of as copies of the base context of the process group. {\sc mpi} provides a procedure which allows users to create contexts. This procedure accepts the handle of a group of which the calling process is a member, and returns a handle to the new context. \begin{verbatim} context = mpi_context_creation(group). \end{verbatim} This procedure must be called loosely synchronously by all members of {\tt group}. The procedure may not actually synchronize the member processes --- it is suggested that this is a lightweight procedure that can be implemented so as to not require interprocess communication. {\sc mpi} provides a procedure which allows users to delete user created contexts. The procedure accepts a context handle that was created by the calling process and deletes the identified context. \begin{verbatim} mpi_context_deletion(context) \end{verbatim} This procedure has the same synchronization behavior as context creation. \subsubsection*{Context Attributes} {\sc mpi} provides a procedure which allows users to determine the process group that is the frame of a context. \begin{verbatim} group = mpi_context_frame(context) \end{verbatim} {\sc mpi} provides a group descriptor cache facility which allows user to attach attributes to group descriptors. %---------------------------------------------------------------------- % Descriptors \section{Descriptor Facilities} This section describes the descriptor transmission and user cache facilities. \subsection*{Transmission Facility} {\sc mpi} provides a mechanism whereby the user can transmit a valid descriptor in a message such that the received descriptor handle is valid. This can be integrated with the capability to transmit typed messages, and it is suggested that a notional data type should be introduced for this purpose, e.g. {\tt MPI\_DSCR\_TYPE}. There are other reasonable approaches to providing this facility. The descriptor is translated as necessary to be meaningful on the destination process, storage is allocated for it, and a handle to that storage is returned. Decorations are not transmitted. Handles are guaranteed to be unique within each process --- if processes A and B independently send to process C a descriptor for an object D, then process C will get two copies of the same handle. As with all transfers of descriptors, the receiving process is responsible for releasing the descriptor and its handle when it is no longer needed or becomes stale. MPI provides a procedure which frees a descriptor. \begin{verbatim} mpi_free_dscr(handle) \end{verbatim} A descriptor registry service which allows descriptors to be identified by name would be a useful additional feature. This service can be implemented at the user level using the point-to-point chapter of {\sc mpi} and the descriptor transmission facilities. These services can be deferred in this session of {\sc mpi}. \subsection*{Cache Facility} {\sc mpi} provides a ``cache'' facility that allows an application to attach arbitrary pieces of information, called {\em decorations}, to context and group descriptors. Decorations are local to the process and are not included if the descriptor is sent to another process. This facility is intended to support optimizations such as saving persistent communication handles and recording topology-based decisions by adaptive algorithms. {\sc mpi} provides the following services related to cacheing: \begin{description} \item [Generate key:] Generate cache key. \begin{verbatim} keyval = mpi_GetDecorationKey() \end{verbatim} \item [Store decoration:] Store decoration in cache by key. \begin{verbatim} mpi_SetDecoration(handle, keyval, decoration_val, decoration_destructor_routine) \end{verbatim} \item [Retrieve decoration:] Retrieve decoration from cache by key. \begin{verbatim} mpi_TestDecoration(handle,keyval,decoration) \end{verbatim} \item [Delete decoration:] Delete decoration from cache by key. \begin{verbatim} mpi_DeleteDecoration(handle,keyval) \end{verbatim} \end{description} Each decoration consists of a pointer or a value of the same size as a pointer, and would typically be a reference to a larger block of storage managed by the module. As an example, a global operation using cacheing to be more efficient for aall contexts of a group after the first call might look like this: {\small \begin{verbatim} static int gop_key_assigned = 0; /* 0 only on first entry */ static MPI_KEY_TYPE gop_key; /* key for this module's stuff */ efficient_global_op (context, ...) int context_handle; { struct gop_stuff_type *gop_stuff; /* whatever we need */ int group_handle = mpi_context_frame(context_handle); if (!gop_key_assigned) /* get a key on first call ever */ { gop_key_assigned = 1; if ( ! (gop_key = MPI_GetDecorationKey()) ) { MPI_abort ("Insufficient keys available"); } } if (MPI_TestDecoration (group_handle,gop_key,&gop_stuff)) { /* This module has executed in this group before. We will use the cached information */ } else { /* This is a group that we have not yet cached anything in. We will now do so. */ gop_stuff = /* malloc a gop_stuff_type */ /* ... fill in *gop_stuff with whatever we want ... */ MPI_SetDecoration (group_handle, gop_key, gop_stuff, gop_stuff_destructor); } /* ... use contents of *gop_stuff to do the global op ... */ } gop_stuff_destructor (gop_stuff) /* called by MPI on group delete */ struct gop_stuff_type *gop_stuff; { /* ... free storage pointed to by gop_stuff ... */ } \end{verbatim} } The cache facility could also be provided for process descriptors, but it is less clear how such provision would be useful. It is suggested that the cache store, retrieve and delete decoration procedures should fail when applied to a process descriptor handle. %---------------------------------------------------------------------- % Point-to-point \section{Point-to-Point Communication} This proposal recommends three forms for {\sc mpi} point-to-point message addressing and selection: null context; closed context; open context. It is further recommended that messages communicated in each form are distinguished such that a {\tt Send} operation of form X cannot match with a {\tt Receive} operation of form Y, requiring that form is embedded into the message envelope. The three forms are described, followed by considerations of uniform integration of these forms in the point-to-point communication chapter of {\sc mpi}. \subsection*{Null Context Form} The {\it null context\/} form contains no message context. Message selection and addressing are expressed by \begin{verbatim} (process, tag) \end{verbatim} where: {\tt process} is a process handle; {\tt tag} is a message tag. {\tt Send} supplies the {\tt process} of the receiver. {\tt Receive} supplies the {\tt process} of the sender. {\tt Receive} can wildcard on {\tt process} by supplying the wildcard descriptor handle value {\tt MPI\_WILDCARD}. In this case the receiver may have obtained the process descriptor of the sender, and the null descriptor handle {\tt MPI\_NULL} is returned in the relevant point-to-point enquiry procedure. \subsection*{Closed Context Form} The {\it closed context\/} form permits communication between members of the same context. Message selection and addressing are expressed by \begin{verbatim} (context, rank, tag) \end{verbatim} where: {\tt context} is a context handle; {\tt rank} is a process rank in the frame of {\tt context}; {\tt tag} is a message tag. The calling process must be a member of the frame of {\tt context}. {\tt Send} supplies the {\tt context} of the receiver (and sender), and the {\tt rank} of the receiver. {\tt Receive} supplies the {\tt context} of the sender (and receiver), and the rank of the sender. The {\tt (context, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process identifier of the receiver (sender). {\tt Receive} cannot wildcard on {\tt context}. {\tt Receive} can wildcard on {\tt rank} by supplying the wildcard integer {\tt MPI\_DONTCARE}. This proposal makes no statement about the provision for wildcard on {\tt tag}. \subsection*{Open Context Form} The {\it open context\/} form permits communication between members of any two contexts. Message selection and addressing are expressed by \begin{verbatim} (lcontext, rcontext, rank, tag) \end{verbatim} where: {\tt lcontext} is a context handle; {\tt rcontext} is a context handle; {\tt rank} is a process rank in the frame of {\tt rcontext}; {\tt tag} is a message tag. The calling process must be a member of the frame of {\tt lcontext} and need not be a member of the frame of {\tt rcontext}. {\tt Send} supplies the context of the sender in {\tt lcontext}, the context of the receiver in {\tt rcontext}, and the {\tt rank} of the receiver in the frame of {\tt rcontext}. {\tt Receive} supplies the context of the receiver in {\tt lcontext}, the context of the sender in {\tt rcontext}, and the {\tt rank} of the sender in the frame of {\tt rcontext}. The {\tt (rcontext, rank)} pair in {\tt Send} ({\tt Receive}) is sufficient to determine the process identifier of the receiver (sender). {\tt Receive} cannot wildcard on {\tt lcontext}. {\tt Receive} can wildcard on {\tt rcontext} by supplying the wildcard descriptor handle value {\tt MPI\_WILDCARD}, in which case it must also wildcard on {\tt rank} since the process descriptor of the sender cannot be determined. In this case the receiver may not have obtained the context descriptor of the sender, and the null descriptor handle {\tt MPI\_NULL} is returned in the relevant point-to-point enquiry procedure. {\tt Receive} can wildcard on {\tt rank} by supplying the wildard intger value {\tt MPI\_DONTCARE}. \subsection*{Uniform Integration} The three forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point chapter of {\sc mpi} by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with form. This is at the expense of multiplying the number of {\tt Send} and {\tt Receive} procedures by a factor of three, and some further but trivial work with details of the current point-to-point chapter which uniformly assumes a single addressing and selection form. There are various approaches to unification of the syntactic frameworks which may simplify integration. Two options are now described, each based on retention and extension of the framework of the closed and open contexts forms. The framework of the open context form could be adopted and extended. The null context form is expressed as {\tt (MPI\_NULL, MPI\_NULL, process, tag)}, which is a little clumsy. The closed context form is expressed as {\tt (MPI\_NULL, context, rank, tag)}, which is marginally inconvenient. The open context form is expressed as {\tt (lcontext, rcontext, rank, tag}), which is of course natural. The framework of the closed context form could be adopted and extended. The null context form is expressed as {\tt (MPI\_NULL, process, tag)}, which is marginally inconvenient, and requires that descriptor handles are expressed as intgers. The closed context form is expressed as {\tt (context, rank, tag)}, which is of course natural. Expression of the open context form requires a little more work. We can use the {\tt context} field as ``shorthand notation'' for the {\tt (lcontext, rcontext)} pair at the expense of introducing some trickery. We define a ``duplet descriptor'' which is formally composed of two references to contexts, and provide a procedure which constructs such a descriptor given two context descriptors. Both {\tt Send} and {\tt Receive} accept a duplet descriptor in {\tt context}, are able to distinguish the duplet descriptor from a singlet descriptor, and treat the duplet as shorthand notation. It is conjectured that using this framework is the best choice for {\sc mpi}. %---------------------------------------------------------------------- % Point-to-point \section{Collective Communication} Symmetric collective communication operations are compliant with the closed context form described above. This proposal recommends that such operations accept a context descriptor which identifies the context (thus frame) in which they are to operate. {\sc mpi} does plan to describe symmetric collective communication operations. It is not possible to determine whether this proposal is sufficient to allow implementation of the collective communication chapter of {\sc mpi} in terms of the point-to-point chapter of {\sc mpi} without loss of generality, since the collective operations are not yet defined. Asymmetric collective communication operations, especially those in which sender(s) and receiver(s) are distinct processes, are compliant with the open context form described above. This proposal recommends that such operations accept a pair of context descriptors (a duplet descriptor) which identify the contexts (thus frames) in which they are to operate. {\sc mpi} does not plan to describe asymmetric collective communication operations. Such operations are expressive when writing programs beyond the SPMD model, which are composed of communicative functionally distinct process groups. These services can be deferred in this session of {\tt mpi}. %---------------------------------------------------------------------- % Conclusion \section{Conclusion} This chapter presented a proposal for communication contexts and process groups with {\sc mpi}. In the proposal process groups are created dynamically and are static in membership. Associated with each process group are one or more communication contexts which inherit process ranking. The recommendations for point-to-point communication are powerful. The proposal provides process addressed communication which occurs within an extended context. The proposal also contains closed context communication addressed in terms of context and rank which protects messages belonging to one context from those belonging to other contexts. The proposal also contains open context communications adressed in terms of sender context, receiver context, and process rank which provides expressive power for intercommunication between modules within different groups. The proposal is extensible to a number of features which might be included in future sessions of {\sc MPI}, for example: dynamic processes; dynamic groups; multiple group collective communications. % % END "Proposal VII" %======================================================================% % % BEGIN "Proposal III" % Anthony Skjellum % March 1993 % \chapter{Contexts -- Proposal III} \begin{center} A.~Skjellum {\em et al.} \end{center} %---------------------------------------------------------------------- % Introduction \section{Introduction} This chapter takes a slightly different approach to contexts and groups, than does Proposal~VII. It is of roughly equal conceptual ``power'' as Proposal~VII, with some differences. As appropriate, this chapter borrows directly from Proposal~VII, by Clarke and Littlefield. \begin{enumerate} \item Contexts are supported to discriminate between messages in the system. A context is a conceptual extension of the tag space into a system-defined part (not wildcardable), and a totally user-defined part (the traditional 32-bit tag). \item A context is a lower-level concept than a group, so that contexts not associated with groups are permitted. This permits the user to develop codes that build on the server model, or which build up groups dynamically (not otherwise supported by MPI1). \item Groups are used to describe cooperative communication in the system. Groups have one or more context of communication associated with them. When created, a group is given a context of communication. \item Context and group descriptors can be explicitly transferred to processes that are not members of the context or group. \item In point-to-point messages, processes can be identified in either of two ways: by opaque process identifier, or by rank in a group; in either case, communication scope is within a given context. \item The cache facility, allowing groups to add additional information (described in Proposal~VII) is embraced by this Proposal, with reservations as noted. The possible need to omit this cacheing feature from MPI1 should not invalidate the remainder of Proposal~VII from further consideration (severability). \end{enumerate} %---------------------------------------------------------------------- % Processes \section{Processes} This Proposal views processes in the familiar way, as one thinks of processes in Unix or NX for example. Each process is a distinct space of instructions and data. Each process is allowed to compose multiple concurrent threads and {\sc MPI\/} does not distinguish such threads. {\sc MPI\/} shall be thread-aware, but not thread supporting, so we make every attempt to make thread safe programs possible in defining what follows. \subsection*{Process Identifier} Each process is identified by an opaque process identifier, which is associable to a {\it process descriptor} of undefined size and opaque structure, through {\sc MPI} accessor calls. In a static process model process identifiers can be obtained by mapping from a group and rank. In a future extension for dynamic processes, identifiers may be returned by process creation functions. {\sc MPI} provides a procedure that returns an identifier for the calling process. \begin{verbatim} my_process = mpi_my_process() \end{verbatim} This identifier can be converted to a transmittable form by {\sc MPI} converter functions, though opaque, it is conceptually a pointer. \subsection*{Process Creation \& Destruction} This proposal makes no statements regarding creation and destruction of processes. {\sc MPI} provides facilities for identifier transmission allowing the user explicitly to transfer a process identifier from one process to another. These facilities are described below. MPI also provides means to transfer underlying information about the opaque process descriptor underlying. %---------------------------------------------------------------------- % Groups \section{Process Groups} This proposal views a process group as an ordered collection of distinct processes (via process identifiers), the membership and ordering of which does not change over the lifetime of the group. The canonical representation of a group is a one-to-one map from the integers $(0, 1, \ldots, N-1)$ to identifiers of the $N$ processes composing the group. There may be structure associated with a process group defined by a process topology. This proposal makes no further statements regarding such structures. There may be non-enumerative ways to construct and manipulate special groups, or for special machine architectures ({\em e.g.}, cohorts). This proposal makes no further statements about such special groups, other than the desirability of avoiding group-name enumeration, when possible. \subsection*{Group Identifier} Each group is identified by an opaque {\it group identifier}, which is a associable to a {\it group descriptor} of undefined size and opaque structure, through {\sc MPI} accessor functions. The initialization of {\sc MPI} makes each process a member of the ``initial'' group. {\sc MPI} provides a procedure that returns an identifier to this group. \begin{verbatim} group_ident = mpi_initial_group() \end{verbatim} {\sc MPI} provides facilities for descriptor transmission allowing the user explicitly transfer a group descriptor from one process to another. \subsection*{Group Creation and Deletion} {\sc MPI} provides facilities which allow users dynamically to create and delete process groups. The procedures described here generate groups which are static in membership. {\sc mpi} provides a procedure that allows users to create one or more groups which are subsets of existing groups. \begin{verbatim} new_group = mpi_group_partition(old_group, key) \end{verbatim} This procedure creates one or more new groups {\tt new\_group} which are distinct subsets of an existing group {\tt old\_group} according to the supplied values of {\tt key}. This procedure is called by and synchronises all members of {\tt new\_group}. No overlapping is permitted, so that exactly one {\tt new\_group} is achieved in each {\tt old\_group} process. The new groups have new contexts of communication. The number of new contexts depends on the number of different key values asserted. {\sc MPI} provides a procedure which allows users to create a group by permutation of an existing group. \begin{verbatim} new_group = mpi_group_permutation(old_group, rank) \end{verbatim} This procedure creates one new group with the same membership as {\tt old\_group} with a permutation of process ranking, and returns the created group descriptor in {\tt new\_group}. It is called by and synchronises all members of {\tt gha}. The new group has a new context of communication. {\sc MPI} provides a procedure that allows users to create a group by explicit definition of its membership as a list of process identifiers. \begin{verbatim} new_group = mpi_group_definition(array_of_process_ids,length, context_to_use) \end{verbatim} This procedure creates one new group {\tt new\_group} with membership and ordering described by the process handle list {\tt array\_of\_process\_ids}, of length {\tt length}. If {\tt context\_to\_use} is specified as the special context {\tt MPI\_GET\_CONTEXT}, then the system allocates the new context. Else, the system trusts the user to have allocated the context legally (see below). This procedure must be called by and synchronises all processes identified in the list. A further approach to new group definition is as follows \begin{verbatim} new_group = mpi_group_def_by_leader(leader_id, in_length, in_array_of_process_ids, context_to_use, out_array_of_process_ids, out_length) \end{verbatim} This weaker form requires all future group members to have identified only a leader (specified by {\tt leader\_id}). The leader knows the length a names of all participants. This is a synchronization of all participants. Same semantics for {\tt context\_to\_use} as above. {\sl MPI} provides a way formally to duplicate a group, in order to obtain a separate context of communication. This can be achieved by using other operations, but this procedure allows optimization in some implementations (no explicit group copy, for instance). \begin{verbatim} new_group = mpi_group_duplicate(old_group) \end{verbatim} The only difference between the old and new groups is that there is a new context of communication. {\sc MPI} provides a procedure which allows users to delete user created groups. \begin{verbatim} mpi_group_deletion(group) \end{verbatim} This procedure deletes an existing group {\tt group}. It is called by and synchronises all members of {\tt group}. {\sc MPI} may provide additional procedures which allow users to construct process groups with a process group topology. \subsection*{Group Attributes/Accessors} {\sc MPI} provides a procedure that accepts a valid group identifier and returns the rank of the calling process within the identified group. \begin{verbatim} rank = mpi_group_rank(group) \end{verbatim} {\sc MPI} provides a procedure that accepts a valid group identifier and returns the number of members, or {\it size}, of the identified group. \begin{verbatim} size = mpi_group_size(group) \end{verbatim} {\sc MPI} provides a procedure that accepts a valid group identifier {\it rank-in-group}, and returns the valid process identifier to which the supplied rank maps within the identified group. \begin{verbatim} process_id = mpi_group_process(group, rank) \end{verbatim} {\sc MPI} may provide additional procedures which allow users to determine the process group topology attributes. {\sc MPI} provides a group descriptor cache facility thath allows the user to attach attributes to group descriptors. See Proposal~VII for details. %---------------------------------------------------------------------- % Contexts \section{Communication Contexts} This proposal views a communication context as a partition of the tag space, which is a protection mechanism that avoids collision between messages sent between processes. Process groups have one or more contexts in {\sl MPI}. Unlike Proposal~VII, more contexts are obtained for a group using the above-discussed group creation and replication functions. Replication may only be formal for good implementations. \subsubsection*{Context Identifier} Each context is identified by an opaque process identifier. It is conceptually an integer assigned by the system to partition a large tag space into a user-defined and system-controlled subspaces. This stategy provides the minimal level of isolation needed to build large libraries, and is close to practice. \subsubsection*{Context Creation and Deletion} {\sc MPI} provides facilities that allow user dynamically to allocate and free contexts. When contexts are used with groups, these calls are not needed. For more advanced users (such as building your own dynamic groups), these calls will be used. Above, where {\tt context\_to\_use} appears as an argument, the following call would have been used to secure such a context in advance. \begin{verbatim} mpi_context_creation(number_of_contexts_wanted,array_of_contexts, number_of_contexts_provided) \end{verbatim} This call is called by any process, with no synchronization to other processes. {\sc MPI} provides a procedure that allows users to delete user- created contexts. The procedure accepts a context identifier array, containing zero or more contexts created previously in the system. \begin{verbatim} mpi_context_deletion(context_array,length) \end{verbatim} No synchronization occurs here. The user can do erroneous things by freeing contexts that are still in use. For general applications, it may be nice to have a name service for contexts (necessary for building dynamic groups and servers, for yourself). Herewith: \begin{verbatim} mpi_associate_contexts_with_name(string_name,context_array,length) mpi_disassociate_contexts_with_name(string_name) mpi_get_contexts_by_name(string_name,max_length,out_length, context_array \end{verbatim} As with context generation, the above calls assume a simple reactive, global server, or shared name space mechanism (both achieveable easily in practice). %---------------------------------------------------------------------- % Descriptors \section{Descriptor Facilities} This section describes the descriptor transmission and user cache facilities. \subsection*{Conversion Facility} {\sc MPI} provides a mechanism whereby the user can convert a valid descriptor ({\em e.g.}, a group descriptor) idenfied through an identifier ({\em e.g.}, a group identifier) for use in a message such that the received descriptor can be reconstructed on the remote end. This can be integrated with message transmission as the user sees fit, without additional complication to the send/receive semantics of {\sc MPI}. An example follows: \begin{verbatim} error = mpi_group_group_transmit(group, group_buffer, max_length, act_length) \end{verbatim} If the buffer is not long enough to hold the information, an error occurs. A network independent format can be assumed in the {\tt group\_buffer}. Cached ``attributes'' are not transmitted (see below). \subsection*{Cache Facility} {\sc MPI} provides a ``cache'' facility that allows an application to attach arbitrary pieces of information, called {\em attributes}, to context and group descriptors. Attributes are local to the process and are not included if the descriptor is sent to another process. This facility is intended to support optimizations such as saving persistent communication handles and recording topology-based decisions by adaptive algorithms. {\sc MPI} provides the following services related to cacheing. We call our attributes `attributes'; Proposal~VII calls them (equivalently) decorations (no big difference, except naming, is anticipated). \begin{description} \item [Generate key:] Generate cache key. \begin{verbatim} keyval = mpi_get_attribute_key() \end{verbatim} \item [Store attribute:] Store attribute in cache by key. \begin{verbatim} mpi_set_attribute(handle, keyval, attribute_val, attribute_destructor_routine) \end{verbatim} \item [Retrieve attribute:] Retrieve attribute from cache by key. \begin{verbatim} mpi_Test_Attribute(handle,keyval,attribute) \end{verbatim} \item [Delete attribute:] Delete attribute from cache by key. \begin{verbatim} mpi_delete_attribute(handle,keyval) \end{verbatim} \end{description} Each attribute consists of a pointer or a value of the same size as a pointer, and would typically be a reference to a larger block of storage managed by the module. Our example will appear in a later draft, because we have semantic differences from some of the ancillary aspects of the example of Proposal~VII. The cache facility could also be provided for process identifiers, but it is less clear how such provision would be useful. It is suggested that the cache store, retrieve and delete attribute procedures should fail when applied to a process identifiers. Implementations should use AVL trees, or similar efficient data structures to provide relatively efficient access to attributes. %---------------------------------------------------------------------- % Point-to-point \section{Point-to-Point Communication} This proposal recommends two forms for {\sc MPI} point-to-point message addressing and selection: by-group notation, by process-ID notation; always, one is working in a context, as this is the fundamental management tactic of {\sc MPI} for messages. As a group always has a context at creation, and an ALL group is anticipated, this should prove fine for a static process model. We disagree significantly from Proposal~VII in what follows. The two forms are described, followed by considerations of uniform integration of these forms in the point-to-point communication chapter of {\sc MPI}. \subsection*{Group-Rank Form} The {\it group-rank\/} form permits communication between members of the same context and group. Message selection and addressing are expressed by \begin{verbatim} (group, rank, tag) \end{verbatim} where: {\tt group} is a group identifier; {\tt rank} is a process rank in that group; {\tt tag} is a message tag. The calling process must be a member of the frame of {\tt context}. {\tt Send} determines the context using information in the group identifier. It does all necessary mappings to the process identifier space. {\tt Receive} cannot wildcard on context, so a valid matching receive must refer to the same group information. {\tt Receive} can wildcard on {\tt rank} by supplying the wildcard integer {\tt MPI\_DONTCARE}. This proposal makes the following statement about the provision for wildcard on {\tt tag}. Two integer-form wildcard is needed for layering: {\tt care\_bits}, {\tt dont\_care\_bits}. Tags are matched if and only if \begin{equation} (received\_tag AND NOT dont\_care_bits) XOR care\_bits== 0 \end{equation} This general format can be used to partition the tag space for virtual topologies or other user-defined needs, and is quite important to the standard's flexibility. \subsection*{Process-Identifier Form} Communication takes place using the following parameters: \begin{verbatim} (context, process_identifier, tag) \end{verbatim} where: {\tt context} is a context identifier, {\tt process\_identifier} is a process identifier, {\tt tag} is a message tag. The calling process must be a member of the same context as the recipient. There is no reference to groups here. Contexts can have been shared by using a least common ancestor prior to this call, or by the above-mentioned context naming service. There is never wildcarding on context. Wildcarding on {\tt process\_identifier} is through {\tt MPI\_DONTCARE}. Tag wildcarding is through the integer pair described above. \subsection*{Uniform Integration} The two forms of addressing and selection described have different syntactic frameworks. We can consider integrating these forms into the point-to-point chapter of {\sc MPI} by defining a further orthogonal axis (as in the multi-level proposal of Gropp \& Lusk) which deals with form. This is at the expense of multiplying the number of {\tt Send} and {\tt Receive} procedures by a factor of two, and some further but trivial work with details of the current point-to-point chapter which uniformly assumes a single addressing and selection form. No further details, other than naming that disambiguates the rank-group form from the process-id-context form is really needed, and the naming would seem uncontroversial. %---------------------------------------------------------------------- % Collective \section{Collective Communication} Symmetric collective communication operations are compliant with the group-rank form described above. This proposal recommends that such operations accept a group-identifier (which contains context and other information) needed to operate correctly. We recommend that the tag argument be included in collective calls where this could help with debugging. {\sc MPI} does plan to describe symmetric collective communication operations. It is impossible to determine whether this proposal is sufficient to allow implementation of the collective communication chapter of {\sc MPI} in terms of the point-to-point chapter of {\sc mpi} without loss of generality, since the collective operations are not yet defined. Asymmetric collective communication operations, especially those in which sender(s) and receiver(s) are distinct processes, should be made compliant with the group-rank form described above. {\sc MPI1} should forego non-blocking collective operations, but ask vendors to support thread models in lieu of such operations. %---------------------------------------------------------------------- % Conclusion \section{Conclusion} This proposal is substantially different than either Proposal~VII or I. Contexts are integer tag partitions here, and are fundamentally lower-level objects than groups. Groups have contexts in this proposal, but not all operations require the presence of group-scope. To avoid invidious comparisons here, more substantial comparisons of all three proposals are deferred the following appendix. \appendix \chapter{Summary of context subcommittee proposals} The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. \section{Common Features} \subsection{Process group management} In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. In each proposal there is a defined group containing all (or perhaps all initial) processes. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. \subsection{Provision for point-to-point communication within group} In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. \subsection{Provision for collective communication within group} In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. \subsection{Opacity of group and process description} In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. \subsection{Fields of point-to-point communication} In each proposal point-to-point communication accepts three fields, inclusive of message tag, in addressing and selection. \section{Separable Features} \subsection{Tag usage in point-to-point communication} Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. \subsection{Tag usage in collective communication} Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. \subsection{Context or Group cache} Proposal VII describes a "cache" facility associated with contexts and groups. Proposal III describes a similar "cache" facility associated with groups. This feature can be placed in all Proposals I, III and VII. \subsection{Opaque object (descriptor) transmission} Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. \subsection{Context registry} Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. \section{Concept Differences} \subsection{Concept of CONTEXT and GROUP} In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept than GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. \subsection{Scope of point-to-point communication} In Proposal I the scope of point-to-point communication is limited to the group. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communication is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. \subsection{Transmission of group or context} In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) \section{Detail differences} \subsection{Manifestation of context} In Proposals I and VII context is an opaque object. In Proposal III context is an integer. \subsection{Deletion of group} In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). \subsection{Duplication of group} In Proposals I and III there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). \subsection{Global shared variables} Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) \subsection{Process identifier addressed communication} Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. \subsection{Inter-group communication} Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in a triplet addressing form: sender (receiver) group, receiver (sender) group, sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. \end{document} From owner-mpi-context@CS.UTK.EDU Fri Mar 26 13:11:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19032; Fri, 26 Mar 93 13:11:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02142; Fri, 26 Mar 93 13:11:03 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 26 Mar 1993 13:11:01 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02125; Fri, 26 Mar 93 13:10:58 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA01869; Fri, 26 Mar 93 18:10:53 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05020; Fri, 26 Mar 93 11:09:35 MST Date: Fri, 26 Mar 93 11:09:35 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303261809.AA05020@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: Re: A new proposal for context and groups. Hi all, A while back, Mark Sears wrote the beginnings of a "Proposal VI": > Proposal VI > Mark P. Sears > Sandia National Laboratories > > The following proposal for context and group definitions is > intended to fill a gap in Tony Skjellum's list. It is most > closely related to Rik Littlefield's proposal V. The main > difference is that in my proposal, context and groups are > completely orthogonal, both in purpose and function... > [remainder deleted...] Any particular reason why this proposal hasn't been pushed any farther? (Maybe no one has time to do it?? I'm not volunteering! :-) I think "Proposal VI" would fit into Lyndon's summary like this: > In Proposal I CONTEXT and GROUP are identical concepts and are not > distinguished. > > In Proposal III CONTEXT is a lower degree concept that GROUP. The > GROUP concept inherits aspects of the CONTEXT concept. > > In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT > concept inherits aspects of the GROUP concept. In Proposal VI CONTEXT and GROUP are othogonal and unrelated. Is this worth pursuing? Tom (I'm STILL reading the proposals... :-) Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-context@CS.UTK.EDU Fri Mar 26 16:43:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25014; Fri, 26 Mar 93 16:43:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12680; Fri, 26 Mar 93 16:41:48 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 26 Mar 1993 16:41:47 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12672; Fri, 26 Mar 93 16:41:46 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA12136; Fri, 26 Mar 93 15:35:39 CST Date: Fri, 26 Mar 93 15:35:39 CST From: Tony Skjellum Message-Id: <9303262135.AA12136@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, hender@macaw.fsl.noaa.gov Subject: CONTEXT agenda @ Dallas; Sears proposal Tom! Concerning Sears proposal (Proposal VIII by my nomenaclature). I have not had time to read this proposal yet. I except any responsiblity for not addressing it. I will read it prior to meeting, and give Mark Sears' proposal time during our 3-hour slot (see below), if he wishes. If it is a workable closure for MPI, I would consider adding it to straw poll, of course. I appreciate Mark's efforts, and I do not mean to snub him! Schedule (we initiate the meeting on Wednesday) 3hr context discussion before committee of the whole... Assume we start at 1:30+epsilon: intro to context sub-committee proposals - Skjellum / 10 mins Proposal VII - Littlefield & Clarke presents / 30 mins discussion on VII - lead by Littlefield & Clarke / 15 mins Proposal I - Snir / 30 mins discussion on I - lead by Snir / 15 mins Proposal III & VIII - Skjellum presents / 30 mins discussion on III & VIII - lead by Skjellum & Sears / 15 mins Overall discussion / Recommendations / Ranking poll 35 mins - Tony From owner-mpi-context@CS.UTK.EDU Mon Mar 29 13:08:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19903; Mon, 29 Mar 93 13:08:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05771; Mon, 29 Mar 93 13:07:54 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 29 Mar 1993 13:07:53 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05763; Mon, 29 Mar 93 13:07:50 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA02203; Mon, 29 Mar 93 18:07:46 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00768; Mon, 29 Mar 93 11:06:27 MST Date: Mon, 29 Mar 93 11:06:27 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303291806.AA00768@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: My (Preliminary) Vote Hi all, I FINALLY finished reading the context proposals. Right now I am leaning towards "Proposal VIII" (Mark Sears' proposal for orthogonal context and group concepts). However, I would like to see more detail in that proposal. Of course, the discussion on Wednesday may change my opinions! :-) See you all then... Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-context@CS.UTK.EDU Mon Mar 29 13:24:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20278; Mon, 29 Mar 93 13:24:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06707; Mon, 29 Mar 93 13:23:33 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 29 Mar 1993 13:23:32 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06699; Mon, 29 Mar 93 13:23:23 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA11427 (5.65c/IDA-1.4.4 for mpi-context@cs.utk.edu); Mon, 29 Mar 1993 19:23:11 +0100 Date: Mon, 29 Mar 1993 19:23:11 +0100 From: James Cownie Message-Id: <199303291823.AA11427@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02335; Mon, 29 Mar 93 19:19:34 BST To: hender@macaw.fsl.noaa.gov Cc: mpi-context@cs.utk.edu In-Reply-To: Tom Henderson's message of Mon, 29 Mar 93 11:06:27 MST <9303291806.AA00768@macaw.fsl.noaa.gov> Subject: My (Preliminary) Vote Content-Length: 621 It woudl be really nice if the could be kept separate. Unfortunately they interact because 1) the context needs to be agreed by a set of processors, so context creation is naturally a group operation 2) if collective operations are built on point to point they need a context to remove the ambiguity, and to work safely. See you on Dallas -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-context@CS.UTK.EDU Mon Mar 29 13:34:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20599; Mon, 29 Mar 93 13:34:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07253; Mon, 29 Mar 93 13:33:35 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 29 Mar 1993 13:33:34 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07243; Mon, 29 Mar 93 13:33:32 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA14097; Mon, 29 Mar 93 12:27:00 CST Date: Mon, 29 Mar 93 12:27:00 CST From: Tony Skjellum Message-Id: <9303291827.AA14097@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: straw poll I invite straw poll votes on MPI context proposals. Note that Lyndon has introduced proposal X, but I recommend we defer discussion on that till Wednesday night. - TOny From owner-mpi-context@CS.UTK.EDU Mon Mar 29 13:57:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21095; Mon, 29 Mar 93 13:57:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08541; Mon, 29 Mar 93 13:57:27 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 29 Mar 1993 13:57:26 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08530; Mon, 29 Mar 93 13:57:24 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA02384; Mon, 29 Mar 93 18:57:20 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00904; Mon, 29 Mar 93 11:56:01 MST Date: Mon, 29 Mar 93 11:56:01 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303291856.AA00904@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: Re: My (Preliminary) Vote > Mark;s is essentially the same as mine, except that he has weakened > a few things. It is so vague (Clintonlike) that perhaps it has to get > elected :-) > - Tony Unfortunately, I agree that this proposal is vague. Maybe the reason I like it is that I'm confusing vagueness with simplicity... :-) {I'm trying to think which "promises" might be broken within the first week...} I've made a few suggestions for "de-vagueifying" to Mark. I hope a more detailed proposal could be available by Wednesday (i.e. I hope Mark doesn't have any "real" work to do before then...). Tom From owner-mpi-context@CS.UTK.EDU Mon Mar 29 18:27:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27367; Mon, 29 Mar 93 18:27:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21729; Mon, 29 Mar 93 18:22:30 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 29 Mar 1993 18:22:29 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21715; Mon, 29 Mar 93 18:22:26 -0500 Received: from newton.sandia.gov (newton.cs.sandia.gov) by cs.sandia.gov (4.1/SMI-4.1) id AA12159; Mon, 29 Mar 93 16:22:23 MST Received: by newton.sandia.gov (5.57/Ultrix3.0-C) id AA05331; Mon, 29 Mar 93 16:23:17 -0700 Date: Mon, 29 Mar 93 16:23:17 -0700 From: mpsears@newton.cs.sandia.gov (Mark P. Sears) Message-Id: <9303292323.AA05331@newton.sandia.gov> To: mpi-context@cs.utk.edu Subject: Response to comments on Pr. VIII Tom (and other colleagues in MPI), Thanks for your interest and let me try to answer some of your questions, according to my point of view. I apologize for any lack of clarity in the proposal (I had the flu when I wrote it). My proposal was certainly not at the level of detail needed for insertion in the MPI document -- it was intended more as a position statement. In the following, my comments begin with **** in response to Tom's statements and questions. I end with some responses to Jim Cownie's comments. Mark P. Sears Sandia National Laboratories ********************************************************************** (from Tom Henderson's email:) 1) Process ID's are globally unique integers. I see this as a minor advantage. When PID is an opaque data type, we get the "identity" problem: how do I tell if PID A is the same as PID B? I need to use a special routine (like MPI_RANK() in Proposal I). **** I agree with this. **** In addition, our current model of a parallel task in MPI is rather **** static: all the processes in the task are created and destroyed **** simultaneously and all can communicate with one another. So why not **** just number them? That is current common practice anyway. The numbering **** encodes the system's idea of the best process ordering. Opaque id's **** promote a sense that any ordering is just as good as any other, which **** is not the case for high performance applications. 2) Contexts are globally unique integers. Global uniqueness makes context creation "slow". Given the intended use of contexts, I do not see this as a problem. Your current proposal does not explain how contexts are "created" in much detail. Here's three (non-exclusive) possibilities: A) make_group_contexts(group, num_contexts, new_contexts) num_contexts new contexts are placed in array new_contexts. This routine must be called "loosely-synchronously" in every process in the list of processes called "group". This is a convenient way for a user to create one or more contexts. It does not imply any relationship between groups and contexts. B) get_registered_contexts(context_name, num_contexts, new_contexts) This is a "name-server" version. If one or more contexts are registered under the name "context_name", then the first num_contexts contexts are copied into array new_contexts. If "context_name" is not currently registered, allocate num_contexts new contexts and register them under name "context_name". Some folks will not need groups. There must be a way to allow access to contexts without using groups. (However, see #3 below...) C) make_list_contexts(list_of_PIDs, num_contexts, new_contexts) This could easily be (A). The only reason for having this version would be if "group" has other stuff "cached" with it. This one could be left out if "group" is just a "list of PIDs". **** Here I had in mind a hardware or low level model of context, just **** implemented as a few bits in what is now the tag field of the **** message envelope. From this point of view, contexts are a scarce **** resource, but this matches well with the idea that contexts are **** associated with different software components of a program - most **** programs use only a handful of independent software components **** (the main app, a few libraries). My model of how the underlying **** implementation uses context is that a message (or part thereof) **** arrives at a process and must then be queued or matched with an **** outstanding receive. If there are not many possible context values then **** the number of possible queues is not large either, and all of the **** resources needed to support all context values can be preallocated **** in process startup. For 16 or 256 context values we can easily imagine **** this not taking very long, order milliseconds. Then context values can **** be used like tag values -- they preexist and can be utilized and allocated **** according to any desired scheme. If we must specify an allocation scheme, **** then I would recommend something like **** **** context_value = MPI_get_context() **** **** where the routine must be called synchronously on every processor. **** This routine would typically be called a few times by the various **** initialization code for the various software components. **** The bottom line is that contexts preexist and a very simple (or no) **** allocation/deallocation scheme is needed. 3) I did not completely understand this: > 5) This proposal requires no server code and most of the > group code is not even parallel. I did a test implementation of > groups as defined here and was able to build code for identity > groups, permutation groups, linear and bilinear groups, general > set groups (Tony's favorite), composition groups, Cartesian products > and cartesian subsets (my favorite), all in about 700 lines of code. > The group definition really lends itself to object-oriented user > extensibility, like X widgets but easier. Are you talking about groups only here? If not, how are you proposing to maintain globally synchronous allocation/deallocation of contexts? **** Yes, groups only. See above comments on context. 4) Minor questions: > 6) Groups are easily constructed and destroyed, since no > global communication is required. Dynamic groups are not excluded, > although they must be used carefully. Since groups have no associated > context, there are no resources limiting their construction other than > memory and CPU time. You say "global" communication is not required. I assume that communication among the members of a created group is required? (Seems obvious...) **** There is no magic, of course. Sometimes, group creation requires **** communication and sometimes not. Many interesting groups can be **** computed rather than communicated, for example the row and column **** subgroups of a group representing a rectangle. Other groups require **** communication in order for each process to know whether it belongs **** to a group or not, the random list group being a good example, but I **** see that communication as independent of the construction of the group. **** That is, you communicate the list of processes wherever needed, then **** each process that needs the group builds it from the list. 5) Clarification: > There is no reason to disallow user-defined groups (e.g. dynamic > groups). The term "dynamic group" has been used to mean different things in the mpi-context discussion. Are you talking about a user-created group whose membership does not change? Are you talking about a user-created group whose membership can change after creation? Are you talking about something else? **** A group is just a mapping or function. This function could be **** one predefined by MPI or one defined by a (knowlegeable) user. MPI could **** restrict itself to a few simple kinds of groups and allow the user to **** implement code for groups whose membership changes (what I meant by **** dynamic groups) or groups with properties we haven't thought of yet. 6) My understanding of this proposal is that "group" is an array of integers, probably including range(group). You mention that other information could be cached with "group". Are you proposing a "handle" or "opaque object"? **** A group is not explictly an array. Most groups in my view (this is **** biased by the kind of work I do, of course) will be computed rather **** than given as explicit lists. So a group is a union of possible **** structures. Each kind of structure knows how to compute the **** needed functions (element(group, rank), rank(group, element, etc) -- all **** very straightforward with modern C programming techniques. Maybe **** that computation uses a list, maybe it doesn't. **** What the user gets from a group creation operation **** is a pointer to the object defining the group. The pointer is of **** course an opaque type valid only on the process that created that **** group. **** The kind of information likely to be cached with a group is probably **** a spanning tree or set of exchanges which are optimized for the **** particular group. Alternatively it is possible to cache optimized methods **** for operations such as broadcast or synchronization. 7) How are you proposing groups be created? It would be nice to see a few proposed routines. (Or, "it's done just like proposal XXX.) **** I hope you don't want code! (I will mail it to anyone that asks). **** Let me begin with defining a classification of groups. Each class **** here is defined in terms of the function element(group, rank). Each **** type of group will have its own constructor with the parameters **** needed for that type. The output of the constructor is a pointer **** to the actual group object. **** **** Class: element(rank) parameters: **** ------ ------------- ----------- **** Identity rank. order **** Permutation p(rank) for some permutation p. order, p **** List list(rank) order, list **** Linear start + rank*delta order, start, delta **** Bilinear start + r1 * d1 + r2 * d2 order, n1, d1, d2 **** where rank = r1 + n1 * r2 **** Composition g1(g2(rank)) (group) g1, g2 **** Product g1(r1) + range(g1)*g2(r2) (group) g1, g2 **** So the constructor for a linear group would be defined like **** **** Group MPI_make_linear_group(order, start, delta), **** **** which returns a pointer to the actual group structure. Each **** class of groups would have a specific constructor. **** **** The functions common to all groups would be defined like **** **** int MPI_group_order(Group g) **** int MPI_group_range(Group g) **** int MPI_group_iselement(Group g, int element) **** int MPI_group_rank(Group g, int element) **** int MPI_group_element(Group g, int rank) **** **** In addition in my test implementation I defined some other **** stuff like a destructor, copy constructor, etc. **** Although I have made heavy use of C pointers here, a Fortran **** binding is not too hard to construct. 8) Is this what you had in mind? (My understanding of some of the things you've said...): * All point-to-point communication uses (PID, context, tag). **** Yes. * All collective communication uses (group, context, tag) (with the caveat that tag may be eliminated by the collcomm committee). **** Here the committee could eliminate tag and context together but not **** independently, I think. How could I call such a routine not knowing **** which tags it was going to use for the operation? If neither is **** available then the MPI routines must use an internal context and **** set of tags. * Third-party library routines that use collective communication must have (group, context) passed in as arguments. * Third-party library routines that use point-to-point communication within a group must have (group, context) passed in as arguments. (PID's are then extracted inside the routine using element(), etc.) * Third-party library routines that use point-to-point communication without reference to groups must have (PID's, context) passed in as arguments. **** Not quite. **** In my view there are two categories of third party routines: one **** category which establishes and uses its own context and tag spaces **** and a second category which uses the context and tag spaces given **** to it. Both kinds are useful -- MPI must be careful not to preclude **** one or the other. * Group-based point-to-point calls will look like: send(element(group,rank), context, tag, ...) recv(element(group,rank), context, tag, ...) **** Yes. * Default group and default context are provided. When used, these look like: send(element(group,rank), MPI_DEF_CONTEXT, tag, ...) barrier(MPI_DEF_GROUP, MPI_DEF_CONTEXT) **** Yes. Default context (free-for-all) especially is intended for **** the poor hapless end-user slob (I wear this hat occasionally) **** who hasn't got the time to understand all our machinations. :-) **** Default group is a trivial group (what I would call an identity **** group) which maps rank into itself, assuming PID == rank. 9) The examples above imply that context appears in all communication routines. There could be special versions of all communication routines that use the default application context. I'd prefer to avoid this. No one has proposed special versions of the collective communication routines that use the default group! **** Hear, hear. How many routines do we need anyway? Delete my **** suggestion. 10) If I've got the points above right, inter-group communication seems to be no problem with this proposal. Everything is converted into a globally unique PID. **** Right. 11) Clarification: > 2) Processes are addressed by their rank within the parallel task. > This global rank is fixed and assigned in an implementation-defined way. What do you mean by "parallel task"? Is there only one? Is this extensible to "more than one"? (Seems like it could be as long as "global rank" == PID is globally unique for any process.) A bit more detail on how this would extend would be nice. > [ This is an area where MPI 2.0 could really break some new ground: > define interaction between different parallel tasks, define creation > and deletion of parallel tasks, ...] I think "one" parallel task in MPI 1.0 is definitely a good idea. :-) **** There are several options: **** 1) There exists one parallel task (collection of processes). **** In this case processes can be addressed by rank within the task. **** 2) There exists more than one parallel task. Now we open up **** a Pandora's box of options: can a process belong to more than **** one parallel task (yech), can groups cross task boundaries **** (yech**2). If the answer to both these questions is no, then **** I would support modifying everything so that processes are **** addressed by (parallel-task-id, rank). But this significantly **** extends our work. **** I had a hard time with this one -- am pulled both ways. 12) Suggestion: > 4) Groups have an implicit topology, defined by the ordering of the > elements. Any other ordering can be defined by constructing a new > group with the same elements in the new order. There is no need for > any other topology function. It seems to me that having standard routines that produce "good" orderings for a few common logical topologies is a good idea. > There must exist environment functions which obtain the topology of > the global process assignment (hypercube, mesh, random network, ...) I think that this is more difficult than specifying a few "logical topology" ordering routines. Arguments? :-) **** What you mean by a logical topology ordering routine is (correct **** me if I misinterpret) something that selects, for example a Gray **** ordering or natural ordering for a group. The choice depends on **** the underlying topology and the application and cannot be done **** automatically. Suppose you have constructed a group and want to **** impose Gray ordering on it. In this proposal you would compose **** your group with a permutation group (where the permutation was **** the Gray ordering), creating thus a new group with the required **** order. **** The implementation knows what the global topology is. A routine **** like the following could be easily implemented (I think) to return **** this information: **** **** char * MPI_global_topology() **** **** which in various implementations could return a string containing **** something like **** **** "N 564" -- a random network with 564 processes. **** "H 5" -- a 5 dimensional hypercube. **** "R 16 13" -- a 16 by 13 mesh. **** **** or generally **** "topology-class parameter ..." **** **** I don't anticipate very many topology classes. (from James Cownie's email:) It woudl be really nice if the could be kept separate. Unfortunately they interact because 1) the context needs to be agreed by a set of processors, so context creation is naturally a group operation **** In this proposal contexts are global quantities agreed to by **** all processes. Also, contexts are a scarce and therefore limited **** resource. I see no problem in simply creating them all at process **** startup. **** If we insist that contexts do need to be "created" before being **** used, and if we accept the model (in this proposal) that a context **** is just an integer, then there is no problem with implementing **** a routine **** context = MPI_get_context() **** which tells MPI internally that this process is now willing to **** send and receive messages with the specified context value. Such **** a routine could be implemented with other point-to-point routines **** which used one preexisting context (MPI_INTERNAL_CONTEXT). 2) if collective operations are built on point to point they need a context to remove the ambiguity, and to work safely. **** I agree, and my proposal states this. I would feel badly about **** any proposal where collective communications could not be built **** on top of point to point. (the end) From owner-mpi-context@CS.UTK.EDU Tue Mar 30 11:36:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14063; Tue, 30 Mar 93 11:36:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12397; Tue, 30 Mar 93 11:36:11 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 30 Mar 1993 11:36:09 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12386; Tue, 30 Mar 93 11:36:07 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04528; Tue, 30 Mar 93 16:35:59 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA02355; Tue, 30 Mar 93 09:34:39 MST Date: Tue, 30 Mar 93 09:34:39 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303301634.AA02355@macaw.fsl.noaa.gov> To: mpi-context@cs.utk.edu Subject: A few more ramblings Cc: mpsears@newton.cs.sandia.gov Mark, I had a few more thoughts about Proposal VIII... One of the arguments that was unresolved at the last meeting was whether contexts should be a "scarce" or "plentiful" resource. One concern was a situation like one we have dealt with here. One of my co-workers (Bernardo Rodriguez) has developed a high-level library for finite difference approximation (FDA) problems (common with the weather forecast models we are working with). One well-known strategy for getting decent performance from a data-parallel FDA is to compute boundary values, start non-blocking boundary exchange, compute interior values while exchange is happening, and complete the exchange. An example of how this code looks with contexts is shown below. Routines that start with "LIB" are our "third-party" library routines. Routines written by a user of the library start with "USER". Array "A" contains the data. "myContext" is the context used by the LIB_XXX() routines. USER_COMPUTE_BOUNDARY(A, ...) LIB_START_EXCHANGE(A, myContext) USER_COMPUTE_BOUNDARY(A, ...) LIB_END_EXCHANGE(A, myContext) Unfortunately, weather models have lots and lots of arrays (A, B, C, ...). If we use the same idea we need contexts "myContextA", "myContextB", ... The code looks like: USER_COMPUTE_BOUNDARY(A, B, C, ...) LIB_START_EXCHANGE(A, myContextA) LIB_START_EXCHANGE(B, myContextB) LIB_START_EXCHANGE(C, myContextC) ... USER_COMPUTE_BOUNDARY(A, B, C, ...) LIB_END_EXCHANGE(A, myContextA) LIB_END_EXCHANGE(B, myContextB) LIB_END_EXCHANGE(C, myContextC) ... Here's a situation where a "large number" of contexts might be needed. Now, here's how we fix it when contexts are scarce: USER_COMPUTE_BOUNDARY(A, B, C, ...) LIB_START_EXCHANGE(A, myContext, keyA) LIB_START_EXCHANGE(B, myContext, keyB) LIB_START_EXCHANGE(C, myContext, keyC) ... USER_COMPUTE_BOUNDARY(A, B, C, ...) LIB_END_EXCHANGE(A, myContext, keyA) LIB_END_EXCHANGE(B, myContext, keyB) LIB_END_EXCHANGE(C, myContext, keyC) ... The "keys" are defined by the library to be in some range. The library guarantees that communication operations initiated by calls to any routines using different keys will not collide. This might be done by using "key" inside each routine as a base to select a range of tags. It is the user's responsibility to ensure that the "keys" are unique. The bottom line is: the library developer is responsible for managing tags used by a library. The context insures that communication internal to a library routine will not collide with communication external to that routine. OK, now I'm comfortable with "context" as a scarce resource that is allocated ONCE at the beginning of an application. Comments? Flames? Tom From owner-mpi-context@CS.UTK.EDU Tue Mar 30 13:38:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17175; Tue, 30 Mar 93 13:38:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA18165; Tue, 30 Mar 93 13:37:43 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 30 Mar 1993 13:37:42 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA18157; Tue, 30 Mar 93 13:37:41 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA14756; Tue, 30 Mar 93 12:30:38 CST Date: Tue, 30 Mar 93 12:30:38 CST From: Tony Skjellum Message-Id: <9303301830.AA14756@Aurora.CS.MsState.Edu> To: hender@macaw.fsl.noaa.gov, mpi-context@cs.utk.edu Subject: Re: A few more ramblings Cc: mpsears@newton.cs.sandia.gov Given all the discussion since I left for Salishamn, I will need to rely on Mark to present his part of XIII (oops, VIII) tomorrow. - Tony From owner-mpi-context@CS.UTK.EDU Tue Mar 30 13:54:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17387; Tue, 30 Mar 93 13:54:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19035; Tue, 30 Mar 93 13:54:12 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 30 Mar 1993 13:54:11 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19026; Tue, 30 Mar 93 13:54:09 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA14808; Tue, 30 Mar 93 12:47:29 CST Date: Tue, 30 Mar 93 12:47:29 CST From: Tony Skjellum Message-Id: <9303301847.AA14808@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: agenda of context committeee we will prorate all speaker times in our agenda, to give equal time to Mark Sears. - I will give exact details tomorrow, Tony From owner-mpi-context@CS.UTK.EDU Fri Apr 2 01:51:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24312; Fri, 2 Apr 93 01:51:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08735; Fri, 2 Apr 93 01:51:15 -0500 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 2 Apr 1993 01:51:14 EST Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08727; Fri, 2 Apr 93 01:51:13 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01906; Fri, 2 Apr 93 00:50:53 CST Date: Fri, 2 Apr 93 00:50:53 CST From: Tony Skjellum Message-Id: <9304020650.AA01906@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: the gathering Cc: mpi-collcomm@cs.utk.edu Dear Context sub-committee members (and observers from collcomm, etc), The meeting this week underscored the need for convergence to a unifying proposal that captures the features of Proposal I, VIII, and III+VII=X. The following work will be accomplished before May 12 to that end, while respecting the current separateness of I and VIII. I regret having to leave current MPI meeting early, but the context discussions were quite sufficient to put me in a higher gear on the problems before us... . Rik Littlefield agrees to organize a set of test cases to be coded for each proposal; proposers including codings in their proposals. Deadline for such examples is April 21, 8pm EST. This will be discussed on mpi-context over next three weeks. . I will develop a unified proposal X (with sensible names, and rationale, details, performance discussion, and examples). . I will ask for help, as needed, from Lyndon/Mark/Marc etc, on understanding nuances of their proposals, . Marc Snir / Lyndon Clarke will discuss changes/enhancements (if any) to Proposal I . Mark Sears will complete (presumably) a full proposal VIII (Tacit in this discussion is the accepted merger of III+VII as X, despite its incomplete state, so we have eliminated some proposals from consideration this round). To be considered for a straw vote (before next meeting), all proposals must be complete in that they must . Address their interactions with the first-reading of pt2pt, and current status of collcomm, including needed changes if any . Provide specific syntax/semantics, as needed for pt2pt & collcomm chapters . Describe any known flaws in syntax / semantics . Describe logical subsets, if any, for MPI1 . Implement the examples that Rik organizes, and upon which we agree together (including those from Wednesday night discussion session) . Include discussions of how starting works, and what the spawning semantics must provide them (or through an initial message) so that they can work. . The meaning of the MPI_ALL group in the proposal, if any, or weaker substitutes for same. . The existence/non-existence/requirement for servers or shared-memory locations to effect some features . Include expectations for performance of key operations (eg, how much does it cost to get a new context?, can this be done outside of loops and cached?) . Describe their use of a "cacheing facility," if any . Describe their syntax/semantics of a "cacheing facility" . Describe their reliance on any other MPI1 features not specifically part of context/group/tag/pid nature - - - - - Presumably Proposals I, VIII, and X will fill all requirements to reach the next straw poll deadline. Whichever do make this Straw poll deadline, (May 10, 1993, 5pm EST), can be considered by the voting subcommittee. A ranking will be developed, with the bottom N-2 proposals dropped. We will meet on the evening of Wednesday, May 12, 8:00pm CST, for as long as it takes to choose the final proposal, possibly by further merger of the remaining strong proposals. On Thursday, May 13, we will present our first reading of the Context subcommittee (with possible spill over to Friday, May 14). Actual context sub-committee members will vote, only, in all cases. Please recall the two-sub-committee voting limit of the MPIF (as well as sub-committee membership; observers are always welcome). I will strive not to send fine-grain changes to proposal X's around, but will wait to circulate my product in complete form, prior to May 10, so there is a lower e-mail burden for next weeks; perhaps others will like to keep their updates coarse grain, but share important things with everyone, for sure. If agreements/compromises occur between proposals and/or proposers, please share this with me and the sub-committee in a timely fashion; I do not desire surprises at the next meeting. For instance, if Marc Snir were willing to consider a separate context feature (separate from group) in Proposal I, a lot of effort could be averted, because his proposal is pretty good otherwise (except in re inter-group issues). I think Lyndon will be talking to Marc about making inter-group communication easier in Proposal I, also. If any breakthroughs are made, please let me know. - Tony PS Please copy mpi-collcomm on context-related matters for the duration of MPIF. . . . . . . . . . . "There is no lifeguard at the gene pool." - C. H. Baldwin "In the end ... there can be only one." - Ramirez (Sean Connery) in Anthony Skjellum, MSU/ERC, (601)325-8435; FAX: 325-8997; tony@cs.msstate.edu From owner-mpi-context@CS.UTK.EDU Tue Apr 6 15:32:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25155; Tue, 6 Apr 93 15:32:28 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09620; Tue, 6 Apr 93 15:31:25 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 6 Apr 1993 15:31:24 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09604; Tue, 6 Apr 93 15:31:20 -0400 Date: Tue, 6 Apr 93 20:31:15 BST Message-Id: <826.9304061931@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-context: comment and suggestion To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-colcomm@cs.utk.edu Dear MPI context colleagues. I'd like to say something about contexts and groups and the extant proposals ... First off, we have two major concepts floating around, which I need to define here for purpose of the discussion below. Group --- is an ordered collection of distinct processes, or formally of references to distinct processes. It provides a naming scheme for processes in terms of a group name and rank of process within group. Context --- is a distinct space of messages, or more formally of message tags. It provides management of messages as a message in context A cannot be received as a message in context B. Within these definitions there are exactly two themes in the extant proposals. Marc Snir, in Proposal I, views Group and Context as identical. This simplifies the number of concepts in MPI, but does mean that we can have intragroup communication and no way at all can you have intergroup communication within the above definition of Group and Context. Rik and I amusingly coined the term "grountext" to describe the group/context entity in this proposal. Tony Skjellum, in Proposal III, views Group and Context as independent. This means two concepts instead of one, but does mean that we can allow intragroup communication and some intergroup communication with restriction on how flexible we can make such communication. Proposals VIII and X are identical to III in the manner in which they treat Context and Group as independent concepts. Please consider Proposal VII as not compliant with the above definitions of Group and Context. We need to decide: 1) Are context and group identical or different? 2) Is intergroup communication provided? Now I want to point out something about intergroup communication which we have in our system and find most expressive and convenient, but does not fit in with the above frameworks and the assumption that the message envelope always contains just (context, process, tag). Receive in intergroup communication can wildcard on (sender group and sender rank) or (sender rank), in addition to message tag. We (at EPCC) do, and want to do (in MPI) (written out in longhand notation) receive(group, group', rank, tag) where group is the receiver group, group' is the sender group, rank is the sender rank in group' and tag is the message tag. The receiver can never wildcard group. The receiver can always wildcard tag. The receiver can always wildcard either (rank) or (group' and rank). (In fact, group and group' in this expression are more like the grountext of Marc's proposal or the "context" of historical proposal VII, but never mind on that point.) In the framework of Marc we can reasonably do intergroup communication without wildcard on group'. To do this we transmit group information in messages and form a group which is the union of group and group'. We cannot add wildcard on group' by saying that to do that one forms a union of group and all cases of group'. This requires the sender to always know too much about the detail of the recieve call with which it is to match (i.e., that the receiver is or is not doing a wildcard). If you disbelive this, then you should probably argue that we do not need source selection in point-to-point as you can use tag to choose the source, as it is the same argument (and bogus in my opinion). In the framework of Tony we can reasonably do intergroup communication without wildcard on group'. To do this we transmit group information in messages and choose a context for the pair of groups to use for intergroup communication. We cannot add wildcard on group' by using a context agreed for such use between group and all cases of group'. The argument is the same as that above after a little substitution. If we are serious about intergroup communication then in my opinion we really should provide the facility to wildcard on sender group. This throws up a small number issues, some of which I now address. No process addressed: I didn't mention process addressed communication at all. Perhaps the demons of speed are bothered by this. Well, we could do such as (context,process,tag), and the above does not exclude it. We can fit it in, of course. Size of point-to-point section: I said above "longhand notation". Well that is the most expressive and convenient notation, and if you ask me then I think that (group,group,rank,tag) or (NULL,group,rank,tag) are both acceptable for intragroup communication. On the other hand one can introduce some grunge syntax for intergroup communication which use the same framework as intragroup communication and replaces group in (group,rank,tag) with some glob object which is "shorthand" for (group, group'). This is not the best syntax in the world but we can live with it. We can even fit in the process addressed stuff with this kind of syntax as I have shown in Proposal X. Message envelope: You probably spot that this needs the sender group id to go into the message envelope. Perhaps the demons of speed are bothered by this. Well, you could have a different enevelope for groupless communication, intragroup communication and intergroup communication, and only pay the cost of the bigger envelope when you need it. This is going to take two bits for envelope identification. Big deal! It will anyway be natural not to match communications of different kinds (e.g. intergroup cannot match with intragroup, groupless cannot match with intergroup) so the extra header bits would be useful anyway. Unknown group: You probably also spot that the receive with wildcard on group can pick up a group that the receiver knows nothing of. I would be happiest if the implementation of MPI at the receiver asked the implementation of MPI at the sender about the group in this case, so that the receiver never has to bother about the eventuality. We (at EPCC) could accept that the returned group identifier is a NULL identifier. This means that groups have to exchange flattened group descriptions in messages in a reasonable way before they can make a great deal of sense of intergroup communication. Not ideal, but we can live with it. Comments please? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Apr 8 10:57:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11782; Thu, 8 Apr 93 10:57:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15395; Thu, 8 Apr 93 10:56:38 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 8 Apr 1993 10:56:37 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15386; Thu, 8 Apr 93 10:56:27 -0400 Date: Thu, 8 Apr 93 15:56:22 BST Message-Id: <2310.9304081456@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-context: context and group (medium) To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-collcomm@cs.utk.edu Dear MPI Colleagues This letter is about groups, contexts, independence and coupling thereof, the kinds of point-to-point communication which we have talked about, and to brief extent libraries. Before embarking on the guts of the letter, I should like to express very strong support for the suggestion that MPI users can cleanly program in the host-node model. In my opinion, this model of programming is of considerable commercial significance, and I observer that their are a number of important programs around which use this model. o--------------------o I understand three different kinds of point-to-point communication which have been discussed by various people in MPIF. I write these out with separate group and context concepts, as per a previous message to mpi-context [Subject: mpi-context: comment and suggestion]. I will then discuss coupling of group and context. I refer the reader to my prevous message to mpi-comm which described classes of MPI user libraries [Subject: mpi-comm: various (long)], as their is some follow on discussion below. Groupless (process addressed) ----------------------------- (process, context, tag) Wildcard on process, tag. No wildcard on context Intragroup (closed group) ------------------------- (group, rank, context, tag) Wildcard on rank, tag. No wildcard on group, context. Intergroup (open group) ----------------------- (lgroup, rgroup, rank, context, tag) Wildcard on rgroup, rank, tag. No wildcard on lgroup, context. Observe that "group" in intragroup and "lgroup" in intergroup are the same thing, they are the group of the callling process. Since neither "group" nor "context" in intragroup can be wildcard then there may appear to be appeal in some coupling of them in order to provide shorter syntax and easier context/group management. This implies that we couple context to group of calling process. Now this coulping is not compatible with intergroup since the two calling processes have different groups, thus different contexts, thus the send and receive can never match. We can resolve this difficulty by a more careful statement of where the context of the message is coupled. In particular we can state that the context of the message is coupled to the group of the message receiver. In this way we would express intragroup as a coupling of (group,context), and we would express intergroup as a pair of such couplings. The claim we have heard that context and group must be strongly coupled, resulting in a proposal which asserts that context and group are identical, is possibly nothing more than a consequence of an assumption that messages may only be distinguished on the basis of (process, context, tag) (here process is a process label which can be a rank within a group). Given that assumption, we can only use context to distinguish messages within different groups and the two entities become strongly coupled. Examining records of the early meetings of MPI, I find that this "decision" was made by the point-to-point subcommittee in a straw poll which rejected selection by group by a narrow majority of 10 to 11. Please note also that the same meeting rejected context modifying process identifier --- a "decision" which we are already often ignoring. These "decisions" predate the existence of the contexts subcommittee and the vigorous discussion of contexts and groups which has been and continues to take place. We should uniformly be open minded enough to allow ourselves to question all such "decisions", and to change them if we see fit. The description of MPI user libraries which has been given by Mark Sears and myself strongly suggests that context and group must be independent entities. Provision of the process addressed communication immediately suggests that a context can appear without coupling to a group in which case it seems (to me) that they are independent entities. There is an argument against process addressed communication which says that process addressed communication gains nothing in performance over intragroup communication in the group of all processes. If the process description in process addressed communication will, for sake of generality and thus portability, have to be an some kind of pointer to a process description object which contains whatever information is needed to route a message to the intended recipient. It could be just that (in C, at least), a pointer. Sometimes, on some machines, it will actually be implementable with some other kind of magic which is more scalable, but it must always appear the same way. It could be an index, representable as an integer in the host language, into a table of process description objects (better for F77, for sure). It could be a rank in a group of (all) processes, used as an index into a process description object table, which is just fine for a static process model (and reflects existing practice). It could be some kind of global unique process identifier which is again user as a table index somewhere. If tables grow too large in either of the latter cases, then there may be some hashing and/or caching involved. There are counter arguments. I give one, and invite you to give more. On some machines, the global unique process identifier is sufficient to route the message, and is representable as an integer in the host language. For example, the global process id can be a composite of two bit fields (nodeid, procid) where nodeid is a physical processor node number and procid is a process number on the node, and the nodeid bit field is sufficient to route. In these cases, there is no need for a process description object table, and no need to do a table lookup. We probably all have used machines just like this. For me the arguments have piled up in favour of context and group being separate and independent entities. This letter therefore makes the recommendation that context and group are separate and independent entities. In that light I propose further discussion on management of contexts within and between processes, and within and between groups, and on the subject of the use objects which bind one or more contexts and one or more groups in order to keep the communcation syntax compact by overloading. I shall post another letter to you tomorrow. o--------------------o Comments, questions, (flames :-) please?! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Apr 8 14:32:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15572; Thu, 8 Apr 93 14:32:12 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26381; Thu, 8 Apr 93 14:31:06 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 8 Apr 1993 14:31:05 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26243; Thu, 8 Apr 93 14:29:58 -0400 Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA01330; Thu, 8 Apr 93 11:29:43 PDT Message-Id: <9304081829.AA01330@SSD.intel.com> To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu, prp@SSD.intel.com Subject: Re: mpi-context: context and group (longer) In-Reply-To: Your message of "Thu, 08 Apr 93 15:56:22 BST." <2310.9304081456@subnode.epcc.ed.ac.uk> Date: Thu, 08 Apr 93 11:29:42 -0700 From: prp@SSD.intel.com > From: L J Clarke > Subject: mpi-context: context and group (medium) > > ... > > For me the arguments have piled up in favour of context and group > being separate and independent entities. > > Lyndon I agree, although I also see merit in associating a context with a group. I would like to share my thoughts about context which lead me to think we need two differently managed forms of context. Some of you have already heard this. Most of the discussion about context has revolved around protecting two different entities: libraries and groups. I think this is required, but I think they need very differently managed contexts. One form is not adequate to cover both needs without sacrificing performance. Consider a SPMD program with these calls. Assume the calls are loosely synchronous. call to LibA (Group1) call to LibB (Group1) call to LibB (Group2) In a loosely synchronous environment, messages for the next call can come in before the previous one has completed. Here we see two forms of overlap. Within the call to LibA, we might get messages from processes which have already entered LibB. If LibA and LibB are independently written, they might use some of the same tags. To avoid messages from LibB matching receives in LibA, we must use different contexts. If we have static contexts, allocated when the libraries are initialized, each call in the library can quickly provide the context to its point-to-point calls. If we only have dynamic contexts, especially if contexts are carried inside groups, then a library must be prepared to dynamically allocate a new context on any call when it sees a new group. I know we discussed ways to do this locally, so the context could be created and cached locally on the fly without communication, but I find the idea of incorporating such code into every library call horrifying. Within the first call to LibB, we might get messages from processes which have entered the second call to LibB. Since these calls are in different groups, it might be difficult to code LibB in such a way that messages could not intermix, since a process' position in Group2 might be quite different from its position in Group1. (I would hope that libraries would be coded so that multiple sequential calls to the same library with the same group would be safe. That seems to be current practice.) To keep the two calls from interfering, it would be convenient to have a different context for each group. If each group contains a dynamically allocated context, thats easy. But if contexts are statically allocated, especially if they require a name server, getting a new context for each new group might be a global operation that wouldn't scale well. So I propose that we need two forms of context, one that is quite static for protecting code, and one that is more dynamic for protecting groups. The only mechanism I know of that is adequate for protecting code is context alloctated via a nameserver. In MIMD programs, one cannot say much about the order in which libraries are initialized. Thus, if context is statically allocated at initialization time, there must be a way to obtain the global context value for a piece of code independently of other processes. A more static method, such as a MPI registry or a "dollar bill server" has the disadvantage of requiring a much larger value range for context. That uses precious bits in the envelope of every message. Once a context is allocated to a piece of code, it can be safely stored in a global variable without endangering thread safety or shared memory implementations, because no matter how many instantiations of the library store into the variable, they will always store the same value. There are nice dynamic mechanisms for allocating context for groups, which require only communication within the group. This can piggyback on the communication which is probably required to set up and synchronize the group when it is created. For instance, one might set aside a small number of context values for use by groups. When a group is created, every process in the group could provide its current set of free context values, possibly as a bit vector. After a groupwide reduction, each process chooses the smallest value from the intersection, resulting in every process choosing the same value. Other forms of context protection might be required in the future. I don't predict any, and expect that with both a static and a dynamic form, it is likely that future needs would be covered. The point-to-point calls might be configured to accept (group, rank, context). In this configuration, the static context protecting the code is passed in explicitly, and the context protecting the group is inside the group object. I'm not sure how this interacts with cross-group message passing. Perhaps the simplest solution is to use a well-known group context in such cases, which effectively disables group protection. Those are my thoughts on context. Although I think the methods outlined here are simple enough, I would be happy to see simpler mechanisms that solve the same problems. I am not comfortable with any solution that requires active participation by every library call, no matter how local. Paul From owner-mpi-context@CS.UTK.EDU Fri Apr 9 11:48:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29402; Fri, 9 Apr 93 11:48:39 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24713; Fri, 9 Apr 93 11:48:25 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 11:48:24 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24704; Fri, 9 Apr 93 11:48:22 -0400 Date: Fri, 9 Apr 93 16:48:19 BST Message-Id: <3201.9304091548@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-context: context and group (medium) To: mpi-context@cs.utk.edu In-Reply-To: L J Clarke's message of Thu, 8 Apr 93 15:56:22 BST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-collcomm@cs.utk.edu Dear MPI Colleagues This is a short letter. First: a colleague here pointed out to me that I left an unfinished point, ie failed to draw to conclusion, in the mail message "Subject: mpi-context: context and group (medium)". Apologies to you all for my slipshod work. I conclude that discussion here. Regarding the process identifier, for which there are arguments for and against its appearance as a global unique process identifier and as a process local handle to opaque process descriptor object. The discussion in the referenced letter should have concluded that MPI should say that it is a process local identifier of a process expressed as an integer, and no more. This allows the implementation of MPI to choose the "best" form, which may be a global unique process identifier or may be a process local opaque reference to a process description object or may be an index into a table of subject objects describing all processes. Second: the letter I sent to you all "Subject: mpi-context: comment and suggestion" contained. Apologies again. I correct those errors here. * The claim that the conceptual framework of Tony regarding Group and Context restricts the possibilities for inter(group)communication is false. It is the restriction of the message envelope to (context,process,tag) which creates the limitation in this case. * When I explained how intergroup communication can be done within the conceptual framework of Marc (Snir) I should have said that this is a method for *simulating* intergroup communication without wildcard on group'. * When I explained how intergroup communication can be done within the framework Tony Ishould have said that this is a method for *implementing* intergroup communication without wildcard on group'. Final: Regarding the same letter whihc really deals with the subject of inter(group)communication I may have made errors or at least unhelpful assumptions in the latter couple of paragraphs of the message. Again I apologise. I plan to go into deep thought on the subject of inter(group,context)communication, and promise to deliver some quality discussion to you all next week. Please bear with me. Until such time I shall omit inter(group,context)communication from my discussions. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Apr 9 12:20:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29769; Fri, 9 Apr 93 12:20:59 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26161; Fri, 9 Apr 93 12:20:16 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 12:20:15 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26150; Fri, 9 Apr 93 12:20:12 -0400 Date: Fri, 9 Apr 93 17:20:03 BST Message-Id: <3227.9304091620@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-context: Why scarce contexts? To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-collcomm@cs.utk.edu Dear MPI Colleagues. This question is primarily directed at Mark Sears. Mark, in Proposal VII you say that contexts will be a scarce resource, in fact you suggest 16 which is in my mind very scarce indeed. Why do you say this? It will help me/us if I/we understand, I am sure. Please reply. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Apr 9 13:06:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00684; Fri, 9 Apr 93 13:06:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28334; Fri, 9 Apr 93 13:06:08 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 13:06:07 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28326; Fri, 9 Apr 93 13:06:06 -0400 Received: from newton.sandia.gov (newton.cs.sandia.gov) by cs.sandia.gov (4.1/SMI-4.1) id AA04322; Fri, 9 Apr 93 11:06:04 MDT Received: by newton.sandia.gov (5.57/Ultrix3.0-C) id AA01816; Fri, 9 Apr 93 11:06:59 -0600 Message-Id: <9304091706.AA01816@newton.sandia.gov> To: mpi-context@cs.utk.edu Subject: Re: mpi-context: Why scarce contexts? In-Reply-To: Your message of Fri, 09 Apr 93 17:20:03 -0000. <3227.9304091620@subnode.epcc.ed.ac.uk> Date: Fri, 09 Apr 93 11:06:58 MST From: mpsears@newton.cs.sandia.gov Lyndon asks why I think context values will be a scarce resource. First, we don't have any right now, so that is a data point :-) I think there are several reasons. The first is that a context requires underlying resources in the implementation (e.g. queues) which may be limited. A message arrives at a process, it goes into a queue matching the assigned context value in the envelope. Both support for the queue and the matching function take some effort. (16 queues is not too bad; 1000 is a lot.) One way to limit the effort required is to limit the number of supported contexts. Second, the bits in the envelope that support the context value have to come from somewhere, probably the existing tag field. If the tag field is only 16 bits to begin with (for argument's sake), then taking more than 4 bits for a context value might have a large impact. This is a question vendors might answer: how many context values and tag values are you willing to support on future platforms and how many are you willing to back fit on existing ones? Last, I don't see a need for billions of contexts. My model calls for most programs to use handfuls, not thousands. I would also like to think (this is a hopeless cause, but here goes) that much of MPI could be implemented in hardware, not just the communications part but the part that we now think of as overhead. This would greatly extend the class of programs that could benefit from parallelization, and I oppose for this reason things which add unnecessary complexity to the communications process. Mark Sears Sandia National Laboratories P.S. My proposal is VIII, not VII. From owner-mpi-context@CS.UTK.EDU Fri Apr 9 13:32:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01034; Fri, 9 Apr 93 13:32:25 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29401; Fri, 9 Apr 93 13:31:47 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 13:31:46 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29393; Fri, 9 Apr 93 13:31:43 -0400 Date: Fri, 9 Apr 93 18:31:38 BST Message-Id: <3385.9304091731@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-context: Why scarce contexts? To: mpsears@newton.cs.sandia.gov, mpi-context@cs.utk.edu In-Reply-To: mpsears@newton.cs.sandia.gov's message of Fri, 09 Apr 93 11:06:58 MST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-collcomm@cs.utk.edu Dear Mark First, apologies for getting the proposal number wrong. > > Lyndon asks why I think context values will be a scarce resource. > [stuff deleted] > I think there are several reasons. The first is that a context > requires underlying resources in the implementation (e.g. queues) > which may be limited. A message arrives at a process, it goes > into a queue matching the assigned context value in the > envelope. Both support for the queue and the matching function > take some effort. (16 queues is not too bad; 1000 is a lot.) > One way to limit the effort required is to > limit the number of supported contexts. What you seem to be asking in this argument is that each process should use a limited number of contexts, which is different to asking that the system as a whole should use a limited number of contexts. Okay, that is just perhaps a subtle point. You are assuming details of an implementation of context. For example, in a different approach there could be just one queue which is searched through (in some fashion) in receive for a matching message, testing for context in no different way to testing for tag and sender. In that implementation contexts do not require resource, and the number of contexts is bounded only by the bit length of the context identifier. I imagine that you must have good reasons for the assumed implementation of context. Please do let me/us know why you make the assumption, I am sure that I am not alone in my concern that the number of contexts should be so scarce, but perhaps you know of very good reasons why they should so be. > Second, the bits in the envelope that support the context value > have to come from somewhere, probably the existing tag field. If > the tag field is only 16 bits to begin with (for argument's sake), > then taking more than 4 bits for a context value might have a > large impact. I must be missing something here again. This seems to say that the bit length of the envelope is fixed to some number of bits and the more fields we want to cram into the envelope the shorter the bit lengths of fields must be. Is there a good reason why the bit length of the envelope shoud be fixed in this fashion, or perhaps are you arguing that the bit length of the envelope should be as short as possible? > This is a question vendors might answer: how many > context values and tag values are you willing to support on future > platforms and how many are you willing to back fit on existing ones? > Yes, this would be a good question for the vendors indeed. VENDORS - PLEASE PLEASE PLEASE DO ADVISE US ON THIS ONE. > Last, I don't see a need for billions of contexts. My model calls > for most programs to use handfuls, not thousands. Yes, your model demands that programs use a handfull, the concern which I have is that complex and highly modular software will not be able to conform with your model, inhibiting the development of third party software. > I would also like to > think (this is a hopeless cause, but here goes) that much of > MPI could be implemented in hardware, not just the communications > part but the part that we now think of as overhead. This would > greatly extend the class of programs that could benefit from > parallelization, and I oppose for this reason things which add > unnecessary complexity to the communications process. I am sure that vendors do take very seriously the possibility of implementing relevant parts of MPI in hardware. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Apr 9 15:33:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05144; Fri, 9 Apr 93 15:33:07 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04518; Fri, 9 Apr 93 15:32:29 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 15:32:28 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04510; Fri, 9 Apr 93 15:32:25 -0400 Date: Fri, 9 Apr 93 20:32:21 BST Message-Id: <3457.9304091932@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-context: context management and group binding (long) To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-collcomm@cs.utk.edu Dear MPI Colleagues I now discuss context management and group binding. As promised I omit inter(group,context)communication for the present. This letter is further to letters of today and yesterday to mpi-comm and mpi-context. Some of the people I talked with about contexts at the recent meeting wanted to be able to generate some contexts values themselves, i.e. not by calling context constructor procedures. This is accomadated in the recommendations of this letter. In my letter to mpi-comm "Subject: various (long)" I suggested that the question of secure/insecure point-to-point and collective communications could be described as a property of a context, with some advantage. In this letter I will incorporate this feature. I will also be discussing communicator objects as in Proposal X, but with sensible names. Tony Skjellum has made the valued suggestion to me, privately, that it is better to attributise the communicator object with the secure/insecure stuff, rather than the context. I shall adopt this suggestion in this letter and attributise the communicator object rather than the context. o--------------------o Message Contexts ---------------- In this proposal a (message) context identifier is (like a message tag) just an integer which is used in message selection and (unlike a message tag) may not be wildcard. The interval of context identifiers (1, ..., MPI_NUMUSR_CONTEXTS) are reserved for the MPI user to manage as she sees fit. Use of these contexts allows the user to write programs which do not make use of the provided context creation and deletion facilities. How big should MPI_NUMUSR_CONTEXTS be? Say 1, 2, 4, 8, 16, 32, 64, 128, ... Steve and Tom, and friends, can you advise? The MPI system provides a procedure which creates a unique context outside the interval of reserved user context identifiers, and a procedure which deletes a created context (it does not delete user reserved contexts). For example: context = mpi_create_context() mpi_delete_context(context) There may be advantage in defining the context create and delete functions such that they create and delete more than one context at a time, in order to amortise creation/deletion overhead. Please note that these context generation calls are made by a single process and are asynchronous. They can be implemented as a process local operation by attatching the global process identifier to a process local context allocator, at the expense of needing a lot of bits in the context. They can also be implemented via access to shared data (or a reactive server) in which case the bit length of the context can be made smaller. [I view this as an implementation detail which we should not dwell on in MPI, and should be the freedom of the implementor to choose any formally correct method which hopefully optimises execution on the target platform.] The user program may make use of the user reserved contexts. ClassB libraries (encapsulated objects) are expected to use system created contexts. These can be created as above or through the Communicator object constructors described below. Communicator Objects -------------------- The context acquired by the user in either of the above ways is not valid for communication. Communication is effected by use of a Communicator object, which is a binding of context, zero or more groups (just zero or one in this letter), and communicator attributes (just one in this letter). Two classes of communicator are described in this letter: * WorldCommunicator - an instance of a WorldCommunicator is a binding of context to nothing. This communicator allows the user to intracommunicate within the world of processes comprising the user application, labelling processes with their (process local) process identifier. * GroupCommunicator - an instance of a GroupCommunicator is a binding of context to a process group. This communicator allows the user to intracommunicate within the group of processes comprising the group, labelling processes with their (group global) rank within group. Communicator creation defines the SECURITY attribute of the communicator to be created, which may be any of the following: * MPI_DEFAULT_COMMUNICATOR - the default Security attribute specified in environmental management. * MPI_REGULAR_COMMUNICATOR - the regular Security attribute which provides regular point-to-point and collective semantics * MPI_SECURE_COMMUNICATOR - the secure Security attribute which provides secure point-to-point and collective semantics Communicator objects are opaque objects of undefined size referenced by an object handle which is expressed as in integer in the host language. Communicator creation will create a context for the Communicator, or will accept and bind a user managed context. MPI should provide procedures for creation of each class of Communicator objects, and for deletion of any class of Communicator object. For example, handle = mpi_create_world_communicator(context, security) handle = mpi_create_group_communicator(group, context, security) mpi_delete_communicator(handle) In each creation procedure "security" is the security attribute described above. It is the responsibility of the user to ensure that all communicators with the same context also have the same security. In each creation procedure "context" may be a user managed context or may take the value MPI_NULL_CONTEXT (or something like that :-) in which case the creation procedure also creates a context for the communicator. If the creation procedure creates a context then the procedure synchronises the calling processes (all processes for a WorldCommunicator and the group of processes for a GroupCommunicator) and returns the same context to each copy of the communicator object. If a user managed context was supplied then the procedure is process local and it is the responsibility of the user to ensure that each user managed context is bound to no more than one communicator at any time. In the GroupCommunicator creation procedure "group" is a handle to a group description. The communicator deletion procedure deletes the bound context if that context was created in the communicator creation procedure but does not delete a user managed context. Short Examples -------------- A user program which only makes use of two user reserved contexts and makes no use of process groupings can "enable" the user reserved contexts by creating WorldCommunicator objects. For example, c0 = mpi_create_world_communicator(0,MPI_DEFAULT_COMMUNICATOR) c1 = mpi_create_world_communicator(1,MPI_DEFAULT_COMMUNICATOR) A ClassA library can accept a communicator object as argument. For example, void class_a_procedure(int communicator, ...) { /* do it */ } A ClassB library can accept a group as argument and create private GroupCommunicator objects. For example, void class_b_procedure(int group, ...) { static int communicator = MPI_NULL_COMMUNICATOR; if (communicator != MPI_NULL_COMMUNICATOR) { communicator = mpi_create_group_communicator(group, MPI_NULL_CONTEXT, MPI_SECURE_COMMUNICATOR); } /* do it */ } This example could be generalised by adding a group "cache" facility as described by Rik Littlefield. Point-to-point communication ---------------------------- The point-to-point (intra)communication procedures have a generic process and message addressing form (communicator, process_label,message_label). I shall deal with Send and Receive separately. Send(communicator, process-label, message-label) ---- * communicator is a WorldCommunicator or a GroupCommunicator * process-label is { the (process local) identifier of the receiver when { communicator is a WorldCommunicator { { the rank in communicator.group of the receiver when { communicator is a GroupCommunicator * message-label is the message tag in communicator.context. The point-to-point communication is REGULAR if communicator.security has the value MPI_REGULAR_COMMUNICATOR, and SECURE if communicator.security has the value MPI_SECURE_COMMUNICATOR. Recv(communicator, process-label, message-label) ---- * communicator is a WorldCommunicator or a GroupCommunicator * process-label is { the (process local) identifier of the receiver when { communicator is a WorldCommunicator { { the rank in communicator.group of the receiver when { communicator is a GroupCommunicator { { a wildcard value in either case * message-label is the message tag in communicator.context or a wildcard value The point-to-point communication is REGULAR if communicator.security has the value MPI_REGULAR_COMMUNICATOR, and SECURE if communicator.security has the value MPI_SECURE_COMMUNICATOR. Collective communication ------------------------ The WorldCommunicator is not valid for MPI collective communication. The GroupCommunicator is valid for MPI collective communication procedures. The collective communication is REGULAR if communicator.security has the value MPI_REGULAR_COMMUNICATOR, and SECURE if communicator.security has the value MPI_SECURE_COMMUNICATOR. o--------------------o Comments, questions, (flames :-), please! For your conveniene, my plan now is to go into a session of deep thought regarding intercommunication, the work we have done at EPCC, and MPI. I will then discuss these thoughts with my colleagues here, and promise to return quality discussion of intercommunication to you sometime next week. [If anyone wants to discuss intercommunication with me, I prefer to do so privately until I have really thought longer and harder than before.] I have an oustanding reply to Paul Pierce's recent letter, which I shall make now. I'll be off-line for a while, probably come on-line again Sunday, and will reply to letters which I hope you will write in a reactive and less prolific fashion. Happy reading :-) Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Apr 9 16:00:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AB06763; Fri, 9 Apr 93 16:00:26 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05670; Fri, 9 Apr 93 16:00:03 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 16:00:02 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05655; Fri, 9 Apr 93 16:00:00 -0400 Date: Fri, 9 Apr 93 20:59:57 BST Message-Id: <3490.9304091959@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-context: CORRECTION to previous message To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues An astute colleague here has pointed out two silly errors and some exceptionally bad phrasing in my previous letter "Subject: mpi-context: context management and group binding (long)" When describing point-to-point receive, please replace the two erroneous occurences of "receiver" by "sender". Cut and paste errors, sorry. In the final paragraph I am inviting your replies and informing that I personally will be in a reactive and less prolific mode of operation. The wording implies that I am asking you to be reactive and less prolific, which of course I would not ask. Tired and hungry errors (its 9pm here now, Easter Friday), sorry. Best Wishes Lyndon "the prolific" ps thanks Al :-) /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Apr 9 16:20:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07110; Fri, 9 Apr 93 16:20:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06508; Fri, 9 Apr 93 16:19:39 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 16:19:38 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06500; Fri, 9 Apr 93 16:19:36 -0400 Date: Fri, 9 Apr 93 21:19:33 BST Message-Id: <3512.9304092019@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-context: context and group (longer) To: prp@SSD.intel.com In-Reply-To: prp@SSD.intel.com's message of Thu, 08 Apr 93 11:29:42 -0700 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Paul Pierce writes: > > For me the arguments have piled up in favour of context and group > > being separate and independent entities. > > > > Lyndon > > I agree, although I also see merit in associating a context with a group. > Hey, some consensus here. Magic! BTW, Paul, I wanted to ask what you thought about my suggestions for secure send/receive being bound to a context kind of thing (now communicator object as of last mail to mpi-context), as opposed to having different calls. I think you are right about the microscopic effect on the code. I just tried to give both global default control and per module instance control over the security question. > Consider a SPMD program with these calls. Assume the calls are loosely > synchronous. > > call to LibA (Group1) > call to LibB (Group1) > call to LibB (Group2) > > In a loosely synchronous environment, messages for the next call can come in > before the previous one has completed. Here we see two forms of overlap. > > Within the call to LibA, we might get messages from processes which have already > entered LibB. If LibA and LibB are independently written, they might use some > of the same tags. To avoid messages from LibB matching receives in LibA, we > must use different contexts. If we have static contexts, allocated when the > libraries are initialized, each call in the library can quickly provide the > context to its point-to-point calls. If we only have dynamic contexts, > especially if contexts are carried inside groups, then a library must be > prepared to dynamically allocate a new context on any call when it sees a new > group. I know we discussed ways to do this locally, so the context could be > created and cached locally on the fly without communication, but I find the > idea of incorporating such code into every library call horrifying. Paul, I have a model for libraries like this, which in my mail to mpi-comm "Subject: mpi-comm: various (long)" I referred to as ClassB libraries, which maybe you might want to think about. It's quite simple. We write libraries just like this, which are akin to encapsulated objects. We think in terms of library instances. The library provides is an instance constructor which accepts a group, creates context(s) for the instance and constructs the instance, returning an instance id to the user which is used to refer to the instance for all calls. That is, all calls including and up to the instance destructor, which asks an instance to detruct itself. Our experience is that users do not find it difficult to manage this model for ClassB libraries. > > So I propose that we need two forms of context, one that is quite static for > protecting code, and one that is more dynamic for protecting groups. I cannot see any difference between the latter of these two contexts "more dynamic for protecting groups" and a global group identifier. > The only mechanism I know of that is adequate for protecting code is context > alloctated via a nameserver. In MIMD programs, one cannot say much about the > order in which libraries are initialized. Thus, if context is statically > allocated at initialization time, there must be a way to obtain the global > context value for a piece of code independently of other processes. A more > static method, such as a MPI registry or a "dollar bill server" has the > disadvantage of requiring a much larger value range for context. That uses > precious bits in the envelope of every message. Once a context is allocated to > a piece of code, it can be safely stored in a global variable without > endangering thread safety or shared memory implementations, because no matter > how many instantiations of the library store into the variable, they will > always store the same value. We find that with regard to operations within a process group, and in particular to library instance construction and desctruction decribed above, the main user program has a highly SPMD nature. So we can exploit sequencing. This is a most valuable learning experience, because we had similar thoughts to those you express here, implemented a name server, and really didn't need it once (for this purpose). > > The point-to-point calls might be configured to accept (group, rank, context). > In this configuration, the static context protecting the code is passed in > explicitly, and the context protecting the group is inside the group object. > > I'm not sure how this interacts with cross-group message passing. Perhaps the > simplest solution is to use a well-known group context in such cases, which > effectively disables group protection. As I point out above, your "group protecting context hidden inside group" really does just seem to me to be a global group identifier. Within the definition of context I see no reason why we necessarily will cause a problem with intercommunication. When you say "use a well know group context in such cases" I take it you mean a common ancestor like the "group context" of all processes or something? I have promised, I will return quality discussion on intercommunication next week. Did the points in this reply letter help, Paul? Best Wishes Lyndon "the temporarily less prolific" /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri Apr 9 18:45:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08745; Fri, 9 Apr 93 18:45:09 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12895; Fri, 9 Apr 93 18:44:25 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 18:44:24 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12872; Fri, 9 Apr 93 18:43:49 -0400 Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA24612; Fri, 9 Apr 93 15:43:36 PDT Message-Id: <9304092243.AA24612@SSD.intel.com> To: lyndon@epcc.ed.ac.uk Cc: prp@SSD.intel.com, mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Subject: Re: mpi-context: context and group (longer) In-Reply-To: Your message of "Fri, 09 Apr 93 21:19:33 BST." <3512.9304092019@subnode.epcc.ed.ac.uk> Date: Fri, 09 Apr 93 15:43:35 -0700 From: prp@SSD.intel.com > From: L J Clarke > > Paul Pierce writes: > > > Consider a SPMD program with these calls. Assume the calls are loosely > > synchronous. > > > > call to LibA (Group1) > > call to LibB (Group1) > > call to LibB (Group2) > > > > In a loosely synchronous environment, messages for the next call can come in > > before the previous one has completed. Here we see two forms of overlap. > > > > Within the call to LibA, we might get messages from processes which have already > > entered LibB. > > Paul, I have a model for libraries like this, which in my mail to > mpi-comm "Subject: mpi-comm: various (long)" I referred to as ClassB > libraries ... Yes, that is a matching concept. > We think in terms of library instances. ... I found this part hard to understand. However, if you propose to use the mechanisms in your just previous mail: > A ClassB library can accept a group as argument and create private > GroupCommunicator objects. > For example, > void class_b_procedure(int group, ...) > { > static int communicator = MPI_NULL_COMMUNICATOR; > > if (communicator != MPI_NULL_COMMUNICATOR) > { > communicator = mpi_create_group_communicator(group, > MPI_NULL_CONTEXT, > MPI_SECURE_COMMUNICATOR); > } > > /* do it */ > } > This example could be generalised by adding a group "cache" facility > as described by Rik Littlefield. First of all this code doesn't work - if the library is called with a different group (see the LibB(Group{1,2}) example above) it will mistakenly use the communicator for Group1 when called for Group2. This problem can be fixed using cacheing. But... This is exactly the sort of too-dynamic, too-intrusive mechanism I find horrifying. I can't conceive of unleashing on the unsuspecting world a standard that requires you to put code like that in every library call (its even more complex with cacheing.) We _must_ come up with a better mechanism. The example mechanism I talked about might look like this: int my_context; void class_b_initialize() /* Called once at the beginning of time */ { my_context = create_and_or_lookup_context("mylib"); } void class_b_procedure(int group, ...) { /* do it using (group, rank, my_context) */ } Note the total absence of context maintenance in the arbitrary library procedure. For group protection, the group must contain an additional embedded context. > > So I propose that we need two forms of context, one that is quite static for > > protecting code, and one that is more dynamic for protecting groups. > > I cannot see any difference between the latter of these two contexts > "more dynamic for protecting groups" and a global group identifier. You are right, its the same. My point is that group context is necessary but _not_ sufficient. > > The only mechanism I know of that is adequate for protecting code is context > > alloctated via a nameserver. ... > > We find that with regard to operations within a process group, and in > particular to library instance construction and desctruction decribed > above, the main user program has a highly SPMD nature. So we can > exploit sequencing. This is a most valuable learning experience, > because we had similar thoughts to those you express here, implemented a > name server, and really didn't need it once (for this purpose). You learned that you have a SPMD universe. We have a mostly SPMD universe, but we have customers already with MPMD applications. One can argue that sequencing is acceptable for a static-process model, but it is not adequate for dynamic processes. We have talked about defining MPI in such a way that it is complete for a static process model but without limiting its extension to a dynamic process model. So we must be careful - if we assume sequencing now, we must do it in a way that allows for a nameserver later. > Lyndon "the temporarily less prolific" Paul From owner-mpi-context@CS.UTK.EDU Fri Apr 9 22:58:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11834; Fri, 9 Apr 93 22:58:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20936; Fri, 9 Apr 93 22:58:02 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 9 Apr 1993 22:58:01 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20883; Fri, 9 Apr 93 22:56:53 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Fri, 9 Apr 93 19:45 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA09922; Fri, 9 Apr 93 19:43:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA22459; Fri, 9 Apr 93 19:43:26 PDT Date: Fri, 9 Apr 93 19:43:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: proposal -- context and tag limits To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, mpsears@newton.cs.sandia.gov Cc: d39135@carbon.pnl.gov, gropp@mcs.anl.gov, mpi-collcomm@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu Message-Id: <9304100243.AA22459@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Lyndon et.al. write: > ... This seems to say that the bit > length of the envelope is fixed to some number of bits and the more > fields we want to cram into the envelope the shorter the bit lengths of > fields must be. Is there a good reason why the bit length of the > envelope shoud be fixed in this fashion, or perhaps are you arguing > that the bit length of the envelope should be as short as possible? > > > This is a question vendors might answer: how many > > context values and tag values are you willing to support on future > > platforms and how many are you willing to back fit on existing ones? > > Yes, this would be a good question for the vendors indeed. > > VENDORS - PLEASE PLEASE PLEASE DO ADVISE US ON THIS ONE. I wonder what kind of useful advice vendors could really give us. Hardware support boils down to a question of getting faster performance in exchange for some relatively small resource limit. But in almost every case I can think of, such limits are made functionally transparent to the user by automatic fallback to some slower mechanism without the resource limit. Thus we have.. fixed size register sets with compilers that spill to memory, fixed size caches with automatic flush/reload from main memory, fixed size TLB's with cpu traps for TLB reload, fixed size physical memory with virtual memory support, and so on. The only counterexample that pops to mind is fixed-length numeric values, for which reasonably well established conventions exist. No such conventions currently exist regarding tag and context values. ============ PROPOSAL TO ENVIRONMENT COMMITTEE ============== The MPI specification should 1. require that all MPI implementations provide functional support for specified generous limits (e.g., 32 bits) on tag and context values, and 2. suggest that vendors provide a system-specific mechanism by which the user can optionally specify tag and context limits that the program agrees to abide by. Even the form of these limits should remain unspecified since they may vary from system to system. ======================== END PROPOSAL ======================== Further discussion... If a vendor wishes to provide hardware support to enhance performance for some stricter limits, and if some people are able and willing to write programs within those limits, that's great. Those people on those machines will be lark happy. If the performance increase is substantial, and I'm on one of those machines, and my program is simple enough, I'll probably be one of those people. However, I am not aware of any system on which generous limits could not be supported, albeit with some loss of performance compared to staying within the (currently hypothetical) hardware-supported limits. Everyone I know would MUCH prefer suboptimal performance over HAVING to rewrite applications to conform to varying and inconsistent hard limits. Yes, I recall the many arguments against mandating specific limits. But, I claim that those arguments are misdirected. They are based on analogy to things like word length and memory size, which I again note are subject to well established conventions and principles. (You can't run big programs on small machines, and we pretty much agree about what "big" and "small" mean.) In the case of context and tag values, such conventions do not exist, and a very wide range of conflicting limits have been discussed at various times and places. I believe that we will not meet our goal of portability if we do not specify usable limits on tag and context values. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Apr 12 13:57:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19566; Mon, 12 Apr 93 13:57:41 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24449; Mon, 12 Apr 93 13:56:46 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 12 Apr 1993 13:56:35 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24426; Mon, 12 Apr 93 13:56:31 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 12 Apr 93 10:42 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA11608; Mon, 12 Apr 93 10:40:32 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA24711; Mon, 12 Apr 93 10:40:28 PDT Date: Mon, 12 Apr 93 10:40:28 PDT From: rj_littlefield@pnlg.pnl.gov Subject: contexts examples/problems 1-3 To: jwf@parasoft.com, lyndon@epcc.ed.ac.uk, mpi-collcomm@cs.utk.edu, mpi-context@cs.utk.edu, mpsears@cs.sandia.gov, snir@watson.ibm.com, tony@cs.msstate.edu Cc: d39135@carbon.pnl.gov Message-Id: <9304121740.AA24711@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Folks, As Tony Skjellum noted, I am organizing a set of test cases & issues to be addressed by the various context proposals. I have formulated these as a set of "problems" such as might be found on an essay test. Here are draft versions of the first three "problem statements" for the context proposals. I anticipate that at least one more problem will be submitted. Please tell me about defects and inadequacies in these problems. If you have a favorite concern, now is the time to get it reflected in the problem set. Thanks, --Rik Littlefield BACKGROUND INFO . Be sure that your point-to-point and group/context control calls are specified elsewhere in your proposal. PROBLEM 1 (simple): . Specify your calling sequence for an MPI circular-shift routine that operates on a contiguous buffer of double precision float values. E.g. you might specify MPI_CSHIFTB (inbuf,outbuf,datatype,len,group,shift) where IN inbuf input buffer OUT outbuf output buffer IN datatype symbolic constant MPI_DOUBLE IN len length of inbuf (# of elements) IN group handle to group descriptor IN shift number of processes to shift . Assume that a user desires to write a new collective communication routine with the same calling sequence as cshift, but with different semantics. To be definite, this routine exchanges data in the pattern needed for one stage in a butterfly. I.e., the process of rank i exchanges data with the process of rank i+shift*(1-2*(i%(2*shift)/shift). Call this routine bflyexchange. . Show an implementation of bflyexchange in terms of your point-to-point and group/context control calls. . Specify the conditions necessary to ensure correct operation of this implementation. E.g., you might say "safe under all conditions", "safe if and only if no other routine issues wildcard receives in the same group/context", "safe if and only if context and tag are unique", or something like that. Making these conditions simple and broad is good. Getting caught stating conditions that are too broad is bad. . Discuss the performance of this implementation. Note that the semantics of bflyexchange require only a single send and receive per process. Explain how this level of performance can be achieved or approached by your implementation. If you assert that group control operations can be done without communications, explain how this works and what implications it has on other system parameters, e.g., the number and range of context values. PROBLEM 2 (medium) . Write a "guidelines for library developers and users" document that explains how to write and call libraries in order to maintain message-passing isolation between the various libraries and between the libraries and the user program. Be sure to explain how to achieve good efficiency. Be complete, but brief. (Long explanations can be interpreted as indicating a complex design.) You may wish to describe two or more self-consistent strategies, along the lines of Lyndon's "ClassA" and "ClassB" libraries as discussed earlier on mpi-context. PROBLEM 3 (hard?) This problem is paraphrased from one posed by Jon Flower. The task is to simulate the host-node programming model through the use of "host" and "node" groups. This is interesting both for backward- compatibility and for its inter-group communication requirements. As stated by Jon, this problem really spans subcommittees. For the sake of the present discussion, I have reformulated it in terms of an SPMD programming model in which a black-box function is used to tell each process whether it's the host or a node. Note in particular that nodes don't know the id of the host. Here is pseudo-code for the desired program: main() { if (I_am_the_host()) host (); else node (); } host (); /* * Form two groups containing: * i) only the host process. * ii) the node processes. */ host_group = mpi_...; node_group = mpi_...; /* * Broadcast from host to all nodes; using "ALL" group. * (It would be nice to have inter-group broadcast for * this since that is more like "current practice".) */ myrank = mpi_...; mpi_bcast( ...., myrank, MPI_GROUP_ALL, ...); /* * Send individual message to each node in turn. */ for(node=0; node < MPI_ORDER(node_group); node++) { mpi_send( ..., (node_group, node), ...); } /* * Receive result from node 0. */ mpi_recv( ..., (node_group, 0), ...); } node() { /* * Form two groups containing: * i) only the host process. * ii) the node processes. */ host_group = mpi_...; node_group = mpi_...; /* * Receive bcast from host using ALL group. */ host_rank = mpi_...; mpi_bcast(..., host_rank, MPI_GROUP_ALL, ...); /* * Receive single message from host. */ mpi_recv(..., 0, host_group, ...); /* * Send point-to-point messages in node group. */ myrank = mpi_... (node_group); nnodes = mpi_... (node_group); sendhandle = mpi_isend( ..., (node_group,(myrank+1)%nnodes), ...); mpi_recv ( ..., (node_group,(myrank-1+nnodes)%nnodes), ...); mpi_complete (sendhandle); /* * Compute global sum in nodes only. */ mpi_reduce(... , node_group, MPI_SUM_OP, ...); /* * Node 0 sends sum to host. */ if(myrank == 0) mpi_send(..., 0, host_group, ...); } . Show how to implement this pseudo-code using your point-to-point and group calls. Note that this code wants to think of node processes in terms of their rank in the node_group, not the ALL group. Be sure to show all details of any translations that are required. . Discuss how the collective comms and point-to-point messages are kept separate, even if the point-to-point calls are changed to used wildcards. ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Apr 12 17:56:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24573; Mon, 12 Apr 93 17:56:06 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10975; Mon, 12 Apr 93 17:54:58 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 12 Apr 1993 17:54:57 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10955; Mon, 12 Apr 93 17:54:02 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13925; Mon, 12 Apr 93 16:52:04 CDT Date: Mon, 12 Apr 93 16:52:04 CDT From: Tony Skjellum Message-Id: <9304122152.AA13925@Aurora.CS.MsState.Edu> To: tony@aurora@cs.msstate.edu, mpsears@newton.cs.sandia.gov Subject: Re: the gathering Cc: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Mark, You should explain what your model implies about the starting of processes. If you assume that processes have been started by MPI, that is OK (generally a tacit assumption of MPI1), but in any event you should tell us what the process is told at the moment of spawning (eg, about ALL groups, or its name, etc), that will help it become part of MPI-based communication. We need to see how "safe"/"unsafe" it will be to start MPI in every model. If it is extremely difficult/simple to get from the "just-spawned" state to the "MPI-up-and-running" sate, that should be made clear. I am happy to answer more questions! Please shoot away. - Tony PS Because this is of general interest to all readers, I am echoing to the reflector. I hope that is OK with you. ----- Begin Included Message ----- From mpsears@newton.cs.sandia.gov Mon Apr 12 15:08:18 1993 To: tony@aurora@cs.msstate.edu Subject: Re: the gathering Date: Mon, 12 Apr 93 14:10:36 MST From: mpsears@newton.cs.sandia.gov Content-Length: 243 Tony, I need a little clarification of what you mean by "Include discussion of how starting works and what the spawning semantics must provide them (or through an initial message) so that they can work." Starting and spawning what? mark ----- End Included Message ----- From owner-mpi-context@CS.UTK.EDU Tue Apr 13 05:34:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15699; Tue, 13 Apr 93 05:34:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24925; Tue, 13 Apr 93 05:33:23 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 13 Apr 1993 05:33:22 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24917; Tue, 13 Apr 93 05:33:19 -0400 Date: Tue, 13 Apr 93 10:32:55 BST Message-Id: <5989.9304130932@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-context: context and group (longer) To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues This never got to you, looks like I hit the wrong key in my mailer on April 10. Oops. Regards Lyndon ---- Start of forwarded text ---- > From lyndon Sat Apr 10 17:50:29 1993 > Date: Sat, 10 Apr 93 17:50:29 > From: L J Clarke > Subject: Re: mpi-context: context and group (longer) > To: prp@SSD.intel.com Paul Pierce writes: > > We think in terms of library instances. ... > > I found this part hard to understand. However, if you propose to use the > mechanisms in your just previous mail: > First off no way do I intend to use the simple example mechanism in my previous email. Of course it does not work if called with different groups, it is just a silly little simple example, it was never meant to work with different groups. Why did you find that part hard to understand? It is just an object oriented approach with a parallel object. I will try to explain again, hopefully making myself more clear, at the foot of this letter. I hope I can make myself clear. > This is exactly the sort of too-dynamic, too-intrusive mechanism I find > horrifying. I can't conceive of unleashing on the unsuspecting world a > standard that requires you to put code like that in every library call > (its even more complex with cacheing.) We _must_ come up with a > better mechanism. Fine, my approach will not do all of this anyway, please see below. > The example mechanism I talked about might look like this: Interesting! This example works in exactly one group/context and therefore is utterly incapable of supporting multiple instances operating on different data. > > int my_context; > > void class_b_initialize() /* Called once at the beginning of time */ > { > my_context = create_and_or_lookup_context("mylib"); > } > > void class_b_procedure(int group, ...) > { > > /* do it using (group, rank, my_context) */ > } > > I cannot see any difference between the latter of these two contexts > > "more dynamic for protecting groups" and a global group identifier. > > You are right, its the same. My point is that group context is necessary but > _not_ sufficient. I cannot concur. I know I mailed a lot of discussion letters recently. I hope you can find time to read them because the mechanisms discussed do mean that you only need the one context. > > > The only mechanism I know of that is adequate for protecting code is context > > > alloctated via a nameserver. ... > > > > We find that with regard to operations within a process group, and in > > particular to library instance construction and desctruction decribed > > above, the main user program has a highly SPMD nature. So we can > > exploit sequencing. This is a most valuable learning experience, > > because we had similar thoughts to those you express here, implemented a > > name server, and really didn't need it once (for this purpose). > > You learned that you have a SPMD universe. Negative! How can you read that from my words? We have a highly MPMD universe, with lots of groups both overlapping and distinct, which do lots of things of closed scope within the groups and lots of things of open scope between groups. We call this MSPMD or M(SPMD), meaning multiple (single program multiple data). The observation we make, and it is hardly surprising, it that the computations and communications associated with a process group are locally SPMD-like in nature. This is an entirely different observation. It is this property of a process group which allows parallel object constructors to be called in sequence. > One can argue that sequencing is acceptable for a static-process model, but it > is not adequate for dynamic processes. Eh? I don't understand what you are suggesting sequencing can be used for in static process model that it cannot in dynamic process model. Please do let me/us know. I'll second guess the sequencing you are thinking about, but I fail to understand how it relates to static/dynamic process model. This is sequencing within groups of calls of context allocators. This means that when the processes of a group allocate contexts as a concerted (and synchronous?) action they each bind same usage to the N'th context generated by the group, for all N. Of course we must realise that we distinguish the N'th context generated by the process P (member of group G) from the N'th generated by the group G, since the P will generally also be a member of other groups (H, ...) and such groups (H, ...) are not identical in membership to G. > We have talked about defining MPI in > such a way that it is complete for a static process model but without limiting > its extension to a dynamic process model. So we must be careful - if we assume > sequencing now, we must do it in a way that allows for a nameserver later. > Well I have no problem with this statement. It would be trivial to incorporate a name server into the discussions I have recently posted. In fact, when I get on to some helpful discussion of intercommunication I will probably be asking for some name service anyway. The explanation promised above now follows. I really will do it mainly by an exceptionally simple worked example. We view ClassB libraries as encapsulated parallel objects. Each object class has a constructor, which creates an object, and destructor, which destroys the object. Each object also provides a number of public operators which you can use to do things with the object. We allow the object to do things with user data as well as its encapsulate data, so we have a hybrid between object oriented and the usual ad hoc. The user calls constructors in all processes within the group G in sequence. Of course, processes which are also in another group say H interleave calls to consructors across H with those across G. The constructor called within G synchronises processes within G. Its a complete doddle. Let me give a very simple example for class Simple in C, assuming that object identifiers are expressed as type "void *" and group and communicator handles are expressed also as type "long". In practice we may implement object identifiers as opaque entities, but it's simpler to show this (void *) stuff here. I omit all error checks and standard speed optimisations as well. #---------------------------------------- # Library fragment void * Simple_contructor(long G) { Simple_state *state Simple_alloc_state(); /* Class state memory allocator */ Simple_state->communicator = mpi_create_group_communicator(G, MPI_NULL_CONTEXT, MPI_SECURE_CONTEXT); return (void *)Simple_state; } void Simple_destructor(void *state) { Simple_state *state = (Simple_state *)state; /* Macros can be faster */ mpi_delete_communicator(Simple_state->communicator); Simple_state_dealloc(Simple_state); /* Class state memory deallocator */ return; } void Simple_maketea_double(void *state, double *data, int items) { Simple_state *state = (Simple_state *)state; /* Macros can be faster */ /* next line is just to show how easy it is to get at the communicator */ long communicator = Simple_state->communicator; /* make "items" cups of double precision tea :-) with communication using communicator */ return; } void Simple_maketea_float(void *state, float *data, int items) { Simple_state *state = (Simple_state *)state; /* Macros can be faster */ /* next line is just to show how easy it is to get at the communicator */ long communicator = Simple_state->communicator; /* make "items" cups of single precision tea :-) with communication using communicator */ return; } # #---------------------------------------- #---------------------------------------- # User fragment /* some decls */ void *simple_ga, *simple_gb, *simple_ha, *simple_hb; double data[16]; float fata[16]; /* stuff */ simple_ga = Simple_constructor(G); if (in_H) { simple_ha = Simple_constructor(H); simple_maketea_double(simple_hb, data, 16); simple_maketea_float(simple_hb, fata, 16); simple_hb = Simple_constructor(H); } simple_maketea_float(simple_ga, fata, 16); simple_gb = Simple_constructor(G); simple_maketea_double(simple_gb, data, 7); /* more stuff */ # #---------------------------------------- ---- End of forwarded text ---- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Apr 13 05:59:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21553; Tue, 13 Apr 93 05:59:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26245; Tue, 13 Apr 93 05:58:56 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 13 Apr 1993 05:58:56 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26237; Tue, 13 Apr 93 05:58:53 -0400 Date: Tue, 13 Apr 93 10:58:49 BST Message-Id: <6027.9304130958@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: some corrections to example To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu In-Reply-To: L J Clarke's message of Tue, 13 Apr 93 10:32:55 BST Reply-To: lyndon@epcc.ed.ac.uk There were some silly mistakes in the example I sent out. Here I just correct that. Regards Lyndon Let me give a very simple example for class Simple in C, assuming that object identifiers are expressed as type "void *" and group and communicator handles are expressed also as type "long". In practice we may implement object identifiers as opaque entities, but it's simpler to show this (void *) stuff here. I omit all error checks and standard speed optimisations as well. #---------------------------------------- # Library fragment void * Simple_contructor(long G) { Simple_state *state Simple_alloc_state(); /* Class state memory allocator */ state->communicator = mpi_create_group_communicator(G, MPI_NULL_CONTEXT, MPI_SECURE_CONTEXT); return (void *)state; } void Simple_destructor(void *id) { Simple_state *state = (Simple_state *)id; /* Macros can be faster */ mpi_delete_communicator(state->communicator); Simple_dealloc_state(state); /* Class state memory deallocator */ return; } void Simple_maketea_double(void *id, double *data, int items) { Simple_state *state = (Simple_state *)id; /* Macros can be faster */ /* next line is just to show how easy it is to get at the communicator */ long communicator = state->communicator; /* make "items" cups of double precision tea :-) with communication using communicator */ return; } void Simple_maketea_float(void *id, float *data, int items) { Simple_state *state = (Simple_state *)id; /* Macros can be faster */ /* next line is just to show how easy it is to get at the communicator */ long communicator = state->communicator; /* make "items" cups of single precision tea :-) with communication using communicator */ return; } # #---------------------------------------- #---------------------------------------- # User fragment /* some decls */ void *simple_ga, *simple_gb, *simple_ha, *simple_hb; double data[16]; float fata[16]; /* stuff */ simple_ga = Simple_constructor(G); if (in_H) { simple_ha = Simple_constructor(H); simple_maketea_double(simple_hb, data, 16); simple_maketea_float(simple_hb, fata, 16); simple_hb = Simple_constructor(H); } simple_maketea_float(simple_ga, fata, 16); simple_gb = Simple_constructor(G); simple_maketea_double(simple_gb, data, 7); /* more stuff */ # #---------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Mon Apr 19 14:22:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07511; Mon, 19 Apr 93 14:22:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03824; Mon, 19 Apr 93 14:21:04 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 14:21:03 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03816; Mon, 19 Apr 93 14:21:02 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17610; Mon, 19 Apr 1993 14:21:01 -0400 Date: Mon, 19 Apr 1993 14:21:01 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9304191821.AA17610@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Names for communicator objects Has anyone come up with sensible names for communicator objects. At the last MPI meeting they were called floopy, bongo, and bingo. I suggest they be called C1, C2, and C3 communicators instead. David -------------------------------------------------------------------------- | David W. Walker | Office : (615) 574-7401 | | Oak Ridge National Laboratory | Fax : (615) 574-0680 | | Building 6012/MS-6367 | Messages : (615) 574-1936 | | P. O. Box 2008 | Email : walker@msr.epm.ornl.gov | | Oak Ridge, TN 37831-6367 | | -------------------------------------------------------------------------- From owner-mpi-context@CS.UTK.EDU Mon Apr 19 14:30:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07582; Mon, 19 Apr 93 14:30:17 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04340; Mon, 19 Apr 93 14:29:56 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 14:29:55 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04332; Mon, 19 Apr 93 14:29:54 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA11638; Mon, 19 Apr 93 13:29:47 CDT Date: Mon, 19 Apr 93 13:29:47 CDT From: Tony Skjellum Message-Id: <9304191829.AA11638@Aurora.CS.MsState.Edu> To: walker@rios2.epm.ornl.gov Subject: Re: Names for communicator objects Cc: mpi-context@cs.utk.edu We have sensible names for these things. It is part of the proposal X rewrite The highest level would be inter-group-communicator, the next lowest is intra-group-communication, and I am not sure what the lowest should be (will review notes). C1, C2, C3 do not contain enough semantic information to be helpful, to my mind... - Tony ----- Begin Included Message ----- From owner-mpi-context@CS.UTK.EDU Mon Apr 19 13:22:12 1993 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 14:21:03 EDT Date: Mon, 19 Apr 1993 14:21:01 -0400 From: walker@rios2.epm.ornl.gov (David Walker) To: mpi-context@cs.utk.edu Subject: Names for communicator objects Content-Length: 729 Has anyone come up with sensible names for communicator objects. At the last MPI meeting they were called floopy, bongo, and bingo. I suggest they be called C1, C2, and C3 communicators instead. David -------------------------------------------------------------------------- | David W. Walker | Office : (615) 574-7401 | | Oak Ridge National Laboratory | Fax : (615) 574-0680 | | Building 6012/MS-6367 | Messages : (615) 574-1936 | | P. O. Box 2008 | Email : walker@msr.epm.ornl.gov | | Oak Ridge, TN 37831-6367 | | -------------------------------------------------------------------------- ----- End Included Message ----- From owner-mpi-context@CS.UTK.EDU Mon Apr 19 14:43:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07750; Mon, 19 Apr 93 14:43:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05458; Mon, 19 Apr 93 14:43:10 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 14:43:09 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05450; Mon, 19 Apr 93 14:43:08 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17381; Mon, 19 Apr 1993 14:43:07 -0400 Date: Mon, 19 Apr 1993 14:43:07 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9304191843.AA17381@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Re: Names for communicator objects If you're going to use long names like inter-group-communicator, intra-group-communication, and ??? (how about no-group-communicator), at least make sure they can be shortened to distinct acronyms. I think it is not too much to ask people to remember that C1, C2, and C3 stand for the three communicators in increasing order of complexity. "Inter" and "intra" will be a problem because there will be a class of people who don't remember that the former is Latin for "between", and the latter is Latin for "within." Also if you're listening to a talk or squinting at someones illegible viewgraphs you might not discern the distinction. David ----- Begin Included Message ----- From root Mon Apr 19 14:29:49 1993 Received: from Aurora.CS.MsState.Edu by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17617; Mon, 19 Apr 1993 14:29:48 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA11638; Mon, 19 Apr 93 13:29:47 CDT Date: Mon, 19 Apr 93 13:29:47 CDT From: Tony Skjellum Message-Id: <9304191829.AA11638@Aurora.CS.MsState.Edu> To: walker@rios2.epm.ornl.gov Subject: Re: Names for communicator objects Cc: mpi-context@cs.utk.edu Status: R We have sensible names for these things. It is part of the proposal X rewrite The highest level would be inter-group-communicator, the next lowest is intra-group-communication, and I am not sure what the lowest should be (will review notes). C1, C2, C3 do not contain enough semantic information to be helpful, to my mind... - Tony From owner-mpi-context@CS.UTK.EDU Mon Apr 19 16:36:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11316; Mon, 19 Apr 93 16:36:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14100; Mon, 19 Apr 93 16:35:21 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 16:35:20 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14081; Mon, 19 Apr 93 16:35:17 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 19 Apr 93 13:31 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA04004; Mon, 19 Apr 93 13:28:59 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA03387; Mon, 19 Apr 93 13:28:56 PDT Date: Mon, 19 Apr 93 13:28:56 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: Names for communicator objects To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Cc: d39135@carbon.pnl.gov Message-Id: <9304192028.AA03387@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu > We have sensible names for these things. > It is part of the proposal X rewrite > > The highest level would be inter-group-communicator, the next lowest > is intra-group-communication, and I am not sure what the lowest > should be (will review notes). > > C1, C2, C3 do not contain enough semantic information to be helpful, > to my mind... > > - Tony I share David's concerns about the subtle difference between "inter" and "intra", and likewise Tony's about "C1" etc not being mnemonic. How about "G0", "G1", and "G2", for both level of complexity and the number of groups involved? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-context@CS.UTK.EDU Mon Apr 19 18:28:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15382; Mon, 19 Apr 93 18:28:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23910; Mon, 19 Apr 93 18:26:50 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 18:26:46 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23900; Mon, 19 Apr 93 18:26:42 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA18691; Mon, 19 Apr 93 17:26:36 CDT Date: Mon, 19 Apr 93 17:26:36 CDT From: Tony Skjellum Message-Id: <9304192226.AA18691@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov, rj_littlefield@pnlg.pnl.gov Subject: RE: Names for communicator objects Cc: d39135@carbon.pnl.gov I would suggest GRP0, GRP1, GRP2, to distinguish between GRIDS (eg, synonym for virtual topologies) versus GROUPS. - Tony ----- Begin Included Message ----- From owner-mpi-context@CS.UTK.EDU Mon Apr 19 15:36:10 1993 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 16:35:20 EDT Date: Mon, 19 Apr 93 13:28:56 PDT From: rj_littlefield@pnlg.pnl.gov Subject: RE: Names for communicator objects To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov Cc: d39135@carbon.pnl.gov X-Envelope-To: mpi-context@cs.utk.edu Content-Length: 878 > We have sensible names for these things. > It is part of the proposal X rewrite > > The highest level would be inter-group-communicator, the next lowest > is intra-group-communication, and I am not sure what the lowest > should be (will review notes). > > C1, C2, C3 do not contain enough semantic information to be helpful, > to my mind... > > - Tony I share David's concerns about the subtle difference between "inter" and "intra", and likewise Tony's about "C1" etc not being mnemonic. How about "G0", "G1", and "G2", for both level of complexity and the number of groups involved? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 ----- End Included Message ----- From owner-mpi-context@CS.UTK.EDU Tue Apr 20 06:04:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09503; Tue, 20 Apr 93 06:04:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16292; Tue, 20 Apr 93 06:03:48 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 20 Apr 1993 06:03:47 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16284; Tue, 20 Apr 93 06:03:43 -0400 Date: Tue, 20 Apr 93 11:03:35 BST Message-Id: <4613.9304201003@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: RE: Names for communicator objects To: Tony Skjellum , mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Mon, 19 Apr 93 17:26:36 CDT Reply-To: lyndon@epcc.ed.ac.uk Cc: d39135@carbon.pnl.gov Well, to pitch in my penny, I had already suggested some kind of name for the two simplest communicator objects in a previous letter. Well, never mind, because I am composing a letter which, among other things, suggests that: 1) The first two ones are called an "intracommunicator" object, which is composed of a context and a group (possibly null group). 2) The third one is called an "intercommunicator" object, which is composed of two intracommunicator objects. I understand the point Dave makes about missing the difference between "er" and "ra" embedded in the string "inxxcommunicator". I don't rate it as importantly as Dave, but that's just my opinion. I would have thought that few enough people educated enough to come into contact with this MPI stuff would not know the difference between the English prefix "intra" and the English prefix "inter" --- I do believe I understood this before leaving high school and my English education was rather poor, and my school never did teach Latin (it did teach Russian :-). Okay, so this is just my opinion, but I really don't think that enumerating the objects is very helpful to the user. If we want to make up concise short names, so users have less to type (which of course is irrelevant to me as I really don't mind typing, as you may have noticed, and at any rate my editor can type the long things for me :-), then imho they really should convey a bit more than just groups. How about ... RACOM - intRACOMmunicator - is Floopy and Bongo as above. ERCOM - intERCOMmunicator - is Bingo as above. Or even ... ACOM - intrACOMmunicator - is Floopy and Bongo as above. ECOM - intErCOMmunicator - is Bingo as above. Or along the cryptic line of thinking ... CG - ContextGroup - is Floopy and Bongo. CGG - ContextGroupGroup - is Bingo. Regards Lyndon > > I would suggest GRP0, GRP1, GRP2, to distinguish between GRIDS > (eg, synonym for virtual topologies) versus GROUPS. > - Tony > > > ----- Begin Included Message ----- > > >From owner-mpi-context@CS.UTK.EDU Mon Apr 19 15:36:10 1993 > X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 19 Apr 1993 16:35:20 EDT > Date: Mon, 19 Apr 93 13:28:56 PDT > From: rj_littlefield@pnlg.pnl.gov > Subject: RE: Names for communicator objects > To: mpi-context@cs.utk.edu, walker@rios2.epm.ornl.gov > Cc: d39135@carbon.pnl.gov > X-Envelope-To: mpi-context@cs.utk.edu > Content-Length: 878 > > > We have sensible names for these things. > > It is part of the proposal X rewrite > > > > The highest level would be inter-group-communicator, the next lowest > > is intra-group-communication, and I am not sure what the lowest > > should be (will review notes). > > > > C1, C2, C3 do not contain enough semantic information to be helpful, > > to my mind... > > > > - Tony > > I share David's concerns about the subtle difference between > "inter" and "intra", and likewise Tony's about "C1" etc not > being mnemonic. > > How about "G0", "G1", and "G2", for both level of complexity > and the number of groups involved? > > --Rik > > ---------------------------------------------------------------------- > rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield > Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 > Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 > > > ----- End Included Message ----- > > /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Apr 20 13:26:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02987; Tue, 20 Apr 93 13:26:47 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA18378; Tue, 20 Apr 93 13:24:45 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 20 Apr 1993 13:24:44 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA18365; Tue, 20 Apr 93 13:24:25 -0400 Date: Tue, 20 Apr 93 18:23:45 BST Message-Id: <4968.9304201723@subnode.epcc.ed.ac.uk> From: L J Clarke To: mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-collcomm@cs.utk.edu Subject: mpi-context; intercommunication etc (long) Dear mpi-context colleagues I previously wrote regarding context management and binding of contexts for intracommunication in the letter [Subject: mpi-context: context management and group binding (long)] and sent out a short correction in the letter [Subject: mpi-context: CORRECTION to previous message] to which I draw your attention. In this letter I wish to briefly revisit and recap the above subjects, then move on to briefly discuss and make a concrete suggestion for intercommunication. This is a long letter. Probably best to print it and read over a coffee. I really must clarify the nature of the context which I am assuming. In this letter contexts are assumed to be global in the sense that if a process P creates a context C, it can send C to another process Q, and Q can both send and receive messages of context C. This is the model adopted by Zipcode, which I view as the exemplar of existing practice regarding message context. Regarding intracommunication I hope to slightly simplify the content of my suggestion compared to the letters referred to above. Regarding intercommunication the particular suggestion I make is motivated by conformity with intracommunication both in the point-to-point syntax class, and in the content of the message envelope. o--------------------o 1. Communicator and Communication ================================= Communicator objects provide point-to-point and collective communication in MPI. A communicator object is a binding of a message context and one or more process worlds. Two subclasses of communicator object are defined below, intracommunicator and intercommunicator. Communicator objects are identified by process local object identifiers. 1.1 Construction, Destruction and Information --------------------------------------------- MPI provides subclass specific communicator constructors described below. MPI provides a subclass generic communicator object destructor procedure. mpi_delete_communicator(id) id is identifier of a communicator purpose deletes the communicator object identified by id See Note 1) under intracommunicator construction and Note 1) under intercommunicator construction Notes: 1) This procedure could be replaced with MPI_FREE if we wish to fit in with the manipulation of communication handles and buffer descriptor handles described in the point-to-point chapter. MPI provides a subclass generic procedure which returns the context identifier of a communicator object. context = mpi_communicator_context(id) context is the context bound to the communicator id is the identifier of a communicator See Note 1) under intracommunicator construction and Note 1) under intercommunicator construction purpose informs the caller of the context bound to a communicator 1.2 Discussion -------------- 2. Intracommunicator and Intracommunication =========================================== Intracommunicator objects provide point-to-point communication between processes of the same process world in MPI. Intracommunicator objects also provide collective communication in MPI. 2.1 Construction and Information -------------------------------- MPI provides a subclass intracommunicator constructor. id = mpi_create_intracommunicator(context, world) id is identifier of created communicator context is message context for communications world is process world of receiver and sender in both send and recv purpose creates an intracommunicator object Notes: 1) The context of an intracommunicator is either an actual context or the null context (MPI_NULL). If the context is an actual context then the call does not synchronise processes in the process world of the intracommunicator. If the context is the null context then the call synchronises the process world of the communicator and creates a context for the communicator. In this case the context is deleted when the communicator is itself deleted calling mpi_delete_communicator, and that call will synchronise the process world. In this case the information procedure mpi_communicator_context will return MPI_NULL to the caller --- the caller is not allowed to have knowledge of the context created. 2) The process world of an intracommunicator object is either an actual process group or the null group (MPI_NULL). If the world is an actual process group then the world is understood to contain all processes composing the process group and the communicator object identifies processes in a relative sense, i.e. as a rank within the process group. If the world is the null group then the world is dunerstood to contain all processes composing the program and the communicator object identifies processes in an absolute sense, i.e. as a process identifier. MPI provides a subclass information procedure which returns the identifier of the world of the intracommunicator. world = mpi_intracommunicator_world(id) world is process world of the communicator id is identifier of created communicator purpose returns the world identifier of the intracommunicator, either an actual group identifier or the null group identifier (MPI_NULL) 2.2 Point-to-point ------------------ I deal with generic "send" and "recv" seperately, and can ignore the multiple flavours thereof. send(id, process, label, ...) id is identifier of intracommunicator object process is identifier of receiver in world of object label is message tag in context of object recv(id, process, label, ...) id is identifier of intracommunicator object, and cannot be wildcard process is identifier of sender in world of object, and can be wildcard label is message tag in context of object, and can be wildcard Notes: 1) The caller must be in the world of the intracommunicator, i.e. either it is the null process group or an actual process group of which the caller is a member. 2.3 Collective -------------- I deal with a generic collective "operation", and can ignore the multiple flavours thereof. operation(id, ...) id is identifier of intracommunicator object Notes: 1) The intracommunicator must have a world which is an actual process group of which the caller is a member. 2.4 Envelope ------------ The message envelope for intracommunication consists of: * sender identifier within process world of communicator (pid or rank) * receiver routing (implementation defined) * message context of communicator * message tag * message length (implementation defined) The sender and reciever must bind the context to the same process world in an intracommunicator, thus the world is determinable. 2.5 Discussion -------------- The facilities for intracommunication, coupled with the context model, provide a convenient and powerful interface for communications which are closed within the scope of a group and for the serial client-server model. The ability to create an intracommunicator without synchronisation of processes simplifies the construction of libraries in highly MIMD programs, and can be used to advantage in conjunction with the association and location facilities described below. 3. Association, Dissociation, Location, Passivation and Activation ================================================================== 3.1 Association, Dissociation and Location ------------------------------------------ These facilities allow the user to bind names to process, group, and context objects. mpi_associate(name, id) name is a string which is the name bound to the given object id is the object identifier (process, group or context) prupose associates name with object identified by id id = mpi_locate(name, wait) id is the object identifier (process, group or context) name is a string which is the name bound to the given object wait is a boolean value determining whether the caller waits for the name to become associated with an object of given class purpose creates a copy of the object associated with name mpi_dissociate(id) id is the object identifier (process, group or context) purpose removes the association of name with object id, and can only be performed by the process which previously associated name. Notes: 1) These facilities are a name service. This could be implemented by a name server process which can run on a host or login node, and need not consume expensive numerical computation resources. 3.2 Passivation and Activation ------------------------------ These facilities allow the user to transmit a process, group and context objects. Passivation and activation produce a "portable" description of the object in a memory buffer (conventionally these operations produce a description in a file, but a memory buffer is more convenient for transmission in a message :-). mpi_passivate(id, buf, len) id is the object identifier (process, group or context) buf is an array of character len is the length of the array buf purpose writes a portable description of object identified by id in the memory buffer buf id = mpi_activate(buf, len) id is the object identifier (process, group or context) buf is an array of character len is the length of the array buf purpose reads a portable description of an object and creates a copy of the object Notes: 1) The detailed type of the memory buffer is not of great importance provided that we define that type. I have used character above, we could choose integer, for example. 3.3 Discussion -------------- I have assumed that MPI can distinguish the class of the object (process, group or context) given the object identifier. If this cannot be the case then we can describe a different set of procedures for each class or we can add a class argument to the above procedures. The name association and location service is the most manageable way of describing which groups communicate with one another. The passivation activation facilities are potentially a building block in the implementation of the name association and location service. Deletion of objects created by activation or location should only delete the process local copy of the object. It should not delete the original copy. When location and activation "create" an object and the object already exists within the calling process, a new object should not be created and the id of the existing object should be returned. This means that such object have multiple references, so we should define the destructors in terms of deleting references to objects, leaving the implementation to delete the object when there are zero references. 4. Intercommunicator and Intercommunication =========================================== Intercommunicator objects provide point-to-point communication between processes of different process worlds in MPI. Intercommunicator objects do not provide collective communication in MPI (yet :-). 4.1 Construction ---------------- id = mpi_create_intercommunicator(context, local_world, remote_world) id is identifier of created communicator context is message context for communications local_world is process world of sender in send and receiver in recv remote_world is process world of receiver in send and sender in recv purpose creates an intercommunicator object Notes: 1) The context can be an actual context or the null context (MPI_NULL). If the context is an actual context then the call does not synchronise processes within the two process worlds of the communicator. If the context is the null context then the call synchronises the two process worlds of the communicator and creates a context for the communicator. In this case the context is deleted when the communicator is itself deleted calling mpi_delete_communicator, and that call will synchronise the process world. In this case the information procedure mpi_communicator_context will return MPI_NULL to the caller --- the caller is not allowed to have knowledge of the context created. 2) Each process world of an intercommunicator object is either an actual process group or the null group (MPI_NULL). If the world is an actual process group then the world is understood to contain all processes composing the process group and the communicator object identifies processes in that world in a relative sense, i.e. as a rank within the process group. If the world is the null group then the world is understood to contain all processes composing the program and the communicator object identifies processes in that world in an absolute sense, i.e. as a global process identifier. MPI provides subclass information procedures which return the identifier of the local_world and remote_world of the intercommunicator. world = mpi_intercommunicator_local_world(id) world is local process world of the communicator id is identifier of created communicator purpose returns the local world identifier of the intercommunicator, either an actual group identifier or the null group identifier (MPI_NULL) world = mpi_intercommunicator_remote_world(id) world is remote process world of the communicator id is identifier of created communicator purpose returns the remote world identifier of the intercommunicator, either an actual group identifier or the null group identifier (MPI_NULL) 4.2 Point-to-point ------------------ I deal with generic "send" and "recv" seperately, and can ignore the multiple flavours thereof. send(id, process, label, ...) id is identifier of intracommunicator object process is identifier of receiver in remote_world of object label is message tag in context of object recv(id, process, label, ...) id is identifier of intracommunicator object, and cannot be wildcard process is identifier of sender in remote_world of object, and can be a wildcard label is message tag in context of object, and can be a wildcard 1) The caller must be in the local_world of the intracommunicator, i.e. either it is the null process group or an actual process group of which the caller is a member. 4.3 Envelope ------------ The message envelope for intercommunication consists of: * sender identifier within process world of communicator (pid or rank) * receiver routing (implementation defined) * message context of communicator * message tag * message length (implementation defined) The sender and reciever must bind the context to the same process worlds in an intercommunicator, thus both the local_world and the remote_world are determinable. This is identical to the envelope of intracommunication. 4.4 Discussion -------------- The facilities for intercommunication, coupled with the context model, and the name service, provide a convenient interface for the parallel client-server model and parallel modular-application software, provided that the WAIT_ANY() facilities of point-to-point communication are fair. The ability to create an intercommunicator without synchronisation of processes simplifies the programming of parallel client-server software, and avoids a dependency graph problem when writing parallel modular-application software in which the module graph contains loops. 5. Discussion ============= I find it a wee bit amusing that an intercommunicator in which local_world and remote_world are the same is no different to an intracommunicator. This suggests to me that either (a) there should only be an intercommunicator class or (b) we think of the intracommunicator class as simply syntactic sugar around the intercommunicator class. The communicator object class names are rather long. Perhaps programmers would prefer shorter names in programs. We could take the approach of deriving names from the list of objects which a communicator binds, for example: "intracommunicator" becomes "CW" as it is a binding of a context and a world; "intercommunicator" becomes "CWW" as it is a binding of a context and a world and another world. On the other hand we could take collections of letters from the long names, for example: "intracommunicator" becomes "RACO" or "ACO"; "intercommunicator" becomes "ERCO" or "ECO". o--------------------o Comments, questions, flames, please! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Tue Apr 20 14:07:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04375; Tue, 20 Apr 93 14:07:05 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22093; Tue, 20 Apr 93 14:06:22 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 20 Apr 1993 14:06:16 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22052; Tue, 20 Apr 93 14:06:10 -0400 Date: Tue, 20 Apr 93 19:06:06 BST Message-Id: <5045.9304201806@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: proposal -- context and tag limits To: rj_littlefield@pnlg.pnl.gov, mpi-context@cs.utk.edu In-Reply-To: rj_littlefield@pnlg.pnl.gov's message of Fri, 9 Apr 93 19:43:26 PDT Reply-To: lyndon@epcc.ed.ac.uk Cc: d39135@carbon.pnl.gov, gropp@mcs.anl.gov, mpi-collcomm@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu Rik writes: > ============ PROPOSAL TO ENVIRONMENT COMMITTEE ============== Yes, I support the spirit and detail of the proposal. > Everyone I know would MUCH prefer suboptimal performance > over HAVING to rewrite applications to conform to varying and > inconsistent hard limits. Yes, this claim is true of everyone I know except for one very small community of academic scientists who will write their relatively simple programs from scratch for every machine on which they will do major scientific production runs. I know a whole lot more academics and commercials who just will not write programs from scratch in this way. > Yes, I recall the many arguments against mandating specific > limits. But, I claim that those arguments are misdirected. Indeed I believe that your claim is valid. > I believe that we will not meet our goal of portability > if we do not specify usable limits on tag and context values. I have the same belief. I also believe that if we fail on portability then we fail period. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Wed Apr 21 08:36:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23891; Wed, 21 Apr 93 08:36:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11515; Wed, 21 Apr 93 08:34:50 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 21 Apr 1993 08:34:49 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11507; Wed, 21 Apr 93 08:34:48 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17335; Wed, 21 Apr 1993 08:34:47 -0400 Date: Wed, 21 Apr 1993 08:34:47 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9304211234.AA17335@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Names for communicators I think either C0, C1, C2 or G0, G1, G2 are the best names for communicator objects. Things like "RACOM" and "ERCOM" are not suitable. Does the C0 (aka Floopy) communicator still exist in the latest proposal X. In pt2pt communication using C0 we use process handle to designate source and destination, whereas with C1 and C2 the rank is used. Does this imply we need separate pt2pt routines for C0 communicators, thereby doubling the number of pt2pt routines? David From owner-mpi-context@CS.UTK.EDU Wed Apr 21 08:57:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24371; Wed, 21 Apr 93 08:57:41 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14688; Wed, 21 Apr 93 08:57:09 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 21 Apr 1993 08:57:09 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14674; Wed, 21 Apr 93 08:57:04 -0400 Date: Wed, 21 Apr 93 13:56:02 BST Message-Id: <5987.9304211256@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Names for communicators To: walker@rios2.epm.ornl.gov (David Walker), mpi-context@cs.utk.edu In-Reply-To: David Walker's message of Wed, 21 Apr 1993 08:34:47 -0400 Reply-To: lyndon@epcc.ed.ac.uk Hi David > I think either C0, C1, C2 or G0, G1, G2 are the best names for communicator > objects. Things like "RACOM" and "ERCOM" are not suitable. Okay, whatever you think. > Does the C0 (aka Floopy) communicator still exist in the latest proposal X. Hard to say :-) In the letter (long) I posted yesterday I spoke only of two subclasses, which I called intracommunicator and intercommunicator, and which we can call C1, C2. The capability of the C0 (aka Floopy) is in the suggestion of the letter, for sure. The question is how things end up presented in X (aka malcolm :-), which we (Tony, Rik, myself, ...) have not yet discussed. > In pt2pt > communication using C0 we use process handle to designate source and > destination, whereas with C1 and C2 the rank is used. Yes, although we haven't really made a decision about whether its a global process identifier (pid) or a local process identifier (handle), or whether (my preference) we define it such that it can be implemented either way. > Does this imply > we need separate pt2pt routines for C0 communicators, thereby doubling the > number of pt2pt routines? > I think not, and this is reflected in the letter of yesterday. The way I see it, a process identifier or process handle should be expressed as an integer which is the same type as a rank. Thus the pt2pt syntax class is the same for all of the cases, and we do not need to multiply the number of pt2pt routines, if we use genericity and overloading. In essence we describe pt2pt as (communicator, process-label, message-label, ...) where communicator is an instance of a Communicator object process-labek is a process label interpreted according to communicator as a process identifier or rank label is a message label which is a tag in the context of the communicator > David > /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Fri May 7 14:28:21 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA01084; Fri, 7 May 93 14:28:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21927; Fri, 7 May 93 14:27:59 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 7 May 1993 14:27:58 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21894; Fri, 7 May 93 14:27:08 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA27019; Fri, 7 May 93 13:27:41 CDT Date: Fri, 7 May 93 13:27:41 CDT From: Tony Skjellum Message-Id: <9305071827.AA27019@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: Re: mpi-context: context and group (longer) Cc: mpi-collcomm@cs.utk.edu For your information, the following was sent (by me) to the IAC subcommittee. - Tony ----- Begin Included Message ----- From tony Fri May 7 13:25:17 1993 Date: Fri, 7 May 93 13:24:37 CDT From: Tony Skjellum To: mpi-iac@CS.UTK.EDU, stoessel@irsun21.ifp.fr Subject: Re: Subset comments Cc: tony Content-Length: 6858 IACers, I think that a subset is essential, because of the number of features in MPI that are so remote from current practice; furthermore, if one were to look at the liberal vs. conservative nature of committee (as others have observed), it is not equal over all features/proposed features. Hence, I offer my thoughts. This is not meant to be a "flame." For instance, I have argued for the addition of a two-word match against tags, in order to allow easier layerability. A tag would be matched as follows (following Jim Cownie): (received_tag xor (not dont_care_bits)) and care_bits This would allow the user, not only complete freedom in use of tags, but also the ability to develop further layers on top of MPI that partition the use of tag. I will bring this idea up again at the next meeting, at the second reading of pt-2-pt. It is necessary to have this, and it is a small step away from usual practice. However, it is hard to convince people to add this, despite its negligible impact on performance (two logical operations, instead of one, assuming user passes the one's complement of the dont_care_bits.) However, its impact on MPI flexibility is immense. Hence, I view this feature as essential to the subset and full MPI likewise. Another instance. Contexts. We are arguing for/not for contexts that are independent of groups. Contexts as an extended, system-registered part of the tag field help us to build libraries that can co-exist, register at runtime, and do not interfere with the message-passing of other parts of the system. I want an "open system"; hence, I want to see the tag partitioning. Contexts work very well in Zipcode (my message passing system, developed at Caltech, LLNL), and are helpful with the libraries we develop on Zipcode. Because vendor systems do not have contexts, Zipcode, when it layers on vendor systems must requeue messages. This is undesirable from a performance standpoint. Hence, it is highly desirable for MPI to provide contexts of the type I describe, as a simple tag registration/partitioning mechanism that is understandable as an extension of existing practice. If contexts are limited, and their is a mechanism to find this out (environment), then messaging systems like Zipcode could do requeueing of messages as necessary, or manage contexts themselves at times, and use the precious "fast" contexts on user-specified communications, leaving others to be requeued and slower. Hence, I view contexts (whether plentiful or scarce in a given implementation) as essential to the subset, and the full standard. As Don Heller of Shell has noted... "contexts allow the development of a software industry [for multicomputers]." Groups. Yes, we need them too. They are important for managing who is communicating with others. So, they have to stay in the subset as well. Rik Littlefield, Lyndon Clarke, and I have argued (and will continue to do so) for attributes based on group/context-scope. This would allow the methods implementing communication to be changed in MPI for each group/context scope, permitting optimizations. This is not current practice, except in our Zipcode 1.0 release, which has this useful capability, but it justifiably useful. I think these ideas can/should remain in the standard and in the subset. There are multitudinous types of send/receive that we are currently proposing, but not using in practice currently, which have been proposed and accepted with relative ease by MPI. Practically, send, receive, receive unblocked, is enough, provided the kernel is smart enough to do overlapping of communication and computation. Actually, if the semantics of the Reactive Kernel were taken, which allows the system to handle all memory management, then receive would provide the pointer to the data, and send would be like free, with an allocate mechanism like malloc. These reduce the number of copies of data, except when extremely regular data structures are in use (less and less likely). The RK semantics thought out by Seitz et al are remarkably simple, but highly optimizable, and can even work very fast in shared memory. These semantics do not appear as options in MPI; we only have multitudinous buffer-oriented operations. When memory management units are involved, binding control of the memory and messaging operations gives even more opportunity for the system to optimize. Allowing the user to receive messages without having first to know their size is elegant, and simplifies error issues. As we all know, there are faster implementation strategies than RK semantics for message-passing that are low level, such as channels, active messages (unchampioned in this standard), and shared-memory (eg, CRAY Tera 3D). These need not be part of this standard, but it would be helpful if the standard were unhostile to such possibly efficient implementation mechanisms. The "buffer descriptor" approach in MPI is the best match (being the highest level interface) to a runtime system that exploits channels, and/or active messages, and/or remote memory writes, etc. The optimizability of the highest level is complemented by the fact that the user no longer knows if a buffer is ever formed on the local or remote end [well it should be written to make that so]. Furthermore, heterogeneity can be encapsulated in transfers at this level. Hence, I am convinced that "buffer descriptor" stuff should remain in the subset. The committee has shied away from defining the process model, and this has led not only to a very static model (arguably OK), but a predilection to the SPMD (handling of groups, definition of subgroups, need for dynamic contexts diminished, etc). All of these factors make the standard backward-looking if so adopted, and make it really difficult to justify in the distributed environment. I am not sure why this has happened, but it is unfortunate. It means that MPI codes will be partially portable, but not totally, as each system will have different process management. SPMD programs will be reasonably portable, as the process management is simple, and therefore localized. The handling of the host/node model is not well established in MPI, and may not be suitably supported. That would be a big problem to my mind. To summarize, it is my view that the enabling mechanisms: group, context, tag selection, and buffer descriptors described above are essential aspects of a standard and subset, and should not be sacrificed. MPMD programming, the host/node model, should be supported. - Tony . . . . . . . . . . "There is no lifeguard at the gene pool." - C. H. Baldwin "In the end ... there can be only one." - Ramirez (Sean Connery) in Anthony Skjellum, MSU/ERC, (601)325-8435; FAX: 325-8997; tony@cs.msstate.edu ----- End Included Message ----- From owner-mpi-context@CS.UTK.EDU Sun May 9 22:31:43 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA03781; Sun, 9 May 93 22:31:43 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00862; Sun, 9 May 93 22:31:06 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Sun, 9 May 1993 22:31:05 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00810; Sun, 9 May 93 22:30:38 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA29012; Sun, 9 May 93 21:31:10 CDT Date: Sun, 9 May 93 21:31:10 CDT From: Tony Skjellum Message-Id: <9305100231.AA29012@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Subject: Mea culpa on previous letter to Ho/Pierce; Question about proposal VIII (Sears) Cc: tony@CS.UTK.EDU Dear friends/colleagues: 1) I think I misunderstood who was saying what in my last message, re Ho/Pierce. But, I hope my point was clear. I disagree with Howard on the concept that the context is convenient, but not important/essential. I thank Paul for his examples. If I was sounding inflamed, ignore that, I had low blood sugar. I think that the context should be a separate, logical extension to the tag field, the latter should be 32 bits long, for sufficient flexibility in layering, and the tag matching should permit layering by selective bit inclusion/exclusion. Please see that my tag matching and context/tag (as a logical partitioning) are correlated. The MPI layer partitions the context field from the "logical tag," and controls it. The user has control over the format of the rest of the tag, and may build layers (or use other layers), that subsequently partition the rest of the tag to continue to build safe service layers. That is why I keep pounding the receipt selectivity mechanism based on the dont_care bits, and the care_bits. I do not view the context as a large quantity in some implementations. In certain cases, there might be as few as 16, but the implementer (like a compiler writer using registers), allows the code to work if the hard limit is exceeded, by reserving a context that supports a higher-cost propocal (eg, requeueing of messages). ------------------------------------------- 2) Lyndon suggested some time ago that I revive my two-dimensional grid example, vis a vis Rik Littlefield's collections of examples, and ask how that applies to Mark Sear's proposal VIII... Basically, in this case, there is a two-dimensional logical array of processes (a virtual topology), of shape PxQ. In a given position (p,q), there are three possible contexts for a given global operation G (eg, combine): 1) G over the whole collective, PxQ topology 2) G over the row that includes process (p,q) [the pth row] 3) G over a column that includes process (p,q) [the pth column] How many contexts are needed to provide for safe intermixing of operation G over the three possible combinations. Assume that the operations of 1, 2, 3 may operate in sequence correctly, for now (ie, two G's over the whole collection work correctly). This is true of multiple combines, deterministic broadcasts, etc. It is not true if there are non-deterministic G operations included. If there are three contexts of communication, then everything is fine, because row G, col G, and whole G cannot interfere. By extension, a total of three unique contexts are needed for the whole PxQ topology, reusing the same context on parallel (row, column) entities, and an additional one for the "whole." I pointed up this example at an earlier MPI meeting. In the Proposal VIII model, contexts and groups are disjoint, so no property of group will provide the safety needed. Hence, I assert that either three contexts should be needed here, or tag values would be needed to disambiguate the G operations. Now, in an earlier mail message, Ho/Pierce bantered about "how many contexts are needed for a given group." I argued, one per unique library, with the discussion of non-deterministic broadcast, because I wanted my non-deterministic broadcast to work safely when other communication was going on. At least, there must be a point-to-point context for a group, and a global operations context for a group [which is what Zipcode 1.0 has]. Further non-deterministic operations would need more contexts. In the light of this issue, I would suggest that my PxQ topology above reasonably needs 6 contexts of communication to be implemented safely [and assuming that all global operations are deterministic, as asserted]. In short, six contexts are needed for this group. In the PxQxR three-dimensional case, the number is larger: 1) G over the whole collective, PxQxR topology 2) G over PxQ plane 3) G over PxR plane 4) G over QxR plane By simple study, this requires 1 + 3 * 6 = 19 contexts to be implemented safely, and sets a limit on the number of contexts that one would want to have, to support a single 3D topology safely. Remember, I am assuming that we are talking about Proposal VIII, which does not offer group-based safety for messaging. Proposal I, by Snir, does offer this safety inherently [which masks the need for explicit contexts when dealing with SPMD-type calculations]. I am arguing that the assumption of small numbers of contexts (where small is about 8 or 16, and where the code breaks if the number is exceeded) is not reasonable. The code must continue to work, if slower, if many contexts are needed, beyond that which is available as fast, hardware contexts. Proposal VIII must accomodate that to be reasonable. In an application at Livermore, one of my colleagues uses multiple two-dimensional topologies to implement stages of a calculation. They overlap. So, he wants at least two unique topologies, completely supported, or at least 12 contexts. If he goes to three dimensions, he want 38 minimum, for his code to work. Summary: Proposal VIII cannot cope with practical situations illustrated above, because it breaks when there are "too many contexts," and contexts are assumed rare (eg, 8 or 16). To be reasonable, it would have to have a fall-back to support many contexts in some way. Lyndon and others have made this request, in the past weeks. Comments? - Tony From owner-mpi-context@CS.UTK.EDU Mon May 10 03:03:53 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA03996; Mon, 10 May 93 03:03:53 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14410; Mon, 10 May 93 02:59:44 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 10 May 1993 02:59:42 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14402; Mon, 10 May 93 02:59:37 -0400 Message-Id: <9305100659.AA14402@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6287; Mon, 10 May 93 03:00:12 EDT Date: Mon, 10 May 93 03:00:11 EDT From: "Marc Snir" To: MPI-CONTEXT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: CON-03.DVI.* %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 16 df15 D E /Fb 58 123 df35 D<13E01201EA07C013005A121E5A123812781270A312F05AA77E1270A312781238123C7E7E7E13 C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780120313C01201A313E01200A7 120113C0A3120313801207EA0F00121E121C127C12F05A0B217C9C16>II<12 38127C127EA2123E120E121E123C127C12F81260070B798416>44 DI< 127012F8A312700505788416>I III<1238127CA312381200A81238127CA3123C121C123C123812F812 F012600618799116>59 D61 D<13E0487EA213B0A2EA03B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF 13E0007F13C013197F9816>65 DI<3801F180EA07 FF5AEA1F0FEA3C0712781303127000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C 5AEA01F011197E9816>II<387FFFC0B5FC7EEA1C01A490C7FCA2 131CA2EA1FFCA3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816>I71 D73 D<387F0FE038FF8FF0387F0FE0381C0780EB0F00131E131C133C5B5BEA1DE07F121F7F1338EA1E 3C131CEA1C1E7F7F14801303387F07E038FF8FF0387F07E01419809816>75 DI<38FC07E0EAFE0FA2383A0B80EA3B 1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313197F9816>I<387E1FC038FF3FE038 7F1FC0381D07001387A313C7A2121CA213E7A31367A21377A21337A31317EA7F1FEAFF9FEA7F0F 13197F9816>III82 DI<387FFFE0B5FCA2EAE0E0A400 001300AFEA07FC487E6C5A13197F9816>I<387F07F038FF8FF8387F07F0381C01C0B0EA1E0300 0E1380EA0F8F3807FF006C5AEA00F81519809816>I<38FC07E0EAFE0FEAFC07387001C0A30030 1380EA3803A313E3EA39F3A213B300191300A61313EA1B1BEA0F1EA2EA0E0E13197F9816>87 D<387F1F80133F131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B81207 131CA2EA0E0EA2487E387F1FC000FF13E0007F13C013197F9816>I<38FE0FE0EAFF1FEAFE0F38 1C0700A2EA0E0EA26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I<387FFF 80B5FCA238E007005B131E131CEA003C5B137013F0485A5B1203485A90C7FC5A381E0380121C12 3C12781270B5FCA311197E9816>I95 D97 D<127E12FE127E120EA4133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13 F014E01381EB83C013FF000E1300EA067C1419809816>II<133F5B7F1307A4EA 03E7EA0FFF123FEA3C1F487E1270EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807 C7E014197F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA 7FFFB5FCA2EA01C0ACEA3FFE487E6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E 00EA380EA3EA3C1E6C5AEA1FFC485AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001 E0EAE000A3EAF001387C07C0383FFF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4 137CEA0FFF148013871303A2120EA9387FC7F038FFE7F8387FC7F01519809816>II<127E12FE127E120EA4EB7F E0A3EB0F00131E5B5B5BEA0FF8A213BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F87141980 9816>107 DI<38FBC78038FFEFC0EBFFE0EA3E7CEA3C 78EA3870AA38FE7CF8A31512809116>IIII<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F 9116>114 DI<12035AA4EA7FFFB5FCA20007C7FCA75BEB03 80A2130713873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F 0FEBFFF06C13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E 0EA36C5AA4EA03B8A3EA01F0A26C5A13127F9116>I<38FF1FE013BF131F38380380A413E33819 F300A213B3EA1DB7A4EA0F1EA313127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813 F012016C5A12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>I<38 7F1FC038FF9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35B A3485A1279127BEA7F806CC7FC123C131B7F9116>I<383FFFC05AA238700780EB0F00131EC65A 13F8485A485A485A48C7FC381E01C0123C1278B5FCA312127F9116>I E /Fc 48 123 df11 D<127012F812FCA2127C120CA31218A2123012601240060D 7D9C0C>39 D<13C0EA0180EA03001206120E120C121C121812381230A21270A21260A212E0AC12 60A21270A21230A212381218121C120C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E 7E121C120C120E120612077EA21380A21201A213C0AC1380A21203A21300A25A1206120E120C12 1C12185A5A5A0A2A7E9E10>I<127012F012F8A212781218A31230A2127012601240050D7D840C> 44 DI<127012F8A3127005057D840C>I<12035A123FB4FC12C71207B3 A3EAFFF8A20D1C7C9B15>49 DII<13F0 EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EAFC1CEAF81E130E12F013 0FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>54 D<127012F8A312701200A812 7012F8A3127005127D910C>58 D<127012F8A312701200A8127012F012F8A212781218A31230A2 127012601240051A7D910C>I<1306130FA3497EA4EB33C0A3EB61E0A3EBC0F0A338018078A2EB FFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C1D7F9C1F>65 D69 D<39FFF3FFC0A2390F003C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F> 72 DI76 D80 D<3807E080EA1FF9EA3C1FEA7007130312E01301 A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA07FFC61380130FEB03C0A2130112C0A300E013801303 00F01300EAFC0EEACFFCEA83F8121E7E9C17>83 D<007FB512C0A238780F030070130100601300 00E014E000C01460A400001400B03803FFFCA21B1C7F9B1E>I<3AFFE0FFE1FFA23A1F001E007C 6C1530143FA20180147000079038678060A32603C0E713C0ECC3C0A2D801E0EBC1809038E181E1 A3D800F3EBF3001400A2017B13F6017E137EA3013C133CA3011C133801181318281D7F9B2B>87 D97 D<12FCA2121CA9137EEA1DFF381F8780381E01C0001C13E013 0014F0A614E01301001E13C0381F07803819FF00EA187C141D7F9C17>IIII<1378EA01FCEA039EEA071EEA0E0C1300A6EAFFE0A2EA0E 00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F07381E0380A2121CAB38FF9FF0 A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C0C>II<12FC A2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA1C781338133C131C7F130F38FF9F E0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF9FF8391F83 B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124>IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0A612781307123CEA1E1FEA0FFB EA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA2123CEAFFE0A2EA1C00A81330A5EA 1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA03E314127F 9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA31312 7F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B061486A2EB318E000713CCA2136038 03E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038070600EA038EEA01DC13D8EA00 F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8A21512809116>I<38FF0FE0A2 381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA35BA3EAF180A200C7C7FC12 7E123C131A7F9116>II E /Fd 18 118 df<1238127C12FEA312 7C12381200A41238127C12FEA3127C123807127D910D>58 D68 D73 D<3BFFFC7FFE0FFCA23B0FC007E000C081D9 E003130100071680EC07F8D803F0EC0300A29039F80CFC0700011506EC1CFE9039FC187E0E0000 150CEC387F90397E303F18A290397F601FB8013F14B002E013F0ECC00F011F5CA26D486C5AA2EC 00036D5CA22E1C7F9B31>87 D97 D99 D101 D<3803F0F0380FFFF8383E1F38383C0F30007C13 80A4003C1300EA3E1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F0 0078A3007813F0383E03E0381FFFC03803FE00151B7F9118>103 D<121E123FA25A7EA2121EC7 FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>105 D108 D<39FF1FC0FE90387FE3FF3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F91 28>I<38FF1FC0EB7FE0381FE1F0EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1FC0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7F E0EB1F8090C7FCA6EAFFE0A2171A7F911B>I114 DI<1203A35AA25AA2123FEA FFFCA2EA1F00A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F 03EBFEFFEA03F818127F911B>I E /Fe 55 126 df<137013F01201EA03C0EA0780EA0F00121E 121C123C123812781270A212F05AA87E1270A212781238123C121C121E7EEA0780EA03C0EA01F0 120013700C24799F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370 A813F013E0A2120113C0120313801207EA0F00121E5A5A5A12600C247C9F18>I<123C127E127F A3123F120F120E121E127C12F81270080C788518>44 D<127812FCA412780606778518>46 D48 DII<123C127EA4123C1200A812 38127C127EA3123E120E121E123C127812F01260071A789318>59 D<387FFFC0B512E0A3C8FCA4 B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4 380FFF80A3EA0E03A2381C01C0A2387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA 1E03EA3C01EA78001270A200F013005AA87E007013E0A21278EA3C01001E13C0EA0F073807FF80 6C1300EA01FC131C7E9B18>I69 DI<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7 FC5AA5EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C 7E9B18>I73 D76 D<38FC01F8EAFE03A2383B06E0A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03 F8A3151C7F9B18>I<387E07F038FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A2 13311339A31319A2131D130DA3EA7F07EAFF87EA7F03151C7F9B18>III82 D<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A290C7FC7E1278123FEA1FF0EA0FFEEA01FF38 001F80EB03C0EB01E01300A2126012E0130100F013C0EAFC07B512801400EAE7FC131C7E9B18> I<387FFFF8B5FCA238E07038A400001300B2EA07FFA3151C7F9B18>I<38FF07F8A3381C01C0A4 380E0380A4EA0F0700071300A4EA038EA4EA018C13DCA3EA00D813F8A21370151C7F9B18>86 D<38FE03F8A338700070A36C13E0A513F8A2EA39DCA2001913C0A3138CEA1D8DA4000D13801305 EA0F07A2EA0E03151C7F9B18>I<387F8FE0139F138F380E0700120FEA070E138EEA039C13DCEA 01F8A26C5AA2137013F07F120113DCEA039E138EEA070F7F000E13801303001E13C0387F07F038 FF8FF8387F07F0151C7F9B18>I<38FF07F8A3381C01C0EA1E03000E1380EA0F0700071300A2EA 038EA2EA01DCA213FC6C5AA21370A9EA01FC487E6C5A151C7F9B18>I95 D97 D<127E12FE127E120EA5133EEBFF 80000F13C0EBE3E0EB80F0EB00701478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E13 8038067E00151C809B18>IIII< EB1FC0EB7FE013FFEA01F1EBC0C01400A3387FFFC0B5FCA23801C000AEEA7FFFA3131C7F9B18> I<3803F1F03807FFF85A381E1F30383C0F00EA3807A5EA3C0FEA1E1EEA1FFC485AEA3BF00038C7 FC123CEA1FFF14C04813E0387801F038F00078481338A36C1378007813F0EA7E03383FFFE0000F 13803803FE00151F7F9318>I<127E12FE127E120EA5133FEBFF80000F13C0EBE1E013801300A2 120EAA387FC3FC38FFE7FE387FC3FC171C809B18>II<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B 5B5BEA0FF87F139C131EEA0E0FEB0780130314C038FFC7F8A3151C7F9B18>107 DI<387DF1F038FFFBF86CB4 7E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBFBF80397F1F1F001914819318>IIII<387F87E038FF9FF8EA7FBF3803FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A 15147F9318>114 DI<487E1203A4387FFFC0 B5FCA238038000A9144014E0A21381EBC3C0EA01FF6C1380EB7E0013197F9818>I<387E07E0EA FE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13FE3801FCFC1714809318>I<387F8FF000FF13 F8007F13F0381E03C0000E1380A338070700A3EA038EA4EA01DCA3EA00F8A2137015147F9318> I<38FF8FF8A3383800E0A3381C01C0A2137113F9A213D9A2380DDD80A3138DEA0F8FA238070700 15147F9318>I<387F8FF0139F138F38070700138EEA039EEA01DC13F81200137013F07FEA01DC EA039E138EEA0707000F1380387F8FF000FF13F8007F13F015147F9318>I<387F8FF000FF13F8 007F13F0380E01C0EB0380A21207EB0700A2EA03871386138EEA01CEA2EA00CCA213DC1378A313 70A313F05B1279EA7BC0EA7F806CC7FC121E151E7F9318>I<383FFFF05AA2387001E0EB03C0EB 078038000F00131E137C5B485A485AEA0780380F0070121E5A5AB512F0A314147F9318>II<127CB47E7FEA07E01200AB7FEB7FC0EB3FE0A2EB7FC0EBF0005BAB1207B45A5B007CC7 FC13247E9F18>125 D E /Ff 25 124 df<1238127CA2127E123E120CA31218A21230126012C0 1280070E789F0D>39 D<1230127812F81278127005057C840D>46 D<130C131C13FCEA0FF81338 1200A41370A613E0A6EA01C0A61203EA7FFE12FF0F1E7C9D17>49 D<133FEBFFC03801C1E03803 00F000061378A2000F137C1380A2EB00781206C712F814F0130114E0EB03C0EB0780EB0F00131C 5B5B13C0485A380300601206001C13C05AEA7FFFB51280A2161E7E9D17>I<137F3801FFC03803 83E0EA0701EB00F05A1301A2000013E0A2EB03C0EB0780EB0F0013FE13F8130E7F1480EB03C0A3 EA3007127812F8A238F00F8000C01300EA601EEA383CEA1FF8EA07E0141F7D9D17>II<14181438A21478147C14FCA2EB01BCA2 EB033C143EEB061EA2130CA21318141F497EA21360EB7FFF90B5FCEBC00F3901800780A2EA0300 A21206A2001F14C039FFC07FFCA21E207E9F22>65 D<0007B5FC15C039003C01E015F090387800 F8A515F0EBF001EC03E0EC07C0EC0F809038FFFE00ECFF803901E007C0EC03E0A21401A215F0D8 03C013E01403A2EC07C0A2EC0F800007EB3F00387FFFFEB512F01D1F7E9E20>I<903803F80890 380FFE1890383F073890387801F83801F000D803C013F0000714705B48C7FC5A121E003E146012 3C007C1400A45AA415C01278EC0180127C003CEB0300A26C13065C6C6C5A3807E0703801FFC06C 6CC7FC1D217B9F21>I<0007B5FC15E039003C01F0EC00F849137C153C151EA3151F5BA6484813 1E153EA3153C157C4848137815F0A2EC01E0EC03C0EC0F800007EB3F00387FFFFCB512E0201F7E 9E23>I<0007B512F8A238003C001578491338A5EC0C30EBF0181500A21438EBFFF8A23801E070 1430A2151815301400485A1560A215E0EC01C014030007130F007FB51280B6FC1D1F7E9E1F>I< 3A07FFC7FFC0A23A003C007800A2495BA649485AA490B5FCA23901E003C0A64848485AA6000713 0F397FFCFFF8485A221F7E9E22>72 D<3807FFE0A238003C00A25BA65BA6485AA6485AA61207EA FFFCA2131F7F9E10>I<3A07FFE0FFE0A23A003C003E001538495B5DEC01804AC7FC14065C495A 5C5CEBF1F013F3EBF778EA01EC13F8497E13E080A248487EA36E7EA26E7E0007497E397FFC1FFC 00FF133F231F7E9E23>75 D<3807FFF0A2D8003CC7FCA25BA65BA6485AA3EC0180A2EC0300EA03 C0A25C1406140E141E0007137E387FFFFCB5FC191F7E9E1C>II<3A07FC03FFC0A23A003E007C001538016F1330A3EB6780A2EB63C001C35BA2EB C1E0A2EBC0F0A2D801805B1478A2143CA33903001F80A2140FA3481307D80F8090C7FC387FF003 12FF221F7E9E22>II<0007B5FC15C039003C03E0 EC01F0EB780015F8A59038F001F0A215E0EC03C0EC0F809038FFFE004813F801E0C7FCA5485AA6 1207EA7FFC12FF1D1F7E9E1F>I<3807FFFC14FF39003C07C0EC03E0EB780115F0A415E0EBF003 15C0EC0780EC1F00EBFFFC14F03801E038143C141CA2141EA23803C03EA41506A20007140C397F FC1F1800FFEB0FF8C7EA03E01F207E9E21>82 DI<001FB512F8A2381E03 C000381438EB078012300070141812601538153038C00F0000001400A5131EA65BA6137C381FFF F05A1D1F7B9E21>I<39FFF007FEA2390F0001F0EC00C0A2EC0180EA0780EC03005C1406140EEB C00C0003131C14185CA25CEA01E05CA2EBE180A2D800F3C7FCA213F6A213FCA21378A21370A21F 207A9E22>86 D<3A03FFC1FFC0A23A003E007C00011E137015606D5B1401EC8380010790C7FC14 C6EB03CC14FC6D5A5C1300801301497E143C1306497E131CEB381FEB300F01607FEBC007000180 38038003D80FC07F39FFF01FFE13E0221F7F9E22>88 D123 D E /Fg 30 122 df<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA212 01B3A7387FFFE0A213207C9F1C>49 DI I<14E013011303A21307130F131FA21337137713E7EA01C71387EA03071207120E120C12181238 127012E0B512FEA2380007E0A7EBFFFEA217207E9F1C>I67 D70 D76 D 85 D97 D II I<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA212 7E003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F0007B512 80380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC 121CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6C B45A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A2 0C247FA30F>I108 D<3AFF87F00FE090399FFC3FF8 3A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFFF1FFE3FFC0A22A167E952F>I< 38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0A21A167E951F>I<13FE3807FF C0380F83E0381E00F0003E13F848137CA300FC137EA7007C137CA26C13F8381F01F0380F83E038 07FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEBC03E497E1580A2EC0FC0A8EC1F 80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A21A207E951F>I114 DI<13C0A41201A212031207120F121FB5FCA2EA0FC0ABEBC180A512 07EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF14FEA2EA0F833907FF7FC0EA01 FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015B A26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E>I<3AFFE3FF87F8A23A1F807C 00C0D80FC0EB0180147E13E0000790387F030014DF01F05B00031486EBF18FD801F913CC13FB90 38FF07DC6C14F8EBFE03017E5BA2EB7C01013C5BEB380001185B25167F9528>I<39FFF01FE0A2 390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA2 6DC7FCA2130EA2130CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B207F951E>121 D E /Fh 15 122 df97 DI<137E48B4FC38038380EA0F07121E001C1300EA3C0248C7FCA35AA5EA70 021307EA381EEA1FF8EA07E011147C9315>I<1478EB03F814F0EB0070A314E0A4EB01C0A213F1 EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133CEA707C3878FCC0EA3F CF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB0300EA780EEA7FFC13E0 00F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I103 DI<136013F0A213E01300A7120FEA1F80123113C0EA6380 A212C3EA0700A3120EA3EA1C301360A2EA38C01218EA1F80EA0F000C1F7D9E0E>I108 D<381E07C0383F1FE03833B870EA63E0EBC038138000C71370EA0700A3000E13E0A3EB01C3001C 13C6A2EB038C1301003813F8381800F018147D931A>110 D<137C48B4FC38038380380F01C012 1E001C13E0123C1278A338F003C0A3EB07801400EA700F131EEA3838EA1FF0EA07C013147C9317 >I114 D116 D<38078780380FCFC03818F8E0EA3070EA6071A238C0 E1C03800E000A3485AA30071136038F380C0A238E3818038C7C300EA7CFEEA387C13147D9315> 120 D<000F1360381F8070003113E013C0EA6380A238C381C0EA0701A3380E0380A4EB0700A25B 5BEA07FEEA03EEEA000EA25B12785BEA7070EA60E0EA3FC06CC7FC141D7D9316>I E /Fi 2 16 df0 D15 D E /Fj 1 111 df<380F07C0381F8FE03831D8703861 F03813E013C000C35BEA0380A348485AA3903801C180000EEBC300EB03831486EB018E001C13FC 380C00F019147F931B>110 D E /Fk 52 123 df<90390FF03FFC90387FFDFF9038F83FE0D801 E013810003137FD807C01300806E137CA5B712FCA23A07C01F007CB03B3FF8FFE3FF80A2292080 9F2C>15 D<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123CA25AA412F85AA97E12 78A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA114>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA01E0A2 13C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<1238127C12FE12FFA2127F123B12 03A21206A2120E120C12181270122008107C860F>44 D<146014E0130114C0A213031480A21307 14005B130EA2131E131C133C1338A213781370A213F05B12015BA212035BA2120790C7FC5A120E A2121E121C123C1238A212781270A212F05AA2132D7DA11A>47 D<137013F0120F12FF12F31203 B3A4B51280A2111D7C9C1A>49 DI<14E0A2497EA3497EA2497EA2497E130CA2EB187FA201307F143F01707F EB601FA201C07F140F48B57EA2EB800748486C7EA20006801401000E803AFFE01FFFE0A2231F7E 9E28>65 DI<9038 07FC0290383FFF0E9038FE03DE3903F000FE4848133E4848131E485A48C7120EA2481406127EA2 00FE1400A7127E1506127F7E150C6C7E6C6C13186C6C13386C6C13703900FE01C090383FFF8090 3807FC001F1F7D9E26>I69 DI<903807FC0290383FFF0E9038FE03DE3903F000FE4848133E48 48131E485A48C7120EA2481406127EA200FE91C7FCA591387FFFE0A2007E903800FE00A2127F7E A26C7E6C7E6C7E3803F0013900FE03BE90383FFF1E903807FC06231F7D9E29>I73 D75 DIIIII82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400 B4FC13F86CB4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200 F813F038FE01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060 140CA200E0140E00C01406A400001400B10007B512C0A21F1E7E9D24>II87 DII<003FB51280A2EB807F393E 00FF00383801FEA248485A5CEA6007495AA2C6485A495AA2495A91C7FC5B485AEC0180EA03FCEA 07F8A2380FF00313E0001F140048485A5C48485A38FF007F90B5FCA2191F7D9E20>I97 DIII II<3801FC3C3807FFFE380F07DEEA1E03003E13E0A5001E13C0380F0780EBFF00EA19FC 0018C7FCA2121C381FFF8014F06C13F8003F13FC387C007C0070133E00F0131EA30078133CA238 3F01F8380FFFE000011300171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121FB0EAFFE0A20B217E A00E>I107 DI<3AFE0FE03F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB00 7CAC3AFFE3FF8FFEA227147D932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3 FFA218147D931D>I<48B4FC000713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA2 6C13F8A2381F83F03807FFC00001130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB00 7EA2143E143FA6143E147E147CEB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D> I114 DII<38FF07F8A2EA1F00AD1301A2EA0F07 3807FEFFEA03F818147D931D>I<39FFE07F80A2391F001C00380F8018A26C6C5AA26C6C5AA26C 6C5AA213F900005B13FF6DC7FCA2133EA2131CA219147F931C>I<3AFFE7FE1FE0A23A1F00F007 006E7ED80F801306A23907C1BC0CA214BE3903E31E18A23901F60F30A215B03900FC07E0A29038 7803C0A3903830018023147F9326>I<38FFE1FFA2380F80706C6C5A6D5A3803E180EA01F36CB4 C7FC137E133E133F497E136FEBC7C0380183E0380381F0380701F8380E00FC39FF81FF80A21914 7F931C>I<39FFE07F80A2391F001C00380F8018A26C6C5AA26C6C5AA26C6C5AA213F900005B13 FF6DC7FCA2133EA2131CA21318A2EA783012FC5BEAC0E0EAE1C0EA7F80001EC8FC191D7F931C> I<383FFFE0A2383C0FC0EA381F0070138038603F00137E13FEC65A485A485A3807E060120F13C0 381F80E0383F00C0EA7F01EA7E03B5FCA213147F9317>I E /Fl 31 124 df<121C127FEAFF80A213C0A3127F121C1200A212011380A21203EA07001206120E5A5A12300A 157BA913>39 D<121C127FEAFF80A5EA7F00121C09097B8813>46 D<13075B137FEA07FFB5FCA2 12F8C6FCB3AB007F13FEA317277BA622>49 DII<1407 5C5C5C5C5CA25B5B497E130F130E131C1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A 5AB612F8A3C71300A7017F13F8A31D277EA622>I65 DI<91393FF00180903903FFFE07010FEBFF8F90393FF007FF9038FF8001 4848C7127FD807FC143F49141F4848140F485A003F15075B007F1503A3484891C7FCAB6C7EEE03 80A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007F0010FB512C0 010391C7FC9038003FF829297CA832>I69 D79 DI82 D<48B47E000F13F0381F81FC486C7E147FA2EC 3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0 393F83DFFC380FFF0F3801FC031E1B7E9A21>97 DIII< EB3FE03801FFF83803F07E380FE03F391FC01F80393F800FC0A2EA7F00EC07E05AA390B5FCA290 C8FCA47E7F003F14E01401D81FC013C0380FE0033903F81F803900FFFE00EB1FF01B1B7E9A20> I<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14> 105 D 107 DI<3BFFC07F800FF0903AC1FFE03FFC903AC7 83F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3 FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A3 13E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F 140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8 FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FC A3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013 FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038 F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD 1438A73803F870000113F03800FFE0EB1F8015267FA51B>I<3AFFFE03FF80A33A07F0007000A2 6D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A3 6D5AA26D5AA26D5A211B7F9A24>118 D<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7 FCEBFF1E6D5AEB3FF86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F 00036D7E3AFFF01FFF80A3211B7F9A24>120 D123 D E /Fm 67 124 df<90380FC3E090387FEFF09038E07C783801C0F8D8038013303907007000A7 B61280A23907007000B0387FE3FFA21D20809F1B>11 DI<90380F80F890387FE7FE9038E06E 063901C0FC0F380380F8380700F00270C7FCA6B7FCA23907007007B03A7FE3FE3FF0A22420809F 26>14 D<90380FC0FFEB7FE79038E07E0F3801C0FC4848487E38070070A7B7FCA23907007007B0 3A7FE3FE3FF0A22420809F26>I34 D<127012F812FCA2127C120CA31218A21238123012601240 060E7C9F0D>39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A212E0 AC1260A21270A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I<12C0 12607E7E121C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A21300 5A12065A121C12185A5A5A0B2E7DA112>I<127012F812FCA2127C120CA31218A2123812301260 1240060E7C840D>44 DI<127012F8A3127005057C840D>I48 DIII<13 0EA2131E133EA2136E13EE13CEEA018E1203130E1206120E120C121812381230126012E0B512F0 A238000E00A7EBFFE0A2141E7F9D17>II<137CEA01FEEA0783380E0380EA0C07121C3838030090C7FC127812 70A2EAF3F8EAF7FEEAFC0E487EEB0380A200F013C0A51270A214801238EB0700121CEA0E1EEA07 FCEA01F0121F7E9D17>I<1260387FFFC0A21480EA600138C003001306A2C65A5BA25B5BA213E0 5B1201A3485AA41207A76CC7FC121F7D9D17>III<127012F8A312701200AA127012F8A312700514 7C930D>I<127012F8A312701200AA127012F8A312781218A41230A21260A21240051D7C930D>I< EB0380A3497EA3EB0DE0A3EB18F0A3EB3078A3497EA3EBE01E13C0EBFFFE487FEB800FA2000314 80EB0007A24814C01403EA0F8039FFE03FFEA21F207F9F22>65 DI<90381FC04090387FF0C03801F8393803C00D38078007380F0003121E003E1301123C 127C1400127812F81500A8007814C0127CA2123C003EEB0180121E6CEB0300EA07803803C00E38 01F81C38007FF0EB1FC01A217D9F21>I69 DI73 D<39FFFC1FFCA239078007C0150014065C5C5C14705CEB81C0EB83800187C7FC80138FEB9BC0EB B1E013E1EBC0F01380147880A280A280EC0780A215C039FFFC3FFCA21E1F7E9E23>75 D77 D<39FF807FF813C00007EB07809038E00300A2EA06F0A21378133CA2131EA2130FA2EB078314C3 1303EB01E3A2EB00F3A2147BA2143F80A280A2000F7FEAFFF0801D1F7E9E22>III82 D<3807E080EA0FF9EA1C1FEA300FEA7007EA600312E01301A36CC7FCA21278127FEA 3FF0EA1FFC6C7EEA03FF38001F801307EB03C0A2130112C0A400E01380EAF00338F80700EAFE0E EACFFCEA81F812217D9F19>I<007FB512E0A238780F010070130000601460A200E0147000C014 30A400001400B23807FFFEA21C1F7E9E21>I<39FFFC7FF8A23907800780EC0300B3A300031302 EBC006A200015B6C6C5AEB7830EB3FE0EB0FC01D207E9E22>I<3BFFF07FF83FF0A23B0F000780 0F80EE0300A23A07800FC006A3913819E00ED803C0140CA214393A01E030F018A33A00F0607830 A3ECE07C903978C03C60A390393D801EC0A390383F000F6D5CA3010E6DC7FCA32C207F9E2F>87 D92 D97 D<120E12FEA2120EA9133FEBFF80380FC3C0EB00E000 0E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E0015207F9F19>IIII<13 3C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0A2120EAD38FFE7FCA216 207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFFC0A20A1F809E0C>I<13E0 EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EAF1C0EA7F80EA3E000C2882 9E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B137013F0EA0FF81338EA0E1C13 1E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA2120EB3A9EAFFE0A20B20809F 0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E13E0AD3AFFE7FE7FE0A223 147F9326>IIII<3803E180EA 0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA1E1FEA0FF3EA03E3EA0003A7 EB3FF8A2151D7E9318>I II<1206A4120EA2121E123EEAFFF8A2EA0E00AA1318A5EA07 3013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC1303A2EA070FEBFDFCEA01F11614 7F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA213FC6C 5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560ECC0E0D80E0F13C0130C14E00007 EBE180EB186114713903987300EBB033A2143F3801F03EEBE01EA20000131CEBC00C1E147F9321 >I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13781338133C137C13EEEA01C713 8738030380380701C0000F13E038FF87FEA21714809318>I<38FF87F8A2381E01E0000E13C014 80A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A31360A35B12F0EAF18012F3007FC7 FC123C151D7F9318>III E /Fn 14 124 df67 D73 D80 D<903807FFFE017FEBFFE048B612F84815FE489039001FFF8003077F48D980017F6F7FA2 707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC130F013FEBF03F3901FFFE004813F048 13C04890C7FC485A485A485AA212FF5BA3167FA26D14FF6C6CEB01EF003F140301FF90380FCFFF 6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEBF0010107903980007FF8362E7DAD3A> 97 D101 D108 D<903A7FC001FFC0B5010F13F8 033F13FE92B6FC9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC 5CA25CA25CB3A8B6D8C07FEBFFE0A53B2E7CAD42>110 DI<90397FC01FFEB590B512E002C314F802CF14FE9139DFF01FFF9126FF80077F00 0349486C7F6C496D7F02F06D7F717E4A81173F84171F84A3711380AB19005FA34D5AA260177F6E 5D6E4A5A6E495B6E495B9126FF800F5BDBE03F90C7FC02EFB512FC02E314F002E014C0DB1FFCC8 FC92CAFCAFB612C0A539427CAD42>I<9039FF803FC0B5EBFFF0028313FC02877F91388FE7FFEC 9F070003D99E0F13806C13BCA214F8A214F06F13004A6C5A6F5A92C8FCA25CB3A6B612E0A5292E 7CAD31>114 D<90390FFF81E090B512F7000314FF5A380FFC01391FE0003FD83F80130F007F14 0790C7FC15035AA27F6D90C7FC13F013FF14F86CEBFFC015F86C14FE6C806C15806C15C06C15E0 C615F0013F14F81307EB001F020013FC153F0078140F00F81407A26C1403A27E16F86C14076D14 F06D130F01F0EB1FE001FEEBFFC090B6128048ECFE00D8F83F13F8D8F00313C0262E7CAD2F>I< EB01F0A61303A31307A3130FA2131F133FA2137FEA01FF5A000F90B512C0B7FCA4C601F0C7FCB3 A5ED01F0A91503D97FF813E01507D93FFC13C090391FFE1F806DB5FC6D1400010113FC9038003F F024427EC12E>I<007FB5D8801FB5FCA528007FF8000390C7FC6EEB01FC6D6C495A6D6C495A6D 5D6D6D485A6D6D485AEDE03F6D6D48C8FC6DEBF8FE91387FF9FC6EB45A5E6E5B6E5B80806E7F82 824A7F825C91380FEFFFDA1FCF7FDA3F877FDA7F037FECFE0149486C7F4A6D7E49488001076E7E 49486D7E49487F49486D7FD9FFC081B500F8013FEBFFC0A53A2E7EAD3F>120 D123 D E /Fo 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007 B3B3A7007FB612E0A4233879B732>49 D 67 D97 D<903801FFC0011F13F8017F13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED 0FF0485A003FEC07F8A2485AED03FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C 147C000F15786C6C14F86C6CEB01F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E0 26267DA52D>101 D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C0102 7814804A7E4A14C05CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE 3FF891C7121F17FC160F17FEA3EE07FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E 481380DAFC07130091383FFFFC020F13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF8091387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F 80ED0E0091C8FCB3A3B512FEA423267DA529>114 D116 D E /Fp 36 119 df38 D<127012F8A212F012E005057A840F>46 D<14035CA25C1580141FA2143714771467 14C7A2EB0187A2EB0307A21306130E130C1318A2133090383FFFC05BEB600313C012011380EA03 00A21206A2121E39FFC03FFC13801E237DA224>65 D<90B512E015F890380F003C151E131E150E 150FA249130E151EA2153C49137815F0EC01E090387FFFC090B5FC9038F003E0EC00F01578485A 1538153CA248481378A315F039078001E01403EC07C0EC1F00B512FE14F020227DA122>I<90B5 12F015FC90380F003E150F011EEB0780150316C015015B16E0A35BA449EB03C0A44848EB0780A2 16005D4848130E5D153C5D48485B4A5AEC0780021FC7FCB512FC14F023227DA125>68 D<91387E0180903903FF810090380F80C390383E00670178133F49133ED801C0131E485A120748 C7121C120E121E5A15185A92C7FCA25AA4EC3FFC5AEC01E0A26C495AA312700078495A1238003C 130F6C131B260FC0F3C7FC3803FFC1C690C8FC212479A226>71 D73 D<903807FFC04913809038003C00 A25CA45CA4495AA4495AA4495AA449C7FCA212381278EAF81EA2485AEA6078EA70F0EA3FE0EA1F 801A237CA11A>I<9039FFF80FFCA290390F0007C01600011E130E5D5D1560495B4A5A4AC7FC14 0E495A5C147814FCEBF1BCEBF33CEBFE1E13FC3801F01F497EA2813803C007A26E7EA2EA07806E 7EA28139FFF80FFEA226227DA125>III<01FFEB1FFC1480010FEB03C01680D91BC01300A3EB19E001 311306A2EB30F0A201605B1478A3496C5AA3141ED801805BA2140FA2D803005BEC07E0A300065C 1403A2121F39FFE00180A226227DA124>I<14FE903807FF8090380F03E090383C00F001701378 4913384848133C4848131C48C7FC48141E121EA25AA25AA348143CA31578A34814F0A26CEB01E0 15C01403EC07800078EB0F00141E6C5B6C13F8380F83E03807FF80D801FCC7FC1F2479A225>I< 90B512C015F090380F0078153C011E131E150EA349131EA3153C491338157815F0EC03C090B512 005CEBF00FEC07803901E003C0A43903C00780A43907800F001503A2EC0706D8FFF8138EEC03FC C7EA01F020237DA124>82 D<903801F06090380FFC4090381E0EC0EB3807EB700301E0138013C0 1201A2D803801300A26DC7FCA27FEA01F813FF6C13E06D7EEB1FF8EB03FCEB007C143C80A30030 131CA31418007013385C00781360387C01C038EF0380D8C7FFC7FCEA81FC1B247DA21B>I<001F B512F8A2391E03C07800381438EB0780123000601430A2EB0F0012C0A3D8001E1300A45BA45BA4 5BA4485AA31203B57E91C7FC1D2277A123>I<393FFE07FFA23903C000F015E0484813C0A4390F 000180A4001EEB0300A4481306A4485BA4485BA25C12705C5C6C485A49C7FCEA1E0EEA0FFCEA03 F0202377A124>I<3BFFF03FF81FF8D9E07FEB3FF03B1F0007800780001E010FEB03001606141F 5E14375E14675E14C75E381F0187000F5DEB0307ED818013060383C7FC130C158613180138138C 0130139C0160139815B801C013B015E013805D13005D120E92C8FC120C2D2376A131>87 D<39FFF003FFA2000FC712F015E090388001C000071480EC0300EBC0060003130E5CEBE0180001 5B5CEBF0E000005BEBF18001FBC7FC13FF137E137C1378A45BA4485AA4EA3FFE5B202276A124> 89 D97 D<137E48B4FC3803C380EA0703 EA0E07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03 383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2 19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA 701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>II<136013F0 13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA 1F80EA0E000C217CA00F>105 D<13F0EA0FE0A21200A2485AA4485AA448C7FCEB01E0EB07F0EB 0E30380E1870EB30F01360EBC060381D8000121F13E0EA1CF0EA3838133CEB1C20143038703860 A21440EB18C038E01F8038600F0014237DA216>107 DI<381E0780383F1F E0EA63B8EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140C EB0318003813F0381801E018157C941B>110 D<137E48B4FC3803C380380701C0120E001C13E0 123CA21278A338F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803 C1F03807E3F8380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EB E3C038077F80EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863 B0E013E0EAC3C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114 D<13FC48B4FC38038380EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA70 07EAF00EA2485AEA7038EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2 EA0700120EA45AA45AA213101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F13 30381F8070EA31C0006113E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2 130FEB1F183807F3F03803E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC3 80A2000313C0EA0700A3380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I E /Fq 53 122 df35 D<127012F812FCA2127C12 0CA41218A21230A212601240060F7C840E>44 DI<127012F8A3127005 057C840E>I48 DI51 D<00101380EA1C07381FFF 005B5B13F00018C7FCA613F8EA1BFEEA1F0F381C0780EA180314C0EA000114E0A4126012F0A214 C0EAC0031260148038300700EA1C1EEA0FFCEA03F013227EA018>53 D<137E48B4FC3803C18038 0701C0EA0E03121CEB018048C7FCA2127812701320EAF1FCEAF3FEEAF60738FC038000F813C013 0112F014E0A51270A3003813C0130300181380381C0700EA0E0EEA07FCEA01F013227EA018>I< EA01F0EA07FCEA0E0F38180780EA3803383001C01270A31278EB0380123E383F0700EA1FCEEA0F FCEA03F87FEA0F7F381C3F80EA380F387007C0130338E001E01300A5387001C0A238380380381E 0F00EA0FFEEA03F013227EA018>56 DI<497E497EA3497EA3497E130CA2EB1CF8EB1878 A2EB383C1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F000 1F130139FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038F81C703801E0063903C003 F03807800148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C14 60123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>II 70 D<903807F00890383FFC189038FC0E383801E0033903C001F83807800048C71278121E1538 5AA2007C14181278A212F81500A6EC1FFF1278007CEB0078A2123CA27EA27E6C7E6C6C13F83801 F0013900FC079890383FFE08903807F80020247DA226>I<39FFFC3FFFA239078001E0AD90B5FC A2EB8001AF39FFFC3FFFA220227EA125>I<3803FFF0A238000F00B3A6127012F8A3EAF01EEA60 1CEA3878EA1FF0EA07C014237EA119>74 D<39FFFC07FFA239078001F015C05D4AC7FC14065C5C 14385C5CEB81C0EB8380EB87C080138DEB98F013B0EBE078497E13808080A26E7E8114036E7EA2 6E7E4A7E3AFFFC07FF80A221227EA126>III<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7FA2 EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000EA FFF0156020227EA125>III82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3F F0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF801 38EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200E0 141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<3BFFF03FFC07FEA23B0F0007C001 F00203EB00E01760D807806D13C0A33B03C007F001801406A216032701E00C781300A33A00F018 3C06A3903978383E0CEC301EA2161C90393C600F18A390391EC007B0A3010F14E0EC8003A36D48 6C5AA32F237FA132>87 D<387FFFFEA2EB003C007C137C0070137814F84813F0EB01E0EAC00314 C01307148038000F005B131E133E133C5B13F85B0001130313E0EA03C012071380000F13071300 001E1306003E130E003C131E007C133E007813FEB5FCA218227DA11E>90 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0 F0EB0078000E1338143C141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237F A21B>II<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA 38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>II<133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700 B1EA7FF8A2102380A20F>I<14F03801F1F83807FFB8380F1F38381E0F00EA1C07003C1380A500 1C1300EA1E0FEA0F1EEA1FFCEA19F00018C7FCA2121CEA1FFF6C13C04813E0383801F038700070 481338A400701370007813F0381E03C0380FFF803801FC0015217F9518>I<120E12FEA2121E12 0EAAEB1F80EB7FC0380FC1E0EB80F0EB0070120EAE38FFE7FFA218237FA21B>I<121C121E123E 121E121CC7FCA8120E12FEA2121E120EAFEAFFC0A20A227FA10E>II<120E12FEA212 1E120EAAEB0FFCA2EB07E0EB0380EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03 C0130114E014F038FFE3FEA217237FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E> I<390E1FC07F3AFE7FE1FF809039C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8F FEA227157F942A>I<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA21815 7F941B>II<380E1F8038FE7FE0 38FFC1F0380F0078120E143CA2141EA7143CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FC A8EAFFE0A2171F7F941B>I114 DI<1206A5120EA3121E123EEAFFF8A2 EA0E00AA130CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E 1370AC14F01301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2 391E01C07CD81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D8 1980141DA2EBF00F00001400A2497EEB600620157F9423>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A21330A213701360A2EAF0C012F1 EAF380007FC7FC123E171F7F941A>121 D E /Fr 20 118 df45 D68 D73 D77 D80 D<90387F80203901FFE0603807C0F8390F001CE0001E130F481307003813030078130112701400 12F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013C0EC 1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039E780 070038C1F01E38C07FFC38800FF01D337CB125>83 D97 D99 DIII<15F090387F03F83901FFCF1C3803C1FC390780F818 390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0 D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB 00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F 7E9F21>I<1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8 A20D307EAF12>105 D<260781FEEB3FC03BFF87FF80FFF0903A8E07C1C0F83B0F9803E3007C27 07B001E6133C9026E000FC7F495BA3495BB3486C486C133F3CFFFC1FFF83FFF0A2341F7E9E38> 109 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E 3AFFFC1FFF80A2211F7E9E25>II<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE0 38EBC000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813 705A1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2 141C7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A312 03A21207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB 19>II E /Fs 5 85 df<1630167016F0A21501A21503A2150715 0FA2151B821531A2156115E115C1EC0181A2EC0301A21406A2140C141C14181430A202607FA2EC C000A249B5FC5B91C7FC1306A25BA25BA25B1370136013E01201000381D80FF01301D8FFFE9038 3FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E8314 7E188002FE140FA24A15C0A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA201 1F157E17FE4A5CA2013F4A5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495A ED07E00001EC3F80B600FEC8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7 121EA24A140CA2147EA214FEA25CA20101151CEEC0184A1400A201031301A202F05B1503010713 0F91B5FC93C7FCECE00F010F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA213 7EA213FEA25BA21201B512FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092 380003F0EE01F84AEB00FCA2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804A EB3F00167E0107EB03F891B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA201 3F1301A21400A249495AA2137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0 C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C13 0F121800385CA20030011F1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FE A25CA21301A25CA21303A25CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D31 74B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 795 696 a Fs(D)26 b(R)g(A)f(F)h(T)225 787 y Fr(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)685 981 y Fq(Scott)c(Berryman,)e Fp(Y)l(ale)19 b(Univ)700 1040 y Fq(James)c(Co)o(wnie,)h Fp(Meiko)i(Ltd)474 1098 y Fq(Jac)o(k)e(Dongarra,)i Fp(Univ.)23 b(of)17 b(T)l(ennesse)n(e)j(and)d(ORNL)801 1156 y Fq(Al)f(Geist,)f Fp(ORNL)795 1214 y Fq(Bill)f(Gropp,)j Fp(ANL)767 1272 y Fq(Rolf)f(Hemp)q(el,)e Fp(GMD)762 1330 y Fq(Bob)i(Knigh)o(ten,)g Fp(Intel)786 1388 y Fq(Rust)o(y)g(Lusk,)h Fp(ANL)614 1446 y Fq(Stev)o(e)e(Otto,)h Fp(Or)n(e)n(gon)h(Gr)n(aduate)g(Inst)578 1504 y Fq(T)l(on)o(y)g(Skjellum,)c Fp(Missisippi)j(State)j(Univ)653 1563 y Fq(Marc)d(Snir,)g Fp(IBM)h(T.)g(J.)g(Watson)744 1621 y Fq(Da)o(vid)f(W)l(alk)o(er,)f Fp(ORNL)626 1679 y Fq(Stev)o(e)g(Zenith,)g Fp(Kuck)k(&)f(Asso)n(ciates)844 1803 y Fq(Ma)o(y)e(9,)g(1993)87 1861 y(This)g(w)o(ork)g(w)o(as)h(supp)q(orted)g(b)o(y)f(ARP)l(A)g(and)g(NSF)g (under)g(con)o(tract)g(n)o(um)o(b)q(er)f(###,)g(b)o(y)g(the)192 1919 y(National)h(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o(hnology) f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)650 1977 y(Agreemen)o(t)f(No.)21 b(CCR-8809615.)p eop %%Page: 1 2 bop 75 356 a Fo(Chapter)32 b(1)75 564 y Fn(Con)m(texts)40 b({)g(Prop)s(osal)h (I)876 786 y Fm(Marc)14 b(Snir)75 927 y Fl(1.1)70 b(Con)n(texts)75 1029 y Fm(A)17 b Fk(comm)o(unication)k(con)o(text)c Fm(\(for)f(short,)g Fk(con)o(text)p Fm(\))h(is)g(a)g(mec)o(hanism)g(for)g(the)g(mo)q (dularization)75 1085 y(of)h(MPI)g(comm)o(unication.)29 b(An)o(y)18 b(MPI)g(comm)o(unication)h(o)q(ccurs)f(within)h(a)f(con)o(text,)g(and)g(do)q (es)g(not)75 1142 y(in)o(terfere)j(with)g(comm)o(unication)h(executed)g (within)g(another)e(con)o(text.)37 b(F)l(urthermore,)21 b(a)g(con)o(text)75 1198 y(sp)q(eci\014es)c(a)e(lo)q(cal)h(name)f(space)h(for)e(pro)q(cesses)i (that)e(comm)o(unicate)h(in)h(this)g(con)o(text.)j(The)d(pro)q(cesses)75 1255 y(that)i(participate)i(in)g(a)f(con)o(text)f(are)h(asso)q(ciated)g(with) g(a)g Fk(rank)p Fm(,)g(whic)o(h)h(ranges)e(from)h(0)f(to)h Fj(n)13 b Fi(\000)g Fm(1,)75 1311 y(where)18 b Fj(n)h Fm(is)f(the)h(n)o(um)o (b)q(er)f(of)g(pro)q(cesses)g(that)g(participate)h(in)g(the)f(con)o(text.)28 b(This)19 b(rank)f(is)g(used)h(in)75 1368 y(in)o(terpro)q(cess)d(comm)o (unication)h(as)e(the)h(lo)q(cal)h(address)f(of)f(the)h(pro)q(cess)g(within)h (that)e(con)o(text.)21 b(Th)o(us,)75 1424 y(comm)o(unication)16 b(within)g(a)f(con)o(text)g(is)h(una\013ected)f(b)o(y)g(comm)o(unication)h (outside)g(that)e(con)o(text.)166 1480 y(A)22 b(pro)q(cess)g(ma)o(y)g(comm)o (unicate)g(sim)o(ultaneously)h(in)g(sev)o(eral)f(con)o(texts.)40 b(The)22 b(con)o(text)g(of)f(a)75 1537 y(comm)o(unication)16 b(is)g(explicitly)i(stated)c(as)h(a)g(parameter)f(of)h(the)g(comm)o (unication)h(call.)166 1593 y(A)h(pro)q(cess)g(that)f(participates)i(in)g(a)e (comm)o(unication)i(con)o(text)f(accesses)g(this)g(con)o(text)g(using)g(a)75 1650 y Fh(c)n(ontext)h(hand)r(le)g Fm(\(i.e.,)f(a)h(handle)h(to)e(an)g (opaque)h(ob)s(ject)f(that)g(iden)o(ti\014es)j(a)d(con)o(text\).)26 b(This)19 b(handle)75 1706 y(can)c(b)q(e)h(used)g(to)143 1788 y Fi(\017)23 b Fm(Find)14 b(information)f(ab)q(out)g(this)g(con)o(text,)g (suc)o(h)g(as)g(the)g(n)o(um)o(b)q(er)h(of)e(pro)q(cesses)i(that)e (participate)189 1844 y(in)k(the)f(con)o(text,)f(or)h(the)g(rank)g(of)g(the)g (calling)i(pro)q(cess)f(within)g(the)f(con)o(text.)143 1933 y Fi(\017)23 b Fm(Comm)o(unicate)12 b(with)h(other)f(pro)q(cesses)h(that)e (participate)j(in)f(the)f(con)o(text;)h(these)f(pro)q(cesses)h(are)189 1989 y(addressed)i(using)h(their)g(con)o(text)f(rank.)143 2078 y Fi(\017)23 b Fm(Create)14 b(new)i(con)o(texts.)166 2160 y(Con)o(text)i (handles)h(cannot)g(b)q(e)g(transferred)f(for)g(one)h(pro)q(cess)f(to)g (another;)i(they)e(can)h(b)q(e)g(used)75 2216 y(only)12 b(on)g(the)f(pro)q (cess)h(where)g(they)g(w)o(ere)f(created.)19 b(Th)o(us,)11 b(\\kno)o(wledge")h(ab)q(out)f(a)h(con)o(text)f(exists)g(only)75 2273 y(lo)q(cally)l(,)17 b(at)e(the)h(pro)q(cesses)g(that)f(participate)h(in) h(that)d(con)o(text.)21 b(Op)q(erations)16 b(within)h(a)e(comm)o(unica-)75 2329 y(tion)h(con)o(texts)e(\(including)k(the)d(generation)h(of)f(new)g(sub)q (con)o(texts\))g(do)g(not)g(require)h(comm)o(unication)75 2385 y(with)g(pro)q(cesses)f(that)g(do)g(not)g(participate)g(in)i(that)d(con)o (text.)166 2442 y(F)l(ollo)o(ws)h(examples)h(of)f(p)q(ossible)i(uses)e(for)g (con)o(texts.)75 2561 y Fg(1.1.1)55 b(Lo)r(osely)18 b(sync)n(hronous)h (library)h(call)g(in)n(terface)75 2647 y Fm(Consider)13 b(the)g(case)g(where) g(a)f(parallel)i(application)h(executes)e(a)f(\\parallel)i(call")g(to)e(a)g (library)i(routine,)75 2704 y(i.e.,)g(where)h(all)g(pro)q(cesses)g(transfer)f (con)o(trol)g(to)g(the)h(library)g(routine.)20 b(If)15 b(the)f(library)i(w)o (as)d(dev)o(elop)q(ed)964 2828 y(1)p eop %%Page: 2 3 bop 75 -100 a Fm(2)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)75 45 y Fm(separately)l(,)23 b(then)e(one)h(should)g(b) q(ew)o(are)g(of)e(the)i(p)q(ossibilit)o(y)h(that)e(the)g(library)h(co)q(de)g (ma)o(y)f(receiv)o(e)75 102 y(b)o(y)f(mistak)o(e)g(messages)f(send)i(b)o(y)f (the)g(caller)h(co)q(de,)g(and)f(vice-v)o(ersa.)35 b(The)20 b(problem)h(is)g(solv)o(ed)f(b)o(y)75 158 y(allo)q(cating)c(a)f(di\013eren)o (t)h(con)o(text)e(to)h(the)g(library)l(,)h(th)o(us)f(prev)o(en)o(ting)h(un)o (w)o(an)o(ted)e(in)o(terference.)75 277 y Fg(1.1.2)55 b(F)-5 b(unctional)21 b(decomp)r(osition)e(and)g(mo)r(dular)g(co)r(de)f(dev)n (elopmen)n(t)75 363 y Fm(Often,)i(a)f(parallel)i(application)g(is)e(dev)o (elop)q(ed)i(b)o(y)e(in)o(tegrating)h(sev)o(eral)f(distinct)h(functional)h (mo)q(d-)75 419 y(ules,)c(that)f(is)h(eac)o(h)g(dev)o(elop)q(ed)h(separately) l(.)24 b(Eac)o(h)16 b(mo)q(dule)i(is)f(a)f(parallel)i(program)d(that)h(runs)h (on)f(a)75 476 y(dedicated)g(set)e(of)g(pro)q(cesses,)g(and)g(the)h (computation)f(consists)h(of)e(phases)i(where)f(mo)q(dules)i(compute)75 532 y(separately)l(,)d(in)o(termixed)h(with)f(global)h(phases)f(where)g(all)h (pro)q(cesses)f(comm)o(unicate.)19 b(It)13 b(is)g(con)o(v)o(enien)o(t)75 589 y(to)g(allo)o(w)h(eac)o(h)g(mo)q(dule)g(to)f(use)h(its)g(o)o(wn)f(priv)m (ate)i(pro)q(cess)f(n)o(um)o(b)q(ering)g(sc)o(heme,)g(for)f(the)h(in)o(tramo) q(dule)75 645 y(computation.)24 b(This)17 b(is)g(ac)o(hiev)o(ed)g(b)o(y)g (using)g(a)f(priv)m(ate)i(mo)q(dule)f(con)o(text)f(for)g(in)o(tramo)q(dule)i (compu-)75 701 y(tation,)c(and)i(a)f(global)h(con)o(text)e(for)h(in)o(termo)q (dule)h(comm)o(unication.)75 820 y Fg(1.1.3)55 b(Collectiv)n(e)20 b(comm)n(unication)75 906 y Fm(MPI)g(supp)q(orts)h(collectiv)o(e)h(comm)o (unication)f(within)h(dynamically)g(created)f(groups)f(of)g(pro)q(cesses.)75 963 y(Eac)o(h)15 b(suc)o(h)h(group)f(can)g(b)q(e)h(represen)o(ted)g(b)o(y)f (a)g(distinct)h(comm)o(unication)g(con)o(text.)k(This)c(pro)o(vides)f(a)75 1019 y(simple)g(mec)o(hanism)f(to)f(ensure)h(that)f(comm)o(unication)h(that)f (p)q(ertains)h(to)f(collectiv)o(e)i(comm)o(unication)75 1076 y(within)h(one)f(group)f(is)h(not)f(confused)i(with)f(collectiv)o(e)h(comm)o (unication)g(within)f(another)g(group,)f(and)75 1132 y(a)o(v)o(oids)h(the)g (in)o(tro)q(duction)h(of)f(t)o(w)o(o)f(di\013eren)o(t)h(mec)o(hanisms)h(with) g(similar)g(functionalit)o(y)l(.)75 1251 y Fg(1.1.4)55 b(Ligh)n(t)n(w)n(eigh) n(t)21 b(gang)f(sc)n(heduling)75 1337 y Fm(Consider)14 b(an)f(en)o(vironmen)o (t)g(where)g(pro)q(cesses)h(are)e(m)o(ultith)o(treaded.)20 b(Con)o(texts)12 b(can)h(b)q(e)h(used)g(to)e(pro-)75 1393 y(vide)h(a)e(mec)o (hanism)i(whereb)o(y)f(all)h(pro)q(cesses)f(are)f(time-shared)i(b)q(et)o(w)o (een)f(sev)o(eral)g(parallel)h(executions,)75 1450 y(and)19 b(can)h(con)o(text)e(switc)o(h)i(from)e(one)h(parallel)i(execution)f(to)f (another,)g(in)h(a)f(lo)q(osely)h(sync)o(hronous)75 1506 y(manner.)27 b(A)17 b(thread)g(is)h(allo)q(cated)h(on)e(eac)o(h)g(pro)q(cess)h(to)f(eac)o (h)g(parallel)i(execution,)g(and)f(a)f(di\013eren)o(t)75 1562 y(con)o(text)d(is)i(used)f(to)f(iden)o(tify)i(eac)o(h)f(parallel)i (execution.)k(Th)o(us,)14 b(tra\016c)g(from)g(one)h(execution)h(cannot)75 1619 y(b)q(e)i(confused)g(with)g(tra\016c)f(from)f(another)h(execution.)28 b(The)18 b(blo)q(c)o(king)g(and)g(un)o(blo)q(c)o(king)h(of)e(threads)75 1675 y(due)g(to)f(comm)o(unication)h(ev)o(en)o(ts)f(pro)o(vide)h(a)f(\\lazy") g(con)o(text)g(switc)o(hing)h(mec)o(hanism.)24 b(This)17 b(can)f(b)q(e)75 1732 y(extended)j(to)f(the)h(case)f(where)h(the)f(parallel)i(executions)f (are)g(spanning)g(distinct)h(pro)q(cess)e(subsets.)75 1788 y(\(MPI)d(do)q(es)g(not)g(require)h(m)o(ultithreaded)h(pro)q(cesses.\))75 1929 y Fl(1.2)70 b(Basic)22 b(Con)n(text)g(Op)r(erations)75 2030 y Fm(A)17 b(global)g(con)o(text)g Fk(MPI)p 534 2030 16 2 v 18 w(ALL)g Fm(is)g(prede\014ned.)26 b(All)19 b(pro)q(cesses)e (participate)g(in)h(this)f(con)o(text)f(when)75 2087 y(computation)k(starts.) 32 b(MPI)20 b(do)q(es)g(not)f(sp)q(ecify)i(ho)o(w)e(pro)q(cesses)h(are)g (initially)i(rank)o(ed)e(within)h(the)75 2143 y(con)o(text)11 b Fe(MPI)p 308 2143 15 2 v 17 w(ALL)p Fm(.)g(It)h(is)g(exp)q(ected)h(that)f (the)g(start-up)f(pro)q(cedure)i(used)f(to)f(initiate)i(an)f(MPI)g(program)75 2200 y(\(at)j(load-time)j(or)d(run-time\))i(will)h(pro)o(vide)f(information)g (or)e(con)o(trol)h(on)h(this)f(initial)j(ranking)e(\(e.g.,)75 2256 y(b)o(y)12 b(sp)q(ecifying)j(that)c(pro)q(cesses)i(are)f(rank)o(ed)h (according)g(to)e(their)i(pid's,)h(or)d(according)i(to)f(the)h(ph)o(ysical)75 2312 y(addresses)h(of)f(the)h(executing)g(pro)q(cessors,)f(or)g(according)h (to)f(a)h(n)o(um)o(b)q(ering)g(sc)o(heme)g(sp)q(eci\014ed)i(at)d(load)75 2369 y(time\).)166 2508 y Fd(Discussion:)h Fc(If)e(w)o(e)h(think)e(of)h (adding)f(new)i(pro)q(cesses)i(at)d(run-time,)e(then)j Fb(MPI)p 1454 2508 14 2 v 15 w(ALL)f Fc(con)o(v)o(eys)g(the)h(wrong)75 2564 y(impression,)f(since)j(it)f(is)f(just)h(the)h(initial)d(set)j(of)e(pro) q(cesses.)166 2704 y Fm(The)i(follo)o(wing)h(op)q(erations)g(are)e(a)o(v)m (ailable)j(for)e(creating)g(new)h(con)o(texts.)p eop %%Page: 3 4 bop 75 -100 a Ff(1.2.)34 b(BASIC)16 b(CONTEXT)f(OPERA)l(TIONS)964 b Fm(3)166 45 y Fk(MPI)p 275 45 16 2 v 18 w(COPY)p 446 45 V 18 w(CONTEXT\(new)o(con)o(text,)17 b(con)o(text\))166 137 y Fm(Create)g(a)g(new)g(con)o(text)g(that)g(includes)j(all)e(pro)q(cesses)g(in) g(the)f(old)h(con)o(text.)26 b(The)18 b(rank)f(of)g(the)75 193 y(pro)q(cesses)g(in)g(the)f(previous)h(con)o(text)f(is)g(preserv)o(ed.)24 b(The)16 b(call)i(m)o(ust)d(b)q(e)i(executed)g(b)o(y)f(all)i(pro)q(cesses)75 250 y(in)f(the)g(old)g(con)o(text.)22 b(It)17 b(is)g(a)f(blo)q(c)o(king)h (call:)24 b(No)16 b(call)h(returns)f(un)o(til)i(all)f(pro)q(cesses)g(ha)o(v)o (e)f(called)i(the)75 306 y(function.)j(The)15 b(parameters)f(are)75 406 y Fk(OUT)k(new)o(con)o(text)k Fm(handle)16 b(to)d(newly)h(created)g(con)o (text.)19 b(The)14 b(handle)h(should)g(not)e(b)q(e)i(asso)q(ciated)189 462 y(with)g(an)g(ob)s(ject)g(b)q(efore)g(the)h(call.)75 554 y Fk(IN)h(con)o(text)23 b Fm(handle)17 b(to)d(old)i(con)o(text)166 689 y Fk(MPI)p 275 689 V 18 w(NEW)p 422 689 V 19 w(CONTEXT\(new)o(con)o (text,)h(con)o(text,)g(arra)o(y)p 1338 689 V 18 w(of)p 1398 689 V 19 w(ranks,)f(size\))75 824 y(OUT)i(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h(created)g(con)o(text)f(at)h(calling)h(pro)q (cess.)19 b(This)14 b(handle)g(should)189 880 y(not)g(b)q(e)i(asso)q(ciated)g (with)f(an)g(ob)s(ject)g(b)q(efore)h(the)f(call.)75 972 y Fk(IN)i(con)o(text) 23 b Fm(handle)17 b(to)d(old)i(con)o(text)75 1064 y Fk(IN)h(arra)o(y)p 277 1064 V 18 w(of)p 337 1064 V 19 w(ranks)22 b Fm(ranks)15 b(in)h(the)f(old)h(con)o(text)e(of)h(the)g(pro)q(cesses)h(that)e(join)i(the)f (new)h(con)o(text)75 1156 y Fk(IN)h(size)23 b Fm(size)16 b(of)f(new)h(con)o (text)e(\(in)o(teger\))166 1255 y(A)20 b(new)g(con)o(text)f(is)i(created)e (for)h(the)g(pro)q(cesses)g(in)h(the)f(old)g(con)o(text)f(that)g(are)h (listed)h(in)g(the)75 1312 y(arra)o(y)l(.)f(The)c(pro)q(cesses)f(are)h (listed)g(according)g(to)f(their)h(rank)g(in)g(the)f(old)i(con)o(text.)j(The) c(rank)f(of)g(the)75 1368 y(pro)q(cesses)h(in)g(the)f(new)g(con)o(text)g(is)h (determined)g(b)o(y)f(their)h(place)g(in)g(the)f(list.)166 1424 y(The)h(call)g(has)g(to)f(b)q(e)h(executed)h(b)o(y)e(all)i(pro)q(cesses) f(listed)g(in)h(the)f(arra)o(y;)e(all)i(mak)o(e)f(the)h(call)h(with)75 1481 y(the)h(same)g(list)i(of)d(parameters.)29 b(Pro)q(cesses)18 b(in)h(the)g(old)g(con)o(text)e(that)h(do)g(not)g(b)q(elong)i(to)d(the)i(new) 75 1537 y(con)o(text)14 b(need)i(not)f(mak)o(e)g(the)g(call.)21 b(The)15 b(call)h(is)g(blo)q(c)o(king;)g(no)f(pro)q(cess)g(returns)g(from)f (the)i(call)g(un)o(til)75 1594 y(all)g(pro)q(cesses)g(ha)o(v)o(e)f(executed)h (the)f(call.)166 1686 y Fk(MPI)p 275 1686 V 18 w(SPLIT)p 445 1686 V 19 w(CONTEXT\(new)o(con)o(text,)i(con)o(text,)g(k)o(ey)l(,)f(index\)) 75 1821 y(OUT)i(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h(created)g(con) o(text)f(at)h(calling)h(pro)q(cess.)19 b(This)14 b(handle)g(should)189 1877 y(not)g(b)q(e)i(asso)q(ciated)g(with)f(an)g(ob)s(ject)g(b)q(efore)h(the) f(call.)75 1969 y Fk(IN)i(con)o(text)23 b Fm(handle)17 b(to)d(old)i(con)o (text)75 2061 y Fk(IN)h(k)o(ey)22 b Fm(in)o(teger)75 2153 y Fk(IN)17 b(index)23 b Fm(in)o(teger)166 2252 y(A)18 b(new)g(con)o(text)f(is)h (created)g(for)g(eac)o(h)g(distinct)h(v)m(alue)g(of)e Fe(key)p Fm(;)h(this)h(con)o(text)e(is)h(shared)g(b)o(y)g(all)75 2308 y(pro)q(cesses)13 b(that)f(made)h(the)g(call)h(with)f(this)g(k)o(ey)f(v)m (alue.)21 b(Within)13 b(eac)o(h)g(new)g(con)o(text)f(the)h(pro)q(cesses)g (are)75 2365 y(rank)o(ed)j(according)g(to)g(the)g(order)g(of)f(the)h Fe(index)g Fm(v)m(alues)h(they)f(pro)o(vided;)h(in)g(case)f(of)f(ties,)i(pro) q(cesses)75 2421 y(are)e(rank)o(ed)g(according)h(to)e(their)i(rank)f(in)h (the)f(old)h(con)o(text.)166 2478 y(This)e(call)h(is)f(blo)q(c)o(king:)21 b(No)13 b(call)i(returns)f(un)o(til)h(all)f(pro)q(cesses)g(in)h(the)f(old)g (con)o(text)f(executed)i(the)75 2534 y(call.)166 2591 y(P)o(articular)g(uses) h(of)e(this)i(function)g(are:)166 2647 y(\(i\))g(Reordering)h(pro)q(cesses:) 22 b(All)c(pro)q(cesses)e(pro)o(vide)h(the)f(same)g Fe(key)g Fm(v)m(alue,)h(and)g(pro)o(vide)f(their)75 2704 y(index)g(in)h(the)e(new)g (order.)p eop %%Page: 4 5 bop 75 -100 a Fm(4)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)166 45 y Fm(\(ii\))e(Splitting)h(a)e(con)o(text)f(in)o (to)i(sub)q(con)o(texts,)f(while)i(preserving)f(the)f(old)h(relativ)o(e)g (order)f(among)75 102 y(pro)q(cesses:)21 b(All)c(pro)q(cesses)f(pro)o(vide)g (the)g(same)f Fe(index)g Fm(v)m(alue,)i(and)f(pro)o(vide)g(a)f(k)o(ey)h(iden) o(tifying)h(their)75 158 y(new)e(sub)q(con)o(text.)166 214 y Fe(MPI)p 241 214 15 2 v 17 w(COPY)p 354 214 V 16 w(CONTEXT)g Fm(is)h(a)f(particular)i(case)e(of)h Fe(MPI)p 1069 214 V 16 w(SPLIT)p 1205 214 V 17 w(CONTEXT)p Fm(,)e(when)i(all)h(pro)q(cesses)f(pro-) 75 271 y(vide)g(the)g(same)e(k)o(ey)h(and)h(index)g(parameter.)166 363 y Fk(MPI)p 275 363 16 2 v 18 w(RANK\(rank,)g(con)o(text\))75 504 y(OUT)i(rank)23 b Fm(in)o(teger)75 598 y Fk(IN)17 b(con)o(text)23 b Fm(con)o(text)15 b(handle)166 705 y(Return)h(the)f(rank)g(of)g(the)g (calling)i(pro)q(cess)e(within)i(the)e(sp)q(eci\014ed)i(con)o(text.)166 796 y Fk(MPI)p 275 796 V 18 w(SIZE\(size,)h(con)o(text\))75 938 y(OUT)g(size)23 b Fm(in)o(teger)75 1032 y Fk(IN)17 b(con)o(text)23 b Fm(con)o(text)15 b(handle)166 1138 y(Return)h(the)f(n)o(um)o(b)q(er)h(of)e (pro)q(cesses)i(that)e(b)q(elong)j(to)d(the)h(sp)q(eci\014ed)j(con)o(text.) 166 1195 y(A)d(con)o(text)g(ob)s(ject)f(is)i(destro)o(y)o(ed)f(using)h(the)f Fe(MPI)p 1036 1195 15 2 v 17 w(FREE)f Fm(function.)75 1316 y Fg(1.2.1)55 b(Usage)19 b(note)75 1402 y Fm(Use)e(of)g(con)o(texts)f(for)h (libraries:)25 b(Eac)o(h)17 b(library)h(ma)o(y)e(pro)o(vide)i(an)f (initialization)j(routine)e(that)e(is)i(to)75 1459 y(b)q(e)c(called)i(b)o(y)e (all)g(pro)q(cesses,)g(and)g(that)f(generate)h(a)g(con)o(text)f(for)g(the)h (use)g(of)f(that)g(library)l(.)21 b(A)14 b(sc)o(heme)75 1515 y(for)j(allo)o(wing)i(eac)o(h)g(link)o(ed)g(library)g(to)f(ha)o(v)o(e)g(its)g (o)o(wn)f(initializati)q(on)k(co)q(de)d(w)o(ould)h(can)f(b)q(e)h(used)g(for) 75 1572 y(this)d(purp)q(ose)f(\(assuming)h(the)f(library)h(will)h(not)e(ha)o (v)o(e)f(sev)o(eral)i(concurren)o(t)f(instan)o(tiations\).)166 1628 y(Use)g(of)g(con)o(texts)g(for)f(functional)j(decomp)q(osition:)k(A)15 b(harness)g(program,)f(running)i(in)g(the)g(con-)75 1684 y(text)e Fe(ALL)g Fm(generates)f(a)h(sub)q(con)o(text)h(for)e(eac)o(h)i(mo)q(dule)g (and)g(then)f(starts)f(the)i(submo)q(dule)g(within)h(the)75 1741 y(corresp)q(onding)g(con)o(text.)166 1797 y(Use)i(of)f(con)o(texts)g (for)g(collectiv)o(e)j(comm)o(unication:)26 b(A)18 b(con)o(text)f(is)h (created)g(for)f(eac)o(h)h(group)f(of)75 1854 y(pro)q(cesses)f(where)f (collectiv)o(e)i(comm)o(unication)f(is)g(to)e(o)q(ccur.)166 1910 y(Use)k(of)g(con)o(texts)f(for)h(con)o(text-switc)o(hing)g(among)g(sev)o (eral)g(parallel)i(executions:)26 b(A)18 b(pream)o(ble)75 1967 y(co)q(de)d(is)g(used)g(to)f(generate)g(a)g(di\013eren)o(t)g(con)o(text)g (for)g(eac)o(h)g(execution;)i(this)e(pream)o(ble)h(co)q(de)g(needs)h(to)75 2023 y(use)g(a)e(m)o(utual)i(exclusion)h(proto)q(col)e(to)f(mak)o(e)h(sure)g (eac)o(h)h(thread)f(claims)h(the)f(righ)o(t)g(con)o(text.)166 2156 y Fd(Implemen)o(tati)o(on)d(note:)166 2205 y Fc(W)m(e)18 b(outline)f(here)i(t)o(w)o(o)f(p)q(ossible)g(implemen)o(tations)d(of)j(con)o (texts.)31 b(They)19 b(are)f(b)o(y)g(no)g(means)f(the)i(only)75 2255 y(p)q(ossible)c(ones.)23 b(In)15 b(eac)o(h)g(implemen)o(tation)d(w)o(e)j (assume)g(that)g(a)g(con)o(text)h(ob)r(ject)g(is)f(a)g(p)q(oin)o(ter)g(to)g (a)g(structure)75 2305 y(that)d(describ)q(es)i(the)f(con)o(text.)18 b(A)12 b(comp)q(onen)o(t)f(of)g(this)h(structure)i(is)e(a)g(table)f(of)h(the) g(pro)q(cesses)j(that)d(participate)75 2355 y(in)17 b(the)i(con)o(text,)f (ordered)h(b)o(y)f(rank.)29 b(W)m(e)17 b(assume)h(that)f(the)i(n)o(um)o(b)q (er)e(of)g(concurren)o(tly)i(activ)o(e)e(con)o(texts)i(at)75 2405 y(eac)o(h)12 b(pro)q(cess)i(is)d(relativ)o(ely)g(small;)e(sa)o(y)j (16-32.)k(In)c(either)g(implemen)o(tatio)o(n)d(one)j(migh)o(t)d(ha)o(v)o(e)j (disjoin)o(t)e(message)75 2455 y(queues)j(for)e(eac)o(h)i(con)o(text,)f(or)g (ha)o(v)o(e)f(shared)i(queues,)g(with)e(the)h(righ)o(t)g(mec)o(hanisms)d(for) j(con)o(text)g(matc)o(hing)e(and)75 2504 y(bu\013er)15 b(allo)q(cation.)166 2554 y(Prop)q(osal)f(1:)j(Large)d(con)o(text)h(tags.)166 2604 y(In)d(this)g(implem)o(en)o(tation)d(w)o(e)j(use)h(large)e(con)o(text)i(tags) f(\(sa)o(y)f(32)h(bits\),)g(so)g(that)f(matc)o(hing)f(of)i(an)f(incoming)75 2654 y(tag)19 b(with)g(a)h(lo)q(cal)e(pro)q(cess)k(requires)f(to)e(p)q (erform)g(a)g(searc)o(h)i(in)e(a)g(hash)h(table)f(or)h(another)g(similar)d (searc)o(h)75 2704 y(structure)e(\(this)d(o)q(ccurs)i(whenev)o(er)g(a)e (message)g(is)h(receiv)o(ed\).)19 b(All)11 b(messages)i(sen)o(t)g(within)f(a) g(con)o(text)h(carry)g(the)p eop %%Page: 5 6 bop 75 -100 a Ff(1.3.)34 b(AD)o(V)-5 b(ANCED)14 b(CONTEXT)i(OPERA)l(TIONS)841 b Fm(5)75 45 y Fc(same)13 b(tag)h(v)n(alue.)k(W)m(e)c(use)h(as)g(con)o(text)f (tag)g(the)h(pid)f(of)f(the)i(lo)o(w)o(est)f(n)o(um)o(b)q(ered)g(pro)q(cess)i (in)e(the)h(con)o(text)g(\(let's)75 95 y(call)g(it)g(the)h(con)o(text)g (leader\),)h(concatenated)g(with)e(a)g(coun)o(ter)i(that)e(is)h(incremen)o (ted)g(whenev)o(er)h(this)e(pro)q(cess)75 145 y(allo)q(cates)h(a)g(new)g (tag.)24 b(This)16 b(guaran)o(tee)g(a)g(unique)g(tag)g(for)f(eac)o(h)i(group) f(\(sp)q(ecial)g(co)q(de)h(needed)g(for)f(coun)o(ter)75 195 y(wraparound\).)166 247 y Fb(MPI)p 235 247 14 2 v 15 w(COPY)p 338 247 V 15 w(CONTEXT)11 b Fc(-)h(A)h(new)g(tag)f(is)g(generated)i(b)o(y)f (incremen)o(ting)e(the)i(old)f(tag)h(b)o(y)f(one;)h(one)f(can)h(either)75 296 y(cop)o(y)f(the)g(old)f(con)o(text)h(table,)g(or)f(create)j(a)d(new)h(p)q (oin)o(ter)g(to)g(the)g(old)f(table.)17 b(A)12 b(global)e(barrier)i(sync)o (hronization)75 346 y(is)i(needed)h(to)f(mak)o(e)e(sure)j(the)g(call)e(is)h (blo)q(c)o(king.)166 398 y Fb(MPI)p 235 398 V 15 w(SPLIT)p 360 398 V 15 w(CONTEXT)j Fc(-)h(A)h(naiv)o(e)f(implemen)o(tatio)o(n)e(is)j (to)f(ha)o(v)o(e)g(an)h(all-to-all)d(comm)o(unicati)o(on)g(where)75 448 y(eac)o(h)h(pro)q(cess)i(gathers)f(the)f Fb(\(key,)k(index\))15 b Fc(pairs)i(of)f(all)g(pro)q(cesses)j(in)e(the)g(old)f(group.)27 b(Eac)o(h)17 b(pro)q(cess)i(can)75 498 y(determine)14 b(whether)h(it)e(is)h (the)g(leader)h(of)e(a)g(new)i(group)e(and)h(broadcast)g(the)h(new)f(group)g (tag)f(to)h(all)e(mem)o(b)q(ers)75 548 y(\(using)j(p)q(oin)o(t)g(to)g(p)q (oin)o(t)g(comm)o(unicatio)o(n)e(in)i(the)h(old)e(group,)h(or)g(using)h (another)f(all-to-all)e(comm)o(unicatio)o(n\).)75 597 y(Algorithmic)f(minds)g (will)g(think)i(of)f(man)o(y)f(p)q(ossible)i(optimizations.)166 649 y Fb(MPI)p 235 649 V 15 w(NEW)p 316 649 V 15 w(CONTEXT)c Fc(-)g(The)i(lo)o(w)o(est)f(n)o(um)o(b)q(ered)f(pro)q(cess)j(in)e(the)g(list) g(broadcast)g(the)h(new)f(con)o(text)h(tag)e(to)h(all)75 699 y(pro)q(cesses)i(in)d(the)g(list)g(\(using)g(p)q(oin)o(t)g(to)g(p)q(oin)o(t)f (comm)o(unication)e(op)q(erations\).)17 b(A)11 b(more)e(robust)h(implemen)o (tation)75 749 y(ma)o(y)i(en)o(tail)h(the)i(broadcast)f(of)f(the)i(mem)o(b)q (er)d(list,)h(for)h(error)g(c)o(hec)o(king.)166 801 y Fb(MPI)p 235 801 V 15 w(RANK,)21 b(MPI)p 447 801 V 15 w(SIZE)13 b Fc(-)g(require)i(lo) q(cal)e(access)j(to)e(the)g(con)o(text)h(ob)r(ject)166 853 y(Prop)q(osal)f(2:)j(Small)12 b(con)o(text)i(tags.)166 905 y(In)19 b(that)h(implemen)o(tatio)o(n)d(the)j(n)o(um)o(b)q(er)f(of)g (distinct)h(con)o(text)g(tag)f(v)n(alues)g(is)h(equal)f(to)g(the)h(maxima)o (l)75 955 y(n)o(um)o(b)q(er)d(of)g(con)o(texts)i(that)f(can)g(b)q(e)h(activ)o (e)f(at)f(the)i(same)e(no)q(de.)30 b(Th)o(us,)19 b(the)f(con)o(text)h(tag)e (of)h(an)f(incoming)75 1005 y(message)i(can)g(b)q(e)g(used)h(to)f(index)f (directly)i(in)o(to)e(a)g(con)o(text)i(table,)g(a)o(v)o(oiding)c(the)k(need)g (for)e(a)h(searc)o(h)h(in)e(a)75 1055 y(hash)13 b(table.)18 b(Eac)o(h)13 b(pro)q(cess)i(has)e(a)f(unique)h(con)o(text)h(tag)e(for)h (incoming)d(comm)o(unication)g(within)i(this)h(con)o(text.)75 1104 y(Ho)o(w)o(ev)o(er,)h(di\013eren)o(t)h(con)o(text)g(tag)e(v)n(alues)h (ma)o(y)e(b)q(e)j(used)g(b)o(y)f(di\013eren)o(t)h(pro)q(cesses)h(for)e(the)h (same)e(con)o(text)i(\(this)75 1154 y(is)e(necessary)i(in)e(order)h(to)f (densely)h(p)q(opulate)f(the)h(con)o(text)f(tag)g(range\).)18 b(The)c(con)o(text)g(table)f(carries,)h(for)e(eac)o(h)75 1204 y(mem)o(b)q(er)g(of)i(the)g(con)o(text,)g(the)h(con)o(text)f(tag)g(to)f(b)q (e)i(used)g(when)f(sending)g(messages)g(to)g(it.)166 1256 y Fb(MPI)p 235 1256 V 15 w(COPY)p 338 1256 V 15 w(CONTEXT)c Fc(-)h(A)h(new)g (tag)f(is)h(generated)h(b)o(y)e(eac)o(h)h(pro)q(cess)i(for)d(the)i(new)f(con) o(text,)g(and)f(broadcast)75 1306 y(to)18 b(all)g(other)h(mem)o(b)q(ers)f(of) g(the)h(con)o(text,)h(using)e(an)h(all-to-all)d(comm)o(uni)o(cation.)29 b(A)19 b(new)g(con)o(text)h(table)e(is)75 1356 y(created,)d(with)e(these)j (new)e(tags.)166 1408 y Fb(MPI)p 235 1408 V 15 w(SPLIT)p 360 1408 V 15 w(CONTEXT)j Fc(-)h(A)h(naiv)o(e)f(implemen)o(tatio)o(n)e(is)j(to)f (ha)o(v)o(e)g(an)h(all-to-all)d(comm)o(unicati)o(on)g(where)75 1457 y(eac)o(h)e(pro)q(cess)h(gathers)f(the)h Fb(\(key,)20 b(index,)h(new)p 881 1457 V 15 w(context)p 1050 1457 V 14 w(tag\))13 b Fc(triples)h(of)e(all)h(pro)q(cesses)j(in)d(the)h(old)f(group.)75 1507 y(Eac)o(h)g(pro)q(cess)h(then)f(creates)i(a)d(new)h(con)o(text)g(table)g (for)f(the)h(pro)q(cesses)i(that)e(pro)o(vided)f(the)i(same)d(k)o(ey)i(v)n (alue)f(as)75 1557 y(it.)166 1609 y Fb(MPI)p 235 1609 V 15 w(NEW)p 316 1609 V 15 w(CONTEXT)g Fc(-)i(Similar)d(to)j Fb(MPI)p 785 1609 V 15 w(COPY)p 888 1609 V 15 w(CONTEXT)p Fc(.)166 1661 y Fb(MPI)p 235 1661 V 15 w(RANK,)21 b(MPI)p 447 1661 V 15 w(SIZE)13 b Fc(-)g(require)i(lo)q(cal)e(access)j(to)e(the)g(con)o(text)h(ob)r(ject)75 1899 y Fl(1.3)70 b(Adv)l(anced)23 b(con)n(text)f(op)r(erations)75 2005 y Fm(Additional)e(functions)e(are)f(required)i(to)e(supp)q(ort)h(a)f (less)i(static)e(mo)q(del,)i(where)f(pro)q(cesses)g(ma)o(y)f(b)q(e)75 2062 y(created)i(or)f(deleted)i(during)g(execution.)31 b(This)20 b(requires)f(con)o(texts)f(to)h(gro)o(w)e(or)h(b)q(e)i(merged.)30 b(This)75 2118 y(section)12 b(outlines)g(p)q(ossible)h(mec)o(hanisms)f(for)e (suc)o(h)i(extension.)19 b(The)11 b(lac)o(k)h(of)f(in)o(terupt)g(driv)o(en)h (comm)o(u-)75 2175 y(nication)j(mec)o(hanisms)g(in)g(MPI)f(restricts)g(the)g (functionalit)o(y)h(of)f(suc)o(h)g(mec)o(hanisms:)20 b(A)14 b(new)h(pro)q(cess)75 2231 y(can)g(join)f(an)h(existing)g(con)o(text)f(only)h (if)g(all)g(pro)q(cesses)g(that)f(participate)h(in)g(the)f(old)h(con)o(text)f (execute)75 2287 y(a)h(call)h(to)f(add)g(this)h(new)f(pro)q(cess.)166 2346 y(A)j(prede\014ned)i(lo)q(cal)g(con)o(text)e Fe(MPI)p 792 2346 15 2 v 16 w(ME)g Fm(where)h(only)g(one)f(pro)q(cess)h(participates,) g(is)g(prede\014ned)75 2403 y(for)c(eac)o(h)g(pro)q(cess.)166 2497 y Fk(MPI)p 275 2497 16 2 v 18 w(SP)l(A)-6 b(WN\()17 b(new)o(con)o(text,) f(oldcon)o(text,)i(arra)o(y)p 1202 2497 V 18 w(of)p 1262 2497 V 19 w(en)o(vironmen)o(ts,)d(len\))166 2591 y Fm(Spa)o(wn)h(new)h(pro)q (cesses)g(and)g(create)f(a)g(new)h(con)o(text)f(that)g(includes)j(the)d(pro)q (cesses)h(in)h(the)e(old)75 2647 y(con)o(text,)d(follo)o(w)o(ed)h(b)o(y)g (the)g(newly)h(spa)o(wned)f(pro)q(cesses.)19 b(The)14 b(arra)o(y)f(pro)o (vides)h(en)o(vironmen)o(t)g(param-)75 2704 y(eters)j(for)f(eac)o(h)h(newly)h (generated)e(pro)q(cess.)26 b(The)17 b(form)f(of)g(the)h(arra)o(y)f(en)o (tries)h(are)g(implemen)o(tation)p eop %%Page: 6 7 bop 75 -100 a Fm(6)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)75 45 y Fm(dep)q(enden)o(t)i({)e(they)g(ma)o(y)f (include)k(information)d(on)h(the)f(pro)q(cessor)g(that)f(is)i(to)f(run)g (the)h(new)f(tasks,)75 102 y Fe(argv,)23 b(argc)15 b Fm(argumen)o(ts,)f(etc.) 75 214 y Fk(OUT)k(new)o(con)o(text)k Fm(handle)17 b(to)d(new)i(con)o(text)75 314 y Fk(OUT)i(oldcon)o(text)24 b Fm(handle)16 b(to)f(old)h(con)o(text)75 413 y Fk(IN)h(arra)o(y)p 277 413 16 2 v 18 w(of)p 337 413 V 19 w(en)o(vironmen)o(ts)k Fm(list)16 b(of)f(en)o(vironmen)o(ts)g(for)g(new)g (pro)q(cesses)75 512 y Fk(IN)i(len)23 b Fm(n)o(um)o(b)q(er)16 b(of)f(new)g(pro)q(cesses)h(\(in)o(teger\))166 625 y(As)23 b(particular)g(cases,)h Fe(MPI)p 669 625 15 2 v 17 w(SPAWN\()f(newcontext,)f (MPI)p 1211 625 V 17 w(ME,...\))42 b Fm(allo)o(ws)23 b(one)g(pro)q(cess)g(to) 75 681 y(spa)o(wn)f(new)g(pro)q(cesses,)j(and)d(create)g(a)g(new)h(con)o (text)e(that)h(consists)g(of)g(the)g(spa)o(wning)h(pro)q(cess,)75 737 y(follo)o(w)o(ed)13 b(b)o(y)g(the)g(spa)o(wned)g(pro)q(cesses;)h Fe(MPI)p 849 737 V 17 w(SPAWN\()23 b(newcontext,)f(MPI)p 1391 737 V 17 w(ALL,...\))c Fm(allo)o(ws)13 b(to)g(add)75 794 y(to)21 b(create)g(a)g(new)h(\\univ)o(ersal")g(con)o(text)e(that)h(con)o(tains)h(all) g(previous)g(pro)q(cesses)g(and)g(the)f(newly)75 850 y(spa)o(wned)15 b(pro)q(cesses.)166 944 y Fk(MPI)p 275 944 16 2 v 18 w(BCAST)p 473 944 V 19 w(CONTEXT\()d(new)o(con)o(text,)g(con)o(text,)g(ro)q(ot,)h(arra) o(y)p 1514 944 V 18 w(of)p 1574 944 V 19 w(ranks,)e(coun)o(t\))166 1093 y Fm(Beha)o(v)o(es)i(lik)o(e)h Fe(MPI)p 495 1093 15 2 v 16 w(NEW)p 583 1093 V 17 w(CONTEXT)p Fm(,)e(except)h(that)f(only)h(one)g (pro)q(cess)g(in)h(the)f(old)g(con)o(text,)f(namely)75 1150 y(the)21 b(pro)q(cess)g(with)h(rank)e Fe(root)p Fm(,)i(has)e(to)h(compute)g (the)g(list)h(of)e(the)h(ranks)g(of)f(the)h(pro)q(cesses)h(that)75 1206 y(participate)c(in)g(the)f(new)h(con)o(text.)25 b(The)17 b(new)g(con)o(text)g(do)q(es)h(not)e(necessarily)j(include)g(the)f(pro)q (cess)75 1263 y Fe(root)p Fm(.)h(The)c(call)h(is)f(blo)q(c)o(king,)h(and)f (it)g(has)f(to)g(b)q(e)i(executed)f(b)o(y)g(all)g(pro)q(cesses)h(in)f(the)g (list)g(and)g(b)o(y)g(the)75 1319 y(ro)q(ot)f(pro)q(cess.)75 1432 y Fk(OUT)k(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h(created)g(con) o(text)f(at)h(calling)h(pro)q(cess.)19 b(This)14 b(handle)g(should)189 1488 y(not)g(b)q(e)i(asso)q(ciated)g(with)f(an)g(ob)s(ject)g(b)q(efore)h(the) f(call.)75 1587 y Fk(IN)i(con)o(text)23 b Fm(handle)17 b(to)d(old)i(con)o (text)75 1687 y Fk(IN)h(ro)q(ot)23 b Fm(index)17 b(of)e(new)g(con)o(text)g (creator)f(in)i(old)g(con)o(text)75 1786 y Fk(IN)h(arra)o(y)p 277 1786 16 2 v 18 w(of)p 337 1786 V 19 w(ranks)22 b Fm(ranks)17 b(in)h(old)g(con)o(texts)e(of)h(mem)o(b)q(ers)h(of)f(new)g(con)o(text;)g (signi\014can)o(t)i(only)f(at)189 1842 y(ro)q(ot)75 1941 y Fk(IN)f(coun)o(t)23 b Fm(size)16 b(of)f(new)h(con)o(text;)e(signi\014can)o(t) i(only)g(at)e(ro)q(ot)166 2130 y Fd(Implemen)o(tati)o(on)e(note:)166 2181 y Fc(The)19 b(call)e(can)i(b)q(e)g(implemen)o(ted)d(as)i(a)g(broadcast)h (from)e(the)i(ro)q(ot)f(to)g(all)g(new)g(con)o(text)h(participan)o(ts,)75 2231 y(follo)o(w)o(ed)12 b(b)o(y)h(the)h(co)q(de)g(of)f Fb(MPI)p 574 2231 14 2 v 15 w(NEW)p 655 2231 V 15 w(CONTEXT)f Fc(\(the)i(broadcast)g (is)f(executed)i(using)e(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(uni-)75 2281 y(cation\).)166 2421 y Fm(Example:)34 b(Supp)q(ose)24 b(one)e(has)g(a)g(serv)o(er)g(con)o(text)g Fe(SERVER)p Fm(,)f(and)h(a)g (clien)o(t)i(con)o(text)d Fe(CLIENT)p Fm(.)75 2478 y(Pro)q(cesses)16 b(in)i(either)f(con)o(texts)e(ma)o(y)h(not)g(kno)o(w)g(ab)q(out)g(pro)q (cesses)h(in)g(the)g(other)f(con)o(text,)f(but)i(they)75 2534 y(all)h(b)q(elong)g(to)e(con)o(text)g Fe(MPI)p 582 2534 15 2 v 17 w(ALL)p Fm(,)g(and)h(all)g(kno)o(w)g(ab)q(out)f(pro)q(cess)h Fe(root)g Fm(\(the)f(\\nameserv)o(er"\).)24 b(The)75 2591 y(ro)q(ot)17 b(pro)q(cess)g(kno)o(ws)g(ab)q(out)h(eac)o(h)f(of)g(the)h(t)o(w)o(o)e(con)o (texts.)26 b(A)17 b(call)i(to)e Fe(MPI)p 1408 2591 V 17 w(BCAST)p 1545 2591 V 16 w(CONTEXT)g Fm(can)g(b)q(e)75 2647 y(used)e(to)f(create)g(a)g (new)h(con)o(text)e(that)h(is)h(the)g(union)g(of)f(the)g Fe(CLIENT)g Fm(and)h Fe(SERVER)e Fm(con)o(texts,)h(so)g(as)g(to)75 2704 y(allo)o(w)h(these)h(to)e(comm)o(unicate.)p eop %%Page: 7 8 bop 75 -100 a Ff(1.4.)29 b(EXAMPLES)15 b({)g(PR)o(OBLEMS)h(IN)g(RICK'S)g (LIST)756 b Fm(7)166 45 y Fd(Discussion:)166 96 y Fc(P)o(ossible)12 b(extension:)18 b(the)13 b(ro)q(ot)f(has)g(t)o(w)o(o)g(lists;)g(the)h(list)e (of)h(pro)q(cesses)j(that)d(call)f(to)h(join)f(the)i(new)g(con)o(text,)75 146 y(and)j(the)g(list)f(of)g(pro)q(cesses)k(that)c(are)i(to)e(join)o(t)g (the)h(next)h(con)o(text.)24 b(The)16 b(remianing)e(pro)q(cesses)k(are)e (returned)75 196 y(a)g(v)n(alue)f(indicating)g(they)i(w)o(ere)g(left)e(out.) 25 b(This)16 b(could)g(b)q(e)g(useful)h(for)e(master-sla)o(v)o(e)g(co)q(de.) 26 b(The)16 b(ro)q(ot)g(is)g(the)75 246 y(master)f(that)h(creates)i(new)e (con)o(texts)h(for)e(the)h(parallel)f(executuin)i(of)e(new)h(tasks.)24 b(All)15 b(disp)q(onible)g(pro)q(cesses)75 296 y(execute)h(the)e(call,)f(but) h(only)f(a)h(subset)h(is)f(allo)q(cated.)166 472 y Fk(MPI)p 275 472 16 2 v 18 w(MER)o(GE)p 490 472 V 19 w(CONTEXT\()k(new)o(con)o(text,)f (con)o(text1,)g(con)o(text2,)g(ro)q(ot\))166 565 y Fm(This)e(function)g(allo) o(ws)g(to)e(merge)h(t)o(w)o(o)f(con)o(texts)h(ev)o(en)h(when)g(there)f(is)h (no)f(con)o(text)g(that)g(encom-)75 621 y(passes)k(the)g(t)o(w)o(o)f(merged)h (con)o(texts;)g(it)g(is)h(only)f(required)h(that)e(one)i(pro)q(cess,)f (namely)h Fe(root)p Fm(,)e(b)q(e)i(in)75 678 y(b)q(oth)13 b(con)o(texts)e(to) h(b)q(e)h(merged.)19 b(The)13 b(call)h(is)f(blo)q(c)o(king)h(and)e(m)o(ust)g (b)q(e)h(executed)h(b)o(y)e(all)i(pro)q(cesses)f(that)75 734 y(participate)k(in)f(the)g(merged)g(con)o(text.)21 b(Eac)o(h)16 b(pro)q(cess)g(pro)o(vides)g(its)g(old)g(con)o(text,)f(and)h(the)g(index)h (of)75 791 y(pro)q(cess)c(ro)q(ot)f(within)i(this)g(old)f(con)o(text)f(\(the) h(index)h(ma)o(y)e(b)q(e)i(di\013eren)o(t)f(in)g(the)g(t)o(w)o(o)f(con)o (texts)g(that)g(are)75 847 y(merged\).)19 b(The)14 b(ro)q(ot)g(pro)o(vides)g (handles)h(to)e(b)q(oth)i(con)o(texts)e(that)g(are)h(to)f(b)q(e)i(merged.)k (The)c(pro)q(cesses)75 904 y(in)h Fe(context1)e Fm(precede)i(the)g(pro)q (cesses)f(in)h Fe(context2)e Fm(in)i(the)g(ranking)f(of)g(the)g(new)h(con)o (text.)75 1017 y Fk(OUT)i(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h (created)g(con)o(text)f(at)h(calling)h(pro)q(cess.)19 b(This)14 b(handle)g(should)189 1073 y(not)g(b)q(e)i(asso)q(ciated)g(with)f(an)g(ob)s (ject)g(b)q(efore)h(the)f(call.)75 1173 y Fk(IN)i(con)o(text1)23 b Fm(handle)17 b(to)d(old)i(con)o(text)75 1272 y Fk(IN)h(con)o(text2)23 b Fm(handle)17 b(to)d(second)i(con)o(text;)e(signi\014can)o(t)i(only)g(at)f (ro)q(ot)75 1372 y Fk(IN)i(ro)q(ot)23 b Fm(index)17 b(of)e(ro)q(ot)f(in)i (old)g(con)o(text)166 1485 y(Example:)23 b(a)17 b(serv)o(er)f(\(the)h(\\ro)q (ot"\))e(and)i(a)f(set)h(of)f(clien)o(t)i(pro)q(cesses)f(participate)h(in)f Fe(context1)p Fm(.)75 1541 y(The)j(serv)o(er)f(spa)o(wns)g(new)h(\\help)q (ers",)h(or)e(has)h(other)f(a)o(v)m(ailable)i(serv)o(er)e(pro)q(cesses)h (join)g(it.)33 b(A)20 b(call)75 1598 y(to)e Fe(MPI)p 209 1598 15 2 v 16 w(MERGE)p 345 1598 V 17 w(CONTEXT)f Fm(can)i(b)q(e)g(used)g(to)e (create)h(a)h(new)f(con)o(text)g(that)f(mak)o(e)h(these)h(new)f(serv)o(ers)75 1654 y(a)o(v)m(ailable)f(to)d(the)i(parallel)g(clinet.)166 1788 y Fd(Discussion:)166 1839 y Fc(Here,)h(to)q(o,)f(the)h(function)e(can)h (b)q(e)h(extended)h(to)d(allo)o(w)g(the)h(creation)h(of)e(a)h(new)g(con)o (text)h(where)g(only)e(a)75 1889 y(subset)g(of)f(the)g(pro)q(cesses)j(in)c (the)i(old)e(group)g(participate)166 1940 y(If)k Fb(MPI)p 280 1940 14 2 v 15 w(MERGE)p 405 1940 V 14 w(CONTEXT)f Fc(is)h(a)o(v)n(ailable)e (then)j(the)g(functionalit)o(y)d(of)i Fb(MPI)p 1344 1940 V 15 w(SPAWN)f Fc(can)i(b)q(e)f(reduced:)27 b(It)17 b(is)75 1990 y(su\013cien)o(t)e(to)e(create)i(a)f(new)g(con)o(text)g(that)g(consists)h(of) e(a)g(spa)o(wning)g(pro)q(cess)j(\(rather)e(than)g(sp)o(wning)f(con)o(text\)) 75 2040 y(and)h(the)g(spa)o(wned)h(pro)q(cesses.)20 b(This)14 b(new)g(con)o(text)h(can)f(then)h(b)q(e)f(merged)g(with)f(a)h(preexisting)g (con)o(text.)166 2256 y Fd(Implemen)o(tati)o(on)e(note:)166 2308 y Fc(Implemen)o(tation)f(is)i(similar)f(to)i(the)g(implemen)o(tation)c (of)k Fb(MPI)p 1179 2308 V 15 w(BCAST)p 1304 2308 V 14 w(CONTEXT)75 2542 y Fl(1.4)70 b(Examples)22 b({)h(Problems)f(in)g(Ric)n(k's)g(list)75 2646 y Fm(Only)16 b(the)g(basic)g(con)o(text)e(functions)i(are)f(used)h(in)g (these)f(examples.)166 2704 y Fd(W)l(arning)p Fc(:)h(late)e(nigh)o(t)f(w)o (ork)p eop %%Page: 8 9 bop 75 -100 a Fm(8)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)75 45 y Fk(F)l(unction)h(that)h(implemen)o(ts)e(sh)o (u\017e)f(p)q(erm)o(utation)i(in)g(group/con)o(text)75 204 y Fe(void)23 b(function)g(mpi_shuffle\(inbuf,)e(outbuf,)i(datatype,)g(count,) g(context,)g(size\))75 317 y(...)75 430 y({)75 486 y (mpi_copy_context\(newcontex)o(t,)e(context\);)75 543 y(mpi_rank\(context,)h (i\);)75 599 y(dest)h(=)h(shuffle\(i,)f(size\);)75 656 y(source)g(=)h (unshuffle\(i,)e(size\);)75 712 y(mpi_irecvc)g(\(handle,)h(inbuf,)g(count,)g (datatype,)g(source,)g(0,)g(nexcontext\);)75 769 y(mpi_sendc)g(\(outbuf,)f (count,)h(datatype,)g(dest,)g(0,)h(newcontext\);)75 825 y(mpi_wait\(handle,)e (ret_stat\);)75 882 y(mpi_free\(newcontext\);)75 938 y(})166 1095 y Fm(This)15 b(w)o(orks)f(OK,)h(if)g(pro)q(cesses)g(are)g (single-threaded.)21 b(If)15 b(they)g(are)g(m)o(ultithreaded,)g(one)g(has)g (to)75 1152 y(mak)o(e)f(sure)g(that)g(other)f(concurren)o(t)i(threads)f(do)g (not)g(create)g(new)g(con)o(texts.)19 b(New)14 b(con)o(text)g(creation)75 1208 y(can)h(b)q(e)h(skipp)q(ed)h(if)f(there)f(are)g(no)g(p)q(ending)i (messages)e(in)h(the)f(con)o(text)g(when)g(mpi)p 1539 1208 14 2 v 17 w(sh)o(u\017e)h(is)g(called.)75 1338 y Fk(Use)25 b(of)h(con)o(texts)g(for)f(library)h(dev)o(elopmen)o(t)44 b Fm(The)23 b(co)q(de)g(of)f(a)g(parallel)i(library)f(function)75 1395 y(should)18 b(b)q(e)f(written)g(so)f(that)g(eac)o(h)h(message)f(that)g (is)h(pro)q(duced)h(b)o(y)e(some)h(pro)q(cess)g(is)g(consumed)g(b)o(y)75 1451 y(another)d(pro)q(cess)h(during)h(the)f(execution)g(of)g(the)f(library)i (co)q(de)f({)f(i.e.)21 b(the)14 b(library)i(should)f(\\clean)h(its)75 1508 y(garbage".)166 1566 y(A)d(parallel)j(library)e(is)g(in)o(v)o(ok)o(ed)g (collectiv)o(ely)i(b)o(y)d(a)g(group)g(of)h(pro)q(cesses.)19 b(An)o(y)14 b(in)o(v)o(ok)m(ation)g(of)f(the)75 1622 y(library)k(o)q(ccurs)g (within)g(a)f(con)o(text)g(de\014ned)i(b)o(y)e(the)h(user.)23 b(All)18 b(pro)q(cesses)f(that)e(participate)i(in)h(suc)o(h)75 1679 y(con)o(text)g(in)o(v)o(ok)o(e)h(the)g(parallel)h(library)l(,)h(and)e (matc)o(hing)g(in)o(v)o(ok)m(ations)g(o)q(ccur)g(at)f(all)i(of)e(them)h(in)h (the)75 1735 y(same)15 b(order.)166 1793 y(The)10 b(library)h(has)f(to)g (generate)g(a)f(new)i(con)o(text,)f(using)h(the)f(function)h Fe(MPI)p 1427 1793 15 2 v 17 w(COPY)p 1540 1793 V 17 w(CONTEXT\(newcontext,) 75 1850 y(context\))p Fm(,)16 b(for)h(eac)o(h)g(con)o(text)g Fe(context)f Fm(where)h(from)g(the)g(library)h(ma)o(y)f(b)q(e)h(in)o(v)o(ok)o (ed.)26 b(The)17 b(library)75 1906 y(will)g(then)f(use)g(the)g(con)o(text)f Fe(newcontext)f Fm(for)h(its)h(comm)o(unication)h(whenev)o(er)f(it)g(is)g(in) o(v)o(ok)o(ed)g(within)75 1963 y(the)h(con)o(text)f Fe(context)p Fm(.)24 b(If)17 b(the)g(library)g(uses)h(collectiv)o(e)g(comm)o(unication)g (within)g(dynamically)g(de-)75 2019 y(\014ned)h(subgroups,)f(then)g(these)g (subgroups)g(will)i(b)q(e)f(created)e(b)o(y)h(splitting)i(the)d(group)h(of)g (pro)q(cesses)75 2076 y(de\014ned)f(b)o(y)e Fe(newcontext)p Fm(.)166 2134 y(In)23 b(the)g(general)g(case,)h(the)e(con)o(text)g(of)g(the)h (in)o(v)o(ok)m(ation)g(has)f(to)g(b)q(e)h(passed)g(as)f(an)h(explicit)75 2190 y(parameter)14 b(when)i(the)f(library)h(is)f(in)o(v)o(ok)o(ed.)21 b(The)15 b(library)h(co)q(de)f(will)i(generate)e(a)g(new)g(con)o(text)f(when) 75 2247 y(it)h(starts)f(executing)j(and)e(will)i(free)e(this)h(con)o(text)e (when)i(it)g(terminates.)166 2305 y(There)f(are)g(sev)o(eral)h(sp)q(ecial)h (cases)e(where)g(dynamic)h(con)o(text)f(creation)g(can)h(b)q(e)f(a)o(v)o (oided.)166 2363 y(Case)10 b(1:)18 b(The)11 b(library)g(is)h(alw)o(a)o(ys)e (in)o(v)o(ok)o(ed)h(within)h(the)e(con)o(text)h Fe(MPI)p 1343 2363 V 16 w(ALL)p Fm(.)f(Then)i(the)f(new)g(con)o(text)75 2420 y(can)18 b(b)q(e)g(\\cac)o(hed":)25 b(A)17 b(library)i(collectiv)o(e)g (initialization)i(routine)d(should)g(b)q(e)h(in)o(v)o(ok)o(ed)f(b)o(y)f(the)h (user)75 2476 y(at)c(the)g(start)f(of)h(the)g(program;)f(this)i(routine)f (creates)g(a)g(cop)o(y)g(of)g(the)g(con)o(text)g Fe(MPI)p 1537 2476 V 17 w(ALL)f Fm(and)i(stores)e(a)75 2532 y(handle)k(to)e(it)h(in)g(a)g (C)f(static)h(v)m(ariable)h(\(F)l(ortran)d(77)h(COMMON\).)g(The)h(con)o(text) f Fe(MPI)p 1602 2532 V 17 w(ALL)g Fm(need)h(not)75 2589 y(b)q(e)g(passed)f (as)g(a)g(parameter)f(when)i(the)f(library)h(is)g(in)o(v)o(ok)o(ed.)166 2647 y(Case)g(2:)22 b(The)17 b(library)g(is)g(alw)o(a)o(ys)e(in)o(v)o(ok)o (ed)i(in)g(a)f(unique)i Fh(libr)n(ary)f(c)n(al)r(ling)f(c)n(ontext)g Fm(on)h(eac)o(h)f(pro-)75 2704 y(cess.)29 b(Then)18 b(a)g(library)h (initialization)i(routine)e(should)g(b)q(e)g(in)o(v)o(ok)o(ed)f(b)o(y)g(the)g (user)g(on)h(eac)o(h)f(pro)q(cess)p eop %%Page: 9 10 bop 75 -100 a Ff(1.4.)34 b(EXAMPLES)16 b({)e(PR)o(OBLEMS)j(IN)e(RICK'S)h (LIST)751 b Fm(9)75 45 y(where)16 b(the)g(library)g(ma)o(y)f(b)q(e)h(in)o(v)o (ok)o(ed;)g(the)f(initialization)k(routine)d(is)g(called)h(after)e(the)h (user)g(de\014ned)75 102 y(the)i(library)g(calling)i(con)o(texts)d(and)h(b)q (efore)g(the)g(library)g(is)g(in)o(v)o(ok)o(ed.)28 b(Eac)o(h)18 b(initialization)i(call)f(is)f(a)75 158 y(collectiv)o(e)d(call)f(within)g (the)f(library)h(calling)g(con)o(text.)19 b(A)13 b(cop)o(y)g(of)f(the)h (calling)i(con)o(text)d(is)h(created)g(\(b)o(y)75 214 y Fe(MPI)p 150 214 15 2 v 17 w(COPY)p 263 214 V 16 w(CONTEXT)p Fm(\))i(and)i(stored)e (in)j(a)d(static)h(library)h(v)m(ariable.)25 b(Subsequen)o(t)17 b(calls)g(to)f(the)g(library)75 271 y(need)g(not)f(pass)g(the)g(calling)i (con)o(text)e(as)g(a)f(parameter.)166 327 y(Case)19 b(3:)27 b(The)20 b(library)g(ma)o(y)e(b)q(e)i(in)o(v)o(ok)o(ed)f(on)h(eac)o(h)f(pro)q (cess)g(within)i(a)d(\014xed)i(\(small\))g(n)o(um)o(b)q(er)75 384 y(of)f(library)h(calling)h(con)o(texts.)32 b(Copies)20 b(of)f(these)g(con)o(texts)g(can)h(b)q(e)g(created)f(b)q(efore)h(the)f (library)h(is)75 440 y(in)o(v)o(ok)o(ed)e(and)g(\\cac)o(hed")g(in)h(static)f (v)m(ariables.)29 b(Subsequen)o(t)19 b(in)o(v)o(ok)m(ations)f(to)g(the)g (library)h(need)f(not)75 497 y(create)d(new)g(copies,)h(but)f(only)h(select)g (the)f(righ)o(t)g(preexisting)i(cop)o(y)l(.)166 547 y Fc(This)9 b(assumes)h(one)f(can)h(test)g(con)o(texts)h(\(con)o(text)f(handles\))g(for)f (equalit)o(y)m(.)15 b(This)9 b(should)g(b)q(e)h(said)g(explicitely)75 596 y(in)j(the)i(draft)75 704 y Fk(Use)i(of)h(con)o(texts)f(for)g(a)h(host)f (no)q(de)h(computation)i(mo)q(del)42 b Fc(Let's)14 b(assume)g(t)o(w)o(o)f (con)o(texts:)145 779 y Fa(\017)23 b Fb(MPI)p 258 779 14 2 v 15 w(ALL)13 b Fc(with)g(host)i(b)q(eing)e(no)q(de)i(zero)145 854 y Fa(\017)23 b Fb(MPI)p 258 854 V 15 w(NODE)13 b Fc(that)h(do)q(es)g(not) g(include)g(the)h(host)166 928 y(W)m(e)e(assume)h(the)g(load)f(pro)q(cedure)j (ensures)g(that)e(host)g(is)g(pro)q(cess)i(zero)e(in)g(ALL)75 1078 y Fb(host/node)20 b(code)479 b(translation)75 1128 y(--------------)e (-------------)75 1227 y(I_am_the_host\(\))455 b(\(mpi_rank\(MPI_A)o(LL,)19 b(rank\);)860 1277 y(rank==0;\))75 1377 y(form)i(node)g(group)457 b(mpi_rank\(MPI_AL)o(L,)19 b(task\))860 1427 y(mpi_split_conte)o(xt\(MP)o (I_NOD)o(E,)g(MPI_ALL,)903 1476 y(\(task==0\),0\);)75 1576 y(Broadcast)h(from)h(host)g(to)g(nodes)174 b(mpi_bcast\(MPI_A)o(LL,0,)o (...\))75 1676 y(regular)20 b(communications)f(btwn)196 b(mpi_send\(buffer)o (,len,)o(dest,)o(tag,M)o(PI_NO)o(DE\))75 1725 y(nodes)675 b(mpi_recv\(buffer) o(,len,)o(sourc)o(e,tag)o(,MPI_)o(NODE)o(\))184 1775 y(/*)21 b(source,)g(dest)g(are)g(ranging)f(from)h(0)h(to)f(#nodes-1)f(*/)75 1875 y(sum)h(values)g(from)g(each)g(node)239 b(mpi_reduce\(inbu)o(f,out)o (buf,l)o(en,MP)o(I_ALL)o(,)75 1925 y(at)21 b(host)893 b(0,MPI_ISUM\))228 1975 y(/*)21 b(host)g(\(node)g(zero\))f(calls)h(mpi_reduce)f(with)h(inbuf)g (=)g(0)h(*/)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-context@CS.UTK.EDU Mon May 10 05:13:04 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA04059; Mon, 10 May 93 05:13:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14410; Mon, 10 May 93 02:59:44 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Mon, 10 May 1993 02:59:42 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14402; Mon, 10 May 93 02:59:37 -0400 Message-Id: <9305100659.AA14402@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6287; Mon, 10 May 93 03:00:12 EDT Date: Mon, 10 May 93 03:00:11 EDT From: "Marc Snir" To: MPI-CONTEXT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: CON-03.DVI.* %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 16 df15 D E /Fb 58 123 df35 D<13E01201EA07C013005A121E5A123812781270A312F05AA77E1270A312781238123C7E7E7E13 C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780120313C01201A313E01200A7 120113C0A3120313801207EA0F00121E121C127C12F05A0B217C9C16>II<12 38127C127EA2123E120E121E123C127C12F81260070B798416>44 DI< 127012F8A312700505788416>I III<1238127CA312381200A81238127CA3123C121C123C123812F812 F012600618799116>59 D61 D<13E0487EA213B0A2EA03B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF 13E0007F13C013197F9816>65 DI<3801F180EA07 FF5AEA1F0FEA3C0712781303127000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C 5AEA01F011197E9816>II<387FFFC0B5FC7EEA1C01A490C7FCA2 131CA2EA1FFCA3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816>I71 D73 D<387F0FE038FF8FF0387F0FE0381C0780EB0F00131E131C133C5B5BEA1DE07F121F7F1338EA1E 3C131CEA1C1E7F7F14801303387F07E038FF8FF0387F07E01419809816>75 DI<38FC07E0EAFE0FA2383A0B80EA3B 1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313197F9816>I<387E1FC038FF3FE038 7F1FC0381D07001387A313C7A2121CA213E7A31367A21377A21337A31317EA7F1FEAFF9FEA7F0F 13197F9816>III82 DI<387FFFE0B5FCA2EAE0E0A400 001300AFEA07FC487E6C5A13197F9816>I<387F07F038FF8FF8387F07F0381C01C0B0EA1E0300 0E1380EA0F8F3807FF006C5AEA00F81519809816>I<38FC07E0EAFE0FEAFC07387001C0A30030 1380EA3803A313E3EA39F3A213B300191300A61313EA1B1BEA0F1EA2EA0E0E13197F9816>87 D<387F1F80133F131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B81207 131CA2EA0E0EA2487E387F1FC000FF13E0007F13C013197F9816>I<38FE0FE0EAFF1FEAFE0F38 1C0700A2EA0E0EA26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I<387FFF 80B5FCA238E007005B131E131CEA003C5B137013F0485A5B1203485A90C7FC5A381E0380121C12 3C12781270B5FCA311197E9816>I95 D97 D<127E12FE127E120EA4133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13 F014E01381EB83C013FF000E1300EA067C1419809816>II<133F5B7F1307A4EA 03E7EA0FFF123FEA3C1F487E1270EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807 C7E014197F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA 7FFFB5FCA2EA01C0ACEA3FFE487E6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E 00EA380EA3EA3C1E6C5AEA1FFC485AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001 E0EAE000A3EAF001387C07C0383FFF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4 137CEA0FFF148013871303A2120EA9387FC7F038FFE7F8387FC7F01519809816>II<127E12FE127E120EA4EB7F E0A3EB0F00131E5B5B5BEA0FF8A213BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F87141980 9816>107 DI<38FBC78038FFEFC0EBFFE0EA3E7CEA3C 78EA3870AA38FE7CF8A31512809116>IIII<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F 9116>114 DI<12035AA4EA7FFFB5FCA20007C7FCA75BEB03 80A2130713873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F 0FEBFFF06C13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E 0EA36C5AA4EA03B8A3EA01F0A26C5A13127F9116>I<38FF1FE013BF131F38380380A413E33819 F300A213B3EA1DB7A4EA0F1EA313127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813 F012016C5A12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>I<38 7F1FC038FF9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35B A3485A1279127BEA7F806CC7FC123C131B7F9116>I<383FFFC05AA238700780EB0F00131EC65A 13F8485A485A485A48C7FC381E01C0123C1278B5FCA312127F9116>I E /Fc 48 123 df11 D<127012F812FCA2127C120CA31218A2123012601240060D 7D9C0C>39 D<13C0EA0180EA03001206120E120C121C121812381230A21270A21260A212E0AC12 60A21270A21230A212381218121C120C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E 7E121C120C120E120612077EA21380A21201A213C0AC1380A21203A21300A25A1206120E120C12 1C12185A5A5A0A2A7E9E10>I<127012F012F8A212781218A31230A2127012601240050D7D840C> 44 DI<127012F8A3127005057D840C>I<12035A123FB4FC12C71207B3 A3EAFFF8A20D1C7C9B15>49 DII<13F0 EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EAFC1CEAF81E130E12F013 0FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>54 D<127012F8A312701200A812 7012F8A3127005127D910C>58 D<127012F8A312701200A8127012F012F8A212781218A31230A2 127012601240051A7D910C>I<1306130FA3497EA4EB33C0A3EB61E0A3EBC0F0A338018078A2EB FFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C1D7F9C1F>65 D69 D<39FFF3FFC0A2390F003C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F> 72 DI76 D80 D<3807E080EA1FF9EA3C1FEA7007130312E01301 A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA07FFC61380130FEB03C0A2130112C0A300E013801303 00F01300EAFC0EEACFFCEA83F8121E7E9C17>83 D<007FB512C0A238780F030070130100601300 00E014E000C01460A400001400B03803FFFCA21B1C7F9B1E>I<3AFFE0FFE1FFA23A1F001E007C 6C1530143FA20180147000079038678060A32603C0E713C0ECC3C0A2D801E0EBC1809038E181E1 A3D800F3EBF3001400A2017B13F6017E137EA3013C133CA3011C133801181318281D7F9B2B>87 D97 D<12FCA2121CA9137EEA1DFF381F8780381E01C0001C13E013 0014F0A614E01301001E13C0381F07803819FF00EA187C141D7F9C17>IIII<1378EA01FCEA039EEA071EEA0E0C1300A6EAFFE0A2EA0E 00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F07381E0380A2121CAB38FF9FF0 A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C0C>II<12FC A2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA1C781338133C131C7F130F38FF9F E0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF9FF8391F83 B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124>IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0A612781307123CEA1E1FEA0FFB EA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA2123CEAFFE0A2EA1C00A81330A5EA 1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA03E314127F 9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA31312 7F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B061486A2EB318E000713CCA2136038 03E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038070600EA038EEA01DC13D8EA00 F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8A21512809116>I<38FF0FE0A2 381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA35BA3EAF180A200C7C7FC12 7E123C131A7F9116>II E /Fd 18 118 df<1238127C12FEA312 7C12381200A41238127C12FEA3127C123807127D910D>58 D68 D73 D<3BFFFC7FFE0FFCA23B0FC007E000C081D9 E003130100071680EC07F8D803F0EC0300A29039F80CFC0700011506EC1CFE9039FC187E0E0000 150CEC387F90397E303F18A290397F601FB8013F14B002E013F0ECC00F011F5CA26D486C5AA2EC 00036D5CA22E1C7F9B31>87 D97 D99 D101 D<3803F0F0380FFFF8383E1F38383C0F30007C13 80A4003C1300EA3E1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F0 0078A3007813F0383E03E0381FFFC03803FE00151B7F9118>103 D<121E123FA25A7EA2121EC7 FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>105 D108 D<39FF1FC0FE90387FE3FF3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F91 28>I<38FF1FC0EB7FE0381FE1F0EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1FC0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7F E0EB1F8090C7FCA6EAFFE0A2171A7F911B>I114 DI<1203A35AA25AA2123FEA FFFCA2EA1F00A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F 03EBFEFFEA03F818127F911B>I E /Fe 55 126 df<137013F01201EA03C0EA0780EA0F00121E 121C123C123812781270A212F05AA87E1270A212781238123C121C121E7EEA0780EA03C0EA01F0 120013700C24799F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370 A813F013E0A2120113C0120313801207EA0F00121E5A5A5A12600C247C9F18>I<123C127E127F A3123F120F120E121E127C12F81270080C788518>44 D<127812FCA412780606778518>46 D48 DII<123C127EA4123C1200A812 38127C127EA3123E120E121E123C127812F01260071A789318>59 D<387FFFC0B512E0A3C8FCA4 B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4 380FFF80A3EA0E03A2381C01C0A2387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA 1E03EA3C01EA78001270A200F013005AA87E007013E0A21278EA3C01001E13C0EA0F073807FF80 6C1300EA01FC131C7E9B18>I69 DI<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7 FC5AA5EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C 7E9B18>I73 D76 D<38FC01F8EAFE03A2383B06E0A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03 F8A3151C7F9B18>I<387E07F038FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A2 13311339A31319A2131D130DA3EA7F07EAFF87EA7F03151C7F9B18>III82 D<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A290C7FC7E1278123FEA1FF0EA0FFEEA01FF38 001F80EB03C0EB01E01300A2126012E0130100F013C0EAFC07B512801400EAE7FC131C7E9B18> I<387FFFF8B5FCA238E07038A400001300B2EA07FFA3151C7F9B18>I<38FF07F8A3381C01C0A4 380E0380A4EA0F0700071300A4EA038EA4EA018C13DCA3EA00D813F8A21370151C7F9B18>86 D<38FE03F8A338700070A36C13E0A513F8A2EA39DCA2001913C0A3138CEA1D8DA4000D13801305 EA0F07A2EA0E03151C7F9B18>I<387F8FE0139F138F380E0700120FEA070E138EEA039C13DCEA 01F8A26C5AA2137013F07F120113DCEA039E138EEA070F7F000E13801303001E13C0387F07F038 FF8FF8387F07F0151C7F9B18>I<38FF07F8A3381C01C0EA1E03000E1380EA0F0700071300A2EA 038EA2EA01DCA213FC6C5AA21370A9EA01FC487E6C5A151C7F9B18>I95 D97 D<127E12FE127E120EA5133EEBFF 80000F13C0EBE3E0EB80F0EB00701478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E13 8038067E00151C809B18>IIII< EB1FC0EB7FE013FFEA01F1EBC0C01400A3387FFFC0B5FCA23801C000AEEA7FFFA3131C7F9B18> I<3803F1F03807FFF85A381E1F30383C0F00EA3807A5EA3C0FEA1E1EEA1FFC485AEA3BF00038C7 FC123CEA1FFF14C04813E0387801F038F00078481338A36C1378007813F0EA7E03383FFFE0000F 13803803FE00151F7F9318>I<127E12FE127E120EA5133FEBFF80000F13C0EBE1E013801300A2 120EAA387FC3FC38FFE7FE387FC3FC171C809B18>II<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B 5B5BEA0FF87F139C131EEA0E0FEB0780130314C038FFC7F8A3151C7F9B18>107 DI<387DF1F038FFFBF86CB4 7E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBFBF80397F1F1F001914819318>IIII<387F87E038FF9FF8EA7FBF3803FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A 15147F9318>114 DI<487E1203A4387FFFC0 B5FCA238038000A9144014E0A21381EBC3C0EA01FF6C1380EB7E0013197F9818>I<387E07E0EA FE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13FE3801FCFC1714809318>I<387F8FF000FF13 F8007F13F0381E03C0000E1380A338070700A3EA038EA4EA01DCA3EA00F8A2137015147F9318> I<38FF8FF8A3383800E0A3381C01C0A2137113F9A213D9A2380DDD80A3138DEA0F8FA238070700 15147F9318>I<387F8FF0139F138F38070700138EEA039EEA01DC13F81200137013F07FEA01DC EA039E138EEA0707000F1380387F8FF000FF13F8007F13F015147F9318>I<387F8FF000FF13F8 007F13F0380E01C0EB0380A21207EB0700A2EA03871386138EEA01CEA2EA00CCA213DC1378A313 70A313F05B1279EA7BC0EA7F806CC7FC121E151E7F9318>I<383FFFF05AA2387001E0EB03C0EB 078038000F00131E137C5B485A485AEA0780380F0070121E5A5AB512F0A314147F9318>II<127CB47E7FEA07E01200AB7FEB7FC0EB3FE0A2EB7FC0EBF0005BAB1207B45A5B007CC7 FC13247E9F18>125 D E /Ff 25 124 df<1238127CA2127E123E120CA31218A21230126012C0 1280070E789F0D>39 D<1230127812F81278127005057C840D>46 D<130C131C13FCEA0FF81338 1200A41370A613E0A6EA01C0A61203EA7FFE12FF0F1E7C9D17>49 D<133FEBFFC03801C1E03803 00F000061378A2000F137C1380A2EB00781206C712F814F0130114E0EB03C0EB0780EB0F00131C 5B5B13C0485A380300601206001C13C05AEA7FFFB51280A2161E7E9D17>I<137F3801FFC03803 83E0EA0701EB00F05A1301A2000013E0A2EB03C0EB0780EB0F0013FE13F8130E7F1480EB03C0A3 EA3007127812F8A238F00F8000C01300EA601EEA383CEA1FF8EA07E0141F7D9D17>II<14181438A21478147C14FCA2EB01BCA2 EB033C143EEB061EA2130CA21318141F497EA21360EB7FFF90B5FCEBC00F3901800780A2EA0300 A21206A2001F14C039FFC07FFCA21E207E9F22>65 D<0007B5FC15C039003C01E015F090387800 F8A515F0EBF001EC03E0EC07C0EC0F809038FFFE00ECFF803901E007C0EC03E0A21401A215F0D8 03C013E01403A2EC07C0A2EC0F800007EB3F00387FFFFEB512F01D1F7E9E20>I<903803F80890 380FFE1890383F073890387801F83801F000D803C013F0000714705B48C7FC5A121E003E146012 3C007C1400A45AA415C01278EC0180127C003CEB0300A26C13065C6C6C5A3807E0703801FFC06C 6CC7FC1D217B9F21>I<0007B5FC15E039003C01F0EC00F849137C153C151EA3151F5BA6484813 1E153EA3153C157C4848137815F0A2EC01E0EC03C0EC0F800007EB3F00387FFFFCB512E0201F7E 9E23>I<0007B512F8A238003C001578491338A5EC0C30EBF0181500A21438EBFFF8A23801E070 1430A2151815301400485A1560A215E0EC01C014030007130F007FB51280B6FC1D1F7E9E1F>I< 3A07FFC7FFC0A23A003C007800A2495BA649485AA490B5FCA23901E003C0A64848485AA6000713 0F397FFCFFF8485A221F7E9E22>72 D<3807FFE0A238003C00A25BA65BA6485AA6485AA61207EA FFFCA2131F7F9E10>I<3A07FFE0FFE0A23A003C003E001538495B5DEC01804AC7FC14065C495A 5C5CEBF1F013F3EBF778EA01EC13F8497E13E080A248487EA36E7EA26E7E0007497E397FFC1FFC 00FF133F231F7E9E23>75 D<3807FFF0A2D8003CC7FCA25BA65BA6485AA3EC0180A2EC0300EA03 C0A25C1406140E141E0007137E387FFFFCB5FC191F7E9E1C>II<3A07FC03FFC0A23A003E007C001538016F1330A3EB6780A2EB63C001C35BA2EB C1E0A2EBC0F0A2D801805B1478A2143CA33903001F80A2140FA3481307D80F8090C7FC387FF003 12FF221F7E9E22>II<0007B5FC15C039003C03E0 EC01F0EB780015F8A59038F001F0A215E0EC03C0EC0F809038FFFE004813F801E0C7FCA5485AA6 1207EA7FFC12FF1D1F7E9E1F>I<3807FFFC14FF39003C07C0EC03E0EB780115F0A415E0EBF003 15C0EC0780EC1F00EBFFFC14F03801E038143C141CA2141EA23803C03EA41506A20007140C397F FC1F1800FFEB0FF8C7EA03E01F207E9E21>82 DI<001FB512F8A2381E03 C000381438EB078012300070141812601538153038C00F0000001400A5131EA65BA6137C381FFF F05A1D1F7B9E21>I<39FFF007FEA2390F0001F0EC00C0A2EC0180EA0780EC03005C1406140EEB C00C0003131C14185CA25CEA01E05CA2EBE180A2D800F3C7FCA213F6A213FCA21378A21370A21F 207A9E22>86 D<3A03FFC1FFC0A23A003E007C00011E137015606D5B1401EC8380010790C7FC14 C6EB03CC14FC6D5A5C1300801301497E143C1306497E131CEB381FEB300F01607FEBC007000180 38038003D80FC07F39FFF01FFE13E0221F7F9E22>88 D123 D E /Fg 30 122 df<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA212 01B3A7387FFFE0A213207C9F1C>49 DI I<14E013011303A21307130F131FA21337137713E7EA01C71387EA03071207120E120C12181238 127012E0B512FEA2380007E0A7EBFFFEA217207E9F1C>I67 D70 D76 D 85 D97 D II I<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA212 7E003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F0007B512 80380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC 121CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6C B45A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A2 0C247FA30F>I108 D<3AFF87F00FE090399FFC3FF8 3A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFFF1FFE3FFC0A22A167E952F>I< 38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0A21A167E951F>I<13FE3807FF C0380F83E0381E00F0003E13F848137CA300FC137EA7007C137CA26C13F8381F01F0380F83E038 07FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEBC03E497E1580A2EC0FC0A8EC1F 80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A21A207E951F>I114 DI<13C0A41201A212031207120F121FB5FCA2EA0FC0ABEBC180A512 07EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF14FEA2EA0F833907FF7FC0EA01 FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015B A26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E>I<3AFFE3FF87F8A23A1F807C 00C0D80FC0EB0180147E13E0000790387F030014DF01F05B00031486EBF18FD801F913CC13FB90 38FF07DC6C14F8EBFE03017E5BA2EB7C01013C5BEB380001185B25167F9528>I<39FFF01FE0A2 390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA2 6DC7FCA2130EA2130CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B207F951E>121 D E /Fh 15 122 df97 DI<137E48B4FC38038380EA0F07121E001C1300EA3C0248C7FCA35AA5EA70 021307EA381EEA1FF8EA07E011147C9315>I<1478EB03F814F0EB0070A314E0A4EB01C0A213F1 EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133CEA707C3878FCC0EA3F CF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB0300EA780EEA7FFC13E0 00F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I103 DI<136013F0A213E01300A7120FEA1F80123113C0EA6380 A212C3EA0700A3120EA3EA1C301360A2EA38C01218EA1F80EA0F000C1F7D9E0E>I108 D<381E07C0383F1FE03833B870EA63E0EBC038138000C71370EA0700A3000E13E0A3EB01C3001C 13C6A2EB038C1301003813F8381800F018147D931A>110 D<137C48B4FC38038380380F01C012 1E001C13E0123C1278A338F003C0A3EB07801400EA700F131EEA3838EA1FF0EA07C013147C9317 >I114 D116 D<38078780380FCFC03818F8E0EA3070EA6071A238C0 E1C03800E000A3485AA30071136038F380C0A238E3818038C7C300EA7CFEEA387C13147D9315> 120 D<000F1360381F8070003113E013C0EA6380A238C381C0EA0701A3380E0380A4EB0700A25B 5BEA07FEEA03EEEA000EA25B12785BEA7070EA60E0EA3FC06CC7FC141D7D9316>I E /Fi 2 16 df0 D15 D E /Fj 1 111 df<380F07C0381F8FE03831D8703861 F03813E013C000C35BEA0380A348485AA3903801C180000EEBC300EB03831486EB018E001C13FC 380C00F019147F931B>110 D E /Fk 52 123 df<90390FF03FFC90387FFDFF9038F83FE0D801 E013810003137FD807C01300806E137CA5B712FCA23A07C01F007CB03B3FF8FFE3FF80A2292080 9F2C>15 D<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123CA25AA412F85AA97E12 78A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA114>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA01E0A2 13C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<1238127C12FE12FFA2127F123B12 03A21206A2120E120C12181270122008107C860F>44 D<146014E0130114C0A213031480A21307 14005B130EA2131E131C133C1338A213781370A213F05B12015BA212035BA2120790C7FC5A120E A2121E121C123C1238A212781270A212F05AA2132D7DA11A>47 D<137013F0120F12FF12F31203 B3A4B51280A2111D7C9C1A>49 DI<14E0A2497EA3497EA2497EA2497E130CA2EB187FA201307F143F01707F EB601FA201C07F140F48B57EA2EB800748486C7EA20006801401000E803AFFE01FFFE0A2231F7E 9E28>65 DI<9038 07FC0290383FFF0E9038FE03DE3903F000FE4848133E4848131E485A48C7120EA2481406127EA2 00FE1400A7127E1506127F7E150C6C7E6C6C13186C6C13386C6C13703900FE01C090383FFF8090 3807FC001F1F7D9E26>I69 DI<903807FC0290383FFF0E9038FE03DE3903F000FE4848133E48 48131E485A48C7120EA2481406127EA200FE91C7FCA591387FFFE0A2007E903800FE00A2127F7E A26C7E6C7E6C7E3803F0013900FE03BE90383FFF1E903807FC06231F7D9E29>I73 D75 DIIIII82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400 B4FC13F86CB4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200 F813F038FE01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060 140CA200E0140E00C01406A400001400B10007B512C0A21F1E7E9D24>II87 DII<003FB51280A2EB807F393E 00FF00383801FEA248485A5CEA6007495AA2C6485A495AA2495A91C7FC5B485AEC0180EA03FCEA 07F8A2380FF00313E0001F140048485A5C48485A38FF007F90B5FCA2191F7D9E20>I97 DIII II<3801FC3C3807FFFE380F07DEEA1E03003E13E0A5001E13C0380F0780EBFF00EA19FC 0018C7FCA2121C381FFF8014F06C13F8003F13FC387C007C0070133E00F0131EA30078133CA238 3F01F8380FFFE000011300171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121FB0EAFFE0A20B217E A00E>I107 DI<3AFE0FE03F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB00 7CAC3AFFE3FF8FFEA227147D932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3 FFA218147D931D>I<48B4FC000713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA2 6C13F8A2381F83F03807FFC00001130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB00 7EA2143E143FA6143E147E147CEB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D> I114 DII<38FF07F8A2EA1F00AD1301A2EA0F07 3807FEFFEA03F818147D931D>I<39FFE07F80A2391F001C00380F8018A26C6C5AA26C6C5AA26C 6C5AA213F900005B13FF6DC7FCA2133EA2131CA219147F931C>I<3AFFE7FE1FE0A23A1F00F007 006E7ED80F801306A23907C1BC0CA214BE3903E31E18A23901F60F30A215B03900FC07E0A29038 7803C0A3903830018023147F9326>I<38FFE1FFA2380F80706C6C5A6D5A3803E180EA01F36CB4 C7FC137E133E133F497E136FEBC7C0380183E0380381F0380701F8380E00FC39FF81FF80A21914 7F931C>I<39FFE07F80A2391F001C00380F8018A26C6C5AA26C6C5AA26C6C5AA213F900005B13 FF6DC7FCA2133EA2131CA21318A2EA783012FC5BEAC0E0EAE1C0EA7F80001EC8FC191D7F931C> I<383FFFE0A2383C0FC0EA381F0070138038603F00137E13FEC65A485A485A3807E060120F13C0 381F80E0383F00C0EA7F01EA7E03B5FCA213147F9317>I E /Fl 31 124 df<121C127FEAFF80A213C0A3127F121C1200A212011380A21203EA07001206120E5A5A12300A 157BA913>39 D<121C127FEAFF80A5EA7F00121C09097B8813>46 D<13075B137FEA07FFB5FCA2 12F8C6FCB3AB007F13FEA317277BA622>49 DII<1407 5C5C5C5C5CA25B5B497E130F130E131C1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A 5AB612F8A3C71300A7017F13F8A31D277EA622>I65 DI<91393FF00180903903FFFE07010FEBFF8F90393FF007FF9038FF8001 4848C7127FD807FC143F49141F4848140F485A003F15075B007F1503A3484891C7FCAB6C7EEE03 80A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007F0010FB512C0 010391C7FC9038003FF829297CA832>I69 D79 DI82 D<48B47E000F13F0381F81FC486C7E147FA2EC 3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0 393F83DFFC380FFF0F3801FC031E1B7E9A21>97 DIII< EB3FE03801FFF83803F07E380FE03F391FC01F80393F800FC0A2EA7F00EC07E05AA390B5FCA290 C8FCA47E7F003F14E01401D81FC013C0380FE0033903F81F803900FFFE00EB1FF01B1B7E9A20> I<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14> 105 D 107 DI<3BFFC07F800FF0903AC1FFE03FFC903AC7 83F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3 FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A3 13E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F 140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8 FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FC A3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013 FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038 F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD 1438A73803F870000113F03800FFE0EB1F8015267FA51B>I<3AFFFE03FF80A33A07F0007000A2 6D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A3 6D5AA26D5AA26D5A211B7F9A24>118 D<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7 FCEBFF1E6D5AEB3FF86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F 00036D7E3AFFF01FFF80A3211B7F9A24>120 D123 D E /Fm 67 124 df<90380FC3E090387FEFF09038E07C783801C0F8D8038013303907007000A7 B61280A23907007000B0387FE3FFA21D20809F1B>11 DI<90380F80F890387FE7FE9038E06E 063901C0FC0F380380F8380700F00270C7FCA6B7FCA23907007007B03A7FE3FE3FF0A22420809F 26>14 D<90380FC0FFEB7FE79038E07E0F3801C0FC4848487E38070070A7B7FCA23907007007B0 3A7FE3FE3FF0A22420809F26>I34 D<127012F812FCA2127C120CA31218A21238123012601240 060E7C9F0D>39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A212E0 AC1260A21270A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I<12C0 12607E7E121C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A21300 5A12065A121C12185A5A5A0B2E7DA112>I<127012F812FCA2127C120CA31218A2123812301260 1240060E7C840D>44 DI<127012F8A3127005057C840D>I48 DIII<13 0EA2131E133EA2136E13EE13CEEA018E1203130E1206120E120C121812381230126012E0B512F0 A238000E00A7EBFFE0A2141E7F9D17>II<137CEA01FEEA0783380E0380EA0C07121C3838030090C7FC127812 70A2EAF3F8EAF7FEEAFC0E487EEB0380A200F013C0A51270A214801238EB0700121CEA0E1EEA07 FCEA01F0121F7E9D17>I<1260387FFFC0A21480EA600138C003001306A2C65A5BA25B5BA213E0 5B1201A3485AA41207A76CC7FC121F7D9D17>III<127012F8A312701200AA127012F8A312700514 7C930D>I<127012F8A312701200AA127012F8A312781218A41230A21260A21240051D7C930D>I< EB0380A3497EA3EB0DE0A3EB18F0A3EB3078A3497EA3EBE01E13C0EBFFFE487FEB800FA2000314 80EB0007A24814C01403EA0F8039FFE03FFEA21F207F9F22>65 DI<90381FC04090387FF0C03801F8393803C00D38078007380F0003121E003E1301123C 127C1400127812F81500A8007814C0127CA2123C003EEB0180121E6CEB0300EA07803803C00E38 01F81C38007FF0EB1FC01A217D9F21>I69 DI73 D<39FFFC1FFCA239078007C0150014065C5C5C14705CEB81C0EB83800187C7FC80138FEB9BC0EB B1E013E1EBC0F01380147880A280A280EC0780A215C039FFFC3FFCA21E1F7E9E23>75 D77 D<39FF807FF813C00007EB07809038E00300A2EA06F0A21378133CA2131EA2130FA2EB078314C3 1303EB01E3A2EB00F3A2147BA2143F80A280A2000F7FEAFFF0801D1F7E9E22>III82 D<3807E080EA0FF9EA1C1FEA300FEA7007EA600312E01301A36CC7FCA21278127FEA 3FF0EA1FFC6C7EEA03FF38001F801307EB03C0A2130112C0A400E01380EAF00338F80700EAFE0E EACFFCEA81F812217D9F19>I<007FB512E0A238780F010070130000601460A200E0147000C014 30A400001400B23807FFFEA21C1F7E9E21>I<39FFFC7FF8A23907800780EC0300B3A300031302 EBC006A200015B6C6C5AEB7830EB3FE0EB0FC01D207E9E22>I<3BFFF07FF83FF0A23B0F000780 0F80EE0300A23A07800FC006A3913819E00ED803C0140CA214393A01E030F018A33A00F0607830 A3ECE07C903978C03C60A390393D801EC0A390383F000F6D5CA3010E6DC7FCA32C207F9E2F>87 D92 D97 D<120E12FEA2120EA9133FEBFF80380FC3C0EB00E000 0E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E0015207F9F19>IIII<13 3C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0A2120EAD38FFE7FCA216 207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFFC0A20A1F809E0C>I<13E0 EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EAF1C0EA7F80EA3E000C2882 9E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B137013F0EA0FF81338EA0E1C13 1E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA2120EB3A9EAFFE0A20B20809F 0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E13E0AD3AFFE7FE7FE0A223 147F9326>IIII<3803E180EA 0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA1E1FEA0FF3EA03E3EA0003A7 EB3FF8A2151D7E9318>I II<1206A4120EA2121E123EEAFFF8A2EA0E00AA1318A5EA07 3013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC1303A2EA070FEBFDFCEA01F11614 7F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA213FC6C 5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560ECC0E0D80E0F13C0130C14E00007 EBE180EB186114713903987300EBB033A2143F3801F03EEBE01EA20000131CEBC00C1E147F9321 >I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13781338133C137C13EEEA01C713 8738030380380701C0000F13E038FF87FEA21714809318>I<38FF87F8A2381E01E0000E13C014 80A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A31360A35B12F0EAF18012F3007FC7 FC123C151D7F9318>III E /Fn 14 124 df67 D73 D80 D<903807FFFE017FEBFFE048B612F84815FE489039001FFF8003077F48D980017F6F7FA2 707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC130F013FEBF03F3901FFFE004813F048 13C04890C7FC485A485A485AA212FF5BA3167FA26D14FF6C6CEB01EF003F140301FF90380FCFFF 6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEBF0010107903980007FF8362E7DAD3A> 97 D101 D108 D<903A7FC001FFC0B5010F13F8 033F13FE92B6FC9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC 5CA25CA25CB3A8B6D8C07FEBFFE0A53B2E7CAD42>110 DI<90397FC01FFEB590B512E002C314F802CF14FE9139DFF01FFF9126FF80077F00 0349486C7F6C496D7F02F06D7F717E4A81173F84171F84A3711380AB19005FA34D5AA260177F6E 5D6E4A5A6E495B6E495B9126FF800F5BDBE03F90C7FC02EFB512FC02E314F002E014C0DB1FFCC8 FC92CAFCAFB612C0A539427CAD42>I<9039FF803FC0B5EBFFF0028313FC02877F91388FE7FFEC 9F070003D99E0F13806C13BCA214F8A214F06F13004A6C5A6F5A92C8FCA25CB3A6B612E0A5292E 7CAD31>114 D<90390FFF81E090B512F7000314FF5A380FFC01391FE0003FD83F80130F007F14 0790C7FC15035AA27F6D90C7FC13F013FF14F86CEBFFC015F86C14FE6C806C15806C15C06C15E0 C615F0013F14F81307EB001F020013FC153F0078140F00F81407A26C1403A27E16F86C14076D14 F06D130F01F0EB1FE001FEEBFFC090B6128048ECFE00D8F83F13F8D8F00313C0262E7CAD2F>I< EB01F0A61303A31307A3130FA2131F133FA2137FEA01FF5A000F90B512C0B7FCA4C601F0C7FCB3 A5ED01F0A91503D97FF813E01507D93FFC13C090391FFE1F806DB5FC6D1400010113FC9038003F F024427EC12E>I<007FB5D8801FB5FCA528007FF8000390C7FC6EEB01FC6D6C495A6D6C495A6D 5D6D6D485A6D6D485AEDE03F6D6D48C8FC6DEBF8FE91387FF9FC6EB45A5E6E5B6E5B80806E7F82 824A7F825C91380FEFFFDA1FCF7FDA3F877FDA7F037FECFE0149486C7F4A6D7E49488001076E7E 49486D7E49487F49486D7FD9FFC081B500F8013FEBFFC0A53A2E7EAD3F>120 D123 D E /Fo 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007 B3B3A7007FB612E0A4233879B732>49 D 67 D97 D<903801FFC0011F13F8017F13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED 0FF0485A003FEC07F8A2485AED03FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C 147C000F15786C6C14F86C6CEB01F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E0 26267DA52D>101 D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C0102 7814804A7E4A14C05CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE 3FF891C7121F17FC160F17FEA3EE07FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E 481380DAFC07130091383FFFFC020F13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF8091387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F 80ED0E0091C8FCB3A3B512FEA423267DA529>114 D116 D E /Fp 36 119 df38 D<127012F8A212F012E005057A840F>46 D<14035CA25C1580141FA2143714771467 14C7A2EB0187A2EB0307A21306130E130C1318A2133090383FFFC05BEB600313C012011380EA03 00A21206A2121E39FFC03FFC13801E237DA224>65 D<90B512E015F890380F003C151E131E150E 150FA249130E151EA2153C49137815F0EC01E090387FFFC090B5FC9038F003E0EC00F01578485A 1538153CA248481378A315F039078001E01403EC07C0EC1F00B512FE14F020227DA122>I<90B5 12F015FC90380F003E150F011EEB0780150316C015015B16E0A35BA449EB03C0A44848EB0780A2 16005D4848130E5D153C5D48485B4A5AEC0780021FC7FCB512FC14F023227DA125>68 D<91387E0180903903FF810090380F80C390383E00670178133F49133ED801C0131E485A120748 C7121C120E121E5A15185A92C7FCA25AA4EC3FFC5AEC01E0A26C495AA312700078495A1238003C 130F6C131B260FC0F3C7FC3803FFC1C690C8FC212479A226>71 D73 D<903807FFC04913809038003C00 A25CA45CA4495AA4495AA4495AA449C7FCA212381278EAF81EA2485AEA6078EA70F0EA3FE0EA1F 801A237CA11A>I<9039FFF80FFCA290390F0007C01600011E130E5D5D1560495B4A5A4AC7FC14 0E495A5C147814FCEBF1BCEBF33CEBFE1E13FC3801F01F497EA2813803C007A26E7EA2EA07806E 7EA28139FFF80FFEA226227DA125>III<01FFEB1FFC1480010FEB03C01680D91BC01300A3EB19E001 311306A2EB30F0A201605B1478A3496C5AA3141ED801805BA2140FA2D803005BEC07E0A300065C 1403A2121F39FFE00180A226227DA124>I<14FE903807FF8090380F03E090383C00F001701378 4913384848133C4848131C48C7FC48141E121EA25AA25AA348143CA31578A34814F0A26CEB01E0 15C01403EC07800078EB0F00141E6C5B6C13F8380F83E03807FF80D801FCC7FC1F2479A225>I< 90B512C015F090380F0078153C011E131E150EA349131EA3153C491338157815F0EC03C090B512 005CEBF00FEC07803901E003C0A43903C00780A43907800F001503A2EC0706D8FFF8138EEC03FC C7EA01F020237DA124>82 D<903801F06090380FFC4090381E0EC0EB3807EB700301E0138013C0 1201A2D803801300A26DC7FCA27FEA01F813FF6C13E06D7EEB1FF8EB03FCEB007C143C80A30030 131CA31418007013385C00781360387C01C038EF0380D8C7FFC7FCEA81FC1B247DA21B>I<001F B512F8A2391E03C07800381438EB0780123000601430A2EB0F0012C0A3D8001E1300A45BA45BA4 5BA4485AA31203B57E91C7FC1D2277A123>I<393FFE07FFA23903C000F015E0484813C0A4390F 000180A4001EEB0300A4481306A4485BA4485BA25C12705C5C6C485A49C7FCEA1E0EEA0FFCEA03 F0202377A124>I<3BFFF03FF81FF8D9E07FEB3FF03B1F0007800780001E010FEB03001606141F 5E14375E14675E14C75E381F0187000F5DEB0307ED818013060383C7FC130C158613180138138C 0130139C0160139815B801C013B015E013805D13005D120E92C8FC120C2D2376A131>87 D<39FFF003FFA2000FC712F015E090388001C000071480EC0300EBC0060003130E5CEBE0180001 5B5CEBF0E000005BEBF18001FBC7FC13FF137E137C1378A45BA4485AA4EA3FFE5B202276A124> 89 D97 D<137E48B4FC3803C380EA0703 EA0E07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03 383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2 19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA 701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>II<136013F0 13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA 1F80EA0E000C217CA00F>105 D<13F0EA0FE0A21200A2485AA4485AA448C7FCEB01E0EB07F0EB 0E30380E1870EB30F01360EBC060381D8000121F13E0EA1CF0EA3838133CEB1C20143038703860 A21440EB18C038E01F8038600F0014237DA216>107 DI<381E0780383F1F E0EA63B8EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140C EB0318003813F0381801E018157C941B>110 D<137E48B4FC3803C380380701C0120E001C13E0 123CA21278A338F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803 C1F03807E3F8380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EB E3C038077F80EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863 B0E013E0EAC3C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114 D<13FC48B4FC38038380EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA70 07EAF00EA2485AEA7038EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2 EA0700120EA45AA45AA213101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F13 30381F8070EA31C0006113E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2 130FEB1F183807F3F03803E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC3 80A2000313C0EA0700A3380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I E /Fq 53 122 df35 D<127012F812FCA2127C12 0CA41218A21230A212601240060F7C840E>44 DI<127012F8A3127005 057C840E>I48 DI51 D<00101380EA1C07381FFF 005B5B13F00018C7FCA613F8EA1BFEEA1F0F381C0780EA180314C0EA000114E0A4126012F0A214 C0EAC0031260148038300700EA1C1EEA0FFCEA03F013227EA018>53 D<137E48B4FC3803C18038 0701C0EA0E03121CEB018048C7FCA2127812701320EAF1FCEAF3FEEAF60738FC038000F813C013 0112F014E0A51270A3003813C0130300181380381C0700EA0E0EEA07FCEA01F013227EA018>I< EA01F0EA07FCEA0E0F38180780EA3803383001C01270A31278EB0380123E383F0700EA1FCEEA0F FCEA03F87FEA0F7F381C3F80EA380F387007C0130338E001E01300A5387001C0A238380380381E 0F00EA0FFEEA03F013227EA018>56 DI<497E497EA3497EA3497E130CA2EB1CF8EB1878 A2EB383C1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F000 1F130139FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038F81C703801E0063903C003 F03807800148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C14 60123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>II 70 D<903807F00890383FFC189038FC0E383801E0033903C001F83807800048C71278121E1538 5AA2007C14181278A212F81500A6EC1FFF1278007CEB0078A2123CA27EA27E6C7E6C6C13F83801 F0013900FC079890383FFE08903807F80020247DA226>I<39FFFC3FFFA239078001E0AD90B5FC A2EB8001AF39FFFC3FFFA220227EA125>I<3803FFF0A238000F00B3A6127012F8A3EAF01EEA60 1CEA3878EA1FF0EA07C014237EA119>74 D<39FFFC07FFA239078001F015C05D4AC7FC14065C5C 14385C5CEB81C0EB8380EB87C080138DEB98F013B0EBE078497E13808080A26E7E8114036E7EA2 6E7E4A7E3AFFFC07FF80A221227EA126>III<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7FA2 EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000EA FFF0156020227EA125>III82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3F F0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF801 38EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200E0 141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<3BFFF03FFC07FEA23B0F0007C001 F00203EB00E01760D807806D13C0A33B03C007F001801406A216032701E00C781300A33A00F018 3C06A3903978383E0CEC301EA2161C90393C600F18A390391EC007B0A3010F14E0EC8003A36D48 6C5AA32F237FA132>87 D<387FFFFEA2EB003C007C137C0070137814F84813F0EB01E0EAC00314 C01307148038000F005B131E133E133C5B13F85B0001130313E0EA03C012071380000F13071300 001E1306003E130E003C131E007C133E007813FEB5FCA218227DA11E>90 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0 F0EB0078000E1338143C141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237F A21B>II<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA 38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>II<133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700 B1EA7FF8A2102380A20F>I<14F03801F1F83807FFB8380F1F38381E0F00EA1C07003C1380A500 1C1300EA1E0FEA0F1EEA1FFCEA19F00018C7FCA2121CEA1FFF6C13C04813E0383801F038700070 481338A400701370007813F0381E03C0380FFF803801FC0015217F9518>I<120E12FEA2121E12 0EAAEB1F80EB7FC0380FC1E0EB80F0EB0070120EAE38FFE7FFA218237FA21B>I<121C121E123E 121E121CC7FCA8120E12FEA2121E120EAFEAFFC0A20A227FA10E>II<120E12FEA212 1E120EAAEB0FFCA2EB07E0EB0380EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03 C0130114E014F038FFE3FEA217237FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E> I<390E1FC07F3AFE7FE1FF809039C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8F FEA227157F942A>I<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA21815 7F941B>II<380E1F8038FE7FE0 38FFC1F0380F0078120E143CA2141EA7143CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FC A8EAFFE0A2171F7F941B>I114 DI<1206A5120EA3121E123EEAFFF8A2 EA0E00AA130CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E 1370AC14F01301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2 391E01C07CD81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D8 1980141DA2EBF00F00001400A2497EEB600620157F9423>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A21330A213701360A2EAF0C012F1 EAF380007FC7FC123E171F7F941A>121 D E /Fr 20 118 df45 D68 D73 D77 D80 D<90387F80203901FFE0603807C0F8390F001CE0001E130F481307003813030078130112701400 12F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013C0EC 1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039E780 070038C1F01E38C07FFC38800FF01D337CB125>83 D97 D99 DIII<15F090387F03F83901FFCF1C3803C1FC390780F818 390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0 D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB 00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F 7E9F21>I<1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8 A20D307EAF12>105 D<260781FEEB3FC03BFF87FF80FFF0903A8E07C1C0F83B0F9803E3007C27 07B001E6133C9026E000FC7F495BA3495BB3486C486C133F3CFFFC1FFF83FFF0A2341F7E9E38> 109 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E 3AFFFC1FFF80A2211F7E9E25>II<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE0 38EBC000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813 705A1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2 141C7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A312 03A21207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB 19>II E /Fs 5 85 df<1630167016F0A21501A21503A2150715 0FA2151B821531A2156115E115C1EC0181A2EC0301A21406A2140C141C14181430A202607FA2EC C000A249B5FC5B91C7FC1306A25BA25BA25B1370136013E01201000381D80FF01301D8FFFE9038 3FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E8314 7E188002FE140FA24A15C0A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA201 1F157E17FE4A5CA2013F4A5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495A ED07E00001EC3F80B600FEC8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7 121EA24A140CA2147EA214FEA25CA20101151CEEC0184A1400A201031301A202F05B1503010713 0F91B5FC93C7FCECE00F010F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA213 7EA213FEA25BA21201B512FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092 380003F0EE01F84AEB00FCA2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804A EB3F00167E0107EB03F891B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA201 3F1301A21400A249495AA2137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0 C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C13 0F121800385CA20030011F1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FE A25CA21301A25CA21303A25CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D31 74B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 795 696 a Fs(D)26 b(R)g(A)f(F)h(T)225 787 y Fr(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)685 981 y Fq(Scott)c(Berryman,)e Fp(Y)l(ale)19 b(Univ)700 1040 y Fq(James)c(Co)o(wnie,)h Fp(Meiko)i(Ltd)474 1098 y Fq(Jac)o(k)e(Dongarra,)i Fp(Univ.)23 b(of)17 b(T)l(ennesse)n(e)j(and)d(ORNL)801 1156 y Fq(Al)f(Geist,)f Fp(ORNL)795 1214 y Fq(Bill)f(Gropp,)j Fp(ANL)767 1272 y Fq(Rolf)f(Hemp)q(el,)e Fp(GMD)762 1330 y Fq(Bob)i(Knigh)o(ten,)g Fp(Intel)786 1388 y Fq(Rust)o(y)g(Lusk,)h Fp(ANL)614 1446 y Fq(Stev)o(e)e(Otto,)h Fp(Or)n(e)n(gon)h(Gr)n(aduate)g(Inst)578 1504 y Fq(T)l(on)o(y)g(Skjellum,)c Fp(Missisippi)j(State)j(Univ)653 1563 y Fq(Marc)d(Snir,)g Fp(IBM)h(T.)g(J.)g(Watson)744 1621 y Fq(Da)o(vid)f(W)l(alk)o(er,)f Fp(ORNL)626 1679 y Fq(Stev)o(e)g(Zenith,)g Fp(Kuck)k(&)f(Asso)n(ciates)844 1803 y Fq(Ma)o(y)e(9,)g(1993)87 1861 y(This)g(w)o(ork)g(w)o(as)h(supp)q(orted)g(b)o(y)f(ARP)l(A)g(and)g(NSF)g (under)g(con)o(tract)g(n)o(um)o(b)q(er)f(###,)g(b)o(y)g(the)192 1919 y(National)h(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o(hnology) f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)650 1977 y(Agreemen)o(t)f(No.)21 b(CCR-8809615.)p eop %%Page: 1 2 bop 75 356 a Fo(Chapter)32 b(1)75 564 y Fn(Con)m(texts)40 b({)g(Prop)s(osal)h (I)876 786 y Fm(Marc)14 b(Snir)75 927 y Fl(1.1)70 b(Con)n(texts)75 1029 y Fm(A)17 b Fk(comm)o(unication)k(con)o(text)c Fm(\(for)f(short,)g Fk(con)o(text)p Fm(\))h(is)g(a)g(mec)o(hanism)g(for)g(the)g(mo)q (dularization)75 1085 y(of)h(MPI)g(comm)o(unication.)29 b(An)o(y)18 b(MPI)g(comm)o(unication)h(o)q(ccurs)f(within)h(a)f(con)o(text,)g(and)g(do)q (es)g(not)75 1142 y(in)o(terfere)j(with)g(comm)o(unication)h(executed)g (within)g(another)e(con)o(text.)37 b(F)l(urthermore,)21 b(a)g(con)o(text)75 1198 y(sp)q(eci\014es)c(a)e(lo)q(cal)h(name)f(space)h(for)e(pro)q(cesses)i (that)e(comm)o(unicate)h(in)h(this)g(con)o(text.)j(The)d(pro)q(cesses)75 1255 y(that)i(participate)i(in)g(a)f(con)o(text)f(are)h(asso)q(ciated)g(with) g(a)g Fk(rank)p Fm(,)g(whic)o(h)h(ranges)e(from)h(0)f(to)h Fj(n)13 b Fi(\000)g Fm(1,)75 1311 y(where)18 b Fj(n)h Fm(is)f(the)h(n)o(um)o (b)q(er)f(of)g(pro)q(cesses)g(that)g(participate)h(in)g(the)f(con)o(text.)28 b(This)19 b(rank)f(is)g(used)h(in)75 1368 y(in)o(terpro)q(cess)d(comm)o (unication)h(as)e(the)h(lo)q(cal)h(address)f(of)f(the)h(pro)q(cess)g(within)h (that)e(con)o(text.)21 b(Th)o(us,)75 1424 y(comm)o(unication)16 b(within)g(a)f(con)o(text)g(is)h(una\013ected)f(b)o(y)g(comm)o(unication)h (outside)g(that)e(con)o(text.)166 1480 y(A)22 b(pro)q(cess)g(ma)o(y)g(comm)o (unicate)g(sim)o(ultaneously)h(in)g(sev)o(eral)f(con)o(texts.)40 b(The)22 b(con)o(text)g(of)f(a)75 1537 y(comm)o(unication)16 b(is)g(explicitly)i(stated)c(as)h(a)g(parameter)f(of)h(the)g(comm)o (unication)h(call.)166 1593 y(A)h(pro)q(cess)g(that)f(participates)i(in)g(a)e (comm)o(unication)i(con)o(text)f(accesses)g(this)g(con)o(text)g(using)g(a)75 1650 y Fh(c)n(ontext)h(hand)r(le)g Fm(\(i.e.,)f(a)h(handle)h(to)e(an)g (opaque)h(ob)s(ject)f(that)g(iden)o(ti\014es)j(a)d(con)o(text\).)26 b(This)19 b(handle)75 1706 y(can)c(b)q(e)h(used)g(to)143 1788 y Fi(\017)23 b Fm(Find)14 b(information)f(ab)q(out)g(this)g(con)o(text,)g (suc)o(h)g(as)g(the)g(n)o(um)o(b)q(er)h(of)e(pro)q(cesses)i(that)e (participate)189 1844 y(in)k(the)f(con)o(text,)f(or)h(the)g(rank)g(of)g(the)g (calling)i(pro)q(cess)f(within)g(the)f(con)o(text.)143 1933 y Fi(\017)23 b Fm(Comm)o(unicate)12 b(with)h(other)f(pro)q(cesses)h(that)e (participate)j(in)f(the)f(con)o(text;)h(these)f(pro)q(cesses)h(are)189 1989 y(addressed)i(using)h(their)g(con)o(text)f(rank.)143 2078 y Fi(\017)23 b Fm(Create)14 b(new)i(con)o(texts.)166 2160 y(Con)o(text)i (handles)h(cannot)g(b)q(e)g(transferred)f(for)g(one)h(pro)q(cess)f(to)g (another;)i(they)e(can)h(b)q(e)g(used)75 2216 y(only)12 b(on)g(the)f(pro)q (cess)h(where)g(they)g(w)o(ere)f(created.)19 b(Th)o(us,)11 b(\\kno)o(wledge")h(ab)q(out)f(a)h(con)o(text)f(exists)g(only)75 2273 y(lo)q(cally)l(,)17 b(at)e(the)h(pro)q(cesses)g(that)f(participate)h(in) h(that)d(con)o(text.)21 b(Op)q(erations)16 b(within)h(a)e(comm)o(unica-)75 2329 y(tion)h(con)o(texts)e(\(including)k(the)d(generation)h(of)f(new)g(sub)q (con)o(texts\))g(do)g(not)g(require)h(comm)o(unication)75 2385 y(with)g(pro)q(cesses)f(that)g(do)g(not)g(participate)g(in)i(that)d(con)o (text.)166 2442 y(F)l(ollo)o(ws)h(examples)h(of)f(p)q(ossible)i(uses)e(for)g (con)o(texts.)75 2561 y Fg(1.1.1)55 b(Lo)r(osely)18 b(sync)n(hronous)h (library)h(call)g(in)n(terface)75 2647 y Fm(Consider)13 b(the)g(case)g(where) g(a)f(parallel)i(application)h(executes)e(a)f(\\parallel)i(call")g(to)e(a)g (library)i(routine,)75 2704 y(i.e.,)g(where)h(all)g(pro)q(cesses)g(transfer)f (con)o(trol)g(to)g(the)h(library)g(routine.)20 b(If)15 b(the)f(library)i(w)o (as)d(dev)o(elop)q(ed)964 2828 y(1)p eop %%Page: 2 3 bop 75 -100 a Fm(2)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)75 45 y Fm(separately)l(,)23 b(then)e(one)h(should)g(b) q(ew)o(are)g(of)e(the)i(p)q(ossibilit)o(y)h(that)e(the)g(library)h(co)q(de)g (ma)o(y)f(receiv)o(e)75 102 y(b)o(y)f(mistak)o(e)g(messages)f(send)i(b)o(y)f (the)g(caller)h(co)q(de,)g(and)f(vice-v)o(ersa.)35 b(The)20 b(problem)h(is)g(solv)o(ed)f(b)o(y)75 158 y(allo)q(cating)c(a)f(di\013eren)o (t)h(con)o(text)e(to)h(the)g(library)l(,)h(th)o(us)f(prev)o(en)o(ting)h(un)o (w)o(an)o(ted)e(in)o(terference.)75 277 y Fg(1.1.2)55 b(F)-5 b(unctional)21 b(decomp)r(osition)e(and)g(mo)r(dular)g(co)r(de)f(dev)n (elopmen)n(t)75 363 y Fm(Often,)i(a)f(parallel)i(application)g(is)e(dev)o (elop)q(ed)i(b)o(y)e(in)o(tegrating)h(sev)o(eral)f(distinct)h(functional)h (mo)q(d-)75 419 y(ules,)c(that)f(is)h(eac)o(h)g(dev)o(elop)q(ed)h(separately) l(.)24 b(Eac)o(h)16 b(mo)q(dule)i(is)f(a)f(parallel)i(program)d(that)h(runs)h (on)f(a)75 476 y(dedicated)g(set)e(of)g(pro)q(cesses,)g(and)g(the)h (computation)f(consists)h(of)e(phases)i(where)f(mo)q(dules)i(compute)75 532 y(separately)l(,)d(in)o(termixed)h(with)f(global)h(phases)f(where)g(all)h (pro)q(cesses)f(comm)o(unicate.)19 b(It)13 b(is)g(con)o(v)o(enien)o(t)75 589 y(to)g(allo)o(w)h(eac)o(h)g(mo)q(dule)g(to)f(use)h(its)g(o)o(wn)f(priv)m (ate)i(pro)q(cess)f(n)o(um)o(b)q(ering)g(sc)o(heme,)g(for)f(the)h(in)o(tramo) q(dule)75 645 y(computation.)24 b(This)17 b(is)g(ac)o(hiev)o(ed)g(b)o(y)g (using)g(a)f(priv)m(ate)i(mo)q(dule)f(con)o(text)f(for)g(in)o(tramo)q(dule)i (compu-)75 701 y(tation,)c(and)i(a)f(global)h(con)o(text)e(for)h(in)o(termo)q (dule)h(comm)o(unication.)75 820 y Fg(1.1.3)55 b(Collectiv)n(e)20 b(comm)n(unication)75 906 y Fm(MPI)g(supp)q(orts)h(collectiv)o(e)h(comm)o (unication)f(within)h(dynamically)g(created)f(groups)f(of)g(pro)q(cesses.)75 963 y(Eac)o(h)15 b(suc)o(h)h(group)f(can)g(b)q(e)h(represen)o(ted)g(b)o(y)f (a)g(distinct)h(comm)o(unication)g(con)o(text.)k(This)c(pro)o(vides)f(a)75 1019 y(simple)g(mec)o(hanism)f(to)f(ensure)h(that)f(comm)o(unication)h(that)f (p)q(ertains)h(to)f(collectiv)o(e)i(comm)o(unication)75 1076 y(within)h(one)f(group)f(is)h(not)f(confused)i(with)f(collectiv)o(e)h(comm)o (unication)g(within)f(another)g(group,)f(and)75 1132 y(a)o(v)o(oids)h(the)g (in)o(tro)q(duction)h(of)f(t)o(w)o(o)f(di\013eren)o(t)h(mec)o(hanisms)h(with) g(similar)g(functionalit)o(y)l(.)75 1251 y Fg(1.1.4)55 b(Ligh)n(t)n(w)n(eigh) n(t)21 b(gang)f(sc)n(heduling)75 1337 y Fm(Consider)14 b(an)f(en)o(vironmen)o (t)g(where)g(pro)q(cesses)h(are)e(m)o(ultith)o(treaded.)20 b(Con)o(texts)12 b(can)h(b)q(e)h(used)g(to)e(pro-)75 1393 y(vide)h(a)e(mec)o (hanism)i(whereb)o(y)f(all)h(pro)q(cesses)f(are)f(time-shared)i(b)q(et)o(w)o (een)f(sev)o(eral)g(parallel)h(executions,)75 1450 y(and)19 b(can)h(con)o(text)e(switc)o(h)i(from)e(one)h(parallel)i(execution)f(to)f (another,)g(in)h(a)f(lo)q(osely)h(sync)o(hronous)75 1506 y(manner.)27 b(A)17 b(thread)g(is)h(allo)q(cated)h(on)e(eac)o(h)g(pro)q(cess)h(to)f(eac)o (h)g(parallel)i(execution,)g(and)f(a)f(di\013eren)o(t)75 1562 y(con)o(text)d(is)i(used)f(to)f(iden)o(tify)i(eac)o(h)f(parallel)i (execution.)k(Th)o(us,)14 b(tra\016c)g(from)g(one)h(execution)h(cannot)75 1619 y(b)q(e)i(confused)g(with)g(tra\016c)f(from)f(another)h(execution.)28 b(The)18 b(blo)q(c)o(king)g(and)g(un)o(blo)q(c)o(king)h(of)e(threads)75 1675 y(due)g(to)f(comm)o(unication)h(ev)o(en)o(ts)f(pro)o(vide)h(a)f(\\lazy") g(con)o(text)g(switc)o(hing)h(mec)o(hanism.)24 b(This)17 b(can)f(b)q(e)75 1732 y(extended)j(to)f(the)h(case)f(where)h(the)f(parallel)i(executions)f (are)g(spanning)g(distinct)h(pro)q(cess)e(subsets.)75 1788 y(\(MPI)d(do)q(es)g(not)g(require)h(m)o(ultithreaded)h(pro)q(cesses.\))75 1929 y Fl(1.2)70 b(Basic)22 b(Con)n(text)g(Op)r(erations)75 2030 y Fm(A)17 b(global)g(con)o(text)g Fk(MPI)p 534 2030 16 2 v 18 w(ALL)g Fm(is)g(prede\014ned.)26 b(All)19 b(pro)q(cesses)e (participate)g(in)h(this)f(con)o(text)f(when)75 2087 y(computation)k(starts.) 32 b(MPI)20 b(do)q(es)g(not)f(sp)q(ecify)i(ho)o(w)e(pro)q(cesses)h(are)g (initially)i(rank)o(ed)e(within)h(the)75 2143 y(con)o(text)11 b Fe(MPI)p 308 2143 15 2 v 17 w(ALL)p Fm(.)g(It)h(is)g(exp)q(ected)h(that)f (the)g(start-up)f(pro)q(cedure)i(used)f(to)f(initiate)i(an)f(MPI)g(program)75 2200 y(\(at)j(load-time)j(or)d(run-time\))i(will)h(pro)o(vide)f(information)g (or)e(con)o(trol)h(on)h(this)f(initial)j(ranking)e(\(e.g.,)75 2256 y(b)o(y)12 b(sp)q(ecifying)j(that)c(pro)q(cesses)i(are)f(rank)o(ed)h (according)g(to)e(their)i(pid's,)h(or)d(according)i(to)f(the)h(ph)o(ysical)75 2312 y(addresses)h(of)f(the)h(executing)g(pro)q(cessors,)f(or)g(according)h (to)f(a)h(n)o(um)o(b)q(ering)g(sc)o(heme)g(sp)q(eci\014ed)i(at)d(load)75 2369 y(time\).)166 2508 y Fd(Discussion:)h Fc(If)e(w)o(e)h(think)e(of)h (adding)f(new)i(pro)q(cesses)i(at)d(run-time,)e(then)j Fb(MPI)p 1454 2508 14 2 v 15 w(ALL)f Fc(con)o(v)o(eys)g(the)h(wrong)75 2564 y(impression,)f(since)j(it)f(is)f(just)h(the)h(initial)d(set)j(of)e(pro) q(cesses.)166 2704 y Fm(The)i(follo)o(wing)h(op)q(erations)g(are)e(a)o(v)m (ailable)j(for)e(creating)g(new)h(con)o(texts.)p eop %%Page: 3 4 bop 75 -100 a Ff(1.2.)34 b(BASIC)16 b(CONTEXT)f(OPERA)l(TIONS)964 b Fm(3)166 45 y Fk(MPI)p 275 45 16 2 v 18 w(COPY)p 446 45 V 18 w(CONTEXT\(new)o(con)o(text,)17 b(con)o(text\))166 137 y Fm(Create)g(a)g(new)g(con)o(text)g(that)g(includes)j(all)e(pro)q(cesses)g(in) g(the)f(old)h(con)o(text.)26 b(The)18 b(rank)f(of)g(the)75 193 y(pro)q(cesses)g(in)g(the)f(previous)h(con)o(text)f(is)g(preserv)o(ed.)24 b(The)16 b(call)i(m)o(ust)d(b)q(e)i(executed)g(b)o(y)f(all)i(pro)q(cesses)75 250 y(in)f(the)g(old)g(con)o(text.)22 b(It)17 b(is)g(a)f(blo)q(c)o(king)h (call:)24 b(No)16 b(call)h(returns)f(un)o(til)i(all)f(pro)q(cesses)g(ha)o(v)o (e)f(called)i(the)75 306 y(function.)j(The)15 b(parameters)f(are)75 406 y Fk(OUT)k(new)o(con)o(text)k Fm(handle)16 b(to)d(newly)h(created)g(con)o (text.)19 b(The)14 b(handle)h(should)g(not)e(b)q(e)i(asso)q(ciated)189 462 y(with)g(an)g(ob)s(ject)g(b)q(efore)g(the)h(call.)75 554 y Fk(IN)h(con)o(text)23 b Fm(handle)17 b(to)d(old)i(con)o(text)166 689 y Fk(MPI)p 275 689 V 18 w(NEW)p 422 689 V 19 w(CONTEXT\(new)o(con)o (text,)h(con)o(text,)g(arra)o(y)p 1338 689 V 18 w(of)p 1398 689 V 19 w(ranks,)f(size\))75 824 y(OUT)i(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h(created)g(con)o(text)f(at)h(calling)h(pro)q (cess.)19 b(This)14 b(handle)g(should)189 880 y(not)g(b)q(e)i(asso)q(ciated)g (with)f(an)g(ob)s(ject)g(b)q(efore)h(the)f(call.)75 972 y Fk(IN)i(con)o(text) 23 b Fm(handle)17 b(to)d(old)i(con)o(text)75 1064 y Fk(IN)h(arra)o(y)p 277 1064 V 18 w(of)p 337 1064 V 19 w(ranks)22 b Fm(ranks)15 b(in)h(the)f(old)h(con)o(text)e(of)h(the)g(pro)q(cesses)h(that)e(join)i(the)f (new)h(con)o(text)75 1156 y Fk(IN)h(size)23 b Fm(size)16 b(of)f(new)h(con)o (text)e(\(in)o(teger\))166 1255 y(A)20 b(new)g(con)o(text)f(is)i(created)e (for)h(the)g(pro)q(cesses)g(in)h(the)f(old)g(con)o(text)f(that)g(are)h (listed)h(in)g(the)75 1312 y(arra)o(y)l(.)f(The)c(pro)q(cesses)f(are)h (listed)g(according)g(to)f(their)h(rank)g(in)g(the)f(old)i(con)o(text.)j(The) c(rank)f(of)g(the)75 1368 y(pro)q(cesses)h(in)g(the)f(new)g(con)o(text)g(is)h (determined)g(b)o(y)f(their)h(place)g(in)g(the)f(list.)166 1424 y(The)h(call)g(has)g(to)f(b)q(e)h(executed)h(b)o(y)e(all)i(pro)q(cesses) f(listed)g(in)h(the)f(arra)o(y;)e(all)i(mak)o(e)f(the)h(call)h(with)75 1481 y(the)h(same)g(list)i(of)d(parameters.)29 b(Pro)q(cesses)18 b(in)h(the)g(old)g(con)o(text)e(that)h(do)g(not)g(b)q(elong)i(to)d(the)i(new) 75 1537 y(con)o(text)14 b(need)i(not)f(mak)o(e)g(the)g(call.)21 b(The)15 b(call)h(is)g(blo)q(c)o(king;)g(no)f(pro)q(cess)g(returns)g(from)f (the)i(call)g(un)o(til)75 1594 y(all)g(pro)q(cesses)g(ha)o(v)o(e)f(executed)h (the)f(call.)166 1686 y Fk(MPI)p 275 1686 V 18 w(SPLIT)p 445 1686 V 19 w(CONTEXT\(new)o(con)o(text,)i(con)o(text,)g(k)o(ey)l(,)f(index\)) 75 1821 y(OUT)i(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h(created)g(con) o(text)f(at)h(calling)h(pro)q(cess.)19 b(This)14 b(handle)g(should)189 1877 y(not)g(b)q(e)i(asso)q(ciated)g(with)f(an)g(ob)s(ject)g(b)q(efore)h(the) f(call.)75 1969 y Fk(IN)i(con)o(text)23 b Fm(handle)17 b(to)d(old)i(con)o (text)75 2061 y Fk(IN)h(k)o(ey)22 b Fm(in)o(teger)75 2153 y Fk(IN)17 b(index)23 b Fm(in)o(teger)166 2252 y(A)18 b(new)g(con)o(text)f(is)h (created)g(for)g(eac)o(h)g(distinct)h(v)m(alue)g(of)e Fe(key)p Fm(;)h(this)h(con)o(text)e(is)h(shared)g(b)o(y)g(all)75 2308 y(pro)q(cesses)13 b(that)f(made)h(the)g(call)h(with)f(this)g(k)o(ey)f(v)m (alue.)21 b(Within)13 b(eac)o(h)g(new)g(con)o(text)f(the)h(pro)q(cesses)g (are)75 2365 y(rank)o(ed)j(according)g(to)g(the)g(order)g(of)f(the)h Fe(index)g Fm(v)m(alues)h(they)f(pro)o(vided;)h(in)g(case)f(of)f(ties,)i(pro) q(cesses)75 2421 y(are)e(rank)o(ed)g(according)h(to)e(their)i(rank)f(in)h (the)f(old)h(con)o(text.)166 2478 y(This)e(call)h(is)f(blo)q(c)o(king:)21 b(No)13 b(call)i(returns)f(un)o(til)h(all)f(pro)q(cesses)g(in)h(the)f(old)g (con)o(text)f(executed)i(the)75 2534 y(call.)166 2591 y(P)o(articular)g(uses) h(of)e(this)i(function)g(are:)166 2647 y(\(i\))g(Reordering)h(pro)q(cesses:) 22 b(All)c(pro)q(cesses)e(pro)o(vide)h(the)f(same)g Fe(key)g Fm(v)m(alue,)h(and)g(pro)o(vide)f(their)75 2704 y(index)g(in)h(the)e(new)g (order.)p eop %%Page: 4 5 bop 75 -100 a Fm(4)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)166 45 y Fm(\(ii\))e(Splitting)h(a)e(con)o(text)f(in)o (to)i(sub)q(con)o(texts,)f(while)i(preserving)f(the)f(old)h(relativ)o(e)g (order)f(among)75 102 y(pro)q(cesses:)21 b(All)c(pro)q(cesses)f(pro)o(vide)g (the)g(same)f Fe(index)g Fm(v)m(alue,)i(and)f(pro)o(vide)g(a)f(k)o(ey)h(iden) o(tifying)h(their)75 158 y(new)e(sub)q(con)o(text.)166 214 y Fe(MPI)p 241 214 15 2 v 17 w(COPY)p 354 214 V 16 w(CONTEXT)g Fm(is)h(a)f(particular)i(case)e(of)h Fe(MPI)p 1069 214 V 16 w(SPLIT)p 1205 214 V 17 w(CONTEXT)p Fm(,)e(when)i(all)h(pro)q(cesses)f(pro-) 75 271 y(vide)g(the)g(same)e(k)o(ey)h(and)h(index)g(parameter.)166 363 y Fk(MPI)p 275 363 16 2 v 18 w(RANK\(rank,)g(con)o(text\))75 504 y(OUT)i(rank)23 b Fm(in)o(teger)75 598 y Fk(IN)17 b(con)o(text)23 b Fm(con)o(text)15 b(handle)166 705 y(Return)h(the)f(rank)g(of)g(the)g (calling)i(pro)q(cess)e(within)i(the)e(sp)q(eci\014ed)i(con)o(text.)166 796 y Fk(MPI)p 275 796 V 18 w(SIZE\(size,)h(con)o(text\))75 938 y(OUT)g(size)23 b Fm(in)o(teger)75 1032 y Fk(IN)17 b(con)o(text)23 b Fm(con)o(text)15 b(handle)166 1138 y(Return)h(the)f(n)o(um)o(b)q(er)h(of)e (pro)q(cesses)i(that)e(b)q(elong)j(to)d(the)h(sp)q(eci\014ed)j(con)o(text.) 166 1195 y(A)d(con)o(text)g(ob)s(ject)f(is)i(destro)o(y)o(ed)f(using)h(the)f Fe(MPI)p 1036 1195 15 2 v 17 w(FREE)f Fm(function.)75 1316 y Fg(1.2.1)55 b(Usage)19 b(note)75 1402 y Fm(Use)e(of)g(con)o(texts)f(for)h (libraries:)25 b(Eac)o(h)17 b(library)h(ma)o(y)e(pro)o(vide)i(an)f (initialization)j(routine)e(that)e(is)i(to)75 1459 y(b)q(e)c(called)i(b)o(y)e (all)g(pro)q(cesses,)g(and)g(that)f(generate)h(a)g(con)o(text)f(for)g(the)h (use)g(of)f(that)g(library)l(.)21 b(A)14 b(sc)o(heme)75 1515 y(for)j(allo)o(wing)i(eac)o(h)g(link)o(ed)g(library)g(to)f(ha)o(v)o(e)g(its)g (o)o(wn)f(initializati)q(on)k(co)q(de)d(w)o(ould)h(can)f(b)q(e)h(used)g(for) 75 1572 y(this)d(purp)q(ose)f(\(assuming)h(the)f(library)h(will)h(not)e(ha)o (v)o(e)f(sev)o(eral)i(concurren)o(t)f(instan)o(tiations\).)166 1628 y(Use)g(of)g(con)o(texts)g(for)f(functional)j(decomp)q(osition:)k(A)15 b(harness)g(program,)f(running)i(in)g(the)g(con-)75 1684 y(text)e Fe(ALL)g Fm(generates)f(a)h(sub)q(con)o(text)h(for)e(eac)o(h)i(mo)q(dule)g (and)g(then)f(starts)f(the)i(submo)q(dule)g(within)h(the)75 1741 y(corresp)q(onding)g(con)o(text.)166 1797 y(Use)i(of)f(con)o(texts)g (for)g(collectiv)o(e)j(comm)o(unication:)26 b(A)18 b(con)o(text)f(is)h (created)g(for)f(eac)o(h)h(group)f(of)75 1854 y(pro)q(cesses)f(where)f (collectiv)o(e)i(comm)o(unication)f(is)g(to)e(o)q(ccur.)166 1910 y(Use)k(of)g(con)o(texts)f(for)h(con)o(text-switc)o(hing)g(among)g(sev)o (eral)g(parallel)i(executions:)26 b(A)18 b(pream)o(ble)75 1967 y(co)q(de)d(is)g(used)g(to)f(generate)g(a)g(di\013eren)o(t)g(con)o(text)g (for)g(eac)o(h)g(execution;)i(this)e(pream)o(ble)h(co)q(de)g(needs)h(to)75 2023 y(use)g(a)e(m)o(utual)i(exclusion)h(proto)q(col)e(to)f(mak)o(e)h(sure)g (eac)o(h)h(thread)f(claims)h(the)f(righ)o(t)g(con)o(text.)166 2156 y Fd(Implemen)o(tati)o(on)d(note:)166 2205 y Fc(W)m(e)18 b(outline)f(here)i(t)o(w)o(o)f(p)q(ossible)g(implemen)o(tations)d(of)j(con)o (texts.)31 b(They)19 b(are)f(b)o(y)g(no)g(means)f(the)i(only)75 2255 y(p)q(ossible)c(ones.)23 b(In)15 b(eac)o(h)g(implemen)o(tation)d(w)o(e)j (assume)g(that)g(a)g(con)o(text)h(ob)r(ject)g(is)f(a)g(p)q(oin)o(ter)g(to)g (a)g(structure)75 2305 y(that)d(describ)q(es)i(the)f(con)o(text.)18 b(A)12 b(comp)q(onen)o(t)f(of)g(this)h(structure)i(is)e(a)g(table)f(of)h(the) g(pro)q(cesses)j(that)d(participate)75 2355 y(in)17 b(the)i(con)o(text,)f (ordered)h(b)o(y)f(rank.)29 b(W)m(e)17 b(assume)h(that)f(the)i(n)o(um)o(b)q (er)e(of)g(concurren)o(tly)i(activ)o(e)e(con)o(texts)i(at)75 2405 y(eac)o(h)12 b(pro)q(cess)i(is)d(relativ)o(ely)g(small;)e(sa)o(y)j (16-32.)k(In)c(either)g(implemen)o(tatio)o(n)d(one)j(migh)o(t)d(ha)o(v)o(e)j (disjoin)o(t)e(message)75 2455 y(queues)j(for)e(eac)o(h)i(con)o(text,)f(or)g (ha)o(v)o(e)f(shared)i(queues,)g(with)e(the)h(righ)o(t)g(mec)o(hanisms)d(for) j(con)o(text)g(matc)o(hing)e(and)75 2504 y(bu\013er)15 b(allo)q(cation.)166 2554 y(Prop)q(osal)f(1:)j(Large)d(con)o(text)h(tags.)166 2604 y(In)d(this)g(implem)o(en)o(tation)d(w)o(e)j(use)h(large)e(con)o(text)i(tags) f(\(sa)o(y)f(32)h(bits\),)g(so)g(that)f(matc)o(hing)f(of)i(an)f(incoming)75 2654 y(tag)19 b(with)g(a)h(lo)q(cal)e(pro)q(cess)k(requires)f(to)e(p)q (erform)g(a)g(searc)o(h)i(in)e(a)g(hash)h(table)f(or)h(another)g(similar)d (searc)o(h)75 2704 y(structure)e(\(this)d(o)q(ccurs)i(whenev)o(er)g(a)e (message)g(is)h(receiv)o(ed\).)19 b(All)11 b(messages)i(sen)o(t)g(within)f(a) g(con)o(text)h(carry)g(the)p eop %%Page: 5 6 bop 75 -100 a Ff(1.3.)34 b(AD)o(V)-5 b(ANCED)14 b(CONTEXT)i(OPERA)l(TIONS)841 b Fm(5)75 45 y Fc(same)13 b(tag)h(v)n(alue.)k(W)m(e)c(use)h(as)g(con)o(text)f (tag)g(the)h(pid)f(of)f(the)i(lo)o(w)o(est)f(n)o(um)o(b)q(ered)g(pro)q(cess)i (in)e(the)h(con)o(text)g(\(let's)75 95 y(call)g(it)g(the)h(con)o(text)g (leader\),)h(concatenated)g(with)e(a)g(coun)o(ter)i(that)e(is)h(incremen)o (ted)g(whenev)o(er)h(this)e(pro)q(cess)75 145 y(allo)q(cates)h(a)g(new)g (tag.)24 b(This)16 b(guaran)o(tee)g(a)g(unique)g(tag)g(for)f(eac)o(h)i(group) f(\(sp)q(ecial)g(co)q(de)h(needed)g(for)f(coun)o(ter)75 195 y(wraparound\).)166 247 y Fb(MPI)p 235 247 14 2 v 15 w(COPY)p 338 247 V 15 w(CONTEXT)11 b Fc(-)h(A)h(new)g(tag)f(is)g(generated)i(b)o(y)f (incremen)o(ting)e(the)i(old)f(tag)h(b)o(y)f(one;)h(one)f(can)h(either)75 296 y(cop)o(y)f(the)g(old)f(con)o(text)h(table,)g(or)f(create)j(a)d(new)h(p)q (oin)o(ter)g(to)g(the)g(old)f(table.)17 b(A)12 b(global)e(barrier)i(sync)o (hronization)75 346 y(is)i(needed)h(to)f(mak)o(e)e(sure)j(the)g(call)e(is)h (blo)q(c)o(king.)166 398 y Fb(MPI)p 235 398 V 15 w(SPLIT)p 360 398 V 15 w(CONTEXT)j Fc(-)h(A)h(naiv)o(e)f(implemen)o(tatio)o(n)e(is)j (to)f(ha)o(v)o(e)g(an)h(all-to-all)d(comm)o(unicati)o(on)g(where)75 448 y(eac)o(h)h(pro)q(cess)i(gathers)f(the)f Fb(\(key,)k(index\))15 b Fc(pairs)i(of)f(all)g(pro)q(cesses)j(in)e(the)g(old)f(group.)27 b(Eac)o(h)17 b(pro)q(cess)i(can)75 498 y(determine)14 b(whether)h(it)e(is)h (the)g(leader)h(of)e(a)g(new)i(group)e(and)h(broadcast)g(the)h(new)f(group)g (tag)f(to)h(all)e(mem)o(b)q(ers)75 548 y(\(using)j(p)q(oin)o(t)g(to)g(p)q (oin)o(t)g(comm)o(unicatio)o(n)e(in)i(the)h(old)e(group,)h(or)g(using)h (another)f(all-to-all)e(comm)o(unicatio)o(n\).)75 597 y(Algorithmic)f(minds)g (will)g(think)i(of)f(man)o(y)f(p)q(ossible)i(optimizations.)166 649 y Fb(MPI)p 235 649 V 15 w(NEW)p 316 649 V 15 w(CONTEXT)c Fc(-)g(The)i(lo)o(w)o(est)f(n)o(um)o(b)q(ered)f(pro)q(cess)j(in)e(the)g(list) g(broadcast)g(the)h(new)f(con)o(text)h(tag)e(to)h(all)75 699 y(pro)q(cesses)i(in)d(the)g(list)g(\(using)g(p)q(oin)o(t)g(to)g(p)q(oin)o(t)f (comm)o(unication)e(op)q(erations\).)17 b(A)11 b(more)e(robust)h(implemen)o (tation)75 749 y(ma)o(y)i(en)o(tail)h(the)i(broadcast)f(of)f(the)i(mem)o(b)q (er)d(list,)h(for)h(error)g(c)o(hec)o(king.)166 801 y Fb(MPI)p 235 801 V 15 w(RANK,)21 b(MPI)p 447 801 V 15 w(SIZE)13 b Fc(-)g(require)i(lo) q(cal)e(access)j(to)e(the)g(con)o(text)h(ob)r(ject)166 853 y(Prop)q(osal)f(2:)j(Small)12 b(con)o(text)i(tags.)166 905 y(In)19 b(that)h(implemen)o(tatio)o(n)d(the)j(n)o(um)o(b)q(er)f(of)g (distinct)h(con)o(text)g(tag)f(v)n(alues)g(is)h(equal)f(to)g(the)h(maxima)o (l)75 955 y(n)o(um)o(b)q(er)d(of)g(con)o(texts)i(that)f(can)g(b)q(e)h(activ)o (e)f(at)f(the)i(same)e(no)q(de.)30 b(Th)o(us,)19 b(the)f(con)o(text)h(tag)e (of)h(an)f(incoming)75 1005 y(message)i(can)g(b)q(e)g(used)h(to)f(index)f (directly)i(in)o(to)e(a)g(con)o(text)i(table,)g(a)o(v)o(oiding)c(the)k(need)g (for)e(a)h(searc)o(h)h(in)e(a)75 1055 y(hash)13 b(table.)18 b(Eac)o(h)13 b(pro)q(cess)i(has)e(a)f(unique)h(con)o(text)h(tag)e(for)h (incoming)d(comm)o(unication)g(within)i(this)h(con)o(text.)75 1104 y(Ho)o(w)o(ev)o(er,)h(di\013eren)o(t)h(con)o(text)g(tag)e(v)n(alues)h (ma)o(y)e(b)q(e)j(used)g(b)o(y)f(di\013eren)o(t)h(pro)q(cesses)h(for)e(the)h (same)e(con)o(text)i(\(this)75 1154 y(is)e(necessary)i(in)e(order)h(to)f (densely)h(p)q(opulate)f(the)h(con)o(text)f(tag)g(range\).)18 b(The)c(con)o(text)g(table)f(carries,)h(for)e(eac)o(h)75 1204 y(mem)o(b)q(er)g(of)i(the)g(con)o(text,)g(the)h(con)o(text)f(tag)g(to)f(b)q (e)i(used)g(when)f(sending)g(messages)g(to)g(it.)166 1256 y Fb(MPI)p 235 1256 V 15 w(COPY)p 338 1256 V 15 w(CONTEXT)c Fc(-)h(A)h(new)g (tag)f(is)h(generated)h(b)o(y)e(eac)o(h)h(pro)q(cess)i(for)d(the)i(new)f(con) o(text,)g(and)f(broadcast)75 1306 y(to)18 b(all)g(other)h(mem)o(b)q(ers)f(of) g(the)h(con)o(text,)h(using)e(an)h(all-to-all)d(comm)o(uni)o(cation.)29 b(A)19 b(new)g(con)o(text)h(table)e(is)75 1356 y(created,)d(with)e(these)j (new)e(tags.)166 1408 y Fb(MPI)p 235 1408 V 15 w(SPLIT)p 360 1408 V 15 w(CONTEXT)j Fc(-)h(A)h(naiv)o(e)f(implemen)o(tatio)o(n)e(is)j(to)f (ha)o(v)o(e)g(an)h(all-to-all)d(comm)o(unicati)o(on)g(where)75 1457 y(eac)o(h)e(pro)q(cess)h(gathers)f(the)h Fb(\(key,)20 b(index,)h(new)p 881 1457 V 15 w(context)p 1050 1457 V 14 w(tag\))13 b Fc(triples)h(of)e(all)h(pro)q(cesses)j(in)d(the)h(old)f(group.)75 1507 y(Eac)o(h)g(pro)q(cess)h(then)f(creates)i(a)d(new)h(con)o(text)g(table)g (for)f(the)h(pro)q(cesses)i(that)e(pro)o(vided)f(the)i(same)d(k)o(ey)i(v)n (alue)f(as)75 1557 y(it.)166 1609 y Fb(MPI)p 235 1609 V 15 w(NEW)p 316 1609 V 15 w(CONTEXT)g Fc(-)i(Similar)d(to)j Fb(MPI)p 785 1609 V 15 w(COPY)p 888 1609 V 15 w(CONTEXT)p Fc(.)166 1661 y Fb(MPI)p 235 1661 V 15 w(RANK,)21 b(MPI)p 447 1661 V 15 w(SIZE)13 b Fc(-)g(require)i(lo)q(cal)e(access)j(to)e(the)g(con)o(text)h(ob)r(ject)75 1899 y Fl(1.3)70 b(Adv)l(anced)23 b(con)n(text)f(op)r(erations)75 2005 y Fm(Additional)e(functions)e(are)f(required)i(to)e(supp)q(ort)h(a)f (less)i(static)e(mo)q(del,)i(where)f(pro)q(cesses)g(ma)o(y)f(b)q(e)75 2062 y(created)i(or)f(deleted)i(during)g(execution.)31 b(This)20 b(requires)f(con)o(texts)f(to)h(gro)o(w)e(or)h(b)q(e)i(merged.)30 b(This)75 2118 y(section)12 b(outlines)g(p)q(ossible)h(mec)o(hanisms)f(for)e (suc)o(h)i(extension.)19 b(The)11 b(lac)o(k)h(of)f(in)o(terupt)g(driv)o(en)h (comm)o(u-)75 2175 y(nication)j(mec)o(hanisms)g(in)g(MPI)f(restricts)g(the)g (functionalit)o(y)h(of)f(suc)o(h)g(mec)o(hanisms:)20 b(A)14 b(new)h(pro)q(cess)75 2231 y(can)g(join)f(an)h(existing)g(con)o(text)f(only)h (if)g(all)g(pro)q(cesses)g(that)f(participate)h(in)g(the)f(old)h(con)o(text)f (execute)75 2287 y(a)h(call)h(to)f(add)g(this)h(new)f(pro)q(cess.)166 2346 y(A)j(prede\014ned)i(lo)q(cal)g(con)o(text)e Fe(MPI)p 792 2346 15 2 v 16 w(ME)g Fm(where)h(only)g(one)f(pro)q(cess)h(participates,) g(is)g(prede\014ned)75 2403 y(for)c(eac)o(h)g(pro)q(cess.)166 2497 y Fk(MPI)p 275 2497 16 2 v 18 w(SP)l(A)-6 b(WN\()17 b(new)o(con)o(text,) f(oldcon)o(text,)i(arra)o(y)p 1202 2497 V 18 w(of)p 1262 2497 V 19 w(en)o(vironmen)o(ts,)d(len\))166 2591 y Fm(Spa)o(wn)h(new)h(pro)q (cesses)g(and)g(create)f(a)g(new)h(con)o(text)f(that)g(includes)j(the)d(pro)q (cesses)h(in)h(the)e(old)75 2647 y(con)o(text,)d(follo)o(w)o(ed)h(b)o(y)g (the)g(newly)h(spa)o(wned)f(pro)q(cesses.)19 b(The)14 b(arra)o(y)f(pro)o (vides)h(en)o(vironmen)o(t)g(param-)75 2704 y(eters)j(for)f(eac)o(h)h(newly)h (generated)e(pro)q(cess.)26 b(The)17 b(form)f(of)g(the)h(arra)o(y)f(en)o (tries)h(are)g(implemen)o(tation)p eop %%Page: 6 7 bop 75 -100 a Fm(6)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)75 45 y Fm(dep)q(enden)o(t)i({)e(they)g(ma)o(y)f (include)k(information)d(on)h(the)f(pro)q(cessor)g(that)f(is)i(to)f(run)g (the)h(new)f(tasks,)75 102 y Fe(argv,)23 b(argc)15 b Fm(argumen)o(ts,)f(etc.) 75 214 y Fk(OUT)k(new)o(con)o(text)k Fm(handle)17 b(to)d(new)i(con)o(text)75 314 y Fk(OUT)i(oldcon)o(text)24 b Fm(handle)16 b(to)f(old)h(con)o(text)75 413 y Fk(IN)h(arra)o(y)p 277 413 16 2 v 18 w(of)p 337 413 V 19 w(en)o(vironmen)o(ts)k Fm(list)16 b(of)f(en)o(vironmen)o(ts)g(for)g(new)g (pro)q(cesses)75 512 y Fk(IN)i(len)23 b Fm(n)o(um)o(b)q(er)16 b(of)f(new)g(pro)q(cesses)h(\(in)o(teger\))166 625 y(As)23 b(particular)g(cases,)h Fe(MPI)p 669 625 15 2 v 17 w(SPAWN\()f(newcontext,)f (MPI)p 1211 625 V 17 w(ME,...\))42 b Fm(allo)o(ws)23 b(one)g(pro)q(cess)g(to) 75 681 y(spa)o(wn)f(new)g(pro)q(cesses,)j(and)d(create)g(a)g(new)h(con)o (text)e(that)h(consists)g(of)g(the)g(spa)o(wning)h(pro)q(cess,)75 737 y(follo)o(w)o(ed)13 b(b)o(y)g(the)g(spa)o(wned)g(pro)q(cesses;)h Fe(MPI)p 849 737 V 17 w(SPAWN\()23 b(newcontext,)f(MPI)p 1391 737 V 17 w(ALL,...\))c Fm(allo)o(ws)13 b(to)g(add)75 794 y(to)21 b(create)g(a)g(new)h(\\univ)o(ersal")g(con)o(text)e(that)h(con)o(tains)h(all) g(previous)g(pro)q(cesses)g(and)g(the)f(newly)75 850 y(spa)o(wned)15 b(pro)q(cesses.)166 944 y Fk(MPI)p 275 944 16 2 v 18 w(BCAST)p 473 944 V 19 w(CONTEXT\()d(new)o(con)o(text,)g(con)o(text,)g(ro)q(ot,)h(arra) o(y)p 1514 944 V 18 w(of)p 1574 944 V 19 w(ranks,)e(coun)o(t\))166 1093 y Fm(Beha)o(v)o(es)i(lik)o(e)h Fe(MPI)p 495 1093 15 2 v 16 w(NEW)p 583 1093 V 17 w(CONTEXT)p Fm(,)e(except)h(that)f(only)h(one)g (pro)q(cess)g(in)h(the)f(old)g(con)o(text,)f(namely)75 1150 y(the)21 b(pro)q(cess)g(with)h(rank)e Fe(root)p Fm(,)i(has)e(to)h(compute)g (the)g(list)h(of)e(the)h(ranks)g(of)f(the)h(pro)q(cesses)h(that)75 1206 y(participate)c(in)g(the)f(new)h(con)o(text.)25 b(The)17 b(new)g(con)o(text)g(do)q(es)h(not)e(necessarily)j(include)g(the)f(pro)q (cess)75 1263 y Fe(root)p Fm(.)h(The)c(call)h(is)f(blo)q(c)o(king,)h(and)f (it)g(has)f(to)g(b)q(e)i(executed)f(b)o(y)g(all)g(pro)q(cesses)h(in)f(the)g (list)g(and)g(b)o(y)g(the)75 1319 y(ro)q(ot)f(pro)q(cess.)75 1432 y Fk(OUT)k(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h(created)g(con) o(text)f(at)h(calling)h(pro)q(cess.)19 b(This)14 b(handle)g(should)189 1488 y(not)g(b)q(e)i(asso)q(ciated)g(with)f(an)g(ob)s(ject)g(b)q(efore)h(the) f(call.)75 1587 y Fk(IN)i(con)o(text)23 b Fm(handle)17 b(to)d(old)i(con)o (text)75 1687 y Fk(IN)h(ro)q(ot)23 b Fm(index)17 b(of)e(new)g(con)o(text)g (creator)f(in)i(old)g(con)o(text)75 1786 y Fk(IN)h(arra)o(y)p 277 1786 16 2 v 18 w(of)p 337 1786 V 19 w(ranks)22 b Fm(ranks)17 b(in)h(old)g(con)o(texts)e(of)h(mem)o(b)q(ers)h(of)f(new)g(con)o(text;)g (signi\014can)o(t)i(only)f(at)189 1842 y(ro)q(ot)75 1941 y Fk(IN)f(coun)o(t)23 b Fm(size)16 b(of)f(new)h(con)o(text;)e(signi\014can)o(t) i(only)g(at)e(ro)q(ot)166 2130 y Fd(Implemen)o(tati)o(on)e(note:)166 2181 y Fc(The)19 b(call)e(can)i(b)q(e)g(implemen)o(ted)d(as)i(a)g(broadcast)h (from)e(the)i(ro)q(ot)f(to)g(all)g(new)g(con)o(text)h(participan)o(ts,)75 2231 y(follo)o(w)o(ed)12 b(b)o(y)h(the)h(co)q(de)g(of)f Fb(MPI)p 574 2231 14 2 v 15 w(NEW)p 655 2231 V 15 w(CONTEXT)f Fc(\(the)i(broadcast)g (is)f(executed)i(using)e(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(uni-)75 2281 y(cation\).)166 2421 y Fm(Example:)34 b(Supp)q(ose)24 b(one)e(has)g(a)g(serv)o(er)g(con)o(text)g Fe(SERVER)p Fm(,)f(and)h(a)g (clien)o(t)i(con)o(text)d Fe(CLIENT)p Fm(.)75 2478 y(Pro)q(cesses)16 b(in)i(either)f(con)o(texts)e(ma)o(y)h(not)g(kno)o(w)g(ab)q(out)g(pro)q (cesses)h(in)g(the)g(other)f(con)o(text,)f(but)i(they)75 2534 y(all)h(b)q(elong)g(to)e(con)o(text)g Fe(MPI)p 582 2534 15 2 v 17 w(ALL)p Fm(,)g(and)h(all)g(kno)o(w)g(ab)q(out)f(pro)q(cess)h Fe(root)g Fm(\(the)f(\\nameserv)o(er"\).)24 b(The)75 2591 y(ro)q(ot)17 b(pro)q(cess)g(kno)o(ws)g(ab)q(out)h(eac)o(h)f(of)g(the)h(t)o(w)o(o)e(con)o (texts.)26 b(A)17 b(call)i(to)e Fe(MPI)p 1408 2591 V 17 w(BCAST)p 1545 2591 V 16 w(CONTEXT)g Fm(can)g(b)q(e)75 2647 y(used)e(to)f(create)g(a)g (new)h(con)o(text)e(that)h(is)h(the)g(union)g(of)f(the)g Fe(CLIENT)g Fm(and)h Fe(SERVER)e Fm(con)o(texts,)h(so)g(as)g(to)75 2704 y(allo)o(w)h(these)h(to)e(comm)o(unicate.)p eop %%Page: 7 8 bop 75 -100 a Ff(1.4.)29 b(EXAMPLES)15 b({)g(PR)o(OBLEMS)h(IN)g(RICK'S)g (LIST)756 b Fm(7)166 45 y Fd(Discussion:)166 96 y Fc(P)o(ossible)12 b(extension:)18 b(the)13 b(ro)q(ot)f(has)g(t)o(w)o(o)g(lists;)g(the)h(list)e (of)h(pro)q(cesses)j(that)d(call)f(to)h(join)f(the)i(new)g(con)o(text,)75 146 y(and)j(the)g(list)f(of)g(pro)q(cesses)k(that)c(are)i(to)e(join)o(t)g (the)h(next)h(con)o(text.)24 b(The)16 b(remianing)e(pro)q(cesses)k(are)e (returned)75 196 y(a)g(v)n(alue)f(indicating)g(they)i(w)o(ere)g(left)e(out.) 25 b(This)16 b(could)g(b)q(e)g(useful)h(for)e(master-sla)o(v)o(e)g(co)q(de.) 26 b(The)16 b(ro)q(ot)g(is)g(the)75 246 y(master)f(that)h(creates)i(new)e (con)o(texts)h(for)e(the)h(parallel)f(executuin)i(of)e(new)h(tasks.)24 b(All)15 b(disp)q(onible)g(pro)q(cesses)75 296 y(execute)h(the)e(call,)f(but) h(only)f(a)h(subset)h(is)f(allo)q(cated.)166 472 y Fk(MPI)p 275 472 16 2 v 18 w(MER)o(GE)p 490 472 V 19 w(CONTEXT\()k(new)o(con)o(text,)f (con)o(text1,)g(con)o(text2,)g(ro)q(ot\))166 565 y Fm(This)e(function)g(allo) o(ws)g(to)e(merge)h(t)o(w)o(o)f(con)o(texts)h(ev)o(en)h(when)g(there)f(is)h (no)f(con)o(text)g(that)g(encom-)75 621 y(passes)k(the)g(t)o(w)o(o)f(merged)h (con)o(texts;)g(it)g(is)h(only)f(required)h(that)e(one)i(pro)q(cess,)f (namely)h Fe(root)p Fm(,)e(b)q(e)i(in)75 678 y(b)q(oth)13 b(con)o(texts)e(to) h(b)q(e)h(merged.)19 b(The)13 b(call)h(is)f(blo)q(c)o(king)h(and)e(m)o(ust)g (b)q(e)h(executed)h(b)o(y)e(all)i(pro)q(cesses)f(that)75 734 y(participate)k(in)f(the)g(merged)g(con)o(text.)21 b(Eac)o(h)16 b(pro)q(cess)g(pro)o(vides)g(its)g(old)g(con)o(text,)f(and)h(the)g(index)h (of)75 791 y(pro)q(cess)c(ro)q(ot)f(within)i(this)g(old)f(con)o(text)f(\(the) h(index)h(ma)o(y)e(b)q(e)i(di\013eren)o(t)f(in)g(the)g(t)o(w)o(o)f(con)o (texts)g(that)g(are)75 847 y(merged\).)19 b(The)14 b(ro)q(ot)g(pro)o(vides)g (handles)h(to)e(b)q(oth)i(con)o(texts)e(that)g(are)h(to)f(b)q(e)i(merged.)k (The)c(pro)q(cesses)75 904 y(in)h Fe(context1)e Fm(precede)i(the)g(pro)q (cesses)f(in)h Fe(context2)e Fm(in)i(the)g(ranking)f(of)g(the)g(new)h(con)o (text.)75 1017 y Fk(OUT)i(new)o(con)o(text)k Fm(handle)15 b(to)d(newly)h (created)g(con)o(text)f(at)h(calling)h(pro)q(cess.)19 b(This)14 b(handle)g(should)189 1073 y(not)g(b)q(e)i(asso)q(ciated)g(with)f(an)g(ob)s (ject)g(b)q(efore)h(the)f(call.)75 1173 y Fk(IN)i(con)o(text1)23 b Fm(handle)17 b(to)d(old)i(con)o(text)75 1272 y Fk(IN)h(con)o(text2)23 b Fm(handle)17 b(to)d(second)i(con)o(text;)e(signi\014can)o(t)i(only)g(at)f (ro)q(ot)75 1372 y Fk(IN)i(ro)q(ot)23 b Fm(index)17 b(of)e(ro)q(ot)f(in)i (old)g(con)o(text)166 1485 y(Example:)23 b(a)17 b(serv)o(er)f(\(the)h(\\ro)q (ot"\))e(and)i(a)f(set)h(of)f(clien)o(t)i(pro)q(cesses)f(participate)h(in)f Fe(context1)p Fm(.)75 1541 y(The)j(serv)o(er)f(spa)o(wns)g(new)h(\\help)q (ers",)h(or)e(has)h(other)f(a)o(v)m(ailable)i(serv)o(er)e(pro)q(cesses)h (join)g(it.)33 b(A)20 b(call)75 1598 y(to)e Fe(MPI)p 209 1598 15 2 v 16 w(MERGE)p 345 1598 V 17 w(CONTEXT)f Fm(can)i(b)q(e)g(used)g(to)e (create)h(a)h(new)f(con)o(text)g(that)f(mak)o(e)h(these)h(new)f(serv)o(ers)75 1654 y(a)o(v)m(ailable)f(to)d(the)i(parallel)g(clinet.)166 1788 y Fd(Discussion:)166 1839 y Fc(Here,)h(to)q(o,)f(the)h(function)e(can)h (b)q(e)h(extended)h(to)d(allo)o(w)g(the)h(creation)h(of)e(a)h(new)g(con)o (text)h(where)g(only)e(a)75 1889 y(subset)g(of)f(the)g(pro)q(cesses)j(in)c (the)i(old)e(group)g(participate)166 1940 y(If)k Fb(MPI)p 280 1940 14 2 v 15 w(MERGE)p 405 1940 V 14 w(CONTEXT)f Fc(is)h(a)o(v)n(ailable)e (then)j(the)g(functionalit)o(y)d(of)i Fb(MPI)p 1344 1940 V 15 w(SPAWN)f Fc(can)i(b)q(e)f(reduced:)27 b(It)17 b(is)75 1990 y(su\013cien)o(t)e(to)e(create)i(a)f(new)g(con)o(text)g(that)g(consists)h(of) e(a)g(spa)o(wning)g(pro)q(cess)j(\(rather)e(than)g(sp)o(wning)f(con)o(text\)) 75 2040 y(and)h(the)g(spa)o(wned)h(pro)q(cesses.)20 b(This)14 b(new)g(con)o(text)h(can)f(then)h(b)q(e)f(merged)g(with)f(a)h(preexisting)g (con)o(text.)166 2256 y Fd(Implemen)o(tati)o(on)e(note:)166 2308 y Fc(Implemen)o(tation)f(is)i(similar)f(to)i(the)g(implemen)o(tation)c (of)k Fb(MPI)p 1179 2308 V 15 w(BCAST)p 1304 2308 V 14 w(CONTEXT)75 2542 y Fl(1.4)70 b(Examples)22 b({)h(Problems)f(in)g(Ric)n(k's)g(list)75 2646 y Fm(Only)16 b(the)g(basic)g(con)o(text)e(functions)i(are)f(used)h(in)g (these)f(examples.)166 2704 y Fd(W)l(arning)p Fc(:)h(late)e(nigh)o(t)f(w)o (ork)p eop %%Page: 8 9 bop 75 -100 a Fm(8)867 b Ff(CHAPTER)15 b(1.)35 b(CONTEXTS)15 b({)g(PR)o(OPOSAL)i(I)75 45 y Fk(F)l(unction)h(that)h(implemen)o(ts)e(sh)o (u\017e)f(p)q(erm)o(utation)i(in)g(group/con)o(text)75 204 y Fe(void)23 b(function)g(mpi_shuffle\(inbuf,)e(outbuf,)i(datatype,)g(count,) g(context,)g(size\))75 317 y(...)75 430 y({)75 486 y (mpi_copy_context\(newcontex)o(t,)e(context\);)75 543 y(mpi_rank\(context,)h (i\);)75 599 y(dest)h(=)h(shuffle\(i,)f(size\);)75 656 y(source)g(=)h (unshuffle\(i,)e(size\);)75 712 y(mpi_irecvc)g(\(handle,)h(inbuf,)g(count,)g (datatype,)g(source,)g(0,)g(nexcontext\);)75 769 y(mpi_sendc)g(\(outbuf,)f (count,)h(datatype,)g(dest,)g(0,)h(newcontext\);)75 825 y(mpi_wait\(handle,)e (ret_stat\);)75 882 y(mpi_free\(newcontext\);)75 938 y(})166 1095 y Fm(This)15 b(w)o(orks)f(OK,)h(if)g(pro)q(cesses)g(are)g (single-threaded.)21 b(If)15 b(they)g(are)g(m)o(ultithreaded,)g(one)g(has)g (to)75 1152 y(mak)o(e)f(sure)g(that)g(other)f(concurren)o(t)i(threads)f(do)g (not)g(create)g(new)g(con)o(texts.)19 b(New)14 b(con)o(text)g(creation)75 1208 y(can)h(b)q(e)h(skipp)q(ed)h(if)f(there)f(are)g(no)g(p)q(ending)i (messages)e(in)h(the)f(con)o(text)g(when)g(mpi)p 1539 1208 14 2 v 17 w(sh)o(u\017e)h(is)g(called.)75 1338 y Fk(Use)25 b(of)h(con)o(texts)g(for)f(library)h(dev)o(elopmen)o(t)44 b Fm(The)23 b(co)q(de)g(of)f(a)g(parallel)i(library)f(function)75 1395 y(should)18 b(b)q(e)f(written)g(so)f(that)g(eac)o(h)h(message)f(that)g (is)h(pro)q(duced)h(b)o(y)e(some)h(pro)q(cess)g(is)g(consumed)g(b)o(y)75 1451 y(another)d(pro)q(cess)h(during)h(the)f(execution)g(of)g(the)f(library)i (co)q(de)f({)f(i.e.)21 b(the)14 b(library)i(should)f(\\clean)h(its)75 1508 y(garbage".)166 1566 y(A)d(parallel)j(library)e(is)g(in)o(v)o(ok)o(ed)g (collectiv)o(ely)i(b)o(y)d(a)g(group)g(of)h(pro)q(cesses.)19 b(An)o(y)14 b(in)o(v)o(ok)m(ation)g(of)f(the)75 1622 y(library)k(o)q(ccurs)g (within)g(a)f(con)o(text)g(de\014ned)i(b)o(y)e(the)h(user.)23 b(All)18 b(pro)q(cesses)f(that)e(participate)i(in)h(suc)o(h)75 1679 y(con)o(text)g(in)o(v)o(ok)o(e)h(the)g(parallel)h(library)l(,)h(and)e (matc)o(hing)g(in)o(v)o(ok)m(ations)g(o)q(ccur)g(at)f(all)i(of)e(them)h(in)h (the)75 1735 y(same)15 b(order.)166 1793 y(The)10 b(library)h(has)f(to)g (generate)g(a)f(new)i(con)o(text,)f(using)h(the)f(function)h Fe(MPI)p 1427 1793 15 2 v 17 w(COPY)p 1540 1793 V 17 w(CONTEXT\(newcontext,) 75 1850 y(context\))p Fm(,)16 b(for)h(eac)o(h)g(con)o(text)g Fe(context)f Fm(where)h(from)g(the)g(library)h(ma)o(y)f(b)q(e)h(in)o(v)o(ok)o (ed.)26 b(The)17 b(library)75 1906 y(will)g(then)f(use)g(the)g(con)o(text)f Fe(newcontext)f Fm(for)h(its)h(comm)o(unication)h(whenev)o(er)f(it)g(is)g(in) o(v)o(ok)o(ed)g(within)75 1963 y(the)h(con)o(text)f Fe(context)p Fm(.)24 b(If)17 b(the)g(library)g(uses)h(collectiv)o(e)g(comm)o(unication)g (within)g(dynamically)g(de-)75 2019 y(\014ned)h(subgroups,)f(then)g(these)g (subgroups)g(will)i(b)q(e)f(created)e(b)o(y)h(splitting)i(the)d(group)h(of)g (pro)q(cesses)75 2076 y(de\014ned)f(b)o(y)e Fe(newcontext)p Fm(.)166 2134 y(In)23 b(the)g(general)g(case,)h(the)e(con)o(text)g(of)g(the)h (in)o(v)o(ok)m(ation)g(has)f(to)g(b)q(e)h(passed)g(as)f(an)h(explicit)75 2190 y(parameter)14 b(when)i(the)f(library)h(is)f(in)o(v)o(ok)o(ed.)21 b(The)15 b(library)h(co)q(de)f(will)i(generate)e(a)g(new)g(con)o(text)f(when) 75 2247 y(it)h(starts)f(executing)j(and)e(will)i(free)e(this)h(con)o(text)e (when)i(it)g(terminates.)166 2305 y(There)f(are)g(sev)o(eral)h(sp)q(ecial)h (cases)e(where)g(dynamic)h(con)o(text)f(creation)g(can)h(b)q(e)f(a)o(v)o (oided.)166 2363 y(Case)10 b(1:)18 b(The)11 b(library)g(is)h(alw)o(a)o(ys)e (in)o(v)o(ok)o(ed)h(within)h(the)e(con)o(text)h Fe(MPI)p 1343 2363 V 16 w(ALL)p Fm(.)f(Then)i(the)f(new)g(con)o(text)75 2420 y(can)18 b(b)q(e)g(\\cac)o(hed":)25 b(A)17 b(library)i(collectiv)o(e)g (initialization)i(routine)d(should)g(b)q(e)h(in)o(v)o(ok)o(ed)f(b)o(y)f(the)h (user)75 2476 y(at)c(the)g(start)f(of)h(the)g(program;)f(this)i(routine)f (creates)g(a)g(cop)o(y)g(of)g(the)g(con)o(text)g Fe(MPI)p 1537 2476 V 17 w(ALL)f Fm(and)i(stores)e(a)75 2532 y(handle)k(to)e(it)h(in)g(a)g (C)f(static)h(v)m(ariable)h(\(F)l(ortran)d(77)h(COMMON\).)g(The)h(con)o(text) f Fe(MPI)p 1602 2532 V 17 w(ALL)g Fm(need)h(not)75 2589 y(b)q(e)g(passed)f (as)g(a)g(parameter)f(when)i(the)f(library)h(is)g(in)o(v)o(ok)o(ed.)166 2647 y(Case)g(2:)22 b(The)17 b(library)g(is)g(alw)o(a)o(ys)e(in)o(v)o(ok)o (ed)i(in)g(a)f(unique)i Fh(libr)n(ary)f(c)n(al)r(ling)f(c)n(ontext)g Fm(on)h(eac)o(h)f(pro-)75 2704 y(cess.)29 b(Then)18 b(a)g(library)h (initialization)i(routine)e(should)g(b)q(e)g(in)o(v)o(ok)o(ed)f(b)o(y)g(the)g (user)g(on)h(eac)o(h)f(pro)q(cess)p eop %%Page: 9 10 bop 75 -100 a Ff(1.4.)34 b(EXAMPLES)16 b({)e(PR)o(OBLEMS)j(IN)e(RICK'S)h (LIST)751 b Fm(9)75 45 y(where)16 b(the)g(library)g(ma)o(y)f(b)q(e)h(in)o(v)o (ok)o(ed;)g(the)f(initialization)k(routine)d(is)g(called)h(after)e(the)h (user)g(de\014ned)75 102 y(the)i(library)g(calling)i(con)o(texts)d(and)h(b)q (efore)g(the)g(library)g(is)g(in)o(v)o(ok)o(ed.)28 b(Eac)o(h)18 b(initialization)i(call)f(is)f(a)75 158 y(collectiv)o(e)d(call)f(within)g (the)f(library)h(calling)g(con)o(text.)19 b(A)13 b(cop)o(y)g(of)f(the)h (calling)i(con)o(text)d(is)h(created)g(\(b)o(y)75 214 y Fe(MPI)p 150 214 15 2 v 17 w(COPY)p 263 214 V 16 w(CONTEXT)p Fm(\))i(and)i(stored)e (in)j(a)d(static)h(library)h(v)m(ariable.)25 b(Subsequen)o(t)17 b(calls)g(to)f(the)g(library)75 271 y(need)g(not)f(pass)g(the)g(calling)i (con)o(text)e(as)g(a)f(parameter.)166 327 y(Case)19 b(3:)27 b(The)20 b(library)g(ma)o(y)e(b)q(e)i(in)o(v)o(ok)o(ed)f(on)h(eac)o(h)f(pro)q (cess)g(within)i(a)d(\014xed)i(\(small\))g(n)o(um)o(b)q(er)75 384 y(of)f(library)h(calling)h(con)o(texts.)32 b(Copies)20 b(of)f(these)g(con)o(texts)g(can)h(b)q(e)g(created)f(b)q(efore)h(the)f (library)h(is)75 440 y(in)o(v)o(ok)o(ed)e(and)g(\\cac)o(hed")g(in)h(static)f (v)m(ariables.)29 b(Subsequen)o(t)19 b(in)o(v)o(ok)m(ations)f(to)g(the)g (library)h(need)f(not)75 497 y(create)d(new)g(copies,)h(but)f(only)h(select)g (the)f(righ)o(t)g(preexisting)i(cop)o(y)l(.)166 547 y Fc(This)9 b(assumes)h(one)f(can)h(test)g(con)o(texts)h(\(con)o(text)f(handles\))g(for)f (equalit)o(y)m(.)15 b(This)9 b(should)g(b)q(e)h(said)g(explicitely)75 596 y(in)j(the)i(draft)75 704 y Fk(Use)i(of)h(con)o(texts)f(for)g(a)h(host)f (no)q(de)h(computation)i(mo)q(del)42 b Fc(Let's)14 b(assume)g(t)o(w)o(o)f (con)o(texts:)145 779 y Fa(\017)23 b Fb(MPI)p 258 779 14 2 v 15 w(ALL)13 b Fc(with)g(host)i(b)q(eing)e(no)q(de)i(zero)145 854 y Fa(\017)23 b Fb(MPI)p 258 854 V 15 w(NODE)13 b Fc(that)h(do)q(es)g(not) g(include)g(the)h(host)166 928 y(W)m(e)e(assume)h(the)g(load)f(pro)q(cedure)j (ensures)g(that)e(host)g(is)g(pro)q(cess)i(zero)e(in)g(ALL)75 1078 y Fb(host/node)20 b(code)479 b(translation)75 1128 y(--------------)e (-------------)75 1227 y(I_am_the_host\(\))455 b(\(mpi_rank\(MPI_A)o(LL,)19 b(rank\);)860 1277 y(rank==0;\))75 1377 y(form)i(node)g(group)457 b(mpi_rank\(MPI_AL)o(L,)19 b(task\))860 1427 y(mpi_split_conte)o(xt\(MP)o (I_NOD)o(E,)g(MPI_ALL,)903 1476 y(\(task==0\),0\);)75 1576 y(Broadcast)h(from)h(host)g(to)g(nodes)174 b(mpi_bcast\(MPI_A)o(LL,0,)o (...\))75 1676 y(regular)20 b(communications)f(btwn)196 b(mpi_send\(buffer)o (,len,)o(dest,)o(tag,M)o(PI_NO)o(DE\))75 1725 y(nodes)675 b(mpi_recv\(buffer) o(,len,)o(sourc)o(e,tag)o(,MPI_)o(NODE)o(\))184 1775 y(/*)21 b(source,)g(dest)g(are)g(ranging)f(from)h(0)h(to)f(#nodes-1)f(*/)75 1875 y(sum)h(values)g(from)g(each)g(node)239 b(mpi_reduce\(inbu)o(f,out)o (buf,l)o(en,MP)o(I_ALL)o(,)75 1925 y(at)21 b(host)893 b(0,MPI_ISUM\))228 1975 y(/*)21 b(host)g(\(node)g(zero\))f(calls)h(mpi_reduce)f(with)h(inbuf)g (=)g(0)h(*/)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-context@CS.UTK.EDU Wed Jun 9 17:32:25 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA20612; Wed, 9 Jun 93 17:32:25 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02043; Wed, 9 Jun 93 17:32:15 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 9 Jun 1993 17:32:14 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02035; Wed, 9 Jun 93 17:32:13 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA16583; Wed, 9 Jun 93 16:31:59 CDT Date: Wed, 9 Jun 93 16:31:59 CDT From: Tony Skjellum Message-Id: <9306092131.AA16583@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: June 22 context meeting Gentlemen, I will arrive on June 21, and hope to see as many of the context subcommittee members for pre-meeting discussion on June 22, at the Bristol Suites. I am writing this so everyone on the sub-committee will know that we are trying to roll up our sleeves again, this time, pre-meeting, to be sure we're on track by the time the meeting starts. -Tony From owner-mpi-context@CS.UTK.EDU Thu Jun 10 06:48:29 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA21785; Thu, 10 Jun 93 06:48:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25168; Thu, 10 Jun 93 06:48:33 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 10 Jun 1993 06:48:32 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25160; Thu, 10 Jun 93 06:48:29 -0400 Date: Thu, 10 Jun 93 11:48:07 BST Message-Id: <21723.9306101048@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: June 22 context meeting To: Tony Skjellum , mpi-context@cs.utk.edu In-Reply-To: Tony Skjellum's message of Wed, 9 Jun 93 16:31:59 CDT Reply-To: lyndon@epcc.ed.ac.uk I will arrive evening of June 21. > Gentlemen, I will arrive on June 21, and hope to see as many of the context > subcommittee members for pre-meeting discussion on June 22, at the Bristol Suites. > I am writing this so everyone on the sub-committee will know that we are trying > to roll up our sleeves again, this time, pre-meeting, to be sure we're on track > by the time the meeting starts. > -Tony > /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Jun 10 08:38:38 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA22177; Thu, 10 Jun 93 08:38:38 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01083; Thu, 10 Jun 93 08:35:56 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 10 Jun 1993 08:35:55 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01075; Thu, 10 Jun 93 08:35:54 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA18321; Thu, 10 Jun 1993 08:35:55 -0400 Date: Thu, 10 Jun 1993 08:35:55 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9306101235.AA18321@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu Subject: Next MPI meeting Unfortunately I won't be at the next MPI meeting, which is why I'm keen to understand the current context ideas and proposal(s) now. I've not had much feedback on the examples I sent out a couple of days ago. I'm particularly keen to get a correct version of the convolution example. I think examples are important if we are to make others understand a fairly complex proposal. They also help us understand what is really needed. Here's a question that I've asked myself, and which I'm sure others will ask. Are naked contexts really necessary? Communicators provide the scope of a communication operation. Why would an application writer use mpi_alloc_context and mpi_make_comm, rather than mpi_safemake_comm? David From owner-mpi-context@CS.UTK.EDU Thu Jun 10 11:02:58 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA22663; Thu, 10 Jun 93 11:02:58 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10074; Thu, 10 Jun 93 11:03:09 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 10 Jun 1993 11:03:08 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09781; Thu, 10 Jun 93 11:00:02 -0400 Date: Thu, 10 Jun 93 15:59:53 BST Message-Id: <22013.9306101459@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Next MPI Meeting (etc) To: walker@rios2.epm.ornl.gov (David Walker), lyndon@epcc.ed.ac.uk In-Reply-To: David Walker's message of Wed, 9 Jun 1993 08:50:17 -0400 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Hi David Seems to be just me and thee on the contexts and examples front at the moment. I guess people are just too busy. > A more complicated example based on the convolution example could > use overlapping groups. Assume for example that the parallel FFT > requires the number of processes to be a power of two, and we have > a total of 12 processes. Then we could put 0,1,2,..,7 in one group > and 4,5,6,...,11 in the other group. I'm not sure how much this > complicates things. I was thinking, in terms of this example, of something like having arbitrarily different distributions and numbers of nodes in each of the two groups. This would introduce an arbitrary communication pattern between the two groups making sender choice more important to the example. I was really thinking of abritrarily different computations in two or more groups. Trouble is, its real hard to write a complete yet sshort example - conceptually simple is easy, small simple code is not so easy. > Sorry, in my corrected version there was still an extraneous > mpi_safemake_comm inside the first conditional branch. This has > been corrected in the revised version below. Thanks. > The arguments to > my version of mpi_safemake_comm are: > > mpi_safemake_comm (local_group, remote_group, communicator) > IN local_group > IN remote_group > OUT communicator Well of course I support this kind of communicator constructor, as my previous messages make obvious. I guess the point is that this is *not* the kind of constructor in the working document from Marc Snir - this is something that Marc identified as left out for now. Further this is *not* the sort of thing that Tony/Mark were proposing to include in the working document - from what I can gather. Would you like me to type in explanation of what I think is the kind of thing that they are suggesting? Would that help anyone? > In the sends the rank of the destination process is relative to the > remote group, i.e., the receiving group. In the receives the rank > of the source process is relative to the remote group, i.e., the > sending group. This seems the natural way to do things. Yes this is the natural way to do communication between different groupings, of course, in my opinion. > As you > point out, the source rank in the receives can be replace by > MPI_DONTCARE. Yes, in the example given, because the communication is a particular pattern - i.e. one process in A group sends to one process in B group and vica versa. > It's not obvious to me why a C2 receive can't > specify a particular source in the remote group. Please explain > this to me. The comment I made was a result of confusing your example with my understanding of the working document and what Tony/Mark are suggesting. In particular the way in which your example assumes to use publish/subscribe is quite incompatible with what I think is meant my Tony/Mark. Are you aware of this? I can send an explanation of what I believe thes guys intend, if that will help. (Frankly, it will mean that you can write your example program in a half-sensible way, but more complicated examples like those I suggested above cannot be written in any sensible way.) I should make a general point about your example. When building the communicator for intercommunication you are using the init_group and the fft_group. Now the init_group is a superset of the fft_group, in fact it encloses the two different fft_group's. This is a rather strange thing to be doing since it means that calculation of the destination rank in the other group refers to the enclosing group. It is *much* more natural and expressive to think of the destination rank in the destination group, so that a better thing to do is to make a communicator with the local group being the fft_group and the remote group being the other (remote) fft_group. Would you like me to adapt your example program to this way of working? It will be easy for me to do - of course I will be inventing definitions of some things like publish and subscribe. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-context@CS.UTK.EDU Thu Jun 10 12:02:54 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-UTK) id AA22952; Thu, 10 Jun 93 12:02:54 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14511; Thu, 10 Jun 93 12:03:09 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 10 Jun 1993 12:03:08 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14503; Thu, 10 Jun 93 12:03:04 -0400 Date: Thu, 10 Jun 93 17:03:00 BST Message-Id: <22074.9306101603@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Next MPI meeting To: walker@rios2.epm.ornl.gov (David Walker), mpi-context@cs.utk.edu In-Reply-To: David Walker's message of Thu, 10 Jun 1993 08:35:55 -0400 Reply-To: lyndon@epcc.ed.ac.uk Dear David > I think examples > are important if we are to make others understand a fairly complex proposal. I agree. I fear that we may need complex examples, and people will have time to study them. > They also help us understand what is really needed. > I agree again, and have the same fear - i.e. that whereas the examples can be conceptually simple the example code cannot be. > Here's a question that I've asked myself, and which I'm sure others will > ask. Are naked contexts really necessary? Communicators provide the scope > of a communication operation. Why would an application writer use > mpi_alloc_context and mpi_make_comm, rather than mpi_safemake_comm? > Naked groups allow the user to perform group manipulations on groups. This is conceptually cleaner than doing it on communicators, in my humble opinion. The context-free :-) communicator constructors synchronise the processes invloved in the communicator construction. The context-oriented constructors do not. Naked contexts allow preallocation of contexts and avoidance of synchronisation for communicator construction. I think that a modification of your example program in line with a comment about destination rank addressing will reveal another use for naked context. I'll have to redefine publish/subscribe a little, but I dont see this as a problem as the example redefines them relative to my understanding of the intentions of Tony and Mark (Sears). I'll send that modification after this message. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ >From owner-mpi-context@CS.UTK.EDU Tue May 11 21:56:56 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA22309; Tue, 11 May 93 21:56:54 -0500 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA01509; Tue, 11 May 93 21:58:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15824; Tue, 11 May 93 22:52:51 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Tue, 11 May 1993 22:52:50 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15806; Tue, 11 May 93 22:52:47 -0400 Received: from anasazi.cs.sandia.gov.noname ([132.175.76.10]) by cs.sandia.gov (4.1/SMI-4.1) id AA26565; Tue, 11 May 93 20:52:45 MDT Received: by anasazi.cs.sandia.gov.noname (4.1/SMI-4.1) id AA13505; Tue, 11 May 93 20:52:44 MDT Date: Tue, 11 May 93 20:52:44 MDT From: mpsears@cs.sandia.gov (Mark P. Sears) Message-Id: <9305120252.AA13505@anasazi.cs.sandia.gov.noname> To: mpi-context@cs.utk.edu Subject: Proposal VIII -- response to Riks probs. Status: R The following is my response to Rik Littlefields problem sets. I will bring copies to the meeting. Mark Sears Sandia National Laboratories Comments on Rik Littlefields problems: 1) (describe a circular shift routine) Here I give a contiguous byte version only. Extensions to other data types and to non contigous buffers is straightforward. Error handling is left out, and I use an aggressive style of message passing (send before receive). Note that tag must be provided in addition to context. // MPI_CSHIFT_B -- shift a contiguous buffer around a group in circular order // // limitations: // uses aggressive style of message passing. // no check for len being same on all processes. // // assumptions: // MPI_MYNODE() returns process id // MOD(a,b) = a modulo b // SEND(buf, len, dest, tag, context) sends buffer to destination. // RECV(buf, len, src, tag, context) receives buffer from source. // void MPI_CSHIFT_B ( char *in, char *out, int len, int shift, MPIGroup group, int context, int tag) { int myrank; // rank of this process in the group int left, right; // process ids for sending and receiving // exit routine if this process is not a group member: if(!MPI_groupiselement(group, MPI_MYNODE())) return; // get my rank in group and compute left and right neighbors myrank = MPI_grouprank(group, MPI_MYNODE()); left = MPI_groupelement(group, MOD((myrank - shift), MPI_grouporder(group))); right = MPI_groupelement(group, MOD((myrank + shift), MPI_grouporder(group))); // now execute the shift: SEND(in, len, right, tag, context); RECV(out, len, left, tag, context); } 2) (bflyexchange) // MPI_BFLY_B -- butterfly exchange // // limitations: // uses aggressive style of message passing. // no check for len being same on all processes. // // assumptions: // MPI_MYNODE() returns process id // MOD(a,b) = a modulo b // SEND(buf, len, dest, tag, context) sends buffer to destination. // RECV(buf, len, src, tag, context) receives buffer from source. // void MPI_BFLY_B ( char *in, char *out, int len, int shift, MPIGroup group, int context, int tag) { int myrank; // rank of this process in the group int partner; // process id for exchanging int j; // exit routine if this process is not a group member: if(!MPI_groupiselement(group, MPI_MYNODE())) return; // get my rank in group and compute partner for exchange: myrank = MPI_grouprank(group, MPI_MYNODE()); j = (myrank % (shift<<1))/shift; partner = MPI_groupelement(group, myrank + shift(1-2*j)); // (the problem statement was a little unclear about what to do if // the exchange partner was not in the group. Here I just assume // that it is in the group.) // now execute the exchange: SEND(in, len, partner, tag, context); RECV(out, len, partner, tag, context); } Discussion: Safety: This routine (and other similarly implemented global operations) are safe if no process in the group has a send or receive using the same context and tag. The routine is unsafe if a process in the group issues another send outside the routine to its partner in the dual exchange (using the same tag), or if a process posts a wildcard receive prior to calling this routine which would match the incoming message. Thus, the safety of this routine is the same as the safety associated with normal point to point message passing. Performance: Performance of this routine acheives the limit of one send/receive per process. The computational overhead of the group operations will depend on the type of group used, but with use of code inlining and for simple kinds of groups this overhead will be moderate. Note that group operations require no communications, since a group is defined algorithmically. 2) Guidelines for library developers. There are two possible models for developing library or third party software which uses MPI, but which is in turn intended for use by other independently developed software. The first kind uses context and tags provided to it by the calling software, and is typical of communication services. An example of such software that would not be provided by MPI would be a matrix transpose. Writing software of this type is similar to current practice, but the author must avoid using tags and context for which permission has not been granted. One possibility for the author is to simply specify a context in the interface and make the explicitly declared assumption that tags within that context are usable in any way within the library module. An author of multiple library modules that need to work together can develop an internal scheme for tag management which allows the use of minimum numbers of contexts. The second kind of library software establishes and manages its own context and tag spaces. This kind of software is typical of higher level operations which provide computational functions (which implicitly use communications). There are a number of ways for a library to do this management. For example, to begin with a library module might make use of static contexts and assume it has only one 'instantiation' within a program. More advanced libraries might bind context to data objects that are being manipulated and allow multiple simultaneous use of the library code. In both cases, a primary source of problems will be receives where the tag filter is wild-carded. Such receives should be used with care, and independent context values associated with each. A secondary source of problems will be collective communication operations among groups which intersect. Again, such operations can be made non-interfering by choosing different contexts for each. 3) host-node programming The description of this problem makes some assumptions that I don't agree with. First is that the ALL group contains the host. My opinion is that there might be two preexisting groups, the ALL group containing all the nodes, and the ALLH group containing all the nodes and the host process. Here I make the assumption that in the ALLH group the host is the last process. To make the problem interesting, the program uses a static context. These groups can be built as follows, assuming constructors for building linear groups and list groups: // build the ALL group MPIGroup ALL; // all processes MPIGroup ALLH; // all processes plus host ... // Build ALL, ALLH groups // // Assume // MPI_HOSTID() -- returns id of host // MPI_NNODES() -- returns number of node processes // MPI_makelineargroup(int order, int start, int stride) // -- constructor for linear group // MPI_makelistgroup(int order, int *list) // -- constructor for list group // void MPIBuildAlls() { int *list; int n; int i; n = MPI_NNODES(); ALL = MPI_makelineargroup( /* order: */ n, /* start */ 0, /* delta */ 1); list = (int *) malloc(sizeof(int) * (n + 1)) for(i=0;i MPIGroup hgroup; // group containing host only MPIGroup ngroup; // group containing all nodes static int me; static int hostid; static int context = 100; // a randomly chosen context static int tags = 13367; // a randomly chosen tag base main() { me = MPI_NODE(); hostid = MPI_HOSTID(); MPI_opencontext(context); // use a static context if(me == hostid) host(); else node(); MPI_closecontext(context); // context is no longer in use } host() { int myrank; // build host, node groups: hgroup = MPI_makelistgroup(1, &hostid); ngroup = ALL; // lets not do more work than necessary // compute rank of host in ALLH group myrank = MPI_grouprank(ALLH, hostid); // broadcast from host to all nodes (perfectly legitimate) MPI_BCAST( ..., myrank, ALLH, tags, context); // send a message to each node in turn for(i=0;iFrom weeks@mozart.convex.com Wed Jun 9 14:53:36 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA14989; Wed, 9 Jun 93 14:53:35 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12451; Wed, 9 Jun 93 14:53:05 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01222; Wed, 9 Jun 93 07:51:39 -0500 Received: by mozart.convex.com (5.64/1.28) id AA26626; Wed, 9 Jun 93 07:53:48 -0500 Date: Wed, 9 Jun 93 07:53:48 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091253.AA26626@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 12 mpi mail #2 Status: R LaTeX source (sender forgotten: I deleted the verbiage before the LaTeX stuff so I could just print it.. I think it was sent by Steve Otto) DW --------------------------- cut here ---------------------------------------- \documentstyle[twoside,11pt]{report} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface} \author{Scott Berryman, {\em Yale Univ} \\ James Cownie, {\em Meiko Ltd} \\ Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\ Al Geist, {\em ORNL} \\ Bill Gropp, {\em ANL} \\ Rolf Hempel, {\em GMD} \\ Bob Knighten, {\em Intel} \\ Rusty Lusk, {\em ANL} \\ Steve Otto, {\em Oregon Graduate Inst} \\ Tony Skjellum, {\em Missisippi State Univ} \\ Marc Snir, {\em IBM T. J. Watson} \\ David Walker, {\em ORNL} \\ Steve Zenith, {\em Kuck \& Associates} } \date{May 10, 1993 \\ This work was supported by ARPA and NSF under contract number \#\#\#, by the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615. } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \label{sec:context} %======================================================================% \chapter{Contexts -- Proposal X} \begin{center} {\em Lyndon J.~Clarke, Tony Skjellum, Rik Littlefield}\medskip \end{center} This chapter describes the definition of context, its relation to group and tag, and describes communication safety, while indicating how all this relates to point-to-point and global communication. We include examples that illustrate the features incorporated, including the ``Rik Littlefield'' examples, plus the logical grid topology context example of Skjellum. We have utilized C-language semantics to describe the functionality, whereas other authors have utilized Fortran semantics. This should not be construed to denigrate (or render impossible) a Fortran interface. % %---------------------------------------------------------------------- % Context, Tag, Group and Process \section{Context, Tag, Group and Process} This section describes the view of context, tag, group and process taken by MPI. MPI views group and context as different and unrelated entities. Group is used for management of processes and context is used for management of messages. The purpose of these definitions are to promote the generation of library modules, and to allow users safely to combine programs written at different times, or by different programmers, without globalizing the semantics of how messaging is done. To do this, we introduce scope to message passing. This is accomplished through contexts and groups, primarily, and then by tags, within the user's perview. Note that we take a {\em laissez faire\/} view of the context resource (in the same spirit as Unix ports), in that we provide an resource allocation mechanism, but also provide for correct programs that define their own ``well-known'' contexts. Depending on implementation, either ``well-known'' or system-allocated contexts might be of higher performance, or the first allocated might always be faster, depending on the implementation details. (In doing so, we have removed a major incompatibility of this MPI formulation with the alternate Proposal VIII (Mark Sears)). We do not view groups, contexts, and tags as totally ``orthogonal,'' but we seek to make them orthogonal as possible, consistent with providing a safe, optimizable environment for programming. Since key events in the formation of groups may require unique contexts for their creation, and because contexts are often naturally associated with groups, their is a tendency to link context heavily with group. Actually, we have found that the group-to-context relationship is not one to one, both based on experience in experimental systems, and in recent discussions. Except where needed, contexts and groups are separate concepts, but we build up communicator objects in the following, that intentionally bind them together for programming convenience. In the end, this formulation remains at odds with Proposal I (Marc Snir), which uses the word ``context'' in a semantically distinct sense. Here, context always means a logical partition of the tag space into a user-defined and system-defined space. The latter is subsequently allocated/deallocated under either a system- or user-controlled safety mechanism, or a combination of both. Though we permit contexts to be integer values, and hence transmittable like tags, the operating system may choose internal optimizations by hashing such contexts to make reference to hardware message tags, or other facilities that we cannot anticipate in a portable standard. These runtime optimizations are not excluded. Furthermore, in what follows, we note at times that simplifications occur for the SPMD model. These simplifications imply a simpler implementation, and less potential overhead for that programming model, while including (rather than excluding) more general programming in the MPMD model. MPI libraries for SPMD and MPMD may be appropriately optimized at compile-time and/or run-time by the implementor. \subsection{Tag and Context} \subsubsection{Tag} MPI defines a {\em tag\/} as the familiar non-negative integer label of a message. The tag is a field in message selection which may be wildcard. There is no concept of creation or deletion of tags. Any non-negative integer value may be used as a tag. The MPI user is responsible for management of tag usage. It is correct for a process {\cal P} to transmit a tag value T to another process {\cal Q}, and, subsequently, for {\cal P} to then send messages of tag T, and for {\cal Q} to then receive messages of tag T. A tag is a 32-bit integer. Users assign non-negative tag values. {\small {\bf Discussion:} This definition appears compatible with the point-to-point definition of tag, at present. We will introduce an amendment to tag matching to point-to-point, to allow bit selection (which was narrowly defeated last time, and justify that better this time). Other than that, tag definition is in no apparent conflict with point-to-point.} \subsubsection{Context} MPI defines a {\em context\/} as the familiar integer label of a distinct message space, or more formally as a distinct message tag space. The context is a field in message selection that may not be wildcarded. There is no concept of creation or deletion of contexts. Key features are as follows: \begin{itemize} \item A context is exactly analogous to a tag, except that it may not be wildcarded in selection. \item In MPI, a context is stored in a 32-bit integer. Implementations that facilitate context management need not span the full 32 bits. Users are constrained to use non-negative contexts. System-provided contexts (see below), will be negative. \end{itemize} The MPI user is responsible for the management of context usage. Any non-negative integer value may be used as a context. It is correct for a process {\cal P} to transmit a context value {\bf C} to another process {\cal Q}, and, subsequently, for {\cal P} to send and receive messages of context {\bf C}, and for {\cal Q} to send and receive messages of context C. {\small {\bf Discussion:} A switch to 16-bit contexts would be a reasonable implementation alternative; we are convinced that a compliant implementation should permit at least 16 bits of context. Because these numbers are essentially a hash to hardware resource, in some implementations, 16 bits may provide nominal performance improvements. It is also thought that 16 bits would provide a sufficiently large number of contexts for practical applications, as currently envisaged, even those that have a number of objects. We strongly oppose less than 16 bits, because they could be insufficient for reasonable applications. We suggest the natural symmetry of tag and context sizes may simplify alignment in implementations, and reduce the number of data types in MPI by one. Again, we view the MPI contexts are software contexts; the mapping to ``hardware contexts'' is an implementation issue; there may only be an infinitesmal number of such hardware contexts. Still, in analogy to register variables in a C program, a program continues to compile and run after the fast register resource is exhausted\ldots one suggests to the implementor that at least one context support hooks to a slower, but bigger internal context map.} \subsubsection{Context Management} MPI provides services that facilitate context management and recommends, although does not mandate, use of these services. MPI provides a context allocation and deallocation facility that allows user processes to generate unique context values. \begin{verbatim} int context_array[]; /* OUT */ int number_to_alloc; /* IN */ mpi_alloc_contexts(context_array, number_to_alloc) \end{verbatim} This procedure allocates \verb|number_to_alloc| unique context values and and stores them in \verb|context_array|. None of the context values allocated have been returned by a previous call to the procedure in any process unless such values have also previously been deallocated by calling \verb|mpi_free_contexts|. \begin{verbatim} int context_array[]; /* IN */ int number_to_free; /* IN */ mpi_free_contexts(context_array, number_to_free) \end{verbatim} This procedure deallocates \verb|number_to_free| unique context values stored in the array \verb|context_array|. Each context value must have previously been allocated by calling \verb|mpi_alloc_contexts| or else the program is erroneous. The context values deallocated may be allocated with future calls of \verb|mpi_alloc_contexts|. {\small {\bf Discussion:} Allocation and deallocation of contexts involves a global synchronization in the SPMD model, and the possibility of a context server in the full MPMD programming model. The degree of implementation difficulty depends on which model the programmer is using. SPMD programming involves no server. Thus, extra overhead (however minimal) is removed from the SPMD model. An appropriate MPI implementation could provide an SPMD and MPMD version to facilitate this optimization, or via runtime checking.} {\small {\bf Discussion:} We allow the user to write programs with complicated and possibly unpredictable behavior by allocating a context C in one process P and sending the context to another process Q which frees the context. P continues to use context C while a third process R allocates C for another purpose. Therefore, the user is expected to maintain safety in such cases, through systematic programming. Programs that do not maintain their own context coherence after \verb|mpi_free_contexts| are said to be erroneous.} MPI permits correct programs to include a mixture of code that do their own allocation of contexts with code that utilizes the automatic context registration mechanisms. The ``fastest'' contexts are implementation-defined, and implementations may provide hints as to which contexts are faster, or undertake heuristics to utilize faster contexts before slower contexts. {\small {\bf Discussion:} Selection of a range of system- and user-allocated contexts, as opposed to positive/negative (equi-partition of bits) would be viewed as a friendly amendment, if this further articulation is seen as sufficiently important to merit the additional mechanisms needed to implement it. The functionality is equivalent, but the latter form may be more flexible for implementors, and more useful to users. This issue could be remanded to the Environment Subcommittee.} \subsubsection{Context Global Association} MPI provides name services which allow the user to make global associations of names with contexts and locate contexts by associated names. \begin{verbatim} char *name; /* IN */ int context; /* IN */ mpi_associate_context(name, context) \end{verbatim} This procedure associates \verb|name| with context value \verb|context|. The procedure fails if \verb|name| is associated with any context value. \begin{verbatim} char *name; /* IN */ int context; /* IN */ mpi_dissociate_context(name, context) \end{verbatim} This procedure dissociates \verb|name| from the context value \verb|context|. The procedure fails if \verb|name| is not associated with the context value \verb|context|. \begin{verbatim} char *name; /* IN */ int context; /* OUT */ int wait; /* IN */ mpi_locate_context(name, &context, wait) \end{verbatim} This procedure locates the context value \verb|context| associated with \verb|name|. If \verb|name| is not associated with a context value then the behavior is determined by the logical value of \verb|wait|. If \verb|wait| is \verb|true| then the procedure waits until \verb|name| becomes associated with a context value. If \verb|wait| is \verb|false| the procedure returns \verb|MPI_NULL_CONTEXT| in \verb|context|. \subsection{Group and Process} \subsubsection{Process} MPI views a process in the familiar sense, for example as a Unix or Intel NX process. MPI defines a {\em process identifier\/} (pid for short) as a process-local integer entity that refers to an opaque process description object of undefined size. This means that if two processes {\cal P} and {\cal Q} both know the identifier of a process {\cal R} then the relationship between the values of the identifiers known to {\cal P} and {\cal Q} is undefined. \begin{small} {\bf Discussion:} This pid definition leaves freedom for the implementor to do whatever is efficient for the machine(s) at hand, and does not exclude implementations in which the pid is global. \end{small} \subsubsection{Group Definition} MPI defines a group as an ordered collection of processes, or more formally as an ordered collection of pids. MPI defines a {\em group identifier\/} (gid for short) as a process-local integer entity that refers to an opaque group description object of undefined size. This means that if two processes {\cal P} and {\cal Q} both know the identifier of a group {\bf G} then the relationship between the values of the identifiers known to {\cal P} and {\cal Q} is undefined. This MPI formulation defines a pid as formally the identifier of a singleton group containing only the process. \subsubsection{Group Services} MPI provides services that allow the user to create and delete groups. \begin{verbatim} int new_group; /* OUT */ int old_group; /* IN */ int subset_key; /* IN */ mpi_subset_group(&new_group, old_group, subset_key) \end{verbatim} This procedure creates one or more new groups as subsets of an existing group according to a subset key. The procedure synchronizes the member processes of the existing group. \begin{verbatim} mpi_permute_group(&new_group, old_group, permute_rank) \end{verbatim} This procedure creates one new group by permuting the member ranks of an existing group. The procedure synchronizes the member processes of the existing group. \begin{verbatim} int new_group; /* OUT */ int old_group_array[]; /* IN */ int number_of_old_group; /* IN */ mpi_union_group(&new_group, old_group_array, number_of_old_group) \end{verbatim} This procedure creates one new group by a union of one or more distinct existing groups. The procedure synchronizes the member processes of the union. Note in particular that each existing group can of course be the singleton group of a process. \begin{verbatim} mpi_delete_group(group) \end{verbatim} This procedure deletes a group. The procedure synchronizes the member processes of the group. \begin{small} {\bf Discussion: } We have concluded that at least one default context is needed, safely to implement the above group creation, deletion, and union operations. This context may be a hidden context, per the implementation. It is at this point that group and context become non-orthogonal, because a safe messaging mechanism must be supported, deterministically to accomplish these operations. The communicators described below reveal context and group merger explicitly to the user. The above operations do not. \end{small} \subsubsection{Group Transmission} MPI provides services which allow the user to write a portable group description into a memory buffer and read a portable group description from a memory buffer. The memory buffer can be transmitted in a message, allowing transmission of group descriptions. \begin{verbatim} char *buf_ptr; /* OUT */ int buf_len; /* IN */ int group; /* IN */ mpi_passivate_group(group, buf_ptr, buf_len) \end{verbatim} This procedure writes a portable description of \verb|group| into the buffer \verb|buf_ptr| of length \verb|buf_len| bytes. \begin{verbatim} char *buf_ptr; /* OUT */ int buf_len; /* IN */ int group; /* OUT */ mpi_activate_group(&group, buf_ptr, buf_len) \end{verbatim} This procedure reads a portable description of a group from the buffer \verb|buf_ptr| of length \verb|buf_len| bytes and returns a group descriptor identifier for the read group in \verb|group|. MPI provides name services that allow the user to make global associations of names with groups and locate groups by associated names. \begin{verbatim} char *name; /* IN */ int group; /* IN */ mpi_associate_group(name, group) \end{verbatim} This procedure associates \verb|name| with group identifier by \verb|group|. The procedure fails if \verb|name| is associated with any group. \begin{verbatim} char *name; /* IN */ int group; /* IN */ mpi_dissociate_group(name, group) \end{verbatim} This procedure dissociates \verb|name| from the group identified by \verb|group|. The procedure fails if \verb|name| is not associated with the group identified by \verb|group|. \begin{verbatim} char *name; /* IN */ int group; /* OUT */ int wait; /* IN */ mpi_locate_group(name, &group, wait) \end{verbatim} This procedure locates the group associated with \verb|name| and returns a group identifier for the located group in \verb|group|. If \verb|name| is not associated with a context value then the behavior is determined by the logical value of \verb|wait|. If \verb|wait| is \verb|true| then the procedure waits until \verb|name| becomes associated with a group. If \verb|wait| is \verb|false| the procedure returns \verb|MPI_NULL_GROUP| in \verb|group|. \begin{small} {\bf Discussion:} When a process receives a group description either by means of \verb|mpi_locate_group| or \verb|mpi_activate_group|, it may later delete that group using \verb|mpi_delete_group|. If so, then the group deletion should simply delete the received copy of the group description rather than the actual group. \end{small} \subsubsection{Default Groups} MPI provides each process with two predefined groups, which cannot be deleted. These are the singleton group of which the process is the only member and a single group of which all processes are a member. \begin{verbatim} int pid; mpi_my_pid(&pid) \end{verbatim} This process returns the singleton group identifier, that is, process identifier, of the calling process. \begin{verbatim} int gid; mpi_all_group(&gid) \end{verbatim} This process returns the group identifier of the group of all processes. \begin{small} {\bf Discussion:} The concept of the ``all'' group is not really extensible to a dynamic process world. We could replace this with an ``initial'' group for example. It might be more general to replace the all group by the following. Creation of the set of processes composing the program is implementation specific. Require an implementation specific method of assigning each process to a {\em base\/} process group, with different and distinct groups of processes in different base groups. Default is to make all processes members of the same base group. Provide a procedure to return the base group of the process. \end{small} %---------------------------------------------------------------------- % Communicator and Communication \section{Communicator and Communication} Now that we have established the fundamental workings of tag, context, and group, we can turn to the standard user-level communication objects of MPI. In particular, communication between processes is provided by communicator objects that are a binding of a message context and one or more process ``worlds.'' A process world is either a process group or all processes composing the user program. \subsection{Intra-communication} Intra-communication is communication between processes within the same process world. This service is provided by an intra-communicator object, which is a binding of a context, and a process world. The process world is either a group or all processes composing the user program. \begin{verbatim} int communicator; /* OUT */ int context; /* IN */ int world; /* IN */ mpi_create_intra_communicator(&communicator, context, world) \end{verbatim} This procedure creates an intra-communicator and returns the communicator identifier in \verb|communicator|. The intra-communicator world supplied in \verb|world| may be an actual group identifier or the null group \verb|MPI_NULL_GROUP|. If \verb|world| is an actual group identifier, then the world is the identified group, and processes within the world are identified relatively by {\bf rank} within that group. Otherwise the world contains all processes within the user program and processes within the world are identified absolutely by process identifier. The communicator context supplied in \verb|context| may be an actual context value or the null context \verb|MPI_NULL_CONTEXT|. If \verb|context| is an actual context then the procedure completes asynchronously. If \verb|context| is the null context then the procedure allocates a single context using the context allocator described above, synchronizes all members of the process world, and uses the allocated context in each intra-communicator created. \begin{small} {\bf Discussion:} The name ``intra-communicator'' is long. We propose ``C1'' as the nick-name for an intra-communicator object. {\bf Discussion:} We conserve the Proposal I rank-mechanism for referring to processes in groups, for most cases (see above), and under communication primitives. We see this as the most easily understood, intuitive way to describe communication relative to the group; the level of virtualization implied by this mapping is also desirable from the point of view of safety. \end{small} \begin{verbatim} int communicator; /* IN */ int world; /* OUT */ mpi_intra_communicator_world(communicator, &world) \end{verbatim} This procedure returns in \verb|world| the process world of the intra-communicator identified by \verb|communicator|. \begin{verbatim} int communicator; /* IN */ int context; /* OUT */ mpi_communicator_context(communicator, &context) \end{verbatim} This procedure returns in \verb|context| the context of a communicator object identified by \verb|communicator|. If the context supplied in communicator creation was the null context, then the procedure returns the context allocated during communicator creation. \begin{verbatim} int communicator; /* IN */ mpi_delete_communicator(communicator) \end{verbatim} This procedure deletes a communicator object. If the context supplied in communicator creation was the null context then this procedure synchronizes the process members of process world(s) of the communicator and deallocates the context allocated for the communicator. Otherwise the procedure completes asynchronously. \subsection{Inter-communication} Intercommunication is communication between processes within two different process worlds. This service is provided by an inter-communicator object, which is a binding of a context, a local process world, and a remote process world. Each process world is either a group or all processes composing the user program. \begin{verbatim} int communicator; /* OUT */ int context; /* IN */ int local; /* IN */ int remote; /* IN */ mpi_create_intercommunicator(&communicator, context, local, remote) \end{verbatim} This procedure creates an inter-communicator and returns the communicator identifier in \verb|communicator|. The intercommunicator local world supplied in \verb|local| may be an actual group identifier or the null group \verb|MPI_NULL_GROUP|. If \verb|local| is an actual group identifier then the local world is the identified group and processes within the local world are identified relatively by rank within that group. Otherwise the local world contains all processes within the user program and processes within the local world are identified absolutely by process identifier. The inter-communicator remote world supplied in \verb|remote| may be an actual group identifier or the null group \verb|MPI_NULL_GROUP|. If \verb|remote| is an actual group identifier then the remote world is the identified group and processes within the remote world are identified relatively by rank within that group. Otherwise the remote world contains all processes within the user program and processes within the remote world are identified absolutely by process identifier. The communicator context supplied in \verb|context| may be an actual context value or the null context \verb|MPI_NULL_CONTEXT|. If \verb|context| is an actual context then the procedure completes asynchronously. If \verb|context| is the null context then the procedure allocates a single context using the context allocator described above, synchronizes all members of the local and remote process worlds, and uses the allocated context in each intercommunicator created. Where the user explicitly manages context values then he or she must ensure that the same process world(s) are bound to the same context value in each process which uses that context value, otherwise the program is erroneous. \begin{small} {\bf Discussion:} We propose the nickname ``C2'' for the inter-communicator. \end{small} \begin{verbatim} int communicator; /* IN */ int local; /* OUT */ mpi_intercommunicator_local(communicator, &local) \end{verbatim} This procedure returns in \verb|local| the local world of the inter-communicator identified by \verb|communicator|. \begin{verbatim} int communicator; /* IN */ int remote; /* OUT */ mpi_intercommunicator_remote(communicator, &remote) \end{verbatim} This procedure returns in \verb|remote| the remote world of the intercommunicator identified by \verb|communicator|. The context of an inter-communicator may be determined using the procedure\linebreak[4]\verb|mpi_communicator_context| described above. The procedure will again return the context allocated dynamically when the context supplied in communicator creation was the null context. An inter-communicator may be deleted using the procedure \verb|mpi_delete_communicator| described above. The procedure will synchronize all members of both the local and remote process worlds when the context supplied in communicator creation was the null context, and will otherwise complete asynchronously. \subsection{Point-to-point} The generic point-to-point communication send \verb|send| accepts a single communicator, a process label, and a message label as arguments and send-specific arguments: \begin{verbatim} send(communicator, process_label, message_label, ...) \end{verbatim} \verb|communicator| is the identifier of a communicator object. If the communicator is an intra-communicator then the sender must be a member of the intra-communicator world. If the communicator is an inter-communicator then the sender must be a member of the inter-communicator local world. \verb|process_label| identifies the receiver process. If the communicator is an intra-communicator and the world is an actual group, or if the communicator is an inter-communicator and the remote world is an actual group, then \verb|process-label| is a rank within the group. If the communicator is an intra-communicator and the world is the null group, or if the communicator is an inter-communicator and the remote world is the null group, then \verb|process-label| is a process identifier. \verb|message_label| is the message tag in the context+tag space of the communicator. The generic point-to-point communication receive \verb|recv| accepts a single communicator, a process label, and a message label as arguments and receive-specific arguments: \begin{verbatim} recv(communicator, process_label, message_label, ...) \end{verbatim} \verb|communicator| is the identifier of a communicator object. If the communicator is an intra-communicator then the sender must be a member of the intra-communicator world. If the communicator is an intercommunicator then the sender must be a member of the intercommunicator local world. \verb|process_label| identifies the sender process. If the communicator is an intra-communicator and the world is an actual group, or if the communicator is an intercommunicator and the remote world is an actual group, then \verb|process-label| is a rank within the group. If the communicator is an intra-communicator and the world is the null group, or if the communicator is an intercommunicator and the remote world is the null group, then \verb|process-label| is a process identifier. This field may be wildcarded by supplying the value \verb|MPI_PID_ANY|. \verb|message_label| is the message tag in the tag space of the communicator. This field may be wildcard by supplying the value \verb|MPI_TAG_ANY|. \subsection{Collective} The generic collective communication operation \verb|operation| accepts a single communicator as argument and operation specific arguments. \begin{verbatim} operation(communicator, ...) \end{verbatim} Communicator must be an intra-communicator with a process world which is a process group of which the caller is a member, or else the program is erroneous. \subsection{Message Envelope} To provide the services and capabilities we envisage for MPI, the prototypical message envelope is to consist of : \begin{description} \item[context], which is the identifier of the communicator \item[sender], which is the label of the sender according to the world, or remote world, of the communicator \item[tag], which is the message tag \item[receiver], which is implementation- specific information required to route the message to the receiver \end{description} \begin{small} {\bf Discussion:} The name of sender and receiver, has to be chosen in a way that is useful to the MPI implementation, rather than the user. The user works in ranks, but the system may actually transfer its own internal name for the sender. This remains an implementation detail, with a standard interface based on rank. \end{small} \section{Safety} The user of collective communications must be aware that the operations have no mechanism for protecting the point-to-point messages of which they are composed from those of the user program within the same context. The user is responsible for ensuring that there are no oustanding communications with the same communicator when the collective communication procedure is called. The writer of collective communications cannot use wildcard on either tag or sender in point-to-point receive calls, and strict sequencing of point-to-point messages is assumed. This is more easily handled by utilizing a duplicate communicator: one for point-to-point only, and one for deterministic global operations, all of which have the strict ordering assumptions. Each additional global operation (non-deterministic) will in general need its own communicator (new context, plus copy of the same group). This issue is now well understood, and the features included in this formulation provide a straightforward means for user and library code to achieve safety. %---------------------------------------------------------------------- % Point-to-point Chapter \section{Point-to-point Chapter} {\bf Changes to draft to go here... we will work these out based on newest Snir chapter, on Wednesday night, May 12} %---------------------------------------------------------------------- % Collective Chapter \section{Collective Chapter} {\bf Changes to draft to go here... we will work these out based on newest Geist chapter, on Wednesday night, May 12 \& 13} %---------------------------------------------------------------------- % Worked Examples \section{Worked Examples} {\bf Worked examples to go here... we will add these for you ASAP} {\bf Littlefield examples, Skjellum examples} %---------------------------------------------------------------------- % Conclusion \section{Conclusion} This formulation of context, group, and tag, provide a powerful, forward-looking standard within MPI, that ``closes the loops'' needed to complete the standard. It does so by providing tiers of services, which keep context and group as separate entities as much as possible at the low level, but which subsequently unite them for the user's benefit in intra-communicator (C1) and inter-communicator (C2) objects, that cope with group and context simultaneously. The latter provides a group union capability, especially important in MPMD and distributed-computing situations. Through careful choice of features, we are able to support safe programs in which both system-provided and user-defined contexts are permissible. We are able to provide a tag field that is totally at the user's disposal, and big enough to allow sophisticated application-dependent selectivity (32 bits). This formulation achieves the goal of enabling software module formulation, and provides servers to allow name/context matching, another feature of potential impact to both MPMD and distributed-computing arenas. It further has the clear potential to drop servers whenever the SPMD model is elected, replacing same with synchronizations over all processes. We have left out some things compared to earlier versions. The cacheing facility can be implemented later, in MPI2, or as an implementation-specific set of features, for now. We have omitted discussion of virtual topologies, because they can be layered on top of MPI1, using two features: \begin{itemize} \item contexts for each sub-topology, \item layerability of tag selectivity (dont\_care bits, match bits). \end{itemize} Given these capabilities, we see no need to stipulate virtual topologies in the base MPI is apparent, and so we have not gone into this further here. %======================================================================% \end{document} >From owner-mpi-context@CS.UTK.EDU Sat May 15 22:36:57 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA19231; Sat, 15 May 93 22:36:55 -0500 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA07218; Sat, 15 May 93 22:37:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28086; Sat, 15 May 93 23:35:17 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Sat, 15 May 1993 23:35:16 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28076; Sat, 15 May 93 23:35:14 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA02134; Sat, 15 May 93 22:35:13 CDT Date: Sat, 15 May 93 22:35:13 CDT From: Tony Skjellum Message-Id: <9305160335.AA02134@Aurora.CS.MsState.Edu> To: mpi-context@cs.utk.edu Subject: the gathering (part II) Status: R We have a unified proposal between Marc Snir, myself, Lyndon Clarke, and Mark Sears, which we worked out at the last meeting, and which we will flesh out better over the next weeks, and present to you here. This progress report is mainly for the benefit of those who did not attend the meeting. One point that did not appear in the pre-meeting proposal rounds was the cacheing facility. It will definitely be needed to support virtual topologies. We will send you update soon, Tony >From owner-mpi-context@CS.UTK.EDU Fri May 21 13:21:09 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA13482; Fri, 21 May 93 13:20:58 -0500 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA10187; Fri, 21 May 93 13:19:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09039; Fri, 21 May 93 14:07:48 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Fri, 21 May 1993 14:07:46 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09031; Fri, 21 May 93 14:07:40 -0400 Message-Id: <9305211807.AA09031@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 8875; Fri, 21 May 93 14:07:43 EDT Date: Fri, 21 May 93 14:07:42 EDT From: "Marc Snir" To: MPI-CONTEXT@CS.UTK.EDU Status: R %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: CON-V6.DVI.* %%Pages: 11 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 6 119 df97 D99 D101 D<13C0EA01E013C01380C7FCA6120E123FEA33801263 EAC700A21207120EA35A1340EA38C0A3EA3980EA3F00121E0B1C7D9B0D>105 D<13C01201A3EA0380A4EAFFE0A2EA0700A2120EA45AA31320EA3860A213C0A2EA1F80EA0F000B 1A7D990E>116 D118 D E /Fb 1 27 df<1470EB01F0EB03E0EB0780EB0F0013 1E5B5BA25BB3AD485AA25B1203485A90C7FC120E123C12705A1270123C120E7E7F6C7E12017FA2 6C7EB3AD1378A27F7F7FEB0780EB03E0EB01F0EB007014637B811F>26 D E /Fc 31 121 df<1238127C12FE12FFA2127F123B1203A212071206A2120C121C121812701220 08117CA210>39 D<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA21201 B3A7387FFFE0A213207C9F1C>49 DII< 14E013011303A21307130F131FA21337137713E7EA01C71387EA03071207120E120C1218123812 7012E0B512FEA2380007E0A7EBFFFEA217207E9F1C>I<12601278387FFFFEA214FC14F8A214F0 38E0006014C038C00180EB0300A2EA00065B131C131813381378A25BA31201A31203A76C5A1722 7DA11C>55 D67 D77 DIII87 D97 D99 DI<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FC C7FCA3127CA2127E003E13186C1330380FC0703803FFC0C6130015167E951A>I<3801FE1F0007 B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018 C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80 FC6CB45A000113C019217F951C>103 DI<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2 EAFFF0A20C247FA30F>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFF F1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0 A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7007C137C A26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEB C03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A2 1A207E951F>I114 DI<13C0A41201A212031207120F12 1FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF 14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<3AFFE3FF87F8A23A1F807C00C0D80FC0EB01 80147E13E0000790387F030014DF01F05B00031486EBF18FD801F913CC13FB9038FF07DC6C14F8 EBFE03017E5BA2EB7C01013C5BEB380001185B25167F9528>119 D<39FFF07FC0A2390FC01C00 6C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D7E497E80EB67F013E33801C1F8380381FC 48C67E000E137E39FF81FFE0A21B167F951E>I E /Fd 30 122 df40 D<12E07E12787E7E7E7E13801203A213C01201A712031380A2120713005A121E5A5A5A5A0A 1D7D9914>I<1238127C127EA2123E121EA2127C12F81260070A7A8414>44 D65 D67 DIII<38FE3F80A338380E00A7EA3FFEA3EA380EA738FE 3F80A311177F9614>72 DI75 DI<38FE0FE0A3383B1B80A413 BBA2EA39B3A313F3EA38E3A21303A538FE0FE0A31317809614>I<38FE3F80A3383B0E00A4138E 1239A213CEA31238A213EE136EA4EAFE3EA311177F9614>III82 DII<38FE3F80A338380E00A26C5AA56C5AA4EA0630EA0770A3EA0360A213 E0A26C5A11177F9614>86 D88 D<38FE3F80A3383C1E00EA1C1CEA 1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>I97 D99 D<133C13FE1201EA03DE138C1380A2EAFFFEA3EA0380AAEA7FFCA30F177F9614> 102 D109 D111 DI<38FF1F80EB7FC013FF3807E080EBC0005B A290C7FCA6EAFFFCA31210808F14>114 D<38FF3F80A3381C1C00A2120E5BA212071330A2EA03 70A26C5AA35BA3EA7B80127F90C7FC127E123C11187F8F14>121 D E /Fe 1 106 df105 D E /Ff 57 123 df<1218123C123E121E120EA3121E121C 123C127812F01260070D799816>39 D<13E01201EA07C013005A121E5A123812781270A312F05A A77E1270A312781238123C7E7E7E13C0EA01E012000B217A9C16>I<12E07E127C121C121E7EEA 0780120313C01201A313E01200A7120113C0A3120313801207EA0F00121E121C127C12F05A0B21 7C9C16>II<1238127C127EA2123E120E121E123C127C12F81260070B798416> 44 D47 D49 DI<1238127CA312381200A81238127CA3123C121C123C123812F812F0126006 18799116>59 D<13E0487EA213B0A2EA03B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3 387F1FC000FF13E0007F13C013197F9816>65 DI< 3801F180EA07FF5AEA1F0FEA3C0712781303127000F0C7FC5AA77E387003801278A2EA3C07381F 0F00EA0FFE6C5AEA01F011197E9816>II<387FFFC0B5FC7EEA1C 01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816>II<387F1FC038 FFBFE0387F1FC0381C0700A7EA1FFFA3EA1C07A9387F1FC038FFBFE0387F1FC013197F9816>72 DI<387F0FE038FF8FF0387F0FE0381C0780EB0F00 131E131C133C5B5BEA1DE07F121F7F1338EA1E3C131CEA1C1E7F7F14801303387F07E038FF8FF0 387F07E01419809816>75 DI<38FC07 E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313197F9816 >I<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA213E7A31367A21377A21337A3 1317EA7F1FEAFF9FEA7F0F13197F9816>III82 DI<387FFFE0B5FCA2 EAE0E0A400001300AFEA07FC487E6C5A13197F9816>I<387F07F038FF8FF8387F07F0381C01C0 B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519809816>I<38FE0FE0EAFF1FEAFE0F3838 0380381C0700A4EA0E0EA4EA060CEA071CA4EA031813B8A3EA01B013F0A26C5A13197F9816>I< 387F1F80133F131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B8120713 1CA2EA0E0EA2487E387F1FC000FF13E0007F13C013197F9816>88 D<38FE0FE0EAFF1FEAFE0F38 1C0700A2EA0E0EA26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I<387FFF 80B5FCA238E007005B131E131CEA003C5B137013F0485A5B1203485A90C7FC5A381E0380121C12 3C12781270B5FCA311197E9816>I95 D<120C121E123C1278127012F0 12E0A312F012F812781230070D789B16>II<127E12FE12 7E120EA4133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E01381EB83C013FF 000E1300EA067C1419809816>II<133F5B7F1307A4EA03E7EA0FFF123FEA3C1F 487E1270EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E014197F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FCA2EA01C0ACEA 3FFE487E6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3EA3C1E6C5A EA1FFC485AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3EAF001387C 07C0383FFF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0FFF1480138713 03A2120EA9387FC7F038FFE7F8387FC7F01519809816>II<127E12FE127E120EA4EB7FE0A3EB0F00131E5B5B5B EA0FF8A213BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816>107 DI<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870 AA38FE7CF8A31512809116>IIII<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F9116>114 DI<12035AA4EA7FFFB5FCA20007C7FCA75BEB0380A2130713 873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C 13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4 EA03B8A3EA01F0A26C5A13127F9116>I<38FF1FE013BF131F38380380A413E33819F300A213B3 EA1DB7A4EA0F1EA313127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813F012016C5A 12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>I<387F1FC038FF 9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35BA3485A1279 127BEA7F806CC7FC123C131B7F9116>I<383FFFC05AA238700780EB0F00131EC65A13F8485A48 5A485A48C7FC381E01C0123C1278B5FCA312127F9116>I E /Fg 58 124 df11 D<137CEA01FEEA0387485A120E130690C7FCA4B5FCA2EA0E07AC387F0FE0A213 1A809915>I 34 D<126012F012F812781218A31230127012E01240050B7D990B>39 D<13C0EA0180EA030012 06120E120C5A1238A212301270A21260A212E0AA1260A21270A212301238A212187E120E12067E EA0180EA00C00A267E9B0F>I<12C012607E7E121C120C7E1207A27E1380A21201A213C0AA1380 A21203A213005AA212065A121C12185A5A5A0A267E9B0F>I<126012F0A212701230A31260A212 C01240040B7D830B>44 DI<126012F0A2126004047D830B>I<130CA213 1C1318A213381330A213701360A213E013C0A212011380A2120313005A1206A2120E120CA2121C 1218A212381230A212701260A212E05AA20E257E9B13>I<126012F0A212601200A8126012F0A2 126004107D8F0B>58 D<126012F0A212601200A8126012F0A212701230A31260A212C012400417 7D8F0B>I63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0 A348B47EA2130000061378A3487F121E39FF81FFC0A21A1B7F9A1D>65 D67 DIIII<39FFF3FFC0A2390F003C00A9EBFFFCA2EB003CAB39FF F3FFC0A21A1A7F991D>II<39FFF07FC0A2390F00 3E0014185C5C5C495A49C7FC13065B131E133F136FEBC780EB87C013036D7E8013001478A28014 3E39FFF0FFC0A21A1A7F991E>75 D<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3EB78 18A3EB3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>77 DI<137F3801FFC03807C1 F0380F0078001E7F001C131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C13 1C001E133C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>II82 DI<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FCAF3803FFF0A2191A 7F991C>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC5DD807801306EB819E A2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE13F0EBFC03017C5BA2 EB7801A201385BEB3000241B7F9927>87 D92 D97 D<12FCA2121CA813F8EA1FFE130F381C0380A2 EB01C0A6EB0380A2381F0F00EA1BFEEA18F8121A7F9915>II<137EA2130EA8EA07CEEA1FFEEA3C 1EEA700EA212E0A61270131EEA3C3E381FFFC0EA07CF121A7F9915>II<13F8EA01FCEA 03BCEA073CEA0E181300A5EAFFC0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA 3877EA7038A5EA3870EA3FE0EA7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781E EA1FF8EA07E010197F9013>I<12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A 7F9915>I<1218123CA21218C7FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00 C01300A6EA07E0A21200B0126012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F 80A2EB3C0013305B5B121DEA1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12 FCA2121CB3A4EAFF80A2091A80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301 AA39FF9FE7F8A21D107F8F20>II< EA07E0EA1FF8EA381CEA700EEA6006EAE007A6EA700EA2EA3C3CEA1FF8EA07E010107F8F13>I< EAFCF8EAFFFEEA1F0F381C07801303EB01C0A6EB03801307381F0F0013FEEA1CF890C7FCA5B47E A212177F8F15>IIII<120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B 177F960F>II<38FF3F80A2381C0E 00130CA26C5AA21338EA0730A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00 381C3E0C13361337000E5B13631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1F C0A2380E1E00EA071CEA03B813B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A213 10808F14>I<38FF3F80A2381C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA21261 00F3C7FC12C712FE127811177F8F14>III E /Fh 8 118 df<127812FCA412781200A5127812FCA4127806117D900C>58 D68 D99 D<121E123FA4121EC7FCA4127FA212 1FADEAFFC0A20A1B809A0C>105 D110 DI115 D<38FF1FE0A2EA1F03AB1307130F380FFFFCEA03F316117F9019>117 D E /Fi 3 34 df0 D<127012F8A3127005057D8C0C>I<1506A281168015 01ED00E0B712F816F0C912E0ED0180150316001506A2250E7E902A>33 D E /Fj 11 123 df<127012F8A3127005057D840C>58 D<127012F012F8A212781218A31230A212 7012601240050D7D840C>I I<13F8EA03FEEA0F06121E123C1238EA781CEA7FF8EAFFE0EAF000A4EA7004130EEA383CEA1FF0 EA0FC00F127F9113>101 DI<13E01201A2 EA00C01300A6120EEA1F8012331263124312C3EA0700A2120EA3EA1C201360A2EA38E013C0EA1F 80EA0F000B1C7F9B0E>105 D<1307130FA213061300A613F0EA01FCEA031C1206A2120C1200A2 1338A41370A413E0A4EA01C01261EAF380EAF70012FE127C1024809B11>I<391C0FC0F8393F1F E3FC396730770EEB607C38C7C078138038070070A2000E495AA3ED3840261C01C013C0A2157116 803A3803803F00D81801131E22127F9124>109 D<381C0F80383F3FC0386770E013C0EAC780A2 EA0700A2380E01C0A3EB0384001C138CA2EB071C1418383803F0381801E016127F9119>I115 D<3801E180EA07F1380FFF00EA0C1EEA180CC65A5B 5B5B485A48C7FCEA0601485A1218EA3F0EEA7FFCEA61F8EAC0F011127F9113>122 D E /Fk 42 121 df<1238127C12FCA2127C120CA21218A21230126012C01280060D799C0C>39 D<1230127812F81278127005057D840C>46 D<13181370EA03F0120FEA1C701200A213E0A6EA01 C0A6EA0380A61207EAFFF8A20D1C7C9B15>49 D<137E3801FF80380387C0EA0603EB01E0120FA2 121FA2120E1200EB03C0A2EB0780EB0F00130E5B5B13605BEA038038070180120E381C03001230 EA7FFF485AA2131C7E9B15>I<137C48B4FC38038780EA0603000F13C0A3380E07801200140013 0E133CEA01F8485AEA001C7F130FA4127012F8A2EAF01E12E05BEA7078EA3FE0EA0F80121D7D9B 15>II<133E13FF3801C380 EA0383EA0707120E001C130090C7FC123C1238EA79F8EA7FFCEA7C1C487EA2EAF00FA4EAE01EA4 131C5BEA70705BEA3FC0EA0F80111D7C9B15>54 D<137C48B4FC38038780EA0703120EEB01C0A2 381E0380A2EB0700EA0F8613DCEA07F81203487EEA0E7E487EEA380F12707F12E0A31306130EEA 701CEA3878EA1FF0EA0FC0121D7D9B15>56 D<14301470A214F0801301A2EB0378A21306147CEB 0C3CA21318A21330A2497EA2EBFFFEA23801801EA23803001F801206120F397FC0FFF012FF1C1D 7F9C1F>65 D<903807F030EB1FFC90387C0E603901F003E0EA03C038078001EA0F00000E130000 1E14C05AA25A1500A25AA5EC0180EC03001270127814066C5B001C131C000F5B3807C0E06CB45A D8007EC7FC1C1E7C9C1E>67 D<380FFFFCECFF803900F007C0EC01E0EC00F015701578485A1538 A3153C1538485A1578A3157015F0484813E0140115C0EC03801407EC0E00380F007CB512F014C0 1E1C7E9B20>I<000FB512E0A23800F003EC01C01400A33801E060A31500EBE1E013FF485B13C1 13C0A215C0ECC18038078001A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>I<000FB512 C0A23800F007EC03801401A3EA01E01461A2EC600014E013E148B45AA213C113C0A33807818001 80C7FCA5120FEAFFFC5B1A1C7E9B1B>I<903807F030EB1FFC90387C0E603901F003E0EA03C038 078001EA0F00000E1300001E14C05AA25A1500A25AA3ECFFF0A2EC0F80150012701278A27E001C 5B000F133E3807C0F63803FFC2C66CC7FC1C1E7C9C21>I<390FFF9FFE143F3900F003C0A53901 E00780A590B5FC481400EBC00FA53807801EA6000F133E39FFF3FFE015C01F1C7E9B1F>I<380F FF80A23800F000A5485AA6485AA6485AA6120FEAFFF8A2111C7F9B0F>I<390FFF87FEA23900F0 03E0EC0180EC030014065C48485A14705CEBE1C013E313E73803CDE013D9EBF1F013E013C01478 EA078080A280A280000FEB1F8039FFF87FF013F01F1C7E9B20>75 D<380FFFC0A23800F000A548 5AA6485AA41406140CEA0780A2141C141814381478380F01F8B512F0A2171C7E9B1A>II<390FF80FFEA23900FC01E0EC00C0A213DEA239018F0180A214811387A2EB83C1390303C300 EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38FFE00C13C01F1C7E9B1F>II<380FFFFC14FF3900F00F80140315C01401A23801E003A315801407EC0F003803C03E EBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B1C>I<380FFFF814FE3900F00F801403 15C01401A23801E003A3EC0780EC0F00143E3803FFF85CEBC07880A2143E3807803CA51518000F 1430EAFFF89038F01FE0C7EA07C01D1D7E9B1F>82 DI<001FB512F05A383C07810038EB8060 12701260A2EB0F0012C0A200001400A3131EA65BA6137C381FFFE0A21C1C7C9B1E>I<39FFF8FF E0A2390F001E00140CA4001E5BA6485BA6485BA300385BA2383C0180D81C03C7FCEA0E0EEA07FC EA01F01B1D7A9B1F>I<3BFFF0FFE1FF805B3B1F001F007C006C15384A13301670026F1360A202 CF5BA23A07818F018015819026838783C7FC140715861386158E01CC138C120301D813D8140301 F013F0A25D13E05DEA01C05DEB8001291D7B9B2B>87 D<3907FF8FFEA239007C03E0013C138001 3E1300EB1E06140EEB1F1CEB0F1814B0EB07E0A26D5A80A2497E130E130C497EEB3078EB707CEB E03CEBC03E3801801E3803001FEA0F8039FFE07FF014FF1F1C7F9B1F>I97 D<13FCEA03FEEA0F0FEA1E0EEA3C0CEA38001278A35AA31270EA7806EA380CEA3E18EA1F F0EA07E010127E9112>99 DII105 D<391F8FC0FC9038BFE3FE3907E0F60F9038C07C07EB8078EB0070A4000EEBE0 0EA64848485A3AFF8FF8FF80A221127F9124>109 D<381F8F80EBBFC03807E0E013C013801300 A4380E01C0A6381C038038FF8FF0139F14127F9117>I<13FCEA03FF380F0780381E03C0EA3C01 1238007813E0A338F003C0A300701380130738780F00EA3C1EEA1FF8EA07E013127E9115>I<38 0FC7E0EBFFF03803F078EBC03C1380A2141EA33807003CA31478A2EB80F0380FC3E0380EFF80EB 3E0090C7FCA35AA3B47EA2171A809117>I114 DI<1206A3120EA2120C121C123CEA FFE0A2EA1C005AA65A13C0A4EA7180A2EA7F00123C0B1A7C9910>I<38FC1F80A2EA3C07003813 00A6EA700EA4131E133E137E383FDF80EA1F1F11127C9117>I<381FE3FCEA3FE7380383C03801 C380EBC700EA00EE13EC13781370137813F8EA01DCEA039EEA070EEA0607121E38FF1FF0A21612 7F9116>120 D E /Fl 51 123 df<13301360EA01C0EA038013005A120E121E121C123CA21238 1278A312F85AA97E1278A31238123CA2121C121E120E7E7E1380EA01C0EA006013300C297D9E13 >40 D<12C0126012387E120C120E7E1380120313C0A2120113E0A313F01200A9120113E0A313C0 1203A2138012071300120E120C121C5A12605A0C297D9E13>I<1238127C12FE12FFA2127F123B 120312071206A2120C121812301220080F7D860D>44 D<1360EA01E0120F12FF12F31203B3A238 7FFF80A2111B7D9A18>49 DI63 D65 D I<90381FE0209038FFF8E03803F80F3807E007380FC001EA1F8048C7FCA2007E1460A212FE1500 A7007E1460A27E15C06C7E390FC001803907E003003803F80E3800FFFCEB1FE01B1C7D9B22>I< B512F814FF390FC01FC0EC07E0EC03F0EC01F8A2EC00FCA315FEA815FCA3EC01F8A2EC03F0EC07 E0EC1FC0B6120014F81F1C7E9B25>III73 D<39FFFC07FFA2390FC000E0EC01C0 EC0380EC06005C14385C5CEBC1C013C3EBC7E013CFEBFBF0EBF3F813E1EBC0FC80147E80EC1F80 15C0140FEC07E015F039FFFC3FFFA2201C7E9B25>75 DII<39FFE003FFA2390FF000307FEA0DFCEA0CFE137E7FEB1F8014C0 EB0FE0EB07F01303EB01F814FCEB00FE147F143FEC1FB015F0140F1407140314011400A2D8FFC0 13701530201C7E9B25>III82 D<3807F860381FFEE0EA3C07EA7801EA700012F01460A26C130012FEEAFFE0EA7F FE6C7E1480000F13C06C13E0EA007FEB03F01301130012C0A214E07E38F001C0EAFC0338EFFF00 EAC3FC141C7D9B1B>I<007FB512E0A238781F81007013800060146000E0147000C01430A40000 1400B03807FFFEA21C1C7E9B21>I<39FFFC03FFA2390FC00030B3120715606C6C13E03901F001 C03900FC078090387FFE00EB0FF8201C7E9B25>I<3AFFFC01FF80A23A0FC00018006C6C5BA26D 1370000314606D13E000015C7F0000495AA2D97E03C7FCA2EB7F07EB3F06148EEB1F8C14CCEB0F D8A2EB07F0A36D5AA26D5AA2211C7F9B24>I<397FFE1FFEA23907F001803803F8036C6C48C7FC 000013066D5AEB7F1C6D5A14B0EB1FE0130FA26D7E6D7E1307497EEB0CFEEB187EEB387F496C7E EB601F01C07F00016D7E496C7EEA03003AFFF03FFF80A2211C7F9B24>88 D<3AFFFC01FF80A23A0FE00038006C6C13306C6C137015606C6C5B3800FE015DD97F03C7FCEB3F 871486EB1FCEEB0FFC5C13076D5AAAEB3FFFA2211C7F9B24>I<387FFFFCA2387E01F8EA780300 7013F038E007E0130F00C013C0131F148038003F005B137E13FE485A5B0003130613F0EA07E012 0FEBC00E121FEB801CEA3F0048133C007E13FCB5FCA2171C7D9B1D>I97 DIIII<133F3801FF803803E7C0EA07C7EA0F87EB8380EB8000A5EAFFF8A2EA0F80AEEA7FF8 A2121D809C0F>I<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E1FEA1FFC EA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F0383E03E038 1FFFC03803FE00151B7F9118>II<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>I107 DI<39FF1FC0FE90387FE3FF 3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1FC0EB7FE038 1FE1F0EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1F C0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090C7FCA6EAFF E0A2171A7F911B>I 114 DI<1203A35AA25AA2123FEAFFFCA2EA1F00A9130CA4 EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA03F818127F 911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C 5AA21378A2133016127F9119>I<39FF8FF8FEA2391F03E030A201831370000FEBF0601386D807 C613C0EBCEF8EBEC790003EB7D80EBF83D0001EB3F00A2497E0000131EEBE00EA21F127F9122> I<38FFC7FCA2381F8180EA0F833807C700EA03EEEA01FC5B1200137C13FEEA01DFEA039F38070F 80380607C0380C03E038FF07FCA216127F9119>I<38FFC1FCA2381F0060EB80E0000F13C013C0 3807C180A23803E300A2EA01F6A213FE6C5AA21378A21330A25B1270EAF8E0EAC0C0EAE380007F C7FC123E161A7F9119>I<383FFF80A2383C1F00EA303EEA707EEA60FC5BEA61F012033807E180 13C1EA0F81EA1F83383F0300EA3E07485AB5FCA211127F9115>I E /Fm 70 124 df11 D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B512C0A2 EA0E01AE387F87F8A2151D809C17>I<90383F03F09038FFCFF83903C0FC1C390701F03CA2390E 00E01892C7FCA5B612FCA2390E00E01CAE3A7FC7FCFF80A2211D809C23>14 D34 D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C>39 D<13C0EA0180EA030012 06120E120C121C121812381230A21270A21260A212E0AC1260A21270A21230A212381218121C12 0C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C120C120E120612077EA21380 A21201A213C0AC1380A21203A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>I<1306 ADB612E0A2D80006C7FCAD1B1C7E9720>43 D<127012F012F8A212781218A31230A21270126012 40050D7D840C>II<127012F8A3127005057D840C>I<1303A213071306 A2130E130C131C1318A213381330A213701360A213E013C0A21201138012031300A25A1206A212 0E120CA2121C1218A21238123012701260A212E05AA210297E9E15>II<12035A12 3FB4FC12C71207B3A3EAFFF8A20D1C7C9B15>III<131CA2133C137CA213DC1201139C1203EA071C1206120E120C121812381230126012 E0B512C0A238001C00A63801FFC0A2121C7F9B15>II<13F0EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EA FC1CEAF81E130E12F0130FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>I<1260 387FFF80A21400EA6003EAC0065BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B 15>III<127012F8A312701200A81270 12F8A3127005127D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A21270 12601240051A7D910C>I<007FB512C0B612E0C9FCA8B612E06C14C01B0C7E8F20>61 D<1306130FA3497EA4EB33C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3 001F131F39FFC0FFF0A21C1D7F9C1F>65 DI<90381F80 80EBFFE13803F0333807801B380F000F001E1307001C1303123C5A1401127012F091C7FCA70070 EB01801278A27E001CEB0300121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E9C1E>IIII<39FFF3FFC0 A2390F003C00AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>72 DI76 DIIII 82 D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA 07FFC61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F8121E7E9C 17>I<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B03803FFFC A21B1C7F9B1E>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA20180147000079038678060A3 2603C0E713C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13F6017E137E A3013C133CA3011C133801181318281D7F9B2B>87 D92 D97 D<12FCA2121CA9137EEA1DFF381F8780381E01C0001C13E0130014F0A614E01301001E13C0381F 07803819FF00EA187C141D7F9C17>IIII<1378EA01FCEA039EEA071EEA0E0C1300A6EAFFE0A2EA0E00AEEA7FE0A20F1D809C0D>II<12FC A2121CA9137CEA1DFFEA1F07381E0380A2121CAB38FF9FF0A2141D7F9C17>I<1218123C127C12 3C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C0C>II<12FCA2121CA9EB7FC0A2EB3E0013185B 5B5BEA1DE0121FEA1E70EA1C781338133C131C7F130F38FF9FE0A2131D7F9C16>I<12FCA2121C B3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF9FF8391F83B838391E01E01CA2001C13C0AB 3AFF8FF8FF80A221127F9124>IIII<3803E180EA0F F9EA1E1FEA3C071278130312F0A612781307123CEA1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A 7F9116>III<120CA5121CA2123CEAFFE0A2EA1C00A81330A5EA1E60EA0FC0EA07800C1A7F9910>I< 38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA03E314127F9117>I<38FF0FE0A2381C0780EB03 00EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA313127F9116>I<39FF3FCFE0A2391C0F07 80EC0300131F380E1B061486A2EB318E000713CCA213603803E0F8A33801C070A31B127F911E> I<387F8FF0A2380F078038070600EA038EEA01DC13D8EA00F01370137813F8EA01DCEA038E130E EA0607380F038038FF8FF8A21512809116>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA213 1CEA0398A213F86C5AA26C5AA35BA3EAF180A200C7C7FC127E123C131A7F9116>III E /Fn 35 121 df<121C127FEAFF80A213C0A3127F 121C1200A212011380A21203EA07001206120E5A5A12300A157BA913>39 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FEA317277BA622>49 DII<14075C5C5C5C5CA25B5B497E130F130E131C1338 137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F13F8A31D277EA6 22>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC0380FFFF8EB80FC38 0E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300007E137F007814806CEBFF 00381F01FE380FFFF8000313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E04814C01580A21500387000 1E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213FEA41201A86C5A13781C29 7CA822>II<91393FF00180903903FFFE07010FEBFF8F90393FF007FF90 38FF80014848C7127FD807FC143F49141F4848140F485A003F15075B007F1503A3484891C7FCAB 6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007F001 0FB512C0010391C7FC9038003FF829297CA832>67 D69 D73 D76 DI87 D<48B47E000F13F0381F81FC486C7E147FA2EC3F80A2EA0F00 C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F83DFFC38 0FFF0F3801FC031E1B7E9A21>97 D99 DIII<9038FF81F00003EBE7FC390FC1FE7C391F80FCFC003FEB FE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1F8381FFFE0001813800038C8FC123C A2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007F8007C13015A1400A36C1301007EEB 03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A22>II< 1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14>I< EAFFE0A3120FACEC1FFCA3EC07C0EC0F80EC1E00147C5CEBE1F0EBE3E0EBE7C0EBEFE0EBFFF0A2 80EBF3FCEBE1FE13C080EC7F80143F15C0EC1FE0EC0FF039FFFC3FFEA31F2A7EA924>107 DI<3BFFC07F800FF0903AC1FFE03FFC903AC783F0 F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3FFF8 A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A313E0 AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F1400 81A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8FCA9 EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FCA318 1B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013FE38 7FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038F7FF C000C11300161B7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD1438 A73803F870000113F03800FFE0EB1F8015267FA51B>I<39FFE03FF8A3000F1303B11407A2140F 0007131F3A03F03BFF803801FFF338003FC3211B7D9A26>I<3BFFFE7FFC0FFEA33B0FE007E000 E03B07F003F001C0A29039F807F80300031680A23B01FC0EFC0700A2D9FE1E5B000090381C7E0E A29039FF383F1E017F141C0278133C90393FF01FB8A216F86D486C5AA26D486C5AA36D486C5AA2 2F1B7F9A32>119 D<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3F F86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01F FF80A3211B7F9A24>I E /Fo 24 122 df<127012F812FCA2127C120CA41218A21230A2126012 40060F7C840E>44 D49 D51 D57 D<90380FE01090383FF8309038F81C703801E0063903 C003F03807800148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA200 3C1460123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>67 D76 DI<38 03F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3FF0EA1FFE 6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF80138EF0380 38C7FF00EA81FC14247DA21B>83 D<007FB512F8A2387C07800070143800601418A200E0141C00 C0140CA500001400B3A20003B5FCA21E227EA123>I 97 D99 D<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA 38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>II<121C121E123E121E121CC7FCA8120E12FEA2121E120E AFEAFFC0A20A227FA10E>105 DI<120E12FEA2121E120EAAEB0FFCA2EB07E0EB0380 EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03C0130114E014F038FFE3FEA21723 7FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>I<390E1FC07F3AFE7FE1FF809039 C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>I<380E1F8038 FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II114 DI<000E137038FE07F0A2 EA1E00000E1370AC14F01301380703783803FE7FEA01F818157F941B>117 D<38FFC3FEA2381E00F8000E1360A26C13C0A338038180A213C300011300A2EA00E6A3137CA313 38A21330A213701360A2EAF0C012F1EAF380007FC7FC123E171F7F941A>121 D E /Fp 15 121 df<127812FCA212FEA2127E1206A5120CA31218A2123012701260124007147A 8512>44 D<91383FE001903901FFF803903807F01E90391F80070790393E00018F49EB00CF01F0 147F4848143F0003151F485A4848140FA248C8FC16075A123E007E1503A3127C00FC1500AB127C 007E1503A3123E123F6C1506A26C7E160C6C7E6C6C141812016C6C1430017C14606DEB01C09039 1F800380903907F01E00903801FFFC9038003FE028337CB130>67 D71 D77 D97 D100 DI<380781FE39FF87FF8090 388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E3AFFFC1FFF80A2211F7E9E25> 110 DI<380780FC39FF87FF8090389E07C0390FB803E03907E000F04913F8157C5B 153EA3151FA9153EA3157C6D137815F89038E001F09038B803E090389E0FC0903887FF00EB81FC 0180C7FCAB487EEAFFFCA2202D7E9E25>I<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE038EB C000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813705A 1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2141C 7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A31203A2 1207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB19> II<39FFF807FFA2390FE003F83903C001E001E05B0001495A6C6C 48C7FCEB7806EB7C0CEB3C1C6D5AEB0F3014E013076D5A80801307EB0E78EB0C7C497EEB381E49 7E01607F496C7E0001130348486C7E000780391FC003F83AFFE007FFC0A2221F7F9E23>120 D E /Fq 5 85 df<1630167016F0A21501A21503A21507150FA2151B821531A2156115E115C1EC 0181A2EC0301A21406A2140C141C14181430A202607FA2ECC000A249B5FC5B91C7FC1306A25BA2 5BA25B1370136013E01201000381D80FF01301D8FFFE90383FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E83147E188002FE140FA24A15C0 A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA2011F157E17FE4A5CA2013F4A 5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495AED07E00001EC3F80B600FE C8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7121EA24A140CA2147EA214 FEA25CA20101151CEEC0184A1400A201031301A202F05B15030107130F91B5FC93C7FCECE00F01 0F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA2137EA213FEA25BA21201B512 FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092380003F0EE01F84AEB00FC A2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804AEB3F00167E0107EB03F891 B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA2013F1301A21400A249495AA2 137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C130F121800385CA20030011F 1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FEA25CA21301A25CA21303A2 5CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D3174B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 795 220 a Fq(D)26 b(R)g(A)f(F)h(T)570 311 y Fp(Maps,)20 b(Groups)i(and)e(Con)n(texts)391 433 y Fo(Lyndon)d(Clark)o(e,)e(Mark)h (Sears,)h(T)l(on)o(y)f(Skjellum,)d(Marc)j(Snir)832 530 y(Ma)o(y)g(13,)g(1993) 75 715 y Fn(1)69 b(In)n(tro)r(duction)75 807 y Fm(MPI)15 b(pro)o(vides)f (supp)q(ort)h(for)f(the)h(execution)g(of)f Fl(parallel)f(pro)q(cedures)p Fm(.)k(A)e(parallel)e(pro)q(cedure)j(is)e(executed)75 857 y(collectiv)o(ely)g (b)o(y)g(a)f(set)j(of)d(comm)o(unicating)e(pro)q(cesses.)22 b(T)m(ransfer)14 b(of)g(con)o(trol)g(to)g(the)g(parallel)f(pro)q(cedure,)j (and)75 907 y(bac)o(k,)d(is)g(ac)o(hiev)o(ed)g(b)o(y)g(ha)o(ving)e(eac)o(h)j (executing)g(pro)q(cess)g(transfer)g(con)o(trol)f(to)g(the)h(lo)q(cal)e(pro)q (cedure)j(co)q(de,)e(and)75 957 y(return)j(from)e(it.)22 b(Not)15 b(all)f(pro)q(cesses)k(need)e(transfer)g(con)o(trol)f(to)h(the)f(same)g (parallel)f(pro)q(cedure;)j(the)f(parallel)75 1006 y(pro)q(cedure)k(ma)o(y)d (in)o(v)o(olv)o(e)g(only)h(a)g(subset)i(of)d(pro)q(cesses,)22 b(or)d(di\013eren)o(t)g(parallel)e(calls)h(ma)o(y)f(b)q(e)i(executed)h(b)o(y) 75 1056 y(di\013eren)o(t)15 b(subsets)g(of)e(pro)q(cesses.)21 b(The)14 b(collectiv)o(e)f(comm)o(unication)e(calls)i(pro)o(vided)g(b)o(y)h (MPI)g(are)g(examples)f(of)75 1106 y(suc)o(h)i(parallel)d(pro)q(cedures.)21 b(Libraries)14 b(suc)o(h)g(as)g(parallel)f(scien)o(ti\014c)i(libraries)e (will)g(b)q(e)h(another)h(example.)158 1157 y(It)j(is)g(highly)e(desirable)j (to)f(allo)o(w)e(the)i(pro)q(cesses)j(that)d(execute)i(a)d(parallel)g(pro)q (cedure)j(use)e(a)g(\\virtual)75 1206 y(pro)q(cess)i(name)d(space")j(lo)q (cal)d(to)i(the)g(in)o(v)o(ok)n(ation.)29 b(Th)o(us,)19 b(the)g(co)q(de)h(of) d(the)i(parallel)f(pro)q(cedure)i(will)d(lo)q(ok)75 1256 y(iden)o(tical,)c (irresp)q(ectiv)o(e)j(of)e(the)h(absolute)f(addresses)j(of)d(the)h(executing) g(pro)q(cesses.)22 b(It)14 b(is)g(often)h(the)g(case)g(that)75 1306 y(parallel)10 b(application)g(co)q(de)j(is)e(built)g(b)o(y)g(comp)q (osing)f(sev)o(eral)i(parallel)e(mo)q(dules)h(\(e.g.,)f(a)i(n)o(umerical)e (solv)o(er,)h(and)75 1356 y(a)j(graphic)g(displa)o(y)f(mo)q(dule\).)18 b(Supp)q(ort)d(of)e(a)h(virtual)f(name)g(space)j(for)e(eac)o(h)g(mo)q(dule)f (will)g(allo)o(w)f(to)i(comp)q(ose)75 1406 y(mo)q(dules)19 b(that)g(w)o(ere)i(dev)o(elop)q(ed)f(separately)g(without)f(c)o(hanging)g (all)g(message)g(passing)g(calls)h(within)e(eac)o(h)75 1455 y(mo)q(dule.)e(The)e(set)g(of)e(pro)q(cesses)k(that)d(execute)i(a)e(parallel) f(pro)q(cedure)j(ma)o(y)c(b)q(e)j(\014xed,)f(or)g(ma)o(y)e(b)q(e)j (determined)75 1505 y(dynamically)9 b(b)q(efore)k(the)g(in)o(v)o(ok)n(ation.) i(Th)o(us,)e(MPI)f(has)h(to)f(pro)o(vide)g(a)g(mec)o(hanism)d(for)j(creating) h(dynamically)75 1555 y(sets)20 b(of)f(lo)q(cally)e(named)h(pro)q(cesses.)36 b(W)m(e)18 b(alw)o(a)o(ys)g(n)o(um)o(b)q(er)g(pro)q(cesses)k(that)d(execute)i (a)d(parallel)g(pro)q(cedure)75 1605 y(consecutiv)o(ely)m(,)d(starting)g (form)e(zero.)23 b(Th)o(us,)15 b(a)f Fl(group)g Fm(is)g(an)h(ordered)h(set)g (of)f(pro)q(cesses.)24 b(Eac)o(h)15 b(pro)q(cess)i(in)d(a)75 1655 y(group)i(is)g(asso)q(ciated)h(with)f(a)f Fl(rank)p Fm(,)h(starting)g (from)e(zero.)26 b(Pro)q(cesses)19 b(are)d(iden)o(ti\014ed)h(b)o(y)f(their)g (ranks)g(when)75 1705 y(comm)o(unication)10 b(o)q(ccurs)16 b(within)d(the)h(group.)158 1755 y(Another)h(imp)q(ortan)o(t)d(goal)h(is)h (the)h(supp)q(ort)g(of)f(a)g Fl(lo)q(osely)g(sync)o(hronous)e Fm(transfer)j(of)e(con)o(trol:)19 b(no)14 b(syn-)75 1805 y(c)o(hronization)f (o)q(ccurs)i(either)f(b)q(efore)g(or)f(after)h(the)g(call.)j(Th)o(us,)c(a)g (pro)q(cess)i(ma)o(y)d(start)i(sending)f(messages)h(that)75 1855 y(p)q(ertain)g(to)g(the)h(execution)g(of)e(a)h(parallel)f(pro)q(cedure)j (b)q(efore)f(all)e(other)i(participating)e(pro)q(cesses)k(ha)o(v)o(e)c (joined)75 1904 y(the)j(execution;)h(a)f(pro)q(cess)i(ma)o(y)13 b(receiv)o(e)18 b(a)d(message)h(that)f(p)q(ertains)i(to)f(the)g(execution)h (of)e(a)g(parallel)g(pro)q(ce-)75 1954 y(dure,)f(while)g(participating)f(in)g (another)i(parallel)e(execution.)19 b(A)14 b Fl(comm)o(unicatio)o(n)f(con)o (text)f Fm(mec)o(hanism)g(is)75 2004 y(needed)k(to)e(distinguish)g(comm)o (unicatio)o(n)e(that)i(b)q(elongs)h(to)f(distinct)g(parallel)g(pro)q(cedure)i (executions.)k(\(Ev)o(en)75 2054 y(if)11 b(parallel)h(transfer)h(of)f(con)o (trol)g(is)g(executed)i(sync)o(hroneously)m(,)f(one)f(still)g(needs)h(a)f (con)o(text)h(mec)o(hanism)d(for)i(the)75 2104 y(sync)o(hronization)i(calls)f (themselv)o(es.\))158 2154 y(Normally)m(,)i(a)i(parallel)f(pro)q(cedure)k(is) d(written)h(so)g(that)f(all)f(messages)i(pro)q(duced)h(during)e(its)g (execution)75 2204 y(are)e(also)f(consumed)g(b)o(y)g(the)h(pro)q(cesses)i (that)d(execute)j(the)e(pro)q(cedure.)21 b(Ho)o(w)o(ev)o(er,)15 b(if)e(one)i(parallel)e(pro)q(cedure)75 2254 y(calls)i(another,)h(then)h(it)e (migh)o(t)f(b)q(e)i(desirable)g(to)g(allo)o(w)e(suc)o(h)i(call)f(to)h(pro)q (ceed)h(while)e(messages)h(are)g(p)q(ending)75 2304 y(\(the)21 b(messages)f(will)f(b)q(e)i(consumed)f(b)o(y)g(the)h(pro)q(cedure)h(after)e (the)h(call)e(returns\).)39 b(In)20 b(suc)o(h)h(case,)i(a)d(new)75 2354 y(comm)o(unication)11 b(con)o(text)k(is)f(needed)i(for)d(the)i(called)f (parallel)f(pro)q(cedure,)j(ev)o(en)f(if)e(the)i(transfer)g(of)f(con)o(trol)g (is)75 2403 y(sync)o(hronized.)158 2454 y(A)j Fl(con)o(text)f Fm(is)h(the)h(MPI)f(mec)o(hanism)e(for)i(partitioning)f(comm)o(unication)e (space.)29 b(A)17 b(send)h(made)e(in)h(a)75 2504 y(con)o(text)e(cannot)f(b)q (e)h(receiv)o(ed)h(in)d(another)i(con)o(text.)k(Con)o(texts)c(are)g(iden)o (ti\014ed)f(in)g(MPI)g(using)g(in)o(teger-v)n(alued)75 2553 y Fl(con)o(text)p 234 2553 15 2 v 16 w(id)p Fm('s.)158 2604 y(The)k(comm)o(unication)c(domain)h(used)k(b)o(y)e(a)h(parallel)e(pro)q (cedure)k(is)d(iden)o(ti\014ed)h(b)o(y)f(a)g Fl(comm)o(unicator)p Fm(.)75 2654 y(Comm)o(uni)o(cators)c(bring)h(together)i(the)f(concepts)i(of)d (pro)q(cess)j(group)d(and)h(comm)o(unicatio)o(n)d(con)o(text.)22 b(A)14 b(com-)75 2704 y(m)o(unicator)g(is)i(an)g(explicit)g(parameter)g(in)f (eac)o(h)i(p)q(oin)o(t-to-p)q(oin)o(t)d(comm)o(unication)f(op)q(eration.)24 b(The)17 b(comm)o(u-)965 2828 y(1)p eop %%Page: 2 2 bop 75 -100 a Fm(2)1596 b Fk(2)42 b(MAPS)75 45 y Fm(nicator)16 b(iden)o(ti\014es)g(the)g(comm)o(unicatio)o(n)d(con)o(text)j(of)f(that)h(op)q (eration;)g(it)f(iden)o(ti\014es)i(the)f(group)f(of)g(pro)q(cesses)75 95 y(that)j(can)f(b)q(e)h(in)o(v)o(olv)o(ed)f(in)g(this)g(comm)o(unication;)f (and)h(it)g(pro)o(vides)h(the)g(translation)f(from)e(virtual)i(pro)q(cess)75 145 y(names,)e(whic)o(h)g(are)h(ranks)g(within)f(the)h(group,)f(in)o(to)g (absolute)h(addresses.)25 b(Collectiv)o(e)15 b(comm)o(unication)d(calls)75 195 y(also)i(tak)o(e)g(a)g(comm)o(unicator)d(as)k(parameter;)e(it)h(is)h(exp) q(ected)h(that)e(parallel)f(libraries)h(will)f(b)q(e)i(built)f(to)g(accept)75 244 y(a)g(comm)o(uni)o(cator)e(as)i(parameter.)j(Comm)o(unicators)11 b(are)k(represen)o(ted)h(b)o(y)e(opaque)g(MPI)g(ob)r(jects.)158 297 y(MPI)i(do)q(es)h(not)f(pro)o(vide)g(for)g(absolute)g(pro)q(cess)i (names.)24 b(Rather,)17 b(pro)q(cesses)h(are)f(alw)o(a)o(ys)e(iden)o (ti\014ed)h(b)o(y)75 347 y(their)d(rank)g(inside)g(a)f(group.)18 b(A)13 b(univ)o(ersal)g(group)f(that)h(includes)h(all)d(pro)q(cesses)16 b(a)o(v)n(ailable)10 b(when)k(computation)75 397 y(starts)21 b(is)g(the)g(nearest)h(equiv)n(alen)o(t)e(to)g(an)g(absolute)h(pro)q(cess)h (name)d(space)j(in)e(MPI.)g(Relativ)o(e)g(naming)e(is)75 447 y(consisten)o(t)d(with)f(MPI)g(implem)o(en)o(tations)d(where)k(pro)q(cesses)i (can)d(b)q(e)h(added)f(dynamically)l(.)158 500 y(New)g(pro)q(cess)h(groups)e (are)h(built)e(b)o(y)h(subsetting)h(and)f(reordering)h(pro)q(cesses)i(within) c(existing)h(groups)h(\(as)75 549 y(de\014ned)20 b(b)o(y)e(comm)o (unicators\).)29 b(A)19 b(new)g(group)f(is)h(de\014ned)g(from)e(an)h(old)g (group)g(b)o(y)h(sp)q(ecifying)f(the)h(corre-)75 599 y(sp)q(ondence)e(b)q(et) o(w)o(een)f(the)g(rank)e(of)h(eac)o(h)g(pro)q(cess)i(in)d(the)i(new)f(group)g (and)g(its)g(rank)f(in)h(the)g(old)g(group.)21 b(Suc)o(h)75 649 y(corresp)q(ondence)c(is)d(called)f(a)h Fl(map)p Fm(;)f(it)g(is)h (represen)o(ted)j(in)c(MPI)h(b)o(y)g(an)g(opaque)f(ob)r(ject.)75 802 y Fn(2)69 b(Maps)75 899 y Fm(A)14 b Fl(map)g Fm(is)h(a)f(one-to)g (one-mapping)e(from)h(0)p Fj(:::m)8 b Fi(\000)i Fm(1)k(in)o(to)g(the)h(set)g (of)f(non)g(negativ)o(e)h(in)o(tegers;)g Fj(m)g Fm(is)f(the)h Fl(size)75 949 y Fm(of)g(map)g Fj(f)t Fm(.)25 b(A)16 b(p)q(ossible)g (represen)o(tation)i(for)d(suc)o(h)i(map)d(is)i(a)g(list)f(of)h Fj(m)g Fm(in)o(tegers;)h(it)f(is)g(often)g(con)o(v)o(enien)o(t)g(to)75 999 y(think)d(of)g(a)g(map)e(as)j(b)q(eing)f(suc)o(h)h(list.)j(Another)e(p)q (ossible)e(represen)o(tation)i(is)e(as)h(an)f(algorithmic)d(sp)q (eci\014cation)75 1048 y(of)j(the)i(map)d(\(e.g.,)h(hash)h(function\).)k(A)c (map)e(is)i(represen)o(ted)i(b)o(y)e(an)g(opaque)g(map)e(ob)r(ject.)158 1101 y(Maps)i(are)g(used)h(in)f(order)g(to)g(represen)o(t)i(groups.)i(If)c (pro)q(cesses)i(are)f(n)o(um)o(b)q(ered)e(from)g(0)g(to)h Fj(n)9 b Fi(\000)h Fm(1,)j(then)h(a)75 1151 y(map)f Fj(f)18 b Fm(:)13 b(0)p Fj(:::m)c Fi(\000)h Fm(1)j Fi(!)f Fm(0)p Fj(:::n)d Fi(\000)h Fm(1)k(represen)o(ts)k(a)c(subset)j(group)d(of)h Fj(m)g Fm(pro)q(cesses:)23 b(pro)q(cess)16 b Fj(f)t Fm(\()p Fj(i)p Fm(\))g(has)f(rank)g Fj(i)g Fm(in)75 1201 y(the)e(new)f(group.)18 b(A)12 b(map,)e(b)o(y)i(itself,) g(do)q(es)h(not)f(represen)o(t)i(a)e(group;)g(it)g(do)q(es)h(so)f(only)f (relativ)o(e)h(to)g(a)g(pre-de\014ned)75 1251 y(con)o(taining)h(group,)g (i.e.,)f(relativ)o(e)i(to)g(another)g(ranking)f(of)h(the)g(pro)q(cesses.)158 1382 y Fh(Discussion:)38 b Fg(Should)16 b(decide)f(if)f(opaque)h(ob)r(jects)f (are)f(justi\014ed,)i(or)f(whether)g(an)g(arra)o(y)g(of)g(indices)i(is)e(OK.) f(On)75 1428 y(the)i(\\opaque)h(ob)r(ject")f(side:)21 b(Ma)o(y)15 b(ha)o(v)o(e)g(algorithmic)i(map)e(de\014nitions)i(and)f(ma)o(y)e(spread)i (the)f(de\014nition)i(o)o(v)o(er)e(more)75 1473 y(than)d(one)g(no)q(de,)h(on) f(a)f(v)o(ery)h(large)h(system.)k(On)11 b(the)h(explicit)i(represen)o(tation) g(side:)j(The)12 b(represen)o(tation)i(is)e(simple,)h(w)o(e)75 1519 y(a)o(v)o(oid)i(an)f(additional)i(ob)r(ject,)e(and)g(there)g(is)g(no)f (m)o(uc)o(h)h(w)o(aste.)k(Also,)c(with)g(an)g(explicit)i(represen)o(tation,)f (the)f(user)f(can)75 1565 y(easily)j(add)e(his/her)i(o)o(wn)e(map)g (constructors,)h(and)g(building)i(comm)o(unicators)f(is)e(faster.)20 b(Need)14 b(to)g(understand)i(what)75 1610 y(an)d(opaque)i(map)e(w)o(ould)h (b)q(e)f(in)h(F)m(ortran.)158 1663 y(If)e(maps)i(are)f(explictly)j(represen)o (ted)e(as)f(arra)o(ys,)g(then)h(one)f(needs)h(map)f(size)h(to)f(b)q(e)g(an)h (additional)i(parameter.)158 1799 y Fm(F)m(or)h(eac)o(h)g Ff(m)p Fm(,)g(an)g(iden)o(tit)o(y)f(map)f Ff(MPI)p 764 1799 14 2 v 15 w(IDENT\(m\))h Fm(of)g(size)i Ff(m)e Fm(is)h(prede\014ned:)26 b Ff(MPI)p 1475 1799 V 15 w(IDENT)p Fm(\()p Ff(m)p Fm(\))o(\()p Fj(i)p Fm(\))17 b(=)g Fj(i)g(;)7 b(i)17 b Fm(=)75 1849 y(0)p Fj(;)7 b(:::;)g(m)g Fi(\000)i Fm(1.)158 1984 y Fh(Discussion:)35 b Fg(If)11 b(maps)i(are)g(non)g(opaque)h(then)f(the)f(iden)o(t)i(map)f(is)g (just)f(a)h(long)g(arra)o(y)g(with)g Fe(i)g Fg(stored)g(at)f(the)h Fe(i)p Fg(-th)75 2034 y(en)o(try)m(,)f(and)g(it)g(is)g(just)f Fd(MPI)p 465 2034 12 2 v 13 w(IDENT)p Fg(.)e(If)i(maps)h(are)g(opaque,)h(w)o (e)e(need)h(a)f(di\013eren)o(t)j(map)d(for)h(eac)o(h)g(domain)h(size)f(\(a)f (function)75 2084 y(from)i(in)o(tegers)h(to)f(maps?\))18 b(Need)13 b(to)g(think)h(of)f(F)m(ortran)g(implemen)o(tation.)75 2299 y Fc(2.1)56 b(Op)r(erations)18 b(on)h(maps)75 2381 y Fm(The)14 b(follo)o(wing)e(op)q(erations)i(are)g(de\014ned)h(on)f(maps.)i(Eac)o(h)f (function)e(is)h(in)o(v)o(ok)o(ed)f(lo)q(cally)f(b)o(y)i(a)g(pro)q(cess.)158 2469 y Fl(MPI)p 257 2469 15 2 v 17 w(MAP)p 388 2469 V 17 w(SIZE\(map,)i (size\))75 2608 y(IN)g(map)21 b Fm(handle)13 b(to)h(map)e(ob)r(ject)75 2704 y Fl(OUT)k(size)k Fm(size)14 b(of)g(map)e(\(in)o(teger\))p eop %%Page: 3 3 bop 75 -100 a Fk(2.2)41 b(Map)13 b(constructors)1369 b Fm(3)158 45 y Fl(MPI)p 257 45 15 2 v 17 w(MAP)p 388 45 V 17 w(APPL)l(Y\()15 b(map,)h(argumen)o(t,)e(v)m(alue\))75 174 y(IN)i(map)21 b Fm(handle)13 b(to)h(map)e(ob)r(ject)75 259 y Fl(IN)k(argumen)o(t)j Fm(map)12 b(argumen)o(t)h(\(in)o(teger\))75 345 y Fl(OUT)j(v)m(alue)k Fm(image)12 b(of)h Ff(argument)f Fm(under)j Ff(map)e Fm(\(in)o(teger\))158 474 y Fl(MPI)p 257 474 V 17 w(MAP)p 388 474 V 17 w(INVERSE\()k(map,)e (argumen)o(t,)f(v)m(alue\))75 603 y(IN)i(map)21 b Fm(handle)13 b(to)h(map)e(ob)r(ject)75 688 y Fl(OUT)k(argumen)o(t)i Fm(in)o(v)o(erse)d (image)d(of)h Ff(value)g Fm(under)h Ff(map)p Fm(;)f(-1)h(if)f Ff(value)f Fm(is)i(not)g(in)f(the)i(range)f(of)f Ff(map)p Fm(.)75 773 y Fl(IN)j(v)m(alue)21 b Fm(map)12 b(v)n(alue)h(\(in)o(teger\))158 902 y Fl(MPI)p 257 902 V 17 w(MAP)p 388 902 V 17 w(LIST\(map,)i(len,)g(arra)o (y)p 850 902 V 17 w(of)p 906 902 V 17 w(in)o(teger,)e(size\))158 988 y Fm(Creates)i(an)f(explicit)f(list)h(represen)o(tation)h(of)e(a)h(map.) 75 1073 y Fl(IN)i(map)21 b Fm(handle)13 b(to)h(map)e(ob)r(ject)75 1158 y Fl(IN)k(LEN)22 b Fm(length)13 b(of)h(arra)o(y)f(\(in)o(teger\))75 1243 y Fl(OUT)j(arra)o(y)p 310 1243 V 16 w(of)p 365 1243 V 17 w(in)o(teger)i Fm(list)c(of)f(v)n(alues)g(in)h(the)g(range)h(of)e Ff(map)75 1329 y Fl(OUT)j(size)k Fm(length)13 b(of)h(returned)h(list)f({)f (size)i(of)e(map)g(\(in)o(teger\).)158 1492 y Fh(Discussion:)158 1542 y Fg(I)g(c)o(hanged)i(terminology)h(from)e(\\elemen)o(t,)h(rank")f(to)g (\\argumen)o(t,)g(v)n(alue")i(since)e(b)q(oth)h(argumen)o(t)f(and)h(v)n(alue) g(are)75 1592 y(ranks.)75 1794 y Fc(2.2)56 b(Map)19 b(constructors)75 1872 y Fm(The)14 b(map)f(constructors)i(ma)o(y)d(b)q(e)j(either)g(lo)q(cal)e (or)h(ma)o(y)d(require)k(comm)o(unication.)158 1957 y Fl(MPI)p 257 1957 V 17 w(MAP)p 388 1957 V 17 w(BUILD\(map,)g(arra)o(y)p 807 1957 V 17 w(of)p 863 1957 V 16 w(in)o(teger,)f(size\))158 2043 y Fm(Build)f(a)g(map)f(from)g(an)h(explicit)g(list)g(represen)o(tation.) 20 b(This)13 b(function)g(is)g(called)h(lo)q(cally)e(b)o(y)h(one)h(pro)q (cess.)75 2128 y Fl(OUT)i(map)k Fm(handle)14 b(to)f(map)g(ob)r(ject)75 2213 y Fl(IN)j(arra)o(y)p 259 2213 V 17 w(of)p 315 2213 V 17 w(in)o(teger)i Fm(list)13 b(of)h(v)n(alues)f(in)h(the)g(range)g(of)f Ff(map)75 2298 y Fl(IN)j(size)k Fm(n)o(um)o(b)q(er)13 b(of)h(en)o(tries)h(in) e(arra)o(y)h({)f(map)g(size)h(\(in)o(teger\))158 2419 y Fl(MPI)p 257 2419 V 17 w(MAP)p 388 2419 V 17 w(SPLIT\(comm,)h(map,)h(k)o(ey)l(,)g (index,)e(newmap\))158 2504 y Fm(This)g(is)f(a)h(collectiv)o(e)g(function)f (that)h(is)g(called)f(b)o(y)h(all)e(pro)q(cesses)17 b(in)c(the)i(group)e (asso)q(ciated)i(with)e Ff(\(comm,)75 2554 y(map\))p Fm(.)23 b(All)14 b(calling)h(pro)q(cesses)j(pro)o(vide)e(the)g(same)f(v)n(alues)g (for)h(the)g(parameters)g Ff(comm)f Fm(and)g Ff(map)p Fm(.)23 b(A)16 b(separate)75 2604 y(group)e(of)f(pro)q(cesses)k(is)d(formed)f(for)g (eac)o(h)i(distinct)f(v)n(alue)f(of)h Ff(key)p Fm(,)f(with)g(the)i(pro)q (cesses)i(ordered)e(according)f(to)75 2654 y(the)g(v)n(alue)g(of)f Ff(index)p Fm(.)k(Eac)o(h)d(pro)q(cess)i(in)d(suc)o(h)h(group)g(is)g (returned)h(in)f Ff(newmap)e Fm(the)j(map)d(of)h(this)h(group)g(within)75 2704 y Ff(comm)p Fm(.)p eop %%Page: 4 4 bop 75 -100 a Fm(4)1442 b Fk(3)42 b(CONTEXT)p 1815 -100 13 2 v 15 w(ID)75 45 y Fl(IN)16 b(comm)21 b Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75 149 y Fl(IN)h(map)21 b Fm(map)12 b(ob)r(ject)j (handle)75 253 y Fl(IN)h(k)o(ey)21 b Fm(\(in)o(teger\))75 357 y Fl(IN)16 b(index)k Fm(\(in)o(teger\))75 460 y Fl(OUT)c(newmap)k Fm(map)12 b(ob)r(ject)j(handle)158 572 y(Additional)d(map)h(creation)h (functions)g(ma)o(y)e(b)q(e)i(pro)o(vided.)k(F)m(ollo)o(ws)12 b(a)i(p)q(ossible)g(list)g(constructors.)158 710 y Fh(Discussion:)k Fg(Do)13 b(w)o(e)g(w)o(an)o(t)g(to)g(mandate)h(them)f(in)g(MPI?)158 883 y Fl(MPI)p 257 883 15 2 v 17 w(MAP)p 388 883 V 17 w(UNION\()j(map1,)g (map2,)f(newmap\))158 974 y Ff(map1)i Fm(and)g Ff(map2)g Fm(are)h(maps)e (with)h(disjoin)o(t)g(ranges.)30 b(The)18 b(resulting)f(concatenation)h(is)g (a)f(map)f(with)h(a)75 1024 y(range)d(whic)o(h)h(is)f(the)h(union)e(of)h(the) h(ranges)g(of)e Ff(map1)h Fm(and)g Ff(map2)p Fm(,)f(with)h(elemen)o(ts)g(in)g (the)g(range)h(of)f Ff(map1)f Fm(listed)75 1073 y(\014rst,)18 b(and)g(original)d(order)j(preserv)o(ed)h(within)e(eac)o(h)h(range.)28 b(The)18 b(union)f(op)q(eration)g(is)g(asso)q(ciativ)o(e)h(but)f(not)75 1123 y(comm)o(utativ)o(e.)459 1245 y Ff(newmap)o Fm(\()p Fj(i)p Fm(\))12 b(=)692 1186 y Fb(\032)744 1219 y Ff(map1)o Fm(\()p Fj(i)p Fm(\))287 b(if)13 b Fj(i)e(<)h(siz)r(e)p Fm(\()p Ff(map1)q Fm(\))744 1269 y Ff(map2)o Fm(\()p Fj(i)e Fi(\000)f Fj(siz)r(e)p Fm(\()p Ff(map1)q Fm(\)\))42 b(otherwise)75 1369 y Fl(IN)16 b(map1)21 b Fm(map)12 b(ob)r(ject)i(handle)75 1473 y Fl(IN)i(map2)21 b Fm(map)12 b(ob)r(ject)i(handle)75 1577 y Fl(OUT)i(newmap)k Fm(map)12 b(ob)r(ject)j(handle)158 1724 y Fl(MPI)p 257 1724 V 17 w(MAP)p 388 1724 V 17 w(PR)o(ODUCT\()g(map1,)h(map2,)f(range,)g (newmap\))158 1815 y Ff(map1)c Fm(is)i(a)f(map)e(with)i(v)n(alues)g(in)g(the) h(range)g(0)p Fj(;)7 b(:::;)g Ff(ran)o(ge)s Fi(\000)f Fm(1.)18 b Ff(newmap)11 b Fm(is)h(the)h(cartesian)g(pro)q(duct)g(of)f Ff(map1)75 1864 y Fm(and)k Ff(map2)p Fm(,)f(whic)o(h)h(maps)f(the)i(pair)f (\()p Fj(i;)7 b(j)r Fm(\))16 b(in)o(to)g(the)g(pair)g(\()p Ff(map1)o Fm(\()p Fj(i)p Fm(\))p Fj(;)7 b Ff(map2)p Fm(\()p Fj(j)r Fm(\)\).)25 b(P)o(airs)16 b(in)g(the)h(domain)c(and)j(in)75 1914 y(the)d(range)f(of)g(this)h(mapping)d(are)i(n)o(um)o(b)q(ered)g(in)g(ro) o(w)g(ma)r(jor)f(order,)i(i.e.,)e(pair)h(\()p Fj(i;)7 b(j)r Fm(\))13 b(has)f(n)o(um)o(b)q(er)g Fj(i)6 b Fi(\001)g Ff(range)f Fm(+)h Fj(j)r Fm(.)75 1964 y(The)14 b(pro)q(duct)h(op)q(eration)f(is)g(asso)q (ciativ)o(e)f(but)i(not)e(comm)o(utativ)o(e.)523 2079 y Ff(newmap)o Fm(\()p Fj(i)c Fi(\001)g Ff(range)f Fm(+)i Fj(j)r Fm(\))i(=)g Ff(map1)o Fm(\()p Fj(i)p Fm(\))e Fi(\001)e Ff(range)h Fm(+)g Ff(map2)o Fm(\()p Fj(j)r Fm(\))75 2181 y Fl(IN)16 b(map1)21 b Fm(map)12 b(ob)r(ject)i(handle)75 2285 y Fl(IN)i(map2)21 b Fm(map)12 b(ob)r(ject)i(handle)75 2388 y Fl(OUT)i(newmap)k Fm(map)12 b(ob)r(ject)j(handle)75 2553 y Fn(3)69 b(Con)n(text)p 424 2553 21 2 v 24 w(id)75 2654 y Fm(A)13 b Fl(con)o(text)p 278 2654 15 2 v 15 w(id)e Fm(is)i(an)f(in)o(teger.)18 b(The)13 b(range)g(of)f(v)n(alid)f(v)n(alues)h(for)g Ff(context)p 1274 2654 14 2 v 14 w(id)g Fm(is)h(implem)o(en)o(tation)d(dep)q(enden)o(t,)75 2704 y(and)k(can)g(b)q(e)g(found)g(b)o(y)f(calling)g(a)g(suitable)h(query)h (function,)e(as)h(describ)q(ed)h(in)f Fl(??)p Fm(.)p eop %%Page: 5 5 bop 75 -100 a Fk(3.1)41 b(Op)q(erations)14 b(on)g(con)o(text)p 576 -100 13 2 v 16 w(id's)1201 b Fm(5)75 45 y Fc(3.1)56 b(Op)r(erations)18 b(on)h(con)n(text)p 753 45 17 2 v 20 w(id's)75 168 y Fl(MPI)p 174 168 15 2 v 17 w(CONTEXT)p 431 168 V 19 w(ALLOCA)l(TE\(comm,)d(map,)g (arra)o(y)p 1112 168 V 16 w(of)p 1167 168 V 17 w(con)o(textids,)d(len\))158 259 y Fm(Allo)q(cates)e(an)g(arra)o(y)g(of)g(con)o(text)p 676 259 13 2 v 15 w(id's.)17 b(This)11 b(is)g(a)g(collectiv)o(e)g(op)q(eration)g (that)g(is)g(executed)i(b)o(y)e(all)f(pro)q(cesses)75 309 y(in)g(the)i(group) f(de\014ned)h(b)o(y)e Ff(\(comm,)21 b(map\))p Fm(.)16 b(The)11 b(con)o(text)p 984 309 V 16 w(ids)g(that)g(are)g(returned)h(are)g(unique)f (within)f(the)h(group)75 359 y(asso)q(ciated)i(with)f Ff(\(comm,)21 b(map\))p Fm(.)16 b(The)d(arra)o(y)f(returned)i(is)e(the)h(same)e(on)h(all)g (pro)q(cesses)i(that)f(call)e(the)i(function)75 409 y(\(same)f(order,)i(same) e(n)o(um)o(b)q(er)g(of)g(elemen)o(ts\).)18 b(The)13 b(call)f(ma)o(y)f(blo)q (c)o(k)i(un)o(til)f(all)g(pro)q(cesses)j(within)e(the)g(executing)75 458 y(group)h(ha)o(v)o(e)f(in)o(v)o(ok)o(ed)g(the)i(call.)75 573 y Fl(IN)h(comm)21 b Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75 679 y Fl(IN)h(map)21 b Fm(map)12 b(ob)r(ject)j(handle)75 786 y Fl(OUT)h(arra)o(y)p 310 786 15 2 v 16 w(of)p 365 786 V 17 w(con)o(text)p 538 786 V 16 w(ids)j Fm(\(arra)o(y)14 b(of)f(in)o(tegers\))75 892 y Fl(IN)j(len)k Fm(n)o(um)o(b)q(er)13 b(of)g(con)o(text)p 562 892 13 2 v 16 w(id's)g(to)h(allo)q(cate)f(\(in)o(teger\))158 1042 y Fl(MPI)p 257 1042 15 2 v 17 w(CONTEXT)p 514 1042 V 19 w(RESER)-5 b(VE\(con)o(text)p 931 1042 V 15 w(id\))158 1133 y Fm(Reserv)o(e)21 b(a)e(con)o(text)p 492 1133 13 2 v 16 w(id)g(v)n(alue.)34 b(This)20 b(is)f(a)g(lo)q(cal)g(call)g(that)g(is)h(in)o(v)o(ok)o(ed)e(b)o(y)i (a)f(single)g(pro)q(cess.)37 b(A)19 b(re-)75 1183 y(serv)o(ed)i(con)o(text)p 343 1183 V 15 w(id)e(v)n(alue)g(will)f(not)h(b)q(e)h(allo)q(cated)f(b)o(y)g (a)g(subsequen)o(t)i(call)d(to)h Ff(MPI)p 1457 1183 14 2 v 15 w(CONTEXT)p 1626 1183 V 15 w(ALLOCATE)e Fm(b)o(y)75 1233 y(the)g(same)e(pro)q(cess.)27 b(It)16 b(is)g(erroneous)i(to)e(reserv)o(e)i(a) e(con)o(text)p 1070 1233 13 2 v 16 w(id)f(that)i(has)f(b)q(een)h(already)f(b) q(een)h(allo)q(cated)f(b)o(y)75 1282 y Ff(MPI)p 144 1282 14 2 v 15 w(CONTEXT)p 313 1282 V 14 w(ALLOCATE)p Fm(.)75 1397 y Fl(IN)g(con)o(text)p 305 1397 15 2 v 16 w(id)k Fm(\(in)o(teger\))158 1590 y Fh(Discussion:)158 1642 y Fg(Ma)o(y)13 b(w)o(an)o(t)g(to)g(reserv)o(e) h(an)f(arra)o(y)g(of)g(v)n(alues,)h(rather)g(than)f(one)h(v)n(alue.)158 1697 y(Implemen)o(tations)i(ma)o(y)d(ha)o(v)o(e)g(there)h(o)o(wn)f(reserv)o (ed)g(v)n(alues.)19 b(The)13 b(e\013ect)g(is)g(as)h(if)f(calls)h(to)f Fd(MPI)p 1575 1697 12 2 v 13 w(CONTEXT)p 1728 1697 V 12 w(RESERVE)75 1747 y Fg(where)g(executed)h(in)g(a)f(pream)o(ble.)158 1921 y Fl(MPI)p 257 1921 15 2 v 17 w(CONTEXT)p 514 1921 V 19 w(FREE\(con)o(text)p 836 1921 V 16 w(id\))158 2012 y Fm(F)m(ree)j(a)e(reserv)o(ed)j(con)o(text)p 583 2012 13 2 v 16 w(id)d(v)n(alue.)21 b(The)15 b(v)n(alue)f(b)q(ecomes)h(a)o (v)n(ailable)e(for)h(a)h(subsequen)o(t)i(reserv)n(ation)e(b)o(y)75 2062 y Ff(MPI)p 144 2062 14 2 v 15 w(CONTEXT)p 313 2062 V 14 w(RESERVE)e Fm(or)g(subsequen)o(t)j(allo)q(cation)c(b)o(y)i Ff(MPI)p 1071 2062 V 15 w(CONTEXT)p 1240 2062 V 14 w(ALLOCATE)p Fm(.)d(It)j(is)g(erroneous)h(to)f(free)g(a)75 2112 y(v)n(alue)f(that)h(is)g (asso)q(ciated)h(with)e(an)h(activ)o(e)g(comm)o(unicator.)75 2226 y Fl(IN)i(con)o(text)p 305 2226 15 2 v 16 w(id)k Fm(\(in)o(teger\))158 2419 y Fh(Discussion:)158 2471 y Fg(Here,)12 b(to)q(o,)h(w)o(e)g(migh)o(t)h (prefer)f(to)g(free)f(an)i(arra)o(y)f(of)g(v)n(alues,)h(rather)f(than)h(one.) 158 2522 y(Comm)o(unicators,)i(lik)o(e)h(all)f(opaque)g(ob)r(jects,)f(are)g (freed)g(b)o(y)g(a)f(call)j(to)d Fd(MPI)p 1286 2522 12 2 v 13 w(FREE)p Fg(.)f(W)m(e)i(ma)o(y)g(prefer)g(to)f(ha)o(v)o(e)i(that)75 2568 y(same)c(call)g(free)g(also)g(the)g(asso)q(ciated)h(con)o(text)p 757 2568 V 14 w(id.)18 b(With)12 b(the)g(curren)o(t)g(design,)h(if)e(w)o(e)g (w)o(an)o(t)h(to)f(free)g(b)q(oth)h(comm)o(unicator)75 2614 y(and)i(con)o(text)p 275 2614 V 14 w(id)g(w)o(e)f(need)g(to)g(store)g(the)g (con)o(text)p 810 2614 V 15 w(id)h(v)n(alue,)g(free)e(the)i(comm)o(unicator,) g(next)f(free)g(the)g(con)o(text)p 1730 2614 V 15 w(id.)p eop %%Page: 6 6 bop 75 -100 a Fm(6)1328 b Fk(4)41 b(COMMUNICA)m(TORS)75 45 y Fn(4)69 b(Comm)n(unicators)75 144 y Fm(A)17 b Fl(comm)o(unicator)d Fm(is)j(an)g(opaque)g(ob)r(ject)h(that)f(iden)o(ti\014es)g(a)g(group)g(of)f (pro)q(cesses)k(and)d(a)f(comm)o(unication)75 194 y(con)o(text)i(for)f(that)g (group.)28 b(Lik)o(e)17 b(other)h(opaque)g(ob)r(jects,)g(comm)o(unicators)d (cannot)j(b)q(e)g(transfered)h(b)q(et)o(w)o(een)75 244 y(pro)q(cesses;)d(con) o(text)p 401 244 13 2 v 16 w(id's)d(are)h(used)h(to)f(transfer)h(information) c(on)i(con)o(text.)158 298 y(As)g(a)f(short-hand,)g(w)o(e)h(shall)e(iden)o (ti\014y)h(the)h(group)f(of)g(pro)q(cesses)j(asso)q(ciated)e(with)f(a)g(comm) o(unicator)e Ff(comm)75 348 y Fm(with)17 b(the)h(comm)o(unicator)d(itself.)28 b(Th)o(us,)18 b(\\the)g(pro)q(cess)h(with)f(rank)f Fj(i)h Fm(in)f Ff(comm)p Fm(")f(should)h(b)q(e)h(understo)q(o)q(d)h(as)75 398 y(meaning)14 b(\\)h(the)h(pro)q(cess)h(with)e(rank)g Fj(i)h Fm(in)f(the)h(group)f(asso)q(ciated)i(with)e Ff(comm)p Fm(".)21 b(In)16 b(the)g(same)e(manner,)h(\\the)75 447 y(comm)o(unication)7 b(con)o(text)k Ff(comm)p Fm(")e(should)h(b)q(e)h(understo)q(o)q(d)h(to)e (mean)f(\\the)i(comm)o(unication)c(con)o(text)k(asso)q(ciated)75 497 y(with)j Ff(comm)p Fm(".)158 551 y(An)19 b(initial)e(comm)o(unicator)f Ff(MPI)p 701 551 14 2 v 15 w(COMM)p 804 551 V 15 w(INIT)i Fm(is)h(de\014ned)h (when)f(the)h(program)d(starts.)34 b(Its)19 b(asso)q(ciated)75 601 y(group)e(con)o(tains)h(all)e(pro)q(cesses)k(that)e(start)g(the)g (computation.)28 b(Applications)16 b(that)i(do)f(not)h(need)g(m)o(ultiple)75 651 y(pro)q(cess)e(groups)e(or)g(m)o(ultiple)d(con)o(texts,)k(will)d(only)h (use)i(this)f(comm)o(unicator.)158 705 y(Let)f Ff(comm)g Fm(b)q(e)g(a)g(comm) o(unicator)d(that)j(is)g(asso)q(ciated)g(with)g(a)g(group)g(of)f(size)i Fj(n)p Fm(,)e(and)h(let)g Ff(map)e Fm(:)g(0)p Fj(::m)c Fi(\000)g Fm(1)k Fi(!)75 755 y Fm(0)p Fj(::n)t Fi(\000)t Fm(1)g(b)q(e)h(a)f(map.)16 b(Then)c(the)g(pair)f Ff(\(comm,)20 b(map\))11 b Fm(de\014nes)i(a)e (subgroup,)h(namely)e(the)i(subgroup)f(of)g(pro)q(cesses)75 805 y(with)j(ranks)g Ff(map)o Fm(\(0\))p Fj(;)7 b(:::;)g Ff(map)m Fm(\()p Fj(m)j Fi(\000)f Fm(1\))14 b(in)g Ff(comm)p Fm(.)75 944 y Fc(4.1)56 b(Op)r(erations)18 b(on)h(comm)n(unicators)75 1064 y Fl(MPI)p 174 1064 15 2 v 17 w(COMM)p 351 1064 V 18 w(MAP\(comm,)d(sub) q(comm,)f(map\))158 1154 y Fm(Returns)d(a)e(map)f(suc)o(h)i(that)g Ff(\(comm,)21 b(map\))9 b Fm(is)i(the)g(group)g(asso)q(ciated)g(with)f Ff(subcomm)p Fm(;)g(i.e.,)g(if)g Ff(i)g Fm(is)h(the)g(rank)75 1204 y(of)i(a)h(pro)q(cess)h(in)e(the)i(group)e(asso)q(ciated)i(with)e Ff(subcomm)p Fm(,)f(then)i Ff(map\(i\))f Fm(is)g(the)i(rank)e(of)g(that)h (same)f(pro)q(cess)i(in)75 1254 y(the)g(group)e(asso)q(ciated)i(with)f Ff(comm)p Fm(.)j(The)e(call)e(is)h(erroneous)h(if)f Ff(subcomm)e Fm(has)i(a)g(pro)q(cess)i(that)e(is)g(not)g(mem)o(b)q(er)75 1304 y(of)f Ff(comm)p Fm(.)75 1412 y Fl(IN)j(comm)21 b Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75 1513 y Fl(IN)h(sub)q(comm)k Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75 1613 y Fl(OUT)h(map)k Fm(map)12 b(ob)r(ject)j(handle) 158 1722 y(The)h Ff(MPI)p 314 1722 14 2 v 15 w(COMM)p 417 1722 V 15 w(MAP)f Fm(function)g(retriev)o(es)i(the)f(group)f(of)g(pro)q(cesses)j (that)e(is)f(asso)q(ciated)i(with)e(a)g(comm)o(u-)75 1772 y(nicator.)36 b(Ho)o(w)o(ev)o(er,)21 b(since)f(absolute)g(pro)q(cess)i(names)d(are)h(not)g (visible)f(in)g(MPI,)g(the)i(group)e(can)h(only)f(b)q(e)75 1822 y(de\014ned)h(relativ)o(e)e(to)h(another)g(encompassing)f(group.)33 b(The)19 b(\\absolute")f(pro)q(cess)j(n)o(um)o(b)q(er)d(is)h(obtained)f(b)o (y)75 1872 y(using)c Ff(MPI)p 253 1872 V 15 w(COMM)p 356 1872 V 15 w(INIT)f Fm(as)h(a)f(reference)j(comm)o(unicator.)158 1961 y Fl(MPI)p 257 1961 15 2 v 17 w(COMM)p 434 1961 V 18 w(SIZE\(comm,)g (size\))158 2051 y Fm(Returns)f(the)f(size)h(of)e(the)i(group)e(asso)q (ciated)i(with)f Ff(comm)p Fm(.)75 2160 y Fl(IN)i(comm)21 b Fm(handle)14 b(to)f(comm)o(unicator)75 2260 y Fl(OUT)j(size)k Fm(group)13 b(size)i(\(in)o(teger\))158 2405 y Fl(MPI)p 257 2405 V 17 w(COMM)p 434 2405 V 18 w(CONTEXTID\(comm,)i(con)o(text)p 1077 2405 V 16 w(id\))158 2494 y Fm(Returns)e(the)f(con)o(text)p 522 2494 13 2 v 16 w(id)f(asso)q(ciated)i(with)f(the)g(comm)o(unicator)d Ff(comm)p Fm(.)75 2603 y Fl(IN)16 b(comm)21 b Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75 2704 y Fl(OUT)h(con)o(text)p 356 2704 15 2 v 15 w(id)k Fm(con)o(text)p 564 2704 13 2 v 16 w(id)p eop %%Page: 7 7 bop 75 -100 a Fk(4.2)41 b(Comm)n(unicator)11 b(constructors)1182 b Fm(7)75 45 y Fc(4.2)56 b(Comm)n(unicator)19 b(constructors)75 157 y Fl(MPI)p 174 157 15 2 v 17 w(COMM)p 351 157 V 18 w(MAKE\(comm,)e(map,)f (con)o(text)p 967 157 V 15 w(id,)f(new)o(comm\))158 242 y Fm(Creates)20 b(a)f(new)g(comm)o(unicator)d(ob)r(ject,)k(whic)o(h)f(is)g(asso)q(ciated)g (with)g(the)g(group)g(de\014ned)h(b)o(y)e Ff(\(comm,)75 292 y(map\))p Fm(,)e(and)g(the)h(con)o(text)p 483 292 13 2 v 15 w(id)f Ff(context)p 704 292 14 2 v 15 w(id)p Fm(.)25 b(This)16 b(is)g(a)g(lo)q(cal)g(call)f(executed)j(b)o(y)f(one)f(pro)q(cess.)27 b(Ho)o(w)o(ev)o(er,)17 b(the)75 342 y(new)c(comm)o(unicator)e(ob)r(ject)i (should)g(not)g(b)q(e)g(used)h(for)f(comm)o(unicatio)o(n)d(b)q(et)o(w)o(een) 15 b(t)o(w)o(o)d(pro)q(cesses)k(unless)d(they)75 392 y(b)q(oth)h(ha)o(v)o(e)g (called)f(the)i(function.)158 442 y(It)h(is)g(erroneous)h(to)f(create)h(on)f (the)g(same)f(pro)q(cess)j(t)o(w)o(o)d(distinct)h(comm)o(unicators)e(with)h (the)i(same)e(con-)75 491 y(text)p 149 491 13 2 v 16 w(id.)75 582 y Fl(IN)h(comm)21 b Fm(comm)o(unicator)11 b(ob)r(ject)k(handle)75 665 y Fl(IN)h(map)21 b Fm(map)12 b(ob)r(ject)j(handle)75 747 y Fl(IN)h(con)o(text)p 305 747 15 2 v 16 w(id)k Fm(con)o(text)p 514 747 13 2 v 15 w(id)75 830 y Fl(OUT)c(new)o(comm)k Fm(comm)o(unicator)11 b(ob)r(ject)j(handle)158 920 y(The)c(con)o(tect)p 370 920 V 16 w(id)f(used)h(in)f(this)g(call)g(ma)o(y)e(b)q(e)j(an)f(in)o(teger)h (returned)h(b)o(y)e(a)g(previous)g(call)g(to)g Ff(MPI)p 1627 920 14 2 v 15 w(ALLOC)p 1752 920 V 15 w(CONTEXT)p Fm(,)75 970 y(executed)18 b(within)d(the)h(same)f(group.)23 b(In)16 b(suc)o(h)h(case,)f (it)g(is)f(guaran)o(teed)i(that)e(the)i(con)o(text)p 1552 970 13 2 v 15 w(id)f(is)f(unique,)h(and)75 1020 y(has)i(not)f(b)q(een)i(used)f (already)f(to)h(create)h(a)e(comm)o(unicator.)26 b(Ho)o(w)o(ev)o(er,)18 b(the)h(user)f(is)g(free)g(to)f(manage)f(on)h(it)75 1070 y(o)o(wn)e(the)h (con)o(text)p 371 1070 V 16 w(id's,)f(and)g(use)i(other)f(mec)o(hanisms)e (for)h(their)h(allo)q(cation.)21 b(F)m(or)16 b(example,)e(one)i(could)f(ha)o (v)o(e)75 1119 y(a)f(single)g Fl(con)o(text)p 385 1119 15 2 v 16 w(id)h(serv)o(er)e Fm(generate)j(unique)e(con)o(text)p 1031 1119 13 2 v 16 w(id's)g(for)g(the)h(en)o(tire)g(system;)f(one)g(can)h (preallo)q(cate)75 1169 y(statically)e(some)g(con)o(text)p 493 1169 V 16 w(id)g(v)n(alues,)g(for)h(the)g(use)h(of)e(libraries;)g(etc.) 158 1298 y Fh(Discussion:)158 1343 y Fg(Do)g(w)o(e)f(w)o(an)o(t)g(to)g (prohibit)j(implemen)o(tations)g(that)e(sync)o(hronize)h(the)f(creation)g(of) f(comm)o(unicators)j(\(i.e.,)c(where)i(a)75 1389 y(call)h(to)f Fd(MPI)p 254 1389 12 2 v 13 w(MAKE)p 347 1389 V 13 w(COMM)e Fg(blo)q(c)o(ks)k(un)o(til)f(all)h(mem)o(b)q(ers)e(of)g(the)g(group)h(ha)o(v) o(e)f(made)h(the)f(call\)?)158 1435 y(Is)g(it)g(preferable)i(to)d(ha)o(v)o(e) i(a)f Fd(array)p 686 1435 V 12 w(of)p 738 1435 V 14 w(comm)e Fg(as)i(parameter,)g(rather)h(than)f(one)h(comm)o(unicator?)158 1484 y(W)m(e)g(don't)h(ha)o(v)o(e)g(no)o(w)f(the)h(abilit)o(y)h(to)f(ha)o(v)o (e)g(t)o(w)o(o)e(di\013eren)o(t)j(groups)g(with)e(the)h(same)f(con)o(text;)i (e.g.,)d(t)o(w)o(o)h(di\013eren)o(t)75 1534 y(n)o(um)o(b)q(erings)i(of)d(the) h(pro)q(cesses,)g(used)g(with)g(the)g(same)g(comm)o(unication)i(con)o(text.)j (The)14 b(alternativ)o(e)h(is)f(to)g(remo)o(v)o(e)g(the)75 1584 y(restriction)f(that)e(t)o(w)o(o)g(distinct)i(comm)o(unicators)g(cannot) f(use)f(the)g(same)h(con)o(text)p 1282 1584 V 14 w(id.)17 b(It)11 b(is)h(no)o(w)f(the)g(user)h(resp)q(onsibili)q(t)o(y)75 1634 y(to)17 b(disam)o(biguate)i(these)f(t)o(w)o(o)e(comm)o(unicators;)k(the)d (system)g(do)q(es)h(not)f(guaran)o(tee)h(that)f(a)g(message)g(sen)o(t)g(with) g(one)75 1684 y(comm)o(unicator)d(can)f(b)q(e)g(receiv)o(ed)h(only)f(with)g (this)h(same)e(comm)o(unicator,)i(if)f(they)g(use)f(the)h(same)g(con)o(text)p 1668 1684 V 14 w(id.)18 b(W)m(e)12 b(sa)o(v)o(e)75 1733 y(on)i(con)o(text)p 254 1733 V 15 w(id's)g(but)g(lo)q(ose)h(safet)o(y)f(\(the)g(sender)h(and)f (receiv)o(er)h(ma)o(y)f(b)q(e)g(using)h(di\013eren)o(t)h(pro)q(cess)e(n)o(um) o(b)q(erings,)i(if)e(t)o(w)o(o)75 1783 y(di\013eren)o(t)h(comm)o(unicators)f (are)g(used)f({)g(the)g(receiv)o(er)h(ma)o(y)g(not)f(iden)o(tify)i(correctly) f(the)f(sender\).)75 2003 y Fn(5)69 b(W)-6 b(orking)24 b(without)e(con)n (text)p 940 2003 21 2 v 24 w(id's)75 2094 y Fm(It)9 b(is)h(p)q(ossible)f(to)g (create)i(comm)o(unicators)c(directly)m(,)j(without)f(using)g(con)o(text)p 1296 2094 13 2 v 15 w(id's.)16 b(A)10 b(single)f(function)g(com)o(bines)75 2144 y(the)k(allo)q(cation)e(of)g(a)h(new)h(con)o(text)p 631 2144 V 16 w(id)e(and)h(the)h(generation)g(of)e(a)h(new)h(comm)o(unicator.)i (This)d(allo)o(ws)f(a)h(\\naiv)o(e")75 2194 y(user)22 b(that)e(do)q(es)i(not) e(need)i(to)f(customize)f(con)o(text)p 942 2194 V 16 w(id)g(allo)q(cation)f (to)h(ignore)h(this)f(MPI)h(feature.)39 b(Direct)75 2243 y(comm)o(unicator)11 b(creation)j(is)g(safer,)g(since)h(uniqueness)g(of)e(con)o(text)p 1150 2243 V 16 w(id's)g(is)h(guaran)o(teed,)g(b)o(y)g(construction.)158 2329 y Fl(MPI)p 257 2329 15 2 v 17 w(COMM)p 434 2329 V 18 w(SAFEMAKE\(comm,)j (map,)f(new)o(comm\))158 2414 y Fm(Creates)h(a)f(new)g(comm)o(unicator)d (\(with)j(an)f(attac)o(hed)i(con)o(text)p 1186 2414 13 2 v 15 w(id\))f(that)g(is)g(asso)q(ciated)g(with)g(the)g(group)75 2464 y(de\014ned)h(b)o(y)g(the)f(pair)g Ff(\(comm,)21 b(map\))p Fm(.)j(This)17 b(is)f(a)g(collectiv)o(e)g(call)g(that)g(has)g(to)h(b)q(e)g (in)o(v)o(ok)o(ed)e(b)o(y)h(all)f(mem)o(b)q(ers)75 2514 y(in)h(this)g(group.) 24 b(All)15 b(of)h(them)f(are)i(pro)o(viding)d(the)j(same)e(input)h (parameters)g Ff(comm)f Fm(and)h Ff(map)p Fm(.)24 b(The)17 b(call)e(ma)o(y)75 2563 y(blo)q(c)o(k)g(un)o(til)g(all)f(pro)q(cesses)k(in)d (the)h(group)f(ha)o(v)o(e)h(in)o(v)o(ok)o(ed)e(it.)23 b(The)16 b(new)g(comm)o(unicator)c(ma)o(y)i(b)q(e)i(safely)f(used)75 2613 y(for)f(comm)o(uni)o(cation)d(with)j(an)o(y)f(mem)o(b)q(er)f(of)h(the)i (new)f(group,)g(once)g(the)h(call)e(returns.)75 2704 y Fl(IN)j(comm)21 b Fm(comm)o(unicator)11 b(handle)p eop %%Page: 8 8 bop 75 -100 a Fm(8)710 b Fk(6)42 b(W)o(ORKING)13 b(WITHOUT)h(MAPS)g(AND)g (CONTEXT)p 1780 -100 13 2 v 16 w(ID'S)75 45 y Fl(IN)i(map)21 b Fm(map)12 b(handle)75 126 y Fl(OUT)k(new)o(comm)k Fm(comm)o(unicator)11 b(handle)158 213 y Ff(MPI)p 227 213 14 2 v 15 w(COMM)p 330 213 V 15 w(SAFEMAKE)h Fm(is)75 300 y Ff(MPI_CONTEXT_ALLOC)o(\(comm)o(,)19 b(map,)i(context_id,)f(1\);)75 350 y(MPI_COMM_MAKE\(com)o(m,)f(map,)i (context_id,)e(newcomm\);)75 400 y(``MPI_SYNCH\(comm,)o(map\)')o(')g(/*)i(by) h(which)f(we)g(mean)g(a)g(synchronization)533 450 y(operation)f(in)h(the)g (subgroup)f(defined)h(by)g(\(comm,map\))f(*/)158 615 y Fh(Discussion:)158 661 y Fg(Do)g(w)o(e)e(prefer)i(to)f(create)g(an)h(arra)o(y)g(of)f(comm)o (unicators,)j(rather)e(than)f(one?)37 b(\(T)m(o)19 b(b)q(e)g(consisten)o(t)i (with)f(the)75 707 y Fd(MPI)p 137 707 12 2 v 13 w(CONTEXT)p 290 707 V 11 w(ALLOC)11 b Fg(function\).)158 752 y Fd(MPI)p 220 752 V 13 w(SYNCH\(comm)o(,m)o(ap\))e Fg(is)k(not)g(curren)o(tly)h (de\014ned)f(in)h(the)f(collectiv)o(e)i(comm)o(unication)g(library;)f(it)f (can)g(b)q(e)g(co)q(ded)75 798 y(using)h(p)q(oin)o(t-to-p)q(oin)o(t)i(calls,) e(but)g(seems)f(imp)q(ortan)o(t)h(enough)g(to)f(b)q(e)h(included)h(there.)158 848 y(General)e(though)o(t)f({)g(if)f(w)o(e)g(supp)q(ort)i(collectiv)o(e)g (comm)o(unication)i(calls)d(with)g(a)g(map)f(as)h(an)g(additional)i (parameter,)75 898 y(then)h(w)o(e)f(ha)o(v)o(e)h(disasso)q(ciated)i(the)e (\\con)o(text")g(of)f(a)h(collectiv)o(e)h(comm)o(unication,)h(from)e(its)g (supp)q(orting)h(\\group".)23 b(W)m(e)75 948 y(need)14 b(to)f(do)g(it)g(when) h(w)o(e)e(build)j(new)e(comm)o(unicators,)i(in)f(order)f(to)g(b)q(o)q (otstrap)h({)f(if)h(w)o(e)e(w)o(an)o(t)h(the)g(call)i(that)e(generates)75 997 y(a)g(new)g(comm)o(unicator)i(to)e(in)o(v)o(olv)o(e)i(only)f(the)f(pro)q (cesses)h(in)g(the)f(new)g(group.)75 1217 y Fn(6)69 b(W)-6 b(orking)24 b(without)e(maps)h(and)g(con)n(text)p 1251 1217 21 2 v 25 w(id's)75 1307 y Fm(It)14 b(is)g(p)q(ossible)g(to)f(a)o(v)o(oid)g (b)q(oth)h(the)g(use)h(of)e(map)g(ob)r(jects)i(and)e(of)h(con)o(text)p 1264 1307 13 2 v 15 w(id's)g(altogether.)k(A)c(single)f(function)75 1357 y(com)o(bines)20 b(the)h(generation)g(of)f(a)h(map,)f(the)h(allo)q (cation)e(of)i(a)f(con)o(text)p 1263 1357 V 16 w(id)g(and)h(the)g(generation) g(of)f(a)h(new)75 1407 y(comm)o(unicator.)14 b(This)e(allo)o(ws)e(\\naiv)o (e")g(users)j(to)e(use)h(MPI)g(without)f(b)q(eing)g(exp)q(osed)i(to)e(maps)f (or)h(con)o(text)p 1787 1407 V 16 w(id's.)158 1492 y Fl(MPI)p 257 1492 15 2 v 17 w(COMM)p 434 1492 V 18 w(BUILD\(comm,)k(arra)o(y)p 889 1492 V 17 w(of)p 945 1492 V 16 w(ranks,)h(size,)f(new)o(comm\))158 1578 y Fm(Build)j(a)f(map)g(from)f(an)h(explicit)h(list)g(represen)o(tation.) 31 b(This)18 b(is)g(a)g(collectiv)o(e)g(function)f(that)h(is)g(called)75 1627 y(b)o(y)d(all)e(mem)o(b)q(ers)h(of)g(the)h(new)g(group.)21 b(All)14 b(pro)o(vide)g(the)i(same)e(parameters)g(for)h Ff(comm)p Fm(,)f Ff(array)p 1614 1627 14 2 v 14 w(of)p 1672 1627 V 15 w(ranks)g Fm(and)75 1677 y Ff(size)p Fm(.)j(The)d(call)f(ma)o(y)e(blo)q(c)o (k)j(un)o(til)e(it)i(w)o(as)f(in)o(v)o(ok)o(ed)g(on)g(all)f(the)j(group)e (mem)o(b)q(ers.)k(When)c(it)h(returns,)g(the)h(new)75 1727 y(comm)o(unicator)c(ma)o(y)h(b)q(e)j(used)f(to)g(comm)o(unicate)e(with)h(an)o (y)h(mem)o(b)q(er)e(of)h(the)i(new)f(group.)75 1807 y Fl(IN)i(comm)21 b Fm(handle)14 b(to)f(comm)o(unicator)75 1888 y Fl(IN)j(arra)o(y)p 259 1888 15 2 v 17 w(of)p 315 1888 V 17 w(ranks)k Fm(list)13 b(of)h(v)n(alues)f(in)h(the)g(range)g(of)g Ff(map)75 1969 y Fl(IN)i(size)k Fm(n)o(um)o(b)q(er)13 b(of)h(en)o(tries)h(in)e(arra)o(y)h({)f (map)g(size)h(\(in)o(teger\))75 2050 y Fl(OUT)i(new)o(comm)k Fm(handle)13 b(to)h(new)g(comm)o(unicator)158 2130 y Ff(MPI)p 227 2130 14 2 v 15 w(COMM)p 330 2130 V 15 w(BUILD)f Fm(is)75 2217 y Ff(MPI_MAP_BUILD\(com)o(m,)19 b(array_of_ranks,)g(size,)h(newmap\);)75 2267 y(MPI_COMM_SAFEMAKE)o(\(comm)o(,)f(newmap,)h(newcomm\);)158 2389 y Fl(MPI)p 257 2389 15 2 v 17 w(COMM)p 434 2389 V 18 w(COPY\(comm,)c (new)o(comm\))158 2475 y Fm(Creates)d(a)f(new)g(comm)o(unicator)d(with)j(the) h(same)e(group)g(as)h(the)h(old)e(comm)o(unicator.)k(This)d(is)f(a)h (collectiv)o(e)75 2524 y(function)17 b(that)g(is)h(called)f(b)o(y)g(all)f (mem)o(b)q(ers)g(of)h(new)h(group.)28 b(All)16 b(pro)o(vide)h(the)h(same)f (parameters)g(for)g Ff(comm)p Fm(.)75 2574 y(The)j(call)f(ma)o(y)f(blo)q(c)o (k)h(un)o(til)g(it)g(w)o(as)h(in)o(v)o(ok)o(ed)f(on)g(all)g(the)h(group)g (mem)o(b)q(ers.)34 b(When)20 b(it)f(returns,)j(the)f(new)75 2624 y(comm)o(unicator)11 b(ma)o(y)h(b)q(e)j(used)f(to)g(comm)o(unicate)e (with)h(an)o(y)h(mem)o(b)q(er)e(of)h(the)i(group.)75 2704 y Fl(IN)h(comm)21 b Fm(handle)14 b(to)f(comm)o(unicator)p eop %%Page: 9 9 bop 1854 -100 a Fm(9)75 45 y Fl(OUT)16 b(new)o(comm)k Fm(handle)13 b(to)h(new)g(comm)o(unicator)158 127 y Ff(MPI)p 227 127 14 2 v 15 w(COMM)p 330 127 V 15 w(COPY)f Fm(is)75 217 y Ff(MPI_COMM_SIZE\(com)o (m,)19 b(size\);)75 267 y(MPI_COMM_SAFEMAKE)o(\(comm)o(,)g (MPI_IDENT\(size\),)g(newcomm\);)158 392 y Fl(MPI)p 257 392 15 2 v 17 w(COMM)p 434 392 V 18 w(SPLIT\(comm,)c(k)o(ey)l(,)h(index,)f(new)o (comm\))158 477 y Fm(Split)g(the)h(group)g(asso)q(ciated)g(with)g Ff(comm)p Fm(;)f(creates)j(a)d(new)h(group)g(for)f(eac)o(h)i(distinct)f(v)n (alue)f(of)g Ff(key)g Fm(that)75 527 y(con)o(tains)d(the)h(pro)q(cesses)i (that)d(supplied)g(that)g(k)o(ey)g(v)n(alue;)g(the)h(pro)q(cesses)h(are)f (rank)o(ed)f(according)g(to)g(the)h Ff(index)75 577 y Fm(v)n(alues)f(they)h (supplied.)18 b(A)13 b(new)g(comm)o(unicator)d(is)i(created)i(for)e(eac)o(h)h (subgroup.)18 b(Eac)o(h)13 b(pro)q(cess)i(is)d(returned)i(a)75 627 y(handle)e(to)g(the)h(comm)o(unicator)c(for)j(the)h(new)g(subgroup)f(it)g (b)q(elongs)g(to.)17 b(This)c(is)f(a)f(collectiv)o(e)i(call)e(executed)j(b)o (y)75 676 y(all)c(pro)q(cesses)15 b(in)c(the)h(group)f(asso)q(ciated)i(with)e Ff(comm)p Fm(.)16 b(They)c(call)f(ma)o(y)f(blo)q(c)o(k)h(un)o(til)g(all)f (pro)q(cesses)k(ha)o(v)o(e)e(in)o(v)o(ok)o(ed)75 726 y(the)j(function.)21 b(When)15 b(the)g(call)f(returns)i(the)g(new)f(comm)o(unicator)d(ma)o(y)h(b)q (e)i(safely)f(used)i(for)e(comm)o(unication)75 776 y(in)f(the)i(new)f(group.) 75 866 y Fl(IN)i(comm)21 b Fm(handle)14 b(to)f(comm)o(unicator)75 948 y Fl(IN)j(k)o(ey)21 b Fm(\(in)o(teger\))75 1031 y Fl(IN)16 b(index)k Fm(\(in)o(teger\))75 1113 y Fl(OUT)c(new)o(comm)k Fm(handle)13 b(to)h(new)g(comm)o(unicator)158 1203 y Ff(MPI)p 227 1203 14 2 v 15 w(COMM)p 330 1203 V 15 w(SPLIT\(comm,)19 b(key,)i(index,)g(newcomm\))12 b Fm(is)75 1293 y Ff(MPI_COMM_SIZE\(com)o(m,) 19 b(size\);)75 1343 y(MPI_MAP_SPLIT\(com)o(m,)g(MPI_IDENT\(size\),)f(key,)j (index,)g(newmap\);)75 1393 y(MPI_COMM_SAFEMAKE)o(\(comm)o(,)e(newmap,)h (newcomm\);)158 1561 y Fh(Discussion:)158 1607 y Fg(Ma)o(y)13 b(w)o(an)o(t)f(a)g(sp)q(ecial)i(DONTCARE)e(k)o(ey)g(v)n(alue)h(that)g (indicates)h(that)f(the)f(caller)h(need)g(not)g(ha)o(v)o(e)f(a)h(new)f(comm)o (u-)75 1652 y(nicator)75 1872 y Fn(7)69 b(Examples)75 1963 y Fm(W)m(e)20 b(sa)o(y)g(that)h(a)f(parallel)f(pro)q(cedure)k(is)d Fa(active)h Fm(at)f(a)g(pro)q(cess)i(if)e(the)h(pro)q(cess)h(b)q(elongs)f(to) f(a)h(group)f(that)75 2013 y(ma)o(y)13 b(collectiv)o(ely)h(execute)i(the)f (pro)q(cedure,)i(and)d(some)g(mem)o(b)q(er)f(of)h(that)g(group)h(is)f(curren) o(tly)i(executing)f(the)75 2063 y(pro)q(cedure)j(co)q(de.)27 b(If)16 b(a)h(parallel)e(pro)q(cedure)j(is)f(activ)o(e)f(at)h(a)f(pro)q (cess,)j(then)e(this)f(pro)q(cess)j(ma)o(y)14 b(b)q(e)k(receiving)75 2113 y(messages)10 b(p)q(ertaining)f(to)h(this)g(pro)q(cedure,)i(ev)o(en)e (if)f(it)g(do)q(es)i(not)e(curren)o(tly)i(execute)h(the)e(co)q(de)h(of)e (this)h(pro)q(cedure.)75 2228 y Fc(7.1)56 b(Nonreen)n(tran)n(t)18 b(parallel)j(pro)r(cedures)75 2305 y Fm(This)13 b(co)o(v)o(ers)h(the)f(case)h (where,)g(at)f(an)o(y)g(p)q(oin)o(t)f(in)h(time,)e(at)i(most)f(one)h(in)o(v)o (ok)n(ation)e(of)h(a)h(parallel)f(pro)q(cedure)j(can)75 2355 y(b)q(e)e(activ)o(e)g(at)g(an)o(y)f(pro)q(cess.)20 b(I.e.,)12 b(concurren)o(t)j(in)o(v)o(ok)n(ations)c(of)h(the)h(same)f(parallel)g(pro)q (cedure)j(ma)o(y)c(o)q(ccur)j(only)75 2405 y(within)h(disjoin)o(t)h(groups)g (of)g(pro)q(cesses.)27 b(F)m(or)16 b(example,)f(all)g(in)o(v)o(ok)n(ations)f (of)i(parallel)f(pro)q(cedures)j(in)o(v)o(olv)o(e)d(all)75 2455 y(pro)q(cesses,)h(pro)q(cesses)h(are)d(single-threaded,)g(and)g(there)h (are)f(no)g(recursiv)o(e)h(in)o(v)o(ok)n(ations.)158 2504 y(In)e(suc)o(h)h(a) f(case,)h(a)f(con)o(text)p 604 2504 13 2 v 15 w(id)g(can)g(b)q(e)h (statically)e(allo)q(cated)h(to)g(eac)o(h)h(pro)q(cedure.)19 b(The)14 b(static)g(allo)q(cation)75 2554 y(can)c(b)q(e)h(done)f(in)g(a)g (pream)o(ble,)f(as)h(part)g(of)g(initialization)d(co)q(de.)17 b(Or,)11 b(it)f(can)g(b)q(e)h(done)f(a)g(compile/link)d(time,)i(if)g(the)75 2604 y(implemen)o(tatio)o(n)h(has)j(additional)f(mec)o(hanisms)e(to)j(reserv) o(e)i(con)o(text)p 1190 2604 V 16 w(id)d(v)n(alues.)18 b(Comm)n(unicators)11 b(to)i(b)q(e)g(used)75 2654 y(b)o(y)f(the)g(di\013eren)o(t)h(pro)q(cedures)h (can)e(b)q(e)g(build)f(in)h(a)f(pream)o(ble,)g(if)g(the)i(executing)f(groups) g(are)g(statically)f(de\014ned;)75 2704 y(if)17 b(the)i(executing)f(groups)h (c)o(hange)f(dynamically)m(,)d(then)k(a)e(new)i(comm)o(unicator)c(has)j(to)g (b)q(e)g(built)g(whenev)o(er)p eop %%Page: 10 10 bop 75 -100 a Fm(10)1584 b Fk(8)42 b(LEFT)75 45 y Fm(the)15 b(executing)h(group)e(c)o(hanges,)h(but)g(this)g(new)g(comm)o(uni)o(cator)d (can)j(b)q(e)g(built)f(using)h(the)g(same)f(preallo)q(cated)75 95 y(con)o(text)p 210 95 13 2 v 16 w(id.)19 b(If)14 b(the)h(parallel)e(pro)q (cedures)j(can)f(b)q(e)g(organized)f(in)o(to)g(libraries,)f(so)i(that)f(only) g(one)g(pro)q(cedure)j(of)75 145 y(eac)o(h)c(library)e(can)i(b)q(e)g (concurren)o(tly)g(activ)o(e)g(at)f(eac)o(h)h(pro)q(cessor,)h(then)f(it)f(is) g(su\016cien)o(t)h(to)f(allo)q(cate)g(one)g(con)o(text)75 195 y(p)q(er)j(library)m(.)75 318 y Fc(7.2)56 b(P)n(arallel)14 b(pro)r(cedures)e(that)g(are)g(nonreen)n(tran)n(t)h(within)h(eac)n(h)f (executing)f(group)75 397 y Fm(This)i(co)o(v)o(ers)i(the)f(case)h(where,)f (at)g(an)o(y)f(p)q(oin)o(t)g(in)g(time,)f(for)h(eac)o(h)h(pro)q(cess)i (group,)d(there)i(can)f(b)q(e)g(at)f(most)g(one)75 447 y(activ)o(e)h(in)o(v)o (ok)n(ation)e(of)h(a)g(parallel)g(pro)q(cedure)j(b)o(y)d(a)h(pro)q(cess)i (mem)o(b)q(er.)i(Ho)o(w)o(ev)o(er,)c(it)g(migh)o(t)d(b)q(e)k(p)q(ossible)f (that)75 497 y(the)e(same)e(pro)q(cedure)j(is)e(concurren)o(tly)i(in)o(v)o (ok)o(ed)d(in)h(t)o(w)o(o)g(partially)e(\(or)j(completely\))e(o)o(v)o (erlapping)g(groups.)17 b(F)m(or)75 547 y(example,)12 b(the)h(same)g (collectiv)o(e)g(comm)o(unicati)o(on)d(function)j(ma)o(y)e(b)q(e)j(concurren) o(tly)h(in)o(v)o(ok)o(ed)d(on)h(t)o(w)o(o)g(partially)75 596 y(o)o(v)o(erlapping)g(groups.)158 648 y(In)e(suc)o(h)i(a)e(case,)h(a)f(con)o (text)p 595 648 V 16 w(id)g(is)g(asso)q(ciated)h(with)f(eac)o(h)h(parallel)e (pro)q(cedure)j(and)f(eac)o(h)f(executing)i(group,)75 697 y(so)k(that)h(o)o (v)o(erlapping)e(execution)i(groups)f(ha)o(v)o(e)g(distinct)h(comm)o(unicati) o(on)c(con)o(texts.)30 b(\(One)18 b(do)q(es)g(not)f(need)75 747 y(a)f(di\013eren)o(t)i(con)o(text)p 414 747 V 16 w(id)e(from)f(eac)o(h)i (group;)h(one)f(merely)e(needs)j(a)f(\\coloring")e(of)h(the)i(groups,)f(so)f (that)h(One)75 797 y(can)c(generate)i(the)f(comm)o(unicators)d(for)i(eac)o(h) g(parallel)f(pro)q(cedure)k(when)d(the)h(execution)g(groups)g(are)f (de\014ned.)75 847 y(Here,)i(again,)d(one)i(only)g(need)h(one)f(con)o(text)g (for)g(eac)o(h)h(library)m(,)d(if)h(no)h(t)o(w)o(o)f(pro)q(cedures)k(from)12 b(the)j(same)e(library)75 897 y(can)h(b)q(e)h(concurren)o(tly)g(activ)o(e)f (in)f(the)i(same)e(group.)158 948 y(Note)18 b(that,)f(for)g(collectiv)o(e)h (comm)o(unicati)o(on)c(libraries,)k(w)o(e)f(do)g(allo)o(w)f(sev)o(eral)i (concurren)o(t)h(in)o(v)o(ok)n(ations)75 998 y(within)e(the)i(same)e(group:) 25 b(a)18 b(broadcast)g(in)g(a)f(group)h(ma)o(y)e(b)q(e)j(started)g(at)e(a)h (pro)q(cess)h(b)q(efore)g(the)g(previous)75 1047 y(broadcast)h(in)f(that)h (group)g(ended)g(at)g(another)g(pro)q(cess.)37 b(In)20 b(suc)o(h)g(a)g(case,) h(one)f(cannot)g(rely)g(on)f(con)o(text)75 1097 y(mec)o(hanisms)12 b(to)i(disam)o(biguate)e(successiv)o(e)17 b(in)o(v)o(ok)n(ations)12 b(of)h(the)i(same)f(parallel)f(pro)q(cedure)j(within)d(the)i(same)75 1147 y(group:)31 b(the)21 b(pro)q(cedure)i(need)e(b)q(e)h(implem)o(en)o(ted)d (so)h(as)h(to)f(a)o(v)o(oid)g(confusion.)37 b(E.g.,)21 b(for)f(broadcast,)j (one)75 1197 y(ma)o(y)13 b(need)k(to)e(carry)h(additional)d(information)f(in) j(messages,)h(suc)o(h)g(as)f(the)h(broadcast)g(ro)q(ot,)f(to)g(help)g(in)g (suc)o(h)75 1247 y(disam)o(biguation;)9 b(one)i(also)g(relies)h(on)f(preserv) n(ation)h(of)f(message)g(order)i(b)o(y)e(MPI.)g(With)g(suc)o(h)h(an)f (approac)o(h,)h(w)o(e)75 1297 y(ma)o(y)g(b)q(e)i(gaining)e(p)q(erformance,)h (but)h(w)o(e)g(lo)q(ose)g(mo)q(dularit)o(y)m(.)h(It)f(is)f(not)h(su\016cien)o (t)g(to)g(implemen)o(t)d(the)j(parallel)75 1346 y(pro)q(cedure)f(so)e(that)h (it)e(w)o(orks)i(correctly)g(in)f(isolation,)e(when)j(in)o(v)o(ok)o(ed)e (only)h(once;)h(it)f(needs)h(to)f(b)q(e)h(implemen)o(ted)75 1396 y(so)j(that)h(an)o(y)f(n)o(um)o(b)q(er)f(of)h(successiv)o(e)i(in)o(v)o (ok)n(ations)d(will)g(execute)j(correctly)m(.)23 b(Of)15 b(course,)i(the)f (same)e(approac)o(h)75 1446 y(can)g(b)q(e)h(used)f(for)g(other)g(parallel)f (libraries.)75 1569 y Fc(7.3)56 b(W)-5 b(ell)19 b(nested)f(parallel)j(pro)r (cedures)75 1649 y Fm(Calls)12 b(of)h(parallel)f(pro)q(cedures)j(are)f(w)o (ell)f(nested)h(if)f(a)g(new)g(parallel)f(pro)q(cedure)j(is)e(alw)o(a)o(ys)g (in)o(v)o(ok)o(ed)f(in)h(a)g(subset)75 1698 y(of)j(a)g(group)g(executing)h (the)g(same)e(parallel)g(pro)q(cedure.)27 b(Th)o(us,)17 b(pro)q(cesses)i (that)d(execute)i(the)f(same)e(parallel)75 1748 y(pro)q(cedure)h(ha)o(v)o(e)e (the)g(same)f(execution)i(stac)o(k.)158 1799 y(In)i(suc)o(h)g(a)f(case,)i(a)e (new)h(con)o(text)g(need)h(to)e(b)q(e)h(dynamically)d(allo)q(cated)i(for)g (eac)o(h)h(new)g(in)o(v)o(ok)n(ation)d(of)i(a)75 1849 y(parallel)e(pro)q (cedure.)24 b(Ho)o(w)o(ev)o(er,)16 b(a)f(stac)o(k)h(mec)o(hanism)d(can)i(b)q (e)h(used)g(for)f(allo)q(cating)f(new)h(con)o(texts.)24 b(Th)o(us,)15 b(a)75 1899 y(p)q(ossible)e(mec)o(hanism)d(is)j(to)f(allo)q(cate)g(\014rst)i (a)e(large)g(n)o(um)o(b)q(er)g(of)g(con)o(text)p 1233 1899 V 16 w(id's)g(\(up)h(to)f(the)i(upp)q(er)f(b)q(ound)g(on)f(the)75 1949 y(depth)f(of)f(nested)i(parallel)e(pro)q(cedure)i(calls\),)f(and)f(then) h(use)h(a)e(lo)q(cal)g(stac)o(k)h(managemen)o(t)d(of)i(these)i(con)o(text)p 1799 1949 V 16 w(id's)75 1999 y(on)i(eac)o(h)g(pro)q(cess)i(to)e(create)h(a)e (new)i(comm)o(unicator)c(\(using)j Ff(MPI)p 1129 1999 14 2 v 15 w(COMM)p 1232 1999 V 15 w(MAKE)p Fm(\))f(for)g(eac)o(h)i(new)f(in)o(v)o (ok)n(ation.)158 2132 y Fh(Discussion:)k Fg(General)c(case)158 2266 y Fm(In)g(the)h(general)f(case,)h(there)g(ma)o(y)d(b)q(e)j(m)o(ultiple)c (concurren)o(tly)16 b(activ)o(e)e(in)o(v)o(ok)n(ations)e(of)h(the)i(same)e (parallel)75 2316 y(pro)q(cedure)18 b(within)d(the)i(same)e(group;)h(in)o(v)o (ok)n(ations)e(ma)o(y)g(not)i(b)q(e)h(w)o(ell)e(nested.)26 b(A)16 b(new)g(con)o(text)h(need)g(to)e(b)q(e)75 2366 y(created)i(for)e(eac)o (h)h(in)o(v)o(ok)n(ation.)21 b(It)16 b(is)f(the)h(user)h(resp)q(onsibilit)o (y)e(to)g(mak)o(e)f(sure)j(that,)f(if)e(t)o(w)o(o)h(distinct)h(parallel)75 2416 y(pro)q(cedures)h(are)e(in)o(v)o(ok)o(ed)f(concurren)o(tly)i(on)f(o)o(v) o(erlapping)f(sets)i(of)e(pro)q(cesses,)j(then)f(con)o(text)p 1584 2416 13 2 v 15 w(id)f(allo)q(cation)e(or)75 2466 y(comm)o(unicator)e (creation)j(is)g(prop)q(erly)g(co)q(ordinated.)75 2610 y Fn(8)69 b(Left)75 2704 y Fm(Remote)13 b(pro)q(cedure)j(call)d(and)g(C2)h(ob)r(jects.) p eop %%Page: 11 11 bop 1833 -100 a Fm(11)158 45 y(Ho)o(w)14 b(to)f(handle)h(gro)o(wing)f(pro)q (cess)j(set.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF >From weeks@mozart.convex.com Wed Jun 9 15:37:19 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15217; Wed, 9 Jun 93 15:37:19 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA13402; Wed, 9 Jun 93 15:37:04 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA02892; Wed, 9 Jun 93 08:36:37 -0500 Received: by mozart.convex.com (5.64/1.28) id AA01496; Wed, 9 Jun 93 08:38:50 -0500 Date: Wed, 9 Jun 93 08:38:50 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091338.AA01496@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: An inquiry you may not have received Status: R On May 31 there was a post in comp.parallel asking "what is MPI?" -- I responded to the guy, telling him to contact you if he wanted to get in one or more of the email lists and/or to attend the meeting. He is from exxon. Now it occurs to me that he might have sent you mail at cs.utk and maybe it didn't get forwarded to france or whatever, so just for reference here is the guy's name and email: Denny Willen dew@custer.exxon.com >From owner-mpi-context@CS.UTK.EDU Wed May 26 15:38:20 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA09084; Wed, 26 May 93 15:38:17 -0500 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA09337; Wed, 26 May 93 15:38:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21630; Wed, 26 May 93 16:35:42 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Wed, 26 May 1993 16:35:41 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21536; Wed, 26 May 93 16:35:08 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19257; Wed, 26 May 1993 16:35:06 -0400 Date: Wed, 26 May 1993 16:35:06 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9305262035.AA19257@rios2.epm.ornl.gov> To: mpi-context@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Group nakedness Status: R It has occurred to me, while reading the "Maps, Groups, and Contexts" (MGC) draft, that groups no longer really exist in MPI in the sense that there is no routine whose input or output is a group. It seems we have discarded the intuitively appealing concept of a group, and replaced it with lesser obvious concept of a map. I would like to get rid of maps, and restore groups to MPI. For most of the routines in MGC this can be done by replacing the word "map" with "group", and clarifying the names of some routines. I would like to see the MGC draft rewritten with the routines below. In most cases there is a clear correspondence between a "map" routine and a "group" routine that does something very similar. David MPI_MAP_DOMAIN --> MPI_GROUP_SIZE size_of_group = MPI_GROUP_SIZE(group) IN group - handle to group object OUT size_of_group - number of processes in group (integer) MPI_MAP_ELEMENT --> MPI_RANK_IN_PARENT old_rank = MPI_RANK_IN_PARENT(group,rank) IN group - handle to group object IN rank - rank in group (integer) OUT old_rank - rank of process (group,rank) in parent group MPI_MAP_RANK --> MPI_RANK_IN_GROUP rank = MPI_RANK_IN_GROUP(group,old_rank) IN group - handle to group object IN old_rank - rank in parent group (integer) OUT rank - rank in group of process whose rank in the parent group is (group,old_rank) (integer) MPI_MAP_FLATTEN --> MPI_INQ_GROUP MPI_INQ_GROUP(group,array_of_integer,max_size) IN group - handle to group object OUT array_of_integer - list of the ranks in the parent group of the processes in group INOUT max_dim - on entry the maximum number of integers that can be put into array_of_integer. on exit the size of the group, i.e. the actual number of integers put into array_of_integer (integer). MPI_MAP_BUILD --> MPI_GROUP_BUILD MPI_MAKE_GROUP(group,array_of_integer) OUT group - handle to new group object IN array_of_integer - ranks in parent group of processes in new group (Note: this is still a local group constructor since communicators will stiil be used to all communication routines) MPI_MAP_CONCAT --> MPI_GROUP_CONCAT MPI_GROUP_CONCAT(group1,group2,new_group) IN group1 - handle to first group IN group2 - handle to second group (does not overlap with group1) OUT new_group - handle to new group that is the concatentation of group1 and group2 MPI_MAP_SPLIT --> MPI_GROUP_SPLIT MPI_GROUP_SPLIT(group,key,index,new_group) IN group - handle to group IN key - key used to partition group (integer) IN index - value whose relative order over all processes in new_group determines the rank within new_group (integer) OUT new_group - handle to new group (Note: this is a collective group constructor) MPI_ALLOC_CONTEXT --> MPI_ALLOC_CONTEXT MPI_ALLOC_CONTEXT(comm,array_of_contextids,len) IN comm - handle to communicator IN len - number of contexts to be created (integer) OUT array_of_contextids - array of len context IDs (Note: this is a collective operation involving all the process in the group in comm) MPI_MAKE_COMM --> MPI_MAKE_COMM MPI_MAKE_COMM(group,context_id,new_comm) IN group - handle to group IN context_id - context ID OUT new_comm - handle to communicator containing group and context_id MPI_SAFEMAKE_COMM --> MPI_SAFEMAKE_COMM MPI_SAFEMAKE_COMM(group,new_comm) IN group - handle to group OUT new_comm - handle to communicator (Note: this is a collective operation) MPI_MAP --> not needed MPI_CONTEXT --> what is this for? nothing --> MPI_PARENT (a new routine, may be useful) MPI_PARENT(group,parent_group) IN group - handle to group OUT parent_group - handle to parent of group Date: Fri, 28 May 93 14:42:06 BST Message-Id: <7507.9305281342@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Group nakedness To: walker@rios2.epm.ornl.gov (David Walker) In-Reply-To: David Walker's message of Wed, 26 May 1993 16:35:06 -0400 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Status: RO David writes: > It has occurred to me, while reading the "Maps, Groups, and Contexts" (MGC) > draft, that groups no longer really exist in MPI in the sense that there is > no routine whose input or output is a group. > > It seems we have discarded the intuitively appealing concept of a group, and > replaced it with lesser obvious concept of a map. I would like to get rid > of maps, and restore groups to MPI. Me too! I think this makes a lot of sense. I explained it to future users here and they just said "Eh?". I can't comment on the detailed suggestions which was given below, I aint had time to read it yet. Best Wishes Lyndon (the ex-prolific :-) /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ >From owner-mpi-context@CS.UTK.EDU Thu Jun 3 10:29:00 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA09255; Thu, 3 Jun 93 10:28:45 -0500 Received: from [128.169.201.1] by convex.convex.com (5.64/1.35) id AA21898; Thu, 3 Jun 93 10:26:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08266; Thu, 3 Jun 93 11:14:59 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 3 Jun 1993 11:14:58 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08257; Thu, 3 Jun 93 11:14:47 -0400 Date: Thu, 3 Jun 93 16:14:37 BST Message-Id: <14127.9306031514@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-context: group nakedness and deconvolution To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Status: R Hi Tony How's the context draft coming along between yourself and Mark (Sears)? I'll be interested to hear what's happening on the "group nakedness" which was mentioned over the net by David, yourself and I. I found time to consider the concrete suggestion put forward by David, at last, and have some input which I hope will be of use. First off I think that the deletion of MAP and exposure of GROUP is a Good Thing. It helps to deconvolve the conceptual context of the draft which I now see as a major difficulty. I explained it to people and the convolution was quite a problem. I'll go through David's suggestion making the detailed comments, and then make the overall comments after all that. Lines of David's begin with "> ", and I'll include pointers to the overall comments where appropriate. o--------------------o Detailed Comments - ----------------- > MPI_MAP_DOMAIN --> MPI_GROUP_SIZE > size_of_group = MPI_GROUP_SIZE(group) > IN group - handle to group object > OUT size_of_group - number of processes in group (integer) Looks fine to me. > MPI_MAP_ELEMENT --> MPI_RANK_IN_PARENT > old_rank = MPI_RANK_IN_PARENT(group,rank) > IN group - handle to group object > IN rank - rank in group (integer) > OUT old_rank - rank of process (group,rank) in parent group This introduces the idea of ranking within the space of a parent group, presumably that within which the (child) group was created (overall comment below). The group object must store the handle of the parent group, which doesnt seem to be a problem. > MPI_MAP_RANK --> MPI_RANK_IN_GROUP > rank = MPI_RANK_IN_GROUP(group,old_rank) > IN group - handle to group object > IN old_rank - rank in parent group (integer) > OUT rank - rank in group of process whose rank in the parent group is > (group,old_rank) (integer) Is consistent. > MPI_MAP_FLATTEN --> MPI_INQ_GROUP > MPI_INQ_GROUP(group,array_of_integer,max_size) > IN group - handle to group object > OUT array_of_integer - list of the ranks in the parent group of the > processes in group > INOUT max_dim - on entry the maximum number of integers that can be > put into array_of_integer. > on exit the size of the group, i.e. the actual number of integers put into array_of_integer (integer). Is consistent. > MPI_MAP_BUILD --> MPI_GROUP_BUILD > MPI_MAKE_GROUP(group,array_of_integer) > OUT group - handle to new group object > IN array_of_integer - ranks in parent group of processes in new group > (Note: this is still a local group constructor since communicators will stiil > be used to all communication routines) Is not consistent. Needs the parent group as an argument in order to record the parent group in the created group (at least). Needs the size of the group to create as argument. Its far from clear how much use MPI_MAKE_GROUP and MPI_INQ_GROUP really are with the business of always referencing the parent. In order to send a group in a message this way the user will have to send the entire ancestry of the group in the message(s). Please see overall comments below. > MPI_MAP_CONCAT --> MPI_GROUP_CONCAT > MPI_GROUP_CONCAT(group1,group2,new_group) > IN group1 - handle to first group > IN group2 - handle to second group (does not overlap with group1) > OUT new_group - handle to new group that is the concatentation of group1 and > group2 Within the parent reference framework the groups must have the same parent, or with more complication a common ancestor which becomes the parent of the created group. Decision should be made and documented. > MPI_MAP_SPLIT --> MPI_GROUP_SPLIT > MPI_GROUP_SPLIT(group,key,index,new_group) > IN group - handle to group > IN key - key used to partition group (integer) > IN index - value whose relative order over all processes in new_group > determines the rank within new_group (integer) > OUT new_group - handle to new group > (Note: this is a collective group constructor) Is consistent. Should document that "group" is parent of each "new_group" (obvious to us, I know, but ...). > MPI_ALLOC_CONTEXT --> MPI_ALLOC_CONTEXT > MPI_ALLOC_CONTEXT(comm,array_of_contextids,len) > IN comm - handle to communicator > IN len - number of contexts to be created (integer) > OUT array_of_contextids - array of len context IDs > (Note: this is a collective operation involving all the process in the > group in comm) Please see overall comments below. > MPI_MAKE_COMM --> MPI_MAKE_COMM > MPI_MAKE_COMM(group,context_id,new_comm) > IN group - handle to group > IN context_id - context ID > OUT new_comm - handle to communicator containing group and context_id Is consistent (although see following message on intercommuncation). > MPI_SAFEMAKE_COMM --> MPI_SAFEMAKE_COMM > MPI_SAFEMAKE_COMM(group,new_comm) > IN group - handle to group > OUT new_comm - handle to communicator > (Note: this is a collective operation) Is inconsistent. There is no communicator with which to call MPI_ALLOC_CONTEXT. Can be fixed, for example, by redefining MPI_ALLOC_CONTEXT to accept a group rather than a communicator. > MPI_MAP --> not needed I assume this is a reference to MPI_COMM_MAP. Equivalent operator still needed, returning group. MPI_COMM_GROUP(comm, group) IN comm communicator handle OUT group handle of group bound in communicator MPI_CONTEXT --> what is this for? I assume this is a reference to MPI_COMM_CONTEXTID. Should keep it. > nothing --> MPI_PARENT (a new routine, may be useful) > MPI_PARENT(group,parent_group) > IN group - handle to group > OUT parent_group - handle to parent of group In the preant reference framework this is essential. Overall Comments - ---------------- 1. Parent framework. I mentioned that MPI_INQ_GROUP and MPI_BUILD_GROUP will be difficult to use. Recommendation is to drop the parent reference framework and have absolute groups. This means absolute process identification which can be, for example, indices within a group containing all processes, but not mandated to be such. 2. Context allocation. I mentioned that MPI_ALLOC_CONTEXT and MPI_SAFEMAKE_COMM are inconsistent. Recommendation is to replace the communicator argument of MPI_ALLOC_CONTEXT with a group. MPI implementation provides the context or whatever for the implied collective operation, as it does for MPI_SAFEMAKE_COMM as described. This recommendation effects deconvolution of the conceptual content of the draft. It will make the whole thing much easier to understand. o--------------------o Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ >From weeks@mozart.convex.com Wed Jun 9 15:47:16 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15290; Wed, 9 Jun 93 15:47:15 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA13569; Wed, 9 Jun 93 15:46:55 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA03206; Wed, 9 Jun 93 08:46:28 -0500 Received: by mozart.convex.com (5.64/1.28) id AA01658; Wed, 9 Jun 93 08:48:42 -0500 Date: Wed, 9 Jun 93 08:48:42 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091348.AA01658@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: June 3 mpi mail #1 of 1 Status: R >From owner-mpi-context@CS.UTK.EDU Thu Jun 3 12:29:51 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA14746; Thu, 3 Jun 93 12:29:33 -0500 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA27165; Thu, 3 Jun 93 12:26:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16388; Thu, 3 Jun 93 13:22:04 -0400 X-Resent-To: mpi-context@CS.UTK.EDU ; Thu, 3 Jun 1993 13:22:02 EDT Errors-To: owner-mpi-context@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16375; Thu, 3 Jun 93 13:21:35 -0400 Date: Thu, 3 Jun 93 18:21:28 BST Message-Id: <14220.9306031721@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-context: intercommunication etcetara To: tony@aurora.cs.msstate.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-context@cs.utk.edu Status: R Hi Tony Here are thoughts on how we can proceed with intercommunication. I write this letter as though maps were deleted and groups exposed, in anticipation :-) In summary this letter says what we do not need two different kinds of communicator objects - indeed we can have just one communicator object which is described in the letter. We do not really need different constructors to build communicators for intracommunication and intercommunication, but we should provide them for the convenience of the user. The letter adds at most nine procedures to this section of MPI, and suggests deletion of a few others :-) The "context" in the draft from Marc (Snir) is weaker than that in Zipcode and Proopsal X, as I am sure you are aware. The allocator ensures only that the context is unique within the set of processes performing the allocation. The context (or context_id in the draft) can be sent in a message. Presumably this means the following. A group G could allocate a context C, send it to a group H. H can then send messages to G using context C which in the knowledge that G knows such messages come from H. However, G cannot send messages to H using context C in the knowledge that such messages have come from G. I have no complaint with this context concept. It does have consequences, one of which is that attempting to simulate intercommunication by group union intracommunication not particularly palatable. Never mind, I hope this letter can largely resolve difficulties with intercommunication. With this kind of context in mind, I write out generic send and receive in longhand, in two slightly different ways. Please bear with me on this. send(sender-group, send(local-group, receiver-group, remote-group, receiver-rank, OR remote-rank, receiver-context, remote-context, tag, tag, ...) ...) receive(receiver-group, receive(local-group, sender-group, OR remote-group, sender-rank, remote-rank, receiver-context, local-context, tag, tag, ...) ...) The receive operation is allowed to wildcard on "sender-rank" (OR "remote-rank") and "tag" (OR "tag" :-) only. The "sender-group" (OR "local-group") argument to "send" is just there so that the rank of the sender can be placed in the message envelope. In principle this could just be some identifier for the sender, but the natural identifier to use in MPI terms in the rank within a group, and putting the group there is good programming practice since it documents the send. The two group arguments to receive, "receiver-group" and "sender-group" (OR "local-group" and "remote-group") appear at first to serve no purpose since "receive" cannot select on them. However they may be useful in optimisations of interpretation of "sender-rank" (OR "remote-rank") and the context, and are good programming practice since they document the receive. In general people at MPI (excepting myself) have been thinking and talking about the (special) case where it just so happens that "sender-group = receiver-group" (OR "remote-group = local-group"). This point is key. I suggest that we can simply let this be a special case, provide convenience functions for this case, and retain efficiency. Sounds groovy to me :-) So what is a communicator then? No suprises to learn that the idea is something like typedef struct { group_t local_group; context_t local_context; group_t remote_group; context_t remote_context; } communicator_t; where I have written "group_t" for the type of a group and "context_t" for the type of a context (and yes, I am talking a bit too close to implementation here, but that is just to demonstrate the potential for a single implementation). If the communicator is to be used for intracommunication, then the construction of the communicator sets "remote_group = local_group" and "remote_context = local_context". If the communicator is to be used for intercommunication, then the construction of the communicator sets "remote_group" and "local_group" differently. It may or may not set "remote_context" and "local_context" differently depending on the scope within which contexts were allocated. Now the interesting case of the two "group_t" being set the same and the two "context_t" being set different. Yes, it even means something, it means two different modules within the same process group which communicate with one another, and need private context for such communications (it's intercommunication really). Now we're motoring :-) How is it used in point-to-point? For pedagogical purposes, I now write out the generic send and receive using the communicator object described and show how they correspond to the above. send(communicator, rank, tag, ...) sender-group = local-group = communicator.local_group receiver-group = remote-group = communicator.remote_group receiver-rank = remote-rank = rank receiver-context = remote-context = communicator.remote_context tag = tag = tag receive(communicator, rank, tag, ...) receiver-group = local-group = communicator.local_group sender-group = remote-group = communicator.remote_group sender-rank = remote-rank = rank receiver-context = local-context = communicator.local_context tag = tag = tag So how is it used in collective? Well we have only collective operations appliacble to the case where the local and remote parts of the communicator are the same, so they are all errors if the communicator supplied is not of that nature. We could invent jargon to describe a communicator with that nature, if we really wanted to. I think that deals with what the object is and how it is used. Better lets think about the constructors and accessors and all the such now. Okay, I said we keep functions for making the special case of remote and local parts the same for user convenience. So just two new procedures for constructors. Names for these? Well, as you know I'm not very good at that part :-) I added the letter 'A' to 'COMM' (a for assymetric, and use of the string "comma" is of course silly). MPI_COMMA_MAKE(local_group, local_context, remote_group, remote_context, comm) IN local_group local group for communicator which caller must be member of IN local_context for communicator which local_group processes must be able to use for receive IN remote_group for communicator which caller moy or may not be a member of IN remote_context for communicator which remote_group processes must be able to use for receive OUT comm communicator created MPI_COMMA_SAFEMAKE(lcoal_group, remote_group, comm) IN local_group as for MPI_COMMA_MAKE IN remote_group as for MPI_COMMA_MAKE OUT comm as for MPI_COMMA_MAKE (Note that this is a collective operation synchronising all members of both process groups) [Aside - I *much* prefer to drop the "safemake" stuff and be able to pass a NULL or MPI_NULL_CONTEXT to the "make" stuff in which case it allocates the contexts for the user. It allows just one or other of the two contexts in these cases to be allocated automatically. I propose this.] Six new procedures for accessors. They are pretty self explanantory so I do not detail the argument lists here, it can be added trivially later. MPI_COMM_LOCAL_GROUP(comm, local_group) MPI_COMM_REMOTE_GROUP(comm, remote_group) MPI_COMM_LOCAL_CONTEXTID(comm, local_context) MPI_COMM_REMOTE_CONTEXTID(comm, remote_context) MPI_COMM_LOCAL_SIZE(comm, local_size) MPI_COMM_REMOTE_SIZE(comm, remote_size) The three accessors in the draft should probably be documented as errors if the remote and local parts of the communicator are not identical. On the other hand, they could just be deleted :-) [Aside - why have accessors for the group size when the user can easily get at the group and ask that object what the size is? If we have this function it should be in Section 6.] Regarding Section 6, "Working without maps and context_id's". I guess we'd have a new procedure which is pretty obvious by now MPI_COMMA_BUILD(comm, local_arrayofranks, local_size, remote_arrayofranks, remote_size, newcomm) [Aside - I'd like to see MPI_COMM[A]_BUILD dropped. This should be done at the group level. I propose this.] MPI_COMM_SPLIT should be documented as an error if the local and remote parts of the existing communicator are not identical. Finally I cannot avoid the m