From owner-mpi-pt2pt@CS.UTK.EDU Mon Nov 23 14:37:41 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18480; Mon, 23 Nov 92 14:37:41 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA23460; Mon, 23 Nov 92 14:28:46 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23454; Mon, 23 Nov 92 14:28:42 -0500 Received: from LOCALHOST.cs.utk.edu by thud.cs.utk.edu with SMTP (5.61++/2.7c-UTK) id AA06042; Mon, 23 Nov 92 14:28:41 -0500 Message-Id: <9211231928.AA06042@thud.cs.utk.edu> To: mpi-pt2pt@cs.utk.edu Cc: wade@cs.utk.edu Subject: a test Date: Mon, 23 Nov 92 14:28:40 EST From: Reed Wade this is a test of the mpi-pt2pt mailing list. -reed Mon Nov 23 16:03:12 EST 1992 From owner-mpi-pt2pt@CS.UTK.EDU Tue Nov 24 23:08:30 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15646; Tue, 24 Nov 92 23:08:30 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA25775; Tue, 24 Nov 92 22:28:54 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25771; Tue, 24 Nov 92 22:28:51 -0500 Received: from Electryon.Think.COM by mail.think.com; Tue, 24 Nov 92 22:28:43 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA20255; Tue, 24 Nov 92 22:28:41 EST Date: Tue, 24 Nov 92 22:28:41 EST Message-Id: <9211250328.AA20255@electryon.think.com> To: mpi-pt2pt@cs.utk.edu Subject: Nonblocking functions and handlers. I would like to float a few questions about what we mean by the modifier nonblocking it is applied to the MPI specification. Nonblocking Send: This is the curious one. Does is mean: a) The user may alter the contents of the buffer once the send function returns control? or b) The user must check the status of the send operation to determine when it is safe to alter the contents of the buffer? Nonblocking receive: This seems pretty clear. The user must inquire of the system to determine if the user specified buffer contains the desired message. In either case, it may be desirable to specify an action that is to occur once the operation is complete. Consider adding two arguments to the send and receive functions: MPI__n( int , int tag, void * buffer, int len, void (* handler)(), void * arg ) Here, handler is a pointer to a function which accepts two arguments: handler( int msg_id, * arg ) Upon completion of the send or receive operation, the handler function is invoked and provided with the id of the message operation just completed and the argument address provided by the user. I raise the issue of handlers because we have ben asked repeatedly to provide such fuctionality. Thoughts? moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:38:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17471; Wed, 25 Nov 92 10:38:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06314; Wed, 25 Nov 92 10:13:00 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06310; Wed, 25 Nov 92 10:12:58 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA08435; Wed, 25 Nov 92 09:12:54 CST Received: by teak18.cray.com id AA05289; 4.1/CRI-5.6; Wed, 25 Nov 92 09:12:52 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251512.AA05289@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 9:12:49 CST Cc: par@teak.cray.com (Peter Rigsbee) In-Reply-To: <9211250328.AA20255@electryon.think.com>; from "Adam Greenberg" at Nov 24, 92 10:28 pm X-Mailer: ELM [version 2.3 PL11b-CRI] Adam Greenberg writes: > In either case, it may be desirable to specify an action that is to occur once > the operation is complete. Consider adding two arguments to the send and > receive functions: > > MPI__n( int , int tag, void * buffer, int len, > void (* handler)(), void * arg ) > > Here, handler is a pointer to a function which accepts two arguments: > handler( int msg_id, * arg ) > > Upon completion of the send or receive operation, the handler function is > invoked and provided with the id of the message operation just completed and > the argument address provided by the user. I've never really liked this approach. I think it adds complexity without really adding much value. A couple of issues: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) - second, even if you have a signal handler, the mainline of the application must still poll something to see if the operation has completed; the difference is that the "something" if often a local variable rather than the messaging system. But the logic in the program is similar and it is likely that the number and decision of where to poll is driven by the algorithm. - third, do all architectures provide the user-level interrupts needed to implement this? If one doesn't, adding this functionality may simply not be possible. IMO, this seems like something that isn't desirable as a part of MPI1. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:40:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17537; Wed, 25 Nov 92 10:40:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06692; Wed, 25 Nov 92 10:30:22 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06686; Wed, 25 Nov 92 10:30:19 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03147; Wed, 25 Nov 92 09:28:46 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA12985; Wed, 25 Nov 92 09:28:43 CST Date: Wed, 25 Nov 92 09:28:43 CST Message-Id: <9211251528.AA12985@godzilla.mcs.anl.gov> To: moose@think.com Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Adam Greenberg's message of Tue, 24 Nov 92 22:28:41 EST <9211250328.AA20255@electryon.think.com> Subject: Nonblocking functions and handlers. I'd love to see a standard implementation that invoked a routine on receipt, particularly if that routine was not restricted in any way (for example, restricted to not using any MPI calls). Unfortunately, doing this right makes demands on the underlying OS that may be more than some vendors are willing to commit to. We need to be careful here. Note that Unix does not provide this functionality to users, though VMS has for a long time. (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. Bill Gropp From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:07:46 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19010; Wed, 25 Nov 92 11:07:46 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07267; Wed, 25 Nov 92 11:02:46 -0500 Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07263; Wed, 25 Nov 92 11:02:43 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Wed, 25 Nov 1992 11:02:31 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA26484; Wed, 25 Nov 1992 11:02:30 -0500 Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> To: mpi-pt2pt@cs.utk.edu, par@teak.cray.com Subject: Re: Nonblocking functions and handlers. Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Yale Computer Science Department From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:38:31 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19507; Wed, 25 Nov 92 11:38:31 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07714; Wed, 25 Nov 92 11:22:36 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07710; Wed, 25 Nov 92 11:22:33 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA13731; Wed, 25 Nov 92 10:22:31 CST Received: by teak18.cray.com id AA05599; 4.1/CRI-5.6; Wed, 25 Nov 92 10:22:30 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251622.AA05599@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 10:22:26 CST In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU>; from "Harry Berryman" at Nov 25, 92 11:02 am X-Mailer: ELM [version 2.3 PL11b-CRI] Scott Berryman writes: > - first, like most signal handling, this kind of capability tends > to be more oriented towards C programs and away from Fortran > programs (note that your prototype for "handler" is such that > it could not be programmed in Fortran) > > Although the prototype given by Greenberg precludes the use of > Fortran, the approach certainly does not preclude Fortran. > Obviously Fortran can pass pointers to functions as parameters. Point well taken. But there are two aspects of the interface to consider. Fortran passing pointers to functions as parameters is fine. But a Fortran programmer might also want to write the handler in Fortran, which would then require that the handler's arguments all be pointers to things. But this would bother C programmers and would be inconsistent with other "signal handlers". > In particular, the iPSC series (including the Delta) support > a similar mechanism in Fortran through the hrecv(), et al. My iPSC documentation (which may be out of date) shows that hrecv() can be called by Fortran but requires that the handler be written in C: proc is the procedure executed when the receive is complete. This procedure must be written by the user and must be a C procedure with four arguments. I'd argue that having to write the handler in C makes this less useful to Fortran programmers, especially if you are trying to write something portable. There is no real standard defining how C and Fortran interact. The C handler could call a subhandler written in Fortran, but this is getting kind of messy for something that should be simple. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:43:10 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21772; Wed, 25 Nov 92 13:43:10 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA10110; Wed, 25 Nov 92 13:30:11 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10106; Wed, 25 Nov 92 13:30:09 -0500 Received: from Electryon.Think.COM by mail.think.com; Wed, 25 Nov 92 13:30:05 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA22742; Wed, 25 Nov 92 13:30:04 EST Date: Wed, 25 Nov 92 13:30:04 EST Message-Id: <9211251830.AA22742@electryon.think.com> To: berryman-harry@CS.YALE.EDU Cc: mpi-pt2pt@cs.utk.edu, par@teak.cray.com In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> "berryman-harry@CS.YALE.EDU" Subject: Nonblocking functions and handlers. Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Mea culpa. The prototype was not meant to exclude Fortran (in fact we through a discussion of this issue here last week). The handler should viewed as taking two pointers: void handler( int * msg_id, * arg ) Sorry for any confusion: moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:44:47 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21797; Wed, 25 Nov 92 13:44:47 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA09803; Wed, 25 Nov 92 13:14:38 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09784; Wed, 25 Nov 92 13:14:29 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA22170; Wed, 25 Nov 92 13:14:31 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 131340.4011; Wed, 25 Nov 1992 13:13:40 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA12688; Wed, 25 Nov 1992 12:06:04 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA08958; Wed, 25 Nov 92 12:06:02 -0600 Date: Wed, 25 Nov 92 12:06:02 -0600 Message-Id: <9211251806.AA08958@brisk.kai.com> To: mpi-pt2pt@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-formal@cs.utk.edu, mpi-ptop@cs.utk.edu Reply-To: William.Gropp's.message.of.Wed@kai.com, 25 Nov 92 09:28:43 CST <9211251528.AA12985@godzilla.mcs.anl.gov> Subject: Nonblocking functions and handlers. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Bill Gropp writes: (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. I'm not so sure this is a radical position Bill since even nonsynchronized communication will need to be defined formally this way. Nonsynchronized communication is in effect creating a parallel process that has the job of passing the communication on. Al Geist earlier asked the question wheather buffers used by nonsynchronized communication should be accessible after the communication has started - the answer should be - no, unless by some explicit mechanism that formally amounts to a communication with the process mentioned above. Any nonexplicit interaction (e.g. a write to the buffer) would have to be specified as formally equivalent to an explicit interaction. Also, there is quite a range of terminology in use. One common error: "Asynchronous" and "synchronous" has quite a particular meaning in EE and when CS people use the terms in relation to message passing they usually mean NONSYNCHRONIZED and SYNCHRONIZED. Also BLOCKING = SYNCHRONIZED. Let us begin a glossary that defines the terms we use - if no-one else volunteers I'll take this to be the responsibility of the Formal Specification Subcommittee. So I'm looking for volunteers from that subcommittee. Steven From owner-mpi-pt2pt@CS.UTK.EDU Sat Nov 28 19:07:59 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08243; Sat, 28 Nov 92 19:07:59 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22503; Sat, 28 Nov 92 18:46:34 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22499; Sat, 28 Nov 92 18:46:32 -0500 Message-Id: <9211282346.AA22499@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6099; Sat, 28 Nov 92 18:45:57 EST Date: Sat, 28 Nov 92 18:35:53 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: Our work Reply-To: SNIR@watson.ibm.com First, a technical issue: If somebody cannot print postscript, or has strong preferences for Latex, pls let me know. Now for content. I would like to use the outline I just sent as a working document for our subcommittee, and modify it periodically to reflect input I receive from the remaining group members. I shall update this document every weekend, or every 2 weekends, according to the number of comments I get, and the amount of spare time my family grants me. Of course, decisions will be taken when we meet, but I feel it is useful to have a working document that reflects in a somewhat coherent manner the state of our deliberations. So please, start sending your comments to mpi-pt2pt, assuming the next checkpoint is end of next week. I want to replace as soon as possible the I's by We's. From owner-mpi-pt2pt@CS.UTK.EDU Thu Dec 3 10:39:15 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01501; Thu, 3 Dec 92 10:39:15 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA16013; Thu, 3 Dec 92 10:20:18 -0500 Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16008; Thu, 3 Dec 92 10:20:13 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA25545 (5.65c/IDA-1.4.4 for ); Thu, 3 Dec 1992 16:19:46 +0100 Received: by f1neuman.gmd.de id AA14744; Thu, 3 Dec 1992 16:19:30 +0100 Date: Thu, 3 Dec 1992 16:19:30 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9212031519.AA14744@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: gmap10@f1neuman.gmd.de I would like to make some comments on the document Marc Snir has distributed. 1. I agree on all points in the list of goals. I find it especially important to define something which is not too far away from common practice, otherwise nobody would follow us. A point which is missing on the list (perhaps because it is too obvious): the interface should not contain more functions than are really necessary. We must avoid to end up with a mixed bag which contains multiple mechanisms for the same thing. 2. I like Marc's typification scheme for message passing functions based on the 4 suboperations. I understand those suboperations more as a way to describe the semantics rather than the implementation of the (compound) functions which the user will see. I'm not sure that the channel based operations are necessary on the user level. Basically there are the two addressing schemes for message passing: either a message is sent to a named process, or it is sent along a named connection (channel), in which case the processes don't even need to have identifiers. We had a long discussion here in Europe on which scheme is better. In the end it turns out that both are more or less equivalent. Most existing user interfaces tend towards the named process approach. The only reason for supporting both concepts in the same interface seems to be the possibility of some additional optimization. If suboperations (1) and (4) are separated, ressource allocation and deallocation can be saved. However, I think that a clever operating system can detect frequently used communication paths automatically, and it can establish a permanent link for them. (Link caching) 3. We will have various message passing mechanisms, especially synchronous and asynchronous communication (global/local send termination). I strongly recommend that the corresponding send and receive functions must match. On one hand this gives the opportunity for some implementation optimization, and also it might be used for correctness checking. 4. As Marc said, in MPI1 termination is either local or global, at the user's choice. I really think that this is necessary, and a restriction on either synchronous or asynchronous communication would be hard to accept. 5. Marc suggests that there should be no "unsuccessful return" of a message passing operation. I agree. 6. On page 7, Marc discusses the different protocols for message passing. I'm not sure that we want to define this in detail, since it is very much implementation dependent. On the other hand we will have to accept the fact that a program will run on a machine with huge system buffers and not on a smaller system. Somehow this is true with other system resources, also. 7. Page 7, further down: if we deal with heterogeneous networks, a contiguous message buffer may contain sections with different data type. The reason is always that one wants to send as few messages as possible (startup minimization). Personally I prefer to have a separate library call before the send which specifies the contents of the message buffer. This information is then used or ignored by the send routine depending on the data representations in the sender and receiver processors. 8. If our goal is to define a scalable interface, setting up groups by specifying member lists in all participating processes is a problem. This is especially so for the handling of very regular group structures, like grids or tori, since these will most likely be used by applications with very large process numbers. I think the Virtual Topology subcommittee has to deal with this problem. 9. Communication context: I very much favor option 1. (i.e. context switch by some special functions) against explicitely specifying the context in a (collective) communication statement. I do not see that the explicit specification would lead to less user errors, and it would add another item to the argument lists. 10. Receive criteria: I do not see a good point for having "don't care" addresses in sends. For "don't care" in receive I prefer named constants over "magic values". Be sure not to use arguments for both input to and output from a function, especially if it may contain wildcards on input. There is no better way to make sure your constants get overwritten! 11. Matching of data: In Fortran I think option 2. is acceptable. If the receive buffer is too short, this is a user error. In C there is another option which should be supported by the interface: if the receive buffer pointer is NULL, the system allocates one with the right size automatically. 12. Syntax: multiple modes like in MPI1 are okay with me, but I hate to see character strings there. Talk to any implementor who cares about efficiency, and you know what I mean. 13. Also, in Fortran I find it better to use procedure calls for most functions. 14. Miscelania: a) We will need a nonblocking AWAIT operation. This is a great help in organizing the overlap of computation and communication. b) How could we survive without a PROBE function? c) What about providing an inquiry subroutine, which returns the message length, the sender, and the type of the "last message" which was received? For non-blocking messages this is not well defined, but a solution should be possible. This has become a longer message than I had planned. Sorry for bothering you all so much! Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 16:39:48 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21532; Wed, 9 Dec 92 16:39:48 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22026; Wed, 9 Dec 92 16:32:54 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22022; Wed, 9 Dec 92 16:32:51 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA05364; Wed, 9 Dec 92 15:32:41 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA00525; Wed, 9 Dec 92 15:32:39 CST Date: Wed, 9 Dec 92 15:32:39 CST Message-Id: <9212092132.AA00525@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: lusk@antares.mcs.anl.gov Thanks to Marc for laying out many of the issues so clearly. We find that we agree with many and disagree with some. Goals: (1) We think that those most in need of a standard are those writing libraries, and therefor the standard should cater to their needs as well as those of end users. It will be easy to design simple application programming interfaces on top of the MPI standard. (2) We strongly agree with this point (allow efficient communication). (3) We are not sure what "allow (but no mandate)" for heterogeneous communications means. If this means that vendors of homogeneous massively parallel systems may implement only the homogeneous operations, then we agree, as long as the heterogeneous operations are defined by the MPI standard. We agree in principle with the others. Framework: We also find the described framework useful for thinking about the operations, but not necessarily the right structure for the MPI. Issues: An exchange operation can be useful. At least two MPP vendors (TMC and Intel) provide such operations. We also agree an leaving out RPC, active messages, etc, as these are topics of rapidly moving research. We believe that there may be something more inclusive than channels that can serve as the basis for specifying high-performance communication operations. We do not find the discussion of "local versus global" termination to be a good way to consider the issues of blocking, nonblocking, and synchronous communication. In particular, blocking and nonblocking refer to the state of the buffer containing the message on return from the routine (be it send, receive, or wait), and synchronous (in the MPI proposal) really refers to an alternate protocol for the delivery of a message. We believe that relying on an exception handling mechanism for error handling will make the system too fragile. We propose allowing the return of error conditions. An alternate proposal allows the user to specify an error routine to be called in the event of an error. Note that library implementors may need to handle errors explicitly. Correctness Criteria: We believe that it is important to allow the user to determine (at least a bound on) the amount of resources available. Further, it MAY be appropriate to specify a minimum size that is supported (otherwise zero may be consistent with the letter of the standard). Communication Parameters: Our draft implementation includes an extension to the current proposal that permits the communication of messages of mixed datatype (we interpret "type" on page 7 to mean "datatype" rather than "message type"). This is a simple extension to the data description vector that allows for heterogeneous communication. Groups and context: We agree that the collective communication committee should be the one to discuss these issues. We do feel that the destination and source processor ids should not be relative to some "current" group. Note that such a functionality (with some restrictions) can be provided by a simple layer on top of MPI. For an example of a problem with making processor id's relative to a current "group", consider recv(source=DONT_CARE,...). Does this mean only in the current group? What is the result of MPI_INFOS() on the received message? What if two processors A and B are both in two groups alpha and beta, but with different rankings (ie., A = 2 and B = 5 in alpha, and A = 12 and B = 0 in beta). Now A (current group alpha) sends to B (current group beta). B does recv(source=DONT_CARE,...). Does B receive this message? What is the value of MPI_INFOS()? If B doesn't receive this message, then how is a communication context different from a group that is a copy of the current group (which would seem to supply the same semantics)? Matching of send and receive: We believe that there should be two types of matching operations: a simple (efficient) one on message type, and a general one using a user-defined predicate (matching function). Our experience has been that once the users have "select by source", they'll then want "select by source range and/or type range and/or message size and/or ...". We also agree that truncation of a received message into a too-short buffer should be an error (or at least detectable!). Syntax: We agree with multiple function names rather than multiple modes. It is easier to provide a efficient implementation and clear semantics in this case (for example, having the return code always have the same meaning independent of the input arguments). Note also that it is easy, given a multiple function names approach, to produce one-routine-does-everything interfaces but not vice-versa. Miscelania: We must have a non-blocking AWAIT. We'd like to AWAIT several messages. We must have a PROBE. We agree that the issue of returning data on received messages (particularly those done with nonblocking operations) is very important. One approach (used by one MPP vendor) is to ask for this info by handle, and then explicitly free the handle. This eliminates unnecessary global state (the currently most recently received message) from the semantics, and makes it possible to easily describe what exactly is returned by these "INFO" routines. Bill Gropp and Rusty Lusk From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 17:09:55 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22659; Wed, 9 Dec 92 17:09:55 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22401; Wed, 9 Dec 92 16:51:16 -0500 Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22397; Wed, 9 Dec 92 16:51:13 -0500 Received: from b125 (b125.super.org) by super.super.org (4.1/SMI-4.1) id AA08797; Wed, 9 Dec 92 16:51:11 EST Received: by b125 (4.1/SMI-4.1) id AA04506; Wed, 9 Dec 92 16:51:10 EST Date: Wed, 9 Dec 92 16:51:10 EST From: lederman@b125.super.org (Steve Lederman) Message-Id: <9212092151.AA04506@b125> To: mpi-pt2pt@cs.utk.edu Subject: thoughts I have a few thoughts on the ideas that have been passed around. I'm sorry for not responding sooner but a cold has slowed me down. First, on Marc's document of Nov. 28: - Goal 4 is for a C, C++, F77 and F90 binding. The draft Intro that Jack sent around on Nov. 30 states in the subsection for Who Should Use This Standard? that it is for Fortran 77 and/or C. Do we wish to include C++ and F90 which would likely to have different interfaces? - I find the SEND_RECV operation an interesting option. On the old Mark/?? machines from Caltech/JPL there was a call (if my recollection is correct) that allowed for the same user buffer to serve as the send and receive buffer. I assume this is not what is intended here? I think SEND_RECV would be used with some frequency by users. Would it simply be a convenience or can it be made faster than issuing the two calls. I would think it would be faster on some architectures (even more than the two call overhead). If it is faster then my inclination is to include it even though it adds another call. comments? - I have a question regarding when a communication operation is completed. On a send, does it terminate locally if the user buffer is copied into a system buffer but has not left the node? (I believe this is done on the CM5.) Analogously, does it terminate globally if it gets to a system buffer on the receiving node or until it is in the user buffer? As for choosing local, global or both I vote for the current choice of both. I can see applications for both. - I think that Marc's suggestion that MPI functions not return unsuccessfully has merit. It would be pain to be constantly checking the status after each operation to be safe. Along these lines is what to do if the send and recv message lengths are different. The current standard allows for fewer bytes to be received and the recv returns the actual number. Again, I think it would be a pain to have to check every time. If this is a desirable feature to allow different lengths, then I vote to make it users selectable. I know I have found many bugs by not having messages that were a different size being received. - I think that a PROBE function would be a good idea. Given that the standard is not currently going to support active messages, etc. it seems that being able periodically check if any messages have arrived would be a useful feature. Another thought: At the meeting in Minneapolis, several people, including myself, discussed the advantage of having a send where the sending process knows the receiving process has already posted a receive. This could relieve the send process of having to check with the receiving process since it knows it will not have to buffer the message in system space and it knows it is ready to receive the message. If a system could not take advantage of this then it could revert to a normal send. That is, the standard would not specify the way the send would occur, it would allow for an alternative protocol if the vendor wished. This is the rr (receiver ready) option in Bill Gropp and Ewing Lusk's implementation. I would like to propose we include this in the standard. Thoughts? Well enough for one message. Steve From owner-mpi-pt2pt@CS.UTK.EDU Mon Dec 14 15:48:42 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24886; Mon, 14 Dec 92 15:48:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17233; Mon, 14 Dec 92 15:48:21 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Dec 1992 20:48:20 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17227; Mon, 14 Dec 92 15:48:19 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA03749; Mon, 14 Dec 92 15:48:17 -0500 Date: Mon, 14 Dec 92 15:48:17 -0500 Message-Id: <9212142048.AA03749@thud.cs.utk.edu> To: mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum Forwarding: Mail from '"Dr. C.D. Wright" ' dated: Mon, 14 Dec 92 12:16:10 GMT ---------- Begin Forwarded Message ---------- >From @ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK Mon Dec 14 07:20:05 1992 Return-Path: <@ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK> Received: from mail.liv.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22696; Mon, 14 Dec 92 07:19:55 -0500 Received: from ibm.liverpool.ac.uk by mailhub.liverpool.ac.uk via JANET with NIFTP (PP) id <21042-0@mailhub.liverpool.ac.uk>; Mon, 14 Dec 1992 12:19:28 +0000 Received: from UK.AC.LIVERPOOL by MAILER(4.4.t); 14 Dec 1992 12:20:02 GMT Date: Mon, 14 Dec 92 12:16:10 GMT From: "Dr. C.D. Wright" Subject: Re: Message Passing Interface Forum To: dongarra@edu.utk.cs Message-Id: <"mailhub.li.044:14.11.92.12.19.28"@liverpool.ac.uk> Status: RO Hi. Since I am in the UK it is clear that I can't actively participate in the MPI Forum. I do, however, have one particular problem with every comms library I have used so far that I would like to see addressed in any new "standard", and I hope you can pass this on to whoever is the appropriate person to deal with it. In many packages such as PVM, PARMACS, p4, etc, it is possible to probe for and/or receive messages selectively, the selection being based on the message type (usually in integer) and/or the sender. This is overly restrictive. It would be far more useful if the message's format were sufficiently well defined for the user to be able to provide their own selection function to be passed in and used as the basis for reception and/or probing. That's it. Hope you can do something with this gripe/suggestion. Colin. ----------- End Forwarded Message ----------- From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 15 19:28:34 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21803; Tue, 15 Dec 92 19:28:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15587; Tue, 15 Dec 92 19:28:11 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 16 Dec 1992 00:28:10 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from helios.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15573; Tue, 15 Dec 92 19:28:06 -0500 Received: by helios.llnl.gov (4.1/LLNL-1.18) id AA11599; Tue, 15 Dec 92 16:30:03 PST Date: Tue, 15 Dec 92 16:30:03 PST From: tony@helios.llnl.gov (Anthony Skjellum) Message-Id: <9212160030.AA11599@helios.llnl.gov> To: dongarra@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum That is what we have been talking about in Zipcode for a long time. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Dec 25 19:15:36 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20495; Fri, 25 Dec 92 19:15:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09422; Fri, 25 Dec 92 19:15:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 26 Dec 1992 00:15:05 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09400; Fri, 25 Dec 92 19:15:04 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA14461; Fri, 25 Dec 92 19:14:59 -0500 Date: Fri, 25 Dec 92 19:14:59 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9212260014.AA14461@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: point-to-point proposal consistent with collective proposal Point-to-point Communication Proposal ------------------------------------- After reading Marc Snir's Message Passing Interface Outline and the other point-to-point comments, I would like to propose the following minimum set of point-to-point routines for discussion at the January meeting in Dallas. Stated Goals: 1. Design an application programming interface. 2. Design an interface that is not too different from current practice 3. Define an interface that can be quickly implemented on many vendor platforms. 4. Focus on a proposal that can be agreed upon in 6 months. 5. Provide a reliable communication interface. 6. Allow efficient communication. 7. Interface should not contain more functions than are really necessary. ----------------------------------------------------------------------------- General Comments: The collective communication interface should be an extension of the point-to-point interface. a. As Marc points out on page 2 "SEND and RECV are a particular case of broadcast in a group of size 2; this observation can be used to check if the definition of collective communication semantics are consistent with the definition of point-to-point communication." b. Collective routines like broadcast should provide the same message data format (contiguous, buffer with stride, etc.) as point-to-point routines. c. By using a structured name space (described on pages 7-8 of [Snir]) where all processes are identified by a (group,rank) pair for both the point-to-point and collective routines, then the users will have a consistent naming scheme across all the MPI communication routines. And those desiring a flat name space can have it by using the default group "ALL". d. Syntax guidelines - short, mnemonic names; systematic ordering of parameters; systematic naming convention. ------------------------------------------------------------------------------- Proposed point-to-point routines: 1. info = MPI_SEND( buf, bytes, type, gid, dest ) Function: Sends a contiguous message of length "bytes" pointed to by "buf" to the process identified by ("gid","dest") with a message tag of "type". On return "info" contains the error code. Conceptually the user can view this function as "I want this message delivered to the specified process in the fastest reliable manner." Implementation comments: On return user can consider buf is safe for reuse. MPI_SEND need not be strictly blocking. Users do not want to have to dance around the send buffer with STAUS and WAIT checking for when the send buffer is safe to change. If a compiler can determine that (or rearrange code so that) the send buffer is not touched before the send completes, then the implementation could be non-blocking. If a multiprocessor requires synchronous communication or routing along a particular topology, then this could be set in a global routine that affects all point-to-point and collective routines. 2. info = MPI_RECV( buf, bytes, type, gid, source ) Function: Receive the message specified by "type", "gid", and "source" into "buf". The message must be <= "bytes" in size or "info" returns error. Routine waits until specified message has arrived [like blocking recv]. On return "info" contains the error code. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. Conceptually the user can view this function as "I must have the specified message before I can do anything else. I will wait until it arrives." 3. info = MPI_NRECV( buf, bytes, type, gid, source ) Function: Checks to see if the specified message has arrived [like probe]. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. If the message has arrived it is returned in "buf" and "info" = 1. The message must be <= "bytes" in size or "info" returns error. If the message has not arrived, then the routine returns with "info" = 0. Values less than 0 returned by "info" indicate an error. MPI_NRECV can be called multiple times to check on the arrival of a message, and the user can later call MPI_RECV to wait on the same message. Conceptually the user can view this function as "I would like to receive the specified message now, but if it is not here, I will not wait for it. I will go do something else." Implementation is not specified here, but for efficiency, implementors may wish to post a receive on the first call to MPI_NRECV and just do checks [like status] on repeat calls. If implemented like this, MPI_RECV must be able to check both the system queue and the posted queue. 4. type = MPI_TYPE() 5. bytes = MPI_BYTES() 6. gid = MPI_GID() 7. source = MPI_SOURCE() Function: These routines return the actual values for the last message returned by MPI_RECV or MPI_NRECV. Values < 0 indicate an error. Allows the user to determine the size of the received message and the values of "don't care" arguments. 8. bytes = MPI_PACK( data_type, nitems, stride, inbuf, outbuf ) 9. bytes = MPI_UNPACK( data_type, nitems, stride, inbuf, outbuf ) Function: General purpose routines for packing and unpacking an arbitrarily complex message buffer. Allows for non-contiguous messages, mixed data types in a single message, and heterogeneity between machines. Same routines can be used to build collective communication buffers, for example for MPI_BCAST. On return "outbuf" contains the (potentially) encoded data, and "bytes" is the size in bytes of the encoded data. "bytes" < 0 indicates an error. The following data types would be supported: BYTE INTEGER*2 INTEGER*4 REAL*4 REAL*8 COMPLEX*8 COMPLEX*16 I would also like to see the following data types considered: CHARACTER_STRING INTEGER*8 REAL*16 COMPLEX*32 "nitems" is the total number of items of this data type to pack or unpack from "inbuf". "stride" can be used for packing rows of a matrix in Fortran or columns of a matrix in C. MPI_PACK can be called multiple times to construct a single message buffer. MPI_UNPACKs must be called by the user to decode such a buffer on receipt. Neither of these routines need be called if the user does not require non-contiguous messages, mixed data types, or heterogeneity. Having separate packing and unpacking routines allows send and receive to be optimized for a contiguous stream of bytes, and simplifies future extensions of MPI into areas such as heterogeneity. =============================================================================== For completeness here is the present proposal from the collective communication subcommittee. 1. info = MPI_BCAST( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return the contents of "buf" on "root" is contained in "buf" on all group members. On return "info" contains the error code. 2. info = MPI_GATHER( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return all the individual "buf" are concatenated into the "root" buf, which must be of size at least gsize*bytes where gsize is group size. The data is laid in the "root" buf in rank order that is | gid,0 data | gid,1 data | ...| gid, root data | ...| gid, gsize-1 data | Other member's "buf" are unchanged on return. On return "info" contains the error code. 3. info = MPI_GLOBAL_OP( inbuf, bytes, type, gid, op, outbuf ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "op". On return the "outbuf" of all group members contains the result of the global operation "op" applied pointwise to the collective "inbuf". For example, if the op is max and inbuf contains two float point numbers then outbuf(1) = global max( inbuf(1)) and outbuf(2) = global max( inbuf(2)) A set of standard operations are supplied with MPI including: global max - for each data type global min - for each data type global sum - for each data type global mult- for each data type global AND - for integer and logical type global OR - for integer and logical type global XOR - for integer and logical type Optionally the users may define their own global functions for this routine. On return "info" contains the error code. 4. info = MPI_SYNCH( gid ) Function: Called by all members of the group "gid" Returns only when all members have called this function. On return "info" contains the error code. 5. gid = MPI_MKGROUP( list_of_processes ) Function: Called by all processes in the list. Forms a logical group containing the listed processes and assigns each process a unique rank in the group. The ranks are consecutively numbered from 0 to gsize-1. On return "gid" is an MPI assigned group ID (or error code if < 0) 6. gsize = MPI_GROUPSIZE( gid ) Function: Can be called by any process. On return "gsize" is the number of members in the group "gid" (or error code if < 0). 7. rank = MPI_MYRANK( gid ) Function: Can be called only by members of group "gid". On return "rank" is the rank of the calling process in group "gid" (an integer between 0 and gsize-1) or error code if < 0. =========================================================================== From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:50:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19230; Mon, 4 Jan 93 08:50:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07603; Mon, 4 Jan 93 08:49:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:49:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07594; Mon, 4 Jan 93 08:49:43 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06801 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:49:38 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15719; Mon, 4 Jan 93 13:49:35 GMT Date: Mon, 4 Jan 93 13:49:35 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041349.AA15719@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02659; Mon, 4 Jan 93 13:49:02 GMT To: mpi-pt2pt@cs.utk.edu Subject: Multifarious Content-Length: 6062 Gentlepeople, Firstly a general point ======================= I'd like to add an additional criterion to the goals of the message passing interface. 8) The implementation and semantics of the interface should remain simple and "obvious" when multiple "lightweight" threads share the same address space. I'd like this because 1) Lightweight thread libraries on the Posix model are becoming available on most workstations (networks of which are one of our targets). 2) There is a clear industry trend to having a few processors on a logically (and often physically) shared memory both in workstations (e.g. SparcStation 10), and servers (SparcServer 690, RS6000 ??). It would be foolish to go down a path which made exploitation of such machines hard. Undoubtedly this trend will also occur in future MPP machines (it would "only" take a software change to the code on the Paragon to make it be like this. After all it has all the hardware there already). As a consequence of this I'd like to remove as much hidden state as possible from the message passing system, so 1) I dislike the functions which return information about "the last message", since in a single process with multiple threads there can be many "last" messages. The suggestion that these operations be performed on descriptors returned from the probe is much preferable (and semantically cleaner). 2) I dislike the idea of setting options in a global routine, because there may be other threads performing operations at the same time as the options are changing, and the semantics becomes very hard to pin down. Secondly A few comments on Al Geist's proposal of Christmas Day =============================================================== 1. MPI_SEND I liked the separation in the original draft between the concepts of blocking (the buffer is now free) and synchronous (the message has been received at the other end) communications. I'd like to preserve both possibilities. 3. MPI_NRECV I don't understand how this is intended to work. Consider the following code. char buffer[100]; int i; for (i=0; i<10; i++) { if (&MPI_NRECV(buffer[10*i],10,1,1,1)) break; } Questions : 1) Into which elements of buffer will the receive occur ? Al's implementation note suggests just the first, so what then do the other (up to) 9 calls achieve ? Will they receive the next nine matching messages ? (this doesn't appear very intuitive for a "polling" routine) 2) Should I now do a blocking MPI_RECV as well ? Or does this only make sense if the MPI_NRECVs all fail ? But if they do fail, is the receive buffer still queued to receive a matching message when it becomes available ? (and if so how do I cancel the outstanding receive and get back ownership of my buffer ?) 4,5,6,7 See comments above about this style of working. 8,9 If we're only having one form of packing, then it should probably be the more general one with arrays of pointers and counts, since it can subsume (admittedly at some cost) the strided style. Conclusion ========== I prefer the original draft to Al's version. I'd like to change the mode argument in the original draft (see my mail to mpi-lang). Removing internal state ======================= I'd also like to remove the hidden state ("last message"), there are various ways of achieving this, none of which is exceptionally attractive. 1) Return a structure from MPI_RECV (and friends) This is fine in C, but not good in Fortran 77 (and may be more expensive than we want when you don't need any of the information). e.g. typedef struct MPI_REPLY { int status; int nbytes; int whoFrom; int tag; /* More ? */ } MPI_REPLY; { MPI_REPLY res; res = MPI_RECV(...); if (res.whoFrom == 0) ... etc ... } 2) Return results indirect through individual parameters (arguments for the Fortran inclined) This works, but is very error prone, particularly in Fortran when the arguments are INOUT. (We have an interface like this at the moment, and it does cause problems). 3) Return a tag on which queries can be made. This means that you must then have another call to the system to dispose of the tag (and associated system storage). This is unpleasant on a simple blocking receive when you don't want any of the information (since it's another call to the library). 4) Have additional (possibly optional) OUT arguments through which the results are returned if required. Using optional keyword arguments in F90 this works fine. C can be made to work more or less. F77 can't (within the standard). 5) Have one required argument which is a pointer to a reply structure like that used in 1) which is then filled in by the function. This can work in all the languages. In C you could also allow NULL to mean no such results are required. IMHO Solution 5 is best. If we wanted to (and we may) then we could disallow direct access to this structure and provide functions to pick it apart (these could be macros in C, inline functions in C++). This would avoid having to specify the contents of the structure and allow for future expansion. Comments on Steve Lederman's note ================================= > At the meeting in Minneapolis, several people, including myself, > discussed the advantage of having a send where the sending process > knows the receiving process has already posted a receive. We have this in our current message passing model. It has proved to be an awful mistake. 98% of user's don't know when to use it, and then complain that their code no longer works when they've used it incorrectly. Please let's not do this. (I'm speaking from experience here). See you all in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:59:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19312; Mon, 4 Jan 93 08:59:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08031; Mon, 4 Jan 93 08:59:34 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:59:32 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08019; Mon, 4 Jan 93 08:59:30 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06920 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:59:25 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15744; Mon, 4 Jan 93 13:59:21 GMT Date: Mon, 4 Jan 93 13:59:21 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041359.AA15744@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02661; Mon, 4 Jan 93 13:58:46 GMT To: mpi-pt2pt@cs.utk.edu Subject: Incorrect Example Content-Length: 311 Appologies, the example should (of course) read for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 09:33:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19844; Mon, 4 Jan 93 09:33:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09262; Mon, 4 Jan 93 09:32:48 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 14:32:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09242; Mon, 4 Jan 93 09:32:36 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA07241 (5.65c/IDA-1.4.4); Mon, 4 Jan 1993 09:32:32 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15874; Mon, 4 Jan 93 14:32:28 GMT Date: Mon, 4 Jan 93 14:32:28 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041432.AA15874@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02690; Mon, 4 Jan 93 14:31:52 GMT To: mpi-pt2pt@cs.utk.edu, mpi-lang@cs.utk.edu Subject: Profilers etc. Content-Length: 2528 Gentlepeople, I have an implementation issue which I would like to raise in the MPI forum, since it is unclear where it fits into the sub-committee structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to those of you who receive this message twice. Issue ===== The major objective of MPI1 is to achieve portability of applications. This has major benefits for us all (not least in legitimising and therefore growing the MPP marketplace). One of the benefits which it would also be nice to achieve would be the wide availability of different tools which support programming in the MPI1 model. The most immediately obvious such tools (to me at least !) are 1) HPF to Fortran + MPI1 translators 2) Performance monitoring/tuning tools 3) Debuggers Support for the first is easy (since it just requires what we're already doing). Portable support for the second is not so trivial, since the collection of useful performance information is much more intrusive. Portable support for the third is harder still, and I won't discuss it further. Options ======= We have various possible options which we can take. 1) Ignore the problem Provide no support for portable performance monitoring tools, and leave each tool provider with a large porting problem. I don't like this solution, it loses some of the benefit of the standard, which should be attracting people to build tools. 2) Document specific implementation hooks as part of MPI1. In effect these would be callbacks from the library to profiling code which could then do whatever it liked. 3) As 2, but without REQUIRING that a conforming implementation provide the functions. They're there as a recommendation, rather than being mandatory. I think we should be concerned about this, and I'd like us at least to make some recommendation. Personally I'd probably implement two separate interface to the library, one of which provided the hooks, and the other of which didn't so that you don't pay the cost of checking the profiling hooks unless you asked to. Of course even if we do nothing it's not too hard to escape (using horrible macros) in C, but Fortran doesn't always have macros, so a properly specified internal solution is definitely preferable. Thoughts ??? Flame me at Dallas. I'm travelling tomorrow. When are we having a meeting in Europe ??? -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:26:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21722; Mon, 4 Jan 93 10:26:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11937; Mon, 4 Jan 93 10:25:39 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:25:38 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11921; Mon, 4 Jan 93 10:25:36 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA02891; Mon, 4 Jan 93 10:25:20 -0500 Date: Mon, 4 Jan 93 10:25:20 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9301041525.AA02891@msr.EPM.ORNL.GOV> To: jim@meiko.co.uk, mpi-pt2pt@cs.utk.edu Subject: NRECV example explained. Hi Jim, Your points are well taken given your suggested additional goals for MPI. You asked some questions about an example you gave. char buffer[100]; int i; for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } > 1) Into which elements of buffer will the receive occur ? The MPI_NRECV call says I want to receive a message not to exceed 10 bytes which matches type = 1, groupID = 1, source = 1. If the message has already arrived before the NRECV call, then it will be placed in buffer[0:9]. If it arrives while the 'for' loop is still executing, then it will be placed in buffer[10*i:10*i+10]. If it arrives later, then the user will need to ask for it later with either MPI_NRECV or MPI_RECV. All he can say for sure is the message did not arrive while I was polling for it. It is a strange example in that the user asks for a single message but specifies 10 different places where it should be stored. Ultimately, it the successful RECV or NRECV that specifies where the message is stored. > 2) Should I now do a blocking MPI_RECV as well ? Or does this only > make sense if the MPI_NRECVs all fail ? But if they do fail, is > the receive buffer still queued to receive a matching message when > it becomes available ? (and if so how do I cancel the outstanding > receive and get back ownership of my buffer ?) It only makes sense to call MPI_RECV if all MPI_NRECVs fail. It is not specified that the recv buffer ever has to be queued. An efficient immplementation may queue a failed NRECV in the educated guess that the user will ask for the message later in the same buffer. Then when the user does ask and the RECV or NRECV is successful the system avoids one copy of the message. Technically the user never loses ownership of the buffer. He can do anything he wants with it. If he is sure the posted message will never arrive, then he can post a message from another source to that same buffer. Cheers! Al From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:44:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22293; Mon, 4 Jan 93 10:44:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12552; Mon, 4 Jan 93 10:44:12 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:44:11 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12544; Mon, 4 Jan 93 10:44:08 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA08401 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 10:44:05 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA16038; Mon, 4 Jan 93 15:44:00 GMT Date: Mon, 4 Jan 93 15:44:00 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041544.AA16038@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02725; Mon, 4 Jan 93 15:43:24 GMT To: geist@msr.EPM.ORNL.GOV Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Mon, 4 Jan 93 10:25:20 -0500 <9301041525.AA02891@msr.EPM.ORNL.GOV> Subject: NRECV example explained. Content-Length: 1350 Thanks Al, I now understand what you're trying to do, but I still don't think that your suggested "efficient implementation" is safe. Consider an example like this (less perverted than my previous one, which was of course constructed solely to make a point). strcpy(buffer0,"Something"); if (MPI_NRECV(buffer0, 10, 1, 1, 1)) /* Whatever */ else { sleep(20); /* Wait for the message to arrive, * For the sake of argument assume it does. */ MPI_RECV(buffer1, 10, 1, 1, 1); if (strcmp(buffer0,"Something") != SAME) bug("Buffer changed"); } Your suggested "efficient" implementation will surely overwrite buffer0 (which it should NOT do in your semantic model, since "Ultimately, it the successful RECV or NRECV that specifies where the message is stored."). I have to say I much prefer a model in which the queuing and dequeuing of buffers is explicit. Then it is clear where the current ownership of the buffer resides at any time, and when the additional copy can be avoided. (Though of course there is a lovely race to be resolved in the cancel...) Maybe we can talk about this further over a beer if you're going to be in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 12:37:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20368; Tue, 5 Jan 93 12:37:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13588; Tue, 5 Jan 93 12:37:19 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 17:37:18 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13576; Tue, 5 Jan 93 12:37:16 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA22236; Tue, 5 Jan 93 17:37:12 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00742; Tue, 5 Jan 93 10:36:12 MST Date: Tue, 5 Jan 93 10:36:12 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301051736.AA00742@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Multifarious I have a few short comments on Jim's comments... > > Removing internal state > ======================= > > I'd also like to remove the hidden state ("last message"), there are > various ways of achieving this, none of which is exceptionally > attractive. > > 1) Return a structure from MPI_RECV (and friends) > This is fine in C, but not good in Fortran 77 (and may be more > expensive than we want when you don't need any of the information). > > e.g. > typedef struct MPI_REPLY > { > int status; > int nbytes; > int whoFrom; > int tag; > /* More ? */ > } MPI_REPLY; > > { > MPI_REPLY res; > > res = MPI_RECV(...); > > if (res.whoFrom == 0) > > ... etc ... > } > > > 2) Return results indirect through individual parameters (arguments for the > Fortran inclined) > > This works, but is very error prone, particularly in Fortran when > the arguments are INOUT. (We have an interface like this at the > moment, and it does cause problems). > > 3) Return a tag on which queries can be made. > > This means that you must then have another call to the system to > dispose of the tag (and associated system storage). This is > unpleasant on a simple blocking receive when you don't want any of > the information (since it's another call to the library). > > 4) Have additional (possibly optional) OUT arguments through which the > results are returned if required. > > Using optional keyword arguments in F90 this works fine. > C can be made to work more or less. > F77 can't (within the standard). > > 5) Have one required argument which is a pointer to a reply structure > like that used in 1) which is then filled in by the function. > > This can work in all the languages. > In C you could also allow NULL to mean no such results are > required. > > > IMHO Solution 5 is best. > > If we wanted to (and we may) then we could disallow direct access to > this structure and provide functions to pick it apart (these could be > macros in C, inline functions in C++). This would avoid having to > specify the contents of the structure and allow for future expansion. > > > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England I also like Solution 5 and like the idea of providing functions to access the reply structure. "Hiding" the data structure in this way makes it easy to change things in later revisions when we (inevitably) find out we've forgotten something. Tom Henderson Forecast Systems Laboratory (NOAA) hender@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 15:52:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01504; Tue, 5 Jan 93 15:52:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22365; Tue, 5 Jan 93 15:51:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 20:51:34 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22325; Tue, 5 Jan 93 15:50:59 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA21624; Tue, 5 Jan 93 14:50:42 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA17798; Tue, 5 Jan 93 14:50:40 CST Date: Tue, 5 Jan 93 14:50:40 CST Message-Id: <9301052050.AA17798@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Comments on "ready receiver" part of draft implementation Here is some text supporting the idea of "ready receivers". Bill Gropp and Rusty Lusk %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: rr.dvi %%Pages: 3 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}B /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}{initclip}ifelse ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}B /@endspecial{grestore clear SpecialSave restore end}B /@defspecial{SDict begin}B /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}B /st{stroke SaveX SaveY moveto}B /fil{fill SaveX SaveY moveto}B /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 58 123 df<001F83E000F06E3001C078780380F8 780300F03007007000070070000700700007007000070070000700700007007000FFFFFF800700 700007007000070070000700700007007000070070000700700007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070007FE3FF001D20809F 1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<7038F87CFC7EFC7E743A 0402040204020804080410081008201040200F0E7E9F17>34 D<70F8FCFC740404040808101020 40060E7C9F0D>39 D<0020004000800100020006000C000C001800180030003000300070006000 60006000E000E000E000E000E000E000E000E000E000E000E000E0006000600060007000300030 003000180018000C000C000600020001000080004000200B2E7DA112>I<800040002000100008 000C00060006000300030001800180018001C000C000C000C000E000E000E000E000E000E000E0 00E000E000E000E000E000C000C000C001C001800180018003000300060006000C000800100020 00400080000B2E7DA112>I<70F8FCFC74040404080810102040060E7C840D>44 DI<70F8F8F87005057C840D>I<000100030003000600060006000C000C 000C00180018001800300030003000600060006000C000C000C001800180018003000300030006 00060006000C000C000C00180018001800300030003000600060006000C000C000C000102D7DA1 17>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F003C0F003C0 F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0700380700380700380 7807803807001806001C0E000E1C0003F000121F7E9D17>I<018003800F80F380038003800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 80038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0F803C0F8 03C0F803C02007C00007C0000780000780000F00000E00001C0000380000700000600000C00001 80000300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F0000C1C00 100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C0000380003F000 003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F00780400780400F00 200E001C3C0003F000121F7E9D17>I<007C000182000701000E03800C07801C07803803003800 00780000700000700000F1F000F21C00F40600F80700F80380F80380F003C0F003C0F003C0F003 C0F003C07003C07003C07003803803803807001807000C0E00061C0001F000121F7E9D17>54 D<03F0000C0C001006003003002001806001806001806001807001807803003E03003F06001FC8 000FF00003F80007FC000C7E00103F00300F806003804001C0C001C0C000C0C000C0C000C0C000 806001802001001002000C0C0003F000121F7E9D17>56 D<70F8F8F87000000000000000000000 70F8F8F87005147C930D>58 D<000100000003800000038000000380000007C0000007C0000007 C0000009E0000009E0000009E0000010F0000010F0000010F00000207800002078000020780000 403C0000403C0000403C0000801E0000801E0000FFFE0001000F0001000F0001000F0002000780 0200078002000780040003C00E0003C01F0007E0FFC03FFE1F207F9F22>65 D<000FC040007030C001C009C0038005C0070003C00E0001C01E0000C01C0000C03C0000C07C00 00407C00004078000040F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000407C0000403C0000401C0000401E0000800E00008007000100 0380020001C0040000703800000FC0001A217D9F21>67 DI70 D73 D76 DII<001F800000F0F00001C0380007801E000F000F000E0007001E0007803C 0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0 F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E0007 800E0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C198010078030038060 0180600180E00180E00080E00080E00080F00000F000007800007F00003FF0001FFC000FFE0003 FF00001F800007800003C00003C00001C08001C08001C08001C08001C0C00180C00380E00300F0 0600CE0C0081F80012217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F0030 800F0010800F0010800F0010800F0010000F0000000F0000000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21>I87 D<080410082010201040204020804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>92 D<1FE000303000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E 00F00E10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E3E000EC3800F01C00F00E0 0E00E00E00700E00700E00780E00780E00780E00780E00780E00780E00700E00700E00E00F00E0 0D01C00CC300083E0015207F9F19>I<03F80E0C1C1E381E380C70007000F000F000F000F000F0 00F00070007000380138011C020E0C03F010147E9314>I<000380003F80000380000380000380 00038000038000038000038000038000038000038003E380061B801C0780380380380380700380 700380F00380F00380F00380F00380F00380F003807003807003803803803807801C07800E1B80 03E3F815207E9F19>I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF 80F00000F00000F000007000007000003800801800800C010007060001F80011147F9314>I<00 7C00C6018F038F07060700070007000700070007000700FFF00700070007000700070007000700 070007000700070007000700070007000700070007007FF01020809F0E>I<0000E003E3300E3C 301C1C30380E00780F00780F00780F00780F00780F00380E001C1C001E380033E0002000002000 003000003000003FFE001FFF800FFFC03001E0600070C00030C00030C00030C000306000603000 C01C038003FC00141F7F9417>I<0E0000FE00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E3E000E43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16207F9F19>I< 1C001E003E001E001C000000000000000000000000000E007E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000 E0000000000000000000000000007007F000F00070007000700070007000700070007000700070 007000700070007000700070007000700070007000706070F060F0C061803F000C28829E0E>I< 0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF0 0E03C00E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E00 0E07000E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E81C81C000F00F00E000F 00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00FFE7FE7F E023147F9326>I<0E3E00FE43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16147F9319>I<01F800 070E001C03803801C03801C07000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E0 7000E03801C03801C01C0380070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00 E00E00F00E00700E00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01 C00EC3000E3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319 >I<03E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380F0 03807003807803803803803807801C0B800E138003E38000038000038000038000038000038000 0380000380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E000E000E000E000E 000E000E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030704030C010C010C010E0 0078007F803FE00FF00070803880188018C018C018E030D0608F800D147E9312>I<0200020002 00060006000E000E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E080E08 0E080E080E080610031001E00D1C7F9B12>I<0E01C0FE1FC00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00603C0030DC001F1 FC16147F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C00 007C00004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000E001E001C00380078007010E01 1E011C0338027006700EFFFE10147F9314>I E /Fb 14 122 df<01FFFFC0001E00F0001E0078 001E0038001E003C003C003C003C003C003C003C003C003C0078007800780078007800F0007801 E000F0078000FFFE0000F00F8000F003C001E001C001E001E001E001E001E001E003C001E003C0 01E003C001E003C001C0078003C00780078007800F0007801E000F007800FFFFE0001E1F7D9E20 >66 D<0000FC040007030C001C00980030007800E0007801C00038038000300380003007000030 0E0000301E0000201E0000203C0000003C00000078000000780000007800000078000000F00000 00F000FFF0F0000780F0000780F0000F0070000F0070000F0070000F0070001E0038001E001800 3E001C002E000E00CC000383040000FC00001E217A9F23>71 D<01FFF800001F0000001E000000 1E0000001E0000003C0000003C0000003C0000003C000000780000007800000078000000780000 00F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0008003C0010003C001 0003C0030003C00200078006000780060007800C0007801C000F007800FFFFF800191F7D9E1D> 76 D<01FFFF00001E03C0001E00E0001E0070001E0078003C0078003C0078003C0078003C0078 007800F0007800F0007801E0007801C000F0070000F01E0000FFF00000F0380001E01C0001E01E 0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C0807803C080780 3C100F801C10FFF00C20000007C01D207D9E21>82 D<00C001E001E001C0000000000000000000 00000000000E003300230043804300470087000E000E000E001C001C001C003840388030807080 310033001C000B1F7C9E0E>105 D<01E0000FE00001C00001C00001C00001C000038000038000 0380000380000700000700000703C00704200E08E00E11E00E21E00E40C01C80001D00001E0000 1FC00038E000387000387000383840707080707080707080703100E03100601E0013207D9F15> 107 D<03C01FC0038003800380038007000700070007000E000E000E000E001C001C001C001C00 38003800380038007000700070007100E200E200E200E200640038000A207C9F0C>I<007C0001 C3000301800E01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F0 0700700F00700E0030180018700007C00013147C9317>111 D<01C1E002621804741C04781C04 701E04701E08E01E00E01E00E01E00E01E01C03C01C03C01C03C01C0380380780380700380E003 C1C0072380071E000700000700000E00000E00000E00000E00001C00001C0000FFC000171D8093 17>I<1C1E002661004783804787804707804703008E00000E00000E00000E00001C00001C0000 1C00001C000038000038000038000038000070000030000011147C9313>114 D<00FC030206010C030C070C060C000F800FF007F803FC003E000E700EF00CF00CE00840102060 1F8010147D9313>I<018001C0038003800380038007000700FFF007000E000E000E000E001C00 1C001C001C003800380038003820704070407080708031001E000C1C7C9B0F>I<0E00C03300E0 2301C04381C04301C04701C08703800E03800E03800E03801C07001C07001C07001C07101C0E20 180E20180E201C1E200C264007C38014147C9318>I<0E00C03300E02301C04381C04301C04701 C08703800E03800E03800E03801C07001C07001C07001C07001C0E00180E00180E001C1E000C3C 0007DC00001C00001C00003800F03800F07000E06000C0C0004380003E0000131D7C9316>121 D E /Fc 20 117 df45 D<000003800000000007C00000000007C0000000000FE0000000000FE0000000000FE000000000 1FF0000000001FF0000000003FF8000000003FF8000000003FF80000000073FC0000000073FC00 000000F3FE00000000E1FE00000000E1FE00000001C0FF00000001C0FF00000003C0FF80000003 807F80000007807FC0000007003FC0000007003FC000000E003FE000000E001FE000001E001FF0 00001C000FF000001FFFFFF000003FFFFFF800003FFFFFF80000780007FC0000700003FC000070 0003FC0000E00001FE0000E00001FE0001E00001FF0001C00000FF0001C00000FF00FFFE001FFF FEFFFE001FFFFEFFFE001FFFFE2F297EA834>65 D 72 D77 DI80 D<01FF800007FFF0000F81F8001FC0 7E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007 FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F80 7F00DF803F839FFC0FFF0FFC01FC03FC1E1B7E9A21>97 D<001FF80000FFFE0003F01F0007E03F 800FC03F801F803F803F801F007F800E007F0000007F000000FF000000FF000000FF000000FF00 0000FF000000FF000000FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007 E0070003F01E0000FFFC00001FE0001A1B7E9A1F>99 D<003FE00001FFF80003F07E0007C01F00 0F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFF E0FF000000FF000000FF0000007F0000007F0000007F0000003F8000E01F8000E00FC001C007E0 038003F81F0000FFFE00001FF0001B1B7E9A20>101 D<0007F0003FFC00FE3E01F87F03F87F03 F07F07F07F07F03E07F00007F00007F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007 F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007 F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF807FFF80182A7EA915>I< 00FF81F003FFE7F80FC1FE7C1F80FC7C1F007C383F007E107F007F007F007F007F007F007F007F 007F007F007F007F003F007E001F007C001F80FC000FC1F8001FFFE00018FF8000380000003800 00003C0000003E0000003FFFF8001FFFFF001FFFFF800FFFFFC007FFFFE01FFFFFF03E0007F07C 0001F8F80000F8F80000F8F80000F8F80000F87C0001F03C0001E01F0007C00FC01F8003FFFE00 007FF0001E287E9A22>II<07000F801FC0 3FE03FE03FE01FC00F8007000000000000000000000000000000FFE0FFE0FFE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFE FFFE0F2B7DAA14>I108 DII<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E0 7F0007F07F0007F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007 F8FF0007F87F0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003F E0001D1B7E9A22>I114 D<03FE300FFFF01E03F03800F0700070F00070 F00070F80070FC0000FFE0007FFE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007C E0003CF0003CF00038F80038FC0070FF01E0F7FFC0C1FF00161B7E9A1B>I<0070000070000070 0000700000F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F0 3807F03807F03807F03807F03803F03803F87001F86000FFC0001F8015267FA51B>I E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 117 7 a Fc(A)23 b(Note)f(on)i(Access)e(to)h(High-P)n(erformance)e (Message-P)n(assing)832 94 y(Proto)r(cols)869 240 y Fb(Bil)r(l)16 b(Gr)n(opp)864 297 y(R)o(usty)g(Lusk)526 353 y Fa(Mathematics)f(and)g (Computer)g(Science)i(Division)678 409 y(Argonne)e(National)h(Lab)q(oratory) 145 546 y(The)22 b(MPI)h(Implemen)o(tation)g(T)l(estb)q(ed)g(con)o(tains)g (what)f(are)g(called)i(there)e(\\Ready-Receiv)o(er")74 602 y(v)o(ersions)17 b(of)g(the)g(message-passing)g(primitiv)o(es.)27 b(The)17 b(purp)q(ose)h(of)e(this)i(brief)f(note)g(is)h(to)e(giv)o(e)h(some) 74 659 y(explanation)e(of)f(what)g(wh)o(y)f(w)o(e)h(think)h(something)g(lik)o (e)g(this)f(needs)h(to)f(b)q(e)h(considered)g(as)f(part)f(of)h(the)74 715 y(ev)o(en)o(tual)i(standard)e(and)i(to)e(rep)q(ort)h(on)g(some)g(exp)q (erimen)o(ts)h(that)f(supp)q(ort)g(this)h(opinion.)145 802 y(It)21 b(is)g(quite)h(p)q(ossible)h(to)e(enco)q(de)h(message-passing)f (algorithms)g(with)h(only)f(a)g(single)h(t)o(yp)q(e)g(of)74 858 y(send)d(and)g(receiv)o(e,)h(sa)o(y)e(a)g(blo)q(c)o(king)i(send)f(and)g (a)f(blo)q(c)o(king)i(receiv)o(e.)30 b(Most)18 b(of)g(the)g(v)m(ariations)h (are)74 914 y(to)c(enable)h(increased)g(p)q(erformance,)f(p)q(ossibly)i(at)d (the)h(exp)q(ense)i(of)d(increased)i(program)e(complexit)o(y)l(.)74 971 y(An)19 b(example)h(of)f(this)g(is)g(the)g(general)h(consensus)f(that)g (w)o(e)f(need)i(non-blo)q(c)o(king)h(send)e(and)g(receiv)o(e)74 1027 y(op)q(erations)c(in)i(order)d(to)h(o)o(v)o(erlap)g(computation)g(with)h (comm)o(unication.)145 1114 y(It)11 b(is)h(imp)q(ortan)o(t)f(not)g(to)g(stop) g(to)q(o)g(so)q(on,)g(ho)o(w)o(ev)o(er,)g(in)h(pro)o(viding)h(supp)q(ort)e (for)g(high)h(p)q(erformance.)74 1170 y(As)k(time)g(go)q(es)f(b)o(y)l(,)h(v)o (endors)f(will)j(pro)o(vide)e(op)q(erations)g(that)f(are)g(faster)g(than)g (the)h(ones)g(that)f(can)g(b)q(e)74 1227 y(sp)q(eci\014ed)i(with)d(the)g(op)q (erations)h(in)g(the)f(initial)j(standard)c(prop)q(osal.)20 b(It)15 b(is)f(p)q(ossible)i(to)e(an)o(ticipate)h(at)74 1283 y(least)g(some)g(of)g(these)h(op)q(erations)f(and)g(pro)o(vide)h(for)f(them)g (in)h(the)f(MPI)g(standard.)145 1370 y(One)j(example)g(o)q(ccurs)f(in)h(the)g (situation)f(when)h(receiv)o(es)g(ha)o(v)o(e)f(b)q(een)h(issued)g(prior)g(to) e(the)i(send)74 1426 y(op)q(eration)e(that)f(is)i(exp)q(ected)g(to)e(satisfy) h(them.)22 b(In)16 b(suc)o(h)g(a)g(case)g(a)f(considerable)j(amoun)o(t)d(of)h (hand-)74 1483 y(shaking)23 b(can)f(b)q(e)g(eliminated)i(since)f(the)f (bu\013ers)g(ha)o(v)o(e)g(already)g(b)q(een)h(allo)q(cated.)41 b(On)22 b(the)g(In)o(tel)74 1539 y(IPSC/860)c(and)g(the)g(Delta,)g(one)h(can) f(tak)o(e)f(adv)m(an)o(tage)h(of)g(this)g(b)o(y)g(using)h(\\force)f(t)o(yp)q (es".)28 b(Almost)74 1595 y(all)20 b(users)f(of)f(the)h(Delta)f(who)h(are)f (seeking)i(go)q(o)q(d)e(p)q(erformance)h(use)g(this)g(tec)o(hnique.)32 b(Its)19 b(name)g(is)74 1652 y(misleading,)d(since)f(it)f(do)q(esn't)g(ha)o (v)o(e)g(an)o(ything)g(to)f(do)h(with)h(t)o(yp)q(es.)k(The)14 b(imp)q(ortan)o(t)g(thing)g(to)g(realize)74 1708 y(is)19 b(that)f(it)i(is)f (not)f(just)h(an)f(arcane)h(v)o(endor-sp)q(eci\014c)i(optimization,)f(but)f (a)f(general)i(situation)f(that)74 1765 y(transcends)13 b(an)o(y)g (particular)h(implemen)o(tation.)20 b(The)13 b(general)h(situation)f(is)h (that)e(when)i(receiv)o(es)g(ha)o(v)o(e)74 1821 y(b)q(een)i(issued)h(ahead)e (of)g(time,)g(message)g(latency)h(can)f(b)q(e)h(greatly)f(decreased.)145 1908 y(W)l(e)e(conducted)h(some)f(preliminary)h(exp)q(erimen)o(ts)g(on)f(the) g(IPSC/860)g(with)g(the)g(MPI)g(implemen-)74 1964 y(tation)19 b(testb)q(ed,)g(using)h(the)f(\\rr")f(v)o(ersions)h(of)f(send)i(and)f(receiv) o(e)h(that)e(it)h(supplies.)33 b(\(\\rr")17 b(stands)74 2021 y(for)d(\\ready)g(receiv)o(er".\))20 b(The)15 b(test)f(program)f(just)i (ping-p)q(ongs)g(a)g(message)f(bac)o(k)g(and)h(forth)f(b)q(et)o(w)o(een)74 2077 y(t)o(w)o(o)20 b(no)q(des.)39 b(The)22 b(follo)o(wing)g(graphs)f(sho)o (w)g(the)h(p)q(erformance)f(impro)o(v)o(emen)o(ts)g(pro)o(vided)i(b)o(y)e (the)74 2134 y(ready-receiv)o(er)16 b(op)q(erations.)145 2220 y(This)g(next)f(test)f(sho)o(ws)h(the)g(b)q(eha)o(vior)h(in)g(sending)g (around)g(a)e(ring.)963 2790 y(1)p eop %%Page: 2 2 bop 74 2161 a @beginspecial 72 @llx 72 @lly 504 @urx 504 @ury 4320 @rwi @setspecial %%BeginDocument: ftime.ps .24 .24 scale /g0dict 40 dict def g0dict begin /m { moveto } bind def /a { rmoveto } bind def /l { lineto } bind def /v { 0 exch rlineto } bind def /h { 0 rlineto } bind def /s { stroke } bind def /n { newpath } bind def /r { rlineto } bind def /c { closepath } bind def /f { fill } bind def /p { copypage erasepage } def /g { setgray } bind def /b { newpath moveto lineto stroke } bind def /d { newpath moveto 0 exch rlineto stroke } bind def /e { newpath moveto 0 rlineto stroke } bind def /i { newpath moveto rlineto stroke } bind def /rs { dup stringwidth pop 0 exch sub 0 a show } bind def /cs { dup stringwidth pop 2 div 0 exch sub 0 a show } bind def /rshow { gsave currentpoint translate rotate 0 0 moveto show grestore } bind def /rrs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 0 exch sub 0 a show grestore } bind def /rcs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 2 div 0 exch sub 0 a show grestore } bind def 1440 569 569 e 36 569 569 d 18 710 569 d 36 850 569 d 18 991 569 d 36 1132 569 d 18 1272 569 d 36 1413 569 d 18 1553 569 d 36 1694 569 d 18 1834 569 d 36 1975 569 d n 568 558 m -4 -2 r -2 -3 r 560 546 l -4 v 562 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 568 558 m -3 -2 r -1 -1 r -1 -2 r 562 546 l -4 v 563 536 l 1 -3 r 1 -1 r 3 -1 r s n 571 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 817 553 m 1 -2 r -1 -1 r -1 1 r 2 v 1 2 r 1 1 r 4 2 r 5 h 4 -2 r 1 -1 r 2 -2 r -3 v -2 -2 r -3 -3 r -7 -3 r -2 -1 r -3 -2 r -1 -4 r -4 v s n 827 558 m 3 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -5 -3 r s n 816 533 m 1 2 r 3 h 6 -3 r 4 h 2 1 r 2 2 r s n 820 535 m 6 -4 r 5 h 1 1 r 2 3 r 2 v s n 849 558 m -4 -2 r -2 -3 r 841 546 l -4 v 843 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 849 558 m -2 -2 r -2 -1 r -1 -2 r 843 546 l -4 v 844 536 l 1 -3 r 2 -1 r 2 -1 r s n 852 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 875 558 m -4 -2 r -3 -3 r 867 546 l -4 v 868 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 875 558 m -3 -2 r -1 -1 r -1 -2 r 868 546 l -4 v 870 536 l 1 -3 r 1 -1 r 3 -1 r s n 877 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s -24 1108 555 d -27 1110 558 d n 1110 558 m 1096 539 l 20 h s 9 1105 531 e n 1130 558 m -4 -2 r -2 -3 r 1123 546 l -4 v 1124 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1130 558 m -2 -2 r -2 -1 r -1 -2 r 1124 546 l -4 v 1125 536 l 1 -3 r 2 -1 r 2 -1 r s n 1133 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1156 558 m -4 -2 r -2 -3 r 1148 546 l -4 v 1150 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1156 558 m -3 -2 r -1 -1 r -1 -2 r 1150 546 l -4 v 1151 536 l 1 -3 r 1 -1 r 3 -1 r s n 1159 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1393 554 m -1 -1 r 1 -2 r 2 2 r 1 v -2 2 r -2 2 r -4 h -4 -2 r -2 -2 r -2 -3 r -1 -5 r -7 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 1 v -1 4 r -3 2 r -4 2 r -1 h -4 -2 r -2 -2 r -2 -4 r s n 1387 558 m -3 -2 r -2 -2 r -1 -3 r -2 -5 r -7 v 2 -4 r 2 -3 r 3 -1 r s n 1388 531 m 3 1 r 2 3 r 2 4 r 1 v -2 4 r -2 2 r -3 2 r s n 1411 558 m -4 -2 r -2 -3 r 1404 546 l -4 v 1405 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1411 558 m -2 -2 r -2 -1 r -1 -2 r 1405 546 l -4 v 1406 536 l 1 -3 r 2 -1 r 2 -1 r s n 1414 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1437 558 m -4 -2 r -2 -3 r 1429 546 l -4 v 1431 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1437 558 m -3 -2 r -1 -1 r -1 -2 r 1431 546 l -4 v 1432 536 l 1 -3 r 1 -1 r 3 -1 r s n 1440 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1665 558 m -3 -2 r -2 -2 r -4 v 2 -2 r 3 -2 r 6 h 3 2 r 2 2 r 4 v -2 2 r -3 2 r c s n 1665 558 m -2 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r s n 1671 546 m 2 2 r 1 2 r 4 v -1 2 r -2 2 r s n 1665 546 m -3 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 3 -1 r 6 h 3 1 r 2 1 r 1 3 r 5 v -1 3 r -2 1 r -3 1 r s n 1665 546 m -2 -1 r -1 -1 r -2 -3 r -5 v 2 -3 r 1 -1 r 2 -1 r s n 1671 531 m 2 1 r 1 1 r 2 3 r 5 v -2 3 r -1 1 r -2 1 r s n 1692 558 m -3 -2 r -3 -3 r 1685 546 l -4 v 1686 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1692 558 m -2 -2 r -1 -1 r -2 -2 r 1686 546 l -4 v 1687 536 l 2 -3 r 1 -1 r 2 -1 r s n 1695 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1718 558 m -4 -2 r -2 -3 r 1710 546 l -4 v 1712 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 1718 558 m -2 -2 r -2 -1 r -1 -2 r 1712 546 l -4 v 1713 536 l 1 -3 r 2 -1 r 2 -1 r s n 1721 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1931 553 m 3 1 r 4 4 r -27 v s -25 1936 556 d 12 1931 531 e n 1961 558 m -4 -2 r -3 -3 r 1953 546 l -4 v 1954 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1961 558 m -3 -2 r -1 -1 r -1 -2 r 1954 546 l -4 v 1956 536 l 1 -3 r 1 -1 r 3 -1 r s n 1963 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1986 558 m -3 -2 r -3 -3 r 1979 546 l -4 v 1980 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1986 558 m -2 -2 r -1 -1 r -2 -2 r 1980 546 l -4 v 1981 536 l 2 -3 r 1 -1 r 2 -1 r s n 1989 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 2012 558 m -4 -2 r -2 -3 r 2004 546 l -4 v 2006 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 2012 558 m -2 -2 r -2 -1 r -1 -2 r 2006 546 l -4 v 2007 536 l 1 -3 r 2 -1 r 2 -1 r s n 2015 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s 1440 569 569 d 36 569 569 e 18 569 713 e 36 569 857 e 18 569 1001 e 36 569 1145 e 18 569 1289 e 36 569 1433 e 18 569 1577 e 36 569 1721 e 18 569 1865 e 36 569 2009 e n 528 585 m -4 -2 r -2 -3 r 521 573 l -4 v 522 563 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 528 585 m -2 -2 r -2 -1 r -1 -2 r 522 573 l -4 v 523 563 l 1 -3 r 2 -1 r 2 -1 r s n 531 558 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 393 867 m 2 -1 r -2 -1 r -1 1 r 1 v 1 3 r 2 1 r 3 2 r 6 h 3 -2 r 2 -1 r 1 -3 r -2 v -1 -3 r -4 -2 r 398 857 l -2 -1 r -3 -3 r -1 -4 r -3 v s n 404 873 m 2 -2 r 1 -1 r 2 -3 r -2 v -2 -3 r -3 -2 r 398 857 l s n 392 848 m 1 1 r 3 h 6 -2 r 4 h 3 1 r 1 1 r s n 396 849 m 6 -3 r 5 h 2 1 r 1 2 r 3 v s n 425 873 m -3 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 425 873 m -2 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -1 r s n 428 846 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 2 r s n 451 873 m -4 -2 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 2 r c s n 451 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 454 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 477 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 477 873 m -3 -2 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -1 r s n 479 846 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 2 r s n 503 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 503 873 m -3 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -1 r s n 505 846 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 2 r s n 528 873 m -4 -2 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 528 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 531 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s -25 404 1158 d -27 405 1160 d n 405 1160 m 391 1141 l 20 h s 9 400 1133 e n 425 1160 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1160 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -2 r s n 428 1133 m 3 2 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1160 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 454 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1160 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 479 1133 m 3 2 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1160 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -2 r s n 505 1133 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1160 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 531 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 407 1444 m -1 -1 r 1 -1 r 2 1 r 1 v -2 3 r -2 1 r -4 h 397 1447 l -2 -3 r -2 -2 r -1 -5 r -8 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 1 v -1 4 r -3 3 r -4 1 r -1 h 397 1437 l -2 -3 r -2 -4 r s n 401 1448 m -3 -1 r -2 -3 r -1 -2 r -2 -5 r -8 v 2 -4 r 2 -2 r 3 -2 r s n 402 1421 m 3 2 r 2 2 r 2 4 r 1 v -2 4 r -2 3 r -3 1 r s n 425 1448 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 3 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1448 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 2 -2 r s n 428 1421 m 3 2 r 1 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1448 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 454 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1448 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -1 r 3 -2 r s n 479 1421 m 3 2 r 1 1 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1448 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 3 -2 r s n 505 1421 m 3 2 r 1 1 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1448 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 531 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 398 1736 m -3 -1 r -2 -3 r -4 v 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 4 v -2 3 r -3 1 r c s n 398 1736 m -2 -1 r -1 -3 r -4 v 1 -2 r 2 -1 r s n 404 1725 m 2 1 r 1 2 r 4 v -1 3 r -2 1 r s n 398 1725 m -3 -2 r -2 -1 r -1 -3 r -5 v 1 -2 r 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 1 2 r 5 v -1 3 r -2 1 r -3 2 r s n 398 1725 m -2 -2 r -1 -1 r -2 -3 r -5 v 2 -2 r 1 -2 r 2 -1 r s n 404 1709 m 2 1 r 1 2 r 2 2 r 5 v -2 3 r -1 1 r -2 2 r s n 425 1736 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1736 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1709 m 3 1 r 1 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1736 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1736 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1709 m 3 1 r 1 2 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1736 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1709 m 3 1 r 1 2 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1736 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 370 2019 m 3 1 r 4 4 r -27 v s -26 375 2023 d 12 370 1997 e n 400 2024 m 396 2023 l -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 400 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 402 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 425 2024 m -3 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 425 2024 m -2 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1997 m 3 1 r 1 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -1 2 r -3 1 r s n 451 2024 m -4 -1 r -2 -4 r -2 -7 r -3 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 3 v -1 7 r -2 4 r -4 1 r c s n 451 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s n 477 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 477 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1997 m 3 1 r 1 2 r 2 2 r 1 7 r 3 v -1 7 r -2 2 r -1 2 r -3 1 r s n 503 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 503 2024 m -3 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 528 2024 m -4 -1 r -2 -4 r -1 -7 r -3 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 528 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s 1440 569 2009 e -36 569 2009 d -18 710 2009 d -36 850 2009 d -18 991 2009 d -36 1132 2009 d -18 1272 2009 d -36 1413 2009 d -18 1553 2009 d -36 1694 2009 d -18 1834 2009 d -36 1975 2009 d 1440 2009 569 d -36 2009 569 e -18 2009 713 e -36 2009 857 e -18 2009 1001 e -36 2009 1145 e -18 2009 1289 e -36 2009 1433 e -18 2009 1577 e -36 2009 1721 e -18 2009 1865 e -36 2009 2009 e -27 1139 510 d -27 1141 510 d 6 1135 510 e 9 1135 483 e n 1152 494 m 16 h 2 v -2 3 r -1 1 r -3 1 r -3 h -4 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r s n 1166 494 m 3 v -1 3 r s n 1159 501 m -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r s -18 1178 501 d -18 1179 501 d n 1179 497 m 3 3 r 4 1 r 2 h 4 -1 r 1 -3 r -14 v s n 1188 501 m 3 -1 r 1 -3 r -14 v s 5 1174 501 e 9 1174 483 e 9 1188 483 e n 1210 501 m -3 -1 r -1 -1 r -1 -3 r -2 v 1 -3 r 1 -1 r 3 -2 r 2 h 3 2 r 1 1 r 2 3 r 2 v -2 3 r -1 1 r -3 1 r c s n 1207 500 m -1 -3 r -5 v 1 -2 r s n 1215 490 m 1 2 r 5 v -1 3 r s n 1216 499 m 2 1 r 2 1 r -1 v -2 h s n 1206 491 m -1 -1 r -2 -3 r -1 v 2 -3 r 4 -1 r 6 h 4 -1 r 1 -2 r s n 1203 486 m 2 -1 r 4 -2 r 6 h 4 -1 r 1 -3 r -1 v -1 -2 r -4 -2 r -8 h -4 2 r -1 2 r 1 v 1 3 r 4 1 r s n 1230 510 m -22 v 2 -3 r 2 -2 r 3 h 2 2 r 2 2 r s n 1232 510 m -22 v 1 -3 r 1 -2 r s 10 1227 501 e -27 1250 510 d -27 1251 510 d n 1251 497 m 3 3 r 3 1 r 3 h 4 -1 r 1 -3 r -14 v s n 1260 501 m 3 -1 r 1 -3 r -14 v s 5 1246 510 e 9 1246 483 e 9 1260 483 e n 1306 515 m -2 -2 r -3 -4 r -2 -5 r 1297 497 l -5 v 1299 486 l 1301 481 l 3 -4 r 2 -3 r s n 1304 513 m -3 -5 r -1 -4 r 1299 497 l -5 v 1300 486 l 1 -4 r 1304 477 l s -27 1317 510 d -27 1318 510 d n 1318 497 m 2 3 r 3 1 r 3 h 3 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -3 -2 r -3 h -3 2 r -2 2 r s n 1326 501 m 2 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -2 -2 r s 5 1313 510 e 8 -18 1342 501 i 6 -15 1344 501 i n 1358 501 m 1350 483 l 1347 478 l -2 -2 r -3 -2 r -1 h -1 2 r 1 1 r 1 -1 r s 7 1340 501 e 7 1353 501 e n 1368 510 m -22 v 1 -3 r 3 -2 r 2 h 3 2 r 1 2 r s n 1369 510 m -22 v 2 -3 r 1 -2 r s 10 1364 501 e n 1386 494 m 15 h 2 v -1 3 r -1 1 r -3 1 r -4 h -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r 3 h 4 2 r 2 2 r s n 1400 494 m 3 v -1 3 r s n 1392 501 m -2 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 2 -2 r s n 1422 499 m 1 2 r -5 v -1 3 r -1 1 r -3 1 r -5 h -3 -1 r -1 -1 r -3 v 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -2 r s n 1409 497 m 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -1 r -4 v -1 -1 r -3 -2 r -5 h -2 2 r -2 1 r -1 2 r -5 v 1 3 r s n 1431 515 m 3 -2 r 2 -4 r 3 -5 r 1440 497 l -5 v 1439 486 l 1436 481 l -2 -4 r -3 -3 r s n 1434 513 m 2 -5 r 1 -4 r 1439 497 l -5 v 1437 486 l -1 -4 r 1434 477 l s n 870 2060 m 2 -5 r 10 v -2 -5 r -3 3 r -6 2 r -3 h -5 -2 r -4 -3 r -1 -3 r -2 -6 r -8 v 2 -5 r 1 -4 r 4 -3 r 5 -2 r 3 h 6 2 r 3 3 r 2 4 r s n 858 2065 m -3 -2 r -4 -3 r -2 -3 r -1 -6 r -8 v 1 -5 r 2 -4 r 4 -3 r 3 -2 r s n 892 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 892 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 896 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 920 2053 d -24 921 2053 d n 921 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 933 2053 m 4 -2 r 2 -3 r -19 v s n 940 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 952 2053 m 4 -2 r 1 -3 r -19 v s 6 915 2053 e 12 915 2029 e 12 933 2029 e 12 952 2029 e -24 976 2053 d -24 978 2053 d n 978 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 990 2053 m 3 -2 r 2 -3 r -19 v s n 997 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1009 2053 m 3 -2 r 2 -3 r -19 v s 7 971 2053 e 12 971 2029 e 12 990 2029 e 12 1009 2029 e n 1033 2053 m -19 v 2 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1035 2053 m -19 v 1 -3 r 4 -2 r s -24 1052 2053 d -24 1053 2053 d 7 1028 2053 e 6 1047 2053 e 7 1052 2029 e -24 1071 2053 d -24 1072 2053 d n 1072 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1084 2053 m 4 -2 r 1 -3 r -19 v s 7 1065 2053 e 12 1065 2029 e 12 1084 2029 e n 1108 2065 m -2 -2 r 2 -1 r 2 1 r c s -24 1108 2053 d -24 1110 2053 d 7 1103 2053 e 12 1103 2029 e n 1144 2048 m -2 -2 r 2 -1 r 2 1 r 2 v -4 3 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1134 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1160 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1175 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1175 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 4 2 r 3 3 r s n 1163 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s n 1194 2065 m -29 v 2 -5 r 3 -2 r 3 h 4 2 r 2 3 r s n 1196 2065 m -29 v 1 -5 r 2 -2 r s 13 1189 2053 e n 1220 2065 m -2 -2 r 2 -1 r 1 1 r c s -24 1220 2053 d -24 1221 2053 d 7 1214 2053 e 12 1214 2029 e n 1245 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1245 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1249 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1273 2053 d -24 1274 2053 d n 1274 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1286 2053 m 4 -2 r 2 -3 r -19 v s 6 1268 2053 e 12 1268 2029 e 12 1286 2029 e n 1324 2050 m 2 3 r -7 v -2 4 r -2 1 r -3 2 r -7 h -3 -2 r -2 -1 r -4 v 2 -1 r 3 -2 r 9 -4 r 3 -1 r 2 -2 r s n 1307 2048 m 2 -2 r 3 -1 r 9 -4 r 3 -2 r 2 -1 r -5 v -2 -2 r -3 -2 r -7 h -4 2 r -1 2 r -2 3 r -7 v 2 4 r s -36 1367 2065 d -36 1369 2065 d n 1362 2065 m 20 h 5 -2 r 2 -1 r 2 -4 r -5 v -2 -3 r -2 -2 r -5 -2 r -13 h s n 1382 2065 m 4 -2 r 1 -1 r 2 -4 r -5 v -2 -3 r -1 -2 r -4 -2 r s 12 1362 2029 e n 1403 2043 m 20 h 3 v -1 4 r -2 1 r -3 2 r -6 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1422 2043 m 5 v -2 3 r s n 1411 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s -24 1437 2053 d -24 1439 2053 d n 1439 2043 m 2 5 r 3 3 r 3 2 r 6 h 1 -2 r -1 v -1 -2 r -2 2 r 2 1 r s 7 1432 2053 e 12 1432 2029 e n 1475 2063 m -2 -1 r 2 -2 r 2 2 r 1 v -2 2 r -4 h -3 -2 r -2 -3 r -31 v s n 1471 2065 m -1 -2 r -2 -3 r -31 v s 14 1461 2053 e 12 1461 2029 e n 1495 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1495 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s n 1499 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1523 2053 d -24 1525 2053 d n 1525 2043 m 1 5 r 4 3 r 3 2 r 5 h 2 -2 r -1 v -2 -2 r -1 2 r 1 1 r s 7 1518 2053 e 12 1518 2029 e -24 1552 2053 d -24 1554 2053 d n 1554 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1566 2053 m 3 -2 r 2 -3 r -19 v s n 1573 2048 m 3 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1584 2053 m 4 -2 r 2 -3 r -19 v s 7 1547 2053 e 12 1547 2029 e 12 1566 2029 e 12 1584 2029 e n 1608 2050 m -2 v -1 h 2 v 1 1 r 4 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1624 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1624 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 3 2 r 4 3 r s n 1612 2043 m -4 -2 r -1 -3 r -4 v 1 -3 r 4 -2 r s -24 1643 2053 d -24 1644 2053 d n 1644 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1656 2053 m 4 -2 r 2 -3 r -19 v s 6 1638 2053 e 12 1638 2029 e 12 1656 2029 e n 1698 2048 m -2 -2 r 2 -1 r 1 1 r 2 v -3 3 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1687 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1711 2043 m 21 h 3 v -2 4 r -2 1 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1730 2043 m 5 v -2 3 r s n 1720 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s 27 320 1110 e 27 320 1112 e n 320 1106 m 16 v 1 4 r 2 1 r 2 1 r 3 h 2 -1 r 2 -1 r 1 -4 r -10 v s n 320 1122 m 1 2 r 2 2 r 2 1 r 3 h 2 -1 r 2 -2 r 1 -2 r s 9 347 1106 d n 333 1118 m 1 3 r 1 1 r 9 4 r 2 1 r 1 v -2 2 r s n 334 1121 m 3 1 r 9 2 r 1 2 r 2 v -3 2 r -1 h s n 332 1139 m 1 h -2 v -1 h -2 2 r -1 2 r 5 v 1 3 r 2 1 r 2 1 r 9 h 3 2 r 1 1 r s n 332 1150 m 11 h 3 1 r 1 3 r 1 v s n 334 1150 m 1 -1 r 2 -8 r 1 -4 r 3 -1 r 2 h 3 1 r 1 4 r 4 v -1 3 r -3 2 r s n 337 1141 m 1 -2 r 3 -2 r 2 h 3 2 r 1 2 r s n 320 1164 m 22 h 4 2 r 1 2 r 3 v -1 2 r -3 2 r s n 320 1166 m 22 h 4 1 r 1 1 r s 11 329 1160 d n 337 1182 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1196 m -4 h -3 -1 r s n 329 1189 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 315 1236 m 2 -2 r 4 -3 r 5 -2 r 7 -2 r 5 h 6 2 r 6 2 r 3 3 r 3 2 r s n 317 1234 m 6 -3 r 3 -1 r 7 -1 r 5 h 6 1 r 4 1 r 5 3 r s 27 320 1247 e 27 320 1248 e n 333 1248 m -3 2 r -1 3 r 3 v 1 3 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -3 r -3 v -1 -3 r -3 -2 r s n 329 1256 m 1 2 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -2 r s 5 320 1243 d 18 8 329 1272 i 15 6 329 1274 i n 329 1288 m 18 -8 r 5 -3 r 3 -2 r 1 -3 r -1 v -1 -1 r -2 1 r 2 1 r s 7 329 1270 d 7 329 1283 d n 320 1298 m 22 h 4 1 r 1 3 r 2 v -1 3 r -3 1 r s n 320 1299 m 22 h 4 2 r 1 1 r s 10 329 1294 d n 337 1316 m 15 v -3 h -2 -1 r -2 -1 r -1 -3 r -4 v 1 -3 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 3 r 3 v -1 4 r -3 2 r s n 337 1330 m -4 h -3 -1 r s n 329 1322 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 332 1352 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1339 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s 356 1360 315 1383 b n 332 1402 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1389 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s n 337 1412 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1426 m -4 h -3 -1 r s n 329 1419 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 333 1451 m 1 -1 r 1 1 r -1 1 r -1 h -3 -2 r -1 -3 r -4 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 3 v -1 4 r -3 2 r s n 329 1443 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 315 1460 m 2 2 r 4 3 r 5 3 r 7 1 r 5 h 6 -1 r 6 -3 r 3 -3 r 3 -2 r s n 317 1462 m 6 3 r 3 1 r 7 2 r 5 h 6 -2 r 4 -1 r 5 -3 r s 657 974 614 806 b 20 -7 647 983 i 21 -7 647 984 i n 657 981 m -1 2 r 3 v 2 v 2 2 r 3 1 r 3 h 2 h 3 -2 r 1 -3 r -3 v -1 -2 r -2 -2 r -2 -1 r s n 656 988 m 2 1 r 3 1 r 3 h 2 h 2 -2 r 1 -3 r 1 -2 r s 1 4 646 980 i 21 -8 654 1004 i 20 -7 655 1004 i 2 3 653 1001 i 2 7 674 993 i n 666 1016 m -4 v 2 -2 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 2 r -2 2 r -2 1 r -3 h -3 -1 r -2 -3 r c s n 666 1016 m 1 -3 r 1 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s n 681 1013 m -1 2 r -1 3 r -2 2 r -2 h -4 h -2 -1 r -2 -2 r s n 678 1040 m 1 -2 r 1 1 r 1 v -1 1 r -3 -2 r -2 -1 r -1 -3 r -3 v 2 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 3 r s n 673 1035 m 1 -2 r 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s 20 -8 672 1052 i 21 -8 672 1053 i 6 -13 683 1060 i 9 3 687 1053 i 10 3 686 1052 i 1 4 671 1049 i 3 6 681 1057 i 3 6 691 1042 i 697 1058 695 1052 b n 679 1072 m 1 -1 r 1 h 2 v c s 14 -5 686 1070 i 14 -5 686 1071 i 1 4 685 1067 i 701 1069 699 1062 b 13 -5 690 1080 i 14 -5 690 1081 i n 693 1080 m -1 3 r 3 v 1 2 r 2 3 r 2 h 11 -4 r s n 693 1088 m 1 2 r 3 h 10 -4 r s 1 4 689 1077 i 3 7 702 1072 i 709 1090 706 1083 b n 699 1104 m -2 v 1 -1 r 1 -2 r 2 -1 r 2 1 r 2 h 1 2 r 1 2 r 2 v -1 1 r -1 2 r -2 1 r -3 h -1 -1 r -2 -2 r c s n 699 1102 m 1 -1 r 4 -2 r 3 h s n 709 1105 m -2 2 r -4 1 r -2 h s n 702 1109 m 1 v -1 2 r 1 h -2 v s n 705 1099 m 1 -2 r 2 -1 r -1 v 3 1 r 2 2 r 2 5 r 2 3 r 1 h s n 708 1095 m 2 1 r 2 3 r 2 4 r 2 3 r 2 h 1 h 1 -2 r -3 v -2 -6 r -2 -2 r -2 -1 r -1 1 r -2 1 r 4 v s 41 -155 708 1114 i 7 10 749 959 i 18 -12 755 991 i 18 -12 756 992 i n 764 986 m 3 v 2 v 1 2 r 3 2 r 3 h 3 -1 r 1 -1 r 2 -2 r 1 -3 r -1 -3 r -1 -2 r -2 -1 r -3 -1 r s n 765 993 m 2 1 r 3 h 3 h 2 -2 r 2 -2 r -3 v -2 v s 756 992 754 988 b 18 -12 767 1009 i 18 -12 768 1010 i 2 4 766 1006 i 787 1000 783 994 b n 782 1018 m -1 -4 r 1 -2 r 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 2 r 1 3 r -1 3 r -2 3 r -1 1 r -4 1 r -2 -1 r -3 -2 r c s n 782 1018 m -3 v -3 v 2 -2 r 2 -1 r 3 -1 r 3 h 2 1 r s n 795 1011 m 2 v 3 v -2 3 r -2 1 r -3 1 r -3 -1 r -2 -1 r s n 799 1038 m 1 -1 r 1 h 1 v -1 1 r -3 -1 r -2 -1 r -2 -2 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 1 r 1 3 r 3 v s n 793 1035 m -3 v 1 -3 r 2 -2 r 1 -1 r 3 -1 r 3 h 2 2 r s 18 -12 796 1051 i 17 -12 797 1052 i 3 -15 808 1057 i 10 1 811 1049 i 10 1 810 1048 i 3 3 794 1049 i 3 5 807 1054 i 4 6 812 1037 i 822 1051 818 1046 b n 808 1069 m -1 v 2 h -1 2 r c s 12 -8 814 1065 i 12 -8 815 1066 i 3 3 812 1063 i 828 1061 824 1055 b 12 -8 820 1075 i 12 -9 821 1076 i n 823 1074 m 3 v 1 3 r 1 1 r 2 2 r 3 h 9 -6 r s n 825 1081 m 2 2 r 2 -1 r 10 -6 r s 2 4 819 1072 i 4 6 831 1064 i 841 1079 837 1073 b n 835 1096 m -1 -2 r 1 -2 r 1 -2 r 1 -1 r 3 h 1 h 2 1 r 1 2 r 1 2 r -1 2 r -1 2 r -1 1 r -3 h -1 h -2 -1 r c s n 834 1094 m 2 -2 r 3 -3 r 2 h s n 845 1094 m -1 2 r -4 3 r -2 h s n 839 1099 m 2 v 2 v 1 -1 r -1 -1 r s n 840 1089 m -2 v 1 -2 r 1 h 2 h 3 2 r 3 4 r 2 2 r 2 h s n 842 1085 m 1 h 3 2 r 3 4 r 2 2 r 3 h 1 -1 r 1 -2 r -1 -3 r -4 -5 r -2 -2 r -2 1 r -1 h -1 2 r 3 v s 32 42 852 1113 i 45 52 884 1155 i 36 36 929 1207 i 16 -15 968 1264 i 15 -15 969 1265 i n 976 1258 m 3 v 1 2 r 2 1 r 2 2 r 3 h 3 -2 r 2 -1 r 1 -3 r -3 v -1 -3 r -2 -1 r -2 -1 r -3 h s n 979 1264 m 2 1 r 3 h 3 -1 r 1 -2 r 1 -3 r -3 v -2 v s 969 1265 966 1262 b 16 -15 983 1280 i 16 -15 984 1280 i 3 3 981 1277 i 1002 1267 997 1262 b n 999 1286 m -1 -3 r -3 v 1 -3 r 2 -2 r 3 -1 r 3 h 3 2 r 1 1 r 2 3 r -1 3 r -1 3 r -1 1 r -3 2 r -3 h -3 -2 r c s n 999 1286 m -3 v -2 v 1 -3 r 2 -2 r 3 -1 r 2 h 3 1 r s n 1011 1277 m 1 2 r 3 v -2 3 r -1 1 r -3 2 r -3 h -2 -1 r s n 1020 1303 m -2 v 2 h 2 v -1 h -3 h -2 h -2 -3 r -2 -3 r -2 v 2 -3 r 1 -2 r 3 -1 r 3 h 3 1 r 1 2 r 2 3 r 3 v s n 1014 1300 m -1 -2 r -3 v 2 -3 r 1 -1 r 3 -2 r 3 1 r 2 h s 16 -15 1019 1316 i 16 -15 1020 1317 i 1 -14 1032 1319 i 10 -1 1033 1311 i 10 -2 1033 1311 i 3 3 1017 1314 i 5 5 1030 1317 i 5 5 1033 1299 i 1045 1312 1041 1307 b n 1035 1332 m -2 v 1 h 2 v c s 10 -10 1040 1327 i 11 -11 1040 1328 i 3 3 1037 1325 i 1053 1320 1048 1314 b 10 -10 1048 1335 i 11 -10 1048 1336 i n 1050 1333 m 3 v 2 3 r 1 2 r 3 1 r 2 h 8 -8 r s n 1053 1341 m 2 h 3 h 8 -8 r s 3 3 1045 1333 i 5 5 1056 1323 i 1069 1336 1064 1331 b n 1066 1353 m -1 -2 r -1 v 1 -3 r 1 -1 r 2 -1 r 2 h 2 1 r 1 1 r 1 3 r 1 v -1 2 r -1 2 r -2 h -2 h -2 h c s n 1065 1351 m 1 -2 r 3 -3 r 2 -1 r s n 1075 1350 m -1 2 r -3 3 r -2 h s n 1071 1355 m 2 v 2 v 1 -1 r -1 -1 r s n 1069 1345 m -1 v 1 -2 r 1 -1 r 2 -1 r 3 2 r 3 3 r 3 2 r 2 h s n 1071 1341 m 1 h 3 1 r 4 4 r 3 2 r 2 -1 r 1 -1 r -2 v -1 -3 r -4 -4 r -3 -2 r -3 1 r 1 v -1 2 r 1 3 r s 23 22 1086 1366 i 45 37 1109 1388 i 45 34 1154 1425 i 21 15 1199 1459 i 11 -19 1229 1494 i 11 -19 1230 1494 i n 1235 1485 m 1 3 r 1 2 r 2 1 r 3 1 r 3 -1 r 3 -2 r 1 -2 r -3 v -1 -3 r -2 -3 r -1 -1 r -3 h -3 1 r s n 1239 1491 m 3 h 2 -1 r 3 -2 r 1 -2 r -3 v -3 v -2 -2 r s 1230 1494 1227 1492 b 11 -19 1248 1504 i 11 -19 1249 1505 i 4 2 1245 1503 i 1262 1488 1256 1484 b n 1265 1506 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 1 r 2 1 r 2 3 r 1 2 r -1 4 r -1 1 r -2 2 r -3 1 r -3 h c s n 1265 1506 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s n 1274 1494 m 1 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 1 r -2 h s n 1290 1516 m -1 v 1 -1 r 2 v 1 v -3 h -2 h -3 -1 r -2 -3 r -1 -2 r 1 -4 r 1 -1 r 2 -3 r 3 h 3 h 2 1 r 2 3 r 1 2 r s n 1283 1516 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 11 -19 1293 1530 i 10 -19 1294 1530 i -4 -14 1306 1529 i 9 -4 1305 1521 i 9 -5 1304 1521 i 4 2 1290 1528 i 5 3 1304 1527 i 6 4 1301 1509 i 1316 1518 1311 1515 b n 1312 1540 m -1 -1 r 2 -1 r 2 v c s 7 -13 1315 1534 i 7 -13 1316 1535 i 3 3 1313 1532 i 1326 1524 1320 1520 b 7 -13 1325 1540 i 7 -12 1326 1540 i n 1327 1537 m 1 3 r 2 3 r 2 1 r 3 h 2 -1 r 6 -10 r s n 1332 1544 m 2 h 2 -2 r 6 -9 r s 4 2 1322 1538 i 7 3 1329 1526 i 1346 1535 1339 1531 b n 1347 1552 m -1 -2 r -1 v -2 v 1 -2 r 2 -1 r 1 -1 r 2 h 2 1 r 2 2 r 2 v 2 v -1 2 r -2 1 r -2 h -2 h c s n 1346 1550 m -2 v 2 -4 r 2 -1 r s n 1356 1546 m -1 3 r -2 3 r -2 1 r s n 1353 1553 m 1 v 2 2 r -1 v -2 -1 r s n 1349 1544 m -1 -2 r 1 -2 r -1 v 2 -1 r 3 h 5 3 r 3 1 r 1 -1 r s n 1349 1539 m 1 h 4 h 4 3 r 3 h 2 -1 r 1 -1 r -2 v -2 -2 r -6 -4 r -3 h -2 1 r 1 v 2 v 2 3 r s 9 4 1370 1559 i 45 31 1379 1563 i 45 26 1424 1594 i 45 23 1469 1620 i 6 2 1514 1643 i 7 -20 1532 1663 i 7 -21 1533 1664 i n 1536 1654 m 1 3 r 2 1 r 2 1 r 3 h 3 -1 r 2 -3 r 1 -1 r -4 v -2 -2 r -2 -2 r -2 -1 r -2 h -3 1 r s n 1541 1659 m 2 h 3 -1 r 2 -3 r 1 -2 r -3 v -1 -3 r -2 -1 r s 1533 1664 1529 1662 b 7 -20 1552 1671 i 7 -20 1553 1671 i 4 1 1549 1670 i 1563 1652 1557 1650 b n 1569 1670 m -3 -2 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r 3 v 2 v -2 2 r -3 2 r -3 -1 r c s n 1569 1670 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 2 h s n 1576 1657 m 1 2 r 1 2 r 4 v 2 v -2 2 r -3 1 r -2 h s n 1595 1676 m -1 -1 r 1 -1 r 1 2 r 1 v -3 1 r -2 h -3 -1 r -3 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 3 1 r 2 h 2 2 r 2 3 r s n 1588 1677 m -2 -2 r -1 -2 r -4 v 1 -1 r 2 -3 r 3 -1 r 2 h s 7 -20 1600 1689 i 7 -21 1601 1690 i -6 -14 1613 1687 i 9 -5 1610 1679 i 9 -6 1609 1679 i 4 2 1597 1688 i 6 3 1610 1685 i 6 3 1605 1668 i 1621 1674 1615 1672 b n 1620 1697 m -1 -1 r 2 -1 r 1 v c s 6 -13 1622 1690 i 6 -14 1623 1691 i 3 2 1620 1689 i 1631 1678 1625 1676 b 5 -13 1633 1694 i 5 -14 1634 1695 i n 1635 1692 m 1 2 r 3 2 r 2 1 r 3 h 2 -1 r 4 -11 r s n 1641 1697 m 2 h 2 -2 r 4 -10 r s 4 2 1630 1693 i 7 2 1635 1680 i 1653 1686 1646 1684 b n 1657 1703 m -2 -1 r -2 v -2 v -2 v 2 -1 r 1 -1 r 3 h 2 1 r 1 1 r 1 2 r 2 v -1 2 r -1 1 r -2 1 r -2 h c s n 1655 1702 m -3 v 2 -3 r 1 -2 r s n 1664 1696 m 3 v -1 3 r -2 2 r s n 1663 1703 m 2 v 2 1 r -1 v -2 h s n 1657 1695 m -1 -2 r -2 v 1 -1 r 1 -1 r 4 h 4 2 r 4 h 1 -1 r s n 1657 1690 m 1 h 3 h 5 1 r 3 1 r 2 -2 r -1 v -2 v -2 -2 r -6 -2 r -3 -1 r -2 2 r 1 v 2 v 2 2 r s 13 4 1681 1707 i 45 21 1694 1711 i 45 18 1739 1732 i 45 17 1784 1750 i 12 3 1829 1767 i 5 -21 1855 1787 i 5 -20 1856 1787 i n 1858 1777 m 2 3 r 1 2 r 2 h 4 h 2 -1 r 2 -3 r 1 -2 r -1 -3 r -1 -3 r -3 -2 r -2 h -2 h -2 2 r s n 1863 1782 m 3 h 2 -2 r 2 -2 r 1 -2 r -1 -4 r -1 -2 r -2 -2 r s 1856 1787 1852 1786 b 6 -21 1875 1793 i 6 -20 1876 1793 i 4 1 1872 1792 i 1885 1773 1878 1771 b n 1892 1790 m -3 -2 r -1 -2 r -3 v -2 v 2 -3 r 3 -1 r 3 -1 r 2 1 r 3 2 r 1 2 r 4 v 2 v -2 2 r -3 2 r -3 h c s n 1892 1790 m -2 -1 r -1 -3 r -3 v -2 v 2 -3 r 3 -1 r 2 -1 r s n 1898 1777 m 2 2 r 1 2 r 3 v 2 v -2 3 r -3 1 r -2 1 r s n 1919 1795 m -1 -2 r 1 h 1 1 r 1 v -3 1 r -2 1 r -3 -1 r -3 -2 r -1 -2 r -4 v -2 v 2 -2 r 3 -2 r 3 h 2 1 r 2 2 r 2 2 r s n 1912 1796 m -2 -2 r -1 -2 r -3 v -2 v 2 -3 r 3 -1 r 2 -1 r s 6 -21 1925 1807 i 6 -20 1926 1807 i -7 -12 1938 1803 i 9 -6 1934 1796 i 9 -7 1933 1796 i 4 1 1922 1806 i 6 2 1935 1802 i 7 2 1928 1785 i 1944 1790 1939 1789 b n 1945 1813 m -1 v 1 -1 r 1 1 r c s 4 -14 1947 1806 i 4 -13 1948 1806 i 3 1 1945 1805 i 1955 1793 1948 1791 b 4 -14 1958 1809 i 4 -13 1959 1809 i n 1960 1807 m 2 2 r 2 2 r 2 h 4 h 1 -1 r 3 -11 r s n 1966 1811 m 3 h 1 -2 r 3 -11 r s 4 1 1955 1808 i 7 1 1959 1795 i 1977 1800 1970 1798 b n 1983 1816 m -2 -1 r -2 v -1 -2 r 1 -2 r 1 -1 r 2 -1 r 2 -1 r 2 1 r 2 1 r 2 v 1 2 r -1 2 r -1 2 r -2 h -2 1 r c s n 1981 1815 m -3 v 1 -4 r 2 -1 r s n 1990 1808 m 3 v -1 4 r -2 1 r s n 1989 1816 m 1 v 2 2 r -1 v -2 -1 r s n 1982 1808 m -2 v -1 -2 r 1 -1 r 1 -2 r 3 h 5 2 r 3 -1 r 2 h s n 1982 1803 m 1 -1 r 3 h 5 2 r 3 -1 r 2 -1 r -1 v -3 v -3 -1 r -6 -2 r -3 h -2 2 r 1 v 2 v 3 2 r s 2 2007 1818 e 45 144 614 748 i 8 21 659 892 i 8 -12 634 897 i 8 -12 635 897 i n 636 895 m 1 2 r 2 3 r 2 1 r 3 1 r 2 -1 r 652 892 l s n 641 901 m 2 1 r 2 -2 r 651 891 l s 4 2 631 895 i 6 4 639 883 i 655 893 649 889 b n 656 912 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 2 1 r 2 3 r 2 v -1 3 r -1 2 r -2 2 r -3 1 r -3 -1 r c s n 656 912 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 2 1 r s n 666 901 m 1 2 r 3 v -1 3 r -1 2 r -2 2 r -3 h -2 h s 8 -12 670 921 i 8 -12 670 922 i n 672 919 m 1 3 r 2 2 r 1 2 r 3 h 2 -1 r 7 -9 r s n 676 926 m 3 h 2 -1 r 687 915 l s 3 3 667 919 i 6 4 675 907 i 690 918 684 914 b 12 -18 684 940 i 12 -18 685 940 i n 691 932 m 3 v 1 2 r 2 1 r 3 1 r 3 -1 r 3 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -2 r -1 -1 r -3 -1 r -3 1 r s n 694 938 m 2 h 3 h 3 -2 r 1 -2 r 1 -3 r -1 -3 r -1 -2 r s 685 940 682 938 b 12 -18 702 952 i 12 -17 703 952 i 4 2 699 950 i 718 936 712 932 b n 719 955 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 2 1 r 2 2 r 3 v -1 3 r -1 2 r -3 2 r -2 1 r -4 -1 r c s n 719 955 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 2 1 r s n 729 944 m 1 2 r 3 v -1 3 r -1 2 r -2 1 r -3 1 r -3 h s n 743 967 m -1 -1 r 2 -1 r 2 v -1 1 r -2 h -3 h -2 -2 r -2 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 -1 r 3 1 r 2 1 r 2 3 r 3 v s n 736 966 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 3 h 2 h s 13 -18 744 981 i 12 -17 745 981 i -3 -14 758 981 i 10 -3 757 973 i 10 -4 756 973 i 3 2 742 979 i 5 4 755 979 i 6 4 754 961 i 768 971 763 968 b 19 -42 730 956 i 45 68 749 914 i 28 36 794 982 i 11 -9 825 1030 i 11 -9 826 1031 i n 828 1029 m 3 v 1 3 r 1 2 r 3 1 r 2 h 9 -8 r s n 830 1037 m 2 1 r 3 -1 r 8 -7 r s 3 3 823 1028 i 5 5 834 1019 i 846 1033 841 1027 b n 842 1051 m -1 -3 r -3 v 2 -3 r 1 -1 r 4 -1 r 2 h 3 1 r 2 2 r 1 3 r -1 3 r -1 3 r -2 1 r -3 1 r -3 h -3 -2 r c s n 842 1051 m -2 v -3 v 2 -3 r 1 -1 r 3 -2 r 3 1 r 2 h s n 855 1043 m 2 v 3 v -2 3 r -1 1 r -3 1 r -3 h -3 -1 r s 11 -9 853 1063 i 11 -9 853 1064 i n 856 1062 m 3 v 1 3 r 1 2 r 3 1 r 2 h 9 -7 r s n 858 1070 m 2 1 r 2 -1 r 9 -7 r s 2 3 851 1061 i 4 5 862 1052 i 874 1066 869 1060 b 16 -14 862 1085 i 17 -14 862 1086 i n 870 1079 m 3 v 1 3 r 1 1 r 3 2 r 3 h 3 -1 r 1 -1 r 2 -3 r -3 v -1 -3 r -1 -2 r -2 -1 r -3 h s n 872 1086 m 2 1 r 3 h 3 -1 r 2 -1 r 1 -3 r 1 -3 r -1 -2 r s 862 1086 860 1083 b 16 -14 876 1102 i 17 -14 876 1103 i 2 4 874 1099 i 895 1091 890 1086 b n 891 1109 m -1 -3 r -3 v 2 -3 r 1 -1 r 4 -1 r 2 h 3 2 r 2 1 r 1 3 r -1 3 r -1 3 r -2 1 r -3 1 r -3 h -2 -1 r c s n 891 1109 m -2 v -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 2 1 r s n 904 1101 m 2 v 3 v -2 3 r -1 1 r -3 2 r -3 -1 r -2 h s n 911 1127 m -1 v 1 h 2 v -1 h -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 2 -3 r 2 -1 r 3 -2 r 3 1 r 2 1 r 2 2 r 1 3 r 3 v s n 904 1125 m -3 v -2 v 2 -3 r 1 -2 r 3 -1 r 3 h 2 1 r s 16 -14 909 1141 i 16 -14 910 1142 i 1 -14 922 1145 i 10 923 1137 e 11 922 1136 e 3 3 907 1139 i 4 4 920 1143 i 5 5 923 1125 i 935 1138 931 1133 b 41 39 933 1150 i 45 48 974 1189 i 36 35 1019 1237 i 9 -11 1061 1283 i 10 -11 1061 1284 i n 1063 1282 m 1 2 r 1 3 r 2 2 r 3 1 r 2 -1 r 7 -9 r s n 1067 1289 m 2 h 2 -1 r 7 -9 r s 3 3 1058 1281 i 6 5 1067 1270 i 1082 1282 1076 1277 b n 1081 1300 m -2 -2 r -3 v 1 -3 r 2 -2 r 2 -2 r 3 h 3 1 r 2 2 r 2 2 r 3 v -1 3 r -2 2 r -2 2 r -3 h -3 -1 r c s n 1081 1300 m -1 -2 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 2 h s n 1092 1291 m 1 2 r 3 v -1 3 r -1 1 r -3 2 r -3 h -2 h s 9 -11 1094 1311 i 9 -11 1095 1311 i n 1097 1309 m 3 v 2 3 r 1 1 r 3 1 r 2 -1 r 7 -8 r s n 1100 1316 m 2 1 r 2 -1 r 8 -9 r s 4 2 1091 1309 i 5 4 1101 1298 i 1115 1310 1109 1305 b 13 -17 1107 1331 i 14 -16 1107 1331 i n 1114 1324 m 2 v 1 3 r 2 1 r 3 1 r 3 h 2 -2 r 2 -2 r 1 -3 r -2 v -2 -3 r -2 -2 r -2 h -3 h s n 1117 1330 m 2 h 3 h 3 -2 r 1 -1 r 1 -3 r -3 v -1 -2 r s 1107 1331 1104 1329 b 14 -17 1123 1345 i 14 -16 1124 1345 i 3 2 1121 1343 i 1140 1331 1135 1326 b n 1140 1349 m -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 3 1 r 2 1 r 1 3 r 1 3 r -2 3 r -1 1 r -3 2 r -3 h -3 -1 r c s n 1140 1349 m -1 -2 r -3 v 1 -3 r 1 -2 r 3 -1 r 3 -1 r 2 1 r s n 1151 1339 m 2 v 1 3 r -1 3 r -2 2 r -3 1 r -2 1 r -3 -1 r s n 1162 1364 m -2 v 2 h 1 v -1 1 r -3 h -2 h -2 -2 r -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 h 3 1 r 1 1 r 2 3 r 3 v s n 1156 1362 m -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -1 r 3 -1 r 2 1 r s 14 -16 1163 1377 i 13 -17 1164 1378 i -1 -14 1176 1379 i 10 -2 1176 1371 i 10 -2 1175 1370 i 3 3 1161 1375 i 5 4 1174 1377 i 6 4 1174 1359 i 1188 1370 1183 1366 b 11 8 1188 1382 i 45 30 1199 1390 i 45 28 1244 1420 i 45 23 1289 1448 i 1 1 1334 1471 i 8 -12 1342 1482 i 8 -12 1343 1482 i n 1345 1480 m 1 3 r 2 2 r 1 1 r 4 1 r 2 -1 r 5 -10 r s n 1349 1486 m 3 h 2 -1 r 6 -10 r s 3 2 1340 1480 i 6 4 1347 1468 i 1363 1478 1357 1474 b n 1365 1496 m -2 -3 r -3 v -3 v 1 -1 r 3 -3 r 3 h 3 h 2 1 r 2 3 r 3 v 3 v -1 2 r -3 2 r -3 1 r -3 -1 r c s n 1365 1496 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 2 -1 r 3 h s n 1375 1484 m 1 2 r 3 v 3 v -1 2 r -3 2 r -3 1 r -2 h s 8 -12 1379 1504 i 8 -13 1380 1505 i n 1382 1502 m 1 3 r 2 2 r 1 1 r 4 1 r 2 -1 r 5 -10 r s n 1386 1508 m 3 1 r 2 -2 r 6 -9 r s 3 2 1377 1503 i 6 4 1384 1490 i 1400 1500 1394 1496 b 11 -18 1395 1522 i 11 -19 1396 1523 i n 1401 1514 m 1 3 r 1 2 r 2 1 r 3 h 3 h 2 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -3 r -1 -1 r -3 h -3 1 r s n 1405 1520 m 2 h 3 -1 r 3 -2 r 1 -2 r -3 v -3 v -2 -2 r s 1396 1523 1392 1520 b 11 -18 1414 1533 i 12 -19 1414 1534 i 3 2 1411 1532 i 1428 1517 1422 1513 b n 1430 1535 m -2 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 -1 r 3 1 r 2 1 r 2 2 r 3 v 3 v -1 2 r -3 2 r -3 1 r -3 -1 r c s n 1430 1535 m -1 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 h 2 h s n 1440 1524 m 1 2 r 1 3 r -1 3 r -1 1 r -3 3 r -2 h -3 h s n 1455 1546 m -1 v 1 -1 r 2 v 1 v -3 h -2 h -3 -1 r -2 -3 r -1 -3 r 1 -3 r 1 -2 r 3 -2 r 2 h 4 h 1 1 r 2 3 r 1 3 r s n 1448 1546 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 h s 11 -19 1458 1560 i 12 -18 1458 1560 i -4 -14 1471 1559 i 9 -3 1470 1551 i 9 -4 1469 1551 i 3 2 1455 1558 i 6 3 1468 1558 i 6 3 1466 1540 i 1481 1549 1476 1546 b 31 15 1483 1561 i 45 19 1514 1576 i 45 13 1559 1595 i 39 18 1604 1608 i 5 -13 1651 1635 i 5 -14 1652 1636 i n 1653 1633 m 1 3 r 3 2 r 2 h 3 1 r 2 -2 r 4 -10 r s n 1659 1638 m 2 h 2 -1 r 4 -11 r s 4 2 1648 1634 i 6 3 1654 1621 i 1671 1628 1664 1625 b n 1676 1645 m -3 -2 r -1 -2 r -4 v 1 -1 r 2 -3 r 3 -1 r 3 h 2 1 r 3 2 r 1 3 r -1 3 r 2 v -2 2 r -3 1 r -3 h c s n 1676 1645 m -2 -1 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s n 1683 1633 m 2 2 r 1 2 r 4 v -1 1 r -2 3 r -3 1 r -2 h s 5 -14 1691 1652 i 5 -13 1692 1652 i n 1693 1649 m 1 3 r 3 2 r 2 1 r 3 h 2 -2 r 4 -10 r s n 1699 1655 m 2 h 2 -2 r 4 -10 r s 4 1 1688 1651 i 6 3 1694 1637 i 1711 1644 1704 1641 b 8 -20 1709 1667 i 8 -20 1710 1667 i n 1714 1658 m 1 2 r 2 2 r 2 1 r 3 h 3 -1 r 2 -2 r -2 v 1 -4 r -1 -2 r -3 -2 r -2 -1 r -2 h -3 1 r s n 1719 1663 m 2 h 3 -1 r 2 -3 r -2 v 1 -3 r -1 -3 r -2 -1 r s 1710 1667 1706 1666 b 8 -20 1729 1675 i 8 -20 1730 1675 i 4 1 1726 1674 i 1741 1657 1735 1654 b n 1746 1674 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1746 1674 m -1 -2 r -1 -2 r -3 v 1 -2 r 2 -3 r 2 -1 r 3 h s n 1754 1662 m 1 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1772 1681 m -1 v 1 h 1 v 1 v -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 4 1 r 2 h 2 3 r 1 2 r s n 1765 1682 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 3 h s 8 -20 1777 1694 i 8 -20 1778 1695 i -6 -13 1790 1692 i 8 -5 1788 1684 i 8 -5 1787 1684 i 4 2 1774 1693 i 6 2 1787 1691 i 7 3 1782 1673 i 1798 1680 1793 1678 b 27 8 1802 1692 i 45 9 1829 1700 i 45 22 1874 1709 i 45 15 1919 1731 i 1 1964 1746 e 44 12 1965 1746 i 45 143 614 750 i 6 16 659 893 i 20 -8 658 924 i 20 -8 658 925 i n 657 921 m 4 12 r 2 2 r 2 1 r 2 h 2 -1 r 1 -1 r 1 -2 r -3 v -3 -8 r s n 661 933 m 2 1 r 1 1 r 3 h 1 -1 r 2 -1 r 1 -2 r -2 v s 2 7 677 913 i n 670 926 m 1 2 r 2 h 7 h 2 1 r 1 v 1 v s n 671 928 m 3 h 7 -1 r 1 1 r 1 2 r -1 1 r -1 1 r s n 678 939 m 5 12 r -2 1 r -2 -1 r -2 h -2 -2 r -1 -3 r -3 v 1 -3 r 3 -2 r 2 h 3 -1 r 3 2 r 2 2 r 2 v 1 3 r -1 3 r s n 682 950 m -3 1 r -2 h s n 674 946 m 1 -2 r 1 -3 r 2 -2 r 2 h 3 -1 r 3 2 r 2 1 r s n 682 960 m 1 h -1 v -1 h -1 1 r 3 v 1 3 r 2 2 r 1 1 r 3 h 6 -3 r 3 h 1 1 r s n 685 969 m 9 -4 r 2 h 2 2 r 1 v s n 687 968 m 1 -2 r -1 -6 r -1 -3 r 2 -2 r 2 h 2 h 2 2 r 1 3 r 2 v -1 3 r s n 687 960 m -2 v 1 -2 r 2 h 2 h 2 1 r s 20 -8 685 992 i 20 -8 685 993 i n 694 988 m -2 -1 r -2 -1 r -1 -2 r -4 v 1 -2 r 3 -2 r 2 -1 r 3 h 2 1 r 3 2 r 2 v 2 v -1 3 r s n 689 984 m -3 v 2 -2 r 2 -2 r 2 -1 r 3 h 3 1 r 2 1 r s 1 4 684 989 i 706 988 705 984 b 16 695 999 e 13 696 1000 e n 700 1010 m 711 999 l 3 -3 r 1 -3 r -2 v -1 v -1 -1 r -1 2 r 1 h s 3 6 694 997 i 701 1012 698 1007 b 20 -8 696 1021 i 20 -8 697 1022 i n 695 1018 m 5 11 r 2 3 r 1 h 2 1 r 2 -1 r 2 -2 r -1 v -4 v -3 -7 r s n 700 1029 m 1 2 r 2 h 2 1 r 2 -1 r 1 -2 r 1 -1 r -3 v s 3 6 715 1010 i n 708 1023 m 2 1 r 1 1 r 8 h 1 h 1 1 r -1 2 r s n 710 1024 m 2 h 7 h 2 h 1 2 r -2 2 r -1 h s n 718 1047 m -1 v 2 1 r -1 1 r -1 h -2 -1 r -2 -1 r -1 -3 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 1 r 4 v -1 2 r s n 713 1043 m -2 v 1 -3 r 3 -2 r 1 -1 r 4 h 2 1 r 2 2 r s 15 1 718 1055 i 13 718 1056 e 11 -11 722 1067 i 2 6 717 1053 i 723 1069 721 1063 b 14 -5 725 1074 i 13 -5 726 1075 i n 731 1073 m -2 2 r -1 3 r 2 v 1 3 r 1 1 r 1 -1 r 1 -1 r -2 -1 r 2 v s 2 4 724 1071 i 740 1073 738 1066 b 12 28 737 1090 i 45 90 749 1118 i 18 31 794 1208 i 18 -12 808 1255 i 18 -12 808 1256 i n 806 1253 m 7 10 r 2 2 r 2 h 2 h 2 -1 r 1 -2 r -2 v -1 -3 r -4 -7 r s n 813 1263 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r 1 -1 r -1 -3 r s 4 6 824 1241 i n 820 1255 m 2 1 r 1 h 8 -1 r 1 h 1 1 r 1 v s n 822 1256 m 2 h 7 -3 r 2 1 r 1 1 r -1 2 r -1 1 r s n 831 1266 m 6 10 r -1 1 r -3 1 r -1 -1 r -2 -1 r -2 -2 r -4 v -2 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 2 1 r 4 v 2 v s n 837 1275 m -3 2 r -2 h s n 828 1274 m -3 v 1 -3 r 2 -2 r 1 -1 r 4 -1 r 2 1 r 2 1 r s n 839 1285 m -1 v -1 1 r 1 v 2 v 2 4 r 2 1 r 2 h 2 h 6 -4 r 2 h 2 h s n 844 1293 m 7 -5 r 3 h 2 1 r 1 v s n 845 1292 m 1 -1 r -3 -6 r -1 -3 r 1 -2 r 2 -1 r 2 -1 r 3 2 r 2 3 r 2 v -1 3 r s n 843 1285 m -2 v 1 -2 r 2 -2 r 2 h 2 1 r s 18 -12 848 1316 i 18 -12 848 1317 i n 856 1311 m -3 -1 r -2 -1 r -1 -2 r -1 -3 r 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 h 2 2 r 1 2 r 1 2 r -1 3 r s n 850 1307 m -2 v 1 -3 r 2 -2 r 1 -2 r 3 h 3 h 2 1 r s 2 3 846 1314 i 868 1308 866 1304 b 16 -3 859 1321 i 13 -3 860 1322 i n 866 1331 m 9 -13 r 2 -4 r 1 -3 r -1 -2 r -1 v -2 h 1 v 1 1 r s 4 5 858 1319 i 867 1333 864 1328 b 19 -12 864 1342 i 18 -12 865 1343 i n 863 1340 m 7 10 r 2 2 r 1 h 3 h 2 -1 r 1 -2 r -2 v -1 -3 r -4 -7 r s n 870 1350 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r -1 v -3 v s 4 6 881 1328 i n 876 1342 m 2 1 r 2 h 8 -1 r 1 h 1 1 r -1 1 r s n 878 1343 m 3 h 7 -3 r 1 1 r 1 1 r -1 2 r -1 1 r s n 891 1364 m -1 v 2 h -1 1 r 1 v -3 -1 r -2 -1 r -2 -2 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 1 r 1 4 r 2 v s n 885 1361 m -3 v -3 v 2 -2 r 2 -1 r 3 -1 r 3 1 r 2 1 r s 16 -3 892 1372 i 13 -3 893 1373 i 9 -13 899 1382 i 4 5 891 1370 i 900 1384 897 1379 b 12 -8 904 1389 i 12 -8 904 1390 i n 909 1386 m -2 3 r 3 v 2 v 2 3 r 1 h 1 -1 r -1 v -1 h 1 v s 2 4 902 1386 i 918 1385 914 1379 b 11 15 918 1402 i 45 59 929 1417 i 45 46 974 1476 i 7 7 1019 1522 i 14 -17 1027 1545 i 14 -17 1028 1546 i n 1025 1543 m 9 8 r 3 1 r 2 h 2 -1 r 1 -2 r 1 -2 r -1 v -2 -3 r -7 -5 r s n 1034 1551 m 3 h 1 h 2 -1 r 1 -1 r 1 -2 r -2 v -1 -2 r s 6 5 1038 1526 i n 1038 1541 m 3 1 r 1 -1 r 7 -3 r 2 h 2 v s n 1041 1542 m 2 -1 r 6 -5 r 1 h 2 1 r -1 3 r s n 1052 1548 m 10 8 r -1 2 r -2 1 r -2 h -2 -1 r -3 -2 r -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -2 r 3 h 3 1 r 1 2 r 2 2 r 3 v s n 1061 1555 m -2 3 r -2 1 r s n 1052 1556 m -2 v -1 -3 r 1 -3 r 2 -1 r 2 -2 r 3 h 3 h s n 1066 1565 m -1 v -1 v -1 1 r 1 v 1 2 r 3 3 r 2 h 2 h 2 -1 r 5 -5 r 2 -1 r 1 h s n 1073 1570 m 6 -7 r 2 -1 r 2 1 r 1 h s n 1074 1569 m -2 v -4 -4 r -2 -3 r 1 -2 r 1 -2 r 2 -1 r 3 1 r 3 2 r 2 v 1 3 r s n 1070 1563 m -1 -2 r -3 v 2 -1 r 2 -1 r 2 h s 13 -17 1084 1591 i 13 -17 1085 1592 i n 1090 1583 m -3 h -2 h -1 -1 r -2 -3 r -3 v 1 -3 r 1 -2 r 3 -1 r 3 -1 r 3 1 r 1 2 r 1 2 r 1 3 r s n 1084 1582 m -1 -2 r -1 -3 r 2 -3 r 1 -2 r 3 -2 r 2 h 3 h s 4 3 1081 1589 i 1101 1577 1097 1574 b 14 -7 1096 1592 i 12 -7 1097 1593 i n 1106 1600 m 4 -15 r 1 -5 r -3 v -1 -2 r -1 h -1 h 1 v 1 h s 5 4 1095 1591 i 1108 1601 1103 1597 b 13 -17 1108 1611 i 13 -17 1109 1611 i n 1105 1609 m 10 7 r 3 2 r 1 -1 r 3 -1 r 1 -1 r -2 v -2 v -2 -3 r -6 -5 r s n 1115 1616 m 2 1 r 2 h 2 -1 r 1 -2 r 1 -2 r -1 -2 r -1 -2 r s 6 4 1119 1592 i n 1119 1606 m 2 1 r 2 h 7 -4 r 1 h 1 1 r 1 v s n 1121 1607 m 2 -1 r 7 -4 r 1 h 2 1 r -1 2 r -1 1 r s n 1140 1623 m -1 v 1 h 1 v 1 v -3 h -3 h -2 -2 r -2 -3 r -3 v 1 -3 r 1 -1 r 3 -2 r 3 -1 r 3 2 r 2 1 r 2 3 r 3 v s n 1133 1622 m -1 -2 r -3 v 1 -3 r 1 -2 r 3 -2 r 3 h 2 1 r s 14 -7 1143 1630 i 12 -6 1144 1631 i 4 -15 1153 1638 i 5 4 1142 1629 i 1155 1639 1150 1635 b 10 -11 1159 1643 i 9 -11 1160 1644 i n 1164 1639 m -1 3 r 3 v 1 2 r 3 2 r 1 h 1 -1 r -1 v -2 h 1 v s 3 3 1157 1641 i 1172 1635 1166 1630 b 22 20 1177 1651 i 45 32 1199 1671 i 45 26 1244 1703 i 24 18 1289 1729 i 10 -19 1317 1763 i 10 -20 1318 1764 i n 1314 1762 m 11 5 r 4 1 r 1 -1 r 2 -1 r 1 -2 r -2 v -1 -2 r -2 -2 r -7 -4 r s n 1325 1767 m 3 h 1 h 2 -2 r 1 -1 r -3 v -1 -1 r -1 -2 r s 7 4 1324 1742 i n 1327 1757 m 3 h 1 -1 r 6 -5 r 1 h 1 h 1 2 r s n 1330 1757 m 1 -2 r 5 -5 r 2 -1 r 2 1 r 3 v -1 1 r s n 1342 1761 m 11 6 r -1 1 r -1 2 r -2 h -2 h -3 -1 r -2 -3 r -1 -2 r -4 v 1 -1 r 3 -3 r 2 -1 r 4 1 r 2 1 r 2 2 r 1 3 r s n 1352 1766 m -1 3 r -2 1 r s n 1344 1769 m -1 -2 r -1 -3 r -3 v 1 -2 r 3 -2 r 2 -1 r 3 h s n 1359 1774 m -1 v -1 h 2 v 2 2 r 3 2 r 3 h 1 -1 r 2 -1 r 3 -7 r 2 -1 r 1 h s n 1367 1778 m 4 -8 r 2 -1 r 2 h 1 h s n 1368 1776 m -1 v -5 -4 r -3 -2 r -2 v 1 -2 r 2 -2 r 3 1 r 3 1 r 1 2 r 1 3 r s n 1363 1771 m -2 -2 r -2 v 1 -2 r 2 -1 r 2 h s 10 -19 1382 1796 i 10 -20 1383 1797 i n 1387 1787 m -3 1 r -2 h -2 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r s n 1380 1787 m -1 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 2 h s 4 2 1379 1795 i 1396 1779 1392 1777 b 12 -10 1395 1794 i 11 -9 1395 1795 i n 1406 1800 m 1 -16 r -4 v -1 -3 r -2 -2 r -1 h -1 h 2 v 2 -1 r s 5 3 1393 1793 i 1407 1801 1402 1798 b 9 -19 1410 1810 i 9 -20 1411 1811 i n 1407 1809 m 11 5 r 3 1 r 1 -1 r 2 -1 r 1 -2 r -2 v -1 v -2 -3 r -8 -4 r s n 1418 1814 m 2 h 2 h 1 -1 r 1 -2 r -3 v -1 v -1 -2 r s 6 3 1417 1790 i n 1420 1804 m 2 h 1 -1 r 6 -5 r 2 h 1 h 2 v s n 1422 1804 m 2 -2 r 5 -5 r 1 h 2 h 3 v 1 v s n 1444 1816 m -1 -1 r 2 -1 r 1 v 1 v -3 1 r -3 h -2 -1 r -3 -2 r -3 v -3 v 1 -2 r 2 -3 r 3 h 3 h 2 1 r 2 2 r 1 3 r s n 1437 1816 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 12 -10 1449 1822 i 10 -8 1450 1822 i 1 -16 1460 1828 i 5 3 1447 1821 i 1461 1828 1456 1826 b 6 -13 1467 1831 i 6 -13 1468 1832 i n 1471 1826 m -1 3 r 1 3 r 1 2 r 3 1 r 2 h -1 v -1 v -2 h 1 2 r s 4 2 1464 1830 i 1477 1820 1471 1817 b 29 10 1485 1835 i 45 6 1514 1845 i 45 17 1559 1851 i 40 17 1604 1868 i 6 -21 1651 1900 i 6 -20 1652 1900 i n 1648 1899 m 11 4 r 4 h 1 -1 r 2 -2 r -2 v -2 v -1 -1 r -2 -2 r -8 -3 r s n 1659 1903 m 3 -1 r 1 h 2 -2 r -2 v -2 v -1 -1 r -1 -2 r s 7 3 1654 1878 i n 1660 1892 m 2 h 1 -1 r 5 -6 r 2 -1 r 1 1 r 1 v s n 1662 1892 m 1 -2 r 5 -6 r 1 -1 r 2 1 r 2 v 1 v s n 1675 1894 m 12 4 r -1 1 r -1 2 r -2 1 r -2 h -3 -1 r -2 -2 r -2 -2 r -4 v 1 -1 r 2 -3 r 2 -1 r 4 h 2 h 2 2 r 2 3 r s n 1686 1897 m -1 3 r -2 2 r s n 1678 1901 m -1 -1 r -2 -3 r -3 v 1 -2 r 2 -3 r 2 -1 r 3 h s n 1694 1904 m -1 v -1 h 1 v 1 v 2 2 r 4 1 r 2 -1 r 1 h 2 -2 r 2 -7 r 2 -1 r 1 -1 r s n 1702 1907 m 3 -9 r 2 -2 r 2 h 1 h s n 1703 1905 m -1 -2 r -5 -2 r -3 -2 r -3 v -2 v 2 -1 r 3 h 3 1 r 2 1 r 1 3 r s n 1697 1901 m -2 -2 r -2 v -2 v 2 -2 r 2 h s 7 -20 1720 1922 i 7 -20 1721 1922 i n 1723 1912 m -2 2 r -3 h -2 -1 r -2 -2 r -1 -2 r -1 -3 r 1 -2 r 2 -3 r 3 -1 r 3 h 2 h 1 2 r 2 2 r s n 1716 1913 m -1 -1 r -2 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s 4 1 1717 1921 i 1731 1903 1727 1902 b 10 -12 1732 1918 i 9 -11 1733 1919 i n 1744 1922 m -2 -16 r -4 v -2 -3 r -1 -1 r -1 -1 r -2 1 r 1 1 r 1 h s 6 2 1730 1918 i 1746 1923 1740 1921 b 6 -20 1750 1931 i 6 -21 1751 1932 i n 1747 1930 m 11 4 r 4 h 1 h 2 -2 r -2 v -2 v -1 -2 r -3 -2 r -7 -2 r s n 1758 1934 m 3 h 1 -1 r 2 -1 r -2 v -3 v -1 -1 r -2 -2 r s 7 2 1753 1910 i n 1759 1924 m 2 -1 r 1 -1 r 5 -5 r 2 -1 r 1 h 1 v s n 1761 1923 m 1 -1 r 5 -7 r 1 h 2 h 2 v 1 v s n 1784 1932 m -1 -2 r 2 h 1 v 1 v -3 1 r -2 1 r -3 -1 r -2 -2 r -2 -3 r -3 v 1 -2 r 2 -3 r 2 -1 r 4 h 2 1 r 2 1 r 2 3 r s n 1777 1933 m -1 -2 r -2 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 3 h s 10 -12 1790 1937 i 9 -10 1791 1937 i -2 -15 1802 1940 i 6 2 1788 1936 i 1804 1941 1798 1939 b 4 -14 1810 1943 i 4 -13 1811 1943 i n 1812 1937 m 4 v 2 2 r 1 2 r 3 1 r 2 -1 r -1 v -1 -1 r -1 h 1 2 r s 4 1 1807 1942 i 1818 1930 1811 1928 b 1 1828 1944 e 45 11 1829 1944 i 45 11 1874 1955 i 45 15 1919 1966 i 26 5 1964 1981 i 19 4 1990 1986 i end %%EndDocument @endspecial 80 2259 a Fa(Figure)16 b(1:)j(Comm)o(unications)d(rates)e(for)h (\\ping-p)q(ong")h(b)q(et)o(w)o(een)f(t)o(w)o(o)f(no)q(des)i(on)f(an)g(In)o (tel)h(ipsc/860.)963 2790 y(2)p eop %%Page: 3 3 bop 74 2133 a @beginspecial 72 @llx 72 @lly 504 @urx 504 @ury 4320 @rwi @setspecial %%BeginDocument: ring.ps .24 .24 scale /g0dict 40 dict def g0dict begin /m { moveto } bind def /a { rmoveto } bind def /l { lineto } bind def /v { 0 exch rlineto } bind def /h { 0 rlineto } bind def /s { stroke } bind def /n { newpath } bind def /r { rlineto } bind def /c { closepath } bind def /f { fill } bind def /p { copypage erasepage } def /g { setgray } bind def /b { newpath moveto lineto stroke } bind def /d { newpath moveto 0 exch rlineto stroke } bind def /e { newpath moveto 0 rlineto stroke } bind def /i { newpath moveto rlineto stroke } bind def /rs { dup stringwidth pop 0 exch sub 0 a show } bind def /cs { dup stringwidth pop 2 div 0 exch sub 0 a show } bind def /rshow { gsave currentpoint translate rotate 0 0 moveto show grestore } bind def /rrs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 0 exch sub 0 a show grestore } bind def /rcs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 2 div 0 exch sub 0 a show grestore } bind def 1440 569 569 e 36 569 569 d 18 713 569 d 36 857 569 d 18 1001 569 d 36 1145 569 d 18 1289 569 d 36 1433 569 d 18 1577 569 d 36 1721 569 d 18 1865 569 d 36 2009 569 d n 568 558 m -4 -2 r -2 -3 r 560 546 l -4 v 562 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 568 558 m -3 -2 r -1 -1 r -1 -2 r 562 546 l -4 v 563 536 l 1 -3 r 1 -1 r 3 -1 r s n 571 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 814 553 m 2 1 r 4 4 r -27 v s -25 819 556 d 11 814 531 e n 843 558 m -4 -2 r -2 -3 r 835 546 l -4 v 837 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 843 558 m -3 -2 r -1 -1 r -1 -2 r 837 546 l -4 v 838 536 l 1 -3 r 1 -1 r 3 -1 r s n 846 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 869 558 m -4 -2 r -3 -3 r 861 546 l -4 v 862 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 869 558 m -3 -2 r -1 -1 r -1 -2 r 862 546 l -4 v 864 536 l 1 -3 r 1 -1 r 3 -1 r s n 871 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 894 558 m -3 -2 r -3 -3 r 887 546 l -4 v 888 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 894 558 m -2 -2 r -1 -1 r -2 -2 r 888 546 l -4 v 889 536 l 2 -3 r 1 -1 r 2 -1 r s n 897 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1099 553 m 1 -2 r -1 -1 r -2 1 r 2 v 2 2 r 1 1 r 4 2 r 5 h 4 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -6 -3 r -3 -1 r -2 -2 r -2 -4 r -4 v s n 1109 558 m 3 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -5 -3 r s n 1097 533 m 2 2 r 2 h 7 -3 r 4 h 2 1 r 1 2 r s n 1101 535 m 7 -4 r 5 h 1 1 r 1 3 r 2 v s n 1131 558 m -4 -2 r -3 -3 r 1123 546 l -4 v 1124 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1131 558 m -3 -2 r -1 -1 r -1 -2 r 1124 546 l -4 v 1126 536 l 1 -3 r 1 -1 r 3 -1 r s n 1133 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s n 1157 558 m -4 -2 r -3 -3 r 1149 546 l -4 v 1150 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1157 558 m -3 -2 r -1 -1 r -2 -2 r 1150 546 l -4 v 1151 536 l 2 -3 r 1 -1 r 3 -1 r s n 1159 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1182 558 m -4 -2 r -2 -3 r 1175 546 l -4 v 1176 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1182 558 m -2 -2 r -2 -1 r -1 -2 r 1176 546 l -4 v 1177 536 l 1 -3 r 2 -1 r 2 -1 r s n 1185 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1387 554 m 1 -1 r -1 -2 r -2 2 r 1 v 3 2 r 4 2 r 5 h 4 -2 r 1 -2 r -4 v -1 -2 r -4 -2 r -4 h s n 1397 558 m 2 -2 r 2 -2 r -4 v -2 -2 r -2 -2 r s n 1397 546 m 2 -1 r 3 -3 r 1 -2 r -4 v -1 -3 r -1 -1 r -4 -1 r -5 h -4 1 r -1 1 r -2 3 r 1 v 2 2 r 1 -2 r -1 -1 r s n 1401 544 m 1 -4 r -4 v -1 -3 r -2 -1 r -2 -1 r s n 1419 558 m -4 -2 r -3 -3 r 1411 546 l -4 v 1412 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1419 558 m -3 -2 r -1 -1 r -1 -2 r 1412 546 l -4 v 1414 536 l 1 -3 r 1 -1 r 3 -1 r s n 1421 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1444 558 m -3 -2 r -3 -3 r 1437 546 l -4 v 1438 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1444 558 m -2 -2 r -1 -1 r -2 -2 r 1438 546 l -4 v 1439 536 l 2 -3 r 1 -1 r 2 -1 r s n 1447 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1470 558 m -4 -2 r -2 -3 r 1462 546 l -4 v 1464 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 1470 558 m -2 -2 r -2 -1 r -1 -2 r 1464 546 l -4 v 1465 536 l 1 -3 r 2 -1 r 2 -1 r s n 1473 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s -24 1685 555 d -27 1686 558 d n 1686 558 m 1672 539 l 20 h s 9 1681 531 e n 1707 558 m -4 -2 r -3 -3 r 1699 546 l -4 v 1700 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1707 558 m -3 -2 r -1 -1 r -2 -2 r 1700 546 l -4 v 1701 536 l 2 -3 r 1 -1 r 3 -1 r s n 1709 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1732 558 m -4 -2 r -2 -3 r 1725 546 l -4 v 1726 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1732 558 m -2 -2 r -2 -1 r -1 -2 r 1726 546 l -4 v 1727 536 l 1 -3 r 2 -1 r 2 -1 r s n 1735 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1758 558 m -4 -2 r -2 -3 r 1750 546 l -4 v 1752 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1758 558 m -3 -2 r -1 -1 r -1 -2 r 1752 546 l -4 v 1753 536 l 1 -3 r 1 -1 r 3 -1 r s n 1761 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s 1961 545 1964 558 b n 1961 545 m 3 3 r 3 1 r 4 h 4 -1 r 3 -3 r 1 -4 r -2 v -1 -4 r -3 -3 r -4 -1 r -4 h -3 1 r -2 1 r -1 3 r 1 v 1 2 r 2 -2 r -2 -1 r s n 1971 549 m 3 -1 r 2 -3 r 2 -4 r -2 v -2 -4 r -2 -3 r -3 -1 r s 12 1964 558 e n 1964 556 m 6 h 6 2 r s n 1994 558 m -3 -2 r -3 -3 r 1987 546 l -4 v 1988 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1994 558 m -2 -2 r -1 -1 r -2 -2 r 1988 546 l -4 v 1989 536 l 2 -3 r 1 -1 r 2 -1 r s n 1997 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 2020 558 m -4 -2 r -2 -3 r 2012 546 l -4 v 2014 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 2020 558 m -2 -2 r -2 -1 r -1 -2 r 2014 546 l -4 v 2015 536 l 1 -3 r 2 -1 r 2 -1 r s n 2023 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 2046 558 m -4 -2 r -3 -3 r 2038 546 l -4 v 2039 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 2046 558 m -3 -2 r -1 -1 r -1 -2 r 2039 546 l -4 v 2041 536 l 1 -3 r 1 -1 r 3 -1 r s n 2048 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s 1440 569 569 d 36 569 569 e 18 569 641 e 18 569 713 e 18 569 785 e 18 569 857 e 36 569 929 e 18 569 1001 e 18 569 1073 e 18 569 1145 e 18 569 1217 e 36 569 1289 e 18 569 1361 e 18 569 1433 e 18 569 1505 e 18 569 1577 e 36 569 1649 e 18 569 1721 e 18 569 1793 e 18 569 1865 e 18 569 1937 e 36 569 2009 e n 528 585 m -4 -2 r -2 -3 r 521 573 l -4 v 522 563 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 528 585 m -2 -2 r -2 -1 r -1 -2 r 522 573 l -4 v 523 563 l 1 -3 r 2 -1 r 2 -1 r s n 531 558 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s 392 932 395 945 b n 392 932 m 3 2 r 3 2 r 4 h 4 -2 r 3 -2 r 1 -4 r -3 v -1 -4 r -3 -2 r -4 -1 r -4 h -3 1 r -2 1 r -1 3 r 1 v 1 1 r 2 -1 r -2 -1 r s n 402 936 m 3 -2 r 2 -2 r 2 -4 r -3 v -2 -4 r -2 -2 r -3 -1 r s 12 395 945 e n 395 943 m 6 h 6 2 r s n 425 945 m -3 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 425 945 m -2 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -1 r s n 428 918 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 2 r s n 451 945 m -4 -2 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 2 r c s n 451 945 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 454 918 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 477 945 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 477 945 m -3 -2 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -1 r s n 479 918 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 2 r s n 503 945 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 503 945 m -3 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -1 r s n 505 918 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 2 r s n 528 945 m -4 -2 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 528 945 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 531 918 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 370 1299 m 3 2 r 4 3 r -27 v s -26 375 1303 d 12 370 1277 e n 400 1304 m 396 1303 l -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 400 1304 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 402 1277 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 425 1304 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1304 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -2 r s n 428 1277 m 3 2 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1304 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1304 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 454 1277 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1304 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1304 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 479 1277 m 3 2 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1304 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1304 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -2 r s n 505 1277 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1304 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1304 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 531 1277 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 370 1659 m 3 1 r 4 4 r -27 v s -26 375 1663 d 12 370 1637 e 392 1651 395 1664 b n 392 1651 m 3 3 r 3 1 r 4 h 4 -1 r 3 -3 r 1 -4 r -2 v -1 -4 r -3 -3 r -4 -1 r -4 h -3 1 r -2 2 r -1 2 r 2 v 1 1 r 2 -1 r -2 -2 r s n 402 1655 m 3 -1 r 2 -3 r 2 -4 r -2 v -2 -4 r -2 -3 r -3 -1 r s 12 395 1664 e n 395 1663 m 6 h 6 1 r s n 425 1664 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1664 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1637 m 3 1 r 1 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1664 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1664 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1637 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1664 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1664 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1637 m 3 1 r 1 2 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1664 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1664 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1637 m 3 1 r 1 2 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1664 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1664 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1637 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 368 2019 m 1 -1 r -1 -2 r -2 2 r 1 v 2 2 r 1 2 r 4 1 r 5 h 4 -1 r 1 -2 r 1 -2 r -3 v -1 -2 r 379 2011 l 373 2009 l -3 -2 r -2 -2 r -2 -4 r -4 v s n 378 2024 m 2 -1 r 2 -2 r 1 -2 r -3 v -1 -2 r 378 2011 l 373 2009 l s n 366 2000 m 2 1 r 2 h 7 -3 r 3 h 3 2 r 1 1 r s n 370 2001 m 7 -4 r 5 h 1 1 r 1 3 r 2 v s n 400 2024 m 396 2023 l -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 400 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 402 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 425 2024 m -3 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 425 2024 m -2 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1997 m 3 1 r 1 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -1 2 r -3 1 r s n 451 2024 m -4 -1 r -2 -4 r -2 -7 r -3 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 3 v -1 7 r -2 4 r -4 1 r c s n 451 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s n 477 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 477 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1997 m 3 1 r 1 2 r 2 2 r 1 7 r 3 v -1 7 r -2 2 r -1 2 r -3 1 r s n 503 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 503 2024 m -3 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 528 2024 m -4 -1 r -2 -4 r -1 -7 r -3 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 528 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s 1440 569 2009 e -36 569 2009 d -18 713 2009 d -36 857 2009 d -18 1001 2009 d -36 1145 2009 d -18 1289 2009 d -36 1433 2009 d -18 1577 2009 d -36 1721 2009 d -18 1865 2009 d -36 2009 2009 d 1440 2009 569 d -36 2009 569 e -18 2009 641 e -18 2009 713 e -18 2009 785 e -18 2009 857 e -36 2009 929 e -18 2009 1001 e -18 2009 1073 e -18 2009 1145 e -18 2009 1217 e -36 2009 1289 e -18 2009 1361 e -18 2009 1433 e -18 2009 1505 e -18 2009 1577 e -36 2009 1649 e -18 2009 1721 e -18 2009 1793 e -18 2009 1865 e -18 2009 1937 e -36 2009 2009 e -27 1139 510 d -27 1141 510 d 6 1135 510 e 9 1135 483 e n 1152 494 m 16 h 2 v -2 3 r -1 1 r -3 1 r -3 h -4 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r s n 1166 494 m 3 v -1 3 r s n 1159 501 m -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r s -18 1178 501 d -18 1179 501 d n 1179 497 m 3 3 r 4 1 r 2 h 4 -1 r 1 -3 r -14 v s n 1188 501 m 3 -1 r 1 -3 r -14 v s 5 1174 501 e 9 1174 483 e 9 1188 483 e n 1210 501 m -3 -1 r -1 -1 r -1 -3 r -2 v 1 -3 r 1 -1 r 3 -2 r 2 h 3 2 r 1 1 r 2 3 r 2 v -2 3 r -1 1 r -3 1 r c s n 1207 500 m -1 -3 r -5 v 1 -2 r s n 1215 490 m 1 2 r 5 v -1 3 r s n 1216 499 m 2 1 r 2 1 r -1 v -2 h s n 1206 491 m -1 -1 r -2 -3 r -1 v 2 -3 r 4 -1 r 6 h 4 -1 r 1 -2 r s n 1203 486 m 2 -1 r 4 -2 r 6 h 4 -1 r 1 -3 r -1 v -1 -2 r -4 -2 r -8 h -4 2 r -1 2 r 1 v 1 3 r 4 1 r s n 1230 510 m -22 v 2 -3 r 2 -2 r 3 h 2 2 r 2 2 r s n 1232 510 m -22 v 1 -3 r 1 -2 r s 10 1227 501 e -27 1250 510 d -27 1251 510 d n 1251 497 m 3 3 r 3 1 r 3 h 4 -1 r 1 -3 r -14 v s n 1260 501 m 3 -1 r 1 -3 r -14 v s 5 1246 510 e 9 1246 483 e 9 1260 483 e n 1306 515 m -2 -2 r -3 -4 r -2 -5 r 1297 497 l -5 v 1299 486 l 1301 481 l 3 -4 r 2 -3 r s n 1304 513 m -3 -5 r -1 -4 r 1299 497 l -5 v 1300 486 l 1 -4 r 1304 477 l s -27 1317 510 d -27 1318 510 d n 1318 497 m 2 3 r 3 1 r 3 h 3 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -3 -2 r -3 h -3 2 r -2 2 r s n 1326 501 m 2 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -2 -2 r s 5 1313 510 e 8 -18 1342 501 i 6 -15 1344 501 i n 1358 501 m 1350 483 l 1347 478 l -2 -2 r -3 -2 r -1 h -1 2 r 1 1 r 1 -1 r s 7 1340 501 e 7 1353 501 e n 1368 510 m -22 v 1 -3 r 3 -2 r 2 h 3 2 r 1 2 r s n 1369 510 m -22 v 2 -3 r 1 -2 r s 10 1364 501 e n 1386 494 m 15 h 2 v -1 3 r -1 1 r -3 1 r -4 h -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r 3 h 4 2 r 2 2 r s n 1400 494 m 3 v -1 3 r s n 1392 501 m -2 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 2 -2 r s n 1422 499 m 1 2 r -5 v -1 3 r -1 1 r -3 1 r -5 h -3 -1 r -1 -1 r -3 v 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -2 r s n 1409 497 m 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -1 r -4 v -1 -1 r -3 -2 r -5 h -2 2 r -2 1 r -1 2 r -5 v 1 3 r s n 1431 515 m 3 -2 r 2 -4 r 3 -5 r 1440 497 l -5 v 1439 486 l 1436 481 l -2 -4 r -3 -3 r s n 1434 513 m 2 -5 r 1 -4 r 1439 497 l -5 v 1437 486 l -1 -4 r 1434 477 l s -36 645 2065 d -36 646 2065 d n 640 2065 m 20 h 5 -2 r 2 -1 r 2 -4 r -3 v -2 -4 r -2 -1 r -5 -2 r -14 h s n 660 2065 m 4 -2 r 1 -1 r 2 -4 r -3 v -2 -4 r -1 -1 r -4 -2 r s 12 640 2029 e n 655 2048 m 3 -2 r 2 -1 r 5 -12 r 2 -2 r 2 h 1 2 r s n 658 2046 m 2 -3 r 4 -12 r 1 -2 r 4 h 1 4 r 1 v s n 682 2065 m -1 -2 r 1 -1 r 2 1 r c s -24 682 2053 d -24 684 2053 d 7 677 2053 e 12 677 2029 e -24 701 2053 d -24 703 2053 d n 703 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 715 2053 m 3 -2 r 2 -3 r -19 v s 7 696 2053 e 12 696 2029 e 12 715 2029 e n 744 2053 m -3 -2 r -2 -1 r -2 -4 r -3 v 2 -4 r 2 -1 r 3 -2 r 4 h 3 2 r 2 1 r 1 4 r 3 v -1 4 r -2 1 r -3 2 r c s n 741 2051 m -2 -3 r -7 v 2 -3 r s n 751 2038 m 2 3 r 7 v -2 3 r s n 753 2050 m 1 1 r 4 2 r -2 v -4 h s n 739 2039 m -2 -1 r -1 -4 r -1 v 1 -4 r 5 -2 r 9 h 5 -1 r 2 -2 r s n 736 2033 m 1 -2 r 5 -2 r 9 h 5 -2 r 2 -3 r -2 v -2 -3 r -5 -2 r -10 h -5 2 r -2 3 r 2 v 2 3 r 5 2 r s n 819 2060 m 2 -5 r 10 v -2 -5 r -3 3 r -5 2 r -3 h -6 -2 r -3 -3 r -2 -3 r -1 -6 r -8 v 1 -5 r 2 -4 r 3 -3 r 6 -2 r 3 h 5 2 r 3 3 r 2 4 r s n 808 2065 m -4 -2 r -3 -3 r -2 -3 r -2 -6 r -8 v 2 -5 r 2 -4 r 3 -3 r 4 -2 r s n 842 2053 m -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -5 2 r c s n 842 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 845 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 869 2053 d -24 871 2053 d n 871 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 883 2053 m 3 -2 r 2 -3 r -19 v s n 890 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 902 2053 m 3 -2 r 2 -3 r -19 v s 7 864 2053 e 12 864 2029 e 12 883 2029 e 12 902 2029 e -24 926 2053 d -24 927 2053 d n 927 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 939 2053 m 4 -2 r 2 -3 r -19 v s n 946 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 958 2053 m 4 -2 r 1 -3 r -19 v s 6 921 2053 e 12 921 2029 e 12 939 2029 e 12 958 2029 e n 982 2053 m -19 v 2 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 984 2053 m -19 v 2 -3 r 3 -2 r s -24 1001 2053 d -24 1003 2053 d 7 977 2053 e 7 996 2053 e 7 1001 2029 e -24 1020 2053 d -24 1022 2053 d n 1022 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1034 2053 m 3 -2 r 2 -3 r -19 v s 7 1015 2053 e 12 1015 2029 e 12 1034 2029 e n 1058 2065 m -2 -2 r 2 -1 r 1 1 r c s -24 1058 2053 d -24 1059 2053 d 6 1053 2053 e 12 1053 2029 e n 1094 2048 m -2 -2 r 2 -1 r 1 1 r 2 v -3 3 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1083 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1109 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 4 -2 r 1 -1 r 2 -4 r -12 v 2 -3 r 2 -2 r s n 1124 2050 m -16 v 2 -3 r 4 -2 r 1 h s n 1124 2046 m -1 -1 r -11 -2 r -5 -2 r -1 -3 r -4 v 1 -3 r 5 -2 r 6 h 3 2 r 3 3 r s n 1112 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s n 1143 2065 m -29 v 2 -5 r 3 -2 r 4 h 3 2 r 2 3 r s n 1145 2065 m -29 v 2 -5 r 1 -2 r s 14 1138 2053 e n 1169 2065 m -2 -2 r 2 -1 r 2 1 r c s -24 1169 2053 d -24 1171 2053 d 7 1164 2053 e 12 1164 2029 e n 1195 2053 m -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -5 2 r c s n 1195 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1198 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 1222 2053 d -24 1224 2053 d n 1224 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1236 2053 m 3 -2 r 2 -3 r -19 v s 7 1217 2053 e 12 1217 2029 e 12 1236 2029 e n 1274 2050 m 1 3 r -7 v -1 4 r -2 1 r -4 2 r -6 h -4 -2 r -2 -1 r -4 v 2 -1 r 4 -2 r 8 -4 r 4 -1 r 1 -2 r s n 1256 2048 m 2 -2 r 4 -1 r 8 -4 r 4 -2 r 1 -1 r -5 v -1 -2 r -4 -2 r -7 h -3 2 r -2 2 r -2 3 r -7 v 2 4 r s -36 1316 2065 d -36 1318 2065 d n 1311 2065 m 21 h 5 -2 r 2 -1 r 1 -4 r -5 v -1 -3 r -2 -2 r -5 -2 r -14 h s n 1332 2065 m 3 -2 r 2 -1 r 2 -4 r -5 v -2 -3 r -2 -2 r -3 -2 r s 12 1311 2029 e n 1352 2043 m 21 h 3 v -2 4 r -2 1 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1371 2043 m 5 v -2 3 r s n 1361 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s -24 1387 2053 d -24 1388 2053 d n 1388 2043 m 2 5 r 3 3 r 4 2 r 5 h 2 -2 r -1 v -2 -2 r -2 2 r 2 1 r s 7 1381 2053 e 12 1381 2029 e n 1424 2063 m -1 -1 r 1 -2 r 2 2 r 1 v -2 2 r -3 h -4 -2 r -1 -3 r -31 v s n 1421 2065 m -2 -2 r -2 -3 r -31 v s 13 1411 2053 e 12 1411 2029 e n 1445 2053 m -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r 2 5 r 4 v -2 5 r -4 3 r -5 2 r c s n 1445 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1448 2029 m 4 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -4 2 r s -24 1472 2053 d -24 1474 2053 d n 1474 2043 m 2 5 r 3 3 r 4 2 r 5 h 1 -2 r -1 v -1 -2 r -2 2 r 2 1 r s 7 1467 2053 e 12 1467 2029 e -24 1501 2053 d -24 1503 2053 d n 1503 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1515 2053 m 4 -2 r 1 -3 r -19 v s n 1522 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1534 2053 m 3 -2 r 2 -3 r -19 v s 7 1496 2053 e 12 1496 2029 e 12 1515 2029 e 12 1534 2029 e n 1558 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 4 -2 r 1 -1 r 2 -4 r -12 v 2 -3 r 2 -2 r s n 1573 2050 m -16 v 2 -3 r 4 -2 r 1 h s n 1573 2046 m -1 -1 r -11 -2 r -5 -2 r -1 -3 r -4 v 1 -3 r 5 -2 r 6 h 3 2 r 3 3 r s n 1561 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s -24 1592 2053 d -24 1594 2053 d n 1594 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1606 2053 m 3 -2 r 2 -3 r -19 v s 7 1587 2053 e 12 1587 2029 e 12 1606 2029 e n 1647 2048 m -2 -2 r 2 -1 r 2 1 r 2 v -4 3 r -3 2 r -5 h -5 -2 r -4 -3 r -2 -5 r -4 v 2 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1637 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1661 2043 m 20 h 3 v -1 4 r -2 1 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1680 2043 m 5 v -2 3 r s n 1669 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1733 2072 m -4 -3 r -3 -6 r -4 -6 r -1 -9 r -7 v 1 -8 r 4 -7 r 3 -5 r 4 -4 r s n 1729 2069 m -3 -7 r -2 -5 r -2 -9 r -7 v 2 -8 r 2 -6 r 3 -6 r s -24 1746 2053 d -24 1748 2053 d n 1748 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1760 2053 m 4 -2 r 1 -3 r -19 v s 7 1741 2053 e 12 1741 2029 e 12 1760 2029 e -36 1784 2053 d -36 1786 2053 d n 1786 2048 m 3 3 r 4 2 r 3 h 5 -2 r 4 -3 r 1 -5 r -4 v -1 -5 r -4 -3 r -5 -2 r -3 h -4 2 r -3 3 r s n 1796 2053 m 4 -2 r 3 -3 r 2 -5 r -4 v -2 -5 r -3 -3 r -4 -2 r s 7 1779 2053 e 12 1779 2017 e 31 1846 2050 e 31 1846 2039 e -33 1904 2062 d -36 1906 2065 d n 1906 2065 m 1887 2039 l 27 h s 12 1899 2029 e n 1923 2072 m 3 -3 r 4 -6 r 3 -6 r 2 -9 r -7 v -2 -8 r -3 -7 r -4 -5 r -3 -4 r s n 1926 2069 m 4 -7 r 1 -5 r 2 -9 r -7 v -2 -8 r -1 -6 r -4 -6 r s 27 320 1110 e 27 320 1112 e n 320 1106 m 16 v 1 4 r 2 1 r 2 1 r 3 h 2 -1 r 2 -1 r 1 -4 r -10 v s n 320 1122 m 1 2 r 2 2 r 2 1 r 3 h 2 -1 r 2 -2 r 1 -2 r s 9 347 1106 d n 333 1118 m 1 3 r 1 1 r 9 4 r 2 1 r 1 v -2 2 r s n 334 1121 m 3 1 r 9 2 r 1 2 r 2 v -3 2 r -1 h s n 332 1139 m 1 h -2 v -1 h -2 2 r -1 2 r 5 v 1 3 r 2 1 r 2 1 r 9 h 3 2 r 1 1 r s n 332 1150 m 11 h 3 1 r 1 3 r 1 v s n 334 1150 m 1 -1 r 2 -8 r 1 -4 r 3 -1 r 2 h 3 1 r 1 4 r 4 v -1 3 r -3 2 r s n 337 1141 m 1 -2 r 3 -2 r 2 h 3 2 r 1 2 r s n 320 1164 m 22 h 4 2 r 1 2 r 3 v -1 2 r -3 2 r s n 320 1166 m 22 h 4 1 r 1 1 r s 11 329 1160 d n 337 1182 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1196 m -4 h -3 -1 r s n 329 1189 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 315 1236 m 2 -2 r 4 -3 r 5 -2 r 7 -2 r 5 h 6 2 r 6 2 r 3 3 r 3 2 r s n 317 1234 m 6 -3 r 3 -1 r 7 -1 r 5 h 6 1 r 4 1 r 5 3 r s 27 320 1247 e 27 320 1248 e n 333 1248 m -3 2 r -1 3 r 3 v 1 3 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -3 r -3 v -1 -3 r -3 -2 r s n 329 1256 m 1 2 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -2 r s 5 320 1243 d 18 8 329 1272 i 15 6 329 1274 i n 329 1288 m 18 -8 r 5 -3 r 3 -2 r 1 -3 r -1 v -1 -1 r -2 1 r 2 1 r s 7 329 1270 d 7 329 1283 d n 320 1298 m 22 h 4 1 r 1 3 r 2 v -1 3 r -3 1 r s n 320 1299 m 22 h 4 2 r 1 1 r s 10 329 1294 d n 337 1316 m 15 v -3 h -2 -1 r -2 -1 r -1 -3 r -4 v 1 -3 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 3 r 3 v -1 4 r -3 2 r s n 337 1330 m -4 h -3 -1 r s n 329 1322 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 332 1352 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1339 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s 356 1360 315 1383 b n 332 1402 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1389 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s n 337 1412 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1426 m -4 h -3 -1 r s n 329 1419 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 333 1451 m 1 -1 r 1 1 r -1 1 r -1 h -3 -2 r -1 -3 r -4 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 3 v -1 4 r -3 2 r s n 329 1443 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 315 1460 m 2 2 r 4 3 r 5 3 r 7 1 r 5 h 6 -1 r 6 -3 r 3 -3 r 3 -2 r s n 317 1462 m 6 3 r 3 1 r 7 2 r 5 h 6 -2 r 4 -1 r 5 -3 r s 579 675 569 569 b 586 757 579 675 b 22 -2 575 770 i 22 -2 575 771 i n 585 770 m -1 3 r -1 2 r 2 v 1 3 r 3 2 r 3 h 2 h 3 -1 r 2 -2 r -3 v -3 v -1 -1 r -2 -2 r s n 583 777 m 1 2 r 2 2 r 4 h 2 h 3 -1 r 1 -2 r 1 -2 r s 4 575 767 d 22 -2 577 792 i 22 -2 577 793 i 4 577 789 d 1 7 598 787 i n 586 807 m 1 -4 r 1 -2 r 3 -1 r 2 h 4 h 2 2 r 1 3 r 2 v 3 v -2 3 r -3 1 r -2 h -3 -1 r -3 -1 r -1 -3 r c s n 586 807 m 1 -3 r 1 -2 r 3 -1 r 3 h 3 h 2 2 r 1 2 r s n 600 807 m 2 v -2 3 r -3 1 r -2 h -4 -1 r -2 -1 r -1 -2 r s n 591 833 m 1 -1 r 1 1 r 1 v -1 h -3 -2 r -1 -2 r -3 v 1 -3 r 1 -2 r 3 -2 r 2 h 4 1 r 2 2 r 1 3 r 2 v 3 v -2 2 r s n 588 827 m 1 -2 r 1 -2 r 3 -2 r 2 h 4 1 r 2 2 r 1 2 r s 22 -2 582 843 i 22 -2 582 844 i 10 -12 590 854 i 9 5 596 848 i 9 5 596 847 i 4 582 840 d 1 6 590 851 i 1 7 603 838 i 1 6 604 849 i 4 597 865 d 9 -125 597 869 i 9 35 606 744 i 7 23 615 779 i 21 -7 613 817 i 20 -7 614 818 i n 623 814 m -1 3 r 2 v 2 v 2 3 r 3 1 r 3 h 2 h 3 -2 r 1 -3 r -3 v -1 -2 r -1 -2 r -3 -1 r s n 622 821 m 2 2 r 3 1 r 3 h 2 h 2 -2 r 2 -3 r -2 v s 2 4 612 814 i 21 -7 620 837 i 21 -7 620 838 i 1 4 619 834 i 2 6 640 828 i n 632 850 m -4 v 1 -2 r 3 -2 r 2 -1 r 3 h 2 2 r 2 2 r 1 2 r 3 v -1 3 r -3 2 r -2 1 r -3 h -3 -2 r -2 -2 r c s n 632 850 m -3 v 1 -2 r 3 -2 r 2 -1 r 3 h 3 2 r 1 1 r s n 646 847 m 2 v -2 3 r -2 2 r -2 1 r -3 h -3 -2 r -2 -1 r s n 643 874 m 1 -1 r 1 h -1 2 r -1 h -2 -1 r -2 -2 r -1 -3 r -3 v 1 -3 r 3 -2 r 2 h 3 h 3 1 r 2 3 r 2 v 3 v -1 2 r s n 638 869 m -2 v 2 -3 r 2 -2 r 2 h 4 h 2 1 r 2 2 r s 21 -7 636 886 i 21 -7 636 887 i 7 -13 646 895 i 9 3 651 888 i 10 3 650 887 i 1 4 635 883 i 2 6 645 892 i 2 7 656 876 i 2 6 659 887 i 6 20 655 903 i 10 25 661 923 i 9 16 671 948 i 9 31 680 964 i 9 17 689 995 i 10 18 698 1012 i 9 13 708 1030 i 9 1 717 1043 i 9 25 726 1044 i 11 -18 737 1084 i 12 -19 737 1085 i n 743 1076 m 3 v 2 2 r 1 1 r 4 h 2 h 3 -2 r 1 -2 r 1 -3 r -1 -3 r -2 -3 r -2 -1 r -2 h -3 1 r s n 746 1082 m 3 h 3 -1 r 2 -2 r 1 -2 r 1 -3 r -1 -3 r -1 -2 r s 3 2 734 1083 i 11 -18 755 1095 i 11 -19 756 1096 i 4 2 752 1094 i 6 4 764 1075 i n 772 1097 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 1 r 2 1 r 2 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 1 r -3 -1 r c s n 772 1097 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 h 2 h s n 781 1086 m 1 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 h -2 h s n 796 1108 m -1 v 2 -1 r 2 v -1 1 r -2 h -3 h -2 -1 r -3 -3 r -3 v -3 v 2 -2 r 2 -2 r 3 h 3 h 2 1 r 2 3 r 1 3 r s n 790 1108 m -2 -2 r -3 v -3 v 1 -2 r 3 -2 r 3 -1 r 2 h s 11 -19 799 1122 i 11 -18 800 1122 i -3 -14 812 1121 i 10 -4 811 1114 i 10 -4 810 1113 i 4 2 796 1120 i 5 3 810 1120 i 6 3 808 1102 i 6 3 817 1108 i 4 3 823 1122 i 9 4 827 1125 i 10 10 836 1129 i 9 -1 846 1139 i 9 1 855 1138 i 37 12 864 1139 i 37 21 901 1151 i 37 6 938 1172 i 20 -11 975 1178 i -7 -21 1007 1177 i -7 -20 1008 1176 i n 1005 1167 m 2 1 r 2 h 2 h 3 -2 r 1 -3 r 1 -3 r -1 -2 r -2 -3 r -2 -1 r -4 h -2 1 r -1 1 r -2 3 r s n 1011 1168 m 2 -2 r 1 -2 r 1 -4 r -1 -2 r -2 -2 r -2 -2 r -3 h s 4 -2 1004 1178 i -6 -20 1027 1170 i -6 -21 1028 1170 i 4 -1 1024 1171 i 7 -2 1018 1151 i n 1040 1159 m -3 h -3 -1 r -2 -3 r -2 v -3 v 1 -3 r 3 -2 r 2 h 3 h 3 1 r 1 3 r 1 2 r 3 v -1 3 r -3 2 r c s n 1040 1159 m -2 h -3 -2 r -2 -2 r -2 v -4 v 1 -2 r 2 -2 r s n 1038 1145 m 2 h 3 1 r 1 3 r 1 2 r 3 v -1 3 r -2 2 r s n 1065 1148 m -2 h 1 -2 r 1 1 r 1 1 r -2 3 r -1 1 r -3 1 r -4 h -2 -1 r -2 -3 r -1 -2 r -3 v 2 -3 r 2 -2 r 2 h 4 h 2 1 r s n 1060 1153 m -3 h -2 -2 r -2 -2 r -1 -2 r -3 v 2 -3 r 1 -2 r s -7 -21 1077 1156 i -6 -20 1077 1155 i -13 -6 1085 1145 i 3 -10 1079 1141 i 3 -9 1078 1141 i 3 -1 1074 1156 i 6 -1 1082 1146 i 7 -2 1067 1136 i 6 -2 1078 1133 i 30 30 1092 1138 i 37 -13 1122 1168 i 33 103 1159 1155 i 9 -20 1194 1272 i 9 -19 1195 1272 i n 1199 1263 m 2 3 r 1 1 r 2 1 r 3 h 3 -1 r 2 -2 r 1 -2 r -3 v -1 -3 r -2 -2 r -2 -1 r -3 h -2 1 r s n 1204 1268 m 2 h 3 -1 r 2 -2 r 1 -2 r -4 v -1 -2 r -1 -2 r s 1195 1272 1192 1271 b 9 -19 1214 1280 i 9 -20 1215 1281 i 4 2 1211 1279 i 1227 1262 1220 1259 b n 1231 1280 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 3 1 r 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1231 1280 m -1 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 2 h s n 1239 1268 m 1 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1257 1288 m -2 v 1 h 1 v 1 v -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 2 r s n 1250 1288 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 3 -1 r 2 h s 9 -20 1261 1301 i 9 -20 1262 1301 i -6 -14 1275 1299 i 9 -5 1272 1291 i 9 -5 1271 1290 i 3 1 1259 1300 i 6 3 1272 1297 i 7 3 1267 1280 i 1283 1287 1278 1284 b 22 14 1284 1297 i 37 -19 1306 1311 i 37 -16 1343 1292 i 37 6 1380 1276 i 37 9 1417 1282 i 5 1454 1291 e -2 -21 1470 1302 i -2 -22 1471 1302 i n 1470 1292 m 2 1 r 2 1 r 2 h 3 -1 r 2 -3 r 1 -3 r -2 v -2 -3 r -2 -2 r -3 h -2 h -2 1 r -2 2 r s n 1476 1294 m 2 -1 r 2 -2 r 1 -4 r -2 v -2 -3 r -2 -1 r -2 -1 r s 4 1467 1302 e -2 -22 1491 1300 i -2 -22 1492 1300 i 4 1488 1300 e 1493 1278 1486 1279 b n 1506 1291 m -3 -1 r -2 -2 r -2 -3 r -2 v 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 3 r 1 2 r -1 3 r -2 2 r -3 2 r c s n 1506 1291 m -2 -1 r -2 -2 r -2 -3 r -2 v 1 -3 r 2 -2 r 1 -1 r s n 1507 1276 m 2 1 r 2 2 r 1 3 r 2 v 3 v -2 2 r -2 2 r s n 1532 1285 m -1 -1 r 1 -1 r 1 1 r 1 v -2 2 r -2 1 r -3 1 r -3 -1 r -2 -2 r -1 -3 r -1 -2 r 1 -3 r 2 -2 r 3 -2 r 2 h 3 1 r 2 2 r s n 1526 1289 m -2 -1 r -2 -2 r -1 -3 r -1 -2 r 1 -3 r 2 -2 r 2 -2 r s -3 -21 1543 1294 i -3 -21 1544 1294 i -11 -9 1553 1286 i 6 -9 1547 1280 i 5 -8 1546 1280 i 5 -1 1539 1295 i 6 1550 1286 e 7 -1 1537 1273 i 1555 1271 1548 1272 b 2 1562 1280 e 37 5 1564 1280 i 37 -21 1601 1285 i 37 -36 1638 1264 i 36 -36 1675 1228 i 3 3 1711 1192 i 34 44 1714 1195 i 10 89 569 569 i 8 83 579 658 i 1 -14 520 742 i 1 -14 521 742 i n 522 739 m 2 3 r 3 1 r 2 h 3 -1 r 1 -2 r 534 729 l s n 529 743 m 2 -1 r 1 -2 r 533 729 l s 4 517 742 e 7 518 728 e 537 729 529 728 b n 547 744 m -3 -1 r -2 -2 r -1 -4 r -2 v 2 -3 r 2 -2 r 3 h 2 h 3 1 r 2 2 r 1 3 r 2 v -2 3 r -2 2 r -3 1 r c s n 547 744 m -2 -1 r -2 -2 r -1 -4 r -2 v 2 -3 r 2 -2 r 2 h s n 550 730 m 2 1 r 2 2 r 1 3 r 2 v -2 3 r -2 2 r -2 1 r s 1 -14 563 745 i -14 565 745 d n 565 742 m 2 2 r 3 1 r 2 h 3 h 1 -2 r 577 731 l s n 572 745 m 2 -1 r 1 -2 r 576 731 l s 5 560 745 e 7 1 561 730 i 7 573 731 e 1 -21 586 753 i 1 -21 587 753 i n 587 743 m 2 2 r 2 2 r 2 h 3 -1 r 3 -2 r 1 -3 r -2 v -1 -3 r -2 -3 r -3 -1 r -2 h -2 1 r -2 2 r s n 593 747 m 2 -1 r 3 -2 r 1 -3 r -2 v -1 -3 r -2 -3 r -2 -1 r s 4 583 753 e 1 -22 607 755 i 1 -22 608 755 i 4 1 604 754 i 8 605 733 e n 623 748 m -3 -1 r -2 -2 r -1 -3 r -2 v 2 -3 r 2 -2 r 3 -1 r 2 h 3 1 r 2 3 r 1 3 r 2 v -2 3 r -2 2 r -3 1 r c s n 623 748 m -2 -1 r -2 -2 r -1 -3 r -2 v 2 -3 r 2 -2 r 2 -1 r s n 626 734 m 2 1 r 2 2 r 1 4 r 2 v -2 3 r -2 2 r -2 1 r s n 650 747 m -1 -1 r 1 -1 r 1 1 r 1 v -2 2 r -2 1 r -3 h -3 -2 r -2 -2 r -1 -3 r -2 v 1 -3 r 2 -2 r 3 -1 r 2 h 3 2 r 2 2 r s n 644 750 m -2 -2 r -2 -2 r -1 -3 r -2 v 1 -3 r 2 -2 r 2 -1 r s 1 -22 659 758 i 1 -22 660 758 i -9 -11 670 751 i 7 -8 665 745 i 7 -8 664 745 i 5 1 655 757 i 6 667 751 e 7 657 736 e 6 668 737 e 2 -25 604 742 i 9 35 606 717 i 10 34 615 752 i 9 44 625 786 i 5 30 634 830 i 14 -4 637 872 i 14 -4 637 873 i n 640 872 m -2 2 r 4 v 1 2 r 1 2 r 2 1 r 11 -3 r s n 639 880 m 1 1 r 2 1 r 11 -3 r s 1 4 636 869 i 2 7 650 865 i 654 883 652 876 b n 643 898 m -4 v 2 -2 r 2 -2 r 2 h 4 h 2 1 r 2 3 r 2 v 3 v -1 3 r -3 2 r -2 h -3 h -3 -2 r -2 -2 r c s n 643 898 m -3 v 2 -2 r 3 -2 r 2 h 3 h 2 1 r 2 2 r s n 657 896 m 2 v -2 3 r -2 2 r -2 h -4 h -2 -2 r -2 -1 r s 14 -4 647 914 i 14 -4 647 915 i n 650 914 m -2 2 r 4 v 1 2 r 1 2 r 3 1 r 11 -3 r s n 649 922 m 1 1 r 2 1 r 11 -3 r s 1 4 646 911 i 2 7 660 907 i 664 925 663 918 b 21 -5 645 937 i 21 -5 645 938 i n 655 936 m -1 2 r -1 3 r 1 2 r 1 2 r 3 2 r 3 h 2 h 3 -2 r 1 -3 r 1 -3 r -1 -2 r -1 -2 r -3 -1 r s n 654 943 m 1 1 r 3 2 r 3 h 2 h 3 -2 r 1 -3 r 1 -2 r s 645 938 644 934 b 21 -5 650 958 i 21 -5 650 959 i 1 4 649 955 i 672 957 670 950 b n 661 972 m -4 v 1 -2 r 3 -2 r 2 h 3 h 3 1 r 2 3 r 2 v 3 v -2 3 r -2 2 r -2 h -4 h -2 -2 r -2 -2 r c s n 661 972 m -3 v 2 -2 r 2 -2 r 2 h 4 h 2 1 r 2 2 r s n 675 970 m 2 v -2 3 r -3 2 r -2 h -3 h -2 -2 r -2 -1 r s n 670 997 m 1 -1 r 1 h -1 2 r -1 h -2 -2 r -2 -1 r -1 -3 r 1 -4 r 1 -2 r 3 -2 r 2 h 3 h 3 1 r 1 3 r 1 2 r 3 v -2 3 r s n 665 992 m 1 -3 r 2 -2 r 2 -2 r 2 h 4 h 2 1 r 1 2 r s 21 -5 662 1008 i 21 -5 662 1009 i 7 -13 672 1018 i 9 4 677 1011 i 10 4 676 1010 i 1 4 661 1005 i 1 6 671 1015 i 2 7 682 1000 i 686 1017 685 1011 b 1 4 679 1027 i 9 24 680 1031 i 9 689 1055 e 10 6 698 1055 i 9 15 708 1061 i 9 11 717 1076 i 9 9 726 1087 i 9 7 735 1096 i 10 -24 744 1103 i 9 -18 754 1079 i 9 2 763 1061 i 9 2 772 1063 i 1 781 1065 d 13 -5 758 1023 i 13 -5 758 1024 i n 761 1023 m -1 3 r 3 v 1 2 r 2 2 r 2 h 11 -4 r s n 761 1031 m 2 1 r 2 h 10 -4 r s 2 4 756 1020 i 3 6 770 1015 i 777 1032 774 1025 b n 768 1048 m -3 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r 1 2 r 3 v -1 3 r -2 2 r -2 1 r -3 h -3 -1 r -2 -2 r c s n 768 1048 m -2 v 1 -3 r 3 -2 r 2 -1 r 3 h 3 1 r 1 1 r s n 782 1044 m 2 v -1 3 r -3 2 r -1 1 r -4 h -2 -1 r -2 -1 r s 13 -6 775 1063 i 13 -6 775 1064 i n 778 1063 m -1 2 r 4 v 1 2 r 2 2 r 2 h 11 -4 r s n 778 1071 m 1 1 r 3 h 10 -4 r s 2 4 773 1060 i 2 6 787 1055 i 794 1072 791 1065 b 20 -9 777 1087 i 20 -9 777 1088 i n 787 1084 m -1 2 r -1 3 r 1 1 r 2 3 r 3 1 r 3 h 2 -1 r 3 -2 r 1 -3 r -1 -3 r -2 v -2 -2 r -3 -1 r s n 786 1090 m 2 2 r 3 1 r 3 h 2 -1 r 2 -2 r 1 -3 r -2 v s 777 1088 776 1084 b 20 -8 785 1106 i 19 -8 786 1107 i 2 3 784 1104 i 807 1102 804 1095 b n 798 1118 m -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 2 r 1 2 r 3 v -1 3 r -2 2 r -2 1 r -4 h -2 -1 r -2 -2 r c s n 798 1118 m -2 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 1 r s n 812 1114 m 2 v -1 3 r -3 2 r -2 1 r -3 h -3 -1 r -1 -1 r s n 811 1141 m 1 -1 r 1 h -1 2 r -1 h -2 -1 r -2 -1 r -1 -3 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 2 r 1 2 r 3 v -1 3 r s n 806 1137 m -2 v 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 1 r s 20 -9 805 1154 i 19 -9 806 1155 i 6 -13 816 1161 i 10 3 820 1154 i 9 3 820 1153 i 2 4 804 1151 i 2 5 815 1159 i 3 6 824 1143 i 831 1159 828 1153 b 7 70 802 1115 i 9 -5 809 1185 i 9 21 818 1180 i 9 7 827 1201 i 10 -19 836 1208 i 9 20 846 1189 i 9 5 855 1209 i 2 -4 864 1214 i -7 -12 865 1216 i -7 -12 866 1216 i n 864 1213 m 3 1 r 3 -1 r 2 -1 r 2 -2 r -3 v -6 -9 r s n 872 1212 m 1 -2 r -2 v -6 -10 r s 3 -2 863 1218 i 6 -4 855 1206 i 871 1196 865 1200 b n 888 1202 m -3 1 r -3 -1 r -3 -2 r -1 -1 r -4 v -2 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 3 2 r 1 2 r 3 v 3 v -2 2 r c s n 888 1202 m -2 1 r -3 -1 r -3 -2 r -1 -2 r -1 -3 r 1 -3 r 1 -2 r s n 882 1189 m 2 h 3 h 3 3 r 1 1 r 1 3 r -1 3 r -1 2 r s -8 -13 902 1194 i -8 -12 903 1193 i n 901 1191 m 3 h 3 -1 r 2 -1 r 2 -2 r -2 v -6 -10 r s n 909 1189 m 1 -2 r -2 v -6 -9 r s 4 -2 899 1195 i 6 -4 892 1183 i 907 1173 901 1177 b -11 -18 925 1188 i -12 -18 926 1187 i n 920 1179 m 3 h 3 h 1 -1 r 2 -2 r 1 -3 r -1 -3 r -1 -2 r -2 -2 r -3 -1 r -3 1 r -2 1 r -1 2 r -1 3 r s n 927 1178 m 2 -2 r -3 v -1 -3 r -1 -2 r -2 -2 r -3 h -2 h s 926 1187 922 1190 b -11 -19 943 1177 i -11 -18 944 1176 i 3 -2 941 1178 i 935 1156 929 1160 b n 953 1162 m -4 1 r -2 -1 r -3 -2 r -1 -1 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 2 r 1 3 r -1 3 r -2 2 r c s n 953 1162 m -3 1 r -2 -1 r -3 -2 r -1 -2 r -1 -3 r 1 -3 r 1 -2 r s n 947 1149 m 2 h 3 1 r 2 2 r 2 1 r 3 v 3 v -2 2 r s n 974 1146 m -2 -1 r 1 -1 r 1 h 1 1 r -1 3 r -1 2 r -3 2 r -3 h -3 h -2 -2 r -1 -2 r -1 -3 r 1 -3 r 2 -3 r 1 -1 r 4 -1 r 2 1 r s n 970 1152 m -2 h -3 -1 r -2 -2 r -2 -2 r -3 v -3 v 2 -2 r s -11 -19 987 1150 i -11 -18 988 1149 i -14 -4 993 1138 i 1 -10 985 1135 i 1 -11 984 1136 i 4 -2 984 1151 i 6 -3 990 1139 i 6 -4 973 1133 i 988 1124 983 1127 b 24 6 987 1135 i 37 14 1011 1141 i 37 5 1048 1155 i 37 55 1085 1160 i 5 1 1122 1215 i 11 -9 1117 1212 i 11 -9 1118 1213 i n 1120 1211 m 3 v 1 3 r 1 1 r 3 2 r 2 h 9 -7 r s n 1122 1218 m 2 1 r 3 h 8 -7 r s 3 3 1115 1210 i 5 5 1126 1201 i 1138 1215 1133 1209 b n 1134 1233 m -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 1 r 2 3 r -1 3 r -2 3 r -1 1 r -3 1 r -3 h -3 -2 r c s n 1134 1233 m -1 -2 r 1 -3 r 1 -3 r 2 -1 r 3 -2 r 3 1 r 2 1 r s n 1146 1225 m 1 3 r 2 v -2 3 r -2 2 r -3 1 r -3 -1 r -2 -1 r s 11 -9 1144 1246 i 11 -9 1145 1246 i n 1147 1244 m 3 v 1 3 r 1 2 r 3 2 r 2 -1 r 9 -7 r s n 1149 1252 m 2 1 r 3 -1 r 9 -7 r s 3 3 1142 1243 i 5 6 1153 1234 i 1165 1249 1161 1243 b 17 -14 1153 1268 i 17 -13 1153 1268 i n 1161 1262 m 3 v 1 2 r 1 2 r 3 2 r 3 h 3 -1 r 1 -2 r 2 -2 r -3 v -1 -3 r -1 -2 r -2 -1 r -3 h s n 1163 1269 m 2 1 r 3 h 3 -1 r 2 -1 r 1 -3 r 1 -3 r -1 -2 r s 1153 1268 1151 1265 b 17 -13 1166 1284 i 17 -13 1167 1285 i 3 3 1164 1282 i 1186 1274 1181 1269 b n 1182 1292 m -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 2 r 1 3 r 2 v -2 3 r -1 1 r -3 2 r -3 -1 r -3 -1 r c s n 1182 1292 m -1 -2 r 1 -3 r 1 -3 r 2 -1 r 3 -1 r 3 h 2 1 r s n 1194 1285 m 1 2 r -1 3 r -1 2 r -2 2 r -3 1 r -3 h -2 -1 r s n 1201 1311 m -2 v 2 1 r -1 1 r 1 v -3 -1 r -2 -1 r -2 -2 r -2 -3 r 1 -3 r 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 1 r 1 4 r 2 v s n 1195 1308 m -1 -2 r -3 v 2 -3 r 2 -1 r 3 -1 r 3 h 2 1 r s 17 -13 1199 1324 i 17 -13 1199 1325 i 2 -15 1211 1329 i 10 -1 1213 1321 i 11 1212 1320 e 2 3 1197 1322 i 4 5 1209 1326 i 4 6 1214 1308 i 1225 1322 1221 1317 b 23 -25 1210 1319 i 36 39 1233 1294 i 37 4 1269 1333 i 37 -12 1306 1337 i 9 2 1343 1325 i 5 -13 1353 1333 i 5 -14 1354 1334 i n 1355 1331 m 1 3 r 2 2 r 2 h 4 1 r 1 -2 r 4 -10 r s n 1360 1336 m 3 h 1 -1 r 5 -11 r s 4 2 1350 1332 i 7 2 1355 1319 i 1372 1326 1366 1323 b n 1378 1343 m -3 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 2 r 4 v -1 2 r -2 2 r -2 1 r -4 h c s n 1378 1343 m -2 -1 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 2 h s n 1385 1331 m 1 1 r 1 3 r 3 v 2 v -3 3 r -2 1 r -3 h s 5 -13 1393 1349 i 5 -14 1394 1350 i n 1395 1347 m 1 2 r 3 3 r 1 h 4 1 r 1 -2 r 5 -11 r s n 1400 1352 m 3 h 1 -1 r 5 -11 r s 4 2 1390 1348 i 7 2 1395 1335 i 1412 1342 1406 1339 b 8 -20 1411 1364 i 8 -20 1412 1365 i n 1416 1355 m 1 3 r 2 2 r 2 h 3 1 r 2 -2 r 3 -2 r -2 v -3 v -1 -3 r -2 -2 r -2 -1 r -2 h -3 2 r s n 1421 1360 m 2 h 2 -1 r 3 -3 r -1 v 1 -4 r -2 -2 r -1 -2 r s 1412 1365 1408 1363 b 8 -20 1431 1372 i 8 -20 1432 1373 i 4 2 1428 1371 i 1443 1354 1436 1351 b n 1448 1371 m -2 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 4 h 1 1 r 3 2 r 1 2 r 4 v -1 2 r -2 2 r -3 1 r -3 h c s n 1448 1371 m -1 -1 r -1 -3 r -3 v -2 v 3 -3 r 2 -1 r 3 h s n 1455 1359 m 2 1 r 1 3 r 3 v -1 2 r -2 3 r -3 1 r -2 h s n 1474 1378 m -1 v 1 -1 r 1 2 r -1 1 r -3 1 r -2 h -3 -1 r -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 2 -2 r 4 1 r 2 h 2 2 r 1 3 r s n 1467 1379 m -1 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r 2 -1 r 3 h s 8 -20 1479 1391 i 8 -20 1480 1392 i -6 -14 1492 1389 i 8 -5 1490 1381 i 8 -6 1489 1381 i 4 2 1476 1390 i 6 2 1489 1388 i 7 3 1484 1370 i 1500 1377 1495 1374 b 30 16 1497 1385 i 37 -10 1527 1401 i 37 16 1564 1391 i 37 -3 1601 1407 i 23 4 1638 1404 i 87 5 1661 1408 i 10 54 569 569 i 9 50 579 623 i 9 49 588 673 i 4 23 597 722 i 21 -4 591 756 i 21 -4 591 757 i n 591 753 m 2 13 r 2 2 r 1 1 r 2 1 r 2 -1 r 2 -1 r 1 -1 r -3 v 601 755 l s n 593 766 m 1 1 r 2 1 r 2 1 r 2 -1 r 2 -1 r -1 v 1 -2 r s 1 7 612 749 i n 602 761 m 2 1 r 1 1 r 8 2 r 1 1 r 1 v -1 1 r s n 604 762 m 2 1 r 7 1 r 2 h 2 v -2 2 r -1 h s 22 -4 597 789 i 21 -4 598 790 i n 608 787 m -3 -2 r -1 -2 r -1 -2 r 1 -3 r 1 -2 r 3 -2 r 2 h 3 h 3 2 r 1 2 r 1 3 r -1 2 r -1 2 r s n 603 781 m 1 -2 r 2 -2 r 2 -2 r 2 h 4 h 2 2 r 1 1 r s 1 4 597 786 i 4 619 785 d 15 4 607 797 i 13 3 607 798 i n 609 810 m 622 801 l 3 -3 r 2 -3 r 1 -2 r -1 v -2 -1 r 2 v 1 h s 1 6 606 795 i 609 812 608 805 b 22 -4 603 819 i 21 -4 604 820 i n 603 816 m 2 12 r 2 3 r 1 1 r 2 h 2 h 2 -1 r 1 -2 r -3 v -1 -8 r s n 605 828 m 2 2 r 1 1 r 2 h 2 h 2 -2 r 1 -1 r -2 v s 1 7 624 812 i n 615 823 m 1 2 r 1 1 r 8 1 r 1 1 r 1 v 1 v s n 616 825 m 2 h 8 1 r 1 1 r 2 v -1 1 r -1 1 r s n 620 849 m 1 -1 r 1 1 r -1 1 r -1 h -2 -1 r -2 -2 r -3 v -3 v 2 -3 r 3 -1 r 2 -1 r 3 1 r 2 1 r 2 3 r 2 v 3 v -2 3 r s n 616 844 m -2 v 2 -3 r 3 -1 r 2 -1 r 3 1 r 2 1 r 2 2 r s 16 3 618 857 i 13 3 619 858 i 13 -9 621 869 i 1 6 618 855 i 621 871 620 865 b 14 -3 622 877 i 14 -3 623 878 i n 629 877 m -3 2 r -2 2 r -1 2 r 1 3 r 1 1 r 1 h 1 -1 r -1 -1 r -1 1 r s 1 4 622 874 i 637 878 636 871 b 3 22 631 893 i 9 34 634 915 i 9 40 643 949 i 9 23 652 989 i 10 5 661 1012 i 9 -2 671 1017 i 9 21 680 1015 i 5 10 689 1036 i 18 -12 681 1050 i 18 -11 682 1050 i n 680 1047 m 6 10 r 3 2 r 1 1 r 3 -1 r 1 -1 r 2 -2 r -1 v -1 -3 r -4 -7 r s n 686 1057 m 2 2 r 2 h 2 h 2 -2 r 1 -2 r -1 v -2 v s 4 6 698 1035 i n 693 1049 m 2 1 r 2 1 r 8 -2 r 1 h 1 v 2 v s n 695 1050 m 3 h 7 -2 r 1 h 1 2 r -1 2 r -1 h s 18 -12 700 1077 i 18 -12 700 1078 i n 708 1071 m -3 h -2 -1 r -1 -2 r -1 -3 r 1 -3 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 2 r 1 2 r -1 3 r s n 702 1068 m -2 v -3 v 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 1 r s 2 3 698 1075 i 720 1069 718 1065 b 16 -3 711 1082 i 13 -3 712 1083 i n 718 1092 m 727 1079 l 2 -4 r -3 v -2 v -1 v -2 h 1 v 1 h s 3 5 710 1080 i 719 1094 716 1089 b 18 -12 716 1103 i 18 -12 717 1104 i n 715 1100 m 7 11 r 2 2 r 2 h 2 h 2 -2 r 1 -1 r -2 v -1 -3 r -4 -7 r s n 722 1111 m 2 1 r 1 h 2 h 2 -1 r 1 -2 r -2 v -2 v s 4 6 733 1088 i n 728 1102 m 2 2 r 2 h 8 -2 r 1 1 r 1 1 r -1 1 r s n 730 1104 m 3 -1 r 7 -2 r 1 h 1 2 r -1 2 r -1 1 r s n 743 1125 m -2 v 2 1 r 1 v -1 1 r -3 -1 r -2 -1 r -2 -3 r -1 -3 r 1 -3 r 2 -2 r 2 -1 r 3 -1 r 3 h 2 2 r 1 2 r 1 3 r 3 v s n 737 1121 m -2 v -3 v 3 -2 r 1 -2 r 3 h 3 h 2 1 r s 15 -2 745 1132 i 13 -2 745 1133 i 9 -13 751 1143 i 4 5 743 1131 i 752 1144 749 1139 b 12 -8 756 1150 i 12 -7 756 1150 i n 762 1147 m -2 3 r -1 2 r 3 v 2 2 r 1 1 r 1 -1 r 1 -1 r -2 -1 r 2 v s 2 3 754 1147 i 770 1145 766 1139 b 1 1 762 1149 i 9 28 763 1150 i 9 26 772 1178 i 9 -3 781 1204 i 10 12 790 1201 i 9 9 800 1213 i 9 7 809 1222 i 9 10 818 1229 i 9 9 827 1239 i 10 5 836 1248 i 9 4 846 1253 i 9 20 855 1257 i 2 864 1277 e 9 -19 869 1292 i 9 -20 869 1293 i n 866 1291 m 11 5 r 3 h 2 h 1 -2 r 1 -1 r -3 v -1 v -3 -2 r -7 -4 r s n 877 1296 m 2 h 2 -1 r 1 -1 r 1 -2 r -2 v -1 v -2 -2 r s 6 3 875 1271 i n 878 1285 m 3 h 1 h 6 -5 r 1 -1 r 1 1 r 1 1 r s n 881 1285 m 2 -1 r 4 -6 r 2 h 2 1 r 2 v -1 1 r s 9 -20 898 1306 i 9 -19 899 1306 i n 903 1297 m -3 1 r -2 h -2 -1 r -3 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -3 r 3 -1 r 3 1 r 2 1 r 1 1 r 1 3 r s n 896 1297 m -2 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 3 1 896 1305 i 911 1288 907 1286 b 11 -11 911 1304 i 10 -9 912 1304 i n 922 1309 m -16 v -4 v -1 -3 r -1 -2 r -1 h -2 h 1 1 r 1 h s 6 2 909 1303 i 924 1310 918 1307 b 9 -20 926 1319 i 9 -19 927 1319 i n 924 1317 m 11 6 r 3 h 1 -1 r 2 -1 r 1 -2 r -2 v -2 v -3 -2 r -7 -3 r s n 935 1323 m 2 h 2 -1 r 1 -1 r 1 -2 r -3 v -1 v -2 -2 r s 6 3 933 1298 i n 936 1312 m 3 h 1 -1 r 6 -5 r 1 h 1 h 1 1 r s n 939 1312 m 1 -2 r 5 -5 r 2 -1 r 2 1 r 2 v -1 1 r s n 961 1323 m -1 -1 r 2 -1 r 2 v 1 v -3 1 r -2 h -3 -2 r -3 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r s n 954 1323 m -2 -1 r -1 -3 r 1 -3 r 1 -2 r 2 -3 r 2 -1 r 3 h s 12 -11 966 1329 i 10 -9 967 1329 i 1 -16 977 1334 i 6 3 964 1328 i 979 1335 973 1332 b 6 -13 985 1337 i 6 -13 986 1338 i n 988 1332 m 4 v 1 2 r 1 2 r 3 1 r 1 h 1 -1 r -1 -1 r -1 h 1 v s 4 2 982 1336 i 994 1326 988 1323 b 10 8 1001 1339 i 37 -3 1011 1347 i 37 14 1048 1344 i 37 23 1085 1358 i 37 1122 1381 e 7 6 1159 1381 i 20 -6 1156 1396 i 20 -6 1156 1397 i n 1155 1393 m 3 12 r 2 3 r 2 h 2 1 r 2 -1 r 1 -2 r 1 -1 r -3 v -2 -8 r s n 1158 1405 m 2 2 r 1 h 2 1 r 2 -1 r 2 -2 r 1 -1 r -2 v s 2 7 1175 1387 i n 1167 1399 m 2 2 r 1 h 8 1 r 1 1 r 1 v 1 v s n 1169 1401 m 2 h 8 h 1 h 2 v -1 2 r -1 h s 21 -6 1165 1427 i 20 -6 1166 1428 i n 1175 1424 m -2 -1 r -2 -2 r -1 -2 r -3 v 2 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 3 r 1 2 r -1 2 r -1 3 r s n 1170 1419 m 1 -2 r 1 -2 r 3 -2 r 2 -1 r 3 h 3 1 r 1 2 r s 2 4 1164 1424 i 1187 1425 1186 1421 b 16 2 1175 1435 i 13 1 1176 1436 i n 1179 1447 m 12 -10 r 3 -3 r 2 -3 r -2 v -1 v -2 -1 r 1 v 1 1 r s 2 6 1175 1433 i 1180 1449 1178 1443 b 20 -7 1175 1457 i 20 -7 1175 1458 i n 1174 1454 m 3 12 r 2 2 r 2 1 r 2 h 2 h 1 -2 r 1 -1 r -3 v -2 -8 r s n 1177 1466 m 2 1 r 1 1 r 2 h 2 h 2 -2 r 1 -1 r -2 v s 2 6 1194 1448 i n 1186 1460 m 2 1 r 1 1 r 8 1 r 1 1 r 2 v s n 1188 1461 m 2 1 r 8 h 1 h 2 v -1 2 r -1 h s n 1194 1485 m 1 -1 r 1 1 r -1 1 r -1 h -2 -1 r -2 -2 r -1 -3 r -3 v 2 -3 r 2 -1 r 2 -1 r 4 h 2 1 r 2 3 r 1 2 r 3 v -2 3 r s n 1189 1480 m 1 -2 r 1 -3 r 3 -1 r 2 -1 r 3 h 3 1 r 1 2 r s 16 2 1193 1493 i 13 1 1194 1494 i 12 -10 1197 1505 i 2 6 1193 1491 i 1198 1507 1196 1501 b 14 -5 1199 1513 i 14 -5 1200 1514 i n 1206 1512 m -3 2 r -1 2 r -1 3 r 1 2 r 2 1 r 1 h -1 v -1 -1 r -1 1 r s 1 4 1199 1510 i 1214 1512 1212 1505 b 21 177 1208 1524 i -13 -17 1210 1731 i -13 -17 1211 1730 i n 1208 1732 m 9 -7 r 2 -3 r -1 v -3 v -1 -1 r -3 -1 r -1 h -3 1 r -7 5 r s n 1217 1725 m 1 -2 r -2 v -2 v -1 -1 r -2 -1 r -2 -1 r -2 1 r s 6 -4 1194 1715 i n 1209 1719 m 1 -2 r -2 v -2 -7 r -2 v 1 h 1 h s n 1210 1717 m -1 -3 r -3 -7 r -1 v 2 -1 r 2 1 r 1 h s -13 -17 1236 1710 i -13 -17 1237 1710 i n 1230 1702 m -1 3 r -1 2 r -1 1 r -3 2 r -3 -1 r -3 -2 r -1 -1 r -1 -3 r -3 v 2 -3 r 1 -1 r 3 -1 r 3 1 r s n 1227 1708 m -3 1 r -2 h -3 -2 r -1 -2 r -2 -3 r 1 -3 r 1 -2 r s 3 -2 1234 1712 i 1226 1691 1223 1693 b -4 -15 1240 1698 i -3 -13 1240 1698 i n 1249 1691 m 1236 1683 l -4 -2 r -3 h -3 h 1 v -1 1 r 2 1 r -2 v s 5 -4 1238 1700 i 1251 1690 1246 1693 b -13 -17 1260 1691 i -13 -17 1261 1691 i n 1258 1693 m 10 -7 r 1 -3 r 1 -1 r -1 -3 r -1 -1 r -2 -1 r -2 h -3 1 r -6 5 r s n 1268 1686 m 1 -2 r -2 v -1 -2 r -1 -2 r -2 -1 r -1 h -3 1 r s 5 -4 1245 1676 i n 1259 1680 m 1 -3 r -1 v -2 -8 r -1 v 1 -1 r 2 1 r s n 1260 1677 m -1 -2 r -2 -7 r -1 v 1 -1 r 3 1 r s n 1280 1663 m -1 h -1 v 1 h 1 1 r -1 2 r -1 3 r -2 1 r -3 1 r -3 h -3 -2 r -1 -1 r -1 -3 r -3 v 2 -3 r 2 -1 r 3 -1 r 3 h s n 1277 1669 m -2 1 r -3 -1 r -3 -1 r -1 -2 r -1 -3 r -3 v 1 -2 r s -4 -15 1288 1661 i -3 -13 1288 1661 i -13 -8 1297 1654 i 5 -3 1286 1662 i 1299 1652 1294 1656 b -9 -12 1304 1649 i -9 -11 1305 1648 i n 1301 1643 m 3 2 r 2 h 3 h 2 -2 r -2 v -2 -1 r 2 v 1 h s 4 -2 1301 1650 i 1298 1635 1293 1639 b 25 -176 1280 1660 i -8 -20 1305 1497 i -8 -20 1306 1497 i n 1303 1498 m 11 -5 r 2 -2 r 1 -1 r -2 v -1 -2 r -1 -2 r -2 h -3 h -8 3 r s n 1314 1493 m 1 -1 r 1 -2 r -2 v -1 -2 r -1 -1 r -2 -1 r -2 h s 7 -2 1294 1478 i n 1307 1485 m 2 -2 r -1 v -8 v 1 -1 r 1 -1 r 1 1 r s n 1309 1483 m -2 v -1 -7 r -2 v 2 h 2 1 r 1 v s -9 -20 1336 1484 i -9 -20 1337 1484 i n 1332 1475 m -1 2 r -2 2 r -2 1 r -3 h -3 -1 r -2 -2 r -1 -2 r -3 v 1 -3 r 3 -2 r 2 -1 r 2 h 3 1 r s n 1327 1480 m -2 h -3 -1 r -2 -3 r -1 -2 r -3 v 1 -3 r 2 -1 r s 4 -1 1333 1485 i 1331 1463 1327 1464 b -16 1342 1474 d -13 1343 1473 d n 1354 1469 m -12 -11 r -3 -3 r -3 -1 r -2 h -1 h -1 1 r 2 1 r -1 v s 5 -2 1341 1474 i 1356 1468 1350 1470 b -8 -20 1364 1472 i -8 -20 1365 1472 i n 1361 1473 m 12 -4 r 2 -3 r 1 -1 r -2 v -1 -2 r -2 -2 r -1 h -3 h -8 3 r s n 1373 1469 m 1 -2 r 1 -2 r -2 v -1 -2 r -2 -1 r -1 -1 r -2 h s 7 -3 1353 1454 i n 1366 1460 m 1 -1 r 1 -2 r -8 v -1 v 1 h 2 h s n 1367 1459 m -3 v -7 v -2 v 2 h 2 1 r 1 v s n 1390 1450 m -1 -1 r 1 -1 r 1 1 r -1 3 r -1 2 r -3 1 r -3 h -3 -1 r -2 -2 r -1 -2 r -3 v 1 -3 r 2 -2 r 2 -1 r 3 h 3 1 r s n 1386 1455 m -2 h -3 -1 r -2 -3 r -1 -1 r -4 v 1 -2 r 1 -2 r s -16 1398 1450 d -14 1399 1450 d -12 -11 1410 1445 i 6 -3 1396 1451 i 1412 1444 1406 1447 b -5 -13 1417 1442 i -5 -14 1418 1442 i n 1416 1436 m 2 2 r 3 1 r 2 1 r 3 -2 r -1 v -1 v -1 -1 r -1 2 r 1 h s 4 -1 1414 1443 i 1415 1427 1409 1430 b 36 14 1418 1436 i 36 4 1454 1450 i 37 2 1490 1454 i 37 2 1527 1456 i 29 2 1564 1458 i 3 -22 1600 1474 i 3 -21 1601 1474 i n 1597 1473 m 12 2 r 3 -1 r 2 h 1 -2 r -2 v -1 -3 r -1 v -3 -1 r -9 -1 r s n 1609 1475 m 2 -1 r 2 -1 r 1 -1 r -3 v -1 -2 r -1 v -2 -1 r s 7 1 1600 1452 i n 1608 1464 m 2 h 1 -1 r 4 -7 r 1 -1 r 1 h 1 2 r s n 1610 1464 m 1 -2 r 3 -7 r 1 -1 r 2 h 1 3 r 1 v s 3 -21 1633 1478 i 3 -21 1634 1478 i n 1634 1468 m -2 2 r -2 1 r -2 -1 r -3 -1 r -2 -2 r -1 -4 r 1 -2 r 1 -3 r 2 -1 r 3 -1 r 3 h 1 2 r 2 2 r s n 1628 1470 m -2 -1 r -2 -2 r -1 -4 r 1 -2 r 1 -2 r 2 -2 r 2 -1 r s 4 1630 1478 e 4 1636 1457 e 8 -13 1644 1472 i 7 -12 1645 1473 i n 1656 1474 m 1652 1459 l -2 -4 r -1 -3 r -2 -1 r -1 h -1 1 r 1 v 2 -1 r s 6 1 1642 1472 i 6 1652 1474 e 3 -21 1663 1482 i 3 -22 1664 1483 i n 1660 1482 m 12 2 r 4 -1 r 1 -1 r 1 -2 r -2 v -2 v -1 -1 r -3 -2 r -8 -1 r s n 1672 1484 m 3 -1 r 1 -1 r 1 -2 r -2 v -2 v -1 -1 r -2 -2 r s 7 1 1663 1461 i n 1671 1473 m 2 -1 r 1 -1 r 4 -6 r 1 -1 r 1 h 1 1 r s n 1673 1472 m 1 -2 r 3 -6 r 1 -1 r 2 h 1 2 r 1 v s n 1697 1477 m -1 -2 r 1 h 1 1 r 1 v -2 2 r -2 h -3 h -3 -2 r -2 -2 r -1 -3 r 1 -2 r 1 -3 r 2 -2 r 4 h 2 h 3 1 r 1 3 r s n 1691 1479 m -2 -1 r -2 -3 r -1 -3 r 1 -2 r 1 -3 r 2 -2 r 3 h s 8 -14 1704 1481 i 7 -12 1705 1481 i -4 -15 1716 1482 i 6 1 1702 1480 i 1718 1483 1712 1482 b 2 -14 1724 1483 i 2 -15 1725 1484 i n 1726 1477 m 1 4 r 2 2 r 1 1 r 3 1 r 2 -1 r -1 v -1 -1 r -1 1 r 1 1 r s 4 1 1721 1483 i 1730 1470 1723 1469 b 7 3 1741 1481 i p end %%EndDocument @endspecial 98 x Fa(Figure)15 b(2:)k(Comm)o(unications)c(rates)f(for)h (\\buc)o(k)o(et-brigade")f(messages)h(in)g(a)g(ring)g(of)f(four)g(pro)q (cessors)74 2287 y(on)h(an)g(In)o(tel)h(ipsc/860.)963 2790 y(3)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 17:49:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08019; Tue, 5 Jan 93 17:49:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27933; Tue, 5 Jan 93 17:49:38 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 22:49:37 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27924; Tue, 5 Jan 93 17:49:36 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA22393; Tue, 5 Jan 93 15:49:25 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0n9N5Q-0016ZKC; Tue, 5 Jan 93 15:49 MST Message-Id: Date: Tue, 5 Jan 93 15:49 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Comments on "ready receiver" part of draft implementation We would like to second the position taken by Gropp and Lusk. With our Sandia/Univ. of New Mexico OS (SUNMOS) project, we have also discovered the value of posting non-blocking receives for anticipated messages. In SUNMOS 1, we provided this facility at the kernel level. However, we also saw the benefit of having permanently posted receives. That is, once such a receive is posted, any matching message would get delivered to the specified location (overwriting previous data), with an associated integer flag being incremented upon message receipt. Many applications utilize repetitive boundary data exchanges; this strategy avoids the costs of independent receives. We have proceeded to a new kernel design that uses posted receives as the basis for all message communication. A message passing design document is currently in the works :-). We will be happy to electronically forward that document to you as it becomes available. Stephen Wheat and Barney Maccabe SNL UNM/SNL From owner-mpi-profile@CS.UTK.EDU Wed Jan 13 17:46:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11465; Wed, 13 Jan 93 17:46:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01528; Wed, 13 Jan 93 17:25:49 -0500 X-Resent-To: mpi-profile@CS.UTK.EDU ; Wed, 13 Jan 1993 17:25:47 EST Errors-To: owner-mpi-profile@CS.UTK.EDU Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01521; Wed, 13 Jan 93 17:25:46 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA01688; Wed, 13 Jan 93 17:25:45 -0500 Date: Wed, 13 Jan 93 17:25:45 -0500 Message-Id: <9301132225.AA01688@thud.cs.utk.edu> To: mpi-profile@cs.utk.edu Subject: this is a test testing From owner-mpi-pt2pt@CS.UTK.EDU Fri Jan 15 12:26:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14370; Fri, 15 Jan 93 12:26:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03545; Fri, 15 Jan 93 12:25:16 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 15 Jan 1993 12:25:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03526; Fri, 15 Jan 93 12:25:09 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA19456; Fri, 15 Jan 93 17:24:54 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14219; Fri, 15 Jan 93 10:23:53 MST Date: Fri, 15 Jan 93 10:23:53 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301151723.AA14219@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: "Message Capsule" Proposal (long) Hi all, Here is a proposal that we think may help reduce the number of separate send and receive routines to something much less than "512" without eliminating any of the nice features that we haven't been able to say "no" to. The proposal also allows for "simple" versions of send and receive routines for novice users. The proposal borrows ideas from Jim Cownie, Paul Pierce, and others (without their consent-- don't blame anyone but us!). We hope that this proposal will at least start a few good arguments. :) Leslie Hart hart@fsl.noaa.gov Tom Henderson hender@fsl.noaa.gov Bernardo Rodriguez bernardo@fsl.noaa.gov NOAA Forecast Systems Lab ----------------------------------------------------------------------------- A PROPOSAL FOR "MESSAGE CAPSULES" 1 Introduction to Message Capsules To send a message, a user must first create and initialize a data structure called a "message capsule". (In Fortran77 a message capsule might be an integer array, possibly containing pointers used by underlying C routines. Another way might be to use the POSIX approach mentioned by Bob Knighten.) The message capsule is an opaque object: a user should only access the contents of a message capsule using a special set of routines. A message capsule contains a complete description of the user data in the message including user data location, data type(s), and optional stride information. The capsule will also contain information describing the data in an incoming message. (In the current proposal this information is transmitted with each message. This is only necessary if we allow reception of a message by a process that does not know its complete data description.) A message capsule also contains message tag, context, control flags, and "matching" information. "Matching" information is used by a receiving process to select a message for receipt. "Matching" criteria has not been defined for MPI. A receiving process must also create and initialize a message capsule. When a message is received, the receive routine uses the information in the message capsule to move data from system buffers (or communication hardware) to location(s) in user space described in the capsule. The information in the message capsule is used by the receive routine to guide any data translation that may be necessary in a heterogeneous system. Note that the use of message capsules does not imply extra data movement. The message capsule just tells the system "Here's where to put the data when it arrives...". In the current proposal, message capsules are dynamic data objects. A message capsule can be reused once the data described by it is no longer required for message passing communication. Routines are provided to create and destroy message capsules, attach descriptions of user data, inquire about the contents of a message capsule, and modify message context. Some proposed routines are described below. The send and receive routines have "message capsule" as an additional parameter. Context is not required as an explicit input parameter to these routines. Proposed (incomplete) syntax for these routines is described below along with syntax for check, wait, probe, and release routines. Proposed syntax for "simplified" versions of send and receive routines are also described below. 2 Program Flow General steps for typical send -> receive communications using message capsules: Case 1. Receiving process knows the description of data in the message. 1) SENDING PROCESS: Sending process creates and initializes a message capsule by calling capsule creation and data description routines. Message context specification routine may also be called to select context if "default" context is not desired. A flag is set in the message capsule to indicate that it has been initialized by the user. If multiple messages with identical data descriptions are to be sent sequentially, this step only needs to be done for the first message. 2) RECEIVING PROCESS: Receiving process creates and initializes a message capsule by calling capsule creation and data description routines. Message context specification routine may also be called to select context if "default" context is not desired. A flag is set in the message capsule to indicate that it has been initialized by the user. If multiple messages with identical data descriptions are to be sent sequentially, this step only needs to be done for the first message. 3) SENDING PROCESS: "Send" message. During a send operation, the send (or wait for a non-blocking send) routine uses information in the message capsule to collect user data, construct a "message" (using system buffer(s) if necessary), and send the message to a specified destination. Depending on the implementation, the send routine may also perform data translation in a heterogeneous system. While a send operation is in progress, a flag is set in the capsule to indicate that the capsule should not be used for anything else. After the operation completes, this flag is cleared. A send routine will return an error if the message capsule has not been properly initialized. 4) RECEIVING PROCESS: "Receive" message. During a receive operation, the receive (or possibly wait for a non-blocking receive) routine distributes received data to user location(s) described in the message capsule. Depending on the implementation, the receive routine may also perform data translation in a heterogeneous system. While a receive operation is in progress, a flag is set in the capsule to indicate that the capsule should not be used for anything else. After the operation completes, this flag is cleared. The receive (or wait) routine will return an error if the data in the received message does not match the description in the capsule (ie. mismatch in data length(s) or type(s)). An error will also be returned if the message capsule has not been properly initialized. 5) RECEIVING PROCESS: (OPTIONAL) "Information" about message described by a capsule. Information about a message may be obtained by calling various "info" routines with the corresponding message capsule specified as an argument. Information returned by these routines includes data types, data sizes, data strides, sending process, message tag, and message context. "Info" routines will return an error if a message has not been successfully "probed" or received. Case 2. Receiving process does not know the description of data in the message. 1) SENDING PROCESS: Sending process creates and initializes a message capsule by calling capsule creation and data description routines (same as Case 1). 2) RECEIVING PROCESS: Receiving process creates a message capsule by calling the capsule creation routine. The message capsule may optionally be initialized in this step by calling the data description routines (though it may need to be re-initialized after the message arrives). Message context specification routine may also be called to select context if "default" context is not desired. 3) SENDING PROCESS: "Send" message. Same as Case 1. 4) RECEIVING PROCESS: (NOT OPTIONAL) "Probe" message (inquire if a message with "matching" characteristics has arrived-- "matching" criteria have not yet been defined for MPI). If a message with "matching" characteristics has arrived, data description information is loaded into a message capsule specified as an argument to the probe routine. The message is removed from the system message queue so no other thread may probe or receive the message unless it is "released" by the thread (see "release" in step 5 below). A flag is set in the message capsule to indicate that the data description of the arrived message is valid. This flag is used by the "info" routines. Note that the message capsule must be able to separately store the user's data description and the data description of the "arrived" message. "Info" routines will only return information about "arrived" messages. 5) RECEIVING PROCESS: (NOT OPTIONAL) "Info" routines MUST be called before a receive is posted when data description is not known by the receiving process. In a C program, information returned by these routines may be used to guide dynamic allocation of user data structures to store message data. In a Fortran77 program, this information can be used to check if available static arrays are large enough to hold message data before a receive is posted. 6) RECEIVING PROCESS: (OPTIONAL) "Release" message. If a thread does not wish to receive a message after a successful probe, it must release the message so other threads will have an opportunity to probe or receive the message. The release routine clears the flag mentioned in step 4 and puts the message back into the system message queue. Return to step 4. 7) RECEIVING PROCESS: Receiving process initializes the message capsule by calling data description routine(s) using information returned by the "info" routines. A flag is set in the message capsule to indicate that it has been initialized by the user. If the message capsule was initialized in step 2 and it does not have the same description as the arrived message, then the data description must be "reset" using a special routine, and then initialized. 8) RECEIVING PROCESS: "Receive" message. Same as Case 1. 3 Proposed Message Capsule Initialization and Inquiry Routines Basic functionality of proposed routines are described below. A sample syntax is shown just to make explanation easier. (For example, the issue of whether these routines are functions or subroutines is not important at this point, parameter ordering has not been given any thought, etc.) Also, routine names have been chosen to be descriptive an do not conform to any string length limitations. 3.1 Message Capsule Creation and Destruction Routines MPI_CREATE_CAPSULE (CAPSULE) Create message capsule CAPSULE. MPI_DESTROY_CAPSULE (CAPSULE) Destroy message capsule CAPSULE. 3.2 Data Description Routines Each message capsule contains one or more "data items" to describe the data in the corresponding message. Each data item contains information about data type, data length, and storage location(s). Each data item is created by a single call to a MPI_ATTACH_xxx() routine. Two kinds of data items are supported: "vector" and "strided". A vector data item is used for user data stored in contiguous memory. A strided data item is used for non-contiguous user data that can be described as a sequence of identically sized data blocks separated by a fixed stride. More general user data arrangements may be described by multiple vector and strided data items. (Note that the vector data item is a subset of the strided data item. Vector data items could be removed to get rid of almost half of the MPI_ATTACH_xxx() routines. We included the vector data item because it simplifies the most commonly used data description.) MPI_ATTACH_INT_VECTOR (CAPSULE, LENGTH, ARRAY) The next data item in the message described by message capsule CAPSULE is LENGTH integers. Storage location is user array ARRAY. MPI_ATTACH_REAL_VECTOR (CAPSULE, LENGTH, ARRAY) The next data item in the message described by message capsule CAPSULE is LENGTH real numbers (floats) stored in user array ARRAY. (etc. for other data types)... MPI_ATTACH_INT_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY) The next data item in the message described by message capsule CAPSULE is NUM_BLOCKS blocks of integers. Each block has BLOCK_LENGTH integers. Starting elements in each block are separated by STRIDE integers. Storage location of first block is user array ARRAY. MPI_ATTACH_REAL_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY) The next data item in the message described by message capsule CAPSULE is NUM_BLOCKS blocks of real numbers (floats). Each block has BLOCK_LENGTH real numbers. Starting elements in each block are separated by STRIDE real numbers. Storage location of first block is user array ARRAY. (etc. for other data types)... MPI_ATTACH_RESET (CAPSULE) All data items are removed from message capsule CAPSULE. The flag that indicates that CAPSULE has been initialized by the user is cleared. This routine can be used in step 7 of Case 2 in Section 2 above to remove any existing data item(s) before re-initializing the message capsule. It must be used in this situation if the message capsule contains a user data description that does not match the data description of the arrived message. This routine is also used to discard data descriptions that are no longer needed allowing message capsules to be re-used. 3.3 Context Modification Routine MPI_SET_CONTEXT (CAPSULE, CONTEXT) The context for the message described by message capsule CAPSULE is set to context CONTEXT. 3.4 "Information" Routines These routines must be used when a receiving process does not know the data description of a message it wants to receive. Use of these routines in other cases is optional. In all cases, "length" refers to number of elements (integers, floats, ...) not number of bytes. All of these routines return an error if the specified message has not been successfully probed or received. Only information about messages that have actually arrived is returned. Note that no "info" routine is required for context since this must always be known by the receiving routine. MPI_INFO_NUM_DATA_ITEMS (CAPSULE) Returns the number of data items in the message. MPI_INFO_DATA_ITEM_DESCRIPTION (CAPSULE, ITEM_NUMBER) Returns a description of data item number ITEM_NUMBER in the message described by message capsule CAPSULE. Valid return values include INT_VECTOR_ITEM, REAL_VECTOR_ITEM, INT_STRIDE_ITEM, REAL_STRIDE_ITEM, etc. MPI_INFO_VECTOR_ITEM (CAPSULE, ITEM_NUMBER, LENGTH) LENGTH is set to the length of the vector data described by data item number ITEM_NUMBER in message capsule CAPSULE. An error is returned if the data item is not a vector data item. MPI_INFO_STRIDE_ITEM (CAPSULE, ITEM_NUMBER, NUM_BLOCKS, BLOCK_LENGTH) NUM_BLOCKS is set to the number of blocks in the strided data described by data item number ITEM_NUMBER in message capsule CAPSULE. BLOCK_LENGTH is set to the length of each block. An error is returned if the data item is not a strided data item. Note that stride is not returned since there is no requirement that strides used by the sending and receiving processes match. MPI_INFO_TAG (CAPSULE) Returns the message tag for the message described by message capsule CAPSULE. MPI_INFO_SENDER (CAPSULE) Returns the sending process for the message described by message capsule CAPSULE. 4 Proposed (Partial) Syntax for Send, Receive, Check, Wait, Probe, and Release Routines Syntax examples showing how message capsules affect send, receive, check, wait, probe, and release routines are shown below. Routine naming is for clarity only and does not conform to any of the MPI naming styles. 4.1 Send Syntaxes for the three flavors of send discussed at the Dallas meeting ("synchronous", "blocking", and "non-blocking") are shown below. (These three routines could be reduced to one if the "communication mode" parameter were used. It is also possible to embed even more information in the message capsule such as "communication mode", message tag, and "matching" criteria. We show examples for this case in Section 8.) MPI_SEND_SYNCH (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "synchronous" send. MPI_SEND_BLOCKING (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "blocking" send. MPI_SEND_NON_BLOCKING (CAPSULE, DESTINATION, TAG) Send the message described by message capsule CAPSULE to process DESTINATION using message tag TAG. This is a "non-blocking" send. 4.2 Receive Syntaxes for the two flavors of receive discussed at the Dallas meeting ("blocking" and "non-blocking") are shown below. (These two routines could be reduced to one if the "communication mode" parameter were used.) "Matching" criteria have been left undefined. MPI_RECV_BLOCKING (CAPSULE, "MATCHING CRITERIA") Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in location(s) described in message capsule CAPSULE. This is a "blocking" receive. Error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). MPI_RECV_NON_BLOCKING (CAPSULE, "MATCHING CRITERIA") Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in location(s) described in message capsule CAPSULE. This is a "non-blocking" receive. Error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). (This will only happen if this routine is called when a "matching" message has already arrived.) 4.3 Wait and Check The wait and check routines use the message capsule in a way that is very similar to Jim Cownie's proposal (option 5 in "Removing internal state"). These routines are used with the non-blocking send and receive routines. MPI_CHECK (CAPSULE) CALLED BY A SENDING PROCESS: Return value indicates whether a send operation on the message described by message capsule CAPSULE has completed or not. An error will be returned if a non-blocking send has not been previously posted for CAPSULE. If the send operation has completed, clear the flag in CAPSULE that indicates that communication is in progress. CALLED BY A RECEIVING PROCESS: Return value indicates whether a receive operation on the message described by message capsule CAPSULE has completed or not. An error will be returned if a non-blocking receive has not been previously posted for CAPSULE. If the receive operation has completed, error conditions may be returned if the user-provided data description in the message capsule is not consistent with the data description of the received message (message too long, wrong data types, etc.). If the receive operation has completed, clear the flag in CAPSULE that indicates that communication is in progress. (Depending on the implementation, user data locations described in CAPSULE may be loaded by this routine, or they may already be loaded by the time this routine is called.) MPI_WAIT (CAPSULE) Same as MPI_CHECK() except that the routine will not return until a "matching" message arrives. 4.4 Probe and Release The probe routines also uses the message capsule in a way that is very similar to Jim Cownie's proposal (option 5 in "Removing internal state"). Both wait and check styles of probe are proposed. MPI_PROBE_CHECK (CAPSULE, "MATCHING CRITERIA") If a message with characteristics satisfying "MATCHING CRITERIA" has arrived, copy its description into message capsule CAPSULE. The message is removed from the system "message queue" and may not be probed or received by another thread unless explicitly released using the release routine. "Information" routines can be called using CAPSULE to get information about the message. A receive can also be posted for this message by the thread that owns CAPSULE (after modifying CAPSULE using information from the "info" routines). If a "matching" message has not arrived, return immediately without modifying CAPSULE. Return value indicates whether a message has arrived or not. MPI_PROBE_WAIT (CAPSULE, "MATCHING CRITERIA") Same as MPI_PROBE_CHECK() except that the routine will not return until a "matching" message arrives. MPI_RELEASE (CAPSULE) The previously probed message described by message capsule CAPSULE is put back in the system "message queue". It may now be probed or received by any thread. An error will be returned if the message has not been probed. Data description of arrived message in CAPSULE is invalidated. 4.5 Simplified Versions of Send and Receive Routines Good arguments have been made (we think) for "simplified" versions of send and receive routines that are less flexible and "easier to use". These routines could be built on top of the routines proposed above or implemented directly. The simplified routines proposed below cannot be used with contexts or probe or "info" routines. Only contiguous user data is supported. Receiving process must know message data description. Actually, it is not clear to us that these routines are really easier to use. They do conform more closely to "common practice" though. Simplified versions of the check and wait routines can also be made (we've left them out-- this thing is long enough already!). MPI_SEND_SYNCH_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Send message consisting of LENGTH elements from location ARRAY to process DESTINATION using message tag TAG. DATA_TYPE is the data type for ARRAY. (Pre-defined values for DATA_TYPE are MPI_INT, MPI_REAL, etc. DATA_TYPE is required for heterogeneous communication.) This is a "synchronous" send. MPI_SEND_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "blocking" send. MPI_SEND_NON_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE) Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "non-blocking" send. MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE) Receive the message with characteristics satisfying "MATCHING CRITERIA" and store message data in array ARRAY. Received data type must be DATA_TYPE. Data length must be LENGTH. This is a "blocking" receive. Error conditions will be returned if LENGTH or DATA_TYPE are not consistent with the content of the received message. MPI_RECV_NON_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE) Same as MPI_RECV_BLOCKING_SIMPLE() except that this is a "non-blocking" receive. 5 Code Fragment Examples in "C" Here are some code fragment examples of how programs using message capsules might look. Lots of stuff has been left out (some variable declarations and initializations, error control using return values, etc.) for brevity. Code segments for sending and receiving processes are shown separately for clarity. "Matching" criteria for receives is left undefined. Process ID is assumed to be an integer. 5.1 Blocking Send -> Receive Vector Data Item: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA"); /* receive message */ mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.2 Blocking Send -> Receive Vector Data Item: Receiver Does Not Know Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int tag, receiver_pid, vec_length, num_items, item, data_item; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_PROBE_WAIT(capsule, "MATCHING CRITERIA"); /* probe until message arrives*/ /* Get data description of arrived message using "info" routines and */ /* initialize message capsule. */ num_items = MPI_INFO_NUM_DATA_ITEMS(capsule); /* get number of data items */ /* (Resetting capsule is optional if this is the only place this occurs in */ /* the code. Resetting is necessary if this code fragment (excluding */ /* capsule creation and destruction) is in a loop. Resetting does not */ /* affect information contained in the capsule about the arrived message.) */ MPI_ATTACH_RESET(capsule); if (num_items == 1) /* Deal with single data item message. */ { /* What kind of data item is in the message? */ data_item = MPI_INFO_DATA_ITEM_DESCRIPTION(capsule, 0); if (data_item == REAL_VECTOR_ITEM) /* Deal with real vector data item. */ { /* Get length of vector item and store in "vec_length". */ MPI_INFO_VECTOR_ITEM (capsule, 0, &vec_length); if (vec_length <= 1000) /* Data fits in existing user array. */ { /* Initialize first data item to be "vec_length" floats stored */ /* at location "my_vector". */ MPI_ATTACH_REAL_VECTOR(capsule, vec_length, my_vector); } /* end of vec_length if */ else { ... /* allocate a larger buffer to hold message data */ } /* end of vec_length else */ } /* end of data_item if */ else { ... /* deal with other possibilities for first data item */ } /* end of data_item else */ } /* end of num_items if */ else { ... /* deal with multiple data items */ } /* end of num_items if */ MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA"); /* receive message */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.3 Blocking Send -> Non-Blocking Receive Complicated Message With Context: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid, my_context; float my_vector[1000], my_array[2000]; /* user data */ int my_indices[200]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_SET_CONTEXT (capsule, my_context); /* my_context must be set up earlier */ MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array); MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices); ... MPI_SEND_BLOCKING(capsule, receiver_pid, tag); /* send msg to receiver */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... MPI_SET_CONTEXT (capsule, my_context); /* must be same as sender's context */ MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array); MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices); ... MPI_RECV_NON_BLOCKING(capsule, "MATCHING CRITERIA"); /* post receive */ ... /* "do something" */ MPI_WAIT (capsule); /* wait for message when done */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... 5.4 Blocking Send -> Receive With "Simplified" Routines DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ /* send msg to receiver */ MPI_SEND_BLOCKING_SIMPLE (receiver_pid, tag, my_vector, 1000, MPI_REAL); ... } /* end of i for loop */ ... RECEIVING PROCESS CODE: ... for (i=0; i<10; i++) /* loop 10 times */ { ... /* receive message */ MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", my_vector, 1000, MPI_REAL); mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... 6 Advantages and Disadvantages of "Message Capsule" Approach 6.1 Advantages - The number of separate send and receive routines is greatly reduced without sacrificing functionality. - A user who is used to "common practice" can use the simplified versions of the routines. - A user who wants more flexibility only needs to learn about the features required for his/her specific application. (For example, if I only need contiguous messages, then I don't need to know anything about strided data items. If the receiving process always knows the data description of received messages, then I don't need to know about the probe and "info" routines.) - "Hidden states" are removed so multi-threaded applications won't get confused. - Encapsulation of features in message capsules allows new features to be added later without modifying syntax of existing routines. A new feature would require addition of one or more new routines to modify and examine message capsules. 6.2 Disadvantages - The use of message capsules does not conform exactly to "common practice". It's fairly close to PVM though. - There is no way to prevent a user from messing with a message capsule directly. This is the same problem encountered in C with (FILE *) structures. The problem can be partially alleviated by including "magic numbers" in message capsules. - Possible slight performance degradation due to "casing" in receive routines. - Data description must be sent along with each message. This is only a significant problem if lots of short messages are sent by a program running on a machine with low communication latency. Sending data description information could be avoided if we do not allow a receiving process to be ignorant of the data description of incoming messages. 7 Unresolved Issues - Can the message capsule concept work well with collective communication routines? - Is it really necessary to support multiple data items in a message capsule? It is if we want to allow mixed-type messages. 8 Code Fragment Example With Even More Information Hidden In the Message Capsule The code fragment presented in Section 5.1 has been modified slightly to indicate how the capsule could possibly contain additional information to further reduce the number of send/receive calls and the number of parameters. Message tag and "matching" criteria for receive operations are now "hidden" in the message capsule. Communication "mode" (synchronous, blocking, or non-blocking) is used to reduce the number of send and receive routines. Communication "mode" is also "hidden" in the message capsule. Four new routines have been added. Routine MPI_ASSIGN_TAG() sets message tag. Routines MPI_SET_SEND_MODE() and MPI_SET_RECV_MODE() set communication "mode" (synchronous, blocking, or non-blocking for send, blocking or non-blocking for receive). Routine MPI_SET_MATCH() sets matching characteristics for receive operations. The send routines (MPI_SEND_SYNCH(), MPI_SEND_BLOCKING(), and MPI_SEND_BLOCKING()) have been combined into one routine (MPI_SEND()). The receive routines (MPI_RECV_BLOCKING() and MPI_RECV_NON_BLOCKING()) have been combined into one routine (MPI_RECV()). Both MPI_SEND() and MPI_RECV() have a reduced number of parameters. In the example below, lines containing "###" indicate changes from the example in Section 5.1. (Example from 5.1) Blocking Send -> Receive Vector Data Item: Receiver Knows Data Description DECLARATIONS FOR SENDING AND RECEIVING PROCESSES: Capsule *capsule; /* pointer to message capsule */ int i, tag, receiver_pid; float my_vector[1000]; /* user data */ SENDING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... /* ### Following 4 lines added ### */ MPI_ASSIGN_TAG (capsule, tag); /* Assign the tag to the capsule */ /* Set send mode to "blocking". Valid values for send mode are */ /* MPI_MODE_SEND_NON_BLOCK, MPI_MODE_SEND_BLOCK, and MPI_MODE_SEND_SYNC. */ MPI_SET_SEND_MODE (capsule, MPI_MODE_SEND_BLOCK); MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... compute(my_vector, ...); /* user does something to data */ /* ### tag parameter disappears from parameter list ### */ MPI_SEND(capsule, receiver_pid); /* send msg to receiver */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... RECEIVING PROCESS CODE: ... MPI_CREATE_CAPSULE(capsule); /* create message capsule */ ... /* ### Following 5 lines added ### */ /* Set matching criteria in capsule */ MPI_ASSIGN_MATCH(capsule, "MATCHING CRITERIA"); /* Set the receive mode to "blocking". Valid values for receive mode are */ /* MPI_MODE_RECV_NONBLOCK and MPI_MODE_RECV_BLOCK. */ MPI_SET_RECV_MODE (capsule, MPI_MODE_RECV_BLOCK); MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector); /* initialize capsule */ ... for (i=0; i<10; i++) /* loop 10 times */ { ... /* ### tag parameter disappears from parameter list ### */ MPI_RECV(capsule); /* receive message */ mess_with(my_vector, ...); /* user does something with data */ ... } /* end of i for loop */ ... MPI_DESTROY_CAPSULE(capsule); /* destroy capsule before end of program */ ... From owner-mpi-pt2pt@CS.UTK.EDU Sat Jan 16 05:16:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00760; Sat, 16 Jan 93 05:16:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24719; Sat, 16 Jan 93 05:16:13 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 05:16:08 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24593; Sat, 16 Jan 93 05:15:56 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21412; Sat, 16 Jan 93 02:15:53 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01226; Sat, 16 Jan 93 02:21:14 PST Message-Id: <9301161021.AA01226@jewel.us.oracle.com> Date: Sat, 16 Jan 93 02:21:14 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: mpi Dear Sirs, Over the past 4 years, some of us at Oracle have been addressing many of the same problems that you are addressing in your MPI effort. We have approached the problem from a C/Unix/Systems Software point of view instead of a Fortran/Scientific Software point of view. We learned of your effort from Sandia, whom we are hoping to work with in developing systems software for MP machines, and the communications interface will be a substantial part of this effort. I sent the following comments about MPI to Jack Dongarra. He suggested dropping the note on the 'pt2pt' mailing list. My primary hope is that my radically different background and point of view will stimulate new ideas. My expectations, however, are that you will find me obnoxious and ask me to go away. :-) Cheers, Chuck Simmons csimmons@oracle.com General Comments ---------------- The proposed interface specification is at best disappointing. From our point of view, it essentially standardizes and extends the existing nCUBE model of a message passing interface. We explicitly rejected this model two years ago. In this document, I'll describe why I think this interface model is inadequate. The most important aspect of the proposed standard is its lack of reference to prior related work. Communications interfaces have been standardized before. Thus, the most important question that the proposed standard must address is: Why does no existing interface meet the requirements of our applications? Associated with this are the questions: What concepts can we steal from existing interfaces? What can we learn from past mistakes? The current work on the MPI interface appears to be occuring in a vaccuum, ignoring prior work. The introduction to the draft standard states: "The main advantages of establishing a message passing standard are portability and ease-of-use." If that is the case, then the MPI effort is seriously misguided. For example, the existing TCP/IP standard is far more portable and easier to use than MPI will ever be. However, I would agree that TCP/IP is not adequate as a communications interface for an MP machine. The reason for this is that MP machines provide relatively high-bandwidth low-latency communications. To take advantage of this, a very lightweight communications protocol is required. We first need a highly efficient communications interface; we then need a portable and easy-to-use interface. An efficient communications interface needs to provide the following properties: (1) We require the ability to copy messages from a user address space directly onto a wire without an intervening memcpy() of the message into a system buffer. Similarily, we require that received messages be able to move from a wire into a user address space without an intervening memcpy(). (2) We require that messages be packetized. That is, in TCP/IP, individual message packets are concatenated together at the sender and sent out as a stream of data. This requires the receiver to parse the incoming data stream to recover the individual packets. In an efficient interface, we want the packet boundaries to be preserved. Once we have outlined our requirements from the communications interface, the next question to ask is whether we are ready to standardize on an approach. Standards bodies work best when they standardize existing work, and work worst when they attempt to design. Standardizing too early on an inferior interface will impede research efforts to improve the interface, and additional work will be needed when it becomes time to standardize the desired interface. If it is decided that a standard is desired, then the scope of the standard needs to be carefully delineated. A reasonable scope is that of defining a communications interface at about the ISO network or transport layer. Unfortunately, some of the text in the draft standard suggest that some of the researchers want to address all of the issues associated with building an MP operating system. For example, section 5 refers to creating and destroying processes and to addressing distributed computing in a heterogeneous environment. Appendix A contains routines that are completely unrelated to communications, and which may well conflict with other existing standards, or which are already well defined in other existing standards. (Consider, for example, the date and time routines. Compare and constrast these to functionality provided by ANSI C and/or Posix.) In general, most of the draft standard stays within a reasonable scope. Attempts to extend the standard beyond that reasonable scope should be brutally (:-) suppressed. The core issues of the standard are: 1) Point-to-point communications. This is the heart and soul of the standard. 2) Communications amongst multiple instantiations of a single program (a group or collection), and communications between these collections. This is a reasonable topic to explore since these collections will be common entities on MP machines. However, this is far less important than point-to-point communications. Given a good portable interface for point-to-point communications, an application can fairly easily implement completely portable and reasonably efficient inter- and intra- collection communications. 3) Reliable messaging and sequencing of messages. This is, for the most part, explicitly ignored by the draft standard. However, this is far more important than (2). As mentioned before, applications don't need (2) to be standardized in order to be portable. However, it is extremely difficult to write the code needed to provide robust, efficient reliable messaging. 4) Threads (aka "Communications Contexts"). For the most part, this is orthogonal to communications. Threads need to be addressed to the extent that the communications interface model must be capable of coexisting with a reasonable threads implementation. In summary, the draft standard should contain a section describing the scope of the standard. Issues such as heterogeneity, performance tracing, parallel I/O, process creation, load balancing, and environmental queries should be placed explicitly outside the scope of the standard, except to the extent that the standard needs to coexist with these features. [Note: On page 3, a reference is made to dynamic load balancing. The interesting problem of dynamic load balancing from the communications interface point of view is that some addresses known to some processes may become invalid at arbitrary points in time. The operating system may need to be able to forward messages that were sent to an obsolete address.] Specific Comments ----------------- My specific objections to details of the MPI standard fall into the following areas: 1) Reliability and sequencing. I feel that the standard should pay far more attention to these subjects. Reliability and sequencing of messages are extremely important in industrial strength applications. Because of the importance of this area and its difficulty of implementation, the standard should pressure hardware vendors and operating system vendors to provide this functionality. On page 6 of the draft standard, for example, it is explicitly allowed that an operating system may discard a message if that message would overflow system buffers, and the operating system need not inform the sender that the message has been discarded. This immediately requires most applications to make use of inefficient algorithms to detect when a message may have been discarded and retry the send of the message as well as to detect when a message has been successfully received but also retried. This is so onerous, that we have asked nCUBE to implement hardware for their next generation system so that when an OS receives a message, it can immediately return one of the following stati to the sending OS: message accepted; message tossed due to parity error; message tossed due to lack of space. Allowing unreliable messages to be implemented encourages the implementation of unreliable applications. 2) The mechanisms used to receive a message are inefficient. In particular, many applications make use of variable length messages. The current proposal only works well for fixed length messages. In the face of variable length messages, the current proposal requires that an application either over allocate storage; or it requires that the application allow the OS to buffer the message, and then the application will probe for the message to determine the length, allocate a correctly sized buffer, and copy the message into the new buffer. Essentially, we want the OS to be able to copy messages off of a wire directly into user address space, so that it is then easy for the application to access variable length packets. We are willing to accept interfaces that require the hardware to implement paging for efficient implementation. 3) The 'mode' argument on the send and receive functions is inefficient. We strongly suspect that this argument will virtually always be a constant. If that is the case, it is more appropriate to make this constant part of the function name instead of an argument. For example, the obvious implementation of the mpi_csend() routine would look like: int mpi_csend (mode, buf, dest, type, len) ... { switch (mode) { case NON_BLOCKING: return mpi_csend_blocking (buf, dest, type, len); case BLOCKING: return mpi_csend_nonblocking (buf, dest, type, len); case SYNCHRONOUS: return mpi_csend_synchronous (buf, dest, type, len); default: return (save_errno, -1); } } static int mpi_csend_nonblocking (buf, dest, type, len) ... { trap to os; save any error; return value; } static int mpi_csend_blocking (buf, dest, type, len) ... { int msgid; msgid = mpi_csend_nonblocking (buf, dest, type, len); mpi_wait (msgid); return status/length sent; } ... The costs of this approach are: (A) An extraneous error condition when the value of 'mode' is out of bounds. Making the mode part of the function name allows this error to be detected at compile time. (B) Additional cpu cycles wasted passing an additional argument to various routines. (C) Cpu cycles as we decide which piece of code to execute in response to the 'mode'. Further, we take the point of view that the non-blocking operations are of fundamental importance, and that the other modes of operation are relatively minor, uninteresting envelopes around this fundamental operation. 4) The send and receive routines for scatter-gather messages ignore prior work. Consider, for example, the closely related Unix readv() and writev() operations. These are specified as: int readv (int fd, struct iovec *iov, int iovcnt); Since an 'iovec' structure is already defined, this structure should be used instead of inventing a new paradigm that separates this into two distinct arrays. In particular, the draft standard pretty much requires system software to perform an expensive translation to convert iovec's into separate arrays. [Note also that we consider the sending of a contiguous message as a relatively uninteresting envelope around the more fundamental transmission of a scatter-gather message.] 5) The send and receive routines specify the use of a pair of arguments. At a conceptual level, these arguments are a single concept: the address to which a message should be sent (or from which the message should be received). This approach has numerous problems: A) Passing two arguments to specify an address is slower than passing a single argument. B) This implies an implementation whereby each process has one message receive queue. To pull messages off this queue in an order other than that in which the message was received requires a potentially time-consuming queue scan to search for a desired message. A much better approach would allow the OS to take a received message and quickly hash it to one of a few receive queues for a process. Since the code that sends a message is generally closely related to the code that receives a message, the sender can make sure that it puts the message on a queue in such a fashion that the receiver will never be interested in anything other than the head of one of its queues. C) Allocation of message type constants by library code in a robust fashion is exceedingly difficult. If message types are to be required, the standard must address the mechanisms that are to be used to allocate message types so that separately developed libraries are guaranteed to have no conflicts. We prefer the concept of a "port". Essentially, this allows a process to ask the OS to create another receive queue for the process. Sending processes then send messages to a specific queue. A receiving process can retrieve a message from the head of a specific queue, or it can use a routine similar to the Unix select() system call to provide a bitmap of interesting queues and retrieve a message from the head of one of these queues. Taking the above into account, we would then specify the interface as: /* non-blocking scatter/gather recv */ int mpi_recv (port_t local_port, struct iovec *iovecp, int iovec_len); /* blocking version of the above */ int mpi_recvb (port_t local_port, struct iovec *iovecp, int iovec_len); /* synchronous version of the above */ int mpi_recvs (port_t local_port, struct iovec *iovecp, int iovec_len); /* vector versions of the above */ int mpi_recvv (port_t local_port, void *bufp, int blklen, int stride, int nblks); int mpi_recvvb (port_t local_port, void *bufp, int blklen, int stride, int nblks); int mpi_recvvs (port_t local_port, void *bufp, int blklen, int stride, int nblks); /* contiguous versions of the above */ int mpi_recvc (port_t local_port, void *bufp, int buflen); int mpi_recvcb (port_t local_port, void *bufp, int buflen); int mpi_recvcs (port_t local_port, void *bufp, int buflen); Also, in the above, the vector versions of the above routines may need additional work. We envision an rpc protocol whereby an rpc header is prepended to a vector that is to be scattered. The application would like to receive the rpc header into one buffer and then scatter the vector into numerous locations. The most general approach would be for the primitive operation to specify a list of arrays into which the received buffer is to be vector scattered. For mpi_infos() and mpi_infot()... Here you are addressing the problem of how to handle an OS header that is prepended to every message. We agree that the general problem is somewhat tricky. Which is probably why the proposed implementation has some constraints on it that are less than trivially easy to use. Note that in a C program running on Unix, the application may need to disable signals before it calls a recv routine, and re-enable the signals after calling a routine to retrieve header information. One question that needs to be answered is the purpose of this header information. If the purpose is application specific, then perhaps the application should have embedded this information in its RPC header. (Our most generally used RPC protocol does this. This approach also allows a thread context pointer to be easily embedded in the message and appropriately retrieved.) If the purpose of this header is for security and verification that a sender has permission to perform an operation, then the simple source/type information available here is probably inadequate. (The existing Unix paradigm is to provide a cred_t structure in this header that specifies the credentials of the sender. This header information is sufficiently burdensome that an application may want to tell the operating system whether or not to attach credentials information to each message.) For mpi_probe(), we strongly desired both blocking and non-blocking versions of this. The non-blocking version simply tests for the presence of a message and returns. The blocking version blocks until a desirable message is present. Consider again the select() system call which provides for both of these functionalities as well as the ability to specify a timeout on the blocking version of the operation. I'm not qualified to comment on groups since we don't use those much, and when we do, we completely ignore little things like topology and locality. However, lack of qualification never stopped me before... We've been toying with the idea of an "array port". The concept is not well defined. For certain types of applications, we may be able to create a port in a number of processes such that the ports have the property that the bit pattern of one port is easily computable from the bit pattern of another port. The primary goal here is to use a very small amount of storage in each process to specify the addresses of all processes. I would appreciate a lot more information about the intended use of groups. Certainly the proposal is far more complicated than anything I would ever use. Also, it wouldn't work for things that I would use. In particular, when I use a group, the first thing that I am doing is creating a list of addresses for all processes in the group. I cannot implement this using MPI_DEFRG. In particular, the processes in the group don't know the addresses of the processes in the group until after all processes have become part of the group. Typically, what we do is have one process in the group store its address in a well known location (a nameserver). Other processes in the group wait for the first process to appear, and then they send their addresses to the first process. The first process generates the list of addresses and broadcasts it back to all of the processes (now that all the processes are known). Thus, the list of addresses is our canonical data structure. Our canonical operations are the routines that help create this data structure, the routines that help us broadcast the data structure, and the routines that allow us to communicate pt-to-pt to an indexed address in the data structure. This approach is easily extended to allow processes outside the group to communicate with processes inside the group. The mpi_*unpack routines specify that the 'msg' argument is an output argument. It should really be an input argument. For the general pack/unpack routines; again, using a 'struct iovec *' instead of would prevent conflicts with existing paradigms. For section A.4, this section is completely outside the scope of the document. The communications contexts essentially implement threads. The communications routines should not specify how you get information about threads, nor how you go about creating a new thread. Also, the PUSH and POP approach to communications and group contexts looks like a design flaw waiting to bite. In section A.5, the cpu,date,machine,infmn,time, and wall routines are outside the scope of the document. The date and time functions are particularly suspect. Compare then to the ANSI C and Posix routines for manipulating the date and time. The mpi_etext and mpi_error routines are reasonable. Note that in our heavily layered applications, we will constantly need to convert mpi_error codes into Unix error codes, and it might be nice if the MPI error handling paradigm was better integrated into Posix error handling. (All we care is that in addition to implementing mpi_error and mpi_etext, the implementation set 'errno' to a reasonable value.) Note that it is not at all clear that communications contexts will work except for in highly restricted circumstances. They are less general than either threads or ports. In fact, I suspect that ports can provide all the needed functionality of communications contexts, and message types, and pids with less hassle. From owner-mpi-pt2pt@CS.UTK.EDU Sat Jan 16 06:04:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11435; Sat, 16 Jan 93 06:04:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00620; Sat, 16 Jan 93 06:03:55 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 16 Jan 1993 06:03:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00612; Sat, 16 Jan 93 06:03:37 -0500 Received: from jewel.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21711; Sat, 16 Jan 93 03:03:35 PST Received: by jewel.us.oracle.com (5.59.10/37.3) id AA01231; Sat, 16 Jan 93 03:08:56 PST Message-Id: <9301161108.AA01231@jewel.us.oracle.com> Date: Sat, 16 Jan 93 03:08:56 PST From: Charles Simmons To: mpi-pt2pt@cs.utk.edu Subject: RE: Communication contexts > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather Allow me to clarify my sentiments. The example of context usage given in the MPI draft documentation appears to explicitly assume the existance of threads, and here contexts appear to be being used as threads. > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. I could strongly support almost all of the above. [The part I wouldn't support is a "tag" as I will explain below.] We do our work on the nCUBE, which, as you may know, has a message passing interface essentially equivalent to that proposed by MPI. We ran into the exact problem quoted above: it was difficult to write libraries that we're guaranteed not to conflict in their usage of message types. The usage of the word "contexts" in the above is equivalent to the usage of the word "ports" amongst operating systems programmers. Because "port" is slightly less overloaded than "context", I prefer the word "port". Note that ports are sufficiently powerful that they subsume the concepts of "contexts", "pids", and "message types" or "tags" as used by MPI. >The only expression stated so far about the difference between tags and >contexts I had gleaned was that context should now be wildcardable (IE no >-1 for All contexts) while a receive ALL or some other MASK variant would >be allowed on tags. I agree that there is little difference between tags and contexts. This helps explain why ports are so good at subsuming both contexts and tags. > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) I don't particularly understand the above, probably because I haven't been listening in on enough of the discussion. I think I understand what you mean by stack-based management, and we do use the word "registration"... When using ports, stack-based management isn't an issue for the same reasons you don't manage pids and message types using stack based management. For example, when sending a message, instead of using the MPI push_context (contextp); send (dest, type, buf, buflen); pop_context (contextp); you use send (port, buf, buflen); In our usage, a "port" is simply a queue to which messages can be sent. The receiver can remove a message from the head of the queue. Wildcards are neither needed nor implemented, making ports more efficient. The efficiency arises from the fact that a process can have multiple ports. In MPI, you essentially implement one receive queue for each pid. If you want to use a wildcard to access messages out-of-order, you need to waste time scanning the queue. With multiple ports, however, you can set up multiple queues so that you never need to access anything other than the head of a queue. Sometimes ports do need to be "registered". In our terminology, registration is the process of publishing in a well-known location sufficient information about a "port" created by one process that another process can send messages to that port. This is a very high level action that is almost outside the scope of MPI. We use a nameserver for this purpose. (The nameserver is accessed via a well-known port, i.e. a port whose bit-pattern is a well-known constant.) [Do note that we require that the bit-pattern representing a port be allowed to be transmitted between to processes without the operating system needing to know that the bit-pattern represents a port. This means that accessing the nameserver is not required in order to use a port.] For groups, we have thought about extending the concept of a port to an array-port. The simple implementation of this allows each process in a group to create a port. These ports are all sent to a central location which creates an array of ports and broadcasts the array to all ports in the array. The complex memory efficient implementation may require processes of a group to be created in a special fashion. Basically, it allows the address of a port for a specific process in a group to be easily computed by another process given the port of the first process in the group and the index of the target process in the group. -- Chuck From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 11:38:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17983; Tue, 19 Jan 93 11:38:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21908; Tue, 19 Jan 93 11:38:02 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 11:37:57 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21879; Tue, 19 Jan 93 11:37:21 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13277 (5.65c/IDA-1.4.4); Tue, 19 Jan 1993 11:36:49 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA27411; Tue, 19 Jan 93 16:36:38 GMT Date: Tue, 19 Jan 93 16:36:37 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301191636.AA27411@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA05440; Tue, 19 Jan 93 16:35:18 GMT To: csimmons@us.oracle.com Cc: mpi-pt2pt@cs.utk.edu, mpi-context@cs.utk.edu In-Reply-To: Charles Simmons's message of Sat, 16 Jan 93 02:21:14 PST <9301161021.AA01226@jewel.us.oracle.com> Subject: mpi Content-Length: 3447 To: mpi-pt2pt@meiko.co.uk, mpi-context.@meiko.co.uk (Apologies to those who get it twice) Gentlepeople, Chuck Simmons has raised some interesting issues, with which I have a great deal of sympathy. The original messaging model which we implemented on our machines (and which we still support, and expect to continue to support) is in exactly the vein which Chuck is asking for (and has been used for an Oracle port !). In particular CSN supports 1) multiple end points for communication in a single process (know as "transports") 2) No tagging of messages 3) No system buffering, but the ability for users to queue multiple non-blocking receives on a transport, thus providing the buffering they require. 4) Send and receive by struct iovec. (As Chuck observes, the implementation ends up building an iovec (on the stack) for the simpler forms) 5) Both blocking and non-blocking tests for I/O completion. (our test actually has a timeout value). 6) The ability to pass transport addresses around the machine without the system being involved. 7) A standard name server service to associate textual names with transport addresses. If people are really interested then I can probably send the man pages. (Though this is not the main point of this note). HOWEVER (and this is one of the points) although this system had clean, specified semantics and a fast implementation (on our hardware), it hasn't helped to sell machines, and we have now produced an NX style interface as an alternative. I think that the problem with popularising such an interface among users is that it appears to do less for them than a model with implicit buffering, and is therefore harder to start to use. The fact that it allows them greater control and can ultimately produce higher performance is not their immediate concern and does not therefore count for much. Many FORTRAN programmers in particular to do not wish to be concerned with "system programming issues" like buffer management. I would certainly like MPI to be able to support such an interface (so that we can achieve the higher performance it offers, and also make MPI applicable in non-scientific application areas). (Though those who were present in Dallas will have noted that I wasn't trying to push such an approach there, mainly because I think it's a lost cause. NX style is what we have to live with...) (Second point coming up...) However it crosses my mind that there may be some potential for embedding such an interface within the MPI model. (This is a vague thought, but I'm throwing it out so others can think about it as well). Observe that 1) Marc's persistent communication descriptors seem to remove the tag matching requirement (though they're still rather vague !) 2) One way of viewing contexts would be to implement a context as a particular queue of messages (or in other words an entirely separate communication end point) within a process. (The implications of this for contexts would be a) contexts must be declared before use b) their number may be limited c) they should be freed after use ) The combination of the two things could then come somewhere near to what Chuck is asking for... -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 13:04:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21335; Tue, 19 Jan 93 13:04:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25741; Tue, 19 Jan 93 13:04:21 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 13:04:20 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25733; Tue, 19 Jan 93 13:04:18 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26371; Tue, 19 Jan 93 18:04:14 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA19328; Tue, 19 Jan 93 11:03:13 MST Date: Tue, 19 Jan 93 11:03:13 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301191803.AA19328@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: "Message Capsule" Proposal Tony Skjellum pointed out that Zipcode has been successfully using "invoices" that work in the same way as "capsules" for several years. Tony's comments were not sent to the entire mailing list and are repeated below, with his permission: > Please read about our message packing in the following uuencoded, > compressed postscript file. We already have created this technology. > (we've omitted the postscript file) > > We are able to deal effectively with fortran packing and unpacking > of messages, and 'pack+send' and 'unpack+receive' are inherently > more optimizable on some systems. > > - Tony Chuck Simmons points out that: > ... > Standards bodies work best when they standardize existing work, and work > worst when they attempt to design. > ... > > Cheers, Chuck Simmons We think that the success of "invoices" in Zipcode supports the argument that the "Message Capsule" proposal is an attempt to standardize existing work. (BTW, we aren't really that satisfied with the term "capsule". Another term may be better.) Tom Henderson Leslie Hart hender@fsl.noaa.gov hart@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 19 13:13:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21564; Tue, 19 Jan 93 13:13:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26145; Tue, 19 Jan 93 13:12:59 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 19 Jan 1993 13:12:57 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26131; Tue, 19 Jan 93 13:12:55 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26408; Tue, 19 Jan 93 18:12:52 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA19336; Tue, 19 Jan 93 11:11:50 MST Date: Tue, 19 Jan 93 11:11:50 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301191811.AA19336@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: "Message Capsule" Proposal The following is excerpts from an email discussion we have been having with Tony Skjellum regarding message "capsules" or "invoices". We have forwarded it with Tony's permission. Our comments are marked with ">>" and Tony's are marked with ">". Tom Henderson Leslie Hart hender@fsl.noaa.gov hart@fsl.noaa.gov > > Please pass along my comments. Invoices are important, in that they > permit additional possible optimizations. The more the user says > "what" and not "how," to accomplish transfer, the more opportunity for > faster performance. Invoices make Fortran-based message passing much > nicer, as well as providing more sophisticated packing/unpacking for C > people. > >> Several people (including us) were uneasy with the "hidden state" associated >> with stack-based context management. Any particular reason for using stacks >> in Zipcode? > Stacks are a detail of conext handling. They are not essential. > - Tony From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 26 18:04:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23239; Tue, 26 Jan 93 18:04:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26207; Tue, 26 Jan 93 18:02:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 26 Jan 1993 18:02:55 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26199; Tue, 26 Jan 93 18:02:53 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.9) id AA16157; Tue, 26 Jan 93 17:02:49 CST Received: by teak18.cray.com id AA29628; 4.1/CRI-5.6; Tue, 26 Jan 93 17:02:48 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9301262302.AA29628@teak18.cray.com> Subject: MPI task identifiers To: mpi-pt2pt@cs.utk.edu Date: Tue, 26 Jan 93 17:02:44 CST X-Mailer: ELM [version 2.3 PL11b-CRI] At the meeting in Dallas, I asked a question about how tasks were identified, referring (for example) to the argument that one would pass to one of the variants of send. I was surprised to hear that the October draft specified this, and that it was defined as sequential integers starting at 0. I've scanned through the document since then, and didn't notice where this was stated. (The only indication I saw to such identifiers was an example included as "Table 1" on page 9, which listed "PID"s of 0-7.) Each place I'd expect to see a definition, says only "the ID number of the process..." and leaves it as that. But I'm willing to accept that it is there and that I just missed it (or the authors forgot to print it ;-). In any case, I'd like to propose that the final spec should *not* define the internals of task identifiers. Please consider this as such a proposal. I'd appreciate hearing any comments. Assuming the October spec does describe what a regular task identifier is, I'd like someone to point out where and to hear reasons why it is important to dictate a 0..n ranges for task identifiers. Thanks for your time. - Peter Rigsbee par@cray.com PROPOSAL -- OPAQUE TASK IDENTIFIERS 1. Proposal TASK IDENTIFIER All tasks in MPI are identified by unique integer identifiers that are assigned by the system upon which MPI is implemented. The MPI spec does not define the contents of these identifiers and users writing portable applications should not make any decisions about nor perform any operations based on their contents. 2. Discussion a. The major benefit of a 0..n range of identifiers is that users on many systems (most notably those supporting one process per CPU) can write efficient applications without having to deal with groups or virtual topologies. If the relationship between process identifier and process location is well-defined for a particular system, codes can operate directly on identifiers (send to "myid()+1", for example, and have the data sent to a nearest neighbor CPU). b. The drawback is that the above advantage leads programmers away from writing efficient, portable code. A code written using the convenient method above may work great on a few systems, acceptably on another set of systems, and horribly on a third set. It should get the correct answers in each case, but, for most people writing MPI codes, horrible performance would be considered a bug. (This of course assumes that the MPI specs for groups and virtual topologies will allow for efficient, portable applications. If not, we need to fix those specs, and not simply introduce a faster alternative.) c. For an MPP system with one process per CPU, a 0..n assignment is easy and natural. For almost any other system, it will be more complex. For a network implementation, it may require the use of a "process-identifier server" or some sort of globally shared "next identifier" variable. These can be very costly to implement. It also raises ugly questions about where and how to map these identifiers to the information actually locating the task. d. In contrast, the proposal allows an implementation to use the most appropriate and efficient method. It may simply be a Unix PID. It could be a range of 0..n integers. Or it could be a complex encoding of host/PID information that would allow a network implementation to get better performance. It's up to the implementation. e. The MPI spec does not address the fact that some implementations may be provided on top of systems allowing dynamic creation and termination of tasks. A 0..n requirement assumes a static system. What happens when task goes away? There is no longer a sequential list from 0..n, so does this implemenation still adhere to the MPI spec? (If so, one could argue that the 0..n spec is then just a red herring, since one could assign any numbers at all, and claim simply that all missing numbers represented tasks that never started.) With the proposal, the answer is simple -- that identifier no longer represents a task, and the implementation still adheres to the spec. f. The above proposal does not prohibit implementations from assigning 0..n identifiers, or for users of such implementations relying on this assignment. But it makes it clear that such reliance is non-standard and non-portable. 3. Summary The above proposal has a number of benefits, particularly when compared to the 0..n alternative: - allows implementations to use the most appropriate and/or efficient method for assigning task identifiers (including 0..n) - encourages portable programming that does not rely on operations on task identifiers - no confusing semantics for MPI implementations built on top of dynamic systems From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 26 19:25:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23808; Tue, 26 Jan 93 19:25:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29304; Tue, 26 Jan 93 19:24:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 26 Jan 1993 19:24:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29296; Tue, 26 Jan 93 19:24:13 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05866; Tue, 26 Jan 93 18:24:11 CST Date: Tue, 26 Jan 93 18:24:11 CST From: Tony Skjellum Message-Id: <9301270024.AA05866@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, par@teak.cray.com Subject: Re: MPI task identifiers Concerning Peter's proposal... It may have been a mistake to simplify the naming of processes to a single integer when MPI was defined. Commonly, systems like RK, and so on had used pairs of integers on multicomputers and networks before that; the two numbers did have semantic content: the first number, a node number, connoted the location of the process. The second, a PID, could provide an implementation-dependent way to name a process on a processor. Hence, at the low level, specification of WHERE processes ran was permitted, but it was not necessary to restrict how they were named (beyond it being an integer quantity) on a processor. At a low level, it is convenient to be able to control process placement. When processes are created, the user could specify the node and PID on some systems, with either or both picked by the system on some implementations. Such composite information could then be used as part of message passing. nCUBE, for instance, specifies the PID's but does not restrict the locations (nodes are selected by user). The Mosaic system will generally place processes at will, and a message-passing interface for it will have neither option. For scalable spawning of large numbers of identical processes, some mechanism of this form is needed (user specifies collection of process locations, gets back identifier describing these as a group; eg, a common PID). In early systems, and arguably still, it is important to provide low-level control over process placement. Arguably, this need not be done in the message passing interface, but it must be done portably somewhere in the system. Otherwise, it will be difficult to provide guarantees of performance behavior (ie, force one process per processor, if desired; clump processes together to get overlapping capability when desired, etc). Our Zipcode system uses {node,pid} in addressee lists, but these are not used by the programmer directly, in most message classes (eg, 3D logical process grids) but are hidden with further mappings to application-relevant names for processes. In fact, we would like to generalize these addressee lists to include other information besides this naming, but that remains a future-release feature. It does not matter so much to us what convention names processes, but it is interesting to have access to semantic information about processes named by the system (eg, accessors to location, etc). Such information would be used when providing services, like Zipcode, that are higher level. The user might not want them at all for direct use. Forbidding a user code or application layer like Zipcode to take advantage of architecture (albeit transparently to the user code) is a disadvantage. Typically, process placement remains an aspect of architecture we need to control. Just some thoughts... - Tony Skjellum From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 08:37:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02588; Wed, 27 Jan 93 08:37:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01030; Wed, 27 Jan 93 08:37:18 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 08:37:17 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01022; Wed, 27 Jan 93 08:37:16 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17101; Wed, 27 Jan 1993 08:37:15 -0500 Date: Wed, 27 Jan 1993 08:37:15 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301271337.AA17101@rios2.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Task Indentifers I assume that "task" in Peter's proposal are the same as "processes" elsewhere in the MPI discussion. I think there is some confusion over the terms "PID" and "rank". If PIDS do not convey any information to the application then as Peter says: "users writing portable applications should not make any decisions about nor perform any operations based on [task identifiers]." In fact there's almost no point in having task identifiers accessible to an application. Instead applications typically use what was referred to at the Dallas meeting as the "rank", which is a one-to-one mapping of PIDs to the integers 0,1,2,...n-1. The user needs ranks so s/he can say things like, if ( myID() .EQ. 0 ) output(data) PID (or task ID if you like) and rank are synonymous in MPI1 since we could see no reason for distinguishing them. Note that "rank" is essentially an arbitrary enumeration of processes, and contains no information about relative placement of processes on physical processors. A ranking is the same as a 1-D topology. MPI does not currently address how a topology is mapped to physical processors, in fact the whole topic of process creation has so far been avoided. I guess my main question with Peter's proposal is : "If an application shouldn't make any decisions about nor perform any operations based on task IDs, then why does an application even need to know about task IDs?" From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 09:00:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03235; Wed, 27 Jan 93 09:00:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01779; Wed, 27 Jan 93 08:59:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 08:59:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01771; Wed, 27 Jan 93 08:59:40 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA00815 (5.65c/IDA-1.4.4 for ); Wed, 27 Jan 1993 08:59:36 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA13095; Wed, 27 Jan 93 13:59:32 GMT Date: Wed, 27 Jan 93 13:59:32 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301271359.AA13095@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02254; Wed, 27 Jan 93 13:58:07 GMT To: walker@rios2.epm.ornl.gov Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: David Walker's message of Wed, 27 Jan 1993 08:37:15 -0500 <9301271337.AA17101@rios2.epm.ornl.gov> Subject: Opaque Task Identifers Content-Length: 1945 I tend to agree with David here. It is clear that the underlying system level identification of a process can potentially be arbitrarily complex, and will almost certainly require at least a (node,process within node) tuple. However, this is not the way the user wants to think about her program. She wants to think about a set of processors with an obvious mapping onto the positive integers. (Fortran programmers would probably prefer that to read "strictly positive integers" ...). At the user level we shouldn't be exposing the internals of the system's process mapping scheme, since it will likely be different on each machine. (On a net doubtless there will be whole ip addresses kicking around, and so on). I would suggest that we let the user think about a simple enumeration of processors, but stipulate that this is a random enumeration (in other words there is no guarantee whatever that an N to N+1 communication is faster than an N to N+x communication). The system can then use a simple table lookup to translate from the user's process number into the system address. Most of the time this will be necessary anyway (on all systems which need more than this simple numbering scheme), and in any case it's only a single (unsigned) compare followed by one store access. If the user needs to have guarantees of "nearness", then she should use the virtual topologies mechanism to obtain a good process enumeration. If we were feeling mean, then we'd add an implementation recommendation that the intial mapping should be random, so that people get an incentive to use the virtual topology mechanisms even to implement a straight line of processes. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Wed Jan 27 10:29:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05985; Wed, 27 Jan 93 10:29:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05149; Wed, 27 Jan 93 10:28:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 27 Jan 1993 10:28:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05141; Wed, 27 Jan 93 10:28:50 -0500 Message-Id: <9301271528.AA05141@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 7863; Wed, 27 Jan 93 10:28:42 EST Date: Wed, 27 Jan 93 09:50:23 EST From: "Shlomo Kipnis ((914) 945-1281)" To: mpi-pt2pt@cs.utk.edu Subject: Process enumeration The main goal of the MPI standard is the portability of applications between different machines. Linear ordering of process (task) ids in the range 0..n is the simplest and most portable numbering scheme I can think of. As David and Jim state, this numbering scheme is used by many applications, and most machines provide a virtual domain of processes in the range 0..n to be used. Any implementation of MPI, obviously, need to translate this numbering scheme to the physical process/task/thread ids used in the underlying system. However, a portable user's application cannot rely on such a physical numbering scheme. The performance of an application may be very different on different machines, but at least an application that uses MPI is guaranteed to run on different machines (unlike the state of the art today). An interesting feature of the "process groups" idea is that pids in a process group can always be addressed in the range of 0..n using the ranks of processes within the group. By using process groups, one can avoid the issue of noncontiguous pids range which may be caused by particular partitions of the machine, processes leaving or joining an application, running several application codes concurrently under the same job, etc... As currently defined, process groups do not address the issue of an application's topology. However, topologies can be defined on top of process groups. (I hope the topology subcommittee discusses these issues.) To summarize, I think that process ids in the range 0..n are the right approach for ensuring portability of applications. This scheme also addresses the virtual organization of processes in a process group. The virtual-to-physical translation of pids is done in many systems and is typically very fast. Shlomo Kipnis IBM T.J Watson Research Center Yorktown Heights, NY 10598 (914) 945-1281 kipnis@watson.ibm.com From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:12:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08710; Thu, 28 Jan 93 09:12:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02034; Thu, 28 Jan 93 09:11:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:11:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02026; Thu, 28 Jan 93 09:11:42 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA03474; Thu, 28 Jan 93 09:11:39 -0500 Date: Thu, 28 Jan 93 09:11:39 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9301281411.AA03474@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: My two cents worth on process numbering. >To summarize, I think that process ids in the range 0..n are the right >approach for ensuring portability of applications. With the ability to have multiple groups each with processes numbered 0...p then no process can be uniquely identified by a single integer [0,1,...n]. The above process naming scheme will require some group context information down in the point to point routines. It can be explicit as in a (group_id, rank) pair which I prefer because it makes codes more readable and easier to debug, or it can be implicit as in the original MPI draft. Peter Rigsbee is correct in noting that on large MPP systems looking up this mapping of 0-p to real process location and id can be expensive in both time and memory. If the high priests want to be efficient and go fast then Peter's suggestion of having a more sophisticated task naming should be considered seriously. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:18:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08787; Thu, 28 Jan 93 09:18:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02247; Thu, 28 Jan 93 09:18:02 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:18:01 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02239; Thu, 28 Jan 93 09:17:56 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA24455 (5.65c/IDA-1.4.4 for ); Thu, 28 Jan 1993 15:16:56 +0100 Received: by f1neuman.gmd.de id AA15219; Thu, 28 Jan 1993 15:16:01 +0100 Date: Thu, 28 Jan 1993 15:16:01 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9301281416.AA15219@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: MPI task identifiers Cc: gmap10@f1neuman.gmd.de During the last two days there was an interesting discussion about the naming of processes in MPI. I just want to throw some thoughts into the discussion. If we number the processes in a group from 0 to n-1, I see two benefits: 1. The applications programmer immediately knows how to address the other processes in the same group, instead of first having to look up some strange id numbers. Sure, the translation to the underlying (h/w) or network addresses has to be done somewhere, but it's not necessarily the applications programmer who wants to do this. 2. The 0,...,n-1 ranking of processes provides a natural order for collective communications, like gather. Without nicely ordered id numbers, almost every reasonable MPI program would have to use the virtual topologies. If you look at the proposal of the PTOP subcommittee, you will find no assumption about the definition of the process identifiers in a group. All ordering and addressing of processes is provided by the topologies. So, as a proponent of virtual topologies I could be happy with that solution. At the Dallas meeting somebody already suggested that there be always a topology associated with a process group. At group creation this would be just a linear ordering. Since topologies can be overwritten, this default could be changed by the user at any time. Note that this would not mean that the pid numbers are arranged as 0,..,n-1. A process would use the topology look-up functions to get the pids of its neighbors, whereas the pid numbers could have any value (as Peter Ringsbee proposes). I think we have to choose between: a. We force programmers to use the topology functions in virtually all applications. The pid numbers can then have any values. The default topology of a group is a linear ordering. The mapping of processes to processors causes some (usually small) overhead at group creation. b. The pid numbers are numbered as 0,..,n-1. This does not mean the processes are mapped in a kind of nearest-neighbor fashion to the processors. It is rather a (random) enumeration of the processes in a group, in order to get benefits 1. and 2. (see above). If one wants to have a good mapping, one should use the virtual topology functions (which I recommend, of course). On the other hand, this introduces another level of address indirection. I tend towards alternative b., although there are some tricky problems. For example: when a topology is assigned to a group, does this then change the pid numbers? If not, then the buffer structure in collective operations will not be aligned with the topology. On the other hand, changing the pid number of a process looks ugly to me. Perhaps it helps if we avoid the word "pid" alltogether, and only refer to the "rank" of a process, as David Walker suggests. I suggest that at the next Dallas meeting we have a session in which the subcommittees for collective communication and process topologies meet together. To me this seems to be the ideal forum where we can try to solve the problem. Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Thu Jan 28 09:57:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09245; Thu, 28 Jan 93 09:57:56 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04162; Thu, 28 Jan 93 09:57:00 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 28 Jan 1993 09:56:59 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04154; Thu, 28 Jan 93 09:56:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA11324 (5.65c/IDA-1.4.4 for ); Thu, 28 Jan 1993 09:56:52 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA25589; Thu, 28 Jan 93 14:56:47 GMT Date: Thu, 28 Jan 93 14:56:47 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301281456.AA25589@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02620; Thu, 28 Jan 93 14:55:18 GMT To: geist@msr.EPM.ORNL.GOV Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Thu, 28 Jan 93 09:11:39 -0500 <9301281411.AA03474@msr.EPM.ORNL.GOV> Subject: Re Al's two cents worth on process numbering. Content-Length: 2327 (Come on Al give us the whole $1 worth !) Al asserts that > Peter Rigsbee is correct in noting that on large MPP systems looking > up this mapping of 0-p to real process location and id can be > expensive in both time and memory. I disagree that this is necessarily expensive, for the following reasons :- 1) At some level the system has to know the lowest level address, otherwise it cannot send the data. (I know this could be a route, rather than an absolute address, but it comes to the same thing) So it's a cost you have to pay somewhere. 2) In the worst case the cost is constant in time and linear in number of nodes in memory. (Just index into an array of low level addresses). How bad is this really ? Suppose you have a 16K node machine with 128 bit low level addresses (4 32bit words) this occupies 256K bytes, which doesn't seem so bad. 3) You can trade off the store requirement against a larger time cost. Rather than holding all of the destination addresses you implement an address cache (using standard symbol table technology), and fill in the addresses as you use them. (Of course you need a distributed scalable lookup to find them the first time). This then requires store which depends on the working set of targets for communication, and whatever CPU time is taken in the lookup. (Should be logarithmic in number of processors if you use a tree). The issue I'd like to clarify here is what is actually being asked for by Peter ? Is he asking that the user deal with genuine low level addresses (which are potentially of arbitrary size, and certainly different on different vendors machines) ? or Is he asking that users deal with integer identifiers, but these are opaque in the sense that modifying then makes no sense ? The former seems extremely unattractive (especially in Fortran !), and if it's the latter, then you still have to do a translation somewhere to get to the real low level address, so why not let the user have what they really wanted (i.e. 0..N) ? -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 00:33:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29228; Fri, 5 Feb 93 00:33:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27483; Fri, 5 Feb 93 00:31:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 00:31:54 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27473; Fri, 5 Feb 93 00:31:52 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA09343; Fri, 5 Feb 93 00:31:59 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 002919.13027; Fri, 5 Feb 1993 00:29:19 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA09357; Thu, 4 Feb 1993 19:11:45 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA05200; Thu, 4 Feb 93 19:11:44 -0600 Date: Thu, 4 Feb 93 19:11:44 -0600 Message-Id: <9302050111.AA05200@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Composition. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 What is the meaning of these sequential compositions? 1) two non-blocking sends to the same destination? 2) a non-blocking send and a blocking send to the same destination? 3) a non-blocking send and a broadcast? 4) two non-blocking receives? 5) a non-blocking receive and a blocking receive? and so on ... I suspect different machine vendors may have different views. Steven ps. mpi-formal members are refered to mpi-pt2pt. From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 00:33:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29230; Fri, 5 Feb 93 00:33:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27500; Fri, 5 Feb 93 00:32:03 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 00:32:01 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27492; Fri, 5 Feb 93 00:31:59 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AB09405; Fri, 5 Feb 93 00:32:06 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 002916.13009; Fri, 5 Feb 1993 00:29:16 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA09184; Thu, 4 Feb 1993 18:56:01 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA05164; Thu, 4 Feb 93 18:56:00 -0600 Date: Thu, 4 Feb 93 18:56:00 -0600 Message-Id: <9302050056.AA05164@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Concerns. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 I have to agree with Steve Otto and confess more than a little anxiety at this end. It seems to me that we are much too concerned with being all things to all men and I am absolutely convinced that a mix and match philosophy is a recipe for disaster: not that it can't be implemented or specified but purely from the usability point of view. We should be able to provide more wizardly and wise power by providing less not more! We would do better to ensure that the issues we address are those that we all currently understand well or that we have time to defer to assure ourselves that new forms or compositions are at least well concieved. The latter is impossible in the six months we have allotted ourselves. If we really want people to use MPI it has to be straight forward and at least no more complex than existing systems. If we can't decide on a *concise* set of wisely restricted forms for send and recieve, if you really believe we have to provide all forms, then I suggest this effort may be premature. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 05:48:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13673; Fri, 5 Feb 93 05:48:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18966; Fri, 5 Feb 93 05:44:22 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 05:44:21 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18958; Fri, 5 Feb 93 05:44:20 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA24268; Fri, 5 Feb 93 05:44:19 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 054326.7477; Fri, 5 Feb 1993 05:43:26 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA16580; Thu, 4 Feb 1993 23:45:25 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA06410; Thu, 4 Feb 93 23:45:23 -0600 Date: Thu, 4 Feb 93 23:45:23 -0600 Message-Id: <9302050545.AA06410@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 It will probably come as no surprise that I, for one, have no fear of being politically incorrect in these matters ;-) I am not afraid to say that I think at this stage it would be a very good idea to define only process composition and blocking send/recv. The rest comes - more or less - as a construction of the other two. Process ids are simply a many to one channel descriptor - implementable by a single process, non-synchronized communication is simply a process creation, broadcast is a composition of N sends with N receives, and such an interface can be implemented by all the available technologies. In addition this simplification would greatly aid the task of the poor programmer. Where is the innovation in this forum? It seems that we are pandering to the lowest common denominator. Has noone learnt anything in the past 7/8 years? Otherwise we should simply take an existing system (PVM for example, though there are many well designed MP systems) and adopt it as the accepted standard, work from that point and be done with it. The interface efforts of the Unix community (e.g., DCE and sockets) may well out run this one. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 06:05:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17447; Fri, 5 Feb 93 06:05:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20412; Fri, 5 Feb 93 06:04:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 06:04:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20403; Fri, 5 Feb 93 06:04:45 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA27476 (5.65c/IDA-1.4.4); Fri, 5 Feb 1993 06:04:40 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA16231; Fri, 5 Feb 93 11:04:37 GMT Date: Fri, 5 Feb 93 11:04:36 GMT From: jim@meiko.co.uk (Paul Kelly) Message-Id: <9302051104.AA16231@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA06680; Fri, 5 Feb 93 11:02:53 GMT To: zenith@kai.com Cc: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu In-Reply-To: Steven Ericsson Zenith's message of Thu, 4 Feb 93 23:45:23 -0600 <9302050545.AA06410@brisk.kai.com> Subject: Be brave. Be sure. Content-Length: 896 Steve, > I am not afraid to say that I think at this stage it would be a very > good idea to define only process composition and blocking send/recv. > The rest comes - more or less - as a construction of the other two. > Process ids are simply a many to one channel descriptor - > implementable by a single process, non-synchronized communication is > simply a process creation, broadcast is a composition of N sends with > N receives, and such an interface can be implemented by all the > available technologies. You just described occam. One thing I have learnt from the last 10 years is that that's NOT ENOUGH. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 08:24:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02543; Fri, 5 Feb 93 08:24:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27610; Fri, 5 Feb 93 08:23:34 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 08:23:33 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27602; Fri, 5 Feb 93 08:23:31 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA19627; Fri, 5 Feb 93 08:17:03 EST Date: Fri, 5 Feb 93 08:17:03 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302051317.AA19627@cs.wmich.edu> To: mpi-pt2pt@CS.UTK.EDU hi; A good offline note to me from > From: Alan Mainwaring together with some other recent posts, indicates that we are using at least two valid but very different meanings for the term "overlappping communication and computation". -------- 1) Overlapping IO and computation means transfers from the communication processor to and from the node's computation processor memory are carried out by the node's memory system system while a computation is actively running on the node. 2) Overlapping IO and computation means receives and sends requests in a process may be pending while a computation is actively running in a process. -------- Note 2) does not require 1) , some ways 2) could (has) been provided: 2) could be cone by 1) 2) could be done by the OS while the unknowing computation process is interupted and placed in a wait state by the OS when it wants to prefrom a transfer. 2) could be done when the computation process makes any system call. -------- I believe MPI must have 2), that is non-blocking sends and receives. I also think it would pass any straw vote by the widest of margins, I do not have any real feeling about how important 1) currently is. I would expect 1) could become an issue if communications become fast enough to really support distributed virtual memory (that is, commmunication speed approaches computation speed). Then much finer grained parallelism can be applied. It would seem that most interfaces that support direct transfers into memory, rather than a buffer (iovec, stride etc.), could logically also well support 2), if the hardware provides it. It would be best if the interface specifed could use 2) if available, without extra complication. cheers - john From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 13:13:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09267; Fri, 5 Feb 93 13:13:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16600; Fri, 5 Feb 93 13:09:51 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 13:09:50 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16588; Fri, 5 Feb 93 13:09:48 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA25537; Fri, 5 Feb 93 13:09:50 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 130847.19757; Fri, 5 Feb 1993 13:08:47 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA04173; Fri, 5 Feb 1993 10:16:24 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA07622; Fri, 5 Feb 93 10:16:22 -0600 Date: Fri, 5 Feb 93 10:16:22 -0600 Message-Id: <9302051616.AA07622@brisk.kai.com> To: jim@meiko.co.uk Cc: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu In-Reply-To: Paul Kelly's message of Fri, 5 Feb 93 13:20:43 GMT <9302051320.AA16757@hub.meiko.co.uk> Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Thanks Jim (or is it Paul) for the name correction (via private mail)... I know what I'm suggesting - and yes, though I hate to admit it, it is the Occam model more or less. Though you shouldn't allow your distaste for the language to deflect you from the qualities of it's communication model. The inadequacies of that language are many but when it comes to the communication model (CSP) it is fundamental. All I'm suggesting here is that we do not make the other mistake, that of providing so much that it is neither managable by us or our intended audience. Either we are prepared to provide something better than currently exists based on a good understanding of our collective experience with mature concepts or we should start from an existing system, with all its flaws, and work from there. This is not a forum for invention. I should qualify my comments to reiterate that in fact I am looking to this forum to provide a consistent interface for low level implementation in operating systems and code generation. I never want any of my general purpose users using message passing as a means to program machines. And if you gave me just processes and blocking send and receive that *would* be enough. As someone who has written many message passing programs (using Express and other libraries, as well as Occam and CSP), and made it my business over the years to talk to others who have, I have to tell you that the fewer well conceived choices you give me the happier I am and, in my experience, the user is. I just don't believe those people who say we have to provide every form - it is counter to all my experience with users (no doubt, there are users here who will insist on being provided with every variant ;-) - but you will have seen that manifestation before. ). One way to look for evidence to support or denounce this is to undertake an investigation. Find an array of *applications*, not systems tools etc. but "real" applications, and identify the subset of functionality used. My bet is that you will discover the subset is small, the place where non-blocking communication is used is on machines where the process model implementation is inadequate, and that code that does not fit into my simple model is not readily portable. Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 14:07:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10311; Fri, 5 Feb 93 14:07:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20802; Fri, 5 Feb 93 14:07:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 14:07:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20794; Fri, 5 Feb 93 14:07:03 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA11334; Fri, 5 Feb 93 12:06:57 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nKYO9-0016ZKC; Fri, 5 Feb 93 12:06 MST Message-Id: Date: Fri, 5 Feb 93 12:06 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Be brave. Be sure. Steven: While I am impressed that you have enough influence over your users to convince them that they need not see the messaging environment themselves, I must confess that I am not so fortunate. My users are developing their codes in C, F77, and C++. The current trend here at Sandia is towards C++. Our REAL codes (REAL code is one that we make money off of, hence very important :-) are done by those that are still waiting for compilers to do everything. But, while they continue to wait, their livelyhoods depend upon their ability to get the best performance from these MP beasts as possible. Even assembly language is still a useful tool for them, case in point: tuned libraries where interfaces are defined but the library code is certainly not portable. In addition, it is my opinion that there is way too little experience with compiler generated messaging to have such a concept meet the "maturity/experience" criteria upon which we wish to base MPI. As for overlapped messaging, I dare say that EVERY code here at Sandia makes use of overlapped messaging. At the very least, the programmers want to have multiple I/O channels active simultaneously, which is at least possible on our two 1024-processor nCUBE 2 hypercubes. Furthermore, it is clear that my users are already digging into what possibilities for performance enhancements await them in our new Paragon. These are general users! Not just the wizards. Again, the "food chain" is quite clear. We do want simply understood features for the "novice" or "don't care" people. However, we must have what the experienced general users want. Furthermore, we must have the "buy-in" of the wizards. To make this more clear, I use the term wizard to refer to application types doing real, dollar/pound income-oriented work. That is, they are not of the CS lunatic fringe type, but are real-world applications people. All of this is just to state that the standard should be primarily for human use, not for compilers only. As has been voiced by others, I strongly suspect that overlapped I/O and computation is a feature that general users will (AND DO) want. Furthermore, I believe that "our collective experience with mature concepts" can tackle this problem in a very successful manner. Stephen From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 17:00:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13516; Fri, 5 Feb 93 17:00:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01026; Fri, 5 Feb 93 16:59:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 16:59:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01011; Fri, 5 Feb 93 16:59:03 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00846; Fri, 5 Feb 93 15:58:52 CST Date: Fri, 5 Feb 93 15:58:52 CST From: Tony Skjellum Message-Id: <9302052158.AA00846@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, srwheat@cs.sandia.gov Subject: Re: Be brave. Be sure. To follow up on Stephen's comment, I do not believe that compilers would make good use of this standard. Active-message-level programming seems a much better level for standardizing compiler interfaces. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 18:13:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14881; Fri, 5 Feb 93 18:13:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04250; Fri, 5 Feb 93 18:13:18 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 18:13:17 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04238; Fri, 5 Feb 93 18:13:15 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA24078; Fri, 5 Feb 93 18:13:19 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 181248.24121; Fri, 5 Feb 1993 18:12:48 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701) id AA01338; Fri, 5 Feb 1993 17:10:16 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA08601; Fri, 5 Feb 93 17:10:15 -0600 Date: Fri, 5 Feb 93 17:10:15 -0600 Message-Id: <9302052310.AA08601@brisk.kai.com> To: mpi-formal@cs.utk.edu, mpi-pt2pt@cs.utk.edu Cc: In-Reply-To: Steven Ericsson Zenith's message of Fri, 5 Feb 93 10:16:22 EST Subject: Be brave. Be sure. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 From: srwheat@cs.sandia.gov (Stephen R. Wheat) ... But, while they continue to wait, their livelyhoods depend upon their ability to get the best performance from these MP beasts as possible. Even assembly language is still a useful tool for them, case in point: tuned libraries where interfaces are defined but the library code is certainly not portable. The fact that your users are programming using message passing suggests that they are certainly also prepared to use assembler when necessary. The reason they use MP is the same reason they use assembler at times: there is no higher level way currently to achieve what they want with the performance they require on the machines they are using. ;-) Evenso, when I've asked people (physicists and so on) with some experience programming such machines this question they have confirmed that they use MP as reluctantly as they use assembler - it's just that sometimes it's unavoidable. Some of those people are at Sandia. I certainly would not want to deprive them of their lively hood and there are exceptions - of course. Some people really enjoy the challenge of writing MP code, just as they enjoy the challenge of writing assembler - me too at times ;-) But the pro's and con's of message passing as a programming model, and the alternatives, are not the issues here. While we may have different motivations we do all understand the importance of MP at some level. Certainly it should be usable by "users" in the absence of anything better. In fact I'd contend that my recent calls serve those that do have to use it well - wizard or not. I am not calling for us to take power away from users - on the contrary - I'm calling for greater utility and usability through simplicity. I am calling for us to focus on things we understand well and to avoid invention. I do not feel that such simplicity will fail to serve wizards - your implication is that wizards are only attracted to complexity: this surely is not true. In addition, it is my opinion that there is way too little experience with compiler generated messaging to have such a concept meet the "maturity/experience" criteria upon which we wish to base MPI. No argument here - but then this isn't an objective of MPI. As for overlapped messaging, I dare say that EVERY code here ... I am not saying that non-synchronizing send is a "bad thing" - I am saying there is a simpler way of looking at these things. Define the fundamentals and work from there. We have been arguing about the nuances of buffer behavior without getting a real feel for our base objectives. You can build all you want - if you really want it - with fundamental and well understood components - it makes a great opportunity for third party software vendors like Parasoft and DSL. Buffering characteristics are likely to remain a choice based on hardware design and we should leave such things alone in our interface design. Now I understand that most machine vendors have a variety of approaches - and that is exactly why we should should be careful. But I cannot believe the solution for MPI is the union of those approaches - though it may be some component of the intersection. Isn't our objective to provide a Message Passing Interface that will allow people who want to write portable (or transportable) MP code to do so? And to provide for compiler writers and architects - people like me - who are just looking for a consistent interface on MP machines? Doesn't usability have a right to be at the fore of our concerns? If this is the case then we do need some well conceived abstraction or we must adopt an existing system and refine it. When it comes right down to it, I really don't mind which. I would like though to take a program written in MPI and simply transform it into another MPI program for execution on a machine with different characteristics. What I've seen so far is going to make it very very hard to do that. I'm only likely to want to do literal translation on a machine identical to the original target - it would be nice to permit simple transformations and checking (for deadlock for example) ... wouldn't it? Incidently, in addition to my architects point of view I do still design message passing systems and program them - so I hope I qualify as a user too. ;-) Steven From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 22:04:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17274; Fri, 5 Feb 93 22:04:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13646; Fri, 5 Feb 93 22:04:22 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 22:04:21 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13638; Fri, 5 Feb 93 22:04:19 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Fri, 5 Feb 1993 22:04:17 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA05848; Fri, 5 Feb 1993 22:04:16 -0500 Date: Fri, 5 Feb 1993 22:04:16 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302060304.AA05848@YOGI.NA.CS.YALE.EDU> To: mpi-pt2pt@cs.utk.edu Subject: compiler target (was Be brave. Be sure.) Tony says: To follow up on Stephen's comment, I do not believe that compilers would make good use of this standard. Active-message-level programming seems a much better level for standardizing compiler interfaces. There are some compiler folks who might disagree with you. The group at Rice is working on generating message-passing, as well as the group at Syracuse, U. Maryland, and a few other places. Perhaps you could be more specific as to why you think that compilers cannot make good use of this standard? -scott berryman From owner-mpi-pt2pt@CS.UTK.EDU Fri Feb 5 23:21:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17474; Fri, 5 Feb 93 23:21:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15951; Fri, 5 Feb 93 23:20:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Feb 1993 23:20:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15940; Fri, 5 Feb 93 23:20:42 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01319; Fri, 5 Feb 93 22:20:29 CST Date: Fri, 5 Feb 93 22:20:29 CST From: Tony Skjellum Message-Id: <9302060420.AA01319@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, berryman-harry@CS.YALE.EDU Subject: Re: compiler target (was Be brave. Be sure.) Cc: bert@athena.llnl.gov, tsw@mpci.llnl.gov Yes Scott, I can be more specific. I think that MPI[1] is not interesting for compiler writers. Active messages are finer grain message passing, which have lower overhead. There is naturally a larger overhead in a full system call like a send()/recv() pair. If the compiler knows it wants to drop a single real number on a process, it could use a finer grain mechanism. Compilers would be good at cacheing limited open channel resources, such as supported at the low-level by CMMD. Thus, they could take advantage of not having to build up and tear down connections between processes, as is needed for send()/recv(). A good example of this is neighbor communication for boundary sharing in domain decomposition. They could take advantage of special properties of the network(s) of a system, that are lost when a robust send()/recv() mechanism is provided, each of which implies a library [if not system] call. The overhead of a subroutine call send()/recv() implies a minimal granularity, because stack frames must be assembled, etc. The compiler itself could be responsible for the non-deadlock properties of the communication, rather than insisting that the messaging calls provide this capability. Furthermore, compilers could make use of the "force semantics" per Intel, but still provide the user with a guarantee of correctness (ie, use the cheapest calls, but build their own appropriate protocols, but as weak as possible). Mixing communication and computation is another example of where the messaging libraries with send/recv semantics are too heavy duty, compared to active messages. A compiler might want to add two values and store them remotely. It does not need a full stack of messages, with rendezvous to accomplish this. I think the people who are building compilers to generate straight message- passing code should read the work from Berkeley and TMC on active messages (Culler et al). and build compilers that generate code using these excellent ideas. Performance will be superior. Message-passing libraries are for people who write explicit message-passing code without the benefit of compiler support. They need guarantees like non deadlock of messages, they need queues so messages are there when you look for them, and so on. Compilers can manage these things, and avoid the necessary compromises made to present robust message passing to users through send/recv. It is difficult to cope with the many sequential calls that occur in message passing code, without the need to build things like send+recv, in order to reduce overhead. Compilers could optimize away such overheads if they controlled the messaging resource directly (ie, stamp out unneeded overheads). Also, they could inline conversions for heterogeneity, and reduce the number of copies of messages by driving the network as they convert data (or just drive the network as they build parts of a message). Furthermore, they could balance the work done by sender and receiver based on the relative power of the sender and receiver to do conversions. This gets quite complex for hand-written message passing code. Active messaging can also be standardized (eg, create MPI-AM) to provide portable code to machines which are message passing, and which are hierarchically shared memory. So, one could build a model that unifies machines like the Paragon, CM-5, and Cray Tera, for the purpose of compiler code generation, though these are different machines, to be sure. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 08:54:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26068; Mon, 8 Feb 93 08:54:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20830; Mon, 8 Feb 93 08:52:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 08:52:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20822; Mon, 8 Feb 93 08:52:39 -0500 Date: Mon, 8 Feb 93 13:52:17 GMT Message-Id: <16939.9302081352@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.)) To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi Here comes my 5p worth on the subject of buffered and unbuffered comms. For completeness: by unbuffered I mean that the sender blocks until the message has been (or certainly will be) copied into the space of the receiver, a la occam for example; by buffered comms I mean that the message is copied away into a system buffer somewhere and the sender returns, a la PVM for example. It seems to me that there are a few different issues in this subject which discussions may have touched on. a) Ease of programming. I don't think anyone can really disagree that many programmers will report that they find buffered comms easier to use. These users have been fortunate enough not to have come up against the boundedness of buffering provided with the system. With high probability they are programming (or adapting) applications by inserting message passing calls directly into the application source. b) Portability of programs using MPI between different machines. [Here I give away my bias.] It is very difficult to make statements about portability (and reliability/correctness) of programs which use buffered messages and rely on system storage capacity and structure thereof. This becomes particularly difficult when the application makes use of substantive libraries, which themselves are written using buffered comms and place requirements on system buffering. It can just become to difficult to work out how much buffering you need. We played with a system which allowed the programmer to configure the system buffering, and this feature was only used (properly) by high priests. In my work I will only be able to use MPI if I can write substantive libraries in the confidence that they will not be subject to failure due to running out of such buffer space. Therefore we use only unbuffered message passing with a mixture of blocking/nonblocking (like Intel NX isend/irecv/msgwait) calls. Therefore my bias :-) c) Portability of existing applications (using existing message passing interfaces) to MPI. This is a different subject to (c). Following the discussion it seems that an argument in favour of adopting buffered comms is the number of existing programs (eg, lotsa programs written using Express) which use unbuffered comms. It has been my experience that migration of applications between different, broadly similar, message passing interfaces is not so difficult, except for this point. In a nutshell, the surgery you have to perform to move programs/libraries between an interface that forces buffered comms and one which forces unbuffered comms and blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. Given the above, I come to the conclusions: i) MPI must contain unbuffered communications with blocking/nonblocking (irecv/isend/msgwait kinda thing) calls, for reliability and portability. ii) If a goal of MPI is that existing applications using message passing interfaces (eg Express, PVM) should easily port to MPI, then MPI must also contain buffered comms. This seems to be a matter for the full committee, hence I have crossposted. Incidentally, to pitch in 2ps on some other lines of discussion: * I can see no particular benefit in allowing a buffered snd/rcv match an unbuffered rcv/snd. I can see considerable inconvenience in forbidding a blocking send/recv to match with a nonblocking recv/send. * Yes, we do try to overlap communications with calculation. (Sometimes it doesnt buy you, sometimes it does. Sometimes we have specifically needed to avoid such overlap in order to maximise performance, but that was a weird one :-) Just as important, we frequently "overlap" communication with communication (ie, use nonblocking calls) to avoid deadlock. * Please, oh please, let MPI decide that communications should be adressed using a rank relative to a group/context (0 ... GroupSize - 1) or (1 ... GroupSize). We do this all the time, and its very very convenient. In fact, when we can't do this we end up having to create arrays of the task identifiers - we end up doing it ourselves anyway. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 09:25:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27616; Mon, 8 Feb 93 09:25:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22596; Mon, 8 Feb 93 09:24:49 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 09:24:46 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22575; Mon, 8 Feb 93 09:24:40 -0500 Message-Id: <9302081424.AA22575@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9817; Mon, 08 Feb 93 09:24:38 EST Date: Mon, 8 Feb 93 09:24:37 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 7422; Mon, 8 Feb 1993 09:19:14 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R2) with TCP; Mon, 08 Feb 93 09:19:11 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930113) id AA24152; Mon, 8 Feb 1993 09:19:15 -0500 Date: Mon, 8 Feb 1993 09:19:15 -0500 From: snir@snir.watson.ibm.com (Marc Snir) Message-Id: <9302081419.AA24152@snir.watson.ibm.com> To: snir@yktvmv.vnet.ibm.com \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. We have not discussed in our last meeting the choice of a syntax for the various communication operations. In particular, there have been no discussion on the choice of function names. There has been no decision whether its preferable to use distinct function names or use additional parameters to distinguish between different variants of the same operation. There has been no discussion whether parameters should be kept separated or bundled in one ``capsule'' that is passed as one parameter. There has been no discussion whether we prefer, in Fortran, functions or subroutines. Thus, the choice of a particular syntax in this draft is merely for the sake of exposition, and is subject to change. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. The current draft borrows heavily on the proposal of Gropp and Lusk for a Multi-Level Message Passing Interface and on my previous outline. \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} \paragraph*{Untyped Data} The simplest (and most used) message data is untyped. Such data consists of a sequence of bytes. Untyped messages can be used to transfer arbitrary typed values between processors with identical data representations; thus, untyped messages are sufficient in a homogeneous environment, where no type conversion is required. \discuss{ Is it legitimate to require that message length be a multiple of 4 or 8? -- or should we require that any number of bytes be OK, with communication perhaps faster for multiples of 4 or 8?} \paragraph*{Typed data} \missing{Need to add here typed messages.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The id of the sending task \item[dest] The id of the receiving task \item[tag] User defined \item[context] \end{description} All these fields are of type integer. The range of valid values for the {\bf sender} and {\bf receiver} fields is {\tt 0 ... number\_of\_tasks-1}, where {\tt number\_of\_tasks} is the number of tasks currently participating in the computation (i.e., both fields must include a valid task number). The ranges of valid values for {\tt tag} and {\tt context} are implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. \discuss{ Users would prefer to be able to use any integer as {\tt tag} or {\tt context}. Vendors may prefer small ranges (e.g. 8 or 16 bits) so as to use fast tag and/or context matching hardware. A defined arithmetic type could be used in C for {\tt tag} and {\tt context}. } The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The {\tt context} field is used to distinguish messages produced by different applications or different libraries. Consider the case where an application calls in parallel on all executing nodes a library function. One needs to make sure that a message generated by the library code will not be consumed by the application and vice-versa. To do so, one can \begin{enumerate} \item Synchronize all nodes before and after the call, or \item Follow a programming convention that prevents such confusion (e.g., make sure that the library code generate messages with tags that are distinct from the tags used by the application, and accepts only messages with correct tags). \end{enumerate} The first solution may lead to less efficient code (it prevents a ``loosely synchronous'' library call). The second solution prevents modular code development. The use of a {\tt context} field, together with the context setting mechanisms described in Section~\ref{sec:context}, solve this problem. Finally, we would like to note that the actual mechanism used to associate data with envelope is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit in some context, and may not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \paragraph*{Untyped Messages} There are three kinds of buffers for untyped messages. \begin{description} \item[contiguous buffer] A sequence of contiguous bytes in memory, specified by \begin{description} \item[buf] Initial address \item[len] Number of bytes \end{description} \item[constant stride buffer] A sequence of blocks equally spaced and equally sized blocks, specified by \begin{description} \item[buf] Initial address \item[numblk] Number of blocks \item[lenblk] Number of bytes in each block \item[stride] Number of bytes between the start of each block \end{description} Note that a constant stride buffer becomes a contiguous buffer when {\tt stride = lenblk}. \item[general scatter gather] A sequence of constant stride buffers, specified by a {\bf data\_vector} of the form {\tt (number\_of\_items, buf1, numblk1, lenblk1, stride1, ... bufn, numblkn, lenblkn, striden)}. \end{description} \discuss{ Since Fortran 77 does not have a pointer type, an MPI library that is strictly Fortran 77 compliant would need a different send routine for each possible type of the variable {\tt buf}. This is not very practical. I suggest to ignore the issue and rely on the fact that Fortran 77 compilers and linkers are very unlikely to do type checking across separately compiled modules and are passing routine parameters by reference. The C binding can use pointers, and the Fortran 90 binding can use overloaded subroutines (for basic types). Data vectors are more of a problem, since we now need pointers as components of the data vector. The Posix Fortran 77 Language Interface IEEE standard does not provide a general enough mechanism for that purpose. We might restrict the use of data vectors to C. (Of course, sophisticated programmers would be able to call C routines from Fortran, and nonstandard solutions might also be provided.) Or, we might require that all buffer addresses are integer displacements with respect to the first address, in which case all entries are integer. Is it acceptable to have a (small) fixed upper bound on the length of data vectors? } \paragraph*{Typed Buffers} \missing{Data buffers for typed messages.} \subsection{Receive Criteria} The selection of a message by a receive operation is done according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \discuss{ The DONTCARE value can be a fixed value (e.g. -1), and/or a named constant. It might be useful to have the option to require exact size match of incoming message and receive buffer. Some would also like to see truncation supported. } \subsection{Communication Handles} One can consider (most) communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). A handle is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The handle, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., a handle needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all handles are freed at program termination, by default). A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Handle Creation} A handle for a send operation is created by a call to {\bf MPI\_INIT\_SEND}, followed by a set of calls that set the various parameters associated with the handle. The handle cannot be used until all parameters have been set. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating a handle for a receive operation. Handle creation is a local operation that need not involve communication with a remote process. {\bf handle = MPI\_INIT\_SEND (dest, tag, context)} \\ Create a send handle. All three parameters must be specified. {\bf handle = MPI\_INIT\_RECV (source, tag, context)} \\ Create a send handle. DONTCARE values may be used for the first two parameters. See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ An alternative design, proposed by Gropp and Lusk, is to have each newly created handle associated with default parameters that can be later modified. In their proposal, too, all modifications must occur before the first use.} \subsubsection{Buffer specification} The {\bf MPI\_BUFFER} function associates a buffer with a handle. {\bf MPI\_BUFFER(handle, buftype, params)}\\ \begin{tabbing} {\bf buftype} \ \ \ \ \ \ \ \ \ \ \ \= {\bf params} \\ CONTIGUOUS \> buf, len \\ STRIDED \> buf, numblk, lenblk, stride \\ GENERAL \> data\_vector \end{tabbing} See Section~\ref{subsec:buffers} for a description of these buffer types and parameters. The {\tt buftype} parameter is an integer, with named constants provided for the three values. The {\tt params} parameter list consists of an address (pointer) for {\tt buf}, and a one-dimensional array of integers to specify the remaining values. See Section~\ref{subsec:buffers} for a discussion of data vectors. \missing{typed buffers} \discuss{ Some may want specific values for {\tt buftype}, e.g. CONTIGUOUS =0, STRIDED=1, GENERAL=2. } \subsubsection{Start Mode} The {\bf MPI\_START\_MODE} function selects a mode for starting a communication operation. {\bf MPI\_START\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation may start whether or not a matching communication operation has been posted. \item[READY] The operation my start only if a matching communication operation has been posted. \end{description} Thus, a {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. A {\bf ready receive} can start only if a matching send is already posted; otherwise the operation is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and READY=1. We discussed only ready send, not ready receive. Ready send is useful when message passing is implemented in a ``push'' style, where the sender moves the message to the receiver (common case). Ready receive would be useful when message passing is implemented in a ``pull'' style, where the receiver moves the data from the sender. } \subsubsection{Completion Mode} \label{subsec:completion} The {\bf MPI\_COMPLETE\_MODE} function selects a mode for completing a communication operation. {\bf MPI\_COMPLETE\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation completes locally, irrespective of the completion of the matching operation. \item[SYNCHRONOUS] The operation completes only if the matching operation completes too. \end{description} A regular send can complete as soon as the message has been composed: data has been copied out of the send buffer and the values used for the envelope have been stored. The sender may now update the send buffer. A regular send can complete before the matching receive has completed. A regular receive can complete as soon as the message has been consumed: data is available in the receive buffer and all values returned by the send complete operation are available. The receiver may now access the receive buffer. A regular receive may complete before the matching send has completed (It may seem strange, at first blush, that a receive may complete before the matching send completes. However, one should keep in mind that an operation completes only when the {\tt complete} suboperation terminates. Thus, even though data has been copied out and the message was sent, a send has not not yet completed if the {\tt send complete} suboperation has not yet been invoked. Informally, the send (receive) completes only when the sender (receiver) knows the operation has completed.) A synchronous communication operation completes only if the matching operation completes too. Thus, a synchronous send can complete as soon as the all operations preceding the completion of the matching receive have terminated. A synchronous receive may terminate as soon as the message has been consumed and all operations preceding the completion of the matching send have terminated. Informally, the completion of the matching send and receive operations are synchronized. Matching communication operations must have the same completion mode. I.e., either both send and receive are regular or both are synchronous. A program where a regular receive matches a synchronous send (or vice versa) is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and SYNCHRONOUS=1. } \subsubsection{Communication Start} {\bf MPI\_START(handle)} \\ The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \discuss{ Some might prefer two distinct functions: {\bf MPI\_SEND\_START} and {\bf MPI\_RECV\_START}. } \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf MPI\_COMPLETE\_SEND ( handle)} \\ A call to {\bf MPI\_COMPLETE\_SEND} returns when the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). {\bf MPI\_COMPLETE\_RECV ( handle, len, source, tag)} \\ A call to {\bf MPI\_COMPLETE\_RECV} returns when the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). The call returns the number of bytes received, and the source and tag of the message received. {\bf MPI\_CHECK\_SEND ( handle, flag)} \\ A call to {\bf MPI\_CHECK\_SEND} returns {\tt flag=true} if the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). It returns {\tt flag=false}, otherwise. A successful return of {\bf MPI\_CHECK\_SEND} has the same semantics as a return of {\bf MPI\_COMPLETE\_SEND}. {\bf MPI\_CHECK\_RECV ( handle, flag, len, source, tag)} \\ A call to {\bf MPI\_CHECK\_RECV} returns {\tt flag=true} if the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). In such case the call returns the number of bytes received, and the source and tag of the message received. The call returns {\tt flag=false}, otherwise. In such case, the return values of {\tt len}, {\tt source} and {\tt tag} are undefined. Implementation notes: A call to {\tt MPI\_COMPLETE} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. A call to {\tt MPI\_COMPLETE\_SEND(handle)} has the same semantics as a busy loop of the form \begin{verbatim} repeat MPI_CHECK_SEND(handle, flag) until(flag=true) \end{verbatim} and similarly for receives. The use of a blocking receive operation ({\tt MPI\_COMPLETE}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_CHECK}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_CHECK} is for that operation to return as soon as possible. Note that it is correct, but inefficient, to implement {\tt MPI\_CHECK} via a call to {\tt MPI\_COMPLETE}, in which case, {\tt MPI\_CHECK} always returns {\tt true}. \discuss{ One might use {\tt len=-1} to indicate unsuccessful return of {\tt MPI\_CHECK\_RECV} } \subsection{Blocking Communication} Blocking send and receive operations combine all four suboperations into one call. The operation returns only when the communication completes and no communication handle persists after the call completed. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf send \\ \bf recv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. {\bf MPI\_SEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle,len,source,tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, soure, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} Note that all receive calls return the actual length, tag and source of the received message. This, the {\tt source} and {\tt tag} parameters are INOUT. The {\tt len} parameter is INOUT for contiguous receive buffers, OUT for strided and general receive buffers. {\bf Implementation note:} While these 24 functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \discuss{ Not all 24 functions are equally important (ready-synchronous is somewhat esoteric). We can decide to organize the functins so that the most important are in a ``core subset''. But (i) users will use whatever you provide, and will find uses for what you believed to be useless; (ii) there is a virtue in being systematic -- no need to remember what works and what does not; (iii) what is not important can be implemented using multiple calls to the basic functions. The need for an additional {\tt len} parameter for strided and general receives is somewhat annoying. Suggestions? } \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call, and the last two suboperations ({\tt COMPLETE} or {\tt CHECK} and {\tt FREE}) into another suboperation. Thus a nonblocking send or receive consists of two suboperations: One that starts the communication, and one that completes the operation and free any associated resources. No communication handle persists after these two suboperations are executed. \subsubsection{Initiation} We use the following naming convention for the first suboperation of a nonblocking communication: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf isend \\ \bf irecv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. The letter {\bf i} in {\bf isend} and {\bf irecv} indicates that the operation is ``immediate'' and that the call may return before the operation is completed. We thus have 12 immediate send operations and 12 immediate receive operations. {\bf MPI\_ISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_ISEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} \subsubsection{Completion} A communication operation can be completed in two ways: (i) By a call to {\tt MPI\_WAIT}. Such call returns when the corresponding operation has completed. (ii) by a call to {\tt MPI\_STATUS}. Such call returns successfully if the communication operation has completed; it may also return unsuccessfully if the operation has not completed. See Section~\ref{subsec:complete_ops} for a discussion of the two completion operation types. {\bf MPI\_WAIT\_SEND(handle)} \\ is \begin{verbatim} MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_WAIT\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_COMPLETE_RECV(handle, len, source, type) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_SEND(handle, flag)} \\ is \begin{verbatim} MPI_CHECK_SEND(handle, flag) if (flag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_CHECK_RECV(handle, flag, len, source, type) if (flag) MPI_FREE(handle) \end{verbatim} \paragraph*{Multiple Receives} It is convenient to be able to wait for any successful receive in a set, rather than having to wait for specific message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ {\bf list\_of\_handles} is an array containing $(\rm n, handle_1 , ... , handle_n)$. Blocks until one of the operations associated with handles in the array has completed. Returns the index of that handle in the array, and the length, source and tag of the received message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], len, source, tag); index = 1} || ... || {MPI_WAIT_RECV (handle[n], len, source, tag); index = n} \end{verbatim} (``$||$'' indicates alternation; one of the alternatives is chosen, nondeterministically.) \discuss{ An alternative definition of {\tt WAITANY} is to require that the first available message in the list is received. I.e. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ is \\ {\tt for(index=1; i$<=$n; i++) \{ \\ \ \ \ MPI\_STATUS\_RECV (handle[index], flag, len, source, tag); \\ \ \ \ if(flag) break; \\ \ \ \ \} } A difference would occur in the case where two receives in the list match the same message; in the original definition any of the two may succeed, whereas in the amended definition only the first one can succeed. This additional determinism, may, on the other hand, restrict concurrency in the implementation of {\tt WAITANY}. It would be nice to have a WAITALL that waits for the completion of all operations listed in the list\_of\_handles. The most useful option might be, in fact, ``wait for the completion of ALL pending operations in the current context'', which could be specified by a ``wildcard'' list\_of\_handles (e.g., a length zero list). } \paragraph*{Probe} It is convenient for a receiver to be able to check for pending messages that can be received, before executing a receive operation. This, for example, would allow the receiver to allocate a receive buffer of the right size, when the size of incoming messages is not known ahead of time. The {\bf PROBE} call specifies an envelope pattern. It returns a handle to an available message with an envelope that matches the pattern, and that does not match any pending receive. It also returns the actual values of {\tt len} {\tt source} and {\tt tag}. The message is now ``locked'' and can be received only by an operation that uses the handle returned. If there is no matching message, then {\tt probe} returns with a NULL handle. {\bf MPI\_PROBE(len, source, tag, context, handle)} \\ If there is a matching message that is not matched by a previously posted nonblocking receive, returns message length, the actual values of source and tag, and a handle that must be used to receive the message. Otherwise, returns a handle with value {\bf NULL} (NULL is a named constant). {\bf MPI\_PRECV ( buf, len, handle)} \\ Nonblocking receive of probed message into a contiguous buffer. {\bf MPI\_PRECVS ( buf, numblk, lenblk, stride, handle)} \\ Nonblocking receive of probed message into a fixed stride buffer. {\bf MPI\_PRECVG ( datavector, handle)} \\ Nonblocking receive of probed message into a buffer specified by a data vector. \discuss{ May want a specific value for NULL, e.g., NULL=0 May need to return more than length, for typed messages. } \subsection{Correctness} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active handles. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item Each process can create at least one communication handle. I.e., if a process executes an {\tt MPI\_INIT\_SEND} or {\tt MPI\_INIT\_RECV} operation, and has no other active handle, then the operation eventually succeeds. It is highly desirable to have generous bounds on the number of concurrently active communication handles each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active handle. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE\_SEND} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE\_RECV} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ids 1 and 2) The following program is safe, and should always succeed. \begin{verbatim} IF (GETID() == 1) { MPI_SEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_RECV(buf=recvbuf, len=1000, source=1, tag=0); MPI_SEND(buf=sendbuf, len=1000, dest=1, type=0); } \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} IF (GETID() == 1) { MPI_SSEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SSEND(buf=sendbuf, len=1000, dest=1, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=1, tag=0); } \end{verbatim} The send operation of the 1st process can complete only if the matching receive of the second processor is executed; the send operation of the second processor can complete only if the matching receive of the first processor is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} IF (GETID() == 1) THEN { MPI_SEND(buf=sendbuf, len=1000000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SEND(buf=sendbuf, len=1000000, dest=1, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=1, tag=0); } \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer 1 MgB of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \subsection{Error Handling} \subsubsection{Communication Errors} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. The other errors can be divided in two classes: \begin{description} \item[resource errors] Errors that occur in an unsafe programs because of limitations on the resources of the communication subsystem, but would not occur if these subsystem had unbounded resources (buffer overflow, no available handles, etc.) \item[program errors] Errors that result in an erroneous program that will always fail. (send with no matching receive, send to nonexistent destination, etc.) \end{description} Program errors always result in an exception, if detected. MPI will provide two mechanisms for handling resource errors: In {\em system mode} each MPI call may return with a return code that indicates that a resource error was detected; no exception occur, and the program may recover in normal execution mode. In {\em user mode} a detected resource error causes an exception. \discuss{ Program errors cannot always be distinguished from resource errors. Have to decide how to handle doubtful cases in system mode. Need to change the syntax so that all return values (error code excepted) are returned via parameters. It's reasonable to restrict system mode to C and C++ bindings. } \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 09:37:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27880; Mon, 8 Feb 93 09:37:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23139; Mon, 8 Feb 93 09:37:24 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 09:37:22 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23131; Mon, 8 Feb 93 09:37:18 -0500 Message-Id: <9302081437.AA23131@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9955; Mon, 08 Feb 93 09:37:17 EST Date: Mon, 8 Feb 93 09:37:16 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU Some (all?) of you may have received a truncated version of this doc at my previous attempt. Trying again. \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. We have not discussed in our last meeting the choice of a syntax for the various communication operations. In particular, there have been no discussion on the choice of function names. There has been no decision whether its preferable to use distinct function names or use additional parameters to distinguish between different variants of the same operation. There has been no discussion whether parameters should be kept separated or bundled in one ``capsule'' that is passed as one parameter. There has been no discussion whether we prefer, in Fortran, functions or subroutines. Thus, the choice of a particular syntax in this draft is merely for the sake of exposition, and is subject to change. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. The current draft borrows heavily on the proposal of Gropp and Lusk for a Multi-Level Message Passing Interface and on my previous outline. \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} \paragraph*{Untyped Data} The simplest (and most used) message data is untyped. Such data consists of a sequence of bytes. Untyped messages can be used to transfer arbitrary typed values between processors with identical data representations; thus, untyped messages are sufficient in a homogeneous environment, where no type conversion is required. \discuss{ Is it legitimate to require that message length be a multiple of 4 or 8? -- or should we require that any number of bytes be OK, with communication perhaps faster for multiples of 4 or 8?} \paragraph*{Typed data} \missing{Need to add here typed messages.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The id of the sending task \item[dest] The id of the receiving task \item[tag] User defined \item[context] \end{description} All these fields are of type integer. The range of valid values for the {\bf sender} and {\bf receiver} fields is {\tt 0 ... number\_of\_tasks-1}, where {\tt number\_of\_tasks} is the number of tasks currently participating in the computation (i.e., both fields must include a valid task number). The ranges of valid values for {\tt tag} and {\tt context} are implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. \discuss{ Users would prefer to be able to use any integer as {\tt tag} or {\tt context}. Vendors may prefer small ranges (e.g. 8 or 16 bits) so as to use fast tag and/or context matching hardware. A defined arithmetic type could be used in C for {\tt tag} and {\tt context}. } The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The {\tt context} field is used to distinguish messages produced by different applications or different libraries. Consider the case where an application calls in parallel on all executing nodes a library function. One needs to make sure that a message generated by the library code will not be consumed by the application and vice-versa. To do so, one can \begin{enumerate} \item Synchronize all nodes before and after the call, or \item Follow a programming convention that prevents such confusion (e.g., make sure that the library code generate messages with tags that are distinct from the tags used by the application, and accepts only messages with correct tags). \end{enumerate} The first solution may lead to less efficient code (it prevents a ``loosely synchronous'' library call). The second solution prevents modular code development. The use of a {\tt context} field, together with the context setting mechanisms described in Section~\ref{sec:context}, solve this problem. Finally, we would like to note that the actual mechanism used to associate data with envelope is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit in some context, and may not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \paragraph*{Untyped Messages} There are three kinds of buffers for untyped messages. \begin{description} \item[contiguous buffer] A sequence of contiguous bytes in memory, specified by \begin{description} \item[buf] Initial address \item[len] Number of bytes \end{description} \item[constant stride buffer] A sequence of blocks equally spaced and equally sized blocks, specified by \begin{description} \item[buf] Initial address \item[numblk] Number of blocks \item[lenblk] Number of bytes in each block \item[stride] Number of bytes between the start of each block \end{description} Note that a constant stride buffer becomes a contiguous buffer when {\tt stride = lenblk}. \item[general scatter gather] A sequence of constant stride buffers, specified by a {\bf data\_vector} of the form {\tt (number\_of\_items, buf1, numblk1, lenblk1, stride1, ... bufn, numblkn, lenblkn, striden)}. \end{description} \discuss{ Since Fortran 77 does not have a pointer type, an MPI library that is strictly Fortran 77 compliant would need a different send routine for each possible type of the variable {\tt buf}. This is not very practical. I suggest to ignore the issue and rely on the fact that Fortran 77 compilers and linkers are very unlikely to do type checking across separately compiled modules and are passing routine parameters by reference. The C binding can use pointers, and the Fortran 90 binding can use overloaded subroutines (for basic types). Data vectors are more of a problem, since we now need pointers as components of the data vector. The Posix Fortran 77 Language Interface IEEE standard does not provide a general enough mechanism for that purpose. We might restrict the use of data vectors to C. (Of course, sophisticated programmers would be able to call C routines from Fortran, and nonstandard solutions might also be provided.) Or, we might require that all buffer addresses are integer displacements with respect to the first address, in which case all entries are integer. Is it acceptable to have a (small) fixed upper bound on the length of data vectors? } \paragraph*{Typed Buffers} \missing{Data buffers for typed messages.} \subsection{Receive Criteria} The selection of a message by a receive operation is done according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \discuss{ The DONTCARE value can be a fixed value (e.g. -1), and/or a named constant. It might be useful to have the option to require exact size match of incoming message and receive buffer. Some would also like to see truncation supported. } \subsection{Communication Handles} One can consider (most) communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). A handle is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The handle, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., a handle needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all handles are freed at program termination, by default). A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Handle Creation} A handle for a send operation is created by a call to {\bf MPI\_INIT\_SEND}, followed by a set of calls that set the various parameters associated with the handle. The handle cannot be used until all parameters have been set. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating a handle for a receive operation. Handle creation is a local operation that need not involve communication with a remote process. {\bf handle = MPI\_INIT\_SEND (dest, tag, context)} \\ Create a send handle. All three parameters must be specified. {\bf handle = MPI\_INIT\_RECV (source, tag, context)} \\ Create a send handle. DONTCARE values may be used for the first two parameters. See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ An alternative design, proposed by Gropp and Lusk, is to have each newly created handle associated with default parameters that can be later modified. In their proposal, too, all modifications must occur before the first use.} \subsubsection{Buffer specification} The {\bf MPI\_BUFFER} function associates a buffer with a handle. {\bf MPI\_BUFFER(handle, buftype, params)}\\ \begin{tabbing} {\bf buftype} \ \ \ \ \ \ \ \ \ \ \ \= {\bf params} \\ CONTIGUOUS \> buf, len \\ STRIDED \> buf, numblk, lenblk, stride \\ GENERAL \> data\_vector \end{tabbing} See Section~\ref{subsec:buffers} for a description of these buffer types and parameters. The {\tt buftype} parameter is an integer, with named constants provided for the three values. The {\tt params} parameter list consists of an address (pointer) for {\tt buf}, and a one-dimensional array of integers to specify the remaining values. See Section~\ref{subsec:buffers} for a discussion of data vectors. \missing{typed buffers} \discuss{ Some may want specific values for {\tt buftype}, e.g. CONTIGUOUS =0, STRIDED=1, GENERAL=2. } \subsubsection{Start Mode} The {\bf MPI\_START\_MODE} function selects a mode for starting a communication operation. {\bf MPI\_START\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation may start whether or not a matching communication operation has been posted. \item[READY] The operation my start only if a matching communication operation has been posted. \end{description} Thus, a {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. A {\bf ready receive} can start only if a matching send is already posted; otherwise the operation is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and READY=1. We discussed only ready send, not ready receive. Ready send is useful when message passing is implemented in a ``push'' style, where the sender moves the message to the receiver (common case). Ready receive would be useful when message passing is implemented in a ``pull'' style, where the receiver moves the data from the sender. } \subsubsection{Completion Mode} \label{subsec:completion} The {\bf MPI\_COMPLETE\_MODE} function selects a mode for completing a communication operation. {\bf MPI\_COMPLETE\_MODE(mode)} The mode parameter is an integer, with named constants provided for its two values. There are two modes: \begin{description} \item[REGULAR] The operation completes locally, irrespective of the completion of the matching operation. \item[SYNCHRONOUS] The operation completes only if the matching operation completes too. \end{description} A regular send can complete as soon as the message has been composed: data has been copied out of the send buffer and the values used for the envelope have been stored. The sender may now update the send buffer. A regular send can complete before the matching receive has completed. A regular receive can complete as soon as the message has been consumed: data is available in the receive buffer and all values returned by the send complete operation are available. The receiver may now access the receive buffer. A regular receive may complete before the matching send has completed (It may seem strange, at first blush, that a receive may complete before the matching send completes. However, one should keep in mind that an operation completes only when the {\tt complete} suboperation terminates. Thus, even though data has been copied out and the message was sent, a send has not not yet completed if the {\tt send complete} suboperation has not yet been invoked. Informally, the send (receive) completes only when the sender (receiver) knows the operation has completed.) A synchronous communication operation completes only if the matching operation completes too. Thus, a synchronous send can complete as soon as the all operations preceding the completion of the matching receive have terminated. A synchronous receive may terminate as soon as the message has been consumed and all operations preceding the completion of the matching send have terminated. Informally, the completion of the matching send and receive operations are synchronized. Matching communication operations must have the same completion mode. I.e., either both send and receive are regular or both are synchronous. A program where a regular receive matches a synchronous send (or vice versa) is erroneous and its outcome is undefined. \discuss{ May want to fix the values, e.g., REGULAR=0 and SYNCHRONOUS=1. } \subsubsection{Communication Start} {\bf MPI\_START(handle)} \\ The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \discuss{ Some might prefer two distinct functions: {\bf MPI\_SEND\_START} and {\bf MPI\_RECV\_START}. } \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf MPI\_COMPLETE\_SEND ( handle)} \\ A call to {\bf MPI\_COMPLETE\_SEND} returns when the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). {\bf MPI\_COMPLETE\_RECV ( handle, len, source, tag)} \\ A call to {\bf MPI\_COMPLETE\_RECV} returns when the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). The call returns the number of bytes received, and the source and tag of the message received. {\bf MPI\_CHECK\_SEND ( handle, flag)} \\ A call to {\bf MPI\_CHECK\_SEND} returns {\tt flag=true} if the send operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). It returns {\tt flag=false}, otherwise. A successful return of {\bf MPI\_CHECK\_SEND} has the same semantics as a return of {\bf MPI\_COMPLETE\_SEND}. {\bf MPI\_CHECK\_RECV ( handle, flag, len, source, tag)} \\ A call to {\bf MPI\_CHECK\_RECV} returns {\tt flag=true} if the receive operation identified by {\tt handle} is complete (see Section~\ref{subsec:completion} for a description of completion criteria). In such case the call returns the number of bytes received, and the source and tag of the message received. The call returns {\tt flag=false}, otherwise. In such case, the return values of {\tt len}, {\tt source} and {\tt tag} are undefined. Implementation notes: A call to {\tt MPI\_COMPLETE} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. A call to {\tt MPI\_COMPLETE\_SEND(handle)} has the same semantics as a busy loop of the form \begin{verbatim} repeat MPI_CHECK_SEND(handle, flag) until(flag=true) \end{verbatim} and similarly for receives. The use of a blocking receive operation ({\tt MPI\_COMPLETE}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_CHECK}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_CHECK} is for that operation to return as soon as possible. Note that it is correct, but inefficient, to implement {\tt MPI\_CHECK} via a call to {\tt MPI\_COMPLETE}, in which case, {\tt MPI\_CHECK} always returns {\tt true}. \discuss{ One might use {\tt len=-1} to indicate unsuccessful return of {\tt MPI\_CHECK\_RECV} } \subsection{Blocking Communication} Blocking send and receive operations combine all four suboperations into one call. The operation returns only when the communication completes and no communication handle persists after the call completed. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf send \\ \bf recv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. {\bf MPI\_SEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSSENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_SEND(handle,len,source,tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_SRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, soure, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_RSRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) MPI_COMPLETE_RECV(handle, len, source, tag) MPI_FREE(handle) \end{verbatim} Note that all receive calls return the actual length, tag and source of the received message. This, the {\tt source} and {\tt tag} parameters are INOUT. The {\tt len} parameter is INOUT for contiguous receive buffers, OUT for strided and general receive buffers. {\bf Implementation note:} While these 24 functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \discuss{ Not all 24 functions are equally important (ready-synchronous is somewhat esoteric). We can decide to organize the functins so that the most important are in a ``core subset''. But (i) users will use whatever you provide, and will find uses for what you believed to be useless; (ii) there is a virtue in being systematic -- no need to remember what works and what does not; (iii) what is not important can be implemented using multiple calls to the basic functions. The need for an additional {\tt len} parameter for strided and general receives is somewhat annoying. Suggestions? } \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call, and the last two suboperations ({\tt COMPLETE} or {\tt CHECK} and {\tt FREE}) into another suboperation. Thus a nonblocking send or receive consists of two suboperations: One that starts the communication, and one that completes the operation and free any associated resources. No communication handle persists after these two suboperations are executed. \subsubsection{Initiation} We use the following naming convention for the first suboperation of a nonblocking communication: \[ \left[ \begin{array}{c} - \\ \bf r \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \end{array} \right] \left[ \begin{array}{c} \bf isend \\ \bf irecv \end{array} \right] \left[ \begin{array}{c} - \\ \bf s \\ \bf g \end{array} \right] \] The first letter (void or {\bf r}) indicates the start mode (regular or ready). The second letter (void or {\bf s}) indicates the completion mode (regular or synchronous). The last letter (void, {\bf s} or {\bf g}) indicates the buffer type (contiguous, strided or general). The corresponding 12 send and 12 receive operations are listed below. The letter {\bf i} in {\bf isend} and {\bf irecv} indicates that the operation is ``immediate'' and that the call may return before the operation is completed. We thus have 12 immediate send operations and 12 immediate receive operations. {\bf MPI\_ISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_ISEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_ISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISEND(buf, len, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONEOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDS(buf, numblk, lenblk, stride, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSISENDG(vector, dest, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_SEND(dest, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_IRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_SIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, REGULAR) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, REGULAR) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECV(buf, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, CONTIGUOUS, (len)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVS(buf, numblk, lenblk, stride, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, STRIDED, buf, (numblk, lenblk, stride)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} {\bf MPI\_RSIRECVG(vector, len, source, tag, context)} \\ is \begin{verbatim} handle= MPI_INIT_RECV(source, tag, context) MPI_BUFFER(handle, GENERAL, (vector)) MPI_START_MODE(handle, READY) MPI_COMPLETE_MODE(handle, SYNCHRONOUS) MPI_START(handle) \end{verbatim} \subsubsection{Completion} A communication operation can be completed in two ways: (i) By a call to {\tt MPI\_WAIT}. Such call returns when the corresponding operation has completed. (ii) by a call to {\tt MPI\_STATUS}. Such call returns successfully if the communication operation has completed; it may also return unsuccessfully if the operation has not completed. See Section~\ref{subsec:complete_ops} for a discussion of the two completion operation types. {\bf MPI\_WAIT\_SEND(handle)} \\ is \begin{verbatim} MPI_COMPLETE_SEND(handle) MPI_FREE(handle) \end{verbatim} {\bf MPI\_WAIT\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_COMPLETE_RECV(handle, len, source, type) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_SEND(handle, flag)} \\ is \begin{verbatim} MPI_CHECK_SEND(handle, flag) if (flag) MPI_FREE(handle) \end{verbatim} {\bf MPI\_STATUS\_RECV(handle, len, source, type)} \\ is \begin{verbatim} MPI_CHECK_RECV(handle, flag, len, source, type) if (flag) MPI_FREE(handle) \end{verbatim} \paragraph*{Multiple Receives} It is convenient to be able to wait for any successful receive in a set, rather than having to wait for specific message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ {\bf list\_of\_handles} is an array containing $(\rm n, handle_1 , ... , handle_n)$. Blocks until one of the operations associated with handles in the array has completed. Returns the index of that handle in the array, and the length, source and tag of the received message. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], len, source, tag); index = 1} || ... || {MPI_WAIT_RECV (handle[n], len, source, tag); index = n} \end{verbatim} (``$||$'' indicates alternation; one of the alternatives is chosen, nondeterministically.) \discuss{ An alternative definition of {\tt WAITANY} is to require that the first available message in the list is received. I.e. {\bf MPI\_WAITANY\_RECV ( list\_of\_handles, index, len, source, tag)} \\ is \\ {\tt for(index=1; i$<=$n; i++) \{ \\ \ \ \ MPI\_STATUS\_RECV (handle[index], flag, len, source, tag); \\ \ \ \ if(flag) break; \\ \ \ \ \} } A difference would occur in the case where two receives in the list match the same message; in the original definition any of the two may succeed, whereas in the amended definition only the first one can succeed. This additional determinism, may, on the other hand, restrict concurrency in the implementation of {\tt WAITANY}. It would be nice to have a WAITALL that waits for the completion of all operations listed in the list\_of\_handles. The most useful option might be, in fact, ``wait for the completion of ALL pending operations in the current context'', which could be specified by a ``wildcard'' list\_of\_handles (e.g., a length zero list). } \paragraph*{Probe} It is convenient for a receiver to be able to check for pending messages that can be received, before executing a receive operation. This, for example, would allow the receiver to allocate a receive buffer of the right size, when the size of incoming messages is not known ahead of time. The {\bf PROBE} call specifies an envelope pattern. It returns a handle to an available message with an envelope that matches the pattern, and that does not match any pending receive. It also returns the actual values of {\tt len} {\tt source} and {\tt tag}. The message is now ``locked'' and can be received only by an operation that uses the handle returned. If there is no matching message, then {\tt probe} returns with a NULL handle. {\bf MPI\_PROBE(len, source, tag, context, handle)} \\ If there is a matching message that is not matched by a previously posted nonblocking receive, returns message length, the actual values of source and tag, and a handle that must be used to receive the message. Otherwise, returns a handle with value {\bf NULL} (NULL is a named constant). {\bf MPI\_PRECV ( buf, len, handle)} \\ Nonblocking receive of probed message into a contiguous buffer. {\bf MPI\_PRECVS ( buf, numblk, lenblk, stride, handle)} \\ Nonblocking receive of probed message into a fixed stride buffer. {\bf MPI\_PRECVG ( datavector, handle)} \\ Nonblocking receive of probed message into a buffer specified by a data vector. \discuss{ May want a specific value for NULL, e.g., NULL=0 May need to return more than length, for typed messages. } \subsection{Correctness} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active handles. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item Each process can create at least one communication handle. I.e., if a process executes an {\tt MPI\_INIT\_SEND} or {\tt MPI\_INIT\_RECV} operation, and has no other active handle, then the operation eventually succeeds. It is highly desirable to have generous bounds on the number of concurrently active communication handles each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active handle. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE\_SEND} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE\_RECV} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ids 1 and 2) The following program is safe, and should always succeed. \begin{verbatim} IF (GETID() == 1) { MPI_SEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_RECV(buf=recvbuf, len=1000, source=1, tag=0); MPI_SEND(buf=sendbuf, len=1000, dest=1, type=0); } \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} IF (GETID() == 1) { MPI_SSEND(buf=sendbuf, len=1000, dest=2, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SSEND(buf=sendbuf, len=1000, dest=1, tag=0); MPI_SRECV(buf=recvbuf, len=1000, source=1, tag=0); } \end{verbatim} The send operation of the 1st process can complete only if the matching receive of the second processor is executed; the send operation of the second processor can complete only if the matching receive of the first processor is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} IF (GETID() == 1) THEN { MPI_SEND(buf=sendbuf, len=1000000, dest=2, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=2, tag=0); } ELSE \* (GETID() == 2) *\ { MPI_SEND(buf=sendbuf, len=1000000, dest=1, tag=0); MPI_RECV(buf=recvbuf, len=1000000, source=1, tag=0); } \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer 1 MgB of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \subsection{Error Handling} \subsubsection{Communication Errors} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. The other errors can be divided in two classes: \begin{description} \item[resource errors] Errors that occur in an unsafe programs because of limitations on the resources of the communication subsystem, but would not occur if these subsystem had unbounded resources (buffer overflow, no available handles, etc.) \item[program errors] Errors that result in an erroneous program that will always fail. (send with no matching receive, send to nonexistent destination, etc.) \end{description} Program errors always result in an exception, if detected. MPI will provide two mechanisms for handling resource errors: In {\em system mode} each MPI call may return with a return code that indicates that a resource error was detected; no exception occur, and the program may recover in normal execution mode. In {\em user mode} a detected resource error causes an exception. \discuss{ Program errors cannot always be distinguished from resource errors. Have to decide how to handle doubtful cases in system mode. Need to change the syntax so that all return values (error code excepted) are returned via parameters. It's reasonable to restrict system mode to C and C++ bindings. } \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:12:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28591; Mon, 8 Feb 93 10:12:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24749; Mon, 8 Feb 93 10:11:20 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:11:19 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24741; Mon, 8 Feb 93 10:11:17 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25516 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:11:15 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06314; Mon, 8 Feb 93 07:06:07 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16745; Mon, 8 Feb 93 07:10:39 PST Date: Mon, 8 Feb 93 07:10:39 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081510.AA16745@lion.parasoft> To: mpi-pt2pt@cs.utk.edu To: mpi-pt2pt@cs.utk.edu Re: Buffering requirements I agree with Jim Cownie's comment that trying to figure out buffering requirements in an application is difficult for the receiver and hence the problems in creating REALLY reliable code with buffered systems. A potentially easier option is the one that Rik Littlefield mentioned - buffering at the sender instead. For most applications it is MUCH easier to compute how much is going to be sent than how much will be received. However, buffering at the sender seems to impose an implementation feature, i.e., that there be an intial message describing the actual transaction followed by a data request and finally the data itself. Is there a way around this? If so, it seems like a great idea to me. If not, is it going to be acceptable to the High Priests to have this three-hop version? Jon Flower From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:36:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29236; Mon, 8 Feb 93 10:36:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25820; Mon, 8 Feb 93 10:35:47 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:35:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25812; Mon, 8 Feb 93 10:35:43 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25693 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:35:40 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06436; Mon, 8 Feb 93 07:30:33 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16763; Mon, 8 Feb 93 07:35:05 PST Date: Mon, 8 Feb 93 07:35:05 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081535.AA16763@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: /dev/null, /dev/full? To: mpi-pt2pt@cs.utk.edu Re: /dev/null, /dev/full? Despite the fact that we still don't seem to have settled on a model for specifying process identifiers I would like to raise a different issue on this topic ..... the idea of having a special identifier for a node that doesn't actually exist. Sending a message to this destination would always succeed and return as though the data were transmitted, and reading from this node would also return immediately, without modifying the user buffer, but with a return code indicating that no error occurred. I think this offers users great simplifications in their coding. A classic example is the standard domain decomposed PDE solution. In one dimension you have to write code for the guard strip exchange such as if(I_am_on_left_edge) { send_to_right(); read_from_right(); } else if(I_am_on_right_edge) { send_to_left(); read_from_left(); } else if(there_is_more_than_one_node) { send_to_right(); read_from_right(); send_to_left(); read_from_left(); } Note that this code assumes buffered communication. If you have to code unbuffered messages then the logic becomes considerably worse since you have to deal with even and odd parity and the case of an odd number of nodes, etc..... If we have the "no node" concept then the above becomes: send_to_right(); read_from_right(); send_to_left(); read_from_left(); in all cases. Given an "exchange" or "sendrecv" function this code becomes even simpler. When dealing with virtual topologies this is invaluable because the virtual topology code can return this special value as the process I.D. and it can then be passed directly to the communication routines without having to be interpreted at all. I think this can be implemented without cost to the higher level software by making the "no node" identifier illegal for normal communication. Then the basic communication functions which, I imagine, do checking such as if( ! destination_is_legal_node ) { return ILLEGAL_NODE; } can be simply changed to if( ! destination_is_legal_node ) { if( destination_is_no_node ) { /* Pretend this was OK */ return OK; } return ILLEGAL_NODE; } without any impact on the regular code. EXTENSION: Marc Snir raised an interesting point in connection with this idea. Since it is normally used to deal with boundary conditions in the physical world why not extend the idea by allowing a read from the "no node" source to actually implement the user boundary condition either by filling an array with predefined values, or even calling a user-defined function? Again I think this could be implemented at no cost to the higher levels by simply invoking some code in the error handling for normal communication, and extending the "no node" identifier to a range of identifiers. I think in this case you might also be able to run user-supplied code, even when you have a communication co-processor, assuming that the error checking shown above occurs early enough in the sequence that it would still be in the main processor(?) I don't have any experience with this extension but it certainly seems "cute". One possible objection might be that the boundary case code is almost certainly already coded in any existing application and pushing it down into the message passing layer might be a step backwards. For applications written from scratch, however, it is a nice abstraction. Jon Flower From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:36:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29242; Mon, 8 Feb 93 10:36:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25849; Mon, 8 Feb 93 10:36:36 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:36:34 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25840; Mon, 8 Feb 93 10:36:33 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA20721; Mon, 8 Feb 93 08:36:26 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nLaX2-0016ZKC; Mon, 8 Feb 93 08:36 MST Message-Id: Date: Mon, 8 Feb 93 08:36 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-pt2pt@cs.utk.edu Subject: Re: Buffering requirements Jon asks: >> Is there a way around this? If so, it seems like a great idea >> to me. If not, is it going to be acceptable to the High Priests >> to have this three-hop version? This brings us back to the question of the "Ready Receiver". That is, should one KNOW that a matched receive has already been posted, then one should be able perform the send without having to go through the heavy duty "RTS/CTS/Data" sequence. One other point I would like to raise is that the phrase "system buffering" implies an implementation where the system provides the buffering rather than the application. I believe that one could argue that "application buffering" would be more portable, in that the application/library would depend on more basic services, such as application memory allocation to provide the buffering. Stephen From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 10:54:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29472; Mon, 8 Feb 93 10:54:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26520; Mon, 8 Feb 93 10:53:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 10:53:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26512; Mon, 8 Feb 93 10:53:49 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA25873 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Mon, 8 Feb 1993 07:53:47 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA06638; Mon, 8 Feb 93 07:48:40 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA16875; Mon, 8 Feb 93 07:53:12 PST Date: Mon, 8 Feb 93 07:53:12 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9302081553.AA16875@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: System vs. Application buffers Having application level buffers certainly seems to give the user more control over the amount of memory they hve to play with but is it consistent with layered (library) packages? Assume that we have a simple three-tier library calling sequence with buffer requirements as follows User-application (Needs buffer space X) Library-1 (Needs buffer space Y) Library-2 (Needs buffer space Z) Are we then going to ask for memory (X+Y+Z) or simply max(X,Y,Z)? I suspect that if the system is buffering we'll probably get the latter. In the application buffered case you could probably have either if buffers are allocated with malloc/realloc. But if buffers are declared explicitly then X+Y+Z is probably going to be required. One option would be to have one of the arguments to the library package be its message buffer space in the same way that IMSL routines have you pass in workspace as arguments. Then you could potentially re-use an outer layer buffer. This seems ugly but would probably work. Jon From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 8 12:19:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01665; Mon, 8 Feb 93 12:19:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00736; Mon, 8 Feb 93 12:18:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 8 Feb 1993 12:18:14 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00728; Mon, 8 Feb 93 12:18:11 -0500 Received: from carbon.pnl.gov (130.20.65.121) by pnlg.pnl.gov; Mon, 8 Feb 93 09:04 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA23874; Mon, 8 Feb 93 09:03:34 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA18935; Mon, 8 Feb 93 09:03:26 PST Date: Mon, 8 Feb 93 09:03:26 PST From: d39135@sodium.pnl.gov Subject: Re: Buffering requirements To: jwf@lion.Parasoft.COM, mpi-pt2pt@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9302081703.AA18935@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Jon Flower says: > I agree with Jim Cownie's comment that trying to figure out buffering > requirements in an application is difficult for the receiver and hence > the problems in creating REALLY reliable code with buffered systems. > A potentially easier option is the one that Rik Littlefield mentioned - > buffering at the sender instead. For most applications it is MUCH > easier to compute how much is going to be sent than how much will > be received. > > However, buffering at the sender seems to impose an implementation > feature, i.e., that there be an intial message describing the actual > transaction followed by a data request and finally the data itself. > > Is there a way around this? If so, it seems like a great idea > to me. If not, is it going to be acceptable to the High Priests > to have this three-hop version? Let's see if I can remember what it was I proposed. I think it went like this: 1. Provide an MPI hook that an application could call to declare the maximum amount of data that might be sent to unready receivers. 2. Require that an application calling the hook also provide buffer space for that much data. 3. Require that if the hook is called, MPI then must *act as if* it were working in "sender-buffers" mode. 4. One possible implementation of this approach is for MPI to simply transform all blocking send calls into . copy data to (application-provided) buffer space . issue non-blocking send from the buffer copy . return to application and . check completion on subsequent MPI call(s) This implementation does not seem to impose any extra costs at the protocol level. In particular, if non-blocking send can be accomplished without an "RTS/CTS/data" sequence, say by assuming some amount of buffering at the receiver as in the Intel iPSC short-message and DELTA protocols, then the sender-buffers implementation proposed here should be able to do the same. No doubt many optimizations within MPI are possible -- the proposal just requires that MPI act as described here. I am not sure how the sender-buffers declaration should look in order to best support libraries. One possibility would be to allow the declarations to be nested, so that each library could declare its own needs at the point where those needs are determined. I'm not convinced that this is much easier than just (re)coding the library to use non-blocking sends directly, but it seems to fit with the basic idea of portable support for non-blocking csend's. --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Tue Feb 9 02:04:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18444; Tue, 9 Feb 93 02:04:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06135; Tue, 9 Feb 93 02:03:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 9 Feb 1993 02:03:05 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06127; Tue, 9 Feb 93 02:03:03 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA20719; Tue, 9 Feb 93 01:02:59 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA04746; Tue, 9 Feb 93 01:02:55 CST Message-Id: <9302090702.AA04746@donner.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: A suggestion for a multi-level MPI Date: Tue, 09 Feb 93 01:02:54 CST From: Rusty Lusk Last week Tom Henderson asked for more detail about the multiple-level proposal, so we wrote up a slightly more detailed explanation of the idea, with some examples, which is appended at the end of this note. It is partly subsumed by Marc Snir's posting of a thoroughly worked out point-to-point proposal, which has a lot in common with what we described as Levels 1 and 2. A few notes: In Level 1, we use a very minimal set of required parameters in the "init" calls, so that more of the parameters can be reset inside a loop without acquiring a new handle. We really think that Levels 3 and 4 (the simpler levels, where there are fewer options and parameters) are necessary in order to provide a simple interface for basic users. We didn't change the way heterogeneity is handled, but Tom Henderson's suggestion (that the datatypes *always* be specified, perhaps with a BYTES type for the "raw" datatype) is a quite sensible one. Here is a medium-sized description of what we mean by a "multi-level" standard. The specific routines and their parameters are not as important as the idea that complexity can be managed in this way. ---------------------------------------------------------------------- \documentstyle[11pt]{article} \textwidth=6.0in \textheight=9.0in \hoffset=-0.5in \voffset=-1.0in \parskip=.1in \def\explicitspace{\hbox{\vrule height 2pt width .5pt \vbox to 8pt{\vfil\hrule height .5pt width 6pt}% \vrule height 2pt width .5pt}} \begin{document} \title{\Large \bf A Proposal for a Multi-Level Message-Passing Interface Standard} \author{ {\it William D. Gropp and Ewing L. Lusk} \\ Argonne National Laboratory \\ Argonne, IL 60439 \\ gropp@mcs.anl.gov \\ lusk@mcs.anl.gov } \date{} \maketitle \begin{center} {\large \it DRAFT} \end{center} \section{Introduction} \label{sec:introduction} In \cite{gropp-lusk:mpi1} we described the organization of a library of routines that could be used to implement a standard set of message-passing functions. We used the library to rapidly implement most of the routines described in \cite{dongarra-hempel-hey-walker:october-draft}. The implementation testbed was not originally offered as a counterproposal to the routines described in \cite{dongarra-hempel-hey-walker:october-draft} since no real discussion had taken place in the MPI Committee yet, but it did contain many capabilities we believed would ultimately be desired. It was constructed in a way that offered a wide range of capabilities with a minimum of complexity in syntax. Basically, it consisted of a large number of separate routines whose names were orthogonally organized along a set of axes corresponding to synchronization type, buffer structure, high-performance mode, and heterogeneity specification. See \cite{gropp-lusk:mpi1} for details. At the MPI meeting in January, it became apparent that \begin{itemize} \item Many of the committee members were uncomfortable with having a large number of routines. \item A very wide range of capabilities and options was nonetheless desired. \item A very wide range of users was to be targeted, from experienced users requiring access to high-performance operations to beginning users requiring simplicity in both syntax and semantics. \item While discussion of precise syntax was deferred, it might be useful to suggest one, in order that everyone could see what the ultimate standard might look like. \end{itemize} We added a further consideration of our own: \begin{itemize} \item The standard needs a way to expand in order to add functionality or options that have not been envisioned yet, without becoming incompatible with itself. \item An implementation would have to be extremely adaptable if it was expected to be able to track the discussions of the MPI Committee. \end{itemize} The present draft is a reflection of these considerations. It is still incomplete (as are the committee's discussions) but is offered in the hope that it might be useful. It focuses primarily on the point-to-point operations. \section{The Basic Idea: Orthogonality of Function} \label{sec:orthogonality} The MPI committee has discussed a large number of options for point-to-point message passing. The fact that there could be as many as a thousand different ``send'' routines has become a joke, and some believe that the scope of MPI way have to restricted in order that it not become unwieldy. This problem has at least two solutions, and we explore each of them here. What makes the porblem tractable is that the options that are being discussed are {\it orthogonal\/}; that is, choice of one value of an option does not determine what value another opetion must have. For example, whether a send is blocking or nonblocking is completely independent of whether the message consists of contiguous memory or not. This orthogonality can be exploited to make the interface simple to specify, implement, and use, while at the same time not sacrificing flexibility. The first way is to set the options with a ``universal'' {\tt set\_option(handle,option,value)} routine. There will be a small number of allowed values for {\tt option}, and for most of these, a small number of values for {\tt value}. (This is the approach taken in Level I, below.) The second way is to absorb the options and their values into the routine names. This leads to a large {\em number\/} of routines, but their names can be made predictable and descriptive, so that they are easy to understand and use. (This is the approach taken in Level II, below.) The principal organizing idea of this library is that it proposes multiple sets of routines, roughly grouped by complexity level, {\it all of which are proposed for simultaneous inclusion as part of the MPI Standard}. That is, although it is possible to implement some of the higher-level routines in terms of the lower ones, either by combining short sequences of low-level calls or by supplying specific values for parameters, it is strongly suggested here that all levels be adopted together. This is the way to attain a set of routines that, taken altogether, offers complete flexibility for the advanced user while not imposing any unnecessary complexity in either syntax or semantics on the beginner. The relationships among the levels remain extremely useful for defining the semantics of the higher-level routines in terms of the lower level ones, which have simpler semantics. We now describe the four ``levels'' of MPI routines. For expository purposes, we describe them in what might be considered an unnatural order: \begin{description} \item[Level IV] The highest level, consisting of only four routines, suitable for beginning users. \item[Level I] The lowest level, giving complete access to all of the facilities offered by the MPI Standard. The actual number of routines is small. \item[Level III] A middle level, required in order to write programs that are more efficient and portable than those expressible with Level IV. This is the approximate level of most current message-passing systems. \item[Level II] A level between I and III, chosen for convenience of use without giving up the features of Level I and possible added efficiency in the implementation. It corresponds roughly to the set of routines described in \cite{gropp-lusk:mpi1}. \end{description} It is to be emphasized that this layered structure is for ease of definition and convenience of use, and does not imply a layered structure for the implementation, which could result in suboptimal efficiency. Likewise, it is not suggested here that a user would have to pick one of the layers and stick with it. In all cases where it makes sense, the layers should be interoperable; that is, routines from all layers may appear in the same program. \section{The Point-to-point MPI routines} \label{sec:pt-to-pt} Note on syntax: we have to pick {\em some\/} syntax in order to write these down clearly. We leave the selection of an elegant, expressive, complete, intuitive syntax and appropriate language bindings to the further discussions of the MPI Committee. \subsection{Level IV: The Four-function Interface} \label{sec:four-func} A large number of parallel algorithms can be expressed in terms of four simple functions. In the quest for simplicity, it is useful to see how far the standard might be able to go. It appears to us that the following set of routines represents a minimum. (Strictly speaking, only two of these belong in this point-to-point section, but it is not worth separating this tiny set of routines into parts.) \begin{verbatim} MPI_numpids() returns the number of active process ID's MPI_mypid() returns the id of the calling process MPI_send(dest,tag,buffer,length) sends the contiguous set of length bytes addressed by buffer to process dest and tags the message with tag. The buffer is available for reuse when this returns. MPI_recv(dest,tag,buffer,length) waits for a message from process dest with tag tag and puts in buffer, truncating it to length if necessary for it to fit into buffer. \end{verbatim} More precise definitions of these can be made once we have defined the lower levels. Basically, the point-to-point component consists of an asynchronous send that blocks only until the buffer can be reused, and a blocking receive that waits for the arrival of a message. \subsection{Level I: Access to complete functionality in a small number of routines} \label{sec:level1} The above trivial interface is enough to specify a parallel program, and in many cases, is actually sufficient. The only reason for adding more point-to-point routines is to improve efficiency, portability, and convenience. Before defining what perhaps will be the most commonly-used layer for most programmers (Level III), we go down to Level I, where all of the MPI capabilities will be exposed. We can then describe the intermediate layers more easily, in terms of the lowest one. The idea here is to fully exploit for the lowest level two ideas that were discussed favorably at the January meeting: \begin{itemize} \item separation of the ``setup'' part of an operation from the ``initiate'' part. This idea is described in detail in \cite{snir:notes}. \item the use of a control structure (either in system space or user space) to hold the parameters for an operation that has been ``set up'', but is not complete. (It may or may not have been initiated.) This structure serves as the ``handle'' for the operation. Its precise format depends on the MPI implementation, and it is accessed by the user only through MPI routines. \end{itemize} \begin{verbatim} handle = MPI_init_send(dest,tag) handle = MPI_init_recv() MPI_mod_send(handle,option,value) MPI_mod_recv(handle,option,value) MPI_do_send(handle) MPI_do_recv(handle) MPI_free(handle) \end{verbatim} The idea is that the ``init'' routines return a handle that is usable {\em as is\/}, because it selects a reasonable set of defaults, but that can be modified before the operations is actually initiated. If the {\tt do\_send} or {\tt do\_recv} operations are issued without any intervening calls to {\tt mod} routines, then the default behavior occurs, which we will have to decide on. However a wide range of options can be set by the {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} routines. Although we are not intending to specify an implementation, it will perhaps clarify things to envision how this might work. The {\tt init} routines will allocate a block of storage (or, alternatively, deal with a block a storage allocated by the user). The ``do'' routines will take their parameters from this control block. The control block will be immediately usable, since the minimal parameters will be specified on the ``init'' call. (A send doesn't make sense without a destination and a tag, so they are its parameters. A receive need specify no filtering by tag or sender, and could receive a message of 0 bytes, requiring no buffer, hence the {\tt recv\_init} has no parameters.) The {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} are used to modify the default parameters set up by the {\tt MPI\_init\_send} and {\tt MPI\_init\_recv} routines. The parameters to the {\tt MPI\_mod} routines are as follows: \begin{description} \item[handle] the value returned by a previous {\tt MPI\_init\_send} or {\tt MPI\_init\_recv}. \item[option] one of a collection of symbolic constants specifying the parameter being set. \item[value] one of a collection of symbolic constants or integers or addresses, depending on the option. \end{description} One suggestion for a set of options and their allowed values might be: \begin{verbatim} option value ------- ------- BUFTYPE - one of CONTIG, STRIDED, VEC, HVEC BLOCKING - one of NONE, LOCAL, GLOBAL TRANSLAT - one of NONE, XDR PROTOCOL - one of PLAIN, RCVRRDY BUFADD - address of buffer, or NULL BUFLEN - integer lengh of buffer DATATYPE - one of INT, FLOAT, DOUBLE, LONGINT, BYTES, CHAR TAG - integer or (on receive) ANY BUFVEC - address of structure describing the buffer data, or NULL SOURCE - (on receive) integer process id, or ANY \end{verbatim} The {\tt MPI\_init\_send(dest,tag)} could reasonably set the following defaults, in addition to setting the {\tt dest} and {\tt tag} fields in the control block: \begin{verbatim} BUFTYPE = CONTIG BLOCKING = LOCAL TRANSLAT = NONE PROTOCOL = PLAIN BUFADD = NULL BUFLEN = 0 DATATYPE = BYTES BUFVEC = NULL \end{verbatim} The {\tt MPI\_init\_recv()} could set the same default values, together with {\tt SOURCE = ANY} and {\tt TAG = ANY}. (In a real proposed standard, these symbolic constants should have an MPI prefix, in order to prevent the name-space pollution problem, but here we leave them off for simplicity.) The large number of routines listed in \cite{gropp-lusk:mpi1}, some of which will reappear in Level II, have in Level IV been absorbed into parameter values of the {\tt MPI\_mod\_send} and {\tt MPI\_mod\_recv} routines, in particular the first four values of {\tt option}. The advantages of this approach are: \begin{itemize} \item a small number of routines at the lowest level \item extensibility: new options, even implementation-specific ones, can be handled by adding more options without adding more routines. \end{itemize} The disadvantage is that to send a message requires a sequence of subroutine calls, adding both program complexity and a moderate amount of overhead to each operation. It is this problem that is addressed by level II. \subsection{Level III: A Medium-Level Interface} \label{sec:level3} In this section we describe a level that is designed to provide access to a sufficient set of MPI routines that efficient, portable programs may be written in it, without accessing all of the flexibility of the complete MPI interface. This level will, however, support both blocking and non-blocking sends and receives as well as messages between machines from different vendors. By imposing a few restrictions on what can be done at this level, however, we can reduce the number of routines and shorten parameter lists, thus reducing complexity. A possible set of restrictions for this level might be: \begin{itemize} \item No noncontiguous buffer structures \item No mixed datatypes in messages destined for heterogeneous processors \item No ``rendezvous'' send \item No ``receiver-ready'' messages \end{itemize} Programmers who need more functionality than is offered at this level can use the routines in levels I or II. The set of Level III routines, given the above restrictions, looks like \[ \left[ \begin{array}{c} n \\ b \end{array} \right] \left[ \begin{array}{c} send \\ recv \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ h \end{array} \right], \] \noindent together with the {\tt wait} and {\tt status} routines. The first column specifies whether the opertion is to block or not, and the last column specifies whether the message is to be translated for transmission to a machine with a different data representation. That is, the above diagram ellustrates the way the names are to made up. Since in this case the list is relatively short, we can also list them: \begin{verbatim} bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) \end{verbatim} \subsection{Level II: Flexibility and Efficiency in a large number of routines} \label{sec:level2} For Level II, we use only the naming schematic, since the list becomes rather long. The first column represents the choice of buffer structure ({\tt BUFTYPE} in Level I). The second column specifies the blocking option ({\tt BLOCKING}). Then whether it is a send or receive, then the {\tt TRANSLAT} option, and finally the receiver-ready protocol ({\tt PROTOCOL = RCVRRDY}). \[ \left[ \begin{array}{c} c \\ s \\ g \end{array} \right] \left[ \begin{array}{c} n \\ b \\ s \end{array} \right] \left[ \begin{array}{c} send \\ recv \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ h \end{array} \right] \left[ \begin{array}{c} \explicitspace \\ rr \end{array} \right], \] \noindent See \cite{gropp-lusk:mpi1} for details. \section{Examples of the Multi-Level Approach} \label{sec:examples} Before giving the details, we give three examples of how certain operations would be expressed at teach of the levels. Note that some operations are not available at the higher levels. The point is that simple operations should be simple to express while more sophisticated ones may be more complicated. \paragraph{Example 1.} A blocking send of a contiguous stream of uninterpreted bytes. At Level IV (the four-function level) nothing but the basic parameters needs to specified, since this is the only type of send operation. \begin{verbatim} MPI_send(dest,tag,buf,len) \end{verbatim} At Level III, where both blocking and non-blocking sends are possible, we need to specify that this is a blocking send. We choose to incorporate the specification into the routine name. \begin{verbatim} MPI_sendb(dest,tag,buf,len) \end{verbatim} At Level II, noncontiguous buffers are allowed, so we need to specify that the buffer is contiguous, which we again do with the name. \begin{verbatim} MPI_csendb(dest,tag,buf,len) \end{verbatim} At Level I, we need to set up the operation first, then modify the default parameters of the operation before actually initiating it. \begin{verbatim} handle = MPI_init_send(dest,tag) MPI_mod_send(handle,BUFADD,buf) MPI_mod_send(handle,BUFLEN,len) MPI_mod_send(handle,BLOCKING,LOCAL) MPI_do_send(handle) MPI_send_wait(handle) MPI_send_free(handle) \end{verbatim} \paragraph{Example 2.} A non-blocking receive of data that is all of the same type into a contiguous buffer, with translation for receipt from a process on a machine with a possibly different data representation. At Level IV, this operation is impossible to express. At Level III, the specification that is it is non-blocking and heterogeneous is encoded in the name, which specifies a routine that expects a datatype and number of items, rather than a length. \begin{verbatim} handle = MPI_nrecvh(source,tag,buf,datatype,numvals) \end{verbatim} At Level II, we must also specify that the buffer is contiguous. \begin{verbatim} handle = MPI_cnrecvh(source,tag,buf,datatype,numvals) \end{verbatim} At Level I, we set up the operation and then modify its parameters. \begin{verbatim} handle = MPI_init_recv() MPI_mod_recv(handle,SOURCE,source) MPI_mod_recv(handle,TAG,tag) MPI_mod_recv(handle,BUFADD,buf) MPI_mod_recv(handle,BUFTYPE,CONTIG) MPI_mod_recv(handle,TRANSLATE,XDR) MPI_mod_recv(handle,DATATYPE,datatype) MPI_mod_recv(handle,NUMVALS,numvals) MPI_do_recv(handle) MPI_recv_status(handle) \end{verbatim} \paragraph{Example 3.} A non-blocking send of non-contiguous, mixed heterogeneous data, with the assumption that the receiver has already issued his receive. At Levels IV and III, this is impossible to express. At Level II, most of the options are encoded in the name. \begin{verbatim} MPI_gsendhrr(dest,tag,...) \end{verbatim} At Level I, we use the same basic routines to set the options. \begin{verbatim} handle = MPI_init_send(dest,tag) MPI_mod(handle,BUFTYPE,HVEC) (Build vector describing data locations, types, and numbers of items in each clump.) MPI_mod_send(handle,HVECADD,vec) MPI_mod_send(handle,TRANSLATE,XDR) MPI_mod_send(handle,RECVRRDY,TRUE) MPI_do_send(handle) \end{verbatim} \paragraph{Example 4.} Set up a channel from one process to another, and then use it for repeated send operations of different data. This operation cannot be specified at Levels II, III, or IV. At Level I, it can be specified in the following way. \begin{verbatim} handle = MPI_init_send(dest,tag) begin loop MPI_mod_send(handle,BUFADD,buf) MPI_mod_send(handle,BUFLEN,len) MPI_do_send(handle) end loop \end{verbatim} \section{Collective Communication} \label{sec:collective} The idea of a separate setup routine, together with multiple levels to accomodate both and basic operations as well as more elaborate ones, should prove useful for collective operations as well. We will try to extend the point-to-point operations to collective operations in as natural a way as possible. \bibliographystyle{plain} \begin{thebibliography}{1} \bibitem{dongarra-hempel-hey-walker:october-draft} J.~Dongarra, R.~Hempel, T.~Hey, and D.~Walker. \newblock A proposal for a message passing interface standard. \newblock (circulated before November meeting). \bibitem{gropp-lusk:mpi1} Bill Gropp and Rusty Lusk. \newblock A test implementation of the {MPI} draft message-passing interface standard. \newblock Technical Report ANL--92--47, Argonne National Laboratory, December 1992. \newblock (circulated at January meeting). \bibitem{snir:notes} Marc Snir. \newblock Message-passing interface--outline. \newblock (circulated at January meeting). \end{thebibliography} \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 06:33:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22237; Mon, 15 Feb 93 06:33:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11881; Mon, 15 Feb 93 06:32:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 06:32:39 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11873; Mon, 15 Feb 93 06:32:34 -0500 Date: Mon, 15 Feb 93 11:32:26 GMT Message-Id: <21543.9302151132@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: various To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear colleagues I circulated the documents prepared by Marc, and Bill & Rusty, around interested people locally, for discussion. Here are some resulting considerations of the point-to-point component of MPI. a) Synchronous messages and blocking/nonblocking communication calls: The properties of end to end synchronisation, and blocking/nonblocking communication calls are "orthogonal" in the sense described by Bill and Rusty. Thus, another "axis" in the procedure naming (section 3.3) should be added, which perhaps is either nothing or 's'. The approach taken by Marc is a little, different and appears to implicitly recognise this. This doesnt seem (to me) to fit in with the framework which Marc uses, but it might be appropriate to consider synchronous and ready-receiver as points on the same axis. They both refer to the knowledge which the sender has about receipt of the message. In the synchronous case, the sender absolutely knows that the receiver has got the message; in the ready-receiver case the sender (presumably) has no knowledge of whether the receiver has or has not got the message. This approach would also remove the curious "synchronous ready receiver" from Marc's document. b) Cancellation of nonblocking communications: This subject has appeared before. We strongly support the points made by Jim Cownie regarding the requirement for cancellation of nonblocking communications. An example of its usefullness arises when writing libraries encapsulating common communication patterns, we view it as desirable that the communication call should be able to return with an error code if something is wrong. A routine XXX may not be able to detect this until after some nonblocking communications have been started, and some of those which have been started have also been completed and the information therein examined. At this point it becomes known that the operation cannot succeed, however there are these nonblockinf communications outstanding. It may not be possible to wait for them to complete, as they may never complete since the matching operation may never be started, so th natural thing to do is cancel the outstanding communications and return cleanly with an error code. The importance of the clean return is apparent when one considers the possibility that the memory into which pending nonblocking receives may write (or may not, unknown) is a local variable of the caller of XXX, which will itself then wish to return to its caller, which will in turn call some user error procedure. In this case the stack can become corrupted by data from the uncancelled nonblocking receive started in XXX and the whole program barfs up. Not nice. c) Matching of buffer structures: The proposals contain three provisions for buffer structures: contiguous, strided and iovec (ignoring heterogeneous issues for the moment, see below). Each of these should be defined to provide a byte stream (which may not map to a contiguous set of bytes in memory), and the programmer should thus be allowed to mix'n'match sends and receives of different types freely. This provides a handy way of performing scatter and gather type operations. d) Heterogeneous issues and data conversion: Provision of data type conversion in MPI is gaining support. This makes sense to me, since a lot of PVM users are (presumably) taking advantage of such provisions in PVM. The proposal of Bill and Rusty contains a set of routines which provide data type conversion. Unfortunately the capabilities provided are weaker than PVM, since they allow data conversion in messages of the form "array of datatype" only. This raises the issue of user defined data types in C/Fortran90 (of course there is no issue in Fortran77). At least with the PVM interface the programmer can portably communicate such data types, albeit with a little effort and overhead. e) Completion/checking functions: In sections 1.5.5 and 1.5.6 of Marc's document, the START and COMPLETE operations are described. I cannot agree that there should be seperate MPI_SEND_XXX and MPI_RECV_XXX routines (XXX is one of {START,COMPLETE,CHECK}). The handle contains all of the information regarding the kind of operation which has been started, so it is not necessary to have different routines. It can be most convenient to be able to wait for a mixture of sends and receives to complete, more specifically to wait for any of such a mixture. In order to do this, you must have an interface which allows send and receive handles to be passed to the same wait routine. Another advatage, perhaps, is that htis wait function does not have to raise an error if the wrong kind of handle is supplied, although this advantage is minimal since it will have to check validity of handles in a more general sense. There has been an argument of readability for having seperate routines for send and receive. We could resolve this by defining MPI_SEND_XXX and MPI_RECV_XXX to be wrappers around MPI_XXX, which check that all handles are of the appropriate nature. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 10:09:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04339; Mon, 15 Feb 93 10:09:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20259; Mon, 15 Feb 93 10:08:42 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 10:08:41 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20251; Mon, 15 Feb 93 10:08:40 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15846; Mon, 15 Feb 93 09:08:27 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA27168; Mon, 15 Feb 93 09:08:24 CST Date: Mon, 15 Feb 93 09:08:24 CST Message-Id: <9302151508.AA27168@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 15 Feb 93 11:32:26 GMT <21543.9302151132@subnode.epcc.ed.ac.uk> Subject: various X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 06:32:39 EST Date: Mon, 15 Feb 93 11:32:26 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk ... Just one clarification: ... d) Heterogeneous issues and data conversion: Provision of data type conversion in MPI is gaining support. This makes sense to me, since a lot of PVM users are (presumably) taking advantage of such provisions in PVM. The proposal of Bill and Rusty contains a set of routines which provide data type conversion. Unfortunately the capabilities provided are weaker than PVM, since they allow data conversion in messages of the form "array of datatype" only. Actually, our proposal is somewhat stronger than this, at least at the "expert" level (level 1): The user can provide an array of structures, each of which is essentially an "iovec" (that is, pointer and length) with an additional field for the datatype. Thus, a message consisting of a pair of integers and an array of doubles could be sent as (please excuse the syntax; this can be cleaned up in any number of ways): mvec[0].len = 2 * sizeof(int); mvec[0].ptr = &i[0]; mvec[0].datatype = MPI_INT; mvec[1].len = 347 * sizeof(double); mvec[1].ptr = &array[0]; mvec[1].datatype = MPI_DOUBLE; Alternate forms of this structure can basically define a "channel program" that allows for things like non-unit-stride access and scatter/gathers. Thus, just as with PVM, a single message may portably contain multiple data types; the difference is in when the data is presented to the interface (one datatype at a time, as in PVM, or all at once, as in our proposal). This raises the issue of user defined data types in C/Fortran90 (of course there is no issue in Fortran77). At least with the PVM interface the programmer can portably communicate such data types, albeit with a little effort and overhead. This is our intent as well. Note that it would be easy to write a tool that would take a structure/derived type definition and generate the appropriate setup/calls in either the PVM case or our proposal (A compiler could do it to, but I suspect that initially many would be happy not to have the compiler tinkered with, particularly since such enhancements would not be portable). Bill and Rusty From owner-mpi-pt2pt@CS.UTK.EDU Mon Feb 15 10:21:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04647; Mon, 15 Feb 93 10:21:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20871; Mon, 15 Feb 93 10:21:27 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Feb 1993 10:21:25 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20820; Mon, 15 Feb 93 10:20:39 -0500 Date: Mon, 15 Feb 93 15:19:51 GMT Message-Id: <21840.9302151519@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: various To: gropp@antares.mcs.anl.gov (William Gropp) In-Reply-To: William Gropp's message of Mon, 15 Feb 93 09:08:24 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Hi Bill and Rusty Thanks for the clarification. I hadn't picked up the 'datatype' field in the descriptor structure for iovec messages - I can't find it in the proposal document. What you propose here for level 1 is indeed as strong as PVM - it may be more convenient also. Of course the iovec is then a bit less like the kind of iovec that Un*x programmers are familiar with, and used in this way it makes sense to check that the iovec structures at sender and receiver have some kind of consonance. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Wed Mar 3 15:26:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08992; Wed, 3 Mar 93 15:26:59 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03642; Wed, 3 Mar 93 15:25:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 3 Mar 1993 15:25:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03627; Wed, 3 Mar 93 15:25:50 -0500 Date: Wed, 3 Mar 93 20:25:31 GMT Message-Id: <6861.9303032025@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: safe, unsafe and erroneous MPI programs To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk MPI, more specifically the point-to-point component, describes three classes of programs: safe, unsafe and erroneous. I'm thinking about buffer resource exhaustion here. It is possible to write safe programs in MPI. It is not possible to identify a subset of MPI which is guaranteed to provide either safe or erroneous programs, and cannot produce unsafe programs. We think that MPI should contain such an identifiable subset. This subset was provided by the SYNCHRONOUS completion mode proposed by Marc Snir, which was rejected at the last meeting. I could not support the proposal because I do not believe that the completion mode proposed is actually what users want, and did not reflect common practice. Primary proposal ---------------- I propose that a further completion mode should be added to MPI, in addition to the surviving REGULAR completion mode, which reflects common practice and creates the identifiable subset of MPI which. I propose that a SECURE (for want of a better name) completion mode be added, which has the following semantics: (a) The COMPLETE operation will wait until the user buffer for send is clear for use. In send the data has been copied from the user buffer (without specification of where it has been copied to), and in receive the data has been copied to the user buffer. AND (b) The COMPLETE operation will wait until the sender (receiver) has performed the START operation which matches the receiver (sender). Note that (a) is just the REGULAR completion mode, and (b) has no additional meaning for a receiver. It really just applies to senders. With this addition, the identifiable subset of MPI which produces either safe or erroneous programs is that which contains the SECURE complete mode and does not contain the READY start mode. The program Process 1 Process 2 --------- --------- secure send to 2 secure send to 1 blah blah blah secure recv from 2 secure recv from 1 is erroneous (by definition?). It will deadlock with every correct implementation of MPI. The program Process 1 Process 2 --------- --------- nonblocking secure send to 2 nonblocking secure send to 1 nonblocking secure recv from 2 nonblocking secure revc from 1 blah blah blah wait for send to 2 wait for send to 1 wait for recv from 2 wait for recv from 1 is safe (by definition?). It will not deadlock with any correct implementation of MPI. Secondary proposal ------------------ MPI should not contain a combination of SECURE complete mode and READY start mode. I can see no sense in such a combination, and leaving it out decreases the overlap between the safe subset of MPI and the potentially unsafe subset of MPI. /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Wed Mar 3 16:54:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11917; Wed, 3 Mar 93 16:54:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08053; Wed, 3 Mar 93 16:53:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 3 Mar 1993 16:53:07 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08044; Wed, 3 Mar 93 16:53:05 -0500 Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA20435; Wed, 3 Mar 93 13:52:55 PST Message-Id: <9303032152.AA20435@SSD.intel.com> To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu, prp@SSD.intel.com Subject: Re: safe, unsafe and erroneous MPI programs In-Reply-To: Your message of "Wed, 03 Mar 93 20:25:31 GMT." <6861.9303032025@subnode.epcc.ed.ac.uk> Date: Wed, 03 Mar 93 13:52:31 -0800 From: prp@SSD.intel.com I agree that there should be a way to produce programs which are known to be safe. However, I also think that once a program is proven safe, it should be easy to allow it to take advantage of buffering when available. To make it easy, I recommend a global safeness mode, rather that one that must be selected on each send and receive. I would suggest that the environmental management section contain a call in which the user can specify that all operations are to have SECURE semantics if coded as normal semantics, and those coded with READY semantics would either be proved correct (might be hard) or flagged as erroneous. This way, a program can be proved safe and then released from SECURE semantics with a single line change, instead of going through all the sends and receives and changing them from SECURE to normal. This is particularly important if we decide to distinguish modes by syntax rather than by a mode parameter. (I think we should use syntax because it can be more efficient.) So, I would propose a call like this: MPI_safe() which, if present, will cause the implementation to flag an unsafe program as erroneous. Alternatively, it could be a link option (that is, the MPI standard would require that the implementor provide a means of specifying that a program is to be created or executed in such a way as to report an error if any unsafe operation is attempted.) This would be analogous to lint for C programs. You use it when you need it, and the rest of the time it is not in the way. Paul From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 06:57:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13100; Thu, 4 Mar 93 06:57:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19151; Thu, 4 Mar 93 06:55:09 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 06:55:08 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19137; Thu, 4 Mar 93 06:55:05 -0500 Date: Thu, 4 Mar 93 11:55:00 GMT Message-Id: <7359.9303041155@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: safe, unsafe and erroneous MPI programs To: prp@SSD.intel.com In-Reply-To: prp@SSD.intel.com's message of Wed, 03 Mar 93 13:52:31 -0800 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu I'm afraid that I must disagree with the notion of a global SECURE mode as Paul has suggested, which can be switched on or off as the programmer/user desires. In the first case, we really must think about programs being written to include compositions of parallel libraries which have already been written and determined to be correct and secure. So the library writer should have control of how the library communications are performed, and this kind of "global change the semantics of everything" should be avoided and discouraged. In the second case, once a module has correctly and safely been written using secure communications I really don't think there will be so much to "take advatage of" in use of system buffering. All of my colleagues at EPCC program in this way, and all of our library work is done in this way. Using the secure communications and blocking/nonblocking forms one can write, without difficulty, secure and efficient software. In fact, more efficiently, in principle, since less memory copies of message data are needed. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 07:17:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20778; Thu, 4 Mar 93 07:17:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19895; Thu, 4 Mar 93 07:16:45 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 07:16:43 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19885; Thu, 4 Mar 93 07:16:36 -0500 Date: Thu, 4 Mar 93 12:16:27 GMT Message-Id: <7397.9303041216@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Task identifiers To: mpi-pt2pt@cs.utk.edu In-Reply-To: Tony Skjellum's message of Wed, 3 Mar 93 16:01:25 CST Reply-To: lyndon@epcc.ed.ac.uk > I think that we agreed to defer dynamic issues to a future MPI standard > level, but I do agree that this is never guaranteed to happen, as Rusty > has said. Unless of course we all want to and can afford from our funding more beano trips to Dallas ;-) (joking, honest guv!) > The lack of control for processes, and the lack of dynamic > group syntax/semantics worries me. They will almost certainly appear as > non-portable extensions. Yup, probably, over what timescale do we think? > Perhaps the original N-month limit was too short to permit the best > possible standard. > Perhaps, but on the other hand dynamic groups (i.e., those where the membership changes in time during the existence of the group, if I understand the emergent MPI jargon correctly (hey, we need a glossary of MPI terminology, any volunteers?)) really do seem to be at an experimental stage in our worlds. We have dynamic groups in the system we have implemented (CHIMP), but to be quite honest hardly any users have really used them to implement dynamic groups, rather they have implemented static groups (i.e., those where the membership does not change in time during the existence of the group, if I understand the emergent MPI jargon correctly (hey, we need a glossary of MPI terminology, any volunteers?)). Dynamic groups have been put into PVM3, as experimental functions which may or may not survive in the current or some other form. In short, it seems to be just too early to try to define a standard for dynamic groups (glossary), which is of course unfortunate. I'd suggest strongly that we must have an open mind toward - or even plan for - a later invocation of MPI, at which point subjects such as dynamic group and dynamic process models are dealt with. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 09:48:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28327; Thu, 4 Mar 93 09:48:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27405; Thu, 4 Mar 93 09:47:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 09:47:39 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27396; Thu, 4 Mar 93 09:47:38 -0500 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17713; Thu, 4 Mar 1993 09:47:36 -0500 Date: Thu, 4 Mar 1993 09:47:36 -0500 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9303041447.AA17713@gstws.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: So many proposals, so few standards... I just wanted to put in my endorsement for Bill and Rusty's proposal to have three levels. The lowest having Marc's init, start, wait, free, etc. constructs. The next level having send and receives using buffer descriptors. The highest level send and receives specify the buffer and the datatype and send only contiguous bytes. I also like the idea of having opaque task id. This has been brought up by Peter, Tony, and now Bill and Rusty. There are nice features that arise from this approach that makes computing in a dynamic, heterogeneous environment much easier and efficient. PVM has now gone to this method of task naming because of the advantages. There is still a need for users to think in terms of 0-[p-1] and this is logically provided by having a rank in a group. I also agree with Bill and Rusty that to write portable code we have to have a standard way to start up one or more tasks. The existing interfaces all provide some mechanism for doing this. Are we going to propose a standard that doesn't even provide some of the basic features of the exiting interfaces? With regard to opaque task id, spawning tasks, and dynamic groups, I see a double standard being applied that I do not like. When we discuss communication routines, many people vote for new totally untried (maybe unworkable) concepts because "they could lead to more efficient programs in the future." Fine. But when we talk about things like dynamic groups, people start saying "gosh, never been tried before. We better not do it." Some of our standard effort is future looking and other parts are backwards looking. I feel it is time to try to be consistent. One final thought. If we do not design an interface that is as flexible, intuitive, and easy to use as existing interfaces, then we will have simply created Yet Another Message Passing interface (YAMP) and not a standard. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 11:17:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00544; Thu, 4 Mar 93 11:17:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01967; Thu, 4 Mar 93 11:14:31 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 11:14:29 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01959; Thu, 4 Mar 93 11:14:28 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05686; Thu, 4 Mar 93 10:11:41 CST Date: Thu, 4 Mar 93 10:11:41 CST From: Tony Skjellum Message-Id: <9303041611.AA05686@Aurora.CS.MsState.Edu> To: mpi-pt2pt@cs.utk.edu, geist@gstws.epm.ornl.gov Subject: Re: So many proposals, so few standards... I concur with Al. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 11:21:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00629; Thu, 4 Mar 93 11:21:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02226; Thu, 4 Mar 93 11:19:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 11:19:50 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02173; Thu, 4 Mar 93 11:19:42 -0500 Date: Thu, 4 Mar 93 16:19:30 GMT Message-Id: <7649.9303041619@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: So many proposals, so few standards... To: geist@gstws.epm.ornl.gov (Al Geist), mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Thu, 4 Mar 1993 09:47:36 -0500 Reply-To: lyndon@epcc.ed.ac.uk Okay, this is mainly about task identifiers again. [stuff deleted] > I also like the idea of having opaque task id. This has been brought up > by Peter, Tony, and now Bill and Rusty. There are nice features that > arise from this approach that makes computing in a dynamic, heterogeneous > environment much easier and efficient. PVM has now gone to this > method of task naming because of the advantages. > There is still a need for users to think in terms of 0-[p-1] > and this is logically provided by having a rank in a group. if (MPI will provide process groups) then One thing processes do really find useful - from hard experience - is to be able to receive a message from any member of the process group. If we frame point-to-point in terms of some opaque (global) task identifier type then how can we provide the very expressive receive selection above? If we can't, then opaque types do not seem capable of providing expressive power for process groups. fi > I also agree with Bill and Rusty that to write portable code > we have to have a standard way to start up one or more tasks. > The existing interfaces all provide some mechanism for doing this. > Are we going to propose a standard that doesn't even provide > some of the basic features of the exiting interfaces? Either one includes the task creation in the source code of the program in which case MPI somehow ought to look at a mechanism compatible with the overall conceptual content of MPI Or one puts the task creation outside the program and outside the domain of MPI. This also seems to boil down to whether we have a dynamic process model or a static process model. When we decide this then we can look at the details, but I really do think MPI should agree which the goal is before trying to reach consensus on the attack plan. Why don't we email vote on this? My vote: static model. > With regard to opaque task id, spawning tasks, and dynamic groups, > I see a double standard being applied that I do not like. > When we discuss communication routines, many people vote for > new totally untried (maybe unworkable) concepts because > "they could lead to more efficient programs in the future." Can we find examples of this? Let me see off the top of my head ... a) Communication handles, unless I missed somthing. b) Ready sender communication (symmetry partner of ready receiver) c) Buffer descriptiors with data types > Fine. But when we talk about things like dynamic groups, > people start saying "gosh, never been tried before. We better not do it." > Some of our standard effort is future looking and other parts > are backwards looking. I feel it is time to try to be consistent. Unfortunately it is the burden of a standards committee that it should standardise only agreed existing practice, in such a manner that imminent developments are anticipated and a route for extension of the standard at such a time that such developments become common practice is practical, insofar as the standards committee is able to so do. My phrasing may not be the best, but the principle should be perfectly clear. > One final thought. If we do not design an interface that is > as flexible, intuitive, and easy to use as existing interfaces, > then we will have simply created Yet Another Message Passing interface (YAMP) > and not a standard. i.e., it will not be acceptable to a sufficiently large portion of the market and therefore will not be accepted by the market as a 'de facto' industry standard. How about some serious suggestions regarding how MPI can ensure to avoid this apocalypse? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 4 12:53:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03951; Thu, 4 Mar 93 12:53:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07822; Thu, 4 Mar 93 12:50:08 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 4 Mar 1993 12:50:06 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07811; Thu, 4 Mar 93 12:50:05 -0500 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA17511; Thu, 4 Mar 1993 12:50:03 -0500 Date: Thu, 4 Mar 1993 12:50:03 -0500 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9303041750.AA17511@gstws.epm.ornl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Reply to Lyndon >is to be able to receive a message from any member of the process group. >If we frame point-to-point in terms of some opaque (global) task >identifier type then how can we provide the very expressive >receive selection above? The receive side is easy, and if you REALLY mean process group above then the send side is also. But if you refuse to use the rank in a group then knowing who to send to is the trick. >Either one includes the task creation in the source code of the program >Or one puts the task creation outside the program... Putting it outside only works for multprocessors. What if you have multiple multiprocessors or just a bunch of workstations on a network? Users would have to write a separate (non-portable) program just to start up their applications. The argument still boils down to portability. >Why don't we email vote on this? My vote is dynamic model >Unfortunately it is the burden of a standards committee that it should >standardise only agreed existing practice, This is what I read as one of our original GOALS in MPI. But that is not what I see happening. We either need to stick to the original goal and drop all the "fringe" stuff out of the communication specification, or we take a consistently forward-looking approach (which can't possibly be agreed on in two more meetings) but may be the only way that we >can ensure to avoid this apocalypse? Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 03:26:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23608; Fri, 5 Mar 93 03:26:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22531; Fri, 5 Mar 93 03:25:56 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 03:25:55 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22521; Fri, 5 Mar 93 03:25:52 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 4 Mar 93 13:22 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA11676; Thu, 4 Mar 93 13:21:48 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA10514; Thu, 4 Mar 93 13:21:44 PST Date: Thu, 4 Mar 93 13:21:44 PST From: d39135@sodium.pnl.gov Subject: RE: Reply to Lyndon To: geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303042121.AA10514@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Al Geist writes: > Putting it outside only works for multprocessors. What if you have > multiple multiprocessors or just a bunch of workstations on a network? > Users would have to write a separate (non-portable) program just > to start up their applications. I am puzzled by these words. There is at least one package (TCGMSG, by Robert Harrison) that uses external invocation and works quite nicely on workstation networks as well as multiprocessors. In TCGMSG, the programming interface is that the application just calls PBEGIN_(argc,argv) to initialize, then finds out the number of participating processes by calling NNODES(). (This is a static process model.) For workstations, TCGMSG supplies a general purpose program to start up application processes. It works by passing them command-line flags that allow them to find each other. The application does not need to worry about these flags, because they are handled by the PBEGIN_ . On multiprocessors, other procedures may be used. For example, on the DELTA, the application processes are initiated by the Intel utility 'mexec', and PBEGIN_ works by calling the Intel intrinsic numnodes(). > The argument still boils down to portability. Exactly. As discussed above, the start-em-up-externally approach used by TCGMSG allows a program to execute without change across widely varying environments, with no need for the application programmer to write non-portable code. It would have been extremely difficult, verging on impossible, to support an interface that explicitly allows an application to create new processes on the fly. (That capability is not provided by the DELTA operating system. But they do support sockets, so I guess you could have a demon on some other system that would rsh back to the delta and do another mexec, then use some undocumented and risky system interfaces to pass messages between the nodes of the two mexec's, and then... But that doesn't seem very robust.) > >Why don't we email vote on this? I propose that MPI simply mandate the existence of a capability to start up processes externally. Those processes find themselves executing spontaneously and then find out about the others by making MPI calls. The syntax and semantics of the MPI calls should be specified completely, but we should not say anything about the external capability except that it exists. --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 09:45:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27927; Fri, 5 Mar 93 09:45:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14009; Fri, 5 Mar 93 09:43:32 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 09:43:30 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14000; Fri, 5 Mar 93 09:43:26 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA22099 (5.65c/IDA-1.4.4 for ); Fri, 5 Mar 1993 09:43:22 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA05607; Fri, 5 Mar 93 14:43:17 GMT Date: Fri, 5 Mar 93 14:43:16 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303051443.AA05607@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02088; Fri, 5 Mar 93 14:40:29 GMT To: mpi-pt2pt@cs.utk.edu Subject: Cancel Content-Length: 4971 People, I believe that any of the non-blocking operations which MPI mandates should be cancellable. Why have cancel ? ================= There are various reasons for this (in random order) 1) Resource liberation Some (? ALL ?) of these operations commit system resources. If the user knows that the operation can never complete, then she should be able to release those resources. 2) Access revocation Some of these operations give the MPI system authority to modify areas of the user's store. (non blocking receive in particular). The user should be able to revoke this permission. [See examples with receive buffers on the stack for the true potential horror here !] 3) Cleanliness In some of Marc's early documents (and maybe still) there is an implication that all messages MUST have been received at program exit, otherwise the program is erroneous. It seems reasonable also to require all non-blocking operations to have completed (even MPI -1.1's non blocking exit call completes at this point !), if this is so then we need to be able to force the completion of these operations. (This is particularly the case for server type objects which may wish to queue many non-blocking receives [so as to avoid system buffering and additional data copying], but then wish to exit, possibly returning to the rest of the code...). What does cancel mean ? ======================= Cancelling an operation forces its completion with an error status of CANCELLED. One can reasonably require that the cancelled operation then be tested for completion in the normal way that any such non-blocking operation would be. Cancelling an operation can FAIL. This occurs if the operation to be cancelled has already successfully completed (Entered the state in which it could be successfully found as complete) or the system can guarantee its completion in a bounded time (in which case the cancel call should block until the operation completes). After an operation has been cancelled, I know that either 1) The operation completed normally or 2) The operation will never now proceed (it has been cancelled) The two cases can be separated by testing the completion return from the normal WAIT or STATUS call. (This is preferable to returning this result as the status of the cancel call as it makes the case of one thread cancelling a call on which another thread is already waiting resolve correctly). Consider for example cancelling a non-blocking send operation, there are three possible cases :- 1) data transfer has not actually started => it never starts, the operation is really cancelled, returning a CANCELLED kind of result. 2) data transfer has already completed => cancel failed and returns a COMPLETED kind of result 3) data transfer has started, so it is allowed to complete then reduces to case 2) Note that cancellation is an operation which happens to a REQUEST, it is not an operation which is applied to a MESSAGE. Therefore once a particular receive has been (successfully) cancelled the system state is as if that receive had never existed. There is NO implication that the cancel has an effect on any future incoming message which would have matched the, now cancelled, receive. Such a message can still be received in the normal way by any suitable matching receive call. Cancelling Group operations =========================== If we have non-blocking group operations we should be able to cancel them. It seems entirely reasonable to require that all the participants who have started the operation have also to cancel it. Issues ====== People are nervous about cancel because there is a race condition involved. This race is between the cancel operation itself, and the external event which signals completion of the operation being cancelled. (e.g. the cancel and the incoming message on a non-blocking receive). However this race is no worse than the existing one between a receive and an incoming message. (Does the data go straight into the user buffer, or require system buffering ??) Implementing a full cancel as described above is non-trivial, since it may be necessary to send out a cancel message after a non-blocking send. (This is the case if one uses receiver buffering). If this is viewed as too complex, (however note that various people already have implementations of cancel which implement semantics similar to those described above), then at the very least we should still seriously consider including the ability to cancel operations which gave the system permission to write into our address space. (Thanks to Lyndon for some of the examples and other discussion). -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 10:05:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28310; Fri, 5 Mar 93 10:05:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15328; Fri, 5 Mar 93 10:04:29 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 10:04:26 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15320; Fri, 5 Mar 93 10:04:25 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA06600; Fri, 5 Mar 93 09:01:26 CST Date: Fri, 5 Mar 93 09:01:26 CST From: Tony Skjellum Message-Id: <9303051501.AA06600@Aurora.CS.MsState.Edu> To: geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu, d39135@sodium.pnl.gov Subject: RE: Reply to Lyndon Rik, Using the Delta's lack of an elegant process model to defend your point is not really valid. They could have supported dynamic process creation, but just were too busy with other things, and did not see how useful the previously supported feature(s) were from iPSC960 and iPSC/1&2. The real issue is whether anything is lost/gained by separating spawning and MPI messaging. Why might there be? 1) ability to spawn processes in a way to take advantage of special group properties (hierarchy in the memory/network access). Oh well, the user can still do this him/herself, and then tell MPI that the group has a special property. 2) ability to support dynamic process model to allow MPI to serve as a base layer for additional services. For true portability, I would like to avoid having MEXEC on Delta, and spawn() functions on iPSC860 and CE/RK, and so on. The lack of a well-understood need for group, so that group-scope operations, can be attributed to a group, have not been recognized in MPI. As each group might want to implement a different broadcast or combine or other operations, the lack of a well-formed group concept is the bottleneck to this discussion. I am sleepy, I can't think of more now. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 11:07:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00459; Fri, 5 Mar 93 11:07:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20433; Fri, 5 Mar 93 11:06:47 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 11:06:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20425; Fri, 5 Mar 93 11:06:37 -0500 Date: Fri, 5 Mar 93 16:06:03 GMT Message-Id: <8555.9303051606@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Cancel To: jim@meiko.co.uk (James Cownie), mpi-pt2pt@cs.utk.edu In-Reply-To: James Cownie's message of Fri, 5 Mar 93 14:43:16 GMT Reply-To: lyndon@epcc.ed.ac.uk Hello world :-) I just like to say that I generally support Jim's proposal. I see three sensible options, at this point: (1) Cancel for nonblocking receive operations (2) Cancel for nonblocking point-to-point operations (i.e., send and receive) (3) Cancel for nonblocking operations (i.e., point-to-point and collective) [How about cancellable nonblocking group partitions? :-)] I will certainly vote for (2) - I don't suppose there are any surprises there. I expect that I will find in favour of (3), but I really have to think through implementation first, so I'm reserving judgement. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 13:07:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03718; Fri, 5 Mar 93 13:07:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27705; Fri, 5 Mar 93 13:06:17 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 13:06:15 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27686; Fri, 5 Mar 93 13:06:11 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Fri, 5 Mar 93 10:01 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA13638; Fri, 5 Mar 93 10:00:28 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA11163; Fri, 5 Mar 93 10:00:25 PST Date: Fri, 5 Mar 93 10:00:25 PST From: d39135@sodium.pnl.gov Subject: static process creation To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303051800.AA11163@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Tony Skjellum writes: > Rik, > > Using the Delta's lack of an elegant process model to defend your > point is not really valid. They could have supported dynamic process > creation, but just were too busy with other things, and did not see > how useful the previously supported feature(s) were from iPSC960 and > iPSC/1&2. I was addressing the question of whether it is OK for MPI to REQUIRE dynamic process creation, or whether it should include a capability that requires only static processes. My position is that a static process model should be supported. I have no problem with MPI also providing a standard interface to dynamic processes in environments where that can be done. But I strongly feel that a dynamic process interface should not be the only one. In the note that Tony refers to, I presented existence arguments that: a) The static process model can be supported cleanly on systems that also support dynamic processes, such as workstation networks. (Someone else had asserted the contrary.) b) There exists at least one system (the Delta) that MPI seems to target, and that does not support dynamic process creation. This still seems like a valid argument for supporting static processes. I agree that the argument would be stronger with a longer list of systems that do not adequately support dynamic process creation. I used the Delta because that's one example that I'm sure of. I am also sure that the iPSC/860 under NX does not have dynamic process creation. I believe that the iPSC/1 and /2 did have it, but I am not so sure that their model/implementation allowed a process running on a node to create a process on another node. That capability seems to be assumed by the arguments about why we want dynamic process creation. If more examples are really needed, perhaps they can be provided by people who have the facts. I am not sure, as examples only, that the CM-5, NCube-2, or Meiko systems provide all the dynamic process support that's required. I'd be pretty surprised if Intel was the only vendor with current offerings that don't. But my guess is that this is one of those issues where we have an extended discussion and then all vote one way. If we're going to vote, then I propose that the issue be phrased "Should MPI support a static process model, or should it support ONLY a dynamic process model?" Is there actually debate on this issue? Can anyone argue why a static process model should NOT be supported? --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 13:21:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04185; Fri, 5 Mar 93 13:21:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28589; Fri, 5 Mar 93 13:21:06 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 13:21:04 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28519; Fri, 5 Mar 93 13:20:55 -0500 Date: Fri, 5 Mar 93 18:20:50 GMT Message-Id: <8662.9303051820@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: static process creation To: mpi-pt2pt@cs.utk.edu In-Reply-To: d39135@sodium.pnl.gov's message of Fri, 5 Mar 93 10:00:25 PST Reply-To: lyndon@epcc.ed.ac.uk Rik writes: > I used the Delta because that's one example that I'm sure of. I am > also sure that the iPSC/860 under NX does not have dynamic process > creation. I believe that the iPSC/1 and /2 did have it, but I am not > so sure that their model/implementation allowed a process running on a > node to create a process on another node. That capability seems to be > assumed by the arguments about why we want dynamic process creation. > > If more examples are really needed, perhaps they can be provided by > people who have the facts. I am not sure, as examples only, that the > CM-5, NCube-2, or Meiko systems provide all the dynamic process > support that's required. I'd be pretty surprised if Intel was the > only vendor with current offerings that don't. > FYI: Meiko Computing Surface - i.e., the machine they can deliver today as opposed to the "new" machine - don't allow dynamic process creation. I believe that CM-5 only allows you to kick off an SPMD program, not very dynamic really. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 17:47:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11144; Fri, 5 Mar 93 17:47:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12931; Fri, 5 Mar 93 17:45:54 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 17:45:53 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12922; Fri, 5 Mar 93 17:45:51 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA07602; Fri, 5 Mar 93 17:45:31 -0500 Date: Fri, 5 Mar 93 17:45:31 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303052245.AA07602@msr.EPM.ORNL.GOV> To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, mpi-pt2pt@cs.utk.edu, tony@Aurora.CS.MsState.Edu Subject: Re: static process creation Hi Rik, Just a point of fact: we just got through testing a fully dynamic model on the Intel i860. It is a part of the PVM 3.0 software. Maybe the delta is the ONLY machine. Good thing only one was built. I think your phrasing is ment to sway the reader to vote a particular way (your way of course) We shouldn't loose track of the fact that a static model is just a subset of the dynamic model. There is no such thing as ONLY a dynamic model. I can not argue that a static model should not be supported. This is not the debate. The issue is should we limit MPI to a static model (We haven't even voted to have process creation of any kind yet) given that a more flexible model interface already exists (PVM and others) but maybe a better argument is the ability to write more fault tolerant applications with a dynamic model. Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 20:48:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14007; Fri, 5 Mar 93 20:48:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20019; Fri, 5 Mar 93 20:46:55 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 20:46:54 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20011; Fri, 5 Mar 93 20:46:51 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Fri, 5 Mar 93 17:38 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA14836; Fri, 5 Mar 93 17:36:52 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA12820; Fri, 5 Mar 93 17:36:39 PST Date: Fri, 5 Mar 93 17:36:39 PST From: d39135@sodium.pnl.gov Subject: static process subset? To: d39135@sodium.pnl.gov, geist@gstws.epm.ornl.gov, geist@msr.EPM.ORNL.GOV, mpi-pt2pt@cs.utk.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303060136.AA12820@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Al Geist writes: > Just a point of fact: we just got through testing a fully dynamic model > on the Intel i860. It is a part of the PVM 3.0 software. Great! Can you share with us how you did it, and what is the performance of the communications between dynamically allocated processes on two i860 nodes? I have visions of layers of daemons (like my tongue-in-cheek description of how to do dynamic processes on the Delta), but then I'm a notorious skeptic. > I think your phrasing is ment to sway the reader to vote a particular way > (your way of course) This strikes me as a personal attack, although of course I have no idea how it was intended. I will resist responding other than to ask, we can focus on the technical issues? > We shouldn't loose track of the fact that > a static model is just a subset of the dynamic model. > There is no such thing as ONLY a dynamic model. Here's an example of what I mean by "supporting ONLY a dynamic model". Suppose we define an interface that says the only way to get P processes running is to write code that says for (i=1; i I can not argue that a static model should not be supported. > This is not the debate. The issue is should we limit MPI > to a static model (We haven't even voted to have process creation > of any kind yet) given that a more flexible model interface > already exists (PVM and others) Perhaps this is an area where MPI needs to have a defined subset. I think it is important that implementations can call themselves "MPI-compliant" without having dynamic process creation. I also think that the MPI interfaces should be defined such that static process creation is cleanly supported. At the same time, I agree that dynamic process models are important. I would NOT vote to restrict the design of MPI to a static process model, in the sense of defining fundamental features that are not compatible with dynamic processes. For this reason, I prefer process ID's to be opaque types rather than sequential integers. I might however vote to do only static now because of time constraints. This is a question of priority. My personal top priority is getting pt-pt and group/context/topology definitions such that rock-solid libraries can be written that will run fast at least in static process environments. Even that much seems to be taking a gosh-awful long time. Dynamic processes would probably be my second priority. Given the current schedule and rate of progress, I don't think I need a third (low) priority. However, (to switch topics a bit) if I did need to name low priority stuff, then certainly figuring out how to implement multiple non-blocking scatter-gather operations on overlapping groups would fall someplace below third. That stuff sounds great for an application, but it's the devil's own time to implement correctly, even given a good definition. So that stuff scares me -- I'm afraid that if we require it, NOBODY will have a compliant implementation. Does anybody else worry about this? > Al Geist --Rik Littlefield From owner-mpi-pt2pt@CS.UTK.EDU Fri Mar 5 22:26:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14706; Fri, 5 Mar 93 22:26:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23643; Fri, 5 Mar 93 22:26:00 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 5 Mar 1993 22:25:59 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23635; Fri, 5 Mar 93 22:25:56 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA22624 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Fri, 5 Mar 1993 19:25:53 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA19714; Fri, 5 Mar 93 19:18:53 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA09061; Fri, 5 Mar 93 19:25:16 PST Date: Fri, 5 Mar 93 19:25:16 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303060325.AA09061@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: Agreeing with Rik I fully agree with Rik Littlefield's comments and am very worried that we are running out of time. My personal preferences at this point would be to prioritize various things as: a) Get collective communication down properly. Include all the (blocking) routines that applications could benefit from - even obscure ones. b) Try to agree on some minimal way of creating a program and identifying a process in it. c) Define point-to-point message passing. d) Worry about "other things" like dynamic process models and non-blocking concatenate and its friends. I put collective communication ahead of point-to-point in my list because I am EXTREMELY concerned about the complexity of the current point-to-point design. My feeling is that a complex point-to-point layer is only important if it allows applications to be written that execute faster than they would with a simpler layer. In my experience, especially with number-crunching programs, users most benefit from highly optimized collective communication routines, not a vast array of point-to-point routines. One might argue, of course, that the sophistication in the point-to-point layer is exactly so that implementors can make highly optimized collective comm. layers. I don't agree with this for the simple reason that I don't think it's obvious that any particular layer in the point-to-point spec. is faster than any other. If this is so I would imagine that collective communication calls will be optimized independently for each machine (as they are now) rather than being built "portably" from the point-to-point layers. Thus the sophistication in the low level is probably wasted. Of course, many people will want to build their codes from simple point-to-point routines, but I wonder if this doesn't just mean that we're missing some important collective ones, at least in some cases. To respond to one of Al Geist's comments it bothers me too that we seem to be accepting vast quantites of point-to-point stuff that certainly isn't current practice (some of it doesn't even exist in any other system) and yet are wary of adding things like smart collective routines and toplogy support that are in day-to-day use in a lot of applications. Jon Flower p.s. By the way, my vote would be for opaque task ID's for hypothetical forward-compatibility, ..... but I think they're a royal pain to use!! In every case where I've had to do this I've ended up making an array in each node, collecting the opaque ID's and converting them to consecutive integers. From owner-mpi-pt2pt@CS.UTK.EDU Sat Mar 6 13:58:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02798; Sat, 6 Mar 93 13:58:21 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08785; Sat, 6 Mar 93 13:57:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 6 Mar 1993 13:57:13 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08762; Sat, 6 Mar 93 13:57:10 -0500 Date: Sat, 6 Mar 93 18:57:01 GMT Message-Id: <328.9303061857@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: static process creation To: mpi-pt2pt@cs.utk.edu In-Reply-To: d39135@sodium.pnl.gov's message of Fri, 5 Mar 93 10:00:25 PST Reply-To: lyndon@epcc.ed.ac.uk I sent this yesterday, but it seems to have dissapaered into the great bit bucket in the sky (nice phrase, Jim :-). Rik writes: > If more examples are really needed, perhaps they can be provided by > people who have the facts. I am not sure, as examples only, that the > CM-5, NCube-2, or Meiko systems provide all the dynamic process > support that's required. I'd be pretty surprised if Intel was the > only vendor with current offerings that don't. FYI: Meiko Computing Surface - i.e., the machine they can deliver today as opposed to the "new" machine - don't allow dynamic process creation. I believe that CM-5 only allows you to kick off an SPMD program, not very dynamic really. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Sat Mar 6 14:17:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03026; Sat, 6 Mar 93 14:17:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09402; Sat, 6 Mar 93 14:17:20 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 6 Mar 1993 14:17:19 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09388; Sat, 6 Mar 93 14:17:17 -0500 Date: Sat, 6 Mar 93 19:17:14 GMT Message-Id: <346.9303061917@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: static process creation To: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Fri, 5 Mar 93 17:45:31 -0500 Reply-To: lyndon@epcc.ed.ac.uk Hi Al, Rik, mpi-pt2pt world ... I mailed out two other peices of kit which I understand don't allow dynamic processes. I should have pointed out, I guess, that to the best of my knowledge this a restriction placed on the user by the "system" software, and not by the hardware. If my guess is correct The system architects within the respective companies will have made a decision that this would be the interface which they present to the user, and they *could* have made a different decision. So I hardly think that the argument that there are existing systems which do not provide dynamic process creation really is an argument that those systems should not be asked to provide dynamic process creation within MPI, unless the vendors among us advise that if MPI contains dynamic process creation they will not implement that part of MPI for those systems *and* the existence of such systems which do not implement MPI, or part thereof, will have a significant affect on the market impact and penetration of MPI. I'm hardly a person who is able to say much of value on the last point. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Sat Mar 6 15:37:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03409; Sat, 6 Mar 93 15:37:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12480; Sat, 6 Mar 93 15:36:46 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 6 Mar 1993 15:36:45 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12470; Sat, 6 Mar 93 15:36:41 -0500 Date: Sat, 6 Mar 93 20:36:37 GMT Message-Id: <390.9303062036@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: dynamic processes and dynamic groups To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Hi again Well, we certainly are having some fun now in this discussion of static vs dynamic processes in MPI. This is as much about dynamic groups as dynamic processes, and the latter part is really about MPI as a standards committee. I have a lot of sympathy with John Flower's concerns. There appears to be reasonable consensus among those contributing to the discussion that in a world process model where processes dynamically appear and dissapear in an asynchronous fashion an enumeration of processes is not a useful way of identifying processes - what sense could we make of the enumeration? An opaque type process (or task) identifier is better. The same considerations surely lead us to the observation that in a world group model where processes dynamically join and leave groups in an asynchronous fashion an enumeration of members of a group is not a useful way of identifying members - what sense could we make of the enumeration? Rolf previously pointed this out on mpi-comm - March 4, subject "Process ids". What are asynchronously created (terminated) processes to do? I guess they either join (leave) an existing group, or create (destroy) a group for their own purposes. The primitives for these two are really process group resizing (or destruction/recreation) and process group creation (termination). I guess their must be heavyweight PVM users who have other experiences of usage of dynamic process creation/termination - Al?. These thoughts/questions, however interesting and discussion provoking they may or may not be, do not take us any further on answering some of the BIG questions about what MPI will be, regarding for example: a) Process model; static v dynamic; ... b) Group model; static v dynamic; ... I've previously stated that I'd vote for static process model in (a). This "seemed to be" the intent of MPI (although I couldn't find it written down and recorded as agreed), would be useful, and could be agreed upon within the MPI time schedule. This is not to say that I think limitation to a static process model would be the most useful thing upon which to agree to describe a 'de facto' standard. I have pretty well the same opinions regarding process groups. I'd like us all agree on how we're going to move forward and get the details of the standard described, as I'm concerned that at the current rate we may not. I'll broadly support John Flower's suggestions, primarily because I don't see any others, and I'll do my best to implement whatever strategy we can agree on. Final thought, perhaps we should move the discussion of MPI strategy to mpi-comm. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 11 18:14:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10133; Thu, 11 Mar 93 18:14:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23324; Thu, 11 Mar 93 18:13:15 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 11 Mar 1993 18:13:13 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23312; Thu, 11 Mar 93 18:13:05 -0500 Via: uk.ac.southampton.ecs; Thu, 11 Mar 1993 18:45:16 +0000 Via: brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:38:09 GMT From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:46:23 GMT Date: Thu, 11 Mar 93 18:46:26 GMT Message-Id: <19625.9303111846@holt.ecs.soton.ac.uk> To: mpi-pt2pt@cs.utk.edu Subject: Re: safe, unsafe and erroneous MPI programs L J Clarke writes: > This subset was provided by the SYNCHRONOUS completion mode proposed by > Marc Snir, which was rejected at the last meeting. I could not support > the proposal because I do not believe that the completion mode proposed > is actually what users want, and did not reflect common practice. It was with some dismay that I learned from the above message that the SYNCHRONOUS completion mode had been dropped, having been unfortunately unable to attend the last meeting, for which minutes have not yet been posted. Was this decision due to some awkward aspect of the precise semantics in Marc's proposal, or was it just that the committee does not see the need for any form of end-to-end synchronization mode at all? If the latter is the case then I am astounded. In any case, I would strongly advocate the inclusion of a synchronous, mode of communication, with both blocking and non-blocking variants, for the same reasons elucidated by Lyndon in a message posted before the February meeting: > Given the above, I come to the conclusions: > > i) MPI must contain unbuffered communications with blocking/nonblocking > (irecv/isend/msgwait kinda thing) calls, for reliability and > portability. > > ii) If a goal of MPI is that existing applications using message passing > interfaces (eg Express, PVM) should easily port to MPI, then MPI must > also contain buffered comms. This seems to be a matter for the full > committee, hence I have crossposted. What went wrong at the February meeting? I've not been very vocal on the mpi mailing lists of late, having got behind in reading the mound of messages arriving daily, but I'm back on top of it and hope to be contributing more from now on. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-pt2pt@CS.UTK.EDU Sun Mar 14 22:10:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12927; Sun, 14 Mar 93 22:10:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02224; Sun, 14 Mar 93 22:09:44 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 14 Mar 1993 22:09:42 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02216; Sun, 14 Mar 93 22:09:37 -0500 Message-Id: <9303150309.AA02216@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 8667; Sun, 14 Mar 93 22:09:35 EST Date: Sun, 14 Mar 93 22:09:34 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir \\ William Gropp and Ewing Lusk} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. It does not yet include a description of the Fortran 77 and C bindings. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. \discuss{ The following subsections of the introduction contain general information on the design of MPI procedures. The material should be moved to a general introduction for the entire document. } \subsection{Data Types} \subsubsection{Handle} MPI procedures use at various places {\em handles}. Handles are used to access opaque objects. Such object can be created, updated and destroyed only by by calling suitable MPI procedures, and providing the handle as parameter. Opaque objects hide from the user the internal representation used for various MPI objects, thus allowing to have similar calls in C and Fortran, allowing to overcome problems with the typing rules in these languages, and allowing for future extension of their functionality. Handles are of type {\tt void *} in C and of type {\tt integer} in Fortran. An opaque object can be {\em persistent} or {\em ephemeral}. A persistent object persists until destroyed by an explicit operation. An ephemeral object is good for a single use; thus an ephemeral object associated with a communication operation disappears once this operation is completed (or once this object is not needed anymore for the completion of the operation). An opaque object is created by a call to {\tt MPI\_CREATE}, and destroyed by a call to {\tt MPI\_FREE}. Additional MPI functions are available to create, access and update specific opaque objects. {\bf \ \\ MPI\_CREATE(handle, type, persistence)} \begin{description} \item[OUT handle] handle to object \item[IN type] state value that identifies the type of object to be created (e.g., {\tt MPI\_COMMUNICATION, MPI\_BUFFER, MPI\_CONTEXT}, etc.). \item[IN persistence] state value; either {\tt MPI\_PERSISTENT} or {\tt MPI\_EPHEMERAL}. \end{description} {\bf \ \\ MPI\_FREE(handle)} \begin{description} \item[IN handle] handle to object \end{description} An object can be destroyed only if there is no pending operation that is using this object; after successful return of the routine, the handle is undefined. {\bf \ \\ MPI\_ASSOCIATED(handle, type)} \begin{description} \item[IN handle] handle to object \item[OUT type] state \end{description} Returns the type of the object the handle is currently associated with, if such exists. Returns the special type {\tt MPI\_NULL} if the handle is not currently associated with any object. MPI may provide predefined opaque objects and predefined, static handles to these objects. Such objects may not be destroyed. \paragraph*{List of handles} An MPI call may need a parameter that is a {\em list of handles}. In C, such list will be a record with one component being the length of the list, the other components being an array of pointers. In Fortran, the list will be an array of integers, the first one of which is the length of the list. \discuss{ The mechanism for opaque objects used here follows the POSIX Fortran binding standard. An alternative choice is to have different type declarations for each type of opaque object. Then, opaque objects are created/destroyed like regular variables, rather than by MPI calls; they are still accessed and updated only via MPI functions. } \subsubsection{State} MPI procedures use at various places arguments with {\em state} types. The values of such data type are all identified by names, and no operation is defined on them. For example, the {\tt MPI\_CREATE} routine has a state type parameter with values {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}. An {\tt enumeration} declared in an included MPI.h file will be used in C for state datatypes. The Fortran 77 mechanism needs to be decided. \discuss{ Named integer constants can be used in Fortran 90, using the {\tt PARAMETER} mechanism. The constant declarations can be made available via an {\tt INCLUDE} file. Fortran 77 does not seem to offer any convenient mechanism. One possibility is to specify explicit integer values, and allow the use of named constants with those Fortran 77 compilers that support them conveniently. Another possibility is to use character strings, rather than integers. } \subsubsection{Named constants} MPI procedures sometimes assign a special meaning to a special value of a basic type parameter; e.g. {\tt tag} is an integer valued parameter of point-to-point communication operations, with a special {\tt DONTCARE} value. Such parameters will have a range of regular values, which is a proper subrange of the range of values of the corresponding basic type; special values (such as DONTCARE) will be outside the regular range. The range of regular values can be queried, and sometimes set, using environment inquiry or environment setting functions (Section~\ref{sec:inquiry}). The special values are provided by named constant, that are made available via an MPI.h include file in a C binding. \discuss{ Need to agree on a Fortran mechanism for named constants (see the discussion above). Implementers should try to detect illegal uses of ``special values''. Thus, the use of the {\tt DONTCARE} value to tag a message sent should be flagged as an error. } \subsubsection{Choice} MPI functions sometimes use parameters with a {\em choice} (or union) data type. I.e., distinct calls to the same routine may pass by reference actual parameters of different types. The mechanism for providing such parameters will differ from language to language. In C, a formal parameter of type {\tt void *} will be used, with an actual pointer parameter. in Fortran, we shall cheat. \discuss{ The Fortran 77 standard specifies that the type of actual arguments need to agree with the type of dummy arguments; no construct equivalent to C pointers is available. Thus, it would seem that there is no standard conforming mechanism to support choice parameters. However, most Fortran compiler either don't check type consistency of calls to external routines, or support a special mechanism to link foreign (e.g., C) routines. I suggest that we accept this nonconformity with Fortran 77 standard. I.e., we accept that the same routine may be passed an actual parameter of a different type at distinct calls. Generic routines can be used in Fortran 90 to provide a standard conforming solution. This solution will be consistent with our nonstandard conforming Fortran 77 solution. } \subsection{Processes} An MPI program is executed by several autonomous processes that execute each their own code, in a MIMD style. The codes executed by each process need not be identical. The processes communicate via calls to MPI communication primitives. Typically, each processor executes in its own address space, although shared-memory implementations of MPI are possible. This document specifies the behavior of a parallel program assuming that only MPI calls are used for communication. The interaction of an MPI program with other possible means of communication (e.g., shared memory) is not specified. In particular, it is assumed that message buffers at distinct processors are disjoint. MPI does not specify the execution model for each process. A process can be sequential, or can be multithreaded, with threads possibly executing concurrently. Care has been taken to make MPI ``thread-safe'', by avoiding the use of implicit global states. The initial allocation of processes to an MPI computation and their binding to physical processors is not specified by the program itself. It is expected that vendors will provide mechanisms to do so either at load time or at run time. Such mechanisms will allow to specify the initial number of required processes, the code to be executed by each initial process, and the allocation of processes to processors. Also, the current proposal does not provide for dynamic creation or deletion of processes during program execution, although it is intended to be consistent with such extension. Finally, the current proposal does not specify a naming scheme for processes. We propose to always identify processes according to their relative rank in a context (group), so that, effectively, processes are identified by consecutive integers. Absolute, system-wide unique process id's are (will be) needed only if dynamic process creation is to be supported (in such eventuality we propose to use handles to opaque {\em process structures} for that purpose). \subsection{Contexts} \discuss{ This section contains a proposal for use of contexts that will subsume groups. It borrows heavily on the current group proposal. This proposal has not yet been discussed in MPI meetings.} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \paragraph*{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \paragraph*{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \paragraph*{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \paragraph*{Lightweight gang scheduling} Consider an environment where processes are multithtreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \subsubsection{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \paragraph*{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ If process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. A possible addition is a function of the form {\bf MPI\_CREATE\_CONTEXT(newcontext, list\_of\_process\_handles)} which creates a new context out of an explicit list of members (and rank them in their order of occurrence in the list). This, coupled with a mechanism for requiring the spawning of new processes to the computation, will allow to create a new all inclusive context that includes the additional processes. However, I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } \subsubsection{Error Handling} It is assumed that MPI is implemented on top of an error-free communication subsystem: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, and the likes. In other words, MPI does not provide mechanisms to deal with failures in the underlying communication subsystem -- it is the responsibility of the MPI implementer to insulate the user from such errors (or to reflect them as global program failures). The same holds true for node failures. Of course, MPI programs may still be erroneous. A {\bf program error} can occur when an MPI call is called with an incorrect parameter (non-existing destination in a send operation, buffer too small in a receive operation, etc.) This type of error would occur in any implementation. In addition, a {\bf resource error} may occur when a program exceeds the amount of available system resources (number of pending messages, system buffers, etc.). The occurrence of this type of error depends on the amount of available resources in the system and the resource allocation mechanism used; this may differ from system to system. The recommended implementation profile provides several mechanisms to alleviate the portability problem this represents. One can also write {\bf safe} programs, that are not subject to resource errors. All MPI procedure calls return an error parameter that indicates successful completion of the operation, or the error condition that occurred, otherwise. The recommended implementation profile in a POSIX environment is for any MPI routine that encounters a recoverable error to store an error number in a global variable ({\em errno} in a C environment) and generate an {\em MPI error signal}, using a special signal value. The default handler for this signal terminates the execution of all involved processes, with a suitable error message being returned to the user. However, the user can provide his or her own signal handling routine. In particular, the user can specify a ``noop'' signal handler, thus relegating all error handling to the user code, using the error parameters returned by the MPI calls. MPI calls may initiate operations that continue asynchronously after the call returned. Thus, the operation may return with a code indicating successful completion, yet later cause an error exception to be raised. If there is a subsequent call that relates to the same operation (e.g., a call that verifies that an asynchronous operation has completed) then the error parameter associated with this call will be used to indicate the nature of the error. In a few cases, the error may occur after all calls that relate to the operation have completed, so that no error parameter can be used to indicate the nature of the error (e.g., an error in a send with the ready mode). In such cases, an error will be undetected, if the user disabled the MPI error signal. \discuss{ The alternative choice is to have fatal and non-fatal signals. } \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} The data part of a message consists of a sequence of values, each of a basic datatype in the host language. Thus, in Fortran, a message consists of a sequence of values that are each of type {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt LOGICAL}, or (length 1) {\tt CHARACTER}. A message may mix values of different types. \discuss{ May also need {\tt DOUBLE COMPLEX} in Fortran.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The rank the sending process \item[destination] The rank of the receiving process \item[tag] User defined \item[context] handle \end{description} The range of valid values for the {\bf source} and {\bf destination} fields is {\tt 0 ... n-1}, where {\tt n} is the number of processes in the current context. The ranges of valid values for {\tt tag} is implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. {\tt Context} should be a context shared by both source and destination. The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The actual mechanism used to associate an envelope with a message is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit, and need not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. A buffer consists of a sequence {\bf buffer components}. Each component consists of a sequence variables of the same basic type. There are three basic types of buffer components: \begin{description} \item[block] A sequence of contiguous values of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[datatype] Type of elements \end{description} \item[vector] A sequence of equally spaced and equally sized blocks of elements of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[stride] Number of elements between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk \le stride$) \item[datatype] Type of elements \end{description} Note that a constant stride becomes contiguous when {\tt stride = lenblk}. A vector buffer component can be an arbitrary submatrix of a two-dimensional matrix. \item[indexed] A sequence of elements of the same basic type, specified by \begin{description} \item[start] initial element \item[list\_of\_indices] List of displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} \end{description} For example, if a Fortran array is declared as {\tt double precision a(10)}, then the tuple {\tt $<$(a(3), (0,3,6,2), DOUBLE$>$} specifies a buffer component with entries {\tt a(3) a(6), a(9), a(5)}. \discuss{ Do we allow entries to be repeated in an indexed buffer component? Do we allow different buffer components to overlap? Do we require in an vector buffer component that {\tt len} be a multiple of {\tt lenblk}? } A buffer is described by an opaque object accessed via a {\bf buffer handle}. Such object is created and destroyed via calls to {\tt MPI\_CREATE} and {\tt MPI\_FREE}. It is associated with successive buffer components by calling in succession one of the functions {\tt MPI\_ADD\_BLOCK}, {\tt MPI\_ADD\_VECTOR} or {\tt MPI\_ADD\_INDEX}, in order to append a component to the buffer associated with a buffer handle. A buffer object can be destroyed only if there is no pending communication operation using it. After a buffer object is destroyed the associated buffer handle is undefined. {\bf \ \\ MPI\_ADD\_BLOCK( buffer, start, len, datatype)} \\ Append a block component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_VEC( buffer, len, stride, lenblk, datatype )} \\ Append a vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN stride] Number of elements between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_INDEX( buffer, start, list\_of\_indices, datatype)} \\ Append an indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[start] initial position for indexing (choice) \item[list\_of\_indices] list of relative indices of entries (array of integers) \item[IN datatype] datatype identifier (status) \end{description} Consider, for example, the following fragment of Fortran code \begin{verbatim} DOUBLE PRECISION A(10,20) INTEGER B, C(5,10) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, B, 1, MPI_INT) CALL MPI_ADD_VEC (BH, A(1,3), 11, 4, 2, MPI_DOUBLE) CALL MPI_ADD_INDEX(BH, C(3,7), (4,2,1), MPI_INT) \end{verbatim} Then the buffer associated with the handle {\tt BH} consists of the sequence of variables {\tt B, A(1,3), A(2,3), A(5,3), A(6,3), A(9,3), A(10,3), A(3,4), A(4,4), A(7,4), A(8,4), A(1,5), C(2,8), C(5,7), C(4,7)}. A message created from this buffer will consist of a sequence of one integer, followed by eleven double precision reals, followed by three integers. A buffer handle can be used for communication, even if it is not associated with any variables (i.e., even if it was not set by any {\tt MPI\_ADD\_xxx} call). Such handle is associated with an empty buffer, and a message created from it contains no data. \discuss{ The main modifications w.r.t. the proposal of Gropp and Lusk is measuring length in elements, rather than bytes. Seems more natural, since type is known. Also, object creation uses a generic function, rather than a function specific to buffer descriptors. As result, I gave up on the size parameter. This may not be such a loss: it is not clear that specifying a maximum length at buffer object creation is useful, since indices of arbitrary size may need to be stored in the object. } \subsubsection{Data Conversion} The types and the locations of the entries used to create a message is solely determined from the information in the buffer descriptor, using the storage association rules specified by the host language and its implementation; the type and the locations of these entries do not depend on the declaration for the corresponding variables in the calling program. It is not required that the data types specified in a buffer descriptor match the data types of the corresponding elements in the host program. However, in case of mismatches, the correspondence between entries in the host program and entries in a message created with the buffer descriptor may be implementation dependent. No data conversion occurs when data is moved from a sender buffer into a message. Consider the following fragment of Fortran code \begin{verbatim} REAL A(100) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, A(1), 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, A(2), 1, MPI_INT) CALL MPI_ADD_BLOCK (BH, A(3), 1, MPI_LOGICAL) CALL MPI_ADD_BLOCK (BH, A(4), 1, MPI_COMPLEX) CALL MPI_ADD_BLOCK (BH, A(6), 1, MPI_DOUBLE) CALL MPI_ADD_BLOCK(BH, A(8), 4, MPI_CHAR) \end{verbatim} A message created from this buffer will consist of a sequence containing one real, one integer, one logical, one complex, one double, and four characters. No data conversion occurs when values are copied from the sender buffer to the message. Thus, the first entry in the message is a real number equal to {\tt A(1)}; the second entry is an integer number that happens to have the same binary representation as {\tt A(2)}; the third entry is a logical value that happens to have the same binary representation as {\tt A(3)}; the fourth entry in a complex number with a binary representation identical to {\tt A(4), A(5)}; the fifth entry is a double precision value with a binary representation identical to {\tt A(6), A(7)}; and if words have four bytes then the last four entries are bytes that make the binary representation of {\tt A(8)}, in the byte order of the executing machine. The correspondence between the first seven entries of the array {\tt A} and the first five entries of the message created from this buffer is determined by the rules of Fortran 77 on storage association: Each variable of type {\tt INTEGER}, {\tt REAL}, or {\tt LOGICAL} occupy one {\em numeric storage unit}; a variable of type {\tt DOUBLE} or {\tt COMPLEX} occupy two numeric storage units. Thus, the same correspondence will hold for any implementation. However, different implementations may have different binary encodings of integer, real and logical values, so that the actual values transferred by the message may differ. The correspondence between the entry {\tt A(8)} of the array, and the last four character entries in the message is implementation dependent, since the Fortran language does not specify a correspondence between character storage units and numeric storage units (an array of characters cannot be equivalenced with an array of integers). Different results may occur in big-endians or small-endians machines, or in 32 bit or 64 bit machines. The same holds, symmetrically, when a message is received. Entries are moved from the message into the receiver memory according to the information provided by the buffer descriptor, with no regard to the way the corresponding variables are declared in the receiving program. When data is moved in a homogeneous environment between nodes having the same architecture, then no data conversion occur at any point during data transfer. Assume, in the previous example, that an identically declared buffer descriptor is used to receive data into an identically declared array at the receiving process. Then, when these two nodes communicate, the values in the first eight entries of array {\tt A} of the sender will be copied into the first eight entries of array {\tt A} at the receiver (assuming that reals occupy the same storage as four bytes). In particular, in a homogeneous environment, it is possible to communicate using only character typed messages. When data is moved into a heterogeneous environment between nodes having distinct architectures, data conversion may occur during the transfer: Each entry is converted from the data representation used on the sending node to the data representation used in the receiving node. Consider the following fragment of Fortran code. \begin{verbatim} REAL X, Y CHARACTER (4) Z INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, X, 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, Y, 4, MPI_CHAR) CALL MPI_ADD_BLOCK (BH, Z, 4, MPI_CHAR) \end{verbatim} Assume that the same arrays {\tt X, Y, Z} and buffer handle {\tt BH} are created at two processes A and B; this handle is used by A to create and send a message for B, by B to receive this message. Further assume that both A and B run on distinct nodes, with possibly different 32 bit architectures. Then {\tt X} at process B is assigned the value of {\tt X} at process A (up to rounding errors that may occur during conversion); {\tt Z} on process B is assigned the character string value of {\tt Z} on process A; if both nodes use ASCII encoding, then no conversion is required for the characters. On the other hand, variable {\tt Y} on process B may be allocated a value that differs from the value of variable {\tt Y} on process A. This may occur if the two nodes use a different byte sequence (little-endian vs big-endian), or use a different binary representation for reals (IEEE vs HEX). Thus, in order to ensure correct execution in a heterogeneous environment, it is important that the types of values in a message match the types of the corresponding values in the sending and in the receiving program. \subsection{Receive Criteria} The selection of a message by a receive operation is done uniquely according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \subsection{Communication Mode} A sending operation can occur in one of two modes: \begin{description} \item[REGULAR] The send may start whether or not a matching receive has been posted. \item[READY] The send may start only if a matching receive has been posted. \end{description} A {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. In some systems, this will allow to optimize communication and avoid a hand-shaking operation that is otherwise required. \discuss{I deleted the symmetric ready receive. Will revive it if there is a requirement for it.} \subsection{Communication Objects} An opaque communication object identifies various properties of a communication operation, such as the buffer descriptor that is associated with it, its context, the tag and destination parameters to be used for a send, or the tag and source parameters to be used for a receive. In addition, this object stores information about the status of the last communication operation that was performed with this object. This object is accessed using a communication handle. One can consider communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). An object is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The communication object, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., an object needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all objects are freed at program termination, by default). The above scenario pertains to {\em persistent} objects. One can also create {\em ephemeral} objects. Such object persists only until the communication operation is completed, at which point it is destroyed. Thus, correct invocation of suboperations with an ephemeral object is {\bf INIT START COMPLETE}. A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Communication Object Creation} An object for a send operation is created by a call to {\bf MPI\_INIT\_SEND}. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating an object for a receive operation. The creation of a communication object is a local operation that need not involve communication with a remote process. {\bf \ \\ MPI\_INIT\_SEND (handle, buffer\_handle, dest, tag, context, mode, persistence)} \\ Creates a send communication object. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to send buffer descriptor \item[IN dest] rank in context of destination (integer) \item[IN tag] user tag for messages sent with this handle (integer) \item[IN context] handle to context of messages sent with this handle \item[IN mode] send mode (state type, with two values: {\tt MPI\_REGULAR} and {\tt MPI\_READY}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} {\bf \ \\ MPI\_INIT\_RECV (handle, buffer\_handle, source, tag, context, persistence)} \\ Create a receive handle. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to receive buffer descriptor. \item[IN source] rank in context of source, or DONTCARE (integer). \item[IN tag] user tag for messages received with this handle, or DONTCARE (integer). \item[IN context] handle to context of messages received with this handle. \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ I have not included proposals for partially specified message handles, that some peoples seem to desire. I have merged all handle setup into one call. } \subsubsection{Communication Start} {\bf \ \\ MPI\_START(handle)} \begin{description} \item[IN handle] communication handle \end{description} The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf \ \\ MPI\_WAIT ( handle, return\_status\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_WAIT} returns when the send operation identified by {\tt handle} is complete. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer, or any other location that can be referenced by the send operation. However, it does not indicate that the message has been received; rather it may have been buffered by the communication subsystem. The completion of a receive operation indicates that the receiver is now free to access the locations in the receive buffer, which contain the received message, or any other location that can be referenced by the receive operation. It does not indicate that the matching send operation has completed. The call returns a handle to an opaque object that contains information on the completed operation -- the {\bf return status} object. {\bf \ \\ MPI\_STATUS (handle, flag, return\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT flag] logical \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_STATUS} returns {\tt flag=true} if the operation identified by {\tt handle} is complete, In such case, the return handle points to an opaque object that contains information on the completed information. It returns {\tt flag=false}, otherwise. In such case, the value of the return handle is undefined. Implementation notes: A call to {\tt MPI\_WAIT} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. The use of a blocking receive operation ({\tt MPI\_WAIT}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_STATUS}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_STATUS} is for that operation to return as soon as possible. However, if repeatedly called for an operation that is enabled, it must eventually succeed. The return status object for a send operation carries no information. The return status object for a receive operation carries information on the source, tag and length of the received message. These fields are required because the receive operation may have specified {\tt DONTCARE} in either source or tag field, and the message may have been shorter than the receive buffer. {\bf \ \\ MPI\_RETURN\_STAT( handle, len, source, tag)} \begin{description} \item[IN handle] handle to return status object \item[OUT len] difference between length of receive buffer and length of received message, in bytes. Thus, the value returned is zero if the received message matches the the receive buffer, positive if it is shorter (integer). \item[OUT source] rank of message sender in message context (integer). \item[OUT tag] tag of received message (integer). \end{description} \discuss{ I put the difference between message buffer and message length as the value returned, rather than length of received message, so that it might be easy to test for exact match. The use of a return status object, rather than a list of parameters may simplify the use of MPI routines, if the values stored in the object are seldom checked. A predefined return status object should be provided, to ease programming.} \subsubsection{Multiple Completions} It is convenient to be able to wait for the completion of any or all the operations in a set, rather than having to wait for specific message. A call to {\tt MPI\_WAITANY} or {\tt MPI\_STATUSANY} can be used to wait for the completion of one out of several operations; a call to {\tt MPI\_WAITALL} can be used to wait for all pending operations in a list. {\bf \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_handle)} \\ Blocks until one of the operations associated with the communication handles in the array has completed. Returns the index of that handle in the array, and returns the status of that operation in the object associated with the return\_handle. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed. \end{description} The successful execution of {\tt MPI\_WAITANY(list\_of\_handles, index, return\_handle)} is equivalent to the successful execution of {\tt MPI\_WAIT(handle[i], return\_handle)}, where {\tt i} is the value returned by {\tt index} and {\tt handle[i]} is the {\tt i}-th handle in the list. If more then one operation is enabled and can terminate, one is arbitrarily chosen (subject to the restrictions on operation termination order, see Section~\ref{subsec:correct}). {\tt \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_status\_handle)} \\ is \begin{verbatim} {MPI_WAIT_RECV (handle[1], return_handle); index = 0} || ... || {MPI_WAIT_RECV (handle[n], return_handle); index = n-1} \end{verbatim} (``$||$'' indicates choice; one of the alternatives is chosen, nondeterministically.) {\bf \ \\ MPI\_STATUSANY ( list\_of\_handles, index, return\_handle)} \\ Causes either one or none of the operations associated with the communication handles to return. In the former case, it has the same return semantics as a call to {\tt MPI\_WAIT\_ANY}. In the later case, it returns a value of -1 in {\tt index} and {\tt return\_handle} is undefined. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed, or -1 if none completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed, if any; undefined when {\tt index = -1}. \end{description} {\bf MPI\_WAITALL(list\_of\_handles, list\_of\_return\_handles)} \\ Blocks until all communication operations associated with handles in the list complete, and return the status of all these operations. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT list\_of\_return\_handles] Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. \end{description} \subsection{Blocking Communication} Blocking send and receive operations combine all communication suboperations into one call. The operation returns only when the communication completes and no communication object persists after the call completed. However, the buffer descriptor object needs be created ahead of the call. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf R \end{array} \right] \left[ \begin{array}{c} \bf SEND \\ \bf RECV \end{array} \right] \] The first letter (void or {\bf R}) indicates the start mode (regular or ready). {\bf \ \\ MPI\_SEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RSEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RECV(buffer\_handle, source, tag, context, return\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} {\bf Implementation note:} While these functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call. They use ephemeral communication objects, so that the operation is completed, and the associated resources are freed, by using one of the functions {\tt MPI\_WAIT, MPI\_STATUS, MPI\_WAITANY, MPI\_STATUSANY}, or {\tt MPI\_WAITALL}. Here, too, a buffer object has to be created ahead of the communication initiation operation. We use the same naming convention as for blocking operations: a prefix of {\bf R} indicates the {\tt READY} mode. In addition, a prefix of {\bf I} is used to indicate {\em immediate} (i.e., nonblocking) execution. {\bf \ \\ MPI\_ISEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRSEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) \end{verbatim} \subsection{Block Sending Operations} The most frequent type of buffer used is a contiguous buffer of numeric storage units, i.e., a contiguous buffer of words that may contain either INTEGER, REAL or LOGICAL values (in FORTRAN). In a homogeneous environment such messages can be used to send arbitrary sequences of contiguous items, where each item occupies an integer number of words. We specialize the functions in the two previous subsections to this case, thus avoiding the need for the creation of a buffer descriptor object. We use the same naming scheme used in the previous subsections, and append a {\bf B} in the function name, for {\tt BLOCK}. {\bf \ \\ MPI\_SENDB (start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_SEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_RSENDB (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RSEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf MPI\_RECVB(buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RECV(buffer_handle, source, tag, context, return_status_handle) \end{verbatim} {\bf \ \\ MPI\_ISENDB (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_ISEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRSENDB (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRSEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRECVB(handle, buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRECV(handle, buffer_handle, source, tag, context) \end{verbatim} \discuss{ I use word count, rather than byte count. I believe word messages are much more prevalent than byte messages, and it's a blessing not to have to multiply by 4 for each message. Byte messages are still available the hard way. Also, word messages fit well with the Fortran numeric storage unit concept. If we are to add more functions, my next addition would be a vector send operation. } \subsection{Correctness} \label{subsec:correct} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. Thus, if a two messages from the same source can satisfy a pending receive, the first message sent is accepted; if a message can satisfy two pending receives, the first receive posted is satisfied. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active communication objects. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item If a process executes an {\tt INIT} operation, then the operation eventually succeeds, or a {\em resource exception} occurs. The standard does not specify when a resource exception is allowed to occur. It is expected that an operational definition will be made available, in the form of test programs that have to execute with no resource exceptions. It is highly desirable to have generous bounds on the number of concurrently active communication objects each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active communication object. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation. {\bf An enabled operation either completes successfully or becomes permanently disabled.} \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ranks 0 and 1) The following program is safe, and should always succeed. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_RECVB(recvbuf, len, 1, tag, context) CALL MPI_SENDB(sendbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed; the receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_SENDB(sendbuf, len, 0, tag, context) CALL MPI_RECVB(recvbuf, len, 0, tag, context) END IF \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer {\tt len} words of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Sun Mar 14 22:16:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12993; Sun, 14 Mar 93 22:16:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02504; Sun, 14 Mar 93 22:16:17 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sun, 14 Mar 1993 22:16:16 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02496; Sun, 14 Mar 93 22:16:15 -0500 Message-Id: <9303150316.AA02496@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 8685; Sun, 14 Mar 93 22:16:15 EST Date: Sun, 14 Mar 93 22:10:08 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: draft of point-to-point section was sent Reply-To: SNIR@watson.ibm.com Main additions: 1. A more systematic treatment of the various datatypes used by MPI. 2. A proposal for context=group. 3. The incorporation of (a variant of) the Argonne approach to buffer handles. 4. The introduction of two levels of functios, for general buffers and for contiguous data. Comments are welcome. A much as possible, I suggest that we start arguing from the current draft, rather than ab initio. From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 11:33:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25262; Mon, 15 Mar 93 11:33:21 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03012; Mon, 15 Mar 93 11:31:24 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 11:31:22 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03004; Mon, 15 Mar 93 11:31:21 -0500 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA15829; Mon, 15 Mar 93 08:31:13 PST Message-Id: <9303151631.AA15829@ocfmail.ocf.llnl.gov> Date: Mon, 15 Mar 1993 08:39:09 -0800 To: "Marc Snir" From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: Pt2Pt document Cc: MPI-PT2PT@CS.UTK.EDU Marc, I tried to process your point to point document with LaTeX, but it blew up. Since I am not a LaTeX expert and since I believe LaTeX can output postscript (or at least it outputs something that can be turned into postscript), could you run your document through the necessary processing and send out a postscript document? I believe more of us have postscript printers than have LaTeX processors or LaTeX expertise. Thanks, Regards, Dan From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:06:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02388; Mon, 15 Mar 93 17:06:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14644; Mon, 15 Mar 93 17:05:50 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:05:49 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14636; Mon, 15 Mar 93 17:05:48 -0500 Message-Id: <9303152205.AA14636@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6357; Mon, 15 Mar 93 17:05:47 EST Date: Mon, 15 Mar 93 17:04:54 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: pt2pt draft Reply-To: SNIR@watson.ibm.com Just sent a postcript file, for the benefit of those that don't have access to latex. From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:06:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02386; Mon, 15 Mar 93 17:06:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14541; Mon, 15 Mar 93 17:04:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:04:49 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14533; Mon, 15 Mar 93 17:04:42 -0500 Message-Id: <9303152204.AA14533@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 6333; Mon, 15 Mar 93 17:04:35 EST Date: Mon, 15 Mar 93 17:04:31 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT_1.DVI.* %%Pages: 35 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 2 36 df34 DI E /Fb 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EAE738EA0700A30D0E7E 8E12>3 D E /Fc 1 16 df15 D E /Fd 4 111 df<127012F8A3127005057C840E>58 D<15181578EC01E0EC0780EC1E001478EB03E0EB0F80013CC7FC13F0EA03C0000FC8FC123C12F0 A2123C120FEA03C0EA00F0133CEB0F80EB03E0EB0078141EEC0780EC01E0EC007815181D1C7C99 26>60 D<12C012F0123C120FEA03C0EA00F0133EEB0F80EB01E0EB0078141EEC0780EC01E0EC00 78A2EC01E0EC0780EC1E001478EB01E0EB0F80013EC7FC13F0EA03C0000FC8FC123C12F012C01D 1C7C9926>62 D<381E03E0383F0FF038639838EBB01CEAC3E013C0138000075B1300A3000E5BA2 ECE08015C048EBE180EB01C1ECC300EB00E74813FE001813781A157F941D>110 D E /Fe 5 107 df0 D15 D<150C153C15F0EC03C0EC0F00143C14F0EB07 C0011FC7FC1378EA01E0EA0780001EC8FC127812E01278121EEA0780EA01E0EA0078131FEB07C0 EB00F0143C140FEC03C0EC00F0153C150C1500A8B612FCA21E277C9F27>20 D<12C012F0123C120FEA03C0EA00F0133CEB0F80EB03E0EB0078141EEC0780EC01E0EC0078151C 1578EC01E0EC0780EC1E001478EB03E0EB0F80013CC7FC13F0EA03C0000FC8FC123C12F012C0C9 FCA8B612FCA21E277C9F27>I<12C0B3B3AD02317AA40E>106 D E /Ff 26 121 df<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C 01C0A2387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA8 7E007013E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>III73 D76 D<38FC01F8EAFE03A2383B06E0A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03 F8A3151C7F9B18>I<387E07F038FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A2 13311339A31319A2131D130DA3EA7F07EAFF87EA7F03151C7F9B18>III82 D<387FFFF8B5FCA238E07038A400001300B2EA07FFA3151C7F9B18>84 D<38FF83FEA3381C0070 B2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA007C171C809B18>I<387F8FE0139F 138F380E0700120FEA070E138EEA039C13DCEA01F8A26C5AA2137013F07F120113DCEA039E138E EA070F7F000E13801303001E13C0387F07F038FF8FF8387F07F0151C7F9B18>88 D97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3 E0EB80F0EB00701478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E138038067E00151C 809B18>II101 D<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C13 1EEA0E0FEB0780130314C038FFC7F8A3151C7F9B18>107 DI110 DI<38 7F87E038FF9FF8EA7FBF3803FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A15147F9318>114 D<487E1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01FF6C1380EB7E001319 7F9818>116 D<387F8FF0139F138F38070700138EEA039EEA01DC13F81200137013F07FEA01DC EA039E138EEA0707000F1380387F8FF000FF13F8007F13F015147F9318>120 D E /Fg 72 126 df<127012F8B212701200A5127012F8A31270051E769D1A>33 D<1338137813F8EA01E0EA03C0EA0780EA0F00121E121C123C123812781270A312F05AA87E1270 A312781238123C121C121E7EEA0780EA03C0EA01E0EA00F8137813380D2878A21A>40 D<126012F012787E7E7EEA0780EA03C0120113E0120013F01370A313781338A813781370A313F0 13E0120113C01203EA0780EA0F00121E5A5A5A12600D287CA21A>I<13E0A40040134038F0E1E0 EAF8E3387EEFC0381FFF00EA07FC6C5A487EEA1FFF387EEFC038F8E3E0EAF0E13840E040000013 00A413157D991A>I<123C127E127FA3123F1207120F120E123E12FC12F812E0080D77851A>44 D<387FFFC0B512E0A26C13C013047D901A>I<127812FCA41278060676851A>I48 D<13C012011203A21207120F127F12FD 12791201B2EA7FFFA3101E7B9D1A>IIII<383FFFC05AA20070C7FCA8EA73F8EA7FFE7F387E0F80387803 C0EA3001000013E01300A3126012F01301EB03C038700780EA7C1F383FFF00EA1FFCEA07F0131E 7D9D1A>I<137F3801FF804813C03807C1E0EA0F01121E383C00C0003813001278127012F0EAE3 F8EAEFFEB5FC38FE0F8038F803C0EAF00114E01300A31270A2EA7801003813C0EA3C03381F0F80 380FFF006C5AEA01F8131E7D9D1A>I<12E0B512F8A338E001E014C0EA0003EB0780EB0F00130E 131E5B133813781370A213F05BA212015BA312035BA7151F7E9E1A>III<123C127EA4123C1200A9123C127C127EA3123E120E12 1E121C123C12F812F012E0071C77941A>59 D<387FFFF0B512F8A26C13F0C8FCA4387FFFF0B512 F8A26C13F0150C7E941A>61 D<1338137CA2136C13EEA313C6A2EA01C7A438038380A4380701C0 A213FFA24813E0EA0E00A4481370387F01FC38FF83FE387F01FC171E7F9D1A>65 DIIII<387FFFFCB5FC7E380E001CA41400A3EB0380A3EA0FFFA3EA0E03A390C7 FCA8EA7FE012FF127F161E7F9D1A>I<3801F8E0EA03FEEA07FFEA0F0FEA1E03EA3C011238EA78 001270A200F013005AA5EB0FF8A338F000E01270130112781238EA3C03121EEA0F0FEA07FFEA03 FEEA01F8151E7E9D1A>I<38FF83FEA3381C0070AA381FFFF0A3381C0070AB38FF83FEA3171E7F 9D1A>II<387F03F838FF87FC387F03F8381C01 E0EB03C014801307EB0F00131E131C133C5B5B7FEA1DFC121F139E130E130FEA1E07001C138013 0314C0EB01E0A2EB00F01470007F13FC38FF81FE387F00FC171E7F9D1A>75 DI<007E133FB4EB7F806C 1400381D80DCA313C1A2001C139CA213E3A2EB631C1377A21336A2133E131CA21300A7007F137F 39FF80FF80397F007F00191E809D1A>I<38FE03FE12FFA2381D8070A213C0121CA213E0A21360 1370A213301338A21318131CA2130C130EA21306A213071303A238FF81F0A21380171E7F9D1A> IIIII<3807F1C0EA1FFDEA3FFFEA7C1FEA7007EAF003EAE001A390C7FC7E1278123F EA1FF8EA0FFEEA01FF38000F80EB03C0130114E01300126012E0A2EAF001EB03C038FE0780B5FC EBFE00EAE3FC131E7D9D1A>I<387FFFFEB5FCA238E0380EA400001300B3A23803FF80A3171E7F 9D1A>I<38FF83FEA3381C0070B3A2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA00 7C171E7F9D1A>I<38FF01FEA3381C0070A3001E13F0000E13E0A3380701C0A438038380A43801 C700A4EA00C613EEA3136C137CA21338171E7F9D1A>I<00FE13FEEAFF01EAFE000070131C0078 133C00381338A7137C001C137013EEA513C6A2380DC760A31383A3000F13E0A2380701C0171E7F 9D1A>I<387F87F8A3380F01C03807038013833803870013C7EA01CE13EEEA00FCA21378A3137C A213FE13EEEA01CF13C7380387801383380703C01301380E00E0A2387F01FC38FF83FE387F01FC 171E7F9D1A>I<38FF01FEA3381C00706C13E0A2380701C0A213830003138013C700011300A2EA 00EEA2137CA21338AA48B4FCA3171E7F9D1A>I<383FFFF85AA23870007014F0EB01E014C0EA00 03EB0780EB0F00130E131E5B133813785B5B1201485A5B120748C7FC001E1338121C123C5A1270 B512F8A3151E7E9D1A>II93 D<387FFFC0B512E0A26C13C013047D7E1A>95 D97 D<12FEA3120EA6133FEBFFC0000F13E0EBE1 F0EB8070EB00781438000E133C141CA5000F133C14381478EB80F0EBC3E0EBFFC0000E13803806 7E00161E7F9D1A>I<3801FF80000713C04813E0EA1F01383C00C0481300127012F05AA57E1270 007813707E381F01F0380FFFE06C13C00001130014157D941A>II II<3801F8FC3807FFFE5A381F0F8C381C0380003C13C0EA3801A3EA3C03001C 1380EA1F0FEBFF00485AEA39F80038C7FC123C121C381FFF8014F04813F8387C00FC0070131C00 F0131E48130EA36C131E0078133C383F01F8381FFFF06C13E00001130017217F941A>I<12FEA3 120EA6133FEBFF80000F13C0EBE1E013801300A2120EAB38FFE3FE13E713E3171E7F9D1A>II<12FEA3120E A6EB0FFCEB1FFEEB0FFCEB03C0EB0780EB0F00131E5B5B13FC120F13DE138F380E0780EB03C0A2 EB01E0EB00F038FFE3FE14FF14FE181E7F9D1A>107 DI<387DF1F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAC387F1F1F39FF9F9F80397F1F1F 00191580941A>IIII<387F87F038FF9FFCEA7FBF3803FC3CEBF018EBE000A2 5BA25BA9EA7FFFB5FC7E16157E941A>114 D<380FFB80EA3FFF5AEAF80FEAE003A300F8C7FCEA 7FC0EA3FFCEA0FFF38007F80EB07C0EA600112E012F0130338FC0F80B512005BEAE7F812157C94 1A>I<13C01201A6387FFFE0B5FCA23801C000AA1470A314F0EBE1E0EA00FFEB7FC0EB3F00141C 7F9B1A>I<38FE0FE0A3EA0E00AC1301A2EA0F073807FFFE7EEA01FC17157F941A>I<387F83FC38 FFC7FE387F83FC380E00E0A3380701C0A338038380A33801C700A3EA00EEA3137CA2133817157F 941A>I<387FC7F8EBCFFCEBC7F8380703C038038380EBC700EA01EFEA00FE137C13781338137C 13EE120113C738038380000713C0EA0F01387FC7FC00FF13FE007F13FC17157F941A>120 D<387FC3FC38FFC7FE387FC3FC380E00E0A27EEB01C013811203EB838013C31201EBC700EA00E7 A213E61366136E133CA31338A35BA21230EA78E01271EA7FC06C5A001EC7FC17207F941A>I123 D<126012F0B3B31260042775A21A>I<127CB4FC7FEA07C0EA01E012 00AB7F137CEB3FE0131F133FEB7C0013F05BAB1201EA07C0B45A90C7FC127C13277DA21A>I E /Fh 22 121 df73 D77 D<90B512E015F890380F003C151E011E130FA21507A249130FA3150E49131E153C153815F09038 F003E090B51280ECFE0001F0C7FC485AA4485AA4485AA4EAFFF8A220227DA121>80 D97 D<137E48B4FC3803C380EA0703EA0E 07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03 383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2 19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA 701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>II<13F0EA0F E0A21200A2485AA4485AA448C7FC131FEB7F80EBE1C0380F80E0A21300120E381E01C0121CA338 380380A21484EB07060070130C130E1418EB063038E007E0386003C017237DA219>I<136013F0 13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA 1F80EA0E000C217CA00F>I108 D<391E07C07C393F1FE1FE3963B8738790 39E03E038012C3EBC03CEB803800079038780700EB0070A3000EEBE00EA21610ED1C18261C01C0 133015381660ED18C03A3803801F80D81801EB0F0025157C9428>I<381E0780383F1FE0EA63B8 EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140CEB031800 3813F0381801E018157C941B>I<137E48B4FC3803C380380701C0120E001C13E0123CA21278A3 38F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803C1F03807E3F8 380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EBE3C038077F80 EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863B0E013E0EAC3 C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114 D<13FC48B4FC38038380 EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA7007EAF00EA2485AEA7038 EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2EA0700120EA45AA45AA2 13101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F1330381F8070EA31C00061 13E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2130FEB1F183807F3F038 03E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC380A2000313C0EA0700A3 380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I<3803C1C0380FE7E0381C 7E3038303C70EB38F01260146038007000A45BA214200070133038F1C060A200E113C038E3E180 387E7F00EA3C3E14157D9416>120 D E /Fi 65 123 df<903807F83F017FB512C03A01FC0FE3 E03903F01FC7EA07E0D80FC01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A22323 7FA221>11 D13 D<13181330136013C01201EA0380120713005A121EA2123E123CA2127CA31278 12F8AD1278127CA3123CA2123E121EA27E7E13801203EA01C012001360133013180D317BA416> 40 D<12C012607E7E121C7E120F7E1380EA03C0A213E01201A213F0A3120013F8AD13F01201A3 13E0A2120313C0A2EA078013005A120E5A12185A5A5A0D317DA416>I<1238127C12FE12FFA212 7F123B1203A212071206A2120C121C12181270122008117C8610>44 D I<1238127C12FEA3127C123807077C8610>I<13181378EA01F812FFA21201B3A7387FFFE0A213 207C9F1C>49 DII<14E013011303A213 07130F131FA21337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA238 0007E0A7EBFFFEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC00018 C7FCA4EA19FCEA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F838 6007E0003013C0381C1F80380FFF00EA03F815207D9F1C>II57 D<1238127C12FEA3127C12381200A81238127C12FEA3127C12380716 7C9510>I63 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB300FA201607F 140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A225227EA12A >65 DII IIIIII75 DIIIII82 D<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400B4FC13E06C B4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA36C131EA26C 133C12FCB413F838C7FFE00080138018227DA11F>I<007FB61280A2397E03F80F007814070070 14030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127>IIII<3A7FFFC1FFF0A23A03FC00 0C006C6C5B000014386D5B90387F8060013F5B14C190381FE380010F90C7FC14F7EB07FE6D5AA2 6D7E13008081497F14BF9038031FE0496C7E130E90380C07F8496C7E133890383001FE496C7E13 E04848EB7F8049EB3FC03AFFFC03FFFEA227227FA12A>II<003FB512E0A29038801F C0383E003F003C14800038EB7F00485B5C1301386003FC5C130700005B495A131F5C133F495A91 C7FC5B491360485A12035B000714E0485A5B001FEB01C013C0383F8003007F1307EB003FB6FCA2 1B227DA122>I97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3 127CA2127E003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F 0007B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC 0018C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B38 1F80FC6CB45A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2 EAFFF0A20C247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC13 3E137EEA78FCEA7FF0EA1FC0102E83A311>III<3AFF87F00FE090399FFC3FF83A1F B87E70FC9039E03EC07C9039C03F807EA201801300AE3BFFF1FFE3FFC0A22A167E952F>I<38FF 87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0A21A167E951F>I<13FE3807FFC038 0F83E0381E00F0003E13F848137CA300FC137EA7007C137CA26C13F8381F01F0380F83E03807FF C03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2 EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A21A207E951F>I114 DI<13C0A41201A212031207120F121FB5FCA2EA0FC0ABEBC180A51207EB E300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF14FEA2EA0F833907FF7FC0EA01FC1A 167E951F>I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C 6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E>I<3AFFE3FF87F8A23A1F807C00C0 D80FC0EB0180147E13E0000790387F030014DF01F05B00031486EBF18FD801F913CC13FB9038FF 07DC6C14F8EBFE03017E5BA2EB7C01013C5BEB380001185B25167F9528>I<39FFF07FC0A2390F C01C006C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D7E497E80EB67F013E33801C1F838 0381FC48C67E000E137E39FF81FFE0A21B167F951E>I<39FFF01FE0A2390FC00600A2EBE00E00 07130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA2130CA2 5B1278EAFC3813305BEA69C0EA7F80001FC8FC1B207F951E>I<387FFFF0A2387C07E038700FC0 EA601F00E0138038C03F005B13FEC65A1201485AEBF0301207EA0FE0EBC070EA1F80003F1360EB 00E0EA7E03B5FCA214167E9519>I E /Fj 64 123 df<90380FC3E090387FEFF09038E07C7838 01C0F8D8038013303907007000A7B61280A23907007000B0387FE3FFA21D20809F1B>11 DII<90 380F80F890387FE7FE9038E06E063901C0FC0F380380F8380700F00270C7FCA6B7FCA239070070 07B03A7FE3FE3FF0A22420809F26>I34 D<127012F812FCA2127C120CA31218A2123812301260 1240060E7C9F0D>39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A2 12E0AC1260A21270A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I< 12C012607E7E121C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A2 13005A12065A121C12185A5A5A0B2E7DA112>I<127012F812FCA2127C120CA31218A212381230 12601240060E7C840D>44 DI<127012F8A3127005057C840D>I<1303A2 13071306A2130E130CA2131C1318A213381330A213701360A213E013C0A212011380A312031300 A25A1206A2120E120CA2121C1218A212381230A212701260A212E05AA2102D7DA117>II<130EA2131E133EA2136E13EE13CEEA018E1203130E1206 120E120C121812381230126012E0B512F0A238000E00A7EBFFE0A2141E7F9D17>52 D<1260387FFFC0A21480EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207A7 6CC7FC121F7D9D17>55 D57 D<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A312701200 AA127012F8A312781218A41230A21260A21240051D7C930D>I63 D65 DI<90381FC04090387FF0C03801F8393803C00D380780 07380F0003121E003E1301123C127C1400127812F81500A8007814C0127CA2123C003EEB018012 1E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D9F21>IIII<90380FC02090387FF8603901F81CE03803E00638078003380F0001121E14005A127C156012 7812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E38078003EA03E03901F80E6039007FFC 2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90B5FCA2EB800FAD39FFF8FFF8A2 1D1F7E9E22>II76 DI<39FF807FF813C00007EB07809038E00300A2EA06F0A21378133CA2 131EA2130FA2EB078314C31303EB01E3A2EB00F3A2147BA2143F80A280A2000F7FEAFFF0801D1F 7E9E22>III<3807E080EA0FF9EA1C1FEA300FEA70 07EA600312E01301A36CC7FCA21278127FEA3FF0EA1FFC6C7EEA03FF38001F801307EB03C0A213 0112C0A400E01380EAF00338F80700EAFE0EEACFFCEA81F812217D9F19>83 D<007FB512E0A238780F010070130000601460A200E0147000C01430A400001400B23807FFFEA2 1C1F7E9E21>I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800FC006A3913819E00ED8 03C0140CA214393A01E030F018A33A00F0607830A3ECE07C903978C03C60A390393D801EC0A390 383F000F6D5CA3010E6DC7FCA32C207F9E2F>87 D<397FF83FF8A23907C00F800003EB06003801 E00EEBF00C00005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C13078080130DEB1DF0EB18F8EB 3878EB307CEB603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FFE01FFEA21F1F7F9E 22>I92 D97 D<120E12FEA2120EA9133FEBFF80380FC3C0EB00 E0000E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E0015207F9F19>III I<133C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0A2120EAD38FFE7FC A216207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFFC0A20A1F809E0C>I< 13E0EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EAF1C0EA7F80EA3E000C 28829E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B137013F0EA0FF81338EA0E 1C131E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA2120EB3A9EAFFE0A20B20 809F0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E13E0AD3AFFE7FE7FE0 A223147F9326>IIII<3803E1 80EA0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA1E1FEA0FF3EA03E3EA00 03A7EB3FF8A2151D7E9318>III<1206A4120EA2121E123EEAFFF8A2EA0E00AA1318 A5EA073013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC1303A2EA070FEBFDFCEA01 F116147F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA2 13FC6C5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560ECC0E0D80E0F13C0130C14 E00007EBE180EB186114713903987300EBB033A2143F3801F03EEBE01EA20000131CEBC00C1E14 7F9321>I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13781338133C137C13EEEA 01C7138738030380380701C0000F13E038FF87FEA21714809318>I<38FF87F8A2381E01E0000E 13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A31360A35B12F0EAF18012F3 007FC7FC123C151D7F9318>II E /Fk 37 121 df<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123CA25AA412F85A A97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA114>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA01E0A2 13C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<1238127C12FE12FFA2127F123B12 03A21206A2120E120C12181270122008107C860F>44 D<1238127C12FEA3127C12381200A61238 127C12FEA3127C123807147C930F>58 D63 D<14E0A2497EA3497EA2497EA2497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F 48B57EA2EB800748486C7EA20006801401000E803AFFE01FFFE0A2231F7E9E28>65 D<903807FC0290383FFF0E9038FE03DE3903F000FE4848133E4848131E485A48C7120EA2481406 127EA200FE1400A7127E1506127F7E150C6C7E6C6C13186C6C13386C6C13703900FE01C090383F FF80903807FC001F1F7D9E26>67 DII73 D77 DIII 82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC13F86CB4FC14C0 6C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813F038FE01E038E7 FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140CA200E0140E00C0 1406A400001400B10007B512C0A21F1E7E9D24>I88 D97 D99 DII I104 D<121C123F5AA37E121CC7FCA6B4FCA2121FB0EAFFE0A20B217EA00E>I107 DI<3AFE0FE03F80 90393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8FFEA227147D932C >I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D>I<48B4FC000713 C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F03807FFC0000113 0017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA6143E147E147CEB80 FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I<3AFFE7FE1FE0A2 3A1F00F007006E7ED80F801306A23907C1BC0CA214BE3903E31E18A23901F60F30A215B03900FC 07E0A290387803C0A3903830018023147F9326>119 D<38FFE1FFA2380F80706C6C5A6D5A3803 E180EA01F36CB4C7FC137E133E133F497E136FEBC7C0380183E0380381F0380701F8380E00FC39 FF81FF80A219147F931C>I E /Fl 73 124 df<90380F83E090387FE7F09038F07E783801C0F8 EA0380EC7000EA0700A8B612C0A23907007000B1397FE3FF80A21D2380A21C>11 DII<903807E03F90393FF0FF809039F03BC1C03A01C01F00E039 03803E01A23A07001C00C01600A7B712E0A23907001C011500B03A7FF1FFCFFEA2272380A229> I34 D<127012F812FCA2127C120CA41218A21230A212601240060F7CA20E>39 D<1330136013C0EA0180EA03005A1206120E120C121C12181238A212301270A3126012E0AE1260 1270A312301238A21218121C120C120E120612077EEA0180EA00C0136013300C327DA413>I<12 C012607E7E7E120E120612077E1380120113C0A2120013E0A313601370AE136013E0A313C01201 A21380120313005A1206120E120C5A5A5A5A0C327DA413>I<127012F812FCA2127C120CA41218 A21230A212601240060F7C840E>44 DI<127012F8A3127005057C840E> IIIIII<130EA2131EA2133EA2 136E13EE13CE1201138EEA030E12071206120E120C1218A212301270126012E0B512F8A238000E 00A73801FFF0A215217FA018>I<00101380EA1C07381FFF005B5B13F00018C7FCA613F8EA1BFE EA1F0F381C0780EA180314C0EA000114E0A4126012F0A214C0EAC0031260148038300700EA1C1E EA0FFCEA03F013227EA018>I<137E48B4FC3803C180380701C0EA0E03121CEB018048C7FCA212 7812701320EAF1FCEAF3FEEAF60738FC038000F813C0130112F014E0A51270A3003813C0130300 181380381C0700EA0E0EEA07FCEA01F013227EA018>I<12601270387FFFE0A214C0EA600038E0 018038C00300A21306C65AA25BA25BA25BA213E0A3485AA51203A86C5A13237DA118>III<127012F8A312701200AB127012F8A3127005157C940E>I< 127012F8A312701200AB127012F8A312781218A41230A3126012E01240051F7C940E>I61 D<497E497EA3497EA3497E130CA2EB1CF8EB1878A2EB383C 1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F0001F130139 FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038F81C703801E0063903C003F0380780 0148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C1460123E12 1E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>IIII<903807F00890383FFC189038FC0E3838 01E0033903C001F83807800048C71278121E15385AA2007C14181278A212F81500A6EC1FFF1278 007CEB0078A2123CA27EA27E6C7E6C6C13F83801F0013900FC079890383FFE08903807F8002024 7DA226>I<39FFFC3FFFA239078001E0AD90B5FCA2EB8001AF39FFFC3FFFA220227EA125>II76 DI<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7F A2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000 EAFFF0156020227EA125>II< B512F014FC3807803FEC0F801407EC03C0A215E0A515C0A2EC0780140FEC3F00EBFFFC14F00180 C7FCADEAFFFCA21B227EA121>I82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA 3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF8 0138EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200 E0141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<39FFFC0FFFA239078001F8EC00 60B3A5000314406D13C0A20001EB01803900E00300EB7007EB3C1EEB1FF8EB07E020237EA125> I<3BFFF03FFC07FEA23B0F0007C001F00203EB00E01760D807806D13C0A33B03C007F001801406 A216032701E00C781300A33A00F0183C06A3903978383E0CEC301EA2161C90393C600F18A39039 1EC007B0A3010F14E0EC8003A36D486C5AA32F237FA132>87 D<397FF807FFA23907E003F00003 EB01C000015C3800F00301F890C7FCEB7806EB7C0EEB3E0CEB1E18EB1F38EB0FB0EB07E0A21303 1301497E80EB0778EB0E7CEB0C3EEB1C1EEB181F496C7EEB700701607F496C7E0001130101807F 00031300D80FC07F3AFFE007FFC0A222227FA125>I92 D 97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C141C141EA7141C 143C000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II<14 E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA38001278127012F0A712701278EA3801EA 3C03381E0EF0380FFCFEEA03F017237EA21B>II< 133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700B1EA7FF8A2102380A20F>I<14F03801F1 F83807FFB8380F1F38381E0F00EA1C07003C1380A5001C1300EA1E0FEA0F1EEA1FFCEA19F00018 C7FCA2121CEA1FFF6C13C04813E0383801F038700070481338A400701370007813F0381E03C038 0FFF803801FC0015217F9518>I<120E12FEA2121E120EAAEB1F80EB7FC0380FC1E0EB80F0EB00 70120EAE38FFE7FFA218237FA21B>I<121C121E123E121E121CC7FCA8120E12FEA2121E120EAF EAFFC0A20A227FA10E>II<120E12FEA2121E120EAAEB0FFCA2EB07E0EB0380EB0700 130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03C0130114E014F038FFE3FEA217237FA21A >I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E>I<390E1FC07F3AFE7FE1FF809039C0F303 C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>I<380E1F8038FE7FC0 38FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II<380E1F8038FE7FE038FFC1F0380F0078120E143CA2141EA714 3CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FCA8EAFFE0A2171F7F941B>I<3801F060EA 07FCEA1F06381C03E0EA3C01EA7800A25AA712781301123C1303EA1F0EEA0FFCEA03F0C7FCA8EB 0FFEA2171F7E941A>I II<1206A5120EA3121E123EEAFFF8A2EA0E00AA13 0CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E1370AC14F0 1301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A26C13C0A338 038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2391E01C07C D81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D81980141DA2 EBF00F00001400A2497EEB600620157F9423>I<387FC1FFA2380780F8000313E03801C1C01480 3800E3001377133E133C131C133E13771367EBC3803801C1C0380380E0380700F0EA0F8038FFC1 FFA2181580941A>I<38FFC3FEA2381E00F8000E1360A26C13C0A338038180A213C300011300A2 EA00E6A3137CA31338A21330A213701360A2EAF0C012F1EAF380007FC7FC123E171F7F941A>I< 383FFFC0A2383C038038380700EA300EEA701EEA603C13385BEA00F0485A3803C0C01380EA0700 5AEA1E01001C1380EA3803EA7007B5FCA212157F9416>II E /Fm 44 122 df<903901FF81FF011F01EF13C0903A7F80FF87E0D9FE01EB0FF03903FC03FE13 F8D807F013FCA2EE07E0020190C7FCA6B712F8A32707F001FCC7FCB3A33A7FFF1FFFE0A32C2A7F A928>11 D<121C127FEAFF80A5EA7F00121C09097B8813>46 D48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FE A317277BA622>III<14075C5C5C5C5CA25B5B497E13 0F130E131C1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F 13F8A31D277EA622>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC038 0FFFF8EB80FC380E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300007E137F 007814806CEBFF00381F01FE380FFFF8000313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E04814C015 80A215003870001E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213FEA41201 A86C5A13781C297CA822>III66 D<91393FF00180903903FFFE07010FEBFF8F 90393FF007FF9038FF80014848C7127FD807FC143F49141F4848140F485A003F15075B007F1503 A3484891C7FCAB6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F8 90393FF007F0010FB512C0010391C7FC9038003FF829297CA832>II73 D77 DIII82 D<90387F80603903FFF0E0000F13FF381F807F383F001F003E1307007E1303127C00FC1301A214 007E7E6D130013F8EBFF806C13F814FE6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F 1407A200E013031401A37E15F06C13036C14E0B413079038E01FC090B5120000E05B38C01FF01D 297CA826>I<007FB712C0A39039807FC03FD87C00140700781503A20070150100F016E0A24815 00A5C71500B3A490B612E0A32B287EA730>I<48B47E000F13F0381F81FC486C7E147FA2EC3F80 A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F 83DFFC380FFF0F3801FC031E1B7E9A21>97 DIIII<90 38FF81F00003EBE7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80 FE001F5B380FC1F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14 F0121F397E0007F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038 007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3 A3EAFFFEA30F2B7DAA14>105 DIII<3BFFC07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F90 3ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FF C07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B 7D9A26>II<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A2 5DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26> I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C 13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038F7FFC000C11300161B 7E9A1B>I<1370A413F0A312011203A21207381FFFF0B5FCA23807F000AD1438A73803F8700001 13F03800FFE0EB1F8015267FA51B>I<39FFE03FF8A3000F1303B11407A2140F0007131F3A03F0 3BFF803801FFF338003FC3211B7D9A26>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEB FC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D 5A211B7F9A24>I<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3FF8 6D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01FFF 80A3211B7F9A24>120 D<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C 6C485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA38 07007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>I E /Fn 11 118 df49 D<913A03FF800380023FEBF00749B5EAFC0F0107ECFF1F011F9038803FBF903A3F F80007FFD9FFE07F48497F48497F4890C8127F4848153F49151F121F49150F123F5B007F1607A3 4992C7FC12FFAB127F7FEF0780A2123F7F001F160F6D1600120F6D5D6C6C153E6C6D5C6C6D14FC 6C6D495AD93FF8495A903A1FFF801FC0010790B55A01014AC7FCD9003F13F80203138031337BB1 3C>67 D80 D97 D99 D105 D<2703F007F8EB0FF000FFD93FFFEB7FFE4A6DB5FC903CF1F03FC3E07F8090 3CF3C01FE7803FC0260FF780EBEF0000079026000FFEEB1FE001FE5C495CA2495CB2B500C1B500 83B5FCA440207D9F45>109 D<3903F007F800FFEB3FFF4A7F9039F1F03FC09039F3C01FE0380F F7800007496C7E13FE5BA25BB2B500C1B51280A429207D9F2E>II<1378A513F8A41201A212031207120F381FFFFEB5FCA33807F800AF140FA7141F 3803FC1EEBFE3E3801FFFC38007FF0EB1FC0182E7EAD20>116 DI E /Fo 27 120 df<1238127C12FEA212FF127F123B1203A41206A2120CA21218 12381270122008137B8611>44 D<1318133813F8120712FF12F81200B3AD487E387FFFF0A21428 7CA71E>49 D<137F3801FFC0380781F0380E00F80018137C121E003F137EEB803EA3381F007E00 0E137CC7FCA25C5C495AEB07C001FFC7FCA2EB01E06D7E147C80A280A21580123C127EB4FCA315 00485B007C133E00305B001C5B380F01F06CB45AC690C7FC19297EA71E>51 D<00181318001F13F8EBFFF014E014C01400EA1BF80018C7FCA8137E3819FF80381F83C0381E01 E0381C00F0001813F8C71278147CA2147EA31238127C12FEA25A147C12F0006013F81270383001 F0381803E0380E07C03807FF00EA01FC17297DA71E>53 D<137F3801FFC03807C1E0380F007000 1E7F001C133C003C131C48131EA200F87FA41580A41278141F127C003C133F121C001E136F6C13 CF3807FF8F0001130FD8001013001300A2141EA2121E003F5BA25C1470003E5B381801C0380E07 80D807FEC7FCEA01F819297EA71E>57 D69 D<02FF13100107EBE03090391FC0707090387E001C01F8EB0E F048481303485A4848130148481300A248C812705A123E1630127E127CA200FC1500A84AB5FC12 7C007E90380007F01503123EA2123F7E6C7EA26C7E6C7E6C6C13076C7E017E131C90391FC07870 903907FFE0100100EB8000282B7DA92F>71 D76 DI83 D87 D97 D99 D<140F49B4FCA2EB001F 80AC137E3801FFCF3807C0EF380F003F001E7F487FA2127C127812F8A81278127C123C5C6C5B6C 5B3907C1EF803903FF8FF83800FE0F1D2A7EA921>I<140F3901FC3F803907FF73C0380F07E338 1E03C3003EEBE180393C01E000007C7FA6003C5BEA3E03001E5B381F078001FFC7FCEA39FC0030 C8FCA21238123C383FFFC06C13F86C7F487F3838003F48130FEC0780481303A40070EB07000078 5B6C131E380F80F86CB45AC613801A287E9A1E>103 DI<120FEA1F80A213C0 1380A2EA0F00C7FCA8EA0780127FA2120F1207B3A2EAFFF8A20D297FA811>I107 DI<3A0783F801FC3AFF8FFE07FF903A9C 0F0E07803B0FB0079803C03B07E003F001E001C013E0A2018013C0B13BFFFC7FFE3FFFA2301A7F 9933>I<380783F838FF8FFCEB9C1E380FB00F3907E0078013C0A21380B139FFFCFFFCA21E1A7F 9921>I<137F3801FFC03807C1F0380F0078001E7F487FA2487FA200F81480A800781400007C5B 003C131EA26C5B6C5B3807C1F03801FFC06C6CC7FC191A7E991E>I<380783F838FF8FFEEBBC1F 390FE00780D807C013C090388003E0140115F0A2EC00F8A8EC01F0A215E0EBC003EC07C09038E0 0F809038B83F00EB8FFCEB83F00180C7FCAAEAFFFCA21D267F9921>I<380787C038FF9FE0EBB9 F0EA0FF1EA07E1EBC0E01400A25BAF7FEAFFFEA2141A7F9917>114 D<3807F840381FFFC0EA3C 07EA7003EA6001EAE000A36C1300127EEA7FF0EA3FFC6CB4FC0007138038003FC0130738C001E0 13007EA36C13C0EAF80138FE078038C7FF00EA83F8131A7E9918>I<390780078000FF13FFA200 0F130F00071307AF140FA2141F0003133F9038E077C03900FFC7FCEB7F071E1A7F9921>117 D<3AFFF1FFC3FFA23A0F803E00FCD9001C13703807801E023E1360A22603C03F13C01467A2D801 E0EB818014C3A201F013C32600F1811300A201F913E790387B00E6A2017F13FE013E137CA3011E 1378011C1338A2281A7F992B>119 D E /Fp 10 118 df 67 D80 D97 D99 D105 D<2701F803F8EB03F800FFD91FFFEB1FFF913B7C0FC07C0FC0913BE0 07E0E007E03C07F9C003E1C0032601FB80D9F3807FD9FF00EBF70049D901FE6D7EA2495CA3495C B3A4486C496C497EB500F0B500F0B512F0A344267EA549>109 D<3901F807F800FFEB1FFE9138 781F809138E00FC03A07F9C007E03801FB80EBFF00496D7EA25BA35BB3A4486C497EB500F1B512 E0A32B267EA530>II<1318A51338A41378A213F8A2120112031207001FB5FCB6FCA2D801F8C7FC B2EC0180AA3800FC031500137CEB7E07EB3F0EEB0FFCEB03F019367EB421>116 DI E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 392 509 a Fp(P)n(oin)n(t)29 b(to)g(P)n(oin)n(t)g(Comm)n(unication)843 656 y Fo(Marc)20 b(Snir)554 731 y(William)d(Gropp)k(and)f(Ewing)f(Lusk)772 848 y(Marc)n(h)i(15,)e(1993)164 1068 y Fn(1)83 b(P)n(oin)n(t)27 b(to)h(P)n(oin)n(t)f(Comm)n(unication)164 1192 y Fm(1.1)70 b(In)n(tro)r(duction)164 1285 y Fl(This)15 b(section)f(is)h(a)g(draft)g(of)g (the)g(curren)o(t)f(prop)q(osal)i(for)f(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o (unic)o(a-)164 1345 y(tion.)21 b(It)15 b(do)q(es)h(not)g(y)o(et)f(include)f (a)i(description)f(of)h(the)f(F)l(ortran)h(77)g(and)g(C)g(bindings.)237 1405 y(I)c(ha)o(v)o(e)g(tried)f(to)i(indicate,)e(wherev)o(er)g(appropriate,)j (gaps)f(and)g(unresolv)o(ed)e(issues,)164 1465 y(using)17 b(small)d(t)o(yp)q (e.)164 1646 y Fk(Discussion:)54 b Fj(The)19 b(follo)o(wing)h(subsections)f (of)f(the)h(in)o(tro)q(duction)h(con)o(tain)f(general)g(in-)164 1706 y(formation)d(on)h(the)g(design)h(of)f(MPI)g(pro)q(cedures.)26 b(The)17 b(material)h(should)g(b)q(e)f(mo)o(v)o(ed)g(to)f(a)164 1766 y(general)g(in)o(tro)q(duction)g(for)e(the)i(en)o(tire)f(do)q(cumen)o (t.)164 2031 y Fm(1.2)70 b(Data)23 b(T)n(yp)r(es)164 2123 y Fi(1.2.1)55 b(Handle)164 2216 y Fl(MPI)19 b(pro)q(cedures)h(use)g(at)g(v)m (arious)h(places)e Fh(hand)r(les)p Fl(.)34 b(Handles)19 b(are)h(used)g(to)g (access)164 2276 y(opaque)c(ob)s(jects.)21 b(Suc)o(h)16 b(ob)s(ject)f(can)i (b)q(e)f(created,)f(up)q(dated)i(and)f(destro)o(y)o(ed)g(only)f(b)o(y)164 2336 y(b)o(y)h(calling)f(suitable)h(MPI)g(pro)q(cedures,)g(and)h(pro)o (viding)f(the)g(handle)h(as)g(parameter.)164 2396 y(Opaque)11 b(ob)s(jects)g(hide)g(from)f(the)h(user)g(the)g(in)o(ternal)f(represen)o (tation)g(used)i(for)f(v)m(arious)164 2457 y(MPI)17 b(ob)s(jects,)h(th)o(us)f (allo)o(wing)h(to)g(ha)o(v)o(e)f(similar)f(calls)h(in)g(C)h(and)h(F)l (ortran,)f(allo)o(wing)961 2599 y(1)p eop %%Page: 2 2 bop 164 307 a Fl(to)15 b(o)o(v)o(ercome)e(problems)g(with)i(the)g(t)o(yping)g (rules)f(in)h(these)f(languages,)j(and)e(allo)o(wing)164 367 y(for)j(future)f(extension)g(of)g(their)g(functionalit)o(y)l(.)23 b(Handles)17 b(are)h(of)f(t)o(yp)q(e)g Fg(void)25 b(*)17 b Fl(in)g(C)164 428 y(and)g(of)f(t)o(yp)q(e)g Fg(integer)e Fl(in)i(F)l(ortran.) 237 488 y(An)21 b(opaque)h(ob)s(ject)f(can)h(b)q(e)f Fh(p)n(ersistent)h Fl(or)g Fh(ephemer)n(al)p Fl(.)37 b(A)21 b(p)q(ersisten)o(t)g(ob)s(ject)164 548 y(p)q(ersists)12 b(un)o(til)f(destro)o(y)o(ed)f(b)o(y)h(an)i(explicit)c (op)q(eration.)21 b(An)11 b(ephemeral)e(ob)s(ject)i(is)h(go)q(o)q(d)164 608 y(for)17 b(a)g(single)g(use;)f(th)o(us)h(an)h(ephemeral)c(ob)s(ject)i (asso)q(ciated)i(with)f(a)g(comm)o(unicati)o(on)164 668 y(op)q(eration)f (disapp)q(ears)g(once)f(this)h(op)q(eration)g(is)f(completed)e(\(or)i(once)g (this)g(ob)s(ject)g(is)164 729 y(not)i(needed)e(an)o(ymore)g(for)i(the)f (completion)e(of)i(the)g(op)q(eration\).)237 789 y(An)j(opaque)g(ob)s(ject)f (is)h(created)f(b)o(y)g(a)h(call)f(to)i Fg(MPI)p 1213 789 16 2 v 17 w(CREATE)p Fl(,)c(and)j(destro)o(y)o(ed)f(b)o(y)164 849 y(a)g(call)e(to)h Fg(MPI)p 437 849 V 18 w(FREE)p Fl(.)e(Additional)i(MPI) f(functions)h(are)h(a)o(v)m(ailable)e(to)i(create,)e(access)164 909 y(and)h(up)q(date)g(sp)q(eci\014c)e(opaque)i(ob)s(jects.)164 1029 y Fi(MPI)p 279 1029 17 2 v 20 w(CREA)-5 b(TE\(handle,)20 b(t)n(yp)r(e,)d(p)r(ersistence\))164 1121 y(OUT)i(handle)25 b Fl(handle)16 b(to)h(ob)s(ject)164 1219 y Fi(IN)h(t)n(yp)r(e)24 b Fl(state)d(v)m(alue)f(that)h(iden)o(ti\014es)e(the)i(t)o(yp)q(e)f(of)g(ob)s (ject)g(to)h(b)q(e)g(created)f(\(e.g.,)286 1279 y Fg(MPI)p 367 1279 16 2 v 17 w(COMMUNICATI)o(ON)o(,)j(MPI)p 847 1279 V 17 w(BUFFER,)g(MPI)p 1147 1279 V 18 w(CONTEXT)p Fl(,)13 b(etc.\).)164 1377 y Fi(IN)18 b(p)r(ersistence)24 b Fl(state)16 b(v)m(alue;)g(either)f Fg(MPI)p 1020 1377 V 17 w(PERSISTENT)e Fl(or)j Fg(MPI)p 1447 1377 V 18 w(EPHEMERAL)o Fl(.)164 1529 y Fi(MPI)p 279 1529 17 2 v 20 w(FREE\(handle\))164 1620 y(IN)i(handle)26 b Fl(handle)16 b(to)g(ob)s(ject)237 1712 y(An)h(ob)s(ject)f(can)h(b)q(e)g(destro)o(y)o(ed)f (only)h(if)f(there)h(is)f(no)i(p)q(ending)f(op)q(eration)h(that)f(is)164 1772 y(using)h(this)g(ob)s(ject;)f(after)h(successful)f(return)g(of)h(the)g (routine,)f(the)h(handle)f(is)h(unde-)164 1832 y(\014ned.)164 1952 y Fi(MPI)p 279 1952 V 20 w(ASSOCIA)-5 b(TED\(handle,)21 b(t)n(yp)r(e\))164 2044 y(IN)d(handle)26 b Fl(handle)16 b(to)g(ob)s(ject)164 2142 y Fi(OUT)j(t)n(yp)r(e)k Fl(state)237 2233 y(Returns)16 b(the)g(t)o(yp)q(e)f(of)h(the)g(ob)s(ject)f(the)h(handle)f(is)h(curren)o(tly) e(asso)q(ciated)j(with,)e(if)164 2293 y(suc)o(h)g(exists.)20 b(Returns)c(the)f(sp)q(ecial)g(t)o(yp)q(e)f Fg(MPI)p 1041 2293 16 2 v 18 w(NULL)g Fl(if)g(the)h(handle)h(is)f(not)h(curren)o(tly)164 2354 y(asso)q(ciated)h(with)f(an)o(y)g(ob)s(ject.)237 2414 y(MPI)h(ma)o(y)f(pro)o(vide)h(prede\014ned)h(opaque)g(ob)s(jects)f(and)h (prede\014ned,)g(static)f(han-)164 2474 y(dles)f(to)g(these)g(ob)s(jects.)21 b(Suc)o(h)16 b(ob)s(jects)g(ma)o(y)f(not)h(b)q(e)h(destro)o(y)o(ed.)961 2599 y(2)p eop %%Page: 3 3 bop 164 307 a Fi(List)13 b(of)g(handles)50 b Fl(An)11 b(MPI)g(call)f(ma)o(y)g (need)h(a)g(parameter)f(that)i(is)f(a)h Fh(list)h(of)g(hand)r(les)p Fl(.)164 367 y(In)k(C,)g(suc)o(h)g(list)g(will)f(b)q(e)h(a)h(record)f(with)g (one)h(comp)q(onen)o(t)e(b)q(eing)h(the)g(length)h(of)f(the)164 428 y(list,)j(the)g(other)g(comp)q(onen)o(ts)f(b)q(eing)h(an)h(arra)o(y)f(of) g(p)q(oin)o(ters.)33 b(In)20 b(F)l(ortran,)h(the)f(list)164 488 y(will)15 b(b)q(e)h(an)h(arra)o(y)g(of)f(in)o(tegers,)f(the)h(\014rst)h (one)f(of)h(whic)o(h)e(is)h(the)g(length)g(of)h(the)f(list.)164 657 y Fk(Discussion:)54 b Fj(The)19 b(mec)o(hanism)h(for)e(opaque)h(ob)s (jects)f(used)h(here)g(follo)o(ws)g(the)g(POSIX)164 714 y(F)l(ortran)14 b(binding)j(standard.)j(An)15 b(alternativ)o(e)h(c)o(hoice)g(is)g(to)f(ha)o (v)o(e)f(di\013eren)o(t)i(t)o(yp)q(e)f(declara-)164 770 y(tions)g(for)f(eac)o (h)h(t)o(yp)q(e)h(of)e(opaque)h(ob)s(ject.)k(Then,)d(opaque)f(ob)s(jects)f (are)h(created/destro)o(y)o(ed)164 826 y(lik)o(e)e(regular)f(v)m(ariables,)h (rather)f(than)f(b)o(y)h(MPI)g(calls;)i(they)e(are)f(still)j(accessed)e(and)g (up)q(dated)164 883 y(only)k(via)f(MPI)g(functions.)164 1193 y Fi(1.2.2)55 b(State)164 1286 y Fl(MPI)12 b(pro)q(cedures)i(use)e(at)i(v)m (arious)f(places)g(argumen)o(ts)f(with)h Fh(state)g Fl(t)o(yp)q(es.)20 b(The)13 b(v)m(alues)164 1346 y(of)j(suc)o(h)f(data)i(t)o(yp)q(e)e(are)g(all) g(iden)o(ti\014ed)g(b)o(y)g(names,)f(and)i(no)g(op)q(eration)h(is)e (de\014ned)g(on)164 1406 y(them.)k(F)l(or)14 b(example,)f(the)h Fg(MPI)p 759 1406 16 2 v 17 w(CREATE)e Fl(routine)i(has)i(a)e(state)h(t)o(yp) q(e)f(parameter)f(with)164 1466 y(v)m(alues)j Fg(MPI)p 390 1466 V 18 w(PERSISTEN)o(T)d Fl(and)k Fg(MPI)p 853 1466 V 17 w(EPHEMERAL)p Fl(.)237 1527 y(An)e Fg(enumeratio)o(n)d Fl(declared)j(in)g(an) h(included)e(MPI.h)g(\014le)h(will)f(b)q(e)i(used)f(in)g(C)h(for)164 1587 y(state)g(datat)o(yp)q(es.)22 b(The)17 b(F)l(ortran)f(77)h(mec)o(hanism) c(needs)j(to)h(b)q(e)f(decided.)164 1767 y Fk(Discussion:)67 b Fj(Named)22 b(in)o(teger)h(constan)o(ts)e(can)h(b)q(e)g(used)h(in)g(F)l (ortran)d(90,)j(using)g(the)164 1827 y Ff(PARAMETER)17 b Fj(mec)o(hanism.)28 b(The)18 b(constan)o(t)f(declarations)i(can)f(b)q(e)g(made)g(a)o(v)m(ailable) i(via)e(an)164 1888 y Ff(INCLUDE)e Fj(\014le.)25 b(F)l(ortran)16 b(77)g(do)q(es)h(not)f(seem)h(to)f(o\013er)g(an)o(y)h(con)o(v)o(enien)o(t)g (mec)o(hanism.)25 b(One)164 1948 y(p)q(ossibilit)o(y)c(is)e(to)f(sp)q(ecify)j (explicit)g(in)o(teger)e(v)m(alues,)h(and)f(allo)o(w)g(the)g(use)g(of)f (named)i(con-)164 2008 y(stan)o(ts)15 b(with)i(those)f(F)l(ortran)f(77)h (compilers)i(that)d(supp)q(ort)i(them)f(con)o(v)o(enien)o(tly)l(.)25 b(Another)164 2068 y(p)q(ossibilit)o(y)17 b(is)f(to)f(use)g(c)o(haracter)g (strings,)f(rather)h(than)g(in)o(tegers.)164 2319 y Fi(1.2.3)55 b(Named)18 b(constan)n(ts)164 2411 y Fl(MPI)j(pro)q(cedures)i(sometime)o(s)c (assign)k(a)g(sp)q(ecial)e(meaning)g(to)h(a)g(sp)q(ecial)g(v)m(alue)g(of)164 2471 y(a)f(basic)g(t)o(yp)q(e)f(parameter;)h(e.g.)34 b Fg(tag)20 b Fl(is)h(an)g(in)o(teger)e(v)m(alued)i(parameter)f(of)h(p)q(oin)o(t-)961 2599 y(3)p eop %%Page: 4 4 bop 164 307 a Fl(to-p)q(oin)o(t)23 b(comm)o(unic)o(ation)d(op)q(erations,)25 b(with)d(a)h(sp)q(ecial)f Fg(DONTCARE)d Fl(v)m(alue.)39 b(Suc)o(h)164 367 y(parameters)14 b(will)f(ha)o(v)o(e)h(a)h(range)g(of)g(regular)f(v)m (alues,)h(whic)o(h)f(is)g(a)h(prop)q(er)g(subrange)h(of)164 428 y(the)i(range)h(of)g(v)m(alues)f(of)h(the)f(corresp)q(onding)h(basic)f(t) o(yp)q(e;)h(sp)q(ecial)f(v)m(alues)g(\(suc)o(h)g(as)164 488 y(DONTCARE\))d(will)e(b)q(e)i(outside)g(the)g(regular)g(range.)21 b(The)15 b(range)g(of)g(regular)g(v)m(alues)164 548 y(can)d(b)q(e)g(queried,) f(and)h(sometimes)d(set,)j(using)g(en)o(vironmen)o(t)d(inquiry)i(or)h(en)o (vironmen)o(t)164 608 y(setting)21 b(functions)g(\(Section)f Fi(??)p Fl(\).)37 b(The)21 b(sp)q(ecial)f(v)m(alues)h(are)g(pro)o(vided)g(b)o (y)f(named)164 668 y(constan)o(t,)c(that)h(are)f(made)f(a)o(v)m(ailable)h (via)g(an)g(MPI.h)g(include)f(\014le)g(in)h(a)h(C)f(binding.)164 838 y Fk(Discussion:)237 894 y Fj(Need)e(to)e(agree)g(on)h(a)g(F)l(ortran)f (mec)o(hanism)h(for)f(named)h(constan)o(ts)f(\(see)h(the)g(discussion)164 951 y(ab)q(o)o(v)o(e\).)237 1007 y(Implemen)o(ters)19 b(should)g(try)e(to)g (detect)h(illegal)i(uses)e(of)g(\\sp)q(ecial)h(v)m(alues".)29 b(Th)o(us,)18 b(the)164 1064 y(use)e(of)e(the)i Ff(DONTCARE)e Fj(v)m(alue)i(to)f(tag)f(a)h(message)g(sen)o(t)f(should)j(b)q(e)f(\015agged)f (as)f(an)h(error.)164 1374 y Fi(1.2.4)55 b(Choice)164 1466 y Fl(MPI)13 b(functions)i(sometime)o(s)d(use)i(parameters)f(with)g(a)i Fh(choic)n(e)f Fl(\(or)h(union\))f(data)h(t)o(yp)q(e.)164 1527 y(I.e.,)10 b(distinct)g(calls)g(to)i(the)e(same)g(routine)h(ma)o(y)e(pass)j (b)o(y)f(reference)e(actual)i(parameters)164 1587 y(of)18 b(di\013eren)o(t)f (t)o(yp)q(es.)25 b(The)17 b(mec)o(hanism)e(for)i(pro)o(viding)h(suc)o(h)f (parameters)g(will)f(di\013er)164 1647 y(from)e(language)i(to)g(language.)22 b(In)15 b(C,)g(a)h(formal)e(parameter)g(of)i(t)o(yp)q(e)e Fg(void)25 b(*)15 b Fl(will)f(b)q(e)164 1707 y(used,)i(with)g(an)h(actual)f(p)q(oin)o (ter)g(parameter.)k(in)c(F)l(ortran,)g(w)o(e)g(shall)g(c)o(heat.)164 1876 y Fk(Discussion:)237 1933 y Fj(The)i(F)l(ortran)e(77)h(standard)g(sp)q (eci\014es)i(that)d(the)i(t)o(yp)q(e)f(of)g(actual)h(argumen)o(ts)e(need)i (to)164 1989 y(agree)f(with)g(the)g(t)o(yp)q(e)h(of)e(dumm)o(y)h(argumen)o (ts;)g(no)g(construct)g(equiv)m(alen)o(t)i(to)e(C)f(p)q(oin)o(ters)164 2046 y(is)e(a)o(v)m(ailable.)21 b(Th)o(us,)14 b(it)g(w)o(ould)g(seem)g(that)f (there)g(is)i(no)e(standard)g(conforming)h(mec)o(hanism)164 2102 y(to)g(supp)q(ort)h(c)o(hoice)g(parameters.)k(Ho)o(w)o(ev)o(er,)14 b(most)g(F)l(ortran)f(compiler)j(either)f(don't)g(c)o(hec)o(k)164 2159 y(t)o(yp)q(e)i(consistency)h(of)f(calls)h(to)e(external)h(routines,)h (or)e(supp)q(ort)h(a)g(sp)q(ecial)i(mec)o(hanism)e(to)164 2215 y(link)i(foreign)e(\(e.g.,)g(C\))f(routines.)27 b(I)18 b(suggest)e(that)h(w)o (e)g(accept)h(this)f(nonconformit)o(y)h(with)164 2272 y(F)l(ortran)10 b(77)h(standard.)18 b(I.e.,)12 b(w)o(e)f(accept)h(that)f(the)g(same)h (routine)g(ma)o(y)e(b)q(e)j(passed)e(an)h(actual)164 2328 y(parameter)i(of)h (a)g(di\013eren)o(t)g(t)o(yp)q(e)h(at)e(distinct)j(calls.)237 2385 y(Generic)g(routines)g(can)f(b)q(e)h(used)f(in)h(F)l(ortran)e(90)g(to)h (pro)o(vide)g(a)g(standard)g(conforming)164 2441 y(solution.)j(This)12 b(solution)g(will)g(b)q(e)g(consisten)o(t)f(with)g(our)g(nonstandard)g (conforming)g(F)l(ortran)961 2599 y Fl(4)p eop %%Page: 5 5 bop 164 307 a Fj(77)15 b(solution.)164 632 y Fm(1.3)70 b(Pro)r(cesses)164 725 y Fl(An)16 b(MPI)h(program)f(is)h(executed)e(b)o(y)h(sev)o(eral)g (autonomous)h(pro)q(cesses)g(that)g(execute)164 785 y(eac)o(h)f(their)g(o)o (wn)i(co)q(de,)e(in)h(a)g(MIMD)f(st)o(yle.)22 b(The)17 b(co)q(des)g(executed) f(b)o(y)g(eac)o(h)g(pro)q(cess)164 845 y(need)21 b(not)h(b)q(e)f(iden)o (tical.)35 b(The)21 b(pro)q(cesses)h(comm)o(unic)o(ate)d(via)i(calls)f(to)i (MPI)f(com-)164 905 y(m)o(unication)13 b(primitiv)o(es.)18 b(T)o(ypically)l(,)13 b(eac)o(h)i(pro)q(cessor)h(executes)e(in)h(its)g(o)o (wn)h(address)164 965 y(space,)i(although)g(shared-memory)e(implem)o(en)o (tati)o(ons)g(of)i(MPI)f(are)h(p)q(ossible.)26 b(This)164 1025 y(do)q(cumen)o(t)20 b(sp)q(eci\014es)h(the)f(b)q(eha)o(vior)h(of)h(a)f (parallel)g(program)g(assuming)g(that)g(only)164 1086 y(MPI)c(calls)f(are)i (used)f(for)h(comm)o(uni)o(cation.)k(The)17 b(in)o(teraction)f(of)i(an)f(MPI) g(program)164 1146 y(with)k(other)g(p)q(ossible)g(means)f(of)h(comm)o (unication)d(\(e.g.,)j(shared)h(memory\))c(is)i(not)164 1206 y(sp)q(eci\014ed.)29 b(In)18 b(particular,)h(it)g(is)f(assumed)h(that)g (message)f(bu\013ers)i(at)f(distinct)f(pro-)164 1266 y(cessors)f(are)f (disjoin)o(t.)237 1326 y(MPI)k(do)q(es)g(not)h(sp)q(ecify)e(the)h(execution)f (mo)q(del)g(for)h(eac)o(h)f(pro)q(cess.)34 b(A)19 b(pro)q(cess)164 1387 y(can)e(b)q(e)f(sequen)o(tial,)f(or)i(can)g(b)q(e)f(m)o(ultithreaded,)e (with)i(threads)h(p)q(ossibly)g(executing)164 1447 y(concurren)o(tly)l(.)23 b(Care)18 b(has)g(b)q(een)f(tak)o(en)g(to)h(mak)o(e)d(MPI)i(\\thread-safe",)i (b)o(y)d(a)o(v)o(oiding)164 1507 y(the)g(use)g(of)h(implici)o(t)d(global)i (states.)237 1567 y(The)e(initial)e(allo)q(cation)h(of)h(pro)q(cesses)g(to)g (an)g(MPI)e(computation)h(and)h(their)f(bind-)164 1627 y(ing)f(to)h(ph)o (ysical)f(pro)q(cessors)h(is)g(not)g(sp)q(eci\014ed)e(b)o(y)h(the)h(program)f (itself.)19 b(It)12 b(is)g(exp)q(ected)164 1688 y(that)17 b(v)o(endors)f (will)g(pro)o(vide)g(mec)o(hanism)o(s)e(to)j(do)g(so)g(either)f(at)h(load)g (time)d(or)j(at)g(run)164 1748 y(time.)26 b(Suc)o(h)18 b(mec)o(hanism)o(s)e (will)i(allo)o(w)g(to)h(sp)q(ecify)e(the)i(initial)e(n)o(um)o(b)q(er)f(of)j (required)164 1808 y(pro)q(cesses,)c(the)g(co)q(de)g(to)g(b)q(e)f(executed)g (b)o(y)g(eac)o(h)g(initial)g(pro)q(cess,)h(and)g(the)g(allo)q(cation)164 1868 y(of)d(pro)q(cesses)h(to)g(pro)q(cessors.)21 b(Also,)12 b(the)g(curren)o(t)f(prop)q(osal)j(do)q(es)f(not)f(pro)o(vide)f(for)i(dy-)164 1928 y(namic)j(creation)h(or)g(deletion)g(of)g(pro)q(cesses)h(during)g (program)f(execution,)f(although)164 1989 y(it)g(is)f(in)o(tended)g(to)i(b)q (e)f(consisten)o(t)g(with)g(suc)o(h)f(extension.)21 b(Finally)l(,)14 b(the)i(curren)o(t)f(pro-)164 2049 y(p)q(osal)h(do)q(es)g(not)f(sp)q(ecify)f (a)i(naming)e(sc)o(heme)e(for)k(pro)q(cesses.)21 b(W)l(e)15 b(prop)q(ose)h(to)f(alw)o(a)o(ys)164 2109 y(iden)o(tify)j(pro)q(cesses)j (according)g(to)f(their)g(relativ)o(e)e(rank)j(in)f(a)g(con)o(text)g (\(group\),)h(so)164 2169 y(that,)f(e\013ectiv)o(ely)l(,)d(pro)q(cesses)i (are)h(iden)o(ti\014ed)d(b)o(y)i(consecutiv)o(e)f(in)o(tegers.)29 b(Absolute,)164 2229 y(system-wide)13 b(unique)h(pro)q(cess)h(id's)g(are)f (\(will)g(b)q(e\))h(needed)f(only)g(if)g(dynamic)f(pro)q(cess)164 2290 y(creation)h(is)g(to)h(b)q(e)f(supp)q(orted)i(\(in)e(suc)o(h)g(ev)o(en)o (tualit)o(y)e(w)o(e)h(prop)q(ose)j(to)f(use)f(handles)h(to)164 2350 y(opaque)i Fh(pr)n(o)n(c)n(ess)f(structur)n(es)g Fl(for)g(that)h(purp)q (ose\).)961 2599 y(5)p eop %%Page: 6 6 bop 164 307 a Fm(1.4)70 b(Con)n(texts)164 520 y Fk(Discussion:)46 b Fj(This)18 b(section)f(con)o(tains)g(a)f(prop)q(osal)h(for)f(use)i(of)e (con)o(texts)g(that)g(will)i(sub-)164 580 y(sume)13 b(groups.)19 b(It)13 b(b)q(orro)o(ws)g(hea)o(vily)h(on)f(the)g(curren)o(t)g(group)g(prop)q (osal.)20 b(This)14 b(prop)q(osal)f(has)164 640 y(not)i(y)o(et)g(b)q(een)h (discussed)h(in)f(MPI)f(meetings.)237 821 y Fl(A)h Fi(con)n(text)f Fl(consists)i(of:)237 923 y Fe(\017)24 b Fl(A)d(set)g(of)h(pro)q(cesses)g (that)g(curren)o(tly)d(b)q(elong)j(to)g(the)f(con)o(text)g(\(p)q(ossibly)g (all)286 983 y(pro)q(cesses,)16 b(or)h(a)g(prop)q(er)f(subset\).)237 1084 y Fe(\017)24 b Fl(A)19 b Fi(ranking)g Fl(of)h(the)f(pro)q(cesses)g (within)g(that)g(con)o(text,)g(i.e.,)f(a)h(n)o(um)o(b)q(ering)f(of)286 1145 y(the)g(pro)q(cesses)g(in)g(that)g(con)o(text)g(from)f(0)h(to)g Fd(n)13 b Fe(\000)f Fl(1,)18 b(where)g Fd(n)g Fl(is)g(the)g(n)o(um)o(b)q(er) 286 1205 y(of)e(pro)q(cesses)h(in)f(that)h(con)o(text.)237 1307 y(A)f(pro)q(cess)h(ma)o(y)e(b)q(elong)h(to)h(sev)o(eral)e(con)o(texts)h (at)g(the)g(same)g(time.)237 1367 y(An)o(y)i(in)o(terpro)q(cess)f(comm)o (unication)f(o)q(ccurs)i(within)g(a)h(con)o(text,)e(and)i(messages)164 1427 y(sen)o(t)i(within)f(one)h(con)o(text)f(can)h(b)q(e)g(receiv)o(ed)e (only)i(within)f(the)h(same)f(con)o(text.)34 b(A)164 1487 y(con)o(text)13 b(is)h(sp)q(eci\014ed)g(using)h(a)f Fh(c)n(ontext)j(hand)r(le)f Fl(\(i.e.,)d(a)h(handle)g(to)h(an)g(opaque)f(ob)s(ject)164 1547 y(that)23 b(iden)o(ti\014es)d(a)j(con)o(text\).)38 b(Con)o(text)22 b(handles)g(cannot)h(b)q(e)f(transferred)g(for)h(one)164 1608 y(pro)q(cess)18 b(to)g(another;)g(they)f(can)h(b)q(e)f(used)h(only)f(on)h (the)f(pro)q(cess)h(where)f(they)g(where)164 1668 y(created.)237 1728 y(F)l(ollo)o(ws)f(examples)e(of)j(p)q(ossible)f(uses)h(for)f(con)o (texts.)164 1858 y Fi(Lo)r(osely)h(sync)n(hronous)i(library)g(call)h(in)n (terface)49 b Fl(Consider)16 b(the)f(case)h(where)g(a)164 1918 y(parallel)e(application)i(executes)e(a)h(\\parallel)g(call")g(to)h(a)f (library)g(routine,)g(i.e.,)e(where)164 1978 y(all)i(pro)q(cesses)i(transfer) f(con)o(trol)g(to)g(the)g(library)f(routine.)21 b(If)16 b(the)f(library)h(w)o (as)g(dev)o(el-)164 2038 y(op)q(ed)k(separately)l(,)f(then)h(one)f(should)h (b)q(ew)o(are)f(of)h(the)f(p)q(ossibilit)o(y)g(that)g(the)h(library)164 2099 y(co)q(de)12 b(ma)o(y)f(receiv)o(e)f(b)o(y)i(mistak)o(e)e(messages)h (send)i(b)o(y)e(the)h(caller)f(co)q(de,)i(and)g(vice-v)o(ersa.)164 2159 y(T)l(o)19 b(prev)o(en)o(t)f(suc)o(h)g(o)q(ccurrence)h(one)g(migh)o(t)e (use)i(a)g(barrier)f(sync)o(hronization)h(b)q(efore)164 2219 y(and)f(after)f(the)h(parallel)e(library)h(call.)24 b(Instead,)17 b(one)h(can)g(allo)q(cate)f(a)h(di\013eren)o(t)e(con-)164 2279 y(text)e(to)g(the)h(library)l(,)e(th)o(us)h(prev)o(en)o(ting)f(un)o(w)o(an)o (ted)h(in)o(terference.)k(No)o(w,)d(the)f(transfer)164 2339 y(of)j(con)o(trol)e(to)i(the)f(library)g(need)f(not)i(b)q(e)f(sync)o (hronized.)961 2599 y(6)p eop %%Page: 7 7 bop 164 307 a Fi(F)-5 b(unctional)25 b(decomp)r(osition)e(and)g(mo)r(dular)g (co)r(de)f(dev)n(elopmen)n(t)49 b Fl(Often,)164 367 y(a)18 b(parallel)f(application)g(is)h(dev)o(elop)q(ed)e(b)o(y)h(in)o(tegrating)h (sev)o(eral)e(distinct)h(functional)164 428 y(mo)q(dules,)h(that)h(is)g(eac)o (h)f(dev)o(elop)q(ed)g(separately)l(.)28 b(Eac)o(h)19 b(mo)q(dule)f(is)g(a)h (parallel)f(pro-)164 488 y(gram)13 b(that)h(runs)g(on)g(a)g(dedicated)f(set)h (of)g(pro)q(cesses,)g(and)g(the)g(computation)f(consists)164 548 y(of)j(phases)h(where)f(mo)q(dules)g(compute)e(separately)l(,)i(in)o (termixe)o(d)e(with)i(global)g(phases)164 608 y(where)i(all)g(pro)q(cesses)h (comm)o(unicate.)25 b(It)18 b(is)g(con)o(v)o(enien)o(t)f(to)i(allo)o(w)f(eac) o(h)g(mo)q(dule)g(to)164 668 y(use)g(its)g(o)o(wn)g(priv)m(ate)g(pro)q(cess)h (n)o(um)o(b)q(ering)d(sc)o(heme,)g(for)i(the)g(in)o(tramo)q(dule)f(compu-)164 729 y(tation.)27 b(This)18 b(is)f(ac)o(hiev)o(ed)g(b)o(y)g(using)h(a)h(priv)m (ate)e(mo)q(dule)g(con)o(text)g(for)h(in)o(tramo)q(dule)164 789 y(computation,)d(and)i(a)g(global)f(con)o(text)g(for)g(in)o(termo)q(dule) e(comm)o(unic)o(ation.)164 919 y Fi(Collectiv)n(e)f(comm)n(unication)51 b Fl(MPI)10 b(supp)q(orts)i(collectiv)o(e)c(comm)o(unic)o(ation)h(within)164 979 y(dynamically)h(created)i(groups)h(of)g(pro)q(cesses.)20 b(Eac)o(h)13 b(suc)o(h)f(group)h(can)g(b)q(e)f(represen)o(ted)164 1039 y(b)o(y)h(a)i(distinct)e(con)o(text.)19 b(This)c(pro)o(vides)e(a)h (simple)e(mec)o(hanism)f(to)j(ensure)g(that)g(com-)164 1099 y(m)o(unication)20 b(that)i(p)q(ertains)g(to)g(collectiv)o(e)c(comm)o (unication)h(within)i(one)h(group)g(is)164 1159 y(not)17 b(confused)f(with)g (collectiv)o(e)e(comm)o(uni)o(cation)g(within)h(another)i(group.)164 1289 y Fi(Ligh)n(t)n(w)n(eigh)n(t)i(gang)f(sc)n(heduling)50 b Fl(Consider)15 b(an)g(en)o(vironmen)o(t)d(where)i(pro)q(cesses)164 1349 y(are)j(m)o(ultith)o(treaded.)k(Con)o(texts)c(can)h(b)q(e)f(used)g(to)h (pro)o(vide)e(a)i(mec)o(hanism)c(whereb)o(y)164 1410 y(all)21 b(pro)q(cesses)i(are)f(time-shared)e(b)q(et)o(w)o(een)h(sev)o(eral)g (parallel)g(executions,)h(and)h(can)164 1470 y(con)o(text)h(switc)o(h)h(from) f(one)h(parallel)g(execution)f(to)h(another,)j(in)d(a)g(lo)q(osely)h(syn-)164 1530 y(c)o(hronous)d(manner.)39 b(A)22 b(thread)h(is)f(allo)q(cated)h(on)g (eac)o(h)f(pro)q(cess)h(to)g(eac)o(h)f(parallel)164 1590 y(execution,)14 b(and)i(a)g(di\013eren)o(t)e(con)o(text)g(is)i(used)f(to)h(iden)o(tify)d(eac) o(h)i(parallel)f(execution.)164 1650 y(Th)o(us,)f(tra\016c)g(from)e(one)i (execution)f(cannot)i(b)q(e)f(confused)f(with)h(tra\016c)g(from)e(another)164 1711 y(execution.)31 b(The)20 b(blo)q(c)o(king)g(and)h(un)o(blo)q(c)o(king)e (of)h(threads)h(due)f(to)g(comm)o(unicati)o(on)164 1771 y(ev)o(en)o(ts)14 b(pro)o(vide)g(a)i(\\lazy")f(con)o(text)f(switc)o(hing)h(mec)o(hanism)o(.)j (This)d(can)h(b)q(e)f(extended)164 1831 y(to)j(the)f(case)g(where)g(the)h (parallel)e(executions)h(are)g(spanning)i(distinct)d(pro)q(cess)i(sub-)164 1891 y(sets.)j(\(MPI)16 b(do)q(es)h(not)g(require)e(m)o(ultithreaded)e(pro)q (cesses.\))164 2061 y Fk(Discussion:)39 b Fj(A)14 b(con)o(text)f(handle)i (migh)o(t)f(b)q(e)g(implemen)o(ted)i(as)e(a)f(p)q(oin)o(ter)h(to)g(a)f (structure)164 2117 y(that)f(consists)h(of)g(con)o(text)f(lab)q(el)j(\(that)d (is)h(carried)h(b)o(y)f(messages)f(sen)o(t)h(within)h(this)g(con)o(text\))164 2173 y(and)20 b(a)g(con)o(text)g(mem)o(b)q(er)g(table,)i(that)d(translates)h (pro)q(cess)h(ranks)e(within)j(a)e(con)o(text)f(to)164 2230 y(absolute)g(addresses)g(or)g(to)f(routing)h(information.)31 b(Of)19 b(course,)h(other)f(implemen)o(tations)164 2286 y(are)c(p)q(ossible,) i(including)h(implemen)o(tations)e(that)f(do)g(not)g(require)h(eac)o(h)g(con) o(text)f(mem)o(b)q(er)164 2343 y(to)g(store)f(a)h(full)h(list)g(of)f(the)g (con)o(text)g(mem)o(b)q(ers.)237 2399 y(Con)o(texts)g(can)h(b)q(e)g(used)g (only)h(on)e(the)h(pro)q(cess)g(where)g(they)g(w)o(ere)f(created.)22 b(Since)17 b(the)164 2456 y(con)o(text)e(carries)g(information)h(on)f(the)g (group)g(of)g(pro)q(cesses)h(that)e(b)q(elong)i(to)f(this)h(con)o(text,)961 2599 y Fl(7)p eop %%Page: 8 8 bop 164 307 a Fj(a)14 b(pro)q(cess)g(can)g(send)g(a)g(message)g(within)h(a)e (con)o(text)h(only)g(to)g(other)f(pro)q(cesses)i(that)e(b)q(elong)164 364 y(to)i(that)g(con)o(text.)20 b(Th)o(us,)15 b(eac)o(h)h(pro)q(cess)f (needs)i(to)e(k)o(eep)g(trac)o(k)g(only)h(of)f(the)h(con)o(texts)f(that)164 420 y(where)j(created)f(at)g(that)g(pro)q(cess;)h(the)g(total)f(n)o(um)o(b)q (er)h(of)f(con)o(texts)f(p)q(er)i(pro)q(cess)g(is)g(lik)o(ely)164 477 y(to)d(b)q(e)g(small.)237 533 y(The)23 b(only)f(di\013erence)i(I)e(see)h (b)q(et)o(w)o(een)f(this)h(curren)o(t)f(de\014nition)i(of)e(con)o(text,)h (whic)o(h)164 589 y(subsumes)18 b(the)f(group)h(concept,)g(and)f(a)g(pared)h (do)o(wn)f(de\014nition,)j(if)e(that)e(I)i(assume)f(here)164 646 y(that)g(pro)q(cess)g(n)o(um)o(b)q(ering)h(is)g(relativ)o(e)g(to)f(the)g (con)o(text,)g(rather)g(then)g(b)q(eing)i(global,)f(th)o(us)164 702 y(requiring)f(a)e(con)o(text)g(mem)o(b)q(er)h(table.)22 b(I)16 b(argue)g(that)f(this)h(is)g(not)g(m)o(uc)o(h)f(added)i(o)o(v)o (erhead,)164 759 y(and)e(giv)o(es)h(m)o(uc)o(h)f(additional)i(needed)f (functionalit)o(y)l(.)239 853 y Fc(\017)24 b Fj(If)14 b(a)f(new)g(con)o(text) g(is)h(created)g(b)o(y)f(cop)o(ying)h(a)f(previous)h(con)o(text,)f(then)h (one)f(do)q(es)h(not)286 909 y(need)19 b(a)e(new)h(mem)o(b)q(er)g(table;)i (rather,)d(one)h(needs)h(just)e(a)h(new)g(con)o(text)f(lab)q(el)j(and)286 966 y(a)e(new)h(p)q(oin)o(ter)g(to)e(the)i(same)f(old)h(con)o(text)f(mem)o(b) q(er)g(table.)30 b(This)19 b(holds)g(true,)g(in)286 1022 y(particular,)c(for) g(con)o(texts)f(that)h(include)j(all)e(pro)q(cesses.)239 1116 y Fc(\017)24 b Fj(A)13 b(con)o(text)g(mem)o(b)q(er)h(table)g(mak)o(es)f(sure) h(that)f(a)g(message)g(is)h(sen)o(t)f(only)h(to)f(a)h(pro)q(cess)286 1172 y(that)f(can)h(execute)g(in)h(the)f(con)o(text)f(of)g(the)h(message.)19 b(The)14 b(alternativ)o(e)g(mec)o(hanism,)286 1229 y(whic)o(h)e(is)g(c)o(hec) o(king)g(at)f(reception,)i(is)f(less)g(e\016cien)o(t,)g(and)g(requires)g (that)f(eac)o(h)g(con)o(text)286 1285 y(lab)q(el)18 b(b)q(e)g(system-wide)f (unique.)26 b(This)17 b(requires)g(that,)f(to)g(the)h(least,)g(all)g(pro)q (cesses)286 1342 y(in)e(a)f(con)o(text)f(execute)i(a)f(collectiv)o(e)i (agreemen)o(t)d(algorithm)i(at)e(the)h(creation)h(of)e(this)286 1398 y(con)o(text.)239 1492 y Fc(\017)24 b Fj(The)14 b(use)g(of)f(relativ)o (e)h(addressing)g(within)h(eac)o(h)f(con)o(text)f(is)h(needed)h(to)e(supp)q (ort)h(true)286 1548 y(mo)q(dular)19 b(dev)o(elopmen)o(t)g(of)f(sub)q (computations)h(that)e(execute)i(on)f(a)g(subset)h(of)f(the)286 1605 y(pro)q(cesses.)h(There)14 b(is)f(also)g(a)g(big)h(adv)m(an)o(tage)e(in) i(using)g(the)f(same)g(con)o(text)f(construct)286 1661 y(for)i(collectiv)o(e) j(comm)o(unications)f(as)f(w)o(ell.)164 2009 y Fi(1.4.1)55 b(Con)n(text)19 b(Op)r(erations)164 2101 y Fl(A)h(global)i(con)o(text)e Fi(ALL)g Fl(is)h(prede\014ned.)35 b(All)20 b(pro)q(cesses)h(b)q(elong)h(to)f (this)g(con)o(text)164 2162 y(when)d(computation)g(starts.)28 b(MPI)18 b(do)q(es)h(not)g(sp)q(ecify)e(ho)o(w)i(pro)q(cesses)g(are)f (initially)164 2222 y(rank)o(ed)h(within)f(the)h(con)o(text)f(ALL.)h(It)g(is) g(exp)q(ected)f(that)h(the)g(start-up)h(pro)q(cedure)164 2282 y(used)d(to)g(initiate)f(an)i(MPI)e(program)h(\(at)g(load-time)f(or)h (run-time\))e(will)h(pro)o(vide)g(in-)164 2342 y(formation)d(or)h(con)o(trol) g(on)g(this)g(initial)e(ranking)j(\(e.g.,)e(b)o(y)g(sp)q(ecifying)g(that)h (pro)q(cesses)164 2402 y(are)20 b(rank)o(ed)g(according)g(to)h(their)e (pid's,)h(or)h(according)f(to)g(the)g(ph)o(ysical)f(addresses)961 2599 y(8)p eop %%Page: 9 9 bop 164 307 a Fl(of)15 b(the)g(executing)f(pro)q(cessors,)i(or)f(according)g (to)h(a)f(n)o(um)o(b)q(ering)e(sc)o(heme)g(sp)q(eci\014ed)i(at)164 367 y(load)i(time\).)164 548 y Fk(Discussion:)24 b Fj(If)17 b(w)o(e)g(think)h(of)f(adding)h(new)f(pro)q(cesses)h(at)e(run-time,)i(then)f Ff(ALL)g Fj(con)o(v)o(eys)164 608 y(the)e(wrong)g(impression,)h(since)g(it)g (is)f(just)g(the)g(initial)j(set)d(of)f(pro)q(cesses.)237 789 y Fl(The)i(follo)o(wing)g(op)q(erations)i(are)e(a)o(v)m(ailable)g(for)g (creating)g(new)g(con)o(texts.)164 909 y Fi(MPI)p 279 909 17 2 v 20 w(COPY)p 461 909 V 22 w(CONTEXT\(new)n(con)n(text,)i(con)n(text\))237 969 y Fl(Create)f(a)g(new)f(con)o(text)g(that)h(includes)e(all)h(pro)q (cesses)i(in)e(the)g(old)h(con)o(text.)k(The)164 1029 y(rank)c(of)f(the)g (pro)q(cesses)h(in)f(the)h(previous)f(con)o(text)f(is)h(preserv)o(ed.)21 b(The)16 b(call)g(m)o(ust)f(b)q(e)164 1090 y(executed)j(b)o(y)i(all)f(pro)q (cesses)i(in)e(the)h(old)g(con)o(text.)31 b(It)19 b(is)h(a)g(blo)q(c)o(king)f (call:)28 b(No)20 b(call)164 1150 y(returns)c(un)o(til)f(all)h(pro)q(cesses)h (ha)o(v)o(e)e(called)h(the)g(function.)21 b(The)16 b(parameters)f(are)164 1264 y Fi(OUT)k(new)n(con)n(text)24 b Fl(handle)12 b(to)h(newly)f(created)g (con)o(text.)19 b(The)13 b(handle)f(should)h(not)286 1324 y(b)q(e)j(asso)q (ciated)i(with)e(an)g(ob)s(ject)g(b)q(efore)g(the)g(call.)164 1426 y Fi(IN)i(con)n(text)24 b Fl(handle)16 b(to)h(old)f(con)o(text)164 1649 y Fk(Discussion:)35 b Fj(I)11 b(considered)h(adding)f(a)f(string)g (parameter,)g(to)g(pro)o(vide)h(a)e(unique)j(iden)o(ti\014er)164 1706 y(to)j(the)h(next)f(con)o(text.)21 b(But,)15 b(in)i(an)e(en)o(vironmen)o (t)h(where)g(pro)q(cesses)g(are)f(single)i(threaded,)164 1762 y(this)j(is)h(not)f(m)o(uc)o(h)g(help:)31 b(Either)20 b(all)h(pro)q(cesses)g (agree)f(on)g(the)g(order)g(they)g(create)g(new)164 1819 y(con)o(texts,)15 b(or)h(the)g(application)i(deadlo)q(c)o(ks.)23 b(A)16 b(k)o(ey)g(ma)o(y)f (help)i(in)g(an)f(en)o(vironmen)o(t)h(where)164 1875 y(pro)q(cesses)f(are)g (m)o(ultithreaded,)h(to)e(distinguish)j(call)f(from)e(distinct)i(threads)f (of)f(the)h(same)164 1932 y(pro)q(cess;)f(but)g(it)h(migh)o(t)f(b)q(e)h (simpler)g(to)f(use)g(a)g(m)o(utex)g(algorithm)g(at)g(eac)o(h)g(pro)q(cess.) 237 1992 y Fk(Implemen)o(tation)23 b(note:)k Fj(No)19 b(comm)o(unication)h (is)f(needed)i(to)d(create)h(a)f(new)i(con-)164 2052 y(text,)g(b)q(ey)o(ond)g (a)f(barrier)h(sync)o(hronization;)i(all)e(pro)q(cesses)g(can)g(agree)f(to)g (use)h(the)f(same)164 2112 y(naming)c(sc)o(heme)f(for)g(successiv)o(e)i (copies)f(of)f(the)g(same)g(con)o(text.)19 b(Also,)c(no)f(new)g(rank)g(table) 164 2172 y(is)i(needed,)g(just)f(a)g(new)g(con)o(text)g(lab)q(el)h(and)g(a)f (new)g(p)q(oin)o(ter)h(to)e(the)i(same)f(old)g(table.)164 2413 y Fi(MPI)p 279 2413 V 20 w(NEW)p 438 2413 V 20 w(CONTEXT\(new)n(con)n(text,)j (con)n(text,)g(k)n(ey)-5 b(,)18 b(index\))961 2599 y Fl(9)p eop %%Page: 10 10 bop 164 307 a Fi(OUT)19 b(new)n(con)n(text)24 b Fl(handle)13 b(to)g(newly)g(created)f(con)o(text)g(at)i(calling)e(pro)q(cess.)21 b(This)286 367 y(handle)16 b(should)h(not)g(b)q(e)f(asso)q(ciated)h(with)f (an)h(ob)s(ject)f(b)q(efore)g(the)g(call.)164 469 y Fi(IN)i(con)n(text)24 b Fl(handle)16 b(to)h(old)f(con)o(text)164 571 y Fi(IN)i(k)n(ey)24 b Fl(in)o(teger)164 672 y Fi(IN)18 b(index)25 b Fl(in)o(teger)237 787 y(A)20 b(new)g(con)o(text)f(is)g(created)h(for)g(eac)o(h)f(distinct)g(v)m (alue)h(of)g Fg(key)p Fl(;)g(this)g(con)o(text)f(is)164 847 y(shared)e(b)o(y)g(all)f(pro)q(cesses)h(that)h(made)d(the)i(call)f(with)h (this)g(k)o(ey)e(v)m(alue.)23 b(Within)16 b(eac)o(h)164 907 y(new)k(con)o(text)g(the)g(pro)q(cesses)h(are)g(rank)o(ed)f(according)h(to)g (the)f(order)h(of)f(the)h Fg(index)164 967 y Fl(v)m(alues)16 b(they)g(pro)o(vided;)f(in)h(case)g(of)h(ties,)e(pro)q(cesses)i(are)f(rank)o (ed)g(according)g(to)h(their)164 1027 y(rank)f(in)g(the)g(old)h(con)o(text.) 237 1088 y(This)e(call)e(is)h(blo)q(c)o(king:)20 b(No)14 b(call)f(returns)i (un)o(til)e(all)h(pro)q(cesses)g(in)g(the)g(old)h(con)o(text)164 1148 y(executed)g(the)h(call.)237 1208 y(P)o(articular)g(uses)g(of)h(this)f (function)g(are:)237 1268 y(\(i\))h(Reordering)h(pro)q(cesses:)25 b(All)16 b(pro)q(cesses)i(pro)o(vide)f(the)h(same)e Fg(key)h Fl(v)m(alue,)g(and)164 1328 y(pro)o(vide)e(their)h(index)f(in)h(the)g(new)g (order.)237 1389 y(\(ii\))e(Splitting)g(a)h(con)o(text)f(in)o(to)h(sub)q(con) o(texts,)f(while)g(preserving)g(the)h(old)g(relativ)o(e)164 1449 y(order)23 b(among)f(pro)q(cesses:)35 b(All)21 b(pro)q(cesses)i(pro)o (vide)f(the)g(same)g Fg(index)f Fl(v)m(alue,)i(and)164 1509 y(pro)o(vide)15 b(a)i(k)o(ey)e(iden)o(tifying)g(their)g(new)h(sub)q(con)o (text.)164 1629 y Fi(MPI)p 279 1629 17 2 v 20 w(RANK\(rank,)i(con)n(text\)) 164 1743 y(OUT)h(rank)24 b Fl(in)o(teger)164 1845 y Fi(IN)18 b(con)n(text)24 b Fl(con)o(text)15 b(handle)237 1959 y(Return)h(the)g(rank)h (of)f(the)g(calling)g(pro)q(cess)g(within)g(the)g(sp)q(eci\014ed)g(con)o (text.)164 2080 y Fi(MPI)p 279 2080 V 20 w(SIZE\(size,)j(con)n(text\))164 2194 y(OUT)g(size)24 b Fl(in)o(teger)164 2296 y Fi(IN)18 b(con)n(text)24 b Fl(con)o(text)15 b(handle)237 2410 y(Return)h(the)g(n)o(um)o(b)q(er)f(of)h (pro)q(cesses)h(that)g(b)q(elong)f(to)h(the)f(sp)q(eci\014ed)g(con)o(text.) 949 2599 y(10)p eop %%Page: 11 11 bop 164 307 a Fi(Usage)25 b(note)49 b Fl(Use)21 b(of)i(con)o(texts)e(for)i (libraries:)32 b(Eac)o(h)22 b(library)f(ma)o(y)g(pro)o(vide)g(an)164 367 y(initialization)15 b(routine)i(that)g(is)g(to)g(b)q(e)g(called)f(b)o(y)h (all)f(pro)q(cesses,)h(and)h(that)f(generate)164 428 y(a)g(con)o(text)e(for)i (the)f(use)g(of)g(that)h(library)l(.)237 488 y(Use)11 b(of)h(con)o(texts)e (for)i(functional)f(decomp)q(osition:)18 b(A)11 b(harness)h(program,)g (running)164 548 y(in)18 b(the)g(con)o(text)g Fg(ALL)f Fl(generates)i(a)g (sub)q(con)o(text)f(for)h(eac)o(h)f(mo)q(dule)f(and)i(then)f(starts)164 608 y(the)e(submo)q(dule)f(within)h(the)g(corresp)q(onding)h(con)o(text.)237 668 y(Use)k(of)g(con)o(texts)g(for)g(collectiv)o(e)e(comm)o(unic)o(ation:)29 b(A)21 b(con)o(text)f(is)h(created)g(for)164 729 y(eac)o(h)16 b(group)h(of)f(pro)q(cesses)h(where)f(collectiv)o(e)e(comm)o(uni)o(cation)g (is)i(to)g(o)q(ccur.)237 789 y(Use)i(of)g(con)o(texts)g(for)g(con)o (text-switc)o(hing)f(among)h(sev)o(eral)f(parallel)g(executions:)164 849 y(A)h(pream)o(ble)e(co)q(de)i(is)g(used)g(to)g(generate)g(a)h(di\013eren) o(t)e(con)o(text)g(for)i(eac)o(h)e(execution;)164 909 y(this)j(pream)o(ble)d (co)q(de)j(needs)g(to)g(use)f(a)i(m)o(utual)d(exclusion)g(proto)q(col)j(to)f (mak)o(e)e(sure)164 969 y(eac)o(h)e(thread)g(claims)e(the)i(righ)o(t)g(con)o (text.)164 1139 y Fk(Discussion:)59 b Fj(If)20 b(pro)q(cess)g(handles)h(are)e (made)h(explicit)i(in)e(MPI,)g(then)g(an)f(additional)164 1195 y(function)c(needed)h(is)f Fk(MPI)p 646 1195 16 2 v 18 w(PR)o(OCESS\(pro)q (cess,)i(con)o(text,)f(rank\))p Fj(,)e(whic)o(h)i(returns)e(a)164 1252 y(handle)i(to)f(the)g(pro)q(cess)h(iden)o(ti\014ed)h(b)o(y)e(the)g Ff(rank)g Fj(and)g Ff(context)g Fj(parameters.)237 1312 y(A)10 b(p)q(ossible)i(addition)f(is)g(a)f(function)h(of)f(the)g(form)f Fk(MPI)p 1199 1312 V 18 w(CREA)l(TE)p 1434 1312 V 19 w(CONTEXT\(new)o(con)o (text,)164 1372 y(list)p 236 1372 V 20 w(of)p 298 1372 V 19 w(pro)q(cess)p 484 1372 V 18 w(handles\))19 b Fj(whic)o(h)g(creates)f(a)g (new)h(con)o(text)e(out)h(of)g(an)g(explicit)i(list)f(of)164 1432 y(mem)o(b)q(ers)e(\(and)g(rank)g(them)g(in)h(their)g(order)f(of)g(o)q (ccurrence)h(in)g(the)f(list\).)26 b(This,)18 b(coupled)164 1492 y(with)13 b(a)f(mec)o(hanism)h(for)e(requiring)j(the)e(spa)o(wning)h(of) f(new)g(pro)q(cesses)h(to)f(the)g(computation,)164 1553 y(will)20 b(allo)o(w)e(to)g(create)g(a)g(new)g(all)h(inclusiv)o(e)i(con)o(text)c(that)h (includes)i(the)e(additional)i(pro-)164 1613 y(cesses.)k(Ho)o(w)o(ev)o(er,)16 b(I)h(opp)q(ose)g(the)f(idea)h(of)g(requiring)g(dynamic)h(pro)q(cess)f (creation)f(as)g(part)164 1673 y(of)f(MPI.)f(Man)o(y)h(implemen)o(ters)h(w)o (an)o(t)e(to)g(run)i(MPI)f(in)h(an)f(en)o(vironmen)o(t)g(where)g(pro)q (cesses)164 1733 y(are)g(statically)h(allo)q(cated)g(at)f(load-time.)164 1983 y Fi(1.4.2)55 b(Error)18 b(Handling)164 2076 y Fl(It)12 b(is)g(assumed)g(that)g(MPI)g(is)g(implem)o(en)o(te)o(d)e(on)i(top)h(of)g(an) f(error-free)g(comm)o(unicati)o(on)164 2136 y(subsystem:)18 b(A)12 b(message)g(sen)o(t)g(is)g(alw)o(a)o(ys)g(receiv)o(ed)e(correctly)l(,) i(and)h(the)f(user)g(do)q(es)h(not)164 2196 y(need)e(to)g(c)o(hec)o(k)e(for)i (transmission)g(errors,)g(time-outs,)g(and)g(the)g(lik)o(es.)18 b(In)11 b(other)g(w)o(ords,)164 2256 y(MPI)21 b(do)q(es)h(not)g(pro)o(vide)e (mec)o(hanisms)f(to)i(deal)g(with)h(failures)e(in)i(the)f(underlying)164 2316 y(comm)o(unic)o(ation)15 b(subsystem)g({)j(it)e(is)h(the)f(resp)q (onsibilit)o(y)g(of)h(the)f(MPI)h(implem)o(en)n(ter)164 2377 y(to)i(insulate)g(the)g(user)f(from)g(suc)o(h)h(errors)g(\(or)g(to)h (re\015ect)e(them)f(as)j(global)f(program)164 2437 y(failures\).)i(The)16 b(same)f(holds)i(true)f(for)g(no)q(de)h(failures.)949 2599 y(11)p eop %%Page: 12 12 bop 237 307 a Fl(Of)15 b(course,)h(MPI)f(programs)g(ma)o(y)f(still)g(b)q(e)i (erroneous.)21 b(A)15 b Fi(program)j(error)d Fl(can)164 367 y(o)q(ccur)i(when)h(an)f(MPI)g(call)g(is)g(called)f(with)h(an)h(incorrect)e (parameter)g(\(non-existing)164 428 y(destination)f(in)g(a)h(send)g(op)q (eration,)g(bu\013er)f(to)q(o)i(small)c(in)i(a)h(receiv)o(e)d(op)q(eration,)j (etc.\))164 488 y(This)k(t)o(yp)q(e)g(of)h(error)f(w)o(ould)h(o)q(ccur)f(in)g (an)o(y)g(impleme)o(n)o(tation.)31 b(In)20 b(addition,)h(a)g Fi(re-)164 548 y(source)g(error)e Fl(ma)o(y)e(o)q(ccur)i(when)g(a)h(program)f (exceeds)e(the)i(amoun)o(t)f(of)i(a)o(v)m(ailable)164 608 y(system)e (resources)i(\(n)o(um)o(b)q(er)e(of)i(p)q(ending)h(messages,)f(system)e (bu\013ers,)j(etc.\).)31 b(The)164 668 y(o)q(ccurrence)13 b(of)i(this)f(t)o (yp)q(e)f(of)i(error)f(dep)q(ends)g(on)h(the)f(amoun)o(t)f(of)i(a)o(v)m (ailable)e(resources)164 729 y(in)19 b(the)g(system)f(and)j(the)e(resource)g (allo)q(cation)h(mec)o(hanism)c(used;)21 b(this)e(ma)o(y)f(di\013er)164 789 y(from)f(system)g(to)h(system.)26 b(The)18 b(recomme)o(nded)d(impleme)o (n)o(tation)h(pro\014le)h(pro)o(vides)164 849 y(sev)o(eral)d(mec)o(hanism)o (s)f(to)i(alleviate)e(the)i(p)q(ortabilit)o(y)f(problem)f(this)i(represen)o (ts.)20 b(One)164 909 y(can)c(also)h(write)f Fi(safe)g Fl(programs,)g(that)h (are)f(not)h(sub)s(ject)e(to)i(resource)f(errors.)237 969 y(All)21 b(MPI)g(pro)q(cedure)h(calls)f(return)h(an)h(error)e(parameter)g(that)h (indicates)g(suc-)164 1029 y(cessful)g(completion)f(of)i(the)f(op)q(eration,) j(or)e(the)f(error)h(condition)f(that)h(o)q(ccurred,)164 1090 y(otherwise.)237 1150 y(The)h(recommende)o(d)d(implem)o(en)o(tation)g (pro\014le)j(in)f(a)h(POSIX)f(en)o(vironmen)o(t)e(is)164 1210 y(for)h(an)o(y)f(MPI)g(routine)g(that)g(encoun)o(ters)g(a)h(reco)o(v)o (erable)e(error)h(to)h(store)f(an)h(error)164 1270 y(n)o(um)o(b)q(er)f(in)h (a)h(global)g(v)m(ariable)f(\()p Fh(errno)h Fl(in)f(a)h(C)g(en)o(vironmen)o (t\))d(and)j(generate)f(an)164 1330 y Fh(MPI)d(err)n(or)e(signal)p Fl(,)i(using)g(a)f(sp)q(ecial)g(signal)g(v)m(alue.)27 b(The)18 b(default)g(handler)g(for)g(this)164 1391 y(signal)13 b(terminates)e(the)h (execution)g(of)h(all)f(in)o(v)o(olv)o(ed)e(pro)q(cesses,)k(with)f(a)g (suitable)f(error)164 1451 y(message)19 b(b)q(eing)i(returned)f(to)g(the)g (user.)33 b(Ho)o(w)o(ev)o(er,)19 b(the)h(user)h(can)f(pro)o(vide)f(his)i(or) 164 1511 y(her)14 b(o)o(wn)g(signal)h(handling)f(routine.)20 b(In)14 b(particular,)g(the)g(user)g(can)g(sp)q(ecify)f(a)i(\\no)q(op")164 1571 y(signal)i(handler,)f(th)o(us)g(relegating)g(all)g(error)h(handling)f (to)h(the)g(user)f(co)q(de,)g(using)h(the)164 1631 y(error)f(parameters)f (returned)h(b)o(y)g(the)g(MPI)g(calls.)237 1692 y(MPI)f(calls)f(ma)o(y)g (initiate)g(op)q(erations)i(that)g(con)o(tin)o(ue)e(async)o(hronously)h (after)g(the)164 1752 y(call)k(returned.)30 b(Th)o(us,)20 b(the)f(op)q (eration)i(ma)o(y)d(return)h(with)g(a)h(co)q(de)g(indicating)f(suc-)164 1812 y(cessful)14 b(completion,)e(y)o(et)i(later)g(cause)h(an)g(error)f (exception)g(to)h(b)q(e)f(raised.)21 b(If)14 b(there)g(is)164 1872 y(a)i(subsequen)o(t)f(call)g(that)h(relates)f(to)g(the)h(same)e(op)q (eration)i(\(e.g.,)f(a)h(call)e(that)i(v)o(eri\014es)164 1932 y(that)f(an)f(async)o(hronous)i(op)q(eration)f(has)g(completed\))d(then)i (the)g(error)g(parameter)f(as-)164 1993 y(so)q(ciated)18 b(with)f(this)g (call)g(will)f(b)q(e)i(used)f(to)h(indicate)e(the)h(nature)h(of)g(the)f (error.)24 b(In)18 b(a)164 2053 y(few)13 b(cases,)h(the)g(error)f(ma)o(y)f(o) q(ccur)i(after)g(all)f(calls)g(that)h(relate)f(to)h(the)f(op)q(eration)i(ha)o (v)o(e)164 2113 y(completed,)e(so)k(that)g(no)f(error)g(parameter)f(can)h(b)q (e)h(used)f(to)g(indicate)f(the)h(nature)h(of)164 2173 y(the)h(error)h (\(e.g.,)f(an)h(error)f(in)h(a)g(send)f(with)h(the)f(ready)g(mo)q(de\).)28 b(In)18 b(suc)o(h)g(cases,)h(an)164 2233 y(error)d(will)f(b)q(e)i (undetected,)e(if)g(the)h(user)h(disabled)e(the)h(MPI)g(error)g(signal.)164 2414 y Fk(Discussion:)40 b Fj(The)16 b(alternativ)o(e)f(c)o(hoice)h(is)g(to)f (ha)o(v)o(e)f(fatal)h(and)h(non-fatal)f(signals.)949 2599 y Fl(12)p eop %%Page: 13 13 bop 164 452 a Fm(1.5)70 b(Messages)164 544 y Fl(A)16 b(message)f(consists)i (of)g(an)f Fh(envelop)n(e)i Fl(and)f Fh(data)p Fl(.)164 674 y Fi(1.5.1)55 b(Data)164 766 y Fl(The)17 b(data)h(part)g(of)f(a)h(message)e (consists)i(of)f(a)h(sequence)e(of)h(v)m(alues,)g(eac)o(h)g(of)g(a)h(basic) 164 826 y(datat)o(yp)q(e)k(in)f(the)g(host)h(language.)37 b(Th)o(us,)22 b(in)f(F)l(ortran,)i(a)f(message)e(consists)i(of)g(a)164 887 y(sequence)14 b(of)h(v)m(alues)h(that)f(are)g(eac)o(h)g(of)g(t)o(yp)q(e)g Fg(INTEGER)p Fl(,)d Fg(REAL)p Fl(,)h Fg(DOUBLE)24 b(PRECISION)o Fl(,)164 947 y Fg(COMPLEX)p Fl(,)16 b Fg(LOGICAL)p Fl(,)f(or)k(\(length)g (1\))f Fg(CHARACTER)p Fl(.)d(A)k(message)e(ma)o(y)g(mix)g(v)m(alues)i(of)164 1007 y(di\013eren)o(t)c(t)o(yp)q(es.)164 1188 y Fk(Discussion:)40 b Fj(Ma)o(y)15 b(also)g(need)h Ff(DOUBLE)23 b(COMPLEX)14 b Fj(in)i(F)l(ortran.)164 1438 y Fi(1.5.2)55 b(En)n(v)n(elop)r(e)164 1530 y Fl(The)16 b(follo)o(wing)g(information)f(is)h(asso)q(ciated)i(with)e (eac)o(h)f(message:)164 1632 y Fi(source)24 b Fl(The)16 b(rank)h(the)f (sending)g(pro)q(cess)164 1734 y Fi(destination)25 b Fl(The)17 b(rank)f(of)h(the)f(receiving)e(pro)q(cess)164 1835 y Fi(tag)24 b Fl(User)16 b(de\014ned)164 1937 y Fi(con)n(text)24 b Fl(handle)237 2039 y(The)15 b(range)g(of)g(v)m(alid)f(v)m(alues)h(for)g(the)f Fi(source)g Fl(and)h Fi(destination)h Fl(\014elds)e(is)h Fg(0)25 b(...)164 2099 y(n-1)p Fl(,)16 b(where)h Fg(n)f Fl(is)h(the)g(n)o(um)o(b)q (er)f(of)h(pro)q(cesses)h(in)f(the)g(curren)o(t)f(con)o(text.)23 b(The)17 b(ranges)164 2159 y(of)j(v)m(alid)g(v)m(alues)g(for)h Fg(tag)e Fl(is)g(impleme)o(n)o(tation)e(dep)q(enden)o(t,)k(and)f(can)g(b)q(e) h(found)f(b)o(y)164 2219 y(calling)13 b(a)h(suitable)g(query)f(function,)h (as)g(describ)q(ed)g(in)f(Section)h Fi(??)p Fl(.)21 b Fg(Context)11 b Fl(should)164 2279 y(b)q(e)16 b(a)h(con)o(text)e(shared)i(b)o(y)f(b)q(oth)h (source)f(and)h(destination.)237 2340 y(The)h Fg(tag)e Fl(\014eld)h(can)h(b)q (e)g(arbitrarily)e(set)i(b)o(y)f(the)g(application,)g(and)h(can)g(b)q(e)g (used)164 2400 y(to)f(distinguish)f(di\013eren)o(t)f(messages.)949 2599 y(13)p eop %%Page: 14 14 bop 237 307 a Fl(The)12 b(actual)g(mec)o(hanism)d(used)j(to)g(asso)q(ciate)h (an)f(en)o(v)o(elop)q(e)f(with)g(a)i(message)e(is)g(im-)164 367 y(plemen)o(tation)h(dep)q(enden)o(t;)i(some)f(of)i(the)f(information)f (\(e.g.,)h Fi(sender)f Fl(or)i Fi(receiv)n(er)p Fl(\))164 428 y(ma)o(y)g(b)q(e)h(implicit,)c(and)17 b(need)f(not)h(b)q(e)f(explicitly)e (carried)h(b)o(y)h(a)h(message.)164 572 y Fm(1.6)70 b(Data)23 b(Bu\013ers)164 664 y Fl(The)18 b(basic)h(p)q(oin)o(t)f(to)h(p)q(oin)o(t)f (comm)o(unicati)o(on)e(op)q(erations)k(are)e Fi(send)g Fl(and)h Fi(receiv)n(e)p Fl(.)164 724 y(A)13 b Fi(send)h Fl(op)q(eration)h(creates)f (a)g(message;)g(the)f(message)h(data)g(is)g(assem)o(bled)e(from)h(the)164 785 y Fi(send)k(bu\013er)p Fl(.)j(A)14 b Fi(receiv)n(e)f Fl(op)q(eration)j (consumes)d(a)i(message;)f(the)g(message)g(data)h(is)164 845 y(mo)o(v)o(ed)g(in)o(to)i(the)f Fi(receiv)n(e)j(bu\013er)p Fl(.)k(The)17 b(sp)q(eci\014cation)g(of)g(send)g(or)h(receiv)o(e)c(bu\013ers) 164 905 y(uses)i(the)g(same)g(syn)o(tax.)237 965 y(A)j(bu\013er)g(consists)h (of)g(a)f(sequence)f Fi(bu\013er)k(comp)r(onen)n(ts)p Fl(.)30 b(Eac)o(h)19 b(comp)q(onen)o(t)164 1025 y(consists)14 b(of)h(a)f(sequence)f (v)m(ariables)h(of)h(the)f(same)f(basic)h(t)o(yp)q(e.)20 b(There)13 b(are)h(three)g(basic)164 1086 y(t)o(yp)q(es)i(of)h(bu\013er)f(comp)q(onen)o (ts:)164 1199 y Fi(blo)r(c)n(k)25 b Fl(A)15 b(sequence)h(of)g(con)o(tiguous)h (v)m(alues)f(of)h(the)f(same)f(basic)h(t)o(yp)q(e,)f(sp)q(eci\014ed)h(b)o(y) 286 1301 y Fi(start)24 b Fl(Initial)15 b(elemen)o(t)286 1382 y Fi(len)25 b Fl(Num)o(b)q(er)14 b(of)i(elemen)o(ts)e(\()p Fg(len)f Fe(\025)g Fg(0)p Fl(\))286 1462 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)164 1564 y Fi(v)n(ector)24 b Fl(A)14 b(sequence)f(of)i(equally)f(spaced)h(and)g(equally)e(sized)h(blo)q (c)o(ks)h(of)g(elemen)o(ts)d(of)286 1624 y(the)k(same)f(basic)h(t)o(yp)q(e,)g (sp)q(eci\014ed)f(b)o(y)286 1726 y Fi(start)24 b Fl(Initial)15 b(elemen)o(t)286 1807 y Fi(len)25 b Fl(Num)o(b)q(er)14 b(of)i(elemen)o(ts)e (\()p Fg(len)f Fe(\025)g Fg(0)p Fl(\))286 1887 y Fi(stride)24 b Fl(Num)o(b)q(er)15 b(of)h(elemen)o(ts)e(b)q(et)o(w)o(een)h(the)h(start)h (of)f(eac)o(h)g(blo)q(c)o(k)286 1968 y Fi(len)n(blk)25 b Fl(Num)o(b)q(er)15 b(of)h(elemen)o(ts)e(in)h(eac)o(h)h(blo)q(c)o(k)g(\()p Fg(lenblk)c Fe(\024)h Fg(stride)p Fl(\))286 2049 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)286 2151 y(Note)10 b(that)i(a)f (constan)o(t)g(stride)f(b)q(ecomes)g(con)o(tiguous)h(when)g Fg(stride)24 b(=)h(lenblk)p Fl(.)286 2211 y(A)c(v)o(ector)g(bu\013er)g(comp)q (onen)o(t)g(can)h(b)q(e)f(an)h(arbitrary)g(submatrix)e(of)i(a)g(t)o(w)o(o-) 286 2271 y(dimensional)14 b(matrix.)164 2372 y Fi(indexed)24 b Fl(A)16 b(sequence)f(of)i(elemen)o(t)o(s)d(of)j(the)f(same)f(basic)h(t)o (yp)q(e,)f(sp)q(eci\014ed)h(b)o(y)286 2474 y Fi(start)24 b Fl(initial)15 b(elemen)o(t)949 2599 y(14)p eop %%Page: 15 15 bop 286 307 a Fi(list)p 363 307 17 2 v 21 w(of)p 429 307 V 20 w(indices)26 b Fl(List)12 b(of)g(displacemen)o(ts)d(of)j(the)f(elemen)o (ts)e(in)j(the)f(bu\013er)h(com-)393 367 y(p)q(onen)o(ts,)17 b(relativ)o(e)d(to)j(the)f(initial)f(elemen)n(t.)286 443 y Fi(datat)n(yp)r(e)24 b Fl(T)o(yp)q(e)16 b(of)g(elemen)o(ts)237 534 y(F)l(or)f(example,)d(if)i(a)h(F)l(ortran)g(arra)o(y)g(is)f(declared)g (as)h Fg(double)24 b(precision)e(a\(10\))p Fl(,)164 594 y(then)15 b(the)f(tuple)g Fd(<)p Fg(\(a\(3\),)24 b(\(0,3,6,2\))o(,)f(DOUBLE)p Fd(>)12 b Fl(sp)q(eci\014es)j(a)g(bu\013er)g(comp)q(onen)o(t)164 655 y(with)h(en)o(tries)f Fg(a\(3\))24 b(a\(6\),)g(a\(9\),)g(a\(5\))p Fl(.)164 824 y Fk(Discussion:)56 b Fj(Do)19 b(w)o(e)g(allo)o(w)g(en)o(tries)h (to)f(b)q(e)h(rep)q(eated)f(in)h(an)g(indexed)g(bu\013er)g(comp)q(o-)164 880 y(nen)o(t?)237 937 y(Do)15 b(w)o(e)g(allo)o(w)g(di\013eren)o(t)g (bu\013er)h(comp)q(onen)o(ts)f(to)f(o)o(v)o(erlap?)237 993 y(Do)e(w)o(e)g(require)h(in)g(an)f(v)o(ector)g(bu\013er)g(comp)q(onen)o(t)g (that)g Ff(len)g Fj(b)q(e)g(a)g(m)o(ultiple)i(of)e Ff(lenblk)p Fj(?)237 1234 y Fl(A)j(bu\013er)h(is)f(describ)q(ed)f(b)o(y)h(an)h(opaque)f (ob)s(ject)g(accessed)g(via)g(a)h Fi(bu\013er)h(handle)p Fl(.)164 1294 y(Suc)o(h)f(ob)s(ject)g(is)g(created)f(and)i(destro)o(y)o(ed)f(via)g (calls)f(to)i Fg(MPI)p 1292 1294 16 2 v 17 w(CREATE)e Fl(and)h Fg(MPI)p 1652 1294 V 18 w(FREE)p Fl(.)164 1354 y(It)k(is)h(asso)q(ciated)h (with)f(successiv)o(e)e(bu\013er)i(comp)q(onen)o(ts)f(b)o(y)g(calling)h(in)f (succession)164 1415 y(one)f(of)g(the)g(functions)g Fg(MPI)p 695 1415 V 17 w(ADD)p 790 1415 V 18 w(BLOCK)p Fl(,)d Fg(MPI)p 1046 1415 V 18 w(ADD)p 1142 1415 V 17 w(VECTOR)h Fl(or)i Fg(MPI)p 1472 1415 V 17 w(ADD)p 1567 1415 V 18 w(INDEX)p Fl(,)d(in)164 1475 y(order)h(to)g(app)q(end)h(a)g(comp)q(onen)o(t)e(to)h(the)g(bu\013er)g (asso)q(ciated)h(with)f(a)g(bu\013er)g(handle.)164 1535 y(A)f(bu\013er)h(ob)s (ject)f(can)g(b)q(e)h(destro)o(y)o(ed)e(only)h(if)g(there)g(is)g(no)h(p)q (ending)g(comm)o(unicati)o(on)164 1595 y(op)q(eration)j(using)g(it.)31 b(After)19 b(a)h(bu\013er)g(ob)s(ject)f(is)g(destro)o(y)o(ed)g(the)g(asso)q (ciated)i(bu\013er)164 1655 y(handle)16 b(is)g(unde\014ned.)164 1776 y Fi(MPI)p 279 1776 17 2 v 20 w(ADD)p 427 1776 V 21 w(BLOCK\()i (bu\013er,)g(start,)g(len,)h(datat)n(yp)r(e\))237 1896 y Fl(App)q(end)d(a)h (blo)q(c)o(k)f(comp)q(onen)o(t)f(to)i(bu\013er.)k(The)16 b(parameters)g(are:) 164 1981 y Fi(INOUT)i(bu\013er)24 b Fl(bu\013er)17 b(handle)164 2077 y Fi(IN)h(start)25 b Fl(bu\013er)16 b(comp)q(onen)o(t)f(initial)g (elemen)o(t)e(\(c)o(hoice\))164 2173 y Fi(IN)18 b(len)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(\(in)o(teger\))164 2269 y Fi(IN)18 b(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)d(\(status\))164 2414 y Fi(MPI)p 279 2414 V 20 w(ADD)p 427 2414 V 21 w(VEC\()19 b(bu\013er,)f(len,)g(stride,)g(len)n (blk,)i(datat)n(yp)r(e)e(\))949 2599 y Fl(15)p eop %%Page: 16 16 bop 237 307 a Fl(App)q(end)16 b(a)h(v)o(ector)e(bu\013er)i(comp)q(onen)o(t)e (to)i(bu\013er.)k(The)c(parameters)e(are:)164 409 y Fi(INOUT)j(bu\013er)24 b Fl(bu\013er)17 b(handle)164 511 y Fi(IN)h(start)25 b Fl(bu\013er)16 b(comp)q(onen)o(t)f(initial)g(elemen)o(t)e(\(c)o(hoice\))164 612 y Fi(IN)18 b(len)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(\(in)o (teger\))164 714 y Fi(IN)18 b(stride)25 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(b)q(et)o(w)o(een)i(the)h(start)h(of)g(eac)o(h)e(blo)q(c) o(k)h(\(in)o(teger\))164 816 y Fi(IN)i(len)n(blk)26 b Fl(Num)o(b)q(er)14 b(of)j(elemen)o(ts)c(in)j(eac)o(h)g(blo)q(c)o(k)f(\(in)o(teger\))164 917 y Fi(IN)j(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)d (\(status\))164 1079 y Fi(MPI)p 279 1079 17 2 v 20 w(ADD)p 427 1079 V 21 w(INDEX\()k(bu\013er,)g(start,)g(list)p 1077 1079 V 21 w(of)p 1143 1079 V 20 w(indices,)i(datat)n(yp)r(e\))237 1200 y Fl(App)q(end)c(an)h(indexed)e(bu\013er)i(comp)q(onen)o(t)e(to)i (bu\013er.)k(The)c(parameters)e(are:)164 1301 y Fi(INOUT)j(bu\013er)24 b Fl(bu\013er)17 b(handle)164 1403 y Fi(start)24 b Fl(initial)15 b(p)q(osition)i(for)f(indexing)g(\(c)o(hoice\))164 1505 y Fi(list)p 241 1505 V 21 w(of)p 307 1505 V 20 w(indices)26 b Fl(list)15 b(of)i(relativ)o(e)d(indices)i(of)g(en)o(tries)f(\(arra)o(y)h(of)h(in)o (tegers\))164 1606 y Fi(IN)h(datat)n(yp)r(e)24 b Fl(datat)o(yp)q(e)17 b(iden)o(ti\014er)d(\(status\))237 1708 y(Consider,)i(for)h(example,)c(the)j (follo)o(wing)g(fragmen)o(t)f(of)i(F)l(ortran)g(co)q(de)164 1822 y Fg(DOUBLE)23 b(PRECISION)g(A\(10,20\))164 1883 y(INTEGER)g(B,)i (C\(5,10\))164 1943 y(INTEGER)e(BH)164 2003 y(...)164 2063 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f(MPI_BUFFE)o(R,)f(MPI_PERSIST)o(EN)o(T\))164 2123 y(CALL)i(MPI_ADD_BL)o(OCK)e(\(BH,)i(B,)h(1,)g(MPI_INT\))164 2183 y(CALL)f(MPI_ADD_VE)o(C)f(\(BH,)h(A\(1,3\),)f(11,)h(4,)h(2,)g (MPI_DOUBLE\))164 2244 y(CALL)f(MPI_ADD_IN)o(DEX)o(\(BH)o(,)f(C\(3,7\),)g (\(4,2,1\),)f(MPI_INT\))949 2599 y Fl(16)p eop %%Page: 17 17 bop 237 307 a Fl(Then)15 b(the)g(bu\013er)g(asso)q(ciated)g(with)g(the)f (handle)h Fg(BH)f Fl(consists)h(of)g(the)g(sequence)f(of)164 367 y(v)m(ariables)237 428 y Fg(B,)25 b(A\(1,3\),)e(A\(2,3\),)g(A\(5,3\),)g (A\(6,3\),)g(A\(9,3\),)g(A\(10,3\),)g(A\(3,4\),)g(A\(4,4\),)164 488 y(A\(7,4\),)g(A\(8,4\),)g(A\(1,5\),)g(C\(2,8\),)g(C\(5,7\),)g(C\(4,7\))p Fl(.)237 548 y(A)11 b(message)f(created)g(from)g(this)h(bu\013er)g(will)f (consist)h(of)g(a)g(sequence)f(of)h(one)g(in)o(teger,)164 608 y(follo)o(w)o(ed)k(b)o(y)h(elev)o(en)e(double)j(precision)e(reals,)h(follo)o (w)o(ed)f(b)o(y)h(three)f(in)o(tegers.)237 668 y(A)d(bu\013er)g(handle)g(can) g(b)q(e)g(used)g(for)g(comm)o(unic)o(ation,)e(ev)o(en)h(if)g(it)h(is)g(not)g (asso)q(ciated)164 729 y(with)i(an)o(y)f(v)m(ariables)h(\(i.e.,)e(ev)o(en)h (if)g(it)h(w)o(as)g(not)g(set)g(b)o(y)g(an)o(y)f Fg(MPI)p 1354 729 16 2 v 18 w(ADD)p 1450 729 V 17 w(xxx)g Fl(call\).)19 b(Suc)o(h)164 789 y(handle)i(is)g(asso)q(ciated)h(with)e(an)i(empt)o(y)d(bu\013er,)j(and)f (a)h(message)e(created)g(from)g(it)164 849 y(con)o(tains)c(no)h(data.)164 1018 y Fk(Discussion:)237 1075 y Fj(The)h(main)g(mo)q(di\014cations)h(w.r.t.) 25 b(the)18 b(prop)q(osal)f(of)g(Gropp)h(and)f(Lusk)h(is)g(measuring)164 1131 y(length)g(in)g(elemen)o(ts,)f(rather)g(than)g(b)o(ytes.)25 b(Seems)17 b(more)g(natural,)g(since)h(t)o(yp)q(e)f(is)g(kno)o(wn.)164 1188 y(Also,)k(ob)s(ject)e(creation)h(uses)f(a)h(generic)g(function,)i (rather)d(than)g(a)g(function)i(sp)q(eci\014c)g(to)164 1244 y(bu\013er)c(descriptors.)28 b(As)17 b(result,)h(I)g(ga)o(v)o(e)f(up)h(on)f (the)h(size)g(parameter.)26 b(This)18 b(ma)o(y)f(not)g(b)q(e)164 1301 y(suc)o(h)k(a)g(loss:)32 b(it)21 b(is)g(not)g(clear)h(that)e(sp)q (ecifying)j(a)d(maxim)o(um)h(length)h(at)e(bu\013er)h(ob)s(ject)164 1357 y(creation)h(is)g(useful,)h(since)g(indices)g(of)e(arbitrary)g(size)i (ma)o(y)d(need)j(to)d(b)q(e)j(stored)d(in)j(the)164 1413 y(ob)s(ject.)164 1724 y Fi(1.6.1)55 b(Data)19 b(Con)n(v)n(ersion)164 1816 y Fl(The)d(t)o(yp)q(es)g(and)h(the)f(lo)q(cations)h(of)g(the)f(en)o(tries)f (used)h(to)h(create)e(a)i(message)f(is)g(solely)164 1876 y(determined)f(from) i(the)g(information)g(in)g(the)h(bu\013er)g(descriptor,)f(using)h(the)g (storage)164 1937 y(asso)q(ciation)g(rules)e(sp)q(eci\014ed)g(b)o(y)g(the)g (host)i(language)f(and)h(its)e(implem)o(en)n(tation;)e(the)164 1997 y(t)o(yp)q(e)22 b(and)h(the)g(lo)q(cations)g(of)g(these)g(en)o(tries)e (do)i(not)h(dep)q(end)e(on)i(the)e(declaration)164 2057 y(for)g(the)g (corresp)q(onding)g(v)m(ariables)g(in)g(the)f(calling)g(program.)38 b(It)21 b(is)h(not)g(required)164 2117 y(that)h(the)f(data)h(t)o(yp)q(es)f (sp)q(eci\014ed)f(in)h(a)h(bu\013er)g(descriptor)e(matc)o(h)g(the)h(data)h(t) o(yp)q(es)164 2177 y(of)h(the)f(corresp)q(onding)i(elemen)o(ts)20 b(in)j(the)h(host)g(program.)43 b(Ho)o(w)o(ev)o(er,)23 b(in)g(case)h(of)164 2238 y(mismatc)o(hes,)c(the)i(corresp)q(ondence)g(b)q(et)o(w)o(een)f(en)o (tries)g(in)h(the)g(host)g(program)g(and)164 2298 y(en)o(tries)10 b(in)g(a)h(message)f(created)h(with)f(the)h(bu\013er)g(descriptor)g(ma)o(y)e (b)q(e)i(implem)o(en)o(tati)o(on)164 2358 y(dep)q(enden)o(t.)34 b(No)20 b(data)h(con)o(v)o(ersion)f(o)q(ccurs)h(when)g(data)g(is)f(mo)o(v)o (ed)f(from)g(a)i(sender)164 2418 y(bu\013er)c(in)o(to)e(a)i(message.)949 2599 y(17)p eop %%Page: 18 18 bop 237 307 a Fl(Consider)17 b(the)f(follo)o(wing)g(fragmen)o(t)f(of)h(F)l (ortran)h(co)q(de)164 421 y Fg(REAL)24 b(A\(100\))164 482 y(INTEGER)f(BH)164 542 y(...)164 602 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f(MPI_BUFFE)o(R,)f (MPI_PERSIST)o(EN)o(T\))164 662 y(CALL)i(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(1\),)g (1,)h(MPI_REAL\))164 722 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(2\),)g(1,)h (MPI_INT\))164 782 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(3\),)g(1,)h (MPI_LOGICA)o(L\))164 843 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(4\),)g(1,)h (MPI_COMPLE)o(X\))164 903 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(A\(6\),)g(1,)h (MPI_DOUBLE)o(\))164 963 y(CALL)f(MPI_ADD_BL)o(OCK)o(\(BH)o(,)f(A\(8\),)g(4,) i(MPI_CHAR\))237 1077 y Fl(A)17 b(message)g(created)g(from)f(this)h(bu\013er) h(will)f(consist)g(of)h(a)g(sequence)e(con)o(taining)164 1137 y(one)k(real,)f(one)h(in)o(teger,)f(one)h(logical,)f(one)h(complex,)e(one)i (double,)g(and)g(four)g(c)o(har-)164 1198 y(acters.)31 b(No)19 b(data)i(con)o(v)o(ersion)d(o)q(ccurs)i(when)g(v)m(alues)f(are)h(copied)f (from)f(the)h(sender)164 1258 y(bu\013er)e(to)g(the)f(message.)22 b(Th)o(us,)16 b(the)h(\014rst)g(en)o(try)e(in)i(the)f(message)g(is)g(a)h (real)g(n)o(um)o(b)q(er)164 1318 y(equal)g(to)h Fg(A\(1\))p Fl(;)f(the)h(second)g(en)o(try)f(is)g(an)i(in)o(teger)d(n)o(um)o(b)q(er)g (that)j(happ)q(ens)g(to)f(ha)o(v)o(e)164 1378 y(the)13 b(same)f(binary)g (represen)o(tation)h(as)g Fg(A\(2\))p Fl(;)g(the)f(third)h(en)o(try)f(is)h(a) g(logical)g(v)m(alue)f(that)164 1438 y(happ)q(ens)k(to)f(ha)o(v)o(e)g(the)f (same)g(binary)h(represen)o(tation)g(as)g Fg(A\(3\))p Fl(;)f(the)h(fourth)g (en)o(try)f(in)164 1499 y(a)g(complex)e(n)o(um)o(b)q(er)g(with)h(a)i(binary)e (represen)o(tation)g(iden)o(tical)f(to)j Fg(A\(4\),)23 b(A\(5\))p Fl(;)13 b(the)164 1559 y(\014fth)k(en)o(try)f(is)h(a)g(double)g(precision)f (v)m(alue)h(with)f(a)i(binary)e(represen)o(tation)h(iden)o(tical)164 1619 y(to)j Fg(A\(6\),)j(A\(7\))p Fl(;)c(and)h(if)f(w)o(ords)h(ha)o(v)o(e)f (four)g(b)o(ytes)g(then)g(the)g(last)h(four)g(en)o(tries)e(are)164 1679 y(b)o(ytes)f(that)g(mak)o(e)f(the)h(binary)g(represen)o(tation)g(of)h Fg(A\(8\))p Fl(,)d(in)i(the)h(b)o(yte)e(order)h(of)h(the)164 1739 y(executing)d(mac)o(hine.)237 1800 y(The)f(corresp)q(ondence)g(b)q(et)o (w)o(een)g(the)g(\014rst)g(sev)o(en)f(en)o(tries)g(of)i(the)e(arra)o(y)i Fg(A)e Fl(and)i(the)164 1860 y(\014rst)g(\014v)o(e)g(en)o(tries)f(of)h(the)g (message)g(created)f(from)g(this)h(bu\013er)h(is)f(determined)d(b)o(y)j(the) 164 1920 y(rules)k(of)h(F)l(ortran)g(77)g(on)g(storage)h(asso)q(ciation:)29 b(Eac)o(h)19 b(v)m(ariable)g(of)h(t)o(yp)q(e)f Fg(INTEGER)p Fl(,)164 1980 y Fg(REAL)p Fl(,)11 b(or)j Fg(LOGICAL)c Fl(o)q(ccup)o(y)j(one)g Fh(numeric)i(stor)n(age)g(unit)p Fl(;)f(a)f(v)m(ariable)g(of)h(t)o(yp)q(e)e Fg(DOUBLE)164 2040 y Fl(or)21 b Fg(COMPLEX)c Fl(o)q(ccup)o(y)j(t)o(w)o(o)g(n) o(umeric)e(storage)j(units.)33 b(Th)o(us,)21 b(the)f(same)f(corresp)q(on-)164 2100 y(dence)c(will)f(hold)i(for)g(an)o(y)g(implem)o(e)o(n)o(tation.)i(Ho)o (w)o(ev)o(er,)c(di\013eren)o(t)h(implem)o(en)n(tations)164 2161 y(ma)o(y)20 b(ha)o(v)o(e)g(di\013eren)o(t)g(binary)i(enco)q(dings)f(of)h (in)o(teger,)f(real)g(and)h(logical)e(v)m(alues,)i(so)164 2221 y(that)17 b(the)f(actual)g(v)m(alues)g(transferred)g(b)o(y)g(the)g(message)g (ma)o(y)e(di\013er.)237 2281 y(The)21 b(corresp)q(ondence)h(b)q(et)o(w)o(een) e(the)h(en)o(try)f Fg(A\(8\))g Fl(of)h(the)g(arra)o(y)l(,)h(and)g(the)f(last) 164 2341 y(four)j(c)o(haracter)g(en)o(tries)e(in)i(the)f(message)g(is)h (implem)o(en)o(tation)d(dep)q(enden)o(t,)k(since)164 2401 y(the)19 b(F)l(ortran)i(language)f(do)q(es)h(not)f(sp)q(ecify)f(a)h(corresp)q(ondence) g(b)q(et)o(w)o(een)e(c)o(haracter)164 2462 y(storage)j(units)g(and)g(n)o (umeric)d(storage)j(units)f(\(an)h(arra)o(y)g(of)g(c)o(haracters)f(cannot)h (b)q(e)949 2599 y(18)p eop %%Page: 19 19 bop 164 307 a Fl(equiv)m(alenced)16 b(with)i(an)g(arra)o(y)g(of)g(in)o (tegers\).)26 b(Di\013eren)o(t)17 b(results)h(ma)o(y)e(o)q(ccur)i(in)g(big-) 164 367 y(endians)e(or)h(small-endians)e(mac)o(hines,)f(or)i(in)g(32)h(bit)f (or)h(64)g(bit)f(mac)o(hines.)237 428 y(The)i(same)e(holds,)i(symmetri)o (call)o(y)l(,)c(when)k(a)g(message)e(is)i(receiv)o(ed.)k(En)o(tries)17 b(are)164 488 y(mo)o(v)o(ed)i(from)g(the)i(message)f(in)o(to)h(the)f(receiv)o (er)f(memory)f(according)j(to)g(the)g(infor-)164 548 y(mation)h(pro)o(vided)g (b)o(y)g(the)g(bu\013er)i(descriptor,)f(with)g(no)g(regard)g(to)g(the)g(w)o (a)o(y)f(the)164 608 y(corresp)q(onding)17 b(v)m(ariables)f(are)h(declared)e (in)h(the)g(receiving)f(program.)237 668 y(When)c(data)h(is)f(mo)o(v)o(ed)e (in)h(a)i(homogeneous)f(en)o(vironmen)o(t)d(b)q(et)o(w)o(een)i(no)q(des)i(ha) o(ving)164 729 y(the)21 b(same)f(arc)o(hitecture,)g(then)h(no)g(data)h(con)o (v)o(ersion)e(o)q(ccur)h(at)h(an)o(y)f(p)q(oin)o(t)g(during)164 789 y(data)16 b(transfer.)21 b(Assume,)13 b(in)i(the)g(previous)g(example,)d (that)k(an)f(iden)o(tically)e(declared)164 849 y(bu\013er)j(descriptor)g(is)f (used)h(to)g(receiv)o(e)e(data)i(in)o(to)g(an)g(iden)o(tically)e(declared)h (arra)o(y)h(at)164 909 y(the)g(receiving)e(pro)q(cess.)21 b(Then,)16 b(when)g(these)f(t)o(w)o(o)h(no)q(des)g(comm)o(unicate,)c(the)k(v)m(alues)164 969 y(in)g(the)h(\014rst)g(eigh)o(t)f(en)o(tries)f(of)i(arra)o(y)g Fg(A)f Fl(of)h(the)g(sender)f(will)g(b)q(e)h(copied)f(in)o(to)g(the)h (\014rst)164 1029 y(eigh)o(t)e(en)o(tries)f(of)i(arra)o(y)g Fg(A)f Fl(at)h(the)f(receiv)o(er)e(\(assuming)i(that)h(reals)f(o)q(ccup)o(y)h (the)f(same)164 1090 y(storage)21 b(as)g(four)f(b)o(ytes\).)33 b(In)20 b(particular,)g(in)g(a)g(homogeneous)h(en)o(vironmen)o(t,)c(it)j(is) 164 1150 y(p)q(ossible)c(to)h(comm)o(unic)o(ate)d(using)i(only)g(c)o (haracter)g(t)o(yp)q(ed)g(messages.)237 1210 y(When)22 b(data)g(is)f(mo)o(v)o (ed)e(in)o(to)i(a)h(heterogeneous)g(en)o(vironmen)o(t)d(b)q(et)o(w)o(een)h (no)q(des)164 1270 y(ha)o(ving)14 b(distinct)f(arc)o(hitectures,)g(data)i (con)o(v)o(ersion)e(ma)o(y)g(o)q(ccur)h(during)g(the)g(transfer:)164 1330 y(Eac)o(h)21 b(en)o(try)e(is)i(con)o(v)o(erted)e(from)h(the)g(data)h (represen)o(tation)f(used)h(on)g(the)g(sending)164 1391 y(no)q(de)c(to)f(the) g(data)i(represen)o(tation)d(used)i(in)e(the)h(receiving)f(no)q(de.)237 1451 y(Consider)i(the)f(follo)o(wing)g(fragmen)o(t)f(of)h(F)l(ortran)h(co)q (de.)164 1565 y Fg(REAL)24 b(X,)h(Y)164 1625 y(CHARACTER)d(\(4\))j(Z)164 1685 y(INTEGER)e(BH)164 1746 y(...)164 1806 y(CALL)h(MPI_CREATE)o(\(BH)o(,)f (MPI_BUFFE)o(R,)f(MPI_PERSIST)o(EN)o(T\))164 1866 y(CALL)i(MPI_ADD_BL)o(OCK)e (\(BH,)i(X,)h(1,)g(MPI_REAL\))164 1926 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i (Y,)h(4,)g(MPI_CHAR\))164 1986 y(CALL)f(MPI_ADD_BL)o(OCK)e(\(BH,)i(Z,)h(4,)g (MPI_CHAR\))237 2100 y Fl(Assume)11 b(that)i(the)g(same)e(arra)o(ys)i Fg(X,)25 b(Y,)g(Z)12 b Fl(and)h(bu\013er)g(handle)g Fg(BH)f Fl(are)g(created)h(at)164 2161 y(t)o(w)o(o)g(pro)q(cesses)h(A)e(and)i(B;)e (this)h(handle)g(is)g(used)g(b)o(y)g(A)f(to)i(create)e(and)i(send)f(a)h (message)164 2221 y(for)e(B,)f(b)o(y)h(B)f(to)i(receiv)o(e)c(this)j(message.) 19 b(F)l(urther)12 b(assume)f(that)i(b)q(oth)g(A)e(and)i(B)e(run)h(on)164 2281 y(distinct)g(no)q(des,)j(with)e(p)q(ossibly)g(di\013eren)o(t)g(32)h(bit) f(arc)o(hitectures.)18 b(Then)c Fg(X)f Fl(at)g(pro)q(cess)164 2341 y(B)19 b(is)g(assigned)h(the)g(v)m(alue)f(of)g Fg(X)h Fl(at)f(pro)q(cess)h(A)f(\(up)h(to)g(rounding)g(errors)g(that)f(ma)o(y)164 2401 y(o)q(ccur)12 b(during)g(con)o(v)o(ersion\);)f Fg(Z)h Fl(on)g(pro)q(cess)g(B)f(is)h(assigned)g(the)f(c)o(haracter)g(string)h(v)m (alue)164 2462 y(of)18 b Fg(Z)f Fl(on)h(pro)q(cess)g(A;)f(if)f(b)q(oth)j(no)q (des)f(use)g(ASCI)q(I)f(enco)q(ding,)g(then)h(no)g(con)o(v)o(ersion)e(is)949 2599 y(19)p eop %%Page: 20 20 bop 164 307 a Fl(required)14 b(for)h(the)g(c)o(haracters.)21 b(On)15 b(the)g(other)h(hand,)f(v)m(ariable)g Fg(Y)g Fl(on)h(pro)q(cess)g(B)e (ma)o(y)164 367 y(b)q(e)19 b(allo)q(cated)g(a)g(v)m(alue)f(that)h(di\013ers)g (from)f(the)g(v)m(alue)h(of)g(v)m(ariable)f Fg(Y)g Fl(on)h(pro)q(cess)h(A.) 164 428 y(This)12 b(ma)o(y)f(o)q(ccur)h(if)g(the)g(t)o(w)o(o)g(no)q(des)h (use)f(a)h(di\013eren)o(t)e(b)o(yte)g(sequence)g(\(little-endian)g(vs)164 488 y(big-endian\),)i(or)g(use)f(a)h(di\013eren)o(t)e(binary)i(represen)o (tation)e(for)i(reals)f(\(IEEE)h(vs)f(HEX\).)164 548 y(Th)o(us,)18 b(in)f(order)h(to)g(ensure)g(correct)f(execution)g(in)g(a)h(heterogeneous)g (en)o(vironmen)o(t,)164 608 y(it)e(is)g(imp)q(ortan)o(t)g(that)g(the)h(t)o (yp)q(es)f(of)g(v)m(alues)h(in)f(a)h(message)e(matc)o(h)g(the)h(t)o(yp)q(es)g (of)h(the)164 668 y(corresp)q(onding)g(v)m(alues)f(in)g(the)g(sending)h(and)g (in)f(the)g(receiving)e(program.)164 813 y Fm(1.7)70 b(Receiv)n(e)20 b(Criteria)164 905 y Fl(The)15 b(selection)f(of)h(a)g(message)f(b)o(y)h(a)g (receiv)o(e)e(op)q(eration)i(is)g(done)g(uniquely)f(according)164 965 y(to)23 b(the)g(v)m(alue)g(of)g(the)g(message)f(en)o(v)o(elop)q(e.)40 b(The)22 b(receiv)o(e)f(op)q(eration)j(sp)q(eci\014es)e(an)164 1025 y Fi(en)n(v)n(elop)r(e)k(pattern)p Fl(;)g(a)d(message)f(can)h(b)q(e)h (receiv)o(ed)c(b)o(y)j(that)g(receiv)o(e)e(op)q(eration)164 1086 y(only)c(if)f(its)h(en)o(v)o(elop)q(e)f(matc)o(hes)f(that)j(pattern.)24 b(A)17 b(pattern)g(sp)q(eci\014es)g(v)m(alues)g(for)g(the)164 1146 y Fg(source)p Fl(,)h Fg(tag)h Fl(and)h Fg(context)d Fl(\014elds)j(of)g (the)g(message)f(en)o(v)o(elop)q(e.)30 b(In)20 b(addition,)g(the)164 1206 y(v)m(alue)15 b(for)h(the)f Fg(dest)e Fl(\014eld)i(is)g(set,)g (implicitl)o(y)l(,)d(to)k(b)q(e)f(equal)g(to)h(the)f(receiving)e(pro)q(cess) 164 1266 y(id.)38 b(The)22 b(receiv)o(er)e(ma)o(y)g(sp)q(ecify)i(a)g Fi(DONTCARE)g Fl(v)m(alue)g(for)g Fg(source)p Fl(,)f(or)h Fg(tag)p Fl(,)164 1326 y(indicating)e(that)i(an)o(y)e(source)h(and/or)h(tag)g(are)f (acceptable.)34 b(It)21 b(cannot)g(sp)q(ecify)g(a)164 1387 y(DONTCARE)d(v)m(alue)h(for)f Fg(context)e Fl(or)j Fg(dest)p Fl(.)26 b(Th)o(us,)19 b(a)g(message)f(can)h(b)q(e)f(receiv)o(ed)164 1447 y(b)o(y)k(a)g(receiv)o(e)e(op)q(eration)j(only)e(if)h(it)f(is)h (addressed)h(to)f(the)g(receiving)e(task,)k(has)f(a)164 1507 y(matc)o(hing)18 b(con)o(text,)h(has)h(matc)o(hing)e(source)h(unless)h (source=DONTCARE)f(in)g(the)164 1567 y(pattern,)d(and)h(has)g(a)f(matc)o (hing)f(tag)i(unless)f(tag=DONTCARE)h(in)f(the)g(pattern.)237 1627 y(The)k(length)f(of)h(the)g(receiv)o(ed)d(message)i(m)o(ust)f(b)q(e)i (less)g(or)g(equal)f(the)g(length)h(of)164 1688 y(the)d(receiv)o(e)f (bu\013er.)26 b(I.e.,)16 b(all)h(incoming)f(data)i(m)o(ust)f(\014t,)g (without)h(truncation,)g(in)o(to)164 1748 y(the)f(receiv)o(e)f(bu\013er.)25 b(It)17 b(is)h(erroneous)g(to)g(receiv)o(e)d(a)j(message)f(whic)o(h)g(length) g(exceed)164 1808 y(the)e(receiv)o(e)d(bu\013er,)j(and)g(the)g(outcome)e(of)j (program)e(where)h(this)f(o)q(ccurs)h(is)g(undeter-)164 1868 y(mined.)164 2013 y Fm(1.8)70 b(Comm)n(unicati)o(on)21 b(Mo)r(de)164 2105 y Fl(A)16 b(sending)g(op)q(eration)h(can)g(o)q(ccur)f(in)g(one)g(of)h(t) o(w)o(o)f(mo)q(des:)164 2207 y Fi(REGULAR)24 b Fl(The)e(send)g(ma)o(y)f (start)i(whether)f(or)g(not)h(a)g(matc)o(hing)d(receiv)o(e)g(has)286 2267 y(b)q(een)c(p)q(osted.)164 2369 y Fi(READ)n(Y)25 b Fl(The)16 b(send)h(ma)o(y)d(start)j(only)f(if)g(a)g(matc)o(hing)f(receiv)o(e)f(has)j(b) q(een)f(p)q(osted.)949 2599 y(20)p eop %%Page: 21 21 bop 237 307 a Fl(A)21 b Fi(ready)k(send)c Fl(can)h(start)g(only)g(if)f(a)h (matc)o(hing)e(receiv)o(e)g(is)h(already)h(p)q(osted;)164 367 y(otherwise)17 b(the)g(op)q(eration)i(is)e(erroneous)h(and)g(its)f(outcome)g (is)g(unde\014ned.)25 b(In)17 b(some)164 428 y(systems,)12 b(this)g(will)g(allo)o(w)h(to)g(optimize)d(comm)o(unic)o(ation)h(and)i(a)o(v) o(oid)f(a)h(hand-shaking)164 488 y(op)q(eration)k(that)g(is)f(otherwise)g (required.)164 668 y Fk(Discussion:)28 b Fj(I)19 b(deleted)i(the)e(symmetric) g(ready)g(receiv)o(e.)32 b(Will)21 b(reviv)o(e)f(it)f(if)g(there)h(is)f(a)164 729 y(requiremen)o(t)d(for)e(it.)164 993 y Fm(1.9)70 b(Comm)n(unicati)o(on)21 b(Ob)t(jects)164 1086 y Fl(An)13 b(opaque)i(comm)o(uni)o(cation)c(ob)s(ject)i (iden)o(ti\014es)g(v)m(arious)h(prop)q(erties)g(of)g(a)g(comm)o(uni-)164 1146 y(cation)j(op)q(eration,)g(suc)o(h)g(as)g(the)g(bu\013er)g(descriptor)g (that)g(is)g(asso)q(ciated)g(with)g(it,)f(its)164 1206 y(con)o(text,)e(the)g (tag)h(and)g(destination)f(parameters)g(to)h(b)q(e)f(used)h(for)g(a)g(send,)f (or)h(the)f(tag)164 1266 y(and)f(source)g(parameters)f(to)h(b)q(e)g(used)g (for)h(a)f(receiv)o(e.)18 b(In)12 b(addition,)h(this)g(ob)s(ject)f(stores)164 1326 y(information)17 b(ab)q(out)h(the)g(status)g(of)g(the)g(last)f(comm)o (unication)e(op)q(eration)j(that)g(w)o(as)164 1387 y(p)q(erformed)f(with)g (this)h(ob)s(ject.)26 b(This)18 b(ob)s(ject)f(is)h(accessed)f(using)i(a)f (comm)o(unicati)o(on)164 1447 y(handle.)237 1507 y(One)i(can)h(consider)f (comm)o(uni)o(cation)e(op)q(erations)j(to)g(consist)g(of)f(the)g(follo)o (wing)164 1567 y(sub)q(op)q(erations:)164 1681 y Fi(INIT\(op)r(eration,)d (params,)i(handle\))25 b Fl(Pro)q(cess)15 b(pro)o(vides)f(all)f(relev)m(an)o (t)g(parame-)286 1742 y(ters)18 b(for)g(its)g(participation)g(in)g(the)f (comm)o(unication)e(op)q(eration)k(\(t)o(yp)q(e)f(of)g(op-)286 1802 y(eration,)e(data)i(bu\013er,)f(tag,)g(participan)o(ts,)f(etc.\).)21 b(An)c(ob)s(ject)f(is)g(created)g(that)286 1862 y(iden)o(ti\014es)f(the)h(op) q(eration.)164 1964 y Fi(ST)-5 b(AR)g(T\(handle\))26 b Fl(The)16 b(comm)o(unic)o(ation)e(op)q(eration)j(is)f(started)164 2065 y Fi(COMPLETE\(handle\))25 b Fl(The)16 b(comm)o(unication)d(op)q(eration)k (is)f(completed.)164 2167 y Fi(FREE\(handle\))24 b Fl(The)e(comm)o(unication) d(ob)s(ject,)j(and)h(asso)q(ciated)g(resources)f(are)286 2227 y(freed.)164 2341 y(Correct)16 b(in)o(v)o(o)q(cation)g(of)g(these)g(sub)q(op) q(erations)i(is)f(a)f(sequence)f(of)i(the)f(form)520 2451 y Fi(INIT)i Fl(\()p Fi(ST)-5 b(AR)g(T)20 b(COMPLETE)p Fl(\))1224 2431 y Fb(\003)1263 2451 y Fi(FREE)p Fd(:)949 2599 y Fl(21)p eop %%Page: 22 22 bop 164 307 a Fl(I.e.,)10 b(an)i(ob)s(ject)f(needs)h(b)q(e)f(created)h(b)q (efore)f(comm)o(unicati)o(on)f(o)q(ccurs;)j(it)e(can)g(b)q(e)h(reused)164 367 y(only)h(after)g(the)g(previous)g(use)g(has)h(completed;)d(and)j(it)f (needs)g(to)g(b)q(e)h(freed)e(ev)o(en)o(tually)164 428 y(\(of)i(course,)g (one)h(can)f(assume)f(that)i(all)e(ob)s(jects)h(are)g(freed)g(at)g(program)g (termination,)164 488 y(b)o(y)i(default\).)237 548 y(The)21 b(ab)q(o)o(v)o(e)f(scenario)g(p)q(ertains)h(to)g Fh(p)n(ersistent)g Fl(ob)s(jects.)33 b(One)20 b(can)h(also)g(create)164 608 y Fh(ephemer)n(al)15 b Fl(ob)s(jects.)20 b(Suc)o(h)14 b(ob)s(ject)f(p)q (ersists)i(only)f(un)o(til)f(the)h(comm)o(uni)o(cation)e(op)q(era-)164 668 y(tion)j(is)g(completed,)d(at)k(whic)o(h)e(p)q(oin)o(t)h(it)g(is)g (destro)o(y)o(ed.)20 b(Th)o(us,)15 b(correct)f(in)o(v)o(o)q(cation)h(of)164 729 y(sub)q(op)q(erations)j(with)e(an)h(ephemeral)d(ob)s(ject)i(is)g Fi(INIT)i(ST)-5 b(AR)g(T)19 b(COMPLETE)p Fl(.)237 789 y(A)i(user)g(ma)o(y)e (directly)h(in)o(v)o(ok)o(es)f(these)i(sub)q(op)q(erations.)37 b(This)22 b(w)o(ould)f(allo)o(w)g(to)164 849 y(amortize)16 b(the)i(o)o(v)o(erhead)f(of)h(setting)g(up)g(a)g(comm)o(unication)d(o)o(v)o (er)i(man)o(y)f(successiv)o(e)164 909 y(uses)23 b(of)h(the)e(same)g(handle,)i (and)g(allo)o(ws)f(to)g(o)o(v)o(erlap)g(comm)o(uni)o(cation)d(and)k(com-)164 969 y(putation.)29 b(Simpler)16 b(comm)o(unication)g(op)q(erations)k(com)o (bine)c(sev)o(eral)i(of)h(these)f(sub-)164 1029 y(op)q(erations)24 b(in)o(to)e(one)g(op)q(eration,)j(th)o(us)d(simplifying)e(the)i(use)g(of)h (comm)o(unicati)o(on)164 1090 y(primitiv)o(e)o(s.)f(Th)o(us,)c(one)f(only)h (needs)f(to)h(sp)q(ecify)e(precisely)g(the)h(seman)o(tics)f(of)h(these)164 1150 y(sub)q(op)q(erations)i(in)e(order)g(to)h(sp)q(ecify)e(the)h(seman)o (tics)f(of)h(MPI)g(p)q(oin)o(t)g(to)h(p)q(oin)o(t)f(com-)164 1210 y(m)o(unication)d(op)q(erations.)237 1270 y(W)l(e)h(sa)o(y)f(that)h(a)g (comm)o(unication)d(op)q(eration)j(\(send)g(or)g(receiv)o(e\))e(is)h Fi(p)r(osted)g Fl(once)164 1330 y(a)23 b Fi(start)g Fl(sub)q(op)q(eration)i (w)o(as)f(in)o(v)o(ok)o(ed;)g(the)f(op)q(eration)h(is)f Fi(completed)f Fl(once)h(the)164 1391 y Fi(complete)17 b Fl(sub)q(op)q(eration)j(completes.) k(A)17 b(send)h(and)h(a)f(receiv)o(e)e(op)q(eration)j Fi(matc)n(h)164 1451 y Fl(if)d(the)h(receiv)o(e)d(pattern)j(sp)q(eci\014ed)g(b)o(y)f(the)h (receiv)o(e)d(matc)o(hes)i(the)g(message)g(en)o(v)o(elop)q(e)164 1511 y(created)g(b)o(y)f(the)h(send.)164 1640 y Fi(1.9.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)164 1733 y Fl(An)g(ob)s(ject)g (for)g(a)h(send)f(op)q(eration)h(is)f(created)g(b)o(y)g(a)g(call)g(to)g Fi(MPI)p 1452 1733 17 2 v 21 w(INIT)p 1598 1733 V 20 w(SEND)p Fl(.)164 1793 y(A)g(call)g(to)g Fi(MPI)p 487 1793 V 21 w(INIT)p 633 1793 V 20 w(RECV)h Fl(is)f(similarly)d(used)k(for)g(creating)f(an)h(ob)s (ject)f(for)h(a)164 1853 y(receiv)o(e)8 b(op)q(eration.)21 b(The)11 b(creation)f(of)h(a)g(comm)o(unication)d(ob)s(ject)i(is)h(a)g(lo)q (cal)g(op)q(eration)164 1913 y(that)17 b(need)e(not)i(in)o(v)o(olv)o(e)d (comm)o(unicati)o(on)g(with)i(a)h(remote)e(pro)q(cess.)164 2034 y Fi(MPI)p 279 2034 V 20 w(INIT)p 424 2034 V 20 w(SEND)g(\(handle,)g (bu\013er)p 972 2034 V 20 w(handle,)h(dest,)f(tag,)g(con)n(text,)f(mo)r(de,) 164 2094 y(p)r(ersistence\))237 2214 y Fl(Creates)j(a)f(send)h(comm)o(uni)o (cation)d(ob)s(ject.)20 b(P)o(arameters)15 b(are)164 2313 y Fi(OUT)k(handle)25 b Fl(message)19 b(handle.)31 b(The)20 b(handle)f(should)i (not)f(b)q(e)f(asso)q(ciated)i(with)286 2373 y(an)o(y)16 b(ob)s(ject)g(b)q (efore)g(the)g(call.)164 2474 y Fi(IN)i(bu\013er)p 394 2474 V 20 w(handle)25 b Fl(handle)16 b(to)h(send)f(bu\013er)h(descriptor)949 2599 y(22)p eop %%Page: 23 23 bop 164 307 a Fi(IN)18 b(dest)24 b Fl(rank)17 b(in)f(con)o(text)f(of)i (destination)f(\(in)o(teger\))164 409 y Fi(IN)i(tag)25 b Fl(user)16 b(tag)h(for)g(messages)e(sen)o(t)h(with)g(this)g(handle)h(\(in)o(teger\))164 511 y Fi(IN)h(con)n(text)24 b Fl(handle)16 b(to)h(con)o(text)e(of)i(messages) e(sen)o(t)h(with)g(this)h(handle)164 612 y Fi(IN)h(mo)r(de)24 b Fl(send)11 b(mo)q(de)f(\(state)h(t)o(yp)q(e,)g(with)f(t)o(w)o(o)h(v)m (alues:)19 b Fg(MPI)p 1324 612 16 2 v 17 w(REGULAR)8 b Fl(and)k Fg(MPI)p 1699 612 V 17 w(READY)p Fl(\))164 714 y Fi(IN)18 b(p)r(ersistence)24 b Fl(handle)10 b(p)q(ersistence)g(\(state)h(t)o(yp)q(e,)g(with)g(t)o(w)o(o)g (v)m(alues:)18 b Fg(MPI)p 1623 714 V 18 w(PERSISTEN)o(T)286 774 y Fl(and)f Fg(MPI)p 462 774 V 17 w(EPHEMERAL)p Fl(\))164 936 y Fi(MPI)p 279 936 17 2 v 20 w(INIT)p 424 936 V 20 w(RECV)c(\(handle,)h (bu\013er)p 975 936 V 19 w(handle,)g(source,)f(tag,)h(con)n(text,)e(p)r(er-) 164 996 y(sistence\))237 1117 y Fl(Create)k(a)h(receiv)o(e)d(handle.)21 b(P)o(arameters)15 b(are)164 1218 y Fi(OUT)k(handle)25 b Fl(message)19 b(handle.)31 b(The)20 b(handle)f(should)i(not)f(b)q(e)f(asso)q(ciated)i(with) 286 1279 y(an)o(y)16 b(ob)s(ject)g(b)q(efore)g(the)g(call.)164 1380 y Fi(IN)i(bu\013er)p 394 1380 V 20 w(handle)25 b Fl(handle)16 b(to)h(receiv)o(e)d(bu\013er)i(descriptor.)164 1482 y Fi(IN)i(source)24 b Fl(rank)17 b(in)f(con)o(text)f(of)i(source,)f(or)g(DONTCARE)g(\(in)o (teger\).)164 1584 y Fi(IN)i(tag)25 b Fl(user)d(tag)h(for)f(messages)g (receiv)o(ed)e(with)i(this)g(handle,)h(or)f(DONTCARE)286 1644 y(\(in)o(teger\).)164 1746 y Fi(IN)c(con)n(text)24 b Fl(handle)16 b(to)h(con)o(text)e(of)i(messages)e(receiv)o(ed)f(with)j(this)f(handle.)164 1847 y Fi(IN)i(p)r(ersistence)24 b Fl(handle)10 b(p)q(ersistence)g(\(state)h (t)o(yp)q(e,)g(with)g(t)o(w)o(o)g(v)m(alues:)18 b Fg(MPI)p 1623 1847 16 2 v 18 w(PERSISTEN)o(T)286 1907 y Fl(and)f Fg(MPI)p 462 1907 V 17 w(EPHEMERAL)p Fl(\))237 2009 y(See)f(Section)g(1.5.2)g(for)h(a) f(discussion)h(of)f(source,)g(tag)h(and)g(con)o(text.)164 2178 y Fk(Discussion:)48 b Fj(I)18 b(ha)o(v)o(e)f(not)f(included)k(prop)q(osals)e (for)e(partially)i(sp)q(eci\014ed)i(message)d(han-)164 2235 y(dles,)f(that)e(some)h(p)q(eoples)i(seem)e(to)g(desire.)237 2291 y(I)h(ha)o(v)o(e)e(merged)i(all)g(handle)g(setup)g(in)o(to)f(one)g (call.)949 2599 y Fl(23)p eop %%Page: 24 24 bop 164 307 a Fi(1.9.2)55 b(Comm)n(unication)21 b(Start)164 460 y(MPI)p 279 460 17 2 v 20 w(ST)-5 b(AR)g(T\(handle\))164 561 y(IN)18 b(handle)26 b Fl(comm)o(uni)o(cation)14 b(handle)237 663 y(The)j Fg(MPI)p 419 663 16 2 v 17 w(START)f Fl(function)g(starts)i(the)f (execution)e(of)j(a)f(comm)o(unic)o(ation)e(op)q(era-)164 723 y(tion)k(\(send)h(or)f(receiv)o(e\).)28 b(A)19 b(sender)g(should)h(not)g(up)q (date)g(the)f(send)h(bu\013er)f(after)h(a)164 784 y(send)e(op)q(eration)g (has)h(started)f(and)g(un)o(til)f(it)g(is)g(completed.)24 b(A)17 b(receiv)o(er)e(should)j(not)164 844 y(access)f(the)f(receiv)o(e)f(bu\013er)i (after)f(a)h(receiv)o(e)e(op)q(eration)i(w)o(as)h(started)f(and)g(un)o(til)f (it)g(is)164 904 y(completed.)21 b(A)16 b(program)h(that)g(do)q(es)h(not)f (satisfy)g(this)g(condition)f(is)h(erroneous)g(and)164 964 y(its)f(outcome)f(is)h(undetermined.)164 1094 y Fi(1.9.3)55 b(Comm)n(unication)21 b(Completion)164 1247 y(MPI)p 279 1247 17 2 v 20 w(W)-6 b(AIT)19 b(\()f(handle,)h(return)p 864 1247 V 20 w(status)p 1030 1247 V 20 w(handle\))164 1348 y(IN)f(handle)26 b Fl(comm)o(uni)o(cation)14 b(handle)164 1450 y Fi(OUT)19 b(return)p 466 1450 V 19 w(handle)26 b Fl(handle)16 b(that)g(is)g(asso)q(ciated)i(with)e (return)g(status)h(ob)s(ject.)237 1552 y(A)23 b(call)f(to)h Fi(MPI)p 574 1552 V 21 w(W)-6 b(AIT)23 b Fl(returns)g(when)g(the)g(send)g(op) q(eration)h(iden)o(ti\014ed)d(b)o(y)164 1612 y Fg(handle)16 b Fl(is)i(complete.)24 b(The)18 b(completion)e(of)i(a)g(send)h(op)q(eration)f (indicates)g(that)g(the)164 1672 y(sender)h(is)g(no)o(w)h(free)e(to)i(up)q (date)f(the)g(lo)q(cations)h(in)f(the)g(send)g(bu\013er,)h(or)g(an)o(y)f (other)164 1732 y(lo)q(cation)e(that)f(can)h(b)q(e)f(referenced)f(b)o(y)h (the)g(send)g(op)q(eration.)23 b(Ho)o(w)o(ev)o(er,)14 b(it)h(do)q(es)i(not) 164 1792 y(indicate)11 b(that)i(the)f(message)g(has)h(b)q(een)f(receiv)o(ed;) f(rather)h(it)g(ma)o(y)f(ha)o(v)o(e)h(b)q(een)g(bu\013ered)164 1853 y(b)o(y)k(the)g(comm)o(uni)o(cation)e(subsystem.)237 1913 y(The)k(completion)e(of)h(a)i(receiv)o(e)c(op)q(eration)j(indicates)f(that)h (the)g(receiv)o(er)d(is)j(no)o(w)164 1973 y(free)e(to)i(access)f(the)g(lo)q (cations)h(in)f(the)g(receiv)o(e)e(bu\013er,)i(whic)o(h)g(con)o(tain)g(the)g (receiv)o(ed)164 2033 y(message,)f(or)i(an)o(y)g(other)f(lo)q(cation)h(that)g (can)f(b)q(e)h(referenced)e(b)o(y)h(the)g(receiv)o(e)e(op)q(era-)164 2093 y(tion.)21 b(It)16 b(do)q(es)h(not)g(indicate)e(that)i(the)f(matc)o (hing)e(send)j(op)q(eration)g(has)g(completed.)237 2154 y(The)g(call)e (returns)i(a)g(handle)f(to)h(an)g(opaque)g(ob)s(ject)f(that)h(con)o(tains)g (information)164 2214 y(on)g(the)f(completed)e(op)q(eration)j({)g(the)f Fi(return)i(status)e Fl(ob)s(ject.)164 2334 y Fi(MPI)p 279 2334 V 20 w(ST)-5 b(A)g(TUS)20 b(\(handle,)f(\015ag,)g(return)p 1029 2334 V 19 w(handle\))164 2436 y(IN)f(handle)26 b Fl(comm)o(uni)o(cation) 14 b(handle)949 2599 y(24)p eop %%Page: 25 25 bop 164 307 a Fi(OUT)19 b(\015ag)25 b Fl(logical)164 409 y Fi(OUT)19 b(return)p 466 409 17 2 v 19 w(handle)26 b Fl(handle)16 b(that)g(is)g(asso)q(ciated)i(with)e(return)g(status)h(ob)s(ject.)237 511 y(A)h(call)g(to)h Fi(MPI)p 561 511 V 20 w(ST)-5 b(A)g(TUS)20 b Fl(returns)f Fg(flag=true)c Fl(if)j(the)g(op)q(eration)h(iden)o(ti\014ed) 164 571 y(b)o(y)d Fg(handle)f Fl(is)i(complete,)e(In)h(suc)o(h)h(case,)g(the) g(return)f(handle)h(p)q(oin)o(ts)h(to)f(an)h(opaque)164 631 y(ob)s(ject)h(that)h(con)o(tains)g(information)e(on)j(the)e(completed)e (information.)31 b(It)19 b(returns)164 691 y Fg(flag=false)o Fl(,)12 b(otherwise.)21 b(In)14 b(suc)o(h)h(case,)f(the)h(v)m(alue)g(of)g (the)g(return)f(handle)h(is)g(unde-)164 751 y(\014ned.)237 812 y(Implem)o(en)o(tation)e(notes:)237 872 y(A)g(call)f(to)h Fg(MPI)p 510 872 16 2 v 17 w(WAIT)f Fl(blo)q(c)o(ks)g(only)h(the)f(executing) g(thread.)20 b(If)13 b(the)f(executing)g(pro-)164 932 y(cess)h(is)f(m)o (ultithreaded,)f(then)i(other)g(threads)g(within)g(the)f(pro)q(cess)i(can)f (b)q(e)g(sc)o(heduled)164 992 y(for)j(execution.)237 1052 y(The)h(use)h(of)f (a)h(blo)q(c)o(king)f(receiv)o(e)e(op)q(eration)j(\()p Fg(MPI)p 1197 1052 V 17 w(WAIT)p Fl(\))e(allo)o(ws)h(the)g(op)q(erating)164 1112 y(system)k(to)h(desc)o(hedule)f(the)h(blo)q(c)o(k)o(ed)f(thread)i(and)f (sc)o(hedule)f(another)i(thread)g(for)164 1173 y(execution,)c(if)h(suc)o(h)g (is)g(a)o(v)m(ailable.)32 b(The)20 b(use)g(of)g(a)h(non)o(blo)q(c)o(king)e (receiv)o(e)f(op)q(eration)164 1233 y(\()p Fg(MPI)p 264 1233 V 17 w(STATUS)p Fl(\))11 b(allo)o(ws)i(the)f(user)h(to)g(sc)o(hedule)f (alternativ)o(e)g(activities)f(within)h(a)h(single)164 1293 y(thread)j(of)h(execution.)237 1353 y(The)h(in)o(tended)e(implem)o(en)o (tation)f(of)j Fg(MPI)p 1027 1353 V 17 w(STATUS)d Fl(is)j(for)g(that)g(op)q (eration)g(to)g(re-)164 1413 y(turn)e(as)g(so)q(on)h(as)g(p)q(ossible.)k(Ho)o (w)o(ev)o(er,)14 b(if)h(rep)q(eatedly)g(called)g(for)h(an)g(op)q(eration)g (that)164 1474 y(is)g(enabled,)f(it)h(m)o(ust)f(ev)o(en)o(tually)f(succeed.) 237 1534 y(The)g(return)f(status)h(ob)s(ject)f(for)g(a)h(send)g(op)q(eration) g(carries)f(no)h(information.)19 b(The)164 1594 y(return)13 b(status)h(ob)s(ject)f(for)h(a)f(receiv)o(e)e(op)q(eration)j(carries)f (information)f(on)i(the)f(source,)164 1654 y(tag)h(and)h(length)e(of)h(the)g (receiv)o(ed)d(message.)20 b(These)14 b(\014elds)f(are)h(required)e(b)q (ecause)i(the)164 1714 y(receiv)o(e)h(op)q(eration)j(ma)o(y)e(ha)o(v)o(e)h (sp)q(eci\014ed)g Fg(DONTCARE)e Fl(in)i(either)f(source)i(or)g(tag)g (\014eld,)164 1775 y(and)f(the)f(message)f(ma)o(y)g(ha)o(v)o(e)g(b)q(een)i (shorter)f(than)h(the)f(receiv)o(e)e(bu\013er.)164 1895 y Fi(MPI)p 279 1895 17 2 v 20 w(RETURN)p 546 1895 V 20 w(ST)-5 b(A)g(T\()19 b(handle,)g(len,)g(source,)f(tag\))164 1997 y(IN)g(handle)26 b Fl(handle)16 b(to)g(return)g(status)h(ob)s(ject)164 2098 y Fi(OUT)i(len)24 b Fl(di\013erence)15 b(b)q(et)o(w)o(een)h(length)g(of)g (receiv)o(e)e(bu\013er)i(and)h(length)f(of)g(receiv)o(ed)286 2159 y(message,)24 b(in)f(b)o(ytes.)42 b(Th)o(us,)25 b(the)e(v)m(alue)g (returned)g(is)g(zero)g(if)g(the)g(receiv)o(ed)286 2219 y(message)11 b(matc)o(hes)g(the)g(the)h(receiv)o(e)e(bu\013er,)j(p)q(ositiv)o(e)e(if)h(it) f(is)h(shorter)g(\(in)o(teger\).)164 2320 y Fi(OUT)19 b(source)24 b Fl(rank)16 b(of)h(message)e(sender)h(in)g(message)g(con)o(text)f(\(in)o (teger\).)164 2422 y Fi(OUT)k(tag)24 b Fl(tag)17 b(of)g(receiv)o(ed)d (message)h(\(in)o(teger\).)949 2599 y(25)p eop %%Page: 26 26 bop 164 420 a Fk(Discussion:)237 477 y Fj(I)18 b(put)g(the)g(di\013erence)h (b)q(et)o(w)o(een)f(message)f(bu\013er)h(and)g(message)f(length)h(as)g(the)g (v)m(alue)164 533 y(returned,)e(rather)f(than)g(length)h(of)g(receiv)o(ed)g (message,)f(so)h(that)e(it)i(migh)o(t)g(b)q(e)g(easy)f(to)g(test)164 589 y(for)g(exact)f(matc)o(h.)237 650 y(The)f(use)g(of)e(a)i(return)f(status) f(ob)s(ject,)h(rather)g(than)h(a)f(list)h(of)f(parameters)f(ma)o(y)h (simplify)164 710 y(the)k(use)g(of)g(MPI)f(routines,)i(if)f(the)g(v)m(alues)h (stored)e(in)i(the)f(ob)s(ject)f(are)h(seldom)g(c)o(hec)o(k)o(ed.)23 b(A)164 770 y(prede\014ned)17 b(return)e(status)f(ob)s(ject)h(should)h(b)q(e) g(pro)o(vided,)g(to)e(ease)h(programming.)164 1020 y Fi(1.9.4)55 b(Multiple)20 b(Completions)164 1113 y Fl(It)f(is)g(con)o(v)o(enien)o(t)e(to) i(b)q(e)g(able)g(to)h(w)o(ait)e(for)i(the)f(completion)e(of)i(an)o(y)g(or)h (all)e(the)h(op-)164 1173 y(erations)f(in)g(a)g(set,)g(rather)g(than)g(ha)o (ving)g(to)g(w)o(ait)g(for)g(sp)q(eci\014c)g(message.)25 b(A)17 b(call)h(to)164 1233 y Fg(MPI)p 245 1233 16 2 v 17 w(WAITANY)g Fl(or)j Fg(MPI)p 604 1233 V 17 w(STATUSANY)c Fl(can)k(b)q(e)f(used)h(to)f(w)o (ait)h(for)f(the)g(completion)f(of)164 1293 y(one)e(out)g(of)f(sev)o(eral)g (op)q(erations;)h(a)g(call)f(to)g Fg(MPI)p 1079 1293 V 18 w(WAITALL)e Fl(can)i(b)q(e)h(used)g(to)f(w)o(ait)h(for)164 1353 y(all)f(p)q(ending)g(op)q (erations)i(in)e(a)g(list.)164 1474 y Fi(MPI)p 279 1474 17 2 v 20 w(W)-6 b(AIT)h(ANY)20 b(\()e(list)p 710 1474 V 21 w(of)p 776 1474 V 21 w(handles,)h(index,)f(return)p 1338 1474 V 20 w(handle\))237 1594 y Fl(Blo)q(c)o(ks)k(un)o(til)g(one)g(of)h(the)g(op)q (erations)h(asso)q(ciated)f(with)g(the)f(comm)o(unicati)o(on)164 1654 y(handles)16 b(in)f(the)g(arra)o(y)h(has)g(completed.)j(Returns)d(the)f (index)g(of)g(that)h(handle)g(in)f(the)164 1715 y(arra)o(y)l(,)h(and)g (returns)g(the)g(status)g(of)h(that)f(op)q(eration)h(in)e(the)h(ob)s(ject)f (asso)q(ciated)i(with)164 1775 y(the)f(return)p 384 1775 15 2 v 17 w(handle.)21 b(The)c(parameters)e(are:)164 1889 y Fi(IN)j(list)p 324 1889 17 2 v 22 w(of)p 391 1889 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to)f(comm)o(unication)d(ob)s(jects.)164 1991 y Fi(OUT)19 b(index)24 b Fl(index)16 b(of)g(handle)g(for)h(op)q(eration)g (that)g(completed)d(\(in)o(teger\).)164 2092 y Fi(OUT)19 b(return)p 466 2092 V 19 w(handle)26 b Fl(handle)19 b(that)g(is)g(asso)q(ciated)i(with)e (return)g(status)h(ob)s(ject.)286 2153 y(Set)c(to)h(return)f(status)h(of)f (op)q(eration)h(that)g(completed.)237 2267 y(The)11 b(successful)f(execution) g(of)h Fg(MPI)p 892 2267 16 2 v 17 w(WAITANY\(lis)o(t)p 1220 2267 V 15 w(of)p 1287 2267 V 18 w(handles,)23 b(index,)g(return)p 1871 2267 V 17 w(handle\))164 2327 y Fl(is)11 b(equiv)m(alen)o(t)e(to)i(the)g (successful)f(execution)g(of)h Fg(MPI)p 1127 2327 V 17 w(WAIT\(handl)o(e[i)o (],)22 b(return)p 1710 2327 V 16 w(handle\))p Fl(,)164 2387 y(where)15 b Fg(i)g Fl(is)h(the)f(v)m(alue)g(returned)g(b)o(y)g Fg(index)f Fl(and)i Fg(handle[i])d Fl(is)i(the)g Fg(i)p Fl(-th)h(handle)f(in) 164 2447 y(the)h(list.)949 2599 y(26)p eop %%Page: 27 27 bop 237 307 a Fl(If)15 b(more)f(then)h(one)g(op)q(eration)i(is)e(enabled)g (and)g(can)h(terminate,)d(one)i(is)g(arbitrar-)164 367 y(ily)20 b(c)o(hosen)i(\(sub)s(ject)f(to)g(the)h(restrictions)f(on)h(op)q(eration)g (termination)e(order,)i(see)164 428 y(Section)16 b(1.13\).)164 548 y Fg(MPI)p 245 548 16 2 v 17 w(WAITANY)23 b(\()j(list)p 623 548 V 17 w(of)p 692 548 V 17 w(handles,)d(index,)g(return)p 1275 548 V 17 w(status)p 1448 548 V 16 w(handle\))164 608 y Fl(is)241 690 y Fg({MPI_WAIT_)o(RE)o(CV)f(\(handle[1],)g(return_han)o(dle)o (\);)g(index)i(=)h(0})g(||)g(...)164 750 y(||)241 810 y({MPI_WAIT_)o(RE)o(CV) d(\(handle[n],)g(return_han)o(dle)o(\);)g(index)i(=)h(n-1})237 892 y Fl(\(\\)p Fe(jj)p Fl(")18 b(indicates)f(c)o(hoice;)f(one)i(of)f(the)g (alternativ)o(es)g(is)g(c)o(hosen,)g(nondeterministi-)164 953 y(cally)l(.\))164 1073 y Fi(MPI)p 279 1073 17 2 v 20 w(ST)-5 b(A)g(TUSANY)21 b(\()d(list)p 777 1073 V 22 w(of)p 844 1073 V 20 w(handles,)h(index,)f(return)p 1405 1073 V 20 w(handle\))237 1193 y Fl(Causes)c(either)e(one)h(or)h(none)f(of)g(the)g(op)q(erations)h (asso)q(ciated)g(with)f(the)g(comm)o(uni-)164 1254 y(cation)g(handles)f(to)h (return.)20 b(In)12 b(the)g(former)g(case,)g(it)g(has)i(the)e(same)f(return)i (seman)o(tics)164 1314 y(as)j(a)g(call)f(to)h Fg(MPI)p 492 1314 16 2 v 17 w(WAIT)p 613 1314 V 17 w(ANY)p Fl(.)e(In)i(the)f(later)g (case,)g(it)g(returns)h(a)g(v)m(alue)f(of)h(-1)g(in)g Fg(index)164 1374 y Fl(and)h Fg(return)p 418 1374 V 16 w(handle)d Fl(is)i(unde\014ned.)21 b(The)c(parameters)e(are:)164 1462 y Fi(IN)j(list)p 324 1462 17 2 v 22 w(of)p 391 1462 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to) f(comm)o(unication)d(ob)s(jects.)164 1557 y Fi(OUT)19 b(index)24 b Fl(index)e(of)h(handle)g(for)g(op)q(eration)h(that)g(completed,)d(or)j(-1)f (if)g(none)286 1617 y(completed)14 b(\(in)o(teger\).)164 1712 y Fi(OUT)19 b(return)p 466 1712 V 19 w(handle)26 b Fl(handle)19 b(that)g(is)g(asso)q(ciated)i(with)e(return)g(status)h(ob)s(ject.)286 1772 y(Set)11 b(to)h(return)f(status)i(of)f(op)q(eration)g(that)g(completed,) e(if)h(an)o(y;)h(unde\014ned)g(when)286 1833 y Fg(index)24 b(=)h(-1)p Fl(.)237 1920 y Fi(MPI)p 352 1920 V 21 w(W)-6 b(AIT)h(ALL\(list)p 728 1920 V 20 w(of)p 793 1920 V 21 w(handles,)19 b(list)p 1106 1920 V 21 w(of)p 1172 1920 V 20 w(return)p 1348 1920 V 20 w(handles\))237 2041 y Fl(Blo)q(c)o(ks)14 b(un)o(til)f(all)i(comm)o(uni)o(cation)d(op)q (erations)k(asso)q(ciated)g(with)e(handles)h(in)f(the)164 2101 y(list)i(complete,)e(and)j(return)g(the)f(status)h(of)g(all)f(these)h(op)q (erations.)23 b(The)17 b(parameters)164 2161 y(are:)164 2249 y Fi(IN)h(list)p 324 2249 V 22 w(of)p 391 2249 V 20 w(handles)25 b Fl(list)16 b(of)g(handles)h(to)f(comm)o(unication)d(ob)s(jects.)164 2344 y Fi(OUT)19 b(list)p 384 2344 V 21 w(of)p 450 2344 V 20 w(return)p 626 2344 V 20 w(handles)25 b Fl(Must)18 b(ha)o(v)o(e)f(the)h(same) f(length)h(as)h(the)f(\014rst)g(list.)286 2404 y(Eac)o(h)11 b(return)g(status)g(ob)s(ject)g(is)g(set)f(to)i(the)e(return)h(status)h(of)f (the)g(corresp)q(onding)286 2464 y(op)q(eration)17 b(in)f(the)g(\014rst)g (list.)949 2599 y(27)p eop %%Page: 28 28 bop 164 307 a Fm(1.10)70 b(Blo)r(c)n(king)22 b(Comm)n(unication)164 400 y Fl(Blo)q(c)o(king)d(send)h(and)h(receiv)o(e)d(op)q(erations)j(com)o (bine)d(all)h(comm)o(unication)e(sub)q(op)q(er-)164 460 y(ations)k(in)o(to)f (one)h(call.)32 b(The)21 b(op)q(eration)g(returns)f(only)g(when)h(the)f(comm) o(unicati)o(on)164 520 y(completes)g(and)i(no)g(comm)o(unic)o(ation)e(ob)s (ject)h(p)q(ersists)h(after)f(the)h(call)f(completed.)164 580 y(Ho)o(w)o(ev)o(er,)14 b(the)i(bu\013er)h(descriptor)e(ob)s(ject)h(needs)g(b) q(e)h(created)e(ahead)i(of)g(the)f(call.)237 640 y(W)l(e)g(use)g(the)g(follo) o(wing)g(naming)g(con)o(v)o(en)o(tion)f(for)h(suc)o(h)g(op)q(erations:)777 695 y Fa(")824 737 y Fe(\000)822 797 y Fi(R)885 695 y Fa(#)8 b(")966 737 y Fi(SEND)962 797 y(RECV)1145 695 y Fa(#)237 898 y Fl(The)16 b(\014rst)h(letter)e(\(v)o(oid)h(or)g Fi(R)p Fl(\))g(indicates)g (the)g(start)h(mo)q(de)e(\(regular)h(or)h(ready\).)164 1018 y Fi(MPI)p 279 1018 17 2 v 20 w(SEND)i(\(bu\013er)p 639 1018 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1078 y Fl(is)164 1170 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(REGULAR,)g(EPHEMERAL)o(\))164 1231 y(MPI_START\()o(han)o (dl)o(e\))164 1291 y(MPI_WAIT\(h)o(and)o(le)o(,)g(null\))164 1443 y Fi(MPI)p 279 1443 V 20 w(RSEND)c(\(bu\013er)p 681 1443 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1503 y Fl(is)164 1596 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(READY,)g(EPHEMERAL\))164 1656 y(MPI_START\()o(han)o(dl)o (e\))164 1716 y(MPI_WAIT\(h)o(and)o(le)o(,)g(null\))164 1868 y Fi(MPI)p 279 1868 V 20 w(RECV\(bu\013er)p 626 1868 V 20 w(handle,)c (source,)f(tag,)g(con)n(text,)g(return)p 1514 1868 V 19 w(handle\))164 1929 y Fl(is)164 2021 y Fg(MPI_INIT_R)o(ECV)o(\(h)o(and)o(le,)k(buffer_han)o (dl)o(e,)g(source,)h(tag,)i(context,)d(EPHEMERAL\))164 2081 y(MPI_START\()o(han)o(dl)o(e\))164 2141 y(MPI_WAIT\(h)o(and)o(le)o(,re)o(tur) o(n_h)o(an)o(dle)o(\))237 2233 y Fi(Implemen)n(tation)d(note:)237 2293 y Fl(While)k(these)g(functions)h(can)g(b)q(e)g(impleme)o(n)o(ted)d(via)i (calls)g(to)i(functions)e(that)164 2354 y(implem)o(en)o(t)c(sub)q(op)q (erations,)25 b(as)e(describ)q(ed)f(in)g(this)g(subsection,)h(an)g(e\016cien) o(t)d(im-)164 2414 y(plemen)o(tation)d(ma)o(y)h(optimize)f(a)o(w)o(a)o(y)i (these)g(m)o(ultiple)d(calls,)j(pro)o(vided)f(it)h(do)q(es)h(not)164 2474 y(c)o(hange)c(the)g(b)q(eha)o(vior)g(of)h(correct)e(programs.)949 2599 y(28)p eop %%Page: 29 29 bop 164 307 a Fm(1.11)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)164 400 y Fl(Non)o(blo)q(c)o(king)g(send)h(and)g(receiv)o(e)d(op)q(erations)k (com)o(bine)d(the)h(\014rst)h(t)o(w)o(o)g(sub)q(op)q(era-)164 460 y(tions)d(\()p Fg(INIT)e Fl(and)i Fg(START)p Fl(\))e(in)o(to)h(one)h (call.)30 b(They)19 b(use)h(ephemeral)d(comm)o(unicati)o(on)164 520 y(ob)s(jects,)e(so)h(that)f(the)g(op)q(eration)i(is)e(completed,)d(and)k (the)f(asso)q(ciated)i(resources)e(are)164 580 y(freed,)g(b)o(y)h(using)h (one)f(of)h(the)f(functions)g Fg(MPI)p 1014 580 16 2 v 17 w(WAIT,)24 b(MPI)p 1263 580 V 17 w(STATUS,)f(MPI)p 1563 580 V 18 w(WAITANY,)164 640 y(MPI)p 245 640 V 17 w(STATUSANY)p Fl(,)11 b(or)j Fg(MPI)p 656 640 V 17 w(WAITALL)p Fl(.)d(Here,)i(to)q(o,)i(a)g(bu\013er)f(ob)s(ject)f (has)i(to)f(b)q(e)g(created)164 700 y(ahead)j(of)f(the)g(comm)o(unication)d (initiation)j(op)q(eration.)237 761 y(W)l(e)h(use)g(the)f(same)g(naming)g (con)o(v)o(en)o(tion)g(as)i(for)f(blo)q(c)o(king)f(op)q(erations:)24 b(a)17 b(pre\014x)164 821 y(of)f Fi(R)g Fl(indicates)g(the)f Fg(READY)g Fl(mo)q(de.)20 b(In)c(addition,)g(a)g(pre\014x)g(of)g Fi(I)g Fl(is)g(used)g(to)h(indicate)164 881 y Fh(imme)n(diate)f Fl(\(i.e.,)e(non)o(blo)q(c)o(king\))i(execution.)164 1001 y Fi(MPI)p 279 1001 17 2 v 20 w(ISEND)j(\(handle,)g(bu\013er)p 856 1001 V 19 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1062 y Fl(is)164 1163 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o (e,)g(dest,)i(tag,)g(context,)f(REGULAR,)g(EPHEMERAL)o(\))164 1224 y(MPI_START\()o(han)o(dl)o(e\))164 1385 y Fi(MPI)p 279 1385 V 20 w(IRSEND)18 b(\(handle,)h(bu\013er)p 897 1385 V 20 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 1446 y Fl(is)164 1547 y Fg(MPI_INIT_S)o(END)o(\(h)o(and)o(le,)k(buffer_han)o(dl)o(e,)g(dest,)i (tag,)g(context,)f(READY,)g(EPHEMERAL\))164 1608 y(MPI_START\()o(han)o(dl)o (e\))164 1769 y Fi(MPI)p 279 1769 V 20 w(IRECV\(handle,)15 b(bu\013er)p 839 1769 V 19 w(handle,)f(source,)f(tag,)g(con)n(text,)g(return) p 1706 1769 V 19 w(status)p 1871 1769 V 21 w(handle\))164 1830 y Fl(is)164 1931 y Fg(MPI_INIT_R)o(ECV)o(\(h)o(and)o(le,)22 b(buffer_han)o(dl)o(e,)g(source,)h(tag,)i(context,)d(EPHEMERAL\))164 1991 y(MPI_START\()o(han)o(dl)o(e\))164 2136 y Fm(1.12)70 b(Blo)r(c)n(k)22 b(Sending)h(Op)r(erations)164 2228 y Fl(The)h(most)g(frequen)o(t)f(t)o(yp)q (e)g(of)i(bu\013er)f(used)g(is)g(a)h(con)o(tiguous)g(bu\013er)f(of)h(n)o (umeric)164 2288 y(storage)18 b(units,)f(i.e.,)f(a)h(con)o(tiguous)h (bu\013er)g(of)f(w)o(ords)h(that)g(ma)o(y)d(con)o(tain)i(either)f(IN-)164 2349 y(TEGER,)23 b(REAL)g(or)h(LOGICAL)f(v)m(alues)g(\(in)g(F)o(OR)l(TRAN\).) e(In)h(a)i(homogeneous)164 2409 y(en)o(vironmen)o(t)14 b(suc)o(h)j(messages)g (can)g(b)q(e)g(used)h(to)f(send)h(arbitrary)f(sequences)f(of)h(con-)164 2469 y(tiguous)g(items,)d(where)i(eac)o(h)f(item)g(o)q(ccupies)h(an)g(in)o (teger)g(n)o(um)o(b)q(er)e(of)j(w)o(ords.)949 2599 y(29)p eop %%Page: 30 30 bop 237 307 a Fl(W)l(e)17 b(sp)q(ecialize)f(the)i(functions)f(in)g(the)h(t)o (w)o(o)f(previous)g(subsections)h(to)g(this)f(case,)164 367 y(th)o(us)f(a)o(v)o(oiding)f(the)g(need)g(for)h(the)g(creation)f(of)h(a)g (bu\013er)g(descriptor)f(ob)s(ject.)21 b(W)l(e)15 b(use)164 428 y(the)i(same)f(naming)g(sc)o(heme)f(used)i(in)g(the)f(previous)h (subsections,)g(and)h(app)q(end)g(a)f Fi(B)164 488 y Fl(in)f(the)g(function)g (name,)f(for)h Fg(BLOCK)p Fl(.)164 608 y Fi(MPI)p 279 608 17 2 v 20 w(SENDB)j(\(start,)f(len,)g(dest,)g(tag,)g(con)n(text\))164 668 y Fl(is)164 770 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)k (MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 830 y(MPI_ADD\(bu)o(ffe)o(r_)o(han)o (dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 890 y(MPI_SEND)f(\(buffer_h)o(and)o (le,)f(len,)i(dest,)g(tag,)g(context\))164 1052 y Fi(MPI)p 279 1052 V 20 w(RSENDB)18 b(\(bu\013er)p 720 1052 V 20 w(handle,)h(dest,)f (tag,)g(con)n(text\))164 1112 y Fl(is)164 1214 y Fg(MPI_CREATE)o(\(bu)o(ff)o (er_)o(han)o(dle)o(,)k(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 1274 y(MPI_ADD\(bu)o(ffe)o(r_)o(han)o(dle)o(,)h(start,)g(len,)h(MPI_REAL\)) 164 1335 y(MPI_RSEND)e(\(buffer_han)o(dle)o(,)g(len,)j(dest,)e(tag,)i (context\))237 1436 y Fi(MPI)p 352 1436 V 21 w(RECVB\(bu\013er)p 740 1436 V 19 w(handle,)14 b(source,)f(tag,)g(con)n(text,)g(return)p 1607 1436 V 20 w(status)p 1773 1436 V 20 w(handle\))164 1496 y Fl(is)164 1598 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 1658 y(MPI_ADD\(bu)o(ffe)o(r_)o (han)o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 1719 y(MPI_RECV\(b)o(uff)o (er)o(_ha)o(ndl)o(e,)e(source,)h(tag,)h(context,)f(return_sta)o(tus)o(_ha)o (nd)o(le\))164 1880 y Fi(MPI)p 279 1880 V 20 w(ISENDB)c(\(handle,)f(bu\013er) p 895 1880 V 20 w(handle,)h(dest,)f(tag,)g(con)n(text\))164 1941 y Fl(is)164 2042 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)k (MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 2103 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 2163 y(MPI_ISEND\()o(han)o(dl)o (e,)e(buffer_hand)o(le,)g(dest,)i(tag,)g(context\))164 2325 y Fi(MPI)p 279 2325 V 20 w(IRSENDB)18 b(\(handle,)h(bu\013er)p 937 2325 V 20 w(handle,)g(dest,)f(tag,)g(con)n(text\))164 2385 y Fl(is)949 2599 y(30)p eop %%Page: 31 31 bop 164 307 a Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 367 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 428 y(MPI_IRSEND)o(\(ha)o(nd)o (le,)e(buffer_han)o(dle)o(,)g(dest,)i(tag,)g(context\))164 587 y Fi(MPI)p 279 587 17 2 v 20 w(IRECVB\(handle,)14 b(bu\013er)p 878 587 V 20 w(handle,)g(source,)f(tag,)g(con)n(text,)g(return)p 1746 587 V 19 w(status)p 1911 587 V 21 w(handle\))164 647 y Fl(is)164 746 y Fg(MPI_CREATE)o(\(bu)o(ff)o(er_)o(han)o(dle)o(,)22 b(MPI_BUFFER,)g(MPI_EPHEME)o(RAL)o(\))164 807 y(MPI_ADD\(bu)o(ffe)o(r_)o(han) o(dle)o(,)h(start,)g(len,)h(MPI_REAL\))164 867 y(MPI_IRECV\()o(han)o(dl)o(e,) e(buffer_hand)o(le,)g(source,)h(tag,)h(context\))164 1075 y Fk(Discussion:)237 1132 y Fj(I)18 b(use)f(w)o(ord)g(coun)o(t,)g(rather)f (than)h(b)o(yte)g(coun)o(t.)26 b(I)18 b(b)q(eliev)o(e)h(w)o(ord)d(messages)h (are)g(m)o(uc)o(h)164 1188 y(more)e(prev)m(alen)o(t)g(than)g(b)o(yte)g (messages,)f(and)h(it's)g(a)g(blessing)h(not)f(to)f(ha)o(v)o(e)h(to)f(m)o (ultiply)j(b)o(y)164 1245 y(4)h(for)g(eac)o(h)h(message.)30 b(Byte)19 b(messages)f(are)g(still)j(a)o(v)m(ailable)f(the)f(hard)f(w)o(a)o (y)l(.)30 b(Also,)20 b(w)o(ord)164 1301 y(messages)15 b(\014t)g(w)o(ell)h (with)f(the)h(F)l(ortran)e(n)o(umeric)i(storage)e(unit)i(concept.)237 1357 y(If)21 b(w)o(e)f(are)g(to)g(add)h(more)f(functions,)j(m)o(y)d(next)g (addition)i(w)o(ould)f(b)q(e)g(a)f(v)o(ector)g(send)164 1414 y(op)q(eration.)164 1739 y Fm(1.13)70 b(Correctness)164 1951 y Fk(Discussion:)27 b Fj(The)18 b(material)h(in)g(this)g(section)g(has)g(not) f(y)o(et)g(b)q(een)h(discussed)h(b)o(y)f(MPIF.)164 2011 y(Some)d(or)g(all)h (of)f(it)h(is)f(lik)o(ely)i(to)e(mo)o(v)o(e)g(to)f(Section)i Fk(??)p Fj(.)23 b(It)16 b(is)h(incorp)q(orated)g(here)f(for)g(com-)164 2072 y(pleteness.)164 2322 y Fi(1.13.1)55 b(Order)164 2414 y Fl(MPI)15 b(preserv)o(es)g(the)h(order)g(of)g(messages)f(b)q(et)o(w)o(een)g (an)o(y)h(\014xed)f(pair)h(of)h(pro)q(cesses.)k(In)164 2474 y(other)16 b(w)o(ords,)g(if)f(pro)q(cess)h(A)f(executes)f(t)o(w)o(o)i (successiv)o(e)e(send)i Fg(start)d Fl(sub)q(op)q(erations,)949 2599 y(31)p eop %%Page: 32 32 bop 164 307 a Fl(pro)q(cess)13 b(B)e(executes)g(t)o(w)o(o)h(successiv)o(e)e (receiv)o(e)g Fg(start)g Fl(op)q(erations,)k(and)e(b)q(oth)h(receiv)o(es)164 367 y(matc)o(h)19 b(either)h(sends,)i(then)f(the)f(\014rst)h(receiv)o(e)e (will)h(receiv)o(e)e(the)j(message)f(sen)o(t)g(b)o(y)164 428 y(the)g(\014rst)g(send,)h(and)g(the)f(second)g(receiv)o(e)e(will)h(receiv)o (e)f(the)i(message)g(sen)o(t)f(b)o(y)h(the)164 488 y(second)h(send.)36 b(Th)o(us,)22 b(if)e(a)i(t)o(w)o(o)f(messages)f(from)g(the)h(same)f(source)h (can)g(satisfy)h(a)164 548 y(p)q(ending)d(receiv)o(e,)c(the)j(\014rst)h (message)e(sen)o(t)h(is)g(accepted;)g(if)g(a)g(message)g(can)g(satisfy)164 608 y(t)o(w)o(o)e(p)q(ending)h(receiv)o(es,)c(the)j(\014rst)h(receiv)o(e)d(p) q(osted)j(is)f(satis\014ed.)237 668 y(The)h(last)g(paragraph)i(assumes)e (that)g(the)g(send)g Fg(start)f Fl(op)q(erations)i(are)f(ordered)164 729 y(b)o(y)k(the)g(program)h(order)f(at)h(pro)q(cess)g(A,)f(and)h(the)f (receiv)o(e)f Fg(start)f Fl(op)q(erations)k(are)164 789 y(ordered)f(b)o(y)f (the)h(program)f(order)h(at)g(pro)q(cess)h(B.)e(If)g(a)h(pro)q(cess)h(is)e(m) o(ultithreaded)164 849 y(and)f(the)f(op)q(erations)h(are)f(executed)f(b)o(y)g (distinct)h(threads,)g(then)g(the)g(seman)o(tics)f(of)164 909 y(the)d(threaded)g(system)e(ma)o(y)h(not)h(de\014ne)g(an)g(order)h(b)q(et)o (w)o(een)e(the)g(t)o(w)o(o)h(op)q(erations,)h(in)164 969 y(whic)o(h)g(case)g (the)g(condition)g(is)g(v)o(oid.)164 1099 y Fi(1.13.2)55 b(Progress)19 b(and)g(F)-5 b(airness)164 1192 y Fl(W)l(e)22 b(can)h(mo)q(del)f(the)g (execution)g(of)h(MPI)f(programs)h(as)h(an)f(in)o(teraction)f(b)q(et)o(w)o (een)164 1252 y(executing)h(pro)q(cesses)h(that)g(execute)e(eac)o(h)h(their)g (o)o(wn)h(program,)h(and)f(the)g Fi(com-)164 1312 y(m)n(unication)c (subsystem)p Fl(.)g(The)c(comm)o(unic)o(ation)d(subsystem)i(ma)o(y)f(ha)o(v)o (e)h(v)m(arious)164 1372 y(constrain)o(ts)h(on)h(the)f(amoun)o(t)g(of)g (resources)g(it)g(can)h(use.)k(E.g.:)237 1432 y(Bounds)e(on)f(the)g(n)o(um)o (b)q(er)e(and)i(total)h(sizes)e(of)h(activ)o(e)f(comm)o(unic)o(ation)f(ob)s (jects.)164 1492 y(Suc)o(h)g(b)q(ound)h(can)g(b)q(e)f(global,)g(p)q(er)h(no)q (de,)f(or)h(p)q(er)f(pair)g(of)h(comm)o(unic)o(ating)d(no)q(des.)237 1553 y(Bounds)g(on)g(the)f(n)o(um)o(b)q(er)f(and)i(total)g(sizes)f(of)h (messages)f(bu\013ered)g(in)h(the)f(system.)164 1613 y(Suc)o(h)19 b(b)q(ound)h(can,)g(again,)g(b)q(e)g(global,)g(p)q(er)f(no)q(de,)h(or)g(p)q (er)f(pair)h(of)f(comm)o(unicati)o(ng)164 1673 y(no)q(de.)j(In)16 b(addition,)f(a)i(message)e(ma)o(y)g(b)q(e)h(bu\013ered)g(at)g(the)g(sender,) g(at)g(the)g(receiv)o(er,)164 1733 y(at)h(b)q(oth,)f(or)h(p)q(erhaps)g(at)g (another)g(place)e(altogether.)237 1793 y(Th)o(us,)g(it)g(will)f(b)q(e)h (di\016cult)e(to)j(set)f(rules)f(on)i(resource)e(managemen)o(t)f(of)i(the)g (com-)164 1854 y(m)o(unication)g(subsystem.)21 b(Ho)o(w)o(ev)o(er,)15 b(it)h(is)g(generally)g(exp)q(ected)f(that)i(impleme)o(n)o(ters)164 1914 y(will)e(pro)o(vide)f(information)h(on)h(the)g(mec)o(hanism)c(used)k (for)g(resource)g(allo)q(cation,)f(and)164 1974 y(that)22 b(query)g(and)g (set)g(functions)g(will)f(allo)o(w)h(to)g(query)g(and)g(p)q(ossibly)g(con)o (trol)g(the)164 2034 y(amoun)o(t)15 b(of)i(a)o(v)m(ailable)f(resources.)237 2094 y(W)l(e)d(pro)o(vide)f(in)g(this)h(section)g(a)g(set)g(of)g(minim)o(al)d (requiremen)o(ts)g(on)j(the)g(comm)o(uni-)164 2155 y(cation)h(subsystem.)19 b(Programs)14 b(that)g(execute)e(on)j(an)o(y)e(subsystem)g(that)h(ful\014ls)f (these)164 2215 y(minim)o(al)18 b(requiremen)o(ts)g(are)j Fi(safe)g Fl(and)h(will)d(p)q(ort)j(to)f(an)o(y)g(MPI)f(impleme)o(n)o(tation.)164 2275 y Fi(Unsafe)h Fl(programs)h(ma)o(y)d(execute)h(on)i(some)e(MPI)h(implem) o(e)o(n)o(tations,)f(dep)q(ending)164 2335 y(on)h(the)f(amoun)o(t)f(of)i(a)o (v)m(ailable)e(resources)h(and)h(the)f(implem)o(en)o(tation)d(used)k(for)f (the)164 2395 y(MPI)14 b(comm)o(unication)e(subsystem.)19 b(Finally)14 b Fi(erroneous)g Fl(programs)h(nev)o(er)e(execute)164 2456 y(correctly)l(.)19 b(\(While)14 b(it)g(is)g(desirable)g(to)g(detect)g (erroneous)h(programs,)g(it)f(is)g(not)h(p)q(ossi-)949 2599 y(32)p eop %%Page: 33 33 bop 164 307 a Fl(ble)13 b(to)g(do)h(so)g(at)g(compile)d(time,)h(and)i(often)f (prohibitiv)o(e)f(to)i(do)f(so)i(a)e(run)h(time.)k(Th)o(us,)164 367 y(the)e(do)q(cumen)o(t)f(do)q(es)j(not)f(sp)q(ecify)e(a)i(b)q(eha)o(vior) g(for)g(erroneous)g(programs,)f(although)164 428 y(the)g(desired)g(b)q(eha)o (vior)g(is)g(to)g(return)g(a)h(useful)f(error)g(message.\))224 542 y(1.)24 b(If)15 b(a)h(pro)q(cess)h(executes)d(an)j Fg(INIT)d Fl(op)q(eration,)i(then)g(the)f(op)q(eration)i(ev)o(en)o(tually)286 602 y(succeeds,)c(or)g(a)g Fh(r)n(esour)n(c)n(e)h(exc)n(eption)g Fl(o)q(ccurs.)21 b(The)13 b(standard)h(do)q(es)g(not)g(sp)q(ecify)286 662 y(when)23 b(a)h(resource)g(exception)e(is)h(allo)o(w)o(ed)g(to)h(o)q (ccur.)43 b(It)23 b(is)g(exp)q(ected)f(that)286 722 y(an)g(op)q(erational)h (de\014nition)e(will)g(b)q(e)h(made)f(a)o(v)m(ailable,)h(in)f(the)h(form)f (of)h(test)286 782 y(programs)14 b(that)f(ha)o(v)o(e)g(to)h(execute)e(with)h (no)h(resource)g(exceptions.)19 b(It)13 b(is)g(highly)286 843 y(desirable)e(to)i(ha)o(v)o(e)e(generous)i(b)q(ounds)h(on)e(the)g(n)o(um)o(b) q(er)f(of)h(concurren)o(tly)f(activ)o(e)286 903 y(comm)o(unic)o(ation)20 b(ob)s(jects)j(eac)o(h)f(pro)q(cess)h(ma)o(y)e(ha)o(v)o(e,)i(so)h(that,)g(in) e(practice,)286 963 y Fg(INIT)15 b Fl(op)q(erations)i(will)e(alw)o(a)o(ys)h (b)q(e)h(guaran)o(teed)f(to)h(succeed.)224 1065 y(2.)24 b(Eac)o(h)d(pro)q (cess)g(can)g(initiate)f(a)h(comm)o(uni)o(cation)d(op)q(eration)k(for)f(eac)o (h)f(activ)o(e)286 1125 y(comm)o(unic)o(ation)e(ob)s(ject.)32 b(I.e.)f(correct)20 b Fg(START)e Fl(op)q(erations)j(alw)o(a)o(ys)f(succeed) 286 1185 y(\(ev)o(en)o(tually\).)224 1287 y(3.)k(A)10 b(send)h(op)q(eration)h (is)f Fi(enabled)g Fl(if)f(the)h(sending)g(pro)q(cess)g(has)h(issued)e(a)i Fg(COMPLETE)286 1347 y Fl(op)q(eration)20 b(and)g(the)g(receiving)e(pro)q (cess)i(has)g(issued)g(a)f Fg(START)f Fl(op)q(eration)j(for)286 1407 y(a)e(matc)o(hing)f(receiv)o(e.)28 b(Symmetri)o(call)o(y)l(,)16 b(a)k(receiv)o(e)d(op)q(eration)j(is)f Fi(enabled)g Fl(if)286 1467 y(the)h(receiving)e(pro)q(cess)j(has)f(issued)g(a)g Fg(COMPLETE)d Fl(op)q(eration)k(and)g(the)f(send-)286 1528 y(ing)j(pro)q(cess)h(has)g (issued)f(a)g Fg(START)f Fl(op)q(eration)i(for)f(a)h(matc)o(hing)d(send.)42 b(An)286 1588 y(enabled)19 b(op)q(eration)i(ma)o(y)d(b)q(ecome)g Fi(disabled)j Fl(either)e(b)q(ecause)h(it)f(completes)286 1648 y(successfully)c(or,)h(in)g(the)g(case)g(of)h(a)g(receiv)o(e,)c(b)q(ecause)k (the)f(matc)o(hing)f(message)286 1708 y(is)h(successfully)f(receiv)o(ed)f(b)o (y)i(another)g(receiv)o(e)e(op)q(eration.)286 1789 y Fi(An)30 b(enabled)h(op)r(eration)f(either)f(completes)g(successfully)i(or)f(b)r(e-) 286 1849 y(comes)18 b(p)r(ermanen)n(tly)g(disabled.)224 1951 y Fl(4.)24 b(A)16 b Fg(FREE)f Fl(op)q(eration)i(alw)o(a)o(ys)f(succeeds)f (\(ev)o(en)o(tually\).)237 2065 y(The)f(four)h(conditions)f(guaran)o(tee)h (progress)g(in)e(the)h(comm)o(unication)d(subsystem.)164 2125 y(The)19 b(third)f(condition)h(guaran)o(tee)g(\(w)o(eak\))f(fairness)h(among) f(comp)q(eting)g(comm)o(uni-)164 2186 y(cation)e(requests.)237 2246 y(Examples)f(\(in)o(v)o(olving)g(t)o(w)o(o)h(pro)q(cessors)h(with)f (ranks)h(0)g(and)f(1\))237 2306 y(The)g(follo)o(wing)g(program)g(is)g(safe,)g (and)h(should)g(alw)o(a)o(ys)f(succeed.)164 2468 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))949 2599 y Fl(33)p eop %%Page: 34 34 bop 164 307 a Fg(IF)25 b(\(rank.EQ.0)o(\))d(THEN)241 367 y(CALL)i (MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))241 428 y(CALL)g(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 488 y(ELSE)101 b(!)25 b(rank.EQ.1)241 548 y(CALL)f(MPI_RECVB\()o(rec)o(vbu)o (f,)e(len,)i(0,)h(tag,)f(context\))241 608 y(CALL)g(MPI_SENDB\()o(sen)o(dbu)o (f,)e(len,)i(0,)h(tag,)f(context\))164 668 y(END)h(IF)237 765 y Fl(The)16 b(follo)o(wing)g(program)g(is)g(erroneous,)h(and)g(should)f(alw)o (a)o(ys)g(fail.)164 934 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))164 994 y(IF)i(\(rank.EQ.0)o(\))d(THEN)241 1054 y(CALL)i(MPI_RECVB\()o(rec)o(vbu) o(f,)e(len,)i(1,)h(tag,)f(context\))241 1114 y(CALL)g(MPI_SENDB\()o(sen)o (dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 1174 y(ELSE)101 b(!)25 b(rank.EQ.1)241 1235 y(CALL)f(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,)i (0,)h(tag,)f(context\))241 1295 y(CALL)g(MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,) i(0,)h(tag,)f(context\))164 1355 y(END)h(IF)237 1463 y Fl(The)17 b(receiv)o(e)e(op)q(eration)j(of)f(the)f(\014rst)i(pro)q(cess)f(m)o(ust)f (complete)e(b)q(efore)k(its)e(send,)164 1523 y(and)k(can)g(complete)e(only)h (if)h(the)f(matc)o(hing)f(send)i(of)g(the)g(second)g(pro)q(cessor)h(is)e(ex-) 164 1584 y(ecuted;)24 b(the)e(receiv)o(e)f(op)q(eration)i(of)g(the)f(second)h (pro)q(cess)g(m)o(ust)e(complete)f(b)q(efore)164 1644 y(its)f(send)h(and)g (can)g(complete)d(only)j(if)f(the)g(matc)o(hing)f(send)i(of)g(the)f(\014rst)h (pro)q(cess)g(is)164 1704 y(executed.)g(This)c(program)g(will)f(deadlo)q(c)o (k.)237 1764 y(The)i(follo)o(wing)h(program)f(is)g(unsafe,)g(and)h(ma)o(y)e (succeed)h(or)g(fail,)g(dep)q(ending)g(on)164 1824 y(implem)o(en)o(tati)o (on.)164 1993 y Fg(CALL)24 b(MPI_RANK\(r)o(ank)o(,)f(context\))164 2053 y(IF)i(\(rank.EQ.0)o(\))d(THEN)241 2113 y(CALL)i(MPI_SENDB\()o(sen)o (dbu)o(f,)e(len,)i(1,)h(tag,)f(context\))241 2173 y(CALL)g(MPI_RECVB\()o(rec) o(vbu)o(f,)e(len,)i(1,)h(tag,)f(context\))164 2233 y(ELSE)101 b(!)25 b(rank.EQ.1)241 2293 y(CALL)f(MPI_SENDB\()o(sen)o(dbu)o(f,)e(len,)i (0,)h(tag,)f(context\))241 2354 y(CALL)g(MPI_RECVB\()o(rec)o(vbu)o(f,)e(len,) i(0,)h(tag,)f(context\))164 2414 y(END)h(IF)949 2599 y Fl(34)p eop %%Page: 35 35 bop 237 307 a Fl(The)19 b(message)f(sen)o(t)g(b)o(y)g(eac)o(h)h(pro)q(cess)g (has)g(to)g(b)q(e)g(copied)f(out)i(b)q(efore)e(the)h(send)164 367 y(op)q(eration)d(returns)g(and)g(the)f(receiv)o(e)e(op)q(eration)j (starts.)22 b(F)l(or)16 b(the)f(program)g(to)h(com-)164 428 y(plete,)i(it)h(is)g(necessary)g(that)h(at)f(least)g(one)h(of)f(the)g(t)o(w)o (o)g(messages)g(sen)o(t)g(is)g(bu\013ered)164 488 y(out)e(of)g(either)f(pro)q (cesses')h(address)g(space.)23 b(Th)o(us,)17 b(this)g(program)f(can)h (succeed)f(only)164 548 y(if)h(the)g(comm)o(uni)o(cation)e(system)g(has)j (su\016cien)o(t)e(bu\013er)i(space)f(to)h(bu\013er)f Fg(len)f Fl(w)o(ords)164 608 y(of)h(data.)237 668 y(If)d(additional)h(requiremen)o(ts) d(will)i(b)q(ecome)f(part)i(of)g(the)g(standard)h(\(e.g.,)e(b)q(ounds)164 729 y(on)23 b(the)f(minimal)d(n)o(um)o(b)q(er)i(of)i(concurren)o(tly)e(activ) o(e)h(handles)g(that)h(need)f(b)q(e)h(sup-)164 789 y(p)q(orted,)16 b(then)g(further)g(programs)h(b)q(ecome)d(safe.)949 2599 y(35)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:17:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02635; Mon, 15 Mar 93 17:17:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15024; Mon, 15 Mar 93 17:16:49 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:16:47 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15005; Mon, 15 Mar 93 17:16:44 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA26351; Mon, 15 Mar 93 17:16:44 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 171443.29617; Mon, 15 Mar 1993 17:14:43 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701 for ) id AA16876; Mon, 15 Mar 1993 15:15:28 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA02007; Mon, 15 Mar 93 15:15:27 -0600 Date: Mon, 15 Mar 93 15:15:27 -0600 Message-Id: <9303152115.AA02007@brisk.kai.com> To: mpi-pt2pt@CS.UTK.EDU Cc: Subject: READY Send Rationale Please. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Someone please give me a rationale for Ready Send. Doesn't Ready Send add overhead to every point-to-point communication? In a dumb implementation (i.e. one that isn't able to analyse the usage and detect that I never - ever - use Ready Send) receives will feel constrained to notify senders in all cases on the off chance that one is a Ready Send. This restricts good implementations that do not require such a dumb primitive. Further, given the non-ready state caveat "outcome is undefined" and a system where I cannot know, or choose not to know, the readiness of receives - am I able to optimize away the use of Ready Sends? If I am, as seems likely, I suggest we optimize it out of the standard. It is certainly going to complicate the formal specification and its removal further simplifies the number of calls. The only rationale I could think of (excepting that I have a dumb implementation of communication ;-) ) is memory mapped I/O devices - but a) I thought for some reason that that was not within the scope of MPI, and b) in any case, such should not be allowed to impact on the general primitives - there are better solutions. But please, if someone has a plausible rationale to keep Ready Send in this standard, let's hear it. Steven PS. We'll get to a single well defined "send" and "recv" yet! ;-) From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:39:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02984; Mon, 15 Mar 93 17:39:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15838; Mon, 15 Mar 93 17:38:49 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:38:48 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15830; Mon, 15 Mar 93 17:38:47 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA17275; Mon, 15 Mar 93 17:38:45 EST Received: by b125.super.org (4.1/SMI-4.1) id AA00438; Mon, 15 Mar 93 17:38:44 EST Date: Mon, 15 Mar 93 17:38:44 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303152238.AA00438@b125.super.org> To: mpi-pt2pt@CS.UTK.EDU In-Reply-To: Steven Ericsson Zenith's message of Mon, 15 Mar 93 15:15:27 -0600 <9303152115.AA02007@brisk.kai.com> Subject: READY Send Rationale Please. The Ready Send was added because it can reduce overhead in some systems. As the current draft states: "On some systems, this will allow to optimize communication and avoid a hand-shaking operation that is otherwise required." On many message passing systems the sender must probe the receiver to make sure that space is available before the send can occur. On some system(s) this is done for each set of packets sent. With Ready Send, the sender knows that the receive is posted and does not need to send a probe message to check and this can reduce latency and increase throughput. A good algorithmic example of its use is for a global combine where the result winds up in every node. If you use a minimum spanning tree to a root node, the receives can be posted before the nodes contribution to the global value is passed on. When the root node uses the inverse tree to fan the answer back out, it can use a Ready Send without any chance of losing data. I think it has reasonable uses. One can question whether is deserves to be in the standard given the added complexity of additional choices. However, after two votes of yes for this construct, I think we should leave it unless there is a global move to simplify MPI. Steve From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 17:59:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04186; Mon, 15 Mar 93 17:59:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16444; Mon, 15 Mar 93 17:58:16 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 17:58:15 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16436; Mon, 15 Mar 93 17:58:09 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 15 Mar 93 14:56 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA01275; Mon, 15 Mar 93 14:54:26 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA06683; Mon, 15 Mar 93 14:54:24 PST Date: Mon, 15 Mar 93 14:54:24 PST From: rj_littlefield@pnlg.pnl.gov Subject: context creation To: mpi-pt2pt@CS.UTK.EDU, snir@watson.ibm.com Cc: d39135@sodium.pnl.gov Message-Id: <9303152254.AA06683@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@CS.UTK.EDU The new point-to-point proposal distributed on 3/15/93 seems to be generally well thought out. However, there is a simple application program design that I do not see how to implement using only the proposed context-forming routines. The same situation arose with the old group proposal, and I have posted to mpi-collcom and mpi-context a suggested fix. What follows here is a similar suggestion for the new point-to-point proposal. The application design in question uses a master-slaves strategy to asynchronously parcel out chunks of work, with each chunk being done by several processes working collaboratively. The idea is that as chunks of work finish, the processes that were assigned to them go into an idle pool, and when the pool gets big enough, some processes are pulled out and assigned to the next chunk of work. Collective communication between processes working on each chunk is required, so it seems natural to organize them into MPI contexts. However, using a synchronous context partitioning routine, such as MPI_NEW_CONTEXT, is not attractive because the varying chunk size implies that each group of processes can finish their work at different times, and synchronous partitioning would delay the assignment of new work until all currently active chunks had finished. This prompts me to suggest the addition of a less synchronous context-forming routine. This routine would be called only by those processes wishing to join the new context, rather than all processes belonging to the old context. MPI_FORM_CONTEXT (newcontext,oldcontext,nprocs,knownmembers,tag) where newcontext is output as a handle for the new context. oldcontext is a handle for the context that contains all of the participating processes. nprocs is the number of processes that will belong to the new context. knownmembers is a set of ranks, within oldcontext, of some or all members of the new context. Each member of the new context must provide the same set of knownmembers. tag is a user-provided integer tag, sufficiently unique to disambiguate overlapping contexts that might be formed simultaneously (see discussion below) In the appication design discussed above, the master would use point-to-point messages in the ALL context to inform individual processes of their assignments and collaborators, and the processes assigned to each chunk would organize themselves by calling MPI_FORM_CONTEXT. The knownmembers argument specifies a set of processes that can act as organizers of the context. In earlier discussions, some people have argued that group- or context-creation calls should not have to specify all members because that would violate scalability. If knownmembers is allowed to be less than the full set, then a tag is required to handle the case of overlapping process sets being handled simultaneously. Suppose that processes 2 and 3 each attempt to form a 2-process context with process 1, and process 1 is the only specified knownmember. (I.e., process 1 makes two calls, each specifying a 2-process context but not indicating which other process.) Then messages from 2 and 3 can be received at 1 in either order, and the tag is required to keep things straight. Personally, I would consider it acceptable to require that knownmembers specify all the members of the context, in which case there is no need for tag because things are kept straight by sequencing. Have I missed something about how to use MPI_NEW_CONTEXT? Is there a better way to implement this type of application than MPI_FORM_CONTEXT, or does MPI_FORM_CONTEXT have some drawback that I don't see? --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Mon Mar 15 21:40:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07549; Mon, 15 Mar 93 21:40:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23801; Mon, 15 Mar 93 21:39:23 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 15 Mar 1993 21:39:22 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from relay1.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23793; Mon, 15 Mar 93 21:39:20 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay1.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA08956; Mon, 15 Mar 93 21:39:18 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 213832.9336; Mon, 15 Mar 1993 21:38:32 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-93012701 for ) id AA22840; Mon, 15 Mar 1993 18:28:24 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA02416; Mon, 15 Mar 93 18:28:23 -0600 Date: Mon, 15 Mar 93 18:28:23 -0600 Message-Id: <9303160028.AA02416@brisk.kai.com> To: mpi-pt2pt@CS.UTK.EDU Cc: Subject: READY Send rationale From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Ah yes, I recall this discussion now, Marc gave a good description of the spanning tree algorithm using it in Dallas last. My instinct is to complain about the implementations that benefit from it. And I don't really like our design being driven to provide efficiency in certain implementations because of a system characteristic. However, I should have come to understand this earlier perhaps. We need to say something else to guide implementors least they make the mistake I mentioned earlier. For example, we should say that Ready Send can always be replaced by Send but that Send may never be replaced by Ready Send. This gives us some greater portability and permits vendors to implement Ready Send as Send if they choose. Further, we should include the rationale that drove us to design in Ready Send. The above caveat solves my specification problem. Steven From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:22:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10313; Thu, 18 Mar 93 08:22:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03936; Thu, 18 Mar 93 08:20:28 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:20:27 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03919; Thu, 18 Mar 93 08:20:23 -0500 Date: Thu, 18 Mar 93 13:20:05 GMT Message-Id: <9135.9303181320@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: document of March 15 To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all I just got back from a session of leave and customer visits, and what a lovely lot of email I find! I read the document of March 15, "Point to Point Communication", by Marc Bill and Rusty. I suggest that we delete certain sections of this document as they are not within the remit of the point-to-point subcommittee. This material is: a) Much of section 1.2 and in particular from "List of handles" to end of section 1.2.4, as it is the remit of the bindings subcommittee to make proposals to the committee on the subject of bindings. b) Much of section 1.4 and in particular from beginning to end of section 1.4.1 as it is the remit of the contexts subcommittee to make proposals to the committee on the subject of contexts. Comments? Flames?? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:28:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10347; Thu, 18 Mar 93 08:28:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04209; Thu, 18 Mar 93 08:26:59 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:26:58 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04201; Thu, 18 Mar 93 08:26:47 -0500 Date: Thu, 18 Mar 93 13:26:37 GMT Message-Id: <9150.9303181326@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: btw To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all Should have mentioned, that apart from the material referred to in the previous message, on a first skim-read the document looks generally great. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:47:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10749; Thu, 18 Mar 93 08:47:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04894; Thu, 18 Mar 93 08:45:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:45:06 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04866; Thu, 18 Mar 93 08:45:03 -0500 Message-Id: <9303181345.AA04866@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1703; Thu, 18 Mar 93 08:44:53 EST Date: Thu, 18 Mar 93 08:33:45 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Reply-To: SNIR@watson.ibm.com Subject: *************** Referenced Note *************** Received: from CS.UTK.EDU by watson.ibm.com (IBM VM SMTP V2R3) with TCP; Thu, 18 Mar 93 08:22:12 EST Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03936; Thu, 18 Mar 93 08:20:28 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:20:27 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03919; Thu, 18 Mar 93 08:20:23 -0500 Date: Thu, 18 Mar 93 13:20:05 GMT Message-Id: <9135.9303181320@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: document of March 15 To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi all I just got back from a session of leave and customer visits, and what a lovely lot of email I find! I read the document of March 15, "Point to Point Communication", by Marc Bill and Rusty. I suggest that we delete certain sections of this document as they are not within the remit of the point-to-point subcommittee. This material is: a) Much of section 1.2 and in particular from "List of handles" to end of section 1.2.4, as it is the remit of the bindings subcommittee to make proposals to the committee on the subject of bindings. b) Much of section 1.4 and in particular from beginning to end of section 1.4.1 as it is the remit of the contexts subcommittee to make proposals to the committee on the subject of contexts. Comments? Flames?? >>> The sections on language objects and on context need to be moved >>> to a general introduction in the final document. The section on >>> context should be construed as my proposal to the context committee >>> and should be discussed in this subcommittee. The section(s) on >>> language objects identifies which objects are used by the point to >>> point communication functions. The comments on their actual >>> embodiment in Fortran and C should be construed as my proposal to the >>> language binding subcommittee and should be discussed there. I >>> willfully trespassed in my document, in order to provide a fuller >>> view and speed up the process. >>> I shall be happy to see this text replaced by a more elaborate >>> proposal once the relevant subcommittees catch up. Best Wishes Lyndon >>> Marc /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 18 08:57:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10932; Thu, 18 Mar 93 08:57:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05304; Thu, 18 Mar 93 08:55:04 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:55:03 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05274; Thu, 18 Mar 93 08:54:58 -0500 Date: Thu, 18 Mar 93 13:54:52 GMT Message-Id: <9194.9303181354@subnode.epcc.ed.ac.uk> From: L J Clarke To: SNIR@watson.ibm.com, mpi-pt2pt@cs.utk.edu In-Reply-To: Marc Snir's message of Thu, 18 Mar 93 08:33:45 EST Reply-To: lyndon@epcc.ed.ac.uk Hi Marc Fine! Then, can we please a) annotate the document to make this clear, and b) exclude such sections from discussion in the meeting of the point-to-point subcommittee. Best Wishes Lyndon > *************** Referenced Note *************** > > Received: from CS.UTK.EDU by watson.ibm.com (IBM VM SMTP V2R3) with TCP; > Thu, 18 Mar 93 08:22:12 EST > Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) > id AA03936; Thu, 18 Mar 93 08:20:28 -0500 > X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 18 Mar 1993 08:20:27 EST > Errors-To: owner-mpi-pt2pt@CS.UTK.EDU > Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) > id AA03919; Thu, 18 Mar 93 08:20:23 -0500 > Date: Thu, 18 Mar 93 13:20:05 GMT > Message-Id: <9135.9303181320@subnode.epcc.ed.ac.uk> > From: L J Clarke > Subject: document of March 15 > To: mpi-pt2pt@cs.utk.edu > Reply-To: lyndon@epcc.ed.ac.uk > > Hi all > > I just got back from a session of leave and customer visits, and what a > lovely lot of email I find! > > I read the document of March 15, "Point to Point Communication", by Marc > Bill and Rusty. > > I suggest that we delete certain sections of this document as they are > not within the remit of the point-to-point subcommittee. This material > is: > > a) Much of section 1.2 and in particular from "List of handles" to end > of section 1.2.4, as it is the remit of the bindings subcommittee to > make proposals to the committee on the subject of bindings. > > b) Much of section 1.4 and in particular from beginning to end of > section 1.4.1 as it is the remit of the contexts subcommittee to make > proposals to the committee on the subject of contexts. > > Comments? Flames?? > > >>> The sections on language objects and on context need to be moved > >>> to a general introduction in the final document. The section on > >>> context should be construed as my proposal to the context committee > >>> and should be discussed in this subcommittee. The section(s) on > >>> language objects identifies which objects are used by the point to > >>> point communication functions. The comments on their actual > >>> embodiment in Fortran and C should be construed as my proposal to the > >>> language binding subcommittee and should be discussed there. I > >>> willfully trespassed in my document, in order to provide a fuller > >>> view and speed up the process. > >>> I shall be happy to see this text replaced by a more elaborate > >>> proposal once the relevant subcommittees catch up. > > Best Wishes > Lyndon > > >>> Marc > > > /--------------------------------------------------------\ > e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) > c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c > \--------------------------------------------------------/ > > > /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Mar 23 22:49:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03843; Tue, 23 Mar 93 22:49:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15784; Tue, 23 Mar 93 22:48:53 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 23 Mar 1993 22:48:51 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15774; Tue, 23 Mar 93 22:48:47 -0500 Message-Id: <9303240348.AA15774@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1885; Tue, 23 Mar 93 22:48:48 EST Date: Tue, 23 Mar 93 22:48:47 EST From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU \documentstyle[12pt]{article} \newcommand{\discuss}[1]{ \ \\ \ \\ {\small {\bf Discussion:} #1} \ \\ \ \\ } \newcommand{\missing}[1]{ \ \\ \ \\ {\small {\bf Missing:} #1} \\ \ \\ } \begin{document} \title{ Point to Point Communication } \author{Marc Snir \\ William Gropp and Ewing Lusk} \maketitle \section{Point to Point Communication} \subsection{Introduction} This section is a draft of the current proposal for point-to-point communication. It does not yet include a description of the Fortran 77 and C bindings. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. \discuss{ The following subsections of the introduction contain general information on the design of MPI procedures. The material should be moved to a general introduction for the entire document. The actual binding of these objects in Fortran and C will be discussed in the language binding subcommittee. } \subsection{Data Types} \subsubsection{Handle} MPI procedures use at various places {\em handles}. Handles are used to access opaque objects. Such object can be created, updated and destroyed only by by calling suitable MPI procedures, and providing the handle as parameter. Opaque objects hide from the user the internal representation used for various MPI objects, thus allowing to have similar calls in C and Fortran, allowing to overcome problems with the typing rules in these languages, and allowing for future extension of their functionality. Handles are of type {\tt void *} in C and of type {\tt integer} in Fortran. An opaque object can be {\em persistent} or {\em ephemeral}. A persistent object persists until destroyed by an explicit operation. An ephemeral object is good for a single use; thus an ephemeral object associated with a communication operation disappears once this operation is completed (or once this object is not needed anymore for the completion of the operation). An opaque object is created by a call to {\tt MPI\_CREATE}, and destroyed by a call to {\tt MPI\_FREE}. Additional MPI functions are available to create, access and update specific opaque objects. {\bf \ \\ MPI\_CREATE(handle, type, persistence)} \begin{description} \item[OUT handle] handle to object \item[IN type] state value that identifies the type of object to be created (e.g., {\tt MPI\_COMMUNICATION, MPI\_BUFFER, MPI\_CONTEXT}, etc.). \item[IN persistence] state value; either {\tt MPI\_PERSISTENT} or {\tt MPI\_EPHEMERAL}. \end{description} {\bf \ \\ MPI\_FREE(handle)} \begin{description} \item[IN handle] handle to object \end{description} An object can be destroyed only if there is no pending operation that is using this object; after successful return of the routine, the handle is undefined. {\bf \ \\ MPI\_ASSOCIATED(handle, type)} \begin{description} \item[IN handle] handle to object \item[OUT type] state \end{description} Returns the type of the object the handle is currently associated with, if such exists. Returns the special type {\tt MPI\_NULL} if the handle is not currently associated with any object. MPI may provide predefined opaque objects and predefined, static handles to these objects. Such objects may not be destroyed. \paragraph*{List of handles} An MPI call may need a parameter that is a {\em list of handles}. In C, such list will be a record with one component being the length of the list, the other components being an array of pointers. In Fortran, the list will be an array of integers, the first one of which is the length of the list. \discuss{ The mechanism for opaque objects used here follows the POSIX Fortran binding standard. An alternative choice is to have different type declarations for each type of opaque object. Then, opaque objects are created/destroyed like regular variables, rather than by MPI calls; they are still accessed and updated only via MPI functions. Some people proposed that {\tt list\_of\_handles} would not contain the list length, and the list length be passed as an additional, separate parameter. There are really three choices: \begin{enumerate} \item {\tt list\_of\_handles} contains a list length entry \item {\tt list\_of\_handles} is terminated with a special {\tt end-of-list} entry. \item {\tt list\_of\_handles} is not self-delimiting; and additional length parameter is needed whenever a list of handles is used. \end{enumerate} Comments? } \subsubsection{State} MPI procedures use at various places arguments with {\em state} types. The values of such data type are all identified by names, and no operation is defined on them. For example, the {\tt MPI\_CREATE} routine has a state type parameter with values {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}. An {\tt enumeration} declared in an included MPI.h file will be used in C for state datatypes. The Fortran 77 mechanism needs to be decided. \discuss{ Named integer constants can be used in Fortran 90, using the {\tt PARAMETER} mechanism. The constant declarations can be made available via an {\tt INCLUDE} file. Fortran 77 does not seem to offer any convenient mechanism. One possibility is to specify explicit integer values, and allow the use of named constants with those Fortran 77 compilers that support them conveniently. Another possibility is to use character strings, rather than integers. } \subsubsection{Named constants} MPI procedures sometimes assign a special meaning to a special value of a basic type parameter; e.g. {\tt tag} is an integer valued parameter of point-to-point communication operations, with a special {\tt DONTCARE} value. Such parameters will have a range of regular values, which is a proper subrange of the range of values of the corresponding basic type; special values (such as DONTCARE) will be outside the regular range. The range of regular values can be queried, and sometimes set, using environment inquiry or environment setting functions (Section~\ref{sec:inquiry}). The special values are provided by named constant, that are made available via an MPI.h include file in a C binding. \discuss{ Need to agree on a Fortran mechanism for named constants (see the discussion above). Implementers should detect, whenever possible, illegal uses of ``special values''. Thus, the use of the {\tt DONTCARE} value to tag a message sent will be flagged as an error. } \subsubsection{Choice} MPI functions sometimes use parameters with a {\em choice} (or union) data type. I.e., distinct calls to the same routine may pass by reference actual parameters of different types. The mechanism for providing such parameters will differ from language to language. In C, a formal parameter of type {\tt void *} will be used, with an actual pointer parameter. in Fortran, we shall cheat. \discuss{ The Fortran 77 standard specifies that the type of actual arguments need to agree with the type of dummy arguments; no construct equivalent to C pointers is available. Thus, it would seem that there is no standard conforming mechanism to support choice parameters. However, most Fortran compiler either don't check type consistency of calls to external routines, or support a special mechanism to link foreign (e.g., C) routines. I suggest that we accept this nonconformity with Fortran 77 standard. I.e., we accept that the same routine may be passed an actual parameter of a different type at distinct calls. Generic routines can be used in Fortran 90 to provide a standard conforming solution. This solution will be consistent with our nonstandard conforming Fortran 77 solution. } \subsection{Processes} An MPI program is executed by several autonomous processes that execute each their own code, in a MIMD style. The codes executed by each process need not be identical. The processes communicate via calls to MPI communication primitives. Typically, each processor executes in its own address space, although shared-memory implementations of MPI are possible. This document specifies the behavior of a parallel program assuming that only MPI calls are used for communication. The interaction of an MPI program with other possible means of communication (e.g., shared memory) is not specified. In particular, it is assumed that message buffers at distinct processors are disjoint. MPI does not specify the execution model for each process. A process can be sequential, or can be multithreaded, with threads possibly executing concurrently. Care has been taken to make MPI ``thread-safe'', by avoiding the use of implicit global states. The initial allocation of processes to an MPI computation and their binding to physical processors is not specified by the program itself. It is expected that vendors will provide mechanisms to do so either at load time or at run time. Such mechanisms will allow to specify the initial number of required processes, the code to be executed by each initial process, and the allocation of processes to processors. Also, the current proposal does not provide for dynamic creation or deletion of processes during program execution (total number of processes is fixed), although it is intended to be consistent with such extension. Finally, the current proposal does not specify a naming scheme for processes. We propose to always identify processes according to their relative rank in a context (group), so that, effectively, processes are identified by consecutive integers. Absolute, system-wide unique process id's are (will be) needed only if dynamic process creation is to be supported (in such eventuality we propose to use handles to opaque {\em process structures} for that purpose). \subsection{Contexts} \discuss{ This section contains a proposal for use of contexts that will subsume groups. It borrows heavily on the current group proposal. This proposal has will be discussed in the ``context'' subcommittee.} A {\bf context} consists of: \begin{itemize} \item A set of processes that currently belong to the context (possibly all processes, or a proper subset). \item A {\bf ranking} of the processes within that context, i.e., a numbering of the processes in that context from 0 to $n-1$, where $n$ is the number of processes in that context. \end{itemize} A process may belong to several contexts at the same time. Any interprocess communication occurs within a context, and messages sent within one context can be received only within the same context. A context is specified using a {\em context handle} (i.e., a handle to an opaque object that identifies a context). Context handles cannot be transferred for one process to another; they can be used only on the process where they where created. Follows examples of possible uses for contexts. \paragraph*{Loosely synchronous library call interface} Consider the case where a parallel application executes a ``parallel call'' to a library routine, i.e., where all processes transfer control to the library routine. If the library was developed separately, then one should beware of the possibility that the library code may receive by mistake messages send by the caller code, and vice-versa. To prevent such occurrence one might use a barrier synchronization before and after the parallel library call. Instead, one can allocate a different context to the library, thus preventing unwanted interference. Now, the transfer of control to the library need not be synchronized. \paragraph*{Functional decomposition and modular code development} Often, a parallel application is developed by integrating several distinct functional modules, that is each developed separately. Each module is a parallel program that runs on a dedicated set of processes, and the computation consists of phases where modules compute separately, intermixed with global phases where all processes communicate. It is convenient to allow each module to use its own private process numbering scheme, for the intramodule computation. This is achieved by using a private module context for intramodule computation, and a global context for intermodule communication. \paragraph*{Collective communication} MPI supports collective communication within dynamically created groups of processes. Each such group can be represented by a distinct context. This provides a simple mechanism to ensure that communication that pertains to collective communication within one group is not confused with collective communication within another group. \paragraph*{Lightweight gang scheduling} Consider an environment where processes are multithreaded. Contexts can be used to provide a mechanism whereby all processes are time-shared between several parallel executions, and can context switch from one parallel execution to another, in a loosely synchronous manner. A thread is allocated on each process to each parallel execution, and a different context is used to identify each parallel execution. Thus, traffic from one execution cannot be confused with traffic from another execution. The blocking and unblocking of threads due to communication events provide a ``lazy'' context switching mechanism. This can be extended to the case where the parallel executions are spanning distinct process subsets. (MPI does not require multithreaded processes.) \discuss{ A context handle might be implemented as a pointer to a structure that consists of context label (that is carried by messages sent within this context) and a context member table, that translates process ranks within a context to absolute addresses or to routing information. Of course, other implementations are possible, including implementations that do not require each context member to store a full list of the context members. Contexts can be used only on the process where they were created. Since the context carries information on the group of processes that belong to this context, a process can send a message within a context only to other processes that belong to that context. Thus, each process needs to keep track only of the contexts that where created at that process; the total number of contexts per process is likely to be small. The only difference I see between this current definition of context, which subsumes the group concept, and a pared down definition, if that I assume here that process numbering is relative to the context, rather then being global, thus requiring a context member table. I argue that this is not much added overhead, and gives much additional needed functionality. \begin{itemize} \item If a new context is created by copying a previous context, then one does not need a new member table; rather, one needs just a new context label and a new pointer to the same old context member table. This holds true, in particular, for contexts that include all processes. \item A context member table makes sure that a message is sent only to a process that can execute in the context of the message. The alternative mechanism, which is checking at reception, is less efficient, and requires that each context label be system-wide unique. This requires that, to the least, all processes in a context execute a collective agreement algorithm at the creation of this context. \item The use of relative addressing within each context is needed to support true modular development of subcomputations that execute on a subset of the processes. There is also a big advantage in using the same context construct for collective communications as well. \end{itemize} } \subsubsection{Context Operations} A global context {\bf ALL} is predefined. All processes belong to this context when computation starts. MPI does not specify how processes are initially ranked within the context ALL. It is expected that the start-up procedure used to initiate an MPI program (at load-time or run-time) will provide information or control on this initial ranking (e.g., by specifying that processes are ranked according to their pid's, or according to the physical addresses of the executing processors, or according to a numbering scheme specified at load time). \discuss{If we think of adding new processes at run-time, then {\tt ALL} conveys the wrong impression, since it is just the initial set of processes.} The following operations are available for creating new contexts. {\bf \ \\ MPI\_COPY\_CONTEXT(newcontext, context)} Create a new context that includes all processes in the old context. The rank of the processes in the previous context is preserved. The call must be executed by all processes in the old context. It is a blocking call: No call returns until all processes have called the function. The parameters are \begin{description} \item[OUT newcontext] handle to newly created context. The handle should not be associated with an object before the call. \item[IN context] handle to old context \end{description} \discuss{ I considered adding a string parameter, to provide a unique identifier to the next context. But, in an environment where processes are single threaded, this is not much help: Either all processes agree on the order they create new contexts, or the application deadlocks. A key may help in an environment where processes are multithreaded, to distinguish call from distinct threads of the same process; but it might be simpler to use a mutex algorithm at each process. {\bf Implementation note:} No communication is needed to create a new context, beyond a barrier synchronization; all processes can agree to use the same naming scheme for successive copies of the same context. Also, no new rank table is needed, just a new context label and a new pointer to the same old table. However, each context creation operation is likely to be a blocking collective operation that requires barrier synchronization. It is important to guarantee that no message is sent to a process in a new context before that process has joined the new context. Thus, a call to a context creation function will not return until all processes in the new context have called the function. (Of course, one can imagine ingenious methods to relax this requirement.) } {\bf \ \\ MPI\_NEW\_CONTEXT(newcontext, context, key, index)} \begin{description} \item[OUT newcontext] handle to newly created context at calling process. This handle should not be associated with an object before the call. \item[IN context] handle to old context \item[IN key] integer \item[IN index] integer \end{description} A new context is created for each distinct value of {\tt key}; this context is shared by all processes that made the call with this key value. Within each new context the processes are ranked according to the order of the {\tt index} values they provided; in case of ties, processes are ranked according to their rank in the old context. This call is blocking: No call returns until all processes in the old context executed the call. Particular uses of this function are: (i) Reordering processes: All processes provide the same {\tt key} value, and provide their index in the new order. (ii) Splitting a context into subcontexts, while preserving the old relative order among processes: All processes provide the same {\tt index} value, and provide a key identifying their new subcontext. {\bf \ \\ MPI\_RANK(rank, context)} \begin{description} \item[OUT rank] integer \item[IN context] context handle \end{description} Return the rank of the calling process within the specified context. {\bf \ \\ MPI\_SIZE(size, context)} \begin{description} \item[OUT size] integer \item[IN context] context handle \end{description} Return the number of processes that belong to the specified context. \paragraph*{Usage note} Use of contexts for libraries: Each library may provide an initialization routine that is to be called by all processes, and that generate a context for the use of that library. Use of contexts for functional decomposition: A harness program, running in the context {\tt ALL} generates a subcontext for each module and then starts the submodule within the corresponding context. (In a MIMD environment, ``harness program'' may be replaced by ``initial coordination code''.) Use of contexts for collective communication: A context is created for each group of processes where collective communication is to occur. Use of contexts for context-switching among several parallel executions: A preamble code is used to generate a different context for each execution; this preamble code needs to use a mutual exclusion protocol to make sure each thread claims the right context. \discuss{ A possible additional context creation function is {\bf MPI\_CREATE(newcontext, oldcontext, list\_of\_ranks)}, which picks an explicit list of members for a new context (identified by their rank in a previous context) and create that new context. This function must be called by all members of the list, and all must supply the same list. The processes are ranked in the new context according to their rank in the list. Dynamic process deletion can be handled by creating first a new context that does not include processes to be deleted, next having those processes terminate. Dynamic process creation can be handled by allowing {\em group extension}: A possible mechanism is a function of the form {\bf MPI\_EXTEND\_CONTEXT(context, number)} which adds {\tt number} processes to {\tt context}, with the new processes ranked above the old ones. Additional functionality is required for process creation/deletion in a heterogeneous environment, in order to control where new processes are allocated. None of this requires explicit, absolute, process names. However, if process handles are made explicit in MPI, then an additional function needed is {\bf MPI\_PROCESS(process, context, rank)}, which returns a handle to the process identified by the {\tt rank} and {\tt context} parameters. I oppose the idea of requiring dynamic process creation as part of MPI. Many implementers want to run MPI in an environment where processes are statically allocated at load-time. } \subsubsection{Error Handling} MPI provides the user with reliable message transmission: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, or other error conditions. In other words, MPI does not provide mechanisms for dealing with failures in the communication system. Where the MPI implementation is built on an unreliable underlying mechanism, then it is the job of the implementer of the MPI subsystem to insulate the user from this unreliability, or to reflect unrecoverable errors as global program failures. Similarly MPI itself provides no mechanisms for handling node failures. Of course, MPI programs may still be erroneous. A {\bf program error} can occur when an MPI call is called with an incorrect parameter (non-existing destination in a send operation, buffer too small in a receive operation, etc.) This type of error would occur in any implementation. In addition, a {\bf resource error} may occur when a program exceeds the amount of available system resources (number of pending messages, system buffers, etc.). The occurrence of this type of error depends on the amount of available resources in the system and the resource allocation mechanism used; this may differ from system to system. The recommended implementation profile provides several mechanisms to alleviate the portability problem this represents. One can also write {\bf safe} programs, that are not subject to resource errors. All MPI procedure calls return an error parameter that indicates successful completion of the operation, or the error condition that occurred, otherwise. The recommended implementation profile in a POSIX environment is for any MPI routine that encounters a recoverable error to store an error number in a global variable ({\em errno} in a C environment) and generate an {\em MPI error signal}, using a special signal value. The default handler for this signal terminates the execution of all involved processes, with a suitable error message being returned to the user. However, the user can provide his or her own signal handling routine. In particular, the user can specify a ``noop'' signal handler, thus relegating all error handling to the user code, using the error parameters returned by the MPI calls. MPI calls may initiate operations that continue asynchronously after the call returned. Thus, the operation may return with a code indicating successful completion, yet later cause an error exception to be raised. If there is a subsequent call that relates to the same operation (e.g., a call that verifies that an asynchronous operation has completed) then the error parameter associated with this call will be used to indicate the nature of the error. In a few cases, the error may occur after all calls that relate to the operation have completed, so that no error parameter can be used to indicate the nature of the error (e.g., an error in a send with the ready mode). In such cases, an error will be undetected, if the user disabled the MPI error signal. \discuss{ The alternative choice is to have fatal and non-fatal signals. One might want different signals for different modules. The details of such proposal need be elaborated in an appropriate ``profile'' subcommittee. } \subsection{Messages} A message consists of an {\em envelope} and {\em data}. \subsubsection{Data} The data part of a message consists of a sequence of values, each of a basic datatype in the host language. Thus, in Fortran, a message consists of a sequence of values that are each of type {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt LOGICAL}, or (length 1) {\tt CHARACTER}. A message may mix values of different types. \discuss{ May also need {\tt DOUBLE COMPLEX} in Fortran.} \subsubsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The rank the sending process \item[destination] The rank of the receiving process \item[tag] User defined \item[context] handle \end{description} The range of valid values for the {\bf source} and {\bf destination} fields is {\tt 0 ... n-1}, where {\tt n} is the number of processes in the specified context. The ranges of valid values for {\tt tag} is implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. {\tt Context} should be a context shared by both source and destination. The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. The actual mechanism used to associate an envelope with a message is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit, and need not be explicitly carried by a message. \subsection{Data Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is assembled from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is moved into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. A buffer consists of a sequence {\bf buffer components}. Each component consists of a sequence variables of the same basic type. There are three basic types of buffer components: \begin{description} \item[block] A sequence of contiguous values of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[datatype] Type of elements \end{description} \item[vector] A sequence of equally spaced and equally sized blocks of elements of the same basic type, specified by \begin{description} \item[start] Initial element \item[len] Number of elements ($\tt len \ge 0$) \item[stride] Number of elements between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk \le stride$) \item[datatype] Type of elements \end{description} Note that a constant stride becomes contiguous when {\tt stride = lenblk}. A vector buffer component can be an arbitrary submatrix of a two-dimensional matrix. \item[indexed] A sequence of elements of the same basic type, specified by \begin{description} \item[start] initial element \item[list\_of\_indices] List of displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} \end{description} For example, if a Fortran array is declared as {\tt double precision a(10)}, then the tuple {\tt $<$(a(3), (0,3,6,2), DOUBLE$>$} specifies a buffer component with entries {\tt a(3) a(6), a(9), a(5)}. \discuss{ Do we allow negative displacements? Do we allow entries to be repeated in an indexed buffer component? Do we allow different buffer components to overlap? Do we require in an vector buffer component that {\tt len} be a multiple of {\tt lenblk}? } A buffer is described by an opaque object accessed via a {\bf buffer handle}. Such object is created and destroyed via calls to {\tt MPI\_CREATE} and {\tt MPI\_FREE}. It is associated with successive buffer components by calling in succession one of the functions {\tt MPI\_ADD\_BLOCK}, {\tt MPI\_ADD\_VECTOR} or {\tt MPI\_ADD\_INDEX}, in order to append a component to the buffer associated with a buffer handle. A buffer object can be destroyed only if there is no pending communication operation using it. After a buffer object is destroyed the associated buffer handle is undefined. {\bf \ \\ MPI\_ADD\_BLOCK( buffer, start, len, datatype)} \\ Append a block component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_VEC( buffer, len, stride, lenblk, datatype )} \\ Append a vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN len] Number of elements (integer) \item[IN stride] Number of elements between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} {\bf \ \\ MPI\_ADD\_INDEX( buffer, start, list\_of\_indices, datatype)} \\ Append an indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[start] initial position for indexing (choice) \item[list\_of\_indices] list of relative indices of entries (array of integers) \item[IN datatype] datatype identifier (status) \end{description} Consider, for example, the following fragment of Fortran code \begin{verbatim} DOUBLE PRECISION A(10,20) INTEGER B, C(5,10) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, B, 1, MPI_INT) CALL MPI_ADD_VEC (BH, A(1,3), 11, 4, 2, MPI_DOUBLE) CALL MPI_ADD_INDEX(BH, C(3,7), (4,2,1), MPI_INT) \end{verbatim} Then the buffer associated with the handle {\tt BH} consists of the sequence of variables {\tt B, A(1,3), A(2,3), A(5,3), A(6,3), A(9,3), A(10,3), A(3,4), A(4,4), A(7,4), A(8,4), A(1,5), C(2,8), C(5,7), C(4,7)}. A message created from this buffer will consist of a sequence of one integer, followed by eleven double precision reals, followed by three integers. A buffer handle can be used for communication, even if it is not associated with any variables (i.e., even if it was not set by any {\tt MPI\_ADD\_xxx} call). Such handle is associated with an empty buffer, and a message created from it contains no data. \discuss{ The main modifications w.r.t. the proposal of Gropp and Lusk is measuring length in elements, rather than bytes. Seems more natural, since type is known. Also, object creation uses a generic function, rather than a function specific to buffer descriptors. As result, I gave up on the size parameter. This may not be such a loss: it is not clear that specifying a maximum length at buffer object creation is useful, since indices of arbitrary size may need to be stored in the object. } \subsubsection{Data Conversion} The types and the locations of the entries used to create a message is solely determined from the information in the buffer descriptor, using the storage association rules specified by the host language and its implementation; the type and the locations of these entries do not depend on the declaration for the corresponding variables in the calling program. It is not required that the data types specified in a buffer descriptor match the data types of the corresponding elements in the host program. However, in case of mismatches, the correspondence between entries in the host program and entries in a message created with the buffer descriptor may be implementation dependent. No data conversion occurs when data is moved from a sender buffer into a message. Consider the following fragment of Fortran code \begin{verbatim} REAL A(100) INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, A(1), 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, A(2), 1, MPI_INT) CALL MPI_ADD_BLOCK (BH, A(3), 1, MPI_LOGICAL) CALL MPI_ADD_BLOCK (BH, A(4), 1, MPI_COMPLEX) CALL MPI_ADD_BLOCK (BH, A(6), 1, MPI_DOUBLE) CALL MPI_ADD_BLOCK(BH, A(8), 4, MPI_CHAR) \end{verbatim} A message created from this buffer will consist of a sequence containing one real, one integer, one logical, one complex, one double, and four characters. No data conversion occurs when values are copied from the sender buffer to the message. Thus, the first entry in the message is a real number equal to {\tt A(1)}; the second entry is an integer number that happens to have the same binary representation as {\tt A(2)}; the third entry is a logical value that happens to have the same binary representation as {\tt A(3)}; the fourth entry in a complex number with a binary representation identical to {\tt A(4), A(5)}; the fifth entry is a double precision value with a binary representation identical to {\tt A(6), A(7)}; and if words have four bytes then the last four entries are bytes that make the binary representation of {\tt A(8)}, in the byte order of the executing machine. The correspondence between the first seven entries of the array {\tt A} and the first five entries of the message created from this buffer is determined by the rules of Fortran 77 on storage association: Each variable of type {\tt INTEGER}, {\tt REAL}, or {\tt LOGICAL} occupy one {\em numeric storage unit}; a variable of type {\tt DOUBLE} or {\tt COMPLEX} occupy two numeric storage units. Thus, the same correspondence will hold for any implementation. However, different implementations may have different binary encodings of integer, real and logical values, so that the actual values transferred by the message may differ. The correspondence between the entry {\tt A(8)} of the array, and the last four character entries in the message is implementation dependent, since the Fortran language does not specify a correspondence between character storage units and numeric storage units (an array of characters cannot be equivalenced with an array of integers). Different results may occur in big-endians or small-endians machines, or in 32 bit or 64 bit machines. The same holds, symmetrically, when a message is received. Entries are moved from the message into the receiver memory according to the information provided by the buffer descriptor, with no regard to the way the corresponding variables are declared in the receiving program. When data is moved in a homogeneous environment between nodes having the same architecture, then no data conversion occur at any point during data transfer. Assume, in the previous example, that an identically declared buffer descriptor is used to receive data into an identically declared array at the receiving process. Then, when these two nodes communicate, the values in the first eight entries of array {\tt A} of the sender will be copied into the first eight entries of array {\tt A} at the receiver (assuming that reals occupy the same storage as four bytes). In particular, in a homogeneous environment, it is possible to communicate using only character typed messages. When data is moved into a heterogeneous environment between nodes having distinct architectures, data conversion may occur during the transfer: Each entry is converted from the data representation used on the sending node to the data representation used in the receiving node. Consider the following fragment of Fortran code. \begin{verbatim} REAL X, Y CHARACTER (4) Z INTEGER BH ... CALL MPI_CREATE(BH, MPI_BUFFER, MPI_PERSISTENT) CALL MPI_ADD_BLOCK (BH, X, 1, MPI_REAL) CALL MPI_ADD_BLOCK (BH, Y, 4, MPI_CHAR) CALL MPI_ADD_BLOCK (BH, Z, 4, MPI_CHAR) \end{verbatim} Assume that the same arrays {\tt X, Y, Z} and buffer handle {\tt BH} are created at two processes A and B; this handle is used by A to create and send a message for B, by B to receive this message. Further assume that both A and B run on distinct nodes, with possibly different 32 bit architectures. Then {\tt X} at process B is assigned the value of {\tt X} at process A (up to rounding errors that may occur during conversion); {\tt Z} on process B is assigned the character string value of {\tt Z} on process A; if both nodes use ASCII encoding, then no conversion is required for the characters. On the other hand, variable {\tt Y} on process B may be allocated a value that differs from the value of variable {\tt Y} on process A. This may occur if the two nodes use a different byte sequence (little-endian vs big-endian), or use a different binary representation for reals (IEEE vs HEX). Thus, in order to ensure correct execution in a heterogeneous environment, it is important that the types of values in a message match the types of the corresponding values in the sending and in the receiving program. \subsection{Receive Criteria} The selection of a message by a receive operation is done uniquely according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\bf DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. The length of the received message must be less or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \subsection{Communication Mode} A sending operation can occur in one of two modes: \begin{description} \item[REGULAR] The send may start whether or not a matching receive has been posted. \item[READY] The send may start only if a matching receive has been posted. \end{description} A {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. In some systems, this will allow to optimize communication and avoid a hand-shaking operation that is otherwise required. \discuss{I deleted the symmetric ready receive. Will revive it if there is a requirement for it.} \subsection{Communication Objects} An opaque communication object identifies various properties of a communication operation, such as the buffer descriptor that is associated with it, its context, the tag and destination parameters to be used for a send, or the tag and source parameters to be used for a receive. In addition, this object stores information about the status of the last communication operation that was performed with this object. This object is accessed using a communication handle. One can consider communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). An object is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The communication object, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., an object needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all objects are freed at program termination, by default). The above scenario pertains to {\em persistent} objects. One can also create {\em ephemeral} objects. Such object persists only until the communication operation is completed, at which point it is destroyed. Thus, correct invocation of suboperations with an ephemeral object is {\bf INIT START COMPLETE}. A user may directly invokes these suboperations. This would allow to amortize the overhead of setting up a communication over many successive uses of the same handle, and allows to overlap communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation was invoked; the operation is {\bf completed} once the {\bf complete} suboperation completes. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsubsection{Communication Object Creation} An object for a send operation is created by a call to {\bf MPI\_INIT\_SEND}. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating an object for a receive operation. The creation of a communication object is a local operation that need not involve communication with a remote process. {\bf \ \\ MPI\_INIT\_SEND (handle, buffer\_handle, dest, tag, context, mode, persistence)} \\ Creates a send communication object. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to send buffer descriptor \item[IN dest] rank in context of destination (integer) \item[IN tag] user tag for messages sent with this handle (integer) \item[IN context] handle to context of messages sent with this handle \item[IN mode] send mode (state type, with two values: {\tt MPI\_REGULAR} and {\tt MPI\_READY}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} {\bf \ \\ MPI\_INIT\_RECV (handle, buffer\_handle, source, tag, context, persistence)} \\ Create a receive handle. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to receive buffer descriptor. \item[IN source] rank in context of source, or DONTCARE (integer). \item[IN tag] user tag for messages received with this handle, or DONTCARE (integer). \item[IN context] handle to context of messages received with this handle. \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ I have not included proposals for partially specified message handles, that some peoples seem to desire. I have merged all handle setup into one call. } \subsubsection{Communication Start} {\bf \ \\ MPI\_START(handle)} \begin{description} \item[IN handle] communication handle \end{description} The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started and until it is completed. A receiver should not access the receive buffer after a receive operation was started and until it is completed. A program that does not satisfy this condition is erroneous and its outcome is undetermined. \subsubsection{Communication Completion} \label{subsec:complete_ops} {\bf \ \\ MPI\_WAIT ( handle, return\_status\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_WAIT} returns when the send operation identified by {\tt handle} is complete. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer, or any other location that can be referenced by the send operation. However, it does not indicate that the message has been received; rather it may have been buffered by the communication subsystem. The completion of a receive operation indicates that the receiver is now free to access the locations in the receive buffer, which contain the received message, or any other location that can be referenced by the receive operation. It does not indicate that the matching send operation has completed. The call returns a handle to an opaque object that contains information on the completed operation -- the {\bf return status} object. {\bf \ \\ MPI\_STATUS (handle, flag, return\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT flag] logical \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_STATUS} returns {\tt flag=true} if the operation identified by {\tt handle} is complete, In such case, the return handle points to an opaque object that contains information on the completed information. It returns {\tt flag=false}, otherwise. In such case, the value of the return handle is undefined. Implementation notes: A call to {\tt MPI\_WAIT} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. The use of a blocking receive operation ({\tt MPI\_WAIT}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_STATUS}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_STATUS} is for that operation to return as soon as possible. However, if repeatedly called for an operation that is enabled, it must eventually succeed. The return status object for a send operation carries no information. The return status object for a receive operation carries information on the source, tag and length of the received message. These fields are required because the receive operation may have specified {\tt DONTCARE} in either source or tag field, and the message may have been shorter than the receive buffer. {\bf \ \\ MPI\_RETURN\_STAT( handle, len, source, tag)} \begin{description} \item[IN handle] handle to return status object \item[OUT len] difference between length of receive buffer and length of received message, in bytes. Thus, the value returned is zero if the received message matches the the receive buffer, positive if it is shorter (integer). \item[OUT source] rank of message sender in message context (integer). \item[OUT tag] tag of received message (integer). \end{description} \discuss{ I put the difference between message buffer and message length as the value returned, rather than length of received message, so that it might be easy to test for exact match. ``Accepted practice'' seems to be a byte count. The use of a return status object, rather than a list of parameters may simplify the use of MPI routines, if the values stored in the object are seldom checked. A predefined return status object should be provided, to ease programming.} \subsubsection{Multiple Completions} It is convenient to be able to wait for the completion of any or all the operations in a set, rather than having to wait for specific message. A call to {\tt MPI\_WAITANY} or {\tt MPI\_STATUSANY} can be used to wait for the completion of one out of several operations; a call to {\tt MPI\_WAITALL} can be used to wait for all pending operations in a list. {\bf \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_handle)} \\ Blocks until one of the operations associated with the communication handles in the array has completed. Returns the index of that handle in the array, and returns the status of that operation in the object associated with the return\_handle. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed. \end{description} The successful execution of {\tt MPI\_WAITANY(list\_of\_handles, index, return\_handle)} is equivalent to the successful execution of {\tt MPI\_WAIT(handle[i], return\_handle)}, where {\tt i} is the value returned by {\tt index} and {\tt handle[i]} is the {\tt i}-th handle in the list. If more then one operation is enabled and can terminate, one is arbitrarily chosen (subject to the restrictions on operation termination order, see Section~\ref{subsec:correct}). {\tt \ \\ MPI\_WAITANY ( list\_of\_handles, index, return\_status\_handle)} \\ is \begin{verbatim} {MPI_WAIT (handle[1], return_handle); index = 0} || ... || {MPI_WAIT (handle[n], return_handle); index = n-1} \end{verbatim} (``$||$'' indicates choice; one of the alternatives is chosen, nondeterministically.) {\bf \ \\ MPI\_STATUSANY ( list\_of\_handles, index, return\_handle)} \\ Causes either one or none of the operations associated with the communication handles to return. In the former case, it has the same return semantics as a call to {\tt MPI\_WAIT\_ANY}. In the later case, it returns a value of -1 in {\tt index} and {\tt return\_handle} is undefined. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT index] index of handle for operation that completed, or -1 if none completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed, if any; undefined when {\tt index = -1}. \end{description} {\bf MPI\_WAITALL(list\_of\_handles, list\_of\_return\_handles)} \\ Blocks until all communication operations associated with handles in the list complete, and return the status of all these operations. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[OUT list\_of\_return\_handles] Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. \end{description} \subsection{Blocking Communication} Blocking send and receive operations combine all communication suboperations into one call. The operation returns only when the communication completes and no communication object persists after the call completed. However, the buffer descriptor object needs be created ahead of the call. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf R \end{array} \right] \left[ \begin{array}{c} \bf SEND \\ \bf RECV \end{array} \right] \] The first letter (void or {\bf R}) indicates the start mode (regular or ready). {\bf \ \\ MPI\_SEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RSEND (buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} {\bf \ \\ MPI\_RECV(buffer\_handle, source, tag, context, return\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} {\bf Implementation note:} While these functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. \subsection{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call. They use ephemeral communication objects, so that the operation is completed, and the associated resources are freed, by using one of the functions {\tt MPI\_WAIT, MPI\_STATUS, MPI\_WAITANY, MPI\_STATUSANY}, or {\tt MPI\_WAITALL}. Here, too, a buffer object has to be created ahead of the communication initiation operation. We use the same naming convention as for blocking operations: a prefix of {\bf R} indicates the {\tt READY} mode. In addition, a prefix of {\bf I} is used to indicate {\em immediate} (i.e., nonblocking) execution. {\bf \ \\ MPI\_ISEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, REGULAR, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRSEND (handle, buffer\_handle, dest, tag, context)} \\ is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, READY, EPHEMERAL) MPI_START(handle) \end{verbatim} {\bf \ \\ MPI\_IRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, EPHEMERAL) MPI_START(handle) \end{verbatim} \subsection{Block Sending Operations} The most frequent type of buffer used is a contiguous buffer of numeric storage units, i.e., a contiguous buffer of words that may contain either INTEGER, REAL or LOGICAL values (in FORTRAN). In a homogeneous environment such messages can be used to send arbitrary sequences of contiguous items, where each item occupies an integer number of words. We specialize the functions in the two previous subsections to this case, thus avoiding the need for the creation of a buffer descriptor object. We use the same naming scheme used in the previous subsections, and append a {\bf B} in the function name, for {\tt BLOCK}. {\bf \ \\ MPI\_SENDB (start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_SEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_RSENDB (handle, start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RSEND (buffer_handle, len, dest, tag, context) \end{verbatim} {\bf MPI\_RECVB(start, len, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_RECV(buffer_handle, source, tag, context, return_status_handle) \end{verbatim} {\bf \ \\ MPI\_ISENDB (handle, start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_ISEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRSENDB (handle, start, len, dest, tag, context)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRSEND(handle, buffer_handle, dest, tag, context) \end{verbatim} {\bf \ \\ MPI\_IRECVB(handle, start, len, source, tag, context, return\_status\_handle)} \\ is \begin{verbatim} MPI_CREATE(buffer_handle, MPI_BUFFER, MPI_EPHEMERAL) MPI_ADD(buffer_handle, start, len, MPI_REAL) MPI_IRECV(handle, buffer_handle, source, tag, context) \end{verbatim} \discuss{ I use word count, rather than byte count. I believe word messages are much more prevalent than byte messages, and it's a blessing not to have to multiply by 4 for each message. Byte messages are still available the hard way. Also, word messages fit well with the Fortran numeric storage unit concept. If we are to add more functions, my next addition would be a vector send operation. } \subsection{Correctness} \label{subsec:correct} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsubsection{Order} MPI preserves the order of messages between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. Thus, if a two messages from the same source can satisfy a pending receive, the first message sent is accepted; if a message can satisfy two pending receives, the first receive posted is satisfied. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsubsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active communication objects. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item If a process executes an {\tt INIT} operation, then the operation eventually succeeds, or a {\em resource exception} occurs. The standard does not specify when a resource exception is allowed to occur. It is expected that an operational definition will be made available, in the form of test programs that have to execute with no resource exceptions. It is highly desirable to have generous bounds on the number of concurrently active communication objects each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active communication object. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation or, in the case of a send, because the matching receive successfully receives another message. {\bf An enabled operation either completes successfully or becomes permanently disabled.} I.e., an enabled operation either eventually succeeds, or becomes disabled (progress); and an operation cannot be enabled infinitely many times without ever succeeding (fairness). \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ranks 0 and 1) The following program is safe, and should always succeed. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_RECVB(recvbuf, len, 1, tag, context) CALL MPI_SENDB(sendbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVB(recvbuf, len, 0, tag, context) CALL MPI_SENDB(sendbuf, len, 0, tag, context) END IF \end{verbatim} The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed; the receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDB(sendbuf, len, 1, tag, context) CALL MPI_RECVB(recvbuf, len, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_SENDB(sendbuf, len, 0, tag, context) CALL MPI_RECVB(recvbuf, len, 0, tag, context) END IF \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer {\tt len} words of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \end{document} From owner-mpi-pt2pt@CS.UTK.EDU Tue Mar 23 22:54:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03906; Tue, 23 Mar 93 22:54:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15939; Tue, 23 Mar 93 22:53:30 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 23 Mar 1993 22:53:29 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15931; Tue, 23 Mar 93 22:53:28 -0500 Message-Id: <9303240353.AA15931@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1909; Tue, 23 Mar 93 22:53:29 EST Date: Tue, 23 Mar 93 22:49:11 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: new draft Reply-To: SNIR@watson.ibm.com Has few changes w.r.t. previous draft, mostly typos and clarifications, or alternative proposals acknowledged in dicussion parts. Thanks to James Cownie, Jon Flower, Rik Littlefield, Eric Barszcz, Tom Henderson, and whoever else I forgot, for corrections and suggestions. From owner-mpi-pt2pt@CS.UTK.EDU Thu Mar 25 12:41:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21419; Thu, 25 Mar 93 12:41:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27041; Thu, 25 Mar 93 12:41:13 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 25 Mar 1993 12:41:12 EST Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27031; Thu, 25 Mar 93 12:41:07 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA02886 (5.65c/IDA-1.4.4); Thu, 25 Mar 1993 12:41:03 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA18733; Thu, 25 Mar 93 17:40:59 GMT Date: Thu, 25 Mar 93 17:40:59 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303251740.AA18733@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA07372; Thu, 25 Mar 93 17:37:30 GMT To: snir@watson.ibm.com Cc: mpi-pt2pt@cs.utk.edu Cc: mpi-context@cs.utk.edu Subject: PT2PT draft (MARCH 23) Content-Length: 5131 Marc, [This is a reconstruction from memory of some mail I sent this morning, which seems to have disappeared... If anyone got the original they can test how good my memory is by comparing the two mails :-)] Here are some comments and queries on the current (23 March) point to point proposal. To avoid duplication of lots of text I will reference the proposal bye page number. Apologies to those of you who haven't printed it ! Page 3 : Discussion of options for Lists of Handles My preference is for option 3 (A separate length parameter). I prefer this over 1 (length as first element) because this leads to off by one errors, or a funny structure in C with an arbitrary sized array. I prefer it over 2 (a delimited list) because it avoids a whole pass over the list to work out its length when this is required. (e.g. if a copy needs to be made. cf the horrible C code char * copy = strcpy(malloc(strlen(string)+1), string); which accesses every byte of the string twice) Page 4: Discussion of named constants in FORTRAN. The F90 discussion should mention MODULEs (e.g. "... can be made available via an INCLUDE file, or MODULE.") PLEASE PLEASE PLEASE don't use character strings. This is guaranteed to be slow. I'd rather have literal values in place than strings ! Page 8 et seq: Contexts. I don't understand how to implement your context model in the way you seem to be describing. On page 8 you suggest that you can check the context at transmission, and avoid a collective agreement alogrothm at context creation. Similarly on page 10 in the implementation note you say "No communication is needed to create a new context beyond a barrier synchronisation". Surely you MUST check the context at the receiver (you seemed to agree with this in previous mail in MPI-context), therefore the sender and receiver must agree on the context name. (Or at least the sender must know the value by which the context is known at the receiver, which could be different for each receiver if you like [I don't !]) To achive this surely you need a group co-operation on context creation. Consider Process number i.e. Rank in ALL 0 1 Group 0 0 1 Group 1 1 0 The number in the table is the rank of the process in the group. Certainly when process 1 receives a message from zero he must know the group/context from which it was sent, since the result of the sender inquiry will be different depending on the group/context. Even if you only allow partitioning, I don't think you can avoid needing a cooperation, consider a scenario like this Process number Time 0 0 1 2 Time 1 0 | 1 2 Partition of all Time 2 1 | 2 Partition of subgroup Time 3 0 1 | 2 Partition of all Where the | denotes a partition of the group. Now with only a sequnce number process 0 has has been involved in 2 partitions, but process 1 (which should be in the same group) has been involved in 3. Ah, you can get the right answer by using a count of the partitions of the the parent group, and its depth in the tree. (Somehow...) Anyway it all falls apart when you let in the group construction by list (I believe) ! Page 12: I agree that we shouldn't REQUIRE dynamic process creation. Page 13: A global errno is the CLASSIC example of global data which gives threads a problem. Do we REALLY want to do it like this ? Page 14: Data types I think we should have both MPI_CHAR and MPI_BYTE. The difference being that in a heterogeneous environment MPI_CHAR would be translated according to the local character sets (e.g. ASCII -> EBCDIC), MPI_BYTE would be an 8 bit integer. What should we do about Fortran 90 KINDs ? Page 15: Data buffers. Vector buffer : Do we allow a zero or negative inter block stride ? Zero is a neat way to perform a fill operation... Page 17: Last line on the page... What is the (4,2,1) ? It's not Fortran 77. Looks a bit like an F90 array constructor ... Page 18: Data conversion You state "No data conversion occurs when data is moved from a sender buffer into a message". In a heterogeneous environment surely this is an implementation issue. The implementation should be free to translate at source, at dest, or on the moon if it likes. Page 36: You should delete "out of either process' address space." The example only requires buffering, it doesn't matter where the implementation chooses to do it. General question ================ Where is the store which is referenced by handles allocated ? At present it appears that all of this (even for ephemeral objects) is MPI system managed. I would very much like it to be possible to have the user manage this store, as she can often do it cheaper. (e.g. allocating ephemeral objects on the stack). Hmmm... came out more or less the same I think. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 10:09:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18013; Tue, 6 Apr 93 10:09:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22609; Tue, 6 Apr 93 10:07:28 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 10:07:26 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22587; Tue, 6 Apr 93 10:07:23 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA27788; Tue, 6 Apr 93 10:07:20 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA05994; Tue, 6 Apr 93 10:07:19 EDT Date: Tue, 6 Apr 93 10:07:19 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304061407.AA05994@b125.super.org> To: mpi-pt2pt@cs.utk.edu Subject: ready receive We have now met three times in Dallas and each time we have had a vote on ready receive. The votes have been: Jan: 13 Y 10 N Feb: 20 Y 12 N Apr: 10 Y 10 N Given these votes, I assume that ready receive remains in the draft by default. However, since the discussion was limited at the last meeting, I thought it would be useful to outline the arguments on this topic. That way, if we need to vote again, we can hopefully do it quickly and with the pros/cons at hand. Ready receive is a special type of send that tells the processor that an appropriate receive has already been posted for this message. On some systems this allows the sender to transmit the message without having to send system request/acknowledge packets (or significantly reduces the number). This effectively raises the bandwidth that the user achieves on the system. For example, on the Intel Delta ready receive (forced type) can increase the bandwidth by over 20%. An example of the use of ready receive is for a fan in/fan out collection/distribution of information. A standard algorithm would use a tree structure to recursively collect the information at the root node. Then the inverse tree is used to distribute the information back out to the other nodes in the tree. The modification with ready receive is to post an asynchronous receive for the fan out step at each node before the node sends the data to the next node during the fan in steps. Thus, when the fan out occurs, the sender knows that a receive is already posted and can use ready receive to decrease the overhead associated with this step. Some arguments for ready receive: - it reduces system overhead for message passing on some systems - variants of it are used on current generation systems (Intel NX) Some arguments against ready receive: - when improperly used a message can be lost. worse yet, this may only happen on a larger problem where synchronization is worse. this leads to users filling bug reports with the vendor. - it is yet another variant of send and adds more calls to MPI Now, IMHO: Lots of things in MPI will fail if used improperly by the user. Ready receive is not unique in this fashion. We left in asynchronous calls for several reasons but if the user does not properly issue a wait then he/she can get the wrong answer. I don't think we should penalize a knowledgeable user because some people may do the wrong thing. As for it being yet another call, I do have sympathize here. However, ready receive can simply do a standard send and that would comply with the MPI standard. Thus, on systems that don't care, the extra work is minimal. This is the same as the extra call for every regular call to get profiling. On the other hand, it can be a big win on some systems and there the implementor can spend the time to do it right. For these reasons I support ready receive (with a new name :-). I hope this summarizes the arguments but fear I have left something out. I welcome additions and comments. Steve From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 11:05:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19536; Tue, 6 Apr 93 11:05:10 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25029; Tue, 6 Apr 93 11:04:10 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 11:04:09 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25021; Tue, 6 Apr 93 11:04:06 -0400 Received: by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA18478; Tue, 6 Apr 93 08:04:05 PDT Date: Tue, 6 Apr 93 08:04:05 PDT From: nessett@ocfmail.ocf.llnl.gov (Danny Nessett) Message-Id: <9304061504.AA18478@ocfmail.ocf.llnl.gov> To: mpi-pt2pt@cs.utk.edu Subject: timeouts - a proposal The Problem ----------- There are a number of situations in which it is desirable/necessary to timeout communications supported by MPI. For example, if a process wishes to test for non-MPI I/O completion, such as non-blocking file I/O, keyboard/mouse interrupts or other asynchronous events, it cannot afford to block inside an MPI routine forever. Another example is when a long job (say an atmospheric code that runs for weeks), is submitted via batch and encounters either a hardware or software error. There will be no one who will be monitoring the progress of this job to terminate it. If there is no programming facility within MPI to detect such errors, it is conceivable that such a job could continue "running" consuming memory and perhaps CPU resources for many hours before corrective action is taken. In regards to the last problem, several people I have talked to have commented that it is not the job of MPI to detect and recover from either hardware errors or erroneous programs. I believe this is acceptable if we are designing a prototype or an academic toy, but in real production environments, such an approach will lead to much less acceptance of MPI as a standard programming interface. The Issues As I See Them ------------------------ There seems to be several issues in regards to timeout that are important. If I have missed some, I encourage others to point them out. The issues I see are: o People are reluctant to add a timeout parameter to every MPI call that could possibly block. Not only does it complicate the interface, it also requires those programmers who are not interested in timeout to think about it. o People want MPI to conform to standard practice. Current practice in the homogeneous MPP (NX, CMMD, etc) and workstation cluster world (PVM, P4, etc) is not to provide a timeout or to time out communications in the supporting library and crash. Current practice in the networking world (say sockets) is to provide for a timeout in the programming library. Consequently, there is a collision of approaches in current practice. o Some MPP manufacturers may find it difficult to provide timeouts. Either there is no hardware support in the machine's message passing mechanism or it is not easily used to provide the necessary high-performance message passing semantics. o Timeouts require the programmer to catch timeout errors in if statements or other conditional statements and provide the appropriate exception handling. Scientific programmers have a habit of not coding for errors. o Timeouts move the responsibility for failure atomicity to the application programmer. The underlying MPI library implementation cannot simply crash all of the parallel executables when a timeout occurs (at least in all cases). A Proposal ---------- Given the above issues, consider the following proposed change to the MPI interface to support timeouts. To each of the functions MPI_WAIT, MPI_WAITANY, and MPI_WAITALL add a timeout argument. This argument should be a double value representing the number of seconds to wait before timing out the WAIT call. If the conditions for the WAIT to return are not satisfied before the timeout occurs, the function returns a return value indicating timeout. In detail the following changes are proposed : MPI_WAIT( handle, return_status_handle, timeout) IN handle communication handle OUT return_handle handle that is associated with return status object. IN timeout timeout value (double) MPI_WAITANY( list_of_handles, index, return_handle, timeout) IN list_of_handles list of handles to communication objects OUT index index of handle for for operation that completed (integer) OUT return_handle handle that is associated with return status object. Set to return status of operation that completed. IN timeout timeout value (double) MPI_WAITALL( list_of_handles, list_of_return_handles, timeout) IN list_of_handles list of handles to communication objects OUT list_of_return_handles Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. IN timeout timeout value (double) MPI_WAIT, MPI_WAITANY and MPI_WAITALL return a status value as their return value. Thus, in order to use them properly, the programmer must check their return value in a conditional statement. A suitable value should represent a TIMEOUT and another value SUCCESS. Since supplying a timeout value of 0 transforms these wait functions into the equivalent of the MPI functions MPI_STATUS, MPI_STATUS_ANY and MPI_STATUS_ALL (NB: I didn't find the last function in the current draft, but I think it was added at the last meeting), these function calls can be eliminated from the MPI standard. Specifying a timeout value of -1 indicates a "wait forever" request (i.e., do not timeout the calls). Analysis of the proposal ------------------------ An examination of the proposal according to the issues described above follows: o Only three MPI calls take a timeout parameter. Consequently, the MPI interface may be used without knowledge of or concern with user supplied timeouts. o Use of timeouts in the WAIT family of function calls follows the specification of a timeout value in the UNIX I/O "select" call. Current practice for networked message passing is followed in this family, while current practice for MPP and cluster library message passing is followed in the other MPI calls. o Provision of a timeout in the WAIT family of calls for MPPs may or may not be problematic. Specific vendor advice is required to clarify this issue. o Scientific programmers not using the WAIT family of MPI calls (probably most) need not check return values. Use of the WAIT family is considered to be "advanced MPI programming" and so should only be used by "experts." o Failure atomicity can still be provided by the underlying MPI implementation as long as the programmer only uses blocking MPI calls. Use of non-blocking I/O is advanced use of MPI and so requires the programmer to exercise more skill in developing programs. From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 11:30:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19957; Tue, 6 Apr 93 11:30:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26307; Tue, 6 Apr 93 11:29:32 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 11:29:31 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26299; Tue, 6 Apr 93 11:29:28 -0400 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.16) id AA11026; Tue, 6 Apr 93 10:29:21 CDT Received: by teak18.cray.com id AA10963; 4.1/CRI-5.6; Tue, 6 Apr 93 10:29:20 CDT From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9304061529.AA10963@teak18.cray.com> Subject: Re: ready receive To: mpi-pt2pt@cs.utk.edu Date: Tue, 6 Apr 93 10:29:16 CDT In-Reply-To: <9304061407.AA05994@b125.super.org>; from "Steve Huss-Lederman" at Apr 6, 93 10:07 am X-Mailer: ELM [version 2.3 PL11b-CRI] Steve Huss-Lederman writes: > > Ready receive is a special type of send that tells the processor that > an appropriate receive has already been posted for this message. On > some systems this allows the sender to transmit the message without > having to send system request/acknowledge packets (or significantly > reduces the number). This effectively raises the bandwidth that the > user achieves on the system. For example, on the Intel Delta ready > receive (forced type) can increase the bandwidth by over 20%. I think this is a good description of ready receive. It matches the description given in each meeting. One thing that bothers me, and that is behind the reason I vote *against* the feature, is that while the proponents say: "some systems [can benefit]" they always mention the same example -- Intel systems. No other vendor's MPP or cluster system ever gets mentioned, nor any of the workstation packages. Will ready receive be of benefit to other current or soon-to-be systems or packages? I'll start. I do not anticipate that ready receive will be of benefit to the CRAY T3D (MPP system). As we all know, defining a standard requires balancing several opposing pressures. I can live with MPI including a familiar, commonly-used performance feature even if our architecture may not be able to use it optimally. But I dislike the idea of standardizing an uncommon feature that is only beneficial to one kind of system. (Especially a competitor! ;-) Thanks. - Peter Rigsbee Cray Research, Inc. par@cray.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 12:08:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21123; Tue, 6 Apr 93 12:08:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28443; Tue, 6 Apr 93 12:07:05 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 12:07:01 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28429; Tue, 6 Apr 93 12:06:58 -0400 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA18922 (5.65c/IDA-1.4.4 for ); Tue, 6 Apr 1993 11:06:56 -0500 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05217; Tue, 6 Apr 93 11:06:55 CDT Date: Tue, 6 Apr 93 11:06:55 CDT Message-Id: <9304061606.AA05217@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu In-Reply-To: Peter Rigsbee's message of Tue, 6 Apr 93 10:29:16 CDT <9304061529.AA10963@teak18.cray.com> Subject: ready receive ... I'll start. I do not anticipate that ready receive will be of benefit to the CRAY T3D (MPP system). Why? I've heard arguments that other systems operate in what I'll call receiver-probably-ready which may (or may not) reduce the need for receiver-ready, but I have not heard anyone offer EVIDENCE that receiver-ready will not help. The only evidence that we have is that the one vendor that has implemented it does see a significant performance improvement. Because of the strong assertion that receiver-ready makes about the communication, there is a some reason to believe that any system could take advantage of the additional semantic information, though it need not do so. Does anyone care to provide some evidence that ready-receive can not provide additional performance on other systems? Bill From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 13:10:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21974; Tue, 6 Apr 93 13:10:09 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01876; Tue, 6 Apr 93 13:09:09 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 13:09:08 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA01868; Tue, 6 Apr 93 13:09:05 -0400 Date: Tue, 6 Apr 93 18:08:47 BST Message-Id: <723.9304061708@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: ready receive To: par@teak.cray.com (Peter Rigsbee), mpi-pt2pt@cs.utk.edu In-Reply-To: Peter Rigsbee's message of Tue, 6 Apr 93 10:29:16 CDT Reply-To: lyndon@epcc.ed.ac.uk Peter Rigsbee writes: > Steve Huss-Lederman writes: > > > > Ready receive is a special type of send that tells the processor that > > an appropriate receive has already been posted for this message. On > > some systems this allows the sender to transmit the message without > > having to send system request/acknowledge packets (or significantly > > reduces the number). This effectively raises the bandwidth that the > > user achieves on the system. For example, on the Intel Delta ready > > receive (forced type) can increase the bandwidth by over 20%. > > I think this is a good description of ready receive. It matches the > description given in each meeting. > > One thing that bothers me, and that is behind the reason I vote *against* > the feature, is that while the proponents say: > "some systems [can benefit]" > they always mention the same example -- Intel systems. No other vendor's > MPP or cluster system ever gets mentioned, nor any of the workstation > packages. I agree with this. I want to make a strong proposal about the argument that a 20% improvement on the Delta is attainable, but more precisely about the Delta in relation to MPI. There is one Delta. There will never be another (conjecture, but true). We should place the Delta outside the set of machines which MPI does consider. It is of no future importance and is of trivial importance in the present. This means that arguments such as "You can't do XXX on the Delta" are not acceptable as arguments for excluding XXX from MPI, and that arguments such as "XXX goes ? faster than YYY on the Delta" is not acceptable as an argument for including XXX in MPI. > I'll start. I do not anticipate that ready receive will be of benefit to > the CRAY T3D (MPP system). I encourage vendors to add their statements. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 13:31:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22354; Tue, 6 Apr 93 13:31:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02816; Tue, 6 Apr 93 13:29:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 13:29:57 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02729; Tue, 6 Apr 93 13:28:38 -0400 Date: Tue, 6 Apr 93 18:28:33 BST Message-Id: <754.9304061728@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: interrupt driven To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Dear MPI Colleagues We have, reasonably in my opinion, decided not to provide interrupt driven receive/send (as in Intel hrecv/hsend) in MPI, or active messages. I have heard, can't recall from whom, that this carries the consequence that we should not define any feature of MPI which asks for the implementor of MPI to make use of interrupt driven receives, or active messages. This seems to me to be a bogus argument. In my mind the primary reason for not providing hrecv/hsend type facilities is that it is very difficult to standardise these facilities across different operating systems, particularly in terms of what the handler procedure is and is not allowed to do (e.g., can it do I/O, can it use MPI, ...). It seems to me clear that on many (all) machines of interest the system itself will be making use of similar facilities. Who would write MPI for CM-5 without using active messages? It seems to me that for example non blocking communications really do ask for use of interrupt/active messages. Is there agreement, disagreement, or what on this point? Please do let me know. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 16:01:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25987; Tue, 6 Apr 93 16:01:12 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11304; Tue, 6 Apr 93 15:59:27 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 15:59:26 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11296; Tue, 6 Apr 93 15:59:24 -0400 Received: from elephant (elephant.parasoft.com) by sampson.ccsf.caltech.edu with SMTP id AA19208 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Tue, 6 Apr 1993 12:59:22 -0700 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA09481; Tue, 6 Apr 93 12:50:01 PDT Received: by lion.parasoft (4.1/SMI-4.1) id AA01378; Tue, 6 Apr 93 12:50:00 PDT Date: Tue, 6 Apr 93 12:50:00 PDT From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9304061950.AA01378@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: Timeouts. I like the idea of a timeout too. I also spoke to a handful of people during the last meeting and got the distinct impression that this would be a useful addition. I also like the idea of collapsing the STATUS and WAIT calls together by using the timeout value. Not only does this reduce the number of routines but it also eliminates the need to remember which is which! Jon From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 16:07:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26079; Tue, 6 Apr 93 16:07:52 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11792; Tue, 6 Apr 93 16:07:20 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 16:07:18 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11784; Tue, 6 Apr 93 16:07:17 -0400 Received: from elephant (elephant.parasoft.com) by sampson.ccsf.caltech.edu with SMTP id AA19865 (5.65c/IDA-1.4.4 for mpi-pt2pt@cs.utk.edu); Tue, 6 Apr 1993 13:07:15 -0700 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA09487; Tue, 6 Apr 93 12:57:54 PDT Received: by lion.parasoft (4.1/SMI-4.1) id AA01393; Tue, 6 Apr 93 12:57:53 PDT Date: Tue, 6 Apr 93 12:57:53 PDT From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9304061957.AA01393@lion.parasoft> To: mpi-pt2pt@cs.utk.edu Subject: Ready receive. While I tend to agree that implementing ready-receive might well improve the performance of many vendors systems I would like to make the following observations which relate to my continued voting against this idea: a) I don't believe that vendors will actually go to the trouble of implementing this. All the discussions we've had in Dallas keep coming back to the "feature" of allowing ready-receive which is that it can actually be a no-op. To my mind this implies that vendors will take this approach wherever possible. (As suggested by Peter -- thank you Peter!!) b) If the best that can be expected from this is a 20% improvement in communication speed I (personally) don't care very much. I don't think that such an improvement is worth fighting for, especially since it only means a tiny overall improvement in an "efficient" program. Of these two I think point a) is where I have the biggest problem. Why add a feature to MPI that those outside our meetings might expect to be implemented, when we know, in our hearts, that it probably isn't going to be? At best it's misleading and at worst someone will go out of their way to use it (by re-ordering their communication logic, for example) only to find that it doesn't help!. I think this is the classic case of an "optional" feature that should be pushed to the back of the manual and left in peace. Jon From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 16:38:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26562; Tue, 6 Apr 93 16:38:49 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13254; Tue, 6 Apr 93 16:38:15 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 16:38:14 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13246; Tue, 6 Apr 93 16:38:11 -0400 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04157; Tue, 6 Apr 93 20:38:07 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA10433; Tue, 6 Apr 93 14:36:46 MDT Date: Tue, 6 Apr 93 14:36:46 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9304062036.AA10433@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Timeouts. > I like the idea of a timeout too. I also spoke to a handful of people > during the last meeting and got the distinct impression that this > would be a useful addition. > > I also like the idea of collapsing the STATUS and WAIT calls together > by using the timeout value. Not only does this reduce the number of > routines but it also eliminates the need to remember which is which! > > Jon I agree. At last, a proposal that REDUCES the number of routines! Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Apr 6 21:26:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29246; Tue, 6 Apr 93 21:26:02 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26159; Tue, 6 Apr 93 21:24:11 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 6 Apr 1993 21:24:10 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26151; Tue, 6 Apr 93 21:24:07 -0400 Received: from ernie.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA12606; Tue, 6 Apr 93 18:23:57 PDT Message-Id: <9304070123.AA12606@SSD.intel.com> To: lederman@b125.super.org (Steve Huss-Lederman) Cc: mpi-pt2pt@cs.utk.edu, prp@SSD.intel.com Subject: Re: ready receive In-Reply-To: Your message of "Tue, 06 Apr 93 10:07:19 EDT." <9304061407.AA05994@b125.super.org> Date: Tue, 06 Apr 93 18:23:57 -0700 From: prp@SSD.intel.com First a little history of "force types", which is our form of ready receive. It was first implemented on the iPSC/2 system to solve a flow control problem which has not been discussed here and might have been specific to the NX protocols. It had the side effect of being faster, and was used for that reason in applications. Because it was useful, it was propogated to the iPSC/860 and Delta systems where the performance effects were more pronounced. Since the Delta system is effectively a prototype of the Paragon system, we expect to continue to see a performance win. We would certainly implement ready receive in MPI if it is part of the standard. So, it will not be ignored in all implementations. Certainly vendor input is valuable in assessing the (near-term) benefit of including ready receive in the first version of MPI, but I think that users should make the decision to keep it in or not. After all, it is a very small burden for vendors; once implemented or ignored, it is done. The bigger question is, will it be used and provide benefit enough to make it worth having. Application writers should make the call. Thats the meat of my comments, as a postscript I will try to motivate the notion that ready receive is inherently advantageous in at least the simplest implementations. A compliant implementation of MPI should run this program. Otherwise, what is the point in having tags (I oversimplify, that is a separate argument): 1 2 async send to 2, tag=1 async send to 2, tag=2 async send to 2, tag=3 blocking receive from 1, tag=3 A simple implementation with no buffering must use a protocol where the receiver somehow lets the sender know which message can be sent. Otherwise, if the sender simply sent the first message (tag=1) and waited for a matching receive, the program would deadlock because the receiver is blocked waiting for the (tag=3) message and won't post any other receive until it comes in. For simple implementations with buffering assume the message sizes are very large (too large to buffer more than part of 1 message) and the same arguments hold. Only complex protocols which can partly buffer messages can get around the problem, and then one can simply increase the number of messages sent to defeat the buffering, although this might not fall within the spirit of the standard. So, the argument is that, in general, a compliant implementation must use a protocol where information travels from a receiving process back to the sender. This is more true for simple implementations than complex ones. Such a protocol incurs a minimum 2-trip penalty for at least some messages. Ready receive semantics eliminate the need for this step. With ready receive, the implementation does not have to use two or three trips, or does not have to figure out how many trips to use. It just sends the message, and the receiver gives an error or drops the message if there is no matching receive. Paul Pierce From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 7 11:25:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14072; Wed, 7 Apr 93 11:25:25 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07583; Wed, 7 Apr 93 11:24:09 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 7 Apr 1993 11:24:08 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07575; Wed, 7 Apr 93 11:24:06 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA29865; Wed, 7 Apr 93 08:24:02 PDT Message-Id: <9304071524.AA29865@ocfmail.ocf.llnl.gov> Date: Wed, 7 Apr 1993 08:32:56 -0800 To: mpi-pt2pt@cs.utk.edu From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: ready receive For those interested in understanding the reasoning behind ready receive and regular receive, note that the problems they solve were addressed over 20 years ago in a paper by David Walden in the April 1972 issue of CACM (pp. 221-230). This paper discusses (among other things) the notion of a rendezvous site for interprocess communication. Section 3 of the paper contains the relevant information. Dan From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 7 12:42:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17089; Wed, 7 Apr 93 12:42:36 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12113; Wed, 7 Apr 93 12:41:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 7 Apr 1993 12:41:57 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12105; Wed, 7 Apr 93 12:41:55 -0400 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.17) id AA20978; Wed, 7 Apr 93 11:41:51 CDT Received: by teak18.cray.com id AA12541; 4.1/CRI-5.6; Wed, 7 Apr 93 11:41:48 CDT From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9304071641.AA12541@teak18.cray.com> Subject: Re: ready receive To: mpi-pt2pt@cs.utk.edu Date: Wed, 7 Apr 93 11:41:42 CDT In-Reply-To: <9304070123.AA12606@SSD.intel.com>; from "prp@SSD.intel.com" at Apr 6, 93 6:23 pm X-Mailer: ELM [version 2.3 PL11b-CRI] prp@SSD.intel.com writes: > > Certainly vendor input is valuable in assessing the (near-term) benefit of > including ready receive in the first version of MPI, but I think that > users should make the decision to keep it in or not. After all, it is a > very small burden for vendors; once implemented or ignored, it is done. > The bigger question is, will it be used and provide benefit enough to > make it worth having. Application writers should make the call. I agree that it is trivial to write a ready-receive send that just looks like or invokes the regular send. But that may not be the only cost involved. To properly use ready receive, an application must ensure that the receive is ready before the message is sent. If the application would normally be written in such a way that this just "falls out" (that is, you can simply change a "send" to a "ready-receive send"), then I agree there is minimal impact involved to the application. But what if the application required special synchronization or was written in such a way that some additional overhead was required in order to properly use ready receive? If the cost of this extra overhead is less than the benefit from the ready receive, this would make sense. But now you have an application whose optimal performance *depends* on ready-receive send working faster than the regular send. If they port this to a system in which ready-receive send is nothing more than the regular send, then they are paying the cost of the extra overhead, but getting no benefit from it. And unfortunately, the people doing the port may not realize or remember this. So, one might argue, the vendor supplying that system ought to implement the ready-receive send! But if the underlying architecture doesn't lend itself to any benefit, what should the vendor do? Use a different mechanism so that the ready-receive send is faster than the (now slower) regular send? (This isn't hypothetical -- I've thought hard the last couple days about how I would implement a faster ready-receive on the CRAY T3D, and the only way I can think of how to do it is to use a protocol similar to what Paul describes. But I have a different protocol for the T3D that I believe is faster and just as reliable; one that is made possible by features of the T3D architecture that aren't in most other systems.) If ready-receive isn't in MPI, Intel still has the option of extending MPI to support a ready-receive send on their systems. It would be clear to a user choosing to use that extension that they are making a choice that will reduce portability. But by using ready-receive in any case, they are making this choice; this would make the decision explicit. In a sense, this is a general issue. People have argued for several MPI functions because they offer the enticing prospect of both portability and optimization. But this analysis is based on the design of the current architectures that people are familiar with, and in the long run may prove specious. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 7 13:49:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18831; Wed, 7 Apr 93 13:49:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15376; Wed, 7 Apr 93 13:48:39 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 7 Apr 1993 13:48:38 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15367; Wed, 7 Apr 93 13:48:34 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 7 Apr 93 10:48 PDT Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06918; Wed, 7 Apr 93 10:46:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA17058; Wed, 7 Apr 93 10:46:26 PDT Date: Wed, 7 Apr 93 10:46:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: Re: interrupt driven To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@carbon.pnl.gov, mpi-pt2pt@cs.utk.edu Message-Id: <9304071746.AA17058@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. > > I have heard, can't recall from whom, that this carries the consequence > that we should not define any feature of MPI which asks for the > implementor of MPI to make use of interrupt driven receives, or active > messages. This seems to me to be a bogus argument. I (Rik Littlefield) have regularly made arguments that are close enough to this to be worth clarifying or defending. Think of the MPI specification as being divided into two parts: 1) things you can implement with portable code, and 2) things you can't. My bias is to make the POTENTIALLY portable part of MPI as large as possible. (I emphasize "potentially" to make it clear that I am talking about interface definitions. I encourage implementors to do whatever they want internally.) As an example, consider groups and collective communication. All other factors being equal, I would strongly prefer a specification that permitted groups and collective communication, including group formation and disbanding, to be implemented using just the standardized MPI point-to-point capabilities. This sort of layerability potentially has several good effects. Three of these are 1) earlier widespread availability via porting, 2) better focusing of vendor effort on tuning the point-to-point layer, and 3) the capability to create portable MPI-plug-compatible implementations that are tuned for different requirements, e.g., debugging versus performance. Of course, all other factors are not equal. Strict layerability also has some potentially bad effects. Two of these (depending on where you draw the line between layers) are 1) the loss of integrated syntax (e.g., being able to use rank directly in the point-to-point calls), and 2) loss of functionality that is not supported by lower layers (e.g., asynchronous name servers). Whether the good outweighs the bad is a matter of opinion. The applications that I deal with now and anticipate dealing with over the lifetime of MPI-1 can be characterized as follows: . sensitive to performance, . intended to run on lots of platforms, . not sensitive to syntax, and . not needing name server support. Applications like these are best served by layerability. Others' mileage may differ. Lyndon continues: > In my mind the primary reason for not providing hrecv/hsend type > facilities is that it is very difficult to standardise these facilities > across different operating systems, particularly in terms of what the > handler procedure is and is not allowed to do (e.g., can it do I/O, can > it use MPI, ...). It seems to me clear that on many (all) machines of > interest the system itself will be making use of similar facilities. > Who would write MPI for CM-5 without using active messages? It seems to > me that for example non blocking communications really do ask for use of > interrupt/active messages. > > Is there agreement, disagreement, or what on this point? Please do let > me know. I agree that it is important to distinguish between exporting a standardized hrecv/hsend and and exporting a standardized asynchronous name server interface. The former seems hopeless; the latter is doable, but at the cost of potentially detracting from other features such as performance and prompt widespread availability. Thus, I do not agree that it is bogus to argue against requiring asynchronous servers et.al. in MPI-1. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 07:35:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08643; Thu, 8 Apr 93 07:35:03 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06247; Thu, 8 Apr 93 07:33:36 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 07:33:35 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06237; Thu, 8 Apr 93 07:33:32 -0400 Date: Thu, 8 Apr 93 12:33:28 BST Message-Id: <2110.9304081133@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-pt2pt: ready receiver To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI colleagues. This is the promised "ready-receiver" follow up to my "mpi-comm: various (long)" message. It is not so long :-) The first part follows through from the mentioned letter. The second part is more thoughts on ready receiver unrelated to the mentioned letter. o--------------------o I begin by describing ready-receiver communication for secure and insecure contexts. For an insecure context this is exactly as we have discussed in the meetings. For a secure context I suggest that the ready-reciever is exactly the same as an insecure-context, except that send gets an acknowledge from the receiver MPI saying whether the ready-receiver message is accepted or discarded (although I am happy to drop the suggested acknowledge for secure ready-receiver in which case it really does become less than secure.) o--------------------o Now I wish to make a more general comment about ready-receiver which is not in any way related to the secure/insecure context suggestion. My understanding of the argument for ready-receiver is that, if used in MPI user software, will provide performance benefit on some machines (where ready-receiver is implemented with essentially no protocol) and will not incur penalty on other machines (where ready-receiver is implemented as regular). I claim that on the latter machines software written with ready-receiver will actually go slower than equivalent software written without ready-receiver. The reason for this claim is quite simple, here is an example. Consider a "classic" case where ready-receiver can optimise a program, a fan-in and fan-out algorithm over a binary tree. Each node in the tree, which is not a leaf and not the root, executes the following in versions with ready-receive and without ready-receive. With Ready-receiver | Without Ready-receiver ------------------- | ---------------------- | regular receive from each child | regular receive from each child | process child messages | process child messages | start nbrecv from parent | | send to parent | send to parent | wait for nbrecv from parent | recv from parent | ready-receiver send to each child | send to each child When ready-receiver is implemented as send, the version on the left (written with ready-receiver) is slower than the version on the right (written without ready-receiver) since the node must perform more operations, and in particular may have to hit the system one more time, as the diagram clearly shows. The summary of this is that we can expect actual use of ready-receiver to sometimes make software go faster and sometimes make software go slower. I guess we beg the questions: A) How much faster is the left version than the right version when ready-receiver is not the same as regular? B) How much slower is the left version than the right version when ready-receiver is the same as regular? o--------------------o Comments, questions, (flames :-) please? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 08:57:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09502; Thu, 8 Apr 93 08:57:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10253; Thu, 8 Apr 93 08:56:32 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 08:56:31 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10245; Thu, 8 Apr 93 08:56:29 -0400 Date: Thu, 8 Apr 93 13:56:24 BST Message-Id: <2167.9304081256@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: timeouts - a proposal To: nessett@ocfmail.ocf.llnl.gov (Danny Nessett), mpi-pt2pt@cs.utk.edu In-Reply-To: Danny Nessett's message of Tue, 6 Apr 93 08:04:05 PDT Reply-To: lyndon@epcc.ed.ac.uk Regarding time-outs in the non-blocking wait calls, proposed by Danny Nesset. Some of the programmers either within, or assocaited with, EPCC have specifically asked for this capability. Exactly this facility is provided in the Meiko system CSN. I support this proposal, and agree that we need to seek vendor advice on whether they can sensible implement this, subject to a single amendment. The amendment is that the value to pass for an "infinite timeout" should be a named constant as opposed to the literal constant -1 (although I realise this is really a langauge binding issues). Vendors - please advise us. I should point out that I strongly disagree with the authors statement that nonblocking communication is advanced MPI usage that scientific programmers do not have to think about. In order for the programmer to write a portable (i.e. safe) program using the point-to-point communications the programmer will have to write programs using a mixture of blocking and nonblocking sends and receives. Of course, the programmer is free to write an unsafe program (which is potentailly simpler) using only blocking sends and receives. Let us hope that such programmers realise that such programs are not really portable. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 11:46:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12590; Thu, 8 Apr 93 11:46:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17500; Thu, 8 Apr 93 11:43:29 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 11:43:28 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17492; Thu, 8 Apr 93 11:43:25 -0400 Date: Thu, 8 Apr 93 16:43:20 BST Message-Id: <2373.9304081543@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-pt2pt: unsafe program? (short) To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues We have agreed that the following is an unsafe program. Process 1 Process 2 --------- --------- Send to Process 2 Tag 0 Send to Process 1 Tag 0 Receive from Process 2 Tag 0 Receive from Process 1 Tag 0 So it seems that the following is also an unsafe program. Process 1 Process 2 --------- --------- Send to Process 2 Tag 1 Receive from Process 1 Tag 2 Send to Process 2 Tag 2 Receive from Process 1 Tag 1 Does anyone disagree? Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 13:11:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13793; Thu, 8 Apr 93 13:11:30 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21714; Thu, 8 Apr 93 13:09:57 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 13:09:56 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21702; Thu, 8 Apr 93 13:09:54 -0400 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA04999; Thu, 8 Apr 93 13:09:52 -0400 Date: Thu, 8 Apr 93 13:09:52 -0400 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9304081709.AA04999@msr.EPM.ORNL.GOV> To: mpi-pt2pt@CS.UTK.EDU Subject: Re:mpi-pt2pt: unsafe program? (short) Lyndon "the prolific" writes: So it seems that the following is also an unsafe program. Process 1 Process 2 --------- --------- Send to Process 2 Tag 1 Receive from Process 1 Tag 2 Send to Process 2 Tag 2 Receive from Process 1 Tag 1 Does anyone disagree? >We will not serve our purpose if we declare every program >that requires buffering to be unsafe. >The vast majority of message-passing programs are written now >assuming some buffering. >Many programs suffer a big performance hit if they have to be >written with synchronous communication exclusivily. (to be "safe") Al Geist From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 14:54:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15960; Thu, 8 Apr 93 14:54:19 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27575; Thu, 8 Apr 93 14:53:43 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 14:53:42 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27567; Thu, 8 Apr 93 14:53:39 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA23393; Thu, 8 Apr 93 14:53:37 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA06629; Thu, 8 Apr 93 14:53:37 EDT Date: Thu, 8 Apr 93 14:53:37 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304081853.AA06629@b125.super.org> To: mpi-pt2pt@cs.utk.edu Subject: ready receive I am glad to see that ready receive is being discussed. I agree with the comments that if a lot of machines don't implement a separate ready receive then there is the chance that users will write programs assuming it is there and then possible get slower results. I can see this especially happening if ones adds synchronizations that would not otherwise be present. Lyndon raised the issue of the extra expense in the example of fan-in/fan-out if one uses forced type semantics but only gets a regular receive. I wrote a program to test this on the Delta since one can get ready receive (forced type) on that machine. All tests had these things in common. They did 10 trials to reduce the noise. The program was run twice to make sure it was reproducible. I gsync() before each trial and test to make sure each node starts at about the same time. After that I also send another long blocking message to really make sure they started about the same time. I think the code is correct :-) There were only 2 nodes to keep the case simple and clean. Node 0 can be thought of as the leaf and node 1 as the parent in Lyndon's example. Node 0 sends its information to node 1 who then returns it to node 0. With more nodes, node 1 would pass the data on and receive it again before it sends it back to node 0. The difference in the 3 cases is whether you use ready receive semantics and if you actually do the special type send. For the pseudo code below irecv is non-blocking, crecv is blocking and csend is blocking. forced (ready receive): node 0 node 1 ------ ------ irecv (tag=2) crecv(tag=1) csend(tag=1, reg) csend(tag=2, forced) wait(tag=2) regular: node 0 node 1 ------ ------ csend(tag=1, reg) crecv(tag=1) crecv (tag=2) csend(tag=2, regular) forced/regular (ready receive semantics without the special send): node 0 node 1 ------ ------ irecv (tag=2) crecv(tag=1) csend(tag=1, reg) csend(tag=2, regular) wait(tag=2) The times in seconds for various message sizes is: node # bytes forced regular reg/forced 1 100 2.946200e-04 2.283100e-04 2.955100e-04 0 100 4.382000e-04 3.820900e-04 4.548500e-04 1 500 4.092900e-04 4.174500e-04 4.086200e-04 0 500 6.818600e-04 6.904600e-04 6.711400e-04 1 2500 8.961900e-04 8.948600e-04 8.995300e-04 0 2500 1.155420e-03 1.288320e-03 1.223410e-03 1 12500 3.110890e-03 3.712720e-03 3.702740e-03 0 12500 3.494100e-03 4.035370e-03 3.953160e-03 1 62500 1.482880e-02 1.787127e-02 1.755866e-02 0 62500 1.542180e-02 1.819615e-02 1.786163e-02 1 312500 7.446586e-02 8.802909e-02 8.647874e-02 0 312500 7.508341e-02 8.834550e-02 8.674206e-02 1 1562500 3.713712e-01 4.369691e-01 4.320925e-01 0 1562500 3.719815e-01 4.372853e-01 4.323565e-01 I note a few general things about these results. Node 0 is generally slower because it is receiving the second message. There is a greater difference in time between the two nodes for smaller messages. Several general conclusions can be drawn: 1) forced type is faster than regular once the message is several thousand bytes long. The times are about 17% more for regular than forced for long messages. This is as advertised (remember that only one of the two messages was forced type). 2) Doing the irecv/wait is the same or faster than the "regular" send semantics even without doing a forced type send. This is the opposite of what was probably expected. With the caveat that these results are for the Delta, it seems that ready receive (forced type) semantics are better in general. Thus, if you don't have to add synchronization to your code and you are sending messages of sufficient length, then using ready receive is a win. If you add a synchronization to achieve ready receive then it could clearly slow your code down. I hope this answers the question Lyndon raised. There is no penalty (on the Delta) for solely using ready receive semantics. I clearly read all of his messages carefully and even spent several hours to answer one of his many questions. I wonder if I really get the beer :-) From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 15:24:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16483; Thu, 8 Apr 93 15:24:27 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28843; Thu, 8 Apr 93 15:23:20 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 15:23:19 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28835; Thu, 8 Apr 93 15:23:13 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 8 Apr 93 12:13 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA08178; Thu, 8 Apr 93 12:11:40 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA19332; Thu, 8 Apr 93 12:11:37 PDT Date: Thu, 8 Apr 93 12:11:37 PDT From: rj_littlefield@pnlg.pnl.gov Subject: buffering proposal to mpi-envir To: geist@msr.EPM.ORNL.GOV, gropp@mcs.anl.gov, mpi-envir@CS.UTK.EDU, mpi-pt2pt@CS.UTK.EDU Cc: d39135@carbon.pnl.gov Message-Id: <9304081911.AA19332@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@CS.UTK.EDU, mpi-envir@CS.UTK.EDU Al Geist writes: > We will not serve our purpose if we declare every program > that requires buffering to be unsafe. > The vast majority of message-passing programs are written now > assuming some buffering. > Many programs suffer a big performance hit if they have to be > written with synchronous communication exclusivily. (to be "safe") I agree, and I have previously informally proposed a way to get around this problem in a system-independent fashion. This seems like a point-to-point issue, but the minutes of the Feb.17-19 meeting say that it was deferred to the Environment subcommittee. So, I hereby officially offer this proposal to the Environment subcommittee, with cross-posting to both groups. PROPOSAL TO SUPPORT MPI MESSAGE BUFFERING IN USER-PROVIDED SPACE 1. INTERFACE An application program can optionally provide buffer space for MPI's use: e.g. MPI_USER_PROVIDES_BUFFER (len,buffer) where IN len is the length of 'buffer', in bytes. OUT buffer is a scratch array of len bytes for MPI's use in buffering messages 2. FUNCTIONALITY If the above routine is called, then for subsequent sends, MPI must *act as if* message data is being buffered by the sender. That is, user-provided buffer space may be consumed by outgoing messages, but not by incoming ones. 3. POSSIBLE IMPLEMENTATION One approach is for MPI to simply transform all blocking sends into . copy data to (application-provided) buffer space . issue non-blocking send from the buffer copy . return to application and . check completion on subsequent MPI call(s) No doubt many optimizations within MPI are possible -- the proposal just requires that MPI act as described here. 4. DISCUSSION The intended usage is that an application will provide buffer space once, immediately after MPI initialization. This is a minimalist proposal. An application may not be able to compute tight bounds on how much buffer space is actually required, but at least the proposed interface provides a single point-of-modification to aid in porting. No mechanism is proposed for canceling MPI access to the user-provided buffer. No mechanism is proposed to support multiple buffer spaces, although this might be convenient for some libraries. Allowing user-provided buffer space to be bound to a particular context (per Lyndon's recent suggestion regarding 'secure' communications) might meet both of these needs, but further study of that extension would be required. --Rik Littlefield ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 19:38:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20014; Thu, 8 Apr 93 19:38:57 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11689; Thu, 8 Apr 93 19:36:23 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 19:36:21 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11681; Thu, 8 Apr 93 19:36:18 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 8 Apr 93 16:27 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA08549; Thu, 8 Apr 93 16:25:04 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20464; Thu, 8 Apr 93 16:25:00 PDT Date: Thu, 8 Apr 93 16:25:00 PDT From: rj_littlefield@pnlg.pnl.gov Subject: receiver-ready send, performance data To: lederman@b125.super.org, mpi-pt2pt@cs.utk.edu Cc: d39135@carbon.pnl.gov Message-Id: <9304082325.AA20464@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu Steve Lederman writes: > Several general conclusions can be drawn: > > 1) forced type is faster than regular once the message is several > thousand bytes long. The times are about 17% more for regular than > forced for long messages. This is as advertised (remember that only > one of the two messages was forced type). > > 2) Doing the irecv/wait is the same or faster than the "regular" send > semantics even without doing a forced type send. This is the opposite > of what was probably expected. I think it's a bit optimistic to say that these conclusions are "general". The situation that Steve outlines is highly asymmetric -- process 0 does basically all the receiving while 1 does the sending. I've run literally hundreds of tests under symmetric conditions, both two-party and three-party transfers (i.e., exchange and shift), and I get results that are different from Steve's. I agree that forced types have higher bandwidth. However, the amount of the increase depends on whether you're doing two- or three-party transfers, as well as what machine you're on. On the Delta, 20% is a typical number; in some situations it's closer to 2X. On the iPSC/860, things are even more confusing -- overall, using forced types can be about the same or 2X faster than unforced, or something in between on average, and it's not even deterministic. Oh, and on the iPSC/860, breakeven for forced versus unforced is at 100 bytes even with the extra sync, versus several thousand bytes on the Delta. (A quick summary and discussion of my data is available in "Characterising and Tuning Communications Performance on the Touchstone DELTA and iPSC/860 (extended abstract)", available via anonymous ftp from delilah.ccsf.caltech.edu as file delta/documents/PNL_papers/IUGpaper.ps) I disagree with Steve's conclusion about irecv/wait being the same or faster than crecv. While that's usually true in his tests (but not always -- see bytes=100), I have consistently found the opposite. Steve continues: > ... If > you add a synchronization to achieve ready receive then it could > clearly slow your code down. > > ... There is no penalty > (on the Delta) for solely using ready receive semantics. These two statements seem contradictory. In the general case, you do have to add synchronization to use receiver-ready semantics, and that can slow you down if the messages are short. There ain't no such thing as a free lunch. My specific conclusion about timing is that On Intel systems, long messages are faster if forced and short messages aren't, and if you really care, your code had better be prepared to adapt. My general conclusion about timing is that General conclusions about timing are really dangerous to make. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Thu Apr 8 19:45:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20045; Thu, 8 Apr 93 19:45:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11921; Thu, 8 Apr 93 19:44:45 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Thu, 8 Apr 1993 19:44:44 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11913; Thu, 8 Apr 93 19:44:42 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Thu, 8 Apr 93 16:41 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA08563; Thu, 8 Apr 93 16:39:21 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20547; Thu, 8 Apr 93 16:39:16 PDT Date: Thu, 8 Apr 93 16:39:16 PDT From: rj_littlefield@pnlg.pnl.gov Subject: receiver-ready send, opinion To: lederman@b125.super.org, mpi-pt2pt@cs.utk.edu Cc: d39135@carbon.pnl.gov Message-Id: <9304082339.AA20547@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu I confess to having conflicting opinions about receiver-ready send. On the one hand, I like it a lot because it runs faster on some machines and it isn't much harder to use than regular send. If it's guaranteed safe with no extra sync, then I just use it. Otherwise I write adaptive codes that figure out whether to use it or not based on the message length. That strategy ports fine to systems where receiver-ready send doesn't help. On the other hand, I don't like the concept as it appears in MPI. I see the basic issue as Should MPI support a variety of protocols with different performance characteristics? In the specific case of receiver-ready send, we (the MPI committee) have answered "yes". But in the general case, we have answered "no" by specifying that any form of send can match with any form of receive. For example, an MPI implementation effectively can't use a "two-trip receiver pulls" (2TRP) protocol for exact-match messages, even though this would naturally handle many of the situations for which I now use receiver-ready send. This is because under the MPI spec, the sender doesn't know whether the receiver is going to do exact-match or wildcard, and using 2TRP with wildcard receive would be prohibitively expensive. It's true that the semantics of receiver-ready send allows MPI to use a more efficient protocol. But I suspect that an even greater improvement could be obtained if we required matching of send and receive modes, which we already voted is not acceptable. Perhaps receiver-ready send is just one of those borderline questions that won't be resolved except by some random process like flipping a coin or taking an MPI vote in the afternoon. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 10:28:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28034; Fri, 9 Apr 93 10:28:05 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21283; Fri, 9 Apr 93 10:25:49 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 10:25:47 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21268; Fri, 9 Apr 93 10:25:45 -0400 Date: Fri, 9 Apr 93 15:25:41 BST Message-Id: <3089.9304091425@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re:mpi-pt2pt: unsafe program? (short) To: geist@msr.EPM.ORNL.GOV (Al Geist), mpi-pt2pt@CS.UTK.EDU In-Reply-To: Al Geist's message of Thu, 8 Apr 93 13:09:52 -0400 Reply-To: lyndon@epcc.ed.ac.uk Al writes: > > Lyndon "the prolific" writes: > So it seems that the following is also an unsafe program. > > > Process 1 Process 2 > --------- --------- > > Send to Process 2 Tag 1 Receive from Process 1 Tag 2 > Send to Process 2 Tag 2 Receive from Process 1 Tag 1 > > Does anyone disagree? > > >We will not serve our purpose if we declare every program > >that requires buffering to be unsafe. I do not with to dispute your point. I asked a simple question, as I was unclear, and a colleague at EPCC asked me the question. Do I take your sentence as agreement that the program is unsafe? > >The vast majority of message-passing programs are written now > >assuming some buffering. I do not with to dispute your point. > >Many programs suffer a big performance hit if they have to be > >written with synchronous communication exclusivily. (to be "safe") > This seems to be a rather strong conjecture. I would be happy to receive significant justification. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 11:27:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29032; Fri, 9 Apr 93 11:27:53 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23903; Fri, 9 Apr 93 11:27:21 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 11:27:20 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23895; Fri, 9 Apr 93 11:27:18 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA24572; Fri, 9 Apr 93 08:27:01 PDT Message-Id: <9304091527.AA24572@ocfmail.ocf.llnl.gov> Date: Fri, 9 Apr 1993 08:36:04 -0800 To: lyndon@epcc.ed.ac.uk From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: timeouts - a proposal Cc: mpi-pt2pt@CS.UTK.EDU I agree with Lyndon's suggestion that MPI should use a named constant for representing an "infinite timeout". Whether or not non-blocking communication is advanced MPI usage rests on a subjective opinion of what is elementary and what is advanced. Dan From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 11:29:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29039; Fri, 9 Apr 93 11:29:22 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23929; Fri, 9 Apr 93 11:28:55 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 11:28:54 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23921; Fri, 9 Apr 93 11:28:52 -0400 Date: Fri, 9 Apr 93 16:28:49 BST Message-Id: <3185.9304091528@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: timeouts - a proposal To: nessett@ocfmail.ocf.llnl.gov (Dan Nessett), lyndon@epcc.ed.ac.uk In-Reply-To: Dan Nessett's message of Fri, 9 Apr 1993 08:36:04 -0800 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@CS.UTK.EDU Dan writes: > I agree with Lyndon's suggestion that MPI should use a named constant for > representing an "infinite timeout". Whether or not non-blocking > communication is advanced MPI usage rests on a subjective opinion of what > is elementary and what is advanced. > I concur! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 11:44:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29311; Fri, 9 Apr 93 11:44:17 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24508; Fri, 9 Apr 93 11:43:33 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 11:43:32 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24500; Fri, 9 Apr 93 11:43:30 -0400 Received: from newton.sandia.gov (newton.cs.sandia.gov) by cs.sandia.gov (4.1/SMI-4.1) id AA02495; Fri, 9 Apr 93 09:43:28 MDT Received: by newton.sandia.gov (5.57/Ultrix3.0-C) id AA01706; Fri, 9 Apr 93 09:44:23 -0600 Message-Id: <9304091544.AA01706@newton.sandia.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: mpi-pt2pt: unsafe program? (short) In-Reply-To: Your message of Thu, 08 Apr 93 16:43:20 -0000. <2373.9304081543@subnode.epcc.ed.ac.uk> Date: Fri, 09 Apr 93 09:44:22 MST From: mpsears@newton.cs.sandia.gov Lyndon asks if the program Process 1 Process 2 send(2, tag=1) recv(1, tag=2) send(2, tag=2) recv(1, tag=1) is unsafe. The answer is yes. With no buffering, process 2 cannot complete the first recv, since process 1 has not sent the first msg yet and process 1 cannot complete the first send, since process 2 has not posted a receive for it. Even if buffering is available, the program can still hang with large enough messages. The only safe way to write this program is for one of the processes to post its first operation in unblocked mode. The basic reason for the strong requirement on a safe program is that there is no good theoretical description of what a safe program is in the presence of buffering. One possible weakening would be to allow an MPI programmer to assume that when a message is sent by one process to another, the second process can detect if the message envelope has arrived (via probe), without blocking the first process. Therefore, zero length messages would always be safe. Al Geist's comments are appropriate. MPI is defined so that if you want to write a safe program you must assume no buffering, which requires a LOT of programmer effort, and like the ready receiver argument this programmer effort may indeed be counterproductive on machines which do have significant buffering. The existing constraint, while theoretically attractive, may be too limiting. We can legislate whatever we want in meetings and build a useless specification. Further opinions: 1) time-outs are a very bad idea. They add unlimited ways for things to go wrong. Remember that a timeout is a fixed threshold (unless you want to be able to change the timeout dynamically) and will always be chosen wrong in some situation. Your program will fail just when your boss or your customer is looking over your shoulder. There are usually better ways to accomplish the same thing without timeouts. For example, suppose the program has a process waiting for a task and there are no more tasks. Rather than using a time-out, a better way is to send a task request message that tells the waiting process to quit waiting for tasks. Timeouts are useful in network software which utilizes unreliable protocols, because they are the only way to detect certain classes of errors. MPI assumes reliable delivery of messages; therefore timeouts are inappropriate. 2) ready-receiver is a moderately bad idea, (this is my own position) for reasons already posted by others on the net. Mark Sears Sandia National Laboratories - ------- End of Unsent Draft ------- End of Forwarded Message From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 12:01:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AB29589; Fri, 9 Apr 93 12:01:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25338; Fri, 9 Apr 93 12:00:05 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 12:00:04 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25311; Fri, 9 Apr 93 12:00:01 -0400 Date: Fri, 9 Apr 93 16:59:58 BST Message-Id: <3212.9304091559@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-pt2pt: unsafe program? (short) To: mpsears@newton.cs.sandia.gov, mpi-pt2pt@cs.utk.edu In-Reply-To: mpsears@newton.cs.sandia.gov's message of Fri, 09 Apr 93 09:44:22 MST Reply-To: lyndon@epcc.ed.ac.uk Mark writes: > Al Geist's comments are appropriate. This point is well taken. > MPI is defined so that > if you want to write a safe program you must assume no buffering, > which requires a LOT of programmer effort, This is another subjective thing. Please believe me that I know and work with a lot of programmers with varying levels of experience and scientific backgrounds who find this way of programming is LITTLE effort indeed. > and like the ready > receiver argument this programmer effort may indeed be counterproductive > on machines which do have significant buffering. The existing > constraint, while theoretically attractive, may be too limiting. > We can legislate whatever we want in meetings and build a useless > specification. Understood. I am trying to do things to separate the secure communication I propose from the regular communication which provides buffering. Please do see my long message to mpi-comm of April 8 "Subject: mpi-comm: various (long)". I strongly encourage the inclusion of secure mode of communication in MPI. I also strongly encourage the inclusion of buffered mode and effort to allow the user management of how much buffering and where. I believe they are two ways of programming both of which MPI must support, rather than mandate one at the expense of the other, and that this is critical to the long term success of MPI as a de facto standard. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 12:24:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29791; Fri, 9 Apr 93 12:24:40 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26324; Fri, 9 Apr 93 12:24:14 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 12:24:13 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26316; Fri, 9 Apr 93 12:24:11 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA11445; Fri, 9 Apr 93 12:24:06 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA06800; Fri, 9 Apr 93 12:24:06 EDT Date: Fri, 9 Apr 93 12:24:06 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304091624.AA06800@b125.super.org> To: rj_littlefield@pnlg.pnl.gov Cc: mpi-pt2pt@cs.utk.edu, d39135@carbon.pnl.gov In-Reply-To: rj_littlefield@pnlg.pnl.gov's message of Thu, 8 Apr 93 16:39:16 PDT <9304082339.AA20547@sodium.pnl.gov> Subject: receiver-ready send, opinion I agree with Rik's comments that the performance does vary with the circumstance that you use forced type on the Delta. I mostly wrote that code to check Lyndon's question of the performance difference in a blocking receive vs. a non-blocking receive followed by a wait. It was not great. So it often comes down to (as Rik noted) the point that ready receive requires user skill if you want to be sure you get better performance and this will not easily port to other machines in many circumstances. The flip side is that "us" performance seekers want the extra performance and have routinely been willing to pay for it in more complex code. We do lots of ifs on message size, type of communication, etc. if we want optimal performance. I have concerns about typical users and ready receive. I have even thought about only making it available with message buffers but this may reduce performance. I would like to see ready receive but would be pleased by a better overall proposal on how to fit it into MPI. Steve From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 13:01:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00631; Fri, 9 Apr 93 13:01:48 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28117; Fri, 9 Apr 93 13:01:03 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 13:01:01 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28093; Fri, 9 Apr 93 13:00:56 -0400 Received: by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA25915; Fri, 9 Apr 93 10:00:54 PDT Date: Fri, 9 Apr 93 10:00:54 PDT From: nessett@ocfmail.ocf.llnl.gov (Danny Nessett) Message-Id: <9304091700.AA25915@ocfmail.ocf.llnl.gov> To: mpi-lang@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: cross-language support - a proposal (long) Cc: nessett@ocfmail.ocf.llnl.gov I am cross posting this message to the point-to-point list, since it contains a proposed modification to the point-to-point proposal. Those interested in point-to-point, but not language binding issues should skip to section 3, and evaluate whether the changes are acceptable. 1. The Problem -------------- At the recent MPI meeting in Dallas (31 March - 2 April), the language binding subcommittee proposed that the MPI standard make no provision for interoperability between MPI-based programs written in different programming languages. I objected to that suggestion and volunteered to develop a proposal that would allow such language interoperability. My objections were and are based on the following considerations : o MPI is a mulit-use standard, being developed for homogeneous multi- processors, homogeneous computer clusters and heterogeneous distributed systems. The objective is to provide a message passing interface that can be used in all of these environments. In addition, there is an objective to promote portability of MPI-based codes from one environment to another. Correct operation of MPI-based codes in a heterogeneous environment requires the MPI implementation to accommodate different formats for the communicated data. As a side effect, programs written in different programming languages will interoperate as long as the data types are conceptually the same. For example, REALs in Fortran and floats in C are conceptually the same. Unless there is a requirement that MPI support cross-language interoperability, some programs will work in heterogeneous environments, but not work in homogeneous environments. This will give the appearance that MPI is poorly designed. o MPI can be used in at least two different ways. It can be used for intercommunication between application executables that are aware (from the programmer's point of view) of each other's internal data structures and algorithms. It can also be used to provide services, much like a library provides services, in which the internal data structures and algorithms are not visible to the service client. In this second class of use, MPI acts as a generic service interface, supporting a wide variety of applications. Commonly, this generic service interface is known as client/server. Traditionally, libraries define their interface data structures and procedure prototypes in a specific programming language. Some libraries provide multiple interface definitions, one for each particular programming language they support. However, when using a generic service interface to access what is the equivalent of library services, this approach becomes problematic. The MPI interface is itself specified in terms of one or more programming language bindings. There is no useful way to specify another language binding for the client/server interface. Consequently, another approach is used for the client/server interface, which is programming language independent. Generally, server writers specify an interface in terms of a message parameter that represents the function or procedure to execute and in conjunction with a particular value of that parameter, specify a set of parameters that are the function's or procedure's arguments. Note that with a message passing based generic service interface, such as MPI (as opposed to a remote procedure call generic service interface), the service call is not constrained to obey function or procedure semantics, since the client is not obliged to block while the server provides the service. However, that is an aside. Since the server's interface is specified in terms of MPI data types, rather than in terms of the data types of the programming language used to write the server, there is a complete decoupling of the programming language used to write a server from the programming language used to write the client (and vice versa). The server can use one MPI language binding to offer service and the client can use another MPI language binding to access those services. An example may clarify this somewhat. Suppose I wish to write a server that provides a matrix manipulation service. I define my interface as follows : first parameter : operation to be performed (integer) - 1 = matrix sum 2 = matrix multiply 3 = matrix inversion 4 = matrix transposition 5 = return result second parameter: number of rows (integer) in matrix (or matrices) third parameter : number of columns (integer) in matrix (or matrices) fourth parameter: first matrix for all operations (single precision real array, row major order) fifth parameter : second matrix for operations 1 and 2 (single precision real array, row major order) For this interface, the operation and parameters will be specified as MPI data types. For a C or C++ language binding of MPI, these data types will be "int" and "float". For a Fortran language binding of MPI, these data types will be "INTEGER" and "REAL". This means there really is no way to prevent cross language interactions, other than to provide a "caveat emptor" warning to the programmer. Some programmers will not read the manual that closely; will implement a cross language service interface; will use it on various machines with no problems; and finally will curse the MPI implementors when it doesn't work on other machines. This will reduce confidence in MPI as a standard message passing interface. Note the property described in the previous bullet. In a heterogeneous environment, even if both client and server are written in the same language, the MPI implementation must accommodate conversion between different ranges of values for a particular type. For example, if one machine represents integers as 32 bit quantities, while another represents them as 64 bit quantities, sending an integer parameter from the second to the first requires a check to ensure the 64 bit value can be represented as a 32 bit value (e.g., that 100,000 represented as a 64 bit number fits into a 32 bit representation). This check also allows a client written in one language (say, C) to call a server written in another language (say, Fortran). 2. My view of the issues ------------------------ Below I summarize my view of the issues in regards to cross-language MPI service. I am sure that there are other issues that I have not thought of and welcome others to contribute them to the discussion. o As I have stated above, MPI is a multi-use standard being targeted for homogeneous multi-processors, homogeneous computer clusters and heterogeneous distributed systems. Vendor and user communities representing each of these environments are participating in the MPI standardization process. This invariably leads to a conflict of objectives. For example, the community most interested in homogeneous multi-processors does not want to sacrifice the performance of parallel programs running on these machines in order to accommodate heterogenous distributed processing or cross-language MPI support. Alternatively, those interested in heterogenous distributed systems don't want to constrain the MPI standard in such a way that severly limits its applicability in those environments. Those interested in homogeneous computer clusters are probably somewhere in between with their objectives. Consequently, if the MPI standard is to provide cross-language support, it should do so in a way that doesn't penalize the performance of programs running on homogeneous multi-processors, while at the same time minimizing the implementation effort for cross-language and heterogenous distributed system support. Also, programmers who intend to use only one language should not have to think about cross-language issues. o Cross-language support raises the question of translation between data values that are of a fundamentally different type (as opposed to being of a different "kind" in the Fortran 90 sense). For example, should the MPI standard provide support that would allow a Fortran program to send a "complex" value to a program written in C? (By this I mean support the transmission of the Fortran "complex" type to a C program, not the transmission of a "complex" value represented in some user specified way, like two real values). A cross-language facility may or may not allow this, depending on the degree of automation the standard is designed to provide. It is conceivable for a cross-language standard to support the communication of only those data types that are common to all of the target programming languages. There is also the issue of type coercion of transmitted values within a single language. For example C allows you to coerce a real value into an integer value when you assign a real value to an integer variable. Taking this approach would imply a sender could specify a real value in an MPI_ADD_BLOCK call, while the receiver specified an integer variable and expect MPI to perform the conversion. Allowing such coercion in the MPI interface would significantly increase its complexity, since the programmer would now have to specify not only the type being sent or received, but also the type being coerced. For this reason, I believe we have decided not to support such type coercion in MPI. o Translation between values of the same type (but perhaps different "kind") across languages requires knowledge of the mapping between a particular language type and its machine representation. For example, in Fortran 77 a REAL might map to an IEEE 32 and a DOUBLE PRECISION to an IEEE 64. In C a float might map to an IEEE 32, a double to an IEEE 64 and a long double to an IEEE 64. In Fortran 90 a REAL with a given kind parameter might map to either an IEEE 32 or an IEEE 64. Similar mapping of int, long, INTEGER (with different kind parameters in Fortran 90), char, CHARACTER, LOGICAL and COMPLEX to underlying representations is required to properly support cross-language interoperability. Some language types that have been suggested as MPI types would require definition in certain languages. For example, LOGICAL in Fortran has no direct type analog in C. Similarly, there is no COMPLEX data type in C. It would be possible to simply declare use of these MPI types in a program written in an incompatible programming language as erroneous or the standard could specify a mapping between these type and non-native types in the appropriate languages (e.g., LOGICAL could map to unsigned char and COMPLEX to a structure containing two reals). In a heterogeneous environment the mapping information can be more complex. On a SUN Workstation REALs, DOUBLE PRECISIONS, floats, doubles and long doubles may map as specified above. On a CRAY, however, a REAL may map to a CRAY 64, a DOUBLE PRECISION to a CRAY 64, a float to a CRAY 64, a double to a CRAY 64 and a long double to a CRAY 128. o Cross-language interoperability may require an MPI implementation that supports a particular language binding to be aware in some way of the existence of other language bindings. This may cause a problem when new MPI language bindings are developed. The features of MPI that support cross-language interoperability should allow the graceful integration of new language bindings into the standard. 3. A Proposal ------------- In order to support cross-language use of MPI, I propose the following modification to the point-to-point draft. The proposal is in two completely independent parts. One can be accepted without accepting the other. 3.1 First Part Modify the datatype parameter values of the MPI_ADD_... functions so that they come from the following set : MPI_REAL MPI_DOUBLEPRECISION MPI_COMPLEX MPI_INTEGER MPI_LOGICAL MPI_CHARACTER MPI_FLOAT MPI_DOUBLE MPI_LONGDOUBLE MPI_SHORT MPI_INT MPI_LONG MPI_CHAR (a character array) MPI_UCHAR MPI_BYTE These values for the datatype parameter are allowed in any MPI implementation irrespective of its language binding. They specify the type of the data that is being sent/received. Notice that the first group (i.e., MPI_REAL,..., MPI_CHARACTER) are Fortran types; the second group (MPI_FLOAT,...,MPI_UCHAR) are C (C++) types and MPI_BYTE is a language independent type. The use of a type from the first group in a Fortran program indicates that the data being sent/received is in Fortran format and consequently does not require translation. The use of a type from the first group in a C (C++) program indicates that the data being sent/received is in Fortran format and so may require translaton. Similarly, the use of a type from the second group in a Fortran program indicates the data being sent/received is in C (C++) format and so may require translation. The use of a type from the second group in a C program indicates the data being sent/received is in C (C++) format and so need not be translated. A client/server interface would specify its interface in terms of these MPI types. When a MPI_ADD_... function is called, the specified type would be used as the datatype parameter irregardless of the MPI language binding. Thus, if a parameter is specified as MPI_REAL, that datatype would be specified both in Fortran and C programs using the client/server interface. To properly pass messages, both the client and server must use the same datatype for the parameter. In order for the MPI implementation to take advantage of the provided type information in a cross-language communication, it must know which Fortran types map into which C types. Therefore, the standard should specify this information. Let me make the following strawman proposal : MPI_REAL maps to MPI_FLOAT; MPI_DOUBLEPRECISION maps to MPI_DOUBLE; MPI_COMPLEX maps either to a C struct or has no mapping, which would cause an error in a cross-language communication; MPI_INTEGER maps to MPI_INT; MPI_LOGICAL maps to MPI_UCHAR or has no mapping, which would cause an error; and MPI_CHARACTER maps to MPI_CHAR. There are three C types that have no natural analogs in Fortran : MPI_SHORT, MPI_LONG, and MPI_LONGDOUBLE. MPI_SHORT and MPI_LONG probably should map to MPI_INTEGER, since it is the only integer type available in Fortran 77. MPI_LONGDOUBLE probably should map to MPI_DOUBLEPRECISION. However, that means that MPI_DOUBLE and MPI_LONGDOUBLE map to the same Fortran data type. An alternate mapping would map MPI_DOUBLE to MPI_REAL and MPI_LONGDOUBLE to MPI_DOUBLEPRECISION. This requires discussion. In addition to a standard mapping between programming language types, a particular MPI implementation must know how the supported language types map into underlying machine representations. For example, it must know that REAL maps into IEEE 32, int maps into a 32 bit 2's complement value, etc. An MPI implementation would operate as follows. I categorize the implementations by environment in order to demonstrate properties of the previously discussed issues. 3.1.1 Homogeneous Mulit-processors and Computer Clusters -------------------------------------------------------- I describe the behavior of an MPI implementation with a Fortran language binding. The corresponding actions of an MPI implemenation with a C language binding should be obvious. 3.1.1.1 Send If the datatype parameter is in the Fortran set, block copy the data using the underlying system network. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type and use the per implementation machine representation mapping to decide the underlying representation for both the Fortran and C type. If they are the same, block copy the data using the underlying system network. If not, convert the data to the appropriate underlying representation for the C type and send the converted data. 3.1.1.2 Receive If the datatype parameter is in the Fortran set, the buffer in which the data arrived is in the proper format. Make it available to the MPI caller. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type and use the per implementation machine representation maping to decide the underlying representation for both the Fortran and C type. If they are the same, the buffer in which the data arrived is in the proper format. Make it available to the MPI caller. Otherwise, convert the buffer to the appropriate underlying representation and make it available to the MPI caller. A comment on implementation strategy. It is possible for both a send and receive operation to know before hand whether it needs to translate the buffer data or not (i.e., the above algorithms can be run before the actual machine transmission is sent or received). Consequently, in those situations in which translation is necessary, the implementation can supply an intermediate buffer in which to translate or from which to translate the data. By knowing both the Fortran and C types as well as their underlying representations, the implementation can precalculate the size of the translation buffer. 3.1.2 Heterogeneous Distributed Systems --------------------------------------- Supporting MPI communications in a heterogeneous distributed system is more complicated than in a homogeneous environment. Not only must differences in programming language data types be accommodated, differences in underlying machine represenations are also a concern. The exact algorithms to use when sending and receiving depend on the particular presentation-level protocol employed. Protocols like XDR and ASN.1 use a intermediate representation of data for transmission purposes. A protocol like NDR (used in OSF DCE) transmits the data in the format of the sender, placing the burden on the receiver to translate it. In the following discussion I finesse the protocol issue by using the generic description "call the off-machine protocol translation module" to mean execute the appropriate presentation protocol algorithm. Some implementations will combine the protocol translation activity with sending or receiving the data. 3.1.2.1 Send If the datatype parameter is in the Fortran set, determine the underlying machine represenation and call the off-machine protocol translation module to put it into the proper format. Send the result to the receiver. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type. Determine the Fortran type's underlying representation and call the off-machine protocol translation module to put it into the proper format. Send the result to the receiver. 3.1.2.2 Receive If the datatype parameter is in the Fortran set, determine the underlying machine representation and call the off-machine protocol translation module to put it into the proper format. Return this result to the MPI caller. If the datatype parameter is in the C set, use the type mapping to decide what is the corresponding Fortran type. Determine the Fortran type's underlying representation and call the off-machine protocol translation module to put it into the proper format. Return this result to the MPI caller. 3.2 Second Part Cross-language interoperability raises the issue of handling data types in one programming language that have no exact analog in another programming language. With the current suggested language bindings for MPI (i.e., Fortran 77, Fortran 90, C and C++), I think the following types fall into this category, one way or another : Fortran 77 LOGICAL COMPLEX Fortran 90 REAL(SELECTED_REAL_KIND(--,--)) INTEGER(SELECTED_INT_KIND(--)) LOGICAL COMPLEX COMPLEX(SELECTED_REAL_KIND(--,--)) C and C++ types included in the MPI standard have natural analogs in Fortran 77 and Fortran 90. There is at least two ways to handle these type mismatches. The simplest strategy is to generate an error when these types are referenced in an inappropriate language. This approach has the advantage of being simple to implement. It has the disadvantage that MPI-based programs written in one language without thought of using it from a program written in another language will likely use inappropriate types. The second strategy is to define analogs in each language for the non-common types. For example, LOGICAL in both Fortran 77 and Fortran 90 could map to unsigned char in C. COMPLEX in Fortran 77 and Fortran 90 could map to a struct with two float members in C. However, REAL, INT and COMPLEX with KIND parameter information cannot be handled in this way, since the "length" of the type is a machine dependent quantity. For example, on some machines INTEGER(SELECTED_INT_KIND(4)) might be equivalent to an int, while on other machines it is equivalent to a long (and not an int). Similar "length" problems exist with the REAL(SELECTED_REAL_KIND(--,--)) type. With these points in mind I propose the following. Map LOGICAL into unsigned char, COMPLEX into a C struct with two floats and disallow specification of the Fortran 90 types that specify a KIND parameter. Since Fortran 90 supports a KIND function call that the programmer can use to determine when INTEGER and INTERGER(SELECTED_INT_KIND(--)), REAL and REAL(SELECTED_REAL_KIND(--,--)), and COMPLEX and COMPLEX(SELECTED_REAL_KIND(--,--)) are equivalent, there is a work around for most cases. 4. Analysis of the Proposal --------------------------- Following is an analysis of the proposal according to the issues specified in section 2. o As described in section 3.1.1 the proposal can be implemented in such a way that it does not adversely impact the performance of either homogeneous multi-processors or homogeneous computer clusters. The programmer who writes programs in one language need never consider cross- language issues. Furthermore, the MPI types he/she would use would be natural for the programming language in which he/she is working. o The issue of mapping data types not found in one programming language into data types found in another is addressed by the second part, which covers mapping of data types by defining analagous types in the programming language from which a type is missing. Type coercion is not supported between types, since it is simple to first coerce the type (e.g., float to int) before sending it. It is not simple to support this kind of type coercion from within the MPI implementation. o The proposal handles translation of values of the same type by requiring the programmer to specify the type of data in the datatype parameter of the MPI_ADD_... functions. There is a limited amount of translation between types of different kinds due to the mapping required to associate a type in one language with a type in another language. This has the advantage of automating much of the work required to communicate values in a cross-language situation. However, there is also a possible disadvantage. In a heterogeneous distributed system the mapping could lead to the loss of information or to an error. For example, the suggested mapping associates a float with a REAL. On a CRAY a REAL is a 64-bit floating point number, while on a SUN a float is a 32-bit floating point. Transfering a REAL on a CRAY to a float on a SUN causes loss of precision and potentially could result in a translation error because the value on the CRAY cannot be represented by a 32-bit floating point. This problem also occurs on a single machine if a REAL maps to, say, an IEEE 64 and a float to an IEEE 32 (which doesn't seem likely). o Adding a new language binding to the MPI standard requires the definition of new MPI_ values and the mapping between these new values and the existing MPI_ values. Introduction of MPI implementations with support for the new language bindings can be accomplished gradually, since they will support the old MPI_, eventhough the old implementations do not support the new MPI_. As vendors upgrade their MPI implementations to conform to the new language binding standards, the new MPI_s can be used more and more until all useful implementations support them. Thus, the proposal supports the gradual introduction of new language bindings without requiring all implementations to immediately support them. From owner-mpi-pt2pt@CS.UTK.EDU Fri Apr 9 22:57:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11822; Fri, 9 Apr 93 22:57:54 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20891; Fri, 9 Apr 93 22:56:58 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Fri, 9 Apr 1993 22:56:56 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20883; Fri, 9 Apr 93 22:56:53 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Fri, 9 Apr 93 19:45 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA09922; Fri, 9 Apr 93 19:43:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA22459; Fri, 9 Apr 93 19:43:26 PDT Date: Fri, 9 Apr 93 19:43:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: proposal -- context and tag limits To: lyndon@epcc.ed.ac.uk, mpi-context@cs.utk.edu, mpsears@newton.cs.sandia.gov Cc: d39135@carbon.pnl.gov, gropp@mcs.anl.gov, mpi-collcomm@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-pt2pt@cs.utk.edu Message-Id: <9304100243.AA22459@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-envir@cs.utk.edu, mpi-context@cs.utk.edu, mpi-collcomm@cs.utk.edu Lyndon et.al. write: > ... This seems to say that the bit > length of the envelope is fixed to some number of bits and the more > fields we want to cram into the envelope the shorter the bit lengths of > fields must be. Is there a good reason why the bit length of the > envelope shoud be fixed in this fashion, or perhaps are you arguing > that the bit length of the envelope should be as short as possible? > > > This is a question vendors might answer: how many > > context values and tag values are you willing to support on future > > platforms and how many are you willing to back fit on existing ones? > > Yes, this would be a good question for the vendors indeed. > > VENDORS - PLEASE PLEASE PLEASE DO ADVISE US ON THIS ONE. I wonder what kind of useful advice vendors could really give us. Hardware support boils down to a question of getting faster performance in exchange for some relatively small resource limit. But in almost every case I can think of, such limits are made functionally transparent to the user by automatic fallback to some slower mechanism without the resource limit. Thus we have.. fixed size register sets with compilers that spill to memory, fixed size caches with automatic flush/reload from main memory, fixed size TLB's with cpu traps for TLB reload, fixed size physical memory with virtual memory support, and so on. The only counterexample that pops to mind is fixed-length numeric values, for which reasonably well established conventions exist. No such conventions currently exist regarding tag and context values. ============ PROPOSAL TO ENVIRONMENT COMMITTEE ============== The MPI specification should 1. require that all MPI implementations provide functional support for specified generous limits (e.g., 32 bits) on tag and context values, and 2. suggest that vendors provide a system-specific mechanism by which the user can optionally specify tag and context limits that the program agrees to abide by. Even the form of these limits should remain unspecified since they may vary from system to system. ======================== END PROPOSAL ======================== Further discussion... If a vendor wishes to provide hardware support to enhance performance for some stricter limits, and if some people are able and willing to write programs within those limits, that's great. Those people on those machines will be lark happy. If the performance increase is substantial, and I'm on one of those machines, and my program is simple enough, I'll probably be one of those people. However, I am not aware of any system on which generous limits could not be supported, albeit with some loss of performance compared to staying within the (currently hypothetical) hardware-supported limits. Everyone I know would MUCH prefer suboptimal performance over HAVING to rewrite applications to conform to varying and inconsistent hard limits. Yes, I recall the many arguments against mandating specific limits. But, I claim that those arguments are misdirected. They are based on analogy to things like word length and memory size, which I again note are subject to well established conventions and principles. (You can't run big programs on small machines, and we pretty much agree about what "big" and "small" mean.) In the case of context and tag values, such conventions do not exist, and a very wide range of conflicting limits have been discussed at various times and places. I believe that we will not meet our goal of portability if we do not specify usable limits on tag and context values. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-pt2pt@CS.UTK.EDU Mon Apr 12 14:54:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20748; Mon, 12 Apr 93 14:54:43 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28670; Mon, 12 Apr 93 14:53:50 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 12 Apr 1993 14:53:49 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28662; Mon, 12 Apr 93 14:53:45 -0400 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA02194; Mon, 12 Apr 93 18:53:21 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA19563; Mon, 12 Apr 93 12:54:38 MDT Date: Mon, 12 Apr 93 12:54:38 MDT From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9304121854.AA19563@nipmuc.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: mpi-pt2pt: unsafe program? (short) > Subject: Re: mpi-pt2pt: unsafe program? (short) > Date: Fri, 09 Apr 93 09:44:22 MST > From: mpsears@newton.cs.sandia.gov > > Lyndon asks if the program > > Process 1 Process 2 > > send(2, tag=1) recv(1, tag=2) > send(2, tag=2) recv(1, tag=1) > > is unsafe. The answer is yes. With no buffering, process 2 cannot > complete the first recv, since process 1 has not sent the first msg yet > and process 1 cannot complete the first send, since process 2 has > not posted a receive for it. Even if buffering is available, the program > can still hang with large enough messages. The only safe way to > write this program is for one of the processes to post its first > operation in unblocked mode. I have a similar question to Lyndon's. Given that I don't think MPI has yet specified a lower limit on the number of outstanding requests that may exist, is the unblocked version safe. Suppose you can only have one outstanding request. Again, clearly only one outstanding request is too few, but how many is enough? I would personally expect "most" programs written with non-blocking receives to work, but they may well be unsafe. In an algorithm where many boundary exchanges are happening there may be a lot of pending non-blocking receives and this is the problem I think of when the question of safe versus unsafe arises. Regards, Leslie Hart (hart@fsl.noaa.gov) From owner-mpi-pt2pt@CS.UTK.EDU Mon Apr 12 16:26:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22237; Mon, 12 Apr 93 16:26:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05506; Mon, 12 Apr 93 16:25:17 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 12 Apr 1993 16:25:16 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05497; Mon, 12 Apr 93 16:25:14 -0400 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.17) id AA25777; Mon, 12 Apr 93 15:25:11 CDT Received: by teak18.cray.com id AA18699; 4.1/CRI-5.6; Mon, 12 Apr 93 15:25:08 CDT From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9304122025.AA18699@teak18.cray.com> Subject: Re: mpi-pt2pt: unsafe program? (short) To: mpi-pt2pt@CS.UTK.EDU Date: Mon, 12 Apr 93 15:25:03 CDT In-Reply-To: <9304121854.AA19563@nipmuc.fsl.noaa.gov>; from "Leslie Hart" at Apr 12, 93 12:54 pm X-Mailer: ELM [version 2.3 PL11b-CRI] Leslie Hart writes: > > > Subject: Re: mpi-pt2pt: unsafe program? (short) > > Date: Fri, 09 Apr 93 09:44:22 MST > > From: mpsears@newton.cs.sandia.gov > > > > Lyndon asks if the program > > > > Process 1 Process 2 > > > > send(2, tag=1) recv(1, tag=2) > > send(2, tag=2) recv(1, tag=1) > > > > is unsafe. The answer is yes. With no buffering, process 2 cannot > > complete the first recv, since process 1 has not sent the first msg yet > > and process 1 cannot complete the first send, since process 2 has > > not posted a receive for it. Even if buffering is available, the program > > can still hang with large enough messages. The only safe way to > > write this program is for one of the processes to post its first > > operation in unblocked mode. > > I have a similar question to Lyndon's. Given that I don't think MPI has > yet specified a lower limit on the number of outstanding requests that may > exist, is the unblocked version safe. Suppose you can only have one > outstanding request. Again, clearly only one outstanding request is too > few, but how many is enough? I would personally expect "most" programs > written with non-blocking receives to work, but they may well be unsafe. > In an algorithm where many boundary exchanges are happening there may be > a lot of pending non-blocking receives and this is the problem I think of > when the question of safe versus unsafe arises. I find these discussions quite frustrating and mostly theoretical. I'd suggest we note these as issues and move on. Although opposed to recent political trends, I'd suggest we take the approach of letting the marketplace settle issues like this. The Fortran 77 standard does not specify a minimum number of COMMON blocks that a compiler must support. There is probably an implicit minimum of two (blank common and a named common block). So anyone could write and market a standard-conforming Fortran compiler that only lets you have two common blocks per compilation unit. How many users worry about this? How many users jam all their data into one named COMMON block so it will be compilable by such a brain-dead compiler? Not many. Why? Because such compilers aren't going to survive in the marketplace -- they will either be fixed (by increasing the limit), or they won't have any users. So someone trying to write portable Fortran uses a "reasonable" number of COMMON blocks (i.e., a number that is supported by all the systems of immediate interest) and figures he'll wait in case he hits a compiler limit with some new machine before worrying about it further. The same thing will be true of MPI. Who cares if someone writes an MPI implementation that supports at most one active send and receive in the entire system? If there is an official MPI test suite, the test suite will probably fail. Even if there isn't, there won't be any MPI applications that will run. Either that implementation will be fixed, or it will die on its own. The MPI standard doesn't have to solve this problem, the marketplace will. Taking a less extreme example, assume a new version of MPI is developed that can't run a code that runs on every other MPI system. I suggest that the developer of that code will point this out to the new MPI vendor. Ideally, the end result of this discussion is that the final application runs everywhere. Maybe this means the new MPI version is changed, or the application is changed, or both are changed. There may also be cases where the result is that the application is not ported to the new version of MPI, perhaps to the detriment of both vendor and developer, but maybe not. (For example, the new system may simply be too small to run that particular application. In this case, the presence of the smaller limit helped identify this situation.) The major problem with setting bounds for these things is one of coming to agreement. Using Leslie's question as an example, I suggest that we'd all agree that only one outstanding request is too few. And we'd probably agree that having 1,000,000 is more than enough. But trying to agree on a single minimum value (or even minimum order of magnitude) between 1 and 1,000,000 will take forever. And whatever we agreed on would be wrong for some situations, discouraging MPI's implementation. The other problem is that of units. I think this came up in the discussion of enquiry functions. IMO, we don't want MPI forcing a particular implementation. Let us consider the question of how do you define the maximum number of outstanding requests. One MPP vendor might want to define a value which means "number per CPU". Another would want to define his as "number per thread". A third uses a shared memory pool and wants to have a value which means "number for system as a whole". A vendor with a network implementation might want to support a different value for each machine in the network. Which unit(s) do you use for the definition? And how will this effect the other (and future) implementations? In summary, I think questions about the minimum maximum number of outstanding requests, or how many contexts and/or groups should be supported, or about whether or not a particular code sequence is safe for all sizes of data and all possible implementation styles are interesting questions that vendors and users should think about. But if MPI tries to choose and dictate the answers, it will take forever to finish and be less widely accepted and implemented. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Apr 14 18:20:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23206; Wed, 14 Apr 93 18:20:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00352; Wed, 14 Apr 93 18:19:23 -0400 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 14 Apr 1993 18:19:20 EDT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00344; Wed, 14 Apr 93 18:19:11 -0400 Message-Id: <9304142219.AA00344@CS.UTK.EDU> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1697; Wed, 14 Apr 93 18:19:05 EDT Date: Wed, 14 Apr 93 18:19:01 EDT From: "Marc Snir" To: MPI-PT2PT@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-V2.DVI.* %%Pages: 30 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 8 56 df34 DI50 DI<12E0B3B3B3B1 EAFFFCA30E4A73811C>I<131CB3B3B3B1EAFFFCA30E4A80811C>I<12E0B3A9031B73801C>I<12 E0B3A9031B75801C>I E /Fb 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EA E738EA0700A30D0E7E8E12>3 D E /Fc 2 36 df<1506A28116801501ED00E0B712F816F0C912 E0ED0180150316001506A2250E7E902A>33 D<1203B3ABEAC30CEAF33CEA3B70EA1FE0EA0780A2 EA03007E0E257D9C15>35 D E /Fd 2 63 df<127012F8A3127005057C840D>58 D<12E01278121EEA0780EA01E0EA0078131C130FEB03C0EB00F0143C140FEC03C0A2EC0F00143C 14F0EB03C0010FC7FC131C1378EA01E0EA0780001EC8FC127812E01A1A7C9723>62 D E /Fe 4 107 df0 D15 D<12C012F0123C120FEA03C0EA00F0133C130E6D7EEB 01E0EB0078141EEC0780EC01C0EC0780EC1E001478EB01E0EB0780010EC7FC133C13F0EA03C000 0FC8FC123C12F012C0C9FCA7B612C0A21A247C9C23>21 D<12C0B3B3A9022D7BA10D>106 D E /Ff 32 122 df<1238127C127EA2123E120E121E123C127C12F81260070B798416>44 D<127012F8A312700505788416>46 D48 DI<13E0487EA2 13B0A2EA03B8A31318EA071CA5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C0 13197F9816>65 DI<3801F180EA07FF5AEA1F0FEA 3C0712781303127000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C5AEA01F01119 7E9816>II<387FFFC0B5FC7EEA1C01A490C7FCA2131CA2EA1FFC A3EA1C1CA290C7FC14E0A5EA7FFFB5FC7E13197F9816>III73 D76 D<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3 EA38E3A21303A538FE0FE0A313197F9816>I<387E1FC038FF3FE0387F1FC0381D07001387A313 C7A2121CA213E7A31367A21377A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>III82 DI<387FFFE0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816>I< 387F07F038FF8FF8387F07F0381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F8151980 9816>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803A313E3EA39F3A213B300191300 A61313EA1B1BEA0F1EA2EA0E0E13197F9816>87 D<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0E A26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>89 D91 D93 D97 D101 D105 D110 D112 D<12035AA4EA7FFFB5FCA20007C7 FCA75BEB0380A2130713873803FF005BEA00F811177F9616>116 D<387F1FC038FF9FE0387F1F C0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0A35BA3485A1279127BEA7F80 6CC7FC123C131B7F9116>121 D E /Fg 53 122 df<90380FE0FE90387FF3FF9039F87F8F8039 01E0FF1F000313FEEA07C091387E0F00023EC7FCA5B612F0A22607C03EC7FCB0393FF9FFE0A221 20809F1E>11 D13 D<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123CA2 5AA412F85AA97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA114 >40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA01 E0A213C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<1238127C12FE12FFA2127F12 3B1203A21206A2120E120C12181270122008107C860F>44 DI<123812 7C12FEA3127C123807077C860F>I63 D<14E0A2497EA3497EA2497EA2497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F 48B57EA2EB800748486C7EA20006801401000E803AFFE01FFFE0A2231F7E9E28>65 DI<903807FC0290 383FFF0E9038FE03DE3903F000FE4848133E4848131E485A48C7120EA2481406127EA200FE1400 A7127E1506127F7E150C6C7E6C6C13186C6C13386C6C13703900FE01C090383FFF80903807FC00 1F1F7D9E26>IIII<903807FC0290383FFF0E9038FE03DE3903 F000FE4848133E4848131E485A48C7120EA2481406127EA200FE91C7FCA591387FFFE0A2007E90 3800FE00A2127F7EA26C7E6C7E6C7E3803F0013900FE03BE90383FFF1E903807FC06231F7D9E29 >III76 DIIIIII<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC 13F86CB4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813 F038FE01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140C A200E0140E00C01406A400001400B10007B512C0A21F1E7E9D24>IIIIII97 DIIIII<3801FC3C38 07FFFE380F07DEEA1E03003E13E0A5001E13C0380F0780EBFF00EA19FC0018C7FCA2121C381FFF 8014F06C13F8003F13FC387C007C0070133E00F0131EA30078133CA2383F01F8380FFFE0000113 00171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121FB0EAFFE0A20B217EA00E>I107 DI<3AFE0FE0 3F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8FFEA227147D 932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D>I<48B4FC00 0713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F03807FFC000 01130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA6143E147E147C EB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I<39FFE07F80 A2391F001C00380F8018A26C6C5AA26C6C5AA26C6C5AA213F900005B13FF6DC7FCA2133EA2131C A219147F931C>I<38FFE1FFA2380F80706C6C5A6D5A3803E180EA01F36CB4C7FC137E133E133F 497E136FEBC7C0380183E0380381F0380701F8380E00FC39FF81FF80A219147F931C>120 D<39FFE07F80A2391F001C00380F8018A26C6C5AA26C6C5AA26C6C5AA213F900005B13FF6DC7FC A2133EA2131CA21318A2EA783012FC5BEAC0E0EAE1C0EA7F80001EC8FC191D7F931C>I E /Fh 28 86 df<1230127812F81278127005057C840D>46 D<130C131C13FCEA0FF813381200 A41370A613E0A6EA01C0A61203EA7FFE12FF0F1E7C9D17>49 D<133FEBFFC03801C1E0380300F0 00061378A2000F137C1380A2EB00781206C712F814F0130114E0EB03C0EB0780EB0F00131C5B5B 13C0485A380300601206001C13C05AEA7FFFB51280A2161E7E9D17>I<137F3801FFC0380383E0 EA0701EB00F05A1301A2000013E0A2EB03C0EB0780EB0F0013FE13F8130E7F1480EB03C0A3EA30 07127812F8A238F00F8000C01300EA601EEA383CEA1FF8EA07E0141F7D9D17>II<00031370EBFFF014E01480EBFE000006C7FC A6EA0C7CEA0DFF380F8780EA0E03380C01C0A2000013E0A4387003C012F0A21480EAC00700E013 00EA600EEA383CEA1FF0EA0FC0141F7D9D17>I<133FEBFF803801C1E0EA038038070070120EA3 14E0120FEB81C0EBC3803807F700EA03FC120148B4FC380F3F80EA1C0F383807C01303EA7001A2 12E0A3EB0380007013005BEA3C1CEA1FF8EA07E0141F7D9D17>56 D<137E48B4FC380383803807 01C0120E121E001C13E0123CA21278A41303A2383807C0A2EA1C0FEA1FFB380FE380EA0007A214 00130E1260EAF01C5B485A5BEA7FC06CC7FC131F7C9D17>I<14181438A21478147C14FCA2EB01 BCA2EB033C143EEB061EA2130CA21318141F497EA21360EB7FFF90B5FCEBC00F3901800780A2EA 0300A21206A2001F14C039FFC07FFCA21E207E9F22>65 D<0007B5FC15C039003C01E015F09038 7800F8A515F0EBF001EC03E0EC07C0EC0F809038FFFE00ECFF803901E007C0EC03E0A21401A215 F0D803C013E01403A2EC07C0A2EC0F800007EB3F00387FFFFEB512F01D1F7E9E20>I<903803F8 0890380FFE1890383F073890387801F83801F000D803C013F0000714705B48C7FC5A121E003E14 60123C007C1400A45AA415C01278EC0180127C003CEB0300A26C13065C6C6C5A3807E0703801FF C06C6CC7FC1D217B9F21>I<0007B5FC15E039003C01F0EC00F849137C153C151EA3151F5BA648 48131E153EA3153C157C4848137815F0A2EC01E0EC03C0EC0F800007EB3F00387FFFFCB512E020 1F7E9E23>I<0007B512F8A238003C001578491338A5EC0C30EBF0181500A21438EBFFF8A23801 E0701430A2151815301400485A1560A215E0EC01C014030007130F007FB51280B6FC1D1F7E9E1F >I<0007B512F8A238003C001578491338A5EC0C30EBF0181500A21438EBFFF8A23801E0701430 A491C7FC485AA61207EA7FFE12FF1D1F7E9E1E>I<903801FC0490380FFF0C90383F039C903878 00FC5BD803E013785B4848133848C7FC5A121E003E1430A2481400A45AA2EC7FFCA2EC01E01278 EC03C0127C123CA27E6C13073907800F803803E0393801FFF039003F80001E217B9F24>I<3A07 FFC7FFC0A23A003C007800A2495BA649485AA490B5FCA23901E003C0A64848485AA60007130F39 7FFCFFF8485A221F7E9E22>I<3807FFE0A238003C00A25BA65BA6485AA6485AA61207EAFFFCA2 131F7F9E10>II<3A07FFE0FFE0A23A003C003E001538495B5DEC 01804AC7FC14065C495A5C5CEBF1F013F3EBF778EA01EC13F8497E13E080A248487EA36E7EA26E 7E0007497E397FFC1FFC00FF133F231F7E9E23>I<3807FFF0A2D8003CC7FCA25BA65BA6485AA3 EC0180A2EC0300EA03C0A25C1406140E141E0007137E387FFFFCB5FC191F7E9E1C>II<3A07FC03FFC0A23A003E007C001538016F1330A3EB6780 A2EB63C001C35BA2EBC1E0A2EBC0F0A2D801805B1478A2143CA33903001F80A2140FA3481307D8 0F8090C7FC387FF00312FF221F7E9E22>II<0007 B5FC15C039003C03E0EC01F0EB780015F8A59038F001F0A215E0EC03C0EC0F809038FFFE004813 F801E0C7FCA5485AA61207EA7FFC12FF1D1F7E9E1F>I<3807FFFC14FF39003C07C0EC03E0EB78 0115F0A415E0EBF00315C0EC0780EC1F00EBFFFC14F03801E038143C141CA2141EA23803C03EA4 1506A20007140C397FFC1F1800FFEB0FF8C7EA03E01F207E9E21>82 DI< 001FB512F8A2381E03C000381438EB078012300070141812601538153038C00F0000001400A513 1EA65BA6137C381FFFF05A1D1F7B9E21>I<39FFFC7FF8A23907800F80EC0700380F0006A6001E 5BA6485BA600385BA35C003C5B121C381E0180D80F07C7FCEA07FCEA01F81D20799E22>I E /Fi 73 126 df<127012F8B012701200A5127012F8A31270051C779B18>33 D38 D<137013F01201EA03C0EA0780EA0F00121E121C123C123812781270A212F05AA87E1270 A212781238123C121C121E7EEA0780EA03C0EA01F0120013700C24799F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370A813F013E0A2120113C012 0313801207EA0F00121E5A5A5A12600C247C9F18>II<136013 F0A7387FFFC0B512E0A26C13C03800F000A7136013147E9718>I<123C127E127FA3123F120F12 0E121E127C12F81270080C788518>II<127812FCA41278 0606778518>I48 DIII<131F5B5B137713F7EA01E713 C71203EA07871307120E121E123C1238127812F0B512F8A338000700A6EB7FF0EBFFF8EB7FF015 1C7F9B18>I<381FFF805AA20038C7FCA8EA3BFCEA3FFE7F383E0780381803C0380001E01300A2 126012F0130100E013C0EAF003387C0F80383FFF006C5AEA07F8131C7E9B18>I<137E48B4FC00 071380380FC3C0EA1F03123C383801800078C7FC1270A2EAF3F8EAEFFEB5FC38FE0F8038F803C0 EAF00114E01300A312701301007813C0EA3C03381E0F80380FFF006C5AEA03F8131C7E9B18>I< 12E0B512E0A338E003C0EB078038000F00130E131E131C133C5B1370A213F05BA212015BA31203 5BA7131D7E9C18>III<123C127EA4123C1200A81238127C127EA3123E120E121E123C12 7812F01260071A789318>59 D<387FFFC0B512E0A3C8FCA4B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C01C0A2 387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA87E0070 13E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>IIII<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7FC5AA5 EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C7E9B18 >I<387F07F038FF8FF8387F07F0381C01C0A9EA1FFFA3EA1C01AA387F07F038FF8FF8387F07F0 151C7F9B18>II<387F07 F038FF87F8387F07F0381C03C0EB078014005B131E5B133813785B121D7F121F13BC131CEA1E1E 130EEA1C0F7F1480130314C01301387F03F038FF87F8387F03F0151C7F9B18>75 DI<38FC01F8EAFE03A238 3B06E0A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03F8A3151C7F9B18>I<38 7E07F038FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A213311339A31319A2131D 130DA3EA7F07EAFF87EA7F03151C7F9B18>IIIII<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A290C7FC7E1278123FEA1FF0EA0FFE EA01FF38001F80EB03C0EB01E01300A2126012E0130100F013C0EAFC07B512801400EAE7FC131C 7E9B18>I<387FFFF8B5FCA238E07038A400001300B2EA07FFA3151C7F9B18>I<38FF83FEA3381C 0070B2001E13F0000E13E0EA0F013807C7C03803FF806C1300EA007C171C809B18>I<38FF07F8 A3381C01C0A4380E0380A4EA0F0700071300A4EA038EA4EA018C13DCA3EA00D813F8A21370151C 7F9B18>I<38FE03F8A338700070A36C13E0A513F8A2EA39DCA2001913C0A3138CEA1D8DA4000D 13801305EA0F07A2EA0E03151C7F9B18>I<387F8FE0139F138F380E0700120FEA070E138EEA03 9C13DCEA01F8A26C5AA2137013F07F120113DCEA039E138EEA070F7F000E13801303001E13C038 7F07F038FF8FF8387F07F0151C7F9B18>I<38FF07F8A3381C01C0EA1E03000E1380EA0F070007 1300A2EA038EA2EA01DCA213FC6C5AA21370A9EA01FC487E6C5A151C7F9B18>I91 D93 D95 D97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3E0EB80F0EB00701478000E1338A5120F1478 1470EB80F0EBC3E0EBFFC0000E138038067E00151C809B18>IIIII<3803F1F03807FFF85A381E1F30383C0F00EA3807A5EA3C0F EA1E1EEA1FFC485AEA3BF00038C7FC123CEA1FFF14C04813E0387801F038F00078481338A36C13 78007813F0EA7E03383FFFE0000F13803803FE00151F7F9318>I<127E12FE127E120EA5133FEB FF80000F13C0EBE1E013801300A2120EAA387FC3FC38FFE7FE387FC3FC171C809B18>II<12FEA3120EA5EB3F F0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C131EEA0E0FEB0780130314C038FFC7F8A3 151C7F9B18>107 DI<387DF1 F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBFBF80397F1F1F001914819318 >IIII<387F87E038FF9FF8EA7FBF3803FC78EBF030EBE0005BA35BA8 EA7FFEB5FC6C5A15147F9318>114 DI<487E 1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01FF6C1380EB7E0013197F9818 >I<387E07E0EAFE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13FE3801FCFC1714809318>I<38 7F8FF000FF13F8007F13F0381E03C0000E1380A338070700A3EA038EA4EA01DCA3EA00F8A21370 15147F9318>I<387F8FF0139F138F38070700138EEA039EEA01DC13F81200137013F07FEA01DC EA039E138EEA0707000F1380387F8FF000FF13F8007F13F015147F9318>120 D<387F8FF000FF13F8007F13F0380E01C0EB0380A21207EB0700A2EA03871386138EEA01CEA2EA 00CCA213DC1378A31370A313F05B1279EA7BC0EA7F806CC7FC121E151E7F9318>I 123 D<126012F0B3B012600424769F18>I<127CB47E7FEA07E01200AB7FEB7FC0EB3FE0A2EB7F C0EBF0005BAB1207B45A5B007CC7FC13247E9F18>I E /Fj 23 122 df<3801FFF0A238001E00 A35BA45BA45BA4485AA4485AA4485AA4EAFFF8A2141F7D9E12>73 D<48B4ECFFC0A2D8001F9038 01F800A2ED037801375C1506150CA20167EB19E0EB63801531A201C3495A15C3A2EC8183D80183 EB8780EC8307A214862703038C0FC7FCA21498EB01D80006EBF01EA214E0000E13C03AFFE1C3FF E0D9C1835B2A1F7D9E29>77 D<48B5128015E039001E00F01578153849133CA4491378A2157015 E0EBF001EC078090B5120014F8D801E0C7FCA4485AA4485AA4EAFFF85B1E1F7D9E1F>80 D97 D<137E48B4FC38038380EA0F07121E001C1300EA3C02 48C7FCA35AA5EA70021307EA381EEA1FF8EA07E011147C9315>99 D<1478EB03F814F0EB0070A3 14E0A4EB01C0A213F1EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133C EA707C3878FCC0EA3FCF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB03 00EA780EEA7FFC13E000F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I<14F8 EB01FCEB03BC143CEB07181400A2130EA53801FFE0A238001C00A35BA55BA65BA4485AA35B1233 127B00F3C7FC127E123C1629829F0E>III<136013F0A213E01300A7120FEA1F80123113C0EA6380A212C3EA0700A3120EA3EA1C30 1360A2EA38C01218EA1F80EA0F000C1F7D9E0E>I108 D<391E07C0F8393F1FE1FC3933 B8730E3963E07607EBC03CEB803800C7EB780E38070070A3000E495AA3ED3860261C01C013C0A2 ED718015313A3803803F00D81801131E23147D9325>I<381E07C0383F1FE03833B870EA63E0EB C038138000C71370EA0700A3000E13E0A3EB01C3001C13C6A2EB038C1301003813F8381800F018 147D931A>I<137C48B4FC38038380380F01C0121E001C13E0123C1278A338F003C0A3EB078014 00EA700F131EEA3838EA1FF0EA07C013147C9317>I<3803C1E03807E7F838067E3C380C7C1CEB 781E1370EA18E01200A33801C03CA314781203147014E0EBE3C038077F80EB1E0090C7FCA2120E A45AEAFFC0A2171D809317>I114 D<13FCEA03FEEA0707EA0E0FA2130EEA1E00EA0F8013 F0EA07F8EA03FCEA003E131E1270EAF01CA2EAE038EA6070EA3FE0EA1F8010147D9313>II<000F1360381F8070003113E013C0EA6380A238C381C0EA0701A3380E0380A314 8CEB0718A2130FEB1F303807F3F03803E1E016147D9318>I<380F01C0EA1F83003113E013C1EA 6380A200C313C0EA0700A3380E0180A3EB0300A21306A2EA0F0CEA07F8EA01E013147D9315>I< 38078780380FCFC03818F8E0EA3070EA6071A238C0E1C03800E000A3485AA30071136038F380C0 A238E3818038C7C300EA7CFEEA387C13147D9315>120 D<000F1360381F8070003113E013C0EA 6380A238C381C0EA0701A3380E0380A4EB0700A25B5BEA07FEEA03EEEA000EA25B12785BEA7070 EA60E0EA3FC06CC7FC141D7D9316>I E /Fk 40 122 df<1238127C12FEA3127C123807077C86 10>46 D<13181378EA01F812FFA21201B3A7387FFFE0A213207C9F1C>49 DII<14E013011303A21307130F131FA2 1337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA2380007E0A7EBFF FEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC00018C7FCA4EA19FC EA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F8386007E0003013 C0381C1F80380FFF00EA03F815207D9F1C>I<13FE3803FFC0380703E0380E00F05A1478123C12 3E123F1380EBE0F0381FF9E0EBFFC06C13806C13C06C13E04813F0381E7FF8383C1FFCEA7807EB 01FEEAF000143E141EA2141C7E007813387E381F01F0380FFFC00001130017207E9F1C>56 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB300FA201607F 140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A225227EA12A >65 D67 DIII72 D77 DIII<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400 B4FC13E06CB4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA3 6C131EA26C133C12FCB413F838C7FFE00080138018227DA11F>83 D<007FB61280A2397E03F80F 00781407007014030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127 >I97 DI< EBFF80000713E0380F83F0EA1F03123E127E387C01E090C7FC12FCA6127C127EA2003E13306C13 60380FC0E03807FF803800FE0014167E9519>II< 13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA2127E 003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F0007B51280 380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC12 1CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6CB4 5A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A20C 247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC133E137EEA78 FCEA7FF0EA1FC0102E83A311>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFF F1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0 A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7007C137C A26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEB C03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A2 1A207E951F>I114 DI<13C0A41201A212031207120F12 1FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF 14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E000713 0CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E >I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C6C5AA2EB 7EC0A2137F6D5AA26DC7FCA2130EA2130CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B20 7F951E>121 D E /Fl 66 123 df11 D<133FEBFF803803C1C0EA0703A2380E 018090C7FCA5B512C0A2EA0E01AE387F87F8A2151D809C17>II<90383F03F09038FFCFF83903C0FC1C3907 01F03CA2390E00E01892C7FCA5B612FCA2390E00E01CAE3A7FC7FCFF80A2211D809C23>I34 D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C>39 D<13C0EA0180EA030012 06120E120C121C121812381230A21270A21260A212E0AC1260A21270A21230A212381218121C12 0C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C120C120E120612077EA21380 A21201A213C0AC1380A21203A21300A25A1206120E120C121C12185A5A5A0A2A7E9E10>I<1270 12F012F8A212781218A31230A2127012601240050D7D840C>44 DI<12 7012F8A3127005057D840C>I<1303A213071306A2130E130C131C1318A213381330A213701360 A213E013C0A21201138012031300A25A1206A2120E120CA2121C1218A21238123012701260A212 E05AA210297E9E15>II<12035A123FB4FC12C71207B3A3EAFFF8A20D1C7C9B15>I< EA07C0EA1FF0EA3878EA603C131E12F0EAF80FA312701200130E131E131C133C1378137013E0EA 01C0EA0380EA0700EA0E03120C1218EA3006EA7FFE12FFA2101C7E9B15>II<1260387FFF80A21400EA6003EAC006 5BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B15>55 D57 D<127012F8A312701200A8127012F8A3127005 127D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A2127012601240051A 7D910C>I63 D<1306130FA3497EA4EB33C0A3EB61E0A3EB C0F0A338018078A2EBFFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C1D7F9C1F>65 DI<90381F8080EBFFE13803F0333807801B380F000F00 1E1307001C1303123C5A1401127012F091C7FCA70070EB01801278A27E001CEB0300121E6C1306 6C6C5A3803F0383800FFF0EB1F80191E7E9C1E>IIII<90381F8080EBFFE13803F0333807801B380F000F001E 1307001C1303123C5A1401127012F091C7FCA5ECFFF0A20070EB07801278A27E121C121E7E3807 800F3803F0393800FFF090381FC0001C1E7E9C21>I<39FFF3FFC0A2390F003C00AAEBFFFCA2EB 003CAC39FFF3FFC0A21A1C7E9B1F>II77 DIIIII<3807E080EA1FF9EA3C1FEA7007 130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA07FFC61380130FEB03C0A2130112C0A3 00E01380130300F01300EAFC0EEACFFCEA83F8121E7E9C17>I<007FB512C0A238780F03007013 010060130000E014E000C01460A400001400B03803FFFCA21B1C7F9B1E>I<3AFFE0FFE1FFA23A 1F001E007C6C1530143FA20180147000079038678060A32603C0E713C0ECC3C0A2D801E0EBC180 9038E181E1A3D800F3EBF3001400A2017B13F6017E137EA3013C133CA3011C133801181318281D 7F9B2B>87 D92 D97 D<12FCA2121CA9137EEA1DFF381F8780381E01C000 1C13E0130014F0A614E01301001E13C0381F07803819FF00EA187C141D7F9C17>IIII<1378EA01FCEA039EEA071EEA0E0C1300A6EAFF E0A2EA0E00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F07381E0380A2121CAB 38FF9FF0A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121CAEEAFF80A2091D7F9C 0C>II<12FCA2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA1C781338133C131C7F 130F38FF9FE0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C>I<39FC7E07E039FDFF 9FF8391F83B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124>IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0A612781307123CEA 1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA2123CEAFFE0A2EA1C00 A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307EA0C0F380FFBF0EA 03E314127F9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA2 6C5AA313127F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B061486A2EB318E000713 CCA213603803E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038070600EA038EEA01 DC13D8EA00F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8A21512809116>I< 38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C5AA35BA3EAF180A2 00C7C7FC127E123C131A7F9116>II E /Fm 18 118 df<1238127C12FEA3127C12381200A41238127C12FEA3127C123807127D910D>58 D63 D68 D73 D77 D97 D99 D101 D<3803F0F0380FFFF8383E1F 38383C0F30007C1380A4003C1300EA3E1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E048 13F0387801F838F00078A3007813F0383E03E0381FFFC03803FE00151B7F9118>103 D<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F9D0E>105 D108 D<39FF1FC0FE90387FE3FF3A1FE1F70F809039 80FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1FC0EB7FE0381FE1F0EB80F8A213 00AB38FFE7FFA218127F911B>II<38FF1FC0EBFFE0381FC1F8 130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090C7FCA6EAFFE0A2171A7F911B> I115 D<1203A35AA25AA2123FEAFFFCA2EA1F00A9130CA4 EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA03F818127F 911B>I E /Fn 43 122 df<903901FF81FF011F01EF13C0903A7F80FF87E0D9FE01EB0FF03903 FC03FE13F8D807F013FCA2EE07E0020190C7FCA6B712F8A32707F001FCC7FCB3A33A7FFF1FFFE0 A32C2A7FA928>11 D<121C127FEAFF80A5EA7F00121C09097B8813>46 D48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB 007F13FEA317277BA622>III<14075C5C5C5C5CA25B 5B497E130F130E131C1338137013F013E0EA01C0EA0380EA07005A120E5A5A5A5AB612F8A3C713 00A7017F13F8A31D277EA622>I<000C1303380F803FEBFFFEA25C5C14E05C49C7FC000EC8FCA6 EB7FC0380FFFF8EB80FC380E007F000C1480C7123F15C0A215E0A2123E127FEAFF80A315C01300 007E137F007814806CEBFF00381F01FE380FFFF8000313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E0 4814C01580A215003870001E5C5C48137014F0495AC6485A13075C130F91C7FC5B5BA3137EA213 FEA41201A86C5A13781C297CA822>III66 D<91393FF00180903903FFFE0701 0FEBFF8F90393FF007FF9038FF80014848C7127FD807FC143F49141F4848140F485A003F15075B 007F1503A3484891C7FCAB6C7EEE0380A2123F7F001F15076C6C15006C6C5C6D141ED801FE5C6C 6C6C13F890393FF007F0010FB512C0010391C7FC9038003FF829297CA832>II73 D77 D III82 D<90387F80603903FFF0E0000F13FF381F807F383F001F003E1307007E1303127C00FC1301A214 007E7E6D130013F8EBFF806C13F814FE6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F 1407A200E013031401A37E15F06C13036C14E0B413079038E01FC090B5120000E05B38C01FF01D 297CA826>I<007FB712C0A39039807FC03FD87C00140700781503A20070150100F016E0A24815 00A5C71500B3A490B612E0A32B287EA730>I<48B47E000F13F0381F81FC486C7E147FA2EC3F80 A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F 83DFFC380FFF0F3801FC031E1B7E9A21>97 DIIII<90 38FF81F00003EBE7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80 FE001F5B380FC1F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14 F0121F397E0007F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038 007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3 A3EAFFFEA30F2B7DAA14>105 DIII<3BFFC07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F90 3ADC01FB803F01F8D9FF00138001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FF C07F9038C1FFC09038C787E0390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B 7D9A26>II<38FFE1FE9038E7FF809038FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A2 5DEBF0014A5A01F85B9038FE0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26> I<38FFC3F0EBCFFCEBDC7E380FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C 13F0000713F8C613FC1303130000E0137C143C7EA26C13787E38FF01F038F7FFC000C11300161B 7E9A1B>I<1370A413F0A312011203A21207381FFF