From owner-mpi-pt2pt@CS.UTK.EDU Mon Nov 23 14:37:41 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18480; Mon, 23 Nov 92 14:37:41 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA23460; Mon, 23 Nov 92 14:28:46 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23454; Mon, 23 Nov 92 14:28:42 -0500 Received: from LOCALHOST.cs.utk.edu by thud.cs.utk.edu with SMTP (5.61++/2.7c-UTK) id AA06042; Mon, 23 Nov 92 14:28:41 -0500 Message-Id: <9211231928.AA06042@thud.cs.utk.edu> To: mpi-pt2pt@cs.utk.edu Cc: wade@cs.utk.edu Subject: a test Date: Mon, 23 Nov 92 14:28:40 EST From: Reed Wade this is a test of the mpi-pt2pt mailing list. -reed Mon Nov 23 16:03:12 EST 1992 From owner-mpi-pt2pt@CS.UTK.EDU Tue Nov 24 23:08:30 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15646; Tue, 24 Nov 92 23:08:30 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA25775; Tue, 24 Nov 92 22:28:54 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25771; Tue, 24 Nov 92 22:28:51 -0500 Received: from Electryon.Think.COM by mail.think.com; Tue, 24 Nov 92 22:28:43 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA20255; Tue, 24 Nov 92 22:28:41 EST Date: Tue, 24 Nov 92 22:28:41 EST Message-Id: <9211250328.AA20255@electryon.think.com> To: mpi-pt2pt@cs.utk.edu Subject: Nonblocking functions and handlers. I would like to float a few questions about what we mean by the modifier nonblocking it is applied to the MPI specification. Nonblocking Send: This is the curious one. Does is mean: a) The user may alter the contents of the buffer once the send function returns control? or b) The user must check the status of the send operation to determine when it is safe to alter the contents of the buffer? Nonblocking receive: This seems pretty clear. The user must inquire of the system to determine if the user specified buffer contains the desired message. In either case, it may be desirable to specify an action that is to occur once the operation is complete. Consider adding two arguments to the send and receive functions: MPI__n( int , int tag, void * buffer, int len, void (* handler)(), void * arg ) Here, handler is a pointer to a function which accepts two arguments: handler( int msg_id, * arg ) Upon completion of the send or receive operation, the handler function is invoked and provided with the id of the message operation just completed and the argument address provided by the user. I raise the issue of handlers because we have ben asked repeatedly to provide such fuctionality. Thoughts? moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:38:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17471; Wed, 25 Nov 92 10:38:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06314; Wed, 25 Nov 92 10:13:00 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06310; Wed, 25 Nov 92 10:12:58 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA08435; Wed, 25 Nov 92 09:12:54 CST Received: by teak18.cray.com id AA05289; 4.1/CRI-5.6; Wed, 25 Nov 92 09:12:52 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251512.AA05289@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 9:12:49 CST Cc: par@teak.cray.com (Peter Rigsbee) In-Reply-To: <9211250328.AA20255@electryon.think.com>; from "Adam Greenberg" at Nov 24, 92 10:28 pm X-Mailer: ELM [version 2.3 PL11b-CRI] Adam Greenberg writes: > In either case, it may be desirable to specify an action that is to occur once > the operation is complete. Consider adding two arguments to the send and > receive functions: > > MPI__n( int , int tag, void * buffer, int len, > void (* handler)(), void * arg ) > > Here, handler is a pointer to a function which accepts two arguments: > handler( int msg_id, * arg ) > > Upon completion of the send or receive operation, the handler function is > invoked and provided with the id of the message operation just completed and > the argument address provided by the user. I've never really liked this approach. I think it adds complexity without really adding much value. A couple of issues: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) - second, even if you have a signal handler, the mainline of the application must still poll something to see if the operation has completed; the difference is that the "something" if often a local variable rather than the messaging system. But the logic in the program is similar and it is likely that the number and decision of where to poll is driven by the algorithm. - third, do all architectures provide the user-level interrupts needed to implement this? If one doesn't, adding this functionality may simply not be possible. IMO, this seems like something that isn't desirable as a part of MPI1. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 10:40:02 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17537; Wed, 25 Nov 92 10:40:02 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA06692; Wed, 25 Nov 92 10:30:22 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06686; Wed, 25 Nov 92 10:30:19 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03147; Wed, 25 Nov 92 09:28:46 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA12985; Wed, 25 Nov 92 09:28:43 CST Date: Wed, 25 Nov 92 09:28:43 CST Message-Id: <9211251528.AA12985@godzilla.mcs.anl.gov> To: moose@think.com Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Adam Greenberg's message of Tue, 24 Nov 92 22:28:41 EST <9211250328.AA20255@electryon.think.com> Subject: Nonblocking functions and handlers. I'd love to see a standard implementation that invoked a routine on receipt, particularly if that routine was not restricted in any way (for example, restricted to not using any MPI calls). Unfortunately, doing this right makes demands on the underlying OS that may be more than some vendors are willing to commit to. We need to be careful here. Note that Unix does not provide this functionality to users, though VMS has for a long time. (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. Bill Gropp From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:07:46 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19010; Wed, 25 Nov 92 11:07:46 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07267; Wed, 25 Nov 92 11:02:46 -0500 Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07263; Wed, 25 Nov 92 11:02:43 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Wed, 25 Nov 1992 11:02:31 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA26484; Wed, 25 Nov 1992 11:02:30 -0500 Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> To: mpi-pt2pt@cs.utk.edu, par@teak.cray.com Subject: Re: Nonblocking functions and handlers. Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Yale Computer Science Department From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 11:38:31 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19507; Wed, 25 Nov 92 11:38:31 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA07714; Wed, 25 Nov 92 11:22:36 -0500 Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07710; Wed, 25 Nov 92 11:22:33 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.2) id AA13731; Wed, 25 Nov 92 10:22:31 CST Received: by teak18.cray.com id AA05599; 4.1/CRI-5.6; Wed, 25 Nov 92 10:22:30 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9211251622.AA05599@teak18.cray.com> Subject: Re: Nonblocking functions and handlers. To: mpi-pt2pt@cs.utk.edu Date: Wed, 25 Nov 92 10:22:26 CST In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU>; from "Harry Berryman" at Nov 25, 92 11:02 am X-Mailer: ELM [version 2.3 PL11b-CRI] Scott Berryman writes: > - first, like most signal handling, this kind of capability tends > to be more oriented towards C programs and away from Fortran > programs (note that your prototype for "handler" is such that > it could not be programmed in Fortran) > > Although the prototype given by Greenberg precludes the use of > Fortran, the approach certainly does not preclude Fortran. > Obviously Fortran can pass pointers to functions as parameters. Point well taken. But there are two aspects of the interface to consider. Fortran passing pointers to functions as parameters is fine. But a Fortran programmer might also want to write the handler in Fortran, which would then require that the handler's arguments all be pointers to things. But this would bother C programmers and would be inconsistent with other "signal handlers". > In particular, the iPSC series (including the Delta) support > a similar mechanism in Fortran through the hrecv(), et al. My iPSC documentation (which may be out of date) shows that hrecv() can be called by Fortran but requires that the handler be written in C: proc is the procedure executed when the receive is complete. This procedure must be written by the user and must be a C procedure with four arguments. I'd argue that having to write the handler in C makes this less useful to Fortran programmers, especially if you are trying to write something portable. There is no real standard defining how C and Fortran interact. The C handler could call a subhandler written in Fortran, but this is getting kind of messy for something that should be simple. - Peter Rigsbee From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:43:10 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21772; Wed, 25 Nov 92 13:43:10 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA10110; Wed, 25 Nov 92 13:30:11 -0500 Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10106; Wed, 25 Nov 92 13:30:09 -0500 Received: from Electryon.Think.COM by mail.think.com; Wed, 25 Nov 92 13:30:05 -0500 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA22742; Wed, 25 Nov 92 13:30:04 EST Date: Wed, 25 Nov 92 13:30:04 EST Message-Id: <9211251830.AA22742@electryon.think.com> To: berryman-harry@CS.YALE.EDU Cc: mpi-pt2pt@cs.utk.edu, par@teak.cray.com In-Reply-To: <199211251602.AA26484@YOGI.NA.CS.YALE.EDU> "berryman-harry@CS.YALE.EDU" Subject: Nonblocking functions and handlers. Date: Wed, 25 Nov 1992 11:02:30 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Peter Rigsbee says: - first, like most signal handling, this kind of capability tends to be more oriented towards C programs and away from Fortran programs (note that your prototype for "handler" is such that it could not be programmed in Fortran) Although the prototype given by Greenberg precludes the use of Fortran, the approach certainly does not preclude Fortran. Obviously Fortran can pass pointers to functions as parameters. In particular, the iPSC series (including the Delta) support a similar mechanism in Fortran through the hrecv(), et al. -scott berryman Mea culpa. The prototype was not meant to exclude Fortran (in fact we through a discussion of this issue here last week). The handler should viewed as taking two pointers: void handler( int * msg_id, * arg ) Sorry for any confusion: moose From owner-mpi-pt2pt@CS.UTK.EDU Wed Nov 25 13:44:47 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21797; Wed, 25 Nov 92 13:44:47 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA09803; Wed, 25 Nov 92 13:14:38 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09784; Wed, 25 Nov 92 13:14:29 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA22170; Wed, 25 Nov 92 13:14:31 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 131340.4011; Wed, 25 Nov 1992 13:13:40 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA12688; Wed, 25 Nov 1992 12:06:04 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA08958; Wed, 25 Nov 92 12:06:02 -0600 Date: Wed, 25 Nov 92 12:06:02 -0600 Message-Id: <9211251806.AA08958@brisk.kai.com> To: mpi-pt2pt@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-formal@cs.utk.edu, mpi-ptop@cs.utk.edu Reply-To: William.Gropp's.message.of.Wed@kai.com, 25 Nov 92 09:28:43 CST <9211251528.AA12985@godzilla.mcs.anl.gov> Subject: Nonblocking functions and handlers. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Bill Gropp writes: (Warning: radical position that I'm not sure even I hold follows:) An interesting issue is whether we should defer all nonblocking communications to a thread-based execution model. I'm not so sure this is a radical position Bill since even nonsynchronized communication will need to be defined formally this way. Nonsynchronized communication is in effect creating a parallel process that has the job of passing the communication on. Al Geist earlier asked the question wheather buffers used by nonsynchronized communication should be accessible after the communication has started - the answer should be - no, unless by some explicit mechanism that formally amounts to a communication with the process mentioned above. Any nonexplicit interaction (e.g. a write to the buffer) would have to be specified as formally equivalent to an explicit interaction. Also, there is quite a range of terminology in use. One common error: "Asynchronous" and "synchronous" has quite a particular meaning in EE and when CS people use the terms in relation to message passing they usually mean NONSYNCHRONIZED and SYNCHRONIZED. Also BLOCKING = SYNCHRONIZED. Let us begin a glossary that defines the terms we use - if no-one else volunteers I'll take this to be the responsibility of the Formal Specification Subcommittee. So I'm looking for volunteers from that subcommittee. Steven From owner-mpi-pt2pt@CS.UTK.EDU Sat Nov 28 19:07:59 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08243; Sat, 28 Nov 92 19:07:59 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22503; Sat, 28 Nov 92 18:46:34 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22499; Sat, 28 Nov 92 18:46:32 -0500 Message-Id: <9211282346.AA22499@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6099; Sat, 28 Nov 92 18:45:57 EST Date: Sat, 28 Nov 92 18:35:53 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-pt2pt@cs.utk.edu Subject: Our work Reply-To: SNIR@watson.ibm.com First, a technical issue: If somebody cannot print postscript, or has strong preferences for Latex, pls let me know. Now for content. I would like to use the outline I just sent as a working document for our subcommittee, and modify it periodically to reflect input I receive from the remaining group members. I shall update this document every weekend, or every 2 weekends, according to the number of comments I get, and the amount of spare time my family grants me. Of course, decisions will be taken when we meet, but I feel it is useful to have a working document that reflects in a somewhat coherent manner the state of our deliberations. So please, start sending your comments to mpi-pt2pt, assuming the next checkpoint is end of next week. I want to replace as soon as possible the I's by We's. From owner-mpi-pt2pt@CS.UTK.EDU Thu Dec 3 10:39:15 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01501; Thu, 3 Dec 92 10:39:15 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA16013; Thu, 3 Dec 92 10:20:18 -0500 Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16008; Thu, 3 Dec 92 10:20:13 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA25545 (5.65c/IDA-1.4.4 for ); Thu, 3 Dec 1992 16:19:46 +0100 Received: by f1neuman.gmd.de id AA14744; Thu, 3 Dec 1992 16:19:30 +0100 Date: Thu, 3 Dec 1992 16:19:30 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9212031519.AA14744@f1neuman.gmd.de> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: gmap10@f1neuman.gmd.de I would like to make some comments on the document Marc Snir has distributed. 1. I agree on all points in the list of goals. I find it especially important to define something which is not too far away from common practice, otherwise nobody would follow us. A point which is missing on the list (perhaps because it is too obvious): the interface should not contain more functions than are really necessary. We must avoid to end up with a mixed bag which contains multiple mechanisms for the same thing. 2. I like Marc's typification scheme for message passing functions based on the 4 suboperations. I understand those suboperations more as a way to describe the semantics rather than the implementation of the (compound) functions which the user will see. I'm not sure that the channel based operations are necessary on the user level. Basically there are the two addressing schemes for message passing: either a message is sent to a named process, or it is sent along a named connection (channel), in which case the processes don't even need to have identifiers. We had a long discussion here in Europe on which scheme is better. In the end it turns out that both are more or less equivalent. Most existing user interfaces tend towards the named process approach. The only reason for supporting both concepts in the same interface seems to be the possibility of some additional optimization. If suboperations (1) and (4) are separated, ressource allocation and deallocation can be saved. However, I think that a clever operating system can detect frequently used communication paths automatically, and it can establish a permanent link for them. (Link caching) 3. We will have various message passing mechanisms, especially synchronous and asynchronous communication (global/local send termination). I strongly recommend that the corresponding send and receive functions must match. On one hand this gives the opportunity for some implementation optimization, and also it might be used for correctness checking. 4. As Marc said, in MPI1 termination is either local or global, at the user's choice. I really think that this is necessary, and a restriction on either synchronous or asynchronous communication would be hard to accept. 5. Marc suggests that there should be no "unsuccessful return" of a message passing operation. I agree. 6. On page 7, Marc discusses the different protocols for message passing. I'm not sure that we want to define this in detail, since it is very much implementation dependent. On the other hand we will have to accept the fact that a program will run on a machine with huge system buffers and not on a smaller system. Somehow this is true with other system resources, also. 7. Page 7, further down: if we deal with heterogeneous networks, a contiguous message buffer may contain sections with different data type. The reason is always that one wants to send as few messages as possible (startup minimization). Personally I prefer to have a separate library call before the send which specifies the contents of the message buffer. This information is then used or ignored by the send routine depending on the data representations in the sender and receiver processors. 8. If our goal is to define a scalable interface, setting up groups by specifying member lists in all participating processes is a problem. This is especially so for the handling of very regular group structures, like grids or tori, since these will most likely be used by applications with very large process numbers. I think the Virtual Topology subcommittee has to deal with this problem. 9. Communication context: I very much favor option 1. (i.e. context switch by some special functions) against explicitely specifying the context in a (collective) communication statement. I do not see that the explicit specification would lead to less user errors, and it would add another item to the argument lists. 10. Receive criteria: I do not see a good point for having "don't care" addresses in sends. For "don't care" in receive I prefer named constants over "magic values". Be sure not to use arguments for both input to and output from a function, especially if it may contain wildcards on input. There is no better way to make sure your constants get overwritten! 11. Matching of data: In Fortran I think option 2. is acceptable. If the receive buffer is too short, this is a user error. In C there is another option which should be supported by the interface: if the receive buffer pointer is NULL, the system allocates one with the right size automatically. 12. Syntax: multiple modes like in MPI1 are okay with me, but I hate to see character strings there. Talk to any implementor who cares about efficiency, and you know what I mean. 13. Also, in Fortran I find it better to use procedure calls for most functions. 14. Miscelania: a) We will need a nonblocking AWAIT operation. This is a great help in organizing the overlap of computation and communication. b) How could we survive without a PROBE function? c) What about providing an inquiry subroutine, which returns the message length, the sender, and the type of the "last message" which was received? For non-blocking messages this is not well defined, but a solution should be possible. This has become a longer message than I had planned. Sorry for bothering you all so much! Rolf Hempel From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 16:39:48 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21532; Wed, 9 Dec 92 16:39:48 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22026; Wed, 9 Dec 92 16:32:54 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22022; Wed, 9 Dec 92 16:32:51 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA05364; Wed, 9 Dec 92 15:32:41 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA00525; Wed, 9 Dec 92 15:32:39 CST Date: Wed, 9 Dec 92 15:32:39 CST Message-Id: <9212092132.AA00525@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Marc Snir's paper Cc: lusk@antares.mcs.anl.gov Thanks to Marc for laying out many of the issues so clearly. We find that we agree with many and disagree with some. Goals: (1) We think that those most in need of a standard are those writing libraries, and therefor the standard should cater to their needs as well as those of end users. It will be easy to design simple application programming interfaces on top of the MPI standard. (2) We strongly agree with this point (allow efficient communication). (3) We are not sure what "allow (but no mandate)" for heterogeneous communications means. If this means that vendors of homogeneous massively parallel systems may implement only the homogeneous operations, then we agree, as long as the heterogeneous operations are defined by the MPI standard. We agree in principle with the others. Framework: We also find the described framework useful for thinking about the operations, but not necessarily the right structure for the MPI. Issues: An exchange operation can be useful. At least two MPP vendors (TMC and Intel) provide such operations. We also agree an leaving out RPC, active messages, etc, as these are topics of rapidly moving research. We believe that there may be something more inclusive than channels that can serve as the basis for specifying high-performance communication operations. We do not find the discussion of "local versus global" termination to be a good way to consider the issues of blocking, nonblocking, and synchronous communication. In particular, blocking and nonblocking refer to the state of the buffer containing the message on return from the routine (be it send, receive, or wait), and synchronous (in the MPI proposal) really refers to an alternate protocol for the delivery of a message. We believe that relying on an exception handling mechanism for error handling will make the system too fragile. We propose allowing the return of error conditions. An alternate proposal allows the user to specify an error routine to be called in the event of an error. Note that library implementors may need to handle errors explicitly. Correctness Criteria: We believe that it is important to allow the user to determine (at least a bound on) the amount of resources available. Further, it MAY be appropriate to specify a minimum size that is supported (otherwise zero may be consistent with the letter of the standard). Communication Parameters: Our draft implementation includes an extension to the current proposal that permits the communication of messages of mixed datatype (we interpret "type" on page 7 to mean "datatype" rather than "message type"). This is a simple extension to the data description vector that allows for heterogeneous communication. Groups and context: We agree that the collective communication committee should be the one to discuss these issues. We do feel that the destination and source processor ids should not be relative to some "current" group. Note that such a functionality (with some restrictions) can be provided by a simple layer on top of MPI. For an example of a problem with making processor id's relative to a current "group", consider recv(source=DONT_CARE,...). Does this mean only in the current group? What is the result of MPI_INFOS() on the received message? What if two processors A and B are both in two groups alpha and beta, but with different rankings (ie., A = 2 and B = 5 in alpha, and A = 12 and B = 0 in beta). Now A (current group alpha) sends to B (current group beta). B does recv(source=DONT_CARE,...). Does B receive this message? What is the value of MPI_INFOS()? If B doesn't receive this message, then how is a communication context different from a group that is a copy of the current group (which would seem to supply the same semantics)? Matching of send and receive: We believe that there should be two types of matching operations: a simple (efficient) one on message type, and a general one using a user-defined predicate (matching function). Our experience has been that once the users have "select by source", they'll then want "select by source range and/or type range and/or message size and/or ...". We also agree that truncation of a received message into a too-short buffer should be an error (or at least detectable!). Syntax: We agree with multiple function names rather than multiple modes. It is easier to provide a efficient implementation and clear semantics in this case (for example, having the return code always have the same meaning independent of the input arguments). Note also that it is easy, given a multiple function names approach, to produce one-routine-does-everything interfaces but not vice-versa. Miscelania: We must have a non-blocking AWAIT. We'd like to AWAIT several messages. We must have a PROBE. We agree that the issue of returning data on received messages (particularly those done with nonblocking operations) is very important. One approach (used by one MPP vendor) is to ask for this info by handle, and then explicitly free the handle. This eliminates unnecessary global state (the currently most recently received message) from the semantics, and makes it possible to easily describe what exactly is returned by these "INFO" routines. Bill Gropp and Rusty Lusk From owner-mpi-pt2pt@CS.UTK.EDU Wed Dec 9 17:09:55 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22659; Wed, 9 Dec 92 17:09:55 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22401; Wed, 9 Dec 92 16:51:16 -0500 Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22397; Wed, 9 Dec 92 16:51:13 -0500 Received: from b125 (b125.super.org) by super.super.org (4.1/SMI-4.1) id AA08797; Wed, 9 Dec 92 16:51:11 EST Received: by b125 (4.1/SMI-4.1) id AA04506; Wed, 9 Dec 92 16:51:10 EST Date: Wed, 9 Dec 92 16:51:10 EST From: lederman@b125.super.org (Steve Lederman) Message-Id: <9212092151.AA04506@b125> To: mpi-pt2pt@cs.utk.edu Subject: thoughts I have a few thoughts on the ideas that have been passed around. I'm sorry for not responding sooner but a cold has slowed me down. First, on Marc's document of Nov. 28: - Goal 4 is for a C, C++, F77 and F90 binding. The draft Intro that Jack sent around on Nov. 30 states in the subsection for Who Should Use This Standard? that it is for Fortran 77 and/or C. Do we wish to include C++ and F90 which would likely to have different interfaces? - I find the SEND_RECV operation an interesting option. On the old Mark/?? machines from Caltech/JPL there was a call (if my recollection is correct) that allowed for the same user buffer to serve as the send and receive buffer. I assume this is not what is intended here? I think SEND_RECV would be used with some frequency by users. Would it simply be a convenience or can it be made faster than issuing the two calls. I would think it would be faster on some architectures (even more than the two call overhead). If it is faster then my inclination is to include it even though it adds another call. comments? - I have a question regarding when a communication operation is completed. On a send, does it terminate locally if the user buffer is copied into a system buffer but has not left the node? (I believe this is done on the CM5.) Analogously, does it terminate globally if it gets to a system buffer on the receiving node or until it is in the user buffer? As for choosing local, global or both I vote for the current choice of both. I can see applications for both. - I think that Marc's suggestion that MPI functions not return unsuccessfully has merit. It would be pain to be constantly checking the status after each operation to be safe. Along these lines is what to do if the send and recv message lengths are different. The current standard allows for fewer bytes to be received and the recv returns the actual number. Again, I think it would be a pain to have to check every time. If this is a desirable feature to allow different lengths, then I vote to make it users selectable. I know I have found many bugs by not having messages that were a different size being received. - I think that a PROBE function would be a good idea. Given that the standard is not currently going to support active messages, etc. it seems that being able periodically check if any messages have arrived would be a useful feature. Another thought: At the meeting in Minneapolis, several people, including myself, discussed the advantage of having a send where the sending process knows the receiving process has already posted a receive. This could relieve the send process of having to check with the receiving process since it knows it will not have to buffer the message in system space and it knows it is ready to receive the message. If a system could not take advantage of this then it could revert to a normal send. That is, the standard would not specify the way the send would occur, it would allow for an alternative protocol if the vendor wished. This is the rr (receiver ready) option in Bill Gropp and Ewing Lusk's implementation. I would like to propose we include this in the standard. Thoughts? Well enough for one message. Steve From owner-mpi-pt2pt@CS.UTK.EDU Mon Dec 14 15:48:42 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24886; Mon, 14 Dec 92 15:48:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17233; Mon, 14 Dec 92 15:48:21 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 14 Dec 1992 20:48:20 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17227; Mon, 14 Dec 92 15:48:19 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA03749; Mon, 14 Dec 92 15:48:17 -0500 Date: Mon, 14 Dec 92 15:48:17 -0500 Message-Id: <9212142048.AA03749@thud.cs.utk.edu> To: mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum Forwarding: Mail from '"Dr. C.D. Wright" ' dated: Mon, 14 Dec 92 12:16:10 GMT ---------- Begin Forwarded Message ---------- >From @ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK Mon Dec 14 07:20:05 1992 Return-Path: <@ibm.liv.ac.uk:CDW10@LIVERPOOL.AC.UK> Received: from mail.liv.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22696; Mon, 14 Dec 92 07:19:55 -0500 Received: from ibm.liverpool.ac.uk by mailhub.liverpool.ac.uk via JANET with NIFTP (PP) id <21042-0@mailhub.liverpool.ac.uk>; Mon, 14 Dec 1992 12:19:28 +0000 Received: from UK.AC.LIVERPOOL by MAILER(4.4.t); 14 Dec 1992 12:20:02 GMT Date: Mon, 14 Dec 92 12:16:10 GMT From: "Dr. C.D. Wright" Subject: Re: Message Passing Interface Forum To: dongarra@edu.utk.cs Message-Id: <"mailhub.li.044:14.11.92.12.19.28"@liverpool.ac.uk> Status: RO Hi. Since I am in the UK it is clear that I can't actively participate in the MPI Forum. I do, however, have one particular problem with every comms library I have used so far that I would like to see addressed in any new "standard", and I hope you can pass this on to whoever is the appropriate person to deal with it. In many packages such as PVM, PARMACS, p4, etc, it is possible to probe for and/or receive messages selectively, the selection being based on the message type (usually in integer) and/or the sender. This is overly restrictive. It would be far more useful if the message's format were sufficiently well defined for the user to be able to provide their own selection function to be passed in and used as the basis for reception and/or probing. That's it. Hope you can do something with this gripe/suggestion. Colin. ----------- End Forwarded Message ----------- From owner-mpi-pt2pt@CS.UTK.EDU Tue Dec 15 19:28:34 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21803; Tue, 15 Dec 92 19:28:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15587; Tue, 15 Dec 92 19:28:11 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Wed, 16 Dec 1992 00:28:10 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from helios.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15573; Tue, 15 Dec 92 19:28:06 -0500 Received: by helios.llnl.gov (4.1/LLNL-1.18) id AA11599; Tue, 15 Dec 92 16:30:03 PST Date: Tue, 15 Dec 92 16:30:03 PST From: tony@helios.llnl.gov (Anthony Skjellum) Message-Id: <9212160030.AA11599@helios.llnl.gov> To: dongarra@cs.utk.edu, mpi-collcomm@cs.utk.edu, mpi-pt2pt@cs.utk.edu Subject: Re: Message Passing Interface Forum That is what we have been talking about in Zipcode for a long time. - Tony From owner-mpi-pt2pt@CS.UTK.EDU Fri Dec 25 19:15:36 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20495; Fri, 25 Dec 92 19:15:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09422; Fri, 25 Dec 92 19:15:07 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Sat, 26 Dec 1992 00:15:05 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09400; Fri, 25 Dec 92 19:15:04 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA14461; Fri, 25 Dec 92 19:14:59 -0500 Date: Fri, 25 Dec 92 19:14:59 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9212260014.AA14461@msr.EPM.ORNL.GOV> To: mpi-pt2pt@cs.utk.edu Subject: point-to-point proposal consistent with collective proposal Point-to-point Communication Proposal ------------------------------------- After reading Marc Snir's Message Passing Interface Outline and the other point-to-point comments, I would like to propose the following minimum set of point-to-point routines for discussion at the January meeting in Dallas. Stated Goals: 1. Design an application programming interface. 2. Design an interface that is not too different from current practice 3. Define an interface that can be quickly implemented on many vendor platforms. 4. Focus on a proposal that can be agreed upon in 6 months. 5. Provide a reliable communication interface. 6. Allow efficient communication. 7. Interface should not contain more functions than are really necessary. ----------------------------------------------------------------------------- General Comments: The collective communication interface should be an extension of the point-to-point interface. a. As Marc points out on page 2 "SEND and RECV are a particular case of broadcast in a group of size 2; this observation can be used to check if the definition of collective communication semantics are consistent with the definition of point-to-point communication." b. Collective routines like broadcast should provide the same message data format (contiguous, buffer with stride, etc.) as point-to-point routines. c. By using a structured name space (described on pages 7-8 of [Snir]) where all processes are identified by a (group,rank) pair for both the point-to-point and collective routines, then the users will have a consistent naming scheme across all the MPI communication routines. And those desiring a flat name space can have it by using the default group "ALL". d. Syntax guidelines - short, mnemonic names; systematic ordering of parameters; systematic naming convention. ------------------------------------------------------------------------------- Proposed point-to-point routines: 1. info = MPI_SEND( buf, bytes, type, gid, dest ) Function: Sends a contiguous message of length "bytes" pointed to by "buf" to the process identified by ("gid","dest") with a message tag of "type". On return "info" contains the error code. Conceptually the user can view this function as "I want this message delivered to the specified process in the fastest reliable manner." Implementation comments: On return user can consider buf is safe for reuse. MPI_SEND need not be strictly blocking. Users do not want to have to dance around the send buffer with STAUS and WAIT checking for when the send buffer is safe to change. If a compiler can determine that (or rearrange code so that) the send buffer is not touched before the send completes, then the implementation could be non-blocking. If a multiprocessor requires synchronous communication or routing along a particular topology, then this could be set in a global routine that affects all point-to-point and collective routines. 2. info = MPI_RECV( buf, bytes, type, gid, source ) Function: Receive the message specified by "type", "gid", and "source" into "buf". The message must be <= "bytes" in size or "info" returns error. Routine waits until specified message has arrived [like blocking recv]. On return "info" contains the error code. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. Conceptually the user can view this function as "I must have the specified message before I can do anything else. I will wait until it arrives." 3. info = MPI_NRECV( buf, bytes, type, gid, source ) Function: Checks to see if the specified message has arrived [like probe]. A minus one (-1) in "type", "gid", or "source" means don't care and that argument will match any message. If the message has arrived it is returned in "buf" and "info" = 1. The message must be <= "bytes" in size or "info" returns error. If the message has not arrived, then the routine returns with "info" = 0. Values less than 0 returned by "info" indicate an error. MPI_NRECV can be called multiple times to check on the arrival of a message, and the user can later call MPI_RECV to wait on the same message. Conceptually the user can view this function as "I would like to receive the specified message now, but if it is not here, I will not wait for it. I will go do something else." Implementation is not specified here, but for efficiency, implementors may wish to post a receive on the first call to MPI_NRECV and just do checks [like status] on repeat calls. If implemented like this, MPI_RECV must be able to check both the system queue and the posted queue. 4. type = MPI_TYPE() 5. bytes = MPI_BYTES() 6. gid = MPI_GID() 7. source = MPI_SOURCE() Function: These routines return the actual values for the last message returned by MPI_RECV or MPI_NRECV. Values < 0 indicate an error. Allows the user to determine the size of the received message and the values of "don't care" arguments. 8. bytes = MPI_PACK( data_type, nitems, stride, inbuf, outbuf ) 9. bytes = MPI_UNPACK( data_type, nitems, stride, inbuf, outbuf ) Function: General purpose routines for packing and unpacking an arbitrarily complex message buffer. Allows for non-contiguous messages, mixed data types in a single message, and heterogeneity between machines. Same routines can be used to build collective communication buffers, for example for MPI_BCAST. On return "outbuf" contains the (potentially) encoded data, and "bytes" is the size in bytes of the encoded data. "bytes" < 0 indicates an error. The following data types would be supported: BYTE INTEGER*2 INTEGER*4 REAL*4 REAL*8 COMPLEX*8 COMPLEX*16 I would also like to see the following data types considered: CHARACTER_STRING INTEGER*8 REAL*16 COMPLEX*32 "nitems" is the total number of items of this data type to pack or unpack from "inbuf". "stride" can be used for packing rows of a matrix in Fortran or columns of a matrix in C. MPI_PACK can be called multiple times to construct a single message buffer. MPI_UNPACKs must be called by the user to decode such a buffer on receipt. Neither of these routines need be called if the user does not require non-contiguous messages, mixed data types, or heterogeneity. Having separate packing and unpacking routines allows send and receive to be optimized for a contiguous stream of bytes, and simplifies future extensions of MPI into areas such as heterogeneity. =============================================================================== For completeness here is the present proposal from the collective communication subcommittee. 1. info = MPI_BCAST( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return the contents of "buf" on "root" is contained in "buf" on all group members. On return "info" contains the error code. 2. info = MPI_GATHER( buf, bytes, type, gid, root ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "root". On return all the individual "buf" are concatenated into the "root" buf, which must be of size at least gsize*bytes where gsize is group size. The data is laid in the "root" buf in rank order that is | gid,0 data | gid,1 data | ...| gid, root data | ...| gid, gsize-1 data | Other member's "buf" are unchanged on return. On return "info" contains the error code. 3. info = MPI_GLOBAL_OP( inbuf, bytes, type, gid, op, outbuf ) Function: Called by all members of the group "gid" using the same arguments for "bytes", "type", "gid", and "op". On return the "outbuf" of all group members contains the result of the global operation "op" applied pointwise to the collective "inbuf". For example, if the op is max and inbuf contains two float point numbers then outbuf(1) = global max( inbuf(1)) and outbuf(2) = global max( inbuf(2)) A set of standard operations are supplied with MPI including: global max - for each data type global min - for each data type global sum - for each data type global mult- for each data type global AND - for integer and logical type global OR - for integer and logical type global XOR - for integer and logical type Optionally the users may define their own global functions for this routine. On return "info" contains the error code. 4. info = MPI_SYNCH( gid ) Function: Called by all members of the group "gid" Returns only when all members have called this function. On return "info" contains the error code. 5. gid = MPI_MKGROUP( list_of_processes ) Function: Called by all processes in the list. Forms a logical group containing the listed processes and assigns each process a unique rank in the group. The ranks are consecutively numbered from 0 to gsize-1. On return "gid" is an MPI assigned group ID (or error code if < 0) 6. gsize = MPI_GROUPSIZE( gid ) Function: Can be called by any process. On return "gsize" is the number of members in the group "gid" (or error code if < 0). 7. rank = MPI_MYRANK( gid ) Function: Can be called only by members of group "gid". On return "rank" is the rank of the calling process in group "gid" (an integer between 0 and gsize-1) or error code if < 0. =========================================================================== From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:50:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19230; Mon, 4 Jan 93 08:50:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07603; Mon, 4 Jan 93 08:49:52 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:49:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07594; Mon, 4 Jan 93 08:49:43 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06801 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:49:38 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15719; Mon, 4 Jan 93 13:49:35 GMT Date: Mon, 4 Jan 93 13:49:35 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041349.AA15719@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02659; Mon, 4 Jan 93 13:49:02 GMT To: mpi-pt2pt@cs.utk.edu Subject: Multifarious Content-Length: 6062 Gentlepeople, Firstly a general point ======================= I'd like to add an additional criterion to the goals of the message passing interface. 8) The implementation and semantics of the interface should remain simple and "obvious" when multiple "lightweight" threads share the same address space. I'd like this because 1) Lightweight thread libraries on the Posix model are becoming available on most workstations (networks of which are one of our targets). 2) There is a clear industry trend to having a few processors on a logically (and often physically) shared memory both in workstations (e.g. SparcStation 10), and servers (SparcServer 690, RS6000 ??). It would be foolish to go down a path which made exploitation of such machines hard. Undoubtedly this trend will also occur in future MPP machines (it would "only" take a software change to the code on the Paragon to make it be like this. After all it has all the hardware there already). As a consequence of this I'd like to remove as much hidden state as possible from the message passing system, so 1) I dislike the functions which return information about "the last message", since in a single process with multiple threads there can be many "last" messages. The suggestion that these operations be performed on descriptors returned from the probe is much preferable (and semantically cleaner). 2) I dislike the idea of setting options in a global routine, because there may be other threads performing operations at the same time as the options are changing, and the semantics becomes very hard to pin down. Secondly A few comments on Al Geist's proposal of Christmas Day =============================================================== 1. MPI_SEND I liked the separation in the original draft between the concepts of blocking (the buffer is now free) and synchronous (the message has been received at the other end) communications. I'd like to preserve both possibilities. 3. MPI_NRECV I don't understand how this is intended to work. Consider the following code. char buffer[100]; int i; for (i=0; i<10; i++) { if (&MPI_NRECV(buffer[10*i],10,1,1,1)) break; } Questions : 1) Into which elements of buffer will the receive occur ? Al's implementation note suggests just the first, so what then do the other (up to) 9 calls achieve ? Will they receive the next nine matching messages ? (this doesn't appear very intuitive for a "polling" routine) 2) Should I now do a blocking MPI_RECV as well ? Or does this only make sense if the MPI_NRECVs all fail ? But if they do fail, is the receive buffer still queued to receive a matching message when it becomes available ? (and if so how do I cancel the outstanding receive and get back ownership of my buffer ?) 4,5,6,7 See comments above about this style of working. 8,9 If we're only having one form of packing, then it should probably be the more general one with arrays of pointers and counts, since it can subsume (admittedly at some cost) the strided style. Conclusion ========== I prefer the original draft to Al's version. I'd like to change the mode argument in the original draft (see my mail to mpi-lang). Removing internal state ======================= I'd also like to remove the hidden state ("last message"), there are various ways of achieving this, none of which is exceptionally attractive. 1) Return a structure from MPI_RECV (and friends) This is fine in C, but not good in Fortran 77 (and may be more expensive than we want when you don't need any of the information). e.g. typedef struct MPI_REPLY { int status; int nbytes; int whoFrom; int tag; /* More ? */ } MPI_REPLY; { MPI_REPLY res; res = MPI_RECV(...); if (res.whoFrom == 0) ... etc ... } 2) Return results indirect through individual parameters (arguments for the Fortran inclined) This works, but is very error prone, particularly in Fortran when the arguments are INOUT. (We have an interface like this at the moment, and it does cause problems). 3) Return a tag on which queries can be made. This means that you must then have another call to the system to dispose of the tag (and associated system storage). This is unpleasant on a simple blocking receive when you don't want any of the information (since it's another call to the library). 4) Have additional (possibly optional) OUT arguments through which the results are returned if required. Using optional keyword arguments in F90 this works fine. C can be made to work more or less. F77 can't (within the standard). 5) Have one required argument which is a pointer to a reply structure like that used in 1) which is then filled in by the function. This can work in all the languages. In C you could also allow NULL to mean no such results are required. IMHO Solution 5 is best. If we wanted to (and we may) then we could disallow direct access to this structure and provide functions to pick it apart (these could be macros in C, inline functions in C++). This would avoid having to specify the contents of the structure and allow for future expansion. Comments on Steve Lederman's note ================================= > At the meeting in Minneapolis, several people, including myself, > discussed the advantage of having a send where the sending process > knows the receiving process has already posted a receive. We have this in our current message passing model. It has proved to be an awful mistake. 98% of user's don't know when to use it, and then complain that their code no longer works when they've used it incorrectly. Please let's not do this. (I'm speaking from experience here). See you all in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 08:59:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19312; Mon, 4 Jan 93 08:59:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08031; Mon, 4 Jan 93 08:59:34 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 13:59:32 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08019; Mon, 4 Jan 93 08:59:30 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA06920 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 08:59:25 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15744; Mon, 4 Jan 93 13:59:21 GMT Date: Mon, 4 Jan 93 13:59:21 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041359.AA15744@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02661; Mon, 4 Jan 93 13:58:46 GMT To: mpi-pt2pt@cs.utk.edu Subject: Incorrect Example Content-Length: 311 Appologies, the example should (of course) read for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 09:33:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19844; Mon, 4 Jan 93 09:33:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09262; Mon, 4 Jan 93 09:32:48 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 14:32:47 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09242; Mon, 4 Jan 93 09:32:36 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA07241 (5.65c/IDA-1.4.4); Mon, 4 Jan 1993 09:32:32 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA15874; Mon, 4 Jan 93 14:32:28 GMT Date: Mon, 4 Jan 93 14:32:28 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041432.AA15874@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02690; Mon, 4 Jan 93 14:31:52 GMT To: mpi-pt2pt@cs.utk.edu, mpi-lang@cs.utk.edu Subject: Profilers etc. Content-Length: 2528 Gentlepeople, I have an implementation issue which I would like to raise in the MPI forum, since it is unclear where it fits into the sub-committee structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to those of you who receive this message twice. Issue ===== The major objective of MPI1 is to achieve portability of applications. This has major benefits for us all (not least in legitimising and therefore growing the MPP marketplace). One of the benefits which it would also be nice to achieve would be the wide availability of different tools which support programming in the MPI1 model. The most immediately obvious such tools (to me at least !) are 1) HPF to Fortran + MPI1 translators 2) Performance monitoring/tuning tools 3) Debuggers Support for the first is easy (since it just requires what we're already doing). Portable support for the second is not so trivial, since the collection of useful performance information is much more intrusive. Portable support for the third is harder still, and I won't discuss it further. Options ======= We have various possible options which we can take. 1) Ignore the problem Provide no support for portable performance monitoring tools, and leave each tool provider with a large porting problem. I don't like this solution, it loses some of the benefit of the standard, which should be attracting people to build tools. 2) Document specific implementation hooks as part of MPI1. In effect these would be callbacks from the library to profiling code which could then do whatever it liked. 3) As 2, but without REQUIRING that a conforming implementation provide the functions. They're there as a recommendation, rather than being mandatory. I think we should be concerned about this, and I'd like us at least to make some recommendation. Personally I'd probably implement two separate interface to the library, one of which provided the hooks, and the other of which didn't so that you don't pay the cost of checking the profiling hooks unless you asked to. Of course even if we do nothing it's not too hard to escape (using horrible macros) in C, but Fortran doesn't always have macros, so a properly specified internal solution is definitely preferable. Thoughts ??? Flame me at Dallas. I'm travelling tomorrow. When are we having a meeting in Europe ??? -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:26:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21722; Mon, 4 Jan 93 10:26:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11937; Mon, 4 Jan 93 10:25:39 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:25:38 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11921; Mon, 4 Jan 93 10:25:36 -0500 Received: by msr.EPM.ORNL.GOV (5.61/1.34) id AA02891; Mon, 4 Jan 93 10:25:20 -0500 Date: Mon, 4 Jan 93 10:25:20 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9301041525.AA02891@msr.EPM.ORNL.GOV> To: jim@meiko.co.uk, mpi-pt2pt@cs.utk.edu Subject: NRECV example explained. Hi Jim, Your points are well taken given your suggested additional goals for MPI. You asked some questions about an example you gave. char buffer[100]; int i; for (i=0; i<10; i++) { if (MPI_NRECV(&buffer[10*i],10,1,1,1)) break; } > 1) Into which elements of buffer will the receive occur ? The MPI_NRECV call says I want to receive a message not to exceed 10 bytes which matches type = 1, groupID = 1, source = 1. If the message has already arrived before the NRECV call, then it will be placed in buffer[0:9]. If it arrives while the 'for' loop is still executing, then it will be placed in buffer[10*i:10*i+10]. If it arrives later, then the user will need to ask for it later with either MPI_NRECV or MPI_RECV. All he can say for sure is the message did not arrive while I was polling for it. It is a strange example in that the user asks for a single message but specifies 10 different places where it should be stored. Ultimately, it the successful RECV or NRECV that specifies where the message is stored. > 2) Should I now do a blocking MPI_RECV as well ? Or does this only > make sense if the MPI_NRECVs all fail ? But if they do fail, is > the receive buffer still queued to receive a matching message when > it becomes available ? (and if so how do I cancel the outstanding > receive and get back ownership of my buffer ?) It only makes sense to call MPI_RECV if all MPI_NRECVs fail. It is not specified that the recv buffer ever has to be queued. An efficient immplementation may queue a failed NRECV in the educated guess that the user will ask for the message later in the same buffer. Then when the user does ask and the RECV or NRECV is successful the system avoids one copy of the message. Technically the user never loses ownership of the buffer. He can do anything he wants with it. If he is sure the posted message will never arrive, then he can post a message from another source to that same buffer. Cheers! Al From owner-mpi-pt2pt@CS.UTK.EDU Mon Jan 4 10:44:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22293; Mon, 4 Jan 93 10:44:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12552; Mon, 4 Jan 93 10:44:12 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Mon, 04 Jan 1993 15:44:11 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12544; Mon, 4 Jan 93 10:44:08 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA08401 (5.65c/IDA-1.4.4 for ); Mon, 4 Jan 1993 10:44:05 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA16038; Mon, 4 Jan 93 15:44:00 GMT Date: Mon, 4 Jan 93 15:44:00 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9301041544.AA16038@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02725; Mon, 4 Jan 93 15:43:24 GMT To: geist@msr.EPM.ORNL.GOV Cc: mpi-pt2pt@cs.utk.edu In-Reply-To: Al Geist's message of Mon, 4 Jan 93 10:25:20 -0500 <9301041525.AA02891@msr.EPM.ORNL.GOV> Subject: NRECV example explained. Content-Length: 1350 Thanks Al, I now understand what you're trying to do, but I still don't think that your suggested "efficient implementation" is safe. Consider an example like this (less perverted than my previous one, which was of course constructed solely to make a point). strcpy(buffer0,"Something"); if (MPI_NRECV(buffer0, 10, 1, 1, 1)) /* Whatever */ else { sleep(20); /* Wait for the message to arrive, * For the sake of argument assume it does. */ MPI_RECV(buffer1, 10, 1, 1, 1); if (strcmp(buffer0,"Something") != SAME) bug("Buffer changed"); } Your suggested "efficient" implementation will surely overwrite buffer0 (which it should NOT do in your semantic model, since "Ultimately, it the successful RECV or NRECV that specifies where the message is stored."). I have to say I much prefer a model in which the queuing and dequeuing of buffers is explicit. Then it is clear where the current ownership of the buffer resides at any time, and when the additional copy can be avoided. (Though of course there is a lovely race to be resolved in the cancel...) Maybe we can talk about this further over a beer if you're going to be in Dallas. -- Jim James Cownie Meiko Limited 650 Aztec West Bristol BS12 4SD England Phone : +44 454 616171 FAX : +44 454 618188 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 12:37:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20368; Tue, 5 Jan 93 12:37:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13588; Tue, 5 Jan 93 12:37:19 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 17:37:18 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13576; Tue, 5 Jan 93 12:37:16 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA22236; Tue, 5 Jan 93 17:37:12 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00742; Tue, 5 Jan 93 10:36:12 MST Date: Tue, 5 Jan 93 10:36:12 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301051736.AA00742@macaw.fsl.noaa.gov> To: mpi-pt2pt@cs.utk.edu Subject: Re: Multifarious I have a few short comments on Jim's comments... > > Removing internal state > ======================= > > I'd also like to remove the hidden state ("last message"), there are > various ways of achieving this, none of which is exceptionally > attractive. > > 1) Return a structure from MPI_RECV (and friends) > This is fine in C, but not good in Fortran 77 (and may be more > expensive than we want when you don't need any of the information). > > e.g. > typedef struct MPI_REPLY > { > int status; > int nbytes; > int whoFrom; > int tag; > /* More ? */ > } MPI_REPLY; > > { > MPI_REPLY res; > > res = MPI_RECV(...); > > if (res.whoFrom == 0) > > ... etc ... > } > > > 2) Return results indirect through individual parameters (arguments for the > Fortran inclined) > > This works, but is very error prone, particularly in Fortran when > the arguments are INOUT. (We have an interface like this at the > moment, and it does cause problems). > > 3) Return a tag on which queries can be made. > > This means that you must then have another call to the system to > dispose of the tag (and associated system storage). This is > unpleasant on a simple blocking receive when you don't want any of > the information (since it's another call to the library). > > 4) Have additional (possibly optional) OUT arguments through which the > results are returned if required. > > Using optional keyword arguments in F90 this works fine. > C can be made to work more or less. > F77 can't (within the standard). > > 5) Have one required argument which is a pointer to a reply structure > like that used in 1) which is then filled in by the function. > > This can work in all the languages. > In C you could also allow NULL to mean no such results are > required. > > > IMHO Solution 5 is best. > > If we wanted to (and we may) then we could disallow direct access to > this structure and provide functions to pick it apart (these could be > macros in C, inline functions in C++). This would avoid having to > specify the contents of the structure and allow for future expansion. > > > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England I also like Solution 5 and like the idea of providing functions to access the reply structure. "Hiding" the data structure in this way makes it easy to change things in later revisions when we (inevitably) find out we've forgotten something. Tom Henderson Forecast Systems Laboratory (NOAA) hender@fsl.noaa.gov From owner-mpi-pt2pt@CS.UTK.EDU Tue Jan 5 15:52:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01504; Tue, 5 Jan 93 15:52:33 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22365; Tue, 5 Jan 93 15:51:40 -0500 X-Resent-To: mpi-pt2pt@CS.UTK.EDU ; Tue, 05 Jan 1993 20:51:34 GMT Errors-To: owner-mpi-pt2pt@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22325; Tue, 5 Jan 93 15:50:59 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA21624; Tue, 5 Jan 93 14:50:42 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA17798; Tue, 5 Jan 93 14:50:40 CST Date: Tue, 5 Jan 93 14:50:40 CST Message-Id: <9301052050.AA17798@godzilla.mcs.anl.gov> To: mpi-pt2pt@cs.utk.edu Subject: Comments on "ready receiver" part of draft implementation Here is some text supporting the idea of "ready receivers". Bill Gropp and Rusty Lusk %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: rr.dvi %%Pages: 3 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}B /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}{initclip}ifelse ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}B /@endspecial{grestore clear SpecialSave restore end}B /@defspecial{SDict begin}B /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}B /st{stroke SaveX SaveY moveto}B /fil{fill SaveX SaveY moveto}B /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 58 123 df<001F83E000F06E3001C078780380F8 780300F03007007000070070000700700007007000070070000700700007007000FFFFFF800700 700007007000070070000700700007007000070070000700700007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070007FE3FF001D20809F 1B>11 D<003F0000E0C001C0C00381E00701E00701E00700000700000700000700000700000700 00FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E07FC3FE1720809F19>I<7038F87CFC7EFC7E743A 0402040204020804080410081008201040200F0E7E9F17>34 D<70F8FCFC740404040808101020 40060E7C9F0D>39 D<0020004000800100020006000C000C001800180030003000300070006000 60006000E000E000E000E000E000E000E000E000E000E000E000E0006000600060007000300030 003000180018000C000C000600020001000080004000200B2E7DA112>I<800040002000100008 000C00060006000300030001800180018001C000C000C000C000E000E000E000E000E000E000E0 00E000E000E000E000E000C000C000C001C001800180018003000300060006000C000800100020 00400080000B2E7DA112>I<70F8FCFC74040404080810102040060E7C840D>44 DI<70F8F8F87005057C840D>I<000100030003000600060006000C000C 000C00180018001800300030003000600060006000C000C000C001800180018003000300030006 00060006000C000C000C00180018001800300030003000600060006000C000C000C000102D7DA1 17>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F003C0F003C0 F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0700380700380700380 7807803807001806001C0E000E1C0003F000121F7E9D17>I<018003800F80F380038003800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 80038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0F803C0F8 03C0F803C02007C00007C0000780000780000F00000E00001C0000380000700000600000C00001 80000300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F0000C1C00 100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C0000380003F000 003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F00780400780400F00 200E001C3C0003F000121F7E9D17>I<007C000182000701000E03800C07801C07803803003800 00780000700000700000F1F000F21C00F40600F80700F80380F80380F003C0F003C0F003C0F003 C0F003C07003C07003C07003803803803807001807000C0E00061C0001F000121F7E9D17>54 D<03F0000C0C001006003003002001806001806001806001807001807803003E03003F06001FC8 000FF00003F80007FC000C7E00103F00300F806003804001C0C001C0C000C0C000C0C000C0C000 806001802001001002000C0C0003F000121F7E9D17>56 D<70F8F8F87000000000000000000000 70F8F8F87005147C930D>58 D<000100000003800000038000000380000007C0000007C0000007 C0000009E0000009E0000009E0000010F0000010F0000010F00000207800002078000020780000 403C0000403C0000403C0000801E0000801E0000FFFE0001000F0001000F0001000F0002000780 0200078002000780040003C00E0003C01F0007E0FFC03FFE1F207F9F22>65 D<000FC040007030C001C009C0038005C0070003C00E0001C01E0000C01C0000C03C0000C07C00 00407C00004078000040F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000780000007C0000407C0000403C0000401C0000401E0000800E00008007000100 0380020001C0040000703800000FC0001A217D9F21>67 DI70 D73 D76 DII<001F800000F0F00001C0380007801E000F000F000E0007001E0007803C 0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0 F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E0007 800E0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C198010078030038060 0180600180E00180E00080E00080E00080F00000F000007800007F00003FF0001FFC000FFE0003 FF00001F800007800003C00003C00001C08001C08001C08001C08001C0C00180C00380E00300F0 0600CE0C0081F80012217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F0030 800F0010800F0010800F0010800F0010000F0000000F0000000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21>I87 D<080410082010201040204020804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>92 D<1FE000303000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E 00F00E10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00000E3E000EC3800F01C00F00E0 0E00E00E00700E00700E00780E00780E00780E00780E00780E00780E00700E00700E00E00F00E0 0D01C00CC300083E0015207F9F19>I<03F80E0C1C1E381E380C70007000F000F000F000F000F0 00F00070007000380138011C020E0C03F010147E9314>I<000380003F80000380000380000380 00038000038000038000038000038000038000038003E380061B801C0780380380380380700380 700380F00380F00380F00380F00380F00380F003807003807003803803803807801C07800E1B80 03E3F815207E9F19>I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF 80F00000F00000F000007000007000003800801800800C010007060001F80011147F9314>I<00 7C00C6018F038F07060700070007000700070007000700FFF00700070007000700070007000700 070007000700070007000700070007000700070007007FF01020809F0E>I<0000E003E3300E3C 301C1C30380E00780F00780F00780F00780F00780F00380E001C1C001E380033E0002000002000 003000003000003FFE001FFF800FFFC03001E0600070C00030C00030C00030C000306000603000 C01C038003FC00141F7F9417>I<0E0000FE00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E3E000E43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16207F9F19>I< 1C001E003E001E001C000000000000000000000000000E007E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000 E0000000000000000000000000007007F000F00070007000700070007000700070007000700070 007000700070007000700070007000700070007000706070F060F0C061803F000C28829E0E>I< 0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF0 0E03C00E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E00 0E07000E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E81C81C000F00F00E000F 00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00FFE7FE7F E023147F9326>I<0E3E00FE43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16147F9319>I<01F800 070E001C03803801C03801C07000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E0 7000E03801C03801C01C0380070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00 E00E00F00E00700E00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01 C00EC3000E3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319 >I<03E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380F0 03807003807803803803803807801C0B800E138003E38000038000038000038000038000038000 0380000380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E000E000E000E000E 000E000E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030704030C010C010C010E0 0078007F803FE00FF00070803880188018C018C018E030D0608F800D147E9312>I<0200020002 00060006000E000E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E080E08 0E080E080E080610031001E00D1C7F9B12>I<0E01C0FE1FC00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00603C0030DC001F1 FC16147F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C00 007C00004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000E001E001C00380078007010E01 1E011C0338027006700EFFFE10147F9314>I E /Fb 14 122 df<01FFFFC0001E00F0001E0078 001E0038001E003C003C003C003C003C003C003C003C003C0078007800780078007800F0007801 E000F0078000FFFE0000F00F8000F003C001E001C001E001E001E001E001E001E003C001E003C0 01E003C001E003C001C0078003C00780078007800F0007801E000F007800FFFFE0001E1F7D9E20 >66 D<0000FC040007030C001C00980030007800E0007801C00038038000300380003007000030 0E0000301E0000201E0000203C0000003C00000078000000780000007800000078000000F00000 00F000FFF0F0000780F0000780F0000F0070000F0070000F0070000F0070001E0038001E001800 3E001C002E000E00CC000383040000FC00001E217A9F23>71 D<01FFF800001F0000001E000000 1E0000001E0000003C0000003C0000003C0000003C000000780000007800000078000000780000 00F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0008003C0010003C001 0003C0030003C00200078006000780060007800C0007801C000F007800FFFFF800191F7D9E1D> 76 D<01FFFF00001E03C0001E00E0001E0070001E0078003C0078003C0078003C0078003C0078 007800F0007800F0007801E0007801C000F0070000F01E0000FFF00000F0380001E01C0001E01E 0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C0807803C080780 3C100F801C10FFF00C20000007C01D207D9E21>82 D<00C001E001E001C0000000000000000000 00000000000E003300230043804300470087000E000E000E001C001C001C003840388030807080 310033001C000B1F7C9E0E>105 D<01E0000FE00001C00001C00001C00001C000038000038000 0380000380000700000700000703C00704200E08E00E11E00E21E00E40C01C80001D00001E0000 1FC00038E000387000387000383840707080707080707080703100E03100601E0013207D9F15> 107 D<03C01FC0038003800380038007000700070007000E000E000E000E001C001C001C001C00 38003800380038007000700070007100E200E200E200E200640038000A207C9F0C>I<007C0001 C3000301800E01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F0 0700700F00700E0030180018700007C00013147C9317>111 D<01C1E002621804741C04781C04 701E04701E08E01E00E01E00E01E00E01E01C03C01C03C01C03C01C0380380780380700380E003 C1C0072380071E000700000700000E00000E00000E00000E00001C00001C0000FFC000171D8093 17>I<1C1E002661004783804787804707804703008E00000E00000E00000E00001C00001C0000 1C00001C000038000038000038000038000070000030000011147C9313>114 D<00FC030206010C030C070C060C000F800FF007F803FC003E000E700EF00CF00CE00840102060 1F8010147D9313>I<018001C0038003800380038007000700FFF007000E000E000E000E001C00 1C001C001C003800380038003820704070407080708031001E000C1C7C9B0F>I<0E00C03300E0 2301C04381C04301C04701C08703800E03800E03800E03801C07001C07001C07001C07101C0E20 180E20180E201C1E200C264007C38014147C9318>I<0E00C03300E02301C04381C04301C04701 C08703800E03800E03800E03801C07001C07001C07001C07001C0E00180E00180E001C1E000C3C 0007DC00001C00001C00003800F03800F07000E06000C0C0004380003E0000131D7C9316>121 D E /Fc 20 117 df45 D<000003800000000007C00000000007C0000000000FE0000000000FE0000000000FE000000000 1FF0000000001FF0000000003FF8000000003FF8000000003FF80000000073FC0000000073FC00 000000F3FE00000000E1FE00000000E1FE00000001C0FF00000001C0FF00000003C0FF80000003 807F80000007807FC0000007003FC0000007003FC000000E003FE000000E001FE000001E001FF0 00001C000FF000001FFFFFF000003FFFFFF800003FFFFFF80000780007FC0000700003FC000070 0003FC0000E00001FE0000E00001FE0001E00001FF0001C00000FF0001C00000FF00FFFE001FFF FEFFFE001FFFFEFFFE001FFFFE2F297EA834>65 D 72 D77 DI80 D<01FF800007FFF0000F81F8001FC0 7E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007 FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F80 7F00DF803F839FFC0FFF0FFC01FC03FC1E1B7E9A21>97 D<001FF80000FFFE0003F01F0007E03F 800FC03F801F803F803F801F007F800E007F0000007F000000FF000000FF000000FF000000FF00 0000FF000000FF000000FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007 E0070003F01E0000FFFC00001FE0001A1B7E9A1F>99 D<003FE00001FFF80003F07E0007C01F00 0F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFF E0FF000000FF000000FF0000007F0000007F0000007F0000003F8000E01F8000E00FC001C007E0 038003F81F0000FFFE00001FF0001B1B7E9A20>101 D<0007F0003FFC00FE3E01F87F03F87F03 F07F07F07F07F03E07F00007F00007F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007 F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007 F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF807FFF80182A7EA915>I< 00FF81F003FFE7F80FC1FE7C1F80FC7C1F007C383F007E107F007F007F007F007F007F007F007F 007F007F007F007F003F007E001F007C001F80FC000FC1F8001FFFE00018FF8000380000003800 00003C0000003E0000003FFFF8001FFFFF001FFFFF800FFFFFC007FFFFE01FFFFFF03E0007F07C 0001F8F80000F8F80000F8F80000F8F80000F87C0001F03C0001E01F0007C00FC01F8003FFFE00 007FF0001E287E9A22>II<07000F801FC0 3FE03FE03FE01FC00F8007000000000000000000000000000000FFE0FFE0FFE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFE FFFE0F2B7DAA14>I108 DII<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E0 7F0007F07F0007F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007 F8FF0007F87F0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003F E0001D1B7E9A22>I114 D<03FE300FFFF01E03F03800F0700070F00070 F00070F80070FC0000FFE0007FFE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007C E0003CF0003CF00038F80038FC0070FF01E0F7FFC0C1FF00161B7E9A1B>I<0070000070000070 0000700000F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F0 3807F03807F03807F03807F03803F03803F87001F86000FFC0001F8015267FA51B>I E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 117 7 a Fc(A)23 b(Note)f(on)i(Access)e(to)h(High-P)n(erformance)e (Message-P)n(assing)832 94 y(Proto)r(cols)869 240 y Fb(Bil)r(l)16 b(Gr)n(opp)864 297 y(R)o(usty)g(Lusk)526 353 y Fa(Mathematics)f(and)g (Computer)g(Science)i(Division)678 409 y(Argonne)e(National)h(Lab)q(oratory) 145 546 y(The)22 b(MPI)h(Implemen)o(tation)g(T)l(estb)q(ed)g(con)o(tains)g (what)f(are)g(called)i(there)e(\\Ready-Receiv)o(er")74 602 y(v)o(ersions)17 b(of)g(the)g(message-passing)g(primitiv)o(es.)27 b(The)17 b(purp)q(ose)h(of)e(this)i(brief)f(note)g(is)h(to)e(giv)o(e)h(some) 74 659 y(explanation)e(of)f(what)g(wh)o(y)f(w)o(e)h(think)h(something)g(lik)o (e)g(this)f(needs)h(to)f(b)q(e)h(considered)g(as)f(part)f(of)h(the)74 715 y(ev)o(en)o(tual)i(standard)e(and)i(to)e(rep)q(ort)h(on)g(some)g(exp)q (erimen)o(ts)h(that)f(supp)q(ort)g(this)h(opinion.)145 802 y(It)21 b(is)g(quite)h(p)q(ossible)h(to)e(enco)q(de)h(message-passing)f (algorithms)g(with)h(only)f(a)g(single)h(t)o(yp)q(e)g(of)74 858 y(send)d(and)g(receiv)o(e,)h(sa)o(y)e(a)g(blo)q(c)o(king)i(send)f(and)g (a)f(blo)q(c)o(king)i(receiv)o(e.)30 b(Most)18 b(of)g(the)g(v)m(ariations)h (are)74 914 y(to)c(enable)h(increased)g(p)q(erformance,)f(p)q(ossibly)i(at)d (the)h(exp)q(ense)i(of)d(increased)i(program)e(complexit)o(y)l(.)74 971 y(An)19 b(example)h(of)f(this)g(is)g(the)g(general)h(consensus)f(that)g (w)o(e)f(need)i(non-blo)q(c)o(king)h(send)e(and)g(receiv)o(e)74 1027 y(op)q(erations)c(in)i(order)d(to)h(o)o(v)o(erlap)g(computation)g(with)h (comm)o(unication.)145 1114 y(It)11 b(is)h(imp)q(ortan)o(t)f(not)g(to)g(stop) g(to)q(o)g(so)q(on,)g(ho)o(w)o(ev)o(er,)g(in)h(pro)o(viding)h(supp)q(ort)e (for)g(high)h(p)q(erformance.)74 1170 y(As)k(time)g(go)q(es)f(b)o(y)l(,)h(v)o (endors)f(will)j(pro)o(vide)e(op)q(erations)g(that)f(are)g(faster)g(than)g (the)h(ones)g(that)f(can)g(b)q(e)74 1227 y(sp)q(eci\014ed)i(with)d(the)g(op)q (erations)h(in)g(the)f(initial)j(standard)c(prop)q(osal.)20 b(It)15 b(is)f(p)q(ossible)i(to)e(an)o(ticipate)h(at)74 1283 y(least)g(some)g(of)g(these)h(op)q(erations)f(and)g(pro)o(vide)h(for)f(them)g (in)h(the)f(MPI)g(standard.)145 1370 y(One)j(example)g(o)q(ccurs)f(in)h(the)g (situation)f(when)h(receiv)o(es)g(ha)o(v)o(e)f(b)q(een)h(issued)g(prior)g(to) e(the)i(send)74 1426 y(op)q(eration)e(that)f(is)i(exp)q(ected)g(to)e(satisfy) h(them.)22 b(In)16 b(suc)o(h)g(a)g(case)g(a)f(considerable)j(amoun)o(t)d(of)h (hand-)74 1483 y(shaking)23 b(can)f(b)q(e)g(eliminated)i(since)f(the)f (bu\013ers)g(ha)o(v)o(e)g(already)g(b)q(een)h(allo)q(cated.)41 b(On)22 b(the)g(In)o(tel)74 1539 y(IPSC/860)c(and)g(the)g(Delta,)g(one)h(can) f(tak)o(e)f(adv)m(an)o(tage)h(of)g(this)g(b)o(y)g(using)h(\\force)f(t)o(yp)q (es".)28 b(Almost)74 1595 y(all)20 b(users)f(of)f(the)h(Delta)f(who)h(are)f (seeking)i(go)q(o)q(d)e(p)q(erformance)h(use)g(this)g(tec)o(hnique.)32 b(Its)19 b(name)g(is)74 1652 y(misleading,)d(since)f(it)f(do)q(esn't)g(ha)o (v)o(e)g(an)o(ything)g(to)f(do)h(with)h(t)o(yp)q(es.)k(The)14 b(imp)q(ortan)o(t)g(thing)g(to)g(realize)74 1708 y(is)19 b(that)f(it)i(is)f (not)f(just)h(an)f(arcane)h(v)o(endor-sp)q(eci\014c)i(optimization,)f(but)f (a)f(general)i(situation)f(that)74 1765 y(transcends)13 b(an)o(y)g (particular)h(implemen)o(tation.)20 b(The)13 b(general)h(situation)f(is)h (that)e(when)i(receiv)o(es)g(ha)o(v)o(e)74 1821 y(b)q(een)i(issued)h(ahead)e (of)g(time,)g(message)g(latency)h(can)f(b)q(e)h(greatly)f(decreased.)145 1908 y(W)l(e)e(conducted)h(some)f(preliminary)h(exp)q(erimen)o(ts)g(on)f(the) g(IPSC/860)g(with)g(the)g(MPI)g(implemen-)74 1964 y(tation)19 b(testb)q(ed,)g(using)h(the)f(\\rr")f(v)o(ersions)h(of)f(send)i(and)f(receiv) o(e)h(that)e(it)h(supplies.)33 b(\(\\rr")17 b(stands)74 2021 y(for)d(\\ready)g(receiv)o(er".\))20 b(The)15 b(test)f(program)f(just)i (ping-p)q(ongs)g(a)g(message)f(bac)o(k)g(and)h(forth)f(b)q(et)o(w)o(een)74 2077 y(t)o(w)o(o)20 b(no)q(des.)39 b(The)22 b(follo)o(wing)g(graphs)f(sho)o (w)g(the)h(p)q(erformance)f(impro)o(v)o(emen)o(ts)g(pro)o(vided)i(b)o(y)e (the)74 2134 y(ready-receiv)o(er)16 b(op)q(erations.)145 2220 y(This)g(next)f(test)f(sho)o(ws)h(the)g(b)q(eha)o(vior)h(in)g(sending)g (around)g(a)e(ring.)963 2790 y(1)p eop %%Page: 2 2 bop 74 2161 a @beginspecial 72 @llx 72 @lly 504 @urx 504 @ury 4320 @rwi @setspecial %%BeginDocument: ftime.ps .24 .24 scale /g0dict 40 dict def g0dict begin /m { moveto } bind def /a { rmoveto } bind def /l { lineto } bind def /v { 0 exch rlineto } bind def /h { 0 rlineto } bind def /s { stroke } bind def /n { newpath } bind def /r { rlineto } bind def /c { closepath } bind def /f { fill } bind def /p { copypage erasepage } def /g { setgray } bind def /b { newpath moveto lineto stroke } bind def /d { newpath moveto 0 exch rlineto stroke } bind def /e { newpath moveto 0 rlineto stroke } bind def /i { newpath moveto rlineto stroke } bind def /rs { dup stringwidth pop 0 exch sub 0 a show } bind def /cs { dup stringwidth pop 2 div 0 exch sub 0 a show } bind def /rshow { gsave currentpoint translate rotate 0 0 moveto show grestore } bind def /rrs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 0 exch sub 0 a show grestore } bind def /rcs { gsave currentpoint translate rotate 0 0 moveto dup stringwidth pop 2 div 0 exch sub 0 a show grestore } bind def 1440 569 569 e 36 569 569 d 18 710 569 d 36 850 569 d 18 991 569 d 36 1132 569 d 18 1272 569 d 36 1413 569 d 18 1553 569 d 36 1694 569 d 18 1834 569 d 36 1975 569 d n 568 558 m -4 -2 r -2 -3 r 560 546 l -4 v 562 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 568 558 m -3 -2 r -1 -1 r -1 -2 r 562 546 l -4 v 563 536 l 1 -3 r 1 -1 r 3 -1 r s n 571 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 817 553 m 1 -2 r -1 -1 r -1 1 r 2 v 1 2 r 1 1 r 4 2 r 5 h 4 -2 r 1 -1 r 2 -2 r -3 v -2 -2 r -3 -3 r -7 -3 r -2 -1 r -3 -2 r -1 -4 r -4 v s n 827 558 m 3 -2 r 1 -1 r 1 -2 r -3 v -1 -2 r -4 -3 r -5 -3 r s n 816 533 m 1 2 r 3 h 6 -3 r 4 h 2 1 r 2 2 r s n 820 535 m 6 -4 r 5 h 1 1 r 2 3 r 2 v s n 849 558 m -4 -2 r -2 -3 r 841 546 l -4 v 843 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 849 558 m -2 -2 r -2 -1 r -1 -2 r 843 546 l -4 v 844 536 l 1 -3 r 2 -1 r 2 -1 r s n 852 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 875 558 m -4 -2 r -3 -3 r 867 546 l -4 v 868 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 875 558 m -3 -2 r -1 -1 r -1 -2 r 868 546 l -4 v 870 536 l 1 -3 r 1 -1 r 3 -1 r s n 877 531 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -3 2 r s -24 1108 555 d -27 1110 558 d n 1110 558 m 1096 539 l 20 h s 9 1105 531 e n 1130 558 m -4 -2 r -2 -3 r 1123 546 l -4 v 1124 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1130 558 m -2 -2 r -2 -1 r -1 -2 r 1124 546 l -4 v 1125 536 l 1 -3 r 2 -1 r 2 -1 r s n 1133 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1156 558 m -4 -2 r -2 -3 r 1148 546 l -4 v 1150 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1156 558 m -3 -2 r -1 -1 r -1 -2 r 1150 546 l -4 v 1151 536 l 1 -3 r 1 -1 r 3 -1 r s n 1159 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1393 554 m -1 -1 r 1 -2 r 2 2 r 1 v -2 2 r -2 2 r -4 h -4 -2 r -2 -2 r -2 -3 r -1 -5 r -7 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 1 v -1 4 r -3 2 r -4 2 r -1 h -4 -2 r -2 -2 r -2 -4 r s n 1387 558 m -3 -2 r -2 -2 r -1 -3 r -2 -5 r -7 v 2 -4 r 2 -3 r 3 -1 r s n 1388 531 m 3 1 r 2 3 r 2 4 r 1 v -2 4 r -2 2 r -3 2 r s n 1411 558 m -4 -2 r -2 -3 r 1404 546 l -4 v 1405 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1411 558 m -2 -2 r -2 -1 r -1 -2 r 1405 546 l -4 v 1406 536 l 1 -3 r 2 -1 r 2 -1 r s n 1414 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1437 558 m -4 -2 r -2 -3 r 1429 546 l -4 v 1431 536 l 2 -4 r 4 -1 r 3 h 3 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -3 2 r c s n 1437 558 m -3 -2 r -1 -1 r -1 -2 r 1431 546 l -4 v 1432 536 l 1 -3 r 1 -1 r 3 -1 r s n 1440 531 m 2 1 r 1 1 r 2 3 r 1 6 r 4 v -1 7 r -2 2 r -1 1 r -2 2 r s n 1665 558 m -3 -2 r -2 -2 r -4 v 2 -2 r 3 -2 r 6 h 3 2 r 2 2 r 4 v -2 2 r -3 2 r c s n 1665 558 m -2 -2 r -1 -2 r -4 v 1 -2 r 2 -2 r s n 1671 546 m 2 2 r 1 2 r 4 v -1 2 r -2 2 r s n 1665 546 m -3 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 3 -1 r 6 h 3 1 r 2 1 r 1 3 r 5 v -1 3 r -2 1 r -3 1 r s n 1665 546 m -2 -1 r -1 -1 r -2 -3 r -5 v 2 -3 r 1 -1 r 2 -1 r s n 1671 531 m 2 1 r 1 1 r 2 3 r 5 v -2 3 r -1 1 r -2 1 r s n 1692 558 m -3 -2 r -3 -3 r 1685 546 l -4 v 1686 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1692 558 m -2 -2 r -1 -1 r -2 -2 r 1686 546 l -4 v 1687 536 l 2 -3 r 1 -1 r 2 -1 r s n 1695 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 1718 558 m -4 -2 r -2 -3 r 1710 546 l -4 v 1712 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 1718 558 m -2 -2 r -2 -1 r -1 -2 r 1712 546 l -4 v 1713 536 l 1 -3 r 2 -1 r 2 -1 r s n 1721 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 1931 553 m 3 1 r 4 4 r -27 v s -25 1936 556 d 12 1931 531 e n 1961 558 m -4 -2 r -3 -3 r 1953 546 l -4 v 1954 536 l 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 7 r -3 3 r -4 2 r c s n 1961 558 m -3 -2 r -1 -1 r -1 -2 r 1954 546 l -4 v 1956 536 l 1 -3 r 1 -1 r 3 -1 r s n 1963 531 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 7 r -1 2 r -1 1 r -3 2 r s n 1986 558 m -3 -2 r -3 -3 r 1979 546 l -4 v 1980 536 l 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 1986 558 m -2 -2 r -1 -1 r -2 -2 r 1980 546 l -4 v 1981 536 l 2 -3 r 1 -1 r 2 -1 r s n 1989 531 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -1 1 r -3 2 r s n 2012 558 m -4 -2 r -2 -3 r 2004 546 l -4 v 2006 536 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 7 r -2 3 r -4 2 r c s n 2012 558 m -2 -2 r -2 -1 r -1 -2 r 2006 546 l -4 v 2007 536 l 1 -3 r 2 -1 r 2 -1 r s n 2015 531 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s 1440 569 569 d 36 569 569 e 18 569 713 e 36 569 857 e 18 569 1001 e 36 569 1145 e 18 569 1289 e 36 569 1433 e 18 569 1577 e 36 569 1721 e 18 569 1865 e 36 569 2009 e n 528 585 m -4 -2 r -2 -3 r 521 573 l -4 v 522 563 l 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 7 r -2 3 r -4 2 r c s n 528 585 m -2 -2 r -2 -1 r -1 -2 r 522 573 l -4 v 523 563 l 1 -3 r 2 -1 r 2 -1 r s n 531 558 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 7 r -1 2 r -2 1 r -2 2 r s n 393 867 m 2 -1 r -2 -1 r -1 1 r 1 v 1 3 r 2 1 r 3 2 r 6 h 3 -2 r 2 -1 r 1 -3 r -2 v -1 -3 r -4 -2 r 398 857 l -2 -1 r -3 -3 r -1 -4 r -3 v s n 404 873 m 2 -2 r 1 -1 r 2 -3 r -2 v -2 -3 r -3 -2 r 398 857 l s n 392 848 m 1 1 r 3 h 6 -2 r 4 h 3 1 r 1 1 r s n 396 849 m 6 -3 r 5 h 2 1 r 1 2 r 3 v s n 425 873 m -3 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 425 873 m -2 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -1 r s n 428 846 m 3 1 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 2 r s n 451 873 m -4 -2 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 2 r c s n 451 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 454 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s n 477 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 477 873 m -3 -2 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -1 r s n 479 846 m 3 1 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 2 r s n 503 873 m -4 -2 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 2 r c s n 503 873 m -3 -2 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -1 r s n 505 846 m 3 1 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 2 r s n 528 873 m -4 -2 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 2 r c s n 528 873 m -2 -2 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -1 r s n 531 846 m 2 1 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 2 r s -25 404 1158 d -27 405 1160 d n 405 1160 m 391 1141 l 20 h s 9 400 1133 e n 425 1160 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 3 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1160 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 2 -2 r s n 428 1133 m 3 2 r 1 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1160 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 1 6 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 454 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1160 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -6 r 1 -3 r 1 -1 r 3 -2 r s n 479 1133 m 3 2 r 1 1 r 2 3 r 1 6 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1160 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -6 r 3 -4 r 4 -2 r 2 h 4 2 r 3 4 r 1 6 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1160 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -6 r 2 -3 r 1 -1 r 3 -2 r s n 505 1133 m 3 2 r 1 1 r 1 3 r 2 6 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1160 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -6 r 2 -4 r 4 -2 r 3 h 4 2 r 2 4 r 2 6 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1160 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -6 r 1 -3 r 2 -1 r 2 -2 r s n 531 1133 m 2 2 r 2 1 r 1 3 r 1 6 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 407 1444 m -1 -1 r 1 -1 r 2 1 r 1 v -2 3 r -2 1 r -4 h 397 1447 l -2 -3 r -2 -2 r -1 -5 r -8 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 1 v -1 4 r -3 3 r -4 1 r -1 h 397 1437 l -2 -3 r -2 -4 r s n 401 1448 m -3 -1 r -2 -3 r -1 -2 r -2 -5 r -8 v 2 -4 r 2 -2 r 3 -2 r s n 402 1421 m 3 2 r 2 2 r 2 4 r 1 v -2 4 r -2 3 r -3 1 r s n 425 1448 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 3 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1448 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 2 -2 r s n 428 1421 m 3 2 r 1 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1448 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 454 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1448 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -1 r 3 -2 r s n 479 1421 m 3 2 r 1 1 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1448 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -3 r 4 -2 r 2 h 4 2 r 3 3 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1448 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -1 r 3 -2 r s n 505 1421 m 3 2 r 1 1 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1448 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -3 r 4 -2 r 3 h 4 2 r 2 3 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1448 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -1 r 2 -2 r s n 531 1421 m 2 2 r 2 1 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 398 1736 m -3 -1 r -2 -3 r -4 v 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 4 v -2 3 r -3 1 r c s n 398 1736 m -2 -1 r -1 -3 r -4 v 1 -2 r 2 -1 r s n 404 1725 m 2 1 r 1 2 r 4 v -1 3 r -2 1 r s n 398 1725 m -3 -2 r -2 -1 r -1 -3 r -5 v 1 -2 r 2 -2 r 3 -1 r 6 h 3 1 r 2 2 r 1 2 r 5 v -1 3 r -2 1 r -3 2 r s n 398 1725 m -2 -2 r -1 -1 r -2 -3 r -5 v 2 -2 r 1 -2 r 2 -1 r s n 404 1709 m 2 1 r 1 2 r 2 2 r 5 v -2 3 r -1 1 r -2 2 r s n 425 1736 m -3 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 425 1736 m -2 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1709 m 3 1 r 1 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -1 1 r -3 1 r s n 451 1736 m -4 -1 r -2 -4 r -2 -6 r -4 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 4 v -1 6 r -2 4 r -4 1 r c s n 451 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 477 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 477 1736 m -3 -1 r -1 -1 r -1 -3 r -2 -6 r -4 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1709 m 3 1 r 1 2 r 2 2 r 1 7 r 4 v -1 6 r -2 3 r -1 1 r -3 1 r s n 503 1736 m -4 -1 r -3 -4 r -1 -6 r -4 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 4 v -1 6 r -3 4 r -4 1 r c s n 503 1736 m -3 -1 r -1 -1 r -2 -3 r -1 -6 r -4 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1709 m 3 1 r 1 2 r 1 2 r 2 7 r 4 v -2 6 r -1 3 r -1 1 r -3 1 r s n 528 1736 m -4 -1 r -2 -4 r -1 -6 r -4 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 4 v -2 6 r -2 4 r -4 1 r c s n 528 1736 m -2 -1 r -2 -1 r -1 -3 r -1 -6 r -4 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1709 m 2 1 r 2 2 r 1 2 r 1 7 r 4 v -1 6 r -1 3 r -2 1 r -2 1 r s n 370 2019 m 3 1 r 4 4 r -27 v s -26 375 2023 d 12 370 1997 e n 400 2024 m 396 2023 l -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 400 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 402 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 425 2024 m -3 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 3 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 425 2024 m -2 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 2 -1 r s n 428 1997 m 3 1 r 1 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -1 2 r -3 1 r s n 451 2024 m -4 -1 r -2 -4 r -2 -7 r -3 v 2 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 1 7 r 3 v -1 7 r -2 4 r -4 1 r c s n 451 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 454 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s n 477 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 477 2024 m -3 -1 r -1 -2 r -1 -2 r -2 -7 r -3 v 2 -7 r 1 -2 r 1 -2 r 3 -1 r s n 479 1997 m 3 1 r 1 2 r 2 2 r 1 7 r 3 v -1 7 r -2 2 r -1 2 r -3 1 r s n 503 2024 m -4 -1 r -3 -4 r -1 -7 r -3 v 1 -7 r 3 -4 r 4 -1 r 2 h 4 1 r 3 4 r 1 7 r 3 v -1 7 r -3 4 r -4 1 r c s n 503 2024 m -3 -1 r -1 -2 r -2 -2 r -1 -7 r -3 v 1 -7 r 2 -2 r 1 -2 r 3 -1 r s n 505 1997 m 3 1 r 1 2 r 1 2 r 2 7 r 3 v -2 7 r -1 2 r -1 2 r -3 1 r s n 528 2024 m -4 -1 r -2 -4 r -1 -7 r -3 v 1 -7 r 2 -4 r 4 -1 r 3 h 4 1 r 2 4 r 2 7 r 3 v -2 7 r -2 4 r -4 1 r c s n 528 2024 m -2 -1 r -2 -2 r -1 -2 r -1 -7 r -3 v 1 -7 r 1 -2 r 2 -2 r 2 -1 r s n 531 1997 m 2 1 r 2 2 r 1 2 r 1 7 r 3 v -1 7 r -1 2 r -2 2 r -2 1 r s 1440 569 2009 e -36 569 2009 d -18 710 2009 d -36 850 2009 d -18 991 2009 d -36 1132 2009 d -18 1272 2009 d -36 1413 2009 d -18 1553 2009 d -36 1694 2009 d -18 1834 2009 d -36 1975 2009 d 1440 2009 569 d -36 2009 569 e -18 2009 713 e -36 2009 857 e -18 2009 1001 e -36 2009 1145 e -18 2009 1289 e -36 2009 1433 e -18 2009 1577 e -36 2009 1721 e -18 2009 1865 e -36 2009 2009 e -27 1139 510 d -27 1141 510 d 6 1135 510 e 9 1135 483 e n 1152 494 m 16 h 2 v -2 3 r -1 1 r -3 1 r -3 h -4 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r s n 1166 494 m 3 v -1 3 r s n 1159 501 m -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r s -18 1178 501 d -18 1179 501 d n 1179 497 m 3 3 r 4 1 r 2 h 4 -1 r 1 -3 r -14 v s n 1188 501 m 3 -1 r 1 -3 r -14 v s 5 1174 501 e 9 1174 483 e 9 1188 483 e n 1210 501 m -3 -1 r -1 -1 r -1 -3 r -2 v 1 -3 r 1 -1 r 3 -2 r 2 h 3 2 r 1 1 r 2 3 r 2 v -2 3 r -1 1 r -3 1 r c s n 1207 500 m -1 -3 r -5 v 1 -2 r s n 1215 490 m 1 2 r 5 v -1 3 r s n 1216 499 m 2 1 r 2 1 r -1 v -2 h s n 1206 491 m -1 -1 r -2 -3 r -1 v 2 -3 r 4 -1 r 6 h 4 -1 r 1 -2 r s n 1203 486 m 2 -1 r 4 -2 r 6 h 4 -1 r 1 -3 r -1 v -1 -2 r -4 -2 r -8 h -4 2 r -1 2 r 1 v 1 3 r 4 1 r s n 1230 510 m -22 v 2 -3 r 2 -2 r 3 h 2 2 r 2 2 r s n 1232 510 m -22 v 1 -3 r 1 -2 r s 10 1227 501 e -27 1250 510 d -27 1251 510 d n 1251 497 m 3 3 r 3 1 r 3 h 4 -1 r 1 -3 r -14 v s n 1260 501 m 3 -1 r 1 -3 r -14 v s 5 1246 510 e 9 1246 483 e 9 1260 483 e n 1306 515 m -2 -2 r -3 -4 r -2 -5 r 1297 497 l -5 v 1299 486 l 1301 481 l 3 -4 r 2 -3 r s n 1304 513 m -3 -5 r -1 -4 r 1299 497 l -5 v 1300 486 l 1 -4 r 1304 477 l s -27 1317 510 d -27 1318 510 d n 1318 497 m 2 3 r 3 1 r 3 h 3 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -3 -2 r -3 h -3 2 r -2 2 r s n 1326 501 m 2 -1 r 3 -3 r 1 -3 r -3 v -1 -4 r -3 -2 r -2 -2 r s 5 1313 510 e 8 -18 1342 501 i 6 -15 1344 501 i n 1358 501 m 1350 483 l 1347 478 l -2 -2 r -3 -2 r -1 h -1 2 r 1 1 r 1 -1 r s 7 1340 501 e 7 1353 501 e n 1368 510 m -22 v 1 -3 r 3 -2 r 2 h 3 2 r 1 2 r s n 1369 510 m -22 v 2 -3 r 1 -2 r s 10 1364 501 e n 1386 494 m 15 h 2 v -1 3 r -1 1 r -3 1 r -4 h -3 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 3 -2 r 3 h 4 2 r 2 2 r s n 1400 494 m 3 v -1 3 r s n 1392 501 m -2 -1 r -3 -3 r -1 -3 r -3 v 1 -4 r 3 -2 r 2 -2 r s n 1422 499 m 1 2 r -5 v -1 3 r -1 1 r -3 1 r -5 h -3 -1 r -1 -1 r -3 v 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -2 r s n 1409 497 m 1 -1 r 3 -1 r 6 -3 r 3 -1 r 1 -1 r -4 v -1 -1 r -3 -2 r -5 h -2 2 r -2 1 r -1 2 r -5 v 1 3 r s n 1431 515 m 3 -2 r 2 -4 r 3 -5 r 1440 497 l -5 v 1439 486 l 1436 481 l -2 -4 r -3 -3 r s n 1434 513 m 2 -5 r 1 -4 r 1439 497 l -5 v 1437 486 l -1 -4 r 1434 477 l s n 870 2060 m 2 -5 r 10 v -2 -5 r -3 3 r -6 2 r -3 h -5 -2 r -4 -3 r -1 -3 r -2 -6 r -8 v 2 -5 r 1 -4 r 4 -3 r 5 -2 r 3 h 6 2 r 3 3 r 2 4 r s n 858 2065 m -3 -2 r -4 -3 r -2 -3 r -1 -6 r -8 v 1 -5 r 2 -4 r 4 -3 r 3 -2 r s n 892 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 892 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 896 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 920 2053 d -24 921 2053 d n 921 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 933 2053 m 4 -2 r 2 -3 r -19 v s n 940 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 952 2053 m 4 -2 r 1 -3 r -19 v s 6 915 2053 e 12 915 2029 e 12 933 2029 e 12 952 2029 e -24 976 2053 d -24 978 2053 d n 978 2048 m 3 3 r 6 2 r 3 h 5 -2 r 2 -3 r -19 v s n 990 2053 m 3 -2 r 2 -3 r -19 v s n 997 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1009 2053 m 3 -2 r 2 -3 r -19 v s 7 971 2053 e 12 971 2029 e 12 990 2029 e 12 1009 2029 e n 1033 2053 m -19 v 2 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1035 2053 m -19 v 1 -3 r 4 -2 r s -24 1052 2053 d -24 1053 2053 d 7 1028 2053 e 6 1047 2053 e 7 1052 2029 e -24 1071 2053 d -24 1072 2053 d n 1072 2048 m 4 3 r 5 2 r 3 h 5 -2 r 2 -3 r -19 v s n 1084 2053 m 4 -2 r 1 -3 r -19 v s 7 1065 2053 e 12 1065 2029 e 12 1084 2029 e n 1108 2065 m -2 -2 r 2 -1 r 2 1 r c s -24 1108 2053 d -24 1110 2053 d 7 1103 2053 e 12 1103 2029 e n 1144 2048 m -2 -2 r 2 -1 r 2 1 r 2 v -4 3 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1134 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s n 1160 2050 m -2 v -2 h 2 v 2 1 r 3 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1175 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1175 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 4 2 r 3 3 r s n 1163 2043 m -3 -2 r -2 -3 r -4 v 2 -3 r 3 -2 r s n 1194 2065 m -29 v 2 -5 r 3 -2 r 3 h 4 2 r 2 3 r s n 1196 2065 m -29 v 1 -5 r 2 -2 r s 13 1189 2053 e n 1220 2065 m -2 -2 r 2 -1 r 1 1 r c s -24 1220 2053 d -24 1221 2053 d 7 1214 2053 e 12 1214 2029 e n 1245 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1245 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1249 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1273 2053 d -24 1274 2053 d n 1274 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1286 2053 m 4 -2 r 2 -3 r -19 v s 6 1268 2053 e 12 1268 2029 e 12 1286 2029 e n 1324 2050 m 2 3 r -7 v -2 4 r -2 1 r -3 2 r -7 h -3 -2 r -2 -1 r -4 v 2 -1 r 3 -2 r 9 -4 r 3 -1 r 2 -2 r s n 1307 2048 m 2 -2 r 3 -1 r 9 -4 r 3 -2 r 2 -1 r -5 v -2 -2 r -3 -2 r -7 h -4 2 r -1 2 r -2 3 r -7 v 2 4 r s -36 1367 2065 d -36 1369 2065 d n 1362 2065 m 20 h 5 -2 r 2 -1 r 2 -4 r -5 v -2 -3 r -2 -2 r -5 -2 r -13 h s n 1382 2065 m 4 -2 r 1 -1 r 2 -4 r -5 v -2 -3 r -1 -2 r -4 -2 r s 12 1362 2029 e n 1403 2043 m 20 h 3 v -1 4 r -2 1 r -3 2 r -6 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1422 2043 m 5 v -2 3 r s n 1411 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s -24 1437 2053 d -24 1439 2053 d n 1439 2043 m 2 5 r 3 3 r 3 2 r 6 h 1 -2 r -1 v -1 -2 r -2 2 r 2 1 r s 7 1432 2053 e 12 1432 2029 e n 1475 2063 m -2 -1 r 2 -2 r 2 2 r 1 v -2 2 r -4 h -3 -2 r -2 -3 r -31 v s n 1471 2065 m -1 -2 r -2 -3 r -31 v s 14 1461 2053 e 12 1461 2029 e n 1495 2053 m -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r 2 5 r 4 v -2 5 r -3 3 r -5 2 r c s n 1495 2053 m -3 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 3 -2 r s n 1499 2029 m 3 2 r 4 3 r 1 5 r 4 v -1 5 r -4 3 r -3 2 r s -24 1523 2053 d -24 1525 2053 d n 1525 2043 m 1 5 r 4 3 r 3 2 r 5 h 2 -2 r -1 v -2 -2 r -1 2 r 1 1 r s 7 1518 2053 e 12 1518 2029 e -24 1552 2053 d -24 1554 2053 d n 1554 2048 m 3 3 r 5 2 r 4 h 5 -2 r 2 -3 r -19 v s n 1566 2053 m 3 -2 r 2 -3 r -19 v s n 1573 2048 m 3 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1584 2053 m 4 -2 r 2 -3 r -19 v s 7 1547 2053 e 12 1547 2029 e 12 1566 2029 e 12 1584 2029 e n 1608 2050 m -2 v -1 h 2 v 1 1 r 4 2 r 7 h 3 -2 r 2 -1 r 2 -4 r -12 v 1 -3 r 2 -2 r s n 1624 2050 m -16 v 2 -3 r 3 -2 r 2 h s n 1624 2046 m -2 -1 r -10 -2 r -5 -2 r -2 -3 r -4 v 2 -3 r 5 -2 r 5 h 3 2 r 4 3 r s n 1612 2043 m -4 -2 r -1 -3 r -4 v 1 -3 r 4 -2 r s -24 1643 2053 d -24 1644 2053 d n 1644 2048 m 4 3 r 5 2 r 3 h 6 -2 r 1 -3 r -19 v s n 1656 2053 m 4 -2 r 2 -3 r -19 v s 6 1638 2053 e 12 1638 2029 e 12 1656 2029 e n 1698 2048 m -2 -2 r 2 -1 r 1 1 r 2 v -3 3 r -4 2 r -5 h -5 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 5 -2 r 4 h 5 2 r 3 3 r s n 1687 2053 m -3 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 3 -2 r s n 1711 2043 m 21 h 3 v -2 4 r -2 1 r -3 2 r -5 h -5 -2 r -4 -3 r -1 -5 r -4 v 1 -5 r 4 -3 r 5 -2 r 3 h 5 2 r 4 3 r s n 1730 2043 m 5 v -2 3 r s n 1720 2053 m -4 -2 r -3 -3 r -2 -5 r -4 v 2 -5 r 3 -3 r 4 -2 r s 27 320 1110 e 27 320 1112 e n 320 1106 m 16 v 1 4 r 2 1 r 2 1 r 3 h 2 -1 r 2 -1 r 1 -4 r -10 v s n 320 1122 m 1 2 r 2 2 r 2 1 r 3 h 2 -1 r 2 -2 r 1 -2 r s 9 347 1106 d n 333 1118 m 1 3 r 1 1 r 9 4 r 2 1 r 1 v -2 2 r s n 334 1121 m 3 1 r 9 2 r 1 2 r 2 v -3 2 r -1 h s n 332 1139 m 1 h -2 v -1 h -2 2 r -1 2 r 5 v 1 3 r 2 1 r 2 1 r 9 h 3 2 r 1 1 r s n 332 1150 m 11 h 3 1 r 1 3 r 1 v s n 334 1150 m 1 -1 r 2 -8 r 1 -4 r 3 -1 r 2 h 3 1 r 1 4 r 4 v -1 3 r -3 2 r s n 337 1141 m 1 -2 r 3 -2 r 2 h 3 2 r 1 2 r s n 320 1164 m 22 h 4 2 r 1 2 r 3 v -1 2 r -3 2 r s n 320 1166 m 22 h 4 1 r 1 1 r s 11 329 1160 d n 337 1182 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1196 m -4 h -3 -1 r s n 329 1189 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 315 1236 m 2 -2 r 4 -3 r 5 -2 r 7 -2 r 5 h 6 2 r 6 2 r 3 3 r 3 2 r s n 317 1234 m 6 -3 r 3 -1 r 7 -1 r 5 h 6 1 r 4 1 r 5 3 r s 27 320 1247 e 27 320 1248 e n 333 1248 m -3 2 r -1 3 r 3 v 1 3 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -3 r -3 v -1 -3 r -3 -2 r s n 329 1256 m 1 2 r 3 3 r 4 1 r 2 h 4 -1 r 3 -3 r 1 -2 r s 5 320 1243 d 18 8 329 1272 i 15 6 329 1274 i n 329 1288 m 18 -8 r 5 -3 r 3 -2 r 1 -3 r -1 v -1 -1 r -2 1 r 2 1 r s 7 329 1270 d 7 329 1283 d n 320 1298 m 22 h 4 1 r 1 3 r 2 v -1 3 r -3 1 r s n 320 1299 m 22 h 4 2 r 1 1 r s 10 329 1294 d n 337 1316 m 15 v -3 h -2 -1 r -2 -1 r -1 -3 r -4 v 1 -3 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 3 r 3 v -1 4 r -3 2 r s n 337 1330 m -4 h -3 -1 r s n 329 1322 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 332 1352 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1339 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s 356 1360 315 1383 b n 332 1402 m -3 1 r 5 h -2 -1 r -2 -1 r -1 -3 r -5 v 1 -3 r 2 -1 r 2 h 1 1 r 2 3 r 2 6 r 2 3 r 1 1 r s n 333 1389 m 1 1 r 1 3 r 3 6 r 1 3 r 2 1 r 3 h 2 -1 r 1 -3 r -5 v -1 -2 r -2 -2 r -2 -1 r 5 h -3 1 r s n 337 1412 m 16 v -3 h -2 -2 r -2 -1 r -1 -2 r -4 v 1 -4 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 4 r 2 v -1 4 r -3 3 r s n 337 1426 m -4 h -3 -1 r s n 329 1419 m 1 -3 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 3 r s n 333 1451 m 1 -1 r 1 1 r -1 1 r -1 h -3 -2 r -1 -3 r -4 v 1 -4 r 3 -2 r 4 -2 r 2 h 4 2 r 3 2 r 1 4 r 3 v -1 4 r -3 2 r s n 329 1443 m 1 -2 r 3 -3 r 4 -1 r 2 h 4 1 r 3 3 r 1 2 r s n 315 1460 m 2 2 r 4 3 r 5 3 r 7 1 r 5 h 6 -1 r 6 -3 r 3 -3 r 3 -2 r s n 317 1462 m 6 3 r 3 1 r 7 2 r 5 h 6 -2 r 4 -1 r 5 -3 r s 657 974 614 806 b 20 -7 647 983 i 21 -7 647 984 i n 657 981 m -1 2 r 3 v 2 v 2 2 r 3 1 r 3 h 2 h 3 -2 r 1 -3 r -3 v -1 -2 r -2 -2 r -2 -1 r s n 656 988 m 2 1 r 3 1 r 3 h 2 h 2 -2 r 1 -3 r 1 -2 r s 1 4 646 980 i 21 -8 654 1004 i 20 -7 655 1004 i 2 3 653 1001 i 2 7 674 993 i n 666 1016 m -4 v 2 -2 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 2 r -2 2 r -2 1 r -3 h -3 -1 r -2 -3 r c s n 666 1016 m 1 -3 r 1 -2 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s n 681 1013 m -1 2 r -1 3 r -2 2 r -2 h -4 h -2 -1 r -2 -2 r s n 678 1040 m 1 -2 r 1 1 r 1 v -1 1 r -3 -2 r -2 -1 r -1 -3 r -3 v 2 -3 r 2 -2 r 2 -1 r 3 h 3 1 r 2 3 r 1 2 r 3 v -2 3 r s n 673 1035 m 1 -2 r 1 -3 r 2 -2 r 2 -1 r 4 h 2 1 r 2 2 r s 20 -8 672 1052 i 21 -8 672 1053 i 6 -13 683 1060 i 9 3 687 1053 i 10 3 686 1052 i 1 4 671 1049 i 3 6 681 1057 i 3 6 691 1042 i 697 1058 695 1052 b n 679 1072 m 1 -1 r 1 h 2 v c s 14 -5 686 1070 i 14 -5 686 1071 i 1 4 685 1067 i 701 1069 699 1062 b 13 -5 690 1080 i 14 -5 690 1081 i n 693 1080 m -1 3 r 3 v 1 2 r 2 3 r 2 h 11 -4 r s n 693 1088 m 1 2 r 3 h 10 -4 r s 1 4 689 1077 i 3 7 702 1072 i 709 1090 706 1083 b n 699 1104 m -2 v 1 -1 r 1 -2 r 2 -1 r 2 1 r 2 h 1 2 r 1 2 r 2 v -1 1 r -1 2 r -2 1 r -3 h -1 -1 r -2 -2 r c s n 699 1102 m 1 -1 r 4 -2 r 3 h s n 709 1105 m -2 2 r -4 1 r -2 h s n 702 1109 m 1 v -1 2 r 1 h -2 v s n 705 1099 m 1 -2 r 2 -1 r -1 v 3 1 r 2 2 r 2 5 r 2 3 r 1 h s n 708 1095 m 2 1 r 2 3 r 2 4 r 2 3 r 2 h 1 h 1 -2 r -3 v -2 -6 r -2 -2 r -2 -1 r -1 1 r -2 1 r 4 v s 41 -155 708 1114 i 7 10 749 959 i 18 -12 755 991 i 18 -12 756 992 i n 764 986 m 3 v 2 v 1 2 r 3 2 r 3 h 3 -1 r 1 -1 r 2 -2 r 1 -3 r -1 -3 r -1 -2 r -2 -1 r -3 -1 r s n 765 993 m 2 1 r 3 h 3 h 2 -2 r 2 -2 r -3 v -2 v s 756 992 754 988 b 18 -12 767 1009 i 18 -12 768 1010 i 2 4 766 1006 i 787 1000 783 994 b n 782 1018 m -1 -4 r 1 -2 r 2 -3 r 1 -1 r 3 -1 r 3 h 3 2 r 1 2 r 1 3 r -1 3 r -2 3 r -1 1 r -4 1 r -2 -1 r -3 -2 r c s n 782 1018 m -3 v -3 v 2 -2 r 2 -1 r 3 -1 r 3 h 2 1 r s n 795 1011 m 2 v 3 v -2 3 r -2 1 r -3 1 r -3 -1 r -2 -1 r s n 799 1038 m 1 -1 r 1 h 1 v -1 1 r -3 -1 r -2 -1 r -2 -2 r -1 -4 r 1 -2 r 2 -3 r 2 -1 r 3 -1 r 3 1 r 2 2 r 1 1 r 1 3 r 3 v s n 793 1035 m -3 v 1 -3 r 2 -2 r 1 -1 r 3 -1 r 3 h 2 2 r s 18 -12 796 1051 i 17 -12 797 1052 i 3 -15 808 1057 i 10 1 811 1049 i 10 1 810 1048 i 3 3 794 1049 i 3 5 807 1054 i 4 6 812 1037 i 822 1051 818 1046 b n 808 1069 m -1 v 2 h -1 2 r c s 12 -8 814 1065 i 12 -8 815 1066 i 3 3 812 1063 i 828 1061 824 1055 b 12 -8 820 1075 i 12 -9 821 1076 i n 823 1074 m 3 v 1 3 r 1 1 r 2 2 r 3 h 9 -6 r s n 825 1081 m 2 2 r 2 -1 r 10 -6 r s 2 4 819 1072 i 4 6 831 1064 i 841 1079 837 1073 b n 835 1096 m -1 -2 r 1 -2 r 1 -2 r 1 -1 r 3 h 1 h 2 1 r 1 2 r 1 2 r -1 2 r -1 2 r -1 1 r -3 h -1 h -2 -1 r c s n 834 1094 m 2 -2 r 3 -3 r 2 h s n 845 1094 m -1 2 r -4 3 r -2 h s n 839 1099 m 2 v 2 v 1 -1 r -1 -1 r s n 840 1089 m -2 v 1 -2 r 1 h 2 h 3 2 r 3 4 r 2 2 r 2 h s n 842 1085 m 1 h 3 2 r 3 4 r 2 2 r 3 h 1 -1 r 1 -2 r -1 -3 r -4 -5 r -2 -2 r -2 1 r -1 h -1 2 r 3 v s 32 42 852 1113 i 45 52 884 1155 i 36 36 929 1207 i 16 -15 968 1264 i 15 -15 969 1265 i n 976 1258 m 3 v 1 2 r 2 1 r 2 2 r 3 h 3 -2 r 2 -1 r 1 -3 r -3 v -1 -3 r -2 -1 r -2 -1 r -3 h s n 979 1264 m 2 1 r 3 h 3 -1 r 1 -2 r 1 -3 r -3 v -2 v s 969 1265 966 1262 b 16 -15 983 1280 i 16 -15 984 1280 i 3 3 981 1277 i 1002 1267 997 1262 b n 999 1286 m -1 -3 r -3 v 1 -3 r 2 -2 r 3 -1 r 3 h 3 2 r 1 1 r 2 3 r -1 3 r -1 3 r -1 1 r -3 2 r -3 h -3 -2 r c s n 999 1286 m -3 v -2 v 1 -3 r 2 -2 r 3 -1 r 2 h 3 1 r s n 1011 1277 m 1 2 r 3 v -2 3 r -1 1 r -3 2 r -3 h -2 -1 r s n 1020 1303 m -2 v 2 h 2 v -1 h -3 h -2 h -2 -3 r -2 -3 r -2 v 2 -3 r 1 -2 r 3 -1 r 3 h 3 1 r 1 2 r 2 3 r 3 v s n 1014 1300 m -1 -2 r -3 v 2 -3 r 1 -1 r 3 -2 r 3 1 r 2 h s 16 -15 1019 1316 i 16 -15 1020 1317 i 1 -14 1032 1319 i 10 -1 1033 1311 i 10 -2 1033 1311 i 3 3 1017 1314 i 5 5 1030 1317 i 5 5 1033 1299 i 1045 1312 1041 1307 b n 1035 1332 m -2 v 1 h 2 v c s 10 -10 1040 1327 i 11 -11 1040 1328 i 3 3 1037 1325 i 1053 1320 1048 1314 b 10 -10 1048 1335 i 11 -10 1048 1336 i n 1050 1333 m 3 v 2 3 r 1 2 r 3 1 r 2 h 8 -8 r s n 1053 1341 m 2 h 3 h 8 -8 r s 3 3 1045 1333 i 5 5 1056 1323 i 1069 1336 1064 1331 b n 1066 1353 m -1 -2 r -1 v 1 -3 r 1 -1 r 2 -1 r 2 h 2 1 r 1 1 r 1 3 r 1 v -1 2 r -1 2 r -2 h -2 h -2 h c s n 1065 1351 m 1 -2 r 3 -3 r 2 -1 r s n 1075 1350 m -1 2 r -3 3 r -2 h s n 1071 1355 m 2 v 2 v 1 -1 r -1 -1 r s n 1069 1345 m -1 v 1 -2 r 1 -1 r 2 -1 r 3 2 r 3 3 r 3 2 r 2 h s n 1071 1341 m 1 h 3 1 r 4 4 r 3 2 r 2 -1 r 1 -1 r -2 v -1 -3 r -4 -4 r -3 -2 r -3 1 r 1 v -1 2 r 1 3 r s 23 22 1086 1366 i 45 37 1109 1388 i 45 34 1154 1425 i 21 15 1199 1459 i 11 -19 1229 1494 i 11 -19 1230 1494 i n 1235 1485 m 1 3 r 1 2 r 2 1 r 3 1 r 3 -1 r 3 -2 r 1 -2 r -3 v -1 -3 r -2 -3 r -1 -1 r -3 h -3 1 r s n 1239 1491 m 3 h 2 -1 r 3 -2 r 1 -2 r -3 v -3 v -2 -2 r s 1230 1494 1227 1492 b 11 -19 1248 1504 i 11 -19 1249 1505 i 4 2 1245 1503 i 1262 1488 1256 1484 b n 1265 1506 m -2 -3 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 3 1 r 2 1 r 2 3 r 1 2 r -1 4 r -1 1 r -2 2 r -3 1 r -3 h c s n 1265 1506 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s n 1274 1494 m 1 2 r 1 3 r -1 3 r -1 2 r -2 2 r -3 1 r -2 h s n 1290 1516 m -1 v 1 -1 r 2 v 1 v -3 h -2 h -3 -1 r -2 -3 r -1 -2 r 1 -4 r 1 -1 r 2 -3 r 3 h 3 h 2 1 r 2 3 r 1 2 r s n 1283 1516 m -1 -2 r -1 -3 r 1 -3 r 1 -2 r 2 -2 r 3 -1 r 2 h s 11 -19 1293 1530 i 10 -19 1294 1530 i -4 -14 1306 1529 i 9 -4 1305 1521 i 9 -5 1304 1521 i 4 2 1290 1528 i 5 3 1304 1527 i 6 4 1301 1509 i 1316 1518 1311 1515 b n 1312 1540 m -1 -1 r 2 -1 r 2 v c s 7 -13 1315 1534 i 7 -13 1316 1535 i 3 3 1313 1532 i 1326 1524 1320 1520 b 7 -13 1325 1540 i 7 -12 1326 1540 i n 1327 1537 m 1 3 r 2 3 r 2 1 r 3 h 2 -1 r 6 -10 r s n 1332 1544 m 2 h 2 -2 r 6 -9 r s 4 2 1322 1538 i 7 3 1329 1526 i 1346 1535 1339 1531 b n 1347 1552 m -1 -2 r -1 v -2 v 1 -2 r 2 -1 r 1 -1 r 2 h 2 1 r 2 2 r 2 v 2 v -1 2 r -2 1 r -2 h -2 h c s n 1346 1550 m -2 v 2 -4 r 2 -1 r s n 1356 1546 m -1 3 r -2 3 r -2 1 r s n 1353 1553 m 1 v 2 2 r -1 v -2 -1 r s n 1349 1544 m -1 -2 r 1 -2 r -1 v 2 -1 r 3 h 5 3 r 3 1 r 1 -1 r s n 1349 1539 m 1 h 4 h 4 3 r 3 h 2 -1 r 1 -1 r -2 v -2 -2 r -6 -4 r -3 h -2 1 r 1 v 2 v 2 3 r s 9 4 1370 1559 i 45 31 1379 1563 i 45 26 1424 1594 i 45 23 1469 1620 i 6 2 1514 1643 i 7 -20 1532 1663 i 7 -21 1533 1664 i n 1536 1654 m 1 3 r 2 1 r 2 1 r 3 h 3 -1 r 2 -3 r 1 -1 r -4 v -2 -2 r -2 -2 r -2 -1 r -2 h -3 1 r s n 1541 1659 m 2 h 3 -1 r 2 -3 r 1 -2 r -3 v -1 -3 r -2 -1 r s 1533 1664 1529 1662 b 7 -20 1552 1671 i 7 -20 1553 1671 i 4 1 1549 1670 i 1563 1652 1557 1650 b n 1569 1670 m -3 -2 r -1 -3 r -3 v 1 -2 r 2 -3 r 3 -1 r 3 h 2 1 r 2 2 r 1 3 r 3 v 2 v -2 2 r -3 2 r -3 -1 r c s n 1569 1670 m -2 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 2 h s n 1576 1657 m 1 2 r 1 2 r 4 v 2 v -2 2 r -3 1 r -2 h s n 1595 1676 m -1 -1 r 1 -1 r 1 2 r 1 v -3 1 r -2 h -3 -1 r -3 -2 r -1 -3 r -3 v 1 -2 r 2 -2 r 3 -2 r 3 1 r 2 h 2 2 r 2 3 r s n 1588 1677 m -2 -2 r -1 -2 r -4 v 1 -1 r 2 -3 r 3 -1 r 2 h s 7 -20 1600 1689 i 7 -21 1601 1690 i -6 -14 1613 1687 i 9 -5 1610 1679 i 9 -6 1609 1679 i 4 2 1597 1688 i 6 3 1610 1685 i 6 3 1605 1668 i 1621 1674 1615 1672 b n 1620 1697 m -1 -1 r 2 -1 r 1 v c s 6 -13 1622 1690 i 6 -14 1623 1691 i 3 2 1620 1689 i 1631 1678 1625 1676 b 5 -13 1633 1694 i 5 -14 1634 1695 i n 1635 1692 m 1 2 r 3 2 r 2 1 r 3 h 2 -1 r 4 -11 r s n 1641 1697 m 2 h 2 -2 r 4 -10 r s 4 2 1630 1693 i 7 2 1635 1680 i 1653 1686 1646 1684 b n 1657 1703 m -2 -1 r -2 v -2 v -2 v 2 -1 r 1 -1 r 3 h 2 1 r 1 1 r 1 2 r 2 v -1