From owner-mpi-comm@CS.UTK.EDU Mon Nov 23 16:08:35 1992 Return-Path: Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA26326; Mon, 23 Nov 92 15:50:10 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26317; Mon, 23 Nov 92 15:50:05 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA06935; Mon, 23 Nov 92 15:50:04 -0500 Date: Mon, 23 Nov 92 15:50:04 -0500 Message-Id: <9211232050.AA06935@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: mpi committees and mailing list Status: R The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail will be collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi etc. Also try: send index from mpi I'm in the process of collecting information on how the HPF committee operates and will pass along the details in the next few days. I would like to suggest January 6 - 8 for the next meeting in Dallas at the airport hotel. We should plan on starting around 1:00 on January 6th and finishing around 3:00 on January 8th. Below is a list of the subcommittees as I have them from our meeting last week. Please let me know if you have any changes. Regards, Jack Introduction Subcommittee ---------------------- Jack Dongarra - Chair dongarra@cs.utk.edu Daniel Frye danielf@kgnvma.vnet.ibm.com Tony Hey ajgh@ecs.soton.ac.uk Rusty Lusk lusk@mcs.anl.gov Steve Zenith zenith@kai.com Point-To-Point Communication Subcommittee ----------------------------------------- Marc Snir - Chair snir@watson.ibm.com Scott Berryman berryman@cs.yale.edu Jack Dongarra dongarra@cs.utk.edu Al Geist geist@msr.epm.ornl.gov Adam Greenberg moose@think.com Bill Gropp gropp@mcs.anl.gov Leslie Hart hart@fsl.noaa.gov Rolf Hempel hempel@gmd.de Tom Henderson hender@fsl.noaa.gov Bob Knighten knighten@ssd.intel.com Oliver McBryan mcbryan@piper.cs.colorado.edu Peter Rigsbee par@cray.com David Walker walker@msr.epm.ornl.gov Collective Communication Subcommittee ------------------------------------- Al Geist - Chair geist@msr.epm.ornl.gov Leslie Hart hart@fsl.noaa.gov Jon Flowers jwf@parasoft.com Adam Greenberg moose@think.com Bob Knighten knighten@ssd.intel.com Steve Otto otto@cse.ogi.edu Peter Rigsbee par@cray.com Marc Snir snir@watson.ibm.com David Walker walker@msr.epm.ornl.gov Process Topology Subcommittee ----------------------------- Rolf Hempel - Chair hempel@gmd.de Jon Flowers jwf@parasoft.com Tom Henderson hender@fsl.noaa.gov Oliver McBryan mcbryan@piper.cs.colorado.edu Steve Otto otto@cse.ogi.edu Lew Tucker tucker@think.com Language Binding Subcommittee ----------------------------- Scott Berryman - Chair berryman@cs.yale.edu Bruce Leisure bleasure@kai.com Formal Language Description Subcommittee ---------------------------------------- Steve Zenith - Chair zenith@kai.com Tony Hey ajgh@cs.ston.ac.uk Rusty Lusk lusk@mcs.anl.gov Environment Inquiry Subcommittee --------------------------------- Bill Gropp - Chair gropp@mcs.anl.gov Daniel Frye danielf@kgnvma.vnet.ibm.com From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 15:37:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02830; Tue, 24 Nov 92 15:37:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA19627; Tue, 24 Nov 92 15:12:24 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19622; Tue, 24 Nov 92 15:12:21 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA25860; Tue, 24 Nov 92 14:12:19 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA11600; Tue, 24 Nov 92 14:12:17 CST Date: Tue, 24 Nov 92 14:12:17 CST Message-Id: <9211242012.AA11600@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Test Implementation of the Draft Standard We have completed a first and rough draft of a test implementation. We have included (and run!) three sample programs written in MPI on a sun network and on an Intel ipsc/i860. It is available by anonymous ftp from info.mcs.anl.gov in pub/mpi . The README file there gives more information, including how to get, install, and run the examples. This is a rough draft; the file mpi.man.ps.Z contains a description of the implementations architecture and man pages for all of the routines. We will be updating this implementation with additional examples and fewer restrictions soon. Enjoy! Bill Gropp and Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 17:07:49 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06915; Tue, 24 Nov 92 17:07:49 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA21596; Tue, 24 Nov 92 16:45:52 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21572; Tue, 24 Nov 92 16:45:38 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA21258; Tue, 24 Nov 92 16:45:35 -0500 Date: Tue, 24 Nov 92 16:45:35 -0500 Message-Id: <9211242145.AA21258@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: working procedures Chuck Koelbel has passed on some notes from the HPF experence. I think we will find them useful in carring out our meetings. Jack HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From chk@cs.rice.edu Mon Nov 23 15:45:48 1992 Return-Path: Received: from cs.rice.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26184; Mon, 23 Nov 92 15:45:39 -0500 Received: from DialupEudora (charon.rice.edu) by cs.rice.edu (AA22750); Mon, 23 Nov 92 14:44:21 CST Message-Id: <9211232044.AA22750@cs.rice.edu> Date: Mon, 23 Nov 1992 14:47:55 -0600 To: dongarra@cs.utk.edu From: chk@cs.rice.edu Subject: HPFF organization X-Attachments: :Macintosh HD:4227:HPFF rules: Status: RO Here it is. If anything is unclear, just ask me or (for logistical details) Theresa Chatman (tlc@cs.rice.edu). Chuck HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 18:09:18 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08323; Tue, 24 Nov 92 18:09:18 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22602; Tue, 24 Nov 92 17:43:42 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22596; Tue, 24 Nov 92 17:43:35 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA11950; Tue, 24 Nov 92 17:43:40 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 174208.17287; Tue, 24 Nov 1992 17:42:08 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA07130; Tue, 24 Nov 1992 16:14:37 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA06815; Tue, 24 Nov 92 16:14:36 -0600 Date: Tue, 24 Nov 92 16:14:36 -0600 Message-Id: <9211242214.AA06815@brisk.kai.com> To: dongarra@cs.utk.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Jack Dongarra's message of Tue, 24 Nov 92 15:04:05 -0500 <9211242004.AA20345@thud.cs.utk.edu> Subject: more on the intro From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Concerning the introduction: "The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds." I think this sentence is unnecessarily defensive. I'd delete the sentence and the following "thus". I earlier raised an objection to \item A simple way to create processes for the SPMD model there seems to be some redundancy here with \item Process groups The former is a subset of the latter. I'll repeat my noncirculated comments: I would rather not see us say too much about the process model at this stage; except to say that there is some process model defined by the language using the standard and to say something about side effects between processes that affect the message passing semantics (i.e., that there should be none). That does, of course, leave us with some interesting problems for identifying our communication; traditionally a difficult subject in message passing (hence my distribution of this message to the whole committee). The easy route would be to have a global name space of shared "channels" each with a communication characteristic [type] (e.g., "point-to-point" or "one-to-many"): a compiler can check correct usage if we identify the right rules (e.g. a synchronized unidirectional point-to-point communication is shared by only two processes). Such logical naming would map well across all the existing systems and avoid the problems of name servers, processor id, topologies and so forth since any topology can be described by the set of names and its users. Processor ids are, in effect, names in a global name space - and are often channels that have a "many-to-one" characteristic. Steven From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:38:03 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07921; Sat, 28 Nov 92 18:38:03 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22407; Sat, 28 Nov 92 18:35:38 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22403; Sat, 28 Nov 92 18:35:36 -0500 Message-Id: <9211282335.AA22403@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6064; Sat, 28 Nov 92 18:35:01 EST Date: Sat, 28 Nov 92 18:32:55 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: previously sent postcript file Reply-To: SNIR@watson.ibm.com Contains an outline I want to use for discussing issues in the poin-to-point subcommittee. I send it to the entire committee since it touches many issues related to collective communication and formal definitions. If somebody cannot print postscript and prefers latex, pls let me know. From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:40:52 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07938; Sat, 28 Nov 92 18:40:52 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22399; Sat, 28 Nov 92 18:33:55 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22394; Sat, 28 Nov 92 18:33:40 -0500 Message-Id: <9211282333.AA22394@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6053; Sat, 28 Nov 92 18:33:04 EST Date: Sat, 28 Nov 92 18:32:22 EST From: "Marc Snir" To: MPI-COMM@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: MPIOUTLN.DVI.* %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 16 df15 D E /Fb 51 125 df<127012F8B012701200A5 127012F8A31270051C779B18>33 D<137013F01201EA03C0EA0780EA0F00121E121C123C123812 781270A212F05AA87E1270A212781238123C121C121E7EEA0780EA03C0EA01F0120013700C2479 9F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370A813F013E0A212 0113C0120313801207EA0F00121E5A5A5A12600C247C9F18>I<123C127E127FA3123F120F120E 121E127C12F81270080C788518>44 DI48 DII<387FFFC0B512E0A3C8FCA4B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C01C0A2 387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA87E0070 13E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>IIII<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7FC5AA5 EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C7E9B18 >I<387F07F038FF8FF8387F07F0381C01C0A9EA1FFFA3EA1C01AA387F07F038FF8FF8387F07F0 151C7F9B18>II76 D<38FC01F8EAFE03A2383B06E0 A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03F8A3151C7F9B18>I<387E07F0 38FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A213311339A31319A2131D130DA3 EA7F07EAFF87EA7F03151C7F9B18>III82 D<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A2 90C7FC7E1278123FEA1FF0EA0FFEEA01FF38001F80EB03C0EB01E01300A2126012E0130100F013 C0EAFC07B512801400EAE7FC131C7E9B18>I<387FFFF8B5FCA238E07038A400001300B2EA07FF A3151C7F9B18>I<38FF83FEA3381C0070B2001E13F0000E13E0EA0F013807C7C03803FF806C13 00EA007C171C809B18>I<38FF07F8A3381C01C0A4380E0380A4EA0F0700071300A4EA038EA4EA 018C13DCA3EA00D813F8A21370151C7F9B18>I<38FE03F8A338700070A36C13E0A513F8A2EA39 DCA2001913C0A3138CEA1D8DA4000D13801305EA0F07A2EA0E03151C7F9B18>I97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3E0EB80F0EB0070 1478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E138038067E00151C809B18>IIIII<3803F1F03807FFF85A381E1F30 383C0F00EA3807A5EA3C0FEA1E1EEA1FFC485AEA3BF00038C7FC123CEA1FFF14C04813E0387801 F038F00078481338A36C1378007813F0EA7E03383FFFE0000F13803803FE00151F7F9318>I<12 7E12FE127E120EA5133FEBFF80000F13C0EBE1E013801300A2120EAA387FC3FC38FFE7FE387FC3 FC171C809B18>II<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C131EEA0E0FEB 0780130314C038FFC7F8A3151C7F9B18>107 DI<387DF1F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBF BF80397F1F1F001914819318>IIII<387F87E038FF9FF8EA7FBF3803 FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A15147F9318>114 DI<487E1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01 FF6C1380EB7E0013197F9818>I<387E07E0EAFE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13 FE3801FCFC1714809318>I<38FF8FF8A3383800E0A3381C01C0A2137113F9A213D9A2380DDD80 A3138DEA0F8FA23807070015147F9318>119 D<387F8FF000FF13F8007F13F0380E01C0EB0380 A21207EB0700A2EA03871386138EEA01CEA2EA00CCA213DC1378A31370A313F05B1279EA7BC0EA 7F806CC7FC121E151E7F9318>121 D<126012F0B3B012600424769F18>124 D E /Fc 12 119 df97 DI<137E48B4FC38038380EA0F07121E001C1300EA3C0248C7FCA35AA5EA70 021307EA381EEA1FF8EA07E011147C9315>I<1478EB03F814F0EB0070A314E0A4EB01C0A213F1 EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133CEA707C3878FCC0EA3F CF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB0300EA780EEA7FFC13E0 00F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I103 D<136013F0A213E01300A7120FEA1F8012 3113C0EA6380A212C3EA0700A3120EA3EA1C301360A2EA38C01218EA1F80EA0F000C1F7D9E0E> 105 D108 D<381E07C0383F1FE03833B870EA63E0EBC038138000C71370EA0700A3000E 13E0A3EB01C3001C13C6A2EB038C1301003813F8381800F018147D931A>110 D<137C48B4FC38038380380F01C0121E001C13E0123C1278A338F003C0A3EB07801400EA700F13 1EEA3838EA1FF0EA07C013147C9317>I116 D<380F01C0EA1F83003113E0 13C1EA6380A200C313C0EA0700A3380E0180A3EB0300A21306A2EA0F0CEA07F8EA01E013147D93 15>118 D E /Fd 28 121 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07E0D80F C01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D<13181330136013C01201EA0380120713005A121EA2123E123CA2127CA3127812F8AD1278127C A3123CA2123E121EA27E7E13801203EA01C012001360133013180D317BA416>40 D<12C012607E7E121C7E120F7E1380EA03C0A213E01201A213F0A3120013F8AD13F01201A313E0 A2120313C0A2EA078013005A120E5A12185A5A5A0D317DA416>I63 D67 D71 D77 D87 D97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014 F812FCA2B5FCA200FCC7FCA3127CA2127E003E13186C1330380FC0703803FFC0C6130015167E95 1A>II<3801FE1F0007B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03 380F87C0EBFF80D819FEC7FC0018C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F 140FA3007CEB1F00007E5B381F80FC6CB45A000113C019217F951C>II<120E121FEA3F80A3EA1F00 120EC7FCA7EAFF80A2121FB2EAFFF0A20C247FA30F>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA2018013 00AE3BFFF1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39 FFF1FFC0A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7 007C137CA26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF838 1FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8 EAFFF0A21A207E951F>I114 DI<13C0A41201A2120312 07120F121FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA238 1F807EAF14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE0 0E0007130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B 167F951E>I<39FFF07FC0A2390FC01C006C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D 7E497E80EB67F013E33801C1F8380381FC48C67E000E137E39FF81FFE0A21B167F951E>120 D E /Fe 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EAE738EA0700A30D0E 7E8E12>3 D E /Ff 33 118 df<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123C A25AA412F85AA97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA1 14>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA 01E0A213C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<14E0B0B712C0A3C700E0C7 FCB022237D9C29>43 D<1238127C12FE12FFA2127F123B1203A21206A2120E120C121812701220 08107C860F>I<137013F0120F12FF12F31203B3A4B51280A2111D7C9C1A>49 DIII<14E0A2497EA3497EA2497EA2 497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F48B57EA2EB800748486C7EA200 06801401000E803AFFE01FFFE0A2231F7E9E28>65 D69 DI73 D78 D82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC13F86C B4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813F038FE 01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140CA200E0 140E00C01406A400001400B10007B512C0A21F1E7E9D24>I87 D97 D100 DII<3801FC3C3807FFFE380F07DEEA1E03003E13E0A5001E13C0380F 0780EBFF00EA19FC0018C7FCA2121C381FFF8014F06C13F8003F13FC387C007C0070133E00F013 1EA30078133CA2383F01F8380FFFE000011300171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121F B0EAFFE0A20B217EA00E>I108 D<3AFE0FE03F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8F FEA227147D932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D> I<48B4FC000713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F0 3807FFC00001130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA614 3E147E147CEB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I E /Fg 77 124 df<90380FC3E090387FEFF09038E07C783801C0F8D8038013303907007000A7B6 1280A23907007000B0387FE3FFA21D20809F1B>11 DII<90380F80F890387FE7FE9038E06E063901C0 FC0F380380F8380700F00270C7FCA6B7FCA23907007007B03A7FE3FE3FF0A22420809F26>I<90 380FC0FFEB7FE79038E07E0F3801C0FC4848487E38070070A7B7FCA23907007007B03A7FE3FE3F F0A22420809F26>I34 D<127012F812FCA2127C120CA31218A21238123012601240060E7C9F0D >39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A212E0AC1260A212 70A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I<12C012607E7E12 1C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A213005A12065A12 1C12185A5A5A0B2E7DA112>I<1306AFB612F0A2D80006C7FCAF1C207D9A23>43 D<127012F812FCA2127C120CA31218A21238123012601240060E7C840D>II<127012F8A3127005057C840D>I<1303A213071306A2130E130CA2131C1318A213381330A2 13701360A213E013C0A212011380A312031300A25A1206A2120E120CA2121C1218A212381230A2 12701260A212E05AA2102D7DA117>IIIII<130EA2131E133EA2136E13EE13CEEA018E120313 0E1206120E120C121812381230126012E0B512F0A238000E00A7EBFFE0A2141E7F9D17>II<137CEA01FEEA07 83380E0380EA0C07121C3838030090C7FC12781270A2EAF3F8EAF7FEEAFC0E487EEB0380A200F0 13C0A51270A214801238EB0700121CEA0E1EEA07FCEA01F0121F7E9D17>I<1260387FFFC0A214 80EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207A76CC7FC121F7D9D17> II I<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A312701200AA127012F8 A312781218A41230A21260A21240051D7C930D>I<007FB512E0B612F0C9FCA8B612F06C14E01C 0C7D9023>61 D63 D65 DI<90381FC04090387F F0C03801F8393803C00D38078007380F0003121E003E1301123C127C1400127812F81500A80078 14C0127CA2123C003EEB0180121E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D 9F21>IIII<90380FC02090387FF8603901F81CE03803E0063807800338 0F0001121E14005A127C1560127812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E380780 03EA03E03901F80E6039007FFC2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90 B5FCA2EB800FAD39FFF8FFF8A21D1F7E9E22>II< EAFFFEA2EA0780B11406A4140EA2140C141C143C14FCB5FCA2171F7E9E1C>76 DI<39FF807FF813C0 0007EB07809038E00300A2EA06F0A21378133CA2131EA2130FA2EB078314C31303EB01E3A2EB00 F3A2147BA2143F80A280A2000F7FEAFFF0801D1F7E9E22>III82 D<3807E080EA0FF9EA1C1FEA300FEA7007EA600312E01301A36CC7FCA21278127FEA3FF0EA1FFC 6C7EEA03FF38001F801307EB03C0A2130112C0A400E01380EAF00338F80700EAFE0EEACFFCEA81 F812217D9F19>I<007FB512E0A238780F010070130000601460A200E0147000C01430A4000014 00B23807FFFEA21C1F7E9E21>I<39FFFC7FF8A23907800780EC0300B3A300031302EBC006A200 015B6C6C5AEB7830EB3FE0EB0FC01D207E9E22>I<39FFF007FEA2390F8001F090C712E06C6C13 C0A2EBC00100031480A2EBE00300011400A23800F006A3EB780CA36D5AA36D5AA2EB1F70EB0F60 A214E06D5AA26D5AA31F207F9E22>I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800F C006A3913819E00ED803C0140CA214393A01E030F018A33A00F0607830A3ECE07C903978C03C60 A390393D801EC0A390383F000F6D5CA3010E6DC7FCA32C207F9E2F>I<397FF83FF8A23907C00F 800003EB06003801E00EEBF00C00005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C1307808013 0DEB1DF0EB18F8EB3878EB307CEB603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FF E01FFEA21F1F7F9E22>I92 D97 D<120E12FEA2120EA9133FEB FF80380FC3C0EB00E0000E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E001520 7F9F19>IIII<133C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020 809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0 A2120EAD38FFE7FCA216207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFF C0A20A1F809E0C>I<13E0EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EA F1C0EA7F80EA3E000C28829E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B1370 13F0EA0FF81338EA0E1C131E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA212 0EB3A9EAFFE0A20B20809F0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E 13E0AD3AFFE7FE7FE0A223147F9326>IIII<3803E180EA0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA 1E1FEA0FF3EA03E3EA0003A7EB3FF8A2151D7E9318>III<1206A4120EA2121E123E EAFFF8A2EA0E00AA1318A5EA073013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC13 03A2EA070FEBFDFCEA01F116147F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3 EA0386A2138EEA01CCA213FC6C5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560EC C0E0D80E0F13C0130C14E00007EBE180EB186114713903987300EBB033A2143F3801F03EEBE01E A20000131CEBC00C1E147F9321>I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13 781338133C137C13EEEA01C7138738030380380701C0000F13E038FF87FEA21714809318>I<38 FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A313 60A35B12F0EAF18012F3007FC7FC123C151D7F9318>III E /Fh 29 122 df<48B4FC000F13E0383E03F8007813FCEA7E01 00FF13FEA5387E03FC1200EB07F0EB0FE01480EB1F00131E5BA25BA21370A690C7FCA61370EA01 FC487EA56C5AEA0070172A7CA920>63 D 70 D<91387FE003903903FFFC0F011FEBFF1F90397FF00FFF9038FF8001D803FEC7FC48488048 48804980485A003F815B007F81A3484891C7FCA90203B512F8A2EA7FC0DA00011300A2123F7F12 1F6C7E7F6C7E6C6C5B3800FF8090387FF00F011FB5123F0103EBFC0F9039007FE0032D297CA836 >I73 D77 D<90387F80603903FFF0E0000F13FF38 1F807F383F001F003E1307007E1303127C00FC1301A214007E7E6D130013F8EBFF806C13F814FE 6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F1407A200E013031401A37E15F06C1303 6C14E0B413079038E01FC090B5120000E05B38C01FF01D297CA826>83 D87 D<48B47E000F13F0381F 81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012 FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97 D99 DIII<9038FF81F00003EB E7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1 F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007 F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A 22>II<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3 120FB3A3EAFFFEA30F2B7DAA14>I107 DI<3BFFC07F800FF0 903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA3 01E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F0 9038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E039 0FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EF FF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F8 5BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F013 70A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7E A26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203A21207381F FFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B>I<39FFE03F F8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A26>I<3AFF FE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB 3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3BFFFE7FFC0FFEA33B0FE007 E000E03B07F003F001C0A29039F807F80300031680A23B01FC0EFC0700A2D9FE1E5B000090381C 7E0EA29039FF383F1E017F141C0278133C90393FF01FB8A216F86D486C5AA26D486C5AA36D486C 5AA22F1B7F9A32>I<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3F F86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01F FF80A3211B7F9A24>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C 485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807 007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>I E /Fi 18 119 df<127012F812FCA2127C120CA41218A21230A212601240060F7C840E>44 D49 DI56 DI77 D<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E13 1E7FA2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F 8000EAFFF0156020227EA125>I<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460 A36C1300A21278127FEA3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A4 6C136014E06C13C0EAF80138EF038038C7FF00EA81FC14247DA21B>83 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C 141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II101 D<121C121E123E121E121CC7FC A8120E12FEA2121E120EAFEAFFC0A20A227FA10E>105 D<390E1FC07F3AFE7FE1FF809039C0F3 03C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>109 D<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II114 D<38FFC3FEA2381E00F8000E1360A26C13C0A338038180 A213C300011300A2EA00E6A3137CA31338A217157F941A>118 D E /Fj 17 124 df73 D77 D79 DI97 D 99 D101 DI<15F090387F03F83901FFCF1C3803C1FC390780F818390F007800 48137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0D81C7FC7FC 0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB0070157848 1438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F7E9F21>I< 1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8A20D307EAF 12>105 D108 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E3AFF FC1FFF80A2211F7E9E25>110 D<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE038EBC000A35B B3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813705A1430A37E 6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2141C7EA27E14 186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A31203A21207120F 121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB19>II123 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 457 227 a Fj(Message)21 b(P)n(assing)g(In)n(terface)i({)f(Outline)869 354 y Fi(Marc)16 b(Snir)772 456 y(No)o(v)o(em)o(b)q(er)d(28,)k(1992)75 643 y Fh(In)n(tro)r(duction)75 745 y Fg(The)f(purp)q(ose)h(of)f(this)g(do)q (cumen)o(t)h(is)f(to)g(pro)o(vide)g(a)g(framew)o(ork)f(that)g(w)o(e)h(can)g (use)h(to)e(discuss)i(issues)75 801 y(in)i(the)e(de\014nition)j(of)d(MPI,)h (to)f(list)i(questions)f(w)o(e)f(need)i(to)e(answ)o(er,)h(and)g(prop)q(ose)f (some)h(answ)o(ers.)75 857 y(I)g(plan)g(to)e(use)i(this)g(do)q(cumen)o(t)f (as)g(a)g(framew)o(ork)f(for)h(the)g(discussions)i(in)f(the)f(p)q(oin)o (t-to-p)q(oin)o(t)h(sub-)75 914 y(committee;)d(I)g(broadcast)g(it)g(to)g(the) g(en)o(tire)g(committee,)g(b)q(ecause)h(man)o(y)f(issues)h(are)f(relev)m(an)o (t)g(to)g(the)75 970 y(other)g(sub)q(committees.)75 1113 y Fh(Goals)131 1214 y Fg(1.)22 b(Design)17 b(an)h(application)h(programming)d (in)o(terface)i(\(not)e(necessarily)j(a)e(target)f(for)h(compilers)189 1271 y(or)d(a)h(system)g(implemen)o(tation)h(library\))131 1363 y(2.)22 b(Allo)o(w)15 b(e\016cien)o(t)h(comm)o(unication:)21 b(Av)o(oid)16 b(memory)e(to)h(memory)g(cop)o(ying)g(and)h(allo)o(w)f(o)o(v)o (erlap)189 1420 y(of)e(computation)g(and)h(comm)o(unication)g(and)g(o\017oad) f(to)g(comm)o(unication)h(copro)q(cessor,)g(where)189 1476 y(a)o(v)m(ailable)131 1569 y(3.)22 b(Allo)o(w)15 b(\(but)g(no)h(mandate\))e (extensions)i(for)f(use)g(in)h(heterogeneous)f(en)o(vironmen)o(t.)131 1661 y(4.)22 b(Allo)o(w)15 b(con)o(v)o(enien)o(t)h(C,)f(C++,)g(F77)f(and)i (F90)e(bindings)j(for)e(in)o(terface.)131 1754 y(5.)22 b(Pro)o(vide)15 b(a)g(reliable)i(comm)o(unication)f(in)o(terface:)21 b(User)15 b(need)h(not)f(cop)q(e)h(with)f(comm)o(unication)189 1810 y(failures.)21 b(Suc)o(h)15 b(failures)i(are)d(dealt)i(b)o(y)f(the)h(underlying)h(comm)o (unication)f(subsystem.)131 1902 y(6.)22 b(F)l(o)q(cus)15 b(on)g(a)g(prop)q (osal)h(that)e(can)h(b)q(e)h(agreed)f(up)q(on)h(in)g(6)f(mon)o(ths.)131 1995 y(7.)22 b(De\014ne)10 b(an)g(in)o(terface)h(that)e(is)i(not)f(to)q(o)f (di\013eren)o(t)i(from)e(curren)o(t)h(practice)h(\(PVM/Express/P)o(armacs\)) 131 2087 y(8.)22 b(De\014ne)13 b(an)g(in)o(terface)f(that)g(can)h(b)q(e)h (quic)o(kly)g(implemen)o(ted)g(on)f(man)o(y)f(v)o(endors)g(platforms,)h(with) 189 2144 y(no)i(signi\014can)o(t)h(c)o(hanges)f(in)h(the)f(underlying)j(comm) o(unication)d(and)h(system)f(soft)o(w)o(are.)146 2245 y(Do)f(w)o(e)h(agree)g (on)g(this?)75 2388 y Fh(F)-6 b(ramew)n(ork)75 2489 y Fg(I)11 b(prop)q(ose)g(to)g(consider)g(comm)o(unication)h(op)q(erations)f(as)g (consisting)g(of)g(the)g(follo)o(wing)g(sub)q(op)q(erations:)75 2591 y Ff(INIT\(op)q(eration,)19 b(params,)d(handle\))24 b Fg(Pro)q(cess)15 b(pro)o(vides)f(all)i(relev)m(an)o(t)e(parameters)g(for)g (its)g(par-)189 2647 y(ticipation)19 b(in)f(the)g(comm)o(unication)g(op)q (eration)g(\(data)f(bu\013er,)h(t)o(yp)q(e,)g(participan)o(ts,)g(etc.\).)27 b(A)189 2704 y(handle)16 b(is)g(created)f(that)g(iden)o(ti\014es)h(the)g(op)q (eration.)964 2828 y(1)p eop %%Page: 2 2 bop 75 45 a Ff(ST)l(AR)l(T\(handle\))24 b Fg(The)16 b(comm)o(unication)g(op)q (eration)f(is)h(started)75 139 y Ff(A)-6 b(W)g(AIT\(handle\))24 b Fg(The)15 b(comm)o(unication)h(op)q(eration)g(is)g(completed.)75 232 y Ff(FREE\(handle\))25 b Fg(The)16 b(handle,)g(and)f(asso)q(ciated)g (resources)h(are)f(freed.)75 337 y(Correct)g(in)o(v)o(o)q(cation)h(of)g (these)g(sub)q(op)q(erations)g(is)h(a)e(sequence)i(of)e(the)h(form)f(1\(23\)) 1539 321 y Fe(\003)1558 337 y Fg(4.)21 b(I.e.,)15 b(a)h(handle)75 394 y(need)g(b)q(e)g(created)g(b)q(efore)g(comm)o(unication)g(o)q(ccurs;)g (it)f(can)h(b)q(e)g(reused)g(only)g(after)f(the)g(previous)i(use)75 450 y(has)12 b(completed;)h(and)f(it)g(need)h(to)e(b)q(e)h(freed)g(ev)o(en)o (tually)h(\(of)e(course,)h(one)g(can)g(assume)f(that)g(all)i(handles)75 507 y(are)i(freed)g(at)g(program)f(termination,)h(b)o(y)g(default\).)146 563 y(The)f(c)o(hoice)i(of)e(these)h(sub)q(op)q(erations)g(is)h(somewhat)d (arbitrary)l(.)20 b(The)15 b(justi\014cation)g(is)g(that)f(com-)75 619 y(m)o(unication)k(ma)o(y)f(b)q(e)h(a)g(length)o(y)g(pro)q(cess)f(one)h (desires)h(to)d(o)o(v)o(erlap)i(with)g(computation,)f(hence)i(the)75 676 y(separation)g(of)g(2)h(and)f(3;)i(also)e(comm)o(unication)i(setup)e(ma)o (y)g(in)o(v)o(olv)o(e)h(a)f(signi\014can)o(t)i(o)o(v)o(erhead)e(one)75 732 y(desires)i(to)f(amortize)h(o)o(v)o(er)e(man)o(y)h(successiv)o(e)i(comm)o (unications)f(with)g(the)g(same)f(c)o(haracteristics,)75 789 y(hence)c(the)g(separation)f(of)f(1)h(and)h(4)f(from)f(2)h(and)g(3.)75 932 y Fh(Issues)75 1035 y Fd(Whic)n(h)20 b(op)r(erations?)75 1121 y Fg(SEND)15 b(and)g(RECEIVE)g(for)g(p)q(oin)o(t)g(to)f(p)q(oin)o(t)i (comm)o(unication.)k(Longer)15 b(list)h(for)e(collectiv)o(e)j(comm)o(u-)75 1177 y(nication)g(\(note:)k(SEND)16 b(and)g(RECEIVE)g(are)g(particular)g (case)g(of)g(broadcast,)f(for)g(group)g(of)h(size)h(2;)75 1234 y(this)i(observ)m(ation)g(can)g(b)q(e)g(used)h(to)e(c)o(hec)o(k)h(if)g (de\014nition)i(of)d(collectiv)o(e)i(comm)o(unication)g(seman)o(tics)75 1290 y(are)15 b(consisten)o(t)g(with)h(de\014nition)h(of)e(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication\).)146 1347 y(One)g(can)f(argue)g(for)f(a)h(SEND) p 684 1347 14 2 v 17 w(RECV)g(\(or)f(EX)o(CHANGE\))h(op)q(eration.)146 1403 y(I)g(prop)q(ose)g(to)g(lea)o(v)o(e)g(out,)f(for)g(the)i(time)f(b)q (eing,)h(things)f(lik)o(e)i(remote)d(pro)q(cedure)i(calls,)g(monitors,)75 1460 y(atomic)h(op)q(erations,)i(semaphores,)e(activ)o(e)h(messages,)g (message)f(handlers,)i(etc.)27 b(This,)19 b(in)f(order)g(to)75 1516 y(fo)q(cus)f(initially)j(on)c(a)h(smaller)g(set)g(of)f(issues,)i(with)f (less)g(dep)q(endencie)q(s)i(on)e(the)g(underlying)i(pro)q(cess)75 1572 y(mo)q(del.)146 1629 y(Opinions?)75 1750 y Fd(What)g(calls)h(are)e(made) h(a)n(v)m(ailable)h(to)f(the)f(user?)75 1836 y Fg(Options:)75 1941 y Ff(\(1+2+3+4\))23 b Fg(blo)q(c)o(king)16 b(send/receiv)o(e)75 2035 y Ff(\(1+2\))i(\(3+4\))23 b Fg(non)o(blo)q(c)o(king)17 b(send/receiv)o(e,)f(follo)o(w)o(ed)f(b)o(y)h(w)o(ait)75 2128 y Ff(\(1\))i(\(2+3\))g(\(4\))24 b Fg(c)o(hannel)16 b(creation;)f(blo)q(c)o (king)i(c)o(hannel)f(send/receiv)o(e;)g(c)o(hannel)h(cancelation)75 2222 y Ff(\(1\))h(\(2\))g(\(3\))h(\(4\))k Fg(c)o(hannel)17 b(creation;)e(non)o(blo)q(c)o(king)h(send/receiv)o(e;)g(w)o(ait;)f(c)o (hannel)h(destruction)146 2327 y(An)o(y)f(more)g(needed?)22 b(An)o(y)15 b(less?)75 2448 y Fd(When)j(is)i(a)f(comm)n(unication)h(op)r (eration)e(completed?)75 2534 y Fg(The)12 b(main)f(distinction)j(is)e(b)q(et) o(w)o(een)f Fc(lo)n(c)n(al)g Fg(termination)g(and)h Fc(glob)n(al)f Fg(termination.)19 b(A)11 b(send)h(terminates)75 2591 y(lo)q(cally)23 b(when)f(data)f(has)h(b)q(een)h(copied)g(out)e(of)g(the)h(sender)g(bu\013er)g (\(formally)l(,)h(when)f(the)f(sender)75 2647 y(cannot)e(a\013ect)g(an)o (ymore)g(the)h(outcome)g(of)f(the)h(send)g(op)q(eration\).)33 b(It)20 b(terminates)g(globally)g(when)75 2704 y(data)d(has)h(b)q(een)h (receiv)o(ed)g(b)o(y)f(the)f(receiv)o(er)i(\(formally)l(,)f(when)h(receiv)o (er)f(has)g(terminated)g(executing)964 2828 y(2)p eop %%Page: 3 3 bop 75 45 a Fg(an)o(y)15 b(op)q(eration)h(that)f(logically)j(precedes)f(the)f (execution)g(of)g(the)g(receiv)o(e\).)22 b(The)16 b(distinction)h(has)f(no)75 102 y(meaning)g(for)e(a)h(receiv)o(e)h({)f(since)i(lo)q(cal)f(termination)g (of)e(a)h(receiv)o(e)h(implies)i(global)d(termination.)146 158 y(Choices:)131 252 y(1.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g(lo)q (cal.)131 346 y(2.)22 b(T)l(ermination)e(is)g(either)g(lo)q(cal)h(or)e (global,)i(at)e(user)g(c)o(hoice)i(\(e.g.,)e(user)g(can)h(c)o(ho)q(ose)g(b)q (et)o(w)o(een)189 402 y(sync)o(hronous)15 b(and)g(async)o(hronous)g(comm)o (unication\))131 496 y(3.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g (global.)146 590 y(The)21 b(curren)o(t)f(MPI)h(prop)q(osal)g(c)o(ho)q(oses)g (2.)37 b(If)21 b(w)o(e)g(stic)o(k)g(to)f(this)h(c)o(hoice)h(then)f(I)g (suggest)g(that)75 646 y(termination)h(mo)q(de)g(\(global)g(or)f(lo)q(cal\))h (is)g(part)f(of)g(the)h(op)q(eration)f(parameters)g(that)g(are)g(set)g(up)75 703 y(in)g(sub)q(op)q(eration)g(1.)34 b(F)l(urthermore,)21 b(I)f(w)o(ould)h(suggest)e(to)h(o\013er)f(the)h(same)g(option)h(for)e (collectiv)o(e)75 759 y(comm)o(unications.)146 816 y(Another)13 b(issue)h(is)g(whether)g(termination)g(mo)q(de)f(need)h(b)q(e)h(consisten)o (t)e(for)g(all)h(participan)o(ts.)20 b(Cur-)75 872 y(ren)o(t)c(MPI)g(prop)q (osal)g(requires)h(that)e(a)g(sync)o(hronous)h(send)h(b)q(e)g(matc)o(hed)e(b) o(y)h(a)g(sync)o(hronous)g(receiv)o(e)75 928 y(\(either)i(all)h(participan)o (ts)f(use)g(lo)q(cal)h(completion)g(or)e(all)i(use)f(global)g(completion\).) 29 b(There)18 b(seem)g(to)75 985 y(b)q(e)d(no)g(comp)q(elling)i(logical)f (reason)f(for)f(this)h(restriction,)g(and)g(I)g(am)g(not)f(sure)h(it)g(has)g (signi\014can)o(t)g(im-)75 1041 y(plemen)o(tation)i(adv)m(an)o(tages.)k(The) 16 b(alternativ)o(e)g(is)g(to)f(allo)o(w)h(eac)o(h)g(participan)o(t)g(in)h(a) e(comm)o(unication)75 1098 y(op)q(eration)h(to)g(c)o(ho)q(ose)g(either)h(lo)q (cal)h(or)e(global)h(termination)f(\(this)h(b)q(ecomes)g(more)e(signi\014can) o(t)j(for)d(a)75 1154 y(collectiv)o(e)i(op)q(eration,)e(where)g(the)h(c)o (hoice)g(is)g(meaningful)g(at)f(more)f(than)i(one)f(pro)q(cess\).)146 1211 y(I)i(assume)h(lo)q(cal)g(termination)g(for)f(sub)q(op)q(eration)h(1)f ({)g(no)h(sync)o(hronization)g(with)g(another)f(pro-)75 1267 y(cessor)e(is)h(required)g(for)f(this)g(phase.)146 1324 y(Opinions?)146 1380 y(Should)h(w)o(e)f(c)o(ho)q(ose)g(1)g(or)g(3,)f(rather)h(than)g(2?)146 1437 y(Should)g(w)o(e)f(use)g(another)g(mec)o(hanism)g(to)g(c)o(ho)q(ose)g (lo)q(cal)h(and)f(global)h(termination)f(\(e.g.,)f(a)g(global)75 1493 y(\015ag\)?)146 1549 y(Should)g(w)o(e)e(require)h(the)g(c)o(hoice)h(of)e (\\sync)o(hronous")g(to)g(b)q(e)h(done)g(consisten)o(tly)h(b)q(e)f(all)h (participan)o(ts)75 1606 y(in)j(a)f(comm)o(unication?)75 1728 y Fd(When)j(do)r(es)g(a)h(call)h(return?)75 1813 y Fg(The)14 b(ob)o(vious)f(c)o(hoice)i(is:)k(\\A)13 b(call)i(return)e(when)h(the)f (corresp)q(onding)i(sub)q(op)q(eration\(s\))e(terminated".)75 1870 y(An)21 b(alternativ)o(e)f(is)h(to)f(allo)o(w)h(for)f(unsuccessful)i (return)f(as)f(w)o(ell.)37 b(E.g.,)20 b(send)h(can)g(return)f(unsuc-)75 1926 y(cessfully)l(,)d(if)f(the)f(system)g(cannot)g(accept)g(the)h(message,)e (for)h(whatev)o(er)g(reasons;)f(or)h(receiv)o(e)h(returns)75 1983 y(unsuccessfully)f(if)e(the)g(receiv)o(e)h(cannot)e(execute)i(within)g (a)e(set)h(time;)g(and)g(so)g(on.)19 b(This)13 b(is)g(the)g(curren)o(t)75 2039 y(MPI)i(de\014nition.)146 2096 y(My)j(p)q(ersonal)i(c)o(hoice)g(is)g (against)e(unsuccessful)j(return.)32 b(This)20 b(is)f(more)g(adequate)g(for)f (system)75 2152 y(programming,)f(rather)h(than)f(application)j(programming;)e (it)g(in)o(tro)q(duces)h(seman)o(tic)f(dep)q(endencies)75 2209 y(on)h(implemen)o(tation;)i(it)e(forces)f(the)h(user)f(to)g(c)o(hec)o(k)h (for)f(success)i(at)e(eac)o(h)g(op)q(eration,)i(rather)e(then)75 2265 y(deal)j(with)g(failure)g(of)f(comm)o(unication)h(using)g(an)f (exception)i(handling)g(mec)o(hanism.)36 b(Of)20 b(course,)75 2322 y(implemen)o(ters)f(are)f(w)o(elcome)g(to)f(pro)o(vide)i(a)e(reasonable) i(exception)f(handling)i(mec)o(hanism)f(for)e(the)75 2378 y(MPI)e(library)h ({)f(I)h(suggest)e(to)h(lea)o(v)o(e)g(op)q(en)h(the)f(de\014nition)i(of)e (suc)o(h)h(mec)o(hanism.)146 2434 y(Opinions?)75 2556 y Fd(Correctness)i (criteria)75 2642 y Fg(I)e(use)f(the)g(follo)o(wing)h(terminology:)964 2828 y(3)p eop %%Page: 4 4 bop 75 45 a Ff(safe)17 b(program)22 b Fg(A)c(program)f(that)h(should)h (execute)f(successfully)i(in)f(an)o(y)f(implemen)o(tation,)i(inde-)189 102 y(p)q(enden)o(tly)c(of)f(the)h(amoun)o(t)e(of)h(a)o(v)m(ailable)i(system) d(resources.)75 195 y Ff(unsafe)j(program)23 b Fg(A)d(program)f(that)g(will)j (succeed)f(or)f(fail,)i(dep)q(ending)g(on)e(implemen)o(tation)h(or)189 252 y(a)o(v)m(ailable)c(system)d(resources.)75 346 y Ff(erroneous)i(program) 22 b Fg(A)16 b(program)e(that)g(will)j(alw)o(a)o(ys)e(fail.)146 452 y(Examples)g(\(in)o(v)o(olving)h(t)o(w)o(o)e(pro)q(cessors)h(with)h(ids)g (1)f(and)g(2,)f(and)i(using)g(MPI)f(lik)o(e)h(syn)o(tax\))146 508 y(The)f(follo)o(wing)h(program)e(is)i(safe,)e(and)i(should)g(alw)o(a)o (ys)e(succeed.)75 659 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 715 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000\))147 772 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000\))75 884 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 941 y(CALL)g(CRECV\(mode=blocking,)e(buf=recbuf,)h(source=1,)h(type=0,)g (len=1000\))147 997 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000\))146 1148 y Fg(The)15 b(follo)o(wing)h(program)e(is)i (erroneous,)e(and)i(should)g(alw)o(a)o(ys)e(fail.)75 1310 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 1367 y(CALL)f (CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g(len=1000\))147 1423 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recdbuf,)h(source=2,)g(type=0,)h (len=1000\))75 1536 y(ELSE)47 b(!)24 b(\(GETID\(\))f(==)g(2\))147 1593 y(CALL)g(CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))147 1649 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recbuf,)h (source=1,)h(type=0,)g(len=1000\))146 1755 y Fg(The)12 b(follo)o(wing)h (program)d(is)j(unsafe,)f(and)h(ma)o(y)e(succeed)i(or)f(fail,)h(dep)q(ending) h(on)e(implemen)o(tation.)75 1918 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN) 147 1975 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000000\))147 2031 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000000\))75 2144 y(ELSE)g(!)h(\(GETID\(\))f(==)g (2\))147 2200 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000000\))147 2257 y(CALL)g(CRECV\(mode=blocking,)e (buf=recbuf,)h(source=1,)h(type=0,)g(len=1000000\))146 2363 y Fg(This)14 b(program)f(can)h(succeed)h(only)g(if)f(the)g(system)g(has)g (su\016cien)o(t)g(bu\013er)g(space)g(to)g(bu\013er)g(1)f(MgB)75 2420 y(of)i(data.)146 2476 y(The)j(strict)g(de\014nition)i(of)e(\\safe")f (needs)i(to)f(b)q(e)g(somewhat)g(temp)q(ered)h(in)g(practice.)29 b(A)18 b(correct)75 2532 y(sequen)o(tial)13 b(program)d(ma)o(y)h(fail)i(b)q (ecause)f(of)g(time)g(limits)h(or)e(other)g(resource)h(restrictions.)19 b(Lik)o(ewise,)13 b(a)75 2589 y(safe)h(parallel)i(program)d(ma)o(y)l(,)g (exceptionally)l(,)k(exceed)e(system)f(resources.)19 b(Still,)d(common)e (sense)h(can)75 2645 y(di\013eren)o(tiate)i(b)q(et)o(w)o(een)g(resources)f (that)g(are)h(\\practically)g(un)o(b)q(ounded",)h(i.e.)25 b(with)17 b(limits)g(that)f(are)75 2702 y(unlik)o(ely)k(to)d(b)q(e)h(normally)g(encoun) o(tered)g(b)o(y)g(correct)f(programs,)g(and)g(resources)h(that)f (\\practically)964 2828 y(4)p eop %%Page: 5 5 bop 75 45 a Fg(b)q(ounded",)23 b(i.e.)37 b(with)21 b(limits)h(one)f(encoun)o (ters)g(frequen)o(tly)g(with)g(correct)g(programs,)f(and)h(whic)o(h)75 102 y(often)13 b(impinges)i(on)f(program)e(p)q(ortabilit)o(y)l(.)20 b(A)14 b(formal)f(de\014nition)i(will)h(assume)d(that)g(some)g(resources)75 158 y(are)19 b(unlimited,)k(and)c(implemen)o(tations)i(will)g(striv)o(e)f(to) f(pro)o(vide)h(a)f(reasonable)h(appro)o(ximation)f(of)75 214 y(this)f(assumption.)26 b(F)l(urthermore,)17 b(the)g(user)h(should)g(b)q(e)g (able)g(to)f(query)g(and)g(p)q(ossibly)i(con)o(trol)e(an)o(y)75 271 y(implemen)o(tation)f(sp)q(eci\014c)i(resource)d(limitation)h(that)f (a\013ects)f(program)g(b)q(eha)o(vior.)146 327 y(One)k(need)g(to)f(allo)o(w)h (\(indeed,)h(one)f(cannot)f(disallo)o(w\))h(v)o(endors)g(to)e(supp)q(ort)i (some)f(unsafe)h(pro-)75 384 y(grams)c(as)h(w)o(ell.)21 b(The)15 b(requiremen)o(ts)h(there)f(should)h(b)q(e)g(that)143 478 y Fa(\017)23 b Fg(The)15 b(seman)o(tics)g(of)g(successful)i(program)d (execution)i(b)q(e)g(w)o(ell-de\014ned)143 571 y Fa(\017)23 b Fg(The)16 b(system)f(pro)o(vide)i(clear)f(information)h(and)f(p)q(ossibly)h (con)o(trol)f(on)g(those)g(parameters)f(that)189 628 y(determine)h(the)f(b)q (eha)o(vior)h(of)f(unsafe)g(programs)f(\(e.g.,)g(bu\013er)h(sizes\).)146 722 y(A)g(handle)j(is)e Fc(active)g Fg(\(at)f(a)g(pro)q(cess\))h(from)f(the)h (start)f(of)g(sub)q(op)q(eration)i(1)e(to)g(the)h(completion)h(of)75 778 y(sub)q(op)q(eration)f(4)g(for)f(this)h(handle.)22 b(A)15 b(comm)o(unication)i(is)f Fc(active)f Fg(\(at)g(a)g(pro)q(cess\))h(from)e (the)i(start)e(of)75 835 y(sub)q(op)q(eration)i(2)f(to)g(the)g(completion)h (of)f(sub)q(op)q(eration)h(3.)k(I)15 b(suggest)g(the)g(follo)o(wing)h(design) g(p)q(oin)o(t:)131 941 y(1.)22 b(The)13 b(n)o(um)o(b)q(er)g(of)g(activ)o(e)g (comm)o(unication)g(handles)i(p)q(er)e(pro)q(cess)g(is)h(\\practically)g(un)o (b)q(ounded".)131 1035 y(2.)22 b(The)d(n)o(um)o(b)q(er)g(of)f(initiated)j (comm)o(unications)e(p)q(er)h(pro)q(cessor)f(is)g(\\practically)h(un)o(b)q (ounded".)189 1091 y(\(Note)12 b(that)h(there)g(ma)o(y)g(b)q(e)g(only)h(one)g (activ)o(e)f(comm)o(unication)h(op)q(eration)f(p)q(er)h(activ)o(e)f (handle\).)131 1185 y(3.)22 b(The)16 b(amoun)o(t)g(of)g(a)o(v)m(ailable)i (system)d(bu\013er)i(space)f(is)h(limited)h({)e(a)g(safe)g(program)f(cannot)h (rely)189 1241 y(on)f(it.)146 1348 y(This)g(design)h(p)q(oin)o(t)g(has)f(the) h(follo)o(wing)f(implications:)131 1454 y(1.)22 b(A)11 b(t)o(yp)q(e)g(1)f(or) h(t)o(yp)q(e)g(2)g(sub)q(op)q(eration)g(will)i(alw)o(a)o(ys)d(complete,)i (indep)q(enden)o(tl)q(y)h(of)e(other)g(pro)q(cesses)189 1510 y(activities.)21 b(Of)15 b(course,)g(the)g(same)g(is)h(true)f(for)g(a)f(t)o (yp)q(e)i(4)f(sub)q(op)q(eration.)131 1604 y(2.)22 b(The)15 b(successful)h(completion)h(of)d(a)h(t)o(yp)q(e)g(3)g(sub)q(op)q(eration)h (ma)o(y)e(dep)q(end)j(on)e(the)g(participation)189 1661 y(of)h(other)g(pro)q (cesses.)24 b(A)17 b(send)g(ma)o(y)f(b)q(e)h(blo)q(c)o(k)o(ed)g(un)o(til)h(a) e(matc)o(hing)h(receiv)o(e)g(o)q(ccurs.)24 b(And,)17 b(of)189 1717 y(course,)h(a)g(receiv)o(e)h(ma)o(y)e(not)h(complete)g(un)o(til)i(a)d (matc)o(hing)h(send)h(o)q(ccurs.)29 b(\(This)18 b(extends)h(to)189 1774 y(collectiv)o(e)e(comm)o(unication.\))146 1880 y(T)l(o)e(this,)h(w)o(e)f (need)i(add)f(p)q(ositiv)o(e)g(requiremen)o(ts)g(for)f(progress)g(and/or)h (fairness.)21 b(Let's)16 b(sa)o(y)f(that)75 1936 y(a)k(comm)o(unication)h(is) f Fc(enable)n(d)f Fg(\(at)h(a)f(pro)q(cess\))h(if)h(a)f(matc)o(hing)g(comm)o (unication)h(is)f(activ)o(e)h(\(i.e.)31 b(a)75 1993 y(matc)o(hing)16 b(receiv)o(e)h(for)e(a)h(send)g(or)f(a)h(matc)o(hing)g(send)g(for)g(a)f (receiv)o(e\).)23 b(An)16 b(enabled)h(comm)o(unication)75 2049 y(ma)o(y)g(b)q(ecome)i(disabled)h(either)e(b)q(ecause)h(it)g(completes)f (successfully)i(\(the)e(comm)o(unication)h(o)q(ccurs)75 2106 y(and)d(sub)q(op)q(eration)h(3)e(terminates\))g(or)h(b)q(ecause)g(the)g(matc) o(hing)g(comm)o(unication)h(b)q(ecomes)f(inactiv)o(e)75 2162 y(\(e.g.,)e(a)h(receiv)o(e)h(that)e(matc)o(hes)h(one)h(send)g(ma)o(y)e(b)q(e) i(satis\014ed)g(b)o(y)f(another)g(send,)h(th)o(us)f(disabling)i(the)75 2219 y(\014rst)e(send\).)75 2325 y Ff(progress)21 b Fg(If)14 b(some)f(comm)o(unication)h(in)g(the)f(system)g(is)h(enabled)h(then)e(some)g (comm)o(unication)h(ev)o(en-)189 2381 y(tually)i(o)q(ccurs)f(in)h(the)g (system.)75 2475 y Ff(fairness)22 b Fg(An)c(activ)o(e)f(comm)o(unication)h (either)g(completes)g(successfully)h(or)d(b)q(ecomes)i(p)q(ermanen)o(tly)189 2532 y(disabled.)964 2828 y(5)p eop %%Page: 6 6 bop 146 45 a Fg(W)l(e)19 b(assume)g(in)h(these)f(de\014nitions)i(that)d(eac)o (h)h(pro)q(cess)g(in)o(v)o(ok)o(e)g(sub)q(op)q(erations)h(in)g(the)f(correct) 75 102 y(order,)c(i.e.)23 b(1\(23\))393 85 y Fe(\003)411 102 y Fg(4.)f(Of)16 b(course,)f(a)h(program)f(where)h(an)g(activ)o(e)g(comm)o (unication)g(b)q(ecomes)h(p)q(erma-)75 158 y(nen)o(tly)f(disabled)h(\(e.g.,)c (a)i(send)h(is)f(left)h(forev)o(er)e(with)i(no)f(matc)o(hing)g(receiv)o(e\))h (is)f(erroneous,)g(and)g(will)75 214 y(not)g(terminate)g(successfully)l(.)146 271 y(The)20 b(enforcemen)o(t)g(of)g(these)g(t)o(w)o(o)f(conditions)i (requires)g(a)f(\015o)o(w)g(con)o(trol)g(proto)q(col:)29 b(A)21 b(pro)q(cess)75 327 y(should)c(not)f(b)q(e)h(prev)o(en)o(ted)g(from)e (accepting)j(one)e(message)g(that)g(w)o(as)f(sen)o(t)h(b)q(ecause)i(its)e (bu\013ers)g(are)75 384 y(full)21 b(with)f(other)g(messages,)g(p)q(ossibly)h (sen)o(t)f(earlier.)34 b(When)21 b(a)e(receiv)o(e)i(matc)o(hes)e(sev)o(eral)h (p)q(ossible)75 440 y(messages)d(\(e.g.,)f(a)h(receiv)o(e)h(with)g(a)f (\\don't)f(care")h(sender)h(\014eld\))g(then)g(alternativ)o(e)g(options)f (should)75 497 y(b)q(e)f(considered)g(fairly)g(\(e.g.,)e(b)o(y)h(using)h(an)f (LR)o(U)h(p)q(olicy)h(when)e(considering)i(matc)o(hing)e(senders\).)146 553 y(A\014cionados)d(of)f(formalism)h(will)i(note)d(that)g(I)h(used)g(the)g (w)o(eak)o(est)f(de\014nition)j(of)d(fairness)h(\(ev)o(en)o(tual)75 610 y(fairness\),)j(rather)f(than)h(b)q(ounded)i(fairness)f(or)e(other)h (alternativ)o(es.)146 666 y(Examples:)146 723 y(The)g(follo)o(wing)h (programs)e(are)h(safe,)f(and)i(should)g(succeed:)75 823 y Fb(!)24 b(assume)f(only)g(two)h(processes,)e(numbered)h(1)h(and)f(2)75 936 y(IF)h(\(GETID\(\))e(==)i(1\))g(THEN)147 992 y(DO)f(I=0,)g(N)218 1048 y(CALL)g(CSEND\(mode=nonblocking,)e(buf=buf\(I\),)i(dest=2,)f(type=I,)h (len=1000\))147 1105 y(END)g(DO)75 1218 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 1274 y(DO)g(I=0,)g(N)218 1331 y(CALL)g(CRECV\(mode=nonblocking,)e (buf=buf\(I\),)i(dest=2,)f(type=N-I,)h(len=1000\))147 1387 y(END)g(DO)146 1487 y Fg(Pro)q(cess)16 b(1)g(uses)h(non)o(blo)q(c)o(king)h (sends)f(to)f(send)h(a)f(sequence)i(of)e(messages)g(to)f(pro)q(cess)i(2;)g (pro)q(cess)75 1544 y(2)d(receiv)o(es)h(these)f(messages)f(in)i(the)f(rev)o (erse)g(order,)g(using)h(non)o(blo)q(c)o(king)g(receiv)o(es.)20 b(In)15 b(practice,)g(suc)o(h)75 1600 y(program)e(ma)o(y)g(fail)h(if)g(the)g (n)o(um)o(b)q(er)g(of)g(p)q(ending)h(messages)f(implied)i(b)o(y)d(this)i (program)d(is)i(larger)g(than)75 1657 y(the)20 b(system)g(limit.)36 b(But)21 b(the)f(system)g(limit)h(should)h(b)q(e)e(large)h(enough)f(as)g(to)g (mak)o(e)f(suc)o(h)i(failure)75 1713 y(extremely)16 b(unlik)o(ely)l(.)75 1813 y Fb(!)24 b(Assume)f(a)g(large)h(number)f(of)g(processes)75 1926 y(!)h(Consumer)e(code)75 1983 y(IF)i(\(GETID\(\))e(==)i(1\))g(THEN)123 2039 y(DO)f(I=)h(1,)f(INFOG\(\))194 2095 y(CALL)h(CRECV\(mode=blocking,)d (buf=buf\(I\),)h(source=I,)h(type=0,)g(len=1000\))123 2152 y(END)g(DO)75 2265 y(!)h(Producers)75 2321 y(ELSE)123 2378 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))146 2478 y Fg(All)17 b(pro)q(cesses)f(send)g(a)f(message)h(to)f (pro)q(cess)h(1.)21 b(Pro)q(cess)15 b(1)h(receiv)o(es)g(these)g(messages)f (in)i(a)e(\014xed)75 2534 y(order.)31 b(A)20 b(\015o)o(w)e(con)o(trol)h (proto)q(col)g(is)h(needed)g(to)f(mak)o(e)f(sure)h(that)g(bu\013ers)g(at)f (pro)q(cess)i(1)f(will)h(not)75 2591 y(o)o(v)o(er\015o)o(w.)e(The)c(sending)h (pro)q(cesses)f(ma)o(y)g(b)q(e)g(dela)o(y)o(ed)h(un)o(til)g(the)f(receiv)o(e) g(pro)q(cess)g(is)h(ready)f(to)f(accept)75 2647 y(their)j(message.)146 2704 y(Consider)f(the)h(follo)o(wing)g(implemen)o(tation)g(alternativ)o(es)f (for)g(this)h(co)q(de:)964 2828 y(6)p eop %%Page: 7 7 bop 143 45 a Fa(\017)23 b Fg(Messages)18 b(are)h(sen)o(t)g(immediately)i(and) e(stored)g(in)h(the)f(ph)o(ysical)i(memory)d(of)h(the)g(receiving)189 102 y(no)q(de)g({)g(no)g(pro)o(vision)g(for)g(retry)l(.)31 b(This)19 b(is)h(a)e(bad)i(implemen)o(tation)g(since)g(ph)o(ysical)g(storage) 189 158 y(limits)c(are)f(lik)o(ely)i(to)e(b)q(e)g(encoun)o(tered.)143 252 y Fa(\017)23 b Fg(Messages)c(are)g(sen)o(t)h(immediately)h(and)f(stored)f (in)i(the)f(receiv)o(er)g(ph)o(ysical)i(memory)l(.)33 b(If)20 b(the)189 308 y(receiv)o(er)e(is)g(unable)h(to)f(receiv)o(e)g(the)g(message,) g(the)f(send)i(is)f(retried.)28 b(This)19 b(is)f(a)f(correct,)h(but)189 365 y(p)q(ossibly)e(ine\016cien)o(t)h(implemen)o(tation.)143 459 y Fa(\017)23 b Fg(A)16 b(\\request-to-send")h(proto)q(col)g(is)g(used;)h (the)f(receiv)o(er)g(allo)o(ws)g(the)g(send)g(to)g(pro)q(ceed)g(only)g(if)189 515 y(it)g(has)g(storage)f(for)h(the)g(message.)26 b(The)18 b(receiv)o(ers)g(need)g(to)e(store)h(information)g(on)h(p)q(ending,)189 571 y(unsatis\014ed)13 b(requests)g(to)f(send.)19 b(The)13 b(maxim)o(um)f(n)o(um)o(b)q(er)h(of)f(p)q(ending)j(requests)d(at)g(a)g(no)q (de)h(is)g(a)189 628 y(system)f(limit)i(that)e(should)i(b)q(e)f(set)f(large)h (enough)g(as)f(to)g(b)q(e)i(considered)g(\\practically)g(in\014nite".)146 722 y(Opinions?)146 778 y(Should)k(w)o(e)e(b)q(e)h(ev)o(en)h(more)e (restrictiv)o(e)h(and)g(require)g(that)g(safe)f(programs)f(run)i(correctly)l (,)h(ev)o(en)75 835 y(when)c(exceeding)h(\\v)o(ery)d(large")h(resource)h (limits?)21 b(\(Or)13 b(should)i(w)o(e)e(a)o(v)o(oid)g(to)f(burden)j(the)e (implemen-)75 891 y(tation)i(with)g(requiremen)o(ts)h(to)f(handle)h(what)f (most)f(lik)o(ely)j(are)e(pathologic)h(programs?\))146 948 y(Should)h(w)o(e)f(try)g(to)g(b)q(e)h(more)f(formal)g(in)h(de\014ning)h(when) f(some)f(of)g(the)h(\\unsafe")f(programs)f(will)75 1004 y(run)h(to)g (completions?)24 b(\(Can)16 b(w)o(e)g(do)g(it)g(without)g(b)q(eing)i(more)d (sp)q(eci\014c)j(ab)q(out)e(the)h(implemen)o(tation)75 1060 y(mo)q(del?\))146 1117 y(Should)f(w)o(e)f(require)h(fairness?)21 b(Should)c(w)o(e)d(ha)o(v)o(e)h(a)g(stronger)f(requiremen)o(t)i(of)f (fairness?)75 1260 y Fh(What)23 b(are)g(the)g(comm)n(unication)d(parameters?) 75 1363 y Fd(Message)e(data)i(\(bu\013er\))131 1449 y Fg(1.)i(scalar)15 b(\(b)o(yte/halfw)o(ord/w)o(ord/doublew)o(ord\),)e(presumably)j(coming)f (from)g(a)g(register)131 1543 y(2.)22 b(Con)o(tiguous)15 b(space)g(in)h (memory)f(\(initial)i(address/length\))131 1637 y(3.)22 b(Bu\013er)15 b(with)g(constan)o(t)g(stride)g(\(initial)i(address/size)f(of)f(eac)o(h)g (blo)q(c)o(k/stride/n)o(um)o(b)q(er)i(blo)q(c)o(ks\))131 1731 y(4.)22 b(A)15 b(t)o(yp)q(ed)g(message)g(\(of)g(a)f(t)o(yp)q(e)i(supp)q (orted)g(in)g(the)f(domain)g(language\).)131 1824 y(5.)22 b(A)d(union)i(of)d (sev)o(eral)i(messages)f(\(p)q(ossibly)i(sp)q(eci\014ed)g(b)o(y)e(an)h(arra)o (y)e(of)h(p)q(oin)o(ters)h(to)e(message)189 1881 y(descriptors\).)g(The)13 b(seman)o(tics)f(of)f(a)h(union)h(is)f(probably)h(conjunctiv)o(e)f(at)g(the)g (sender)g(end)h(\(send)189 1937 y(all\);)g(at)f(the)h(receiv)o(e)g(end,)h(it) f(ma)o(y)e(b)q(e)j(conjunctiv)o(e)f(\(receiv)o(e)g(all\))h(or)e(disjunctiv)o (e)i(\(receiv)o(e)f(an)o(y\).)146 2044 y(The)j(curren)o(t)g(MPI)f(prop)q (osal)i(has)e(2)h(and)g(3)g(\(2)f(is)h(really)h(a)f(particular)g(case)g(of)g (3\).)21 b(Do)15 b(w)o(e)h(w)o(an)o(t)75 2100 y(more?)k(In)c(particular,)g (do)f(w)o(e)g(w)o(an)o(t)f(t)o(yp)q(ed)h(messages?)20 b(Ho)o(w)15 b(general?)75 2222 y Fd(Groups)k(and)g(con)n(text)75 2307 y Fg(Messages)c(are)g(tagged)g(b)o(y)g(pro)q(cess)h(names)f(\(sender)h(and)f (receiv)o(er\))h(and)g(t)o(yp)q(e)f(names.)21 b(Both)15 b(names)75 2364 y(spaces)g(can)f(b)q(e)i(\015at)e(or)g(structured.)20 b(If)14 b(a)h(\015at)f(pro)q(cess)h(name)f(space)h(is)g(used,)g(then)g(a)f (pro)q(cess)h(names)75 2420 y(is)e(an)f(in)o(teger)g(\(from)f(0)h(to)f(n)o (um)o(b)q(er)p 687 2420 14 2 v 17 w(of)p 741 2420 V 16 w(pro)q(cesses-1,)i (for)e(C)h(think)o(ers\).)19 b(If)13 b(a)e(structured)i(pro)q(cess)f(name)75 2477 y(space)18 b(is)g(used)f(then)h(a)f(pro)q(cess)h(name)f(is)h(a)f(pair)h (of)f(the)g(form)g(\(group,)g(rank\).)26 b(Similarly)19 b(in)f(a)f(\015at)75 2533 y(t)o(yp)q(e)g(name)h(space,)g(a)f(t)o(yp)q(e)g(is)h(a)f(n)o(um)o(b)q (er)h(in)g(a)f(range;)h(in)g(a)f(structured)g(t)o(yp)q(e)h(name)f(space,)h (it)f(is)h(a)75 2590 y(pair)e(of)e(the)i(form)e(\(group,)g(lo)q(cal)p 659 2590 V 18 w(t)o(yp)q(e\).)146 2646 y(The)21 b(usefulness)h(of)f(groups)f (seem)h(clear)h(in)f(the)g(con)o(text)g(of)f(collectiv)o(e)j(op)q(erations,)f (where)f(a)75 2703 y(groupid)c(is)g(a)f(handle)h(to)f(sp)q(ecify)h(the)g(set) f(of)f(participan)o(ts)i(in)g(a)f(collectiv)o(e)i(comm)o(unication)f(\(Note:) 964 2828 y(7)p eop %%Page: 8 8 bop 75 45 a Fg(some)15 b(of)h(the)f(discussions)j(on)d(groups)g(seem)h(to)f (assume)h(that)f(a)g(group)g(is)i(alw)o(a)o(ys)e(represen)o(ted)h(b)o(y)f(a) 75 102 y(mem)o(b)q(ership)f(list)g(that)f(is)h(presen)o(t)f(at)f(eac)o(h)i (group)f(mem)o(b)q(er.)19 b(While)14 b(is)g(the)f(represen)o(tation)h(lik)o (ely)h(to)75 158 y(b)q(e)g(used)h(on)e(small)i(systems,)e(it)h(is)g(b)o(y)g (no)g(means)f(the)h(unique)h(p)q(ossible)h(one.)j(An)o(y)15 b(connected)g(graph)75 214 y(structure)h(can)h(b)q(e)g(used)g(to)f(k)o(eep)h (trac)o(k)f(of)g(group)g(mem)o(b)q(ership,)i(with)f(ob)o(vious)f(tradeo\013s) g(b)q(et)o(w)o(een)75 271 y(storage)h(and)h(p)q(erformance.)28 b(A)18 b(complete)g(list)h(at)e(eac)o(h)h(no)q(de)h(is)f(one)g(extreme)g(of)g (this,)g(namely)l(,)h(a)75 327 y(complete)g(graph.)28 b(Therefore,)19 b(a)e(group)h(handle)i(is)e(a)g(more)g(abstract)f(and)h(more)g(general)g (concept)75 384 y(than)d(a)g(list)h(of)f(group)g(mem)o(b)q(ers.\))146 440 y(The)h(collectiv)o(e)j(comm)o(unication)e(sub)q(committee)g(has)g(to)f (deal)h(with)g(mec)o(hanisms)g(for)f(creating)75 497 y(and)j(destro)o(ying)f (groups)g(\(for)g(the)g(record,)h(I)g(fa)o(v)o(or)e(dynamic)i(group)g (creation)f(b)q(oth)h(b)o(y)f(explicitly)75 553 y(listing)j(group)f(mem)o(b)q (ers)f(and)h(b)o(y)g(partitioning)h(an)e(existing)i(group.)33 b(I)20 b(do)g(not)f(fa)o(v)o(or)g(a)g(complex)75 610 y(stac)o(k)e(or)h(tree)g (mec)o(hanism)h(to)e(manage)h(groups)f(and)i(na)o(vigate)e(from)h(one)g (group)g(to)f(another)h({)g(at)75 666 y(least)d(not)g(in)h(the)f(basic)h(MPI) f(prop)q(osal\).)146 723 y(In)i(the)f(con)o(text)g(of)g(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication,)g(the)f(issue)h(is)g(whether)g(w)o(e)f(w)o(an)o (t)f(to)h(use)g(a)75 779 y(structured)f(name)g(space)h(for)e(t)o(yp)q(e)i (and)f(pro)q(cess)h(argumen)o(ts)e(and,)h(if)h(so,)e(ho)o(w.)146 835 y(The)21 b(adv)m(an)o(tage)h(of)f(ha)o(ving)h(a)f(structured)h(name)g (space)f(for)h(pro)q(cesses)g(is)g(that)f(it)h(simpli\014es)75 892 y(com)o(bining)14 b(of)e(distinct)i(parallel)g(co)q(des)f(in)h(a)e(lo)q (osely)i(coupled)g(application:)20 b(Eac)o(h)13 b(functional)h(subset)75 948 y(can)j(use)h(message)f(passing)h(within)g(its)g(con)o(text,)f(with)g(no) h(c)o(hanges)f(in)h(the)f(lo)q(cal)i(co)q(de.)27 b(The)17 b(same)75 1005 y(concern)k(for)f(mo)q(dularit)o(y)h(militates)h(for)e(a)g(structured)g (t)o(yp)q(e)h(space.)36 b(This)21 b(do)q(es)g(not)g(necessarily)75 1061 y(implies)j(or)d(requires)h(a)f(stac)o(k)g(mec)o(hanism)h(for)f (managing)g(groups,)i(or)e(a)g(tree)g(structure)h(among)75 1118 y(groups.)146 1174 y(First,)14 b(t)o(w)o(o)g(remarks:)143 1268 y Fa(\017)23 b Fg(The)c(preexisting)h(group)f Fb(ALL)f Fg(that)g(includes)j(all)f(pro)q(cesses)g(pro)o(vides)f(a)g(global)g(con)o (text.)31 b(If)189 1324 y(all)17 b(comm)o(unications)f(use)h(this)f(con)o (text,)f(then)i(one)f(defaults)g(to)g(\015at)f(name)h(space.)23 b(With)16 b(the)189 1381 y(righ)o(t)g(syn)o(tax,)h(the)g(existence)h(of)f(a)g (structured)g(name)g(space)g(can)g(b)q(e)h(ignored)g(b)o(y)f(users)g(that)189 1437 y(prefer)e(a)g(\015at,)f(global)i(name)f(space.)143 1531 y Fa(\017)23 b Fg(A)c(structured)f(name)h(space)g(can)g(alw)o(a)o(ys)g(fak)o (ed,)g(with)g(no)g(c)o(hanges)g(in)g(the)g(comm)o(unication)189 1588 y(op)q(erations:)g(Replace)d(eac)o(h)e Fb(process)p 868 1588 15 2 v 16 w(id)g Fg(actual)h(parameter)e(b)o(y)h(a)g(a)g(reference)h(to) e(a)h(function)189 1644 y Fb(process\(rank,)22 b(group)p 646 1644 V 16 w(id\))p Fg(;)c(replace)h(eac)o(h)e Fb(type)h Fg(actual)f (parameter)g(b)o(y)h(a)f(reference)h(to)f(a)189 1701 y(function)f Fb(type\(local)p 610 1701 V 15 w(type,)24 b(group)p 889 1701 V 16 w(id\))p Fg(.)146 1794 y(F)l(or)16 b(simplicit)o(y)l(,)j(I)e(shall)h (assume)f(the)f(same)h(group)f(con)o(text)h(is)g(used)g(b)q(oth)g(for)f (relativ)o(e)i(pro)q(cess)75 1851 y(n)o(um)o(b)q(ering)h(and)g(for)f(relativ) o(e)h(t)o(yp)q(e)g(naming.)30 b(I)19 b(b)q(eliev)o(e)h(this)f(simpli\014es)i (the)d(discussion,)j(without)75 1907 y(in)o(tro)q(ducing)13 b(undue)g(restrictions.)20 b(A)12 b(con)o(text)f(for)g(a)h(comm)o(unication)g (op)q(eration)h(can)f(b)q(e)g(established)75 1964 y(in)k(t)o(w)o(o)e(w)o(a)o (ys:)131 2070 y(1.)22 b(The)f(curren)o(t)h(scop)q(e)g(\(curren)o(t)f(group\)) g(of)g(a)g(comm)o(unication)h(op)q(eration)g(is)g(implicit;)27 b(it)21 b(is)189 2126 y(established)16 b(b)o(y)g(the)f(last)g(executed)h(con) o(text)f(con)o(trolling)h(call)g(\()p Fb(MPI)p 1401 2126 V 17 w(PUSHC)e Fg(and)i Fb(MPI)p 1713 2126 V 16 w(POPC)f Fg(in)189 2183 y(the)e(curren)o(t)h(draft\).)k(Th)o(us,)c(an)o(y)f(pro)q(cess)h(n)o(um) o(b)q(er)h(is)f(understo)q(o)q(d)g(to)f(b)q(e)i(relativ)o(e)f(rank)f(in)i (the)189 2239 y(curren)o(t)g(group,)f(and)h(an)o(y)g(t)o(yp)q(e)g(name)g(is)h (understo)q(o)q(d)f(to)g(b)q(e)g(relativ)o(e)h(to)e(curren)o(t)h(group,)g (and)189 2296 y(do)q(es)j(not)f(con\015icts)i(with)f(t)o(yp)q(e)h(names)e (used)i(within)g(other)f(groups.)27 b(Of)18 b(course,)h(the)f(initial)189 2352 y(scop)q(e)d(is)g Fb(ALL)p Fg(,)e(and)i(the)g(system)f(defaults)h(to)f (a)g(\015at)g(name)h(space)g(if)g(scop)q(e)g(setting)f(commands)189 2409 y(are)h(nev)o(er)g(used.)131 2503 y(2.)22 b(The)g(con)o(text)g(of)f(a)h (comm)o(unication)h(op)q(erations)f(need)h(to)f(b)q(e)h(explicitly)i(sp)q (eci\014ed)f(in)f(the)189 2559 y(comm)o(unication)15 b(op)q(eration)f (itself.)21 b(A)14 b(pro)q(cess)h(ma)o(y)e(b)q(elong)j(to)d(sev)o(eral)i (groups)f(at)f(an)o(y)h(p)q(oin)o(t)189 2615 y(in)19 b(time,)g(and)g(ma)o(y)f (use)h(an)o(y)f(of)g(these)h(groups)f(as)g(scop)q(e)h(for)f(a)g(comm)o (unication)h(op)q(eration.)189 2672 y(Again,)14 b(the)f(initial)j(scop)q(e)e Fb(ALL)f Fg(is)h(prede\014ned,)h(and)f(can)g(b)q(e)g(used)g(as)g(a)f(\015at)g (name)h(space.)19 b(\(In)14 b(a)964 2828 y(8)p eop %%Page: 9 9 bop 189 45 a Fg(language)13 b(where)g(argumen)o(ts)f(are)h(optional,)h(con)o (text)e(w)o(ould)i(b)q(e)g(an)f(optional)g(parameter)g(with)189 102 y(default)i(v)m(alue)i Fb(ALL)p Fg(.\))146 208 y(The)h(adv)m(an)o(tage)f (of)g(an)h(explicit)i(con)o(text)d(parameter)g(is)h(that)g(it)g(prev)o(en)o (ts)f(the)h(confusion)h(that)75 264 y(o)q(ccurs)13 b(when)g(an)g(implicit)i (con)o(text)d(is)h(c)o(hanged)g(mid)g(comm)o(unication.)20 b(More)12 b(generally)l(,)i(it)f(prev)o(en)o(ts)75 321 y(the)e(confusion)h (arising)g(when)g(the)f(seman)o(tics)h(of)e(an)i(op)q(eration)f(is)h (quali\014ed)h(b)o(y)e(a)g(scop)q(e)h(that)e(dep)q(ends)75 377 y(on)18 b(the)g(con)o(trol)g(\015o)o(w)g(of)g(the)g(program,)g(rather)f (than)h(its)h(static)f(structure.)29 b(The)18 b(disadv)m(an)o(tage)g(is)75 434 y(the)e(need)g(for)f(an)g(additional)i(parameter,)d(in)j(languages)e (that)g(don't)g(ha)o(v)o(e)g(optional)h(parameters)e(or)75 490 y(o)o(v)o(erloading)h(of)g(pro)q(cedures.)146 547 y(Opinions?)75 690 y Fh(Matc)n(hing)23 b(of)g(send)g(and)h(receiv)n(e)75 793 y Fd(receiv)n(e)18 b(criteria)75 879 y Fg(Messages)c(are)h(matc)o(hed)g(b)o (y)h(sender)f(and)h(t)o(yp)q(e,)f(where)g(eac)o(h)g(can)h(b)q(e)g(a)e (\\don't)h(care".)146 935 y(Our)f(exp)q(erience)j(is)e(that)f(matc)o(hing)g (b)o(y)g(t)o(yp)q(e)h(only)l(,)g(or)f(disallo)o(wing)i(\\don't)e(care")g(in)h (the)f(sender)75 992 y(\014eld)20 b(signi\014can)o(tly)h(simpli\014es)g(the)e (implemen)o(tation)i({)d(but)h(some)g(users)g(complained)i(ab)q(out)e(suc)o (h)75 1048 y(restriction.)146 1105 y(Also,)c(do)g(w)o(e)g(w)o(an)o(t)f(to)g (b)q(e)i(explicit)i(ab)q(out)d(the)g(don't)g(care)g(v)m(alue,)h(or)e(use)i (named)f(constan)o(ts?)146 1161 y(Opinions?)75 1283 y Fd(Matc)n(hing)20 b(of)f(data)75 1369 y Fg(F)l(or)c(un)o(t)o(yp)q(ed)g(messages)g(the)g (options)h(are)131 1462 y(1.)22 b(Send)16 b(and)f(receiv)o(e)h(bu\013er)f(ha) o(v)o(e)g(same)g(size)131 1556 y(2.)22 b(Receiv)o(e)16 b(bu\013er)g(is)f(no)g (shorter)g(than)g(send)h(bu\013er)131 1650 y(3.)22 b(No)15 b(constrain)o(ts)f(\(messages)h(ma)o(y)f(b)q(e)i(truncated\).)131 1744 y(4.)22 b(User)15 b(c)o(ho)q(ose)g(one)g(of)g(the)g(ab)q(o)o(v)o(e)g (options)h(\(one)f(more)f(call)j(parameter,)d(or)g(a)h(global)h(\015ag\).)146 1838 y(Choices)f(should)g(b)q(e)g(the)f(same)g(for)f(all)j(t)o(yp)q(es)e(of)g (send)h(op)q(eration)f(\(the)g(curren)o(t)g(MPI)g(draft)g(seem)75 1894 y(to)h(allo)o(w)g(truncation)g(for)g(con)o(tiguous)g(data,)f(but)i (disallo)o(w)g(it)f(for)g(receiv)o(e)h(with)g(stride\))146 1951 y(The)d(EUI)h(prop)q(osal)f(uses)g(option)h(4.)19 b(I)13 b(w)o(ould)h(b)q(e)g(happier)g(with)f(2,)g(but)h(am)e(uncomfortable)i(with)75 2007 y(3)i(\(truncation)h(seems)g(to)f(fall)i(in)f(the)g(category)f(of)g (comm)o(unication)i(errors,)e(for)g(whic)o(h)h(I)h(assume)e(a)75 2063 y(global)g(exception)g(handling)h(mec)o(hanism\).)146 2120 y(Opinions?)146 2176 y(If)12 b(t)o(yp)q(ed)g(messages)g(are)g(supp)q (orted)g(then)h(one)f(should)h(sp)q(ell)h(out)e(requiremen)o(ts)g(on)g(t)o (yp)q(e)g(compat-)75 2233 y(ibilit)o(y)18 b(\(e.g.,)d(can)h(t)o(w)o(o)e(real) i(n)o(um)o(b)q(ers)g(b)q(e)h(receiv)o(ed)g(in)o(to)f(a)g(complex)g(n)o(um)o (b)q(er?\))23 b({)16 b(this)g(is)g(probably)75 2289 y(sp)q(eci\014c)h(to)e (eac)o(h)g(language)g(binding.)146 2346 y(Opinions?)75 2489 y Fh(Syn)n(tax)75 2590 y Fg(This)h(ma)o(y)e(b)q(e,)i(of)f(course,)f(a)h(sub)s (ject)g(of)g(endless)i(dispute.)k(A)15 b(few)g(ob)o(vious)g(guidelines:)143 2684 y Fa(\017)23 b Fg(Short,)14 b(mnemonic)i(names)964 2828 y(9)p eop %%Page: 10 10 bop 143 45 a Fa(\017)23 b Fg(Systematic)15 b(naming)h(con)o(v)o(en)o(tion)143 137 y Fa(\017)23 b Fg(Systematic)15 b(ordering)g(of)g(parameters)143 230 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)g(where)g(this)f(is)h(p)q (ossible,)h(of)e(optional)h(parameters)f(and)h(k)o(eyw)o(ord)e(param-)189 286 y(eters.)143 379 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)f(where)g (this)h(p)q(ossible,)h(of)d(o)o(v)o(erloading.)146 469 y(I)d(heard)g(in)h (our)f(last)g(meeting)g(a)g(preference)h(for)f(m)o(ultiple)h(function)g (names,)g(rather)e(than)h(m)o(ultiple)75 525 y(mo)q(des.)146 582 y(Th)o(us,)f(in)h(C++)g(or)f(F90,)f(one)i(could)g(ha)o(v)o(e)f(the)g(c)o (hoice)h(of)f(the)g(t)o(yp)q(e)g(of)g(datum)g(sen)o(t)g(\(scalar/con)o (tiguous)75 638 y(v)o(ector/noncon)o(tiguous\))15 b(b)q(e)i(dictated)g(b)o(y) f(the)h(t)o(yp)q(e)f(and)h(n)o(um)o(b)q(er)f(of)g(parameters,)f(rather)h (than)g(b)q(e)75 695 y(enco)q(ded)f(in)f(the)g(function)g(name.)19 b(Then)14 b(a)f(p)q(ossible)i(naming)f(c)o(hoice)h(is)f Fb(mode)23 b(|)h(operation)p Fg(,)12 b(where)75 751 y Fb(mode)18 b Fg(=)g Fb(init)g Fg(\(1\),)f Fb(nonblocking)g Fg(\(1+2\),)h Fb(blocking)f Fg(\(1+2+3+4,)h(lo)q(cal)h(termination\),)g(or)f Fb(sync)75 808 y Fg(\(1+2+3+4)d(global)h(termination\),)f(and)g Fb(operation)f Fg(=)h(send,)h(or)e(recv.)146 864 y(In)h(addition,)h(one)f(has)g Fb(DO\(handle\))p Fg(,)f Fb(START\(handle\))p Fg(,)e Fb(AWAIT\(handle\))i Fg(and)h Fb(FREE\(handle\))p Fg(.)146 921 y(Another)21 b(issue)i(is)f (function)g(vs)f(pro)q(cedure)i(call,)h(in)e(F)l(ortran.)38 b(Pro)q(cedure)22 b(calls)h(seem)e(more)75 977 y(natural)15 b(in)h(F)l(ortran,)e(where)h(function)h(in)o(v)o(o)q(cations)g(cannot)f(b)q (e)h(used)g(as)e(statemen)o(ts.)146 1033 y(Opinions?)22 b(Preferences)16 b(for)f(mnemonics?)21 b(Suggestions)16 b(for)e(F77)h(and)g(C?)75 1176 y Fh(Miscelania)75 1277 y Fg(Do)h(w)o(e)g(w)o(an)o(t)g(a)g Fb(TEST)p Fg(,)f(a)i(non)o(blo)q(c)o(king)h Fb(AWAIT)d Fg(op)q(eration?)25 b(\(Alw)o(a)o(ys)16 b(returns;)h(a)f(successful)i(return)75 1334 y(is)e(equiv)m(alen)o(t)h(to)d(a)h(successful)i(return)e(of)g(a)f(blo)q (c)o(king)j Fb(AWAIT)p Fg(\).)146 1390 y(Do)d(w)o(e)h(w)o(an)o(t)f(to)h Fb(AWAIT)f Fg(sev)o(eral)i(messages?)146 1447 y(Do)h(w)o(e)h(w)o(an)o(t)f(a)g Fb(PROBE)h Fg(function)h(\(tells)f(me)g(whic)o(h)h(messages)f(are)g(w)o (aiting)g(to)f(b)q(e)i(receiv)o(ed)g(b)o(y)75 1503 y(me\)?)146 1560 y(Where)12 b(do)h(w)o(e)f(return)h(information)g(on)g(parameters)f(of)g (receiv)o(ed)i(messages)e(suc)o(h)h(as)f(length,)i(and)75 1616 y(t)o(yp)q(e?)24 b(The)16 b(curren)o(t)g(MPI)h(prop)q(osal)f(has)g(length)h (b)q(e)g(the)f(v)m(alue)i(returned)e(b)o(y)h(the)f(receiv)o(e)h(function)75 1673 y(and)d(t)o(yp)q(e)g(returned)h(b)o(y)f(a)f(subsequen)o(t)i(call)g(to)e Fb(MP)p 964 1673 15 2 v 17 w(INFOT)p Fg(.)g(Another)h(natural)g(c)o(hoice)h (w)o(ould)f(seem)h(to)75 1729 y(b)q(e)f(that)f(that)f(length)i(and)g(t)o(yp)q (e)f(are)g(returned)h(b)o(y)f(the)h(len)g(and)g(t)o(yp)q(e)f(parameters)f(of) h(the)h(initial)h(call,)75 1786 y(but)g(this)h(is)g(error)e(prone:)143 1876 y Fa(\017)23 b Fg(The)d(v)m(alues)h(are)e(returned)i(async)o(hronously)f (for)f(non)o(blo)q(c)o(king)j(receiv)o(es.)35 b(The)20 b(user)g(has)g(to)189 1932 y(remem)o(b)q(er)11 b(not)h(to)f(touc)o(h)g(the)h(actual)g(len)g(and)g (t)o(yp)q(e)g(parameters,)f(in)h(addition)h(of)e(not)h(touc)o(hing)189 1989 y(the)j(receiv)o(e)h(bu\013er,)f(un)o(til)h(the)f(op)q(eration)h (completes.)143 2081 y Fa(\017)23 b Fg(A)f(natural)h(implemen)o(tation)h(w)o (ould)f(b)q(e)g(to)f(alw)o(a)o(ys)g(return)g(the)h(length)g(and)g(t)o(yp)q(e) g(of)f(the)189 2138 y(receiv)o(ed)f(messages.)36 b(Ho)o(w)o(ev)o(er,)20 b(most)g(users)g(will)i(assume)f(that)e(the)i(actual)g(len)g(and)g(t)o(yp)q (e)189 2194 y(parameters)14 b(are)h(not)g(up)q(dated)h(when)f(their)h(v)m (alue)h(is)e(not)g(\\don't)f(care".)143 2286 y Fa(\017)23 b Fg(The)13 b(user)h(will)h(not)e(b)q(e)h(able)g(to)f(use)h(constan)o(ts)e(as)h (actual)h(\\don't)f(care")g(parameter)f(v)m(alues)j(\(or)189 2343 y(an)o(ytime,)g(if)g(the)g(\\natural")g(implemen)o(tation)i(is)e(allo)o (w)o(ed\))143 2435 y Fa(\017)23 b Fg(This)17 b(c)o(hoice)h(b)q(ecomes)g(ev)o (en)f(less)h(app)q(etizing)h(when)e(a)g(handle)h(is)g(reused;)g(then)g(the)f (len)h(and)189 2492 y(t)o(yp)q(e)d(v)m(ariables)h(w)o(ould)g(b)q(e)g(reused,) f(to)q(o.)75 2582 y(It)g(seems)g(safer)f(to)g(return)h(these)g(v)m(alues)h (with)f(the)g(call)g(that)g(concludes)h(the)f(execution)h(of)e(sub)q(op)q (er-)75 2638 y(ation)h(3)g({)g(but)g(w)o(e)g(need)h(to)f(think)h(syn)o(tax.) 146 2695 y(Opinions?)952 2828 y(10)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Mon Nov 30 18:08:53 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22099; Mon, 30 Nov 92 18:08:53 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA03436; Mon, 30 Nov 92 17:54:53 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03434; Mon, 30 Nov 92 17:54:51 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA26470; Mon, 30 Nov 92 17:54:50 -0500 Date: Mon, 30 Nov 92 17:54:50 -0500 Message-Id: <9211302254.AA26470@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: draft of introduction Enclosed is a draft the introduction subcommittee has put together. We would like to get your comments and input. We feel that it is in a partial state and will be expanded. Hopefully this draft will help keep the discussion of goals, target, audience, etc. from becoming fragmented in each subcommittee. Jack \section{Introduction} \label{sec:intro} \subsection{Overview and Goals} Message passing is a paradigm used widely on certain classes of parallel machines; especially those with distributed memory. Although there are many variations, the basic concept of processes communicating through messages is well understood. Over the last ten years, substantial progress has been made in casting significant applications in this paradigm. Each vendor has implemented their own variant. More recently, several systems \cite{need references} have demonstrated that a message passing system can be efficiently and be portably implemented. It is thus an appropriate time to try to define both the syntax and semantics of a core of library routines that will be useful to a wide range of users and efficiently implementable on a wide range of computers. In designing MPI we have sought to make use of the most attractive features of a number of existing message passing systems, rather than selecting one of them and adopting it as the standard. Thus, MPI has been strongly influenced by work at the IBM T. J. Watson Research Center by Bala, Kipnis, Snir and colleagues \cite{}, Intel's NX/2 \cite{}, Express \cite{}, nCUBE's Vertex \cite{}, and PARMACS \cite{}. Other important contributions have come from Zipcode \cite{}, Chimp \cite{}, PVM \cite{}, and PICL \cite{}. One of the objectives of this paper is to promote a discussion within the concurrent computing research community of the issues that must be addressed in establishing a practical, portable, and flexible standard for message passing. This cooperative process began with a workshop on standards for message passing held in April 1992 \cite{}. The main advantages of establishing a message passing standard are portability and ease-of-use. In a distributed memory communication environment in which the higher level routines and/or abstractions are build upon lower level message passing routines the benefits of standardization are particularly apparent. Furthermore, the definition of a message passing standard, such as that proposed here, provides vendors with a clearly defined base set of routines that they can implement efficiently, or in some cases provide hardware support for, thereby enhancing scalability. The goal of the Message Passing Interface simply stated is to develop a \it{de facto} standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. \subsection{Who Should Use This Standard?} This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. \subsection{What Platforms Are Targets For Implementation?} The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run efficiently on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those "machines" consisting of collections of other machines, parallel or not, connected by a communication network. \subsection{What Is Included In The Standard?} The standard includes (this is temporarily as inclusive as possible): \begin{itemize} \item Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication \item Collective operations \item Process groups \item Communication contexts \item A simple way to create processes for the SPMD model \item Bindings for both Fortran and C \item A model implementation \item A formal specification. \end{itemize} \subsection{What Is Not Included In The Standard?} The standard does not specify: \begin{itemize} \item Explicit shared-memory operations \item Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages \item Program construction tools \item Debugging facilities \item Tracing facilities \end{itemize} Features that are not included can always be offered as extensions by specific implementations. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 16:39:05 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17944; Tue, 1 Dec 92 16:39:05 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27367; Tue, 1 Dec 92 16:14:27 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27363; Tue, 1 Dec 92 16:14:24 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA00550; Tue, 1 Dec 92 16:14:28 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 161350.23711; Tue, 1 Dec 1992 16:13:50 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301 for ) id AA22501; Tue, 1 Dec 1992 15:07:53 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA16316; Tue, 1 Dec 92 15:07:51 -0600 Date: Tue, 1 Dec 92 15:07:51 -0600 Message-Id: <9212012107.AA16316@brisk.kai.com> To: mpi-comm@cs.utk.edu Subject: A proposal. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 I've seen several proposals and made a few suggestions myself. I have now had a chance to look at Gropp and Lusk's Manual for the test implementation and I like it as a starting point. I propose to all subcommittees that we use this document as the focus for discussion along with the introduction Jack sent around yesterday. This shouldn't be taken as an endorsement by me of the Gropp and Lusk manual - I have several remarks I would like to make at a later stage. However, it is clear that we need a well considered and simple starting point and I see that in this document plus the introduction. Do I have any seconders? In the HPFF committe structure description we received a few days ago it was noted that much success came from someone taking charge of the central document. I implore someone to do this (Jack?). Steven P.S. I am not ignoring the other comments made in the committees - I think they too can - after debate - be folded into the suggested starting point. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 19:08:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21233; Tue, 1 Dec 92 19:08:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA01862; Tue, 1 Dec 92 18:43:11 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01858; Tue, 1 Dec 92 18:43:08 -0500 Message-Id: <9212012343.AA01858@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 7311; Tue, 01 Dec 92 18:42:33 EST Date: Tue, 1 Dec 92 18:43:02 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: HPF process Reply-To: SNIR@watson.ibm.com The HPF process did not start with a central document. Various peoples wrote proposals for various parts of the language design and, after several iterations, when the number of proposals starting being reasonable and these proposals started being coherent, they were collated into one document that evolved (mostly by successive pruning) into a draft. I suppose we can follow the same approach -- with the initial MPI Draft being our first frame of reference. From owner-mpi-comm@CS.UTK.EDU Fri Dec 4 16:10:12 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06597; Fri, 4 Dec 92 16:10:12 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27740; Fri, 4 Dec 92 15:56:38 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27738; Fri, 4 Dec 92 15:56:36 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA07974; Fri, 4 Dec 92 15:56:35 -0500 Date: Fri, 4 Dec 92 15:56:35 -0500 Message-Id: <9212042056.AA07974@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack Name: Address: Email: Fax: Office phone number: Home phone number: (I'll use this only in an emergency and won't circulate.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 21:39:33 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10542; Mon, 7 Dec 92 21:39:33 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29754; Mon, 7 Dec 92 21:28:26 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29750; Mon, 7 Dec 92 21:28:21 -0500 Message-Id: <9212080228.AA29750@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 3637; Mon, 07 Dec 92 21:26:25 EST Date: Mon, 7 Dec 92 14:02:48 CST From: panda@vnet.ibm.com To: mpi-comm@cs.utk.edu To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack NAME: RAJ PANDA ADDRESS: E39/4305 IBM, 11400 BURNET RD., AUSTIN, TX 78758 EMAIL: PANDA@VNET.IBM.COM FAX: 1-512-823-6144 OFFICE PHONE NUMBER: 1-512-838-6632 HOME PHONE NUMBER: 1-512-835-1959 e.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 22:10:17 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10659; Mon, 7 Dec 92 22:10:17 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29855; Mon, 7 Dec 92 21:41:36 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29851; Mon, 7 Dec 92 21:41:26 -0500 Message-Id: <9212080241.AA29851@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 4638; Mon, 07 Dec 92 21:39:36 EST Date: Mon, 7 Dec 92 14:51:31 CST From: LBWARD@AUSVM6.VNET.IBM.COM To: mpi-comm@cs.utk.edu Subject: address for mpi mailing list Please add my name to the MPI forum. Linton Ward Name: Linton Ward Address: 11400 Burnet Rd, Austin TX 78758 Email: lbward@ausvm6.vmnet.ibm.com Fax: (512) 823-6144 Office phone number: (512) 838-5116 Home phone number: (512) 244-1949 From owner-mpi-comm@CS.UTK.EDU Thu Dec 10 13:10:06 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08984; Thu, 10 Dec 92 13:10:06 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA18058; Thu, 10 Dec 92 12:47:20 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18051; Thu, 10 Dec 92 12:47:17 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA27178; Thu, 10 Dec 92 12:47:15 -0500 Date: Thu, 10 Dec 92 12:47:15 -0500 Message-Id: <9212101747.AA27178@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: Information on the MPI meeting Here are some details on the MPI meeting which is set for January 6th-8th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm January 6th and finishing about noon on January 8th. The format of the meeting is: Wednesday, January 6 1:00 pm to 8:00 pm 2 Breakouts for 10 people 1 Breakout for 20 people 5:00 pm to 6:00 pm--On our own for dinner. Thursday, January 7 8:00 am to 10:00 pm 1 U-shape room for 40 people 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, January 8 8:00 am to 12:00 pm 1 U-shape room for 40 people This is an open meeting and all are welcome. For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: January 6-8 February 24-26 April 7-9 May 19-21 June 30-July2 If you have any questions, please feel free to contact me. Jack From owner-mpi-comm@CS.UTK.EDU Wed Dec 23 16:46:29 1992 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA20114; Wed, 23 Dec 92 16:46:25 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA05394; Wed, 23 Dec 92 16:46:20 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14697; Wed, 23 Dec 92 17:31:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 23 Dec 1992 22:31:10 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14687; Wed, 23 Dec 92 17:31:08 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19206; Wed, 23 Dec 1992 17:29:47 -0500 Date: Wed, 23 Dec 1992 17:29:47 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9212232229.AA19206@rios2.epm.ornl.gov> To: abmacca@cs.sandia.gov, benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu, peter@sun.math.usfca.edu, srwheat@cs.sandia.gov Subject: MPI Information Status: RO Dear MPI committee member or observer, I need to know how many people to expect at the Dallas so please let me know if you intend to show up by sending email to walker@msr.epm.ornl.gov Please find below some information about the use of the MPI mailing lists, and who's involved in the various subcommittees. Please check that your entry is correct. If there are any errors please sent email to walker@msr.epm.ornl.gov and to dongarra@cs.utk.edu. Also there's some information about the January 6-8, 1993, meeting in Dallas. 1. Use of Mailing Lists ======================= The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee (subcommitttes + observers) mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail is being collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi ...etc. Also try: send index from mpi 2. Membership of Subcommittees (as of Dec. 22, 1993) ==================================================== 2.1. Introduction Subcommittee: Jack Dongarra (Chair) dongarra@cs.utk.edu Univ. of TN & ORNL Jim Cownie jim@meiko.co.uk Meiko Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@ecs.soton.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gob Argonne National Lab. Peter A. Rigsbee par@cray.com Cray Research, Inc. Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.2. Point-To-Point Communication Subcommittee Marc Snir (Chair) snir@watson.ibm.com IBM T. J. Watson Scott Berryman berryman@cs.yale.edu Yale Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Jack Dongarra dongarra@cs.utk.edu Univ. of TN & ORNL Anne Elster elster@cs.cornell.edu Cornell Univ. Al Geist geist@msr.epm.ornl.gov ORNL Adam Greenberg moose@think.com Thinking Machines Corp. Bill Gropp gropp@mcs.anl.gov Argonne National Lab. Leslie Hart hart@fsl.noaa.gov NOAA/FSL Rolf Hempel hempel@gmd.de GMD Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. of Colorado Charles Mosher ccm@arco.com ARCO Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.3. Collective Communication Subcommittee Al Geist (Chair) geist@msr.epm.ornl.gov ORNL Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Mark Debbage md@ecs.soton.ac.uk Southampton Univ. Anne Elster elster@cs.cornell.edu Cornell Univ. Jon Flower jwf@parasoft.com Parasoft Corp. Adam Greenberg moose@think.com Thinking Machines Corp. Leslie Hart hart@fsl.noaa.gov NOAA/FSL C. T. Howard Ho ho@almaden.ibm.com IBM Almaden John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Charles Mosher ccm@arco.com ARCO Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Marc Snir snir@watson.ibm.com IBM T. J. Watson Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.4. Process Topology Subcommittee Rolf Hempel (Chair) hempel@gmd.de GMD Jim Cownie jim@meiko.co.uk Meiko Jon Flower jwf@parasoft.com Parasoft Corp. Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. Robert L. Knighten knighten@ssd.intel.com Intel SSD Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. Of Colorado Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Lew Tucker tucker@think.com Thinking Machines Corp. 2.5. Language Binding Subcommittee Scott Berryman (Chair) berryman@cs.yale.edu Yale Bruce Leisure bleisure@kai.com Kuck & Associates, Inc. Robert L. Knighten knighten@ssd.intel.com Intel SSD Cherri M. Pancake pancake@cs.orst.edu Oregon State Univ. Peter Rigsbee par@cray.com Cray Research, Inc. 2.6. Formal Language Description Subcommittee Steve Zenith (Chair) zenith@kai.com Kuck & Associates, Inc. Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@cs.ston.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gov Argonne National Lab. 2.7. Environment Inquiry Subcommittee Bill Gropp (Chair) gropp@mcs.anl.gov Argonne National Lab. Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Robert L. Knighten knighten@ssd.intel.com Intel SSD Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.8. Observers Ed Benson benson@rdvax.enet.dec.com DEC Clemens H. Cap cap@ifi.unizh.ch University of Zurich Michel J. Dayde dayde@enseeiht.fr ENSEEIHT-IRIT Jim Feeney feeneyj@gdlvm6.vnet.ibm.com IBM Kyle Gallivan gallivan@csrd.uiuc.edu CSRD, Univ. of Illinois Michael Heath heath@ncsa.uiuc.edu NCSA, Univ. of Illinois Mark Bowen Hill mbh@ecs.soton.ac.uk Southampton Univ. Shlomo Kipnis kipnis@watson.ibm.com IBM T. J. Watson ??? langley@dirac.scri.fsu.edu Florida State Univ. Bob Leary leary@sdsc.edu San Diego Super. Ctr. Lorie Liebrock lorie@cs.rice.edu Rice University Miron Livny miron@cs.wisc.edu Univ. of Wisconsin Ken Kennedy ken@rice.edu Rice University Neil MacDonald nbm@epcc.ed.ac.uk Univ. of Edinburgh Dan Cristian Marinescu dcm@cs.purdue.edu Purdue University Harish Nag harish@ssd.intel.com Intel SSD Dan Nessett nessett@llnl.gov LLNL Angela Ovearly fsang@kira.lerc.nasa.gov NASA Lewis Res. Ctr. Peter S. Pacheco peter@sun.math.usfca.edu Univ. of CA, San Franc. Raj Panda panda@vnet.ibm.com IBM Arnulfo Perez arperez@mtecv2.mty.itesm.mx Center for AI, Mexico Matthew Peters mpeters@vnet.ibm.com IBM UK Sci. Ctr. Jean-Laurent Philippe philippe@archipel.fr Archipel Robert S. Schreiber schreibr@riacs.edu RIACS, NASA Ames Mike Surridge ms@pac.soton.ac.uk Southampton Univ. Tricot tricot@archipel.fr Archipel Robert Voigt rvoigt@note.nsf.gov NSF Linton Ward lbward@ausvm6.vmnet.ibm.com IBM Dennis Weeks weeks@convex.com Convex Francis Wray falk@parsytec.de Parsytec 3. Institutions Involved in MPI =============================== 3.1. Universities Univ. of Edinburgh Emory University N.L. Mexico Centro de Intelligencia Artifical, Mexico Mississippi State University Oregon Graduate Institute Oregon State University Purdue University Rice University Western Michigan University Yale University University of Illinois, CSRD University of Illinois, NCSA University of Southampton University of Tennessee University of Texas at Austin University of Wisconsin University of Zurich University of Colorado 3.2. Laboratories ARCO Exploration and Production Technology Argonne National Laboratory Cerfacs ENSEEIHT-IRIT, France GMD Lawrence Livermore National Laboratory NASA RIACS NOAA Oak Ridge National Laboratory San Diego Supercomputer Center Sandia National Laboratories Supercomputer Research Center 3.3. Companies ARCHIPEL S.A. Cray Research, Inc. Digital Equipment Corporation IBM Austin IBM Almaden Research Center IBM Kingston IBM T.J. Watson Research Center IBM UK Scientific Centre Intel Supercomputer Systems Division Kuck and Associates, Inc. ParaSoft Corporation Parsytec Computer GmbH Thinking Machines Corporation 4. MPI Committee Meeting, Jan 6-8, 1993 ======================================= The next meeting of the MPI subcommittees will take place at Bristol Suites Hotel 7800 Alpha Road Dallas, Texas The meeting will start at 1pm on Wednesday January 6, 1993, and finish at noon on Friday, January 8, 1993. Rooms are $89 per night and reservations may be made by calling (214) 233-7600 (mention MPI meeting). The meeting registration fee will be $75. Please make checks and POs payable to University of Tennessee. Jack Dongarra will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle refer to the MPI meeting. We have made arrangements with Delta Airlines to get a deal on airfares. When you book your flights please call Delta or have your travel agent call 1-800-241-6760 for reservations and refer to File Number S2145. The file goes by the name Message Passing Interface. A certain restrictions apply and seats are limited. An agenda will be sent out soon to all subcommittee members. We intend to rent a laser printer for use by attendees with protable computers. From owner-mpi-comm@CS.UTK.EDU Mon Jan 4 11:02:02 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA06142; Mon, 4 Jan 93 11:02:00 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA08877; Mon, 4 Jan 93 11:01:56 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15255; Mon, 4 Jan 93 11:54:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 04 Jan 1993 16:54:19 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15247; Mon, 4 Jan 93 11:54:16 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.4) id AA21055; Mon, 4 Jan 93 10:54:14 CST Received: by teak18.cray.com id AA05581; 4.1/CRI-5.6; Mon, 4 Jan 93 10:54:13 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9301041654.AA05581@teak18.cray.com> Subject: Profilers etc. (fwd) To: mpi-comm@cs.utk.edu Date: Mon, 4 Jan 93 10:54:10 CST X-Mailer: ELM [version 2.3 PL11b-CRI] Status: R James Cownie writes: > > I have an implementation issue which I would like to raise in the MPI > forum, since it is unclear where it fits into the sub-committee > structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to > those of you who receive this message twice. > > Issue > ===== > The major objective of MPI1 is to achieve portability of applications. > This has major benefits for us all (not least in legitimising and > therefore growing the MPP marketplace). > > One of the benefits which it would also be nice to achieve would be > the wide availability of different tools which support programming in > the MPI1 model. The most immediately obvious such tools (to me at > least !) are > > 1) HPF to Fortran + MPI1 translators > 2) Performance monitoring/tuning tools > 3) Debuggers > > Support for the first is easy (since it just requires what we're > already doing). > > Portable support for the second is not so trivial, since the > collection of useful performance information is much more intrusive. > > Portable support for the third is harder still, and I won't discuss it > further. First, I think this is an important issue. I'd argue that the need isn't necessarily to support portable tools, but it would be nice to define a mechanism by which performance and debugging tools could obtain at least the most basic information about the communications. Second, I don't think it really fits into either group that Jim sent the message to. The same issue, for example, also applies for global communications. This is why I copied the entire committee, and I think there should be some discussion in Dallas was to whether this is a topic that should be covered by the standard. Third, I fear this will be a very difficult subject to address. Clearly Jim had some specific use in mind when he made his suggestions, but didn't go into much detail (and properly so -- this isn't intended as a criticism). But there are some very basic differences between the ways that different tools are designed that need to be considered. What information should be recorded? Is data needed for each transmission (a "trace") or should it be accumulated by the library? Should the library pass information to the tool or should the tool query the library? I don't think there are clear cut answers here. But I think this is an important issue that should receive some airtime in Dallas. - Peter Rigsbee par@cray.com I've included the rest of Jim's proposal for those who didn't see the original mailing... > > Options > ======= > We have various possible options which we can take. > > 1) Ignore the problem > Provide no support for portable performance monitoring tools, and > leave each tool provider with a large porting problem. > > I don't like this solution, it loses some of the benefit of the > standard, which should be attracting people to build tools. > > 2) Document specific implementation hooks as part of MPI1. > In effect these would be callbacks from the library to profiling > code which could then do whatever it liked. > > 3) As 2, but without REQUIRING that a conforming implementation > provide the functions. They're there as a recommendation, rather > than being mandatory. > > > I think we should be concerned about this, and I'd like us at least to > make some recommendation. Personally I'd probably implement two > separate interface to the library, one of which provided the hooks, > and the other of which didn't so that you don't pay the cost of > checking the profiling hooks unless you asked to. > > Of course even if we do nothing it's not too hard to escape (using > horrible macros) in C, but Fortran doesn't always have macros, so a > properly specified internal solution is definitely preferable. > > Thoughts ??? > Flame me at Dallas. I'm travelling tomorrow. When are we having a > meeting in Europe ??? > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England > > Phone : +44 454 616171 > FAX : +44 454 618188 > E-Mail: jim@meiko.co.uk or jim@meiko.com > > From owner-mpi-comm@CS.UTK.EDU Tue Jan 5 17:33:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07561; Tue, 5 Jan 93 17:33:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27130; Tue, 5 Jan 93 17:32:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 05 Jan 1993 22:32:45 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27122; Tue, 5 Jan 93 17:32:42 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA23201; Tue, 5 Jan 93 22:32:36 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01387; Tue, 5 Jan 93 15:31:37 MST Date: Tue, 5 Jan 93 15:31:37 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301052231.AA01387@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: A few general comments Here are a few comments/questions regarding the MPI1 draft proposal from folks here (Forecast Systems Lab). Apologies if you have seen them before! 1) Currently, communication between groups operating in different communication contexts appears to be very difficult. One way to alleviate this problem might be to provide a way to "name" contexts. Communicating groups could then refer to a context by name to ensure that the same message types were used. Message registration would also accomplish this and might be easier to implement (see our "Proposal 1" below). 2) We believe that MPI1 should include a specification of low level protocols that vendors should implement so that heterogeneous computing can be made available. Each media type (standard ethernet, FDDI, HIPPI, etc.) should have a specification. This would greatly aid in this standard being useful in a workstation environment. The protocol document would be a companion document to the syntactical standard for MPI1. 3) On p. 9 "No reference may be made to any process or group outside the current group context." How do different groups that are simultaneously working on different stages of a problem communicate? MPI_PARTG()?? 4) The use of stacks to manage contexts and groups may be a bad idea if MPI ever intends to support MIMD programs (how do groups synchronize stacks of message contexts?). Message registration would do the job without stacks. Registration of blocks of messages at the beginning of a program (or large module) would not affect performance very much. For groups, we like Al Geist's approach of identifying a process by one or more (group ID, process ID) pairs. We feel that communication between groups is very useful, even for SPMD programs. 5) It is clear that some way of responding to asynchronous events must be provided. We believe that "interrupt" type routines (such as Intel's hrecv() and hsend()) may not be the best approach. We have a brief alternative proposal for "Remote Services" below ("Proposal 2"). 6) The consensus seems to be that "parallel I/O" should not be a part of MPI1. Should it be a part of MPI2 or MPI3? If our intent is to create a standard that supports the development of portable code then we believe that I/O must be considered at some point. We should make sure that the specification of MPI1 includes the hooks necessary to include I/O at a later date. If our intent is to create only a low level library, then we can leave all this up to us users. We have some ideas about I/O and Data Decomposition below ("Proposal 3"). Successful implementation of such a proposal would require process topology information (yes, here's a plug for the Process Topology subcommittee). The proposals follow. Regards, Leslie Hart (hart@fsl.noaa.gov) Tom Henderson (hender@fsl.noaa.gov) Bernardo Rodriguez (bernardo@fsl.noaa.gov) PROPOSALS Proposal 1. Message Type Registration. Message types are managed using a global registry. Routines are provided that allow the user to "reserve" blocks of message types for use by the application. Message type registration is intended to prevent message type "collisions" when user code and third-party library routines are used together (historically, a serious problem with typed message passing systems). Initial concepts for message type registration are due to Tony Skjellum of Lawrence Livermore National Laboratory. Two proposed routines to register and monitor message types are described below. Arguments have been omitted for brevity. Message type registration routine: MPI_ReserveMsgType() Reserve one or more message types for use by the calling process. This routine is used to prevent message type "collisions" between user applications and libraries. Inputs include an ID string and the number of message types to be reserved. The routine first checks the locally maintained message type registry and returns an error condition if the ID string matches an existing entry. The routine then goes to the global message type registry to get the message types. If the ID string matches an existing entry in the global message type registry then the message types are copied from that entry to the calling process. Otherwise, new message types are registered with the ID string, entered into the global message type registry, and copied to the calling process. The new ID string and message types are entered into the locally maintained message type registry. Message type registration inquiry routine: MPI_InquireMsgTypeByName() Ask the message type registry to obtain the message types reserved under an ID string. An error is returned if no message types have been reserved under the ID string. Proposal 2. Remote Services (brief concept only). A remote service is similar in concept to a RPC (remote procedure call). Remote services are most often used to respond to specified asynchronous events. The actions taken in response to asynchronous events are performed by agent processes known as protocol handlers. A remote service is requested of one process by the same or another process. The remote service request is accompanied by a small amount of data to guide the action of the service. Routines are provided to allow user-created routine to be attached to specified requests. An attached user-created routine is called a protocol handler. A protocol handler does not have access to the address space of the its associated process. The protocol handler has the status of a full process (ie. it may perform I/O and message-passing communication). This is an advantage over some current interrupt-driven approaches. Remote service requests are handled one at a time on a first-come first-served basis (ie. a new remote service request will not be handled until all pending remote service requests are handled). This may be a drawback to this approach. Proposal 3. Some ideas about Cartesian Data Decomposition and Parallel I/O. Due to length, we have just included a few ideas. If there is any interest, we can provide (lots) more detail. 1) We propose Cartesian data decomposition routines to support the scalable distribution of multidimensional arrays among the processes in a group. These routines isolate the user from the details of implementing scalable data parallelism in a message passing environment. The routines also enhance performance by distributing data in a way that is most advantageous for a particular architecture. 2) A Cartesian data decomposition defines the distribution of a multidimensional data array onto a "logical" array of processes with the same dimension. The processes are members of a specified group. A Cartesian data decomposition is defined by information provided by the user and by run-time parameters determined by the system. The user specifies the group, array dimension, and number of elements in each dimension of the data array. The user may also optionally specify re-ordering of processes in the group and boundary behavior. The Cartesian data decomposition routines then combine this information with the number of processes in the group (provided by the system at run time) to determine which portion of the data array will reside on each process in the group. All information about the decomposition is then stored in a structure (or FORTRAN77 array). The "logical" array of processors is created using the Processor Topology routines (this idea has already been discussed in some detail in that subcommittee). 3) Cartesian data decomposition and I/O should be intimately related. Once a decomposition structure has been created, I/O should be possible using routines with syntax similar to this: cget(file, array, decomp) cput(file, array, decomp) where decomp is the decomposition structure. 4) Data decomposition tools have been available for several years, most notably from Parasoft's Express software package. These tools are extremely useful for almost all data parallel applications. Use of these tools in conjunction with I/O greatly simplifies program development on message passing systems. 5) Few areas in parallel processing have been dealt with in less detail by standardization efforts than parallel I/O. We would like to see behavior defined for such operations as seek/read and seek/write when several processes are accessing the same file. The concept of "loose synchronization" (Express and elsewhere) has proven to be very useful and should be included in a standard. Encapsulation of I/O in standardized routines would also permit performance optimization by the people who (should) know best how to do it (ie. the hardware vendors). From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 11:42:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24005; Wed, 6 Jan 93 11:42:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12383; Wed, 6 Jan 93 11:41:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 16:41:34 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12375; Wed, 6 Jan 93 11:41:32 -0500 Received: from carbon.pnl.gov (130.20.65.121) by pnlg.pnl.gov; Wed, 6 Jan 93 08:39 PST Received: from fermi.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA25662; Wed, 6 Jan 93 08:38:48 PST Received: by fermi.pnl.gov (4.1/SMI-4.1) id AA18870; Wed, 6 Jan 93 08:38:46 PST Date: Wed, 06 Jan 93 08:38:45 -0800 From: Robert J Harrison Subject: Re: Comments on Bill Gropp's comments To: mpi-comm@cs.utk.edu Message-Id: <9301061638.AA18870@fermi.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu At the end of message <9301051955.AA01014@SSD.intel.com> Paul Pierce wrote > > I strongly recommend that we avoid separated info calls. Info can be returned > through output parameters, although that does annoyingly complicate receive > calls. Alternatively, info can be associated with handles, but that seriously > complicates handle management. (Handle management is a good topic for a > separate discussion.) I also strongly favour not having separate info calls. The necessary information is brief and does not excessively clutter argument lists, and can always be hidden in a wrapper if it is routinely not required. Also, by removing sources of ambiguity we give more freedom for implementation specific optimizations. > > If we decide we must have separated info calls, we should use dynamic context > semantics. > Yes to this too. Robert Harrison. From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 17:17:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10712; Wed, 6 Jan 93 17:17:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29683; Wed, 6 Jan 93 17:16:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 22:16:46 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from p.lanl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29672; Wed, 6 Jan 93 17:16:45 -0500 Received: from beta.lanl.gov by p.lanl.gov (5.65/1.14) id AA11282; Wed, 6 Jan 93 15:16:41 -0700 Received: by beta.lanl.gov (5.57/Ultrix2.4-C) id AA29872; Wed, 6 Jan 93 15:15:46 -0700 Date: Wed, 6 Jan 93 15:15:46 -0700 From: sp@beta.lanl.gov (Stephen W Poole) Message-Id: <9301062215.AA29872@beta.lanl.gov> To: mpi-comm@cs.utk.edu I would like to have my name added to the mailing list. Steve... From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:31:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17910; Fri, 15 Jan 93 15:31:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Fri, 15 Jan 93 15:29:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:29:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13112; Fri, 15 Jan 93 15:29:50 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13694; Fri, 15 Jan 1993 15:29:49 -0500 Date: Fri, 15 Jan 1993 15:29:49 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301152029.AA13694@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts I thought the discussion of communication contexts at the last Dallas meeting was lacking in focus, and I'm afraid the existing subcommittees may not deal with this area. If there are enough people interested perhaps there should be a separate communication context subcommittee. If you have any strong opinions on this please let me know. Also let me know if you would like to be on the communication context subcommittee if one is established. On the topic of communication contexts I think (at least) 4 approaches have been proposed. 1) Implicit communication contexts as in MPI1 controlled by push/pop. 2) Explicit registration as in Zipcode 3) Explicit communication contexts should be merged with explicit group contexts. Since groups cannot communicate with each other a new communication context can be created by creating a new group with the same members as the current group. 4) Chuck Simmons of Oracle has suggested that communication contexts are really a particular type of thread, and so can be handled using existing threads packages. If you have any ideas, or are strongly for or against any of the above approaches let mpi-comm know so we can initiate a discussion of this topic. David Walker From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:48:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18263; Fri, 15 Jan 93 15:48:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14054; Fri, 15 Jan 93 15:47:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:47:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14046; Fri, 15 Jan 93 15:47:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03636; Fri, 15 Jan 93 14:47:35 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00709; Fri, 15 Jan 93 14:47:32 CST Message-Id: <9301152047.AA00709@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: walker@rios2.epm.ornl.gov Date: Fri, 15 Jan 93 14:47:31 CST From: Rusty Lusk | | On the topic of communication contexts I think (at least) 4 approaches have | been proposed. | 1) Implicit communication contexts as in MPI1 controlled by push/pop. | 2) Explicit registration as in Zipcode | 3) Explicit communication contexts should be merged with explicit | group contexts. Since groups cannot communicate with each other | a new communication context can be created by creating a new group | with the same members as the current group. | 4) Chuck Simmons of Oracle has suggested that communication contexts | are really a particular type of thread, and so can be handled | using existing threads packages. | If you have any ideas, or are strongly for or against any of the above | approaches let mpi-comm know so we can initiate a discussion of this topic. I think that 1) contradicts our genral agreement to avoid state as much as possible. 4) is a problem on systems that don't support threads, and we have more-or-less agreed to be consistent with thread packages but not depend upon them. 3) should be discussed along with the converse: that contexts rather than groups should be the primary concept and groups defined in terms of contexts. David, perhaps you could create a mailing list on which this discussion can take place, parallel to the subcommittee mailing lists, so it doesn't have to go to the whole committee. Thanks. Rusty From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 16:13:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18561; Fri, 15 Jan 93 16:13:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16784; Fri, 15 Jan 93 16:12:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 16:12:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16768; Fri, 15 Jan 93 16:12:01 -0500 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA21882; Fri, 15 Jan 93 15:11:54 -0600 Received: by mozart.convex.com (5.64/1.28) id AA01950; Fri, 15 Jan 93 15:11:17 -0600 Date: Fri, 15 Jan 93 15:11:17 -0600 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9301152111.AA01950@mozart.convex.com> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: lusk@antares.mcs.anl.gov, walker@rios2.epm.ornl.gov Rusty Lusk writes: > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. I believe the default inclusion of everyone is a good idea, although creating a separate mailing list might be appropriate if some mpi-comm members wish to "unsubscribe". In the meantime, I will add one brief opinion: I felt that the discussions of contexts in the topology subgroup were quite sufficient, and it would probably be better to have the single subgroup which covers both groups and contexts, so that both will appear in the standard in a style which does not make them conflict with each other, and/or to avoid a proposal where groups and contexts would be totally redundant. I definitely agree that it is an important topic, and that the discussions last week in Dallas were far away from getting a proposal for groups and contexts that would be unanimously acceptable, so much work remains to be done. From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 17:01:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19329; Fri, 15 Jan 93 17:01:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21190; Fri, 15 Jan 93 17:00:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 17:00:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21182; Fri, 15 Jan 93 17:00:54 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA20492; Fri, 15 Jan 93 22:00:50 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14823; Fri, 15 Jan 93 14:59:49 MST Date: Fri, 15 Jan 93 14:59:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Communication contexts > | > | On the topic of communication contexts I think (at least) 4 approaches have > | been proposed. > | 1) Implicit communication contexts as in MPI1 controlled by push/pop. > | 2) Explicit registration as in Zipcode > | 3) Explicit communication contexts should be merged with explicit > | group contexts. Since groups cannot communicate with each other > | a new communication context can be created by creating a new group > | with the same members as the current group. > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > | If you have any ideas, or are strongly for or against any of the above > | approaches let mpi-comm know so we can initiate a discussion of this topic. > > I think that 1) contradicts our genral agreement to avoid state as much as > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather > than groups should be the primary concept and groups defined in terms of > contexts. > > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. > > Rusty I'm not sure, but I believe that Zipcode uses push/pop to control contexts (Tony, please correct me if I'm wrong). I'd like to add a 5th approach, basically the one proposed by Paul Pierce: 5) Contexts are used exclusively to insure that message collisions will not occur if independently developed sub-programs are combined. Contexts and groups are orthogonal. Contexts and threads are orthogonal. Each message has an associated context and tag. Message context is managed by library routines and is completely out of a user's control. Message tag is selected by the user. The method of managing message contexts is a separate issue (assuming we want contexts). Existing proposals are: 5a) Stack-based management (objected to due to hidden states). 5b) Explicit registration with user-defined "names" (probably requires some communication). 5c) Explicit registration by a central authority ("dollar bill" registration mentioned by Jim Cownie.) There is enough confusion about contexts and groups that we may want to keep this discussion open to everyone. If the concensus is to make a separate mailing list, please both of us to it. Thanks, Tom Henderson hender@fsl.noaa.gov Leslie Hart hart@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 19:06:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21669; Fri, 15 Jan 93 19:06:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27214; Fri, 15 Jan 93 19:05:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 19:05:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27206; Fri, 15 Jan 93 19:05:40 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00450; Fri, 15 Jan 93 18:05:36 CST Date: Fri, 15 Jan 93 18:05:36 CST From: Tony Skjellum Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu Subject: Re: Communication contexts In Zipcode, contexts have a group-wide scope. Groups of processes can involve more than one context. For instance, if 10 processes are involved in several stages of a calculation, each stage might have a different notion of how to do messaging, but each would use all the processes. Groups are important, because they allow control over the scope of global operations. Contexts are important because they provide guarantees to sub-programs about restricted scope of messages. Contexts are added typing-like information, but which is controlled by the system, not ad hoc by each user program. When they determine to start messaging, sub-programs request a context from a "postmaster general" in Zipcode. Currently, this is a group-oriented request (loose synchronization) resulting in each group member getting the context, and other information. In a lower-level system, simpler tactics might be possible. The way we did it was supposed to avoid possibility of races or other problems like that. To build large-scale software, without globalization of the local use of messaging resources, and to provide scalable algorithms with efficient global operations, both groups and contexts are needed. The push/pop state is an implementation detail of little importance. -Tony From owner-mpi-comm@CS.UTK.EDU Sat Jan 16 02:07:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26026; Sat, 16 Jan 93 02:07:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11696; Sat, 16 Jan 93 02:06:16 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 16 Jan 1993 02:06:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11688; Sat, 16 Jan 93 02:06:13 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA06036; Sat, 16 Jan 93 02:01:08 EST Date: Sat, 16 Jan 93 02:01:08 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301160701.AA06036@cs.wmich.edu> To: mpi-comm@cs.utk.edu hi; It was my feeling that the concepts of group and context, that were clearly understood differently by different people at the start of the discussions, became somewhat better agreed on as the pt2pt discussion took place. Groups - used to define groups of processors for collective communication primitives and to allow mapping topologies onto processors. This allows a group of processors to be used in isolation to compute together. Contexts - As Tom (and Tony) express below, > From: hender@macaw.fsl.noaa.gov (Tom Henderson) > Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> > To: mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > I'd like to add a 5th approach, basically the one proposed by Paul Pierce: > > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. The only expression stated so far about the difference between tags and contexts I had gleaned was that context should now be wildcardable (IE no -1 for All contexts) while a receive ALL or some other MASK variant would be allowed on tags. > > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) Add: 5d) A context is not strictly managed, but use should follow suggested guidelines. This could allow static allocation of contexts when reasonable, but also support a context server (as the Postmaster General in Zipcode). Think of the static contexts or non-registrar provided contexts as reserved with respect to any registration system. This could be part of the MPI initialization call, if a registrar is included. The drawback here of course is the user could violate whatever guidelines were given. It seems to me most types of name registration or stack allocation schemes, that are not too restrictive and allows user code to interact, are likely to allow errant management of contexts as well. I expect that allowing 5d) would let strictly controlled context systems to be built over MPI. > From: Tony Skjellum > Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> > To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > In Zipcode, contexts have a group-wide scope. Groups of processes can Of course if we eliminate groups from the pt2pt call then contexts may not be explicitly limited to groups in MPI. (though a context server could be asked to provide a context relative to a group). > involve more than one context. For instance, if 10 processes are involved > in several stages of a calculation, each stage might have a different notion > of how to do messaging, but each would use all the processes. Groups > are important, because they allow control over the scope of global operations. > Contexts are important because they provide guarantees to sub-programs > about restricted scope of messages. Contexts are added typing-like > information, but which is controlled by the system, not ad hoc by each > user program. When they determine to start messaging, sub-programs > request a context from a "postmaster general" in Zipcode. Currently, this ... > To build large-scale software, without globalization of the local use of > messaging resources, and to provide scalable algorithms with efficient > global operations, both groups and contexts are needed. The push/pop > state is an implementation detail of little importance. I expect the use push/pop for context state would make implementation in some environments hard and I have some concern about requiring registration on very large systems. I do think a strictly controlled stack context system could be built over 5d). The question about context management relates to how high a level MPI should be at. I see it at a low level, that higher level systems can be built on. That is one reason for keeping context management "advisory". > > -Tony > I liked the zipcode documentation. I think it would be a good idea to make it available on netlib. john From owner-mpi-comm@CS.UTK.EDU Sun Jan 17 11:46:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04093; Sun, 17 Jan 93 11:46:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09441; Sun, 17 Jan 93 11:43:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 17 Jan 1993 11:43:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09433; Sun, 17 Jan 93 11:43:55 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA08023; Sun, 17 Jan 93 11:38:43 EST Date: Sun, 17 Jan 93 11:38:43 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301171638.AA08023@cs.wmich.edu> To: mpi-comm@cs.utk.edu Although the question of IO specification in the MPI-1 has been raised before, I think it still needs to be addressed. Omission of any IO requirements means no program can be portable. If IO specifications are not included the introduction should make a strong case for the reasons. Any attempt to specify high level parallel IO at this point in time seems premature. Perhaps there can be a middle ground. We could specify low level IO, compatible with ANSI/ISO/POSIX standards and perhaps allow an inquire function to test for its presence. I will suggest every process should have access to Level-1 IO, given in the following list, which works for c and c++. -------------------------------------------------------------------- Level-1 Input and Output A process should have access to: 1) open/close/flush read/write lseek 2) readv/writev 3) fcntl dup dup2 ioctl 4) fopen/fclose/fflush scanf/printf fscanf/fprintf sscanf/sprintf 5) varargs vscanf/vprintf vfscanf/vfprintf vsscanf/vsprintf Behavior of these functions is to match that in current standards (specific standard to be chosen). -------------------------------------------------------------------- This could be loosened to only require defined behavior when access is by a single process. A Like Level-1 IO can be specified for Fortran-77 as well. This type of facility is provided in many vendor environments and several portable environments as well. A prototype version of the Level-1 IO could be built using gnu sprintf/sscanf, MPI-1 message passing and the assumption that some process can do IO. -------------------------------------------------------------------- Level-2 Input and Output A case can be made for Level-2 Input and Output as well. This would be group IO, no claim would be made for efficiency, only that the behavior was defined and would not overload the communication network. All processes in a group would execute the same IO calls as in level-1 and the joint behavior would be defined. To avoid state a modified name for each function could be used. A generic level-2 IO could be built on level-1 and the rest of MPI-1. -------------------------------------------------------------------- I propose we include a Level-1 specification and I think we should consider Level-2 as well. I am willing to draft a Level-1 suggestion and a Level-2 suggestion. I'm not sure which mailing list to put this on, but I think the issue needs to be addressed quickly (even if the decision remains to ignore IO). john From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 09:30:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14975; Mon, 18 Jan 93 09:30:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16251; Mon, 18 Jan 93 09:29:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 09:29:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16243; Mon, 18 Jan 93 09:29:24 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12246; Mon, 18 Jan 1993 09:29:23 -0500 Date: Mon, 18 Jan 1993 09:29:23 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181429.AA12246@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI news 1) NEW SUBCOMMITTEE A new subcommittee on profiling has been set up. It is chaired by Jim Cownie of Meiko, and you can send email to it at mpi-profile@cs.utk.edu. Please let me know if you want to join 2) NEWCOMERS FILE If you are a newcomer to the MPI forum or want general information send the message send mpi-newcomers from mpi to netlib@ornl.gov 3) REVISED VERSION OF MPI1 TECH REPORT A revised version of the MPI1 paper by Dongarra, Hempel, Hey , and Walker is available from netlib. This is basically the original document with some unnecessary and inappropriate routines removed. The PostScript file can be obtained by sending the message send mpi1.ps from mpi to netlib@ornl.gov 4) NEXT MPI MEETING Here are some details on the MPI meeting which is set for February 17th-19th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm February 17th and finishing about noon on February 19th. The format of the meeting is: Wednesday, February 17 1:00 pm to 8:00 pm point to point subcommittee meeting 5:00 pm to 6:00 pm--On our own for dinner. after dinner: other subcommittees meet Thursday, Febrauary 18 8:00 am to 12:00 pm Collective communication subcommittee meeting 1:00 pm to 6:00 pm Subcommittee reports presented to the main group 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, Febraury 19 8:00 am to 12:00 pm Subcommittee reports presented to the main group For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: March 31-April 2 May 19-21 June 30-July2 From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 14:20:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21493; Mon, 18 Jan 93 14:20:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27089; Mon, 18 Jan 93 14:19:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 14:19:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27075; Mon, 18 Jan 93 14:18:59 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA15953; Mon, 18 Jan 1993 14:18:58 -0500 Date: Mon, 18 Jan 1993 14:18:58 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181918.AA15953@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Contexts group A new subcommittee and associated email group has been set up to responsible for MPI communication contexts. The current group membership is as follows: tony@cs.msstate.edu dongarra@cs.utk.edu lusk@mcs.anl.gov weeks@convex.com hender@fsl.noaa.gov john@cs.wmich.edu csimmons@us.oracle.com walker@msr.epm.ornl.gov If you would to join (or leave) this subcommittee please send email to me. You can send email to the group at mpi-context@cs.utk.edu. You can get the archived email for this group by sending the following message: send mpi-context from mpi to netlib@ornl.gov Regards, David From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:29:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19476; Tue, 19 Jan 93 12:29:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24138; Tue, 19 Jan 93 12:28:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:28:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24130; Tue, 19 Jan 93 12:28:25 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26202; Tue, 19 Jan 93 17:28:13 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02004; Tue, 19 Jan 93 10:30:14 MST Date: Tue, 19 Jan 93 10:30:14 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02004@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu > From: john@cs.wmich.edu (John Kapenga) > To: mpi-comm@cs.utk.edu > > Although the question of IO specification in the MPI-1 has been > raised before, I think it still needs to be addressed. Omission > of any IO requirements means no program can be portable. If IO > specifications are not included the introduction should make a > strong case for the reasons. We (Tom Henderson and myself) agree and have also suggested such a proposal. Ours had a little more high level stuff, but not much. We do think you need some specification of how basic I/O performs in a parallel environment to provide a high level of portability. We think opportunities for parallelism can be expressed as well by routines that are loosely snychronous. > ... > Any attempt to specify high level parallel IO at this point in time > seems premature. Perhaps there can be a middle ground. We could > specify low level IO, compatible with ANSI/ISO/POSIX standards > and perhaps allow an inquire function to test for its presence. > I am willing to draft a Level-1 suggestion and a Level-2 suggestion. > I think that is a good idea and would be interested in cooperating on such an endeavor. > I'm not sure which mailing list to put this on, but I think the > issue needs to be addressed quickly (even if the decision remains > to ignore IO). Neither am I, let's hope it's not another subcommittee ;-) > > john > Leslie Hart (hart@fsl.noaa.gov) From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:32:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19727; Tue, 19 Jan 93 12:32:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24283; Tue, 19 Jan 93 12:31:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:31:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24275; Tue, 19 Jan 93 12:31:22 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26204; Tue, 19 Jan 93 17:28:14 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02007; Tue, 19 Jan 93 10:30:15 MST Date: Tue, 19 Jan 93 10:30:15 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02007@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov, csimmons@us.oracle.com Subject: Re: "Message Capsule" Proposal (long) > From: Charles Simmons > To: bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov > Subject: Re: "Message Capsule" Proposal (long) > > As feedback... > > The message capsule proposal might be a bit high level. E.g. it > seems to implement the topmost layer of the ISO 7-layer communications > model which deals with transmitting data between processes running > on heterogeneous machines. I'm not familiar with with the ISO 7-layer approach, but what we were proposing was a place for each local process (or thread) to store information about one or more messages. Other words for capsule might be "handle" or "envelop". > > " > 6.1 Advantages > > - The number of separate send and receive routines is greatly reduced > without sacrificing functionality. > - A user who is used to "common practice" can use the simplified > versions of the routines. > - A user who wants more flexibility only needs to learn about the features > required for his/her specific application. (For example, if I only need > contiguous messages, then I don't need to know anything about strided data > items. If the receiving process always knows the data description of > received messages, then I don't need to know about the probe and "info" > routines.) > - "Hidden states" are removed so multi-threaded applications won't get > confused. > - Encapsulation of features in message capsules allows new features to be > added later without modifying syntax of existing routines. A new feature > would require addition of one or more new routines to modify and examine > message capsules. > " > > While the number of separate send and receive routines is reduced, > the number of other routines are correspondingly increased (e.g. attach > and info routines). The increase in routines is not combinatorial, but rather additive. For three new modes each of which have two possible settings, there is a maximum of six (6) new routines (ok, maybe 12 if there are other inquiry functions). In another approach the number of routines would be *mulitplied* by two for each new mode (i.e. multiplied by 8). > > Even in the MPI approach, users can limit themselves to simple versions > of various rotuines. Agreed, but I don't think the document has been taken to heart as a syntactical or semantical starting point. > > I don't see how capsules keep multi-threaded applications from > getting confused. If you have two threads in a receiver that are > each waiting for messages that look pretty much the same to the OS > or compiler or library code, it would appear to be fairly easy to > give the messages to the wrong threads. The sender really needs > to attach a thread identifier to a message under certain circumstances. > This responds to some issues that hidden state cause with multithreading. Such as what does the last message mean? We don't claim that it deals with which thread should be a target in a send. It just removes confusion if you probe and then wish to receive the message that you probed about. > > [There was a brief thought I had that this proposal reminded me of... > For certain types of applications, it would be possible for the sender > to know the layout of data in a receiver's address space. In an RPC > application, the receiver could transmit a description of the layout > to the sender. In an SPMD application, the sender might just know without > being told. This would allow the sender to prepend the layout description > to her message, and would allow the receiving OS to splatter received data > directly into user address space.] > > G'luck, Chuck Thanks for your input, we wanted to start a discussion about these issues. (BTW, your response was directly to us, so we included it in its entirety to mpi-comm). Leslie Hart Tom Henderson Bernardo Rodriguez hart@fsl.noaa.gov hender@fsl.noaa.gov bernardo@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Tue Jan 26 16:04:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21467; Tue, 26 Jan 93 16:04:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20893; Tue, 26 Jan 93 16:03:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 26 Jan 1993 16:03:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20885; Tue, 26 Jan 93 16:03:47 -0500 Received: by enet-gw.pa.dec.com; id AA27890; Tue, 26 Jan 93 13:03:06 -0800 Message-Id: <9301262103.AA27890@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Tue, 26 Jan 93 13:03:45 PST Date: Tue, 26 Jan 93 13:03:45 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: POSIX 1003.4 message queues considered ? Has anyone involved in the MPI effort looked into POSIX 1003.4 (Realtime) message queues ? -Ed From owner-mpi-comm@CS.UTK.EDU Wed Jan 27 12:03:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08441; Wed, 27 Jan 93 12:03:40 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09490; Wed, 27 Jan 93 12:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 27 Jan 1993 12:02:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09480; Wed, 27 Jan 93 12:02:24 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02128; Wed, 27 Jan 93 11:02:21 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA16675; Wed, 27 Jan 93 11:02:16 CST Message-Id: <9301271702.AA16675@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: minutes of January MPI meeting Date: Wed, 27 Jan 93 11:02:15 CST From: Rusty Lusk Minutes of the Message Passing Interface Standard Meeting Dallas, January 6-8, 1993 The MPI Standards Committee met in Dallas on January 6-8, 1993, at the Bristol Suites Hotel in North Dallas. This was the third meeting of the MPI committee, but the first following the format used by the High Performance Fortran Forum. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. Because interest in the Point-to-Point communications and the Collective communications was so general, these met as committees of the whole. No formal decisions were taken at this meeting, but a number of straw votes were taken in the subcommittees. These are included as part of the reports on the work of the subcommittees. These minutes were taken by Rusty Lusk (lusk@mcs.anl.gov) with some additions by Bob Knighten. Marc Snir's notes on the point-to-point subcommittee meetings are included here as well. These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly the lead to each topic (and a few other things.) January 6 --------- ------------------------------------------------------------------------------- General Meeting ------------------------------------------------------------------------------- The meeting was called to order by Jack Dongarra at 1:30. Jack Dongarra presented the rules and procedures that had been circulated in the mailing list. In general, they say that we intend to operate in very open fashion, following the example set by the High-Performance Fortran Committee. He also described the subcommittee structure. For details, see the mailing list, A tentative schedule for future meetings was presented, which was amended on the last day (see there). All meetings will be in Dallas at the Bristol Suites. Steve Otto will coordinate the production of the document. He will obtain a set of LaTeX macros from the HPF Committee and distribute them to the subcommittee heads. It was suggested by Bob Knighten that the Executive Director arrange for copies of all pertinent documents be provided at the meetings. Dennis Weeks, who is somewhat local (Convex), volunteered to help with the relevant copying. The attendees were: Ed Anderson Cray Research eca@cray.com James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Jim Feeney IBM-Endicott feeneyj@gdlvm6.vnet.ibm.com Jon Flower ParaSoft jwf@parasoft.com Daniel Frye IBM-Kingston danielf@kgnvma.vnet.ibm.com Al Geist ORNL gst@ornl.gov Ian Glendinning Univ. of Southampton igl@ecs.soton.ac.uk Adam Greenberg TMC moose@think.com Bill Gropp ANL gropp@mcs.anl.gov Robert Harrison PNL rj_harrison@pnl.gov Leslie Hart NOAA/FSL hart@fsl.noaa.gov Tom Haupt Syracuse U. haupt@npac.syr.edu Rolf Hempel GMD hempel@gmd.de Tom Henderson NOAA/FSL hender@fsl.noaa.gov C. T. Howard Ho IBM Almaden ho@almaden.ibm.com Steven Huss-Lederman SRC lederman@super.org John Kapenga Western Michigan Univ. john@cs.wmich.edu Bob Knighten Intel SSD knighten@ssd.intel.com Bob Leary SDSC leary@sdsc.edu Rik Littlefield PNL rj_littlefield@pnl.gov Rusty Lusk ANL lusk@mcs.anl.gov Barney Maccabe Sandia abmacca@cs.sandia.gov Phil McKinley Michigan State mckinlehy@cps.msu.edu Chuck Mosher ARCO ccm@arco.com Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Institute otto@cse.ogi.edu Paul Pierce Intel prp@ssd.intel.com Peter Rigsbee Cray Research par@cray.com Ambuj Singh UC Santa Barbara ambuj@cs.ucsb.edu Marc Snir IBM snir@watson.ibm.com Robert G. Voigt NSF rvoigt@nsf.gov David Walker ORNL walker@msr.epm.ornl.gov Dennis Weeks Convex weeks@convex.com Stephen Wheat Sandia NL srwheat@cs.sandia.gov ------------------------------------------------------------------------------- Point-to-point subcommittee ------------------------------------------------------------------------------- Mark Snir called the meeting to order at 1:40 p.m. It adjourned at 4:10 p.m. It resumed the following morning at 9:10 a.m. and adjourned at 4:15 p.m. Marc Snir began by summarizing the decisions that we have to make: * which operations? send receive channels? sendreceive? info arguments operations on queues probe? * operation modes sync async local and/or global termination interrupt-driven? * message types (data types) structure of data in core buffer packing * send-receive matching type (We later decided to call this "tag".) sender? * correctness criteria (See Marc Snir's paper in handouts) * heterogeneous operations * name space how processes are addressed flat? structured? implicit/explicit * error handling * interaction with threads, interrupt handlers, remote signalling * special operations for high performance ready receiver? * process startup * syntax/style (The plan is to postpone this for this meeting.) We will prioritize this list and then go through them one by one. (The priorities assigned were more or less in the order listed above.) Two preliminary questions were then discussed: A. Must we worry about multithreaded environments? James Cownie pointed out that threads were coming, in almost all new systems. Most systems have threads now. It was proposed that a process, which could send and receive messages, should be an address space, so that individual threads would not be (MPI-) addressable. B. What about signals? Paul Pierce suggested that we discuss signals first: do we want to support send/receive from interrupt handlers? These two questions were then discussed at length. Dealing with threads argues against the notion of "last message", since that implies state is maintained by the system. There was general agreement that "state" was a ` bad thing, but arguments in favor of state are: Sometimes one doesn't want all of the information available after an operations, so it shouldn't be returned. Having lots of arguments to calls is bad, especially inout arguments. Ways to avoid state are: Structures could be returned Return individual arguments Return tag to do queries on (but they one needs to free it.) Additional out arguments (OK in Fortran 90, but not in C or f77) User passes in storage to be used (so he knows the address), and MPI provides inquiry functions [For more details, see Jim Cownie's mail message of January 4, 1993 entitled: Multifarious] There was a general agreement that system state decreases portability and manageability, and we should decrease it when we can. James Cownie said that We need a reentrant style, and Mark Snir suggested that we try to make all function calls reentrant. When queried no one in the group objected to trying to make all the functions that are introduced in MPI reentrant. Now we began going through the above-mentioned major topics. Which Operations? ----- ---------- We have send and receive. How about send-receive (also called shift)? It can be efficiently implemented, and buffer can be reused. There was a discussion of the "two-body" send-receive (exchange) and the "three-body" version (ring-shift). Variations include those in which the send-buffer is required to be the same as the received- buffer and those in which is is required to be disjoint from the receive-buffer. Al Geist: We should focus on *required* operations. Steve Otto replied that send-receive *is* a required operation. Using "exchange" can help avoid deadlock. It was agreed that there was no consensus on these issues and it was decided to defer this to the collective communication subcommittee. Operation Modes --------- ----- The next topic that Marc Snir raised for discussion was when do send and receive return. Marc described several options: For send: 1) return as soon as possible 2) return when send-buffer is clear 3) return when the corresponding receive has completed For receive: 1) return as soon as possible 2) return when the receive-buffer is full "Receive has completed" means "when the user knows". In other words, when the sender has returned from send, the receiver has returned from receive. There was a general discussion about whether 3) was necessary? dangerous? Robert Harrison said he believed that 3) was the minimal version that was truly portable. Steve Otto pointed out that 3) is CSP-like. Rusty Lusk said that 3) would be easier to prove things about than the others. Adam Greenberg and Paul Pierce pointed out that neither TMC nor Intel have implemented an operation depending on the behavior of the receiver. A straw vote was taken and the vote was 17-3 in favor of having 3) as an option. Marc Snir pointed out that in his original proposal send returns a handle and the status of the handle is then tested for completion of the send operation, and asked if this is this desirable. There was general agreement that something of this sort was desirable, but a variety of alternatives were mentioned It was pointed out that sometimes one wants to wait on multiple outstanding operations. Al Geist prefers separating "wait" into "sendwait" and receivewait" for code readability. Bill Gropp suggested that instead of using handles, one could supply a routine to be called when an operation completes. James Cownie: "This gets really hairy in Fortran". There was a discussion of probing multiple outstanding receives: If the receives return handles, h1 = recv( ... ) h2 = recv( ... ) wait ( h1 or h2) ? wait ( h1 and h2 ) is not needed. Jame Cownie proposed that we supply an operations to *wait* on a vector of handles, which would return one of those that have succeeded. It would return the handle, not the status. A straw vote as taken on this proposal, which passed 17 - 0. So we have: status (handle) wait (array of handles) The send specifies what completion of send means. Handles need to be freed. It was pointed out the only the existence of such an operation has been decided, the semantics are yet unspecified - e.g. issues such as fairness or what wait returns when several complete are not yet specified. There was a long discussion of cancellation of send and receive. It was observed that there are serious implementation problems because of race conditions, freeing resources, etc. A straw poll was taken on including cancel in the initial MPI. It failed 7-19. This was the end of the Wednesday afternoon point-to-point meeting. January 7 --------- The point-to-point subcommittee (now a Committee of the Whole) resumed at 9:15 a.m. on Thursday morning. Marc Snir opened the meeting and summarized the progress so far: 3 ways in which send can terminate sendreceive postponed no cancel of incomplete send operation status and wait (successful status accomplishes same as wait) We did not get to: channels (the idea of trying to bind as soon as possible as many parameters as possible, so that they can be reused.) probe readyreceive Marc noted that channels and readyrecv address similar issues. Probably want only one of these. Do we want either? Rolf Hempel observed that we don't need channels - can depend on operating system to cache the connection information when doing synchronous communication. Adam Greenberg replied: NO! Want to be able to do this all at user level without "smart" OS. Channel creation and use might look like: handle = send_init( ... ) start(handle) wait(handle) free(handle) This is an intermediate point between bundled send/receive and full named channels. Indeed there are many intermediate points based on various early bindings. Is there enough experience to justify a standard? Bob Knighten observed that there has been substantial experience with channels on the iWarp system. There was next a discussion of the ready-receiver semantics proposed by Lusk and Gropp in the handouts. Steve Huss-Lederman said that such operations could make a difference of as much as 25% for matrix multiplication on the Delta. Some doubt was expressed about the universality of this optimization. Question of use of readyrecv by naive users again. Cownie mentions experience again. Greenberg: facilities for efficiency should not make it difficult to write correct programs. Wheat: Don't penalize users who do understand and can take advantage of efficient procedures. General back and forth discussion. Two straw votes were taken: Ready-receiver operations passed 13 - 10 Channels passed 19 - 2 (Marc Snir will write up a detailed proposal) The next topic discussed was the probe operation. Do we want such an operation, and if so, what should be its semantics? Probing must "lock" the message that it finds, else the information returned by the probe may be unreliable. (Consider the multithreaded environment.) Bill Gropp pointed out that probe is often used to find out the length of a message so that a buffer of the appropriate size can be allocated. Marc Snir pointed out that this is a problem with the November document, that we need to know the length of a message ahead of time. Jon Flowers suggested the need for a blocking probe. What is needed is to probe and then to receive the message found via the probe: handle = probe(params) . . . recv(handle) release(handle) Marc Snir pointed out that the handle serves as a lock on the message. James Cownie pointed out that while we agreed to not have a cancel for a send, we do need to be able to cancel receives, since an outstanding receive is permission for the system to write in the user's address space, which is a permission the user may want to revoke. A straw vote was taken on the existence of some form of probe, and it passed 25 to 0. Send-Receive Matching ------------ -------- The next topic is the matching of send and receive. Currently we have to discuss matching on: tag sender group id context id We will also need to discuss the name space issue for messages. Here are three proposals for the predicate that determine whether a message matches a particular receive: 1) simple matching on fields 2) more general, with mask, etc. 3) user defined function Adam Greenberg said that at TMC: A user defined function is used by the system whenever a message is received by a node to decide if it is to actually be received by the application. The parameters to the user defined receive predicate are tag and group. Issue: If most information is encoded in the tag, then the tag protocol must be understood by all users involved in writing a particular application. True, but not a serious problem. Best to identify small class of specific matching parameters (e.g. group) and use tag for everything else. James Cownie pointed out that the matching function, if not too complicated, can (and is, on many systems) done by special communications processors. There was further discussion of the difficulties of having the system call user code for screening messages. Paul Pierce pointed out that receipt of a message by the hardware is a crucial point for performance. There was general discussion of alternative approaches to getting at least some of this. The question of need for this generality was also raised. TMC has a user who wants and uses his own predicate function. Possibilities: (a) select on mask for fields (including a don't care); (b) simple static logical operations on fields; (c) user defined (b) might be match = AND (( message(i) = pattern(i) ) OR mask(i)) fields A straw vote was taken on whether to pursue allowing user-defines predicates. It was decided 26-1 not to allow user-defined functions for this purpose. (b) deferred until a proposal is available. Marc Snir summarized that matching by tag is generally agreed on and that this is not the only item for selection. After some discussion, matching by sender was also generally agreed on. So now, how do we identify a sender? Rusty Lusk spoke in favor of a flat name space, so that processes could be addressed independently of group, etc. There ensued a general discussion of groups, contexts, and the name space. It was pointed out that the name space expected by send could be flat and groups could be implemented by a function that converted any structured name into a flat integer id. Other proposals were to to have name=(rank,gid) with the restriction that this name be usable only within the given group (gid) and the sender must be a member of this group. By default the group would be ALL. Other alternatives mentioned were name=(rank,ALL)=pid and name=(pid,context). This led to a general discussion of context and the relation to groups. Marc Snir pointed out that we could have pid pid,context in which context did not change the meaning of pid. Paul Pierce said that tags and contexts should be separated since they need to be handled in different ways. Marc Snir pointed out that there should be no "don't care" on context. There was a discussion of servers that can process "any" message. This also led into discussion of flat name space vs. hierarchical name space where we would have pid(group, rank) function. Can use context to define groups, but there are other uses as well. Why groups as well as context? What is the difference between context and groups? Cownie: Context is just another integer used in the same manner as tag. Not quite - it is reserved, but what is the meaning of "reserved"? Greenberg was concerned about connecting send/receive behavior with groups. Snir: Suppose a users wants to have two independently written subroutines that use the usual rank notation. Wheat: Similarly want to use rank notation when partitioning machine. Snir: Both contexts and groups are nice, but do we need both? Gropp: Problem with mixing two applications both of which use 0-based indexing will need a larger common name space when they need to communicate. There was a general discussion of the cost of contexts. Cownie observed that context is cheap if only used to distinguish code - obtain a unique context id for the code by means of the "one-dollar random number generator": each author obtains a one-dollar bill, copies the serial number, and then burns the bill. But in general context is not cheaper than groups. Someone asked about spawning additional processes while program is running? Various people raised the question: If use name=(pid,context), does context change the meaning of the pid (i.e. is pid context {or gid or ???} relative.) There was some discussion of message registration. Paul Pierce observed that tag vs. context is only matter of registration. He wants to divorce tag and context for safety. This implies that one cannot use wild card for selecting on context. Various people noted difficulties with mixing tag and context. Adam Greenberg offered: Proposal - always separate tag and context. Have a context, NONE, so that pid with context NONE is unmodified, but with other contexts the pid may be relative. [NONE, GLOBAL, BASE] tag, context - must match on context Several people noted that there are two very different uses of context - identification of distinct code and identification of a group of processors. There is state, even distributed state, associated with remapping of processors with groups. POSSIBLE FIELDS FOR SEND/RECEIVE: tag context id group - no wild card - set of processes - registration management - receive only from group - managed by system Marc Snir asked whether we could agree on what would be carried with a message: tag context (like tag, except no wild card; management to be determined) Two straw votes were taken: Having contexts passed unanimously. Having the context *not* modify the process id passed unanimously. Groups ------ Three alternatives: no groups (use send(pid(group, rank), ...) instead) group as explicit parameter in operations use contexts to implement groups The basic difference is: do we want to be able to select on group? Straw vote: yes: 10 no:11 on the capability of selecting by group. (Thursday lunch occurred here) Message Data Types ------- ---- ----- WHAT IS A BUFFER? (Language bindings are going to be important here.) There are many options to consider: a) contiguous bytes (non-controversial) General agreement that 0-length messages should be allowed. b) contiguous buffers of other (implementation specific?) units? b) stride? (parameters: base-address, element-length, stride, number-of-elements) c) basic data types? d) arbitrary structures? e) How will we specify data to be communicated in a heterogeneous environment? f) iovec structures (array of pointers and lengths, as in un*x iovec for reads and writes) Marc Snir pointed out that one possibility is to have separate pack/unpack routines and then just send contiguous buffers. Rusty Lusk pointed out that this requires a copy that may be unnecessary on certain machines. Two choices - pack scattered buffer and send OR send scattered buffer. If the second, then may need a pack that produces the descriptor of a scattered buffer to be used by the send scattered buffer. Straw poll: Use IOVEC type send. Passed 18-1 Basic data types were deferred. Marc Snir observed that up to this point, a message is a set of bytes in storage, but now we are about to consider more meanings: message = sequence of *values* Should we use the same calls for homogeneous and heterogeneous calls? Can we have a fast homogeneous implementation of the heterogeneous calls? Bill Gropp pointed out that the current testbed implementation does this. SEND vs. SENDCHAR, SENDREAL, . . . To be compliant with F77 need to have at least SENDCHAR for correctness (and this is a real issue, e.g. on VAX.) Strictly need to have different for each basic data type (but in practice this is not an issue.) But for other than CHARACTER there is also an efficiency issue. 1. F77 conformance 2. Special problem of CHARACTER 3. Performance 4. Heterogeneity (?) Postpone to language binding discussion. This led into the issue of the general problem of converting types between languages and machines! This in turn led to a discussion of XDR (and mention of other systems such as NDR, ...) XDR supports the basic types (INT, REAL, COMPLEX, CHAR, etc.), array constructors, pack/unpack routines, etc. Do we use the same calls for homogeneous and heterogeneous systems? Can we have a fast implementation of heterogeneous procedures for a homogeneous system? What about a "message envelope" that specifies the environmental aspects of messages (e.g. heterogeneity features such as XDR.) When we talk about heterogeneity, do we expect MPI libraries from different vendors on different machines to cooperate? Include general SEND as SENDBYTES? Agreed that do not want SEND in homogeneous to require type information needed for heterogeneous environment. There was a discussion of whether we have to pick an interchange format, for example XDR. There seemed to be some agreement that we do (as MPI implementations from different vendors have to be able to communicate with one another), but no vote was taken. Error Handling ----- -------- The main issue here is whether an error detected by an MPI routine should result in the calling of an error-handler or return of a return code. Other issues are how much of error handling should be standardized as opposed to implementation-dependent, and how much user control there should be over error-handling. There are two types of error environments - soft (recoverable) and hard (unrecoverable). In a soft error environment there is the opportunity for cleanup on the part of both the "application" and the system, while in the hard error environment the system will cleanup and terminate the application. Choices: An mpi routine always returns (though it may return with an error code.) An mpi routine may call an exception handler There may be a default exception handler and there could be a user-installable one as well. Library writers may want to handle errors differently from how a user program wants to handle them (or have them handled by the system). Robert Harrison described the error modes used in TCGMSG and p4: A process has a user-settable state that determines whether an error should result in a (hopefully) graceful takedown of the program or in a error return code. Paul Pierce described the Intel method which uses two syntactically distinct classes of functions. For one class an error results in a message being printed and the process in which the error occurred terminating. For the other class an error code is set. There was some discussion of the problem of maintaining state in a multithreaded environment. Two straw votes were taken: Do we want a version of MPI that calls an exception handler: yes: 23 no: 0 Do we want a version with return codes: yes: 19 no: 1 Specific discussion of modes or "shadow" routines was deferred. Correctness Criteria ----------- -------- This concerns defining what is a correct implementation of MPI An assumption that had to be restated several times during the meeting is that MPI assumes a reliable underlying communication system, i.e. MPI does NOT address what happens if that fails. Two specific topics are order of messages and resource bounds. There was discussion about whether order preservation is required; that is, for messages from one process to another, messages are received in the order they are sent. Maintaining message ordering is troublesome, but seems essential for conveniently writing reliable portable programs. But then comes the question of what exactly this means, particularly with multithreaded processes! What is the effect of probe on the ordering of messages? Straw vote in favor of requiring order preservation: yes: 23 no: 4 On the issue of correctness with regard to resource exhaustion, Marc Snir suggested the following example: Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv What should an implementation be required to do with this program? On the CM-5 this will always deadlock. On Intel and Meiko machines this will "usually" work (but how does one specify exactly when it will work.) Exchange is an even nastier case. ------------------------------------------------------------------------------ Summary of both Wednesday and Thursday point-to-point subgroup meetings by Marc Snir 1. Multithreaded systems and signal handlers. Should these be of concern to us? No vote was taken, but the general feeling was that we should try to define the various communication calls so that they do not rule out the case where the communicating process is multithreaded. The implications seems to be that all calls should be made reentrant, and the communication subsystem is, from the view-point of the application code, stateless. (With one with one obvious exception, namely that due to posted receive or send buffers, and perhaps additional exceptions having to do with global "modes", like error handling mode. 2. Small or large library? No vote taken. The general feeling is that we should provide many options for the advanced programmer that wants to optimize code (otherwise, all "interesting" codes will use non-portable features, but set the syntax so that the use that uses only the "core" functions need not be burdened by the availability of advanced options. 3. What functions? Clearly, SEND and RECEIVE General sentiment that a combined send-receive would be nice ("most used function on CM"), but discussion postponed until we have a proposed definition: Do we we want an exchange (source=dest), or a 3-body function (source != dest), or allow for both? do we want send_buffer identical to receive_buffer, or disjoint from receive_buffer, or allow arbitrary overlap between the two? What attributes are shared by sent message and received message, if at all? WAIT, STATUS and PROBE functions, and persistent handles are discussed later. 4. What modes? We want blocking and nonblocking sends and receives (blocking -- returns when operation terminated; nonblocking -- returns as soon as possible and a different call is needed to terminate the operation). We want synchronous and asynchronous modes (Synchronous -- operation terminated when terminated at all participating nodes. Asynchronous -- operation terminated when terminated at the calling node; e.g. a send terminates asynchronously when the sender buffer can be reused. Please let me know if you dislike this terminology and prefer something like "local" and "global".) The vote went 17-2 toward having a synchronous SEND (completes when RECEIVE has completed, i.e. when the corresponding WAIT has returned, or STATUS has returned successfully.) We did not discuss whether we want all 4 combinations of blocking-nonblocking and synchronous-asynchronous, or just 3 (blocking synchronous, blocking asynchronous and nonblocking asynchronous). We did not discussed explicitly, but "kind of assumed" that any SEND mode can match any RECEIVE mode. 5. How does one complete a nonblocking operation? The SEND and RECEIVE nonblocking operations return a handle that can be used to query for completion. WAIT(handle) blocks until operation completed; STATUS(handle) returns as soon as possible, and returns an indication for successful completion. In addition, these operations return information on completed RECEIVES: tag, message length, etc. for the received message. The information is returned in a structure provided by the caller. After return of a WAIT or successful return of a STATUS the operation handle is freed; the system has no more information on the completed operation, and has freed all associated resources. A more complex WAIT is needed, that waits for the completion of one out of several pending operations. Proposed syntax is WAIT(array_of_handles) that returns information on which operation succeeded and its parameters (voted 17 to 0). No CANCEL operation -- Once a SEND or RECEIVE is posted, it must complete. (Voted 19 to 7. Some peoples asked to reconsider at least canceling posted RECEIVEs, even if posted SENDs must complete). 6. Additional operations "ready-receive" SEND. SEND with a promise that a matching RECEIVE is already posted (A program where such SEND occurs with no preceding matching RECEIVE is erroneous and, hopefully, the implementation detects this error.) The justification is "it exists on some machine" and "it can improve performance by 25% on Delta". Accepted by 13 against 10. Persistent handles. Created by SEND_INIT(params) (resp RECV_INIT(params). can now be repeatedly used to send/receive messages with these parameters, and then explicitly destroyed. Supported by 19 against 2. PROBE. Allows probing for messages available to receive. Justification - "provides a mechanism to allocate memory to a message of unknown length, before it is received". The proposed mechanism is PROBE(params) that returns a lock to a matching message if there is a matching message that can be received. This message is now locked and can only be received using this lock. This was voted 25 to 0. Some level of uncertainty whether we should also allow to unlock without receiving (why should one want to do this?) 7. What is the buffer argument in SENDs and RECEIVEs? A message is a sequence of values, and as a particular case which is of most interest for homogeneous systems, and for which the syntax ought be simpler, a message is a sequence of bytes. There are various ways of specifying this sequence of bytes. a. Contiguous message: Starting address and length b. Regular stride message: Starting address, number blocks, length of blocks, stride. Voted with no opposition. c. IOVEC: a list of entries, each of which describes a type a or type b message. Voted 18 against 1. There was no discussion on a concrete proposal for typed messages, short of agreement that there should be such. The standard is not going to propose a concrete encoding of typed messages, and a concrete mechanism for message exchange in heterogeneous systems. 8. Matching of SENDs and RECEIVEs. A SEND operation associates with a message the following attributes. a. Sender id. b. Tag c. Context The idea of associated a group id, too, was rejected 11 to 10. The RECEIVE criterion is a Boolean predicate on these attributes of the form. (SENDER_ID = param1) and (TAG = param2) and (CONTEXT = param3). Don't cares are allowed for sender_id and tag, but not for context. Sender_id is determined by system, in the obvious manner, and is absolute (not relative to a group or a context). Tag is under sender control. Context is under sender control, but a yet to be determined mechanism is used to allocate valid context values to processes so as to prevent conflicts. All this was approved with no opposition. The idea of allowing the user to provide its own Boolean function as a receive predicate was rejected 26 to 1 (Reason: "hard to do if the matching is done by a communication coprocessor".) 9. Error handling a. We need a version of MPI where errors generate exceptions (user program halts when an error is detected in an MPI call, or a specific exception handling mechanism is invoked). Voted 19 to 1. b. we need to provide a version of MPI where calls return error codes, and do not cause exceptions, whenever possible. Voted 23 to 0. 10. Ordering of messages Messages sent from the same source to the same destination "arrive in the order they where sent". Voted 23 to 0. The exact implications in terms of order in which RECEIVEs can occur has to be worked out. It was pointed out that this condition may be somewhat hard to define in a multithreaded environment. End of Marc Snir's summary --------------------------------------------------------------------------- Collective Communication Subcommittee --------------------------------------------------------------------------- The Collective Communication Subcommittee was called to order by Al Geist at 4:30 p.m. on Wednesday. It continued until 6:40 p.m. when there was a break for dinner. The meeting resumed at 8:25 p.m. and finally adjourned at 10:10 p.m. Al Geist introduced this as the first meeting, since no real discussion on groups and collective communication took place in Minneapolis. One goal of this committee is to maintain consistency with the point-to-point operations. Any discussion of groups necessarily involves this subcommittee. Collective communication operations can be constructed out of the point-to-point primitives, but are desired because they can be implemented efficiently they are convenient for programmers. The committee then went through the set of collective communication primitives that had been proposed by Al Geist during the email discussions. Broadcast: info = MPI_BCAST(buf,bytes,tag,gid,root) On return, contents of buf for root is in buf for all processes. Al Geist pointed out that the group id here is explicit. Root has to be a member of the group. It was at this point that the committee decided that it would use the word "tag" for message type from now on to distinguish it from "type", which will now always mean type of data. Marc Snir pointed out that for consistency with point-to-point operations, there should be both local termination (the operation returns when the local process has done its part) and global termination (the operation returns when all processes have finished their participation) versions. There followed a discussion of the fact that the point-to-point committee seems to be adopting many different versions of send and receive, and that total compatibility will require many different versions of broadcast. There was a discussion of the reason for the tag parameter in the call. It is needed to disentangle multiple broadcasts occurring at approximately the same time. Paul Pierce described how the system can do this by generating sequence numbers. Others argued that the tag was useful for the programmer in any case, particularly for verifying program correctness. Marc Snir argued that there is a problem (because of the intuition that bcast provides barrier) 1 2 3 send(3) bcast rec(don't care) bcast send(3) bcast rec(don't care) Note that 3 may receive from 2 before 1, i.e. no barrier. Al Geist replied that we need barrier, but broadcast is NOT a barrier. James Cownie initiated a general discussion of whether broadcasts could be received by general receives. This would make it simpler to inherit some of the point-to-point semantics. Al Geist said that broadcast should be consistent with the other collective operations, all of which are symmetric. Paul Pierce suggested we specify collective communication routines in terms of model P-P implementation. This has consequences in terms of what options can be supported. Marc Snir pointed out that one can't actually specify collective communication in terms of point-to-point operations because they need dynamically-allocated additional space. It was decided to postpone a straw vote on whether all processes participating in a broadcast should do "broadcast" or only the root should "broadcast" and the others should "receive" because of concern about remaining issues, e.g. different varieties of recieves. The discussion of "error code" was deferred until the issue is settled in the Point-to-point communication subcommittee. MPI_GATHER: (see mail archives for details) It was proposed to have a version in which each participant contributes a different amount of information (a general "concatenate" function). Issues raised: How handle the situation where the number of bytes on each processor is different. How specify the type of data? For example one needs to know the size of the data type for various purposes, e.g. when doing recursive bisection. MPI_GLOBAL_OP: (see archives for definition) This does not include the data types. There was a discussion of how the forwarding processors know where to break buffers if the data type is not specified. Paul Pierce suggested that we should separate the case of user-defined combining operations from the system ones, which could be optimized. Robert Harrison suggested that the buffer be specified as (#items, length) at least for the user-defined operations. (Tag would be retained) Someone noted that "bytes" would be different on each processor in the heterogeneous case. Back to GATHER. Many agreed that the interface should be changed, but no proposal was offered. Straw vote on having separate general concatenation, to go along with the gather operation: yes: 18 no: 0 MPI_SYNCH There was general agreement that "BARRIER" would be a better name. James Cownie suggested that a tag argument would be helpful for debugging. There was also some discussion of failure of such a barrier, e.g. because some node fails. It was agreed that this was not a problem peculiar to this particular function. One individual nonetheless argued strongly for some kind of timeout for the barrier. Groups ------ gid = MPI_MKGRP (list of processes) There was much discussion of the format of the process list. As defined MKGRP defines a group as a subset of a pre-existing group. One alternative would be to allow creating a group consisting of processes from a number of other groups. (NB Identification of processes is unspecified. This is a task for the Point-to-point Communication Subcommittee.) MKGRP provides an implicit barrier among the processes joining the group. There are a number of problems about making sure that gid is uniform and known across the system. This is an efficiency issue. Should it be possible to SEND to a (gid,rank) pair? Marc argued that one should do Point-to-point communication only within a group, not between groups. Note that groups are constant - cannot add or delete members from a group. Also group creation is a barrier for the processes that are part of the group. This raises the question of how the processes joining the group know that they are joining. What is utility of groups? Certainly at present the only commonly used group is ALL. MPI_FREEGRP(gid) MPI_GRPSIZE MPI_MYRANK There was a general discussion of how group id's would be generated. Also a discussion of the mapping information: How to map back from my_rank and gid to rank in ALL? (In order to actually do a SEND.) ----- At this point the group broke for dinner ----- The continuation after dinner was an informal general discussion. There were some general question about experience from Al Geist to Paul Pierce. Adam Greenberg expressed interest in discussing channels. Channels are seen as an early binding, (Curryification) of various of the SEND/RECV functions which offer a number of gains in efficiency. There was a discussion of Fortran language bindings (F77, F90, HPF) of MPI. It was agreed by those knowledgable in the area that there are no special issues in regard to HPF. Steve Wheat discussed the Sandia implementation of channels on the Ncube. Sounds very similar to iWarp channels except that they are dynamic in creation. Jim Cownie noted that global-ops are going to result in non-determinism in numeric routines. Jim also elaborated on Meiko's BAD experience with ready_receive function - lots of user problems. Commonly user's try it on small problems and it works and speeds up. But then on large problems things erratically break and the user bitches. Paul Pierce noted that this is essentially Intel's force type and the Intel experience has not been so bad. In particular it is harder to use and does not generally work easily on small problems. Cownie: In general what to do when a ready_receive fails? No reasonable way to raise error. Response: Use a signal. Cownie: GAACK! This is implementation and not viable on all systems. John Kapenga listed six collective communication issues that he considers particularly important. [Missed the list] Other desirable collective communication features that were mentioned: global-exchange; all-to-all communication. What are criteria for including? Proposal: Difficulty of implementation; frequency of use; efficiency gain John Kapenga asked about 2-D and 3-D mesh operations - e.g. shifts? Adam Greenberg said this should be left to compilers. John: No Way! Adam argued that the compiler can recognize opportunity to avoid memory copies. Unless that same facility is available to user the compiler can do much better. The group adjourned at 10:10 p.m. --------------------------------------------------------------------------- Topologies Subcommittee --------------------------------------------------------------------------- The Topologies Subcommittee was called to order by Rolf Hempel at 4:00 on Wednesday. It lasted until dinner. --------------------------------------------------------------------------- Other Subcommittees --------------------------------------------------------------------------- The other subcommittees (Introduction, Formal Semantics, Environmental Enquiry, Language Binding) met informally after dinner on Wednesday. --------------------------------------------------------------------------- Meeting of the Whole Committee --------------------------------------------------------------------------- Thursday, January 7, 4:30 The Agenda for the rest of the meeting was presented: Introduction subgroup report Collective-communications subgroup report Process Topology subgroup report Environmental Inquiry subgroup report Formal Language subgroup report Language Binding subgroup report Profiling (Jim Cownie) Dates for future meetings Report of the Introduction Subcommittee: ------ -- --- ------------ ------------ Jack Dongarra presented the results of the subcommittee meeting that took place Wednesday night. This is essentially the draft that has been available from netlib for the last six weeks. There was some on-the-fly editing by the group at large. The goal of the Message Passing Interface simply stated is to develop a *de facto* standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. Goals ----- Design an application programming interface (not necessarily for compilers or a system implementation library). Allow efficient communication: Avoid memory to memory copying and allow overlap of computation and communication and offload to communication coprocessor, where available. Allow (but not mandate) extensions for use in heterogeneous environment. Allow convenient C, Fortran 77, Fortran 90, and C++ bindings for interface. Provide a reliable communication interface. User need not cope with communication failures. Such failures are dealt by the underlying communication subsystem. Define an interface that is not too different from current practice, such as PVM, Express, P4, etc. Define an interface that can be quickly implemented on many vendor's platforms, with no significant changes in the underlying communication and system software. The interface should not contain more functions than are really necessary. (Based on the latest count of send/receive variants, this drew a large laugh from the crowd.) Focus on a proposal that can be agreed upon in 6 months. Added: Semantics of the MPI should be programming language independent. Who Should Use This Standard? --- ------ --- ---- --------- This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers of software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. What Platforms Are Targets For Implementation? ---- --------- --- ------- --- --------------- The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those ``machines" consisting of collections of other machines, parallel or not, connected by a communication network. It was agreed that explicit remarks that MPI is intended to be usable with multithreaded processes and with MIMD (not just SPMD) programs should be added somewhere. What Is Included In The Standard? ---- -- -------- -- --- --------- The standard includes: Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication Collective operations Process groups Communication contexts A simple way to create processes for the SPMD model Bindings for both Fortran and C Support for Process Topologies In addition A model implementation and A formal specification. will be provided. It was proposed that explanation and rationale for the standard would also be provided as would sample programs and a validation suite. This is getting very ambitious. Jim Cownie also wants wrappers available for use by, for example, profiling. The suggestion is to provide "name shift", e.g. __MPI_SEND, etc. so the profiler can have MPI_SEND call __MPI_SEND after doing whatever is useful for profiling. What Is Not Included In The Standard? ---- -- --- -------- -- --- --------- The standard does not specify: Explicit shared-memory operations Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages Program construction tools Debugging facilities Tracing facilities Features that are not included can always be offered as extensions by specific implementations. Report of the Collective Communication Subcommittee: ------ -- --- ---------- ------------- ------------ Al Geist summarized the meeting that took place Wednesday afternoon (described above). Global functions beyond those discussed by the subcommittee, such as all2all or total_exchange, await written proposals. The (whole) committee added that Fortran 90 and HPF would be a good place to look for more combining functions (other than max, min, sum, etc.) It was agreed that a way to supply user-supplied functions would be useful. Issues mentioned include: What is a group? How are groups formed? Are group elements addressable, if so how? Are groups ordered (e.g. for prefix/suffix operations)? Group always an ordered subset of the ALL group? Partitioning? Connection with virtual topologies? This will be discussed when topology group reports. Friday, January 8 ------ ------- - Jack Dongarra called the meeting to order at 9:00. Report of the Process Topologies Subcommittee: ------ -- --- ------- ---------- ------------ Rolf Hempel reported on the meeting held Wednesday afternoon: Motivation: Applications have structures of processes Most natural way to address processes Processor topology is valuable to user Creation of subgroups is a natural way to implement topologies Draft proposal for MPI functions in support of process topologies (by Rolf Hempel) is in the handout bundle. The subcommittee made some changes to the draft. What functions should MPI contain? specification of logical process structure lookup functions for process id's clean interface to other parts of MPI (process groups) What should it not contain? any reference to particular hardware architectures algorithms for mapping of processes to processors If it does this, the user program will be portable, but will contain full information for processes mapping at the logical level. Claim: The use of process topologies is not an obstruction to quick implementation of MPI, since the first implementation can make random assignments. A process topology is assigned to a process group. Copying groups can be used to overlay different topologies on the same processes. All processes in a group call the topology definition function. Inquiry functions provide the translation of logical process location to process id. Supported Topologies: General graph structure: For each process, define the complete set of neighbors for each node. In principle this is sufficient as it covers all topologies. But it is not scalable as all processes have knowledge of all others. we should investigate a scalable version. However, important special cases should be treated explicitly, because regular structures can be specified in a scalable way easier to implement the mapping they cover a large number of applications. A special case: Cartesian structures grids/tori hypercube is a special case Support for creation of subgroups for regular structures will be useful. Special treatment for trees? deferred User-defined topology definition functions? deferred It will be necessary for the inquiry functions to provide information on the hardware topology, so that a user can provide his own mapping function. Marc Snir: We need to consider consistency of mapping alignments, for example an octtree for image processing with a grid structure. Al Geist: What is connection between group and topology. Recall that a group is a linear *ordered* array which is a kind of topology. General discussion of copying topologies and groups Proposal is to have at most one topology per group so can use group id as name for topology. This is reason that there must be a group copy. David Walker: We need closer coordination between the collective communication subcommittee and the topology subcommittee, since groups are central to both. Report of the Environmental Enquiry Subcommittee: ------ -- --- ------------- ------- ------------ Bill Gropp reported that the Environmental Enquiry subcommittee needs to wait and get a better picture of what MPI will contain. Jon Flower again asked for cpu_time. This was discussed, and we were reminded that these were more-or-less rejected at the Minneapolis meeting as not being part of MPI. Standardization should come from POSIX. Marc Snir: Part of the subcommittee's job should be to decide *what* can be enquired about as well as how it will be done. There was general discussion about inquiring about both MPI parameters and implementation parameters. Also if parameter *setting* as well as enquiry should be supported. (Buffer pool sizes, for example). Jon Flower also asked about system hints. He suggested it should be possible to tell the system about implementation specific tuning in a system independent way. Report of the Formal Specification Subcommittee: ------ -- --- ------ ------------- ------------ Rusty Lusk reported the committee was without its chairman, Steven Zenith, but that it viewed its mission as to try to formalize what the other subcommittees decide on. It will probably use CSP, for lack of experience with any other formal specification language. Bob Knighten suggested that the subcommittee look into LIS (Language Independent Specification) that POSIX defined in order to separate semantics from language bindings. Report on MPI -1 (minus one) ------ -- ------ ----------- James Cownie presented an MPI anti-specification. Ya hadda be there, but in case you weren't or just want to be reminded, here is a transcription of Jim's slides. MPI -1 (Jim Cownie) In the spirit of LPF (Low Performance Fortran) * Bindings ONLY for Mathematica Occam ML * No function take arguments or returns result * Point to Pointless communication * 1024 different sends NO receives * Full support for 0 dimensional topologies * User data in a message limited to 1 byte (of 6 data types) BUT 1 KByte of TAG, CONTEXT * Informal semantics - Formal Syntax * All groups are contexts * All contexts are groups * Non blocking wait * Non blocking barrier * All user programs are unsafe & erroneous, they therefore do all their work in the exception handler. --------------------------------------------------------------------------- A Profile/Instrumentation subgroup was formed with Jim Cownie as chairman. Steve Otto, as general editor, will contact subgroup chairmen to begin discussion of editing concerns. Discussion of meeting format. The following was proposed as a format for subsequent meetings, based on the experience with this meeting. Wed. afternoon: point-to-point Wed. night: all subcommittees other than pt-to-pt and collective comm. Thurs. morning: collective communication Thurs. afternoon: subcommittee reports Fri. afternoon: subcommittee reports Meeting Dates: It was decided to moved the next two meetings up a week from when they were tentatively scheduled. The next meeting will be Feb 17-19. The next one after that will be Mar 31-Apr 2 The currently-scheduled May 19-21 and June 30-July 2 meetings may also be moved up as well. Note that July 2 will be a holiday in the United States. From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 01:42:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07543; Wed, 3 Feb 93 01:42:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03537; Wed, 3 Feb 93 01:41:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03528; Wed, 3 Feb 93 01:41:05 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA28816; Wed, 3 Feb 93 00:41:03 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA23933; Wed, 3 Feb 93 00:41:00 CST Message-Id: <9302030641.AA23933@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Date: Wed, 03 Feb 93 00:40:59 CST From: Rusty Lusk At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:16:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27255; Wed, 3 Feb 93 06:16:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23277; Wed, 3 Feb 93 06:14:47 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:14:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23269; Wed, 3 Feb 93 06:14:40 -0500 Received: from wrpyr.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21815; Wed, 3 Feb 93 03:14:37 PST Received: by wrpyr.us.oracle.com (5.59.11/37.7) id AA15597; Wed, 3 Feb 93 03:17:14 PST Message-Id: <9302031117.AA15597@wrpyr.us.oracle.com> Date: Wed, 3 Feb 93 03:17:14 PST From: "Chuck Simmons " To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI Reply-To: csimmons@us.oracle.com Original-To: WRPYR:mpi-comm@cs.utk.edu In-Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU's message of 02-03-93 00:40 Rusty, Bill -- I do view the interface as being layered, but I would organize the layers differently. And I have a couple of arguments that I haven't tried out yet against some of the send/recv routines. First, it sounds to me like you are suggesting that the 72 routines be implemented as 72 simple routines that each make a small number of subroutine calls to very simple routines. And then the do_send routine would do a lot. I guess the do_send routine on a typical machine might look like: /* convert the data into a vector of buffers */ if (options & MPI_OPT_ONEBUF) { iovlen = 1; iovp = bufp; } else if (options & MPI_OPT_STRIDED) { /* malloc local buffer, gather data, set up iovlen and iovp */ } /* translate the data */ if (options & MPI_OPT_HETER) { /* malloc new buffer, translate data into new buffer */ } /* send the data non-blocking and unreliable */ if (options & MPI_OPT_RR) { sendrr (...); } else send (...); /* block if so requested */ if (options & MPI_OPT_BLOCK) { block (...); } My view of the layering is: basic functionality: non-blocking vector send non-blocking vector receive easy-to-use basic functionality: non-blocking single buffer send non-blocking single buffer receive reliable: synchronous vector send synchronous vector receive easy and reliable: synchronous single buffer send synchronous single buffer receive In the above, each of the later listed routines can be implemented on top of an earlier listed routine. I would then examine each of the remaining aspects of the functionality: heterogeneity, ready-receiver, strided messages, and blocking non-synchronous messages. For heterogeneity, I would argue that it is wrong to put the datatype translation filter in the communications channel. In your model, it is relatively easy to do things like: bsendh (dest, tag, buf, FLOAT, 1); or even: bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); but what about bsendh (dest, tag, buf, rpc_t, 1); where 'rpc_t' is some user defined data structure? Thus, this model lacks generality. Only certain simple messages can be heterogeneously sent efficiently through the interface. A completely different paradigm must be used if you want to send a structure instead of an array. Further, consider how the "bsendh (dest, tag, buf, ARRAY|FLOAT, nitems);" routine is going to be implemented on a typical machine: buflen = sizeof (net_float) * nitems; sendbuf = (net_float *) malloc (buflen); if (!sendbuf) return ENOMEM; for (i = 0; i < nitems; i++) { sendbuf[i] = float2netfloat (buf[i]); } bsend (dest, tag, (void *)sendbuf, buflen); free (sendbuf); (In the above, we assume that a "net_float" holds the network normalized format of a floating point number.) It would be as efficient to implement the translation routines completely separate from the communications routines. But, wait, you say, our machine is not typical and we can implement this as: bsend (dest, tag, &header, sizeof (header)); for (i = 0; i < nitems; i++) { bsend (dest, tag, float2netfloat (buf[i]), sizeof (net_float)); } where 'bsend' is assumed to be a machine instruction. My argument against this is that the gain in efficiency is sufficiently small that it's not clear its worth the added complexity. In both implementations, the cpu time will be dominated by the data conversion. In the "typical" implementation, there is an additional cost for moving data between two buffers, but that data movement occurs at the speed of local memory access which is likely to be much higher than network bandwidth. Thus, it is much easier for the hardware to DMA the first implementation and run some other process while that DMA is in progress. In the second implementation, the sending process is essentially hogging the cpu for the full time it takes to transmit each piece of the message. Obviously, if there exists a standard for network normal formats of datatypes, and if hardware vendors implement the data translation facilities in hardware, then we could DMA out the whole message heterogeneously. But surely there are other things that we would rather have hardware vendors implementing for us. The above is essentially the same argument that I'm going to use for strided messages. Strided messages are only efficient if you have hardware that can DMA the strided vector, and then its not so efficient that it's obviously worth the effort. For ready-receiver, I'm going to argue that the communications subsystem should always assume the receiver is ready and then handle the case where this isn't true. If the application has set things up so that this is true, that application will run efficiently whether or not the sender specifies that the receiver is ready. If the application isn't being careful, it will run slowly in your model, but in my model, it will usually run quickly since memory is usually availabe in the receiver, and my model will run as slowly as your model when congestion is present at the receiver. Implementing the ready-receiver places strong semantics on the sender and receiver. The sender has to know whether or not the receiver was implemented in a fashion that allows the sender to assume that the receiver is ready. Finally, let's consider synchronous messages versus blocking messages. For most applications, you want reliability. Implementing blocking non-synchronous messages doesn't help you implement this. You can't reuse the buffer being sent until you know that it has been received at the other side. So you might as well either send the message synchronously, or send it non-blocking and go off and do something else while you're waiting to see if you'll need to retransmit the buffer. [Hmmm... I did mention, didn't I, that I was still trying out a couple of these arguments...] This reduces the number of interface routines to 8. I could probably be convinced that adding 4 more interface routines to support strided messages doesn't overly complicate the interface. -- Chuck *** We use Oracle*Mail running on Oracle V6.2 and an nCUBE2 supercomputer. *** ---- Included Message ---- Received: 02-02-93 22:48 Sent: 02-03-93 00:40 From: WRPYR:owner-mpi-comm@CS.UTK.EDU To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:35:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01865; Wed, 3 Feb 93 06:35:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23802; Wed, 3 Feb 93 06:35:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:34:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23794; Wed, 3 Feb 93 06:34:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13771 (5.65c/IDA-1.4.4 for ); Wed, 3 Feb 1993 06:34:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26850; Wed, 3 Feb 93 11:34:48 GMT Date: Wed, 3 Feb 93 11:34:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302031134.AA26850@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03425; Wed, 3 Feb 93 11:33:11 GMT To: lusk@antares.mcs.anl.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Wed, 03 Feb 93 00:40:59 CST <9302030641.AA23933@donner.mcs.anl.gov> Subject: A suggestion for a multi-level MPI Content-Length: 581 The fundamental approach seems fine to me (though Chuck makes some good points...) However there's at least one point to note :- 1) The Fortran binding issue suggests that in the heterogeneous cases there should be a routine for each data type, rather than having the data type as an argument. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 07:27:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11163; Wed, 3 Feb 93 07:27:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25571; Wed, 3 Feb 93 07:26:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 07:26:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ifi.uio.no by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25557; Wed, 3 Feb 93 07:26:08 -0500 Received: from byflak.ifi.uio.no by ifi.uio.no with SMTP id ; Wed, 3 Feb 1993 13:26:05 +0100 From: "\ystein Gran Larsen" Received: by byflak.ifi.uio.no ; Wed, 3 Feb 93 13:26:04 +0100 Date: Wed, 3 Feb 93 13:26:04 +0100 Message-Id: <9302031226.AA26562@byflak.ifi.uio.no> To: mpi-comm@cs.utk.edu Subject: data types and heterogeneity Hi, I have read the minutes from the MPI meeting, January 6-8 1993. The section on message data types mentiones the use of XDR as a protocol for converting types between languages and machines. The standardization of Scalable Coherent Interface (SCI) has lead to a proposed standard for "Shared-Data Formats Optimized for Scalable Coherent Interface Processors", P1596.5. [Working group P1596.5 is chaired by David V. James (dvj@apple.com)] This proposed standard may be of relevance to the MPI-initiative, when it comes to supporting heterogeneous machines. -{\O}ystein From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 11:56:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17274; Wed, 3 Feb 93 11:56:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07365; Wed, 3 Feb 93 11:54:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 11:54:55 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07357; Wed, 3 Feb 93 11:54:53 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01169; Wed, 3 Feb 93 10:54:51 CST Date: Wed, 3 Feb 93 10:54:51 CST From: Tony Skjellum Message-Id: <9302031654.AA01169@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, gran@ifi.uio.no Subject: Re: data types and heterogeneity XDR as currently implemented causes one function call per datum. This feature must be changed (loops moved inside fundamental converter routines) to provide for better performance (less bandwidth impact of converting data types). I am eager to see the report mentioned by Oystein. - Tony From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 13:33:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19846; Wed, 3 Feb 93 13:33:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12323; Wed, 3 Feb 93 13:31:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 13:31:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12315; Wed, 3 Feb 93 13:31:56 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Wed, 3 Feb 93 10:31 PST Message-Id: Date: Wed, 3 Feb 93 10:31 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: Mix + Match, NOT At first look, I definitely like the proposal of Lusk and Gropp. It seems to be the right way to set up a hierarchy of the 1024+ routines that we are inventing. One thing that bothers me: at the last meeting it was said by several people that one would be allowed to mix+match different kinds of sends and recvs to each other. That is, I could do a "cnsend()" and have that consumed by a "sbrecvhrr()". I think we are asking for lots of trouble if the standard states that this is allowed. If it is allowed, an implementor has to worry about 72 x 72 = 5184 different types of behavior for pt-pt communication. I don't see a strong reason for allowing mix+match, and on the implementation side it seems to me that there are many reasons for wanting the restriction that a send of flavor 53 (out of a possible 72) should be matched by a recv of the same flavor, 53. It is true that it seems like many flavors could be mixed, but consider the synchronization choices. If we allow mixing, then the implementor of the sbrecv() routine has to insure that "sensible" behavior results no matter what send routine was used. As I understand it, this means the implementation cannot be done in an orthogonal manner. I'm not even sure we could define "sensible" for all 5184 combinations! Comments? Am I missing something? --Steve Otto From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 18:43:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29539; Wed, 3 Feb 93 18:43:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27997; Wed, 3 Feb 93 18:41:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 18:41:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27989; Wed, 3 Feb 93 18:41:32 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA14014; Wed, 3 Feb 93 18:35:11 EST Date: Wed, 3 Feb 93 18:35:11 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302032335.AA14014@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: A suggestion for a multi-level MPI James Cownie remarks on needing a seperate send routine for each data type in the Fortran Binding. For both the C and Fortran versions there could be seperate calls setting up the Data Addresses (no Data need be moved) and then a single send could take the value returned by the Data Address Setup Call. No system call required on the Data Setup Call. This might require that the array passed to the Data Setup Call not be modified before the send call is made, to avoid recopying it. This also allows later extensions to other Data Addressing modes, without changing the send/recieve syntax. This reduces the number of calls by almost a factor of three, uncouples the data location from the send/recieve syntax and allows arguement types on all calls to be correct and fixed. It does make a bit more processing (trival) and the need for the user to mamage the Data Setup return value and not modify the address array (two of its bad points). john From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 23:45:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02591; Wed, 3 Feb 93 23:45:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09607; Wed, 3 Feb 93 23:43:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 23:43:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09599; Wed, 3 Feb 93 23:43:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00850; Wed, 3 Feb 93 22:43:44 CST Date: Wed, 3 Feb 93 22:43:44 CST From: Tony Skjellum Message-Id: <9302040443.AA00850@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: Anonymous ftp site aurora.cs.msstate.edu A paper of relevance to the MPI project, describing "Zipcode 1.0" is available by anonymous FTP, in the pub/reports subdirectory: zipcode_sep92.ps.Z The site is aurora.cs.msstate.edu. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 00:02:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02797; Thu, 4 Feb 93 00:02:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10374; Thu, 4 Feb 93 00:01:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 00:01:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10364; Thu, 4 Feb 93 00:01:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA08517; Wed, 3 Feb 93 23:01:36 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA26443; Wed, 3 Feb 93 23:01:34 CST Message-Id: <9302040501.AA26443@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels Date: Wed, 03 Feb 93 23:01:33 CST From: Rusty Lusk Chuck -- Thanks for your comments. They are of two kinds. The first kind are about the suggested multiple-level organization of the library. There I think I didn't explain some things well enough, and will try to do better here. The other kind are about what should be in or out of the library altogether, and there I can only try to summarize what people talked about in Dallas; the things we incorporated are just the things that the straw votes indicated people wanted in the standard. | | I do view the interface as being layered, but I would organize the layers | differently. And I have a couple of arguments that I haven't tried out | yet against some of the send/recv routines. | | First, it sounds to me like you are suggesting that the 72 routines | be implemented as 72 simple routines that each make a small number of | subroutine calls to very simple routines. This is not at all what we intended to suggest. We used the word "levels" rather than "layers". The upper levels could be *defined* in terms of the lower ones (That's what makes them upper-level) but the intention is that they would *not* be implemented as calls to the lower-level routines. In fact what we called level 2 (large number of routines) exists partly to avoid the multiple subroutine calls necessary in level 1 (where there are a small number of routines.) In fact, level 2 routines don't need to use the data structures implied by the get_handle-mod_handle-use_handle-free_handle at all. They just have to do what they say they do, as efficiently as possible. | | For heterogeneity, I would argue that it is wrong to put the datatype | translation filter in the communications channel. | Our discussions seemed to indicate that people want it there. The popularity of systems like PVM, p4, Express, and others that handle this for the user attests to this. I think the reason is portability; people like being able to move code unchanged from a heterogeneous network to a multicomputer and back. The system can (relatively easily) handle all the translations, including figuring out whether it is necessary at all or not. Many users hardly even know about the non-portability of data formats; they think (in my opinion rightly) that the communication system should handle that if they want it to. | In your model, it | is relatively easy to do things like: | | bsendh (dest, tag, buf, FLOAT, 1); | or even: | bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); | but what about | bsendh (dest, tag, buf, rpc_t, 1); | where 'rpc_t' is some user defined data structure? Thus, this model | lacks generality. Not quite true, although not entirely false. In the "draft implementation" of mpi that you can get from here, we use an iovec-like structure which is a vector of (addr,datatype,nitems) triples to describe a scattered, mixed type buffer. An arbitrary structure can be sent this way, provided the user builds the structure descriptor (It's too bad compilers don't do this for us.) | | It would be as efficient to implement the translation routines completely | separate from the communications routines. | .... | | The above is essentially the same argument that I'm going to use for | strided messages. Strided messages are only efficient if you have | hardware that can DMA the strided vector, and then its not so efficient | that it's obviously worth the effort. Machines do have this kind of hardware, and we can expect to see more of it. One thing that I will continue to argue in favor of is giving the user access to as much of the speed the hardware can deliver as possible. | | For ready-receiver, I'm going to argue that the communications subsystem | should always assume the receiver is ready and then handle the case where | this isn't true. | Some existing machines can perform *significantly* faster if the user can relieve their system of some buffer-management chores. If we don't give the power users access to this efficiency, they simply won't use MPI. And it will come to pass that everyone will agree that to get real efficiency, you have to use the vendor's unique routines. Or you have to use "MPI with the XXX extension package". Portability will be destroyed. This is a way the MPI effort could fail. | | Implementing the ready-receiver places strong semantics on the sender | and receiver. The sender has to know whether or not the receiver was | implemented in a fashion that allows the sender to assume that the receiver | is ready. | Agreed. The naive user can use what we call the Level 3 or Level 4 routines, and never read the chapter in the manual that describes "ready-receiver" semantics and requirements. The power user can use them to write *portable* fast code. | | Finally, let's consider synchronous messages versus blocking messages. For | most applications, you want reliability. Implementing blocking | non-synchronous messages doesn't help you implement this. You can't reuse | the buffer being sent until you know that it has been received at the other | side. So you might as well either send the message synchronously, or send | it non-blocking and go off and do something else while you're waiting to see | if you'll need to retransmit the buffer. | Most people want to compute while the message is in transit, and even while the buffer is being emptied. They don't want this capability taken from them. | | This reduces the number of interface routines to 8. I could probably | be convinced that adding 4 more interface routines to support strided | messages doesn't overly complicate the interface. | The Level 4 routines are only 2. More routines are needed only for more efficiency and functionality. In the last few weeks I have talked to lots of people about the MPI effort, and they are mainly afraid that we will leave something important out. If we don't supply all the functionality and efficiency that we possibly can, MPI will fail. The problem is to do so without making the package too complicated. Our posting about multiple levels was merely an attempt to demonstrate that it could be done. Regards, Rusty Lusk & Bill Gropp From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 01:30:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04177; Thu, 4 Feb 93 01:30:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13427; Thu, 4 Feb 93 01:29:10 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 01:29:09 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13419; Thu, 4 Feb 93 01:29:08 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12505; Thu, 4 Feb 93 01:29:05 -0500 Date: Thu, 4 Feb 93 01:29:05 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040629.AA12505@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Steve Otto proposes that the 72 (or 648 if each data type is separate) sends be matched to the same type of recv. While I agree with Steve that we will not be able to define much less test the 5184 combinations if we allow otherwise, There are obvious cases where the user would want to mismatch send/recv For example, she may have to do a blocking send, but can get away with a non-blocking recv. Not all combinations make sense, but we can't restrict the user to no mismatches. Al Geist From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:10:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05537; Thu, 4 Feb 93 02:10:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14590; Thu, 4 Feb 93 02:08:44 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:08:43 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14582; Thu, 4 Feb 93 02:08:41 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12583; Thu, 4 Feb 93 02:08:39 -0500 Date: Thu, 4 Feb 93 02:08:39 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040708.AA12583@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. >Most people want to compute even while the buffer is being emptied... Actually most people don't want to think or know about how messages are sent they just want it reliable and fast. Only high priests are concerned with the pain of non-blocking send. >If we don't supply all the functionality that we possibly can, >MPI will fail. The problem is to do so without making the package >too complicated. The complication I am seeing in MPI will also make it fail. 72 sends?? Appears to violate several of our goals, like similar to existing packages. small set of routines. >The naive user never has to read the chapter in the manual... I hope the user doesn't have to make room on his shelves for volumes of the MPI manual. Al From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:41:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05708; Thu, 4 Feb 93 02:41:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15558; Thu, 4 Feb 93 02:40:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:40:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15544; Thu, 4 Feb 93 02:40:30 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA08675 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 08:39:03 +0100 Received: by f1neuman.gmd.de id AA14246; Thu, 4 Feb 1993 08:38:42 +0100 Date: Thu, 4 Feb 1993 08:38:42 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9302040738.AA14246@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Cc: gmap10@f1neuman.gmd.de I was glad to read Steve's proposal not to allow mismatches between different types of sends and receives. If we don't follow that line, we will most likely cause many problems to the implementors of MPI which we cannot forsee at the moment. Also, the requirement of matching sends and receives could be used to check the correct behaviour of the user program in some debugging mode. Al's counter-example of course makes sense, but if you have to use a blocking send, you can get the same by using the non-blocking version followed directly by the wait-for-completion. I'm looking forward to seeing an example where you really need the mismatch. Rolf From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 03:16:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05912; Thu, 4 Feb 93 03:16:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17215; Thu, 4 Feb 93 03:14:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 03:14:53 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17205; Thu, 4 Feb 93 03:14:51 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA15627; Thu, 4 Feb 93 03:08:28 EST Date: Thu, 4 Feb 93 03:08:28 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302040808.AA15627@cs.wmich.edu> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Rolf asks -- > ... > followed directly by the wait-for-completion. I'm looking forward to > seeing an example where you really need the mismatch. > > Rolf By using the wait_for_completion you never need a blocking send/receive operation. The code looks a bit ugly. Right now we have to have a "mismatch" because we don't have a synchronized send/receive pair ;-). Actually I thought the reason that the synchronized send/recieve pair was there initially was to help insure correctness in matching calls, as well as a possible simpler implementation. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 07:03:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07047; Thu, 4 Feb 93 07:03:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01249; Thu, 4 Feb 93 07:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 07:02:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01241; Thu, 4 Feb 93 07:02:28 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA09794 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 07:02:22 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA02743; Thu, 4 Feb 93 12:02:17 GMT Date: Thu, 4 Feb 93 12:02:17 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041202.AA02743@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03570; Thu, 4 Feb 93 12:00:32 GMT To: otto@iliamna.cse.ogi.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Steve Otto's message of Wed, 3 Feb 93 10:31 PST Subject: ~ (Mix + Match, NOT) Content-Length: 2480 The only place where I can see a need for matching of send and receive formats is in the heterogeneous case. Here I would be happy to insist that a heterogeneous send be matched to a heterogeneous receive. Since I don't want to enforce in MPI the implemntation decision of whether the data type conversion occurs at sender or receiver I need the data type information in both places. I can see NO reason to restrict any other combinations. Discussion ---------- There are two major areas which cause the many sends 1) layout of data in store. (contiguous, strided, by iovec) 2) synchronisation behaviour (return asap, return when buffer free, return when data received at other end) Data layout ----------- Data layout is an issue which is entirely local. Why does it matter to the receiver how the sender chose to layout her buffer (or the converse) ? This information hiding is one of the things which is an advantage of message passing. The data is necessarily serialised to some degree over the communications medium, so where's the problem ? I believe it is actually HARDER to insist that the buffer sepcifications match at each end than not to do this. (Certainly I have to pass additional information with the message if I'm to check this.) Synchronisation behaviour ------------------------- Once again this seems to me to be largely to do with local issues of buffer management. (Though of course my algorithm may need certain synchronisation behaviour, in which case I must code it using the correct forms of send and recv at both ends.) However I can see no reason to INSIST that the same style of send and receive is used. Indeed there are strong reasons NOT to do so. Consider (for example) a server with multiple clients. I'd certainly like to write it like this :-- Set up lots of non-blocking receives while (running) { wait for a recv service the request get a reply buffer (may need to wait for one) queue a reply requeue the receive buffer } However the clients can simply make requests thus blocking send request blocking receive reply Here is an immediate mismatch in the synchronisations. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:44:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11973; Thu, 4 Feb 93 09:44:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07037; Thu, 4 Feb 93 09:42:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:42:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07029; Thu, 4 Feb 93 09:42:30 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12554; Thu, 4 Feb 93 07:42:27 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK7mc-0016ZKC; Thu, 4 Feb 93 07:42 MST Message-Id: Date: Thu, 4 Feb 93 07:42 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... # Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. # >Most people want to compute even while the buffer is being emptied... # Actually most people don't want to think or know about how messages # are sent they just want it reliable and fast. # Only high priests are concerned with the pain of non-blocking send. Not so; we have those that are not only concerned about being able to compute while messages are in transit, but also wanting to assure that minimal system space is required. That is, they want to manage memory much more tightly than a "buffered" system allows. We have a lot of applications types that find it very hard to fit an application into memory and still allocate sufficient message buffers. By the way, paging is not a viable solution :-). Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:59:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12166; Thu, 4 Feb 93 09:59:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07696; Thu, 4 Feb 93 09:58:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:58:23 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07688; Thu, 4 Feb 93 09:58:22 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12746; Thu, 4 Feb 93 07:58:18 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK81y-0016ZKC; Thu, 4 Feb 93 07:58 MST Message-Id: Date: Thu, 4 Feb 93 07:58 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: john@cs.wmich.edu, mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... as an implementor of blocked and non-blocked messaging systems, I have not experienced any "system" problems associated with matching mixed send/receives. While worrying about the "potential" implementation difficulties, I fear that we make use of the system so difficult that it won't be used (so would that make it easy to use?). Furthermore, as for the theme that seems to prevail concerning features that only "wizards" would use, if we do not provide wizardly type features, then the wizards will not use MPI. Then to whom will the "novice" types refer when they finally see the light of low-performance codes running in a high-performance compute environment? Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:28:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12742; Thu, 4 Feb 93 10:28:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09131; Thu, 4 Feb 93 10:26:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:26:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09123; Thu, 4 Feb 93 10:25:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01127; Thu, 4 Feb 93 09:25:56 CST Date: Thu, 4 Feb 93 09:25:56 CST From: Tony Skjellum Message-Id: <9302041525.AA01127@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu Subject: Re: 5184 bottles of sends on the wall... Please remember that overlapped send/receive will only make sense on systems with excess memory bandwidth. Hence, it might make more sense to implement a system where this is permitted by the implementor, but not required. Otherwise, calculations might slow down instead of achieving pipelined improvements. As Al points out, it is only the heavy-duty guys who try this sort of stuff... - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:30:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12830; Thu, 4 Feb 93 10:30:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09278; Thu, 4 Feb 93 10:28:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:28:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09270; Thu, 4 Feb 93 10:28:53 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA16295; Thu, 4 Feb 93 10:22:30 EST Date: Thu, 4 Feb 93 10:22:30 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302041522.AA16295@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: 5184 bottles of sends on the wall... My initial concept of mixing IO types was that blocking and non-blocking could be freely mixed, but that synchronized was special and a synchronized send would only match a synchronized receive. The rationale for dropping the synchronized recieve is OK with me. I have no problems with the current proposal. Overlappping communication and computation is the only way to get good prerformance in many applications. It must be assumed to be the norm rather than the exception. I would not be able to use a system that did not support it. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:46:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13213; Thu, 4 Feb 93 10:46:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10063; Thu, 4 Feb 93 10:45:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:45:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10042; Thu, 4 Feb 93 10:45:06 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA12579 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 10:45:01 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA06100; Thu, 4 Feb 93 15:44:56 GMT Date: Thu, 4 Feb 93 15:44:56 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041544.AA06100@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03785; Thu, 4 Feb 93 15:43:15 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-comm@cs.utk.edu, john@cs.wmich.edu In-Reply-To: Tony Skjellum's message of Thu, 4 Feb 93 09:25:56 CST <9302041525.AA01127@Aurora.CS.MsState.Edu> Subject: 5184 bottles of sends on the wall... Content-Length: 1025 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. True, and all the more reason to avoid as many copies as possible. After all if taking it from the user buffer consumes bandwidth 1, then the bandwidth cost of a send which needed a copy will be 2+1 = 3. This can be significant. (Of course I'm assuming a "fair" memory system here, in which taking from one port only loses the same from another. I've seen systems where this wasn't true, and this can change the balance.) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:33:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19567; Thu, 4 Feb 93 15:33:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25767; Thu, 4 Feb 93 15:30:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:30:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25752; Thu, 4 Feb 93 15:30:34 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04724; Thu, 4 Feb 93 20:30:30 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05190; Thu, 4 Feb 93 13:29:24 MST Date: Thu, 4 Feb 93 13:29:24 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042029.AA05190@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI I especially like "Level 1" of Rusty and Bill's proposal. I would like to see more detail in this section, like examples of mod_xxx() and do_xxx() routines. I think there will be a good deal of support for Level 1 with all the talk that's been going on about handles/invoices/envelopes (of course there's that annoying syntax thing...). I am concerned about how heterogeneous communications are handled (or avoided) in Levels 3 and 4. I would be a lot more comfortable if ALL send() and recv() routines were heterogeneous. Here's a slight modification to the original proposal: Level 4 ("novice users"): send(dest,tag,bufadd,datatype,numitems) recv(dest,tag,bufadd,datatype,numitems) Level 3 ("dangerous users" [like me :) ]): bsend(tag,dest,bufadd,datatype,numitems) nsend(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,datatype,numitems) nrecv(tag,dest,bufadd,datatype,numitems) or [n][send] [b][recv] in Bill and Rusty's shorthand. Level 2 ("really frightening users"): [c][n][send][ ] [s][b][recv][rr] [g][s] Here's the arguments: - I don't like the idea of having to tell novice users that they must change all the MPI communication routine calls in their code before their program will run on a heterogeneous system. I think a lot of novice users will be using heterogeneous workstation networks. The current proposal forces these folks to learn at least two versions of send() and recv(). - I don't think there are any significant performance reasons for having separate heterogeneous versions of these calls, as long as we require a receiving process to know what incoming message contents will be (which seems to be consistent with the original proposal). - I don't think that the additional "datatype" argument reduces "ease of use" very much. An incorrect datatype should be no more difficult to debug than an incorrect "bufadd" or "buflen" (and should be easier than a bad "tag" or "dest", especially if deadlock results!). In FORTRAN77, "numitems" should be easier to code portably than "buflen". Even in C, there is little difference in complexity between portable calls to heterogeneous and non-heterogeneous versions: (non-heterogeneous) float mybuf[NUMITEMS]; ... send(dest, tag, mybuf, NUMITEMS * sizeof(float)); (heterogeneous) send(dest, tag, mybuf, MPI_FLOAT, NUMITEMS); - This gets rid of a whole bunch of send() and recv() variant routines (at least 22). - If we really want non-heterogeneous communication, it could be stuck in Level 1 with all the other stuff that's really dangerous for "novices". Another option would be to allow a MPI_BYTE datatype (not sure how this would work in FORTRAN). This may be useful for library developers. I think it's a good idea to keep "raw bytes" away from novice users (especially in FORTRAN). Tom Henderson hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:49:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20005; Thu, 4 Feb 93 15:49:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27824; Thu, 4 Feb 93 15:48:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:47:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27816; Thu, 4 Feb 93 15:47:57 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01577; Thu, 4 Feb 93 14:47:56 CST Date: Thu, 4 Feb 93 14:47:56 CST From: Tony Skjellum Message-Id: <9302042047.AA01577@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, hender@macaw.fsl.noaa.gov Subject: Re: A suggestion for a multi-level MPI Tom, It is an artificial requirement to restrict heterogeneous messages to a single data type. - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:53:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20074; Thu, 4 Feb 93 15:53:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28380; Thu, 4 Feb 93 15:52:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:52:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28367; Thu, 4 Feb 93 15:52:36 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04817; Thu, 4 Feb 93 20:52:32 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05241; Thu, 4 Feb 93 13:51:27 MST Date: Thu, 4 Feb 93 13:51:27 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042051.AA05241@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI > Tom, > > It is an artificial requirement to restrict heterogeneous messages to a single > data type. > - Tony > I agree. My interpretation of Rusty and Bill's proposal is that messages that contain multiple data types would be constructed using the Level 1 routines-- something similar to Zipcode's invoices. Tom From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:02:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21865; Thu, 4 Feb 93 17:02:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03500; Thu, 4 Feb 93 17:01:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:01:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03492; Thu, 4 Feb 93 17:01:20 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Thu, 4 Feb 93 14:01 PST Message-Id: Date: Thu, 4 Feb 93 14:01 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: 5184; one small point I got into my office late today, and I must say, how exciting to see all the mail concerning 5184! Jim Cownie writes: >The data is necessarily serialised to some degree over the >communications medium, so where's the problem ? I believe it is >actually HARDER to insist that the buffer sepcifications match at each >end than not to do this. (Certainly I have to pass additional >information with the message if I'm to check this.) I did not mean to imply that the standard should RULE OUT mix+match or that it should check for it at run time, merely that the standard should not make the guarantee that one IS allowed to do this. There is an important difference. I really don't believe we can foresee everything, so we should be conservative on what the standard guarantees. Jim agrees that in the case of heterogeneity, it does seem that we want matching: >The only place where I can see a need for matching of send and receive >formats is in the heterogeneous case. Here I would be happy to insist >that a heterogeneous send be matched to a heterogeneous receive. So maybe we don't want to allow all 5184 combinations. OK, at least we are admitting that there is an issue here. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:52:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23307; Thu, 4 Feb 93 17:52:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06478; Thu, 4 Feb 93 17:50:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:50:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06466; Thu, 4 Feb 93 17:50:53 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA17493; Thu, 4 Feb 93 14:50:39 PST Message-Id: <9302042250.AA17493@SSD.intel.com> To: otto@iliamna.cse.ogi.edu (Steve Otto) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Thu, 04 Feb 93 14:50:38 -0800 From: prp@SSD.intel.com > >The data is necessarily serialised to some degree over the > >communications medium, so where's the problem ? I believe it is > >actually HARDER to insist that the buffer sepcifications match at each > >end than not to do this. (Certainly I have to pass additional > >information with the message if I'm to check this.) > > I did not mean to imply that the standard should RULE OUT > mix+match or that it should check for it at run time, > merely that the standard should not make the > guarantee that one IS allowed to do this. There is an > important difference. I really don't believe we can foresee > everything, so we should be conservative on what the standard > guarantees. > > --Steve Otto > One of the important differences is that if the standard does not guarantee that you can mix, you can't write a portable program that mixes. Since we have seen one or two good examples of useful mixing, I think its important to guarantee the ability to mix at least for some combinations. My preference would be to guarantee mixing in all but specific cases, as few as possible. In fact, I would recommend against including features that cannot be mixed. I am optimistic that we can take care of the heterogeneous case in a sufficiently uniform way that mixing heterogeneous levels with non-heterogeneous becomes a non-issue. There was a good argument for not having a non-heterogeneous level, that would take care of it. The one case where mixing probably can't be justified is synchronous operation, where the send does not complete until the receive does. I've never liked this option, and worry that some support comes from misunderstanding about the difference between blocking and synchronous, as evidenced by Chuck's comment "You can't reuse the buffer being sent until you know that it has been received at the other side". This is of course not true. If the blocking send semantics are that 1) the send completes when the send buffer can be written safely and 2) message delivery is guaranteed, then you may happily overwrite the send buffer when the send completes with the assurance that the undamaged message will eventually appear at the receiver if its not already there. The only interesting difference between blocking send and synchronous send is in the amount of system buffering that may be required by applications. A correct application using blocking send might require some amount of system buffering, while a correct application that exclusively uses synchronous send requires no system buffering. In my opinion, we should focus on the problem of implied system buffering in the system environment area and eliminate synchronous send from the point-to-point portion of the standard. With reasonable care, we can make a clear and simple guarantee that appropriate different sends and receives can be mixed freely. This will allow people to write portable applications which use mixed structures. Paul From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:56:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23412; Thu, 4 Feb 93 17:56:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:06:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23575; Thu, 4 Feb 93 18:06:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23912; Thu, 4 Feb 93 18:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 19:10:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24655; Thu, 4 Feb 93 19:10:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11923; Thu, 4 Feb 93 19:09:15 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from almaden.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11915; Thu, 4 Feb 93 19:09:11 -0500 Message-Id: <9302050009.AA11915@CS.UTK.EDU> Received: from almaden.ibm.com by almaden.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9078; Thu, 04 Feb 93 16:09:16 PST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 20:10:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25133; Thu, 4 Feb 93 20:10:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14638; Thu, 4 Feb 93 20:09:30 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 20:09:29 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14630; Thu, 4 Feb 93 20:09:28 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00403; Thu, 4 Feb 93 19:09:20 CST Date: Thu, 4 Feb 93 19:09:20 CST From: Tony Skjellum Message-Id: <9302050109.AA00403@Aurora.CS.MsState.Edu> To: ho@almaden.ibm.com Subject: re: my comments on overlapping Cc: mpi-comm@cs.utk.edu BTW, I do not mean blocking send / blocking receive only in my comments. I mean asynchronous reading of a specific message (aka intel irecv) going on at the same time as processing. The message appears in the buffer, and one can poll for that to be complete. In practice, message passing does happen in background on all these systems, but the algorithms are not explicitly using this feature, or encouraging this overlap, necessarily. They might process messages at high priority, and only compute when message input queues are empty. Lennart has always repeated this warning that the overlap is only useful from a performance sense, and therefore from the point of view of a programmer trying to ask for this, if the system can perform better with the overlap. I DO NOT ADVOCATE block send / block receive only. Best wishes, Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:18:36 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: Content-Length: 831 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:05:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14845; Fri, 5 Feb 93 18:05:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03769; Fri, 5 Feb 93 18:03:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:03:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03760; Fri, 5 Feb 93 18:03:04 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA06572; Fri, 5 Feb 93 15:02:57 PST Message-Id: <9302052302.AA06572@SSD.intel.com> To: jwf@lion.parasoft.com (Jon Flower) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Fri, 05 Feb 93 15:02:57 -0800 From: prp@SSD.intel.com Folks, here is an offline exchange Jon and I had that you might enjoy. To: prp@SSD.intel.com Subject: Re: 5184; one small point Paul, I saw your message about blocking and synchronous messages. Without wanting to speak for Chuck I think there is a pretty important difference between the two, particularly for applications such as Oracle. Using the blocking send it is true that the send buffer can be reused as soon as it is "clear" AS LONG AS IT IS GUARANTEED that it will be delivered. Unfortuntely this can't be guaranteed in the case where system buffers are required because the system may simply be out of memory at the receiver. In this case MPI and other systems will simply drop the offending message on the ground. For a database I assume this would be fatal. On the other hand the synchronous scheme has the advantage that you know it's worked when it returns, on both send and receive sides. Jon Flower This is an area where we must be clear about the semantics. We have agreed to some extent that message passing in MPI will be reliable. If I remember correctly, this passed on a straw vote early in the first point-to-point discussion. When I say "agreed to some extent" I mean that not everyone has the same understanding of what the standard is about on this issue. To me it means that if I have a send, and a receive that matches the message sent, that the message will absolutely be delivered. This is how NX works. If there isn't enough buffer space, the send must block so that the message waits in the send buffer until it can be delivered. Flow control is required in the underlying protocol. There seem to be a lot of people that expect no underlying protocol. When you say "send", the system sends - whether the message can be delivered or not. Apparently some systems work this way, and therefore it is expected behavior for people who have been users of these systems. To me, it is not at all compatible with the notion of reliable message passing. If you have reliable message passing as I understand it, clearly the important difference you mention goes away, leaving the lesser difference I wrote about. Without reliable message passing, I see no point whatsoever in blocking send. In fact, I see little point in having a typed send/receive standard, because such an interface is loaded up with friendly help for the user and it seems ridiculous to turn around and expect that same user to worry about reliable delivery. Does any of this make sense? Paul Hi Paul. I think what you're saying makes some sense. I have two more questions though.... a) Having the sender block until memory is available potentially invalidates some algorithms, particularly as the number of nodes grows. It's quite easy to come up with a sort of "ring" algorithm in which everyone ends up hung. I would agree that the trivial case (a single one-dimensional ring) in which this happens is a "user error" but there are more complex multi-dimensional cases in which it is quite hard for users to guarantee that it won't come up, even with quite reasonable programs. b) What are the implications of the flow control that you are using on performance? Is your routing hardware doing the claiming of memory blocks on the receiver or is there an initial software handshake to get started. The latter probably costs a fair amount that the "high priests" don't like??? I agree that the definition of "reliable devliery" in the MPI picture is not solid. I thought I remembered a piece that actually said that it was OK to drop a message if there wasn't enough memory on the receiver but I could be hallucinating. I wonder if it's worth trying to straighten this out? I have a nasty feeling that we'll get down to implementation details. If we can't get IBM and TMC to agree that Node 0: send to 1 read from 1 Node 1: send to 0 read from 0 is a reasonable thing to do I don't see much hope for anything else at this level....... Jon > a) Having the sender block until memory is available... This is inherently a hard problem that is all about system buffering. You can't solve this problem by careful or clever definition of the semantics of a message passing interface (whether and when sends block, and whether message passing is reliable.) If you push it down here, it pops up over there - depending on the semantics defined for the interface, the problem must be solved by the system or by the user. My preference (for a typed send/receive interface) is to push the problem to the system side, as much as possible. That way most users won't have to deal with it. But you can't completely solve the problem on the system side, because the user can always write a program that will confound any system buffering that has finite buffer space. With a reasonably good system implementation, it is actually rather difficult to come up with an algorithm that breaks it. Also, it is possible to define straightforward rules that will guarantee correct execution. That way, although the user must be careful being careful is easy. > b) What are the implications of the flow control... There is a performance penalty. People have claimed that NX overheads are too high, and it is true that they are noticeably higher than raw hardware. But there is quite a bit of overhead from message matching too, so for those who want ultimate performance there should be an alternative to the typed send/receive interface. Active messages is a good candidate. Such an interface is outside the scope of MPI. > I agree that the definition of "reliable devliery" in the MPI > picture is not solid. I thought I remembered a piece that actually > said that it was OK to drop a message if there wasn't enough > memory on the receiver but I could be hallucinating. I wonder It was in the first draft. > if it's worth trying to straighten this out? I have a nasty > feeling that we'll get down to implementation details. If we > can't get IBM and TMC to agree that > > Node 0: > send to 1 > read from 1 > > Node 1: > send to 0 > read from 0 > > is a reasonable thing to do I don't see much hope for anything > else at this level....... We have to develop and agree on a way to manage system buffering. To get Intel, IBM, and TMC together it must allow for the case where the above works and the case where it doesn't. > > Jon Paul Do you have any plan for how to reconcile the difference between the three camps on the simple program I sent you? My impression is that: NX (& Express, in fact) want that program to work as well as it possibly can. I believe that >95% of Express users code that way and are successful. I think you mentioned a similar fact in Dallas. TMC is prepared to say that the program is incorrect and won't work at all. IBM is not convinced either way. I don't see how to reconcile these camps in any good way. If we allow for the possibility that the code doesn't work AT ALL then the 95% of Express codes I mentioned above ALL have to be re-written in order to become portable. Furthermore, if the only way to guarantee portability, even within MPI, is to assume "half-duplex" communication then why not toss the other kind completely -- it would certainly reduce the complexity of the draft? Jon I think the answer (if there is one) is to provide a mechanism where the user declares a need for system buffering. This is a job for the system environment group to work out properly. With a declaration, you get: - Portability: The code is written for a specific buffering model but will port between environments because it tells the system what it needs. - Efficiency: If the user declares no buffering, the system can eliminate the associated overhead. (How to do this is a trick.) - Bloat: Every vendor must provide system buffering in case the user declares a need for it. Paul I agree....... do you think TMC will? Should we attempt to make a formal suggestion along these lines to the entire group. "Prepare to be flamed!!!!" Jon [I'm ready] Paul From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:59:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15545; Fri, 5 Feb 93 18:59:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06448; Fri, 5 Feb 93 18:57:54 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:57:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06440; Fri, 5 Feb 93 18:57:50 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA18688 sendmail 4.1/UCSB-2.0-sun Fri, 5 Feb 93 15:55:42 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA23498; Fri, 5 Feb 93 14:42:13 PST Date: Fri, 5 Feb 93 14:42:13 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302052242.AA23498@garuda> To: mpi-comm@cs.utk.edu Subject: order preservation Hello: The issue of order preservation of messages when there are multiple threads at the sender was not clarified at the January meeting. What do people have to say about this? It seems that there is no need to require message orderings across different threads. In other words, the order that the messages are received should be consistent with the ``program order'' which is defined by the order in which statements get executed. Thus messages in different threads may not be related and there should not be a need to preserve their order. On a different note, there was some question regarding the need for a synchronous send operation. One reason that they may be useful is that they might be the only means to ensure true portability. As illustrated by the following program that we discussed at the January meeting, asynchronous send operations are not portable across CM-5 and Intel. Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv In other words, the standard could inquire that programs employing synchronous send/receive operations be portable across machines. It would be difficult to meet the same requirement for the asynchronous operations. --Ambuj Singh Dept. of CS, UCSB From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 06:54:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18563; Mon, 8 Feb 93 06:54:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17241; Mon, 8 Feb 93 06:53:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 06:53:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17231; Mon, 8 Feb 93 06:52:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03743 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 06:52:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26581; Mon, 8 Feb 93 11:52:48 GMT Date: Mon, 8 Feb 93 11:52:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081152.AA26581@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00721; Mon, 8 Feb 93 11:50:58 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show Content-Length: 1971 While I am no fan of arbitrary system buffering of messages, I believe that if it is to succeed MPI must allow the easy importation of existing Express and NX style codes. This necessarily implies that system buffering is required, so that code like this Blocking send to myself Blocking receive from myself works. (This is an even simpler version of the test case given by Jon/Paul). I agree with the approach that the user should declare her buffering requirement up front. This seems to be the only way to ensure portability. (Even if all it achieves is ensuring that she gets an "MPI-INSFMEM insufficient buffer space available" message on some systems, this is still preferable to a crash or hang). This is exactly the set of issues Marc was addressing in his discussion on SAFE and UNSAFE programs, and the amounts of buffering such codes could assume (he also addressed issues such as "How many outstanding messages can a code assume it is allowed ?"). I've a feeling that a lot of users (most ?) don't actually know what their buffer requirement is, however I still think that requiring a user specification of required buffer space is the correct way forward. (It might also be nice to have a system enquiry function which could return the buffer space currently used and the high water mark. This would at least let users find out about the behaviour of their codes. Unfortunately this information is rather implementation specific, and would not be accessible through the currently proposed profiling interface. Is this an issue for the profiling committee, or the environmental enquiry one ??? [As chair of the Profiling committe I'd vote for Environmental Enquiry !]) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 07:10:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22055; Mon, 8 Feb 93 07:10:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17649; Mon, 8 Feb 93 07:09:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 07:09:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17641; Mon, 8 Feb 93 07:09:23 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03842 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 07:09:20 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26809; Mon, 8 Feb 93 12:09:13 GMT Date: Mon, 8 Feb 93 12:09:13 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081209.AA26809@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00724; Mon, 8 Feb 93 12:07:22 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show [more] Content-Length: 2029 One more point I ommitted before... Paul says :- > to work out properly. With a declaration, you get: > > - Portability: The code is written for a specific buffering model but will > port between environments because it tells the system what it needs. > > - Efficiency: If the user declares no buffering, the system can eliminate > the associated overhead. (How to do this is a trick.) > > - Bloat: Every vendor must provide system buffering in case the user declares > a need for it. The last point is not actually true. It's entirely reasonable to have a standard which contains a buffering request and allow it to be refused. (There's not actually any point in having the request unless it can be refused !) I don't see a huge logical distinction between an implementation which permits system buffering of 1 byte and one which permits no system buffering. If the user needed 1Mb, then neither system is any use ! So it's merely an implementation quality issue. If someone has been able to sell their current system and keep their users happy without system buffering, then they should still be able to keep them happy with an MPI implementation in which the amount of system buffering is zero, and any request for more fails. Such an implementation would be standard conforming too. (Though such a vendor might expect some moaning from people trying to import code onto the machine, and maybe market pressure would push them into an implementation which did support system buffering !) (As an analogy, I believe that it is possible to have a standard conforming ADA implementation whose sole action when presented with any program is to raise the "Implementation limit exceeded" exception. Not useful, not saleable, but standard !) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 08:54:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26134; Mon, 8 Feb 93 08:54:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20868; Mon, 8 Feb 93 08:54:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 08:54:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20822; Mon, 8 Feb 93 08:52:39 -0500 Date: Mon, 8 Feb 93 13:52:17 GMT Message-Id: <16939.9302081352@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.)) To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi Here comes my 5p worth on the subject of buffered and unbuffered comms. For completeness: by unbuffered I mean that the sender blocks until the message has been (or certainly will be) copied into the space of the receiver, a la occam for example; by buffered comms I mean that the message is copied away into a system buffer somewhere and the sender returns, a la PVM for example. It seems to me that there are a few different issues in this subject which discussions may have touched on. a) Ease of programming. I don't think anyone can really disagree that many programmers will report that they find buffered comms easier to use. These users have been fortunate enough not to have come up against the boundedness of buffering provided with the system. With high probability they are programming (or adapting) applications by inserting message passing calls directly into the application source. b) Portability of programs using MPI between different machines. [Here I give away my bias.] It is very difficult to make statements about portability (and reliability/correctness) of programs which use buffered messages and rely on system storage capacity and structure thereof. This becomes particularly difficult when the application makes use of substantive libraries, which themselves are written using buffered comms and place requirements on system buffering. It can just become to difficult to work out how much buffering you need. We played with a system which allowed the programmer to configure the system buffering, and this feature was only used (properly) by high priests. In my work I will only be able to use MPI if I can write substantive libraries in the confidence that they will not be subject to failure due to running out of such buffer space. Therefore we use only unbuffered message passing with a mixture of blocking/nonblocking (like Intel NX isend/irecv/msgwait) calls. Therefore my bias :-) c) Portability of existing applications (using existing message passing interfaces) to MPI. This is a different subject to (c). Following the discussion it seems that an argument in favour of adopting buffered comms is the number of existing programs (eg, lotsa programs written using Express) which use unbuffered comms. It has been my experience that migration of applications between different, broadly similar, message passing interfaces is not so difficult, except for this point. In a nutshell, the surgery you have to perform to move programs/libraries between an interface that forces buffered comms and one which forces unbuffered comms and blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. Given the above, I come to the conclusions: i) MPI must contain unbuffered communications with blocking/nonblocking (irecv/isend/msgwait kinda thing) calls, for reliability and portability. ii) If a goal of MPI is that existing applications using message passing interfaces (eg Express, PVM) should easily port to MPI, then MPI must also contain buffered comms. This seems to be a matter for the full committee, hence I have crossposted. Incidentally, to pitch in 2ps on some other lines of discussion: * I can see no particular benefit in allowing a buffered snd/rcv match an unbuffered rcv/snd. I can see considerable inconvenience in forbidding a blocking send/recv to match with a nonblocking recv/send. * Yes, we do try to overlap communications with calculation. (Sometimes it doesnt buy you, sometimes it does. Sometimes we have specifically needed to avoid such overlap in order to maximise performance, but that was a weird one :-) Just as important, we frequently "overlap" communication with communication (ie, use nonblocking calls) to avoid deadlock. * Please, oh please, let MPI decide that communications should be adressed using a rank relative to a group/context (0 ... GroupSize - 1) or (1 ... GroupSize). We do this all the time, and its very very convenient. In fact, when we can't do this we end up having to create arrays of the task identifiers - we end up doing it ourselves anyway. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Tue Feb 9 02:23:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19163; Tue, 9 Feb 93 02:23:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06697; Tue, 9 Feb 93 02:22:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Feb 1993 02:22:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06689; Tue, 9 Feb 93 02:22:45 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA20792; Tue, 9 Feb 93 01:22:44 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA04771; Tue, 9 Feb 93 01:22:41 CST Message-Id: <9302090722.AA04771@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: The Jon and Paul show (was: 5184; one small point) In-Reply-To: Your message of "Fri, 05 Feb 93 15:02:57 PST." <9302052302.AA06572@SSD.intel.com> Date: Tue, 09 Feb 93 01:22:40 CST From: Rusty Lusk | | Folks, here is an offline exchange Jon and I had that you might enjoy. .... | Should we attempt to make a formal suggestion along these lines | to the entire group. "Prepare to be flamed!!!!" (not a flame) I did enjoy this exchange, because it is a tutorial on the buffering problem. Paul says it perfectly: You can't solve the system buffering problem by careful or clever definition of the semantics of a message passing interface. I agree that most users would rather not worry about it (and don't) and we should allow systems that try to handle the problem for the user to do so (the example programs in the discussion should run). I agree with Jim Cownie on this point. For portability, an enquiry function can be provided to alert the user who checks to find out how much (or how little) buffering is provided. I think the environmental enquiry subcommittee should propose something along these lines. As Paul says, handling this robustly has a cost, and some users certainly want to avoid it. After all, the user can know some things about his program that the system cannot, and should be able to take over responsibility for the buffering problem if he wants to. This is just what is done in the case of the "ready-receiver" communication. I admit that this is not a particularly euphonious name, but some way is needed for the user to relieve the system of buffer management if he wants to do so in the interest of greater efficiency. Rusty From owner-mpi-comm@CS.UTK.EDU Wed Feb 10 01:16:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13938; Wed, 10 Feb 93 01:16:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11091; Wed, 10 Feb 93 01:15:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Feb 1993 01:15:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11083; Wed, 10 Feb 93 01:15:33 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02367; Wed, 10 Feb 93 00:15:31 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA06777; Wed, 10 Feb 93 00:15:28 CST Message-Id: <9302100615.AA06777@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels of collective operations Date: Wed, 10 Feb 93 00:15:27 CST From: Rusty Lusk A few days ago, we posted a suggestion for a multi-level MPI specification as a way to manage the complexity inherent in providing a highly functional interface, and to make MPI accessible to users who did not need or want all of its capabilities. The examples there were only for point-to-point communication. We thought it worth exploring whether the same ideas might be useful in the context of the group operations. Here are a few thoughts. The idea here is not to propose specific syntax, but to suggest how simultaneous adoption of multiple levels of collective operations might help provide both a simple interface and a full-featured one. Level 4: (very, very simple) Here there would be only one group, the default group of all processes, so there is no group id in the parameter lists. MPI_barrier MPI_combine(...,SUM,...) MAX ... (That is, only a fixed set of combining operations) MPI_broadcast(....) All operations would block until globally completed. Message types would be just like level 4 of the point-to-point operations: either contiguous strings of bytes, or arrays of specific data types specified like (...,datatype,numitems) The main point is that using just this level of group operations one could port many existing programs that use global operations, without introducing any explicit management of groups. Level 3: (still pretty simple, but more functionality) All the utilities proposed by Al Geist at the last meeting for the management of groups (creation, inquiry, etc.) Operations at this level have group id as an explicit argument (Again as Al suggested). User-defined combining functions. Gather operations. Main restriction: for simplicity, still simple contiguous buffers made up of either untyped byte strings or arrays of fixed types. All operations still block. In fact, we propose that there not be non-blocking collective operations, even at lower levels. The necessity of cooperating in the global operation beyond just making one's own explicit contribution puts non-blocking global operations more in the "threads" domain. Level 2: (More options for buffer structure, as in the point-to-point routines at this level) Here buffers specified for broadcasting, combining, etc., could be completely general, including mixed-type, non-contiguous buffers described by vectors of (address,datatype,numitems) triples. Level 1: (Full functionality, plus separate "setup" parts of the operations, like in Marc's point-to-point proposal.) Here operations would have a separate "init" call. Being separate has the same advantages as for the point-to-point operations, in that resources could be reused in loops. The "init" call would return a handle that could be further modified (setting buffer type, address, etc.) before use. The payoff seems potentially greater here because some of the "init" calls may turn out to be inherently non-scalable, and therefore reuse of their results is important. Summary: Multiple levels might help deal with conflicting demands for both simplicity and functionality in the case of collective operations as well as for point-to-point operations. What was generally discussed at the last meeting is very much like Level 3 above. An even simpler interface (so that groups need not be explicit at all) is provided by Level 4, while the somewhat grubby aspects of multiple and complicated buffer formats is pushed down into Level 2. Finally, Level 1 provides the opportunity to absorb some of the difficult-to-scale parts of global operations into separate, reusable calls. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Fri Feb 12 04:14:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05929; Fri, 12 Feb 93 04:14:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23402; Fri, 12 Feb 93 04:12:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Feb 1993 04:12:57 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enseeiht.enseeiht.fr by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23393; Fri, 12 Feb 93 04:12:50 -0500 Received: from marylin (marylin.enseeiht.fr) by enseeiht.enseeiht.fr, Fri, 12 Feb 1993 10:12:45 +0100 Message-Id: <199302120912.AA08776@enseeiht.enseeiht.fr> Date: Fri, 12 Feb 93 10:10:29 +0100 From: Michel DAYDE To: mpi-comm@cs.utk.edu Subject: Re Multiple levels of cllective operations I like the idea of multi-level MPI specifications since it is a nice way of dealing with functionality and simplicity. Also the fact that Level 4 of collective operations avoids to introduce any explicit management of groups will simplify the life. Michel Dayde, ENSEEIHT-IRIT and CERFACS. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 07:56:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02001; Mon, 15 Feb 93 07:56:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14309; Mon, 15 Feb 93 07:54:13 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14301; Mon, 15 Feb 93 07:54:07 -0500 Date: Mon, 15 Feb 93 12:54:01 GMT Message-Id: <21675.9302151254@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Mixed F77/C, MIMD(not SPMD) To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. The January minutes state that It was agreed that explicit remarks that MPI is intended to be usable with multithreaded proesses and with MIMD (not just SPMD) programs should be added somewhere. We at EPCC most strongly support the proposal that MPI should provide for MIMD programs and not restrict itself to the world of SPMD programming. The SPMD model has been useful, and can be expected to remain useful, for tweaking serial (usually Fortran77) programs in a strictly data parallel fashion to derive a parallel program. However, this is not a string enough argument for restricting MPI to SPMD. There are a lot of applications out there which are not really suited to such simple tweaking. Further, we believe that MPI needs to provide support for library writers, where libraries can consist of a procedure library linked into user processes along with one or more library service processes. The concern felt by users and gurus alike is that our perception of much of the discussion in subcommittees is that it seems to be proceeding with scant consideration of issues in MIMD programming as opposed to SPMD programming. I suggest that this should be prioritised within the MPI. I am happy to write a proposal or discussion document on these issues, but I have no time to write either before the next meeting as my travel schedule starts at 6am tomorrow. So, I'm afraid this point was simply raising a flag that there is concern, without a proposal on the matter, for which I apologise. Looking forward to meeting and discussing at the meeting later this week! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04616; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20711; Mon, 15 Feb 93 10:19:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20702; Mon, 15 Feb 93 10:19:36 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15992; Mon, 15 Feb 93 09:19:33 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA27181; Mon, 15 Feb 93 09:19:31 CST Date: Mon, 15 Feb 93 09:19:31 CST Message-Id: <9302151519.AA27181@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 15 Feb 93 12:54:01 GMT <21675.9302151254@subnode.epcc.ed.ac.uk> Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Again, a few clarifications: b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. My interpretation (and in fact, the reason that Rusty and I included such a method in our implementation of the November draft) is that there needs to be some way to write, entirely in MPI, a running program, and there is a chance that we could agree on a method for running SPMD programs. It was definately NOT meant to exclude other models; just a position that MPI needs to include some way to write a complete (at the source code level; we do not intend to specify the OS interface to "launching" the program) parallel program for a common and simple model. I'd be quite happy adding text that indicates that MPI is intended for MIMD models as well. I would be very interested in seeing a proposal for an interface for MIMD programs. Bill From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04617; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20721; Mon, 15 Feb 93 10:19:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20713; Mon, 15 Feb 93 10:19:40 -0500 Received: from jadoube.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15995; Mon, 15 Feb 93 09:19:37 CST From: levine@antares.mcs.anl.gov (David Levine) Received: by jadoube.mcs.anl.gov (4.1/GeneV4) id AA21322; Mon, 15 Feb 93 09:19:36 CST Date: Mon, 15 Feb 93 09:19:36 CST Message-Id: <9302151519.AA21322@jadoube.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. Let me second this, if for no other reason than the lack of dynamic memory allocation in Fortran 77, and the relatively small physical memory available on the nodes of distributed-memory machines. In porting Fortran 77 codes to distributed-memory machines I consider the use of C for dynamic memory allocation to be "mandatory." --dave levine From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:20:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07231; Mon, 15 Feb 93 12:20:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25990; Mon, 15 Feb 93 12:18:19 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 12:18:17 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25979; Mon, 15 Feb 93 12:18:15 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA03629; Mon, 15 Feb 93 09:18:12 PST Date: Mon, 15 Feb 93 09:18:12 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151718.AA03629@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility I agree with Lyndon Clarke that interlanguage compatibility needs to be an explicitly stated goal of the standard. The user community is assuming that this is true.... Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 13:17:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09917; Mon, 15 Feb 93 13:17:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29053; Mon, 15 Feb 93 13:16:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29045; Mon, 15 Feb 93 13:16:09 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 13:16:06 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28124; Mon, 15 Feb 1993 13:16:04 -0500 Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151816.AA28124@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:00:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11570; Mon, 15 Feb 93 14:00:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01123; Mon, 15 Feb 93 13:59:09 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:59:07 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01110; Mon, 15 Feb 93 13:59:04 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00836; Mon, 15 Feb 93 12:58:50 CST Date: Mon, 15 Feb 93 12:58:50 CST From: Tony Skjellum Message-Id: <9302151858.AA00836@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, berryman-harry@CS.YALE.EDU Subject: Re: C/Fortran Compatibility I agree with Scott. I think that a valid, related point, is that there need to be a C interface which is convenient for C programmers, not just a FORTRAN interface, which is usable by C programmers, but not natural. - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:41:33 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Content-Length: 937 Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:08:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11680; Mon, 15 Feb 93 14:08:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01543; Mon, 15 Feb 93 14:07:50 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:07:49 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01534; Mon, 15 Feb 93 14:07:48 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA01566; Mon, 15 Feb 93 14:00:38 EST Date: Mon, 15 Feb 93 14:00:38 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302151900.AA01566@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: C/Fortran Compatibility Asking for compatibility is a lot in some respects. going far beyond the MPI interface - such as asking a malloc from a C subroutine to "work" when called from a Fortran program. (library name collisions - need for initializations for both environments, cleaning up both environments etc.) I want to see them together too, but.... Perhaps we should not make it a full "requirement", but rather be careful to set up the MPI language bindings so that they could work together from C and Fortran. and indicate that this is intended on environments supporting mixed programs. I know of no standards work on mixed programs and recall the the discussion of it (ANSI?) as a general area for standardization a couple years back being postponed as too "hairy". cheers - john From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:24:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12649; Mon, 15 Feb 93 14:24:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02252; Mon, 15 Feb 93 14:23:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:23:10 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02243; Mon, 15 Feb 93 14:23:08 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA06780; Mon, 15 Feb 93 11:23:02 PST Date: Mon, 15 Feb 93 11:23:02 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151923.AA06780@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: hat is Interlanguage Compatibility? To me, multilanguage support means that: (a) we provide both C and Fortran programmatic interfaces, and (b) operationally, there will be no perceptible difference whether the programmer uses the C interface, the Fortran, or both in his/her program. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:36:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13372; Mon, 15 Feb 93 14:36:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02815; Mon, 15 Feb 93 14:34:51 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:34:50 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02805; Mon, 15 Feb 93 14:34:49 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:34:46 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28237; Mon, 15 Feb 1993 14:34:44 -0500 Date: Mon, 15 Feb 1993 14:34:44 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151934.AA28237@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Any opinions? In my earlier message, I suggested that a C++ interface was not needed if an ANIS C interface was available. Are there any objections to this? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:44:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13826; Mon, 15 Feb 93 14:44:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03257; Mon, 15 Feb 93 14:43:38 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:43:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03248; Mon, 15 Feb 93 14:43:36 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:43:34 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28255; Mon, 15 Feb 1993 14:43:33 -0500 Date: Mon, 15 Feb 1993 14:43:33 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151943.AA28255@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Perhaps another place? Cherri, Perhaps we should carry on the thread about C++, ANSI C, F77, and F90 to the mpi-lang mailing list. At any rate, I'm very much for having the interface consistent across languages. This does however, limit what we can do in C++ and F90. We cannot, for example, use methods in C++ or optional arguments in F90. Will such limitations cause much gnashing of teeth in the user community? Will we hate ourselves in a year or two for not taking advantage of the truly rightous F90 features? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 15:11:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14568; Mon, 15 Feb 93 15:11:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04587; Mon, 15 Feb 93 15:09:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 15:09:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04578; Mon, 15 Feb 93 15:09:49 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA25287; Mon, 15 Feb 93 11:39:31 PST Date: Mon, 15 Feb 93 11:39:31 PST Message-Id: <9302151939.AA25287@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA06755; Mon, 15 Feb 93 11:39:30 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: john@cs.wmich.edu Cc: mpi-comm@CS.UTK.EDU Subject: Re: C/Fortran Compatibility In-Reply-To: <9302151900.AA01566@cs.wmich.edu> References: <9302151900.AA01566@cs.wmich.edu> Reply-To: knighten@SSD.intel.com (Bob Knighten) POSIX Standardized Profiles are one standard arena where some effort is being made to guarantee some expected level of interoperability. My POSIX working group (P1003.14) is producing the P1003.18 profile for "classic POSIX" covering P1003.1 (system interfaces) and P1003.2 (shell and utilities). This does not include networks or any of the serious problems regarding interoperability. In this context we have been struggling just to find reasonable requirements to express such expectations as that a file written by a C program can be read by a FORTRAN program or an Ada program. We would not even dream of trying to stadardized the way you might link routines produced by different language compilers! -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:26:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18114; Mon, 15 Feb 93 17:26:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10980; Mon, 15 Feb 93 17:25:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:25:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10972; Mon, 15 Feb 93 17:25:09 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA11917; Mon, 15 Feb 93 14:25:03 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA10017; Mon, 15 Feb 93 14:24:59 PST Date: Mon, 15 Feb 93 14:24:59 PST Message-Id: <9302152224.AA10017@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Response to Bob Knighten's comments Bob said: ?We would not even dream of trying to >stadardized the way you might link routines produced by different language >compilers! I don't think that's necessarily the goal of multilanguage support. I should be able to write a program that combines routines written in Fortran and C, using MPI for the message-passing. That does not mean that the function calls look identical in the two source languages, nor that there be a single set of library routines capable of servicing both. But it does mean that I should be able to send a message from a Fortran routine, calling the Fortran library module, and receive it from a C routine that uses the corresponding C library module. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:45:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19007; Mon, 15 Feb 93 17:45:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11672; Mon, 15 Feb 93 17:43:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:43:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11664; Mon, 15 Feb 93 17:43:32 -0500 Received: by enet-gw.pa.dec.com; id AA27000; Mon, 15 Feb 93 14:41:53 -0800 Message-Id: <9302152241.AA27000@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Mon, 15 Feb 93 14:43:30 PST Date: Mon, 15 Feb 93 14:43:30 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: Some notes on MPI 15-FEB-1993 Some general comments from the Massively Parallel Systems Group of Digital Equipment Corporation on MPI: The MPI effort must strive to Keep It Simple !!! The multiple levels idea is good as long as it is clearly stated what the model is in terms of mixing the use of levels within an application. Language independence must be worried about.... C and FORTRAN are a MUST today, C++, ADA, and others will be of interest in the future. MPI must be a reliable message passing system. i.e. The user must not be burdened with running a protocol on top of MPI to support retransmission, etc..... (An optional non-reliable mode should not be ruled out.) MPI should be thread safe. All functions should return error codes not just a fail/succeed status. All subroutines should have a status argument into which an error code can be returned. Groups and communication contexts discussions and specifications must include thread usage rules and guidelines. When in doubt give the user the choice about buffering and other less than obvious characteristics.... But, consistently warn about portability issues WHENEVER a choice is offered. FLAG ALL CHOICES THAT CAN CAUSE PORTABILITY PROBLEMS !!! Be particularly careful with the homogeneous vs heterogeneous environment issues. i.e. Packing verses raw data, XDR etc.... Models / guidelines should be included to address the issue of sending structures and endian sensitive data between heterogeneous environments. When the syntax effort gets serious: All names should be readable, descriptive, and consistent. No numeric constants should exist in the MPI standard. All constants should be defined symbolically only !!! (POSIX inherited a lot of stuff from SVID and should not be use as a strict model) Data hiding via opaque types should be used unless there is a truly justifiable reason. A message type registration facility should be considered as an alternative to communication contexts. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 18:56:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20138; Mon, 15 Feb 93 18:56:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14806; Mon, 15 Feb 93 18:54:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 18:54:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14798; Mon, 15 Feb 93 18:54:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01171; Mon, 15 Feb 93 17:54:04 CST Date: Mon, 15 Feb 93 17:54:04 CST From: Tony Skjellum Message-Id: <9302152354.AA01171@Aurora.CS.MsState.Edu> To: benson@rdvax.enet.dec.com Subject: Re: Some notes on MPI Cc: mpi-comm@cs.utk.edu Please elaborate on these points so we can discuss them at the meeting, in the context subcommitee. >Data hiding via opaque types should be used unless there is a truly >justifiable reason. -- what does this mean???? >A message type registration facility should be considered as an alternative >to communication contexts. I see "current typing practice without any registration," "current typing practice plus context registration," and "typing registration with no degrees of freedom left for user," as the three alternatives. I think the latter is too restrictive, because user cannot encode semantically meaningful bits into types at all. In the first (current) option, this can be done, but there are big risks of conflicts in real programs. Hence, I continue to favor the middle option. - Tony From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 08:18:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16998; Tue, 16 Feb 93 08:18:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19183; Tue, 16 Feb 93 08:16:05 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 08:16:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from inet-gw-1.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19171; Tue, 16 Feb 93 08:15:51 -0500 Received: by inet-gw-1.pa.dec.com; id AA16702; Tue, 16 Feb 93 05:15:46 -0800 Received: by mpsg.mps.mlo.dec.com; id AA14150; Tue, 16 Feb 93 08:15:38 -0500 Date: Tue, 16 Feb 93 08:15:38 -0500 From: linden@mps.mlo.dec.com (David C.P. Linden) Message-Id: <9302161315.AA14150@mpsg.mps.mlo.dec.com> To: tony@Aurora.CS.MsState.Edu Cc: benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu In-Reply-To: Tony Skjellum's message of Mon, 15 Feb 93 18:57:10 -0500 <9302152357.AA13311@mpsg.mps.mlo.dec.com> Subject: Some notes on MPI [I work with Ed in Digital's Massively Parallel Systems Group.] >Data hiding via opaque types should be used unless there is a truly >justifiable reason. This means that, unless there is good reason, the user gets a cookie and must use cookie-manipulation routines to perform actions. A major example is process ids. We think PVM took the better approach by having PIDs be opaque integers. This allows processes to join and leave, which MPI seems to rule out, but which we think should be reconsidered. Having the notion of 0..n-1 or 1..n may have some level of convenience, but we think it is too restrictive. (Additionally, there is the paradigm problem. 0..n-1 is the C way of numbering, but 1..n is the FORTRAN.) >A message type registration facility should be considered as an alternative >to communication contexts. A big problem we see with communication contexts is that they aren't thread-safe. Their purported purpose is to ensure different pieces of code (e.g., libraries, user codes) don't conflict with each other in the space of types. That's a real problem worthy of solution, but the lack of thread-safety in communication contexts casts doubt on CCs as a solution. A type registration system (compare (very loosely) the X windows color space registration system) would allow users and libraries to allocate and deallocate a range of types. With appropriate allocate routines, you could specify alignment constraints in order to bit-encode semantics. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 10:29:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25120; Tue, 16 Feb 93 10:29:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25032; Tue, 16 Feb 93 10:27:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 10:27:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25024; Tue, 16 Feb 93 10:27:25 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA02715; Tue, 16 Feb 93 07:27:19 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA11099; Tue, 16 Feb 93 07:27:18 PST Date: Tue, 16 Feb 93 07:27:18 PST Message-Id: <9302161527.AA11099@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Continued comments on multilanguage support In response to Scott Berryman's comments: >At any rate, I'm very much for having the interface consistent >across languages. This does however, limit what we can do in C++ >and F90. We cannot, for example, use methods in C++ or optional >arguments in F90. Will such limitations cause much gnashing of teeth >in the user community? Will we hate ourselves in a year or two for >not taking advantage of the truly rightous F90 features? I'm not convinced that optional arguments are all that good for the general technical computing community, anyway, unless there is VERY GOOD interprocedural argument checking going on (which lets out most current systems). Consistency is much more important at this point in the evolution cycle. Cherri Pancake P.S. I tried to post this to mpi-lang, but got rejected. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 15:03:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08057; Tue, 16 Feb 93 15:03:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10592; Tue, 16 Feb 93 15:00:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 15:00:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10578; Tue, 16 Feb 93 15:00:29 -0500 Received: from s307.cs.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA07659; Tue, 16 Feb 93 14:53:15 EST Date: Tue, 16 Feb 93 14:53:15 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302161953.AA07659@cs.wmich.edu> To: mpi-comm@cs.utk.edu Hi; A late pre-meeting post. I hope we can talk about the possibility of adding an io specification to MPI in Dallas. A couple of us (Leslie Hart, ... and myself) have been talking about possible MPI io specifications. Here is a brief outline of IO specifications for one option, a loosely synchronized version of multiple IO based on the ANSI-C librtary. Placing "mp_" before all names is fine (It is my preference to do that in all libraries). Leslie and others have a seek and read/write primative whuich has merit if processes were going to read from differeent "random" positions in a file. Assuming a message is sent for every call this could be high overhead, but it keeps life simple. This also allows IO request to be asynchronous (assuming the seeks are absolute), although we may wear out the disk head actuator ;-). Perhaps we should set basic goals for an IO system: 1) interaction with a terminal 2) writing and reading buffers to files, which should correspond closely to sending and recieving messages. 3) three modes of operation - single_io(), ordered_io() and unordered_io() 4) some specification of conditions for deadlock free use. Some questions: Should all IO be loosely synchronized? Should all IO "require" a message be sent with that request (implying no buffering / queueing)? If not "require" then, should the user have any control over the buffering / queueing? I'm not sure about how to address hetrogenious access to a file. I have seen a couple options. I would tend to not want to force a "binary" file compatable with all architectures. Text files should not pose as large a problem. It does appear binary conversions by the message system in pt-2-pt has supporters, if that works out then soemthing could here too. This is still a thought exercise, not a proposal. john ---------------------------------------------------------------- The Three MPI IO Modes The three levels of routines below and the three IO-modes are orthogonal. All IO routines must be executed by all members of a group in a loosely synchronized manner. All processes must invoke the same primative on the same stream. The following IO routines operate in three modes. Initializing IO and switching between modes using: single_io(group) ordered_io(group) unordered_io(group) Terminating IO from a group is done by: end_io(group) Each of these calls must be executed by all members of a group. In single_io() mode the IO routines should function from a process as described in the relevant standard. On output or in setting the file position the results from only one of the processes will be done. On input all processes receive the same values. In ordered_io() and unordered_io() the results of setting file positions are done for only one process. Input and Output from a group of size p results in p separate items being read or written. The ordered_io() mode requires the p IO operations to occur in process order. (If we have a group order this should be group order) The order of the p IO operations in unordered_io() mode is unspecified. **************************************************************** The POSIX IEEE Std 1003.1-1988 C functions From Chapter 8, 4.9 (Which is a pointer to the ANSI-C standard X3.159-1989 which is essentially the same as ISO 9899-1990) Those functions not mentioned in POSIX which were later included in the ANSI-C are still be be considered required by POSIX. Below: the functions marked with a "*" are non-ANSI-C the functions marked with a 1 provide a complete set, without access to buffer cantrol. Level-1 IO -------------------------------------------------------- 1 fopen(), FILE * fopen(const char *name, const char *mode) freopen(), FILE * fopen(const char *name, const char *mode, FILE *stream) 1 fclose(), int fclose(FILE *stream) 1 clearerr(), int clearerr(FILE *stream) 1 feof(), int feof(FILE *stream) 1 ferror(), int ferror(FILE *stream) setvbuf(), (not mentioned in POSIX, but it is in ANSI-C) int setvbuf(FILE *stream, char *buf, int smode, size_t size) fflush(), int fflush(FILE *stream) rewind(), int rewind(FILE *stream) fsetpos(), (Not in POSIX, but in ANSI-C) int fsetpos(FILE *stream, const f_pos *pos) 1 fseek(), int fseek(FILE *stream, long offset, int ptrnane) ftell(), long ftell(FILE *stream) fgetpos() (Not in POSIX, but in ANSI-C), int fgetpos(FILE *stream, fpos_t *pos) 1 fread(), int fread(char *ptr, int size, int nitems, FILE *stream); 1* sfread(), (a possible non-ANSI-C stride version) 1* ifread(), (a possible non-ANSI_C iovec version) fwrite(), int fwrite(char *ptr, int size, int nitems, FILE *stream); 1* sfwrite(), (a possible non-ANSI-C stride version) 1* ifwrite(), (a possible non-ANSI_C iovec version) Level-2 IO -------------------------------------------------------- setbuf(), void setbuf(FILE *stream, char *buff) perror(), int perror(const char *err) 1 printf(), fprintf(), sprintf(), 1 scanf(), fscanf(), sscanf(). Other ANSI-C IO -------------------------------------------------------- putc(), putchar(), puts(), fputc(), fputs(), getc(), getchar(), gets(), fgetc(), fgets(), vprintf(), vfprintf(), vsprintf(), (Not in POSIX, but in ANSI-C) remove(), rename+(), tmpfile(), tmpname(), ungetc(), ****************************************************************** Remarks: Deadlock ------------------------------------------------ Something must be said about deadlock, this needs to be addressed as part of the collective communications and pt 2 tp too. Some options are No deadlock will occur if: no messages are "active" while any IO mode is enabled (too strong but safe) no messages are "active" in the IO group. no messages are "active" in the IO group, during the execution of each IO opeation. all processes in the IO group have free receive/transmit buffers during the execution of each IO operation. Some type of inquiry function might help. All processes in the group may execute a probe to find out if it is safe to call the IO operations (humm....) Remarks: Sizes of objects ---------------------------------------- Restriction on the sizes of objects generated by the different processes could be made. Possible examples: In fread()/fwrite() size and nitems must be identical in all processes. In printf()/scanf() the same format string appears in all processes. These would make implementation faster and easier (and should be true in single_io() mode IO.) It does make slightly irregular data distribution harder. (Why can't P always divide N :-) ). Remarks: Buffering ----------------------------------------------- There are two types of buffering to consider, queueing multiple output requests before sending them and buffering the response to input requests (as the normal ANSI-C buffered IO rouitines do.) Buffering works in single_io() mode, but it may not make sense in ordered_io() or unordered_io() modes. Queueing input requests could make sense, for non-blocking requests. In our distributed graphics system we have some asynchronious commands, which are not loosely synchronized. For these commnads each process gives an independent stream to the server, which it must de-mux into a single valid stream. Any calls which change "state" in the graphics server are loosely synchronized. So the server dose not have to keep state for each process, just insure a stream can be interupted from one process at a valid place before switching to another processes stream. I don't think that type of IO is needed most. Since the most heavy use of IO would be the ordered_io() / unordered_io(), and that would seem to be large size chuncks of data, buffering multiple IO requests may not be needed. For terminal IO, the usual line buffering should be OK. What I'm really trying to figure out is, if setting up things with no buffering/queueing specified has a negitive enough impact on any applications to worry about. That boils down to how applications would use IO routines. Remarks: Higher level IO ----------------------------------------- Mapping r-dimensional data onto a logical s-dimensional process array is one fundimental high level IO operation. The ordered_io() and unordered_io() calls above allow a fixed dimension data file to be operated on by a fixed dimension array. I'm not sure how often dimensions above 3 are needed. That is, how often problems of higher dimension can't be well handled by projecting into 1-3 dimensions first (for the data/process distribution). If the size of the array was changed, a single_io() read of the file could be done, disgarding items not needed. For non-parallel disks (from a host or other single source) on an MIMD machine, I don't think this posses a serious problem. For high preformance parallel disks, or disks distributed on a network, that could be unacceptable. The only way to support that type of IO would be to define the process array and the data file and let the system sort it out - as well as choose the files internal distribution. (As in Hart's system propoasal). The EXPRESS model is successful and should be considered. Remarks: Multiple process IO ------------------------------------- What if anything to say about multiple processes operating on files? limit the total number of open files? (eg System Constant 12) limit any file to be open by only one group for writing, for defined behavior? allow any number of groups to open the same file for reading? (requires the server or group to keep state for each group). Remarks: The three buffering modes in ANSI-C. --------------------- The three buffering modes are: unbuffered block buffered line buffered by default: stdout is line buffered stderr is unbuffered all others are block buffered The setbuf(FILE *stream, char *buf) call can set buffering on a newly opened stream to unbuffered (buf == NULL) or block buffered, using a user allocated buffer. he buffer buf must be of a fixed system supplied size. Useful buffering calls, not mentioned in POSIX: setbuffer(FILE *stream, char *buf, int size), allows user specified buffer sizes to be used. setlinebuf(FILE* stream) puts a stream into line buffered mode. setvbuf(FILE *stream, char *buf, int type, int size) sets any of the three modes (using type) and allows a user sized buffer to be used. setvbuf is in ANSI-C and I suggest it be included. The tie in of the buffer and sending a message could be made, implying a message is sent when the buffer is filled. Giving the user some control. I'm not sure I would like to state something like this as a requirement. Remarks: non-C Language IO ----------------------------------------- The functions above are clearly C oriented. Compatioble calls could be available in other language bindings. No attempt to specify native Fortran IO in MPI is suggested. Remarks: The varargs forms ----------------------------------------- The varags form of the buffered output family (vprintf(), vfprintf() and vsprintf() ) is in ANSI-C (it was not mentioned in POSIX). They could be useful in avoiding extra memory copies. There are not corresponding varagr buffered input versions defined in ANSI-C. I would suggest that they would not be as useful as a stride and iovec version of the fread() and fwrite() calls. I think iovev/stride versions of fread() and fwrite() should might be the first extension to the ANSI-C standard to consider including in Level-1 IO. The format for these would be compatible with whatever the pt-2-pt calls become. Remarks: Blocking, Non-Blocking and Synchronized modes ---------- Some of these calls could be used in all 3 forms. The setvbuf() call already could give a lot of control over buffering. It might be wise to only consider "extra" modes for the fread() and fwrite(), which would correspond the modes finally in pt2pt. (though if extra modes are implemented here they should be easy for the implementer to add to calls such as fprintf() and fscanf() ) Remarks: The modes in the fopen call ---------------------------- Limiting the fopen modes to read only and write only would simplify some things. Remarks: The unbuffered IO calls. ------------------------------- Although the POSIX IEEE Std 1003.1-1988 C functions do include the usual UNIX unbuffered IO functions, I think we could take the same approach ANSI-C does and only specify the buffered IO functions. Because of the fread() and fwrite() calls no functionality is lost. If the unbuffered calls were included I would suggest the calls readv() and writev() be included as well. int readv(fd, iov, iovcnt) int fd; struct iovec *iov; /* pointer to an array of iovec structures */ int iovcnt; /* number of iovec structures */ where, struct iovec { caddr_t iov_base; int iov_len; }; From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25996; Thu, 25 Feb 93 12:19:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06338; Thu, 25 Feb 93 12:17:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06280; Thu, 25 Feb 93 12:16:46 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:32 GMT Message-Id: <2150.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26012; Thu, 25 Feb 93 12:19:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06370; Thu, 25 Feb 93 12:18:21 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06357; Thu, 25 Feb 93 12:18:10 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:58 GMT Message-Id: <2160.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: third part of document >-------------------------------Cut-Here-------------------------------< M3.MBW=4X7R!M"=FS[?T0"!P-9;F G&V9^JE.Y=8!$JUT,N0>,&HSGU:"W Q? M74UY1V+0@-I&:2=3'@V7AD_#)65_,*=O/LPF1:NN;XC-4J5D)^?8$M:NA3:K M"JU#[-7^+U+W_SMD+6*\V'R6?,_EX%(7H]==I.4%TIZ]]\L%WYZPI@PL;"Y# M\6[*!V29\*AQJXCLI18RE?UQ_\_4(;KYB%Q()ND9^C!Z,E7XW-2X+"<&H!H\ MAR/*J5^U0UQE3F-M5LCVZ?)IC.2I[.!4M4IV35S8F'>%K>)XH6Y8(E8X?WL[-+F'>J/_4/4R>)/\,.!6NS.9[:G]!9O.JD2DG MG&M*B9N:\W[V/SP 5 4+B).>B[H4,]%GT;BA2Q"#)PVG!F0T[.//ES MD$N_#>7*7^Z60WQ?E=\!]3H$N=;M*764C\8AF((8<4<# M:2W%D+3K[E.21LS=M1%G.*-VQMUU[IVW@ERS=)YG\PW%""J0A2XX]UCJYJX4AN*:Q-+E]]V*$8E87R73MPOSM_^ MB_&[>F)[,C:P3B.LF()*8I1;K#-P>?KVISV)/9+18UENXJ20J4ZW%<=_MJ+88R\8MKMQZ MB[EZR^=.7K$WW%QOQM!9 65U.%UGKRHYVIM$(-U6C(LX\D?*+R#UK"D-?1<# M3CNDW=YY<<$XEOKF0-Z2>Y..G.R,:?,6Z61,E%+1EO?$_&1T*7+:U19;PU9-\>H%_&#\R8\=&W!UVZLQ'7 MC&&^*<44'^ M,%>$,&@@J:L2F+H/36+"G^?&1%R[,;9CRG8BUKZF$*_/>^-5#1+5\-I]%AS[ M?8O%Z-=EI?BY(IK%_?:]7YO%043'\G)$(;9KIHE#1] O[/?Y" M.\.;U6./K+S%ZY(KR4^4V8<._D33RK_DW,)?^!7AV MCU?1;LU+HL9U(+J-V^4&5.@-/14&!\,^O1P]L8QK?Z/BVT?(&SJGSR*9Q:I3VJ M[\BJYM6+85Q.E0P##0UB&+N6V(8F\K+-]A=R;"2,'%^9XDS)J9!QD28@P@M4 M>56_: NO] '6") $>)3"%>FU_T_9Y8J79HNRJPYO=8ZQ #[4!FXXTNK9E0J# MI/=[_+$:(,OW?EMDS:EF9B>4ES=R,$[:]UF"/"1_Y:2 _\9ZE1Z8ENPKQ0XS MAVO2F$\'J;?6++L?@J>V!6* )>9A+[Q4BAS3O3>^6_D#7M_G87T3-#VQA:.B M#M=LVUM"]-Q9O9S:>TMGEV5\N&&08:.. -KIRS]OUO+2)\8.](0X+[>M^Q8N M"@?3C\)KI@E2@N:9MMVP.=:M8&F9D/10J#@:O@($ >*'.1QS+VP.L;R65B][ M13]T5N2:[F)!=??_PR\3**1_M;VHG>W/(IVMW;W>+],$0F;+=-OD6?CI ZOA MIB>?Q6-)(T3Y'(R3'B+W[9S.FZ-S%@T_ M 9S3\4.:\ALY)-R'.P>CA57!6+Y(F^XUP<<&XB\?>9=TJ^(&8,:9[:?9PU^" M/Z&30N9DW7V3SG<];0L[A?/3(-2[].%/]5>@-C0J6F'%"[[>G\L0;@#9#9J. MEP>A]&DV)6AR5#=Q!NXAAM>'P<0R,F NU3<<;;!FIC]])Q_TE+DZ]&&!.M'ZNR[F#\*@P S^KZU#:G MD:5W+KL%'DMX8?CF6>PAJS3,H,TPP1D ]\!BML@Z$J8#5]JX2EYM--SRX[4] M>^_+3]HX]7_:Q/RA)>H6"=/-M,M:GU* U,9K3+2:[X ,8]>38\KQU"Q@,=%5 MI0V&%S8[\ \ULJ,#K2:J4TV:U&?,33&5B1JL/6DZY'#2SDL4@:AR$+II;!T* MJ1O+H,MS7A[U27P&?$=W+EMOBIPQ01TU*]W2:U3_IJ74Q] ='C&RVBHP63U2 M\V"J56('Z3WA]7C,Y)Q"I>=XIYP'K\.VDS8C;=6&B>&M%+\)M;:3WRN.3JR] M" .:1%ZYW(8U_\PI[D^#>8"U643D=,;5)\CM:?VJ $W31%>XWM%UM?:P3K3> MI2O6NMDI[YU94-S?/=?6;+:!PL#F]'/ZS?#P-7UVF/V,W>HSX[IPMH?7,\:% MY7(,-]2"()LCEK-$!2(.K5F]?%>\,1(7$XWW%3S;>E4UTM3O>*I13['#C&#O/GR>WOMZHYMFQ]39^"/^M M54-QB & @5FU+;HCY@SK/7LV+,$ K\+Z^-IZPX.Z M][S5E.FO[I>:+,>WIA@;KY);VW0C,=-F\>:S9#S,[-AH#&W-[XJL7U@M7Q?/EE3 M--ESD#V6,>J:L B%CEFWKL&1XDC?9V:RGE!>!2/4F9ELW!SV+:H0!>@?[ER_ M&7R??=8:=>EZ_];C4S>O;XO&Z&)/( ,78?VQYH42 5K7L==0Z!5U<6V(F[=) M5P_)7^,09O;:VO2I"6#N*9%Q@]24<"$5 MU4O#@I< F=.9R/(KM884P_9!C; M7JN9IU"(,<$W>JME45];-AX8YA?UJEE_.<^3F>(QZJQ:,[U%?= MJ''!$.BM(>OD?MUYG6&'?1&=\.OLH8[TL)LV3E7B< '8;>-%]) 4;LP4'5O; M:PR'61/JLR4ZN<>T=C[>;*^AZ.[,=O;1I7+@+79/N"]>GPH^VG\7' .F8C_]UB$-/59.6 M?=FD!=C,L9TS#72+[N32<7FPP&,RXAYQQ?:?[@569>XWRF@JXNN7C]V!'0PJ M?P&0A]R=*$-'']W[?>3V_EBPP6/@L> T.C"#3?X22FEIS%^4WYK0>2R$1?G] M8*>_TV-7A'FC,BR/YF/GH\N_605^M%#TO=E!--C>;^38[]]D*,,SYUC_/4C_ M_9C5^M]@+K3TI4L!M,#VJQ-W;%C2-:V:E[DJ_N!N/R^-VF?D&T_-N_=%I0P! M:2O.R@)E*4=3]P=D_2KZ_CS*C^8[M7UWGAQ/[B6_>R%I8E2NL,]5,V?CBT,U=@LCW8&JQ"CBAW M8E7!ZF N=#Z-;]I@A.V.B!4GYXTC*E1(;ISI_(_F7UW$9^YV M=XF[X,S[;(*CH\@2O!M229O$\D?'-? M7BF\+&MW("X4K3VQG45^7/-I?#Z4W4V3"ENW"2'B K^21MPE-J\8^]RF"1"\ M:5:!4^SR:Q>3BAOX/1Q;K>MH[E0^ MH]&(Y^C:L0@V+#C^A6774@-SC-S ')#/.=?[Z^."U8S'@MP_(A"VX'G\?67O MHX&BLVQVM"-[VHG]W5@&MTNYTU]9;A+3P:$#M$Q;(),^H-*IZ"BR)/!(VBQ: MMU62<]^-&N37C&L2G?M",B_;-^@^'-O.?N!,C52*GW7(LQT=L %B M(N:^%S6O RD-W]@K/1V@ME?'J^TOXF_;M0W>U.+("1_9LFU$;L^OMNT9NFU; M):N2CJ^+7V!.-;@U@&+'*+>Y_ MN-?1X$WGY7+;".N$O6D.K1_;I,4;=[$4GYS37."-*W/)6%>)HS0%,0%4.XQ=W7*S M(^S#)6V$(I@[G[8W<,HU"?%S"N63F9HU3DN^MC%WE[O#TUMX''UJG'^$G[<0GH(=?/5JJ^;X\6X9;HGF9KQ=8CAYB M6I/.9FKE7><.NRHEY#3C5W&R5J)Y6XO.)P:7)=CM^ 1YF(33H:YNTDLY7;2^ M92F@L&7LW4HFZRPB?CEZY*Z.T6KUY/@T6QP30%O(,7O/2=LOIR^GD/:TJU)+ M:,O(A>1-'YP8Q/?%0VWTCP'(/N1,JO;X:/L$SGJW >5WB.(]HC'4U?@%KN)I MO!]K+P VIS:X8V@O*F=[+FY3X4AQ!S%=_;5>X1FHRK.9U[6T[%#7P M=I>WJV)I8<:Y>GV"G"QZI&>V$F*ZJ_!&2RF2)/(9<3L$#L_";5!SJBUX=>J5 M.7RM(A6QPU?24 ++9Z=_0R9B/)0WTW MMH@-O%,1=7C2D^OMT$ M!MV_<,SI9@!VLEW'-(E:MO?8_F[UMFC[CUJ.'G*'MC7;A[19]Q\R#92!E0*J M !BP?MP_*2;;=4QZ4P'D<57'5.ZS)*Y[V7WE/DLVNP>YP^U":20;][OF+O_^ M_.9>'+8G0'M3_!.8 QYC^C2_W49*;NI8*TS(WG4?CS6Y>VZ$HY\;EDWM'G$; M$F]KS]]0=F P'LW(L;X0E^O7[G MKF^SI=:TO]FMQ7A0X M*#^K?MT.9K)ARO?0I%G:+XO/"(<<-7?; GQ>+B277(/5@0F\;J8PHNK0=GI7 M% F'6<>D3YBV-' )S6P,0?/#66_,P5VW]>CU+IPRP;/9!%3D\XJ6+FJ;ME%* M1\VTB%48L0=2_PQ':W.&00W.D^4D-)Q62ZT!_()'2X?*06(6]A891RU08P@* M?9&C/>H*WFURM?9)QBJ7'!]K3H#[&A.@3UJGHWQB^0+5W6R\G/E20GD&7#P? M6\&>"N'Y 1) Q _1.VID/AO%!?="D\*?RA&U6GPI6'2)"J*P!7$/[ [I\: MPH>7B/!YF^V/GKC+0Y>&L.-U%L?;G?_81@P,/Q<2XH;AP;E_AF,)".%+7B1P.)\M^ MY3 MPI&FIW"!^*!:=9DTI/96BA'B$;UON 6O(8[W)"F\G%.\@CR/L]]S(UO.W2V' MAWNZ]G_4RLG55FO[%+GB0NYV<;97?JW6[+JR-7&N74,_-/O0VBP,=&N# M0G_:8LX?KMO#<9O55@CZ4Y38P6\F-O/QM!B]$;'@:;1J@6TI+K&8L'TL=GYC M<7MRESAFL:U2_LI$@JX%U_AP^43W'B!.0)G@$_9R]MS89^3WUVF %7*;^2Z_2\3,+^D8=3S_WNV]\O*X/VN46C]OZ,#_.TM^ M!<0 [6_9W73@* FX"8 GN*G1/3]%KG7@_JC;SC1_8@'9(+;[-P7<*>@K!;%A MA%"_NW&2*@:<=NPIU/RJ5-=KSU],MDKUE"T]=L$.8>W1\FAM=XZ-JEH:"&V# M?WVEK5SR.-"IT"V/AHFNH\GCL0A)WO-W#*!.0SINQL4.G?%"X,PY\2N+2"*$ M-.NNT4V5I&8&WCG>[G2W20?D>T<:K^47-9[P4XW_L7FWQM_7^.FXX#VBW-'1 MQH5IL%^$ #N3M;W'FPY\H^O;1;_[0@\)IPVO94:'?VFB"0[:[_\1MS:-)H!# M 7#'O-_Y5''<&\WY#61/P /<[@7GN*I OTH?'T?#J&6_XDY+)X QF>23#^%G_NPU9T:'QT_ MR#'9@NS4L7]PY'VZNY"'N6_C$?+87'3.[X\666' !R MV#2_F#G@KW*\=6PW=1#H5Y_CDKDUN:,;NDDD5Y+7-YW:M>?W*(1-0"3?='=& MQ:#DG&XIN4GT45XEIR(NR+'D?L!DSI8\=-PEKXZOQC%^R/'-[R!;ZDPFMY!S M_S#D:/+X-Y]G3<[M#KZ>H]_DMU^+*HQ<=YR5M),/TR"Y>7+@L8+WX\:G7P4'NG&_6;M!N=$FV29*G#LW0,-CIUM _9UO/&W_EF7C9?)3^9E<0ZXJ[Y&3 MN;4X*G(V(JP\V$V ''8[N+$,[\6AZ'8.R9>P/(XOQV]U+ @=^9B 1RXLQZ$V MRPOES/*]J^%P(0B^S*Z!!B*0A7)N-]AY6" :)V]3RR?FH>XM]\28O##,HN8()=[N'M_%/+9N+H(%XN30:3EXOUT?K?@-S5H!\.:V49D8CK^16P'WES]%AXE8.9FXZ?G82 M0A_F3')CN4V <[.X79B'-B3:9,<60NO;77?G,]T9O.,K37QRV+#IT?8,OF=^8$=N77U'WJKM6, M:TWG$O;)_-SN5,N70Y]&/ZUS'/C'?+)N1_Q2ZLSAV3' MRGOFY=^?.8..\3XX'0)E'SH/EPG.R;7PS0 Z5.ZIH MSD_$+)N?*L23W1VS>:V^N]]T3TMY]TJ/WLWNV\/F,Q;"/.N]MR-A-_D$M@.2 M#U/ W6DQ*M>SF_M>KDLW=.^*^&P.L>3S+7>>Y#K#S26N9(0(8+T6YET/GRV3 M5P&?"MVM6%D84HV:NXW39 >A>[R)LYZVFZN_>\S5J^"R0\\-[ P!-S??742@ M#:.TI9K^ IK.ZZ'-5/1YPPU'7RM9:$_ MBM%^35V\7IC [N<_S WOIW=I2)\ ,[^42^>4':72XBKHC6F0[$_\3XA 59Q: M!JS9?6]5)@CP%XO0QJ!;&R. ML)%H=(WGT82ER('E@,USF;8Z@B[:E@7WDBG M09NZCSF,4+!/5QG\9O;=?9V/YKV/@GG/DA88O]MB2:W8S6\L-ONU^ :M/$4O MQBGFGD_WGCMYD!>,:!@$=2C56KQVMEB4C#Y%QVHZ$TF0O 5J*?DN+#<@2!Z> M5I_ H[];GLB3.*Q\IKQIP:])*C_;W0WT*%H&[Y[^#".)-&XI3F P47-S;&#: M! T"N9Y4V[Y<*PGC:+U)"MIX0-[3UMC+G>9B\X*55UJ-M2H+W$T9>';:RM>17QO!QO%M\ ;AQK5@LX-I["! M0;O5;DIJNAY:Q1RJ53BJ+G,XX_3&M DUJL>M/ER[*2ETUFPQ7=293LW1M4/C M@L&@,Y)%BAJ50>KVKJ<_GBNX8X*Q76M.K+K/G@!2!&B9*<_[&WRW/-R8#JA; M7,O5ZC1N>+IZ>2=6)H":J!$")665J;X<1AQ3"OENS5,O9]S'7]KVQPWUG M1H/62\K\:>62RWLWSHOOM3/1CIN:C6F$*MB)5GX/QIG?HFC#>!8;T[8L-J0/ M;AGCG#@QMB>N8>C=LXR[T_&OM@9R.,?0JNRL?1K&U=7>2\LJ[[6PF8 MYY[1:@Y/-G4\<&X['L&68"&EMG+$*H?MXPI"UM$Y,RY6. MLFNPGG(?[/J-E(W*566#QTO@*O!T$Q,V/.X"WW9#,OM:U>/)7L+/%5%8B:[K MUOV(^075^AI15[=7UWS'E CK=9I^WXN;QAW JUQN:DJEE=(4.<6\0#XM7RZI M)TWC[_/HHB4,44GCY9(WR+WDDW4(N:A\ I[:KG(ONS?KZYHG\1<1>YX;%ZT/ MS/VXI[R].IY;U'A2G9>WUL."PO$I *V\3EX^<K%Y86<1^Q%1_1;U^YJWUV5"RSIXJ[39U U9OZ][ MRK_D$G#/.FQ\/@7@)OT"V'ZY[R)XTW[#O MU?6K)W)"J$*MW?ET'$@K4W,;GL MY7D;O>ZJB=1\;_DU&#)6'R97C_?71[[*[5@-8\ 6:Z(3L,//0NFX]39Y@-ZVC M' ?=^6L]]VH=V/U@U[)SV2?L0]'8.JYI0\S_Y1R'1V%,GFX6^.>0F'6^XOF)UC M"T3FF$-F[;A\O[W^=KM=UOWKEES4,:K]R XH9ZBB6L?DEG8R-T[ I'KF9JW/ MMC>!A'/<]M7++.DG:(#K@+Z>O'+B\6==8+Y;'R.*$H'@DS\SZ M6A-$47.0:!MUTVW'-I![O]NH&7,0.9%[Z7TVCZ^[SINHF[38N=OQNVQ\[_=HU+SBGMDW8B.W<]V^XW%SV^RM'

);9_=V/Y^HCEG>9?MI>VRK\0=TT=QC[:# MRM?M*C7 @D=1$^=9Y["CCMWM2O-"=M$/!CBWD>*X#TWDSNCAN3S\X^YM5W"_ MR+?L5@!(Z9M'$MP 7[GKQL?1J78SNZ4=S9YV)]ORW%7L(LG#Z^=;*.%K57#X MM6_M>9N_(=!=L5[9_AL&W)7)>_X[LYX;)[ CV#/NH?67^[L=CLG YQSVK_M!? F@/+ZH7*GF.^^*,F+V!V<]Q$ MK0?;<'7B)I?T,(YI4QPS=:;?7FSRL2\;*H??"V:#L+<$.5G>'8%3]YT$1^I1 M"ZGI4R'G7.X/['Y,W$QSB)GH._&4U]'.U'=)7TS3>4^F>D]WWYROKP=) [M; M/@_%?CHFZV-TZ(Q8JT^^HR^&1LHRY[)MT\=^OUTWO%FLAK6=M,5S">@Z%1/. MW]U]^+U!,6=2^M>3;4TS7>&[R^:8^A(PA9L/+WH3_U3JU3FQK-OSH2[76[7[ MA'UT]%W2VW:.9"E/I(UVU^W3K9NT..22>[IGHZZ)5*KF1'6]S3_TYIC#,9\Z MY'#IHH8HA!LP]>CE_*5K;D6\Y3N6'3$=0\>@V[\%U-&-RG1/[JB8GGXH=O02 M.*G.QMRH;4XY!9SW-GHW6M^6JLO\@M>W1\[;U:9W;3^]H]XS@+Q8KA3G)*C7 MJ^':2T CI#B3!9^8E<,'TV8DFIDZ?'<];)B!MGKEX:M\XDS_^\!0!5]+[*2; MP3N(. $=*CF-;>/4YJVFGP&OTN)M'7L[$1JET0&NODTXZ;>#8+3OJDZARZHC M/[?J(DJONNY[EMN)EVA':=:VQ_>"Z!T=6(DH8A[6XK]I??1A\1]=;QM(G[[/ MU8MOY6?I]_DY^Q[OV\3MXIYUO;C$P!SG,5!9(, M;R#;*DF>I(MX=.YO_[9IXX728T34L7+YH;O4D%R (4(,V" ;F%91ZW7V>.)ZE^)7WC8<1%3Z44P_O\3:[7 M#:WK>>LDIC5^)3E>N"3ZV1B>#ASM.P^Y>BT?!UT7NHB9;\\,[JEXIAGQXE@%@G_XICQPC@9-Y$31\$R!0MH M_B)Y1<;EW=-R7.B?^\E.&_VG(C6J)[<7AA<#!@&&&IJ"_S1\;!>:C[E/O&6. MO2EI\/>GY)RP#7Q40M/=:3N&/^(=\:86\=)D'3S<9Y95 =X+OKM:=;RRVN\H<:V8*GR0@O$"M]UG6F<_#J9@C=7ZRRL M/K=[2NK"ZHHW&NO.W5EWD-F&'K@>+59S,9JP95O2E1N?=^5,88!9S0AT_-*9 M6]N>3NHP:%QX=RLW5(50U,C,.V(-AZ8,H]6@V:V]-+ADE97:ULW23=8]?,- M&2,>34L9Y^J44H@"+2U3$':[4+[=:?@+.]JCI$9_.[K)-40!N^"( !!-V _1;&H9 "H"7&Z6I M19FS(U[>PK249CLR'3DGM1UVQ\S2Z>EY+SV6LWESV!Y-#0*XJ@R/7U@$_WI. M&M6<:UT:?><'..@/8.0RW'2ZU^ MUN9J.&\P^J!X>)GIJ['%T1)"9,9V,M@63;T4>%+P$\.R=P C9GW>7G>?5]C= MDY.&'&HRH,!;!3@%H/F]WD2,/$/ WO-PJ"JD 2 Z#[N"0Z)TL#/\Z.VI\VV9F46 MU\J<,W3U'.7OQBP[#+YM-A2X].ICZ_YT">I1*\$C4-_'YE7-W:TI=.9A'O0&OK]PK&NS_I9 M"6EIMK]W/9S.*DNEOX@G2./5W=70.!]U>6B!!J5^ R>8%-S'9!(Y:/SEH?>Z M4GEX:O%K4K<>9JQ [-?KB]G$Z]V?,?#V-FVQYX=^ZQ&OUM?R>0Z0$DU'E\4W ML6GQA9HA7Z&&H):+'QS_?4/1TO( 7:E>T)ZB"D3G9C6RVNSP^Y'YZK/D5 M 9@ "_"4>QN@D\UV/T87!$<"9=)"X$A@J#Z%I8KVLB?RV#LO!_I80:\^MNK& ME96-<;6!<\94;U=<[;&UYF;7U>4B)?A6T*Q=AL8VO+^#=M5D>*C->%H)F^@:,^5C=KU MR[/2@;F67G2.PV9H=0/Z(=U[;F?4N0[N(ROFKP,QTU;?O[947F'6D;Y<5JE_<2:MB738%';M$5/QOUHO[P= MA9US -DA MJ6-\NU5@]@Y^Z[QRO([> ,/.HX$Q;?H->UFN_FQGO8?UPE XO+'1[%N&54E3 MDC'?9-@-O=Y;@;QD1+ YNK?E1,U#8K+<44/_35DR.86:GGJWYHD=O_"6)-5[ M'8\J#D\1*0[_ .L6SV%_.7FRQ%XX&E?8(V\B-![8+\N*Q3RU>/9.KBE4K&TS M3?F]?G5^X&H-RY>[K^%)%FO4"[Z_(M0F!YJQOW:32,/:[^L&-%D<7NS1-)+2 M\"^7Q$?\@E,\^XI\G\6?Y$(#_,M/30GG^;[\GEKWXG7V@W2G2L^^B\W43OQ& M$A^#(%'M-U(0K294PPL:.,RX-,GOMW9S3'@1(*C>Z* %C7Q!J<.N42\$B$K> M_#R5E;2_9,^/@9V M+&!>=""]$5F&05/K PU/ .8HX&P%%Z)X8&@;$BQ1--= M+&U^9$F-)5^0E ^D1#K^\466:OO ^B419:GN7C!N!N?V,DY3YN049RF>W%E. MW-S3:_1EJXI.U"NB3R_<)^O@N6&QEOP H5<^-BLOB&,1R,$-MJFO8WC!QOX8 M,PNKR"I-Y/&TE=Z#_QE>$UWQ7RF@8.!P.YK@:.$[Y!CY:+7TJ0PQF5^=E[J? M":?(U. Q_'*TPPB'O$8*(4/CI7PC*"LP 2EB,_C@3#>->F@N=6>.4L?@Y:T3 M^%CQZ7QT/AKS*6Y1S-;?1QV'RNF&N8G><4O2M%&:%,^ 7U"_%2.FL?0]\)# M&NGYP>, \$VS%^**K]E(HIND#T&W[5X_S%]BV.+LDI9=H[?LW.WDV/NX'\X#OYO6R?6OO8*KH_YB2_"E#)3R]> M\E5>?<&PH $RZ.?9#LRMW@_KT_$UHK+.GJ_:O&0N[9^BO#FVO9O-;?_PM(IB M[[*BXNN('F 2%UFU6S G336.SL'0KF./!$U=G<>B&1_8_TKB\^Y#>[ MYJ3)I^88([CMU/KR[%UJ(]1T]^-JKL]T-:Q7%#D>##F>['LF'\O&R2>:%&Q& MTE:\#M8[&F[4XC@_1_"Z&2%VM[S4X G P]@=W%G:_@##"6"9=I^1:M<<)''B MD;.-'TK#KA760@B;@Z3M57=]D],7(0N>4!UA]NX%[+7S/^-J)EOYLVO8;P0; ML\V3OL(.\.\O.@=-].P/325$#CD&??AOO?,$GP,3#*W2EWT9868?Y=CDNX$" M4?-J9L;;7G;V?4H7RH7_;U5TLV&3\OO4?\"5AB,I (W&F:\2X &V40]S*0), M :Y>#H[E[8< ./!9"W,6\?WGGV5W[KN5N\%2Q>^+-AC\$ULCP'.:"5 %. 4R MTM#2IJ$!_]6PN+8EJ!HKG^JL=QF"'@U&MOM=@ENMY$.Y=JG789E-=*J ZD.+;?R7_4G.:!,@D.1) M<.CX;W4[?EQ=D/[J_&P$]5'10_V!:%'_*9K]GAS3&)/ZD\.E/A-7CZW='-': M'_>81\&H_FN[NTC)M^3__+#Z1\%-8!! "-#S) HZYVRE@NX+N';=,#C6C[A7 M.@N"4_ZX9*'F4SH0]62^[;6PYF.XO@//2M=)QO.66%ET83N&+H91L!I0"P-? M9-V0$L[#K$>87I@>KN%B1D.[_T9:WMT>:KR7!>)FZ;BJB\:+(9+O0<_$$U#Z MZ>*@3\+('&$>H6F4!M&QU EW(_E,^&=O.FWQ9.SV8!']B=9N?D#MTE7 M&46-AEFU<>4UCE<=;.I>Y87.^,D:*V@W"IY(MOV=6GF10.OEIUDN8CI>CO5' MU!"MC3JF,^1>5:VKOT$NJ/.W(?A^MXV?DDZBQL[.(8?S'M#JOB8NO1_I'<>R M]]-WDKWY\*>:GHR1O1(OTO"_5Z6T=D\QO\^V"?#R]]D;__T+O^11P^^8-1O& MQ9>^1T+Z9_.M*WW?1Q?G]V^![D,)?ZNPPH_5Q_!?T0G\YL3BSH?_QI_PM_;? M?V& G'H&/@3NQ]]1S(J+!FJ4\D?WJ$T0OB_C]V#RU$B5\WZ]IY%2X9].A4<" M"*;U'<68#_#;93_3I\7W;&YKH1I)'I)_IX^SM^+Z]'MRT>_K^S#>60RTY\== MXIYUI8@( ?1A?B?-T^,$]$KT:'2+_,:19C9N)?#U=86,O\)W\A="/#S8^3:[?2(JTQ=N>'?E6NI2?P? MY5'<*+_C,G$=>QS>6@[::O,H#7 P@% MB5^>GCD;,!I4X)BA-PIG1O.F8L)7D&2H\+_71UL:@_)^<3^>921OM<9Z=*@I M03OHJE$1Z#X["MY?-4:A09&J'.AEHE(7HNYO1H%FQ"]OY590JW*P'0F&^ZNA M(P.-467?7=J2N+LZ5%%27<%M2 "B6:=345! ?-([ U#B:2T]P0G]6AY/]W][3T)*B%EJ?WLSNM.^%="WRQ7;4; M.7?A2^A'0%"X7<16J'J/;XQ>YW3V=')0KGIP6CAI,%\=@'!AIT/\.>QL4G > M@#X!2S9?2CQM46*,9^9TY4+H=+)DS$+G4@M>RS;\;&YFG3N\>A]"]6N9#7YG M&'4X08QG&W4@;+1ID&E@==IRA5?U+L9W"W-!.5A8?U<7?59'93Q8<]!J4V^S M0T(;:&LK"MUBD5?Y-!4XJ4OG2.11(7D4/IHUF% ]-'WBO>O!JQ&R7 :5+C5K? M7UU\""H-7!U[#WNP2\L[N8 "0R5T+5^?>]12(7M:5$-JMSJE2_Q0)DU3-DE2 M'V(_&SM_,GW[2,!*31PL24T<(C@U>Y]HA'%7;+YX@W1($$]^C6-$?6=]9E*Z M3G9_%G/830%>E0%[?U9[ 7E2?:YW6'8$2#0_C'^+7.U-CW\-2V%]M5]'>=%+ M>7Z:3'1??30X2KU\H4P-;3]P07!S-1)&4G@C1U,"%D0Y'T5M7T^"2;AP[D:W M6,U]NW"D1E]1'SP4.K=,NUC'8$=:?![74BE/5SKJ<-K3?^'Q$]E#IH@&\>HX W38=LZ'P1?,ERE0F90TAUO#U+=4D[ M.5X0-DH;I0$V?.9+$7E=?-@.8W.7;3U\\4L%@"!LC'M&? @VYDN30P=H43&J MX8U M?H%C@("!WDPL-A-[P5LE>\-RCX'S2,1'1QR*@#M ,7U+:SU_S3UB'!\%FAI\ M=#E]?G2$8T5_@E_)8F0U=0'J@&)2?GPA2S%!@7Q6?GAV0G?R@+1HPT[A/(=1 M_$Z7-YIFY47.3H-_'')B2[A2DGSY@'!R:F[V-8E2YT6!.>E%^DZ04FY C4R? M6.U([X%R?FQWVT4O+Q\R%:J?YY$S%+ 7/U3 M^FS$>6UE^W.O3[!0$TZ&6.ML!SM'7A>")$\(?Z]B)X%K=,$THX"W1U)J3EG/ M3&)A\3IZ:+]^P410.[P\D#115B):1SH=4Z-5Z6@;4[ Z5CL3@+\^>F1C5Q0$ M-#:56$)645]%W)+^'3I-M1PJ$PG:9I(F&D0 M/5]_=GM*@6Q?^E N@-)!4(%G?\Y@4X$/ U6!25E7@==:68$">A='*UH:;UZ! M&W]A@9ADW&W[+A=49H%)@)\_2X#^?SL;S6IP@10U34I*6_-/4&J+0D50K'B:;0P[;G8]59]N+7R)@8M 5TI12B12"SS#2)2 )D#? M-FAK U*_7IJ A&^-11QM[DF/:B5Y.'K1(3M"(WMZ17 [R$F+:D!O'(+20)@1 MU%Z;4,>"J8!A?&-M8WP/:&9QK%N9#2U+(E(L&U]\CE2*2CY\*DP47"M2E501 M8YQ']2X?!:&!K0C6@GU'@FBX5#ML\E"]5$%,:E]";$R!=WY+2F@C2%)=1ZV" M:WR80$J0'=2T'/,=@55'U7^0WA<<%Z#?]9V941A /^! ME5*P;8E_6WLNIHSG4C< =NR'4@@P>"CFW7>)%NF1O8;01I)TFI?^MN MHDE)3EU8,DXD:TA&%5IP3VUEQD:O4+Y&&H+X7!R"&$J%/^=1)$^]6EEAK%'/ M@ M>#%7);+II7$WR6%5&-1_S2>9I=EU%<"53'$PS@B=EI3SX=^1PB%,<5.Q, MACSD0HY@Z$BS.ZE6\3I @KA3)&1Y:!R"56>".11"TEV4@EQ&3D89.WY[#S5- M@FM98&I+9:AYDF%E#XYMR%-T62N FIO8TV!7H+N4U&!(DII?_ML:W]6@54\ M.8 J9UJ!*&GA1FV"+4YU>N9\$55Z3V>!DV=64VJ!NU5S9D^ SFIW8#9N'45( M/X""7DG@/3>YJ1GEB1()7-G^0 DIR)6)O=B$*7&OV43!\&'UP M=AI]0!LF?TE2T5U;3/5GM(*76[:"3W@R?^Y2KCK1@[N"^V,/"D&"Z$S^5BT\ MT6$:<7"#LE,T->!=NVMY1Y9PM&AB<6)\Q4/L4T-YRD,$M50X;#-]R W5?-/4D;>0'#2+T<_H+0=TI^.7M' M?VPT@0$%@Y) 6TRC-$YXT0+6;*%%4#V0-$,ZRCBT-#QZ.41\-1<#KC50-7@Y MCS6#-4!F/65* !$T,#Q' #UZ935?(Z [; YJ;GA5WCU5($2 FS5$-8\UVH+( M.^X]QSD_ CLXQD-I.9("QC@Q4BM)@#W_@M TX6<@-D4 GP.\,Y@_MT:"-/]3 MRT (-ALZT 'G OQ$0P ?*[,YOFAAA%@@%SX..38UR#N&-1%#!#9NA#%2IA%* M958D_SM.92@ H#D09HHU5!^)-62$&#IF-3Y&'S_X-2,V&CA#(+ U2@ ]>J@- M*#9-@5E/53P4-@ $)$=6+W@YQCG4/"P['C:H4W](T3RI0PLVE#FP 9Q/!D(D M?7:$8'FH.7J$14A[&B0;P41_A'LA.3]# /I5@X27 86$;D#81GL!40**A,M/ MC82* A):D(1/9 DY(#;>-2L!C3KX ?L%_S3Y*96@:1#KN M-KN$%C9835, 7(33/!HVLSG#A)$\%#N1 UD$ GE"-3X\B@(B3AYY!S4/#'MK MAC7T'NA6S(1I/,Z$L&9-7&PTT82I2KDYUH2T/QLF7V4I+U@ 6W95 #<_5@!% M /D>%#SN/=N$UB%-0K98J W@A/Y,"#;60($4 *2A'M:! HR'4Y7 MD#4, -------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:20:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26029; Thu, 25 Feb 93 12:20:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06397; Thu, 25 Feb 93 12:18:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06309; Thu, 25 Feb 93 12:17:22 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:14 GMT Message-Id: <2156.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: second part of document >-------------------------------Cut-Here-------------------------------< M(#?)K;\YTI"T."TE/@J'],.MM=75'0(M/04$ >KU:JTCI@ <-F6PZ#W6I4AC ML!+$=!=!7!UG.]+M]FK=<=/;'S? [7*SW,+VO[URC]SY]M3] 3ZC1XP74O7%+W?YV MU;UU]]M=-]9-+6OSW8?WUYULFQE^-\Q]@S?B+6]KVS0W"D #\-F6 M,*=]3HO3C70D+4T'W3GU;DAI>]-?*1Q= WC3201_4%5WVB@ QIQL]$QHU>P M'! :MH) K56SVWLW'.UUI]Y9]]7-=9?>8/?TC7H'WM=WQWUU<]^ -_;==R?> M"CV]1U^:]^F]]\=ZTW?P!\RX3;-?NMM;G?0W?1?2TD MUE- 'G"'H $;-FH#%F )Z7:HYG]SW-1W"0Y^Q]S5M@(N>B/@*S@ OH"7!@WX M#?" (Q$E@03^>H_;[S?E#7>'X)0M]/UY2]\F>,OM@A/@+3@1'H1[WZIW]VUX MO^#K=I;]W.%F;3D0#7B&0-V#:]RR M]D=!:Y_<2C@+OH07X8:W"KZ&O^'_-@.>3PLK,SAI,%Y(X#: ^ST% I3!N;- M1A\!PPE-C6>[W97WG^IT/]]D^!Q-2Q_@0S@ [^=A=TN_>G-H+SW4HX)&Z$+]N,^"H^@!_ACKC M3;)MVPMW@HI^Q[O<@@2. [C? ET?7G3W!9H""^B'6[W7PD"H1B_4/K@:#7H_ MXJ^X&JZ*B])7MP%^>C/CK;B\/8NWWF[B#(XVBB@2> [@?IO5C8&6&HQ;#$IU M,DZ"I^+,MAONC#?CU/@TKGT+X-4X.AZ/4^(Y 39^?K]">,R9D(MOX@[XSK95 M$R@;=AP-*)P%I39EW15!Q>CW/;WCAU/1<(*_EXG%T2. JFQ9--%CQJII'8465OTA*<,- M/X:ZUWQOTG/!^IU/:\BC=?^- E@%Q5XH?4L7A)Q@7#U*Y\RW-!T3:5O>FS52 M($R_TGGW0TW1\=]6=4, 6YOEC[5@*$Q/W(]UL*A'NP "=:C-T_'?M#18+@-> M!$1W' U[NXFA05ZM=TK+ON%<;7GT"^GV8PV7V]Z3M# =\3D&,=T<0%_?TA*B M I"0/PQ_M44 6ZL@70!>WK:IU8!Y0IX"C '#M!NP>3??M[2)'9=3$)NU"Q*8 M!]/+MFQ1,<3590 *8):SY"0!N"V7H]>WMS"]!)0!7<8M77$7V,'!.]=EC-(& M0F ^QW6_JT%LWF.[#5L=F/=8OZLN]2A]&7SF,T!H#GOS:YJX<,V:U]*;-6(0 M:6L*7CDI#9O?TE/&K#AO4X#"- 1D%,&+0$;^)I;Y\SR;(Y5C])F>6Z>$@K3 M6'DZ)YJGY9AL8BU"= E+B#J=F\NKN_C]&%.&L+>GT$-ODT+BK"UYP#FW=)WP$I@!E[>;Z5K.,H6_G M3?I_7I?GT3WZ8_VCN]5_:IE^.4IP?_0HG: JY?_X>7[\>>A\-'^ -HKJN?FG MS7:+VE0&J7UFG]J485Q]2[/:4L:#K4T%.@\.\4U2C]()0K2>4B_;U'JJ#5MC MZ^KT+6UC6^%+MG:N:JOG*SHI+2MNUO#Y*.V9W]*(@=)M H)Y8+5],QH@3%.#/X(7'IQGUQ%/MXX"1 C.>;6ND*_9*$"7X%+[Z=EY?IZ0 M.P:+WV9-]1$:01U>'DB3U*@YQ%YT5^MP^ACP"@;D"OE7/F]CX:/TQ5)2G]_Z=M!E/-8^NP7N$=CIJC9J4ZHB"5-OM1C>&#K6GZ(3X M#MX@%.E1@<&.L:?MFX::'KF;[K=ZI+ZBHS:4^T6P!N#EG7>:#N8Y<&QT\6ZY MA^6?A;4QH\/FHW3FQ[+SV7C,4854%^ZDN)<.1U\&F[+.AZ'_Z7])-0MX;EN<:N(1Z8+/M?KI6SBQ7!E.#V>>K2Y6:&;,> M;@_K7WJYO5V%0K39ODM;;,W MUU@[P;ZU?^47NP6/P3?7)#R8EYL[[EWXLHV%N^H*0N5^O.OPL@A2?06D &L& MF!=(L^R<.8;NF?OGU#O-C'#0U(1[UQZW/.SS.HX" M4 E]N)_NE:\H8/47;\4?\/)W1!VT0])X^6<#1S?G)[NYGEV/!D@U'\_(9_"7 M=WY%2[_ICKE]T#\@UH$\F#=*-P8H 7[^F/_G^'HZ;;8[Z;ZA=JZ_,^J;NYJN MFK:8GY <@[%V9]$%_=+,^@3_K M$_=H;O"A\1:W0@Z5YS8-?""M=@?:O=SJKEZ'XCIB[(YT^P5U-]/]^RGS(&H- MGD\OC8_!XOY8N_#F^F-ML\/P.3.>'9R3WHH\[(ZW#]/Z^YN.F<_F%8-O2Y"# MU6!YSR[% ^CD_-)]=S\K0CM;SD93Y07Y59[0V]V N;O^MD/EC[S"\9[[! MQ^^XO"!.2LOHI7S=+L7_[R"JV>U)A]OM=S>O'13B\W7%]!+[W[ZS'['IX%(?40- M5N?KY?S=711DV0B[P3ZAP^KU^DY_1]/P\@--#4F_\?"Z,J_9<-K1H0[>O9O8 M0O5V#E8?Z)!A,)V;8]#I^;V^I]\3FW4N_UB+?U9B\\V?6^D]_?M>NP_V;H!H M'FI#]+?(]'1-P@?/<8>RP_3,#Q8@+_/ MI\M!+.^UP^A]_59^?&#Q>'S$GFKCY1XY3M"[!>7Q DGN9 -U9 &W4&8T;@ @ M==V2Y]/-]#/M94_3U71-/F;?Y&;V-XUF[^04^^M-7;_9^560;CV6D%@7]G[!?2U0UZG6_2.>5+0SM_2V6NJC5#'YSR]1-^>>P<8>ESO MN0/=+0 6'HM<;YHX%W^AO^_;;3]]GQOJ@CU9OI4?ZM4(@*^V#^VT=,4MHQ?7 MK'PF$RS[$A_AU_+XN<4NOE/QON$_;^!7 MZ 0^)(V%1P:I^7>>IN75?CE:;[LO"O][A["/PX$).UB-$$CY>[9O'[5S"45\ M=2ZAHS:\<]6@OR_F=+I4 *@+\8\U2A_GZ^I5184MQ-_26W%ZWL3W]2RVG<_9 MD^AO=E[-V(OLROMI+J]CU;GY/7B'6/9V(1%?X,?18WW7GIH'^:[YI*^>8_G; M@9\H3.?R87EMDZ+3+Z#[1 TX#W'EO>HYE)?DS'U&T+K5:N?B2EZH01DNN98] M!,3DUSTTG=V+V60V3N[=Z^1($\7^>X_W0+EY/Y33V4;YG5U\J]1"=32OEW?H MTCL?;=KDXFE!;FY\/^O)]R@"@C??TGU<5>R+'>@]2% HE :5R8/SRKDB1]QH MG!8X^.HT$@%'_PB+GMGQ;1?\Q#,]#14F':EY5&!C2QE\.1NM C %9\ 8L&+S M>9+#7ZT"C/PF)PEQO20C5$+]_+@!:#_N[+9_$U M]G:Q4ROLD?G[_M7?TKL!8O#!/_HG=G NTX?U3?MP#WN/!@X[4PU T.N>^9E_-^D-GIHPV]_IDC\<3_F/#T8 :L[QOWGE0I8MQ/?_^C_[CAW K =)H^6Y MZIQ_V;\925[-EY>;:R04=U!_4X'(7D4O5A>O@U!4#>QT$CY%'?$/@[:YD]'1 M60=33@WGZH@P<9&>\Q5 M_U -, ^W7SJ9#8!_"P(\])\=KU;FJIO7J6D2PN@YE9GF3I(7E:M=V/:*,P= M_&)X9X !(!*P]49A"\Q% 2-ZW+^[VSOO3D>02_7-YW1 6@$0(!BPZ#>GL-TX M:D04V#5.CJ2FZ;<#G."4.:1^QK6QGA<.6157>Z=I4T@V%C6BW":G5#.W.>]9 MPN)=9ZLSH!F-DE;T^PQ@ B-^J(*)GPJP"! $8 )( 4B!1 "_T!"G]99_6_4= MV@QK0 &6'4\@;8=!(\@E./ 8V(V_G2:PY]=FRVQT<78V%[6G0\HK"" %. )< MO9 CK;@P5B_K)R<;5;8*!#?_-'<\AM GN!GL!I8"0PFC,*O ;2 M5)* ; ;G7W'-+]?5,]EYX%!V0< S@2[0"& -Y 5V ADUZD 00#"0XO<$6 +0 M_OIO^;0G0(0HW;9>8Z,5?9Y_ZIYJG8^H%]@'Y--D4\0.-[=^7WZEYD,(1.\- M^XY[(KECWW*O!* HRPBHF 18\0)GWY4MVE?=Z[+-Y*Y]-KDRF[I-N@;>HZ7E MXZ&0*>TY9Z=C>K'^=&PI9E,_NY M][: ,3W:'KU.5E?22]OI[&Y_2KS-7X8/%+2RX_OQZ_A[MS1%'NHN9Y:;P\*! MU?1Y< #DW6K-:)>P.[/A!C=KD@-873WOL6>&N-"9Z7)ND[L(VL$NI"=,\^=5 M[)88K+KY7TVOI1=I>_NQV"!HWJ]671;PUI>> PWB^?!N<#3>H,%.FB<;Y.W9 M_GZ#R[;@X$5OF/1MRZMA@58#Y8<[FJHND::Z\_!!^)Z#BXCH(-?NCO:;0VW( MJ\ "6$#LH 3-OS:@@Z-!]%!SWL%:7T5 IE.9,_5YYB!IK$''P?6.CZ9?T,3) MY7YW8#];G^UOA^8;BN+]['*#8,$D@)7.H]9 0+55A7Q$(#:P'&H#O!4!'*5) M]KQ\Z$'1H)H/UJ<>X!5U^)B#8;FJ@OR/;+=+*WM(9R_!= M"*& ![TE763//H"0$\RI_X)S([WC ^BN3H-\&Q$:UBH#)4*!H)H@1;C?\XG- MYUA" ;7'6F)@/O=5BP[9_9IT;SIRFE BK\:R&^LU?V!O,@$3G8CP=-?/>]>I M(\8_P 4M1)&..;AL2PS2\42#Z#\!GPW/KE?68__-]*I"'S\>GQ=0!A05I.,M MTV)X;L"LG.PN3F?HP]HM?BYOV;7P1I8.-O>J8:/-A4QY*PK4W#70["#)XP^4 M<-HV$K\4( K@"I $8 +\O48$63;4G8=OLF?\Z^GIZO( Y*CQ'VKC^( :& ND MYTZ#?D(-X'M0,:A-B^19Y3!9G;NI'J/0JX:]0YKLXX:%'CR/VA/0E-?J$Q3N MA[2 =\#-VIA@/N>KTXIX O-J$+WK'+O/8Q<[^.#AYVJ!\[]06@,01AW;@.M3$K5-NQ M[0AWD"(9G]ONL<:R6T),#-YU+"' 7+].1N>0\_I9'J1_",-O8;V0"_@"#%% M]49V]AUH8)L-PJ:T:PB.^( VF @?H"4P"(@9@8"= ?^"@0Y+VH1M"C $B.A$ M],8(!#Z2X6S.'!A'L]AYUF!K,<(=VAA0JJ2TJP8N"Q6#VC5P(*$-_X!6"\71 M,

8#F6FS0=*>=ZQ#JYKQ_B\)182&)O,>RN=N< %\YXP45(*O05;@8M-U M1<0."PY+X *OEI,2I SR:31JYS1)@2)P$)>OLP&F]P1$@<"Z&.<&(9CT 0+V M;E@G!C\C !MM4Q>TVZCU &!KE[C\"BHMRY86@@_Z! QI,#@4 (LP7M>?2PZN M!GQ#M\A!0,P/O8P>&!;&!C!K:(1X#*/C*$0:2 HN!QT!\H -O&3B.0+"- M Y^!FK^M84S 3:0 \!K*#7V!-@%$T3QP%$@$J ($W82'D#2FFBR0#-!E<$6P MT6QV=4#NH$^/[Q>["P(2541U[4!.H%2)47,^G!ZFO'Y^0;\4 #(PZX>K2ZF= M_=2#%#[GWRTM(=#.F]!-VNQ]LKH9R3*O=I@^Q!W6Q9B'[L,B0!- @?CXFA]F MV<8/S,+B&OZPU;<]!-3Q#[N$A**!G&_H66'TDQSB)P $@L-237GC;C,9+ 1^ MY"""QC[EWK,BV<=DT+*@)S2"T[TL6T=0)G=[H\EA^[A[(\'O7K>/EB:#0PGN M^T*(4"$4(K\O,@@3!"S,!*])202;(,9/X<=&8_BUV9PJ:YN?X!"1&X@DC-VI M'11%!"%X?D%P05-03(>A"Q;"!;$=#CO"8%:0D496,P+0 M_&Q^.#^=GU5N$S@6+ LB"7]V@;FFX/?O+9@" S&!<%M:D3#(+^&EH8T)/C! M!0")++_"8-3+A-_JU M!Z=R^L(*7H)POP?*\]J)[J67HR0K^>VB]EU&8Z#[RO7(4G'U!<5. ' \- ,'SVPVNJP\N<<7!B^ M"3=IHP%N7'C-:PYV M"#UP<$,0H-R0)&"[B2=.<*:()0&]8:OP[_5(+"@R>X2&H6PFR@30H[8V3:';38C@.<0=,A_&QU" MTDJ'2+$C8.IPF:BYR\UQ"#%T[T0R(+!N%Z@^A!X&#H-OO$-1X# 0>"@_Q![^ MO(B'S4 XVA,P"A@-) /.!)J'/$4#XH(#[99 K!Y>#\&!VL/9W"P0=0A' Q_Z MAN!X6+ZLG0?N A<$# R8 6V'S\-LH%E1'BA4+ + #]\,#D1(VM@06-.CNQ\" M#F!KO+S]H2=C5O<_W/HEYDR :#?GH3MP?3C!>2HF$!>(38 &HE$1@@@V-*Q- M$&.!6D7NH3#M@@BGRR#.[SB(!4&V29L&Z0-$S&P!(< GXK :0 3P#K(1- Y<5A"P(< MO3AL10"S8& N+Z*QP6 M GDH%T5RHGU/@,4DRN]1$#IY[KW3(#PO]]<_$ '!ZFQZ@SR*@&NO4/CU^_"! M_=R)!SW9WJOO".3:V]'MV3"$S<']'I]N?*BUV_5E"_EHG)M-(9C'(6?B2R7V M$W=Y83SC7>^O3N?;FR0T]O2%XD)[86?/D$:@TQ%@CYSH71N&9@NO*.1 M[UIX[L(&(+R0[<='4W"8Z'Z#4L,.'K1@-C?;X^NI!P>&[;Q1FE=Q0$=+>P+H M[1)K?KD)8WIN@KBR\6P., M[ZIXY+N;GI9Q-G=F5"76ZA1-'L8.W]<(MM8FS"A2\6!K#@)0^">Y4P_FYBB+T,0&XY"0Q(A 6]R%Y1P("+J#@3JMT_?54R7* M\Z9V L,WX]UNT3-*&^49[ )YE+\]HF*PUKAHU-4%!#L&>+F?H69F,&BEZ_RA M!E%V^3L&X9[-B'?8&QVX&3]V[@3'W^+.B\,8CGDTQQVBY4Q)F_T(U>;7_W-F.-Z#KFZ_9Z:)VE<:BFP;0.=AH%.@- M?PQIH;V?EY50J;A94^39![.-L#I2070NE <;X$V<^0)[HSC+6TW/ PA0A!,V MA\!M>:!P8UU."HCE\S8.'<@ ]+5^'6S.(1?F0T4TUQ)\"8&"X5?@PC9GA*V% M"\H Y[EGXK9NB",#I"9&=3B*6I'77S810* #%!!9#F5".#5PHAB136=/A*0I M& %S9[]SXFE0G3C;Z^K]!AN*)46[P,R1=1*2TP$J!/>A%0>+/L6Z8W.Q=TCQ^QT: XN*M#BY(E+1>.@,C!0F M#X. K0B[8P'Q'5CO&B\D$-^*146Q8?UP>>=8O"OF#]5^W,85( #1%AA\X]P0 M$&^'C,= !]:QK<@$> (, 4J!*8!D3NNM0%=!=K/#IN*LH,@0X!8B6P$!A\A ?V%DV(HL5]'VD1V4<1I.04 MD@I);I6LP6H1R_:2V[*]%FN((,'MGD@P)S>!VR&RT3IR/[F4X&Y1B.A;W-/D M;:HR*$#?34W0N)CP0RXZ$1U^4<3>35"1BO@R!"[8V&X1ZT0.VU30--)=#"3& M_,*+;T2OH!PQ@@<%* )@ : 3P I@- /!9 7[",B'=U$%IRD7R&0,&=@[/>I M.V'H;@TXZ'P7?=E5/#%^-R&48%WW;Q0^=:HT^2Q M;!!XASNX'&K#"F=,_-!) >-_-#:Y7_I/?!ADI!F"$4M],,:C3P^/-0$S]*BE M\CAY/CH-G#+(.8=[J/ZL$FF%8 T"GZ%0N+?$:]+9&FV0*$.(@09P_0=[%.:! M[IY^WR-P6TQ-,K?9<_&)!J^#LJ*HP'YH8)C+,\VY['J,"#9[G9&PI5=_?.6- M$H&#ED=$WU[1A,<3V-M1"05':0 =1(5QF&BFV].X!V]\JS4U8VQ.%LCJ\^>A M-B)$9*$U)![O*S!+] #R(+UZDX3_G68FFX=1]&UI%!N'CD"]4B0PNR8$!!Q6 M#B^!-X%,X-VQI^@)3$7N':%""@ 58#WP'HA5/-(5U]2,"KE;&NY1F,:\V,#- MYYJ0G,=U36 ?0AZ)"SR!X:1CT>@'US1J#A7U*-)UFAI$\2\8G6N?ZAY]"LJ M#^$"Z\#%HS$2&ZD 2""6 D^!08!48.BN]38$^.I)$"V/%,3(H@5QKTCWZT,R MV3J(90)3C9RBLZCP RWVV8A]R+WEHVG1^2B_N6^@D:)[5K888O5QVG=]_+)E M'[-]W3W:8O>QO$%6<\#E%L5]"D'UGEI@$T? D[C!WS9Y_,"\ !NM_I882-NU M KF C4 K&X<@3G-\O"2&-P2,PL6,FMD "5)&HO8CSXK 1 =:+>4$HX'R1,I!'/$JJ0L!M_-'NEBBOI=!JE^"\8S =9/AIBCZ!XZ MU 9WLD9QGHQPU,B98]GYAO9Z,CK4'$<@9T;0J_QA&25ZFT0YX-L0#PD9F H@ M[QQJ^CW:'[_QEJ:(%#-F,:QUNT+8FRK$8?<;G-!1'.F, +T:X4"/OY>;6R:" MU7)Y]C\IH%AQUHB\FZO-"2MX(D@J9'IN]@C#X^6]&F>#SS_H() .IYC$N\#M M#^T#?[\W35ZMPBBV2ZP)"3U] SG<'6(MUA@LE/4)AEAV2(&/WX.O8J&[$*F MYQA[M3Q'I"LQHA:SVT%<[W9W9;GK)&RM,PEP9"5F,0ARF3WB'=R1Q3AK3!3J M$@]ZF4+"85[-.0B_D]_MAWB-I+S)'.51%OF0=C\E)5!.* &^Z2PJ M..Y;)4F19/(Q'RD17"$V'Z%L,A^9CU8$,3%]Y BZ%CV"V,?8HO9QMD@25-8U M) -M#TFAG$*0M^AT)")*$B&#\4@D8DL2X8<3/.+H!'F"5QSFHN_&N:@"S$E^ M!FB"6D2?I IPAX9=Y*E9Y9B"'[^BX"N/5+=9&_FQ*_N(V4HT8O/0NZA(7$IR M!>&(7\%.85C0"1 $6"#*%]&",KKZHAY1F+A_O/4@$K^+6D&'6B$Q,'A(S*N% M)<>2=\&L)%J2D7843%+^%YU[CD&>#61PT4:M'"(ZZ"R#@#0L UY2,_A)=."% MT@I[B#\)WW[OJ# 0E$/"UG8#M,9B6<#P2$EK1&VP]9AZYSSV7(:O-"E,A-7% M_A!ZO#/?V1.(35?CTQ<*U>Z0P,2EW7R.(] X-#:N!BP\3I@]P) M$]]R]$GB'VYP:)=YXZE]Z=!V9;FU7=K.EP>%G K,_^"%1LHZY))0R-@M; Y) M$U.6,T"=H34Q5)=-]"R:(DF*_(&CC\XQ=I^NAYK2&_3](6KUJX2A,T]DA M^N) @;DVGF9O.L )XB<>";6&8[Y8X\0-*WDBBXJ)R\!C16RKJ M#*6!#HZS8OJ249/ 9"M2$5U^4H"KXNJ14VA8NRFB+F-W33K>Y&;--QE'PP/@ MHNB.#K>H8NC1LP&^?&!"'B,AGDM&)0@2CK:JQ#Q2(RF5UDBBX5,Q4SG"I"IV M*C^5D44@-I?ZN53E9W$.*+6.5O\=ZU_CR%+G ^RQ. M!B.*]T@AHC[25WE:Q-OX)--J[0.6'&M1AGBLI"$:))65",D<8FWQRT%+6R&\ M**N7TLJ5(/EQ AFO(-RD)+.5Q$4E8OO1*O=^%&?%'U^15$0SXA>1>==?A ,T M+/N/^D7P(E-RO!BPG"/V_!"+W!(I *;.^G?0TS_B*U<(IA'\(K]RO_BN9$LF M,0.,;DPV)J[2Z?@S1# B A6,&\C$HZ@.DF;VD=#U*(>V3\S;>@O1N1R>I[.MY$J^,![W<'!LOWAA' MZ[8Z5[M081\SI2@&=+?=)MU[MK]:8)E0@WEYA*WM(AN3ND9'89Q/XZC%XUEB MZ !S.[IBH]X20CE,$C52,X68PK1(X\2 OD9ZHQ02,R> ;SIA9FHR13E',Q"& M$C^#XT(^I.IN[>?F.3[(^,*6Y+O5'L:/< D(3CPYU@VA*.-+M.)N3W391J-Q9;/.^@A'85O6D2BW%?*^%BM)$4& M?;*7BD K9?<22PFBPB82Y2"!(\7RI?PQYA.F'%.J%,>.+$71(3B0L9A/#,5! M"[U_OC\OGS-QMA>GI(EQ!A>/4!M&C9O(;S-_S%,*,)F1!,SCH>%14!E\PVP0 M(].*8LV83P+QO"C!!"52'N%H%DQO9@;S#9"^XV"J,T-I9<4F#@W3&'E4,6J: M,)61@,=DX.0QHM?"Y!&^,/N*X(8 H@E06M&_7&!.<.(5"41/Y0$RASEAVV&V M&GN8'-)&+V)]MYL$HPSTT3J4F]Z2R2;-)[O44H)L\&'QD15"&: MY"HYD(22!22A'I?%?/91'Z5]U#[8HG8OC,E]3+.1,<%I/CD'7;@OC=FW6V,2 M&,V/$LCAXOK1)3G'_%8J%^^8X\KV91]S#G!__&0U( &#_,(!.,*\M.8NN&=9E/HU)* M[\Y^QTM7)A,/L\>S?-?)YA)RTP&V'4%/$>3:R\VA[FQ_Z[]=X4N/LR>:$ZI) M]=Q[CTGG(&* 8^8;>OH)(1UU7@'7GG9.G/FU<_.\!KIR@4M5G@:0K/"?Y%#^ M?IY\]LPWW9GME"FV3&9Z# >7YT)_IMN0\5?(0R=B(AMU0D*YG"<13:>XW-"- M$C61W U172#-O_F)[%Y:V68"G$J)XMHF<$-27)48#36'8LJ48ID2!6!V="FB M .27^D3ZY700/W?5+,L5&^6.6KMWW;10F%9OM!'.2$ #<\IB)/00SGFG)%>^ M#Q&;B\K%YCGR4>G"[&7",".;P4@V29P3KMD6N-UD-G&8?,I1Y<+OL\G74UPF M^!J;MS339J227(C$A$=^I72 WXW.XE>J9 &EC&*.%GN5M\V#8FXC>S?Q*U:V M%JV/R,HOYG 3AUC -^Z4=TY;9RC^E^E$G" M'Y&(>$QA8&23!0BKD2[F-!<%$"6 M%S>!Q8@I !5 O1F8LQX&YJ8 0S\')'SSUHE^?&X^*6.;!,:O9.00JX*2S'7. M<*!]DTQ!YK_2VOF4W 2^_+:=4E+$&9^LV84@WN4>@9A*/=#^YH5K[]I(?.HO>=E/$) M"94_YDG\G-=OV.EE\,#E+0^<.CY'W_B.P1FOPPVJ(%6,<3SNG>6M6RBM,-%- M!A!K+4/JW:+0")" ;!($ 7!>;ST(H4NM*"%0) #B]E8"#C8['0WK?F?C!'DB M'#A\;+];&LDS>0GB[-B-TER>'S]>8722:@G/"Y-[<+1[JHY^^R7]C& ]M-)XE_4\^V8WEN[?GB)#(V MX42%637/9?L/.O5>^!M&OUU.\^>IU@2Y_4!A*1Y@8:)MT?> G_,OU:VE-11 M /]J:0*L6L<3/4E1RZ;L*T^9<<.>W[U3O>GM#'K:];J- +V5 '_LB,?+;!18 MZ!X^";[J9$53[QEGO DF!F9LBT^Z)TVM6*C#8U*"VPIT@30C7_"R5>>U4W-: M%J^%SD_?UO&!MP;:/!+>(=%\/KKNI-M0&NG+$[GI+&=S,CZ97:Q1YIEN(T5> M\TTB6^J1Y!:&78VX.EIEBG#) /TMZC,^Z)Z30-&?">S_H M!UEL\L\X6E]2,C?WVPZ./#^8MS27IPYO3H.F.W(V"]]]UT]]H180R5CC#%S> M]@B7/TZ"W%<.<4G\4URRV(Z0A[L2Y*OR82?VPS:H'"?LK"=B%;UIM+2.(\&!=/G^@0#BY^)Z(#83FSPS(3?6HZ]! MTH:9YSPCWKKPT!?^/(#&\ H'ACKBY;RP2,G?8U[6N_YV@[P/Y@!4G(GR9 TH M.'>%+,\[&A=2 XIXC ]Y#0V>*DW%)@L"=?C)2PPM/X-%T$^"WN4S>AE\+ GX M0)V;'Z7 S>T&J#EU?+^!(HERD1! H$@RT*$#)%_>'!-$MS6;(*$3$QKLI >^ M+V5^$S9;9.6Q,IF^VT7J%<.,C\@@!8+R)FB)/!**%GQTQ3A1U?QO$&H^O'^* M,,.:-H$KCC>2"5!Z' *D'HV*#B:_)1SM,><6M)5M#2L3D[/&$R(_60?'5F M-LZ8RTV58'.S AF0;&/")=V=<4SV(T^MURD"A"(".[.;0L6<)+_&V"D/Y"*J M #%"RTX^YG81VNG2DG;V*_F+XD7SYK53]QF5' @ O3AL6@#VHGOS2-F5I(DZ M+ >>-]%*IA7Q8UG% BX.&#NBFTR39>O&%?')S&]B"L<+HDSBF]E/=$D";7/^ M&.-U;5!UVBEO":BYZ_RA.6F3/\CD0&(-<.F8$UPJ&0N7*E"[$ OT-,@!M%EN MZ%R&J,I0VB\3XLCNZW_"U%1!7$4VVHKQ.@>8NV!>"PYYLTQLXYKSZCF? _]\ MUO!U];L9$ :MRS#!TTP*1D6@&D\FG0?.+7K0U,U1+5MV_+WCH(EN#E!I.BTT M>%Z6P]"\Y9$P+/K*FTT*)C%"^,+_IST1-=>+3-CY/L^?W$G*0 [2T-?Q[#(6 M\3B&T0,[9&%TG#EWM%GN&<%M"T).T-ON?9=_4(Z.1<<$-$^.YC31HVEE2W!< MUW2:-1\:I5*S+0"BPK(105.:TIK6V\"'%!K1:QTR)G%[X :W(6#2]VCIQ"]T M([&C/$VIJ,SGIYDXU%XV.4.2#+LF#DBQ=J._J55N1YF4#L^P(YD2JFFFE&J2 M(]T*A,:1*"T-(X3Q)'!Z_;J:8\VW)MY1?3D(W"9J-_V.T=!DX#044+G6K &: M+XMY*PK+9A!'Z=A6](9*'L=T=TTVFCB4LCBIA&QV?BR5N0W!I@B3L*D?O6&Z M0_MOHDIX*)"ST.?H'&VB,R.="H(B)FKSB*G:9(IF*=^2"4&B'+\&MMG3]'0J M'T&=+$0'3J #%QCY:H@.)(.;R!&6=ND5:)R*4-^<4?6.J M'Y.((E'4(1W3UVG'/(F6-2E^)$^UPYN-N1@,3'96)[&+@3F>FG91C)AQ_"(^ MUD9^=U*C*""3X-E&;$H6,@F0V4XJ &(Q,,+ (H M2AF(@3G#X-<33\IAL_MQV/ <,%LBDV4DOF]4RE@JY".7PY#Z,_PRS$F+2UM M,3.0"KT-I+2",TC*#"5>VC*CL+5AYYA3F*;\W,HA2!&<*L\%9X@S8]@EA/NE MT99MT,ON:+YP>1=JHR=2^ 2<8SYRVO"MAL>^[I1TN43\7(;S76?V3,AA M 6>#O#U J(VQN]!0HZ7Y19=J.40L**%Q7%?^U!=>_L"@)LI1HYNPPZDX\7'X MYF1T>[TXGC0SQ8@Q9-P5/?%_ID_Z19]4B ?'L\XI,WVA&%/4H/MO_TDK/'QV M[@)YSD%^Z5KO\2F04WD&EORE>3E[YO).>(B:XWDJ($>?.KRO5 =29$KX9'T^ M(YN YL&67N_N3PC>ZI/&/M6%/TY16NX3"I H163"_*Y>D8$LFV>N^7>[N]7Y MXJJ&%,=>9MS.)S"WPVK^!I-UZ7Q5/,#2S[-P)#)>F]TZG:=3K-CHN7!DR M&EV#DR $'[L4VV@N3+5']N0!,UA\A0&FIN[??^3,C% M165V/;UVYH33#)G_C Y,/UV 8$'6HWLO %H#7(/F)]-SVL^OIP)TS=D 9:L= MTV9[H%,):$=R [I)$_4Q.@M]%U"L&HNM6IH9K1GBU,"%^L+7) F4+7KC?(NF M0'5[5D#K6A3M1.J/OR'NBY9*K]*=EH==.K)@F2JR DW>]M M&71ILST5:L547A@:\H+R2P6(W W%XS(49I-&Y8;V'4N!O=!?J. M@1IK L05PN>1T#E-N8XB.T>!T%#.YO#03UG #%1Z2,^ABB(:)BY4Y>)'[8;B M.;^A*%+&)CJ2_9EY_'.^2*4; L0YS1YU0NH3[([.-2.8E32C(NO1L+;59#&. M%8]N5SF^ICWTEO;7/(:BQB2DADHD8B,UV5D$T&R"*HV*BLXFXN_TCM8C%8?B M0XV8=;\B:0]_)Q;ALK*7"S(!D1U9(R M*W6(#,DR9@\13 J1E 26^^1]SSRA6THSU):!\T5^X)AOCKR?(8@J/FHX_ R< M -U$I,5\9:A&CCD279,V_"*9GII0#4J4BJ@2513] =VH+U$E)!H@)KHG-;J! MV#!^"3[] WW-P:E. ZM]"H^2<3:D**]TWNF4-&0.!+( -3\DP,&262:CN^6M M!6=T8 00VT.5)LJO4?EA2I6B%[AL)*J],Y.%60>$+>Q/3/$CF76 X.25>?/(3S) DOU^C\]!=@ M[V8":+HG'H%/FL=C+%!^Y4 $EI^/H=3..W!O_-GP_")SHTXVFG4Q?6?[4P[0 M$\R<>\\-W\W.C>?ZK+M-&1@1&#K8)'^/,4H^O,"I"=%T2U4"GT-N=BE!DS;& M&E^A,3OD)&=R:DE,^S'J)W&CQTR*@-F4^.=7]3-2/9V>,Z#!:JWQP;A*S/Z% MZD1] ,)C70+U,4=9/4.8ZPAW,CSOUPO0NM9)2]-A^6B<6;WK%KVN;Q2'!%'& M5G=ONCKA),LF+FIBXWJ2WO*:,4+:*:.0#D#J0Q:">5!SPS79JK9N78.#H?-= M[P)II=3](:!.40@"U#]83(6>_T00X*?PO+H A0)" Y%WH;;,Y)EM-7IH8:,) ;J'M;^6GE<@.@=6C*LI3X6,;U,J:(UP;)=0/>;A M\LQ_P\05'_0N/W@[#8/V!,9S@,M=7H8/=X 9H*FU[KJ+507B (R3LA:ZC*7A M 2145J*#JC@S-UEB7=0%1V%X%L X9&B2L=IB=$^^ZQP$=R&V7=XN7G9>LNF9-NC./A!OND DK3W([N71<&7. M$EMZ40BU!GYN]Z@/5=U%&D-I8+6$0!O22;A?'4%*Y_*E?<7;78(2[:?_B4B] M +,I=L<6I>0R?/FSB1RN9$Z 1Y7;&OE4(OD=19_V'(&J0$MS(C@3/;I.M.>= M+KV9;I7CWL_&:VA>?5UV!D.)I$I*H7) !X'Q/$_Z^=R47K_3 :2HN49ZTS_\ M!01JR9RZRR95?4@:@ <*B/:IPL"7'RU2#/ H/..M&8*CP[@7X7GPH.IS7<#DAAFMQH\S=P=(%]W,0UW7H^'&RV6VT"'%<$ M)XFLRU66J\=TUE??BZB=*%N#]M(40/:SRY#SM,XM"LVK(5,#IH#NX,D8_+6* M'8*M)D3N!D]5/JH0.99V(J-VW,M&H#J50Q#E+-5@>Q""AYJ;8Q+5?"H@?6J2 M';F<9TKQ*)N2?7I./%XN* VCBQX!(N9FT,E)G=MH-NJ<%]) :IZS=+EIE+NR M-1%)51EE*!\5"<*I?&!^*GN>U,[6&R4/]S=:X=!&. M"P6(1QS.:RPUG1<@2%0N(T^D8K]&)0MSD@JI9)%6(P&=&T2F*(@JZ$,3-!Q. M7Z.BUM==96VSM(@0!5;>;KX;@1L^#4!2B_G;G"%6^VR(LL6$9+,RFPI.PRUR M4YF;Z;UOZB;N]0;-*T.FVT*1;(YB*13J+;E6/"+6;+:8W$HUJ77S66$2_;]* M42MVHX.8J%5N,7E0734RR^0)#5C]G5OPXX=__">>)8$(.\:&:H;Q@D=2K=.T M5,F;%-5#*7IS"E!+)0+<'], @;DP@'[U!60TA<"6YPH('[\$PE%2DGE2A:,M M)L&;F!%-G%*29"A/"'C&;+;2O/&P[%>R+0L(>L'](]H(L+>#W8$.FD=.6GN2NE%H:K/P= M":M&&D!,7= .-?<"8F;^URRF7\ >; LPQ_:_NPETZY29"02E:9;M8M$@H.== M[_IUUXE(9/@P#6N!6\.&)V-S%"4USR%Y25(AWW-,(9 ^*J<*$XJIPH6CT/!+B M8;N9;[KD)YSU(B5M;'Y:[]9TF%>^995P&$=HU0/V8$.F D+I*:!P:8H-K*H= M9"]J"MF](4,1%TC>FIV):5I MOU"&JM#A'RPQUBBB;+2>\T9_>M:*G4#U%_DBG;+BT=BGYD\;J ;P](DQ?,6* M!FN/&U %X*AQC0,4>D8L>IIT"0&EP%ZQ8RJ[@_'E+U6,9A_ ZI+N;GKB&S4Z M8PN?E[[=7J0NGHF M3:(/.-XKP)UK%Y4$(C:J[3!T< -W-:E:Y!4<[=L8]G- MZ32(O,A\!O+N*$C]::Z1[DBRV()-)%0HM$-=3,DR9+$%<<+)[+$3)BN3)1RU M !>JJD/5Y^S/W0A)>PPT$,)\_$%$8Y/N@6HWK=#A31%L?3NM;%%V\2,ARLUI M,&N/4L#::@50&5LME&[PYP24)\JW[%$VV[AA=,M>8\6-C$)#:QX69$@J4%'R M:X*)W]@U'1\S'+M8Y=!5V+2#X* [6I4!Z5D#=!%. 'E\;<&)[#I3%4N0[2EJ M1B8X9T>4+,5/)=L+-(F@9W6HJD(#)#'"R_#'@QU"9@V!=L/T;.1+H<@W[ 0J M.+)W^EE584PVH]H+U.*(-IZS+]F^(Q9@/ML"O+7&;?)JC=FH(;SN9(K::*O= M @7*49R%$\ L!K(&U#L/?5U"8$/)>IOZ !I9+"A9E]S9-5!@6^6^/>0U<-B MZ"*KA5,^;$CQ[/B'5<,:[(ZJFS5!X>.T-1=5B(N6!U>)=#_QK/[NSOI_<[/V M])IT+=H]*X9.&LF')4\>"8"L3Q];QJ)JK/.JF/(^N1Y.$DSZ.(8YP _C;NV4R M8G5S407AH# -"V<^!-1X#3NQ:%IPH)HV EA<,X]BZ$J7GK_(7#QV[B@-O)@0 M(_&Q@=K6V]'KW@H_+:[539NM[5,?I^2.5L?I.XO&[A:N8->&Z]W/K,>)_;9)/&4ZUF&;)0&')B7+>C8UA!IELBO'EA-^>DZ!9<> M]"BN=3%;*,:UUL-&U84*%=FSZX)##3;T/>J*& 4N:/%KBJ+66ZTV(L61S=7& M/5VG9,A>Q(\A(7C)G'0I+G4E*; MU*@.5J_.;.IQ# =)F]929'%\C[_/9K74Y,F@C-V!U:Z0"[JE+!/RWA=1>[BN M;^Z?:KL1HW#6Z4;&G--(;7RNYUH.*$!U(=N?9==>DZ0V[UJ*WX!V)ONLN'%( M%S&VQ#-G[7PVWU-^2\?::@&;N-K>9:)5^4GU=(WN\[BUMM,YH:3#T. M<;*(:+HK9%A.8%MT:T*FVTAO&CSC+&T6=*=Y?=B> 7^QUM;LH2R0?;KI*X8V MULB39-@CK'(@LI>R=-A83(V2[5IC?-4,'QF6'UH7T %J M-NAIZM2Z0 ^UIWI"9*9R7T.==1K 3=R6LM7;W B>.@F2J*3=_Q86V;L5':@H3'+I\7:6.0Y^OE]3A+7+DXFK9S)XF$'NAJ$?5HUU31_IZO-/> M9HEWMD?B;.[Q9LG*Y#T21^&,CEC$ZBWPJ5>N93RR;P^+L%(^Y2@UN :\H[FF M 1Z@NMLFG29V4=B)_%GM@EV_(O/>?7]&#:UI((F@T%YNLV@8N\;0(D M :8 30!!:K,T15JQ(XOJ/8FRKD'A; I@#H "J$%F5J6?&L!_X(+.NB-PC=,M M"E.QI4]D9BMV:\C7:M\2%ENX[=!;ZA'T[,?#;'123^NAM]59$B4NEE/'<"V2<65;]*7J+-13LH2K9.N M*\4 "E@;+",B LOF=,'R,J6 $UC2I7?S3XJ!?>WE:!\#2S:M2+2S"#M15942 M,EFEFT 2K!/ !#N?0L%RV%2PC$87[*[6!2NUA>2RV&2P1E$:['@S]MK(S9EE M*P6#A%R7*N\6O GQ \(R$G5 *M4(W$RUDIN$I8FF_$*YJ=)4K*S2WP?&E<(N M_3*BAD.H'WXS,FGU^W.,,GVK?EDV&HTS#$L:)!3697><05CE+!J6 ]=87<-B M,QNYC[N8:S\Q#LMNG,."!\>EEM'D[!G617O*P_Z-%P!_OCEF;BW68">(E<36 M;PNQ6,!#+)TV"@FNU4UV]"YWBDG$*B1V#HF?F\3Z8]V&$=!+K&MMR,: 9.B5 M:G5 _EO\7R@6H)M/(\6Z_@:R3MCY=?MZ6*Q:LA9K#CW;F>+O41B M+?MI+L)L(:P0-?>+9>B) >%I2+4"W0BT&)M*9>CR\F)VQ]AWXVRO(#2?0[ I M:5EL;8"Z31 M&^>9-L+5ZK7QS2K[K!0^(NF QLQ6J(P5LC?2Y0(6,5>MU,*VZNRN/TFUP],A M*,&6>]FV0*&@SDHM_.F&!K%\NMGY'R#/\,?QK-@Q_L:N0D*67>LOUOB+18"" M2(K6RR8B'9G 5N+<5PUN^3BM_X\FM98A.0;!W:I!2 M2X6TQLM\;;^QV@B];-U%7*>CNU8(6^\MFR@3P$^<;?^C1QST92=7&;>CS+(- M:BFXAEK@G*(6HFGK\_#%;-FY7$L.'NVU+)>G=3+N:4UW1$,VG> M&.-\1DKTH'_PGCNJW>B.2:5R*[=^K@DJK%=)6 M:X=X"+9)+@?5M$MF/:6F:I<]_-W!XL\UT'$0=!^N:U>J$-[8+;Q6/@NM9RUMUK!J+6VN$H\?;%-<>]WIO$%MXX&S7)2K@;;-A M>"NHAM"0;;R60*NTD]8:>/&U*MM&W;[V@NO-_-?F[NYHM=&)[CDUW6:P70B: MZ&I]L]F)3JKVOQCA9;=J7.<4.5XJXH7W4"/KR/+":S>VGMF&F^T&R^O W/ ^ M:]UH9CN;!N),4M3:NHR9D>W.^XJ8I.[[MPF[R-I'Z>S5VZ%03&] V:A?EW7PJ#Z45GT>D;="'#;BT/?LU;>^H:;LQK+.QDDNU#3>: M#T-]6-MVGM:V]9:Y;6DV 0FE:UN6 M$V6NOKP6'E27/_CX.Q*V8O5W=\A;8!;QJ_FZG?:6;D6O:$U"ZMEO=;MG*]K* M$[6A-1O#IN]P=DM3\5S.;\.(<\?D:J@V_EF]*AS-!JC_%5@HM+2[Z"(*&1S-<^)U^Q' I]/8:2!KZ]U=Z#+[/6=XB]Q:]I M;Z^M%-QO+8NWRUK2A=6==,6W%U8D;?E60VAR1=^J5]6W*8@NZ?<0!KC"M@/O!%" ^ MZ%2OG5<.P='GL/C C>#*;VVW;#0&:5F4^(?7T^!R<#VXB+N[KI[N^R%M;*K]W5$E+J^'R2&^X/M+2Y@YWTIG:_.'& M; B'0ESKJ[*F"NLDY54F<4^+[9OWS:RM;BN0G*;F;:U]$E&^[?KUU1GQV.+. M.KNX=-^LTV;1P9$J/)/N.H^+]M3KIIOT)DGQHY69_/:8"]@Y;@$42>>"7;8- M-)>%%-@][@46!OLN1;5N8'M_::!>X$C V+GPI?B]\E0 0+XW '=Q5RKO-.3J M1 66F\!3X!# "G""3<&N8"6YH-\7K'(@!OL8T$[2]^Z+E])-+G_Q!ELI!>6: MYG]RQYRH7@)O*34IN*K?',G&7Y.7J-\L>ADA0HHS3*A?:+?3^ M"LFWRMS4+"#68 B,\7]/ZUR25I H="VAW%6#4/6 M:,>Y_5/;G^0.G?NU)7[^.(%[C%BYK$!WMPLKA*0%6'5S>49.8[4Q=YO/%>1A M8H5IFEB9[\TWRT9?(Z?]5VV^!-TLFT$WXX::2\4BXB!I6U^DK& NR]B,//4#ZIG;)C*5 N]$J*G[[LN=MKXI'Z2 M(O^K^L;4K,?4(]N+S;*%9'5Z_KS[;#:%41,00\BB $,#_-F5+ V8S#?H>])I M$91:.^<+ \ET08"?6]]D3$%'NARZ[2+6IZ)@UK(+S=+3XVA?O_<_?F&7-V_\G[K/I-<#.+4,\F M@@-K4E/6*B68D1@ C@2C@#>@/+S4G=J5N-L)GLT)?RRQP-,X'BG81MB(_?CJ M[/)I%4=G\#46>4>QBUO,V-AVE5$^IHK6=KG;]=\" $>YOD\A7AYPH$N?]?^! M8X]T#+OOK_U6/QC$4[5!9B.9=YN/7(+V%ZP*=K,.@Q_!R]G.Z3'X'RRYB\M> MY5BF_SW?*)S.$:RCO>UQ@%&,?;MD\%%WS=?#"#9>6L%MJ3+,)W VL.D(4K13P2!CE;:[I+EVM#C[$J!-4VAA2S(&6 M)Y^>YTF#)E*0 5S+0P*SZ/B$G5HQL%_N'QB M@U.Q"MT)XI%PO7H.EALB&$6'PE)5H1/@"4 $" )0 4AJOD\W*S=S=UO'; Y- MAK.\"EG+[^7.!#PYM0(+T\*LID:QY:()N->BO:HV"_L+)KH*HALOP\7>0/X&YKB]3<3(3,-'U MAHVYR=X)L$S7#]IVY472=X7 %('*7^OPL2M519U"Y4X>VM/(Y9;V0\ST>V+V M^T!4WLK[3:% \:I']8[N=JN[OMW)J\#2">@ME9^^&1V\*;W$Z!CU RRAO=/J M,NEYSUT;,6".>:FV2:3>9TD#?1NFW]W0;?,+SJ1V1Z>[.J!-KW571FQ8"^XR M6]6CP3GO[IQW,CD"'A6N:ZBOFE3\,'LVDVHDGB=2=\F[^;3S;DSS['>I;>\N M6T^#FUJ8I[P43LL"+L[)@&_!F#M%G2[88B)+=6 B@FN\1=MLI%:8[-?;5=IJ M%=FTY%L69Z37$EGH!= ]@>^X,CII[RM$4)RU11.3#@6TD#@F+S\*(WGQ:WT+T6:J/#5\A;G7,83M?^&0I3[TRC.*AULB*E0.'T>7;-S>78>XET P96S\;KE!/9R>C'%H3C.[:9887NUT$/2;7:3QFNZ'6!J>ZFA[M7#XY]F>GL+31 MB=FK\5Z>[P&W,5B\[;P>;].U?4<3J6(SDKKGS.$Z-OV<+E+)I@(@>XKOS=3< MBTF/IL?LK2@5Z^SK]Y;\\7@1M8G)%" M#QFX1%\(KLR8@FOFTPXN".4),1V"7,NPH]IL')NB#MZL[=*Z[M8RUD@';N/- M2%JXU-O7+0S71BK#U6&V'G>I/DQZ:-H7]WJO6OO.0'VXP-(@[N.V$/CE\-MX M:8U[NV(HJ1(WRC:S0=[,;'B;TKTMYI64FFHLWMM>4\>86=SD9NLF(SI^3/Q^ M<:_%H0'I)IJ4C"N C4F61)>+4D18K0I0-LS&!7:Z<2=W<%S,KQP7DDM&W.#9 MWG3?+?1P+<2/'M]S$KRZW*FK_O8I>_7ZYH+TO+$DW#41AM?-^ M''&9"5[R,LFS).!\V@E8-^ !QL)1 M"%]_$E?Z[J=1@MS2>P&/TF[&_UQ/+%N09SR*_0NI7;&^ES=_PBIV7(Q;5>Q* MB+N,A.$U[*&1+GO172SP%>]U3N"I;8,63CC6+?55=<=SYTQ>GOLSN*K2)4/" M#9QS+MTS9 '5WM@<-)WV!,8"-UV[GMPS!;K3/0KV=!=^]E?['G<4J6;LO;L9 M=5&3.E8JLONS8W@TY1'"[9:]3MJI;E,7-ZIJ!;(V![5R_RZ M(96RN4+'+J1W^)<-WMA1=E^'?%.M[)QP-3P"QL>J@O_ ?S5'I$!QL#@(EOF4 M%:>(?^)U[2+X/M<(3MM%@#?(D^!=H2/X<+KI>Y F"6W![6!(6C,8\CG,4YR MJ%1^HF"OWS1X#?R_305C@]-SK. GWZ!MW\A&BP7;=V?!]5D,'/<-OLIB8(=L/%@8K@XG!VUQC, ?YQ.IG$YMV&0)I,<)M3^@VN-Y<)&U'OR_E1LZ5>XV%$A?<(VWGQQF_">;82'! F$.,D&XVGCI.PA[ M@A?(JT&&<*_P "?)QUC@7J-8!RDG<#+F M5MV1A+L^G<]218ODHZEA2Q.S-.':98;1D=!6YF8J_R3 -5G%28EN::<*ZAL] M?.9J+L@;@(8 .XRO&Q"F5F_"?N-10N-/LX(,.& ?OQ_4ZQ> M8EVMR3HL';B->:HL:)=V"7(4-L;Y0%!(?-#FY8@QY,OGUU%@= M=GM]XYV[K'/0?)<#VI@^_R004D#SG\;J?=Q?Q07VYO"?JE L8'.51>MH>D9( M:%.-DT;5'U93M6QP_%IJ*-'*J($UP/MXK?SC_$+&7]%T436 <+QNL_R,K1_; MA=2%8U;X,679!F!9)J4)/WF70[8FGXAVOT==]CVP8YFQ%64U7M(T-]"_.Z&]E-8Y^5#QEXC]66W6'?XM8IPA.])[_ZUS,'#WA(#;2C=,%*2 M"IJ$*341XVAW'5MI:K[EYK">_S=8HD/NXOND/1A^/6/,R\(6@ YO50)9U0Y> M_.0YMW%-[M]W 'J84/\U\-/VGWO#7> MB9G,>,>(A^2F3M,?CM7R#>.N&;OS7-[4.UN;5<]Y:+O,?+L?ZFM590QF5A^R M.>@TH %GJ']8F?S1%2$&!O:5,37L\H#OLV:5:Q07+M]U0H8W?1&TL(0X&R%P.6Q5/61"88R8P >-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 15:16:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09733; Tue, 2 Mar 93 15:16:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24112; Tue, 2 Mar 93 15:14:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 15:14:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24104; Tue, 2 Mar 93 15:13:57 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA26860 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 14:13:54 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA08480; Tue, 2 Mar 93 14:13:52 CST Date: Tue, 2 Mar 93 14:13:52 CST Message-Id: <9303022013.AA08480@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A proposal for buffer descriptors The following postscript file contains a more detailed account of the buffer descriptors that we discussed at the last meeting. These allow one to send non-contiguous structures of mixed typed data between heterogeneous machines. Bill and Rusty %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: bd.dvi %%Pages: 8 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 46 123 df<003FE3F801F03F1C03C03E3E07C07C3E0F807C3E0F807C1C0F807C000F80 7C000F807C000F807C000F807C00FFFFFFC0FFFFFFC00F807C000F807C000F807C000F807C000F 807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C00 7FE1FFC07FE1FFC01F1D809C1C>11 D<78FCFCFCFC7806067D850D>46 D<00600001E0000FE000 FFE000F3E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000 03E00003E00003E00003E00003E00003E00003E00003E00003E0007FFF807FFF80111B7D9A18> 49 D<07F8001FFE00383F80780FC0FC07C0FC07E0FC03E0FC03E07803E00007E00007C00007C0 000F80001F00001E0000380000700000E0000180600300600600600800E01FFFC03FFFC07FFFC0 FFFFC0FFFFC0131B7E9A18>I<03F8001FFE003C1F003C0F807C07C07E07C07C07C03807C0000F 80000F80001E00003C0003F800001E00000F800007C00007C00007E03007E07807E0FC07E0FC07 E0FC07C0780F80781F001FFE0007F800131B7E9A18>I<000180000380000780000F80001F8000 3F80006F8000CF80008F80018F80030F80060F800C0F80180F80300F80600F80C00F80FFFFF8FF FFF8000F80000F80000F80000F80000F80000F8001FFF801FFF8151B7F9A18>I<000380000003 80000007C0000007C0000007C000000FE000000FE000001FF000001BF000001BF0000031F80000 31F8000061FC000060FC0000E0FE0000C07E0000C07E0001803F0001FFFF0003FFFF8003001F80 03001F8006000FC006000FC00E000FE00C0007E0FFC07FFEFFC07FFE1F1C7E9B24>65 DI<001FE02000FFF8E003F80FE007C003E00F8001E01F0000E03E0000E03E0000607E000060 7C000060FC000000FC000000FC000000FC000000FC000000FC000000FC000000FC0000007C0000 607E0000603E0000603E0000C01F0000C00F80018007C0030003F80E0000FFFC00001FE0001B1C 7D9B22>IIII<000FF008007FFE3801FC07F807E001F80F8000781F0000783F000038 3E0000387E0000187C000018FC000000FC000000FC000000FC000000FC000000FC000000FC007F FFFC007FFF7C0001F87E0001F83E0001F83F0001F81F0001F80F8001F807E001F801FC07F8007F FE78000FF818201C7D9B26>I73 D76 DII<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E0003E07E0003 F07C0001F07C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC00 01F87C0001F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E0001F07C0000 3FE0001D1C7D9B24>II82 D<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE0000FFE0007FFE 003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000E0E000E0F001 C0FC03C0EFFF0083FC00141C7D9B1B>I<7FFFFFE07FFFFFE0781F81E0701F80E0601F8060E01F 8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F8000001F800000 1F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000 001F800007FFFE0007FFFE001C1C7E9B21>III89 D<0FF8001C1E003E0F803E07803E07C01C07C00007C0007FC007E7C01F07C03C07C07C07 C0F807C0F807C0F807C0780BC03E13F80FE1F815127F9117>97 DI<03FC000E0E001C1F003C1F00781F00780E00F80000F80000F80000F80000F80000F800 007800007801803C01801C03000E0E0003F80011127E9115>I<000FF0000FF00001F00001F000 01F00001F00001F00001F00001F00001F00001F001F9F00F07F01C03F03C01F07801F07801F0F8 01F0F801F0F801F0F801F0F801F0F801F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C 1B>I<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F80000 7800007C00603C00601E00C00F038001FC0013127F9116>I<03F8F00E0F381E0F381C07303C07 803C07803C07803C07801C07001E0F000E0E001BF8001000001800001800001FFF001FFFC00FFF E01FFFF07801F8F00078F00078F000787000707800F01E03C007FF00151B7F9118>103 D<1E003F003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D107 DIII<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800F8F8 00F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>II114 D<1FD830786018E018E018F000FF807FE07F F01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007000F 000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08079803F0 0E1A7F9913>II119 DII<3FFF803C1F00303F00 303E00607C0060FC0060F80001F00003F00007E00007C1800F81801F81801F03803E03007E0700 7C0F00FFFF0011127F9115>I E /Fc 58 126 df<03800007E0000FE0001E70001C70001C7000 1C70001C77E01CE7E01DE7E00FC7000F8E000F0E001E0E003F1C007F1C00739C00E3F800E1F800 E0F1C0E0F1C071F9C07FFFC03F9F801E070013197F9816>38 D<00E001E0038007000E001C001C 0038003800700070007000E000E000E000E000E000E000E000E000E00070007000700038003800 1C001C000E000700038001E000E00B217A9C16>40 DI<01C00001C00001C00001C00071C700F9CF807FFF001F FC0007F00007F0001FFC007FFF00F9CF8071C70001C00001C00001C00001C00011127E9516>I< 387C7E7E3E0E1E1C78F060070B798416>44 D<70F8F8F8700505788416>46 D<03E0000FF8001FFC001E3C00380E00780F00700700700700E00380E00380E00380E00380E003 80E00380E00380E00380F00780700700700700780F003C1E001E3C001FFC000FF80003E0001119 7E9816>48 D<01800380038007800F807F80FF8073800380038003800380038003800380038003 80038003800380038003807FF87FFC7FF80E197C9816>I<07E0001FF8003FFC00783E00E00700 F00780F00380600380000380000380000700000700000E00001C0000380000700000E00001C000 0380000F00001E03803803807FFF80FFFF807FFF8011197E9816>I<07E0001FF8003FFC00781E 00780700300700000700000700000E00003E0007FC0007F00007FC00001E000007000003000003 80000380600380F00380E00700781E003FFC001FF80007E00011197E9816>I<007C0000FC0000 DC0001DC00039C00039C00071C000F1C000E1C001E1C003C1C00381C00781C00F01C00FFFFE0FF FFE0FFFFE0001C00001C00001C00001C00001C0001FFC001FFC001FFC013197F9816>I<07F000 1FFC003FFE007C1F00F00780E00380E00380E003807007007C1F001FFC0007F0001FFC003C1E00 700700F00780E00380E00380E00380F007807007007C1F003FFE001FFC0007F00011197E9816> 56 D<70F8F8F870000000000000000070F8F8F8700512789116>58 D<387C7C7C380000000000 00000038787C7C3C1C1C3870E0400618799116>I<7FFF00FFFF80FFFF80000000000000000000 000000000000FFFF80FFFF807FFF00110B7E9116>61 D<7FF800FFFE007FFF001C0F001C07801C 03801C03801C03801C07801C07001FFF001FFE001FFE001C1F001C03801C03C01C01C01C01C01C 01C01C01C01C03C01C07807FFF80FFFF007FFC0012197F9816>66 D<7FF800FFFE007FFF001C0F 001C07801C03C01C01C01C01C01C01E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00 E01C01C01C01C01C03C01C07801C0F807FFF00FFFE007FF8001319809816>68 D<7FFFC0FFFFC07FFFC01C01C01C01C01C01C01C01C01C00001C00001C1C001C1C001FFC001FFC 001FFC001C1C001C1C001C00001C00E01C00E01C00E01C00E01C00E07FFFE0FFFFE07FFFE01319 7F9816>I<7F1FC0FFBFE07F1FC01C07001C07001C07001C07001C07001C07001C07001FFF001F FF001FFF001C07001C07001C07001C07001C07001C07001C07001C07001C07007F1FC0FFBFE07F 1FC013197F9816>72 DI76 DI<7E1FC0FF3FE07F1FC01D07001D87001D87001D87001DC7001DC7001CC7001CC7001C E7001CE7001CE7001C67001C67001C77001C77001C37001C37001C37001C17007F1F00FF9F007F 0F0013197F9816>I<1FFC003FFE007FFF00780F00F00780E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780F00780780F007FFF00 3FFE001FFC0011197E9816>I<7FF800FFFE007FFF001C0F801C03801C03C01C01C01C01C01C01 C01C03C01C03801C0F801FFF001FFE001FF8001C00001C00001C00001C00001C00001C00001C00 007F0000FF80007F000012197F9816>I<7FE000FFF8007FFC001C1E001C0F001C07001C07001C 07001C07001C0F001C1E001FFC001FF8001FFC001C1C001C0E001C0E001C0E001C0E001C0E201C 0E701C0E707F07E0FF87E07F03C014197F9816>82 D<07E3001FFF003FFF00781F00F00700E007 00E00700E00000F000007800003F80001FF00007FC0000FE00000F000007000003800003806003 80E00380E00700F80F00FFFE00FFFC00C7F00011197E9816>I<7FFFE0FFFFE0FFFFE0E0E0E0E0 E0E0E0E0E0E0E0E000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00007FC000FFE0007FC0013197F9816>I86 D91 D93 D95 D<1FE0003FF0007FF800783C00300E00000E00000E00 03FE001FFE003E0E00700E00E00E00E00E00E00E00783E007FFFE03FE7E00F83E013127E9116> 97 D<7E0000FE00007E00000E00000E00000E00000E00000E3E000EFF000FFF800F83C00F00E0 0E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00FFF800EFF00063C00 1419809816>I<03F80FFC1FFE3C1E780C7000E000E000E000E000E000F000700778073E0E1FFC 0FF803F010127D9116>I<003F00007F00003F0000070000070000070000070003C7000FF7001F FF003C1F00780F00700700E00700E00700E00700E00700E00700E00700700F00700F003C1F001F FFE00FE7F007C7E014197F9816>I<03E00FF81FFC3C1E780E7007E007FFFFFFFFFFFFE000E000 700778073C0F1FFE0FFC03F010127D9116>I<001F00007F8000FF8001E78001C30001C00001C0 007FFF00FFFF00FFFF0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C0003FFE007FFF003FFE0011197F9816>I<03E3C007F7E00FFFE01C1CC0380E0038 0E00380E00380E00380E001C1C000FF8001FF0001BE0003800001800001FFC001FFF003FFF8078 03C0E000E0E000E0E000E0E000E07001C07C07C03FFF800FFE0003F800131C7F9116>I<7E0000 FE00007E00000E00000E00000E00000E00000E3C000EFE000FFF000F87800F03800E03800E0380 0E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01519809816> I<018003C003C0018000000000000000007FC07FC07FC001C001C001C001C001C001C001C001C0 01C001C001C001C07FFFFFFF7FFF101A7D9916>I<7E0000FE00007E00000E00000E00000E0000 0E00000E7FE00E7FE00E7FE00E0F000E1E000E3C000E78000EF0000FF0000FF8000FBC000F1E00 0E0E000E07000E07807F87F0FFCFF07F87F01419809816>107 DII<7E3C00FEFE007FFF000F87800F03800E03800E03800E03800E 03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01512809116>I<03E000 0FF8001FFC003C1E00780F00700700E00380E00380E00380E00380E00380F00780700700780F00 3C1E001FFC000FF80003E00011127E9116>I<7E3E00FEFF007FFF800F83C00F00E00E00E00E00 700E00700E00700E00700E00700E00700E00E00F01E00F83C00FFF800EFF000E3C000E00000E00 000E00000E00000E00000E00007FC000FFE0007FC000141B809116>I114 D<0FEC3FFC7FFCF03CE01CE01C70007F801FF007F8003C600EE0 0EF00EF81EFFFCFFF8C7E00F127D9116>I<0300000700000700000700000700007FFF00FFFF00 FFFF00070000070000070000070000070000070000070000070100070380070380070380078700 03FE0001FC0000F80011177F9616>I<7E1F80FE3F807E1F800E03800E03800E03800E03800E03 800E03800E03800E03800E03800E03800E03800E0F800FFFF007FBF803E3F01512809116>I<7F 1FC0FF1FE07F1FC01C07001E0F000E0E000E0E000E0E00071C00071C00071C00071C0003B80003 B80003B80001F00001F00000E00013127F9116>II<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001F00000E00001E00001F00003B8 00073C00071C000E0E007F1FC0FF3FE07F1FC013127F9116>I<7F1FC0FF9FE07F1FC01C07000E 07000E0E000E0E00070E00071C00071C00039C00039C0003980001B80001B80000F00000F00000 F00000E00000E00000E00001C00079C0007BC0007F80003F00003C0000131B7F9116>I<3FFFC0 7FFFC07FFFC0700780700F00701E00003C0000780001F00003E0000780000F00001E01C03C01C0 7801C0FFFFC0FFFFC0FFFFC012127F9116>I<001F80007F8000FF8001E00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00003C0007F8000FF0000FF00007F800003C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001E00000FF80007F80001F801120 7E9C16>I<7C0000FF0000FF800003C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001E00000FF00007F80007F8000FF0001E00001C00001C00001C00001C00001C00001 C00001C00001C00001C00003C000FF8000FF00007C000011207E9C16>125 D E /Fd 59 123 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 0E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00 000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<003F07E00001C0 9C18000380F018000701F03C000E01E03C000E00E018000E00E000000E00E000000E00E000000E 00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C00 0E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>14 D<6060F0F0F8F868680808 08080808101010102020404080800D0C7F9C15>34 D<004000800100020006000C000C00180018 00300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000E0006000 60006000700030003000180018000C000C00060002000100008000400A2A7D9E10>40 D<800040002000100018000C000C000600060003000300038001800180018001C001C001C001C0 01C001C001C001C001C001C001C001C0018001800180038003000300060006000C000C00180010 002000400080000A2A7E9E10>I<01800180018001804182F18F399C0FF003C003C00FF0399CF1 8F4182018001800180018010127E9E15>I<60F0F0701010101020204080040C7C830C>44 DI<60F0F06004047C830C>I<00010003000600060006000C000C000C00 18001800180030003000300060006000C000C000C0018001800180030003000300060006000C00 0C000C00180018001800300030003000600060006000C000C00010297E9E15>I<03C00C301818 300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E007600660 06700E300C300C18180C3007E0101D7E9B15>I<030007003F00C7000700070007000700070007 0007000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C 7C9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C00380070 006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<60F0F060000000000000 0000000060F0F06004127C910C>58 D<60F0F0600000000000000000000060F0F0701010101020 204080041A7C910C>I<000600000006000000060000000F0000000F0000000F00000017800000 178000001780000023C0000023C0000023C0000041E0000041E0000041E0000080F0000080F000 0180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E0004001E000C001F 001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E0618001 801980070007800E0003801C0003801C00018038000180780000807800008070000080F0000000 F0000000F0000000F0000000F0000000F0000000F0000000F00000007000008078000080780000 80380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E> II70 D72 DI77 DI80 D<07E0801C1980300580700380600180E00180E00080E0 0080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C000 01C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>83 D<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F0020000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003FFFC001B1C7F9B 1E>IIII91 D<08081010202040404040808080808080B0B0 F8F8787830300D0C7A9C15>II<1FC000307000783800781C00301C00001C 00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F87001212 7E9115>97 DI<07E00C301878307870306000E000E000E0 00E000E000E00060007004300418080C3007C00E127E9112>I<003F0000070000070000070000 070000070000070000070000070000070000070003E7000C1700180F00300700700700600700E0 0700E00700E00700E00700E00700E00700600700700700300700180F000C370007C7E0131D7E9C 17>I<03E00C301818300C700E6006E006FFFEE000E000E000E00060007002300218040C1803E0 0F127F9112>I<00F8018C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C0 0C38C01C3880181800381C00381C00381C00381C001818001C38000C300013C000100000300000 1800001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007F800 121C7F9215>II<18003C003C0018000000000000000000 000000000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C 00FF80091D7F9C0C>I107 DIII<03F0000E1C00180600300300700380600180E001C0E0 01C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F9115>I< FC7C001D86001E03001C01801C01C01C00C01C00E01C00E01C00E01C00E01C00E01C00E01C01C0 1C01C01C01801E03001D06001CF8001C00001C00001C00001C00001C00001C00001C0000FF8000 131A7F9117>I<03C1000C3300180B00300F00700700700700E00700E00700E00700E00700E007 00E00700600700700700300F00180F000C370007C7000007000007000007000007000007000007 00000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE00FF000 F880388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C00FF E01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C1A7F99 10>IIII<7F8FF00F03800F030007020003840001C80001D80000F000007000007800 00F800009C00010E00020E000607000403801E07C0FF0FF81512809116>II<7F FC70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E127F91 12>I E /Fe 14 118 df<000FF83F00007FFDFFC001F81FE3E003E03F87E007C03F87E00F803F 07E00F803F03C00F801F00000F801F00000F801F00000F801F00000F801F00000F801F0000FFFF FFFC00FFFFFFFC000F801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F 801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F801F0000 0F801F00000F801F00000F801F00000F801F00007FF0FFF0007FF0FFF00023237FA221>11 D<387CFEFEFE7C3807077C8610>46 D<00180000780001F800FFF800FFF80001F80001F80001F8 0001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F8 0001F80001F80001F80001F80001F80001F80001F80001F80001F8007FFFE07FFFE013207C9F1C >49 D66 D<00FF8007FFE00F83F01F03F03E03 F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00007C00007E00007E00003E00 301F00600FC0E007FF8000FE0014167E9519>99 D<0001FE000001FE0000003E0000003E000000 3E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0001FC3E0007 FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC003E00FC003E00FC003E00 FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E001E00FE000F83BE0007FF3F C001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8 FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F00300FC07003FFC000FF00 15167E951A>I<1C003E007F007F007F003E001C000000000000000000000000000000FF00FF00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FF E00B247EA310>105 D<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC00 7EFC007EFC007EFC007EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE001716 7E951C>111 DI114 D<0FF3003FFF00781F00600700E00300E003 00F00300FC00007FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F007 00FC0E00EFFC00C7F00011167E9516>I<01800001800001800001800003800003800007800007 80000F80003F8000FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F 80000F80000F80000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F 16>II E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 210 105 a Fe(1.)18 b(Bu\013er)g(descriptors)210 216 y Fd(The)h (message-passing)f(routines)h(describ)q(ed)h(ab)q(o)o(v)o(e)e(are)h (su\016cien)o(t)g(to)f(write)h(relativ)o(ely)e(e\016cien)o(t)i(pro-)210 286 y(grams.)25 b(In)17 b(man)o(y)e(applications,)h(ho)o(w)o(ev)o(er,)h(the)h (data)e(to)h(b)q(e)g(sen)o(t)h(or)e(receiv)o(ed)i(is)f(not)g(in)f(consequtiv) o(e)210 355 y(lo)q(cations)11 b(in)g(memory)m(.)k(While)c(an)o(y)g(data,)g (ho)o(w)o(ev)o(er)h(laid)f(out)h(in)f(memory)m(,)e(ma)o(y)g(b)q(e)k(sen)o(t)f (or)g(receiv)o(ed)h(this)210 425 y(w)o(a)o(y)m(,)h(there)j(are)e(a)g(n)o(um)o (b)q(er)g(of)f(sp)q(ecial)i(cases)h(for)d(whic)o(h)i(sp)q(ecial)f(arrangemen) o(ts)g(ma)o(y)e(b)q(e)j(made)e(b)o(y)h(the)210 495 y(op)q(erating)e(system)g (or)f(the)i(hardw)o(are.)k(These)c(include)f(constan)o(t,)g(non-unit)g (strides)h(\(useful)f(for)g(sending)210 565 y(a)k(ro)o(w)g(from)e(a)i(F)m (ortran)g(arra)o(y\),)g(scatter/gathers)i(\(useful)f(for)e(sparse)j(v)o (ectors\),)g(and)d(non-con)o(tiguous)210 634 y(blo)q(c)o(ks)j(\(useful)g(for) g(sending)g(structures)j(with)c(p)q(oin)o(ters\).)34 b(These)20 b(op)q(erations)g(ma)o(y)d(b)q(e)i(pro)o(vided)g(b)o(y)210 704 y(the)c(lo)o(w-lev)o(el)e(hardw)o(are)i(and)g(soft)o(w)o(are)g(b)q (ecause)h(they)f(are)g(common)d(and)i(b)q(ecause)j(imp)q(ortan)o(t)c(p)q (erfor-)210 774 y(mance)d(optimizations)e(can)j(b)q(e)g(made)e(for)h(them,)g (particularly)g(the)h(elimination)d(of)i(unnecessary)i(memory)210 844 y(motion.)22 b(F)m(or)16 b(Unix)g(programmers,)e(the)j(non-con)o(tiguous) e(blo)q(c)o(k)h(case)h(is)f(simply)e(the)j(analogue)e(of)g(the)210 913 y(routines)d Fc(readv)e Fd(and)g Fc(writev)p Fd(.)16 b(Sev)o(eral)c (message-passing)e(implemen)o(tatio)o(ns)f(already)i(pro)o(vide)f(non-unit) 210 983 y(stride)h(v)o(ector)h(op)q(erations,)f(and)f(researc)o(h)j(w)o(ork)d (with)g(scatter/gather)j(op)q(erations)d(has)h(sho)o(wn)g(that)g(these)210 1053 y(are)j(also)g(of)f(great)h(p)q(oten)o(tial)f(b)q(ene\014t.)272 1123 y(MPI)e(pro)o(vides)g(access)i(to)d(all)g(of)g(these)i(in)e(an)o(y)g (com)o(bination)e(b)o(y)j(w)o(a)o(y)f(of)g(a)g(general)h(data-arrangemen)o(t) 210 1192 y(descriptor.)210 1319 y Fb(1.1.)16 b(Organization)210 1415 y Fd(A)i(data-arrangemen)o(t)e(is)i(de\014ned)h(in)e(2)g(steps:)27 b(\(1\))18 b(create)h(a)f(data-arrangemen)o(t)e(descriptor)j(and)f(\(2\))210 1485 y(add)13 b(descriptions)i(of)e(the)h(data)f(arrangemen)o(t)g(to)g(it.)k (A)d(data-arrangemen)o(t)e(descriptor)j(is)e(created)i(with)210 1555 y Fc(MPI)p 279 1555 14 2 v 15 w(NewBD)p Fd(.)e(The)i(routines)g Fc(MPI)p 741 1555 V 15 w(BDVec)p Fd(,)e Fc(MPI)p 957 1555 V 15 w(BDBlk)p Fd(,)h(and)g Fc(MPI)p 1255 1555 V 15 w(BDIdx)f Fd(add)i(descriptions)g(for)f(non-unit)210 1624 y(stride)e(v)o(ectors,)h(blo) q(c)o(ks,)e(and)h(indexed)g(v)o(ectors)g(\(these)i(are)d(describ)q(ed)j(in)d (more)f(detail)h(on)h(the)g(man)e(pages)210 1694 y(for)17 b(these)i (routines\).)29 b(A)17 b(send)h(or)f(receiv)o(e)i(op)q(eration)e(then)h(tak)o (es)g(the)f(data-arrangemen)o(t)g(descriptor)210 1764 y(instead)d(of)f(the)i (\(bu\013er,size\))g(that)f(the)h(send)g(and)e(receiv)o(e)j(routines)e(use)h (for)e(con)o(tiguous)h(data.)210 1890 y Fb(1.2.)i(V)l(ectors)210 1987 y Fd(The)d(\\v)o(ector")g(data-arrangemen)o(t)f(allo)o(ws)f(sending)i (and)g(receiving)g(of)f(non-con)o(tiguous)g(data)g(where)i(the)210 2057 y(elemen)o(ts)g(are)g(separated)h(b)o(y)f(a)f(constan)o(t)i(distance)g (or)e(stride.)272 2126 y(T)m(o)g(add)h(a)g(v)o(ector)g(to)g(the)h(data)e (arrangemen)o(t,)g(use)210 2222 y Fc(MPI_BDVec\()20 b(da,)h(buffer,)f (elmlen,)h(stride,)f(elmcount,)g(datatype)g(\))210 2318 y Fd(The)14 b(parameters)g(are:)210 2415 y Fb(bu\013er)251 b Fd(p)q(oin)o(ter)14 b(to)g(bu\013er)210 2514 y Fb(elmlen)236 b Fd(size)15 b(of)e(eac)o(h)h (elemen)o(t,)f(in)h(b)o(ytes)210 2614 y Fb(stride)254 b Fd(distance)15 b(b)q(et)o(w)o(een)g(successiv)o(e)h(elemen)o(ts,)d(in)h(b)o(ytes)p eop %%Page: 2 2 bop 210 105 a Fb(elmcoun)o(t)182 b Fd(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts) 210 208 y Fb(datat)o(yp)q(e)190 b Fd(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 932 208 14 2 v 15 w(INT)p Fd(\))272 311 y(F)m(or)h(example,)e(if)h (a)h(F)m(ortran)f(arra)o(y)h(is)g(declared)g(as)210 414 y Fc(double)21 b(precision)e(a\(10,20\))210 517 y Fd(and)14 b(the)g(ro)o(w)g Fc(a\(3,1:20\))e Fd(is)i(to)f(b)q(e)i(sen)o(t,)f(use)210 620 y Fc(bd)21 b(=)h(MPI_BDNew\(1\))210 690 y(MPI_BDVec\()e(bd,)h(a\(3,1\),)f(8,) i(8*10,20)e(\))210 759 y(MPI_bsend\()g(to,)h(tag,)g(bd,)g(context)f(\))210 862 y Fd(Here)15 b(w)o(e)f(assume)g(that)g(a)f(double)h(precision)g(quan)o (tit)o(y)f(tak)o(es)i(eigh)o(t)e(b)o(ytes.)210 990 y Fb(1.3.)j(Blo)q(c)o(ks) 210 1087 y Fd(Blo)q(c)o(ks)d(of)f(con)o(tiguous)g(data)g(ma)o(y)f(b)q(e)i (added)g(to)f(the)h(data)f(arrangemen)o(t)g(with)g Fc(MPI)p 1564 1087 V 15 w(BDBlk)p Fd(.)17 b(The)c(format)210 1156 y(is)210 1259 y Fc(MPI_BDBlk\()20 b(bd,)h(buffer,)f(len,)h(datatype)f(\))272 1362 y Fd(Multiple)12 b(uses)i(of)e Fc(MPI)p 640 1362 V 15 w(BDBlk)g Fd(ma)o(y)e(b)q(e)j(used)h(to)e(send)i(complicated)d(data)h (structures.)21 b(F)m(or)12 b(example,)210 1432 y(to)i(send)h(the)f (structure)210 1535 y Fc(struct)21 b({)297 1605 y(int)g(n[3];)297 1674 y(double)g(a[4];)297 1744 y(char)65 b(s;)297 1814 y(})22 b(S;)210 1917 y Fd(w)o(e)14 b(can)g(use)h(the)f(follo)o(wing)e(co)q(de:)210 2020 y Fc(bd)21 b(=)h(MPI_BDNew\()e(3)h(\);)210 2090 y(MPI_BDBlk\()f(bd,)h (&S.n[0],)f(3)i(*)f(sizeof\(int\),)85 b(MPI_INT)20 b(\);)210 2159 y(MPI_BDBlk\()g(bd,)h(&S.a,)86 b(4)22 b(*)f(sizeof\(double\),)e(MPI_DBL) h(\);)210 2229 y(MPI_BDBlk\()g(bd,)h(&S.s,)86 b(sizeof\(char\),)150 b(MPI_OTHER)20 b(\);)210 2299 y(MPI_bsend\()g(to,)h(tag,)g(bd,)g(context)f (\);)210 2402 y Fd(T)m(o)13 b(receiv)o(e)i(the)g(structure,)g(the)g(exact)f (same)f(co)q(de)i(is)f(used,)g(with)g(the)g Fc(MPI)p 1432 2402 V 15 w(bsend)f Fd(replaced)i(with)210 2505 y Fc(MPI_brecv\()20 b(source,)g(tag,)h(bd,)g(context,)f(recv_handle)g(\);)p eop %%Page: 3 3 bop 210 105 a Fb(1.4.)16 b(Indexed)210 202 y Fd(Another)e(t)o(yp)q(e)g(of)f (non-con)o(tiguous)g(comm)o(unicatio)o(n)e(pattern)j(that)f(is)h(common)c(in) j(applications)g(is)g(\\in-)210 271 y(dexed.")18 b(F)m(or)11 b(example,)f(if)g(elemen)o(ts)h(of)f(one)i(v)o(ector,)g(indexed)g(b)o(y)e (another,)i(are)g(used.)18 b(These)12 b(are)g(de\014ned)210 341 y(with)i Fc(MPI)p 374 341 14 2 v 15 w(BDIdx)p Fd(.)j(The)d(format)e(is) 210 444 y Fc(MPI_BDIdx\()20 b(bd,)h(buffer,)f(idxarray,)g(numitems,)g (datatype)g(\))p eop %%Page: 4 4 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(NewBD)117 b Fd(Create)15 b(a)f(new)g(bu\013er)h(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(MPI)p 397 314 13 2 v 15 w(BD)f(*MPI)p 589 314 V 15 w(NewBD\()h(n)o(umitems)c(\))314 384 y(in)o(t)i(n)o(umitems;)210 454 y Fb(INPUT)j(AR)o(GUMENTS)314 524 y Fd(n)o(umitesm)215 b(Maxim)o(um)10 b(n)o(um)o(b)q(er)j(of)h(bu\013er)h(descriptions)g(for)e(the) i(bu\013er)f(descriptor.)210 593 y Fb(DESCRIPTION)314 663 y(MPI)p 413 663 15 2 v 17 w(NewBD)h Fd(creates)h(a)e(new)g(bu\013er)i(descriptor.)k (The)15 b(v)n(alue)f(of)f Fc(numitems)g Fd(is)h(the)h(maxim)n(um)314 733 y(n)o(um)o(b)q(er)c(of)g(bu\013er)h(descriptions)h(that)e(can)h(b)q(e)g (added)g(with)g(the)g(routines)g Fc(MPI)p 1574 733 14 2 v 15 w(BDVec)p Fd(,)e Fc(MPI)p 1787 733 V 16 w(BDBlk)p Fd(,)314 802 y(and)j Fc(MPI)p 463 802 V 16 w(BDIdx)p Fd(.)210 872 y Fb(RETURN)j(V)-5 b(ALUE)314 942 y(MPI)p 413 942 15 2 v 17 w(NewBD)14 b Fd(returns)i(a)d(p)q(oin)o(ter)h(to)g(a)g(bu\013er)g(descriptor,)h(or)f(n)o (ull)f(if)g(an)h(error)g(o)q(ccurs.)p eop %%Page: 5 5 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BD)o(V)l(ec)136 b Fd(Add)14 b(a)g(stride)g(v)o(ector)h(to)f(a)f(bu\013er)i(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BD)o(V)m(ec\()i(b)q(d,)f(bu\013er,)g(elmlen,)e(stride,)i(elmcoun)o(t,) e(datat)o(yp)q(e)j(\);)314 384 y(MPI)p 397 384 V 15 w(BD)f(b)q(d;)314 454 y(c)o(har)g(*bu\013er;)314 524 y(in)o(t)f(elmlen,)f(stride,)i(elmcoun)o (t,)f(datat)o(yp)q(e;)210 593 y Fb(INPUT)j(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(elmlen)275 b(size)15 b(of)e(eac)o(h)h(elemen)o(t,)f(in)h(b)o(ytes)314 872 y(stride)293 b(distance)15 b(b)q(et)o(w)o(een)g(successiv)o(e)h(elemen)o(ts,)d(in)h(b)o (ytes)314 942 y(elmcoun)o(t)228 b(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)314 1012 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 1012 14 2 v 15 w(INT)p Fd(\))210 1081 y Fb(DESCRIPTION)314 1151 y(MPI)p 413 1151 15 2 v 17 w(BD)o(V)l(ec)d Fd(adds)g(a)g(description)h (of)f(a)g(v)o(ector)h(\(with)f(non-unit-stride\))h(to)f(a)g(previously)g (created)314 1221 y(bu\013er)15 b(descriptor.)210 1291 y Fb(RETURN)h(V)-5 b(ALUE)314 1360 y(MPI)p 413 1360 V 17 w(BD)o(V)l(ec)13 b Fd(returns)j(0,)d (or)h Fa(\000)p Fd(1)g(if)f(an)g(error)i(o)q(ccurs.)p eop %%Page: 6 6 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BDBlk)137 b Fd(Add)14 b(a)g(con)o(tiguous)f(blo)q(c)o(k)h(to)g(a)f(bu\013er)i (descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BDBlk\()h(b)q(d,)g(bu\013er,)g(len,)g(datat)o(yp)q(e)g (\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)314 454 y(c)o(har)g (*bu\013er;)314 524 y(in)o(t)f(len,)h(datat)o(yp)q(e;)210 593 y Fb(INPUT)i(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(len)341 b(n)o(um)o(b)q(er)13 b(of)h(b)o(ytes)314 872 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 872 14 2 v 15 w(INT)p Fd(\))210 942 y Fb(DESCRIPTION)314 1012 y(MPI)p 413 1012 15 2 v 17 w(BDBlk)18 b Fd(adds)h(a)f(description)i(of)e (a)h(con)o(tiguous)f(blo)q(c)o(k)h(to)f(a)h(previously)g(created)h(bu\013er) 314 1081 y(descriptor.)210 1151 y Fb(RETURN)c(V)-5 b(ALUE)314 1221 y(MPI)p 413 1221 V 17 w(BDBlk)13 b Fd(returns)i(0,)e(or)h Fa(\000)p Fd(1)g(if)f(an)h(error)h(o)q(ccurs.)p eop %%Page: 7 7 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BDIdx)140 b Fd(Add)14 b(a)g(scatter/gather)h(to)f(a)g(bu\013er)h(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BDIdx\()h(b)q(d,)g(bu\013er,)h(idxarra)o(y)m(,)d(n)o(umitems,)f(datat) o(yp)q(e)j(\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)314 454 y(c)o(har)g(*bu\013er;)314 524 y(in)o(t)f(idxarra)o(y[],)f(n)o(umitems,)f (datat)o(yp)q(e;)210 593 y Fb(INPUT)16 b(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(idxarra)o(y)242 b(p)q(oin)o(ter)14 b(to)g(index)g(arra)o(y)314 872 y(n)o(umitems)215 b(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)314 942 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 942 14 2 v 15 w(INT)p Fd(\))210 1012 y Fb(DESCRIPTION)314 1081 y(MPI)p 413 1081 15 2 v 17 w(BDIdx)d Fd(adds)g(a)h(description)g(of)f(a)g(scatter)i(or)e (gather)h(blo)q(c)o(k)f(to)h(a)f(previously)g(created)i(bu\013er)314 1151 y(descriptor.)19 b(\(It)14 b(is)g(a)g(gather)g(when)g(used)h(in)e(a)h (send)h(and)f(a)f(scatter)j(when)e(used)h(in)e(a)h(receiv)o(e.\))210 1221 y Fb(RETURN)i(V)-5 b(ALUE)314 1291 y(MPI)p 413 1291 V 17 w(BDBlk)13 b Fd(returns)i(0,)e(or)h Fa(\000)p Fd(1)g(if)f(an)h(error)h(o)q (ccurs.)p eop %%Page: 8 8 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(F)l(reeBD)120 b Fd(F)m(ree)15 b(a)e(bu\013er)i(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(F)m(reeBD\()i(b)q(d)f(\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)210 454 y Fb(INPUT)i(AR)o(GUMENTS)314 524 y Fd(b)q(d)347 b(Data)13 b(bu\013er)210 593 y Fb(DESCRIPTION)314 663 y(MPI)p 413 663 15 2 v 17 w(F)l(reeBD)g Fd(frees)i(a)f(bu\013er)g (descriptor)i(created)f(with)f Fc(MPI)p 1354 663 14 2 v 15 w(NewBD)p Fd(.)210 733 y Fb(RETURN)i(V)-5 b(ALUE)314 802 y(MPI)p 413 802 15 2 v 17 w(NewBD)14 b Fd(returns)i(0,)d(or)h Fa(\000)p Fd(1)f(if)g(an)h(error)h(o)q(ccurs.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:03:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10713; Tue, 2 Mar 93 16:03:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26723; Tue, 2 Mar 93 16:01:55 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:01:54 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26694; Tue, 2 Mar 93 16:01:49 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA28558 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:01:47 -0600 Received: from localhost by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00600; Tue, 2 Mar 93 15:01:45 CST Message-Id: <9303022101.AA00600@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Revised Multi-level MPI Suggestion Date: Tue, 02 Mar 1993 15:01:44 -0600 From: Rusty Lusk Before the last meeting we sent out a suggestion on how the potential complexity of the evolving MPI library could be managed by dividing the routines into layers. The idea was to provide a very simple interface at the highest layer, which we call Level 4, and provide all of the flexibility necessary at lower levels, which we called 1 and 2. Since then the February meeting has taken place, and some of our thinking has changed. The basic idea still seems like a good one to us, so here is a new proposal for multiple levels in MPI. At the February meeting, Marc Snir's detailed point-to-point proposal basically covered what we called level 1 (where there are a small number of routines, including "init" routines for operations) and level 2 (where sets of the level 1 routines are bundled together for efficiency, but almost all functionality is preserved). There also seemed to be general agreement that there was no need to have separate routines for supporting communication among heterogeneous machines; rather, contiguous buffers could be specified by (addr,datatype,numitems) instead of (addr,len). The concept of buffer-descriptor (details in another message) would be used to describe noncontiguous buffers and buffers of mixed data types. Therefore it seems unnecessary to have *two* higher levels, since one of the differences between our level 3 and level 4 was the specification of data types. At the same time, it still seems useful to us to have a level of MPI routines that are simpler than the full set specified in Marc's document. We therefore suggest here a level 3 for MPI that imposes a set of restrictions in the interest of simplicity. (So, for example, it will not use buffer descriptors, but restrict users to contiguous buffers of data of the same type.) It corresponds roughly to "current practice", but adds thread safety (no "last message" semantics). It assumes the existence of only one default "all" group and one "universal" context, so that these need not be part of the calls. It does include nonblocking sends and receives, but simplified versions of wait, status, and probe. Users requiring more flexibility can use the routines of level 2 instead. Level 3 MPI routines: (syntax only specified for definiteness) numids() returns number of processes myrank() returns caller's rank in group "all" bsend(dest,tag,buf,datatype,numitems) blocking send (until buffer is empty) brecv(from,tag,buf,datatype,maxitems, blocking receive, with separate actual_from,actual_tag, output arguments to handle wild-card actual_numitems) input arguments Note that lots of programs have been written with just these four calls. handle = nsend(dest,tag,buf,datatype,numitems) non-blocking send handle = nrecv(from,tag,buf,datatype,maxitems, non-blocking receive actual_from,actual_tag, output arguments to handle wild-card actual_numitems) input arguments status(handle) check status of non-blocking operation wait(handle) wait for a non-blocking operation probe(from,tag,actual_from,actual_tag,actual_length) look in queue for matching message, get length so can allocate buffer This is a larger set than the "four-function" interface we proposed as level 4 earlier, but not much, and has the advantage that one can write correct programs even on machines with no buffering. It corresponds roughly to current practice in that most existing programs can be easily converted to this level. It gains simplicity and ease of use by not offering access to MPI's advanced features like noncontiguous buffers of various and mixed types, explicit handle management, high-performance protocols, and elaborate status and wait calls. It has no explicit mention of groups or contexts. The collective operations seem to be getting complicated too, and would benefit from a restricted, simple level. One way to do this would be to add a (blocking) barrier and collective operations in their simplest forms, without explicit mention of group-id's. These Level 3 operations add nothing to MPI that cannot be done at lower levels. However, it seems to us that adopting these simplified versions into the standard will make it easier for many people to use and hence pave the way for its acceptance. Any comments? Bill Gropp & Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:17:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11504; Tue, 2 Mar 93 16:17:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27414; Tue, 2 Mar 93 16:16:32 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:16:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27402; Tue, 2 Mar 93 16:16:25 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA29060 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:16:23 -0600 Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00668; Tue, 2 Mar 93 15:16:21 CST Message-Id: <9303022116.AA00668@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Task identifiers Date: Tue, 02 Mar 93 15:16:20 CST From: Rusty Lusk One issue that has been postponed several times is that of what "process id's" will look like and what they will mean. We would like to offer a suggestion. In this short note we discuss some of the issues and make a concrete proposal. Since the word "process" is so overloaded, we use the word "task" here (possibly only a marginal improvement). We define a task as a program unit that receives a message. For most systems, this is a single process. In a multi-threaded environment, this may be a process or a thread. To allow a process to decide whether to use threads to handle some operations, we believe that a threads within a process should have the same task id; different threads can be identifed by different tags, groups, or contexts. For example, an obvious way to provide a nonblocking user-defined operation is to execute that operation in a separate thread. This should not require a separate task id. The most important issue is whether the number of tasks is fixed or not. If the number is fixed, a sequential numbering of the tasks is possible and simple. However, the trend seems to be toward allowing for the dynamic modificaiton of the number of tasks, and this is impossible if a sequential numbering is used. To avoid restricting MPI to a fixed collection of tasks, we advocate using an opaque task id. This raises the issue of how the task ids of other processes are determined. We suggest that the rank of the task in a specified group be used as a cannonical representation. This rank is required to be in the range 0 to (size of group)-1. The usual arithmetic operations can be performed on the rank. For example, you can define your right neighbor in a ring as ((myrank + 1) mod p), where p is the number of tasks in the ring. Proposal In general, task ids should be an opaque type. A routine gets its own task id with MPI_MyTid. To get the task ids of other tasks, an application may use MPI_TidFromRank(group, rank). A routine gets its own rank with MPI_MyRank(group). It is the task id that is used as a source or destination in the point-to-point message-passing routines. In Fortran it might as well be an integer. Thus the new MPI routines being proposed are: MPI_MyTid() Get caller's task id MPI_MyRank(group) Get caller's rank in group MPI_TidFromRank(group,rank) Get task id of a task from its rank in a group MPI_RankFromTid(tid,group) Get rank in a group from a task id An exception is made for the level 3 routines described in the recent earlier message; these only have one group and we propose that task id and rank in the "all" group be the same for them. This is current common practice. Discussion Note that since a task id is an opaque type, it cannot be sent to another task and used as a task id on that other task. Only the rank is meaningful across tasks. Making the task id an opaque type allows an implementation to optimize the task ids. For example, an implementation may choose to make the task ids the physical node ids used by the hardware. In addition, it encourages users to use library routines to determine neighbors rather than arithmetic on the task id (note that they can still get a neighbor by adding one to their rank). For example, the "best" neighbor in a virtual ring depends on details of the underlying hardware and software; calling a routine such as MPI_NbrInRing to get the neighbor is a more portable way of getting a good neighbor than MPI_RankToTask(MPI_MyRank(group)+1, group ). Bill Gropp and Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:33:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12253; Tue, 2 Mar 93 16:33:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28299; Tue, 2 Mar 93 16:32:14 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:32:13 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28287; Tue, 2 Mar 93 16:32:04 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA29509 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:32:01 -0600 Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00744; Tue, 2 Mar 93 15:31:59 CST Message-Id: <9303022131.AA00744@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: MPI Job startup Date: Tue, 02 Mar 93 15:31:58 CST From: Rusty Lusk We feel that the MPI standard should provide enough information to specify an entire, source-code portable program. To do this, it is necessary to specify how parallel programs start and end. This is the analogue of PROGRAM MAIN ... END in Fortran and main(argc,argv) {...exit(rc); }. This proposal does not deal with methods for spawning tasks during the execution of an MPI program, or for handling explicitly MIMD models (beyond the usual way of starting a Turing machine on all nodes and sending the appropriate tape). This is not meant to suggest that MPI not have such interfaces, just that we are only proposing an SPMD model at this time. Issues It should be possible to write a program that is source-language portable to any system, particularly if it requires only a simple interface to the operating system. An example is an environment that starts up a program with a fixed number of processors. Without something like this, it is not possible to write a portable MPI program. However, no standard should restrict programs to a particular interface. For example, some applications may want to dynamically added and remove tasks during the execution of an application. We leave it to others on the committee to propse a mechanism for this case. Below, we discuss three different approaches; we note that all of these have been tried in some existing system or systems. The seemingly simplest approach is to use an explicit initialization routine, MPI_Init. There are several problems with this approach. The major one is that the parallel part of the program is the code that "follows" the MPI_Init. This is potentially error prone; further, it can be awkward to implement on some systems. This is the way p4 does it. Another approach is to insist that the entire program be run in parallel; a simple way to do this is to replace the "main" program with a routine. In C, this could be MPI_Main( argc, argv ). The linking step then provides a "main" that handles all appropriate startup (including the processing of command line arguments where appropriate for information on the number of tasks etc.) and then calls the user's MPI_main routine. Yet another approach is to provide a routine that runs a user-defined routine in parallel. For example, MPI_Call( argc, argv, routine ) can be used to run routine( argc, argv ) in parallel. A generalization of this would allow the running of a particular routine on a defined group: MPI_Call( groupid, argc, argv, routine ). This is more flexible than the MPI_Main approach but retains the simplicity of running a well-defined program unit (the routine) in parallel. All three of these approaches are currently in use. These are not the most general methods. For example, these provide no mechanism for dynamically creating (remote spawn) tasks on other processors. Also, there is no direct support for MIMD programming. Finally, there is no support for adding an already-running process to a group of running processes. Proposal We propose the MPI_Call form. A slight variation would allow the use of a general argument descriptor instead of argc/argv. This would allow the more convenient passing arguments to the routine. We have used this mechanism (in Bill's Chameleon system) to write programs for nX, the CM-5, the nCube, and networks of workstations using both p4 and PVM, and it has worked well for us. The point is that all source code is portable. The only thing that is different is the shell command by which you start your program. Note that in this case, program termination is handled by having (a) the routine exit and (b) the calling routine exit. Thus an MPI program would look something like this: main(argc,argv) int argc; char **argv; { ret_code = MPI_Call( argc, argv, worker ); exit( irc ); } int worker( argc, argv ) int argc; char **argv; { ... if (error condition) return 1; ... return 0; } In Fortran it would like something like: program main external worker irc = MPI_call(worker) stop end integer function worker ... worker = 0 return end It may be useful to have a more drastic exit. We suggest MPI_exitall(code); the code here is an exit code that may be ignored by the invoking environment (but UNIX-like environments should treat it like exit(code)). In the case of heterogeneous environments, it is necessary to specify the set of workstations to be used. Our only requirement is that this be done, in the C/POSIX case, through command-line arguments (thus maintaining source-code portability). We could specify an extensible, common format for specifying the machines, executables, and resource limits, but, to avoid additional debate, we suggest only that the command line option -mpi be reserved for future extensions. Bill & Rusty From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 18:09:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15496; Tue, 2 Mar 93 18:09:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02859; Tue, 2 Mar 93 18:07:43 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 18:07:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02851; Tue, 2 Mar 93 18:07:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA08222; Tue, 2 Mar 93 23:07:34 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA17367; Tue, 2 Mar 93 16:06:22 MST Date: Tue, 2 Mar 93 16:06:22 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303022306.AA17367@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised Multi-level MPI Suggestion A Suggestion for probe() in the New Rusty and Bill Scheme I'm a bit confused about probe() in Rusty and Bill's latest proposal: ... > probe(from,tag,actual_from,actual_tag,actual_length) > look in queue for matching message, > get length so can allocate buffer > ... > Any comments? > > Bill Gropp & Rusty Lusk > I believe that there was general agreement that probe() should provide the caller with a "handle" to avoid "hidden state" (the thread-safety issue). (The probe() in the current point-to-point proposal does this.) Should the probe() proposed above return "handle"?: handle = probe(from,tag,actual_from,actual_tag,actual_length) If so, we need to add a "recv()" that has "handle" as an argument at Level 3 (basically precv() as I understand it). (This "handle" was also occaisionally referred to as "lock" in the last meeting.) Also, the current proposals all require that a receiving process know the data type(s) in an incoming message before that message is received. This means that datatype can be included in probe() and "numitems" can be used instead of "length". Syntax is then consistent with other routines in the new "Level 3". Here's a counter-proposal (in the spirit of Rusty and Bill's syntax): handle = probe(from, tag, datatype, actual_from, actual_tag, actual_numitems) precv(handle, buf) The use of "handle" guarantees that input parameters to probe() do not need to be repeated in precv(). "maxitems" (as in brecv()) doesn't seem to be necessary since the caller will know how much memory is required before buf is specified. It might be a good idea to add "unlock(handle)" to Level 3. (OR remove probe() from Level 3.) Of course my preference is to dump the whole mess, including buffer descriptor, into an opaque data structure (as in Leslie's and my "message capsule" proposal on Jan 15 in mpi-pt2pt :-). Tom Henderson NOAA Forecast Systems Lab hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 3 16:35:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11369; Wed, 3 Mar 93 16:35:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07168; Wed, 3 Mar 93 16:34:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Mar 1993 16:34:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07160; Wed, 3 Mar 93 16:34:22 -0500 Date: Wed, 3 Mar 93 21:34:17 GMT Message-Id: <6940.9303032134@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Task identifiers To: Rusty Lusk In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:16:20 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Question: How do Rusty and Bill write so many proposals? Answer: Surely they never sleep:-) (Oh well, perhaps I won't get featured on the jokes newsgroups after all.) But seriously, fellow committee members, I recall that the contexts subcommittee is about to prepare one or more proposals which will have considerable bearing on this subject, so perhaps we should again defer "task identifiers" until this has happened. One crucial point raised (again), which I do not see evidence of having really been addressed, is whether MPI will provide a reasonable interface for a static process model, or for a dynamic process model. We have to decide what we want to do here, and I think we ought to clearly state this as potential future users whom I have spoken to are asking just this kind of question. What are the arguments for and against a dynamic process model? To get started: a) The MPP programming world has a whole lot more experience with the static model, and as such it is better aligned with common practice. b) We anticipate people moving toward use of a dynamic process model. c) There are a whole load of issues in the dynamic model which complicate implementations and imply lesser efficiency. d) In order for programs to be portable in the MPI + dynamic process model, the interface to dynamic process control has to be uniform as well as the interface to message passing. Would MPI really consider addressing the question of a uniform interface to process control at this point? At least in the static case the process creation is not part of the program and we can "ignore" it. /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 3 17:06:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12272; Wed, 3 Mar 93 17:06:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08663; Wed, 3 Mar 93 17:04:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Mar 1993 17:04:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08641; Wed, 3 Mar 93 17:04:18 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05267; Wed, 3 Mar 93 16:01:25 CST Date: Wed, 3 Mar 93 16:01:25 CST From: Tony Skjellum Message-Id: <9303032201.AA05267@Aurora.CS.MsState.Edu> To: lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk Subject: Re: Task identifiers Cc: mpi-comm@cs.utk.edu I think that we agreed to defer dynamic issues to a future MPI standard level, but I do agree that this is never guaranteed to happen, as Rusty has said. The lack of control for processes, and the lack of dynamic group syntax/semantics worries me. They will almost certainly appear as non-portable extensions. Perhaps the original N-month limit was too short to permit the best possible standard. - Tony From owner-mpi-comm@CS.UTK.EDU Thu Mar 4 08:24:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27412; Thu, 4 Mar 93 08:24:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22467; Thu, 4 Mar 93 08:22:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Mar 1993 08:22:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22379; Thu, 4 Mar 93 08:22:07 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA02215 (5.65c/IDA-1.4.4 for ); Thu, 4 Mar 1993 14:20:32 +0100 Received: by f1neuman.gmd.de id AA16175; Thu, 4 Mar 1993 14:21:56 GMT Date: Thu, 4 Mar 1993 14:21:56 GMT From: Rolf.Hempel@gmd.de Message-Id: <9303041421.AA16175@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Process ids Cc: gmap10@f1neuman.gmd.de recently there has been a discussion on process identifiers, groups, and dynamic process creation. I would like to add a few comments. 1. It is not so much the dynamic creation of new processes which calls for something like opaque process ids, but rather the dynamic removal of processes. In the former case process ids can be added at the end (n, n+1, ...) without changing the 0,..,n-1 numbering of the existing processes. If a process terminates, however, there is a gap in the numbering scheme. On the other hand, the same gap occurs in the group ranking which also Bill and Rusty want to keep. So, I don't see how the opaque process ids resolve the problem. 2. As Bill and Rusty say at the end of their note, inquiries like "give me my right neighbor" should be done by using the process topology functions, and not on the ranks themselves. Only this way an optimization of the process ordering is possible. The ranks mainly serve to define the buffer ordering in collective communication. 3. At our last meeting we already saw that the MPI time schedule is becoming tight. I don't think it would be wise to deal with dynamic process creation in MPI-1, and as far as I remember there was already some consensus on that. Of course, in the design of the static interface we have to keep in mind that dynamic extensions will be added later. It should not be seen as a problem if on the basis of MPI-1 non-standard extensions will emerge. After all, only this kind of experimenting can make a later standardization possible. Rolf Hempel From owner-mpi-comm@CS.UTK.EDU Sat Mar 6 15:38:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03415; Sat, 6 Mar 93 15:38:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12502; Sat, 6 Mar 93 15:37:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 6 Mar 1993 15:37:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12470; Sat, 6 Mar 93 15:36:41 -0500 Date: Sat, 6 Mar 93 20:36:37 GMT Message-Id: <390.9303062036@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: dynamic processes and dynamic groups To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Hi again Well, we certainly are having some fun now in this discussion of static vs dynamic processes in MPI. This is as much about dynamic groups as dynamic processes, and the latter part is really about MPI as a standards committee. I have a lot of sympathy with John Flower's concerns. There appears to be reasonable consensus among those contributing to the discussion that in a world process model where processes dynamically appear and dissapear in an asynchronous fashion an enumeration of processes is not a useful way of identifying processes - what sense could we make of the enumeration? An opaque type process (or task) identifier is better. The same considerations surely lead us to the observation that in a world group model where processes dynamically join and leave groups in an asynchronous fashion an enumeration of members of a group is not a useful way of identifying members - what sense could we make of the enumeration? Rolf previously pointed this out on mpi-comm - March 4, subject "Process ids". What are asynchronously created (terminated) processes to do? I guess they either join (leave) an existing group, or create (destroy) a group for their own purposes. The primitives for these two are really process group resizing (or destruction/recreation) and process group creation (termination). I guess their must be heavyweight PVM users who have other experiences of usage of dynamic process creation/termination - Al?. These thoughts/questions, however interesting and discussion provoking they may or may not be, do not take us any further on answering some of the BIG questions about what MPI will be, regarding for example: a) Process model; static v dynamic; ... b) Group model; static v dynamic; ... I've previously stated that I'd vote for static process model in (a). This "seemed to be" the intent of MPI (although I couldn't find it written down and recorded as agreed), would be useful, and could be agreed upon within the MPI time schedule. This is not to say that I think limitation to a static process model would be the most useful thing upon which to agree to describe a 'de facto' standard. I have pretty well the same opinions regarding process groups. I'd like us all agree on how we're going to move forward and get the details of the standard described, as I'm concerned that at the current rate we may not. I'll broadly support John Flower's suggestions, primarily because I don't see any others, and I'll do my best to implement whatever strategy we can agree on. Final thought, perhaps we should move the discussion of MPI strategy to mpi-comm. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 08:01:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04198; Mon, 8 Mar 93 08:01:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19241; Mon, 8 Mar 93 07:59:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 07:59:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19232; Mon, 8 Mar 93 07:59:19 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03238 (5.65c/IDA-1.4.4 for ); Mon, 8 Mar 1993 07:59:13 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA10283; Mon, 8 Mar 93 12:59:09 GMT Date: Mon, 8 Mar 93 12:59:09 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303081259.AA10283@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02380; Mon, 8 Mar 93 12:56:19 GMT To: lusk@mcs.anl.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:31:58 CST <9303022131.AA00744@donner.mcs.anl.gov> Subject: MPI Job startup Content-Length: 2990 > We feel that the MPI standard should provide enough information to > specify an entire, source-code portable program. I agree. MPI_INIT style ============== > The seemingly simplest approach is to use an explicit initialization > routine, MPI_Init. There are several problems with this approach. > The major one is that the parallel part of the program is the code > that "follows" the MPI_Init. This is potentially error prone; > further, it can be awkward to implement on some systems. This is > the way p4 does it. I agree that this is simplest. I also prefer it to the others. I don't see the "major problem" outlined above. The whole of the code is parallel even before the MPI_INIT, it's just that it can't communicate yet. I'd expect code like this (given an I/O everywhere model) to execute fine int main(int argc,char ** argv) { printf("Hello world\n"); } and print as many "Hello world" messages as the number of nodes on which I ran it. It also seems to me to be quite an easy way to allow the dynamicness everyone seems to want. The newly created process starts up, and then attaches to the existing MPI environment at the time it calls MPI_INIT. In my view, then, what happens is 1) By some unspecified means a number of processes start to execute 2) They decide to collaborate by executing MPI_INIT [If dynamic process creation is permitted 2a) New processes are forked/execed 2b) The new processes join the existing collaboration by executing MPI_INIT ] 3) They each exit (by calling exit), or you can have a drastic exit if you like. MPI_CALL style ============== I don't think I understand what this is intended to mean, which may be why I don't like it ! Therefore please can you explain it to me. 1) Is the idea that there is one user process which then executes MPI_CALL at which point all of the other processes are created,and magically arrive in a user subroutine, or do all the processes execute main ? 2) If all the processes execute main, what can they do before calling MPI_CALL ? 3) What global state is accessible to the MPI_CALLed routine ? e.g. (your example with minor change...) extern int foo; /* Of course this could be an arbitrarily complex data structure ! */ main(argc,argv) int argc; char **argv; { foo = time(); /* Something which may differ on each node ! */ ret_code = MPI_Call( argc, argv, worker ); exit( irc ); } int worker( argc, argv ) int argc; char **argv; { printf("%d: %d\n", MPI_MYTID(), foo); ... if (error condition) return 1; ... return 0; } At the moment I definitely prefer MPI_INIT ! -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 09:19:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05513; Mon, 8 Mar 93 09:19:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22555; Mon, 8 Mar 93 09:18:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 09:18:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22546; Mon, 8 Mar 93 09:18:01 -0500 Date: Mon, 8 Mar 93 14:17:41 GMT Message-Id: <1439.9303081417@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Job startup To: Rusty Lusk , mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:31:58 CST Reply-To: lyndon@epcc.ed.ac.uk Hi there Spring has arrived in Scotland :-) For the thrid say this year we have beautiful sunshine :-) The snow on the Caringorms is meling fast :-( Now it's MPI job startup. Thanks to Bill and Rusty for starting this one off. The MPI_INIT style is familiar. It makes it explicit that the bit of the program before MPI_INIT is executing in parallel, and no MPI services can be used before MPI_INIT. As Jim correctly points out. It documents to the user what s/he is using. The trouble is, that the user will almost always end up calling MPI_INIT right after entry into main, because what can the user usefully do before MPI_INIT? We should consider providing this approach. I really don't like the MPI_CALL approach. It serves to confuse the user. It looks like there is one main running, and the function reference in MPI_CALL magically runs in parallel. What of use can the user do before MPI_CALL? It looks to the user like the static (global) data is all accessible everywhere. I don't think this is intended, and I don't think this is what should be provided. What about the other approach, in which the user provides a "virtual main" and the MPI library supplies a "actual main", which calls the virtual main. I guess the virtual main calls MPI_INIT and then calls the actual main. This removes the necessity for the user to call MPI_INIT, and means that s/he cannot make the mistake of attempting to use MPI services before they are available. We *might* be able to get this past C programmers. I believe that we would have enormous problems trying to get this past crusty fortran programmers --- "What, you mean I have to write a subroutine instead of a program?". Wait up though, this is what is happening anyway. Your process never starts in your own main. Your "actual main" is a "virtual main", and the compiler/stdlib/runtime executes a preamble to main. FOURTH suggestion ----------------- There is no MPI initialisation procedure. The user program is "attatched to MPI" and can use MPI services as soon as the user program starts to execute. This means that we require system vendors to embed any MPI initialisation his/her implementation requires in the preamble before main. Dead simple. Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 09:25:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05746; Mon, 8 Mar 93 09:25:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22903; Mon, 8 Mar 93 09:24:39 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 09:24:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22895; Mon, 8 Mar 93 09:24:31 -0500 Date: Mon, 8 Mar 93 14:24:19 GMT Message-Id: <1449.9303081424@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Job startup To: lyndon@epcc.ed.ac.uk, Rusty Lusk , mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 8 Mar 93 14:17:41 GMT Reply-To: lyndon@epcc.ed.ac.uk Ooops - small oversight. > This means that we require system vendors to embed any MPI > initialisation his/her implementation requires in the preamble before > main. > Not quite true, Lyndon, you missed something :-) On the other hand all of the MPI service routines could do one check of a static boolean to see if MPI has been initialised, and if not then call an initialisation routine which inverts the static boolean. I just prefer if it happened for sure in the preamble. Best Wishes Lyndon ps replying to my own mail, hmmm :-) /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 10:51:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09115; Mon, 8 Mar 93 10:51:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28354; Mon, 8 Mar 93 10:50:07 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 10:50:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28346; Mon, 8 Mar 93 10:50:04 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA00863; Mon, 8 Mar 93 10:49:41 EST Date: Mon, 8 Mar 93 10:49:41 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9303081549.AA00863@cs.wmich.edu> To: mpi-comm@cs.utk.edu hi; To add to the "thread" on MPI init. I think there should be some form of MPI init call before any other MPI call. This allows MPI implementations over other systems, without a small but built in overhead. I don't feel religous about this though. Assuming an MPI init call of some form is added, the only questions I would have would be - 1) Does each thread need to make the call - or is one call per process OK? and 2) In the case of only requiring on call per process, Is it OK to make more than one call to the init routine - as in the case of each thread in a process making a call before starting. If there is any state information kept on a per thread basis - than one call per thread may be reasonable. I know we tried to keep all state per thread information out (which I view as very reasonable), but unless it is a 100% requirement, one MPI init call per process may not be enough. john From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 15:11:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17797; Mon, 8 Mar 93 15:11:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13204; Mon, 8 Mar 93 15:08:17 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 15:08:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13156; Mon, 8 Mar 93 15:08:08 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 8 Mar 93 12:04 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA16386; Mon, 8 Mar 93 12:02:56 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA14497; Mon, 8 Mar 93 12:02:53 PST Date: Mon, 8 Mar 93 12:02:53 PST From: d39135@sodium.pnl.gov Subject: RE: MPI Job startup To: lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303082002.AA14497@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Here are some comments I haven't seen yet about MPI job startup suggestions. Lyndon Clarke's FOURTH suggestion: > There is no MPI initialisation procedure. The user program is > "attatched to MPI" and can use MPI services as soon as the user program > starts to execute. > > This means that we require system vendors to embed any MPI > initialisation his/her implementation requires in the preamble before > main. This approach has the disadvantage that it requires intimate collaboration with the vendor's runtime language-support library. The interfaces to those libraries are usually undocumented and subject to change without notice. This means that one needs vendor cooperation to write and maintain an MPI implementation using this approach. From the MPI user's standpoint, "FOURTH suggestion" seems indistinguishable from the earlier proposal that there be no explicit initialization call, and that initialization be done by having each MPI routine check a static boolean. A minor problem with the static boolean suggestion is that then the initialization routine does not have access to those parts of the environment passed as (argc,argv). Such access is sometimes useful. (For example... TCGMSG's static process initializer uses command-line arguments to inform each process of its place in the grand scheme of things. In C, the command-line arguments are accessible to TCGMSG because the application has to pass them to the TCGMSG initialization procedure. (In Fortran, TCGMSG's initialization procedure is called without arguments, and it has to hunt up the argc and argv that presumably stored by Fortran initialization in global variables. See above comments about undocumented interfaces.) The MPI_Main approach (MPI does the initialization, user provides a subroutine) seems to have the disadvantage that it works for only one package. If MPI takes this approach, then I don't see how any other package FOO can too, if an application wants to use both MPI and FOO. MPI_Call has the slight problem that if another package FOO adopts the MPI_Call approach, then we end up with main (argc,argv) { MPI_main (argc,argv,myMPIinitializer); } myMPIinitializer (argc,argv) { FOO_main (argc,argv,myFOOinitializer); } myFOOinitializer (argc,argv) { myapplication (argc,argv); } compared to main (argc,argv) { MPI_Init (argc,argv); FOO_Init (argc,argv); myapplication (argc,argv); } Certainly this is not an enormous problem, but I don't see where the extra complexity buys anything. I like MPI_Init (augmented if necessary for dynamic processes). --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 15:52:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19445; Mon, 8 Mar 93 15:52:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18884; Mon, 8 Mar 93 15:51:31 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 15:51:30 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18876; Mon, 8 Mar 93 15:51:28 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA06355 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Mon, 8 Mar 1993 12:51:24 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA18286; Mon, 8 Mar 93 12:44:12 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA15256; Mon, 8 Mar 93 12:44:15 PST Date: Mon, 8 Mar 93 12:44:15 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303082044.AA15256@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: MPI_INIT To throw in my two cents worth..... The business with argc and argv (and, of course environ) has been a thorn in the side of Express for many years. We've adopted a standard two-faced approach in which C programmers get their "main" routine renamed (silently) to exp_main which we then call from our own main. In this case argc and argv get dealt with by Express and passed to the main routine. This approach has the disease mentioned by Rik, that two or more different software suppliers may want to pull the same trick causing confusion. In fortran we went the way of MPI_INIT with no arguments. This seems to work but has the difficulty that things beyond the ken of the message passing system can be affected by it. For example, the Express program program foo write(6,*) 'I have started' call kxinit -- i.e., MPI_INIT write(6,*) 'Done with kxinit' stop end is actually wrong because the I/O system needs to know parameters that don't get set till the call to kxinit. We could, of course, enforce the rule in MPI that only MPI routines are affected by the MPI_INIT call and that, therefore, you can put anything you like before the call to MPI_INIT as long as its name doesn't start with MPI_!!!!! I dont think this is going to work in the long run since it actively discourages library vendors from building their codes with MPI routines!!! Also, in Rik's example, I think the code has to look sort of like main(argc, argv) int argc; char **argv; { MPI_INIT(&argc, &argv); FOO_INIT(&argc, &argv); /* etc... */ if we want to allow for the chance that it's actually MPI_INIT that passes down the arguments....... this is pretty ugly...... My personal vote would be for making the user rename their top level entry point: MPI_main(int argc, char **argv, char **environ) in C and MPI_MAIN in fortran Perhaps some compiler level support for potentially changing the name of the entry point so that third party software vendors could build their own "front-ends" that puts their own code first?? I appreciate that this is definitately NOT current practice and has all sorts of diseases of its own with different software suppliers providing layer upon layer of "fake" compiler front-ends. Another option that I would probably support would be bagging argc/argv altogether at the MPI level. In this case a function call with no arguments that had to be the first thing you did would be OK -- if you do it wrong you probably get a core dump! Express' attempts to deal with argc and argv date back to the days when hardware vendors didn't provide these things and we wanted to ourselves. Now all the systems I know of start up C programs apropriately and the need for Express (or some other system) to get in behind them is not so crucial...?.... A slight counter-example to my own argument, however, is that some systems give you mangled argc/argv and particularly environ. This can lead to bizarre results. (Systems which permit a remote host capability sometimes give you the environment that you would have had on the local host rather than the one you actually have on the remote host.) Jon From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 20:07:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23824; Mon, 8 Mar 93 20:07:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02443; Mon, 8 Mar 93 20:06:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 20:06:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02433; Mon, 8 Mar 93 20:05:58 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 8 Mar 93 17:01 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA17034; Mon, 8 Mar 93 17:00:40 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15033; Mon, 8 Mar 93 17:00:38 PST Date: Mon, 8 Mar 93 17:00:38 PST From: d39135@sodium.pnl.gov Subject: Re: MPI_INIT To: jwf@lion.Parasoft.COM, mpi-comm@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303090100.AA15033@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Jon Flower says: > Also, in Rik's example, I think the code has to look sort of > like > > main(argc, argv) > int argc; > char **argv; > { > MPI_INIT(&argc, &argv); > FOO_INIT(&argc, &argv); > /* etc... */ Yeah, sort of. The example I showed was based on one version of current practice (TCGMSG). It doesn't let the INIT's alter the argument list. In general, this implies far worse ugliness than Jon's example, because it requires that applications know enough to ignore arguments intended for the INIT's. A better version of current practice is the X window system, one of whose initializers looks like XtInitialize ( ... &argc, argv) This allows deleting stuff from the argument list, but not adding. Jon's syntax goes farther and allows both. But maybe this is working the wrong issue. I said: > A minor problem with the static boolean suggestion is that then the > initialization routine does not have access to those parts of the > environment passed as (argc,argv). Such access is sometimes useful. It could also be said that the problem is with the (argc,argv) approach. I defended the (argc,argv) approach, sort of, but I confess that I don't like it very much precisely because it DOES prevent calling the MPI initializer from inside some other routine. I ran afoul of this just a few days ago, when I tried to emulate another package using TCGMSG, and discovered that my application called the other package's initializer from several procedures down. What I would *really* like (putting on my user's hat), is an explicit MPI_INIT with no arguments, like Jon suggests. I guess this means that implementors have to find some way to pass environment info other than through the argv list. Can anyone come up with a convincing reason why this is a big problem? --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 00:01:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27466; Tue, 9 Mar 93 00:01:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10468; Tue, 9 Mar 93 00:00:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 00:00:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10455; Tue, 9 Mar 93 00:00:22 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08899; Mon, 8 Mar 93 22:56:45 CST Date: Mon, 8 Mar 93 22:56:45 CST From: Tony Skjellum Message-Id: <9303090456.AA08899@Aurora.CS.MsState.Edu> To: jwf@lion.Parasoft.COM, mpi-comm@cs.utk.edu, d39135@sodium.pnl.gov Subject: Re: MPI_INIT; environment information Guys, What about using envp? That would provide the Unix-compatible, open-ended approach to passing information on the environment? Info could be added etc, as was suggested as a need. This is a layerable-friendly approach. ie, main(argc,argv,envp) int argc; char **argv; char **envp; 1) For Fortran, the getenv_() (or equivalent) call provides the interface. C could use the direct interface to envp, or the getenv() call. We could have mpi_getenv() to avoid conflicts with general system support, if absolutely needed to avoid POSIX interactions. 2) Handling of argc/argv A) argc, argv is made accessible to C through standard access. Made accessible to Fortran through accessor functions (eg, mpi_argc(), mpi_argv(i,string,maxlen)). B) argc, argv could be obliterated for both models (ie, left empty for C, inaccessible to Fortran). All info through mpi_getenv(). May I throw in a comment about main precursors (niam() in the BBN parlance)? This is great if supported by the linker, and a pain if not supported by the linker. If niam() is absent, then the default niam() is added, which just calls main(). If niam() is present, then the user model can choose. However, what if different programs presume the abilility to use niam(). They would not stack up... Without niam(), has to assume preprocessor substitution, or macro substitution. This requires that user uses an MPI-compatible make procedure, or compiler pseudonym, other than naked cc/f**. This turns out to be a thorny problem to handle transparently. Also, the execution model of the CM-5, for instance, does not [did not] require main to exist at all. Execution of arbitrary functions (not only main) is supported by CMMD 2.0, specifically. I don't know what CMMD 3.0 chooses to do, but you get the idea. main/program main access is an assumption true most everywhere, but not everywhere. - Tony From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 15:24:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13894; Tue, 9 Mar 93 15:24:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24607; Tue, 9 Mar 93 15:14:24 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 15:14:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24595; Tue, 9 Mar 93 15:14:18 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA24881; Tue, 9 Mar 93 20:13:49 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA06208; Tue, 9 Mar 93 13:12:34 MST Date: Tue, 9 Mar 93 13:12:34 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303092012.AA06208@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: MPI Goals and Scope: More Details Please? As a potential user of MPI, the discussion of I/O below disturbs me a bit and leads to a whole bunch of questions... > > We feel that the MPI standard should provide enough information to > > specify an entire, source-code portable program. > > Bill & Rusty > I agree. > > I'd expect code like this (given an I/O everywhere model) to execute > fine > > int main(int argc,char ** argv) > { > printf("Hello world\n"); > } > > and print as many "Hello world" messages as the number of nodes on > which I ran it. > > -- Jim I also agree with Bill and Rusty's statement about portability. Jim's example brings up a whole bunch of interesting problems. First, should the printf() really print one "Hello world" message per node (process)? I can think of several places where I wouldn't expect that behavior (Express "single" I/O mode for example). The next problem is "ordering". Suppose Jim's example is modified slightly: int main (int argc, char **argv) { int me; MPI_INIT(???); /* Yes, I'm casting my vote for MPI_INIT(). */ me = MPI_MYRANK(); /* another guess at syntax */ printf("Hello world from process %d\n", me); } Now what happens? Do we see first-come first-served? Do we see the "intuitive" ordering? Do we get an error message? At the last meeting a straw vote lead to the adoption of "one processor can do standard I/O". If we do this, then users will use vendor-supplied "parallel I/O" routines to get decent performance. We cannot claim that the MPI standard provides enough information to specify an entire, source-code portable program unless we include I/O. I think that there is general agreement that "parallel I/O" will not be included in MPI-1. Maybe we could re-state the portability goal: The ultimate goal of the MPI standardization effort is to provide enough information to specify an entire, source-code portable program. I certainly won't use MPI unless I believe that this will happen with some version of MPI. I think we should re-examine the scope of the MPI effort and write down what we want to accomplish with MPI-1, MPI-2, etc. in much more detail than is currently in the draft introduction. John Flower has proposed one prioritization. Here's another (from my perspective as a user): 1) Static process creation and identification (and associated environmental inquiry routines). 2) Contexts. 3) Heterogeneous communication semantics (I think we already have this). All communication routines should support heterogeneous communication. 4) Blocking send and receive, contiguous data. 5) Non-blocking send and receive, contiguous data (status(), wait(), cancel()?). 6) Creation/destruction of "static" groups (by "static" I mean processes cannot join/leave an existing group). Grid/toroidal process topology would go here if we decide to include it. 7) Blocking collective communication, contiguous data (broadcast, multiple-broadcast, concatenate, barrier, etc.). 8) "Simple" commonly used blocking collective operations (sum, max, min, etc.). 9) Blocking send and receive, strided data. 10) Non-blocking send and receive, strided data. 11) Blocking collective communication, strided data. 12) Blocking send and receive with buffer-descriptor. 13) Non-blocking send and receive with buffer-descriptor. 14) Blocking collective communication with buffer-descriptor. 15) "User-defined" blocking collective operations (Express "excombine()" style). 16) Probe (and associated "precv()" routines, if necessary). 17) Non-blocking collective communication, contiguous data, strided data, and buffer-descriptor (and "multi-cancel()"?). 18) Definition of parallel I/O. 19) Blocking parallel I/O. 20) Non-blocking parallel I/O. 21) Additional environmental inquiry and "setting" routines for performance optimization (suggestions for system buffering, relative node/interconnect performance, etc.). 22) Dynamic process creation and identification. 23) Dynamic groups (processes can join/leave an existing group). 24) "Safe" send and receive. 25) Ready-send and ready-receive. 26) General process topologies. 27) Even more neat stuff. Once a detailed list like this is agreed on, we need to decide where to draw the line between MPI-1 and MPI-2 (and maybe MPI-3, etc.). Various straw polls about I/O indicate that 18-20 will probably be in MPI-2. I imagine an absolute minimum might be 1-5 for MPI-1. Here's three proposals for the list above: MPI-1 MPI-2 MPI-3, etc. "Ambitious" 1-17 18-26 27 "Less Ambitious" 1-14 15-23 24-27 "Conservative" 1-11 12-19 20-27 I share John Flower's "EXTREME concern" about the complexity of the current design. I believe that the MPIF must agree on SOME prioritization list and explicitly decide what is going to be in MPI-1 as soon as possible. I also feel strongly that we should decide what will be in future MPI versions to give potential users some confidence that MPI will eventually be useful for developing source-code portable programs. Tom Henderson (a potential MPI user) NOAA Forecast Systems Laboratory hender@fsl.noaa.gov (Thanks to Leslie Hart for some helpful suggestions.) From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 20:46:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17983; Tue, 9 Mar 93 20:46:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10433; Tue, 9 Mar 93 20:45:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 20:45:19 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10412; Tue, 9 Mar 93 20:45:16 -0500 Date: Wed, 10 Mar 93 01:45:05 GMT Message-Id: <2554.9303100145@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Goals and Scope: More Details Please? To: hender@macaw.fsl.noaa.gov (Tom Henderson), mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Tue, 9 Mar 93 13:12:34 MST Reply-To: lyndon@epcc.ed.ac.uk Hi all There seems to be something to be discussed in what Tom is saying. Here is my 2 pennies worth. Please understand my multicomputer world bias, or flame me for it, whichever your prefer. > > > We feel that the MPI standard should provide enough information to > > > specify an entire, source-code portable program. > > > Bill & Rusty > > > I agree. This is going to sound awful picky, but I'm afraid I gotta say that I disagree with this. It's a grand ambition, and its also a grand challenge, in fact I conjecture that MPI is implicitly incapable of completing the stated mission even given arbitrary time, and that it is not a valid goal for MPI. The goal requires MPI to describe a whole raft of issues which are outwith MPI. I reckon John Flowers could spend the rest of his MPI time telling us about his experiences in Parasoft and Express with the issues that MPI cannot address! So MPI needs a more restricted mission. I'm too sleepy to try right now to find a suitable modification of the above sentence which might serve as a better mission for MPI. > > At the last meeting a straw vote lead to the adoption of "one processor can do > standard I/O". If we do this, then users will use vendor-supplied > "parallel I/O" routines to get decent performance. We cannot claim that the > MPI standard provides enough information to specify an entire, source-code > portable program unless we include I/O. I think that there is general > agreement that "parallel I/O" will not be included in MPI-1. Don't you think it will be some time before the world is ready to even think about standardisation of "parallel I/O". I can't imagine trying to do such a thing at the moment. > Maybe we could > re-state the portability goal: > The ultimate goal of the MPI standardization effort is to provide enough > information to specify an entire, source-code portable program. I have the same problem with this mission as that above. > I certainly won't use MPI unless I believe that this will happen with some > version of MPI. I reckon you will, actually, because although it won't give you the portability asked for above, it will be your best way of getting some portability. > I think we should re-examine the scope of the MPI effort and write down what > we want to accomplish with MPI-1, MPI-2, etc. in much more detail than is > currently in the draft introduction. I concur, in an ambivalent fashion. The introduction is vague, I'll agree with that. I think we are currently working out what we should be doing, in a rather ad hoc fashion. > John Flower has proposed one > prioritization. Here's another (from my perspective as a user): I'm going to walk down this list and give marks out of ten, with ten as highest priority and 0 as lowest priority, from my point of view, and some justification, from my point of view. > 1) Static process creation and identification (and associated environmental > inquiry routines). 10 - bare minimum MPI can acheive without being complete waste of time. > 2) Contexts. 9 - can get away with just tags but would impede library development. > 3) Heterogeneous communication semantics (I think we already have this). > All communication routines should support heterogeneous communication. 6 - static SPMD programs are most sensible in homogeneous machines. > 4) Blocking send and receive, contiguous data. 10 - bare minimum MPI can acheive without being complete waste of time. > 5) Non-blocking send and receive, contiguous data (status(), wait(), > cancel()?). 10 - bare minimum MPI can acheive without being complete waste of time. > 6) Creation/destruction of "static" groups (by "static" I mean processes > cannot join/leave an existing group). Grid/toroidal process topology > would go here if we decide to include it. 8 - can get away with just tags but would impede library development. > 7) Blocking collective communication, contiguous data (broadcast, > multiple-broadcast, concatenate, barrier, etc.). 10 - bare minimum MPI can acheive without being complete waste of time. > 8) "Simple" commonly used blocking collective operations (sum, max, min, > etc.). 10 - bare minimum MPI can acheive without being complete waste of time. > 9) Blocking send and receive, strided data. 8 - useful, but not essential. > 10) Non-blocking send and receive, strided data. 8 - useful, bit not essential. > 11) Blocking collective communication, strided data. 5 - makes collective communications too big. > 12) Blocking send and receive with buffer-descriptor. 8 - useful, but not essential. > 13) Non-blocking send and receive with buffer-descriptor. 8 - useful, but not essential. > 14) Blocking collective communication with buffer-descriptor. 5 - makes collective communications too big. > 15) "User-defined" blocking collective operations (Express "excombine()" > style). 10 - bare minimum MPI can acheive without being complete waste of time. > 16) Probe (and associated "precv()" routines, if necessary). 0 (boy, am I biased! :-) > 17) Non-blocking collective communication, contiguous data, strided data, and > buffer-descriptor (and "multi-cancel()"?). 5 - makes collective communication to big, and has technical complications. > 18) Definition of parallel I/O. 2 - too early to think about standardisation. > 19) Blocking parallel I/O. 2 - too early to think about standardisation. > 20) Non-blocking parallel I/O. 2 - too early to think about standardisation. > 21) Additional environmental inquiry and "setting" routines for performance > optimization (suggestions for system buffering, relative > node/interconnect performance, etc.). 2 - useless without better knowledge of how to use them. > 22) Dynamic process creation and identification. 6 - process group creation and resize is tractable model > 23) Dynamic groups (processes can join/leave an existing group). 2 - asynchronous join/leave very experimental 6 - process group creation and resize is tractable model > 24) "Safe" send and receive. 10 - bare minimum MPI can acheive without being complete waste of time. > 25) Ready-send and ready-receive. 0 - what benefit except for some old machines? > 26) General process topologies. 8 - if we have topologies at all, then should do this. > 27) Even more neat stuff. ? - ??? [stuff deleted] I think you can see what my staged schedule would be from the above. > > I share John Flower's "EXTREME concern" about the complexity of the current > design. I believe that the MPIF must agree on SOME prioritization list and > explicitly decide what is going to be in MPI-1 as soon as possible. I concur, ambivalently, as I think that it will yet take a little time for a strong consensus to emerge. I applaude efforts to generate such a consensus. > I also > feel strongly that we should decide what will be in future MPI versions to > give potential users some confidence that MPI will eventually be useful for > developing source-code portable programs. I am again ambivalent. I think it important to identify what items should be suitable for future MPI, but I'm not at all confident that it will be practical to commit ourselves to a schedule for trying to standardise such items. It will be harder to agree on standards for the items which are deferred. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 08:14:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28523; Wed, 10 Mar 93 08:14:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11106; Wed, 10 Mar 93 08:13:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 08:13:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11098; Wed, 10 Mar 93 08:13:21 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA20589; Wed, 10 Mar 93 08:13:19 EST Received: by b125.super.org (4.1/SMI-4.1) id AA01744; Wed, 10 Mar 93 08:13:18 EST Date: Wed, 10 Mar 93 08:13:18 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303101313.AA01744@b125.super.org> To: mpi-comm@cs.utk.edu Subject: MPI directions & priorities I was in the process of composing a note to the committee about what MPI-1 should compose and was letting it sit overnight when Tom sent out his message. Since he has done a nice job I will shorten my comments. At the last meeting there were several group and private discussions about the growing complexity of MPI. Al has posted his feeling that point-to-point and collective communications get complex and untested ideas added but other areas such as dynamic processes are not considered. Jon posted about the complexity and now Tom has started a list of priorities. I add my voice to these concerns. I have been hesitant to post because I don't want it to seem that I think MPI is hopeless. However, I think it is clear that we cannot adequately complete all the current proposals in 3 more meetings. I would propose that we continue the discussion concerning priorities via e-mail and begin the next meeting with a whole group meeting to set our course. At the higher level we need to decide: 1) How long will the MPI-1 process be allowed to take? Are we going to stick to the 6 month deadline or try and extend it? 2) What is going to be in MPI-1? 3) Of the things we leave out of MPI-1 are they going to be proposed for a later version of MPI? Should we make sure that MPI-1 is compatible with the future goals so that major changes in the base routines is not needed later? 4) If we want to have MPI-#, what mechanism and time scale will be used to get it done. I think it is past time to find the forest for the trees. Global planning to focus our efforts now could significantly reduce our discussions and efforts in the short term as deadlines approach. Furthermore, with a global blueprint we might see a more even handed approach to different aspects of the standard. These are just some thoughts. I hope the discussion will continue. Steve From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 10:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02371; Wed, 10 Mar 93 10:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16993; Wed, 10 Mar 93 10:13:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:13:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16985; Wed, 10 Mar 93 10:13:18 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA03655 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 09:13:16 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05373; Wed, 10 Mar 93 09:13:13 CST Date: Wed, 10 Mar 93 09:13:13 CST Message-Id: <9303101513.AA05373@godzilla.mcs.anl.gov> To: lederman@b125.super.org Cc: mpi-comm@cs.utk.edu In-Reply-To: Steve Huss-Lederman's message of Wed, 10 Mar 93 08:13:18 EST <9303101313.AA01744@b125.super.org> Subject: MPI directions & priorities One approach that we could take is to describe a set of MPI that is extensible to the more elaborate facilities but does not yet specify them. For example, we could keep the goal of being "thread-safe" and resolve (at least in an extensible way) the group/context/process-id issues, and then settle on, for example, somethine like what Rusty and I have called level 3, plus some collective communication routines. This would not have everything that I would want or even consider important, but would be a usable set for casual use. This is a very different process than just defining a simple set of operations; it MUST be possible to extend, in a consistent way, MPI to meet new needs. I'd like to point out that many of the so-called new features are ones that are in use in some (usually only one) system; in most cases they fulfill a perceived need. For MPI to be successful, it must not be incompatible with these needs. I realize that the other side of this is that, by specifying only a subset of desired functionality, there is the danger of mutually incompatible extensions. We need to weigh this against the possibility that MPI will become too complex to be accepted. Bill From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 10:24:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02512; Wed, 10 Mar 93 10:24:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17488; Wed, 10 Mar 93 10:22:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:22:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17476; Wed, 10 Mar 93 10:22:29 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA03990 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 09:22:24 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05385; Wed, 10 Mar 93 09:22:22 CST Date: Wed, 10 Mar 93 09:22:22 CST Message-Id: <9303101522.AA05385@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Wed, 10 Mar 93 01:45:05 GMT <2554.9303100145@subnode.epcc.ed.ac.uk> Subject: MPI Goals and Scope: More Details Please? I'd like to clarify what we mean by > We feel that the MPI standard should provide enough information to > specify an entire, source-code portable program. This is NOT intended to mean that MPI must provide a way to specify ANY (or even most) source-code portable parallel program. Rather, given restrictions strong enough to satisfy a 2/3rds majority ( :) ), a way to specify at least one source-code portable program. In the spirit of extensibility, whatever mechanism is proposed for this simple interface should not preclude non-MPI solutions for more demanding situations. Bill From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 11:33:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03903; Wed, 10 Mar 93 11:33:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21975; Wed, 10 Mar 93 11:31:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 11:31:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21960; Wed, 10 Mar 93 11:31:39 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA10050; Wed, 10 Mar 93 10:27:36 CST Date: Wed, 10 Mar 93 10:27:36 CST From: Tony Skjellum Message-Id: <9303101627.AA10050@Aurora.CS.MsState.Edu> To: lederman@b125.super.org, gropp@mcs.anl.gov Subject: Re: MPI directions & priorities Cc: mpi-comm@cs.utk.edu We must agree to reconvene about extensions in future. I also agree that it would be preferable to specify a whole portable program, but I think the I/O subject is hard to address right now... - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 09:42:01 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:13:21 EST From: William Gropp Date: Wed, 10 Mar 93 09:13:13 CST To: lederman@b125.super.org Cc: mpi-comm@cs.utk.edu Subject: MPI directions & priorities Content-Length: 1193 One approach that we could take is to describe a set of MPI that is extensible to the more elaborate facilities but does not yet specify them. For example, we could keep the goal of being "thread-safe" and resolve (at least in an extensible way) the group/context/process-id issues, and then settle on, for example, somethine like what Rusty and I have called level 3, plus some collective communication routines. This would not have everything that I would want or even consider important, but would be a usable set for casual use. This is a very different process than just defining a simple set of operations; it MUST be possible to extend, in a consistent way, MPI to meet new needs. I'd like to point out that many of the so-called new features are ones that are in use in some (usually only one) system; in most cases they fulfill a perceived need. For MPI to be successful, it must not be incompatible with these needs. I realize that the other side of this is that, by specifying only a subset of desired functionality, there is the danger of mutually incompatible extensions. We need to weigh this against the possibility that MPI will become too complex to be accepted. Bill ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:23:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05096; Wed, 10 Mar 93 12:23:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24792; Wed, 10 Mar 93 12:22:05 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:22:04 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24784; Wed, 10 Mar 93 12:22:01 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26984; Wed, 10 Mar 93 17:21:17 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07764; Wed, 10 Mar 93 10:20:02 MST Date: Wed, 10 Mar 93 10:20:02 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101720.AA07764@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI Goals and Scope: More Details Please? Cc: lyndon@epcc.edinburgh.ac.uk Lyndon writes, > Don't you think it will be some time before the world is ready to even > think about standardisation of "parallel I/O". I can't imagine trying > to do such a thing at the moment. I agree. I'm not proposing we do this now. I'm proposing that we do it when it is as well understood as message passing is now. I think "eventual standardization of parallel I/O" should be a stated goal to give potential MPI users confidence that MPI won't be just another message-passing library. > [stuff deleted...] > > > I certainly won't use MPI unless I believe that this will happen with some > > version of MPI. > > I reckon you will, actually, because although it won't give you the > portability asked for above, it will be your best way of getting some > portability. I think Lyndon has a good point here. I will use MPI if: A) It gives me as much "portability" as existing solutions OR I believe that this will eventually happen. B) It gives me equal or better "performance". I will borrow/build/buy whatever additional stuff I need to make my programs more "portable", if necessary (required for my current project). I am hopeful that A will happen with some version of MPI. Vendor participation gives me some confidence that B will happen. > > John Flower has proposed one > > prioritization. Here's another (from my perspective as a user): > > I'm going to walk down this list and give marks out of ten, with ten as > highest priority and 0 as lowest priority, from my point of view, and > some justification, from my point of view. > > [list deleted...] Thanks Lyndon, this is exactly what I was looking for. At this point I feel it is extremely important that the MPIF come to some agreement about a list like this. It is a lot more important than my actually liking what's on the list! :) > [stuff deleted...] > > I am again ambivalent. I think it important to identify what items > should be suitable for future MPI, but I'm not at all confident that it > will be practical to commit ourselves to a schedule for trying to > standardise such items. It will be harder to agree on standards for the > items which are deferred. > > Best Wishes > Lyndon > I didn't mean to imply an absolute time-frame for the introduction of future MPI features. (It doesn't make sense for the more "researchy" stuff.) Prioritization is more important to me than a strict schedule. Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:32:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05213; Wed, 10 Mar 93 12:32:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25472; Wed, 10 Mar 93 12:30:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:30:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25460; Wed, 10 Mar 93 12:30:47 -0500 Date: Wed, 10 Mar 93 17:30:30 GMT Message-Id: <3289.9303101730@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Goals and Scope: More Details Please? To: hender@macaw.fsl.noaa.gov (Tom Henderson), mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Wed, 10 Mar 93 10:20:02 MST Reply-To: lyndon@epcc.ed.ac.uk Cc: lyndon@epcc.edinburgh.ac.uk > > I'm going to walk down this list and give marks out of ten, with ten as > > highest priority and 0 as lowest priority, from my point of view, and > > some justification, from my point of view. > > > > [list deleted...] > > Thanks Lyndon, this is exactly what I was looking for. Oh, good, glad I did that then! > At this point I feel > it is extremely important that the MPIF come to some agreement about a list > like this. AGREED!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > I didn't mean to imply an absolute time-frame for the introduction of future > MPI features. (It doesn't make sense for the more "researchy" stuff.) > Prioritization is more important to me than a strict schedule. Sorry, Tom, I misread your email. It's over and out for a week from me, colleagues, as I'm visiting my folks for the weekend and a couple of work meetings - surprise surprise, one of them to talk about MPI to a potential user, urgle urgle urgle! May your mail boxes be temporarily lighter ;-) Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:43:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05464; Wed, 10 Mar 93 12:43:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26174; Wed, 10 Mar 93 12:41:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:41:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26164; Wed, 10 Mar 93 12:41:55 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27092; Wed, 10 Mar 93 17:41:49 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07798; Wed, 10 Mar 93 10:40:35 MST Date: Wed, 10 Mar 93 10:40:35 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101740.AA07798@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI Goals and Scope: More Details Please? Cc: gropp@mcs.anl.gov Bill writes, > I'd like to clarify what we mean by > > > We feel that the MPI standard should provide enough information to > > specify an entire, source-code portable program. > > This is NOT intended to mean that MPI must provide a way to specify ANY (or > even most) source-code portable parallel program. Rather, given restrictions > strong enough to satisfy a 2/3rds majority ( :) ), a way to specify at least > one source-code portable program. In the spirit of extensibility, whatever > mechanism is proposed for this simple interface should not preclude non-MPI > solutions for more demanding situations. > Bill > Could you give an example of the one source-code portable program (or class of programs) you expect MPI to support in current and future versions? I think that might help me understand your statement a bit better. Also, what do you mean by "non-MPI solution" and "demanding situations"? Thanks, Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:19:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06396; Wed, 10 Mar 93 13:19:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28116; Wed, 10 Mar 93 13:18:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:18:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28057; Wed, 10 Mar 93 13:18:02 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:10 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA22798; Wed, 10 Mar 93 10:08:19 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20772; Wed, 10 Mar 93 10:08:13 PST Date: Wed, 10 Mar 93 10:08:13 PST From: d39135@sodium.pnl.gov Subject: implementation assumptions To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303101808.AA20772@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-comm@cs.utk.edu In a discussion we are having within the contexts subcommittee, Tony Skjellum commented that: > ... I want to make sure that we do > not provide illusory improvements to the standard, while letting other > parts of it exceed ours in the unscalability department, or let > implementations qualify as compliant, whose unscalability disables the > implied scalability of our syntax/semantics... I agree. This is why (I think) proposals should explicitly address performance. If a proposal assumes that the cost of a particular function is irrelevant, it should say that. If a proposal assumes that a particular function is going to be cheap, it should justify that assumption by naming a time/memory price and outlining an implementation that can achieve it. Sometimes the suggested implementation will reveal further assumptions that have wide implications. Ideally, each proposal will eventually evolve a set of bottom-line assumptions that are consistent and believable. These comments are particularly addressed to those of us beating on contexts and groups, where there seem to be lots of subtle interactions. But I am cross-posting them to the whole committee as food for thought... --Rik ("The Skeptic") Littlefield From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:22:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06459; Wed, 10 Mar 93 13:22:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28196; Wed, 10 Mar 93 13:20:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:20:18 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28188; Wed, 10 Mar 93 13:20:16 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27261; Wed, 10 Mar 93 18:20:08 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07839; Wed, 10 Mar 93 11:18:54 MST Date: Wed, 10 Mar 93 11:18:54 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101818.AA07839@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities Cc: lederman@b125.super.org Steve writes, > 1) How long will the MPI-1 process be allowed to take? Are we going > to stick to the 6 month deadline or try and extend it? I'd like to stick to 6 months for MPI-1. I'm willing to have less functionality in MPI-1 as long as there is a plan for further upgrade. > 2) What is going to be in MPI-1? Good question. :) > 3) Of the things we leave out of MPI-1 are they going to be proposed > for a later version of MPI? Should we make sure that MPI-1 is > compatible with the future goals so that major changes in the base > routines is not needed later? Yes and Yes. > 4) If we want to have MPI-#, what mechanism and time scale will be > used to get it done. Maybe next year we meet again for MPI-2? We should have some experience with MPI-1 by then. (...wait a minute, I only signed on for a six-month tour...) > I think it is past time to find the forest for the trees. Global > planning to focus our efforts now could significantly reduce our > discussions and efforts in the short term as deadlines approach. > Furthermore, with a global blueprint we might see a more even handed > approach to different aspects of the standard. These are just some > thoughts. I hope the discussion will continue. Me too. > Steve Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:22:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06475; Wed, 10 Mar 93 13:22:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28313; Wed, 10 Mar 93 13:21:38 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:21:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28305; Wed, 10 Mar 93 13:21:34 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27273; Wed, 10 Mar 93 18:21:30 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA10163; Wed, 10 Mar 93 11:24:00 MST Date: Wed, 10 Mar 93 11:24:00 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303101824.AA10163@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities Bill Gropp sez: > One approach that we could take is to describe a set of MPI that is extensible > to the more elaborate facilities but does not yet specify them. For example, > we could keep the goal of being "thread-safe" and resolve (at least in an > extensible way) the group/context/process-id issues, and then settle on, for > example, somethine like what Rusty and I have called level 3, plus some > collective communication routines. This would not have everything that I would > want or even consider important, but would be a usable set for casual use. > > This is a very different process than just defining a simple set of operations; > it MUST be possible to extend, in a consistent way, MPI to meet new needs. > I'd like to point out that many of the so-called new features are ones that > are in use in some (usually only one) system; in most cases they fulfill > a perceived need. For MPI to be successful, it must not be incompatible with > these needs. > > I realize that the other side of this is that, by specifying only a subset of > desired functionality, there is the danger of mutually incompatible extensions. > We need to weigh this against the possibility that MPI will become too complex > to be accepted. > > Bill I agree with Bill, I think whatever we produce for MPI-1 should be extensible. I think we should consider carefully our semantics and syntax to not prohibit extensibility. An approach that has been proposed by us (Tom Henderson and myself) as well as by other people is opaque messages. This allows an initial implementation to support a relatively simple to use subset while allowing vendor extensions and future upgrades to MPI-1. We could perhaps specify some core of MPI-1 (not a new idea) as well as provide semantic and syntatical guidance to vendors for their extensions (e.g. a clue about MPI-2 if you will). The opaque message object allows these extensions to be put in place without an explosion of syntax. Leslie Hart (hart@fsl.noaa.gov) P.S This may fit into Rusty & Bill's Level 1 or Level 2 proposal neatly. From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:26:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06583; Wed, 10 Mar 93 13:26:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28684; Wed, 10 Mar 93 13:25:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:25:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28669; Wed, 10 Mar 93 13:25:55 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA10195 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 12:25:52 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA22431; Wed, 10 Mar 93 12:25:51 CST Date: Wed, 10 Mar 93 12:25:51 CST Message-Id: <9303101825.AA22431@godzilla.mcs.anl.gov> To: hender@macaw.fsl.noaa.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Wed, 10 Mar 93 10:40:35 MST <9303101740.AA07798@macaw.fsl.noaa.gov> Subject: MPI Goals and Scope: More Details Please? X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:41:58 EST Date: Wed, 10 Mar 93 10:40:35 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Could you give an example of the one source-code portable program (or class of programs) you expect MPI to support in current and future versions? I think that might help me understand your statement a bit better. Basically, SPMD programs that have "simple" I/O requirements. For example, we could ask the following: IF fprintf (C) or write (Fortran) are supported, THEN each use writes to the selected file, with NO guarentees on the ordering of output. Something similiar for fscanf/read . Thus, we can now write the following two programs (using the MPI_main startup approach): MPI_main() { printf( "Hello world\n" ); } and MPI_main() { int i; for (i=0; i To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities > From: Tony Skjellum > To: lederman@b125.super.org, gropp@mcs.anl.gov > Subject: Re: MPI directions & priorities > Cc: mpi-comm@cs.utk.edu > > We must agree to reconvene about extensions in future. I also agree > that it would be preferable to specify a whole portable program, but I > think the I/O subject is hard to address right now... > > - Tony I also understood that the process would be ongoing and hope that it will continue to track existing practice. I think some parts of the I/O subject are hard to address right now, but there is some existing common practice. Parasoft, Intel, nCUBE and TMC all have some sort of parallel I/O specification. There are ways to force the effect of a printf or of an fseek/fread/fwrite to be deterministic. I believe that the common practice could be standardized, maybe not in the time frame for MPI-1 but probably for MPI-2 (which is where it appeared in Tom's posting). Regards, Leslie From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 14:03:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07595; Wed, 10 Mar 93 14:03:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01003; Wed, 10 Mar 93 14:02:53 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 14:02:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00994; Wed, 10 Mar 93 14:02:47 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA08764 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Wed, 10 Mar 1993 11:02:44 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA26008; Wed, 10 Mar 93 10:55:24 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA26815; Wed, 10 Mar 93 10:55:34 PST Date: Wed, 10 Mar 93 10:55:34 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303101855.AA26815@lion.parasoft> To: mpi-comm@cs.utk.edu To: mpi-comm@cs.utk.edu Re: A way of prioritizing? Something Bill said seemed to strike a chord in my mind: > This is a very different process than just defining a simple > set of operations; it MUST be possible to extend, in a > consistent way, MPI to meet new needs. I'd like to point > out that many of the so-called new features are ones that > are in use in some (usually only one) system; in most cases > they fulfill a perceived need. For MPI to be successful, > it must not be incompatible with these needs. I also have the feeling that a lot of the things that we're trying to put in MPI are really only based on experience (or even need) on a single machine. Sometimes that machine may even be "pre-history". My question is: Is incorporating into MPI a feature that is only based on a single machine any better than allowing a vendor-specific extension on that machine? Take "ready-receive", for example. Adding this to MPI means that users can justifiably put it in their portable codes. If they do, however, what should they assume about it? That it's faster or slower than the non-ready receive? I assume that there will be systems in which it's slower than the non-ready recv. because of the extra protocol that the application might need to ensure that the ready-recv.'s don't crash. In this case, are we expecting users to put code in their applications that can switch between ready-recv, and non-ready-recv semantics based on the hardware they're running on? Although I think there are some users who will do this (and I would encourage library developers to do so) I think this is far beyond the lengths that most users will go to. So what am I suggesting? --------------------------------------------------------- My suggestion (which I expect to be roundly rejected) is that the MPI document ONLY contain the stuff that is current practice. This includes EVERYTHING required to run at least a common class of programs - i.e., MPI_INIT or some such. These things should be designed to maximize ease of use and learning and encourage portability. By this I am including performance level portability. I would like to not have things in the spec. that run well on some machines and poorly on others. I would include a requirement that a conforming implementation include either documents or (preferably) programs from which a user could assess the performance of the various routines. All the other things that are off-shoots of requirements for odd-machines or otherwise more advanced should go in Appendices or "Annexes" (Ada notation(!)) ***AND WOULD NOT BE REQUIRED BY A CONFORMING IMPLEMENTATION OF MPI ***. --------------------------------------------------------- The effect I am after is that the core MPI standard is simple, effective and encourages users to write code. The annexes essentially say to vendors; "If you want to support an advanced feature that will really help on your machine, please do it this way so that my programs that use it will be portable." This is similar to Bill & Rusty's multi-level scenario except that I wouldn't make the lower levels compulsory. My hope is that two things will happen if we adopt this strategy: a) Details of things that go in Annexes can be postponed or left vague, or even left to specially interested groups to define, but we will definitely have something that could be used by an application developer at the end of the six months. b) The MPI core would be smaller which will encourage users/vendors to make implementations, and to optimize the #$%$% out of them. If this concept is acceptable I would suggest that someone prepare a list of topics, in quite great detail, and we all give them grades out of ten (via e-mail), just as Lyndon did. Someone with experience with statistics could analyze the results and give us weighted rankings at the next meeting. We could then prioritize according to this list and also make guided decisions as to what should go in the Annexes. I would then suggest that we go back to the small sub-committee approach to putting the Annexes together rather than our current "committee-of-the-whole" style which seems to encourage hour-long "arguments" followed by 35-0 votes! Jon Flower p.s. Having suggested a pile of work I should say who's going to do it. In the absence of any volunteers I might be persuaded to come up with a list of issues on which we should score. I'm not sure my statistical skills amount to anything much more that simple averages, so someone else could certainly do better than I. I would be very interested in working with a group/sub-committee trying to get up a set of MPI-compliant performance evaluation standards. From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 15:32:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09642; Wed, 10 Mar 93 15:32:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07600; Wed, 10 Mar 93 15:30:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 15:30:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07590; Wed, 10 Mar 93 15:30:35 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 12:29 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24307; Wed, 10 Mar 93 12:27:24 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21996; Wed, 10 Mar 93 12:27:21 PST Date: Wed, 10 Mar 93 12:27:21 PST From: d39135@sodium.pnl.gov Subject: RE: MPI directions & priorities To: gropp@mcs.anl.gov, lederman@b125.super.org, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov, mpi-comm@cs.utk.edu Message-Id: <9303102027.AA21996@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu > We must agree to reconvene about extensions in future. I also agree > that it would be preferable to specify a whole portable program, but I > think the I/O subject is hard to address right now... > > - Tony I presume that Tony speaks of parallel I/O. I agree that parallel I/O is too hard for now. But I think that we have to specify at least some minimum serial I/O capability. I would buy off on something as simple as . there exists one process that can do serial I/O in the standard fashion for the language (e.g., printf() or WRITE(*,*) ) . the tid of that process can be obtained from MPI (in any process) Example: if (mytid == MPI_IOROOT()) { printf ("I'm the designated serial I/O process\n"); } --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 15:37:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09805; Wed, 10 Mar 93 15:37:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08036; Wed, 10 Mar 93 15:36:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 15:36:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08022; Wed, 10 Mar 93 15:36:25 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27653; Wed, 10 Mar 93 20:36:21 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA08016; Wed, 10 Mar 93 13:35:06 MST Date: Wed, 10 Mar 93 13:35:06 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303102035.AA08016@macaw.fsl.noaa.gov> To: jwf@lion.parasoft.com Subject: Annexes, etc. Cc: mpi-comm@cs.utk.edu Jon, > My suggestion (which I expect to be roundly rejected) is > that the MPI document ONLY contain the stuff that is current > practice. This includes EVERYTHING required to run at least > a common class of programs - i.e., MPI_INIT or some such. > > These things should be designed to maximize ease of use and > learning and encourage portability. By this I am including > performance level portability. I would like to not have > things in the spec. that run well on some machines and > poorly on others. I would include a requirement that a > conforming implementation include either documents or > (preferably) programs from which a user could assess > the performance of the various routines. > > All the other things that are off-shoots of requirements > for odd-machines or otherwise more advanced should go in > Appendices or "Annexes" (Ada notation(!)) ***AND WOULD NOT > BE REQUIRED BY A CONFORMING IMPLEMENTATION OF MPI ***. Would the MPIF make any commitment to including "Annex" features in future MPI versions? Would there be any future MPI versions? Would you mind providing a bit more detail on what you feel are a "common class of programs"? (Gee, I don't ask for much... :-) > If this concept is acceptable I would suggest that someone > prepare a list of topics, in quite great detail, and we > all give them grades out of ten (via e-mail), just as > Lyndon did. Someone with experience with statistics > could analyze the results and give us weighted rankings > at the next meeting. What level of detail do you have in mind? Could you give an example? I like this idea. Hashing this out over e-mail could save us a lot of pointless wrangling at the next meeting. Tom P.S. Jon, my apologies for mis-spellings in previous messages! :) From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 05:10:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25347; Thu, 11 Mar 93 05:10:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11372; Thu, 11 Mar 93 05:08:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 05:08:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11362; Thu, 11 Mar 93 05:08:16 -0500 Date: Thu, 11 Mar 93 10:07:58 GMT Message-Id: <3792.9303111007@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: minutes? To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk I never saw any minutes for the February meeting yet, perhaps I've been expecting them too early. Please, can someone let me know when they are expected to be circulated. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 06:02:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07862; Thu, 11 Mar 93 06:02:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18258; Thu, 11 Mar 93 06:01:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 06:01:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18250; Thu, 11 Mar 93 06:01:20 -0500 Date: Thu, 11 Mar 93 11:01:15 GMT Message-Id: <3852.9303111101@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: regarding discussion of directions and priorities To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I am shortly to give a presentation regarding MPI to a potential end user of some imprtance within the UK. In preparation for this, I have spent the last hour of this morning looking back over the discussions on mpi-comm and various subcommittees. The single thing which most struck me was that since March 9 the content on mpi-comm has been dominated by discussion of directions and priorities for MPI. I recorded 20 mail messages in total from 8 individuals. There were preceeding discussions of this nature on mpi-pt2pt dating from March 4. The discussions indicate to me that there are some rather concerned committee members. It would appear, to my mind at least, that different committee members have different views of the extant directions and priorities of MPI, and of desired directions and priorities for MPI. I am sure that I have a different view from some other members of MPI. In order to make satisfactory progress in the process of interface definition, which is apparently not occuring at the moment, I believe that there would indeed be advantage in describing and formally agreeing the directions of MPI and a set of priorities within those directions. I applaude efforts to create consensus regarding directions and priorities. I also believe that, from the point of view of the perception of MPI of individuals and organisations not directly involved in the committee, there is advantage in publicising the agreed directions of MPI. In order to make satisfactory progress in the process of determining directions and priorities, I believe that there would be advantage in describing and formally agreeing a framework within which we can agree on such directions and priorities. Sadly, I do not observe such a framework in operation, and I am skeptical whether MPI can succeed without such a framework. I would therefore suggest that the immediate priorities for MPI as as follows: 1. Discuss and establish framework within which directions for MPI are assessed. 2. Discuss and establish directions for MPI within agreed framework. 3. Discuss and establish priorities for MPI within agreed directions. 4. Review proposals in all subcommittees within agreed priorities. 5. Extend and complete proposals as necessary within terms of 3 and 4. As a suggestion, can we all describe what we mean by "common practice", a starting point for framework discussion? Some of the 8 mentioned above have already done this, to some extent, which I also applaud. It seems to me that time demands 1,2 and 3 must occur primarily by email. I would suggest that these should be formally accepted at the beginning of the next meeting. From a practical point of view, I further suggest that the next meeting should call a meeting of the whole committee for this purpose, and we should be prepared to hold said meeting on the Wednesday morning, on order to allow the subcommittees to make progress with 4 and 5. Now my taxi is waiting, and I must go off-line for a week. I apologise to you all that I will not be able to follow up this somewhat despondent message for 6 days. Good luck! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 11:20:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29874; Thu, 11 Mar 93 11:20:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01139; Thu, 11 Mar 93 11:17:49 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 11:17:48 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01131; Thu, 11 Mar 93 11:17:47 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13509; Thu, 11 Mar 93 10:13:56 CST Date: Thu, 11 Mar 93 10:13:56 CST From: Tony Skjellum Message-Id: <9303111613.AA13509@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: minutes? Yes, having the minutes is becoming essential. - Tony ----- Begin Included Message ----- I never saw any minutes for the February meeting yet, perhaps I've been expecting them too early. Please, can someone let me know when they are expected to be circulated. Best Wishes Lyndon ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 11:23:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00145; Thu, 11 Mar 93 11:23:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01349; Thu, 11 Mar 93 11:21:30 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 11:21:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01340; Thu, 11 Mar 93 11:21:27 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA00895; Thu, 11 Mar 93 11:21:24 -0500 Date: Thu, 11 Mar 93 11:21:24 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303111621.AA00895@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Jon's suggestion Jon writes: > My suggestion (which I expect to be roundly rejected) is > that the MPI document ONLY contain the stuff that is current > practice. This includes EVERYTHING required to run at least > a common class of programs - i.e., MPI_INIT or some such. > > These things should be designed to maximize ease of use and > learning and encourage portability. By this I am including > performance level portability. I would like to not have > things in the spec. that run well on some machines and > poorly on others. I would include a requirement that a > conforming implementation include either documents or > (preferably) programs from which a user could assess > the performance of the various routines. > > All the other things that are off-shoots of requirements > for odd-machines or otherwise more advanced should go in I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. Al Geist From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 12:23:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01995; Thu, 11 Mar 93 12:23:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04568; Thu, 11 Mar 93 12:19:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:19:57 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04559; Thu, 11 Mar 93 12:19:56 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Thu, 11 Mar 1993 12:19:53 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA21933; Thu, 11 Mar 1993 12:19:51 -0500 Date: Thu, 11 Mar 1993 12:19:51 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199303111719.AA21933@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Message Passing not for the masses Al say's I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. \begin{flame} I generally agree with Al, but not this time. While I tend to think that there should be some way to figure out how to use MPI in one sitting, I certainly do not think that we should be "maximizing ease of use" as the principle design goal. If I wanted things to be simple and slow, I'd be a Linda user. Message passing is nasty. Message passing sucks. Message passing is the way to get machines to go fast, at least for the time being. \end{flame} That being said, even if MPI turns into a huge, ugly mass of routines, it should still be possible to isolate a few of these routines and present them as the "simple subset" for people who don't need to drive their communication costs down. I don't see "limiting the size of" being the same as "making easy to use". -scott berryman From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 12:32:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02217; Thu, 11 Mar 93 12:32:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05147; Thu, 11 Mar 93 12:30:50 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:30:48 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05132; Thu, 11 Mar 93 12:30:46 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA29711; Thu, 11 Mar 93 17:30:24 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA05324; Thu, 11 Mar 93 10:32:53 MST Date: Thu, 11 Mar 93 10:32:53 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303111732.AA05324@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, geist@msr.epm.ornl.gov Subject: Re: Jon's suggestion > Jon writes: > > My suggestion (which I expect to be roundly rejected) is > > that the MPI document ONLY contain the stuff that is current > > practice. This includes EVERYTHING required to run at least > > a common class of programs - i.e., MPI_INIT or some such. > > > > These things should be designed to maximize ease of use and > > learning and encourage portability. By this I am including > > performance level portability. I would like to not have > > things in the spec. that run well on some machines and > > poorly on others. I would include a requirement that a > > conforming implementation include either documents or > > (preferably) programs from which a user could assess > > the performance of the various routines. > > > > All the other things that are off-shoots of requirements > > for odd-machines or otherwise more advanced should go in > > I do not reject this in fact I applaud your suggestion. > This may be the only way we can get MPI-1 off the ground. > I am a big advocate of maximizing ease of use > this is only way to get the large user base necessary > for MPI to become a standard. > > Al Geist I also agree. I think that there is significant concern regarding the near term and long term success of MPI. I think some of the concern can be aleviated if we settle on a goal that can be reached by this summer. It may be wise to set up some sort of electronic straw poll procedure to help get us past some of the issues regarding what should be in the first MPI, what are our goals for MPI-1, is there going to be MPI-N (for N>1), what features are most important for MPI-1, etc. Perhaps we can use similar rules to those agreed upon in the first meeting. You must have been in Dallas for 2 of the last three meetings (1 of the first 2, I guess). Calls for vote should be made through the committee chair (or someone appointed by the committee chair). The votes will be counted over a three day period by the comittee chair (or the designee) and reported. (Only your "last" vote counts :-). These will, of course, be straw polls and will be reported again at the beginning of the next Dallas meeting. Regards, Leslie Hart (hart@fsl.noaa.gov) P.S. We may wish to use a separate mail list for voting (mpi-vote??) which includes those eligible to vote via the voter elibibilty rules. Of course, votes for a particular topic should be clearly identified and the sense of the vote should be clear (i.e yes or no). The suggestion of electronic votes is important to me, but the mechanics is not that important. > From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 13:04:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03015; Thu, 11 Mar 93 13:04:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08235; Thu, 11 Mar 93 13:00:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 13:00:08 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08214; Thu, 11 Mar 93 13:00:05 -0500 Received: from jadoube.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA19857 (5.65c/IDA-1.4.4 for ); Thu, 11 Mar 1993 12:00:02 -0600 From: David Levine Received: by jadoube.mcs.anl.gov (4.1/GeneV4) id AA15971; Thu, 11 Mar 93 12:00:00 CST Date: Thu, 11 Mar 93 12:00:00 CST Message-Id: <9303111800.AA15971@jadoube.mcs.anl.gov> To: berryman-harry@cs.yale.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Harry Berryman's message of Thu, 11 Mar 1993 12:19:51 -0500 <199303111719.AA21933@YOGI.NA.CS.YALE.EDU> Subject: Message Passing not for the masses X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:19:57 EST Date: Thu, 11 Mar 1993 12:19:51 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Al say's I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. \begin{flame} : Message passing is nasty. Message passing sucks. Message passing is the way to get machines to go fast, at least for the time being. \end{flame} I'd like to add my two cents in agreement with this comment. Message passing is NOT for the masses. Its just too hard and grungy. My opinion is that message passing is the assembly language of parallel programming and it will evolve that way. I think the "large user base necessary for MPI to become a standard." will be *expert* users, vendors, and HPF compiler writers. If MPI doesn`t support what they need it will not be useful (it may still, however, become yet-another-standard.) That said, I am in agreement that there should be an easy-to-use MPI layer and a nice tutorial document for novice users, but don't handicap MPI to accomodate novice users who will migrate away from message-passing as higher-level software tools (HPF, MIMDizer, BlockComm, ....) become available. --dave levine From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 13:27:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04067; Thu, 11 Mar 93 13:27:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09727; Thu, 11 Mar 93 13:26:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 13:26:19 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09719; Thu, 11 Mar 93 13:26:17 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA01747; Thu, 11 Mar 93 13:26:15 -0500 Date: Thu, 11 Mar 93 13:26:15 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303111826.AA01747@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Message Passing not for the masses scott berryman flames... >even if MPI turns into a huge, ugly mass of routines, There are two problems with this argument. a. at the rate we are going we will never be able to agree on (or document) MPI so there wont be a subset. b. existing packages like NX were not designed to spare performance yet it doesn't have 4 layers and 34 different send variants. Jon and my comment reflect the concern that we will not be able to define MPI-1 in any reasonable time frame if it is a huge, ugly mass of routines, Al From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 14:23:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05158; Thu, 11 Mar 93 14:23:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12286; Thu, 11 Mar 93 14:21:53 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 14:21:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12277; Thu, 11 Mar 93 14:21:50 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Thu, 11 Mar 1993 14:21:48 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA22157; Thu, 11 Mar 1993 14:21:47 -0500 Date: Thu, 11 Mar 1993 14:21:47 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199303111921.AA22157@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Huge ugly mass of routines Al Giest simmers... Jon and my comment reflect the concern that we will not be able to define MPI-1 in any reasonable time frame if it is a huge, ugly mass of routines, At the risk of agreeing with you, there is a real danger at this point of the huge ugly mass of routines preventing us from do the job in the time alloted. This is a different concern than ease of use. What would you suggest eliminating to simplify the standard? -scott From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 14:37:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05455; Thu, 11 Mar 93 14:37:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12897; Thu, 11 Mar 93 14:35:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 14:35:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12882; Thu, 11 Mar 93 14:35:29 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA16681; Thu, 11 Mar 93 11:35:26 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA06719; Thu, 11 Mar 93 11:35:25 PST Date: Thu, 11 Mar 93 11:35:25 PST Message-Id: <9303111935.AA06719@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Response to Berryman's comments Scott says (and I love his "\begin{flame}") -- While I tend to think that there should be some way to figure out how to use MPI in one sitting, I certainly do not think that we should be "maximizing ease of use" as the principle design goal. and then even if MPI turns into a huge, ugly mass of routines, it should still be possible to isolate a few of these routines and present them as the "simple subset" for people who don't need to drive their communication costs down. I don't see "limiting the size of" being the same as "making easy to use". It's certainly try that ease-of-use does not necessarily mean a small number of routines. Ease-of-use is more dependent on: a) If there are subsets, there must be subsets that can function as a reasonal mini-language, in the sense that applications can be developed using just those routines - and getting reasonable performance. b) There must be syntactic/semantic consistency both among routines in a subset and across subset boundaries. Otherwise, users will not be able to learn subsets incrementally with any ease. c) Perhaps most importantly, the names of routines, order and type of arguments, and user-level semantics (by which I mean what can go wrong with respect to the user program) must be consistent and as simple as possible. Cherri From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 15:42:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07612; Thu, 11 Mar 93 15:42:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15959; Thu, 11 Mar 93 15:40:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 15:40:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15920; Thu, 11 Mar 93 15:40:35 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA16701 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Thu, 11 Mar 1993 12:40:32 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA19196; Thu, 11 Mar 93 12:33:06 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA00810; Thu, 11 Mar 93 12:33:21 PST Date: Thu, 11 Mar 93 12:33:21 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303112033.AA00810@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: To fan the flames.... To add another layer of hypothetical justification to my argument .......... I think that looking at what we can achieve in the next few meetings is actually only a part of the issue. Probably a larger part of my concern is how long after we produce a draft some reasonably large percentage of the vendors will actually be supporting whatever it is we come up with. Having commitments from vendors to support MPI1 is a long way from actually having an implementation that "kicks #$%$%^$%" in our hot little hands. (Cf. the vendor "acceptance" of OSF for a good example.) My biggest fear is that if MPI-1 is a massive system then the first implementations might be a year after we finish and the first "good" ones (i.e., that actually use some of the really advanced and "neat" stuff that we put into MPI-1 just for that purposes) may take two or more years!! Is that worth it? I don't think so. Of course, I'm probably slandering the vendors and if so I apologise. Perhaps they are better off than I think they are. My impression, however, is that they are struggling to get other stuff solid which they probably would be totally justified in thinking is more important than implementing MPI. Of course this argument doesn't include third parties, either commercial or not, from implementing MPI on top of other systems and giving them to the world. This could clearly happen on a shorter time-scale. However, I expect MPI to fail if this is the best we can offer because people will still find it "convenient" to use vendor specific alternatives that have higher performance. Jon p.s., Just for the sake of it someone asked for a definition of "common practice" - a term that I've bandied about without justification. Without too much thought I think a viable definition might be to take the intersection of all the systems that are currently in use to develop application codes. This is a conceptual intersection, of course, not a strict one since all systems differ in details. This differs in my mind from what we have in the current MPI spec. which might be termed the transitive closure of all the ideas in current use!!! From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 18:15:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10139; Thu, 11 Mar 93 18:15:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23334; Thu, 11 Mar 93 18:13:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 18:13:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23325; Thu, 11 Mar 93 18:13:18 -0500 Via: uk.ac.southampton.ecs; Thu, 11 Mar 1993 19:07:28 +0000 Via: brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:08:41 GMT From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:16:56 GMT Date: Thu, 11 Mar 93 18:16:58 GMT Message-Id: <19579.9303111816@holt.ecs.soton.ac.uk> To: mpi-comm@cs.utk.edu Subject: Re: Jon's suggestion > From: hart@gov.noaa.fsl.nipmuc (Leslie Hart) > I also agree. I think that there is significant concern regarding > the near term and long term success of MPI. I think some of the > concern can be aleviated if we settle on a goal that can be reached > by this summer. It may be wise to set up some sort of electronic > straw poll procedure to help get us past some of the issues regarding > what should be in the first MPI, what are our goals for MPI-1, is there > going to be MPI-N (for N>1), what features are most important for MPI-1, etc. I agree. Just when I thought things were converging there's been a flurry of alternative proposals. For example, can we please agree *not* to address concurrent I/O in MPI1, else we'll never get finished within the allotted timescale. On the other hand, I do think that we should decide to have an MPI2, and that that would be the proper way to deal with such issues. I don't think concurrent I/O is an intractible problem, its just that to do a reasonable job on will require more time than we've got right now. Another case in point is dynamic vs static process creation. My vote is for static now, dynamic later. If we can agree that there will definitely be an MPI2, then maybe we can keep everyone happy in the sense that their favourite features *will* all be included, but in a staged fashion. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-comm@CS.UTK.EDU Fri Mar 12 15:14:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29560; Fri, 12 Mar 93 15:14:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20854; Fri, 12 Mar 93 15:12:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Mar 1993 15:12:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20838; Fri, 12 Mar 93 15:12:23 -0500 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA00567; Fri, 12 Mar 93 12:12:15 PST Message-Id: <9303122012.AA00567@ocfmail.ocf.llnl.gov> Date: Fri, 12 Mar 1993 12:20:03 -0800 To: mpi-comm@cs.utk.edu From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Buffer descriptors and the simple program Cc: rathkopf@llnl.gov, mirin@llnl.gov, jlowens@llnl.gov, geltgroth@llnl.gov, tsw@llnl.gov, seager@llnl.gov, trj@llnl.gov, lstanberry@llnl.gov, bolstad@llnl.gov, zosel@llnl.gov, jbrown@nersc.gov, frobose@llnl.gov, hendrickson1@llnl.gov, batkinson@llnl.gov I met with a number of application and system programmers here at the Lab to get some feedback on the decisions made at the last MPI meeting. One aspect that they felt uncomfortable with was being forced to use the buffer descriptor routines (i.e., bd_create, bd_contiguous, bd_stride and bd_free) in order to just send a simple message. The application programmers want both simple send and receive primitives as well as more general primitives based on buffer descriptors. For example, the Parallel Monte Carlo and Atmospheric Chemistry codes they are working on would benefit from the more general buffer descriptor based primitives, while the Atmospheric Dynamics code uses only simple send and receive. The programmers that use only a single contiguous buffer for their codes would be upset if MPI forced them to make two bd calls to set up the descriptor and then another bd call to free it. (BTW the last call could be eliminated if there were some way to "clear" a buffer descriptor so it could be reused). Dan Nessett From owner-mpi-comm@CS.UTK.EDU Fri Mar 12 15:52:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00970; Fri, 12 Mar 93 15:52:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22518; Fri, 12 Mar 93 15:51:15 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Mar 1993 15:51:14 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22510; Fri, 12 Mar 93 15:51:09 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA07519 (5.65c/IDA-1.4.4 for ); Fri, 12 Mar 1993 14:51:06 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA21843; Fri, 12 Mar 93 14:51:04 CST Date: Fri, 12 Mar 93 14:51:04 CST Message-Id: <9303122051.AA21843@godzilla.mcs.anl.gov> To: nessett@ocfmail.ocf.llnl.gov Cc: mpi-comm@cs.utk.edu, rathkopf@llnl.gov, mirin@llnl.gov, jlowens@llnl.gov, geltgroth@llnl.gov, tsw@llnl.gov, seager@llnl.gov, trj@llnl.gov, lstanberry@llnl.gov, bolstad@llnl.gov, zosel@llnl.gov, jbrown@nersc.gov, frobose@llnl.gov, hendrickson1@llnl.gov, batkinson@llnl.gov In-Reply-To: Dan Nessett's message of Fri, 12 Mar 1993 12:20:03 -0800 <9303122012.AA00567@ocfmail.ocf.llnl.gov> Subject: Buffer descriptors and the simple program [ message about having to use the "bd_xxx" routines ] This is what the level 2 (or 3, depending on where you draw the line) routines are for. Level 3 contains the "simple snd and receive" with contiguous buffers, while level 1 has everything at the cost of greater complexity. Bill From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 15:54:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29714; Tue, 16 Mar 93 15:54:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19013; Tue, 16 Mar 93 15:51:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 15:51:36 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19005; Tue, 16 Mar 93 15:51:34 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19519; Tue, 16 Mar 1993 15:51:33 -0500 Date: Tue, 16 Mar 1993 15:51:33 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303162051.AA19519@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI test implementation An Argonne report by William Gropp and Ewing Lusk dated December 1992 ("A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47) has recently been distributed. Please note that the MPI1 routines referred to in this document are those in a draft paper that was circulated as a discussion document. Many of these routines differ from those in the final version of MPI1, or are omitted. The final version of the MPI1 draft standard may be obtained by sending the email message "send mpi1.ps from mpi" to netlib@ornl.gov. For those of you interested in correctly referencing this the bibtex item is as follows: @techreport{MPI1, author = "J. J. Dongarra and R. Hempel and A. J. G. Hey and D. W. Walker", title = "A Proposal for a User-Level, Message Passing Interface in a Distributed Memory Environment", institution = "Oak Ridge National Laboratory", number = "{TM}-12231", month = "March", year = 1993} Regards, David Walker. From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 16:43:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01882; Tue, 16 Mar 93 16:43:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22479; Tue, 16 Mar 93 16:41:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 16:41:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22461; Tue, 16 Mar 93 16:41:41 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA13171 (5.65c/IDA-1.4.4 for ); Tue, 16 Mar 1993 15:41:34 -0600 Message-Id: <199303162141.AA13171@antares.mcs.anl.gov> To: walker@rios2.epm.ornl.gov (David Walker) Cc: mpi-comm@cs.utk.edu Subject: Re: MPI test implementation In-Reply-To: Your message of "Tue, 16 Mar 1993 15:51:33 EST." <9303162051.AA19519@rios2.epm.ornl.gov> Date: Tue, 16 Mar 1993 15:41:32 -0600 From: Rusty Lusk Thanks for setting the record straight. That Tech. Report went into the Argonne mill in early December, and we had no idea that it would take it this long to be distributed and therefore how out-of-date it would be when it finally did get mailed. - Rusty From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 20:54:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06431; Tue, 16 Mar 93 20:54:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09400; Tue, 16 Mar 93 20:53:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 20:53:01 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09390; Tue, 16 Mar 93 20:53:00 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Tue, 16 Mar 93 17:52 PST Message-Id: Date: Tue, 16 Mar 93 17:52 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: PostScript version of the current MPI Draft Document A (compressed) PostScript version of the current MPI Draft Document is available via anonyomous ftp at cse.ogi.edu: ftp cse.ogi.edu cd pub/otto binary get draft.ps.Z quit This should get you a copy. You then need to uncompress it. All the source is there also if you are interested. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 08:39:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10600; Thu, 18 Mar 93 08:39:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04620; Thu, 18 Mar 93 08:36:29 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 08:36:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04612; Thu, 18 Mar 93 08:36:21 -0500 Date: Thu, 18 Mar 93 13:36:16 GMT Message-Id: <9166.9303181336@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MINUTES! To: mpi-comm@cs.utk.edu In-Reply-To: David Walker's message of Wed, 17 Mar 1993 09:57:50 -0500 Reply-To: lyndon@epcc.ed.ac.uk Hi all On Thursday of last week I enquired about the minutes of the last meeting. Dave Walker writes: > The minutes of the last meeting in February are not yet available > electronically, but should be soon. I observe that subcommittee chairs have already circulated revised "working documents" for subcommittees. The minutes of previous meeting are the primary reference for results of subcommittee straw polls which must surely be the primary determinants of working document revisions. It is now past time when the minutes were essential. Please, can these be distributed. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 11:18:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14049; Thu, 18 Mar 93 11:18:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11559; Thu, 18 Mar 93 11:17:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 11:16:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11550; Thu, 18 Mar 93 11:16:51 -0500 Date: Thu, 18 Mar 93 16:16:47 GMT Message-Id: <9298.9303181616@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: AGENDA! To: walker@rios2.epm.ornl.gov (David Walker) In-Reply-To: David Walker's message of Thu, 18 Mar 1993 10:48:16 -0500 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu David writes: > > I don't think Lyndon's suggestion to make the timetable session the first of > the meeting is a good idea. How can a timetable be set if we don't know > what's to be included, > and if we don't know how much of the current proposals > are acceptable. > I see your point on the matter of establishing a firm timetable. It seems to me, and there has been a fair bit of traffic on it, that we need to establish boundaries for what will be in MPI. It seems to me that this needed to be the first thing to thrash out at the next meeting. I agree with you that timetables should be sorted out later. > We could move the session on contexts, groups etc. up front I guess. This seems to mean that the context subcommittee has to come to the "report and proposal to whole committee" stage without the opportunity to meet. My understanding was that we had intended to reach that stage at the next meeting, after a subcommittee meeting. Personally I can handle doing this by email over the next 10 days, but I'm not sure that the whole of the subcommittee can. Thanks for the comments, and pointing out my error in supposing that the timetable could be determined at the beginning of the meeting. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 12:54:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15727; Thu, 18 Mar 93 12:54:40 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16896; Thu, 18 Mar 93 12:50:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 12:50:43 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16880; Thu, 18 Mar 93 12:50:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA14802; Thu, 18 Mar 93 17:44:46 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA20644; Thu, 18 Mar 93 10:43:30 MST Date: Thu, 18 Mar 93 10:43:30 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303181743.AA20644@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: MPI Priorities and Scope: Opinion Survey Cc: jwf@lion.parasoft.com, lederman@b125.super.org, igl@ecs.soton.ac.uk, lyndon@epcc.ed.ac.uk Hi all, Steve Huss-Lederman writes: > I would like to propose that we have a meeting of all participants at > the start of the next meeting to settle the issue of what MPI-1 will > cover. Several mail messages have expressed concern that we cannot > conclude by June if we try to incorporate everything that everyone > wants. Since the scope and time frame will effect most of the other > discussions I think it would be best to begin with this. My only > concern is that it will take too long to decide the global picture. > I would hope it could be done in 1-2 hours and help to refocus the > group. I fully support this idea. Here's a proposal for an opinion survey that could shorten and focus the discussion. This proposal originated with Jon Flower and has had input from Ian Glendinning, Steve Huss-Lederman, Leslie Hart, myself, and others. If you think this is a good idea, please complete the survey and return to me at: hender@fsl.noaa.gov If you think this is a bad idea, please say so! We will collect responses and distribute a summary via email before the next meeting if there is sufficient interest. Tom Henderson ----------------------------------------------------------------------------- MPI PRIORITIES OPINION SURVEY Several MPI committee members have expressed concern about the current and future directions of MPI. The MPI committee began discussion using an existing document (Dongarra et al) as a starting point. This document contains background rationale and a statement of direction for the MPI standardization effort. Since these issues were consistent with the original MPI proposal, they have not recently been discussed in detail by the MPIF. The current MPI proposals are very different from the original MPI document. As a result, we feel that the original rationale and direction need to be re-examined. In particular we need to prioritize and limit the issues that MPI will try to resolve so that o the initial MPI standard document ("MPI-1") is delivered in a timely fashion and o the functions that it defines can (and will) be implemented on major platforms within a reasonable time frame. To get discussion of these issues started, we propose an informal, email "opinion survey". The objective of the survey is to find out which MPI features are most important. High-priority features would be included in MPI-1. Lower-priority features might be deferred to a later version of MPI. Some features might never be included. The compiled results of this survey could be used as a basis for discussion at the next MPI meeting. *** This survey is in no way binding and is a completely unofficial "straw poll" arranged by its authors for their own personal information. It is not intended to distract attention from the current proposals. Any data collected will be made available to the group at large if requested. *** Following is a ballot containing possible MPI features and goals. Please rate each item (or sub-item) using the following rating scheme: 0 This feature should not appear in any version of MPI ever. 1-2 MPIF should consider including this feature in some future version of MPI (after MPI-1). 3-4 This feature should be included as an Annex to MPI-1 for possible inclusion in a future version of MPI. 5-6 This feature should be included as an Annex to MPI-1 for definite inclusion in a future version of MPI. 7-10 This feature should be included in MPI-1. In this context an "Annex" is a part of the standard document that will not be required by a conforming implementation of MPI. Rather it should be taken as a recommendation by the MPIF to vendors for the way in which particular features should be implemented, if they are implemented at all. This way features that are not usefully implementable on particular hardware can be eliminated while machines that do support these features do so in a portable manner. When replying to the survey, please use the response form attached to the end of this message. (This will make compilation of the results a lot less painful!) :-) +-------------------------------------------------------+ | POSSIBLE FEATURES FOR INITIAL AND FUTURE MPI VERSIONS | +-------------------------------------------------------+ Ordering is for convenience only. This list is not meant to suggest any priority. GENERAL FEATURES 1) Routines for static process creation: A) Same program in every node. B) Host-node programming model in which there is one "special node". C) Static model but with potentially different programs in different groups of nodes. D) Static model with a way to create more than one process per processor. (Heavy duty processes, not threads). 2) Routines for identification of statically created processes. 3) Associated environmental inquiry routines. 4) Some way of avoiding message tag conflicts between independently developed libraries and user code (contexts). 5) Heterogeneous communication. A) All communication routines should support heterogeneous communication. B) A special set of communication routines should support heterogeneous communication. 6) Creation/destruction of "static" groups (processes cannot join/leave an existing group). Groups are used to limit the scope of collective operations. A) No logical process topology is associated with a group. B) Grid/toroidal process topology is associated with a group at the time the group is created. (The default "ALL" group has a default topology.) C) General graph topology is also supported. D) Provide access to all relevant information (via topological environmental inquiry routines) to allow users to define custom topologies. 7) Support for communication between processes in different groups without reference to the "ALL" group. 8) Dynamic groups (processes can join/leave an existing group while the total number of processes remains static.) 9) Dynamic process creation and identification. 10) Additional environmental inquiry and "setting" routines for performance optimization. A) Inquire/suggest system buffering resource allocation. B) Inquire relative node performance. C) Inquire relative interconnect performance. 11) Termination of programs. A) Kill all others nodes. B) Wait for all the other nodes to complete. C) Automatically de-allocate resources. 12) Standard organization. A) "Multi-level" organization of routines (users may use simple features without having to understand all of MPI). B) Simple "core" standard with (optional) Annexes. C) Simple "core" standard without Annexes. D) Should an MPI "subset" be defined that can be quickly implemented as a layer on top of existing systems? (This question was inspired by section 1.5 of Bill and Rusty's "A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47.) 13) Should there be future version(s) of MPI (MPI-2, etc.) (yes/no)? POINT-TO-POINT COMMUNICATION 14) Blocking send and receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). 15) Non-blocking receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 16) Non-blocking send. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 17) Low-level functionality (from Marc Snir's proposal) visible to users. A) Initialize. B) Start. C) Complete. D) Free. 18) "SECURE" versions of send and receive as proposed by Lyndon Clarke on 3/3/93. (Correct programs using SECURE routines are guaranteed to work regardless of system buffering resources.) 19) Ready-send. 20) Ready-receive. 21) Message handlers (a la iNTEL hrecv, TMC active messages, Express exhandle). A) Simple definition sufficient to support anticipated MPI requirements and make a self-consistent system. B) More advanced routines for remote procedure calls, etc... 22) Additional point-to-point primitives. A) Exchange between two processors. B) "Shift-exchange": read from one, send to another (superset of "A"). C) Contiguous data. D) Strided data. E) Buffer-descriptor (multiple data locations, multiple data types). F) Non-blocking versions? G) Check for completion (status()). H) Wait for completion (wait()). I) Cancel operation (cancel()). COLLECTIVE OPERATIONS 23) Blocking collective communication. A) Broadcast. B) Concatenate. C) Barrier. D) Sum. E) Extrema (max, min). F) Average. G) Sum of squared differences (error). H) Should strided data be allowed for these operations? I) Should buffer-descriptor be allowed for these operations? 24) "User-defined" blocking collective operations (Express "excombine()" style). 25) Non-blocking collective communication, collective operations, and user-defined operations (if this is desired, detailed ranking can be done later). 26) If any non-blocking collective operations are supported, what should completion/cancel options be? A) Check for completion (status()). B) Wait for completion (wait()). C) "Multiple cancel" operation (cancel()). 27) Should status()/wait()/cancel() syntax be consistent across all types of non-blocking operations? OTHER COMMUNICATION OPERATIONS 28) Probe (and associated "precv()" routines, if necessary). I/O 29) Standard I/O is supported in the standard fashion for the language on one process (no file I/O). This process can be identified by an environmental inquiry routine. (This is the current proposal.) 30) Serial I/O is supported in the standard fashion for the language on one process. This process can be identified by an environmental inquiry routine. (Rik Littlefield's proposal.) 31) Multiplexed I/O similar to that currently provided in systems such as CMMD (TMC), VERTEX (nCUBE) and Express (ParaSoft). Basically a way of defining the behavior of multi-node I/O statements which interact with a sequential device. No special considerations for parallel I/O channels. 32) Blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. 33) Non-blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. LANGUAGE BINDINGS 34) Ada 35) C 36) C++ 37) Fortran 77 38) Fortran 90 39) Other IMPLEMENTATION PERIOD There will be a delay between the generation of the standard and its implementation on various platforms. If this delay is too long users will continue to rely more and more heavily on vendor routines and specialized calls and MPI will become less and less attractive, particularly if new technology such as Fortran 90/HPF becomes effective. 40) What is the maximum delay that you think MPI can stand and still succeed (please feel free to comment if you wish)? ---------------------------------------------------------------------------- SURVEY RESPONSE FORM 1) A) B) C) D) 2) 3) 4) 5) A) B) 6) A) B) C) D) 7) 8) 9) 10) A) B) C) 11) A) B) C) 12) A) B) C) D) 13) (yes/no) 14) A) B) C) 15) A) B) C) D) E) F) 16) A) B) C) D) E) F) 17) A) B) C) D) 18) 19) 20) 21) A) B) 22) A) B) C) D) E) F) G) H) I) 23) A) B) C) D) E) F) G) H) I) 24) 25) 26) A) B) C) 27) 28) 29) 30) 31) 32) 33) 34) 35) 36) 37) 38) 39) 40) (free form): Comments? From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 14:28:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17761; Thu, 18 Mar 93 14:28:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22028; Thu, 18 Mar 93 14:22:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 14:22:45 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22020; Thu, 18 Mar 93 14:22:43 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA23272; Thu, 18 Mar 93 11:22:16 PST Date: Thu, 18 Mar 93 11:22:16 PST Message-Id: <9303181922.AA23272@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA10070; Thu, 18 Mar 93 11:22:13 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Subject: Re: MINUTES! In-Reply-To: <9166.9303181336@subnode.epcc.ed.ac.uk> References: <9166.9303181336@subnode.epcc.ed.ac.uk> Reply-To: knighten@SSD.intel.com (Bob Knighten) Yes, the minutes are late and yes, they will be available soon. Rusty has been traveling extensively since the meeting and now that I am supposed to finish them I have had some personal emergencies and my company has decided there are more important ways to spend my time. That is just an explanation of the delay, I am well aware of the importance of the minutes and will have them out shortly. -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:48:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17196; Thu, 25 Mar 93 10:48:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21805; Thu, 25 Mar 93 10:46:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21766; Thu, 25 Mar 93 10:45:43 -0500 Date: Thu, 25 Mar 93 15:44:59 GMT Message-Id: <18119.9303251544@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 12:41:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21413; Thu, 25 Mar 93 12:41:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26938; Thu, 25 Mar 93 12:40:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 12:40:10 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26850; Thu, 25 Mar 93 12:39:35 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07566; Thu, 25 Mar 93 11:33:39 CST Date: Thu, 25 Mar 93 11:33:39 CST From: Tony Skjellum Message-Id: <9303251733.AA07566@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Thank you for this additional work, Lyndon. Why don't we include this as the preface to our proposal? It looks good, but I will have to read it in detail, before rendering my complete view on this matter. Who are these other, new cc people... - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 14:01:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23412; Thu, 25 Mar 93 14:01:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00787; Thu, 25 Mar 93 13:59:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 13:59:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00747; Thu, 25 Mar 93 13:58:34 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07654; Thu, 25 Mar 93 12:52:38 CST Date: Thu, 25 Mar 93 12:52:38 CST From: Tony Skjellum Message-Id: <9303251852.AA07654@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Lyndon, as you know I was immensely under the gun yesterday, and my 9-hour trip was quite an ordeal, but I did get the sub-committee's proposal in on time, and it is basically quite good. I do feel that it is appropriate to include comparisons and even criticisms in the conclusions, but it was not done evenly, since mine was last. To this extent, I see why you did not like that. In same light, I can see why we should drop your name and Rik's name from proposal III, as this might unfairly indicate your support for proposal III (which is admittedly, an extended version of the programming model I support most, and is what Zipcode does, plus some additions). Please restrain yourself from completely dominating this subcommittee, because of the remarkable amount of time you are able to devote to it. To be fair, I cannot devote more than 2hr/day, and you are able to devote something like 12hr/day to it. People at the SIAM meeting were overwhelmed by the volume of mail you generate on different topics. You are a star performer, but you are also very demanding, and I must say that I have had some aggravations in this last weeks, but not only from you. I am trying not to push my personal viewpoint too hard. Please work from a viewpoint of cooperation with me, as we move towards the meeting next week. In this vein, see next paragraph. I will read your summary, but I already agree (note: agree) to drop all conclusion comments from all three subproposals that are "judgemental" (a loaded word). Since we are making a report out of this, the comparison should go in as latex. I will give this to Otto today (I will give it to him). Do you want me to latex your contribution, or will you (it is not too late there). Tell me. Ala Frankie & Johnny, I have to hang up now, since there is more mail from you to read :-)... I will not respond to that which is outdated by this letter. Please work on a latex version of your excellent new contribution, which also makes our joint report much better. I thank you. PS For rest of committee, the new contribution by Lyndon does not invalidate yesterday's tripartite proposal, it will merely be included in it. ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Mar 26 18:57:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28001; Fri, 26 Mar 93 18:57:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17683; Fri, 26 Mar 93 18:54:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 26 Mar 1993 18:54:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17674; Fri, 26 Mar 93 18:54:43 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Fri, 26 Mar 93 15:54 PST Message-Id: Date: Fri, 26 Mar 93 15:54 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: Draft available on netlib A PostScript version of the current MPI Draft Document is available on netlib. It is called draft326.ps.z.uu and is a postscript, compressed, uuencoded file. The size is 1830521 bytes. send draft326.ps.z.uu from mpi will get the file, or you may want to pull it over using xnetlib. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Fri Mar 26 19:42:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28702; Fri, 26 Mar 93 19:42:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19291; Fri, 26 Mar 93 19:40:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 26 Mar 1993 19:40:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from wk46.nas.nasa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19262; Fri, 26 Mar 93 19:40:26 -0500 Received: by wk46.nas.nasa.gov (5.61/1.34) id AA15348; Fri, 26 Mar 93 16:40:19 -0800 Date: Fri, 26 Mar 93 16:40:19 -0800 From: barszcz@nas.nasa.gov (Eric Barszcz) Message-Id: <9303270040.AA15348@wk46.nas.nasa.gov> To: mpi-comm@cs.utk.edu Subject: Multidisciplinary Position Paper %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 34 122 df45 D<000FC000106000603800801800801C01001C02201E02101E04101E04101E04101E08203C0820 3C0840380840780880F00700E00001C000030000060000180000200000C0000100000200000400 100800301000202000605F80C063FFC040FF80807F00801E0017227CA019>50 D<000FC000386000703000E03001C0380380380780380700380F00380F00380F00381E00781E00 781E00781E00F81E00F01C00F00E01F00E02F00605E00309E001F1E00003C00003C00003800007 00000700600E00F00C00F01800E0300080600041C0003F000015227BA019>57 D<0000030000000300000007000000070000000F0000000F0000001F0000002F0000002F000000 4F0000004F80000087800000878000010780000207800002078000040780000407800008078000 080780001007800030078000200780007FFF80004007C0008007C0008003C0010003C0030003C0 020003C0040003C0040003C00C0003C03C0007C0FF003FFC1E237DA224>65 D<00007F00800003808100000E00630000380027000070001F0000E0000E0001C0000E00038000 0E000700000E000F000004000E000004001E000004003C000004003C0000080078000000007800 0000007800000000F000000000F000000000F000000000F000000000F000000000E000000000E0 00002000E000002000E000004000E000004000F000008000700000800070000100003800020000 18000400001C0008000006003000000381C0000000FE000000212479A223>67 D<00FFFFFF000F000F000F0003000F0003001E0003001E0003001E0002001E0002003C0002003C 0002003C0102003C010000780200007802000078060000780E0000FFFC0000F00C0000F00C0000 F00C0001E0080001E0080001E0080001E0000003C0000003C0000003C0000003C0000007800000 0780000007800000078000000F800000FFFC000020227DA120>70 D<00007F0080000380810000 0E00630000380027000070001F0000E0000E0001C0000E000380000E000700000E000F00000400 0E000004001E000004003C000004003C00000800780000000078000000007800000000F0000000 00F000000000F000000000F000000000F0003FFC00E00001E000E00001E000E00001E000E00003 C000E00003C000F00003C000700003C0007000078000380007800018000F80001C001380000600 2300000381C1000000FE000000212479A226>I<00FFF87FFC000F000780000F000780000F0007 80001E000F00001E000F00001E000F00001E000F00003C001E00003C001E00003C001E00003C00 1E000078003C000078003C000078003C000078003C0000FFFFF80000F000780000F000780000F0 00780001E000F00001E000F00001E000F00001E000F00003C001E00003C001E00003C001E00003 C001E000078003C000078003C000078003C000078003C0000F8007C000FFF87FFC0026227DA124 >I<00FFF8000F00000F00000F00001E00001E00001E00001E00003C00003C00003C00003C0000 780000780000780000780000F00000F00000F00000F00001E00001E00001E00001E00003C00003 C00003C00003C0000780000780000780000780000F8000FFF80015227DA113>I<0007FFC00000 3C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F00000 00F0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C00000078000 000780000007800000078000000F0000000F0000380F0000780F0000F81E0000F81E0000F03C00 00403800004070000021E000001F8000001A237CA11A>I<00FF800007FC000F80000F80000F80 001780000F80001780001780002F000013C0002F000013C0004F000013C0008F000023C0009E00 0023C0011E000023C0011E000023C0021E000043C0043C000043C0043C000043C0083C000041E0 083C000081E01078000081E02078000081E02078000081E04078000101E040F0000101E080F000 0101E100F0000101E100F0000200F201E0000200F201E0000200F401E0000200F801E0000400F8 03C0000400F003C0000400F003C0000C00E003C0001E00C007C000FFC0C07FFC002E227DA12C> 77 D<00FFFFE0000F0038000F001E000F000E001E0007001E0007001E0007001E0007003C000F 003C000F003C000F003C001E0078001E0078003C00780078007800E000F003C000FFFE0000F000 0000F0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C000000780 00000780000007800000078000000F800000FFF8000020227DA121>80 D<0001F020000E0C4000 1802C0003001C0006001C000E0018000C0018001C0018001C0018003C0010003C0010003C00000 03C0000003E0000001F8000001FF000000FFE000007FF000001FF8000003FC0000007C0000003C 0000001E0000001E0000001E0020001C0020001C0020001C002000180060003800600030007000 60007000C000C8018000C607000081FC00001B247DA21B>83 D<1FFFFFF81E03C0381803C01830 03C01820078018200780184007801040078010400F0010800F0010800F0010000F0000001E0000 001E0000001E0000001E0000003C0000003C0000003C0000003C00000078000000780000007800 000078000000F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0000003E0 0000FFFF00001D2277A123>I<01020408102020404080B8FCFCF870080F76A20F>96 D<00F8C00185C00705C00E03800E03801C03803C0380380700780700780700780700F00E00F00E 00F00E00F00E10F01C20701C20703C20305C40308C400F078014157B9419>I<03C03F80038003 80038007000700070007000E000E000E000E001C001CF81D0C1E0E3C0638073807380F700F700F 700F700FE01EE01EE01EE03CE038E038607060E031C01F0010237BA216>I<007E0001C1000301 800703800E07801C07803C0000380000780000780000780000F00000F00000F00000F00000F001 00700100700200300C001830000FC00011157B9416>I<00003C0003F800003800003800003800 00700000700000700000700000E00000E00000E00000E00001C000F9C00185C00705C00E03800E 03801C03803C0380380700780700780700780700F00E00F00E00F00E00F00E10F01C20701C2070 3C20305C40308C400F078016237BA219>I<00F803840E021C023C0238027804F018FFE0F000F0 00E000E000E000E000E002E0026004701830600F800F157A9416>I<00003E0000470000CF0001 8F000186000380000380000380000700000700000700000700000700000E0000FFF0000E00000E 00000E00001C00001C00001C00001C00001C000038000038000038000038000038000070000070 0000700000700000700000E00000E00000E00000E00000C00001C00001C000718000F18000F300 006200003C0000182D82A20F>I<001F180030B800E0B801C07001C0700380700780700700E00F 00E00F00E00F00E01E01C01E01C01E01C01E01C01E03800E03800E0780060B8006170001E70000 0700000700000E00000E00000E00701C00F01800F0300060E0003F8000151F7E9416>I<00F000 0FE00000E00000E00000E00001C00001C00001C00001C000038000038000038000038000070000 071F0007218007C0C00F00E00F00E00E00E00E00E01C01C01C01C01C01C01C01C0380380380380 380380380704700708700E08700E10700610E006206003C016237DA219>I<00C001E001C001C0 000000000000000000000000000000001C002300430043008700870087000E000E001C001C001C 00380038003840708070807080710032001C000C217BA00F>I<01E01FC001C001C001C0038003 800380038007000700070007000E000E000E000E001C001C001C001C0038003800380038007000 700070007100E200E200E200E200640038000B237CA20C>108 D<1C0F80F8002610C10C004760 66060087807807008780780700870070070087007007000E00E00E000E00E00E000E00E00E000E 00E00E001C01C01C001C01C01C001C01C01C001C01C03820380380384038038070403803807080 380380308070070031003003001E0024157B9428>I<1C0F002631C04740C08780E08780E08700 E08700E00E01C00E01C00E01C00E01C01C03801C03801C03801C0704380708380E08380E103806 107006203003C017157B941B>I<007E0001C3000381800701C00E01C01C01E03C01E03801E078 01E07801E07801E0F003C0F003C0F00380F00780700700700E00700C0030180018700007C00013 157B9419>I<01C1F002621804741C08780C08700E08700E08701E00E01E00E01E00E01E00E01E 01C03C01C03C01C03C01C07803807003807003C0E003C1C0072380071E000700000700000E0000 0E00000E00000E00001C00001C00001C0000FFC000171F7F9419>I<1C1F002620804741C08783 C08703C08701808700000E00000E00000E00000E00001C00001C00001C00001C00003800003800 0038000038000070000030000013157B9415>114 D<00FC000183000200800401800C03800C03 000C00000F00000FF00007FC0003FE00003E00000F00000700700700F00600F00600E004004008 002030001FC00011157D9414>I<00C001C001C001C001C003800380038003800700FFF8070007 000E000E000E000E001C001C001C001C003800380038003810702070207040708031001E000D1F 7C9E10>I<1E00602300E04380E04381C08381C08701C08701C00703800E03800E03800E03801C 07001C07001C07001C07081C0E10180E101C0E101C1E200C262007C3C016157B941A>I<1E0030 2300704380704380E08380E08700E08700E00701C00E01C00E01C00E01C01C03801C03801C0380 1C03801C07001C07001C07001C0F000C3E0003CE00000E00000E00001C00601C00F03800F03000 E0600080C0004380003E0000151F7B9418>121 D E /Fb 1 81 df80 D E /Fc 3 21 df0 D2 D<0000000C0000003C000000F0000003C000000F0000003C000000F0000007C000001F0000 0078000001E00000078000001E00000078000000E0000000780000001E0000000780000001E000 0000780000001F00000007C0000000F00000003C0000000F00000003C0000000F00000003C0000 000C0000000000000000000000000000000000000000000000000000000000000000FFFFFFFCFF FFFFFC1E277C9F27>20 D E /Fd 3 52 df<07C018303018701C600C600CE00EE00EE00EE00EE0 0EE00EE00EE00EE00E600C600C701C30181C7007C00F157F9412>48 D<0F8030E040708030C038 E0384038003800700070006000C00180030006000C08080810183FF07FF0FFF00D157E9412>50 D<0FE030306018701C701C001C00180038006007E000300018000C000E000EE00EE00EC00C4018 30300FE00F157F9412>I E /Fe 3 106 df<07F0000FE000F0001E0000B8001E0000B8002E0000 B8004E000138005C000138009C000138011C00011C011C00021C023800021C043800021C043800 021C083800041C107000040E107000040E207000040E407000080E40E000080E80E000080F00E0 00080700E000180601C000FE040FF80023177F9622>77 D<07F007F800F000C000B8008000B800 80009C0080011C0100011E0100010E0100010E0100020702000207020002038200020382000401 C4000401C4000400E4000400E4000800780008007800080038000800380018001000FE0010001D 177F961C>I<0300038003000000000000000000000000001C002400460046008C000C00180018 00180031003100320032001C0009177F960C>105 D E /Ff 9 84 df<0F00187F00380F00380E 00700E00700E00700E00E01C00E01C01C01C01801C0380380700380600380C0038180070300070 600070C000730000FC0000F0000015157D9418>23 D<003F0001FFC00381E00400400800001000 001000001000001060000B98000FF800100000200000400000400000400000800000C000804000 80600100380E001FFC0007E00013177F9517>34 D<70F8F8F87005057C840E>58 D<70F8FCFC7404040404080810102040060F7C840E>I<00007F00400003C0C080000E00218000 1C0013800070000F8000E000070001C0000700038000070007000007000F000002000E00000200 1E000002003C000002003C00000400780000000078000000007800000000F000000000F0000000 00F000000000F000000000F0003FFF00E00000F000E00000F000E00000F000E00001E000F00001 E000F00001E000700001E000700003C000380003C000180007C0000C0009C00006001180000380 E08000007F00000022247DA226>71 D<007FC00001FF0007C00003E00007C00005E00007C00005 E00009E0000BC00009E0000BC00009E00013C00009E00023C00011E00027800011E00047800011 E00047800011E00087800021E0010F000020F0010F000020F0020F000020F0040F000040F0041E 000040F0081E000040F0081E000040F0101E000080F0203C00008078203C00008078403C000080 78803C0001007880780001007900780001007900780001007A00780002007C00F00002007C00F0 0002003800F00006003800F0000F003001F000FFE0203FFF0030227EA12F>77 D<007FC003FF0007C000780007C000600005E000200009E000400009E000400008F000400008F0 00400010F800800010780080001078008000103C008000203C010000203E010000201E01000020 1E010000400F020000400F020000400F020000400782000080078400008007C400008003C40000 8003C400010001E800010001E800010001F800010000F800020000F00002000070000200007000 06000070000F00002000FFE000200028227EA127>I<007FFFF0000007801C000007800F000007 800700000F000380000F000380000F000380000F000380001E000780001E000780001E00078000 1E000F00003C000F00003C001E00003C003C00003C007000007801E000007FFF00000078000000 007800000000F000000000F000000000F000000000F000000001E000000001E000000001E00000 0001E000000003C000000003C000000003C000000003C000000007C0000000FFFC00000021227E A11F>80 D<0003F010000E0C2000180260002001E0004000E000C000C0008000C0018000C00180 00C00380008003800080038000000380000003C0000001F0000001FE000000FFE000007FF00000 1FF8000001FC0000003C0000001C0000000E0000000E0000000E0020000C0020000C0020000C00 2000080060001800600010007000200070004000C8008000C603000081FC00001C247DA21E>83 D E /Fg 31 122 df45 D<60F0F06004047C830C>I<030007003F00C7 000700070007000700070007000700070007000700070007000700070007000700070007000700 0700070007000F80FFF80D1C7C9B15>49 D<07C01830201C400C400EF00FF80FF807F807700700 0F000E000E001C001C00380070006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B 15>I<00F0030C06040C0E181E301E300C700070006000E3E0E430E818F00CF00EE006E007E007 E007E007E007600760077006300E300C18180C3003E0101D7E9B15>54 D<03C00C301818300C70 0C600EE006E006E007E007E007E007E0076007700F300F18170C2707C700060006000E300C780C 78187010203030C00F80101D7E9B15>57 D<000600000006000000060000000F0000000F000000 0F00000017800000178000001780000023C0000023C0000023C0000041E0000041E0000041E000 0080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E 0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 D<001F808000E06180018019800700 07800E0003801C0003801C00018038000180780000807800008070000080F0000000F0000000F0 000000F0000000F0000000F0000000F0000000F000000070000080780000807800008038000080 1C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>67 D78 D<07E0801C1980300580700380600180E00180E00080E00080E00080F00000F800007C00 007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001C08001C08001C08001C0C001 80C00180E00300D00200CC0C0083F800121E7E9C17>83 D<7FFFFFC0700F01C0600F00C0400F00 40400F0040C00F0020800F0020800F0020800F0020000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000001F800003FFFC001B1C7F9B1E>I<1FC000307000783800781C0030 1C00001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F 870012127E9115>97 D<07E00C301878307870306000E000E000E000E000E000E0006000700430 0418080C3007C00E127E9112>99 D<003F00000700000700000700000700000700000700000700 00070000070000070003E7000C1700180F00300700700700600700E00700E00700E00700E00700 E00700E00700600700700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C 700E6006E006FFFEE000E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C 071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C 00381C00381C00381C001818001C38000C300013C0001000003000001800001FF8001FFF001FFF 803003806001C0C000C0C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C 001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I107 DIII<03F0000E1C00180600300300700380600180E001C0E001C0E001C0E001C0E001C0 E001C06001807003803003001806000E1C0003F00012127F9115>II 114 D<1F9030704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608FC0 0D127F9110>I<04000400040004000C000C001C003C00FFE01C001C001C001C001C001C001C00 1C001C001C101C101C101C101C100C100E2003C00C1A7F9910>II119 D121 D E /Fh 2 122 df<040004000400C460E4E0 3F800E003F80E4E0C4600400040004000B0D7E8D11>3 D<0C000C000C000C000C000C00FFC0FF C00C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000A1A 7E9310>121 D E /Fi 41 122 df<0007F80FF000007FFE7FFC0001F80FF80E0003E00FE01F00 07C01FC03F000F801F803F000F801F803F000F800F801E000F800F800C000F800F8000000F800F 8000000F800F8000000F800F800000FFFFFFFFFF00FFFFFFFFFF000F800F801F000F800F801F00 0F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F 801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F00 0F800F801F000F800F801F000F800F801F007FF07FF0FFE07FF07FF0FFE02B237FA22F>14 D<387CFEFFFF7F3B03030706060C1C18702008117CA210>39 D<387CFEFEFE7C3807077C8610> 46 D<00180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC070 07E07C07F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C000 0F80001F00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07F FFF0FFFFF0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F8 1F03F81F03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC 3C00FE7E00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C> I<0000E00001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187 E00307E00707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007 E00007E00007E00007E00007E000FFFE00FFFE17207E9F1C>I<1000201E01E01FFFC01FFF801F FF001FFE001FF8001BC00018000018000018000018000019FC001FFF001E0FC01807E01803E000 03F00003F00003F80003F83803F87C03F8FE03F8FE03F8FC03F0FC03F07007E03007C01C1F800F FF0003F80015207D9F1C>I<001F8000FFE003F07007C0F00F01F81F01F83E01F83E01F87E00F0 7C00007C0000FC0800FC7FC0FCFFE0FD80F0FF00F8FE007CFE007CFC007EFC007EFC007EFC007E 7C007E7C007E7C007E3C007C3E007C1E00F80F00F00783E003FFC000FF0017207E9F1C>I<0000 70000000007000000000F800000000F800000000F800000001FC00000001FC00000003FE000000 03FE00000003FE00000006FF000000067F0000000E7F8000000C3F8000000C3F800000183FC000 00181FC00000381FE00000300FE00000300FE00000600FF000006007F00000E007F80000FFFFF8 0000FFFFF800018001FC00018001FC00038001FE00030000FE00030000FE000600007F00060000 7F00FFE00FFFF8FFE00FFFF825227EA12A>65 D<0003FE0080001FFF818000FF01E38001F8003F 8003E0001F8007C0000F800F800007801F800007803F000003803F000003807F000001807E0000 01807E00000180FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE00 000000FE000000007E000000007E000001807F000001803F000001803F000003801F800003000F 8000030007C000060003F0000C0001F800380000FF00F000001FFFC0000003FE000021227DA128 >67 DII72 DI76 DI80 D82 D<01FC0407FF8C1F03FC3C007C7C003C78001C78001CF8000CF8000CFC000CFC00 00FF0000FFE0007FFF007FFFC03FFFF01FFFF80FFFFC03FFFE003FFE0003FF00007F00003F0000 3FC0001FC0001FC0001FE0001EE0001EF0003CFC003CFF00F8C7FFE080FF8018227DA11F>I<07 FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F0003FF003FDF01FC1F03F01F07E 01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18167E951B>97 DI<00FF8007FFE00F83 F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00007C00007E00 007E00003E00301F00600FC0E007FF8000FE0014167E9519>I<0001FE000001FE0000003E0000 003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E00 01FC3E0007FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC003E00FC003E 00FC003E00FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E001E00FE000F83 BE0007FF3FC001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC 00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F00300FC07003 FFC000FF0015167E951A>I<003F8000FFC001E3E003C7E007C7E00F87E00F83C00F80000F8000 0F80000F80000F80000F8000FFFC00FFFC000F80000F80000F80000F80000F80000F80000F8000 0F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80007FF8007FF800 13237FA211>I<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03 C03E07C01F0F801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC7000 3EF0001EF0001EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C>II<1C003F007F007F007F003F001C 000000000000000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F00FFE0FFE00B247EA310>I108 DII<00FE0007FFC00F83E01E00 F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007EFC007E7C007C7C00 7C3E00F81F01F00F83E007FFC000FE0017167E951C>II<00FE030007FF87000FC1C7001F006F003F003F007E003F007E001F007C001F00FC001F00FC 001F00FC001F00FC001F00FC001F00FC001F00FC001F007E001F007E001F003E003F001F007F00 0FC1DF0007FF9F0001FC1F0000001F0000001F0000001F0000001F0000001F0000001F0000001F 0000001F000000FFE00000FFE01B207E951E>II<0FF3003FFF00781F00600700E00300E00300F00300FC0000 7FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00 C7F00011167E9516>I<0180000180000180000180000380000380000780000780000F80003F80 00FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>II119 D121 D E /Fj 2 122 df<020002000200C218F2783AE00F800F803AE0F278C2180200020002000D0E 7E8E12>3 D<06000600060006000600060006000600FFF0FFF006000600060006000600060006 000600060006000600060006000600060006000600060006000C1D7E9611>121 D E /Fk 83 124 df<001F83E000706E3000C07C780180F8780380F07807007000070070000700 7000070070000700700007007000070070000700700007007000FFFFFFC0070070000700700007 007000070070000700700007007000070070000700700007007000070070000700700007007000 070070000700700007007000070070000700700007007000070078007FE3FF801D2380A21C>11 D<001FC0000070200000C010000180380003807800070078000700300007000000070000000700 000007000000070000000700000007000000FFFFF8000700780007003800070038000700380007 003800070038000700380007003800070038000700380007003800070038000700380007003800 07003800070038000700380007003800070038007FE1FF80192380A21B>I<001FD80000703800 00C078000180780003807800070038000700380007003800070038000700380007003800070038 000700380007003800FFFFF8000700380007003800070038000700380007003800070038000700 380007003800070038000700380007003800070038000700380007003800070038000700380007 00380007003800070038007FF3FF80192380A21B>I<000FC07F00007031C08000E00B00400180 1E00E003803E01E007003C01E007001C00C007001C000007001C000007001C000007001C000007 001C000007001C000007001C0000FFFFFFFFE007001C01E007001C00E007001C00E007001C00E0 07001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00 E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E07FF1FF CFFE272380A229>I<7038F87CFC7EFC7E743A0402040204020402080408041008100820104020 0F0F7EA218>34 D<0780000C001840001C0018200018003010007800701C00F0006013FF6000E0 0800E000E00800C000E00801C000E008038000E008030000E008070000E0080E0000E0080C0000 60101C000070101800003010380000182070000018406000000780E03C000001C04200000180C1 00000381810000070380800006030080000E030040000C070040001C0700400038070040003007 0040007007004000E007004000C007004001C00300400180030080038003808007000181000600 00C1000E000042000C00003C0022287DA429>37 D<70F8FCFC7404040404080810102040060F7C A20E>39 D<00200040008001000300060004000C000C0018001800300030003000700060006000 6000E000E000E000E000E000E000E000E000E000E000E000E000E000E000600060006000700030 0030003000180018000C000C0004000600030001000080004000200B327CA413>I<8000400020 00100018000C000400060006000300030001800180018001C000C000C000C000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000C000C000C001C00180018001800300030006 00060004000C00180010002000400080000B327DA413>I<000180000001800000018000000180 000001800000018000000180000001800000018000000180000001800000018000000180000001 80000001800000018000FFFFFFFEFFFFFFFE000180000001800000018000000180000001800000 018000000180000001800000018000000180000001800000018000000180000001800000018000 000180001F227D9C26>43 D<70F8FCFC7404040404080810102040060F7C840E>II<70F8F8F87005057C840E>I<00018000018000038000030000030000070000060000 0600000E00000C00000C00001C0000180000180000180000380000300000300000700000600000 600000E00000C00000C00001C0000180000180000380000300000300000700000600000600000E 00000C00000C00000C00001C0000180000180000380000300000300000700000600000600000E0 0000C00000C0000011317DA418>I<01F000071C000C06001803003803803803807001C07001C0 7001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0 F001E0F001E0F001E07001C07001C07001C07803C03803803803801C07000C0600071C0001F000 13227EA018>I<008003800F80F380038003800380038003800380038003800380038003800380 03800380038003800380038003800380038003800380038003800380038007C0FFFE0F217CA018 >I<03F0000C1C001007002007804003C04003C08003E0F003E0F801E0F801E0F801E02003E000 03E00003C00003C0000780000700000E00001C0000180000300000600000C00001800001000002 00200400200800201800603000403FFFC07FFFC0FFFFC013217EA018>I<03F8000C1E00100700 2007804007C07807C07803C07807C03807C0000780000780000700000F00000E0000380003F000 001C00000F000007800007800003C00003C00003E02003E07003E0F803E0F803E0F003C04003C0 400780200780100F000C1C0003F00013227EA018>I<000200000600000E00000E00001E00001E 00002E00004E00004E00008E00008E00010E00020E00020E00040E00040E00080E00100E00100E 00200E00200E00400E00800E00FFFFF8000E00000E00000E00000E00000E00000E00000E00001F 0001FFF015217FA018>I<1000801E07001FFF001FFE001FF80013E00010000010000010000010 000010000010000010F800130E001407001803801003800001C00001C00001E00001E00001E000 01E07001E0F001E0F001E0E001C08001C04003C04003802007001006000C1C0003F00013227EA0 18>I<007E0001C1000300800601C00E03C01C03C0180180380000380000780000700000700000 F0F800F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E0 7001E07001E03801C03801C01803801C03000C0600070C0001F00013227EA018>I<4000006000 007FFFE07FFFC07FFFC0400080C001008001008002008002000004000008000008000010000030 0000200000600000600000600000E00000C00000C00001C00001C00001C00001C00003C00003C0 0003C00003C00003C00003C00003C00003C00001800013237DA118>I<01F800060E0008030010 01802001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003F80003 FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C0006060004060 00C03000801803000E0E0003F00013227EA018>I<01F000060C000C0600180700380380700380 700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E07001E07003E03803E01805E0 0C05E00619E003E1E00001C00001C00001C0000380000380300300780700780600700C00201800 1030000FC00013227EA018>I<70F8F8F870000000000000000000000070F8F8F87005157C940E> I61 D<07E01838201C400E800FF00FF00FF00F000F000E00 1C00380030006000C000C000800080018001000100010001000100010000000000000000000000 038007C007C007C0038010237DA217>63 D<0001800000018000000180000003C0000003C00000 03C0000005E0000005E000000DF0000008F0000008F0000010F800001078000010780000203C00 00203C0000203C0000401E0000401E0000401E0000800F0000800F0000FFFF0001000780010007 80030007C0020003C0020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF80 0FFF20237EA225>65 DI<0007E0 100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C0000303C00 00307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F800000078000000780000107C0000103C0000103C0000101C0000201E0000200E000040 070000400380008001C0010000E0020000381C000007E0001C247DA223>IIII<0007F008003C0C18 00E0021801C001B8038000F8070000780F0000381E0000381E0000183C0000183C0000187C0000 087800000878000008F8000000F8000000F8000000F8000000F8000000F8000000F8000000F800 1FFF780000F8780000787C0000783C0000783C0000781E0000781E0000780F0000780700007803 8000B801C000B800E00318003C0C080007F00020247DA226>III<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00700F00F80F00F80F00F80E00F01E00401C0020380018700007C0001423 7EA119>IIIII<000FE00000783C0000E00E0003C007800780 03C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C7800003C7800003CF8 00003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003E7800003C 7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E0078003C003C0078000E00E 0000783C00000FE0001F247DA226>II<000FE00000783C0000E00E0003C00780078003C00F0001E00E0000E01E0000F03E0000F8 3C0000787C00007C7C00007C7800003C7800003CF800003EF800003EF800003EF800003EF80000 3EF800003EF800003EF800003EF800003E7800003C7C00007C7C00007C3C0000783C0000781E03 80F00E0420E00F0801E0078813C003C8178000E80E00007C3C02000FEC0200000C0200000C0200 000E0600000F0E000007FC000007FC000007F8000003F8000001E01F2D7DA226>II<03F0200C0C601802603001E07000E0600060 E00060E00060E00020E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80 003FC00007E00001E00000F00000F0000070800070800070800070800070C00060C00060E000C0 F000C0C80180C6070081FC0014247DA21B>I<7FFFFFF878078078600780184007800840078008 40078008C007800C80078004800780048007800480078004000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000FC00003FFFF001E 227EA123>IIII<7FF807FF0007E001F80003C000E00003E000C00001E000800000F001000000F803 00000078020000007C040000003E0C0000001E080000001F100000000FB000000007A000000007 C000000003E000000001E000000001F000000003F80000000278000000047C0000000C3E000000 081E000000101F000000300F80000020078000004007C00000C003E000008001E000010001F000 030000F000070000F8001F8001FC00FFE007FFC022227FA125>II91 D<080410082010201040204020 8040804080408040B85CFC7EFC7E7C3E381C0F0F7AA218>II<08102020404080808080B8FCFC7C38060F7DA20E>96 D<0FE0001838003C0C003C0E0018 070000070000070000070000FF0007C7001E07003C0700780700700700F00708F00708F00708F0 0F087817083C23900FC1E015157E9418>I<0E0000FE00001E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C 0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C8060 0C41C0083F0017237FA21B>I<01FE000703000C07801C0780380300780000700000F00000F000 00F00000F00000F00000F00000F000007000007800403800401C00800C010007060001F8001215 7E9416>I<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00000E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F0 00E0F000E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I< 01FC000707000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000 F000007000007800203800201C00400E008007030000FC0013157F9416>I<003C00C6018F038F 030F070007000700070007000700070007000700FFF80700070007000700070007000700070007 0007000700070007000700070007000700070007807FF8102380A20F>I<00007001F198071E18 0E0E181C07001C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F000100000 1000001800001800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C00018600030 6000303800E00E038003FE0015217F9518>I<0E0000FE00001E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00 700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E0070FFE7FF18237FA21B>I<1C001E003E001E001C00000000000000000000000000000000 000E00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E00FFC00A227FA10E>I<01C003E003E003E001C00000000000000000000000000000000001E0 0FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E060E0F0C0F18061803E000B2C82A10F>I<0E0000FE00001E00000E00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01 C00E01800E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03 C00E01C00E01E00E00F00E00F8FFE3FE17237FA21A>I<0E00FE001E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E00FFE00B237FA20E>I<0E1FC07F00FE60E183801E807201C00F00 3C00E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0 0E003800E0FFE3FF8FFE27157F942A>I<0E1F80FE60C01E80E00F00700F00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FF E7FF18157F941B>I<01FC000707000C01801800C03800E0700070700070F00078F00078F00078 F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC0015157F9418> I<0E1F00FE61C00E80600F00700E00380E003C0E001C0E001E0E001E0E001E0E001E0E001E0E00 1E0E001E0E003C0E003C0E00380F00700E80E00E41C00E3F000E00000E00000E00000E00000E00 000E00000E00000E00000E0000FFE000171F7F941B>I<01F8200704600E02601C01603801E078 00E07800E0F000E0F000E0F000E0F000E0F000E0F000E0F000E07000E07800E03801E01C01E00C 02E0070CE001F0E00000E00000E00000E00000E00000E00000E00000E00000E00000E0000FFE17 1F7E941A>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E 000E000E000F00FFF010157F9413>I<0F8830786018C018C008C008E008F0007F803FE00FF001 F8003C801C800C800CC00CC008E018D0308FC00E157E9413>I<02000200020002000600060006 000E001E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E040E040E040E04 0E040E040708030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I IIII<3FFFC03803803007802007 00600E00401C00403C0040380000700000E00001E00001C0000380400700400F00400E00C01C00 80380080780180700780FFFF8012157F9416>II E /Fl 23 123 df<000003800000000007C00000000007C0000000000FE0000000000FE000000000 0FE0000000001FF0000000001FF0000000003FF8000000003FF8000000003FF80000000073FC00 00000073FC00000000F3FE00000000E1FE00000000E1FE00000001C0FF00000001C0FF00000003 C0FF80000003807F80000007807FC0000007003FC0000007003FC000000E003FE000000E001FE0 00001E001FF000001C000FF000001FFFFFF000003FFFFFF800003FFFFFF80000780007FC000070 0003FC0000700003FC0000E00001FE0000E00001FE0001E00001FF0001C00000FF0001C00000FF 00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE2F297EA834>65 D69 D77 D80 D<01FF800007FFF0000F81FC001FC0FE001FC07F001FC07F001FC03F800F80 3F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F801FE03F803F803F807F803F807F 003F80FE003F80FE003F80FE003F80FE007F80FF007F807F00FFC03F83DFFC0FFF0FFC01FC03FC 1E1B7E9A21>97 D<001FF80000FFFE0003F01F000FE03F801FC03F803F803F803F803F807F801F 007F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00 00007F0000007F8000003F8001C03FC001C01FC003C00FE0078003F01F0000FFFC00001FE0001A 1B7E9A1F>99 D<00003FF80000003FF80000003FF800000003F800000003F800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800001FE3F80000FFFBF80003F03FF8000FE00FF8001FC007F8003F8003F8003F8003F800 7F8003F8007F0003F800FF0003F800FF0003F800FF0003F800FF0003F800FF0003F800FF0003F8 00FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8003F8003F8003F8007F8001FC00F F8000FE01FF80003F03FFF8000FFF3FF80003FC3FF80212A7EA926>I<003FE00001FFF80003F0 7E000FE03F001FC01F803F800FC03F800FC07F000FC07F0007E0FF0007E0FF0007E0FF0007E0FF FFFFE0FFFFFFE0FF000000FF000000FF000000FF0000007F0000007F8000003F8000E03F8001E0 1FC001C00FE003C003F81F8000FFFE00001FF0001B1B7E9A20>I<0007F0003FFC00FE3E01FC7F 03F87F03F87F07F07F07F03E07F00007F00007F00007F00007F00007F00007F000FFFFC0FFFFC0 FFFFC007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF807FFF80182A7E A915>I<00FF81F003FFE7FC0FC1FE7C1F80FC7C3F80FE7C3F007E107F007F007F007F007F007F 007F007F007F007F007F007F003F007E003F80FE001F80FC000FC1F8001FFFE00018FF80003800 00003C0000003C0000003E0000003FFFF8003FFFFF001FFFFFC00FFFFFE007FFFFF01FFFFFF07E 0007F87C0001F8F80001F8F80000F8F80000F8F80000F8FC0001F87E0003F03F0007E00FC01F80 03FFFE00007FF0001E287E9A22>I<07001FC01FE03FE03FE03FE01FE01FC00700000000000000 0000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE0F2B7DAA14>105 D108 DII<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F0 7F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007 F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE0001D1B7E9A22> II114 D<03FE300FFFF03E03F07800F07000F0F00070F0 0070F80070FC0000FFE000FFFE007FFFC03FFFE01FFFF007FFF800FFFC0003FC0000FCE0007CE0 003CF0003CF0003CF80078FC0078FF01F0F7FFC0C1FF00161B7E9A1B>I<007000007000007000 00700000F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F038 07F03807F03807F03807F03807F03803F87001F8F000FFE0001F8015267FA51B>II120 DI<3FFFFF803FFFFF803F00FF803C00FF003801FE007803FC00 7807FC007007F800700FF000701FE000001FE000003FC000007F800000FF800000FF000001FE03 8003FC038003FC038007F803800FF007801FF007801FE007003FC00F007F801F00FF807F00FFFF FF00FFFFFF00191B7E9A1F>I E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 432 658 a Fl(A)23 b(Mo)r(del)f(for)i(Executing)e(Multidisci)o(pl)o(i)o (nary)663 732 y(and)i(Multizonal)d(Programs)391 841 y Fk(Eric)16 b(Barszcz)658 823 y Fj(\003)792 841 y Fk(Sisira)g(K.)g(W)l(eeratunga)1247 823 y Fj(y)1383 841 y Fk(Eddy)g(Pramono)1705 823 y Fj(y)719 960 y Fk(NAS)g(Applied)f(Researc)o(h)g(Branc)o(h)731 1018 y(NASA)g(Ames)f (Researc)o(h)i(Cen)o(ter)788 1076 y(Mo\013ett)g(Field,)f(CA)h(94035)884 1240 y(Marc)o(h)g(26,)h(1993)941 1647 y Fi(Abstract)345 1733 y Fk(This)10 b(pap)q(er)i(presen)o(ts)e(the)h(mo)q(del)e(of)j(computing)d (for)i(m)o(ultidiscipli)o(nary)d(and/or)271 1793 y(m)o(ultizonal)24 b(applications)i(on)h(parallel)e(mac)o(hines)f(b)q(eing)i(pursued)h(at)f (NASA)271 1853 y(Ames)15 b(Researc)o(h)h(Cen)o(ter.)21 b(The)16 b(mo)q(del)f(describ)q(es)h(execution)f(on)i(a)g(parallel)f(ma-)271 1914 y(c)o(hine)c(with)h(some)f(n)o(um)o(b)q(er)g(of)h(pro)q(cessors,)i(eac)o (h)d(with)h(lo)q(cal)g(memory)l(,)e(connected)271 1974 y(b)o(y)20 b(a)g(net)o(w)o(ork.)31 b(It)20 b(do)q(es)g(not)g(discuss)g(pro)q(cessor)h (rates,)g(size)e(of)h(memory)l(,)d(net-)271 2034 y(w)o(ork)h(latency)l(,)g (net)o(w)o(ork)f(bandwidth,)i(net)o(w)o(ork)e(connectivit)o(y)l(,)f(disk)i (space,)g(I/O)271 2094 y(bandwidth)k(or)f(mass)f(storage.)36 b(It)20 b(also)h(do)q(es)h(not)f(discuss)g(language)g(and)h(dis-)271 2154 y(cipline)17 b(sp)q(eci\014c)g(algorithm)g(issues.)27 b(Mo)q(del)18 b(rationalit)o(y)f(and)i(functionalit)o(y)e(are)271 2214 y(presen)o(ted.)p 149 2594 719 2 v 205 2625 a Fh(\003)224 2640 y Fg(The)d(author)g(is)g(an)g(emplo)o(y)o(ee)e(of)i(NASA.)207 2675 y Fh(y)224 2690 y Fg(The)k(author)f(is)g(an)g(emplo)o(y)o(ee)f(of)h (Computer)f(Sciences)j(Co.)28 b(This)17 b(w)o(ork)g(w)o(as)g(funded)g (through)g(NASA)149 2740 y(con)o(tract)e(NAS)f(2-12961)p eop %%Page: 2 2 bop 149 42 a Fi(1)56 b(In)n(tro)r(duction)149 151 y Fk(The)18 b(problem)e(domain)g(of)i(particular)f(in)o(terest)f(to)h(NASA)f(Ames)g (Researc)o(h)g(Cen)o(ter)h(is)g(aero-)149 211 y(nautical)i(researc)o(h)g(and) h(engineering.)29 b(Some)18 b(of)h(the)g(disciplines)f(in)o(v)o(olv)o(ed)f (in)h(mo)q(deling)g(an)149 271 y(aircraft)h(though)h(its)e(\015igh)o(t)h(en)o (v)o(elop)q(e)e(are)i(\015uid)g(dynamics,)e(structural)i(dynamics,)f(thermal) 149 332 y(analysis,)i(propulsion,)g(con)o(trol)f(theory)l(,)g (electromagnetic)e(analysis,)j(and)f(acoustics.)31 b(All)18 b(of)149 392 y(the)c(computational)g(aeroscience)f(\(CAS\))h(grand)h(c)o (hallenges)e(of)i(the)f(HPCC)g(program)g(in)o(v)o(olv)o(e)149 452 y(coupling)j(t)o(w)o(o)f(or)g(more)f(of)i(these)f(disciplines.)223 512 y(In)d(addition,)h(geometrically)d(complex)h(aircraft)i (con\014gurations,)h(often)f(with)g(one)g(or)g(more)149 572 y(comp)q(onen)o(ts)d(in)f(relativ)o(e)f(motion)h(with)h(resp)q(ect)f(to)h (others,)h(require)e(comp)q(osite)f(mesh)h(metho)q(d-)149 633 y(ologies)20 b(for)f(e\016cien)o(t)e(spatial)i(discretization)e([4].)29 b(A)19 b(m)o(ultizonal)d(application)j(meshes)e(indi-)149 693 y(vidual)j(comp)q(onen)o(ts)f(of)h(the)f(con\014guration)i(separately)f(and)g (forms)f(a)h(comp)q(osite)f(mesh)f(b)o(y)149 753 y(o)o(v)o(erlapping)c(comp)q (onen)o(t)f(meshes.)19 b(The)c(o)o(v)o(erlapping)e(meshes)g(in)o(teract)g (through)j(data)f(in)o(ter-)149 813 y(p)q(olation)i(at)g(in)o(ter-mesh)d(b)q (oundary)k(p)q(oin)o(ts)e([8].)223 873 y(Curren)o(tly)l(,)22 b(there)g(are)g(no)h(applications)f(that)h(com)o(bine)d(all)h(of)i(the)f (disciplines.)38 b(Ho)o(w-)149 934 y(ev)o(er,)22 b(sev)o(eral)f(co)q(des)h (exist)f(that)h(com)o(bine)e(at)i(least)f(t)o(w)o(o)h(of)g(these)f (disciplines.)37 b(Some)20 b(ex-)149 994 y(amples)14 b(in)g(the)h (aeronautical)g(domain)f(are)g(\015uid-thermal,)f(\015uid-structures,)i (\015uid-acoustics,)149 1054 y(and)i(\015uid-electromagnetic.)j(A)o(t)15 b(NASA)h(Ames)e(Researc)o(h)i(Cen)o(ter,)f(\015uid-thermal)g([5],)h(\015uid-) 149 1114 y(structures)i([10],)e(and)i(m)o(ultizonal)d(\015uid)i(dynamics)f ([11])h(co)q(des)g(ha)o(v)o(e)g(b)q(een)g(implem)o(en)n(ted)d(on)149 1174 y(a)j(message)f(passing)h(parallel)e(computer,)g(the)h(In)o(tel)e (iPSC/860.)223 1234 y(This)21 b(pap)q(er)i(presen)o(ts)e(the)g(mo)q(del)g(of) h(computing)e(for)i(m)o(ultidiscipl)o(inary)d(and/or)k(m)o(ul-)149 1295 y(tizonal)g(applications)f(on)h(parallel)f(mac)o(hines)e(b)q(eing)j (pursued)g(at)g(NASA)e(Ames)g(Researc)o(h)149 1355 y(Cen)o(ter.)40 b(The)22 b(mo)q(del)f(describ)q(es)i(execution)e(on)i(a)g(parallel)f(mac)o (hine)e(with)i(some)g(n)o(um)o(b)q(er)149 1415 y(of)h(pro)q(cessors,)i(eac)o (h)d(with)g(lo)q(cal)g(memory)l(,)f(connected)h(b)o(y)g(a)h(net)o(w)o(ork.)39 b(It)22 b(do)q(es)h(not)g(dis-)149 1475 y(cuss)c(pro)q(cessor)h(rates,)f (size)e(of)i(memory)l(,)d(net)o(w)o(ork)i(latency)l(,)f(net)o(w)o(ork)h (bandwidth,)i(net)o(w)o(ork)149 1535 y(connectivit)o(y)l(,)e(disk)i(space,)g (I/O)g(bandwidth,)h(or)f(mass)f(storage.)33 b(It)20 b(also)g(do)q(es)g(not)h (discuss)149 1596 y(language)d(and)f(discipline)d(sp)q(eci\014c)h(algorithm)h (issues.)223 1656 y(The)g(pap)q(er)h(has)f(three)g(sections.)21 b(Section)16 b(2)g(describ)q(es)g(general)g(m)o(ultidisci)o(plinary)d(prob-) 149 1716 y(lem)24 b(c)o(haracteristics.)47 b(Section)25 b(3)h(discusses)f(p)q (ossible)h(solution)f(approac)o(hes.)50 b(Section)24 b(4)149 1776 y(presen)o(ts)16 b(three)g(computational)f(mo)q(dels)h(and)g(their)g (requiremen)o(ts.)149 1908 y Fi(2)56 b(Problem)17 b(Characteristics)149 2017 y Fk(A)i(m)o(ultidiscipli)o(nary)e(problem)g(is)i(comp)q(osed)g(of)h(a)g (collection)d(of)j(t)o(w)o(o)f(or)h(more)e(in)o(teracting)149 2078 y(disciplines.)31 b(A)19 b(giv)o(en)g(discipline)f(ma)o(y)h(in)o(teract) f(directly)g(with)i(one)g(or)g(more)f(of)h(the)g(other)149 2138 y(disciplines.)k(Also,)17 b(these)g(in)o(teractions)g(o)q(ccur)h(sim)o (ultaneously)d(across)j(all)f(in)o(terdisciplinary)149 2198 y(b)q(oundaries.)30 b(Similar)16 b(remarks)i(apply)g(to)i(comp)q(onen)o(t)d (meshes)h(in)g(a)h(m)o(ultizonal)d(computa-)149 2258 y(tion.)223 2318 y(A)11 b(m)o(ultidiscipli)o(nary)e(problem)i(is)h(naturally)g(decomp)q (osed)f(in)o(to)h(di\013eren)o(t)g(in)o(teracting)f(dis-)149 2379 y(ciplines,)17 b(eac)o(h)h(of)h(whic)o(h)f(ma)o(y)f(mo)q(del)g(widely)h (di\013eren)o(t)f(ph)o(ysical)h(phenomena.)27 b(Therefore,)149 2439 y(the)d(equations)f(used)h(to)f(describ)q(e)g(the)g(underlying)g(ph)o (ysics)g(\(PDEs/ODEs)i(and/or)g(in)o(te-)149 2499 y(gral)19 b(equations\))f(v)m(ary)g(b)q(et)o(w)o(een)f(disciplines.)25 b(As)18 b(a)h(result,)e(the)h(computational)f(tec)o(hniques)149 2559 y(used)i(for)g(computer)e(sim)o(ulations)f(ma)o(y)h(ha)o(v)o(e)h(widely) f(di\013eren)o(t)h(n)o(umerical)d(algorithm)j(c)o(har-)149 2619 y(acteristics,)c(data)h(structures,)f(and)h(computational)e(requiremen)o (ts.)18 b(In)c(addition,)g(on)h(parallel)149 2680 y(computers)e(with)i(ph)o (ysically)d(distributed)i(memory)l(,)d(the)j(ab)q(o)o(v)o(e)g(di\013erences)g (ma)o(y)e(induce)i(dis-)149 2740 y(tinct)19 b(data)h(partitioning)f (strategies)g(for)h(eac)o(h)e(discipline.)28 b(F)l(or)19 b(example,)e(some)i (disciplines)1036 2864 y(2)p eop %%Page: 3 3 bop 149 42 a Fk(ma)o(y)17 b(b)q(e)h(mo)q(deled)e(using)i(\014nite)f (di\013erence)g(discretization)g(on)h(logically)f(structured)g(meshes)149 102 y(while)i(others)g(ma)o(y)f(use)h(a)g(\014nite)g(elemen)o(t)d (discretization)i(on)h(unstructured)g(meshes.)29 b(Also,)149 162 y(the)23 b(computational)g(requiremen)n(ts)e(within)h(eac)o(h)h (discipline)e(ma)o(y)g(v)m(ary)j(dynamically)c(due)149 222 y(to)g(solution)g(adaptiv)o(e)f(mesh)g(re\014nemen)o(t/coarsening,)f(mo)o (ving)g(b)q(oundaries,)j(and)f(ev)o(olving)149 282 y(material)15 b(non-linearities,)g(etc.)149 415 y Fi(3)56 b(Solution)19 b(Approac)n(hes)149 525 y Fk(Giv)o(en)h(a)h(m)o(ultidiscipl)o(inary)d(and/or)j(m)o(ultizonal)e (problem,)g(what)j(w)o(ould)e(b)q(e)h(the)f(b)q(est)h(ap-)149 585 y(proac)o(h)g(to)f(solving)g(it)g(on)g(a)h(parallel)e(computer?)32 b(On)20 b(a)g(serial)g(computer,)f(the)h(disciplines)149 645 y(and/or)15 b(comp)q(onen)o(t)c(meshes)h(m)o(ust)f(b)q(e)i(pro)q(cessed)g (serially)l(.)19 b(This)13 b(is)g(accomplished)e(b)o(y)h(ha)o(ving)149 706 y(a)21 b(doubly)f(nested)g(lo)q(op.)34 b(The)21 b(outer)f(lo)q(op)h (iterates)f(for)g(the)g(appropriate)h(n)o(um)o(b)q(er)d(of)j(time)149 766 y(steps)c(and)g(the)f(inner)f(lo)q(op)i(iterates)f(o)o(v)o(er)f(the)h (disciplines)f(\(meshes\).)223 826 y(F)l(or)d(a)h(parallel)e(computer,)h(the) g(disciplines)e(\(meshes\))h(ma)o(y)g(b)q(e)h(executed)f(in)h(parallel)g(or)h (se-)149 886 y(quen)o(tially)l(.)19 b(In)14 b(the)g(parallel)f(execution)h (mo)q(de,)f(eac)o(h)h(discipline)e(\(mesh\))h(ma)o(y)g(b)q(e)h(partitioned) 149 946 y(across)19 b(a)f(di\013eren)o(t)f(set)h(of)g(pro)q(cessors)h(and)g (computation)e(pro)q(ceeds)h(in)f(parallel)g(within)h(dis-)149 1007 y(ciplines)i(\(meshes\))g(and)i(across)h(disciplines)c(\(meshes\).)36 b(Both)21 b(data)h(parallelism)e(and)i(task)149 1067 y(parallelism)f(are)h (used.)40 b(In)23 b(the)f(sequen)o(tial)f(execution)g(mo)q(de,)i(eac)o(h)f (discipline)f(\(mesh\))g(is)149 1127 y(partitioned)15 b(across)g(all)f(pro)q (cessors)i(and)f(the)f(computation)f(pro)q(ceeds)i(in)f(parallel)g(within)g (dis-)149 1187 y(ciplines)h(\(meshes\))g(but)h(sequen)o(tially)f(across)i (disciplines)e(\(meshes\).)k(Only)d(data)h(parallelism)149 1247 y(is)f(used.)223 1307 y(The)k(decision)g(whether)g(or)h(not)f(to)h(use)g (task)f(parallelism)e(across)k(disciplines)c(\(meshes\))149 1368 y(is)g(in\015uenced)e(b)o(y)h(sev)o(eral)f(factors:)24 b(memory)15 b(requiremen)o(t)o(s,)g(memory)f(e\016ciency)l(,)h(computa-)149 1428 y(tional)d(requiremen)o(ts,)d(Amdahl's)h(La)o(w,)j(soft)o(w)o(are)f (engineering)f(issues,)i(and)f(m)o(ultidisci)o(plinary)149 1488 y(coupling)17 b(issues.)223 1548 y(In)g(the)g(discussion)g(b)q(elo)o(w)g (when)g(comparing)g(t)o(w)o(o)g(approac)o(hes,)h(a)f(\014xed)g(n)o(um)o(b)q (er)f(of)h(pro-)149 1608 y(cessors)f(and)f(a)g(\014xed)f(problem)f(size)h (are)h(assumed.)20 b(A)14 b(\014xed)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h (is)e(a)h(v)m(alid)149 1669 y(assumption)20 b(b)q(ecause)h(at)f(an)o(y)h (instan)o(t,)f(the)g(goal)h(is)f(to)h(use)f(the)g(a)o(v)m(ailable)f (resources)i(most)149 1729 y(e\016cien)o(tly)l(.)27 b(A)18 b(\014xed)h(problem)e(size)h(is)h(appropriate)g(b)q(ecause)h(once)e(the)h (relev)m(an)o(t)f(ph)o(ysics)g(is)149 1789 y(resolv)o(ed)f(to)g(the)g (desired)g(lev)o(el)e(of)j(accuracy)l(,)e(further)h(re\014nemen)o(t)e(\\w)o (astes")k(computational)149 1849 y(resources)e(\(memory)c(and)k(CPU)f (cycles\).)149 1977 y Fi(3.1)56 b(Memory)17 b(Requirem)o(en)n(ts)149 2069 y Fk(Memory)j(requiremen)o(ts)f(include)i(the)g(size)g(of)h(the)g (executable)e(and)i(the)g(size)f(of)h(the)f(data.)149 2130 y(F)l(or)i(a)f(SIMD)g(mac)o(hine,)f(there)g(is)h(a)g(single)g(cop)o(y)f(of)i (the)f(executable)e(and)j(the)f(disciplines)149 2190 y(are)d(pro)q(cessed)g (sequen)o(tially)e(\(data)i(parallelism)d(only\).)29 b(On)18 b(a)h(MIMD)f(mac)o(hine,)f(there)h(are)149 2250 y(m)o(ultiple)g(copies)i(of)h (the)f(executable.)33 b(If)20 b(the)g(disciplines)f(are)i(pro)q(cessed)g(in)f (parallel)g(\(task)149 2310 y(and)i(data)g(parallelism\),)e(the)h(set)g(of)h (pro)q(cessors)g(asso)q(ciated)g(with)f(a)h(discipline)d(only)i(need)149 2370 y(the)c(executable)f(for)i(that)f(discipline.)23 b(If)16 b(the)h(disciplines)f(are)h(pro)q(cessed)h(sequen)o(tially)d(\(data)149 2431 y(parallelism)j(only\),)j(eac)o(h)e(pro)q(cessor)j(needs)d(a)i(cop)o(y)f (of)g(the)g(executable)f(for)h(all)g(disciplines.)149 2491 y(Dep)q(ending)g(on)f(the)g(amoun)o(t)f(of)h(lo)q(cal)g(memory)l(,)e(whether) h(or)i(not)f(virtual)f(memory)e(is)j(sup-)149 2551 y(p)q(orted,)d(and)f(the)g (parallel)f(I/O)h(bandwidth,)h(this)f(ma)o(y)e(or)i(ma)o(y)f(not)h(b)q(e)g (an)h(issue.)k(Curren)o(tly)l(,)149 2611 y(most)13 b(massiv)o(ely)d(parallel) i(pro)q(cessors)j(should)e(b)q(e)g(though)o(t)h(of)f(as)h(ph)o(ysical)e (memory)e(mac)o(hines)149 2671 y(and)17 b(are)g(considered)e(suc)o(h)h(in)g (this)g(pap)q(er.)1036 2864 y(3)p eop %%Page: 4 4 bop 223 42 a Fk(The)19 b(memory)d(required)i(to)h(store)h(data)g(also)f(v)m (aries.)30 b(On)19 b(a)h(SIMD)f(mac)o(hine,)e(since)h(the)149 102 y(amoun)o(t)d(of)g(data)h(for)g(eac)o(h)e(discipline)g(\(mesh\))f(v)m (ary)l(,)i(some)f(data)i(structures)g(ma)o(y)d(b)q(e)i(padded)149 162 y(dep)q(ending)j(on)g(sp)q(eci\014c)g(mac)o(hine)d(or)j(compiler)d (requiremen)o(ts)g([12].)25 b(Also,)18 b(temp)q(orary)f(v)m(ari-)149 222 y(ables)g(are)g(often)g(the)g(size)f(of)h(the)g(whole)f(mesh.)22 b(On)17 b(a)g(MIMD)f(mac)o(hine,)f(when)i(a)g(domain)f(is)149 282 y(partitioned)c(across)h(m)o(ultiple)c(pro)q(cessors,)14 b(some)d(data)h(are)g(often)h(duplicated)e(to)h(sa)o(v)o(e)g(comm)o(u-)149 342 y(nication)18 b(costs)g([2].)25 b(Also,)18 b(temp)q(orary)e(v)m (ariables,)i(t)o(ypically)e(scalar)i(or)g(v)o(ector)e(temp)q(oraries,)149 403 y(and)k(small)e(global)h(data)h(structures)f(\(i.e.,)f(small)g(lo)q(ok)h (up)h(tables\))f(are)g(replicated)f(to)h(a)o(v)o(oid)149 463 y(b)q(ottlenec)o(ks.)32 b(Bu\013er)20 b(space)g(is)g(also)h(needed)e(on)i (eac)o(h)e(pro)q(cessor)i(to)g(bu\013er)f(messages)g(in)f(a)149 523 y(message)d(passing)h(en)o(vironmen)o(t.)223 583 y(If)f(the)h (disciplines)f(\(meshes\))g(are)h(pro)q(cessed)h(in)e(parallel,)h(eac)o(h)f (discipline)g(\(mesh\))g(has)h(a)149 643 y(b)q(etter)g(surface)f(to)h(v)o (olume)d(ratio)j(for)g(its)f(data)h(implying)d(less)j(duplicated)e(data.)23 b(\(Assuming)149 704 y(a)18 b(\014xed)f(n)o(um)o(b)q(er)f(of)i(pro)q(cessors) g(for)g(the)f(whole)g(application.\))25 b(Also,)17 b(the)g(memory)d(required) 149 764 y(for)23 b(temp)q(orary)f(v)m(ariables)g(dep)q(ends)h(up)q(on)g(the)f (individual)g(discipline.)37 b(If)22 b(the)h(disciplines)149 824 y(\(meshes\))d(are)h(pro)q(cessed)g(sequen)o(tially)l(,)f(the)h(lo)q(cal) g(memory)d(asso)q(ciated)k(with)e(a)i(pro)q(cessor)149 884 y(m)o(ust)17 b(b)q(e)h(shared)g(b)o(y)f(all)g(disciplines)f(\(meshes\).)24 b(This)18 b(implies)d(a)j(higher)f(surface)h(to)g(v)o(olume)149 944 y(ratio)f(for)f(eac)o(h)f(discipline)f(\(mesh\))h(since)g(eac)o(h)h (discipline)e(\(mesh\))g(is)i(spread)g(o)o(v)o(er)g(more)e(pro-)149 1005 y(cessors.)21 b(This)13 b(in)g(turn)g(implies)d(more)i(duplicated)g (data)i(to)f(a)o(v)o(oid)f(comm)o(unication.)17 b(Also,)c(the)149 1065 y(memory)j(requiremen)o(t)g(for)j(temp)q(orary)f(v)m(ariables)h(is)f (dep)q(enden)o(t)h(on)g(the)f(discipline)f(\(mesh\))149 1125 y(with)g(the)f(largest)g(requiremen)o(ts.)149 1252 y Fi(3.2)56 b(Memory)17 b(E\016ciency)149 1344 y Fk(Greater)22 b(parallelism)e(implies)f (more)h(memory)f(is)j(required)f(to)h(solv)o(e)f(the)h(application)f(e\016-) 149 1405 y(cien)o(tly)l(.)f(Let)c(memory)d(e\016ciency)h(b)q(e)j(de\014ned)f (as)932 1531 y Ff(")955 1538 y Fe(M)1008 1531 y Fk(=)1072 1497 y Ff(M)1119 1504 y Fd(0)p 1065 1519 81 2 v 1065 1565 a Ff(M)1112 1572 y Fe(N)1151 1531 y Ff(;)149 1656 y Fk(where)24 b Ff(")321 1663 y Fe(M)383 1656 y Fk(is)g(the)f(memory)d(e\016ciency)l(,)j Ff(M)1002 1663 y Fd(0)1045 1656 y Fk(is)h(amoun)o(t)e(of)i(the)f(memory)e (required)h(to)i(\014t)149 1716 y(the)d(problem)d(on)j(the)f(smallest)f(n)o (um)o(b)q(er)f(of)j(pro)q(cessors)h(\()p Ff(P)1311 1723 y Fd(0)1331 1716 y Fk(\),)f(and)g Ff(M)1531 1723 y Fe(N)1585 1716 y Fk(is)f(the)g(amoun)o (t)g(of)149 1777 y(memory)12 b(used)j(on)g Ff(P)543 1784 y Fe(N)592 1777 y Fk(pro)q(cessors)h(\()p Ff(P)873 1784 y Fd(0)907 1777 y Fc(\024)d Ff(P)990 1784 y Fe(N)1024 1777 y Fk(\).)21 b(Then)15 b(the)f(claim)f(is)h(that)h(memory)d(e\016ciency)149 1837 y(decreases)19 b(as)g(the)g(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h (increase.)28 b(All)18 b(that)h(is)g(necessary)f(for)h(this)g(claim)149 1897 y(to)f(b)q(e)f(true)g(is)f(for)i(a)f(single)g(v)m(ariable/compiler)d (temp)q(orary)j(to)g(b)q(e)g(replicated)f(across)i(t)o(w)o(o)f(or)149 1957 y(more)e(pro)q(cessors.)223 2017 y(As)f(an)h(example)e(of)i(ho)o(w)g (memory)d(requiremen)o(ts)g(can)j(gro)o(w)g(with)g(increased)f(parallelism,) 149 2078 y(consider)h(a)g(32)8 b Fc(\002)g Fk(32)g Fc(\002)g Fk(32)17 b(mesh.)i(\(F)l(airly)14 b(small)f(mesh)h(sizes)g(o)q(ccur)h (frequen)o(tly)e(in)h(m)o(ultizonal)149 2138 y(applications.\))34 b(F)l(orm)19 b(an)i(indep)q(enden)o(t)e(scalar)i(tridiagonal)g(system)e(out)h (of)h(eac)o(h)f(column.)149 2198 y(Then)i(there)f(are)h(32)547 2180 y Fd(2)589 2198 y Fk(indep)q(enden)o(t)f(tridiagonal)h(systems,)f(eac)o (h)g(of)h(length)f(32.)38 b(Compare)149 2258 y(the)20 b(memory)c(requiremen)o (ts)h(to)j(solv)o(e)f(the)g(systems)f(on)i(a)g(serial)f(mac)o(hine,)f(a)i (single)f(v)o(ector)149 2318 y(pro)q(cessor,)j(and)e(a)h(parallel)e(v)o (ector)g(mac)o(hine.)30 b(The)20 b(solution)g(tec)o(hnique)f(will)f(b)q(e)j (Gaussian)149 2379 y(elimination)14 b(without)j(piv)o(oting)e(where)h(the)g (input)g(ma)o(y)f(b)q(e)h(o)o(v)o(erwritten.)223 2439 y(On)g(a)h(serial)e (mac)o(hine,)f(32)725 2421 y Fd(3)757 2439 y Fk(+)d(3)g Fc(\002)g Fk(32)17 b(w)o(ords)g(are)f(required,)f(32)1430 2421 y Fd(3)1467 2439 y Fk(for)h(the)g(righ)o(t)g(hand)h(side)149 2499 y(and)i(3)13 b Fc(\002)f Fk(32)19 b(for)g(a)g(single)e(tridiagonal)i(system.)26 b(The)18 b(systems)f(are)h(formed)f(and)i(solv)o(ed)f(one)149 2559 y(at)f(a)g(time.)223 2619 y(On)g(a)i(single)e(v)o(ector)g(pro)q(cessor,) i(32)910 2601 y Fd(3)942 2619 y Fk(+)13 b(3)f Fc(\002)g Fk(32)1128 2601 y Fd(2)1166 2619 y Fk(w)o(ords)19 b(are)f(required,)e(32)1645 2601 y Fd(3)1684 2619 y Fk(for)i(the)f(righ)o(t)149 2680 y(hand)f(side)f(and) g(3)9 b Fc(\002)f Fk(32)588 2661 y Fd(2)623 2680 y Fk(to)16 b(form)e(32)h(indep)q(enden)o(t)f(tridiagonal)i(systems.)j(A)o(t)14 b(least)h(32)h(of)f(the)149 2740 y(systems)g(are)i(formed)e(at)h(a)h(time)d (to)j(allo)o(w)f(for)g(v)o(ectorization)f(across)i(indep)q(enden)o(t)f (systems.)1036 2864 y(4)p eop %%Page: 5 5 bop 223 42 a Fk(On)13 b(a)h(32)h(no)q(de)f(parallel)f(v)o(ector)g(mac)o (hine,)e(4)6 b Fc(\002)g Fk(32)1157 23 y Fd(3)1191 42 y Fk(w)o(ords)14 b(are)g(required,)f(32)1658 23 y Fd(3)1692 42 y Fk(for)h(the)f(righ)o(t)149 102 y(hand)20 b(side)f(and)h(3)13 b Fc(\002)g Fk(32)610 84 y Fd(3)650 102 y Fk(to)19 b(form)f(32)878 84 y Fd(2)918 102 y Fk(indep)q(enden)o(t)g(tridiagonal)i(systems.)28 b(Thirt)o(y)19 b(t)o(w)o(o)g(of)149 162 y(the)d(systems)f(are)i(formed)e(at)h(a)h(time)d(on) j(eac)o(h)f(pro)q(cessor)h(to)f(allo)o(w)g(for)h(v)o(ectorization.)223 222 y(If)c(the)h(serial)g(mac)o(hine)e(has)j(a)f(memory)d(e\016ciency)h(of)j (1)p Ff(:)p Fk(0,)f(then)g(the)g(single)g(v)o(ector)f(pro)q(ces-)149 282 y(sor)h(and)f(parallel)f(v)o(ector)g(mac)o(hine)e(ha)o(v)o(e)i(memory)e (e\016ciencies)g(of)k(0)p Ff(:)p Fk(92)f(and)g(0)p Ff(:)p Fk(25)h(resp)q (ectiv)o(ely)l(.)149 410 y Fi(3.3)56 b(Computational)18 b(Requirem)o(en)n(ts) 149 502 y Fk(Computational)d(requiremen)o(ts)d(for)j(di\013eren)o(t)e (disciplines)g(and)j(di\013eren)o(t)e(meshes)f(v)m(ary)l(.)21 b(Also,)149 563 y(mesh)16 b(sizes)g(ma)o(y)g(v)m(ary)h(widely)f(in)g(m)o (ultizonal)f(computations)h(yielding)g(v)o(ery)g(di\013eren)o(t)g(com-)149 623 y(putational)f(requiremen)n(ts)c(for)j(di\013eren)o(t)f(meshes.)19 b(This)14 b(leads)f(to)h(load)g(balancing)g(issues.)21 b(F)l(or)149 683 y(example,)14 b(computational)h(requiremen)o(ts)e(in)i(the)h(con)o(trols) g(discipline)e(is)i(m)o(uc)o(h)d(smaller)h(than)149 743 y(the)i (computational)g(requiremen)o(ts)d(to)k(compute)e(a)h(\015uid)g(\015o)o(w)h (\014eld.)223 803 y(If)g(disciplines)f(\(meshes\))h(are)g(pro)q(cessed)i(in)e (parallel)g(\(task)h(and)h(data)g(parallelism\))c(then)149 864 y(the)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h(assigned)f(to)g(eac)o(h)g (discipline)d(\(mesh\))i(can)h(b)q(e)g(adjusted)g(to)g(com-)149 924 y(putationally)g(load)h(balance)f(the)g(application.)21 b(\(Sub)s(ject)16 b(to)g(memory)e(constrain)o(ts.\))223 984 y(If)i(disciplines)f(\(meshes\))g(are)i(pro)q(cessed)g(sequen)o(tially)d (\(data)k(parallelism)c(only\),)i(for)h(dis-)149 1044 y(ciplines)d (\(meshes\))h(with)g(small)f(memory)f(requiremen)n(ts)g(and)j(small)e (computational)h(require-)149 1104 y(men)o(ts,)g(trying)h(to)h(use)g(all)f(a) o(v)m(ailable)g(pro)q(cessors)i(ma)o(y)d(actually)h(slo)o(w)h(do)o(wn)g(the)f (application)149 1165 y(due)h(to)g(the)g(decrease)f(in)g(the)h(computation)f (to)h(comm)o(unicati)o(on)e(ratio.)23 b(If)16 b(the)h(computation)149 1225 y(asso)q(ciated)d(with)f(a)g(discipline)e(\(mesh\))g(is)h(unable)h(to)g (use)g(all)f(a)o(v)m(ailable)g(pro)q(cessors,)i(then)f(some)149 1285 y(pro)q(cessors)18 b(will)d(b)q(e)h(idle)g(during)g(that)h(phase)f(of)h (the)f(computation.)223 1345 y(Ho)q(c)o(kney)h(and)i(Jesshop)q(e)g([7])f (discuss)g(trade)h(o\013s)g(in)f(algorithm)f(selection.)26 b(Whic)o(h)18 b(algo-)149 1405 y(rithm)c(is)i(\\b)q(est")h(dep)q(ends)f(up)q (on)g(the)g(mac)o(hine)d(arc)o(hitecture,)h(actual)i(n)o(um)o(b)q(er)e(of)i (pro)q(cessors)149 1466 y(b)q(eing)e(used,)f(and)h(the)f(problem)e(size.)19 b(An)13 b(algorithm)f(with)h(more)f(inheren)o(t)g(parallelism)f(migh)o(t)149 1526 y(not)20 b(execute)d(as)i(fast)g(as)h(an)f(algorithm)f(with)g(less)h (inheren)o(t)e(parallelism)f(due)j(to)g(an)g(o)o(v)o(erall)149 1586 y(increase)d(in)g(op)q(erations.)149 1714 y Fi(3.4)56 b(Amdahl's)18 b(La)n(w)149 1806 y Fk(E\016ciency)c(also)i(v)m(aries)f(dep)q (ending)h(on)f(whether)g(the)h(disciplines)d(\(meshes\))h(are)h(pro)q(cessed) h(in)149 1866 y(parallel)g(or)g(sequen)o(tially)l(.)j(Amdahl's)14 b(La)o(w)j(sa)o(ys)f(that)h(sp)q(eedup)f(is)g(b)q(ounded)h(b)o(y)e(the)h (fraction)149 1926 y(of)h(the)f(co)q(de)g(that)h(is)f(serial:)847 1998 y Ff(S)h Fc(\024)1069 1965 y Ff(N)p 951 1987 280 2 v 951 2033 a Fk(\()p Ff(N)g Fc(\000)11 b Fk(1\))p Ff(\027)j Fk(+)d(1)1236 1998 y Ff(;)149 2111 y Fk(where)18 b Ff(S)i Fk(is)e(the)f(sp)q(eedup,)h Ff(N)23 b Fk(is)17 b(the)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors,)h(and)f Ff(\027)j Fk(is)c(the)h(fraction)f(of)h(the)149 2171 y(co)q(de)f(that)g(is)f (serial.)k(E\016ciency)l(,)14 b Ff(")p Fk(,)i(is)g(de\014ned)g(as)793 2302 y Ff(")d Fk(=)892 2269 y Ff(S)p 886 2291 45 2 v 886 2336 a(N)949 2302 y Fk(=)1133 2269 y(1)p 1006 2291 280 2 v 1006 2336 a(\()p Ff(N)j Fc(\000)11 b Fk(1\))p Ff(\027)k Fk(+)c(1)1290 2302 y Ff(:)149 2437 y Fk(Amdahl's)h(La)o(w)i(is)f(relev)m(an)o(t)f(since)h (these)g(discussions)g(compare)g(and)h(con)o(trast)f(the)g(pro)q(cessing)149 2498 y(of)k(a)g(\014xed)f(n)o(um)o(b)q(er)e(of)j(m)o(ultiple)c(disciplines)h (\(meshes\).)223 2558 y(If)k(the)h(disciplines)f(\(meshes\))f(are)i(pro)q (cessed)h(in)f(parallel,)f(assume)h(that)g(the)g(n)o(um)o(b)q(er)f(of)149 2618 y(pro)q(cessors)j(assigned)e(to)h(eac)o(h)e(discipline)f(\(mesh\),)h Ff(G)1190 2625 y Fe(i)1204 2618 y Fk(,)i(is)e(suc)o(h)h(that)g(there)g(is)f (near)i(p)q(erfect)149 2678 y(load)c(balance)f(across)h(disciplines)d (\(meshes\).)19 b(Giv)o(en)14 b(the)h(same)f(n)o(um)o(b)q(er)f(of)j(pro)q (cessors)g(\()p Ff(N)j Fk(=)149 2705 y Fb(P)202 2738 y Ff(G)240 2745 y Fe(i)254 2738 y Fk(\),)14 b(pro)q(cessing)g(disciplines)d(\(meshes\))h (sequen)o(tially)f(m)o(ust)h(ha)o(v)o(e)h(a)h(lo)o(w)o(er)e(e\016ciency)l(.) 18 b(Eac)o(h)1036 2864 y(5)p eop %%Page: 6 6 bop 149 42 a Fk(discipline)13 b(\(mesh\))g(is)h(spread)h(o)o(v)o(er)f(more)f (pro)q(cessors)j(and)f(Amdahl's)d(La)o(w)j(implies)d(e\016ciency)149 102 y(decreases)k(as)h(the)f(n)o(um)o(b)q(er)f(of)h(pro)q(cessors)i (increases)e(for)g(a)h(\014xed)f(problem)e(size.)223 162 y(F)l(or)g(example,) e(assume)i(that)h(a)g(m)o(ultizonal)d(application)i(has)h(t)o(w)o(o)g(iden)o (tical)e(sized)g(meshes)149 222 y(with)g(the)g(same)f(computational)g(load)i (for)f(eac)o(h.)20 b(F)l(urther,)13 b(assume)f(that)h(the)g(computation)f(on) 149 282 y(eac)o(h)k(is)f(99.9\045)h(parallel.)k(Giv)o(en)15 b(1000)i(pro)q(cessors,)g(if)e(the)g(meshes)g(are)g(pro)q(cessed)h(in)g (parallel)149 342 y(\(500)g(pro)q(cessors)f(assigned)g(to)f(eac)o(h)f(mesh\)) g(then)h(the)g(e\016ciency)d(is)j(66)p Ff(:)p Fk(7\045.)21 b(If)13 b(the)h(meshes)f(are)149 403 y(pro)q(cessed)k(sequen)o(tially)e (\(1000)j(pro)q(cessors)f(assigned)h(to)e(eac)o(h)g(mesh\))f(then)i(the)f (e\016ciency)e(is)149 463 y(50)p Ff(:)p Fk(0\045.)21 b(This)14 b(means)e(that)i(pro)q(cessing)h(the)e(meshes)f(sequen)o(tially)f(will)i(tak) o(e)g(33\045)h(longer)f(than)149 523 y(pro)q(cessing)k(them)e(in)h(parallel.) 223 583 y(Other)c(sources)h(of)g(ine\016ciency)d(are)i(an)h(increased)f(comm) o(unication)e(to)j(computation)f(ratio)149 643 y(and)24 b(an)g(increase)f(in) g(redundan)o(t)h(computations)f(as)h(the)f(p)q(ercen)o(tage)g(of)g (duplicated)g(data)149 704 y(increases.)149 831 y Fi(3.5)56 b(Soft)n(w)n(are)20 b(Engineering)149 924 y Fk(Some)d(of)i(the)e(soft)o(w)o (are)i(engineering)e(issues)h(are)g(mo)q(dularit)o(y)l(,)e(indep)q(endence,)h (extensibilit)o(y)l(,)149 984 y(mo)q(di\014abilit)o(y)l(,)d(readabilit)o(y)l (,)h(and)i(main)o(tainabilit)o(y)l(.)i(Soft)o(w)o(are)e(engineering)e(ma)o(y) g(b)q(e)i(though)o(t)149 1044 y(of)h(as)f(complexit)o(y)d(managemen)o(t.)20 b(Tw)o(o)e(primary)d(w)o(a)o(ys)i(h)o(umans)f(deal)h(with)f(complexit)o(y)e (are)149 1104 y(divide-and-conquer)22 b(and)h(abstraction.)41 b(The)23 b(trend)f(o)o(v)o(er)g(the)g(last)g(thirt)o(y)g(y)o(ears)g(in)g (soft-)149 1165 y(w)o(are)17 b(engineering)f(has)h(b)q(een)g(to)o(w)o(ards)g (greater)g(co)q(de)g(mo)q(dularit)o(y)e(and)i(greater)g(information)149 1225 y(hiding.)24 b(Structured)16 b(programming,)g(top-do)o(wn)i(design,)f (program)g(mo)q(dules,)e(abstract)j(data)149 1285 y(structures,)c(ob)s(ject)g (orien)o(ted)f(programming)f(are)i(all)g(examples)e(of)i(this)g(trend.)20 b(They)14 b(encour-)149 1345 y(age)i(the)e(partitioning)h(of)g(problems)f(in) o(to)g(simpler)f(pieces,)g(in)o(terfaces)h(to)h(b)q(e)g(de\014ned,)g(and)g (the)149 1405 y(impleme)o(n)o(tation)f(details)h(to)i(b)q(e)f(hidden)g(b)q (ehind)g(the)g(in)o(terfaces.)149 1533 y Fi(3.6)56 b(Multidisciplinary)16 b(Coupling)149 1626 y Fk(The)d(principal)e(metho)q(ds)h(for)g(coupling)h(m)o (ultiple)c(disciplines)h(are)j(global)f(explicit)f(in)o(tegration,)149 1686 y(global)18 b(implicit)c(in)o(tegration,)j(mo)q(dal)f(equation)i(in)o (tegration,)e(and)i(partitioned)f(analysis)h([6].)149 1746 y(Global)c(explicit)d(in)o(tegration)h(applies)h(an)h(explicit)d(time)f(in)o (tegration)j(algorithm)f(to)i(the)f(global)149 1806 y(set)19 b(of)g(equations.)29 b(Ho)o(w)o(ev)o(er,)17 b(stabilit)o(y)g(is)i(a)g (concern)f(and)h(the)g(time)d(step)j(m)o(ust)e(b)q(e)i(c)o(hosen)149 1866 y(based)e(on)g(the)f(discipline)e(with)i(the)g(sti\013est)h(requiremen)n (ts.)223 1926 y(Mo)q(dal)12 b(equation)f(in)o(tegration)h(represen)o(ts)f (the)g(resp)q(onse)i(of)f(a)g(system)e(b)o(y)h(mo)q(des)h(go)o(v)o(erned)149 1987 y(b)o(y)g(mo)q(dal)g(equations)h(of)f(motion.)19 b(Ho)o(w)o(ev)o(er,)11 b(it)h(is)g(hard)h(to)g(determining)d(a)j(set)f(of)h(basis)f(mo)q(des)149 2047 y(for)17 b(a)g(nonlinear)f(system.)223 2107 y(Global)j(implicit)d (coupling)i(sc)o(heme)f(is)i(used,)g(the)g(problem)f(can)h(b)q(e)g(expressed) g(in)f(blo)q(c)o(k)149 2167 y(matrix)d(form)g(where)h(the)f(main)g(diagonal)i (blo)q(c)o(ks)f(represen)o(t)f(the)h(individual)f(disciplines)f(and)149 2227 y(the)e(o\013-diagonal)i(blo)q(c)o(ks)d(represen)o(t)g(the)g(coupling)h (b)q(et)o(w)o(een)f(the)g(disciplines.)19 b(Then)11 b(the)h(whole)149 2288 y(system)18 b(can)g(b)q(e)h(solv)o(ed)f(as)i(a)f(large)f(single)g (matrix.)27 b(Ho)o(w)o(ev)o(er,)17 b(in)h(man)o(y)g(instances,)g(it)g(ma)o(y) 149 2348 y(not)c(b)q(e)g(p)q(ossible)g(to)g(directly)d(couple)i(t)o(w)o(o)h (disciplines.)k(This)c(is)f(due)h(to)f(di\016culties)f(asso)q(ciated)149 2408 y(with)k(the)g(explicit)e(ev)m(aluation)i(of)g(o\013-diagonal)i(blo)q(c) o(ks.)j(F)l(or)16 b(example,)e(it)h(w)o(ould)h(b)q(e)g(di\016cult)149 2468 y(to)j(directly)e(couple)h(a)h(particle)e(sim)o(ulation)g(of)i(a)f (rare\014ed)h(gas)g(to)g(the)f(structural)h(and)g(ther-)149 2528 y(mal)d(analysis)h(of)g(the)g(b)q(o)q(dy)h(o)o(v)o(er)e(whic)o(h)g(it)g (\015o)o(ws.)24 b(Ev)o(en)16 b(if)g(all)h(o\013-diagonal)i(coupling)d(terms) 149 2589 y(can)k(b)q(e)f(ev)m(aluated)g(explicitly)l(,)d(direct)i(in)o(v)o (ersion)f(of)j(suc)o(h)e(a)i(large)f(sparse)g(matrix)f(w)o(ould)h(b)q(e)149 2649 y(prohibitiv)o(ely)14 b(exp)q(ensiv)o(e)h(in)h(terms)e(of)j(b)q(oth)g (memory)c(and)j(CPU)h(time)c(for)k(problems)e(of)h(an)o(y)149 2709 y(consequence)i(to)h(the)f(aeronautics)h(comm)o(unit)o(y)-5 b(.)26 b(As)18 b(a)h(result,)f(it)g(is)h(necessary)f(to)h(resort)g(to)1036 2864 y(6)p eop %%Page: 7 7 bop 149 42 a Fk(some)17 b(form)f(of)h(iterativ)o(e)e(solution)j(approac)o(h)f (based)h(on)g(the)f(concept)f(of)i(sub-structuring)g(or)149 102 y(partitioned)g(analysis.)26 b(A)17 b(natural)h(form)f(of)h (sub-structuring)g(across)h(discipline)d(b)q(oundaries)149 162 y(w)o(ould)24 b(b)q(e)f(inevitable)f(due)h(to)g(widely)f(di\013eren)o(t)h (n)o(umerical)d(c)o(haracteristics)i(across)i(disci-)149 222 y(plines.)c(A)14 b(monolithic)e(iterativ)o(e)h(solv)o(er)g(is)i(unlik)o(ely)d (to)i(b)q(e)h(able)f(to)h(cop)q(e)f(with)g(all)g(the)g(div)o(erse)149 282 y(disciplines)h(in)h(an)h(e\016cien)o(t)d(manner.)223 342 y(In)h(partitioned)f(analysis,)i(the)f(e\013ect)f(of)i(o\013-diagonal)h(blo)q (c)o(ks)e(are)g(represen)o(ted)f(as)i(forcing)149 403 y(functions)23 b(on)g(the)g(righ)o(t)f(hand)i(side.)40 b(The)23 b(computation)f(of)h(these)f (forcing)h(terms)e(w)o(ould)149 463 y(induce)13 b(in)o(ter-disciplinary)e (comm)o(unication.)18 b(P)o(artitioned)13 b(analysis)g(leads)h(to)g(blo)q(c)o (k)f(iterativ)o(e)149 523 y(metho)q(ds)k(\(blo)q(c)o(k)g(Jacobi,)g(blo)q(c)o (k)g(Gauss-Seidel,)g(etc.\).)23 b(Also,)17 b(with)g(partitioned)g(analysis,)g (it)149 583 y(is)i(easier)g(to)g(extend)f(the)h(application)f(to)h(include)f (new)h(disciplines.)27 b(Ho)o(w)o(ev)o(er,)18 b(robust)h(and)149 643 y(e\016cien)o(t)c(m)o(ultidiscipl)o(inary)f(and/or)k(m)o(ultizonal)c (coupling)i(tec)o(hniques)g(is)g(an)h(activ)o(e)e(area)j(of)149 704 y(researc)o(h.)223 764 y(P)o(ark)j(and)h(F)l(elippa)e([9])h(list)g(man)o (y)e(of)j(the)f(problems)f(asso)q(ciated)i(with)g(global)f(implicit)149 824 y(coupling)16 b(and)h(the)f(adv)m(an)o(tages)i(of)e(partitioned)g (analysis.)21 b(F)l(rom)15 b(a)i(soft)o(w)o(are)f(viewp)q(oin)o(t,)f(the)149 884 y(list)i(of)h(disadv)m(an)o(tages)g(of)g(direct)f(coupling)g(and)h(adv)m (an)o(tages)h(of)e(partitioned)g(analysis)h(could)149 944 y(b)q(e)f(said)f (ab)q(out)i(monolithic)c(programming)h(v)o(ersus)h(mo)q(dular)f(programming)g (resp)q(ectiv)o(ely)l(.)149 1078 y Fi(4)56 b(Computational)18 b(Mo)r(dels)149 1187 y Fk(Giv)o(en)d(the)g(ab)q(o)o(v)o(e)h(considerations)g (\(memory)c(requiremen)o(ts,)h(computational)h(requiremen)o(ts,)149 1247 y(Amdahl's)19 b(La)o(w,)j(soft)o(w)o(are)f(engineering,)f(and)i(m)o (ultidisci)o(pli)o(nary)c(and/or)k(m)o(ultizonal)c(cou-)149 1307 y(pling\))d(the)f(most)g(\\natural")j(approac)o(h)e(to)g(implem)o(en)o (ti)o(ng)e(a)i(m)o(ultidisci)o(plinary)d(and/or)k(m)o(ul-)149 1368 y(tizonal)i(application)g(on)h(parallel)e(pro)q(cessor)i(is)f (partitioned)g(analysis)g(using)g(a)h(MIMD)e(st)o(yle)149 1428 y(of)i(execution)d(across)j(disciplines)e(\(meshes\).)24 b(This)18 b(allo)o(ws)g(n)o(umerical)e(algorithms)h(and)h(pro-)149 1488 y(grams)e(p)q(ertaining)g(to)g(individual)f(disciplines)f(to)j(b)q(e)f(dev)o (elop)q(ed)f(and)h(tested)g(indep)q(enden)o(tly)l(.)149 1548 y(After)h(eac)o(h)g(program)h(has)g(b)q(een)g(v)m(alidated,)f(an)h (additional)g(mo)q(dule)e(is)h(added)h(to)g(eac)o(h)f(that)149 1608 y(deals)g(with)f(the)g(required)f(in)o(terdisciplinary)e(\(in)o (terzonal\))j(comm)o(uni)o(cation.)223 1669 y(Sev)o(eral)g(mo)q(dels)h(of)i (computation)e(supp)q(ort)i(the)f(MIMD)f(st)o(yle)g(of)h(execution)f(across)i (dis-)149 1729 y(ciplines.)24 b(Whic)o(h)17 b(one)g(is)h(b)q(est)g(dep)q (ends)f(up)q(on)i(the)e(ph)o(ysical)g(problem,)f(solution)h(algorithm,)149 1789 y(hardw)o(are)f(resources,)g(and)g(op)q(erating)g(system)e(supp)q(ort)j (a)o(v)m(ailable.)j(Belo)o(w)15 b(w)o(e)g(presen)o(t)f(three)149 1849 y(mo)q(dels)i(and)g(describ)q(e)g(the)g(functionalit)o(y)f(that)i(is)f (desired)f(from)g(eac)o(h.)223 1909 y(The)h(discussions)g(assume)g(the)g (follo)o(wing)f(de\014nitions:)21 b Ff(N)5 b(P)24 b Fk(equals)16 b(the)g(total)g(n)o(um)o(b)q(er)f(of)149 1970 y(pro)q(cessors)g(used,)f(a)g (group)h Ff(G)714 1977 y Fe(i)742 1970 y Fk(is)f(the)g(set)f(of)h(pro)q (cessors)h(assigned)g(to)f(a)g(particular)f(discipline,)149 2030 y Ff(N)5 b(G)17 b Fk(is)e(the)h(total)g(n)o(um)o(b)q(er)e(of)i(groups,)g (and)h Ff(GS)1061 2037 y Fe(i)1091 2030 y Fk(is)e(the)h(n)o(um)o(b)q(er)e(of) i(pro)q(cessors)h(in)e(group)i Ff(G)1919 2037 y Fe(i)1933 2030 y Fk(.)223 2090 y(In)c(all)f(of)i(the)f(mo)q(dels,)f(it)h(is)g(assumed)f (that)i(the)f(user)g(has)h(the)f(option)h(to)g(con)o(trol)e(the)h(set)h(of) 149 2150 y(ph)o(ysical)g(pro)q(cessors)h(assigned)g(to)g(a)g(group)g(and)g (the)f(data)i(la)o(y)o(out)d(on)i(that)g(set)f(of)h(pro)q(cessors.)149 2278 y Fi(4.1)56 b(Static)18 b(Mo)r(del)149 2370 y Fk(In)g(the)g(static)g(mo) q(del,)f(all)h(resource)g(requiremen)o(ts)d(are)j(kno)o(wn)h(at)f(startup)h (\(i.e.,)e Ff(N)5 b(P)i Fk(,)19 b Ff(N)5 b(G)p Fk(,)149 2431 y(and)18 b Ff(GS)313 2438 y Fe(i)345 2431 y Fk(are)f(all)f(kno)o(wn\).)24 b(F)l(unctional)16 b(requiremen)o(ts)e(are)j(that)g(the)g(n)o(um)o(b)q(er)e (of)j(pro)q(cessors)149 2491 y(assigned)23 b(to)e(eac)o(h)g(discipline)f(is)h (indep)q(enden)o(t)f(of)i(the)f(other)h(disciplines,)e(a)i(di\013eren)o(t)f (exe-)149 2551 y(cutable)e(ma)o(y)f(b)q(e)i(loaded)g(in)o(to)f(eac)o(h)g (group,)i(eac)o(h)e(group)i(has)f(its)f(o)o(wn)h(logical)f(n)o(um)o(b)q (ering,)149 2611 y(global)f(op)q(erations)h(suc)o(h)e(as)h(SUM)f(or)g(MAX)g (apply)g(to)h(the)f(lo)q(cal)g(group,)h(and)g(some)e(mec)o(ha-)149 2671 y(nism)f(is)i(pro)o(vided)e(to)i(establish)f(comm)o(unication)e(b)q(et)o (w)o(een)h(an)i(individual)e(pro)q(cessor)j(in)e(one)149 2731 y(group)i(and)e(an)h(individual)e(pro)q(cessor)i(in)f(another)h(group.)1036 2864 y(7)p eop %%Page: 8 8 bop 223 42 a Fk(The)12 b(logical)g(n)o(um)o(b)q(ering)f(within)h(a)h(group,)h (global)e(op)q(erations)i(applying)f(only)f(to)h(the)f(lo)q(cal)149 102 y(group,)24 b(and)f(di\013eren)o(t)e(executables)g(all)g(supp)q(ort)i (indep)q(enden)o(t)f(dev)o(elopmen)n(t)d(and)k(testing)149 162 y(of)g(disciplines.)36 b(The)22 b(indep)q(enden)o(t)f(assignmen)o(t)g(of) h(pro)q(cessors)h(to)f(disciplines)e(allo)o(ws)i(the)149 222 y(application)f(to)g(b)q(e)f(statically)g(load)h(balanced.)34 b(P)o(oin)o(t)20 b(to)h(p)q(oin)o(t)f(comm)o(unication)e(b)q(et)o(w)o(een)149 282 y(groups)g(allo)o(w)e(coupling)h(information)e(to)i(b)q(e)f(passed)h (directly)e(b)q(et)o(w)o(een)h(the)g(pro)q(cessors)h(that)149 342 y(need)f(to)h(kno)o(w)f(the)g(information.)223 403 y(Groups)j(are)f (neither)f(created)h(nor)h(deleted)e(dynamically)l(.)24 b(Ev)o(erything)18 b(is)g(initialized)e(at)149 463 y(startup)25 b(and)f(con)o(tin)o(ues)f(un)o (til)g(application)g(termination.)42 b(Curren)o(tly)l(,)24 b(there)f(are)h(sev)o(eral)149 523 y(applications)19 b([11)q(,)f(10)q(,)h(3]) f(using)i(this)f(mo)q(del)e(at)j(NASA)d(Ames)g(Researc)o(h)i(Cen)o(ter)f (based)h(on)149 583 y(the)d(in)o(tercub)q(e)g(comm)o(uni)o(cation)e(soft)o(w) o(are)i(dev)o(elop)q(ed)f(for)i(the)f(In)o(tel)e(iPSC/860)k([1].)223 643 y(There)11 b(are)h(sev)o(eral)f(problems)f(with)i(the)g(iPSC/860)h (implem)o(en)o(tation)c(of)j(the)g(static)g(mo)q(del.)149 704 y(Debuggers)21 b(do)f(not)g(w)o(ork)f(for)h(m)o(ultiple)d(indep)q(enden)o(t)h (groups)j(of)f(pro)q(cessors.)32 b(Groups)21 b(are)149 764 y(restricted)11 b(to)i(a)f(p)q(o)o(w)o(er-of-t)o(w)o(o)h(n)o(um)o(b)q(er)e (of)h(pro)q(cessors)h(and)g(only)f(one)g(group)h(ma)o(y)e(b)q(e)h(assigned) 149 824 y(to)21 b(a)f(pro)q(cessor)i(thereb)o(y)d(limiti)o(ng)f(the)i(abilit) o(y)f(to)h(load)h(balance)f(the)f(computation.)33 b(Also,)149 884 y(existing)24 b(batc)o(h)g(job)g(submission)g(soft)o(w)o(are,)h(Net)o(w)o (ork)e(Queuing)h(System)f(\(NQS\),)g(do)h(not)149 944 y(supp)q(ort)18 b(this)e(st)o(yle)f(of)i(computation.)149 1072 y Fi(4.2)56 b(Dynamic)17 b(Mo)r(del)149 1165 y Fk(In)g(the)f(dynamic)f(mo)q(del,)g (groups)i(ma)o(y)e(b)q(e)i(created)f(or)h(deleted)e(dynamically)l(.)20 b Ff(N)5 b(P)24 b Fk(and)17 b Ff(N)5 b(G)149 1225 y Fk(v)m(ary)19 b(dynamically)l(.)24 b Ff(GS)622 1232 y Fe(i)655 1225 y Fk(is)18 b(\014xed)f(at)i(group)g(creation.)26 b(F)l(unctional)18 b(requiremen)o(ts)d (are)j(that)149 1285 y(the)c(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h(assigned) f(to)h(eac)o(h)e(discipline)f(is)h(indep)q(enden)o(t)g(of)h(the)g(other)f (disci-)149 1345 y(plines,)i(a)i(di\013eren)o(t)e(executable)g(ma)o(y)f(b)q (e)i(loaded)h(in)o(to)e(eac)o(h)h(group,)g(eac)o(h)g(group)h(has)g(its)f(o)o (wn)149 1405 y(logical)g(n)o(um)o(b)q(ering,)e(global)i(op)q(erations)h (apply)f(to)h(the)e(lo)q(cal)h(group,)h(mec)o(hanism)o(s)d(for)i(group)149 1466 y(creation)j(and)f(deletion)g(are)g(needed,)g(and)h(some)e(mec)o(hanism) e(to)k(establish)f(comm)o(unic)o(ation)149 1526 y(b)q(et)o(w)o(een)d(an)h (individual)e(pro)q(cessor)j(in)e(one)g(group)i(and)f(an)g(individual)e(pro)q (cessor)i(in)f(another)149 1586 y(group.)223 1646 y(The)k(reasoning)i(for)f (the)g(functionalit)o(y)e(is)i(the)f(same)g(as)h(in)g(the)f(static)h(mo)q (del)e(with)i(the)149 1706 y(additional)f(requiremen)n(ts)d(of)i(dynamic)f (creation)h(and)g(deletion)g(of)g(groups.)28 b(The)19 b(size)e(of)h(an)149 1766 y(individual)23 b(group)h(is)f(\014xed)g(at)h(group)g(creation.)43 b(If)22 b(a)i(group)h(needs)e(increase)f(its)i(size,)f(a)149 1827 y(new)18 b(group)h(ma)o(y)d(b)q(e)i(created)g(and)g(the)g(relev)m(an)o (t)f(information)g(copied)g(from)g(the)g(old)h(group.)149 1887 y(Then)f(the)g(old)g(group)g(name)f(is)h(reassigned)g(to)g(the)g(new)f(group) i(and)f(the)g(old)g(group)h(deleted.)149 1947 y(The)i(create,)g(cop)o(y)f (and)h(delete)e(approac)o(h)j(to)f(dynamic)e(group)i(size)f(is)h(acceptable)f (b)q(ecause)149 2007 y(disciplines)d(often)h(assume)f(a)h(particular)f(data)i (to)f(pro)q(cessor)h(la)o(y)o(out.)23 b(Adding)16 b(pro)q(cessors)j(to)149 2067 y(the)f(curren)o(t)e(group)j(w)o(ould)e(also)i(require)d(a)i(data)g (reorganization.)26 b(Mo)o(ving)17 b(to)h(a)g(new)f(set)h(of)149 2128 y(pro)q(cessors)k(ma)o(y)c(ease)i(the)g(pro)q(cess.)33 b(This)20 b(form)f(of)h(pro)q(cess)h(migration)e(requires)g(that)h(the)149 2188 y(c)o(hange)d(in)f(lo)q(cation)g(of)h(the)f(group)h(b)q(e)f(transmitted) f(to)i(all)f(groups)h(that)g(need)e(to)i(kno)o(w.)149 2316 y Fi(4.3)56 b(Hybrid)18 b(Mo)r(del)149 2408 y Fk(In)e(the)g(h)o(ybrid)g(mo)q (del,)e(the)i(total)h(n)o(um)o(b)q(er)d(of)j(pro)q(cessors)g(used)g(b)o(y)e (the)h(application,)g Ff(N)5 b(P)i Fk(,)16 b(is)149 2468 y(\014xed)k(at)f (application)h(startup.)31 b(Ho)o(w)o(ev)o(er,)18 b(the)h(n)o(um)o(b)q(er)f (of)i(groups,)h Ff(N)5 b(G)p Fk(,)20 b(and)g(their)f(sizes,)149 2528 y Ff(GS)217 2535 y Fe(i)232 2528 y Fk(,)j(v)m(ary)f(at)g(run)o(time.)33 b(F)l(unctional)21 b(requiremen)o(ts)d(are)j(that)g(the)g(n)o(um)o(b)q(er)e (of)j(pro)q(cessors)149 2589 y(assigned)c(to)g(eac)o(h)e(discipline)f(is)i (indep)q(enden)o(t)f(of)h(other)h(disciplines,)d(a)i(di\013eren)o(t)f (executable)149 2649 y(ma)o(y)k(b)q(e)i(loaded)g(in)o(to)f(eac)o(h)g(group,)i (eac)o(h)e(group)i(has)f(its)f(o)o(wn)h(logical)f(n)o(um)o(b)q(ering,)g (global)149 2709 y(op)q(erations)k(apply)f(to)g(the)f(lo)q(cal)g(group,)j (mec)o(hanisms)21 b(for)j(group)g(creation)g(and)g(deletion)1036 2864 y(8)p eop %%Page: 9 9 bop 149 42 a Fk(are)17 b(needed,)e(and)i(some)f(mec)o(hanism)d(to)k (establish)f(comm)o(unication)e(b)q(et)o(w)o(een)h(an)i(individual)149 102 y(pro)q(cessor)h(in)e(one)g(group)h(and)g(an)g(individual)e(pro)q(cessor) i(in)f(another)h(group.)223 162 y(Ha)o(ving)h(a)h(\014xed)f(n)o(um)o(b)q(er)f (of)i(pro)q(cessors)h(assigned)g(to)f(the)f(application)h(simpli\014es)d (some)149 222 y(problems)d(and)h(creates)g(others.)20 b(Keeping)14 b(the)f(group)i(information)e(consisten)o(t)g(across)i(groups)149 282 y(is)h(simpli\014ed)e(b)q(ecause)j(it)e(is)h(a)h(b)q(ounded)g(problem)d (and)j(tables)f(ma)o(y)f(b)q(e)h(k)o(ept)g(on)g(a)h(pro)q(cessor)149 342 y(basis)e(rather)f(than)h(a)g(group)g(basis.)21 b(Ho)o(w)o(ev)o(er,)12 b(what)j(happ)q(ens)g(when)f(a)h(group)g(is)f(created)f(and)149 403 y(there)19 b(are)g(no)g(idle)f(pro)q(cessors?)30 b(P)o(ossible)19 b(solutions)g(are)g(to)g(either)f(ab)q(ort)i(the)f(application,)149 463 y(inform)d(the)g(application)h(that)g(it)f(is)h(out)g(of)g(pro)q (cessors,)h(or)f(allo)o(w)g(m)o(ultiple)c(groups)18 b(p)q(er)f(pro-)149 523 y(cessor)i(and)h(time)c(slice)i(and)h(share)g(lo)q(cal)g(memory)d(\(mem)o (ory)g(slice\))i(across)h(groups.)30 b(Recall)149 583 y(that)17 b(parallel)f(pro)q(cessors)h(are)f(b)q(eing)h(considered)e(ph)o(ysical)h (memory)d(mac)o(hines.)223 643 y(The)j(most)f(\015exible)g(solution)h(is)g (informing)f(the)h(application)g(that)g(it)g(has)h(run)f(out)h(of)f(idle)149 704 y(pro)q(cessors)24 b(and)e(let)f(it)h(decide)f(if)g(it)h(w)o(an)o(ts)g (to)g(memory)d(slice)i(or)h(ab)q(ort.)40 b(When)22 b(memory)149 764 y(slicing,)16 b(the)g(user)h(should)g(ha)o(v)o(e)f(the)h(option)g(to)g (sp)q(ecify)f(a)i(logical)e(pro)q(cessor)i(la)o(y)o(out)e(and)h(the)149 824 y(set)i(of)g(ph)o(ysical)f(pro)q(cessors)i(that)f(are)g(to)g(b)q(e)g (memory)d(sliced.)28 b(This)19 b(implies)d(the)i(abilit)o(y)g(of)149 884 y(an)f(application)f(to)h(in)o(terrogate)f(the)g(state)g(of)h(an)o(y)f (pro)q(cessor.)223 944 y(With)j(mem)o(ory)e(slicing,)h(it)h(is)g(also)h (desirable)e(to)i(ha)o(v)o(e)e(group)i(migration)e(as)i(an)g(option.)149 1005 y(This)e(is)f(an)h(issue)g(where)f(solution)h(adaptiv)o(e)f(algorithms)f (are)i(used.)25 b(Assume)16 b(that)i(a)g(region)149 1065 y(needs)h (re\014nemen)o(t.)27 b(Then)19 b(a)h(new)f(grid)g(with)g(greater)g (re\014nemen)o(t)d(can)j(b)q(e)h(placed)e(o)o(v)o(er)g(the)149 1125 y(region.)32 b(The)19 b(new)h(grid)f(is)g(treated)h(as)g(a)g(logically)e (separate)i(group.)32 b(In)19 b(other)h(places,)f(the)149 1185 y(grid)k(ma)o(y)d(b)q(e)i(coarsened.)40 b(In)21 b(those)i(places,)g(groups)g (terminate.)37 b(As)22 b(groups)h(terminate,)149 1245 y(pro)q(cessors)c(ma)o (y)c(b)q(ecome)h(idle)g(while)g(other)h(pro)q(cessors)i(are)e(memory)d (slicing.)23 b(Under)16 b(these)149 1306 y(conditions,)g(the)g(user)h(ma)o(y) d(w)o(an)o(t)i(groups)i(to)e(migrate)f(to)i(the)f(idle)f(pro)q(cessors.)223 1366 y(One)d(of)h(the)g(adv)m(an)o(tages)h(of)f(the)f(h)o(ybrid)g(mo)q(del)g (is)g(that)i(it)e(could)g(b)q(e)h(submitted)e(as)j(a)f(batc)o(h)149 1426 y(job.)22 b(The)16 b(resources)h(required)e(are)h(\014xed)g(and)h(sp)q (eci\014able)e(at)i(application)f(instan)o(tiation.)149 1559 y Fi(5)56 b(Conclusion)149 1669 y Fk(Based)22 b(on)f(memory)e(requiremen)n (ts,)h(memory)e(e\016ciency)l(,)i(Amdahl's)f(La)o(w,)k(computational)149 1729 y(requiremen)o(ts,)d(computational)h(e\016ciency)l(,)f(soft)o(w)o(are)i (engineering,)g(and)g(m)o(ultidisci)o(plinary)149 1789 y(\(m)o(ultizonal\))14 b(coupling)i(a)g(computing)f(mo)q(del)g(that)h(supp)q(orts)h(a)g(MIMD)e(st)o (yle)g(of)h(computing)149 1849 y(across)i(disciplines)c(is)i(the)g(most)g (natural.)223 1909 y(Three)c(computational)h(mo)q(dels)f(w)o(ere)g(in)o(tro)q (duced.)20 b(The)13 b(static)g(mo)q(del)e(is)i(curren)o(tly)f(b)q(eing)149 1970 y(used)20 b(at)h(NASA)d(Ames)g(for)i(m)o(ultidiscipli)o(nary)d(and)k(m)o (ultizonal)c(applications.)32 b(W)l(e)20 b(exp)q(ect)149 2030 y(to)k(mo)o(v)o(e)c(to)o(w)o(ards)j(the)g(h)o(ybrid)f(mo)q(del)f(in)h(the)h (next)f(y)o(ear)g(to)h(supp)q(ort)h(solution)f(adaptiv)o(e)149 2090 y(metho)q(ds.)149 2223 y Fi(References)174 2333 y Fk([1])h(E.)15 b(Barszcz.)k(In)o(tercub)q(e)14 b(comm)o(unicati)o(on)f(for)j(the)f (ipsc/860.)21 b(In)15 b Fa(Sc)n(alable)k(High)e(Perfor-)250 2393 y(manc)n(e)h(Computing)g(Confer)n(enc)n(e)g(`92)f(pr)n(o)n(c)n(e)n(e)n (dings)p Fk(,)e(pages)i(307{313,)h(April)d(1992.)174 2495 y([2])24 b(E.)c(Barszcz,)h(T.)g(F.)f(Chan,)i(D.)f(C.)f(Jesp)q(ersen,)i(and)f(R.)g(S.)f (T)l(uminaro.)34 b(Flo52)22 b(on)f(h)o(y-)250 2555 y(p)q(ercub)q(es:)27 b(P)o(erformance)18 b(and)h(mo)q(delling)f(of)i(a)f(m)o(ultigrid)e(euler)h (co)q(de.)30 b Fa(International)250 2615 y(Journal)17 b(of)g(High)h(Sp)n(e)n (e)n(d)f(Computing)p Fk(,)g(1\(3\):481{503,)h(1989.)1036 2864 y(9)p eop %%Page: 10 10 bop 174 42 a Fk([3])24 b(E.)17 b(Barszcz,)g(S.)g(K.)h(W)l(eeratunga,)g(and)g (R.)f(L.)h(Meaking.)25 b(Dynamic)16 b(o)o(v)o(erset)h(grid)h(com-)250 102 y(m)o(unication)c(on)j(distributed)f(memory)d(parallel)j(pro)q(cessors.) 23 b(T)l(o)17 b(b)q(e)g(presen)o(ted)e(at)i(11th)250 162 y(AIAA)d (Computational)i(Fluid)g(Dynamics)e(Conference,)i(Orlando)g(Florida,)g(July)g (1993.)174 264 y([4])24 b(J.)c(A.)g(Benek,)h(P)l(.)f(G.)h(Buning,)g(and)h(J.) e(L.)h(Steger.)35 b(A)20 b(3-d)i(c)o(himera)d(grid)i(em)o(b)q(edding)250 324 y(tec)o(hnique.)k(American)15 b(Institute)j(of)g(Aeronautics)g(and)h (Astronautics)f(pap)q(er)h(85-1523,)250 384 y(July)c(1985.)174 486 y([5])24 b(R.)11 b(F.)g(V)l(an)g(der)h(Wijngaart.)i(E\016cien)o(t)c (implem)o(en)o(tation)f(of)i(a)h(3-dimensional)f(adi)h(metho)q(d)250 546 y(on)22 b(the)f(ipsc/860.)39 b(Submitted)20 b(to)i(Sup)q(erComputing)g (`93",)h(NASA)e(Ames)f(Researc)o(h)250 606 y(Cen)o(ter,)15 b(Mo\013ett)h(Field,)f(CA)h(94035.)174 708 y([6])24 b(C.)f(A.)f(F)l(elippa)h (and)g(T.)h(L.)f(Geers.)42 b(P)o(artitioned)23 b(analysis)g(for)h(coupled)e (mec)o(hanical)250 768 y(systems.)31 b(College)20 b(of)g(Engineering)g(and)h (Applied)e(Science,)g(Univ)o(ersit)o(y)e(of)k(Colorado,)250 828 y(Boulder,)15 b(CO)h(80309.)174 930 y([7])24 b(R.)19 b(W.)g(Ho)q(c)o (kney)g(and)h(C.)g(R.)f(Jesshop)q(e.)32 b Fa(Par)n(al)r(lel)22 b(Coputers)p Fk(.)31 b(Adam)19 b(Hilger,)g(Bristol)250 990 y(and)e(Philadelphia,)e(2nd)h(edition,)g(1988.)174 1092 y([8])24 b(R.)16 b(L.)i(Meakin.)23 b(A)16 b(new)i(metho)q(d)e(for)i(establishing)f(in) o(ter-grid)f(comm)o(unic)o(ation)f(among)250 1152 y(systems)k(of)i(o)o(v)o (erset)e(grids.)35 b(American)18 b(Institute)i(of)g(Aeronautics)h(and)g (Astronautics)250 1212 y(pap)q(er)c(91-1586,)h(June)e(1991.)174 1314 y([9])24 b(K.)e(C.)h(P)o(ark)h(and)f(C.)g(A.)g(F)l(elippa.)41 b(P)o(artitioned)23 b(analysis)g(of)h(coupled)e(systems.)41 b(In)250 1374 y(T.)18 b(Belytsc)o(hk)o(o)f(and)i(T.)f(J.)h(R.)f(Hughes,)h (editors,)g Fa(Computational)h(Metho)n(ds)f(for)g(T)l(r)n(an-)250 1434 y(sient)d(A)o(nalysis)p Fk(,)f(c)o(hapter)f(3,)h(pages)g(157{219.)i (Elsevier)d(Scienc)f(Publishers)h(B.V.,)f(North)250 1494 y(Holland,)i (Amsterdam,)e(New)j(Y)l(ork,)f(and)i(Oxford,)f(1983.)149 1596 y([10])25 b(E.)14 b(Pramono)g(and)i(S.)e(K.)g(W)l(eeratunga.)19 b(Aero)q(elastic)13 b(computations)h(for)h(wings)g(through)250 1656 y(direct)27 b(coupling)h(on)h(distributed-mem)o(ory)d(mim)o(d)g (parallel)h(computers.)56 b(W)l(ork)28 b(in)250 1716 y(progress)20 b(at)f(the)g(Applied)f(Researc)o(h)h(Branc)o(h,)g(NAS)f(System)g(Division,)h (NASA)f(Ames)250 1777 y(Researc)o(h)d(Cen)o(ter,)g(Mo\013ett)i(Field,)d(CA)i (94035.)149 1878 y([11])25 b(J.S.)13 b(Ry)o(an)g(and)h(S.)g(K.)f(W)l (eeratunga.)k(P)o(arallel)c(computation)g(of)h(3-d)g(na)o(vier-stok)o(es)f (\015o)o(w-)250 1939 y(\014eld)f(for)i(sup)q(ersonic)f(v)o(ehicles.)h (American)d(Institute)i(of)g(Aeronautics)g(and)h(Astronautics)250 1999 y(pap)q(er)j(93-0064,)h(Jan)o(uary)f(1993.)149 2100 y([12])25 b(Thinking)c(Mac)o(hines)g(Corp)q(oration,)j(Cam)o(bridge,)d(MA.)36 b Fa(CM)22 b(F)l(ortr)n(an)g(Pr)n(o)n(gr)n(amming)250 2161 y(Guide)p Fk(,)16 b(v)o(ersion)f(1.0)i(edition,)e(Jan)o(uary)i(1991.)1024 2864 y(10)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Sun Mar 28 16:01:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16813; Sun, 28 Mar 93 16:01:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28455; Sun, 28 Mar 93 15:59:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 28 Mar 1993 15:59:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28447; Sun, 28 Mar 93 15:59:45 -0500 Date: Sun, 28 Mar 93 21:59:29 BST Message-Id: <23593.9303282059@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Multidisciplinary Position Paper To: barszcz@nas.nasa.gov (Eric Barszcz), mpi-comm@cs.utk.edu In-Reply-To: Eric Barszcz's message of Fri, 26 Mar 93 16:40:19 -0800 Reply-To: lyndon@epcc.ed.ac.uk Hi Eric Thanks for mailing this. It's a good example of a "parallel module graph" program. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 29 09:46:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12851; Mon, 29 Mar 93 09:46:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22553; Mon, 29 Mar 93 09:43:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 29 Mar 1993 09:43:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22526; Mon, 29 Mar 93 09:43:09 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16002; Mon, 29 Mar 1993 09:43:08 -0500 Date: Mon, 29 Mar 1993 09:43:08 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303291443.AA16002@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Revised agenda for MPI meeting Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-3:00 Intro to context sub-committee proposals - Skjellum (10 mins.) Proposal VII - Littlefield and Clarke present (30 mins.) Discussion on VII - lead by Littlefield and Clarke (15 mins.) Proposal I - Snir (30 mins.) Discussion on I - lead by Snir (15 mins.) Proposals III and VIII - Skjellum presents (30 mins.) Discussion on III and VIII - lead by Skjellum and Sears (15 mins.) Overall discussion, Recommendations, and Ranking poll (35 mins.) 3:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Continued discussion of Snir, Gropp, Lusk point-to-point proposal (Snir) OR Discussion of Snir & Geist collection communication proposal (Otto) 12:00-1:30 Lunch (provided) 1:30-3:00 Discussion of Snir & Geist collection communication proposal (Otto) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) From owner-mpi-comm@CS.UTK.EDU Mon Mar 29 18:14:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27088; Mon, 29 Mar 93 18:14:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20945; Mon, 29 Mar 93 18:02:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 29 Mar 1993 18:01:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20937; Mon, 29 Mar 93 18:01:56 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA03153; Mon, 29 Mar 93 23:01:48 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01225; Mon, 29 Mar 93 16:00:29 MST Date: Mon, 29 Mar 93 16:00:29 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303292300.AA01225@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Resend of unofficial "opinion survey" Hi everyone, Some folks have requested that the unofficial "MPI Opinion Survey" be distributed again. Here it is. Please ignore if you don't need to see it again! :-) Tom ----------------------------------------------------------------------------- Hi all, Steve Huss-Lederman writes: > I would like to propose that we have a meeting of all participants at > the start of the next meeting to settle the issue of what MPI-1 will > cover. Several mail messages have expressed concern that we cannot > conclude by June if we try to incorporate everything that everyone > wants. Since the scope and time frame will effect most of the other > discussions I think it would be best to begin with this. My only > concern is that it will take too long to decide the global picture. > I would hope it could be done in 1-2 hours and help to refocus the > group. I fully support this idea. Here's a proposal for an opinion survey that could shorten and focus the discussion. This proposal originated with Jon Flower and has had input from Ian Glendinning, Steve Huss-Lederman, Leslie Hart, myself, and others. If you think this is a good idea, please complete the survey and return to me at: hender@fsl.noaa.gov If you think this is a bad idea, please say so! We will collect responses and distribute a summary via email before the next meeting if there is sufficient interest. Tom Henderson ----------------------------------------------------------------------------- MPI PRIORITIES OPINION SURVEY Several MPI committee members have expressed concern about the current and future directions of MPI. The MPI committee began discussion using an existing document (Dongarra et al) as a starting point. This document contains background rationale and a statement of direction for the MPI standardization effort. Since these issues were consistent with the original MPI proposal, they have not recently been discussed in detail by the MPIF. The current MPI proposals are very different from the original MPI document. As a result, we feel that the original rationale and direction need to be re-examined. In particular we need to prioritize and limit the issues that MPI will try to resolve so that o the initial MPI standard document ("MPI-1") is delivered in a timely fashion and o the functions that it defines can (and will) be implemented on major platforms within a reasonable time frame. To get discussion of these issues started, we propose an informal, email "opinion survey". The objective of the survey is to find out which MPI features are most important. High-priority features would be included in MPI-1. Lower-priority features might be deferred to a later version of MPI. Some features might never be included. The compiled results of this survey could be used as a basis for discussion at the next MPI meeting. *** This survey is in no way binding and is a completely unofficial "straw poll" arranged by its authors for their own personal information. It is not intended to distract attention from the current proposals. Any data collected will be made available to the group at large if requested. *** Following is a ballot containing possible MPI features and goals. Please rate each item (or sub-item) using the following rating scheme: 0 This feature should not appear in any version of MPI ever. 1-2 MPIF should consider including this feature in some future version of MPI (after MPI-1). 3-4 This feature should be included as an Annex to MPI-1 for possible inclusion in a future version of MPI. 5-6 This feature should be included as an Annex to MPI-1 for definite inclusion in a future version of MPI. 7-10 This feature should be included in MPI-1. In this context an "Annex" is a part of the standard document that will not be required by a conforming implementation of MPI. Rather it should be taken as a recommendation by the MPIF to vendors for the way in which particular features should be implemented, if they are implemented at all. This way features that are not usefully implementable on particular hardware can be eliminated while machines that do support these features do so in a portable manner. When replying to the survey, please use the response form attached to the end of this message. (This will make compilation of the results a lot less painful!) :-) +-------------------------------------------------------+ | POSSIBLE FEATURES FOR INITIAL AND FUTURE MPI VERSIONS | +-------------------------------------------------------+ Ordering is for convenience only. This list is not meant to suggest any priority. GENERAL FEATURES 1) Routines for static process creation: A) Same program in every node. B) Host-node programming model in which there is one "special node". C) Static model but with potentially different programs in different groups of nodes. D) Static model with a way to create more than one process per processor. (Heavy duty processes, not threads). 2) Routines for identification of statically created processes. 3) Associated environmental inquiry routines. 4) Some way of avoiding message tag conflicts between independently developed libraries and user code (contexts). 5) Heterogeneous communication. A) All communication routines should support heterogeneous communication. B) A special set of communication routines should support heterogeneous communication. 6) Creation/destruction of "static" groups (processes cannot join/leave an existing group). Groups are used to limit the scope of collective operations. A) No logical process topology is associated with a group. B) Grid/toroidal process topology is associated with a group at the time the group is created. (The default "ALL" group has a default topology.) C) General graph topology is also supported. D) Provide access to all relevant information (via topological environmental inquiry routines) to allow users to define custom topologies. 7) Support for communication between processes in different groups without reference to the "ALL" group. 8) Dynamic groups (processes can join/leave an existing group while the total number of processes remains static.) 9) Dynamic process creation and identification. 10) Additional environmental inquiry and "setting" routines for performance optimization. A) Inquire/suggest system buffering resource allocation. B) Inquire relative node performance. C) Inquire relative interconnect performance. 11) Termination of programs. A) Kill all others nodes. B) Wait for all the other nodes to complete. C) Automatically de-allocate resources. 12) Standard organization. A) "Multi-level" organization of routines (users may use simple features without having to understand all of MPI). B) Simple "core" standard with (optional) Annexes. C) Simple "core" standard without Annexes. D) Should an MPI "subset" be defined that can be quickly implemented as a layer on top of existing systems? (This question was inspired by section 1.5 of Bill and Rusty's "A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47.) 13) Should there be future version(s) of MPI (MPI-2, etc.) (yes/no)? POINT-TO-POINT COMMUNICATION 14) Blocking send and receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). 15) Non-blocking receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 16) Non-blocking send. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 17) Low-level functionality (from Marc Snir's proposal) visible to users. A) Initialize. B) Start. C) Complete. D) Free. 18) "SECURE" versions of send and receive as proposed by Lyndon Clarke on 3/3/93. (Correct programs using SECURE routines are guaranteed to work regardless of system buffering resources.) 19) Ready-send. 20) Ready-receive. 21) Message handlers (a la iNTEL hrecv, TMC active messages, Express exhandle). A) Simple definition sufficient to support anticipated MPI requirements and make a self-consistent system. B) More advanced routines for remote procedure calls, etc... 22) Additional point-to-point primitives. A) Exchange between two processors. B) "Shift-exchange": read from one, send to another (superset of "A"). C) Contiguous data. D) Strided data. E) Buffer-descriptor (multiple data locations, multiple data types). F) Non-blocking versions? G) Check for completion (status()). H) Wait for completion (wait()). I) Cancel operation (cancel()). COLLECTIVE OPERATIONS 23) Blocking collective communication. A) Broadcast. B) Concatenate. C) Barrier. D) Sum. E) Extrema (max, min). F) Average. G) Sum of squared differences (error). H) Should strided data be allowed for these operations? I) Should buffer-descriptor be allowed for these operations? 24) "User-defined" blocking collective operations (Express "excombine()" style). 25) Non-blocking collective communication, collective operations, and user-defined operations (if this is desired, detailed ranking can be done later). 26) If any non-blocking collective operations are supported, what should completion/cancel options be? A) Check for completion (status()). B) Wait for completion (wait()). C) "Multiple cancel" operation (cancel()). 27) Should status()/wait()/cancel() syntax be consistent across all types of non-blocking operations? OTHER COMMUNICATION OPERATIONS 28) Probe (and associated "precv()" routines, if necessary). I/O 29) Standard I/O is supported in the standard fashion for the language on one process (no file I/O). This process can be identified by an environmental inquiry routine. (This is the current proposal.) 30) Serial I/O is supported in the standard fashion for the language on one process. This process can be identified by an environmental inquiry routine. (Rik Littlefield's proposal.) 31) Multiplexed I/O similar to that currently provided in systems such as CMMD (TMC), VERTEX (nCUBE) and Express (ParaSoft). Basically a way of defining the behavior of multi-node I/O statements which interact with a sequential device. No special considerations for parallel I/O channels. 32) Blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. 33) Non-blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. LANGUAGE BINDINGS 34) Ada 35) C 36) C++ 37) Fortran 77 38) Fortran 90 39) Other IMPLEMENTATION PERIOD There will be a delay between the generation of the standard and its implementation on various platforms. If this delay is too long users will continue to rely more and more heavily on vendor routines and specialized calls and MPI will become less and less attractive, particularly if new technology such as Fortran 90/HPF becomes effective. 40) What is the maximum delay that you think MPI can stand and still succeed (please feel free to comment if you wish)? ---------------------------------------------------------------------------- SURVEY RESPONSE FORM 1) A) B) C) D) 2) 3) 4) 5) A) B) 6) A) B) C) D) 7) 8) 9) 10) A) B) C) 11) A) B) C) 12) A) B) C) D) 13) (yes/no) 14) A) B) C) 15) A) B) C) D) E) F) 16) A) B) C) D) E) F) 17) A) B) C) D) 18) 19) 20) 21) A) B) 22) A) B) C) D) E) F) G) H) I) 23) A) B) C) D) E) F) G) H) I) 24) 25) 26) A) B) C) 27) 28) 29) 30) 31) 32) 33) 34) 35) 36) 37) 38) 39) 40) (free form): Comments? From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 00:51:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02775; Tue, 30 Mar 93 00:51:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06540; Tue, 30 Mar 93 00:49:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 00:49:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06532; Tue, 30 Mar 93 00:49:01 -0500 Received: from Basselope.Think.COM by mail.think.com; Tue, 30 Mar 93 00:48:58 -0500 From: Alan Mainwaring Received: by basselope.think.com (4.1/Think-1.2) id AA15508; Tue, 30 Mar 93 00:48:57 EST Date: Tue, 30 Mar 93 00:48:57 EST Message-Id: <9303300548.AA15508@basselope.think.com> To: mpi-comm@cs.utk.edu In-Reply-To: <9303291443.AA16002@rios2.epm.ornl.gov>, "David Walker's message of Mon, 29 Mar 1993 09:43:08 -0500" Subject: Revised agenda for MPI meeting Date: Mon, 29 Mar 1993 09:43:08 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-3:00 Intro to context sub-committee proposals - Skjellum (10 mins.) Proposal VII - Littlefield and Clarke present (30 mins.) Discussion on VII - lead by Littlefield and Clarke (15 mins.) Proposal I - Snir (30 mins.) Discussion on I - lead by Snir (15 mins.) Proposals III and VIII - Skjellum presents (30 mins.) Discussion on III and VIII - lead by Skjellum and Sears (15 mins.) Overall discussion, Recommendations, and Ranking poll (35 mins.) 3:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Continued discussion of Snir, Gropp, Lusk point-to-point proposal (Snir) OR Discussion of Snir & Geist collection communication proposal (Otto) 12:00-1:30 Lunch (provided) 1:30-3:00 Discussion of Snir & Geist collection communication proposal (Otto) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) After reviewing this schedule, we believe that the last item should be moved so that it is the first order of business at this meeting. If it is a goal for vendors to quickly implement and release MPI, then, it is imperative that we understand what functionality must be included in MPI1, and what functionality may be deferred to subsequent versions. Without this framework we seriously doubt that this process will yield a specification, against which vendors can implement production software this year. Respectfully, Alan and Moose From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 01:16:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03507; Tue, 30 Mar 93 01:16:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07612; Tue, 30 Mar 93 01:15:09 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 01:15:08 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07555; Tue, 30 Mar 93 01:15:05 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA11172; Mon, 29 Mar 93 22:15:02 PST Date: Mon, 29 Mar 93 22:15:02 PST Message-Id: <9303300615.AA11172@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA15164; Mon, 29 Mar 93 22:14:59 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting In-Reply-To: <9303300548.AA15508@basselope.think.com> References: <9303291443.AA16002@rios2.epm.ornl.gov> <9303300548.AA15508@basselope.think.com> Reply-To: knighten@SSD.intel.com (Bob Knighten) Alan Mainwaring writes: > Date: Mon, 29 Mar 1993 09:43:08 -0500 > From: walker@rios2.epm.ornl.gov (David Walker) > > > Provisional Agenda for MPI Meeting, March 31-April 2, 1993 > > . . . > > > After reviewing this schedule, we believe that the last item should be > moved so that it is the first order of business at this meeting. > > If it is a goal for vendors to quickly implement and release MPI, then, it > is imperative that we understand what functionality must be included in MPI1, > and what functionality may be deferred to subsequent versions. Without this > framework we seriously doubt that this process will yield a specification, > against which vendors can implement production software this year. > > Respectfully, > > Alan and Moose > I strongly support this re-ordering of the agenda. -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:35:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12387; Tue, 30 Mar 93 10:35:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08221; Tue, 30 Mar 93 10:33:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:33:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08206; Tue, 30 Mar 93 10:33:26 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA01022; Tue, 30 Mar 93 10:33:20 EST Received: by b125.super.org (4.1/SMI-4.1) id AA03597; Tue, 30 Mar 93 10:33:19 EST Date: Tue, 30 Mar 93 10:33:19 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303301533.AA03597@b125.super.org> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting In response to the recent message by Alan and Moose, I want to reiterate that I support an early discussion of the goals and timetable for MPI. This will allow the general framework to be set when we decide on how to deal with specific proposals. Given the fact that some of the MPI members are already on travel to the meeting, I suspect that this change in agenda would have to be dealt with at the start of the meeting tomorrow. Steve From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:42:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12524; Tue, 30 Mar 93 10:42:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09326; Tue, 30 Mar 93 10:40:32 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:40:30 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09303; Tue, 30 Mar 93 10:40:28 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA19731; Tue, 30 Mar 93 08:40:24 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0ndiQJ-0016bbC; Tue, 30 Mar 93 08:40 MST Message-Id: Date: Tue, 30 Mar 93 08:40 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I must disagree with the suggestion to change the schedule. Without any agreement to even basic, fundamental concepts how are we to decide what will go into MPI1 and what won't. I do indeed understand the desire to know the scope of MPI1, but I really believe the scope depends on an agreed method to achieve contexts, groups, etc. It is those definitions that will define how much work a given set of MPI routines will require. Stephen From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:48:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12768; Tue, 30 Mar 93 10:48:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09919; Tue, 30 Mar 93 10:47:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:47:23 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09910; Tue, 30 Mar 93 10:47:23 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19794; Tue, 30 Mar 1993 10:47:22 -0500 Date: Tue, 30 Mar 1993 10:47:22 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303301547.AA19794@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Revised agenda for MPI meeting I don't think it's a good idea to change the agenda. There seems little point on talking aboutwhat should be in MPI until everyone knows what has been proposed, particularly for contexts. Setting a timetable will be very difficulty if we can't largely agree on point-to-point, collective communication, and contexts, which is why these topics come first. David From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:51:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12832; Tue, 30 Mar 93 10:51:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10133; Tue, 30 Mar 93 10:49:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:49:34 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10095; Tue, 30 Mar 93 10:49:32 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AB04355; Tue, 30 Mar 93 15:49:27 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA02302; Tue, 30 Mar 93 08:48:08 MST Date: Tue, 30 Mar 93 08:48:08 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303301548.AA02302@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting Alan Mainwaring writes, > After reviewing this schedule, we believe that the last item should be > moved so that it is the first order of business at this meeting. > > If it is a goal for vendors to quickly implement and release MPI, then, it > is imperative that we understand what functionality must be included in MPI1, > and what functionality may be deferred to subsequent versions. Without this > framework we seriously doubt that this process will yield a specification, > against which vendors can implement production software this year. > > Respectfully, > > Alan and Moose I agree. It would be very helpful if vendors would speak frankly about how long implementation might take during this discussion (wishful thinking? :-). Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 11:45:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14240; Tue, 30 Mar 93 11:45:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12714; Tue, 30 Mar 93 11:43:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 11:43:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12705; Tue, 30 Mar 93 11:43:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04547; Tue, 30 Mar 93 16:43:34 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA08559; Tue, 30 Mar 93 09:46:14 MST Date: Tue, 30 Mar 93 09:46:14 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303301646.AA08559@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I strongly agree with Steve (see below) on both points. I think that we need to discuss MPI priorities early at the meeting (definitely not the last agenda item!). I agree also since it is likely that a number of people are already traveling, the agenda itself should be the first agenda item. Regards, Leslie Hart (hart@fsl.noaa.gov) ----- Begin Included Message ----- From: lederman@b125.super.org (Steve Huss-Lederman) In response to the recent message by Alan and Moose, I want to reiterate that I support an early discussion of the goals and timetable for MPI. This will allow the general framework to be set when we decide on how to deal with specific proposals. Given the fact that some of the MPI members are already on travel to the meeting, I suspect that this change in agenda would have to be dealt with at the start of the meeting tomorrow. Steve ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 13:55:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17414; Tue, 30 Mar 93 13:55:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19024; Tue, 30 Mar 93 13:53:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 13:53:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19016; Tue, 30 Mar 93 13:53:50 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA14792; Tue, 30 Mar 93 12:43:16 CST Date: Tue, 30 Mar 93 12:43:16 CST From: Tony Skjellum Message-Id: <9303301843.AA14792@Aurora.CS.MsState.Edu> To: amm@Think.COM, mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I have also asked David to make this change. I repeat that this step is important, and I would not mind being delayed in my context group's presentations, to make room for said discussion. Please note, however, contexts must finish by noon on Thursday, if the agenda is changed, because I have to leave (unexpectedly) at 1pm on Thursday. . . . . . . . . . . "There is no lifeguard at the gene pool." - C. H. Baldwin Anthony Skjellum, MSU/ERC, (601)325-8435; FAX: 325-8997; tony@cs.msstate.edu From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 16:15:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20542; Tue, 30 Mar 93 16:15:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26319; Tue, 30 Mar 93 16:13:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 16:13:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26310; Tue, 30 Mar 93 16:13:19 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA27775; Tue, 30 Mar 93 14:13:17 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0ndncS-0016bbC; Tue, 30 Mar 93 14:13 MST Message-Id: Date: Tue, 30 Mar 93 14:13 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I too have other pressing matters that were neatly integrated with the original schedule. I must leave Wed at 4pm, to go elsewhere; I will be back Thursday night. I have used the agenda for its purpose, to make plans about. These plans are worthless if I cannot be at the context discussion. I know this is my own problem for using the agenda for the purposes of what an agenda is for. Furthermore, I repeat that I believe it to be fruitless to discuss the MPI1 contents without agreement on context and group. Stephen From owner-mpi-comm@CS.UTK.EDU Tue Apr 6 10:39:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18806; Tue, 6 Apr 93 10:39:59 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23928; Tue, 6 Apr 93 10:38:00 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 6 Apr 1993 10:37:59 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23920; Tue, 6 Apr 93 10:37:58 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA28082; Tue, 6 Apr 93 10:37:56 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA06018; Tue, 6 Apr 93 10:37:56 EDT Date: Tue, 6 Apr 93 10:37:56 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304061437.AA06018@b125.super.org> To: mpi-comm@cs.utk.edu Subject: subset subcommittee Hello MPIers, At the last MPI meeting, a group met to discuss the possibility of creating a recommended subset of MPI for initial implementation. The idea was presented to the whole group and there were no objections. I have volunteered to coordinate this effort. At the current time I am seeking input to the process and people interested in joining/monitoring this new subcommittee (subset subcommittee for lack of a better name). Send me e-mail (lederman@super.org) to sign up (and suggest a better name :-). At the end of this week I will send all the names of people to David to create a new mail alias. When it is formed you will receive an initial message. You can, of course, join at a later time. Steve From owner-mpi-comm@CS.UTK.EDU Tue Apr 6 13:30:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22321; Tue, 6 Apr 93 13:30:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02738; Tue, 6 Apr 93 13:28:41 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 6 Apr 1993 13:28:40 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02729; Tue, 6 Apr 93 13:28:38 -0400 Date: Tue, 6 Apr 93 18:28:33 BST Message-Id: <754.9304061728@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: interrupt driven To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Dear MPI Colleagues We have, reasonably in my opinion, decided not to provide interrupt driven receive/send (as in Intel hrecv/hsend) in MPI, or active messages. I have heard, can't recall from whom, that this carries the consequence that we should not define any feature of MPI which asks for the implementor of MPI to make use of interrupt driven receives, or active messages. This seems to me to be a bogus argument. In my mind the primary reason for not providing hrecv/hsend type facilities is that it is very difficult to standardise these facilities across different operating systems, particularly in terms of what the handler procedure is and is not allowed to do (e.g., can it do I/O, can it use MPI, ...). It seems to me clear that on many (all) machines of interest the system itself will be making use of similar facilities. Who would write MPI for CM-5 without using active messages? It seems to me that for example non blocking communications really do ask for use of interrupt/active messages. Is there agreement, disagreement, or what on this point? Please do let me know. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Apr 7 13:50:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18878; Wed, 7 Apr 93 13:50:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15424; Wed, 7 Apr 93 13:49:23 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 7 Apr 1993 13:49:22 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15367; Wed, 7 Apr 93 13:48:34 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 7 Apr 93 10:48 PDT Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06918; Wed, 7 Apr 93 10:46:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA17058; Wed, 7 Apr 93 10:46:26 PDT Date: Wed, 7 Apr 93 10:46:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: Re: interrupt driven To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@carbon.pnl.gov, mpi-pt2pt@cs.utk.edu Message-Id: <9304071746.AA17058@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. > > I have heard, can't recall from whom, that this carries the consequence > that we should not define any feature of MPI which asks for the > implementor of MPI to make use of interrupt driven receives, or active > messages. This seems to me to be a bogus argument. I (Rik Littlefield) have regularly made arguments that are close enough to this to be worth clarifying or defending. Think of the MPI specification as being divided into two parts: 1) things you can implement with portable code, and 2) things you can't. My bias is to make the POTENTIALLY portable part of MPI as large as possible. (I emphasize "potentially" to make it clear that I am talking about interface definitions. I encourage implementors to do whatever they want internally.) As an example, consider groups and collective communication. All other factors being equal, I would strongly prefer a specification that permitted groups and collective communication, including group formation and disbanding, to be implemented using just the standardized MPI point-to-point capabilities. This sort of layerability potentially has several good effects. Three of these are 1) earlier widespread availability via porting, 2) better focusing of vendor effort on tuning the point-to-point layer, and 3) the capability to create portable MPI-plug-compatible implementations that are tuned for different requirements, e.g., debugging versus performance. Of course, all other factors are not equal. Strict layerability also has some potentially bad effects. Two of these (depending on where you draw the line between layers) are 1) the loss of integrated syntax (e.g., being able to use rank directly in the point-to-point calls), and 2) loss of functionality that is not supported by lower layers (e.g., asynchronous name servers). Whether the good outweighs the bad is a matter of opinion. The applications that I deal with now and anticipate dealing with over the lifetime of MPI-1 can be characterized as follows: . sensitive to performance, . intended to run on lots of platforms, . not sensitive to syntax, and . not needing name server support. Applications like these are best served by layerability. Others' mileage may differ. Lyndon continues: > In my mind the primary reason for not providing hrecv/hsend type > facilities is that it is very difficult to standardise these facilities > across different operating systems, particularly in terms of what the > handler procedure is and is not allowed to do (e.g., can it do I/O, can > it use MPI, ...). It seems to me clear that on many (all) machines of > interest the system itself will be making use of similar facilities. > Who would write MPI for CM-5 without using active messages? It seems to > me that for example non blocking communications really do ask for use of > interrupt/active messages. > > Is there agreement, disagreement, or what on this point? Please do let > me know. I agree that it is important to distinguish between exporting a standardized hrecv/hsend and and exporting a standardized asynchronous name server interface. The former seems hopeless; the latter is doable, but at the cost of potentially detracting from other features such as performance and prompt widespread availability. Thus, I do not agree that it is bogus to argue against requiring asynchronous servers et.al. in MPI-1. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-comm@CS.UTK.EDU Wed Apr 7 14:29:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20295; Wed, 7 Apr 93 14:29:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17859; Wed, 7 Apr 93 14:27:46 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 7 Apr 1993 14:27:44 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from arthur.cs.purdue.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17851; Wed, 7 Apr 93 14:27:43 -0400 Received: from dragomirna.cs.purdue.edu by arthur.cs.purdue.edu (5.65c/PURDUE_CS-1.2) id ; Wed, 7 Apr 1993 13:27:40 -0500 Received: by dragomirna.cs.purdue.edu (5.65c/PURDUE_CS-1.2) id ; Wed, 7 Apr 1993 13:27:39 -0500 Date: Wed, 7 Apr 1993 13:27:39 -0500 From: dcm@cs.purdue.edu (Dan C Marinescu) Message-Id: <199304071827.AA28416@dragomirna.cs.purdue.edu> To: dcm@cs.purdue.edu, lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Subject: interrupt driven messages Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. Interrupt driven messages are too important to be left out. Any attempt made either by the application or by the compiler to cache data in the local storage of any compute node can only be effective if interrupt driven messages are supported. The interrupt handling procedure should allow both MPI and I/O. Indeed, this opens a Pandora box. For example how many such messages should be queued for a suspended process, should one place a limit upon the size of an interrupt message and upon the time spent in the interrupt handling procedure and so on. Yet it seems reasonable to expect that the standardization effort in OS will make this task easier, for example one could expect that more vendors will base their OS on OSF/1. Moreover, interrupt driven messages are likely to be provided by most vendors. Dan Marinescu From owner-mpi-comm@CS.UTK.EDU Thu Apr 8 07:12:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07522; Thu, 8 Apr 93 07:12:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05594; Thu, 8 Apr 93 07:10:15 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 8 Apr 1993 07:10:14 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05576; Thu, 8 Apr 93 07:10:10 -0400 Date: Thu, 8 Apr 93 12:10:04 BST Message-Id: <2092.9304081110@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-comm: various (long) To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I am sending this letter to mpi-comm as it seems to cover issues in many subcommittees and in particular: point-to-point, collective, context, environmental. This is a long, but I believe important, letter. The letter is a discussion of some issues primarily in point-to-point, collective and context, with some strong and powerful recommendations for each of these subcommittees. o--------------------o 1) In the recent meeting we discussed and narrowly accepted an amendment to the point-to-point chapter which included secure (aka synchronous, aka unbuffered) communication. Paul Pierce made the point that providing different point-to-point procedures does not seem to be the right model for this capability as it has a microscopic effect on MPI user software. This point is actually well taken (by myself at least). Paul went on to suggest that there should be application global control over whether point-to-point communications are all secure (aka synchronous, aka unbuffered) or all insecure (aka regular, aka asynchronous, aka buffered). I opposed this suggestion as it does not respect the requirements of library writers. There appeared to be a number of people who were concerned about the geometric increase in the number of point-to-point communication procedures. The discussion of secure send and secure receive became messy, and I promised to return some more sane discussion of the point. In this letter I will later (after consideration of some issues in collective communications and communication contexts) make a recommendation which: respects the well taken point of Paul; provides a mechanism for the global control suggested by Paul while respecting the independence of library writers from user program writers; avoids geometric growth in the number of point-to-point procedures; and completely side-steps the secure send and secure receive issue. 2) The question of whether collective communication procedures do or do not barrier synchronise the group has been raised a number of times. In the recent meeting this lead to Steve Otto suggesting that MPI user software should be written as though these operations do barrier synchronise although the MPI implementation is not obliged to make the operations so do. This is exactly analagous to the point-to-point insecure (aka regular, ...) communications which may or may not synchronise the sender and receiver depending on the implementation (read depending on system buffering resource availability if you prefer). In order to write safe (i.e. guaranteed portable) programs with insecure (aka regular, ...) communications the MPI user software should be written as though these communications were secure (aka synchronous, ...). In this letter I will later (after consideration of some issues in communication contexts) make a recommendation which: provides for both secure (i.e. barrier synchronous) and insecure (i.e. may or may not be barrier synchronous depending on implementation) collective communications; and avoids geometric growth in the number of collective communication procedures. 3) In the context subcommittee Mark Sears, in Proposal VIII and in presentation at the recent meeting, has pointed out that there are two classes of MPI user libraries. I will discuss this (from my point of view) briefly here for concreteness. There is a class of libraries which accept context(s) from the caller and use these context(s) for communication. Note that since the library uses the contexts(s) given by the caller then it is clearly the responsibility of the caller to ensure that there are no "dangling" communications, either in buffers or nonblocking, which will interfere with the communications of the library, and it is the responsibility of the library to ensure that there are no dangling communications between calls to the library which will interfere with communications of the user. Let me label this class as ClassA. There is a class of libraries which create context(s) for library private use and use only these context(s) for communication. Note that since the library uses private context(s) then the user does not have any concern regarding dangling communications which may interfere with the library communications, and equally the library may leave dangling communications between library calls since they cannot interfere with the user communications. Let me label this class as ClassB. ClassA libraries will generally operate within one (or more) group(s) and accept one (or more) group(s) from the caller, in addition to context(s). ClassB libraries will also generally operate within one (or more) group(s) and accept one (or more) group(s) from the caller, but do not accept context(s). This observation suggests that context and group should be different entities within MPI. (Whether we have an object which binds a group and a context is another issue to which I wish to return in a separate message of narrower circulation.) Observe that the (blocking) collective communications which are described in the MPI collective communication chapter draft are conformant with the ClassA library described and s single group. The kind of libraries which I am writing (and I suspect also Tony, although I am not sure) are conformant with the ClassB library described. 4) I now make a recommendation which ties together the points discussed above and delivers the promises made. First we observe that context is the MPI mechanism for distinguishing the communications of the user from (ClassB) libraries. The recommendation is that secure or insecure modes of operation are a a property of the communication context. This applies equally to both point-to-point communications. In point-to-point communications a secure context provides the secure point-to-point communication which I described at the recent meeting, and an insecure context provides the inseure point-to-point communications which is as discussed (regular) in the current chapter draft. (The semantics and desirability of ready-receiver communication in secure and insecure contexts is a subject to which I wish to return in a separate message of narrower circulation.) In collective communications a secure context provides barrier synchronisation across the group, and an insecure context may or may not provide barrier synchronisation which is as discussed in the recent meeting. There should be an application global default context property, either secure or insecure, which is specified by the user in an environmental management procedure (perhaps as argument to mpi_init()?). Creation of context should specify whether the created context is to have: default property; secure property; insecure property. It is possibe to provide these capabilities in terms of buffering in different ways. One possibility is that the property of a context is the buffering associated with that context, in which case a secure context is one with zero associated buffering. Another possibility is that the property of a context is whether or not it should use whatever buffering is available, in which case a secure context is one which does not use any buffering. My preference is for the latter since it does not prescribe the description and association of buffering. This recommendation does deliver the promises made. The requirements of the (ClassB) library writer are respected while providing global control for the user program writer. The capabilities required are provided without geometric growth of the number of point-to-point and/or collective communications. o--------------------o Free beer to all who can demonstrate that they have read and considered this (long long letter). Comments, questions, (flames :-) please?! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Fri Apr 9 11:26:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29024; Fri, 9 Apr 93 11:26:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23758; Fri, 9 Apr 93 11:24:22 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 9 Apr 1993 11:24:21 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23750; Fri, 9 Apr 93 11:24:19 -0400 Date: Fri, 9 Apr 93 16:24:14 BST Message-Id: <3171.9304091524@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-comm: various (long) To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Thu, 8 Apr 93 12:10:04 BST Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues This message is short. It is two little things which I forgot to say in the "mpi-comm: various (long)" message. We can loosely associate ClassA libraries as collections of procedures for which one can use the substitution model of evaluation, and ClassB procedures as encapsulated objects. The argument lists of of ClassA and ClassB libraries differ primarily in that ClassA libraries accept context(s) whereas ClassB libraries do not. This seems (to me) to convey just the right idea to the library user regarding the responsibilities for management of contexts and messages within those contexts. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Tue Apr 20 16:21:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08357; Tue, 20 Apr 93 16:21:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03609; Tue, 20 Apr 93 16:18:26 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 20 Apr 1993 16:18:25 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03601; Tue, 20 Apr 93 16:18:23 -0400 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA14309; Tue, 20 Apr 1993 16:18:20 -0400 Date: Tue, 20 Apr 1993 16:18:20 -0400 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9304202018.AA14309@gstws.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: 2nd draft of Collective Communication proposal After studying the latest point-to-point draft by Marc and the votes from the last meeting, I have revised the original collective communication draft. Comments can be directed to the mpi-collcomm committee. Here is the LaTeX version, a postscript version will follow in a separate message. Al ----------------------------- __o /\ Al Geist _`\<,_ /\/ \ Oak Ridge National Laboratory (_)/ (_) / \ (615) 574-3153 gst@ornl.gov * * * * * * * * * * ----------------------------- % MPI Authors: % This is MY version of YOUR chapter. It has a wrapper so that you % should be able to simply LaTeX this. % % Please work from this text so that we are in synch. % % --Steve Otto \documentstyle[twoside,11pt]{report} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface} \author{Scott Berryman, {\em Yale Univ} \\ James Cownie, {\em Meiko Ltd} \\ Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\ Al Geist, {\em ORNL} \\ Bill Gropp, {\em ANL} \\ Rolf Hempel, {\em GMD} \\ Bob Knighten, {\em Intel} \\ Rusty Lusk, {\em ANL} \\ Steve Otto, {\em Oregon Graduate Inst} \\ Tony Skjellum, {\em Missisippi State Univ} \\ Marc Snir, {\em IBM T. J. Watson} \\ David Walker, {\em ORNL} \\ Steve Zenith, {\em Kuck \& Associates} } \date{April 20, 1993 \\ This work was supported by ARPA and NSF under contract number \#\#\#, by the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615. } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \chapter{Collective Communication} \label{sec:coll} \begin{center} Al Geist \\ Marc Snir \end{center} \section{Introduction} This section is a draft of the current proposal for collective communication. Collective communication is defined to be communication that involves a group of processes. Examples are broadcast and global sum. A collective operation is executed by having all processes in the group call the communication routine, with matching parameters. Routines can (but are not required to) return as soon as their participation in the collective communication is complete. The completion of a call indicates that the caller is now free to access the locations in the communication buffer, or any other location that can be referenced by the collective operation. However, it does not indicate that other processes in the group have started the operation (unless otherwise indicated in the description of the operation). However, the successful completion of a collective communication call may depend on the execution of a matching call at all processes in the group. The syntax and semantics of the collective operations is defined so as to be consistent with the syntax and semantics of the point to point operations. The reader is referred to the point-to-point communication section of the current MPI draft for information concerning communication buffers and their manipulations. The context section describes the formation, manipulation, and query functions (such as group size) that are available for groups and group objects. The collective communication routines are built above the point-to-point routines. While vendors may optimize certain collective routines for their architectures, a complete library of the collective communication routines can be written entirely using point-to-point communication functions. We are using naive implementations of the collective calls in terms of point to point operations in order to provide an operational definition of their semantics. The following communication functions are proposed. \begin{itemize} \item Broadcast from one member to all members of a group. \item Barrier across all group members \item Gather data from all group members to one member. \item Scatter data from one member to all members of a group. \item Global operations such as sum, max, min, etc., were the result is known by all group members and a variation where the result is known by only one member. The ability to have user defined global operations. \item Simultaneous shift of data around the group, the simplest example being all members sending their data to (rank+1) with wrap around. \item Scan across all members of a group (also called parallel prefix). \item Broadcast from all members to all members of a group. \item Scatter data from all members to all members of a group (also called complete exchange or index). \end{itemize} To simplify the collective communication interface it is designed with two layers. The low level routines have all the generality of, and make use of, the communication buffer routines of the point-to-point section which allows arbitrarily complex messages to be constructed. The second level routines are similar to the upper level point-to-point routines in that they send only a contiguous buffer. \section{Group Functions} A full description of the group formation and manipulation functions can be found in the context chapter of the MPI document. Here we describe only those group functions that are used in the semantic description of the collective communication routines. An initial group containing all processes is supplied by default in MPI. MPI provides a procedure that returns the handle to this initial group. The processes in the inital group each have a unique rank in the group represented by integers (0, 1, 2, ..., [number-of-processes~-~1]. \mpifunc{MPI\_ALLGROUP(group)} Return the descriptor of the inital group containing all processes. \begin{description} \item[OUT group] handle to descriptor object of initial group. \end{description} \mpifunc{MPI\_RANK(group, rank)} Return the rank of the calling process within the specified group. \begin{description} \item[IN group] group handle \item[OUT rank] integer \end{description} \mpifunc{MPI\_GSIZE(group, size)} Return the number of processes that belong to the specified group. \begin{description} \item[IN group] group handle \item[OUT size] integer \end{description} \section{Communication Functions} The proposed communication functions are divided into two layers. The lowest level uses the same buffer descriptor objects available in point-to-point to create noncontiguous, multiple data type messages. The second level is similar to the block send/receive point-to-point operations in that it supports only contiguous buffers of data. For each communication operation, we list these two level of calls together. \section{Synchronization} \subsubsection*{Barrier synchronization} \mpifunc{MPI\_BARRIER( group )} MPI\_BARRIER blocks the calling process until all group members have called it; the call returns at any process only after all group members have entered the call. \begin{description} \item[IN group] group handle \end{description} {\tt MPI\_BARRIER( group )} is \begin{verbatim} MPI_CREATE_BUFFER(buffer_handle, MPI_BUFFER, MPI_PERSISTENT); MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); if (rank==0) { for (i=1; i < size; i++) MPI_RECV(buffer_handle, i, tag, group, return_handle); for (i=1; i < size; i++) MPI_SEND(buffer_handle, i, tag, group); } else { MPI_SEND(buffer_handle, 0, tag, group); MPI_RECV(buffer_handle, 0, tag, group, return_handle); } MPI_FREE(buffer_handle); \end{verbatim} \discuss{ since tag was voted out in the last meeting, a "safe" tag must be selected to use in the above description } \section{Data move functions} \subsubsection*{Broadcast} \mpifunc{ MPI\_BCAST( buffer\_handle, group, root )} {\tt MPI\_BCAST} broadcasts a message from the process with rank {\tt root} to all other processes of the group. It is called by all members of group using the same arguments for {\tt group, and root}. On return the contents of the buffer of the process with rank {\tt root} is contained in buffer of all group members. \begin{description} \item[INOUT buffer\_handle] Handle for buffer where from message is sent or received. \item[IN group] group handle \item[IN root] rank of broadcast root (integer) \end{description} \mpifunc{ MPI\_BCASTC( buf, len, group, root )} {\tt MPI\_BCASTC} behaves like broadcast, restricted to a block buffer. It is called by all processes with the same arguments for {\tt len, group} and {\tt root}. \begin{description} \item[INOUT buffer] Starting address of buffer (choice type) \item[IN len] Number of words in buffer (integer) \item[IN group] group handle \item[in root] rank of broadcast root (integer) \end{description} {\tt MPI\_BCAST( buffer\_handle, group, root )} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_IRECV(handle, buffer_handle, root, tag, group, return_handle); if (rank==root) for (i=0; i < size; i++) MPI_SEND(buffer_handle, i, tag, group); MPI_WAIT(handle) \end{verbatim} \subsubsection*{Circular shift} \mpifunc{MPI\_CSHIFT( inbuf, outbuf, group, shift)} Process with rank {\tt i} sends the data in its input buffer to process with rank $\tt (i+ shift) \bmod group\_size$, who receives the data in its output buffer. All processes make the call with the same values for {\tt group}, and {\tt shift}. The {\tt shift} value can be positive, zero, or negative. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFT1( buf, group, shift)} Process with rank {\tt i} sends the data in its buffer to process with rank $\tt (i+ shift) \bmod group\_size$, who receives the data in the same buffer. All processes make the call with the same values for {\tt group}, and {\tt shift}. The {\tt shift} value can be positive, zero, or negative. \begin{description} \item[INOUT buf] handle to buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFTC( inbuf, outbuf, len, group, shift)} Behaves like {\tt MPI\_CSHIFT}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[IN inbuf] initial location of input buffer \item[OUT outbuf] initial location of output buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFTC1( buf, len, group, shift)} Behaves like {\tt MPI\_CSHIFT1}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[INOUT buf] initial location of buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} {\tt MPI\_CSHIFT( inbuf, outbuf, group, shift)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_ISEND( handle, inbuf, mod(rank+shift, size), tag, group); MPI_RECV( outbuf, mod(rank-shift,size), tag, group, return_handle) MPI_WAIT(handle); \end{verbatim} \subsubsection*{End-off shift} \mpifunc{MPI\_EOSHIFT( inbuf, outbuf, group, shift)} Process with rank {\tt i}, $\tt \max( 0, -shift) \le i < min( size, size - shift)$, sends the data in its input buffer to process with rank {\tt i+ shift}, who receives the data in its output buffer. The output buffer of processes which do not receive data is left unchanged. All processes make the call with the same values for {\tt group}, and {\tt shift}. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFT1( buf, group, shift)} Process with rank {\tt i}, $\tt \max( 0, -shift) \le i < min( size, size - shift)$, sends the data in its buffer to process with rank {\tt i+ shift}, who receives the data in the same buffer. The output buffer of processes which do not receive data is left unchanged. All processes make the call with the same values for {\tt group}, and {\tt shift}. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFTC( inbuf, outbuf, len, group, shift)} Behaves like {\tt MPI\_EOSHIFT}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[IN inbuf] initial location of input buffer \item[OUT outbuf] initial location of output buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFTC1( buf, len, group, shift)} Behaves like {\tt MPI\_EOSHIFT1}, with buffer restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[INOUT buf] initial location of buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \subsubsection*{Gather} \mpifunc{MPI\_GATHER( inbuf, outbuf, group, root, return\_status) } Each process (including the root process) sends the content of its input buffer to the root process. The root process concatenates all the incoming messages in the order of the senders' rank and places the results in its output buffer. It is called by all members of group using the same arguments for {\tt group}, and {\tt root}. The input buffer of each process may have a different length. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor -- significant only at root (choice) \item[IN group] group handle \item[IN root] rank of receiving process (integer) \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_GATHERC( inbuf, inlen, outbuf, group, root) } {\tt MPI\_GATHERC} behaves like {\tt MPI\_GATHER} restricted to block buffers, and with the additional restriction that all input buffers should have the same length. All processes should provided the same values for {\tt inlen, group}, and {\tt root} . \begin{description} \item[IN inbuf] first variable of input buffer (choice) \item[IN inlen] Number of (word) variables in input buffer (integer) \item[OUT outbuf] first variable of output buffer -- significant only at root (choice) \item[IN group] group handle \item[IN root] rank of receiving process (integer) \end{description} {\tt MPI\_GATHERC( inbuf, inlen, outbuf, tag, group, root) } is \begin{verbatim} MPI_GSIZE( &size, group); MPI_RANK( &rank, group); MPI_ISENDC(handle, inbuf, inlen, root, tag, group); if (rank==root) for (i=0; i < size; i++) { MPI_RECVC(outbuf, inlen, i, tag, group, return\_status); outbuf += inlen; } MPI_WAIT(handle); \end{verbatim} \subsubsection*{Scatter} \mpifunc{MPI\_SCATTER( list\_of\_inbufs, outbuf, group, root, return\_status)} The root process sends the content of its {\tt i}-th input buffer to the process with rank {\tt i}; each process (including the root process) stores the incoming message in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt root}. \begin{description} \item[IN list\_of\_inbufs] list of buffer descriptor handles \item[OUT outbuf] buffer descriptor handle \item[IN group] handle \item[IN root] rank of sending process (integer) \item[OUT return\_status] return status handle \end{description} {\tt MPI\_SCATTER( list\_of\_inbufs, outbuf, group, root, return\_status)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_IRECV(handle, outbuf, root, tag, group); if (rank=root) for (i=0; i < size; i++) MPI_SEND(inbuf[i], i, tag, group); MPI_WAIT(handle, return\_status); \end{verbatim} \mpifunc{MPI\_SCATTERC( inbuf, outbuf, len, group, root)} {\tt MPI\_SCATTERC} behaves like {\tt MPI\_SCATTER} restricted to block buffers, and with the additional restriction that all output buffers have the same length. The input buffer block of the root process is partitioned into {\tt n} consecutive blocks, each consisting of {\tt len} words. The {\tt i}-th block is sent to the {\tt i}-th process in the group and stored in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group, len}, and {\tt root}. \begin{description} \item[IN inbuf] first entry in input buffer -- significant only at root (choice). \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries to be stored in output buffer (integer) \item[IN group] handle \item[IN root] rank of sending process (integer) \end{description} {\tt MPI\_SCATTERC( inbuf, outbuf, outlen, group, root) } is \begin{verbatim} MPI_GSIZE( &size, group); MPI_RANK( &rank, group); MPI_IRECVC( handle, outbuf, outlen, root, tag, group); if (rank=root) for (i=0; i < size; i++) { MPI_SENDC(inbuf, outlen, i, tag, group, return\_status); inbuf += outlen; } MPI_WAIT(handle); \end{verbatim} \subsubsection*{All-to-all scatter} \mpifunc{MPI\_ALLSCATTER( list\_of\_inbufs, outbuf, group, return\_status)} Each process in the group sends its {\tt i}-th buffer in its input buffer list to the process with rank {\tt i} (itself included); each process concatenates the incoming messages in its output buffer, in the order of the senders' ranks. The routine is called by all members of the group using the same arguments for {\tt group}. \begin{description} \item[IN list\_of\_inbufs] list of buffer descriptor handles \item[OUT outbuf] buffer descriptor handle \item[IN group] handle \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_ALLSCATTERC( inbuf, outbuf, len, group)} {\tt MPI\_ALLSCATTERC} behaves like {\tt MPI\_ALLSCATTER} restricted to block buffers, and with the additional restriction that all blocks sent from one process to another have the same length. The input buffer block of each process is partitioned into {\tt n} consecutive blocks, each consisting of {\tt len} words. The {\tt i}-th block is sent to the {\tt it}-th process in the group. Each process concatenates the incoming messages, in the order of the senders' ranks, and store them in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt len}. \begin{description} \item[IN inbuf] first entry in input buffer (choice). root (integer) \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries sent from each process to each other (integer). \item[IN group] handle \end{description} {\tt MPI\_ALLSCATTERC( inbuf, outbuf, len, group)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); for (i=0; i < rank; i++) { MPI_IRECVC(recv_handles[i], outbuf, len, tag, group); outbuf += len; } for (i=0; i < size; i++) { MPI_ISENDC(send_handle[i], inbuf, len, i, tag, group); inbuf += len; } MPI_WAITALL(send_handle); MPI_WAITALL(recv_handle); \end{verbatim} \subsubsection*{All-to-all broadcast} \mpifunc{MPI\_ALLCAST( inbuf, outbuf, group, return\_status)} Each process in the group broadcasts its input buffer to all processes (including itself); each process concatenates the incoming messages in its output buffer, in the order of the senders' ranks. The routine is called by all members of the group using the same arguments for {\tt group}. \begin{description} \item[IN inbuf] buffer descriptor handle for input buffer \item[OUT outbuf] buffer descriptor handle for output buffer \item[IN group] handle \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_ALLCASTC( inbuf, outbuf, len, group)} {\tt MPI\_ALLCASTC} behaves like {\tt MPI\_ALLCAST} restricted to block buffers, and with the additional restriction that all blocks sent from one process to another have the same length. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt len}. \begin{description} \item[IN inbuf] first entry in input buffer (choice). root (integer) \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries sent from each process to each other (including itself). \item[IN group] group handle \end{description} {\tt MPI\_ALLCASTC( inbuf, outbuf, len, group)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); for (i=0; i < rank; i++) { MPI_IRECVC(recv_handles[i], outbuf, len, tag, group); outbuf += len; } for (i=0; i < size; i++) { MPI_ISENDC(send_handle[i], inbuf, len, i, tag, group); } MPI_WAITALL(send_handle); MPI_WAITALL(recv_handle); \end{verbatim} \section{Global Compute Operations} \subsubsection*{Reduce} \mpifunc{MPI\_REDUCE( inbuf, outbuf, group, root, op)} Combines the values provided in the input buffer of each process in the group, using the operation {\tt op}, and returns the combined value in the output buffer of the process with rank {\tt root}. Each process can provide one value, or a sequence of values, in which case the combine operation is executed pointwise on each entry of the sequence. For example, if the operation is {\tt max} and input buffers contains two floating point numbers, then outbuf(1) $=$ global max(inbuf(1)) and outbuf(2) $=$ global max(inbuf(2)). All input buffers should define sequences of equal length of entries of types that match the type of the operands of {\tt op}. The output buffer should define a sequence of the same length of entries of types that match the type of the result of {\tt op}. (Note that, here as for all other communication operations, the type of entries inserted in a message depend on the information provided by the input buffer descriptor, and not on the declarations of these variables in the calling program. The types of the variables in the calling program need not match the types defined by the buffer descriptor, but in such case the outcome of a reduce operation may be implementation dependent.) The operation defined by {\tt op} is associative and commutative, and the implementation can take advantage of associativity and commutativity in order to change order of evaluation. The routine is called by all group members using the same arguments for {\tt group, root} and {\tt op}. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer -- significant only at root \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} We list below the operations which are supported for Fortran, each with the corresponding value of the {\tt op} parameter. \begin{description} \item[MPI\_IMAX] integer maximum \item[MPI\_RMAX] real maximum \item[MPI\_DMAX] double precision real maximum \item[MPI\_IMIN] integer minimum \item[MPI\_RMIN] real minimum \item[MPI\_DMIN] double precision real minimum \item[MPI\_ISUM] integer sum \item[MPI\_RSUM] real sum \item[MPI\_DSUM] double precision real sum \item[MPI\_CSUM] complex sum \item[MPI\_DCSUM] double precision complex sum \item[MPI\_IPROD] integer product \item[MPI\_RPROD] real product \item[MPI\_DPROD] double precision real product \item[MPI\_CPROD] complex product \item[MPI\_DCPROD] double precision complex product \item[MPI\_AND] logical and \item[MPI\_IAND] integer (bit-wise) and \item[MPI\_OR] logical or \item[MPI\_IOR] integer (bit-wise) or \item[MPI\_XOR] logical xor \item[MPI\_IXOR] integer (bit-wise) xor \item[MPI\_MAXLOC] rank of process with maximum integer value \item[MPI\_MAXRLOC] rank of process with maximum real value \item[MPI\_MAXDLOC] rank of process with maximum double precision real value \item[MPI\_MINLOC] rank of process with minimum integer value \item[MPI\_MINRLOC] rank of process with minimum real value \item[MPI\_MINDLOC] rank of process with minimum double precision real value \end{description} \mpifunc{MPI\_REDUCEC( inbuf, outbuf, len, group, root, op)} Is same as {\tt MPI\_REDUCE}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer -- significant only at root \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} \mpifunc{MPI\_USER\_REDUCE( inbuf, outbuf, group, root, function)} Same as the reduce operation function above except that a user supplied function is used. {\tt function} is an associative and commutative function with two arguments. The types of the two arguments and of the returned value of the function, and the types of all entries in the input and output buffers all agree. The output buffer has the same length as the input buffer. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer -- significant only at root \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN function] user provided function \end{description} \mpifunc{MPI\_USER\_REDUCEC( inbuf, outbuf, len, group, root, function)} Is same as {\tt MPI\_\_USER\_REDUCE}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer -- significant only at root \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} \discuss{ Do we also want a version of reduce that broadcasts the result to all processes in the group? (This can be achieved by a reduce followed by a broadcast, but a combined function may be somewhat more efficient.) These would be respectively: \mpifunc{MPI\_GOP( inbuf, outbuf, group, op)} \mpifunc{MPI\_GOPC( inbuf, outbuf, len, group, op)} \mpifunc{MPI\_USER\_GOP( inbuf, outbuf, group, function)} \mpifunc{MPI\_USER\_GOPC( inbuf, outbuf, len, group, function)} Do we want a user provided {\em function} (two IN parameters, one OUT value), or a user provided procedure that overwrites the second input (ie. one IN param, one INOUT param, the equivalent of C {\tt a op= b} type assignment)? The second choice may allow a more efficient implementation, without changing the semantics of the MPI call. } \subsubsection*{Scan} \mpifunc{ MPI\_SCAN( inbuf, outbuf, group, op )} MPI\_SCAN is used to perform a parallel prefix with respect to an associative reduction operation on data distributed across the group. The operation returns in the output buffer of the process with rank {\tt i} the reduction of the values in the input buffers of processes with ranks {\tt 0,...,i}. The type of operations supported and their semantic, and the constraints on input and output buffers are as for {\tt MPI\_REDUCE}. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer \item[IN group] handle to group \item[IN op] operation (status) \end{description} \mpifunc{ MPI\_SCANC( inbuf, outbuf, len, group, op )} Same as {\tt MPI\_SCAN}, restricted to block buffers. \begin{description} \item[IN inbuf] first input buffer element (choice) \item[OUT outbuf] first output buffer element (choice) \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN op] operation (status) \end{description} \mpifunc{ MPI\_USER\_SCAN( inbuf, outbuf, group, function )} Same as the scan operation function above except that a user supplied function is used. {\tt function} is an associative and commutative function with two arguments. The types of the two arguments and of the returned values all agree. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer \item[IN group] handle to group \item[IN function] user provided function \end{description} \mpifunc{MPI\_USER\_SCANC( inbuf, outbuf, len, group, function)} Is same as {\tt MPI\_USER\_SCAN}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN function] user provided function \end{description} \discuss{ Do we want scan operations executed by segments? (The HPF definition of prefix and suffix operation might be handy -- in addition to the scanned vector of values there is a mask that tells where segments start and end.) } \section{Correctness} \discuss{ This is still very preliminary} The semantics of the collective communication operations can be derived from their operational definition in terms of point-to-point communication. It is assumed that messages pertaining to one operation cannot be confused with messages pertaining to another operation. Also messages pertaining to two distinct occurrences of the same operation cannot be confused, if the two occurrences have distinct parameters. The relevant parameters for this purpose are {\tt group}, {\tt root} and {\tt op}. messages pertaining to another occurrence of the same operation, with different parameters. The implementer can, of course, use another, more efficient implementation, as long as it has the same effect. \discuss{ This statement does not yet apply to the current, incomplete and somewhat careless definitions I provided in this draft. The definition above means that messages pertaining to a collective communication carry information identifying the operation itself, and the values of the {\tt group} and, where relevant, {\tt root} or {\tt op} parameters. Is this acceptable? } A few examples: \begin{verbatim} MPI_BCAST(buf, len, group, 0); MPI_BCAST(buf, len, group, 1); \end{verbatim} Two consecutive broadcasts, in the same group, with the same tag, but different roots. Since the operations are distinguishable, messages from one broadcast cannot be confused with messages from the other broadcast; the program is safe and will execute as expected. \begin{verbatim} MPI_BCAST(buf, len, group, 0); MPI_BCAST(buf, len, group, 0); \end{verbatim} Two consecutive broadcasts, in the same group, with the same tag and root. Since point-to-point communication preserves the order of messages here, too, messages from one broadcast will not be confused with messages from the other broadcast; the program is safe and will execute as intended. \begin{verbatim} MPI_RANK(&rank, group) if (rank==0) { MPI_BCASTC(buf, len, group, 0); MPI_SENDC(buf, len, 2, tag, group); } elseif (rank==1) { MPI_RECVC(buf, len, MPI_DONTCARE, tag, group); MPI_BCASTC(buf, len, group, 0); MPI_RECVC(buf, len, MPI_DONTCARE, tag, group); } else { MPI_SENDC(buf, len, 2, tag, group); MPI_BCASTC(buf, len, group, 0); } \end{verbatim} Process zero executes a broadcast followed by a send to process one; process two executes a send to process one, followed by a broadcast; and process one executes a receive, a broadcast and a receive. A possible outcome is for the operations to be matched as illustrated by the diagram below. \begin{verbatim} 0 1 2 / - > receive / - send / / broadcast / broadcast / broadcast / / send - receive < - \end{verbatim} The reason is that broadcast is not a synchronous operation; the call at a process may return before the other processes have entered the broadcast. Thus, the message sent by process zero can arrive to process one before the message sent by process two, and before the call to broadcast on process one. \end{document} From owner-mpi-comm@CS.UTK.EDU Tue Apr 20 16:21:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08359; Tue, 20 Apr 93 16:21:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03662; Tue, 20 Apr 93 16:19:33 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 20 Apr 1993 16:19:30 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03654; Tue, 20 Apr 93 16:19:25 -0400 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16360; Tue, 20 Apr 1993 16:19:21 -0400 Date: Tue, 20 Apr 1993 16:19:21 -0400 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9304202019.AA16360@gstws.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Postscript of Collective Communication chapter of MPI %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: cc.dvi %%Pages: 17 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 400 400 @start /Fa 9 118 400 360 dfs[<7FFFF0FFFFF8FFFFF87F FFF00000000000000000000000007FFFF0FFFFF8FFFFF87FFFF0>21 12 126 148 29 61 D[<1FF0003FFC007FFE00781F00300780000380000380007F8007FF801FFF80 3FC3807C0380F00380E00380E00380E00380F007807C1F803FFFFC1FFDFC07F0FC>22 21 125 148 29 97 D[22 30 127 157 29 I[<00F8FC03FFFE07FFFE0F8F8C0E03801E03C01C01C01C01C01C01C01E03C00E03800F8F80 0FFF001FFE001CF8001C00001C00001E00000FFF801FFFF03FFFF87C00FC70001CF0001EE0000E E0000EE0000EF0001E78003C3F01F81FFFF00FFFE001FF00>23 33 127 148 29 103 D[<01F00007FC001FFF003E0F803C07807803C07001C0E000E0E000E0E000E0E000 E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF0007FC0001F000>19 21 125 148 29 111 D[22 32 127 148 29 I[22 21 126 148 29 114 D[<00C00001C00001C00001C00001C00001C00001C0007FFFE0FFFFE0FF FFE001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C07001C07001 C07001C0F001E1E000FFE0007FC0003F00>20 28 127 155 29 116 D[23 21 127 148 29 I E /Fb 7 118 400 360 dfs[<007C01C2030307070E0F1C0F3C003800780078007800F000F000F000F000F001 70037006301C18380FC0>16 21 123 148 26 99 D[<00007C0000CE00019E00039E00038C0003 00000700000700000700000700000E00000E00000E00000E0001FFF001FFF0001C00001C00001C 00001C00001C0000380000380000380000380000380000700000700000700000700000700000E0 0000E00000E00000E00001C00001C00001C00001C000038000738000F30000F300006600003C00 00>23 45 130 162 17 102 D[<00E000E001E000C00000000000000000000000000000000000 001E00330063806380C380C700C70007000E000E000E001C001C001C40386038C070C070803180 31001E00>11 34 124 161 17 105 D[<1E07803318E063B06063E070C3C070C38070C3807007 00E00700E00700E00700E00E01C00E01C00E03820E03831C03861C07061C070C1C030838031818 01E0>24 21 124 148 31 110 D[<007C0001C6000303000603800E03C01C03C03C03C03803C0 7803C07803C07803C0F00780F00780F00780F00F00F00E00701E00701C003038001860000F8000 >18 21 123 148 28 I[<006000E000E000E000E001C001C001C001C00380FFF8FFF803800700 0700070007000E000E000E000E001C001C001C101C18383038303860186018C00F00>13 31 124 158 19 116 D[<0F003011807021C07061C0E0C1C0E0C380E0C380E00381C00701C007 01C00701C00E03800E03800E03840E03860E070C0C070C0E070C0E0B1806131003E1E0>23 21 124 148 30 I E /Fc 2 61 438 432 dfs[<78FCFCFEFE7A020202020404040810102040> 7 18 123 133 17 59 D[<00000000E000000003E00000000FC00000003F00000000FC00000003 F00000000FC00000003F00000000FC00000003F00000000FC00000003F00000000FC00000003F0 0000000FC00000003F00000000FC00000000F000000000FC000000003F000000000FC000000003 F000000000FC000000003F000000000FC000000003F000000000FC000000003F000000000FC000 000003F000000000FC000000003F000000000FC000000003E000000000E0>35 35 123 159 47 I E /Fd 46 124 400 360 dfs[<000FC0000078300000E0080001803C000380 7C0007007C0007007C0007003800070000000700000007000000070000000700000007000000FF FFFC00FFFFFC0007003C0007001C0007001C0007001C0007001C0007001C0007001C0007001C00 07001C0007001C0007001C0007001C0007001C0007001C0007001C0007001C0007001C007FF1FF C07FF1FFC0>26 35 128 162 31 12 D[<000FC03F00007031E0C000E00B802001803E00F00380 7E01F007007C01F007007C01F007003C00E007001C000007001C000007001C000007001C000007 001C000007001C0000FFFFFFFFF0FFFFFFFFF007001C00F007001C007007001C007007001C0070 07001C007007001C007007001C007007001C007007001C007007001C007007001C007007001C00 7007001C007007001C007007001C007007001C007007001C00707FF1FFC7FF7FF1FFC7FF>40 35 128 162 47 14 D[<701CF83EFC3FFC3F741D04010401040104010802080210041004200840 10>16 15 126 162 28 34 D[<001000200040008001000300060004000C001800180018003000 300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000E000600060 00600070003000300030001800180018000C0004000600030001000080004000200010>12 50 125 164 21 40 D[<800040002000100008000C0006000200030001800180018000C000C000 C000E0006000600060007000700070007000700070007000700070007000700070006000600060 00E000C000C000C00180018001800300020006000C0008001000200040008000>12 50 125 164 21 I[<70F8FCFC7404040404080810102040>6 15 124 132 16 44 D[12 2 127 139 19 I[<70F8F8F870>5 5 124 132 16 I[<70F8F8F870000000000000000000000070F8F8F870>5 21 124 148 16 58 D[<07F000181C00200E00400700F00780F80780F80780F80780700780000F00000E0000 1C0000380000700000600000C00000800000800001800001000001000001000001000001000001 000000000000000000000000000000000003800007C00007C00007C000038000>17 35 125 162 27 63 D[<0007F008003FFC1800FC073801F001B803C000F8078000780F0000381E 0000183E0000183C0000187C0000087C00000878000008F8000000F8000000F8000000F8000000 F8000000F8000000F8000000F8000000780000007C0000087C0000083C0000083E0000081E0000 100F0000100780002003C0004001F0018000FC0700003FFE000007F000>29 34 125 161 40 67 D[32 34 126 161 42 I[28 34 126 161 37 70 D[32 34 126 161 41 72 D[14 34 126 161 20 I[41 34 126 161 51 77 D[32 34 126 161 41 I[<000FF00000781E0000E0070003C003C0078001E00F0000F01E0000781E0000783C00003C3C 00003C7C00003E7800001E7800001EF800001FF800001FF800001FF800001FF800001FF800001F F800001FF800001F7800001E7C00003E7C00003E3C00003C3E00007C1E0000781E0000780F0000 F0078001E003C003C000E0070000781E00000FF000>32 34 125 161 43 I[28 34 126 161 38 I[<7FFFFFFC7FFFFFFC7803C03C6003C00C4003C0044003C004C003C006C003C0068003C00280 03C0028003C0028003C0020003C0000003C0000003C0000003C0000003C0000003C0000003C000 0003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0 000003C0000003C0000003C00001FFFF8001FFFF80>31 34 126 161 40 84 D[32 34 126 161 41 I[<1FF000381C007C06007C07007C0380380380000380000380007F8007C3801E03803C0380 780380780380F00384F00384F00384F00784780B843C11C80FE0F0>22 21 126 148 28 97 D[<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E1F800E60E00E80300F00380E001C0E001E0E000E0E000F0E000F 0E000F0E000F0E000F0E000F0E000F0E000E0E001E0E001C0F00380C80700C60E0081F80>24 35 127 162 31 I[<01FE000707000C0F801C0F80380F80780700700000F00000F00000F00000 F00000F00000F00000F000007000007800403800401C00800C010007060001F800>18 21 126 148 24 I[<0000700007F00007F00000F0000070000070000070000070000070000070 00007000007000007000007001F8700706700E01701C00F0380070780070700070F00070F00070 F00070F00070F00070F00070F000707000707800703800701C00F00C017807067F01F87F>24 35 126 162 31 I[<01FC000707000C03801C01C03801C07800E07000E0F000E0FFFFE0F00000 F00000F00000F00000F000007000007800203800201C00400E008007030000FC00>19 21 127 148 24 I[<003E0000E30001C780038F80030F80070700070000070000070000070000 070000070000070000070000FFF800FFF800070000070000070000070000070000070000070000 0700000700000700000700000700000700000700000700000700000700007FF8007FF800>17 35 128 162 17 I[<01F078071D9C0E0E1C1C07001C07003C07803C07803C07803C07801C0700 1C07000E0E000F1C0019F0001000001000001800001C00001FFF000FFFE00FFFF03800F8600018 40001CC0000CC0000CC0000C6000186000183800700E01C001FE00>22 32 127 148 28 I[<0E000000FE000000FE0000001E0000000E0000000E0000000E0000000E000000 0E0000000E0000000E0000000E0000000E0000000E0000000E1F80000E60E0000E8070000F0038 000F0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E00 38000E0038000E0038000E0038000E0038000E003800FFE3FF80FFE3FF80>25 35 127 162 31 I[<1C003E003E003E001C00000000000000000000000000000000000E00FE00 FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC0> 10 34 127 161 16 I[<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E03FC0E03FC0E01E00E01800E02000E04000E08000E10000E38 000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E00E01F0FFE3FEFFE3FE>23 35 127 162 29 107 D[<0E00FE00FE001E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FF E0FFE0>11 35 127 162 16 I[<0E1FC07F00FE60E18380FE807201C01F003C00E00F003C00E0 0E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800 E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFEFFE3FF 8FFE>39 21 127 148 47 I[<0E1F8000FE60E000FE8070001F0038000F0038000E0038000E00 38000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E 0038000E0038000E003800FFE3FF80FFE3FF80>25 21 127 148 31 I[<00FC000703800E01C0 1C00E0380070780078700038F0003CF0003CF0003CF0003CF0003CF0003CF0003C700038780078 3800701C00E00E01C007038000FC00>22 21 127 148 28 I[<0E1F80FE60E0FE80700F00380E 001C0E001E0E001E0E000F0E000F0E000F0E000F0E000F0E000F0E000F0E001E0E001E0E001C0F 00380E80700E60E00E1F800E00000E00000E00000E00000E00000E00000E00000E0000FFE000FF E000>24 31 127 148 31 I[<01F8200704600E02601C01603801E07800E07800E0F000E0F000 E0F000E0F000E0F000E0F000E0F000E07000E07800E03801E01C01E00C02E0070CE001F0E00000 E00000E00000E00000E00000E00000E00000E00000E0000FFE000FFE>23 31 126 148 29 I[<0E1E00FE6300FE87801E87800F03000F00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E0000FFF000FFF000>17 21 127 148 22 I[<0FC4303C600CC00CC004C004E004F0007F803FF00FF800FC001E800E8006 C006C006C004E00CD81887E0>15 21 126 148 22 I[<02000200020002000200060006000600 0E001E003FF8FFF80E000E000E000E000E000E000E000E000E000E000E040E040E040E040E040E 040708030801F0>14 31 127 158 21 I[<0E003800FE03F800FE03F8001E0078000E0038000E 0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E003800 0E0038000E0078000E0078000700BC0003833F8000FC3F80>25 21 127 148 31 I[23 21 127 148 29 I[33 21 127 148 40 I[23 21 127 148 29 I[23 31 127 148 29 I[24 1 128 140 28 123 D E /Fe 29 118 400 360 dfs[<000C0038007000E001C003C0038007800F000F001E001E003E003C003C007C007C007C00 7800F800F800F800F800F800F800F800F800F800F800F80078007C007C007C003C003C003E001E 001E000F000F000780038003C001C000E000700038000C>14 49 124 164 24 40 D[14 49 125 164 24 I[<3C007E00FF00FF00FF80FF807F803D800180018003000300070006000C001C003800 2000>9 18 124 135 18 44 D[<3C7EFFFFFFFF7E3C0000000000003C7EFFFFFFFF7E3C>8 22 124 149 18 58 D[<0001FF0040001FFFC1C0007F80F3C001FC001FC003F0000FC007E00007 C00FC00003C01FC00003C03F800001C03F800001C07F800000C07F000000C07F000000C0FF0000 0000FF00000000FF00000000FF00000000FF00000000FF00000000FF00000000FF000000007F00 0000007F000000C07F800000C03F800000C03F800001C01FC00001800FC000018007E000030003 F000060001FC001C00007F807800001FFFE0000001FF0000>34 34 125 161 46 67 D[38 34 126 161 49 I[33 34 126 161 42 I[<0001FF0020001FFFE0E0007F 8079E001FC001FE003F80007E007E00003E00FC00001E01FC00001E03F800000E03F800000E07F 800000607F000000607F00000060FF00000000FF00000000FF00000000FF00000000FF00000000 FF00000000FF0007FFFEFF0007FFFE7F00000FE07F00000FE07F80000FE03F80000FE03F80000F E01FC0000FE00FE0000FE007E0000FE003F8000FE001FC001FE0007F8073E0001FFFE1E00001FF 8060>39 34 125 161 50 71 D[ 19 34 127 161 24 73 D[47 34 125 161 60 77 D[<0007FE0000003FFFC00000FE07F00003F801FC0007F000FE000FE0007F001FC0003F801F 80001F803F80001FC03F80001FC07F00000FE07F00000FE07F00000FE0FF00000FF0FF00000FF0 FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF07F00000F E07F80001FE07F80001FE03F80001FC01FC0003F801FC0003F800FE0007F0007F000FE0003F801 FC0000FE07F000003FFFC0000007FE0000>36 34 125 161 48 79 D[33 34 126 161 43 I[40 34 126 161 48 82 D[<01FE020007FFCE001F01FE003C007E003C001E0078000E0078000E00F8 000600F8000600FC000600FC000000FF000000FFF000007FFF80003FFFE0003FFFF8001FFFFC00 07FFFE0003FFFF00003FFF000001FF0000003F8000001F8000001F80C0000F80C0000F80C0000F 80E0000F00E0000F00F0001E00FC001C00FF807800E7FFF000807FC000>25 34 125 161 36 I[39 34 126 161 49 85 D[27 35 126 162 36 98 D[<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC00 00FC0000FC0000FC0000FC00007C00007E00007E00003F00301F00600FC0E007FF8000FE00>20 22 126 149 28 I[<00FE0007FF800F83E01F01E03E00F07E00F07C00F8FC00F8FC0078FFFFF8 FFFFF8FC0000FC0000FC0000FC00007E00007E00183E00381F00300F80F003FFC000FF00>21 22 126 149 29 101 D[<001F8000FFE001F1F003E3F007E3F00FC3F00FC1E00FC0000FC0000F C0000FC0000FC0000FC000FFFE00FFFE000FC0000FC0000FC0000FC0000FC0000FC0000FC0000F C0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0007FFC007FFC00> 20 35 126 162 20 I[<00FE0F8003FF9FC00F83E3C01F01F3C01E00F0003E00F8003E00F8003E 00F8003E00F8003E00F8001E00F0001F01F0000F83E0000BFF800008FE00001800000018000000 1C0000001FFFE0001FFFFC000FFFFF0007FFFF001FFFFF807C001FC078000FC0F80007C0F80007 C0F80007C07C000F803E001F001F807E000FFFFC0001FFE000>26 33 127 149 32 I[<0E003F807F807F807F807F803F800E00000000000000000000000000FF80FF801F80 1F801F801F801F801F801F801F801F801F801F801F801F801F801F801F801F801F80FFF0FFF0> 12 36 126 163 17 105 D[12 35 126 162 17 108 D[27 22 125 149 36 110 D[<00FF0007FFE00F81F01F00F83E007C7C003E7C003E7C003EFC003FFC003F FC003FFC003FFC003FFC003FFC003F7C003E7E007E3E007C1F00F80F81F007FFE000FF00>24 22 126 149 32 I[27 32 126 149 36 I[20 22 126 149 27 114 D[<07F9801FFF80380780700380F00180F00180F80000FF0000FFF8007FFE003FFF001FFF8007 FF80003FC0C007C0C003C0E003C0E003C0F00380FC0F00EFFE00C3F800>18 22 126 149 26 I[<00C00000C00000C00000C00001C00001C00003C00007C0000FC0001FC000 FFFF00FFFF000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC000 0FC1800FC1800FC1800FC1800FC18007C18007E30003FE0000FC00>17 32 127 159 24 I[27 22 125 149 36 I E /Ff 65 126 438 432 dfs[<00F0000003F8000003FC000007FC0000071E00000F0E00000E0E00000E 0E00000E0E00000E0E00000E0E00000E1E7FC00E3CFFC00E7CFFC007787FC007F0380007F03800 07E0380007C070000F8070001F8070003FC0E0007DC0E00078E0E00078E1C000F071C000F07B80 00F03B8000F01F0000F01F01C0F00E01C0781F81C0787FC3C03FFBFF803FF1FF801FE0FF000780 3C00>26 37 126 164 31 38 D[<000F001F003E007C00F801F003E007C00F800F001E001E003C 003C003C00780078007800F000F000F000F000F000F000F000F000F000F000F000780078007800 3C003C003C001E001E000F000F8007C003E001F000F8007C003F001F000F>16 47 119 169 31 40 D[<7000F8007C003E001F000F8007C003E001F000F000780078003C003C00 3C001E001E001E000F000F000F000F000F000F000F000F000F000F000F001E001E001E003C003C 003C0078007800F001F003E007C00F801F003E007C00F8007000>16 47 123 169 31 I[<000E0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000 001F0000001F0000001F00007FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF80001F0000001F00 00001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000000E0000>26 27 126 159 31 43 D[<1C003F007F007F807F803F801F8007800F800F001F007E00FC00F80060 00>9 15 117 134 31 I[<7FFFFEFFFFFFFFFFFFFFFFFF7FFFFE>24 5 125 148 31 I[<387CFEFEFE7C38>7 7 116 134 31 I[<00000E00001F00001F00003F00003E0000 7E00007C00007C0000FC0000F80001F80001F00003F00003E00007E00007C0000FC0000F80000F 80001F80001F00003F00003E00007E00007C0000FC0000F80001F80001F00001F00003F00003E0 0007E00007C0000FC0000F80001F80001F00003F00003E00003E00007E00007C0000FC0000F800 00F80000700000>24 47 125 169 31 I[<007E0001FF8003FFC007FFE00FC3F01F00F81E0078 3E007C3C003C7C003E78001E78001E78001EF0000FF0000FF0000FF0000FF0000FF0000FF0000F F0000FF0000FF0000FF8001F78001E78001E78001E7C003E3C003C3E007C1F00F81F81F80FC3F0 07FFE003FFC001FF80007E00>24 37 125 164 31 I[<00700000700000F00000F00001F00003 F00007F0007FF000FFF000FEF000F8F00000F00000F00000F00000F00000F00000F00000F00000 F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000 F00000F0007FFFE0FFFFF0FFFFF07FFFE0>20 37 122 164 31 I[<00FE0003FFC00FFFE01FFF F83E03FC7C007C78003EF0001EF0000FF8000FF8000F70000F00000F00000F00001E00001E0000 1E00003C00007C0000F80001F00003E00007C0000F80001F00003E00007C0001F80003F00007C0 000F800F1F000F3E000F7FFFFFFFFFFFFFFFFF7FFFFF>24 37 125 164 31 I[<1C3E7F7F7F3E1C0000000000000000000000001C3E7E7F7F3F1F0F1F1E3E7CF8F060>8 34 117 153 31 59 D[<00000E00001F00007F0000FF0003FE0007FC001FF0003FE000FF8001FF 0007FC000FF8003FE0007FC000FF0000FE0000FF00007FC0003FE0000FF80007FC0001FF0000FF 80003FE0001FF00007FC0003FE0000FF00007F00001F00000E>24 31 125 161 31 I[<7FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF800000000000000000000000000000 0000000000007FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF80>26 15 126 153 31 I[<700000F80000FE0000FF00007FC0003FE0000FF80007FC0001FF0000FF80003F E0001FF00007FC0003FE0000FF00007F0000FF0003FE0007FC001FF0003FE000FF8001FF0007FC 000FF8003FE0007FC000FF0000FE0000F80000700000>24 31 125 161 31 I[<001E0000003F0000003F0000003F00000073800000738000007380000073800000738000 00F3C00000F3C00000F3C00000E1C00001E1E00001E1E00001E1E00001E1E00001E1E00003C0F0 0003C0F00003C0F00003C0F00007C0F80007FFF80007FFF80007FFF80007FFF8000F003C000F00 3C000F003C000F003C000F003C001E001E00FFC0FFC0FFE1FFC0FFE1FFC0FFC0FFC0>26 37 126 164 31 65 D[25 37 126 164 31 I[<001F81C0007FE1C001FFFBC003FFFFC007F03FC00FC01FC01F80 0FC01F0007C03E0007C03C0003C07C0003C0780003C0780003C078000000F0000000F0000000F0 000000F0000000F0000000F0000000F0000000F0000000F00000007800000078000000780003C0 7C0003C03C0003C03E0003C01F0007801F8007800FC00F0007F03F0003FFFE0001FFFC00007FF0 00001FC000>26 37 126 164 31 I[<7FFF8000FFFFE000FFFFF8007FFFFC000F00FE000F003E 000F001F000F000F800F000F800F0007800F0007C00F0003C00F0003C00F0003E00F0001E00F00 01E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0003E00F0003C00F 0003C00F0003C00F0007C00F000F800F000F800F001F000F003E000F00FE007FFFFC00FFFFF800 FFFFF0007FFF8000>27 37 127 164 31 I[27 37 126 164 31 I[26 37 126 164 31 I[<003F070000FFC70001FFEF0007FF FF0007E0FF000F807F001F003F001E001F003E001F003C000F007C000F0078000F0078000F00F8 000000F0000000F0000000F0000000F0000000F0000000F0000000F001FFC0F001FFE0F001FFE0 F801FFC078000F0078000F0078001F003C001F003E001F001E001F001F003F000F807F0007E0FF 0007FFFF0001FFEF0000FFCF00003F0F00>27 37 126 164 31 I[<7FE07FE0FFF0FFF0FFF0FF F07FE07FE00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F000F000F000F000F000F000FFFFF000FFFFF000FFFFF000FFFFF000F000F000F000F000F 000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F007FE07FE0FFF0FFF0FFF0FFF07FE07FE0>28 37 127 164 31 I[<7FFFF8FFFFFCFFFF FC7FFFF80078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078007FFFF8FFFFFCFFFFFC7FFFF8>22 37 124 164 31 I[<7FC07FC0FFE0FFC0FFE0FFC07FC07FC00E001C000E0038000E0078000E00F0000E00 E0000E01C0000E03C0000E0780000E0700000E0E00000E1E00000E3C00000E3E00000E7F00000E F700000EE780000FC380000FC1C0000F81C0000F00E0000E00E0000E0070000E0070000E003800 0E0038000E001C000E001C000E000E000E000E007FC01FE0FFE03FE0FFE03FE07FC01FE0>27 37 127 164 31 75 D[<7FFC0000FFFE0000FFFE00007FFC000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000780000007800000 078000000780000007800000078000000780000007800000078000000780000007800000078001 80078003C0078003C0078003C0078003C0078003C0078003C07FFFFFC0FFFFFFC0FFFFFFC07FFF FFC0>26 37 126 164 31 I[28 37 127 164 31 I[<7F00FF80FF81FFC0FF81FFC07FC0FF800EC01C000EC01C 000EE01C000E601C000E601C000E701C000E701C000E301C000E381C000E381C000E381C000E18 1C000E1C1C000E1C1C000E0C1C000E0E1C000E0E1C000E061C000E071C000E071C000E071C000E 031C000E039C000E039C000E019C000E019C000E01DC000E00DC000E00DC007FC0FC00FFE07C00 FFE07C007FC03C00>26 37 126 164 31 I[<03FFC01FFFF83FFFFC3FFFFC7E007E7C003E7800 1E78001EF8001FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF000 0FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF8001F78001E78001E7C003E7F00 FE3FFFFC3FFFFC1FFFF803FFC0>24 37 125 164 31 I[25 37 126 164 31 I[<7FFF0000FFFFE000FFFFF000 7FFFF8000F01FC000F007E000F001E000F001F000F000F000F000F000F000F000F000F000F001F 000F001E000F007E000F01FC000FFFF8000FFFF0000FFFE0000FFFF0000F01F8000F0078000F00 7C000F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C780F003C780F 003E787FE01FF0FFF01FF0FFF00FE07FE003C0>29 37 127 164 31 82 D[<01FC1C07FF9C0FFFFC3FFFFC3E03FC7C00FC78007CF0003CF0003CF0003CF0003CF0000078 00007C00003E00003FE0001FFE0007FFC001FFF0001FF80001FC00007C00001E00001E00000F00 000F70000FF0000FF0000FF0001FF8001EFC003EFF00FCFFFFF8FFFFF0E3FFE0E0FF80>24 37 125 164 31 I[<7FFFFFC0FFFFFFC0FFFFFFC0FFFFFFC0F01E03C0F01E03C0F01E03C0F01E 03C0F01E03C0F01E03C0001E0000001E0000001E0000001E0000001E0000001E0000001E000000 1E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000 001E0000001E0000001E0000001E0000001E0000001E000003FFF00007FFF80007FFF80003FFF0 00>26 37 126 164 31 I[28 37 127 164 31 I[<7FC03FE0FFE07FF0FFE07FF07FC03FE00F000F000F000F00 0F000F000F000F0007801E0007801E0007801E0007801E0003C03C0003C03C0003C03C0003C03C 0003E07C0001E0780001E0780001E0780001E0780000F0F00000F0F00000F0F00000F0F0000070 E0000079E0000079E0000079E0000039C0000039C0000039C0000039C000001F8000001F800000 1F8000000F0000>28 37 127 164 31 I[28 37 127 164 31 I[<3FFFFF7FFFFF7FFFFF7FFFFF78001E78003C 78007C7800787800F07801F00001E00003E00003C0000780000F80000F00001E00003E00003C00 007C0000780000F00001F00001E00003C00007C00007800F0F800F0F000F1E000F3E000F3C000F 78000FFFFFFFFFFFFFFFFFFFFFFFFF>24 37 125 164 31 90 D[16 47 116 169 31 I[<700000F80000F80000FC00007C00007E00003E00003E0000 3F00001F00001F80000F80000FC00007C00007E00003E00003F00001F00001F00001F80000F800 00FC00007C00007E00003E00003F00001F00001F80000F80000F80000FC00007C00007E00003E0 0003F00001F00001F80000F80000FC00007C00007C00007E00003E00003F00001F00001F00000E >24 47 125 169 31 I[16 47 126 169 31 I[<7FFFFEFFFFFFFFFFFFFFFFFF7FFFFE>24 5 125 126 31 95 D[<07FC00001FFF00003FFFC0003FFFE0003E03F0001C01F0000000F8000000780000007800 00007800007FF80003FFF8000FFFF8003FE078007E00780078007800F0007800F0007800F00078 00F00078007800F8007E03F8003FFFFFE03FFFFFE00FFE3FE003F00FE0>27 26 125 153 31 97 D[27 37 127 164 31 I[<007FC001FFF007FFF80FFFF81F80F83E00703C00007800007800 00F80000F00000F00000F00000F00000F00000F00000F800007800007C00783E00783F00F81FC1 F00FFFE007FFE001FF80007E00>21 26 123 153 31 I[<0007FC000007FC000007FC000007FC 0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000FC3C0003FF3C0007FF BC000FFFFC001F81FC003E00FC003C007C007C003C0078003C00F8003C00F0003C00F0003C00F0 003C00F0003C00F0003C00F0003C00F8003C0078007C0078007C003C00FC003E01FC001F83FC00 1FFFFFE007FFBFE003FE3FE000F83FE0>27 37 126 164 31 I[<007F0001FFC007FFE00FFFF0 1F81F83F00783C003C7C003C78001E78001EFFFFFEFFFFFEFFFFFEFFFFFEF00000F00000780000 7800007C001E3E001E1F803E1FE07C0FFFF803FFF001FFE0003F80>23 26 125 153 31 I[<0001F80007FC000FFE001FFE003E3E007C1C0078000078000078000078000078 007FFFFCFFFFFCFFFFFCFFFFFC0078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078007FFFF87FFFF87FFFF87FFF F8>23 37 126 164 31 I[<00FC0F8003FF3FC007FFFFE00FFFFFE00F87E1C01F03E0001E01E0 003C00F0003C00F0003C00F0003C00F0003C00F0003C00F0001E01E0001F03E0000F87C0000FFF C0001FFF80001FFF00001CFC00001C0000001C0000000E0000000FFFE0001FFFF8003FFFFE003C 001F007800078070000380E00001C0E00001C0E00001C0E00001C0700003807C000F803F003F00 1FFFFE000FFFFC0003FFF000007F8000>27 40 126 153 31 I[28 37 127 164 31 I[<00300000780000FC00 00FC000078000030000000000000000000000000000000000000007FFC007FFC007FFC007FFC00 003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C007FFFFCFFFFFEFFFFFE7FFFFC>23 38 124 165 31 I[ 27 37 126 164 31 107 D[24 37 125 164 31 I[29 26 128 153 31 I[28 26 127 153 31 I[<00FC0003FF0007FF801FFFE01F87E03E01F07C00F87800787800 78F0003CF0003CF0003CF0003CF0003CF0003CF0003CF8007C7800787C00F87C00F83E01F01F87 E01FFFE007FF8003FF0000FC00>22 26 124 153 31 I[27 39 127 153 31 I[26 26 126 153 31 114 D[<03FC700FFFF03FFFF07FFFF07C03F0F801F0F000F0F000F0F000F07C 00007FE0001FFF0007FFC000FFE00003F00000F870003CF0003CF0003CF8003CFC007CFF01F8FF FFF0FFFFF0E7FFC0E1FE00>22 26 124 153 31 I[<0070000000F0000000F0000000F0000000 F0000000F0000000F000007FFFFE00FFFFFE00FFFFFE00FFFFFE0000F0000000F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F007 8000F0078000F0078000F0078000F80F00007C1F00007FFE00003FFC00001FF8000007E000>25 33 127 160 31 I[28 26 127 153 31 I[<7FE07FE0FFF0FFF0FFF0FFF07FE07FE007000E0007000E0007801E000380 1C0003801C0003C03C0001C0380001C0380001E0780000E0700000E0700000E070000070E00000 70E0000070E0000039C0000039C0000039C000001F8000001F8000001F8000000F0000>28 26 127 153 31 I[<7FF0FFC07FF1FFE07FF1FFE07FF0FFC003C0380001E0700000E0F0000070 E0000079C000003FC000001F8000000F0000000E0000000F0000001F8000003B80000039C00000 70E00000E0F00001E0700001C0380003801C007FE07FE0FFF0FFF0FFF0FFF07FE07FE0>28 26 127 153 31 120 D[<3FFFFF807FFFFF807FFFFF807FFFFF8078003F0078007E007800FC00 7801F8000003F0000007E000000FC000001F8000003F0000007E000000FC000001F8000003F000 0007E007800FC007801F8007803F0007807E000780FFFFFF80FFFFFF80FFFFFF80FFFFFF80>25 26 126 153 31 122 D[<00007F0003FF000FFF001FFF001F00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007C0000F8007FF800FF E000FFC000FFE0007FF80000F800007C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00001F00001FFF000FFF0003FF00007F>24 47 125 169 31 I[<7E0000FFC000FFF000FFF80000F800003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C00003C00003C00003C00003C00003E00001F00001FFE0007FF 0003FF0007FF001FFE001F00003E00003C00003C00003C00003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C00003C0000F800FFF800FFF000FFC0007E0000>24 47 125 169 31 125 D E /Fg 47 123 438 432 dfs[<0001FF81FE00001FFFEFFF80007F80FF 87C000FC00FE0FE001F801FE0FE003F801FC0FE007F001FC0FE007F001FC07C007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC0000FFFFFFFF F800FFFFFFFFF800FFFFFFFFF80007F001FC000007F001FC000007F001FC000007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC 000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC00007FFF1FFFE0007FFF1FFFE0007FFF1FFF E000>43 42 127 169 41 11 D[<00030007000E001C0038007000F001E003C003C007C007800F 800F001F001F003F003E003E007E007E007E007C007C00FC00FC00FC00FC00FC00FC00FC00FC00 FC00FC00FC00FC007C007C007E007E007E003E003E003F001F001F000F000F80078007C003C003 C001E000F000700038001C000E00070003>16 60 122 172 27 40 D[<8000C000E00070003800 1C001E000F000780078007C003C003E001E001F001F001F800F800F800FC00FC00FC007C007C00 7E007E007E007E007E007E007E007E007E007E007E007E007C007C00FC00FC00FC00F800F801F8 01F001F001E003E003C007C0078007800F001E001C0038007000E000C0008000>15 60 123 172 27 I[<1C007F007F00FF80FFC0FFC07FC07FC01CC000C000C00180018001800300 030006000C00180030002000>10 21 123 136 19 44 D[17 6 127 144 23 I[<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE007FFFFE7FFFFE7FFFFE>23 39 123 166 34 49 D[<000003800000000007C000000000 07C0000000000FE0000000000FE0000000000FE0000000001FF0000000001FF0000000003FF800 0000003FF8000000003FF80000000073FC0000000073FC00000000F3FE00000000E1FE00000000 E1FE00000001C0FF00000001C0FF00000003C0FF80000003807F80000007807FC0000007003FC0 000007003FC000000E003FE000000E001FE000001E001FF000001C000FF000001FFFFFF000003F FFFFF800003FFFFFF80000780007FC0000700003FC0000700003FC0000E00001FE0000E00001FE 0001E00001FF0001C00000FF0001C00000FF00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE>47 41 126 168 53 65 D[41 41 125 168 50 I[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE00007F80 03FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803FC000 0003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF8000000000 FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F8000 0000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE000000700 0FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E000000FFF FFC0000003FFFF000000003FF80000>41 41 124 168 51 I[ 45 41 125 168 54 I[39 41 125 168 46 I[36 41 125 168 44 I[<00007FE003000003FFFC0700001FFFFF0F00003FF00FFF0000FF 8001FF0001FE0000FF0003F800003F0007F000003F000FF000001F001FE000000F001FE000000F 003FC000000F003FC0000007007FC0000007007F80000007007F8000000000FF8000000000FF80 00000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000 00FF8001FFFFF87F8001FFFFF87F8001FFFFF87FC00000FF003FC00000FF003FC00000FF001FE0 0000FF001FE00000FF000FF00000FF0007F00000FF0003F80000FF0001FE0000FF0000FF8001FF 00003FF007BF00001FFFFF1F000003FFFE0F0000007FF00300>45 41 124 168 55 I[47 41 125 168 55 I[22 41 126 168 26 I[47 41 125 168 55 75 D[34 41 125 168 42 I[58 41 125 168 66 I[47 41 125 168 55 I[<0000FFE000000007FFFC0000003FC07F8000007F001FC00001FC0007F00003F80003 F80007F00001FC000FF00001FE001FE00000FF001FE00000FF003FC000007F803FC000007F807F C000007FC07F8000003FC07F8000003FC07F8000003FC0FF8000003FE0FF8000003FE0FF800000 3FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF 8000003FE07F8000003FC07FC000007FC07FC000007FC03FC000007F803FC000007F801FE00000 FF001FE00000FF000FF00001FE0007F00001FC0003F80003F80001FC0007F00000FF001FE00000 3FC07F8000000FFFFE00000000FFE00000>43 41 124 168 53 I[ 39 41 125 168 48 I[48 41 125 168 53 82 D[<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC00 01E0FC0001E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001F FFFE000FFFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8 600001F8E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F 80F3FFFF00E0FFFE00C01FF000>29 41 124 168 39 I[<7FFFFFFFFFC07FFFFFFFFFC07FFFFF FFFFC07F803FC03FC07E003FC007C078003FC003C078003FC003C070003FC001C0F0003FC001E0 F0003FC001E0E0003FC000E0E0003FC000E0E0003FC000E0E0003FC000E0E0003FC000E000003F C0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000 00003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003F C0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000 00003FC0000000003FC00000007FFFFFE000007FFFFFE000007FFFFFE000>43 40 126 167 49 I[46 41 125 168 54 I[<7FFFF81FFFF07FFFF81FFFF07FFFF81FFFF001FF0000780000FF8000F000007FC001E00000 7FC001C000003FE003C000001FF0078000000FF80F0000000FF80E00000007FC1E00000003FE3C 00000003FE7800000001FF7000000000FFF0000000007FE0000000007FC0000000003FE0000000 001FF0000000001FF0000000001FF8000000001FFC000000003FFE000000007BFE00000000F1FF 00000000E0FF80000001E0FFC0000003C07FC0000007803FE0000007001FF000000F001FF00000 1E000FF800003C0007FC0000380003FE0000780003FE0000F00001FF0000E00000FF80FFFF801F FFFEFFFF801FFFFEFFFF801FFFFE>47 41 126 168 53 88 D[<3FFFFFFF803FFFFFFF803FFFFF FF803FF000FF003F8001FE003F0003FE003E0003FC003C0007F8007C000FF80078000FF0007800 1FE00070003FE00070003FC00070007F80007000FF80000000FF00000001FE00000003FE000000 03FC00000007F80000000FF80000000FF00000001FE00000003FE00000003FC001C0007F8001C0 00FF8001C000FF0001C001FE0001C003FE0003C003FC0003C007F80003C00FF80007800FF00007 801FE0000F803FE0001F803FC0007F807F8001FF80FFFFFFFF80FFFFFFFF80FFFFFFFF80>34 41 124 168 43 90 D[<01FF800007FFF0000F81F8001FC07E001FC07E001FC03F000F803F8007 003F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F800FE03F803F803F803F003F80 7F003F80FE003F80FE003F80FE003F80FE003F807E007F807F00DF803F839FFC0FFF0FFC01FC03 FC>30 27 126 154 33 97 D[33 42 126 169 39 I[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F803F801F007F80 0E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF000000FF0000007F 0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FFFC00001FE000 >26 27 126 154 31 I[<00003FF80000003FF80000003FF800000003F800000003F800000003 F800000003F800000003F800000003F800000003F800000003F800000003F800000003F8000000 03F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F8001F8003F8003F 8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F800FF0003F800 FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003F8001F8003F8 000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80>33 42 126 169 39 I[<003FE00001FFF80003F07E0007C01F000F801F801F800F803F800FC07F000FC0 7F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F0000 007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF000>27 27 126 154 32 I[<0007F0003FFC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007F000 07F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F0007FFF807FFF807FFF80>24 42 126 169 21 I[<00FF81F003FFE7F80FC1 FE7C1F80FC7C1F007C383F007E107F007F007F007F007F007F007F007F007F007F007F007F003F 007E001F007C001F80FC000FC1F8001FFFE00018FF800038000000380000003C0000003E000000 3FFFF8001FFFFF001FFFFF800FFFFFC007FFFFE01FFFFFF03E0007F07C0001F8F80000F8F80000 F8F80000F8F80000F87C0001F03C0001E01F0007C00FC01F8003FFFE00007FF000>30 40 126 154 34 I[33 42 125 169 39 I[<07000FC01FE03FE03FE03FE01FE00FC007000000000000000000000000000000FFE0 FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 20 I[31 42 126 169 37 107 D[15 42 125 169 20 I[33 27 125 154 39 110 D[<003FE00001FFFC0003F07E000FC01F801F800F C03F800FE03F0007E07F0007F07F0007F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF00 07F8FF0007F8FF0007F8FF0007F87F0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007 F07F0001FFFC00003FE000>29 27 126 154 34 I[33 39 126 154 39 I[24 27 126 154 28 114 D[<03FE300FFFF01E03F03800F0700070F00070F000 70F80070FC0000FFE0007FFE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE000 3CF0003CF00038F80038FC0070FF01E0F7FFC0C1FF00>22 27 126 154 27 I[<00700000700000700000700000F00000F00000F00001F00003F00003F00007F0001FFFF0 FFFFF0FFFFF007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F03807F03807F03807F03807F03807F03803F03803F87001F86000FFC0001F80 >21 38 127 165 27 I[33 27 125 154 39 I[33 39 127 154 37 121 D[<3FFFFF803FFFFF803F007F003C00FE0038 01FE007803FC007803F8007007F800700FF000700FE000001FC000003FC000007F8000007F0000 00FF000001FE038001FC038003F8038007F803800FF007800FE007801FE007003FC00F003F801F 007F007F00FFFFFF00FFFFFF00>25 27 126 154 31 I E /Fh 25 87 438 432 dfs[<387C7EFC7C38>7 6 123 133 17 46 D[<0000C0000180000780007F8007F7800787 80000780000F00000F00000F00000F00000F00000F00001E00001E00001E00001E00001E00001E 00003C00003C00003C00003C00003C00003C0000780000780000780000780000780000780000F0 0000F00000F00000F00000F00000F00001F000FFFFE0FFFFE0>19 40 122 167 30 49 D[<0003FC00000FFF00003C0F80006007C0008003E001E003E001F003E003F003E0 03F003E001E003E000C007E0000007C0000007C000000F8000001F0000001E00000038000000E0 00001FE0000000780000001C0000001E0000000F0000000F8000000F8000000FC000000FC00000 0FC018000FC07C000FC07C000F80FC001F80FC001F80F8001F0040003E0040003E0020007C0038 00F8001E03E00007FFC00000FE0000>27 41 125 167 30 51 D[<00C000C000F0078000FFFF00 00FFFE0001FFF80001FFE000011F00000100000001000000010000000200000002000000020000 000200000002000000021F800004E0E000050070000600380004003C0000003C0000001E000000 1E0000001E0000001E0000003E0000003E0070003E00F8003E00F8003E00F8007C00F0007C0080 0078008000F8008000F000C001E0004003C00060078000381F00000FFC000007F00000>26 41 123 167 30 53 D[<0000FC000007FF00000F0180003C01C0007003C000E007C001C007C003 C0078007800000070000000F0000001F0000001E0000003E0000003E0000003C1F80007C606000 7C8038007D001C00FA001C00FA001E00FC000F00FC000F00F8000F00F8000F00F8001F00F0001F 00F0001F00F0001F00F0001F00F0001E00F0003E00F0003C00F0003C0070007800700070003800 E0003C01C0001E0780000FFE000003F80000>26 41 123 167 30 I[<080000000C0000001FFF FFC01FFFFFC01FFFFF803FFFFF8020000100600002004000040040000800800010008000200000 004000000040000000800000010000000200000006000000040000000800000018000000380000 00300000007000000060000000E0000000E0000001E0000001C0000003C0000003C0000007C000 0007800000078000000F8000000F8000000F8000000F8000001F0000001F0000001F0000000E00 0000>26 42 120 168 30 I[<0000003000000000300000000070000000007000000000F00000 0000F800000001F800000001F800000002F800000002F800000004FC0000000C7C000000087C00 0000107C000000107C000000207C000000203E000000403E000000403E000000803E000000803E 000001003F000001001F000002001F000002001F000004001F000004001F80000FFFFF80000FFF FF800010000F800010000F800020000F8000200007C000400007C000C00007C000800007C00180 0007C001000007E003800003E00FC00007E0FFF000FFFFFFF000FFFF>40 42 126 169 46 65 D[<01FFFFFF0003FFFFFFC0000FC003F0000F8000F8000F8000FC000F8000 7C000F80007E000F80007E001F00003E001F00003E001F00007E001F00007E001F00007C001F00 00FC003E0000F8003E0001F0003E0003E0003E0007C0003E001F00003FFFFC00007C000F80007C 0007C0007C0003F0007C0001F0007C0000F8007C0000F800F80000FC00F80000FC00F80000FC00 F80000FC00F80000FC00F80000F801F00001F801F00001F801F00003F001F00007E001F0000FC0 01F0001F8003F0007F00FFFFFFFC00FFFFFFE000>39 41 126 168 43 I[<00000FF00100007F FE030001FC07070007E0018E001F80005E003E00003E007C00003E00F800001E01F000001E03E0 00000C07C000000C07C000000C0F8000000C1F8000000C1F0000000C3F000000083F000000007E 000000007E000000007E000000007E00000000FC00000000FC00000000FC00000000FC00000000 FC00000000FC00000000FC00000000FC000000207C000000207C000000207C000000403E000000 403E000000801E000000801F000001000F8000020007C000040003E000180001F0003000007E01 E000003FFF80000007FC0000>40 43 122 169 44 I[<01FFFFFF800003FFFFFFE000000FC003 F800000F80007C00000F80003E00000F80001F00000F80000F80000F80000780001F000007C000 1F000007C0001F000003E0001F000003E0001F000003E0001F000003E0003E000003F0003E0000 03F0003E000003F0003E000003F0003E000003F0003E000003F0007C000003E0007C000007E000 7C000007E0007C000007E0007C000007C0007C000007C000F800000FC000F800000F8000F80000 1F8000F800001F0000F800001F0000F800003E0001F000007C0001F00000F80001F00001F00001 F00003E00001F00007C00001F0001F800003F000FE0000FFFFFFF80000FFFFFFC00000>44 41 126 168 47 I[<01FFFFFFFF03FFFFFFFF000FC0007F000F80000F000F800007000F800007 000F800003000F800003001F000003001F000003001F000001001F000001001F000801001F0008 01003E001000003E001000003E001000003E003000003E00F000003FFFF000007FFFE000007C00 E000007C006000007C006000007C002000007C00200200F800400400F800400400F800000400F8 00000800F800000800F800001801F000001001F000003001F000003001F000007001F00000E001 F00003E003F0000FE0FFFFFFFFC0FFFFFFFFC0>40 41 126 168 42 I[<01FFFFFFFE03FFFFFF FE000FC0007E000F80001E000F80000E000F800006000F800006000F800006001F000006001F00 0006001F000002001F000002001F001002001F001002003E002000003E002000003E002000003E 006000003E01E000003FFFE000007FFFC000007C01C000007C00C000007C00C000007C00400000 7C00400000F800800000F800800000F800000000F800000000F800000000F800000001F0000000 01F000000001F000000001F000000001F000000001F000000003F0000000FFFFE00000FFFFE000 00>39 41 126 168 40 I[<00000FF00100007FFE030001FC07070007E0018E001F80005E003E 00003E007C00003E00F800001E01F000001E03E000000C07C000000C07C000000C0F8000000C1F 8000000C1F0000000C3F000000083F000000007E000000007E000000007E000000007E00000000 FC00000000FC00000000FC00000000FC00000000FC0001FFFFFC0001FFFFFC000003F0FC000003 E07C000003E07C000003E07C000003E03E000003E03E000007C01F000007C01F000007C00F8000 07C007C0000FC003E00013C001F8002380007E01C180003FFF00800007FC0000>40 43 122 169 48 I[<01FFFF03FFFE03FFFF07FFFE000FC0001F80000F80001F00000F80001F00 000F80001F00000F80001F00000F80001F00001F00003E00001F00003E00001F00003E00001F00 003E00001F00003E00001F00003E00003E00007C00003E00007C00003E00007C00003E00007C00 003E00007C00003FFFFFFC00007FFFFFF800007C0000F800007C0000F800007C0000F800007C00 00F800007C0000F80000F80001F00000F80001F00000F80001F00000F80001F00000F80001F000 00F80001F00001F00003E00001F00003E00001F00003E00001F00003E00001F00003E00001F000 03E00003F00007E000FFFF81FFFF00FFFF81FFFF00>47 41 126 168 46 I[<03FFFF03FFFF000FC0000F80000F80000F80000F80000F80001F00001F00001F00001F0000 1F00001F00003E00003E00003E00003E00003E00003E00007C00007C00007C00007C00007C0000 7C0000F80000F80000F80000F80000F80000F80001F00001F00001F00001F00001F00001F00003 F000FFFFC0FFFFC0>24 41 126 168 22 I[<01FFFF800003FFFF8000000FC00000000F800000 000F800000000F800000000F800000000F800000001F000000001F000000001F000000001F0000 00001F000000001F000000003E000000003E000000003E000000003E000000003E000000003E00 0000007C000000007C000000007C000000007C000000007C000000007C00002000F800004000F8 00004000F800004000F800008000F800008000F800018001F000018001F000030001F000030001 F000070001F0000E0001F0003E0003F001FE00FFFFFFFC00FFFFFFFC00>35 41 126 168 38 76 D[<01FF80000007FF8003FFC000000BFF80000BC000000FE000000BC00000 17C000000BC0000017C0000009E0000027C0000009E0000027C0000009E0000047C0000011E000 004F80000011E000008F80000010F000010F80000010F000010F80000010F000020F80000010F0 00020F800000207800041F000000207800041F000000207800081F000000207800081F00000020 7800101F000000203C00201F000000403C00203E000000403C00403E000000403C00403E000000 401E00803E000000401E00803E000000401E01003E000000801E01007C000000801E02007C0000 00800F04007C000000800F04007C000000800F08007C000000800F08007C00000100079000F800 000100079000F80000010007A000F80000010007A000F80000010007C000F800000300038000F8 00000FC0038001F80000FFF803007FFFC000FFF803007FFFC000>57 41 126 168 56 I[<01FFC0003FFE03FFC0007FFE000FE00007E0000BF00003800009F00001000009 F80001000008F80001000008FC00010000107C00020000107E00020000103E00020000103F0002 0000101F00020000101F80020000200F80040000200FC00400002007C00400002007E004000020 03E00400002003F00400004001F00800004001F80800004000F80800004000FC08000040007C08 000040007E08000080003E10000080003F10000080001F10000080001F90000080000F90000080 0007D00001000007E00001000003E00001000003E00001000001E00001000001E00003000000E0 000FC00000C000FFF800004000FFF800004000>47 41 126 168 46 I[<00001FF0000000F03C 000003C00700000F0003C0001E0001E0003C0000F000F80000F001F000007801E000007C03C000 007C07C000003C0F8000003E0F8000003E1F0000003E1F0000003E3F0000003F3E0000003F7E00 00003F7E0000003F7E0000003F7E0000003FFC0000007EFC0000007EFC0000007EFC0000007EFC 0000007CFC000000FCFC000000FCFC000000F8FC000001F87C000001F07C000003F07C000003E0 3E000007C03E00000F801E00000F001F00001E000F80003C00078000780003C000F00000F003C0 00007C0F0000000FF80000>40 43 122 169 47 I[<01FFFFFF0003FFFFFFE0000FC003F0000F 8000F8000F80007C000F80003E000F80003F000F80003F001F00003F001F00003F001F00003F00 1F00003F001F00003F001F00007E003E00007E003E00007C003E0000F8003E0001F0003E0007E0 003E001F80007FFFFC00007C000000007C000000007C000000007C000000007C00000000F80000 0000F800000000F800000000F800000000F800000000F800000001F000000001F000000001F000 000001F000000001F000000001F000000003F0000000FFFF800000FFFF800000>40 41 126 168 42 I[<01FFFFFC000003FFFFFF8000000FC00FE000000F8001F000000F8000F800 000F80007C00000F80007C00000F80007E00001F00007E00001F00007E00001F00007E00001F00 007E00001F00007E00001F0000FC00003E0000F800003E0001F800003E0001E000003E0003C000 003E000F0000003E007C0000007FFFE00000007C00700000007C003C0000007C001E0000007C00 0F0000007C000F000000F8000F800000F8000F800000F8000F800000F8000F800000F8000F8000 00F8000F800001F0001F800001F0001F800001F0001F800001F0001F804001F0001F804001F000 1F808003F0000FC080FFFF8007C100FFFF8003C20000000000FC00>42 42 126 168 45 82 D[<0001FC020007FF06001E038E003800DC0070007C00E0003C01E0001C03C0 001C03C0001C0380000807800008078000080780000807C0000807C0000007E0000003F0000003 FE000001FFE00001FFFE0000FFFF00003FFF80000FFFC00000FFE000000FE0000003F0000001F0 000001F0000001F0200000F0200000F0200000F0200000E0600001E0600001E0700001C0700003 C0780007807C000700E6001E00E3C07C00C1FFF000803FC000>31 43 125 169 33 I[<1FFFFFFFFE3FFFFFFFFE3F003F007E38003E001E30003E000C20003E000460003E00 0440003E000440007C000440007C000480007C000480007C000480007C000480007C00040000F8 00000000F800000000F800000000F800000000F800000000F800000001F000000001F000000001 F000000001F000000001F000000001F000000003E000000003E000000003E000000003E0000000 03E000000003E000000007C000000007C000000007C000000007C000000007C000000007C00000 000FC000000FFFFFC0001FFFFFC000>39 41 121 168 44 I[<7FFFC00FFF80FFFFC01FFF8003 F00001F80003E00000E00003E00000400003E00000400003E00000400003E00000400007C00000 800007C00000800007C00000800007C00000800007C00000800007C0000080000F80000100000F 80000100000F80000100000F80000100000F80000100000F80000100001F00000200001F000002 00001F00000200001F00000200001F00000200001F00000200003E00000400003E00000400003E 00000400003E00000400003E00000800003E00000800003E00001800001E00001000001E000020 00000F00006000000F0000C0000007800180000003C00700000001F01C000000007FF800000000 1FC0000000>41 42 120 168 46 I[42 42 120 168 46 I E /Fi 3 21 438 432 dfs[35 3 123 143 47 0 D[<007C0001FF0007FFC00FFFE01FFFF03FFFF83FFFF87FFFFC7FFFFCFFFFFE FFFFFEFFFFFEFFFFFEFFFFFEFFFFFEFFFFFE7FFFFC7FFFFC3FFFF83FFFF81FFFF00FFFE007FFC0 01FF00007C00>23 25 125 154 30 15 D[<000000006000000001E000000007E00000001F8000 00007E00000001F800000007E00000001F800000007E00000001F800000007E00000001F800000 007E00000001F800000007E00000001F800000007E00000000F800000000F8000000007E000000 001F8000000007E000000001F8000000007E000000001F8000000007E000000001F8000000007E 000000001F8000000007E000000001F8000000007E000000001F8000000007C000000001E00000 0000E0000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000007FFFFFFFC0FFFFFFFFE0FFFFFFFFE0>35 48 123 165 47 20 D E /Fj 35 123 576 432 dfs[<1C003E007F00FF80FF80FF807F003E001C00>9 9 123 136 25 46 D[<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 007FFFFE7FFFFE7FFFFE>23 39 123 166 45 49 D[<00FF800003FFF0000FFFFC001F03FE0038 00FF007C007F80FE003FC0FF003FC0FF003FE0FF001FE0FF001FE07E001FE03C003FE000003FE0 00003FC000003FC000007F8000007F000000FE000000FC000001F8000003F0000003E000000780 00000F0000001E0000003C00E0007000E000E000E001C001C0038001C0070001C00FFFFFC01FFF FFC03FFFFFC07FFFFFC0FFFFFF80FFFFFF80FFFFFF80>27 39 125 166 45 I[<007F800003FFF00007FFFC000F81FE001F00FF003F80FF003F807F803F807F803F807F80 1F807F800F007F800000FF000000FF000000FE000001FC000001F8000007F00000FFC00000FFF0 000001FC0000007E0000007F0000007F8000003FC000003FC000003FE000003FE03C003FE07E00 3FE0FF003FE0FF003FE0FF003FC0FF007FC07E007F807C007F003F01FE001FFFFC0007FFF00000 FF8000>27 39 125 166 45 I[<00000E0000001E0000003E0000007E000000FE000000FE0000 01FE000003FE0000077E00000E7E00000E7E00001C7E0000387E0000707E0000E07E0000E07E00 01C07E0003807E0007007E000E007E000E007E001C007E0038007E0070007E00E0007E00FFFFFF F8FFFFFFF8FFFFFFF80000FE000000FE000000FE000000FE000000FE000000FE000000FE000000 FE00007FFFF8007FFFF8007FFFF8>29 39 126 166 45 I[<0C0003000F803F000FFFFE000FFF FC000FFFF8000FFFF0000FFFE0000FFFC0000FFE00000E0000000E0000000E0000000E0000000E 0000000E0000000E7FC0000FFFF8000F80FC000E003E000C003F0000001F8000001FC000001FC0 00001FE000001FE018001FE07C001FE0FE001FE0FE001FE0FE001FE0FE001FC0FC001FC078003F 8078003F803C007F001F01FE000FFFF80003FFF00000FF8000>27 39 125 166 45 I[<0007F000003FFC0000FFFE0001FC0F0003F01F8007E03F800FC03F801FC03F801F80 3F803F801F003F8000007F0000007F0000007F000000FF000000FF0FC000FF3FF800FF707C00FF C03E00FFC03F00FF801F80FF801FC0FF001FC0FF001FE0FF001FE0FF001FE07F001FE07F001FE0 7F001FE07F001FE03F001FE03F001FC01F801FC01F803F800FC03F0007E07E0003FFFC0000FFF0 00003FC000>27 39 125 166 45 I[<380000003E0000003FFFFFF03FFFFFF03FFFFFF07FFFFF E07FFFFFC07FFFFF807FFFFF0070000E0070000E0070001C00E0003800E0007000E000E0000000 E0000001C000000380000007800000078000000F0000000F0000001F0000001F0000003F000000 3E0000003E0000007E0000007E0000007E0000007E000000FE000000FE000000FE000000FE0000 00FE000000FE000000FE000000FE0000007C000000380000>28 41 124 168 45 I[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE0000 7F8003FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803F C0000003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F 80000000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE00000 07000FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E00000 0FFFFFC0000003FFFF000000003FF80000>41 41 124 168 67 67 D[45 41 125 168 71 I[36 41 125 168 57 70 D[<00007FE003000003FFFC0700001FFFFF0F00003FF00FFF0000FF8001FF0001FE0000 FF0003F800003F0007F000003F000FF000001F001FE000000F001FE000000F003FC000000F003F C0000007007FC0000007007F80000007007F8000000000FF8000000000FF8000000000FF800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8001FFFFF87F 8001FFFFF87F8001FFFFF87FC00000FF003FC00000FF003FC00000FF001FE00000FF001FE00000 FF000FF00000FF0007F00000FF0003F80000FF0001FE0000FF0000FF8001FF00003FF007BF0000 1FFFFF1F000003FFFE0F0000007FF00300>45 41 124 168 72 I[22 41 126 168 35 73 D[<0000FFE000000007FFFC0000003FC07F8000007F001FC00001FC0007F0 0003F80003F80007F00001FC000FF00001FE001FE00000FF001FE00000FF003FC000007F803FC0 00007F807FC000007FC07F8000003FC07F8000003FC07F8000003FC0FF8000003FE0FF8000003F E0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF80 00003FE0FF8000003FE07F8000003FC07FC000007FC07FC000007FC03FC000007F803FC000007F 801FE00000FF001FE00000FF000FF00001FE0007F00001FC0003F80003F80001FC0007F00000FF 001FE000003FC07F8000000FFFFE00000000FFE00000>43 41 124 168 69 79 D[<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC00 01E0FC0001E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001F FFFE000FFFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8 600001F8E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F 80F3FFFF00E0FFFE00C01FF000>29 41 124 168 51 83 D[<01FF800007FFF0000F81F8001FC0 7E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007 FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F80 7F00DF803F839FFC0FFF0FFC01FC03FC>30 27 126 154 44 97 D[33 42 126 169 51 I[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F80 3F801F007F800E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF0000 00FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FF FC00001FE000>26 27 126 154 41 I[<00003FF80000003FF80000003FF800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800000003F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F800 1F8003F8003F8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F8 00FF0003F800FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003 F8001F8003F8000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80>33 42 126 169 51 I[<003FE00001FFF80003F07E0007C01F000F801F801F800F803F800FC07F00 0FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F 0000007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF000 >27 27 126 154 43 I[<0007F0003FFC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007 F00007F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007 F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007 F00007F00007F00007F0007FFF807FFF807FFF80>24 42 126 169 28 I[33 42 125 169 51 104 D[<07000FC01FE03FE03FE03FE01FE00FC0070000 00000000000000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0 0FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 27 I[15 42 125 169 27 108 D[53 27 125 154 80 I[33 27 125 154 51 I[<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F07F 0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007F0 7F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE000>29 27 126 154 45 I[33 39 126 154 51 I[24 27 126 154 37 114 D[<03FE300FFFF01E03F03800F0700070F00070F00070F80070FC0000FFE0007FFE007F FF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE0003CF0003CF00038F80038FC0070FF 01E0F7FFC0C1FF00>22 27 126 154 36 I[<00700000700000700000700000F00000F00000F0 0001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F00007F00007F00007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F03807F03807F03807F03807F03807F0 3803F03803F87001F86000FFC0001F80>21 38 127 165 36 I[33 27 125 154 51 I[33 27 127 154 48 I[33 39 127 154 48 121 D[<3FFFFF803FFFFF803F007F003C00FE0038 01FE007803FC007803F8007007F800700FF000700FE000001FC000003FC000007F8000007F0000 00FF000001FE038001FC038003F8038007F803800FF007800FE007801FE007003FC00F003F801F 007F007F00FFFFFF00FFFFFF00>25 27 126 154 41 I E /Fk 70 124 438 432 dfs[<0007F81F80003C067060007003E0F000E007C1F001C00FC1F003C00F80E00780 078040078007800007800780000780078000078007800007800780000780078000078007800007 800780000780078000FFFFFFFF00FFFFFFFF000780078000078007800007800780000780078000 078007800007800780000780078000078007800007800780000780078000078007800007800780 000780078000078007800007800780000780078000078007800007800780000780078000078007 80000780078000078007C000FFF87FFE00FFF87FFE00>36 42 127 169 35 11 D[<0007F800003C06000070010000E0070001C00F8003C00F8007800F80078007000780 000007800000078000000780000007800000078000000780000007800000FFFFFF80FFFFFF8007 800F80078007800780078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800780078007800780FFF87FFCFFF87FFC>30 42 127 169 33 I[<0007F980003C0780007007 8000E00F8001C00F8003C00F800780078007800780078007800780078007800780078007800780 0780078007800780078007800780FFFFFF80FFFFFF800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 0780078007800780078007800780078007800780078007800780078007800780FFFCFFFCFFFCFF FC>30 42 127 169 33 I[<0003F803FC00001E061E030000700138008000E003F0038001C007 E007C003C007E007C0078007C007C0078003C00380078003C00000078003C00000078003C00000 078003C00000078003C00000078003C00000078003C00000078003C00000FFFFFFFFFFC0FFFFFF FFFFC0078003C007C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0 078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003 C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0 078003C003C0078003C003C0078003C003C0FFFC7FFE7FFEFFFC7FFE7FFE>47 42 127 169 51 I[<78FCFCFEFE7A020202020404040810102040>7 18 123 169 17 39 D[<0004000800100020004000C0018003000300060006000E000C001C001800 38003800380030007000700070007000F000F000E000E000E000E000E000E000E000E000E000E0 00E000F000F0007000700070007000300038003800380018001C000C000E000600060003000300 018000C000400020001000080004>14 61 123 172 23 I[<800040002000100008000C000600 030003000180018001C000C000E0006000700070007000300038003800380038003C003C001C00 1C001C001C001C001C001C001C001C001C001C003C003C00380038003800380030007000700070 006000E000C001C0018001800300030006000C0008001000200040008000>14 61 125 172 23 I[<000038000000003800000000380000000038000000003800000000380000 000038000000003800000000380000000038000000003800000000380000000038000000003800 0000003800000000380000000038000000003800000000380000FFFFFFFFFEFFFFFFFFFEFFFFFF FFFE00003800000000380000000038000000003800000000380000000038000000003800000000 380000000038000000003800000000380000000038000000003800000000380000000038000000 00380000000038000000003800000000380000>39 41 125 162 47 43 D[<78FCFCFEFE7A020202020404040810102040>7 18 123 133 17 I[15 3 127 142 20 I[<78FCFCFCFC78>6 6 123 133 17 I[<00000600000E00000E00001C00001C 00001C0000380000380000380000700000700000E00000E00000E00001C00001C00001C0000380 000380000380000700000700000700000E00000E00000E00001C00001C00001C00003800003800 00700000700000700000E00000E00000E00001C00001C00001C000038000038000038000070000 0700000700000E00000E00000E00001C00001C0000380000380000380000700000700000700000 E00000E00000C00000>23 60 125 172 30 I[<007F000001C1C0000780F0000F0078000E0038 001C001C003C001E003C001E003C001E0078000F0078000F0078000F0078000F00F8000F80F800 0F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8 000F80F8000F80F8000F80F8000F8078000F0078000F0078000F0078000F003C001E003C001E00 3C001E001C001C000E0038000F0078000780F00001C1C000007F0000>25 41 126 167 30 I[<00100000700001F0000FF000FEF000F0F00000F00000F00000F00000F000 00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F000 00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F000 00F00001F8007FFFE07FFFE0>19 40 123 167 30 I[<00FE0007FF800E07E01803F02001F820 00F840007C40007CF8007EFC007EFC003EFC003EFC003E78007E00007E00007C00007C0000F800 00F80001F00001E00003C0000780000700000E00001C0000380000700000600000C00001800203 00020600040C000418000410000C3FFFFC7FFFF8FFFFF8FFFFF8>23 40 125 167 30 I[<007F000003FFC0000701F0000C00F80010007C001C007C003E007E003E003E00 3E003E001E003E000C007E0000007C0000007C00000078000000F0000000E0000001C000000700 0000FF00000001E0000000F0000000780000003C0000003E0000001F0000001F0000001F800000 1F8030001F8078001F80FC001F80FC001F80FC001F00F8001F0040003F0040003E0030007C0018 00F8000F01F00003FFC000007F0000>25 41 126 167 30 I[<00006000000060000000E00000 01E0000001E0000003E0000003E0000005E0000009E0000009E0000011E0000021E0000021E000 0041E0000081E0000081E0000101E0000201E0000201E0000401E0000801E0000801E0001001E0 003001E0002001E0004001E000C001E000FFFFFF80FFFFFF800001E0000001E0000001E0000001 E0000001E0000001E0000001E0000001E0000003F000007FFF80007FFF80>25 40 126 167 30 I[<1800181F00F01FFFE01FFFC01FFF801FFF0011F800100000100000100000 100000100000100000100000100000107E001183801600C01800E010007000007800003C00003C 00003C00003E00003E00003E70003EF8003EF8003EF8003EF8003C80003C40007C400078200078 3000F01801E00E07C007FF0001FC00>23 41 125 167 30 I[<000FE000003FF80000F81C0001 E00C0003801E0007803E000F003E000E001C001E0000001C0000003C0000003C0000007C000000 7800000078000000F83F0000F840E000F9807000F9003800FA001C00FC001E00FC001E00FC000F 00F8000F00F8000F80F8000F80F8000F80F8000F8078000F8078000F8078000F807C000F803C00 0F003C000F001C001E001E001E000E003C000700780003C0F00001FFC000007F0000>25 41 126 167 30 I[<20000000380000003FFFFF803FFFFF803FFFFF007FFFFF00600002004000 040040000400400008008000100080002000000020000000400000008000000080000001000000 030000000200000006000000060000000C0000000C0000001C0000001C0000001C000000380000 00380000003800000078000000780000007800000078000000F8000000F8000000F8000000F800 0000F8000000F8000000F8000000F8000000700000>25 42 125 168 30 I[<007F000001FFC0000381F000060078000C003C001C001C0018000E0038000E0038000E0038 000E003C000E003C000E003E001C001F8018001FC038000FF0600007F8C00003FF800001FF0000 007FC00000FFE000030FF8000603FC001C01FE0038007E0030003F0070000F0070000780E00007 80E0000380E0000380E0000380E0000380F0000300700007007800060038000C001E0038000F80 F00003FFE000007F0000>25 41 126 167 30 I[<007F000001FFC00007C1E0000F0070001E00 38001C003C003C001C0078001E0078001E00F8000F00F8000F00F8000F00F8000F00F8000F80F8 000F80F8000F80F8000F8078000F8078001F803C001F803C001F801C002F800E004F800700CF80 03810F80007E0F8000000F0000000F0000000F0000001E0000001E0000001E0000003C001C003C 003E0078003E0070003C00E0001801C0001C0780000FFE000003F80000>25 41 126 167 30 I[<78FCFCFCFC78000000000000000000000000000078FCFCFCFC78>6 26 123 153 17 I[<78FCFCFCFC78000000000000000000000000000070F8FCFCFC7C04040404 0808081010202040>6 38 123 153 17 I[39 15 125 149 47 61 D[<000018000000001800000000 18000000003C000000003C000000003C000000007E000000007E00000000FF000000009F000000 009F000000011F800000010F800000010F8000000207C000000207C000000207C000000403E000 000403E000000403E000000801F000000801F000001801F800001000F800001000F800002000FC 000020007C00003FFFFC00007FFFFE000040003E000040003E000080001F000080001F00008000 1F000100000F800100000F800100000F8002000007C007000007C01F80000FE0FFF000FFFFFFF0 00FFFF>40 42 126 169 46 65 D[36 41 126 168 43 I[<0000FF00100007FFE030001FC07830003E000C7000F80006F001F00003F003E00001 F007C00000F00F800000700F800000701F000000303F000000303E000000303E000000107E0000 00107E000000107C00000000FC00000000FC00000000FC00000000FC00000000FC00000000FC00 000000FC00000000FC00000000FC000000007C000000007E000000007E000000103E000000103E 000000103F000000101F000000200F800000200F8000006007C000004003E000008001F0000180 00F8000300003E000E00001FC038000007FFE0000000FF8000>36 43 125 169 44 I[36 41 126 168 42 69 D[34 41 126 168 40 I[<0000FF00100007FFE030001FC07830003E000C7000F80006F001F00003F0 03E00001F007C00000F00F800000700F800000701F000000303F000000303E000000303E000000 107E000000107E000000107C00000000FC00000000FC00000000FC00000000FC00000000FC0000 0000FC00000000FC00000000FC00000000FC0000FFFF7C0000FFFF7E000003F07E000001F03E00 0001F03E000001F03F000001F01F000001F00F800001F00F800001F007C00001F003E00001F001 F00002F000F80002F0003E000C70001FC038300007FFE0100000FF8000>40 43 125 169 48 I[40 41 126 168 46 I[17 41 126 168 22 I[50 41 126 168 56 77 D[40 41 126 168 46 I[<0001FF0000000F01E000003C00 78000078003C0000E0000E0001E0000F0003C000078007800003C00F800003E01F000001F01F00 0001F03E000000F83E000000F87E000000FC7E000000FC7C0000007C7C0000007CFC0000007EFC 0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007E 7C0000007C7E000000FC7E000000FC7E000000FC3E000000F83F000001F81F000001F01F000001 F00F800003E007800003C007C00007C003E0000F8000F0001E000078003C00003C007800000F01 E0000001FF0000>39 43 125 169 47 I[35 41 126 168 42 I[41 42 126 168 45 82 D[<00FE010003FF83000F81E3001E0037003C001F0038000F00780007007000 0700F0000300F0000300F0000300F0000100F8000100F8000100FC0000007C0000007F0000003F E000001FFF00000FFFE00007FFF80003FFFC00007FFE000007FF0000007F0000001F8000000F80 000007C0000007C0800003C0800003C0800003C0800003C0C00003C0C0000380C0000380E00007 80F0000700F8000E00EE001C00C3C07800C1FFF000803FC000>26 43 125 169 33 I[<7FFFFFFFF87FFFFFFFF87C007C00F870007C003860007C001840007C000840007C00 08C0007C000CC0007C000C80007C000480007C000480007C000480007C000480007C000400007C 000000007C000000007C000000007C000000007C000000007C000000007C000000007C00000000 7C000000007C000000007C000000007C000000007C000000007C000000007C000000007C000000 007C000000007C000000007C000000007C000000007C000000007C000000007C000000007C0000 0000FE000000FFFFFE0000FFFFFE00>38 41 126 168 44 I[58 42 127 168 62 87 D[9 60 122 172 17 91 D[9 60 127 172 17 93 D[<01FC00000E0780001001C0003C00E0003E 00F0003E0078001C00780008007800000078000000780000007800007FF80003E078000F807800 1F0078003E0078007C00780078007820F8007820F8007820F8007820F800F8207C00F8203C013C 401F063FC007F80F00>27 26 126 153 30 97 D[<07800000FF800000FF8000000F8000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 800000078000000783F000078C1C0007B0070007A0038007C003C0078001E0078001E0078000F0 078000F0078000F8078000F8078000F8078000F8078000F8078000F8078000F8078000F0078000 F0078001F0078001E0078001C007C003C00740078007200E0006181C000407E000>29 42 127 169 33 I[<007F8001C0700780080F003C1E007C3C007C3C00387C0010780000F80000 F80000F80000F80000F80000F80000F80000F800007800007C00003C00043C00041E00080F0010 07802001C0C0007F00>22 26 126 153 27 I[<00000F000001FF000001FF0000001F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F00003F0F0001C0CF0003802F000F001F001E001F001C000F003C000F007C000F0078 000F0078000F00F8000F00F8000F00F8000F00F8000F00F8000F00F8000F00F8000F0078000F00 78000F003C000F003C000F001E001F000E002F0007004F8001C18FF8007E0FF8>29 42 126 169 33 I[<007E0003C3800700E00E00F01C00703C00783C003878003C78003CF8003C F8003CFFFFFCF80000F80000F80000F80000F800007800007C00003C00043C00041E00080E0010 07002001C0C0007F00>22 26 126 153 27 I[<001F000070C000E1E001C3E003C3E00381C007 8080078000078000078000078000078000078000078000078000078000FFFE00FFFE0007800007 800007800007800007800007800007800007800007800007800007800007800007800007800007 800007800007800007800007800007800007800007C000FFFE00FFFE00>19 42 127 169 18 I[<0000078001FC1840070721C00E03C1C01E03C0803C01E0003C01E0007C01 F0007C01F0007C01F0007C01F0007C01F0003C01E0003C01E0001E03C0000E0380001707000011 FC0000300000003000000030000000380000001C0000001FFFC0000FFFF80007FFFC001C003E00 30000F007000070060000380E0000380E0000380E0000380E0000380700007007000070038000E 000C0018000780F00000FF8000>26 40 126 154 30 I[<07800000FF800000FF8000000F8000 000780000007800000078000000780000007800000078000000780000007800000078000000780 000007800000078000000783F800078C1C0007900E0007A0070007A0078007C0078007C0078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 07800780078007800780078007800780078007800780078007800780FFFCFFFCFFFCFFFC>30 42 127 169 33 I[<07000F801F801F800F800700000000000000000000000000000000000000 07807F807F800F8007800780078007800780078007800780078007800780078007800780078007 800780078007800780FFF8FFF8>13 41 127 168 17 I[<007800FC00FC00FC00FC0078000000 000000000000000000000000000000007C07FC07FC007C003C003C003C003C003C003C003C003C 003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C00 3C7038F838F870F07060C01F80>14 53 130 168 18 I[<07800000FF800000FF8000000F8000 000780000007800000078000000780000007800000078000000780000007800000078000000780 0000078000000780000007807FF007807FF007801F8007801C0007801800078020000780400007 808000078100000782000007870000079F800007A7800007C7C0000783E0000781E0000781F000 0780F8000780780007807C0007803E0007801E0007801F0007801F80FFFC7FF8FFFC7FF8>29 42 127 169 32 I[<0780FF80FF800F8007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 8007800780078007800780FFFCFFFC>14 42 127 169 17 I[<0781F800FC00FF860E030700FF 98070C03800FA0079003C007A003D001E007C003E001E007C003E001E0078003C001E0078003C0 01E0078003C001E0078003C001E0078003C001E0078003C001E0078003C001E0078003C001E007 8003C001E0078003C001E0078003C001E0078003C001E0078003C001E0078003C001E0078003C0 01E0078003C001E0078003C001E0FFFC7FFE3FFFFFFC7FFE3FFF>48 26 127 153 52 I[<0783F800FF8C1C00FF900E000FA0070007A0078007C0078007C0078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800780078007800780078007800780FFFCFFFCFFFCFFFC>30 26 127 153 33 I[<007F000001C1C000070070000E0038001C001C003C001E003C001E007800 0F0078000F00F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F8078 000F0078000F003C001E003C001E001E003C000E0038000700700001C1C000007F0000>25 26 126 153 30 I[<0783F000FF8C1C00FFB00F0007A0078007C003C0078003E0078001E00780 01F0078001F0078000F8078000F8078000F8078000F8078000F8078000F8078000F8078000F007 8001F0078001F0078001E0078003C007C003C007C0078007A00E0007983C000787E00007800000 078000000780000007800000078000000780000007800000078000000780000007800000FFFC00 00FFFC0000>29 38 127 153 33 I[<003F008001E0C180038021800F0013801E000B801E000F 803C0007807C0007807C00078078000780F8000780F8000780F8000780F8000780F8000780F800 0780F80007807C0007807C0007803C0007803E000F801E000F800F0017800780278001E0C78000 7F0780000007800000078000000780000007800000078000000780000007800000078000000780 000007800000FFFC0000FFFC>30 38 126 153 32 I[<0787C0FF98E0FF91F00FA1F007C1F007 C0E007C00007800007800007800007800007800007800007800007800007800007800007800007 800007800007800007800007800007C000FFFE00FFFE00>20 26 127 153 23 I[<07F8401C06C03001C06000C06000C0E00040E00040F00040F800007E00007FF0003FFE00 0FFF0003FF80003FC00007C08001E08001E0C000E0C000E0C000E0E000C0F001C0F80180C40700 83F800>19 26 126 153 24 I[<00800000800000800000800001800001800001800003800003 80000780000F80001FFF80FFFF8007800007800007800007800007800007800007800007800007 800007800007800007800007800007804007804007804007804007804007804007804003C08001 C08000E100003E00>18 37 127 164 23 I[<07800780FF80FF80FF80FF800F800F8007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800780078007800780078007800F8007800F800380178001C027C000E0 47FC003F87FC>30 26 127 153 33 I[29 26 127 153 32 I[40 26 127 153 44 I[29 26 127 153 32 I[29 38 127 153 32 I[<7FFFF87800F06001F04001E04003C0C007C0800780800F 00801F00001E00003C00007C0000780000F00001F00001E00403C00407C0040780040F000C1F00 081E00083C00187C00387800F8FFFFF8>22 26 126 153 27 I[29 1 128 144 30 I E /Fl 12 119 995 432 dfs[<00003FF001800003FFFE0380000FFFFF8780 003FF007DF8000FF8001FF8001FE00007F8003FC00003F8007F000001F800FF000000F801FE000 0007801FE0000007803FC0000007803FC0000003807FC0000003807F80000003807F8000000000 FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000 000000FF8000000000FF80000000007F80000000007F80000000007FC0000003803FC000000380 3FC0000003801FE0000003801FE0000007000FF00000070007F000000E0003FC00001E0001FE00 003C0000FF8000F800003FF007E000000FFFFFC0000003FFFF000000003FF80000>41 41 124 168 115 67 D[<01FF800007FFF0000F81F8001FC07E001FC07E001FC03F000F803F80 07003F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F800FE03F803F803F803F003F 807F003F80FE003F80FE003F80FE003F80FE003F807E007F807F00DF803F839FFC0FFF0FFC01FC 03FC>30 27 126 154 76 97 D[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F803F 801F007F800E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF000000 FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FFFC 00001FE000>26 27 126 154 71 99 D[<003FE00001FFF80003F07E0007C01F000F801F801F80 0F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF 000000FF0000007F0000007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F00 00FFFE00001FF000>27 27 126 154 74 101 D[<07000FC01FE03FE03FE03FE01FE00FC00700 0000000000000000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 46 105 D[15 42 125 169 46 108 D[53 27 125 154 138 I[33 27 125 154 88 I[<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F00 07F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F 0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE000>29 27 126 154 78 I[<00700000700000700000700000F00000F00000F00001F00003F00003F000 07F0001FFFF0FFFFF0FFFFF007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F03807F03807F03807F03807F03807F03803F03803F87001F860 00FFC0001F80>21 38 127 165 62 116 D[33 27 125 154 88 I[33 27 127 154 83 I E /Fm 8 117 829 432 dfs[<000E00001E00007E0007FE00FFFE00FFFE00F8FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE007FFFFE7FFFFE7FFFFE>23 39 123 166 65 49 D[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE00007F8003 FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803FC00000 03807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF8000000000FF 8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F800000 00007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE0000007000F F00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E000000FFFFF C0000003FFFF000000003FF80000>41 41 124 168 96 67 D[<01FF800007FFF0000F81F8001F C07E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF80 07FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F 807F00DF803F839FFC0FFF0FFC01FC03FC>30 27 126 154 63 97 D[<003FE00001FFF80003F0 7E0007C01F000F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF0007E0FF0007E0FF FFFFE0FFFFFFE0FF000000FF000000FF0000007F0000007F0000007F0000003F8000E01F8000E0 0FC001C007E0038003F81F0000FFFE00001FF000>27 27 126 154 61 101 D[33 42 125 169 73 104 D[33 39 126 154 73 112 D[24 27 126 154 54 114 D[<007000007000007000007000 00F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F03807F038 07F03807F03807F03803F03803F87001F86000FFC0001F80>21 38 127 165 52 116 D E /Fn 36 119 400 300 dfs[<0007800000001C600380003010078000601007 8000E010038000C010010001C07001000380F002000380F0020003806004000700000800070000 10000700006000073C00A00007E20310000781041000070108080007011008000F022008001F84 2008001C782C08003C003E080038003E080078001C0800780000080078000008007000001000F0 00001000700000200070000020007800004000380000800038000100001C000200000E000C0000 0380700000007F800000>33 37 123 163 49 38 D[<70F8F8F0E0>5 5 122 132 20 46 D[<0000030000000300000007000000070000000F0000000F0000001F000000 2F0000002F0000004F0000004F8000008780000087800001078000020780000207800004078000 0407800008078000080780001007800030078000200780007FFF80004007C0008007C0008003C0 010003C0030003C0020003C0040003C0040003C00C0003C03C0007C0FF003FFC>30 35 125 162 48 65 D[<00FFFFE0000F0038000F001C000F001E001E000E001E000F001E000F00 1E000F003C000E003C001E003C001E003C003C00780078007800F0007801E00078078000FFFF80 00F001E000F000F000F0007801E0007801E0003801E0003C01E0003C03C0007803C0007803C000 7803C000F0078000F0078001E0078003C0078007000F801E00FFFFF000>32 34 125 161 45 I[<00FFFFF000000F003C00000F000E00000F000700001E000380001E000380 001E0001C0001E0001C0003C0001C0003C0001E0003C0001E0003C0001E000780001E000780001 E000780001E000780001E000F00003C000F00003C000F00003C000F00003C001E000078001E000 078001E000070001E0000F0003C0000E0003C0001C0003C0003C0003C000380007800070000780 00E000078001C00007800700000F801C0000FFFFF00000>35 34 125 161 49 68 D[<00007F00800003808100000E00630000380027000070001F0000E0000E0001C0000E 000380000E000700000E000F000004000E000004001E000004003C000004003C00000800780000 000078000000007800000000F000000000F000000000F000000000F000000000F0003FFC00E000 01E000E00001E000E00001E000E00003C000E00003C000F00003C000700003C000700007800038 0007800018000F80001C0013800006002300000381C1000000FE000000>33 36 121 162 51 71 D[<00FFF8000F00000F00000F00001E00001E00001E00001E00003C00003C 00003C00003C0000780000780000780000780000F00000F00000F00000F00001E00001E00001E0 0001E00003C00003C00003C00003C0000780000780000780000780000F8000FFF800>21 34 125 161 25 73 D[<0007FFC000003C0000003C0000003C0000007800000078000000780000 0078000000F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0000003C000 0003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000380F00 00780F0000F81E0000F81E0000F03C0000403800004070000021E000001F800000>26 35 124 161 35 I[<00FFF807FC000F0001E0000F0001C0000F000100001E000200001E000400 001E001000001E002000003C004000003C008000003C010000003C040000007808000000781800 00007838000000787C000000F0BC000000F23C000000F41E000000F81E000001F01F000001E00F 000001E00F000001E00F800003C007800003C007800003C003C00003C003C000078003C0000780 01E000078001E000078001E0000F8001F000FFF80FFE00>38 34 125 161 49 I[<00FFFC00000F8000000F0000000F0000001E0000001E0000001E0000001E0000003C0000 003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F000 0000F0000001E0000001E0000001E0002001E0002003C0004003C0004003C0008003C000800780 0180078001000780030007800F000F803E00FFFFFE00>27 34 125 161 41 I[<00FF800007FC000F80000F80000F80001780000F80001780001780002F000013C0002F00 0013C0004F000013C0008F000023C0009E000023C0011E000023C0011E000023C0021E000043C0 043C000043C0043C000043C0083C000041E0083C000081E01078000081E02078000081E0207800 0081E04078000101E040F0000101E080F0000101E100F0000101E100F0000200F201E0000200F2 01E0000200F401E0000200F801E0000400F803C0000400F003C0000400F003C0000C00E003C000 1E00C007C000FFC0C07FFC00>46 34 125 161 59 I[<00FF000FFC000F8001E0000F80018000 0FC000800013C001000013C001000011E001000011E001000021E002000020F002000020F00200 0020F0020000407804000040780400004078040000403C040000803C080000803E080000801E08 0000801E080001001F100001000F100001000F10000100079000020007A000020007A000020003 E000020003E000040003C000040001C000040001C0000C0001C0001E00008000FFC0008000>38 34 125 161 48 I[<0000FE0000078380000C00E0003800700070003800E0003801C0001C0380 001C0700001C0F00001E1E00001E1C00001E3C00001E3C00001E7800001E7800001E7800001EF0 00003CF000003CF000003CF0000078F0000078E0000078E00000F0E00000F0E00001E0E00001C0 F00003C0F00007807000070078000E0038001C001C0038000E00E0000703800001FC0000>31 36 121 162 49 I[<00FFFFC0000F0070000F003C000F001C001E000E001E000E001E000F001E 000F003C001E003C001E003C001E003C003C0078003800780070007801E00078078000FFFC0000 F00E0000F0070000F0038001E003C001E003C001E003C001E003C003C0078003C0078003C00780 03C0078007800F0007800F0107800F01078007020F800702FFF8038C000000F0>32 35 125 161 48 82 D[<0001F020000E0C40001802C0003001C0006001C000E0018000C0018001 C0018001C0018003C0010003C0010003C0000003C0000003E0000001F8000001FF000000FFE000 007FF000001FF8000003FC0000007C0000003C0000001E0000001E0000001E0020001C0020001C 0020001C00200018006000380060003000700060007000C000C8018000C607000081FC0000>27 36 125 162 36 I[<1FFFFFF81E03C0381803C0183003C0182007801820078018400780104007 8010400F0010800F0010800F0010000F0000001E0000001E0000001E0000001E0000003C000000 3C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F00000 00F0000001E0000001E0000001E0000001E0000003E00000FFFF0000>29 34 119 161 47 I[<3FFE03FF03C0007803C0006003C000200780004007800040078000400780 00400F0000800F0000800F0000800F0000801E0001001E0001001E0001001E0001003C0002003C 0002003C0002003C0002007800040078000400780004007800040070000800F0000800F0001000 7000100070002000700040003000400038018000180200000E0C000003F00000>32 35 119 161 48 I[45 35 118 161 65 87 D[32 34 118 161 48 89 D[<00F8C00185C00705C00E03800E03801C03803C0380380700780700780700 780700F00E00F00E00F00E00F00E10F01C20701C20703C20305C40308C400F0780>20 21 123 148 33 97 D[<007E0001C1000301800703800E07801C07803C00003800007800007800 00780000F00000F00000F00000F00000F00100700100700200300C001830000FC000>17 21 123 148 29 99 D[<00003C0003F80000380000380000380000700000700000700000700000 E00000E00000E00000E00001C000F9C00185C00705C00E03800E03801C03803C03803807007807 00780700780700F00E00F00E00F00E00F00E10F01C20701C20703C20305C40308C400F0780>22 35 123 162 33 I[<00F803840E021C023C0238027804F018FFE0F000F000E000E000E000E000 E002E0026004701830600F80>15 21 122 148 29 I[<00003E0000470000CF00018F00018600 0380000380000380000700000700000700000700000700000E0000FFF0000E00000E00000E0000 1C00001C00001C00001C00001C0000380000380000380000380000380000700000700000700000 700000700000E00000E00000E00000E00000C00001C00001C000718000F18000F300006200003C 0000>24 45 130 162 20 I[<001F180030B800E0B801C07001C0700380700780700700E00F00 E00F00E00F00E01E01C01E01C01E01C01E01C01E03800E03800E0780060B8006170001E7000007 00000700000E00000E00000E00701C00F01800F0300060E0003F8000>21 31 126 148 29 I[<00C001E001C001C0000000000000000000000000000000001C0023004300 43008700870087000E000E001C001C001C00380038003840708070807080710032001C00>11 33 123 160 20 105 D[<00F0000FE00000E00000E00000E00001C00001C00001C00001C00003 80000380000380000380000700000701E0070210070C700E10F00E10F00E20600E40001D80001E 00001FC0001C7000383800383800381C00381C20703840703840703840701880E01880600F00> 20 35 125 162 29 107 D[<01E01FC001C001C001C0038003800380038007000700070007000E 000E000E000E001C001C001C001C0038003800380038007000700070007100E200E200E200E200 64003800>11 35 124 162 16 I[<1C0F002631C04740C08780E08780E08700E08700E00E01C0 0E01C00E01C00E01C01C03801C03801C03801C0704380708380E08380E103806107006203003C0 >22 21 123 148 36 110 D[<007E0001C3000381800701C00E01C01C01E03C01E03801E07801 E07801E07801E0F003C0F003C0F00380F00780700700700E00700C0030180018700007C000>19 21 123 148 33 I[<01C1F002621804741C08780C08700E08700E08701E00E01E00E01E00E01E 00E01E01C03C01C03C01C03C01C07803807003807003C0E003C1C0072380071E00070000070000 0E00000E00000E00000E00001C00001C00001C0000FFC000>23 31 127 148 33 I[<1C1F002620804741C08783C08703C08701808700000E00000E00000E00000E00001C 00001C00001C00001C0000380000380000380000380000700000300000>18 21 123 148 28 114 D[<00FC000183000200800401800C03800C03000C00000F00000FF00007 FC0003FE00003E00000F00000700700700F00600F00600E004004008002030001FC000>17 21 125 148 27 I[<00C001C001C001C001C003800380038003800700FFF8070007000E000E00 0E000E001C001C001C001C003800380038003810702070207040708031001E00>13 31 124 158 21 I[<1E00602300E04380E04381C08381C08701C08701C00703800E03800E0380 0E03801C07001C07001C07001C07081C0E10180E101C0E101C1E200C262007C3C0>21 21 123 148 35 I[<1E03802307C04387C04383C08381C08700C08700C00700800E00800E0080 0E00801C01001C01001C01001C02001C02001C04001C08001C08000C300003C000>18 21 123 148 29 I E /Fo 54 122 400 300 dfs[<0000C018000000C018000000C01800000180 300000018030000001803000000180300000030060000003006000000300600000030060000003 006000000600C000000600C000000600C000000600C000000C018000FFFFFFFFC0FFFFFFFFC000 180300000018030000001803000000180300000030060000003006000000300600000030060000 FFFFFFFFC0FFFFFFFFC000600C000000C018000000C018000000C018000000C018000001803000 0001803000000180300000018030000003006000000300600000030060000003006000000600C0 00000600C000000600C00000>34 45 125 162 55 35 D[<70F8FCFC7404040404080810102040 >6 15 124 132 19 44 D[11 2 127 139 21 I[<70F8F8F870>5 5 124 132 19 I[<01F000071C000C06001803003803803803807001C07001C07001C07001C0F0 01E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F0 01E07001C07001C07001C07803C03803803803801C07000C0600071C0001F000>19 34 126 160 32 48 D[<008003800F80F380038003800380038003800380038003800380038003 80038003800380038003800380038003800380038003800380038003800380038007C0FFFE>15 33 124 160 32 I[<03F0000C1C001007002007804003C04003C08003E0F003E0F801E0F801E0 F801E02003E00003E00003C00003C0000780000700000E00001C0000180000300000600000C000 0180000100000200200400200800201800603000403FFFC07FFFC0FFFFC0>19 33 126 160 32 I[<03F8000C1E001007002007804007C07807C07803C07807C03807C0000780 000780000700000F00000E0000380003F000001C00000F000007800007800003C00003C00003E0 2003E07003E0F803E0F803E0F003C04003C0400780200780100F000C1C0003F000>19 34 126 160 32 I[<1000801E07001FFF001FFE001FF80013E000100000100000100000100000 10000010000010F800130E001407001803801003800001C00001C00001E00001E00001E00001E0 7001E0F001E0F001E0E001C08001C04003C04003802007001006000C1C0003F000>19 34 126 160 32 53 D[<007E0001C1000300800601C00E03C01C03C01801803800003800007800 00700000700000F0F800F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001 E0F001E07001E07001E07001E03801C03801C01803801C03000C0600070C0001F000>19 34 126 160 32 I[<01F800060E000803001001802001802000C06000C06000C06000C07000C0 7801803E01003F02001FC4000FF80003F80003FC00067F00083F80100F803007C06001C06000E0 C000E0C00060C00060C00060C000606000406000C03000801803000E0E0003F000>19 34 126 160 32 56 D[<01F000060C000C0600180700380380700380700380F001C0F001C0F001 C0F001E0F001E0F001E0F001E0F001E07001E07003E03803E01805E00C05E00619E003E1E00001 C00001C00001C0000380000380300300780700780600700C002018001030000FC000>19 34 126 160 32 I[<0001800000018000000180000003C0000003C0000003C0000005E0000005 E000000DF0000008F0000008F0000010F800001078000010780000203C0000203C0000203C0000 401E0000401E0000401E0000800F0000800F0000FFFF000100078001000780030007C0020003C0 020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF800FFF>32 35 126 162 49 65 D[29 34 126 161 47 I[<0007E0100038183000E0063001C00170038000F0070000F00E0000701E00 00701C0000303C0000303C0000307C0000107800001078000010F8000000F8000000F8000000F8 000000F8000000F8000000F8000000F800000078000000780000107C0000103C0000103C000010 1C0000201E0000200E000040070000400380008001C0010000E0020000381C000007E000>28 36 125 162 47 I[31 34 126 161 49 I[27 34 126 161 43 70 D[<0007F008003C0C1800E0021801C001B8038000F8070000780F0000381E 0000381E0000183C0000183C0000187C0000087800000878000008F8000000F8000000F8000000 F8000000F8000000F8000000F8000000F8001FFF780000F8780000787C0000783C0000783C0000 781E0000781E0000780F00007807000078038000B801C000B800E00318003C0C080007F000>32 36 125 162 51 I[32 34 126 161 49 I[<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F00 000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00 000F00000F00000F00700F00F80F00F80F00F80E00F01E00401C0020380018700007C000>20 35 126 161 33 74 D[33 34 126 161 51 I[25 34 126 161 40 I[40 34 126 161 60 I[32 34 126 161 49 I[<000FE00000783C0000E00E0003C007800780 03C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C7800003C7800003CF8 00003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003E7800003C 7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E0078003C003C0078000E00E 0000783C00000FE000>31 36 125 162 51 I[27 34 126 161 44 I[33 35 126 161 48 82 D[<03F0200C0C601802603001E07000E0600060E00060E00060E00020E000 20E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E00001E00000 F00000F0000070800070800070800070800070C00060C00060E000C0F000C0C80180C6070081FC 00>20 36 125 162 36 I[<7FFFFFF87807807860078018400780084007800840078008C00780 0C8007800480078004800780048007800400078000000780000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000780000007800000 0780000007800000078000000780000007800000078000000FC00003FFFF00>30 34 126 161 47 I[47 35 127 161 67 87 D[<7FFFFE7E003E78003C7000786000784000F0C000F0C001E08003C08003C08007 80000780000F00001F00001E00003C00003C0000780000780000F00001F00001E00103C00103C0 010780010780030F00031E00021E00023C00063C000E78001EF8007EFFFFFE>24 34 125 161 40 90 D[<0FE0001838003C0C003C0E0018070000070000070000070000FF0007C7 001E07003C0700780700700700F00708F00708F00708F00F087817083C23900FC1E0>21 21 126 148 32 97 D[<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C0E001C0E001E0E00 1E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C80600C41C0083F00>23 35 127 162 36 I[<01FE000703000C07801C0780380300780000700000F00000F00000F00000 F00000F00000F00000F000007000007800403800401C00800C010007060001F800>18 21 126 148 29 I[<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0 F000E0F000E0F000E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE>23 35 126 162 36 I[<01FC000707000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000 F00000F00000F00000F000007000007800203800201C00400E008007030000FC00>19 21 127 148 29 I[<003C00C6018F038F030F070007000700070007000700070007000700FFF8 07000700070007000700070007000700070007000700070007000700070007000700070007807F F8>16 35 128 162 20 I[<00007001F198071E180E0E181C07001C07003C07803C07803C0780 3C07801C07001C07000E0E000F1C0019F0001000001000001800001800001FFE000FFFC00FFFE0 3800F0600030400018C00018C00018C000186000306000303800E00E038003FE00>21 33 127 149 32 I[<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7FF>24 35 127 162 36 I[<1C001E003E001E001C00000000000000000000000000000000000E00FE00 1E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0> 10 34 127 161 19 I[<01C003E003E003E001C00000000000000000000000000000000001E00F E001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E0 00E000E000E000E000E060E0F0C0F18061803E00>11 44 130 161 20 I[<0E0000FE00001E00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01 F00E01C00E01800E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07 800E03C00E01C00E01E00E00F00E00F8FFE3FE>23 35 127 162 35 I[<0E00FE001E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E00FFE0>11 35 127 162 19 I[<0E1FC07F00FE60E183801E807201C00F003C00E00F003C00E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0 0E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFE>39 21 127 148 56 I[<0E1F80FE60C01E80E00F00700F00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7FF>24 21 127 148 36 I[<01FC000707000C01801800C03800E0700070700070F00078F00078F00078 F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC00>21 21 127 148 32 I[<0E1F00FE61C00E80600F00700E00380E003C0E001C0E001E0E001E0E001E 0E001E0E001E0E001E0E001E0E003C0E003C0E00380F00700E80E00E41C00E3F000E00000E0000 0E00000E00000E00000E00000E00000E00000E0000FFE000>23 31 127 148 36 I[<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E00 0E000E000F00FFF0>16 21 127 148 25 114 D[<0F8830786018C018C008C008E008F0007F80 3FE00FF001F8003C801C800C800CC00CC008E018D0308FC0>14 21 126 148 25 I[<02000200020002000600060006000E001E003E00FFF80E000E000E000E000E000E00 0E000E000E000E000E000E040E040E040E040E040E040708030801F0>14 31 127 158 25 I[<0E0070FE07F01E00F00E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800FC7F>24 21 127 148 36 I[23 21 127 148 35 I[32 21 127 148 47 I[23 31 127 148 35 121 D E /Fp 20 118 400 360 dfs[21 4 127 148 29 45 D[53 59 124 186 68 68 D[20 59 124 186 31 73 D[65 59 124 186 81 77 D[46 59 124 186 60 80 D[<000FF00080007FFE018001F00F81 8003C001C380070000E3800E000037801C00003F803C00001F803800000F807800000F80700000 07807000000780F000000380F000000380F000000380F000000380F000000180F800000180F800 000180FC000001807C000000007E000000003F000000003FC00000001FF00000000FFF0000000F FFF0000007FFFF000001FFFFC00000FFFFF000003FFFFC000003FFFE0000007FFF00000007FF00 0000007F800000001FC00000000FC000000007E000000003E000000003E000000001F0C0000001 F0C0000001F0C0000000F0C0000000F0C0000000F0E0000000F0E0000000F0E0000000E0F00000 00E0F0000001E0F8000001C0F8000001C0FC00000380FE00000780F700000700E1C0001E00E0F0 003C00C07E00F000C00FFFE0008001FF0000>36 61 124 187 49 83 D[<003F80000001C0F000 00030038000004001C00000C001E000018000F00001C000F80003E000780003F0007C0003F0007 C0003F0007C0001E0007C000000007C000000007C000000007C00000003FC000000FE7C000007E 07C00001F007C00007E007C0000F8007C0001F0007C0003F0007C0003E0007C0007E0007C0007C 0007C060FC0007C060FC0007C060FC0007C060FC000FC060FC000FC0607C000FC0607E0017C060 3E0023E0C01F0041F18007C180FF0000FE003E00>35 37 124 164 43 97 D[<0007F800003C0E0000F0018001E000C003C00060078000300F0000701F0000F81F0001F83E 0001F83E0001F87E0000F07C0000007C000000FC000000FC000000FC000000FC000000FC000000 FC000000FC000000FC000000FC0000007C0000007C0000007E0000003E0000003E00000C1F0000 0C1F0000180F8000180780003003C0006001E000C000F00180003C0E000007F800>30 37 125 164 39 99 D[<0000000700000000FF00000007FF00000007FF000000003F000000001F 000000001F000000001F000000001F000000001F000000001F000000001F000000001F00000000 1F000000001F000000001F000000001F000000001F000000001F000000001F000000001F000000 001F000000001F000003F81F00001E061F000070019F0001E000DF0003C0007F000780003F000F 80003F000F00001F001F00001F003E00001F003E00001F007E00001F007C00001F007C00001F00 FC00001F00FC00001F00FC00001F00FC00001F00FC00001F00FC00001F00FC00001F00FC00001F 00FC00001F007C00001F007C00001F007E00001F003E00001F003E00001F001E00001F001F0000 1F000F00003F000780007F0003C0005F0001E0009F0000F0031F80003C0E1FFC0007F01FFC>38 60 125 187 49 I[<000FF00000383C0000E00F0001C00780038003C0078001E00F0001F01F00 00F01E0000F83E0000F83E0000F87C00007C7C00007C7C00007CFC00007CFC00007CFFFFFFFCFC 000000FC000000FC000000FC000000FC000000FC0000007C0000007C0000007E0000003E000000 3E00000C1E00000C1F0000180F0000180780003003C0006001E000C000F00180003C0E000007F8 00>30 37 125 164 39 I[<0000FC000003830000070380000E07C0001E0FC0003C0FC0007C0F C0007C07800078000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F8000000F8000000F8000000F8000000F80000FFFFFC00FFFFFC0000F8000000 F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F800 0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000001FC00003FFFF0003FFFF000>26 60 127 187 27 I[<00000007C0000FE01860 003838207000F01E40F001E00F80F003C007806007C007C0000F8003E0000F8003E0000F8003E0 001F8003F0001F8003F0001F8003F0001F8003F0001F8003F0001F8003F0000F8003E0000F8003 E0000F8003E00007C007C00003C007800001E00F000003F01E00000238380000060FE000000400 00000004000000000C000000000C000000000E000000000E000000000700000000078000000003 FFFF000003FFFFF00001FFFFFC0000FFFFFE00078000FF000E00001F801C000007C038000003C0 78000003C070000001E0F0000001E0F0000001E0F0000001E0F0000001E0F0000001E078000003 C038000003803C000007801E00000F000700001C0003C0007800007803C000000FFE0000>36 56 126 165 43 I[<038007C00FE00FE00FE007C0038000000000000000000000000000000000 0000000000000000000000E01FE0FFE0FFE007E003E003E003E003E003E003E003E003E003E003 E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E003E0 07F0FFFFFFFF>16 57 126 184 23 105 D[<00E01FC0007F00001FE060780181E000FFE1803C 0600F000FFE2001E0800780007E4001F10007C0003E4001F10007C0003E8000F20003C0003F000 0FC0003E0003F0000FC0003E0003F0000FC0003E0003E0000F80003E0003E0000F80003E0003E0 000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003 E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E00 03E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E 0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F80003E0003E0000F8000 3E0003E0000F80003E0003E0000F80003E0007F0001FC0007F00FFFF83FFFE0FFFF8FFFF83FFFE 0FFFF8>61 37 125 164 74 109 D[<00E03FC0001FE0C0F000FFE1007800FFE2003C0007E400 3E0003E8001E0003E8001E0003F0001F0003F0001F0003F0001F0003E0001F0003E0001F0003E0 001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003 E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F00 03E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0007F0003F80FFFF87FF FCFFFF87FFFC>38 37 125 164 49 I[<0007F00000003C1E000000F007800001C001C0000380 00E000078000F0000F000078001E00003C001E00003C003E00003E003E00003E007C00001F007C 00001F007C00001F00FC00001F80FC00001F80FC00001F80FC00001F80FC00001F80FC00001F80 FC00001F80FC00001F80FC00001F807C00001F007C00001F007C00001F003E00003E003E00003E 001E00003C001F00007C000F00007800078000F00003C001E00001C001C00000F0078000003C1E 00000007F00000>33 37 125 164 43 I[<00E0FC001FE10600FFE20F00FFE41F8007E81F8003 E81F8003F00F0003F0060003F0000003F0000003E0000003E0000003E0000003E0000003E00000 03E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E000 0003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000007F0 0000FFFFC000FFFFC000>25 37 125 164 33 114 D[<00FF02000700C6000C002E0010001E00 30001E0060000E0060000E00E0000600E0000600E0000600F0000600F8000600FC0000007F0000 003FF000003FFF80001FFFE00007FFF00001FFFC00003FFE000001FE0000003F00C0001F00C000 0F80C0000780E0000380E0000380E0000380E0000380F0000300F0000300F8000700F8000600E4 000C00E2001800C1807000807F8000>25 37 125 164 34 I[<00180000001800000018000000 1800000018000000380000003800000038000000380000007800000078000000F8000000F80000 01F8000003F8000007F800001FFFFE00FFFFFE0000F8000000F8000000F8000000F8000000F800 0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F8000000F8000000F8000000F8000000F8018000F8018000F8018000F8018000F8018000 F8018000F8018000F8018000F801800078018000780300007C0300003C0200001E0600000F0C00 0003F000>25 53 127 180 33 I[<00E00007001FE000FF00FFE007FF00FFE007FF0007E0003F 0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E000 1F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0 001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003E0001F0003 E0003F0003E0003F0001E0003F0001E0005F0001F0009F0000F0009F000078011F80001C061FFC 0007F81FFC>38 37 125 164 49 I E /Fq 5 85 691 432 dfs[<0000001C000000001C000000 003C000000003C000000007C00000000FC00000000FC00000001FC00000001FC000000037C0000 00037C000000067C0000000E7C0000000C7C000000187C000000187C000000307C000000307E00 0000603E000000603E000000C03E000001C03E000001803E000003003E000003003E000006003E 000006003E00000FFFFE00000FFFFE000018003E000038003E000030003E000060003E00006000 3E0000C0003F0000C0001F000180001F000380001F000380001F000FC0003F00FFF007FFF8FFF0 07FFF0>37 42 124 169 70 65 D[<003FFFFF8000003FFFFFF0000001F801F8000001F0007C00 0001F0003E000001F0001F000001F0000F000003E0000F800003E0000F800003E0000F800003E0 0007800007C00007C00007C00007C00007C00007C00007C0000FC0000F80000FC0000F80000FC0 000F80000FC0000F80000FC0001F00000F80001F00001F80001F00001F80001F00001F80003E00 001F00003E00003F00003E00003E00003E00003E00007C00007C00007C00007C00007C0000F800 007C0000F80000F80001F00000F80003E00000F80003C00000F80007800001F0000F000001F000 3E000001F0007C000003F003F000007FFFFFC00000FFFFFE000000>42 41 124 168 72 68 D[<003FFFFFFF003FFFFFFF0001F8003F0001F0000F0001F0000E0001F0000E 0001F000060003E000060003E000060003E0000E0003E0000C0007C0000C0007C0180C0007C018 0C0007C01800000F803000000F803000000F807000000F80F000001FFFE000001FFFE000001F01 E000001F00E000003E00C000003E00C000003E00C000003E00C000007C018000007C000000007C 000000007C00000000F800000000F800000000F800000000F800000001F000000001F000000001 F000000003F00000007FFFC00000FFFFC00000>40 41 124 168 62 70 D[<003FFFFE00003FFFFFC00001F803F00001F001F80001F000F80001F0007C0001F0007C0003 E0007E0003E0007E0003E0007E0003E0007E0007C000FC0007C000FC0007C000F80007C001F800 0F8001F0000F8003E0000F8007C0000F800F00001F007C00001FFFE000001F00E000001F007800 003E003C00003E003C00003E003E00003E003E00007C003E00007C003E00007C003E00007C003E 0000F8007E0000F8007E0000F8007E0000F8007E0001F0007E0601F0007E0601F0007E0603F000 7E0C7FFF803E08FFFF801E1000000007E0>39 42 124 168 70 82 D[<0FFFFFFFF80FFFFFFFF8 1F803E01F81E007C00781C007C003038007C003038007C00303000F800303000F800306000F800 306000F800606001F00060C001F00060C001F000600001F000000003E000000003E000000003E0 00000003E000000007C000000007C000000007C000000007C00000000F800000000F800000000F 800000000F800000001F000000001F000000001F000000001F000000003E000000003E00000000 3E000000003E000000007C000000007C000000007C00000000FC000000FFFFF80000FFFFF80000 >37 41 117 168 69 84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 400 TeXDict begin %%EndSetup %%Page: 0 1 bop 1060 927 a Fq(D)34 b(R)g(A)h(F)g(T)299 1049 y Fp(Do)r(cumen)n(t)30 b(for)g(a)g(Standard)h(Message-P)n(assi)q(ng)i(In)n(terface)913 1308 y Fo(Scott)23 b(Berryman,)c Fn(Y)-5 b(ale)25 b(Univ)933 1386 y Fo(James)c(Co)n(wnie,)g Fn(Meiko)j(Ltd)632 1463 y Fo(Jac)n(k)e (Dongarra,)i Fn(Univ.)32 b(of)23 b(T)-5 b(ennesse)m(e)26 b(and)f(ORNL)1068 1541 y Fo(Al)c(Geist,)f Fn(ORNL)1060 1618 y Fo(Bill)e(Gropp,)k Fn(ANL)1023 1696 y Fo(Rolf)f(Hemp)r(el,)c Fn(GMD)1016 1773 y Fo(Bob)22 b(Knigh)n(ten,)e Fn(Intel)1049 1851 y Fo(Rust)n(y)i(Lusk,)f Fn(ANL)818 1928 y Fo(Stev)n(e)h(Otto,)f Fn(Or)m(e)m(gon)k(Gr)m(aduate)f(Inst) 771 2006 y Fo(T)-5 b(on)n(y)21 b(Skjellum,)c Fn(Missisippi)k(State)26 b(Univ)871 2083 y Fo(Marc)c(Snir,)e Fn(IBM)j(T.)g(J.)f(Watson)992 2161 y Fo(Da)n(vid)g(W)-5 b(alk)n(er,)19 b Fn(ORNL)835 2238 y Fo(Stev)n(e)i(Zenith,)f Fn(Kuck)26 b(&)e(Asso)m(ciates)1099 2404 y Fo(April)c(20,)h(1993)116 2482 y(This)g(w)n(ork)h(w)n(as)h(supp)r (orted)h(b)n(y)d(ARP)-5 b(A)21 b(and)i(NSF)e(under)h(con)n(tract)i(n)n(um)n (b)r(er)c(###,)f(b)n(y)i(the)256 2559 y(National)h(Science)e(F)-5 b(oundation)23 b(Science)d(and)j(T)-5 b(ec)n(hnology)21 b(Cen)n(ter)h(Co)r (op)r(erativ)n(e)867 2636 y(Agreemen)n(t)f(No.)28 b(CCR-8809615.)p eop %%Page: 1 2 bop 100 477 a Fm(Chapter)44 b(1)100 756 y Fl(Colle)o(c)o(ti)n(v)l(e)k(Com)n (m)-10 b(uni)n(c)o(ati)o(on)1189 1054 y Fk(Al)20 b(Geist)1168 1129 y(Marc)g(Snir)100 1331 y Fj(1.1)94 b(In)m(tro)s(duction)100 1469 y Fk(This)28 b(section)h(is)f(a)g(draft)j(of)e(the)g(curren)n(t)i(prop)r (osal)g(for)e(collectiv)n(e)f(comm)n(unication.)51 b(Collectiv)n(e)100 1545 y(comm)n(unication)15 b(is)f(de\014ned)k(to)d(b)r(e)g(comm)n(unication)g (that)i(in)n(v)n(olv)n(es)f(a)f(group)i(of)f(pro)r(cesses.)26 b(Examples)100 1620 y(are)20 b(broadcast)j(and)e(global)f(sum.)26 b(A)18 b(collectiv)n(e)i(op)r(eration)i(is)d(executed)i(b)n(y)f(ha)n(ving)h (all)e(pro)r(cesses)h(in)100 1695 y(the)k(group)i(call)e(the)g(comm)n (unication)g(routine,)i(with)e(matc)n(hing)h(parameters.)38 b(Routines)24 b(can)h(\(but)100 1770 y(are)f(not)g(required)i(to\))e(return)j (as)c(so)r(on)h(as)f(their)h(participation)k(in)23 b(the)h(collectiv)n(e)g (comm)n(unication)100 1846 y(is)i(complete.)43 b(The)27 b(completion)f(of)h (a)f(call)g(indicates)i(that)g(the)f(caller)f(is)g(no)n(w)h(free)g(to)g (access)f(the)100 1921 y(lo)r(cations)j(in)g(the)g(comm)n(unication)f (bu\013er,)k(or)d(an)n(y)g(other)i(lo)r(cation)e(that)h(can)f(b)r(e)f (referenced)j(b)n(y)100 1996 y(the)26 b(collectiv)n(e)e(op)r(eration.)43 b(Ho)n(w)n(ev)n(er,)26 b(it)f(do)r(es)g(not)h(indicate)g(that)h(other)f(pro)r (cesses)g(in)f(the)g(group)100 2072 y(ha)n(v)n(e)c(started)i(the)d(op)r (eration)j(\(unless)e(otherwise)g(indicated)i(in)d(the)h(description)h(of)f (the)g(op)r(eration\).)100 2147 y(Ho)n(w)n(ev)n(er,)i(the)h(successful)f (completion)g(of)g(a)f(collectiv)n(e)h(comm)n(unication)f(call)g(ma)n(y)f (dep)r(end)k(on)e(the)100 2222 y(execution)f(of)f(a)g(matc)n(hing)g(call)f (at)h(all)f(pro)r(cesses)h(in)g(the)g(group.)221 2299 y(The)g(syn)n(tax)h (and)f(seman)n(tics)g(of)g(the)g(collectiv)n(e)f(op)r(erations)k(is)19 b(de\014ned)k(so)e(as)f(to)h(b)r(e)f(consisten)n(t)100 2374 y(with)h(the)g(syn)n(tax)h(and)g(seman)n(tics)e(of)h(the)h(p)r(oin)n(t)g(to)f (p)r(oin)n(t)h(op)r(erations.)221 2452 y(The)e(reader)h(is)e(referred)j(to)e (the)g(p)r(oin)n(t-to-p)r(oin)o(t)k(comm)n(unication)19 b(section)i(of)e(the) i(curren)n(t)h(MPI)100 2527 y(draft)g(for)e(information)i(concerning)g(comm)n (unication)d(bu\013ers)j(and)e(their)h(manipulations.)28 b(The)20 b(con-)100 2602 y(text)d(section)h(describ)r(es)f(the)h(formation,)g (manipulation,)h(and)f(query)g(functions)h(\(suc)n(h)f(as)f(group)i(size\)) 100 2677 y(that)j(are)f(a)n(v)m(ailable)h(for)f(groups)i(and)f(group)h(ob)s (jects.)221 2754 y(The)31 b(collectiv)n(e)g(comm)n(unication)g(routines)j (are)d(built)i(ab)r(o)n(v)n(e)f(the)g(p)r(oin)n(t-to-p)r(oin)n(t)k(routines.) 100 2830 y(While)17 b(v)n(endors)h(ma)n(y)e(optimize)f(certain)j(collectiv)n (e)f(routines)i(for)e(their)h(arc)n(hitectures,)j(a)16 b(complete)g(li-)100 2905 y(brary)22 b(of)e(the)h(collectiv)n(e)e(comm)n(unication)h(routines)i (can)e(b)r(e)g(written)i(en)n(tirely)f(using)g(p)r(oin)n(t-to-p)r(oin)n(t)100 2980 y(comm)n(unication)28 b(functions.)51 b(W)-5 b(e)27 b(are)h(using)h (naiv)n(e)f(implemen)n(tations)g(of)g(the)h(collectiv)n(e)e(calls)g(in)100 3056 y(terms)f(of)i(p)r(oin)n(t)h(to)e(p)r(oin)n(t)i(op)r(erations)h(in)d (order)i(to)f(pro)n(vide)h(an)f(op)r(erational)i(de\014nition)g(of)d(their) 100 3131 y(seman)n(tics.)221 3208 y(The)21 b(follo)n(wing)h(comm)n(unication) e(functions)k(are)d(prop)r(osed.)191 3340 y Fi(\017)31 b Fk(Broadcast)22 b(from)f(one)g(mem)n(b)r(er)d(to)j(all)f(mem)n(b)r(ers)f(of)i(a)f(group.)191 3472 y Fi(\017)31 b Fk(Barrier)22 b(across)f(all)f(group)j(mem)n(b)r(ers)191 3605 y Fi(\017)31 b Fk(Gather)21 b(data)i(from)d(all)g(group)j(mem)n(b)r(ers) 18 b(to)j(one)g(mem)n(b)r(er.)1285 3771 y(1)p eop %%Page: 2 3 bop 100 -134 a Fk(2)968 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)191 60 y Fi(\017)31 b Fk(Scatter)22 b(data)g(from)f (one)g(mem)n(b)r(er)d(to)j(all)g(mem)n(b)r(ers)d(of)j(a)g(group.)191 183 y Fi(\017)31 b Fk(Global)20 b(op)r(erations)j(suc)n(h)e(as)f(sum,)f(max,) g(min,)g(etc.,)h(w)n(ere)g(the)i(result)f(is)f(kno)n(wn)h(b)n(y)g(all)f (group)252 258 y(mem)n(b)r(ers)d(and)k(a)f(v)m(ariation)i(where)e(the)h (result)g(is)f(kno)n(wn)h(b)n(y)f(only)h(one)f(mem)n(b)r(er.)25 b(The)20 b(abilit)n(y)252 333 y(to)h(ha)n(v)n(e)g(user)h(de\014ned)h(global)e (op)r(erations.)191 456 y Fi(\017)31 b Fk(Sim)n(ultaneous)20 b(shift)f(of)f(data)i(around)h(the)e(group,)i(the)e(simplest)e(example)g(b)r (eing)i(all)f(mem)n(b)r(ers)252 531 y(sending)k(their)g(data)g(to)f (\(rank+1\))i(with)e(wrap)h(around.)191 653 y Fi(\017)31 b Fk(Scan)21 b(across)h(all)e(mem)n(b)r(ers)e(of)j(a)g(group)i(\(also)e(called) f(parallel)i(pre\014x\).)191 776 y Fi(\017)31 b Fk(Broadcast)22 b(from)f(all)f(mem)n(b)r(ers)e(to)j(all)f(mem)n(b)r(ers)f(of)i(a)f(group.)191 899 y Fi(\017)31 b Fk(Scatter)c(data)f(from)f(all)f(mem)n(b)r(ers)f(to)i(all) f(mem)n(b)r(ers)f(of)i(a)g(group)i(\(also)f(called)f(complete)f(ex-)252 974 y(c)n(hange)e(or)f(index\).)221 1092 y(T)-5 b(o)21 b(simplify)f(the)h (collectiv)n(e)g(comm)n(unication)g(in)n(terface)j(it)d(is)f(designed)j(with) e(t)n(w)n(o)h(la)n(y)n(ers.)30 b(The)100 1168 y(lo)n(w)c(lev)n(el)f(routines) k(ha)n(v)n(e)e(all)e(the)i(generalit)n(y)h(of,)g(and)f(mak)n(e)e(use)h(of,)i (the)f(comm)n(unication)f(bu\013er)100 1243 y(routines)c(of)f(the)g(p)r(oin)n (t-to-p)r(oin)n(t)k(section)20 b(whic)n(h)h(allo)n(ws)f(arbitrarily)j (complex)c(messages)f(to)j(b)r(e)f(con-)100 1318 y(structed.)29 b(The)20 b(second)g(lev)n(el)f(routines)j(are)d(similar)f(to)i(the)g(upp)r (er)i(lev)n(el)d(p)r(oin)n(t-to-p)r(oin)n(t)24 b(routines)d(in)100 1394 y(that)h(they)g(send)f(only)g(a)g(con)n(tiguous)i(bu\013er.)100 1583 y Fj(1.2)94 b(Group)32 b(F)-8 b(unctions)100 1719 y Fk(A)22 b(full)h(description)i(of)e(the)g(group)i(formation)f(and)f(manipulation)i (functions)g(can)e(b)r(e)f(found)j(in)e(the)100 1794 y(con)n(text)j(c)n (hapter)g(of)f(the)f(MPI)e(do)r(cumen)n(t.)38 b(Here)24 b(w)n(e)f(describ)r (e)i(only)f(those)h(group)h(functions)h(that)100 1869 y(are)21 b(used)h(in)e(the)i(seman)n(tic)e(description)j(of)e(the)g(collectiv)n(e)g (comm)n(unication)g(routines.)221 1944 y(An)h(initial)g(group)j(con)n (taining)g(all)c(pro)r(cesses)i(is)e(supplied)j(b)n(y)f(default)h(in)e(MPI.)e (MPI)h(pro)n(vides)100 2020 y(a)f(pro)r(cedure)j(that)f(returns)h(the)e (handle)h(to)e(this)h(initial)f(group.)30 b(The)20 b(pro)r(cesses)g(in)h(the) g(inital)f(group)100 2095 y(eac)n(h)30 b(ha)n(v)n(e)g(a)g(unique)g(rank)h(in) e(the)h(group)i(represen)n(ted)h(b)n(y)d(in)n(tegers)h(\(0,)h(1,)f(2,)g(...,) f([n)n(um)n(b)r(er-of-)100 2170 y(pro)r(cesses)21 b(-)g(1].)221 2293 y Fg(MPI)p 365 2293 21 3 v 26 w(ALLGR)n(OUP\(group\))f Fk(Return)k(the)f(descriptor)h(of)f(the)f(inital)h(group)h(con)n(taining)h (all)100 2415 y(pro)r(cesses.)100 2534 y Fg(OUT)d(group)32 b Fk(handle)22 b(to)f(descriptor)j(ob)s(ject)e(of)f(initial)g(group.)221 2700 y Fg(MPI)p 365 2700 V 26 w(RANK\(group,)i(rank\))e Fk(Return)h(the)g (rank)f(of)g(the)g(calling)g(pro)r(cess)g(within)h(the)f(sp)r(eci-)100 2822 y(\014ed)g(group.)100 2941 y Fg(IN)i(group)32 b Fk(group)23 b(handle)100 3063 y Fg(OUT)f(rank)31 b Fk(in)n(teger)221 3229 y Fg(MPI)p 365 3229 V 26 w(GSIZE\(group,)25 b(size\))20 b Fk(Return)j(the)e (n)n(um)n(b)r(er)h(of)g(pro)r(cesses)f(that)i(b)r(elong)f(to)f(the)h(sp)r (ec-)100 3352 y(i\014ed)f(group.)100 3470 y Fg(IN)i(group)32 b Fk(group)23 b(handle)100 3593 y Fg(OUT)f(size)30 b Fk(in)n(teger)p eop %%Page: 3 4 bop 100 -134 a Fh(1.3.)47 b(COMMUNICA)-5 b(TION)17 b(FUNCTIONS)1283 b Fk(3)100 60 y Fj(1.3)94 b(Comm)-6 b(uni)n(cati)o(on)27 b(F)-8 b(unctions)100 196 y Fk(The)15 b(prop)r(osed)i(comm)n(unication)e(functions)j (are)d(divided)h(in)n(to)g(t)n(w)n(o)g(la)n(y)n(ers.)26 b(The)15 b(lo)n(w)n(est)h(lev)n(el)e(uses)h(the)100 271 y(same)j(bu\013er)j (descriptor)h(ob)s(jects)f(a)n(v)m(ailable)f(in)f(p)r(oin)n(t-to-p)r(oin)o(t) 24 b(to)19 b(create)i(noncon)n(tiguou)q(s,)i(m)n(ultiple)100 346 y(data)k(t)n(yp)r(e)g(messages.)40 b(The)26 b(second)g(lev)n(el)g(is)f (similar)f(to)i(the)g(blo)r(c)n(k)g(send/receiv)n(e)i(p)r(oin)n(t-to-p)r(oin) n(t)100 422 y(op)r(erations)h(in)e(that)h(it)f(supp)r(orts)i(only)e(con)n (tiguous)j(bu\013ers)e(of)f(data.)47 b(F)-5 b(or)26 b(eac)n(h)i(comm)n (unication)100 497 y(op)r(eration,)23 b(w)n(e)d(list)h(these)g(t)n(w)n(o)g (lev)n(el)f(of)h(calls)f(together.)100 689 y Fj(1.4)94 b(Sync)m(hronization) 100 825 y Fg(Barrier)25 b(sync)n(hronization)100 987 y(MPI)p 244 987 21 3 v 25 w(BARRIER\()c(group)k(\))221 1110 y Fk(MPI)p 345 1110 19 3 v 21 w(BARRIER)18 b(blo)r(c)n(ks)k(the)h(calling)e(pro)r(cess)i (un)n(til)g(all)e(group)j(mem)n(b)r(ers)19 b(ha)n(v)n(e)k(called)f(it;)g(the) 100 1185 y(call)e(returns)k(at)d(an)n(y)g(pro)r(cess)g(only)g(after)i(all)d (group)j(mem)n(b)r(ers)18 b(ha)n(v)n(e)k(en)n(tered)h(the)e(call.)100 1311 y Fg(IN)i(group)32 b Fk(group)23 b(handle)221 1437 y Ff(MPI)p 318 1437 20 3 v 26 w(BARRIER)q(\()37 b(group)f(\))21 b Fk(is)100 1564 y Ff(MPI_CR)q(E)q(A)q(TE)q(_)q(B)q(U)q(FF)q(E)q(R)q(\()q(b)q(uf)q(f)q(e) q(r)q(_h)q(a)q(n)q(d)q(l)q(e,)37 b(MPI_BU)q(F)q(F)q(ER)q(,)g(MPI_PE)q(R)q(SI) q(S)q(T)q(E)q(NT)q(\))q(;)100 1639 y(MPI_GS)q(I)q(Z)q(E\()g(group,)g(&size)f (\);)100 1714 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 1789 y(if)d(\(rank)q(==)q(0)q(\))100 1865 y({)195 1940 y(for)i(\(i=1;)h(i)c (<)h(size;)j(i++\))291 2015 y(MPI_RE)q(C)q(V\()q(b)q(u)q(f)q(f)q(er)q(_)q(h)q (a)q(nd)q(l)q(e)q(,)h(i,)c(tag,)i(group,)i(return)q(_)q(ha)q(n)q(d)q(l)q(e\)) q(;)195 2091 y(for)e(\(i=1;)h(i)c(<)h(size;)j(i++\))291 2166 y(MPI_SE)q(N)q(D\()q(b)q(u)q(f)q(f)q(er)q(_)q(h)q(a)q(nd)q(l)q(e)q(,)h(i,)c (tag,)i(group\))q(;)100 2241 y(})100 2316 y(else)100 2392 y({)195 2467 y(MPI_S)q(EN)q(D)q(\()q(b)q(uf)q(f)q(e)q(r)q(_)q(ha)q(n)q(d)q(l)q(e,)i (0,)d(tag,)h(group\))q(;)195 2542 y(MPI_R)q(EC)q(V)q(\()q(b)q(uf)q(f)q(e)q(r) q(_)q(ha)q(n)q(d)q(l)q(e,)i(0,)d(tag,)h(group,)h(return)q(_)q(h)q(a)q(n)q(dl) q(e)q(\))q(;)100 2617 y(})100 2693 y(MPI_FR)q(E)q(E)q(\(b)q(u)q(f)q(f)q(er)q (_)q(h)q(a)q(n)q(dl)q(e)q(\))q(;)221 2929 y Fe(Discussion:)45 b Fd(since)17 b(tag)g(w)n(as)g(v)n(oted)h(out)f(in)f(the)i(last)f(meeting,)f (a)h("safe")f(tag)h(m)n(ust)g(b)r(e)g(selected)i(to)e(use)100 3004 y(in)h(the)h(ab)r(o)n(v)n(e)g(description)100 3307 y Fj(1.5)94 b(Data)30 b(mo)m(v)m(e)e(functions)100 3443 y Fg(Broadcast)100 3605 y(MPI)p 244 3605 21 3 v 25 w(BCAST\()21 b(bu\013er)p 738 3605 V 25 w(handle,)i(group,)j(ro)r(ot)f(\))p eop %%Page: 4 5 bop 100 -134 a Fk(4)968 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)221 60 y Ff(MPI)p 318 60 20 3 v 26 w(BCAST)20 b Fk(broadcasts)g(a)d(message)f(from)h(the)g(pro)r(cess)h(with)g(rank)g Ff(root)i Fk(to)e(all)e(other)j(pro)r(cesses)100 135 y(of)25 b(the)h(group.)41 b(It)25 b(is)f(called)h(b)n(y)g(all)f(mem)n(b)r(ers)f(of)i (group)i(using)f(the)f(same)e(argumen)n(ts)j(for)g Ff(group,)100 211 y(and)34 b(root)p Fk(.)c(On)21 b(return)j(the)d(con)n(ten)n(ts)i(of)e (the)h(bu\013er)g(of)f(the)g(pro)r(cess)h(with)f(rank)g Ff(root)j Fk(is)c(con)n(tained)100 286 y(in)h(bu\013er)h(of)f(all)f(group)j(mem)n(b)r (ers.)100 422 y Fg(INOUT)f(bu\013er)p 542 422 21 3 v 25 w(handle)29 b Fk(Handle)21 b(for)h(bu\013er)h(where)e(from)f(message)f(is)h(sen)n(t)i(or) f(receiv)n(ed.)100 558 y Fg(IN)i(group)32 b Fk(group)23 b(handle)100 694 y Fg(IN)g(ro)r(ot)33 b Fk(rank)22 b(of)f(broadcast)i(ro)r(ot)f(\(in)n (teger\))221 877 y Fg(MPI)p 365 877 V 26 w(BCASTC\()e(buf,)j(len,)g(group,)i (ro)r(ot)h(\))221 1002 y Ff(MPI)p 318 1002 20 3 v 26 w(BCASTC)33 b Fk(b)r(eha)n(v)n(es)c(lik)n(e)f(broadcast,)34 b(restricted)d(to)e(a)f(blo)r (c)n(k)h(bu\013er.)53 b(It)28 b(is)g(called)h(b)n(y)g(all)100 1078 y(pro)r(cesses)21 b(with)g(the)h(same)c(argumen)n(ts)23 b(for)e Ff(len,)35 b(group)25 b Fk(and)d Ff(root)p Fk(.)100 1214 y Fg(INOUT)g(bu\013er)30 b Fk(Starting)24 b(address)e(of)f(bu\013er)h (\(c)n(hoice)g(t)n(yp)r(e\))100 1350 y Fg(IN)h(len)30 b Fk(Num)n(b)r(er)20 b(of)h(w)n(ords)h(in)e(bu\013er)j(\(in)n(teger\))100 1486 y Fg(IN)g(group)32 b Fk(group)23 b(handle)100 1622 y Fg(in)f(ro)r(ot)33 b Fk(rank)22 b(of)f(broadcast)j(ro)r(ot)d(\(in)n(teger\))221 1758 y Ff(MPI)p 318 1758 V 26 w(BCAST\()36 b(buffer)p 753 1758 V 28 w(handle)q(,)h(group,)f(root)f(\))21 b Fk(is)100 1896 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f(\);)100 1972 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 2047 y(MPI_IR)q(E)q(C)q(V\() q(h)q(a)q(n)q(dl)q(e)q(,)h(buffer)q(_h)q(a)q(n)q(d)q(l)q(e,)g(root,)f(tag,)f (group,)i(return)q(_h)q(a)q(n)q(d)q(l)q(e\))q(;)100 2122 y(if)c(\(rank)q(==)q (r)q(o)q(o)q(t\))195 2197 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))291 2273 y(MPI_SE)q(N)q(D\()q(b)q(u)q(f)q(f)q(er)q(_)q(h)q(a)q(nd)q(l)q(e)q(,)h (i,)c(tag,)i(group\))q(;)100 2348 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n)q(d)q(le)q (\))100 2524 y Fg(Circular)23 b(shift)100 2690 y(MPI)p 244 2690 21 3 v 25 w(CSHIFT\()f(in)n(buf,)h(outbuf,)h(group,)h(shift\))221 2816 y Fk(Pro)r(cess)30 b(with)h(rank)h Ff(i)f Fk(sends)g(the)g(data)h(in)f (its)f(input)j(bu\013er)f(to)f(pro)r(cess)h(with)f(rank)g(\()p Ff(i)22 b Fk(+)100 2891 y Ff(shift)p Fk(\))f(mo)r(d)16 b Ff(group)p 592 2891 20 3 v 27 w(size)p Fk(,)22 b(who)c(receiv)n(es)h(the)g(data)g(in)g (its)f(output)j(bu\013er.)28 b(All)17 b(pro)r(cesses)i(mak)n(e)e(the)100 2966 y(call)k(with)g(the)h(same)e(v)m(alues)h(for)h Ff(group)p Fk(,)j(and)d Ff(shift)p Fk(.)33 b(The)21 b Ff(shift)k Fk(v)m(alue)c(can)g(b)r (e)h(p)r(ositiv)n(e,)f(zero,)h(or)100 3042 y(negativ)n(e.)100 3197 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h (descriptor)100 3333 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i (bu\013er)g(descriptor)100 3469 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 3605 y Fg(IN)i(shift)30 b Fk(in)n(teger)p eop %%Page: 5 6 bop 100 -134 a Fh(1.5.)47 b(D)n(A)-5 b(T)g(A)19 b(MO)n(VE)g(FUNCTIONS)1463 b Fk(5)221 60 y Fg(MPI)p 365 60 21 3 v 26 w(CSHIFT1\()22 b(buf,)h(group,)j (shift\))221 189 y Fk(Pro)r(cess)19 b(with)i(rank)g Ff(i)f Fk(sends)h(the)g(data)g(in)f(its)g(bu\013er)i(to)e(pro)r(cess)h(with)f(rank)i (\()p Ff(i)13 b Fk(+)f Ff(shift)p Fk(\))22 b(mo)r(d)100 264 y Ff(group)p 259 264 20 3 v 27 w(size)p Fk(,)g(who)e(receiv)n(es)f(the)h (data)g(in)g(the)f(same)f(bu\013er.)28 b(All)18 b(pro)r(cesses)i(mak)n(e)e (the)i(call)e(with)i(the)100 339 y(same)f(v)m(alues)h(for)i Ff(group)p Fk(,)i(and)e Ff(shift)p Fk(.)31 b(The)20 b Ff(shift)25 b Fk(v)m(alue)20 b(can)h(b)r(e)g(p)r(ositiv)n(e,)g(zero,)g(or)g(negativ)n(e.) 100 512 y Fg(INOUT)h(buf)30 b Fk(handle)22 b(to)f(bu\013er)i(descriptor)100 662 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 813 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 1033 y Fg(MPI)p 365 1033 21 3 v 26 w(CSHIFTC\()21 b(in)n(buf,)h(outbuf,)i(len,)f(group,)i (shift\))221 1161 y Fk(Beha)n(v)n(es)j(lik)n(e)e Ff(MPI)p 675 1161 20 3 v 26 w(CSHIFT)q Fk(,)k(with)d(bu\013ers)i(restricted)g(to)f(b)r(e)e (blo)r(c)n(ks)h(of)h(n)n(umeric)f(units.)47 b(All)100 1237 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m(alues)i (for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 1387 y Fg(IN)i(in)n(buf)29 b Fk(initial)21 b(lo)r(cation)h(of)f(input)i (bu\013er)100 1537 y Fg(OUT)f(outbuf)30 b Fk(initial)21 b(lo)r(cation)h(of)f (output)j(bu\013er)100 1687 y Fg(IN)f(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(\(and)f(output\))i(bu\013ers)100 1837 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 1987 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 2184 y Fg(MPI)p 365 2184 21 3 v 26 w(CSHIFTC1\()21 b(buf,)j(len,)e(group,)k(shift\))221 2313 y Fk(Beha)n(v)n(es)g(lik)n(e)e Ff(MPI)p 671 2313 20 3 v 25 w(CSHIFT)q(1)q Fk(,)29 b(with)c(bu\013ers)h(restricted)h(to)e(b)r(e)f (blo)r(c)n(ks)h(of)g(n)n(umeric)g(units.)40 b(All)100 2389 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m(alues)i (for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 2539 y Fg(INOUT)h(buf)30 b Fk(initial)21 b(lo)r(cation)g(of)g(bu\013er)100 2689 y Fg(IN)i(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input) i(\(and)f(output\))i(bu\013ers)100 2839 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 2989 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 3139 y Ff(MPI)p 318 3139 V 26 w(CSHIFT\()37 b(inbuf,)g(outbuf)q(,)f(group,)h (shift\))25 b Fk(is)100 3295 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f (\);)100 3371 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 3446 y(MPI_IS)q(E)q(N)q(D\()h(handle)q(,)g(inbuf,)f(mod\(ra)q(n)q(k)q(+)q(sh) q(i)q(f)q(t)q(,)g(size\),)h(tag,)e(group\))q(;)100 3521 y(MPI_RE)q(C)q(V)q (\()h(outbuf)q(,)h(mod\(ra)q(nk)q(-)q(s)q(h)q(i)q(ft)q(,)q(s)q(i)q(ze)q(\))q (,)g(tag,)e(group,)i(return_)q(h)q(a)q(n)q(d)q(le)q(\))100 3596 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n)q(d)q(le)q(\))q(;)p eop %%Page: 6 7 bop 100 -134 a Fk(6)968 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fg(End-o\013)23 b(shift)100 222 y(MPI)p 244 222 21 3 v 25 w(EOSHIFT\()f(in)n(buf,)g(outbuf,)i(group,)i (shift\))221 344 y Fk(Pro)r(cess)21 b(with)h(rank)h Ff(i)p Fk(,)f(max)o(\()p Ff(0)p Fc(;)10 b Fi(\000)p Ff(shif)q(t)q Fk(\))23 b Fi(\024)18 b Ff(i)i Fc(<)e Ff(min)p Fk(\()p Ff(si)q(z)q(e)q Fc(;)10 b Ff(s)q(iz)q(e)19 b Fi(\000)14 b Ff(shift)q Fk(\),)25 b(sends)e(the)f(data)100 420 y(in)17 b(its)f(input)j(bu\013er)g(to)e(pro)r (cess)g(with)g(rank)h Ff(i+)34 b(shift)p Fk(,)20 b(who)d(receiv)n(es)g(the)h (data)g(in)f(its)f(output)k(bu\013er.)100 495 y(The)j(output)j(bu\013er)f(of) e(pro)r(cesses)g(whic)n(h)h(do)g(not)g(receiv)n(e)f(data)h(is)e(left)i(unc)n (hanged.)37 b(All)22 b(pro)r(cesses)100 570 y(mak)n(e)d(the)j(call)e(with)h (the)g(same)e(v)m(alues)h(for)i Ff(group)p Fk(,)i(and)e Ff(shift)p Fk(.)100 707 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h (descriptor)100 831 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i (bu\013er)g(descriptor)100 954 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 1078 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 1262 y Fg(MPI)p 365 1262 V 26 w(EOSHIFT1\()22 b(buf,)h(group,)j(shift\))221 1385 y Fk(Pro)r(cess)21 b(with)h(rank)h Ff(i)p Fk(,)f(max)o(\()p Ff(0)p Fc(;)10 b Fi(\000)p Ff(shif)q(t)q Fk(\))23 b Fi(\024)18 b Ff(i)i Fc(<)e Ff(min)p Fk(\()p Ff(si)q(z)q(e)q Fc(;)10 b Ff(s)q(iz)q(e)19 b Fi(\000)14 b Ff(shift)q Fk(\),)25 b(sends)e(the)f(data)100 1460 y(in)e(its)h(bu\013er)h(to)e(pro)r(cess)h(with)g(rank)h Ff(i+)33 b(shift)p Fk(,)24 b(who)d(receiv)n(es)f(the)h(data)h(in)e(the)h (same)e(bu\013er.)29 b(The)100 1535 y(output)24 b(bu\013er)f(of)e(pro)r (cesses)g(whic)n(h)h(do)f(not)h(receiv)n(e)f(data)h(is)e(left)i(unc)n (hanged.)31 b(All)20 b(pro)r(cesses)h(mak)n(e)100 1610 y(the)g(call)f(with)i (the)f(same)e(v)m(alues)h(for)i Ff(group)p Fk(,)i(and)e Ff(shift)p Fk(.)100 1747 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h (descriptor)100 1871 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i (bu\013er)g(descriptor)100 1995 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 2118 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 2302 y Fg(MPI)p 365 2302 V 26 w(EOSHIFTC\()20 b(in)n(buf,)j(outbuf,)h(len,)e (group,)k(shift\))221 2425 y Fk(Beha)n(v)n(es)g(lik)n(e)e Ff(MPI)p 671 2425 20 3 v 25 w(EOSHIF)q(T)q Fk(,)k(with)d(bu\013ers)h(restricted)h(to)e (b)r(e)g(blo)r(c)n(ks)g(of)g(n)n(umeric)g(units.)40 b(All)100 2500 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m (alues)i(for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 2622 y Fg(IN)i(in)n(buf)29 b Fk(initial)21 b(lo)r(cation)h(of)f (input)i(bu\013er)100 2745 y Fg(OUT)f(outbuf)30 b Fk(initial)21 b(lo)r(cation)h(of)f(output)j(bu\013er)100 2869 y Fg(IN)f(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(\(and)f(output\))i (bu\013ers)100 2993 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 3117 y Fg(IN)i(shift)30 b Fk(in)n(teger)221 3285 y Fg(MPI)p 365 3285 21 3 v 26 w(EOSHIFTC1\()21 b(buf,)j(len,)e(group,)k(shift\))221 3408 y Fk(Beha)n(v)n(es)f(lik)n(e)f Ff(MPI)p 670 3408 20 3 v 25 w(EOSHIF)q(T)q(1)p Fk(,)29 b(with)24 b(bu\013er)i(restricted)g(to)f(b)r (e)f(blo)r(c)n(ks)g(of)g(n)n(umeric)g(units.)39 b(All)100 3483 y(pro)r(cesses)21 b(mak)n(e)e(the)j(call)e(with)h(the)g(same)e(v)m(alues)i (for)g Ff(len,)35 b(group)q Fk(,)24 b(and)d Ff(shift)q Fk(.)100 3605 y Fg(INOUT)h(buf)30 b Fk(initial)21 b(lo)r(cation)g(of)g(bu\013er)p eop %%Page: 7 8 bop 100 -134 a Fh(1.5.)47 b(D)n(A)-5 b(T)g(A)19 b(MO)n(VE)g(FUNCTIONS)1463 b Fk(7)100 60 y Fg(IN)23 b(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h (in)f(input)i(\(and)f(output\))i(bu\013ers)100 182 y Fg(IN)f(group)32 b Fk(handle)23 b(to)e(group)100 303 y Fg(IN)i(shift)30 b Fk(in)n(teger)100 461 y Fg(Gather)100 623 y(MPI)p 244 623 21 3 v 25 w(GA)-6 b(THER\()21 b(in)n(buf,)i(outbuf,)h(group,)h(ro)r(ot,)h(return)p 1643 623 V 26 w(status\))221 746 y Fk(Eac)n(h)d(pro)r(cess)h(\(including)h(the)f(ro)r (ot)g(pro)r(cess\))g(sends)g(the)g(con)n(ten)n(t)h(of)f(its)f(input)i (bu\013er)f(to)g(the)100 821 y(ro)r(ot)f(pro)r(cess.)32 b(The)22 b(ro)r(ot)h(pro)r(cess)f(concatenates)j(all)c(the)i(incoming)f(messages)e(in) i(the)h(order)g(of)g(the)100 896 y(senders')18 b(rank)h(and)f(places)f(the)h (results)g(in)f(its)g(output)j(bu\013er.)28 b(It)17 b(is)g(called)g(b)n(y)g (all)g(mem)n(b)r(ers)e(of)i(group)100 971 y(using)k(the)f(same)e(argumen)n (ts)j(for)f Ff(group)p Fk(,)k(and)c Ff(root)p Fk(.)30 b(The)20 b(input)h(bu\013er)h(of)e(eac)n(h)g(pro)r(cess)g(ma)n(y)e(ha)n(v)n(e)100 1047 y(a)j(di\013eren)n(t)i(length.)100 1163 y Fg(IN)g(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)h(descriptor)100 1284 y Fg(OUT)f(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)g(descriptor)g ({)e(signi\014can)n(t)i(only)e(at)g(ro)r(ot)h(\(c)n(hoice\))100 1406 y Fg(IN)h(group)32 b Fk(group)23 b(handle)100 1527 y Fg(IN)g(ro)r(ot)33 b Fk(rank)22 b(of)f(receiving)g(pro)r(cess)h(\(in)n(teger\))100 1649 y Fg(OUT)g(return)p 475 1649 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1812 y Fg(MPI)p 365 1812 V 26 w(GA)-6 b(THER)n(C\()20 b(in)n(buf,)j(inlen,)e(outbuf,)j(group,)i(ro)r(ot\))221 1934 y Ff(MPI)p 318 1934 20 3 v 26 w(GATHERC)j Fk(b)r(eha)n(v)n(es)c(lik)n(e) f Ff(MPI)p 1027 1934 V 25 w(GATHER)k Fk(restricted)f(to)d(blo)r(c)n(k)g (bu\013ers,)j(and)e(with)g(the)f(addi-)100 2010 y(tional)g(restriction)h (that)g(all)d(input)j(bu\013ers)g(should)f(ha)n(v)n(e)g(the)g(same)d(length.) 35 b(All)22 b(pro)r(cesses)h(should)100 2085 y(pro)n(vided)g(the)f(same)d(v)m (alues)h(for)i Ff(inlen,)37 b(group)p Fk(,)24 b(and)d Ff(root)j Fk(.)100 2201 y Fg(IN)f(in)n(buf)29 b Fk(\014rst)22 b(v)m(ariable)g(of)f (input)i(bu\013er)f(\(c)n(hoice\))100 2322 y Fg(IN)h(inlen)29 b Fk(Num)n(b)r(er)20 b(of)h(\(w)n(ord\))i(v)m(ariables)e(in)g(input)i (bu\013er)f(\(in)n(teger\))100 2444 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(v)m(ariable)g(of)f(output)j(bu\013er)e({)f(signi\014can)n(t)i (only)e(at)g(ro)r(ot)h(\(c)n(hoice\))100 2565 y Fg(IN)h(group)32 b Fk(group)23 b(handle)100 2687 y Fg(IN)g(ro)r(ot)33 b Fk(rank)22 b(of)f(receiving)g(pro)r(cess)h(\(in)n(teger\))221 2803 y Ff(MPI)p 318 2803 V 26 w(GATHERC)q(\()37 b(inbuf,)g(inlen,)f(outbuf)q(,)h(tag,)e (group,)i(root\))56 b Fk(is)100 2919 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(&size,)g(group\))q(;)100 2994 y(MPI_RA)q(N)q(K)q(\()f(&rank,)h(group\))q(;) 100 3069 y(MPI_IS)q(E)q(N)q(DC)q(\()q(h)q(a)q(nd)q(l)q(e)q(,)g(inbuf,)f (inlen,)h(root,)f(tag,)f(group\))q(;)100 3145 y(if)e(\(rank)q(==)q(r)q(o)q(o) q(t\))195 3220 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))195 3295 y({)291 3371 y(MPI_RE)q(C)q(VC)q(\()q(o)q(u)q(t)q(bu)q(f)q(,)h(inlen,)f(i,)e (tag,)h(group,)i(return)q(\\_)q(s)q(t)q(a)q(tu)q(s)q(\))q(;)291 3446 y(outbuf)g(+=)c(inlen;)195 3521 y(})100 3596 y(MPI_WA)q(I)q(T)q(\(h)q(a) q(n)q(d)q(le)q(\))q(;)p eop %%Page: 8 9 bop 100 -134 a Fk(8)975 b Fh(CHAPTER)17 b(1.)41 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fg(Scatter)100 222 y(MPI)p 244 222 21 3 v 25 w(SCA)f(TTER\()21 b(list)p 745 222 V 24 w(of)p 824 222 V 26 w(in)n(bufs,)i(outbuf,)h(group,)i(ro)r(ot,)g(return)p 1899 222 V 26 w(status\))221 345 y Fk(The)21 b(ro)r(ot)g(pro)r(cess)g(sends)g (the)g(con)n(ten)n(t)j(of)d(its)f Ff(i)p Fk(-th)i(input)g(bu\013er)h(to)e (the)g(pro)r(cess)g(with)g(rank)g Ff(i)p Fk(;)100 420 y(eac)n(h)h(pro)r(cess) f(\(including)j(the)d(ro)r(ot)i(pro)r(cess\))f(stores)g(the)f(incoming)g (message)e(in)i(its)g(output)j(bu\013er.)100 495 y(The)19 b(routine)h(is)e (called)h(b)n(y)g(all)f(mem)n(b)r(ers)e(of)j(the)h(group)h(using)e(the)g (same)e(argumen)n(ts)j(for)g Ff(group)p Fk(,)i(and)100 571 y Ff(root)p Fk(.)100 696 y Fg(IN)h(list)p 302 696 V 25 w(of)p 382 696 V 26 w(in)n(bufs)30 b Fk(list)20 b(of)h(bu\013er)h(descriptor)i (handles)100 822 y Fg(OUT)e(outbuf)30 b Fk(bu\013er)23 b(descriptor)g(handle) 100 948 y Fg(IN)g(group)32 b Fk(handle)100 1073 y Fg(IN)23 b(ro)r(ot)33 b Fk(rank)22 b(of)f(sending)h(pro)r(cess)f(\(in)n(teger\))100 1199 y Fg(OUT)h(return)p 475 1199 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1325 y Ff(MPI)p 318 1325 20 3 v 26 w(SCATTER)q(\()37 b(list)p 754 1325 V 26 w(of)p 842 1325 V 25 w(inbufs)q(,)f(outbuf)q(,)h (group,)f(root,)g(return)p 1976 1325 V 28 w(status)q(\))25 b Fk(is)100 1451 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f(\);)100 1526 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 1601 y(MPI_IR)q(E)q(C)q(V\()q(h)q(a)q(n)q(dl)q(e)q(,)h(outbuf)q(,)f(root,)g(tag,)f (group\))q(;)100 1676 y(if)e(\(rank)q(=r)q(o)q(o)q(t)q(\))195 1752 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))291 1827 y(MPI_SE)q(N)q(D\()q(i)q (n)q(b)q(u)q(f[)q(i)q(])q(,)g(i,)e(tag,)h(group\))q(;)100 1902 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n)q(d)q(le)q(,)i(return)q(\\)q(_s)q(t)q(a)q(t)q (u)q(s\))q(;)221 2075 y Fg(MPI)p 365 2075 21 3 v 26 w(SCA)-6 b(TTER)n(C\()19 b(in)n(buf,)k(outbuf,)h(len,)e(group,)k(ro)r(ot\))221 2198 y Ff(MPI)p 318 2198 20 3 v 26 w(SCATTER)q(C)i Fk(b)r(eha)n(v)n(es)c(lik) n(e)e Ff(MPI)p 1055 2198 V 25 w(SCATTE)q(R)27 b Fk(restricted)f(to)d(blo)r(c) n(k)g(bu\013ers,)i(and)f(with)f(the)h(ad-)100 2273 y(ditional)j(restriction)h (that)f(all)e(output)k(bu\013ers)e(ha)n(v)n(e)f(the)g(same)e(length.)43 b(The)26 b(input)h(bu\013er)g(blo)r(c)n(k)100 2348 y(of)d(the)h(ro)r(ot)f (pro)r(cess)h(is)e(partitioned)k(in)n(to)e Ff(n)f Fk(consecutiv)n(e)h(blo)r (c)n(ks,)g(eac)n(h)f(consisting)i(of)e Ff(len)h Fk(w)n(ords.)100 2424 y(The)19 b Ff(i)p Fk(-th)i(blo)r(c)n(k)f(is)f(sen)n(t)h(to)g(the)g Ff(i)p Fk(-th)h(pro)r(cess)f(in)f(the)h(group)i(and)e(stored)h(in)e(its)h (output)i(bu\013er.)29 b(The)100 2499 y(routine)e(is)c(called)h(b)n(y)h(all)f (mem)n(b)r(ers)e(of)j(the)g(group)h(using)f(the)g(same)e(argumen)n(ts)j(for)f Ff(group,)37 b(len)p Fk(,)100 2574 y(and)22 b Ff(root)p Fk(.)100 2700 y Fg(IN)h(in)n(buf)29 b Fk(\014rst)22 b(en)n(try)h(in)e(input)h (bu\013er)h({)d(signi\014can)n(t)k(only)d(at)g(ro)r(ot)g(\(c)n(hoice\).)100 2826 y Fg(OUT)h(outbuf)30 b Fk(\014rst)22 b(en)n(try)h(in)e(output)i (bu\013er)g(\(c)n(hoice\).)100 2951 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(to)f(b)r(e)g(stored)h(in)f(output)j(bu\013er)e(\(in)n (teger\))100 3077 y Fg(IN)h(group)32 b Fk(handle)100 3203 y Fg(IN)23 b(ro)r(ot)33 b Fk(rank)22 b(of)f(sending)h(pro)r(cess)f(\(in)n (teger\))221 3328 y Ff(MPI)p 318 3328 V 26 w(SCATTER)q(C)q(\()37 b(inbuf,)f(outbu)q(f,)h(outlen)q(,)g(group,)f(root\))56 b Fk(is)100 3454 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(&size,)g(group\))q(;)100 3530 y(MPI_RA)q(N)q(K)q(\()f(&rank,)h(group\))q(;)100 3605 y(MPI_IR)q(E)q(C)q(VC)q(\()g(handle)q(,)f(outbuf)q(,)h(outlen)q(,)f(root,)g (tag,)f(group)q(\);)p eop %%Page: 9 10 bop 100 -134 a Fh(1.5.)41 b(D)n(A)-5 b(T)g(A)18 b(MO)n(VE)h(FUNCTIONS)1470 b Fk(9)100 60 y Ff(if)33 b(\(rank)q(=r)q(o)q(o)q(t)q(\))195 135 y(for)i(\(i=0;)h(i)c(<)h(size;)j(i++\))195 211 y({)291 286 y(MPI_SE)q(N)q(DC)q(\()q(i)q(n)q(b)q(uf)q(,)h(outlen)q(,)f(i,)e(tag,)h (group,)i(return)q(\\_)q(s)q(t)q(a)q(tu)q(s)q(\))q(;)291 361 y(inbuf)f(+=)d(outlen)q(;)195 437 y(})100 512 y(MPI_WA)q(I)q(T)q(\(h)q(a)q(n) q(d)q(le)q(\))q(;)100 671 y Fg(All-to-all)23 b(scatter)100 833 y(MPI)p 244 833 21 3 v 25 w(ALLSCA)-6 b(TTER\()20 b(list)p 881 833 V 25 w(of)p 961 833 V 26 w(in)n(bufs,)j(outbuf,)h(group,)h(return)p 1865 833 V 26 w(status\))221 955 y Fk(Eac)n(h)g(pro)r(cess)g(in)f(the)h (group)h(sends)f(its)g Ff(i)p Fk(-th)h(bu\013er)g(in)e(its)h(input)h (bu\013er)g(list)e(to)h(the)g(pro)r(cess)100 1031 y(with)18 b(rank)g Ff(i)g Fk(\(itself)g(included\);)j(eac)n(h)d(pro)r(cess)g (concatenates)h(the)g(incoming)d(messages)g(in)h(its)h(output)100 1106 y(bu\013er,)j(in)e(the)g(order)i(of)e(the)g(senders')h(ranks.)28 b(The)19 b(routine)i(is)d(called)h(b)n(y)g(all)f(mem)n(b)r(ers)f(of)i(the)g (group)100 1181 y(using)j(the)f(same)e(argumen)n(ts)j(for)g Ff(group)p Fk(.)100 1303 y Fg(IN)h(list)p 302 1303 V 25 w(of)p 382 1303 V 26 w(in)n(bufs)30 b Fk(list)20 b(of)h(bu\013er)h(descriptor)i (handles)100 1427 y Fg(OUT)e(outbuf)30 b Fk(bu\013er)23 b(descriptor)g (handle)100 1550 y Fg(IN)g(group)32 b Fk(handle)100 1674 y Fg(OUT)22 b(return)p 475 1674 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1843 y Fg(MPI)p 365 1843 V 26 w(ALLSCA)-6 b(TTER)n(C)o(\()20 b(in)n(buf,)i(outbuf,)i(len,)f(group\))221 1965 y Ff(MPI)p 318 1965 20 3 v 26 w(ALLSCAT)q(T)q(E)q(R)q(C)29 b Fk(b)r(eha)n(v)n(es)e(lik)n(e)e Ff(MPI)p 1158 1965 V 25 w(ALLSCA)q(T)q(TE)q (R)30 b Fk(restricted)e(to)d(blo)r(c)n(k)h(bu\013ers,)i(and)e(with)100 2041 y(the)f(additional)i(restriction)h(that)e(all)e(blo)r(c)n(ks)h(sen)n(t)g (from)g(one)g(pro)r(cess)g(to)g(another)i(ha)n(v)n(e)f(the)f(same)100 2116 y(length.)j(The)18 b(input)i(bu\013er)h(blo)r(c)n(k)d(of)h(eac)n(h)g (pro)r(cess)g(is)e(partitioned)22 b(in)n(to)e Ff(n)e Fk(consecutiv)n(e)i(blo) r(c)n(ks,)f(eac)n(h)100 2191 y(consisting)26 b(of)g Ff(len)g Fk(w)n(ords.)42 b(The)24 b Ff(i)p Fk(-th)j(blo)r(c)n(k)e(is)f(sen)n(t)i(to)f (the)h Ff(it)p Fk(-th)h(pro)r(cess)f(in)f(the)g(group.)43 b(Eac)n(h)100 2267 y(pro)r(cess)24 b(concatenates)h(the)f(incoming)e(messages,)g(in)h(the)g (order)i(of)e(the)h(senders')g(ranks,)g(and)h(store)100 2342 y(them)18 b(in)g(its)g(output)k(bu\013er.)28 b(The)18 b(routine)j(is)d (called)g(b)n(y)h(all)f(mem)n(b)r(ers)e(of)j(the)g(group)i(using)e(the)g (same)100 2417 y(argumen)n(ts)j(for)g Ff(group)p Fk(,)i(and)e Ff(len)p Fk(.)100 2539 y Fg(IN)h(in)n(buf)29 b Fk(\014rst)22 b(en)n(try)h(in)e(input)h(bu\013er)h(\(c)n(hoice\).)29 b(ro)r(ot)21 b(\(in)n(teger\))100 2662 y Fg(OUT)h(outbuf)30 b Fk(\014rst)22 b(en)n(try)h(in)e(output)i(bu\013er)g(\(c)n(hoice\).)100 2786 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(sen)n(t)g(from)e (eac)n(h)i(pro)r(cess)f(to)g(eac)n(h)g(other)i(\(in)n(teger\).)100 2910 y Fg(IN)g(group)32 b Fk(handle)221 3031 y Ff(MPI)p 318 3031 V 26 w(ALLSCAT)q(T)q(E)q(R)q(C)q(\()k(inbuf,)h(outbuf)q(,)f(len,)g (group\))25 b Fk(is)100 3153 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f (\);)100 3228 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);)100 3304 y(for)e(\(i=0;)i(i)d(<)f(rank;)k(i++\))195 3379 y({)227 3454 y(MPI_IR)q(E)q(C)q(V)q(C\()q(r)q(e)q(c)q(v)q(_h)q(a)q(n)q(d)q(le)q(s)q ([)q(i)q(])q(,)g(outbuf)q(,)h(len,)e(tag,)g(group\))q(;)227 3530 y(outbuf)i(+=)c(len;)195 3605 y(})p eop %%Page: 10 11 bop 100 -134 a Fk(10)945 b Fh(CHAPTER)17 b(1.)41 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Ff(for)34 b(\(i=0;)i(i)d(<)f(size;)k(i++\))195 135 y({)227 211 y(MPI_IS)q(E)q(N)q(D)q(C\()q(s)q(e)q(n)q(d)q(_h)q(a)q(n)q(d)q (le)q([)q(i)q(])q(,)g(inbuf,)h(len,)e(i,)f(tag,)h(group\))q(;)227 286 y(inbuf)h(+=)e(len;)195 361 y(})100 437 y(MPI_WA)q(I)q(T)q(AL)q(L)q(\()q (s)q(en)q(d)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 512 y(MPI_WA)q(I)q(T)q(AL)q (L)q(\()q(r)q(ec)q(v)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 671 y Fg(All-to-all)23 b(broadcast)100 833 y(MPI)p 244 833 21 3 v 25 w(ALLCAST\()e(in)n(buf,)i(outbuf,)h(group,)h(return)p 1497 833 V 26 w(status\))221 955 y Fk(Eac)n(h)19 b(pro)r(cess)g(in)f(the)h (group)i(broadcasts)g(its)e(input)h(bu\013er)h(to)d(all)g(pro)r(cesses)h (\(including)i(itself)5 b(\);)100 1031 y(eac)n(h)24 b(pro)r(cess)h (concatenates)h(the)e(incoming)g(messages)e(in)h(its)h(output)j(bu\013er,)f (in)e(the)g(order)i(of)e(the)100 1106 y(senders')19 b(ranks.)27 b(The)18 b(routine)i(is)d(called)g(b)n(y)h(all)g(mem)n(b)r(ers)d(of)j(the)g (group)i(using)f(the)f(same)e(argumen)n(ts)100 1181 y(for)22 b Ff(group)p Fk(.)100 1303 y Fg(IN)h(in)n(buf)29 b Fk(bu\013er)23 b(descriptor)g(handle)g(for)f(input)g(bu\013er)100 1427 y Fg(OUT)g(outbuf)30 b Fk(bu\013er)23 b(descriptor)g(handle)g(for)f(output)h(bu\013er)100 1550 y Fg(IN)g(group)32 b Fk(handle)100 1674 y Fg(OUT)22 b(return)p 475 1674 V 26 w(status)32 b Fk(return)23 b(status)f(handle)221 1843 y Fg(MPI)p 365 1843 V 26 w(ALLCASTC\()e(in)n(buf,)i(outbuf,)i(len,)f (group\))221 1965 y Ff(MPI)p 318 1965 20 3 v 26 w(ALLCAST)q(C)28 b Fk(b)r(eha)n(v)n(es)c(lik)n(e)e Ff(MPI)p 1055 1965 V 25 w(ALLCAS)q(T)27 b Fk(restricted)f(to)d(blo)r(c)n(k)g(bu\013ers,)i(and)f(with)f(the)h(ad-)100 2041 y(ditional)g(restriction)h(that)e(all)f(blo)r(c)n(ks)h(sen)n(t)g(from)f (one)g(pro)r(cess)h(to)g(another)i(ha)n(v)n(e)e(the)g(same)d(length.)100 2116 y(The)f(routine)h(is)e(called)h(b)n(y)g(all)f(mem)n(b)r(ers)e(of)j(the)h (group)h(using)e(the)g(same)e(argumen)n(ts)j(for)g Ff(group)p Fk(,)i(and)100 2191 y Ff(len)p Fk(.)100 2313 y Fg(IN)h(in)n(buf)29 b Fk(\014rst)22 b(en)n(try)h(in)e(input)h(bu\013er)h(\(c)n(hoice\).)29 b(ro)r(ot)21 b(\(in)n(teger\))100 2437 y Fg(OUT)h(outbuf)30 b Fk(\014rst)22 b(en)n(try)h(in)e(output)i(bu\013er)g(\(c)n(hoice\).)100 2560 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(sen)n(t)g (from)e(eac)n(h)i(pro)r(cess)f(to)g(eac)n(h)g(other)i(\(including)g(itself)5 b(\).)100 2684 y Fg(IN)23 b(group)32 b Fk(group)23 b(handle)221 2806 y Ff(MPI)p 318 2806 V 26 w(ALLCAST)q(C)q(\()37 b(inbuf,)f(outbu)q(f,)h (len,)e(group\))25 b Fk(is)100 2927 y Ff(MPI_GS)q(I)q(Z)q(E\()37 b(group,)g(&size)f(\);)100 3003 y(MPI_RA)q(N)q(K)q(\()g(group,)h(&rank)f(\);) 100 3078 y(for)e(\(i=0;)i(i)d(<)f(rank;)k(i++\))195 3153 y({)227 3228 y(MPI_IR)q(E)q(C)q(V)q(C\()q(r)q(e)q(c)q(v)q(_h)q(a)q(n)q(d)q(le)q(s)q ([)q(i)q(])q(,)g(outbuf)q(,)h(len,)e(tag,)g(group\))q(;)227 3304 y(outbuf)i(+=)c(len;)195 3379 y(})100 3454 y(for)h(\(i=0;)i(i)d(<)f (size;)k(i++\))195 3530 y({)227 3605 y(MPI_IS)q(E)q(N)q(D)q(C\()q(s)q(e)q(n)q (d)q(_h)q(a)q(n)q(d)q(le)q([)q(i)q(])q(,)g(inbuf,)h(len,)e(i,)f(tag,)h (group\))q(;)p eop %%Page: 11 12 bop 100 -134 a Fh(1.6.)47 b(GLOBAL)19 b(COMPUTE)f(OPERA)-5 b(TIONS)1171 b Fk(11)195 60 y Ff(})100 135 y(MPI_WA)q(I)q(T)q(AL)q(L)q(\()q (s)q(en)q(d)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 211 y(MPI_WA)q(I)q(T)q(AL)q (L)q(\()q(r)q(ec)q(v)q(_)q(h)q(a)q(nd)q(l)q(e)q(\))q(;)100 407 y Fj(1.6)94 b(Global)29 b(Comput)o(e)d(Op)s(erations)100 544 y Fg(Reduce)100 708 y(MPI)p 244 708 21 3 v 25 w(REDUCE\()21 b(in)n(buf,)h(outbuf,)i(group,)i(ro)r(ot,)g(op\))221 831 y Fk(Com)n(bines)d(the)h(v)m(alues)f(pro)n(vided)j(in)d(the)h(input)i(bu\013er) f(of)e(eac)n(h)h(pro)r(cess)g(in)g(the)g(group,)i(using)100 906 y(the)c(op)r(eration)j Ff(op)p Fk(,)d(and)h(returns)i(the)d(com)n(bined)g (v)m(alue)f(in)h(the)g(output)j(bu\013er)f(of)e(the)g(pro)r(cess)g(with)100 982 y(rank)h Ff(root)p Fk(.)33 b(Eac)n(h)21 b(pro)r(cess)h(can)g(pro)n(vide)i (one)e(v)m(alue,)g(or)g(a)f(sequence)h(of)g(v)m(alues,)g(in)f(whic)n(h)i (case)e(the)100 1057 y(com)n(bine)e(op)r(eration)j(is)c(executed)j(p)r(oin)n (t)n(wise)f(on)g(eac)n(h)g(en)n(try)h(of)f(the)g(sequence.)27 b(F)-5 b(or)20 b(example,)e(if)h(the)100 1132 y(op)r(eration)25 b(is)d Ff(max)j Fk(and)f(input)h(bu\013ers)g(con)n(tains)f(t)n(w)n(o)g (\015oating)h(p)r(oin)n(t)f(n)n(um)n(b)r(ers,)h(then)f(outbuf\(1\))j(=)100 1208 y(global)19 b(max\(in)n(buf\(1\)\))j(and)d(outbuf\(2\))j(=)c(global)h (max\(in)n(buf\(2\)\).)30 b(All)17 b(input)j(bu\013ers)f(should)h(de\014ne) 100 1283 y(sequences)e(of)g(equal)g(length)h(of)f(en)n(tries)h(of)f(t)n(yp)r (es)g(that)i(matc)n(h)d(the)i(t)n(yp)r(e)f(of)g(the)g(op)r(erands)i(of)e Ff(op)p Fk(.)28 b(The)100 1358 y(output)g(bu\013er)e(should)h(de\014ne)f(a)e (sequence)i(of)f(the)g(same)e(length)k(of)e(en)n(tries)h(of)f(t)n(yp)r(es)g (that)i(matc)n(h)100 1433 y(the)e(t)n(yp)r(e)g(of)g(the)h(result)f(of)g Ff(op)p Fk(.)40 b(\(Note)25 b(that,)i(here)f(as)e(for)h(all)g(other)h(comm)n (unication)e(op)r(erations,)100 1509 y(the)e(t)n(yp)r(e)g(of)f(en)n(tries)h (inserted)h(in)e(a)g(message)f(dep)r(end)i(on)g(the)g(information)h(pro)n (vided)g(b)n(y)f(the)g(input)100 1584 y(bu\013er)e(descriptor,)h(and)e(not)f (on)h(the)f(declarations)i(of)e(these)h(v)m(ariables)f(in)g(the)g(calling)g (program.)28 b(The)100 1659 y(t)n(yp)r(es)19 b(of)f(the)h(v)m(ariables)f(in)g (the)h(calling)f(program)h(need)g(not)g(matc)n(h)f(the)h(t)n(yp)r(es)f (de\014ned)i(b)n(y)f(the)g(bu\013er)100 1734 y(descriptor,)32 b(but)e(in)e(suc)n(h)h(case)f(the)g(outcome)g(of)g(a)g(reduce)i(op)r(eration) g(ma)n(y)d(b)r(e)h(implemen)n(tation)100 1810 y(dep)r(enden)n(t.\))221 1886 y(The)22 b(op)r(eration)i(de\014ned)g(b)n(y)e Ff(op)h Fk(is)e(asso)r(ciativ)n(e)h(and)h(comm)n(utativ)n(e,)e(and)i(the)g(implemen)n (tation)100 1961 y(can)h(tak)n(e)h(adv)m(an)n(tage)h(of)e(asso)r(ciativit)n (y)h(and)g(comm)n(utativit)n(y)f(in)g(order)i(to)e(c)n(hange)h(order)h(of)e (ev)m(alu-)100 2036 y(ation.)41 b(The)24 b(routine)j(is)d(called)h(b)n(y)g (all)g(group)i(mem)n(b)r(ers)22 b(using)k(the)f(same)e(argumen)n(ts)j(for)g Ff(group,)100 2112 y(root)d Fk(and)f Ff(op)p Fk(.)100 2241 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 2370 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)g({)d (signi\014can)n(t)k(only)d(at)g(ro)r(ot)100 2498 y Fg(IN)i(group)32 b Fk(handle)23 b(to)e(group)100 2627 y Fg(IN)i(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n(teger\))100 2756 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 2885 y(W)-5 b(e)27 b(list)f(b)r(elo)n(w)h(the)h(op)r(erations)h(whic)n(h)f (are)f(supp)r(orted)j(for)e(F)-5 b(ortran,)31 b(eac)n(h)c(with)h(the)f (corre-)100 2960 y(sp)r(onding)c(v)m(alue)d(of)i(the)f Ff(op)h Fk(parameter.)100 3089 y Fg(MPI)p 244 3089 V 25 w(IMAX)31 b Fk(in)n(teger)22 b(maxim)n(um)100 3218 y Fg(MPI)p 244 3218 V 25 w(RMAX)29 b Fk(real)21 b(maxim)n(um)100 3347 y Fg(MPI)p 244 3347 V 25 w(DMAX)30 b Fk(double)22 b(precision)g(real)f(maxim)n(um)100 3476 y Fg(MPI)p 244 3476 V 25 w(IMIN)31 b Fk(in)n(teger)23 b(minim)n(um)100 3605 y Fg(MPI)p 244 3605 V 25 w(RMIN)30 b Fk(real)21 b(minim)n(um)p eop %%Page: 12 13 bop 100 -134 a Fk(12)938 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fg(MPI)p 244 60 21 3 v 25 w(DMIN)30 b Fk(double)23 b(precision)f(real)f(minim)n(um)100 185 y Fg(MPI)p 244 185 V 25 w(ISUM)31 b Fk(in)n(teger)22 b(sum)100 310 y Fg(MPI)p 244 310 V 25 w(RSUM)29 b Fk(real)21 b(sum)100 435 y Fg(MPI)p 244 435 V 25 w(DSUM)30 b Fk(double)22 b(precision)g(real)f(sum)100 560 y Fg(MPI)p 244 560 V 25 w(CSUM)29 b Fk(complex)20 b(sum)100 685 y Fg(MPI)p 244 685 V 25 w(DCSUM)29 b Fk(double)22 b(precision)g(complex)e (sum)100 810 y Fg(MPI)p 244 810 V 25 w(IPR)n(OD)29 b Fk(in)n(teger)22 b(pro)r(duct)100 935 y Fg(MPI)p 244 935 V 25 w(RPR)n(OD)28 b Fk(real)21 b(pro)r(duct)100 1060 y Fg(MPI)p 244 1060 V 25 w(DPR)n(OD)28 b Fk(double)22 b(precision)g(real)f(pro)r(duct)100 1185 y Fg(MPI)p 244 1185 V 25 w(CPR)n(OD)28 b Fk(complex)19 b(pro)r(duct)100 1310 y Fg(MPI)p 244 1310 V 25 w(DCPR)n(OD)27 b Fk(double)22 b(precision)h(complex)c(pro)r(duct)100 1435 y Fg(MPI)p 244 1435 V 25 w(AND)29 b Fk(logical)21 b(and)100 1560 y Fg(MPI)p 244 1560 V 25 w(IAND)30 b Fk(in)n(teger)22 b(\(bit-wise\))g(and)100 1685 y Fg(MPI)p 244 1685 V 25 w(OR)29 b Fk(logical)21 b(or)100 1810 y Fg(MPI)p 244 1810 V 25 w(IOR)30 b Fk(in)n(teger)22 b(\(bit-wise\))g(or)100 1935 y Fg(MPI)p 244 1935 V 25 w(X)n(OR)29 b Fk(logical)21 b(xor)100 2060 y Fg(MPI)p 244 2060 V 25 w(IX)n(OR)29 b Fk(in)n(teger)23 b(\(bit-wise\))f(xor) 100 2185 y Fg(MPI)p 244 2185 V 25 w(MAXLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(maxim)n(um)c(in)n(teger)22 b(v)m(alue)100 2310 y Fg(MPI)p 244 2310 V 25 w(MAXRLOC)28 b Fk(rank)22 b(of)f(pro)r(cess)g (with)g(maxim)n(um)c(real)k(v)m(alue)100 2435 y Fg(MPI)p 244 2435 V 25 w(MAXDLOC)28 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(maxim)n(um)c (double)23 b(precision)f(real)f(v)m(alue)100 2560 y Fg(MPI)p 244 2560 V 25 w(MINLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)h(with)f(minim)n(um) c(in)n(teger)22 b(v)m(alue)100 2685 y Fg(MPI)p 244 2685 V 25 w(MINRLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(minim)n(um)c(real)k(v)m (alue)100 2810 y Fg(MPI)p 244 2810 V 25 w(MINDLOC)29 b Fk(rank)22 b(of)f(pro)r(cess)g(with)g(minim)n(um)c(double)22 b(precision)g(real)f(v)m (alue)221 2982 y Fg(MPI)p 365 2982 V 26 w(REDUCEC\()e(in)n(buf,)k(outbuf,)h (len,)f(group,)i(ro)r(ot,)h(op\))221 3105 y Fk(Is)20 b(same)f(as)h Ff(MPI)p 610 3105 20 3 v 26 w(REDUCE)q Fk(,)k(restricted)f(to)e(a)f(blo)r(c)n (k)h(bu\013er.)100 3230 y Fg(IN)i(in)n(buf)29 b Fk(\014rst)22 b(lo)r(cation)g(in)f(input)h(bu\013er)100 3355 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(lo)r(cation)g(in)f(output)i(bu\013er)g({)e(signi\014can)n (t)i(only)e(at)g(ro)r(ot)100 3480 y Fg(IN)i(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 3605 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)p eop %%Page: 13 14 bop 100 -134 a Fh(1.6.)47 b(GLOBAL)19 b(COMPUTE)f(OPERA)-5 b(TIONS)1171 b Fk(13)100 60 y Fg(IN)23 b(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n(teger\))100 183 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 350 y Fg(MPI)p 365 350 21 3 v 26 w(USER)p 583 350 V 23 w(REDUCE\()e(in)n(buf,)h(outbuf,)i(group,)i (ro)r(ot,)g(function\))221 472 y Fk(Same)e(as)g(the)h(reduce)g(op)r(eration)i (function)g(ab)r(o)n(v)n(e)f(except)f(that)h(a)e(user)h(supplied)h(function)h (is)100 548 y(used.)g Ff(functi)q(o)q(n)21 b Fk(is)c(an)g(asso)r(ciativ)n(e)h (and)g(comm)n(utativ)n(e)f(function)j(with)d(t)n(w)n(o)h(argumen)n(ts.)28 b(The)17 b(t)n(yp)r(es)100 623 y(of)j(the)h(t)n(w)n(o)f(argumen)n(ts)i(and)f (of)f(the)g(returned)k(v)m(alue)19 b(of)i(the)f(function,)j(and)e(the)f(t)n (yp)r(es)h(of)f(all)f(en)n(tries)100 698 y(in)25 b(the)h(input)h(and)f (output)i(bu\013ers)f(all)d(agree.)41 b(The)25 b(output)j(bu\013er)f(has)f (the)f(same)f(length)i(as)f(the)100 773 y(input)e(bu\013er.)100 893 y Fg(IN)g(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 1016 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)g({)d (signi\014can)n(t)k(only)d(at)g(ro)r(ot)100 1139 y Fg(IN)i(group)32 b Fk(handle)23 b(to)e(group)100 1262 y Fg(IN)i(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n(teger\))100 1385 y Fg(IN)i(function)30 b Fk(user)22 b(pro)n(vided)h(function)221 1552 y Fg(MPI)p 365 1552 V 26 w(USER)p 583 1552 V 23 w(REDUCEC\()d(in)n(buf,) i(outbuf,)i(len,)f(group,)i(ro)r(ot,)i(function\))221 1674 y Fk(Is)20 b(same)f(as)h Ff(MPI)p 610 1674 20 3 v 636 1674 V 49 w(USER)p 783 1674 V 26 w(REDUCE)q Fk(,)k(restricted)f(to)e(a)f(blo)r(c)n (k)h(bu\013er.)100 1794 y Fg(IN)i(in)n(buf)29 b Fk(\014rst)22 b(lo)r(cation)g(in)f(input)h(bu\013er)100 1917 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(lo)r(cation)g(in)f(output)i(bu\013er)g({)e(signi\014can)n (t)i(only)e(at)g(ro)r(ot)100 2039 y Fg(IN)i(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 2162 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 2285 y Fg(IN)i(ro)r(ot)33 b Fk(rank)22 b(of)f(ro)r(ot)h(pro)r(cess)f(\(in)n (teger\))100 2408 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 2629 y Fe(Discussion:)221 2695 y Fd(Do)e(w)n(e)h(also)d(w)n(an)n(t)i(a)g(v)n (ersion)f(of)g(reduce)i(that)f(broadcasts)f(the)h(result)f(to)h(all)e(pro)r (cesses)i(in)f(the)h(group?)100 2762 y(\(This)12 b(can)h(b)r(e)f(ac)n(hiev)n (ed)i(b)n(y)f(a)g(reduce)g(follo)n(w)n(ed)f(b)n(y)h(a)f(broadcast,)h(but)f(a) h(com)n(bined)e(function)h(ma)n(y)g(b)r(e)h(somewhat)100 2828 y(more)18 b(e\016cien)n(t.\))25 b(These)19 b(w)n(ould)f(b)r(e)g(resp)r(ectiv) n(ely:)221 2942 y Fe(MPI)p 352 2942 V 24 w(GOP\()23 b(in)n(buf,)c(outbuf,)g (group,)f(op\))221 3103 y(MPI)p 352 3103 V 24 w(GOPC\()23 b(in)n(buf,)c (outbuf,)g(len,)i(group,)d(op\))221 3264 y(MPI)p 352 3264 V 24 w(USER)p 551 3264 V 22 w(GOP\()k(in)n(buf,)e(outbuf,)f(group,)f (function\))221 3425 y(MPI)p 352 3425 V 24 w(USER)p 551 3425 V 22 w(GOPC\()k(in)n(buf,)e(outbuf,)f(len,)i(group,)d(function\))221 3538 y Fd(Do)g(w)n(e)h(w)n(an)n(t)g(a)f(user)g(pro)n(vided)f Fb(function)g Fd(\(t)n(w)n(o)j(IN)e(parameters,)f(one)h(OUT)g(v)m(alue\),)g (or)f(a)h(user)g(pro)n(vided)100 3605 y(pro)r(cedure)i(that)h(o)n(v)n (erwrites)h(the)f(second)g(input)e(\(ie.)31 b(one)21 b(IN)g(param,)d(one)i (INOUT)i(param,)c(the)k(equiv)m(alen)n(t)p eop %%Page: 14 15 bop 100 -134 a Fk(14)938 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)100 60 y Fd(of)24 b(C)h Fa(a)k(op=)g(b)c Fd(t)n(yp)r(e)g(assignmen)n(t\)?)42 b(The)25 b(second)f(c)n(hoice)i(ma)n(y)e (allo)n(w)f(a)i(more)e(e\016cien)n(t)j(implemen)n(tation,)100 127 y(without)18 b(c)n(hanging)f(the)i(seman)n(tics)g(of)f(the)h(MPI)f(call.) 100 396 y Fg(Scan)100 557 y(MPI)p 244 557 21 3 v 25 w(SCAN\()k(in)n(buf,)g (outbuf,)i(group,)i(op)e(\))221 680 y Fk(MPI)p 345 680 19 3 v 21 w(SCAN)16 b(is)g(used)i(to)g(p)r(erform)g(a)f(parallel)h(pre\014x)h (with)f(resp)r(ect)g(to)f(an)h(asso)r(ciativ)n(e)g(reduction)100 755 y(op)r(eration)k(on)e(data)h(distributed)i(across)d(the)g(group.)29 b(The)19 b(op)r(eration)j(returns)h(in)c(the)h(output)j(bu\013er)100 830 y(of)d(the)f(pro)r(cess)h(with)g(rank)g Ff(i)g Fk(the)f(reduction)k(of)c (the)h(v)m(alues)f(in)g(the)h(input)h(bu\013ers)g(of)f(pro)r(cesses)f(with) 100 906 y(ranks)k Ff(0,...,)q(i)q Fk(.)34 b(The)22 b(t)n(yp)r(e)h(of)f(op)r (erations)i(supp)r(orted)i(and)d(their)g(seman)n(tic,)f(and)h(the)f (constrain)n(ts)100 981 y(on)f(input)i(and)f(output)h(bu\013ers)g(are)e(as)f (for)i Ff(MPI)p 1225 981 20 3 v 25 w(REDUC)q(E)p Fk(.)100 1099 y Fg(IN)h(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 1222 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)100 1344 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 1467 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 1633 y Fg(MPI)p 365 1633 21 3 v 26 w(SCANC\()h(in)n(buf,)j(outbuf,)h(len,)f (group,)j(op)d(\))c Fk(Same)g(as)g Ff(MPI)p 1984 1633 20 3 v 26 w(SCAN)p Fk(,)j(restricted)g(to)100 1755 y(blo)r(c)n(k)21 b(bu\013ers.)100 1888 y Fg(IN)i(in)n(buf)29 b Fk(\014rst)22 b(input)h(bu\013er)g(elemen)n(t)d(\(c)n(hoice\))100 2010 y Fg(OUT)i(outbuf)30 b Fk(\014rst)22 b(output)i(bu\013er)f(elemen)n(t)d(\(c)n (hoice\))100 2133 y Fg(IN)j(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 2255 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 2378 y Fg(IN)i(op)31 b Fk(op)r(eration)23 b(\(status\))221 2557 y Fg(MPI)p 365 2557 21 3 v 26 w(USER)p 583 2557 V 23 w(SCAN\()f(in)n(buf,)g (outbuf,)i(group,)i(function)d(\))221 2680 y Fk(Same)16 b(as)h(the)h(scan)f (op)r(eration)j(function)g(ab)r(o)n(v)n(e)e(except)g(that)g(a)f(user)h (supplied)h(function)h(is)c(used.)100 2755 y Ff(functi)q(o)q(n)29 b Fk(is)24 b(an)h(asso)r(ciativ)n(e)g(and)h(comm)n(utativ)n(e)e(function)j (with)e(t)n(w)n(o)h(argumen)n(ts.)40 b(The)25 b(t)n(yp)r(es)g(of)100 2830 y(the)c(t)n(w)n(o)h(argumen)n(ts)g(and)g(of)f(the)g(returned)j(v)m (alues)d(all)f(agree.)100 2949 y Fg(IN)j(in)n(buf)29 b Fk(handle)23 b(to)e(input)h(bu\013er)100 3072 y Fg(OUT)g(outbuf)30 b Fk(handle)23 b(to)e(output)i(bu\013er)100 3194 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 3316 y Fg(IN)i(function)30 b Fk(user)22 b(pro)n(vided)h (function)221 3482 y Fg(MPI)p 365 3482 V 26 w(USER)p 583 3482 V 23 w(SCANC\()e(in)n(buf,)h(outbuf,)i(len,)f(group,)j(function\))221 3605 y Fk(Is)20 b(same)f(as)h Ff(MPI)p 610 3605 20 3 v 26 w(USER)p 760 3605 V 26 w(SCAN)p Fk(,)j(restricted)g(to)e(a)g(blo)r(c)n(k)g(bu\013er.)p eop %%Page: 15 16 bop 100 -134 a Fh(1.7.)47 b(CORRECTNESS)1733 b Fk(15)100 60 y Fg(IN)23 b(in)n(buf)29 b Fk(\014rst)22 b(lo)r(cation)g(in)f(input)h (bu\013er)100 180 y Fg(OUT)g(outbuf)30 b Fk(\014rst)22 b(lo)r(cation)g(in)f (output)i(bu\013er)100 300 y Fg(IN)g(len)30 b Fk(n)n(um)n(b)r(er)21 b(of)g(en)n(tries)h(in)f(input)i(and)f(output)h(bu\013er)g(\(in)n(teger\))100 420 y Fg(IN)g(group)32 b Fk(handle)23 b(to)e(group)100 540 y Fg(IN)i(function)30 b Fk(user)22 b(pro)n(vided)h(function)221 754 y Fe(Discussion:)221 830 y Fd(Do)e(w)n(e)h(w)n(an)n(t)f(scan)g(op)r (erations)e(executed)k(b)n(y)f(segmen)n(ts?)31 b(\(The)21 b(HPF)g (de\014nition)e(of)h(pre\014x)h(and)f(su\016x)100 905 y(op)r(eration)f(migh)n (t)i(b)r(e)g(handy)f({)h(in)f(addition)g(to)h(the)h(scanned)f(v)n(ector)i(of) e(v)m(alues)g(there)h(is)f(a)f(mask)h(that)g(tells)100 980 y(where)e(segmen)n(ts)g(start)g(and)e(end.\))100 1279 y Fj(1.7)94 b(Correctness)100 1525 y Fe(Discussion:)46 b Fd(This)18 b(is)f(still)g(v)n (ery)j(preliminary)221 1710 y Fk(The)25 b(seman)n(tics)g(of)h(the)g (collectiv)n(e)f(comm)n(unication)h(op)r(erations)h(can)f(b)r(e)f(deriv)n(ed) i(from)e(their)100 1785 y(op)r(erational)i(de\014nition)g(in)d(terms)f(of)h (p)r(oin)n(t-to-p)r(oin)o(t)k(comm)n(unication.)38 b(It)24 b(is)f(assumed)h(that)h(mes-)100 1861 y(sages)c(p)r(ertaining)j(to)d(one)h (op)r(eration)h(cannot)h(b)r(e)c(confused)k(with)d(messages)e(p)r(ertaining) 24 b(to)e(another)100 1936 y(op)r(eration.)29 b(Also)18 b(messages)f(p)r (ertaining)22 b(to)d(t)n(w)n(o)h(distinct)g(o)r(ccurrences)h(of)e(the)g(same) f(op)r(eration)j(can-)100 2011 y(not)h(b)r(e)e(confused,)j(if)e(the)g(t)n(w)n (o)h(o)r(ccurrences)h(ha)n(v)n(e)e(distinct)i(parameters.)28 b(The)21 b(relev)m(an)n(t)h(parameters)100 2086 y(for)f(this)g(purp)r(ose)h (are)e Ff(group)p Fk(,)k Ff(root)f Fk(and)e Ff(op)p Fk(.)28 b(messages)18 b(p)r(ertaining)23 b(to)e(another)h(o)r(ccurrence)g(of)f(the) 100 2162 y(same)h(op)r(eration,)28 b(with)d(di\013eren)n(t)h(parameters.)39 b(The)25 b(implemen)n(ter)e(can,)i(of)g(course,)h(use)e(another,)100 2237 y(more)c(e\016cien)n(t)h(implemen)n(tation,)f(as)g(long)i(as)e(it)h(has) g(the)g(same)e(e\013ect.)221 2414 y Fe(Discussion:)221 2480 y Fd(This)c(statemen)n(t)i(do)r(es)d(not)i(y)n(et)h(apply)d(to)i(the)g (curren)n(t,)g(incomplete)f(and)g(somewhat)f(careless)i(de\014nitions)100 2546 y(I)i(pro)n(vided)g(in)g(this)g(draft.)221 2613 y(The)j(de\014nition)e (ab)r(o)n(v)n(e)i(means)e(that)i(messages)f(p)r(ertaining)e(to)j(a)f (collectiv)n(e)i(comm)n(unication)c(carry)i(in-)100 2679 y(formation)c(iden)n (tifying)h(the)i(op)r(eration)d(itself,)h(and)h(the)h(v)m(alues)f(of)f(the)j Fa(group)e Fd(and,)f(where)i(relev)m(an)n(t,)f Fa(root)h Fd(or)100 2746 y Fa(op)g Fd(parameters.)k(Is)18 b(this)g(acceptable?)221 2931 y Fk(A)i(few)g(examples:)100 3055 y Ff(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f) q(,)36 b(len,)f(group,)i(0\);)100 3131 y(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f)q (,)f(len,)f(group,)i(1\);)221 3255 y Fk(Tw)n(o)18 b(consecutiv)n(e)i (broadcasts,)i(in)d(the)g(same)d(group,)21 b(with)e(the)h(same)c(tag,)k(but)g (di\013eren)n(t)h(ro)r(ots.)100 3330 y(Since)h(the)h(op)r(erations)h(are)e (distinguishable,)k(messages)20 b(from)h(one)h(broadcast)j(cannot)f(b)r(e)d (confused)100 3405 y(with)g(messages)e(from)h(the)i(other)g(broadcast;)h(the) f(program)g(is)e(safe)g(and)i(will)e(execute)h(as)f(exp)r(ected.)100 3530 y Ff(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f)q(,)36 b(len,)f(group,)i(0\);)100 3605 y(MPI_BC)q(A)q(S)q(T\()q(b)q(u)q(f)q(,)f(len,)f(group,)i(0\);)p eop %%Page: 16 17 bop 100 -134 a Fk(16)938 b Fh(CHAPTER)17 b(1.)48 b(COLLECTIVE)17 b(COMMUNICA)-5 b(TION)221 60 y Fk(Tw)n(o)26 b(consecutiv)n(e)h(broadcasts,)j (in)c(the)h(same)d(group,)29 b(with)d(the)h(same)d(tag)j(and)g(ro)r(ot.)44 b(Since)100 135 y(p)r(oin)n(t-to-p)r(oin)o(t)22 b(comm)n(unication)c(preserv) n(es)i(the)g(order)g(of)e(messages)f(here,)j(to)r(o,)f(messages)d(from)i(one) 100 211 y(broadcast)j(will)c(not)j(b)r(e)e(confused)j(with)d(messages)f(from) h(the)h(other)h(broadcast;)i(the)d(program)h(is)d(safe)100 286 y(and)22 b(will)d(execute)i(as)g(in)n(tended.)100 428 y Ff(MPI_RA)q(N)q(K)q(\(&)q(r)q(a)q(n)q(k,)37 b(group\))100 503 y(if)c(\(rank)q(==)q(0)q(\))164 578 y({)195 653 y(MPI_B)q(CA)q(S)q(T)q(C)q (\(b)q(u)q(f)q(,)k(len,)e(group,)h(0\);)195 729 y(MPI_S)q(EN)q(D)q(C)q(\()q (bu)q(f)q(,)h(len,)e(2,)e(tag,)i(group\))q(;)164 804 y(})100 879 y(elseif)i(\(rank=)q(=1)q(\))164 955 y({)195 1030 y(MPI_R)q(EC)q(V)q(C)q (\()q(bu)q(f)q(,)g(len,)e(MPI_DO)q(N)q(TC)q(A)q(R)q(E)q(,)h(tag,)f(group\))q (;)195 1105 y(MPI_B)q(CA)q(S)q(T)q(C)q(\(b)q(u)q(f)q(,)i(len,)e(group,)h (0\);)195 1180 y(MPI_R)q(EC)q(V)q(C)q(\()q(bu)q(f)q(,)h(len,)e(MPI_DO)q(N)q (TC)q(A)q(R)q(E)q(,)h(tag,)f(group\))q(;)164 1256 y(})100 1331 y(else)164 1406 y({)195 1481 y(MPI_S)q(EN)q(D)q(C)q(\()q(bu)q(f)q(,)i(len,)e (2,)e(tag,)i(group\))q(;)195 1557 y(MPI_B)q(CA)q(S)q(T)q(C)q(\(b)q(u)q(f)q(,) i(len,)e(group,)h(0\);)164 1632 y(})221 1774 y Fk(Pro)r(cess)23 b(zero)g(executes)h(a)f(broadcast)k(follo)n(w)n(ed)d(b)n(y)g(a)f(send)h(to)g (pro)r(cess)g(one;)h(pro)r(cess)f(t)n(w)n(o)g(ex-)100 1849 y(ecutes)f(a)f(send)h(to)g(pro)r(cess)g(one,)g(follo)n(w)n(ed)g(b)n(y)g(a)f (broadcast;)k(and)e(pro)r(cess)f(one)g(executes)f(a)h(receiv)n(e,)100 1924 y(a)29 b(broadcast)i(and)f(a)f(receiv)n(e.)52 b(A)27 b(p)r(ossible)j (outcome)e(is)g(for)i(the)g(op)r(erations)h(to)e(b)r(e)g(matc)n(hed)g(as)100 2000 y(illustrated)23 b(b)n(y)e(the)h(diagram)e(b)r(elo)n(w.)227 2292 y Ff(0)733 b(1)701 b(2)609 2442 y(/)33 b(-)f(>)65 b(receiv)q(e)386 b(/)33 b(-)g(send)545 2518 y(/)797 b(/)100 2593 y(broadc)q(a)q(s)q(t)100 b(/)287 b(broadc)q(a)q(s)q(t)227 b(/)96 b(broadc)q(a)q(s)q(t)450 2668 y(/)764 b(/)164 2743 y(send)98 b(-)415 b(receiv)q(e)68 b(<)33 b(-)221 3036 y Fk(The)20 b(reason)g(is)f(that)i(broadcast)i(is)18 b(not)j(a)e(sync)n(hronous)k(op)r(eration;)g(the)d(call)f(at)h(a)f(pro)r (cess)h(ma)n(y)100 3111 y(return)27 b(b)r(efore)e(the)g(other)h(pro)r(cesses) e(ha)n(v)n(e)h(en)n(tered)i(the)d(broadcast.)40 b(Th)n(us,)26 b(the)f(message)d(sen)n(t)j(b)n(y)100 3186 y(pro)r(cess)e(zero)g(can)g(arriv) n(e)h(to)f(pro)r(cess)g(one)g(b)r(efore)h(the)f(message)e(sen)n(t)j(b)n(y)f (pro)r(cess)g(t)n(w)n(o,)g(and)h(b)r(efore)100 3261 y(the)d(call)f(to)i (broadcast)h(on)e(pro)r(cess)h(one.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 11:56:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01628; Wed, 21 Apr 93 11:56:15 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29787; Wed, 21 Apr 93 11:53:34 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 11:53:27 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from almaden.ibm.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29755; Wed, 21 Apr 93 11:52:26 -0400 Message-Id: <9304211552.AA29755@CS.UTK.EDU> Received: from almaden.ibm.com by almaden.ibm.com (IBM VM SMTP V2R2) with BSMTP id 2750; Wed, 21 Apr 93 08:52:59 PDT Date: Wed, 21 Apr 93 08:44:31 PDT From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: the CCL paper by Bala et al. Hi, This is the postscript file of the CCL paper I brought to the last MPI meeting (same version), in case you missed a copy then. It has appeared as an IBM Research Report RJ 9284, April 1993. Let me know if you prefer to have a Latex file of it. All comments are welcome. Regards, -- Howard %!PS-Adobe-2.0 %%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software %%Title: CCL.dvi %%Pages: 24 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N /p 3 def @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}N /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}if ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}N /@endspecial{grestore clear SpecialSave restore end}N /@defspecial{SDict begin}N /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}N /st{stroke SaveX SaveY moveto}N /fil{fill SaveX SaveY moveto}N /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}N end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 108 df<3C000C000C00180018001800187031 903230340038007F00618061906190C1A0C0C00C117E9010>107 D E /Fb 6 116 df<07C00C20107020706000C000C000C00080008000C010C02060C03F000C0E7E8D0F> 99 D<00180038001000000000000000000000000001C0022004300430086000600060006000C0 00C000C000C001800180018001806300E300C60078000D1D80960E>106 D<1F0006000600060006000C000C000C000C00181C1866188E190C32003C003F00318060C060C4 60C460C8C0C8C0700F177E9612>I<383C0044C6004702004602008E06000C06000C06000C0C00 180C00180C40181840181880300880300F00120E7F8D15>110 D<38F04518463846308C000C00 0C000C001800180018001800300030000D0E7F8D10>114 D<07C00C201870187038001E000FC0 03E000606060E060C0C0C1803F000C0E7E8D10>I E /Fc 18 122 df<03CC0E2E181C381C301C 701CE038E038E038E038C072C072C07260F261341E180F107C8F14>97 D<7E000E000E000E001C 001C001C001C00380038003BC03C307830701870187018E038E038E038E038C070C060C0E060C0 63801E000D1A7C9912>I<01F006080C181838301070006000E000E000E000E000E008E0106020 30C01F000D107C8F12>I<001F80000380000380000380000700000700000700000700000E0000 0E0003CE000E2E00181C00381C00301C00701C00E03800E03800E03800E03800C07200C07200C0 720060F2006134001E1800111A7C9914>I<01E006181C08380870087010FFE0E000E000E000E0 00E0086010602030C01F000D107C8F12>I<00F300038B800607000E07000C07001C0700380E00 380E00380E00380E00301C00301C00301C00183C0018780007B800003800003800007000607000 E0E000C1C0007F000011177E8F12>103 D<1F8000038000038000038000070000070000070000 0700000E00000E00000E7C000F86001E07001E07001C07001C0700380E00380E00380E00381C00 701C80701C80703880703900E01900600E00111A7E9914>I<030706000000000000384C4E8E9C 9C1C3838707272E2E4643808197C980C>I<1F8003800380038007000700070007000E000E000E 0E0E131C271C431C801F003C003F8039C038E070E270E270E270E4E0646038101A7E9912>107 D<3F0707070E0E0E0E1C1C1C1C3838383870707070E4E4E4E46830081A7D990A>I<307C005986 009E07009E07009C07009C0700380E00380E00380E00381C00701C80701C80703880703900E019 00600E0011107C8F16>110 D<01F006180C0C180E300E700E600EE00EE00EE00CE01CE018E030 606030C01F000F107C8F14>I<030F000590C009E0C009C06009C06009C0600380E00380E00380 E00380E00701C00701800703800703000E8E000E78000E00000E00001C00001C00001C00001C00 00FF00001317808F14>I<30F059189E389C189C009C0038003800380038007000700070007000 E00060000D107C8F10>114 D<03E004300830187018601C001F801FC00FE000E00060E060E060 80C041803E000C107D8F10>I<06000E000E000E000E001C001C00FFC01C003800380038003800 7000700070007000E100E100E100E200640038000A177C960D>I<38064C074E0E8E0E9C0E9C0E 1C1C381C381C381C7039703970393079389A0F0C10107C8F15>I<38064C074E0E8E0E9C0E9C0E 1C1C381C381C381C703870387038307838F00F700070006060E0E1C0C18047003C0010177C8F13 >121 D E /Fd 19 120 df<00FC000182000703000607000E02000E00000E00000E00000E0000 0E0000FFFF000E07000E07000E07000E07000E07000E07000E07000E07000E07000E07000E0700 0E07000E07000E07007F0FE0131A809915>12 D<60F0F06004047D830B>46 D<0FC21836200E6006C006C002C002C002E00070007E003FE01FF807FC003E000E000700038003 80038003C002C006E004D81887E0101A7E9915>83 D<3F8070C070E020700070007007F01C7030 707070E070E071E071E0F171FB1E3C10107E8F13>97 D<07F80C1C381C30087000E000E000E000 E000E000E0007000300438080C1807E00E107F8F11>99 D<007E00000E00000E00000E00000E00 000E00000E00000E00000E00000E0003CE000C3E00380E00300E00700E00E00E00E00E00E00E00 E00E00E00E00E00E00600E00700E00381E001C2E0007CFC0121A7F9915>I<07C01C3030187018 600CE00CFFFCE000E000E000E0006000300438080C1807E00E107F8F11>I<01F0031807380E10 0E000E000E000E000E000E00FFC00E000E000E000E000E000E000E000E000E000E000E000E000E 000E007FE00D1A80990C>I104 D<18003C003C00180000000000000000000000 0000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80091A80990A >I109 DI<07E01C38300C700E6006E007E007E007E007E007E007 6006700E381C1C3807E010107F8F13>I114 D<1F2060E04020C020C020F0007F003FC01FE000F0 80708030C030C020F0408F800C107F8F0F>I<0400040004000C000C001C003C00FFC01C001C00 1C001C001C001C001C001C001C201C201C201C201C200E4003800B177F960F>IIII E /Fe 2 51 df<0C003C00CC000C000C000C000C 000C000C000C000C000C000C000C000C00FF8009107E8F0F>49 D<1F00618040C08060C0600060 006000C00180030006000C00102020207FC0FFC00B107F8F0F>I E /Ff 4 52 df<003000003000003000003000003000003000003000003000003000003000003000FFFF FCFFFFFC0030000030000030000030000030000030000030000030000030000030000030001618 7E931B>43 D<03000700FF00070007000700070007000700070007000700070007000700070007 000700070007007FF00C157E9412>49 D<0F8030E040708030C038E03840380038007000700060 00C00180030006000C08080810183FF07FF0FFF00D157E9412>I<0FE030306018701C701C001C 00180038006007E000300018000C000E000EE00EE00EC00C401830300FE00F157F9412>I E /Fg 8 83 df<07F0001FFE00383F007C1F80FE0FC0FE0FC0FE0FE0FE07E07C07E03807E0000F E0000FC0000FC0001F80001F00003E0000780000F00000E00001C0000380600700600E00601C00 E01FFFC03FFFC07FFFC0FFFFC0FFFFC0131D7D9C1A>50 D68 D70 D<0007FC0200003FFF0E0000FE03DE0003F000FE0007E000 3E000FC0001E001F80001E003F00000E003F00000E007F000006007E000006007E00000600FE00 000000FE00000000FE00000000FE00000000FE00000000FE003FFFE0FE003FFFE07E00007E007E 00007E007F00007E003F00007E003F00007E001F80007E000FC0007E0007E0007E0003F000FE00 00FE01FE00003FFF8E000007FC0600231F7D9E29>I73 D77 D<001FF80000FFFF0001F81F8007E007E00FC003F01F8001F81F00 00F83F0000FC7F0000FE7E00007E7E00007EFE00007FFE00007FFE00007FFE00007FFE00007FFE 00007FFE00007FFE00007FFE00007F7E00007E7F0000FE7F0000FE3F0000FC3F8001FC1F8001F8 0FC003F007E007E001F81F8000FFFF00001FF800201F7D9E27>79 D82 D E /Fh 61 122 df<001000000800000400000600000200000300000300000300000100000180 000180000180000180000180000180000180000380000380000380000300000300000300000700 000700000600000600000E00000C00000C00001C0000180000380000300000700000600000E000 00C0000180000100000300000600000C0000180000300000600000800000112E80A113>41 D<1C3C3C3C3C040408081020204080060E7D840E>44 D<7FF0FFE07FE00C037D8A10>I<70F8F8 F0E005057B840E>I<00000040000000C000000180000001800000030000000300000006000000 060000000C0000001800000018000000300000003000000060000000C0000000C0000001800000 018000000300000003000000060000000C0000000C000000180000001800000030000000300000 0060000000C0000000C0000001800000018000000300000003000000060000000C0000000C0000 001800000018000000300000003000000060000000C0000000C0000000800000001A2D7FA117> I<000F800030E000E07001C0700380300380380700380F00780F00780E00781E00781E00703C00 F03C00F03C00F03C00F07801E07801E07801E07801C07003C0F003C0F00380F00780F007007007 00700E00701C003038001870000FC000151F7C9D17>I<000200020006000E003C00DC031C001C 0038003800380038007000700070007000E000E000E000E001C001C001C001C003800380038003 800780FFF80F1E7B9D17>I<001F000061800080E00100E00200700220700420700410700820F0 0820F00820F00840E00881E00703C0000380000700000C00001800006000008000030000040000 0800401000401000802001807E030047FF0041FE0080FC00807800141F7C9D17>I<001F800060 E00080700100300200380420380420380410380420700460700380600000E00001C000030000FE 00001C00000600000700000780000780000780300780780780780780F00F00800F00401E00401C 0040380020E0001F8000151F7C9D17>I<0000600000E00000E00000E00001C00001C00001C000 0380000380000300000700000700000600000E00000C0000180000180000300000300000630000 C700008700010700030700060E00040E00080E003F8E00607C00801FC0001C00001C0000380000 380000380000380000700000700000600013277E9D17>I<00C06000FFC001FF8001FE00010000 010000020000020000020000020000040000047800058C000606000C0700080700000780000780 000780000780000F00700F00F00F00F00E00E01E00801C0080380080300040600061C0001F0000 131F7B9D17>I<08E0100BF01017F8201FF8603E19C0380E802000806001004003008003000006 00000E00000C00001C00001C0000380000380000700000700000F00000F00001E00001E00001E0 0003C00003C00003C00007C000078000078000030000141F799D17>55 D<001F000061800080C0 0100600300600600600600600600600E00C00F00800F818007C30007E40003F80001F80003FC00 047E00183F00300F00200700600700C00300C00300C00300800600800600C00C00C00800403000 3060001F8000131F7B9D17>I<001F0000718000C0C00180C00380E00700E00F00E00F01E01E01 E01E01E01E01E01E01C01C03C01C03C01C03C01C07C01C0F800C0F8006378003C700000F00000E 00000E00001C00601C00F03800F07000E0600080C0004380003E0000131F7B9D17>I<070F1F1F 0E0000000000000000000070F8F8F0E008147B930E>I<00000200000006000000060000000E00 00001E0000001E0000003F0000002F0000004F0000004F0000008F0000010F0000010F0000020F 0000020F0000040F00000C0F0000080F0000100F0000100F0000200F80003FFF800040078000C0 07800080078001000780010007800200078002000780060007801E000F80FF807FF81D207E9F22 >65 D<01FFFFC0001E00F0001E0078001E0038001E003C003C003C003C003C003C003C003C003C 0078007800780078007800F0007801E000F0078000FFFE0000F00F8000F003C001E001C001E001 E001E001E001E001E003C001E003C001E003C001E003C001C0078003C00780078007800F000780 1E000F007800FFFFE0001E1F7D9E20>I<0000FE0200078186001C004C0038003C0060003C00C0 001C01C0001803800018070000180F0000181E0000101E0000103C0000003C0000007800000078 0000007800000078000000F0000000F0000000F0000000F0000000F00000807000008070000080 700001003800010038000200180004000C001800060020000381C00000FE00001F217A9F21>I< 01FFFF80001E00E0001E0070001E0038001E001C003C001C003C000E003C000E003C000E007800 0E0078000E0078000E0078000E00F0001E00F0001E00F0001E00F0001E01E0003C01E0003C01E0 003C01E0007803C0007003C0007003C000E003C001C0078001C00780038007800E0007801C000F 007000FFFFC0001F1F7D9E22>I<01FFFFFE001E001C001E000C001E0004001E0004003C000400 3C0004003C0004003C00040078080800780800007808000078180000F0300000FFF00000F03000 00F0300001E0200001E0200001E0200001E0001003C0002003C0002003C0004003C00040078000 800780018007800100078007000F001F00FFFFFE001F1F7D9E1F>I<01FFFFFC001E0038001E00 18001E0008001E0008003C0008003C0008003C0008003C00080078001000780800007808000078 080000F0100000F0300000FFF00000F0300001E0200001E0200001E0200001E0200003C0000003 C0000003C0000003C00000078000000780000007800000078000000F800000FFF800001E1F7D9E 1E>I<0000FC040007030C001C00980030007800E0007801C00038038000300380003007000030 0E0000301E0000201E0000203C0000003C00000078000000780000007800000078000000F00000 00F000FFF0F0000780F0000780F0000F0070000F0070000F0070000F0070001E0038001E001800 3E001C002E000E00CC000383040000FC00001E217A9F23>I<01FFF3FFE0001F003E00001E003C 00001E003C00001E003C00003C007800003C007800003C007800003C007800007800F000007800 F000007800F000007800F00000F001E00000FFFFE00000F001E00000F001E00001E003C00001E0 03C00001E003C00001E003C00003C007800003C007800003C007800003C007800007800F000007 800F000007800F000007800F00000F801F0000FFF1FFE000231F7D9E22>I<01FFF0001F00001E 00001E00001E00003C00003C00003C00003C0000780000780000780000780000F00000F00000F0 0000F00001E00001E00001E00001E00003C00003C00003C00003C0000780000780000780000780 000F8000FFF800141F7D9E12>I<001FFF0000F80000F00000F00000F00001E00001E00001E000 01E00003C00003C00003C00003C0000780000780000780000780000F00000F00000F00000F0000 1E00001E00301E00781E00F83C00F83C00F0780080700040E00021C0001F000018207D9E18>I< 01FFF800001F0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000007800 0000780000007800000078000000F0000000F0000000F0000000F0000001E0000001E0000001E0 000001E0008003C0010003C0010003C0030003C00200078006000780060007800C0007801C000F 007800FFFFF800191F7D9E1D>76 D<01FE00007FC0001E0000FC00001E0000F800001700017800 00170001780000270002F00000270004F00000270004F00000270008F00000470009E000004700 11E00000470021E00000470021E00000870043C00000838043C00000838083C00000838083C000 0103810780000103820780000103820780000103840780000203840F00000203880F0000020390 0F00000203900F00000401E01E00000401E01E00000401C01E00000C01801E00001C01803E0000 FF8103FFC0002A1F7D9E29>I<01FF007FE0001F000F00001F0004000017800400001780040000 278008000023C008000023C008000023C008000041E010000041E010000041F010000040F01000 0080F0200000807820000080782000008078200001003C400001003C400001003C400001001E40 0002001E800002001E800002000F800002000F800004000F0000040007000004000700000C0007 00001C00020000FF80020000231F7D9E22>I<0001FC0000070700001C01C0003000E000E00060 01C000700380007007800038070000380E0000381E0000381C0000383C0000383C000038780000 78780000787800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F00003C07000 03807000070078000F0038001E0038003C001C0070000E00E0000783800001FC00001D217A9F23 >I<01FFFF80001E00E0001E0070001E0038001E003C003C003C003C003C003C003C003C003C00 78007800780078007800F0007800E000F003C000F00F0000FFFC0000F0000001E0000001E00000 01E0000001E0000003C0000003C0000003C0000003C00000078000000780000007800000078000 000F800000FFF000001E1F7D9E1F>I<01FFFF00001E03C0001E00E0001E0070001E0078003C00 78003C0078003C0078003C0078007800F0007800F0007801E0007801C000F0070000F01E0000FF F00000F0380001E01C0001E01E0001E00E0001E00F0003C01E0003C01E0003C01E0003C01E0007 803C0007803C0807803C0807803C100F801C10FFF00C20000007C01D207D9E21>82 D<0007E040001C18C0003005800060038000C0038001C001800180010003800100038001000380 01000380000003C0000003C0000003F8000001FF800001FFE000007FF000001FF0000001F80000 00780000007800000038000000380020003800200038002000300060007000600060006000E000 7000C000E8038000C606000081F800001A217D9F1A>I<0FFFFFF01E0780E01807802010078020 20078020200F0020600F0020400F0020400F0020801E0040001E0000001E0000001E0000003C00 00003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F0 000000F0000001E0000001E0000001E0000001E0000003E00000FFFF00001C1F789E21>I<7FFC 1FF807C003C00780010007800100078001000F0002000F0002000F0002000F0002001E0004001E 0004001E0004001E0004003C0008003C0008003C0008003C000800780010007800100078001000 78001000F0002000F0002000F0002000F0004000F0004000700080007001000030020000380400 000C18000007E000001D20779E22>III<00 FFF07FE0000F801F00000F001C00000F801000000780300000078020000007C040000003C08000 0003C100000003E200000001E400000001EC00000001F800000000F000000000F800000000F800 000000F8000000017C000000023C000000063C000000043E000000081E000000101E000000201F 000000400F000000800F000001800F8000010007800007000780001F000FC000FFC07FF800231F 7E9E22>I<00F1800389C00707800E03801C03803C0380380700780700780700780700F00E00F0 0E00F00E00F00E20F01C40F01C40703C40705C40308C800F070013147C9317>97 D<07803F8007000700070007000E000E000E000E001C001C001CF01D0C3A0E3C0E380F380F700F 700F700F700FE01EE01EE01EE01CE03CE038607060E031C01F0010207B9F15>I<007E0001C100 0300800E07801E07801C07003C0200780000780000780000F00000F00000F00000F00000F00000 70010070020030040018380007C00011147C9315>I<0000780003F80000700000700000700000 700000E00000E00000E00000E00001C00001C000F1C00389C00707800E03801C03803C03803807 00780700780700780700F00E00F00E00F00E00F00E20F01C40F01C40703C40705C40308C800F07 0015207C9F17>I<007C01C207010E011C013C013802780C7BF07C00F000F000F000F000700070 0170023804183807C010147C9315>I<00007800019C00033C00033C000718000700000700000E 00000E00000E00000E00000E0001FFE0001C00001C00001C00001C000038000038000038000038 0000380000700000700000700000700000700000700000E00000E00000E00000E00000C00001C0 0001C0000180003180007B0000F300006600003C00001629829F0E>I<003C6000E27001C1E003 80E00700E00F00E00E01C01E01C01E01C01E01C03C03803C03803C03803C03803C07003C07001C 0F001C17000C2E0003CE00000E00000E00001C00001C00301C00783800F0700060E0003F800014 1D7E9315>I<01E0000FE00001C00001C00001C00001C000038000038000038000038000070000 070000071E000763000E81800F01C00E01C00E01C01C03801C03801C03801C0380380700380700 380700380E10700E20700C20701C20700C40E00CC060070014207D9F17>I<00C001E001E001C0 00000000000000000000000000000E003300230043804300470087000E000E000E001C001C001C 003840388030807080310033001C000B1F7C9E0E>I<01E0000FE00001C00001C00001C00001C0 000380000380000380000380000700000700000703C00704200E08E00E11E00E21E00E40C01C80 001D00001E00001FC00038E000387000387000383840707080707080707080703100E03100601E 0013207D9F15>107 D<03C01FC0038003800380038007000700070007000E000E000E000E001C 001C001C001C0038003800380038007000700070007100E200E200E200E200640038000A207C9F 0C>I<1C0F80F0002630C318004740640C004780680E004700700E004700700E008E00E01C000E 00E01C000E00E01C000E00E01C001C01C038001C01C038001C01C038001C01C070803803807100 3803806100380380E10038038062007007006600300300380021147C9325>I<1C0F802630C047 40604780604700704700708E00E00E00E00E00E00E00E01C01C01C01C01C01C01C038438038838 03083807083803107003303001C016147C931A>I<007C0001C3000301800E01C01E01C01C01E0 3C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00700E00301800187000 07C00013147C9317>I<01C1E002621804741C04781C04701E04701E08E01E00E01E00E01E00E0 1E01C03C01C03C01C03C01C0380380780380700380E003C1C0072380071E000700000700000E00 000E00000E00000E00001C00001C0000FFC000171D809317>I<00F0400388C00705800E03801C 03803C0380380700780700780700780700F00E00F00E00F00E00F00E00F01C00F01C00703C0070 5C0030B8000F380000380000380000700000700000700000700000E00000E0000FFE00121D7C93 15>I<1C1E002661004783804787804707804703008E00000E00000E00000E00001C00001C0000 1C00001C000038000038000038000038000070000030000011147C9313>I<00FC030206010C03 0C070C060C000F800FF007F803FC003E000E700EF00CF00CE008401020601F8010147D9313>I< 018001C0038003800380038007000700FFF007000E000E000E000E001C001C001C001C00380038 0038003820704070407080708031001E000C1C7C9B0F>I<0E00C03300E02301C04381C04301C0 4701C08703800E03800E03800E03801C07001C07001C07001C07101C0E20180E20180E201C1E20 0C264007C38014147C9318>I<0E03803307802307C04383C04301C04700C08700800E00800E00 800E00801C01001C01001C01001C02001C02001C04001C04001C08000E300003C00012147C9315 >I<0E00C1C03300E3C02301C3E04381C1E04301C0E04701C060870380400E0380400E0380400E 0380401C0700801C0700801C0700801C0701001C0701001C0602001C0F02000C0F04000E130800 03E1F0001B147C931E>I<0383800CC4401068E01071E02071E02070C040E00000E00000E00000 E00001C00001C00001C00001C040638080F38080F38100E5810084C60078780013147D9315>I< 0E00C03300E02301C04381C04301C04701C08703800E03800E03800E03801C07001C07001C0700 1C07001C0E00180E00180E001C1E000C3C0007DC00001C00001C00003800F03800F07000E06000 C0C0004380003E0000131D7C9316>I E /Fi 14 113 df0 D<70F8F8F87005057C8D0D>I<400004C0000C6000183000301800600C00C00601800303000186 0000CC0000780000300000300000780000CC000186000303000601800C00C01800603000306000 18C0000C40000416187A9623>I<00020000000600000006000000060000000600000006000000 0600000006000000060000000600000006000000060000000600000006000000060000FFFFFFF0 FFFFFFF00006000000060000000600000006000000060000000600000006000000060000000600 0000060000000600000006000000060000FFFFFFF0FFFFFFF01C207D9E23>6 D<03C00FF01FF83FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF80FF003C01012 7D9317>15 D<000000C0000003C000000F0000003C000000F0000003C00000070000001C000000 78000001E00000078000001E00000078000000E0000000780000001E0000000780000001E00000 00780000001C0000000700000003C0000000F00000003C0000000F00000003C0000000C0000000 000000000000000000000000000000000000000000000000007FFFFF80FFFFFFC01A247C9C23> 20 D<0FC000101FF000103FF80010703E0030E00F0070C007C0E08001FFC08000FF8080003F00 00000000000000000FC000101FF000103FF80010703E0030E00F0070C007C0E08001FFC08000FF 8080003F001C147D9523>25 D<003FF800FFF803C0000700000C00001800003000003000006000 00600000C00000C00000C00000FFFFF8FFFFF8C00000C00000C000006000006000003000003000 001800000C000007000003C00000FFF8003FF8151C7C981E>50 D<00FE0003FF800F01E01C0070 30001860000C60000CC00006C00006C00006C00006C00006C00006C00006C00006C00006C00006 C00006C00006C00006C00006C00006C00006C00006C00006C00006C00006400002171C7D9A1E> 92 D<7FE0FFE0C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000 C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C0 00C000C000C000C000C000C000C00040000B2E78A114>100 DI<000F0038006000E001C001C001C001C001C001C001C001C001C001C001C001 C001C001C001C0038007001E00F8001E000700038001C001C001C001C001C001C001C001C001C0 01C001C001C001C001C001C000E000600038000F102D7DA117>II<0000000008000000001800000000300000000030000000006000000000600000 0000C000000000C000000001800000000180000000030000000003000000000600000000060000 00000C000000000C00000000180000000018000000003000000000300000000060000000006000 000000C000060000C0001E000180002F000180004F000300008780030000078006000003C00600 0003C00C000003C00C000001E018000001E018000000F030000000F03000000078600000007860 0000003CC00000003CC00000001F800000001F800000000F000000000F00000000060000000006 000000252E7E8126>112 D E /Fj 22 117 df<70F8FCFC74040404080810102040060E7C840D> 59 D<000100030003000600060006000C000C000C001800180018003000300030006000600060 00C000C000C00180018001800300030003000600060006000C000C000C00180018001800300030 003000600060006000C000C000C000102D7DA117>61 DI<0000FE0200078186001C004C0038003C0060003C00C0 001C01C0001803800018070000180F0000181E0000101E0000103C0000003C0000007800000078 0000007800000078000000F0000000F0000000F0000000F0000000F00000807000008070000080 700001003800010038000200180004000C001800060020000381C00000FE00001F217E9F20>67 D<00007E0100038183000E00460038002E0070001E00E0000E01C0000C0380000C0700000C0F00 000C1E0000081E0000083C0000003C00000078000000780000007800000078000000F0000000F0 007FFCF00001E0F00001E0F00003C0700003C0700003C0700003C038000780380007801C000F80 0C000B80060033000380C100007F000020217E9F24>71 D<0001FC0000070700001C01C0003000 E000E0006001C000700380007007800038070000380E0000381E0000381C0000383C0000383C00 003878000078780000787800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F0 0003C0700003807000070078000F0038001E0038003C001C0070000E00E0000783800001FC0000 1D217E9F23>79 D<00FFFFC0000F0070000F0038000F001C000F001E001E001E001E001E001E00 1E001E001E003C003C003C003C003C0078003C0070007800E000780380007FFE000078000000F0 000000F0000000F0000000F0000001E0000001E0000001E0000001E0000003C0000003C0000003 C0000003C0000007C00000FFFC00001F1F7E9E1D>I<0007E0800018118000300B000060070000 C0070001C0030001800200038002000380020003800200038000000380000003C0000003F80000 03FF800001FFC00000FFE000003FF0000003F0000000F000000070000000700000007000200070 0020007000200060006000E0006000C0006001C00070018000E8030000C60E000081F800001921 7D9F1C>83 D<0FFFFFFC1E03C0381803C0181003C0082003C00820078008600780084007800840 078008800F0010000F0000000F0000000F0000001E0000001E0000001E0000001E0000003C0000 003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F000 0000F0000001F000007FFFC0001E1F7F9E1B>I<00FFF83FF8000FC00F80000F80060000078004 000007C008000003C010000003C020000003E040000001E080000001F100000000F300000000F6 00000000FC0000000078000000007C000000007C000000007C00000000BE000000011E00000002 1E000000061F0000000C0F000000080F800000100780000020078000004007C000008003C00001 0003E000030003E0000F0007E000FFE01FFE00251F7F9E26>88 DI<00007C0000CE00019E00039E00030C000700000700000700000700000E00000E00000E0000 FFF0000E00000E00001C00001C00001C00001C00001C0000380000380000380000380000380000 700000700000700000700000700000E00000E00000E00000E00000C00001C000318000798000F3 00006200003C000017297E9F16>102 D<00E001E001E000C00000000000000000000000000000 0E00130023804380438043808700070007000E000E001C001C001C20384038403840388019000E 000B1F7E9E10>105 D<0000C00001E00001E00001C00000000000000000000000000000000000 00000000001E00006300004380008380010380010380020700000700000700000700000E00000E 00000E00000E00001C00001C00001C00001C000038000038000038000038000070000070003070 0078E000F1C0006380003E00001328819E13>I<01E0000FE00001C00001C00001C00001C00003 80000380000380000380000700000700000701E00706100E08700E10F00E20F00E40601C80001D 00001E00001FC000387000383800383800381C20703840703840703840701880E01880600F0014 207E9F18>I<1E07C07C00231861860023A032030043C034030043803803804380380380870070 07000700700700070070070007007007000E00E00E000E00E00E000E00E00E000E00E01C101C01 C01C201C01C038201C01C038401C01C0184038038018801801800F0024147E9328>109 D<1E07802318C023A06043C0704380704380708700E00700E00700E00700E00E01C00E01C00E01 C00E03821C03841C07041C07081C03083803101801E017147E931B>I<03C1E004621804741C08 781C08701E08701E10E01E00E01E00E01E00E01E01C03C01C03C01C03C01C03803807803807003 80E003C1C0072380071E000700000700000E00000E00000E00000E00001C00001C0000FFC00017 1D819317>112 D<00F0400388C00705800E03801C03803C0380380700780700780700780700F0 0E00F00E00F00E00F00E00F01C00F01C00703C00705C0030B8000F380000380000380000700000 700000700000700000E00000E0000FFE00121D7E9314>I<1E1E0023210023C38043C780438780 4383008700000700000700000700000E00000E00000E00000E00001C00001C00001C00001C0000 38000018000011147E9315>I<007C018203010603060706060E00078007F803FC01FE001F0007 7007F006F006E004400820301FC010147E9315>I<00C000E001C001C001C001C003800380FFF8 038007000700070007000E000E000E000E001C001C001C001C10382038203820384018800F000D 1C7F9B10>I E /Fk 45 123 df<000FF000007FFC0001F80E0003E01F0007C03F000F803F000F 803F000F801E000F800C000F8000000F8000000F8000000F800000FFFFFF00FFFFFF000F801F00 0F801F000F801F000F801F000F801F000F801F000F801F000F801F000F801F000F801F000F801F 000F801F000F801F000F801F000F801F000F801F000F801F000F801F007FF0FFE07FF0FFE01B23 7FA21F>12 D<387CFEFEFE7C3807077C8610>46 D<00180000780001F800FFF800FFF80001F800 01F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F8007FFFE07FFFE0 13207C9F1C>49 D<03FC000FFF003C1FC07007E07C07F0FE03F0FE03F8FE03F8FE01F87C01F838 03F80003F80003F00003F00007E00007C0000F80001F00003E0000380000700000E01801C01803 80180700180E00380FFFF01FFFF03FFFF07FFFF0FFFFF0FFFFF015207D9F1C>I<00FE0007FFC0 0F07E01E03F03F03F03F81F83F81F83F81F81F03F81F03F00003F00003E00007C0001F8001FE00 01FF000007C00001F00001F80000FC0000FC3C00FE7E00FEFF00FEFF00FEFF00FEFF00FC7E01FC 7801F81E07F00FFFC001FE0017207E9F1C>I<0000E00001E00003E00003E00007E0000FE0001F E0001FE00037E00077E000E7E001C7E00187E00307E00707E00E07E00C07E01807E03807E07007 E0E007E0FFFFFEFFFFFE0007E00007E00007E00007E00007E00007E00007E000FFFE00FFFE1720 7E9F1C>I<1000201E01E01FFFC01FFF801FFF001FFE001FF8001BC00018000018000018000018 000019FC001FFF001E0FC01807E01803E00003F00003F00003F80003F83803F87C03F8FE03F8FE 03F8FC03F0FC03F07007E03007C01C1F800FFF0003F80015207D9F1C>I<000070000000007000 000000F800000000F800000000F800000001FC00000001FC00000003FE00000003FE00000003FE 00000006FF000000067F0000000E7F8000000C3F8000000C3F800000183FC00000181FC0000038 1FE00000300FE00000300FE00000600FF000006007F00000E007F80000FFFFF80000FFFFF80001 8001FC00018001FC00038001FE00030000FE00030000FE000600007F000600007F00FFE00FFFF8 FFE00FFFF825227EA12A>65 DI<0003FE0080 001FFF818000FF01E38001F8003F8003E0001F8007C0000F800F800007801F800007803F000003 803F000003807F000001807E000001807E00000180FE00000000FE00000000FE00000000FE0000 0000FE00000000FE00000000FE00000000FE000000007E000000007E000001807F000001803F00 0001803F000003801F800003000F8000030007C000060003F0000C0001F800380000FF00F00000 1FFFC0000003FE000021227DA128>II<0003 FE0040001FFFC0C0007F00F1C001F8003FC003F0000FC007C00007C00FC00003C01F800003C03F 000001C03F000001C07F000000C07E000000C07E000000C0FE00000000FE00000000FE00000000 FE00000000FE00000000FE00000000FE00000000FE000FFFFC7E000FFFFC7F00001FC07F00001F C03F00001FC03F00001FC01F80001FC00FC0001FC007E0001FC003F0001FC001FC003FC0007F80 E7C0001FFFC3C00003FF00C026227DA12C>71 D73 D76 DII<00 07FC0000003FFF800000FC07E00003F001F80007E000FC000FC0007E001F80003F001F80003F00 3F00001F803F00001F807F00001FC07E00000FC07E00000FC0FE00000FE0FE00000FE0FE00000F E0FE00000FE0FE00000FE0FE00000FE0FE00000FE0FE00000FE0FE00000FE07E00000FC07F0000 1FC07F00001FC03F00001F803F80003F801F80003F000FC0007E0007E000FC0003F001F80000FC 07E000003FFF80000007FC000023227DA12A>II82 D<01FC0407FF8C1F03FC3C 007C7C003C78001C78001CF8000CF8000CFC000CFC0000FF0000FFE0007FFF007FFFC03FFFF01F FFF80FFFFC03FFFE003FFE0003FF00007F00003F00003FC0001FC0001FC0001FE0001EE0001EF0 003CFC003CFF00F8C7FFE080FF8018227DA11F>I<7FFFFFFF807FFFFFFF807E03F80F807803F8 07807003F803806003F80180E003F801C0E003F801C0C003F800C0C003F800C0C003F800C0C003 F800C00003F800000003F800000003F800000003F800000003F800000003F800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800000003F800000003F800000003F800000003F8000003FFFFF80003FFFFF80022227EA1 27>II<07FC001FFF803F07C03F03E03F01E0 3F01F01E01F00001F00001F0003FF003FDF01FC1F03F01F07E01F0FC01F0FC01F0FC01F0FC01F0 7E02F07E0CF81FF87F07E03F18167E951B>97 DI<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC 0000FC0000FC0000FC0000FC00007C00007E00007E00003E00301F00600FC0E007FF8000FE0014 167E9519>I<0001FE000001FE0000003E0000003E0000003E0000003E0000003E0000003E0000 003E0000003E0000003E0000003E0000003E0001FC3E0007FFBE000F81FE001F007E003E003E00 7E003E007C003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E00FC003E 007C003E007C003E003E007E001E00FE000F83BE0007FF3FC001FC3FC01A237EA21F>I<00FE00 07FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC0000 7C00007C00007E00003E00181F00300FC07003FFC000FF0015167E951A>I<003F8000FFC001E3 E003C7E007C7E00F87E00F83C00F80000F80000F80000F80000F80000F8000FFFC00FFFC000F80 000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F80000F80000F80000F80007FF8007FF80013237FA211>I<03FC1E0FFF7F1F0F8F3E07CF3C 03C07C03E07C03E07C03E07C03E07C03E03C03C03E07C01F0F801FFF0013FC0030000030000038 00003FFF801FFFF00FFFF81FFFFC3800FC70003EF0001EF0001EF0001EF0001E78003C7C007C3F 01F80FFFE001FF0018217E951C>II<1C003F007F007F007F003F001C000000000000000000000000000000FF00FF001F 001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE0 0B247EA310>I107 DI< FF07F007F000FF1FFC1FFC001F303E303E001F403E403E001F801F801F001F801F801F001F001F 001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F 001F00FFE0FFE0FFE0FFE0FFE0FFE02B167E9530>II< 00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC007EFC007E FC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE0017167E951C>II<00FE030007FF87000FC1C7001F006F003F003F007E003F007E00 1F007C001F00FC001F00FC001F00FC001F00FC001F00FC001F00FC001F00FC001F007E001F007E 001F003E003F001F007F000FC1DF0007FF9F0001FC1F0000001F0000001F0000001F0000001F00 00001F0000001F0000001F0000001F000000FFE00000FFE01B207E951E>II<0FF3003FFF00781F00600700E0 0300E00300F00300FC00007FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E0 0380F00700FC0E00EFFC00C7F00011167E9516>I<018000018000018000018000038000038000 0780000780000F80003F8000FFFF00FFFF000F80000F80000F80000F80000F80000F80000F8000 0F80000F80000F80000F80000F81800F81800F81800F81800F81800F830007C30003FE0000F800 11207F9F16>IIII<7FFFF07FFFF07C03E07007C0600FC0E01F80C01F00C03E00C07E0000FC0000F80001F00003 F03007E03007C0300F80701F80703F00603E00E07C03E0FFFFE0FFFFE014167E9519>122 D E /Fl 61 126 df<007000F001E003C007800F001E001C00380038007000700070007000E000 E000E000E000E000E000E000E0007000700070007000380038001C001E000F00078003C001F000 F000700C24799F18>40 D<6000F00078003C001E000F000780038001C001C000E000E000E000E0 0070007000700070007000700070007000E000E000E000E001C001C0038007800F001E003C0078 00F00060000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC00 07F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D9718> I<1C3E7E7F3F1F070E1E7CF860080C788518>44 D<7FFF00FFFF80FFFF807FFF0011047D8F18> I<01F00007FC000FFE001F1F001C07003803807803C07001C07001C0E000E0E000E0E000E0E000 E0E000E0E000E0E000E0E000E0E000E0F001E07001C07001C07803C03803801C07001F1F000FFE 0007FC0001F000131C7E9B18>48 D<01800380038007800F803F80FF80FB804380038003800380 03800380038003800380038003800380038003800380038003807FFCFFFE7FFC0F1C7B9B18>I< 03F0000FFE003FFF007C0F807003C0E001C0F000E0F000E06000E00000E00000E00001C00001C0 0003C0000780000F00001E00003C0000780000F00001E00007C0000F80001E00E03C00E07FFFE0 FFFFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007807807803C07801C03001C00001C00003 C0000380000F0003FF0003FE0003FF000007800003C00001C00000E00000E00000E0F000E0F000 E0F001C0F003C07C07803FFF001FFE0003F800131C7E9B18>I<001F00003F0000770000770000 E70001E70001C7000387000787000707000E07001E07003C0700380700780700F00700FFFFF8FF FFF8FFFFF8000700000700000700000700000700000700007FF000FFF8007FF0151C7F9B18>I< 1FFF803FFF803FFF803800003800003800003800003800003800003800003800003BF8003FFE00 3FFF003C07801803C00001C00000E00000E06000E0F000E0F000E0E001C07003C07C0F803FFF00 1FFC0003F000131C7E9B18>I<007E0001FF0007FF800F83C01E03C01C03C03801803800007000 00700000E1F800E7FE00FFFF00FE0780F803C0F001C0F000E0E000E0F000E07000E07000E07000 E03801C03C03C01E07800FFF0007FE0001F800131C7E9B18>I<3078FCFC783000000000000000 003078FCFC78300614779318>58 D<183C7E7E3C180000000000000000183C7E7E3E1E0E1C3C78 F060071A789318>I<000300000780001F80003F00007E0001FC0003F00007E0001FC0003F0000 7E0000FC0000FC00007E00003F00001FC00007E00003F00001FC00007E00003F00001F80000780 00030011187D9918>I<7FFFC0FFFFE0FFFFE0FFFFE0000000000000000000000000FFFFE0FFFF E0FFFFE07FFFC0130C7E9318>I<00700000F80000F80000D80000D80001DC0001DC0001DC0001 8C00038E00038E00038E00038E000306000707000707000707000707000FFF800FFF800FFF800E 03800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<00F8E003FEE007FFE00F07E01E03E03C01E03800E07000E07000E0700000E00000E00000E0 0000E00000E00000E00000E00000E000007000007000E07000E03800E03C00E01E01C00F07C007 FF8003FE0000F800131C7E9B18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C00E0 1C00E01C00F01C00701C00701C00701C00701C00701C00701C00701C00701C00F01C00E01C00E0 1C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18>II<01F1C003FDC00FFFC01F0FC01C03C03803C03801C07001C07001C0700000E00000E00000E0 0000E00000E00000E00FF0E01FF0E00FF07001C07001C07003C03803C03803C01C07C01F0FC00F FFC003FDC001F1C0141C7E9B18>71 D<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01 C01C01C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C00001C00001C00001C00001C0007FFF00FFFF807FFF00111C7D9B 18>I<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E0070 7FFFF0FFFFF07FFFF0141C7F9B18>76 DI<7E07F0FF0FF87F07F0 1D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C01C31C0 1C39C01C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F 9B18>I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380E003 80E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780700700780F 007FFF003FFE000FF800111C7D9B18>II<7FFFF8FFFFF8FFFFF8 E07038E07038E07038E07038007000007000007000007000007000007000007000007000007000 00700000700000700000700000700000700000700000700000700007FF0007FF0007FF00151C7F 9B18>84 D86 D<7F8FE07F9FE07F8FE00E07000F0700070E00078E 00039C0003DC0001F80001F80000F00000F00000700000F00000F80001F80001DC00039E00038E 00070F000707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>88 D91 D93 D<7FFF00FFFF80FFFF807FFF0011047D7F18>95 D<1FE0003FF8007FFC00781E00300E 0000070000070000FF0007FF001FFF007F0700780700E00700E00700E00700F00F00781F003FFF F01FFBF007E1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E0000 0E3E000EFF800FFFC00FC1E00F80E00F00700E00700E00380E00380E00380E00380E00380E0038 0F00700F00700F80E00FC1E00FFFC00EFF80063E00151C809B18>I<01FE0007FF001FFF803E07 80380300700000700000E00000E00000E00000E00000E00000E000007000007001C03801C03E03 C01FFF8007FF0001FC0012147D9318>I<001F80003F80001F8000038000038000038000038000 038003E3800FFB801FFF803C1F80380F80700780700380E00380E00380E00380E00380E00380E0 0380700780700780380F803C1F801FFFF00FFBF803E3F0151C7E9B18>I<01F00007FC001FFE00 3E0F00380780700380700380E001C0E001C0FFFFC0FFFFC0FFFFC0E000007000007001C03801C0 3E03C01FFF8007FF0001FC0012147D9318>I<001F80007FC000FFE000E1E001C0C001C00001C0 0001C0007FFFC0FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<01E1F007FFF80F FFF81E1E301C0E003807003807003807003807003807001C0E001E1E001FFC001FF80039E00038 00001C00001FFE001FFFC03FFFE07801F0700070E00038E00038E00038E000387800F07E03F01F FFC00FFF8001FC00151F7F9318>I<7E0000FE00007E00000E00000E00000E00000E00000E0000 0E3E000EFF800FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C0 000380000000000000000000000000007FC000FFC0007FC00001C00001C00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D 7C9C18>I107 D<7FE000FFE0007FE00000E00000E00000E00000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7CE0E000FFFB F8007FFFF8001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C 1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F0019148193 18>I<7E3E00FEFF807FFFC00FC1C00F80E00F00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF 003E0F803803807001C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07 803E0F801FFF000FFE0001F00013147E9318>I<7E3E00FEFF807FFFC00FC1E00F80E00F00700E 00700E00380E00380E00380E00380E00380E00380F00700F00700F80E00FC1E00FFFC00EFF800E 3E000E00000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I< 7F87E0FF9FF07FBFF803F87803F03003E00003C00003C000038000038000038000038000038000 0380000380000380000380007FFE00FFFF007FFE0015147F9318>114 D<07F7003FFF007FFF00 780F00E00700E00700E007007C00007FE0001FFC0003FE00001F00600780E00380E00380F00380 F80F00FFFF00FFFC00E7F00011147D9318>I<0180000380000380000380000380007FFFC0FFFF C0FFFFC00380000380000380000380000380000380000380000380000380000380400380E00380 E00380E001C1C001FFC000FF80003E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E00E00E 00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F03E007FFFC03 FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E0380070700070700 070700038E00038E00038E00038E0001DC0001DC0001DC0000F80000F80000700015147F9318> II<7F8FF07F9FF07F8FF007 0700078E00039E0001DC0001F80000F80000700000F00000F80001DC00039E00038E000707000F 07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8FF00E01C00E03800E0380070380 070700070700038700038600038E0001CE0001CE0000CC0000CC0000DC00007800007800007800 00700000700000700000F00000E00079E0007BC0007F80003F00001E0000151E7F9318>I<3FFF F07FFFF07FFFF07001E07003C0700780000F00001E00003C0000F80001F00003C0000780000F00 701E00703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I<0007E0001FE0007FE000780000 E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00001E0007FC000FF 8000FF80007FC00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E000007800007FE0001FE00007E013247E9F18>I<7C0000FF0000FFC00003C00000E000 00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000F000007FC0003FE0 003FE0007FC000F00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00003C000FFC000FF00007C000013247E9F18>125 D E /Fm 30 121 df<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE0000FE0000FE0000FE0000 FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000 FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE007FFFFE7FFFFE7F FFFE17277BA622>49 D<00FF800003FFF0000FFFFC001F03FE003800FF007C007F80FE003FC0FF 003FC0FF003FE0FF001FE0FF001FE07E001FE03C003FE000003FE000003FC000003FC000007F80 00007F000000FE000000FC000001F8000003F0000003E00000078000000F0000001E0000003C00 E0007000E000E000E001C001C0038001C0070001C00FFFFFC01FFFFFC03FFFFFC07FFFFFC0FFFF FF80FFFFFF80FFFFFF801B277DA622>I<007F800003FFF00007FFFC000F81FE001F00FF003F80 FF003F807F803F807F803F807F801F807F800F007F800000FF000000FF000000FE000001FC0000 01F8000007F00000FFC00000FFF0000001FC0000007E0000007F0000007F8000003FC000003FC0 00003FE000003FE03C003FE07E003FE0FF003FE0FF003FE0FF003FC0FF007FC07E007F807C007F 003F01FE001FFFFC0007FFF00000FF80001B277DA622>I<00000E0000001E0000003E0000007E 000000FE000000FE000001FE000003FE0000077E00000E7E00000E7E00001C7E0000387E000070 7E0000E07E0000E07E0001C07E0003807E0007007E000E007E000E007E001C007E0038007E0070 007E00E0007E00FFFFFFF8FFFFFFF8FFFFFFF80000FE000000FE000000FE000000FE000000FE00 0000FE000000FE000000FE00007FFFF8007FFFF8007FFFF81D277EA622>I<0C0003000F803F00 0FFFFE000FFFFC000FFFF8000FFFF0000FFFE0000FFFC0000FFE00000E0000000E0000000E0000 000E0000000E0000000E0000000E7FC0000FFFF8000F80FC000E003E000C003F0000001F800000 1FC000001FC000001FE000001FE018001FE07C001FE0FE001FE0FE001FE0FE001FE0FE001FC0FC 001FC078003F8078003F803C007F001F01FE000FFFF80003FFF00000FF80001B277DA622>I<1C 003E007F00FF80FF80FF807F003E001C000000000000000000000000000000000000001C003E00 7F00FF80FF80FF807F003E001C00091B7B9A13>58 D<000003800000000007C00000000007C000 0000000FE0000000000FE0000000000FE0000000001FF0000000001FF0000000003FF800000000 3FF8000000003FF80000000073FC0000000073FC00000000F3FE00000000E1FE00000000E1FE00 000001C0FF00000001C0FF00000003C0FF80000003807F80000007807FC0000007003FC0000007 003FC000000E003FE000000E001FE000001E001FF000001C000FF000001FFFFFF000003FFFFFF8 00003FFFFFF80000780007FC0000700003FC0000700003FC0000E00001FE0000E00001FE0001E0 0001FF0001C00000FF0001C00000FF00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE2F297EA834 >65 D<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE00007F80 03FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803FC000 0003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF8000000000 FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F8000 0000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE000000700 0FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E000000FFF FFC0000003FFFF000000003FF8000029297CA832>67 D<00007FE003000003FFFC0700001FFFFF 0F00003FF00FFF0000FF8001FF0001FE0000FF0003F800003F0007F000003F000FF000001F001F E000000F001FE000000F003FC000000F003FC0000007007FC0000007007F80000007007F800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF 8000000000FF8000000000FF8001FFFFF87F8001FFFFF87F8001FFFFF87FC00000FF003FC00000 FF003FC00000FF001FE00000FF001FE00000FF000FF00000FF0007F00000FF0003F80000FF0001 FE0000FF0000FF8001FF00003FF007BF00001FFFFF1F000003FFFE0F0000007FF003002D297CA8 36>71 D73 D76 D80 D82 D<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC0001E0FC00 01E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001FFFFE000F FFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8600001F8 E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F80F3FFFF 00E0FFFE00C01FF0001D297CA826>I<01FF800007FFF0000F81F8001FC07E001FC07E001FC03F 000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F800FE03F803F80 3F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F807F00DF803F839FFC0F FF0FFC01FC03FC1E1B7E9A21>97 D<001FF80000FFFE0003F01F0007E03F800FC03F801F803F80 3F801F007F800E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF0000 00FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FF FC00001FE0001A1B7E9A1F>99 D<00003FF80000003FF80000003FF800000003F800000003F800 000003F800000003F800000003F800000003F800000003F800000003F800000003F800000003F8 00000003F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F8001F8003 F8003F8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F800FF00 03F800FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003F8001F 8003F8000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80212A7EA926>I<003FE0 0001FFF80003F07E0007C01F000F801F801F800F803F800FC07F000FC07F0007C07F0007E0FF00 07E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F0000007F0000007F0000003F 8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF0001B1B7E9A20>I<0007F0003F FC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007F00007F00007F00007F00007F00007F0 00FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007F00007F00007F00007F00007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF 807FFF80182A7EA915>I<07000FC01FE03FE03FE03FE01FE00FC0070000000000000000000000 00000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE0F2B7DAA14>105 D108 DII<003FE00001FFFC0003 F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F07F0007F0FF0007F8FF0007F8 FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007F07F0007F03F800FE03F800F E01F800FC00FC01F8007F07F0001FFFC00003FE0001D1B7E9A22>II114 D<03FE300FFFF01E03F03800F0700070F00070F00070F80070FC0000FFE0007F FE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE0003CF0003CF00038F80038FC 0070FF01E0F7FFC0C1FF00161B7E9A1B>I<00700000700000700000700000F00000F00000F000 01F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F03807F03807F03807F03807F03807F038 03F03803F87001F86000FFC0001F8015267FA51B>II120 D E /Fn 82 128 df<001F83E000F06E3001C078780380F8780300F03007007000070070000700 700007007000070070000700700007007000FFFFFF800700700007007000070070000700700007 007000070070000700700007007000070070000700700007007000070070000700700007007000 070070000700700007007000070070007FE3FF001D20809F1B>11 D<003F0000E0C001C0C00381 E00701E00701E0070000070000070000070000070000070000FFFFE00700E00700E00700E00700 E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E07FC3FE1720809F19>I<003FE000E0E001C1E00381E00700E00700E00700E00700E007 00E00700E00700E00700E0FFFFE00700E00700E00700E00700E00700E00700E00700E00700E007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E07FE7FE1720809F19>I< 001F81F80000F04F040001C07C06000380F80F000300F00F000700F00F00070070000007007000 000700700000070070000007007000000700700000FFFFFFFF0007007007000700700700070070 070007007007000700700700070070070007007007000700700700070070070007007007000700 70070007007007000700700700070070070007007007000700700700070070070007007007007F E3FE3FF02420809F26>I<70F8F8F8F8F8F8F87070707070707070707020202020200000000000 70F8F8F87005217CA00D>33 D<7038F87CFC7EFC7E743A04020402040208040804100810082010 40200F0E7E9F17>I<70F8FCFC74040404080810102040060E7C9F0D>39 D<0020004000800100020006000C000C00180018003000300030007000600060006000E000E000 E000E000E000E000E000E000E000E000E000E0006000600060007000300030003000180018000C 000C000600020001000080004000200B2E7DA112>I<800040002000100008000C000600060003 00030001800180018001C000C000C000C000E000E000E000E000E000E000E000E000E000E000E0 00E000C000C000C001C001800180018003000300060006000C00080010002000400080000B2E7D A112>I<0006000000060000000600000006000000060000000600000006000000060000000600 00000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF0000600000006 000000060000000600000006000000060000000600000006000000060000000600000006000000 0600000006000000060000000600001C207D9A23>43 D<70F8FCFC74040404080810102040060E 7C840D>II<70F8F8F87005057C840D>I<000100030003000600060006 000C000C000C00180018001800300030003000600060006000C000C000C0018001800180030003 0003000600060006000C000C000C00180018001800300030003000600060006000C000C000C000 102D7DA117>I<03F0000E1C001C0E00180600380700700380700380700380700380F003C0F003 C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07003807003 807003807807803807001806001C0E000E1C0003F000121F7E9D17>I<018003800F80F3800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038007C0FFFE0F1E7C9D17>I<03F0000C1C00100E00200700400780800780F007C0 F803C0F803C0F803C02007C00007C0000780000780000F00000E00001C00003800007000006000 00C0000180000300000600400C00401800401000803FFF807FFF80FFFF80121E7E9D17>I<03F0 000C1C00100E00200F00780F80780780780780380F80000F80000F00000F00000E00001C000038 0003F000003C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F007804007 80400F00200E001C3C0003F000121F7E9D17>I<000600000600000E00000E00001E00002E0000 2E00004E00008E00008E00010E00020E00020E00040E00080E00080E00100E00200E00200E0040 0E00C00E00FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE0141E7F9D17>I< 1803001FFE001FFC001FF8001FE00010000010000010000010000010000010000011F000161C00 180E001007001007800003800003800003C00003C00003C07003C0F003C0F003C0E00380400380 400700200600100E000C380003E000121F7E9D17>I<007C000182000701000E03800C07801C07 80380300380000780000700000700000F1F000F21C00F40600F80700F80380F80380F003C0F003 C0F003C0F003C0F003C07003C07003C07003803803803807001807000C0E00061C0001F000121F 7E9D17>I<4000007FFFC07FFF807FFF8040010080020080020080040000080000080000100000 200000200000400000400000C00000C00001C00001800003800003800003800003800007800007 8000078000078000078000078000078000030000121F7D9D17>I<03F0000C0C00100600300300 2001806001806001806001807001807803003E03003F06001FC8000FF00003F80007FC000C7E00 103F00300F806003804001C0C001C0C000C0C000C0C000C0C000806001802001001002000C0C00 03F000121F7E9D17>I<03F0000E18001C0C00380600380700700700700380F00380F00380F003 C0F003C0F003C0F003C0F003C07007C07007C03807C0180BC00E13C003E3C00003800003800003 80000700300700780600780E00700C002018001070000FC000121F7E9D17>I<70F8F8F8700000 000000000000000070F8F8F87005147C930D>I<70F8F8F8700000000000000000000070F0F8F8 78080808101010202040051D7C930D>I<7FFFFFE0FFFFFFF00000000000000000000000000000 000000000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D9023>61 D<000100000003800000038000000380000007C0000007C0000007C0000009E0000009E0000009 E0000010F0000010F0000010F00000207800002078000020780000403C0000403C0000403C0000 801E0000801E0000FFFE0001000F0001000F0001000F00020007800200078002000780040003C0 0E0003C01F0007E0FFC03FFE1F207F9F22>65 DI<000FC04000 7030C001C009C0038005C0070003C00E0001C01E0000C01C0000C03C0000C07C0000407C000040 78000040F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00780000007C0000407C0000403C0000401C0000401E0000800E000080070001000380020001C0 040000703800000FC0001A217D9F21>IIII<000FE0200078186000E004E0038002E0070001 E00F0000E01E0000601E0000603C0000603C0000207C00002078000020F8000000F8000000F800 0000F8000000F8000000F8000000F8000000F8007FFCF80003E0780001E07C0001E03C0001E03C 0001E01E0001E01E0001E00F0001E0070001E0038002E000E0046000781820000FE0001E217D9F 24>III<0FFFC0007C00003C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0020 3C00F83C00F83C00F83C00F0380040780040700030E0000F800012207E9E17>IIIII<001F800000F0F00001C0380007801E000F000F000E0007001E000780 3C0003C03C0003C07C0003E0780001E0780001E0F80001F0F80001F0F80001F0F80001F0F80001 F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03C0003C01E00 07800E0007000F000F0007801E0001C0380000F0F000001F80001C217D9F23>II82 D<07E0800C1980100780300380 600180600180E00180E00080E00080E00080F00000F000007800007F00003FF0001FFC000FFE00 03FF00001F800007800003C00003C00001C08001C08001C08001C08001C0C00180C00380E00300 F00600CE0C0081F80012217D9F19>I<7FFFFFE0780F01E0600F0060400F0020400F0020C00F00 30800F0010800F0010800F0010800F0010000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000001F800007FFFE001C1F7E9E21>IIII<7FFFF87C00F87000 F06001E04001E0C003C0C003C0800780800F80800F00001E00001E00003C00003C0000780000F8 0000F00001E00001E00003C00403C0040780040F80040F000C1E000C1E00083C00183C00187800 38F801F8FFFFF8161F7D9E1C>90 DI<0804100820102010402040 20804080408040B85CFC7EFC7E7C3E381C0F0E7B9F17>II<1FE0 00303000781800781C00300E00000E00000E00000E0000FE00078E001E0E00380E00780E00F00E 10F00E10F00E10F01E10781E103867200F83C014147E9317>97 D<0E0000FE00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E3E000EC3800F01C00F00E00E00E0 0E00700E00700E00780E00780E00780E00780E00780E00780E00700E00700E00E00F00E00D01C0 0CC300083E0015207F9F19>I<03F80E0C1C1E381E380C70007000F000F000F000F000F000F000 70007000380138011C020E0C03F010147E9314>I<000380003F80000380000380000380000380 00038000038000038000038000038000038003E380061B801C0780380380380380700380700380 F00380F00380F00380F00380F00380F003807003807003803803803807801C07800E1B8003E3F8 15207E9F19>I<03F0000E1C001C0E00380700380700700700700380F00380F00380FFFF80F000 00F00000F000007000007000003800801800800C010007060001F80011147F9314>I<007C00C6 018F038F07060700070007000700070007000700FFF00700070007000700070007000700070007 000700070007000700070007000700070007007FF01020809F0E>I<0000E003E3300E3C301C1C 30380E00780F00780F00780F00780F00780F00380E001C1C001E380033E0002000002000003000 003000003FFE001FFF800FFFC03001E0600070C00030C00030C00030C000306000603000C01C03 8003FC00141F7F9417>I<0E0000FE00000E00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E3E000E43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16207F9F19>I<1C001E 003E001E001C000000000000000000000000000E007E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E00FFC00A1F809E0C>I<00E001F001F001F000E00000 00000000000000000000007007F000F00070007000700070007000700070007000700070007000 700070007000700070007000700070007000706070F060F0C061803F000C28829E0E>I<0E0000 FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FF00E03C0 0E03000E02000E04000E08000E10000E30000E70000EF8000F38000E1C000E1E000E0E000E0700 0E07800E03800E03C00E03E0FFCFF815207F9F18>I<0E00FE000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E00FFE00B20809F0C>I<0E1F01F000FE618618000E81C81C000F00F00E000F00F00E 000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E0 0E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00FFE7FE7FE02314 7F9326>I<0E3E00FE43000E81800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC16147F9319>I<01F800070E00 1C03803801C03801C07000E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07000E0 3801C03801C01C0380070E0001F80014147F9317>I<0E3E00FEC3800F01C00F00E00E00E00E00 F00E00700E00780E00780E00780E00780E00780E00780E00700E00F00E00E00F01E00F01C00EC3 000E3E000E00000E00000E00000E00000E00000E00000E00000E0000FFE000151D7F9319>I<03 E0800619801C05803C0780380380780380700380F00380F00380F00380F00380F00380F0038070 03807803803803803807801C0B800E138003E38000038000038000038000038000038000038000 0380000380003FF8151D7E9318>I<0E78FE8C0F1E0F1E0F0C0E000E000E000E000E000E000E00 0E000E000E000E000E000E000E00FFE00F147F9312>I<1F9030704030C010C010C010E0007800 7F803FE00FF00070803880188018C018C018E030D0608F800D147E9312>I<0200020002000600 06000E000E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E080E080E080E 080E080610031001E00D1C7F9B12>I<0E01C0FE1FC00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00603C0030DC001F1FC1614 7F9319>III<7FC3FC0F01E00701C007018003810001C20000E40000EC00007800003800003C00007C00 004E000087000107000303800201C00601E01E01E0FF07FE1714809318>II<3FFF380E200E201C40384078407000E001E001C00380078007010E011E011C 0338027006700EFFFE10147F9314>III<30 307878F87C787830300E057C9E17>127 D E /Fo 49 123 df<007E0001C1800301800703C00E 03C00E01800E00000E00000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C 17>12 D<003F07E00001C09C18000380F018000701F03C000E01E03C000E00E018000E00E00000 0E00E000000E00E000000E00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E0 1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>14 D<004000800100020006000C000C0018001800300030007000600060006000E000E000E000E000 E000E000E000E000E000E000E000E000600060006000700030003000180018000C000C00060002 000100008000400A2A7D9E10>40 D<800040002000100018000C000C0006000600030003000380 01800180018001C001C001C001C001C001C001C001C001C001C001C001C0018001800180038003 000300060006000C000C00180010002000400080000A2A7E9E10>I<60F0F07010101010202040 80040C7C830C>44 DI<60F0F06004047C830C>I<03C00C301818300C30 0C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E00760066006700E 300C300C18180C3007E0101D7E9B15>48 D<030007003F00C70007000700070007000700070007 000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C7C9B 15>I<00F0030C06040C0E181E301E300C700070006000E3E0E430E818F00CF00EE006E007E007 E007E007E007600760077006300E300C18180C3003E0101D7E9B15>54 D<4000007FFF807FFF00 7FFF0040020080040080040080080000100000100000200000600000400000C00000C00001C000 018000018000038000038000038000038000078000078000078000078000078000078000030000 111D7E9B15>I<03C00C301818300C700C600EE006E006E007E007E007E007E0076007700F300F 18170C2707C700060006000E300C780C78187010203030C00F80101D7E9B15>57 D<60F0F0600000000000000000000060F0F06004127C910C>I<00060000000600000006000000 0F0000000F0000000F00000017800000178000001780000023C0000023C0000023C0000041E000 0041E0000041E0000080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C 0006003E0004001E0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E0618001801980070007800E0003801C0003801C 00018038000180780000807800008070000080F0000000F0000000F0000000F0000000F0000000 F0000000F0000000F0000000700000807800008078000080380000801C0001001C0001000E0002 00070004000180080000E03000001FC000191E7E9C1E>I69 D73 D76 DI<003F800000E0E000 0380380007001C000E000E001C0007003C00078038000380780003C0780003C0700001C0F00001 E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0700001C0780003C07800 03C0380003803C0007801C0007000E000E0007001C000380380000E0E000003F80001B1E7E9C20 >79 DI82 D<07E0801C1980300580700380600180E0 0180E00080E00080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F8000 07C00003C00001C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C 17>I87 D<1FC000307000783800781C00301C00001C00001C0001FC000F1C00381C00701C00601C00E01C 40E01C40E01C40603C40304E801F870012127E9115>97 D I<07E00C301878307870306000E000E000E000E000E000E00060007004300418080C3007C00E12 7E9112>I<003F0000070000070000070000070000070000070000070000070000070000070003 E7000C1700180F00300700700700600700E00700E00700E00700E00700E00700E0070060070070 0700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C700E6006E006FFFEE000 E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C071E061E0E0C0E000E00 0E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C00381C00381C00381C00 1818001C38000C300013C0001000003000001800001FF8001FFF001FFF803003806001C0C000C0 C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C001C001C001C001C001C 001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I108 DII<03F0000E1C00180600300300700380600180E001 C0E001C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F9115 >II<03C1000C3300180B00300F00700700700700E00700E00700E00700E00700 E00700E00700600700700700300F00180F000C370007C700000700000700000700000700000700 000700000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE00F F000F880388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C 00FFE01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C1A 7F9910>IIII<7F8FF00F03800F030007020003840001C80001D80000F00000700000 780000F800009C00010E00020E000607000403801E07C0FF0FF81512809116>I I<7FFC70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E12 7F9112>I E /Fp 7 117 df<00038000000380000007C0000007C0000007C000000FE000000FE0 00001FF000001BF000001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000C0 7E0001803F0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0FF C07FFEFFC07FFE1F1C7E9B24>65 D<0FF8001C1E003E0F803E07803E07C01C07C00007C0007FC0 07E7C01F07C03C07C07C07C0F807C0F807C0F807C0780BC03E13F80FE1F815127F9117>97 DI<03FC000E0E001C1F003C1F00781F00780E00F80000F8 0000F80000F80000F80000F800007800007801803C01801C03000E0E0003F80011127E9115>I< FE3E00FE47001E8F801E8F801E8F801F07001F00001F00001F00001F00001F00001F00001F0000 1F00001F00001F0000FFF000FFF00011127F9114>114 D<1FD830786018E018E018F000FF807F E07FF01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007 000F000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F080798 03F00E1A7F9913>I E /Fq 3 122 df0 D<020002000200C218F2 783AE00F800F803AE0F278C2180200020002000D0E7E8E12>3 D<060006000600060006000600 06000600FFF0FFF006000600060006000600060006000600060006000600060006000600060006 000600060006000C1D7E9611>121 D E /Fr 51 122 df<70F8FCFC7404040404080810102040 060F7C840E>44 DI<70F8F8F87005057C840E>I<01F000071C000C0600 1803003803803803807001C07001C07001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0 F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E07001C07001C07001C07803C0380380 3803801C07000C0600071C0001F00013227EA018>48 D<008003800F80F3800380038003800380 038003800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038007C0FFFE0F217CA018>I<03F0000C1C001007002007804003C04003C08003E0 F003E0F801E0F801E0F801E02003E00003E00003C00003C0000780000700000E00001C00001800 00300000600000C0000180000100000200200400200800201800603000403FFFC07FFFC0FFFFC0 13217EA018>I<1000801E07001FFF001FFE001FF80013E0001000001000001000001000001000 0010000010F800130E001407001803801003800001C00001C00001E00001E00001E00001E07001 E0F001E0F001E0E001C08001C04003C04003802007001006000C1C0003F00013227EA018>53 D<007E0001C1000300800601C00E03C01C03C0180180380000380000780000700000700000F0F8 00F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E07001 E07001E03801C03801C01803801C03000C0600070C0001F00013227EA018>I<01F800060E0008 03001001802001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003 F80003FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C0006060 00406000C03000801803000E0E0003F00013227EA018>56 D<01F000060C000C06001807003803 80700380700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E07001E07003E03803 E01805E00C05E00619E003E1E00001C00001C00001C0000380000380300300780700780600700C 002018001030000FC00013227EA018>I<0001800000018000000180000003C0000003C0000003 C0000005E0000005E000000DF0000008F0000008F0000010F800001078000010780000203C0000 203C0000203C0000401E0000401E0000401E0000800F0000800F0000FFFF000100078001000780 030007C0020003C0020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF800F FF20237EA225>65 DI<0007E010 0038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C0000303C0000 307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8000000F800 0000F800000078000000780000107C0000103C0000103C0000101C0000201E0000200E00004007 0000400380008001C0010000E0020000381C000007E0001C247DA223>III72 DI<03FF F0001F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00700F 00F80F00F80F00F80E00F01E00401C0020380018700007C00014237EA119>II77 DI<000FE00000783C0000E00E00 03C00780078003C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C780000 3C7800003CF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800 003E7800003C7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E0078003C003 C0078000E00E0000783C00000FE0001F247DA226>II82 D<03F0200C0C601802603001E07000E0600060E00060E00060E00020E00020E00020F00000F000 007800007F00003FF0001FFE000FFF0003FF80003FC00007E00001E00000F00000F00000708000 70800070800070800070C00060C00060E000C0F000C0C80180C6070081FC0014247DA21B>I<7F FFFFF87807807860078018400780084007800840078008C007800C800780048007800480078004 800780040007800000078000000780000007800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 80000007800000078000000FC00003FFFF001E227EA123>I86 DI89 D<0FE0001838003C0C003C0E00180700000700000700 00070000FF0007C7001E07003C0700780700700700F00708F00708F00708F00F087817083C2390 0FC1E015157E9418>97 D<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E 00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C0E001C0E001E0E 001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C80600C41C0083F0017 237FA21B>I<01FE000703000C07801C0780380300780000700000F00000F00000F00000F00000 F00000F00000F000007000007800403800401C00800C010007060001F80012157E9416>I<0000 E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000E0F000 E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I<01FC000707000C 03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000F0000070000078 00203800201C00400E008007030000FC0013157F9416>I<00007001F198071E180E0E181C0700 1C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F000100000100000180000 1800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C000186000306000303800E0 0E038003FE0015217F9518>103 D<0E0000FE00001E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00700E0070 0E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070 FFE7FF18237FA21B>I<1C001E003E001E001C00000000000000000000000000000000000E00FE 001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0 0A227FA10E>I<0E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E03FC0E01F00E01C00E01800E02000E04000E08000E10000E38000EF8 000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E00E00F00E00F8FFE3FE17237FA21A >107 D<0E00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE00B237FA2 0E>I<0E1FC07F00FE60E183801E807201C00F003C00E00F003C00E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0 0E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFE27157F942A>I<0E1F 80FE60C01E80E00F00700F00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E00700E00700E00700E00700E00700E0070FFE7FF18157F941B>I<01FC000707000C018018 00C03800E0700070700070F00078F00078F00078F00078F00078F00078F000787000707800F038 00E01C01C00E038007070001FC0015157F9418>I<0E1F00FE61C00E80600F00700E00380E003C 0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E003C0E003C0E00380F00700E80E0 0E41C00E3F000E00000E00000E00000E00000E00000E00000E00000E00000E0000FFE000171F7F 941B>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E000E 000E000F00FFF010157F9413>114 D<0F8830786018C018C008C008E008F0007F803FE00FF001 F8003C801C800C800CC00CC008E018D0308FC00E157E9413>I<02000200020002000600060006 000E001E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E040E040E040E04 0E040E040708030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I IIII E /Fs 25 122 df<78FCFCFCFC780000000000000000000000000000000000000078FCFCFCFC78061F7A 9E12>58 D<00000300000000000300000000000300000000000780000000000780000000000FC0 000000000FC0000000000FC00000000017E00000000013E00000000013E00000000023F0000000 0021F00000000021F00000000040F80000000040F80000000040F800000000807C00000000807C 00000001807E00000001003E00000001003E00000002003F00000002001F00000002001F000000 04000F80000004000F80000004000F800000080007C00000080007C00000180007E000001FFFFF E000001FFFFFE00000200003F00000200001F00000200001F00000400001F80000400000F80000 400000F800008000007C00008000007C00008000007C00010000003E00010000003E0003000000 3F00030000001F00070000001F001F8000003F80FFE00003FFFCFFE00003FFFC2E327EB132>65 D<00003FE0010001FFF8030007F01E03001F800307003E000087007800004F00F000002F01E000 001F03C000000F078000000F0F800000070F000000071F000000031E000000033E000000033C00 0000017C000000017C000000017C000000017800000000F800000000F800000000F800000000F8 00000000F800000000F800000000F800000000F800000000F800000000F800000000F800000000 78000000007C000000007C000000017C000000013C000000013E000000011E000000011F000000 020F000000020F80000006078000000403C000000801E000000800F00000100078000020003E00 00C0001F8003800007F00F000001FFFC0000003FE00028337CB130>67 D76 D80 D<007F802001FFE02007C0 78600F001C601E0006E03C0003E0380001E0780000E0700000E070000060F0000060F0000060F0 000020F0000020F0000020F8000020F80000007C0000007E0000003F0000003FC000001FF80000 0FFF800007FFF80003FFFC0000FFFF00000FFF800000FFC000001FE0000007E0000003F0000001 F0000000F0000000F8000000F88000007880000078800000788000007880000078C0000078C000 0070E00000F0E00000E0F00000E0F80001C0EC000380C7000700C1F01E00807FFC00800FF0001D 337CB125>83 D<7FFFFFFFFFE07FFFFFFFFFE07E000F8007E078000F8001E070000F8000E06000 0F80006040000F80002040000F800020C0000F800030C0000F80003080000F80001080000F8000 1080000F80001080000F80001080000F80001080000F80001000000F80000000000F8000000000 0F80000000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000 0000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000000000 0F80000000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000 0000000F80000000000F80000000000F80000000000F80000000000F80000000000F8000000000 0F80000000000F80000000001FC00000000FFFFF8000000FFFFF80002C317EB030>I<00FE0000 0303C0000C00E00010007000100038003C003C003E001C003E001E003E001E0008001E0000001E 0000001E0000001E00000FFE0000FC1E0003E01E000F801E001F001E003E001E003C001E007C00 1E00F8001E04F8001E04F8001E04F8003E04F8003E0478003E047C005E043E008F080F0307F003 FC03E01E1F7D9E21>97 D<07800000FF800000FF8000000F800000078000000780000007800000 078000000780000007800000078000000780000007800000078000000780000007800000078000 0007800000078000000781FC0007860700078801C0079000E007A0007007C00078078000380780 003C0780003C0780001E0780001E0780001F0780001F0780001F0780001F0780001F0780001F07 80001F0780001F0780001F0780001E0780003E0780003C0780003C0780007807C00070072000F0 072001E00618038006060F000401F80020327EB125>I<003F8000E0600380180700040F00041E 001E1C003E3C003E7C003E7C0008780000F80000F80000F80000F80000F80000F80000F80000F8 0000F800007800007C00007C00003C00011E00011E00020F000207000403801800E060003F8018 1F7D9E1D>I<000001E000003FE000003FE0000003E0000001E0000001E0000001E0000001E000 0001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0 000001E0001F81E000F061E001C019E0078005E00F0003E00E0003E01E0001E03C0001E03C0001 E07C0001E0780001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F800 01E0F80001E0780001E0780001E03C0001E03C0001E01C0001E01E0003E00E0005E0070009E003 8011F000E061FF003F81FF20327DB125>I<003F800000E0E0000380380007003C000E001E001E 001E001C000F003C000F007C000F0078000F8078000780F8000780F8000780FFFFFF80F8000000 F8000000F8000000F8000000F8000000F8000000780000007C0000003C0000003C0000801E0000 800E0001000F0002000780020001C00C0000F03000001FC000191F7E9E1D>I<0007E0001C1000 383800707C00E07C01E07C01C03803C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C000FFFFC0FFFFC003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C00003C00003C00003C00003C00007E0007FFF007FFF0016327FB114>I<07000F801F80 1F800F800700000000000000000000000000000000000000000000000780FF80FF800F80078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 0780078007800FC0FFF8FFF80D307EAF12>105 D<0780FF80FF800F8007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800780078007800780078007800FC0 FFFCFFFC0E327EB112>108 D<0780FE001FC000FF83078060F000FF8C03C18078000F9001E200 3C0007A001E4003C0007A000F4001E0007C000F8001E0007C000F8001E00078000F0001E000780 00F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E 00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000 F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E 38>I<0780FE0000FF83078000FF8C03C0000F9001E00007A001E00007A000F00007C000F00007 C000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F000 078000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0 00078000F000078000F000078000F000078000F000078000F0000FC001F800FFFC1FFF80FFFC1F FF80211F7E9E25>I<001FC00000F0780001C01C00070007000F0007801E0003C01C0001C03C00 01E03C0001E0780000F0780000F0780000F0F80000F8F80000F8F80000F8F80000F8F80000F8F8 0000F8F80000F8F80000F8780000F07C0001F03C0001E03C0001E01E0003C01E0003C00F000780 07800F0001C01C0000F07800001FC0001D1F7E9E21>I<0781FC00FF860700FF8803C00F9001E0 07A000F007C00078078000780780003C0780003C0780003E0780001E0780001F0780001F078000 1F0780001F0780001F0780001F0780001F0780001F0780001F0780003E0780003E0780003C0780 007C0780007807C000F007A000F007A001E00798038007860F000781F800078000000780000007 80000007800000078000000780000007800000078000000780000007800000078000000FC00000 FFFC0000FFFC0000202D7E9E25>I<0783E0FF8C18FF907C0F907C07A07C07C03807C00007C000 07C000078000078000078000078000078000078000078000078000078000078000078000078000 0780000780000780000780000780000780000780000FC000FFFE00FFFE00161F7E9E19>114 D<01FC100E03301800F0300070600030E00030E00010E00010E00010F00010F800007E00003FF0 001FFF000FFFC003FFE0003FF00001F80000F880003C80003C80001CC0001CC0001CE0001CE000 18F00038F00030CC0060C301C080FE00161F7E9E1A>I<00400000400000400000400000400000 C00000C00000C00001C00001C00003C00007C0000FC0001FFFE0FFFFE003C00003C00003C00003 C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C01003C01003C01003C01003C01003C01003C01003C01001C02001E02000E0400078C0001F0014 2C7FAB19>I<078000F000FF801FF000FF801FF0000F8001F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F000078000F00007 8000F000078001F000078001F000078001F000038002F00003C004F00001C008F800007030FF80 001FC0FF80211F7E9E25>II121 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 384 339 a Fs(CCL:)21 b(A)h(P)n(ortable)g(and)f(T)-6 b(unable)21 b(Collectiv)n(e)160 431 y(Comm)n(unication)f(Library)i(for)f(Scalable)g(P)n (arallel)h(Computers)209 588 y Fr(V)l(asan)o(th)17 b(Bala)495 570 y Fq(y)571 588 y Fr(Jehosh)o(ua)g(Bruc)o(k)910 570 y Fq(\003)985 588 y Fr(Rob)q(ert)f(Cypher)1305 570 y Fq(\003)1381 588 y Fr(P)o(ablo)h (Elustondo)1733 570 y Fq(\003)348 647 y Fr(Alex)e(Ho)523 629 y Fq(\003)599 647 y Fr(Ching-Tien)i(Ho)918 629 y Fq(\003)994 647 y Fr(Shlomo)e(Kipnis)1306 629 y Fq(y)1382 647 y Fr(Marc)h(Snir)1595 629 y Fq(y)725 763 y Fr(IBM)f(Researc)o(h)g(Division)1218 745 y Fq(\003)694 821 y Fr(Almaden)f(Researc)o(h)i(Cen)o(ter)800 879 y(650)i(Harry)e(Road)759 937 y(San)h(Jose,)f(CA)g(95120)932 995 y(and)725 1054 y(IBM)f(Researc)o(h)g(Division)1218 1036 y Fq(y)657 1112 y Fr(T.J.)h(W)l(atson)h(Researc)o(h)f(Cen)o(ter)830 1170 y(P)l(.O.)f(Bo)o(x)h(218)660 1228 y(Y)l(orkto)o(wn)g(Heigh)o(ts,)f(NY)g (10598)880 1410 y Fp(Abstract)250 1504 y Fo(A)g(collectiv)o(e)f(comm)o (unication)e(library)h(for)i(parallel)e(computers)i(includes)g(frequen)o(tly) g(used)188 1554 y(op)q(erations)j(suc)o(h)h(as)f(broadcast,)i(reduce,)g (scatter,)h(gather,)e(concatenate,)h(sync)o(hronize,)g(and)188 1604 y(shift.)g(Suc)o(h)15 b(a)f(library)g(pro)o(vides)g(users)i(with)f(a)f (con)o(v)o(enien)o(t)h(programmi)o(ng)d(in)o(terface,)i(e\016cien)o(t)188 1653 y(comm)o(unicatio)o(n)e(op)q(erations,)j(and)f(the)i(adv)n(an)o(tage)e (of)g(p)q(ortabilit)o(y)m(.)19 b(A)c(library)f(of)g(this)h(nature,)188 1703 y(the)e(Collectiv)o(e)g(Comm)o(unicati)o(on)d(Library)j(\(CCL\),)g(in)o (tended)h(for)e(the)i(line)f(of)f(scalable)h(parallel)188 1753 y(computer)g(pro)q(ducts)h(b)o(y)f(IBM,)g(has)g(b)q(een)h(designed.)k(CCL)13 b(is)g(part)g(of)g(the)g(parallel)f(application)188 1803 y(programmi)o(ng)g (in)o(terface)k(of)e(the)i(recen)o(tly)g(announced)f(IBM)h(9076)e(Scalable)g (PO)o(WERparallel)188 1853 y(System)j(1)h(\(SP1\).)30 b(In)18 b(this)g(pap)q(er,)h(w)o(e)f(examine)f(sev)o(eral)h(issues)h(related)g(to)f (the)g(functional-)188 1902 y(it)o(y)m(,)13 b(correctness,)k(and)e(p)q (erformance)f(of)g(a)h(p)q(ortable)g(collectiv)o(e)f(comm)o(unication)d (library)j(while)188 1952 y(fo)q(cusing)i(on)f(three)j(no)o(v)o(el)d(asp)q (ects)j(in)d(the)i(design)f(and)g(implemen)o(tatio)o(n)d(of)j(CCL:)f(\(i\))h (the)g(in-)188 2002 y(tro)q(duction)g(of)e(pro)q(cess)k(groups,)d(\(ii\))g (the)h(de\014nition)f(of)g(seman)o(tics)g(that)h(ensures)h(correctness,)188 2052 y(and)e(\(iii\))g(the)h(design)g(of)f(new)h(and)f(tunable)h(algorithms)d (based)j(on)f(a)h(realistic)f(p)q(oin)o(t-to-p)q(oin)o(t)188 2102 y(comm)o(unicatio)o(n)c(mo)q(del.)960 2727 y Fn(1)p eop %%Page: 2 2 bop 74 157 a Fm(1)69 b(In)n(tro)r(duction)74 290 y Fn(The)13 b(need)h(for)f(collectiv)o(e)i(comm)o(unication)f(arises)f(frequen)o(tly)g (in)h(parallel)h(computation.)k(Collectiv)o(e)74 346 y(comm)o(unication)g(op) q(erations)f(simplify)i(the)e(programming)f(of)h(applications)h(for)f (parallel)h(comput-)74 403 y(ers,)c(facilitate)h(the)f(implemen)o(tation)h (of)e(e\016cien)o(t)i(comm)o(unication)g(sc)o(hemes)f(on)g(v)m(arious)g(mac)o (hines,)74 459 y(promote)d(the)g(p)q(ortabilit)o(y)i(of)e(applications)i (across)e(di\013eren)o(t)g(arc)o(hitectures,)h(and)g(re\015ect)f(conceptual) 74 516 y(grouping)k(of)f(pro)q(cesses.)20 b(In)c(particular,)g(collectiv)o(e) h(comm)o(unication)f(is)g(extensiv)o(ely)h(used)f(in)g(man)o(y)74 572 y(scien)o(ti\014c)f(applications)h(for)c(whic)o(h)j(the)f(in)o(terlea)o (ving)g(of)f(stages)g(of)g(lo)q(cal)i(computations)e(with)h(stages)74 629 y(of)h(global)h(comm)o(unication)g(is)f(p)q(ossible)i(\(see)f([20)o(]\).) 74 716 y(This)21 b(pap)q(er)g(discusses)g(issues)g(related)f(to)g(the)g (design)h(and)g(implemen)o(tation)g(of)f(a)g(p)q(ortable)g(and)74 773 y(tunable)15 b(Collectiv)o(e)g(Comm)o(unication)f(Library)g(\(CCL\).)f (This)h(library)h(is)f(in)o(tended)h(for)e(distributed-)74 829 y(memory)h(parallel)j(computers)e(where)g(explicit)i(comm)o(unication)f (among)e(pro)q(cesses)h(is)g(ac)o(hiev)o(ed)h(via)74 885 y(message-passing.) 32 b(In)20 b(addition)g(to)f(p)q(oin)o(t)h(to)e(p)q(oin)o(t)i(comm)o (unication)g(op)q(erations,)g(suc)o(h)f(as)g Fl(send)74 942 y Fn(and)i Fl(receive)p Fn(,)f(users)g(can)h(program)e(using)i(CCL)g (routines)g(for)e(collectiv)o(e)k(op)q(erations.)35 b(The)21 b(set)74 998 y(of)e(collectiv)o(e)h(comm)o(unication)g(op)q(erations)f(in)h (CCL)e(consists)i(of)e(the)h(follo)o(wing)h(routines:)27 b Fl(bcast)74 1055 y Fn(|)21 b(one-to-all)g(broadcast,)f Fl(reduce)f Fn(|)i(all-to-one)g(reduction,)h Fl(combine)d Fn(|)h(all-to-all)i(reduction,) 74 1111 y Fl(scatter)13 b Fn(|)g(one-to-all)i(p)q(ersonalized)g(comm)o (unication,)f Fl(gather)f Fn(|)h(all-to-one)g(p)q(ersonalized)i(com-)74 1168 y(m)o(unication,)c Fl(concat)d Fn(|)i(all-to-all)g(broadcast,)g Fl(index)e Fn(|)i(all-to-all)g(p)q(ersonalized)h(comm)o(unication\),)74 1224 y Fl(prefix)j Fn(|)i(scanned)g(reduction,)g Fl(shift)e Fn(|)i(one-to-one)f(cyclic)i(p)q(erm)o(utation,)e(and)g Fl(sync)p Fn(|barrier)74 1281 y(sync)o(hronization.)29 b(A)18 b(complete)h(listing)h (with)e(brief)h(descriptions)g(of)f(the)g(functionalit)o(y)h(of)e(all)i(the) 74 1337 y(CCL)c(routines)h(is)g(pro)o(vided)g(in)g(the)f(App)q(endix.)74 1425 y(The)j(library)g(w)o(as)f(designed)i(for)e(the)h(new)f(IBM)h(line)h(of) f(scalable)g(parallel)i(computers.)26 b(\(The)18 b(\014rst)74 1481 y(computer)c(of)g(this)g(line)i(of)e(pro)q(ducts,)g(the)g(IBM)h(9076)d (Scalable)k(PO)o(WERparallel)g(System)e(1)g(\(SP1\),)74 1538 y(has)j(b)q(een)h(announced)f(recen)o(tly)l(.\))25 b(Most)16 b(of)g(CCL)h(is)h(implemen)o(ted)g(as)e(part)g(of)h(the)g(parallel)h(appli-) 74 1594 y(cation)e(programming)e(in)o(terface)i(of)e(SP1.)74 1682 y(A)f(ma)s(jor)f(goal)h(in)h(the)f(design)i(of)d(CCL)i(w)o(as)e(the)h (creation)h(of)e(a)h(truly)h(p)q(ortable)f(library)h(that)f(can)g(run)74 1738 y(correctly)h(and)f(e\016cien)o(tly)i(on)e(a)h(wide)g(range)f(of)g (parallel)i(and)e(distributed)i(computer)f(arc)o(hitectures,)74 1795 y(regardless)i(of)f(the)g(top)q(ology)g(of)g(the)h(in)o(terconnection)h (net)o(w)o(ork)d(and)h(the)h(sp)q(eeds)g(of)f(the)h(pro)q(cessors)74 1851 y(and)e(the)g(comm)o(unication)h(net)o(w)o(ork.)k(Also,)14 b(CCL)g(can)g(b)q(e)h(easily)g(implemen)o(ted)h(on)e(top)f(of)h(message-)74 1907 y(passing)k(libraries)g(suc)o(h)f(as)g(IBM)g(EUI)g([21],)f(IBM)h(VIPER)h (Op)q(erating)g(System)f([18)o(,)f(23],)g(P)o(arasoft)74 1964 y(Express)f([30)o(],)g(and)g(ONRL)i(PVM)e([6)o(].)74 2052 y(F)l(or)d (simplicit)o(y)j(and)e(clarit)o(y)g(of)f(the)g(discussion)j(in)e(this)g(pap)q (er,)g(w)o(e)g(only)g(address)f(scenarios)h(in)h(whic)o(h)74 2108 y(dynamic)23 b(pro)q(cess)f(creation)g(or)g(deletion)h(in)g(run-time)f (is)h(not)e(p)q(ermitted.)41 b(The)22 b(CCL)g(routines)74 2164 y(can)f(either)h(op)q(erate)e(o)o(v)o(er)g(the)h(en)o(tire)g(set)g(of)f(pro)q (cesses)i(that)e(are)g(created)h(at)f(the)h(b)q(eginning)i(of)74 2221 y(an)d(application)i(or)d(o)o(v)o(er)g(user-sp)q(eci\014ed)j(groups)e (of)f(pro)q(cesses.)35 b(F)l(or)19 b(example,)i(a)f(user)g(can)g(view)74 2277 y(sev)o(eral)12 b(pro)q(cesses)f(as)g(forming)g(a)g(t)o(w)o (o-dimensional)h(arra)o(y)e(and)i(p)q(erforming)g(indep)q(enden)o(t)h (broadcast)74 2334 y(op)q(erations)i(in)g(di\013eren)o(t)g(columns)h(of)e (this)h(arra)o(y)l(.)k(Moreo)o(v)o(er,)13 b(the)i(CCL)f(routines)h(can)g (also)g(op)q(erate)74 2390 y(on)g(groups)f(of)h(pro)q(cesses)g(that)f(are)h (determined)h(dynamically)l(,)g(namely)l(,)g(in)f(run-time.)21 b(T)l(o)15 b(facilitate)74 2447 y(the)d(use)g(of)f(suc)o(h)h(groups)f(of)g (pro)q(cesses,)h(CCL)g(includes)h(routines)f(that)f(enable)i(the)e (de\014nition)j(and)d(the)74 2503 y(handling)18 b(of)e(dynamic)h(Pro)q(cess)f (Groups.)22 b(Section)17 b(2)f(examines)h(issues)g(related)g(to)e(pro)q(cess) i(groups)74 2560 y(in)f(CCL.)960 2727 y(2)p eop %%Page: 3 3 bop 74 157 a Fn(CCL)19 b(is)h(a)f(soft)o(w)o(are)e(la)o(y)o(er)i(that)g(can)g (sit)h(on)f(top)g(of)f(p)q(oin)o(t-to-p)q(oin)o(t)i(comm)o(unication.)33 b(A)19 b(crucial)74 214 y(step)g(in)h(the)f(design)h(of)f(CCL)g(w)o(as)f(to)g (de\014ne)i(a)f(simple)i(and)e(realistic)h(mo)q(del)g(for)e(the)i(underlying) 74 270 y(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication.)28 b(This)19 b(allo)o(ws)f(CCL)f(to)h(b)q(e)g(b)q(oth)g(p)q(ortable)g(and)g (e\016cien)o(t)h(o)o(v)o(er)e(a)74 327 y(wide)c(range)f(of)g(parallel)i(mac)o (hines.)20 b(This)13 b(mo)q(del)g(is)g(discussed)h(in)f(Section)g(3.)19 b(In)12 b(particular,)i(Section)74 383 y(3)f(examines)h(sev)o(eral)g (fundamen)o(tal)f(issues)i(related)e(to)g(the)g(seman)o(tics)h(of)f (collectiv)o(e)i(comm)o(unication)74 439 y(op)q(erations)g(o)o(v)o(er)g(Pro)q (cess)g(Groups.)74 527 y(F)l(or)g(CCL)g(to)f(b)q(e)i(scalable)g(and)f (e\016cien)o(t)h(on)f(a)g(v)m(ariet)o(y)g(of)g(parallel)h(mac)o(hines,)g(it)f (is)h(imp)q(ortan)o(t)e(that)74 583 y(the)19 b(algorithms)g(designed)i(for)d (collectiv)o(e)j(comm)o(unication)e(op)q(erations)h(are)e(tunable)i(and)f (exploit)74 640 y(the)h(c)o(haracteristics)h(of)e(the)h(particular)h (parallel)h(computer)e(in)h(a)e(w)o(a)o(y)g(that)h(is)g(hidden)i(from)d(the) 74 696 y(user.)h(In)c(Section)g(4,)e(w)o(e)h(address)g(issues)h(related)g(to) e(the)h(design)i(of)d(tunable)i(algorithms)f(for)g(collec-)74 753 y(tiv)o(e)j(comm)o(unication)g(op)q(erations)g(in)g(CCL.)f(Finally)l(,)j (in)e(Section)g(5)g(w)o(e)f(presen)o(t)g(some)g(concluding)74 809 y(remarks.)74 982 y Fm(2)69 b(Pro)r(cess)23 b(Groups)h(in)e(CCL)74 1115 y Fn(Pro)q(cess)12 b(Groups)f(is)h(a)g(mec)o(hanism)g(for)f(grouping)h (pro)q(cesses)g(in)o(to)g(logical)h(sets)e(and)h(for)f(manipulating)74 1171 y(and)19 b(comm)o(unicating)h(among)e(pro)q(cesses)i(in)f(suc)o(h)h (sets.)31 b(This)19 b(section)h(describ)q(es)g(the)f(concept)h(of)74 1228 y(Pro)q(cess)15 b(Groups)f(in)i(CCL,)e(whic)o(h)i(is)f(based)g(on)g(a)f (similar)i(concept)g(that)e(w)o(as)g(presen)o(ted)h(in)h([2)o(].)j(\(In)74 1284 y(the)c(IBM)h(EUI)f(do)q(cumen)o(tation)h([21)o(],)f(Pro)q(cess)g (Groups)f(are)h(called)i(T)l(ask)e(Groups.\))74 1435 y Fk(2.1)56 b(De\014ning)18 b(Pro)r(cess)g(Groups)74 1552 y Fn(A)13 b(Pro)q(cess)g(Group) g(is)h(an)f(ordered)g(set)g(of)g(pro)q(cesses)g(that)g(has)g(a)g(system-wide) g(unique)i(name.)k(It)13 b(can)74 1609 y(b)q(e)j(op)q(erated)e(up)q(on)i(as)e (a)g(single)j(ob)s(ject.)i(Eac)o(h)14 b(Pro)q(cess)h(Group)f(is)h(iden)o (ti\014ed)i(b)o(y)e(a)f(unique)j(Pro)q(cess)74 1665 y(Group)f(Iden)o (ti\014er)i(\(PGID\))d(in)i(an)f(application)i(program.)k(All)17 b(the)g(pro)q(cesses)f(that)g(are)g(created)g(at)74 1721 y(the)d (initialization)i(of)d(an)g(application)i(b)q(elong)g(to)d(the)i (prede\014ned)h(Pro)q(cess)e(Group)g Fl(ALL)p Fn(.)g(In)h(addition,)74 1778 y(eac)o(h)i(pro)q(cess)h(has)f(a)f(unique)j(Pro)q(cess)e(Iden)o (ti\014er)h(\(PID\))f(named)g Fl(mypid)g Fn(whic)o(h)h(could)g(b)q(e)f(iden)o (ti\014ed)74 1834 y(with)h(the)f(singleton)h(set)f(consisting)h(only)g(of)f (that)f(pro)q(cess.)74 1922 y(There)e(are)f(t)o(w)o(o)f(mec)o(hanisms)i(in)h (CCL)e(for)g(de\014ning)i(new)f(Pro)q(cess)f(Groups.)19 b(One)12 b(mec)o(hanism)g(can)g(b)q(e)74 1978 y(used)g(to)f(de\014ne)i(a)f(new)g(Pro)q (cess)f(Group)h(b)o(y)f(pro)o(viding)i(an)e(explicit)j(list)f(of)e(pro)q (cess)h(iden)o(ti\014ers)h(\(PIDs\))74 2035 y(that)h(comprise)h(the)g(new)g (Pro)q(cess)f(Group.)20 b(This)15 b(mec)o(hanism)g(requires)g(eac)o(h)g(mem)o (b)q(er)g(pro)q(cess)g(of)f(a)74 2091 y(to-b)q(e-formed)j(new)f(Pro)q(cess)g (Group)g(to)g(kno)o(w)g(the)g(PIDs)h(of)e(all)j(the)e(pro)q(cesses)h(that)e (are)h(mem)o(b)q(ers)74 2148 y(of)f(this)g(group.)k(Using)d(this)f(mec)o (hanism,)h(all)f(the)g(pro)q(cesses)h(that)e(comprise)h(a)g(new)g(Pro)q(cess) g(Group)74 2204 y(m)o(ust)g(call)h(the)f Fl(group\(\))g Fn(routine:)146 2290 y Fl(pgid)23 b(=)h(group\(size,)e(list,)h(label\);)74 2377 y Fn(The)18 b(usage)g(of)g Fl(label)f Fn(will)i(b)q(ecome)g(clear)f (later.)28 b(The)18 b(call)i(to)d Fl(group\(\))g Fn(m)o(ust)g(b)q(e)i(made)f (b)o(y)f(eac)o(h)74 2433 y(mem)o(b)q(er)d(pro)q(cess)f(of)g(the)g(to-b)q (e-formed)g(Pro)q(cess)g(Group,)g(and)h(eac)o(h)f(mem)o(b)q(er)g(pro)q(cess)h (m)o(ust)e(supply)74 2490 y(the)19 b(same)g(list)g(of)g(PIDs)g(and)g(in)h (the)f(same)f(order.)31 b(Pro)q(cesses)19 b(that)f(b)q(elong)i(to)e(the)h (new)h(Pro)q(cess)74 2546 y(Group)e(obtain)g(a)f(unique)i(PGID)f(for)f(this)h (group.)27 b(Pro)q(cesses)18 b(that)f(do)h(not)f(wish)i(to)e(b)q(e)h(part)f (of)h(a)74 2602 y(new)e(Pro)q(cess)f(Group)g(do)g(not)g(call)h(the)f Fl(group\(\))f Fn(routine.)960 2727 y(3)p eop %%Page: 4 4 bop 74 157 a Fn(The)17 b(other)f(mec)o(hanism)h(for)f(de\014ning)h(new)g(Pro) q(cess)f(Groups)g(in)i(CCL)e(is)h(b)o(y)f(partitioning)i(existing)74 214 y(Pro)q(cess)d(Groups)f(according)h(to)f(some)g(criteria.)20 b(This)15 b(mec)o(hanism)g(do)q(es)g(not)f(require)i(the)e(mem)o(b)q(ers)74 270 y(of)i(a)f(to-b)q(e-formed)i(Pro)q(cess)e(Group)h(to)f(kno)o(w)h(the)g (PIDs)g(of)g(all)h(the)f(pro)q(cesses)g(that)f(will)j(comprise)74 327 y(this)g(group.)28 b(When)18 b(partitioning)g(an)g(existing)h(Pro)q(cess) f(Group)f Fl(G)p Fn(,)g(sev)o(eral)h(new)g(Pro)q(cess)g(Groups)74 383 y(are)13 b(created)h(as)f(follo)o(ws.)20 b(Ev)o(ery)13 b(pro)q(cess)h(in)g(the)g(group)f Fl(G)g Fn(supplies)j(a)d(lo)q(cal)i(in)o (teger)f(v)m(alue)h(to)e Fl(myval)p Fn(.)74 439 y(All)h(the)e(pro)q(cesses)h (that)e(supply)j(the)e(same)g(v)m(alue)i(to)d Fl(myval)h Fn(are)g(made)g(mem) o(b)q(ers)g(of)g(the)h(same)f(newly)74 496 y(partitioned)j(Pro)q(cess)g (Group.)k(The)c(partitioning)g(of)f(an)h(existing)g(Pro)q(cess)g(Group)f(is)h (p)q(erformed)g(b)o(y)74 552 y(the)g Fl(partition\(\))f Fn(routine:)146 640 y Fl(pgid)23 b(=)h(partition\(G,)e(myval,)h(key\);)74 728 y Fn(The)c(call)g(to)e Fl(partition\(\))g Fn(m)o(ust)h(b)q(e)h(made)f(b)o(y)g (ev)o(ery)g(mem)o(b)q(er)g(of)g(the)h(existing)g(Pro)q(cess)f(Group)74 784 y Fl(G)p Fn(.)e(No)g(mem)o(b)q(er)h(of)f(the)g(existing)i(group)e Fl(G)g Fn(returns)g(from)g(this)h(call)h(un)o(til)f(all)g(pro)q(cesses)g(in)h (group)e Fl(G)74 841 y Fn(ha)o(v)o(e)i(made)h(the)g(call.)32 b(\(In)19 b(that)f(sense,)h(the)g(call)h(to)e Fl(partition\(\))f Fn(imp)q(oses)j(an)e(implicit)j(barrier)74 897 y(sync)o(hronization)h(on)e (the)h(mem)o(b)q(ers)g(of)f Fl(G)p Fn(\).)g(This)h(call)h(results)f(in)h (partitioning)f(the)g(mem)o(b)q(ers)g(of)74 954 y(group)16 b Fl(G)f Fn(in)o(to)h(as)f(man)o(y)g(new)h(groups)f(as)g(the)h(n)o(um)o(b)q (er)g(of)f(distinct)i(in)o(teger)f(v)m(alues)h(supplied)h(b)o(y)d(the)74 1010 y(mem)o(b)q(er)h(pro)q(cesses.)74 1098 y(A)e(pro)q(cess)g(can)g (participate)h(in)f(sev)o(eral)g Fl(group\(\))f Fn(or)h Fl(partition\(\))e Fn(calls)j(and,)f(th)o(us,)f(b)q(e)i(a)e(mem)o(b)q(er)74 1154 y(of)h(sev)o(eral)h(di\013eren)o(t)f(Pro)q(cess)g(Groups.)19 b(Eac)o(h)c(call)g(to)f Fl(group\(\))f Fn(or)h Fl(partition\(\))f Fn(ma)o(y)g(corresp)q(ond)74 1211 y(to)19 b(creating)h(Pro)q(cess)f(Groups)h (based)f(on)h(a)f(new)h(criterion.)34 b(Eac)o(h)19 b(suc)o(h)h(call,)i (therefore,)e(creates)74 1267 y(unique)d(PGIDs)e(for)f(the)i(resultan)o(t)f (groups.)74 1355 y(All)20 b(the)e(pro)q(cesses)g(that)g(b)q(elong)h(to)e(a)h (particular)h(Pro)q(cess)f(Group)g Fl(G)p Fn(,)f(regardless)h(of)g(whether)g (the)74 1411 y(group)e(w)o(as)e(created)i(b)o(y)g(calling)h Fl(group\(\))e Fn(or)g Fl(partition\(\))p Fn(,)f(are)i(rank)o(ed)f(from)g(0)h (to)f Fj(n)10 b Fi(\000)h Fn(1,)k(where)74 1467 y Fj(n)21 b Fn(is)h(the)f(size)h(of)f(the)g(group.)37 b(The)22 b(ranking)f(of)g(the)g (pro)q(cesses)g(in)h(a)f(Pro)q(cess)g(Group)g(that)f(w)o(as)74 1524 y(created)14 b(b)o(y)g(a)f(call)i(to)e Fl(group\(\))g Fn(is)h(the)g(same)f(as)h(the)g(order)f(of)h(the)f(PIDs)h(in)h(the)f(list)g (supplied)i(to)d(the)74 1580 y Fl(group\(\))i Fn(call.)24 b(The)16 b(ranking)g(of)g(the)g(pro)q(cesses)g(in)h(a)f(Pro)q(cess)g(Group)f(that)h(w) o(as)f(created)h(b)o(y)g(a)f(call)74 1637 y(to)j Fl(partition\(\))g Fn(is)h(determined)h(b)o(y)f(the)g(additional)i(parameter,)d Fl(key)p Fn(,)h(that)f(is)i(supplied)h(to)d(the)74 1693 y Fl(partition\(\))f Fn(call)i(as)f(an)g(input)h(argumen)o(t.)27 b(Ties)19 b(in)g(the)f(ranking)g (as)g(a)g(consequence)h(of)f(calling)74 1750 y Fl(partition\(\))c Fn(are)h(to)f(b)q(e)i(brok)o(en)f(b)o(y)g(pro)q(cesses')h(ranking)f(in)h(the) f(paren)o(t)g(group.)74 1837 y(F)l(or)i(example,)i(supp)q(ose)f(that)f(there) h(are)f(7)g(pro)q(cesses,)h(with)g(PID's)f(from)g(0)g(to)g(6)h(\(this)f(is)h (the)g Fl(ALL)74 1894 y Fn(group\).)27 b(F)l(or)17 b(con)o(v)o(enience,)i(w)o (e)f(refer)f(to)g(a)g(pro)q(cess)h(with)g(an)g(ev)o(en)g(\(resp.)27 b(o)q(dd\))18 b(PID)f(as)h(an)f Fh(even)74 1950 y Fn(\(resp.)34 b Fh(o)n(dd)p Fn(\))19 b(pro)q(cess.)34 b(Supp)q(ose)21 b(that)e(w)o(e)h(w)o (an)o(t)e(to)h(create)h(the)g(follo)o(wing)g(t)o(w)o(o)f(t)o(yp)q(es)h(of)f (groups)74 2007 y(while)i(main)o(taining)g(the)f(original)g(ordering:)29 b Fj(G)20 b Fn(in)g(eac)o(h)g(ev)o(en)g(pro)q(cess)g(is)g(de\014ned)h(as)e (the)h(group)74 2063 y(that)c(consists)h(of)f(all)h(ev)o(en)g(pro)q(cesses)g (and)g Fj(G)f Fn(in)h(eac)o(h)g(o)q(dd)g(pro)q(cess)f(is)h(de\014ned)h(as)e (the)h(group)f(that)74 2120 y(consists)g(of)f(all)h(o)q(dd)g(pro)q(cesses.)22 b(There)15 b(are)h(t)o(w)o(o)e(w)o(a)o(ys)g(to)h(do)h(it.)21 b(The)16 b(\014rst)f(\(the)g(static\))g(is)h(to)f(ha)o(v)o(e)74 2176 y(the)g(follo)o(wing)h(co)q(de:)146 2264 y Fl(EXAMPLE)23 b(1:)146 2320 y(if)g(\(pid_is_even\(\)\))f({)217 2377 y(label)i(=)f(EVEN;)217 2433 y(G)h(=)g(group\(4,)f([0,2,4,6],)f(label\);)146 2490 y(})146 2546 y(else)h({)217 2602 y(label)h(=)f(ODD;)960 2727 y Fn(4)p eop %%Page: 5 5 bop 217 157 a Fl(G)24 b(=)g(group\(3,)f([1,3,5],)f(label\);)146 214 y(})74 305 y Fn(The)16 b(second)f(approac)o(h)g(\(the)g(dynamic\))h(is)g (to)e(use)i Fl(partition)p Fn(.)i(The)e(co)q(de)f(in)i(this)e(case)g(is:)146 397 y Fl(G)23 b(=)h(partition\(ALL,)e(is_mypid_odd,)g(key\);)74 489 y Fn(Consider)d(another)e(example)i(where)f(all)h(pro)q(cesses)f(are)g (partitioned)g(in)o(to)g(t)o(w)o(o)f(groups)g(dep)q(ending)74 545 y(on)e(some)g(lo)q(cally)i(computed)f(v)m(alue)g(\(temp)q(erature\).)j (The)d(co)q(de)f(lo)q(oks)h(lik)o(e:)146 637 y Fl(EXAMPLE)23 b(2:)146 693 y(if)g(\(temperature)g(<)g(BOILING\))217 750 y(myval)h(=)f (COLD;)146 806 y(else)217 863 y(myval)h(=)f(HOT;)146 919 y(key)g(=)h (temperature;)146 976 y(G)f(=)h(partition\(ALL,)e(myval,)h(key\);)74 1067 y Fn(In)16 b(this)f(example,)h(eac)o(h)f(of)f(the)h(t)o(w)o(o)f(newly)i (formed)f(groups)f(is)i(sorted)e(according)i(to)e(the)h(ascending)74 1124 y(order)g(of)g Fl(key)p Fn(,)f(whic)o(h)i(w)o(as)f(assigned)g(the)h(in)o (teger)f(v)m(alue)i(of)d(temp)q(erature.)74 1211 y(CCL)j(also)g(pro)o(vides)g (utilit)o(y)i(routines)e(for)f(pro)q(cesses)i(to)e(determine)i(the)f(size)h (of)e(a)h(Pro)q(cess)g(Group)74 1268 y(\()p Fl(getsize\(G\))p Fn(\),)f(to)i(obtain)h(the)g(list)h(of)e(PIDs)g(in)i(a)e(giv)o(en)h(Pro)q (cess)g(Group)f(\()p Fl(getmembers\(G\))p Fn(\),)e(to)74 1324 y(query)f(the)f(rank)g(of)g(a)g(pro)q(cess)h(in)g(a)f(Pro)q(cess)h(Group)f (\()p Fl(getrank\(G,)22 b(pid\))p Fn(\),)13 b(and)i(to)e(query)i(the)f(PID)74 1381 y(of)j(a)g(pro)q(cess)h(with)f(a)h(giv)o(en)f(rank)g(in)i(a)e(Pro)q (cess)g(Group)g(\()p Fl(getpid\(G,)22 b(rank\))p Fn(\).)k(Finally)l(,)19 b(the)e(user)74 1437 y(can)d(call)i Fl(getlabel\(G\))c Fn(to)i(obtain)g(the)g Fl(label)g Fn(supplied)i(to)e Fl(group\(\))f Fn(and)h(the)g Fl(my)p 1550 1437 15 2 v 17 w(val)g Fn(supplied)i(to)74 1494 y Fl(partition\(\))p Fn(.)74 1646 y Fk(2.2)56 b(Using)18 b(Pro)r(cess)g (Groups)74 1763 y Fn(Once)f(established,)g(Pro)q(cess)f(Groups)f(allo)o(w)h (en)o(tire)g(collections)i(of)d(pro)q(cesses)h(to)f(b)q(e)i(iden)o(ti\014ed)g (and)74 1819 y(manipulated)k(in)f(a)g(single)g(call.)34 b(Note)20 b(that)e(when)i(di\013eren)o(t)g(actions)g(are)f(required)h(for)f(di\013eren) o(t)74 1876 y(disjoin)o(t)f(groups,)e(the)h(user)g(can)g(use)g Fl(label)f Fn(to)g(distinguish)j(groups.)24 b(F)l(or)16 b(example,)i(the)f (follo)o(wing)74 1932 y(co)q(de)f(ma)o(y)e(follo)o(w)i(the)f(co)q(de)h(of)f (Example)g(1)g(ab)q(o)o(v)o(e.)146 2024 y Fl(if)23 b(\(getlabel\(G\))g(==)g (EVEN\))g({)217 2080 y(code-segment-1;)146 2137 y(})146 2193 y(else)47 b({)217 2250 y(code-segment-2;)146 2306 y(})74 2398 y Fn(As)15 b(another)g(example,)h(the)f(follo)o(wing)h(co)q(de)g(ma)o(y)e (follo)o(w)i(the)f(co)q(de)h(of)e(Example)i(2)f(ab)q(o)o(v)o(e.)146 2490 y Fl(if)23 b(\(getlabel\(G\))g(==)g(COLD\))g({)217 2546 y(code-segment-1;)146 2602 y(})960 2727 y Fn(5)p eop %%Page: 6 6 bop 146 157 a Fl(else)23 b({)217 214 y(code-segment-2;)146 270 y(})74 365 y Fn(A)e(group-wide)h(transaction)f(of)f(particular)i(in)o (terest)f(is)h(collectiv)o(e)g(comm)o(unication)g(o)o(v)o(er)e(all)i(the)74 421 y(mem)o(b)q(ers)15 b(of)f(a)h(Pro)q(cess)f(Group.)19 b(F)l(or)c(example,) g(a)f(group-wide)i(reduction)f(op)q(eration,)g(whic)o(h)g(tak)o(es)74 478 y(the)g(PGID)g(of)g(a)g(Pro)q(cess)g(Group)g(as)g(a)g(parameter,)f(can)h (b)q(e)h(written)f(as:)146 573 y Fl(combine\(G,)22 b(rfunc\(\),)h(data,)g (result\);)74 668 y Fn(Suc)o(h)15 b(a)e(routine)i(could)g(implemen)o(t)g(a)e (reduction,)i(using)g(a)e(supplied)k(asso)q(ciativ)o(e)d(function)h Fl(rfunc\(\))74 724 y Fn(to)21 b(com)o(bine)i(data)e(from)g(the)h(mem)o(b)q (ers)g(of)g(Pro)q(cess)f(Group)h Fl(G)p Fn(,)f(and)h(return)g(the)g(\014nal)h (result)f(of)74 781 y(the)17 b(reduction)h(to)f(eac)o(h)g(mem)o(b)q(er)g(in)h (the)f(bu\013er)g Fl(result)p Fn(.)24 b(All)19 b(pro)q(cesses)e(in)h(the)f (group)g(mak)o(e)f(the)74 837 y(call)k(with)f(the)g(same)f(actual)h (parameters)f Fl(G)g Fn(and)h Fl(rfunc\(\))p Fn(.)29 b(Similarly)l(,)22 b(a)c(group-wide)h(broadcast)74 894 y(op)q(eration)d(can)f(b)q(e)h(written)f (as:)146 989 y Fl(bcast\(G,)23 b(orig,)g(data\);)74 1084 y Fn(This)14 b(routine)f(w)o(ould)g(implemen)o(t)h(a)f(broadcast)f(comm)o (unication)h(from)f(a)h(particular)g(mem)o(b)q(er)g(of)f(the)74 1140 y(Pro)q(cess)j(Group)f Fl(G)p Fn(,)f(whose)i(PID)f(is)h Fl(orig)p Fn(,)f(to)f(all)j(of)e(the)g(other)g(mem)o(b)q(ers)h(of)f Fl(G)p Fn(.)g(All)h(pro)q(cesses)g(in)g(the)74 1196 y(group)g(mak)o(e)g(the)g (call)h(with)g(the)f(same)g(actual)g(parameters)g Fl(G)g Fn(and)g Fl(orig)p Fn(.)74 1284 y(A)j(participating)h(pro)q(cess)f(can)h(pro)q(ceed)f (past)g(the)g(call)h(to)e(the)h(collectiv)o(e)i(comm)o(unication)f(imme-)74 1341 y(diately)i(after)e(it)h(has)f(\014nished)i(its)f(participation)h(in)g (it)e(\(ev)o(en)h(though)g(the)f(comm)o(unication)i(as)e(a)74 1397 y(whole)e(ma)o(y)e(still)i(b)q(e)g(in)g(progress\).)k(F)l(or)16 b(example)g(in)h(a)f(grid-t)o(yp)q(e)h(problem)f(domain,)g(consider)h(the)74 1453 y(sequence)g(of)d(calls:)146 1548 y Fl(row_pgid)23 b(=)g (partition\(grid_pgid,)e(row_id,)i(key\))146 1605 y(col_pgid)g(=)g (partition\(grid_pgid,)e(col_id,)i(key\))146 1661 y(bcast\(row_pgid,)f (orig1,)h(data1\);)146 1718 y(bcast\(col_pgid,)f(orig2,)h(data2\);)74 1813 y Fn(where)12 b Fl(row)p 277 1813 15 2 v 17 w(pgid)e Fn(and)i Fl(col)p 557 1813 V 17 w(pgid)e Fn(represen)o(t)i(t)o(w)o(o)e(Pro)q(cess)h (Groups)g(for)g(eac)o(h)g(of)g(the)h(pro)q(cesses)f(in)o(v)o(ok-)74 1869 y(ing)17 b(the)f(t)o(w)o(o)e Fl(partition\(\))h Fn(calls.)23 b(Since)17 b Fl(row)p 910 1869 V 17 w(pgid)e Fn(and)h Fl(col)p 1199 1869 V 17 w(pgid)f Fn(are)h(not)g(disjoin)o(t,)g(the)g(second)74 1926 y(broadcast)c(op)q(eration)h(ma)o(y)f(b)q(e)i(partially)g(completed)g (in)f(parallel)i(with)e(the)g(\014rst)f(one.)20 b(The)13 b(program)74 1982 y(execution)g(order)f(guaran)o(tees)f(that)h(the)g(pro)q(cesses)g(in)h Fl(row)p 1102 1982 V 17 w(pgid)f Fi(\\)g Fl(col)p 1341 1982 V 17 w(pgid)f Fn(will)j(reac)o(h)e(the)g(second)74 2038 y(broadcast)18 b(only)i(after)e(they)h(\014nish)h(their)f(participation)h(in)g(the)f (\014rst)g(broadcast.)30 b(CCL)19 b(prev)o(en)o(ts)74 2095 y(p)q(ossible)e(mixing)f(of)f(messages)f(b)q(et)o(w)o(een)i(the)f(t)o(w)o(o)f (broadcast)g(op)q(erations)h(b)o(y)g(using)h(seman)o(tics)f(and)74 2151 y(prop)q(erties)22 b(describ)q(ed)h(in)f(Section)g(3)f(so)g(that)g(the)g (underlying)i(comm)o(unication)f(subsystem)f(can)74 2208 y(distinguish)c(b)q (et)o(w)o(een)f(the)f(t)o(w)o(o)f(messages.)74 2361 y Fk(2.3)56 b(Unique)18 b(Pro)r(cess)g(Groups)h(Iden)n(ti\014ers)74 2478 y Fn(The)h(actual)f(v)m(alue)i(of)e(a)g(PGID)g(is)h(determined)g(b)o(y)g (CCL,)f(and)g(user)h(co)q(de)g(that)e(dep)q(ends)j(on)e(the)74 2534 y(actual)e(v)m(alue)g(of)f(a)g(PGID)g(\(suc)o(h)h(as)f(branc)o(hing)h (based)g(on)f(the)g(v)m(alue)i(of)e(a)g(PGID\))g(is)g(not)g(allo)o(w)o(ed.)74 2591 y(That)h(is,)i(suc)o(h)f(co)q(de)g(ma)o(y)f(op)q(erate)g(correctly)h (with)g(one)g(implemen)o(tation)h(of)e(CCL)h(and)g(not)f(with)960 2727 y(6)p eop %%Page: 7 7 bop 74 157 a Fn(another.)19 b(On)c(the)g(other)f(hand,)h(the)g(user)f(is)h (allo)o(w)o(ed)g(to)f(c)o(hec)o(k)h(equalit)o(y)g(of)f(PGIDs)h(and)f(to)g (include)74 214 y(PGIDs)e(in)h(messages)e(sen)o(t)h(from)f(one)i(pro)q(cess)f (to)f(another.)19 b(As)12 b(a)g(result,)g(CCL)g(m)o(ust)g(guaran)o(tee)f (that)74 270 y(all)19 b(the)f(pro)q(cesses)h(in)f(a)g(single)h(Pro)q(cess)f (Group)g(use)g(the)g(same)g(system-wide)h(unique)g(PGID,)e(and)74 327 y(that)c(distinct)j(Pro)q(cess)e(Groups)f(ha)o(v)o(e)h(distinct)h(PGIDs)f (\(whether)g(or)f(not)h(they)g(ha)o(v)o(e)f(an)o(y)h(pro)q(cesses)74 383 y(in)i(common\).)74 471 y(W)l(e)h(no)o(w)e(describ)q(e)j(a)e(simple)i (mec)o(hanism)f(for)f(generating)g(PGIDs)g(that)f(are)h(system-wide)h (unique.)74 527 y(Eac)o(h)c(pro)q(cess)g(main)o(tains)h(a)e(lo)q(cal)i(coun)o (ter,)f(whic)o(h)h(is)g(initialized)i(to)c(0.)19 b(Logically)l(,)c(a)e(PGID)f (consists)74 583 y(of)g(t)o(w)o(o)g(\014elds:)19 b Fl(\(counter,)k(pid\))p Fn(.)18 b(When)13 b(a)g(new)g(Pro)q(cess)f(Group)g(is)h(created,)g(the)g(pro) q(cess)g(that)f(has)74 640 y(rank)17 b(0)h(in)g(this)g(group)g(determines)g (the)g(PGID)f(b)o(y)h(pairing)g(its)g(lo)q(cal)h(coun)o(ter)e(and)h(its)g (PID.)f(This)74 696 y(pro)q(cess)h(broadcasts)f(the)g(v)m(alue)i(of)e(the)h (PGID)f(to)g(all)h(the)g(pro)q(cesses)g(in)g(the)g(Pro)q(cess)f(Group,)g(and) 74 753 y(it)e(then)g(incremen)o(ts)g(its)g(lo)q(cal)h(coun)o(ter.)k(This)15 b(sc)o(heme)g(guaran)o(tees)f(system-wide)h(uniqueness)h(of)f(all)74 809 y(the)g(PGIDs)g(in)h(CCL.)74 897 y(The)j(implemen)o(tation)h(of)e(the)h (ab)q(o)o(v)o(e)g(mec)o(hanism)g(in)g(CCL)g(is)g(sligh)o(tly)h(di\013eren)o (t.)31 b(Because)19 b(most)74 953 y(existing)i(comm)o(unication)f(subsystems) g(ha)o(v)o(e)f(relativ)o(ely)i(large)f(start-up)f(costs)h(compared)f(to)g (the)74 1010 y(transfer)d(times,)h(CCL)g(app)q(ends)h(the)f(v)m(alue)h(of)e (the)h(lo)q(cal)h(coun)o(ter)f(of)f(eac)o(h)h(pro)q(cess)g(to)f(the)h(con)o (trol)74 1066 y(information)e(comm)o(unicated)f(during)i(the)e(setup)h(of)e (a)h(Pro)q(cess)h(Group.)k(This)c(eliminates)h(the)e(extra)74 1123 y(comm)o(unication)h(steps)g(needed)h(for)e(broadcasting)g(the)h(PID)g (and)f(the)h(coun)o(ter)f(of)h(the)f(pro)q(cess)h(with)74 1179 y(rank)f(0,)g(since)h(eac)o(h)f(pro)q(cess)g(has)g(all)h(the)f(information)h (needed)g(to)e(calculate)j(the)e(PGID)f(of)h(the)g(new)74 1236 y(Pro)q(cess)h(Group)g(to)g(whic)o(h)h(it)f(b)q(elongs.)74 1388 y Fk(2.4)56 b(Common)17 b(Group)h(Structure)g(Routines)74 1505 y Fn(Recen)o(tly)l(,)25 b(a)d(set)g(of)g(Common)g(Group)g(Structure)g (\(CGS\))g(routines)g(has)h(b)q(een)g(prop)q(osed)g(as)f(an)74 1562 y(extension)c(to)f(CCL)g([8].)25 b(The)18 b(CGS)f(routines)h(mak)o(e)e (use)i(of)f(pro)q(cess)h(group)f(routines)g(for)g(de\014ning)74 1618 y(pro)q(cess)i(groups)g(that)f(arise)h(in)h(algorithms)f(with)g(grid)h (and)f(h)o(yp)q(ercub)q(e)i(structures.)30 b(Once)20 b(these)74 1675 y(grid-structured)j(and)f(h)o(yp)q(ercub)q(e-structured)h(groups)f(ha)o (v)o(e)f(b)q(een)i(de\014ned,)i(the)d(standard)f(CCL)74 1731 y(routines)g(can)g(b)q(e)g(used)g(within)g(the)g(structured)f(groups)h(to)e (p)q(erform)i(collectiv)o(e)h(comm)o(unication)74 1788 y(op)q(erations.)27 b(The)18 b(CGS)f(routines)h(also)f(include)j(utilities)g(for)c(con)o(v)o (erting)i(b)q(et)o(w)o(een)g(1-dimensional)74 1844 y(and)e (higher-dimensional)i(addresses.)74 1932 y(As)k(an)h(example,)h(the)f Fg(F)o(ORM2DGRID)f Fn(routine)h(tak)o(es)e(an)h(existing)i(group,)f(views)g (it)g(as)f(an)74 1988 y Fj(X)14 b Fi(\002)e Fj(Y)27 b Fn(2D-grid)17 b(\(where)f Fj(X)k Fn(and)d Fj(Y)27 b Fn(are)17 b(sp)q(eci\014ed)i(b)o(y)d (the)h(user\),)g(and)f(partitions)h(it)g(in)o(to)g(a)f(set)h(of)74 2045 y(nono)o(v)o(erlapping)j(groups)f(corresp)q(onding)h(to)f(the)g(columns) h(of)f(a)g(t)o(w)o(o-dimensional)h(grid)f(and)h(also)74 2101 y(in)o(to)e(a)f(set)g(of)g(nono)o(v)o(erlapping)h(groups)g(corresp)q(onding)g (to)f(the)g(ro)o(ws)g(of)g(a)g(t)o(w)o(o-dimensional)i(grid.)74 2158 y(Th)o(us,)c(eac)o(h)g(pro)q(cess)h(receiv)o(es)g(t)o(w)o(o)e(new)h (PGIDs,)g(one)g(for)g(the)g(column)h(in)g(whic)o(h)g(it)g(is)f(lo)q(cated)h (and)74 2214 y(one)f(for)g(the)g(ro)o(w)g(in)h(whic)o(h)g(it)f(is)h(lo)q (cated)g(in)g(the)f(t)o(w)o(o-dimensional)h(grid.)74 2388 y Fm(3)69 b(Seman)n(tic)21 b(Issues)i(in)f(CCL)74 2521 y Fn(This)e(section)f (discusses)h(sev)o(eral)f(issues)h(related)f(to)f(the)h(seman)o(tics)g(of)f (p)q(oin)o(t-to-p)q(oin)o(t)i(comm)o(uni-)74 2577 y(cation)g(and)g(of)f (collectiv)o(e)j(comm)o(unication)e(op)q(erations)g(on)g(Pro)q(cess)f (Groups.)33 b(These)20 b(issues)h(are)960 2727 y(7)p eop %%Page: 8 8 bop 74 157 a Fn(fundamen)o(tal)16 b(to)e(the)i(design)g(of)f(CCL)g(and)g(of)g (similar)h(libraries.)74 310 y Fk(3.1)56 b(Send)18 b(and)i(Receiv)n(e)c (Seman)n(tics)74 427 y Fn(In)21 b(order)f(for)g(CCL)g(to)g(b)q(e)h(p)q (ortable,)g(it)g(is)g(imp)q(ortan)o(t)e(that)h(it)g(is)h(based)g(on)f(a)g (simple)i(mo)q(del)f(of)74 483 y(p)q(oin)o(t-to-p)q(oin)o(t)16 b(comm)o(unication)g(that)e(is)i(a)o(v)m(ailable)h(on)e(a)g(wide)h(range)f (of)g(mac)o(hines.)74 571 y(The)f(basic)h(mo)q(del)f(of)g(p)q(oin)o(t-to-p)q (oin)o(t)g(comm)o(unication)h(consists)f(of)f(a)h(single)h Fl(send)e Fn(op)q(eration)h(and)g(a)74 627 y(single)i Fl(receive)d Fn(op)q(eration.)20 b(It)14 b(is)h(assumed)f(that)g(a)g(pro)q(cess)g(can)h (send/receiv)o(e)g(to/from)e(an)o(y)h(other)74 684 y(pro)q(cess)g(in)g(the)g (parallel)h(mac)o(hine.)20 b(Namely)l(,)14 b(there)f(is)h(no)g(notion)f(of)g (a)h(top)q(ology)f(at)f(this)i(lev)o(el.)21 b(This)74 740 y(mo)q(del)14 b(is)f(called)i(a)d(fully-connected)j(mo)q(del.)20 b(In)14 b(addition)g(it)f(is)g(assumed)g(that)f(the)h(send)h(and)f(receiv)o(e)74 797 y(op)q(erations)i(ha)o(v)o(e)g(the)h(follo)o(wing)f(four)g(prop)q (erties:)74 884 y(\(1\))h(The)h(op)q(erations)g(are)f(blo)q(c)o(king,)i (namely)l(,)g(a)e Fh(blo)n(cking)h(send)j Fn(op)q(eration)d(and)g(a)g Fh(blo)n(cking)f(r)n(e)n(c)n(eive)74 941 y Fn(op)q(eration)e(are)g(used.)20 b(The)14 b(blo)q(c)o(king)i(send)f(op)q(eration)f(is)g(a)g Fl(send)g Fn(that)f(do)q(es)h(not)g(return)g(to)f(the)h(user)74 997 y(un)o(til)k(the)g(message)e(has)h(b)q(een)i(copied)f(out)f(of)f(the)i (user's)f(bu\013er.)25 b(It)17 b(should)i(b)q(e)e(p)q(oin)o(ted)h(out)f(that) 74 1054 y Fh(blo)n(cking)e(send)20 b Fn(is)c(di\013eren)o(t)g(from)f Fh(synchr)n(onous)h(send)f Fn(in)h(whic)o(h)h(the)f Fl(send)f Fn(do)q(es)h(not)f(return)g(to)g(the)74 1110 y(user)g(un)o(til)h(a)f(matc)o (hing)g Fl(receive)f Fn(op)q(eration)h(has)g(b)q(een)h(issued.)1217 1094 y Ff(1)1258 1110 y Fn(The)f(blo)q(c)o(king)h(receiv)o(e)g(op)q(eration) 74 1167 y(is)g(a)f Fl(receive)g Fn(that)g(do)q(es)g(not)h(return)f(to)g(the)g (user)h(un)o(til)h(the)e(requested)h(message)f(has)h(arriv)o(ed)f(and)74 1223 y(has)g(b)q(een)i(placed)f(in)g(the)f(user's)g(bu\013er.)74 1311 y(\(2\))20 b(The)g(receiv)o(e)i(op)q(eration)f(is)g(a)f Fh(r)n(e)n(c)n(eive-by-sour)n(c)n(e)p Fn(,)g(where)h(the)f(receiv)o(er)h(sp)q (eci\014es)i(the)d(desired)74 1367 y(source)e(of)f(the)h(message.)27 b(Only)19 b(a)f(message)f(from)g(the)h(sp)q(eci\014ed)i(source)d(can)h(b)q(e) h(deliv)o(ered)g(to)e(the)74 1424 y(receiv)o(er.)26 b(The)17 b(desired)h(source)f(m)o(ust)f(alw)o(a)o(ys)g(b)q(e)i(sp)q(eci\014ed)h(and)e (the)g(use)g(of)f(a)h(wildcard)h(v)m(alue)g(for)74 1480 y(the)d(source)h (parameter)e(is)i(not)f(allo)o(w)o(ed.)74 1568 y(\(3\))g(P)o(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication)h(b)q(et)o(w)o(een)f(an)o(y)g(t)o(w)o(o)f(pro)q (cesses)h(is)h(required)g(to)f(b)q(e)g(FIF)o(O)g(suc)o(h)74 1624 y(that)j(m)o(ultiple)i(messages)e(sen)o(t)g(from)g(one)h(pro)q(cess)f (to)g(another)g(are)g(receiv)o(ed)i(in)f(the)g(same)f(order)74 1681 y(in)h(whic)o(h)h(they)e(w)o(ere)g(sen)o(t.)33 b(Ho)o(w)o(ev)o(er,)19 b(the)g Fl(send)g Fn(and)h Fl(receive)e Fn(op)q(erations)i(are)f(async)o (hronous,)74 1737 y(in)i(the)f(sense)h(that)e(no)h(b)q(ounds)h(are)e(assumed) h(on)g(the)g(relativ)o(e)h(sp)q(eeds)g(of)e(the)i(pro)q(cesses)f(or)f(the)74 1794 y(time)c(required)g(to)e(pass)h(messages)g(b)q(et)o(w)o(een)g(pro)q (cesses.)20 b(F)l(urthermore,)13 b(the)h(basic)h(mo)q(del)g(mak)o(es)f(no)74 1850 y(assumptions)j(regarding)g(whether)g(or)g(not)f(the)h(system)g(pro)o (vides)g(bu\013ering)g(of)g(messages)f(b)q(et)o(w)o(een)74 1906 y(a)j(sender)g(and)g(a)g(receiv)o(er.)31 b(Th)o(us,)20 b(a)e(blo)q(c)o(king)j(send)e(op)q(eration)g(ma)o(y)f(or)h(ma)o(y)f(not)g (blo)q(c)o(k)i(un)o(til)g(a)74 1963 y(matc)o(hing)15 b(receiv)o(e)i(is)e (issued.)74 2050 y(\(4\))g(Di\013eren)o(t)g Fl(send)g Fn(op)q(erations)h (issued)h(from)d(m)o(ultiple)k(source)d(pro)q(cesses)h(to)f(a)h(single)h (destination)74 2107 y(pro)q(cess)i(are)e Fh(non-interfering)p Fn(.)28 b(This)19 b(means)f(that)f(the)h(existence)i(of)d(messages)h(that)g (the)g(receiv)o(er)74 2163 y(do)q(es)e(not)f(wish)h(to)f(select)h(\(b)q (ecause)g(these)g(messages)f(do)g(not)g(matc)o(h)g(the)h(source)f(\014eld)i (sp)q(eci\014ed)h(b)o(y)74 2220 y(the)g Fl(receive)e Fn(op)q(eration\))i (cannot)f(prev)o(en)o(t)g(the)h(reception)g(of)f(a)h(message)f(that)f(the)i (receiv)o(er)g(do)q(es)74 2276 y(wish)d(to)g(receiv)o(e.)20 b(F)l(or)14 b(example,)i(if)f(pro)q(cesses)g(1)f(through)h Fj(n)9 b Fi(\000)h Fn(1)k(eac)o(h)h(sends)g(a)g(message)f(to)g(pro)q(cess)74 2333 y(0)h(and)h(if)g(pro)q(cess)g(0)f(issues)h(a)g Fl(receive)e Fn(in)j(order)e(to)g(receiv)o(e)h(a)f(message)g(from)g(pro)q(cess)h Fj(n)11 b Fi(\000)f Fn(1,)15 b(then)74 2389 y(this)g Fl(receive)e Fn(op)q(eration)i(will)h(get)d(the)i(message)f(sen)o(t)g(from)f(pro)q(cess)i Fj(n)8 b Fi(\000)h Fn(1)14 b(ev)o(en)h(if)f(all)i(of)e(the)g(other)74 2446 y(messages)20 b(w)o(ere)h(sen)o(t)f(earlier.)38 b(While)22 b(this)f(non-in)o(terference)h(prop)q(ert)o(y)e(seems)h(v)o(ery)g(natural,)g (it)74 2502 y(could)f(b)q(e)g(violated)f(b)o(y)g(a)g(system)f(that)g(stores)h (messages)f(in)i(bu\013ers)e(on)h(the)g(receiv)o(er)h(side,)g(since)p 74 2542 718 2 v 126 2569 a Fe(1)143 2584 y Fd(Some)14 b(researc)o(hers)g(use) f(the)g(term)g Fc(blo)n(cking)e(send)k Fd(to)e(indicate)i(what)e(w)o(e)g(ha)o (v)o(e)g(de\014ned)h(as)g Fc(synchr)n(onou)o(s)d(send)p Fd(.)960 2727 y Fn(8)p eop %%Page: 9 9 bop 74 157 a Fn(suc)o(h)21 b(bu\013ers)g(could)g(b)q(e)h(\014lled)g(b)o(y)f (messages)f(other)g(than)g(the)h(one)g(that)f(the)g(receiv)o(er)i(selects)f (to)74 214 y(receiv)o(e.)74 301 y(P)o(arallel)c(co)q(de)f(that)g(uses)g (message-passing)g(comm)o(unication)g(with)g(these)g(four)g(prop)q(erties)g (is)h Fh(r)n(ac)n(e-)74 358 y(fr)n(e)n(e)p Fn(:)g(there)10 b(is)h(a)f(unique)i(p)q(ossible)g(matc)o(hing)e(of)g(sends)h(to)e(receiv)o (es,)j(and)e(a)g(unique)i(p)q(ossible)g(successful)74 414 y(execution,)i (indep)q(enden)o(t)g(of)e(the)g(relativ)o(e)g(sp)q(eeds)h(of)f(the)g(pro)q (cesses)g(and)h(the)f(comm)o(unication)h(in)f(the)74 471 y(system.)18 b(Co)q(de)11 b(that)f(is)h(written)f(to)g(b)q(e)h(p)q(ortable)g(mak)o(es)f (no)g(assumption)h(on)g(the)f(amoun)o(t)g(of)g(bu\013ering)74 527 y(pro)o(vided)k(b)o(y)f(the)g(comm)o(unication)h(system;)f(it)h(needs)g (to)e(w)o(ork)g(ev)o(en)i(if)f(eac)o(h)h(send)f(op)q(eration)h(blo)q(c)o(ks) 74 583 y(un)o(til)k(a)f(matc)o(hing)h(receiv)o(e)g(is)f(executed)i(\(i.e.,)e (comm)o(unication)h(is)f(sync)o(hronous\).)26 b(The)17 b(execution)74 640 y(of)e(suc)o(h)g Fh(safe)f Fn(co)q(de)i(is)f(deterministic:)22 b(if)15 b(the)g(program)f(deadlo)q(c)o(ks)i(on)f(some)f(implemen)o(tation,)i (then)74 696 y(it)g(alw)o(a)o(ys)e(deadlo)q(c)o(ks.)74 784 y(Although)k(the)g(basic)h(mo)q(del)g(of)e(p)q(oin)o(t-to-p)q(oin)o(t)h(comm) o(unication)h(de\014ned)g(ab)q(o)o(v)o(e)e(is)i(su\016cien)o(t)f(for)74 840 y(implemen)o(ting)12 b(all)g(of)e(the)g(CCL)h(routines,)g(the)g(actual)f (implemen)o(tation)i(of)e(CCL)g(uses)h(t)o(w)o(o)e(additional)74 897 y(features)j(that)g(are)g(found)h(in)g(man)o(y)f(systems.)19 b(These)13 b(t)o(w)o(o)e(features)h(are)g(a)g Fl(send-receive)f Fn(op)q(eration)74 953 y(and)18 b(the)g(usage)g(of)f(message)g(tags)686 937 y Ff(2)705 953 y Fn(.)27 b(W)l(e)18 b(next)g(discuss)h(these)f(t)o(w)o(o) e(additional)k(features)d(and)h(their)74 1010 y(merits)d(in)h(implemen)o (ting)i(CCL.)74 1097 y(The)f Fl(send-receive)e Fn(op)q(eration)i(sim)o (ultaneously)h(sends)f(a)f(message)g(to)g(one)h(pro)q(cess)g(and)g(receiv)o (es)74 1154 y(a)22 b(message)g(from)f(another)h(pro)q(cess)h(without)f (requiring)i(additional)f(bu\013er)g(space)f(for)g(either)h(of)74 1210 y(the)17 b(messages.)25 b(CCL)17 b(uses)h(the)f Fl(send-receive)e Fn(op)q(eration)j(for)e(implemen)o(ting)j Fl(shift)d Fn(op)q(erations)74 1267 y(within)23 b(Pro)q(cess)f(Groups.)38 b(The)22 b Fl(shift)f Fn(op)q(eration)h(ma)o(y)f(create)g(a)g(cycle)i(of)e(pro)q(cesses,)i(eac)o(h) f(of)74 1323 y(whic)o(h)c(is)g(sending)h(a)e(message)g(to)g(its)h(successor)f (in)i(the)e(cycle)i(and)e(receiving)i(a)f(message)f(from)f(its)74 1380 y(predecessor)h(in)g(the)f(cycle.)24 b(If)17 b(a)f Fl(send-receive)e Fn(op)q(eration)i(is)h(not)f(used,)h(then)f(the)g Fl(send)g Fn(and)g(the)74 1436 y Fl(receive)f Fn(op)q(erations)h(m)o(ust)f(b)q(e)h (paired)g(carefully)h(to)e(a)o(v)o(oid)g(deadlo)q(c)o(ks.)22 b(\(A)16 b(deadlo)q(c)o(k)g(can)g(o)q(ccur)g(if)74 1493 y Fl(send)d Fn(op)q(erations)h(are)f(forced)h(to)f(w)o(ait)g(for)g(matc)o(hing)h Fl(receive)e Fn(op)q(erations)i(that)f(are)g(nev)o(er)h(issued.\))74 1549 y(F)l(or)i(example,)h(to)e(a)o(v)o(oid)h(deadlo)q(c)o(ks)h(in)g Fl(shift)f Fn(op)q(erations,)g(one)g(can)h(pair)f(the)h Fl(send)e Fn(and)i Fl(receive)74 1606 y Fn(op)q(erations,)k(if)g(the)f(cycle)i(is)e(of) g(ev)o(en)g(length,)i(as)e(follo)o(ws.)35 b(Pro)q(cesses)20 b(in)h(ev)o(en)f(p)q(ositions)h(in)g(the)74 1662 y(cycle)16 b(w)o(ould)f(p)q(erform)g(a)f Fl(send)g Fn(follo)o(w)o(ed)h(b)o(y)g(a)f Fl(receive)p Fn(,)f(while)k(pro)q(cesses)e(in)g(o)q(dd)g(p)q(ositions)h(in)f (the)74 1718 y(cycle)h(w)o(ould)f(p)q(erform)f(a)h Fl(receive)e Fn(follo)o(w)o(ed)i(b)o(y)g(a)f Fl(send)p Fn(.)19 b(\(Cycles)c(of)g(o)q(dd)g (lengths)g(can)g(b)q(e)g(handled)74 1775 y(similarly)l(.\))33 b(Although)19 b(suc)o(h)g(an)g(algorithm)g(is)g(p)q(ossible)i(in)f(CCL,)e (since)i(eac)o(h)f(pro)q(cess)g(kno)o(ws)f(its)74 1831 y(p)q(osition)c(in)f (the)g(cycle,)h(it)f(is)g(m)o(uc)o(h)f(simpler)i(and)f(more)f(e\016cien)o(t)h (to)f(use)h(a)f Fl(send-receive)f Fn(op)q(eration)74 1888 y(for)k(implemen)o (ting)i(the)e Fl(shift)p Fn(.)74 1975 y(Some)f(p)q(oin)o(t-to-p)q(oin)o(t)h (comm)o(unication)g(libraries)h(allo)o(w)e(the)g(use)h(of)f(wildcard)h(v)m (alues)g(in)g(the)f(source)74 2032 y(parameter)h(of)g(a)h(receiv)o(e)g(call.) 23 b(Ev)o(en)16 b(if)g(wildcard)h(v)m(alues)g(are)e(not)g(used)i(in)f(the)g (implemen)o(tation)h(of)74 2088 y(CCL,)h(there)h(is)g(still)h(the)f(risk)g (that)f(a)g(message)g(sen)o(t)g(as)g(part)g(of)h(the)f(implemen)o(tation)i (of)e(a)g(CCL)74 2145 y(call)i(will)g(b)q(e)g(mistak)o(enly)f(receiv)o(ed)h (b)o(y)f(a)f(user)h(receiv)o(e)h(with)f(a)f(wildcard)i(source.)31 b(An)19 b(additional)74 2201 y(mec)o(hanism)e(is)g(needed)h(to)e(distinguish) j(CCL)d(messages)g(from)g(user)h(messages.)23 b(Message)16 b(tags)f(can)74 2258 y(pro)o(vide)j(suc)o(h)g(mec)o(hanism.)27 b(These)18 b(are)f(v)m(alues)i(that)e(are)g(app)q(ended)i(to)e(the)g(con)o (ten)o(t)g(of)g(p)q(oin)o(t-to-)74 2314 y(p)q(oin)o(t)f(messages.)k(T)l(ags) 15 b(are)g(t)o(ypically)h(used)g(to)f(distinguish)j(b)q(et)o(w)o(een)d (di\013eren)o(t)h(t)o(yp)q(es)f(of)g(messages)74 2371 y(in)f(an)g (application,)h(as)e(w)o(ell)h(as)f(b)q(et)o(w)o(een)h(m)o(ultiple)h (messages)e(sen)o(t)g(to)f(a)h(destination)i(from)d(the)i(same)74 2427 y(source.)34 b(Eac)o(h)19 b Fl(send)g Fn(op)q(eration)h(sp)q(eci\014es)i (a)d(v)m(alue)i(in)g(the)f(tag)e(\014eld)j(of)f(the)f(message,)i(and)e(eac)o (h)74 2483 y Fl(receive)14 b Fn(op)q(eration)i(sp)q(eci\014es)h(a)e(desired)h (tag)e(v)m(alue)j(to)d(b)q(e)i(matc)o(hed)f(with)h(the)f(receiv)o(ed)h (message.)p 74 2523 718 2 v 126 2550 a Fe(2)143 2566 y Fd(Some)e(researc)o (hers)g(use)f(the)g(term)g Fc(typ)n(e)h Fd(instead)h(of)d Fc(tag)p Fd(.)960 2727 y Fn(9)p eop %%Page: 10 10 bop 74 157 a Fn(W)l(e)21 b(assume)f(that)f(a)h(mec)o(hanism)h(is)g(a)o(v)m (ailable)h(to)e(allo)q(cate)h(tag)f(ranges)g(to)f(v)m(arious)i(libraries,)i (so)74 214 y(that)16 b(user)i(messages)e(and)h(messages)g(generated)g(b)o(y)g (CCL)g(ha)o(v)o(e)f(disjoin)o(t)i(tag)e(ranges.)25 b(If)17 b(wildcard)74 270 y(tag)f(v)m(alues)i(are)f(allo)o(w)o(ed)g(in)h(receiv)o(e)g (op)q(erations,)f(one)g(also)g(need)h(to)f(mak)o(e)f(sure)h(that)f(user)i (receiv)o(e)74 327 y(op)q(erations)j(cannot)f(matc)o(h)f(tag)h(v)m(alues)h (used)g(b)o(y)f(CCL.)g(This)h(can)g(b)q(e)g(ac)o(hiev)o(ed)g(either)g(with)g (an)74 383 y(include/exclu)q(de)c(mec)o(hanism)d(for)f(wildcard)h(tag)f (ranges,)g(as)g(pro)o(vided)i(b)o(y)e(Express)h([30)o(])f(or)g(with)h(an)74 439 y(additional)j(con)o(text)d(mec)o(hanism,)i(as)f(pro)o(vided)h(b)o(y)f (Zip)q(co)q(de)i([28)o(].)74 592 y Fk(3.2)56 b(Implem)o(en)n(tation)16 b(Correctness)74 709 y Fn(There)i(are)f(man)o(y)g(issues)h(related)g(to)f(a)g (correct)g(implemen)o(tation)h(of)f(CCL.)h(These)f(issues)i(include,)74 766 y(for)f(example,)i(separating)f(user)g(p)q(oin)o(t-to-p)q(oin)o(t)g (messages)f(from)g(CCL)h(p)q(oin)o(t-to-p)q(oin)o(t)g(messages,)74 822 y(guaran)o(teeing)11 b(that)g(the)g(correct)g(messages)g(b)q(e)h(deliv)o (ered)h(for)d(collectiv)o(e)j(comm)o(unication)f(op)q(erations)74 879 y(in)g(o)o(v)o(erlapping)f(Pro)q(cess)f(Groups,)h(and)g(distinguishing)i (b)q(et)o(w)o(een)e(messages)f(that)g(b)q(elong)h(to)f(di\013eren)o(t)74 935 y(collectiv)o(e)17 b(comm)o(unication)f(op)q(erations)f(in)h(the)g(same)f (group.)74 1023 y(First,)h(the)h(seman)o(tics)g(of)f(CCL)g(assume)h(that)e (eac)o(h)i(CCL)f(routine)h(b)q(e)h(view)o(ed)f(as)f(a)g(barrier)h(b)o(y)f (the)74 1079 y(user.)26 b(Namely)l(,)18 b(a)e(pro)q(cess)i(can)f(return)g (from)f(a)h(CCL)g(routine)h(only)f(after)g(all)h(other)e(participating)74 1136 y(pro)q(cesses)k(called)i(this)e(routine.)35 b(Therefore,)20 b(in)h(a)f(correct)f(program,)h(an)o(y)f(CCL)h(op)q(eration)g(that)74 1192 y(in)o(v)o(olv)o(es)15 b(pro)q(cesses)h Fj(p)e Fn(and)h Fj(q)i Fn(should)f(b)q(e)f(called)h(in)g(the)f(same)f(order)h(b)o(y)g(pro)q (cesses)g Fj(p)f Fn(and)h Fj(q)r Fn(,)g(and)g(no)74 1248 y(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication)g(b)q(et)o(w)o(een)f(pro)q(cesses)h Fj(p)f Fn(and)g Fj(q)i Fn(should)f(o)o(v)o(erlap)f(a)g(CCL)g(op)q(eration.)74 1336 y(The)j(correctness)f(of)g(the)h(CCL)f(implemen)o(tation)i(is)f(based)g (on)f(the)h(follo)o(wing)g(3)f(prop)q(erties)h(of)f(the)74 1393 y(p)q(oin)o(t-to-p)q(oin)o(t)f(comm)o(unication.)130 1517 y(1.)22 b(The)15 b(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o(unication)g(supp)q (orts)f(receiv)o(e-b)o(y-source)i(op)q(erations.)130 1611 y(2.)22 b(The)15 b(p)q(oin)o(t-to-p)q(oin)o(t)h(comm)o(unication)g(is)g(FIF)o(O)f (and)g(non-in)o(terfering.)130 1705 y(3.)22 b(A)13 b(collectiv)o(e)i(comm)o (unication)e(algorithm)g(in)h(CCL)f(is)g(kno)o(wn)g(to)f(all)i(participating) g(pro)q(cesses.)188 1762 y(In)k(particular,)f(for)g(a)g(sp)q(eci\014c)i (collectiv)o(e)f(comm)o(unication)g(op)q(eration)g(o)o(v)o(er)e(a)h(giv)o(en) g(pro)q(cess)188 1818 y(group)e(with)h(kno)o(wn)g(source)g(and)g(destination) h(pro)q(cesses)f(\(if)g(applicable\))i(and)e(message)f(size,)188 1874 y(a)h(\014xed)h(algorithm)f(is)h(c)o(hosen)g(that)e(is)i(kno)o(wn)f(to)g (all)h(the)f(pro)q(cesses.)24 b(Consequen)o(tly)l(,)17 b(at)f(an)o(y)188 1931 y(giv)o(en)f(p)q(oin)o(t)h(in)f(the)g(algorithm,)g(eac)o(h)g(pro)q(cess) g(kno)o(ws)g(exactly)g(to)f(whic)o(h)i(pro)q(cess)f(a)g(message)188 1987 y(should)j(b)q(e)h(sen)o(t)e(or)g(from)g(whic)o(h)h(pro)q(cess)g(a)f (message)g(should)i(b)q(e)f(receiv)o(ed.)28 b(Therefore,)17 b(the)188 2044 y(sequence)f(of)f(steps)g(for)g(implemen)o(ting)i(eac)o(h)e (CCL)g(op)q(eration)h(is)f(w)o(ell)i(de\014ned.)74 2169 y(The)j(foregoing)f (3)g(prop)q(erties)h(guaran)o(tee)f(that)f(the)i(messages)f(receiv)o(ed-b)o (y-source)i(b)o(y)e(an)o(y)g(giv)o(en)74 2225 y(pro)q(cess)e(are)f(receiv)o (ed)i(in)f(the)g(exact)f(same)h(order)f(for)g(an)o(y)g(giv)o(en)h(execution)h (of)e(a)g(CCL)h(algorithm.)74 2282 y(This)k(order)f(is)g(main)o(tained)h(in)g (the)f(presence)h(of)f(other)f(comm)o(unication,)j(lik)o(e)f(p)q(oin)o (t-to-p)q(oin)o(t)g(or)74 2338 y(other)15 b(CCL)h(calls.)22 b(The)16 b(k)o(ey)f(issue)i(is)f(that)f(the)h(FIF)o(O)f(prop)q(ert)o(y)g(b)q (et)o(w)o(een)h(source)g(and)g(destination)74 2395 y(is)j(main)o(tained)g(b)o (y)g(main)o(taining)g(the)f(order)g(b)q(et)o(w)o(een)h(CCL)f(calls)i(and)e (not)g(allo)o(wing)i(in)o(terference)74 2451 y(b)q(et)o(w)o(een)c(CCL)f (calls)h(and)g(p)q(oin)o(t-to-p)q(oin)o(t)f(calls.)949 2727 y(10)p eop %%Page: 11 11 bop 74 157 a Fk(3.3)56 b(Barrier)17 b(Seman)n(tics)74 274 y Fn(As)e(discussed)i(ab)q(o)o(v)o(e,)d(for)h(program)f(correctness,)h(eac)o(h) g(CCL)g(op)q(eration)g(should)h(b)q(e)g(view)o(ed)g(as)f(im-)74 331 y(p)q(osing)h(a)e(barrier)h(sync)o(hronization)h(on)f(its)g(participan)o (ts.)20 b(Ho)o(w)o(ev)o(er,)14 b(from)g(the)h(p)q(erformance)g(p)q(oin)o(t)74 387 y(of)g(view,)g(imp)q(osing)h(a)e(barrier)h(sync)o(hronization)h(on)f(all) h(the)e(participan)o(ts)i(in)f(CCL)g(op)q(erations)g(ma)o(y)74 443 y(ha)o(v)o(e)j(a)g(negativ)o(e)h(e\013ect)f(on)g(p)q(erformance.)30 b(T)l(o)18 b(address)g(this)h(problem,)g(CCL)g(allo)o(ws)f(t)o(w)o(o)f(mo)q (des)74 500 y(of)h(op)q(eration:)26 b Fh(b)n(arrier)19 b(mo)n(de)j Fn(and)d Fh(non-b)n(arrier)f(mo)n(de)p Fn(.)30 b(Selection)20 b(b)q(et)o(w)o(een)e(these)h(t)o(w)o(o)e(mo)q(des)h(is)74 556 y(done)h(globally)h(at)d(the)i(b)q(eginning)h(of)e(an)h(application.)31 b(F)l(or)18 b(ob)o(vious)g(reasons,)h(the)f(default)h(mo)q(de)74 613 y(is)f(the)f(barrier)h(mo)q(de.)26 b(When)18 b(a)f(CCL)g(op)q(eration)h (is)g(executed)g(in)g(barrier)g(mo)q(de,)g(no)f(pro)q(cess)g(can)74 669 y(complete)g(its)g(call)h(to)e(the)h(routine)g(un)o(til)g(all)h(the)f (pro)q(cesses)g(in)g(the)g(relev)m(an)o(t)g(Pro)q(cess)g(Group)f(ha)o(v)o(e) 74 726 y(executed)h(their)g(\(corresp)q(onding\))f(calls)h(to)f(the)g(same)g (routine.)22 b(Therefore,)16 b(when)h(an)f(op)q(eration)g(is)74 782 y(executed)j(in)f(barrier)g(mo)q(de,)f(there)h(m)o(ust)f(exist)h(some)f (p)q(oin)o(t)h(in)g(time)g(at)f(whic)o(h)h(all)h(the)e(pro)q(cesses)74 839 y(in)g(the)f(group)f(are)h Fh(simultane)n(ously)e Fn(executing)j(the)f (same)g(op)q(eration.)22 b(In)16 b(con)o(trast,)e(when)i(an)g(op)q(er-)74 895 y(ation)h(is)g(executed)h(in)f(non-barrier)g(mo)q(de,)g(eac)o(h)g(pro)q (cess)g(blo)q(c)o(ks)g(only)g(un)o(til)h(it)f(has)g(completed)g(its)74 952 y(participation)h(in)g(the)f(op)q(eration.)26 b(Notice)17 b(that)g(the)g(role)g(of)g(eac)o(h)g(pro)q(cess)g(ma)o(y)g(dep)q(end)h(not)f (only)74 1008 y(on)e(the)h(seman)o(tics)f(of)g(the)g(op)q(eration)g(but)h (also)f(on)g(the)g(sp)q(eci\014c)i(algorithm)f(used)f(to)g(implemen)o(t)h (it.)74 1096 y(The)d(di\013erences)h(b)q(et)o(w)o(een)f(the)f(t)o(w)o(o)g(mo) q(des)g(ma)o(y)g(imply)i(di\013erences)g(in)f(the)g(b)q(eha)o(vior)g(of)f(an) h(applica-)74 1152 y(tion)g(program.)18 b(In)c(a)f(non-barrier)g(mo)q(de,)g (pro)q(cesses)h(ma)o(y)e(return)h(more)g(quic)o(kly)h(from)e(calls)i(to)e (CCL)74 1209 y(routines,)19 b(resulting)g(in)h(a)d(b)q(etter)i(p)q (erformance.)29 b(Ho)o(w)o(ev)o(er,)17 b(there)i(is)f(also)h(some)f(danger)g (in)h(using)74 1265 y(non-barrier)14 b(mo)q(de)f(routines)h(as)f(in)h(some)f (of)g(CCL)g(routines)g(the)h(role)f(of)g(eac)o(h)g(pro)q(cess)h(dep)q(ends)h (also)74 1321 y(on)h(the)f(particular)i(algorithm)e(used)i(to)e(implemen)o(t) h(the)g(routine.)22 b(As)15 b(long)h(as)g(the)f(user's)h(program)74 1378 y(is)i(correct,)g(and)f(no)h(wildcard)h(p)q(oin)o(t-to-p)q(oin)o(t)f (comm)o(unication)h(is)f(used,)g(then)g(non-barrier)g(mo)q(de)74 1434 y(routines)h(will)i(op)q(erate)e(in)h(an)e(implemen)o(tation-indep)r (ende)q(n)o(t)j(manner)e(\(as)f(argued)h(in)h(the)f(ab)q(o)o(v)o(e)74 1491 y(subsection\).)h(Ho)o(w)o(ev)o(er,)14 b(if)g(wildcard)i(p)q(oin)o (t-to-p)q(oin)o(t)f(comm)o(unication)g(is)g(used,)g(then)g(the)f(b)q(eha)o (vior)74 1547 y(of)h(a)f(non-barrier)i(mo)q(de)f(routine)h(can)f(b)q(e)g (implemen)o(tation-dep)q(end)q(en)o(t.)22 b(As)15 b(an)g(example,)h(consider) 74 1604 y(the)e(follo)o(wing)h(piece)h(of)e(co)q(de,)h(in)g(whic)o(h)g Fl(G)f Fn(is)h(the)f(group)g(iden)o(ti\014er)i(of)e(a)g(Pro)q(cess)g(Group)g (consisting)74 1660 y(of)i(pro)q(cesses)h Fl(a)p Fn(,)f Fl(b)p Fn(,)g(and)g Fl(c)p Fn(.)23 b(In)17 b(this)f(example,)h(eac)o(h)g Fl(brecv)e Fn(is)i(a)f(blo)q(c)o(king)i(receiv)o(e)f(op)q(eration)f(with)74 1717 y(a)f(wildcard)i(source)e(parameter.)193 1805 y Fl(process)23 b(a)334 b(process)23 b(b)334 b(process)23 b(c)193 1862 y(---------)333 b(---------)g(---------)193 1918 y(bsend\(c\))906 b(brecv\(*\))193 1975 y(bcast\(G\))357 b(bcast\(G\))g(bcast\(G\))742 2031 y(bsend\(c\))g (brecv\(*\))74 2176 y Fn(Assume)17 b(that)e(the)h(source)g(of)g(the)g Fl(bcast)g Fn(op)q(eration)g(is)g(pro)q(cess)h Fl(b)p Fn(.)22 b(If)17 b(barrier)f(seman)o(tics)g(are)g(used)74 2233 y(for)d(the)h (broadcast)g(op)q(eration,)g(then)g(the)g(\014rst)g(receiv)o(e)g(of)g(pro)q (cess)g Fl(c)g Fn(will)h(receiv)o(e)g(the)f(message)g(sen)o(t)74 2289 y(b)o(y)i(pro)q(cess)g Fl(a)p Fn(,)g(and)g(the)g(second)g(receiv)o(e)h (of)e(pro)q(cess)h Fl(c)g Fn(will)i(receiv)o(e)f(the)f(message)f(send)h(b)o (y)g(pro)q(cess)74 2346 y Fl(b)p Fn(.)38 b(On)22 b(the)f(other)g(hand,)i(if)f (pro)q(cess)g Fl(b)f Fn(can)g(exit)h(the)f(broadcast)g(call)h(b)q(efore)g (pro)q(cess)g Fl(c)f Fn(starts)74 2402 y(executing)d(the)f(broadcast,)f(then) i(it)f(is)g(p)q(ossible)i(for)d(the)h(messages)g(to)f(b)q(e)i(receiv)o(ed)g (in)g(the)f(rev)o(erse)74 2458 y(order:)j(pro)q(cess)15 b Fl(c)f Fn(receiv)o(es)i(b)q(efore)f(the)g(broadcast)f(the)h(message)f(that)g Fl(b)h Fn(sen)o(t)g(after)f(the)h(broadcast,)74 2515 y(and)h(receiv)o(es)g (after)e(the)h(broadcast)g(the)g(message)g(that)f Fl(a)h Fn(sen)o(t)g(b)q (efore)h(the)f(broadcast!)74 2602 y(In)f(summary)l(,)f(non-barrier)h(mo)q(de) g(routines)f(ma)o(y)g(exhibit)i(a)e(di\013eren)o(t)h(b)q(eha)o(vior)f(than)h (barrier)f(mo)q(de)949 2727 y(11)p eop %%Page: 12 12 bop 74 157 a Fn(routines,)14 b(when)f(nondeterministic)i(comm)o(unication)f (op)q(erations)f(are)f(allo)o(w)o(ed.)20 b(F)l(urthermore,)12 b(ev)o(en)74 214 y(if)k(only)f(deterministic)i(comm)o(unication)f(op)q (erations)g(are)e(used,)i(a)e(co)q(de)i(that)f(w)o(ould)g(deadlo)q(c)o(k)h (with)74 270 y(barrier)23 b(mo)q(de)g(routines)g(ma)o(y)f(complete)h (successfully)i(on)e(some)f(executions)i(when)f(non-barrier)74 327 y(mo)q(de)17 b(routines.)26 b(Therefore,)16 b(barrier)i(mo)q(de)f(should) h(b)q(e)f(used)h(when)f(debugging)h(co)q(de)f(\(that)f(is,)i(in)74 383 y(iden)o(tifying)h(bugs)f(that)f(ma)o(y)f(not)h(b)q(e)h(caugh)o(t)f(in)h (non-barrier)g(mo)q(de\).)27 b(Once)18 b(a)f(co)q(de)h(is)g(kno)o(wn)f(to)74 439 y(op)q(erate)h(correctly)g(in)h(barrier)g(mo)q(de,)g(it)f(is)h(p)q (ossible)g(to)f(switc)o(h)g(to)g(non-barrier)h(mo)q(de)f(for)f(b)q(etter)74 496 y(p)q(erformance)22 b(\(pro)o(vided)g(that)e(the)i(barrier)f(mo)q(de)h (is)g(not)f(required)h(for)f(the)g(correctness)g(of)g(the)74 552 y(program\).)e(Finally)l(,)e(notice)e(that)g(it)h(is)f(alw)o(a)o(ys)g(p)q (ossible)i(to)e(use)g(an)g(explicit)j(sync)o(hronization)e(\(i.e.,)74 609 y(to)f(call)h(the)f(routine)h Fl(sync)p Fn(\))e(when)i(a)f(barrier)g(is)h (needed.)74 696 y(Some)d(CCL)g(op)q(erations)f(ha)o(v)o(e)h(seman)o(tics)g (that)f(imply)i(a)e(barrier,)h(while)h(others)e(do)h(not.)19 b(Op)q(erations)74 753 y(that)g(imply)h(a)f(barrier)h(are)e Fl(combine)p Fn(,)h Fl(concat)p Fn(,)g Fl(index)p Fn(,)g(and)h Fl(sync)p Fn(.)31 b(Th)o(us,)20 b(the)f(implemen)o(tation)74 809 y(of)g(these)g(op)q(erations)g(is)g(the)h(same)e(for)h(barrier)g(and)g (for)f(non-barrier)i(mo)q(des.)31 b(Other)19 b(op)q(erations)74 866 y(that)d(do)h(not)f(imply)i(a)e(barrier)h(are)f Fl(bcast)p Fn(,)g Fl(reduce)p Fn(,)g Fl(scatter)p Fn(,)f Fl(gather)p Fn(,)h Fl(prefix)p Fn(,)g(and)h Fl(shift)p Fn(.)23 b(In)74 922 y(non-barrier)15 b(mo)q(de,)g(the)g(latter)f(op)q(erations)h(\(except)g Fl(shift)p Fn(\))e(are)i(implemen)o(ted)h(with)f(a)f(fan-in)i(tree)74 979 y(to)i(a)g(particular)i(destination)f(or)f(a)g(fan-out)h(tree)f(from)g(a) g(particular)i(source.)30 b(F)l(or)18 b(implemen)o(ting)74 1035 y(these)k(op)q(erations)g(in)h(barrier)f(mo)q(de,)h(w)o(e)f(use)g(t)o(w) o(o)f(in)o(ternal)h(partial)h(sync)o(hronization)f(routines:)74 1092 y Fl(sync-in\(dest,G\))11 b Fn(whic)o(h)k(builds)g(a)e(fan-in)h(tree)g (in)g(the)f(group)g Fl(G)h Fn(that)e(con)o(v)o(erges)h(at)g(pro)q(cess)g Fl(dest)p Fn(,)74 1148 y(and)g Fl(sync-out\(src,G\))d Fn(whic)o(h)j(builds)h (a)e(fan-out)g(tree)g(in)h(the)f(group)g Fl(G)g Fn(that)g(div)o(erges)h(from) e(pro)q(cess)74 1205 y Fl(src)p Fn(.)22 b(A)16 b(barrier)g(mo)q(de)h(for)e (the)h(latter)g(op)q(erations)g(is)h(implemen)o(ted)h(b)o(y)e(p)q(erforming)g (a)g Fl(sync-in\(\))74 1261 y Fn(b)q(efore)f(a)f(fan-out)h(t)o(yp)q(e)f(op)q (eration)h(\(lik)o(e)g Fl(bcast)f Fn(and)h Fl(scatter)p Fn(\),)e(or)h(b)o(y)h (p)q(erforming)g(a)f Fl(sync-out\(\))74 1317 y Fn(after)d(a)h(fan-in)g(t)o (yp)q(e)g(op)q(eration)g(\(lik)o(e)h Fl(reduce)p Fn(,)e Fl(gather)p Fn(,)g(and)h Fl(prefix)p Fn(\).)18 b(Notice)12 b(that)f(for)g(the)h Fl(prefix)74 1374 y Fn(op)q(eration,)18 b(the)g(fan-out)f(tree)g(is)h(ro)q (oted)g(at)e(the)i(highest)g(rank)o(ed)g(pro)q(cess)g(in)g(the)g(group.)26 b(Finally)l(,)74 1430 y(regarding)d(the)f(Pro)q(cess)g(Group)g(creation)h(op) q(erations,)h Fl(partition\(\))c Fn(implies)25 b(a)d(barrier)g(mo)q(de)74 1487 y(within)17 b(the)e(paren)o(t)g(group)g(while)h Fl(group\(\))e Fn(do)q(es)i(not.)74 1640 y Fk(3.4)56 b(Dev)n(elop)17 b(and)j(Run)e(Mo)r(des) 74 1757 y Fn(In)h(addition)g(to)f(the)g(c)o(hoice)h(b)q(et)o(w)o(een)f (barrier)g(and)h(non-barrier)f(mo)q(des,)h(the)f(user)h(can)f(select)h(b)q (e-)74 1813 y(t)o(w)o(een)d Fh(develop)i Fn(and)e Fh(run)k Fn(mo)q(des.)i(In)16 b Fh(develop)g Fn(mo)q(de,)g(CCL)g(p)q(erforms)g(more)f (detailed)j(consistency)74 1869 y(c)o(hec)o(kings)f(of)e(the)h(parameters)f (that)h(the)g(user)g(sp)q(eci\014es)i(for)d(CCL)h(calls.)23 b(This)16 b(increases)h(the)f(o)o(v)o(er-)74 1926 y(head)h(of)g(the)g(CCL)g (op)q(erations)g(and)g(ma)o(y)f(require)h(more)g(comm)o(unication)g(b)q(et)o (w)o(een)h(pro)q(cesses.)25 b(In)74 1982 y Fh(run)17 b Fn(mo)q(de,)f(man)o(y) g(of)g(the)h(consistency)g(c)o(hec)o(kings)g(are)f(suppressed,)h(and)g(only)g (c)o(hec)o(kings)g(that)f(are)74 2039 y(required)22 b(b)o(y)e(the)g(seman)o (tics)h(of)f(the)g(op)q(erations)h(are)f(p)q(erformed.)35 b(The)21 b(default)g(in)g(CCL)g(is)f Fh(run)74 2095 y Fn(mo)q(de)d(and)g(the)g(mo)q (de)g(can)g(b)q(e)g(c)o(hanged)g(b)o(y)g(the)g(user.)24 b(Examples)18 b(of)e(consistency)i(c)o(hec)o(kings)f(that)74 2152 y(are)g(done)g(only)h(in) f Fh(develop)g Fn(mo)q(de)h(are)e(c)o(hec)o(king)i(the)f(correctness)g(of)f (the)h(source,)h(the)f(destination)74 2208 y(and)d(the)g(message)g(length)g (supplied)j(to)c(a)h(CCL)f(routine,)i(as)e(w)o(ell)i(as)f(c)o(hec)o(king)h (the)f(mem)o(b)q(ership)h(list)74 2265 y(of)g(a)g(Pro)q(cess)g(Group)g(to)f (b)q(e)i(v)m(alid.)74 2418 y Fk(3.5)56 b(No)18 b(T)-5 b(ags)20 b(for)e(CCL)i(Routines)74 2534 y Fn(In)i(certain)g(comm)o(unication)g (libraries,)i(the)d(predominan)o(t)g(mec)o(hanism)h(for)f(matc)o(hing)g (incoming)74 2591 y(messages)d(to)f(receiv)o(e)j(op)q(erations)e(is)h Fh(matching)g(by)g(tag)p Fn(,)g(rather)e(than)h Fh(matching)h(by)g(sour)n(c)n (e)p Fn(.)29 b(This)949 2727 y(12)p eop %%Page: 13 13 bop 74 157 a Fn(suggests)17 b(a)g(v)o(ersion)h(of)f(CCL)g(where)h(the)f(user) h(is)g(required)g(to)f(pro)o(vide)h(a)f(tag)g(for)g(eac)o(h)g(CCL)h(call;)74 214 y(this)d(approac)o(h)e(is)i(follo)o(w)o(ed,)f(for)f(example,)i(b)o(y)e (Express)h([30)o(].)19 b(The)c(requiremen)o(t)f(on)g(tags)f(is)h(that)g(all) 74 270 y(pro)q(cesses)i(that)e(execute)i(a)f(collectiv)o(e)i(comm)o (unication)f(call)g(pro)o(vide)g(the)f(same)g(tag)f(v)m(alue.)74 358 y(Assume)19 b(that)g(only)g(blo)q(c)o(king)i(p)q(oin)o(t-to-p)q(oin)o(t)e (comm)o(unications)h(are)f(used,)h(and)f(that)g(all)h(receiv)o(e)74 414 y(op)q(erations)e(matc)o(h)g(incoming)h(messages)e(b)o(y)h(tag,)f(with)i (no)f(wildcards.)29 b(F)l(urther)18 b(assume)f(that)h(no)74 471 y(t)o(w)o(o)c(messages)g(with)h(the)g(same)g(tag)f(can)h(b)q(e)h(sim)o (ultaneously)g(w)o(aiting)f(to)f(b)q(e)i(receiv)o(ed)g(b)o(y)f(the)g(same)74 527 y(receiv)o(er.)20 b(Then)14 b(the)f(co)q(de)h(is)g Fh(r)n(ac)n(e-fr)n(e)n (e)p Fn(.)k(It)13 b(follo)o(ws)g(that)g(CCL)g(can)h(b)q(e)g(implemen)o(ted)g (using)g(receiv)o(e-)74 583 y(b)o(y-tag,)h(pro)o(vided)h(that)f(the)h(user)g (guaran)o(tees)f(that)g(the)h(same)f(tag)g(v)m(alue)i(is)f(nev)o(er)g(sim)o (ultaneously)74 640 y(used)g(b)o(y)f(t)o(w)o(o)f(op)q(erations)h(that)g(in)o (v)o(olv)o(e)h(a)e(common)h(pro)q(cess.)74 728 y(This)d(condition)g(ma)o(y)e (b)q(e)h(a)o(wkw)o(ard)f(to)g(ful\014ll.)21 b(F)l(urthermore,)11 b(the)g(e\016cien)o(t)g(implemen)o(tation)h(of)f(some)74 784 y(collectiv)o(e)16 b(comm)o(unication)f(op)q(erations)g(\(suc)o(h)f(as)g(an)h (ascend)f(comm)o(unication)i(on)e(a)g(tree\))g(requires)74 840 y(matc)o(hing)i(b)o(y)g(more)f(than)h(tag;)f(e.g.,)g(using)h Fl(\(source,)23 b(tag\))15 b Fn(to)h(select)g(incoming)h(messages.)22 b(But,)74 897 y(then,)15 b(matc)o(hing-b)o(y-source)h(is)g(su\016cien)o(t,)f (and)h(the)f(tag)f(is)i(sup)q(er\015uous.)74 1071 y Fm(4)69 b(P)n(erformance)23 b(Issues)g(in)f(CCL)74 1204 y Fn(This)14 b(section)f(discusses)h(sev)o(eral)f(p)q(erformance)g(issues)h(and)f(describ) q(es)h(some)f(of)f(the)h(algorithms)g(used)74 1260 y(in)19 b(CCL.)e(The)h(design)h(of)e(comm)o(unication)i(algorithms)f(for)f(CCL)h (attempts)e(to)i(address)f(b)q(oth)h(the)74 1317 y(e\016ciency)j(and)e(the)h (p)q(ortabilit)o(y)g(of)f(the)g(comm)o(unication)h(op)q(erations.)32 b(In)20 b(the)g(discussion)h(of)d(the)74 1373 y(comm)o(unication)e (algorithms)f(in)h(this)g(section,)f(w)o(e)g(assume)g(one)h(pro)q(cess)f(p)q (er)h(pro)q(cessor.)74 1526 y Fk(4.1)56 b(Comm)n(unication)16 b(Mo)r(del)74 1643 y Fn(In)j(designing)g(algorithms)f(for)f(the)h(CCL)g(op)q (erations,)g(w)o(e)f(assume)h(a)f(mo)q(del)i(of)f(a)f(fully-connected)74 1699 y(message-passing)j(system)f(in)h(whic)o(h)g(eac)o(h)g(pro)q(cess)g(can) f(comm)o(unicate)h(directly)h(with)f(an)o(y)f(other)74 1756 y(pro)q(cess)d(and)g(ev)o(ery)g(pair)h(of)e(pro)q(cesses)h(are)g(equally)h (distan)o(t.)22 b(W)l(e)16 b(also)g(assume)g(that)f(eac)o(h)h(pro)q(cess)74 1812 y(can)f(send)h(one)f(message)g(and,)g(sim)o(ultaneously)l(,)h(receiv)o (e)g(another)f(message)g(in)h(the)f(same)g(comm)o(uni-)74 1869 y(cation)h(step.)j(\(This)d(is)g(usually)g(done)g(using)g(a)f Fl(send-receive)e Fn(op)q(eration)j(found)f(in)h(man)o(y)f(parallel)74 1925 y(systems.\))j(In)12 b(most)e(existing)j(message-passing)e(parallel)i (systems,)e(the)h(time)f(for)g(sending)h(an)g Fj(m)p Fn(-b)o(yte)74 1982 y(message)j(from)f(pro)q(cess)h Fj(p)g Fn(to)f(pro)q(cess)h Fj(q)r Fn(,)g(without)g(congestion,)g(can)g(b)q(e)g(mo)q(deled)i(as)d Fj(T)19 b Fn(=)13 b Fj(t)1709 1989 y Fb(s)1737 1982 y Fn(+)d Fj(mt)1838 1989 y Fb(c)1856 1982 y Fn(,)74 2038 y(where)i Fj(t)218 2045 y Fb(s)249 2038 y Fn(is)g(the)g(o)o(v)o(erhead)g(\(start-up)f(time\))h (asso)q(ciated)g(with)g(eac)o(h)g(send)h(and/or)e(receiv)o(e)i(op)q(eration,) 74 2095 y(and)g Fj(t)176 2102 y Fb(c)207 2095 y Fn(is)g(the)g(comm)o (unication)h(time)f(for)f(sending)j(eac)o(h)e(additional)h(b)o(yte)f(\(or)f (an)o(y)g(appropriate)h(data)74 2151 y(unit\).)74 2239 y(Suc)o(h)25 b(a)e(fully-connected)k(mo)q(del)d(addresses)g(emerging)h(trends)f(in)g(man)o (y)g(mo)q(dern)g(distributed-)74 2295 y(memory)g(parallel)i(computers)f(and)f (message-passing)h(comm)o(unication)g(en)o(vironmen)o(ts.)48 b(These)74 2352 y(trends)16 b(are)f(eviden)o(t)i(in)g(systems)e(suc)o(h)h(as) f(Thinking)j(Mac)o(hines')d(CM-5)h([29)o(],)f(In)o(tel's)h(P)o(aragon)e([31)o (],)74 2408 y(NCUBE's)19 b(nCUBE/2)h([27)o(],)f(MIT's)g(J-Mac)o(hine)i([14)o (],)f(IBM's)f(V)l(ulcan)i([7)o(,)e(32],)h(and)f(the)h(recen)o(tly)74 2465 y(announced)e(IBM's)e(Scalable)i(PO)o(WERparallel)g(System)f(1)f (\(SP1\),)f(and)i(in)g(en)o(vironmen)o(ts)g(suc)o(h)g(as)74 2521 y(Express)g([30)o(],)g(P)l(ARMA)o(CS)h([24)o(],)e(PICL)i([22)o(],)f(Zip) q(co)q(de)i([28)o(])d(and)i(V)l(en)o(us)g([2)o(].)25 b(These)18 b(systems)e(and)74 2577 y(en)o(vironmen)o(ts)k(generally)h(ignore)f(the)g(sp) q(eci\014c)i(structure)e(and)g(top)q(ology)f(of)h(the)g(comm)o(unication)949 2727 y(13)p eop %%Page: 14 14 bop 74 157 a Fn(net)o(w)o(ork)18 b(and)h(assume)g(a)g(fully-connected)j (collection)e(of)f(pro)q(cesses,)h(in)g(whic)o(h)g(eac)o(h)f(pro)q(cess)g (can)74 214 y(comm)o(unicate)d(directly)g(with)g(an)o(y)f(other)g(pro)q(cess) g(b)o(y)g(sending)i(and)e(receiving)i(messages.)j(The)15 b(fact)74 270 y(that)j(the)h(mo)q(del)h(do)q(es)f(not)f(assume)h(an)o(y)f(single)i(top) q(ology)e(mak)o(es)g(it)h(more)g(general)g(and)g(\015exible.)74 327 y(This)14 b(mo)q(del,)f(for)g(instance,)h(allo)o(ws)f(the)f(creation)i (of)e(algorithms)h(that)f(are)g(p)q(ortable)i(b)q(et)o(w)o(een)f(di\013er-)74 383 y(en)o(t)g(mac)o(hines,)g(that)f(can)h(op)q(erate)f(within)i(arbitrary)e (and)h(dynamic)g(subsets)g(of)f(pro)q(cesses,)h(and)g(that)74 439 y(can)18 b(op)q(erate)g(in)g(the)g(presence)h(of)e(faults)h(\(assuming)f (connectivit)o(y)i(is)f(main)o(tained\).)28 b(In)19 b(addition,)74 496 y(algorithms)14 b(dev)o(elop)q(ed)i(for)e(this)h(mo)q(del)g(can)f(also)h (b)q(e)f(helpful)j(in)e(designing)h(algorithms)e(for)g(sp)q(eci\014c)74 552 y(top)q(ologies.)74 640 y(T)l(o)k(examine)h(the)g(p)q(erformance)f(of)g (comm)o(unication)h(algorithms,)g(w)o(e)f(de\014ne)h(the)f(follo)o(wing)h (three)74 696 y(complexit)o(y)d(measures:)142 776 y Fi(\017)23 b Fj(C)220 783 y Ff(1)240 776 y Fn(:)18 b(the)13 b(n)o(um)o(b)q(er)f(of)g (comm)o(unication)h(steps)f(required)h(b)o(y)f(an)g(algorithm.)19 b Fj(C)1534 783 y Ff(1)1566 776 y Fn(is)13 b(an)f(imp)q(ortan)o(t)188 833 y(measure)k(when)h(the)f(comm)o(unication)h(start-up)f(time)g(is)h(high)g (relativ)o(e)g(to)e(the)i(transfer)e(time)188 889 y(of)d(one)h(unit)g(of)f (data)g(and)h(the)f(message)g(size)i(p)q(er)f(send/receiv)o(e)h(op)q(eration) e(is)h(relativ)o(ely)h(small.)142 977 y Fi(\017)23 b Fj(C)220 984 y Ff(2)240 977 y Fn(:)j(the)18 b(amoun)o(t)g(of)f(data)h(\(in)h(the)f (appropriate)g(unit)h(of)f(comm)o(unication:)27 b(b)o(ytes,)18 b(\015its,)h(or)188 1034 y(pac)o(k)o(ets\))d(transferred)h(in)i(sequence)f(b) o(y)g(an)o(y)f(pro)q(cess.)27 b Fj(C)1217 1041 y Ff(2)1255 1034 y Fn(is)18 b(an)f(imp)q(ortan)o(t)h(measure)f(when)188 1090 y(the)e(start-up)g(time)g(is)h(small)g(compared)f(to)g(the)g(message)g (size.)142 1178 y Fi(\017)23 b Fj(C)220 1185 y Ff(3)240 1178 y Fn(:)29 b(the)20 b(total)f(amoun)o(t)g(of)g(data)g(comm)o(unicated)h(o)o(v) o(er)f(the)h(net)o(w)o(ork)e(\(in)j(the)f(appropriate)188 1235 y(unit)c(of)f(comm)o(unication:)20 b(b)o(ytes,)15 b(\015its,)g(or)g(pac)o(k)o (ets\).)k(Measures)c Fj(C)1386 1242 y Ff(1)1422 1235 y Fn(and)g Fj(C)1542 1242 y Ff(2)1577 1235 y Fn(do)h(not)e(address)188 1291 y(the)f(issue)h(of)f(load)h(on)f(the)h(net)o(w)o(ork.)k Fj(C)878 1298 y Ff(3)911 1291 y Fn(also)c(considers)g(the)f(fact)g(that)g (comm)o(unicating)h(more)188 1347 y(data)g(o)o(v)o(er)h(the)g(net)o(w)o(ork)f (causes)h(the)h(net)o(w)o(ork)e(to)g(b)q(ecome)i(more)f(congested.)74 1458 y(Th)o(us,)g(under)h(the)g(fully-connected)i(mo)q(del,)e(an)f(algorithm) h(has)f(an)g(estimated)h(time)g(complexit)o(y)g(of)74 1514 y Fj(T)k Fn(=)14 b Fj(C)202 1521 y Ff(1)222 1514 y Fj(t)238 1521 y Fb(s)267 1514 y Fn(+)d Fj(C)345 1521 y Ff(2)365 1514 y Fj(t)381 1521 y Fb(c)399 1514 y Fn(.)21 b(The)16 b(term)g Fj(C)669 1521 y Ff(3)705 1514 y Fn(do)q(es)g(not)f(a\013ect)g(the)h (complexit)o(y)h(here,)e(but)h(ma)o(y)f(b)q(e)i(helpful)h(in)74 1570 y(estimating)d(the)g(congestion)f(b)q(eha)o(vior)h(of)g(a)f(real)h (parallel)h(mac)o(hine.)k(F)l(or)14 b(instance,)h(one)g(can)g(mo)q(del)74 1627 y Fj(T)27 b Fn(=)21 b Fj(C)216 1634 y Ff(1)236 1627 y Fj(t)252 1634 y Fb(s)284 1627 y Fn(+)14 b Fj(C)365 1634 y Ff(2)385 1627 y Fj(t)401 1634 y Fb(c)432 1627 y Fn(+)g Fj(f)5 b Fn(\()p Fj(C)558 1634 y Ff(3)578 1627 y Fn(\),)20 b(where)g(the)g(function)h Fj(f)k Fn(can)20 b(b)q(e)h(deriv)o(ed)g(for)f(a)f(giv)o(en)i(mac)o(hine)g(b)o (y)74 1683 y(curv)o(e-\014tting)16 b(exp)q(erimen)o(tal)g(results.)74 1771 y(It)f(should)h(b)q(e)f(noted)g(that)f(there)h(are)f(more)h(detailed)h (comm)o(unication)g(mo)q(dels,)f(suc)o(h)g(as)f(the)h(P)o(ostal)74 1827 y(mo)q(del)f([3)o(])f(and)f(the)h(LogP)g(mo)q(del)g([13],)f(whic)o(h)i (further)e(tak)o(e)g(in)o(to)h(accoun)o(t)f(that)g(a)h(receiving)h(pro)q (cess)74 1884 y(generally)22 b(completes)f(its)f Fl(receive)g Fn(op)q(eration)g(later)h(than)f(the)g(corresp)q(onding)i(sending)f(pro)q (cess)74 1940 y(\014nishes)c(its)e Fl(send)f Fn(op)q(eration.)20 b(Ho)o(w)o(ev)o(er,)14 b(designing)i(e\016cien)o(t)g(algorithms)f(for)f (these)i(mo)q(dels)f(seems)74 1997 y(to)f(b)q(e)h(more)f(complicated.)21 b(Another)14 b(imp)q(ortan)o(t)g(issue)i(is)f(the)f(uniformit)o(y)h(of)f(the) h(implemen)o(tation.)74 2053 y(F)l(or)20 b(example,)i(in)g(the)e(LogP)g(mo)q (del,)j(the)d(collectiv)o(e)i(comm)o(unication)g(algorithms)e(are)g(designed) 74 2110 y(based)c(on)g Fj(P)23 b Fn(\(the)15 b(n)o(um)o(b)q(er)i(of)e(pro)q (cessors\).)21 b(The)c(optimal)f(algorithms)g(for)f(t)o(w)o(o)g(distinct)i(v) m(alues)g(of)74 2166 y Fj(P)23 b Fn(are)17 b(sometimes)g(v)o(ery)f (di\013eren)o(t.)25 b(This)18 b(presen)o(ts)f(a)f(c)o(hallenge)i(if)g(the)f (goal)f(is)i(to)e(supp)q(ort)h(collec-)74 2223 y(tiv)o(e)i(comm)o(unication)g (algorithms)g(for)e(v)m(arious)i(sizes)h(pro)q(cess)f(groups)f(using)h(the)g (same)f(collectiv)o(e)74 2279 y(comm)o(unication)e(library)l(.)74 2429 y Fk(4.2)56 b(T)-5 b(unable)19 b(Algorithms)74 2546 y Fn(One)12 b(goal)f(in)h(the)f(design)h(of)f(the)g(algorithms)g(for)g(CCL)g(w) o(as)f(that)h(they)g(b)q(e)h(tunable,)g(that)f(is,)h(that)e(they)74 2602 y(exhibit)17 b(a)d(trade-o\013)g(b)q(et)o(w)o(een)h(the)g(di\013eren)o (t)g(comm)o(unication)h(complexit)o(y)f(measures.)20 b(This)15 b(goal)g(is)949 2727 y(14)p eop %%Page: 15 15 bop 74 157 a Fn(imp)q(ortan)o(t)15 b(for)g(suc)o(h)h(a)f(library)h(to)f(b)q (e)h(b)q(oth)g(p)q(ortable)g(and)g(e\016cien)o(t.)21 b(In)16 b(the)g(follo)o(wing)g(discussion,)74 214 y(w)o(e)f(use)g Fj(n)g Fn(to)f(denote)h(the)f(n)o(um)o(b)q(er)h(of)g(pro)q(cesses)g(\(pro)q (cessors\))f(in)o(v)o(olv)o(ed)h(in)h(a)e(CCL)h(op)q(eration,)f(and)74 270 y(w)o(e)k(use)h Fj(m)f Fn(to)g(denote)h(the)f(size)h(of)f(data)g(eac)o(h) g(pro)q(cess)h(has)f(initially)l(.)33 b(All)19 b(CCL)g(routines)f(can)h(b)q (e)74 327 y(implemen)o(ted)g(with)e Fj(C)482 334 y Ff(1)517 327 y Fn(=)f Fi(d)p Fn(log)647 337 y Ff(2)674 327 y Fj(n)p Fi(e)h Fn(comm)o(unication)h(steps,)e(whic)o(h)i(is)f(optimal)h(for)e(this)h (measure.)74 383 y(\(The)c Fl(shift)f Fn(routine)h(can)g(b)q(e)g(implemen)o (ted)h(optimally)g(in)f(one)g(comm)o(unication)h(step,)e(b)o(y)h(using)g(the) 74 439 y Fl(send-receive)g Fn(op)q(eration.\))19 b(Ho)o(w)o(ev)o(er,)13 b(when)h(designing)i(an)e(algorithm)g(to)g(minimize)i Fj(C)1672 446 y Ff(1)1692 439 y Fn(,)e(some)g(of)74 496 y(the)j(corresp)q(onding)g (terms)f(for)g Fj(C)678 503 y Ff(2)715 496 y Fn(or)g Fj(C)804 503 y Ff(3)840 496 y Fn(ma)o(y)g(not)g(b)q(e)i(optimal.)24 b(In)17 b(general,)g(there)g(are)f(tradeo\013s)74 552 y(b)q(et)o(w)o(een)g Fj(C)282 559 y Ff(1)317 552 y Fn(and)f Fj(C)437 559 y Ff(2)458 552 y Fn(,)f(or)h(b)q(et)o(w)o(een)h Fj(C)749 559 y Ff(1)784 552 y Fn(and)f Fj(C)904 559 y Ff(3)925 552 y Fn(.)74 640 y(As)k(an)h (example,)g(consider)h(the)e Fl(index)f Fn(op)q(eration.)33 b(A)19 b(straigh)o(tforw)o(ard)e(implemen)o(tation)k(of)d(the)74 696 y Fl(index)11 b Fn(op)q(eration)g(can)g(b)q(e)h(ac)o(hiev)o(ed)g(with)g Fj(C)852 703 y Ff(1)885 696 y Fn(=)h Fj(n)r Fi(\000)r Fn(1)e(comm)o (unication)h(steps)f(and)h Fj(C)1573 703 y Ff(2)1606 696 y Fn(=)h Fj(m)p Fn(\()p Fj(n)r Fi(\000)r Fn(1\))p Fj(=n)74 753 y Fn(units)22 b(of)g(data.)38 b(This)22 b(is)h(simply)f(b)o(y)g(sending)h (the)e(data)g(directly)i(from)e(source)h(to)f(destination.)74 809 y(Namely)l(,)15 b(eac)o(h)g(pro)q(cess)g(sends)h Fj(m=n)e Fn(units)i(of)e(data)g(to)h Fj(n)9 b Fi(\000)h Fn(1)15 b(other)f(pro)q (cesses.)20 b(Another)15 b(approac)o(h)74 866 y(is)g(b)o(y)f(using)h(a)e (di\013eren)o(t)i(radix)f(for)g(represen)o(ting)g(the)g(PIDs)g(of)g(the)g (pro)q(cesses)h(\(the)f(straigh)o(tforw)o(ard)74 922 y(approac)o(h)20 b(is)g(using)g(a)g(radix)g Fj(n)g Fn(represen)o(tation\).)33 b(In)21 b(the)f(case)g(of)f(a)g(radix)h(2)g(represen)o(tation)g(w)o(e)74 979 y(get)g Fj(C)187 986 y Ff(1)228 979 y Fn(=)h Fi(d)p Fn(log)8 b Fj(n)p Fi(e)20 b Fn(comm)o(unication)h(steps)f(and)g Fj(C)1003 986 y Ff(2)1044 979 y Fi(\031)h Fn(\()p Fj(m)8 b Fn(log)g Fj(n)p Fn(\))p Fj(=)p Fn(2)19 b(units)i(of)e(data.)34 b(In)20 b(general,)74 1035 y(b)o(y)e(c)o(ho)q(osing)g(an)g(appropriate)f(radix)h Fj(r)q Fn(,)g(the)g Fl(index)f Fn(op)q(eration)h(can)f(b)q(e)i(implemen)o (ted)g(with)f Fj(C)1796 1042 y Ff(1)1833 1035 y Fi(\031)74 1092 y Fn(\()p Fj(r)8 b Fi(\000)f Fn(1\))h(log)270 1103 y Fb(r)297 1092 y Fj(n)14 b Fn(comm)o(unication)h(steps)e(and)i(with)f Fj(C)988 1099 y Ff(2)1021 1092 y Fi(\031)f Fn(\()p Fj(m)p Fn(\()p Fj(r)7 b Fi(\000)g Fn(1\))h(log)1322 1103 y Fb(r)1349 1092 y Fj(n)p Fn(\))p Fj(=r)14 b Fn(units)h(of)e(data.)19 b(Hence,)74 1148 y(it)c(is)f(p)q(ossible)i(to)e(implemen)o(t)h(a)f(parameterized)g (algorithm)h(whic)o(h)g(can)f(b)q(e)h(tuned)f(according)h(to)e(the)74 1205 y(start-up)h(time)h Fj(t)367 1212 y Fb(s)386 1205 y Fn(,)f(p)q(er-b)o (yte)h(transfer)e(time)i Fj(t)879 1212 y Fb(c)897 1205 y Fn(,)f(the)h (message)f(size)h Fj(m)p Fn(,)f(and)h(p)q(ossibly)h(the)e(n)o(um)o(b)q(er)h (of)74 1261 y(\\parallel)i(p)q(orts")d(that)h(can)g(supp)q(ort)g(concurren)o (t)h(sends)f(and)h(receiv)o(es)g(e\013ectiv)o(ely)g(\(see)f([12)o(]\).)74 1349 y(As)20 b(a)f(second)i(example,)g(consider)g(the)f(broadcasting)g (problem.)34 b(The)20 b(algorithm)g(for)f(the)h Fl(bcast)74 1405 y Fn(routine)g(is)g(straigh)o(tforw)o(ard)d(when)j(the)g(size)g(of)f (the)h(data)f(is)h Fj(m)f Fn(=)i(1,)e(that)g(is,)i(when)f(the)f(source)74 1461 y(of)e(the)g(broadcast)g(has)g(one)g(item)g(to)g(broadcast.)25 b(In)18 b(this)f(case,)h(a)e(divide-and-conque)q(r)k(algorithm)74 1518 y(pro)o(vides)15 b(an)g(optimal)g(solution,)g(also)g(kno)o(wn)g(as)f Fh(r)n(e)n(cursive)h(doubling)p Fn(.)k(Ho)o(w)o(ev)o(er,)14 b(the)h(broadcasting)74 1574 y(problem)f(b)q(ecomes)f(m)o(uc)o(h)g(more)g (complicated)h(when)f(there)g(are)g Fj(m)f(>)h Fn(1)g(units)h(of)e(data)g(to) h(broadcast.)74 1631 y(In)19 b(this)f(case,)h(a)e(lo)o(w)o(er)h(b)q(ound)h (on)f(the)g(time)g(is)h Fi(d)p Fn(log)8 b Fj(n)p Fi(e)g Fj(t)1102 1638 y Fb(s)1133 1631 y Fn(+)k(\()p Fj(m)g Fn(+)g Fi(d)p Fn(log)d Fj(n)p Fi(e)j(\000)g Fn(1\))p Fj(t)1547 1638 y Fb(c)1564 1631 y Fn(.)29 b(The)18 b(common)74 1687 y(divide-and-conquer)f(algorithm)d(based) g(on)g(a)g(binomial)i(tree)e(tak)o(es)f Fi(d)p Fn(log)c Fj(n)p Fi(e)f Fn(\()p Fj(t)1464 1694 y Fb(s)1490 1687 y Fn(+)g Fj(mt)1589 1694 y Fb(c)1607 1687 y Fn(\))13 b(time,)i(whic)o(h)74 1744 y(ma)o(y)j(b)q(e)h(far)e(from)h(optimal.)29 b(When)19 b Fj(n)f Fn(is)h(a)f(p)q(o)o(w)o(er)g(of)g(t)o(w)o(o,)f(an)h(algorithm)g(based)h(on)f (log)9 b Fj(n)18 b Fn(edge-)74 1800 y(disjoin)o(t)e(spanning)h(trees)e(on)g (a)g(\(log)8 b Fj(n)p Fn(\)-cub)q(e)16 b(is)g(giv)o(en)g(in)g([26)o(,)f(25].) 20 b(This)c(algorithm)f(requires)h Fj(C)1800 1807 y Ff(2)1833 1800 y Fn(=)74 1857 y Fj(m)c Fn(+)h(log)8 b Fj(n)13 b Fi(\000)f Fn(1,)19 b(whic)o(h)g(is)g(optimal,)h(and)e(tak)o(es)g Fj(T)24 b Fn(=)18 b(\()1081 1823 y Fi(p)p 1119 1823 74 2 v 34 x Fj(mt)1175 1864 y Fb(c)1205 1857 y Fn(+)1252 1822 y Fi(p)p 1290 1822 129 2 v 35 x Fn(log)8 b Fj(nt)1399 1864 y Fb(s)1419 1857 y Fn(\))1437 1840 y Ff(2)1474 1857 y Fn(time.)30 b(F)l(or)18 b(arbitrary)74 1913 y(v)m(alues)g(of)f Fj(n)p Fn(,)h(an)f(algorithm)g(based)g(on)g(the)g (generalized)i(Fib)q(onacci)g(trees)e(w)o(as)f(giv)o(en)h(in)h([10)o(])f (with)74 1970 y Fj(C)106 1977 y Ff(2)149 1970 y Fi(\024)24 b Fj(m)14 b Fn(+)h(log)8 b Fj(n)15 b Fn(+)f(3)8 b(log)g(log)g Fj(n)15 b Fn(+)g(15.)38 b(More)21 b(recen)o(tly)l(,)i(an)f(algorithm)f(based) h(on)f(edge-disjoin)o(t)74 2026 y(spanning)g(trees)e(of)g(cascaded)g (decreasing-size)j(h)o(yp)q(ercub)q(es)f(w)o(as)d(giv)o(en)i([4)o(])f(with)h (nearly)g(optimal)74 2082 y Fj(C)106 2089 y Ff(2)139 2082 y Fn(=)13 b Fj(m)6 b Fn(+)g Fi(d)q Fn(log)i Fj(n)p Fi(e)q Fn(.)19 b(The)13 b Fl(reduce)g Fn(op)q(eration)g(is)h(usually)h(implemen)o(ted)g(in)f (a)f(similar)h(manner)g(to)e(the)74 2139 y Fl(bcast)j Fn(op)q(eration)g(b)o (y)g(rev)o(ersing)h(the)f(\015o)o(w)g(of)f(the)i(messages.)74 2227 y(As)11 b(another)f(example,)j(consider)e(the)g Fl(combine)f Fn(and)h Fl(prefix)f Fn(op)q(erations.)19 b(These)11 b(op)q(erations)g (exhibit)74 2283 y(an)17 b(in)o(teresting)g(trade-o\013)f(b)q(et)o(w)o(een)h (measures)f Fj(C)954 2290 y Ff(1)991 2283 y Fn(and)h Fj(C)1113 2290 y Ff(3)1133 2283 y Fn(.)24 b(On)17 b(one)g(hand,)g(these)g(op)q (erations)f(can)74 2339 y(b)q(e)22 b(implemen)o(ted)h(with)f Fj(C)559 2346 y Ff(1)602 2339 y Fn(=)h(2)8 b Fi(d)p Fn(log)g Fj(n)p Fi(e)22 b Fn(and)f Fj(C)972 2346 y Ff(3)1016 2339 y Fn(=)i Fj(O)q Fn(\()p Fj(mn)p Fn(\))e(using)h(a)f(reduction)h(tree)f(follo)o (w)o(ed)74 2396 y(b)o(y)c(a)f(broadcast)f(tree.)24 b(On)17 b(the)f(other)g(hand,)h(these)g(op)q(erations)g(can)f(b)q(e)h(implemen)o(ted) h(with)f Fj(C)1798 2403 y Ff(1)1833 2396 y Fn(=)74 2452 y Fi(d)p Fn(log)9 b Fj(n)p Fi(e)i Fn(and)g Fj(C)335 2459 y Ff(3)368 2452 y Fn(=)i Fj(O)q Fn(\()p Fj(mn)8 b Fn(log)f Fj(n)p Fn(\))k(using)h(a)e (butter\015y-t)o(yp)q(e)h(or)g(circulan)o(t-graph-t)o(yp)q(e)h(comm)o (unication)74 2509 y(pattern.)41 b(In)23 b(fact,)g(h)o(ybrid)g(algorithms)f (for)f(these)i(op)q(erations)f(exist.)42 b(F)l(or)21 b(instance,)k(a)d(h)o (ybrid)74 2565 y(algorithm)c(for)f(the)h Fl(combine)f Fn(op)q(eration)h (requires)h Fj(C)1034 2572 y Ff(1)1071 2565 y Fn(=)e Fi(d)p Fn(log)9 b Fj(n)p Fi(e)j Fn(+)g Fj(k)19 b Fn(comm)o(unication)g(steps)e(and) 949 2727 y(15)p eop %%Page: 16 16 bop 74 157 a Fn(comm)o(unicates)16 b Fj(C)401 164 y Ff(3)433 157 y Fn(=)d Fj(m)534 139 y Fb(n)p 526 146 37 2 v 526 175 a Ff(2)544 166 y Fa(k)568 157 y Fn(\(log)8 b Fj(n)i Fn(+)h(2)758 141 y Fb(k)q Ff(+1)834 157 y Fi(\000)f Fj(k)i Fi(\000)e Fn(2\))15 b(units)h(of)e(data)h(\(see)g([1)o(]\).)74 245 y(The)k Fl(scatter)e Fn(and)i Fl(gather)e Fn(op)q(erations)h(resem)o(ble)h(the)g Fl(bcast)e Fn(and)i Fl(reduce)e Fn(op)q(erations)i(in)g(their)74 301 y(functionalit)o(y)l(.)h(Ho)o(w)o(ev)o(er,)12 b(in)h(terms)e(of)h(their)g (p)q(erformance,)h(these)f(op)q(erations)g(can)g(b)q(e)h(implemen)o(ted)74 358 y(with)e Fj(C)205 365 y Ff(1)238 358 y Fn(=)h Fi(d)q Fn(log)c Fj(n)p Fi(e)j Fn(comm)o(unication)f(steps)h(and)f Fj(C)966 365 y Ff(3)999 358 y Fi(\031)j Fn(\()p Fj(m)8 b Fn(log)f Fj(n)p Fn(\))p Fj(=)p Fn(2)j(units)h(of)e(data,)i(or,)f(alternativ)o(ely)l(,)74 414 y(they)j(can)h(b)q(e)g(implemen)o(ted)h(with)e Fj(C)716 421 y Ff(1)749 414 y Fn(=)g Fj(n)6 b Fi(\000)g Fn(1)14 b(comm)o(unication)g (steps)f(and)g Fj(C)1453 421 y Ff(3)1486 414 y Fn(=)g Fj(m)p Fn(\()p Fj(n)6 b Fi(\000)g Fn(1\))p Fj(=n)13 b Fn(units)74 471 y(of)i(data.)74 558 y(Finally)l(,)i(consider)g(the)f Fl(concat)f Fn(op)q(eration.)21 b(The)16 b(next)g(subsection)h(outlines)g(an)f(algorithm) f(for)g(the)74 615 y Fl(concat)j Fn(op)q(eration)g(that)g(is)h(optimal)g(in)g (all)h(the)e(three)h(complexit)o(y)g(measures,)g(hence,)h(it)e(enables)74 671 y(go)q(o)q(d)d(p)q(erformance)h(o)o(v)o(er)e(a)h(v)m(ariet)o(y)g(of)g (parallel)i(mac)o(hines.)74 824 y Fk(4.3)56 b(An)19 b(Optimal)e(Algorithm)f (for)j(Concatenation)74 941 y Fn(Here,)f(w)o(e)g(outline)g(an)g(algorithm)f (for)g(the)h Fl(concat)f Fn(op)q(eration)h(whic)o(h)g(is)g(optimal)g(with)g (resp)q(ect)g(to)74 997 y(measures)13 b Fj(C)299 1004 y Ff(1)319 997 y Fn(,)g Fj(C)377 1004 y Ff(2)411 997 y Fn(and)g Fj(C)529 1004 y Ff(3)549 997 y Fn(.)19 b(In)14 b(the)f Fl(concat)f Fn(\(all-to-all)j (broadcast\))d(op)q(eration)h(among)f Fj(n)h Fn(pro)q(cesses,)74 1054 y(eac)o(h)k(pro)q(cess)g(has)f(a)g(\014xed-size)j(message)d(\(also)g (called)i(a)f Fh(data)h(blo)n(ck\))f Fn(that)f(it)h(needs)g(to)f(broadcast)74 1110 y(to)e(the)g(other)g Fj(n)8 b Fi(\000)h Fn(1)k(pro)q(cesses.)20 b(Th)o(us,)14 b(at)g(the)g(end,)h(eac)o(h)f(pro)q(cess)h(has)f(all)h Fj(n)f Fn(data)g(blo)q(c)o(ks.)20 b(Because)74 1167 y Fj(n)c Fn(need)g(not)f(necessarily)i(b)q(e)g(a)e(p)q(o)o(w)o(er)g(of)g(t)o(w)o(o,)f (the)h(simple)i(optimal)f(algorithm)g(for)f(concatenation,)74 1223 y(whic)o(h)h(is)g(based)f(on)h(a)e(butter\015y-t)o(yp)q(e)i(comm)o (unication)g(pattern,)e(cannot)h(b)q(e)h(directly)h(used.)74 1311 y(The)i(optimal)g(algorithm)f(for)g(the)h Fl(concat)e Fn(op)q(eration)i(is)g(based)g(on)f(the)h(structure)f(of)g(a)g(circulan)o(t) 74 1367 y(graph.)h(A)14 b Fh(cir)n(culant)h(gr)n(aph)j Fj(G)p Fn(\()p Fj(n;)8 b(S)s Fn(\))k(is)j(c)o(haracterized)g(b)o(y)f(t)o(w)o(o)e (parameters:)19 b(the)14 b(n)o(um)o(b)q(er)g(of)g(no)q(des)74 1424 y Fj(n)p Fn(,)h(and)g(a)f(set)h(of)f(in)o(teger)h(o\013sets)f Fj(S)s Fn(.)19 b(The)c(no)q(des)h(of)e(the)h(graph)g(are)f(lab)q(eled)j(from) d(0)g(through)h Fj(n)10 b Fi(\000)f Fn(1.)74 1480 y(Eac)o(h)20 b(no)q(de)h Fj(i)f Fn(is)g(connected)h(to)f(no)q(des)g(of)g(the)g(form)g (\(\()p Fj(i)12 b Fi(\006)i Fj(s)p Fn(\))e(mo)q(d)h Fj(n)p Fn(\))20 b(for)g(all)h Fj(s)g Fi(2)g Fj(S)h Fn(\(see)f([17)o(]\).)74 1537 y(Circulan)o(t)f(graphs)e(are)g(an)h(imp)q(ortan)o(t)f(class)h(of)g(net) o(w)o(orks)e(whic)o(h)j(can)f(b)q(e)g(used)g(as)g(fault-toleran)o(t)74 1593 y(net)o(w)o(orks)g(for)h(man)o(y)g(other)f(net)o(w)o(orks)h([9)o(,)g(16) o(].)35 b(W)l(e)20 b(use)h(a)f(circulan)o(t)h(graph)f(with)g(p)q(o)o(w)o (er-of-t)o(w)o(o)74 1649 y(o\013sets)c Fj(S)i Fn(=)d Fi(f)p Fn(1)p Fj(;)8 b Fn(2)p Fj(;)g Fn(4)p Fj(;)g Fi(\001)f(\001)g(\001)t Fj(;)h Fn(2)566 1633 y Fb(k)q Fq(\000)p Ff(1)632 1649 y Fi(g)p Fn(,)16 b(where)h Fj(k)g Fn(=)e Fi(d)q Fn(log)8 b Fj(n)p Fi(e)p Fn(,)17 b(as)g(a)f(structure)h(for)f(the)h Fl(concat)f Fn(algorithm)74 1706 y([11)o(])f(\(see)g(Figure)h(1\).)74 1794 y(The)g(algorithm)f(consists)g (of)g Fj(k)h Fn(steps.)k(In)c(step)f(0,)g(eac)o(h)g(pro)q(cess)g Fj(i)g Fn(sends)h(its)f(data)g(blo)q(c)o(k)h(to)e(pro)q(cess)74 1850 y(\()p Fj(i)c Fi(\000)g Fn(1\))i(mo)q(d)h Fj(n)p Fn(,)i(receiv)o(es)i(a) e(data)f(blo)q(c)o(k)i(from)f(pro)q(cess)h(\()p Fj(i)9 b Fn(+)i(1\))h(mo)q(d) h Fj(n)p Fn(,)i(and)g(app)q(ends)i(the)e(receiv)o(ed)74 1906 y(data)f(blo)q(c)o(k)h(to)f(its)g(curren)o(t)g(data.)19 b(In)c(general,)g(in) g(step)g Fj(j)s Fn(,)e(for)h(0)e Fi(\024)h Fj(j)i Fi(\024)e Fj(k)c Fi(\000)g Fn(2,)14 b(eac)o(h)g(pro)q(cess)h Fj(i)f Fn(sends)74 1963 y(all)j(its)f(curren)o(t)g(cum)o(ulated)h(2)606 1946 y Fb(j)640 1963 y Fn(data)f(blo)q(c)o(ks)g(to)g(pro)q(cess)g(\()p Fj(i)10 b Fi(\000)h Fn(2)1212 1946 y Fb(j)1230 1963 y Fn(\))i(mo)q(d)f Fj(n)p Fn(,)k(receiv)o(es)h(2)1607 1946 y Fb(j)1641 1963 y Fn(data)f(blo)q(c)o(ks)74 2019 y(from)f(pro)q(cess)i(\()p Fj(i)10 b Fn(+)h(2)455 2003 y Fb(j)473 2019 y Fn(\))h(mo)q(d)h Fj(n)p Fn(,)j(and)g(app)q(ends)h(the)g(receiv)o(ed)g(data)e(to)h(its)g(cum)o(ulated) h(data)e(blo)q(c)o(ks.)74 2076 y(In)h(the)f(last)g(step,)g(step)g Fj(k)10 b Fi(\000)h Fn(1,)j(eac)o(h)h(pro)q(cess)g Fj(i)g Fn(sends)h(only)f (the)g(last)g Fj(n)10 b Fi(\000)g Fn(2)1411 2059 y Fb(k)q Fq(\000)p Ff(1)1493 2076 y Fn(blo)q(c)o(ks)15 b(of)g(data)f(that)74 2132 y(it)j(cum)o(ulated)g(to)e(pro)q(cess)h(\()p Fj(i)11 b Fi(\000)g Fn(2)672 2116 y Fb(k)q Fq(\000)p Ff(1)738 2132 y Fn(\))h(mo)q(d)h Fj(n)p Fn(,)j(and)h(it)f(receiv)o(es)h(the)f(same)g(n)o(um)o(b)q(er)h(of)e (data)h(blo)q(c)o(ks)74 2189 y(from)f(pro)q(cess)g(\()p Fj(i)10 b Fn(+)g(2)452 2172 y Fb(k)q Fq(\000)p Ff(1)518 2189 y Fn(\))i(mo)q(d)h Fj(n)p Fn(.)20 b(Figure)c(2)f(presen)o(ts)g(an)g(example)h(of)f(this)g (algorithm.)74 2276 y(In)22 b(general,)i(all)e(the)g(homogeneous)g(op)q (erations)f(\(op)q(erations)h(for)f(whic)o(h)i(there)e(is)h(no)g(notion)g(of) 74 2333 y(a)16 b(distinct)i(source)f(and/or)f(destination,)h([19)o(]\))f(in)h (CCL,)f(can)h(b)q(e)g(implemen)o(ted)h(with)f(the)g(minimal)74 2389 y(n)o(um)o(b)q(er)11 b(of)f(start-ups)g(\()p Fi(d)p Fn(log)e Fj(n)p Fi(e)j Fn(comm)o(unication)g(steps\))f(using)i(the)e(same)g(circulan)o (t)i(graph)e(structure.)74 2446 y(Examples)19 b(of)e(other)h(suc)o(h)g(op)q (erations)h(are)e Fl(combine)p Fn(,)h Fl(index)p Fn(,)f(and)i Fl(sync)p Fn(.)28 b(An)18 b(algorithm)g(for)f(the)74 2502 y Fl(combine)f Fn(op)q(eration)h(with)g Fj(T)22 b Fn(=)16 b Fi(d)p Fn(log)8 b Fj(n)p Fi(e)g Fn(\()p Fj(t)843 2509 y Fb(s)873 2502 y Fn(+)k Fj(mt)976 2509 y Fb(c)993 2502 y Fn(\),)17 b(whic)o(h)h(is)f (optimal)h(with)f(resp)q(ect)g(to)g Fj(C)1741 2509 y Ff(1)1778 2502 y Fn(\(and)74 2559 y(also)e(to)g Fj(C)253 2566 y Ff(2)288 2559 y Fn(when)h Fj(m)c Fn(=)h(1\),)i(app)q(ears)g(in)h([5)o(].)949 2727 y(16)p eop %%Page: 17 17 bop 74 157 a Fk(4.4)56 b(Sp)r(ecialized)16 b(Algorithms)74 274 y Fn(F)l(or)g(certain)g(CCL)h(op)q(erations,)f(the)g(b)q(est-kno)o(wn)g (algorithms)h(for)e(the)h(case)h(when)f Fj(n)h Fn(is)f(a)g(p)q(o)o(w)o(er)g (of)74 331 y(2)f(ma)o(y)g(p)q(erform)f(b)q(etter)i(than)f(algorithms)g(for)f (the)i(general)f(case.)20 b(Since,)d(in)f(man)o(y)e(situations,)i Fj(n)f Fn(is)74 387 y(a)h(p)q(o)o(w)o(er)f(of)g(2,)h(it)g(is)g(w)o(orth)o (while)g(to)f(implemen)o(t)j(this)e(case)g(separately)l(.)22 b(F)l(or)15 b(instance,)h(the)g Fl(concat)74 443 y Fn(algorithm)f(for)g Fj(n)g Fn(that)f(is)i(a)f(p)q(o)o(w)o(er)f(of)h(t)o(w)o(o)f(can)h(use)g(a)g (w)o(ell-kno)o(wn)h(h)o(yp)q(ercub)q(e)h(recursiv)o(e)e(exc)o(hange)74 500 y(algorithm)j(whic)o(h)h(eliminates)h(shifting)g(lo)q(cal)f(arra)o(ys)e (at)g(the)i(end)f(of)g(the)h(op)q(eration.)29 b(As)18 b(another)74 556 y(example,)f(when)g Fj(n)g Fn(is)f(a)g(p)q(o)o(w)o(er)g(of)g(t)o(w)o(o,)f (the)h Fl(bcast)g Fn(algorithm)g(describ)q(ed)j(in)e([25)o(])f(is)h(substan)o (tially)74 613 y(simpler,)23 b(in)f(terms)e(of)g(lo)q(cal)i(data)e (structures)g(and)h(con)o(trol,)g(than)g(the)f(algorithm)h(for)f(arbitrary)74 669 y(v)m(alues)d(of)d Fj(n)i Fn(describ)q(ed)h(in)f([4)o(].)74 844 y Fm(5)69 b(Conclusions)74 976 y Fn(W)l(e)16 b(ha)o(v)o(e)f(describ)q(ed) i(the)e(main)h(issues)g(that)f(w)o(e)g(ha)o(v)o(e)g(encoun)o(tered)h(in)h (designing)f(and)g(implemen)o(t-)74 1033 y(ing)i(a)e(Collectiv)o(e)j(Comm)o (unication)e(Library)g(\(CCL\))g(for)f(the)h(recen)o(tly)h(announced)g(IBM)f (Scalable)74 1089 y(PO)o(WERparalel)d(System)f(1,)g(\(SP1\).)18 b(W)l(e)13 b(ha)o(v)o(e)f(fo)q(cused)i(on)f(three)g(no)o(v)o(el)g(asp)q(ects) g(in)h(the)e(design)i(and)74 1146 y(implemen)o(tation)21 b(of)d(CCL:)h(the)g (in)o(tro)q(duction)i(of)d(pro)q(cess)i(groups,)f(de\014nition)i(of)e(seman)o (tics)g(that)74 1202 y(ensures)f(correctness)f(and)h(the)f(design)i(of)e(no)o (v)o(el)g(algorithms)h(based)g(on)f(a)g(realistic)i(p)q(oin)o(t-to-p)q(oin)o (t)74 1259 y(comm)o(unication)c(mo)q(del.)21 b(Eac)o(h)14 b(of)g(these)h(no)o (v)o(el)f(asp)q(ects)h(suggests)f(in)o(teresting)h(a)o(v)o(en)o(ues)f(for)g (further)74 1315 y(learning)j(and)e(researc)o(h.)74 1468 y Fk(Ac)n(kno)n(wledgemen)n(ts)74 1585 y Fn(W)l(e)d(thank)g(the)g(follo)o(wing) h(p)q(eople:)19 b(Dan)12 b(F)l(ry)o(e)f(for)h(his)g(v)m(aluable)i(commen)o (ts)d(and)h(help)i(in)e(revising)h(the)74 1641 y(CCL)j(seman)o(tics)f(as)h (part)f(of)g(the)g(sp)q(ec)i(in)f([21)o(],)f(Magda)g(Konstan)o(tinidou)h(for) f(her)h(con)o(tributions)g(to)74 1698 y(the)c(initial)h(stage)e(of)g(the)h (CCL)f(design,)i(Eric)f(Leu)g(for)f(his)h(commen)o(ts)f(on)h(a)f(draft)g(of)g (this)h(pap)q(er,)g(man)o(y)74 1754 y(in)o(ternal)17 b(and)f(external)h (review)o(ers)f(of)f([21])g(for)h(their)g(v)m(aluable)i(commen)o(ts)d(to)h (the)g(CCL)g(seman)o(tics,)74 1811 y(and)g(Dragutin)e(P)o(etk)o(o)o(vic)h (for)g(his)h(constan)o(t)e(supp)q(ort)h(and)h(help.)949 2727 y(17)p eop %%Page: 18 18 bop 74 157 a Fm(App)r(endix:)30 b(List)23 b(of)g(CCL)f(routines)74 290 y Fn(CCL)14 b(consists)f(of)g(t)o(w)o(o)f(parts,)h(the)h(collectiv)o(e)h (comm)o(unication)f(routines)g(\()p Fl(CC)f Fn(part\))f(and)i(the)f(pro)q (cess)74 346 y(group)k(routines)h(\()p Fl(PG)e Fn(part)g(whic)o(h)i(is)g (called)h Fl(TG)e Fn(for)f(task)g(group)h(in)h(the)f(EUI)h(do)q(cumen)o(t.\)) 25 b(The)18 b Fl(CC)74 403 y Fn(routines)d(pro)o(vide)g(users)g(with)f(the)h (functionalit)o(y)g(of)f(op)q(erations)h(in)o(v)o(olving)h(collectiv)o(e)g (comm)o(unica-)74 459 y(tion.)23 b(The)17 b Fl(PG)f Fn(routines)g(pro)o(vide) h(users)f(with)h(the)f(functionalit)o(y)i(of)d(sp)q(ecifying)k(and)d (manipulating)74 516 y(groups.)74 603 y(The)g(a)o(v)m(ailable)g Fl(CC)f Fn(routines)h(are:)74 691 y Fl(bcast\(\))p Fn(:)23 b(Single)c(source)e(broadcast,)f(i.e.,)h(broadcast)g(a)f(message)h(from)f (one)i(pro)q(cess)f(to)f(all)i(tasks)74 747 y(in)e(a)f(group.)74 835 y Fl(reduce\(\))p Fn(:)k(Apply)c(an)g(asso)q(ciativ)o(e)g(\(but)f(not)g (necessarily)i(comm)o(utativ)o(e\))d(reduction)j(op)q(eration)f(on)74 891 y(all)h(the)g(pro)q(cesses)f(in)h(a)f(group,)g(and)g(place)h(the)f (reduction)i(result)e(in)h(one)g(user-sp)q(eci\014ed)h(pro)q(cess.)74 979 y Fl(combine\(\))p Fn(:)25 b(Apply)20 b(an)e(asso)q(ciativ)o(e)h(\(but)f (not)h(necessarily)g(comm)o(utativ)o(e\))f(reduction)h(op)q(eration)74 1035 y(on)e(all)h(the)f(pro)q(cesses)g(in)h(a)e(group,)h(and)g(place)h(the)f (reduction)h(result)f(in)h(eac)o(h)f(of)f(the)h(pro)q(cesses)h(in)74 1092 y(the)d(group.)74 1179 y Fl(scatter\(\))p Fn(:)k(Distribute)d(distinct)g (messages)f(from)f(a)h(single)i(source)e(to)f(eac)o(h)i(pro)q(cess)f(in)h(a)f (group.)74 1267 y Fl(gather\(\))p Fn(:)23 b(Gather)17 b(distinct)i(messages)e (from)f(eac)o(h)i(pro)q(cess)f(in)i(a)e(group)g(to)g(a)g(single)i (destination)74 1323 y(pro)q(cess.)74 1411 y Fl(concat\(\))p Fn(:)25 b(Concatenate)18 b(to)f(all)i(pro)q(cesses)g(in)g(a)f(group.)29 b(Eac)o(h)18 b(pro)q(cess)g(in)h(a)f(group)g(p)q(erforms)g(a)74 1467 y Fl(bcast)d Fn(within)h(the)f(group.)74 1555 y Fl(index\(\))p Fn(:)k(Eac)o(h)c(task)g(in)h(a)e(group)h(p)q(erforms)g(a)g Fl(scatter)f Fn(op)q(eration.)74 1643 y Fl(prefix\(\))p Fn(:)19 b(P)o(arallel)d(pre\014x)g(op)q(eration.)k(It)15 b(is)h(sometimes)f(called)i Fl(scan)p Fn(.)74 1730 y Fl(shift\(\))p Fn(:)i(Shift)d(data)e(up)i(or)f(do)o (wn)g(some)g(n)o(um)o(b)q(er)g(of)g(steps)g(in)h(a)f(group.)74 1818 y Fl(sync\(\))p Fn(:)k(Barrier)c(sync)o(hronization)i(in)f(a)f(group.)74 1905 y(The)h Fl(PG)e Fn(routines)i(are)f(summarized)h(b)q(elo)o(w.)74 1993 y Fl(group\(\))p Fn(:)h(Create)11 b(a)g(\(static\))g(pro)q(cess)h(group) f(b)o(y)h(explicitly)i(sp)q(ecifying)g(the)d(pro)q(cesses)h(participating)74 2049 y(in)k(the)f(group.)74 2137 y Fl(partition\(\))p Fn(:)23 b(De\014ne)c(a)e(\(dynamic\))h(pro)q(cess)g(group)f(b)o(y)h(partitioning)h (an)e(existing)i(group)e(based)74 2193 y(on)e(a)g(lo)q(cally)i(supplied)h(in) o(teger)d(lab)q(el.)74 2281 y Fl(getsize\(\))p Fn(:)k(Get)14 b(the)i(size)g(\(the)f(n)o(um)o(b)q(er)g(of)g(pro)q(cesses\))g(of)g(an)g (existing)h(pro)q(cess)g(group.)74 2369 y Fl(getmembers\(\))p Fn(:)i(Get)d(the)g(ordered)h(arra)o(y)e(of)g(pro)q(cess)i(ids)g(of)f(an)g (existing)h(pro)q(cess)f(group.)74 2456 y Fl(getrank\(\))p Fn(:)k(Get)14 b(the)i(rank)f(of)f(a)h(pro)q(cess)h(in)g(a)f(pro)q(cess)g (group.)74 2544 y Fl(getpid\(\))p Fn(:)k(Get)c(the)g(pro)q(cess)g(id)h(of)f (the)g(pro)q(cess)h(with)f(a)g(certain)h(rank)f(in)h(a)f(pro)q(cess)g(group.) 949 2727 y(18)p eop %%Page: 19 19 bop 74 157 a Fl(getlabel\(\))p Fn(:)20 b(Get)c(the)g(lab)q(el)i(\(whic)o(h)e (w)o(as)g(supplied)i(b)o(y)e(a)g(user)g(when)g(the)h(group)e(w)o(as)h (formed\))f(of)74 214 y(an)g(existing)h(pro)q(cess)g(group.)949 2727 y(19)p eop %%Page: 20 20 bop 74 157 a Fm(References)97 277 y Fn([1])22 b(A.)14 b(Aggarw)o(al)g(and)g (S.)h(Kipnis,)h(\\Message-Time)f(T)l(radeo\013)f(for)g(Com)o(bining)h(Data)f (in)h(Message-)168 334 y(P)o(assing)g(Systems",)f Fh(IBM)i(R)n(ese)n(ar)n(ch) f(R)n(ep)n(ort)p Fn(,)g(R)o(C-18349,)e(Septem)o(b)q(er)j(1992.)97 428 y([2])22 b(V.)d(Bala)h(and)f(S.)h(Kipnis,)i(\\Pro)q(cess)d(Groups:)29 b(a)19 b(mec)o(hanism)h(for)f(the)h(co)q(ordination)g(of)f(and)168 484 y(comm)o(unication)c(among)e(pro)q(cesses)i(in)g(the)g(V)l(en)o(us)g (collectiv)o(e)h(comm)o(unication)f(library",)f Fh(Pr)n(o-)168 541 y(c)n(e)n(e)n(dings)g(of)i(the)h(7th)g(International)e(Par)n(al)r(lel)g (Pr)n(o)n(c)n(essing)f(Symp)n(osium)p Fn(,)h(IEEE,)g(April)h(1993.)97 634 y([3])22 b(A.)d(Bar-No)o(y)f(and)i(S.)f(Kipnis,)j(\\Designing)e (broadcasting)f(algorithms)h(in)g(the)f(p)q(ostal)h(mo)q(del)168 691 y(for)f(message-passing)g(systems",)h(to)f(app)q(ear)h(in)h Fh(Mathematic)n(al)f(Systems)f(The)n(ory)p Fn(.)g(Also)h(ap-)168 747 y(p)q(eared)15 b(in)g Fh(Pr)n(o)n(c)n(e)n(e)n(dings)e(of)j(the)g(4th)g(A) o(nnual)f(A)o(CM)f(Symp)n(osium)i(on)f(Par)n(al)r(lel)g(A)o(lgorithms)h(and) 168 804 y(A)o(r)n(chite)n(ctur)n(es)p Fn(,)d(June)k(1992,)c(pp.)j(11{22.)97 898 y([4])22 b(A.)34 b(Bar-No)o(y)g(and)h(S.)f(Kipnis,)41 b(\\Broadcasting)35 b(m)o(ultiple)h(messages)e(in)i(sim)o(ultaneous)168 954 y(send/receiv)o(e)18 b(systems",)e(to)g(app)q(ear)h(in)h Fh(Discr)n(ete)f(Applie)n(d)h (Mathematics)p Fn(.)e(Also)h(app)q(eared)h(as)168 1010 y Fh(IBM)d(R)n(ese)n (ar)n(ch)h(R)n(ep)n(ort)p Fn(,)e(R)o(C-18352,)g(Septem)o(b)q(er)i(1992.)97 1104 y([5])22 b(A.)15 b(Bar-No)o(y)l(,)f(S.)i(Kipnis,)h(and)e(B.)g(Sc)o(hieb) q(er,)i(\\An)f(optimal)g(algorithm)f(for)g(computing)h(census)168 1161 y(functions)h(in)g(message-passing)g(systems",)e(to)h(app)q(ear)h(in)g Fh(Par)n(al)r(lel)g(Pr)n(o)n(c)n(essing)e(L)n(etters)p Fn(.)g(Also)168 1217 y(app)q(eared)g(as)g Fh(IBM)h(R)n(ese)n(ar)n(ch)f(R)n(ep)n(ort)p Fn(,)g(R)o(C-18527,)f(No)o(v)o(em)o(b)q(er)g(1992.)97 1311 y([6])22 b(A.)17 b(Beguelin,)i(J.)f(Dongarra,)e(A.)h(Geist,)g(R.)h(Manc)o (hek,)f(and)h(V.)f(Sunderam,)h(\\A)f(user's)g(guide)168 1367 y(to)e(PVM)g(P)o(arallel)i(Virtual)f(Mac)o(hine",)g Fh(ORNL)g(T)m(e)n(chnic)n (al)e(R)n(ep)n(ort)p Fn(,)i(ORNL/TM-11826,)e(Ma)o(y)168 1424 y(1992.)97 1518 y([7])22 b(J.)16 b(Bruc)o(k,)g(R.)g(Cypher,)g(L.)g(Gra)o(v)m (ano,)f(A.)h(Ho,)f(C.)h(T.)f(Ho,)h(S.)g(Kipnis,)h(S.)f(Konstan)o(tinidou,)h (M.)168 1574 y(Snir)g(and)g(E.)g(Upfal,)g(\\Surv)o(ey)g(of)f(routing)h (issues)g(for)f(the)h(V)l(ulcan)h(parallel)h(computer",)d Fh(IBM)168 1631 y(R)n(ese)n(ar)n(ch)f(R)n(ep)n(ort)p Fn(,)g(RJ-8839,)g(June)h(1992.)97 1724 y([8])22 b(J.)c(Bruc)o(k,)h(R.)g(Cypher,)g(P)l(.)f(Elustondo,)i(A.)e (Ho,)h(and)f(C.T.)g(Ho,)h(\\A)f(Prop)q(osal)g(for)g(Common)168 1781 y(Group)13 b(Structures)g(in)i(a)e(Collectiv)o(e)i(Comm)o(unication)e (Library",)h Fh(IBM)g(R)n(ese)n(ar)n(ch)g(R)n(ep)n(ort)p Fn(,)g(RJ-)168 1837 y(9421,)f(Marc)o(h)i(1993.)97 1931 y([9])22 b(J.)g(Bruc)o(k,)i(R.)e (Cypher,)i(and)f(C.T.)e(Ho,)i(\\E\016cien)o(t)g(fault-toleran)o(t)f(mesh)g (and)h(h)o(yp)q(ercub)q(es)168 1988 y(arc)o(hitectures",)g Fh(Pr)n(o)n(c)n(e)n(e)n(dings)d(of)j(the)g(1992)g(International)e(Symp)n (osium)i(on)f(F)m(ault-T)m(oler)n(ant)168 2044 y(Computing)p Fn(,)14 b(pp.)i(162{169.)74 2138 y([10])22 b(J.)16 b(Bruc)o(k,)h(R.)g (Cypher,)g(and)g(C.T.)f(Ho,)g(\\Multiple)j(message)d(broadcasting)h(with)g (generalized)168 2194 y(Fib)q(onacci)f(trees",)e Fh(Pr)n(o)n(c)n(e)n(e)n (dings)f(of)j(the)g(4th)g(IEEE)f(Symp)n(osium)h(on)f(Par)n(al)r(lel)g(and)h (Distribute)n(d)168 2251 y(Pr)n(o)n(c)n(essing)p Fn(,)c(Decem)o(b)q(er)k (1992,)d(pp.)j(424{431.)74 2345 y([11])22 b(J.)16 b(Bruc)o(k,)g(C.T.)f(Ho,)g (and)h(S.)g(Kipnis,)i(\\Concatenating)e(data)f(optimally)i(in)g (message-passing)168 2401 y(systems",)d Fh(IBM)h(R)n(ese)n(ar)n(ch)h(R)n(ep)n (ort)p Fn(,)e(RJ-9191,)h(Jan)o(uary)g(1993.)74 2495 y([12])22 b(J.)g(Bruc)o(k,)i(R.)e(Cypher,)i(C.T.)d(Ho,)i(and)g(S.)f(Kipnis,)k (\\E\016cien)o(t)c(algorithms)g(for)g(the)g(index)168 2551 y(op)q(eration)15 b(in)h(message-passing)f(systems",)f Fh(IBM)i(R)n(ese)n(ar) n(ch)f(R)n(ep)n(ort)p Fn(,)g(in)h(preparation.)949 2727 y(20)p eop %%Page: 21 21 bop 74 157 a Fn([13])22 b(D.)16 b(Culler,)k(R.)d(Karp,)h(D.)f(P)o(atterson,)g (A.)g(Saha)o(y)l(,)g(K.E.)h(Sc)o(hauser,)g(E.)f(San)o(tos,)g(R.)h(Subramo-) 168 214 y(nian,)g(and)g(T.)e(v)o(on)i(Eic)o(k)o(en,)g(\\LogP:)e(to)o(w)o (ards)g(a)h(realistic)i(mo)q(del)f(of)f(parallel)i(computation",)168 270 y Fh(Pr)n(o)n(c)n(e)n(e)n(dings)c(of)k(the)f(4th)h(SIGPLAN)d(Symp)n (osium)j(on)f(Principles)f(and)h(Pr)n(actic)n(es)f(of)h(Par)n(al)r(lel)168 327 y(Pr)n(o)n(gr)n(amming)p Fn(,)c(A)o(CM,)g(Ma)o(y)g(1993.)74 420 y([14])22 b(W.)14 b(J.)h(Dally)l(,)g(A.)g(Chien,)g(S.)g(Fisk)o(e,)g(W.)f (Horw)o(at,)f(J.)i(Keen,)h(M.)e(Lariv)o(ee,)h(R.)g(Lethin,)h(P)l(.)f(Nuth,) 168 477 y(S.)d(Wills,)j(P)l(.)d(Carric)o(k,)h(and)g(G.)f(Fyler)h(\\The)g (J-Mac)o(hine:)20 b(a)12 b(\014ne-grain)i(concurren)o(t)f(computer",)168 533 y Fh(Information)j(Pr)n(o)n(c)n(essing)d(89)p Fn(,)j(Elsevier)g(Science)h (Publishers,)g(IFIP)l(,)e(1989,)f(pp.)h(1147{1153.)74 627 y([15])22 b(J.)11 b(Dongarra,)g(R.)h(Hemp)q(el,)i(A.)d(Hey)l(,)i(and)f(D.)f(W)l(alk)o (er,)h(\\A)g(Prop)q(osal)g(for)f(a)h(user-lev)o(el,)i(message-)168 684 y(passing)23 b(in)o(terface)g(in)h(a)f(distributed)h(memory)e(en)o (vironmen)o(t",)j Fh(ORNL)e(T)m(e)n(chnic)n(al)e(R)n(ep)n(ort)p Fn(,)168 740 y(ORNL/TM-12231,)14 b(Octob)q(er)h(1992.)74 834 y([16])22 b(S.)f(Dutt)f(and)i(J.)f(P)l(.)g(Ha)o(y)o(es,)h(\\Designing)g (fault-toleran)o(t)f(systems)g(using)h(automorphisms",)168 890 y Fh(Journal)16 b(of)g(Par)n(al)r(lel)g(and)g(Distribute)n(d)h(Computing) p Fn(,)d(V)l(ol.)i(12,)e(1991,)g(pp.)h(249{268.)74 984 y([17])22 b(B.)11 b(Elspas)i(and)f(J.)g(T)l(urner,)h(\\Graphs)e(with)i(circulan)o(t)g (adjacency)f(matrices",)g Fh(Journal)i(of)f(Com-)168 1041 y(binatorial)j(The) n(ory)p Fn(,)e(No.)h(9,)f(1970,)g(pp.)h(297{307.)74 1134 y([18])22 b(B.)h(G.)f(Fitc)o(h)i(and)f(M.)g(E.)g(Giampapa,)h(\\The)g(V)l(ulcan)g(Op)q (eration)h(En)o(vironmen)o(t:)36 b(a)23 b(brief)168 1191 y(o)o(v)o(erview)14 b(and)h(status)e(rep)q(ort",)h Fh(Pr)n(o)n(c)n(e)n(e)n(dings)f(of)j(the)g (5th)g(Workshop)g(on)g(Use)f(of)h(Par)n(al)r(lel)e(Pr)n(o-)168 1247 y(c)n(essors)h(in)g(Mete)n(or)n(olo)n(gy)p Fn(,)f(ECMWF,)g(No)o(v)o(em)o (b)q(er)h(1992.)74 1341 y([19])22 b(G.)13 b(F)l(o)o(x)h(and)h(W.)e(F)l (urmanski,)i(\\Optimal)g(comm)o(unication)g(algorithms)g(for)e(regular)i (decomp)q(o-)168 1398 y(sitions)e(on)f(the)g(h)o(yp)q(ercub)q(e",)i Fh(Pr)n(o)n(c)n(e)n(e)n(dings)e(of)h(the)h(3r)n(d)g(Confer)n(enc)n(e)e(on)h (Hyp)n(er)n(cub)n(e)g(Concurr)n(ent)168 1454 y(Computers)j(and)g(Applic)n (ations)p Fn(,)e(A)o(CM,)g(1988,)g(pp.)h(648{713.)74 1548 y([20])22 b(G.)17 b(F)l(o)o(x,)g(M.)g(Johnson,)i(G.)e(Lyzenga,)h(S.)g(Otto,)g(J.)f (Salmon,)i(and)f(D.)f(W)l(alk)o(er,)h Fh(Solving)f(Pr)n(ob-)168 1604 y(lems)e(on)g(Concurr)n(ent)g(Pr)n(o)n(c)n(essors,)f(V)m(olume)h(I:)g (Gener)n(al)g(T)m(e)n(chniques)f(and)h(R)n(e)n(gular)h(Pr)n(oblems)p Fn(,)168 1661 y(Pren)o(tice-Hall,)g(Englew)o(o)q(o)q(d)g(Cli\013s,)f(New)g (Jersey)l(,)h(1988.)74 1755 y([21])22 b(D.)c(F)l(ry)o(e,)h(R.)f(Bry)o(an)o (t,)h(C.T.)f(Ho,)h(P)l(.)f(de)h(Jong,)h(R.)f(La)o(wrence,)g(and)g(M.)f(Snir,) j(\\An)d(External)168 1811 y(User)c(In)o(terface)g(for)f(Scalable)j(P)o (arallel)f(Systems:)k(F)o(OR)l(TRAN)c(In)o(terface",)f Fh(T)m(e)n(chnic)n(al) f(R)n(ep)n(ort)p Fn(,)168 1867 y(IBM)i(Highly)h(P)o(arallel)h(Sup)q (ercomputing)f(Systems)f(Lab)q(oratory)l(,)g(No)o(v)o(em)o(b)q(er)f(1992.)74 1961 y([22])22 b(G.)g(A.)h(Geist,)h(M.)f(T.)f(Heath,)j(B.)e(W.)f(P)o(eyton,)i (and)g(P)l(.)e(H.)h(W)l(orley)l(,)i(\\A)e(user's)g(guide)h(to)168 2018 y(PICL:)c(a)h(P)o(ortable)f(Instrumen)o(ted)h(Comm)o(unication)g (Library",)g Fh(ORNL)g(T)m(e)n(chnic)n(al)e(R)n(ep)n(ort)p Fn(,)168 2074 y(ORNL/TM-11616,)14 b(Octob)q(er)h(1990.)74 2168 y([23])22 b(M.)11 b(E.)g(Giampapa,)h(B.)g(G.)f(Fitc)o(h,)h(G.)f(R.)h(Irwin,)h (and)f(D.)f(G.)g(Shea,)i(\\V)l(ulcan)g(op)q(erating)f(en)o(viron-)168 2224 y(men)o(t:)27 b(programmer's)17 b(reference",)j Fh(Internal)e(Memor)n (andum)p Fn(,)i(IBM)g(T.J.)e(W)l(atson)g(Researc)o(h)168 2281 y(Cen)o(ter,)c(Marc)o(h)h(1991.)74 2375 y([24])22 b(R.)d(Hemp)q(el,)i(\\The)d (ANL/GMD)h(macros)f(\(P)l(ARMA)o(CS\))g(in)i(F)o(OR)l(TRAN)g(for)e(p)q (ortable)i(par-)168 2431 y(allel)h(programming)e(using)h(the)f(message)g (passing)h(programming)f(mo)q(del,)i(user's)e(guide)h(and)168 2488 y(reference)c(man)o(ual",)g Fh(T)m(e)n(chnic)n(al)e(Memor)n(andum)p Fn(,)j(Gesellsc)o(haft)f(f)q(\177)-24 b(ur)16 b(Mathematik)f(und)i(Daten-)168 2544 y(v)o(erab)q(eitung)e(m)o(bH,)g(W)l(est)g(German)o(y)l(.)949 2727 y(21)p eop %%Page: 22 22 bop 74 157 a Fn([25])22 b(C.T.)e(Ho,)j(\\Optimal)g(broadcasting)f(on)g(SIMD)g (h)o(yp)q(ercub)q(es)h(without)f(indirect)i(addressing)168 214 y(capabilit)o(y",)c Fh(Journal)f(of)h(Par)n(al)r(lel)e(and)i(Distribute)n (d)f(Computing)p Fn(,)g(V)l(ol.)g(13,)f(No.)g(2,)h(Octob)q(er)168 270 y(1991,)13 b(pp.)j(246{255.)74 364 y([26])22 b(S.)13 b(L.)h(Johnsson)g (and)g(C.T.)f(Ho,)g(\\Spanning)i(graphs)e(for)h(optim)o(um)f(broadcasting)h (and)g(p)q(erson-)168 420 y(alized)19 b(comm)o(unication)g(in)g(h)o(yp)q (ercub)q(es",)g Fh(IEEE)f(T)m(r)n(ansactions)f(on)i(Computers)p Fn(,)f(V)l(ol.)g(C-38,)168 477 y(No.)c(9,)h(Septem)o(b)q(er)h(1989,)d(pp)j (1249{1268.)74 571 y([27])22 b(J.)f(F.)g(P)o(almer)g(\\The)h(NCUBE)f(family)h (of)f(parallel)i(sup)q(ercomputers",)g Fh(Pr)n(o)n(c)n(e)n(e)n(dings)d(of)i (the)168 627 y(International)15 b(Confer)n(enc)n(e)f(on)i(Computer)h(Design)p Fn(,)d(IEEE,)h(1986.)74 721 y([28])22 b(A.)e(Skjellum)i(and)e(A.)g(P)l(.)g (Leung,)i(\\Zip)q(co)q(de:)32 b(a)20 b(p)q(ortable)h(m)o(ulticomputer)g(comm) o(unication)168 777 y(library)e(atop)e(the)h(Reactiv)o(e)h(Kernel",)h Fh(Pr)n(o)n(c)n(e)n(e)n(dings)d(of)i(the)g(5th)h(Distribute)n(d)f(Memory)g (Com-)168 834 y(puting)d(Confer)n(enc)n(e)p Fn(,)d(IEEE,)i(April)h(1990,)e (pp.)h(328{337.)74 928 y([29])22 b Fh(Conne)n(ction)j(Machine)h(CM-5)h(T)m(e) n(chnic)n(al)d(Summary)p Fn(,)30 b(Thinking)e(Mac)o(hines)g(Corp)q(oration,) 168 984 y(1991.)74 1078 y([30])22 b Fh(Expr)n(ess)15 b(3.0)i(Intr)n(o)n (ductory)f(Guide)p Fn(,)f(P)o(arasoft)f(Corp)q(oration,)g(1990.)74 1172 y([31])22 b Fh(Par)n(agon)16 b(XP/S)f(Overview)p Fn(,)g(In)o(tel)h(Corp) q(oration,)e(1991.)74 1266 y([32])22 b(\\V)l(ulcan)12 b(system)e(summary",)h Fh(Internal)f(Memor)n(andum)p Fn(,)i(IBM)f(T.J.)g(W)l(atson)f(Researc)o(h)h (Cen)o(ter.)949 2727 y(22)p eop %%Page: 23 23 bop 353 679 a 19537183 19800310 7762247 25786449 27299430 45586759 startTexFig 353 679 a %%BeginDocument: Figs/circ16.ps 50 dict begin /arrowHeight 8 def /arrowWidth 4 def /none null def /numGraphicParameters 17 def /stringLimit 65535 def /Begin { save numGraphicParameters dict begin } def /End { end restore } def /SetB { dup type /nulltype eq { pop false /brushRightArrow idef false /brushLeftArrow idef true /brushNone idef } { /brushDashOffset idef /brushDashArray idef 0 ne /brushRightArrow idef 0 ne /brushLeftArrow idef /brushWidth idef false /brushNone idef } ifelse } def /SetCFg { /fgblue idef /fggreen idef /fgred idef } def /SetCBg { /bgblue idef /bggreen idef /bgred idef } def /SetF { /printSize idef /printFont idef } def /SetP { dup type /nulltype eq { pop true /patternNone idef } { /patternGrayLevel idef patternGrayLevel -1 eq { /patternString idef } if false /patternNone idef } ifelse } def /BSpl { 0 begin storexyn newpath n 1 gt { 0 0 0 0 0 0 1 1 true subspline n 2 gt { 0 0 0 0 1 1 2 2 false subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 2 copy false subspline } if n 2 sub dup n 1 sub dup 2 copy 2 copy false subspline patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Circ { newpath 0 360 arc patternNone not { ifill } if brushNone not { istroke } if } def /CBSpl { 0 begin dup 2 gt { storexyn newpath n 1 sub dup 0 0 1 1 2 2 true subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 0 0 false subspline n 2 sub dup n 1 sub dup 0 0 1 1 false subspline patternNone not { ifill } if brushNone not { istroke } if } { Poly } ifelse end } dup 0 4 dict put def /Elli { 0 begin newpath 4 2 roll translate scale 0 0 1 0 360 arc patternNone not { ifill } if brushNone not { istroke } if end } dup 0 1 dict put def /Line { 0 begin 2 storexyn newpath x 0 get y 0 get moveto x 1 get y 1 get lineto brushNone not { istroke } if 0 0 1 1 leftarrow 0 0 1 1 rightarrow end } dup 0 4 dict put def /MLine { 0 begin storexyn newpath n 1 gt { x 0 get y 0 get moveto 1 1 n 1 sub { /i exch def x i get y i get lineto } for patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Poly { 3 1 roll newpath moveto -1 add { lineto } repeat closepath patternNone not { ifill } if brushNone not { istroke } if } def /Rect { 0 begin /t exch def /r exch def /b exch def /l exch def newpath l b moveto l t lineto r t lineto r b lineto closepath patternNone not { ifill } if brushNone not { istroke } if end } dup 0 4 dict put def /Text { ishow } def /idef { dup where { pop pop pop } { exch def } ifelse } def /ifill { 0 begin gsave patternGrayLevel -1 ne { fgred bgred fgred sub patternGrayLevel mul add fggreen bggreen fggreen sub patternGrayLevel mul add fgblue bgblue fgblue sub patternGrayLevel mul add setrgbcolor eofill } { eoclip originalCTM setmatrix pathbbox /t exch def /r exch def /b exch def /l exch def /w r l sub ceiling cvi def /h t b sub ceiling cvi def /imageByteWidth w 8 div ceiling cvi def /imageHeight h def bgred bggreen bgblue setrgbcolor eofill fgred fggreen fgblue setrgbcolor w 0 gt h 0 gt and { l b translate w h scale w h true [w 0 0 h neg 0 h] { patternproc } imagemask } if } ifelse grestore end } dup 0 8 dict put def /istroke { gsave brushDashOffset -1 eq { [] 0 setdash 1 setgray } { brushDashArray brushDashOffset setdash fgred fggreen fgblue setrgbcolor } ifelse brushWidth setlinewidth originalCTM setmatrix stroke grestore } def /ishow { 0 begin gsave fgred fggreen fgblue setrgbcolor /fontDict printFont findfont printSize scalefont dup setfont def /descender fontDict begin 0 [FontBBox] 1 get FontMatrix end transform exch pop def /vertoffset 0 descender sub printSize sub printFont /Courier ne printFont /Courier-Bold ne and { 1 add } if def { 0 vertoffset moveto show /vertoffset vertoffset printSize sub def } forall grestore end } dup 0 3 dict put def /patternproc { 0 begin /patternByteLength patternString length def /patternHeight patternByteLength 8 mul sqrt cvi def /patternWidth patternHeight def /patternByteWidth patternWidth 8 idiv def /imageByteMaxLength imageByteWidth imageHeight mul stringLimit patternByteWidth sub min def /imageMaxHeight imageByteMaxLength imageByteWidth idiv patternHeight idiv patternHeight mul patternHeight max def /imageHeight imageHeight imageMaxHeight sub store /imageString imageByteWidth imageMaxHeight mul patternByteWidth add string def 0 1 imageMaxHeight 1 sub { /y exch def /patternRow y patternByteWidth mul patternByteLength mod def /patternRowString patternString patternRow patternByteWidth getinterval def /imageRow y imageByteWidth mul def 0 patternByteWidth imageByteWidth 1 sub { /x exch def imageString imageRow x add patternRowString putinterval } for } for imageString end } dup 0 12 dict put def /min { dup 3 2 roll dup 4 3 roll lt { exch } if pop } def /max { dup 3 2 roll dup 4 3 roll gt { exch } if pop } def /arrowhead { 0 begin transform originalCTM itransform /taily exch def /tailx exch def transform originalCTM itransform /tipy exch def /tipx exch def /dy tipy taily sub def /dx tipx tailx sub def /angle dx 0 ne dy 0 ne or { dy dx atan } { 90 } ifelse def gsave originalCTM setmatrix tipx tipy translate angle rotate newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath patternNone not { originalCTM setmatrix /padtip arrowHeight 2 exp 0.25 arrowWidth 2 exp mul add sqrt brushWidth mul arrowWidth div def /padtail brushWidth 2 div def tipx tipy translate angle rotate padtip 0 translate arrowHeight padtip add padtail add arrowHeight div dup scale arrowheadpath ifill } if brushNone not { originalCTM setmatrix tipx tipy translate angle rotate arrowheadpath istroke } if grestore end } dup 0 9 dict put def /arrowheadpath { newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath } def /leftarrow { 0 begin y exch get /taily exch def x exch get /tailx exch def y exch get /tipy exch def x exch get /tipx exch def brushLeftArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /rightarrow { 0 begin y exch get /tipy exch def x exch get /tipx exch def y exch get /taily exch def x exch get /tailx exch def brushRightArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /midpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 x1 add 2 div y0 y1 add 2 div end } dup 0 4 dict put def /thirdpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 2 mul x1 add 3 div y0 2 mul y1 add 3 div end } dup 0 4 dict put def /subspline { 0 begin /movetoNeeded exch def y exch get /y3 exch def x exch get /x3 exch def y exch get /y2 exch def x exch get /x2 exch def y exch get /y1 exch def x exch get /x1 exch def y exch get /y0 exch def x exch get /x0 exch def x1 y1 x2 y2 thirdpoint /p1y exch def /p1x exch def x2 y2 x1 y1 thirdpoint /p2y exch def /p2x exch def x1 y1 x0 y0 thirdpoint p1x p1y midpoint /p0y exch def /p0x exch def x2 y2 x3 y3 thirdpoint p2x p2y midpoint /p3y exch def /p3x exch def movetoNeeded { p0x p0y moveto } if p1x p1y p2x p2y p3x p3y curveto end } dup 0 17 dict put def /storexyn { /n exch def /y n array def /x n array def n 1 sub -1 0 { /i exch def y i 3 2 roll put x i 3 2 roll put } for } def Begin [ 0.9 0 0 0.9 0 0 ] concat /originalCTM matrix currentmatrix def Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -31.2139 101.893 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 110.786 -40.107 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 70.107 61.2139 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 26.107 89.3209 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 99.3209 15 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 -31.2139 1101.89 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 70.107 1142.57 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 26.107 1114.47 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 -1 99.3209 1188.79 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 490.786 -40.107 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 531.465 61.2139 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 575.465 89.3209 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 1 502.251 15 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 -1 531.465 1142.57 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 -1 575.465 1114.47 ] concat 332 642 4 4 Elli End Begin %I Elli 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ -1 0 0 -1 502.251 1188.79 ] concat 332 642 4 4 Elli End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 296.428 762 ] concat [ (0) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 363 744 ] concat [ (1) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 407 717 ] concat [ (2) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 435.107 671.893 ] concat [ (3) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 452 606.572 ] concat [ (4) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 443 549 ] concat [ (5) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 407 496 ] concat [ (6) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 363 469 ] concat [ (7) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 296.428 449.786 ] concat [ (8) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 230 469 ] concat [ (9) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 177 496 ] concat [ (10) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 151 540 ] concat [ (11) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 133 606.428 ] concat [ (12) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 151 673 ] concat [ (13) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 177 717 ] concat [ (14) ] Text End Begin %I Text 0 0 0 SetCFg /Helvetica-Bold 14 SetF [ 1 0 0 1 221 744 ] concat [ (15) ] Text End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 729 370 716 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 370 716 414 688 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 414 688 444 641 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 643 455 588 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 455 588 443 533 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 533 412 486 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 412 487 370 458 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 370 458 313 446 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 446 256 458 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 256 458 211 486 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 211 486 182 532 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 182 533 171 588 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 170 588 182 643 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 182 643 211 689 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 211 689 255 718 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 729 455 587 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 369 716 443 531 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 413 688 413 487 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 643 369 459 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 455 588 312 447 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 443 533 255 459 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 414 486 210 487 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 370 458 182 533 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 313 446 171 588 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 255 459 182 643 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 211 485 211 688 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 183 532 256 718 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 256 718 313 730 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 172 588 313 730 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 183 643 370 717 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 212 689 413 688 Line End Begin %I Line 1 0 0 [] 0 SetB 0 0 0 SetCFg 1 1 1 SetCBg 0 SetP [ 1 0 0 1 -11 14 ] concat 256 717 443 643 Line End End %I eop showpage end %%EndDocument 353 679 a endTexFig 349 2031 a Fn(Figure)15 b(1:)20 b(A)15 b(circulan)o(t)h(graph)f(with)h(16)e (no)q(des)i(and)f(o\013sets)f(1)h(and)h(4.)949 2727 y(23)p eop %%Page: 24 24 bop -92 619 a 33564236 36585014 1184071 1512980 35719495 39469056 startTexFig -92 619 a %%BeginDocument: Figs/concat.ps /Times-Roman findfont 12 scalefont setfont 407.929993 448.610016 moveto (After round 2) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After round 2) show 190.970001 448.610016 moveto (After round 1) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After round 1) show 407.929993 575.169983 moveto (After round 0) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After round 0) show 0.400000 setlinewidth newpath 109.609993 674.609985 moveto 109.609993 584.210022 lineto 127.689995 584.210022 lineto 127.689995 674.609985 lineto closepath stroke newpath 109.609993 656.530029 moveto closepath stroke newpath 109.609993 656.530029 moveto 127.689995 656.530029 lineto stroke newpath 109.609993 638.450012 moveto 127.689995 638.450012 lineto stroke newpath 109.609993 620.369995 moveto 127.689995 620.369995 lineto stroke newpath 109.609993 602.289978 moveto 127.689995 602.289978 lineto stroke newpath 145.770004 674.609985 moveto 145.770004 584.210022 lineto 163.850006 584.210022 lineto 163.850006 674.609985 lineto closepath stroke newpath 145.770004 656.530029 moveto closepath stroke newpath 145.770004 656.530029 moveto 163.850006 656.530029 lineto stroke newpath 145.770004 638.450012 moveto 163.850006 638.450012 lineto stroke newpath 145.770004 620.369995 moveto 163.850006 620.369995 lineto stroke newpath 145.770004 602.289978 moveto 163.850006 602.289978 lineto stroke newpath 181.930008 602.289978 moveto 200.010010 602.289978 lineto stroke newpath 181.930008 620.369995 moveto 200.010010 620.369995 lineto stroke newpath 181.930008 638.450012 moveto 200.010010 638.450012 lineto stroke newpath 181.930008 656.530029 moveto 200.010010 656.530029 lineto stroke newpath 181.930008 656.530029 moveto closepath stroke newpath 181.930008 674.609985 moveto 181.930008 584.210022 lineto 200.010010 584.210022 lineto 200.010010 674.609985 lineto closepath stroke newpath 218.089996 674.609985 moveto 218.089996 584.210022 lineto 236.169998 584.210022 lineto 236.169998 674.609985 lineto closepath stroke newpath 218.089996 656.530029 moveto closepath stroke newpath 218.089996 656.530029 moveto 236.169998 656.530029 lineto stroke newpath 218.089996 638.450012 moveto 236.169998 638.450012 lineto stroke newpath 218.089996 620.369995 moveto 236.169998 620.369995 lineto stroke newpath 218.089996 602.289978 moveto 236.169998 602.289978 lineto stroke newpath 254.250000 602.289978 moveto 272.330017 602.289978 lineto stroke newpath 254.250000 620.369995 moveto 272.330017 620.369995 lineto stroke newpath 254.250000 638.450012 moveto 272.330017 638.450012 lineto stroke newpath 254.250000 656.530029 moveto 272.330017 656.530029 lineto stroke newpath 254.250000 656.530029 moveto closepath stroke newpath 254.250000 674.609985 moveto 254.250000 584.210022 lineto 272.330017 584.210022 lineto 272.330017 674.609985 lineto closepath stroke 118.650002 683.650024 moveto (P0) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P0) show 154.809998 683.650024 moveto (P1) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P1) show 190.970001 683.650024 moveto (P2) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P2) show 227.130005 683.650024 moveto (P3) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P3) show 263.290009 683.650024 moveto (P4) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P4) show /Times-Roman findfont 8 scalefont setfont 118.650002 665.570007 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 154.809998 665.570007 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 190.970001 665.570007 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 227.130005 665.570007 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 263.290009 665.570007 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show newpath 326.570007 674.609985 moveto 326.570007 584.210022 lineto 344.649994 584.210022 lineto 344.649994 674.609985 lineto closepath stroke newpath 326.570007 656.530029 moveto closepath stroke newpath 326.570007 656.530029 moveto 344.649994 656.530029 lineto stroke newpath 326.570007 638.450012 moveto 344.649994 638.450012 lineto stroke newpath 326.570007 620.369995 moveto 344.649994 620.369995 lineto stroke newpath 326.570007 602.289978 moveto 344.649994 602.289978 lineto stroke newpath 362.730011 674.609985 moveto 362.730011 584.210022 lineto 380.809998 584.210022 lineto 380.809998 674.609985 lineto closepath stroke newpath 362.730011 656.530029 moveto closepath stroke newpath 362.730011 656.530029 moveto 380.809998 656.530029 lineto stroke newpath 362.730011 638.450012 moveto 380.809998 638.450012 lineto stroke newpath 362.730011 620.369995 moveto 380.809998 620.369995 lineto stroke newpath 362.730011 602.289978 moveto 380.809998 602.289978 lineto stroke newpath 398.890015 602.289978 moveto 416.970001 602.289978 lineto stroke newpath 398.890015 620.369995 moveto 416.970001 620.369995 lineto stroke newpath 398.890015 638.450012 moveto 416.970001 638.450012 lineto stroke newpath 398.890015 656.530029 moveto 416.970001 656.530029 lineto stroke newpath 398.890015 656.530029 moveto closepath stroke newpath 398.890015 674.609985 moveto 398.890015 584.210022 lineto 416.970001 584.210022 lineto 416.970001 674.609985 lineto closepath stroke newpath 435.049988 674.609985 moveto 435.049988 584.210022 lineto 453.130005 584.210022 lineto 453.130005 674.609985 lineto closepath stroke newpath 435.049988 656.530029 moveto closepath stroke newpath 435.049988 656.530029 moveto 453.130005 656.530029 lineto stroke newpath 435.049988 638.450012 moveto 453.130005 638.450012 lineto stroke newpath 435.049988 620.369995 moveto 453.130005 620.369995 lineto stroke newpath 435.049988 602.289978 moveto 453.130005 602.289978 lineto stroke newpath 471.209991 602.289978 moveto 489.290009 602.289978 lineto stroke newpath 471.209991 620.369995 moveto 489.290009 620.369995 lineto stroke newpath 471.209991 638.450012 moveto 489.290009 638.450012 lineto stroke newpath 471.209991 656.530029 moveto 489.290009 656.530029 lineto stroke newpath 471.209991 656.530029 moveto closepath stroke newpath 471.209991 674.609985 moveto 471.209991 584.210022 lineto 489.290009 584.210022 lineto 489.290009 674.609985 lineto closepath stroke 335.610016 665.570007 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 371.769989 665.570007 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 407.929993 665.570007 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 444.089996 665.570007 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 480.250000 665.570007 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 335.610016 647.489990 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 371.769989 647.489990 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 407.929993 647.489990 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 444.089996 647.489990 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 480.250000 647.489990 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show newpath 109.609993 548.049988 moveto 109.609993 457.649994 lineto 127.689995 457.649994 lineto 127.689995 548.049988 lineto closepath stroke newpath 109.609993 529.970032 moveto closepath stroke newpath 109.609993 529.970032 moveto 127.689995 529.970032 lineto stroke newpath 109.609993 511.890015 moveto 127.689995 511.890015 lineto stroke newpath 109.609993 493.809998 moveto 127.689995 493.809998 lineto stroke newpath 109.609993 475.730011 moveto 127.689995 475.730011 lineto stroke newpath 145.770004 548.049988 moveto 145.770004 457.649994 lineto 163.850006 457.649994 lineto 163.850006 548.049988 lineto closepath stroke newpath 145.770004 529.970032 moveto closepath stroke newpath 145.770004 529.970032 moveto 163.850006 529.970032 lineto stroke newpath 145.770004 511.890015 moveto 163.850006 511.890015 lineto stroke newpath 145.770004 493.809998 moveto 163.850006 493.809998 lineto stroke newpath 145.770004 475.730011 moveto 163.850006 475.730011 lineto stroke newpath 181.930008 475.730011 moveto 200.010010 475.730011 lineto stroke newpath 181.930008 493.809998 moveto 200.010010 493.809998 lineto stroke newpath 181.930008 511.890015 moveto 200.010010 511.890015 lineto stroke newpath 181.930008 529.970032 moveto 200.010010 529.970032 lineto stroke newpath 181.930008 529.970032 moveto closepath stroke newpath 181.930008 548.049988 moveto 181.930008 457.649994 lineto 200.010010 457.649994 lineto 200.010010 548.049988 lineto closepath stroke newpath 218.089996 548.049988 moveto 218.089996 457.649994 lineto 236.169998 457.649994 lineto 236.169998 548.049988 lineto closepath stroke newpath 218.089996 529.970032 moveto closepath stroke newpath 218.089996 529.970032 moveto 236.169998 529.970032 lineto stroke newpath 218.089996 511.890015 moveto 236.169998 511.890015 lineto stroke newpath 218.089996 493.809998 moveto 236.169998 493.809998 lineto stroke newpath 218.089996 475.730011 moveto 236.169998 475.730011 lineto stroke newpath 254.250000 475.730011 moveto 272.330017 475.730011 lineto stroke newpath 254.250000 493.809998 moveto 272.330017 493.809998 lineto stroke newpath 254.250000 511.890015 moveto 272.330017 511.890015 lineto stroke newpath 254.250000 529.970032 moveto 272.330017 529.970032 lineto stroke newpath 254.250000 529.970032 moveto closepath stroke newpath 254.250000 548.049988 moveto 254.250000 457.649994 lineto 272.330017 457.649994 lineto 272.330017 548.049988 lineto closepath stroke 118.650002 539.010010 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 154.809998 539.010010 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 190.970001 539.010010 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 227.130005 539.010010 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 263.290009 539.010010 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 118.650002 520.929993 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 154.809998 520.929993 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 190.970001 520.929993 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 227.130005 520.929993 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 263.290009 520.929993 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 118.650002 502.849976 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 118.650002 484.769989 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 154.809998 502.849976 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 154.809998 484.769989 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 190.970001 502.849976 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 190.970001 484.769989 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 227.130005 502.849976 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 227.130005 484.769989 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 263.290009 502.849976 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 263.290009 484.769989 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show newpath 326.570007 548.049988 moveto 326.570007 457.649994 lineto 344.649994 457.649994 lineto 344.649994 548.049988 lineto closepath stroke newpath 326.570007 529.970032 moveto closepath stroke newpath 326.570007 529.970032 moveto 344.649994 529.970032 lineto stroke newpath 326.570007 511.890015 moveto 344.649994 511.890015 lineto stroke newpath 326.570007 493.809998 moveto 344.649994 493.809998 lineto stroke newpath 326.570007 475.730011 moveto 344.649994 475.730011 lineto stroke newpath 362.730011 548.049988 moveto 362.730011 457.649994 lineto 380.809998 457.649994 lineto 380.809998 548.049988 lineto closepath stroke newpath 362.730011 529.970032 moveto closepath stroke newpath 362.730011 529.970032 moveto 380.809998 529.970032 lineto stroke newpath 362.730011 511.890015 moveto 380.809998 511.890015 lineto stroke newpath 362.730011 493.809998 moveto 380.809998 493.809998 lineto stroke newpath 362.730011 475.730011 moveto 380.809998 475.730011 lineto stroke newpath 398.890015 475.730011 moveto 416.970001 475.730011 lineto stroke newpath 398.890015 493.809998 moveto 416.970001 493.809998 lineto stroke newpath 398.890015 511.890015 moveto 416.970001 511.890015 lineto stroke newpath 398.890015 529.970032 moveto 416.970001 529.970032 lineto stroke newpath 398.890015 529.970032 moveto closepath stroke newpath 398.890015 548.049988 moveto 398.890015 457.649994 lineto 416.970001 457.649994 lineto 416.970001 548.049988 lineto closepath stroke newpath 435.049988 548.049988 moveto 435.049988 457.649994 lineto 453.130005 457.649994 lineto 453.130005 548.049988 lineto closepath stroke newpath 435.049988 529.970032 moveto closepath stroke newpath 435.049988 529.970032 moveto 453.130005 529.970032 lineto stroke newpath 435.049988 511.890015 moveto 453.130005 511.890015 lineto stroke newpath 435.049988 493.809998 moveto 453.130005 493.809998 lineto stroke newpath 435.049988 475.730011 moveto 453.130005 475.730011 lineto stroke newpath 471.209991 475.730011 moveto 489.290009 475.730011 lineto stroke newpath 471.209991 493.809998 moveto 489.290009 493.809998 lineto stroke newpath 471.209991 511.890015 moveto 489.290009 511.890015 lineto stroke newpath 471.209991 529.970032 moveto 489.290009 529.970032 lineto stroke newpath 471.209991 529.970032 moveto closepath stroke newpath 471.209991 548.049988 moveto 471.209991 457.649994 lineto 489.290009 457.649994 lineto 489.290009 548.049988 lineto closepath stroke 335.610016 539.010010 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 371.769989 539.010010 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 407.929993 539.010010 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 444.089996 539.010010 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 480.250000 539.010010 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 335.610016 520.929993 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 371.769989 520.929993 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 407.929993 520.929993 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 444.089996 520.929993 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 480.250000 520.929993 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 335.610016 502.849976 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 335.610016 484.769989 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 371.769989 502.849976 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 371.769989 484.769989 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 407.929993 502.849976 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 407.929993 484.769989 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 444.089996 502.849976 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 444.089996 484.769989 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 480.250000 502.849976 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 480.250000 484.769989 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 335.610016 466.690002 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 371.769989 466.690002 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 407.929993 466.690002 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 444.089996 466.690002 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 480.250000 466.690002 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show /Times-Roman findfont 12 scalefont setfont 335.610016 683.650024 moveto (P0) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P0) show 371.769989 683.650024 moveto (P1) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P1) show 407.929993 683.650024 moveto (P2) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P2) show 444.089996 683.650024 moveto (P3) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P3) show 480.250000 683.650024 moveto (P4) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (P4) show newpath 109.609993 421.489990 moveto 109.609993 331.089996 lineto 127.689995 331.089996 lineto 127.689995 421.489990 lineto closepath stroke newpath 109.609993 403.410004 moveto closepath stroke newpath 109.609993 403.410004 moveto 127.689995 403.410004 lineto stroke newpath 109.609993 385.330017 moveto 127.689995 385.330017 lineto stroke newpath 109.609993 367.250000 moveto 127.689995 367.250000 lineto stroke newpath 109.609993 349.170013 moveto 127.689995 349.170013 lineto stroke /Times-Roman findfont 8 scalefont setfont 118.650002 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 118.650002 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 118.650002 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 118.650002 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 118.650002 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 154.809998 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 154.809998 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 154.809998 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 154.809998 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 154.809998 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show newpath 145.770004 349.170013 moveto 163.850006 349.170013 lineto stroke newpath 145.770004 367.250000 moveto 163.850006 367.250000 lineto stroke newpath 145.770004 385.330017 moveto 163.850006 385.330017 lineto stroke newpath 145.770004 403.410004 moveto 163.850006 403.410004 lineto stroke newpath 145.770004 403.410004 moveto closepath stroke newpath 145.770004 421.489990 moveto 145.770004 331.089996 lineto 163.850006 331.089996 lineto 163.850006 421.489990 lineto closepath stroke newpath 181.930008 421.489990 moveto 181.930008 331.089996 lineto 200.010010 331.089996 lineto 200.010010 421.489990 lineto closepath stroke newpath 181.930008 403.410004 moveto closepath stroke newpath 181.930008 403.410004 moveto 200.010010 403.410004 lineto stroke newpath 181.930008 385.330017 moveto 200.010010 385.330017 lineto stroke newpath 181.930008 367.250000 moveto 200.010010 367.250000 lineto stroke newpath 181.930008 349.170013 moveto 200.010010 349.170013 lineto stroke 190.970001 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 190.970001 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 190.970001 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 190.970001 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 190.970001 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 227.130005 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show 227.130005 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 227.130005 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 227.130005 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 227.130005 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show newpath 218.089996 349.170013 moveto 236.169998 349.170013 lineto stroke newpath 218.089996 367.250000 moveto 236.169998 367.250000 lineto stroke newpath 218.089996 385.330017 moveto 236.169998 385.330017 lineto stroke newpath 218.089996 403.410004 moveto 236.169998 403.410004 lineto stroke newpath 218.089996 403.410004 moveto closepath stroke newpath 218.089996 421.489990 moveto 218.089996 331.089996 lineto 236.169998 331.089996 lineto 236.169998 421.489990 lineto closepath stroke newpath 254.250000 421.489990 moveto 254.250000 331.089996 lineto 272.330017 331.089996 lineto 272.330017 421.489990 lineto closepath stroke newpath 254.250000 403.410004 moveto closepath stroke newpath 254.250000 403.410004 moveto 272.330017 403.410004 lineto stroke newpath 254.250000 385.330017 moveto 272.330017 385.330017 lineto stroke newpath 254.250000 367.250000 moveto 272.330017 367.250000 lineto stroke newpath 254.250000 349.170013 moveto 272.330017 349.170013 lineto stroke 263.290009 412.450012 moveto (0) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (0) show 263.290009 394.369995 moveto (1) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (1) show 263.290009 376.290009 moveto (2) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (2) show 263.290009 358.209991 moveto (3) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (3) show 263.290009 340.130005 moveto (4) stringwidth pop 2.0 div -1 mul -2.400000 rmoveto (4) show /Times-Roman findfont 16 scalefont setfont 299.450012 629.409973 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show 299.450012 502.849976 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show 516.409973 629.409973 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show 516.409973 502.849976 moveto (==>) stringwidth pop 2.0 div -1 mul -5.600000 rmoveto (==>) show /Times-Roman findfont 12 scalefont setfont 190.970001 575.169983 moveto (Initial) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (Initial) show 190.970001 322.050018 moveto (After local shift) stringwidth pop 2.0 div -1 mul -4.000000 rmoveto (After local shift) show showpage %%EndDocument -92 619 a endTexFig 286 2090 a Fn(Figure)15 b(2:)20 b(Example)c(of)e(the)i(concatenation)f (algorithm)g(with)h(5)f(pro)q(cessors.)949 2727 y(24)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 16:18:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08358; Wed, 21 Apr 93 16:18:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19305; Wed, 21 Apr 93 16:16:14 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 16:16:12 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19245; Wed, 21 Apr 93 16:16:00 -0400 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA06389 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Wed, 21 Apr 1993 21:15:48 +0100 Date: Wed, 21 Apr 1993 21:15:48 +0100 From: James Cownie Message-Id: <199304212015.AA06389@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA04899; Wed, 21 Apr 93 20:11:18 GMT To: mpi-comm@cs.utk.edu Cc: ajgh@ecs.soton.ac.uk Subject: Meeting in Europe ? Content-Length: 944 Gentle people, At the last MPI meeting we agreed to have a meeting in August (and possibly even September as well). After this decision had been taken, various people approached me and asked whether we could have one of these meetings in Europe. (Not only Rolf and Lyndon !). I have since spoken to Professor Hey, who has kindly offered to host the meeting at Southampton and handle local arrangements if we were to decide to hold a meeting in Europe. Since the arrangements for these meetings are still fluid, now is the time to make this decision, therefore would you care to think about this now. I guess we should aim to make a decision at the next meeting. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 16:34:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08835; Wed, 21 Apr 93 16:34:03 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20629; Wed, 21 Apr 93 16:32:23 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 16:32:18 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20616; Wed, 21 Apr 93 16:32:16 -0400 Received: from Electryon.Think.COM by mail.think.com; Wed, 21 Apr 93 16:32:14 -0400 From: Adam Greenberg Received: by electryon.think.com (4.1/Think-1.2) id AA24033; Wed, 21 Apr 93 16:32:08 EDT Date: Wed, 21 Apr 93 16:32:08 EDT Message-Id: <9304212032.AA24033@electryon.think.com> To: jim@meiko.co.uk Cc: mpi-comm@cs.utk.edu, ajgh@ecs.soton.ac.uk In-Reply-To: <199304212015.AA06389@hub.meiko.co.uk> "jim@meiko.co.uk" Subject: Meeting in Europe ? Date: Wed, 21 Apr 1993 21:15:48 +0100 From: James Cownie Gentle people, At the last MPI meeting we agreed to have a meeting in August (and possibly even September as well). After this decision had been taken, various people approached me and asked whether we could have one of these meetings in Europe. (Not only Rolf and Lyndon !). I have since spoken to Professor Hey, who has kindly offered to host the meeting at Southampton and handle local arrangements if we were to decide to hold a meeting in Europe. Since the arrangements for these meetings are still fluid, now is the time to make this decision, therefore would you care to think about this now. I guess we should aim to make a decision at the next meeting. Heh. I'm fully willing to go to Europe. However, even if Europe is not the eventual venue, somewhere (anywhere?) other than a hotel isolated in North Dallas is to be desired. moose From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 16:42:21 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08999; Wed, 21 Apr 93 16:42:21 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21293; Wed, 21 Apr 93 16:40:53 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 16:40:50 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21278; Wed, 21 Apr 93 16:40:48 -0400 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA09476; Wed, 21 Apr 93 13:40:32 PDT Message-Id: <9304212040.AA09476@ocfmail.ocf.llnl.gov> Date: Wed, 21 Apr 1993 13:50:13 -0800 To: James Cownie From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Re: Meeting in Europe ? Cc: mpi-comm@cs.utk.edu Because of budgetary cuts, foreign travel has been restricted recently for us at LLNL. So, I couldn't make it to a meeting held outside the US. Dan From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 17:27:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10298; Wed, 21 Apr 93 17:27:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24631; Wed, 21 Apr 93 17:26:14 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 17:26:13 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gw1.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24623; Wed, 21 Apr 93 17:26:11 -0400 Received: by gw1.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA16165; Wed, 21 Apr 93 21:26:08 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA26045; Wed, 21 Apr 93 15:24:44 MDT Date: Wed, 21 Apr 93 15:24:44 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9304212124.AA26045@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Meeting in Europe ? > Heh. I'm fully willing to go to Europe. However, even if Europe is not the > eventual venue, somewhere (anywhere?) other than a hotel isolated in North > Dallas is to be desired. > > moose I'll second this. If this is a subcommittee meeting, you've got two votes from this organization! Tom Henderson NOAA Forecast Systems Laboratory From owner-mpi-comm@CS.UTK.EDU Wed Apr 21 23:13:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15052; Wed, 21 Apr 93 23:13:08 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14845; Wed, 21 Apr 93 23:10:44 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 21 Apr 1993 23:10:42 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14828; Wed, 21 Apr 93 23:10:40 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99D) id AA15605; Wed, 21 Apr 93 23:10:34 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA15802; Wed, 21 Apr 93 23:10:42 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Wed, 21 Apr 93 23:10:31 -0400 Message-Id: <9304220310.AA21130@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA21130; Wed, 21 Apr 93 23:10:31 -0400 To: mpi-comm@cs.utk.edu Subject: Re: Meeting in Europe ? James Cownie writes: >At the last MPI meeting we agreed to have a meeting in August (and >possibly even September as well). > >After this decision had been taken, various people approached me and >asked whether we could have one of these meetings in Europe. (Not only >Rolf and Lyndon !). > Hear, hear! If MPI is supposed to be a truely INTERNATIONAL forum, I think it is indeed appropriate to schedule at least one of the meetings outside the U.S. Now, if the organizers (Jack/David) would still sponsor at least some of the airfare for those of us (academics) they now support, there is no way you can keep me away from the meeting! :-) Adam Greenberg follows up: > Heh. I'm fully willing to go to Europe. However, even if Europe is not the > eventual venue, somewhere (anywhere?) other than a hotel isolated in North > Dallas is to be desired. > > moose No kidding! Hmmm .. maybe the HPF peope picked the local to make sure people didn't feel like taking some sessions off to explore less technical things ... ;-) (devious smile) Anne ------------------------------------------------------------------------------- Anne C. Elster | (607) 254-8653 [off] 255-6236 [FAX] Space Plasma Physics | School of Electrical Engineering | (607) 272-2986 [home/answ.mach] Cornell University, 351 Theory Ctr.| Ithaca, NY 14853 | e-mail: elster@cs.cornell.edu USA | na.elster@na-net.ornl.gov From owner-mpi-comm@CS.UTK.EDU Thu Apr 22 01:46:33 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16636; Thu, 22 Apr 93 01:46:33 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23870; Thu, 22 Apr 93 01:45:06 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 22 Apr 1993 01:45:05 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23857; Thu, 22 Apr 93 01:45:02 -0400 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA07638 (5.65c/IDA-1.4.4 for ); Thu, 22 Apr 1993 07:44:53 +0200 Received: by f1neuman.gmd.de id AA15494; Thu, 22 Apr 1993 07:44:49 GMT Date: Thu, 22 Apr 1993 07:44:49 GMT From: Rolf.Hempel@gmd.de Message-Id: <9304220744.AA15494@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Re: Meeting in Europe ? Cc: gmap10@f1neuman.gmd.de It's an excellent idea to hold one of the meetings in Europe. If you need an additional reason to accept it, here it is: in August and probably also in September it's a lot cooler in Europe (especially in England) than at Dallas. Rolf Hempel From owner-mpi-comm@CS.UTK.EDU Thu Apr 22 04:53:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21419; Thu, 22 Apr 93 04:53:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09250; Thu, 22 Apr 93 04:52:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 22 Apr 1993 04:52:11 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09242; Thu, 22 Apr 93 04:52:06 -0400 Date: Thu, 22 Apr 93 09:51:46 BST Message-Id: <6797.9304220851@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Meeting in Europe ? To: James Cownie , mpi-comm@cs.utk.edu In-Reply-To: James Cownie's message of Wed, 21 Apr 1993 21:15:48 +0100 Reply-To: lyndon@epcc.ed.ac.uk Cc: ajgh@ecs.soton.ac.uk Jim writes: > > At the last MPI meeting we agreed to have a meeting in August (and > possibly even September as well). > > After this decision had been taken, various people approached me and > asked whether we could have one of these meetings in Europe. (Not only > Rolf and Lyndon !). I solid proposal if ever I heard one indeed! My support can be assumed. > I have since spoken to Professor Hey, who has kindly offered to host > the meeting at Southampton and handle local arrangements if we were to > decide to hold a meeting in Europe. Southampton would be fine. We could probably hold one here in Edinburgh (much prettier town than Southampton, imho :-). Of course I'd just love that. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ >From owner-mpi-comm@CS.UTK.EDU Wed May 12 04:35:20 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA17462; Wed, 12 May 93 04:35:17 -0500 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA12711; Wed, 12 May 93 04:36:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12047; Wed, 12 May 93 05:14:29 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 12 May 1993 05:14:28 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12017; Wed, 12 May 93 05:14:25 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99D) id AA18254; Wed, 12 May 93 05:14:13 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA14805; Wed, 12 May 93 05:14:13 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Wed, 12 May 93 05:14:09 -0400 Message-Id: <9305120914.AA19664@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA19664; Wed, 12 May 93 05:14:09 -0400 To: mpi-comm@cs.utk.edu Subject: MPI_LOAD_INFO sugestion Cc: elster@cs.cornell.edu Status: R As much as I hate to expand the number of fuctions in MPI, I think this function is a MUST the way I see parallel computing in teh future. Below is a latex-version of the write-up I will be bringing to the meeting today. (Yes, yes, I know I should have discussed this long time ago, but ...). I will bring hard-copies to the meeting, so that people can start thinking about it. Hope you like the idea! - ------------------------------------------------------------------------------- Anne C. Elster | (607) 254-8653 [off] 255-6236 [FAX] Xerox Design Research Institute | (607) 272-2986 [home/answ.mach] Cornell University, 502 Theory Ctr.| Ithaca, NY 14853 | e-mail: elster@cs.cornell.edu USA | na.elster@na-net.ornl.gov \documentstyle[fullpage,12pt]{article} \baselineskip=12pt \oddsidemargin -0.6in \begin{document} \setlength{\parskip}{1ex} % \maketitle %\pagestyle{empty} \begin{center} {\large MPI\_LOAD\_INFO: Implementing load-balanced applications on distributed-memory systems} Anne C. Elster\\ \small Cornell University, Ithaca NY 14853 \date{\today} \end{center} \footnotesize \subsection*{Motivation} Currently, most parallelized algorithms for distributed memory machines assume that they will run on a homogeneous set of processors. (I.e. that each node in the parallel system has approximately the same computing power and I/O speed.) The main focus in parallel computing has thus far been to minimize communication over-head and achieve a load-balance for the particular algorithm or application. However, as each computational node a parallel system become more complex, their individual processor/node performance characteristics will tend to deviate more as some nodes perform I/O while others concentrate on system tasks, etc. This behavior has so far been addressed through multi-tasking on large shared-memory systems such as a network of IBM 3090s, however, no simple, straight-forward solutions exist currently for the distributed memory cases. It is, however, this authors belief that load-balancing application at the ``load-info'' level (i.e. based on each node's dynamic load at run-time) will become very important in the future -- especially if there were mechanisms built into MPI that would facilitate such implementations. \subsection*{Current options} The current options for getting load-info read into a Fortran or C application on Unix-based systems are not wonderful: System load is typically from the OS-side done by {\bf reading kernel memory (-kmem)}. This will, however, usually require {\em setgid} permissions and grubbing through kernel symbol tables. This brute-force approach might fork and exec something with the required kmem permissions. Another alternative is to use {\bf rstat} which does not require such permission, but unfortunately, is not available on all systems. Rstat typically gives you statistics from a remote kernel that should be current disregarding network delay. The third approach I can think of is to use the output from {\bf uptime}. In C, this could possibly be done though fscanf on the return value of popen. \subsection*{MPI suggestion} Since I fond none of the previous methods very programmer-friendly, my suggestion is to add a {\bf MPI\_LOAD\_INFO} that would return the desired information. In the point-to-point case, this would be similar to what one could get from $rstat$, whereas in the collective sense, one could be gathering information on a whole process group (or nodes within a context?) . I see the latter as VERY useful. Clearly the specific format of the info. to be received via such an MPI call needs to be discussed. If the feed-back on this suggestion is positive, this author would gladly write up a more detailed proposal for such a feature. Anne C. Elster Cornell Univ. \end{document} >From weeks@mozart.convex.com Wed Jun 9 14:59:20 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15025; Wed, 9 Jun 93 14:59:18 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12536; Wed, 9 Jun 93 14:58:20 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01415; Wed, 9 Jun 93 07:57:31 -0500 Received: by mozart.convex.com (5.64/1.28) id AA27552; Wed, 9 Jun 93 07:59:43 -0500 Date: Wed, 9 Jun 93 07:59:43 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091259.AA27552@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 12 mpi mail #4 Status: R Another one from Steve Otto (initial verbosity deleted so I could print the LaTeX file --DW): \documentstyle[twoside,11pt]{report} %\pagestyle{headings} %%\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Proposal VIII for contexts and groups} \author{Mark P. Sears, {\em Sandia National Laboratories} } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \label{sec:context} %======================================================================% % BEGIN "Proposal VIII" % % written by Mark P. Sears % Sandia National Laboratories % March 30, 1993 % % I defined these because the discuss macro wont let me % use \verb or verbatim environment. \def\dishdr{\vspace{\discussSpace} \begingroup\setlength{\parindent}{0pt} \setlength{\parskip}{0pt}\leftskip=.6cm \small {\bf Discussion: }} \def\disend{\par\vspace{\discussSpace} \endgroup\par } % end of macro definitions \chapter{Contexts and Groups -- Proposal VIII} \begin{center} {Mark P. Sears, Sandia National Laboratories} \end{center} \section{Introduction and motivation.} \dishdr March 30 draft -- handed out at MPI meeting April 13 draft -- modifications to context allocation scheme, added discussion of limitations, interaction with pt2pt draft, some reorganization. May 11 draft -- added getgroupcontext function which allows the allocation of a dynamic context within a group. On reading this proposal, anything in the discussion sections is considered to be commentary, is open to debate and is probably not part of the proposal as it would go into the MPI document as a whole. \disend The MPI committee has determined that message passing common practice needs to be extended to satisfy two objectives. The first of these is to add facilities which allow development and use of independent software modules (libraries and other third party software). The primary need here is seen to be enabling the establishment of independent communication environment, or context, for independent software components. The second objective is to add capabilities, especially for collective communications, which allow the software developer to use subsets, or groups, of the processes in the parallel task in a convenient way. The purpose of this chapter is to define these concepts more precisely. The development of these ideas follows primarily from the principle that context and group concepts should be as orthogonal as possible, in order to make them easy to understand and implement. The remainder of this chapter defines the terms {\bf process}, {\bf parallel task}, {\bf process address}, {\bf context}, {\bf group }, and {\bf topology} as they are used in this document. In addition this chapter describes the MPI functions needed to assign, create, and manipulate contexts and groups, and describes the overall syntax needed to define point to point and collective communications. Later chapters in the document contain precise descriptions of point to point and collective communication functions syntax and semantics. In the following, a {\bf global quantity} is one which is defined and can be used in the same way on every process of the parallel task. For example, it can be sent as part of a message from one process to another and used on the second process in the same way as on the first. A {\bf local quantity} is defined only on the process which creates it and cannot be sent (in a meaningful way) to another process. Process addresses, tag values, and context values are global quantitites; groups are local quantities. Similarly a {\bf global call} or {\bf group call} is a function call executed (loosely) synchronously on all processes or all processes which are members of a group. A {\bf local call} is a function call executed on a single process independent of any execution on other processes. \dishdr This proposal differs from the other context-group proposals in that context and group are held to be orthogonal and distinct entities. It is closest in flavor to Proposal III, where context is a low level concept, but here the definition of group is unrelated to the definition of context. In this proposal, groups have no context or tags associated with them, default or otherwise. \begin{enumerate} \item The purpose of contexts is to promote software modularity by allowing the construction of independent tag spaces. A context is an integer-valued extension to the tag field which must match exactly between sender and receiver of point-to-point communications. \item In this proposal, contexts are global and there is no concept of a process ``belonging'' to a context. Contexts are also anticipated to be a scarce resource. Context allocation is a rare event, typically limited to program startup. \item The purpose of groups is to provide tools for management of subsets of processes in a parallel task. A group is defined abstractly as an ordered set of unique integers (the elements). Group as a type is a pointer to an object which defines the mapping between rank and the elements of the group, and is a locally defined opaque type. Note that a group can be considered as a ``collection of processes'' only insofar as the elements of the group are process addresses. \item As defined in this proposal, groups contain no context or tag information, nor is a facility for caching user defined information provided. This implies that context and tag must be specified in addition to group for collective communication functions. This accords with the goal of MPI as specifying a lower level implementation that can be used to create different kinds of higher level implementations. \end{enumerate} This proposal is distinguished from proposal III by a much more complete orthogonalization of the concepts of group and context, as well as specifying group at a lower, more fundamental level than proposal III. The advantages of this lower level definition are primarily ease of implementation and understanding and additional flexibility in creating higher level functionality. In addition, because groups use no context or tag resources, they are limited only by available memory and CPU. \disend \section{Parallel task and process models.} An MPI program or {\bf parallel task} consists of a set of {\bf processes}, each executing independent code in an independent address space. The model of a process is similar to that of UNIX. A process can be single threaded or multithreaded, and in a multithreaded environment, MPI makes no distinction between the several threads of a process. Messages can be sent and received between processes, but there is no mechanism to specify the receiving thread or obtain the sending thread of a message. The set of processes in a parallel task is fixed. Each process is addressed by its rank in the task, which is fixed and assigned in an implementation defined way when the parallel task is created. The assignment of processes to processors is also implementation defined, and MPI defines a method (described below) to return a description of the process assignment, or {\bf global topology}. The process address of a process is an integer defined globally within the parallel task, which can be sent in messages to any other process and used with the same meaning on any other process. In addition to the processes in a parallel task, an MPI implementation may also allow communication with a {\bf host process}, if it exists. The host process is not considered part of the parallel task, but point to point communications with it are allowed. The host process may send to or receive messages from any process in the parallel task. The process address of the host process is an integer defined globally within the parallel task, but is not a valid rank in the parallel task. A routine, \verb+MPI_gethost+, returns the host address if available, or returns the constant \verb+MPI_NOHOST+ if there is no host process. An MPI implementation may support a limited subset of point-to-point communications with a host process. The creation and termination of processes and parallel tasks is not defined within MPI. \dishdr This section does not differ greatly from other proposals. This section would modify section 1.3 of the draft Point to Point communication document, mostly in regard to the question of how processes are addressed. I have expanded on the host process stuff a little from the discussion at the last meeting. The committee agreed (I think) to include the possibility of a host process. I feel it is unfortunate but necessary to postpone any discussion of communication between parallel tasks to MPI 2.0. Some of the possible properties that such communication might have are: \begin{itemize} \item A process may belong to at most one parallel task. \item Process addressing must be extended to allow for the inclusion of a parallel task identifier. \item Inter-task communication requires a session layer which intra-task communication does not. \end{itemize} but all this remains open to debate. % i want to think about this a little: % % Inter-task communication can be added to the above proposal as follows. % Each parallel task has a system-wide unique identifier. The definition % of a process address is widened to include both a task id and the process % rank within the task. In other words process-address = (task-id, rank). % The result must be encodable in an integer. No changes would be made % to the syntax or semantics of point-to-point communications, except that % % The concept of dynamic parallel tasks, tasks where the number of processes belonging to the task changes over time, is also full of difficulties and should also be postponed to MPI 2.0, along with any discussion of the means by which parallel tasks are created or destroyed. Note that it would be quite useful to define these things. \disend \section{Contexts.} The purpose of contexts is to promote software modularity by allowing the construction of independent tag spaces. A context is an integer-valued extension to the tag field which must match exactly between sender and receiver of point-to-point communications. By specifying a unique context in addition to tag and other features of a message, the communication between sender and receiver is made independent of other communications which use different contexts. In the following, context values are assigned or allocated by various software components, but they are not pushed or popped during program execution. This more static model avoids the state associated with such a stack mechanism. Contexts as described here do not provide a mechanism for a software component to protect contexts that it has allocated or assigned from misuse. The only protection is that software components which use different contexts are free to use tags within those contexts with a guarantee of no interference. A {\bf context} or {\bf context value} is an integer in a range \verb+[0..MPI_MAXCONTEXT]+ which is part of the message envelope. The sender and receiver of the message must specify exactly matching context values; no wildcard filtering on the context part of a message envelope is allowed. Filtering of messages based on sender address and/or tag is discussed further in the section on point to point communications. The number of available context values, \verb+MPI_NUMCONTEXTS+, is an implementation-defined quantity. This proposal recommends a minimum number of 256, although implementations allowing a greater or lesser number of context values are allowed. \dishdr There has been considerable discussion on the network about the number of available context values. One proposal (by Rik Littlefield) is that MPI should go ahead and specify that context and tag are both 32 bit qtys and that the implementor should provide for the possibility of contexts and tags anywhere in this space. I think these limits are too generous. The other side (e.g. Peter Rigsbee) is to not specify any hard limits and let the marketplace decide. I don't quite agree with this either, since other standards (e.g. Fortran 77) {\it do} specify limits for some things. Perhaps the committee could decide that there is a lower limit (say 256, my original estimate of 16 was seen by many as too small and I have altered it (see above)) for compliance, and a recommended number of $2^{16}$ or $2^{32}$. Then we could identify compliant implementations (satisfies the lower limit) and strongly compliant implementations (satisfies the recommended limit). I propose that this point be left up for grabs and discussed further by the environmental subcommittee. The above statement specifies that there exists an implementation defined limit on the size of the context space. But it does not specify that there is a limit on the number of contexts in the space that can be used at any one time. Should we specify the existence of such a limit? I don't think so. \disend There exists a predefined and allocated context value, \verb+MPI_DEFAULTCONTEXT+ which does not need to be allocated or opened and cannot be freed or closed. This context value is intended for initialization purposes and is free for use by all software components. Context values may either be assigned statically or allocated dynamically, and so we refer to them as static contexts and dynamic contexts. Since context values are integers, it is not appropriate to speak of contexts being created or destroyed. Static context values are chosen by the programmer and should lie in the range \verb+[0, MPI_STATICCONTEXTLIMIT]+, where \verb+MPI_STATICCONTEXTLIMIT+ is an implementation-defined constant. Dynamic context values will be allocated outside this range, within the limit \verb+MPI_CONTEXTLIMIT+ mentioned previously. \dishdr A problem could arise when a static context value is used which would normally be allocated by the dynamic allocation routines. Here we only specify an administrative solution, because that might be the intended program behavior. Generally, we adhere to the notion of allowing any reasonable use of context values rather than specifing strong limitations. \disend Static context values must be opened before use and can be closed when no longer needed. The routine \begin{verbatim} void MPI_opencontext(int context) \end{verbatim} is a local operation that specifies that the context can be used in sending and receiving messages. The routine \begin{verbatim} void MPI_closecontext(int context) \end{verbatim} is a local operation that specifies that the context is no longer needed and no further messages will be sent or received using the context value. Dynamic context values can be allocated with the routine \begin{verbatim} int MPI_getcontext() \end{verbatim} which returns a context value and which must be called synchronously by all processes in the parallel task before messages can be sent or received with the context. Dynamic context values can be freed with the routine \begin{verbatim} void MPI_freecontext(int context-value) \end{verbatim} which is a local operation and which signals to MPI that the given context value will no longer be used. In addition, dynamic context values can be allocated within a group with the routine \begin{verbatim} int MPI_getgroupcontext(Group g) \end{verbatim} This routine must be called synchronously by all processes which are members of the group. It will allocate a context value which has not been previously allocated on any of the processes in the group, so the context value is guaranteed to be unique within the group. Note that if there are two groups which do not intersect then the same context value may be allocated on each. Note also that it is possible for a context value to be allocated in a group and sent to a process not in the group which then opens the context and uses it for communications. There is no guarantee of uniqueness on the second process. Finally, there is no coupling of the context with the group. It is not possible to get the group that was used to allocate the context, nor is the context associated with the group in any way. \dishdr The getgroupcontext function is a collective communication operation. Here it has been specified without reference to a predecessor context or a set of tags, which means that it must be implemented using context and tags internal to the implementation. The semantics needs to be refined to handle the possibility of multithreading (programmer must mutex other threads). Both the getcontext and getgroupcontext functions give the user a lot of flexibility and a lot of room to hang himself or herself. \disend \dishdr The original version of this proposal only allowed for dynamic global context values. After discussion, I have modified the proposal to allow for statically and dynamically allocated contexts, as well as contexts which are allocated only within a group. Internally, MPI implementations may need to set up private data structures or other resources (queues, buffers, etc.) in order to send and receive messages with a particular context value. The \verb+MPI_getcontext()+ routine is the appropriate entry point for such setup to occur. Some implementations will simply set up resources for all possible context values at program initiation. % Deleted: % Obviously, there are many possible schemes for context allocation and % deallocation. The scheme proposed here avoids the need and overhead % of a complex server. Another option is to avoid the allocation problem % altogether, have only static allocation % (as for Fortran unit numbers or TCP/IP ports), and have a routine % \verb+MPI_opencontext(int context-value)+ which initializes use of the % context. % Fancier context allocation could be built on top of such a proposal. It may be necessary for some MPI implementations to preallocate other context values for internal use. Such context values would not be accessible by user code. \disend \section{Groups.} The purpose of groups is to provide tools for management of subsets of processes in a parallel task. A group is defined abstractly as an ordered set of unique integers (the elements); this abstract definition allows groups to be constructed in many different ways and allows the definition of a group as an explicit list (e.g. 7,4,13) or a computational rule (e.g. processes with odd addresses). Groups as program objects may be defined by MPI or may be developed as third party software. A {\bf group} is a one-to-one mapping from the integers $(0, 1, ..., N-1)$ to another set of integers, the {\bf elements} of the group. The number of elements $N$ is the {\bf order} of the group. The {\bf range} of the group is an upper limit on the size of any element. Note that a group is not an ``array of integers'', although that is one possible implementation. An ordered collection of processes is said to be a group if each element of the group defines a process address, or rank in the parallel task. A process belongs to a group if its address is an element of the group. In the following, the opaque type {\tt MPIGroup} is intended to be thought of as a pointer to one of many possible group structures. This {\bf group identifier} is local to a process and may not be legitimately passed from one process to another. \dishdr It would not be too difficult to define functions \verb+MPI_getgroupdsc()+ and \linebreak[4] \verb+MPI_bldgroupfromdsc()+ which would compute an ASCII description of a group or build a group from its ASCII description. Such functions would ease the problem of sending groups in messages. \disend Groups are defined only by construction in a process, so the correct method of passing a group from one process to another is to pass the information needed to construct it. Group construction and destruction is therefore local to a process, and is independent of whether the process belongs to the group or not. There may be many classes of groups, some defined by MPI and some defined as user extensions. Any group class must define the following methods: \begin{verbatim} int MPI_grouporder(MPIGroup g) -- compute order of a group. int MPI_grouprange(MPIGroup g) -- compute range of a group. int MPI_groupelement(MPIGroup g, int rank) -- compute element given rank. int MPI_grouprank(MPIGroup g, int element) -- compute rank of element. int MPI_groupiselement(MPIGroup g, int element) -- membership predicate. MPIGroup MPI_copygroup(MPIGroup g) -- make a copy of a group. void MPI_freegroup(MPIGroup g) -- group destructor. \end{verbatim} \dishdr The public definition of a group would contain a class member defining all of these operations, so, for example, \verb+MPI_grouporder+ would simply execute the function \verb+group->class->order+. In order for the group concept to be user extensible, some common definition of the basic group object and class object needs to be established. This definition is properly done by the binding committee. Thus, although {\bf MPIGroup} is an opaque type it is not entirely hidden -- a sophisticated user can get at it and create new group types. \disend In addition, each class of groups must define a group constructor. The constructor takes as parameters the information needed to construct the group and returns the opaque group identifier. >From the point of view of this proposal, group construction and use are inexpensive operations, limited only by memory and CPU resources. \dishdr For example, let us define a simple class of groups, which we call {\bf linear} groups. For such a group, we define the mapping from rank to element to be the function $(start + delta * rank)$, so the parameters needed to create the group are the order and the constants $start, delta$. The group constructor would then be defined as \begin{verbatim} MPIGroup MPI_makelineargroup(int order, int start, int delta). \end{verbatim} Another simple class would be a {\bf list} group, for which the construction parameters are the order of the group and a list of the elements. The constructor is \begin{verbatim} MPIGroup MPI_makelistgroup(int order, int *elements). \end{verbatim} Note that in the first case, there is probably no need for any communication to create the group. In the second case, the list might have to be distributed using communications, but the group construction is purely local to each process. \disend \section{Using groups in point-to-point communications.} Groups can be used to simplify point-to-point communication. A process in a group can be defined either by its process address (equivalent to the rank in the task) or by its rank in the group. A typical call to send a message using a (group,rank) address might have the form \begin{verbatim} MPI_send(buffer, length, MPI_groupelement(group, rank), tag, context) \end{verbatim} Note that in order for a process to send a message to a process in another group using the second process' group rank, it must construct a local copy of the group. Since a group defines an order for its elements, it is easy to compute the preceding and following processes of a particular process in a group. The preceding and following processes (elements) of the current process are given by \begin{verbatim} my_rank = MPI_grouprank(group, my_pid) pid_precede = MPI_groupelement(group, my_rank-1) pid_follow = MPI_groupelement(group, my_rank+1). \end{verbatim} modulo boundary conditions. \dishdr Two and three dimensional generalizations are straightforward but require additional functions specific to particular types of groups (e.g. Cartesian groups). \disend \section{Groups and collective communication.} Groups are the primary method of organizing collective communication operations. All collective communication operations have the form \begin{verbatim} void MPI_xxx(group, context, ...) \end{verbatim} Such functions must be called by all of the processes in the group in order to complete successfully. \dishdr Previously I proposed that collective communication routines be passed both a context and one or more tags. But a much cleaner solution is to pass the routine just a context under the assumption that the routine may make any use of tags within the context that it likes. Groups may contain private (internal to MPI) information about methods and data to accelerate common collective communication operations. For example, a group implementation may include a private copy of a spanning tree to be used for fast broadcasting. Such information is implementation dependent and not normally accessible to a user. The definition of a group should contain a hook for the collective comm routines to hang such data. \disend \section{Topology.} There are several meanings for the term topology as used in MPI. The first meaning is associated with the global topology or assignment of processes to processors in a particular network. Such a topology might be classified as a hypercube, mesh, or general network topology. Information about the global topology is returned by the routine \begin{verbatim} char *MPI_globaltopology() \end{verbatim} which returns a string containing a description of the global topology. The form of this string is generally {\it topology-class parameter ...}. \dishdr For example, various implementations might return strings like \begin{verbatim} N 564 -- a random (Ethernet) network with 564 processes. H 5 -- a 5 dimensional hypercube. R 2 16 13 -- a two dimensional mesh, 16 by 13. \end{verbatim} \disend The second type of topology is that internal to a group. Since, in this proposal, groups are ordered sets and therefore have an implicitly defined (one dimensional) topology, there is no need for independent topology routines. A different ordering can be selected by building a different group. For example, to select a Gray ordering instead of a natural ordering (useful on a hypercube for some applications), one would build a permutation group implementing the Gray ordering. Alternatively, one could impose a Gray ordering on an existing group by a group composition operation. Third, we define a {\bf virtual topology} as a method of referring to elements of a group other than by straightforward rank. For example, we might want to think of a group as a rectangle or higher order Cartesian object, and reference elements by x-rank, y-rank, etc. Another example might be a tree virtual topology, where we would refer to elements in terms of parent, sibling and child. Virtual topologies can be implemented by layering on top of MPI. \section{Contexts, groups, and third party software.} >From the point of view of this proposal, there are two kinds of third party or library software, i.e. software written to use MPI but intended for use in turn by other applications. The first type is similar to the MPI collective communication operations as described above. This type uses the contexts and tag space provided by whatever software calls it. The second type, typically associated with much more complex functions, establishes and manages its own context and tag spaces. MPI must be careful to support both possibilities. Library software of the first kind, including MPI collective communications, might benefit from the specification of some sort of ``tag manager'' so that only context would need to be provided. This proposal does not attempt to answer this question. \section{Cache facility.} This proposal does not define a cache facility for adding information to groups or contexts. Such a facility, although it has attractive features, is more appropriately layered on top of MPI. \section{Modifications to other documents.} The discussions in the current point-to-point document (sections 1.3 and\ 1.4 of the May 9 draft would be deleted). All of the collective communication routines would be modified to conform to the guideline that a collective communication routine is passed a group and context value separately. \section{Subsets of MPI.} Because this proposal maintains a strong orthogonality between the context and group concepts, it is easy to eliminate all collective communication operations (indeed all group operations) from MPI in order to generate a useful subset. The mechanisms for managing context are simple and do not require a server concept. \end{document} >From weeks@mozart.convex.com Wed Jun 9 15:02:26 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15054; Wed, 9 Jun 93 15:02:23 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12599; Wed, 9 Jun 93 15:01:00 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01533; Wed, 9 Jun 93 08:00:15 -0500 Received: by mozart.convex.com (5.64/1.28) id AA27965; Wed, 9 Jun 93 08:02:23 -0500 Date: Wed, 9 Jun 93 08:02:23 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091302.AA27965@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 12 mpi mail #5 Status: R Another LaTeX source with initial verbosity deleted ... \documentstyle[twoside]{report} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newcommand{\implement}[1]{\vspace{\discussSpace} {\small {\bf Implementation note:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface} \author{Scott Berryman, {\em Yale Univ} \\ James Cownie, {\em Meiko Ltd} \\ Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\ Al Geist, {\em ORNL} \\ Bill Gropp, {\em ANL} \\ Rolf Hempel, {\em GMD} \\ Bob Knighten, {\em Intel} \\ Rusty Lusk, {\em ANL} \\ Steve Otto, {\em Oregon Graduate Inst} \\ Tony Skjellum, {\em Missisippi State Univ} \\ Marc Snir, {\em IBM T. J. Watson} \\ David Walker, {\em ORNL} \\ Steve Zenith, {\em Kuck \& Associates} } \date{May 9, 1993 \\ This work was supported by ARPA and NSF under contract number \#\#\#, by the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615. } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \chapter{Point to Point Communication} \label{sec:pt2pt} \begin{center} Marc Snir \\ William Gropp and Ewing Lusk \end{center} \section{Introduction} This section is a draft of the current proposal for point-to-point communication. It does not yet include a description of the Fortran 77 and C bindings. I have tried to indicate, wherever appropriate, gaps and unresolved issues, using small type. \discuss{ The following sections of the introduction contain general information on the design of MPI procedures. The material should be moved to a general introduction for the entire document. The actual binding of these objects in Fortran and C will be discussed in the language binding subcommittee. } \section{Data Types} \subsection{Handle} MPI procedures use at various places {\em handles}. Handles are used to access opaque objects. Such object can be created, updated and destroyed only by by calling suitable MPI procedures, and providing the handle as parameter. Opaque objects hide from the user the internal representation used for various MPI objects, thus allowing to have similar calls in C and Fortran, allowing to overcome problems with the typing rules in these languages, and allowing for future extension of their functionality. The mechanism for opaque objects used here loosely follows the POSIX Fortran binding standard. An opaque object can be {\em persistent} or {\em ephemeral}. A persistent object persists until destroyed by an explicit operation. An ephemeral object is good for a single use; thus an ephemeral object associated with a communication operation disappears once this operation is completed (or once this object is not needed anymore for the completion of the operation). Opaque object are created by calls that are specific to each object type. They are destroyed by a call to {\tt MPI\_FREE}. Additional MPI functions are available to access and update specific opaque objects. \mpifunc{MPI\_FREE(handle)} \begin{description} \item[IN handle] handle to object \end{description} Mark the handle for destruction. The object can destroyed as soon as there there is no pending operation that is using this object. This is equivalent to changing the persistence of the handle from {\tt persistent} to {\tt ephemeral}. For example, if a communication object is freed, after the communication started but before it completed, then the pending operation is not affected; one can use the handle in calls to {\tt MPI\_WAIT} or {\tt MPI\_STATUS}. However, once the operation completes, it is erroneous to reuse the freed handle for a new communication. \discuss{ An alternative design point was discussed, whereby it is the user responsibility to free a handle only if there are no pending operations on this handle. The current design was deemed more elegant. Note that it is not necessary to destroy the handle as soon as it is not needed anymore. A valid implementation may just mark freed handles and garbage collect them periodically. On the other hand, some objects, such as buffer descriptor objects, can be used by several distinct concurrent communication operations. Thus, to find out whether such object can be freed, one would need a reference count of pending operations. This complication was not observed at our last meeting, and may be a good argument to reconsider the current definition of {\tt MPI\_FREE}. We do not have yet a mechanism to control where from storage is allocated for opaque objects. Some believe it is desirable to allow the user to control this allocation. I now have a generic {\tt MPI\_FREE} function for all kinds of opaque objects. Any reason to have specialized functions for each kind of object, as for {\tt CREATE}? } \mpifunc{MPI\_ASSOCIATED(handle, type)} \begin{description} \item[IN handle] handle to object \item[OUT type] state \end{description} Returns the type of the object the handle is currently associated with, if such exists. Returns the special type {\tt MPI\_NULL} if the handle is not currently associated with any object. MPI may provide predefined opaque objects and predefined, static handles to these objects. Such objects may not be destroyed. \subsection{Array of handles} An MPI call may need a parameter that is an {\em array of handles}. The array-of-handles object is not self-delimiting; it does not carry information on the number of entries in the array. This information has to be provided by an additional {\tt len} parameter, whenever a list-of-handles parameter is used. \discuss{ The reason for not having self-delimiting arrays is that records are not first-class citizens in Fortran and that having an array where one entry encodes length, whereas the other entries encode handles was deemed to be too error-prone. This line of argumentation really implies that we do not view array-of-handles as an opaque object, and that we expect the user to build these objects explicitly, not via MPI calls. The alternative is for array-of-handles to be an opaque object, manipulated via s set of MPI calls. } \subsection{State} MPI procedures use at various places arguments with {\em state} types. The values of such data type are all identified by names, and no operation is defined on them. For example, the {\tt MPI\_APPEND} routine has a state type parameter with values {\tt MPI\_INT, MPI\_REAL,} etc. \subsection{Named constants} MPI procedures sometimes assign a special meaning to a special value of a basic type parameter; e.g. {\tt tag} is an integer valued parameter of point-to-point communication operations, with a special {\tt DONTCARE} value. Such parameters will have a range of regular values, which is a proper subrange of the range of values of the corresponding basic type; special values (such as DONTCARE) will be outside the regular range. The range of regular values can be queried, and sometimes set, using environment inquiry or environment setting functions (Section~\ref{sec:inquiry}). \discuss{ Need to agree on a language mechanism for named constants. Implementers should detect, whenever possible, illegal uses of ``special values''. Thus, the use of the {\tt DONTCARE} value to tag a message sent will be flagged as an error. } \subsection{Choice} MPI functions sometimes use parameters with a {\em choice} (or union) data type. I.e., distinct calls to the same routine may pass by reference actual parameters of different types. The mechanism for providing such parameters will differ from language to language. \discuss{ The Fortran 77 standard specifies that the type of actual arguments need to agree with the type of dummy arguments; no construct equivalent to C pointers is available. Thus, it would seem that there is no standard conforming mechanism to support choice parameters. However, most Fortran compiler either don't check type consistency of calls to external routines, or support a special mechanism to link foreign (e.g., C) routines. I suggest that we accept this nonconformity with Fortran 77 standard. I.e., we accept that the same routine may be passed an actual parameter of a different type at distinct calls. Generic routines can be used in Fortran 90 to provide a standard conforming solution. This solution will be consistent with our nonstandard conforming Fortran 77 solution. } \section{Processes} \discuss{This material belongs to an introduction or environment section} An MPI program is executed by several autonomous processes that each execute their own code, in a MIMD style. The codes executed by each process need not be identical. The processes communicate via calls to MPI communication primitives. Typically, each processor executes in its own address space, although shared-memory implementations of MPI are possible. This document specifies the behavior of a parallel program assuming that only MPI calls are used for communication. The interaction of an MPI program with other possible means of communication (e.g., shared memory) is not specified. In particular, it is assumed that message buffers at distinct processors are disjoint. MPI does not specify the execution model for each process. A process can be sequential, or can be multithreaded, with threads possibly executing concurrently. Care has been taken to make MPI ``thread-safe'', by avoiding the use of implicit global states. The initial allocation of processes to an MPI computation and their binding to physical processors is not specified by the program itself. It is expected that vendors will provide mechanisms to do so either at load time or at run time. Such mechanisms will allow the specification of the initial number of required processes, the code to be executed by each initial process, and the allocation of processes to processors. Also, the current proposal does not provide for dynamic creation or deletion of processes during program execution (the total number of processes is fixed), although it is intended to be consistent with such extension. Finally, the current proposal does not specify a naming scheme for processes. We propose to always identify processes according to their relative rank in a context (group), so that, effectively, processes are identified by consecutive integers. \subsection{Error Handling} \discuss{ Material of this section should be moved to a general introduction section. This material has not yet been approved by MPIF} MPI provides the user with reliable message transmission: A message sent is always received correctly, and the user does not need to check for transmission errors, time-outs, or other error conditions. In other words, MPI does not provide mechanisms for dealing with failures in the communication system. Where the MPI implementation is built on an unreliable underlying mechanism, then it is the job of the implementer of the MPI subsystem to insulate the user from this unreliability, or to reflect unrecoverable errors as global program failures. Similarly MPI itself provides no mechanisms for handling node failures. Of course, MPI programs may still be erroneous. A {\bf program error} can occur when an MPI call is called with an incorrect parameter (non-existing destination in a send operation, buffer too small in a receive operation, etc.) This type of error would occur in any implementation. In addition, a {\bf resource error} may occur when a program exceeds the amount of available system resources (number of pending messages, system buffers, etc.). The occurrence of this type of error depends on the amount of available resources in the system and the resource allocation mechanism used; this may differ from system to system. The recommended implementation profile provides several mechanisms to alleviate the portability problem this represents. One can also write {\bf safe} programs, that are not subject to resource errors. All MPI procedure calls return an error parameter that indicates successful completion of the operation, or the error condition that occurred, otherwise. The recommended implementation profile in a POSIX environment is for any MPI routine that encounters a recoverable error to generate an {\em MPI error signal}, using a special signal value. The default handler for this signal terminates the execution of all involved processes, with a suitable error message being returned to the user. However, the user can provide his or her own signal handling routine. In particular, the user can specify a ``noop'' signal handler, thus relegating all error handling to the user code, using the error parameters returned by the MPI calls. MPI calls may initiate operations that continue asynchronously after the call returned. Thus, the operation may return with a code indicating successful completion, yet later cause an error exception to be raised. If there is a subsequent call that relates to the same operation (e.g., a call that verifies that an asynchronous operation has completed) then the error parameter associated with this call will be used to indicate the nature of the error. In a few cases, the error may occur after all calls that relate to the operation have completed, so that no error parameter can be used to indicate the nature of the error (e.g., an error in a send with the ready mode). In such cases, an error will be undetected, if the user disabled the MPI error signal. \discuss{ The alternative choice is to have fatal and non-fatal signals. One might want different signals for different modules. The details of such proposal need be elaborated in an appropriate ``profile'' subcommittee. } \section{Messages} A message consists of an {\em envelope} and {\em data}. \subsection{Data} The data part of a message consists of a sequence of values, each of a basic datatype in the host language. Thus, in Fortran 77, a message consists of a sequence of values that are each of type {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt DOUBLE PRECISION COMPLEX}, {\tt LOGICAL}, or (length 1) {\tt CHARACTER}. A message may also contain a value of type {\tt BYTE}, which consists of 8 binary digits. A byte is different from a character. Different machines may have different representations for the same character, or may use more than one byte to represent a character. On the other hand, a byte has the same binary value on any machine. A message may mix values of different types. \missing{ Need to agree on the C types (including handling of signed/unsigned), and on Fortran 90 types (including handling of kinds). } \subsection{Envelope} \label{subsec:envelope} The following information is associated with each message: \begin{description} \item[source] The rank of the sending process \item[destination] The rank of the receiving process \item[tag] User defined \item[context] handle \end{description} The range of valid values for the {\bf source} and {\bf destination} fields is {\tt 0 ... n-1}, where {\tt n} is the number of processes in the specified context. {\tt source = destination} is allowed: a process can send a message to itself. The ranges of valid values for {\tt tag} is implementation dependent, and can be found by calling a suitable query function, as described in Section~\ref{sec:inquiry}. The {\tt tag} field can be arbitrarily set by the application, and can be used to distinguish different messages. {\tt Context} should be a context shared by both source and destination. The actual mechanism used to associate an envelope with a message is implementation dependent; some of the information (e.g., {\bf sender} or {\bf receiver}) may be implicit, and need not be explicitly carried by a message. \section{Communication Buffers} \label{subsec:buffers} The basic point to point communication operations are {\bf send} and {\bf receive}. A {\bf send} operation creates a message; the message data is taken from the {\bf send buffer}. A {\bf receive} operation consumes a message; the message data is put into the {\bf receive buffer}. The specification of send or receive buffers uses the same syntax. \subsection{Communication buffer components} A buffer consists of a sequence of {\bf buffer components}. Each component consists of a sequence of (not necessarily distinct) variables of the same basic datatype. The datatype of such a variable is specified by a state parameter. The possible values for Fortran are {\tt MPI\_INT}, {\tt MPI\_REAL}, {\tt MPI\_DOUBLE}, {\tt MPI\_COMPLEX}, {\tt MPI\_DCOMPLEX}, {\tt MPI\_LOGICAL}, {\tt MPI\_CHAR} and {\tt MPI\_BYTE}, corresponding to datatypes {\tt INTEGER}, {\tt REAL}, {\tt DOUBLE PRECISION}, {\tt COMPLEX}, {\tt DOUBLE PRECISION COMPLEX}, {\tt LOGICAL}, {\tt CHARACTER} and to untyped {\tt BYTE}. The possible values for C are {\tt MPI\_CHAR}, {\tt MPI\_SHORT}, {\tt MPI\_INT}, {\tt MPI\_LONG}, {\tt MPI\_FLOAT}, {\tt MPI\_DOUBLE} and {\tt MPI\_BYTE}. \discuss{ May need to take care of kinds in Fortran 90 and of signed/unsigned in C. More elaborate definitions belong to the language binding. } There are five kinds of buffer components: \subsubsection{Contiguous component} A sequence of contiguous values of the same basic type, specified by \begin{description} \item[start] Initial element \item[count] Number of elements ($\tt count \ge 0$) \item[datatype] Type of elements \end{description} Thus, if {\tt A} is an array of integers, then {\tt A(1), 4, MPI\_INT} is the buffer component with entries {\tt A(1), A(2), A(3), A(4)}. \subsubsection{Vector component} A sequence of equally spaced and equally sized blocks of elements of the same basic type, specified by \begin{description} \item[start] Initial element \item[count] Number of elements ($\tt count \ge 0$) \item[stride] Number of elements between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk > 0$) \item[datatype] Type of elements \end{description} Assume that an element of type {\tt datatype} occupies {\tt m} bytes. Then, in a byte addressable machine, the buffer component described by these five parameters consists of the {\tt count} variables at addresses {\tt start, start+m, ... , start+m*(lenblk-1), start+m*stride,} \\ {\tt start+m*(stride+1), ..., start+m*(stride+lenblk-1), ...}. For example, if {\tt A} is an array of integers, then \begin{itemize} \item {\tt A(1), 5, 3, 2, MPI\_INT} is the buffer component with entries {\tt A(1), A(2), A(4), A(5), A(7)}. \item {\tt A(1), 5, 0, 2, MPI\_INT} is the buffer component with entries {\tt A(1), A(2), A(1), A(2), A(1)}. \item {\tt A(10), 5, -1, 2, MPI\_INT} is the buffer component with entries {\tt A(10), A(11), A(9), A(10), A(8)}. \end{itemize} {\tt Contiguous} is a special case of {\tt vector}. A vector buffer component can be used to extract an arbitrary submatrix of a two-dimensional matrix. \subsubsection{Indexed component} A sequence of elements of the same basic type, specified by \begin{description} \item[start] initial element \item[count] Number of elements ($\tt count \ge 0$) \item[array\_of\_indices] Array of displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} For example, if {\tt A} is an array of integers, and {\tt I} is an array with values {\tt 2, -1, 2, 1} then {\tt A(3), 4, I, MPI\_INT} is the buffer component with entries {\tt A(5), A(2), A(5), A(4)}. {\tt Vector} can be thought of as a special case of {\tt indexed} in which the indices are generated with a constant stride. \subsubsection{Heterogeneous vector (hvector) component} Same as {\tt vector}, except that {\tt stride} is given in multiple of bytes, rather than multiples of elements: \begin{description} \item[start] Initial element \item[count] Number of elements ($\tt count \ge 0$) \item[stride] Number of bytes between the start of each block \item[lenblk] Number of elements in each block ($\tt lenblk > 0$) \item[datatype] Type of elements \end{description} Assume that an element of type {\tt datatype} occupies {\tt m} bytes. Then, in a byte addressable machine, the buffer component described by these five parameters consists of the {\tt count} variables at addresses {\tt start, start+m, ... , start+m*(lenblk-1), start+stride, start+stride+m, ..., start+stride+m*(lenblk-1), ...}. Consider a C array declared as \begin{verbatim} struct { char info[3]; short number; double val; } a[100] \end{verbatim} Assume that all structure components are byte aligned, a {\tt short} occupies two bytes, and a {\tt double} occupies eight bytes. Then the buffer component {\tt \&a, 100, 13, 2, MPI\_CHAR} extracts the first two characters of each structure. The buffer component {\tt \&(a.number), 100, 13, 1, MPI\_SHORT} extracts the {\tt short} field of each structure; and the buffer component {\tt \&(a.val), 100, 13, 1} extracts the {\tt double} field of each structure. Thus, heterogeneous vectors can be used to extract vectors of homogeneous values from heterogeneous data. {\tt Vector} is a special case of {\tt heterogeneous vector}. \subsubsection{Heterogeneous indexed (hindexed) component} Same as indexed, except that displacements are in bytes, rather than multiples of element size. \begin{description} \item[start] initial element \item[count] Number of elements ($\tt count \ge 0$) \item[array\_of\_indices] Array of byte displacements of the elements in the buffer components, relative to the initial element. \item[datatype] Type of elements \end{description} For example, if {\tt a} is the C array of the previous example, and {\tt I} is an array with values {\tt 0, 13, 13, 27, 26} then {\tt \&a, 5, I, MPI\_CHAR} is the buffer component with entries {\tt a[0].info[0], a[1].info[0], a[1].info[0], a[1].info[1], a[1].info[0]}. Like {\tt heterogeneous vector}, {\tt heterogeneous indexed} extends the {\tt indexed} facility so as to allow the extraction of homogeneous data from a heterogeneous layout. \discuss{ The use of displacements in indexed buffer components has the same effect as indirect addressing into an array with initial index zero. I.e., if {\tt a} is an array with initial index zero, then a buffer component defined by parameters {\tt a[0], N, i, type} extract the elements {\tt a[i[0]], a[i[1]], ..., a[i[n]]}. This fits well C usage but ill-fits Fortran usage (initial index one). We took a decision of principle to force ``start from zero'' index arithmetic on Fortran users, rather than have different index arithmetic in the C and Fortran binding of MPI. The use of both buffer components with byte and element size displacements is motivated by the observation that some applications require the more general byte displacement, but that most applications would use element size displacement, which is much more natural and convenient (and machine independent). Thus, rather than forcing all users to do byte arithmetics, we decided to provide separately the more advanced function to ``expert programmers''. Leslie Hart proposes to replace ``heterogeneous'' with ``general''. } \subsection{Buffer operations} A buffer is described by an opaque {\bf buffer descriptor} object accessed via a {\bf buffer handle}. Such an object is created by {\tt MPI\_CREATE\_BUFFER}. It is associated with successive buffer components by calling in succession one of the functions {\tt MPI\_APPEND\_xxx}, where {\tt xxx} identifies the type of buffer component appended to the buffer. It becomes available for use in communication operations once committed by the function {\tt MPI\_COMMIT\_BUFFER}. Once committed, the descriptor cannot be modified any more. Note that the commit operation does not create yet a message, and does not bind the data to be sent. It is a commit for the buffer descriptor, not a commit for the buffer content. A buffer descriptor is destroyed after the completion of the first communication operation that uses it, if it is ephemeral, or after it is freed by a call to {\tt MPI\_FREE} and any pending communication operation that uses it has completed, if it is persistent. After the buffer descriptor object has been destroyed, the handle is undefined. \mpifunc{MPI\_CREATE\_BUFFER( buffer, persistence )} Create a new buffer descriptor object. The parameters are: \begin{description} \item[OUT buffer] buffer handle \item[IN persistence] buffer persistence (state type with possible values {\tt MPI\_PERSISTENT} or {\tt MPI\_EPHEMERAL}). \end{description} \discuss{ May want to provide information on maximum object size, and perhaps user space for the object. Leslie Hart proposes {\tt MPI\_CREATE\_BUFFER\_DESC}, rather than {\tt MPI\_CREATE\_BUFFER}. More accurate, but longer. Perhaps {\tt MPI\_CREATE\_BUFDESC}? } \mpifunc{MPI\_APPEND\_CONTIGUOUS( buffer, start, count, datatype)} Append a block component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN count] Number of elements (integer) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_APPEND\_VEC( buffer, count, stride, lenblk, datatype )} Append a vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN count] Number of elements (integer) \item[IN stride] Number of elements between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_APPEND\_INDEXED( buffer, start, count, array\_of\_indices, datatype)} Append an indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] initial position for indexing (choice) \item[IN count] Number of elements (integer) \item[IN array\_of\_indices] array of displacements of entries relative to start (array of integers) \item[IN datatype] datatype identifier (status) \end{description} The index values used to create this component are the values provided at the call; subsequent updates of the array of indices will not affect the buffer descriptor component. \mpifunc{MPI\_APPEND\_HVEC( buffer, start, count, stride, lenblk, datatype )} Append a heterogeneous vector buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] buffer component initial element (choice) \item[IN count] Number of elements (integer) \item[IN stride] Number of bytes between the start of each block (integer) \item[IN lenblk] Number of elements in each block (integer) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_APPEND\_HINDEXED( buffer, start, count, array\_of\_indices, datatype)} Append an heterogeneous indexed buffer component to buffer. The parameters are: \begin{description} \item[INOUT buffer] buffer handle \item[IN start] initial position for indexing (choice) \item[IN count] Number of elements (integer) \item[IN array\_of\_indices] array of byte displacements of entries relative to start (array of integers) \item[IN datatype] datatype identifier (status) \end{description} \mpifunc{MPI\_COMMIT\_BUFFER( bufferin )} Commit a buffer descriptor object. The object cannot be changed anymore after it is committed. A buffer descriptor object can be used for communication only after it is committed. The parameters are: \begin{description} \item[INOUT buffer] buffer descriptor handle \end{description} \discuss{ The commit function may do preprocessing of the buffer descriptor to simplify its use and commit resources in the communication subsystem for its use. The commit function may generate a new object and, hence, modify the object handle. The current proposal does not allow to create a buffer descriptor, use it in a communication after committed, then modify it partially and use it for another communication -- a new buffer descriptor has to be created. Of course, the same buffer descriptor can be reused many times in successive communications that send data from the same buffers; e.g., a new send with a buffer descriptor handle can be initiated once the previous send with this handle has completed. } Example 1: \\ Consider the following fragment of Fortran code \begin{verbatim} DOUBLE PRECISION A(10,20) INTEGER B, C(5,10) INTEGER BH ... CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT) CALL MPI_APPEND_BLOCK (BH, B, 1, MPI_INT) CALL MPI_APPEND_VEC (BH, A(1,3), 11, 4, 2, MPI_DOUBLE) CALL MPI_APPEND_INDEX(BH, C(3,7), (4,2,1), MPI_INT) \end{verbatim} Then the buffer associated with the handle {\tt BH} consists of the sequence of variables {\tt B, A(1,3), A(2,3), A(5,3), A(6,3), A(9,3), A(10,3), A(3,4), A(4,4), A(7,4), A(8,4), A(1,5), C(2,8), C(5,7), C(4,7)}. A message created from this buffer will consist of a sequence of one integer, followed by eleven double precision reals, followed by three integers. Example 2: \\ Suppose one has an array of data declared in C as \begin{verbatim} struct { int type; double position[3]; double velocity[3]; } particle[1000] \end{verbatim} and one wants to transfer in one message the number of particles of type 1, the number of particles of type 2, and the positions of all the particles of these two types. The following code fragment creates a suitable buffer descriptor. \begin{verbatim} int count1 = 0; int count2 = 0; ... mpi_create_buffer( &handle, MPI_PERSISTENT); mpi_append_contiguous( &handle, &count1, MPI_INT); mpi_append_contiguous( &handle, &count2, MPI_INT); for (i=0; i<1000; i++) { if (particle[i].type == 1) { count1++; mpi_append_contiguous( &handle, &particle[i].position, 3, MPI_DOUBLE); } elseif (particle[i].type == 2) { count2++; mpi_append_contiguous( &handle, &particle[i].position, 3, MPI_DOUBLE); } } mpi_commit_buffer(handle) ... \end{verbatim} The buffer descriptor handle can now be used to send (repeatedly) the particle counts for type 1 and 2, and the positions of the particles with these two types, as long as the types of the particles is not modified. Suppose the incoming data is to be stored at the receiver into the two count variables {\tt count1} and {\tt count2} and into the {\tt position} components of the initial entries of the array of particles. The following code generates a suitable buffer descriptor for the receive operation (we assume 4 bytes for integer and 8 bytes for double). \begin{verbatim} ... mpi_create_buffer( &handle, MPI_PERSISTENT); mpi_append_contiguous( &handle, &count1, MPI_INT); mpi_append_contiguous( &handle, &count2, MPI_INT); mpi_append_hvec( &handle, &particle[0].position, 1000, 52, 3, MPI_DOUBLE); mpi_commit_buffer( &handle) ... \end{verbatim} A buffer handle can be used for communication, even if it is not associated with any variables (i.e., even if it was not set by any {\tt MPI\_APPEND\_xxx} call). Such a handle is associated with an empty buffer, and a message created from it contains no data. The handle still need be committed before it is used. The same entry may appear more than once in a buffer component, and distinct buffer components may overlap. If a buffer descriptor with replicated entries is used to receive a message, then the final value of an entry that occurs more than once in the buffer descriptor is undefined. \discuss{ It is not clear to me that overlaps have any useful purpose, and they may complicate implementation. But we had an overwhelming vote in favor of this feature. } \subsection{Type Matching and Data Conversion} \subsubsection{Type matching} One can think of message transmission as consisting of three phases: \begin{enumerate} \item Data is pulled out of the send buffer and a message is assembled \item A message is transferred from sender to receiver \item Data is pulled from the incoming message and disassembled into the receive buffer \end{enumerate} Type matching has to be observed at each of these three phases: The type of each variable in the sender buffer has to match the type of the corresponding entry in the buffer descriptor; the type of an entry in the buffer descriptor used to send a message has to match the type of the corresponding entry in the buffer descriptor used to receive the message; and the type of each variable in the receive buffer has to match the corresponding entry in the buffer descriptor for this buffer. A program that fails to observe these three rules is erroneous. To define type matching more precisely, we need to deal with two issues: matching of types of the host language with buffer descriptor types; and matching of buffer descriptor types at sender and receiver. We restrict ourselves to the case where the sending and receiving programs are written in the same language. The type of an entry in the sender buffer descriptor matches the type of an entry in the receiver buffer descriptor if they have identical names: {\tt MPI\_INT} matches {\tt MPI\_INT}, {\tt MPI\_REAL} matches {\tt MPI\_REAL}, and so on. The type of a variable in a host program matches the type of the entry in a buffer descriptor if the name of the buffer descriptor entry type corresponds to the basic type of the host program variable: an entry with type name {\tt MPI\_INT} matches a Fortran variable of type {\tt INTEGER}, an entry with type name {\tt MPI\_REAL} matches a Fortran variable of type {\tt REAL}, and so on. There is one exception to this last rule: An entry with type name {\tt MPI\_BYTE} can be used to match any byte of storage (on a byte-addressable machine), irrespective of the datatype of the variable that contains this byte. The value of the message entry will be the binary value of the corresponding byte in memory. We thus have two cases: \begin{itemize} \item Communication of typed values (e.g., with datatype different from {\tt MPI\_BYTE}, where the datatypes of the corresponding entries in the sender program, in the sender buffer descriptor, in the receiver buffer descriptor and in the receiver program should all match. \item Communication of untyped values (e.g., of datatype {\tt MPI\_BYTE}, where the the corresponding entries in the sender and the receiver buffer descriptors have both datatype {\tt MPI\_BYTE}. In this case, there are no requirements on the datatypes of the corresponding entries in the sender and the receiver programs, nor is it required that they be the same. \end{itemize} \subsection{Data conversion} We use the following terminology: \begin{description} \item[type conversion] changes the datatype of a value, e.g. by rounding a {\tt REAL} to an {\tt INTEGER}. \item[representation conversion] changes the binary representation of a value, e.g. from Hex floating point to IEEE floating point. \end{description} Representation conversion is needed when data is moved across machines that use different binary encodings for the same datatype. Representation conversion preserves, to the greatest possible extend, the {\bf value} represented. However, rounding errors and overflow and underflow exceptions may occur during floating point conversions; conversion of integers may also lead to exceptions when a value that can represented in one system cannot be represented in the other system. MPI does not specify rules for representation conversion. The type matching rules imply that MPI communication never entails type conversion. On the other hand, MPI requires that a representation conversion be performed when a typed value is transferred across environments that use different representations for the datatype of this value. An exception occurring during representation conversion results in a failure of the communication; an error occurs either in the send operation, or the receive operation, or both. If a value sent in a message is untyped (i.e., of type {\tt MPI\_BYTE}), then the binary representation of the byte stored at the receiver is identical to the binary representation of the byte loaded at the sender. This holds true, whether sender and receiver run in the same or in distinct environments; no representation conversion is required. Note that no conversions ever occur when an MPI program executes in a homogeneous system, where all processes run in the same environment. Also note the different behavior of {\tt MPI\_BYTE} and of {\tt MPI\_CHAR}. A buffer descriptor entry with datatype of {\tt MPI\_CHAR} can only match a Fortran variable of type {\tt CHAR}; and representation conversion may occur when values of type {\tt MPI\_CHAR} are transferred., e.g., from an EBCDIC encoding to an ASCII encoding. Consider the following examples. \begin{verbatim} ! FIRST EXAMPLE INTEGER A(100) ... CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH, A(1), 1, MPI_CHAR) \end{verbatim} This program is erroneous, since the type of the buffer descriptor does not match the type of the variable in the buffer. The behavior or this program is undefined. A desirable behavior is for that error to be detected (at compile time or run time) and result in an error condition. \begin{verbatim} ! SECOND EXAMPLE INTEGER A(100) ... CALL MPI_CREATE_BUFFER(BH, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH, A(1), 1, MPI_BYTE) \end{verbatim} This program is correct. The byte that will be transmitted in a message created with this buffer descriptor is the first byte in the storage occupied by {\tt A(1)}. The value of this byte is implementation dependent, and will vary according to the number of bytes used to store Fortran variables of type {\tt INTEGER} and the byte ordering on the host machine. \begin{verbatim} ! THIRD EXAMPLE INTEGER A(100) ... CALL MPI_CREATE_BUFFER(BH1, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH1, A(1), 1, MPI_INTEGER) ... CALL MPI_CREATE_BUFFER(BH2, MPI_PERSISTENT) CALL MPI_APPEND_CONTIGUOUS (BH2, A(1), 4, MPI_BYTE) \end{verbatim} Consider the following three cases: \begin{enumerate} \item A message is sent using buffer descriptor handle {\tt BH1} and received using buffer descriptor handle {\tt BH2}. The program is erroneous, since the sender and receiver buffer descriptors don't match. The behavior of the program is undefined, and a possible (desirable but unlikely) behavior is for an error condition to occur. \item Both sender and receiver use buffer descriptor handle {\tt BH1}. The program is correct, and an integer value is transmitted from sender to receiver. The value assigned to {\tt A(1)} at the receiver is identical to the value of {\tt A(1)} at the sender, but the binary representation of this value may be different, if sender and receiver run in different environments. \item Both sender and receiver use buffer descriptor handle {\tt BH2}. The program is correct. Assume that both sender and receiver use four bytes to store integers. Then the value stored in {\tt A(1)} at the receiver has the same binary representation as the value of {\tt A(1} at the sender. If the same byte ordering and the same encoding of integer values is used at both ends then {\tt A(1)} at the receiver is set to the value of {\tt A(1)} at the sender; otherwise the same 32 bits may encode different values. \end{enumerate} \implement{ The current definition does not require messages to carry type information. A message can be composed and sent using only the information in the sender buffer descriptor, and can be received and stored using only the information in the receiver buffer descriptor. If messages are sent between different machines then one can either use a ``universal'' data encoding for messages, use knowledge of the receiver environment in order to convert data at the sender, or use knowledge of the sender environment in order to convert data at the receiver. In either case the local buffer descriptor can be used to derive the types of the values transferred. However, additional type information carried by messages can provide better error detection. It is possible to add always such additional type information, for better error checking, or add it when running in ``debug'' mode, where some performance is sacrificed for added error checking, etc. } \discuss{ There is no agreement yet on the need for a definition and, if needed, on the right definition of type matching and conversion between programs written in different languages. The natural interpretation of ``send from C and receive in Fortran'' is either ``send from C, receive in C and convert to Fortran'', or ``convert to Fortran, send from Fortran and receive in Fortran''. However, the correspondence between Fortran types and C types is implementation dependent. Thus, the diagram shown below is not commutative, and the two interpretations of C to Fortran communication may lead to different results. \begin{tabular}{c c c} machine 1 & & machine 2 \\ & send & \\ C program & $\rightarrow$ & C program \\ & & \\ $\downarrow$ & & $\downarrow$ \\ & send & \\ Fortran program & $\rightarrow$ & Fortran program \end{tabular} A good solution would make this diagram commutative. Otherwise, one needs to pick one of the two paths as the canonical definition of interlanguage communication. If the upper path is chosen (message carries sender datatypes), then the receiver of a message needs to know the language the message was sent from; if the lower path is chosen, (message carries receiver datatypes), then the sender needs to know the language the message is received into. A possible solution (proposed by Danny Nessett) is to leave to the user the choice of specifying whether to send ``Fortran datatypes'' or ``C datatypes'', in the above situation. One would still require exact matching of send buffer descriptors and receive buffer descriptors. However, a Fortran program could use a buffer descriptor with ``C datatypes'', and vice versa. The standard would specify which matching of Fortran and C datatypes are legal. For example, it would be correct to send a Fortran variable of type {\tt REAL} using a buffer descriptor of type {\tt MPI\_FLOAT}. Data conversion might be required if C float numbers and Fortran real numbers do not use the same presentation. Another (non)solution is to ignore the issue and let the behavior of C to Fortran communication be undefined. The user can always match language at sending and receiving end if he or she is willing to struggle with the issue of calling MPI C functions from Fortran or vice-versa. } \section{Receive Criteria} The selection of a message by a receive operation is done uniquely according to the value of the message envelope. The receive operation specifies an {\bf envelope pattern}; a message can be received by that receive operation only if its envelope matches that pattern. A pattern specifies values for the {\tt source}, {\tt tag} and {\tt context} fields of the message envelope. In addition, the value for the {\tt dest} field is set, implicitly, to be equal to the receiving process id. The receiver may specify a {\tt DONTCARE} value for {\tt source}, or {\tt tag}, indicating that any source and/or tag are acceptable. It cannot specify a DONTCARE value for {\tt context} or {\tt dest}. Thus, a message can be received by a receive operation only if it is addressed to the receiving task, has a matching context, has matching source unless source=DONTCARE in the pattern, and has a matching tag unless tag=DONTCARE in the pattern. DONTCARE values for source or tag are specified via a named constant for source dontcare and a named constant for tag dontcare. The length of the received message must be less then or equal the length of the receive buffer. I.e., all incoming data must fit, without truncation, into the receive buffer. It is erroneous to receive a message which length exceed the receive buffer, and the outcome of program where this occurs is undetermined. \section{Communication Mode} A send operation can occur in one of three modes: \begin{description} \item[STANDARD] The send may start whether or not a matching receive has been posted. \item[READY] The send may start only if a matching receive has been posted. \item[SECURE] The send operation will not complete successfully until it is guaranteed that the message sent will be received. \end{description} The start and completion of a {\bf standard} send does not depend on the status of a matching receive. A {\bf ready send} can start only if a matching receive is already posted; otherwise the operation is erroneous and its outcome is undefined. In some systems, this allows the removal of a hand-shake operation that is otherwise required, and results in improved performance. Its completion does not depend on the status of a matching receive. A {\bf secure send} will complete if a matching receive is posted, and the receive operation is committed to receive the message sent by the secure send. (I.e., the send and receive operations have already been matched, the receive operation has already started and cannot be cancelled anymore.) The start of a secure send does not depend on the status of a matching receive. A receive operation can occur in one of two modes: {\bf STANDARD} and {\bf SECURE}. A secure send can be matched only by a {\bf secure receive}; a program where a secure send can be matched by a standard receive is erroneous and its behavior is undefined. The secure receive can be posted either before or after the matching send. A standard send and a ready send can both be matched only by a {\bf standard receive}; a program where a secure receive can match a nonsecure send is erroneous and its behavior is undefined. The receive has to be posted ahead of the send for a ready send and can be posted either before or after the send is posted for a standard send. \discuss{ I got criticisms about the previous notation (REGULAR) and the current one (STANDARD). I can also try NORMAL, DEFAULT, ... We should probably vote, at some point. Also, Lyndon proposes to use SYNCHRONOUS, rather than secure, since the other modes can also offer secure communication with the right use. } \implement{ The current definition assumes it is the user responsibility to prevent matching of a regular send with a secure receive and vice-versa. Otherwise, messages need be tagged as standard or secure. Assume an implementation of MPI communication that avoids retries. What protocols can be used? Since a receive may have a wildcard source field, the first step in the protocol must be executed by the sender. For a ready send, this first step may involve sending the data, since the receiver is guaranteed to have space to store it. The protocol is send, possibly followed by an ack-send. The protocol for secure send is likely to be the following: The sender sends a request-to-send message. The receiver stores this request (which contains a copy of the message envelope). When a matching receive is posted, the receiver sends back an acknowledge, and the sender now sends the message. The protocol is (req, ack-req, send), possibly followed by an ack-send. What protocol is used for standard sends? Either the second protocol (req, ack-req, send) or a mix of the first and the second (e.g. first used for short messages, second for long messages). When the receiver does not wildcard the source field then the protocol can be reversed: The receiver sends a clear-to-send message, and the sender can proceed with the short protocol if it has a clear-to-send when the send is executed. What do we gain from the ``secure receive''? Some optimizations for regular receives (e.g., in a regular receive of a short message the receiver knows the short protocol will be used, and need not check for a request-to-send). Q: is that enough justification for the secure receive? A counter argument (Leslie Hart) is that one could conceive of an implementation where a regular receive reserves buffer space, so that a program that use regular receives is not safe, even if only secure sends are used. I doubt this is a strong argument. On the other hand, it indicates that, if the expectation is that programs written using only secure sends are safe, then this is an additional correctness condition on the implementation that need be made explicit. } \section{Communication Objects} An opaque communication object identifies various properties of a communication operation, such as the buffer descriptor that is associated with it, its context, the tag and destination parameters to be used for a send, or the tag and source parameters to be used for a receive. In addition, this object stores information about the status of the last communication operation that was performed with this object. This object is accessed using a communication handle. One can consider communication operations to consist of the following suboperations: \begin{description} \item[INIT(operation, params, handle)] Process provides all relevant parameters for its participation in the communication operation (type of operation, data buffer, tag, participants, etc.). An object is created that identifies the operation. \item[START(handle)] The communication operation is started \item[COMPLETE(handle)] The communication operation is completed. \item[FREE(handle)] The communication object, and associated resources are freed. \end{description} Correct invocation of these suboperations is a sequence of the form \[ \bf INIT \ (START \ COMPLETE )^* \ FREE .\] I.e., an object needs be created before communication occurs; it can be reused only after the previous use has completed; and it needs to be freed eventually (of course, one can assume that all objects are freed at program termination, by default). The above scenario pertains to {\em persistent} objects. One can also create {\em ephemeral} objects. Such an object persists only until the communication operation is completed, at which point it is destroyed. Thus the correct invocation of suboperations with an ephemeral object is {\bf INIT START COMPLETE}. A user may directly invoke these suboperations. This would allow the amortization of the overhead of setting up a communication over many successive uses of the same handle, and allows the overlap of communication and computation. Simpler communication operations combine several of these suboperations into one operation, thus simplifying the use of communication primitives. Thus, one only needs to specify precisely the semantics of these suboperations in order to specify the semantics of MPI point to point communication operations. We say that a communication operation (send or receive) is {\bf posted} once a {\bf start} suboperation has been invoked; the operation is {\bf completed} once the {\bf complete} suboperation has completed. A send and a receive operation {\bf match} if the receive pattern specified by the receive matches the message envelope created by the send. \subsection{Communication Object Creation} \discuss{ This section has not yet been approved} An object for a send operation is created by a call to {\bf MPI\_INIT\_SEND}. A call to {\bf MPI\_INIT\_RECV} is similarly used for creating an object for a receive operation. The creation of a communication object is a local operation that need not involve communication with a remote process. \mpifunc{MPI\_INIT\_SEND (handle, buffer\_handle, dest, tag, context, mode, persistence)} Creates a send communication object. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to send buffer descriptor \item[IN dest] rank in context of destination (integer) \item[IN tag] user tag for messages sent with this handle (integer) \item[IN context] context of messages sent with this handle \item[IN mode] send mode (state type, with three values: {\tt MPI\_STANDARD}, {\tt MPI\_READY} and {\tt MPI\_SECURE}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} \mpifunc{MPI\_INIT\_RECV (handle, buffer\_handle, source, tag, context, mode, persistence)} Create a receive handle. Parameters are \begin{description} \item[OUT handle] message handle. The handle should not be associated with any object before the call. \item[IN buffer\_handle] handle to receive buffer descriptor. \item[IN source] rank in context of source, or {\tt MPI\_ANY\_SOURCE} (integer). \item[IN tag] user tag for messages received with this handle, or {\tt MPI\_ANY\_TAG} (integer). \item[IN context] context of messages received with this handle. \item[IN mode] receive mode (state type, with two values: {\tt MPI\_STANDARD}, and {\tt MPI\_SECURE}) \item[IN persistence] handle persistence (state type, with two values: {\tt MPI\_PERSISTENT} and {\tt MPI\_EPHEMERAL}) \end{description} See Section~\ref{subsec:envelope} for a discussion of source, tag and context. \discuss{ I have not included proposals for partially specified message handles, that some peoples seem to desire. I have merged all handle setup into one call. The functions {\tt MPI\_INIT\_SEND} and {\tt MPI\_INIT\_RECV} have the same list of parameters. Jim Cownie suggests to have only one function with an extra argument. Ephemeral ``persistent handles'' are not terribly useful as such. But their existence allow us to describe all the higher level send/receive operations in terms of basic operations on handles. } \subsection{Communication Start} \mpifunc{MPI\_START(handle)} \begin{description} \item[IN handle] communication handle \end{description} The {\tt MPI\_START} function starts the execution of a communication operation (send or receive). A sender should not update the send buffer after a send operation has started until after it has completed. A receiver should not access the receive buffer after a receive operation was started until after it has completed. A program that does not satisfy these conditions is erroneous and its outcome is undetermined. \subsection{Communication Completion} \label{subsec:complete_ops} \mpifunc{MPI\_WAIT ( handle, return\_status\_handle)} \begin{description} \item[IN handle] communication handle \item[IN return\_handle] handle that is associated with return status object. the return status object is updated by call. \end{description} A call to {\bf MPI\_WAIT} returns when the send operation identified by {\tt handle} is complete. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer, or any other location that can be referenced by the send operation. However, it does not indicate that the message has been received; rather it may have been buffered by the communication subsystem. The completion of a receive operation indicates that the receiver is now free to access the locations in the receive buffer, which contain the received message, or any other location that can be referenced by the receive operation. It does not indicate that the matching send operation has completed. The call returns a handle to an opaque object that contains information on the completed operation -- the {\bf return status} object. \mpifunc{MPI\_STATUS (handle, flag, return\_handle)} \begin{description} \item[IN handle] communication handle \item[OUT flag] logical \item[OUT return\_handle] handle that is associated with return status object. \end{description} A call to {\bf MPI\_STATUS} returns {\tt flag=true} if the operation identified by {\tt handle} is complete. In such case, the return handle points to an opaque object that contains information on the completed operation. It returns {\tt flag=false}, otherwise. In such case, the value of the return handle is undefined. A call to {\tt MPI\_WAIT} blocks only the executing thread. If the executing process is multithreaded, then other threads within the process can be scheduled for execution. The use of a blocking receive operation ({\tt MPI\_WAIT}) allows the operating system to deschedule the blocked thread and schedule another thread for execution, if such is available. The use of a nonblocking receive operation ({\tt MPI\_STATUS}) allows the user to schedule alternative activities within a single thread of execution. The intended implementation of {\tt MPI\_STATUS} is for that operation to return as soon as possible. However, if repeatedly called for an operation that is enabled, it must eventually succeed. The return status object for a send operation carries no information. The return status object for a receive operation carries information on the source, tag and length of the received message. These fields are required because the receive operation may have specified {\tt DONTCARE} in either source or tag field, and the message may have been shorter than the receive buffer. \mpifunc{MPI\_QUERY( handle, count, source, tag)} \begin{description} \item[IN handle] handle to return status object \item[OUT count] number of elements received. \item[OUT source] rank of message sender in message context (integer). \item[OUT tag] tag of received message (integer). \end{description} \mpifunc{MPI\_CREATE\_STATUS( handle)} Creates a new return status object. \begin{description} \item[OUT handle] handle to return status object. \end{description} \discuss{ The use of a return status object, rather than a list of parameters may simplify the use of MPI routines, if the values stored in the object are seldom checked. A predefined return status object should be provided, to ease programming. The main reason for the use of a return-status object is that one wants to be able to use the same {\tt MPI\_WAIT} and {\tt MPI\_STATUS} calls for checking on different types of operations, e.g. both sends and receives, and perhaps new operations in the future. These different operations return different information (albeit we could decree that a send returns the same information as a receive). The issue of memory allocation for return\_status handles is not yet solved. There is a desire that the user will be able to allocate space for them (e.g., on the stack). We return number of elements received, rather than bytes received because ``number of elements'' is closer to the application semantic level (user does not need to be aware of the size of elements), and is more invariant. The return status object returns the number of {\bf elements} received. If there was no truncation, then this is equal to the number of elements sent. On the other hand, the number of bytes collected from the sender memory, the number of bytes sent over the wire, and the number of bytes stored in the receiver memory may all be different, in a heterogeneous environment. The number of elements sent can be computed form the send buffer descriptor; the number of elements received can be computed from the receive buffer descriptor and the length (in bytes) of the received message. {\tt MPI\_STATUS} and {\tt MPI\_QUERY} are not good names -- they can be easily confused. Leslie Hart suggests {\tt MPI\_TEST} or {\tt MPI\_CHECK} for the first. Perhaps we should replace {\tt MPI\_STATUS} with {\tt MPI\_TEST} and {\tt MPI\_QUERY} with {\tt MPI\_STATUS}. } \subsection{Multiple Completions} It is convenient to be able to wait for the completion of any or all the operations in a set, rather than having to wait for a specific message. A call to {\tt MPI\_WAITANY} or {\tt MPI\_STATUSANY} can be used to wait for the completion of one out of several operations; a call to {\tt MPI\_WAITALL} can be used to wait for all pending operations in a list. \mpifunc{MPI\_WAITANY ( list\_of\_handles, count, index, return\_handle)} Blocks until one of the operations associated with the communication handles in the array has completed. Returns the index of that handle in the array, and returns the status of that operation in the object associated with the return\_handle. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[IN count] list length (integer) \item[OUT index] index of handle for operation that completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed. \end{description} The successful execution of {\tt MPI\_WAITANY(list\_of\_handles, index, return\_handle)} has the same effect as the successful execution of {\tt MPI\_WAIT(handle[i], return\_handle)}, where {\tt i} is the value returned by {\tt index} and {\tt handle[i]} is the {\tt i}-th handle in the list, and the cancellation of all remaining wait operations. If more then one operation is enabled and can terminate, one is arbitrarily chosen (subject to the restrictions on operation termination order, and fairness, see Section~\ref{subsec:correct}). {\tt MPI\_WAITANY ( list\_of\_handles, count, index, return\_status\_handle)} is \begin{verbatim} {MPI_WAIT (handle[0], return_handle); index = 0} || ... || {MPI_WAIT (handle[count-1], return_handle); index = count-1} \end{verbatim} (``$||$'' indicates choice; one of the alternatives is chosen, nondeterministically.) \mpifunc{MPI\_STATUSANY ( list\_of\_handles, count, index, return\_handle)} Causes either one or none of the operations associated with the communication handles to return. In the former case, it has the same return semantics as a call to {\tt MPI\_WAIT\_ANY}. In the latter case, it returns a value of --1 in {\tt index} and {\tt return\_handle} is undefined. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[IN count] list length (integer) \item[OUT index] index of handle for operation that completed, or -1 if none completed (integer). \item[OUT return\_handle] handle that is associated with return status object. Set to return status of operation that completed, if any; undefined when {\tt index = -1}. \end{description} \mpifunc{MPI\_WAITALL(list\_of\_handles, count, list\_of\_return\_handles)} Blocks until all communication operations associated with handles in the list complete, and return the status of all these operations. The parameters are: \begin{description} \item[IN list\_of\_handles] list of handles to communication objects. \item[IN count] lists length (integer) \item[OUT list\_of\_return\_handles] Must have the same length as the first list. Each return status object is set to the return status of the corresponding operation in the first list. \end{description} \discuss{ The fairness requirement given in Section~\ref{subsec:correct} implies that the use of {\tt WAIT\_ANY} cannot lead to starvation: If the sending process has issued a send complete operation, and the receiving process repeatedly post a receive for the message sent, then that message must be eventually received. Section~\ref{subsec:correct} has not yet been discussed. The fairness requirement can be attacked either for being too weak (eventually is not good enough), or too strong (hard to implement). In the later case, if the requirement of fairness is dropped from MPI implementations, then some mechanism need be provided to the user to achieve fairness by his or her own devices. One such proposal is to be able to specify a rotating priority order on the operations posted by a {\tt MPI\_WAITANY}: search the list sequentially, starting from a user specified position. } \section{Blocking Communication} Blocking send and receive operations combine all communication suboperations into one call. The operation returns only when the communication completes and no communication object persists after the call completed. However, the buffer descriptor object needs be created ahead of the call. We use the following naming convention for such operations: \[ \left[ \begin{array}{c} - \\ \bf R \\ \bf S \end{array} \right] \left[ \begin{array}{c} \bf SEND \\ \bf RECV \end{array} \right] \] The first letter (void, {\bf R} or {\bf S}) indicates the start mode (standard, ready, or secure). Only two of the combinations (standard and secure) are meaningful for receives. \mpifunc{MPI\_SEND (buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} \mpifunc{MPI\_RSEND (buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_READY, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} \mpifunc{MPI\_SSEND (buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle, null) \end{verbatim} \mpifunc{MPI\_RECV(buffer\_handle, source, tag, context, return\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} \mpifunc{MPI\_SRECV(buffer\_handle, source, tag, context, return\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) MPI_WAIT(handle,return_handle) \end{verbatim} \implement{ While these functions can be implemented via calls to functions that implement suboperations, as described in this subsection, an efficient implementation may optimize away these multiple calls, provided it does not change the behavior of correct programs. } \discuss{ We use a different function name, rather than an additional mode parameter, in order to save on an additional parameter (performance and user convenience). Lyndon and perhaps other prefer fewer functions and an additional parameter. } \section{Nonblocking Communication} Nonblocking send and receive operations combine the first two suboperations ({\tt INIT} and {\tt START}) into one call. They use ephemeral communication objects, so that the operation is completed, and the associated resources are freed, by using one of the functions {\tt MPI\_WAIT, MPI\_STATUS, MPI\_WAITANY, MPI\_STATUSANY}, or {\tt MPI\_WAITALL}. Here, too, a buffer object has to be created ahead of the communication initiation operation. We use the same naming convention as for blocking operations: a prefix of {\bf R} ({\bf S}) indicates the {\tt READY} ({\bf SECURE}) mode. In addition, a prefix of {\bf I} is used to indicate {\em immediate} (i.e., nonblocking) execution. \mpifunc{MPI\_ISEND (handle, buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_IRSEND (handle, buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_READY, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_ISSEND (handle, buffer\_handle, dest, tag, context)} is \begin{verbatim} MPI_INIT_SEND(handle, buffer_handle, dest, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_IRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_STANDARD, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \mpifunc{MPI\_ISRECV(handle, buffer\_handle, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_INIT_RECV(handle, buffer_handle, source, tag, context, MPI_SECURE, MPI_EPHEMERAL) MPI_START(handle) \end{verbatim} \section{Contiguous Block Communication Operations} The most frequent type of buffer used is a buffer with one contiguous component. We specialize the functions in the two previous subsections to this case, thus avoiding the need for the creation of a buffer descriptor object. We use the same naming scheme used in the previous subsections, and append a {\bf C} in the function name, for {\tt CONTIGUOUS}. \mpifunc{MPI\_SENDC (start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_SEND (buffer_handle, count, dest, tag, context) \end{verbatim} \mpifunc{MPI\_RSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_RSEND( buffer_handle, count, dest, tag, context) \end{verbatim} \mpifunc{MPI\_SSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_SSEND( buffer_handle, count, dest, tag, context) \end{verbatim} \mpifunc{MPI\_RECVC (start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_RECV( buffer_handle, source, tag, context, return_status_handle) \end{verbatim} \mpifunc{MPI\_SRECVC (start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_SRECV( buffer_handle, source, tag, context, return_status_handle) \end{verbatim} \mpifunc{MPI\_ISENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_ISEND( handle, buffer_handle, dest, tag, context) \end{verbatim} \mpifunc{MPI\_IRSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_IRSEND( handle, buffer_handle, dest, tag, context) \end{verbatim} \mpifunc{MPI\_ISSENDC (handle, start, count, datatype, dest, tag, context)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_ISSEND( handle, buffer_handle, dest, tag, context) \end{verbatim} \mpifunc{MPI\_IRECVC(handle, start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER (buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_IRECV( handle, buffer_handle, source, tag, context) \end{verbatim} \mpifunc{MPI\_ISRECVC(handle, start, count, datatype, source, tag, context, return\_status\_handle)} is \begin{verbatim} MPI_CREATE_BUFFER( buffer_handle, MPI_EPHEMERAL) MPI_APPEND( buffer_handle, start, count, datatype) MPI_ISRECV( handle, buffer_handle, source, tag, context) \end{verbatim} \section{Probe and Cancel} The {\tt MPI\_PROBE} operation allows incoming messages to be checked for, without actually receiving them. The user can then decide where to receive them, based on the information returned by the probe (basically, the information on the message envelope). An additional function, {\tt MPI\_GET\_LEN} allows the amount of storage needed to receive the message to be computed, when this length is not readily computed from the information returned by {\tt MPI\_PROBE}. The {\tt MPI\_CANCEL} operation allows pending communications to be cancelled. This is required for cleanup. Posting a send or a receive ties user resources (send or receive buffers), and a cancel may be needed to free these resources gracefully. \mpifunc{MPI\_PROBE( source, tag, context, flag, datatype, return\_status)} \begin{description} \item[IN source] rank in context of source, or {\tt MPI\_ANY\_SOURCE} (integer). \item[IN tag] user tag for messages received with this handle, or {\tt MPI\_ANY\_TAG} (integer). \item[IN context] context of messages received with this handle. \item[OUT flag] (logical) \item[IN type] assumed type of data in message (status variable). \item[OUT return\_handle] handle that is associated with return status object. \end{description} {\tt MPI\_PROBE} returns {\tt flag = true} if there is a message that can be received and that matches the pattern specified by the parameters {\tt source}, {\tt tag}, and {\tt context}. It returns {\tt flag = false}, otherwise. If {\tt MPI\_PROBE} returns {\tt flag = true}, then the length, source and tag of the message matched are returned in the return status object. These are the same values that would have been returned by a call to {\tt MPI\_RECV} or to {\tt MPI\_SRECV} executed at the same point in the program (with a caveat concerning length; see below). These values can be decoded from the return status object using the {\tt MPI\_RETURN\_STAT} function. The value returned in the return status object is undefined if {\tt flag=false}. The length value returned by the return status object is (correctly) the number of elements in the message, provided that all elements in the message are of the type specified by the {\tt type} parameter; otherwise the length value returned is undefined. A subsequent receive executed with the same context, and the source and tag returned by the call to {\tt MPI\_PROBE} will receive the message that was matched by the probe, if no other intervening receive occurred after the probe. If the receiving process is multithreaded, it is the user responsibility to ensure that the last condition holds. \discuss{ Do we want {\tt MPI\_PROBE} to return mode (standard or secure)? If yes, we need to carry mode information with messages (it's deja vu all over again). If not, then it is up to the user to encode this information (in the tag) so that receiver can decide whether to use a secure or standard send. MPI guarantees that successive messages sent from a source to a destination within the same context are received in the order they are sent. Thus, MPI must support, either explicitly or implicitly, a FIFO structure to manage messages between each pair of messages, for each context. {\tt MPI\_PROBE} returns information on the first matching message in this FIFO; this will also be the message received by the first subsequent receive with the same source, tag and context as the message matched by {\tt MPI\_PROBE}. Message passing in MPI can be implemented without appending type information to messages. A message is merely a string of bytes and the interpretation of these bytes into a sequence of typed elements is done using the information in the buffer descriptors at each end. The ability to use such implementation strategy is deemed to be an important goal. In such implementation, when a message arrives, it is not be known how many elements it contains, or even how much storage is needed to receive that message (because of possible representation conversion in a heterogeneous environment). The probe function cannot use a buffer descriptor; this defeats the purpose of probing in order to decide where to receive a message. Therefore, probe cannot, in general, return correct length information. Still, it is often the case that probe is used to decide how much storage to allocate in order to receive a message. Encoding such information in the message tag is deemed to be too awkward, and it is deemed important for {\tt MPI\_PROBE} to return some useful size information. The current definition of {\tt MPI\_PROBE} is a compromise between these two goals. For the most common case of messages where all entries have the same type {\tt MPI\_PROBE} returns the correct length information; the more esoteric case is handled by the {\tt MPI\_GET\_LEN} that is described below. The current solution saves us the need for one additional word per message that would otherwise be needed to transfer the message length (in elements) with the message. There was a request for a blocking probe function. } \mpifunc{MPI\_GET\_LEN( count, return\_status, buffer\_descriptor)} Computes the number of elements that were to be received, if the message that is associated with the return status handle would be received in the buffer associated with the buffer descriptor handle. \begin{description} \item[OUT count] number of elements that were to be received (integer) \item[IN return\_status] handle to return status descriptor \item[IN buffer\_descriptor] handle to buffer descriptor \end{description} \discuss { In order to support this function, an additional field is needed in the return status object, i.e., number of bytes in the incoming message matched by probe, or value of datatype parameter provided by probe (unless we add this parameter to the {\tt MPI\_GET\_LEN} function, which is too ugly to bear). } \mpifunc{MPI\_CANCEL( handle, flag)} \begin{description} \item [IN handle] handle to communication object \item[OUT flag] (logical) \end{description} A call to {\tt MPI\_CANCEL} cancels a pending communication operation (send or receive). The call returns {\tt flag = true} if the cancel operation succeeded, {\tt flag = false} otherwise. It must be the case that either the cancel operation succeeds or that the pending communication operation completes (but not both). If a communication operation was cancelled successfully, the effect is as if this operation was never executed. If a send was cancelled successfully then the message sent will not be received, the receive buffer of any posted receive for that message will not be altered, and any such receive has to be satisfied by another send; if a receive was cancelled successfully, then the receive buffer posted will not be altered, and any send matching this receive has to be satisfied by another receive. \implement{ There is not expectation that a cancel operation will be fast. } \section{Correctness} \label{subsec:correct} \discuss{The material in this section has not yet been discussed by MPIF. Some or all of it is likely to move to Section~\ref{sec:formal}. It is incorporated here for completeness.} \subsection{Order} MPI preserves the order of messages within the same context, between any fixed pair of processes. In other words, if process A executes two successive send {\tt start} suboperations within the same context, process B executes two successive receive {\tt start} operations, and both receives match either sends, then the first receive will receive the message sent by the first send, and the second receive will receive the message sent by the second send. Thus, if a two messages from the same source can satisfy a pending receive, the first message sent is accepted; if a message can satisfy two pending receives, the first receive posted is satisfied. The last paragraph assumes that the send {\tt start} operations are ordered by the program order at process A, and the receive {\tt start} operations are ordered by the program order at process B. If a process is multithreaded and the operations are executed by distinct threads, then the semantics of the threaded system may not define an order between the two operations, in which case the condition is void. \subsection{Progress and Fairness} We can model the execution of MPI programs as an interaction between executing processes that execute each their own program, and the {\bf communication subsystem}. The communication subsystem may have various constraints on the amount of resources it can use. E.g.: Bounds on the number and total sizes of active communication objects. Such bound can be global, per node, or per pair of communicating nodes. Bounds on the number and total sizes of messages buffered in the system. Such bound can, again, be global, per node, or per pair of communicating node. In addition, a message may be buffered at the sender, at the receiver, at both, or perhaps at another place altogether. Thus, it will be difficult to set rules on resource management of the communication subsystem. However, it is generally expected that implementers will provide information on the mechanism used for resource allocation, and that query and set functions will allow to query and possibly control the amount of available resources. We provide in this section a set of minimal requirements on the communication subsystem. Programs that execute on any subsystem that fulfils these minimal requirements are {\bf safe} and will port to any MPI implementation. {\bf Unsafe} programs may execute on some MPI implementations, depending on the amount of available resources and the implementation used for the MPI communication subsystem. Finally {\bf erroneous} programs never execute correctly. (While it is desirable to detect erroneous programs, it is not possible to do so at compile time, and often prohibitive to do so a run time. Thus, the document does not specify a behavior for erroneous programs, although the desired behavior is to return a useful error message.) \begin{enumerate} \item If a process executes an {\tt INIT} operation, then the operation eventually succeeds, or a {\em resource exception} occurs. The standard does not specify when a resource exception is allowed to occur. It is expected that an operational definition will be made available, in the form of test programs that have to execute with no resource exceptions. It is highly desirable to have generous bounds on the number of concurrently active communication objects each process may have, so that, in practice, {\tt INIT} operations will always be guaranteed to succeed. \item Each process can initiate a communication operation for each active communication object. I.e. correct {\tt START} operations always succeed (eventually). \item A send operation is {\bf enabled} if the sending process has issued a {\tt COMPLETE} operation and the receiving process has issued a {\tt START} operation for a matching receive. Symmetrically, a receive operation is {\bf enabled} if the receiving process has issued a {\tt COMPLETE} operation and the sending process has issued a {\tt START} operation for a matching send. An enabled operation may become {\bf disabled} either because it completes successfully or, in the case of a receive, because the matching message is successfully received by another receive operation or, in the case of a send, because the matching receive successfully receives another message. {\bf An enabled operation either completes successfully or becomes permanently disabled.} I.e., an enabled operation either eventually succeeds, or becomes disabled (progress); and an operation cannot be enabled infinitely many times without ever succeeding (fairness). \item A {\tt FREE} operation always succeeds (eventually). \end{enumerate} The four conditions guarantee progress in the communication subsystem. The third condition guarantee (weak) fairness among competing communication requests. Examples (involving two processors with ranks 0 and 1) The following program is safe, and should always succeed. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDC(sendbuf, count, 1, tag, context) CALL MPI_RECVC(recvbuf, count, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVC(recvbuf, count, 0, tag, context) CALL MPI_SENDC(sendbuf, count, 0, tag, context) END IF \end{verbatim} The following program is erroneous, and should always fail. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_RECVC(recvbuf, count, 1, tag, context) CALL MPI_SENDC(sendbuf, count, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_RECVC(recvbuf, count, 0, tag, context) CALL MPI_SENDC(sendbuf, count, 0, tag, context) END IF \end{verbatim} The receive operation of the first process must complete before its send, and can complete only if the matching send of the second processor is executed; the receive operation of the second process must complete before its send and can complete only if the matching send of the first process is executed. This program will deadlock. The following program is unsafe, and may succeed or fail, depending on implementation. \begin{verbatim} CALL MPI_RANK(rank, context) IF (rank.EQ.0) THEN CALL MPI_SENDC(sendbuf, count, 1, tag, context) CALL MPI_RECVC(recvbuf, count, 1, tag, context) ELSE ! rank.EQ.1 CALL MPI_SENDC(sendbuf, count, 0, tag, context) CALL MPI_RECVC(recvbuf, count, 0, tag, context) END IF \end{verbatim} The message sent by each process has to be copied out before the send operation returns and the receive operation starts. For the program to complete, it is necessary that at least one of the two messages sent is buffered out of either processes' address space. Thus, this program can succeed only if the communication system has sufficient buffer space to buffer {\tt count} words of data. If additional requirements will become part of the standard (e.g., bounds on the minimal number of concurrently active handles that need be supported, then further programs become safe. \section{Missing} \paragraph*{send-receive} a function of the form (for blocking communication) {\tt MPI\_SEND\_RECV( send\_buffer, destination, recv\_buffer, source, tag, context)} It may be superfluous because the collective communication shift has similar functionality. Yet it allows to generate more easily communication patterns that would lead to deadlock if implemented naively, with blocking communication. An important variant is the case where {\tt send\_buffer = recv\_buffer} (this can either be a subcase of the general send\_recv function, or a separate function, if send buffer and receive buffer are required to be disjoint in the general function). This introduces new functionality that is not available, otherwise, and which seems to be often required. An {\tt exchange} is the special case where source = destination. \end{document} >From weeks@mozart.convex.com Wed Jun 9 15:10:24 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15090; Wed, 9 Jun 93 15:10:24 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12755; Wed, 9 Jun 93 15:10:03 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01731; Wed, 9 Jun 93 08:09:24 -0500 Received: by mozart.convex.com (5.64/1.28) id AA29119; Wed, 9 Jun 93 08:11:37 -0500 Date: Wed, 9 Jun 93 08:11:37 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091311.AA29119@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 14 mpi mail #1 of 1 Status: R >From snir@watson.ibm.com Fri May 14 00:15:16 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA28640; Fri, 14 May 93 00:14:55 -0500 Received: from watson.ibm.com by convex.convex.com (5.64/1.35) id AA02615; Fri, 14 May 93 00:15:01 -0500 Message-Id: <9305140515.AA02615@convex.convex.com> Received: from YKTVMV by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 1089; Fri, 14 May 93 01:14:38 EDT Date: Fri, 14 May 93 01:14:38 EDT From: "Marc Snir" To: weeks@convex.com, tony@cs.msstate.edu Status: R % MPI Authors: % This is MY version of YOUR chapter. It has a wrapper so that you % should be able to simply LaTeX this. % % Please work from this text so that we are in synch. % % --Steve Otto \documentstyle[twoside]{article} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newcommand{\implement}[1]{\vspace{\discussSpace} {\small {\bf Implementation note:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Maps, Groups and Contexts } \author{ Lyndon Clarke, Mark Sears, Tony Skjellum, Marc Snir} \date{May 13, 1993} \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \section{Introduction} Goals Assumptions (closed universe -- to be extended later) Main objects: Maps and communicators \section{Maps} A {\bf map} is a one-to one-mapping $f: 0...m-1 \rightarrow 0...n-1$. A possible representation for such map is a list of $m$ integers in the range $0...n-1$; it is often convenient to think of a map as being such list. Another possible representation is as an algorithmic specification of the map (e.g., hash function). We represent a map by an opaque map object. A {\bf group} is an ordered set of processes (where a process is a primitive of MPI discussed elsewhere). Each process in a group is associated with a {\bf rank}, starting from zero. We shall use maps in order to represent groups. If processes are numbered from $0$ to $n-1$, then a map $f: 0...m-1 \rightarrow 0...n-1$ represents a subset group of $m$ processes. Note that a map, by itself, does not represent a group; it does so only relative to a pre-defined containing group. \section{Context} A {\bf context} is the MPI mechanism for partitioning communication space. A send made in a context cannot be received in another context. Contexts are identified in MPI using {\bf context\_id}'s. \section{Context ID's} A {\bf context\_id} is a integer-like type that can be shared between processes in MPI. \section{Communicators} All MPI communication (point to point and collective) functions use {\bf communicators} {this replaces the word {\bf context} everywhere in current pt2pt and collcomm drafts} to provide a context for the communication. Communicators bring together the concepts of group and context. The source and destination of a message is identified by the rank of that process within the group, and communication using this communicator is restricted to processes within this group. Thus, the communicator restricts the ``spatial'' scope of communication, and provides local process addressing. \discuss{ The current communicators correspond to C1 objects in the previous draft. Future extensions are expected to include C2 objects, that encapsulate two groups: the sender group and the receiver group. } Communicators are represented by opaque {\bf communicator objects}. Such objects are opaque, and cannot be transferred from one node to another. However, communicator objects can be identified using {\bf context\_id}'s; these can be communicated across processes. \discuss{ Is the datatype of context\_id's {\tt INTEGER} or MPI-defined?} Let {\tt comm} be a communicator that is associated with a group $G$ of size $n$, and let $f:0..m-1 \rightarrow 0..n-1$ be a map. Then the pair {\tt (comm, f)} defines a subgroup {\tt G(comm,f)} of $G$, namely the subgroup of processes with ranks $f(0),...,f(m-1)$. We say that {\tt map} is the map of the subgroup {\tt G(comm,f)} within {\tt comm}. An initial communicator {\tt MPI\_COMM\_INIT} is defined when the program starts. its associated group contains all processes that start the computation. \section{Operations on maps} \mpifunc{domain = MPI\_MAP\_DOMAIN(map)} \begin{description} \item[IN map] handle to map object \item[OUT domain] domain of map (integer) -- the domain is the set $0...{\tt domain} -1$. \end{description} \mpifunc{element = MPI\_MAP\_ELEMENT( map, rank)} \begin{description} \item[IN map] handle to map object \item[IN rank] map argument (integer) \item[OUT element] value of map on argument {\tt rank} (integer) \end{description} \mpifunc{rank = MPI\_MAP\_RANK( map, element)} \begin{description} \item[IN map] handle to map object \item[IN element] element in range of map \item[OUT rank] inverse image of element, MPI\_INVALID\_ELEMENT results if element is not in range \end{description} \mpifunc{MPI\_MAP\_FLATTEN(map, array\_of\_integer, domain)} \begin{description} \item[IN map] handle to map object \item[OUT array\_of\_integer] list of values in the range of {\tt map} \item[OUT domain] length of returned list -1 (integer). \end{description} \discuss{Definition of ``mpi\_map\_flatten'' is not definitive.} \section{Map constructors} The map constructors may be either local or may require communication. \subsection{Local map constructors} The execution of the following operation does not require interprocess communication. Examples of local map constructors are as follows: \mpifunc{MPI\_MAP\_BUILD(map, array\_of\_integer)} \begin{description} \item[OUT map] handle to map object \item[IN array\_of\_integer] list of values in the range of {\tt map} \item[IN domain] length of returned list -1 (integer). \end{description} \discuss{Syntax may change, (see also ``flatten'')} \mpifunc{MPI\_MAP\_CONCAT(map1, map2, mapout)} \begin{description} \item[IN map1] map object handle \item[IN map2] map object handle \item[OUT mapout] map object handle \end{description} If map1 and map2 are distinct, then is the concatenation of the maps. $f(i) = f_1(1) \rm{if} i \le {\rm dom}(f_1), {\rm else} = f_2(i-{\rm dom}(f_1))$. \subsection{Collective map constructors} The execution of the following operations require collective communication within a group. Examples are as follows: \mpifunc{MPI\_MAP\_SPLIT(comm, mapin, key, index, mapout)} This is a collective function that is called by all processes in the group associated with {\tt (comm, mapin)}. All calling processes provide the same values for the parameters {\tt comm} and {\tt mapin}. A separate group of processes is formed for each distinct value of {\tt key}, with the processes ordered according to the value of {\tt index}. Each process in such group is returned in {\tt mapout} the map of this group within {\tt comm}. \begin{description} \item[IN comm] communicator object handle \item[IN mapin] map object handle \item[IN key] (integer) \item[IN index] (integer) \item[OUT mapout] map object handle \end{description} \section{Operations on communicators} \mpifunc{MPI\_ALLOC\_CONTEXT(comm, map, array\_of\_contextids, len)} Allocates an array of contextids. This is a collective operation that is executed by all processes in the group defined by {\tt (comm, map)}. The context\_ids that are returned are unique within the group associated with {\tt (comm, map)}. The array returned is the same on all processes that call the function (same order, same number of elements). \begin{description} \item[IN comm] communicator object handle \item[IN map] map object handle \item[OUT array\_of\_contextids] \item[IN len] number of context\_id's to allocate \end{description} \mpifunc{MPI\_MAKE\_COMM(comm, map, context\_id, newcomm)} Creates a new communicator object, which is associated with the group defined by {\tt (comm, map)}, and the contextid {\tt contextid}. The operation does not require communication. However, the new communicator object should not be used for communication between two processes unless they both have called the function. It is erroneous to invoke this function twice in the same process with the same context\_id (unless the context\_id has been freed). \discuss{Mark Sears disagrees with this!} \begin{description} \item[IN comm] communicator object handle \item[IN map] map object handle \item[IN contextid] contextid \item[OUT newcomm] communicator object handle \end{description} \mpifunc{MPI\_SAFEMAKE\_COMM(comm, map, newcomm)} is \begin{verbatim} MPI_ALLOC_CONTEXT(comm, map, context_id, 1); MPI_MAKE_COMM(comm, map, context_id, newcomm); ``MPI_SYNCH(comm,map)'' /* by which we mean a synchronization operation in the subgroup defined by (comm,map) */ \end{verbatim} \discuss{ {\tt MPI\_SYNCH(comm,map)} is important to bootstrap -- but can be coded using point-to-point.} {\tt MPI\_SAFEMAKE\_COMM} creates a new communicator object and synchronizes all processes in the group associated with this object. Thus, when this call returns, the new communicator can be safely used. \mpifunc{MPI\_MAP(comm, subcomm, map)} Returns a map such that {\tt (comm, map)} is the group associated with {\tt subcomm} ({\em i.e.}, returns the indices in the group associated with {\tt comm} for all the processes in the group associated with {\tt subcomm}). \begin{description} \item[IN comm] communicator object handle \item[IN subcomm] communicator object handle \item[OUT map] map object handle \end{description} \mpifunc{MPI\_CONTEXTID(comm, contextid)} Returns the context\_id associated with the communicator {\tt comm}. \begin{description} \item[IN comm] communicator object handle \item[OUT contextid] contextid \end{description} \discuss{Possible addition of functions are as follows: {\tt MPI\_PARENT(comm)} returns the communicator used to create {\tt comm}; } \end{document} >From weeks@mozart.convex.com Wed Jun 9 15:10:57 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15095; Wed, 9 Jun 93 15:10:53 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12679; Wed, 9 Jun 93 15:06:13 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01598; Wed, 9 Jun 93 08:04:35 -0500 Received: by mozart.convex.com (5.64/1.28) id AA28422; Wed, 9 Jun 93 08:06:42 -0500 Date: Wed, 9 Jun 93 08:06:42 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091306.AA28422@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 12 mpi mail #6 Status: R A postscript file from snir or otto (I think this is different from the LaTeX source I forwarded just before this one...) %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: PT2PT-V3.DVI.* %%Pages: 29 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 6 54 df20 DI50 DI<12E0B3B3B3B1EAFFFCA3 0E4A73811C>I<131CB3B3B3B1EAFFFCA30E4A80811C>I E /Fb 1 4 df<12041206A2EAC660EA F7E0EA3F80EA1F00EA3F80EAF7E0EAC660EA0600A212040B0D7E8D11>3 D E /Fc 2 36 df<1560A28181A2150EB71280A2C8EA0E001518A25D5DA2210E7E8F26>33 D<1203B3A7EA8304EAE31CEA7B78EA1FE0EA0FC0EA0780EA03007E0E217E9913>35 D E /Fd 2 63 df<127012F8A3127005057D840C>58 D<12E01278121EEA0780EA01E0EA007813 1EEB0780EB01E0EB0078141EEC0780A2EC1E001478EB01E0EB0780011EC7FC1378EA01E0EA0780 001EC8FC127812E019187D9520>62 D E /Fe 4 107 df0 D15 D<12C012F0123C120FEA03C0EA00F0133C130FEB03C0EB00F0143C140FEC0380EC0F00143C14F0 EB03C0010FC7FC133C13F0EA03C0000FC8FC123C12F012C0C9FCA7B61280A219227D9920>21 D<12C0B3B3A502297B9E0C>106 D E /Ff 36 122 df<1238127C127EA2123E121EA2127C12F8 1260070A7A8414>44 D<127012F8A312700505798414>46 D48 D<12035AA25A123FB4FC12F712471207ACEAFFF0A30C177C9614>I65 DIIIII I<38FE3F80A338380E00A7EA3FFEA3EA380EA738FE3F80A311177F9614>II75 DI<38FE0FE0A3383B1B80A413BBA2EA39B3A313F3EA38E3A21303A538FE0FE0 A31317809614>I<38FE3F80A3383B0E00A4138E1239A213CEA31238A213EE136EA4EAFE3EA311 177F9614>IIIIIII<38FF8FF8A338 1C01C0AEEA1E03000E1380EA0F073807FF006C5AEA00F81517819614>I<38FE3F80A338380E00 A26C5AA56C5AA4EA0630EA0770A3EA0360A213E0A26C5A11177F9614>I<38FE0FE0A338380380 A50018130013E3EA1DF7A213B7A5EA0DB6A21316EA0F1EA31317809614>I<38FE3F80A3383C1E 00EA1C1CEA1E3CEA0E38A26C5AA2EA036013E0A26C5AA7EA07F0A311177F9614>89 D91 D 93 D97 D101 D<1203EA0780A2EA0300C7FCA4EAFF80 A31203AAEAFFFEA30F187D9714>105 D110 D112 D<1207A5EAFFFCA3EA0700A6130EA3139E13FC EA03F8EA00E00F157F9414>116 D<38FF3F80A3381C1C00A2120E5BA212071330A2EA0370A26C 5AA35BA3EA7B80127F90C7FC127E123C11187F8F14>121 D E /Fg 29 86 df<1230127812F81278127005057D840C>46 D<13181370EA03F0120FEA1C701200A213E0A6EA 01C0A6EA0380A61207EAFFF8A20D1C7C9B15>49 D<137E3801FF80380387C0EA0603EB01E0120F A2121FA2120E1200EB03C0A2EB0780EB0F00130E5B5B13605BEA038038070180120E381C030012 30EA7FFF485AA2131C7E9B15>I<137C48B4FC38038780EA0603000F13C0A3380E078012001400 130E133CEA01F8485AEA001C7F130FA4127012F8A2EAF01E12E05BEA7078EA3FE0EA0F80121D7D 9B15>II<380601C03807FF 80140013FC0006C7FCA25AA413F8EA0FFCEA1F0EEA1C0F7F121800001380A3EB0F00127012F013 0EEAE01EEAC01CEA6038EA70F0EA3FE0EA0F80121D7D9B15>I<1218383FFFC0A21480EA700138 6003001306485AC65A5BA25B5B12015B1203A248C7FCA25A120EA2121EA35AA31218121D7B9B15 >55 D<137C48B4FC38038780EA0703120EEB01C0A2381E0380A2EB0700EA0F8613DCEA07F81203 487EEA0E7E487EEA380F12707F12E0A31306130EEA701CEA3878EA1FF0EA0FC0121D7D9B15>I< 13F8EA03FCEA070EEA0E07121CA2003C138012381278A4130F1400A2EA381F5BEA1FEFEA0FDEEA 001E131CA2EA603812F05B485AEAC3C0EA7F80003EC7FC111D7C9B15>I<14301470A214F08013 01A2EB0378A21306147CEB0C3CA21318A21330A2497EA2EBFFFEA23801801EA23803001F801206 120F397FC0FFF012FF1C1D7F9C1F>65 D<380FFFFC14FF3900F00780EC03C0A2EC01E0A2D801E0 13C01403A2EC0780EC1F00EBFFFC5AEBC03E140FEC0780A3EA0780A4EC0F00141E380F007CB55A 14E01B1C7E9B1D>I<903807F030EB1FFC90387C0E603901F003E0EA03C038078001EA0F00000E 1300001E14C05AA25A1500A25AA5EC0180EC03001270127814066C5B001C131C000F5B3807C0E0 6CB45AD8007EC7FC1C1E7C9C1E>I<380FFFFCECFF803900F007C0EC01E0EC00F015701578485A 1538A3153C1538485A1578A3157015F0484813E0140115C0EC03801407EC0E00380F007CB512F0 14C01E1C7E9B20>I<000FB512E0A23800F003EC01C01400A33801E060A31500EBE1E013FF485B 13C113C0A215C0ECC18038078001A2EC0300A25C140E380F003EB512FE5C1B1C7E9B1C>I<000F B512C0A23800F007EC03801401A3EA01E01461A2EC600014E013E148B45AA213C113C0A3380781 800180C7FCA5120FEAFFFC5B1A1C7E9B1B>I<903807F030EB1FFC90387C0E603901F003E0EA03 C038078001EA0F00000E1300001E14C05AA25A1500A25AA3ECFFF0A2EC0F80150012701278A27E 001C5B000F133E3807C0F63803FFC2C66CC7FC1C1E7C9C21>I<390FFF9FFE143F3900F003C0A5 3901E00780A590B5FC481400EBC00FA53807801EA6000F133E39FFF3FFE015C01F1C7E9B1F>I< 380FFF80A23800F000A5485AA6485AA6485AA6120FEAFFF8A2111C7F9B0F>I<3801FFF0A23800 0F00A5131EA65BA65BA2127012F8A2485A485AEAE1C0EA7F80001FC7FC141D7E9B15>I<390FFF 87FEA23900F003E0EC0180EC030014065C48485A14705CEBE1C013E313E73803CDE013D9EBF1F0 13E013C01478EA078080A280A280000FEB1F8039FFF87FF013F01F1C7E9B20>I<380FFFC0A238 00F000A5485AA6485AA41406140CEA0780A2141C141814381478380F01F8B512F0A2171C7E9B1A >II<390FF80FFEA23900FC01E0EC00C0A213DEA239018F0180A214811387A2EB83C1 390303C300EB01E3A3EB00F3A20006137EA2143EA3141E001E131C38FFE00C13C01F1C7E9B1F> II<380FFFFC14FF3900F00F80140315C01401A23801E003A315801407EC0F 003803C03EEBFFF814E001C0C7FCA3485AA6120FEAFFF85B1A1C7E9B1C>I<380FFFF814FE3900 F00F80140315C01401A23801E003A3EC0780EC0F00143E3803FFF85CEBC07880A2143E3807803C A51518000F1430EAFFF89038F01FE0C7EA07C01D1D7E9B1F>82 DI<001FB512F05A383C0781 0038EB806012701260A2EB0F0012C0A200001400A3131EA65BA6137C381FFFE0A21C1C7C9B1E> I<39FFF8FFE0A2390F001E00140CA4001E5BA6485BA6485BA300385BA2383C0180D81C03C7FCEA 0E0EEA07FCEA01F01B1D7A9B1F>I E /Fh 53 122 df<90381FE1F890B512FC3903F03F3E3807 C07EEB807C000F141C1500A5B612C0A2390F807C00AE397FE1FFC0A21F1D809C1C>11 D 13 D<13301360EA01C0EA038013005A120E121E121C123CA212381278A312F85AA97E1278A312 38123CA2121C121E120E7E7E1380EA01C0EA006013300C297D9E13>40 D<12C0126012387E120C 120E7E1380120313C0A2120113E0A313F01200A9120113E0A313C01203A2138012071300120E12 0C121C5A12605A0C297D9E13>I<1238127C12FE12FFA2127F123B120312071206A2120C121812 301220080F7D860D>44 DI<1238127C12FEA3127C123807077D860D>I< EA0FF0EA3FFCEA703EEAF03F12F8A3EA203EEA007C13F813F0EA01C0A21380A5C7FCA4EA0380EA 07C0EA0FE0A3EA07C0EA0380101D7D9C17>63 D65 DI<9038 1FE0209038FFF8E03803F80F3807E007380FC001EA1F8048C7FCA2007E1460A212FE1500A7007E 1460A27E15C06C7E390FC001803907E003003803F80E3800FFFCEB1FE01B1C7D9B22>IIII<90380FF00890387FFE383901FC07783907E001F8380FC0004848137848C71238A2481418 127E12FE1500A5EC7FFFA2007EEB01F8127F7EA2EA1F80EA0FC0EA07F03801FC0739007FFE7890 380FF818201C7D9B26>I<39FFFC3FFFA2390FC003F0AA90B5FCA2EBC003AC39FFFC3FFFA2201C 7E9B25>II76 DI<39FFE003FFA2390FF000307FEA0DFCEA0CFE137E7FEB1F8014C0 EB0FE0EB07F01303EB01F814FCEB00FE147F143FEC1FB015F0140F1407140314011400A2D8FFC0 13701530201C7E9B25>IIIII<3807F86038 1FFEE0EA3C07EA7801EA700012F01460A26C130012FEEAFFE0EA7FFE6C7E1480000F13C06C13E0 EA007FEB03F01301130012C0A214E07E38F001C0EAFC0338EFFF00EAC3FC141C7D9B1B>I<007F B512E0A238781F81007013800060146000E0147000C01430A400001400B03807FFFEA21C1C7E9B 21>I<39FFFC03FFA2390FC00030B3120715606C6C13E03901F001C03900FC078090387FFE00EB 0FF8201C7E9B25>I<3AFFFC01FF80A23A0FC00018006C6C5BA26D1370000314606D13E000015C 7F0000495AA2D97E03C7FCA2EB7F07EB3F06148EEB1F8C14CCEB0FD8A2EB07F0A36D5AA26D5AA2 211C7F9B24>I<3BFFFC7FFE0FFCA23B0FC007E000C081D9E003130100071680EC07F8D803F0EC 0300A29039F80CFC0700011506EC1CFE9039FC187E0E0000150CEC387F90397E303F18A290397F 601FB8013F14B002E013F0ECC00F011F5CA26D486C5AA2EC00036D5CA22E1C7F9B31>I<397FFE 1FFEA23907F001803803F8036C6C48C7FC000013066D5AEB7F1C6D5A14B0EB1FE0130FA26D7E6D 7E1307497EEB0CFEEB187EEB387F496C7EEB601F01C07F00016D7E496C7EEA03003AFFF03FFF80 A2211C7F9B24>I<3AFFFC01FF80A23A0FE00038006C6C13306C6C137015606C6C5B3800FE015D D97F03C7FCEB3F871486EB1FCEEB0FFC5C13076D5AAAEB3FFFA2211C7F9B24>I97 DIIII<133F3801FF803803E7C0EA07C7EA0F87EB8380EB8000A5EAFFF8A2EA0F80 AEEA7FF8A2121D809C0F>I<3803F0F0380FFFF8383E1F38383C0F30007C1380A4003C1300EA3E 1FEA1FFCEA33F00030C7FC12707EEA3FFF14C06C13E04813F0387801F838F00078A3007813F038 3E03E0381FFFC03803FE00151B7F9118>II<121E123FA25A7EA2121EC7FCA5B4FCA2121FAEEAFFE0A20B1E7F 9D0E>I107 DI<39FF1FC0FE 90387FE3FF3A1FE1F70F80903980FC07C0A2EB00F8AB3AFFE7FF3FF8A225127F9128>I<38FF1F C0EB7FE0381FE1F0EB80F8A21300AB38FFE7FFA218127F911B>II<38FF1FC0EBFFE0381FC1F8130014FC147C147EA6147C14FCEB80F8EBC1F0EB7FE0EB1F8090 C7FCA6EAFFE0A2171A7F911B>I114 DI<1203A35AA25AA2123FEAFFFCA2EA1F 00A9130CA4EA0F98EA07F0EA03E00E1A7F9913>I<38FF07F8A2EA1F00AC1301EA0F03EBFEFFEA 03F818127F911B>I<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01 F6A213FE6C5AA21378A2133016127F9119>I<38FFC7FCA2381F8180EA0F833807C700EA03EEEA 01FC5B1200137C13FEEA01DFEA039F38070F80380607C0380C03E038FF07FCA216127F9119> 120 D<38FFC1FCA2381F0060EB80E0000F13C013C03807C180A23803E300A2EA01F6A213FE6C5A A21378A21330A25B1270EAF8E0EAC0C0EAE380007FC7FC123E161A7F9119>I E /Fi 74 126 df<126012F0AF12601200A4126012F0A212600419779816>33 D38 D<13E01201EA07C013005A121E5A123812781270A312F05AA77E1270A312781238123C7E7E7E13 C0EA01E012000B217A9C16>40 D<12E07E127C121C121E7EEA0780120313C01201A313E01200A7 120113C0A3120313801207EA0F00121E121C127C12F05A0B217C9C16>III<1238127C127EA2123E120E121E123C127C12F812 60070B798416>II<127012F8A312700505788416>I48 DIII<137C13FC13DC1201EA039CA2EA071C120F120E121E123C1238127812F0 B512E0A338001C00A53801FFC0A313197F9816>II<13F8EA 03FEEA0FFFEA1F0F123E123CEA78060070C7FC12F0EAF7F8EAFFFEA2EAF80F38F00780A2EAE003 12F0A21270EA7807EB0F006C5AEA1FFEEA0FF8EA03E011197E9816>I<12E0B51280A338E00F00 131EC65A13381378137013F05B12015B12035BA3120790C7FCA7111A7E9916>III<1238127CA312381200A81238127CA3123C121C 123C123812F812F012600618799116>59 DII<13E0487EA213B0A2EA03B8A31318EA071C A5EA0E0EA2EA0FFEA2487EEA1C07A3387F1FC000FF13E0007F13C013197F9816>65 DI<3801F180EA07FF5AEA1F0FEA3C071278130312 7000F0C7FC5AA77E387003801278A2EA3C07381F0F00EA0FFE6C5AEA01F011197E9816>II<387FFFC0B5FC7EEA1C01A490C7FCA2131CA2EA1FFCA3EA1C1CA290C7 FC14E0A5EA7FFFB5FC7E13197F9816>II I<387F1FC038FFBFE0387F1FC0381C0700A7EA1FFFA3EA1C07A9387F1FC038FFBFE0387F1FC013 197F9816>II<387F0FE038FF8FF0387F0FE0381C07 80EB0F00131E131C133C5B5BEA1DE07F121F7F1338EA1E3C131CEA1C1E7F7F14801303387F07E0 38FF8FF0387F07E01419809816>75 D I<38FC07E0EAFE0FA2383A0B80EA3B1BA413BBA2EA39B3A413F3EA38E3A21303A538FE0FE0A313 197F9816>I<387E1FC038FF3FE0387F1FC0381D07001387A313C7A2121CA213E7A31367A21377 A21337A31317EA7F1FEAFF9FEA7F0F13197F9816>IIIIII<387FFFE0B5FCA2EAE0E0A400001300AFEA07FC487E6C5A13197F9816> I<387F07F038FF8FF8387F07F0381C01C0B0EA1E03000E1380EA0F8F3807FF006C5AEA00F81519 809816>I<38FE0FE0EAFF1FEAFE0F38380380381C0700A4EA0E0EA4EA060CEA071CA4EA031813 B8A3EA01B013F0A26C5A13197F9816>I<38FC07E0EAFE0FEAFC07387001C0A300301380EA3803 A313E3EA39F3A213B300191300A61313EA1B1BEA0F1EA2EA0E0E13197F9816>I<387F1F80133F 131F380E1E00131CEA073C1338EA03B813F012015B120012017F120313B81207131CA2EA0E0EA2 487E387F1FC000FF13E0007F13C013197F9816>I<38FE0FE0EAFF1FEAFE0F381C0700A2EA0E0E A26C5AA3EA03B8A2EA01F0A26C5AA8EA03F8487E6C5A13197F9816>I91 D93 D95 D97 D<127E12FE127E120EA4 133EEBFF80000F13C0EB83E01301EB00F0120E1470A4000F13F014E01381EB83C013FF000E1300 EA067C1419809816>II<133F5B7F1307A4EA03E7EA0FFF123FEA3C1F487E1270 EAF00712E0A46C5AA2EA781FEA7C3F383FFFE0381FF7F03807C7E014197F9816>II<131FEB7F8013FFEA01E7EBC30013C0A2EA7FFFB5FCA2EA01C0ACEA3FFE487E 6C5A11197F9816>I<3803E3C0380FFFE05A381E3CC0383C1E00EA380EA3EA3C1E6C5AEA1FFC48 5AEA3BE00038C7FC123CEA1FFC48B4FC4813C0EA780338F001E0EAE000A3EAF001387C07C0383F FF80380FFE00EA03F8131C7F9116>I<127E12FE127E120EA4137CEA0FFF148013871303A2120E A9387FC7F038FFE7F8387FC7F01519809816>II<127E12FE127E120EA4EB7FE0A3EB0F00131E5B5B5BEA0FF8A2 13BC131EEA0E0E130FEB0780387F87F0EAFFCFEA7F871419809816>107 DI<38FBC78038FFEFC0EBFFE0EA3E7CEA3C78EA3870 AA38FE7CF8A31512809116>IIII<38FF0F80EB3FE013FFEA07F1EBE0C0EBC0005BA290C7FCA7EAFFFCA313127F9116>114 DI<12035AA4EA7FFFB5FCA20007C7FCA75BEB0380A2130713 873803FF005BEA00F811177F9616>I<387E1F80EAFE3FEA7E1FEA0E03AA1307EA0F0FEBFFF06C 13F83803F3F01512809116>I<387F1FC000FF13E0007F13C0381C0700EA1E0FEA0E0EA36C5AA4 EA03B8A3EA01F0A26C5A13127F9116>I<387F1FC0133F131F380F1C00EA073CEA03B813F01201 6C5A12017FEA03B8EA073C131CEA0E0E387F1FC038FF3FE0387F1FC013127F9116>120 D<387F1FC038FF9FE0387F1FC0381C0700120E130EA212075BA2EA039CA21398EA01B8A2EA00F0 A35BA3485A1279127BEA7F806CC7FC123C131B7F9116>I123 D<12E0B3AE0320779C16>I<12FCB4FC13C012031201A97F6CB4FCEB7F80A2EBFF00EA01E05BA9 120312FF90C7FC12FC11207E9C16>I E /Fj 23 122 df<3801FFE0A238003C00A45BA45BA448 5AA4485AA4485AA4EAFFF0A2131C7E9B10>73 D<48B4EB03FE1507D8003F14C0150F151BA2016F EB3780EB67801567A201C7EBCF00EC818FA2EC830FD80187131E1486A2148C390307983CA214B0 EB03F00006495AA214C0000E138039FFC38FFF140F271C7E9B25>77 D<3801FFFEECFF8039003C 03C0EC01E0A31378A315C0EBF0031580EC0700141E3801FFFC14F001E0C7FCA2485AA4485AA4EA FFF0A21B1C7E9B1C>80 D97 D99 DIIIII<13C0EA01E0 13C01380C7FCA6120E123FEA33801263EAC700A21207120EA35A1340EA38C0A3EA3980EA3F0012 1E0B1C7D9B0D>I108 D<391E1F07C0393F3F8FE0396761D8703863C1F038C781E0A2380701C0A2390E0380 E0A3EC81C2391C0701C6A2EC038CA239380E01F839180600F01F127D9122>IIII114 DI<13C01201A3EA0380A4EAFFE0A2EA0700A2120EA45AA3 1320EA3860A213C0A2EA1F80EA0F000B1A7D990E>I<380F0180EA1F8312331263EB870012C3EA 0707A2EA0E0EA31420380C1C60121C120C380E3CC0380FEF803803C70013127D9116>II120 D<380F0180EA1F8312331263EB870012C3EA0707A2EA 0E0EA4EA0C1C121C120CEA0E3CEA0FF81203EA0038A2EA3070EA786013E0EA71C0EA3F80001EC7 FC111A7D9114>I E /Fk 42 122 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07 E0D80FC01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D<1238127C12FEA3127C123807077C8610>46 D<13181378EA01F812FFA21201B3A7387FFFE0A2 13207C9F1C>49 DII<14E013011303A2 1307130F131FA21337137713E7EA01C71387EA03071207120E120C12181238127012E0B512FEA2 380007E0A7EBFFFEA217207E9F1C>I<00101320381E01E0381FFFC0148014005B13F8EA1BC000 18C7FCA4EA19FCEA1FFF381E0FC0381807E01303000013F0A214F8A21238127C12FCA214F012F8 386007E0003013C0381C1F80380FFF00EA03F815207D9F1C>I<13FE3803FFC0380703E0380E00 F05A1478123C123E123F1380EBE0F0381FF9E0EBFFC06C13806C13C06C13E04813F0381E7FF838 3C1FFCEA7807EB01FEEAF000143E141EA2141C7E007813387E381F01F0380FFFC0000113001720 7E9F1C>56 D<1470A214F8A3497EA2497EA3EB06FF80010E7FEB0C3FA201187F141F01387FEB30 0FA201607F140701E07F90B5FCA239018001FCA200038090C7FCA20006147FA23AFFE00FFFF8A2 25227EA12A>65 DIIIII72 D77 DIII<3801FC043807FF8C381F03FC383C007C007C133C0078131CA200F8130CA27E1400 B4FC13E06CB4FC14C06C13F06C13F86C13FC000313FEEA003FEB03FFEB007F143FA200C0131FA3 6C131EA26C133C12FCB413F838C7FFE00080138018227DA11F>83 D<007FB61280A2397E03F80F 00781407007014030060140100E015C0A200C01400A400001500B3A20003B512F8A222227EA127 >I97 DI< EBFF80000713E0380F83F0EA1F03123E127E387C01E090C7FC12FCA6127C127EA2003E13306C13 60380FC0E03807FF803800FE0014167E9519>II< 13FE3807FF80380F87C0381E01E0003E13F0EA7C0014F812FCA2B5FCA200FCC7FCA3127CA2127E 003E13186C1330380FC0703803FFC0C6130015167E951A>II<3801FE1F0007B51280 380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03380F87C0EBFF80D819FEC7FC0018C8FC12 1CA2381FFFE014F86C13FE80123F397C003F8048131F140FA3007CEB1F00007E5B381F80FC6CB4 5A000113C019217F951C>II<120E121FEA3F80A3EA1F00120EC7FCA7EAFF80A2121FB2EAFFF0A20C 247FA30F>I<131C133E137FA3133E131C1300A7EA03FFA2EA003FB3A5127812FC133E137EEA78 FCEA7FF0EA1FC0102E83A311>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA201801300AE3BFF F1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39FFF1FFC0 A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7007C137C A26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF8381FF07CEB C03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8EAFFF0A2 1A207E951F>I114 DI<13C0A41201A212031207120F12 1FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA2381F807EAF 14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE00E000713 0CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B167F951E >I<39FFF01FE0A2390FC00600A2EBE00E0007130CEBF01C0003131813F800015BA26C6C5AA2EB 7EC0A2137F6D5AA26DC7FCA2130EA2130CA25B1278EAFC3813305BEA69C0EA7F80001FC8FC1B20 7F951E>121 D E /Fl 71 124 df11 D<137CEA01FEEA0387485A120E130690C7FCA4 B5FCA2EA0E07AC387F0FE0A2131A809915>I<137F485AEA038FEA070FEA0E07A6B5FCA2EA0E07 AC387F9FE0A2131A809915>I<90383E0F803901FF7FC0390383F0E0380707C1000E1381903803 80C01500A4B612E0A2380E0380AC397F8FE3FCA21E1A809920>I34 D<126012F012F812781218A31230127012 E01240050B7D990B>39 D<13C0EA0180EA03001206120E120C5A1238A212301270A21260A212E0 AA1260A21270A212301238A212187E120E12067EEA0180EA00C00A267E9B0F>I<12C012607E7E 121C120C7E1207A27E1380A21201A213C0AA1380A21203A213005AA212065A121C12185A5A5A0A 267E9B0F>I<126012F0A212701230A31260A212C01240040B7D830B>44 DI<126012F0A2126004047D830B>I<130CA2131C1318A213381330A213 701360A213E013C0A212011380A2120313005A1206A2120E120CA2121C1218A212381230A21270 1260A212E05AA20E257E9B13>II<12035AB4FCA21207B3EAFFF0A20C19 7D9813>III<1260EA7FFEA213FCEA600CEAC0181330 A2EA006013C0A21201A2EA0380A41207A8EA03000F1A7E9813>55 D57 D<126012F0A212601200A8126012F0A2126004107D8F0B >I<126012F0A212601200A8126012F0A212701230A31260A212C0124004177D8F0B>I63 D<130C131EA3133FA3497E1367A3EBC3C0A3380181E0A348B47EA213000006 1378A3487F121E39FF81FFC0A21A1B7F9A1D>65 DIIIIII<39FFF3FFC0A2390F003C00A9EBFFFCA2EB003CAB39 FFF3FFC0A21A1A7F991D>III76 D<39FF8001FF6D5A000F14F0A2380DE006A3380CF00CA3 EB7818A3EB3C30A3EB1E60A3EB0FC0A3EB0780121E39FFC78FFFEBC30F201A7F9923>II<137F3801FFC03807C1F0380F00 78001E7F001C131C487F0078130FA200707F00F01480A80078EB0F00A36C131E001C131C001E13 3C6C5B3807C1F03801FFC06C6CC7FC191C7E9A1E>II<137F3801FFC03807C1F0380F0078 001E7F001C131C003C131E487FA200707F00F01480A80070140000785BA2383C1C1E381C3E1C38 1E633C380F61F83807E1F03901FFC080EA007F130014E1ECFF0080143E141C19227E9A1E>III<007FB5FCA238781E0F00601303A200E0148000C01301A3000090C7FC AF3803FFF0A2191A7F991C>I<39FFF0FFC0A2390F001E00140CB17E6D5A12036D5A3801E0E038 007FC0011FC7FC1A1B7F991D>I<3AFFC7FF1FF0A23A1E00F803C091387801806CEC0300A214FC 5DD807801306EB819EA2D803C15B13C3140F01E3131C000114189038E60798A2D800F613B001FE 13F0EBFC03017C5BA2EB7801A201385BEB3000241B7F9927>87 D<39FFE01FF0A2390F800F0000 0713066C6C5A13E000015BEBF038000013306D5A137CEB3CC0133F6D5AA26DC7FCA9EBFFF0A21C 1A80991D>89 D92 D97 D<12FCA2121CA813F8EA1FFE130F381C0380A2EB01C0A6EB0380A2381F0F 00EA1BFEEA18F8121A7F9915>II<137EA2130EA8EA07CEEA1FFEEA3C1EEA700EA212E0A6127013 1EEA3C3E381FFFC0EA07CF121A7F9915>II<13F8EA01FCEA03BCEA073CEA0E181300A5 EAFFC0A2EA0E00ACEA7FE0A20E1A80990C>I<130EEA079FEA1FF7EA3877EA7038A5EA3870EA3F E0EA7780EA7000A2EA3FF013FC13FEEA700FEAE007A3EA700EEA781EEA1FF8EA07E010197F9013 >I<12FCA2121CA813F8EA1FFC131EEA1E0E121CAA38FF9FC0A2121A7F9915>I<1218123CA21218 C7FCA612FCA2121CACEAFF80A2091A80990A>I<13C0EA01E0A2EA00C01300A6EA07E0A21200B0 126012F0EAF1C0EA7F80EA3E000B2183990C>I<12FCA2121CA8EB7F80A2EB3C0013305B5B121D EA1FE0121EEA1C70137813387F131E38FF3FC0A2121A7F9914>I<12FCA2121CB3A4EAFF80A209 1A80990A>I<38FCFC3F39FDFE7F80391F0FC3C0381E0781001C1301AA39FF9FE7F8A21D107F8F 20>IIIIII I<120CA4121CA2123CEAFFC0A2EA1C00A71360A5EA0FC0EA07800B177F960F>II<38FF3F80A2381C0E00130CA26C5AA21338EA07 30A213F06C5AA26C5AA311107F8F14>I<39FF3F9F80A239381E0E00381C3E0C13361337000E5B 13631498000713B013C114F0A2380380E0A319107F8F1C>I<387F1FC0A2380E1E00EA071CEA03 B813B0EA01E012007F1201EA03B8EA071C1206EA0E0E38FF1FE0A21310808F14>I<38FF3F80A2 381C0E00130CA26C5AA21338EA0730A213F06C5AA26C5AA35BA2126100F3C7FC12C712FE127811 177F8F14>III E /Fm 18 118 df<127812FCA412781200A5127812FCA4127806117D900C>58 D 63 D68 D73 D77 D97 D99 D101 D<3807E3C0381FFFE0EA3C3C38381CC038781E00A4EA381CEA3C3CEA3FF8EA37E00070C7FCA2EA 3FFEEBFF806C13C0127F38F003E01301A3387C07C0383FFF803807FC0013197F9016>103 D<121E123FA4121EC7FCA4127FA2121FADEAFFC0A20A1B809A0C>105 D108 D<39FF1F81F890387FE7FE391FE3EE3E903881F81FA2EB01F0AA3AFFE7 FE7FE0A223117F9026>IIII115 D<1203A35AA25A123FEAFFF0A2EA1F00A81318A5EA0FF0EA03E00D187F9711>I<38FF1FE0A2EA 1F03AB1307130F380FFFFCEA03F316117F9019>I E /Fn 42 122 df<903901FF81FF011F01EF 13C0903A7F80FF87E0D9FE01EB0FF03903FC03FE13F8D807F013FCA2EE07E0020190C7FCA6B712 F8A32707F001FCC7FCB3A33A7FFF1FFFE0A32C2A7FA928>11 D<121C127FEAFF80A5EA7F00121C 09097B8813>46 D48 D<13075B137FEA07FFB5FCA212F8C6FCB3AB007F13FEA317277BA622>III<14075C5C5C5C5CA25B5B497E130F130E131C1338137013F013E0EA01C0EA 0380EA07005A120E5A5A5A5AB612F8A3C71300A7017F13F8A31D277EA622>I<000C1303380F80 3FEBFFFEA25C5C14E05C49C7FC000EC8FCA6EB7FC0380FFFF8EB80FC380E007F000C1480C7123F 15C0A215E0A2123E127FEAFF80A315C01300007E137F007814806CEBFF00381F01FE380FFFF800 0313E0C690C7FC1B277DA622>II<1238123E003FB512F0A315E04814C01580A215003870001E5C5C48137014F0495A C6485A13075C130F91C7FC5B5BA3137EA213FEA41201A86C5A13781C297CA822>III66 D<91393FF00180903903FFFE07010FEBFF8F90393FF007FF9038FF80014848C7127FD8 07FC143F49141F4848140F485A003F15075B007F1503A3484891C7FCAB6C7EEE0380A2123F7F00 1F15076C6C15006C6C5C6D141ED801FE5C6C6C6C13F890393FF007F0010FB512C0010391C7FC90 38003FF829297CA832>II73 D77 DIII82 D<007FB712C0A39039807FC03FD87C0014 0700781503A20070150100F016E0A2481500A5C71500B3A490B612E0A32B287EA730>84 D<48B47E000F13F0381F81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA 1FE0EA3F80127F130012FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97 DIIII<9038FF81F00003EBE7FC390FC1FE7C391F80FC FC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1F8381FFFE0001813800038 C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007F8007C13015A1400A36C13 01007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A22>103 D<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3120FB3A3EAFFFEA30F2B7DAA14> 105 DIII<3BFF C07F800FF0903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF0013 8001F05BA301E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0 390FCE07F09038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF8090 38FE07E0390FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE 0FE09038EFFF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E38 0FD8FF13F85BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800 127000F01370A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0 137C143C7EA26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203 A21207381FFFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B> I<39FFE03FF8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A 26>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7 FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3AFFFE03FF80A3 3A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB3F8E14DEEB 1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807007C90C8FCEAFE0F130E131E5BEA7C78EA 3FE0EA0FC021277F9A24>121 D E /Fo 74 124 df11 D<133FEBFF803803C1C0EA0703A2380E018090C7FCA5B512C0A2EA0E01AE387F87F8A2151D809C 17>II<9038 3F03F09038FFCFF83903C0FC1C390701F03CA2390E00E01892C7FCA5B612FCA2390E00E01CAE3A 7FC7FCFF80A2211D809C23>I34 D<127012F812FCA2127C120CA31218A2123012601240060D7D9C0C> 39 D<13C0EA0180EA03001206120E120C121C121812381230A21270A21260A212E0AC1260A212 70A21230A212381218121C120C120E12067EEA0180EA00C00A2A7D9E10>I<12C012607E7E121C 120C120E120612077EA21380A21201A213C0AC1380A21203A21300A25A1206120E120C121C1218 5A5A5A0A2A7E9E10>I<127012F012F8A212781218A31230A2127012601240050D7D840C>44 DI<127012F8A3127005057D840C>I<1303A213071306A2130E130C131C 1318A213381330A213701360A213E013C0A21201138012031300A25A1206A2120E120CA2121C12 18A21238123012701260A212E05AA210297E9E15>II<12035A123FB4FC12C71207 B3A3EAFFF8A20D1C7C9B15>III<131C A2133C137CA213DC1201139C1203EA071C1206120E120C121812381230126012E0B512C0A23800 1C00A63801FFC0A2121C7F9B15>I I<13F0EA03FCEA070CEA0E0EEA1C1E1238130CEA78001270A2EAF3F0EAF7F8EAFC1CEAF81E130E 12F0130FA51270A2130E1238131CEA1C38EA0FF0EA03E0101D7E9B15>I<1260387FFF80A21400 EA6003EAC0065BA2C65A5BA25BA25BA21201A2485AA41207A76CC7FC111D7E9B15>III<127012F8A312701200A8127012F8A312700512 7D910C>I<127012F8A312701200A8127012F012F8A212781218A31230A2127012601240051A7D 910C>I<007FB512C0B612E0C9FCA8B612E06C14C01B0C7E8F20>61 D<1306130FA3497EA4EB33 C0A3EB61E0A3EBC0F0A338018078A2EBFFF8487FEB003CA200067FA3001F131F39FFC0FFF0A21C 1D7F9C1F>65 DI<90381F8080EBFFE13803F033380780 1B380F000F001E1307001C1303123C5A1401127012F091C7FCA70070EB01801278A27E001CEB03 00121E6C13066C6C5A3803F0383800FFF0EB1F80191E7E9C1E>I III<90381F8080EBFFE13803F0333807801B 380F000F001E1307001C1303123C5A1401127012F091C7FCA5ECFFF0A20070EB07801278A27E12 1C121E7E3807800F3803F0393800FFF090381FC0001C1E7E9C21>I<39FFF3FFC0A2390F003C00 AAEBFFFCA2EB003CAC39FFF3FFC0A21A1C7E9B1F>II76 DIIII82 D<3807E080EA1FF9EA3C1FEA7007130312E01301A36CC7FCA2127CEA7FC0EA3FF8EA1FFEEA07FF C61380130FEB03C0A2130112C0A300E01380130300F01300EAFC0EEACFFCEA83F8121E7E9C17> I<007FB512C0A238780F03007013010060130000E014E000C01460A400001400B03803FFFCA21B 1C7F9B1E>I<39FFF0FFC0A2390F001E00140CB36C5B138000035BEA01C03800E0E0EB7FC0011F C7FC1A1D7E9B1F>I<3AFFE0FFE1FFA23A1F001E007C6C1530143FA20180147000079038678060 A32603C0E713C0ECC3C0A2D801E0EBC1809038E181E1A3D800F3EBF3001400A2017B13F6017E13 7EA3013C133CA3011C133801181318281D7F9B2B>87 D<397FF0FFC0A23907C03E0000031338EB E03000011370EBF06000005B1379EB7D80013FC7FC7FA27F80131FEB3BC0EB33E01371EB61F0EB C0F800011378EB807C3803003C487F380F801F39FFE0FFF0A21C1C7F9B1F>I<39FFF00FFCA239 078003C001C0138000031400EBE006EA01F000005BEBF81CEB7818EB7C38EB3C306D5A131F6D5A A26D5AAAEB7FF8A21E1C809B1F>I92 D97 D<12FCA2121CA9137EEA1DFF 381F8780381E01C0001C13E0130014F0A614E01301001E13C0381F07803819FF00EA187C141D7F 9C17>IIII<1378EA01FCEA039EEA07 1EEA0E0C1300A6EAFFE0A2EA0E00AEEA7FE0A20F1D809C0D>II<12FCA2121CA9137CEA1DFFEA1F 07381E0380A2121CAB38FF9FF0A2141D7F9C17>I<1218123C127C123C1218C7FCA612FCA2121C AEEAFF80A2091D7F9C0C>II<12FCA2121CA9EB7FC0A2EB3E0013185B5B5BEA1DE0121FEA1E70EA 1C781338133C131C7F130F38FF9FE0A2131D7F9C16>I<12FCA2121CB3A7EAFF80A2091D7F9C0C> I<39FC7E07E039FDFF9FF8391F83B838391E01E01CA2001C13C0AB3AFF8FF8FF80A221127F9124 >IIII<3803E180EA0FF9EA1E1FEA3C071278130312F0 A612781307123CEA1E1FEA0FFBEA07E3EA0003A6EB1FF0A2141A7F9116>III<120CA5121CA212 3CEAFFE0A2EA1C00A81330A5EA1E60EA0FC0EA07800C1A7F9910>I<38FC1F80A2EA1C03AC1307 EA0C0F380FFBF0EA03E314127F9117>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA 0398A213F86C5AA26C5AA313127F9116>I<39FF3FCFE0A2391C0F0780EC0300131F380E1B0614 86A2EB318E000713CCA213603803E0F8A33801C070A31B127F911E>I<387F8FF0A2380F078038 070600EA038EEA01DC13D8EA00F01370137813F8EA01DCEA038E130EEA0607380F038038FF8FF8 A21512809116>I<38FF0FE0A2381C0780EB0300EA0E06A36C5AA2131CEA0398A213F86C5AA26C 5AA35BA3EAF180A200C7C7FC127E123C131A7F9116>III E /Fp 10 118 df67 D80 D<903807FFFE017FEBFFE048B612F84815FE4890 39001FFF8003077F48D980017F6F7FA2707EA2707E6C90C7FC6C5A6C5AC9FCA40207B5FC91B6FC 130F013FEBF03F3901FFFE004813F04813C04890C7FC485A485A485AA212FF5BA3167FA26D14FF 6C6CEB01EF003F140301FF90380FCFFF6C9026C07F8F13F8000790B5000713FC6CECFC03C66CEB F0010107903980007FF8362E7DAD3A>97 D<91381FFFC049B512FC010714FF011F158090267FFC 0113C0EBFFF048D9C00313E04813804813005A485AA248486D13C06F1380007FED7F0093C7FC5B A212FFAA127F7FA3123F6D15F8121F16016C6C15F06C6D13036C6DEB07E06C6DEB0FC06C01F8EB 3F80903A7FFE01FF00011FB55A010714F8010114E0D9001F90C7FC2D2E7CAD35>99 D<13FCEA03FF4813804813C014E05AA67E14C06C13806C1300EA00FC90C7FCABEB7FC0B5FCA512 037EB3B0B6FCA518497CC820>105 D<90287FC001FFC0ECFFE0B5010F01F8010713FC033F01FE 011F13FF92B6017F809126C1FE07902680FF037F9126C3F0039026C1F8017F00039026C7C00190 26E3E0007F6CD9CF00ECE78002DE03EFC7FC02DC6D01FE6E7E14FC4A5DA24A5DA24A5DB3A8B6D8 C07F9026FFE03FB512F0A55C2E7CAD63>109 D<903A7FC001FFC0B5010F13F8033F13FE92B6FC 9126C1FE077F9126C3F0037F00039026C7C0017F6CEBCF0014DE02DC6D7F14FC5CA25CA25CB3A8 B6D8C07FEBFFE0A53B2E7CAD42>II116 DI E /Fq 8 117 df<140F5C147FEB03FF131FB6FCA313E7EA0007B3B3A7007FB612E0A4233879B7 32>49 D67 D97 D<903801FFC0011F13F8017F 13FE9038FFC1FF00039038007F80D807FCEB1FC0484814E0ED0FF0485A003FEC07F8A2485AED03 FCA212FFA290B6FCA301E0C8FCA5127FA27F003F153CA26C6C147C000F15786C6C14F86C6CEB01 F06C6CEB07E06C9038E03FC0013FB51200010F13FC010013E026267DA52D>101 D<13FFB5FCA412077EB0ED7FC0913803FFF8020F13FE91381F03FFEC3C01027814804A7E4A14C0 5CA25CA291C7FCB3A4B5D8FC3F13FFA4303C7CBB37>104 D<9039FF01FF80B5000F13F0023F13 FC9138FE03FFDAF00113C000039039E0007FE0028014F0EE3FF891C7121F17FC160F17FEA3EE07 FFAAEE0FFEA3EE1FFCA217F86EEB3FF06E137F6EEBFFE06E481380DAFC07130091383FFFFC020F 13F0020190C7FC91C9FCADB512FCA430377DA537>112 D<9038FE03F000FFEB0FFE91383FFF80 91387C7FC014F00007ECFFE06C6C5A5CA25CED7FC0ED3F80ED0E0091C8FCB3A3B512FEA423267D A529>114 D116 D E /Fr 36 119 df38 D<127012F8A212F012E005057A840F>46 D<14035CA25C1580141FA214371477146714C7A2EB01 87A2EB0307A21306130E130C1318A2133090383FFFC05BEB600313C012011380EA0300A21206A2 121E39FFC03FFC13801E237DA224>65 D<90B512E015F890380F003C151E131E150E150FA24913 0E151EA2153C49137815F0EC01E090387FFFC090B5FC9038F003E0EC00F01578485A1538153CA2 48481378A315F039078001E01403EC07C0EC1F00B512FE14F020227DA122>I<90B512F015FC90 380F003E150F011EEB0780150316C015015B16E0A35BA449EB03C0A44848EB0780A216005D4848 130E5D153C5D48485B4A5AEC0780021FC7FCB512FC14F023227DA125>68 D<91387E0180903903FF810090380F80C390383E00670178133F49133ED801C0131E485A120748 C7121C120E121E5A15185A92C7FCA25AA4EC3FFC5AEC01E0A26C495AA312700078495A1238003C 130F6C131B260FC0F3C7FC3803FFC1C690C8FC212479A226>71 D73 D<903807FFC04913809038003C00 A25CA45CA4495AA4495AA4495AA449C7FCA212381278EAF81EA2485AEA6078EA70F0EA3FE0EA1F 801A237CA11A>I<9039FFF80FFCA290390F0007C01600011E130E5D5D1560495B4A5A4AC7FC14 0E495A5C147814FCEBF1BCEBF33CEBFE1E13FC3801F01F497EA2813803C007A26E7EA2EA07806E 7EA28139FFF80FFEA226227DA125>III<01FFEB1FFC1480010FEB03C01680D91BC01300A3EB19E001 311306A2EB30F0A201605B1478A3496C5AA3141ED801805BA2140FA2D803005BEC07E0A300065C 1403A2121F39FFE00180A226227DA124>I<14FE903807FF8090380F03E090383C00F001701378 4913384848133C4848131C48C7FC48141E121EA25AA25AA348143CA31578A34814F0A26CEB01E0 15C01403EC07800078EB0F00141E6C5B6C13F8380F83E03807FF80D801FCC7FC1F2479A225>I< 90B512C015F090380F0078153C011E131E150EA349131EA3153C491338157815F0EC03C090B512 005CEBF00FEC07803901E003C0A43903C00780A43907800F001503A2EC0706D8FFF8138EEC03FC C7EA01F020237DA124>82 D<903801F06090380FFC4090381E0EC0EB3807EB700301E0138013C0 1201A2D803801300A26DC7FCA27FEA01F813FF6C13E06D7EEB1FF8EB03FCEB007C143C80A30030 131CA31418007013385C00781360387C01C038EF0380D8C7FFC7FCEA81FC1B247DA21B>I<001F B512F8A2391E03C07800381438EB0780123000601430A2EB0F0012C0A3D8001E1300A45BA45BA4 5BA4485AA31203B57E91C7FC1D2277A123>I<393FFE07FFA23903C000F015E0484813C0A4390F 000180A4001EEB0300A4481306A4485BA4485BA25C12705C5C6C485A49C7FCEA1E0EEA0FFCEA03 F0202377A124>I<3BFFF03FF81FF8D9E07FEB3FF03B1F0007800780001E010FEB03001606141F 5E14375E14675E14C75E381F0187000F5DEB0307ED818013060383C7FC130C158613180138138C 0130139C0160139815B801C013B015E013805D13005D120E92C8FC120C2D2376A131>87 D<39FFF003FFA2000FC712F015E090388001C000071480EC0300EBC0060003130E5CEBE0180001 5B5CEBF0E000005BEBF18001FBC7FC13FF137E137C1378A45BA4485AA4EA3FFE5B202276A124> 89 D97 D<137E48B4FC3803C380EA0703 EA0E07121C003CC7FC12381278A35AA45BEA7003130EEA383CEA1FF0EA0FC011157B9416>99 D<143CEB03F8A2EB0038A21470A414E0A4EB01C013F9EA01FDEA078F380F0780120E121CEA3C03 383807001278A3EAF00EA214101418EB1C30EA703C137C3838FC60383FCFC0380F078016237BA2 19>I<13F8EA03FCEA0F0EEA1E06123C1238EA780CEAF038EAFFF01380EAF0005AA413021306EA 701C1378EA3FE0EA0F800F157A9416>I<143C147F14CF1301EB03861480A3EB0700A5130EEBFF F0A2EB0E00A25BA55BA55BA55BA45B1201A2EA718012F390C7FC127E123C182D82A20F>II<136013F0 13E0A21300A8120EEA1F801233126312C3A3EA0700A2120EA35A13201330EA3860A213C01239EA 1F80EA0E000C217CA00F>105 D<13F0EA0FE0A21200A2485AA4485AA448C7FCEB01E0EB07F0EB 0E30380E1870EB30F01360EBC060381D8000121F13E0EA1CF0EA3838133CEB1C20143038703860 A21440EB18C038E01F8038600F0014237DA216>107 DI<381E0780383F1F E0EA63B8EBE070EAC3C0A21380000713E01300A3380E01C0A214C2EB0383001C1386EB0706140C EB0318003813F0381801E018157C941B>110 D<137E48B4FC3803C380380701C0120E001C13E0 123CA21278A338F003C0A21480130700701300130E5B6C5AEA1FF0EA07C013157B9419>I<3803 C1F03807E3F8380C761C137C3818781E1370A2EA00E0A43801C03CA314780003137014F014E0EB E3C038077F80EB1E0090C7FCA2120EA45AA2EAFFC0A2171F7F9419>I<381E0F80383F1FC03863 B0E013E0EAC3C1A2EB80C00007130090C7FCA3120EA45AA45A121813157C9415>114 D<13FC48B4FC38038380EA0703EA0E07A2EB0200000FC7FC13F0EA07FC6C7EEA007E130FA2EA70 07EAF00EA2485AEA7038EA3FF0EA1FC011157D9414>I<13C01201A4EA0380A4EA0700EAFFF8A2 EA0700120EA45AA45AA213101318EA7030A21360EA71C0EA3F80EA1E000D1F7C9E10>I<000F13 30381F8070EA31C0006113E012C1EAC380A2380381C0EA0701A3380E0380A214841486EB070CA2 130FEB1F183807F3F03803E1E017157C941A>I<380F01C0381F83E0EA31C3EA61C1EAC1C0EAC3 80A2000313C0EA0700A3380E0180A3EB0300A213061304EA0F1CEA07F8EA01E013157C9416>I E /Fs 53 122 df35 D<127012F812FCA2127C12 0CA41218A21230A212601240060F7C840E>44 DI<127012F8A3127005 057C840E>I48 DI51 D<00101380EA1C07381FFF 005B5B13F00018C7FCA613F8EA1BFEEA1F0F381C0780EA180314C0EA000114E0A4126012F0A214 C0EAC0031260148038300700EA1C1EEA0FFCEA03F013227EA018>53 D<137E48B4FC3803C18038 0701C0EA0E03121CEB018048C7FCA2127812701320EAF1FCEAF3FEEAF60738FC038000F813C013 0112F014E0A51270A3003813C0130300181380381C0700EA0E0EEA07FCEA01F013227EA018>I< EA01F0EA07FCEA0E0F38180780EA3803383001C01270A31278EB0380123E383F0700EA1FCEEA0F FCEA03F87FEA0F7F381C3F80EA380F387007C0130338E001E01300A5387001C0A238380380381E 0F00EA0FFEEA03F013227EA018>56 DI<497E497EA3497EA3497E130CA2EB1CF8EB1878 A2EB383C1330A2497EA3497EA348B51280A2EB800739030003C0A30006EB01E0A3000EEB00F000 1F130139FFC00FFFA220237EA225>65 DI<90380FE01090383FF8309038F81C703801E0063903C003 F03807800148C7FC121E003E1470123C127C15301278A212F81500A700781430A2127CA2003C14 60123E121E6C14C06C7E3903C001803901E003003800F80EEB3FF8EB0FE01C247DA223>II 70 D<903807F00890383FFC189038FC0E383801E0033903C001F83807800048C71278121E1538 5AA2007C14181278A212F81500A6EC1FFF1278007CEB0078A2123CA27EA27E6C7E6C6C13F83801 F0013900FC079890383FFE08903807F80020247DA226>I<39FFFC3FFFA239078001E0AD90B5FC A2EB8001AF39FFFC3FFFA220227EA125>I<3803FFF0A238000F00B3A6127012F8A3EAF01EEA60 1CEA3878EA1FF0EA07C014237EA119>74 D<39FFFC07FFA239078001F015C05D4AC7FC14065C5C 14385C5CEB81C0EB8380EB87C080138DEB98F013B0EBE078497E13808080A26E7E8114036E7EA2 6E7E4A7E3AFFFC07FF80A221227EA126>III<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E131E7FA2 EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F8000EA FFF0156020227EA125>III82 D<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460A36C1300A21278127FEA3F F0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A46C136014E06C13C0EAF801 38EF038038C7FF00EA81FC14247DA21B>I<007FB512F8A2387C07800070143800601418A200E0 141C00C0140CA500001400B3A20003B5FCA21E227EA123>I<3BFFF03FFC07FEA23B0F0007C001 F00203EB00E01760D807806D13C0A33B03C007F001801406A216032701E00C781300A33A00F018 3C06A3903978383E0CEC301EA2161C90393C600F18A390391EC007B0A3010F14E0EC8003A36D48 6C5AA32F237FA132>87 D<387FFFFEA2EB003C007C137C0070137814F84813F0EB01E0EAC00314 C01307148038000F005B131E133E133C5B13F85B0001130313E0EA03C012071380000F13071300 001E1306003E130E003C131E007C133E007813FEB5FCA218227DA11E>90 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0 F0EB0078000E1338143C141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237F A21B>II<14E0130FA213011300AAEA03F0EA07FEEA1F07EA3C01EA 38001278127012F0A712701278EA3801EA3C03381E0EF0380FFCFEEA03F017237EA21B>II<133C13FEEA01CFEA038FA2EA0700A9EAFFF8A2EA0700 B1EA7FF8A2102380A20F>I<14F03801F1F83807FFB8380F1F38381E0F00EA1C07003C1380A500 1C1300EA1E0FEA0F1EEA1FFCEA19F00018C7FCA2121CEA1FFF6C13C04813E0383801F038700070 481338A400701370007813F0381E03C0380FFF803801FC0015217F9518>I<120E12FEA2121E12 0EAAEB1F80EB7FC0380FC1E0EB80F0EB0070120EAE38FFE7FFA218237FA21B>I<121C121E123E 121E121CC7FCA8120E12FEA2121E120EAFEAFFC0A20A227FA10E>II<120E12FEA212 1E120EAAEB0FFCA2EB07E0EB0380EB0700130E13185B137813F8EA0F9C131EEA0E0E7F1480EB03 C0130114E014F038FFE3FEA217237FA21A>I<120E12FEA2121E120EB3ABEAFFE0A20B237FA20E> I<390E1FC07F3AFE7FE1FF809039C0F303C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8F FEA227157F942A>I<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA21815 7F941B>II<380E1F8038FE7FE0 38FFC1F0380F0078120E143CA2141EA7143CA2000F1378EB8070EBC1F0380E7FC0EB1F0090C7FC A8EAFFE0A2171F7F941B>I114 DI<1206A5120EA3121E123EEAFFF8A2 EA0E00AA130CA51308EA0718EA03F0EA01E00E1F7F9E13>I<000E137038FE07F0A2EA1E00000E 1370AC14F01301380703783803FE7FEA01F818157F941B>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A217157F941A>I<39FF8FF9FFA2 391E01C07CD81C031338000EEBE030A2EB06600007EB7060A2130E39038C30C01438139C3901D8 1980141DA2EBF00F00001400A2497EEB600620157F9423>I<38FFC3FEA2381E00F8000E1360A2 6C13C0A338038180A213C300011300A2EA00E6A3137CA31338A21330A213701360A2EAF0C012F1 EAF380007FC7FC123E171F7F941A>121 D E /Ft 20 118 df45 D68 D73 D77 D80 D<90387F80203901FFE0603807C0F8390F001CE0001E130F481307003813030078130112701400 12F0A21560A37E1500127C127E7E13C0EA1FF86CB47E6C13F86C7FC613FF010F1380010013C0EC 1FE01407EC03F01401140015F8A200C01478A57E15706C14F015E07E6CEB01C06CEB038039E780 070038C1F01E38C07FFC38800FF01D337CB125>83 D97 D99 DIII<15F090387F03F83901FFCF1C3803C1FC390780F818 390F00780048137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0 D81C7FC7FC0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB 00701578481438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F 7E9F21>I<1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8 A20D307EAF12>105 D<260781FEEB3FC03BFF87FF80FFF0903A8E07C1C0F83B0F9803E3007C27 07B001E6133C9026E000FC7F495BA3495BB3486C486C133F3CFFFC1FFF83FFF0A2341F7E9E38> 109 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E 3AFFFC1FFF80A2211F7E9E25>II<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE0 38EBC000A35BB3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813 705A1430A37E6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2 141C7EA27E14186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A312 03A21207120F121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB 19>II E /Fu 5 85 df<1630167016F0A21501A21503A2150715 0FA2151B821531A2156115E115C1EC0181A2EC0301A21406A2140C141C14181430A202607FA2EC C000A249B5FC5B91C7FC1306A25BA25BA25B1370136013E01201000381D80FF01301D8FFFE9038 3FFFF0A22C337CB235>65 D<010FB6FC17C0903A003F8007F0EE01F892C7127C177E4A143E8314 7E188002FE140FA24A15C0A21301A25CA21303171F5CA2130718804A143FA2130F18004A5CA201 1F157E17FE4A5CA2013F4A5A5F91C712034C5A495D160F017E4A5A4CC7FC01FE147E16F849495A ED07E00001EC3F80B600FEC8FC15F032317CB036>68 D<010FB612FEA29039003F8000173E92C7 121EA24A140CA2147EA214FEA25CA20101151CEEC0184A1400A201031301A202F05B1503010713 0F91B5FC93C7FCECE00F010F7FA2ECC006A2011F130EA2EC800C92C8FC133FA291C9FCA25BA213 7EA213FEA25BA21201B512FCA22F317CB02F>70 D<010FB512F816FF903A003F801FC0EE07E092 380003F0EE01F84AEB00FCA2147EA214FE16015CA2010115F816034A14F0EE07E01303EE0F804A EB3F00167E0107EB03F891B512E016809138E007C0010FEB03F015014A6C7EA2011F80A25CA201 3F1301A21400A249495AA2137E170601FE150E170C5B171C000102011338B539F000FC70EE7FE0 C9EA1F802F327CB034>82 D<0007B712F8A23A0FE007F00101801400D80E00491370121E001C13 0F121800385CA20030011F1460127000605CA2023F14E000E016C0C790C8FCA25CA2147EA214FE A25CA21301A25CA21303A25CA21307A25CA2130FA25CA2131FA25C133F497E007FB512C0A22D31 74B033>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 795 707 a Fu(D)26 b(R)g(A)f(F)h(T)225 798 y Ft(Do)r(cumen)n(t)20 b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)685 981 y Fs(Scott)c(Berryman,)e Fr(Y)l(ale)19 b(Univ)700 1039 y Fs(James)c(Co)o(wnie,)h Fr(Meiko)i(Ltd)474 1097 y Fs(Jac)o(k)e(Dongarra,)i Fr(Univ.)23 b(of)17 b(T)l(ennesse)n(e)j(and)d(ORNL)801 1155 y Fs(Al)f(Geist,)f Fr(ORNL)795 1213 y Fs(Bill)f(Gropp,)j Fr(ANL)767 1271 y Fs(Rolf)f(Hemp)q(el,)e Fr(GMD)762 1329 y Fs(Bob)i(Knigh)o(ten,)g Fr(Intel)786 1387 y Fs(Rust)o(y)g(Lusk,)h Fr(ANL)614 1445 y Fs(Stev)o(e)e(Otto,)h Fr(Or)n(e)n(gon)h(Gr)n(aduate)g(Inst)578 1504 y Fs(T)l(on)o(y)g(Skjellum,)c Fr(Missisippi)j(State)j(Univ)653 1562 y Fs(Marc)d(Snir,)g Fr(IBM)h(T.)g(J.)g(Watson)744 1620 y Fs(Da)o(vid)f(W)l(alk)o(er,)f Fr(ORNL)626 1678 y Fs(Stev)o(e)g(Zenith,)g Fr(Kuck)k(&)f(Asso)n(ciates)844 1796 y Fs(Ma)o(y)e(9,)g(1993)87 1854 y(This)g(w)o(ork)g(w)o(as)h(supp)q(orted)g(b)o(y)f(ARP)l(A)g(and)g(NSF)g (under)g(con)o(tract)g(n)o(um)o(b)q(er)f(###,)g(b)o(y)g(the)192 1912 y(National)h(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o(hnology) f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)650 1970 y(Agreemen)o(t)f(No.)21 b(CCR-8809615.)p eop %%Page: 1 2 bop 75 359 a Fq(Chapter)32 b(1)75 570 y Fp(P)m(oin)m(t)41 b(to)f(P)m(oin)m(t) h(Comm)m(unication)884 789 y Fo(Marc)15 b(Snir)684 838 y(William)10 b(Gropp)j(and)h(Ewing)f(Lusk)75 991 y Fn(1.1)70 b(In)n(tro)r(duction)75 1088 y Fo(This)17 b(section)h(is)e(a)h(draft)g(of)f(the)i(curren)o(t)g(prop)q (osal)f(for)g(p)q(oin)o(t-to-p)q(oin)o(t)e(comm)o(unication.)24 b(It)17 b(do)q(es)h(not)f(y)o(et)75 1137 y(include)d(a)g(description)g(of)f (the)i(F)m(ortran)f(77)f(and)h(C)f(bindings.)158 1190 y(I)h(ha)o(v)o(e)g (tried)g(to)g(indicate,)f(wherev)o(er)i(appropriate,)f(gaps)g(and)f(unresolv) o(ed)i(issues,)g(using)e(small)f(t)o(yp)q(e.)158 1321 y Fm(Discussion:)38 b Fl(The)13 b(follo)o(wing)j(sections)f(of)f(the)g(in)o(tro)q(duction)i(con)o (tain)f(general)g(information)h(on)e(the)g(design)h(of)75 1367 y(MPI)h(pro)q(cedures.)27 b(The)16 b(material)i(should)g(b)q(e)e(mo)o(v)o(ed) g(to)g(a)g(general)h(in)o(tro)q(duction)i(for)d(the)g(en)o(tire)h(do)q(cumen) o(t.)27 b(The)75 1413 y(actual)14 b(binding)i(of)d(these)g(ob)r(jects)g(in)h (F)m(ortran)g(and)f(C)g(will)h(b)q(e)g(discussed)h(in)e(the)h(language)h (binding)g(sub)q(committee.)75 1648 y Fn(1.2)70 b(Data)23 b(T)n(yp)r(es)75 1753 y Fk(1.2.1)55 b(Handle)75 1835 y Fo(MPI)17 b(pro)q(cedures)j(use)e(at)f (v)n(arious)g(places)h Fj(hand)r(les)p Fo(.)28 b(Handles)18 b(are)g(used)g(to)f(access)i(opaque)e(ob)r(jects.)30 b(Suc)o(h)75 1885 y(ob)r(ject)20 b(can)g(b)q(e)g(created,)i(up)q(dated)e(and)f(destro)o(y) o(ed)i(only)d(b)o(y)h(b)o(y)h(calling)e(suitable)h(MPI)h(pro)q(cedures,)i (and)75 1934 y(pro)o(viding)14 b(the)i(handle)f(as)g(parameter.)21 b(Opaque)16 b(ob)r(jects)g(hide)g(from)d(the)j(user)g(the)g(in)o(ternal)e (represen)o(tation)75 1984 y(used)20 b(for)f(v)n(arious)f(MPI)i(ob)r(jects,)h (th)o(us)e(allo)o(wing)e(to)i(ha)o(v)o(e)g(similar)e(calls)i(in)f(C)h(and)g (F)m(ortran,)h(allo)o(wing)d(to)75 2034 y(o)o(v)o(ercome)f(problems)g(with)h (the)g(t)o(yping)f(rules)i(in)e(these)j(languages,)d(and)h(allo)o(wing)e(for) h(future)i(extension)g(of)75 2084 y(their)13 b(functionalit)o(y)m(.)j(The)e (mec)o(hanism)c(for)j(opaque)g(ob)r(jects)h(used)g(here)g(lo)q(osely)e(follo) o(ws)g(the)h(POSIX)h(F)m(ortran)75 2134 y(binding)f(standard.)158 2186 y(An)i(opaque)h(ob)r(ject)g(can)f(b)q(e)h Fj(p)n(ersistent)f Fo(or)g Fj(ephemer)n(al)p Fo(.)22 b(A)15 b(p)q(ersisten)o(t)i(ob)r(ject)f(p)q (ersists)h(un)o(til)e(destro)o(y)o(ed)75 2236 y(b)o(y)h(an)g(explicit)g(op)q (eration.)25 b(An)17 b(ephemeral)e(ob)r(ject)j(is)e(go)q(o)q(d)g(for)g(a)g (single)g(use;)h(th)o(us)g(an)f(ephemeral)g(ob)r(ject)75 2286 y(asso)q(ciated)i(with)g(a)f(comm)o(unicatio)o(n)e(op)q(eration)i(disapp)q (ears)i(once)f(this)g(op)q(eration)f(is)h(completed)f(\(or)g(once)75 2336 y(this)d(ob)r(ject)h(is)e(not)h(needed)i(an)o(ymore)c(for)i(the)g (completion)e(of)h(the)i(op)q(eration\).)158 2389 y(Opaque)i(ob)r(ject)g(are) g(created)h(b)o(y)e(calls)g(that)g(are)h(sp)q(eci\014c)g(to)g(eac)o(h)f(ob)r (ject)i(t)o(yp)q(e.)25 b(They)17 b(are)g(destro)o(y)o(ed)75 2438 y(b)o(y)d(a)f(call)h(to)f Fi(MPI)p 363 2438 14 2 v 15 w(FREE)p Fo(.)g(Additional)f(MPI)j(functions)f(are)g(a)o(v)n(ailable)e(to)h (access)j(and)e(up)q(date)h(sp)q(eci\014c)g(opaque)75 2488 y(ob)r(jects.)158 2576 y Fh(MPI)p 257 2576 15 2 v 17 w(FREE\(handle\))75 2704 y(IN)h(handle)j Fo(handle)14 b(to)g(ob)r(ject)965 2828 y(1)p eop %%Page: 2 3 bop 75 -100 a Fo(2)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)158 45 y Fo(Mark)e(the)h(handle)g(for)f(destruction.)18 b(The)13 b(ob)r(ject)g(can)g(destro)o(y)o(ed)h(as)e(so)q(on)h(as)f(there)i (there)f(is)g(no)f(p)q(ending)75 95 y(op)q(eration)g(that)g(is)g(using)g (this)g(ob)r(ject.)18 b(This)12 b(is)g(equiv)n(alen)o(t)g(to)g(c)o(hanging)f (the)i(p)q(ersistence)i(of)c(the)i(handle)f(from)75 145 y Fi(persistent)c Fo(to)i Fi(ephemeral)p Fo(.)15 b(F)m(or)9 b(example,)g(if)g(a)h(comm)o (unication)d(ob)r(ject)j(is)g(freed,)i(after)e(the)g(comm)o(unication)75 195 y(started)15 b(but)f(b)q(efore)h(it)f(completed,)f(then)h(the)h(p)q (ending)f(op)q(eration)g(is)f(not)h(a\013ected;)h(one)g(can)f(use)h(the)f (handle)75 244 y(in)f(calls)g(to)g Fi(MPI)p 334 244 14 2 v 15 w(WAIT)f Fo(or)h Fi(MPI)p 565 244 V 15 w(STATUS)p Fo(.)f(Ho)o(w)o(ev)o (er,)h(once)h(the)g(op)q(eration)f(completes,)g(it)g(is)g(erroneous)h(to)f (reuse)75 294 y(the)h(freed)h(handle)f(for)g(a)f(new)h(comm)o(unication.)158 426 y Fm(Discussion:)158 475 y Fl(An)f(alternativ)o(e)i(design)f(p)q(oin)o(t) g(w)o(as)f(discussed,)h(whereb)o(y)g(it)f(is)h(the)e(user)i(resp)q(onsibili)q (t)o(y)i(to)d(free)f(a)h(handle)i(only)f(if)75 520 y(there)h(are)g(no)g(p)q (ending)i(op)q(erations)g(on)e(this)h(handle.)24 b(The)15 b(curren)o(t)g (design)h(w)o(as)f(deemed)g(more)g(elegan)o(t.)24 b(Note)15 b(that)75 566 y(it)f(is)g(not)f(necessary)i(to)e(destro)o(y)h(the)g(handle)h (as)e(so)q(on)h(as)g(it)f(is)h(not)g(needed)g(an)o(ymore.)19 b(A)13 b(v)n(alid)i(implemen)o(tation)i(ma)o(y)75 612 y(just)c(mark)g(freed)g (handles)i(and)f(garbage)g(collect)g(them)f(p)q(erio)q(dical)q(ly)n(.)158 660 y(On)k(the)f(other)h(hand,)h(some)f(ob)r(jects,)h(suc)o(h)f(as)g (bu\013er)g(descriptor)i(ob)r(jects,)e(can)g(b)q(e)g(used)g(b)o(y)g(sev)o (eral)h(distinct)75 706 y(concurren)o(t)c(comm)o(unication)i(op)q(erations.)j (Th)o(us,)13 b(to)g(\014nd)h(out)f(whether)h(suc)o(h)g(ob)r(ject)f(can)g(b)q (e)h(freed,)f(one)g(w)o(ould)h(need)75 752 y(a)f(reference)h(coun)o(t)g(of)f (p)q(ending)j(op)q(erations.)j(This)c(complication)h(w)o(as)d(not)h(observ)o (ed)g(at)g(our)f(last)h(meeting,)h(and)f(ma)o(y)75 797 y(b)q(e)f(a)g(go)q(o)q (d)h(argumen)o(t)g(to)f(reconsider)i(the)e(curren)o(t)g(de\014nition)j(of)d Ff(MPI)p 1130 797 12 2 v 13 w(FREE)p Fl(.)158 846 y(W)m(e)h(do)f(not)h(ha)o (v)o(e)g(y)o(et)g(a)f(mec)o(hanism)i(to)f(con)o(trol)g(where)g(from)f (storage)h(is)g(allo)q(cated)i(for)d(opaque)i(ob)r(jects.)j(Some)75 892 y(b)q(eliev)o(e)d(it)e(is)h(desirable)h(to)e(allo)o(w)h(the)g(user)f(to)g (con)o(trol)h(this)g(allo)q(cation.)158 941 y(I)e(no)o(w)h(ha)o(v)o(e)g(a)g (generic)h Ff(MPI)p 576 941 V 13 w(FREE)d Fl(function)j(for)e(all)i(kinds)g (of)f(opaque)g(ob)r(jects.)18 b(An)o(y)12 b(reason)i(to)e(ha)o(v)o(e)i(sp)q (ecialized)75 987 y(functions)h(for)d(eac)o(h)i(kind)g(of)f(ob)r(ject,)g(as)g (for)g Ff(CREATE)p Fl(?)158 1158 y Fh(MPI)p 257 1158 15 2 v 17 w(ASSOCIA)l(TED\(handle,)h(t)o(yp)q(e\))75 1286 y(IN)i(handle)j Fo(handle)14 b(to)g(ob)r(ject)75 1382 y Fh(OUT)i(t)o(yp)q(e)k Fo(state)158 1474 y(Returns)c(the)f(t)o(yp)q(e)g(of)f(the)i(ob)r(ject)f(the)h (handle)e(is)h(curren)o(tly)h(asso)q(ciated)f(with,)f(if)g(suc)o(h)i(exists.) 21 b(Returns)75 1524 y(the)14 b(sp)q(ecial)h(t)o(yp)q(e)f Fi(MPI)p 444 1524 14 2 v 15 w(NULL)f Fo(if)g(the)i(handle)e(is)h(not)g(curren)o(tly)h (asso)q(ciated)f(with)g(an)o(y)f(ob)r(ject.)158 1577 y(MPI)18 b(ma)o(y)e(pro)o(vide)h(prede\014ned)j(opaque)d(ob)r(jects)i(and)f (prede\014ned,)i(static)e(handles)g(to)f(these)i(ob)r(jects.)75 1627 y(Suc)o(h)14 b(ob)r(jects)h(ma)o(y)d(not)i(b)q(e)h(destro)o(y)o(ed.)75 1760 y Fk(1.2.2)55 b(Arra)n(y)19 b(of)g(handles)75 1843 y Fo(An)14 b(MPI)h(call)f(ma)o(y)e(need)j(a)f(parameter)g(that)g(is)h(an)f Fj(arr)n(ay)g(of)i(hand)r(les)p Fo(.)k(The)14 b(arra)o(y-of-handles)g(ob)r (ject)h(is)f(not)75 1893 y(self-delimiting;)9 b(it)i(do)q(es)i(not)f(carry)g (information)d(on)i(the)i(n)o(um)o(b)q(er)e(of)g(en)o(tries)i(in)e(the)i (arra)o(y)m(.)j(This)c(information)75 1942 y(has)i(to)g(b)q(e)g(pro)o(vided)g (b)o(y)g(an)f(additional)f Fi(len)i Fo(parameter,)f(whenev)o(er)i(a)f (list-of-handles)e(parameter)i(is)g(used.)158 2074 y Fm(Discussion:)158 2123 y Fl(The)d(reason)g(for)g(not)g(ha)o(ving)h(self-delimiti)q(ng)i(arra)o (ys)d(is)h(that)f(records)g(are)g(not)g(\014rst-class)h(citizens)g(in)g(F)m (ortran)f(and)75 2168 y(that)h(ha)o(ving)i(an)f(arra)o(y)f(where)h(one)f(en)o (try)h(enco)q(des)g(length,)g(whereas)g(the)f(other)h(en)o(tries)g(enco)q(de) g(handles)h(w)o(as)e(deemed)75 2214 y(to)k(b)q(e)g(to)q(o)g(error-prone.)27 b(This)17 b(line)g(of)f(argumen)o(tation)i(really)g(implies)g(that)e(w)o(e)g (do)g(not)g(view)h(arra)o(y-of-handles)h(as)75 2260 y(an)f(opaque)g(ob)r (ject,)h(and)f(that)f(w)o(e)g(exp)q(ect)h(the)g(user)g(to)f(build)j(these)e (ob)r(jects)f(explicitl)q(y)m(,)k(not)c(via)i(MPI)e(calls.)29 b(The)75 2305 y(alternativ)o(e)15 b(is)f(for)e(arra)o(y-of-handles)k(to)c(b)q (e)i(an)f(opaque)h(ob)r(ject,)f(manipulated)j(via)e(s)f(set)g(of)g(MPI)g (calls.)75 2521 y Fk(1.2.3)55 b(State)75 2604 y Fo(MPI)14 b(pro)q(cedures)i (use)e(at)f(v)n(arious)g(places)h(argumen)o(ts)f(with)g Fj(state)g Fo(t)o(yp)q(es.)19 b(The)14 b(v)n(alues)f(of)g(suc)o(h)h(data)g(t)o(yp)q(e)g (are)75 2654 y(all)f(iden)o(ti\014ed)h(b)o(y)g(names,)e(and)i(no)g(op)q (eration)g(is)f(de\014ned)i(on)f(them.)k(F)m(or)13 b(example,)f(the)j Fi(MPI)p 1589 2654 V 15 w(APPEND)e Fo(routine)75 2704 y(has)h(a)g(state)g(t)o (yp)q(e)h(parameter)e(with)h(v)n(alues)f Fi(MPI)p 864 2704 V 16 w(INT,)20 b(MPI)p 1054 2704 V 16 w(REAL,)12 b Fo(etc.)p eop %%Page: 3 4 bop 75 -100 a Fg(1.3.)31 b(PR)o(OCESSES)1437 b Fo(3)75 45 y Fk(1.2.4)55 b(Named)18 b(constan)n(ts)75 124 y Fo(MPI)e(pro)q(cedures)j (sometimes)14 b(assign)i(a)g(sp)q(ecial)g(meaning)e(to)i(a)g(sp)q(ecial)g(v)n (alue)g(of)f(a)h(basic)g(t)o(yp)q(e)h(parameter;)75 173 y(e.g.)i Fi(tag)14 b Fo(is)g(an)g(in)o(teger)g(v)n(alued)g(parameter)g(of)g(p)q(oin)o (t-to-p)q(oin)o(t)f(comm)o(unicatio)o(n)f(op)q(erations,)i(with)g(a)g(sp)q (ecial)75 223 y Fi(DONTCARE)g Fo(v)n(alue.)24 b(Suc)o(h)17 b(parameters)f(will)f(ha)o(v)o(e)h(a)g(range)g(of)f(regular)h(v)n(alues,)g (whic)o(h)g(is)g(a)g(prop)q(er)h(subrange)75 273 y(of)f(the)h(range)g(of)f(v) n(alues)h(of)f(the)h(corresp)q(onding)h(basic)f(t)o(yp)q(e;)h(sp)q(ecial)f(v) n(alues)f(\(suc)o(h)i(as)f(DONTCARE\))f(will)75 323 y(b)q(e)f(outside)f(the)h (regular)f(range.)19 b(The)c(range)f(of)g(regular)g(v)n(alues)g(can)g(b)q(e)h (queried,)f(and)g(sometimes)f(set,)h(using)75 373 y(en)o(vironmen)o(t)f (inquiry)g(or)h(en)o(vironmen)o(t)f(setting)h(functions)g(\(Section)g Fh(??)p Fo(\).)158 502 y Fm(Discussion:)158 549 y Fl(Need)f(to)g(agree)g(on)h (a)f(language)i(mec)o(hanism)f(for)f(named)h(constan)o(ts.)158 595 y(Implemen)o(ters)j(should)g(detect,)f(whenev)o(er)g(p)q(ossible,)i (illegal)g(uses)e(of)f(\\sp)q(ecial)j(v)n(alues".)25 b(Th)o(us,)16 b(the)g(use)g(of)f(the)75 641 y Ff(DONTCARE)10 b Fl(v)n(alue)k(to)f(tag)g(a)g (message)h(sen)o(t)f(will)h(b)q(e)g(\015agged)g(as)f(an)g(error.)75 845 y Fk(1.2.5)55 b(Choice)75 924 y Fo(MPI)15 b(functions)h(sometimes)d(use)j (parameters)f(with)g(a)f Fj(choic)n(e)i Fo(\(or)f(union\))g(data)f(t)o(yp)q (e.)23 b(I.e.,)14 b(distinct)h(calls)g(to)75 973 y(the)i(same)f(routine)h(ma) o(y)e(pass)i(b)o(y)f(reference)j(actual)d(parameters)h(of)f(di\013eren)o(t)i (t)o(yp)q(es.)27 b(The)17 b(mec)o(hanism)d(for)75 1023 y(pro)o(viding)f(suc)o (h)h(parameters)g(will)f(di\013er)h(from)e(language)h(to)h(language.)158 1153 y Fm(Discussion:)158 1199 y Fl(The)i(F)m(ortran)g(77)g(standard)h(sp)q (eci\014es)h(that)e(the)g(t)o(yp)q(e)g(of)g(actual)h(argumen)o(ts)g(need)f (to)g(agree)g(with)h(the)f(t)o(yp)q(e)g(of)75 1245 y(dumm)o(y)g(argumen)o (ts;)i(no)f(construct)f(equiv)n(alen)o(t)i(to)e(C)g(p)q(oin)o(ters)h(is)f(a)o (v)n(ailable.)29 b(Th)o(us,)16 b(it)g(w)o(ould)h(seem)f(that)g(there)g(is)75 1291 y(no)f(standard)h(conforming)g(mec)o(hanism)g(to)e(supp)q(ort)i(c)o (hoice)g(parameters.)22 b(Ho)o(w)o(ev)o(er,)14 b(most)h(F)m(ortran)g (compiler)h(either)75 1336 y(don't)g(c)o(hec)o(k)g(t)o(yp)q(e)g(consistency)i (of)e(calls)h(to)f(external)h(routines,)g(or)f(supp)q(ort)h(a)f(sp)q(ecial)h (mec)o(hanism)h(to)d(link)j(foreign)75 1382 y(\(e.g.,)d(C\))f(routines.)25 b(I)15 b(suggest)h(that)f(w)o(e)g(accept)g(this)h(nonconformit)o(y)h(with)f (F)m(ortran)f(77)h(standard.)24 b(I.e.,)15 b(w)o(e)f(accept)75 1428 y(that)f(the)g(same)h(routine)g(ma)o(y)f(b)q(e)g(passed)h(an)g(actual)g (parameter)g(of)e(a)h(di\013eren)o(t)i(t)o(yp)q(e)e(at)g(distinct)i(calls.) 158 1474 y(Generic)h(routines)g(can)f(b)q(e)g(used)g(in)g(F)m(ortran)g(90)g (to)g(pro)o(vide)h(a)e(standard)i(conforming)g(solution.)24 b(This)16 b(solution)75 1520 y(will)f(b)q(e)e(consisten)o(t)h(with)g(our)f (nonstandard)i(conforming)g(F)m(ortran)e(77)g(solution.)75 1745 y Fn(1.3)70 b(Pro)r(cesses)75 1921 y Fm(Discussion:)18 b Fl(This)c(material)g(b)q(elongs)h(to)e(an)g(in)o(tro)q(duction)j(or)d(en)o (vironmen)o(t)i(section)158 2054 y Fo(An)k(MPI)g(program)e(is)i(executed)h(b) o(y)f(sev)o(eral)g(autonomous)e(pro)q(cesses)22 b(that)c(eac)o(h)i(execute)g (their)f(o)o(wn)75 2104 y(co)q(de,)d(in)f(a)h(MIMD)f(st)o(yle.)23 b(The)16 b(co)q(des)h(executed)g(b)o(y)f(eac)o(h)g(pro)q(cess)h(need)f(not)g (b)q(e)g(iden)o(tical.)22 b(The)16 b(pro)q(cesses)75 2154 y(comm)o(unicate)d (via)h(calls)h(to)g(MPI)g(comm)o(unication)d(primitiv)o(es.)19 b(T)o(ypically)m(,)13 b(eac)o(h)j(pro)q(cessor)h(executes)g(in)e(its)75 2203 y(o)o(wn)f(address)i(space,)f(although)f(shared-memory)f(implem)o(en)o (tations)f(of)i(MPI)h(are)g(p)q(ossible.)20 b(This)14 b(do)q(cumen)o(t)75 2253 y(sp)q(eci\014es)i(the)f(b)q(eha)o(vior)f(of)g(a)g(parallel)g(program)e (assuming)h(that)i(only)e(MPI)i(calls)f(are)h(used)g(for)f(comm)o(unica-)75 2303 y(tion.)j(The)d(in)o(teraction)f(of)g(an)g(MPI)h(program)e(with)h(other) h(p)q(ossible)f(means)g(of)g(comm)o(unicatio)o(n)e(\(e.g.,)h(shared)75 2353 y(memory\))e(is)i(not)h(sp)q(eci\014ed.)19 b(In)13 b(particular,)f(it)g (is)g(assumed)g(that)h(message)f(bu\013ers)i(at)e(distinct)h(pro)q(cessors)i (are)75 2403 y(disjoin)o(t.)158 2454 y(MPI)e(do)q(es)g(not)f(sp)q(ecify)h (the)g(execution)g(mo)q(del)e(for)h(eac)o(h)h(pro)q(cess.)20 b(A)12 b(pro)q(cess)i(can)f(b)q(e)g(sequen)o(tial,)f(or)h(can)75 2503 y(b)q(e)h(m)o(ultithreaded,)f(with)g(threads)i(p)q(ossibly)e(executing)i (concurren)o(tly)m(.)j(Care)c(has)g(b)q(een)h(tak)o(en)f(to)f(mak)o(e)g(MPI) 75 2553 y(\\thread-safe",)h(b)o(y)f(a)o(v)o(oiding)f(the)j(use)f(of)g (implicit)d(global)h(states.)158 2604 y(The)h(initial)e(allo)q(cation)f(of)i (pro)q(cesses)j(to)e(an)f(MPI)h(computation)e(and)h(their)h(binding)e(to)i (ph)o(ysical)f(pro)q(ces-)75 2654 y(sors)k(is)f(not)h(sp)q(eci\014ed)g(b)o(y) g(the)f(program)f(itself.)22 b(It)16 b(is)f(exp)q(ected)i(that)f(v)o(endors)g (will)e(pro)o(vide)h(mec)o(hanisms)e(to)75 2704 y(do)h(so)g(either)h(at)e (load)g(time)g(or)h(at)g(run)g(time.)j(Suc)o(h)d(mec)o(hanisms)e(will)h(allo) o(w)f(the)i(sp)q(eci\014cation)h(of)e(the)i(initial)p eop %%Page: 4 5 bop 75 -100 a Fo(4)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)75 45 y Fo(n)o(um)o(b)q(er)i(of)g(required)i(pro)q(cesses,) i(the)d(co)q(de)h(to)f(b)q(e)g(executed)i(b)o(y)d(eac)o(h)i(initial)d(pro)q (cess,)j(and)f(the)h(allo)q(cation)75 95 y(of)g(pro)q(cesses)j(to)e(pro)q (cessors.)34 b(Also,)20 b(the)f(curren)o(t)h(prop)q(osal)e(do)q(es)i(not)e (pro)o(vide)h(for)f(dynamic)f(creation)i(or)75 145 y(deletion)14 b(of)f(pro)q(cesses)j(during)e(program)e(execution)i(\(the)h(total)e(n)o(um)o (b)q(er)g(of)g(pro)q(cesses)j(is)e(\014xed\),)g(although)e(it)75 195 y(is)k(in)o(tended)h(to)f(b)q(e)h(consisten)o(t)h(with)e(suc)o(h)h (extension.)26 b(Finally)m(,)14 b(the)j(curren)o(t)h(prop)q(osal)e(do)q(es)h (not)f(sp)q(ecify)h(a)75 244 y(naming)d(sc)o(heme)j(for)f(pro)q(cesses.)29 b(W)m(e)16 b(prop)q(ose)i(to)e(alw)o(a)o(ys)g(iden)o(tify)g(pro)q(cesses)j (according)e(to)f(their)h(relativ)o(e)75 294 y(rank)d(in)f(a)h(con)o(text)g (\(group\),)g(so)g(that,)f(e\013ectiv)o(ely)m(,)h(pro)q(cesses)j(are)d(iden)o (ti\014ed)g(b)o(y)g(consecutiv)o(e)h(in)o(tegers.)75 443 y Fk(1.3.1)55 b(Error)18 b(Handling)75 614 y Fm(Discussion:)34 b Fl(Material)14 b(of)e(this)h(section)g(should)h(b)q(e)e(mo)o(v)o(ed)h(to)f (a)g(general)h(in)o(tro)q(duction)i(section.)j(This)13 b(material)g(has)75 664 y(not)g(y)o(et)g(b)q(een)h(appro)o(v)o(ed)g(b)o(y)g(MPIF)158 802 y Fo(MPI)i(pro)o(vides)h(the)f(user)h(with)f(reliable)g(message)g (transmission:)21 b(A)16 b(message)g(sen)o(t)h(is)f(alw)o(a)o(ys)f(receiv)o (ed)75 852 y(correctly)m(,)20 b(and)f(the)g(user)h(do)q(es)f(not)g(need)g(to) g(c)o(hec)o(k)g(for)g(transmission)e(errors,)k(time-outs,)d(or)h(other)g (error)75 902 y(conditions.)f(In)c(other)g(w)o(ords,)g(MPI)g(do)q(es)h(not)f (pro)o(vide)g(mec)o(hanisms)e(for)h(dealing)g(with)h(failures)f(in)h(the)g (com-)75 952 y(m)o(unication)9 b(system.)17 b(Where)12 b(the)g(MPI)f (implemen)o(tation)d(is)j(built)f(on)h(an)g(unreliable)g(underlying)g(mec)o (hanism,)75 1002 y(then)16 b(it)f(is)g(the)h(job)f(of)g(the)h(implem)o(en)o (ter)e(of)h(the)h(MPI)f(subsystem)h(to)f(insulate)g(the)h(user)h(from)c(this) j(unrelia-)75 1051 y(bilit)o(y)m(,)11 b(or)j(to)f(re\015ect)j(unreco)o(v)o (erable)f(errors)g(as)e(global)f(program)g(failures.)18 b(Similarly)10 b(MPI)k(itself)f(pro)o(vides)h(no)75 1101 y(mec)o(hanisms)e(for)h(handling)g (no)q(de)h(failures.)158 1157 y(Of)c(course,)i(MPI)e(programs)f(ma)o(y)f (still)i(b)q(e)h(erroneous.)18 b(A)10 b Fh(program)g(error)f Fo(can)h(o)q(ccur)i(when)e(an)g(MPI)h(call)75 1207 y(is)i(called)g(with)g(an) g(incorrect)h(parameter)f(\(non-existing)g(destination)g(in)f(a)h(send)h(op)q (eration,)f(bu\013er)h(to)q(o)f(small)75 1257 y(in)k(a)f(receiv)o(e)j(op)q (eration,)e(etc.\))29 b(This)17 b(t)o(yp)q(e)g(of)g(error)h(w)o(ould)e(o)q (ccur)i(in)f(an)o(y)f(implemen)o(tation.)24 b(In)18 b(addition,)75 1307 y(a)f Fh(resource)i(error)e Fo(ma)o(y)f(o)q(ccur)i(when)h(a)e(program)f (exceeds)k(the)e(amoun)o(t)e(of)i(a)o(v)n(ailable)d(system)j(resources)75 1357 y(\(n)o(um)o(b)q(er)d(of)h(p)q(ending)g(messages,)g(system)g(bu\013ers,) h(etc.\).)26 b(The)16 b(o)q(ccurrence)j(of)c(this)h(t)o(yp)q(e)h(of)e(error)i (dep)q(ends)75 1406 y(on)h(the)g(amoun)o(t)e(of)h(a)o(v)n(ailable)f (resources)k(in)e(the)g(system)g(and)f(the)i(resource)h(allo)q(cation)c(mec)o (hanism)g(used;)75 1456 y(this)g(ma)o(y)d(di\013er)j(from)e(system)h(to)h (system.)22 b(The)16 b(recommended)f(implemen)o(tation)d(pro\014le)k(pro)o (vides)f(sev)o(eral)75 1506 y(mec)o(hanisms)d(to)h(alleviate)g(the)h(p)q (ortabilit)o(y)e(problem)g(this)i(represen)o(ts.)21 b(One)14 b(can)g(also)f(write)h Fh(safe)f Fo(programs,)75 1556 y(that)h(are)g(not)g (sub)r(ject)h(to)f(resource)i(errors.)158 1612 y(All)e(MPI)h(pro)q(cedure)i (calls)e(return)h(an)f(error)h(parameter)f(that)g(indicates)g(successful)i (completion)c(of)h(the)75 1662 y(op)q(eration,)f(or)h(the)h(error)f (condition)g(that)g(o)q(ccurred,)h(otherwise.)158 1718 y(The)e(recommended)f (implem)o(en)o(tation)e(pro\014le)i(in)g(a)g(POSIX)i(en)o(vironmen)o(t)d(is)h (for)g(an)o(y)g(MPI)h(routine)g(that)75 1767 y(encoun)o(ters)18 b(a)f(reco)o(v)o(erable)g(error)g(to)g(generate)g(an)g Fj(MPI)g(err)n(or)f (signal)p Fo(,)h(using)f(a)g(sp)q(ecial)h(signal)e(v)n(alue.)26 b(The)75 1817 y(default)10 b(handler)h(for)g(this)g(signal)f(terminates)g (the)i(execution)f(of)f(all)g(in)o(v)o(olv)o(ed)g(pro)q(cesses,)j(with)e(a)f (suitable)h(error)75 1867 y(message)k(b)q(eing)g(returned)i(to)e(the)h(user.) 22 b(Ho)o(w)o(ev)o(er,)16 b(the)g(user)g(can)f(pro)o(vide)g(his)g(or)g(her)h (o)o(wn)f(signal)f(handling)75 1917 y(routine.)j(In)11 b(particular,)f(the)i (user)g(can)e(sp)q(ecify)i(a)e(\\no)q(op")g(signal)g(handler,)h(th)o(us)g (relegating)f(all)g(error)h(handling)75 1967 y(to)j(the)g(user)h(co)q(de,)f (using)g(the)h(error)f(parameters)g(returned)i(b)o(y)e(the)g(MPI)g(calls.)158 2023 y(MPI)h(calls)g(ma)o(y)e(initiate)h(op)q(erations)h(that)g(con)o(tin)o (ue)g(async)o(hronously)g(after)g(the)h(call)e(returned.)23 b(Th)o(us,)75 2072 y(the)d(op)q(eration)e(ma)o(y)f(return)k(with)d(a)h(co)q (de)h(indicating)d(successful)k(completion,)d(y)o(et)h(later)g(cause)h(an)f (error)75 2122 y(exception)d(to)f(b)q(e)g(raised.)22 b(If)15 b(there)h(is)f(a)g(subsequen)o(t)i(call)d(that)h(relates)h(to)f(the)h(same)e (op)q(eration)h(\(e.g.,)f(a)h(call)75 2172 y(that)i(v)o(eri\014es)i(that)e (an)g(async)o(hronous)i(op)q(eration)e(has)h(completed\))f(then)h(the)g (error)g(parameter)f(asso)q(ciated)75 2222 y(with)e(this)g(call)g(will)f(b)q (e)i(used)g(to)f(indicate)h(the)g(nature)g(of)e(the)i(error.)24 b(In)15 b(a)g(few)h(cases,)g(the)g(error)g(ma)o(y)e(o)q(ccur)75 2272 y(after)h(all)f(calls)h(that)g(relate)h(to)e(the)i(op)q(eration)f(ha)o (v)o(e)g(completed,)f(so)h(that)h(no)e(error)i(parameter)f(can)g(b)q(e)h (used)75 2321 y(to)e(indicate)f(the)i(nature)f(of)f(the)i(error)g(\(e.g.,)d (an)i(error)h(in)e(a)g(send)i(with)e(the)i(ready)f(mo)q(de\).)j(In)d(suc)o(h) g(cases,)h(an)75 2371 y(error)g(will)d(b)q(e)j(undetected,)g(if)e(the)i(user) g(disabled)f(the)g(MPI)g(error)h(signal.)158 2506 y Fm(Discussion:)35 b Fl(The)13 b(alternativ)o(e)i(c)o(hoice)f(is)g(to)f(ha)o(v)o(e)g(fatal)h (and)g(non-fatal)g(signals.)158 2558 y(One)f(migh)o(t)h(w)o(an)o(t)f (di\013eren)o(t)h(signals)i(for)c(di\013eren)o(t)j(mo)q(dules.)158 2614 y(The)e(details)i(of)e(suc)o(h)g(prop)q(osal)i(need)f(b)q(e)f(elab)q (orated)i(in)f(an)f(appropriate)i(\\pro\014le")g(sub)q(committee.)p eop %%Page: 5 6 bop 75 -100 a Fg(1.4.)31 b(MESSA)o(GES)1456 b Fo(5)75 45 y Fn(1.4)70 b(Messages)75 136 y Fo(A)14 b(message)g(consists)g(of)g(an)f Fj(envelop)n(e)i Fo(and)f Fj(data)p Fo(.)75 252 y Fk(1.4.1)55 b(Data)75 329 y Fo(The)17 b(data)f(part)h(of)f(a)g(message)g(consists)h(of)f (a)g(sequence)j(of)d(v)n(alues,)g(eac)o(h)h(of)f(a)g(basic)h(datat)o(yp)q(e)g (in)f(the)h(host)75 379 y(language.)29 b(Th)o(us,)19 b(in)e(F)m(ortran)h(77,) f(a)h(message)f(consists)i(of)e(a)h(sequence)i(of)d(v)n(alues)h(that)g(are)g (eac)o(h)g(of)f(t)o(yp)q(e)75 428 y Fi(INTEGER)p Fo(,)12 b Fi(REAL)p Fo(,)h Fi(DOUBLE)20 b(PRECISION)p Fo(,)12 b Fi(COMPLEX)p Fo(,)g Fi(DOUBLE)20 b(PRECISION)g(COMPLEX)p Fo(,)12 b Fi(LOGICAL)p Fo(,)g(or)i(\(length)g(1\))75 478 y Fi(CHARACTER)p Fo(.)i(A)i(message)f(ma)o (y)f(also)h(con)o(tain)h(a)f(v)n(alue)h(of)f(t)o(yp)q(e)h Fi(BYTE)p Fo(,)f(whic)o(h)h(consists)h(of)e(8)h(binary)f(digits.)75 528 y(A)e(b)o(yte)g(is)g(di\013eren)o(t)h(from)d(a)i(c)o(haracter.)23 b(Di\013eren)o(t)15 b(mac)o(hines)f(ma)o(y)f(ha)o(v)o(e)i(di\013eren)o(t)h (represen)o(tations)h(for)d(the)75 578 y(same)e(c)o(haracter,)i(or)f(ma)o(y)e (use)i(more)f(than)h(one)g(b)o(yte)g(to)g(represen)o(t)i(a)e(c)o(haracter.)19 b(On)13 b(the)g(other)h(hand,)e(a)h(b)o(yte)75 628 y(has)h(the)h(same)e (binary)g(v)n(alue)g(on)h(an)o(y)f(mac)o(hine.)k(A)d(message)g(ma)o(y)d(mix)h (v)n(alues)i(of)f(di\013eren)o(t)i(t)o(yp)q(es.)158 756 y Fm(Missing:)158 802 y Fl(Need)g(to)h(agree)g(on)f(the)h(C)f(t)o(yp)q(es)h(\(including)i (handling)g(of)e(signed/unsign)q(ed\),)i(and)e(on)g(F)m(ortran)g(90)f(t)o(yp) q(es)h(\(in-)75 847 y(cluding)f(handling)h(of)d(kinds\).)75 1046 y Fk(1.4.2)55 b(En)n(v)n(elop)r(e)75 1123 y Fo(The)14 b(follo)o(wing)e(information)f(is)i(asso)q(ciated)i(with)e(eac)o(h)i (message:)75 1205 y Fh(source)20 b Fo(The)14 b(rank)g(of)f(the)i(sending)f (pro)q(cess)75 1288 y Fh(destinatio)o(n)k Fo(The)c(rank)g(of)f(the)i (receiving)f(pro)q(cess)75 1370 y Fh(tag)20 b Fo(User)15 b(de\014ned)75 1453 y Fh(con)o(text)k Fo(handle)158 1536 y(The)14 b(range)g(of)f(v)n(alid)f (v)n(alues)h(for)g(the)h Fh(source)f Fo(and)g Fh(destination)d Fo(\014elds)k(is)f Fi(0)22 b(...)43 b(n-1)p Fo(,)12 b(where)j Fi(n)e Fo(is)g(the)75 1585 y(n)o(um)o(b)q(er)e(of)h(pro)q(cesses)j(in)c(the)i (sp)q(eci\014ed)h(con)o(text.)k Fi(source)j(=)g(destination)10 b Fo(is)i(allo)o(w)o(ed:)k(a)c(pro)q(cess)i(can)e(send)75 1635 y(a)i(message)f(to)h(itself.)158 1685 y(The)g(ranges)h(of)e(v)n(alid)f(v)n (alues)i(for)f Fi(tag)g Fo(is)h(implem)o(en)o(tation)d(dep)q(enden)o(t,)k (and)f(can)g(b)q(e)g(found)f(b)o(y)h(calling)e(a)75 1735 y(suitable)i(query)g (function,)f(as)h(describ)q(ed)i(in)d(Section)i Fh(??)p Fo(.)158 1785 y(The)f Fi(tag)e Fo(\014eld)i(can)f(b)q(e)h(arbitrarily)e(set)j(b)o(y)e (the)h(application,)d(and)i(can)h(b)q(e)g(used)g(to)f(distinguish)g (di\013eren)o(t)75 1835 y(messages.)158 1884 y Fi(Context)f Fo(should)i(b)q(e)h(a)e(con)o(text)i(shared)g(b)o(y)e(b)q(oth)h(source)h(and) f(destination.)158 1934 y(The)g(actual)g(mec)o(hanism)d(used)k(to)f(asso)q (ciate)g(an)g(en)o(v)o(elop)q(e)g(with)g(a)f(message)h(is)g(implem)o(en)o (tation)d(dep)q(en-)75 1984 y(den)o(t;)h(some)f(of)g(the)i(information)8 b(\(e.g.,)j Fh(sender)f Fo(or)i Fh(receiv)o(er)p Fo(\))e(ma)o(y)g(b)q(e)i (implicit,)d(and)i(need)i(not)f(b)q(e)g(explicitly)75 2034 y(carried)j(b)o(y)e(a)h(message.)75 2171 y Fn(1.5)70 b(Comm)n(unication)20 b(Bu\013ers)75 2262 y Fo(The)h(basic)f(p)q(oin)o(t)g(to)g(p)q(oin)o(t)g(comm) o(unicatio)o(n)e(op)q(erations)i(are)h Fh(send)e Fo(and)h Fh(receiv)o(e)p Fo(.)36 b(A)20 b Fh(send)f Fo(op)q(eration)75 2312 y(creates)c(a)d(message;)g (the)i(message)e(data)h(is)f(tak)o(en)h(from)e(the)j Fh(send)f(bu\013er)p Fo(.)j(A)d Fh(receiv)o(e)e Fo(op)q(eration)i(consumes)75 2361 y(a)j(message;)h(the)g(message)f(data)g(is)g(put)h(in)o(to)e(the)i Fh(receiv)o(e)h(bu\013er)p Fo(.)23 b(The)17 b(sp)q(eci\014cation)g(of)f(send) h(or)g(receiv)o(e)75 2411 y(bu\013ers)e(uses)g(the)g(same)e(syn)o(tax.)75 2527 y Fk(1.5.1)55 b(Comm)n(unication)21 b(bu\013er)d(comp)r(onen)n(ts)75 2604 y Fo(A)d(bu\013er)g(consists)h(of)d(a)i(sequence)h(of)e Fh(bu\013er)h(comp)q(onen)o(ts)p Fo(.)j(Eac)o(h)d(comp)q(onen)o(t)e(consists) j(of)e(a)g(sequence)j(of)75 2654 y(\(not)f(necessarily)g(distinct\))g(v)n (ariables)f(of)g(the)h(same)f(basic)h(datat)o(yp)q(e.)23 b(The)16 b(datat)o(yp)q(e)g(of)f(suc)o(h)h(a)g(v)n(ariable)e(is)75 2704 y(sp)q(eci\014ed)g(b)o(y)f(a)f(state)i(parameter.)j(The)d(p)q(ossible)f(v)n (alues)f(for)h(F)m(ortran)f(are)h Fi(MPI)p 1357 2704 14 2 v 15 w(INT)p Fo(,)f Fi(MPI)p 1528 2704 V 15 w(REAL)p Fo(,)g Fi(MPI)p 1721 2704 V 15 w(DOUBLE)p Fo(,)p eop %%Page: 6 7 bop 75 -100 a Fo(6)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)75 45 y Fi(MPI)p 144 45 14 2 v 15 w(COMPLEX)p Fo(,)k Fi(MPI)p 409 45 V 15 w(DCOMPLEX)p Fo(,)g Fi(MPI)p 696 45 V 15 w(LOGICAL)p Fo(,)h Fi(MPI)p 962 45 V 15 w(CHAR)g Fo(and)h Fi(MPI)p 1237 45 V 15 w(BYTE)p Fo(,)f(corresp)q(onding)i(to)f(datat)o(yp)q (es)75 95 y Fi(INTEGER)p Fo(,)e Fi(REAL)p Fo(,)g Fi(DOUBLE)j(PRECISION)p Fo(,)c Fi(COMPLEX)p Fo(,)h Fi(DOUBLE)i(PRECISION)g(COMPLEX)p Fo(,)e Fi(LOGICAL)p Fo(,)g Fi(CHARACTER)75 145 y Fo(and)j(to)g(un)o(t)o(yp)q (ed)g Fi(BYTE)p Fo(.)f(The)h(p)q(ossible)g(v)n(alues)g(for)g(C)g(are)g Fi(MPI)p 1162 145 V 15 w(CHAR)p Fo(,)f Fi(MPI)p 1363 145 V 15 w(SHORT)p Fo(,)f Fi(MPI)p 1585 145 V 15 w(INT)p Fo(,)h Fi(MPI)p 1764 145 V 15 w(LONG)p Fo(,)75 195 y Fi(MPI)p 144 195 V 15 w(FLOAT)p Fo(,)12 b Fi(MPI)p 359 195 V 15 w(DOUBLE)h Fo(and)h Fi(MPI)p 666 195 V 15 w(BYTE)p Fo(.)158 325 y Fm(Discussion:)158 372 y Fl(Ma)o(y)g(need)g(to)g(tak)o(e)g(care)f(of)h(kinds)h(in)f(F)m(ortran)g (90)g(and)g(of)g(signed/unsign)q(ed)j(in)d(C.)f(More)h(elab)q(orate)h (de\014nitions)75 418 y(b)q(elong)g(to)e(the)g(language)i(binding.)158 552 y Fo(There)g(are)f(\014v)o(e)g(kinds)g(of)g(bu\013er)g(comp)q(onen)o(ts:) 75 670 y Fh(Con)o(tiguous)f(comp)q(onen)o(t)75 750 y Fo(A)h(sequence)i(of)d (con)o(tiguous)h(v)n(alues)g(of)f(the)h(same)f(basic)h(t)o(yp)q(e,)g(sp)q (eci\014ed)i(b)o(y)75 839 y Fh(start)k Fo(Initial)12 b(elemen)o(t)75 930 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 1020 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1109 y(Th)o(us,)h(if)e Fi(A)i Fo(is)f(an)h(arra)o(y)f(of)g(in)o(tegers,)h(then)h Fi(A\(1\),)21 b(4,)g(MPI)p 1127 1109 V 15 w(INT)13 b Fo(is)h(the)g(bu\013er)h(comp)q(onen)o (t)e(with)g(en)o(tries)75 1159 y Fi(A\(1\),)21 b(A\(2\),)g(A\(3\),)f(A\(4\))p Fo(.)75 1277 y Fh(V)l(ector)15 b(comp)q(onen)o(t)75 1357 y Fo(A)f(sequence)j(of)c(equally)h(spaced)h(and)f(equally)f(sized)i(blo)q(c)o (ks)f(of)g(elemen)o(ts)g(of)g(the)g(same)g(basic)g(t)o(yp)q(e,)g(sp)q (eci\014ed)75 1407 y(b)o(y)75 1495 y Fh(start)20 b Fo(Initial)12 b(elemen)o(t)75 1586 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 1677 y Fh(stride)18 b Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(b)q(et)o(w)o(een)h (the)g(start)f(of)f(eac)o(h)i(blo)q(c)o(k)75 1767 y Fh(len)o(blk)j Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\()p Fi(lenblk)c Fd(>)i Fi(0)p Fo(\))75 1858 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1947 y(Assume)20 b(that)h(an)f(elemen) o(t)g(of)f(t)o(yp)q(e)i Fi(datatype)e Fo(o)q(ccupies)i Fi(m)f Fo(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75 1996 y(mac)o(hine,)11 b(the)i(bu\013er)h(comp)q(onen)o(t)d(describ)q(ed)k(b)o (y)d(these)i(\014v)o(e)f(parameters)f(consists)i(of)e(the)h Fi(count)e Fo(v)n(ariables)h(at)75 2046 y(addresses)k Fi(start,)21 b(start+m,)f(...)43 b(,)21 b(start+m*\(lenblk-1\),)d(start+m*stride,)75 2096 y(start+m*\(stride+1)o(\),)h(...,)i(start+m*\(stride+l)o(enbl)o(k-1\),)d (...)p Fo(.)i(F)m(or)14 b(example,)f(if)h Fi(A)g Fo(is)h(an)f(arra)o(y)h(of) 75 2146 y(in)o(tegers,)f(then)137 2234 y Fe(\017)21 b Fi(A\(1\),)g(5,)g(3,)g (2,)h(MPI)p 575 2234 V 15 w(INT)c Fo(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e (with)h(en)o(tries)h Fi(A\(1\),)h(A\(2\),)f(A\(4\),)h(A\(5\),)179 2284 y(A\(7\))p Fo(.)137 2375 y Fe(\017)g Fi(A\(1\),)g(5,)g(0,)g(2,)h(MPI)p 575 2375 V 15 w(INT)c Fo(is)h(the)h(bu\013er)g(comp)q(onen)o(t)e(with)h(en)o (tries)h Fi(A\(1\),)h(A\(2\),)f(A\(1\),)h(A\(2\),)179 2425 y(A\(1\))p Fo(.)137 2515 y Fe(\017)g Fi(A\(10\),)f(5,)i(-1,)f(2,)g(MPI)p 618 2515 V 15 w(INT)9 b Fo(is)g(the)h(bu\013er)g(comp)q(onen)o(t)f(with)g(en) o(tries)h Fi(A\(10\),)21 b(A\(11\),)f(A\(9\),)h(A\(10\),)179 2565 y(A\(8\))p Fo(.)158 2654 y Fi(Contiguous)13 b Fo(is)h(a)g(sp)q(ecial)h (case)h(of)e Fi(vector)p Fo(.)19 b(A)c(v)o(ector)g(bu\013er)h(comp)q(onen)o (t)d(can)i(b)q(e)h(used)f(to)f(extract)i(an)75 2704 y(arbitrary)e(submatrix)e (of)i(a)f(t)o(w)o(o-dimensional)e(matrix.)p eop %%Page: 7 8 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1091 b Fo(7)75 45 y Fh(Indexed)15 b(comp)q(onen)o(t)75 122 y Fo(A)f(sequence)i(of) d(elemen)o(ts)h(of)f(the)i(same)e(basic)h(t)o(yp)q(e,)g(sp)q(eci\014ed)h(b)o (y)75 207 y Fh(start)20 b Fo(initial)12 b(elemen)o(t)75 291 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 376 y Fh(arra)o(y)p 188 376 15 2 v 17 w(of)p 244 376 V 16 w(indices)19 b Fo(Arra)o(y)14 b(of)f(displacemen)o(ts)h(of)f(the)i(elemen)o(ts)e(in)h(the)h(bu\013er)g (comp)q(onen)o(ts,)e(relativ)o(e)h(to)f(the)179 425 y(initial)f(elemen)o(t.) 75 510 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 594 y(F)m(or)f(example,)f(if)h Fi(A)g Fo(is)g(an)g(arra)o(y)g(of)g(in)o (tegers,)h(and)g Fi(I)f Fo(is)g(an)g(arra)o(y)h(with)f(v)n(alues)g Fi(2,)21 b(-1,)g(2,)h(1)12 b Fo(then)h Fi(A\(3\),)75 644 y(4,)21 b(I,)h(MPI)p 275 644 14 2 v 15 w(INT)13 b Fo(is)h(the)g(bu\013er)h(comp)q (onen)o(t)e(with)h(en)o(tries)h Fi(A\(5\),)21 b(A\(2\),)f(A\(5\),)h(A\(4\))p Fo(.)158 694 y Fi(Vector)13 b Fo(can)h(b)q(e)g(though)o(t)g(of)f(as)h(a)g(sp) q(ecial)g(case)h(of)e Fi(indexed)g Fo(in)g(whic)o(h)h(the)g(indices)h(are)f (generated)h(with)75 744 y(a)f(constan)o(t)g(stride.)75 854 y Fh(Heterogeneous)f(v)o(ector)i(\(h)o(v)o(ector\))f(comp)q(onen)o(t)75 931 y Fo(Same)d(as)h Fi(vector)p Fo(,)f(except)j(that)e Fi(stride)f Fo(is)h(giv)o(en)g(in)f(m)o(ultiple)f(of)i(b)o(ytes,)h(rather)g(than)f(m)o (ultiples)e(of)i(elemen)o(ts:)75 1024 y Fh(start)20 b Fo(Initial)12 b(elemen)o(t)75 1108 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))75 1193 y Fh(stride)18 b Fo(Num)o(b)q(er)c(of)f(b)o(ytes)i(b)q(et)o(w)o(een)g (the)g(start)f(of)f(eac)o(h)i(blo)q(c)o(k)75 1278 y Fh(len)o(blk)j Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h(eac)o(h)g(blo)q(c)o(k)g(\()p Fi(lenblk)c Fd(>)i Fi(0)p Fo(\))75 1362 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 1455 y(Assume)20 b(that)h(an)f(elemen) o(t)g(of)f(t)o(yp)q(e)i Fi(datatype)e Fo(o)q(ccupies)i Fi(m)f Fo(b)o(ytes.)38 b(Then,)22 b(in)e(a)g(b)o(yte)h(addressable)75 1505 y(mac)o(hine,)15 b(the)i(bu\013er)g(comp)q(onen)o(t)f(describ)q(ed)i(b)o (y)e(these)i(\014v)o(e)e(parameters)h(consists)g(of)f(the)g Fi(count)g Fo(v)n(ariables)75 1554 y(at)c(addresses)j Fi(start,)20 b(start+m,)h(...)43 b(,)21 b(start+m*\(lenblk-1\))o(,)e(start+stride,)g (start+stride+m,)75 1604 y(...,)i(start+stride+m*\(l)o(enblk)o(-1\),)d(...)p Fo(.)158 1654 y(Consider)c(a)g(C)g(arra)o(y)f(declared)i(as)75 1748 y Fi(struct)21 b({)140 1798 y(char)g(info[3];)140 1847 y(short)g(number;)140 1897 y(double)g(val;)228 1947 y(})65 b(a[100])158 2040 y Fo(Assume)19 b(that)f(all)g(structure)j(comp)q(onen)o(ts) d(are)h(b)o(yte)h(aligned,)e(a)h Fi(short)e Fo(o)q(ccupies)j(t)o(w)o(o)e(b)o (ytes,)j(and)d(a)75 2090 y Fi(double)i Fo(o)q(ccupies)i(eigh)o(t)e(b)o(ytes.) 40 b(Then)21 b(the)h(bu\013er)g(comp)q(onen)o(t)e Fi(&a,)h(100,)g(13,)g(2,)g (MPI)p 1610 2090 V 16 w(CHAR)f Fo(extracts)75 2139 y(the)i(\014rst)h(t)o(w)o (o)e(c)o(haracters)i(of)e(eac)o(h)h(structure.)43 b(The)22 b(bu\013er)h(comp)q(onen)o(t)e Fi(&\(a.number\),)e(100,)i(13,)g(1,)75 2189 y(MPI)p 144 2189 V 15 w(SHORT)16 b Fo(extracts)i(the)f Fi(short)f Fo(\014eld)g(of)g(eac)o(h)i(structure;)h(and)e(the)g(bu\013er)h (comp)q(onen)o(t)e Fi(&\(a.val\),)k(100,)75 2239 y(13,)h(1)e Fo(extracts)i(the)f Fi(double)e Fo(\014eld)h(of)g(eac)o(h)h(structure.)36 b(Th)o(us,)21 b(heterogeneous)g(v)o(ectors)f(can)g(b)q(e)g(used)g(to)75 2289 y(extract)15 b(v)o(ectors)g(of)e(homogeneous)g(v)n(alues)h(from)e (heterogeneous)k(data.)158 2339 y Fi(Vector)d Fo(is)g(a)h(sp)q(ecial)g(case)h (of)e Fi(heterogeneous)19 b(vector)p Fo(.)75 2449 y Fh(Heterogeneous)13 b(indexed)h(\(hindexed\))f(comp)q(onen)o(t)75 2526 y Fo(Same)g(as)h(indexed,) g(except)h(that)f(displacemen)o(ts)f(are)i(in)e(b)o(ytes,)h(rather)h(than)f (m)o(ultiples)e(of)h(elemen)o(t)g(size.)75 2619 y Fh(start)20 b Fo(initial)12 b(elemen)o(t)75 2704 y Fh(coun)o(t)19 b Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\()p Fi(count)c Fe(\025)i Fi(0)p Fo(\))p eop %%Page: 8 9 bop 75 -100 a Fo(8)733 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g (COMMUNICA)m(TION)75 45 y Fh(arra)o(y)p 188 45 15 2 v 17 w(of)p 244 45 V 16 w(indices)19 b Fo(Arra)o(y)12 b(of)g(b)o(yte)g(displacemen)o(ts)g (of)g(the)g(elemen)o(ts)h(in)e(the)i(bu\013er)g(comp)q(onen)o(ts,)f(relativ)o (e)g(to)179 95 y(the)i(initial)e(elemen)o(t.)75 189 y Fh(datat)o(yp)q(e)19 b Fo(T)o(yp)q(e)c(of)e(elemen)o(ts)158 292 y(F)m(or)18 b(example,)f(if)h Fi(a)g Fo(is)g(the)h(C)f(arra)o(y)g(of)f(the)i(previous)g(example,)e(and)i Fi(I)e Fo(is)i(an)f(arra)o(y)g(with)f(v)n(alues)h Fi(0,)75 342 y(13,)j(13,)g(27,)h(26)16 b Fo(then)i Fi(&a,)k(5,)f(I,)g(MPI)p 782 342 14 2 v 15 w(CHAR)c Fo(is)g(the)h(bu\013er)h(comp)q(onen)o(t)d(with)h (en)o(tries)i Fi(a[0].info[0],)75 392 y(a[1].info[0],)g(a[1].info[0],)g (a[1].info[1],)g(a[1].info[0])p Fo(.)158 444 y(Lik)o(e)e Fi(heterogeneous)i (vector)p Fo(,)d Fi(heterogeneous)j(indexed)c Fo(extends)k(the)e Fi(indexed)f Fo(facilit)o(y)f(so)i(as)g(to)75 494 y(allo)o(w)12 b(the)j(extraction)f(of)f(homogeneous)g(data)h(from)e(a)h(heterogeneous)j(la) o(y)o(out.)158 625 y Fm(Discussion:)158 674 y Fl(The)d(use)h(of)e (displacemen)o(ts)k(in)e(indexed)h(bu\013er)f(comp)q(onen)o(ts)h(has)e(the)h (same)f(e\013ect)g(as)h(indirect)h(addressing)g(in)o(to)75 720 y(an)d(arra)o(y)g(with)g(initial)j(index)e(zero.)j(I.e.,)11 b(if)h Ff(a)f Fl(is)h(an)g(arra)o(y)g(with)g(initial)j(index)e(zero,)e(then)h (a)g(bu\013er)h(comp)q(onen)o(t)g(de\014ned)75 765 y(b)o(y)g(parameters)g Ff(a[0],)k(N,)i(i,)g(type)11 b Fl(extract)h(the)h(elemen)o(ts)g Ff(a[i[0]],)j(a[i[1]],)h(...,)h(a[i[n]])p Fl(.)c(This)f(\014ts)f(w)o(ell)h(C) 75 811 y(usage)g(but)f(ill-\014ts)i(F)m(ortran)e(usage)h(\(initial)h(index)g (one\).)i(W)m(e)c(to)q(ok)h(a)f(decision)i(of)d(principle)k(to)d(force)g (\\start)g(from)f(zero")75 857 y(index)j(arithmetic)f(on)g(F)m(ortran)f (users,)h(rather)f(than)h(ha)o(v)o(e)f(di\013eren)o(t)i(index)g(arithmetic)f (in)g(the)f(C)g(and)h(F)m(ortran)f(binding)75 902 y(of)h(MPI.)158 951 y(The)19 b(use)f(of)h(b)q(oth)g(bu\013er)h(comp)q(onen)o(ts)f(with)h(b)o (yte)e(and)i(elemen)o(t)f(size)g(displacemen)o(ts)j(is)d(motiv)n(ated)h(b)o (y)f(the)75 996 y(observ)n(ation)c(that)f(some)g(applications)j(require)d (the)g(more)f(general)i(b)o(yte)f(displacemen)o(t,)h(but)f(that)g(most)f (applicatio)q(ns)75 1042 y(w)o(ould)d(use)g(elemen)o(t)g(size)h(displacemen)o (t,)h(whic)o(h)e(is)g(m)o(uc)o(h)g(more)g(natural)g(and)g(con)o(v)o(enien)o (t)i(\(and)e(mac)o(hine)h(indep)q(enden)o(t\).)75 1088 y(Th)o(us,)f(rather)f (than)h(forcing)g(all)g(users)g(to)f(do)g(b)o(yte)h(arithmetics,)h(w)o(e)e (decided)h(to)f(pro)o(vide)i(separately)g(the)e(more)g(adv)n(anced)75 1133 y(function)14 b(to)f(\\exp)q(ert)h(programmers".)158 1182 y(Leslie)g(Hart)f(prop)q(oses)i(to)d(replace)j(\\heterogeneous")g(with)e (\\general".)75 1396 y Fk(1.5.2)55 b(Bu\013er)18 b(op)r(erations)75 1478 y Fo(A)c(bu\013er)i(is)e(describ)q(ed)i(b)o(y)e(an)g(opaque)g Fh(bu\013er)h(descriptor)c Fo(ob)r(ject)k(accessed)h(via)e(a)g Fh(bu\013er)h(handle)p Fo(.)i(Suc)o(h)75 1528 y(an)f(ob)r(ject)g(is)g (created)h(b)o(y)f Fi(MPI)p 582 1528 V 15 w(CREATE)p 729 1528 V 15 w(BUFFER)p Fo(.)e(It)i(is)f(asso)q(ciated)i(with)f(successiv)o(e)i (bu\013er)f(comp)q(onen)o(ts)e(b)o(y)75 1577 y(calling)g(in)i(succession)h (one)f(of)f(the)i(functions)e Fi(MPI)p 914 1577 V 15 w(APPEND)p 1061 1577 V 15 w(xxx)p Fo(,)g(where)i Fi(xxx)e Fo(iden)o(ti\014es)h(the)h(t)o (yp)q(e)f(of)f(bu\013er)75 1627 y(comp)q(onen)o(t)e(app)q(ended)h(to)f(the)h (bu\013er.)20 b(It)14 b(b)q(ecomes)h(a)o(v)n(ailable)d(for)i(use)h(in)f(comm) o(unicatio)o(n)e(op)q(erations)i(once)75 1677 y(committed)c(b)o(y)h(the)i (function)e Fi(MPI)p 635 1677 V 15 w(COMMIT)p 782 1677 V 15 w(BUFFER)p Fo(.)f(Once)j(committed,)c(the)k(descriptor)g(cannot)f(b)q(e)g(mo) q(di\014ed)75 1727 y(an)o(y)h(more.)158 1780 y(Note)18 b(that)g(the)g(commit) c(op)q(eration)k(do)q(es)g(not)g(create)h(y)o(et)f(a)f(message,)h(and)f(do)q (es)h(not)g(bind)f(the)h(data)75 1829 y(to)d(b)q(e)h(sen)o(t.)23 b(It)16 b(is)f(a)g(commit)d(for)j(the)h(bu\013er)g(descriptor,)h(not)e(a)g (commit)d(for)j(the)h(bu\013er)h(con)o(ten)o(t.)23 b(A)15 b(bu\013er)75 1879 y(descriptor)f(is)e(destro)o(y)o(ed)i(after)f(the)g(completion)d(of)i (the)h(\014rst)h(comm)o(unicati)o(on)c(op)q(eration)i(that)h(uses)g(it,)f(if) g(it)g(is)75 1929 y(ephemeral,)g(or)h(after)g(it)g(is)g(freed)h(b)o(y)e(a)h (call)f(to)h Fi(MPI)p 899 1929 V 15 w(FREE)f Fo(and)h(an)o(y)f(p)q(ending)i (comm)o(uni)o(cation)c(op)q(eration)j(that)75 1979 y(uses)j(it)e(has)h (completed,)f(if)f(it)i(is)f(p)q(ersisten)o(t.)22 b(After)15 b(the)h(bu\013er)f(descriptor)h(ob)r(ject)f(has)g(b)q(een)h(destro)o(y)o(ed,) f(the)75 2029 y(handle)f(is)f(unde\014ned.)158 2117 y Fh(MPI)p 257 2117 15 2 v 17 w(CREA)l(TE)p 471 2117 V 19 w(BUFFER\()i(bu\013er,)g(p)q (ersistence)e(\))158 2205 y Fo(Create)i(a)e(new)i(bu\013er)g(descriptor)g(ob) r(ject.)k(The)14 b(parameters)g(are:)75 2296 y Fh(OUT)i(bu\013er)j Fo(bu\013er)c(handle)75 2391 y Fh(IN)h(p)q(ersistence)i Fo(bu\013er)11 b(p)q(ersistence)h(\(state)e(t)o(yp)q(e)g(with)f(p)q(ossible)g(v)n(alues)g Fi(MPI)p 1350 2391 14 2 v 15 w(PERSISTENT)e Fo(or)j Fi(MPI)p 1705 2391 V 15 w(EPHEMERAL)p Fo(\).)158 2561 y Fm(Discussion:)158 2609 y Fl(Ma)o(y)j(w)o(an)o(t)g(to)g(pro)o(vide)i(information)g(on)e(maxim)o (um)h(ob)r(ject)g(size,)f(and)h(p)q(erhaps)g(user)f(space)h(for)f(the)g(ob)r (ject.)158 2658 y(Leslie)20 b(Hart)e(prop)q(oses)i Ff(MPI)p 592 2658 12 2 v 13 w(CREATE)p 725 2658 V 12 w(BUFFER)p 857 2658 V 12 w(DESC)p Fl(,)c(rather)j(than)g Ff(MPI)p 1252 2658 V 13 w(CREATE)p 1385 2658 V 12 w(BUFFER)p Fl(.)c(More)k(accurate,)h(but)75 2704 y(longer.)e(P)o(erhaps)c Ff(MPI)p 414 2704 V 13 w(CREATE)p 547 2704 V 12 w(BUFDESC)p Fl(?)p eop %%Page: 9 10 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1091 b Fo(9)158 45 y Fh(MPI)p 257 45 15 2 v 17 w(APPEND)p 481 45 V 17 w(CONTIGUOUS\()15 b(bu\013er,)g(start,)g(coun)o(t,)f(datat)o(yp)q(e\)) 158 130 y Fo(App)q(end)h(a)e(blo)q(c)o(k)h(comp)q(onen)o(t)f(to)h(bu\013er.) 19 b(The)14 b(parameters)g(are:)75 208 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 289 y Fh(IN)h(start)k Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))75 369 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 450 y Fh(IN)i(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 563 y Fh(MPI)p 257 563 V 17 w(APPEND)p 481 563 V 17 w(VEC\()j(bu\013er,)d(coun)o(t,)h(stride,)f(len)o (blk,)f(datat)o(yp)q(e)i(\))158 648 y Fo(App)q(end)g(a)e(v)o(ector)i (bu\013er)g(comp)q(onen)o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:) 75 726 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 807 y Fh(IN)h(start)k Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h (\(c)o(hoice\))75 887 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 967 y Fh(IN)i(stride)j Fo(Num)o(b)q(er)13 b(of)h(elemen)o(ts)f(b)q(et)o(w)o(een)j(the)e(start)h(of)e (eac)o(h)h(blo)q(c)o(k)g(\(in)o(teger\))75 1048 y Fh(IN)i(len)o(blk)i Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h)i(blo)q(c)o(k)e(\(in)o (teger\))75 1128 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 1242 y Fh(MPI)p 257 1242 V 17 w(APPEND)p 481 1242 V 17 w(INDEXED\()i(bu\013er,)f(start,)g(coun)o(t,)f (arra)o(y)p 1294 1242 V 17 w(of)p 1350 1242 V 17 w(indices,)f(datat)o(yp)q (e\))158 1327 y Fo(App)q(end)i(an)e(indexed)i(bu\013er)g(comp)q(onen)o(t)e (to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 1405 y Fh(INOUT)i (bu\013er)j Fo(bu\013er)c(handle)75 1485 y Fh(IN)h(start)k Fo(initial)12 b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75 1566 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 1646 y Fh(IN)i(arra)o(y)p 259 1646 V 17 w(of)p 315 1646 V 17 w(indices)i Fo(arra)o(y)c(of)f(displacemen)o(ts)g(of)h(en)o (tries)h(relativ)o(e)e(to)h(start)h(\(arra)o(y)e(of)h(in)o(tegers\))75 1727 y Fh(IN)i(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g (\(status\))158 1805 y(The)g(index)f(v)n(alues)g(used)h(to)f(create)i(this)f (comp)q(onen)o(t)e(are)i(the)g(v)n(alues)f(pro)o(vided)g(at)g(the)h(call;)e (subsequen)o(t)75 1854 y(up)q(dates)j(of)e(the)i(arra)o(y)e(of)h(indices)g (will)e(not)i(a\013ect)h(the)f(bu\013er)h(descriptor)g(comp)q(onen)o(t.)158 1940 y Fh(MPI)p 257 1940 V 17 w(APPEND)p 481 1940 V 17 w(HVEC\()i(bu\013er,)e (start,)g(coun)o(t,)f(stride,)g(len)o(blk,)f(datat)o(yp)q(e)i(\))158 2025 y Fo(App)q(end)g(a)e(heterogeneous)j(v)o(ector)f(bu\013er)g(comp)q(onen) o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)75 2103 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 2183 y Fh(IN)h(start)k Fo(bu\013er)15 b(comp)q(onen)o(t)e(initial)f(elemen)o(t)h(\(c)o(hoice\))75 2264 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o (teger\))75 2344 y Fh(IN)i(stride)j Fo(Num)o(b)q(er)13 b(of)h(b)o(ytes)g(b)q (et)o(w)o(een)h(the)g(start)f(of)g(eac)o(h)g(blo)q(c)o(k)g(\(in)o(teger\))75 2425 y Fh(IN)i(len)o(blk)i Fo(Num)o(b)q(er)c(of)f(elemen)o(ts)h(in)f(eac)o(h) i(blo)q(c)o(k)e(\(in)o(teger\))75 2505 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g(\(status\))158 2618 y Fh(MPI)p 257 2618 V 17 w(APPEND)p 481 2618 V 17 w(HINDEXED\()i(bu\013er,)f (start,)g(coun)o(t,)g(arra)o(y)p 1332 2618 V 16 w(of)p 1387 2618 V 17 w(indices,)f(datat)o(yp)q(e\))158 2704 y Fo(App)q(end)h(an)e (heterogeneous)j(indexed)f(bu\013er)g(comp)q(onen)o(t)e(to)h(bu\013er.)19 b(The)14 b(parameters)g(are:)p eop %%Page: 10 11 bop 75 -100 a Fo(10)716 b Fg(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fh(INOUT)i(bu\013er)j Fo(bu\013er)c(handle)75 124 y Fh(IN)h(start)k Fo(initial)12 b(p)q(osition)h(for)h(indexing)f(\(c)o(hoice\))75 203 y Fh(IN)j(coun)o(t)j Fo(Num)o(b)q(er)14 b(of)f(elemen)o(ts)h(\(in)o(teger\))75 282 y Fh(IN)i(arra)o(y)p 259 282 15 2 v 17 w(of)p 315 282 V 17 w(indices)i Fo(arra)o(y)c(of)f(b)o(yte)h(displacemen)o(ts)g(of)f(en)o(tries)i (relativ)o(e)f(to)f(start)i(\(arra)o(y)f(of)f(in)o(tegers\))75 361 y Fh(IN)j(datat)o(yp)q(e)k Fo(datat)o(yp)q(e)14 b(iden)o(ti\014er)g (\(status\))158 471 y Fh(MPI)p 257 471 V 17 w(COMMIT)p 485 471 V 19 w(BUFFER\()h(bu\013erin)e(\))158 557 y Fo(Commi)o(t)7 b(a)j(bu\013er)h(descriptor)g(ob)r(ject.)18 b(The)10 b(ob)r(ject)h(cannot)f (b)q(e)g(c)o(hanged)h(an)o(ymore)d(after)i(it)g(is)g(committed.)75 606 y(A)h(bu\013er)g(descriptor)h(ob)r(ject)g(can)f(b)q(e)g(used)g(for)g (comm)o(uni)o(cation)d(only)i(after)g(it)h(is)f(committed.)15 b(The)c(parameters)75 656 y(are:)75 731 y Fh(INOUT)16 b(bu\013er)j Fo(bu\013er)c(descriptor)g(handle)158 885 y Fm(Discussion:)158 930 y Fl(The)i(commit)h(function)g(ma)o(y)g(do)f(prepro)q(cessing)j(of)d(the) g(bu\013er)h(descriptor)h(to)e(simplify)i(its)f(use)f(and)h(commit)75 976 y(resources)12 b(in)g(the)g(comm)o(unication)i(subsystem)e(for)f(its)h (use.)k(The)c(commit)f(function)i(ma)o(y)f(generate)g(a)f(new)g(ob)r(ject)g (and,)75 1022 y(hence,)i(mo)q(dify)h(the)g(ob)r(ject)f(handle.)158 1067 y(The)18 b(curren)o(t)g(prop)q(osal)h(do)q(es)g(not)f(allo)o(w)h(to)e (create)h(a)g(bu\013er)g(descriptor,)i(use)e(it)h(in)f(a)g(comm)o(unication)i (after)75 1113 y(committed,)14 b(then)g(mo)q(dify)g(it)g(partially)i(and)e (use)g(it)g(for)f(another)h(comm)o(unication)i({)d(a)h(new)f(bu\013er)h (descriptor)h(has)f(to)75 1159 y(b)q(e)h(created.)21 b(Of)13 b(course,)i(the)g(same)f(bu\013er)h(descriptor)h(can)f(b)q(e)f(reused)h(man)o (y)g(times)g(in)g(successiv)o(e)h(comm)o(unications)75 1204 y(that)d(send)g(data)g(from)f(the)g(same)h(bu\013ers;)g(e.g.,)f(a)g(new)h (send)g(with)g(a)f(bu\013er)i(descriptor)g(handle)g(can)f(b)q(e)f(initiated)j (once)75 1250 y(the)e(previous)i(send)f(with)f(this)h(handle)h(has)e (completed.)158 1382 y Fo(Example)f(1:)75 1432 y(Consider)i(the)h(follo)o (wing)c(fragmen)o(t)h(of)i(F)m(ortran)f(co)q(de)75 1513 y Fi(DOUBLE)21 b(PRECISION)e(A\(10,20\))75 1563 y(INTEGER)h(B,)i(C\(5,10\))75 1613 y(INTEGER)e(BH)75 1663 y(...)75 1713 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,) d(MPI_PERSISTENT\))75 1763 y(CALL)j(MPI_APPEND_BLOCK)d(\(BH,)j(B,)h(1,)f (MPI_INT\))75 1812 y(CALL)g(MPI_APPEND_VEC)e(\(BH,)i(A\(1,3\),)f(11,)h(4,)h (2,)f(MPI_DOUBLE\))75 1862 y(CALL)g(MPI_APPEND_INDEX\()o(BH,)e(C\(3,7\),)h (\(4,2,1\),)g(MPI_INT\))158 1943 y Fo(Then)14 b(the)h(bu\013er)g(asso)q (ciated)f(with)g(the)g(handle)g Fi(BH)g Fo(consists)h(of)e(the)h(sequence)i (of)e(v)n(ariables)158 1993 y Fi(B,)21 b(A\(1,3\),)g(A\(2,3\),)f(A\(5,3\),)g (A\(6,3\),)h(A\(9,3\),)f(A\(10,3\),)g(A\(3,4\),)h(A\(4,4\),)f(A\(7,4\),)g (A\(8,4\),)75 2043 y(A\(1,5\),)g(C\(2,8\),)h(C\(5,7\),)f(C\(4,7\))p Fo(.)158 2093 y(A)14 b(message)f(created)j(from)c(this)i(bu\013er)h(will)d (consist)j(of)e(a)g(sequence)j(of)e(one)g(in)o(teger,)f(follo)o(w)o(ed)g(b)o (y)g(elev)o(en)75 2143 y(double)h(precision)g(reals,)g(follo)o(w)o(ed)e(b)o (y)i(three)h(in)o(tegers.)158 2192 y(Example)d(2:)75 2242 y(Supp)q(ose)j(one) f(has)g(an)g(arra)o(y)f(of)g(data)h(declared)h(in)e(C)h(as)75 2323 y Fi(struct)21 b({)140 2373 y(int)h(type;)140 2423 y(double)f (position[3];)140 2473 y(double)g(velocity[3];)228 2523 y(})g(particle[1000]) 158 2604 y Fo(and)d(one)g(w)o(an)o(ts)h(to)f(transfer)h(in)f(one)g(message)g (the)h(n)o(um)o(b)q(er)e(of)h(particles)g(of)g(t)o(yp)q(e)h(1,)f(the)h(n)o (um)o(b)q(er)f(of)75 2654 y(particles)g(of)g(t)o(yp)q(e)g(2,)g(and)g(the)g(p) q(ositions)g(of)f(all)g(the)h(particles)h(of)e(these)i(t)o(w)o(o)f(t)o(yp)q (es.)31 b(The)18 b(follo)o(wing)d(co)q(de)75 2704 y(fragmen)o(t)d(creates)k (a)e(suitable)f(bu\013er)i(descriptor.)p eop %%Page: 11 12 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1070 b Fo(11)75 45 y Fi(int)21 b(count1)g(=)g(0;)75 95 y(int)g(count2)g(=)g(0;)75 145 y(...)75 195 y(mpi_create_buffer)o(\()e(&handle,)h(MPI_PERSISTENT\);)75 244 y(mpi_append_contig)o(uous\()e(&handle,)i(&count1,)h(MPI_INT\);)75 294 y(mpi_append_contig)o(uous\()d(&handle,)i(&count2,)h(MPI_INT\);)75 344 y(for)g(\(i=0;)g(i<1000;)f(i++\))h({)140 394 y(if)h(\(particle[i].typ)o (e)d(==)i(1\))h({)206 444 y(count1++;)206 493 y(mpi_append_conti)o(guous)o (\()d(&handle,)h(&particle[i].posit)o(ion,)e(3,)k(MPI_DOUBLE\);)206 543 y(})140 593 y(elseif)f(\(particle[i].type)d(==)k(2\))f({)206 643 y(count2++;)206 693 y(mpi_append_conti)o(guous)o(\()e(&handle,)h (&particle[i].posit)o(ion,)e(3,)k(MPI_DOUBLE\);)206 742 y(})140 792 y(})75 842 y(mpi_commit_buffer)o(\(hand)o(le\))75 892 y(...)158 980 y Fo(The)14 b(bu\013er)h(descriptor)f(handle)g(can)g(no)o(w)f(b)q(e)h (used)g(to)g(send)g(\(rep)q(eatedly\))h(the)f(particle)g(coun)o(ts)g(for)f(t) o(yp)q(e)75 1030 y(1)g(and)f(2,)h(and)g(the)g(p)q(ositions)g(of)f(the)i (particles)f(with)g(these)h(t)o(w)o(o)f(t)o(yp)q(es,)g(as)g(long)f(as)h(the)h (t)o(yp)q(es)g(of)e(the)h(particles)75 1079 y(is)h(not)g(mo)q(di\014ed.)158 1129 y(Supp)q(ose)i(the)f(incoming)d(data)i(is)h(to)f(b)q(e)h(stored)h(at)e (the)h(receiv)o(er)h(in)o(to)e(the)h(t)o(w)o(o)g(coun)o(t)f(v)n(ariables)g Fi(count1)75 1179 y Fo(and)h Fi(count2)f Fo(and)h(in)o(to)f(the)i Fi(position)d Fo(comp)q(onen)o(ts)i(of)f(the)i(initial)d(en)o(tries)j(of)f (the)g(arra)o(y)g(of)f(particles.)23 b(The)75 1229 y(follo)o(wing)10 b(co)q(de)i(generates)i(a)e(suitable)g(bu\013er)h(descriptor)h(for)d(the)i (receiv)o(e)g(op)q(eration)f(\(w)o(e)h(assume)f(4)f(b)o(ytes)i(for)75 1279 y(in)o(teger)h(and)g(8)g(b)o(ytes)g(for)g(double\).)75 1367 y Fi(...)75 1416 y(mpi_create_buffer)o(\()19 b(&handle,)h (MPI_PERSISTENT\);)75 1466 y(mpi_append_contig)o(uous\()e(&handle,)i (&count1,)h(MPI_INT\);)75 1516 y(mpi_append_contig)o(uous\()d(&handle,)i (&count2,)h(MPI_INT\);)75 1566 y(mpi_append_hvec\()d(&handle,)j (&particle[0].pos)o(itio)o(n,)e(1000,)i(52,)g(3,)g(MPI_DOUBLE\);)75 1616 y(mpi_commit_buffer)o(\()e(&handle\))75 1665 y(...)158 1753 y Fo(A)c(bu\013er)g(handle)f(can)h(b)q(e)g(used)g(for)f(comm)o (unication,)d(ev)o(en)k(if)f(it)g(is)g(not)g(asso)q(ciated)i(with)e(an)o(y)g (v)n(ariables)75 1803 y(\(i.e.,)e(ev)o(en)i(if)e(it)h(w)o(as)g(not)g(set)h(b) o(y)f(an)o(y)g Fi(MPI)p 761 1803 14 2 v 15 w(APPEND)p 908 1803 V 15 w(xxx)f Fo(call\).)17 b(Suc)o(h)d(a)f(handle)g(is)g(asso)q(ciated)h (with)f(an)g(empt)o(y)75 1853 y(bu\013er,)i(and)f(a)g(message)g(created)i (from)c(it)i(con)o(tains)g(no)g(data.)19 b(The)c(handle)f(still)f(need)i(b)q (e)g(committed)d(b)q(efore)75 1903 y(it)i(is)f(used.)158 1953 y(The)h(same)e(en)o(try)i(ma)o(y)d(app)q(ear)i(more)f(than)i(once)f(in)g(a)g (bu\013er)h(comp)q(onen)o(t,)e(and)h(distinct)g(bu\013er)h(comp)q(o-)75 2002 y(nen)o(ts)i(ma)o(y)d(o)o(v)o(erlap.)21 b(If)15 b(a)g(bu\013er)h (descriptor)g(with)f(replicated)h(en)o(tries)g(is)f(used)h(to)f(receiv)o(e)h (a)f(message,)f(then)75 2052 y(the)g(\014nal)g(v)n(alue)f(of)g(an)h(en)o(try) g(that)g(o)q(ccurs)i(more)d(than)g(once)i(in)e(the)i(bu\013er)g(descriptor)g (is)f(unde\014ned.)158 2180 y Fm(Discussion:)158 2226 y Fl(It)f(is)g(not)h (clear)g(to)f(me)g(that)g(o)o(v)o(erlaps)i(ha)o(v)o(e)e(an)o(y)h(useful)g (purp)q(ose,)g(and)g(they)g(ma)o(y)f(complicate)i(implemen)o(tation.)75 2272 y(But)e(w)o(e)g(had)g(an)h(o)o(v)o(erwhelming)h(v)o(ote)e(in)h(fa)o(v)o (or)f(of)g(this)h(feature.)75 2470 y Fk(1.5.3)55 b(T)n(yp)r(e)18 b(Matc)n(hing)i(and)f(Data)g(Con)n(v)n(ersion)75 2547 y Fh(T)o(yp)q(e)d(matc) o(hing)75 2623 y Fo(One)f(can)f(think)f(of)h(message)f(transmission)g(as)h (consisting)g(of)f(three)i(phases:)126 2704 y(1.)20 b(Data)13 b(is)h(pulled)f(out)h(of)f(the)i(send)g(bu\013er)f(and)g(a)g(message)f(is)h (assem)o(bled)p eop %%Page: 12 13 bop 75 -100 a Fo(12)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)126 45 y Fo(2.)20 b(A)14 b(message)f(is)h (transferred)i(from)c(sender)j(to)f(receiv)o(er)126 127 y(3.)20 b(Data)13 b(is)h(pulled)f(from)f(the)j(incoming)c(message)j(and)g(disassem)o (bled)f(in)o(to)g(the)i(receiv)o(e)g(bu\013er)158 209 y(T)o(yp)q(e)c(matc)o (hing)e(has)i(to)f(b)q(e)h(observ)o(ed)h(at)f(eac)o(h)g(of)f(these)i(three)g (phases:)17 b(The)11 b(t)o(yp)q(e)g(of)f(eac)o(h)i(v)n(ariable)d(in)h(the)75 259 y(sender)15 b(bu\013er)g(has)f(to)g(matc)o(h)f(the)h(t)o(yp)q(e)g(of)g (the)g(corresp)q(onding)h(en)o(try)f(in)g(the)g(bu\013er)h(descriptor;)g(the) f(t)o(yp)q(e)h(of)75 309 y(an)e(en)o(try)g(in)f(the)h(bu\013er)h(descriptor)g (used)g(to)e(send)i(a)f(message)f(has)h(to)f(matc)o(h)g(the)h(t)o(yp)q(e)h (of)e(the)h(corresp)q(onding)75 358 y(en)o(try)f(in)e(the)i(bu\013er)g (descriptor)g(used)g(to)f(receiv)o(e)h(the)g(message;)f(and)g(the)h(t)o(yp)q (e)f(of)g(eac)o(h)g(v)n(ariable)f(in)h(the)h(receiv)o(e)75 408 y(bu\013er)i(has)f(to)g(matc)o(h)e(the)j(corresp)q(onding)f(en)o(try)h (in)e(the)i(bu\013er)g(descriptor)g(for)e(this)h(bu\013er.)19 b(A)13 b(program)e(that)75 458 y(fails)i(to)g(observ)o(e)i(these)h(three)f (rules)f(is)g(erroneous.)158 508 y(T)m(o)g(de\014ne)h(t)o(yp)q(e)g(matc)o (hing)d(more)h(precisely)m(,)i(w)o(e)f(need)h(to)g(deal)f(with)g(t)o(w)o(o)f (issues:)20 b(matc)o(hing)13 b(of)g(t)o(yp)q(es)j(of)75 558 y(the)h(host)g(language)f(with)g(bu\013er)h(descriptor)h(t)o(yp)q(es;)g(and)f (matc)o(hing)d(of)i(bu\013er)i(descriptor)g(t)o(yp)q(es)f(at)f(sender)75 607 y(and)c(receiv)o(er.)18 b(W)m(e)12 b(restrict)h(ourselv)o(es)g(to)f(the)g (case)h(where)g(the)f(sending)g(and)g(receiving)g(programs)e(are)i(written)75 657 y(in)h(the)i(same)e(language.)158 707 y(The)21 b(t)o(yp)q(e)h(of)e(an)g (en)o(try)i(in)e(the)h(sender)i(bu\013er)f(descriptor)g(matc)o(hes)e(the)i(t) o(yp)q(e)f(of)f(an)h(en)o(try)g(in)f(the)75 757 y(receiv)o(er)13 b(bu\013er)g(descriptor)g(if)e(they)h(ha)o(v)o(e)g(iden)o(tical)f(names:)16 b Fi(MPI)p 1131 757 14 2 v 15 w(INT)11 b Fo(matc)o(hes)h Fi(MPI)p 1448 757 V 15 w(INT)p Fo(,)e Fi(MPI)p 1617 757 V 15 w(REAL)h Fo(matc)o(hes)75 807 y Fi(MPI)p 144 807 V 15 w(REAL)p Fo(,)i(and)g(so)h(on.) 158 857 y(The)i(t)o(yp)q(e)g(of)e(a)h(v)n(ariable)g(in)g(a)g(host)g(program)f (matc)o(hes)h(the)h(t)o(yp)q(e)g(of)e(the)i(en)o(try)g(in)f(a)g(bu\013er)i (descriptor)75 906 y(if)e(the)i(name)e(of)g(the)i(bu\013er)g(descriptor)g(en) o(try)g(t)o(yp)q(e)g(corresp)q(onds)h(to)e(the)g(basic)g(t)o(yp)q(e)h(of)f (the)g(host)h(program)75 956 y(v)n(ariable:)k(an)15 b(en)o(try)i(with)e(t)o (yp)q(e)h(name)f Fi(MPI)p 794 956 V 15 w(INT)g Fo(matc)o(hes)h(a)f(F)m (ortran)h(v)n(ariable)e(of)i(t)o(yp)q(e)g Fi(INTEGER)p Fo(,)e(an)h(en)o(try) 75 1006 y(with)10 b(t)o(yp)q(e)i(name)d Fi(MPI)p 433 1006 V 15 w(REAL)h Fo(matc)o(hes)h(a)f(F)m(ortran)h(v)n(ariable)e(of)i(t)o(yp)q(e)g Fi(REAL)p Fo(,)e(and)i(so)g(on.)17 b(There)12 b(is)e(one)h(exception)75 1056 y(to)h(this)h(last)f(rule:)18 b(An)13 b(en)o(try)g(with)f(t)o(yp)q(e)h (name)e Fi(MPI)p 917 1056 V 15 w(BYTE)h Fo(can)h(b)q(e)g(used)g(to)g(matc)o (h)e(an)o(y)h(b)o(yte)h(of)f(storage)h(\(on)f(a)75 1106 y(b)o (yte-addressable)j(mac)o(hine\),)d(irresp)q(ectiv)o(e)j(of)e(the)g(datat)o (yp)q(e)h(of)f(the)h(v)n(ariable)e(that)i(con)o(tains)f(this)g(b)o(yte.)19 b(The)75 1155 y(v)n(alue)13 b(of)h(the)g(message)g(en)o(try)g(will)f(b)q(e)h (the)g(binary)g(v)n(alue)f(of)g(the)i(corresp)q(onding)g(b)o(yte)f(in)f (memory)m(.)158 1205 y(W)m(e)h(th)o(us)g(ha)o(v)o(e)g(t)o(w)o(o)f(cases:)137 1287 y Fe(\017)21 b Fo(Comm)o(uni)o(cation)6 b(of)j(t)o(yp)q(ed)h(v)n(alues)f (\(e.g.,)g(with)g(datat)o(yp)q(e)g(di\013eren)o(t)h(from)e Fi(MPI)p 1430 1287 V 15 w(BYTE)p Fo(,)g(where)i(the)g(datat)o(yp)q(es)179 1337 y(of)k(the)h(corresp)q(onding)g(en)o(tries)g(in)f(the)h(sender)h (program,)d(in)h(the)h(sender)h(bu\013er)f(descriptor,)h(in)d(the)i(re-)179 1386 y(ceiv)o(er)g(bu\013er)f(descriptor)i(and)d(in)h(the)g(receiv)o(er)i (program)c(should)i(all)e(matc)o(h.)137 1469 y Fe(\017)21 b Fo(Comm)o(uni)o(cation)13 b(of)j(un)o(t)o(yp)q(ed)h(v)n(alues)f(\(e.g.,)g(of) g(datat)o(yp)q(e)g Fi(MPI)p 1224 1469 V 15 w(BYTE)p Fo(,)f(where)j(the)f(the) g(corresp)q(onding)179 1518 y(en)o(tries)e(in)e(the)i(sender)g(and)f(the)g (receiv)o(er)h(bu\013er)g(descriptors)h(ha)o(v)o(e)d(b)q(oth)h(datat)o(yp)q (e)g Fi(MPI)p 1631 1518 V 16 w(BYTE)p Fo(.)e(In)i(this)179 1568 y(case,)k(there)g(are)f(no)g(requiremen)o(ts)g(on)g(the)g(datat)o(yp)q (es)h(of)e(the)i(corresp)q(onding)f(en)o(tries)h(in)f(the)g(sender)179 1618 y(and)c(the)i(receiv)o(er)g(programs,)e(nor)h(is)f(it)h(required)h(that) f(they)g(b)q(e)g(the)h(same.)75 1734 y Fk(1.5.4)55 b(Data)19 b(con)n(v)n(ersion)75 1811 y Fo(W)m(e)13 b(use)i(the)g(follo)o(wing)c (terminology:)75 1892 y Fh(t)o(yp)q(e)k(con)o(v)o(ersion)j Fo(c)o(hanges)d(the)f(datat)o(yp)q(e)g(of)f(a)h(v)n(alue,)f(e.g.)18 b(b)o(y)13 b(rounding)h(a)f Fi(REAL)g Fo(to)h(an)g Fi(INTEGER)p Fo(.)75 1974 y Fh(represen)o(tati)o(on)e(con)o(v)o(ersion)18 b Fo(c)o(hanges)e(the)g(binary)f(represen)o(tation)i(of)e(a)g(v)n(alue,)f (e.g.)23 b(from)13 b(Hex)j(\015oating)179 2024 y(p)q(oin)o(t)d(to)h(IEEE)g (\015oating)f(p)q(oin)o(t.)158 2106 y(Represen)o(tation)20 b(con)o(v)o(ersion)f(is)g(needed)h(when)g(data)e(is)h(mo)o(v)o(ed)e(across)j (mac)o(hines)e(that)h(use)h(di\013eren)o(t)75 2156 y(binary)9 b(enco)q(dings)h(for)f(the)h(same)f(datat)o(yp)q(e.)16 b(Represen)o(tation)11 b(con)o(v)o(ersion)e(preserv)o(es,)k(to)c(the)h(greatest)h(p)q(ossible)75 2205 y(extend,)h(the)e Fh(v)m(alue)g Fo(represen)o(ted.)20 b(Ho)o(w)o(ev)o(er,)11 b(rounding)e(errors)j(and)e(o)o(v)o(er\015o)o(w)g(and) g(under\015o)o(w)h(exceptions)g(ma)o(y)75 2255 y(o)q(ccur)17 b(during)e(\015oating)f(p)q(oin)o(t)h(con)o(v)o(ersions;)h(con)o(v)o(ersion)g (of)f(in)o(tegers)h(ma)o(y)e(also)g(lead)h(to)h(exceptions)g(when)g(a)75 2305 y(v)n(alue)d(that)h(can)g(represen)o(ted)j(in)c(one)h(system)g(cannot)g (b)q(e)h(represen)o(ted)h(in)e(the)g(other)h(system.)i(MPI)d(do)q(es)h(not)75 2355 y(sp)q(ecify)f(rules)h(for)e(represen)o(tation)j(con)o(v)o(ersion.)158 2405 y(The)j(t)o(yp)q(e)f(matc)o(hing)e(rules)j(imply)d(that)i(MPI)g(comm)o (unication)d(nev)o(er)k(en)o(tails)e(t)o(yp)q(e)i(con)o(v)o(ersion.)31 b(On)75 2455 y(the)15 b(other)g(hand,)f(MPI)g(requires)i(that)e(a)g(represen) o(tation)i(con)o(v)o(ersion)e(b)q(e)h(p)q(erformed)f(when)h(a)f(t)o(yp)q(ed)g (v)n(alue)g(is)75 2504 y(transferred)g(across)g(en)o(vironmen)o(ts)d(that)i (use)g(di\013eren)o(t)g(represen)o(tations)i(for)d(the)h(datat)o(yp)q(e)g(of) e(this)i(v)n(alue.)k(An)75 2554 y(exception)f(o)q(ccurring)g(during)f (represen)o(tation)i(con)o(v)o(ersion)f(results)g(in)f(a)h(failure)e(of)h (the)h(comm)o(unicatio)o(n;)d(an)75 2604 y(error)i(o)q(ccurs)g(either)g(in)e (the)i(send)g(op)q(eration,)e(or)h(the)g(receiv)o(e)h(op)q(eration,)f(or)g(b) q(oth.)158 2654 y(If)g(a)f(v)n(alue)h(sen)o(t)h(in)e(a)h(message)g(is)f(un)o (t)o(yp)q(ed)i(\(i.e.,)e(of)g(t)o(yp)q(e)i Fi(MPI)p 1180 2654 V 15 w(BYTE)p Fo(\),)e(then)h(the)h(binary)e(represen)o(tation)75 2704 y(of)h(the)g(b)o(yte)h(stored)g(at)f(the)h(receiv)o(er)h(is)e(iden)o (tical)f(to)h(the)h(binary)f(represen)o(tation)i(of)d(the)i(b)o(yte)g(loaded) e(at)h(the)p eop %%Page: 13 14 bop 75 -100 a Fg(1.5.)31 b(COMMUNICA)m(TION)14 b(BUFFERS)1070 b Fo(13)75 45 y(sender.)24 b(This)16 b(holds)f(true,)h(whether)h(sender)g (and)e(receiv)o(er)i(run)f(in)f(the)h(same)f(or)g(in)g(distinct)h(en)o (vironmen)o(ts;)75 95 y(no)e(represen)o(tation)h(con)o(v)o(ersion)f(is)g (required.)158 145 y(Note)f(that)g(no)g(con)o(v)o(ersions)g(ev)o(er)h(o)q (ccur)g(when)g(an)e(MPI)h(program)f(executes)j(in)d(a)h(homogeneous)f (system,)75 195 y(where)g(all)e(pro)q(cesses)j(run)e(in)g(the)g(same)f(en)o (vironmen)o(t.)16 b(Also)11 b(note)g(the)g(di\013eren)o(t)h(b)q(eha)o(vior)f (of)f Fi(MPI)p 1654 195 14 2 v 15 w(BYTE)g Fo(and)h(of)75 244 y Fi(MPI)p 144 244 V 15 w(CHAR)p Fo(.)g(A)h(bu\013er)h(descriptor)h(en)o(try) f(with)e(datat)o(yp)q(e)i(of)e Fi(MPI)p 1106 244 V 16 w(CHAR)g Fo(can)h(only)g(matc)o(h)f(a)g(F)m(ortran)h(v)n(ariable)g(of)75 294 y(t)o(yp)q(e)i Fi(CHAR)p Fo(;)e(and)h(represen)o(tation)i(con)o(v)o (ersion)f(ma)o(y)d(o)q(ccur)j(when)g(v)n(alues)f(of)g(t)o(yp)q(e)h Fi(MPI)p 1473 294 V 15 w(CHAR)e Fo(are)i(transferred.,)75 344 y(e.g.,)f(from)f(an)h(EBCDIC)i(enco)q(ding)f(to)f(an)h(ASCI)q(I)g(enco)q (ding.)158 394 y(Consider)g(the)h(follo)o(wing)c(examples.)75 481 y Fi(!)43 b(FIRST)21 b(EXAMPLE)75 531 y(INTEGER)f(A\(100\))75 581 y(...)75 630 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 680 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h(MPI_CHAR\))158 767 y Fo(This)15 b(program)f(is)i(erroneous,)g(since)h(the)f(t)o(yp)q(e)g(of) e(the)j(bu\013er)f(descriptor)h(do)q(es)f(not)f(matc)o(h)g(the)h(t)o(yp)q(e)g (of)75 817 y(the)g(v)n(ariable)f(in)g(the)i(bu\013er.)25 b(The)16 b(b)q(eha)o(vior)f(or)h(this)g(program)e(is)i(unde\014ned.)25 b(A)16 b(desirable)g(b)q(eha)o(vior)g(is)f(for)75 867 y(that)f(error)h(to)e (b)q(e)i(detected)h(\(at)e(compile)e(time)h(or)h(run)g(time\))e(and)i(result) h(in)e(an)h(error)h(condition.)75 954 y Fi(!)43 b(SECOND)21 b(EXAMPLE)75 1004 y(INTEGER)f(A\(100\))75 1054 y(...)75 1104 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH,)d(MPI_PERSISTENT\))75 1153 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH,)j(A\(1\),)g(1,)h(MPI_BYTE\))158 1241 y Fo(This)11 b(program)f(is)h(correct.)19 b(The)12 b(b)o(yte)g(that)f (will)f(b)q(e)i(transmitted)f(in)g(a)g(message)g(created)i(with)e(this)h (bu\013er)75 1290 y(descriptor)h(is)f(the)g(\014rst)h(b)o(yte)f(in)g(the)g (storage)h(o)q(ccupied)f(b)o(y)g Fi(A\(1\))p Fo(.)17 b(The)12 b(v)n(alue)f(of)h(this)f(b)o(yte)i(is)e(implemen)o(tation)75 1340 y(dep)q(enden)o(t,)17 b(and)e(will)f(v)n(ary)h(according)g(to)g(the)h(n) o(um)o(b)q(er)f(of)f(b)o(ytes)i(used)h(to)e(store)h(F)m(ortran)f(v)n (ariables)g(of)f(t)o(yp)q(e)75 1390 y Fi(INTEGER)e Fo(and)i(the)h(b)o(yte)f (ordering)g(on)f(the)i(host)f(mac)o(hine.)75 1477 y Fi(!)43 b(THIRD)21 b(EXAMPLE)75 1527 y(INTEGER)f(A\(100\))75 1577 y(...)75 1627 y(CALL)h(MPI_CREATE_BUFFER)o(\(BH1,)d(MPI_PERSISTENT\))75 1677 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH1,)j(A\(1\),)g(1,)g (MPI_INTEGER\))75 1726 y(...)75 1776 y(CALL)g(MPI_CREATE_BUFFER)o(\(BH2,)d (MPI_PERSISTENT\))75 1826 y(CALL)j(MPI_APPEND_CONTIG)o(UOUS)d(\(BH2,)j (A\(1\),)g(4,)g(MPI_BYTE\))158 1913 y Fo(Consider)14 b(the)h(follo)o(wing)c (three)k(cases:)126 1993 y(1.)20 b(A)10 b(message)g(is)g(sen)o(t)h(using)e (bu\013er)j(descriptor)f(handle)f Fi(BH1)f Fo(and)h(receiv)o(ed)i(using)e (bu\013er)h(descriptor)g(handle)179 2043 y Fi(BH2)p Fo(.)19 b(The)d(program)d(is)h(erroneous,)i(since)f(the)h(sender)g(and)e(receiv)o(er) i(bu\013er)g(descriptors)g(don't)f(matc)o(h.)179 2093 y(The)g(b)q(eha)o(vior) f(of)g(the)h(program)d(is)j(unde\014ned,)g(and)f(a)g(p)q(ossible)h (\(desirable)g(but)g(unlik)o(ely\))e(b)q(eha)o(vior)h(is)179 2142 y(for)f(an)h(error)h(condition)e(to)h(o)q(ccur.)126 2224 y(2.)20 b(Both)d(sender)h(and)f(receiv)o(er)i(use)e(bu\013er)h(descriptor)g (handle)f Fi(BH1)p Fo(.)26 b(The)18 b(program)d(is)i(correct,)h(and)f(an)179 2274 y(in)o(teger)12 b(v)n(alue)f(is)g(transmitted)g(from)f(sender)j(to)f (receiv)o(er.)19 b(The)12 b(v)n(alue)f(assigned)h(to)f Fi(A\(1\))g Fo(at)g(the)h(receiv)o(er)179 2323 y(is)h(iden)o(tical)f(to)h(the)g(v)n(alue) g(of)f Fi(A\(1\))g Fo(at)h(the)h(sender,)g(but)f(the)h(binary)f(represen)o (tation)h(of)e(this)i(v)n(alue)e(ma)o(y)179 2373 y(b)q(e)i(di\013eren)o(t,)h (if)e(sender)i(and)f(receiv)o(er)h(run)g(in)e(di\013eren)o(t)i(en)o(vironmen) o(ts.)126 2455 y(3.)20 b(Both)c(sender)h(and)e(receiv)o(er)i(use)g(bu\013er)f (descriptor)h(handle)f Fi(BH2)p Fo(.)22 b(The)16 b(program)e(is)h(correct.)25 b(Assume)179 2504 y(that)14 b(b)q(oth)h(sender)h(and)e(receiv)o(er)i(use)f (four)f(b)o(ytes)h(to)f(store)i(in)o(tegers.)k(Then)15 b(the)g(v)n(alue)f (stored)h(in)f Fi(A\(1\))179 2554 y Fo(at)i(the)h(receiv)o(er)h(has)e(the)h (same)f(binary)g(represen)o(tation)i(as)e(the)h(v)n(alue)f(of)f Fi(A\(1)h Fo(at)g(the)h(sender.)27 b(If)16 b(the)179 2604 y(same)c(b)o(yte)h (ordering)g(and)g(the)h(same)e(enco)q(ding)h(of)g(in)o(teger)g(v)n(alues)g (is)g(used)g(at)g(b)q(oth)g(ends)h(then)g Fi(A\(1\))e Fo(at)179 2654 y(the)j(receiv)o(er)h(is)f(set)g(to)g(the)g(v)n(alue)f(of)g Fi(A\(1\))g Fo(at)h(the)g(sender;)h(otherwise)g(the)f(same)f(32)g(bits)h(ma)o (y)e(enco)q(de)179 2704 y(di\013eren)o(t)i(v)n(alues.)p eop %%Page: 14 15 bop 75 -100 a Fo(14)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)158 45 y Fm(Implemen)o(tation)f(note:)158 94 y Fl(The)d(curren)o(t)g(de\014nition)j(do)q(es)d(not)h(require)g(messages) g(to)e(carry)i(t)o(yp)q(e)f(information.)18 b(A)9 b(message)i(can)f(b)q(e)h (comp)q(osed)75 140 y(and)g(sen)o(t)f(using)i(only)f(the)f(information)i(in)f (the)g(sender)g(bu\013er)g(descriptor,)h(and)e(can)h(b)q(e)f(receiv)o(ed)i (and)e(stored)h(using)h(only)75 186 y(the)i(information)i(in)e(the)g(receiv)o (er)h(bu\013er)g(descriptor.)21 b(If)13 b(messages)h(are)g(sen)o(t)g(b)q(et)o (w)o(een)g(di\013eren)o(t)i(mac)o(hines)f(then)f(one)75 231 y(can)e(either)h(use)g(a)f(\\univ)o(ersal")j(data)d(enco)q(ding)i(for)e (messages,)h(use)f(kno)o(wledge)i(of)d(the)i(receiv)o(er)g(en)o(vironmen)o(t) h(in)e(order)75 277 y(to)i(con)o(v)o(ert)h(data)f(at)g(the)h(sender,)f(or)h (use)f(kno)o(wledge)i(of)e(the)g(sender)h(en)o(vironmen)o(t)h(in)f(order)f (to)g(con)o(v)o(ert)h(data)g(at)f(the)75 323 y(receiv)o(er.)k(In)c(either)g (case)f(the)h(lo)q(cal)g(bu\013er)g(descriptor)h(can)f(b)q(e)f(used)h(to)f (deriv)o(e)i(the)e(t)o(yp)q(es)h(of)f(the)g(v)n(alues)h(transferred.)75 368 y(Ho)o(w)o(ev)o(er,)f(additional)k(t)o(yp)q(e)c(information)j(carried)e (b)o(y)g(messages)g(can)g(pro)o(vide)h(b)q(etter)f(error)f(detection.)19 b(It)13 b(is)h(p)q(ossible)75 414 y(to)d(add)g(alw)o(a)o(ys)h(suc)o(h)f (additional)j(t)o(yp)q(e)d(information,)i(for)e(b)q(etter)g(error)g(c)o(hec)o (king,)h(or)f(add)h(it)f(when)g(running)i(in)e(\\debug")75 460 y(mo)q(de,)i(where)g(some)g(p)q(erformance)h(is)g(sacri\014ced)h(for)d (added)i(error)g(c)o(hec)o(king,)g(etc.)158 675 y Fm(Discussion:)158 724 y Fl(There)h(is)g(no)g(agreemen)o(t)h(y)o(et)f(on)g(the)g(need)g(for)g(a) g(de\014nition)i(and,)f(if)f(needed,)h(on)f(the)g(righ)o(t)g(de\014nition)j (of)c(t)o(yp)q(e)75 770 y(matc)o(hing)k(and)e(con)o(v)o(ersion)j(b)q(et)o(w)o (een)d(programs)h(written)g(in)g(di\013eren)o(t)g(languages.)29 b(The)16 b(natural)h(in)o(terpretation)i(of)75 815 y(\\send)11 b(from)e(C)g(and)i(receiv)o(e)f(in)h(F)m(ortran")f(is)g(either)h(\\send)g (from)e(C,)g(receiv)o(e)i(in)f(C)f(and)i(con)o(v)o(ert)f(to)g(F)m(ortran",)g (or)g(\\con)o(v)o(ert)75 861 y(to)i(F)m(ortran,)h(send)g(from)f(F)m(ortran)g (and)h(receiv)o(e)g(in)g(F)m(ortran".)18 b(Ho)o(w)o(ev)o(er,)11 b(the)i(corresp)q(ondence)h(b)q(et)o(w)o(een)f(F)m(ortran)f(t)o(yp)q(es)75 907 y(and)g(C)f(t)o(yp)q(es)g(is)h(implemen)o(tation)j(dep)q(enden)o(t.)j(Th) o(us,)11 b(the)h(diagram)g(sho)o(wn)g(b)q(elo)o(w)g(is)g(not)f(comm)o(utativ) o(e,)i(and)f(the)f(t)o(w)o(o)75 952 y(in)o(terpretations)16 b(of)c(C)h(to)g(F)m(ortran)g(comm)o(unication)j(ma)o(y)d(lead)h(to)f (di\013eren)o(t)i(results.)238 999 y(mac)o(hine)g(1)285 b(mac)o(hine)15 b(2)513 1045 y(send)232 1091 y(C)e(program)118 b Fc(!)g Fl(C)13 b(program)313 1182 y Fc(#)436 b(#)513 1228 y Fl(send)183 1273 y(F)m(ortran)13 b(program)69 b Fc(!)g Fl(F)m(ortran)13 b(program)158 1321 y(A)h(go)q(o)q(d)h(solution)h(w)o(ould)f(mak)o(e)f(this)h(diagram)h (comm)o(utativ)o(e.)21 b(Otherwise,)15 b(one)g(needs)g(to)f(pic)o(k)h(one)f (of)g(the)g(t)o(w)o(o)75 1366 y(paths)k(as)f(the)g(canonical)i(de\014nition)h (of)d(in)o(terlanguage)j(comm)o(unication.)31 b(If)16 b(the)h(upp)q(er)h (path)g(is)f(c)o(hosen)h(\(message)75 1412 y(carries)c(sender)g(datat)o(yp)q (es\),)h(then)e(the)h(receiv)o(er)g(of)f(a)g(message)h(needs)g(to)g(kno)o(w)f (the)h(language)h(the)f(message)f(w)o(as)h(sen)o(t)75 1458 y(from;)h(if)g(the)f(lo)o(w)o(er)h(path)h(is)f(c)o(hosen,)g(\(message)h (carries)f(receiv)o(er)h(datat)o(yp)q(es\),)g(then)f(the)f(sender)i(needs)f (to)g(kno)o(w)g(the)75 1503 y(language)g(the)e(message)h(is)f(receiv)o(ed)i (in)o(to.)158 1553 y(A)c(p)q(ossible)i(solution)h(\(prop)q(osed)e(b)o(y)g (Dann)o(y)g(Nessett\))f(is)h(to)f(lea)o(v)o(e)h(to)f(the)g(user)h(the)f(c)o (hoice)h(of)f(sp)q(ecifying)i(whether)75 1599 y(to)d(send)g(\\F)m(ortran)g (datat)o(yp)q(es")i(or)d(\\C)h(datat)o(yp)q(es",)h(in)g(the)f(ab)q(o)o(v)o(e) g(situation.)18 b(One)10 b(w)o(ould)h(still)g(require)g(exact)f(matc)o(hing) 75 1644 y(of)16 b(send)h(bu\013er)g(descriptors)h(and)f(receiv)o(e)g (bu\013er)g(descriptors.)28 b(Ho)o(w)o(ev)o(er,)17 b(a)f(F)m(ortran)h (program)g(could)g(use)g(a)f(bu\013er)75 1690 y(descriptor)f(with)e(\\C)g (datat)o(yp)q(es",)h(and)g(vice)g(v)o(ersa.)j(The)c(standard)h(w)o(ould)g(sp) q(ecify)h(whic)o(h)e(matc)o(hing)i(of)e(F)m(ortran)g(and)75 1735 y(C)i(datat)o(yp)q(es)j(are)e(legal.)27 b(F)m(or)16 b(example,)i(it)e(w) o(ould)h(b)q(e)f(correct)g(to)g(send)h(a)f(F)m(ortran)g(v)n(ariable)i(of)e(t) o(yp)q(e)g Ff(REAL)e Fl(using)k(a)75 1781 y(bu\013er)c(descriptor)g(of)f(t)o (yp)q(e)g Ff(MPI)p 556 1781 12 2 v 13 w(FLOAT)p Fl(.)d(Data)j(con)o(v)o (ersion)i(migh)o(t)f(b)q(e)f(required)h(if)f(C)f(\015oat)i(n)o(um)o(b)q(ers)f (and)h(F)m(ortran)f(real)75 1827 y(n)o(um)o(b)q(ers)h(do)f(not)h(use)f(the)g (same)g(presen)o(tation.)158 1876 y(Another)j(\(non\)solution)i(is)f(to)e (ignore)i(the)e(issue)i(and)f(let)g(the)g(b)q(eha)o(vior)h(of)e(C)g(to)h(F)m (ortran)f(comm)o(unication)k(b)q(e)75 1922 y(unde\014ned.)32 b(The)18 b(user)g(can)g(alw)o(a)o(ys)g(matc)o(h)g(language)h(at)f(sending)h (and)f(receiving)i(end)e(if)f(he)h(or)g(she)g(is)g(willing)i(to)75 1968 y(struggle)14 b(with)g(the)f(issue)h(of)f(calling)i(MPI)e(C)g(functions) i(from)d(F)m(ortran)i(or)f(vice-v)o(ersa.)75 2207 y Fn(1.6)70 b(Receiv)n(e)20 b(Criteria)75 2305 y Fo(The)e(selection)g(of)f(a)g(message)g (b)o(y)g(a)g(receiv)o(e)i(op)q(eration)e(is)h(done)f(uniquely)g(according)h (to)f(the)h(v)n(alue)f(of)f(the)75 2355 y(message)c(en)o(v)o(elop)q(e.)18 b(The)13 b(receiv)o(e)h(op)q(eration)e(sp)q(eci\014es)i(an)e Fh(en)o(v)o(elop)q(e)g(pattern)p Fo(;)f(a)h(message)g(can)h(b)q(e)g(receiv)o (ed)75 2405 y(b)o(y)20 b(that)g(receiv)o(e)h(op)q(eration)f(only)f(if)g(its)h (en)o(v)o(elop)q(e)h(matc)o(hes)e(that)h(pattern.)38 b(A)20 b(pattern)g(sp)q(eci\014es)i(v)n(alues)75 2455 y(for)c(the)h Fi(source)p Fo(,)f Fi(tag)g Fo(and)h Fi(context)e Fo(\014elds)i(of)f(the)h (message)f(en)o(v)o(elop)q(e.)32 b(In)19 b(addition,)f(the)h(v)n(alue)f(for)g (the)75 2504 y Fi(dest)g Fo(\014eld)g(is)g(set,)i(implicitly)m(,)c(to)i(b)q (e)h(equal)f(to)h(the)g(receiving)f(pro)q(cess)j(id.)31 b(The)19 b(receiv)o(er)h(ma)o(y)c(sp)q(ecify)j(a)75 2554 y Fi(DONTCARE)11 b Fo(v)n(alue)i(for)f Fi(source)p Fo(,)g(or)h Fi(tag)p Fo(,)f(indicating)g (that)h(an)o(y)g(source)h(and/or)f(tag)f(are)i(acceptable.)19 b(It)13 b(cannot)75 2604 y(sp)q(ecify)k(a)f(DONTCARE)h(v)n(alue)e(for)h Fi(context)f Fo(or)i Fi(dest)p Fo(.)25 b(Th)o(us,)17 b(a)f(message)g(can)g(b) q(e)i(receiv)o(ed)f(b)o(y)g(a)f(receiv)o(e)75 2654 y(op)q(eration)e(only)f (if)g(it)g(is)h(addressed)h(to)f(the)g(receiving)g(task,)g(has)f(a)h(matc)o (hing)e(con)o(text,)i(has)g(matc)o(hing)e(source)75 2704 y(unless)18 b(source=DONTCARE)i(in)d(the)h(pattern,)h(and)e(has)h(a)f(matc)o(hing)f(tag)h (unless)i(tag=DONTCARE)e(in)p eop %%Page: 15 16 bop 75 -100 a Fg(1.7.)31 b(COMMUNICA)m(TION)14 b(MODE)1136 b Fo(15)75 45 y(the)18 b(pattern.)28 b(DONTCARE)16 b(v)n(alues)h(for)g (source)h(or)f(tag)f(are)i(sp)q(eci\014ed)g(via)e(a)h(named)f(constan)o(t)h (for)g(source)75 95 y(don)o(tcare)e(and)e(a)h(named)f(constan)o(t)h(for)g (tag)f(don)o(tcare.)158 145 y(The)i(length)f(of)g(the)g(receiv)o(ed)i (message)e(m)o(ust)f(b)q(e)i(less)g(then)g(or)f(equal)g(the)h(length)f(of)g (the)h(receiv)o(e)g(bu\013er.)75 195 y(I.e.,)d(all)f(incoming)f(data)i(m)o (ust)f(\014t,)i(without)f(truncation,)g(in)o(to)g(the)h(receiv)o(e)g (bu\013er.)19 b(It)12 b(is)h(erroneous)g(to)g(receiv)o(e)75 244 y(a)h(message)f(whic)o(h)h(length)g(exceed)i(the)e(receiv)o(e)h (bu\013er,)g(and)e(the)i(outcome)e(of)g(program)g(where)i(this)f(o)q(ccurs)h (is)75 294 y(undetermined.)75 432 y Fn(1.7)70 b(Comm)n(unication)20 b(Mo)r(de)75 523 y Fo(A)14 b(send)h(op)q(eration)f(can)g(o)q(ccur)h(in)e(one) h(of)f(three)j(mo)q(des:)75 606 y Fh(ST)l(AND)o(ARD)k Fo(The)14 b(send)h(ma)o(y)d(start)i(whether)i(or)d(not)h(a)g(matc)o(hing)e(receiv)o(e)j (has)f(b)q(een)h(p)q(osted.)75 690 y Fh(READ)o(Y)20 b Fo(The)15 b(send)g(ma)o(y)c(start)k(only)e(if)g(a)h(matc)o(hing)e(receiv)o(e)j(has)f(b) q(een)h(p)q(osted.)75 773 y Fh(SECURE)21 b Fo(The)12 b(send)g(op)q(eration)f (will)f(not)h(complete)g(successfully)h(un)o(til)e(it)h(is)g(guaran)o(teed)h (that)f(the)h(message)179 823 y(sen)o(t)i(will)f(b)q(e)h(receiv)o(ed.)158 906 y(The)k(start)g(and)f(completion)e(of)i(a)g Fh(standard)e Fo(send)j(do)q(es)g(not)f(dep)q(end)h(on)f(the)h(status)g(of)f(a)g(matc)o (hing)75 956 y(receiv)o(e.)158 1006 y(A)e Fh(ready)h(send)e Fo(can)h(start)g(only)f(if)g(a)h(matc)o(hing)e(receiv)o(e)j(is)f(already)f(p) q(osted;)i(otherwise)g(the)f(op)q(eration)75 1056 y(is)e(erroneous)h(and)e (its)h(outcome)f(is)g(unde\014ned.)19 b(In)13 b(some)f(systems,)g(this)h (allo)o(ws)f(the)h(remo)o(v)n(al)d(of)i(a)h(hand-shak)o(e)75 1106 y(op)q(eration)h(that)g(is)f(otherwise)i(required,)g(and)e(results)i(in) f(impro)o(v)o(ed)e(p)q(erformance.)18 b(Its)c(completion)e(do)q(es)j(not)75 1155 y(dep)q(end)g(on)f(the)g(status)h(of)e(a)h(matc)o(hing)e(receiv)o(e.)158 1205 y(A)20 b Fh(secure)h(send)d Fo(will)g(complete)h(if)g(a)g(matc)o(hing)f (receiv)o(e)j(is)e(p)q(osted,)i(and)f(the)g(receiv)o(e)h(op)q(eration)e(is)75 1255 y(committed)c(to)h(receiv)o(e)j(the)e(message)g(sen)o(t)h(b)o(y)f(the)g (secure)i(send.)28 b(\(I.e.,)17 b(the)h(send)g(and)e(receiv)o(e)j(op)q (erations)75 1305 y(ha)o(v)o(e)h(already)h(b)q(een)g(matc)o(hed,)h(the)f (receiv)o(e)h(op)q(eration)e(has)h(already)f(started)i(and)e(cannot)h(b)q(e)g (cancelled)75 1355 y(an)o(ymore.\))c(The)d(start)h(of)e(a)g(secure)j(send)f (do)q(es)g(not)f(dep)q(end)h(on)e(the)i(status)g(of)e(a)g(matc)o(hing)f (receiv)o(e.)158 1405 y(A)j(receiv)o(e)i(op)q(eration)e(can)h(o)q(ccur)g(in)f (one)h(of)f(t)o(w)o(o)g(mo)q(des:)20 b Fh(ST)l(AND)o(ARD)14 b Fo(and)h Fh(SECURE)p Fo(.)h(A)f(secure)75 1454 y(send)i(can)g(b)q(e)g(matc) o(hed)f(only)g(b)o(y)g(a)g Fh(secure)i(receiv)o(e)p Fo(;)e(a)g(program)f (where)j(a)e(secure)i(send)g(can)f(b)q(e)g(matc)o(hed)75 1504 y(b)o(y)e(a)f(standard)h(receiv)o(e)h(is)f(erroneous)h(and)f(its)f(b)q(eha)o (vior)h(is)g(unde\014ned.)22 b(The)15 b(secure)i(receiv)o(e)f(can)f(b)q(e)g (p)q(osted)75 1554 y(either)g(b)q(efore)f(or)g(after)g(the)h(matc)o(hing)d (send.)158 1604 y(A)f(standard)g(send)g(and)f(a)h(ready)f(send)i(can)f(b)q (oth)f(b)q(e)h(matc)o(hed)f(only)g(b)o(y)g(a)g Fh(standard)h(receiv)o(e)p Fo(;)e(a)i(program)75 1654 y(where)k(a)f(secure)i(receiv)o(e)g(can)e(matc)o (h)f(a)h(nonsecure)i(send)f(is)g(erroneous)g(and)f(its)g(b)q(eha)o(vior)g(is) g(unde\014ned.)21 b(The)75 1704 y(receiv)o(e)14 b(has)g(to)f(b)q(e)h(p)q (osted)g(ahead)f(of)f(the)i(send)g(for)f(a)g(ready)g(send)h(and)f(can)h(b)q (e)g(p)q(osted)g(either)g(b)q(efore)g(or)f(after)75 1753 y(the)h(send)h(is)f (p)q(osted)h(for)e(a)h(standard)g(send.)158 1882 y Fm(Discussion:)158 1932 y Fl(I)h(got)h(criticisms)h(ab)q(out)f(the)g(previous)h(notation)g (\(REGULAR\))e(and)h(the)g(curren)o(t)g(one)g(\(ST)m(AND)o(ARD\).)e(I)h(can) 75 1982 y(also)g(try)e(NORMAL,)g(DEF)l(A)o(UL)m(T,)f(...)18 b(W)m(e)c(should)h(probably)h(v)o(ote,)d(at)h(some)g(p)q(oin)o(t.)19 b(Also,)14 b(Lyndon)h(prop)q(oses)g(to)e(use)75 2031 y(SYNCHR)o(ONOUS,)f (rather)i(than)g(secure,)g(since)h(the)e(other)h(mo)q(des)g(can)g(also)h (o\013er)e(secure)i(comm)o(unication)h(with)e(the)75 2081 y(righ)o(t)g(use.) 158 2292 y Fm(Implemen)o(tation)f(note:)158 2338 y Fl(The)g(curren)o(t)h (de\014nition)i(assumes)e(it)g(is)g(the)f(user)h(resp)q(onsibil)q(it)o(y)i (to)e(prev)o(en)o(t)g(matc)o(hing)g(of)f(a)h(regular)g(send)g(with)75 2384 y(a)f(secure)h(receiv)o(e)g(and)f(vice-v)o(ersa.)18 b(Otherwise,)c (messages)g(need)f(b)q(e)g(tagged)h(as)f(standard)i(or)e(secure.)158 2429 y(Assume)i(an)g(implemen)o(tation)j(of)d(MPI)f(comm)o(unication)k(that)d (a)o(v)o(oids)h(retries.)23 b(What)15 b(proto)q(cols)i(can)e(b)q(e)g(used?)75 2475 y(Since)j(a)e(receiv)o(e)i(ma)o(y)f(ha)o(v)o(e)g(a)g(wildcard)h(source)f (\014eld,)i(the)d(\014rst)h(step)g(in)h(the)e(proto)q(col)i(m)o(ust)f(b)q(e)g (executed)h(b)o(y)f(the)75 2521 y(sender.)158 2567 y(F)m(or)c(a)g(ready)h (send,)f(this)h(\014rst)f(step)h(ma)o(y)f(in)o(v)o(olv)o(e)i(sending)g(the)e (data,)h(since)g(the)f(receiv)o(er)h(is)g(guaran)o(teed)g(to)f(ha)o(v)o(e)75 2612 y(space)h(to)f(store)g(it.)k(The)c(proto)q(col)h(is)g(send,)f(p)q (ossibly)j(follo)o(w)o(ed)e(b)o(y)g(an)f(ac)o(k-send.)158 2658 y(The)g(proto)q(col)h(for)e(secure)i(send)f(is)g(lik)o(ely)i(to)e(b)q(e)g (the)g(follo)o(wing:)19 b(The)13 b(sender)g(sends)h(a)e(request-to-send)i (message.)75 2704 y(The)f(receiv)o(er)h(stores)f(this)g(request)h(\(whic)o(h) g(con)o(tains)g(a)f(cop)o(y)g(of)g(the)g(message)g(en)o(v)o(elop)q(e\).)19 b(When)14 b(a)f(matc)o(hing)h(receiv)o(e)p eop %%Page: 16 17 bop 75 -100 a Fo(16)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fl(is)g(p)q(osted,)h(the)f(receiv)o (er)h(sends)f(bac)o(k)h(an)f(ac)o(kno)o(wledge,)i(and)e(the)g(sender)h(no)o (w)e(sends)i(the)f(message.)20 b(The)14 b(proto)q(col)h(is)75 91 y(\(req,)e(ac)o(k-req,)g(send\),)g(p)q(ossibly)j(follo)o(w)o(ed)e(b)o(y)f (an)h(ac)o(k-send.)158 136 y(What)g(proto)q(col)h(is)f(used)g(for)f(standard) i(sends?)k(Either)14 b(the)g(second)g(proto)q(col)h(\(req,)e(ac)o(k-req,)g (send\))h(or)g(a)f(mix)h(of)75 182 y(the)f(\014rst)g(and)h(the)f(second)h (\(e.g.)j(\014rst)c(used)h(for)e(short)i(messages,)f(second)h(for)f(long)h (messages\).)158 228 y(When)h(the)g(receiv)o(er)g(do)q(es)g(not)f(wildcard)i (the)f(source)g(\014eld)g(then)g(the)f(proto)q(col)i(can)f(b)q(e)f(rev)o (ersed:)21 b(The)14 b(receiv)o(er)75 273 y(sends)h(a)f(clear-to-send)h (message,)g(and)f(the)h(sender)f(can)h(pro)q(ceed)g(with)f(the)g(short)h (proto)q(col)g(if)f(it)h(has)f(a)g(clear-to-send)75 319 y(when)f(the)g(send)h (is)g(executed.)158 365 y(What)f(do)g(w)o(e)g(gain)g(from)g(the)g(\\secure)g (receiv)o(e"?)19 b(Some)13 b(optimizations)i(for)e(regular)h(receiv)o(es)g (\(e.g.,)d(in)j(a)e(regular)75 410 y(receiv)o(e)18 b(of)f(a)g(short)g (message)h(the)f(receiv)o(er)h(kno)o(ws)g(the)f(short)g(proto)q(col)i(will)f (b)q(e)g(used,)g(and)g(need)f(not)h(c)o(hec)o(k)f(for)g(a)75 456 y(request-to-send\).)h(Q:)12 b(is)i(that)f(enough)i(justi\014cation)g (for)e(the)g(secure)g(receiv)o(e?)158 502 y(A)k(coun)o(ter)g(argumen)o(t)h (\(Leslie)g(Hart\))f(is)h(that)f(one)g(could)i(conceiv)o(e)f(of)f(an)g (implemen)o(tation)j(where)d(a)g(regular)75 547 y(receiv)o(e)12 b(reserv)o(es)g(bu\013er)g(space,)f(so)h(that)f(a)g(program)h(that)f(use)h (regular)g(receiv)o(es)g(is)g(not)f(safe,)g(ev)o(en)h(if)f(only)h(secure)g (sends)75 593 y(are)h(used.)18 b(I)12 b(doubt)i(this)g(is)f(a)g(strong)h (argumen)o(t.)k(On)13 b(the)g(other)g(hand,)h(it)f(indicates)i(that,)e(if)g (the)g(exp)q(ectation)i(is)f(that)75 639 y(programs)i(written)g(using)g(only) h(secure)e(sends)h(are)g(safe,)f(then)h(this)g(is)f(an)h(additional)i (correctness)e(condition)i(on)e(the)75 684 y(implemen)o(tation)g(that)d(need) h(b)q(e)f(made)h(explicit.)75 903 y Fn(1.8)70 b(Comm)n(unication)20 b(Ob)t(jects)75 994 y Fo(An)15 b(opaque)g(comm)o(unication)d(ob)r(ject)k (iden)o(ti\014es)g(v)n(arious)e(prop)q(erties)j(of)e(a)g(comm)o(uni)o(cation) d(op)q(eration,)j(suc)o(h)75 1043 y(as)h(the)g(bu\013er)h(descriptor)h(that)d (is)h(asso)q(ciated)h(with)e(it,)h(its)g(con)o(text,)g(the)h(tag)e(and)h (destination)g(parameters)75 1093 y(to)f(b)q(e)h(used)g(for)f(a)g(send,)h(or) f(the)h(tag)f(and)g(source)h(parameters)g(to)f(b)q(e)g(used)i(for)d(a)h (receiv)o(e.)24 b(In)15 b(addition,)f(this)75 1143 y(ob)r(ject)g(stores)h (information)10 b(ab)q(out)j(the)h(status)g(of)e(the)i(last)f(comm)o (unication)d(op)q(eration)j(that)g(w)o(as)g(p)q(erformed)75 1193 y(with)h(this)f(ob)r(ject.)19 b(This)14 b(ob)r(ject)h(is)f(accessed)i (using)d(a)h(comm)o(unicatio)o(n)d(handle.)158 1243 y(One)k(can)f(consider)h (comm)o(uni)o(cation)c(op)q(erations)j(to)g(consist)g(of)g(the)g(follo)o (wing)d(sub)q(op)q(erations:)75 1324 y Fh(INIT\(op)q(eration,)j(params,)i (handle\))i Fo(Pro)q(cess)c(pro)o(vides)f(all)f(relev)n(an)o(t)g(parameters)h (for)f(its)h(participation)179 1374 y(in)i(the)i(comm)o(unication)c(op)q (eration)j(\(t)o(yp)q(e)g(of)g(op)q(eration,)g(data)g(bu\013er,)h(tag,)e (participan)o(ts,)i(etc.\).)25 b(An)179 1424 y(ob)r(ject)14 b(is)g(created)i(that)e(iden)o(ti\014es)g(the)g(op)q(eration.)75 1503 y Fh(ST)l(AR)l(T\(handle\))k Fo(The)c(comm)o(unication)c(op)q(eration)k (is)g(started)75 1582 y Fh(COMPLETE\(handle\))19 b Fo(The)14 b(comm)o(unication)c(op)q(eration)k(is)g(completed.)75 1661 y Fh(FREE\(handle\))k Fo(The)d(comm)o(unicati)o(on)c(ob)r(ject,)j(and)g(asso) q(ciated)h(resources)h(are)e(freed.)75 1743 y(Correct)h(in)o(v)o(o)q(cation)e (of)g(these)i(sub)q(op)q(erations)g(is)f(a)f(sequence)k(of)c(the)h(form)587 1825 y Fh(INIT)j Fo(\()p Fh(ST)l(AR)l(T)f(COMPLETE)p Fo(\))1187 1808 y Fb(\003)1223 1825 y Fh(FREE)p Fd(:)75 1907 y Fo(I.e.,)10 b(an)h(ob)r(ject)g(needs)h(b)q(e)f(created)i(b)q(efore)e(comm)o(unication)c (o)q(ccurs;)13 b(it)d(can)h(b)q(e)g(reused)i(only)c(after)i(the)h(previous)75 1956 y(use)k(has)g(completed;)f(and)g(it)g(needs)i(to)e(b)q(e)h(freed)h(ev)o (en)o(tually)d(\(of)h(course,)i(one)f(can)f(assume)g(that)h(all)e(ob)r(jects) 75 2006 y(are)g(freed)h(at)f(program)e(termination,)g(b)o(y)h(default\).)158 2056 y(The)21 b(ab)q(o)o(v)o(e)g(scenario)g(p)q(ertains)g(to)g Fj(p)n(ersistent)f Fo(ob)r(jects.)39 b(One)22 b(can)f(also)f(create)i Fj(ephemer)n(al)e Fo(ob)r(jects.)75 2106 y(Suc)o(h)c(an)g(ob)r(ject)h(p)q (ersists)h(only)d(un)o(til)g(the)i(comm)o(uni)o(cation)c(op)q(eration)j(is)g (completed,)f(at)h(whic)o(h)g(p)q(oin)o(t)f(it)h(is)75 2156 y(destro)o(y)o(ed.)i(Th)o(us)12 b(the)g(correct)i(in)o(v)o(o)q(cation)c(of)h (sub)q(op)q(erations)i(with)e(an)g(ephemeral)g(ob)r(ject)i(is)e Fh(INIT)j(ST)l(AR)l(T)75 2205 y(COMPLETE)p Fo(.)158 2255 y(A)19 b(user)i(ma)o(y)c(directly)j(in)o(v)o(ok)o(e)f(these)h(sub)q(op)q(erations.) 36 b(This)19 b(w)o(ould)g(allo)o(w)e(the)j(amortization)e(of)g(the)75 2305 y(o)o(v)o(erhead)d(of)g(setting)g(up)g(a)g(comm)o(unication)d(o)o(v)o (er)j(man)o(y)e(successiv)o(e)k(uses)f(of)f(the)g(same)g(handle,)f(and)h (allo)o(ws)75 2355 y(the)d(o)o(v)o(erlap)f(of)g(comm)o(unication)e(and)i (computation.)16 b(Simpler)10 b(comm)o(unicatio)o(n)f(op)q(erations)j(com)o (bine)e(sev)o(eral)75 2405 y(of)17 b(these)i(sub)q(op)q(erations)g(in)o(to)e (one)h(op)q(eration,)g(th)o(us)g(simplifying)d(the)j(use)h(of)e(comm)o (unicatio)o(n)e(primitiv)o(es.)75 2455 y(Th)o(us,)h(one)h(only)e(needs)j(to)e (sp)q(ecify)g(precisely)h(the)g(seman)o(tics)f(of)f(these)j(sub)q(op)q (erations)f(in)f(order)g(to)g(sp)q(ecify)75 2504 y(the)e(seman)o(tics)g(of)f (MPI)h(p)q(oin)o(t)g(to)f(p)q(oin)o(t)h(comm)o(unicatio)o(n)d(op)q(erations.) 158 2554 y(W)m(e)i(sa)o(y)g(that)g(a)g(comm)o(unicatio)o(n)d(op)q(eration)j (\(send)i(or)e(receiv)o(e\))h(is)f Fh(p)q(osted)f Fo(once)i(a)f Fh(start)e Fo(sub)q(op)q(eration)75 2604 y(has)18 b(b)q(een)h(in)o(v)o(ok)o (ed;)g(the)g(op)q(eration)f(is)g Fh(completed)e Fo(once)i(the)h Fh(complete)d Fo(sub)q(op)q(eration)j(has)f(completed.)75 2654 y(A)e(send)h(and)f(a)g(receiv)o(e)i(op)q(eration)e Fh(matc)o(h)f Fo(if)g(the)i(receiv)o(e)h(pattern)f(sp)q(eci\014ed)g(b)o(y)f(the)h(receiv)o (e)h(matc)o(hes)d(the)75 2704 y(message)f(en)o(v)o(elop)q(e)g(created)h(b)o (y)f(the)g(send.)p eop %%Page: 17 18 bop 75 -100 a Fg(1.8.)31 b(COMMUNICA)m(TION)14 b(OBJECTS)1073 b Fo(17)75 45 y Fk(1.8.1)55 b(Comm)n(unication)21 b(Ob)s(ject)d(Creation)75 206 y Fm(Discussion:)35 b Fl(This)14 b(section)g(has)f(not)h(y)o(et)f(b)q (een)g(appro)o(v)o(ed)158 339 y Fo(An)c(ob)r(ject)i(for)d(a)h(send)i(op)q (eration)e(is)g(created)i(b)o(y)e(a)g(call)f(to)h Fh(MPI)p 1180 339 15 2 v 18 w(INIT)p 1304 339 V 17 w(SEND)p Fo(.)g(A)g(call)g(to)g Fh(MPI)p 1728 339 V 17 w(INIT)p 1851 339 V 18 w(RECV)75 389 y Fo(is)17 b(similarly)c(used)18 b(for)f(creating)g(an)f(ob)r(ject)i(for)e(a) h(receiv)o(e)h(op)q(eration.)26 b(The)18 b(creation)f(of)f(a)g(comm)o (unication)75 439 y(ob)r(ject)f(is)e(a)h(lo)q(cal)f(op)q(eration)h(that)g (need)h(not)e(in)o(v)o(olv)o(e)g(comm)o(unicatio)o(n)e(with)j(a)f(remote)h (pro)q(cess.)158 525 y Fh(MPI)p 257 525 V 17 w(INIT)p 380 525 V 18 w(SEND)i(\(handle,)e(bu\013er)p 856 525 V 15 w(handle,)g(dest,)h(tag,)h (con)o(text,)f(mo)q(de,)g(p)q(ersistence\))158 611 y Fo(Creates)g(a)f(send)h (comm)o(unicati)o(on)c(ob)r(ject.)19 b(P)o(arameters)14 b(are)75 696 y Fh(OUT)i(handle)i Fo(message)g(handle.)28 b(The)18 b(handle)g(should)f (not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g(ob)r(ject)h(b)q(efore)g(the) 179 746 y(call.)75 832 y Fh(IN)e(bu\013er)p 273 832 V 16 w(handle)j Fo(handle)13 b(to)h(send)h(bu\013er)g(descriptor)75 918 y Fh(IN)h(dest)k Fo(rank)14 b(in)f(con)o(text)i(of)e(destination)h(\(in)o(teger\))75 1004 y Fh(IN)i(tag)21 b Fo(user)15 b(tag)e(for)h(messages)g(sen)o(t)g(with)g (this)g(handle)g(\(in)o(teger\))75 1090 y Fh(IN)i(con)o(text)k Fo(con)o(text)14 b(of)f(messages)h(sen)o(t)h(with)f(this)g(handle)75 1176 y Fh(IN)i(mo)q(de)21 b Fo(send)14 b(mo)q(de)f(\(state)i(t)o(yp)q(e,)f (with)f(three)j(v)n(alues:)h Fi(MPI)p 1119 1176 14 2 v 15 w(STANDARD)p Fo(,)12 b Fi(MPI)p 1400 1176 V 15 w(READY)h Fo(and)h Fi(MPI)p 1685 1176 V 15 w(SECURE)p Fo(\))75 1262 y Fh(IN)i(p)q(ersistence)i Fo(handle)10 b(p)q(ersistence)i(\(state)e(t)o(yp)q(e,)g(with)f(t)o(w)o(o)g(v) n(alues:)15 b Fi(MPI)p 1318 1262 V 15 w(PERSISTENT)8 b Fo(and)h Fi(MPI)p 1703 1262 V 15 w(EPHEMERAL)p Fo(\))158 1383 y Fh(MPI)p 257 1383 15 2 v 17 w(INIT)p 380 1383 V 18 w(RECV)15 b(\(handle,)e(bu\013er)p 859 1383 V 15 w(handle,)g(source,)h(tag,)g(con)o(text,)f(mo)q(de,)i(p)q (ersistence\))158 1469 y Fo(Create)g(a)e(receiv)o(e)j(handle.)i(P)o (arameters)c(are)75 1555 y Fh(OUT)i(handle)i Fo(message)g(handle.)28 b(The)18 b(handle)g(should)f(not)g(b)q(e)h(asso)q(ciated)h(with)e(an)o(y)g (ob)r(ject)h(b)q(efore)g(the)179 1604 y(call.)75 1690 y Fh(IN)e(bu\013er)p 273 1690 V 16 w(handle)j Fo(handle)13 b(to)h(receiv)o(e)h(bu\013er)g (descriptor.)75 1777 y Fh(IN)h(source)k Fo(rank)14 b(in)f(con)o(text)i(of)e (source,)i(or)f Fi(MPI)p 897 1777 14 2 v 15 w(ANY)p 978 1777 V 15 w(SOURCE)f Fo(\(in)o(teger\).)75 1863 y Fh(IN)j(tag)21 b Fo(user)15 b(tag)e(for)h(messages)g(receiv)o(ed)h(with)f(this)f(handle,)h (or)g Fi(MPI)p 1232 1863 V 15 w(ANY)p 1313 1863 V 15 w(TAG)f Fo(\(in)o(teger\).)75 1949 y Fh(IN)j(con)o(text)k Fo(con)o(text)14 b(of)f(messages)h(receiv)o(ed)i(with)d(this)h(handle.)75 2035 y Fh(IN)i(mo)q(de)21 b Fo(receiv)o(e)15 b(mo)q(de)e(\(state)h(t)o(yp)q(e,)g (with)g(t)o(w)o(o)f(v)n(alues:)18 b Fi(MPI)p 1133 2035 V 15 w(STANDARD)p Fo(,)12 b(and)i Fi(MPI)p 1495 2035 V 15 w(SECURE)p Fo(\))75 2121 y Fh(IN)i(p)q(ersistence)i Fo(handle)10 b(p)q(ersistence)i (\(state)e(t)o(yp)q(e,)g(with)f(t)o(w)o(o)g(v)n(alues:)15 b Fi(MPI)p 1318 2121 V 15 w(PERSISTENT)8 b Fo(and)h Fi(MPI)p 1703 2121 V 15 w(EPHEMERAL)p Fo(\))158 2206 y(See)15 b(Section)f(1.4.2)e(for) i(a)f(discussion)i(of)e(source,)i(tag)e(and)h(con)o(text.)158 2335 y Fm(Discussion:)35 b Fl(I)12 b(ha)o(v)o(e)h(not)g(included)i(prop)q (osals)f(for)f(partially)i(sp)q(eci\014ed)f(message)f(handles,)i(that)d(some) h(p)q(eoples)75 2381 y(seem)g(to)g(desire.)158 2427 y(I)g(ha)o(v)o(e)g (merged)h(all)g(handle)h(setup)e(in)o(to)h(one)g(call.)158 2474 y(The)c(functions)i Ff(MPI)p 459 2474 12 2 v 13 w(INIT)p 552 2474 V 12 w(SEND)c Fl(and)j Ff(MPI)p 784 2474 V 13 w(INIT)p 877 2474 V 13 w(RECV)d Fl(ha)o(v)o(e)j(the)f(same)g(list)h(of)f(parameters.) 17 b(Jim)10 b(Co)o(wnie)h(suggests)75 2519 y(to)i(ha)o(v)o(e)g(only)i(one)e (function)i(with)e(an)g(extra)h(argumen)o(t.)158 2566 y(Ephemeral)f(\\p)q (ersisten)o(t)h(handles")f(are)f(not)g(terribly)h(useful)g(as)f(suc)o(h.)17 b(But)12 b(their)g(existence)h(allo)o(w)g(us)f(to)f(describ)q(e)75 2612 y(all)j(the)f(higher)i(lev)o(el)f(send/receiv)o(e)h(op)q(erations)g(in)f (terms)f(of)g(basic)h(op)q(erations)h(on)e(handles.)p eop %%Page: 18 19 bop 75 -100 a Fo(18)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fk(1.8.2)55 b(Comm)n(unication)21 b(Start)75 157 y Fh(MPI)p 174 157 15 2 v 17 w(ST)l(AR)l(T\(handle\))75 274 y(IN)16 b(handle)j Fo(comm)o(unication)10 b(handle)158 355 y(The)16 b Fi(MPI)p 314 355 14 2 v 15 w(START)f Fo(function)h(starts)g (the)h(execution)f(of)g(a)f(comm)o(unication)e(op)q(eration)i(\(send)i(or)f (receiv)o(e\).)75 404 y(A)i(sender)i(should)d(not)h(up)q(date)h(the)f(send)h (bu\013er)g(after)f(a)g(send)h(op)q(eration)f(has)g(started)h(un)o(til)e (after)h(it)g(has)75 454 y(completed.)31 b(A)18 b(receiv)o(er)i(should)e(not) h(access)h(the)f(receiv)o(e)h(bu\013er)f(after)g(a)f(receiv)o(e)h(op)q (eration)g(w)o(as)f(started)75 504 y(un)o(til)c(after)h(it)f(has)h (completed.)21 b(A)15 b(program)e(that)i(do)q(es)g(not)g(satisfy)g(these)h (conditions)e(is)h(erroneous)h(and)f(its)75 554 y(outcome)e(is)h (undetermined.)75 670 y Fk(1.8.3)55 b(Comm)n(unication)21 b(Completion)75 782 y Fh(MPI)p 174 782 15 2 v 17 w(W)-5 b(AIT)17 b(\()e(handle,)f(return)p 675 782 V 15 w(status)p 816 782 V 16 w(handle\))75 898 y(IN)i(handle)j Fo(comm)o(unication)10 b(handle)75 980 y Fh(IN)16 b(return)p 284 980 V 15 w(handle)j Fo(handle)13 b(that)h(is)f(asso)q(ciated)i(with)e (return)i(status)f(ob)r(ject.)19 b(the)14 b(return)g(status)h(ob)r(ject)f(is) 179 1030 y(up)q(dated)g(b)o(y)g(call.)158 1111 y(A)g(call)e(to)i Fh(MPI)p 428 1111 V 17 w(W)-5 b(AIT)14 b Fo(returns)h(when)f(the)g(send)g(op) q(eration)g(iden)o(ti\014ed)f(b)o(y)h Fi(handle)e Fo(is)h(complete.)18 b(The)75 1161 y(completion)13 b(of)h(a)g(send)h(op)q(eration)g(indicates)g (that)f(the)h(sender)i(is)d(no)o(w)g(free)i(to)e(up)q(date)h(the)g(lo)q (cations)f(in)g(the)75 1211 y(send)e(bu\013er,)h(or)e(an)o(y)g(other)h(lo)q (cation)e(that)i(can)f(b)q(e)h(referenced)i(b)o(y)d(the)h(send)g(op)q (eration.)17 b(Ho)o(w)o(ev)o(er,)12 b(it)f(do)q(es)h(not)75 1261 y(indicate)f(that)h(the)g(message)f(has)h(b)q(een)g(receiv)o(ed;)h (rather)g(it)e(ma)o(y)e(ha)o(v)o(e)j(b)q(een)g(bu\013ered)h(b)o(y)e(the)h (comm)o(unication)75 1310 y(subsystem.)158 1360 y(The)20 b(completion)e(of)i (a)f(receiv)o(e)i(op)q(eration)f(indicates)g(that)g(the)h(receiv)o(er)g(is)f (no)o(w)f(free)i(to)f(access)h(the)75 1410 y(lo)q(cations)15 b(in)g(the)h(receiv)o(e)h(bu\013er,)f(whic)o(h)f(con)o(tain)g(the)h(receiv)o (ed)h(message,)e(or)h(an)o(y)f(other)h(lo)q(cation)e(that)i(can)75 1460 y(b)q(e)g(referenced)i(b)o(y)d(the)h(receiv)o(e)h(op)q(eration.)23 b(It)15 b(do)q(es)h(not)g(indicate)f(that)h(the)g(matc)o(hing)d(send)j(op)q (eration)g(has)75 1510 y(completed.)158 1559 y(The)k(call)e(returns)j(a)e (handle)h(to)f(an)g(opaque)g(ob)r(ject)h(that)g(con)o(tains)f(information)d (on)k(the)g(completed)75 1609 y(op)q(eration)14 b({)f(the)i Fh(return)f(status)e Fo(ob)r(ject.)158 1694 y Fh(MPI)p 257 1694 V 17 w(ST)l(A)l(TUS)j(\(handle,)f(\015ag,)i(return)p 898 1694 V 15 w(handle\))75 1811 y(IN)g(handle)j Fo(comm)o(unication)10 b(handle)75 1893 y Fh(OUT)16 b(\015ag)k Fo(logical)75 1975 y Fh(OUT)c(return)p 335 1975 V 15 w(handle)i Fo(handle)c(that)g(is)g(asso)q (ciated)g(with)g(return)h(status)g(ob)r(ject.)158 2056 y(A)e(call)f(to)h Fh(MPI)p 426 2056 V 17 w(ST)l(A)l(TUS)f Fo(returns)j Fi(flag=true)c Fo(if)h(the)h(op)q(eration)g(iden)o(ti\014ed)g(b)o(y)g Fi(handle)e Fo(is)i(complete.)75 2106 y(In)j(suc)o(h)g(case,)g(the)h(return)f(handle)g(p) q(oin)o(ts)f(to)h(an)f(opaque)h(ob)r(ject)g(that)g(con)o(tains)f(information) e(on)i(the)h(com-)75 2156 y(pleted)f(op)q(eration.)k(It)c(returns)h Fi(flag=false)p Fo(,)c(otherwise.)20 b(In)15 b(suc)o(h)g(case,)g(the)g(v)n (alue)f(of)f(the)i(return)h(handle)e(is)75 2205 y(unde\014ned.)158 2255 y(A)f(call)g(to)g Fi(MPI)p 397 2255 14 2 v 15 w(WAIT)f Fo(blo)q(c)o(ks)h(only)f(the)i(executing)g(thread.)k(If)13 b(the)h(executing)g(pro)q(cess)h(is)e(m)o(ultithreaded,)75 2305 y(then)i(other)f(threads)h(within)e(the)i(pro)q(cess)g(can)f(b)q(e)h(sc) o(heduled)g(for)f(execution.)158 2355 y(The)h(use)h(of)f(a)f(blo)q(c)o(king)h (receiv)o(e)h(op)q(eration)f(\()p Fi(MPI)p 977 2355 V 15 w(WAIT)p Fo(\))f(allo)o(ws)g(the)i(op)q(erating)f(system)f(to)h(desc)o(hedule)75 2405 y(the)j(blo)q(c)o(k)o(ed)g(thread)h(and)e(sc)o(hedule)i(another)g (thread)f(for)f(execution,)j(if)c(suc)o(h)j(is)f(a)o(v)n(ailable.)27 b(The)18 b(use)h(of)e(a)75 2455 y(non)o(blo)q(c)o(king)12 b(receiv)o(e)i(op)q (eration)f(\()p Fi(MPI)p 710 2455 V 15 w(STATUS)p Fo(\))f(allo)o(ws)g(the)i (user)g(to)f(sc)o(hedule)h(alternativ)o(e)f(activities)g(within)75 2504 y(a)h(single)f(thread)i(of)e(execution.)158 2554 y(The)g(in)o(tended)g (implemen)o(tation)c(of)j Fi(MPI)p 820 2554 V 16 w(STATUS)f Fo(is)h(for)h(that)f(op)q(eration)h(to)f(return)i(as)f(so)q(on)f(as)h(p)q (ossible.)75 2604 y(Ho)o(w)o(ev)o(er,)h(if)f(rep)q(eatedly)i(called)f(for)f (an)h(op)q(eration)g(that)f(is)h(enabled,)g(it)f(m)o(ust)g(ev)o(en)o(tually)h (succeed.)158 2654 y(The)g(return)h(status)g(ob)r(ject)g(for)e(a)h(send)h(op) q(eration)e(carries)i(no)f(information.)h(The)f(return)h(status)g(ob)r(ject) 75 2704 y(for)h(a)g(receiv)o(e)i(op)q(eration)f(carries)g(information)d(on)i (the)i(source,)g(tag)e(and)g(length)h(of)f(the)h(receiv)o(ed)h(message.)p eop %%Page: 19 20 bop 75 -100 a Fg(1.8.)31 b(COMMUNICA)m(TION)14 b(OBJECTS)1073 b Fo(19)75 45 y(These)13 b(\014elds)g(are)f(required)h(b)q(ecause)h(the)e (receiv)o(e)i(op)q(eration)e(ma)o(y)e(ha)o(v)o(e)h(sp)q(eci\014ed)j Fi(DONTCARE)c Fo(in)i(either)h(source)75 95 y(or)h(tag)f(\014eld,)h(and)g (the)g(message)g(ma)o(y)e(ha)o(v)o(e)h(b)q(een)i(shorter)g(than)f(the)h (receiv)o(e)g(bu\013er.)158 180 y Fh(MPI)p 257 180 15 2 v 17 w(QUER)l(Y\()h(handle,)e(coun)o(t,)g(source,)h(tag\))75 295 y(IN)h(handle)j Fo(handle)14 b(to)g(return)h(status)f(ob)r(ject)75 376 y Fh(OUT)i(coun)o(t)j Fo(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(receiv)o (ed.)75 457 y Fh(OUT)i(source)j Fo(rank)14 b(of)g(message)f(sender)j(in)d (message)h(con)o(text)g(\(in)o(teger\).)75 538 y Fh(OUT)i(tag)k Fo(tag)14 b(of)f(receiv)o(ed)i(message)f(\(in)o(teger\).)158 652 y Fh(MPI)p 257 652 V 17 w(CREA)l(TE)p 471 652 V 19 w(ST)l(A)l(TUS\()h (handle\))158 738 y Fo(Creates)g(a)f(new)g(return)h(status)g(ob)r(ject.)75 817 y Fh(OUT)h(handle)i Fo(handle)c(to)g(return)h(status)g(ob)r(ject.)158 974 y Fm(Discussion:)158 1020 y Fl(The)c(use)g(of)f(a)h(return)g(status)g(ob) r(ject,)g(rather)g(than)g(a)g(list)h(of)e(parameters)i(ma)o(y)f(simplify)i (the)e(use)g(of)f(MPI)h(routines,)75 1066 y(if)i(the)f(v)n(alues)i(stored)f (in)g(the)f(ob)r(ject)h(are)f(seldom)i(c)o(hec)o(k)o(ed.)j(A)12 b(prede\014ned)i(return)f(status)g(ob)r(ject)f(should)j(b)q(e)d(pro)o(vided,) 75 1111 y(to)h(ease)g(programming.)19 b(The)12 b(main)i(reason)g(for)e(the)h (use)g(of)g(a)f(return-status)i(ob)r(ject)f(is)g(that)g(one)h(w)o(an)o(ts)e (to)h(b)q(e)g(able)h(to)75 1157 y(use)c(the)h(same)f Ff(MPI)p 357 1157 12 2 v 13 w(WAIT)e Fl(and)j Ff(MPI)p 590 1157 V 13 w(STATUS)c Fl(calls)12 b(for)e(c)o(hec)o(king)h(on)g(di\013eren)o(t)g(t)o(yp) q(es)g(of)f(op)q(erations,)i(e.g.)k(b)q(oth)10 b(sends)h(and)75 1203 y(receiv)o(es,)k(and)f(p)q(erhaps)h(new)e(op)q(erations)j(in)e(the)g (future.)19 b(These)14 b(di\013eren)o(t)h(op)q(erations)g(return)g (di\013eren)o(t)g(information)75 1248 y(\(alb)q(eit)g(w)o(e)d(could)j(decree) e(that)g(a)g(send)h(returns)g(the)f(same)g(information)i(as)e(a)g(receiv)o (e\).)158 1294 y(The)h(issue)h(of)f(memory)g(allo)q(cation)j(for)d(return)p 868 1294 V 15 w(status)g(handles)i(is)f(not)f(y)o(et)g(solv)o(ed.)22 b(There)14 b(is)g(a)g(desire)h(that)g(the)75 1340 y(user)e(will)i(b)q(e)e (able)h(to)f(allo)q(cate)i(space)f(for)e(them)i(\(e.g.,)e(on)h(the)g(stac)o (k\).)158 1385 y(W)m(e)j(return)g(n)o(um)o(b)q(er)g(of)f(elemen)o(ts)i (receiv)o(ed,)g(rather)e(than)i(b)o(ytes)e(receiv)o(ed)i(b)q(ecause)g(\\n)o (um)o(b)q(er)f(of)f(elemen)o(ts")i(is)75 1431 y(closer)c(to)f(the)g (application)j(seman)o(tic)f(lev)o(el)f(\(user)f(do)q(es)h(not)f(need)h(to)f (b)q(e)g(a)o(w)o(are)g(of)g(the)g(size)h(of)f(elemen)o(ts\),)g(and)h(is)g (more)75 1477 y(in)o(v)n(arian)o(t.)23 b(The)14 b(return)h(status)f(ob)r (ject)h(returns)g(the)f(n)o(um)o(b)q(er)h(of)f Fm(elemen)o(ts)g Fl(receiv)o(ed.)22 b(If)13 b(there)i(w)o(as)f(no)g(truncation,)75 1522 y(then)h(this)g(is)g(equal)g(to)f(the)h(n)o(um)o(b)q(er)g(of)f(elemen)o (ts)h(sen)o(t.)21 b(On)14 b(the)g(other)h(hand,)g(the)f(n)o(um)o(b)q(er)h(of) f(b)o(ytes)h(collected)h(from)75 1568 y(the)d(sender)i(memory)m(,)e(the)g(n)o (um)o(b)q(er)h(of)f(b)o(ytes)h(sen)o(t)g(o)o(v)o(er)f(the)h(wire,)f(and)h (the)f(n)o(um)o(b)q(er)i(of)e(b)o(ytes)g(stored)h(in)g(the)g(receiv)o(er)75 1614 y(memory)c(ma)o(y)g(all)i(b)q(e)e(di\013eren)o(t,)i(in)e(a)g (heterogeneous)i(en)o(vironmen)o(t.)18 b(The)10 b(n)o(um)o(b)q(er)g(of)g (elemen)o(ts)h(sen)o(t)f(can)h(b)q(e)f(computed)75 1659 y(form)k(the)g(send)h (bu\013er)h(descriptor;)g(the)e(n)o(um)o(b)q(er)h(of)f(elemen)o(ts)i(receiv)o (ed)f(can)g(b)q(e)g(computed)g(from)f(the)h(receiv)o(e)g(bu\013er)75 1705 y(descriptor)g(and)e(the)g(length)i(\(in)e(b)o(ytes\))h(of)f(the)g (receiv)o(ed)h(message.)158 1751 y Ff(MPI)p 220 1751 V 13 w(STATUS)g Fl(and)j Ff(MPI)p 505 1751 V 13 w(QUERY)d Fl(are)i(not)h(go)q(o)q(d)g(names)g ({)g(they)f(can)h(b)q(e)g(easily)h(confused.)28 b(Leslie)18 b(Hart)e(suggests)75 1796 y Ff(MPI)p 137 1796 V 13 w(TEST)d Fl(or)i Ff(MPI)p 352 1796 V 13 w(CHECK)e Fl(for)h(the)h(\014rst.)23 b(P)o(erhaps)16 b(w)o(e)e(should)j(replace)f Ff(MPI)p 1229 1796 V 13 w(STATUS)c Fl(with)j Ff(MPI)p 1523 1796 V 13 w(TEST)e Fl(and)j Ff(MPI)p 1766 1796 V 13 w(QUERY)75 1842 y Fl(with)d Ff(MPI)p 224 1842 V 13 w(STATUS)p Fl(.)75 2040 y Fk(1.8.4)55 b(Multiple)20 b(Completions)75 2117 y Fo(It)c(is)g(con)o(v)o(enien)o(t)h(to)f (b)q(e)h(able)e(to)h(w)o(ait)g(for)g(the)g(completion)f(of)g(an)o(y)h(or)g (all)f(the)i(op)q(erations)f(in)g(a)g(set,)h(rather)75 2166 y(than)c(ha)o(ving)f(to)g(w)o(ait)g(for)h(a)g(sp)q(eci\014c)h(message.)j(A)c (call)g(to)f Fi(MPI)p 1096 2166 14 2 v 15 w(WAITANY)g Fo(or)h Fi(MPI)p 1393 2166 V 15 w(STATUSANY)e Fo(can)i(b)q(e)g(used)h(to)75 2216 y(w)o(ait)f(for)h(the)h(completion)d(of)h(one)i(out)f(of)f(sev)o(eral)i (op)q(erations;)e(a)h(call)f(to)h Fi(MPI)p 1340 2216 V 15 w(WAITALL)f Fo(can)h(b)q(e)h(used)g(to)f(w)o(ait)75 2266 y(for)g(all)e(p)q(ending)i(op)q (erations)g(in)g(a)f(list.)158 2351 y Fh(MPI)p 257 2351 15 2 v 17 w(W)-5 b(AIT)l(ANY)17 b(\()f(list)p 628 2351 V 15 w(of)p 682 2351 V 16 w(handles,)e(coun)o(t,)h(index,)f(return)p 1306 2351 V 15 w(handle\))158 2436 y Fo(Blo)q(c)o(ks)g(un)o(til)f(one)h(of)f(the)h (op)q(erations)g(asso)q(ciated)h(with)e(the)i(comm)o(uni)o(cation)c(handles)j (in)f(the)h(arra)o(y)g(has)75 2486 y(completed.)19 b(Returns)c(the)g(index)g (of)e(that)i(handle)f(in)g(the)h(arra)o(y)m(,)e(and)i(returns)h(the)f(status) g(of)f(that)g(op)q(eration)75 2536 y(in)f(the)i(ob)r(ject)g(asso)q(ciated)f (with)g(the)g(return)p 798 2536 13 2 v 16 w(handle.)k(The)d(parameters)f (are:)75 2623 y Fh(IN)i(list)p 215 2623 15 2 v 15 w(of)p 269 2623 V 17 w(handles)j Fo(list)13 b(of)g(handles)h(to)g(comm)o(unication)d(ob) r(jects.)75 2704 y Fh(IN)16 b(coun)o(t)j Fo(list)14 b(length)g(\(in)o (teger\))p eop %%Page: 20 21 bop 75 -100 a Fo(20)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fh(OUT)i(index)j Fo(index)14 b(of)f(handle)h(for)f(op)q(eration)h(that)g(completed)f(\(in)o(teger\).)75 129 y Fh(OUT)j(return)p 335 129 15 2 v 15 w(handle)i Fo(handle)d(that)g(is)g (asso)q(ciated)h(with)e(return)i(status)g(ob)r(ject.)22 b(Set)16 b(to)f(return)h(status)g(of)179 179 y(op)q(eration)e(that)f(completed.)158 271 y(The)h(successful)i(execution)e(of)f Fi(MPI)p 733 271 14 2 v 16 w(WAITANY\(list)p 1013 271 V 13 w(of)p 1070 271 V 15 w(handles,)20 b(index,)h(return)p 1566 271 V 14 w(handle\))12 b Fo(has)i(the)75 321 y(same)i(e\013ect)i(as)f(the)h(successful)g(execution)g (of)e Fi(MPI)p 922 321 V 15 w(WAIT\(handle[i],)j(return)p 1418 321 V 14 w(handle\))p Fo(,)d(where)i Fi(i)f Fo(is)f(the)75 371 y(v)n(alue)g(returned)i(b)o(y)e Fi(index)g Fo(and)g Fi(handle[i])f Fo(is)h(the)h Fi(i)p Fo(-th)f(handle)h(in)f(the)h(list,)f(and)g(the)h (cancellation)f(of)g(all)75 421 y(remaining)c(w)o(ait)h(op)q(erations.)158 471 y(If)i(more)e(then)j(one)f(op)q(eration)g(is)g(enabled)g(and)g(can)g (terminate,)f(one)h(is)g(arbitrarily)f(c)o(hosen)i(\(sub)r(ject)g(to)75 521 y(the)e(restrictions)i(on)d(op)q(eration)h(termination)e(order,)i(and)g (fairness,)g(see)h(Section)f(1.13\).)158 571 y Fi(MPI)p 227 571 V 15 w(WAITANY)20 b(\()i(list)p 548 571 V 15 w(of)p 607 571 V 15 w(handles,)e(count,)h(index,)f(return)p 1255 571 V 15 w(status)p 1402 571 V 14 w(handle\))13 b Fo(is)140 655 y Fi({MPI_WAIT)20 b(\(handle[0],)g(return_handle\);)f(index)h(=)i(0})f(||)h (...)75 705 y(||)140 755 y({MPI_WAIT)e(\(handle[count-1],)f(return_handle\);) f(index)j(=)h(count-1})158 838 y Fo(\(\\)p Fe(jj)p Fo(")13 b(indicates)h(c)o(hoice;)g(one)g(of)f(the)i(alternativ)o(es)f(is)f(c)o (hosen,)i(nondeterministically)m(.\))158 924 y Fh(MPI)p 257 924 15 2 v 17 w(ST)l(A)l(TUSANY)h(\()g(list)p 685 924 V 15 w(of)p 739 924 V 16 w(handles,)e(coun)o(t,)h(index,)f(return)p 1363 924 V 15 w(handle\))158 1009 y Fo(Causes)20 b(either)h(one)f(or)f(none)h (of)f(the)h(op)q(erations)g(asso)q(ciated)g(with)f(the)h(comm)o(unication)c (handles)k(to)75 1059 y(return.)f(In)13 b(the)h(former)d(case,)j(it)f(has)g (the)h(same)e(return)i(seman)o(tics)e(as)i(a)e(call)g(to)h Fi(MPI)p 1449 1059 14 2 v 15 w(WAIT)p 1552 1059 V 15 w(ANY)p Fo(.)f(In)h(the)h(latter)75 1109 y(case,)g(it)g(returns)h(a)f(v)n(alue)f(of)g ({1)h(in)f Fi(index)g Fo(and)h Fi(return)p 991 1109 V 14 w(handle)f Fo(is)h(unde\014ned.)19 b(The)14 b(parameters)g(are:)75 1201 y Fh(IN)i(list)p 215 1201 15 2 v 15 w(of)p 269 1201 V 17 w(handles)j Fo(list)13 b(of)g(handles)h(to)g(comm)o(unication)d(ob)r(jects.)75 1286 y Fh(IN)16 b(coun)o(t)j Fo(list)14 b(length)g(\(in)o(teger\))75 1370 y Fh(OUT)i(index)j Fo(index)14 b(of)f(handle)h(for)f(op)q(eration)h (that)g(completed,)f(or)h(-1)f(if)g(none)h(completed)g(\(in)o(teger\).)75 1454 y Fh(OUT)i(return)p 335 1454 V 15 w(handle)i Fo(handle)d(that)g(is)g (asso)q(ciated)h(with)e(return)i(status)g(ob)r(ject.)22 b(Set)16 b(to)f(return)h(status)g(of)179 1504 y(op)q(eration)e(that)f(completed,)g(if) g(an)o(y;)g(unde\014ned)i(when)g Fi(index)21 b(=)g(-1)p Fo(.)158 1631 y Fh(MPI)p 257 1631 V 17 w(W)-5 b(AIT)l(ALL\(list)p 580 1631 V 15 w(of)p 634 1631 V 17 w(handles,)13 b(coun)o(t,)i(list)p 1047 1631 V 15 w(of)p 1101 1631 V 17 w(return)p 1253 1631 V 15 w(handles\))158 1717 y Fo(Blo)q(c)o(ks)k(un)o(til)e(all)g(comm)o(unicati)o (on)e(op)q(erations)k(asso)q(ciated)g(with)f(handles)g(in)g(the)g(list)g (complete,)g(and)75 1767 y(return)d(the)g(status)f(of)g(all)e(these)j(op)q (erations.)k(The)14 b(parameters)g(are:)75 1859 y Fh(IN)i(list)p 215 1859 V 15 w(of)p 269 1859 V 17 w(handles)j Fo(list)13 b(of)g(handles)h (to)g(comm)o(unication)d(ob)r(jects.)75 1943 y Fh(IN)16 b(coun)o(t)j Fo(lists)14 b(length)g(\(in)o(teger\))75 2027 y Fh(OUT)i(list)p 266 2027 V 15 w(of)p 320 2027 V 16 w(return)p 471 2027 V 15 w(handles)j Fo(Must)h(ha)o(v)o(e)g(the)g(same)f(length)h(as)f(the)i(\014rst)f (list.)35 b(Eac)o(h)20 b(return)h(status)179 2077 y(ob)r(ject)14 b(is)g(set)h(to)f(the)g(return)h(status)g(of)e(the)i(corresp)q(onding)g(op)q (eration)e(in)h(the)g(\014rst)h(list.)158 2248 y Fm(Discussion:)158 2294 y Fl(The)9 b(fairness)i(requiremen)o(t)f(giv)o(en)h(in)f(Section)g(1.13) g(implies)h(that)f(the)f(use)h(of)f Ff(WAIT)p 1376 2294 12 2 v 12 w(ANY)f Fl(cannot)i(lead)g(to)g(starv)n(ation:)75 2340 y(If)15 b(the)i(sending)g(pro)q(cess)g(has)g(issued)g(a)f(send)h(complete)g (op)q(eration,)i(and)d(the)g(receiving)j(pro)q(cess)d(rep)q(eatedly)i(p)q (ost)f(a)75 2385 y(receiv)o(e)i(for)f(the)h(message)g(sen)o(t,)g(then)g(that) f(message)h(m)o(ust)f(b)q(e)h(ev)o(en)o(tually)i(receiv)o(ed.)34 b(Section)19 b(1.13)g(has)f(not)h(y)o(et)75 2431 y(b)q(een)14 b(discussed.)19 b(The)12 b(fairness)i(requiremen)o(t)h(can)e(b)q(e)g(attac)o (k)o(ed)h(either)g(for)e(b)q(eing)j(to)q(o)e(w)o(eak)g(\(ev)o(en)o(tually)i (is)f(not)f(go)q(o)q(d)75 2477 y(enough\),)g(or)f(to)q(o)g(strong)g(\(hard)h (to)f(implemen)o(t\).)18 b(In)12 b(the)g(later)g(case,)g(if)g(the)g (requiremen)o(t)i(of)d(fairness)i(is)g(dropp)q(ed)g(from)75 2522 y(MPI)g(implemen)o(tation)q(s,)j(then)d(some)h(mec)o(hanism)h(need)e(b)q (e)h(pro)o(vided)h(to)e(the)h(user)f(to)h(ac)o(hiev)o(e)g(fairness)h(b)o(y)e (his)h(or)g(her)75 2568 y(o)o(wn)g(devices.)22 b(One)14 b(suc)o(h)h(prop)q (osal)h(is)f(to)f(b)q(e)g(able)i(to)e(sp)q(ecify)h(a)f(rotating)h(priorit)o (y)h(order)f(on)g(the)f(op)q(erations)i(p)q(osted)75 2614 y(b)o(y)d(a)g Ff(MPI)p 222 2614 V 13 w(WAITANY)p Fl(:)d(searc)o(h)k(the)f(list)h(sequen)o (tially)n(,)h(starting)f(from)f(a)g(user)g(sp)q(eci\014ed)i(p)q(osition.)p eop %%Page: 21 22 bop 75 -100 a Fg(1.9.)31 b(BLOCKING)14 b(COMMUNICA)m(TION)1038 b Fo(21)75 45 y Fn(1.9)70 b(Blo)r(c)n(king)21 b(Comm)n(unication)75 140 y Fo(Blo)q(c)o(king)14 b(send)i(and)f(receiv)o(e)h(op)q(erations)g(com)o (bine)d(all)h(comm)o(unication)e(sub)q(op)q(erations)j(in)o(to)g(one)g(call.) 21 b(The)75 190 y(op)q(eration)16 b(returns)j(only)c(when)i(the)h(comm)o (unicati)o(on)c(completes)i(and)g(no)h(comm)o(unicatio)o(n)d(ob)r(ject)j(p)q (ersists)75 239 y(after)d(the)h(call)e(completed.)k(Ho)o(w)o(ev)o(er,)d(the)h (bu\013er)f(descriptor)i(ob)r(ject)e(needs)i(b)q(e)e(created)h(ahead)f(of)g (the)g(call.)158 291 y(W)m(e)g(use)g(the)h(follo)o(wing)c(naming)h(con)o(v)o (en)o(tion)h(for)h(suc)o(h)g(op)q(erations:)793 352 y Fa(2)793 427 y(4)844 386 y Fe(\000)842 435 y Fh(R)846 485 y(S)898 352 y Fa(3)898 427 y(5)933 377 y(\024)979 411 y Fh(SEND)976 460 y(RECV)1135 377 y Fa(\025)158 583 y Fo(The)h(\014rst)g(letter)g(\(v)o(oid,)e Fh(R)h Fo(or)g Fh(S)p Fo(\))g(indicates)g(the)h(start)g(mo)q(de)e (\(standard,)h(ready)m(,)g(or)g(secure\).)21 b(Only)14 b(t)o(w)o(o)75 633 y(of)f(the)i(com)o(binations)d(\(standard)i(and)g(secure\))i(are)e (meaningful)d(for)j(receiv)o(es.)158 720 y Fh(MPI)p 257 720 15 2 v 17 w(SEND)i(\(bu\013er)p 565 720 V 15 w(handle,)e(dest,)h(tag,)h(con)o (text\))c Fo(is)75 846 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h (dest,)i(tag,)g(context,)f(MPI_STANDARD,)75 896 y(MPI_EPHEMERAL\))75 946 y(MPI_START\(handle\))75 996 y(MPI_WAIT\(handle,)e(null\))158 1120 y Fh(MPI)p 257 1120 V 17 w(RSEND)e(\(bu\013er)p 601 1120 V 15 w(handle,)e(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1246 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,) f(MPI_READY,)g(MPI_EPHEMERAL\))75 1296 y(MPI_START\(handle\))75 1346 y(MPI_WAIT\(handle,)e(null\))158 1470 y Fh(MPI)p 257 1470 V 17 w(SSEND)e(\(bu\013er)p 592 1470 V 15 w(handle,)e(dest,)h(tag,)h(con)o (text\))11 b Fo(is)75 1597 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,)f(MPI_SECURE,)g(MPI_EPHEMERAL\))75 1647 y(MPI_START\(handle\))75 1696 y(MPI_WAIT\(handle,)e(null\))158 1821 y Fh(MPI)p 257 1821 V 17 w(RECV\(bu\013er)p 554 1821 V 16 w(handle,)c(source,)h(tag,)h(con)o(text,)f(return)p 1312 1821 V 15 w(handle\))c Fo(is)75 1947 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g(context,)f(MPI_STANDARD,)f (MPI_EPHEMERAL\))75 1997 y(MPI_START\(handle\))75 2047 y(MPI_WAIT\(handle,r)o (eturn)o(_hand)o(le\))158 2171 y Fh(MPI)p 257 2171 V 17 w(SRECV\(bu\013er)p 581 2171 V 16 w(handle,)14 b(source,)h(tag,)h(con)o(text,)e(return)p 1338 2171 V 15 w(handle\))e Fo(is)75 2298 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g(context,)f(MPI_SECURE,)f(MPI_EPHEMERAL\)) 75 2347 y(MPI_START\(handle\))75 2397 y(MPI_WAIT\(handle,r)o(eturn)o(_hand)o (le\))158 2565 y Fm(Implemen)o(tation)13 b(note:)158 2612 y Fl(While)20 b(these)f(functions)g(can)g(b)q(e)f(implemen)o(ted)i(via)f(calls) h(to)e(functions)i(that)e(implemen)o(t)i(sub)q(op)q(erations,)i(as)75 2658 y(describ)q(ed)16 b(in)g(this)f(subsection,)h(an)f(e\016cien)o(t)g (implemen)o(tation)j(ma)o(y)d(optimize)h(a)o(w)o(a)o(y)e(these)h(m)o(ultiple) i(calls,)f(pro)o(vided)75 2704 y(it)d(do)q(es)h(not)f(c)o(hange)h(the)f(b)q (eha)o(vior)i(of)e(correct)g(programs.)p eop %%Page: 22 23 bop 75 -100 a Fo(22)716 b Fg(CHAPTER)14 b(1.)27 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)158 45 y Fm(Discussion:)158 91 y Fl(W)m(e)h(use)g(a)g(di\013eren)o(t)h(function)h(name,)e(rather)g(than)g (an)h(additional)i(mo)q(de)d(parameter,)h(in)f(order)g(to)g(sa)o(v)o(e)g(on)g (an)75 136 y(additional)g(parameter)d(\(p)q(erformance)g(and)g(user)g(con)o (v)o(enience\).)18 b(Lyndon)13 b(and)f(p)q(erhaps)g(other)g(prefer)g(few)o (er)e(functions)75 182 y(and)k(an)f(additional)j(parameter.)75 401 y Fn(1.10)70 b(Non)n(blo)r(c)n(king)22 b(Comm)n(unication)75 492 y Fo(Non)o(blo)q(c)o(king)11 b(send)i(and)f(receiv)o(e)h(op)q(erations)f (com)o(bine)f(the)i(\014rst)g(t)o(w)o(o)e(sub)q(op)q(erations)i(\()p Fi(INIT)f Fo(and)g Fi(START)p Fo(\))f(in)o(to)75 542 y(one)16 b(call.)21 b(They)16 b(use)g(ephemeral)f(comm)o(uni)o(cation)d(ob)r(jects,)17 b(so)e(that)g(the)h(op)q(eration)f(is)h(completed,)e(and)h(the)75 592 y(asso)q(ciated)d(resources)i(are)e(freed,)h(b)o(y)e(using)h(one)f(of)g (the)i(functions)e Fi(MPI)p 1221 592 14 2 v 15 w(WAIT,)21 b(MPI)p 1433 592 V 15 w(STATUS,)g(MPI)p 1689 592 V 15 w(WAITANY,)75 641 y(MPI)p 144 641 V 15 w(STATUSANY)p Fo(,)12 b(or)i Fi(MPI)p 498 641 V 15 w(WAITALL)p Fo(.)e(Here,)j(to)q(o,)f(a)g(bu\013er)h(ob)r(ject)g (has)g(to)f(b)q(e)g(created)i(ahead)e(of)g(the)h(comm)o(u-)75 691 y(nication)e(initiation)f(op)q(eration.)158 741 y(W)m(e)h(use)h(the)h (same)d(naming)f(con)o(v)o(en)o(tion)j(as)f(for)g(blo)q(c)o(king)g(op)q (erations:)18 b(a)13 b(pre\014x)h(of)f Fh(R)g Fo(\()p Fh(S)p Fo(\))h(indicates)f(the)75 791 y Fi(READY)e Fo(\()p Fh(SECURE)p Fo(\))i(mo)q(de.)k(In)12 b(addition,)f(a)h(pre\014x)h(of)f Fh(I)g Fo(is)g(used)h(to)g(indicate)f Fj(imme)n(diate)g Fo(\(i.e.,)f(non)o (blo)q(c)o(king\))75 841 y(execution.)158 926 y Fh(MPI)p 257 926 15 2 v 17 w(ISEND)16 b(\(handle,)e(bu\013er)p 750 926 V 15 w(handle,)g(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1039 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g(context,) f(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1089 y(MPI_START\(handle\))158 1203 y Fh(MPI)p 257 1203 V 17 w(IRSEND)d(\(handle,)e(bu\013er)p 786 1203 V 15 w(handle,)g(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1316 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)f(MPI_READY,)g(MPI_EPHEMERAL\))75 1366 y(MPI_START\(handle\))158 1479 y Fh(MPI)p 257 1479 V 17 w(ISSEND)c(\(handle,)d(bu\013er)p 776 1479 V 16 w(handle,)h(dest,)h(tag,)h(con)o(text\))c Fo(is)75 1593 y Fi(MPI_INIT_SEND\(han)o(dle,)18 b(buffer_handle,)h(dest,)i(tag,)g (context,)f(MPI_SECURE,)g(MPI_EPHEMERAL\))75 1643 y(MPI_START\(handle\))158 1756 y Fh(MPI)p 257 1756 V 17 w(IRECV\(handle,)15 b(bu\013er)p 740 1756 V 15 w(handle,)f(source,)i(tag,)f(con)o(text,)g(return)p 1497 1756 V 15 w(status)p 1638 1756 V 16 w(handle\))c Fo(is)75 1869 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g (context,)f(MPI_STANDARD,)f(MPI_EPHEMERAL\))75 1919 y(MPI_START\(handle\))158 2033 y Fh(MPI)p 257 2033 V 17 w(ISRECV\(handle,)14 b(bu\013er)p 766 2033 V 16 w(handle,)g(source,)h(tag,)h(con)o(text,)e(return)p 1523 2033 V 15 w(status)p 1664 2033 V 16 w(handle\))d Fo(is)75 2146 y Fi(MPI_INIT_RECV\(han)o(dle,)18 b(buffer_handle,)h(source,)i(tag,)g (context,)f(MPI_SECURE,)f(MPI_EPHEMERAL\))75 2196 y(MPI_START\(handle\))75 2332 y Fn(1.11)70 b(Con)n(tiguous)23 b(Blo)r(c)n(k)f(Comm)n(unication)f(Op)r (erations)75 2423 y Fo(The)c(most)f(frequen)o(t)i(t)o(yp)q(e)g(of)e(bu\013er) i(used)g(is)f(a)g(bu\013er)h(with)f(one)g(con)o(tiguous)g(comp)q(onen)o(t.)27 b(W)m(e)16 b(sp)q(ecialize)75 2473 y(the)e(functions)f(in)f(the)i(t)o(w)o(o)e (previous)i(subsections)g(to)f(this)g(case,)h(th)o(us)f(a)o(v)o(oiding)e(the) j(need)g(for)f(the)g(creation)h(of)75 2523 y(a)h(bu\013er)i(descriptor)f(ob)r (ject.)24 b(W)m(e)15 b(use)h(the)g(same)e(naming)g(sc)o(heme)h(used)h(in)f (the)h(previous)g(subsections,)h(and)75 2573 y(app)q(end)d(a)g Fh(C)g Fo(in)g(the)g(function)g(name,)e(for)i Fi(CONTIGUOUS)p Fo(.)158 2658 y Fh(MPI)p 257 2658 V 17 w(SENDC)j(\(start,)d(coun)o(t,)h (datat)o(yp)q(e,)f(dest,)h(tag,)h(con)o(text\))c Fo(is)p eop %%Page: 23 24 bop 75 -100 a Fg(1.11.)26 b(CONTIGUOUS)14 b(BLOCK)h(COMMUNICA)m(TION)g(OPERA) m(TIONS)505 b Fo(23)75 45 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g (MPI_EPHEMERAL\))75 95 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g (datatype\))75 145 y(MPI_SEND)f(\(buffer_handle,)f(count,)h(dest,)h(tag,)g (context\))158 257 y Fh(MPI)p 257 257 15 2 v 17 w(RSENDC)16 b(\(handle,)e(start,)h(coun)o(t,)f(datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o (text\))c Fo(is)75 368 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g (MPI_EPHEMERAL\))75 418 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g (datatype\))75 468 y(MPI_RSEND\()f(buffer_handle,)f(count,)h(dest,)h(tag,)g (context\))158 580 y Fh(MPI)p 257 580 V 17 w(SSENDC)16 b(\(handle,)e(start,)h (coun)o(t,)f(datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))c Fo(is)75 691 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g (MPI_EPHEMERAL\))75 741 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g (datatype\))75 791 y(MPI_SSEND\()f(buffer_handle,)f(count,)h(dest,)h(tag,)g (context\))158 903 y Fh(MPI)p 257 903 V 17 w(RECV)o(C)f(\(start,)e(coun)o(t,) h(datat)o(yp)q(e,)f(source,)h(tag,)g(con)o(text,)g(return)p 1565 903 V 15 w(status)p 1706 903 V 16 w(handle\))75 953 y Fo(is)75 1064 y Fi(MPI_CREATE_BUFFER)o(\()g(buffer_handle,)g(MPI_EPHEMERAL\)) 75 1114 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 1164 y(MPI_RECV\()f(buffer_handle,)f(source,)h(tag,)h(context,)f (return_status_handl)o(e\))158 1276 y Fh(MPI)p 257 1276 V 17 w(SRECV)o(C)c(\(start,)f(coun)o(t,)f(datat)o(yp)q(e,)h(source,)g(tag,)g(con)o (text,)g(return)p 1565 1276 V 15 w(status)p 1706 1276 V 16 w(handle\))75 1326 y Fo(is)75 1437 y Fi(MPI_CREATE_BUFFER)o(\()k (buffer_handle,)g(MPI_EPHEMERAL\))75 1487 y(MPI_APPEND\()h(buffer_handle,)e (start,)j(count,)g(datatype\))75 1537 y(MPI_SRECV\()f(buffer_handle,)f (source,)h(tag,)h(context,)f(return_status_hand)o(le\))158 1649 y Fh(MPI)p 257 1649 V 17 w(ISENDC)d(\(handle,)c(start,)i(coun)o(t,)g (datat)o(yp)q(e,)g(dest,)g(tag,)g(con)o(text\))d Fo(is)75 1761 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 1810 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 1860 y(MPI_ISEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)g(context\))158 1972 y Fh(MPI)p 257 1972 V 17 w(IRSENDC)16 b(\(handle,)e(start,)h(coun)o(t,)g (datat)o(yp)q(e,)f(dest,)h(tag,)h(con)o(text\))c Fo(is)75 2084 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 2134 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 2183 y(MPI_IRSEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)f(context\))158 2295 y Fh(MPI)p 257 2295 V 17 w(ISSENDC)c(\(handle,)e(start,)h(coun)o(t,)f (datat)o(yp)q(e,)h(dest,)g(tag,)h(con)o(text\))c Fo(is)75 2407 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 2457 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 2507 y(MPI_ISSEND\()f(handle,)g(buffer_handle,)f(dest,)i(tag,)f(context\))158 2618 y Fh(MPI)p 257 2618 V 17 w(IRECV)o(C\(handle,)11 b(start,)f(coun)o(t,)h (datat)o(yp)q(e,)f(source,)h(tag,)h(con)o(text,)e(return)p 1679 2618 V 15 w(status)p 1820 2618 V 16 w(handle\))75 2668 y Fo(is)p eop %%Page: 24 25 bop 75 -100 a Fo(24)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fi(MPI_CREATE_BUFFER)k (\(buffer_handle,)h(MPI_EPHEMERAL\))75 95 y(MPI_APPEND\()h(buffer_handle,)e (start,)j(count,)g(datatype\))75 145 y(MPI_IRECV\()f(handle,)g (buffer_handle,)f(source,)h(tag,)h(context\))158 266 y Fh(MPI)p 257 266 15 2 v 17 w(ISRECV)o(C\(handle,)10 b(start,)h(coun)o(t,)f(datat)o(yp) q(e,)h(source,)g(tag,)g(con)o(text,)g(return)p 1706 266 V 15 w(status)p 1847 266 V 15 w(handle\))75 316 y Fo(is)75 439 y Fi(MPI_CREATE_BUFFER)o(\()19 b(buffer_handle,)g(MPI_EPHEMERAL\))75 489 y(MPI_APPEND\()h(buffer_handle,)e(start,)j(count,)g(datatype\))75 539 y(MPI_ISRECV\()f(handle,)g(buffer_handle,)f(source,)h(tag,)h(context\))75 682 y Fn(1.12)70 b(Prob)r(e)23 b(and)h(Cancel)75 775 y Fo(The)18 b Fi(MPI)p 233 775 14 2 v 15 w(PROBE)e Fo(op)q(eration)h(allo)o(ws)f (incoming)f(messages)i(to)g(b)q(e)h(c)o(hec)o(k)o(ed)g(for,)g(without)e (actually)h(receiving)75 825 y(them.)22 b(The)16 b(user)g(can)g(then)g (decide)h(where)f(to)g(receiv)o(e)g(them,)f(based)h(on)f(the)h(information)d (returned)k(b)o(y)e(the)75 874 y(prob)q(e)i(\(basically)m(,)f(the)h (information)d(on)i(the)i(message)e(en)o(v)o(elop)q(e\).)27 b(An)17 b(additional)e(function,)h Fi(MPI)p 1716 874 V 16 w(GET)p 1798 874 V 15 w(LEN)75 924 y Fo(allo)o(ws)11 b(the)j(amoun)o(t)c(of)i (storage)h(needed)i(to)d(receiv)o(e)i(the)f(message)g(to)f(b)q(e)h(computed,) f(when)h(this)g(length)g(is)f(not)75 974 y(readily)h(computed)h(from)e(the)i (information)d(returned)16 b(b)o(y)e Fi(MPI)p 1094 974 V 15 w(PROBE)p Fo(.)158 1025 y(The)f Fi(MPI)p 311 1025 V 15 w(CANCEL)e Fo(op)q(eration)h(allo)o(ws)f(p)q(ending)i(comm)o(unicatio)o(ns)d(to)i(b)q(e) h(cancelled.)19 b(This)12 b(is)g(required)h(for)75 1075 y(clean)o(up.)20 b(P)o(osting)14 b(a)h(send)g(or)g(a)f(receiv)o(e)i(ties)f(user)g(resources)i (\(send)f(or)e(receiv)o(e)i(bu\013ers\),)g(and)e(a)h(cancel)g(ma)o(y)75 1125 y(b)q(e)g(needed)g(to)f(free)g(these)i(resources)g(gracefully)m(.)158 1211 y Fh(MPI)p 257 1211 15 2 v 17 w(PR)o(OBE\()f(source,)g(tag,)h(con)o (text,)f(\015ag,)g(datat)o(yp)q(e,)g(return)p 1367 1211 V 15 w(status\))75 1342 y(IN)h(source)k Fo(rank)14 b(in)f(con)o(text)i(of)e (source,)i(or)f Fi(MPI)p 897 1342 14 2 v 15 w(ANY)p 978 1342 V 15 w(SOURCE)f Fo(\(in)o(teger\).)75 1429 y Fh(IN)j(tag)21 b Fo(user)15 b(tag)e(for)h(messages)g(receiv)o(ed)h(with)f(this)f(handle,)h (or)g Fi(MPI)p 1232 1429 V 15 w(ANY)p 1313 1429 V 15 w(TAG)f Fo(\(in)o(teger\).)75 1517 y Fh(IN)j(con)o(text)k Fo(con)o(text)14 b(of)f(messages)h(receiv)o(ed)i(with)d(this)h(handle.)75 1604 y Fh(OUT)i(\015ag)k Fo(\(logical\))75 1692 y Fh(IN)c(t)o(yp)q(e)k Fo(assumed)14 b(t)o(yp)q(e)g(of)f(data)h(in)f(message)h(\(status)h(v)n (ariable\).)75 1779 y Fh(OUT)h(return)p 335 1779 15 2 v 15 w(handle)i Fo(handle)c(that)g(is)g(asso)q(ciated)g(with)g(return)h(status)g (ob)r(ject.)158 1875 y Fi(MPI)p 227 1875 14 2 v 15 w(PROBE)h Fo(returns)i Fi(flag)j(=)g(true)16 b Fo(if)f(there)j(is)f(a)f(message)g(that) h(can)f(b)q(e)i(receiv)o(ed)f(and)g(that)f(matc)o(hes)75 1924 y(the)21 b(pattern)h(sp)q(eci\014ed)g(b)o(y)e(the)h(parameters)g Fi(source)p Fo(,)g Fi(tag)p Fo(,)g(and)f Fi(context)p Fo(.)37 b(It)21 b(returns)h Fi(flag)f(=)h(false)p Fo(,)75 1974 y(otherwise.)38 b(If)19 b Fi(MPI)p 411 1974 V 16 w(PROBE)g Fo(returns)i Fi(flag)g(=)h(true)p Fo(,)e(then)h(the)g(length,)g(source)g(and)f(tag)g(of)g(the)g(message)75 2024 y(matc)o(hed)13 b(are)i(returned)h(in)d(the)i(return)g(status)g(ob)r (ject.)20 b(These)15 b(are)g(the)f(same)g(v)n(alues)f(that)i(w)o(ould)e(ha)o (v)o(e)h(b)q(een)75 2074 y(returned)k(b)o(y)d(a)h(call)f(to)h Fi(MPI)p 542 2074 V 15 w(RECV)f Fo(or)h(to)g Fi(MPI)p 832 2074 V 15 w(SRECV)f Fo(executed)j(at)e(the)g(same)f(p)q(oin)o(t)h(in)f(the)i (program)d(\(with)75 2124 y(a)g(ca)o(v)o(eat)g(concerning)h(length;)f(see)i (b)q(elo)o(w\).)j(These)c(v)n(alues)f(can)h(b)q(e)g(deco)q(ded)g(from)e(the)i (return)g(status)g(ob)r(ject)75 2173 y(using)f(the)g Fi(MPI)p 324 2173 V 15 w(RETURN)p 471 2173 V 15 w(STAT)f Fo(function.)18 b(The)c(v)n(alue)f(returned)j(in)e(the)g(return)h(status)g(ob)r(ject)g(is)f (unde\014ned)h(if)75 2223 y Fi(flag=false)p Fo(.)158 2274 y(The)j(length)g(v) n(alue)f(returned)i(b)o(y)e(the)i(return)f(status)h(ob)r(ject)f(is)g (\(correctly\))h(the)f(n)o(um)o(b)q(er)f(of)g(elemen)o(ts)75 2324 y(in)h(the)h(message,)f(pro)o(vided)h(that)f(all)f(elemen)o(ts)h(in)g (the)h(message)f(are)h(of)e(the)i(t)o(yp)q(e)g(sp)q(eci\014ed)h(b)o(y)e(the)h Fi(type)75 2374 y Fo(parameter;)13 b(otherwise)i(the)f(length)g(v)n(alue)f (returned)j(is)e(unde\014ned.)158 2425 y(A)g(subsequen)o(t)j(receiv)o(e)e (executed)h(with)e(the)h(same)f(con)o(text,)g(and)h(the)g(source)g(and)f(tag) g(returned)i(b)o(y)e(the)75 2475 y(call)h(to)g Fi(MPI)p 274 2475 V 15 w(PROBE)f Fo(will)g(receiv)o(e)i(the)g(message)f(that)h(w)o(as)f (matc)o(hed)g(b)o(y)g(the)h(prob)q(e,)g(if)e(no)h(other)h(in)o(terv)o(ening) 75 2524 y(receiv)o(e)c(o)q(ccurred)g(after)f(the)g(prob)q(e.)18 b(If)10 b(the)h(receiving)g(pro)q(cess)i(is)d(m)o(ultithreaded,)g(it)g(is)h (the)g(user)g(resp)q(onsibilit)o(y)75 2574 y(to)j(ensure)h(that)f(the)h(last) e(condition)g(holds.)158 2704 y Fm(Discussion:)p eop %%Page: 25 26 bop 75 -100 a Fg(1.12.)31 b(PR)o(OBE)14 b(AND)g(CANCEL)1195 b Fo(25)158 45 y Fl(Do)12 b(w)o(e)f(w)o(an)o(t)g Ff(MPI)p 427 45 12 2 v 13 w(PROBE)e Fl(to)i(return)h(mo)q(de)g(\(standard)h(or)e (secure\)?)18 b(If)10 b(y)o(es,)i(w)o(e)f(need)h(to)f(carry)h(mo)q(de)g (information)75 91 y(with)j(messages)g(\(it's)f(deja)h(vu)g(all)g(o)o(v)o(er) g(again\).)22 b(If)14 b(not,)g(then)h(it)g(is)g(up)g(to)f(the)g(user)h(to)f (enco)q(de)h(this)h(information)g(\(in)75 136 y(the)d(tag\))g(so)g(that)h (receiv)o(er)g(can)f(decide)h(whether)g(to)f(use)g(a)g(secure)g(or)g (standard)i(send.)158 182 y(MPI)10 b(guaran)o(tees)h(that)e(successiv)o(e)j (messages)e(sen)o(t)g(from)f(a)h(source)g(to)g(a)g(destination)i(within)f (the)f(same)g(con)o(text)g(are)75 228 y(receiv)o(ed)i(in)f(the)g(order)g (they)g(are)f(sen)o(t.)17 b(Th)o(us,)11 b(MPI)f(m)o(ust)h(supp)q(ort,)h (either)f(explicitly)j(or)c(implicitl)q(y)m(,)k(a)c(FIF)o(O)g(structure)75 273 y(to)i(manage)h(messages)h(b)q(et)o(w)o(een)e(eac)o(h)h(pair)g(of)f (messages,)h(for)f(eac)o(h)h(con)o(text.)k Ff(MPI)p 1308 273 V 13 w(PROBE)11 b Fl(returns)i(information)h(on)f(the)75 319 y(\014rst)i(matc)o(hing)g(message)g(in)h(this)f(FIF)o(O;)e(this)j(will)g (also)f(b)q(e)g(the)f(message)h(receiv)o(ed)h(b)o(y)f(the)f(\014rst)h (subsequen)o(t)h(receiv)o(e)75 365 y(with)d(the)h(same)f(source,)g(tag)g(and) h(con)o(text)f(as)h(the)f(message)g(matc)o(hed)h(b)o(y)f Ff(MPI)p 1261 365 V 13 w(PROBE)p Fl(.)158 410 y(Message)19 b(passing)g(in)g(MPI)e(can) i(b)q(e)f(implemen)o(ted)i(without)e(app)q(ending)j(t)o(yp)q(e)d(information) i(to)d(messages.)32 b(A)75 456 y(message)14 b(is)g(merely)h(a)f(string)g(of)g (b)o(ytes)g(and)g(the)g(in)o(terpretation)i(of)d(these)h(b)o(ytes)g(in)o(to)h (a)e(sequence)i(of)e(t)o(yp)q(ed)i(elemen)o(ts)75 502 y(is)f(done)f(using)i (the)e(information)i(in)e(the)g(bu\013er)h(descriptors)h(at)e(eac)o(h)g(end.) 18 b(The)13 b(abilit)o(y)i(to)e(use)g(suc)o(h)h(implemen)o(tation)75 547 y(strategy)k(is)f(deemed)h(to)f(b)q(e)g(an)g(imp)q(ortan)o(t)i(goal.)30 b(In)17 b(suc)o(h)g(implemen)o(tation)q(,)j(when)d(a)g(message)h(arriv)o(es,) h(it)e(is)h(not)75 593 y(b)q(e)g(kno)o(wn)g(ho)o(w)f(man)o(y)h(elemen)o(ts)h (it)f(con)o(tains,)i(or)d(ev)o(en)h(ho)o(w)g(m)o(uc)o(h)g(storage)g(is)g (needed)h(to)e(receiv)o(e)h(that)g(message)75 639 y(\(b)q(ecause)13 b(of)f(p)q(ossible)j(represen)o(tation)f(con)o(v)o(ersion)g(in)f(a)g (heterogeneous)h(en)o(vironmen)o(t\).)k(The)12 b(prob)q(e)h(function)h (cannot)75 684 y(use)h(a)g(bu\013er)g(descriptor;)i(this)f(defeats)f(the)g (purp)q(ose)h(of)e(probing)j(in)e(order)g(to)g(decide)h(where)f(to)f(receiv)o (e)i(a)e(message.)75 730 y(Therefore,)d(prob)q(e)h(cannot,)g(in)f(general,)i (return)e(correct)g(length)h(information.)19 b(Still,)13 b(it)e(is)g(often)g (the)g(case)g(that)g(prob)q(e)h(is)75 776 y(used)g(to)f(decide)i(ho)o(w)e(m)o (uc)o(h)g(storage)h(to)f(allo)q(cate)i(in)f(order)g(to)f(receiv)o(e)h(a)f (message.)17 b(Enco)q(ding)d(suc)o(h)d(information)j(in)e(the)75 821 y(message)i(tag)f(is)g(deemed)h(to)f(b)q(e)g(to)q(o)g(a)o(wkw)o(ard,)g (and)g(it)g(is)h(deemed)g(imp)q(ortan)o(t)g(for)f Ff(MPI)p 1396 821 V 13 w(PROBE)d Fl(to)j(return)h(some)f(useful)75 867 y(size)k(information.)28 b(The)16 b(curren)o(t)h(de\014nition)i(of)c Ff(MPI)p 884 867 V 13 w(PROBE)f Fl(is)j(a)f(compromise)i(b)q(et)o(w)o(een)e (these)h(t)o(w)o(o)e(goals.)28 b(F)m(or)16 b(the)75 913 y(most)f(common)g (case)g(of)f(messages)h(where)g(all)h(en)o(tries)f(ha)o(v)o(e)g(the)g(same)g (t)o(yp)q(e)g Ff(MPI)p 1316 913 V 12 w(PROBE)e Fl(returns)i(the)g(correct)f (length)75 958 y(information;)20 b(the)d(more)g(esoteric)g(case)g(is)g (handled)i(b)o(y)e(the)g Ff(MPI)p 1076 958 V 13 w(GET)p 1149 958 V 12 w(LEN)f Fl(that)h(is)g(describ)q(ed)h(b)q(elo)o(w.)29 b(The)16 b(curren)o(t)75 1004 y(solution)g(sa)o(v)o(es)e(us)g(the)f(need)h (for)g(one)g(additional)i(w)o(ord)e(p)q(er)g(message)g(that)g(w)o(ould)g (otherwise)h(b)q(e)f(needed)g(to)g(transfer)75 1050 y(the)f(message)h(length) g(\(in)g(elemen)o(ts\))g(with)f(the)g(message.)158 1095 y(There)g(w)o(as)g(a) g(request)h(for)e(a)h(blo)q(c)o(king)j(prob)q(e)e(function.)158 1263 y Fh(MPI)p 257 1263 15 2 v 17 w(GET)p 376 1263 V 18 w(LEN\()h(coun)o(t,) g(return)p 805 1263 V 15 w(status,)g(bu\013er)p 1098 1263 V 15 w(descriptor\))158 1349 y Fo(Computes)g(the)i(n)o(um)o(b)q(er)e(of)h (elemen)o(ts)g(that)g(w)o(ere)h(to)f(b)q(e)g(receiv)o(ed,)i(if)d(the)i (message)e(that)h(is)g(asso)q(ciated)75 1398 y(with)e(the)i(return)f(status)h (handle)f(w)o(ould)e(b)q(e)j(receiv)o(ed)g(in)e(the)h(bu\013er)h(asso)q (ciated)f(with)g(the)g(bu\013er)h(descriptor)75 1448 y(handle.)75 1529 y Fh(OUT)g(coun)o(t)j Fo(n)o(um)o(b)q(er)13 b(of)g(elemen)o(ts)h(that)g (w)o(ere)h(to)f(b)q(e)g(receiv)o(ed)h(\(in)o(teger\))75 1612 y Fh(IN)h(return)p 284 1612 V 15 w(status)k Fo(handle)13 b(to)h(return)h (status)g(descriptor)75 1694 y Fh(IN)h(bu\013er)p 273 1694 V 16 w(descriptor)i Fo(handle)13 b(to)h(bu\013er)h(descriptor)158 1854 y Fm(Discussion:)158 1899 y Fl(In)d(order)h(to)g(supp)q(ort)g(this)h (function,)f(an)g(additional)j(\014eld)d(is)g(needed)h(in)f(the)g(return)g (status)f(ob)r(ject,)h(i.e.,)f(n)o(um)o(b)q(er)75 1945 y(of)17 b(b)o(ytes)g(in)h(the)f(incoming)j(message)d(matc)o(hed)h(b)o(y)f(prob)q(e,)i (or)e(v)n(alue)h(of)f(datat)o(yp)q(e)h(parameter)g(pro)o(vided)h(b)o(y)e (prob)q(e)75 1991 y(\(unless)d(w)o(e)f(add)h(this)g(parameter)f(to)g(the)g Ff(MPI)p 768 1991 12 2 v 13 w(GET)p 841 1991 V 13 w(LEN)f Fl(function,)i (whic)o(h)g(is)f(to)q(o)h(ugly)g(to)f(b)q(ear\).)158 2159 y Fh(MPI)p 257 2159 15 2 v 17 w(CANCEL\()k(handle,)d(\015ag\))75 2283 y(IN)i(handle)j Fo(handle)14 b(to)g(comm)o(uni)o(cation)d(ob)r(ject)75 2365 y Fh(OUT)16 b(\015ag)k Fo(\(logical\))158 2455 y(A)c(call)f(to)h Fi(MPI)p 405 2455 14 2 v 16 w(CANCEL)e Fo(cancels)j(a)f(p)q(ending)g(comm)o (unication)d(op)q(eration)j(\(send)h(or)f(receiv)o(e\).)26 b(The)17 b(call)75 2504 y(returns)12 b Fi(flag)21 b(=)g(true)9 b Fo(if)h(the)g(cancel)h(op)q(eration)f(succeeded,)j Fi(flag)21 b(=)h(false)9 b Fo(otherwise.)17 b(It)11 b(m)o(ust)e(b)q(e)h(the)h(case)75 2554 y(that)16 b(either)g(the)g(cancel)h(op)q(eration)e(succeeds)j(or)e(that) g(the)g(p)q(ending)g(comm)o(uni)o(cation)d(op)q(eration)i(completes)75 2604 y(\(but)k(not)g(b)q(oth\).)33 b(If)19 b(a)f(comm)o(unication)e(op)q (eration)i(w)o(as)h(cancelled)h(successfully)m(,)g(the)g(e\013ect)g(is)f(as)g (if)f(this)75 2654 y(op)q(eration)12 b(w)o(as)h(nev)o(er)g(executed.)20 b(If)12 b(a)g(send)h(w)o(as)g(cancelled)g(successfully)g(then)g(the)h (message)e(sen)o(t)h(will)e(not)h(b)q(e)75 2704 y(receiv)o(ed,)j(the)f (receiv)o(e)h(bu\013er)g(of)e(an)o(y)g(p)q(osted)i(receiv)o(e)g(for)e(that)h (message)f(will)g(not)g(b)q(e)i(altered,)e(and)h(an)o(y)f(suc)o(h)p eop %%Page: 26 27 bop 75 -100 a Fo(26)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fo(receiv)o(e)i(has)g(to)f(b)q(e)h (satis\014ed)g(b)o(y)f(another)g(send;)h(if)f(a)g(receiv)o(e)h(w)o(as)f (cancelled)h(successfully)m(,)g(then)g(the)g(receiv)o(e)75 95 y(bu\013er)f(p)q(osted)g(will)d(not)i(b)q(e)h(altered,)f(and)f(an)o(y)h (send)h(matc)o(hing)d(this)i(receiv)o(e)h(has)f(to)g(b)q(e)g(satis\014ed)h(b) o(y)f(another)75 145 y(receiv)o(e.)158 276 y Fm(Implemen)o(tation)f(note:)158 324 y Fl(There)g(is)h(not)f(exp)q(ectation)i(that)e(a)g(cancel)h(op)q (eration)h(will)g(b)q(e)e(fast.)75 559 y Fn(1.13)70 b(Correctness)75 737 y Fm(Discussion:)18 b Fl(The)13 b(material)h(in)g(this)f(section)h(has)g (not)f(y)o(et)f(b)q(een)i(discussed)h(b)o(y)e(MPIF.)f(Some)i(or)e(all)j(of)d (it)h(is)h(lik)o(ely)h(to)75 787 y(mo)o(v)o(e)e(to)g(Section)i Fm(??)p Fl(.)h(It)d(is)g(incorp)q(orated)i(here)f(for)f(completeness.)75 1001 y Fk(1.13.1)55 b(Order)75 1083 y Fo(MPI)15 b(preserv)o(es)i(the)e(order) g(of)f(messages)h(within)f(the)h(same)f(con)o(text,)h(b)q(et)o(w)o(een)h(an)o (y)e(\014xed)h(pair)f(of)g(pro)q(cesses.)75 1132 y(In)19 b(other)g(w)o(ords,) h(if)e(pro)q(cess)j(A)e(executes)i(t)o(w)o(o)e(successiv)o(e)i(send)f Fi(start)d Fo(sub)q(op)q(erations)j(within)e(the)i(same)75 1182 y(con)o(text,)13 b(pro)q(cess)i(B)d(executes)j(t)o(w)o(o)d(successiv)o (e)j(receiv)o(e)f Fi(start)e Fo(op)q(erations,)g(and)h(b)q(oth)f(receiv)o(es) j(matc)o(h)c(either)75 1232 y(sends,)17 b(then)f(the)g(\014rst)g(receiv)o(e)h (will)d(receiv)o(e)j(the)f(message)f(sen)o(t)h(b)o(y)f(the)h(\014rst)h(send,) f(and)f(the)h(second)h(receiv)o(e)75 1282 y(will)12 b(receiv)o(e)i(the)g (message)f(sen)o(t)h(b)o(y)e(the)i(second)g(send.)19 b(Th)o(us,)13 b(if)f(a)h(t)o(w)o(o)g(messages)g(from)e(the)j(same)e(source)j(can)75 1332 y(satisfy)h(a)g(p)q(ending)h(receiv)o(e,)h(the)f(\014rst)g(message)f (sen)o(t)h(is)f(accepted;)j(if)d(a)g(message)g(can)h(satisfy)f(t)o(w)o(o)g(p) q(ending)75 1381 y(receiv)o(es,)f(the)g(\014rst)f(receiv)o(e)h(p)q(osted)g (is)f(satis\014ed.)158 1434 y(The)h(last)f(paragraph)f(assumes)h(that)h(the)f (send)i Fi(start)d Fo(op)q(erations)h(are)h(ordered)g(b)o(y)f(the)h(program)d (order)75 1484 y(at)i(pro)q(cess)h(A,)f(and)f(the)i(receiv)o(e)g Fi(start)e Fo(op)q(erations)h(are)g(ordered)h(b)o(y)f(the)g(program)e(order)j (at)e(pro)q(cess)j(B.)e(If)f(a)75 1534 y(pro)q(cess)i(is)f(m)o(ultithreaded)e (and)h(the)h(op)q(erations)g(are)g(executed)h(b)o(y)f(distinct)f(threads,)h (then)h(the)f(seman)o(tics)f(of)75 1583 y(the)e(threaded)h(system)f(ma)o(y)d (not)j(de\014ne)h(an)e(order)h(b)q(et)o(w)o(een)h(the)g(t)o(w)o(o)e(op)q (erations,)h(in)f(whic)o(h)h(case)g(the)h(condition)75 1633 y(is)i(v)o(oid.)75 1764 y Fk(1.13.2)55 b(Progress)19 b(and)g(F)-5 b(airness)75 1846 y Fo(W)m(e)15 b(can)i(mo)q(del)d(the)i(execution)h(of)e (MPI)h(programs)f(as)h(an)g(in)o(teraction)f(b)q(et)o(w)o(een)j(executing)e (pro)q(cesses)j(that)75 1896 y(execute)e(eac)o(h)e(their)h(o)o(wn)e(program,) f(and)i(the)g Fh(comm)o(unication)f(subsystem)p Fo(.)19 b(The)c(comm)o (unication)d(sub-)75 1945 y(system)i(ma)o(y)e(ha)o(v)o(e)h(v)n(arious)h (constrain)o(ts)g(on)g(the)g(amoun)o(t)e(of)i(resources)i(it)d(can)h(use.)19 b(E.g.:)158 1998 y(Bounds)f(on)f(the)g(n)o(um)o(b)q(er)f(and)h(total)g(sizes) h(of)e(activ)o(e)h(comm)o(unication)d(ob)r(jects.)28 b(Suc)o(h)18 b(b)q(ound)f(can)g(b)q(e)75 2048 y(global,)12 b(p)q(er)j(no)q(de,)e(or)h(p)q (er)h(pair)e(of)h(comm)o(unicati)o(ng)d(no)q(des.)158 2100 y(Bounds)18 b(on)e(the)i(n)o(um)o(b)q(er)e(and)h(total)f(sizes)i(of)f (messages)g(bu\013ered)h(in)f(the)h(system.)27 b(Suc)o(h)17 b(b)q(ound)g(can,)75 2150 y(again,)h(b)q(e)h(global,)e(p)q(er)j(no)q(de,)f (or)g(p)q(er)g(pair)f(of)g(comm)o(unicating)d(no)q(de.)32 b(In)19 b(addition,)f(a)g(message)g(ma)o(y)e(b)q(e)75 2200 y(bu\013ered)f(at)f(the)h (sender,)g(at)e(the)i(receiv)o(er,)g(at)f(b)q(oth,)f(or)h(p)q(erhaps)h(at)f (another)g(place)g(altogether.)158 2253 y(Th)o(us,)f(it)g(will)e(b)q(e)j (di\016cult)e(to)h(set)h(rules)f(on)g(resource)i(managemen)o(t)c(of)h(the)i (comm)o(uni)o(cation)c(subsystem.)75 2302 y(Ho)o(w)o(ev)o(er,)i(it)f(is)g (generally)g(exp)q(ected)j(that)d(implemen)o(ters)f(will)g(pro)o(vide)h (information)e(on)i(the)h(mec)o(hanism)d(used)75 2352 y(for)j(resource)i (allo)q(cation,)c(and)i(that)h(query)f(and)g(set)h(functions)g(will)d(allo)o (w)h(to)h(query)h(and)f(p)q(ossibly)f(con)o(trol)h(the)75 2402 y(amoun)o(t)g(of)h(a)o(v)n(ailable)f(resources.)158 2455 y(W)m(e)18 b(pro)o(vide)g(in)g(this)g(section)i(a)e(set)h(of)f(minim)o(al)c(requiremen)o (ts)19 b(on)f(the)h(comm)o(unication)c(subsystem.)75 2504 y(Programs)f(that)h (execute)h(on)f(an)o(y)f(subsystem)h(that)g(ful\014ls)f(these)j(minim)o(al)11 b(requiremen)o(ts)k(are)g Fh(safe)g Fo(and)f(will)75 2554 y(p)q(ort)i(to)g (an)o(y)f(MPI)h(implemen)o(tation.)21 b Fh(Unsafe)15 b Fo(programs)f(ma)o(y)g (execute)k(on)d(some)g(MPI)i(implem)o(en)o(tations,)75 2604 y(dep)q(ending)e(on)f(the)h(amoun)o(t)e(of)g(a)o(v)n(ailable)f(resources)17 b(and)d(the)h(implemen)o(tation)c(used)k(for)f(the)h(MPI)g(comm)o(u-)75 2654 y(nication)f(subsystem.)20 b(Finally)13 b Fh(erroneous)f Fo(programs)h(nev)o(er)i(execute)i(correctly)m(.)j(\(While)14 b(it)g(is)g(desirable)h(to)75 2704 y(detect)i(erroneous)f(programs,)e(it)h (is)g(not)g(p)q(ossible)g(to)g(do)g(so)h(at)f(compile)e(time,)h(and)h(often)g (prohibitiv)o(e)f(to)h(do)p eop %%Page: 27 28 bop 75 -100 a Fg(1.13.)26 b(CORRECTNESS)1328 b Fo(27)75 45 y(so)16 b(a)g(run)g(time.)22 b(Th)o(us,)17 b(the)f(do)q(cumen)o(t)f(do)q(es)i (not)f(sp)q(ecify)g(a)g(b)q(eha)o(vior)f(for)h(erroneous)h(programs,)e (although)75 95 y(the)f(desired)i(b)q(eha)o(vior)d(is)h(to)g(return)h(a)e (useful)h(error)h(message.\))126 186 y(1.)20 b(If)13 b(a)g(pro)q(cess)i (executes)h(an)d Fi(INIT)g Fo(op)q(eration,)g(then)h(the)g(op)q(eration)f(ev) o(en)o(tually)g(succeeds,)j(or)d(a)h Fj(r)n(esour)n(c)n(e)179 236 y(exc)n(eption)20 b Fo(o)q(ccurs.)38 b(The)20 b(standard)h(do)q(es)g(not) e(sp)q(ecify)i(when)f(a)g(resource)i(exception)f(is)e(allo)o(w)o(ed)g(to)179 286 y(o)q(ccur.)33 b(It)18 b(is)h(exp)q(ected)h(that)f(an)f(op)q(erational)f (de\014nition)i(will)d(b)q(e)k(made)d(a)o(v)n(ailable,)g(in)h(the)h(form)e (of)179 336 y(test)e(programs)d(that)i(ha)o(v)o(e)f(to)h(execute)h(with)f(no) f(resource)j(exceptions.)j(It)14 b(is)f(highly)g(desirable)h(to)f(ha)o(v)o(e) 179 386 y(generous)18 b(b)q(ounds)g(on)f(the)h(n)o(um)o(b)q(er)f(of)f (concurren)o(tly)j(activ)o(e)e(comm)o(unication)d(ob)r(jects)k(eac)o(h)g(pro) q(cess)179 435 y(ma)o(y)12 b(ha)o(v)o(e,)h(so)h(that,)f(in)h(practice,)g Fi(INIT)f Fo(op)q(erations)h(will)f(alw)o(a)o(ys)g(b)q(e)h(guaran)o(teed)h (to)e(succeed.)126 519 y(2.)20 b(Eac)o(h)d(pro)q(cess)h(can)f(initiate)f(a)g (comm)o(unicatio)o(n)e(op)q(eration)j(for)f(eac)o(h)h(activ)o(e)f(comm)o (unication)e(ob)r(ject.)179 568 y(I.e.)k(correct)d Fi(START)e Fo(op)q(erations)h(alw)o(a)o(ys)f(succeed)j(\(ev)o(en)o(tually\).)126 652 y(3.)k(A)f(send)i(op)q(eration)e(is)h Fh(enabled)d Fo(if)h(the)j(sending) e(pro)q(cess)j(has)d(issued)i(a)e Fi(COMPLETE)f Fo(op)q(eration)h(and)179 701 y(the)d(receiving)h(pro)q(cess)g(has)f(issued)h(a)f Fi(START)f Fo(op)q(eration)g(for)h(a)g(matc)o(hing)e(receiv)o(e.)25 b(Symmetrically)l(,) 13 b(a)179 751 y(receiv)o(e)h(op)q(eration)f(is)g Fh(enabled)d Fo(if)i(the)i(receiving)f(pro)q(cess)i(has)e(issued)h(a)f Fi(COMPLETE)e Fo(op)q(eration)i(and)f(the)179 801 y(sending)h(pro)q(cess)i(has)e(issued)h (a)f Fi(START)f Fo(op)q(eration)h(for)g(a)f(matc)o(hing)g(send.)18 b(An)c(enabled)f(op)q(eration)g(ma)o(y)179 851 y(b)q(ecome)g Fh(disabled)e Fo(either)k(b)q(ecause)g(it)e(completes)h(successfully)h(or,)e (in)g(the)h(case)h(of)e(a)h(receiv)o(e,)g(b)q(ecause)179 901 y(the)g(matc)o(hing)d(message)i(is)g(successfully)i(receiv)o(ed)f(b)o(y)f (another)h(receiv)o(e)h(op)q(eration)e(or,)g(in)f(the)i(case)h(of)d(a)179 951 y(send,)i(b)q(ecause)i(the)e(matc)o(hing)e(receiv)o(e)j(successfully)g (receiv)o(es)h(another)e(message.)179 1017 y Fh(An)i(enabled)f(op)q(eration)f (either)h(completes)f(successfully)g(or)j(b)q(ecomes)f(p)q(ermanen)o(tl)o(y)e (dis-)179 1067 y(abled.)179 1133 y Fo(I.e.,)g(an)h(enabled)h(op)q(eration)f (either)h(ev)o(en)o(tually)e(succeeds,)k(or)d(b)q(ecomes)g(disabled)g (\(progress\);)i(and)e(an)179 1183 y(op)q(eration)f(cannot)g(b)q(e)g(enabled) g(in\014nitely)f(man)o(y)f(times)h(without)h(ev)o(er)h(succeeding)g (\(fairness\).)126 1266 y(4.)20 b(A)14 b Fi(FREE)f Fo(op)q(eration)h(alw)o(a) o(ys)e(succeeds)17 b(\(ev)o(en)o(tually\).)158 1358 y(The)e(four)g (conditions)f(guaran)o(tee)h(progress)h(in)e(the)i(comm)o(unicati)o(on)c (subsystem.)21 b(The)15 b(third)g(condition)75 1408 y(guaran)o(tee)f(\(w)o (eak\))g(fairness)h(among)d(comp)q(eting)g(comm)o(unication)f(requests.)158 1458 y(Examples)i(\(in)o(v)o(olving)f(t)o(w)o(o)h(pro)q(cessors)j(with)e (ranks)g(0)f(and)h(1\))158 1507 y(The)g(follo)o(wing)e(program)g(is)i(safe,)f (and)h(should)g(alw)o(a)o(ys)f(succeed.)75 1640 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75 1690 y(IF)i(\(rank.EQ.0\))f(THEN)140 1740 y(CALL)h(MPI_SENDC\(sendbuf,)d(count,)j(1,)g(tag,)g(context\))140 1790 y(CALL)g(MPI_RECVC\(recvbuf,)d(count,)j(1,)g(tag,)g(context\))75 1840 y(ELSE)86 b(!)22 b(rank.EQ.1)140 1889 y(CALL)f(MPI_RECVC\(recvbuf,)d (count,)j(0,)g(tag,)g(context\))140 1939 y(CALL)g(MPI_SENDC\(sendbuf,)d (count,)j(0,)g(tag,)g(context\))75 1989 y(END)g(IF)158 2072 y Fo(The)14 b(follo)o(wing)e(program)g(is)i(erroneous,)g(and)g(should)g(alw)o (a)o(ys)f(fail.)75 2214 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75 2263 y(IF)i(\(rank.EQ.0\))f(THEN)140 2313 y(CALL)h(MPI_RECVC\(recvbuf,)d (count,)j(1,)g(tag,)g(context\))140 2363 y(CALL)g(MPI_SENDC\(sendbuf,)d (count,)j(1,)g(tag,)g(context\))75 2413 y(ELSE)86 b(!)22 b(rank.EQ.1)140 2463 y(CALL)f(MPI_RECVC\(recvbuf,)d(count,)j(0,)g(tag,)g(context\))140 2512 y(CALL)g(MPI_SENDC\(sendbuf,)d(count,)j(0,)g(tag,)g(context\))75 2562 y(END)g(IF)158 2654 y Fo(The)13 b(receiv)o(e)h(op)q(eration)f(of)f(the)i (\014rst)f(pro)q(cess)i(m)o(ust)d(complete)g(b)q(efore)i(its)e(send,)i(and)f (can)g(complete)f(only)75 2704 y(if)g(the)h(matc)o(hing)d(send)k(of)e(the)h (second)g(pro)q(cessor)i(is)d(executed;)i(the)f(receiv)o(e)h(op)q(eration)f (of)e(the)j(second)f(pro)q(cess)p eop %%Page: 28 29 bop 75 -100 a Fo(28)712 b Fg(CHAPTER)14 b(1.)31 b(POINT)15 b(TO)f(POINT)g(COMMUNICA)m(TION)75 45 y Fo(m)o(ust)i(complete)h(b)q(efore)h (its)g(send)g(and)f(can)h(complete)e(only)h(if)f(the)i(matc)o(hing)e(send)i (of)f(the)h(\014rst)g(pro)q(cess)h(is)75 95 y(executed.)h(This)14 b(program)e(will)g(deadlo)q(c)o(k.)158 145 y(The)i(follo)o(wing)e(program)g (is)i(unsafe,)f(and)h(ma)o(y)e(succeed)k(or)e(fail,)e(dep)q(ending)i(on)g (implemen)o(tatio)o(n.)75 286 y Fi(CALL)21 b(MPI_RANK\(rank,)e(context\))75 336 y(IF)i(\(rank.EQ.0\))f(THEN)140 385 y(CALL)h(MPI_SENDC\(sendbuf,)d (count,)j(1,)g(tag,)g(context\))140 435 y(CALL)g(MPI_RECVC\(recvbuf,)d (count,)j(1,)g(tag,)g(context\))75 485 y(ELSE)86 b(!)22 b(rank.EQ.1)140 535 y(CALL)f(MPI_SENDC\(sendbuf,)d(count,)j(0,)g(tag,)g(context\))140 585 y(CALL)g(MPI_RECVC\(recvbuf,)d(count,)j(0,)g(tag,)g(context\))75 635 y(END)g(IF)158 776 y Fo(The)15 b(message)g(sen)o(t)h(b)o(y)f(eac)o(h)g (pro)q(cess)i(has)e(to)g(b)q(e)h(copied)f(out)g(b)q(efore)h(the)g(send)g(op)q (eration)f(returns)h(and)75 826 y(the)h(receiv)o(e)g(op)q(eration)e(starts.) 26 b(F)m(or)15 b(the)i(program)d(to)i(complete,)f(it)g(is)h(necessary)i(that) e(at)g(least)g(one)g(of)f(the)75 875 y(t)o(w)o(o)d(messages)g(sen)o(t)h(is)f (bu\013ered)i(out)e(of)f(either)i(pro)q(cesses')h(address)g(space.)k(Th)o (us,)13 b(this)f(program)e(can)j(succeed)75 925 y(only)g(if)g(the)i(comm)o (unicati)o(on)c(system)j(has)g(su\016cien)o(t)g(bu\013er)h(space)g(to)f (bu\013er)h Fi(count)d Fo(w)o(ords)i(of)g(data.)158 975 y(If)c(additional)e (requiremen)o(ts)i(will)f(b)q(ecome)h(part)g(of)f(the)i(standard)f(\(e.g.,)g (b)q(ounds)h(on)e(the)i(minim)o(al)c(n)o(um)o(b)q(er)75 1025 y(of)13 b(concurren)o(tly)i(activ)o(e)f(handles)g(that)g(need)h(b)q(e)g(supp) q(orted,)f(then)h(further)g(programs)d(b)q(ecome)i(safe.)75 1162 y Fn(1.14)70 b(Missing)75 1253 y Fh(send-receiv)o(e)39 b Fo(a)13 b(function)h(of)f(the)i(form)d(\(for)i(blo)q(c)o(king)e(comm)o (unication\))158 1303 y Fi(MPI)p 227 1303 14 2 v 15 w(SEND)p 330 1303 V 15 w(RECV\()21 b(send)p 564 1303 V 15 w(buffer,)f(destination,)f (recv)p 1124 1303 V 15 w(buffer,)h(source,)h(tag,)g(context\))158 1353 y Fo(It)13 b(ma)o(y)e(b)q(e)j(sup)q(er\015uous)h(b)q(ecause)f(the)g (collectiv)o(e)f(comm)o(unicatio)o(n)e(shift)h(has)h(similar)e(functionalit)o (y)m(.)16 b(Y)m(et)75 1402 y(it)10 b(allo)o(ws)g(to)g(generate)i(more)e (easily)g(comm)o(unication)d(patterns)12 b(that)f(w)o(ould)f(lead)g(to)h (deadlo)q(c)o(k)f(if)g(implemen)o(ted)75 1452 y(naiv)o(ely)m(,)i(with)h(blo)q (c)o(king)g(comm)o(unication.)158 1502 y(An)20 b(imp)q(ortan)o(t)f(v)n(arian) o(t)g(is)h(the)h(case)g(where)h Fi(send)p 1020 1502 V 14 w(buffer)f(=)h(recv) p 1319 1502 V 14 w(buffer)d Fo(\(this)i(can)f(either)h(b)q(e)g(a)75 1552 y(sub)q(case)c(of)f(the)g(general)g(send)p 578 1552 13 2 v 16 w(recv)h(function,)f(or)f(a)h(separate)h(function,)f(if)f(send)h (bu\013er)h(and)f(receiv)o(e)h(bu\013er)75 1602 y(are)g(required)g(to)f(b)q (e)h(disjoin)o(t)e(in)g(the)i(general)f(function\).)25 b(This)16 b(in)o(tro)q(duces)i(new)e(functionalit)o(y)f(that)h(is)g(not)75 1651 y(a)o(v)n(ailable,)11 b(otherwise,)k(and)e(whic)o(h)h(seems)g(to)g(b)q (e)g(often)g(required.)158 1701 y(An)g Fi(exchange)e Fo(is)i(the)g(sp)q (ecial)h(case)f(where)i(source)f(=)f(destination.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF >From weeks@mozart.convex.com Wed Jun 9 15:10:59 1993 Return-Path: Received: from ens.ens-lyon.fr by lip.ens-lyon.fr (4.1/SMI-4.1) id AA15096; Wed, 9 Jun 93 15:10:57 +0200 Received: from convex.convex.com by ens.ens-lyon.fr (4.1/SMI-4.1) id AA12736; Wed, 9 Jun 93 15:09:15 +0200 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA01714; Wed, 9 Jun 93 08:08:21 -0500 Received: by mozart.convex.com (5.64/1.28) id AA28959; Wed, 9 Jun 93 08:10:32 -0500 Date: Wed, 9 Jun 93 08:10:32 -0500 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9306091310.AA28959@mozart.convex.com> To: Jack.Dongarra@lip.ens-lyon.fr Subject: May 13 mpi mail #1 of 1 Status: R (received from walker, as I recall; I believe he posted it to mpi but I didn't receive it, so eventually he mailed it directly to me - DW) Minutes of the Message Passing Interface Forum Dallas, Texas March 30 - April 2, 1993 The MPI Forum met March 30 - April 2, 1993, at the Bristol Suites Hotel in North Dallas. This was the fifth meeting of the MPIF and the third of the now regular meetings in Dallas. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. For the first time a number of formal votes were taken at this meeting. All of these are recorded in these minutes (and can be found by searching for VOTE) and have also been published (to the mpi-core mailing list) in a summary of all the formal votes and all of the straw votes for the committee as a whole. The notes for these minutes were taken by Bob Knighten (knighten@ssd.intel.com) and Rusty Lusk (lusk@mcs.anl.gov). These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly the lead to each topic (and a few other things.) Attendees: - --------- Joe Baron IBM Austin jbaron@vnet.ibm.com Eric Barszcz NASA Ames barszcz@nas.nasa.gov Harry Scott Berryman Yale Univ. berryman@cs.yale.edu Rob Bjornson SCA bjornson@sca.com Lyndon Clarke EPCC, U. Edinburgh lyndon@epcc.ed.ac.uk James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Anne C. Elster Cornell U. elster@cs.cornell.edu Sam Fineberg NASA Ames fineberg@nas.nasa.gov Jon Flower ParaSoft jwf@parasoft.com Ian Glendinning U. of Southampton igl@ecs.soton.ac.uk Adam Greenberg TMC moose@think.com Bill Gropp ANL gropp@mcs.anl.gov Leslie Hart NOAA/FSL hart@fsl.noaa.gov Tom Haupt Syracuse U. haupt@npac.syr.edu Rolf Hempel GMD hempel@gmd.de Tom Henderson NOAA/FSL hender@fsl.noaa.gov C. T. Howard Ho IBM Almaden ho@almaden.ibm.com Steven Huss-Lederman SRC lederman@super.org Rusty Lusk ANL lusk@mcs.anl.gov John Kapenga Western Michigan U. john@cs.wmich.edu Bob Knighten Intel SSD knighten@ssd.intel.com Rik Littlefield PNL rj_littlefield@pnl.gov Peter Madams nCube pmadams@ncube.com Arthur B. Maccabe U. of New Mexico maccabe@cs.unm.edu Oliver McBryan U. Colorado mcbryan@cs.colorado.edu Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Instiute otto@cse.ogi.edu Peter Pacheco U. of San Francisco peter@sun.math.usfca.edu Paul Pierce Intel prp@ssd.intel.com Sanjay Ranka Syracuse U. ranka@top.cis.syr.edu Arch Robison Shell Development robison@shell.com Mark Sears Sandia mpsears@cs.sandia.gov Anthony Skjellum Mississippi State U. tony@cs.msstate.edu Marc Snir IBM, T.J. Watson snir@watson.ibm.com Alan Sussman U. of Maryland als@cs.umd.edu David Walker ORNL walker@msr.epm.ornl.gov Dennis Weeks Convex weeks@convex.com Stephen Wheat Sandia NL srwheat@cs.sandia.gov Wednesday, March 30 - --------- -------- - ------------------------------------------------------------------------------- General Meeting - ------------------------------------------------------------------------------- Jack Dongarra called the meeting to order at 1:30. The first topic for discussion was the agenda. David Walker had mailed out the following Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Discussion of Snir & Geist collection communication proposal (everyone) (Otto?) 12:00-1:30 Lunch (provided) 1:30-3:00 Full group meeting for presentation of alternate approaches to groups and contexts, dynamic vs. static process models, and other issues (Volunteer?) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir?) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) Following the discussion on the mpi-core mailing list the question was moving the discussion of the timetable for producding MPI to the beginning of the meeting. After a brief discussion it was decided to proceed first with reports from the Communication Context and Point-to-Point Communication subcommittees in order to have a basis for discussing the schedule. The schedule was discussed on Thursday afternoon, following the completion of the Point-to-Point subcommittee report. The Context subcommittee was alloted two hours, from 1:30 to 3:00, with the Point-to-Point Communications subcommittee scheduled from 3:00 to 6:00 and on Thursday morning. - ------------------------------------------------------------------------------- Report From the Communication Context Subcommittee - ------------------------------------------------------------------------------- Tony Skjellum presided. There was a large volume of activity on the mpi-context mailing list before this meeting and so there were five proposals available for consideration, labeled: I (Marc Snir) III (Tony Skjellum) VII (Lyndon Clarke/Rik Littlefield) VIII (Mark Sears) X (Tony Skjellum/Lyndon Clarke) 25 minutes was alloted for presentation of each of these in the order I (Marc Snir), VII+X (Lyndon Clarke), VIII (Mark Sears) , III (Tony Skjellum), X followed by general discussion. Tonight there will be a subcommittee meeting to produce a single proposal. Proposal I (Marc Snir) - ---------- ----------- {Marc used overhead projector slides and these notes are largely a transcription of those slides.} Group=Context Proposal Goals: + Keep it simple (and keep MPI small) + Keep it efficient Minimal needs: + Protection mechanism + Local name space Group = Context = Ordered set of processes. - -- Method for protecting communication between e.g. libraries. - -- + All point-to-point communication is WITHIN a group and uses (group,rank) address. + All collective communications by a group (which is a context) OPERATIONS: + Group copy + Group partition + Group creation by list + Group deletion + ALL group prexists Group handle has only local (i.e. within group) use and meaning. - -- There is no reason to pass the handle of a group outside the group - it has no use. - -- 1. Impact on "current practice": Need additional argument ALL in all p-p calls. 2. Overhead for p-p send - no impact when ALL use. One lookup for other groups. receive - context id match Overhead at creation Loosely synch collective communication within group affected Storage Member table (good protection) - -- 3. Compatibility with dynamic process creation and deletion Process creation/deletion requires same for group {What is ALL group after process creation or deletion?} 4. Interaction with topology Group has no topology information (but it can be used as a peg for such information.) 5. Inter group communication (e.g. client-server models) +-------------------------------------+ +---+ ---> (---) | * | | * | | * | +---+ +--------------------------------------+ Do the communication within an encompassing group. - - + Encompassing group needed for protection + May not be convenient for naming (e.g. send(server[5], ...)) + Inconvenience does not warrant change in p-p layer + Can be handled by creating and explicityl passing arrays of ranks: MPI_LIST_RANKS(list, subgroup, group) returns the list of the rank of each subgroup member within group - -- Discussion: How have available both subgroup and group? There must be an encompassing group which has full knowledge, e.g. server that is member of both the group and the subgroup. This proposal is orthogonal to question of attaching additional information (e.g. topology information, caching, etc.) Can't deal with situation of contacting an independent pre-existing server. Marc's approach is that dynamically adding processes requires dynamically creating a group containing all of the processes. Opacity vs. accesibility to mechanism. ============================================================================= Lyndon VII and X VII: Context is a higher level mechanism than a group. It is basically a unique identifier together with a reference to a group. This means that as a group changes, all contents that reference that group change as well. Same ability to hang on facilities (e.g. topology, caching) as others. Relation to p-p: Three forms: "closed form", "null form", "open form" Open form is to allow communication between different groups. Experience is that creating encompassing groups is difficult. Disagree with Snir's claim. Addressing is via (context, rank). Need a "context allocation" mechanism - this implies global communication. Relation to c-c: Works very cleanly for all-all using closed form. Two group communcations - for MPI-2 Discussion: How to establish communication between groups? Can send context. What is opaque? Lyndon - not important. Startup/bootstrap - everyone starts in the ALL group. Can use common ancestor or name registry. Power relative to I? Lyndon claims that this is more convenient once communication is setup. Basic idea is to be able to communicate via (my_context, remote_context, rank) ORTHOGONAL ISSUES - caching, tag selection, transfer of ???? X: Attempt to synthesize III & VII A CONTEX is a space of tags. A GROUP is a set of process references. [What does this mean?] Idea is to give method for combining groups and context for purpose of communication. COMMUNICATORS (see pp. 3-4) Silly names, but a serious proposal. Floopy - arbitrary communication between processes allowing wild card on tag. Bongo - basically like Marc's proposal - communication within a group using rank naming ("closed") Bingo - communication between groups. Question: Why do we want ANY of these proposals? Performance, build large scale software safely We need examples for all of these proposals! Collective -> group; express collective in terms of p-p implies need to discriminate message -> context. But there are reasons to have groups and contexts that have nothing to do with collective communication. Argument that in Marc's proposal the need for a context means have to have an additional group - but is this a problem? Lyndon argues that there are good reasons to separate group and context. Static vs. dynamic groups. Ability to move context. Proposal VIII - Mark Sears {{{ MPI-1 char * MPI_ALLOCATE Group and Context Proposal Number(); }}} - -- Context and Groups are orthogonal + Orthogonal purpose + Orthogonal functionality + Orthongonal implementation - -- Contexts Purpose - promote software modularity by allowing construction of independent tag spaces Definition: A CONTEXT is an integer-valued extension to the tag component of the meggage envelope, and must match exactly between sender and reciver - -- Model: + Contexts are global. + No concept of process belonging to a context. + Contexts are scarce resources (16). + Context allocation is a rare event. + MPI p-p requries no reference to groups. - -- Context allocation/deallocation ALLOCATION: int mpi_getcontext() + called synchronously by all (EVERY SINGLE ONE) processes + signals to mpi that use of context is now allowed DEALLOCATON void MPI_free_context(value) MPI_DEFAULT CONTEXT + Preallocated; can't be freed. + Solves initialization + Free-for-all But believes that allocation/deallocation not truly needed - could have entirely static system. + Contexts are global + No concept of process belonging to a context + Context are scarce resources (e.g. 16) + Context allocation is a rare event + MPI p-p requires no reference to groups - -- GROUPS Purpose - provide tools for organizing subsets of processes in a parallel task (i.e. MPI program.) Definition: A groups is a 1-1 mapping from (0..n-1) to another set of integer. A group is a collection of processes only in so far as the elements are process addresses. Groups have no associated have no associated context or tags, default or otherwise. - -- Group Implementation + local to each process based on information needed to construct the mapping + Group type is local and opaque + Groups can be sent in message only by sending the information needed to construct the group. + Groups are objects in the OOP sense - -- Usage: MPI_SEND(n,buf,process,tag,context) MPI_BROADCAST(n,buf,group,tag,context) Group identifier is a local opaque type, thought of as a pointer to one of many possible group structures. MPI_SEND(..., element(group,rank),tag,context) GROUP FUNCTIONS int order (group) int range(group) int element(group, int rank) int iselement(group, int element) int rank(group, int element) CLASSES identity, permutation, linear, list, bilinear, composition, cartesian CONSTRUCTORS group makelineargroup(order,start,delta) - -- Two kinds of 3rd party code 1. Code that inherits context and tag space from caller. Example: MPI collective communication 2. Code which allocates and manges own context and tag space. MPI should allow both of these. - -- Topology Global topology - mapping of processes to processors. provide inquiry function returning a string describing this mapping: char * MPI_global_topology() Examples of output: "N 564" - random network of 564 processes "H 5" - 5 dimensional hypercube "R 2 16 13" - 2D mesh, 16x13 Local topology - implicity within group; no additional functions needed. - -- ADVANTAGES + Ease of implementation + Close to hardware + Good use of resources + Flexibility in implementation of higher level concepts + MPI p-p requires no reference to groups + MPI c-c can be layered on top of MPI p-p - -- Discussion - Serious problem with global communication. This destroys the software modularity. How to do global operations using groups? Responsibility on code to insure there is disambiguation, synchronization, etc. Tony - III + Tag/context partionting message space for "safe" software + Groups encapsulate scope of operations for 1) notation 2) optimization 3) performance ........... ............... / . GROUPS . . . / ......... . CONTEXTS . / lower,higher? ............. Relation between groups and contexts? Groups can be orthogonal except for group creation. - -- Forms of communiations + collective communication ON groups (compatible with I) + p-p A] (group,rank,tag) - analagous to I B] (context,pid,tag) Models 1) Contexts created/destroyed 2) contexts can be published dynamic server implied or shared address area Contexts & groups interelate when creating new groups not necessarily from LCA ^^^{{{WHAT DOES THIS MEAN?}}} - -- contexts & groups interrelate when creating new groups, not necessarily from LCA (what does this mean?) Dispute/reply regarding optimization. Argument that group can be used to provide to information about special situations (e.g. shared memory) that can be used for optimization. Dynamic groups more feasible using III or VII. Can contexts be sent in III? Yes. VII more complex than III because it offers more layers. Tony believes that dynamic groups are essential for the heterogeneous case and so believes that I is inappropriate. John K. notes that other proposals can be built on top of VIII. Proposal to defer straw vote until tomorrow to give people time to ponder. Is global synchronization an essential part of VIII? Sears - no, there are various possibilities. Why is it important that contexts are global? Because a context is not associated with a group of processes. This looses much of the safety. It also looses the local addressing within a group. Sears argues that this complicates p-p, but Snir says that something like his proposal has been implemented and is not complicated or expensive. What level of protection is needed/desirable? Picture of using context for safety - at startup send a context to each library used which it then uses for internal safety. Problem - libraries using libraries using libraries, etc. Importance of receiving on wild cards and relation. Host/Node model? Must support this and all do. (What about loading program - not part of THIS discussion.) Subcommittee will meet tonight and present a more unified proposal tomorrow morning. Rik asked for example showing how to implement safe barrier using p-p, group, context. Adam - How can we evaluate these proposals if we have not agreed on what we want from the context concept? - ----- break 4-4:20 pm - ----- - ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee - ------------------------------------------------------------------------------- First reading of p-p proposal - Marc Snir presiding Presentation with some minimal assumptions about use of context. No language binding included. Use of handles and opaque objects. [Ignore text comments on implementation - issues for language experts.] Discussion of ephemeral/persistant. Greenberg asked about "error to free a handle with pending operation" - change to "handle becomes ephemeral" or some such. How deal with lists - included length, separate length, EOL marker. States [Implementation again - not part of current proposal] If have only one send/receive, what is type of buffer and what to do about this? Marc's proposal is to accept breaking F77 in this case. Skipping 2.4 (Contexts) for now (including error handling.) 2.5.1: What about C/Fortran compatibility for messages? Also skipped for now. 2.6: Greenberg wants at least escape hatch to allow functions (MPI_ADD_?) that add, e.g., other F90 objects. Discussion of len in MPI_ADD_BLOCK as "Number of elements" rather than as bytes. There are language and portability issues. Marc mentioned issues in middle of p. 14 (negative displacements, ...) and notes that these must be settled. Note that delete/commit functions discussed last time are not in this proposal. 2.6.1 (Data Conversion) It just happens - this needs to be stated clearly. - -- back to 2.2 - vote on each section [[NOTE: A proposal, in the form of a proposed chapter is offered. Votes are on amendments and on accepting particular parts of chapter.] How are handles allocated? User or system. There are efficiency advantages to user allocation, e.g. on stack. PRP argues for a create for each data type rather than a generic MPI_CREATE. At the moment there is an admixture in the proposal. ====== Discussion of voting rules. Organizations voting: 24 ===== VOTE: Separate mpi_create for each type - ---- Yes: 20 No: 0 Abstain: 4 Possible meanings of free: free 1.1 can always be done 1.2 can be done only if user will not use handle no pending operation after free 2 if no pending operation, deallocate otherwise free when done 1.2 = current proposal; 2 is Greenberg proposal Greenberg explantion - common thing done in their system is to have handlers with free only taking effect after handler completes. PRP - Has primitive that does essentially 2 (which is not "mark ephemeral") It does not imply cancel. Fire and forget. Cownie: Two arguments - Paul's fire and forget; Adam's handler for messages. Snir: Is buffer available after free? PRP/Adam: No. Why not just have another kind of free? Multiplicity of functions. VOTE: mpi_free is valid even if handle is in use, but effect is to - ---- free the object when the operation completes. Yes: 7 No: 8 Abstain: List of handles - change name to array of handles VOTE: list_length explicit (rather than included in array or EOL)? - ---- Yes: 22 No: 0 Cownie wants a method to provide cheap allocation, e.g. on user's stack rather than in system heap. Need a concrete proposal. RLK wants explict statement of what is erroneous, checked, etc. VOTE: Accept 2.2 - ---- Yes: 23 No: 1 2.3 & 2.4 skipped -- they will be considered elsewhere. 2.5 Cownie - proposes that there be both CHAR and BYTE data types rather than just the basic data types of the host language. Another proposal is to have an MPI_STRING data type. But what length? Null terminated? VOTE: BYTE data type. - ---- Yes: 24 No: 0 Pending action of Context Subcommitte is the content of the envelope. VOTE: Accept 2.5 (minus 2.5.2) - ---- Yes: unanimous Proposals for units of bytes and for consistency. 1. Units are elements 2. bytes in C and elements in F77 (No one favors this. Against: 14) 3. bytes everywhere (and provide some way of getting size of basic types) a) index stays as is, but bytes elsewhere Pro: 9 Con: 13 (VOTE 1) b) truly bytes everywhere, including indices Pro: 10 Con: 8 (VOTE 2) Greenberg will bring forward additional proposal for 2.6 at next meeting. VOTE: In vector, stride may be negative? - ---- Yes: 10 No: 2 VOTE: Allow repetition? - ---- Yes: 13 No: 1 VOTE: Can multiple components overlap? - ---- vote not taken PRP proposes that in vector len is a count of the number of blocks. VOTE: total length is an integer multiple of the block size - ---- Yes: 4 No: 10 Tony proposes adding a COMMIT operation. Note: Current proposal does not clearly state it, but a handle cannot be modified after it is in use. There is nothing that explicitly specifies that the modifications are complete. VOTE: Commit? - ---- Yes: 11 No: 6 VOTE: Accept 2.6 - ---- Yes: 12 No: 4 - ----- Break for dinner - 6:15 p.m. - ----- Subcommittees tonight: Here: Vendor caucus Rm 1: Process topology Rm 2: Collective Communications Subcommittee Rm 3: Context Subcommittee BAR: formal, language binding, profile meet at 8 p.m. ----------------------------------------------------------------------------- Revised agenda Thursday 9 p-p (cont) 12 lunch 1:30 (registration) 1:30 collective communication 3 process topology 6-8 dinner Future: May 12-14; June 23-25; August ? ============================================================================= Thursday, April 1, 1993 9:10 a.m. - - ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee (continued) - ------------------------------------------------------------------------------- Point-to-Point Communication - First Reading (continued): - -------------- ------------- ----- ------- Marc Snir presiding 2.7 (Receive Criteria) & 2.8 (Communication Mode) - ------------------------------------------------- Suggestion that we need more than one DONTCARE, a SOURCE_DONTCARE and a TAG_DONTCARE. Lyndon Clarke proposed a "secure" communication mode where the send that will return once the system can guarantee that the receive will actually complete. (During the discussion of this there was a suggestion that the word REGULAR on p. 16 of Marc Snir's draft be changed to STANDARD so first can always be used. Marc remarked that he welcomes all manner of stylistic improvements, but asked that they be sent to him, not voted on here.) Why such a secure communication mode? To provide a portable manner to write programs that are guaraneed to be safe, even without buffering. This is similar to but weaker than the synchronous functions that were rejected in a straw vote last time. The unease with a proliferation of functions was again mentioned. Adam Greenberg suggested that one could be against this and still specify equivalent function by requiring no buffering throughout the system. Rik Littlefield (as a pseudo Tony Skjellum) proposed receive criteria based on an intag and mask. Variations on this have been discussed in the past. VOTE: Typed DONTCARE? - ---- Yes: 19 No: 1 After the vote, a count showed 25 organizations present. Proposal: Receive selection based on (tag & mask) = (intag & mask) Why do this if have context? Discussion of efficiency. PRP proposes sending tag (exact match by system) and extra-info (for recognition of message category by application) in envelope. Cownie unhappy because of effect on latency of small messages. Alternatives being considered: (1) RECEIVE(..., tag, info, ...) no DONTCARE for tag (2) RECEIVE(..., tag, mask, ...) no DONTCARE for tag (3) RECEIVE(..., tag, ...) DONTCARE for tag Does this imply small tags? PRP: reasonable for implementation to limit size of tag, category, and context. Greenberg argues againt PRP propsal because of existing practice and because it could force user to duplicate some system function. Eric argues in favor of wildcarding because it often allows reuse of buffer and so fewer resources. VOTE: (1) fails for lack of second. - ---- (2) Yes: 6 No: 12 so (3) remains. Proposal: Secure communication mode. Rusty - P4 experience argues against this. Cownie - this is useful for reasoning about program correctness. An alternative is to have a "secure mode". But what happens if using secure mode, but some library may have been written without secure mode? VOTE: Do we want a "secure" communication mode? - ---- Yes: 7 No: 6 VOTE: Delete send/ready_receive? - ---- Yes: 10 No: 10 [Amendment tabled] VOTE: Accept amended versions of 2.7 & 2.8 - ---- Yes: 25 No: 1 Discussion of 2.9 (Communication Objects): - ----------------------------------------- Marc Snir began with an overview of the four subsections. He asked if there should be handles for user space objects? This is not in the current proposal. He noted that the use of a STATUS_ALL compared to using STATUS in a loop is one of convenience. Jim Cownie argued: WAIT_ANY should be told where to start scan because of fairness concerns. [Discussion - this is a strong implementation requirement. MPI requires "fairness", though the requirement is so weak as to be untestable. WAIT_ANY is small part of the fairness problem. Easier for user to pass in this information than for the system to maintain it. The user can always guarantee fairness. But at some cost. Postpone until we discuss correctness and fairness in general.] The current draft has MPI_RETURN_STAT providing the free space in the buffer. The current practice appears to return the number of bytes received. Issues: Byte count of data received; Where do handles come from; Partially specified handles; Make explicit that can a message can be sent to oneself. VOTE: The return status for a receive operation should be the number - ---- of bytes received Yes: 17 No: 3 Abstain: 7 The return status handle should be allocated by the user: - -------------------------------------------------------- Typically the user knows exactly the what needs to be done and the lifetime is often short. Thread safety says can't use global storage, so ... Suggestion that there be an overall proposal to deal with handles in user space. But this is not a general handle - this is a special situation. Postpone for general consideration. Partial handles - --------------- return_status_handle + part of communication handle + separate user space object + separate system object VOTE: Accept 2.9 (with amendments) excluding 2.9.1 and WAIT_ANY - ---- Yes: 19 No: 2 Abstain: 3 - ----- break 10:48 - 11:08 - ----- Discussion of sections 2.10-2.12 - -------------------------------- Marc presented the following table for discussion: |general|contig|vector|contig| |buffer |byte |byte |type | - --------------+-------+------+------+------+-------------------------------- blocking | | | | | send | * | | | | receive | | | | | - --------------+-------+------+------+------+ blocking | | | | | ready-receive | * | | | | send | | | | | - --------------+-------+------+------+------+ immediate | | | | | send | * | | | | receive | | | | | - --------------+-------+------+------+------+ immediate | | | | | ready-receive | * | | | | send | | | | | - --------------+-------+------+------+------+ secure | | | | | send | | | | | receive | | | | | - --------------+-------+------+------+------+ Same buffer types are used in Collective Communication. Discuss probe Issues: Counting units? Vector type? Contig type includes contig byte because have byte type? Lyndon - There should be a secure-receive (for optimization of the protocol.) Marc - Is system or user responsible for insuring this works? Lyndon - User. Various - What is value of secure-receive. Gropp - Because these are different protocols loose performance if don't have different functions as general must always deal with worst case. Marc: error? ssend(2) ---------------------------- recv(1) Proposal is that it is erroneous to attempt to receive a message sent by a secure-send by other than a secure-receive. Rusty argues that this should not be erroneous because ??? Possibilities: 1) secure-send {can, cannot} be received by receive 2) Enforced by MPI or user responsibility Adam argues that this is confusing the secure-send features with pre-acknowledgement and these are independent issues. The purpose of secure-receive is entirely performance - it does not have any semantic content. secure-send/secure-receive can be implemented on top of regular p-p. Note that the proposal is to add 2 secure-sends, 2 secure-receive VOTE: Add both blocking and immediate secure-send/secure-receive - ---- with failure of program to match being erroneous. Yes: 10 No: 8 Abstain: 9 VOTE: 2.10 as amended - ---- Yes: 20 No: 4 Abstain: 3 VOTE: 2.11 - ---- Yes: 26 No: 0 Abstain: 1 2.12 - ---- Are blocks just blocks of bytes or are they typed? How do count the size of blocks? Lusk - Want to have blocks of typed data for use in a heterogeneous environment. PRP - Offers precise propsal that use the same parameters as in MPI_ADD_BLOCK. VOTE: Use exactly the same parameters in MPI_ADD_BLOCK as amended - ---- Yes: 26 No: 0 Abstain: 1 Adam - Proposal to have functions to have strided messages using blocks. Rusty - Argues against becasue of problem of proliferation at this level. [Arguement was clearly on matter of taste. This is syntactic sugar as the most general low level routines certainly can be used for this.] VOTE: Have strided block message functions. - ---- Yes: 5 No: 9 Abstain: 12 - ----- lunch 12:00 - 1:35 - ----- Continuation of discussion of Chapter 2. Proposal that more time discussing 2.10-2.14 needed, so p-p subcommittee will meet after dinner tonight. Discussion of Schedule - ---------------------- Future meetings 4. May 12-14 set 5. June 23-25 set 6. August 11-13 tent. 7. September 22-24 tent. Draft to be available: November 15-19 SC '93 Portland Reading Schedule - ---------------- P-P Snir April & May Collective Otto/Guist April & May Profiling Cownie April & May Process Topolgy Hempel May & June Environmental Mgmt Gropp May & June Lang. Bind. gen. Berryman May & June Context Skjellum May & June Formal Spec Zenith June & August ??? Specific language bindings will follow general material by one meeting. Where is the language binding material? There will be a general principles chapter and also the actual bindings. These will be separate votes. General language Berryman June Specific lang. bind. Berryman Is anything coming in the Formal Spec? Does anyone care. Rusty - one participant and two observers. Zenith told Rusty that he was working on something, but nothing has appeared and he is not here. What about public comment? Discussion of the HPFF model of two opportunities for comment. Proposal to have only one round of public comment with draft released to public at Supercomputing '93. Reference implementation - Gropp/Lusk effort Test suite - Greenberg and Haup will lead an effort Subset - "Implementation Order Recommendation" Huss Goals + Define a reasonable subset of MPI that is recommended for initial implementation + Only a minimum - welcome to implement more + Allow MPI to begin to show up in a timely fashion while still consistent across vendors + Consistent with complete standard - -- Method + Create new "subset committee" + Write an Annex (like HPF) + Present flushed out proposal/annex at next meeting + Hope "other" committee will create initial test suite for minimal implementation :-) Might motivate implementors - -- First shot list + want subcommitte members + Begin discussion via e-mail - ----- NOT IN SUBSET 1 No persistent handles 2 No multicomponent buffer descriptors {only one item described} 3 No indexed component 4 No waitany or waitall 5 No name server model If interested send mail to Huss - ------------------------------------------------------------------------------- Report From the Collective Communication Subcommittee - ------------------------------------------------------------------------------- Collective Communications - First Reading - ----------------------------------------- Steve Otto Subcommitte of three met last night (Otto, Ho, El...) Propose a discussion about safety, semantics and function of collect c NOT: Contexts/groups NOT: detailed questions of data types, lan. bind, p-p Semantic Warm-up - ---------------- Barriere implies a time synchronization of processes, but NOT an emptying of all message buffers. I.e. p-p messages may span (in time) a barrier: 1 2 post-send(2) post-receive(1) barrier barrier complete-send(2) barrier does not imply that message queues were emptied. Object to statement on p. 8. {{{QUOTE}}} - -- If we want collective function that does ensure all messages queues were emptied, let's invent it. wait-for-all-global Example 3. p. 46 Is this safe? 0 1 bcast /--------------- receive(0) send(1) ------------/ bcast Does this deadlock? Depends on implementation of bcast If bcast implemented with buffered contenxt-unique messages, probably won't deadlock If bcast synchronizes strongly, probably will deadlock Otto unhappy about defining sematics of collective communication routines in terms of operational p-p routines because this depends on side effects such as emptying message queues. - -- Cownie/Snir argue that this is not true because ... {{{MISSED}}} - -- Conservative Proposal Instead of mandating that example above is "safe", we propose that no messages are allowed to be "in the air" upon entry to collective communication call. So: We require that c-c routines be used AS IF they implied barrier synch but === the user cannot assume that they actually provide barrier synch - -- Confusion about what is actually intended. Unhappiness about phrase "in the air". The example above is unsafe, but is it erroneous? Claim is that this is unsafe, but an implementation that allows it is compliant. Now what is the situation of the example on p. 46? It is there because Marc want people to be aware that the behavior of a valid (if unsafe) program may be surprising. - -- Related point: May want to have a "barrier mode" for c-c so that they do behave as barriers when this is on. Useful - can detect many erroneous programs. - -- Keeping multiple c-c's separated - -------------------------------- /group 1 /collective operation 2 [ ( ] ) \collective operation 1 ^ \group 2 | can messages in here to go to wrong destination? Intersecting spanning trees ... can they catch each other's messages It seems that good implementations can be constructed so that they don't. ==> NOT dependent on user-tags TAGS ARE SUPERFLUOUS FOR C-C + keep for consistency with p-p? + but, what is the implemntor supposed to do with them? + ignore them? + what do we tell users about tags in c-c? We propose - NO TAGS in c-c! - -- Why have them - for debugging. Anything else? Does lack of tags affect non-blocking c-c? Yes. Steve Huss: Want to make sure that system must insure that successive broadcasts will never result in confusion of messages. Marc: What is the exact semantics of c-c, in particular when is the burden on the user of MPI to insure no ambiguity vs.l when this is guaranteed by the system? This is particularly important if generalize to allow non-blocking c-c. PRP: Might want to guarantee that ordering is guaranteed between c-c. Snir: Certainly want to guarantee that if parameters in c-c are different, then there is no ambiguity. Further may guarantee even if have same parameters. Proposal - no tags and no ambiguity (so matching is via parameters and sequencing.) [System responsiblity vs. user responsiblity] Steve: Wants MPI to guarantee preservation of order Kapenga/Pierce: Order may not be meaningful Gropp: Already have similar issue for p-p. Snir: So what is behavior when order is not meaningful? Cownie: Kapenga made important point that multithreaded libraries cannot use c-c without context information. Various: MIght be able to do this by using duplicated groups. Again, issues is what responsibility falls on system and what on user. One possible solution is to require user to provide ordering of c-c. - -- Functions - --------- bcast barrier gather global reductions cshift/eoshift scans all-to-all bcast index - -- 3.1 Introduction - ---------------- Note that non-blocking c-c is not included VOTE: Include non-blocking c-c? - ---- Yes: 0 No: uncounted Abstain: 9 Note that groups carry no topology information. Weeks proposes adding a perfect shuffle c-c function. Ranka proposes adding a permutation c-c function. Mention of variations on this. Rik observes that the arbitrary buffer descriptor versions will be extremely complex in implementation. He also objected that this cannot be implemented using p-p. Marc responed that it is possible using p-p because can send message to oneself. Discuss this in detail when we get to the reduction section. 3.2 (Group Functions) & 3.3 ( - ----------------------------- LATER 3.4 Synchronization - ------------------- Tag is removed, so examples are now wrong and must be replaced. Jon wants the examples removed as they contain semantics that are not guaranteed. Marc suggests moving to appendix with the hope that this would eventually contain a full specification of c-c in terms of p-p. Agreed. VOTE: Remove tag in all c-c? - ---- Yes: 15 No: 5 Abstain: 5 VOTE: Accept just semantics of 3.4 - ---- Yes: 23 No: 0 Abstain: 2 - --- break 3:15 p.m. - 3:35 p.m. - --- Ho has paper on collective-communication library. He will make it generally available. 3.5 (Data move functions) - ------------------------- Otto: max size to shift? NO Issues: + What if inbuf is outbuf? + Periodicity in topology? [Hempel] + Tie topology to chsift [Hemple] + cshift as sendreceive(source, destination, ...) [Flower] Allowing inbuf and outbuf to be the same violates F77. Rik - user-level double buffering a pain. various: back and forth on responsibility of user vs. system. Steve Huss: Proposes a cshift with only 1 buffer. Disallowing partially overlapping buffers. Note that this new cshift will have an INOUT argument. Does this same argument apply to other reduction functions? Have to check one by one. VOTE: Allow 2nd cshift with only one buffer. No partial overlap on - ---- orig cshift Yes: 21 No: 0 Abstain: 6 Note that earlier encompassing vote imples that we will have types in cshift. Marc remarks that changes to measuring size in bytes affects statement in text that buffers have same number of units. eoshift - ------- Second single buffer form? zero filling - another argument? VOTE: Accept cshift, eoshift proposals with amendment of 2nd form of each. - ---- Yes: 20 No: 2 Abstain: 4 bcast - ----- VOTE: Accept bcast - ---- Yes: 22 No: 0 Abstain: 2 gather - ------ Note - len is number of bytes, not what is written. Long general, rambling discussion of gather. Proposal - separate IN argument to gather of sizes; separate functions to find sizes. Proposal - version of gather with list of outbufs Proposal [Flower] - all-to-all gather(???) Cownie moved to direct gather and scatter back to subcommittee for further consideration. Accepted. 3.6 (Global Compute Operations) - ------------------------------- Issues: + inbuf=outbuf problem? 2nd version + have types, so don't need (R,I)MAX, etc. + return value to all? 3rd version + maxloc (etc.) return location and value + restriction on user defined functions + vectorized user function Returned to committee for further work. scan - nothing to say for today correctness - nothing to say for today Finished with collective communication for today. - ------------------------------------------------------------------------------- Report From the Process Topology Subcommittee - ------------------------------------------------------------------------------- Rolf Hempel presided. No presentation, but need to discuss the direction to go. First question - is topology going to be part of group management at all? Rolf remarks that vast majority of applications have a natural topology. There are implementation efficiencies - e.g. avoiding tables for mapping to processors. Marc: User visible mapping of processes to processors is not likely to be valuable. Trend in hardware is to hide hardware topology. What is the advantage of topology? Convenient in writing programs when topology is natural to problem. This information may be useful for implementation on particular systems. What do vendors have to say about the utility of such information. Cownie: The point is to be as flat as possible. Snir: Topology can certainly be built as a superstructure. What is value of integration? Convenience, safety. Safety does not appear important and he does not see sufficient value to convenience. Major issue is relation of topologies and groups. Can store topology as part of group (current proposal) or as a superstructure (Marc's preference.) Certainly convenient to have standardized method to build e.g. row group, column group, etc. This is what is in current draft, but the issue is one of integration. But is it more efficient or even substantially more convenient. Three possibilies: Not in MPI, in MPI but not integrated, integrated into group mechanism in MPI. Various repetitive arguments for each of these positions. Straw vote: Topologies in MPI? - ---------- If yes integrated with groups (e.g. eoshift), OR separate library, environmental inquiry In: 25 Out: 4 Abstain: 5 integrated: 4 Separate: 23 Abstain: 7 Back to discussion in the subcommittee. Meeting tonight: p-p room 1 c-c room 2 ============================================================================= Friday, April 2, 1993 - ------------------------------------------------------------------------------- Report From the Profiling Subcommittee - ------------------------------------------------------------------------------- Profiling - Jim Cownie - ---------------------- Based on draft section ?? of document. This is a single level approach and is basically static, i.e. based on selecting actual functions at link time. Questions: 1) Is one level OK? - other option is chain of function pointers 2) Debugging support? - Dump all message envelopes - Status of active handles Single level has no extra cost, but limits as might want to use the single level for another purpose e.g. as a network intermediary. Yes, that limit could be significant. Provide multi-level interface in the same manner (i.e. publishing alternative names for each level)? [Problem is setting the multiple interfaces.] Provide environmental facilities for exporting a multi-level dynamic approach? General agreement that single level is better than extra cost for everyone. Note that single level actually can support full dynamic approach as the MPI routines can be replace by functions that do function pointer swizzling. Debugging support? - ------------------ A small amount of discussion last night. What is needed and what can be provided? Besides items mentioned above, need to be able to decode the opaque objects in MPI. Useful to have a recording mechanism. - ------------------------------------------------------------------------------- Report From the Language Binding Subcommittee - ------------------------------------------------------------------------------- Scott Berryman Gross assumptions: No Fortran 90 binding C++ binding = C binding Specification says nothing about language interoperability {The confusion bomb went off!} [sender deals with native message; XDR in general; underspecified buffer descriptors - incl. lengths or incl. lang. spec.; general vs. limited translation; know transl in hetero - don't know in homo; this is not just a language issue - it is a language implementation issue][Need concrete proposals] - ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee (continued) - ------------------------------------------------------------------------------- Marc Snir Byte vs. Element Count + Need ADD_VEC, ADD_INDEX with byte displacement + Most usage will be element displacement - -- [R] [III] [R] [III] [R] [III] [R] [III] (array of records) so odd displacement - -- Have two different components + Block start len - number of elements data type + VEC start len - total number of elements stride - number of elements between blocks lenblk - number of elements per block data type + HVEC start len - total number of elements stride - number of BYTES between blocks lenblk - number of elements per block data type [///] [////] [ ][///] [////] [ ][///] [////] [ ] VEC(..., 5, 3, 2, REAL) HVEC(..., 5, 3*size_of_real, 2, REAL) INDEX start array_of_indices - element index (start has index zero) type HINDEX start array_of_indices - element displacement in bytes type HVEC, HINDEX more general but less convenient and more error prone Returned length of received messages - ------------------------------------ Number of elements IS always meaningful (even for message containing multiple type.) And number of elements sent is always the same as the number of elements received, while the number of bytes sent/received need not be the same. Moreover the number of elements is not too hard to compute. Proposal: Use the element count except for displacements in Hxx buffer components. Organizations present: 22 VOTE: proposal as above - ---- Yes: 19 No: 0 Abstain: 3 Probe - ----- 1) Use to decide where to receive message (allocate memory) 2) Use to debug. Propose to support 1. Do we need probe? Why not receive in a system buffer and return a pointer? Problems - system buffer is untyped; memory management Proposal: MPI_PROBE(source, tag, context, flag, return_status_handle) MPI_RETURN_STATUS(handle, len, source, tag) Assuming no other concurrent receive (single thread) MPI_RECV executed with source/tag returned by PROBE and same context will return message found by probe. Multithread programs need suitable critical region. What is returned in the len field? Should be the number of elements, but unfortunately don't know this without buffer descriptor. So return number of bytes? This may not be useful for deciding size of buffer in a hetero env. So possibilities: 1) len = -1 and provide DECODE(buff_desc, msg_status_handle) 2) Number of bytes ("on the wire") 3) Number of elements (cost of including this info in envelope) Could have 2) and also provide DECODE. The only inconvenience is the difference in units between DECODE and RECEIVE. Another alternative is to provide a data type as part of the PROBE rather than a buffer_descriptor - this would cover the most common case of uniform buffer_descriptors. Cost of rebuilding buffer_descriptor for receive because have to add in pointer for actual storage space. {{???Is this right}} Oliver object proposal again 4) PROBE(source, tag, context, type, return_status_handle) returning number of elements. Straw Vote: Probe? - ---------- Yes: 23 No: 2 Straw Vote: 1) probe - no length 8 - ---------- 2) probe - simple type length 17 3) probe - byte count 11 4) probe - element count 6 5) decode function 22 Cancel - ------ MPI_CANCEL(handle) Either communication succeeds or CANCEL succeeds, but not both nor neither. (This is to cancel a non-blocking send.) + Need CANCEL to recover committed resource + Implementation is not trivial Is it valid for CANCEL to always fail? No, if there is a send with no corresponding posted receive, then CANCEL must succeed. Recognition that CANCEL may be a very expensive operation. For example it may require an interrupt drive mechanism. What is the effect (cost) on normal communication? Straw Vote: CANCEL? - ---------- Yes: 14 No: 6 Type Mismatch - ------------- Suppose send 4 bytes and receive integer or conversely. Is this unsafe or erroneous? Various proposals: type mismatch is always erroneous; type mismatch is never erroneous; BYTE type is an escape hatch. Another proposal is to allow type conversion as well. Straw Vote: - ---------- 1. type mismatch always erroneous 2 2. type mismatch erroneous except BYTE 12 3. never erroneous 9 - ------------------------------------------------------------------------------- Report on MPI -1.2 - ------------------------------------------------------------------------------- Jim Cownie presented. Procedure clarification: All official votes require majority of ABSTENTIONS. Added Features Insecure send for users who lack confidence Tags are opaque objects Message data is opaque All lengths are in bits, measure as floats (for sufficient precision) Messages are not passed in envelopes (they are too small), but packing crates. Context proposal number server - returns 64 bit integer (expected to last at least 1 week) Group simplification: maximum number of elements in a group is 1 All communications occur in a group From owner-mpi-comm@CS.UTK.EDU Mon Aug 9 22:20:29 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA16990; Mon, 9 Aug 93 22:20:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21096; Mon, 9 Aug 93 22:18:03 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 9 Aug 1993 22:18:02 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from noc.usfca.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21088; Mon, 9 Aug 93 22:17:59 -0400 Received: by noc.usfca.edu (5.65/DEC-Ultrix/4.3) id AA16320; Mon, 9 Aug 1993 19:20:01 -0700 Received: by mobydick.usfca.edu.usfca.edu (4.1/SMI-4.1) id AA01133; Mon, 9 Aug 93 19:17:23 PDT Date: Mon, 9 Aug 93 19:17:23 PDT From: peter@mobydick.usfca.edu (Peter Pacheco) Message-Id: <9308100217.AA01133@mobydick.usfca.edu.usfca.edu> To: mpi-comm@cs.utk.edu Subject: user's guide Cc: peter@mobydick.usfca.edu Salutations! If we're to have a successful standard, it seems very important that we have a User's Guide. So I thought I would try to get the ball rolling on this. I've appended a draft table of contents with a few notes. We definitely need suggestions for sample programs that we can include in the Guide. At the beginning of the Guide, it would be good to have some short, easy programs to get the user started. Later on, of course, it would be good to have some longer, more sophisticated programs. Throughout the Guide, it would be useful to have short examples illustrating various features of MPI. So send your questions, comments, criticisms . . . Do your worst, Peter Pacheco Department of Mathematics University of San Francisco San Francisco, CA 94117 (415) 666-6630 peter@mobydick.usfca.edu Introduction. [Brief discussion of history and purpose of MPI.] [Quick overview of MPI.] [Pointers to literature on parallel processing for those users who need more background before reading the Guide.] Chapter I. Basic message passing using MPI. 1. MPI_SEND and MPI_RECV. 2. Basic program. [How about calculating a definite integral using the trapezoidal rule. Include source code with mpi_init, or whatever is decided on. Will we also need to call mpi_iomode?] 3. More short examples. 4. Some short exercises. Chapter 2. Collective communication using MPI. 1. Making the basic program run faster. [Assuming trapezoidal rule:] (a) Roll-your-own global sum. (b) MPI_REDUCE. 2. Other MPI collective communication routines and examples. [Defer detailed discussion of some routines to appendix?] (a) Barrier. (b) Broadcast. (c) Gather. (d) Scatter. (e) Scatter_gather. (f) All_cast. (h) Scan. 3. Short exercises. Chapter 3. Writing larger programs with MPI. 1. Communicators (using naive user functions). 2. Topologies. 3. Writing programs that partition data. [A possible example: Solve a pde using finite differences on a mesh.] 4. Writing programs that partition control. [A possible example: Supervisor-worker type problem, e.g., find the chromatic number of a graph using the chromatic polynomial.] 5. Other Examples. 6. Estimating Performance. 7. Short Exercises [on use of communicators, etc.] Chapter 4. Debugging and Profiling. [Do we need this chapter? MPI, of course, mandates nothing about debuggers or profilers. However, if the Guide is to be useful to the infamous "naive" user, it might be a good idea to include something here.] Chapter 5. Making your program run faster. 1. A quick overview of what happens when a message is sent. 2. Non-blocking communication in MPI. 3. User-defined buffering in MPI. 4. Parallel libraries and contexts. 5. Exercises. Chapter 6. Everything you always wanted to know about MPI. 1. Environmental enquiry functions. 2. More on communicators, groups, and contexts. 3. Exercises. [Do we want a section on what's NOT in MPI, e.g., parallel i/o? It also might be appropriate to include a section with pointers to the literature on parallel algorithms.] Appendix A. A summary of MPI. Bibliography. From owner-mpi-comm@CS.UTK.EDU Wed Aug 11 11:04:58 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA03724; Wed, 11 Aug 93 11:04:58 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24741; Wed, 11 Aug 93 11:01:42 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 11 Aug 1993 11:01:40 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA24726; Wed, 11 Aug 93 11:01:39 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01830; Wed, 11 Aug 93 10:01:37 CDT Date: Wed, 11 Aug 93 10:01:37 CDT From: Tony Skjellum Message-Id: <9308111501.AA01830@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: Netlib MPI Way out of Date Everyone... I have just seen the context chapter from netlib, which someone has brought to the meeting. It is not even the June 24 version of our document, it is something from May. Netlib needs to be up-to-date, or we need to rely solely on anonymous ftp. People are asking me questions about versions of MPI drafts long dead. No wonder there is added confusion. Can this be fixed? Tony From owner-mpi-comm@CS.UTK.EDU Mon Aug 16 09:11:38 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA23739; Mon, 16 Aug 93 09:11:38 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04143; Mon, 16 Aug 93 09:09:37 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 16 Aug 1993 09:09:36 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA04135; Mon, 16 Aug 93 09:09:35 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA23530; Mon, 16 Aug 1993 09:09:34 -0400 Message-Id: <9308161309.AA23530@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Fortran 90 bindings Date: Mon, 16 Aug 93 09:09:33 -0500 From: David W. Walker ------- Forwarded Message Received: from sun2.nsfnet-relay.ac.uk by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA14642; Wed, 11 Aug 1993 06:29:54 -0400 Via: uk.ac.edinburgh.castle; Wed, 11 Aug 1993 11:29:10 +0100 Date: 11 Aug 93 11:27:35 BST From: D Muxworthy Subject: Guidelines for Bindings to Fortran 90 To: mjhanna@somnet.sandia.gov, benson@buffer.enet.dec.com, als@cs.umd.edu, walker@rios2.epm.ornl.gov, donn@bix.com Message-Id: <9308111127.aa25840@uk.ac.ed.castle> Status: O You might be interested to know, or you might not, that the SC22/WG5 bindings guidelines document is available as from today in compressed postscript form from the NCSA anonymous ftp server. server: ftp.ncsa.uiuc.edu file: sc22wg5/N938.ps.Z (use upper and lower case letters exactly as shown) Best wishes, David Muxworthy ------- End of Forwarded Message From owner-mpi-comm@CS.UTK.EDU Tue Aug 17 00:59:20 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA00684; Tue, 17 Aug 93 00:59:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02644; Tue, 17 Aug 93 00:57:02 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 17 Aug 1993 00:56:56 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02631; Tue, 17 Aug 93 00:56:53 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA19016; Mon, 16 Aug 93 23:56:51 CDT Date: Mon, 16 Aug 93 23:56:51 CDT From: Tony Skjellum Message-Id: <9308170456.AA19016@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: proposal: comp.parallel.mpi Colleagues: I would like to have an unmoderated group, like Jack's comp.parallel.pvm, so that post-MPI-meetings, we can promulgate info in a timely and efficient manner. Comments? Anyone willing to give me the text that was used to propose comp.parallel.pvm? Thanks, Tony From owner-mpi-comm@CS.UTK.EDU Tue Aug 17 08:01:34 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA01915; Tue, 17 Aug 93 08:01:34 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29925; Tue, 17 Aug 93 07:59:28 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 17 Aug 1993 07:59:27 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA29917; Tue, 17 Aug 93 07:59:26 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13869; Tue, 17 Aug 1993 07:59:25 -0400 Message-Id: <9308171159.AA13869@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: comp.parallel.mpi Date: Tue, 17 Aug 93 07:59:24 -0500 From: David W. Walker I think Tony's suggestion of a comp.parallel.mpi group is a good idea, but only after the MPI draft has been presented at Supercomputing 93. If it's established before this I think it will just increase the noise level. There's little point, IMHO, of involving "outsiders" at this stage. Regards, David From owner-mpi-comm@CS.UTK.EDU Tue Aug 17 09:54:02 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02883; Tue, 17 Aug 93 09:54:02 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06781; Tue, 17 Aug 93 09:51:01 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 17 Aug 1993 09:51:00 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06768; Tue, 17 Aug 93 09:50:58 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA19531; Tue, 17 Aug 93 08:50:57 CDT Date: Tue, 17 Aug 93 08:50:57 CDT From: Tony Skjellum Message-Id: <9308171350.AA19531@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: comp.parallel.mpi Given the latency of setting up a newsgroup, I will try to time it so it comes into existence after SC93. - Tony From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 08:27:44 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA07742; Fri, 27 Aug 93 08:27:44 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12502; Fri, 27 Aug 93 08:24:43 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 08:24:41 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.meiko.co.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12494; Fri, 27 Aug 93 08:24:35 -0400 Received: from tycho.co.uk (tycho.meiko.co.uk) by hub.meiko.co.uk with SMTP id AA17512 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Fri, 27 Aug 1993 13:24:32 +0100 Date: Fri, 27 Aug 1993 13:24:32 +0100 From: James Cownie Message-Id: <199308271224.AA17512@hub.meiko.co.uk> Received: by tycho.co.uk (5.0/SMI-SVR4) id AA00275; Fri, 27 Aug 93 13:23:06 BST To: mpi-comm@cs.utk.edu Subject: T-Shirts ! Content-Length: 713 If we're serious about a T shirt we probably need to get our act together NOW, for the next meeting. The reasons are 1) It would be good to have them before SC93 2) If the final meeting is in Europe, then there may be hassle over importing large numbers of T shirts. 3) U.S T shirts are better than European ones anyway ! I'm on vacation now for two weeks, and have no strong opinion on design, however I'm sure lots of you do. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 12:41:16 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA09889; Fri, 27 Aug 93 12:41:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16516; Fri, 27 Aug 93 12:37:51 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 12:37:49 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA16508; Fri, 27 Aug 93 12:37:46 -0400 Received: from macaw.fsl.noaa.gov by fslg8.fsl.noaa.gov (5.65/Ultrix3.0-C) id AA02850; Fri, 27 Aug 1993 15:37:34 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA00580; Thu, 26 Aug 93 17:10:23 MDT Date: Thu, 26 Aug 93 17:10:23 MDT From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9308262310.AA00580@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, jim@meiko.co.uk Subject: Re: T-Shirts ! > If we're serious about a T shirt we probably need to get our act > together NOW, for the next meeting. > > -- Jim Sounds like a good idea to me. Who is in charge of the T-shirt design subcommittee (he asks, not volunteering... :-)? Al? Tom From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 13:10:34 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10026; Fri, 27 Aug 93 13:10:34 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19089; Fri, 27 Aug 93 13:08:09 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 13:08:07 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19073; Fri, 27 Aug 93 13:08:04 -0400 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA29076; Fri, 27 Aug 93 12:07:48 -0500 Received: by mozart.convex.com (5.64/1.28) id AA17459; Fri, 27 Aug 93 12:07:07 -0500 From: joelw@mozart.convex.com (Joel Williamson) Message-Id: <9308271707.AA17459@mozart.convex.com> Subject: Re: T-Shirts ! To: hender@macaw.fsl.noaa.gov (Tom Henderson) Date: Fri, 27 Aug 93 12:07:06 CDT Cc: mpi-comm@cs.utk.edu, jim@meiko.co.uk In-Reply-To: <9308262310.AA00580@macaw.fsl.noaa.gov>; from "Tom Henderson" at Aug 26, 93 5:10 pm X-Mailer: ELM [version 2.3 PL11] At HPFF we simply passed around the enclosed Tshirt design file, finding and replacing text and submitting it back to the group for praise/derision. Eventually the group chairman Chuck Keolbel called for a vote. So just strip of this header, and lpr the remaining to a postscript printer, modify and ship it back out. Perhaps we can come up with something more bold than "Don't blame me, I didn't vote for that feature." Best regards, Joel Williamson --------------cut here-------------------------- %!PS-Adobe-2.0 EPSF-1.2 %%DocumentFonts: Helvetica-Bold %%Pages: 1 %%BoundingBox: 5 59 540 782 %%EndComments 50 dict begin /arrowHeight 8 def /arrowWidth 4 def /none null def /numGraphicParameters 17 def /stringLimit 65535 def /Begin { save numGraphicParameters dict begin } def /End { end restore } def /SetB { dup type /nulltype eq { pop false /brushRightArrow idef false /brushLeftArrow idef true /brushNone idef } { /brushDashOffset idef /brushDashArray idef 0 ne /brushRightArrow idef 0 ne /brushLeftArrow idef /brushWidth idef false /brushNone idef } ifelse } def /SetCFg { /fgblue idef /fggreen idef /fgred idef } def /SetCBg { /bgblue idef /bggreen idef /bgred idef } def /SetF { /printSize idef /printFont idef } def /SetP { dup type /nulltype eq { pop true /patternNone idef } { /patternGrayLevel idef patternGrayLevel -1 eq { /patternString idef } if false /patternNone idef } ifelse } def /BSpl { 0 begin storexyn newpath n 1 gt { 0 0 0 0 0 0 1 1 true subspline n 2 gt { 0 0 0 0 1 1 2 2 false subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 2 copy false subspline } if n 2 sub dup n 1 sub dup 2 copy 2 copy false subspline patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Circ { newpath 0 360 arc patternNone not { ifill } if brushNone not { istroke } if } def /CBSpl { 0 begin dup 2 gt { storexyn newpath n 1 sub dup 0 0 1 1 2 2 true subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 0 0 false subspline n 2 sub dup n 1 sub dup 0 0 1 1 false subspline patternNone not { ifill } if brushNone not { istroke } if } { Poly } ifelse end } dup 0 4 dict put def /Elli { 0 begin newpath 4 2 roll translate scale 0 0 1 0 360 arc patternNone not { ifill } if brushNone not { istroke } if end } dup 0 1 dict put def /Line { 0 begin 2 storexyn newpath x 0 get y 0 get moveto x 1 get y 1 get lineto brushNone not { istroke } if 0 0 1 1 leftarrow 0 0 1 1 rightarrow end } dup 0 4 dict put def /MLine { 0 begin storexyn newpath n 1 gt { x 0 get y 0 get moveto 1 1 n 1 sub { /i exch def x i get y i get lineto } for patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Poly { 3 1 roll newpath moveto -1 add { lineto } repeat closepath patternNone not { ifill } if brushNone not { istroke } if } def /Rect { 0 begin /t exch def /r exch def /b exch def /l exch def newpath l b moveto l t lineto r t lineto r b lineto closepath patternNone not { ifill } if brushNone not { istroke } if end } dup 0 4 dict put def /Text { ishow } def /idef { dup where { pop pop pop } { exch def } ifelse } def /ifill { 0 begin gsave patternGrayLevel -1 ne { fgred bgred fgred sub patternGrayLevel mul add fggreen bggreen fggreen sub patternGrayLevel mul add fgblue bgblue fgblue sub patternGrayLevel mul add setrgbcolor eofill } { eoclip originalCTM setmatrix pathbbox /t exch def /r exch def /b exch def /l exch def /w r l sub ceiling cvi def /h t b sub ceiling cvi def /imageByteWidth w 8 div ceiling cvi def /imageHeight h def bgred bggreen bgblue setrgbcolor eofill fgred fggreen fgblue setrgbcolor w 0 gt h 0 gt and { l b translate w h scale w h true [w 0 0 h neg 0 h] { patternproc } imagemask } if } ifelse grestore end } dup 0 8 dict put def /istroke { gsave brushDashOffset -1 eq { [] 0 setdash 1 setgray } { brushDashArray brushDashOffset setdash fgred fggreen fgblue setrgbcolor } ifelse brushWidth setlinewidth originalCTM setmatrix stroke grestore } def /ishow { 0 begin gsave fgred fggreen fgblue setrgbcolor /fontDict printFont findfont printSize scalefont dup setfont def /descender fontDict begin 0 [FontBBox] 1 get FontMatrix end transform exch pop def /vertoffset 0 descender sub printSize sub printFont /Courier ne printFont /Courier-Bold ne and { 1 add } if def { 0 vertoffset moveto show /vertoffset vertoffset printSize sub def } forall grestore end } dup 0 3 dict put def /patternproc { 0 begin /patternByteLength patternString length def /patternHeight patternByteLength 8 mul sqrt cvi def /patternWidth patternHeight def /patternByteWidth patternWidth 8 idiv def /imageByteMaxLength imageByteWidth imageHeight mul stringLimit patternByteWidth sub min def /imageMaxHeight imageByteMaxLength imageByteWidth idiv patternHeight idiv patternHeight mul patternHeight max def /imageHeight imageHeight imageMaxHeight sub store /imageString imageByteWidth imageMaxHeight mul patternByteWidth add string def 0 1 imageMaxHeight 1 sub { /y exch def /patternRow y patternByteWidth mul patternByteLength mod def /patternRowString patternString patternRow patternByteWidth getinterval def /imageRow y imageByteWidth mul def 0 patternByteWidth imageByteWidth 1 sub { /x exch def imageString imageRow x add patternRowString putinterval } for } for imageString end } dup 0 12 dict put def /min { dup 3 2 roll dup 4 3 roll lt { exch } if pop } def /max { dup 3 2 roll dup 4 3 roll gt { exch } if pop } def /arrowhead { 0 begin transform originalCTM itransform /taily exch def /tailx exch def transform originalCTM itransform /tipy exch def /tipx exch def /dy tipy taily sub def /dx tipx tailx sub def /angle dx 0 ne dy 0 ne or { dy dx atan } { 90 } ifelse def gsave originalCTM setmatrix tipx tipy translate angle rotate newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath patternNone not { originalCTM setmatrix /padtip arrowHeight 2 exp 0.25 arrowWidth 2 exp mul add sqrt brushWidth mul arrowWidth div def /padtail brushWidth 2 div def tipx tipy translate angle rotate padtip 0 translate arrowHeight padtip add padtail add arrowHeight div dup scale arrowheadpath ifill } if brushNone not { originalCTM setmatrix tipx tipy translate angle rotate arrowheadpath istroke } if grestore end } dup 0 9 dict put def /arrowheadpath { newpath 0 0 moveto arrowHeight neg arrowWidth 2 div lineto arrowHeight neg arrowWidth 2 div neg lineto closepath } def /leftarrow { 0 begin y exch get /taily exch def x exch get /tailx exch def y exch get /tipy exch def x exch get /tipx exch def brushLeftArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /rightarrow { 0 begin y exch get /tipy exch def x exch get /tipx exch def y exch get /taily exch def x exch get /tailx exch def brushRightArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /midpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 x1 add 2 div y0 y1 add 2 div end } dup 0 4 dict put def /thirdpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 2 mul x1 add 3 div y0 2 mul y1 add 3 div end } dup 0 4 dict put def /subspline { 0 begin /movetoNeeded exch def y exch get /y3 exch def x exch get /x3 exch def y exch get /y2 exch def x exch get /x2 exch def y exch get /y1 exch def x exch get /x1 exch def y exch get /y0 exch def x exch get /x0 exch def x1 y1 x2 y2 thirdpoint /p1y exch def /p1x exch def x2 y2 x1 y1 thirdpoint /p2y exch def /p2x exch def x1 y1 x0 y0 thirdpoint p1x p1y midpoint /p0y exch def /p0x exch def x2 y2 x3 y3 thirdpoint p2x p2y midpoint /p3y exch def /p3x exch def movetoNeeded { p0x p0y moveto } if p1x p1y p2x p2y p3x p3y curveto end } dup 0 17 dict put def /storexyn { /n exch def /y n array def /x n array def n 1 sub -1 0 { /i exch def y i 3 2 roll put x i 3 2 roll put } for } def %%EndProlog %I Idraw 7 Grid 8 %%Page: 1 1 Begin %I b u %I cfg u %I cbg u %I f u %I p u %I t [ 0.8 0 0 0.8 0 0 ] concat /originalCTM matrix currentmatrix def Begin %I MLine %I b 65535 1 0 0 [] 0 SetB %I cfg Black 0 0 0 SetCFg %I cbg White 1 1 1 SetCBg %I p 1 SetP %I t [ 1 0 0 1 20 425.5 ] concat %I 22 66 444 76 436 129 403 153 432 153 135 445 134 447 436 465 412 522 457 466 528 398 549 360 551 335 530 306 522 283 523 264 529 246 546 199 544 120 520 65 444 81 432 82 432 22 MLine End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 3.30769 0 0 2.11765 493 787.029 ] concat %I [ (FRONT) ] Text End Begin %I MLine %I b 65535 1 0 0 [] 0 SetB %I cfg Black 0 0 0 SetCFg %I cbg White 1 1 1 SetCBg %I p 1 SetP %I t [ 1 0 0 1 22 -58.5 ] concat %I 22 66 444 76 436 129 403 153 432 153 135 445 134 447 436 465 412 522 457 466 528 398 549 360 551 335 530 306 522 283 523 264 529 246 546 199 544 120 520 65 444 81 432 82 432 22 MLine End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 3.11905 0 0 2.76471 546 262.441 ] concat %I [ (BACK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1 0 0 2.52941 233 362.382 ] concat %I [ (Don't blame me...) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1 0 0 1.94118 234 275.676 ] concat %I [ (I didn't vote for that feature.) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 4.59975 0 0 4.77176 231.144 821.185 ] concat %I [ (HPFF) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1.08333 0 0 1 234.581 843.5 ] concat %I [ (HIGH PERFORMANCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 1 0 0 1 234.539 766.5 ] concat %I [ (FORTRAN FORUM 1992) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* /Helvetica-Bold 14 SetF %I t [ 2.7907 0 0 4.76471 8.5 923.206 ] concat %I [ (#3) ] Text End End %I eop showpage %%Trailer end From owner-mpi-comm@CS.UTK.EDU Fri Aug 27 13:59:55 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10208; Fri, 27 Aug 93 13:59:55 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22983; Fri, 27 Aug 93 13:57:52 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 27 Aug 1993 13:57:51 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22973; Fri, 27 Aug 93 13:57:50 -0400 Received: by msr.EPM.ORNL.GOV (4.1/1.34) id AA24054; Fri, 27 Aug 93 13:57:49 EDT Date: Fri, 27 Aug 93 13:57:49 EDT From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9308271757.AA24054@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: T-shirts >Who is in charge of the T-shirt design? Al? I am working on it. I have a design down on paper - need to turn into postscript. Al From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 10:27:45 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA18310; Thu, 9 Sep 93 10:27:45 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11629; Thu, 9 Sep 93 10:21:24 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 10:21:21 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sun4.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11606; Thu, 9 Sep 93 10:21:13 -0400 Received: by sun4.epm.ornl.gov (4.1/1.34) id AA21778; Thu, 9 Sep 93 10:21:10 EDT Date: Thu, 9 Sep 93 10:21:10 EDT From: geist@sun4.epm.ornl.gov (Al Geist) Message-Id: <9309091421.AA21778@sun4.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Proposed T-shirt Design Front with back quote. Hi Folks, Here is a possible T-shirt design. There are still a few routines to add that have not been presented to committee yet. The top choice for qoute on the back is: MPI: Defining The Current Practice of Tomorrow Today Runner-up is: Entia non sunt multiplicanda praeter necessitatem I want to stay away from negative comments on MPI or other systems. T-shirts R us Al Geist ------------- %!PS-Adobe-2.0 EPSF-1.2 %%DocumentFonts: Helvetica-Bold %%Pages: 1 %%BoundingBox: 32 2 589 789 %%EndComments /arrowHeight 10 def /arrowWidth 5 def /IdrawDict 51 dict def IdrawDict begin /reencodeISO { dup dup findfont dup length dict begin { 1 index /FID ne { def }{ pop pop } ifelse } forall /Encoding ISOLatin1Encoding def currentdict end definefont } def /ISOLatin1Encoding [ /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /space/exclam/quotedbl/numbersign/dollar/percent/ampersand/quoteright /parenleft/parenright/asterisk/plus/comma/minus/period/slash /zero/one/two/three/four/five/six/seven/eight/nine/colon/semicolon /less/equal/greater/question/at/A/B/C/D/E/F/G/H/I/J/K/L/M/N /O/P/Q/R/S/T/U/V/W/X/Y/Z/bracketleft/backslash/bracketright /asciicircum/underscore/quoteleft/a/b/c/d/e/f/g/h/i/j/k/l/m /n/o/p/q/r/s/t/u/v/w/x/y/z/braceleft/bar/braceright/asciitilde /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef/.notdef /.notdef/dotlessi/grave/acute/circumflex/tilde/macron/breve /dotaccent/dieresis/.notdef/ring/cedilla/.notdef/hungarumlaut /ogonek/caron/space/exclamdown/cent/sterling/currency/yen/brokenbar /section/dieresis/copyright/ordfeminine/guillemotleft/logicalnot /hyphen/registered/macron/degree/plusminus/twosuperior/threesuperior /acute/mu/paragraph/periodcentered/cedilla/onesuperior/ordmasculine /guillemotright/onequarter/onehalf/threequarters/questiondown /Agrave/Aacute/Acircumflex/Atilde/Adieresis/Aring/AE/Ccedilla /Egrave/Eacute/Ecircumflex/Edieresis/Igrave/Iacute/Icircumflex /Idieresis/Eth/Ntilde/Ograve/Oacute/Ocircumflex/Otilde/Odieresis /multiply/Oslash/Ugrave/Uacute/Ucircumflex/Udieresis/Yacute /Thorn/germandbls/agrave/aacute/acircumflex/atilde/adieresis /aring/ae/ccedilla/egrave/eacute/ecircumflex/edieresis/igrave /iacute/icircumflex/idieresis/eth/ntilde/ograve/oacute/ocircumflex /otilde/odieresis/divide/oslash/ugrave/uacute/ucircumflex/udieresis /yacute/thorn/ydieresis ] def /Helvetica-Bold reencodeISO def /none null def /numGraphicParameters 17 def /stringLimit 65535 def /Begin { save numGraphicParameters dict begin } def /End { end restore } def /SetB { dup type /nulltype eq { pop false /brushRightArrow idef false /brushLeftArrow idef true /brushNone idef } { /brushDashOffset idef /brushDashArray idef 0 ne /brushRightArrow idef 0 ne /brushLeftArrow idef /brushWidth idef false /brushNone idef } ifelse } def /SetCFg { /fgblue idef /fggreen idef /fgred idef } def /SetCBg { /bgblue idef /bggreen idef /bgred idef } def /SetF { /printSize idef /printFont idef } def /SetP { dup type /nulltype eq { pop true /patternNone idef } { dup -1 eq { /patternGrayLevel idef /patternString idef } { /patternGrayLevel idef } ifelse false /patternNone idef } ifelse } def /BSpl { 0 begin storexyn newpath n 1 gt { 0 0 0 0 0 0 1 1 true subspline n 2 gt { 0 0 0 0 1 1 2 2 false subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 2 copy false subspline } if n 2 sub dup n 1 sub dup 2 copy 2 copy false subspline patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Circ { newpath 0 360 arc patternNone not { ifill } if brushNone not { istroke } if } def /CBSpl { 0 begin dup 2 gt { storexyn newpath n 1 sub dup 0 0 1 1 2 2 true subspline 1 1 n 3 sub { /i exch def i 1 sub dup i dup i 1 add dup i 2 add dup false subspline } for n 3 sub dup n 2 sub dup n 1 sub dup 0 0 false subspline n 2 sub dup n 1 sub dup 0 0 1 1 false subspline patternNone not { ifill } if brushNone not { istroke } if } { Poly } ifelse end } dup 0 4 dict put def /Elli { 0 begin newpath 4 2 roll translate scale 0 0 1 0 360 arc patternNone not { ifill } if brushNone not { istroke } if end } dup 0 1 dict put def /Line { 0 begin 2 storexyn newpath x 0 get y 0 get moveto x 1 get y 1 get lineto brushNone not { istroke } if 0 0 1 1 leftarrow 0 0 1 1 rightarrow end } dup 0 4 dict put def /MLine { 0 begin storexyn newpath n 1 gt { x 0 get y 0 get moveto 1 1 n 1 sub { /i exch def x i get y i get lineto } for patternNone not brushLeftArrow not brushRightArrow not and and { ifill } if brushNone not { istroke } if 0 0 1 1 leftarrow n 2 sub dup n 1 sub dup rightarrow } if end } dup 0 4 dict put def /Poly { 3 1 roll newpath moveto -1 add { lineto } repeat closepath patternNone not { ifill } if brushNone not { istroke } if } def /Rect { 0 begin /t exch def /r exch def /b exch def /l exch def newpath l b moveto l t lineto r t lineto r b lineto closepath patternNone not { ifill } if brushNone not { istroke } if end } dup 0 4 dict put def /Text { ishow } def /idef { dup where { pop pop pop } { exch def } ifelse } def /ifill { 0 begin gsave patternGrayLevel -1 ne { fgred bgred fgred sub patternGrayLevel mul add fggreen bggreen fggreen sub patternGrayLevel mul add fgblue bgblue fgblue sub patternGrayLevel mul add setrgbcolor eofill } { eoclip originalCTM setmatrix pathbbox /t exch def /r exch def /b exch def /l exch def /w r l sub ceiling cvi def /h t b sub ceiling cvi def /imageByteWidth w 8 div ceiling cvi def /imageHeight h def bgred bggreen bgblue setrgbcolor eofill fgred fggreen fgblue setrgbcolor w 0 gt h 0 gt and { l b translate w h scale w h true [w 0 0 h neg 0 h] { patternproc } imagemask } if } ifelse grestore end } dup 0 8 dict put def /istroke { gsave brushDashOffset -1 eq { [] 0 setdash 1 setgray } { brushDashArray brushDashOffset setdash fgred fggreen fgblue setrgbcolor } ifelse brushWidth setlinewidth originalCTM setmatrix stroke grestore } def /ishow { 0 begin gsave fgred fggreen fgblue setrgbcolor /fontDict printFont printSize scalefont dup setfont def /descender fontDict begin 0 [FontBBox] 1 get FontMatrix end transform exch pop def /vertoffset 1 printSize sub descender sub def { 0 vertoffset moveto show /vertoffset vertoffset printSize sub def } forall grestore end } dup 0 3 dict put def /patternproc { 0 begin /patternByteLength patternString length def /patternHeight patternByteLength 8 mul sqrt cvi def /patternWidth patternHeight def /patternByteWidth patternWidth 8 idiv def /imageByteMaxLength imageByteWidth imageHeight mul stringLimit patternByteWidth sub min def /imageMaxHeight imageByteMaxLength imageByteWidth idiv patternHeight idiv patternHeight mul patternHeight max def /imageHeight imageHeight imageMaxHeight sub store /imageString imageByteWidth imageMaxHeight mul patternByteWidth add string def 0 1 imageMaxHeight 1 sub { /y exch def /patternRow y patternByteWidth mul patternByteLength mod def /patternRowString patternString patternRow patternByteWidth getinterval def /imageRow y imageByteWidth mul def 0 patternByteWidth imageByteWidth 1 sub { /x exch def imageString imageRow x add patternRowString putinterval } for } for imageString end } dup 0 12 dict put def /min { dup 3 2 roll dup 4 3 roll lt { exch } if pop } def /max { dup 3 2 roll dup 4 3 roll gt { exch } if pop } def /midpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 x1 add 2 div y0 y1 add 2 div end } dup 0 4 dict put def /thirdpoint { 0 begin /y1 exch def /x1 exch def /y0 exch def /x0 exch def x0 2 mul x1 add 3 div y0 2 mul y1 add 3 div end } dup 0 4 dict put def /subspline { 0 begin /movetoNeeded exch def y exch get /y3 exch def x exch get /x3 exch def y exch get /y2 exch def x exch get /x2 exch def y exch get /y1 exch def x exch get /x1 exch def y exch get /y0 exch def x exch get /x0 exch def x1 y1 x2 y2 thirdpoint /p1y exch def /p1x exch def x2 y2 x1 y1 thirdpoint /p2y exch def /p2x exch def x1 y1 x0 y0 thirdpoint p1x p1y midpoint /p0y exch def /p0x exch def x2 y2 x3 y3 thirdpoint p2x p2y midpoint /p3y exch def /p3x exch def movetoNeeded { p0x p0y moveto } if p1x p1y p2x p2y p3x p3y curveto end } dup 0 17 dict put def /storexyn { /n exch def /y n array def /x n array def n 1 sub -1 0 { /i exch def y i 3 2 roll put x i 3 2 roll put } for } def %%EndProlog %%BeginIdrawPrologue /arrowhead { 0 begin transform originalCTM itransform /taily exch def /tailx exch def transform originalCTM itransform /tipy exch def /tipx exch def /dy tipy taily sub def /dx tipx tailx sub def /angle dx 0 ne dy 0 ne or { dy dx atan } { 90 } ifelse def gsave originalCTM setmatrix tipx tipy translate angle rotate newpath arrowHeight neg arrowWidth 2 div moveto 0 0 lineto arrowHeight neg arrowWidth 2 div neg lineto patternNone not { originalCTM setmatrix /padtip arrowHeight 2 exp 0.25 arrowWidth 2 exp mul add sqrt brushWidth mul arrowWidth div def /padtail brushWidth 2 div def tipx tipy translate angle rotate padtip 0 translate arrowHeight padtip add padtail add arrowHeight div dup scale arrowheadpath ifill } if brushNone not { originalCTM setmatrix tipx tipy translate angle rotate arrowheadpath istroke } if grestore end } dup 0 9 dict put def /arrowheadpath { newpath arrowHeight neg arrowWidth 2 div moveto 0 0 lineto arrowHeight neg arrowWidth 2 div neg lineto } def /leftarrow { 0 begin y exch get /taily exch def x exch get /tailx exch def y exch get /tipy exch def x exch get /tipx exch def brushLeftArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def /rightarrow { 0 begin y exch get /tipy exch def x exch get /tipx exch def y exch get /taily exch def x exch get /tailx exch def brushRightArrow { tipx tipy tailx taily arrowhead } if end } dup 0 4 dict put def %%EndIdrawPrologue %I Idraw 10 Grid 8 8 %%Page: 1 1 Begin %I b u %I cfg u %I cbg u %I f u %I p u %I t [ 0.799705 0 0 0.799705 0 0 ] concat /originalCTM matrix currentmatrix def Begin %I Pict %I b u %I cfg u %I cbg u %I f u %I p u %I t [ 1 0 0 1 32 4 ] concat Begin %I Text %I cfg Green 0 1 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 2.61386e-08 2.14815 -2.14815 2.61386e-08 637.259 77.1852 ] concat %I [ (Back:) ] Text End Begin %I Text %I cfg Red 1 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 2.06326e-08 1.69565 -1.69565 2.06326e-08 634.783 209 ] concat %I [ ( MPI: Defining ) (The current Practice of Tomorrow) ( Today.) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 129 551 ] concat %I [ (MPI_ADDRESS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.974059 0.226296 -0.226296 0.974059 249.653 331.169 ] concat %I [ (MPI_ALLCAST) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 -3.55271e-14 3.55271e-14 1 364 955 ] concat %I [ (MPI_ALLREDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 239 557 ] concat %I [ (MPI_ALLTOALL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.903278 0.429057 -0.429057 -0.903278 141.341 271.178 ] concat %I [ (MPI_BCAST) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 195 489 ] concat %I [ (MPI_ATTRIBUTE_ALLOC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 41 523 ] concat %I [ (MPI_ATTRIBUTE_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 458 138 ] concat %I [ (MPI_BARRIER) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 112 32.0004 ] concat %I [ (MPI_CANCEL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 373 491 ] concat %I [ (MPI_CART) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.971508 0.237007 -0.237007 -0.971508 220.936 69.8524 ] concat %I [ (MPI_COLL_SUBGROUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 180 551 ] concat %I [ (MPI_COMM_CONTEXT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 570 8.99997 ] concat %I [ (MPI_COMM_DUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 474 587 ] concat %I [ (MPI_COMM_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 152 554 ] concat %I [ (MPI_COMM_GROUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.972983 0.230877 -0.230877 -0.972983 333.252 69.5133 ] concat %I [ (MPI_COMM_INIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.974479 0.224481 -0.224481 0.974479 191.075 417.444 ] concat %I [ (MPI_COMM_MAKE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 569 145 ] concat %I [ (MPI_COMM_NAME_MAKE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 567 345 ] concat %I [ (MPI_CREATE_GROUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 -3.55271e-14 3.55271e-14 1 216.5 957.5 ] concat %I [ (MPI_CREATE_RECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 563 522 ] concat %I [ (MPI_CREATE_RSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 451 580 ] concat %I [ (MPI_CREATE_SEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.0137466 0.999906 -0.999906 0.0137466 478.798 237.077 ] concat %I [ (MPI_DELETE_ATTRIBUTE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 564 839 ] concat %I [ (MPI_EXCHANGE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 449 731 ] concat %I [ (MPI_GATHER) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 114 911 ] concat %I [ (MPI_GET_ATTRIBUTE_KEY) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 336 583 ] concat %I [ (MPI_GET_COUNT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 405 147 ] concat %I [ (MPI_GET_SOURCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 372 164 ] concat %I [ (MPI_GET_TAG) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 253 737 ] concat %I [ (MPI_GRAPH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 387 699 ] concat %I [ (MPI_GRAPHDIMS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 22 545 ] concat %I [ (MPI_GROUP_DUP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.967074 0.254493 -0.254493 -0.967074 372.005 33.4863 ] concat %I [ (MPI_GROUP_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 209 552 ] concat %I [ (MPI_GROUP_RANK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 563 696 ] concat %I [ (MPI_GROUP_SIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 411 585 ] concat %I [ (MPI_INQCART) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 357 731 ] concat %I [ (MPI_INQDIM) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.941742 0.336336 -0.336336 -0.941742 284.527 288.788 ] concat %I [ (MPI_INQGRAPH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.4336e-08 -2.4336e-08 -1 116 79.0001 ] concat %I [ (MPI_INQLOC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 114 956 ] concat %I [ (MPI_INQMAP) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 473 724 ] concat %I [ (MPI_INQRANK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 335 734 ] concat %I [ (MPI_IPROBE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 46 427 ] concat %I [ (MPI_IRECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 482 975 ] concat %I [ (MPI_IRSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 212 750 ] concat %I [ (MPI_ISEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 454.5 571.5 ] concat %I [ (MPI_ISSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 404 295 ] concat %I [ (MPI_MAKDIM) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 44 101 ] concat %I [ (MPI_MAP_CART) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 352.5 977.5 ] concat %I [ (MPI_MAP_GRAPH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 176 404 ] concat %I [ (MPI_PARTC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.894427 0.447214 -0.447214 0.894427 62.304 227.242 ] concat %I [ (MPI_PROBE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 324 176 ] concat %I [ (MPI_PROBE_COUNT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 373 277 ] concat %I [ (MPI_RECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.0167756 0.99986 -0.99986 0.0167756 270.195 187.091 ] concat %I [ (MPI_REDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 154 401 ] concat %I [ (MPI_RSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.966092 0.258195 -0.258195 0.966092 322.997 448.019 ] concat %I [ (MPI_SCAN) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 390 590 ] concat %I [ (MPI_SCATTER) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 193 755 ] concat %I [ (MPI_SEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 67 381 ] concat %I [ (MPI_SENDRECV) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 458 252 ] concat %I [ (MPI_SET_ATTRIBUTE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 482 130 ] concat %I [ (MPI_SHIFT_ID) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 108 401 ] concat %I [ (MPI_SSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 232 748 ] concat %I [ (MPI_START) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 114.5 820.5 ] concat %I [ (MPI_TEST) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 523 584 ] concat %I [ (MPI_TEST_ATTRIBUTE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 36 814 ] concat %I [ (MPI_TEST_CANCELLED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.886225 0.463254 -0.463254 0.886225 170.588 192.927 ] concat %I [ (MPI_TESTALL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 279 726 ] concat %I [ (MPI_TESTANY) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 46 228 ] concat %I [ (MPI_TRANSLATE_RANKS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 21.9999 239 ] concat %I [ (MPI_TYPE_COMMIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 307 908 ] concat %I [ (MPI_TYPE_CONTIGUOUS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.978234 0.207504 -0.207504 0.978234 400.91 467.592 ] concat %I [ (MPI_TYPE_EXTENT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 330.5 570.5 ] concat %I [ (MPI_TYPE_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 542 589 ] concat %I [ (MPI_TYPE_HINDEXED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 13 832 ] concat %I [ (MPI_TYPE_HVECTOR) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 433 143 ] concat %I [ (MPI_TYPE_INDEXED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 113 933 ] concat %I [ (MPI_TYPE_STRUCT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 348 174 ] concat %I [ (MPI_TYPE_VECTOR) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.980581 0.196116 -0.196116 0.980581 209.18 342.635 ] concat %I [ (MPI_USER_ALLREDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 203.5 57.5003 ] concat %I [ (MPI_USER_ALLREDUCEA) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.982102 0.188348 -0.188348 -0.982102 205.261 99.9342 ] concat %I [ (MPI_USER_REDUCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 49.4999 974.5 ] concat %I [ (MPI_USER_REDUCEA) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 22 104 ] concat %I [ (MPI_USER_SCAN) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 297 180 ] concat %I [ (MPI_USER_SCANA) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 111 799 ] concat %I [ (MPI_WAIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 434 303 ] concat %I [ (MPI_WAITALL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 525 318 ] concat %I [ (MPI_WAITANY) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 504 270 ] concat %I [ (MPI_CREATE_SSEND) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 497 592 ] concat %I [ (MPI_GROUP_UNION) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 584 773 ] concat %I [ (MPI_GROUP_INTERSECTION) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 206 10.0001 ] concat %I [ (MPI_GROUP_DIFFERENCE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 211.5 976.5 ] concat %I [ (MPI_GROUP_INCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 410 692 ] concat %I [ (MPI_GROUP_EXCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 546 182 ] concat %I [ (MPI_GROUP_RANGE_INCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.975232 0.221187 -0.221187 -0.975232 531.491 24.6688 ] concat %I [ (MPI_GROUP_RANGE_EXCL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 306 691 ] concat %I [ (MPI_GROUP_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 53.5001 777.5 ] concat %I [ (MPI_COMM_SIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 428 697 ] concat %I [ (MPI_COMM_RANK) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 16 684 ] concat %I [ (MPI_COMM_SPLIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 504 121 ] concat %I [ (MPI_COMM_MERGE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 544 384 ] concat %I [ (MPI_CONTEXTS_ALLOC) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 545 7.00015 ] concat %I [ (MPI_CONTEXTS_FREE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 586 394 ] concat %I [ (MPI_COMM_CONTEXTS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -0.961524 0.274721 -0.274721 -0.961524 504.683 3.24772 ] concat %I [ (MPI_COMM_STAT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 585 584 ] concat %I [ (MPI_INTERCOMM_MAKE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 589 185 ] concat %I [ (MPI_INTERCOMM_START) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 527 116 ] concat %I [ (MPI_INTERCOMM_FINISH) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 591 26.0002 ] concat %I [ (MPI_COMM_SPLITL) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 302.5 30.5001 ] concat %I [ (MPI_INTERCOMM_NAME) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 252 932 ] concat %I [ (MPI_INTERCOMM_NAME_START) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 430 579 ] concat %I [ (MPI_INITIALIZED) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 0.624695 0.780869 -0.780869 0.624695 57.0426 710.807 ] concat %I [ (MPI_INIT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 87.0002 391 ] concat %I [ (MPI_FINALIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 129 399 ] concat %I [ (MPI_ABORT) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 37.0002 696 ] concat %I [ (MPI_VALIDTAGS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1 0 0 1 201 470 ] concat %I [ (MPI_NUMGROUPS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 360 594 ] concat %I [ (MPI_BUFFERSIZE) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ -1 2.43359e-08 -2.43359e-08 -1 424 11.0002 ] concat %I [ (MPI_NUMCOMMUNICATORS) ] Text End Begin %I Text %I cfg Black 0 0 0 SetCFg %I f *-helvetica-bold-r-*-140-* Helvetica-Bold 14 SetF %I t [ 1.2168e-08 1 -1 1.2168e-08 23.0002 392 ] concat %I [ (MPI_ERRORMODE) ] Text End End %I eop End %I eop showpage %%Trailer end From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 12:25:28 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19205; Thu, 9 Sep 93 12:25:28 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21509; Thu, 9 Sep 93 12:22:38 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 12:22:36 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21501; Thu, 9 Sep 93 12:22:34 -0400 Received: from snacker.pnl.gov (130.20.186.18) by pnlg.pnl.gov; Thu, 9 Sep 93 09:16 PDT Received: by snacker.pnl.gov (4.1/SMI-4.1) id AA23525; Thu, 9 Sep 93 09:13:49 PDT Date: Thu, 9 Sep 93 09:13:49 PDT From: rj_littlefield@pnlg.pnl.gov Subject: tee-shirt/ghostview problem To: mpi-comm@cs.utk.edu Cc: rj_littlefield@pnlg.pnl.gov Message-Id: <9309091613.AA23525@snacker.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Al Geist sent us a PostScript file and wrote: % Here is a possible T-shirt design. Be aware that you may have trouble previewing and/or printing that file. I tried previewing with ghostview (version 1.4). It failed with the following diagnostic: Error: /undefined in 2.61386e-08 Operand stack: -savetype- -savetype- -savetype- -marktype- Execution stack: %interp_exit --nostringval-- --nostringval-- false --nostringval-- --nostringval-- --nostringval-- Dictionary stack: 399/479 7/200 37/51 1/17 0/17 5/17 Unrecoverable error, exit code 1 However, when I replaced "2.61386e-08" and all similar strings with "0.", it worked fine. Further experiments suggested that ghostview just doesn't accept negative exponents. This problem is being reported to the ghostview people. The file did print correctly on an Apple LaserWriter II and on a DEC PrintServer 20. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 13:26:26 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19595; Thu, 9 Sep 93 13:26:26 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26105; Thu, 9 Sep 93 13:23:25 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 13:23:24 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26097; Thu, 9 Sep 93 13:23:22 -0400 Date: Thu, 9 Sep 93 18:23:04 BST Message-Id: <25359.9309091723@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: tee-shirt To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Perhaps the subprograms should have arguments. Also it was suggested that there could be an MPI baseball cap with the header files on it :-) (Perhaps I can get one of these!) Anyway, seriously, it's just a bit noisy, don't you think? And, I hope the "back" text is not really a serious suggestion. Cheers Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 13:41:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA19705; Thu, 9 Sep 93 13:41:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27264; Thu, 9 Sep 93 13:38:32 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 13:38:31 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA27256; Thu, 9 Sep 93 13:38:29 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99E) id AA03403; Thu, 9 Sep 93 13:38:27 -0400 Received: from FAFNIR.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA26508; Thu, 9 Sep 93 13:38:32 -0400 From: ken@cs.cornell.edu (Ken Birman) Date: Thu, 9 Sep 93 13:38:14 -0400 Message-Id: <9309091738.AA21632@fafnir.cs.cornell.edu> Received: by fafnir.cs.cornell.edu (5.67/N-0.13) id AA21632; Thu, 9 Sep 93 13:38:14 -0400 To: mpi-comm@cs.utk.edu Subject: Could I get off of this mailing list, please? My group (Horus) hopes to provide an MPI implementation once the standard settles. But this email list seems to be mostly about t-shirt designs. Thanks! From owner-mpi-comm@CS.UTK.EDU Thu Sep 9 16:35:04 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA20563; Thu, 9 Sep 93 16:35:04 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11788; Thu, 9 Sep 93 16:33:02 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 9 Sep 1993 16:33:01 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11780; Thu, 9 Sep 93 16:33:00 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01098; Thu, 9 Sep 93 15:32:52 CDT Date: Thu, 9 Sep 93 15:32:52 CDT From: Tony Skjellum Message-Id: <9309092032.AA01098@Aurora.CS.MsState.Edu> To: ken@cs.cornell.edu Subject: Re: Could I get off of this mailing list, please? Cc: mpi-comm@cs.utk.edu Maybe you should wait for our reference implementation. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Thu Sep 16 03:48:24 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA26807; Thu, 16 Sep 93 03:48:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17304; Thu, 16 Sep 93 03:45:16 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 16 Sep 1993 03:45:14 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from THIALFI.CS.CORNELL.EDU by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17287; Thu, 16 Sep 93 03:45:12 -0400 Received: from CLOYD.CS.CORNELL.EDU by thialfi.cs.cornell.edu (5.67/I-1.99E) id AA04082; Thu, 16 Sep 93 03:45:01 -0400 Received: from ELLI.CS.CORNELL.EDU by cloyd.cs.cornell.edu (5.67/I-1.99D) id AA23988; Thu, 16 Sep 93 03:44:58 -0400 From: elster@cs.cornell.edu (Anne C. Elster) Date: Thu, 16 Sep 93 03:44:56 -0400 Message-Id: <9309160744.AA24612@elli.cs.cornell.edu> Received: by elli.cs.cornell.edu (5.67/N-0.13) id AA24612; Thu, 16 Sep 93 03:44:56 -0400 To: mpi-comm@cs.utk.edu Subject: MPI_LOAD Cc: elster@cs.cornell.edu Well, folks, I promised a nice write-up on my MPI_LOAD suggestion, but am afraid you will have to do with this patch-up job I'm throwing together in this wee hr in the morning. Sorry. I am off to Norway later to day for my Dad's 60th b-day, which also means I will not be able to make next week's meeting in Dallas. Have a nice and productive meeting! Maybe somebody could get a t-shirt sent to me? Take care! Anne ------------------------------------------------------------------------------- Anne C. Elster | (607) 254-8653 [off] 255-6236 [FAX] Xerox Design Research Institute | (607) 272-2986 [home/answ.mach] Cornell University, 502 Theory Ctr.| Ithaca, NY 14853 | e-mail: elster@cs.cornell.edu USA | na.elster@na-net.ornl.gov MPI_(CPU_)LOAD_INFO MOTIVATION Currently, most parallelized algorithms for distributed memory machines assume that they will run on a homogeneous set of processors. (I.e. that each node in the parallel system has approximately the same computing power and I/O speed.) The main focus in parallel computing has thus far been to minimize communication over-head and achieve a load-balance for the particular algorithm or application. However, as each computational node a parallel system become more complex, their individual processor/node performance characteristics will tend to deviate more as some nodes perform I/O while others concentrate on system tasks, etc. This behavior has so far been addressed through multi-tasking on large shared-memory systems such as a network of IBM 3090s, however, no simple, straight-forward solutions exist currently for the distributed memory cases. It is, however, this authors belief that load-balancing application at the ``load-info'' level (i.e. based on each node's dynamic load at run-time) will become very important in the future -- especially if there were mechanisms built into MPI that would facilitate such implementations. CURRRENT OPTIONS Options for getting load-info read into a Fortran or C application on Unix-based systems are not wonderful: System load is typically from the OS-side done by {\bf reading kernel memory (-kmem)}. This will, however, usually require {\em setgid} permissions and grubbing through kernel symbol tables. This brute-force approach might fork and exec something with the required kmem permissions. Another alternative is to use {\bf rstat} which does not require such permission, but unfortunately, is not available on all systems. Rstat typically gives you statistics from a remote kernel that should be current disregarding network delay. The third approach I can think of is to use the output from {\bf uptime}. In C, this could possibly be done though fscanf on the return value of popen. MPI SUGGESTION Since I fond none of the previous methods very programmer-friendly, my suggestion is to add a {\bf MPI\_LOAD\_INFO} that would return the desired information. As I mentioned at the last meeting, I am now convinced that the way ISIS/Horus does "load-info" for CPU load, i.e. to return the no. of jobs the scheduler has outstanding, is a good start. MPI_(CPU_)LOAD_INFO(n) OUT n : the no. of jobs the scheduler has outstanding. This way there is no timing issues associated with the local MPI call (MPI_CPU_LOAD_INFO, if you wish). The notion of needing some kind of time information was the largest objection I heard re. to my suggestion to include an MPI_LOAD_INFO back when I presented it in March. It is up to the applications programmers whether they'd want to do several calls for a statistical average. Global info. can be obtained via an MPI collective comm. call. Users will also be required to know scheduler queue sizes and what they mean for the individual sytstem/node, but at least they will not be stuck with non-standard system-supplied fucntions and "grubbing" through "kmem" tables under super-user priveleges in order to get the information they need. There are of course several other factors that influence the "load" of the processing element, e.g. memory traffic, communication traffic, I/O traffice, etc, all which could be very useful information for applications programmers to get their hands one -- especially in non-homogeneous or multi-user environments. Last time I guess some people objected to the notion of adding another MPI functions, some others felt it was not really an MPI call per se, and hence should be avoided. I can see the arguments for this, but feel the benefits of including such a functions too great to let it pas up. From owner-mpi-comm@CS.UTK.EDU Thu Sep 16 09:03:18 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA28606; Thu, 16 Sep 93 09:03:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15652; Thu, 16 Sep 93 09:00:01 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 16 Sep 1993 09:00:00 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.aggregate.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15635; Thu, 16 Sep 93 08:59:56 -0400 Received: from sirius.aggregate.com by aggregate.com (5.0/SMI-SVR4) id AA01583; Thu, 16 Sep 93 08:00:45 CDT Received: by sirius.aggregate.com (4.1/SMI-SVR4) id AA02006; Thu, 16 Sep 93 08:00:58 CDT From: ted@aggregate.com Message-Id: <9309161300.AA02006@sirius.aggregate.com> Subject: Re: MPI_LOAD To: elster@cs.cornell.edu (Anne C. Elster) Date: Thu, 16 Sep 1993 08:00:57 -0500 (CDT) Cc: mpi-comm@cs.utk.edu In-Reply-To: <9309160744.AA24612@elli.cs.cornell.edu> from "Anne C. Elster" at Sep 16, 93 03:44:56 am X-Mailer: ELM [version 2.4 PL22] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Length: 3171 Anne C. Elster writes: > [text deleted] > > MOTIVATION > > Currently, most parallelized algorithms for distributed memory machines > assume that they will run on a homogeneous set of processors. > (I.e. that each node in the parallel system has approximately > the same computing power and I/O speed.) > The main focus in parallel computing has thus far been to minimize > communication over-head and achieve a load-balance for the > particular algorithm or application. > [text deleted] > > MPI SUGGESTION > > Since I fond none of the previous methods very programmer-friendly, > my suggestion is to add a {\bf MPI\_LOAD\_INFO} that would return > the desired information. As I mentioned at the last meeting, I am now > convinced that the way ISIS/Horus does "load-info" for CPU load, i.e. > to return the no. of jobs the scheduler has outstanding, is a good start. > > MPI_(CPU_)LOAD_INFO(n) > > OUT n : the no. of jobs the scheduler has outstanding. > > This way there is no timing issues associated with the local MPI > call (MPI_CPU_LOAD_INFO, if you wish). The notion of needing some > kind of time information was the largest objection I heard re. > to my suggestion to include an MPI_LOAD_INFO back when I presented it > in March. > > It is up to the applications programmers whether they'd want to do several > calls for a statistical average. Global info. can be obtained via an MPI > collective comm. call. Users will also be required to know scheduler queue > sizes and what they mean for the individual sytstem/node, but at least they > will not be stuck with non-standard system-supplied fucntions and "grubbing" > through "kmem" tables under super-user priveleges in order to get the > information they need. [text deleted] > > Last time I guess some people objected to the notion of adding another > MPI functions, some others felt it was not really an MPI call per se, > and hence should be avoided. I can see the arguments for this, but feel > the benefits of including such a functions too great to let it pas up. I'm merely a lurker on this mailing list, so I'll let others discuss what is and isn't appropriate for MPI. However, I have written a generalized network resource monitoring library for the product I'm working on (NetShare SDK, info@aggregate.com if you are curious). In a real heterogeneous network environment you need to track more than just the load average. For example, a SPARCstation 10 with a load average of 0.6 will give a better response than an idle diskless SPARCstation ELC. Other applications may care more about memory than load -- and then physical memory versus virtual memory. Software properties like licenses and OS type and version may be important. (As an example from NetShare, which does remote execution, you may want to select hosts for a large simulation based on memory, SPECfloat rating, simulator software license availablity, and having an idle keyboard) Resource monitoring is important in a heterogeneous environment, however the simple case of monitoring a single property is not useful in practice. -- Ted Stockwell, Aggregate Computing ted@aggregate.com From owner-mpi-comm@CS.UTK.EDU Fri Sep 17 21:10:54 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA12880; Fri, 17 Sep 93 21:10:54 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19499; Fri, 17 Sep 93 20:53:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 17 Sep 1993 20:53:11 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19491; Fri, 17 Sep 93 20:53:09 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA17279; Fri, 17 Sep 93 20:53:07 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA04681; Fri, 17 Sep 93 20:53:07 EDT Date: Fri, 17 Sep 93 20:53:07 EDT From: lederman@super.org (Steve Huss-Lederman) Message-Id: <9309180053.AA04681@b125.super.org> To: mpi-comm@cs.utk.edu Subject: subset chapter Hello all, Since the upcomming meeting will have our final vote on each chapter, I thought I would let everyone know that the latest version of the subset chapter is available. To get use anonymous ftp to ftp.super.org in pub/mpi. Only the PostScript is there since there are x-refs, etc. The sections reference might differ from the various drafts of chapters that are out there; I used the latest ones I had for each. The major changes from the last meeting: - all intra-communicator and group functionality is in. all inter-communicator and caching is out. - All derived datatype stuff is in. - the new collective communication function MPI_REDUCE_SCATTER is in but MPI_USER_REDUCE_SCATTER and MPI_USER_REDUCE_SCATTERA are out. (I thought that this new name would be the longest yet in MPI but the context people still have a strong hold on this with MPI_LOCAL_SUBGROUP_EXCL_RANGES and After you add the P for profiling you get an amazing 31 characters. I'm surprised it can fit on the Tee Shirt :-) - profiling is in - Environmental management is still up in the air. We will need to discuss this at the meeting to see what we want in/out. If you have thoughts please post to mpi-iac. See you all soon in Dallas. Steve From owner-mpi-comm@CS.UTK.EDU Sun Sep 19 10:50:29 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA17234; Sun, 19 Sep 93 10:50:29 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13805; Sun, 19 Sep 93 10:44:09 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 19 Sep 1993 10:44:08 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA13797; Sun, 19 Sep 93 10:44:07 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12315; Sun, 19 Sep 1993 10:44:06 -0400 Date: Sun, 19 Sep 1993 10:44:06 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9309191444.AA12315@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Next MPI meeting ******************************************************************* ******** ******** ******** Subcommittee chairs!! Please send me agenda items ******** ******** ******** ******************************************************************* The next meeting of the Message Passing Interface (MPI) Forum will take place in Dallas, September 22-24, 1993. Details are given below. The ongoing email discussion on MPI standardization issues can be obtained from netlib. To find out what is available via email send the following message to netlib@ornl.gov: send index from mpi or use xnetlib ("send index from xnetlib" for info). MPI Forum Meeting, September 22-24, 1993 ===================================== The next meeting of the MPI forum will take place at Bristol Suites Hotel 7800 Alpha Road Dallas, Texas The meeting will start at 1pm on Wednesday September 22, 1993, and finish at noon on Friday, September 24, 1993. Rooms are $89 per night and reservations may be made by calling (214) 233-7600 (mention MPI meeting). The meeting registration fee will be $75. Please make checks and POs payable to University of Tennessee. The registration fee will be collected at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. Certain organizations need to see a registration form before giving their employees money for meetings like this. You can get such a form in PostScript after noon on Sept. 20 by sending the following message to netlib@ornl.gov: send mpi-form.ps from mpi There is no need to send me the registration form or bring it to the meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle refer to the MPI meeting. We have NOT been able to make any special arrangements with airlines to get reduced fares. We have secured limited funding from ARPA/NSF for travel expenses of MPI meeting participants who are from U.S. universities. If you would like to apply for financial support to attend the September MPI meeting please send email to me at walker@msr.epm.ornl.gov, with justification of why you need support and an estimate of your travel expenses. Please send comments and/or suggested changes to the agenda below to me at walker@msr.epm.ornl.gov Provisional Agenda for MPI Meeting, September 22-24, 1993 Wednesday TBD Thursday TBD 12:00-1:30 Lunch (provided) TBD 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday TBD From owner-mpi-comm@CS.UTK.EDU Mon Sep 20 16:41:01 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA26506; Mon, 20 Sep 93 16:41:01 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14331; Mon, 20 Sep 93 16:32:35 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 20 Sep 1993 16:32:34 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA14323; Mon, 20 Sep 93 16:32:33 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA11935; Mon, 20 Sep 1993 16:32:32 -0400 Date: Mon, 20 Sep 1993 16:32:32 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9309202032.AA11935@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Agenda for MPI meeting Here is the agenda for this week's MPI meeting. Please let me know if you want any changes made. David Agenda for MPI Meeting September 22-24, 1993 Wednesday 1:00-3:00 Environmental Inquiry Subcommittee Meeting 3:00-4:00 Language Binding Subcommittee Meeting 4:00-5:30 Full MPIF meeting to discuss Collective Communication (Geist) 5:30-6:00 The status of the MPI reference implementation (Lusk and Gropp) 6:00-7:30 Dinner break 7:30-10:30 Break up for subcommittee meetings if necessary Thursday 9:00-12:00 Full meeting to discuss contexts (2nd reading?) (Skjellum) 12:00-1:30 Lunch (provided) 1:30-3:30 Full meeting to discuss contexts (continued from morning) (Skjellum) 3:30-3:45 Break 3:45-5:00 Full meeting to discuss topologies (Hempel) 5:00-6:00 Full meeting to discuss language binding (Lusk?) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:30 Continued informal subcommittee meetings if necessary Friday 9:00-10:30 Full group meeting for second reading of subset chapter. (Huss-Lederman) 10:30-10:45 Break 10:45-12:00 Any other business From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 09:20:06 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA01211; Tue, 21 Sep 93 09:20:06 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25312; Tue, 21 Sep 93 09:14:34 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 09:14:33 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA25303; Tue, 21 Sep 93 09:14:31 -0400 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12662; Tue, 21 Sep 1993 09:14:29 -0400 Date: Tue, 21 Sep 1993 09:14:29 -0400 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9309211314.AA12662@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI Agenda: Second Version Agenda for MPI Meeting September 22-24, 1993 I can't remember what is general discussion, first reading, or second reading. I assume the subcommittee chairs know this. Wednesday 1:00-1:30 Review of the Introduction chapter, and what needs to be done to it. (Dongarra) 1:30-3:00 Full group meeting to discuss point-to-point (Snir) 3:00-4:00 Language Binding Subcommittee Meeting 4:00-5:30 Full MPIF meeting to discuss Collective Communication (Geist) 5:30-6:00 The status of the MPI reference implementation (Lusk and Gropp) 6:00-7:30 Dinner break 7:30-10:30 Environmental Inquiry Subcommittee Meeting. Other subcommittee meetings if necessary. Thursday 9:00-12:00 Full meeting to discuss contexts (2nd reading?) (Skjellum) 12:00-1:30 Lunch (provided) 1:30-3:30 Full meeting to discuss contexts (continued from morning) (Skjellum) 3:30-3:45 Break 3:45-5:00 Full meeting to discuss topologies (Hempel) 5:00-6:00 Full meeting to discuss language binding (Lusk?) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:30 Continued informal subcommittee meetings if necessary Friday 9:00-10:30 Full group meeting for second reading of subset chapter. (Huss-Lederman) 10:30-10:45 Break 10:45-12:00 Any other business From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 11:50:42 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02719; Tue, 21 Sep 93 11:50:42 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08505; Tue, 21 Sep 93 11:46:29 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 11:46:28 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08404; Tue, 21 Sep 93 11:45:34 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08316; Tue, 21 Sep 93 10:45:04 CDT Date: Tue, 21 Sep 93 10:45:04 CDT From: Tony Skjellum Message-Id: <9309211545.AA08316@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, walker@rios2.EPM.ORNL.GOV Subject: MPI meeting ... Context Ladies/Gentlemen: I am coming to Dallas on Wednesday, in time for a post-dinner Context committee meeting. We are reading the chapter on Thursday. The Bristol Suites cancelled my reservation for Tuesday night, so I am coming on Wednesday; this also means I will be bringing the polished (:-)) chapter with me. Those of you who want to see it Wednesday will be able to see copies, or come to the meeting after dinner. I am sorry for the late arrival, but events have conspired against me. Please be patient. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 12:01:14 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02958; Tue, 21 Sep 93 12:01:14 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09341; Tue, 21 Sep 93 11:57:29 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 11:57:26 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09325; Tue, 21 Sep 93 11:57:24 -0400 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08373; Tue, 21 Sep 93 10:56:55 CDT Date: Tue, 21 Sep 93 10:56:55 CDT From: Tony Skjellum Message-Id: <9309211556.AA08373@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, walker@rios2.epm.ornl.gov Subject: Re: MPI Agenda: Second Version The agenda puts a "question mark" after the word second reading with regard to the context chapter. Why? - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 08:37:15 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 09:14:33 EDT Date: Tue, 21 Sep 1993 09:14:29 -0400 From: walker@rios2.epm.ornl.gov (David Walker) To: mpi-comm@cs.utk.edu Subject: MPI Agenda: Second Version Content-Length: 1445 Agenda for MPI Meeting September 22-24, 1993 I can't remember what is general discussion, first reading, or second reading. I assume the subcommittee chairs know this. Wednesday 1:00-1:30 Review of the Introduction chapter, and what needs to be done to it. (Dongarra) 1:30-3:00 Full group meeting to discuss point-to-point (Snir) 3:00-4:00 Language Binding Subcommittee Meeting 4:00-5:30 Full MPIF meeting to discuss Collective Communication (Geist) 5:30-6:00 The status of the MPI reference implementation (Lusk and Gropp) 6:00-7:30 Dinner break 7:30-10:30 Environmental Inquiry Subcommittee Meeting. Other subcommittee meetings if necessary. Thursday 9:00-12:00 Full meeting to discuss contexts (2nd reading?) (Skjellum) 12:00-1:30 Lunch (provided) 1:30-3:30 Full meeting to discuss contexts (continued from morning) (Skjellum) 3:30-3:45 Break 3:45-5:00 Full meeting to discuss topologies (Hempel) 5:00-6:00 Full meeting to discuss language binding (Lusk?) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:30 Continued informal subcommittee meetings if necessary Friday 9:00-10:30 Full group meeting for second reading of subset chapter. (Huss-Lederman) 10:30-10:45 Break 10:45-12:00 Any other business ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Sep 21 22:55:33 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA06018; Tue, 21 Sep 93 22:55:33 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20080; Tue, 21 Sep 93 22:52:05 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 21 Sep 1993 22:52:03 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares9.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20059; Tue, 21 Sep 93 22:51:58 -0400 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA27329 (5.65c/IDA-1.4.4 for ); Tue, 21 Sep 1993 21:51:54 -0500 Message-Id: <199309220251.AA27329@antares.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: minutes of August meeting Date: Tue, 21 Sep 1993 21:51:51 -0500 From: Rusty Lusk Here are the minutes of the August meeting. Bob and Rusty Minutes of the Message Passing Interface Forum Dallas, Texas August 11 - 13, 1993 The MPI Forum met August 11-13, 1993, at the Bristol Suites Hotel in North Dallas. This was the eigth meeting of the MPIF and the sixth regular working meeting in Dallas. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. This meeting included the first reading of the Communication Contexts, Environmental Management and Subset chapters, the second reading of the Process Topologies chapter and formal consideration of various topics in the Point-to-point and Collective Communication chapters. There were a substantial number of formal votes taken at this meeting as well as a few straw votes. All of the votes are recorded in these minutes (and can be found by searching for VOTE) and have also been published in summary form to the mpi-core mailing list. These minutes were written by Bob Knighten (knighten@ssd.intel.com) and Rusty Lusk (lusk@anl.gov). These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly lead to each topic (and a few other things.) The basic document that was used at this meeting are: + DRAFT Document for a Standard Message-Passing Interface (August 10,1993) + MPI Environmental Management section Attendees: --------- Robert G. Babb II U. of Denver babb@cs.du.edu Doreen Cheng NASA/Ames dcheng@nas.nasa.gov Lyndon Clarke University.of Edinburgh lyndon@epcc.ed.ac.uk James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Anne C. Elster Cornell University. elster@cs.cornell.edu Jim Feeney IBM Endicott feenyj@vnet.endicott.ibm.com Al Geist ORNL gst@ornl.gov Ian Glendinning University. of Southampton igl@ecs.soton.ac.uk Brian K. Grant LLNL bkg@llnl.gov Adam Greenberg TMC moose@think.com Leslie Hart NOAA/FSL hart@fsl.noaa.gov Don Heller Shell Development heller@shell.com Tom Henderson NOAA/FSL hender@fsl.noaa.gov Alex Ho IBM Almaden wkh@almaden.ibm.com Gary Howell Florida Tech howell@zach.fit.edu Steven Huss-Lederman SRC lederman@super.org Bob Knighten Intel SSD knighten@ssd.intel.com Rik Littlefield PNL rj_littlefield@pnl.gov Rusty Lusk ANL lusk@mcs.anl.gov Peter Madams nCube pmadams@ncube.com Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Instiute otto@cse.ogi.edu Peter Pacheco U. of San Francisco peter@sun.math.usfca.edu Anthony Skjellum Mississippi State U. tony@cs.msstate.edu Marc Snir IBM, T.J. Watson snir@watson.ibm.com Alan Sussman University. of Maryland als@cs.umd.edu Bob Tomlinson LANL bob@lanl.gov Eric T. Van de Velde CalTech evdv@ama.caltech.edu David Walker ORNL walker@msr.epm.ornl.gov Joel Williamson Convex Computer joelw@convex.com Wednesday, August 11 --------- --------- ------------------------------------------------------------------------------- General Meeting ------------------------------------------------------------------------------- Jack Dongarra opened the meeting by presenting the agenda that was previously sent out by David Walker. AGENDA ------ Wednesday 1:00 - 2:00 Subcommittee meetings 2:00 - 4:00 Point-to-point communications Snir 4:00 - 5:00 Collective communications Geist 6:00 - 7:30 Dinner 7:30 - 10:00 Subcommittee meetings Thursday 9:00 - 12:00 Context Skjellum 12:00 - 1:30 lunch 1:30 - 2:30 Context 3-4 subset Huss 4-6 topology Huss 6-8 dinner 8-10 subcommittees Friday 9-10:30 env Lusk 10:30-12 language Lusk Status of Readings sec\date May June August September --------- +-------------------------------------------------------------- p-p | 2 coll | 1 2 2 profile | 1 2 context | 1 2 topology | 1 2 subset | 1 2 2 lang | 1 2 env | 1 2 The next meeting will be September 22-24. It will again be here in Dallas. Started at 2:10 ------------------------------------------------------------------------------- Report From the Point-to-Point Communication Subcommittee ------------------------------------------------------------------------------- Marc Snir presided. Marc reorganized the chapter to make it more readable. He also added the material in Section 4.13 (Derived datatypes) in line with the straw vote at the last meeting. In response to a question, Marc noted that he welcomes editorial comments, and asks that they be sent to him in e-mail. Derived datatypes (4.3) ----------------------- Marc began by describing the ideas behind "Derived datatypes". What is the relation of this for Fortran 90 data types? Largely orthogonal. Organization count: 20 STRAW VOTE: Should "Derived datatypes" be added to MPI? ---------- Yes: 22 No: 0 Abstain: 0 Introduction (4.13.0) --------------------- VOTE: Approve 4.13.0 (Introduction)? ---- Yes: 19 No: 0 Abstain: 1 Datatype constructors (4.13.1) ------------------------------ Marc gave brief descriptions of the five functions in this section and contrasted them with the earlier buffer construction functions. VOTE: Approve 4.13.1 (Datatype constructors) ---- Yes: 20 No: 0 Abstain: 0 Additional functions (4.13.2) ----------------------------- It was noted that the sentence starting at line 33 on p. 77 is wrong and contradicts what follows. Marc agreed and will repair this. There was some discussion of exactly what is passed in a message using a datatype that contains gaps. There was no disagreement and this will be clarified. The fact that in MPI_ADDRESS has in integer OUT parameter that provides the byte address of location is a problem on some architectures was briefly discussed. One proposal was to use a suitable implementation specific definition of the return type. There was also a discussion of MPI_TYPE_COMMIT primarily to better understand what was intended here. The principal confusion was that MPI_TYPE_COMMIT has to do with completing the definition of the type, NOT committing data. There is an alternative form of MPI_TYPE_COMMIT with only one paramater (type). Then would need commit before communication; can use type in constructors after commit. Yet another alternative is a lazy commit, i.e. a datatype object becomes commited at first use in a communication. There are several issues in considering these alternatives. One is ease of use for the programmer (lazy commit is easy); use of resources (the original allows reclaiming resources as soon as they are not being used); the cost of using a datatype buffer in a communication. Adam Greenberg proposed yet another option - provide an optional function, MPI_TYPE_COMPILE, which can be used for efficiency but otherwise there is lazy commit. The objection to this was efficiency of the communication. Adam's counter was that he expected this to be in the noise of the general overhead of communication. Marc noted that we need to write up full proposals and have a more focused discussion. VOTE: Approve 4.13.2 (Additional functions) EXCEPT MPI_TYPE_COMMIT and ---- MPI_TYPE_FREE? Yes: 16 No: 0 Abstain: 2 Use of general datatypes in communication (4.13.3) -------------------------------------------------- This section specifies how datatypes are used in send and receive, including how type matching working. In particular datatypes match if they are structurally equivalent (i.e. the type signatures are the same.) Is there some function that gives a count (of some suitable sort) of elements in a send when using datatypes? {{{I am confused?}}} VOTE: Have query function that returns ther number of top-most elements ---- received? Yes: 14 No: 0 Abstain: 4 VOTE: Have ONLY a query function that returns the count of top-most level ---- elements in a receive. Yes: No: Abstain: VOTE: Approve 4.13.3 (Use of general datatypes in communication) as amended. ---- Yes: 15 No: 0 Abstain: 4 Examples (4.13.4) ----------------- There are no requirements here and so no vote was taken. Correct use of addresses (4.13.5) --------------------------------- This section is about dealing with addressing when the system does not have limits in using addresses systems which do not have a flat address space. VOTE: Approve section 4.13.5 (Correct use of addresses)? ---- Yes: 18 No: 0 Abstain: 1 Message data (4.2.1) -------------------- Marc asked if the table in this section needs to be expanded to include all of the native C data types. YES! Marc reviewed all of the changes (other than order) that he made in this chapter. More detail in the discussion of the semantics of point-to-point communication. Has added Progress (some guaranteed) and Fairness (none guaranteed) requirements. {{{Where?}}} Can use null when nothing is needed. On p. 61 there is a new function, MPI_TESTALL, as suggested by David Walker. It is included for reason of completeness and symmetry. {{{See Discussion on p. 62}}} VOTE: Allow null pointers in an array of pointers (with the system ---- required to do the right thing? Yes: 20 No: 0 Abstain: 1 On p. 63 there is a new function, MPI_PROBE_COUNT, which uses a datatype to interpret the result of a probe and get a count. On p. 64 there is a typo in MPI_PROBE. There should not be a datatype parameter. On p. 65, the name of MPI_IS_CANCELLED has been changed to MPI_TEST_CANCELLED. The time arrived to make a decision on SENDRECV (section 4.11). Adam Greenberg suggested that there should be two tag arguments (for sending and for receiving) rather than only one. One effect of this is that a wild card can now be used in the receive_tag. VOTE: Have two tag arguments (send_tag and receive_tag) in MPI_EXCHANGE? ---- Yes: 6 No: 3 Abstain: 10 Section 4.12 (Null processes) is a proposal to the suggestion from Jon Flower that was supported in a straw vote at the last meeting. There was a substantial discussion of the utility and cost of MPI_PROCNULL. Steve Huss-Lederman, as a proxy for Rolf Hempel, argued for the value of this in use with process topologies. Various people argued that there would a universal overhead if MPI_PROCNULL were allowed as a source/destination. Three positions: Use as legitimate source/destination; only in send/receive/exchange; not legitimate source/destination. time (based on the suggestion from Jon Flower.) VOTE: (1) Allow MPI_PROCNULL as a source or destination for all communication operations. (2) Allow MPI_PROCNULL only for MPI_SENDRECV and MPI_EXCHANGE. (3) Never allow MPI_PROCNULL as a source or destination in communication operations. (3) Yes: 3 No: 9 Abstain: 8 (2) Yes: 11 No: 5 Abstain: 6 Section 4.14 (Universal communication functions) includes one new convenience function, MPI_COMM_INIT. This was added by Marc Snir to make it part of the base functions in terms of which all other functions can be defined. An alternative approach is to put section 4.14 in an Annex and NOT require these functions as part of VOTE: Approve Chapter 4 (Point to Point Communication) as ammended. ---- Yes: 18 No: 0 Abstain: 1 Note that all that remains to consider in Chapter 4 are the type_commit and free functions. ------------------------------------------------------------------------------- Report From the Collective Communication Subcommittee ------------------------------------------------------------------------------- Al Geist presided. This was a continuation of the second reading of this chapter that was begun at the last meeting. The number of changes was small. The format of the buffer arguments have all been changed to agree with those in the Point-to-point Communication chapter. The material on user_reduce has been rewritten to provide two variants, one assuming commutativity of the user operation and one not. Steve Huss-Lederman asked the question if there are ANY collective operations that are guaranteed to give the same result for repeated runs with the same initial conditions. Al Geist replied "Broadcast. Next question." A long discussion ensued. As in previous discussions of this topic, there was a wide spectrum of opinions expressed. Some insisted that reproducibility, at least in a debug mode, is required. Others insisted this is a quality of implementation issue. Other opinions expressed included that this is outside of the scope of MPI; an implementation must document the extent to which reproducibility is available; etc. In line with the Discussion paragraph on p. 95, it was agreed that it is unneeded to have the completely general ALLTOALL. Rik Littlefield noted that there is a useful kind of reduce that is missing, a "scatter-reduce" that does a reduction of sections of an array to an array of processes. He will write up a proposal and distribute it to the collective communication mailing list. Al Geist noted the change mentioned in the paragraph labeled Missing on p. 99. There was some confusion, so Al Geist promised to add an example to clarify this. Jim Cownie pointed out that it is important that implementors should provide an implementation guide that specifies which of the collective operations that may/may not be synchronizing actually are synchronizing. Adam Greenberg countered that users must assume that these operations are not synchronizing and therefore such a document serves no purpose. Adam Greenberg also expressed unease with section 5.6 and STRAW VOTE: Should MPI require documentation of the implementation ---------- variations in synchronization properties of collective operations. Yes: 18 No: 2 Abstain: 2 Eric Van de Velde asked why two versions of user_reduce were being provided. A brief review of the arguments of last time ensued. VOTE: Approve Chapter 5 (Collective Communications) ---- Yes: 21 No: 0 Abstain: 0 ----------------------------------------------------------------------------- The group adjourned for dinner at 6:10pm ============================================================================= Thursday, August 12, 1993 -------- --------------- ------------------------------------------------------------------------------- Report From the Communication Contexts Subcommittee ------------------------------------------------------------------------------- Anthony Skjellum presented. Group, Contexts and Communicators (Chapter 3) - First Reading ------------------------------------------------------------- Introduction (3.1) ------------------ Objection to use of term intra-communication and inter-communication without definition. This is editorial and will be addressed outside the meeting. There are no requirements in this section, so no vote was taken. Context (3.2) ------------- There was confusion about the term "hypertag". This too is editorial. There are no requirements in this section, so no vote was taken. Groups (3.3) ------------ Predefined Groups (3.3.1) ------------------------- It was proposed to change the wording describing MPI_GROUP_ALL to be "all processes at moment of process creation" and to add another group, MPI_GROUP_SIBLING which is "all processes with the same program text." Don Heller asked about system defined server processes and the like. It was agreed to modify the wording to include these. Jim Cownie noted that there is a problem with the notion of HOST because the host would have to have many different versions of MPI_GROUP_HOST. As an alternative Jim proposed that host should acessible via some constant or function that would give the rank of the host in MPI_ALL. After various proposals for additional predefined groups {MPI_GROUP_PEER "all processes except the host (if there is one)", MPI_GROUP_PARENT "parent of all children spawned"} it was proposed that this section be revised to say something like "There are no predefined groups. The effect of predefined groups is gotten by using the groups associated with the predefined communicators." Adam Greenberg asked that a vote on this section NOT be taken until after the decision on which communicators are predefined. Communicators (3.4) ------------------- Predefined Communicators (3.4.1) -------------------------------- MPI_COMM_ALL MPI_COMM_SIBLING MPI_COMM_PEER MPI_COMM_PARENT MPI_COMM_SELF After dealing for a time with the complexity and inclarity of this situation various alternatives were offered. The simplest was to have only MPI_COMM_ALL. Dan Nessett Organization count: 22 VOTE: Revise sections 3.3 and 3.4 as follows: ---- (1) There are no predefined groups (2) The only predefined communicators are MPI_COMM_ALL and MPI_COMM_PEER. (3) There is a predefined MPI_HOST_RANK which gives the rank of the host in the ALL group. It is MPI_UNDEFINED if there is no host. Yes: 17 No: 2 Abstain: 3 VOTE: Approve sections 3.3 and 3.4 as amended. ---- Yes: 18 No: 0 Abstain: 3 Group Management (3.5) ---------------------- Local Operations (3.5.1) ------------------------ There was some discussion of how MPI manages the coordination between various groups (always by relation to the ALL group), which of the inquiry functions in this section properly belong in the environmental sections (none) and general confusion about what the various functions do. No changes were made. VOTE: Approve section 3.5.1 (Local Operations)? ---- Yes: 19 No: 0 Abstain: 3 Local Group Constructors (3.5.2) -------------------------------- Most of the discussion had to do with clarification and editorial corrections. Eric V. asked for a function to reorder the ranks in a group. After some discussion as to exactly what is needed and why it was noted that the MPI_LOCAL_SUBGROUP provides the desired function. It was agreed to add a remark to this effect. It was noted that all of the functions in section 3.5 need descriptions rather than just names and parameters. Marc Snir asked for a clarifying note that of the functions in this section only MPI_LOCAL_SUBGROUP and MPI_LOCAL_SUBGROUP_RANGES can change the ranks. This has as a side effect that the ranges must not overlap. This was agreed. Lyndon Clarke asked for a clarification that the effect of MPI_LOCAL_SUBGROUP_RANGES is as though the ranges were expanded to a list of ranks and MPI_LOCAL_SUBGROUP were called with these ranks. There should be a similar statement for the relation between MPI_LOCAL_SUBGROUP_EXCL_RANGES and MPI_LOCAL_EXCL_SUBGROUP. This was agreed as well. VOTE: Approve section 3.5.2 (Local Group Constructors) as clarified? ---- Yes: 17 No: 0 Abstain: 5 Collective Group Constructors (3.5.3) ------------------------------------- The phrase "a stable sort is used to determine rank order" on line 23 of p. 18 will be change to say that in the event of ties the rank in the comm group will be used to determine the rank in new_group. After discussion of the meaning of MPI_COLL_SUBGROUP it was proposed to have instead: MPI_COLL_SUBCOMM(comm, key, color, new_comm) which will then appear in section 3.7.3. The effect on section 3.5.3 is that it would simply say there there are no collective group constructors. VOTE: Approve section 3.5.3. (Collective Group Constructors) as amended? ---- Yes: 18 No: 0 Abstain: 3 --- break 10:30 - 11 --- Sections 3.6 and 3.7 (pp. 18-21) were handled by giving a function by function discussion followed by an overview of a "tuning" proposal by Marc Snir. Operations on Contexts (3.6) ---------------------------- Local Operations (3.6.1) ------------------------ Collective Operations (3.6.2) ----------------------------- In MPI_CONTEXTS_ALLOC, the len parameter is removed. The "void *" in the descriptions is removed and better words will be provided. The idea of quiescence that was prominent in the context proposal at the last meeting has largely disappeared. The manner of dealing with the problem that quiescence was designed to solve is discussed at length on p. 19. A discussion of the relation of point-to-point and collective communication was prompted by a dispute between Marc Snir and Jim Cownie. Jim made the point that the collective communication routines can be written using the point-to-point communication routines and the material in the context chapter. It was noted that there is one collective communication routine in the context chapter - MPI_CONTEXTS_ALLOC - and some system magic must insure that this works correctly. Marc Snir noted that the system can provide similar magic throughout for optimization purposes. Operations on Communicators (3.7) --------------------------------- Local communicator Operations (3.7.1) ------------------------------------- There were no issues in this section. Local Constructors (3.7.2) -------------------------- There is a MPI_COMM_BIND functions missing. It was accidently deleted and will be added. The form is: MPI_COMM_BIND(group, context, new_comm) IN group IN context OUT new_comm The details will be provided in the next draft. The name of the function MPI_COMM_UNBIND will be changed to MPI_COMM_FREE (and the function of this name in the point-to-point chapter will be renamed.) The frequent reference to MPI_COMM_GROUP(comm) will be changed to MPI_COMM_GROUP(comm, group). Collective Communicator Constructors (3.7.3) -------------------------------------------- The one collective operation for communicators is MPI_COMM_MAKE. Adam Greenberg noted that as currently written every member of the group associated with sync_comm gets comm_new which has At this point Marc Snir An out of context proposal - Only use of context is for local creation of communicators. - Result can be achieved without explicit context object (some loss of safety) - Either case needs ruls for coordinated context allocation. Communication context - specified by communicator - can be "preallocated" and then locally bound to communicator. MPI_CONTEXTS_ALLOC(comm, n) - preallocates n contexts and "caches" them with comm. {This can be called repeatedly and adds the number of contexts specified on each call. This is a collective operation.} MPI_CONTEXTS_FREE(comm, n) - releases up to n preallocated contexts. {This can be called repeatedly. It is a local operation.} MPI_COMM_CONTEXT(comm, n) - queries the number of available preallocated contexts. {This is a local operation.} MPI_COMM_DUP(comm, new_comm) - duplicates a comunicator. Uses locally cached context, if available, otherwise this is a collective operation {It is erroneous if some but not all have locally cached context available. Note that new_comm does NOT have any cached context.) MPI_COM_LDUP(comm, new_comm) - duplicates a communicator. Uses locally cached context and returns NULL if none available. MPI_COMM_MAKE(sync_comm, comm_group, comm_new) MPI_COMM_LMAKE(sync_comm, comm_group, comm_new) - both of these create new communicator associated with the comm_group which is a subgroup of the group of sync_comm. This must be call by all members of the group of sync_comm. Some unease was expressed about sometime collective sometimes not. Safety? In response Marc noted that there could also be MPI_COMM_GDUP which would always do a collective operation. Correctness rule ---------------- All processes in a comm group must execute the same sequence of calls to MPI_CONTEXT_ALLOC, MPI_COTEXT_FREE, MPI_COMM_xDUP, MPI_COMM_xMAKE with comm as argument. - simple to state - same as for collective communication - too conservative? Note: This is compatible with the existence of static preallocated contexts. At this point Lyndon Clarke, having been waving his hand in the air for several minutes, stood on his chair to try and get Marc to address his question. After a brief discuss between Adam, Marc and Tony, Lyndon was allowed to speak. Lyndon Clarke noted that there is no way for the system to check for proper usage of arguments, so this offers no additional security compared with earlier proposals. Others noted that this did provide some additional safety, but it is hard to make direct comparison. Steve Huss asked if any one in the contex comm wanted to keep the current material. Tony said no, but that would likely not be true if Mark Sears were here. STRAW VOTE: Make this into a full proposal to replace the current 3.7.2 & ---------- 3.7.3 Yes: 25 No: 0 Abstain: 2 -- break for lunch 12:10 - 1:40 -- Cacheing (3.9) -------------- Rik Littlefield presented this material. Attribute Cacheing Function: Safely attach arbitrary informatio to groups (and communicators). Purpose: Allow modules to retain or exchange gropup-specific information WITHOUT complicated calling sequences or correctness ruls for use of module Examples Basic Capabilities: - Attributes are local - Attribute value can be pointer to arbitrary structure - Attributes are referenced by a key value obtained from MPI - Attributes can be defined and retrieved - Destructur routine is called when the group (communicator) is freed - Propagation routine is called when the group (communicator) is duplicated. Funtions: MPI_GET_ATTRIBUTE_KEY( OUT keyval) MPI_GROUP_ATTR(IN group, OUT attr_set_handle) MPI_COMM_ATTR(IN comm, OUT attr_set_handle) MPI_SET_ATTRIBUTE( IN attr_set_handle IN keyval IN attribute_value IN *attribute_destructor_routine IN *attribute_propagation_routine ) MPI_TEST_ATTRIBUTE( IN attr_set_handle IN keyval OUT attribute_value OUT result_status ) attribute_destructor_routine(IN attribute_value) attribute_propagation_routine(IN attribute_value, ......) This list is an updated and organized variant of the text. In particular the two routines, MPI_ATTRIBUTE_ALLOC and MPI_ATTRIBUTE_FREE, have been eliminated. What is the rationale for these functions altogether? These provide a method for managing resources associated with groups and communicators. For example this provides facilities to implement the topology facilities on top of MPI. Rik observed that this allows one to effectively extend MPI, e.g. to provide a user written collective operation that can be safely use with MPI and which looks like an MPI routine. Marc Snir asked for a routine to change the value of attributes without having to provide the destructor and propagation routines. There was a question if this introduced a degree of insecurity. Jim Cownie noted that one might well want at attribute with null destructor and propagation routines. Such a reset routine will be provided in the next draft of this chapter. Do we need attributes for both groups and communicators? Why not just on groups? This would allow elimination of attribute handles. There do seem to be situations where it is need on communicators, not just on groups. Adam noted that this proposal puts a resource burden on the system, so he asked about the possiblity of providing only a single system slot with the remainder of the storage provided by the user. Adam is concerned about the admixture of resources at both user and system level. {{{I'm confused}}} Tony proposed adding MPI_ATTRIBUTE_ALLOC and MPI_ATTRIBUTE_FREE back. It was claimed that providing attribute allocate and free routines together with a call back mechanism associated with the free is sufficient to provide all of the functions in section 3.9. Various people countered that this would introduce new problems of coordination and safety. In particular each library might have independent attribute mechanisms and this would require using multiple callback on each call of free. It was noted that this is very similar to the that was solved in X by using a register of callback functions. That can provide a model for this group to use. STRAW VOTE: Do we want a cacheing mechanism? ---------- Yes: 14 No: 4 Abstain: 7 How would toplogy need this? Steve Huss, as virtual Rolf Hemple, noted that topologies need a variety of information (e.g. dimensions, periodicities) that need to be associated with group (and communicators?). As topologies are a part of MPI, a general cacheing mechanism is not required. But without it there is likely to be conflicts between topology and user written libraries. VOTE: Approve section 3.9 (Cacheing) as amended in the presentation? ---- Yes: 8 No: 7 Abstain: 6 -------- break 2:45 - 3:10 Adam Greenberg asked about having a User's Guide meeting tonight. There are enough to have a meeting here after dinner -------- Introduction to Inter-Communication (3.8) ----------------------------------------- Tom Henderson presented on Inter-communication ( A ) <=> ( B ) <=> ( C ) / / Arank Brank Arank: ----- send(...,Brank,tag, commAB) Brank: ----- recv(...,Arank,tag,commAB,...) Want to be able to send from process in one group to process in another group using the rank in the target group. ALTERNATIVES Local group has acess to remote group and have a rank translation in some common ancestor. User maintains tables and communicators for group-pairs. {{{SLIDES HANDED OUT}}} STRAW VOTE: Hear the full proposal? ---------- Yes: 25 No: 1 Abstain: 4 Basic concepts Local Group Remote Group local group leader remote group leader Peer-group that contains both group leaders. {Not to be confused with MPI_COMM_PEER introduced earlier today.} - All members of both groups must call MPI_COMM_PEER_MAKE. What is reason for tag? - serves as the identifier for this particular inter-communicator. Joel Williamson asked why do this rather than just working in MPI_ALL? So can use names that are convenient for the problem at hand. This is also suitable for generalization to a dynamic situation rather than the static situation that is now in MPI. - Why does everyone call PEER_MAKE? To get a common communicator. - Discussion slide Add an IN argument, my_leader_rank, to MPI_COMM_PEER_MAKE(). This allows later addition of dynamic process creation. The IN argument peer_comm need only be valid in the local group leader. Only the group leaders need to be members of peer_comm. - LOOSELY-SYNCHRONOUS INTER-COMMUNICATOR CONSTRUCTOR - SYNCHRONIZATION PROPERTIES OF MPI_COMM_PEER_MAKE_START() AND MPI_COMM_PEER_MAKE_FINISH() - COMMUNICATOR STATUS (convenience function - make go away) - Synchronization issue: Can a process using an inter-communicator send a message using that inter-communicator as soon as it has the inter-communicator? Something needs to be said. - INTER-COMMUNICATOR SUPPORT - EXAMPLE 1 - "UNDER THE HOOD" - INTRA-COMMUNICATION INTER-COMMUNICATION - POSSIBLE IMPLEMENTATION OF MPI_COMM_PEER_MAKE() - POSSIBLE IMPLEMENTATION OF MPI_COMM_PEER_MAKE_START() - POSSIBLE IMPLEMENTATION OF MPI_COMM_PEER_MAKE_FINISH() - What is comparison with using the common ancestor approach? To do this one would create a union group STRAW VOTE: Have inter-communication? ---------- Yes: 14 No: 3 Abstain: 9 VOTE: Approve section 3.8 (Introduction to Inter-Communication) as ---- amended but minus the name service? Yes: 8 No: 2 Abstain: 10 - ? - EXAMPLE 2 - EXAMPLE 3 - EXAMPLE 4 - Al Geist noted all of the examples should be expanded to show at least one message being sent! - POSSIBLE IMPLEMENTATION OF MPI_COMM_NAME_MAKE() - POSSIBLE IMPLEMENTATION OF MPI_COMM_NAME_MAKE_START() - POSSIBLE IMPLEMENTATION OF MPI_COMM_NAME_MAKE_FINISH() - VOTE: Approve the name serverice material in section 3.8 (as amended)? ---- Yes: 8 No: 1 Abstain: 10 ------------------------------------------------------------------------------- Report From the Process Topology Subcommittee ------------------------------------------------------------------------------- Process Topology - Second Reading --------------------------------- Steve Huss presented. {He put on a pair of Birkenstocks to emphasize that he was acting as a virtual Rolf Hemple.} A couple of proposals that were made verbally at the last meeting were not written and so have not been included. Introduction (6.1) ------------------ VOTE: Approve section 6.1 (Introduction) ---- Yes: 13 No: 0 Abstain: 5 Virtual Topologies (6.2) ------------------------ Terminology - change the name to "process topologies" (Eric Van ) or "application topologies" (David Walker). VOTE: Approve section 6.2 (Virtual Topologies)? ---- Yes: 10 No: 0 Abstain: 6 Embedding in MPI (6.3) ---------------------- There were various editorial remarks which Steve recorded to relay to Rolf Hemple. VOTE: Approve section 6.3 (Embedding in MPI)? ---- Yes: 9 No: 0 Abstain: 8 Overview of the proposed MPI functions (6.4) -------------------------------------------- The initial part of 6.4 (before 6.4.1) will go away. VOTE: Specify that we use row major order? ---- Yes: 4 No: 0 Abstain: 13 Steve Huss pointed out that the extent to which these functions are global functions is not specified. Lyndon Clarke offered the amendment that they be specified as collective. There was a discussion of the group parameters in these functions. Steve agreed to propose to Rolf that they be systematically replaced by communicators. VOTE: Replace all group parameters throughout by communicator? ---- Yes: 8 No: 1 Abstain: 7 VOTE: MPI_MAP_CART and MPI_MAP_GRAPH shall be global routines? ---- Yes: 4 No: 2 Abstain: 12 The large number of abstentions in the recent votes led to a discussion of the value of topology and also to a discussion of our procedures. There was no strong interest in discussing including topology. Neither was there any strong interest in changing procedures. VOTE: Postpone the second reading of the topology chapter until the next ---- meeting? Yes: 3 No: 12 Abstain: 2 VOTE: Approve chapter 6 as amended? ---- Yes: 12 No: 3 Abstain: 2 ------------------------------------------------------------------------------- Report from the Subset Subcommittee ------------------------------------------------------------------------------- Steve Huss presided. This was NOT a second reading. Jim Cownie argued that the profiling material should be included in the subset because this provides important facilities and the cost of providing it in an initial implementation is not large. Several people agreed with this, so there was a quick vote. STRAW VOTE: Include profiling in the subset? ---------- Yes: 24 No: 0 Abstain: 1 A discussion of the parts of environmental management and inquiry to be included lead to an agreement that this should be deferred until the presentation on this material. There was nothing to be said about language binding - there will be F77 and C bindings. STRAW VOTE: Exclude topology from the subset? ---------- Yes: 21 No: 2 Abstain: 3 It was agreed that the list for collective communication in the document is OK. In considering the Point-to-point Functions, the list in the document includes MPI_SENDRECV but excludes MPI_EXCHANGE. It was generally agreed that this is sensible. Steve Otto argued for including hvec type functions in the subset because of common usage. In considering this and the issue of derived datatype several posibilities were considered. The one that got general support is to include all of the material in section 4.13. It was noted that data conversion is not a subset issue - heterogeneous systems have to have it; homogeneous systems do not need it. --- break for dinner at 6:05 --- Subcommitte meetings - starting about 8:30 Subset - immediately after dinner User Guide - after subset meeting Context - after subset meeting ============================================================================= Friday, June 25, 1993 ------ ------- ---- ------------------------------------------------------------------------------- Report from the Environmental Management Subcommittee ------------------------------------------------------------------------------- Rusty Lusk presided. Rusty began by handing out a new version Environmental Management and Inquiry 1 Initialization 2 Environmental query 3 Others Initialization and Termination Current draft: MPI_INIT() "idempotent" MPI_FINALIZE() "last MPI call" Discussion: How does a library know whether to call MPI_FINALIZE? Is MPI_FINALIZE optional? MPI_INIT requires some state not attached to any object; why not a communicator? MPI_INIT requires some state not attached to any object; why not a communicator? A proposed amendment: MPI_INIT(old_comm, new_comm) {if old_comm is null, gets first communicator} MPI_FINALIZE(current_comm, old_comm) Nests MPI invocations Attaches MPI state to communicator STRAW VOTE: We shall provide a mechanism that allows a library written ---------- using to be called from either within or without MPI? Yes: 11 No: 8 Abstain: 3 Steve Huss asks what happens if two libraries are invoked using different numbers of processors, then what is the ALL group? Jim Cownie offered a very simple proposal - All processes must call MPI_INIT at the start and all processes must call MPI_FINALIZE at the end. Note that the picture is that by a vendor provided miracle the MPI system is started and only after this can MPI_INIT be called. This is likely to be a global barrier. STRAW VOTE: MPI_INIT and MPI_FINALIZE must be called exactly once in each ---------- process. Yes: 18 No: 0 Abstain: 5 organization count: 20 Lotes of continuing discussion. It was agreed that, in the context of the straw vote, any program that violated the requirement is erroneous. VOTE: Have an MPI_INITIALIZED flag? Yes: 16 No: 1 Abstain: 2 VOTE: MPI_INIT and MPI_FINALIZE must be called exactly once in each ---- process. MPI_INIT is a global operation. It must be called before any other MPI routine. MPI_FINALIZE is the last MPI call. Yes: 16 No: 1 Abstain: 3 Rusty offered a proposal for MPI_ABORT(error_code) which terminates every process in the ALL group. VOTE: Have MPI_ABORT? ---- Yes: 13 No: 3 Abstain: 2 MPI-Specific (1.1) [Section numbers from chapter handed out at meeting] ------------------ Why are there communicator arguments in MPI_NumCommunicator? Rusty did not know. No one had a convincing argument. VOTE: Remove communicator arguments from MPI_ValidTags and ---- MPI_NumCommunicator? Yes: 14 No: 0 Abstain: 5 There was a fair amount of confusion about the intent and value of the MPI_BufferParams routine. In particular, various alternative proposals were mentioned. Rik Littlefield has proposed that the user be able to specify buffering capability STRAW VOTE: Should there be some way of asking the system about buffering? ---------- Yes: 15 No: 2 Abstain: 7 STRAW VOTE: Should there be some way of telling the system about buffer ---------- requirements? Yes: 6 No: 3 Abstain: 15 Rik will provide a proposal at the next meeting. VOTE: Remove MPI_IOmode? ---- Yes: 18 No: 1 Abstain: 2 Discussion of MPI_Errormode? There was again uncertainty about the communicator argument leading to: VOTE: Remove the communicator argument from MPI_Errormode? ---- Yes: 7 No: 4 Abstain: 8 After further discussion it was realized that while something of this sort is desirable, that is much more detail (e.g. how are error handlers established) that is essential before accepting this function. STRAW VOTE: Should there be a facility to set and query error mode? ---------- Yes: 18 No: 0 Abstain: 0 There was quick agreement that MPI_Has_Nonblocking and MPI_Has_Heterogeneous are not useful. VOTE: Remove MPI_Has_Nonblocking and MPI_Has_Heterogeneous? ---- Yes: 9 No: 0 Abstain: 4 STRAW VOTE: Have functions to inspect receive queue and other interesting ---------- internal structures? Yes: 12 No: 3 Abstain: 2 Anne Elster again asked for MPI_LOAD_INFO. She was proposing a modified version that had less time-specific information, but no written proposal was available at the meeting. A concrete proposal will be seen at the next meeting. VOTE: Remove sections 1.2 (Parallel programming) and 1.3 (non-MPI) except keep for MPI_Ge ---- Yes: 8 No: 3 Abstain: 2 VOTE: Accept MPI Environmental Management chapter as amended. ---- Yes: 8 No: 2 Abstain: 3 ------------------------------------------------------------------------------ MPI Sound Bites Jim Cownie Where's David? Oh No! Don't think about that one too much What's the question again? Those in favor of going to the bar? Shall we accept the chapter as eviscerated? ------------------------------------------------------------------------------ Report from the Language Binding Subcommittee ------------------------------------------------------------------------------- Rusty Lusk presided. Language Binding 7.1-7.4 will go into another chapter (MPI Terms and Conventions) 7.5 will go into an Appendix We need to: 1. Update and read 7.1-7.4 2. Decide on principles for binding presentation 3. Decide on format of definitions in the chapters 4. Decide on format and order of definitions in appendix. 5. Choose a procedure for agreeing on names of C functions, Fortran subroutines, named constants, types. 6. Enforce consistency 1. Modification to 7.1-7.4 (see draft) 2. Presenting the bindings a. Named constants (MPI_SUCCESS) b. Named types (MPI_COMMUNICATOR) c. Functions and arguments i. C - use ANSI C style ii. Fortran - use prototypes and declarations d. Consistency of formal argument names e. IN arguments before OUT arguments f. return code last argument in FORTRAN g. others? 3. Current 7.5 OK modulo name updating? Jim Cownie argued that the principle that "all C functions should return an error code" should be relaxed for those functions that would best be implemented using macros. VOTE: Accept chapter and annex with the modifications outline by Rusty ---- Yes: 12 No: 0 Abstain: 1 Format In the chapters Current Format <- match C binding in order and number of arguments + Fortran Binding <- match appendix + C Binding <- match appendix In the appendix, Sort by appearance order? Alphabetically within chapter? <-- this one chosen Alphabetically? Keep appendix after using if for consistency check? (Note alphabetical index) As a technical issue, Steve Otto would like to have the bindings appear in the chapter source, but only appear printed in the appendices. There was general agreement that in the appendix the functions should appear alphabetically within each chapter. ------------------------------------------------------------------------------ From owner-mpi-comm@CS.UTK.EDU Wed Sep 22 14:31:24 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA10658; Wed, 22 Sep 93 14:31:24 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02812; Wed, 22 Sep 93 14:27:56 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Sep 1993 14:27:54 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02781; Wed, 22 Sep 93 14:27:52 -0400 Via: uk.ac.daresbury.cxa; Wed, 22 Sep 1993 19:26:34 +0100 From: "R.J. Allan" Date: Wed, 22 Sep 93 19:22:20 +0100 Message-Id: <21256.9309221822@uk.ac.dl.cxa> To: owner-mpi-comm@CS.UTK.EDU Cc: mpi-comm@cs.utk.edu, elster@cs.cornell.edu In-Reply-To: Anne C. Elster's message of Thu, 16 Sep 93 03:44:56 -0400 <9309160744.AA24612@elli.cs.cornell.edu> Subject: MPI_LOAD A quick comment on load balancing and message-passing harnesses. We have done load balancing using our own Fortnet harness on, for instance, a domain-decomposed CFD code. This is easy to do (given an algorithm!) providing you have access to CPU time and WALL time, and can time over computation and communication parts of the code. I am pleased to see that MPI has this facility already, more general load tools are very debatable... and probably program dependent in some way Cheers, Robert J. Allan Advanced Research Computing Group, Science and Engineering Research Council, Daresbury Laboratory, Daresbury, Warrington, WA4 4AD, U.K. 'phone +44 925 603207 FAX +44 925 603634 e-mail r.j.allan@daresbury.ac.uk From owner-mpi-comm@CS.UTK.EDU Wed Sep 22 16:19:13 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA11864; Wed, 22 Sep 93 16:19:13 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11362; Wed, 22 Sep 93 16:11:06 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 22 Sep 1993 16:11:05 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA11354; Wed, 22 Sep 93 16:11:03 -0400 Received: from elephant (elephant.parasoft.com) by sampson.ccsf.caltech.edu with SMTP id AA26994 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Wed, 22 Sep 1993 13:11:01 -0700 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA03022; Wed, 22 Sep 93 13:04:11 PDT Received: by lion.parasoft (4.1/SMI-4.1) id AA11731; Wed, 22 Sep 93 13:13:44 PDT Date: Wed, 22 Sep 93 13:13:44 PDT From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9309222013.AA11731@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: MPI_LOAD I agree with Robert Allan -- we have CPU and wallclock statistics inside Express and actually use it when computing grid mappings (i.e., virtual toplogies). It works remarkably well - better than I think it would be reasonable to expect. If anyone remembers the "wall of SGI's" that we had at Supercomputing '92 we had a CFD code that would migrate to and fro between 8 SGI's, and HP and an RS/6000 depending on what other "stuff" was chewing up their resources. So..... I think that MPI would probably benefit a lot from standardizing this type of call because it would encourage people to play, and potentially learn a lot about the rather under-explored field of dynamic load balancing. HOWEVER, having said that I don't think it's really in MPI's charter to include this stuff, just as it doesn't include high resolution timers. Of course, I was one of the people that wanted that feature too! Jon From owner-mpi-comm@CS.UTK.EDU Tue Oct 19 07:36:42 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with SMTP (5.61+IDA+UTK-930125/2.8t-netlib) id AA02869; Tue, 19 Oct 93 07:36:42 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA25225; Tue, 19 Oct 93 07:32:12 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Oct 1993 07:32:11 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930922/2.8s-UTK) id AA25217; Tue, 19 Oct 93 07:32:07 -0400 Date: Tue, 19 Oct 93 12:31:48 BST Message-Id: <29074.9310191131@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi; future availability To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Howdy y'all This may sound a little premature, but its something are frequently asked, so I just thought I'd see if there is an answer. I'd be really grateful for any real answers. We know that there is planned a demonstration of MPI at Supercomputing, and an MPI workshop. What will be the availability status of the demonstrator implementation, and of the TCP/IP mpid which is being written (by Tony et al, I recall)? Are any of the vendors able to give quotable information about when implementations of MPI will be released for their machines? Its actually quite important to know these things for acceptance of MPI in project planning, and we are doing our bit to promote acceptance of MPI, and have a lot of project planning here at the moment. Thanks and Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Sat Nov 27 04:50:26 1993 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id EAA29159; Sat, 27 Nov 1993 04:50:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id EAA19651; Sat, 27 Nov 1993 04:48:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 27 Nov 1993 04:48:32 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id EAA19644; Sat, 27 Nov 1993 04:48:31 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00442; Sat, 27 Nov 93 03:48:45 CST Date: Sat, 27 Nov 93 03:48:45 CST From: Tony Skjellum Message-Id: <9311270948.AA00442@Aurora.CS.MsState.Edu> To: mpi-comm@CS.UTK.EDU Subject: MPI Libraries Paper Dear colleagues, following up on my promise at SC'93 minisymposium, the paper "Writing Libraries in MPI" is now available on anonymous ftp. site: aurora.cs.msstate.edu directory: pub/reports file: mpi_lib_27nov93.ps.Z This will appear in the Proceedings of the Scalable Libraries Conference, to be published by IEEE Press. -Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Wed Jan 12 08:54:53 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id IAA14110; Wed, 12 Jan 1994 08:54:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA15141; Wed, 12 Jan 1994 08:53:44 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 12 Jan 1994 08:53:42 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from dasher.cs.utk.edu by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id IAA15134; Wed, 12 Jan 1994 08:53:41 -0500 From: Jack Dongarra Received: by dasher.cs.utk.edu (5.61+IDA+UTK-930922/2.7c-UTK) id AA26974; Wed, 12 Jan 94 08:52:54 -0500 Date: Wed, 12 Jan 94 08:52:54 -0500 Message-Id: <9401121352.AA26974@dasher.cs.utk.edu> To: mpi-comm@CS.UTK.EDU Subject: ``Final'' MPI meeting We are planning to have the ``Final'' MPI Meeting in Knoxville, Tennessee February 23rd-25th. Plan to fly into the Knoxville airport, called the Knoxville McGhee-Tyson airport. We have made arrangements with the Hilton Hotel in downtown Knoxville. Hilton Hotel 501 W. Church Street Knoxville, TN Phone: 615-523-2300 The room rate is $64.00. When making a reservation tell them you are with the MPI meeting. Rooms under MPI for the above rate will be available until Feb. 1, after that, they will be on an availability basis. The hotel provides a shuttle service from the airport to the hotel. There is a hotel phone located in the airport lower level at baggage pickup. Let the hotel know you have arrived and are with MPI and they will arrange to pick you up; or you can call the hotel in advance, give your flight # and arrival time, and the hotel will go ahead and schedule a pick up. The hotel shuttle leaves the hotel on-the-hour and picks up at the airport on-the-half-hour so there could be a delay. You can rent a car or get a cab from the airport to the hotel as well. Its about a 15-20 minute ride. (There is a Hilton located at the Knoxville airport. Do not confuse this hotel with the meeting Hilton located in downtown Knoxville.) We should plan to start at 1:00 pm Wednesday February 23rd and finish about 12:00 pm on Friday February 25th. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV. The format of the meeting is as before: Wednesday, February 23rd 1:00 pm to 8:00 pm 2 Breakouts for 10 people 1 Breakout for 20 people 5:00 pm to 7:00 pm--Dinner. Thursday, February 24th 8:00 am to 10:00 pm 1 U-shape room for 40 people 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. Friday, February 25th 8:00 am to 12:00 pm 1 U-shape room for 40 people If you have any questions, please feel free to contact me. Jack From owner-mpi-comm@CS.UTK.EDU Sat Jan 15 10:01:22 1994 Received: from CS.UTK.EDU by netlib2.cs.utk.edu with ESMTP (8.6.4/2.8t-netlib) id KAA10048; Sat, 15 Jan 1994 10:01:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id KAA00645; Sat, 15 Jan 1994 10:00:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 15 Jan 1994 10:00:16 EST Errors-to: owner-mpi-comm@CS.UTK.EDU Received: from watson.ibm.com by CS.UTK.EDU with SMTP (8.6.4/2.8s-UTK) id JAA00574; Sat, 15 Jan 1994 09:59:52 -0500 Received: from WATSON by watson.ibm.com (IBM VM SMTP V2R3) with BSMTP id 0223; Sat, 15 Jan 94 09:59:53 EST Received: from YKTVMH by watson.vnet.ibm.com with "VAGENT.V1.0" id 4470; Sat, 15 Jan 1994 09:59:24 EST Received: from snir.watson.ibm.com by yktvmh.watson.ibm.com (IBM VM SMTP V2R3) with TCP; Sat, 15 Jan 94 09:59:15 EST Received: by snir.watson.ibm.com (AIX 3.2/UCB 5.64/930311) id AA40957; Sat, 15 Jan 1994 09:59:14 -0500 From: snir@watson.ibm.com (Marc Snir) Message-Id: <9401151459.AA40957@snir.watson.ibm.com> To: mpi-comm@CS.UTK.EDU Subject: a proposal for MPI-IO (don't worry -- this is for MPI-II) Reply-To: snir@watson.ibm.com Date: Sat, 15 Jan 94 09:59:11 -0500 -:) -:) -:) -:) -:) %!PS-Adobe-2.0 %%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software %%Title: mpi-io-v1.dvi %%Pages: 28 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet %%BeginProcSet: special.pro TeXDict begin /SDict 200 dict N SDict begin /@SpecialDefaults{/hs 612 N /vs 792 N /ho 0 N /vo 0 N /hsc 1 N /vsc 1 N /ang 0 N /CLIP false N /BBcalc false N /p 3 def}B /@scaleunit 100 N /@hscale{@scaleunit div /hsc X}B /@vscale{ @scaleunit div /vsc X}B /@hsize{/hs X /CLIP true N}B /@vsize{/vs X /CLIP true N}B /@hoffset{/ho X}B /@voffset{/vo X}B /@angle{/ang X}B /@rwi{10 div /rwi X} B /@llx{/llx X}B /@lly{/lly X}B /@urx{/urx X}B /@ury{/ury X /BBcalc true N}B /magscale true def end /@MacSetUp{userdict /md known{userdict /md get type /dicttype eq{md begin /letter{}N /note{}N /legal{}N /od{txpose 1 0 mtx defaultmatrix dtransform S atan/pa X newpath clippath mark{transform{ itransform moveto}}{transform{itransform lineto}}{6 -2 roll transform 6 -2 roll transform 6 -2 roll transform{itransform 6 2 roll itransform 6 2 roll itransform 6 2 roll curveto}}{{closepath}}pathforall newpath counttomark array astore /gc xdf pop ct 39 0 put 10 fz 0 fs 2 F/|______Courier fnt invertflag{ PaintBlack}if}N /txpose{pxs pys scale ppr aload pop por{noflips{pop S neg S TR pop 1 -1 scale}if xflip yflip and{pop S neg S TR 180 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{pop S neg S TR pop 180 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{ppr 1 get neg ppr 0 get neg TR}if}{noflips{TR pop pop 270 rotate 1 -1 scale}if xflip yflip and{TR pop pop 90 rotate 1 -1 scale ppr 3 get ppr 1 get neg sub neg ppr 2 get ppr 0 get neg sub neg TR}if xflip yflip not and{TR pop pop 90 rotate ppr 3 get ppr 1 get neg sub neg 0 TR}if yflip xflip not and{TR pop pop 270 rotate ppr 2 get ppr 0 get neg sub neg 0 S TR}if} ifelse scaleby96{ppr aload pop 4 -1 roll add 2 div 3 1 roll add 2 div 2 copy TR .96 dup scale neg S neg S TR}if}N /cp{pop pop showpage pm restore}N end}if} if}N /normalscale{Resolution 72 div VResolution 72 div neg scale magscale{ DVImag dup scale}if}N /psfts{S 65536 div N}N /startTexFig{/psf$SavedState save N userdict maxlength dict begin /magscale false def normalscale currentpoint TR /psf$ury psfts /psf$urx psfts /psf$lly psfts /psf$llx psfts /psf$y psfts /psf$x psfts currentpoint /psf$cy X /psf$cx X /psf$sx psf$x psf$urx psf$llx sub div N /psf$sy psf$y psf$ury psf$lly sub div N psf$sx psf$sy scale psf$cx psf$sx div psf$llx sub psf$cy psf$sy div psf$ury sub TR /showpage{}N /erasepage{}N /copypage{}N /p 3 def @MacSetUp}N /doclip{psf$llx psf$lly psf$urx psf$ury currentpoint 6 2 roll newpath 4 copy 4 2 roll moveto 6 -1 roll S lineto S lineto S lineto closepath clip newpath moveto}N /endTexFig{end psf$SavedState restore}N /@beginspecial{SDict begin /SpecialSave save N gsave normalscale currentpoint TR @SpecialDefaults}N /@setspecial{CLIP{newpath 0 0 moveto hs 0 rlineto 0 vs rlineto hs neg 0 rlineto closepath clip}if ho vo TR hsc vsc scale ang rotate BBcalc{rwi urx llx sub div dup scale llx neg lly neg TR}if /showpage{}N /erasepage{}N /copypage{}N newpath}N /@endspecial{grestore clear SpecialSave restore end}N /@defspecial{SDict begin}N /@fedspecial{end}B /li{lineto}B /rl{rlineto}B /rc{rcurveto}B /np{/SaveX currentpoint /SaveY X N 1 setlinecap newpath}N /st{stroke SaveX SaveY moveto}N /fil{fill SaveX SaveY moveto}N /ellipse{/endangle X /startangle X /yrad X /xrad X /savematrix matrix currentmatrix N TR xrad yrad scale 0 0 1 startangle endangle arc savematrix setmatrix}N end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 3 50 df<00600000600000600000600000600000 6000006000006000006000006000FFFFF0FFFFF000600000600000600000600000600000600000 600000600000600000600014167E9119>43 D<0FC01FE0387070386018E01CE01CE01CE01CE01C E01CE01CE01CE01C6018703838701FE00FC00E137F9211>48 D<06001E00FE00EE000E000E000E 000E000E000E000E000E000E000E000E000E000E00FFE0FFE00B137D9211>I E /Fb 1 2 df<60F0F06004047E880A>1 D E /Fc 10 119 df<60F0F07030303060E040040A7E 830A>59 D<07FFE007FFF801C03801C01C01C01C01C01C0380380380700381E003FFC007FFE007 00700700700700380E00700E00700E00E00E03C0FFFF80FFFE0016147F9319>66 D<07FC7FC007FC7FC001C01C0001C01C0001C01C0001C01C0003803800038038000380380003FF F80007FFF0000700700007007000070070000E00E0000E00E0000E00E0000E00E000FF8FF800FF 8FF8001A147F931B>72 D<3E003E000C000C000C000C00180019E01FF01E303830383030303030 7060606660C660CCC0FCC0700F147E9313>104 D<03000380030000000000000000001C003E00 6600C600CC000C000C0018001980318033003F001C00091480930C>I<00600070006000000000 00000000038007C00CE018C018C000C000C0018001800180018003000300030003006600EE00FC 0078000C1A81930E>I<387C007CFE006F8600CF0600CE06000E06000C06001C0C00180CC01818 C0181980301F80300E00120D808C15>110 D<38E07DF06F30CE70CE300C000C00180018001800 1800300030000C0D808C0F>114 D<07001F8019C0318030001E001F0003804180E180C300FE00 7C000A0D7E8C10>I<1C083E186618C618CC180C180C181830183018201C600FC007800D0D808C 11>118 D E /Fd 27 123 df<70F8F8F87005057C840D>58 D<70F8FCFC7C0C0C0C1818383060 40060E7C840D>I<000001C00000078000001E00000078000001E00000078000000E0000003C00 0000F0000003C000000F0000003C000000F0000000F00000003C0000000F00000003C0000000F0 0000003C0000000E0000000780000001E0000000780000001E0000000780000001C01A1A7C9723 >I<00030003000700060006000E000C000C001C0018001800380030003000700060006000E000 C000C001C00180018001800380030003000700060006000E000C000C001C001800180038003000 3000700060006000E000C000C000102D7DA117>II<00FFFFE000FFFFF8000F007C000F003C000F003E001E003E 001E003E001E003E001E003E003C007C003C007C003C00F8003C01F0007807E0007FFF80007FFF C0007803E000F001F000F001F000F000F000F000F001E001F001E001F001E001F001E001E003C0 03E003C007C003C00F8003C01F007FFFFE00FFFFF0001F1F7E9E22>66 D<00FFF9FFF000FFF9FF F0000F001E00000F001E00000F001E00001E003C00001E003C00001E003C00001E003C00003C00 7800003C007800003C007800003C007800007800F000007FFFF000007FFFF000007800F00000F0 01E00000F001E00000F001E00000F001E00001E003C00001E003C00001E003C00001E003C00003 C007800003C007800003C007800003C00780007FFCFFF800FFF8FFF800241F7E9E26>72 D<00FFFC01FFF8000F00000F00000F00001E00001E00001E00001E00003C00003C00003C00003C 0000780000780000780000780000F00000F00000F00000F00001E00001E00001E00001E00003C0 0003C00003C00003C000FFFC00FFFC00161F7F9E14>I86 D<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E00 F00E00F00E18F01C30F03C30707C3078FC603FCFC00F078015147E9318>97 D<07803F803F000700070007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F 700F700F700FE01EE01EE01EE03CE03CE038607071E03FC01F0010207E9F14>I<007C01FE0787 0F0F1E0F1C0E3C04780078007800F000F000F000F000F00070027007381E1FF80FE010147E9314 >I<007C01FE07870E031E033C033C06781C7FF87FC0F000F000F000F000700070027007381E1F F807E010147E9315>101 D<01E0000FE0000FC00001C00001C00001C000038000038000038000 038000070000070000071F00077F800EE1C00F81C00F00E00E00E01E01C01C01C01C01C01C01C0 38038038038038038038070670070C700E0C700E18700638E007F06003C017207E9F1A>104 D<007000F000F00060000000000000000000000000000007001F8039C031C061C061C0C3800380 0380070007000E000E000E181C301C301C601CE00FC007000D1F7F9E10>I<0000C00001E00001 E00001C0000000000000000000000000000000000000000000003E00007F0000C3800183800303 80030380060700000700000700000700000E00000E00000E00000E00001C00001C00001C00001C 0000380000380000380000380000700000700030700078E000F3C0007F80003E00001328819E13 >I<01E0000FE0000FC00001C00001C00001C00003800003800003800003800007000007000007 01C00707E00E0C300E18700E30F00E60F01CC0E01F80001F80001FE00038F00038380038380038 1C307038607038607038407018C0E01F80600F0014207E9F18>I<03C01FC01F80038003800380 07000700070007000E000E000E000E001C001C001C001C00380038003800380070007000700071 80E300E300E300E6007E003C000A207E9F0E>I<0F03E03E001F8FF0FF0031DC39C38061F03B03 8061E01E01C061C01C01C0C3C03C03800380380380038038038003803803800700700700070070 070007007007000700700E0C0E00E00E180E00E01C180E00E01C300E00E00C701C01C00FE00C00 C0078026147F9328>I<0F07C0001F8FE00031D8700061F0380061E0380061C03800C3C0700003 80700003807000038070000700E0000700E0000700E0000701C1800E01C3000E0383000E038600 0E018E001C01FC000C00F00019147F931B>I<007C0001FF000383800F01C01E01C01C01E03C01 E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F00701E003838001FF00007C0 0013147E9316>I<03C1E007E7F80C7E3C187C1C18781E18701E30E01E00E01E00E01E00E01E01 C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E000700000700000E00000E 00000E00000E00001C0000FFC000FFC000171D819317>I<0F0F001F9F8031F1C061E1C061E3C0 61C3C0C3C3800380000380000380000700000700000700000700000E00000E00000E00000E0000 1C00000C000012147F9315>114 D<007C01FE0387070F070F070E0F00078007F803FC01FE001E 000F700FF00EF00EE01C70383FF01FC010147E9315>I<00C000E001C001C001C001C003800380 FFF8FFF807000700070007000E000E000E000E001C001C001C001C1838303830386038E01FC00F 000D1C7F9B10>I<0780C01FC1E038C1F030E0F060E0F061C070C1C06003806003806003806007 00C00700C00700C0070180070180070300070200078E0003FC0000F00014147F9316>118 D<01E06003F0C007FFC00E1F800C0300000600000C0000180000300000600000C0000180000300 000600C00C01801803803F87007FFE0061FC00C0700013147E9315>122 D E /Fe 5 81 df<0018007800F001E003C007800F001F001E003E003C007C007C007800F800F8 00F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800 F800F8000D25707E25>56 D58 D<007C007C007C007C007C007C007C007C007C007C007C00 7C007C007C007C007C007C007C007C007C007C007C007C007C00F800F800F800F001F001E003E0 03C0078007000E001C003800F000C000F00038001C000E000700078003C003E001E001F000F000 F800F800F8007C007C007C007C007C007C007C007C007C007C007C007C007C007C007C007C007C 007C007C007C007C007C007C007C0E4D798025>60 D62 D80 D E /Ff 1 106 df105 D E /Fg 5 22 df0 D<70F8F8F87005057C8D0D>I<07E01FF8 3FFC7FFE7FFEFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF7FFE7FFE3FFC1FF807E010127D9317>15 D<000000C0000003C000000F0000003C000000F0000003C000000F0000001C00000078000001E0 0000078000001E00000078000000E0000000780000001E0000000780000001E000000078000000 1C0000000F00000003C0000000F00000003C0000000F00000003C0000000C00000000000000000 0000000000000000000000000000000000000000FFFFFFC0FFFFFFC01A247C9C23>20 DI E /Fh 56 123 df<007000F001F003C007800F001E001C003C003800780070007000F000E000E0 00E000E000E000E000E000E000F00070007000780038003C001C001E000F00078003C001F000F0 00700C24799F18>40 D<6000F00078003C001E000F000780038003C001C001E000E000E000F000 70007000700070007000700070007000F000E000E001E001C003C0038007800F001E003C007800 F00060000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC0007 F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C00011147D9718>I< 3C7E7F7F7F3F0F0E1E7CF870080C788518>44 D<000300000F80001F80003F0000FE0001FC0003 F00007E0001FC0003F80007E0000FC0000FC00007E00003F80001FC00007E00003F00001FC0000 FE00003F00001F80000F8000030011187D9918>60 D<600000F80000FC00007E00003F80001FC0 0007E00003F00001FC0000FE00003F00001F80001F80003F0000FE0001FC0003F00007E0001FC0 003F80007E0000FC0000F8000060000011187D9918>62 D<00700000F80000F80000D80000D800 01DC0001DC0001DC00018C00038E00038E00038E00038E00030600070700070700070700070700 0FFF800FFF800FFF800E03800E03801C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 DI<01FCE003FEE007FFE00F07E01E03E03C01E07800E07000E070 00E0F00000E00000E00000E00000E00000E00000E00000E00000E00000F000007000E07000E078 00E03C01E01E01C00F07C007FF8003FF0001FC00131C7E9B18>I<7FF800FFFE007FFF001C0F80 1C03C01C03C01C01E01C00E01C00E01C00F01C00701C00701C00701C00701C00701C00701C0070 1C00701C00F01C00E01C00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF800141C7F9B18> III<01F9C007FFC00FFFC01F0FC0 1C03C03C03C07801C07001C07001C0F00000E00000E00000E00000E00000E00000E00FF0E01FF0 E00FF0F001C07001C07003C07803C03C03C01C07C01F0FC00FFFC007FDC001F9C0141C7E9B18> I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01FFF C01FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C07F07 F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001C00001C00001C00001C00001C00001 C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001C00001C0007FFF00FFFF807FFF00111C7D9B18>I<7F07F0FF87F87F07F01C03C0 1C07801C07001C0F001C1E001C3C001C38001C78001CF0001DF0001DF8001FF8001FBC001F1C00 1E1E001E0E001C0F001C07001C07801C03801C03C01C01C07F03F0FF87F87F03F0151C7F9B18> 75 D<7FE000FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E0070 7FFFF0FFFFF07FFFF0141C7F9B18>II<7E07F0FF0FF87F07F01D 81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C01C31C01C 39C01C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F9B 18>I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780700700780F00 7FFF003FFE000FF800111C7D9B18>II<0FF8003FFE007FFF0078 0F00700700F00780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E0 0380E00380E00380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC00001C0000 1E00000E00000F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F801C03801C03C0 1C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F001C07801C03801C0380 1C03801C03801C03801C039C1C039C1C039C7F03FCFF81F87F00F0161C7F9B18>I<07F3801FFF 803FFF807C1F80700780F00380E00380E00380E00000F000007800003F00001FF0000FFE0001FF 00001F800003C00001E00000E00000E06000E0E000E0E001E0F001C0FC07C0FFFF80FFFF00E7FC 00131C7E9B18>I<7FFFF8FFFFF8FFFFF8E07038E07038E07038E0703800700000700000700000 700000700000700000700000700000700000700000700000700000700000700000700000700000 700000700007FF0007FF0007FF00151C7F9B18>IIII<7F8FE07F9FE07F8FE00E07000F0700070E00 078E00039C0003DC0001F80001F80000F00000F00000700000F00000F80001F80001DC00039E00 038E00070F000707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>II<3FFFE07FFFE07FFFE07001C07003C0700780700700000F00001E00001C0000 3C0000780000700000F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E078 00E07000E0FFFFE0FFFFE0FFFFE0131C7E9B18>I<1FE0003FF8007FFC00783E00300F00000700 00070001FF000FFF003FFF007F0700780700F00700E00700E00700F00F00783F007FFFF03FFBF0 0FE1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E00000E3E000E FF800FFFC00FE3E00F80F00F00700F00780E00380E00380E00380E00380E00380F00380F00780F 00700F80F00FC3E00FFFC00EFF80067E00151C809B18>I<01FE0007FF001FFF803F07803C0300 780000700000F00000E00000E00000E00000E00000F000007000007801C03C01C03F07C01FFF80 07FF0001FC0012147D9318>I<001F80003F80001F8000038000038000038000038000038003F3 800FFB801FFF803E1F80780F80700780F00780E00380E00380E00380E00380E00380E00780F007 80700780780F803E3F801FFFF00FFBF803E3F0151C7E9B18>I<03F0000FFC001FFE003E1F0078 0780700380F003C0E001C0E001C0FFFFC0FFFFC0FFFFC0F000007000007801C03C01C03F07C01F FF8007FF0001FC0012147D9318>I<001FC0007FE000FFE001F1E001C0C001C00001C00001C000 7FFFC0FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<03F1F007FFF80FFFF81E1F 303C0F003807003807003807003807003807003C0F001E1E001FFC003FF8003BF0003800003C00 001FFF001FFFC03FFFE07801F0F00078E00038E00038E00038F000787800F07E03F03FFFE00FFF 8003FE00151F7F9318>I<7E0000FE00007E00000E00000E00000E00000E00000E00000E3F000E FF800FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E 00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C000038000 0000000000000000000000007FC000FFC0007FC00001C00001C00001C00001C00001C00001C000 01C00001C00001C00001C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18> I107 D<7FE000FFE0007FE00000E00000E00000E00000E00000E0 0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0 0000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7DF1F000FFFBF8007FFF FC001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C 1C1C001C1C1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F001914819318>I<7E 3F00FEFF807FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E 00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F80 3803807001C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C07803E0F80 1FFF000FFE0001F00013147E9318>I<7E3E00FEFF807FFFC00FE3E00F80F00F00700F00780E00 380E00380E00380E00380E00380F00380F00780F00700F80F00FC3E00FFFC00EFF800E7E000E00 000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>I<03F3800F FB801FFF803E1F80780F80700780F00780E00380E00380E00380E00380E00380E00380F0078070 0780780F803E1F801FFF800FFB8003F38000038000038000038000038000038000038000038000 3FF8003FF8003FF8151E7E9318>I<7F87E0FF9FF87FBFF803FC7803F03003E00003C00003C000 03C0000380000380000380000380000380000380000380000380007FFE00FFFF007FFE0015147F 9318>I<0FF7003FFF007FFF00F81F00E00700E00700F007007C00007FF0001FFC0007FE00001F 00600780E00380F00380F00780FC0F00FFFF00FFFE00E7F80011147D9318>I<01800003800003 80000380000380007FFFC0FFFFC0FFFFC003800003800003800003800003800003800003800003 80000380000380400380E00380E00381E003C3C001FFC000FF80007E0013197F9818>I<7E07E0 FE0FE07E07E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E0 0E00E00E01E00F07E007FFFC03FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03 800E03800E0380070700070700070700038E00038E00038E00038E0001DC0001DC0001DC0000F8 0000F80000700015147F9318>II<7F8FF07F9FF07F8FF0070700078E00039E0001DC0001F80000F80000700000F00000F800 01DC00039E00038E000707000F07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8F F00E01C00E03800E0380070380070700070700038700038600038E0001CE0001CE0000CC0000CC 0000DC0000780000780000780000700000700000700000F00000E00079E0007BC0007F80003F00 001E0000151E7F9318>I<3FFFF07FFFF07FFFF07001E07003C0700780000F00001E00007C0000 F80001F00003E0000780000F00701E00703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I E /Fi 2 63 df<0000038000000F0000003C000000F0000003C000000F0000003C000000F00000 03C000000F0000003C000000F0000000F00000003C0000000F00000003C0000000F00000003C00 00000F00000003C0000000F00000003C0000000F000000038019187D9520>60 D62 D E /Fj 43 122 df<0003C001F3C003F3C007F3C00700000E00000E00000E00000E00000E00000E00000E 0000FFF1C0FFF1C0FFF1C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E01C00E01C0121E809D16>12 D<7878787838307060E005097D830C >44 D<00030003000700060006000E000C001C0018001800380030003000700060006000E000C0 00C001C001800380030003000700060006000E000C000C001C0018001800380030007000600060 00E000C000C00010297E9E15>47 D<001C0000003E0000003E0000002E00000067000000670000 00E7800000C7800000C3800001C3C0000183C0000181C0000381E0000381E0000700F0000700F0 000600F0000E0078000FFFF8000FFFF8001FFFFC001C003C0018003C0038001E0038001E007000 1F0070000F0070000F00E0000780191D7F9C1C>65 DI<00 3FC000FFF003FFF007C0700F80001E00003E00003C0000780000780000780000F00000F00000F0 0000F00000F00000F00000F00000F00000F000007800007800007800003C00003E00001E00000F 800807C07803FFF800FFF0003F80151F7D9D1B>IIII<003F8001FFF003FFF8 07C0F80F00181E00003E00003C0000780000780000780000F00000F00000F00000F00000F00000 F00000F007F8F007F8F007F87800387800387800383C00383E00381E00380F003807C07803FFF8 01FFF0003F80151F7D9D1C>III75 DIII<003F000001FFE00003FFF00007C0F8000F807C001E001E003E00 1F003C000F00780007807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F0 0003C0F00003C0F00003C0F80007C078000780780007807C000F803C000F003E001F001F003E00 0F807C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F>II<003F000001FFE00003FFF00007C0F8000F807C001F003E003E001F003C000F00780007 807800078078000780F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F000 03C0F00003C07800078078000780781E07803C0F0F003E079F001E07DE000F83FC0007C1F80003 FFF00001FFF800003F7C0000003C0000003E0000001F0000000F801A237E9D1F>II<03F8000FFE001FFF003E0F00780300780000F00000F00000F00000F0 0000F800007C00007F00003FE0001FFC0007FE0001FF00001F800007800007C00003C00003C000 03C00003C00003C0C00780E00780FC1F007FFE001FFC0007F000121F7E9D17>II< F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070 F00070F00070F00070F00070F00070F00070F00070F00070F00070F000F07800E07801E03C03C0 1F07800FFF8007FE0001F800141E7C9C1D>III<78000E007C001E003C003C001E0038000F00 70000F00F0000781E00003C1C00001C3C00001E7800000F70000007E0000003E0000003C000000 3C0000007E00000077000000E7800001E3800003C1C0000381E0000700F0000F00F8000E007800 1C003C003C003E0078001F0070000F00F0000F80191D7F9C1C>I<0FC03FF07FF87038401C001C 001C00FC0FFC3FFC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 D<07E00FF81FFC3C1C70047000E000E000E000E000E000E000700070043C1C1FFC0FF807E00E12 7E9112>99 D<000E000E000E000E000E000E000E000E000E000E000E0F8E1FEE3FFE7C3E700E70 0EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E0F1D7E9C15>I<07C01FE03FF078 787018601CFFFCFFFCFFFCE000E000E000700070043C1C3FFC1FF807E00E127E9112>I<00FC01 FC03FC07000E000E000E000E000E000E000E00FFE0FFE0FFE00E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E1D809C0D>I105 D108 DII<03F0000FFC001FFE003C0F0078 0780700380E001C0E001C0E001C0E001C0E001C0F003C07003807807803C0F001FFE000FFC0003 F00012127F9115>II114 D<1FC03FF07FF0F030E000E000F0007F003FC01FE000F0003800388038F078FFF07FE01FC00D12 7F9110>I<1C001C001C001C001C001C00FFE0FFE0FFE01C001C001C001C001C001C001C001C00 1C001C001C001C201FF00FF007C00C187F970F>III121 D E /Fk 53 123 df<003F0F0000FFBF8003C3F3C00703E3C00703C1 800E01C0000E01C0000E01C0000E01C0000E01C0000E01C000FFFFFC00FFFFFC000E01C0000E01 C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0007F87FC007F87FC001A1D809C18>11 D<003F0000FF8003C1C00703C0 0703C00E01800E00000E00000E00000E00000E0000FFFFC0FFFFC00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F87F87F8151D80 9C17>I<70F8FCFC7C0C0C0C1818306040060D7D9C0C>39 D<00C00180030006000E000C001C00 18003800300030007000700060006000E000E000E000E000E000E000E000E000E000E000E000E0 00600060007000700030003000380018001C000C000E0006000300018000C00A2A7D9E10>II<70F0F8F8781818183030706040050D7D840C>44 DI<70F8F8F87005057D840C>I<00030003000700060006000E000C 001C0018001800380030003000700060006000E000C000C001C001800380030003000700060006 000E000C000C001C001800180038003000700060006000E000C000C00010297E9E15>I<70F8F8 F870000000000000000070F8F8F87005127D910C>58 D<70F8F8F870000000000000000070F0F8 F8781818183030706040051A7D910C>I<0FE03FF8703C601CF01EF01EF01E001E003C007800E0 01C001800380030003000300030003000200000000000000000007000F800F800F8007000F1D7E 9C14>63 D<00060000000F0000000F0000000F0000001F8000001F8000001F8000001F80000033 C0000033C0000033C0000061E0000061E0000061E00000C0F00000C0F00000C0F0000180780001 80780001FFF80003FFFC0003003C0003003C0006001E0006001E0006001E001F001F00FFC0FFF0 FFC0FFF01C1D7F9C1F>65 D68 D70 D72 DI77 DI<003F800000FFE00003E0F80007803C000E000E00 1E000F003C00078038000380780003C0780003C0700001C0F00001E0F00001E0F00001E0F00001 E0F00001E0F00001E0F00001E0F00001E0780003C0780003C0780003C03C0007803C0007801E00 0F000F001E0007803C0003E0F80000FFE000003F80001B1E7E9C20>II<07E0801FF9803C1F80700780700380E00380E00180E00180E00180F00000F000007C0000 7FC0003FF8001FFE0007FF0000FF80000F800003C00003C00001C0C001C0C001C0C001C0E00180 E00380F00300FC0E00CFFC0083F800121E7E9C17>83 D<7FFFFFC07FFFFFC0780F03C0700F01C0 600F00C0E00F00E0C00F0060C00F0060C00F0060C00F0060000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F 0000000F0000000F000003FFFC0003FFFC001B1C7F9B1E>IIII<06000F001F80 39C070E0E07040200C077C9C15>94 D<0FE0001FF8003C3C003C1E00180E00000E00001E0007FE 001FFE003E0E00780E00F00E00F00E60F00E60F01E60783E603FFFC01F878013127F9115>97 DI<03F00FF81E3C383C78187000F000F000F000F000F000 F000780078063C061E0C0FF803E00F127F9112>I<001F80001F80000380000380000380000380 00038000038000038000038000038003E3800FFB801E0F80380780780380700380F00380F00380 F00380F00380F00380F003807003807803803807801E1F800FFBF007E3F0141D7F9C17>I<03E0 0FF01C38381C781E700EFFFEFFFEF000F000F000F000700078063C061E0C0FF803E00F127F9112 >I<007801FC039E071E0E0C0E000E000E000E000E000E00FFE0FFE00E000E000E000E000E000E 000E000E000E000E000E000E000E000E007FE07FE00F1D809C0D>I<00038007E7C00FFDC03C3D C0381C00781E00781E00781E00781E00381C003C3C003FF00037E0007000007000003000003FFC 001FFF003FFF80700780E001C0E001C0E001C0E001C07003803C0F001FFE0007F800121C7F9215 >II<18003C007C003C0018000000000000000000000000 00FC00FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80FF80091D 7F9C0C>I<01C003E003E003E001C00000000000000000000000000FE00FE000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060E0F1C0F1C07F803E 000B25839C0D>IIIII<03F0000FFC001E1E00380700780780700380F003C0F003C0F003 C0F003C0F003C0F003C07003807807803807001E1E000FFC0003F00012127F9115>II<03E1800FF9801E1F803C0780780780780380F00380F00380F00380F00380F00380F00380 7803807807803C07801E1F800FFB8007E380000380000380000380000380000380000380001FF0 001FF0141A7F9116>II<1F903FF07070E030E030E030F8007F803FE00FF000F8C038C0 38E038E038F070DFE08FC00D127F9110>I<0C000C000C000C000C001C001C003C00FFE0FFE01C 001C001C001C001C001C001C001C001C301C301C301C301C301E600FC007800C1A7F9910>IIII<7F8FF07F8FF00F0780070600038E0001DC0001D80000F00000700000780000F80001 DC00038E00030E000607000F0380FF8FF8FF8FF81512809116>II<7FFC7FFC78 38707060F060E061C063C00380070C0F0C0E0C1C1C3C1838187078FFF8FFF80E127F9112>I E /Fl 20 119 df<387CFEFEFE7C3807077D860D>46 D<387CFEFEFE7C3800000000387CFEFEFE 7C3807127D910D>58 D<001FE02000FFF8E003F80FE007E007E00FC001E01F8001E03F0000E03F 0000E07E0000607E000060FE000060FE000000FE000000FE000000FE000000FE000000FE000000 FE0000007E0000607E0000603F0000603F0000C01F8000C00FC0018007E0030003F80E0000FFFC 00001FE0001B1C7D9B22>67 DI73 D<0FF8001FFE003E1F803E07803E07C01C07C00007C003FFC01FFFC03F87C07E07C0FC07C0FC07 C0FC07C0FC0FC07E1FC03FFBF80FE1F815127F9117>97 D<03FC000FFE001F1F003E1F007C1F00 7C0E00FC0000FC0000FC0000FC0000FC0000FC00007C00007E01803E03801F07000FFE0003F800 11127E9115>99 D<01FC000FFF001F0F803E07C07C03C07C03E0FC03E0FFFFE0FFFFE0FC0000FC 0000FC00007C00007E00603E00C01F81C00FFF0001FC0013127F9116>101 D104 D<1E003F003F007F003F003F001E00000000000000 00000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FF E00B1E7F9D0E>I108 DII<01FC000FFF801F07C03E03E07C01F07C01F0FC01F8FC01F8FC01F8FC01F8FC01F8FC01F8 7C01F07C01F03E03E01F07C00FFF8001FC0015127F9118>II114 D<1FD83FF87038E018E018F000FF807FE07FF01FF807FC 007CC01CC01CE01CF038FFF0CFC00E127E9113>I<030003000300070007000F000F003F00FFFC FFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F9807F003E00E1A7F9913 >III E /Fm 55 123 df<007F07F001FF1FF003FF3FF007807800070070000F00F000 0F00F0000F00F0000F00F0000F00F0000F00F0000F00F000FFF8FF80FFF8FF80FFF8FF800F00F0 000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00 F0000F00F0000F00F0000F00F0000F00F0000F00F0000F00F0001C20809F1B>11 D<0000F000F8F001F8F003F8F00780000700000F00000F00000F00000F00000F00000F00000F00 00FFF8F0FFF8F0FFF8F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00 F00F00F00F00F00F00F00F00F00F00F00F00F00F00F0142180A018>I<007000E001C003800780 07000E001E001E003C003C003C0078007800780078007000F000F000F000F000F000F000F000F0 00F000F000F000F000700078007800780078003C003C003C001E001E000E0007000780038001C0 00E000700C2E7EA112>40 DI<000600000006 000000060000000600000006000000060000000600000006000000060000000600000006000000 060000000600000006000000060000FFFFFFF0FFFFFFF000060000000600000006000000060000 000600000006000000060000000600000006000000060000000600000006000000060000000600 00000600001C207D9A23>43 D<787878781830306060E0050A7D830D>II<03F00007F8001FFE001E1E003C0F00380700780780780780700380F003C0F003C0F003 C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07807807807 807807803807003C0F001E1E001FFE000FFC0003F00012207E9E17>48 D<00C001C00FC0FFC0FF C0F3C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 03C003C003C003C0FFFEFFFEFFFE0F1F7C9E17>I61 D<001F0000001F0000003F8000003B8000003B8000007BC0000073C0000071C00000F1E00000E1 E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007 003C000FFFFE000FFFFE000FFFFE001E000F001E000F003C000F803C0007803C000780780007C0 780003C0780003C0F00003E01B207F9F1E>65 DI<001FC000FFF801FFFC03E03C07800C0F00001E00003E00003C00007C0000780000 780000780000F00000F00000F00000F00000F00000F00000F00000F00000780000780000780000 7C00003C00003E00001E00000F000207800E03E03E01FFFC00FFF0001FC017227DA01D>IIII<001FE000FF F801FFFE03E03E07800E0F00001E00003E00003C00007C0000780000780000780000F00000F000 00F00000F00000F00000F00000F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E00 1E0F001E07801E03E03E01FFFE00FFF8001FC017227DA01E>III75 DIII<003F000000FFC00003 FFF00007E1F8000F807C001F003E001E001E003C000F003C000F00780007807800078078000780 F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007 C07800078078000780780007803C000F003C000F001E001E001F003E000F807C0007E1F80003FF F00000FFC000003F00001A227DA021>II82 D<01FC0007FF800FFFC01F 03C03C00C03C00007800007800007800007800007800007C00003C00003F00001FE0000FFC0007 FE0001FF00003F800007C00003C00003E00001E00001E00001E00001E00001E00001C0C003C0F0 07C0FC0F807FFF001FFE0003F80013227EA019>II< F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0 F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F07801E0 7801E03C03C03C03C01F0F800FFF0007FE0001F80014217B9F1F>III89 D<07E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF80FF00FF00FF00FF83F7FFF3FEF 1F8F10147E9316>97 DI<03F00FFC 1FFE3E0E3C0278007800F000F000F000F000F000F000780078003C013E0F1FFF0FFE03F010147E 9314>I<0007800007800007800007800007800007800007800007800007800007800007800007 8007C7800FF7801FFF803E1F807C0780780780F80780F00780F00780F00780F00780F00780F007 80F00780780780780F803E1F801FFF800FF78007C78011207E9F17>I<03F0000FFC001FFE003E 1F003C0700780700700380FFFF80FFFF80FFFF80F00000F00000F000007000007800003C01003E 07001FFF0007FE0001F80011147F9314>I<007E01FE03FE078007000F000F000F000F000F000F 000F00FFF0FFF0FFF00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F000F20809F0E>I104 DI108 DII<01F80007FE001FFF803F0FC03C03 C07801E07801E0F000F0F000F0F000F0F000F0F000F0F000F07801E07801E03C03C03F0FC01FFF 8007FE0001F80014147F9317>II<03C7800FF7801FFF80 3E1F807C0F80780780780780F00780F00780F00780F00780F00780F00780F807807807807C0F80 3E1F801FFF800FF78007C780000780000780000780000780000780000780000780000780000780 111D7E9317>II<07F01FFC3FFC780C7800780078007C003FC01FF00FF803F8 007C003C003CC03CF07CFFF87FF00FC00E147F9311>I<1E001E001E001E001E001E00FFF0FFF0 FFF01E001E001E001E001E001E001E001E001E001E001E001E001E001E201FF00FF007C00C1A7F 9910>II II<7801E07C03C03E07801E0F000F0F00079E0003FC0003F80001F80000F0 0001F00001F80003FC00079E000F0F000E0F001E07803C03C07801E0F801F01414809315>II<7FFF7FFF7FFF003E003C007800F800F001E003E007C00780 0F001F001E003C007C00FFFFFFFFFFFF10147F9314>I E /Fn 35 121 df<000FE000007FF800 00F81C0001E07C0003E07C0007C07C0007C07C0007C0380007C0000007C0000007C0000007C1FE 00FFFFFE00FFFFFE0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C0 3E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E003FF9FFC03F F9FFC01A20809F1D>12 D<0018007000E001C00380038007000E000E001E001C003C003C007800 780078007800F800F000F000F000F000F000F000F000F000F000F80078007800780078003C003C 001C001E000E000E0007000380038001C000E0007000180D2D7DA114>40 DI45 D<00700000F0000FF000FFF000F3F00003F00003F00003F00003F00003F00003F00003F00003F0 0003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F0 0003F000FFFF80FFFF80111D7C9C1A>49 D<07F0001FFE00383F007C1F80FE0FC0FE0FC0FE0FE0 FE07E07C07E03807E0000FE0000FC0000FC0001F80001F00003E0000780000F00000E00001C060 0380600700600C00E01FFFE01FFFC03FFFC07FFFC0FFFFC0FFFFC0131D7D9C1A>I<3803803FFF 803FFF003FFE003FFC003FF0003F800030000030000030000030000033F8003FFE003C1F00380F 80300FC0000FC0000FE0000FE0780FE0FC0FE0FE0FE0FE0FE0FC0FC0780FC0601F80383F001FFC 0007F000131D7D9C1A>53 D<387CFEFEFE7C38000000000000387CFEFEFE7C3807147C930F>58 D<0007FC02003FFF0E00FE03DE03F000FE07E0003E0FC0001E1F80001E3F00000E3F00000E7F00 00067E0000067E000006FE000000FE000000FE000000FE000000FE000000FE000000FE0000007E 0000007E0000067F0000063F0000063F00000C1F80000C0FC0001807E0003803F0007000FE01C0 003FFF800007FC001F1F7D9E26>67 D72 D77 D80 D<03FC080FFF381E03F83C00F8780078780038F80038F80018FC0018FC0000FF0000FFF8007FFF 007FFFC03FFFE01FFFF00FFFF803FFF8001FFC0001FC0000FC0000FCC0007CC0007CC0007CE000 78E00078F800F0FE01E0E7FFC081FF00161F7D9E1D>83 D85 DI<07FC001FFF003F0F803F07C03F03E03F03E00C03E00003E001FFE00FFFE03F83 E07E03E07C03E0F803E0F803E0F803E0FC07E07E0DE03FF9FE07E07E17147F9319>97 DI<01FE0007FF801F0FC03E0FC03E 0FC07C0FC07C0300FC0000FC0000FC0000FC0000FC0000FC00007C00007E00003E00603F00C01F 81C007FF0001FC0013147E9317>I<0007F80007F80000F80000F80000F80000F80000F80000F8 0000F80000F80000F80000F801F8F80FFEF81F83F83E01F87E00F87C00F87C00F8FC00F8FC00F8 FC00F8FC00F8FC00F8FC00F87C00F87C00F87E00F83E01F81F07F80FFEFF03F8FF18207E9F1D> I<01FE0007FF801F83E03F01F07E00F07E00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00 007C00007E00003E00183F00380F807007FFE000FF8015147F9318>I<001F8000FFC001F3E003 E7E003C7E007C7E007C3C007C00007C00007C00007C00007C000FFFC00FFFC0007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C0003FFC003FFC0013207F9F10>I<01FC3C07FFFE0F07DE1E03DE3E03E03E03E03E03E03E03E0 3E03E01E03C00F07800FFF0019FC001800001800001C00001FFF801FFFF00FFFF83FFFFC7C007C 70003EF0001EF0001EF0001E78003C78003C3F01F80FFFE001FF00171E7F931A>II<1C003F007F007F007F003F001C000000000000 00000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00 1F001F00FFE0FFE00B217EA00E>I108 DI< FE0FC0FE3FE01E61F01EC0F81F80F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00F8 1F00F81F00F81F00F81F00F81F00F8FFE3FFFFE3FF18147D931D>I<01FF0007FFC01F83F03E00 F83E00F87C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007E7C007C7C007C3E00F83E00 F81F83F007FFC001FF0017147F931A>II114 D<0FE63FFE701E600EE006E006F800FF C07FF83FFC1FFE03FE001FC007C007E007F006F81EFFFCC7F010147E9315>I<01800180018003 800380038007800F803F80FFFCFFFC0F800F800F800F800F800F800F800F800F800F800F860F86 0F860F860F8607CC03F801F00F1D7F9C14>III120 D E /Fo 30 120 df<000078007C7800FC7801FC7803C40003800007800007800007800007800007800007 8000078000078000FFFC78FFFC78FFFC7807807807807807807807807807807807807807807807 8078078078078078078078078078078078078078078078078078078078078078078078152480A3 1A>12 D46 D<00C001C007C0FFC0FFC0FBC003C003C003C003C003C003 C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0 FFFFFFFFFFFF10227CA118>49 D<03F0000FFC001FFE003C1F003807807007C07003C0F003E0E0 01E06001E02001E00001E00001E00001E00003C00003C0000780000780000F00001E00003C0000 780000F00001E00001C0000380000700000E00001C0000380000700000FFFFE0FFFFE0FFFFE013 227EA118>I<01F00007FC001FFF003E0F003807807003C02003C02003C00003C00003C00003C0 000780000780000F00001E0003FC0003F80003FE00000F000007800003C00003C00001E00001E0 0001E00001E00001E08001E0C003C0E003C07007803C0F801FFF000FFC0003F00013237EA118> I<3FFF803FFF803FFF803C00003C00003C00003C00003C00003C00003C00003C00003C00003CF8 003FFE003FFF003F0F803E07803C03C03803C00001E00001E00001E00001E00001E00001E00001 E04003C04003C0E003C07007807C1F003FFE000FFC0003F00013227EA018>53 D<007E0001FF0003FF0007C1000F00001E00001E00003C00003C000078000078000078F800F3FE 00F7FF00FF0F80FC0780F803C0F803C0F801C0F001E0F001E0F001E0F001E0F001E07001E07001 E07801E07803C03803C03C03801C07801F0F000FFE0007FC0001F00013237EA118>I<01F00007 FC000FFE001E0F003C07803C07807803C07803C07803C07803C07803C03803803C07801E0F000F 1E0007FC0003F8000FFE001E0F003C07807803C07803C0F001E0F001E0F001E0F001E0F001E0F0 01E07803C07803C03C07803E0F801FFF0007FC0001F00013237EA118>56 D<001F0000001F0000003F8000003F8000003B8000007BC0000073C0000071C00000F1E00000F1 E00000E0E00001E0F00001E0F00001C0F00003C0780003C078000380780007803C0007803C0007 003C000F001E000F001E000FFFFE001FFFFF001FFFFF001C000F003C0007803C00078038000780 780003C0780003C0700003C0F00001E0F00001E0E00001E01B237EA220>65 D68 D70 D86 D<07E01FF83FFC3C1E301E200F000F000F000F01FF0FFF3FFF7F0F780FF00FF00FF00FF80F7C3F 7FFF3FFF1F8F10167E9517>97 DI<01FC0007FF000FFF801F07803C0180780000780000700000F00000F00000 F00000F00000F00000F000007800007800007800403C00C01F07C00FFFC007FF8001FC0012167E 9516>I<0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003 C00003C003F3C00FFFC01FFFC03F0FC03C07C07803C07803C0F003C0F003C0F003C0F003C0F003 C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00FFBC003E3C012237EA219>I<03 F00007FC001FFE003E0F003C0780780380780380F001C0FFFFC0FFFFC0FFFFC0F00000F00000F0 00007000007800007800003C00801F07800FFF8007FF0001F80012167E9516>I<01F0F807FFF8 0FFFF81F1F801E0F003C07803C07803C07803C07803C07801E0F001F1F001FFE001FFC0039F000 3800003800003C00003FFE001FFFC01FFFE03FFFF07801F0F800F8F00078F00078F00078F00078 7800F03E03E01FFFC00FFF8001FC0015217F9518>103 D105 D107 DIII<01FC0007FF000FFF801F07C03C01E07800F07800 F0700070F00078F00078F00078F00078F00078F000787800F07800F07C01F03E03E01F07C00FFF 8007FF0001FC0015167F9518>II< F0E0F3E0F7E0FFE0FF00FC00F800F800F000F000F000F000F000F000F000F000F000F000F000F0 00F000F0000B167C9511>114 D<07F01FFE3FFF3C0F7803780078007C003F003FF01FF80FFC01 FE001F000F000F400FE00FF81EFFFE3FFC0FF010167F9513>I<0F000F000F000F000F000F00FF F8FFF8FFF80F000F000F000F000F000F000F000F000F000F000F000F000F000F000F080F1C07FC 07F803E00E1C7F9B12>II 119 D E /Fp 26 87 df<3078F8787005057C840D>46 D<00000300000700000600000E00000C 00001C0000180000380000300000700000600000E00000C00001C0000180000380000300000700 000600000E00000C00001C0000180000380000300000700000600000E00000C00001C000018000 0380000300000700000600000E00000C00001C0000180000380000300000700000600000E00000 C00000182D7FA117>I<000C001C00FC0FF80F3800380038003800380070007000700070007000 7000E000E000E000E000E000E001C001C001C001C001C001C003C07FFEFFFE0F1E7C9D17>49 D<007F0001FFC00383E00701E00700F00F00F00F01F00F01F00001E00001E00003C0000780000F 0000FE0000F800000E000007000007800003C00003C00003C03007C07807C0F807C0F807C0F00F 80C00F00601E00383C001FF80007E000141F7D9D17>51 D<0001C00003C00003C0000780000F80 001B80003B8000738000638000C7000187000307000707000E07000C0700180E00300E00600E00 E00E00FFFFF0FFFFF0001C00001C00001C00001C00001C00001C00003C0003FFC003FFC0141E7D 9D17>I<03007003FFF003FFE003FF8003FE000600000600000600000600000600000600000C7C 000DFF000F87800E03800C01C00C01C00001E00001E00001E00001E07003C0F003C0F003C0F003 80C00780E00700600E00383C001FF0000FC000141F7D9D17>I<000F80007FC000F0E001C0E003 81E00701E00E00C01E00001C00003C000038000079FC007BFE007E07007C0380F80380F803C0F0 03C0F003C0F003C0F003C0F00780E00780E00780E00700E00F00700E00701C003838001FF00007 C000131F7C9D17>I<003F0000FF8001C1E00380E00700700E00700E00700E00700E00E00F00E0 0F81C00FC38007F70003FC0001FC0003FF000F3F801C0F803807C03803C07001C07001C0E001C0 E001C0E001C0E003807003007007003C1C001FF80007E000141F7D9D17>56 D<0000180000003800000038000000780000007C000000FC000000FC000001BC000001BC000003 3C0000033E0000061E0000061E00000C1E00000C1E0000181E0000181F0000300F0000300F0000 600F00007FFF0000FFFF0000C00F00018007800180078003000780030007800600078006000780 1F0007C0FFC07FFCFFC07FFC1E207E9F22>65 D<0003F808000FFE18003F0738007801F801F000 F803C000F007C00070078000700F0000701F0000701E0000703E0000603C0000607C0000007C00 00007C0000007C000000F8000000F8000000F8000000F8000000F80000C0780000C0780001807C 0001803C0003003C0003001E0006001E000C000F80180007E0700001FFC000007F00001D217B9F 21>67 D<07FFFF0007FFFFE0003C01F0003C00F80078007C0078003C0078001E0078001E007800 1E0078001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E0003E01E0 003E01E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003C003C00F8007 C03F007FFFFC00FFFFE000201F7E9E23>I<07FFFFF807FFFFF8003C00F8003C00780078003800 78003800780038007800380078003800780C3000F0183000F0180000F0180000F0380000FFF800 00FFF80001E0700001E0300001E0300001E0301801E0303001E0003003C0003003C0006003C000 6003C000E003C001C003C003C007C00FC07FFFFF80FFFFFF801D1F7E9E1F>I<07FFFFF807FFFF F8003C00F8003C0078007800380078003800780038007800380078003800780C3000F0183000F0 180000F0180000F0380000FFF80000FFF80001E0700001E0300001E0300001E0300001E0300001 E0000003C0000003C0000003C0000003C0000003C0000003C0000007C000007FFE0000FFFE0000 1D1F7E9E1E>I<0001FC04000FFF0C003F039C007800FC00F000FC03E0007803C0007807800038 0F0000381F0000381E0000383E0000303E0000307C0000007C0000007C0000007C000000F80000 00F8000000F8007FFCF8007FFCF80001E0780001E0780003C07C0003C03C0003C03C0003C01E00 03C00F0007C007800F8003E0398001FFF080003F80001E217B9F24>I<07FFC7FFC007FFC7FFC0 003C007800003C007800007800F000007800F000007800F000007800F000007800F000007800F0 0000F001E00000F001E00000F001E00000F001E00000FFFFE00000FFFFE00001E003C00001E003 C00001E003C00001E003C00001E003C00001E003C00003C007800003C007800003C007800003C0 07800003C007800003C007800007C00F80007FFCFFF800FFF8FFF800221F7E9E22>I<07FFE007 FFE0003C00003C0000780000780000780000780000780000780000F00000F00000F00000F00000 F00000F00001E00001E00001E00001E00001E00001E00003C00003C00003C00003C00003C00003 C00007C000FFFC00FFFC00131F7F9E10>I<07FFF00007FFF000003C0000003C00000078000000 7800000078000000780000007800000078000000F0000000F0000000F0000000F0000000F00000 00F0000001E0000001E0000001E0000001E0018001E0018001E0030003C0030003C0030003C007 0003C0060003C00E0003C01E0007C07E007FFFFC00FFFFFC00191F7E9E1C>76 D<07FC0000FFC007FC0001FFC0003E0001F800003E00037800006E0003F000006E0006F000006E 0006F000006E000CF0000067000CF00000670018F00000C70019E00000C70031E00000C70031E0 0000C70061E00000C38061E00000C380C1E000018380C3C00001838183C00001838183C0000181 C303C0000181C303C0000181C603C0000301C60780000301CC0780000301CC0780000300F80780 000300F80780000700F00780000F80F00F80007FF0E0FFF800FFF0E1FFF8002A1F7E9E2A>I<07 FC03FFC007FC03FFC0003E007C00003E003800006F003000006F003000006F0030000067803000 00678030000063C0300000C3C0600000C3C0600000C1E0600000C1E0600000C0F0600000C0F060 000180F0C000018078C000018078C00001803CC00001803CC00001803CC00003001F800003001F 800003000F800003000F800003000F800007000780000F800700007FF0030000FFF0030000221F 7E9E22>I<0003F800001FFE00003C1F0000F0078001E003C003C001E0078001E00F8000F00F00 00F01F0000F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F8 0001F0F80001F0F80001F0F80003E0780003E0780007C07C0007C07C000F803C000F003E001E00 1E003C000F00780007C1F00003FFC00000FE00001D217B9F23>I<07FFFF0007FFFFC0003C03E0 003C01F0007800F0007800F8007800F8007800F8007800F8007800F800F001F000F001F000F001 E000F003C000F00F8000FFFE0001FFF80001E0000001E0000001E0000001E0000001E0000003C0 000003C0000003C0000003C0000003C0000003C0000007C000007FFC0000FFFC00001D1F7E9E1F >I<07FFFC0007FFFF00003C07C0003C03E0007801E0007801F0007801F0007801F0007801F000 7801E000F003E000F003C000F0078000F01F0000FFFC0000FFF00001E0380001E03C0001E01C00 01E01C0001E01E0001E01E0003C03E0003C03E0003C03E0003C03E0003C03E0603C03E0607C03E 0C7FFC1F18FFFC0FF8000003E01F207E9E21>82 D<003F04007FCC01E0FC03807C03003C070038 0600180E00180E00180E00180E00000F00000F00000FE00007FE0003FF8001FFC0007FE00007E0 0001E00000E00000F00000F06000E06000E06000E06000E07001C0700180F80380FE0F00C7FC00 81F80016217D9F19>I<1FFFFFF81FFFFFF81E03C0F83803C03838078038300780387007801860 0780186007803860078030C00F0030000F0000000F0000000F0000000F0000000F0000001E0000 001E0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000003C00 00003C0000007C00001FFFF0003FFFF0001D1F7B9E21>III E /Fq 15 116 df<60F0F0703030306060C0 40040B7D830B>44 D<03000700FF00FF0007000700070007000700070007000700070007000700 07000700070007000700070007000700FFF0FFF00C197D9813>49 D<0F801FE038F07070787878 38787830780070007000E00FC00F8000E000700038003C003C703CF83CF83CF038607870F03FE0 0F800E1A7E9813>51 D<0070007000F000F001F003F00370067006700C701C7018703070307060 70E070FFFFFFFF0070007000700070007007FF07FF10197F9813>I<07801FE0387030306038E0 18E018E01CE01CE01CE01CE01C603C703C307C3FDC1F9C00180018003830307870786071C03F80 1F000E1A7E9813>57 D<0FFF0FFF00780078007800780078007800780078007800780078007800 7800780078007800787078F878F878F878F0F070E03FC00F80101B7F9914>74 D86 D<3F80FFC0F0E0F070607000700FF03FF07870E070E070E073E07370F37FFE3E3C10107E8F13> 97 D<07C01FF038387018701CFFFCFFFCE000E000E000E0007000300C3C180FF007E00E107F8F 11>101 D<00F801FC03BC073C0E180E000E000E000E000E00FFC0FFC00E000E000E000E000E00 0E000E000E000E000E000E000E007FE07FE00E1A80990C>I<18003C003C001800000000000000 000000000000FC00FC001C001C001C001C001C001C001C001C001C001C001C001C00FF80FF8009 1A80990A>105 D110 D<07E01FF8381C700E6006E007E0 07E007E007E007E007700E700E3C3C1FF807E010107F8F13>I114 D<1F207FE070E0E060E060F0 007F003FC01FE001F0C070C070E070F0E0FFC08F800C107F8F0F>I E /Fr 10 58 df<1F003F8060C04040C060C060C060C060C060C060C060C06060C060C03F801F000B10 7F8F0F>48 D<18007800F80098001800180018001800180018001800180018001800FF80FF8009 107E8F0F>I<3F007F80F1C0F0E06060006000E000C00180030006001C0038606060FFC0FFC00B 107F8F0F>I<1F003F8071C071C031C001800F800F0001C000E060E0F0E0F0E0F1C07F801F000B 107F8F0F>I<070007000F001F001B003B0033006300E300FFE0FFE00300030003001FE01FE00B 107F8F0F>I<60807F807F007C00600060006F007F8070C060E000E060E0E0E0E1C07F803F000B 107F8F0F>I<07801FC039C061C06000C000DF80FFC0E060C060C060C060606060C03F801F000B 107F8F0F>I<60007FE07FE0C0C0C1800180030006000E000C000C001C001C001C001C001C0008 000B117E900F>I<1F003F8061C060C060C079C03F801F803FC063E0C0E0C060C06060C03F801F 000B107F8F0F>I<1F003F8060C0C0C0C060C060C060C0E07FE03F60006000C070C071807F003E 000B107F8F0F>I E /Fs 49 122 df<1C3E7E7E3E060C0C18183060C080070E769F0E>39 D<7FF0FFE0FFE00C037D8A10>45 D<70F8F8F0E005057B840E>I<0000006000000060000000E0 000000C0000001C00000038000000300000007000000060000000E0000001C0000001800000038 000000300000007000000060000000E0000001C000000180000003800000030000000700000006 0000000E0000001C0000001800000038000000300000007000000060000000E0000001C0000001 800000038000000300000007000000060000000E0000001C000000180000003800000030000000 70000000E0000000C00000001B2D80A117>I<000F80003FE000F0F001C0700380700380380700 780F00780F00780E00781E00781E00703C00F03C00F03C00F03C00F07801E07801E07801E07801 C07003C0F003C0F00380F00780F00700700F00700E00701C003878001FF0000FC000151F7C9D17 >I<000200060006000E003C00FC07DC071C0038003800380038007000700070007000E000E000 E000E001C001C001C001C00380038003800380FFF8FFF80F1E7B9D17>I<000F80003FC00070E0 00C0700180780330380638380618780C30780C30780C60780CE0780FC0F00701E00001C0000380 000700001E0000380000E00001C0000700000E00300C00301800603000E07F81C07FFFC061FF80 C07F00C03C00151F7D9D17>I<001F80007FE000E0E00180700300380660380670380630380660 7007C0700380600000E00001C000078000FE0000FE000007000007000007800007800007803007 80780780780780F00F00C00F00601E00603C007078003FE0001F8000151F7C9D17>I<001F0000 7F8000E1C001C1C00380E00700E00F00E00F01E01E01E01E01E01E01E01E01C01C03C03C03C03C 07C01C07C01C0F801C1F800F778007E700000F00000E00001E00001C00603C00F03800F07000E0 E000C1C0007F80003E0000131F7B9D17>57 D<070F1F1F0E0000000000000000000070F8F8F0E0 08147B930E>I<000007000000070000000F0000000F0000001F0000003F0000003F0000006F00 00006F000000CF000000CF0000018F0000038F0000030F0000060F0000060F00000C0F80000C07 800018078000180780003FFF80007FFF800060078000C0078000C0078001800780018007800300 0780070007800F0007807FC07FF8FFC07FF81D207E9F22>65 D<0000FC020007FF06001F818C00 3C00DC0070007C01E0007C03C0003803800038070000380F0000381E0000301E0000303C000030 3C00000078000000780000007800000078000000F0000000F0000000F0000000F00000C0F00001 807000018070000300780003007800060038000C001C0018000E0030000780E00003FF800000FE 00001F217A9F21>67 D<01FFFFFE01FFFFFC001E003C001E001C001E001C003C000C003C000C00 3C000C003C001800780C1800780C1800780C0000781C0000F0380000FFF80000FFF80000F03800 01E0300001E0300001E0303001E0306003C0006003C0006003C000C003C001C007800180078003 800780070007801F00FFFFFF00FFFFFE001F1F7D9E1F>69 D<01FFFFFC01FFFFF8001E0078001E 0038001E0038003C0018003C0018003C0018003C003000780C3000780C3000780C0000781C0000 F0380000FFF80000FFF80000F0380001E0300001E0300001E0300001E0300003C0000003C00000 03C0000003C0000007800000078000000780000007800000FFFC0000FFF800001E1F7D9E1E>I< 0001FC040007FE0C001F0398003C01B800F000F801E000F803C0007007800070070000700F0000 701E0000601E0000603C0000603C00000078000000780000007800000078000000F0000000F000 FFF0F000FFF0F0000780F0000F00F0000F0070000F0078000F0078001E0038001E003C003E001E 007E000F81EC0003FF840000FE00001E217A9F23>I<01FFF3FFE001FFF3FFE0001E003C00001E 003C00001E003C00003C007800003C007800003C007800003C007800007800F000007800F00000 7800F000007800F00000F001E00000FFFFE00000FFFFE00000F001E00001E003C00001E003C000 01E003C00001E003C00003C007800003C007800003C007800003C007800007800F000007800F00 0007800F000007800F0000FFF9FFF000FFF1FFE000231F7D9E22>I<01FFF001FFF0001E00001E 00001E00003C00003C00003C00003C0000780000780000780000780000F00000F00000F00000F0 0001E00001E00001E00001E00003C00003C00003C00003C000078000078000078000078000FFF8 00FFF800141F7D9E12>I<01FF00FFE001FF00FFE0001F001E00001F800C00001F800C00003780 18000033C018000033C018000033C018000063C030000061E030000061E030000061E0300000C0 F0600000C0F0600000C0F0600000C0786000018078C000018078C00001803CC00001803CC00003 003D800003001F800003001F800003001F800006000F000006000F000006000F00000E000F0000 FFE0060000FFC0060000231F7D9E22>78 D<0001FC000007FF00001F07C0003801E000F000E001 E0007003C0007007800078070000380F0000381E0000381E0000383C0000383C00003878000078 780000787800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F00003C0700007 807800070078000F0038001E003C003C001E00F0000F83E00007FF800001FC00001D217A9F23> I<01FFFF8001FFFFE0001E00F0001E0078001E0038003C003C003C003C003C003C003C003C0078 00780078007800780070007800E000F001E000F0078000FFFF0000FFF80001E0000001E0000001 E0000001E0000003C0000003C0000003C0000003C0000007800000078000000780000007800000 FFF80000FFF000001E1F7D9E1F>I<01FFFF0001FFFFC0001E01E0001E00F0001E0078003C0078 003C0078003C0078003C0078007800F0007800F0007801E0007803C000F00F0000FFFE0000FFF8 0000F03C0001E01E0001E00E0001E00F0001E00F0003C01E0003C01E0003C01E0003C01E000780 3C0007803C1807803C1807803C30FFF81E30FFF00FE0000007C01D207D9E21>82 D<0007E040001FF8C0003C1D8000700F8000E0078001C007800180030003800300038003000380 0300038000000380000003C0000003F8000001FF800001FFE000007FF000001FF0000001F80000 00780000003800000038000000380030003800300038003000300070007000700060007800E000 7801C000EE078000C7FE000081F800001A217D9F1A>I<0FFFFFF01FFFFFE01E0781E0180780E0 38078060300F0060300F0060600F0060600F00C0C01E00C0C01E00C0001E0000001E0000003C00 00003C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F0 000000F0000001E0000001E0000001E0000003E00000FFFF0000FFFF00001C1F789E21>I<7FFC 3FF87FFC3FF80780078007800300078003000F0006000F0006000F0006000F0006001E000C001E 000C001E000C001E000C003C0018003C0018003C0018003C001800780030007800300078003000 78003000F0006000F0006000F0006000F000C000F00080007001800070030000380600003C1C00 001FF8000007E000001D20779E22>III<00 F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E00F00E00F0 0E30F01C60F03C60707C6078FCC03FCFC00F078014147C9317>97 D<07803F803F000700070007 000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F700F700F700FE01EE01EE01E E03CE03CE038607071E03FC01F0010207B9F15>I<007E0001FF000383800F07801E07801C0700 3C0200780000780000780000F00000F00000F00000F00000F00000700200700700381E001FF800 07E00011147C9315>I<0000780003F80003F00000700000700000700000E00000E00000E00000 E00001C00001C000F1C003FDC0078F800E07801C07803C07803C0700780700780700780700F00E 00F00E00F00E00F00E30F01C60F03C60707C6078FCC03FCFC00F078015207C9F17>I<007C0001 FF000783000F01801E01803C01803C0300780E007FFC007FE000F00000F00000F00000F0000070 00007002007807003C1E001FF80007E00011147C9315>I<0000F80001FC0003BC00033C000718 000700000700000E00000E00000E00000E00000E0001FFE001FFE0001C00001C00001C00003800 00380000380000380000380000700000700000700000700000700000700000E00000E00000E000 00E00001C00001C00001C0000180003380007B8000F300007E00003C00001629829F0E>I<003C 6000FF7001E3E00381E00701E00F01E00F01C01E01C01E01C01E01C03C03803C03803C03803C03 803C07003C0F001C1F001E3F000FFE0003CE00000E00000E00001C00001C00301C00783800F0F0 007FE0003F8000141D7E9315>I<01E0000FE0000FC00001C00001C00001C00003800003800003 8000038000070000070000073E00077F000EC3800F81C00F01C00E01C01E03801C03801C03801C 0380380700380700380700380E18700E30700E30701C60700C60E00FC060078015207D9F17>I< 006000F000F000E000000000000000000000000000000F001F80318031C063806380C380070007 0007000E000E000E001C301C601C6038C018C01F800F000C1F7D9E0E>I<01E0000FE0000FC000 01C00001C00001C0000380000380000380000380000700000700000703C00707E00E0C600E10E0 0E21E00E61E01CC1C01F80001F00001FC00039E0003870003870003838607070C07070C07070C0 703180E03F00601E0013207D9F15>107 D<03C01FC01F8003800380038007000700070007000E 000E000E000E001C001C001C001C0038003800380038007000700070007180E300E300E300E600 7E003C000A207C9F0C>I<1E07C0F8003F1FE1FC0033B8730E0063E076070063C03C0700638038 0700C780780E000700700E000700700E000700700E000E00E01C000E00E01C000E00E01C000E00 E038601C01C038C01C01C038C01C01C071801C01C031803803803F001801801E0023147D9325> I<1E07C03F1FE033B87063E07063C038638038C780700700700700700700700E00E00E00E00E00 E00E01C31C01C61C01C61C038C1C018C3801F81800F018147D931A>I<007C0001FF000383800F 01C01E01C01C01E03C01E07801E07801E07801E0F003C0F003C0F003C0F00780F00700700F0070 1E003838001FF00007C00013147C9317>I<03C1E007E7F8067E3C0C7C1C0C781E0C701E18E01E 00E01E00E01E00E01E01C03C01C03C01C03C01C07803C07803C07003C0E003E3C0077F80071E00 0700000700000E00000E00000E00000E00001C0000FFC000FFC000171D809317>I<1E0F003F3F 8033F1C063C1C063C3C06383C0C783800700000700000700000E00000E00000E00000E00001C00 001C00001C00001C000038000018000012147D9313>114 D<00FC03FE07070E0F0E0F0E0E1E00 0F800FF007F803FC003E001E701EF01CF01CE03860703FE01F8010147D9313>I<018001C00380 03800380038007000700FFF0FFF00E000E000E000E001C001C001C001C00380038003800383070 60706070C071803F001E000C1C7C9B0F>I<0F00601F80703180E031C0E06380E06380E0C381C0 0701C00701C00701C00E03800E03800E03800E038C0E07180E07180E0F180E1F3007F3F003E1E0 16147D9318>I<0F01C01F83C03183E031C1E06380E06380E0C380C00700C00700C00700C00E01 800E01800E01800E03000E03000E06000E06000F0C0007F80001E00013147D9315>I<0F0060E0 1F8071E03180E1F031C0E0F06380E0706380E070C381C0600701C0600701C0600701C0600E0380 C00E0380C00E0380C00E0381800E0381800E0781000E078300070F860007F9FC0001F0F8001C14 7D931E>I<0787800FCFC018F8E03070E06071E06071E0C0E1C000E00000E00000E00001C00001 C00001C00071C060F380C0F380C0E38180C7C3007CFE00387C0013147D9315>I<0F00601F8070 3180E031C0E06380E06380E0C381C00701C00701C00701C00E03800E03800E03800E03800E0700 0E07000E0F000E1F0007FE0003EE00000E00000E00001C00781C0078380070700060E0003FC000 1F0000141D7D9316>I E /Ft 81 124 df<000FC3E0007FEFF000E07C7801C0F8780380F83007 007000070070000700700007007000070070000700700007007000FFFFFF80FFFFFF8007007000 070070000700700007007000070070000700700007007000070070000700700007007000070070 0007007000070070000700700007007000070070007FE3FF007FE3FF001D20809F1B>11 D<001F80007FC001E0E00381E00381E00701E0070000070000070000070000070000070000FFFF E0FFFFE00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700 E00700E00700E00700E00700E07FC3FE7FC3FE1720809F19>I<001FE0007FE001E1E00381E003 80E00700E00700E00700E00700E00700E00700E00700E0FFFFE0FFFFE00700E00700E00700E007 00E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E07F E7FE7FE7FE1720809F19>I<000F80F800007FE7FE0000E06E060001C0FC0F000380F80F000700 F00F00070070000007007000000700700000070070000007007000000700700000FFFFFFFF00FF FFFFFF000700700700070070070007007007000700700700070070070007007007000700700700 070070070007007007000700700700070070070007007007000700700700070070070007007007 0007007007007FE3FE3FF07FE3FE3FF02420809F26>I<7038F87CFC7EFC7E7C3E0C060C060C06 180C180C381C3018603040200F0E7E9F17>34 D<70F8FCFC7C0C0C0C181838306040060E7C9F0D >39 D<006000C001800300070006000C001C00180038003800300070007000700060006000E000 E000E000E000E000E000E000E000E000E000E000E0006000600070007000700030003800380018 001C000C00060007000300018000C000600B2E7DA112>II<000600000006000000060000000600000006000000060000000600000006000000 060000000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF000060000 000600000006000000060000000600000006000000060000000600000006000000060000000600 00000600000006000000060000000600001C207D9A23>43 D<70F8FCFC7C0C0C0C181838306040 060E7C840D>II<70F8F8F87005057C840D>I<0003000300070006 0006000E000C000C001C0018001800380030003000700060006000E000C000C001C00180018001 800380030003000700060006000E000C000C001C0018001800380030003000700060006000E000 C000C000102D7DA117>I<03F0000FFC001E1E001C0E00380700780780700380700380700380F0 03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C070 03807003807003807807803807001C0E001E1E000FFC0003F000121F7E9D17>I<018003801F80 FF80E3800380038003800380038003800380038003800380038003800380038003800380038003 8003800380038003800380FFFEFFFE0F1E7C9D17>I<03F0000FFC00183E00300F00600F80C007 80F007C0F807C0F803C0F803C02003C00007C0000780000780000F00000E00001C000038000070 0000E00001C0000380000300000600C00C00C01800C03001807FFF80FFFF80FFFF80121E7E9D17 >I<03F0000FFC001C1E00300F00780F80780F80780780380F80000F80000F00000F00001E0000 3C0003F80003F000001C00000E00000F000007800007800007C02007C0F807C0F807C0F807C0F0 0780600F80700F003C1E001FFC0003F000121F7E9D17>I<000E00000E00001E00003E00003E00 006E0000EE0000CE00018E00038E00030E00060E000E0E000C0E00180E00380E00300E00600E00 E00E00FFFFF0FFFFF0000E00000E00000E00000E00000E00000E00000E0000FFE000FFE0141E7F 9D17>I<3803003FFF003FFE003FF8003FE00030000030000030000030000030000030000031F0 0037FC003E0E003C07003807803003800003800003C00003C00003C06003C0F003C0F003C0F003 80C00780600700700E003C1E000FF80007E000121F7E9D17>I<007C0001FE000783000E03800C 07801C0780380300380000780000700000700000F3F800F7FE00FC0E00F80700F80380F80380F0 03C0F003C0F003C0F003C0F003C07003C07003C07003803803803807001C07000E1E0007FC0001 F000121F7E9D17>I<6000007FFFC07FFFC07FFF80600180C00300C00600C00600000C00001800 00180000300000600000600000E00000C00001C00001C00001C000038000038000038000038000 078000078000078000078000078000078000078000030000121F7D9D17>I<03F0000FFC001E1E 003807003007007003807003807003807003807807003E07003F0E001FDC000FF80003F80007FC 001E7E00383F00700F80700780E003C0E001C0E001C0E001C0E001C0F001807003803807001E0E 000FFC0003F000121F7E9D17>I<03F00007F8001E1C00380E00380700700700700380F00380F0 0380F003C0F003C0F003C0F003C0F003C07007C07007C0380FC01C0FC01FFBC007F3C000038000 0380000780000700300700780E00780E00701C003078001FF0000FC000121F7E9D17>I<70F8F8 F8700000000000000000000070F8F8F87005147C930D>I<70F8F8F87000000000000000000000 70F8F8F878181818183030606040051D7C930D>I<7FFFFFE0FFFFFFF000000000000000000000 00000000000000000000000000000000000000000000FFFFFFF07FFFFFE01C0C7D9023>61 D<0003800000038000000380000007C0000007C0000007C000000DE000000DE000000DE0000018 F0000018F0000018F00000307800003078000030780000603C0000603C0000603C0000E01E0000 C01E0000FFFE0001FFFF0001800F0001800F0003800F800300078003000780070007C0070003C0 0F8003C0FFE03FFEFFE03FFE1F207F9F22>65 DI<001FC04000 7FF0C001F839C003C00DC0078007C00F0003C01E0003C03E0001C03C0001C07C0001C07C0000C0 780000C0F80000C0F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000 00780000C07C0000C07C0000C03C0000C03E0001801E0001800F0003000780030003C00E0001F8 1C00007FF000001FC0001A217D9F21>IIII<000FC020007FF86001F81CE003E006E0078003 E00F0001E01E0001E01E0000E03C0000E07C0000E07C00006078000060F8000060F8000000F800 0000F8000000F8000000F8000000F8000000F8007FFCF8007FFC780001E07C0001E07C0001E03C 0001E01E0001E01E0001E00F0001E0078003E003E003E001F80E60007FFC20000FE0001E217D9F 24>III<0FFFC00FFFC0003C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0020 3C00F83C00F83C00F83C00F0780060780070F0003FE0000F800012207E9E17>IIIII<001F800000FFF00001E0780007C03E000F801F000F000F001E000780 3C0003C03C0003C07C0003E07C0003E0780001E0F80001F0F80001F0F80001F0F80001F0F80001 F0F80001F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03E0007C01E00 07800F000F000F801F0007C03E0001F0F80000FFF000001F80001C217D9F23>II<001F800000FFF00001E0780007C03E000F801F000F000F001E0007803E0007 C03C0003C07C0003E07C0003E0780001E0F80001F0F80001F0F80001F0F80001F0F80001F0F800 01F0F80001F0F80001F0F80001F0780001E07C0003E07C0003E03C0003C03E0F07C01E1F87800F 38CF000FB0DF0007F0FE0001F0780000FFF010001FF01000007010000078300000387000003FF0 00003FE000001FE000000FC0000007801C297D9F23>II<07E0800FF9801C1F80300F80700780600380E00380E00180E00180E00180F00000F0000078 00007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001C0C001C0C001C0C0 01C0C001C0E00180F00380F80700FE0E00CFFC0081F80012217D9F19>I<7FFFFFE07FFFFFE078 0F01E0700F00E0600F0060600F0060E00F0070C00F0030C00F0030C00F0030C00F0030000F0000 000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F00 00000F0000000F0000000F0000000F0000000F0000000F0000000F000007FFFE0007FFFE001C1F 7E9E21>IIII89 D<7FFFF87FFFF87E00F07801F07001E06003E0 E003C0E00780C00780C00F00C01F00001E00003C00003C0000780000F80000F00001E00001E00C 03C00C07C00C07800C0F001C0F001C1E00183E00183C00387C00787801F8FFFFF8FFFFF8161F7D 9E1C>II<0804180C3018703860306030C060C060C060F87CFC7E FC7E7C3E381C0F0E7B9F17>II<1FE0003FF000783800781C0030 0E00000E00000E00000E0003FE001FFE003E0E00780E00F80E00F00E30F00E30F00E30F01E3078 3F703FEFE00F878014147E9317>97 D<0E0000FE0000FE00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E3F000EFF800FC3C00F00E00E00F00E00700E00780E00780E00 780E00780E00780E00780E00780E00700E00F00F00F00F01E00FC3C00CFF800C3E0015207F9F19 >I<03F80FFC1E1E3C1E380C78007000F000F000F000F000F000F0007000780038033C031F0E0F FC03F010147E9314>I<000380003F80003F800003800003800003800003800003800003800003 8000038000038003E3800FFB801E0F803C0780780380780380700380F00380F00380F00380F003 80F00380F003807003807003807807803807801E1F800FFBF803E3F815207E9F19>I<03F0000F FC001E1E003C0F00380700780700700380F00380FFFF80FFFF80F00000F00000F0000070000078 00003801801C03800F070007FE0001F80011147F9314>I<003C00FE01CF038F03060700070007 000700070007000700FFF0FFF00700070007000700070007000700070007000700070007000700 0700070007007FF07FF01020809F0E>I<0001E003E3F00FFF701C1C703C1E00380E00780F0078 0F00780F00780F00380E003C1E001C1C003FF80033E0003000003000003800003FFE001FFF803F FFC07003E07000F0E00070E00070E00070F000F07801E03E07C00FFF0003FC00141F7F9417>I< 0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E3E00 0EFF000FC3800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C00E01C0FFE7FCFFE7FC16207F9F19>I<1C001E003E001E001C0000000000 00000000000000000E007E007E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E00FFC0FFC00A1F809E0C>I<00E001F001F001F000E000000000000000000000000000 7007F007F000F00070007000700070007000700070007000700070007000700070007000700070 00700070007000706070F0E0F1C07F803E000C28829E0E>I<0E0000FE0000FE00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E1FF00E1FF00E0F800E0E000E0C000E18 000E30000E70000EF0000FF8000F38000E1C000E1E000E0E000E07000E07800E03800E03C0FFCF F8FFCFF815207F9F18>I<0E00FE00FE000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00B2080 9F0C>I<0E3F03F000FEFF8FF800FFC1DC1C000F80F80E000F00F00E000E00E00E000E00E00E00 0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E 000E00E00E000E00E00E000E00E00E00FFE7FE7FE0FFE7FE7FE023147F9326>I<0E3E00FEFF00 FFC3800F01C00F01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0 0E01C00E01C00E01C0FFE7FCFFE7FC16147F9319>I<01F80007FE001E07803C03C03801C07000 E07000E0F000F0F000F0F000F0F000F0F000F0F000F07000E07801E03801C03C03C01E078007FE 0001F80014147F9317>I<0E3F00FEFF80FFC3C00F01E00E00F00E00F00E00780E00780E00780E 00780E00780E00780E00780E00700E00F00F00F00F01E00FC3C00EFF800E3E000E00000E00000E 00000E00000E00000E00000E0000FFE000FFE000151D7F9319>I<03E1800FF9801E1F803C0780 780780780380700380F00380F00380F00380F00380F00380F003807003807803807807803C0F80 1E1F800FF38003E380000380000380000380000380000380000380000380003FF8003FF8151D7E 9318>I<0E78FEFCFF9E0F1E0F0C0F000E000E000E000E000E000E000E000E000E000E000E000E 00FFE0FFE00F147F9312>I<1F903FF07070E030E030E030F00078007F803FE00FF000F0C078C0 38C038E038E038F070DFE08F800D147E9312>I<06000600060006000E000E001E003E00FFF8FF F80E000E000E000E000E000E000E000E000E000E000E180E180E180E180E18073007E003C00D1C 7F9B12>I<0E01C0FE1FC0FE1FC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E 01C00E01C00E01C00E01C00E03C00E03C0070FC007FDFC01F1FC16147F9319>III<7FC7FC7FC7FC0703 E007038003830001C70000EE0000EC00007800003800003C00007C0000EE0001C7000187000303 800701C00F01E0FF87FEFF87FE1714809318>II<3FFF3F FF380E301C703C6038607060F001E001C0038007830F030E031C073C063806701EFFFEFFFE1014 7F9314>II E /Fu 34 119 df46 D<00000C00001C00001C00003C0000380000380000780000700000700000F00000E00000E00001 E00001C00001C00003C0000380000380000780000700000700000F00000E00000E00001E00001C 00001C00003C0000380000780000700000700000F00000E00000E00001E00001C00001C00003C0 000380000380000780000700000700000F00000E00000E00001E00001C00001C00003C00003800 00380000780000700000700000F00000E00000E00000E00000163C7DAC1D>I<00180000380000 F80007F800FFF800FFF800FFF800F8F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000 F80000F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFFF07FFFF014287CA7 1D>49 D<00FE0003FF800FFFC01FFFE01F07F03C01F87800FC78007C70007CF0003EF0003E6000 3E20003E00003E00003E00003E00007C00007C0000780000F80001F00001E00003E00007C0000F 80001F00003E00007C0000F80001F00003E00003C0000780000F00001E00003C00007FFFFE7FFF FE7FFFFE7FFFFE17287EA71D>I<007E0003FF8007FFE00FFFF01F81F83E00F87C007C78007C30 007C10007C00007C00007C0000780000F80000F80001F00007E000FFC000FF8000FFC000FFF000 01F80000FC00007C00003E00003E00001F00001F00001F00001F00001F80001FC0003EE0003EF0 007E7C00FC3F01F81FFFF00FFFE003FFC000FE0018297EA71D>I<0007E0000FE0000BE0001BE0 001BE0003BE0003BE00073E00073E000F3E000F3E001E3E001E3E003E3E003C3E007C3E00F83E0 0F83E01F03E01F03E03E03E03E03E07C03E07C03E0F803E0FFFFFFFFFFFFFFFFFFFFFFFF0003E0 0003E00003E00003E00003E00003E00003E00003E00003E00003E018277EA61D>I<3FFFF83FFF F83FFFF83FFFF83E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E3E 003EFF803FFFE03FFFF03FC1F03F80F83F007C3E007C00007C00003E00003E00003E00003E0000 3E00003E00003E20003E60007C70007CF800F87C01F87F07F03FFFE00FFFC007FF0001FC001728 7EA61D>I<000FE0003FF800FFF801FFF803F81807E0000F80000F80001F00003E00003E00003C 00007C00007C02007C3FC0F8FFE0F9FFF0FBFFF8FF80FCFE007CFE003EFC003EFC003FFC001FF8 001FF8001FF8001FF8001F78001F7C001F7C001F7C001E3C003E3E003E1E007C1F00FC0FC1F807 FFF003FFE001FF80007E0018297EA71D>II<007E0003FFC007FFE00FFFF01F81 F83F00FC3E007C7E007E7C003E7C003E7C003E7C003E7C003E3E007C3E007C1F00F80FC3F007FF E003FFC003FFC00FFFF01F81F83F00FC3E007C7C003E7C003EF8001FF8001FF8001FF8001FF800 1FF8001F7C003E7C003E7E007E3F00FC1F81F80FFFF007FFE003FFC0007E0018297EA71D>I<00 03FE00000FFFC0003FFFF000FFFFF001FE03F003F8006007E0002007C000000F8000001F800000 1F0000003E0000003E0000007C0000007C0000007C0000007C000000F8000000F8000000F80000 00F8000000F8000000F8000000F8000000F8000000F8000000F80000007C0000007C0000007C00 00007C0000003E0000003E0000001F0000001F8000000F80000007C0000007E0000803F8003801 FE01F800FFFFF8003FFFF0000FFFC00003FE001D2C7CAA25>67 D69 DI73 D<0003FC0000000FFF0000003FFFC00000FFFFF00001FE07F80003F801FC0007E0007E000FC000 3F000F80001F001F00000F801F00000F803E000007C03E000007C07C000003E07C000003E07C00 0003E078000001E0F8000001F0F8000001F0F8000001F0F8000001F0F8000001F0F8000001F0F8 000001F0F8000001F0F8000001F0F8000001F07C000003E07C000003E07C000003E07E000007E0 3E000007C03E000007C01F00000F801F80001F800F80001F000FC0003F0007E0007E0003F801FC 0001FE07F80000FFFFF000003FFFC000001FFF80000003FC0000242C7DAA2B>79 D86 D<01FC000FFF003FFF803FFFC03C07E0 3003E00001F00001F00001F00001F00001F00003F001FFF007FFF01FFFF03FC1F07E01F0FC01F0 F801F0F801F0F801F0FC03F07E0FF07FFFF03FFFF01FF9F00FE1F0141B7E9A1C>97 D<007F8001FFE007FFF80FFFF81FC0783F00103E00007C00007C00007C0000F80000F80000F800 00F80000F80000F80000F800007C00007C00007C00003E00003F00181FC0780FFFF807FFF801FF E0007F80151B7E9A1A>99 D<00007C00007C00007C00007C00007C00007C00007C00007C00007C 00007C00007C00007C00007C00007C00007C00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E007C 7C007C7C007CFC007CF8007CF8007CF8007CF8007CF8007CF8007CF8007CF8007C7C007C7C007C 7E00FC3F00FC1F83FC1FFFFC0FFF7C03FE7C01F87C162A7EA91E>I<00FC0003FF0007FF800FFF C01F87E03F01E03E00F07C00F07C0070F80078FFFFF8FFFFF8FFFFF8FFFFF8F80000F80000F800 007800007C00007C00003E00003F00181FC0F80FFFF807FFF801FFE0007F00151B7E9A1A>I<00 1FC0007FC000FFC001FFC003F00003E00007C00007C00007C00007C00007C00007C00007C00007 C00007C000FFFE00FFFE00FFFE00FFFE0007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007 C00007C00007C000122A7FA912>I<00F80F0003FE7F800FFFFF801FFFFF801F07C0003E03E000 3E03E0007C01F0007C01F0007C01F0007C01F0007C01F0003E03E0003E03E0001F07C0001FFFC0 003FFF80003BFE000038F8000078000000780000007C0000003FFF80003FFFF0001FFFF8001FFF FC003FFFFE007C00FE007C003F00F8001F00F8001F00F8001F00F8001F00FC003F007E007E003F 81FC001FFFF8000FFFF00007FFE00000FF000019287E9A1D>III108 DII<007E0001FF8007FFE00FFFF01F81F83F00FC 3E007C7C003E7C003E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001F7C003E7C003E 7C003E3E007C3F00FC1FC3F80FFFF007FFE003FFC0007E00181B7E9A1D>II114 D<07F8001FFF003FFF807FFF807C0780F8 0100F80000F80000F80000FC00007F00007FF0003FFC001FFE0007FF0000FF80001FC0000FC000 07C00007C04007C0E007C0FC1F80FFFF807FFF001FFE0007F800121B7E9A16>I<07C00007C000 07C00007C00007C00007C00007C000FFFF80FFFF80FFFF80FFFF8007C00007C00007C00007C000 07C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C000 07C08007E38003FFC003FFC001FF0000FC0012227FA115>III E /Fv 13 122 df66 D73 D<00000003FFF0000000000000007FFFFF80000000000003FFFFFF F000000000000FFFFFFFFC00000000003FFF003FFF0000000000FFF80007FFC000000003FFE000 01FFF000000007FF8000007FF80000000FFF0000003FFC0000001FFE0000001FFE0000003FFC00 00000FFF0000007FF800000007FF800000FFF000000003FFC00001FFF000000003FFE00003FFE0 00000001FFF00003FFC000000000FFF00007FFC000000000FFF80007FFC000000000FFF8000FFF 80000000007FFC000FFF80000000007FFC001FFF80000000007FFE001FFF00000000003FFE003F FF00000000003FFF003FFF00000000003FFF003FFF00000000003FFF007FFF00000000003FFF80 7FFE00000000001FFF807FFE00000000001FFF807FFE00000000001FFF807FFE00000000001FFF 80FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001F FFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE0000000000 1FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000 001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC07FFE00000000001FFF807FFF000000 00003FFF807FFF00000000003FFF807FFF00000000003FFF803FFF00000000003FFF003FFF0000 0000003FFF003FFF80000000007FFF001FFF80000000007FFE001FFF80000000007FFE001FFF80 000000007FFE000FFFC000000000FFFC000FFFC000000000FFFC0007FFE000000001FFF80003FF E000000001FFF00003FFF000000003FFF00001FFF000000003FFE00000FFF800000007FFC00000 7FFC0000000FFF8000007FFE0000001FFF8000003FFF0000003FFF0000000FFFC00000FFFC0000 0007FFE00001FFF800000003FFF80007FFF000000000FFFF003FFFC0000000007FFFFFFFFF8000 0000000FFFFFFFFC000000000003FFFFFFF00000000000007FFFFF8000000000000003FFF00000 00004A497AC757>79 D<0003FFF0000000003FFFFF00000000FFFFFFE0000001FFFFFFF8000003 FF003FFC000007FF800FFE000007FF8007FF000007FF8003FF800007FF8003FFC00007FF8001FF C00007FF8001FFC00003FF0000FFE00001FE0000FFE00000FC0000FFE00000000000FFE0000000 0000FFE00000000000FFE00000000000FFE00000000000FFE000000000FFFFE00000007FFFFFE0 000007FFFFFFE000003FFFFCFFE00000FFFE00FFE00003FFF000FFE00007FFC000FFE0000FFF00 00FFE0001FFE0000FFE0003FFC0000FFE0007FFC0000FFE0007FF80000FFE000FFF80000FFE000 FFF00000FFE000FFF00000FFE000FFF00000FFE000FFF00001FFE000FFF00001FFE000FFF80003 FFE0007FFC0007FFE0007FFC000FFFE0003FFE001F7FF8001FFF80FE7FFFF007FFFFFC3FFFF003 FFFFF01FFFF0007FFFC00FFFF00007FE0003FFF0342E7DAD38>97 D<007FC000000000FFFFC000 000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC00000000003FFC00000000001 FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000 000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0000000 0001FFC00000000001FFC00000000001FFC00FFE000001FFC0FFFFC00001FFC3FFFFF00001FFCF FFFFFC0001FFFFE01FFF0001FFFF0007FF8001FFFC0001FFC001FFF80000FFE001FFF00000FFF0 01FFE000007FF001FFC000007FF801FFC000003FFC01FFC000003FFC01FFC000003FFC01FFC000 001FFE01FFC000001FFE01FFC000001FFE01FFC000001FFF01FFC000001FFF01FFC000001FFF01 FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC00000 1FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFE01FFC000001FFE01FF C000003FFE01FFC000003FFC01FFC000003FFC01FFC000003FF801FFC000007FF801FFE000007F F001FFF00000FFE001FFF80001FFE001FFFC0003FFC001FF7F0007FF0001FE3FC03FFE0001FC0F FFFFF80001FC03FFFFF00001F800FFFF800001F0001FF8000038487CC741>I<00000000003F00 0000FFF000FFC00007FFFE03FFE0003FFFFFCFFFF0007FFFFFFFCFF001FFE07FFE0FF003FF801F FC1FF007FF000FFE0FF00FFE0007FF0FF00FFE0007FF07E01FFC0003FF81801FFC0003FF80003F FC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FF C0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0001FFC0003FF80001FFC0003FF80000FFE 0007FF00000FFE0007FF000007FF000FFE000003FF801FFC000001FFE07FF8000003FFFFFFE000 0003FFFFFFC000000787FFFE0000000780FFF0000000078000000000000F8000000000000F8000 000000000FC000000000000FC000000000000FE000000000000FF8000000000007FFFFFFF00000 07FFFFFFFF000007FFFFFFFFE00003FFFFFFFFF80003FFFFFFFFFC0001FFFFFFFFFE0000FFFFFF FFFF0003FFFFFFFFFF800FFFFFFFFFFF801FF000001FFFC03FC0000001FFC07F800000007FE07F 800000003FE0FF000000003FE0FF000000001FE0FF000000001FE0FF000000001FE0FF00000000 1FE0FF800000003FE07F800000003FC07FC00000007FC03FE0000000FF801FF8000003FF000FFE 00000FFE0007FFE000FFFC0001FFFFFFFFF000003FFFFFFF8000000FFFFFFE000000007FFFC000 0034447DAE3A>103 D<007FC000000000FFFFC000000000FFFFC000000000FFFFC000000000FF FFC000000000FFFFC00000000003FFC00000000001FFC00000000001FFC00000000001FFC00000 000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0000000 0001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0 07FE000001FFC03FFFC00001FFC0FFFFF00001FFC1FFFFFC0001FFC7F01FFC0001FFCFC00FFE00 01FFDF0007FF0001FFDE0007FF0001FFFC0007FF0001FFF80003FF8001FFF80003FF8001FFF000 03FF8001FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC00003FF8001 FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003 FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF 8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC0 0003FF8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFF FFFFFF81FFFFFF38487BC741>I<00FC0001FF0003FF8007FF800FFFC00FFFC00FFFE00FFFE00F FFE00FFFC00FFFC007FF8003FF8001FF0000FC0000000000000000000000000000000000000000 0000000000000000000000000000000000007FC0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001 FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001 FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001 FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18 497CC820>I<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC00003FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001 FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000 01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0 00FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF8019487CC720>108 D<00000FFE0000000001 FFFFF000000007FFFFFC0000001FFFFFFF0000007FF803FFC00000FFE000FFE00001FF80003FF0 0003FF00001FF80007FE00000FFC000FFE00000FFE000FFC000007FE001FFC000007FF003FFC00 0007FF803FF8000003FF803FF8000003FF807FF8000003FFC07FF8000003FFC07FF8000003FFC0 FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF80000 03FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE07FF8000003FFC07F F8000003FFC07FF8000003FFC07FF8000003FFC03FFC000007FF803FFC000007FF801FFC000007 FF001FFC000007FF000FFE00000FFE0007FF00001FFC0003FF00001FF80001FFC0007FF00000FF E000FFE000007FFC07FFC000001FFFFFFF00000007FFFFFC00000001FFFFF0000000001FFF0000 00332E7DAD3A>111 D<007FC00FFE0000FFFFC0FFFFC000FFFFC3FFFFF000FFFFCFFFFFFC00FF FFFFE03FFF00FFFFFF000FFF8003FFFC0007FFC001FFF80003FFE001FFF00001FFF001FFE00000 FFF001FFC00000FFF801FFC000007FFC01FFC000007FFC01FFC000003FFC01FFC000003FFE01FF C000003FFE01FFC000003FFE01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001F FF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC0 00001FFF01FFC000001FFF01FFC000001FFF01FFC000003FFE01FFC000003FFE01FFC000003FFE 01FFC000007FFC01FFC000007FFC01FFC000007FF801FFC00000FFF801FFE00000FFF001FFF000 01FFE001FFF80003FFE001FFFC0007FFC001FFFF000FFF0001FFFFC07FFE0001FFCFFFFFF80001 FFC3FFFFF00001FFC0FFFF800001FFC01FF8000001FFC00000000001FFC00000000001FFC00000 000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC0000000 0001FFC000000000FFFFFF80000000FFFFFF80000000FFFFFF80000000FFFFFF80000000FFFFFF 8000000038427CAD41>I<00FF807F00FFFF81FFC0FFFF87FFF0FFFF8FFFF8FFFF9F8FFCFFFF9F 1FFE03FFBE1FFE01FFBC1FFE01FFF81FFE01FFF81FFE01FFF01FFE01FFF00FFC01FFE007F801FF E003F001FFE0000001FFE0000001FFE0000001FFC0000001FFC0000001FFC0000001FFC0000001 FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC00000 01FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC000 0001FFC0000001FFC0000001FFC0000001FFC00000FFFFFFE000FFFFFFE000FFFFFFE000FFFFFF E000FFFFFFE000272E7CAD2F>114 D121 D E /Fw 8 117 df<00001E000000003E00000000FE 00000003FE0000003FFE0000FFFFFE0000FFFFFE0000FFFFFE0000FFCFFE0000000FFE0000000F FE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE000000 0FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000 000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE00 00000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE 0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000FFE0000000F FE0000000FFE0000000FFE00007FFFFFFF807FFFFFFF807FFFFFFF807FFFFFFF80213879B730> 49 D<0000001FFE0000E0000003FFFFE001E000001FFFFFF803E000007FFFFFFE07E00001FFFC 00FF0FE00007FFC0001FDFE0000FFF000007FFE0003FFC000001FFE0007FF0000000FFE000FFE0 0000007FE001FFC00000003FE003FF800000001FE007FF800000001FE007FF000000000FE00FFE 0000000007E00FFE0000000007E01FFC0000000007E01FFC0000000003E03FFC0000000003E03F F80000000003E07FF80000000001E07FF80000000001E07FF80000000001E07FF0000000000000 FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF00000000000 00FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF000000000 0000FFF0000000000000FFF0000000000000FFF00000000000007FF00000000000007FF8000000 0000007FF80000000001E07FF80000000001E03FF80000000001E03FFC0000000001E01FFC0000 000001E01FFC0000000003E00FFE0000000003C00FFE0000000007C007FF0000000007C007FF80 0000000F8003FF800000000F8001FFC00000001F0000FFE00000003E00007FF00000007C00003F FC000000F800000FFF000003F0000007FFC0000FE0000001FFFC007FC00000007FFFFFFF000000 001FFFFFFC0000000003FFFFE000000000001FFE0000003B3D7BBB46>67 D<003FFF00000001FFFFE0000007FFFFFC00000FF007FE00001FF801FF80001FF800FFC0001FF8 007FE0001FF8007FE0001FF8007FF0000FF0003FF00007E0003FF00003C0003FF0000000003FF0 000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF000007FF83FF00003FF 803FF00007FE003FF0001FFC003FF0003FF8003FF0007FF0003FF0007FE0003FF000FFE0003FF0 00FFC0003FF000FFC0003FF000FFC0003FF000FFC0007FF000FFC0007FF000FFE000FFF0007FE0 01DFF0003FF003DFFC001FFC0F9FFFE00FFFFE0FFFE001FFF807FFE0003FE001FFE02B267DA52F >97 D<0003FF8000001FFFF000007FFFFC0000FF83FF0003FE00FF8007FC003F800FF8003FC01F F8001FE01FF0001FE03FF0001FF03FF0000FF07FE0000FF07FE0000FF87FE0000FF8FFE0000FF8 FFE0000FF8FFFFFFFFF8FFFFFFFFF8FFFFFFFFF8FFE0000000FFE0000000FFE0000000FFE00000 00FFE00000007FE00000007FE00000007FF00000003FF00000783FF00000781FF80000F80FF800 00F007FC0001F003FE0003E001FF000FC000FFC07F80003FFFFE00000FFFFC000001FFC0002526 7DA52C>101 D<00FF0000000000FFFF0000000000FFFF0000000000FFFF0000000000FFFF0000 00000007FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003 FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000 000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF 000000000003FF003FE0000003FF01FFFC000003FF07FFFE000003FF0F81FF800003FF3C00FF80 0003FF3800FFC00003FF7000FFC00003FFE0007FE00003FFC0007FE00003FFC0007FE00003FF80 007FE00003FF80007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE000 03FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF0000 7FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003 FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007F E00003FF00007FE000FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80313C 7DBB36>104 D<00FF00FF8000FFFF0FFFF800FFFF3FFFFE00FFFFFE03FF00FFFFF000FFC007FF E0007FE003FFC0003FF003FF80003FF803FF00001FF803FF00001FFC03FF00000FFC03FF00000F FE03FF00000FFE03FF00000FFE03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF 000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF00000F FE03FF00000FFE03FF00000FFE03FF00000FFC03FF00001FFC03FF00001FF803FF80003FF003FF C0007FF003FFE000FFE003FFF001FF8003FFFE07FF0003FF3FFFFC0003FF0FFFF00003FF01FF00 0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF 0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000 00FFFFFC000000FFFFFC000000FFFFFC000000FFFFFC00000030377EA536>112 D<00FE01F800FFFE07FF00FFFE1FFF80FFFE3E3FC0FFFE787FE007FE707FE003FEE07FE003FEE0 7FE003FFC07FE003FFC03FC003FF801F8003FF800F0003FF80000003FF80000003FF00000003FF 00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003 FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF000000 03FF00000003FF00000003FF000000FFFFFE0000FFFFFE0000FFFFFE0000FFFFFE000023267EA5 28>114 D<000F0000000F0000000F0000000F0000000F0000001F0000001F0000001F0000001F 0000003F0000003F0000007F0000007F000000FF000001FF000003FF000007FF00001FFFFFF0FF FFFFF0FFFFFFF0FFFFFFF003FF000003FF000003FF000003FF000003FF000003FF000003FF0000 03FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF00 0003FF000003FF000003FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF 003C01FF807801FF807800FFC0F000FFE1F0003FFFE0000FFF800001FE001E377EB626>116 D E /Fx 25 122 df<70F8FCFC7C0C0C0C0C181830306040060F7C840E>44 DI<018003800F80FF80F380038003800380038003800380038003 80038003800380038003800380038003800380038003800380038003800380038003800380FFFE FFFE0F217CA018>49 D<000E00000E00001E00001E00003E00003E00006E0000EE0000CE0001CE 00018E00030E00070E00060E000E0E000C0E00180E00180E00300E00700E00600E00E00E00FFFF F8FFFFF8000E00000E00000E00000E00000E00000E00000E0001FFF001FFF015217FA018>52 D<01F00007FC000E0E001C0700380380780380700380F001C0F001C0F001C0F001E0F001E0F001 E0F001E0F001E07001E07003E07803E03807E01C0DE00FF9E007F1E00081C00001C00003C00003 80000380300700780700780E00701C003838001FF0000FC00013227EA018>57 D<000FE010003FF83000F81C7001E0067003C003F0078001F00F0000F01E0000F03E0000703C00 00707C0000707C0000307800003078000030F8000030F8000000F8000000F8000000F8000000F8 000000F8000000F800000078000030780000307C0000307C0000303C0000603E0000601E000060 0F0000C0078000C003C0018001E0030000F80E00003FF800000FE0001C247DA223>67 DI70 D<03FFF003FFF0000F00000F00000F00000F00000F00000F00 000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00 000F00000F00000F00000F00000F00700F00F80F00F80F00F80F00F01E00601C003878001FF000 07C00014237EA119>74 D77 D80 D<03F0200FFC601C0EE03803 E07001E07001E0E000E0E000E0E00060E00060E00060F00000F000007800007F00003FF0001FFE 000FFF0003FF80003FC00007E00001E00000F00000F0000070C00070C00070C00070C00070E000 60E000E0F000C0F801C0EF0380C7FF0081FC0014247DA21B>83 D<0FE0001FF8003C1C003C0E00 180700000700000700000F0003FF000FFF003F07007C0700780700F00700F00718F00718F00F18 780F187C3FB83FF3F00FC3C015157E9418>97 D<0E0000FE0000FE00001E00000E00000E00000E 00000E00000E00000E00000E00000E00000E00000E00000E1F800E7FE00FC0F00F00780E00380E 003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0F00380F80700F C1F00C7FC00C1F0017237FA21B>I<01FE0007FF000F07801C0780380300780000700000F00000 F00000F00000F00000F00000F00000F000007800007800C03C00C01E01800F030007FE0001F800 12157E9416>I<01FC0007FF000F07801C03C03801C07801E07000E0FFFFE0FFFFE0F00000F000 00F00000F00000F000007800007800603C00601E00C00F838007FF0000FC0013157F9416>101 D<1C001E003E001E001C00000000000000000000000000000000000E00FE00FE001E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC00A227FA10E>105 D<0E00FE00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00B237FA20E> 108 D<0E1F80FE7FC0FFC1E01F80F00F00700E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00700E0070FFE7FFFFE7FF18157F941B>110 D<01FC0007FF000F07801C01C03800E07800F0700070F00078F00078F00078F00078F00078F000 78F000787000707800F03800E01C01C00F078007FF0001FC0015157F9418>I<0E3CFEFEFFCF1F 8F0F060F000E000E000E000E000E000E000E000E000E000E000E000E000E00FFF0FFF010157F94 13>114 D<0F883FF87078E038E018E018E018F0007F003FE01FF001F8003CC01CC01CE01CE01C F018F878DFF08FC00E157E9413>I<060006000600060006000E000E000E001E003E00FFF8FFF8 0E000E000E000E000E000E000E000E000E000E000E0C0E0C0E0C0E0C0E0C0E08071803F001E00E 1F7F9E13>I<0E0070FE07F0FE07F01E00F00E00700E00700E00700E00700E00700E00700E0070 0E00700E00700E00700E00700E00700E00F00E01F007037803FE7F01F87F18157F941B>I121 D E /Fy 20 122 df68 D73 D76 DI< 00003FC000000001FFF800000007C03E0000001F000F8000003C0003C00000F80001F00001F000 00F80003E000007C0003C000003C0007C000003E000F8000001F000F8000001F001F0000000F80 1F0000000F803E00000007C03E00000007C07E00000007E07E00000007E07C00000003E07C0000 0003E0FC00000003F0FC00000003F0FC00000003F0FC00000003F0FC00000003F0FC00000003F0 FC00000003F0FC00000003F0FC00000003F0FC00000003F0FC00000003F07E00000007E07E0000 0007E07E00000007E07E00000007E03E00000007C03F0000000FC01F0000000F801F0000000F80 0F8000001F000F8000001F0007C000003E0003E000007C0003E000007C0001F00000F80000F800 01F000003E0007C000001F000F80000007E07E00000001FFF8000000003FC000002C337CB134> 79 DI<01FE00000FFFC0001C03E0003C00F0003E00 78003E003C003E001C0008001E0000001E0000001E0000001E0000001E000001FE00003FFE0001 FF1E0007F01E000F801E001F001E003E001E007C001E007C001E00F8001E0CF8001E0CF8001E0C F8003E0CF8003E0C7C007E0C3E00DF1C1F038FB80FFF07F003F803C01E1F7D9E21>97 D<07800000FF800000FF8000000F80000007800000078000000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 80FC000787FF80079E07C007B801E007E000F007C0007807C0007C0780003C0780003E0780003E 0780001E0780001F0780001F0780001F0780001F0780001F0780001F0780001F0780001F078000 1F0780001E0780003E0780003E0780003C07C0007807C00078076000F0073801E0071E07C00607 FF000601FC0020327EB125>I<003FC000FFF803E01C07801E0F003E1E003E1E003E3C00087C00 007C0000780000F80000F80000F80000F80000F80000F80000F80000F80000F800007C00007C00 007C00003E00031E00031F00060F800E07C01C03F03800FFE0003F80181F7D9E1D>I<003F8000 00FFE00003E0F80007803C000F001E001E001E001E000F003C000F007C000F007C000F80780007 80F8000780FFFFFF80FFFFFF80F8000000F8000000F8000000F8000000F8000000F80000007800 00007C0000003C0000003C0001801E0001800F0003000F80060007C00E0001F03C0000FFF00000 1FC000191F7E9E1D>101 D<0003E0001FF8003C3800707C00F07C01E07C01E03803C00003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000 03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E000 7FFF007FFF0016327FB114>I<07000F801F801F800F8007000000000000000000000000000000 00000000000000000780FF80FF800F800780078007800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800FC0FFF8FFF80D307EAF12>105 D<0780FF80FF800F80078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 80078007800780078007800780078007800FC0FFFCFFFC0E327EB112>108 D<0781FE003FC000FF87FF80FFF000FF8E07C1C0F8000F9803E3007C0007B001E6003C0007E000 FC001E0007C000F8001E0007C000F8001E0007C000F8001E00078000F0001E00078000F0001E00 078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0 001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E0007 8000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F000 1E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E38>I<0781FE0000FF87FF80 00FF8E07C0000F9803E00007B001E00007E000F00007C000F00007C000F00007C000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078000F000078000F000078000F00007 8000F000078000F000078000F0000FC001F800FFFC1FFF80FFFC1FFF80211F7E9E25>I<001FC0 0000FFF80001E03C0007800F000F0007801E0003C01E0003C03C0001E03C0001E0780000F07800 00F0780000F0F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F8F80000F878 0000F07C0001F03C0001E03C0001E01E0003C01E0003C00F00078007C01F0001F07C0000FFF800 001FC0001D1F7E9E21>I<0783E0FF8FF8FF9C7C0FB07C07F07C07E03807C00007C00007C00007 800007800007800007800007800007800007800007800007800007800007800007800007800007 80000780000780000780000780000780000FC000FFFE00FFFE00161F7E9E19>114 D<00C00000C00000C00000C00000C00001C00001C00001C00003C00003C00007C0000FC0001FC0 00FFFFE0FFFFE003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0 0003C00003C00003C00003C00003C00003C03003C03003C03003C03003C03003C03003C03003C0 3001E06001E06000F0C0007F80001F00142C7FAB19>116 D<078000F000FF801FF000FF801FF0 000F8001F000078000F000078000F000078000F000078000F000078000F000078000F000078000 F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0000780 00F000078000F000078000F000078000F000078000F000078001F000078001F000078003F00003 8007F00003C00EF00001F03CF80000FFF0FF80003FC0FF80211F7E9E25>I121 D E /Fz 5 85 df<00000000600000000000E00000000000E00000000001E00000000001F00000 000003F00000000003F00000000007F0000000000FF0000000000FF0000000001BF0000000001B F00000000033F00000000033F00000000063F00000000063F000000000C3F000000001C3F00000 000183F00000000303F00000000303F00000000603F80000000603F80000000C01F80000000C01 F80000001801F80000003801F80000003001F80000006001F80000006001F8000000C001F80000 00C001F80000018001F8000001FFFFF8000003FFFFF80000070001F80000060001F800000C0001 F800000C0001F80000180001FC0000180001FC0000300000FC0000300000FC0000600000FC0000 E00000FC0000C00000FC0001C00000FC0003C00000FC000FE00003FC00FFFC003FFFE0FFF8003F FFE02B337CB234>65 D<000FFFFFFC00000FFFFFFF8000007F000FC000007E0003E000007E0001 F000007E0000F800007E00007C0000FC00007C0000FC00003E0000FC00003E0000FC00003E0001 F800001E0001F800001F0001F800001F0001F800001F0003F000001F0003F000001F0003F00000 1F0003F000001F0007E000003F0007E000003F0007E000003F0007E000003F000FC000003E000F C000007E000FC000007E000FC000007E001F8000007C001F800000FC001F800000FC001F800000 F8003F000001F8003F000001F0003F000003F0003F000003E0007E000007E0007E000007C0007E 00000F80007E00000F0000FC00001F0000FC00003E0000FC00007C0000FC0000F80001F80001F0 0001F80007C00001F8000F800003F8007E0000FFFFFFF80000FFFFFFC0000030317BB035>68 D<000FFFFFFFF8000FFFFFFFF800007F0003F800007E0000F800007E00007800007E0000780000 7E0000300000FC0000300000FC0000300000FC0000300000FC0000300001F80000300001F80000 300001F80000300001F80000600003F00180600003F00180000003F00180000003F00180000007 E00300000007E00300000007E00700000007E01F0000000FFFFE0000000FFFFE0000000FC01E00 00000FC00E0000001F800C0000001F800C0000001F800C0000001F800C0000003F00180000003F 00180000003F00000000003F00000000007E00000000007E00000000007E00000000007E000000 0000FC0000000000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F8 0000000003F800000000FFFFF0000000FFFFF00000002D317BB02E>70 D<000FFFFFE000000FFF FFFC0000007F003F0000007E000F8000007E0007C000007E0003E000007E0003F00000FC0001F0 0000FC0001F00000FC0001F00000FC0001F00001F80003F00001F80003F00001F80003F00001F8 0003E00003F00007E00003F00007C00003F0000FC00003F0001F800007E0003F000007E0007C00 0007E000F8000007E007E000000FFFFF8000000FFFFF0000000FC00FC000000FC007E000001F80 03F000001F8001F000001F8001F800001F8001F800003F0001F800003F0001F800003F0001F800 003F0001F800007E0003F000007E0003F000007E0003F000007E0003F00000FC0007E00000FC00 07E00000FC0007E00000FC0007E01801F80007E03001F80007E03001F80007E06003F80003E060 FFFFE001F1C0FFFFC000FF80000000003E002D327BB033>82 D<07FFFFFFFFF007FFFFFFFFF00F E007F007F00F8007E001F00E0007E000E00C0007E000E01C0007E000E018000FC000E018000FC0 00E030000FC000C030000FC000C030001F8000C060001F8000C060001F8000C060001F8000C0C0 003F00018000003F00000000003F00000000003F00000000007E00000000007E00000000007E00 000000007E0000000000FC0000000000FC0000000000FC0000000000FC0000000001F800000000 01F80000000001F80000000001F80000000003F00000000003F00000000003F00000000003F000 00000007E00000000007E00000000007E00000000007E0000000000FC0000000000FC000000000 0FC0000000000FC0000000001F80000000001F80000000001F80000000003F800000007FFFFF80 00007FFFFF8000002C3174B032>84 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 0 1 bop 799 988 a Fz(D)25 b(R)g(A)g(F)g(T)388 1079 y Fy(Do)r(cumen)n(t)c(for)h(a) f(P)n(arallel)h(MPI)g(IO)g(Library)207 1274 y Fx(Marc)16 b(Snir)116 b(P)o(eter)16 b(Corb)q(ett)117 b(Dror)17 b(F)l(eitelson)116 b(Jean-Pierre)15 b(Prost)793 1400 y(Jan)o(uary)i(14,)f(1994)p eop %%Page: 1 2 bop 75 362 a Fw(Chapter)31 b(1)75 570 y Fv(IO)75 812 y Fu(1.1)59 b(Intro)r(duction)75 916 y Ft(This)16 b(do)q(cumen)o(t)f(is)g(a)g(prop)q (osal)g(for)g(an)g(extension)g(of)g(MPI)g(in)h(supp)q(ort)f(of)f(\014le)i (I/O)g(\(MPI-IO\).)f(The)75 972 y(need)g(for)e(suc)o(h)i(extension)g(arises)f (from)f(the)i(fact)e(that)g(not)h(all)h(parallel)h(mac)o(hines)f(supp)q(ort)f (Unix)h(\014le)75 1029 y(access)h(at)e(eac)o(h)i(no)q(de;)g(when)f(it)h(is)g (supp)q(orted,)g(the)f(sharing)h(seman)o(tics)f(of)g(Unix)i(\014les)f(are)f (ill-suited)75 1085 y(to)c(parallel)j(computing,)e(and)h(dep)q(end)g(on)f (the)g(t)o(yp)q(e)g(of)f(\014le)i(system)f(accessed)g(and)g(on)

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26006; Thu, 25 Feb 93 12:19:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06312; Thu, 25 Feb 93 12:17:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06274; Thu, 25 Feb 93 12:16:42 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:34 GMT Message-Id: <2153.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12