From owner-mpi-comm@CS.UTK.EDU Mon Nov 23 16:08:35 1992 Return-Path: Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA26326; Mon, 23 Nov 92 15:50:10 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26317; Mon, 23 Nov 92 15:50:05 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA06935; Mon, 23 Nov 92 15:50:04 -0500 Date: Mon, 23 Nov 92 15:50:04 -0500 Message-Id: <9211232050.AA06935@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: mpi committees and mailing list Status: R The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail will be collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi etc. Also try: send index from mpi I'm in the process of collecting information on how the HPF committee operates and will pass along the details in the next few days. I would like to suggest January 6 - 8 for the next meeting in Dallas at the airport hotel. We should plan on starting around 1:00 on January 6th and finishing around 3:00 on January 8th. Below is a list of the subcommittees as I have them from our meeting last week. Please let me know if you have any changes. Regards, Jack Introduction Subcommittee ---------------------- Jack Dongarra - Chair dongarra@cs.utk.edu Daniel Frye danielf@kgnvma.vnet.ibm.com Tony Hey ajgh@ecs.soton.ac.uk Rusty Lusk lusk@mcs.anl.gov Steve Zenith zenith@kai.com Point-To-Point Communication Subcommittee ----------------------------------------- Marc Snir - Chair snir@watson.ibm.com Scott Berryman berryman@cs.yale.edu Jack Dongarra dongarra@cs.utk.edu Al Geist geist@msr.epm.ornl.gov Adam Greenberg moose@think.com Bill Gropp gropp@mcs.anl.gov Leslie Hart hart@fsl.noaa.gov Rolf Hempel hempel@gmd.de Tom Henderson hender@fsl.noaa.gov Bob Knighten knighten@ssd.intel.com Oliver McBryan mcbryan@piper.cs.colorado.edu Peter Rigsbee par@cray.com David Walker walker@msr.epm.ornl.gov Collective Communication Subcommittee ------------------------------------- Al Geist - Chair geist@msr.epm.ornl.gov Leslie Hart hart@fsl.noaa.gov Jon Flowers jwf@parasoft.com Adam Greenberg moose@think.com Bob Knighten knighten@ssd.intel.com Steve Otto otto@cse.ogi.edu Peter Rigsbee par@cray.com Marc Snir snir@watson.ibm.com David Walker walker@msr.epm.ornl.gov Process Topology Subcommittee ----------------------------- Rolf Hempel - Chair hempel@gmd.de Jon Flowers jwf@parasoft.com Tom Henderson hender@fsl.noaa.gov Oliver McBryan mcbryan@piper.cs.colorado.edu Steve Otto otto@cse.ogi.edu Lew Tucker tucker@think.com Language Binding Subcommittee ----------------------------- Scott Berryman - Chair berryman@cs.yale.edu Bruce Leisure bleasure@kai.com Formal Language Description Subcommittee ---------------------------------------- Steve Zenith - Chair zenith@kai.com Tony Hey ajgh@cs.ston.ac.uk Rusty Lusk lusk@mcs.anl.gov Environment Inquiry Subcommittee --------------------------------- Bill Gropp - Chair gropp@mcs.anl.gov Daniel Frye danielf@kgnvma.vnet.ibm.com From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 15:37:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02830; Tue, 24 Nov 92 15:37:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA19627; Tue, 24 Nov 92 15:12:24 -0500 Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19622; Tue, 24 Nov 92 15:12:21 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA25860; Tue, 24 Nov 92 14:12:19 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA11600; Tue, 24 Nov 92 14:12:17 CST Date: Tue, 24 Nov 92 14:12:17 CST Message-Id: <9211242012.AA11600@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Test Implementation of the Draft Standard We have completed a first and rough draft of a test implementation. We have included (and run!) three sample programs written in MPI on a sun network and on an Intel ipsc/i860. It is available by anonymous ftp from info.mcs.anl.gov in pub/mpi . The README file there gives more information, including how to get, install, and run the examples. This is a rough draft; the file mpi.man.ps.Z contains a description of the implementations architecture and man pages for all of the routines. We will be updating this implementation with additional examples and fewer restrictions soon. Enjoy! Bill Gropp and Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 17:07:49 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06915; Tue, 24 Nov 92 17:07:49 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA21596; Tue, 24 Nov 92 16:45:52 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21572; Tue, 24 Nov 92 16:45:38 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA21258; Tue, 24 Nov 92 16:45:35 -0500 Date: Tue, 24 Nov 92 16:45:35 -0500 Message-Id: <9211242145.AA21258@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: working procedures Chuck Koelbel has passed on some notes from the HPF experence. I think we will find them useful in carring out our meetings. Jack HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From chk@cs.rice.edu Mon Nov 23 15:45:48 1992 Return-Path: Received: from cs.rice.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26184; Mon, 23 Nov 92 15:45:39 -0500 Received: from DialupEudora (charon.rice.edu) by cs.rice.edu (AA22750); Mon, 23 Nov 92 14:44:21 CST Message-Id: <9211232044.AA22750@cs.rice.edu> Date: Mon, 23 Nov 1992 14:47:55 -0600 To: dongarra@cs.utk.edu From: chk@cs.rice.edu Subject: HPFF organization X-Attachments: :Macintosh HD:4227:HPFF rules: Status: RO Here it is. If anything is unclear, just ask me or (for logistical details) Theresa Chatman (tlc@cs.rice.edu). Chuck HPFF Working Procedures HPFF never had an explicit vote on adopting any operating procedures. We did, however, informally adopt the following rules. General Organization Most of the technical details were hammered out in subgroups, which made the base proposals. The main group would discuss these proposals and (usually) adopt them. In practice, lots of the technical discussions were done on email lists. Keeping those lists public was a big part of the openness of the forum. For a while, each subgroup brought its own proposal as handouts. Eventually, we ended up with a unified draft thanks to Dave Loveman. The intent (which we never lived up to) was that all proposals discussed at a meeting would be in that draft. Despite always having a loose proposal or two, the draft was extremely useful - among other things, it made editing a final report possible. Main Group / Plenary Session Matters For running discussions, we used loosely-enforced Robert's Rules of Order. Basically, we didn't stand on formalities unless the discussion was becoming unruly. Some of the more often-invoked rules: 1. Moving and voting on amendments before the main proposal. Basically, this kept the discussions focussed. 2. Motions coming out of committee (subgroup) are automatically seconded; others need a second from the floor. 3. Triply-nested amendments are not allowed. This keeps the confusion down. Generally, whoever was making the presentation ran the discussion, recognized comments from the floor, etc. Ken usually came in after the meat of the discussion to run the votes. Peer pressure was fairly effective at keeping people from filibustering (I think once there was a call to limit discussion). More common was lots of people wanting to make comments - this was handled by the presenters, usually by saying "OK, let's go clockwise around the table". Organizations were limited to 2 attendees each (not enforced, but no major problems). We passed around an attendence sheet at each meeting to keep track of who was there; because of various planning problems, I recommend using a pre-registration scheme ("send email to xxx@yyy if you plan to attend"). Organizations were asked to commit to having the same attendee at every meeting, and this was generally followed. It's very important to have this kind of continuity in attendees, else we would have spent too much time in remedial education. Each organization (school, company, lab) got one vote - note this was on an organization basis, not a person. We didn't have trouble with cheating on this rule - just make sure the representatives from an organization have agreed on who is voting. An organization was eligible to vote if it had attended 2 of the last 3 meetings, counting the current meeting (i.e. you could attend every other meeting and still vote; you could not vote at your first meeting). Obviously, we couldn't enforce this rule at the first two meetings. Accepting a section of the HPF language spec was a multi-step process. 1. Someone wrote a draft specification; often there were a couple drafts. Details of these were hashed out in the subgroup. 2. First Reading: The subgroup leader (or occasionally the draft author) presented the subgroup-approved draft to the main group. When there was controversy, it was usually pointed out. The main group discussed, suggested changes (sometimes), and held a series of "straw votes" on the proposal. All attendees were eligible to vote in straw votes, which were not binding on the subgroups. 3. More subgroup discussion, both electronic and in person at the next meeting, producing a revised proposal. 4. Second Reading: The subgroup leader presented the revised proposal to the main group. Sections that were substantially the same as the original (or an alternative presented at the first reading) were amended by motion and eventually voted on. Eligibility for votes was as explained above. Major additions (for example, when PURE functions were added to FORALL between meetings) were treated as first readings at this point. Often most of the proposal was accepted, and a few sections were sent back to subgroup for more work; these came back as second readings at the next meeting. 5. Once a section was accepted at second reading, it was "frozen" until the end of the HPFF process. Revisions were only allowed for clarity or when new information surfaced (like discovering that the draft was self-contradictory). This limitation was enforced as strictly as we could, to avoid backtracking. 6. At the end of the process (December), we promised to allow reconsideration of any feature. See also the discussion of the Journal of Development below. Subgroup Matters Subgroups met independently of the main body, usually the afternoon and evening of the day before. The leader of the subgroup ran these meetings using whatever style he or she was comfortable with. Most of us took votes from whoever showed up, and ran a pseud-democratic style. Also, there was a mailing list for each subgroup where most of the discussions went on. Each subgroup was devoted to one topic from the following list: Data distribution FORALL and other parallel statements Fortran 90, storage & sequence association, and the HPF subset Intrinsic functions Parallel I/O EXTRINSIC (nee LOCAL, nee FOREIGN) functions The groups met in parallel, which caused a little friction but was logstically unavoidable. When a subject straddled two groups (like intrinsic functions for enquiring about distributions), the subgroup leaders would talk to each other and decide who would handle it - this often led to minor anti-turf battles (also known as "after you" deadlock). Both groups would act as sanity checks on the results. When it came time to write the draft, each subgroup became a chapter. The subgroup leader was the editor (and usually major author) of the chapter, and was responsible for making sure the chapter reflected the decisions made in the subgroup and in committee. Logistical Matters Meetings were 2.5 days, starting Wednesday afternoon, in Dallas (there were exceptions to this for logistical reasons, but we would have prefered to keep them all on this schedule). A typical schedule was Wednesday 1:30-6:00 Subgroup meetings, about 3 going on in parallel (reserve 4 "breakout" rooms with the hotel, fo 10-20 people each) 6:00-7:30 Unofficial dinner break (usually the subgroup leaders ate together & planned the meeting agenda) 7:30-10:30 More subgroup meetings (subgroup leaders usually ended up staying up late to revise drafts) Thursday 9:00-12:00 Full group meeting (coffee breaks at natural breaks) 12:00-1:30 Lunch (provided) 1:30-6:00 Full group meeting (and coffee breaks) 6:00-8:00 Dinner (attendees pay, but hotel provided transport to area restaurant) 8:00-10:00 (Sometimes) Full group meeting (when no full meeting, subgroups usually met instead) Friday 9:00-12:00 Full group meeting (and coffee breaks) 12:00-1:30 (Sometimes) Lunch (provided if we thought the meeting would last long) 1:30-3:00 (Sometimes) Full group meeting We tried to finish as early as possible on Friday, to allow people to catch flights. Since we scheduled all the meetings early on, we set up a contract with the hotel. I didn't handle the details, but I think this saved us a little money and a lot of aggravation (talk to Theresa Chatman if you need more info). At any rate, this is a great thing for someone on your staff to set up. We financed the meetings primarily from a per-meeting fee of $95 ($75 if we didn't have lunch the second day) per attendee. Ken offered to foot the bill for academics to attend, based on the expectation of NSF funds. We're still waiting on those funds; highly recommend that you don't make such offers without first getting the cash. Documentation The Draft: The earlier you have one the better. David Loveman served as our general editor, collecting the chapters and trying to smooth format details. I contributed the introduction and credits, mainly cribbed from meeting notes and other announcements. Guy Steele contributed a set of macros for BNF syntax and reproducing code segments. We tried to find some formatting from an ANSI standard, but no dice. Toward the end of the process, version control became a problem. We (tried to) set up the following framework: 1. David had the "official" version of the draft. He set deadlines for receiving the chapters, and edited for formatting. 2. David sent me the whole document when all the chapters were done. I LaTeXed it to check for system dependences (never a problem) 3. I sent the document to the "core" mail group (see below), put it out for anonymous FTP, and announced it on the net. 4. Any further changes were supposed to be made to David's edited copy, not the original version (this to make the next round of edits easier). The system worked pretty well, until the final draft when the wrong version of Chapter 6 got distributed... Each subgroup wrote one chapter, generally written and/or edited by the subgroup leader. While the meetings were going on, the draft contained all proposals still under consideration (Guy provided a "\alternative" macro that marked sections as being alternatives to each other). Also, the author of each section was identified in a footnote at the beginning of the section, along with the date (and occasionally other version information). In the current draft, we've moved summaries of some discarded proposals to the "Journal of Development" (primarily the ones dropped for lack of time). All proposals exist in the archives; we may put together a "rationale" document to explain some of the more controversial decisions. We're also planning to move the authorship information from footnotes to the credits chapter in the December draft. It's too early to know how this organization will work for the reader, but the modularity has helped in writing the document. Other lessons from the draft: 1. Make sure the chapter authors realize they are writing a draft chapter, not a stand-alone document. 2. Make sure the person in charge of the credits chapter has plenty of time to work on it (I spent 50% of my time for a week appeasing various corporate egos, wording things right, and checking spelling of names). 3. Proofread the document before sending it out - both spell checking and careful chapter reading. Mailing lists: Every subgroup had its own mailing list. Those lists are where most of the technical action happened; they were very high-volume (final total was something like 1.5Mb of archived messages). On top of that, there was a list for everybody in the world interested in HPF, and another for the "core" group. The "everybody" list was used for the meeting minutes and a few miscellaneous announcements. The core group list was primarily for the meeting attendees, but a few others were also on it for political and/or practical reasons (for example, Gil Weigand and Theresa Chatman were both on the list); it got meeting details, and copies of the various proposals before the meetings. All lists were kept at cs.rice.edu (there were a couple smaller lists other places, including a European list at GMD and several local lists for particular campuses). People could add or delete themselves to/from any of the lists by mailing to another mail alias at Rice. Automating this was a huge win (although I got 2 or 3 requests sent directly to me every week). FTP: We made everything we could available by anonymous FTP at Rice; eventually other sites picked up some of it as well. This included the various drafts of the language spec; any other handouts from the meetings; supporting (sometimes critical) commentary; papers on Fortran D, Vienna Fortran, ADAPT, and other systems; archived email traffic; and the minutes of the meetings. Generally, I put up a couple versions of any document (particularly text formatting stuff) - TeX or LaTeX, Postscript, and compressed versions of large files. This was important, since I've found out some fascinating things about portability (or lack thereof) of different formats. (Hints: keep all lines 80 characters or less; no fancy macros in TeX; at most one "." in a file name (filename.tex.Z doesn't translate well to IBM!); never include ANY pathname.) Minutes: Extremely popular, but lots of work. I'm convinced that making them available was a key to HPFF's success. Make them as detailed as possible, particularly including counts of the votes taken, major topics discussed, and lists of people (humor helps occasionally, too). "Executive Summary" sections with the major issues at the front seemed to be popular. I also used the minutes to announce additions to the FTP archives. Advertizing: Announcements of major news (new drafts, etc.) went everywhere I could think of. This includes the "world" mailing list; newsgroups comp.lang.fortran, comp.lang.misc, and comp.parallel; (indirectly) na-net, scinet, and hpcwire. Meeting minutes went to the world and core mailing lists and the newsgroups. I've been writing short "What is HPF" articles constantly, and I think the same is true of others. Ken, Dave Loveman, Guy, and I have all given large numbers of talks on HPF; again, the same is probably true of others. This includes invited talks at conferences and visits to various companies. From owner-mpi-comm@CS.UTK.EDU Tue Nov 24 18:09:18 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08323; Tue, 24 Nov 92 18:09:18 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22602; Tue, 24 Nov 92 17:43:42 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22596; Tue, 24 Nov 92 17:43:35 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA11950; Tue, 24 Nov 92 17:43:40 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 174208.17287; Tue, 24 Nov 1992 17:42:08 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301) id AA07130; Tue, 24 Nov 1992 16:14:37 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA06815; Tue, 24 Nov 92 16:14:36 -0600 Date: Tue, 24 Nov 92 16:14:36 -0600 Message-Id: <9211242214.AA06815@brisk.kai.com> To: dongarra@cs.utk.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Jack Dongarra's message of Tue, 24 Nov 92 15:04:05 -0500 <9211242004.AA20345@thud.cs.utk.edu> Subject: more on the intro From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 Concerning the introduction: "The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds." I think this sentence is unnecessarily defensive. I'd delete the sentence and the following "thus". I earlier raised an objection to \item A simple way to create processes for the SPMD model there seems to be some redundancy here with \item Process groups The former is a subset of the latter. I'll repeat my noncirculated comments: I would rather not see us say too much about the process model at this stage; except to say that there is some process model defined by the language using the standard and to say something about side effects between processes that affect the message passing semantics (i.e., that there should be none). That does, of course, leave us with some interesting problems for identifying our communication; traditionally a difficult subject in message passing (hence my distribution of this message to the whole committee). The easy route would be to have a global name space of shared "channels" each with a communication characteristic [type] (e.g., "point-to-point" or "one-to-many"): a compiler can check correct usage if we identify the right rules (e.g. a synchronized unidirectional point-to-point communication is shared by only two processes). Such logical naming would map well across all the existing systems and avoid the problems of name servers, processor id, topologies and so forth since any topology can be described by the set of names and its users. Processor ids are, in effect, names in a global name space - and are often channels that have a "many-to-one" characteristic. Steven From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:38:03 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07921; Sat, 28 Nov 92 18:38:03 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22407; Sat, 28 Nov 92 18:35:38 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22403; Sat, 28 Nov 92 18:35:36 -0500 Message-Id: <9211282335.AA22403@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6064; Sat, 28 Nov 92 18:35:01 EST Date: Sat, 28 Nov 92 18:32:55 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: previously sent postcript file Reply-To: SNIR@watson.ibm.com Contains an outline I want to use for discussing issues in the poin-to-point subcommittee. I send it to the entire committee since it touches many issues related to collective communication and formal definitions. If somebody cannot print postscript and prefers latex, pls let me know. From owner-mpi-comm@CS.UTK.EDU Sat Nov 28 18:40:52 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07938; Sat, 28 Nov 92 18:40:52 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA22399; Sat, 28 Nov 92 18:33:55 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22394; Sat, 28 Nov 92 18:33:40 -0500 Message-Id: <9211282333.AA22394@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 6053; Sat, 28 Nov 92 18:33:04 EST Date: Sat, 28 Nov 92 18:32:22 EST From: "Marc Snir" To: MPI-COMM@CS.UTK.EDU %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: MPIOUTLN.DVI.* %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: texc.pro /TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{ ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]/id ch-image N /rw ch-width 7 add 8 idiv string N /rc 0 N /gp 0 N /cp 0 N{rc 0 ne{rc 1 sub /rc X rw}{G}ifelse}imagemask restore}B /G{{id gp get /gp gp 1 add N dup 18 mod S 18 idiv pl S get exec}loop}B /adv{cp add /cp X}B /chg{rw cp id gp 4 index getinterval putinterval dup gp add /gp X adv}B /nd{/cp 0 N rw exit}B /lsh{rw cp 2 copy get dup 0 eq{pop 1}{dup 255 eq{pop 254}{dup dup add 255 and S 1 and or}ifelse}ifelse put 1 adv}B /rsh{rw cp 2 copy get dup 0 eq{pop 128}{dup 255 eq{pop 127}{dup 2 idiv S 128 and or}ifelse}ifelse put 1 adv}B /clr{rw cp 2 index string putinterval adv}B /set{rw cp fillstr 0 4 index getinterval putinterval adv}B /fillstr 18 string 0 1 17{2 copy 255 put pop}for N /pl[{adv 1 chg}bind{adv 1 chg nd}bind{1 add chg}bind{1 add chg nd}bind{adv lsh}bind{ adv lsh nd}bind{adv rsh}bind{adv rsh nd}bind{1 add adv}bind{/rc X nd}bind{1 add set}bind{1 add clr}bind{adv 2 chg}bind{adv 2 chg nd}bind{pop nd}bind]N /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval (Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail} B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{ /SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 16 df15 D E /Fb 51 125 df<127012F8B012701200A5 127012F8A31270051C779B18>33 D<137013F01201EA03C0EA0780EA0F00121E121C123C123812 781270A212F05AA87E1270A212781238123C121C121E7EEA0780EA03C0EA01F0120013700C2479 9F18>40 D<126012F012787E7E7EEA0780120313C0120113E01200A213F01370A813F013E0A212 0113C0120313801207EA0F00121E5A5A5A12600C247C9F18>I<123C127E127FA3123F120F120E 121E127C12F81270080C788518>44 DI48 DII<387FFFC0B512E0A3C8FCA4B512E0A36C13C0130C7E9318>61 D<137013F8A213D8A2EA01DCA3138CEA038EA41306EA0707A4380FFF80A3EA0E03A2381C01C0A2 387F07F038FF8FF8387F07F0151C7F9B18>65 DI<3801FCE0EA03FEEA07FFEA0F07EA1E03EA3C01EA78001270A200F013005AA87E0070 13E0A21278EA3C01001E13C0EA0F073807FF806C1300EA01FC131C7E9B18>IIII<3801F9C0EA07FF5AEA1F0FEA1C03123CEA78011270A200F0C7FC5AA5 EB0FF0131F130F38F001C0127013031278123CEA1C07EA1F0FEA0FFFEA07FDEA01F9141C7E9B18 >I<387F07F038FF8FF8387F07F0381C01C0A9EA1FFFA3EA1C01AA387F07F038FF8FF8387F07F0 151C7F9B18>II76 D<38FC01F8EAFE03A2383B06E0 A4138EA2EA398CA213DCA3EA38D8A213F81370A21300A638FE03F8A3151C7F9B18>I<387E07F0 38FF0FF8387F07F0381D81C0A313C1121CA213E1A313611371A213311339A31319A2131D130DA3 EA7F07EAFF87EA7F03151C7F9B18>III82 D<3807F380EA1FFF5AEA7C1FEA7007EAF00312E0A2 90C7FC7E1278123FEA1FF0EA0FFEEA01FF38001F80EB03C0EB01E01300A2126012E0130100F013 C0EAFC07B512801400EAE7FC131C7E9B18>I<387FFFF8B5FCA238E07038A400001300B2EA07FF A3151C7F9B18>I<38FF83FEA3381C0070B2001E13F0000E13E0EA0F013807C7C03803FF806C13 00EA007C171C809B18>I<38FF07F8A3381C01C0A4380E0380A4EA0F0700071300A4EA038EA4EA 018C13DCA3EA00D813F8A21370151C7F9B18>I<38FE03F8A338700070A36C13E0A513F8A2EA39 DCA2001913C0A3138CEA1D8DA4000D13801305EA0F07A2EA0E03151C7F9B18>I97 D<127E12FE127E120EA5133EEBFF80000F13C0EBE3E0EB80F0EB0070 1478000E1338A5120F14781470EB80F0EBC3E0EBFFC0000E138038067E00151C809B18>IIIII<3803F1F03807FFF85A381E1F30 383C0F00EA3807A5EA3C0FEA1E1EEA1FFC485AEA3BF00038C7FC123CEA1FFF14C04813E0387801 F038F00078481338A36C1378007813F0EA7E03383FFFE0000F13803803FE00151F7F9318>I<12 7E12FE127E120EA5133FEBFF80000F13C0EBE1E013801300A2120EAA387FC3FC38FFE7FE387FC3 FC171C809B18>II<12FEA3120EA5EB3FF0137F133FEB0780EB0F00131E5B5B5BEA0FF87F139C131EEA0E0FEB 0780130314C038FFC7F8A3151C7F9B18>107 DI<387DF1F038FFFBF86CB47E381F1F1CEA1E1EA2EA1C1CAB387F1F1F39FFBF BF80397F1F1F001914819318>IIII<387F87E038FF9FF8EA7FBF3803 FC78EBF030EBE0005BA35BA8EA7FFEB5FC6C5A15147F9318>114 DI<487E1203A4387FFFC0B5FCA238038000A9144014E0A21381EBC3C0EA01 FF6C1380EB7E0013197F9818>I<387E07E0EAFE0FEA7E07EA0E00AC1301EA0F073807FFFC6C13 FE3801FCFC1714809318>I<38FF8FF8A3383800E0A3381C01C0A2137113F9A213D9A2380DDD80 A3138DEA0F8FA23807070015147F9318>119 D<387F8FF000FF13F8007F13F0380E01C0EB0380 A21207EB0700A2EA03871386138EEA01CEA2EA00CCA213DC1378A31370A313F05B1279EA7BC0EA 7F806CC7FC121E151E7F9318>121 D<126012F0B3B012600424769F18>124 D E /Fc 12 119 df97 DI<137E48B4FC38038380EA0F07121E001C1300EA3C0248C7FCA35AA5EA70 021307EA381EEA1FF8EA07E011147C9315>I<1478EB03F814F0EB0070A314E0A4EB01C0A213F1 EA03FD38078F80EA0E07121C123C14001278A3EAF00EA31430EB1C60133CEA707C3878FCC0EA3F CF380F078015207C9F17>I<137C48B4FCEA0783380F0180121E123CEB0300EA780EEA7FFC13E0 00F0C7FCA412701302EA7807EA3C1EEA1FF8EA07E011147C9315>I103 D<136013F0A213E01300A7120FEA1F8012 3113C0EA6380A212C3EA0700A3120EA3EA1C301360A2EA38C01218EA1F80EA0F000C1F7D9E0E> 105 D108 D<381E07C0383F1FE03833B870EA63E0EBC038138000C71370EA0700A3000E 13E0A3EB01C3001C13C6A2EB038C1301003813F8381800F018147D931A>110 D<137C48B4FC38038380380F01C0121E001C13E0123C1278A338F003C0A3EB07801400EA700F13 1EEA3838EA1FF0EA07C013147C9317>I116 D<380F01C0EA1F83003113E0 13C1EA6380A200C313C0EA0700A3380E0180A3EB0300A21306A2EA0F0CEA07F8EA01E013147D93 15>118 D E /Fd 28 121 df<903807F83F017FB512C03A01FC0FE3E03903F01FC7EA07E0D80F C01387ED83C0ED8000A6B612FCA2390FC01F80B2397FF8FFF8A223237FA221>11 D<13181330136013C01201EA0380120713005A121EA2123E123CA2127CA3127812F8AD1278127C A3123CA2123E121EA27E7E13801203EA01C012001360133013180D317BA416>40 D<12C012607E7E121C7E120F7E1380EA03C0A213E01201A213F0A3120013F8AD13F01201A313E0 A2120313C0A2EA078013005A120E5A12185A5A5A0D317DA416>I63 D67 D71 D77 D87 D97 DIII<13FE3807FF80380F87C0381E01E0003E13F0EA7C0014 F812FCA2B5FCA200FCC7FCA3127CA2127E003E13186C1330380FC0703803FFC0C6130015167E95 1A>II<3801FE1F0007B51280380F87E7EA1F03391E01E000003E7FA5001E5BEA1F03 380F87C0EBFF80D819FEC7FC0018C8FC121CA2381FFFE014F86C13FE80123F397C003F8048131F 140FA3007CEB1F00007E5B381F80FC6CB45A000113C019217F951C>II<120E121FEA3F80A3EA1F00 120EC7FCA7EAFF80A2121FB2EAFFF0A20C247FA30F>I108 D<3AFF87F00FE090399FFC3FF83A1FB87E70FC9039E03EC07C9039C03F807EA2018013 00AE3BFFF1FFE3FFC0A22A167E952F>I<38FF87E0EB9FF8381FB8FCEBE07CEBC07EA21380AE39 FFF1FFC0A21A167E951F>I<13FE3807FFC0380F83E0381E00F0003E13F848137CA300FC137EA7 007C137CA26C13F8381F01F0380F83E03807FFC03800FE0017167E951C>I<38FF8FE0EBBFF838 1FF07CEBC03E497E1580A2EC0FC0A8EC1F80A2EC3F00EBC03EEBE0FCEBBFF8EB8FC00180C7FCA8 EAFFF0A21A207E951F>I114 DI<13C0A41201A2120312 07120F121FB5FCA2EA0FC0ABEBC180A51207EBE300EA03FEC65A11207F9F16>I<38FF83FEA238 1F807EAF14FEA2EA0F833907FF7FC0EA01FC1A167E951F>I<39FFF01FE0A2390FC00600A2EBE0 0E0007130CEBF01C0003131813F800015BA26C6C5AA2EB7EC0A2137F6D5AA26DC7FCA2130EA21B 167F951E>I<39FFF07FC0A2390FC01C006C6C5A6D5A6C6C5A00015B3800FD80017FC7FCA27F6D 7E497E80EB67F013E33801C1F8380381FC48C67E000E137E39FF81FFE0A21B167F951E>120 D E /Fe 1 4 df<1207A3EAE738EAFFF8EA7FF0EA1FC0A2EA7FF0EAFFF8EAE738EA0700A30D0E 7E8E12>3 D E /Ff 33 118 df<1318137013E0EA01C0EA0380A2EA0700120EA2121E121C123C A25AA412F85AA97E1278A47EA2121C121E120EA27EEA0380A2EA01C0EA00E0137013180D2D7DA1 14>40 D<12C012707E7E7EA27EEA0380A213C0120113E0A2EA00F0A413F81378A913F813F0A4EA 01E0A213C012031380A2EA0700120EA25A5A5A12C00D2D7DA114>I<14E0B0B712C0A3C700E0C7 FCB022237D9C29>43 D<1238127C12FE12FFA2127F123B1203A21206A2120E120C121812701220 08107C860F>I<137013F0120F12FF12F31203B3A4B51280A2111D7C9C1A>49 DIII<14E0A2497EA3497EA2497EA2 497E130CA2EB187FA201307F143F01707FEB601FA201C07F140F48B57EA2EB800748486C7EA200 06801401000E803AFFE01FFFE0A2231F7E9E28>65 D69 DI73 D78 D82 D<3803FC08380FFF38381E03F8EA3C00481378143812F814187E1400B4FC13F86C B4FC14C06C13E06C13F06C13F8120338001FFC13011300A200C0137CA36C1378A200F813F038FE 01E038E7FFC000811300161F7D9E1D>I<007FB512FCA2397C0FE07C0070141C0060140CA200E0 140E00C01406A400001400B10007B512C0A21F1E7E9D24>I87 D97 D100 DII<3801FC3C3807FFFE380F07DEEA1E03003E13E0A5001E13C0380F 0780EBFF00EA19FC0018C7FCA2121C381FFF8014F06C13F8003F13FC387C007C0070133E00F013 1EA30078133CA2383F01F8380FFFE000011300171E7F931A>II<121C123F5AA37E121CC7FCA6B4FCA2121F B0EAFFE0A20B217EA00E>I108 D<3AFE0FE03F8090393FF0FFC03A1E70F9C3E09039C07F01F0381F807EA2EB007CAC3AFFE3FF8F FEA227147D932C>I<38FE0FC0EB3FE0381E61F0EBC0F8EA1F801300AD38FFE3FFA218147D931D> I<48B4FC000713C0381F83F0383E00F8A248137CA200FC137EA6007C137CA26C13F8A2381F83F0 3807FFC00001130017147F931A>I<38FF1FC0EB7FF0381FE1F8EB80FCEB007EA2143E143FA614 3E147E147CEB80FCEBC1F8EB7FE0EB1F8090C7FCA7EAFFE0A2181D7E931D>I114 DII<38FF07F8A2EA1F00AD1301A2EA0F073807FEFFEA03F818147D931D>I E /Fg 77 124 df<90380FC3E090387FEFF09038E07C783801C0F8D8038013303907007000A7B6 1280A23907007000B0387FE3FFA21D20809F1B>11 DII<90380F80F890387FE7FE9038E06E063901C0 FC0F380380F8380700F00270C7FCA6B7FCA23907007007B03A7FE3FE3FF0A22420809F26>I<90 380FC0FFEB7FE79038E07E0F3801C0FC4848487E38070070A7B7FCA23907007007B03A7FE3FE3F F0A22420809F26>I34 D<127012F812FCA2127C120CA31218A21238123012601240060E7C9F0D >39 D<136013C0EA0180EA03005A12065A121C12181238A212301270A31260A212E0AC1260A212 70A312301238A21218121C120C7E12077EEA0180EA00C013600B2E7DA112>I<12C012607E7E12 1C120C7E12077E1380A2120113C0A31200A213E0AC13C0A21201A313801203A213005A12065A12 1C12185A5A5A0B2E7DA112>I<1306AFB612F0A2D80006C7FCAF1C207D9A23>43 D<127012F812FCA2127C120CA31218A21238123012601240060E7C840D>II<127012F8A3127005057C840D>I<1303A213071306A2130E130CA2131C1318A213381330A2 13701360A213E013C0A212011380A312031300A25A1206A2120E120CA2121C1218A212381230A2 12701260A212E05AA2102D7DA117>IIIII<130EA2131E133EA2136E13EE13CEEA018E120313 0E1206120E120C121812381230126012E0B512F0A238000E00A7EBFFE0A2141E7F9D17>II<137CEA01FEEA07 83380E0380EA0C07121C3838030090C7FC12781270A2EAF3F8EAF7FEEAFC0E487EEB0380A200F0 13C0A51270A214801238EB0700121CEA0E1EEA07FCEA01F0121F7E9D17>I<1260387FFFC0A214 80EA600138C003001306A2C65A5BA25B5BA213E05B1201A3485AA41207A76CC7FC121F7D9D17> II I<127012F8A312701200AA127012F8A3127005147C930D>I<127012F8A312701200AA127012F8 A312781218A41230A21260A21240051D7C930D>I<007FB512E0B612F0C9FCA8B612F06C14E01C 0C7D9023>61 D63 D65 DI<90381FC04090387F F0C03801F8393803C00D38078007380F0003121E003E1301123C127C1400127812F81500A80078 14C0127CA2123C003EEB0180121E6CEB0300EA07803803C00E3801F81C38007FF0EB1FC01A217D 9F21>IIII<90380FC02090387FF8603901F81CE03803E0063807800338 0F0001121E14005A127C1560127812F81500A6EC7FFCA20078EB01E0127CA2123C7EA27E380780 03EA03E03901F80E6039007FFC2090380FE0001E217D9F24>I<39FFF8FFF8A23907800F00AC90 B5FCA2EB800FAD39FFF8FFF8A21D1F7E9E22>II< EAFFFEA2EA0780B11406A4140EA2140C141C143C14FCB5FCA2171F7E9E1C>76 DI<39FF807FF813C0 0007EB07809038E00300A2EA06F0A21378133CA2131EA2130FA2EB078314C31303EB01E3A2EB00 F3A2147BA2143F80A280A2000F7FEAFFF0801D1F7E9E22>III82 D<3807E080EA0FF9EA1C1FEA300FEA7007EA600312E01301A36CC7FCA21278127FEA3FF0EA1FFC 6C7EEA03FF38001F801307EB03C0A2130112C0A400E01380EAF00338F80700EAFE0EEACFFCEA81 F812217D9F19>I<007FB512E0A238780F010070130000601460A200E0147000C01430A4000014 00B23807FFFEA21C1F7E9E21>I<39FFFC7FF8A23907800780EC0300B3A300031302EBC006A200 015B6C6C5AEB7830EB3FE0EB0FC01D207E9E22>I<39FFF007FEA2390F8001F090C712E06C6C13 C0A2EBC00100031480A2EBE00300011400A23800F006A3EB780CA36D5AA36D5AA2EB1F70EB0F60 A214E06D5AA26D5AA31F207F9E22>I<3BFFF07FF83FF0A23B0F0007800F80EE0300A23A07800F C006A3913819E00ED803C0140CA214393A01E030F018A33A00F0607830A3ECE07C903978C03C60 A390393D801EC0A390383F000F6D5CA3010E6DC7FCA32C207F9E2F>I<397FF83FF8A23907C00F 800003EB06003801E00EEBF00C00005BEB7838EB7C30EB3C70EB3E60EB1EC0130F5C1307808013 0DEB1DF0EB18F8EB3878EB307CEB603CEBE01EEBC01F48487E0003EB0780010013C0EA0F8039FF E01FFEA21F1F7F9E22>I92 D97 D<120E12FEA2120EA9133FEB FF80380FC3C0EB00E0000E13F014701478A7147014F0120FEB01E0EBC3C0380CFF80EB3E001520 7F9F19>IIII<133C13FEEA01CFEA038F1306EA0700A7EAFFF0A2EA0700B0EA7FF0A21020 809F0E>II<120E12FEA2120EA9133E13FF380FC380EB01C0 A2120EAD38FFE7FCA216207F9F19>I<121C121E123E121E121CC7FCA6120E127EA2120EAFEAFF C0A20A1F809E0C>I<13E0EA01F0A3EA00E01300A61370EA07F0A212001370B3A21260EAF0E0EA F1C0EA7F80EA3E000C28829E0E>I<120E12FEA2120EA9EB1FF0A2EB0F80EB0E00130C5B5B1370 13F0EA0FF81338EA0E1C131E130E7F1480130314C038FFCFF8A215207F9F18>I<120E12FEA212 0EB3A9EAFFE0A20B20809F0C>I<390E3F03F039FEFF8FF839FFC1DC1C390F80F80EEB00F0000E 13E0AD3AFFE7FE7FE0A223147F9326>IIII<3803E180EA0FF9EA1E1FEA3C0712781303127012F0A6127012781307EA3C0FEA 1E1FEA0FF3EA03E3EA0003A7EB3FF8A2151D7E9318>III<1206A4120EA2121E123E EAFFF8A2EA0E00AA1318A5EA073013E0EA03C00D1C7F9B12>I<380E01C0EAFE1FA2EA0E01AC13 03A2EA070FEBFDFCEA01F116147F9319>I<38FF87F8A2381E01E0000E13C01480A238070300A3 EA0386A2138EEA01CCA213FC6C5AA21370A315147F9318>I<39FF9FF3FCA2391C0780F01560EC C0E0D80E0F13C0130C14E00007EBE180EB186114713903987300EBB033A2143F3801F03EEBE01E A20000131CEBC00C1E147F9321>I<387FC7FCA2380703E0148038038300EA01C7EA00EE13EC13 781338133C137C13EEEA01C7138738030380380701C0000F13E038FF87FEA21714809318>I<38 FF87F8A2381E01E0000E13C01480A238070300A3EA0386A2138EEA01CCA213FC6C5AA21370A313 60A35B12F0EAF18012F3007FC7FC123C151D7F9318>III E /Fh 29 122 df<48B4FC000F13E0383E03F8007813FCEA7E01 00FF13FEA5387E03FC1200EB07F0EB0FE01480EB1F00131E5BA25BA21370A690C7FCA61370EA01 FC487EA56C5AEA0070172A7CA920>63 D 70 D<91387FE003903903FFFC0F011FEBFF1F90397FF00FFF9038FF8001D803FEC7FC48488048 48804980485A003F815B007F81A3484891C7FCA90203B512F8A2EA7FC0DA00011300A2123F7F12 1F6C7E7F6C7E6C6C5B3800FF8090387FF00F011FB5123F0103EBFC0F9039007FE0032D297CA836 >I73 D77 D<90387F80603903FFF0E0000F13FF38 1F807F383F001F003E1307007E1303127C00FC1301A214007E7E6D130013F8EBFF806C13F814FE 6C7F6C14C07E6C14E0000114F0EA003F010113F8EB001F1407A200E013031401A37E15F06C1303 6C14E0B413079038E01FC090B5120000E05B38C01FF01D297CA826>83 D87 D<48B47E000F13F0381F 81FC486C7E147FA2EC3F80A2EA0F00C7FCA2EB0FFF90B5FC3807FC3FEA1FE0EA3F80127F130012 FEA3147F7E6CEBFFC0393F83DFFC380FFF0F3801FC031E1B7E9A21>97 D99 DIII<9038FF81F00003EB E7FC390FC1FE7C391F80FCFC003FEBFE7C9038007E3848EB7F00A66C137EEB80FE001F5B380FC1 F8381FFFE0001813800038C8FC123CA2123E383FFFF814FF6C14C06C14E06C14F0121F397E0007 F8007C13015A1400A36C1301007EEB03F06CEB07E0390FC01F803903FFFE0038007FF01E287E9A 22>II<1207EA1FC013E0123FA3121F13C0EA0700C7FCA7EAFFE0A3 120FB3A3EAFFFEA30F2B7DAA14>I107 DI<3BFFC07F800FF0 903AC1FFE03FFC903AC783F0F07E3B0FCE03F9C07F903ADC01FB803F01F8D9FF00138001F05BA3 01E05BAF3CFFFE1FFFC3FFF8A3351B7D9A3A>I<38FFC07F9038C1FFC09038C787E0390FCE07F0 9038DC03F813F813F0A313E0AF3AFFFE3FFF80A3211B7D9A26>II<38FFE1FE9038E7FF809038FE07E039 0FF803F8496C7E01E07F140081A2ED7F80A9EDFF00A25DEBF0014A5A01F85B9038FE0FE09038EF FF80D9E1FCC7FC01E0C8FCA9EAFFFEA321277E9A26>I<38FFC3F0EBCFFCEBDC7E380FD8FF13F8 5BA3EBE03C1400AFB5FCA3181B7E9A1C>114 D<3803FE30380FFFF0EA3E03EA7800127000F013 70A27E6C1300EAFFE013FE387FFFC06C13E06C13F0000713F8C613FC1303130000E0137C143C7E A26C13787E38FF01F038F7FFC000C11300161B7E9A1B>I<1370A413F0A312011203A21207381F FFF0B5FCA23807F000AD1438A73803F870000113F03800FFE0EB1F8015267FA51B>I<39FFE03F F8A3000F1303B11407A2140F0007131F3A03F03BFF803801FFF338003FC3211B7D9A26>I<3AFF FE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C485AA2D97F07C7FCA2148FEB 3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5A211B7F9A24>I<3BFFFE7FFC0FFEA33B0FE007 E000E03B07F003F001C0A29039F807F80300031680A23B01FC0EFC0700A2D9FE1E5B000090381C 7E0EA29039FF383F1E017F141C0278133C90393FF01FB8A216F86D486C5AA26D486C5AA36D486C 5AA22F1B7F9A32>I<39FFFC0FFFA33907F003C06C6C485AEA01FC6C6C48C7FCEBFF1E6D5AEB3F F86D5A130FA2130780497E497E131EEB3C7F496C7E496C7ED801E07FEBC00F00036D7E3AFFF01F FF80A3211B7F9A24>I<3AFFFE03FF80A33A07F0007000A26D13F000035CEBFC0100015CA26C6C 485AA2D97F07C7FCA2148FEB3F8E14DEEB1FDCA2EB0FF8A36D5AA26D5AA26D5AA2495AA2EA3807 007C90C8FCEAFE0F130E131E5BEA7C78EA3FE0EA0FC021277F9A24>I E /Fi 18 119 df<127012F812FCA2127C120CA41218A21230A212601240060F7C840E>44 D49 DI56 DI77 D<39FF800FFF13C00007EB01F89038E000607F12061378A27F133E13 1E7FA2EB078014C01303EB01E0A2EB00F01478A2143CA2141E140FA2EC07E0A214031401A2381F 8000EAFFF0156020227EA125>I<3803F020380FFC60381C0EE0EA3803EA7001A2EAE000A21460 A36C1300A21278127FEA3FF0EA1FFE6C7E0003138038003FC0EB07E01301EB00F0A2147012C0A4 6C136014E06C13C0EAF80138EF038038C7FF00EA81FC14247DA21B>83 D97 D<120E12FEA2121E120EAAEB1F80EB7FE0380FC0F0EB0078000E1338143C 141C141EA7141C143C000F1338EB8070EBC1F0380C7FC0EB1F0017237FA21B>II101 D<121C121E123E121E121CC7FC A8120E12FEA2121E120EAFEAFFC0A20A227FA10E>105 D<390E1FC07F3AFE7FE1FF809039C0F3 03C03A1F807E01E0390F003C00000E1338AE3AFFE3FF8FFEA227157F942A>109 D<380E1F8038FE7FC038FFC1E0381F80F0380F0070120EAE38FFE7FFA218157F941B>II114 D<38FFC3FEA2381E00F8000E1360A26C13C0A338038180 A213C300011300A2EA00E6A3137CA31338A217157F941A>118 D E /Fj 17 124 df73 D77 D79 DI97 D 99 D101 DI<15F090387F03F83901FFCF1C3803C1FC390780F818390F007800 48137C001E133C003E133EA7001E133C001F137C6C13786C6C5A380FC1E0380DFFC0D81C7FC7FC 0018C8FCA2121CA2121E380FFFF814FF6C14804814C0391E0007E00038EB01F048EB0070157848 1438A500701470007814F06CEB01E06CEB03C03907C01F003801FFFC38003FE01E2F7E9F21>I< 1207EA0F80121FA2120FEA0700C7FCABEA078012FFA2120F1207B3A6EA0FC0EAFFF8A20D307EAF 12>105 D108 D<380781FE39FF87FF8090388E07C0390F9803E03807B0019038E000F05BA35BB3486C487E3AFF FC1FFF80A2211F7E9E25>110 D<380783E038FF8FF8EB9C7CEA0FB0EA07F0EBE038EBC000A35B B3487EEAFFFEA2161F7E9E19>114 D<3801FC10380FFF30381E03F0EA38004813705A1430A37E 6C1300127EEA3FF06CB4FC6C1380000313E038003FF0EB03F8EB007800C0133CA2141C7EA27E14 186C13386C137038EF01E038C3FFC03880FE00161F7E9E1A>I<13C0A51201A31203A21207120F 121FB512E0A23803C000B01430A83801E060A23800F0C0EB7F80EB1F00142C7FAB19>II123 D E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin %%EndSetup %%Page: 1 1 bop 457 227 a Fj(Message)21 b(P)n(assing)g(In)n(terface)i({)f(Outline)869 354 y Fi(Marc)16 b(Snir)772 456 y(No)o(v)o(em)o(b)q(er)d(28,)k(1992)75 643 y Fh(In)n(tro)r(duction)75 745 y Fg(The)f(purp)q(ose)h(of)f(this)g(do)q (cumen)o(t)h(is)f(to)g(pro)o(vide)g(a)g(framew)o(ork)f(that)g(w)o(e)h(can)g (use)h(to)e(discuss)i(issues)75 801 y(in)i(the)e(de\014nition)j(of)d(MPI,)h (to)f(list)i(questions)f(w)o(e)f(need)i(to)e(answ)o(er,)h(and)g(prop)q(ose)f (some)h(answ)o(ers.)75 857 y(I)g(plan)g(to)e(use)i(this)g(do)q(cumen)o(t)f (as)g(a)g(framew)o(ork)f(for)h(the)g(discussions)i(in)f(the)f(p)q(oin)o (t-to-p)q(oin)o(t)h(sub-)75 914 y(committee;)d(I)g(broadcast)g(it)g(to)g(the) g(en)o(tire)g(committee,)g(b)q(ecause)h(man)o(y)f(issues)h(are)f(relev)m(an)o (t)g(to)g(the)75 970 y(other)g(sub)q(committees.)75 1113 y Fh(Goals)131 1214 y Fg(1.)22 b(Design)17 b(an)h(application)h(programming)d (in)o(terface)i(\(not)e(necessarily)j(a)e(target)f(for)h(compilers)189 1271 y(or)d(a)h(system)g(implemen)o(tation)h(library\))131 1363 y(2.)22 b(Allo)o(w)15 b(e\016cien)o(t)h(comm)o(unication:)21 b(Av)o(oid)16 b(memory)e(to)h(memory)g(cop)o(ying)g(and)h(allo)o(w)f(o)o(v)o (erlap)189 1420 y(of)e(computation)g(and)h(comm)o(unication)g(and)g(o\017oad) f(to)g(comm)o(unication)h(copro)q(cessor,)g(where)189 1476 y(a)o(v)m(ailable)131 1569 y(3.)22 b(Allo)o(w)15 b(\(but)g(no)h(mandate\))e (extensions)i(for)f(use)g(in)h(heterogeneous)f(en)o(vironmen)o(t.)131 1661 y(4.)22 b(Allo)o(w)15 b(con)o(v)o(enien)o(t)h(C,)f(C++,)g(F77)f(and)i (F90)e(bindings)j(for)e(in)o(terface.)131 1754 y(5.)22 b(Pro)o(vide)15 b(a)g(reliable)i(comm)o(unication)f(in)o(terface:)21 b(User)15 b(need)h(not)f(cop)q(e)h(with)f(comm)o(unication)189 1810 y(failures.)21 b(Suc)o(h)15 b(failures)i(are)d(dealt)i(b)o(y)f(the)h(underlying)h(comm)o (unication)f(subsystem.)131 1902 y(6.)22 b(F)l(o)q(cus)15 b(on)g(a)g(prop)q (osal)h(that)e(can)h(b)q(e)h(agreed)f(up)q(on)h(in)g(6)f(mon)o(ths.)131 1995 y(7.)22 b(De\014ne)10 b(an)g(in)o(terface)h(that)e(is)i(not)f(to)q(o)f (di\013eren)o(t)i(from)e(curren)o(t)h(practice)h(\(PVM/Express/P)o(armacs\)) 131 2087 y(8.)22 b(De\014ne)13 b(an)g(in)o(terface)f(that)g(can)h(b)q(e)h (quic)o(kly)g(implemen)o(ted)g(on)f(man)o(y)f(v)o(endors)g(platforms,)h(with) 189 2144 y(no)i(signi\014can)o(t)h(c)o(hanges)f(in)h(the)f(underlying)j(comm) o(unication)d(and)h(system)f(soft)o(w)o(are.)146 2245 y(Do)f(w)o(e)h(agree)g (on)g(this?)75 2388 y Fh(F)-6 b(ramew)n(ork)75 2489 y Fg(I)11 b(prop)q(ose)g(to)g(consider)g(comm)o(unication)h(op)q(erations)f(as)g (consisting)g(of)g(the)g(follo)o(wing)g(sub)q(op)q(erations:)75 2591 y Ff(INIT\(op)q(eration,)19 b(params,)d(handle\))24 b Fg(Pro)q(cess)15 b(pro)o(vides)f(all)i(relev)m(an)o(t)e(parameters)g(for)g (its)g(par-)189 2647 y(ticipation)19 b(in)f(the)g(comm)o(unication)g(op)q (eration)g(\(data)f(bu\013er,)h(t)o(yp)q(e,)g(participan)o(ts,)g(etc.\).)27 b(A)189 2704 y(handle)16 b(is)g(created)f(that)g(iden)o(ti\014es)h(the)g(op)q (eration.)964 2828 y(1)p eop %%Page: 2 2 bop 75 45 a Ff(ST)l(AR)l(T\(handle\))24 b Fg(The)16 b(comm)o(unication)g(op)q (eration)f(is)h(started)75 139 y Ff(A)-6 b(W)g(AIT\(handle\))24 b Fg(The)15 b(comm)o(unication)h(op)q(eration)g(is)g(completed.)75 232 y Ff(FREE\(handle\))25 b Fg(The)16 b(handle,)g(and)f(asso)q(ciated)g (resources)h(are)f(freed.)75 337 y(Correct)g(in)o(v)o(o)q(cation)h(of)g (these)g(sub)q(op)q(erations)g(is)h(a)e(sequence)i(of)e(the)h(form)f(1\(23\)) 1539 321 y Fe(\003)1558 337 y Fg(4.)21 b(I.e.,)15 b(a)h(handle)75 394 y(need)g(b)q(e)g(created)g(b)q(efore)g(comm)o(unication)g(o)q(ccurs;)g (it)f(can)h(b)q(e)g(reused)g(only)g(after)f(the)g(previous)i(use)75 450 y(has)12 b(completed;)h(and)f(it)g(need)h(to)e(b)q(e)h(freed)g(ev)o(en)o (tually)h(\(of)e(course,)h(one)g(can)g(assume)f(that)g(all)i(handles)75 507 y(are)i(freed)g(at)g(program)f(termination,)h(b)o(y)g(default\).)146 563 y(The)f(c)o(hoice)i(of)e(these)h(sub)q(op)q(erations)g(is)h(somewhat)d (arbitrary)l(.)20 b(The)15 b(justi\014cation)g(is)g(that)f(com-)75 619 y(m)o(unication)k(ma)o(y)f(b)q(e)h(a)g(length)o(y)g(pro)q(cess)f(one)h (desires)h(to)d(o)o(v)o(erlap)i(with)g(computation,)f(hence)i(the)75 676 y(separation)g(of)g(2)h(and)f(3;)i(also)e(comm)o(unication)i(setup)e(ma)o (y)g(in)o(v)o(olv)o(e)h(a)f(signi\014can)o(t)i(o)o(v)o(erhead)e(one)75 732 y(desires)i(to)f(amortize)h(o)o(v)o(er)e(man)o(y)h(successiv)o(e)i(comm)o (unications)f(with)g(the)g(same)f(c)o(haracteristics,)75 789 y(hence)c(the)g(separation)f(of)f(1)h(and)h(4)f(from)f(2)h(and)g(3.)75 932 y Fh(Issues)75 1035 y Fd(Whic)n(h)20 b(op)r(erations?)75 1121 y Fg(SEND)15 b(and)g(RECEIVE)g(for)g(p)q(oin)o(t)g(to)f(p)q(oin)o(t)i (comm)o(unication.)k(Longer)15 b(list)h(for)e(collectiv)o(e)j(comm)o(u-)75 1177 y(nication)g(\(note:)k(SEND)16 b(and)g(RECEIVE)g(are)g(particular)g (case)g(of)g(broadcast,)f(for)g(group)g(of)h(size)h(2;)75 1234 y(this)i(observ)m(ation)g(can)g(b)q(e)g(used)h(to)e(c)o(hec)o(k)h(if)g (de\014nition)i(of)d(collectiv)o(e)i(comm)o(unication)g(seman)o(tics)75 1290 y(are)15 b(consisten)o(t)g(with)h(de\014nition)h(of)e(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication\).)146 1347 y(One)g(can)f(argue)g(for)f(a)h(SEND) p 684 1347 14 2 v 17 w(RECV)g(\(or)f(EX)o(CHANGE\))h(op)q(eration.)146 1403 y(I)g(prop)q(ose)g(to)g(lea)o(v)o(e)g(out,)f(for)g(the)i(time)f(b)q (eing,)h(things)f(lik)o(e)i(remote)d(pro)q(cedure)i(calls,)g(monitors,)75 1460 y(atomic)h(op)q(erations,)i(semaphores,)e(activ)o(e)h(messages,)g (message)f(handlers,)i(etc.)27 b(This,)19 b(in)f(order)g(to)75 1516 y(fo)q(cus)f(initially)j(on)c(a)h(smaller)g(set)g(of)f(issues,)i(with)f (less)g(dep)q(endencie)q(s)i(on)e(the)g(underlying)i(pro)q(cess)75 1572 y(mo)q(del.)146 1629 y(Opinions?)75 1750 y Fd(What)g(calls)h(are)e(made) h(a)n(v)m(ailable)h(to)f(the)f(user?)75 1836 y Fg(Options:)75 1941 y Ff(\(1+2+3+4\))23 b Fg(blo)q(c)o(king)16 b(send/receiv)o(e)75 2035 y Ff(\(1+2\))i(\(3+4\))23 b Fg(non)o(blo)q(c)o(king)17 b(send/receiv)o(e,)f(follo)o(w)o(ed)f(b)o(y)h(w)o(ait)75 2128 y Ff(\(1\))i(\(2+3\))g(\(4\))24 b Fg(c)o(hannel)16 b(creation;)f(blo)q(c)o (king)i(c)o(hannel)f(send/receiv)o(e;)g(c)o(hannel)h(cancelation)75 2222 y Ff(\(1\))h(\(2\))g(\(3\))h(\(4\))k Fg(c)o(hannel)17 b(creation;)e(non)o(blo)q(c)o(king)h(send/receiv)o(e;)g(w)o(ait;)f(c)o (hannel)h(destruction)146 2327 y(An)o(y)f(more)g(needed?)22 b(An)o(y)15 b(less?)75 2448 y Fd(When)j(is)i(a)f(comm)n(unication)h(op)r (eration)e(completed?)75 2534 y Fg(The)12 b(main)f(distinction)j(is)e(b)q(et) o(w)o(een)f Fc(lo)n(c)n(al)g Fg(termination)g(and)h Fc(glob)n(al)f Fg(termination.)19 b(A)11 b(send)h(terminates)75 2591 y(lo)q(cally)23 b(when)f(data)f(has)h(b)q(een)h(copied)g(out)e(of)g(the)h(sender)g(bu\013er)g (\(formally)l(,)h(when)f(the)f(sender)75 2647 y(cannot)e(a\013ect)g(an)o (ymore)g(the)h(outcome)g(of)f(the)h(send)g(op)q(eration\).)33 b(It)20 b(terminates)g(globally)g(when)75 2704 y(data)d(has)h(b)q(een)h (receiv)o(ed)g(b)o(y)f(the)f(receiv)o(er)i(\(formally)l(,)f(when)h(receiv)o (er)f(has)g(terminated)g(executing)964 2828 y(2)p eop %%Page: 3 3 bop 75 45 a Fg(an)o(y)15 b(op)q(eration)h(that)f(logically)j(precedes)f(the)f (execution)g(of)g(the)g(receiv)o(e\).)22 b(The)16 b(distinction)h(has)f(no)75 102 y(meaning)g(for)e(a)h(receiv)o(e)h({)f(since)i(lo)q(cal)f(termination)g (of)e(a)h(receiv)o(e)h(implies)i(global)d(termination.)146 158 y(Choices:)131 252 y(1.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g(lo)q (cal.)131 346 y(2.)22 b(T)l(ermination)e(is)g(either)g(lo)q(cal)h(or)e (global,)i(at)e(user)g(c)o(hoice)i(\(e.g.,)e(user)g(can)h(c)o(ho)q(ose)g(b)q (et)o(w)o(een)189 402 y(sync)o(hronous)15 b(and)g(async)o(hronous)g(comm)o (unication\))131 496 y(3.)22 b(T)l(ermination)16 b(is)f(alw)o(a)o(ys)g (global.)146 590 y(The)21 b(curren)o(t)f(MPI)h(prop)q(osal)g(c)o(ho)q(oses)g (2.)37 b(If)21 b(w)o(e)g(stic)o(k)g(to)f(this)h(c)o(hoice)h(then)f(I)g (suggest)g(that)75 646 y(termination)h(mo)q(de)g(\(global)g(or)f(lo)q(cal\))h (is)g(part)f(of)g(the)h(op)q(eration)f(parameters)g(that)g(are)g(set)g(up)75 703 y(in)g(sub)q(op)q(eration)g(1.)34 b(F)l(urthermore,)21 b(I)f(w)o(ould)h(suggest)e(to)h(o\013er)f(the)h(same)g(option)h(for)e (collectiv)o(e)75 759 y(comm)o(unications.)146 816 y(Another)13 b(issue)h(is)g(whether)g(termination)g(mo)q(de)f(need)h(b)q(e)h(consisten)o (t)e(for)g(all)h(participan)o(ts.)20 b(Cur-)75 872 y(ren)o(t)c(MPI)g(prop)q (osal)g(requires)h(that)e(a)g(sync)o(hronous)h(send)h(b)q(e)g(matc)o(hed)e(b) o(y)h(a)g(sync)o(hronous)g(receiv)o(e)75 928 y(\(either)i(all)h(participan)o (ts)f(use)g(lo)q(cal)h(completion)g(or)e(all)i(use)f(global)g(completion\).) 29 b(There)18 b(seem)g(to)75 985 y(b)q(e)d(no)g(comp)q(elling)i(logical)f (reason)f(for)f(this)h(restriction,)g(and)g(I)g(am)g(not)f(sure)h(it)g(has)g (signi\014can)o(t)g(im-)75 1041 y(plemen)o(tation)i(adv)m(an)o(tages.)k(The) 16 b(alternativ)o(e)g(is)g(to)f(allo)o(w)h(eac)o(h)g(participan)o(t)g(in)h(a) e(comm)o(unication)75 1098 y(op)q(eration)h(to)g(c)o(ho)q(ose)g(either)h(lo)q (cal)h(or)e(global)h(termination)f(\(this)h(b)q(ecomes)g(more)e(signi\014can) o(t)j(for)d(a)75 1154 y(collectiv)o(e)i(op)q(eration,)e(where)g(the)h(c)o (hoice)g(is)g(meaningful)g(at)f(more)f(than)i(one)f(pro)q(cess\).)146 1211 y(I)i(assume)h(lo)q(cal)g(termination)g(for)f(sub)q(op)q(eration)h(1)f ({)g(no)h(sync)o(hronization)g(with)g(another)f(pro-)75 1267 y(cessor)e(is)h(required)g(for)f(this)g(phase.)146 1324 y(Opinions?)146 1380 y(Should)h(w)o(e)f(c)o(ho)q(ose)g(1)g(or)g(3,)f(rather)h(than)g(2?)146 1437 y(Should)g(w)o(e)f(use)g(another)g(mec)o(hanism)g(to)g(c)o(ho)q(ose)g (lo)q(cal)h(and)f(global)h(termination)f(\(e.g.,)f(a)g(global)75 1493 y(\015ag\)?)146 1549 y(Should)g(w)o(e)e(require)h(the)g(c)o(hoice)h(of)e (\\sync)o(hronous")g(to)g(b)q(e)h(done)g(consisten)o(tly)h(b)q(e)f(all)h (participan)o(ts)75 1606 y(in)j(a)f(comm)o(unication?)75 1728 y Fd(When)j(do)r(es)g(a)h(call)h(return?)75 1813 y Fg(The)14 b(ob)o(vious)f(c)o(hoice)i(is:)k(\\A)13 b(call)i(return)e(when)h(the)f (corresp)q(onding)i(sub)q(op)q(eration\(s\))e(terminated".)75 1870 y(An)21 b(alternativ)o(e)f(is)h(to)f(allo)o(w)h(for)f(unsuccessful)i (return)f(as)f(w)o(ell.)37 b(E.g.,)20 b(send)h(can)g(return)f(unsuc-)75 1926 y(cessfully)l(,)d(if)f(the)f(system)g(cannot)g(accept)g(the)h(message,)e (for)h(whatev)o(er)g(reasons;)f(or)h(receiv)o(e)h(returns)75 1983 y(unsuccessfully)f(if)e(the)g(receiv)o(e)h(cannot)e(execute)i(within)g (a)e(set)h(time;)g(and)g(so)g(on.)19 b(This)13 b(is)g(the)g(curren)o(t)75 2039 y(MPI)i(de\014nition.)146 2096 y(My)j(p)q(ersonal)i(c)o(hoice)g(is)g (against)e(unsuccessful)j(return.)32 b(This)20 b(is)f(more)g(adequate)g(for)f (system)75 2152 y(programming,)f(rather)h(than)f(application)j(programming;)e (it)g(in)o(tro)q(duces)h(seman)o(tic)f(dep)q(endencies)75 2209 y(on)h(implemen)o(tation;)i(it)e(forces)f(the)h(user)f(to)g(c)o(hec)o(k)h (for)f(success)i(at)e(eac)o(h)g(op)q(eration,)i(rather)e(then)75 2265 y(deal)j(with)g(failure)g(of)f(comm)o(unication)h(using)g(an)f (exception)i(handling)g(mec)o(hanism.)36 b(Of)20 b(course,)75 2322 y(implemen)o(ters)f(are)f(w)o(elcome)g(to)f(pro)o(vide)i(a)e(reasonable) i(exception)f(handling)i(mec)o(hanism)f(for)e(the)75 2378 y(MPI)e(library)h ({)f(I)h(suggest)e(to)h(lea)o(v)o(e)g(op)q(en)h(the)f(de\014nition)i(of)e (suc)o(h)h(mec)o(hanism.)146 2434 y(Opinions?)75 2556 y Fd(Correctness)i (criteria)75 2642 y Fg(I)e(use)f(the)g(follo)o(wing)h(terminology:)964 2828 y(3)p eop %%Page: 4 4 bop 75 45 a Ff(safe)17 b(program)22 b Fg(A)c(program)f(that)h(should)h (execute)f(successfully)i(in)f(an)o(y)f(implemen)o(tation,)i(inde-)189 102 y(p)q(enden)o(tly)c(of)f(the)h(amoun)o(t)e(of)h(a)o(v)m(ailable)i(system) d(resources.)75 195 y Ff(unsafe)j(program)23 b Fg(A)d(program)f(that)g(will)j (succeed)f(or)f(fail,)i(dep)q(ending)g(on)e(implemen)o(tation)h(or)189 252 y(a)o(v)m(ailable)c(system)d(resources.)75 346 y Ff(erroneous)i(program) 22 b Fg(A)16 b(program)e(that)g(will)j(alw)o(a)o(ys)e(fail.)146 452 y(Examples)g(\(in)o(v)o(olving)h(t)o(w)o(o)e(pro)q(cessors)h(with)h(ids)g (1)f(and)g(2,)f(and)i(using)g(MPI)f(lik)o(e)h(syn)o(tax\))146 508 y(The)f(follo)o(wing)h(program)e(is)i(safe,)e(and)i(should)g(alw)o(a)o (ys)e(succeed.)75 659 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 715 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000\))147 772 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000\))75 884 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 941 y(CALL)g(CRECV\(mode=blocking,)e(buf=recbuf,)h(source=1,)h(type=0,)g (len=1000\))147 997 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000\))146 1148 y Fg(The)15 b(follo)o(wing)h(program)e(is)i (erroneous,)e(and)i(should)g(alw)o(a)o(ys)e(fail.)75 1310 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN)147 1367 y(CALL)f (CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g(len=1000\))147 1423 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recdbuf,)h(source=2,)g(type=0,)h (len=1000\))75 1536 y(ELSE)47 b(!)24 b(\(GETID\(\))f(==)g(2\))147 1593 y(CALL)g(CSEND\(mode=synchronous,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))147 1649 y(CALL)g(CRECV\(mode=synchronous,)e(buf=recbuf,)h (source=1,)h(type=0,)g(len=1000\))146 1755 y Fg(The)12 b(follo)o(wing)h (program)d(is)j(unsafe,)f(and)h(ma)o(y)e(succeed)i(or)f(fail,)h(dep)q(ending) h(on)e(implemen)o(tation.)75 1918 y Fb(IF)24 b(\(GETID\(\))e(==)i(1\))g(THEN) 147 1975 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=2,)h(type=0,)g (len=1000000\))147 2031 y(CALL)g(CRECV\(mode=blocking,)e(buf=recdbuf,)h (source=2,)h(type=0,)g(len=1000000\))75 2144 y(ELSE)g(!)h(\(GETID\(\))f(==)g (2\))147 2200 y(CALL)g(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h (type=0,)g(len=1000000\))147 2257 y(CALL)g(CRECV\(mode=blocking,)e (buf=recbuf,)h(source=1,)h(type=0,)g(len=1000000\))146 2363 y Fg(This)14 b(program)f(can)h(succeed)h(only)g(if)f(the)g(system)g(has)g (su\016cien)o(t)g(bu\013er)g(space)g(to)g(bu\013er)g(1)f(MgB)75 2420 y(of)i(data.)146 2476 y(The)j(strict)g(de\014nition)i(of)e(\\safe")f (needs)i(to)f(b)q(e)g(somewhat)g(temp)q(ered)h(in)g(practice.)29 b(A)18 b(correct)75 2532 y(sequen)o(tial)13 b(program)d(ma)o(y)h(fail)i(b)q (ecause)f(of)g(time)g(limits)h(or)e(other)g(resource)h(restrictions.)19 b(Lik)o(ewise,)13 b(a)75 2589 y(safe)h(parallel)i(program)d(ma)o(y)l(,)g (exceptionally)l(,)k(exceed)e(system)f(resources.)19 b(Still,)d(common)e (sense)h(can)75 2645 y(di\013eren)o(tiate)i(b)q(et)o(w)o(een)g(resources)f (that)g(are)h(\\practically)g(un)o(b)q(ounded",)h(i.e.)25 b(with)17 b(limits)g(that)f(are)75 2702 y(unlik)o(ely)k(to)d(b)q(e)h(normally)g(encoun) o(tered)g(b)o(y)g(correct)f(programs,)g(and)g(resources)h(that)f (\\practically)964 2828 y(4)p eop %%Page: 5 5 bop 75 45 a Fg(b)q(ounded",)23 b(i.e.)37 b(with)21 b(limits)h(one)f(encoun)o (ters)g(frequen)o(tly)g(with)g(correct)g(programs,)f(and)h(whic)o(h)75 102 y(often)13 b(impinges)i(on)f(program)e(p)q(ortabilit)o(y)l(.)20 b(A)14 b(formal)f(de\014nition)i(will)h(assume)d(that)g(some)g(resources)75 158 y(are)19 b(unlimited,)k(and)c(implemen)o(tations)i(will)g(striv)o(e)f(to) f(pro)o(vide)h(a)f(reasonable)h(appro)o(ximation)f(of)75 214 y(this)f(assumption.)26 b(F)l(urthermore,)17 b(the)g(user)h(should)g(b)q(e)g (able)g(to)f(query)g(and)g(p)q(ossibly)i(con)o(trol)e(an)o(y)75 271 y(implemen)o(tation)f(sp)q(eci\014c)i(resource)d(limitation)h(that)f (a\013ects)f(program)g(b)q(eha)o(vior.)146 327 y(One)k(need)g(to)f(allo)o(w)h (\(indeed,)h(one)f(cannot)f(disallo)o(w\))h(v)o(endors)g(to)e(supp)q(ort)i (some)f(unsafe)h(pro-)75 384 y(grams)c(as)h(w)o(ell.)21 b(The)15 b(requiremen)o(ts)h(there)f(should)h(b)q(e)g(that)143 478 y Fa(\017)23 b Fg(The)15 b(seman)o(tics)g(of)g(successful)i(program)d (execution)i(b)q(e)g(w)o(ell-de\014ned)143 571 y Fa(\017)23 b Fg(The)16 b(system)f(pro)o(vide)i(clear)f(information)h(and)f(p)q(ossibly)h (con)o(trol)f(on)g(those)g(parameters)f(that)189 628 y(determine)h(the)f(b)q (eha)o(vior)h(of)f(unsafe)g(programs)f(\(e.g.,)g(bu\013er)h(sizes\).)146 722 y(A)g(handle)j(is)e Fc(active)g Fg(\(at)f(a)g(pro)q(cess\))h(from)f(the)h (start)f(of)g(sub)q(op)q(eration)i(1)e(to)g(the)h(completion)h(of)75 778 y(sub)q(op)q(eration)f(4)g(for)f(this)h(handle.)22 b(A)15 b(comm)o(unication)i(is)f Fc(active)f Fg(\(at)g(a)g(pro)q(cess\))h(from)e (the)i(start)e(of)75 835 y(sub)q(op)q(eration)i(2)f(to)g(the)g(completion)h (of)f(sub)q(op)q(eration)h(3.)k(I)15 b(suggest)g(the)g(follo)o(wing)h(design) g(p)q(oin)o(t:)131 941 y(1.)22 b(The)13 b(n)o(um)o(b)q(er)g(of)g(activ)o(e)g (comm)o(unication)g(handles)i(p)q(er)e(pro)q(cess)g(is)h(\\practically)g(un)o (b)q(ounded".)131 1035 y(2.)22 b(The)d(n)o(um)o(b)q(er)g(of)f(initiated)j (comm)o(unications)e(p)q(er)h(pro)q(cessor)f(is)g(\\practically)h(un)o(b)q (ounded".)189 1091 y(\(Note)12 b(that)h(there)g(ma)o(y)g(b)q(e)g(only)h(one)g (activ)o(e)f(comm)o(unication)h(op)q(eration)f(p)q(er)h(activ)o(e)f (handle\).)131 1185 y(3.)22 b(The)16 b(amoun)o(t)g(of)g(a)o(v)m(ailable)i (system)d(bu\013er)i(space)f(is)h(limited)h({)e(a)g(safe)g(program)f(cannot)h (rely)189 1241 y(on)f(it.)146 1348 y(This)g(design)h(p)q(oin)o(t)g(has)f(the) h(follo)o(wing)f(implications:)131 1454 y(1.)22 b(A)11 b(t)o(yp)q(e)g(1)f(or) h(t)o(yp)q(e)g(2)g(sub)q(op)q(eration)g(will)i(alw)o(a)o(ys)d(complete,)i (indep)q(enden)o(tl)q(y)h(of)e(other)g(pro)q(cesses)189 1510 y(activities.)21 b(Of)15 b(course,)g(the)g(same)g(is)h(true)f(for)g(a)f(t)o (yp)q(e)i(4)f(sub)q(op)q(eration.)131 1604 y(2.)22 b(The)15 b(successful)h(completion)h(of)d(a)h(t)o(yp)q(e)g(3)g(sub)q(op)q(eration)h (ma)o(y)e(dep)q(end)j(on)e(the)g(participation)189 1661 y(of)h(other)g(pro)q (cesses.)24 b(A)17 b(send)g(ma)o(y)f(b)q(e)h(blo)q(c)o(k)o(ed)g(un)o(til)h(a) e(matc)o(hing)h(receiv)o(e)g(o)q(ccurs.)24 b(And,)17 b(of)189 1717 y(course,)h(a)g(receiv)o(e)h(ma)o(y)e(not)h(complete)g(un)o(til)i(a)d (matc)o(hing)h(send)h(o)q(ccurs.)29 b(\(This)18 b(extends)h(to)189 1774 y(collectiv)o(e)e(comm)o(unication.\))146 1880 y(T)l(o)e(this,)h(w)o(e)f (need)i(add)f(p)q(ositiv)o(e)g(requiremen)o(ts)g(for)f(progress)g(and/or)h (fairness.)21 b(Let's)16 b(sa)o(y)f(that)75 1936 y(a)k(comm)o(unication)h(is) f Fc(enable)n(d)f Fg(\(at)h(a)f(pro)q(cess\))h(if)h(a)f(matc)o(hing)g(comm)o (unication)h(is)f(activ)o(e)h(\(i.e.)31 b(a)75 1993 y(matc)o(hing)16 b(receiv)o(e)h(for)e(a)h(send)g(or)f(a)h(matc)o(hing)g(send)g(for)g(a)f (receiv)o(e\).)23 b(An)16 b(enabled)h(comm)o(unication)75 2049 y(ma)o(y)g(b)q(ecome)i(disabled)h(either)e(b)q(ecause)h(it)g(completes)f (successfully)i(\(the)e(comm)o(unication)h(o)q(ccurs)75 2106 y(and)d(sub)q(op)q(eration)h(3)e(terminates\))g(or)h(b)q(ecause)g(the)g(matc) o(hing)g(comm)o(unication)h(b)q(ecomes)f(inactiv)o(e)75 2162 y(\(e.g.,)e(a)h(receiv)o(e)h(that)e(matc)o(hes)h(one)h(send)g(ma)o(y)e(b)q(e) i(satis\014ed)g(b)o(y)f(another)g(send,)h(th)o(us)f(disabling)i(the)75 2219 y(\014rst)e(send\).)75 2325 y Ff(progress)21 b Fg(If)14 b(some)f(comm)o(unication)h(in)g(the)f(system)g(is)h(enabled)h(then)e(some)g (comm)o(unication)h(ev)o(en-)189 2381 y(tually)i(o)q(ccurs)f(in)h(the)g (system.)75 2475 y Ff(fairness)22 b Fg(An)c(activ)o(e)f(comm)o(unication)h (either)g(completes)g(successfully)h(or)d(b)q(ecomes)i(p)q(ermanen)o(tly)189 2532 y(disabled.)964 2828 y(5)p eop %%Page: 6 6 bop 146 45 a Fg(W)l(e)19 b(assume)g(in)h(these)f(de\014nitions)i(that)d(eac)o (h)h(pro)q(cess)g(in)o(v)o(ok)o(e)g(sub)q(op)q(erations)h(in)g(the)f(correct) 75 102 y(order,)c(i.e.)23 b(1\(23\))393 85 y Fe(\003)411 102 y Fg(4.)f(Of)16 b(course,)f(a)h(program)f(where)h(an)g(activ)o(e)g(comm)o (unication)g(b)q(ecomes)h(p)q(erma-)75 158 y(nen)o(tly)f(disabled)h(\(e.g.,)c (a)i(send)h(is)f(left)h(forev)o(er)e(with)i(no)f(matc)o(hing)g(receiv)o(e\))h (is)f(erroneous,)g(and)g(will)75 214 y(not)g(terminate)g(successfully)l(.)146 271 y(The)20 b(enforcemen)o(t)g(of)g(these)g(t)o(w)o(o)f(conditions)i (requires)g(a)f(\015o)o(w)g(con)o(trol)g(proto)q(col:)29 b(A)21 b(pro)q(cess)75 327 y(should)c(not)f(b)q(e)h(prev)o(en)o(ted)g(from)e (accepting)j(one)e(message)g(that)g(w)o(as)f(sen)o(t)h(b)q(ecause)i(its)e (bu\013ers)g(are)75 384 y(full)21 b(with)f(other)g(messages,)g(p)q(ossibly)h (sen)o(t)f(earlier.)34 b(When)21 b(a)e(receiv)o(e)i(matc)o(hes)e(sev)o(eral)h (p)q(ossible)75 440 y(messages)d(\(e.g.,)f(a)h(receiv)o(e)h(with)g(a)f (\\don't)f(care")h(sender)h(\014eld\))g(then)g(alternativ)o(e)g(options)f (should)75 497 y(b)q(e)f(considered)g(fairly)g(\(e.g.,)e(b)o(y)h(using)h(an)f (LR)o(U)h(p)q(olicy)h(when)e(considering)i(matc)o(hing)e(senders\).)146 553 y(A\014cionados)d(of)f(formalism)h(will)i(note)d(that)g(I)h(used)g(the)g (w)o(eak)o(est)f(de\014nition)j(of)d(fairness)h(\(ev)o(en)o(tual)75 610 y(fairness\),)j(rather)f(than)h(b)q(ounded)i(fairness)f(or)e(other)h (alternativ)o(es.)146 666 y(Examples:)146 723 y(The)g(follo)o(wing)h (programs)e(are)h(safe,)f(and)i(should)g(succeed:)75 823 y Fb(!)24 b(assume)f(only)g(two)h(processes,)e(numbered)h(1)h(and)f(2)75 936 y(IF)h(\(GETID\(\))e(==)i(1\))g(THEN)147 992 y(DO)f(I=0,)g(N)218 1048 y(CALL)g(CSEND\(mode=nonblocking,)e(buf=buf\(I\),)i(dest=2,)f(type=I,)h (len=1000\))147 1105 y(END)g(DO)75 1218 y(ELSE)g(!)h(\(GETID\(\))f(==)g(2\)) 147 1274 y(DO)g(I=0,)g(N)218 1331 y(CALL)g(CRECV\(mode=nonblocking,)e (buf=buf\(I\),)i(dest=2,)f(type=N-I,)h(len=1000\))147 1387 y(END)g(DO)146 1487 y Fg(Pro)q(cess)16 b(1)g(uses)h(non)o(blo)q(c)o(king)h (sends)f(to)f(send)h(a)f(sequence)i(of)e(messages)g(to)f(pro)q(cess)i(2;)g (pro)q(cess)75 1544 y(2)d(receiv)o(es)h(these)f(messages)f(in)i(the)f(rev)o (erse)g(order,)g(using)h(non)o(blo)q(c)o(king)g(receiv)o(es.)20 b(In)15 b(practice,)g(suc)o(h)75 1600 y(program)e(ma)o(y)g(fail)h(if)g(the)g (n)o(um)o(b)q(er)g(of)g(p)q(ending)h(messages)f(implied)i(b)o(y)d(this)i (program)d(is)i(larger)g(than)75 1657 y(the)20 b(system)g(limit.)36 b(But)21 b(the)f(system)g(limit)h(should)h(b)q(e)e(large)h(enough)f(as)g(to)g (mak)o(e)f(suc)o(h)i(failure)75 1713 y(extremely)16 b(unlik)o(ely)l(.)75 1813 y Fb(!)24 b(Assume)f(a)g(large)h(number)f(of)g(processes)75 1926 y(!)h(Consumer)e(code)75 1983 y(IF)i(\(GETID\(\))e(==)i(1\))g(THEN)123 2039 y(DO)f(I=)h(1,)f(INFOG\(\))194 2095 y(CALL)h(CRECV\(mode=blocking,)d (buf=buf\(I\),)h(source=I,)h(type=0,)g(len=1000\))123 2152 y(END)g(DO)75 2265 y(!)h(Producers)75 2321 y(ELSE)123 2378 y(CALL)f(CSEND\(mode=blocking,)e(buf=sendbuf,)h(dest=1,)h(type=0,)g (len=1000\))146 2478 y Fg(All)17 b(pro)q(cesses)f(send)g(a)f(message)h(to)f (pro)q(cess)h(1.)21 b(Pro)q(cess)15 b(1)h(receiv)o(es)g(these)g(messages)f (in)i(a)e(\014xed)75 2534 y(order.)31 b(A)20 b(\015o)o(w)e(con)o(trol)h (proto)q(col)g(is)h(needed)g(to)f(mak)o(e)f(sure)h(that)g(bu\013ers)g(at)f (pro)q(cess)i(1)f(will)h(not)75 2591 y(o)o(v)o(er\015o)o(w.)e(The)c(sending)h (pro)q(cesses)f(ma)o(y)g(b)q(e)g(dela)o(y)o(ed)h(un)o(til)g(the)f(receiv)o(e) g(pro)q(cess)g(is)h(ready)f(to)f(accept)75 2647 y(their)j(message.)146 2704 y(Consider)f(the)h(follo)o(wing)g(implemen)o(tation)g(alternativ)o(es)f (for)g(this)h(co)q(de:)964 2828 y(6)p eop %%Page: 7 7 bop 143 45 a Fa(\017)23 b Fg(Messages)18 b(are)h(sen)o(t)g(immediately)i(and) e(stored)g(in)h(the)f(ph)o(ysical)i(memory)d(of)h(the)g(receiving)189 102 y(no)q(de)g({)g(no)g(pro)o(vision)g(for)g(retry)l(.)31 b(This)19 b(is)h(a)e(bad)i(implemen)o(tation)g(since)g(ph)o(ysical)g(storage) 189 158 y(limits)c(are)f(lik)o(ely)i(to)e(b)q(e)g(encoun)o(tered.)143 252 y Fa(\017)23 b Fg(Messages)c(are)g(sen)o(t)h(immediately)h(and)f(stored)f (in)i(the)f(receiv)o(er)g(ph)o(ysical)i(memory)l(.)33 b(If)20 b(the)189 308 y(receiv)o(er)e(is)g(unable)h(to)f(receiv)o(e)g(the)g(message,) g(the)f(send)i(is)f(retried.)28 b(This)19 b(is)f(a)f(correct,)h(but)189 365 y(p)q(ossibly)e(ine\016cien)o(t)h(implemen)o(tation.)143 459 y Fa(\017)23 b Fg(A)16 b(\\request-to-send")h(proto)q(col)g(is)g(used;)h (the)f(receiv)o(er)g(allo)o(ws)g(the)g(send)g(to)g(pro)q(ceed)g(only)g(if)189 515 y(it)g(has)g(storage)f(for)h(the)g(message.)26 b(The)18 b(receiv)o(ers)g(need)g(to)e(store)h(information)g(on)h(p)q(ending,)189 571 y(unsatis\014ed)13 b(requests)g(to)f(send.)19 b(The)13 b(maxim)o(um)f(n)o(um)o(b)q(er)h(of)f(p)q(ending)j(requests)d(at)g(a)g(no)q (de)h(is)g(a)189 628 y(system)f(limit)i(that)e(should)i(b)q(e)f(set)f(large)h (enough)g(as)f(to)g(b)q(e)i(considered)g(\\practically)g(in\014nite".)146 722 y(Opinions?)146 778 y(Should)k(w)o(e)e(b)q(e)h(ev)o(en)h(more)e (restrictiv)o(e)h(and)g(require)g(that)g(safe)f(programs)f(run)i(correctly)l (,)h(ev)o(en)75 835 y(when)c(exceeding)h(\\v)o(ery)d(large")h(resource)h (limits?)21 b(\(Or)13 b(should)i(w)o(e)e(a)o(v)o(oid)g(to)f(burden)j(the)e (implemen-)75 891 y(tation)i(with)g(requiremen)o(ts)h(to)f(handle)h(what)f (most)f(lik)o(ely)j(are)e(pathologic)h(programs?\))146 948 y(Should)h(w)o(e)f(try)g(to)g(b)q(e)h(more)f(formal)g(in)h(de\014ning)h(when) f(some)f(of)g(the)h(\\unsafe")f(programs)f(will)75 1004 y(run)h(to)g (completions?)24 b(\(Can)16 b(w)o(e)g(do)g(it)g(without)g(b)q(eing)i(more)d (sp)q(eci\014c)j(ab)q(out)e(the)h(implemen)o(tation)75 1060 y(mo)q(del?\))146 1117 y(Should)f(w)o(e)f(require)h(fairness?)21 b(Should)c(w)o(e)d(ha)o(v)o(e)h(a)g(stronger)f(requiremen)o(t)i(of)f (fairness?)75 1260 y Fh(What)23 b(are)g(the)g(comm)n(unication)d(parameters?) 75 1363 y Fd(Message)e(data)i(\(bu\013er\))131 1449 y Fg(1.)i(scalar)15 b(\(b)o(yte/halfw)o(ord/w)o(ord/doublew)o(ord\),)e(presumably)j(coming)f (from)g(a)g(register)131 1543 y(2.)22 b(Con)o(tiguous)15 b(space)g(in)h (memory)f(\(initial)i(address/length\))131 1637 y(3.)22 b(Bu\013er)15 b(with)g(constan)o(t)g(stride)g(\(initial)i(address/size)f(of)f(eac)o(h)g (blo)q(c)o(k/stride/n)o(um)o(b)q(er)i(blo)q(c)o(ks\))131 1731 y(4.)22 b(A)15 b(t)o(yp)q(ed)g(message)g(\(of)g(a)f(t)o(yp)q(e)i(supp)q (orted)g(in)g(the)f(domain)g(language\).)131 1824 y(5.)22 b(A)d(union)i(of)d (sev)o(eral)i(messages)f(\(p)q(ossibly)i(sp)q(eci\014ed)g(b)o(y)e(an)h(arra)o (y)e(of)h(p)q(oin)o(ters)h(to)e(message)189 1881 y(descriptors\).)g(The)13 b(seman)o(tics)f(of)f(a)h(union)h(is)f(probably)h(conjunctiv)o(e)f(at)g(the)g (sender)g(end)h(\(send)189 1937 y(all\);)g(at)f(the)h(receiv)o(e)g(end,)h(it) f(ma)o(y)e(b)q(e)j(conjunctiv)o(e)f(\(receiv)o(e)g(all\))h(or)e(disjunctiv)o (e)i(\(receiv)o(e)f(an)o(y\).)146 2044 y(The)j(curren)o(t)g(MPI)f(prop)q (osal)i(has)e(2)h(and)g(3)g(\(2)f(is)h(really)h(a)f(particular)g(case)g(of)g (3\).)21 b(Do)15 b(w)o(e)h(w)o(an)o(t)75 2100 y(more?)k(In)c(particular,)g (do)f(w)o(e)g(w)o(an)o(t)f(t)o(yp)q(ed)h(messages?)20 b(Ho)o(w)15 b(general?)75 2222 y Fd(Groups)k(and)g(con)n(text)75 2307 y Fg(Messages)c(are)g(tagged)g(b)o(y)g(pro)q(cess)h(names)f(\(sender)h(and)f (receiv)o(er\))h(and)g(t)o(yp)q(e)f(names.)21 b(Both)15 b(names)75 2364 y(spaces)g(can)f(b)q(e)i(\015at)e(or)g(structured.)20 b(If)14 b(a)h(\015at)f(pro)q(cess)h(name)f(space)h(is)g(used,)g(then)g(a)f (pro)q(cess)h(names)75 2420 y(is)e(an)f(in)o(teger)g(\(from)f(0)h(to)f(n)o (um)o(b)q(er)p 687 2420 14 2 v 17 w(of)p 741 2420 V 16 w(pro)q(cesses-1,)i (for)e(C)h(think)o(ers\).)19 b(If)13 b(a)e(structured)i(pro)q(cess)f(name)75 2477 y(space)18 b(is)g(used)f(then)h(a)f(pro)q(cess)h(name)f(is)h(a)f(pair)h (of)f(the)g(form)g(\(group,)g(rank\).)26 b(Similarly)19 b(in)f(a)f(\015at)75 2533 y(t)o(yp)q(e)g(name)h(space,)g(a)f(t)o(yp)q(e)g(is)h(a)f(n)o(um)o(b)q (er)h(in)g(a)f(range;)h(in)g(a)f(structured)g(t)o(yp)q(e)h(name)f(space,)h (it)f(is)h(a)75 2590 y(pair)e(of)e(the)i(form)e(\(group,)g(lo)q(cal)p 659 2590 V 18 w(t)o(yp)q(e\).)146 2646 y(The)21 b(usefulness)h(of)f(groups)f (seem)h(clear)h(in)f(the)g(con)o(text)g(of)f(collectiv)o(e)j(op)q(erations,)f (where)f(a)75 2703 y(groupid)c(is)g(a)f(handle)h(to)f(sp)q(ecify)h(the)g(set) f(of)f(participan)o(ts)i(in)g(a)f(collectiv)o(e)i(comm)o(unication)f(\(Note:) 964 2828 y(7)p eop %%Page: 8 8 bop 75 45 a Fg(some)15 b(of)h(the)f(discussions)j(on)d(groups)g(seem)h(to)f (assume)h(that)f(a)g(group)g(is)i(alw)o(a)o(ys)e(represen)o(ted)h(b)o(y)f(a) 75 102 y(mem)o(b)q(ership)f(list)g(that)f(is)h(presen)o(t)f(at)f(eac)o(h)i (group)f(mem)o(b)q(er.)19 b(While)14 b(is)g(the)f(represen)o(tation)h(lik)o (ely)h(to)75 158 y(b)q(e)g(used)h(on)e(small)i(systems,)e(it)h(is)g(b)o(y)g (no)g(means)f(the)h(unique)h(p)q(ossible)h(one.)j(An)o(y)15 b(connected)g(graph)75 214 y(structure)h(can)h(b)q(e)g(used)g(to)f(k)o(eep)h (trac)o(k)f(of)g(group)g(mem)o(b)q(ership,)i(with)f(ob)o(vious)f(tradeo\013s) g(b)q(et)o(w)o(een)75 271 y(storage)h(and)h(p)q(erformance.)28 b(A)18 b(complete)g(list)h(at)e(eac)o(h)h(no)q(de)h(is)f(one)g(extreme)g(of)g (this,)g(namely)l(,)h(a)75 327 y(complete)g(graph.)28 b(Therefore,)19 b(a)e(group)h(handle)i(is)e(a)g(more)g(abstract)f(and)h(more)g(general)g (concept)75 384 y(than)d(a)g(list)h(of)f(group)g(mem)o(b)q(ers.\))146 440 y(The)h(collectiv)o(e)j(comm)o(unication)e(sub)q(committee)g(has)g(to)f (deal)h(with)g(mec)o(hanisms)g(for)f(creating)75 497 y(and)j(destro)o(ying)f (groups)g(\(for)g(the)g(record,)h(I)g(fa)o(v)o(or)e(dynamic)i(group)g (creation)f(b)q(oth)h(b)o(y)f(explicitly)75 553 y(listing)j(group)f(mem)o(b)q (ers)f(and)h(b)o(y)g(partitioning)h(an)e(existing)i(group.)33 b(I)20 b(do)g(not)f(fa)o(v)o(or)g(a)g(complex)75 610 y(stac)o(k)e(or)h(tree)g (mec)o(hanism)h(to)e(manage)h(groups)f(and)i(na)o(vigate)e(from)h(one)g (group)g(to)f(another)h({)g(at)75 666 y(least)d(not)g(in)h(the)f(basic)h(MPI) f(prop)q(osal\).)146 723 y(In)i(the)f(con)o(text)g(of)g(p)q(oin)o(t-to-p)q (oin)o(t)h(comm)o(unication,)g(the)f(issue)h(is)g(whether)g(w)o(e)f(w)o(an)o (t)f(to)h(use)g(a)75 779 y(structured)f(name)g(space)h(for)e(t)o(yp)q(e)i (and)f(pro)q(cess)h(argumen)o(ts)e(and,)h(if)h(so,)e(ho)o(w.)146 835 y(The)21 b(adv)m(an)o(tage)h(of)f(ha)o(ving)h(a)f(structured)h(name)g (space)f(for)h(pro)q(cesses)g(is)g(that)f(it)h(simpli\014es)75 892 y(com)o(bining)14 b(of)e(distinct)i(parallel)g(co)q(des)f(in)h(a)e(lo)q (osely)i(coupled)g(application:)20 b(Eac)o(h)13 b(functional)h(subset)75 948 y(can)j(use)h(message)f(passing)h(within)g(its)g(con)o(text,)f(with)g(no) h(c)o(hanges)f(in)h(the)f(lo)q(cal)i(co)q(de.)27 b(The)17 b(same)75 1005 y(concern)k(for)f(mo)q(dularit)o(y)h(militates)h(for)e(a)g(structured)g (t)o(yp)q(e)h(space.)36 b(This)21 b(do)q(es)g(not)g(necessarily)75 1061 y(implies)j(or)d(requires)h(a)f(stac)o(k)g(mec)o(hanism)h(for)f (managing)g(groups,)i(or)e(a)g(tree)g(structure)h(among)75 1118 y(groups.)146 1174 y(First,)14 b(t)o(w)o(o)g(remarks:)143 1268 y Fa(\017)23 b Fg(The)c(preexisting)h(group)f Fb(ALL)f Fg(that)g(includes)j(all)f(pro)q(cesses)g(pro)o(vides)f(a)g(global)g(con)o (text.)31 b(If)189 1324 y(all)17 b(comm)o(unications)f(use)h(this)f(con)o (text,)f(then)i(one)f(defaults)g(to)g(\015at)f(name)h(space.)23 b(With)16 b(the)189 1381 y(righ)o(t)g(syn)o(tax,)h(the)g(existence)h(of)f(a)g (structured)g(name)g(space)g(can)g(b)q(e)h(ignored)g(b)o(y)f(users)g(that)189 1437 y(prefer)e(a)g(\015at,)f(global)i(name)f(space.)143 1531 y Fa(\017)23 b Fg(A)c(structured)f(name)h(space)g(can)g(alw)o(a)o(ys)g(fak)o (ed,)g(with)g(no)g(c)o(hanges)g(in)g(the)g(comm)o(unication)189 1588 y(op)q(erations:)g(Replace)d(eac)o(h)e Fb(process)p 868 1588 15 2 v 16 w(id)g Fg(actual)h(parameter)e(b)o(y)h(a)g(a)g(reference)h(to) e(a)h(function)189 1644 y Fb(process\(rank,)22 b(group)p 646 1644 V 16 w(id\))p Fg(;)c(replace)h(eac)o(h)e Fb(type)h Fg(actual)f (parameter)g(b)o(y)h(a)f(reference)h(to)f(a)189 1701 y(function)f Fb(type\(local)p 610 1701 V 15 w(type,)24 b(group)p 889 1701 V 16 w(id\))p Fg(.)146 1794 y(F)l(or)16 b(simplicit)o(y)l(,)j(I)e(shall)h (assume)f(the)f(same)h(group)f(con)o(text)h(is)g(used)g(b)q(oth)g(for)f (relativ)o(e)i(pro)q(cess)75 1851 y(n)o(um)o(b)q(ering)h(and)g(for)f(relativ) o(e)h(t)o(yp)q(e)g(naming.)30 b(I)19 b(b)q(eliev)o(e)h(this)f(simpli\014es)i (the)d(discussion,)j(without)75 1907 y(in)o(tro)q(ducing)13 b(undue)g(restrictions.)20 b(A)12 b(con)o(text)f(for)g(a)h(comm)o(unication)g (op)q(eration)h(can)f(b)q(e)g(established)75 1964 y(in)k(t)o(w)o(o)e(w)o(a)o (ys:)131 2070 y(1.)22 b(The)f(curren)o(t)h(scop)q(e)g(\(curren)o(t)f(group\)) g(of)g(a)g(comm)o(unication)h(op)q(eration)g(is)g(implicit;)27 b(it)21 b(is)189 2126 y(established)16 b(b)o(y)g(the)f(last)g(executed)h(con) o(text)f(con)o(trolling)h(call)g(\()p Fb(MPI)p 1401 2126 V 17 w(PUSHC)e Fg(and)i Fb(MPI)p 1713 2126 V 16 w(POPC)f Fg(in)189 2183 y(the)e(curren)o(t)h(draft\).)k(Th)o(us,)c(an)o(y)f(pro)q(cess)h(n)o(um) o(b)q(er)h(is)f(understo)q(o)q(d)g(to)f(b)q(e)i(relativ)o(e)f(rank)f(in)i (the)189 2239 y(curren)o(t)g(group,)f(and)h(an)o(y)g(t)o(yp)q(e)g(name)g(is)h (understo)q(o)q(d)f(to)g(b)q(e)g(relativ)o(e)h(to)e(curren)o(t)h(group,)g (and)189 2296 y(do)q(es)j(not)f(con\015icts)i(with)f(t)o(yp)q(e)h(names)e (used)i(within)g(other)f(groups.)27 b(Of)18 b(course,)h(the)f(initial)189 2352 y(scop)q(e)d(is)g Fb(ALL)p Fg(,)e(and)i(the)g(system)f(defaults)h(to)f (a)g(\015at)g(name)h(space)g(if)g(scop)q(e)g(setting)f(commands)189 2409 y(are)h(nev)o(er)g(used.)131 2503 y(2.)22 b(The)g(con)o(text)g(of)f(a)h (comm)o(unication)h(op)q(erations)f(need)h(to)f(b)q(e)h(explicitly)i(sp)q (eci\014ed)f(in)f(the)189 2559 y(comm)o(unication)15 b(op)q(eration)f (itself.)21 b(A)14 b(pro)q(cess)h(ma)o(y)e(b)q(elong)j(to)d(sev)o(eral)i (groups)f(at)f(an)o(y)h(p)q(oin)o(t)189 2615 y(in)19 b(time,)g(and)g(ma)o(y)f (use)h(an)o(y)f(of)g(these)h(groups)f(as)g(scop)q(e)h(for)f(a)g(comm)o (unication)h(op)q(eration.)189 2672 y(Again,)14 b(the)f(initial)j(scop)q(e)e Fb(ALL)f Fg(is)h(prede\014ned,)h(and)f(can)g(b)q(e)g(used)g(as)g(a)f(\015at)g (name)h(space.)19 b(\(In)14 b(a)964 2828 y(8)p eop %%Page: 9 9 bop 189 45 a Fg(language)13 b(where)g(argumen)o(ts)f(are)h(optional,)h(con)o (text)e(w)o(ould)i(b)q(e)g(an)f(optional)g(parameter)g(with)189 102 y(default)i(v)m(alue)i Fb(ALL)p Fg(.\))146 208 y(The)h(adv)m(an)o(tage)f (of)g(an)h(explicit)i(con)o(text)d(parameter)g(is)h(that)g(it)g(prev)o(en)o (ts)f(the)h(confusion)h(that)75 264 y(o)q(ccurs)13 b(when)g(an)g(implicit)i (con)o(text)d(is)h(c)o(hanged)g(mid)g(comm)o(unication.)20 b(More)12 b(generally)l(,)i(it)f(prev)o(en)o(ts)75 321 y(the)e(confusion)h (arising)g(when)g(the)f(seman)o(tics)h(of)e(an)i(op)q(eration)f(is)h (quali\014ed)h(b)o(y)e(a)g(scop)q(e)h(that)e(dep)q(ends)75 377 y(on)18 b(the)g(con)o(trol)g(\015o)o(w)g(of)g(the)g(program,)g(rather)f (than)h(its)h(static)f(structure.)29 b(The)18 b(disadv)m(an)o(tage)g(is)75 434 y(the)e(need)g(for)f(an)g(additional)i(parameter,)d(in)j(languages)e (that)g(don't)g(ha)o(v)o(e)g(optional)h(parameters)e(or)75 490 y(o)o(v)o(erloading)h(of)g(pro)q(cedures.)146 547 y(Opinions?)75 690 y Fh(Matc)n(hing)23 b(of)g(send)g(and)h(receiv)n(e)75 793 y Fd(receiv)n(e)18 b(criteria)75 879 y Fg(Messages)c(are)h(matc)o(hed)g(b)o (y)h(sender)f(and)h(t)o(yp)q(e,)f(where)g(eac)o(h)g(can)h(b)q(e)g(a)e (\\don't)h(care".)146 935 y(Our)f(exp)q(erience)j(is)e(that)f(matc)o(hing)g (b)o(y)g(t)o(yp)q(e)h(only)l(,)g(or)f(disallo)o(wing)i(\\don't)e(care")g(in)h (the)f(sender)75 992 y(\014eld)20 b(signi\014can)o(tly)h(simpli\014es)g(the)e (implemen)o(tation)i({)d(but)h(some)g(users)g(complained)i(ab)q(out)e(suc)o (h)75 1048 y(restriction.)146 1105 y(Also,)c(do)g(w)o(e)g(w)o(an)o(t)f(to)g (b)q(e)i(explicit)i(ab)q(out)d(the)g(don't)g(care)g(v)m(alue,)h(or)e(use)i (named)f(constan)o(ts?)146 1161 y(Opinions?)75 1283 y Fd(Matc)n(hing)20 b(of)f(data)75 1369 y Fg(F)l(or)c(un)o(t)o(yp)q(ed)g(messages)g(the)g (options)h(are)131 1462 y(1.)22 b(Send)16 b(and)f(receiv)o(e)h(bu\013er)f(ha) o(v)o(e)g(same)g(size)131 1556 y(2.)22 b(Receiv)o(e)16 b(bu\013er)g(is)f(no)g (shorter)g(than)g(send)h(bu\013er)131 1650 y(3.)22 b(No)15 b(constrain)o(ts)f(\(messages)h(ma)o(y)f(b)q(e)i(truncated\).)131 1744 y(4.)22 b(User)15 b(c)o(ho)q(ose)g(one)g(of)g(the)g(ab)q(o)o(v)o(e)g (options)h(\(one)f(more)f(call)j(parameter,)d(or)g(a)h(global)h(\015ag\).)146 1838 y(Choices)f(should)g(b)q(e)g(the)f(same)g(for)f(all)j(t)o(yp)q(es)e(of)g (send)h(op)q(eration)f(\(the)g(curren)o(t)g(MPI)g(draft)g(seem)75 1894 y(to)h(allo)o(w)g(truncation)g(for)g(con)o(tiguous)g(data,)f(but)i (disallo)o(w)g(it)f(for)g(receiv)o(e)h(with)g(stride\))146 1951 y(The)d(EUI)h(prop)q(osal)f(uses)g(option)h(4.)19 b(I)13 b(w)o(ould)h(b)q(e)g(happier)g(with)f(2,)g(but)h(am)e(uncomfortable)i(with)75 2007 y(3)i(\(truncation)h(seems)g(to)f(fall)i(in)f(the)g(category)f(of)g (comm)o(unication)i(errors,)e(for)g(whic)o(h)h(I)h(assume)e(a)75 2063 y(global)g(exception)g(handling)h(mec)o(hanism\).)146 2120 y(Opinions?)146 2176 y(If)12 b(t)o(yp)q(ed)g(messages)g(are)g(supp)q (orted)g(then)h(one)f(should)h(sp)q(ell)h(out)e(requiremen)o(ts)g(on)g(t)o (yp)q(e)g(compat-)75 2233 y(ibilit)o(y)18 b(\(e.g.,)d(can)h(t)o(w)o(o)e(real) i(n)o(um)o(b)q(ers)g(b)q(e)h(receiv)o(ed)g(in)o(to)f(a)g(complex)g(n)o(um)o (b)q(er?\))23 b({)16 b(this)g(is)g(probably)75 2289 y(sp)q(eci\014c)h(to)e (eac)o(h)g(language)g(binding.)146 2346 y(Opinions?)75 2489 y Fh(Syn)n(tax)75 2590 y Fg(This)h(ma)o(y)e(b)q(e,)i(of)f(course,)f(a)h(sub)s (ject)g(of)g(endless)i(dispute.)k(A)15 b(few)g(ob)o(vious)g(guidelines:)143 2684 y Fa(\017)23 b Fg(Short,)14 b(mnemonic)i(names)964 2828 y(9)p eop %%Page: 10 10 bop 143 45 a Fa(\017)23 b Fg(Systematic)15 b(naming)h(con)o(v)o(en)o(tion)143 137 y Fa(\017)23 b Fg(Systematic)15 b(ordering)g(of)g(parameters)143 230 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)g(where)g(this)f(is)h(p)q (ossible,)h(of)e(optional)h(parameters)f(and)h(k)o(eyw)o(ord)e(param-)189 286 y(eters.)143 379 y Fa(\017)23 b Fg(Use,)15 b(in)h(languages)f(where)g (this)h(p)q(ossible,)h(of)d(o)o(v)o(erloading.)146 469 y(I)d(heard)g(in)h (our)f(last)g(meeting)g(a)g(preference)h(for)f(m)o(ultiple)h(function)g (names,)g(rather)e(than)h(m)o(ultiple)75 525 y(mo)q(des.)146 582 y(Th)o(us,)f(in)h(C++)g(or)f(F90,)f(one)i(could)g(ha)o(v)o(e)f(the)g(c)o (hoice)h(of)f(the)g(t)o(yp)q(e)g(of)g(datum)g(sen)o(t)g(\(scalar/con)o (tiguous)75 638 y(v)o(ector/noncon)o(tiguous\))15 b(b)q(e)i(dictated)g(b)o(y) f(the)h(t)o(yp)q(e)f(and)h(n)o(um)o(b)q(er)f(of)g(parameters,)f(rather)h (than)g(b)q(e)75 695 y(enco)q(ded)f(in)f(the)g(function)g(name.)19 b(Then)14 b(a)f(p)q(ossible)i(naming)f(c)o(hoice)h(is)f Fb(mode)23 b(|)h(operation)p Fg(,)12 b(where)75 751 y Fb(mode)18 b Fg(=)g Fb(init)g Fg(\(1\),)f Fb(nonblocking)g Fg(\(1+2\),)h Fb(blocking)f Fg(\(1+2+3+4,)h(lo)q(cal)h(termination\),)g(or)f Fb(sync)75 808 y Fg(\(1+2+3+4)d(global)h(termination\),)f(and)g Fb(operation)f Fg(=)h(send,)h(or)e(recv.)146 864 y(In)h(addition,)h(one)f(has)g Fb(DO\(handle\))p Fg(,)f Fb(START\(handle\))p Fg(,)e Fb(AWAIT\(handle\))i Fg(and)h Fb(FREE\(handle\))p Fg(.)146 921 y(Another)21 b(issue)i(is)f (function)g(vs)f(pro)q(cedure)i(call,)h(in)e(F)l(ortran.)38 b(Pro)q(cedure)22 b(calls)h(seem)e(more)75 977 y(natural)15 b(in)h(F)l(ortran,)e(where)h(function)h(in)o(v)o(o)q(cations)g(cannot)f(b)q (e)h(used)g(as)e(statemen)o(ts.)146 1033 y(Opinions?)22 b(Preferences)16 b(for)f(mnemonics?)21 b(Suggestions)16 b(for)e(F77)h(and)g(C?)75 1176 y Fh(Miscelania)75 1277 y Fg(Do)h(w)o(e)g(w)o(an)o(t)g(a)g Fb(TEST)p Fg(,)f(a)i(non)o(blo)q(c)o(king)h Fb(AWAIT)d Fg(op)q(eration?)25 b(\(Alw)o(a)o(ys)16 b(returns;)h(a)f(successful)i(return)75 1334 y(is)e(equiv)m(alen)o(t)h(to)d(a)h(successful)i(return)e(of)g(a)f(blo)q (c)o(king)j Fb(AWAIT)p Fg(\).)146 1390 y(Do)d(w)o(e)h(w)o(an)o(t)f(to)h Fb(AWAIT)f Fg(sev)o(eral)i(messages?)146 1447 y(Do)h(w)o(e)h(w)o(an)o(t)f(a)g Fb(PROBE)h Fg(function)h(\(tells)f(me)g(whic)o(h)h(messages)f(are)g(w)o (aiting)g(to)f(b)q(e)i(receiv)o(ed)g(b)o(y)75 1503 y(me\)?)146 1560 y(Where)12 b(do)h(w)o(e)f(return)h(information)g(on)g(parameters)f(of)g (receiv)o(ed)i(messages)e(suc)o(h)h(as)f(length,)i(and)75 1616 y(t)o(yp)q(e?)24 b(The)16 b(curren)o(t)g(MPI)h(prop)q(osal)f(has)g(length)h (b)q(e)g(the)f(v)m(alue)i(returned)e(b)o(y)h(the)f(receiv)o(e)h(function)75 1673 y(and)d(t)o(yp)q(e)g(returned)h(b)o(y)f(a)f(subsequen)o(t)i(call)g(to)e Fb(MP)p 964 1673 15 2 v 17 w(INFOT)p Fg(.)g(Another)h(natural)g(c)o(hoice)h (w)o(ould)f(seem)h(to)75 1729 y(b)q(e)f(that)f(that)f(length)i(and)g(t)o(yp)q (e)f(are)g(returned)h(b)o(y)f(the)h(len)g(and)g(t)o(yp)q(e)f(parameters)f(of) h(the)h(initial)h(call,)75 1786 y(but)g(this)h(is)g(error)e(prone:)143 1876 y Fa(\017)23 b Fg(The)d(v)m(alues)h(are)e(returned)i(async)o(hronously)f (for)f(non)o(blo)q(c)o(king)j(receiv)o(es.)35 b(The)20 b(user)g(has)g(to)189 1932 y(remem)o(b)q(er)11 b(not)h(to)f(touc)o(h)g(the)h(actual)g(len)g(and)g (t)o(yp)q(e)g(parameters,)f(in)h(addition)h(of)e(not)h(touc)o(hing)189 1989 y(the)j(receiv)o(e)h(bu\013er,)f(un)o(til)h(the)f(op)q(eration)h (completes.)143 2081 y Fa(\017)23 b Fg(A)f(natural)h(implemen)o(tation)h(w)o (ould)f(b)q(e)g(to)f(alw)o(a)o(ys)g(return)g(the)h(length)g(and)g(t)o(yp)q(e) g(of)f(the)189 2138 y(receiv)o(ed)f(messages.)36 b(Ho)o(w)o(ev)o(er,)20 b(most)g(users)g(will)i(assume)f(that)e(the)i(actual)g(len)g(and)g(t)o(yp)q (e)189 2194 y(parameters)14 b(are)h(not)g(up)q(dated)h(when)f(their)h(v)m (alue)h(is)e(not)g(\\don't)f(care".)143 2286 y Fa(\017)23 b Fg(The)13 b(user)h(will)h(not)e(b)q(e)h(able)g(to)f(use)h(constan)o(ts)e(as)h (actual)h(\\don't)f(care")g(parameter)f(v)m(alues)j(\(or)189 2343 y(an)o(ytime,)g(if)g(the)g(\\natural")g(implemen)o(tation)i(is)e(allo)o (w)o(ed\))143 2435 y Fa(\017)23 b Fg(This)17 b(c)o(hoice)h(b)q(ecomes)g(ev)o (en)f(less)h(app)q(etizing)h(when)e(a)g(handle)h(is)g(reused;)g(then)g(the)f (len)h(and)189 2492 y(t)o(yp)q(e)d(v)m(ariables)h(w)o(ould)g(b)q(e)g(reused,) f(to)q(o.)75 2582 y(It)g(seems)g(safer)f(to)g(return)h(these)g(v)m(alues)h (with)f(the)g(call)g(that)g(concludes)h(the)f(execution)h(of)e(sub)q(op)q (er-)75 2638 y(ation)h(3)g({)g(but)g(w)o(e)g(need)h(to)f(think)h(syn)o(tax.) 146 2695 y(Opinions?)952 2828 y(10)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Mon Nov 30 18:08:53 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22099; Mon, 30 Nov 92 18:08:53 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA03436; Mon, 30 Nov 92 17:54:53 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03434; Mon, 30 Nov 92 17:54:51 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA26470; Mon, 30 Nov 92 17:54:50 -0500 Date: Mon, 30 Nov 92 17:54:50 -0500 Message-Id: <9211302254.AA26470@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: draft of introduction Enclosed is a draft the introduction subcommittee has put together. We would like to get your comments and input. We feel that it is in a partial state and will be expanded. Hopefully this draft will help keep the discussion of goals, target, audience, etc. from becoming fragmented in each subcommittee. Jack \section{Introduction} \label{sec:intro} \subsection{Overview and Goals} Message passing is a paradigm used widely on certain classes of parallel machines; especially those with distributed memory. Although there are many variations, the basic concept of processes communicating through messages is well understood. Over the last ten years, substantial progress has been made in casting significant applications in this paradigm. Each vendor has implemented their own variant. More recently, several systems \cite{need references} have demonstrated that a message passing system can be efficiently and be portably implemented. It is thus an appropriate time to try to define both the syntax and semantics of a core of library routines that will be useful to a wide range of users and efficiently implementable on a wide range of computers. In designing MPI we have sought to make use of the most attractive features of a number of existing message passing systems, rather than selecting one of them and adopting it as the standard. Thus, MPI has been strongly influenced by work at the IBM T. J. Watson Research Center by Bala, Kipnis, Snir and colleagues \cite{}, Intel's NX/2 \cite{}, Express \cite{}, nCUBE's Vertex \cite{}, and PARMACS \cite{}. Other important contributions have come from Zipcode \cite{}, Chimp \cite{}, PVM \cite{}, and PICL \cite{}. One of the objectives of this paper is to promote a discussion within the concurrent computing research community of the issues that must be addressed in establishing a practical, portable, and flexible standard for message passing. This cooperative process began with a workshop on standards for message passing held in April 1992 \cite{}. The main advantages of establishing a message passing standard are portability and ease-of-use. In a distributed memory communication environment in which the higher level routines and/or abstractions are build upon lower level message passing routines the benefits of standardization are particularly apparent. Furthermore, the definition of a message passing standard, such as that proposed here, provides vendors with a clearly defined base set of routines that they can implement efficiently, or in some cases provide hardware support for, thereby enhancing scalability. The goal of the Message Passing Interface simply stated is to develop a \it{de facto} standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. \subsection{Who Should Use This Standard?} This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. \subsection{What Platforms Are Targets For Implementation?} The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run efficiently on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those "machines" consisting of collections of other machines, parallel or not, connected by a communication network. \subsection{What Is Included In The Standard?} The standard includes (this is temporarily as inclusive as possible): \begin{itemize} \item Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication \item Collective operations \item Process groups \item Communication contexts \item A simple way to create processes for the SPMD model \item Bindings for both Fortran and C \item A model implementation \item A formal specification. \end{itemize} \subsection{What Is Not Included In The Standard?} The standard does not specify: \begin{itemize} \item Explicit shared-memory operations \item Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages \item Program construction tools \item Debugging facilities \item Tracing facilities \end{itemize} Features that are not included can always be offered as extensions by specific implementations. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 16:39:05 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17944; Tue, 1 Dec 92 16:39:05 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27367; Tue, 1 Dec 92 16:14:27 -0500 Received: from relay2.UU.NET by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27363; Tue, 1 Dec 92 16:14:24 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA00550; Tue, 1 Dec 92 16:14:28 -0500 Received: from kailand.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 161350.23711; Tue, 1 Dec 1992 16:13:50 EST Received: from brisk.kai.com (brisk) by kailand.kai.com via SMTP (5.65d-92031301 for ) id AA22501; Tue, 1 Dec 1992 15:07:53 -0600 Received: by brisk.kai.com (920330.SGI-92101201) id AA16316; Tue, 1 Dec 92 15:07:51 -0600 Date: Tue, 1 Dec 92 15:07:51 -0600 Message-Id: <9212012107.AA16316@brisk.kai.com> To: mpi-comm@cs.utk.edu Subject: A proposal. From: Steven Ericsson Zenith Sender: zenith@kai.com Organization: Kuck and Associates, Inc. 1906 Fox Drive, Champaign IL USA 61820-7334, voice 217-356-2288, fax 217-356-5199 I've seen several proposals and made a few suggestions myself. I have now had a chance to look at Gropp and Lusk's Manual for the test implementation and I like it as a starting point. I propose to all subcommittees that we use this document as the focus for discussion along with the introduction Jack sent around yesterday. This shouldn't be taken as an endorsement by me of the Gropp and Lusk manual - I have several remarks I would like to make at a later stage. However, it is clear that we need a well considered and simple starting point and I see that in this document plus the introduction. Do I have any seconders? In the HPFF committe structure description we received a few days ago it was noted that much success came from someone taking charge of the central document. I implore someone to do this (Jack?). Steven P.S. I am not ignoring the other comments made in the committees - I think they too can - after debate - be folded into the suggested starting point. From owner-mpi-comm@CS.UTK.EDU Tue Dec 1 19:08:43 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21233; Tue, 1 Dec 92 19:08:43 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA01862; Tue, 1 Dec 92 18:43:11 -0500 Received: from cunyvm.cuny.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01858; Tue, 1 Dec 92 18:43:08 -0500 Message-Id: <9212012343.AA01858@CS.UTK.EDU> Received: from YKTVMV by CUNYVM.CUNY.EDU (IBM VM SMTP V2R2) with BSMTP id 7311; Tue, 01 Dec 92 18:42:33 EST Date: Tue, 1 Dec 92 18:43:02 EST From: "Marc Snir" X-Addr: (914) 945-3204 (862-3204) 28-226 IBM T.J. Watson Research Center P.O. Box 218 Yorktown Heights NY 10598 To: mpi-comm@cs.utk.edu Subject: HPF process Reply-To: SNIR@watson.ibm.com The HPF process did not start with a central document. Various peoples wrote proposals for various parts of the language design and, after several iterations, when the number of proposals starting being reasonable and these proposals started being coherent, they were collated into one document that evolved (mostly by successive pruning) into a draft. I suppose we can follow the same approach -- with the initial MPI Draft being our first frame of reference. From owner-mpi-comm@CS.UTK.EDU Fri Dec 4 16:10:12 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06597; Fri, 4 Dec 92 16:10:12 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA27740; Fri, 4 Dec 92 15:56:38 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27738; Fri, 4 Dec 92 15:56:36 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA07974; Fri, 4 Dec 92 15:56:35 -0500 Date: Fri, 4 Dec 92 15:56:35 -0500 Message-Id: <9212042056.AA07974@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack Name: Address: Email: Fax: Office phone number: Home phone number: (I'll use this only in an emergency and won't circulate.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 21:39:33 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10542; Mon, 7 Dec 92 21:39:33 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29754; Mon, 7 Dec 92 21:28:26 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29750; Mon, 7 Dec 92 21:28:21 -0500 Message-Id: <9212080228.AA29750@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 3637; Mon, 07 Dec 92 21:26:25 EST Date: Mon, 7 Dec 92 14:02:48 CST From: panda@vnet.ibm.com To: mpi-comm@cs.utk.edu To: mpi-comm@cs.utk.edu Subject: addresses for mpi I'm trying to put together a complete mailing list for MPI. Can you fill in the following information and send it to me. Thanks, Jack NAME: RAJ PANDA ADDRESS: E39/4305 IBM, 11400 BURNET RD., AUSTIN, TX 78758 EMAIL: PANDA@VNET.IBM.COM FAX: 1-512-823-6144 OFFICE PHONE NUMBER: 1-512-838-6632 HOME PHONE NUMBER: 1-512-835-1959 e.) From owner-mpi-comm@CS.UTK.EDU Mon Dec 7 22:10:17 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10659; Mon, 7 Dec 92 22:10:17 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA29855; Mon, 7 Dec 92 21:41:36 -0500 Received: from vnet.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29851; Mon, 7 Dec 92 21:41:26 -0500 Message-Id: <9212080241.AA29851@CS.UTK.EDU> Received: from AUSVM6 by vnet.ibm.com (IBM VM SMTP V2R2) with BSMTP id 4638; Mon, 07 Dec 92 21:39:36 EST Date: Mon, 7 Dec 92 14:51:31 CST From: LBWARD@AUSVM6.VNET.IBM.COM To: mpi-comm@cs.utk.edu Subject: address for mpi mailing list Please add my name to the MPI forum. Linton Ward Name: Linton Ward Address: 11400 Burnet Rd, Austin TX 78758 Email: lbward@ausvm6.vmnet.ibm.com Fax: (512) 823-6144 Office phone number: (512) 838-5116 Home phone number: (512) 244-1949 From owner-mpi-comm@CS.UTK.EDU Thu Dec 10 13:10:06 1992 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08984; Thu, 10 Dec 92 13:10:06 -0500 Received: by CS.UTK.EDU (5.61++/2.8s-UTK) id AA18058; Thu, 10 Dec 92 12:47:20 -0500 Received: from THUD.CS.UTK.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18051; Thu, 10 Dec 92 12:47:17 -0500 From: Jack Dongarra Received: by thud.cs.utk.edu (5.61++/2.7c-UTK) id AA27178; Thu, 10 Dec 92 12:47:15 -0500 Date: Thu, 10 Dec 92 12:47:15 -0500 Message-Id: <9212101747.AA27178@thud.cs.utk.edu> To: mpi-comm@cs.utk.edu Subject: Information on the MPI meeting Here are some details on the MPI meeting which is set for January 6th-8th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm January 6th and finishing about noon on January 8th. The format of the meeting is: Wednesday, January 6 1:00 pm to 8:00 pm 2 Breakouts for 10 people 1 Breakout for 20 people 5:00 pm to 6:00 pm--On our own for dinner. Thursday, January 7 8:00 am to 10:00 pm 1 U-shape room for 40 people 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, January 8 8:00 am to 12:00 pm 1 U-shape room for 40 people This is an open meeting and all are welcome. For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: January 6-8 February 24-26 April 7-9 May 19-21 June 30-July2 If you have any questions, please feel free to contact me. Jack From owner-mpi-comm@CS.UTK.EDU Wed Dec 23 16:46:29 1992 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA20114; Wed, 23 Dec 92 16:46:25 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA05394; Wed, 23 Dec 92 16:46:20 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14697; Wed, 23 Dec 92 17:31:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 23 Dec 1992 22:31:10 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14687; Wed, 23 Dec 92 17:31:08 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19206; Wed, 23 Dec 1992 17:29:47 -0500 Date: Wed, 23 Dec 1992 17:29:47 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9212232229.AA19206@rios2.epm.ornl.gov> To: abmacca@cs.sandia.gov, benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu, peter@sun.math.usfca.edu, srwheat@cs.sandia.gov Subject: MPI Information Status: RO Dear MPI committee member or observer, I need to know how many people to expect at the Dallas so please let me know if you intend to show up by sending email to walker@msr.epm.ornl.gov Please find below some information about the use of the MPI mailing lists, and who's involved in the various subcommittees. Please check that your entry is correct. If there are any errors please sent email to walker@msr.epm.ornl.gov and to dongarra@cs.utk.edu. Also there's some information about the January 6-8, 1993, meeting in Dallas. 1. Use of Mailing Lists ======================= The following mailing lists have been set up. mpi-comm@cs.utk.edu Whole committee (subcommitttes + observers) mpi-intro@cs.utk.edu Introduction subcommittee mpi-pt2pt@cs.utk.edu Point-to-point communication subcommittee mpi-collcomm@cs.utk.edu Collective communication subcommittee mpi-ptop@cs.utk.edu Process topology subcommittee mpi-lang@cs.utk.edu Language binding subcommittee mpi-formal@cs.utk.edu Formal language description subcommittee mpi-envir@cs.utk.edu Environment inquiry subcommittee All mail is being collected and can be retrieved by sending email to netlib@ornl.gov and in the mail message typing: send mpi-comm from mpi send mpi-intro from mpi ...etc. Also try: send index from mpi 2. Membership of Subcommittees (as of Dec. 22, 1993) ==================================================== 2.1. Introduction Subcommittee: Jack Dongarra (Chair) dongarra@cs.utk.edu Univ. of TN & ORNL Jim Cownie jim@meiko.co.uk Meiko Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@ecs.soton.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gob Argonne National Lab. Peter A. Rigsbee par@cray.com Cray Research, Inc. Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.2. Point-To-Point Communication Subcommittee Marc Snir (Chair) snir@watson.ibm.com IBM T. J. Watson Scott Berryman berryman@cs.yale.edu Yale Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Jack Dongarra dongarra@cs.utk.edu Univ. of TN & ORNL Anne Elster elster@cs.cornell.edu Cornell Univ. Al Geist geist@msr.epm.ornl.gov ORNL Adam Greenberg moose@think.com Thinking Machines Corp. Bill Gropp gropp@mcs.anl.gov Argonne National Lab. Leslie Hart hart@fsl.noaa.gov NOAA/FSL Rolf Hempel hempel@gmd.de GMD Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. of Colorado Charles Mosher ccm@arco.com ARCO Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.3. Collective Communication Subcommittee Al Geist (Chair) geist@msr.epm.ornl.gov ORNL Jaeyoung Choi choi@msr.epm.ornl.gov ORNL Jim Cownie jim@meiko.co.uk Meiko Mark Debbage md@ecs.soton.ac.uk Southampton Univ. Anne Elster elster@cs.cornell.edu Cornell Univ. Jon Flower jwf@parasoft.com Parasoft Corp. Adam Greenberg moose@think.com Thinking Machines Corp. Leslie Hart hart@fsl.noaa.gov NOAA/FSL C. T. Howard Ho ho@almaden.ibm.com IBM Almaden John Kapenga john@cs.wmich.edu Western Michigan Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Arthur B. Maccabe abmacca@cs.sandia.gov Sandia National Labs Charles Mosher ccm@arco.com ARCO Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Paul R. Pierce prp@ssd.intel.com Intel SSD Peter Rigsbee par@cray.com Cray Research, Inc. Anthony Skjellum tony@cs.msstate.edu Mississippi State Univ. Marc Snir snir@watson.ibm.com IBM T. J. Watson Vaidy Sunderam vss@mathcs.emory.edu Emory University Robert van de Geijn rvdg@cs.utexas.edu Univ. of Texas, Austin David Walker walker@msr.epm.ornl.gov ORNL Stephen R. Wheat srwheat@cs.sandia.gov Sandia National Labs Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.4. Process Topology Subcommittee Rolf Hempel (Chair) hempel@gmd.de GMD Jim Cownie jim@meiko.co.uk Meiko Jon Flower jwf@parasoft.com Parasoft Corp. Tom Henderson hender@fsl.noaa.gov NOAA/FSL Steven Huss-Lederman lederman@super.org Supercomput. Res. Ctr. Robert L. Knighten knighten@ssd.intel.com Intel SSD Oliver McBryan mcbryan@piper.cs.colorado.edu Univ. Of Colorado Steve Otto otto@cse.ogi.edu Oregon Graduate Inst. Lew Tucker tucker@think.com Thinking Machines Corp. 2.5. Language Binding Subcommittee Scott Berryman (Chair) berryman@cs.yale.edu Yale Bruce Leisure bleisure@kai.com Kuck & Associates, Inc. Robert L. Knighten knighten@ssd.intel.com Intel SSD Cherri M. Pancake pancake@cs.orst.edu Oregon State Univ. Peter Rigsbee par@cray.com Cray Research, Inc. 2.6. Formal Language Description Subcommittee Steve Zenith (Chair) zenith@kai.com Kuck & Associates, Inc. Ian Glendinning igl@ecs.soton.ac.uk Southampton Univ. Tony Hey ajgh@cs.ston.ac.uk Southampton Univ. Robert L. Knighten knighten@ssd.intel.com Intel SSD Rusty Lusk lusk@mcs.anl.gov Argonne National Lab. 2.7. Environment Inquiry Subcommittee Bill Gropp (Chair) gropp@mcs.anl.gov Argonne National Lab. Daniel Frye danielf@kgnvma.vnet.ibm.com IBM Kingston Robert L. Knighten knighten@ssd.intel.com Intel SSD Steve Zenith zenith@kai.com Kuck & Associates, Inc. 2.8. Observers Ed Benson benson@rdvax.enet.dec.com DEC Clemens H. Cap cap@ifi.unizh.ch University of Zurich Michel J. Dayde dayde@enseeiht.fr ENSEEIHT-IRIT Jim Feeney feeneyj@gdlvm6.vnet.ibm.com IBM Kyle Gallivan gallivan@csrd.uiuc.edu CSRD, Univ. of Illinois Michael Heath heath@ncsa.uiuc.edu NCSA, Univ. of Illinois Mark Bowen Hill mbh@ecs.soton.ac.uk Southampton Univ. Shlomo Kipnis kipnis@watson.ibm.com IBM T. J. Watson ??? langley@dirac.scri.fsu.edu Florida State Univ. Bob Leary leary@sdsc.edu San Diego Super. Ctr. Lorie Liebrock lorie@cs.rice.edu Rice University Miron Livny miron@cs.wisc.edu Univ. of Wisconsin Ken Kennedy ken@rice.edu Rice University Neil MacDonald nbm@epcc.ed.ac.uk Univ. of Edinburgh Dan Cristian Marinescu dcm@cs.purdue.edu Purdue University Harish Nag harish@ssd.intel.com Intel SSD Dan Nessett nessett@llnl.gov LLNL Angela Ovearly fsang@kira.lerc.nasa.gov NASA Lewis Res. Ctr. Peter S. Pacheco peter@sun.math.usfca.edu Univ. of CA, San Franc. Raj Panda panda@vnet.ibm.com IBM Arnulfo Perez arperez@mtecv2.mty.itesm.mx Center for AI, Mexico Matthew Peters mpeters@vnet.ibm.com IBM UK Sci. Ctr. Jean-Laurent Philippe philippe@archipel.fr Archipel Robert S. Schreiber schreibr@riacs.edu RIACS, NASA Ames Mike Surridge ms@pac.soton.ac.uk Southampton Univ. Tricot tricot@archipel.fr Archipel Robert Voigt rvoigt@note.nsf.gov NSF Linton Ward lbward@ausvm6.vmnet.ibm.com IBM Dennis Weeks weeks@convex.com Convex Francis Wray falk@parsytec.de Parsytec 3. Institutions Involved in MPI =============================== 3.1. Universities Univ. of Edinburgh Emory University N.L. Mexico Centro de Intelligencia Artifical, Mexico Mississippi State University Oregon Graduate Institute Oregon State University Purdue University Rice University Western Michigan University Yale University University of Illinois, CSRD University of Illinois, NCSA University of Southampton University of Tennessee University of Texas at Austin University of Wisconsin University of Zurich University of Colorado 3.2. Laboratories ARCO Exploration and Production Technology Argonne National Laboratory Cerfacs ENSEEIHT-IRIT, France GMD Lawrence Livermore National Laboratory NASA RIACS NOAA Oak Ridge National Laboratory San Diego Supercomputer Center Sandia National Laboratories Supercomputer Research Center 3.3. Companies ARCHIPEL S.A. Cray Research, Inc. Digital Equipment Corporation IBM Austin IBM Almaden Research Center IBM Kingston IBM T.J. Watson Research Center IBM UK Scientific Centre Intel Supercomputer Systems Division Kuck and Associates, Inc. ParaSoft Corporation Parsytec Computer GmbH Thinking Machines Corporation 4. MPI Committee Meeting, Jan 6-8, 1993 ======================================= The next meeting of the MPI subcommittees will take place at Bristol Suites Hotel 7800 Alpha Road Dallas, Texas The meeting will start at 1pm on Wednesday January 6, 1993, and finish at noon on Friday, January 8, 1993. Rooms are $89 per night and reservations may be made by calling (214) 233-7600 (mention MPI meeting). The meeting registration fee will be $75. Please make checks and POs payable to University of Tennessee. Jack Dongarra will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle refer to the MPI meeting. We have made arrangements with Delta Airlines to get a deal on airfares. When you book your flights please call Delta or have your travel agent call 1-800-241-6760 for reservations and refer to File Number S2145. The file goes by the name Message Passing Interface. A certain restrictions apply and seats are limited. An agenda will be sent out soon to all subcommittee members. We intend to rent a laser printer for use by attendees with protable computers. From owner-mpi-comm@CS.UTK.EDU Mon Jan 4 11:02:02 1993 Received: from convex.convex.com by mozart.convex.com (5.64/1.28) id AA06142; Mon, 4 Jan 93 11:02:00 -0600 Received: from CS.UTK.EDU by convex.convex.com (5.64/1.35) id AA08877; Mon, 4 Jan 93 11:01:56 -0600 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15255; Mon, 4 Jan 93 11:54:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 04 Jan 1993 16:54:19 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from timbuk.cray.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15247; Mon, 4 Jan 93 11:54:16 -0500 Received: from teak18.cray.com by cray.com (4.1/CRI-MX 2.4) id AA21055; Mon, 4 Jan 93 10:54:14 CST Received: by teak18.cray.com id AA05581; 4.1/CRI-5.6; Mon, 4 Jan 93 10:54:13 CST From: par@teak.cray.com (Peter Rigsbee) Message-Id: <9301041654.AA05581@teak18.cray.com> Subject: Profilers etc. (fwd) To: mpi-comm@cs.utk.edu Date: Mon, 4 Jan 93 10:54:10 CST X-Mailer: ELM [version 2.3 PL11b-CRI] Status: R James Cownie writes: > > I have an implementation issue which I would like to raise in the MPI > forum, since it is unclear where it fits into the sub-committee > structure I have mailed to both mpi-pt2pt and mpi-lang. Apologies to > those of you who receive this message twice. > > Issue > ===== > The major objective of MPI1 is to achieve portability of applications. > This has major benefits for us all (not least in legitimising and > therefore growing the MPP marketplace). > > One of the benefits which it would also be nice to achieve would be > the wide availability of different tools which support programming in > the MPI1 model. The most immediately obvious such tools (to me at > least !) are > > 1) HPF to Fortran + MPI1 translators > 2) Performance monitoring/tuning tools > 3) Debuggers > > Support for the first is easy (since it just requires what we're > already doing). > > Portable support for the second is not so trivial, since the > collection of useful performance information is much more intrusive. > > Portable support for the third is harder still, and I won't discuss it > further. First, I think this is an important issue. I'd argue that the need isn't necessarily to support portable tools, but it would be nice to define a mechanism by which performance and debugging tools could obtain at least the most basic information about the communications. Second, I don't think it really fits into either group that Jim sent the message to. The same issue, for example, also applies for global communications. This is why I copied the entire committee, and I think there should be some discussion in Dallas was to whether this is a topic that should be covered by the standard. Third, I fear this will be a very difficult subject to address. Clearly Jim had some specific use in mind when he made his suggestions, but didn't go into much detail (and properly so -- this isn't intended as a criticism). But there are some very basic differences between the ways that different tools are designed that need to be considered. What information should be recorded? Is data needed for each transmission (a "trace") or should it be accumulated by the library? Should the library pass information to the tool or should the tool query the library? I don't think there are clear cut answers here. But I think this is an important issue that should receive some airtime in Dallas. - Peter Rigsbee par@cray.com I've included the rest of Jim's proposal for those who didn't see the original mailing... > > Options > ======= > We have various possible options which we can take. > > 1) Ignore the problem > Provide no support for portable performance monitoring tools, and > leave each tool provider with a large porting problem. > > I don't like this solution, it loses some of the benefit of the > standard, which should be attracting people to build tools. > > 2) Document specific implementation hooks as part of MPI1. > In effect these would be callbacks from the library to profiling > code which could then do whatever it liked. > > 3) As 2, but without REQUIRING that a conforming implementation > provide the functions. They're there as a recommendation, rather > than being mandatory. > > > I think we should be concerned about this, and I'd like us at least to > make some recommendation. Personally I'd probably implement two > separate interface to the library, one of which provided the hooks, > and the other of which didn't so that you don't pay the cost of > checking the profiling hooks unless you asked to. > > Of course even if we do nothing it's not too hard to escape (using > horrible macros) in C, but Fortran doesn't always have macros, so a > properly specified internal solution is definitely preferable. > > Thoughts ??? > Flame me at Dallas. I'm travelling tomorrow. When are we having a > meeting in Europe ??? > > -- Jim > James Cownie > Meiko Limited > 650 Aztec West > Bristol BS12 4SD > England > > Phone : +44 454 616171 > FAX : +44 454 618188 > E-Mail: jim@meiko.co.uk or jim@meiko.com > > From owner-mpi-comm@CS.UTK.EDU Tue Jan 5 17:33:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07561; Tue, 5 Jan 93 17:33:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27130; Tue, 5 Jan 93 17:32:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 05 Jan 1993 22:32:45 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27122; Tue, 5 Jan 93 17:32:42 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA23201; Tue, 5 Jan 93 22:32:36 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01387; Tue, 5 Jan 93 15:31:37 MST Date: Tue, 5 Jan 93 15:31:37 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301052231.AA01387@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: A few general comments Here are a few comments/questions regarding the MPI1 draft proposal from folks here (Forecast Systems Lab). Apologies if you have seen them before! 1) Currently, communication between groups operating in different communication contexts appears to be very difficult. One way to alleviate this problem might be to provide a way to "name" contexts. Communicating groups could then refer to a context by name to ensure that the same message types were used. Message registration would also accomplish this and might be easier to implement (see our "Proposal 1" below). 2) We believe that MPI1 should include a specification of low level protocols that vendors should implement so that heterogeneous computing can be made available. Each media type (standard ethernet, FDDI, HIPPI, etc.) should have a specification. This would greatly aid in this standard being useful in a workstation environment. The protocol document would be a companion document to the syntactical standard for MPI1. 3) On p. 9 "No reference may be made to any process or group outside the current group context." How do different groups that are simultaneously working on different stages of a problem communicate? MPI_PARTG()?? 4) The use of stacks to manage contexts and groups may be a bad idea if MPI ever intends to support MIMD programs (how do groups synchronize stacks of message contexts?). Message registration would do the job without stacks. Registration of blocks of messages at the beginning of a program (or large module) would not affect performance very much. For groups, we like Al Geist's approach of identifying a process by one or more (group ID, process ID) pairs. We feel that communication between groups is very useful, even for SPMD programs. 5) It is clear that some way of responding to asynchronous events must be provided. We believe that "interrupt" type routines (such as Intel's hrecv() and hsend()) may not be the best approach. We have a brief alternative proposal for "Remote Services" below ("Proposal 2"). 6) The consensus seems to be that "parallel I/O" should not be a part of MPI1. Should it be a part of MPI2 or MPI3? If our intent is to create a standard that supports the development of portable code then we believe that I/O must be considered at some point. We should make sure that the specification of MPI1 includes the hooks necessary to include I/O at a later date. If our intent is to create only a low level library, then we can leave all this up to us users. We have some ideas about I/O and Data Decomposition below ("Proposal 3"). Successful implementation of such a proposal would require process topology information (yes, here's a plug for the Process Topology subcommittee). The proposals follow. Regards, Leslie Hart (hart@fsl.noaa.gov) Tom Henderson (hender@fsl.noaa.gov) Bernardo Rodriguez (bernardo@fsl.noaa.gov) PROPOSALS Proposal 1. Message Type Registration. Message types are managed using a global registry. Routines are provided that allow the user to "reserve" blocks of message types for use by the application. Message type registration is intended to prevent message type "collisions" when user code and third-party library routines are used together (historically, a serious problem with typed message passing systems). Initial concepts for message type registration are due to Tony Skjellum of Lawrence Livermore National Laboratory. Two proposed routines to register and monitor message types are described below. Arguments have been omitted for brevity. Message type registration routine: MPI_ReserveMsgType() Reserve one or more message types for use by the calling process. This routine is used to prevent message type "collisions" between user applications and libraries. Inputs include an ID string and the number of message types to be reserved. The routine first checks the locally maintained message type registry and returns an error condition if the ID string matches an existing entry. The routine then goes to the global message type registry to get the message types. If the ID string matches an existing entry in the global message type registry then the message types are copied from that entry to the calling process. Otherwise, new message types are registered with the ID string, entered into the global message type registry, and copied to the calling process. The new ID string and message types are entered into the locally maintained message type registry. Message type registration inquiry routine: MPI_InquireMsgTypeByName() Ask the message type registry to obtain the message types reserved under an ID string. An error is returned if no message types have been reserved under the ID string. Proposal 2. Remote Services (brief concept only). A remote service is similar in concept to a RPC (remote procedure call). Remote services are most often used to respond to specified asynchronous events. The actions taken in response to asynchronous events are performed by agent processes known as protocol handlers. A remote service is requested of one process by the same or another process. The remote service request is accompanied by a small amount of data to guide the action of the service. Routines are provided to allow user-created routine to be attached to specified requests. An attached user-created routine is called a protocol handler. A protocol handler does not have access to the address space of the its associated process. The protocol handler has the status of a full process (ie. it may perform I/O and message-passing communication). This is an advantage over some current interrupt-driven approaches. Remote service requests are handled one at a time on a first-come first-served basis (ie. a new remote service request will not be handled until all pending remote service requests are handled). This may be a drawback to this approach. Proposal 3. Some ideas about Cartesian Data Decomposition and Parallel I/O. Due to length, we have just included a few ideas. If there is any interest, we can provide (lots) more detail. 1) We propose Cartesian data decomposition routines to support the scalable distribution of multidimensional arrays among the processes in a group. These routines isolate the user from the details of implementing scalable data parallelism in a message passing environment. The routines also enhance performance by distributing data in a way that is most advantageous for a particular architecture. 2) A Cartesian data decomposition defines the distribution of a multidimensional data array onto a "logical" array of processes with the same dimension. The processes are members of a specified group. A Cartesian data decomposition is defined by information provided by the user and by run-time parameters determined by the system. The user specifies the group, array dimension, and number of elements in each dimension of the data array. The user may also optionally specify re-ordering of processes in the group and boundary behavior. The Cartesian data decomposition routines then combine this information with the number of processes in the group (provided by the system at run time) to determine which portion of the data array will reside on each process in the group. All information about the decomposition is then stored in a structure (or FORTRAN77 array). The "logical" array of processors is created using the Processor Topology routines (this idea has already been discussed in some detail in that subcommittee). 3) Cartesian data decomposition and I/O should be intimately related. Once a decomposition structure has been created, I/O should be possible using routines with syntax similar to this: cget(file, array, decomp) cput(file, array, decomp) where decomp is the decomposition structure. 4) Data decomposition tools have been available for several years, most notably from Parasoft's Express software package. These tools are extremely useful for almost all data parallel applications. Use of these tools in conjunction with I/O greatly simplifies program development on message passing systems. 5) Few areas in parallel processing have been dealt with in less detail by standardization efforts than parallel I/O. We would like to see behavior defined for such operations as seek/read and seek/write when several processes are accessing the same file. The concept of "loose synchronization" (Express and elsewhere) has proven to be very useful and should be included in a standard. Encapsulation of I/O in standardized routines would also permit performance optimization by the people who (should) know best how to do it (ie. the hardware vendors). From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 11:42:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24005; Wed, 6 Jan 93 11:42:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12383; Wed, 6 Jan 93 11:41:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 16:41:34 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12375; Wed, 6 Jan 93 11:41:32 -0500 Received: from carbon.pnl.gov (130.20.65.121) by pnlg.pnl.gov; Wed, 6 Jan 93 08:39 PST Received: from fermi.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA25662; Wed, 6 Jan 93 08:38:48 PST Received: by fermi.pnl.gov (4.1/SMI-4.1) id AA18870; Wed, 6 Jan 93 08:38:46 PST Date: Wed, 06 Jan 93 08:38:45 -0800 From: Robert J Harrison Subject: Re: Comments on Bill Gropp's comments To: mpi-comm@cs.utk.edu Message-Id: <9301061638.AA18870@fermi.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu At the end of message <9301051955.AA01014@SSD.intel.com> Paul Pierce wrote > > I strongly recommend that we avoid separated info calls. Info can be returned > through output parameters, although that does annoyingly complicate receive > calls. Alternatively, info can be associated with handles, but that seriously > complicates handle management. (Handle management is a good topic for a > separate discussion.) I also strongly favour not having separate info calls. The necessary information is brief and does not excessively clutter argument lists, and can always be hidden in a wrapper if it is routinely not required. Also, by removing sources of ambiguity we give more freedom for implementation specific optimizations. > > If we decide we must have separated info calls, we should use dynamic context > semantics. > Yes to this too. Robert Harrison. From owner-mpi-comm@CS.UTK.EDU Wed Jan 6 17:17:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10712; Wed, 6 Jan 93 17:17:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29683; Wed, 6 Jan 93 17:16:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 06 Jan 1993 22:16:46 GMT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from p.lanl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29672; Wed, 6 Jan 93 17:16:45 -0500 Received: from beta.lanl.gov by p.lanl.gov (5.65/1.14) id AA11282; Wed, 6 Jan 93 15:16:41 -0700 Received: by beta.lanl.gov (5.57/Ultrix2.4-C) id AA29872; Wed, 6 Jan 93 15:15:46 -0700 Date: Wed, 6 Jan 93 15:15:46 -0700 From: sp@beta.lanl.gov (Stephen W Poole) Message-Id: <9301062215.AA29872@beta.lanl.gov> To: mpi-comm@cs.utk.edu I would like to have my name added to the mailing list. Steve... From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:31:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17910; Fri, 15 Jan 93 15:31:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13121; Fri, 15 Jan 93 15:29:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:29:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13112; Fri, 15 Jan 93 15:29:50 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA13694; Fri, 15 Jan 1993 15:29:49 -0500 Date: Fri, 15 Jan 1993 15:29:49 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301152029.AA13694@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts I thought the discussion of communication contexts at the last Dallas meeting was lacking in focus, and I'm afraid the existing subcommittees may not deal with this area. If there are enough people interested perhaps there should be a separate communication context subcommittee. If you have any strong opinions on this please let me know. Also let me know if you would like to be on the communication context subcommittee if one is established. On the topic of communication contexts I think (at least) 4 approaches have been proposed. 1) Implicit communication contexts as in MPI1 controlled by push/pop. 2) Explicit registration as in Zipcode 3) Explicit communication contexts should be merged with explicit group contexts. Since groups cannot communicate with each other a new communication context can be created by creating a new group with the same members as the current group. 4) Chuck Simmons of Oracle has suggested that communication contexts are really a particular type of thread, and so can be handled using existing threads packages. If you have any ideas, or are strongly for or against any of the above approaches let mpi-comm know so we can initiate a discussion of this topic. David Walker From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 15:48:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18263; Fri, 15 Jan 93 15:48:16 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14054; Fri, 15 Jan 93 15:47:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 15:47:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14046; Fri, 15 Jan 93 15:47:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA03636; Fri, 15 Jan 93 14:47:35 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00709; Fri, 15 Jan 93 14:47:32 CST Message-Id: <9301152047.AA00709@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: walker@rios2.epm.ornl.gov Date: Fri, 15 Jan 93 14:47:31 CST From: Rusty Lusk | | On the topic of communication contexts I think (at least) 4 approaches have | been proposed. | 1) Implicit communication contexts as in MPI1 controlled by push/pop. | 2) Explicit registration as in Zipcode | 3) Explicit communication contexts should be merged with explicit | group contexts. Since groups cannot communicate with each other | a new communication context can be created by creating a new group | with the same members as the current group. | 4) Chuck Simmons of Oracle has suggested that communication contexts | are really a particular type of thread, and so can be handled | using existing threads packages. | If you have any ideas, or are strongly for or against any of the above | approaches let mpi-comm know so we can initiate a discussion of this topic. I think that 1) contradicts our genral agreement to avoid state as much as possible. 4) is a problem on systems that don't support threads, and we have more-or-less agreed to be consistent with thread packages but not depend upon them. 3) should be discussed along with the converse: that contexts rather than groups should be the primary concept and groups defined in terms of contexts. David, perhaps you could create a mailing list on which this discussion can take place, parallel to the subcommittee mailing lists, so it doesn't have to go to the whole committee. Thanks. Rusty From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 16:13:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18561; Fri, 15 Jan 93 16:13:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16784; Fri, 15 Jan 93 16:12:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 16:12:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from convex.convex.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16768; Fri, 15 Jan 93 16:12:01 -0500 Received: from mozart.convex.com by convex.convex.com (5.64/1.35) id AA21882; Fri, 15 Jan 93 15:11:54 -0600 Received: by mozart.convex.com (5.64/1.28) id AA01950; Fri, 15 Jan 93 15:11:17 -0600 Date: Fri, 15 Jan 93 15:11:17 -0600 From: weeks@mozart.convex.com (Dennis Weeks) Message-Id: <9301152111.AA01950@mozart.convex.com> To: mpi-comm@cs.utk.edu Subject: Communication contexts Cc: lusk@antares.mcs.anl.gov, walker@rios2.epm.ornl.gov Rusty Lusk writes: > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. I believe the default inclusion of everyone is a good idea, although creating a separate mailing list might be appropriate if some mpi-comm members wish to "unsubscribe". In the meantime, I will add one brief opinion: I felt that the discussions of contexts in the topology subgroup were quite sufficient, and it would probably be better to have the single subgroup which covers both groups and contexts, so that both will appear in the standard in a style which does not make them conflict with each other, and/or to avoid a proposal where groups and contexts would be totally redundant. I definitely agree that it is an important topic, and that the discussions last week in Dallas were far away from getting a proposal for groups and contexts that would be unanimously acceptable, so much work remains to be done. From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 17:01:57 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19329; Fri, 15 Jan 93 17:01:57 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21190; Fri, 15 Jan 93 17:00:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 17:00:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21182; Fri, 15 Jan 93 17:00:54 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA20492; Fri, 15 Jan 93 22:00:50 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA14823; Fri, 15 Jan 93 14:59:49 MST Date: Fri, 15 Jan 93 14:59:49 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Communication contexts > | > | On the topic of communication contexts I think (at least) 4 approaches have > | been proposed. > | 1) Implicit communication contexts as in MPI1 controlled by push/pop. > | 2) Explicit registration as in Zipcode > | 3) Explicit communication contexts should be merged with explicit > | group contexts. Since groups cannot communicate with each other > | a new communication context can be created by creating a new group > | with the same members as the current group. > | 4) Chuck Simmons of Oracle has suggested that communication contexts > | are really a particular type of thread, and so can be handled > | using existing threads packages. > | If you have any ideas, or are strongly for or against any of the above > | approaches let mpi-comm know so we can initiate a discussion of this topic. > > I think that 1) contradicts our genral agreement to avoid state as much as > possible. 4) is a problem on systems that don't support threads, and we have > more-or-less agreed to be consistent with thread packages but not depend upon > them. 3) should be discussed along with the converse: that contexts rather > than groups should be the primary concept and groups defined in terms of > contexts. > > David, perhaps you could create a mailing list on which this discussion can > take place, parallel to the subcommittee mailing lists, so it doesn't have to > go to the whole committee. Thanks. > > Rusty I'm not sure, but I believe that Zipcode uses push/pop to control contexts (Tony, please correct me if I'm wrong). I'd like to add a 5th approach, basically the one proposed by Paul Pierce: 5) Contexts are used exclusively to insure that message collisions will not occur if independently developed sub-programs are combined. Contexts and groups are orthogonal. Contexts and threads are orthogonal. Each message has an associated context and tag. Message context is managed by library routines and is completely out of a user's control. Message tag is selected by the user. The method of managing message contexts is a separate issue (assuming we want contexts). Existing proposals are: 5a) Stack-based management (objected to due to hidden states). 5b) Explicit registration with user-defined "names" (probably requires some communication). 5c) Explicit registration by a central authority ("dollar bill" registration mentioned by Jim Cownie.) There is enough confusion about contexts and groups that we may want to keep this discussion open to everyone. If the concensus is to make a separate mailing list, please both of us to it. Thanks, Tom Henderson hender@fsl.noaa.gov Leslie Hart hart@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Fri Jan 15 19:06:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21669; Fri, 15 Jan 93 19:06:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27214; Fri, 15 Jan 93 19:05:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 15 Jan 1993 19:05:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27206; Fri, 15 Jan 93 19:05:40 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00450; Fri, 15 Jan 93 18:05:36 CST Date: Fri, 15 Jan 93 18:05:36 CST From: Tony Skjellum Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu Subject: Re: Communication contexts In Zipcode, contexts have a group-wide scope. Groups of processes can involve more than one context. For instance, if 10 processes are involved in several stages of a calculation, each stage might have a different notion of how to do messaging, but each would use all the processes. Groups are important, because they allow control over the scope of global operations. Contexts are important because they provide guarantees to sub-programs about restricted scope of messages. Contexts are added typing-like information, but which is controlled by the system, not ad hoc by each user program. When they determine to start messaging, sub-programs request a context from a "postmaster general" in Zipcode. Currently, this is a group-oriented request (loose synchronization) resulting in each group member getting the context, and other information. In a lower-level system, simpler tactics might be possible. The way we did it was supposed to avoid possibility of races or other problems like that. To build large-scale software, without globalization of the local use of messaging resources, and to provide scalable algorithms with efficient global operations, both groups and contexts are needed. The push/pop state is an implementation detail of little importance. -Tony From owner-mpi-comm@CS.UTK.EDU Sat Jan 16 02:07:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26026; Sat, 16 Jan 93 02:07:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11696; Sat, 16 Jan 93 02:06:16 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 16 Jan 1993 02:06:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11688; Sat, 16 Jan 93 02:06:13 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA06036; Sat, 16 Jan 93 02:01:08 EST Date: Sat, 16 Jan 93 02:01:08 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301160701.AA06036@cs.wmich.edu> To: mpi-comm@cs.utk.edu hi; It was my feeling that the concepts of group and context, that were clearly understood differently by different people at the start of the discussions, became somewhat better agreed on as the pt2pt discussion took place. Groups - used to define groups of processors for collective communication primitives and to allow mapping topologies onto processors. This allows a group of processors to be used in isolation to compute together. Contexts - As Tom (and Tony) express below, > From: hender@macaw.fsl.noaa.gov (Tom Henderson) > Message-Id: <9301152159.AA14823@macaw.fsl.noaa.gov> > To: mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > I'd like to add a 5th approach, basically the one proposed by Paul Pierce: > > 5) Contexts are used exclusively to insure that message collisions will not > occur if independently developed sub-programs are combined. Contexts and > groups are orthogonal. Contexts and threads are orthogonal. Each message > has an associated context and tag. Message context is managed by library > routines and is completely out of a user's control. Message tag is > selected by the user. The only expression stated so far about the difference between tags and contexts I had gleaned was that context should now be wildcardable (IE no -1 for All contexts) while a receive ALL or some other MASK variant would be allowed on tags. > > The method of managing message contexts is a separate issue (assuming we > want contexts). Existing proposals are: > > 5a) Stack-based management (objected to due to hidden states). > 5b) Explicit registration with user-defined "names" (probably requires > some communication). > 5c) Explicit registration by a central authority ("dollar bill" > registration mentioned by Jim Cownie.) Add: 5d) A context is not strictly managed, but use should follow suggested guidelines. This could allow static allocation of contexts when reasonable, but also support a context server (as the Postmaster General in Zipcode). Think of the static contexts or non-registrar provided contexts as reserved with respect to any registration system. This could be part of the MPI initialization call, if a registrar is included. The drawback here of course is the user could violate whatever guidelines were given. It seems to me most types of name registration or stack allocation schemes, that are not too restrictive and allows user code to interact, are likely to allow errant management of contexts as well. I expect that allowing 5d) would let strictly controlled context systems to be built over MPI. > From: Tony Skjellum > Message-Id: <9301160005.AA00450@Aurora.CS.MsState.Edu> > To: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu > Subject: Re: Communication contexts > Status: R > > In Zipcode, contexts have a group-wide scope. Groups of processes can Of course if we eliminate groups from the pt2pt call then contexts may not be explicitly limited to groups in MPI. (though a context server could be asked to provide a context relative to a group). > involve more than one context. For instance, if 10 processes are involved > in several stages of a calculation, each stage might have a different notion > of how to do messaging, but each would use all the processes. Groups > are important, because they allow control over the scope of global operations. > Contexts are important because they provide guarantees to sub-programs > about restricted scope of messages. Contexts are added typing-like > information, but which is controlled by the system, not ad hoc by each > user program. When they determine to start messaging, sub-programs > request a context from a "postmaster general" in Zipcode. Currently, this ... > To build large-scale software, without globalization of the local use of > messaging resources, and to provide scalable algorithms with efficient > global operations, both groups and contexts are needed. The push/pop > state is an implementation detail of little importance. I expect the use push/pop for context state would make implementation in some environments hard and I have some concern about requiring registration on very large systems. I do think a strictly controlled stack context system could be built over 5d). The question about context management relates to how high a level MPI should be at. I see it at a low level, that higher level systems can be built on. That is one reason for keeping context management "advisory". > > -Tony > I liked the zipcode documentation. I think it would be a good idea to make it available on netlib. john From owner-mpi-comm@CS.UTK.EDU Sun Jan 17 11:46:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04093; Sun, 17 Jan 93 11:46:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09441; Sun, 17 Jan 93 11:43:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 17 Jan 1993 11:43:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09433; Sun, 17 Jan 93 11:43:55 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA08023; Sun, 17 Jan 93 11:38:43 EST Date: Sun, 17 Jan 93 11:38:43 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9301171638.AA08023@cs.wmich.edu> To: mpi-comm@cs.utk.edu Although the question of IO specification in the MPI-1 has been raised before, I think it still needs to be addressed. Omission of any IO requirements means no program can be portable. If IO specifications are not included the introduction should make a strong case for the reasons. Any attempt to specify high level parallel IO at this point in time seems premature. Perhaps there can be a middle ground. We could specify low level IO, compatible with ANSI/ISO/POSIX standards and perhaps allow an inquire function to test for its presence. I will suggest every process should have access to Level-1 IO, given in the following list, which works for c and c++. -------------------------------------------------------------------- Level-1 Input and Output A process should have access to: 1) open/close/flush read/write lseek 2) readv/writev 3) fcntl dup dup2 ioctl 4) fopen/fclose/fflush scanf/printf fscanf/fprintf sscanf/sprintf 5) varargs vscanf/vprintf vfscanf/vfprintf vsscanf/vsprintf Behavior of these functions is to match that in current standards (specific standard to be chosen). -------------------------------------------------------------------- This could be loosened to only require defined behavior when access is by a single process. A Like Level-1 IO can be specified for Fortran-77 as well. This type of facility is provided in many vendor environments and several portable environments as well. A prototype version of the Level-1 IO could be built using gnu sprintf/sscanf, MPI-1 message passing and the assumption that some process can do IO. -------------------------------------------------------------------- Level-2 Input and Output A case can be made for Level-2 Input and Output as well. This would be group IO, no claim would be made for efficiency, only that the behavior was defined and would not overload the communication network. All processes in a group would execute the same IO calls as in level-1 and the joint behavior would be defined. To avoid state a modified name for each function could be used. A generic level-2 IO could be built on level-1 and the rest of MPI-1. -------------------------------------------------------------------- I propose we include a Level-1 specification and I think we should consider Level-2 as well. I am willing to draft a Level-1 suggestion and a Level-2 suggestion. I'm not sure which mailing list to put this on, but I think the issue needs to be addressed quickly (even if the decision remains to ignore IO). john From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 09:30:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14975; Mon, 18 Jan 93 09:30:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16251; Mon, 18 Jan 93 09:29:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 09:29:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16243; Mon, 18 Jan 93 09:29:24 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA12246; Mon, 18 Jan 1993 09:29:23 -0500 Date: Mon, 18 Jan 1993 09:29:23 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181429.AA12246@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI news 1) NEW SUBCOMMITTEE A new subcommittee on profiling has been set up. It is chaired by Jim Cownie of Meiko, and you can send email to it at mpi-profile@cs.utk.edu. Please let me know if you want to join 2) NEWCOMERS FILE If you are a newcomer to the MPI forum or want general information send the message send mpi-newcomers from mpi to netlib@ornl.gov 3) REVISED VERSION OF MPI1 TECH REPORT A revised version of the MPI1 paper by Dongarra, Hempel, Hey , and Walker is available from netlib. This is basically the original document with some unnecessary and inappropriate routines removed. The PostScript file can be obtained by sending the message send mpi1.ps from mpi to netlib@ornl.gov 4) NEXT MPI MEETING Here are some details on the MPI meeting which is set for February 17th-19th 1993 in Dallas. The meeting site will be the: Bristol Suites 7800 Alpha Road Dallas, TX 214-233-7600 The room rate is $89.00. When making a reservation tell them you are with the MPI meeting. TBS Shuttle Service will be providing complimentary shuttle service to and from the airports. If you fly into DFW, use their courtesy telephone and dial 03. If you fly into Love Field, you'll have to use a pay phone. They can be reached at 817-267-5150. Upon boarding the shuttle, refer to the MPI meeting. The registration fee for the meeting will be $75. Please make checks and POs payable to University of Tennessee. We will collect this at the meeting. The registration fee will go for coffee breaks, meeting rooms, AV and printer rentals. We should plan to start at 1:00 pm February 17th and finishing about noon on February 19th. The format of the meeting is: Wednesday, February 17 1:00 pm to 8:00 pm point to point subcommittee meeting 5:00 pm to 6:00 pm--On our own for dinner. after dinner: other subcommittees meet Thursday, Febrauary 18 8:00 am to 12:00 pm Collective communication subcommittee meeting 1:00 pm to 6:00 pm Subcommittee reports presented to the main group 6:00 pm to 8:00 pm-- The group dinner somewhere in the area. The hotel will provide round trip van transportation. Friday, Febraury 19 8:00 am to 12:00 pm Subcommittee reports presented to the main group For future planning here is a tentative list of dates, roughly 6 weeks apart, for the series of meetings: March 31-April 2 May 19-21 June 30-July2 From owner-mpi-comm@CS.UTK.EDU Mon Jan 18 14:20:28 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21493; Mon, 18 Jan 93 14:20:28 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27089; Mon, 18 Jan 93 14:19:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 18 Jan 1993 14:19:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27075; Mon, 18 Jan 93 14:18:59 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA15953; Mon, 18 Jan 1993 14:18:58 -0500 Date: Mon, 18 Jan 1993 14:18:58 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9301181918.AA15953@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Contexts group A new subcommittee and associated email group has been set up to responsible for MPI communication contexts. The current group membership is as follows: tony@cs.msstate.edu dongarra@cs.utk.edu lusk@mcs.anl.gov weeks@convex.com hender@fsl.noaa.gov john@cs.wmich.edu csimmons@us.oracle.com walker@msr.epm.ornl.gov If you would to join (or leave) this subcommittee please send email to me. You can send email to the group at mpi-context@cs.utk.edu. You can get the archived email for this group by sending the following message: send mpi-context from mpi to netlib@ornl.gov Regards, David From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:29:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19476; Tue, 19 Jan 93 12:29:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24138; Tue, 19 Jan 93 12:28:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:28:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24130; Tue, 19 Jan 93 12:28:25 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26202; Tue, 19 Jan 93 17:28:13 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02004; Tue, 19 Jan 93 10:30:14 MST Date: Tue, 19 Jan 93 10:30:14 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02004@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu > From: john@cs.wmich.edu (John Kapenga) > To: mpi-comm@cs.utk.edu > > Although the question of IO specification in the MPI-1 has been > raised before, I think it still needs to be addressed. Omission > of any IO requirements means no program can be portable. If IO > specifications are not included the introduction should make a > strong case for the reasons. We (Tom Henderson and myself) agree and have also suggested such a proposal. Ours had a little more high level stuff, but not much. We do think you need some specification of how basic I/O performs in a parallel environment to provide a high level of portability. We think opportunities for parallelism can be expressed as well by routines that are loosely snychronous. > ... > Any attempt to specify high level parallel IO at this point in time > seems premature. Perhaps there can be a middle ground. We could > specify low level IO, compatible with ANSI/ISO/POSIX standards > and perhaps allow an inquire function to test for its presence. > I am willing to draft a Level-1 suggestion and a Level-2 suggestion. > I think that is a good idea and would be interested in cooperating on such an endeavor. > I'm not sure which mailing list to put this on, but I think the > issue needs to be addressed quickly (even if the decision remains > to ignore IO). Neither am I, let's hope it's not another subcommittee ;-) > > john > Leslie Hart (hart@fsl.noaa.gov) From owner-mpi-comm@CS.UTK.EDU Tue Jan 19 12:32:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19727; Tue, 19 Jan 93 12:32:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24283; Tue, 19 Jan 93 12:31:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 19 Jan 1993 12:31:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24275; Tue, 19 Jan 93 12:31:22 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26204; Tue, 19 Jan 93 17:28:14 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA02007; Tue, 19 Jan 93 10:30:15 MST Date: Tue, 19 Jan 93 10:30:15 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9301191730.AA02007@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov, csimmons@us.oracle.com Subject: Re: "Message Capsule" Proposal (long) > From: Charles Simmons > To: bernardo@fsl.noaa.gov, hart@fsl.noaa.gov, hender@fsl.noaa.gov > Subject: Re: "Message Capsule" Proposal (long) > > As feedback... > > The message capsule proposal might be a bit high level. E.g. it > seems to implement the topmost layer of the ISO 7-layer communications > model which deals with transmitting data between processes running > on heterogeneous machines. I'm not familiar with with the ISO 7-layer approach, but what we were proposing was a place for each local process (or thread) to store information about one or more messages. Other words for capsule might be "handle" or "envelop". > > " > 6.1 Advantages > > - The number of separate send and receive routines is greatly reduced > without sacrificing functionality. > - A user who is used to "common practice" can use the simplified > versions of the routines. > - A user who wants more flexibility only needs to learn about the features > required for his/her specific application. (For example, if I only need > contiguous messages, then I don't need to know anything about strided data > items. If the receiving process always knows the data description of > received messages, then I don't need to know about the probe and "info" > routines.) > - "Hidden states" are removed so multi-threaded applications won't get > confused. > - Encapsulation of features in message capsules allows new features to be > added later without modifying syntax of existing routines. A new feature > would require addition of one or more new routines to modify and examine > message capsules. > " > > While the number of separate send and receive routines is reduced, > the number of other routines are correspondingly increased (e.g. attach > and info routines). The increase in routines is not combinatorial, but rather additive. For three new modes each of which have two possible settings, there is a maximum of six (6) new routines (ok, maybe 12 if there are other inquiry functions). In another approach the number of routines would be *mulitplied* by two for each new mode (i.e. multiplied by 8). > > Even in the MPI approach, users can limit themselves to simple versions > of various rotuines. Agreed, but I don't think the document has been taken to heart as a syntactical or semantical starting point. > > I don't see how capsules keep multi-threaded applications from > getting confused. If you have two threads in a receiver that are > each waiting for messages that look pretty much the same to the OS > or compiler or library code, it would appear to be fairly easy to > give the messages to the wrong threads. The sender really needs > to attach a thread identifier to a message under certain circumstances. > This responds to some issues that hidden state cause with multithreading. Such as what does the last message mean? We don't claim that it deals with which thread should be a target in a send. It just removes confusion if you probe and then wish to receive the message that you probed about. > > [There was a brief thought I had that this proposal reminded me of... > For certain types of applications, it would be possible for the sender > to know the layout of data in a receiver's address space. In an RPC > application, the receiver could transmit a description of the layout > to the sender. In an SPMD application, the sender might just know without > being told. This would allow the sender to prepend the layout description > to her message, and would allow the receiving OS to splatter received data > directly into user address space.] > > G'luck, Chuck Thanks for your input, we wanted to start a discussion about these issues. (BTW, your response was directly to us, so we included it in its entirety to mpi-comm). Leslie Hart Tom Henderson Bernardo Rodriguez hart@fsl.noaa.gov hender@fsl.noaa.gov bernardo@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Tue Jan 26 16:04:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21467; Tue, 26 Jan 93 16:04:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20893; Tue, 26 Jan 93 16:03:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 26 Jan 1993 16:03:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20885; Tue, 26 Jan 93 16:03:47 -0500 Received: by enet-gw.pa.dec.com; id AA27890; Tue, 26 Jan 93 13:03:06 -0800 Message-Id: <9301262103.AA27890@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Tue, 26 Jan 93 13:03:45 PST Date: Tue, 26 Jan 93 13:03:45 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: POSIX 1003.4 message queues considered ? Has anyone involved in the MPI effort looked into POSIX 1003.4 (Realtime) message queues ? -Ed From owner-mpi-comm@CS.UTK.EDU Wed Jan 27 12:03:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08441; Wed, 27 Jan 93 12:03:40 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09490; Wed, 27 Jan 93 12:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 27 Jan 1993 12:02:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09480; Wed, 27 Jan 93 12:02:24 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02128; Wed, 27 Jan 93 11:02:21 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA16675; Wed, 27 Jan 93 11:02:16 CST Message-Id: <9301271702.AA16675@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: minutes of January MPI meeting Date: Wed, 27 Jan 93 11:02:15 CST From: Rusty Lusk Minutes of the Message Passing Interface Standard Meeting Dallas, January 6-8, 1993 The MPI Standards Committee met in Dallas on January 6-8, 1993, at the Bristol Suites Hotel in North Dallas. This was the third meeting of the MPI committee, but the first following the format used by the High Performance Fortran Forum. There were both general meetings of the committee as a whole and meetings of several of the subcommittees. Because interest in the Point-to-Point communications and the Collective communications was so general, these met as committees of the whole. No formal decisions were taken at this meeting, but a number of straw votes were taken in the subcommittees. These are included as part of the reports on the work of the subcommittees. These minutes were taken by Rusty Lusk (lusk@mcs.anl.gov) with some additions by Bob Knighten. Marc Snir's notes on the point-to-point subcommittee meetings are included here as well. These minutes are quite long. If you want to see the important topics you can search for --- and this will quickly the lead to each topic (and a few other things.) January 6 --------- ------------------------------------------------------------------------------- General Meeting ------------------------------------------------------------------------------- The meeting was called to order by Jack Dongarra at 1:30. Jack Dongarra presented the rules and procedures that had been circulated in the mailing list. In general, they say that we intend to operate in very open fashion, following the example set by the High-Performance Fortran Committee. He also described the subcommittee structure. For details, see the mailing list, A tentative schedule for future meetings was presented, which was amended on the last day (see there). All meetings will be in Dallas at the Bristol Suites. Steve Otto will coordinate the production of the document. He will obtain a set of LaTeX macros from the HPF Committee and distribute them to the subcommittee heads. It was suggested by Bob Knighten that the Executive Director arrange for copies of all pertinent documents be provided at the meetings. Dennis Weeks, who is somewhat local (Convex), volunteered to help with the relevant copying. The attendees were: Ed Anderson Cray Research eca@cray.com James Cownie Meiko jim@meiko.co.uk Jack Dongarra UT/ORNL dongarra@cs.utk.edu Jim Feeney IBM-Endicott feeneyj@gdlvm6.vnet.ibm.com Jon Flower ParaSoft jwf@parasoft.com Daniel Frye IBM-Kingston danielf@kgnvma.vnet.ibm.com Al Geist ORNL gst@ornl.gov Ian Glendinning Univ. of Southampton igl@ecs.soton.ac.uk Adam Greenberg TMC moose@think.com Bill Gropp ANL gropp@mcs.anl.gov Robert Harrison PNL rj_harrison@pnl.gov Leslie Hart NOAA/FSL hart@fsl.noaa.gov Tom Haupt Syracuse U. haupt@npac.syr.edu Rolf Hempel GMD hempel@gmd.de Tom Henderson NOAA/FSL hender@fsl.noaa.gov C. T. Howard Ho IBM Almaden ho@almaden.ibm.com Steven Huss-Lederman SRC lederman@super.org John Kapenga Western Michigan Univ. john@cs.wmich.edu Bob Knighten Intel SSD knighten@ssd.intel.com Bob Leary SDSC leary@sdsc.edu Rik Littlefield PNL rj_littlefield@pnl.gov Rusty Lusk ANL lusk@mcs.anl.gov Barney Maccabe Sandia abmacca@cs.sandia.gov Phil McKinley Michigan State mckinlehy@cps.msu.edu Chuck Mosher ARCO ccm@arco.com Dan Nessett LLNL nessett@llnl.gov Steve Otto Oregon Graduate Institute otto@cse.ogi.edu Paul Pierce Intel prp@ssd.intel.com Peter Rigsbee Cray Research par@cray.com Ambuj Singh UC Santa Barbara ambuj@cs.ucsb.edu Marc Snir IBM snir@watson.ibm.com Robert G. Voigt NSF rvoigt@nsf.gov David Walker ORNL walker@msr.epm.ornl.gov Dennis Weeks Convex weeks@convex.com Stephen Wheat Sandia NL srwheat@cs.sandia.gov ------------------------------------------------------------------------------- Point-to-point subcommittee ------------------------------------------------------------------------------- Mark Snir called the meeting to order at 1:40 p.m. It adjourned at 4:10 p.m. It resumed the following morning at 9:10 a.m. and adjourned at 4:15 p.m. Marc Snir began by summarizing the decisions that we have to make: * which operations? send receive channels? sendreceive? info arguments operations on queues probe? * operation modes sync async local and/or global termination interrupt-driven? * message types (data types) structure of data in core buffer packing * send-receive matching type (We later decided to call this "tag".) sender? * correctness criteria (See Marc Snir's paper in handouts) * heterogeneous operations * name space how processes are addressed flat? structured? implicit/explicit * error handling * interaction with threads, interrupt handlers, remote signalling * special operations for high performance ready receiver? * process startup * syntax/style (The plan is to postpone this for this meeting.) We will prioritize this list and then go through them one by one. (The priorities assigned were more or less in the order listed above.) Two preliminary questions were then discussed: A. Must we worry about multithreaded environments? James Cownie pointed out that threads were coming, in almost all new systems. Most systems have threads now. It was proposed that a process, which could send and receive messages, should be an address space, so that individual threads would not be (MPI-) addressable. B. What about signals? Paul Pierce suggested that we discuss signals first: do we want to support send/receive from interrupt handlers? These two questions were then discussed at length. Dealing with threads argues against the notion of "last message", since that implies state is maintained by the system. There was general agreement that "state" was a ` bad thing, but arguments in favor of state are: Sometimes one doesn't want all of the information available after an operations, so it shouldn't be returned. Having lots of arguments to calls is bad, especially inout arguments. Ways to avoid state are: Structures could be returned Return individual arguments Return tag to do queries on (but they one needs to free it.) Additional out arguments (OK in Fortran 90, but not in C or f77) User passes in storage to be used (so he knows the address), and MPI provides inquiry functions [For more details, see Jim Cownie's mail message of January 4, 1993 entitled: Multifarious] There was a general agreement that system state decreases portability and manageability, and we should decrease it when we can. James Cownie said that We need a reentrant style, and Mark Snir suggested that we try to make all function calls reentrant. When queried no one in the group objected to trying to make all the functions that are introduced in MPI reentrant. Now we began going through the above-mentioned major topics. Which Operations? ----- ---------- We have send and receive. How about send-receive (also called shift)? It can be efficiently implemented, and buffer can be reused. There was a discussion of the "two-body" send-receive (exchange) and the "three-body" version (ring-shift). Variations include those in which the send-buffer is required to be the same as the received- buffer and those in which is is required to be disjoint from the receive-buffer. Al Geist: We should focus on *required* operations. Steve Otto replied that send-receive *is* a required operation. Using "exchange" can help avoid deadlock. It was agreed that there was no consensus on these issues and it was decided to defer this to the collective communication subcommittee. Operation Modes --------- ----- The next topic that Marc Snir raised for discussion was when do send and receive return. Marc described several options: For send: 1) return as soon as possible 2) return when send-buffer is clear 3) return when the corresponding receive has completed For receive: 1) return as soon as possible 2) return when the receive-buffer is full "Receive has completed" means "when the user knows". In other words, when the sender has returned from send, the receiver has returned from receive. There was a general discussion about whether 3) was necessary? dangerous? Robert Harrison said he believed that 3) was the minimal version that was truly portable. Steve Otto pointed out that 3) is CSP-like. Rusty Lusk said that 3) would be easier to prove things about than the others. Adam Greenberg and Paul Pierce pointed out that neither TMC nor Intel have implemented an operation depending on the behavior of the receiver. A straw vote was taken and the vote was 17-3 in favor of having 3) as an option. Marc Snir pointed out that in his original proposal send returns a handle and the status of the handle is then tested for completion of the send operation, and asked if this is this desirable. There was general agreement that something of this sort was desirable, but a variety of alternatives were mentioned It was pointed out that sometimes one wants to wait on multiple outstanding operations. Al Geist prefers separating "wait" into "sendwait" and receivewait" for code readability. Bill Gropp suggested that instead of using handles, one could supply a routine to be called when an operation completes. James Cownie: "This gets really hairy in Fortran". There was a discussion of probing multiple outstanding receives: If the receives return handles, h1 = recv( ... ) h2 = recv( ... ) wait ( h1 or h2) ? wait ( h1 and h2 ) is not needed. Jame Cownie proposed that we supply an operations to *wait* on a vector of handles, which would return one of those that have succeeded. It would return the handle, not the status. A straw vote as taken on this proposal, which passed 17 - 0. So we have: status (handle) wait (array of handles) The send specifies what completion of send means. Handles need to be freed. It was pointed out the only the existence of such an operation has been decided, the semantics are yet unspecified - e.g. issues such as fairness or what wait returns when several complete are not yet specified. There was a long discussion of cancellation of send and receive. It was observed that there are serious implementation problems because of race conditions, freeing resources, etc. A straw poll was taken on including cancel in the initial MPI. It failed 7-19. This was the end of the Wednesday afternoon point-to-point meeting. January 7 --------- The point-to-point subcommittee (now a Committee of the Whole) resumed at 9:15 a.m. on Thursday morning. Marc Snir opened the meeting and summarized the progress so far: 3 ways in which send can terminate sendreceive postponed no cancel of incomplete send operation status and wait (successful status accomplishes same as wait) We did not get to: channels (the idea of trying to bind as soon as possible as many parameters as possible, so that they can be reused.) probe readyreceive Marc noted that channels and readyrecv address similar issues. Probably want only one of these. Do we want either? Rolf Hempel observed that we don't need channels - can depend on operating system to cache the connection information when doing synchronous communication. Adam Greenberg replied: NO! Want to be able to do this all at user level without "smart" OS. Channel creation and use might look like: handle = send_init( ... ) start(handle) wait(handle) free(handle) This is an intermediate point between bundled send/receive and full named channels. Indeed there are many intermediate points based on various early bindings. Is there enough experience to justify a standard? Bob Knighten observed that there has been substantial experience with channels on the iWarp system. There was next a discussion of the ready-receiver semantics proposed by Lusk and Gropp in the handouts. Steve Huss-Lederman said that such operations could make a difference of as much as 25% for matrix multiplication on the Delta. Some doubt was expressed about the universality of this optimization. Question of use of readyrecv by naive users again. Cownie mentions experience again. Greenberg: facilities for efficiency should not make it difficult to write correct programs. Wheat: Don't penalize users who do understand and can take advantage of efficient procedures. General back and forth discussion. Two straw votes were taken: Ready-receiver operations passed 13 - 10 Channels passed 19 - 2 (Marc Snir will write up a detailed proposal) The next topic discussed was the probe operation. Do we want such an operation, and if so, what should be its semantics? Probing must "lock" the message that it finds, else the information returned by the probe may be unreliable. (Consider the multithreaded environment.) Bill Gropp pointed out that probe is often used to find out the length of a message so that a buffer of the appropriate size can be allocated. Marc Snir pointed out that this is a problem with the November document, that we need to know the length of a message ahead of time. Jon Flowers suggested the need for a blocking probe. What is needed is to probe and then to receive the message found via the probe: handle = probe(params) . . . recv(handle) release(handle) Marc Snir pointed out that the handle serves as a lock on the message. James Cownie pointed out that while we agreed to not have a cancel for a send, we do need to be able to cancel receives, since an outstanding receive is permission for the system to write in the user's address space, which is a permission the user may want to revoke. A straw vote was taken on the existence of some form of probe, and it passed 25 to 0. Send-Receive Matching ------------ -------- The next topic is the matching of send and receive. Currently we have to discuss matching on: tag sender group id context id We will also need to discuss the name space issue for messages. Here are three proposals for the predicate that determine whether a message matches a particular receive: 1) simple matching on fields 2) more general, with mask, etc. 3) user defined function Adam Greenberg said that at TMC: A user defined function is used by the system whenever a message is received by a node to decide if it is to actually be received by the application. The parameters to the user defined receive predicate are tag and group. Issue: If most information is encoded in the tag, then the tag protocol must be understood by all users involved in writing a particular application. True, but not a serious problem. Best to identify small class of specific matching parameters (e.g. group) and use tag for everything else. James Cownie pointed out that the matching function, if not too complicated, can (and is, on many systems) done by special communications processors. There was further discussion of the difficulties of having the system call user code for screening messages. Paul Pierce pointed out that receipt of a message by the hardware is a crucial point for performance. There was general discussion of alternative approaches to getting at least some of this. The question of need for this generality was also raised. TMC has a user who wants and uses his own predicate function. Possibilities: (a) select on mask for fields (including a don't care); (b) simple static logical operations on fields; (c) user defined (b) might be match = AND (( message(i) = pattern(i) ) OR mask(i)) fields A straw vote was taken on whether to pursue allowing user-defines predicates. It was decided 26-1 not to allow user-defined functions for this purpose. (b) deferred until a proposal is available. Marc Snir summarized that matching by tag is generally agreed on and that this is not the only item for selection. After some discussion, matching by sender was also generally agreed on. So now, how do we identify a sender? Rusty Lusk spoke in favor of a flat name space, so that processes could be addressed independently of group, etc. There ensued a general discussion of groups, contexts, and the name space. It was pointed out that the name space expected by send could be flat and groups could be implemented by a function that converted any structured name into a flat integer id. Other proposals were to to have name=(rank,gid) with the restriction that this name be usable only within the given group (gid) and the sender must be a member of this group. By default the group would be ALL. Other alternatives mentioned were name=(rank,ALL)=pid and name=(pid,context). This led to a general discussion of context and the relation to groups. Marc Snir pointed out that we could have pid pid,context in which context did not change the meaning of pid. Paul Pierce said that tags and contexts should be separated since they need to be handled in different ways. Marc Snir pointed out that there should be no "don't care" on context. There was a discussion of servers that can process "any" message. This also led into discussion of flat name space vs. hierarchical name space where we would have pid(group, rank) function. Can use context to define groups, but there are other uses as well. Why groups as well as context? What is the difference between context and groups? Cownie: Context is just another integer used in the same manner as tag. Not quite - it is reserved, but what is the meaning of "reserved"? Greenberg was concerned about connecting send/receive behavior with groups. Snir: Suppose a users wants to have two independently written subroutines that use the usual rank notation. Wheat: Similarly want to use rank notation when partitioning machine. Snir: Both contexts and groups are nice, but do we need both? Gropp: Problem with mixing two applications both of which use 0-based indexing will need a larger common name space when they need to communicate. There was a general discussion of the cost of contexts. Cownie observed that context is cheap if only used to distinguish code - obtain a unique context id for the code by means of the "one-dollar random number generator": each author obtains a one-dollar bill, copies the serial number, and then burns the bill. But in general context is not cheaper than groups. Someone asked about spawning additional processes while program is running? Various people raised the question: If use name=(pid,context), does context change the meaning of the pid (i.e. is pid context {or gid or ???} relative.) There was some discussion of message registration. Paul Pierce observed that tag vs. context is only matter of registration. He wants to divorce tag and context for safety. This implies that one cannot use wild card for selecting on context. Various people noted difficulties with mixing tag and context. Adam Greenberg offered: Proposal - always separate tag and context. Have a context, NONE, so that pid with context NONE is unmodified, but with other contexts the pid may be relative. [NONE, GLOBAL, BASE] tag, context - must match on context Several people noted that there are two very different uses of context - identification of distinct code and identification of a group of processors. There is state, even distributed state, associated with remapping of processors with groups. POSSIBLE FIELDS FOR SEND/RECEIVE: tag context id group - no wild card - set of processes - registration management - receive only from group - managed by system Marc Snir asked whether we could agree on what would be carried with a message: tag context (like tag, except no wild card; management to be determined) Two straw votes were taken: Having contexts passed unanimously. Having the context *not* modify the process id passed unanimously. Groups ------ Three alternatives: no groups (use send(pid(group, rank), ...) instead) group as explicit parameter in operations use contexts to implement groups The basic difference is: do we want to be able to select on group? Straw vote: yes: 10 no:11 on the capability of selecting by group. (Thursday lunch occurred here) Message Data Types ------- ---- ----- WHAT IS A BUFFER? (Language bindings are going to be important here.) There are many options to consider: a) contiguous bytes (non-controversial) General agreement that 0-length messages should be allowed. b) contiguous buffers of other (implementation specific?) units? b) stride? (parameters: base-address, element-length, stride, number-of-elements) c) basic data types? d) arbitrary structures? e) How will we specify data to be communicated in a heterogeneous environment? f) iovec structures (array of pointers and lengths, as in un*x iovec for reads and writes) Marc Snir pointed out that one possibility is to have separate pack/unpack routines and then just send contiguous buffers. Rusty Lusk pointed out that this requires a copy that may be unnecessary on certain machines. Two choices - pack scattered buffer and send OR send scattered buffer. If the second, then may need a pack that produces the descriptor of a scattered buffer to be used by the send scattered buffer. Straw poll: Use IOVEC type send. Passed 18-1 Basic data types were deferred. Marc Snir observed that up to this point, a message is a set of bytes in storage, but now we are about to consider more meanings: message = sequence of *values* Should we use the same calls for homogeneous and heterogeneous calls? Can we have a fast homogeneous implementation of the heterogeneous calls? Bill Gropp pointed out that the current testbed implementation does this. SEND vs. SENDCHAR, SENDREAL, . . . To be compliant with F77 need to have at least SENDCHAR for correctness (and this is a real issue, e.g. on VAX.) Strictly need to have different for each basic data type (but in practice this is not an issue.) But for other than CHARACTER there is also an efficiency issue. 1. F77 conformance 2. Special problem of CHARACTER 3. Performance 4. Heterogeneity (?) Postpone to language binding discussion. This led into the issue of the general problem of converting types between languages and machines! This in turn led to a discussion of XDR (and mention of other systems such as NDR, ...) XDR supports the basic types (INT, REAL, COMPLEX, CHAR, etc.), array constructors, pack/unpack routines, etc. Do we use the same calls for homogeneous and heterogeneous systems? Can we have a fast implementation of heterogeneous procedures for a homogeneous system? What about a "message envelope" that specifies the environmental aspects of messages (e.g. heterogeneity features such as XDR.) When we talk about heterogeneity, do we expect MPI libraries from different vendors on different machines to cooperate? Include general SEND as SENDBYTES? Agreed that do not want SEND in homogeneous to require type information needed for heterogeneous environment. There was a discussion of whether we have to pick an interchange format, for example XDR. There seemed to be some agreement that we do (as MPI implementations from different vendors have to be able to communicate with one another), but no vote was taken. Error Handling ----- -------- The main issue here is whether an error detected by an MPI routine should result in the calling of an error-handler or return of a return code. Other issues are how much of error handling should be standardized as opposed to implementation-dependent, and how much user control there should be over error-handling. There are two types of error environments - soft (recoverable) and hard (unrecoverable). In a soft error environment there is the opportunity for cleanup on the part of both the "application" and the system, while in the hard error environment the system will cleanup and terminate the application. Choices: An mpi routine always returns (though it may return with an error code.) An mpi routine may call an exception handler There may be a default exception handler and there could be a user-installable one as well. Library writers may want to handle errors differently from how a user program wants to handle them (or have them handled by the system). Robert Harrison described the error modes used in TCGMSG and p4: A process has a user-settable state that determines whether an error should result in a (hopefully) graceful takedown of the program or in a error return code. Paul Pierce described the Intel method which uses two syntactically distinct classes of functions. For one class an error results in a message being printed and the process in which the error occurred terminating. For the other class an error code is set. There was some discussion of the problem of maintaining state in a multithreaded environment. Two straw votes were taken: Do we want a version of MPI that calls an exception handler: yes: 23 no: 0 Do we want a version with return codes: yes: 19 no: 1 Specific discussion of modes or "shadow" routines was deferred. Correctness Criteria ----------- -------- This concerns defining what is a correct implementation of MPI An assumption that had to be restated several times during the meeting is that MPI assumes a reliable underlying communication system, i.e. MPI does NOT address what happens if that fails. Two specific topics are order of messages and resource bounds. There was discussion about whether order preservation is required; that is, for messages from one process to another, messages are received in the order they are sent. Maintaining message ordering is troublesome, but seems essential for conveniently writing reliable portable programs. But then comes the question of what exactly this means, particularly with multithreaded processes! What is the effect of probe on the ordering of messages? Straw vote in favor of requiring order preservation: yes: 23 no: 4 On the issue of correctness with regard to resource exhaustion, Marc Snir suggested the following example: Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv What should an implementation be required to do with this program? On the CM-5 this will always deadlock. On Intel and Meiko machines this will "usually" work (but how does one specify exactly when it will work.) Exchange is an even nastier case. ------------------------------------------------------------------------------ Summary of both Wednesday and Thursday point-to-point subgroup meetings by Marc Snir 1. Multithreaded systems and signal handlers. Should these be of concern to us? No vote was taken, but the general feeling was that we should try to define the various communication calls so that they do not rule out the case where the communicating process is multithreaded. The implications seems to be that all calls should be made reentrant, and the communication subsystem is, from the view-point of the application code, stateless. (With one with one obvious exception, namely that due to posted receive or send buffers, and perhaps additional exceptions having to do with global "modes", like error handling mode. 2. Small or large library? No vote taken. The general feeling is that we should provide many options for the advanced programmer that wants to optimize code (otherwise, all "interesting" codes will use non-portable features, but set the syntax so that the use that uses only the "core" functions need not be burdened by the availability of advanced options. 3. What functions? Clearly, SEND and RECEIVE General sentiment that a combined send-receive would be nice ("most used function on CM"), but discussion postponed until we have a proposed definition: Do we we want an exchange (source=dest), or a 3-body function (source != dest), or allow for both? do we want send_buffer identical to receive_buffer, or disjoint from receive_buffer, or allow arbitrary overlap between the two? What attributes are shared by sent message and received message, if at all? WAIT, STATUS and PROBE functions, and persistent handles are discussed later. 4. What modes? We want blocking and nonblocking sends and receives (blocking -- returns when operation terminated; nonblocking -- returns as soon as possible and a different call is needed to terminate the operation). We want synchronous and asynchronous modes (Synchronous -- operation terminated when terminated at all participating nodes. Asynchronous -- operation terminated when terminated at the calling node; e.g. a send terminates asynchronously when the sender buffer can be reused. Please let me know if you dislike this terminology and prefer something like "local" and "global".) The vote went 17-2 toward having a synchronous SEND (completes when RECEIVE has completed, i.e. when the corresponding WAIT has returned, or STATUS has returned successfully.) We did not discuss whether we want all 4 combinations of blocking-nonblocking and synchronous-asynchronous, or just 3 (blocking synchronous, blocking asynchronous and nonblocking asynchronous). We did not discussed explicitly, but "kind of assumed" that any SEND mode can match any RECEIVE mode. 5. How does one complete a nonblocking operation? The SEND and RECEIVE nonblocking operations return a handle that can be used to query for completion. WAIT(handle) blocks until operation completed; STATUS(handle) returns as soon as possible, and returns an indication for successful completion. In addition, these operations return information on completed RECEIVES: tag, message length, etc. for the received message. The information is returned in a structure provided by the caller. After return of a WAIT or successful return of a STATUS the operation handle is freed; the system has no more information on the completed operation, and has freed all associated resources. A more complex WAIT is needed, that waits for the completion of one out of several pending operations. Proposed syntax is WAIT(array_of_handles) that returns information on which operation succeeded and its parameters (voted 17 to 0). No CANCEL operation -- Once a SEND or RECEIVE is posted, it must complete. (Voted 19 to 7. Some peoples asked to reconsider at least canceling posted RECEIVEs, even if posted SENDs must complete). 6. Additional operations "ready-receive" SEND. SEND with a promise that a matching RECEIVE is already posted (A program where such SEND occurs with no preceding matching RECEIVE is erroneous and, hopefully, the implementation detects this error.) The justification is "it exists on some machine" and "it can improve performance by 25% on Delta". Accepted by 13 against 10. Persistent handles. Created by SEND_INIT(params) (resp RECV_INIT(params). can now be repeatedly used to send/receive messages with these parameters, and then explicitly destroyed. Supported by 19 against 2. PROBE. Allows probing for messages available to receive. Justification - "provides a mechanism to allocate memory to a message of unknown length, before it is received". The proposed mechanism is PROBE(params) that returns a lock to a matching message if there is a matching message that can be received. This message is now locked and can only be received using this lock. This was voted 25 to 0. Some level of uncertainty whether we should also allow to unlock without receiving (why should one want to do this?) 7. What is the buffer argument in SENDs and RECEIVEs? A message is a sequence of values, and as a particular case which is of most interest for homogeneous systems, and for which the syntax ought be simpler, a message is a sequence of bytes. There are various ways of specifying this sequence of bytes. a. Contiguous message: Starting address and length b. Regular stride message: Starting address, number blocks, length of blocks, stride. Voted with no opposition. c. IOVEC: a list of entries, each of which describes a type a or type b message. Voted 18 against 1. There was no discussion on a concrete proposal for typed messages, short of agreement that there should be such. The standard is not going to propose a concrete encoding of typed messages, and a concrete mechanism for message exchange in heterogeneous systems. 8. Matching of SENDs and RECEIVEs. A SEND operation associates with a message the following attributes. a. Sender id. b. Tag c. Context The idea of associated a group id, too, was rejected 11 to 10. The RECEIVE criterion is a Boolean predicate on these attributes of the form. (SENDER_ID = param1) and (TAG = param2) and (CONTEXT = param3). Don't cares are allowed for sender_id and tag, but not for context. Sender_id is determined by system, in the obvious manner, and is absolute (not relative to a group or a context). Tag is under sender control. Context is under sender control, but a yet to be determined mechanism is used to allocate valid context values to processes so as to prevent conflicts. All this was approved with no opposition. The idea of allowing the user to provide its own Boolean function as a receive predicate was rejected 26 to 1 (Reason: "hard to do if the matching is done by a communication coprocessor".) 9. Error handling a. We need a version of MPI where errors generate exceptions (user program halts when an error is detected in an MPI call, or a specific exception handling mechanism is invoked). Voted 19 to 1. b. we need to provide a version of MPI where calls return error codes, and do not cause exceptions, whenever possible. Voted 23 to 0. 10. Ordering of messages Messages sent from the same source to the same destination "arrive in the order they where sent". Voted 23 to 0. The exact implications in terms of order in which RECEIVEs can occur has to be worked out. It was pointed out that this condition may be somewhat hard to define in a multithreaded environment. End of Marc Snir's summary --------------------------------------------------------------------------- Collective Communication Subcommittee --------------------------------------------------------------------------- The Collective Communication Subcommittee was called to order by Al Geist at 4:30 p.m. on Wednesday. It continued until 6:40 p.m. when there was a break for dinner. The meeting resumed at 8:25 p.m. and finally adjourned at 10:10 p.m. Al Geist introduced this as the first meeting, since no real discussion on groups and collective communication took place in Minneapolis. One goal of this committee is to maintain consistency with the point-to-point operations. Any discussion of groups necessarily involves this subcommittee. Collective communication operations can be constructed out of the point-to-point primitives, but are desired because they can be implemented efficiently they are convenient for programmers. The committee then went through the set of collective communication primitives that had been proposed by Al Geist during the email discussions. Broadcast: info = MPI_BCAST(buf,bytes,tag,gid,root) On return, contents of buf for root is in buf for all processes. Al Geist pointed out that the group id here is explicit. Root has to be a member of the group. It was at this point that the committee decided that it would use the word "tag" for message type from now on to distinguish it from "type", which will now always mean type of data. Marc Snir pointed out that for consistency with point-to-point operations, there should be both local termination (the operation returns when the local process has done its part) and global termination (the operation returns when all processes have finished their participation) versions. There followed a discussion of the fact that the point-to-point committee seems to be adopting many different versions of send and receive, and that total compatibility will require many different versions of broadcast. There was a discussion of the reason for the tag parameter in the call. It is needed to disentangle multiple broadcasts occurring at approximately the same time. Paul Pierce described how the system can do this by generating sequence numbers. Others argued that the tag was useful for the programmer in any case, particularly for verifying program correctness. Marc Snir argued that there is a problem (because of the intuition that bcast provides barrier) 1 2 3 send(3) bcast rec(don't care) bcast send(3) bcast rec(don't care) Note that 3 may receive from 2 before 1, i.e. no barrier. Al Geist replied that we need barrier, but broadcast is NOT a barrier. James Cownie initiated a general discussion of whether broadcasts could be received by general receives. This would make it simpler to inherit some of the point-to-point semantics. Al Geist said that broadcast should be consistent with the other collective operations, all of which are symmetric. Paul Pierce suggested we specify collective communication routines in terms of model P-P implementation. This has consequences in terms of what options can be supported. Marc Snir pointed out that one can't actually specify collective communication in terms of point-to-point operations because they need dynamically-allocated additional space. It was decided to postpone a straw vote on whether all processes participating in a broadcast should do "broadcast" or only the root should "broadcast" and the others should "receive" because of concern about remaining issues, e.g. different varieties of recieves. The discussion of "error code" was deferred until the issue is settled in the Point-to-point communication subcommittee. MPI_GATHER: (see mail archives for details) It was proposed to have a version in which each participant contributes a different amount of information (a general "concatenate" function). Issues raised: How handle the situation where the number of bytes on each processor is different. How specify the type of data? For example one needs to know the size of the data type for various purposes, e.g. when doing recursive bisection. MPI_GLOBAL_OP: (see archives for definition) This does not include the data types. There was a discussion of how the forwarding processors know where to break buffers if the data type is not specified. Paul Pierce suggested that we should separate the case of user-defined combining operations from the system ones, which could be optimized. Robert Harrison suggested that the buffer be specified as (#items, length) at least for the user-defined operations. (Tag would be retained) Someone noted that "bytes" would be different on each processor in the heterogeneous case. Back to GATHER. Many agreed that the interface should be changed, but no proposal was offered. Straw vote on having separate general concatenation, to go along with the gather operation: yes: 18 no: 0 MPI_SYNCH There was general agreement that "BARRIER" would be a better name. James Cownie suggested that a tag argument would be helpful for debugging. There was also some discussion of failure of such a barrier, e.g. because some node fails. It was agreed that this was not a problem peculiar to this particular function. One individual nonetheless argued strongly for some kind of timeout for the barrier. Groups ------ gid = MPI_MKGRP (list of processes) There was much discussion of the format of the process list. As defined MKGRP defines a group as a subset of a pre-existing group. One alternative would be to allow creating a group consisting of processes from a number of other groups. (NB Identification of processes is unspecified. This is a task for the Point-to-point Communication Subcommittee.) MKGRP provides an implicit barrier among the processes joining the group. There are a number of problems about making sure that gid is uniform and known across the system. This is an efficiency issue. Should it be possible to SEND to a (gid,rank) pair? Marc argued that one should do Point-to-point communication only within a group, not between groups. Note that groups are constant - cannot add or delete members from a group. Also group creation is a barrier for the processes that are part of the group. This raises the question of how the processes joining the group know that they are joining. What is utility of groups? Certainly at present the only commonly used group is ALL. MPI_FREEGRP(gid) MPI_GRPSIZE MPI_MYRANK There was a general discussion of how group id's would be generated. Also a discussion of the mapping information: How to map back from my_rank and gid to rank in ALL? (In order to actually do a SEND.) ----- At this point the group broke for dinner ----- The continuation after dinner was an informal general discussion. There were some general question about experience from Al Geist to Paul Pierce. Adam Greenberg expressed interest in discussing channels. Channels are seen as an early binding, (Curryification) of various of the SEND/RECV functions which offer a number of gains in efficiency. There was a discussion of Fortran language bindings (F77, F90, HPF) of MPI. It was agreed by those knowledgable in the area that there are no special issues in regard to HPF. Steve Wheat discussed the Sandia implementation of channels on the Ncube. Sounds very similar to iWarp channels except that they are dynamic in creation. Jim Cownie noted that global-ops are going to result in non-determinism in numeric routines. Jim also elaborated on Meiko's BAD experience with ready_receive function - lots of user problems. Commonly user's try it on small problems and it works and speeds up. But then on large problems things erratically break and the user bitches. Paul Pierce noted that this is essentially Intel's force type and the Intel experience has not been so bad. In particular it is harder to use and does not generally work easily on small problems. Cownie: In general what to do when a ready_receive fails? No reasonable way to raise error. Response: Use a signal. Cownie: GAACK! This is implementation and not viable on all systems. John Kapenga listed six collective communication issues that he considers particularly important. [Missed the list] Other desirable collective communication features that were mentioned: global-exchange; all-to-all communication. What are criteria for including? Proposal: Difficulty of implementation; frequency of use; efficiency gain John Kapenga asked about 2-D and 3-D mesh operations - e.g. shifts? Adam Greenberg said this should be left to compilers. John: No Way! Adam argued that the compiler can recognize opportunity to avoid memory copies. Unless that same facility is available to user the compiler can do much better. The group adjourned at 10:10 p.m. --------------------------------------------------------------------------- Topologies Subcommittee --------------------------------------------------------------------------- The Topologies Subcommittee was called to order by Rolf Hempel at 4:00 on Wednesday. It lasted until dinner. --------------------------------------------------------------------------- Other Subcommittees --------------------------------------------------------------------------- The other subcommittees (Introduction, Formal Semantics, Environmental Enquiry, Language Binding) met informally after dinner on Wednesday. --------------------------------------------------------------------------- Meeting of the Whole Committee --------------------------------------------------------------------------- Thursday, January 7, 4:30 The Agenda for the rest of the meeting was presented: Introduction subgroup report Collective-communications subgroup report Process Topology subgroup report Environmental Inquiry subgroup report Formal Language subgroup report Language Binding subgroup report Profiling (Jim Cownie) Dates for future meetings Report of the Introduction Subcommittee: ------ -- --- ------------ ------------ Jack Dongarra presented the results of the subcommittee meeting that took place Wednesday night. This is essentially the draft that has been available from netlib for the last six weeks. There was some on-the-fly editing by the group at large. The goal of the Message Passing Interface simply stated is to develop a *de facto* standard for writing message-passing programs. As such the interface should establishing a practical, portable, efficient, and flexible standard for message passing. Goals ----- Design an application programming interface (not necessarily for compilers or a system implementation library). Allow efficient communication: Avoid memory to memory copying and allow overlap of computation and communication and offload to communication coprocessor, where available. Allow (but not mandate) extensions for use in heterogeneous environment. Allow convenient C, Fortran 77, Fortran 90, and C++ bindings for interface. Provide a reliable communication interface. User need not cope with communication failures. Such failures are dealt by the underlying communication subsystem. Define an interface that is not too different from current practice, such as PVM, Express, P4, etc. Define an interface that can be quickly implemented on many vendor's platforms, with no significant changes in the underlying communication and system software. The interface should not contain more functions than are really necessary. (Based on the latest count of send/receive variants, this drew a large laugh from the crowd.) Focus on a proposal that can be agreed upon in 6 months. Added: Semantics of the MPI should be programming language independent. Who Should Use This Standard? --- ------ --- ---- --------- This standard is intended for use by all those who want to write portable message-passing programs in Fortran 77 and/or C. This includes individual application programmers, developers of software designed to run on parallel machines, and creators of higher-level programming languages, environments, and tools. In order to be attractive to this wide audience, the standard must provide a simple, easy-to-use interface for the basic user while not semantically precluding the high-performance message-passing operations available on advanced machines. What Platforms Are Targets For Implementation? ---- --------- --- ------- --- --------------- The attractiveness of the message-passing paradigm at least partially stems from its wide portability. Programs expressed this way can run on distributed-memory multiprocessors, networks of workstations, and combinations of all of these. In addition, shared-memory implementations are possible. The paradigm will not be made obsolete by architectures combining the shared- and distributed-memory views, or by increases in network speeds. It thus should be both possible and useful to implement this standard on a great variety of machines, including those ``machines" consisting of collections of other machines, parallel or not, connected by a communication network. It was agreed that explicit remarks that MPI is intended to be usable with multithreaded processes and with MIMD (not just SPMD) programs should be added somewhere. What Is Included In The Standard? ---- -- -------- -- --- --------- The standard includes: Point-to-point communication in a variety of modes, including modes that allow fast communication and heterogeneous communication Collective operations Process groups Communication contexts A simple way to create processes for the SPMD model Bindings for both Fortran and C Support for Process Topologies In addition A model implementation and A formal specification. will be provided. It was proposed that explanation and rationale for the standard would also be provided as would sample programs and a validation suite. This is getting very ambitious. Jim Cownie also wants wrappers available for use by, for example, profiling. The suggestion is to provide "name shift", e.g. __MPI_SEND, etc. so the profiler can have MPI_SEND call __MPI_SEND after doing whatever is useful for profiling. What Is Not Included In The Standard? ---- -- --- -------- -- --- --------- The standard does not specify: Explicit shared-memory operations Operations that require more operating system support than is currently standard; for example, interrupt-driven receives, remote execution, or active messages Program construction tools Debugging facilities Tracing facilities Features that are not included can always be offered as extensions by specific implementations. Report of the Collective Communication Subcommittee: ------ -- --- ---------- ------------- ------------ Al Geist summarized the meeting that took place Wednesday afternoon (described above). Global functions beyond those discussed by the subcommittee, such as all2all or total_exchange, await written proposals. The (whole) committee added that Fortran 90 and HPF would be a good place to look for more combining functions (other than max, min, sum, etc.) It was agreed that a way to supply user-supplied functions would be useful. Issues mentioned include: What is a group? How are groups formed? Are group elements addressable, if so how? Are groups ordered (e.g. for prefix/suffix operations)? Group always an ordered subset of the ALL group? Partitioning? Connection with virtual topologies? This will be discussed when topology group reports. Friday, January 8 ------ ------- - Jack Dongarra called the meeting to order at 9:00. Report of the Process Topologies Subcommittee: ------ -- --- ------- ---------- ------------ Rolf Hempel reported on the meeting held Wednesday afternoon: Motivation: Applications have structures of processes Most natural way to address processes Processor topology is valuable to user Creation of subgroups is a natural way to implement topologies Draft proposal for MPI functions in support of process topologies (by Rolf Hempel) is in the handout bundle. The subcommittee made some changes to the draft. What functions should MPI contain? specification of logical process structure lookup functions for process id's clean interface to other parts of MPI (process groups) What should it not contain? any reference to particular hardware architectures algorithms for mapping of processes to processors If it does this, the user program will be portable, but will contain full information for processes mapping at the logical level. Claim: The use of process topologies is not an obstruction to quick implementation of MPI, since the first implementation can make random assignments. A process topology is assigned to a process group. Copying groups can be used to overlay different topologies on the same processes. All processes in a group call the topology definition function. Inquiry functions provide the translation of logical process location to process id. Supported Topologies: General graph structure: For each process, define the complete set of neighbors for each node. In principle this is sufficient as it covers all topologies. But it is not scalable as all processes have knowledge of all others. we should investigate a scalable version. However, important special cases should be treated explicitly, because regular structures can be specified in a scalable way easier to implement the mapping they cover a large number of applications. A special case: Cartesian structures grids/tori hypercube is a special case Support for creation of subgroups for regular structures will be useful. Special treatment for trees? deferred User-defined topology definition functions? deferred It will be necessary for the inquiry functions to provide information on the hardware topology, so that a user can provide his own mapping function. Marc Snir: We need to consider consistency of mapping alignments, for example an octtree for image processing with a grid structure. Al Geist: What is connection between group and topology. Recall that a group is a linear *ordered* array which is a kind of topology. General discussion of copying topologies and groups Proposal is to have at most one topology per group so can use group id as name for topology. This is reason that there must be a group copy. David Walker: We need closer coordination between the collective communication subcommittee and the topology subcommittee, since groups are central to both. Report of the Environmental Enquiry Subcommittee: ------ -- --- ------------- ------- ------------ Bill Gropp reported that the Environmental Enquiry subcommittee needs to wait and get a better picture of what MPI will contain. Jon Flower again asked for cpu_time. This was discussed, and we were reminded that these were more-or-less rejected at the Minneapolis meeting as not being part of MPI. Standardization should come from POSIX. Marc Snir: Part of the subcommittee's job should be to decide *what* can be enquired about as well as how it will be done. There was general discussion about inquiring about both MPI parameters and implementation parameters. Also if parameter *setting* as well as enquiry should be supported. (Buffer pool sizes, for example). Jon Flower also asked about system hints. He suggested it should be possible to tell the system about implementation specific tuning in a system independent way. Report of the Formal Specification Subcommittee: ------ -- --- ------ ------------- ------------ Rusty Lusk reported the committee was without its chairman, Steven Zenith, but that it viewed its mission as to try to formalize what the other subcommittees decide on. It will probably use CSP, for lack of experience with any other formal specification language. Bob Knighten suggested that the subcommittee look into LIS (Language Independent Specification) that POSIX defined in order to separate semantics from language bindings. Report on MPI -1 (minus one) ------ -- ------ ----------- James Cownie presented an MPI anti-specification. Ya hadda be there, but in case you weren't or just want to be reminded, here is a transcription of Jim's slides. MPI -1 (Jim Cownie) In the spirit of LPF (Low Performance Fortran) * Bindings ONLY for Mathematica Occam ML * No function take arguments or returns result * Point to Pointless communication * 1024 different sends NO receives * Full support for 0 dimensional topologies * User data in a message limited to 1 byte (of 6 data types) BUT 1 KByte of TAG, CONTEXT * Informal semantics - Formal Syntax * All groups are contexts * All contexts are groups * Non blocking wait * Non blocking barrier * All user programs are unsafe & erroneous, they therefore do all their work in the exception handler. --------------------------------------------------------------------------- A Profile/Instrumentation subgroup was formed with Jim Cownie as chairman. Steve Otto, as general editor, will contact subgroup chairmen to begin discussion of editing concerns. Discussion of meeting format. The following was proposed as a format for subsequent meetings, based on the experience with this meeting. Wed. afternoon: point-to-point Wed. night: all subcommittees other than pt-to-pt and collective comm. Thurs. morning: collective communication Thurs. afternoon: subcommittee reports Fri. afternoon: subcommittee reports Meeting Dates: It was decided to moved the next two meetings up a week from when they were tentatively scheduled. The next meeting will be Feb 17-19. The next one after that will be Mar 31-Apr 2 The currently-scheduled May 19-21 and June 30-July 2 meetings may also be moved up as well. Note that July 2 will be a holiday in the United States. From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 01:42:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07543; Wed, 3 Feb 93 01:42:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03537; Wed, 3 Feb 93 01:41:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03528; Wed, 3 Feb 93 01:41:05 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA28816; Wed, 3 Feb 93 00:41:03 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA23933; Wed, 3 Feb 93 00:41:00 CST Message-Id: <9302030641.AA23933@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Date: Wed, 03 Feb 93 00:40:59 CST From: Rusty Lusk At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:16:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27255; Wed, 3 Feb 93 06:16:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23277; Wed, 3 Feb 93 06:14:47 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:14:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gatekeeper.oracle.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23269; Wed, 3 Feb 93 06:14:40 -0500 Received: from wrpyr.us.oracle.com by gatekeeper.oracle.com (5.59.11/37.7) id AA21815; Wed, 3 Feb 93 03:14:37 PST Received: by wrpyr.us.oracle.com (5.59.11/37.7) id AA15597; Wed, 3 Feb 93 03:17:14 PST Message-Id: <9302031117.AA15597@wrpyr.us.oracle.com> Date: Wed, 3 Feb 93 03:17:14 PST From: "Chuck Simmons " To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI Reply-To: csimmons@us.oracle.com Original-To: WRPYR:mpi-comm@cs.utk.edu In-Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU's message of 02-03-93 00:40 Rusty, Bill -- I do view the interface as being layered, but I would organize the layers differently. And I have a couple of arguments that I haven't tried out yet against some of the send/recv routines. First, it sounds to me like you are suggesting that the 72 routines be implemented as 72 simple routines that each make a small number of subroutine calls to very simple routines. And then the do_send routine would do a lot. I guess the do_send routine on a typical machine might look like: /* convert the data into a vector of buffers */ if (options & MPI_OPT_ONEBUF) { iovlen = 1; iovp = bufp; } else if (options & MPI_OPT_STRIDED) { /* malloc local buffer, gather data, set up iovlen and iovp */ } /* translate the data */ if (options & MPI_OPT_HETER) { /* malloc new buffer, translate data into new buffer */ } /* send the data non-blocking and unreliable */ if (options & MPI_OPT_RR) { sendrr (...); } else send (...); /* block if so requested */ if (options & MPI_OPT_BLOCK) { block (...); } My view of the layering is: basic functionality: non-blocking vector send non-blocking vector receive easy-to-use basic functionality: non-blocking single buffer send non-blocking single buffer receive reliable: synchronous vector send synchronous vector receive easy and reliable: synchronous single buffer send synchronous single buffer receive In the above, each of the later listed routines can be implemented on top of an earlier listed routine. I would then examine each of the remaining aspects of the functionality: heterogeneity, ready-receiver, strided messages, and blocking non-synchronous messages. For heterogeneity, I would argue that it is wrong to put the datatype translation filter in the communications channel. In your model, it is relatively easy to do things like: bsendh (dest, tag, buf, FLOAT, 1); or even: bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); but what about bsendh (dest, tag, buf, rpc_t, 1); where 'rpc_t' is some user defined data structure? Thus, this model lacks generality. Only certain simple messages can be heterogeneously sent efficiently through the interface. A completely different paradigm must be used if you want to send a structure instead of an array. Further, consider how the "bsendh (dest, tag, buf, ARRAY|FLOAT, nitems);" routine is going to be implemented on a typical machine: buflen = sizeof (net_float) * nitems; sendbuf = (net_float *) malloc (buflen); if (!sendbuf) return ENOMEM; for (i = 0; i < nitems; i++) { sendbuf[i] = float2netfloat (buf[i]); } bsend (dest, tag, (void *)sendbuf, buflen); free (sendbuf); (In the above, we assume that a "net_float" holds the network normalized format of a floating point number.) It would be as efficient to implement the translation routines completely separate from the communications routines. But, wait, you say, our machine is not typical and we can implement this as: bsend (dest, tag, &header, sizeof (header)); for (i = 0; i < nitems; i++) { bsend (dest, tag, float2netfloat (buf[i]), sizeof (net_float)); } where 'bsend' is assumed to be a machine instruction. My argument against this is that the gain in efficiency is sufficiently small that it's not clear its worth the added complexity. In both implementations, the cpu time will be dominated by the data conversion. In the "typical" implementation, there is an additional cost for moving data between two buffers, but that data movement occurs at the speed of local memory access which is likely to be much higher than network bandwidth. Thus, it is much easier for the hardware to DMA the first implementation and run some other process while that DMA is in progress. In the second implementation, the sending process is essentially hogging the cpu for the full time it takes to transmit each piece of the message. Obviously, if there exists a standard for network normal formats of datatypes, and if hardware vendors implement the data translation facilities in hardware, then we could DMA out the whole message heterogeneously. But surely there are other things that we would rather have hardware vendors implementing for us. The above is essentially the same argument that I'm going to use for strided messages. Strided messages are only efficient if you have hardware that can DMA the strided vector, and then its not so efficient that it's obviously worth the effort. For ready-receiver, I'm going to argue that the communications subsystem should always assume the receiver is ready and then handle the case where this isn't true. If the application has set things up so that this is true, that application will run efficiently whether or not the sender specifies that the receiver is ready. If the application isn't being careful, it will run slowly in your model, but in my model, it will usually run quickly since memory is usually availabe in the receiver, and my model will run as slowly as your model when congestion is present at the receiver. Implementing the ready-receiver places strong semantics on the sender and receiver. The sender has to know whether or not the receiver was implemented in a fashion that allows the sender to assume that the receiver is ready. Finally, let's consider synchronous messages versus blocking messages. For most applications, you want reliability. Implementing blocking non-synchronous messages doesn't help you implement this. You can't reuse the buffer being sent until you know that it has been received at the other side. So you might as well either send the message synchronously, or send it non-blocking and go off and do something else while you're waiting to see if you'll need to retransmit the buffer. [Hmmm... I did mention, didn't I, that I was still trying out a couple of these arguments...] This reduces the number of interface routines to 8. I could probably be convinced that adding 4 more interface routines to support strided messages doesn't overly complicate the interface. -- Chuck *** We use Oracle*Mail running on Oracle V6.2 and an nCUBE2 supercomputer. *** ---- Included Message ---- Received: 02-02-93 22:48 Sent: 02-03-93 00:40 From: WRPYR:owner-mpi-comm@CS.UTK.EDU To: mpi-comm@cs.utk.edu Subject: A suggestion for a multi-level MPI Reply-To: WRPYR:owner-mpi-comm@CS.UTK.EDU X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 01:41:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU At the last meeting, it became apparent that many people were uncomfortable with the large number of routines (1024 sends?) that we seemed to be going towards. There are good reasons for the standard not to be incomprehensibly complex. There are good reasons also for it not to be crippled by lack of functionality. Are we stuck? There are two ways to deal with this problem. This note is a proposal that we adopt both of them. The reason the problem can be solved is that the sets of options that we have been considering (blocking or not, contiguous buffer or not, heterogeneous machines or not, etc.) are orthogonal, and one of a large number of possible send routines can thus be specified simply and without extra parameters. In addition, certain subsets of the set of all operations can be identified and packaged for ease of use. Here is a suggestion for how to organize the routines and end up with a powerful, flexible, yet easy-to-use library of point-to-point functions. (We assume that these will all be renamed to have an MPI prefix, to avoid name-space pollution) Level 4 (very restrictive, very simple): send(dest,tag,buf,len) recv(dest,tag,buf,len) Both of these block until the operation is locally complete. The buffer is a contiguous sequence of bytes. No special protocols or translation for heterogeneous machines. This level is enough to express lots of parallel algorithms. The main reason we need more is for efficiency and control. We now add these in steps. Level 3 (still simple, but allows for overlap of computation and communication and better buffer management, and allows heterogeneous machines to communicate.) bsend(tag,dest,bufadd,buflen) nsend(tag,dest,bufadd,buflen) bsendh(tag,dest,bufadd,datatype,numitems) nsendh(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,buflen) nrecv(tag,dest,bufadd,buflen) brecvh(tag,dest,bufadd,datatype,numitems) nrecvh(tag,dest,bufadd,datatype,numitems) Here the n or b selects block or nonblocking operations, h specifies translation for heterogeneous machines. The restrictions (which allow the parameter lists to be short and the number of routines to be small) are: no noncontiguous buffers, data in message all the same type, no globally synchronized, CSP-like send-receive, and no special protocol (like the ready-receiver protocol discussed at the last meeting). In order to emphasize the orthogonality, we might organize these routine names in a diagram like this: [n][send][ ] [b][recv][h] That is, one makes a choice in each column. Level 2 (access to almost all MPI functionality via a large number of orthogonally named routines): [c][n][send][ ][ ] [s][b][recv][h][rr] [g][s] the first column designates the buffer type (contiguous, strided, general gather/scatter), the second is the termination type (nonblocking, blocking, synchronized), then send or receive, "h" or not for heterogeneous communication, "rr" or not for "ready-receiver" protocol. This is a lot of routines (3 x 3 x 2 x 2 x 2 = 72) but because of orthogonality it is easy to understand. The only restriction here is that there are no "channels": the initialization of a send operation is combined with initiating it. Level 1 (full MPI functionality, in a small, flexible set of routines) handle = init_send(tag,dest) mod_send(handle,option,value) do_send(handle) free_send(handle) along with the corresponding receive routines. The idea is that the "init" routines will set up a usable set of defaults for all the options, specifying only minimal parameters, and then the "mod" routine can be repeatedly called to specify all the options with regard to buffer type, termination type, etc. The point is that at this level we can offer not only all possible options, with a small number of routines, and include the "setup" routines proposed by Marc for increased efficiency, but we also allow room for growth in the standard through the addition of more values for "option" in the "mod" calls. So here's how it might look, this time from the bottom up: Level 1: full functionality, small number of routines, channel setup Level 2: sequences of Level 1 calls of the form init... mod... mod... mod... do... are given specific names in an organized way. This promotes ease of use and also efficiency (fewer calls). Level 3: several of the Level 2 routines are renamed and promoted to this level to provide a useful subset of MPI functionality in a small number of routines, each having the minimal number of parameters. Level 4: ultra-simple interface All four levels would be part of the standard and it would be possible to mix levels in the same program. We have not considered carefully yet how this interacts with groups and contexts, but similar ideas might be useful there, where there are similar problems in providing both functionality and simplicity. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 06:35:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01865; Wed, 3 Feb 93 06:35:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23802; Wed, 3 Feb 93 06:35:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 06:34:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23794; Wed, 3 Feb 93 06:34:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA13771 (5.65c/IDA-1.4.4 for ); Wed, 3 Feb 1993 06:34:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26850; Wed, 3 Feb 93 11:34:48 GMT Date: Wed, 3 Feb 93 11:34:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302031134.AA26850@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03425; Wed, 3 Feb 93 11:33:11 GMT To: lusk@antares.mcs.anl.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Wed, 03 Feb 93 00:40:59 CST <9302030641.AA23933@donner.mcs.anl.gov> Subject: A suggestion for a multi-level MPI Content-Length: 581 The fundamental approach seems fine to me (though Chuck makes some good points...) However there's at least one point to note :- 1) The Fortran binding issue suggests that in the heterogeneous cases there should be a routine for each data type, rather than having the data type as an argument. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 07:27:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11163; Wed, 3 Feb 93 07:27:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25571; Wed, 3 Feb 93 07:26:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 07:26:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ifi.uio.no by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25557; Wed, 3 Feb 93 07:26:08 -0500 Received: from byflak.ifi.uio.no by ifi.uio.no with SMTP id ; Wed, 3 Feb 1993 13:26:05 +0100 From: "\ystein Gran Larsen" Received: by byflak.ifi.uio.no ; Wed, 3 Feb 93 13:26:04 +0100 Date: Wed, 3 Feb 93 13:26:04 +0100 Message-Id: <9302031226.AA26562@byflak.ifi.uio.no> To: mpi-comm@cs.utk.edu Subject: data types and heterogeneity Hi, I have read the minutes from the MPI meeting, January 6-8 1993. The section on message data types mentiones the use of XDR as a protocol for converting types between languages and machines. The standardization of Scalable Coherent Interface (SCI) has lead to a proposed standard for "Shared-Data Formats Optimized for Scalable Coherent Interface Processors", P1596.5. [Working group P1596.5 is chaired by David V. James (dvj@apple.com)] This proposed standard may be of relevance to the MPI-initiative, when it comes to supporting heterogeneous machines. -{\O}ystein From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 11:56:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17274; Wed, 3 Feb 93 11:56:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07365; Wed, 3 Feb 93 11:54:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 11:54:55 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07357; Wed, 3 Feb 93 11:54:53 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01169; Wed, 3 Feb 93 10:54:51 CST Date: Wed, 3 Feb 93 10:54:51 CST From: Tony Skjellum Message-Id: <9302031654.AA01169@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, gran@ifi.uio.no Subject: Re: data types and heterogeneity XDR as currently implemented causes one function call per datum. This feature must be changed (loops moved inside fundamental converter routines) to provide for better performance (less bandwidth impact of converting data types). I am eager to see the report mentioned by Oystein. - Tony From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 13:33:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19846; Wed, 3 Feb 93 13:33:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12323; Wed, 3 Feb 93 13:31:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 13:31:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12315; Wed, 3 Feb 93 13:31:56 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Wed, 3 Feb 93 10:31 PST Message-Id: Date: Wed, 3 Feb 93 10:31 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: Mix + Match, NOT At first look, I definitely like the proposal of Lusk and Gropp. It seems to be the right way to set up a hierarchy of the 1024+ routines that we are inventing. One thing that bothers me: at the last meeting it was said by several people that one would be allowed to mix+match different kinds of sends and recvs to each other. That is, I could do a "cnsend()" and have that consumed by a "sbrecvhrr()". I think we are asking for lots of trouble if the standard states that this is allowed. If it is allowed, an implementor has to worry about 72 x 72 = 5184 different types of behavior for pt-pt communication. I don't see a strong reason for allowing mix+match, and on the implementation side it seems to me that there are many reasons for wanting the restriction that a send of flavor 53 (out of a possible 72) should be matched by a recv of the same flavor, 53. It is true that it seems like many flavors could be mixed, but consider the synchronization choices. If we allow mixing, then the implementor of the sbrecv() routine has to insure that "sensible" behavior results no matter what send routine was used. As I understand it, this means the implementation cannot be done in an orthogonal manner. I'm not even sure we could define "sensible" for all 5184 combinations! Comments? Am I missing something? --Steve Otto From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 18:43:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29539; Wed, 3 Feb 93 18:43:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27997; Wed, 3 Feb 93 18:41:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 18:41:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27989; Wed, 3 Feb 93 18:41:32 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA14014; Wed, 3 Feb 93 18:35:11 EST Date: Wed, 3 Feb 93 18:35:11 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302032335.AA14014@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: A suggestion for a multi-level MPI James Cownie remarks on needing a seperate send routine for each data type in the Fortran Binding. For both the C and Fortran versions there could be seperate calls setting up the Data Addresses (no Data need be moved) and then a single send could take the value returned by the Data Address Setup Call. No system call required on the Data Setup Call. This might require that the array passed to the Data Setup Call not be modified before the send call is made, to avoid recopying it. This also allows later extensions to other Data Addressing modes, without changing the send/recieve syntax. This reduces the number of calls by almost a factor of three, uncouples the data location from the send/recieve syntax and allows arguement types on all calls to be correct and fixed. It does make a bit more processing (trival) and the need for the user to mamage the Data Setup return value and not modify the address array (two of its bad points). john From owner-mpi-comm@CS.UTK.EDU Wed Feb 3 23:45:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02591; Wed, 3 Feb 93 23:45:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09607; Wed, 3 Feb 93 23:43:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Feb 1993 23:43:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09599; Wed, 3 Feb 93 23:43:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00850; Wed, 3 Feb 93 22:43:44 CST Date: Wed, 3 Feb 93 22:43:44 CST From: Tony Skjellum Message-Id: <9302040443.AA00850@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu Subject: Anonymous ftp site aurora.cs.msstate.edu A paper of relevance to the MPI project, describing "Zipcode 1.0" is available by anonymous FTP, in the pub/reports subdirectory: zipcode_sep92.ps.Z The site is aurora.cs.msstate.edu. - Tony Skjellum From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 00:02:32 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02797; Thu, 4 Feb 93 00:02:32 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10374; Thu, 4 Feb 93 00:01:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 00:01:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10364; Thu, 4 Feb 93 00:01:38 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA08517; Wed, 3 Feb 93 23:01:36 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA26443; Wed, 3 Feb 93 23:01:34 CST Message-Id: <9302040501.AA26443@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels Date: Wed, 03 Feb 93 23:01:33 CST From: Rusty Lusk Chuck -- Thanks for your comments. They are of two kinds. The first kind are about the suggested multiple-level organization of the library. There I think I didn't explain some things well enough, and will try to do better here. The other kind are about what should be in or out of the library altogether, and there I can only try to summarize what people talked about in Dallas; the things we incorporated are just the things that the straw votes indicated people wanted in the standard. | | I do view the interface as being layered, but I would organize the layers | differently. And I have a couple of arguments that I haven't tried out | yet against some of the send/recv routines. | | First, it sounds to me like you are suggesting that the 72 routines | be implemented as 72 simple routines that each make a small number of | subroutine calls to very simple routines. This is not at all what we intended to suggest. We used the word "levels" rather than "layers". The upper levels could be *defined* in terms of the lower ones (That's what makes them upper-level) but the intention is that they would *not* be implemented as calls to the lower-level routines. In fact what we called level 2 (large number of routines) exists partly to avoid the multiple subroutine calls necessary in level 1 (where there are a small number of routines.) In fact, level 2 routines don't need to use the data structures implied by the get_handle-mod_handle-use_handle-free_handle at all. They just have to do what they say they do, as efficiently as possible. | | For heterogeneity, I would argue that it is wrong to put the datatype | translation filter in the communications channel. | Our discussions seemed to indicate that people want it there. The popularity of systems like PVM, p4, Express, and others that handle this for the user attests to this. I think the reason is portability; people like being able to move code unchanged from a heterogeneous network to a multicomputer and back. The system can (relatively easily) handle all the translations, including figuring out whether it is necessary at all or not. Many users hardly even know about the non-portability of data formats; they think (in my opinion rightly) that the communication system should handle that if they want it to. | In your model, it | is relatively easy to do things like: | | bsendh (dest, tag, buf, FLOAT, 1); | or even: | bsendh (dest, tag, buf, ARRAY|FLOAT, nitems); | but what about | bsendh (dest, tag, buf, rpc_t, 1); | where 'rpc_t' is some user defined data structure? Thus, this model | lacks generality. Not quite true, although not entirely false. In the "draft implementation" of mpi that you can get from here, we use an iovec-like structure which is a vector of (addr,datatype,nitems) triples to describe a scattered, mixed type buffer. An arbitrary structure can be sent this way, provided the user builds the structure descriptor (It's too bad compilers don't do this for us.) | | It would be as efficient to implement the translation routines completely | separate from the communications routines. | .... | | The above is essentially the same argument that I'm going to use for | strided messages. Strided messages are only efficient if you have | hardware that can DMA the strided vector, and then its not so efficient | that it's obviously worth the effort. Machines do have this kind of hardware, and we can expect to see more of it. One thing that I will continue to argue in favor of is giving the user access to as much of the speed the hardware can deliver as possible. | | For ready-receiver, I'm going to argue that the communications subsystem | should always assume the receiver is ready and then handle the case where | this isn't true. | Some existing machines can perform *significantly* faster if the user can relieve their system of some buffer-management chores. If we don't give the power users access to this efficiency, they simply won't use MPI. And it will come to pass that everyone will agree that to get real efficiency, you have to use the vendor's unique routines. Or you have to use "MPI with the XXX extension package". Portability will be destroyed. This is a way the MPI effort could fail. | | Implementing the ready-receiver places strong semantics on the sender | and receiver. The sender has to know whether or not the receiver was | implemented in a fashion that allows the sender to assume that the receiver | is ready. | Agreed. The naive user can use what we call the Level 3 or Level 4 routines, and never read the chapter in the manual that describes "ready-receiver" semantics and requirements. The power user can use them to write *portable* fast code. | | Finally, let's consider synchronous messages versus blocking messages. For | most applications, you want reliability. Implementing blocking | non-synchronous messages doesn't help you implement this. You can't reuse | the buffer being sent until you know that it has been received at the other | side. So you might as well either send the message synchronously, or send | it non-blocking and go off and do something else while you're waiting to see | if you'll need to retransmit the buffer. | Most people want to compute while the message is in transit, and even while the buffer is being emptied. They don't want this capability taken from them. | | This reduces the number of interface routines to 8. I could probably | be convinced that adding 4 more interface routines to support strided | messages doesn't overly complicate the interface. | The Level 4 routines are only 2. More routines are needed only for more efficiency and functionality. In the last few weeks I have talked to lots of people about the MPI effort, and they are mainly afraid that we will leave something important out. If we don't supply all the functionality and efficiency that we possibly can, MPI will fail. The problem is to do so without making the package too complicated. Our posting about multiple levels was merely an attempt to demonstrate that it could be done. Regards, Rusty Lusk & Bill Gropp From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 01:30:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04177; Thu, 4 Feb 93 01:30:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13427; Thu, 4 Feb 93 01:29:10 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 01:29:09 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13419; Thu, 4 Feb 93 01:29:08 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12505; Thu, 4 Feb 93 01:29:05 -0500 Date: Thu, 4 Feb 93 01:29:05 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040629.AA12505@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Steve Otto proposes that the 72 (or 648 if each data type is separate) sends be matched to the same type of recv. While I agree with Steve that we will not be able to define much less test the 5184 combinations if we allow otherwise, There are obvious cases where the user would want to mismatch send/recv For example, she may have to do a blocking send, but can get away with a non-blocking recv. Not all combinations make sense, but we can't restrict the user to no mismatches. Al Geist From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:10:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05537; Thu, 4 Feb 93 02:10:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14590; Thu, 4 Feb 93 02:08:44 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:08:43 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14582; Thu, 4 Feb 93 02:08:41 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA12583; Thu, 4 Feb 93 02:08:39 -0500 Date: Thu, 4 Feb 93 02:08:39 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9302040708.AA12583@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. >Most people want to compute even while the buffer is being emptied... Actually most people don't want to think or know about how messages are sent they just want it reliable and fast. Only high priests are concerned with the pain of non-blocking send. >If we don't supply all the functionality that we possibly can, >MPI will fail. The problem is to do so without making the package >too complicated. The complication I am seeing in MPI will also make it fail. 72 sends?? Appears to violate several of our goals, like similar to existing packages. small set of routines. >The naive user never has to read the chapter in the manual... I hope the user doesn't have to make room on his shelves for volumes of the MPI manual. Al From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 02:41:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05708; Thu, 4 Feb 93 02:41:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15558; Thu, 4 Feb 93 02:40:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 02:40:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15544; Thu, 4 Feb 93 02:40:30 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA08675 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 08:39:03 +0100 Received: by f1neuman.gmd.de id AA14246; Thu, 4 Feb 1993 08:38:42 +0100 Date: Thu, 4 Feb 1993 08:38:42 +0100 From: Rolf.Hempel@gmd.de Message-Id: <9302040738.AA14246@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Cc: gmap10@f1neuman.gmd.de I was glad to read Steve's proposal not to allow mismatches between different types of sends and receives. If we don't follow that line, we will most likely cause many problems to the implementors of MPI which we cannot forsee at the moment. Also, the requirement of matching sends and receives could be used to check the correct behaviour of the user program in some debugging mode. Al's counter-example of course makes sense, but if you have to use a blocking send, you can get the same by using the non-blocking version followed directly by the wait-for-completion. I'm looking forward to seeing an example where you really need the mismatch. Rolf From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 03:16:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05912; Thu, 4 Feb 93 03:16:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17215; Thu, 4 Feb 93 03:14:56 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 03:14:53 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17205; Thu, 4 Feb 93 03:14:51 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA15627; Thu, 4 Feb 93 03:08:28 EST Date: Thu, 4 Feb 93 03:08:28 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302040808.AA15627@cs.wmich.edu> To: mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... Rolf asks -- > ... > followed directly by the wait-for-completion. I'm looking forward to > seeing an example where you really need the mismatch. > > Rolf By using the wait_for_completion you never need a blocking send/receive operation. The code looks a bit ugly. Right now we have to have a "mismatch" because we don't have a synchronized send/receive pair ;-). Actually I thought the reason that the synchronized send/recieve pair was there initially was to help insure correctness in matching calls, as well as a possible simpler implementation. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 07:03:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07047; Thu, 4 Feb 93 07:03:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01249; Thu, 4 Feb 93 07:02:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 07:02:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01241; Thu, 4 Feb 93 07:02:28 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA09794 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 07:02:22 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA02743; Thu, 4 Feb 93 12:02:17 GMT Date: Thu, 4 Feb 93 12:02:17 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041202.AA02743@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03570; Thu, 4 Feb 93 12:00:32 GMT To: otto@iliamna.cse.ogi.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Steve Otto's message of Wed, 3 Feb 93 10:31 PST Subject: ~ (Mix + Match, NOT) Content-Length: 2480 The only place where I can see a need for matching of send and receive formats is in the heterogeneous case. Here I would be happy to insist that a heterogeneous send be matched to a heterogeneous receive. Since I don't want to enforce in MPI the implemntation decision of whether the data type conversion occurs at sender or receiver I need the data type information in both places. I can see NO reason to restrict any other combinations. Discussion ---------- There are two major areas which cause the many sends 1) layout of data in store. (contiguous, strided, by iovec) 2) synchronisation behaviour (return asap, return when buffer free, return when data received at other end) Data layout ----------- Data layout is an issue which is entirely local. Why does it matter to the receiver how the sender chose to layout her buffer (or the converse) ? This information hiding is one of the things which is an advantage of message passing. The data is necessarily serialised to some degree over the communications medium, so where's the problem ? I believe it is actually HARDER to insist that the buffer sepcifications match at each end than not to do this. (Certainly I have to pass additional information with the message if I'm to check this.) Synchronisation behaviour ------------------------- Once again this seems to me to be largely to do with local issues of buffer management. (Though of course my algorithm may need certain synchronisation behaviour, in which case I must code it using the correct forms of send and recv at both ends.) However I can see no reason to INSIST that the same style of send and receive is used. Indeed there are strong reasons NOT to do so. Consider (for example) a server with multiple clients. I'd certainly like to write it like this :-- Set up lots of non-blocking receives while (running) { wait for a recv service the request get a reply buffer (may need to wait for one) queue a reply requeue the receive buffer } However the clients can simply make requests thus blocking send request blocking receive reply Here is an immediate mismatch in the synchronisations. -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:44:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11973; Thu, 4 Feb 93 09:44:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07037; Thu, 4 Feb 93 09:42:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:42:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07029; Thu, 4 Feb 93 09:42:30 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12554; Thu, 4 Feb 93 07:42:27 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK7mc-0016ZKC; Thu, 4 Feb 93 07:42 MST Message-Id: Date: Thu, 4 Feb 93 07:42 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Rusty and Bill ride again... # Whoa there Rusty and Bill. Let's not play to fast and loose with the facts. # >Most people want to compute even while the buffer is being emptied... # Actually most people don't want to think or know about how messages # are sent they just want it reliable and fast. # Only high priests are concerned with the pain of non-blocking send. Not so; we have those that are not only concerned about being able to compute while messages are in transit, but also wanting to assure that minimal system space is required. That is, they want to manage memory much more tightly than a "buffered" system allows. We have a lot of applications types that find it very hard to fit an application into memory and still allocate sufficient message buffers. By the way, paging is not a viable solution :-). Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 09:59:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12166; Thu, 4 Feb 93 09:59:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07696; Thu, 4 Feb 93 09:58:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 09:58:23 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [132.175.13.2] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07688; Thu, 4 Feb 93 09:58:22 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA12746; Thu, 4 Feb 93 07:58:18 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0nK81y-0016ZKC; Thu, 4 Feb 93 07:58 MST Message-Id: Date: Thu, 4 Feb 93 07:58 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: john@cs.wmich.edu, mpi-comm@cs.utk.edu Subject: Re: 5184 bottles of sends on the wall... as an implementor of blocked and non-blocked messaging systems, I have not experienced any "system" problems associated with matching mixed send/receives. While worrying about the "potential" implementation difficulties, I fear that we make use of the system so difficult that it won't be used (so would that make it easy to use?). Furthermore, as for the theme that seems to prevail concerning features that only "wizards" would use, if we do not provide wizardly type features, then the wizards will not use MPI. Then to whom will the "novice" types refer when they finally see the light of low-performance codes running in a high-performance compute environment? Stephen From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:28:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12742; Thu, 4 Feb 93 10:28:11 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09131; Thu, 4 Feb 93 10:26:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:26:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09123; Thu, 4 Feb 93 10:25:58 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01127; Thu, 4 Feb 93 09:25:56 CST Date: Thu, 4 Feb 93 09:25:56 CST From: Tony Skjellum Message-Id: <9302041525.AA01127@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, john@cs.wmich.edu Subject: Re: 5184 bottles of sends on the wall... Please remember that overlapped send/receive will only make sense on systems with excess memory bandwidth. Hence, it might make more sense to implement a system where this is permitted by the implementor, but not required. Otherwise, calculations might slow down instead of achieving pipelined improvements. As Al points out, it is only the heavy-duty guys who try this sort of stuff... - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:30:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12830; Thu, 4 Feb 93 10:30:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09278; Thu, 4 Feb 93 10:28:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:28:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09270; Thu, 4 Feb 93 10:28:53 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA16295; Thu, 4 Feb 93 10:22:30 EST Date: Thu, 4 Feb 93 10:22:30 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302041522.AA16295@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: Re: 5184 bottles of sends on the wall... My initial concept of mixing IO types was that blocking and non-blocking could be freely mixed, but that synchronized was special and a synchronized send would only match a synchronized receive. The rationale for dropping the synchronized recieve is OK with me. I have no problems with the current proposal. Overlappping communication and computation is the only way to get good prerformance in many applications. It must be assumed to be the norm rather than the exception. I would not be able to use a system that did not support it. john From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 10:46:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13213; Thu, 4 Feb 93 10:46:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10063; Thu, 4 Feb 93 10:45:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 10:45:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10042; Thu, 4 Feb 93 10:45:06 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA12579 (5.65c/IDA-1.4.4 for ); Thu, 4 Feb 1993 10:45:01 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA06100; Thu, 4 Feb 93 15:44:56 GMT Date: Thu, 4 Feb 93 15:44:56 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302041544.AA06100@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA03785; Thu, 4 Feb 93 15:43:15 GMT To: tony@Aurora.CS.MsState.Edu Cc: mpi-comm@cs.utk.edu, john@cs.wmich.edu In-Reply-To: Tony Skjellum's message of Thu, 4 Feb 93 09:25:56 CST <9302041525.AA01127@Aurora.CS.MsState.Edu> Subject: 5184 bottles of sends on the wall... Content-Length: 1025 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. True, and all the more reason to avoid as many copies as possible. After all if taking it from the user buffer consumes bandwidth 1, then the bandwidth cost of a send which needed a copy will be 2+1 = 3. This can be significant. (Of course I'm assuming a "fair" memory system here, in which taking from one port only loses the same from another. I've seen systems where this wasn't true, and this can change the balance.) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:33:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19567; Thu, 4 Feb 93 15:33:00 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25767; Thu, 4 Feb 93 15:30:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:30:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25752; Thu, 4 Feb 93 15:30:34 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04724; Thu, 4 Feb 93 20:30:30 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05190; Thu, 4 Feb 93 13:29:24 MST Date: Thu, 4 Feb 93 13:29:24 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042029.AA05190@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI I especially like "Level 1" of Rusty and Bill's proposal. I would like to see more detail in this section, like examples of mod_xxx() and do_xxx() routines. I think there will be a good deal of support for Level 1 with all the talk that's been going on about handles/invoices/envelopes (of course there's that annoying syntax thing...). I am concerned about how heterogeneous communications are handled (or avoided) in Levels 3 and 4. I would be a lot more comfortable if ALL send() and recv() routines were heterogeneous. Here's a slight modification to the original proposal: Level 4 ("novice users"): send(dest,tag,bufadd,datatype,numitems) recv(dest,tag,bufadd,datatype,numitems) Level 3 ("dangerous users" [like me :) ]): bsend(tag,dest,bufadd,datatype,numitems) nsend(tag,dest,bufadd,datatype,numitems) brecv(tag,dest,bufadd,datatype,numitems) nrecv(tag,dest,bufadd,datatype,numitems) or [n][send] [b][recv] in Bill and Rusty's shorthand. Level 2 ("really frightening users"): [c][n][send][ ] [s][b][recv][rr] [g][s] Here's the arguments: - I don't like the idea of having to tell novice users that they must change all the MPI communication routine calls in their code before their program will run on a heterogeneous system. I think a lot of novice users will be using heterogeneous workstation networks. The current proposal forces these folks to learn at least two versions of send() and recv(). - I don't think there are any significant performance reasons for having separate heterogeneous versions of these calls, as long as we require a receiving process to know what incoming message contents will be (which seems to be consistent with the original proposal). - I don't think that the additional "datatype" argument reduces "ease of use" very much. An incorrect datatype should be no more difficult to debug than an incorrect "bufadd" or "buflen" (and should be easier than a bad "tag" or "dest", especially if deadlock results!). In FORTRAN77, "numitems" should be easier to code portably than "buflen". Even in C, there is little difference in complexity between portable calls to heterogeneous and non-heterogeneous versions: (non-heterogeneous) float mybuf[NUMITEMS]; ... send(dest, tag, mybuf, NUMITEMS * sizeof(float)); (heterogeneous) send(dest, tag, mybuf, MPI_FLOAT, NUMITEMS); - This gets rid of a whole bunch of send() and recv() variant routines (at least 22). - If we really want non-heterogeneous communication, it could be stuck in Level 1 with all the other stuff that's really dangerous for "novices". Another option would be to allow a MPI_BYTE datatype (not sure how this would work in FORTRAN). This may be useful for library developers. I think it's a good idea to keep "raw bytes" away from novice users (especially in FORTRAN). Tom Henderson hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:49:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20005; Thu, 4 Feb 93 15:49:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27824; Thu, 4 Feb 93 15:48:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:47:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27816; Thu, 4 Feb 93 15:47:57 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01577; Thu, 4 Feb 93 14:47:56 CST Date: Thu, 4 Feb 93 14:47:56 CST From: Tony Skjellum Message-Id: <9302042047.AA01577@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, hender@macaw.fsl.noaa.gov Subject: Re: A suggestion for a multi-level MPI Tom, It is an artificial requirement to restrict heterogeneous messages to a single data type. - Tony From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 15:53:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20074; Thu, 4 Feb 93 15:53:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28380; Thu, 4 Feb 93 15:52:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 15:52:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28367; Thu, 4 Feb 93 15:52:36 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04817; Thu, 4 Feb 93 20:52:32 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA05241; Thu, 4 Feb 93 13:51:27 MST Date: Thu, 4 Feb 93 13:51:27 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9302042051.AA05241@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: A suggestion for a multi-level MPI > Tom, > > It is an artificial requirement to restrict heterogeneous messages to a single > data type. > - Tony > I agree. My interpretation of Rusty and Bill's proposal is that messages that contain multiple data types would be constructed using the Level 1 routines-- something similar to Zipcode's invoices. Tom From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:02:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21865; Thu, 4 Feb 93 17:02:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03500; Thu, 4 Feb 93 17:01:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:01:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03492; Thu, 4 Feb 93 17:01:20 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Thu, 4 Feb 93 14:01 PST Message-Id: Date: Thu, 4 Feb 93 14:01 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: 5184; one small point I got into my office late today, and I must say, how exciting to see all the mail concerning 5184! Jim Cownie writes: >The data is necessarily serialised to some degree over the >communications medium, so where's the problem ? I believe it is >actually HARDER to insist that the buffer sepcifications match at each >end than not to do this. (Certainly I have to pass additional >information with the message if I'm to check this.) I did not mean to imply that the standard should RULE OUT mix+match or that it should check for it at run time, merely that the standard should not make the guarantee that one IS allowed to do this. There is an important difference. I really don't believe we can foresee everything, so we should be conservative on what the standard guarantees. Jim agrees that in the case of heterogeneity, it does seem that we want matching: >The only place where I can see a need for matching of send and receive >formats is in the heterogeneous case. Here I would be happy to insist >that a heterogeneous send be matched to a heterogeneous receive. So maybe we don't want to allow all 5184 combinations. OK, at least we are admitting that there is an issue here. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:52:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23307; Thu, 4 Feb 93 17:52:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06478; Thu, 4 Feb 93 17:50:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:50:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06466; Thu, 4 Feb 93 17:50:53 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA17493; Thu, 4 Feb 93 14:50:39 PST Message-Id: <9302042250.AA17493@SSD.intel.com> To: otto@iliamna.cse.ogi.edu (Steve Otto) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Thu, 04 Feb 93 14:50:38 -0800 From: prp@SSD.intel.com > >The data is necessarily serialised to some degree over the > >communications medium, so where's the problem ? I believe it is > >actually HARDER to insist that the buffer sepcifications match at each > >end than not to do this. (Certainly I have to pass additional > >information with the message if I'm to check this.) > > I did not mean to imply that the standard should RULE OUT > mix+match or that it should check for it at run time, > merely that the standard should not make the > guarantee that one IS allowed to do this. There is an > important difference. I really don't believe we can foresee > everything, so we should be conservative on what the standard > guarantees. > > --Steve Otto > One of the important differences is that if the standard does not guarantee that you can mix, you can't write a portable program that mixes. Since we have seen one or two good examples of useful mixing, I think its important to guarantee the ability to mix at least for some combinations. My preference would be to guarantee mixing in all but specific cases, as few as possible. In fact, I would recommend against including features that cannot be mixed. I am optimistic that we can take care of the heterogeneous case in a sufficiently uniform way that mixing heterogeneous levels with non-heterogeneous becomes a non-issue. There was a good argument for not having a non-heterogeneous level, that would take care of it. The one case where mixing probably can't be justified is synchronous operation, where the send does not complete until the receive does. I've never liked this option, and worry that some support comes from misunderstanding about the difference between blocking and synchronous, as evidenced by Chuck's comment "You can't reuse the buffer being sent until you know that it has been received at the other side". This is of course not true. If the blocking send semantics are that 1) the send completes when the send buffer can be written safely and 2) message delivery is guaranteed, then you may happily overwrite the send buffer when the send completes with the assurance that the undamaged message will eventually appear at the receiver if its not already there. The only interesting difference between blocking send and synchronous send is in the amount of system buffering that may be required by applications. A correct application using blocking send might require some amount of system buffering, while a correct application that exclusively uses synchronous send requires no system buffering. In my opinion, we should focus on the problem of implied system buffering in the system environment area and eliminate synchronous send from the point-to-point portion of the standard. With reasonable care, we can make a clear and simple guarantee that appropriate different sends and receives can be mixed freely. This will allow people to write portable applications which use mixed structures. Paul From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 17:56:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23412; Thu, 4 Feb 93 17:56:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:06:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23575; Thu, 4 Feb 93 18:06:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23912; Thu, 4 Feb 93 18:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06726; Thu, 4 Feb 93 17:55:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 17:55:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06712; Thu, 4 Feb 93 17:55:37 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA06657 sendmail 4.1/UCSB-2.0-sun Thu, 4 Feb 93 14:55:27 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA22402; Thu, 4 Feb 93 13:56:59 PST Date: Thu, 4 Feb 93 13:56:59 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302042156.AA22402@garuda> To: mpi-comm@cs.utk.edu Subject: Asynchronous send operations Hi: Here is a suggestion regarding the various kinds of send operations. From a semantic point of view, the synchronous and the asynchronous versions are meaningful. The further division of the asynchronous send into two kinds based on whether the buffer has been emptied or not is an implementation issue. Implementors of a system where buffer space is not a problem may wish to return without waiting for the buffer to empty whereas implementors of a system with limited buffer space may wish to suspend the operation until the buffer space is emptied. As to which version of asynchronous send operation is being implemented should be transparent and should not affect the correctness of a program. Ambuj Singh Dept. of CS Univ. of California Santa Barbara, CA 93106 ambuj@cs.ucsb.edu From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 19:10:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA24655; Thu, 4 Feb 93 19:10:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11923; Thu, 4 Feb 93 19:09:15 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from almaden.ibm.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11915; Thu, 4 Feb 93 19:09:11 -0500 Message-Id: <9302050009.AA11915@CS.UTK.EDU> Received: from almaden.ibm.com by almaden.ibm.com (IBM VM SMTP V2R2) with BSMTP id 9078; Thu, 04 Feb 93 16:09:16 PST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 20:10:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25133; Thu, 4 Feb 93 20:10:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14638; Thu, 4 Feb 93 20:09:30 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 20:09:29 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14630; Thu, 4 Feb 93 20:09:28 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00403; Thu, 4 Feb 93 19:09:20 CST Date: Thu, 4 Feb 93 19:09:20 CST From: Tony Skjellum Message-Id: <9302050109.AA00403@Aurora.CS.MsState.Edu> To: ho@almaden.ibm.com Subject: re: my comments on overlapping Cc: mpi-comm@cs.utk.edu BTW, I do not mean blocking send / blocking receive only in my comments. I mean asynchronous reading of a specific message (aka intel irecv) going on at the same time as processing. The message appears in the buffer, and one can poll for that to be complete. In practice, message passing does happen in background on all these systems, but the algorithms are not explicitly using this feature, or encouraging this overlap, necessarily. They might process messages at high priority, and only compute when message input queues are empty. Lennart has always repeated this warning that the overlap is only useful from a performance sense, and therefore from the point of view of a programmer trying to ask for this, if the system can perform better with the overlap. I DO NOT ADVOCATE block send / block receive only. Best wishes, Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Feb 4 18:18:36 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Feb 1993 19:09:13 EST Date: Thu, 4 Feb 93 16:09:15 PST From: "Ching-Tien (Howard) Ho" To: mpi-comm@cs.utk.edu Subject: re: Content-Length: 831 > Please remember that overlapped send/receive will only make sense on > systems with excess memory bandwidth. Hence, it might make more sense > to implement a system where this is permitted by the implementor, but > not required. Otherwise, calculations might slow down instead of > achieving pipelined improvements. As Al points out, it is only the > heavy-duty guys who try this sort of stuff... > > - Tony A message passing system with only blocking send and blocking recv is hard to implement a shift or circulant shift efficiently and safely. Either you write an unsafe program bsend (right_nbr) brecv (left_nbr) which may deadlock depending on the size of the system buffer, or you do a two-phase or three-phase algorithm based on parity, which may require pointer jumping and choosing a leader. Regards, -- Howard ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:05:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14845; Fri, 5 Feb 93 18:05:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03769; Fri, 5 Feb 93 18:03:08 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:03:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03760; Fri, 5 Feb 93 18:03:04 -0500 Received: from fuji.ssd.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA06572; Fri, 5 Feb 93 15:02:57 PST Message-Id: <9302052302.AA06572@SSD.intel.com> To: jwf@lion.parasoft.com (Jon Flower) Cc: mpi-comm@cs.utk.edu, prp@SSD.intel.com Subject: Re: 5184; one small point In-Reply-To: Your message of "Thu, 04 Feb 93 14:01:00 PST." Date: Fri, 05 Feb 93 15:02:57 -0800 From: prp@SSD.intel.com Folks, here is an offline exchange Jon and I had that you might enjoy. To: prp@SSD.intel.com Subject: Re: 5184; one small point Paul, I saw your message about blocking and synchronous messages. Without wanting to speak for Chuck I think there is a pretty important difference between the two, particularly for applications such as Oracle. Using the blocking send it is true that the send buffer can be reused as soon as it is "clear" AS LONG AS IT IS GUARANTEED that it will be delivered. Unfortuntely this can't be guaranteed in the case where system buffers are required because the system may simply be out of memory at the receiver. In this case MPI and other systems will simply drop the offending message on the ground. For a database I assume this would be fatal. On the other hand the synchronous scheme has the advantage that you know it's worked when it returns, on both send and receive sides. Jon Flower This is an area where we must be clear about the semantics. We have agreed to some extent that message passing in MPI will be reliable. If I remember correctly, this passed on a straw vote early in the first point-to-point discussion. When I say "agreed to some extent" I mean that not everyone has the same understanding of what the standard is about on this issue. To me it means that if I have a send, and a receive that matches the message sent, that the message will absolutely be delivered. This is how NX works. If there isn't enough buffer space, the send must block so that the message waits in the send buffer until it can be delivered. Flow control is required in the underlying protocol. There seem to be a lot of people that expect no underlying protocol. When you say "send", the system sends - whether the message can be delivered or not. Apparently some systems work this way, and therefore it is expected behavior for people who have been users of these systems. To me, it is not at all compatible with the notion of reliable message passing. If you have reliable message passing as I understand it, clearly the important difference you mention goes away, leaving the lesser difference I wrote about. Without reliable message passing, I see no point whatsoever in blocking send. In fact, I see little point in having a typed send/receive standard, because such an interface is loaded up with friendly help for the user and it seems ridiculous to turn around and expect that same user to worry about reliable delivery. Does any of this make sense? Paul Hi Paul. I think what you're saying makes some sense. I have two more questions though.... a) Having the sender block until memory is available potentially invalidates some algorithms, particularly as the number of nodes grows. It's quite easy to come up with a sort of "ring" algorithm in which everyone ends up hung. I would agree that the trivial case (a single one-dimensional ring) in which this happens is a "user error" but there are more complex multi-dimensional cases in which it is quite hard for users to guarantee that it won't come up, even with quite reasonable programs. b) What are the implications of the flow control that you are using on performance? Is your routing hardware doing the claiming of memory blocks on the receiver or is there an initial software handshake to get started. The latter probably costs a fair amount that the "high priests" don't like??? I agree that the definition of "reliable devliery" in the MPI picture is not solid. I thought I remembered a piece that actually said that it was OK to drop a message if there wasn't enough memory on the receiver but I could be hallucinating. I wonder if it's worth trying to straighten this out? I have a nasty feeling that we'll get down to implementation details. If we can't get IBM and TMC to agree that Node 0: send to 1 read from 1 Node 1: send to 0 read from 0 is a reasonable thing to do I don't see much hope for anything else at this level....... Jon > a) Having the sender block until memory is available... This is inherently a hard problem that is all about system buffering. You can't solve this problem by careful or clever definition of the semantics of a message passing interface (whether and when sends block, and whether message passing is reliable.) If you push it down here, it pops up over there - depending on the semantics defined for the interface, the problem must be solved by the system or by the user. My preference (for a typed send/receive interface) is to push the problem to the system side, as much as possible. That way most users won't have to deal with it. But you can't completely solve the problem on the system side, because the user can always write a program that will confound any system buffering that has finite buffer space. With a reasonably good system implementation, it is actually rather difficult to come up with an algorithm that breaks it. Also, it is possible to define straightforward rules that will guarantee correct execution. That way, although the user must be careful being careful is easy. > b) What are the implications of the flow control... There is a performance penalty. People have claimed that NX overheads are too high, and it is true that they are noticeably higher than raw hardware. But there is quite a bit of overhead from message matching too, so for those who want ultimate performance there should be an alternative to the typed send/receive interface. Active messages is a good candidate. Such an interface is outside the scope of MPI. > I agree that the definition of "reliable devliery" in the MPI > picture is not solid. I thought I remembered a piece that actually > said that it was OK to drop a message if there wasn't enough > memory on the receiver but I could be hallucinating. I wonder It was in the first draft. > if it's worth trying to straighten this out? I have a nasty > feeling that we'll get down to implementation details. If we > can't get IBM and TMC to agree that > > Node 0: > send to 1 > read from 1 > > Node 1: > send to 0 > read from 0 > > is a reasonable thing to do I don't see much hope for anything > else at this level....... We have to develop and agree on a way to manage system buffering. To get Intel, IBM, and TMC together it must allow for the case where the above works and the case where it doesn't. > > Jon Paul Do you have any plan for how to reconcile the difference between the three camps on the simple program I sent you? My impression is that: NX (& Express, in fact) want that program to work as well as it possibly can. I believe that >95% of Express users code that way and are successful. I think you mentioned a similar fact in Dallas. TMC is prepared to say that the program is incorrect and won't work at all. IBM is not convinced either way. I don't see how to reconcile these camps in any good way. If we allow for the possibility that the code doesn't work AT ALL then the 95% of Express codes I mentioned above ALL have to be re-written in order to become portable. Furthermore, if the only way to guarantee portability, even within MPI, is to assume "half-duplex" communication then why not toss the other kind completely -- it would certainly reduce the complexity of the draft? Jon I think the answer (if there is one) is to provide a mechanism where the user declares a need for system buffering. This is a job for the system environment group to work out properly. With a declaration, you get: - Portability: The code is written for a specific buffering model but will port between environments because it tells the system what it needs. - Efficiency: If the user declares no buffering, the system can eliminate the associated overhead. (How to do this is a trick.) - Bloat: Every vendor must provide system buffering in case the user declares a need for it. Paul I agree....... do you think TMC will? Should we attempt to make a formal suggestion along these lines to the entire group. "Prepare to be flamed!!!!" Jon [I'm ready] Paul From owner-mpi-comm@CS.UTK.EDU Fri Feb 5 18:59:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15545; Fri, 5 Feb 93 18:59:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06448; Fri, 5 Feb 93 18:57:54 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 5 Feb 1993 18:57:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from hub.ucsb.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06440; Fri, 5 Feb 93 18:57:50 -0500 Received: from garuda (garuda.ucsb.edu) by hub.ucsb.edu; id AA18688 sendmail 4.1/UCSB-2.0-sun Fri, 5 Feb 93 15:55:42 PST for mpi-comm@cs.utk.edu Received: by garuda (4.1/UCSB-v2) id AA23498; Fri, 5 Feb 93 14:42:13 PST Date: Fri, 5 Feb 93 14:42:13 PST From: ambuj%cs@hub.ucsb.edu (Ambuj Singh) Message-Id: <9302052242.AA23498@garuda> To: mpi-comm@cs.utk.edu Subject: order preservation Hello: The issue of order preservation of messages when there are multiple threads at the sender was not clarified at the January meeting. What do people have to say about this? It seems that there is no need to require message orderings across different threads. In other words, the order that the messages are received should be consistent with the ``program order'' which is defined by the order in which statements get executed. Thus messages in different threads may not be related and there should not be a need to preserve their order. On a different note, there was some question regarding the need for a synchronous send operation. One reason that they may be useful is that they might be the only means to ensure true portability. As illustrated by the following program that we discussed at the January meeting, asynchronous send operations are not portable across CM-5 and Intel. Process 1 Process 2 --------- --------- send to 2 send to 1 recv recv In other words, the standard could inquire that programs employing synchronous send/receive operations be portable across machines. It would be difficult to meet the same requirement for the asynchronous operations. --Ambuj Singh Dept. of CS, UCSB From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 06:54:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18563; Mon, 8 Feb 93 06:54:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17241; Mon, 8 Feb 93 06:53:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 06:53:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17231; Mon, 8 Feb 93 06:52:56 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03743 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 06:52:53 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26581; Mon, 8 Feb 93 11:52:48 GMT Date: Mon, 8 Feb 93 11:52:48 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081152.AA26581@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00721; Mon, 8 Feb 93 11:50:58 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show Content-Length: 1971 While I am no fan of arbitrary system buffering of messages, I believe that if it is to succeed MPI must allow the easy importation of existing Express and NX style codes. This necessarily implies that system buffering is required, so that code like this Blocking send to myself Blocking receive from myself works. (This is an even simpler version of the test case given by Jon/Paul). I agree with the approach that the user should declare her buffering requirement up front. This seems to be the only way to ensure portability. (Even if all it achieves is ensuring that she gets an "MPI-INSFMEM insufficient buffer space available" message on some systems, this is still preferable to a crash or hang). This is exactly the set of issues Marc was addressing in his discussion on SAFE and UNSAFE programs, and the amounts of buffering such codes could assume (he also addressed issues such as "How many outstanding messages can a code assume it is allowed ?"). I've a feeling that a lot of users (most ?) don't actually know what their buffer requirement is, however I still think that requiring a user specification of required buffer space is the correct way forward. (It might also be nice to have a system enquiry function which could return the buffer space currently used and the high water mark. This would at least let users find out about the behaviour of their codes. Unfortunately this information is rather implementation specific, and would not be accessible through the currently proposed profiling interface. Is this an issue for the profiling committee, or the environmental enquiry one ??? [As chair of the Profiling committe I'd vote for Environmental Enquiry !]) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 07:10:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22055; Mon, 8 Feb 93 07:10:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17649; Mon, 8 Feb 93 07:09:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 07:09:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17641; Mon, 8 Feb 93 07:09:23 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03842 (5.65c/IDA-1.4.4 for ); Mon, 8 Feb 1993 07:09:20 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA26809; Mon, 8 Feb 93 12:09:13 GMT Date: Mon, 8 Feb 93 12:09:13 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9302081209.AA26809@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA00724; Mon, 8 Feb 93 12:07:22 GMT To: mpi-comm@cs.utk.edu In-Reply-To: prp@SSD.intel.com's message of Fri, 05 Feb 93 15:02:57 -0800 <9302052302.AA06572@SSD.intel.com> Subject: The Jon and Paul show [more] Content-Length: 2029 One more point I ommitted before... Paul says :- > to work out properly. With a declaration, you get: > > - Portability: The code is written for a specific buffering model but will > port between environments because it tells the system what it needs. > > - Efficiency: If the user declares no buffering, the system can eliminate > the associated overhead. (How to do this is a trick.) > > - Bloat: Every vendor must provide system buffering in case the user declares > a need for it. The last point is not actually true. It's entirely reasonable to have a standard which contains a buffering request and allow it to be refused. (There's not actually any point in having the request unless it can be refused !) I don't see a huge logical distinction between an implementation which permits system buffering of 1 byte and one which permits no system buffering. If the user needed 1Mb, then neither system is any use ! So it's merely an implementation quality issue. If someone has been able to sell their current system and keep their users happy without system buffering, then they should still be able to keep them happy with an MPI implementation in which the amount of system buffering is zero, and any request for more fails. Such an implementation would be standard conforming too. (Though such a vendor might expect some moaning from people trying to import code onto the machine, and maybe market pressure would push them into an implementation which did support system buffering !) (As an analogy, I believe that it is possible to have a standard conforming ADA implementation whose sole action when presented with any program is to raise the "Implementation limit exceeded" exception. Not useful, not saleable, but standard !) -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Feb 8 08:54:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26134; Mon, 8 Feb 93 08:54:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20868; Mon, 8 Feb 93 08:54:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Feb 1993 08:54:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20822; Mon, 8 Feb 93 08:52:39 -0500 Date: Mon, 8 Feb 93 13:52:17 GMT Message-Id: <16939.9302081352@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.)) To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Hi Here comes my 5p worth on the subject of buffered and unbuffered comms. For completeness: by unbuffered I mean that the sender blocks until the message has been (or certainly will be) copied into the space of the receiver, a la occam for example; by buffered comms I mean that the message is copied away into a system buffer somewhere and the sender returns, a la PVM for example. It seems to me that there are a few different issues in this subject which discussions may have touched on. a) Ease of programming. I don't think anyone can really disagree that many programmers will report that they find buffered comms easier to use. These users have been fortunate enough not to have come up against the boundedness of buffering provided with the system. With high probability they are programming (or adapting) applications by inserting message passing calls directly into the application source. b) Portability of programs using MPI between different machines. [Here I give away my bias.] It is very difficult to make statements about portability (and reliability/correctness) of programs which use buffered messages and rely on system storage capacity and structure thereof. This becomes particularly difficult when the application makes use of substantive libraries, which themselves are written using buffered comms and place requirements on system buffering. It can just become to difficult to work out how much buffering you need. We played with a system which allowed the programmer to configure the system buffering, and this feature was only used (properly) by high priests. In my work I will only be able to use MPI if I can write substantive libraries in the confidence that they will not be subject to failure due to running out of such buffer space. Therefore we use only unbuffered message passing with a mixture of blocking/nonblocking (like Intel NX isend/irecv/msgwait) calls. Therefore my bias :-) c) Portability of existing applications (using existing message passing interfaces) to MPI. This is a different subject to (c). Following the discussion it seems that an argument in favour of adopting buffered comms is the number of existing programs (eg, lotsa programs written using Express) which use unbuffered comms. It has been my experience that migration of applications between different, broadly similar, message passing interfaces is not so difficult, except for this point. In a nutshell, the surgery you have to perform to move programs/libraries between an interface that forces buffered comms and one which forces unbuffered comms and blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. Given the above, I come to the conclusions: i) MPI must contain unbuffered communications with blocking/nonblocking (irecv/isend/msgwait kinda thing) calls, for reliability and portability. ii) If a goal of MPI is that existing applications using message passing interfaces (eg Express, PVM) should easily port to MPI, then MPI must also contain buffered comms. This seems to be a matter for the full committee, hence I have crossposted. Incidentally, to pitch in 2ps on some other lines of discussion: * I can see no particular benefit in allowing a buffered snd/rcv match an unbuffered rcv/snd. I can see considerable inconvenience in forbidding a blocking send/recv to match with a nonblocking recv/send. * Yes, we do try to overlap communications with calculation. (Sometimes it doesnt buy you, sometimes it does. Sometimes we have specifically needed to avoid such overlap in order to maximise performance, but that was a weird one :-) Just as important, we frequently "overlap" communication with communication (ie, use nonblocking calls) to avoid deadlock. * Please, oh please, let MPI decide that communications should be adressed using a rank relative to a group/context (0 ... GroupSize - 1) or (1 ... GroupSize). We do this all the time, and its very very convenient. In fact, when we can't do this we end up having to create arrays of the task identifiers - we end up doing it ourselves anyway. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Tue Feb 9 02:23:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19163; Tue, 9 Feb 93 02:23:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06697; Tue, 9 Feb 93 02:22:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Feb 1993 02:22:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06689; Tue, 9 Feb 93 02:22:45 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA20792; Tue, 9 Feb 93 01:22:44 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA04771; Tue, 9 Feb 93 01:22:41 CST Message-Id: <9302090722.AA04771@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: The Jon and Paul show (was: 5184; one small point) In-Reply-To: Your message of "Fri, 05 Feb 93 15:02:57 PST." <9302052302.AA06572@SSD.intel.com> Date: Tue, 09 Feb 93 01:22:40 CST From: Rusty Lusk | | Folks, here is an offline exchange Jon and I had that you might enjoy. .... | Should we attempt to make a formal suggestion along these lines | to the entire group. "Prepare to be flamed!!!!" (not a flame) I did enjoy this exchange, because it is a tutorial on the buffering problem. Paul says it perfectly: You can't solve the system buffering problem by careful or clever definition of the semantics of a message passing interface. I agree that most users would rather not worry about it (and don't) and we should allow systems that try to handle the problem for the user to do so (the example programs in the discussion should run). I agree with Jim Cownie on this point. For portability, an enquiry function can be provided to alert the user who checks to find out how much (or how little) buffering is provided. I think the environmental enquiry subcommittee should propose something along these lines. As Paul says, handling this robustly has a cost, and some users certainly want to avoid it. After all, the user can know some things about his program that the system cannot, and should be able to take over responsibility for the buffering problem if he wants to. This is just what is done in the case of the "ready-receiver" communication. I admit that this is not a particularly euphonious name, but some way is needed for the user to relieve the system of buffer management if he wants to do so in the interest of greater efficiency. Rusty From owner-mpi-comm@CS.UTK.EDU Wed Feb 10 01:16:47 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13938; Wed, 10 Feb 93 01:16:47 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11091; Wed, 10 Feb 93 01:15:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Feb 1993 01:15:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11083; Wed, 10 Feb 93 01:15:33 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA02367; Wed, 10 Feb 93 00:15:31 CST Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA06777; Wed, 10 Feb 93 00:15:28 CST Message-Id: <9302100615.AA06777@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Multiple levels of collective operations Date: Wed, 10 Feb 93 00:15:27 CST From: Rusty Lusk A few days ago, we posted a suggestion for a multi-level MPI specification as a way to manage the complexity inherent in providing a highly functional interface, and to make MPI accessible to users who did not need or want all of its capabilities. The examples there were only for point-to-point communication. We thought it worth exploring whether the same ideas might be useful in the context of the group operations. Here are a few thoughts. The idea here is not to propose specific syntax, but to suggest how simultaneous adoption of multiple levels of collective operations might help provide both a simple interface and a full-featured one. Level 4: (very, very simple) Here there would be only one group, the default group of all processes, so there is no group id in the parameter lists. MPI_barrier MPI_combine(...,SUM,...) MAX ... (That is, only a fixed set of combining operations) MPI_broadcast(....) All operations would block until globally completed. Message types would be just like level 4 of the point-to-point operations: either contiguous strings of bytes, or arrays of specific data types specified like (...,datatype,numitems) The main point is that using just this level of group operations one could port many existing programs that use global operations, without introducing any explicit management of groups. Level 3: (still pretty simple, but more functionality) All the utilities proposed by Al Geist at the last meeting for the management of groups (creation, inquiry, etc.) Operations at this level have group id as an explicit argument (Again as Al suggested). User-defined combining functions. Gather operations. Main restriction: for simplicity, still simple contiguous buffers made up of either untyped byte strings or arrays of fixed types. All operations still block. In fact, we propose that there not be non-blocking collective operations, even at lower levels. The necessity of cooperating in the global operation beyond just making one's own explicit contribution puts non-blocking global operations more in the "threads" domain. Level 2: (More options for buffer structure, as in the point-to-point routines at this level) Here buffers specified for broadcasting, combining, etc., could be completely general, including mixed-type, non-contiguous buffers described by vectors of (address,datatype,numitems) triples. Level 1: (Full functionality, plus separate "setup" parts of the operations, like in Marc's point-to-point proposal.) Here operations would have a separate "init" call. Being separate has the same advantages as for the point-to-point operations, in that resources could be reused in loops. The "init" call would return a handle that could be further modified (setting buffer type, address, etc.) before use. The payoff seems potentially greater here because some of the "init" calls may turn out to be inherently non-scalable, and therefore reuse of their results is important. Summary: Multiple levels might help deal with conflicting demands for both simplicity and functionality in the case of collective operations as well as for point-to-point operations. What was generally discussed at the last meeting is very much like Level 3 above. An even simpler interface (so that groups need not be explicit at all) is provided by Level 4, while the somewhat grubby aspects of multiple and complicated buffer formats is pushed down into Level 2. Finally, Level 1 provides the opportunity to absorb some of the difficult-to-scale parts of global operations into separate, reusable calls. Comments? Rusty Lusk and Bill Gropp From owner-mpi-comm@CS.UTK.EDU Fri Feb 12 04:14:26 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05929; Fri, 12 Feb 93 04:14:26 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23402; Fri, 12 Feb 93 04:12:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Feb 1993 04:12:57 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enseeiht.enseeiht.fr by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23393; Fri, 12 Feb 93 04:12:50 -0500 Received: from marylin (marylin.enseeiht.fr) by enseeiht.enseeiht.fr, Fri, 12 Feb 1993 10:12:45 +0100 Message-Id: <199302120912.AA08776@enseeiht.enseeiht.fr> Date: Fri, 12 Feb 93 10:10:29 +0100 From: Michel DAYDE To: mpi-comm@cs.utk.edu Subject: Re Multiple levels of cllective operations I like the idea of multi-level MPI specifications since it is a nice way of dealing with functionality and simplicity. Also the fact that Level 4 of collective operations avoids to introduce any explicit management of groups will simplify the life. Michel Dayde, ENSEEIHT-IRIT and CERFACS. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 07:56:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02001; Mon, 15 Feb 93 07:56:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14309; Mon, 15 Feb 93 07:54:13 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14301; Mon, 15 Feb 93 07:54:07 -0500 Date: Mon, 15 Feb 93 12:54:01 GMT Message-Id: <21675.9302151254@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Mixed F77/C, MIMD(not SPMD) To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. The January minutes state that It was agreed that explicit remarks that MPI is intended to be usable with multithreaded proesses and with MIMD (not just SPMD) programs should be added somewhere. We at EPCC most strongly support the proposal that MPI should provide for MIMD programs and not restrict itself to the world of SPMD programming. The SPMD model has been useful, and can be expected to remain useful, for tweaking serial (usually Fortran77) programs in a strictly data parallel fashion to derive a parallel program. However, this is not a string enough argument for restricting MPI to SPMD. There are a lot of applications out there which are not really suited to such simple tweaking. Further, we believe that MPI needs to provide support for library writers, where libraries can consist of a procedure library linked into user processes along with one or more library service processes. The concern felt by users and gurus alike is that our perception of much of the discussion in subcommittees is that it seems to be proceeding with scant consideration of issues in MIMD programming as opposed to SPMD programming. I suggest that this should be prioritised within the MPI. I am happy to write a proposal or discussion document on these issues, but I have no time to write either before the next meeting as my travel schedule starts at 6am tomorrow. So, I'm afraid this point was simply raising a flag that there is concern, without a proposal on the matter, for which I apologise. Looking forward to meeting and discussing at the meeting later this week! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04616; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20711; Mon, 15 Feb 93 10:19:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20702; Mon, 15 Feb 93 10:19:36 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15992; Mon, 15 Feb 93 09:19:33 CST From: gropp@antares.mcs.anl.gov (William Gropp) Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA27181; Mon, 15 Feb 93 09:19:31 CST Date: Mon, 15 Feb 93 09:19:31 CST Message-Id: <9302151519.AA27181@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 15 Feb 93 12:54:01 GMT <21675.9302151254@subnode.epcc.ed.ac.uk> Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Again, a few clarifications: b) The working introduction of November 24 states that MPI includes (temporarily?) \item A simple way to create processes for the SPMD model This kinda carries the implication that MPI is for programming in the SPMD model. My interpretation (and in fact, the reason that Rusty and I included such a method in our implementation of the November draft) is that there needs to be some way to write, entirely in MPI, a running program, and there is a chance that we could agree on a method for running SPMD programs. It was definately NOT meant to exclude other models; just a position that MPI needs to include some way to write a complete (at the source code level; we do not intend to specify the OS interface to "launching" the program) parallel program for a common and simple model. I'd be quite happy adding text that indicates that MPI is intended for MIMD models as well. I would be very interested in seeing a proposal for an interface for MIMD programs. Bill From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 10:21:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04617; Mon, 15 Feb 93 10:21:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20721; Mon, 15 Feb 93 10:19:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 10:19:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20713; Mon, 15 Feb 93 10:19:40 -0500 Received: from jadoube.mcs.anl.gov by antares.mcs.anl.gov (4.1/SMI-GAR) id AA15995; Mon, 15 Feb 93 09:19:37 CST From: levine@antares.mcs.anl.gov (David Levine) Received: by jadoube.mcs.anl.gov (4.1/GeneV4) id AA21322; Mon, 15 Feb 93 09:19:36 CST Date: Mon, 15 Feb 93 09:19:36 CST Message-Id: <9302151519.AA21322@jadoube.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Subject: Mixed F77/C, MIMD(not SPMD) X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 07:54:12 EST Date: Mon, 15 Feb 93 12:54:01 GMT From: L J Clarke Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I'd like to raise a couple of points which I hope we can discuss at the meeting later this week, after lenghty discussions with a variety of local users. In general the objectives of the MPI Forum are very well received. Two specific concerns, of a very general nature, became apparent. a) The introduction says that MPI is for programmers who wish to write portable programs in C and/or Fortran77. The choice of languages is fine, although some may wish to use C++ and Fortran90. The concern is whether MPI intends to allow, and if so in what manner, mixed language programming, i.e. programs written in a mixture of C and Fortran77. The observation is that C is most useful for library programming, whereas a large number of applications that we deal with are written in Fortran77. (Users did not expect MPI say anything about cross-calling and such murky business, that is outwith message passing.) I propose that the make a statement of intent, possibly in the introduction, that mixed languages will be allowed/supported (I would hope that we could not make the decision to not allow/support mixture of C and F77 :-). This seems an easy thing to do. Let me second this, if for no other reason than the lack of dynamic memory allocation in Fortran 77, and the relatively small physical memory available on the nodes of distributed-memory machines. In porting Fortran 77 codes to distributed-memory machines I consider the use of C for dynamic memory allocation to be "mandatory." --dave levine From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:20:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07231; Mon, 15 Feb 93 12:20:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25990; Mon, 15 Feb 93 12:18:19 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 12:18:17 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25979; Mon, 15 Feb 93 12:18:15 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA03629; Mon, 15 Feb 93 09:18:12 PST Date: Mon, 15 Feb 93 09:18:12 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151718.AA03629@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility I agree with Lyndon Clarke that interlanguage compatibility needs to be an explicitly stated goal of the standard. The user community is assuming that this is true.... Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 13:17:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09917; Mon, 15 Feb 93 13:17:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29053; Mon, 15 Feb 93 13:16:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA29045; Mon, 15 Feb 93 13:16:09 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 13:16:06 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28124; Mon, 15 Feb 1993 13:16:04 -0500 Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151816.AA28124@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:00:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11570; Mon, 15 Feb 93 14:00:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01123; Mon, 15 Feb 93 13:59:09 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:59:07 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01110; Mon, 15 Feb 93 13:59:04 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA00836; Mon, 15 Feb 93 12:58:50 CST Date: Mon, 15 Feb 93 12:58:50 CST From: Tony Skjellum Message-Id: <9302151858.AA00836@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, berryman-harry@CS.YALE.EDU Subject: Re: C/Fortran Compatibility I agree with Scott. I think that a valid, related point, is that there need to be a C interface which is convenient for C programmers, not just a FORTRAN interface, which is usable by C programmers, but not natural. - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 12:41:33 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 13:16:11 EST Date: Mon, 15 Feb 1993 13:16:04 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) To: mpi-comm@cs.utk.edu Subject: C/Fortran Compatibility Content-Length: 937 Ok, folks, What exactly do you mean by "interlanguage compatibility"? In particular are we going to ask that F77 programs be able to call C routines? Although I agree that this is a good idea, I don't think that dictating compiler (and linker) standards is in the jurisdiction of this committee. On a slightly different issue, it is not clear to me that a C++ interface would be different from an ANSI C interface. Although it is possible to make an object-oriented interface definition, I think that such a definition would be unwise for several reasons. First, if an ANSI C interface is available, anyone wanting an object oriented C++ interface could write one in a portable way. Second, I don't think that enough object-oriented message-passing interfaces have been written to determine what a standard should look like. Of course, I'm willing to persuaded on either point. -scott berryman Chairanimal, Language Binding ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:08:46 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11680; Mon, 15 Feb 93 14:08:46 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01543; Mon, 15 Feb 93 14:07:50 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:07:49 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01534; Mon, 15 Feb 93 14:07:48 -0500 Received: from id.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA01566; Mon, 15 Feb 93 14:00:38 EST Date: Mon, 15 Feb 93 14:00:38 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302151900.AA01566@cs.wmich.edu> To: mpi-comm@CS.UTK.EDU Subject: C/Fortran Compatibility Asking for compatibility is a lot in some respects. going far beyond the MPI interface - such as asking a malloc from a C subroutine to "work" when called from a Fortran program. (library name collisions - need for initializations for both environments, cleaning up both environments etc.) I want to see them together too, but.... Perhaps we should not make it a full "requirement", but rather be careful to set up the MPI language bindings so that they could work together from C and Fortran. and indicate that this is intended on environments supporting mixed programs. I know of no standards work on mixed programs and recall the the discussion of it (ANSI?) as a general area for standardization a couple years back being postponed as too "hairy". cheers - john From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:24:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12649; Mon, 15 Feb 93 14:24:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02252; Mon, 15 Feb 93 14:23:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:23:10 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02243; Mon, 15 Feb 93 14:23:08 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA06780; Mon, 15 Feb 93 11:23:02 PST Date: Mon, 15 Feb 93 11:23:02 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Message-Id: <9302151923.AA06780@research.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: hat is Interlanguage Compatibility? To me, multilanguage support means that: (a) we provide both C and Fortran programmatic interfaces, and (b) operationally, there will be no perceptible difference whether the programmer uses the C interface, the Fortran, or both in his/her program. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:36:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13372; Mon, 15 Feb 93 14:36:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02815; Mon, 15 Feb 93 14:34:51 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:34:50 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02805; Mon, 15 Feb 93 14:34:49 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:34:46 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28237; Mon, 15 Feb 1993 14:34:44 -0500 Date: Mon, 15 Feb 1993 14:34:44 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151934.AA28237@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Any opinions? In my earlier message, I suggested that a C++ interface was not needed if an ANIS C interface was available. Are there any objections to this? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 14:44:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13826; Mon, 15 Feb 93 14:44:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03257; Mon, 15 Feb 93 14:43:38 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 14:43:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA03248; Mon, 15 Feb 93 14:43:36 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Mon, 15 Feb 1993 14:43:34 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA28255; Mon, 15 Feb 1993 14:43:33 -0500 Date: Mon, 15 Feb 1993 14:43:33 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199302151943.AA28255@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Perhaps another place? Cherri, Perhaps we should carry on the thread about C++, ANSI C, F77, and F90 to the mpi-lang mailing list. At any rate, I'm very much for having the interface consistent across languages. This does however, limit what we can do in C++ and F90. We cannot, for example, use methods in C++ or optional arguments in F90. Will such limitations cause much gnashing of teeth in the user community? Will we hate ourselves in a year or two for not taking advantage of the truly rightous F90 features? -scott berryman From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 15:11:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14568; Mon, 15 Feb 93 15:11:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04587; Mon, 15 Feb 93 15:09:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 15:09:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssdintel.ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04578; Mon, 15 Feb 93 15:09:49 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA25287; Mon, 15 Feb 93 11:39:31 PST Date: Mon, 15 Feb 93 11:39:31 PST Message-Id: <9302151939.AA25287@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA06755; Mon, 15 Feb 93 11:39:30 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: john@cs.wmich.edu Cc: mpi-comm@CS.UTK.EDU Subject: Re: C/Fortran Compatibility In-Reply-To: <9302151900.AA01566@cs.wmich.edu> References: <9302151900.AA01566@cs.wmich.edu> Reply-To: knighten@SSD.intel.com (Bob Knighten) POSIX Standardized Profiles are one standard arena where some effort is being made to guarantee some expected level of interoperability. My POSIX working group (P1003.14) is producing the P1003.18 profile for "classic POSIX" covering P1003.1 (system interfaces) and P1003.2 (shell and utilities). This does not include networks or any of the serious problems regarding interoperability. In this context we have been struggling just to find reasonable requirements to express such expectations as that a file written by a C program can be read by a FORTRAN program or an Ada program. We would not even dream of trying to stadardized the way you might link routines produced by different language compilers! -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:26:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18114; Mon, 15 Feb 93 17:26:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10980; Mon, 15 Feb 93 17:25:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:25:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10972; Mon, 15 Feb 93 17:25:09 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA11917; Mon, 15 Feb 93 14:25:03 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA10017; Mon, 15 Feb 93 14:24:59 PST Date: Mon, 15 Feb 93 14:24:59 PST Message-Id: <9302152224.AA10017@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Response to Bob Knighten's comments Bob said: ?We would not even dream of trying to >stadardized the way you might link routines produced by different language >compilers! I don't think that's necessarily the goal of multilanguage support. I should be able to write a program that combines routines written in Fortran and C, using MPI for the message-passing. That does not mean that the function calls look identical in the two source languages, nor that there be a single set of library routines capable of servicing both. But it does mean that I should be able to send a message from a Fortran routine, calling the Fortran library module, and receive it from a C routine that uses the corresponding C library module. Cherri Pancake From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 17:45:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19007; Mon, 15 Feb 93 17:45:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11672; Mon, 15 Feb 93 17:43:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 17:43:35 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from enet-gw.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11664; Mon, 15 Feb 93 17:43:32 -0500 Received: by enet-gw.pa.dec.com; id AA27000; Mon, 15 Feb 93 14:41:53 -0800 Message-Id: <9302152241.AA27000@enet-gw.pa.dec.com> Received: from rdvax.enet; by decwrl.enet; Mon, 15 Feb 93 14:43:30 PST Date: Mon, 15 Feb 93 14:43:30 PST From: MPS ENGINEERING 223-4656 To: mpi-comm@cs.utk.edu Cc: benson@rdvax.enet.dec.com Apparently-To: mpi-comm@cs.utk.edu Subject: Some notes on MPI 15-FEB-1993 Some general comments from the Massively Parallel Systems Group of Digital Equipment Corporation on MPI: The MPI effort must strive to Keep It Simple !!! The multiple levels idea is good as long as it is clearly stated what the model is in terms of mixing the use of levels within an application. Language independence must be worried about.... C and FORTRAN are a MUST today, C++, ADA, and others will be of interest in the future. MPI must be a reliable message passing system. i.e. The user must not be burdened with running a protocol on top of MPI to support retransmission, etc..... (An optional non-reliable mode should not be ruled out.) MPI should be thread safe. All functions should return error codes not just a fail/succeed status. All subroutines should have a status argument into which an error code can be returned. Groups and communication contexts discussions and specifications must include thread usage rules and guidelines. When in doubt give the user the choice about buffering and other less than obvious characteristics.... But, consistently warn about portability issues WHENEVER a choice is offered. FLAG ALL CHOICES THAT CAN CAUSE PORTABILITY PROBLEMS !!! Be particularly careful with the homogeneous vs heterogeneous environment issues. i.e. Packing verses raw data, XDR etc.... Models / guidelines should be included to address the issue of sending structures and endian sensitive data between heterogeneous environments. When the syntax effort gets serious: All names should be readable, descriptive, and consistent. No numeric constants should exist in the MPI standard. All constants should be defined symbolically only !!! (POSIX inherited a lot of stuff from SVID and should not be use as a strict model) Data hiding via opaque types should be used unless there is a truly justifiable reason. A message type registration facility should be considered as an alternative to communication contexts. From owner-mpi-comm@CS.UTK.EDU Mon Feb 15 18:56:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20138; Mon, 15 Feb 93 18:56:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14806; Mon, 15 Feb 93 18:54:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 15 Feb 1993 18:54:46 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA14798; Mon, 15 Feb 93 18:54:45 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA01171; Mon, 15 Feb 93 17:54:04 CST Date: Mon, 15 Feb 93 17:54:04 CST From: Tony Skjellum Message-Id: <9302152354.AA01171@Aurora.CS.MsState.Edu> To: benson@rdvax.enet.dec.com Subject: Re: Some notes on MPI Cc: mpi-comm@cs.utk.edu Please elaborate on these points so we can discuss them at the meeting, in the context subcommitee. >Data hiding via opaque types should be used unless there is a truly >justifiable reason. -- what does this mean???? >A message type registration facility should be considered as an alternative >to communication contexts. I see "current typing practice without any registration," "current typing practice plus context registration," and "typing registration with no degrees of freedom left for user," as the three alternatives. I think the latter is too restrictive, because user cannot encode semantically meaningful bits into types at all. In the first (current) option, this can be done, but there are big risks of conflicts in real programs. Hence, I continue to favor the middle option. - Tony From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 08:18:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16998; Tue, 16 Feb 93 08:18:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19183; Tue, 16 Feb 93 08:16:05 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 08:16:03 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from inet-gw-1.pa.dec.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19171; Tue, 16 Feb 93 08:15:51 -0500 Received: by inet-gw-1.pa.dec.com; id AA16702; Tue, 16 Feb 93 05:15:46 -0800 Received: by mpsg.mps.mlo.dec.com; id AA14150; Tue, 16 Feb 93 08:15:38 -0500 Date: Tue, 16 Feb 93 08:15:38 -0500 From: linden@mps.mlo.dec.com (David C.P. Linden) Message-Id: <9302161315.AA14150@mpsg.mps.mlo.dec.com> To: tony@Aurora.CS.MsState.Edu Cc: benson@rdvax.enet.dec.com, mpi-comm@cs.utk.edu In-Reply-To: Tony Skjellum's message of Mon, 15 Feb 93 18:57:10 -0500 <9302152357.AA13311@mpsg.mps.mlo.dec.com> Subject: Some notes on MPI [I work with Ed in Digital's Massively Parallel Systems Group.] >Data hiding via opaque types should be used unless there is a truly >justifiable reason. This means that, unless there is good reason, the user gets a cookie and must use cookie-manipulation routines to perform actions. A major example is process ids. We think PVM took the better approach by having PIDs be opaque integers. This allows processes to join and leave, which MPI seems to rule out, but which we think should be reconsidered. Having the notion of 0..n-1 or 1..n may have some level of convenience, but we think it is too restrictive. (Additionally, there is the paradigm problem. 0..n-1 is the C way of numbering, but 1..n is the FORTRAN.) >A message type registration facility should be considered as an alternative >to communication contexts. A big problem we see with communication contexts is that they aren't thread-safe. Their purported purpose is to ensure different pieces of code (e.g., libraries, user codes) don't conflict with each other in the space of types. That's a real problem worthy of solution, but the lack of thread-safety in communication contexts casts doubt on CCs as a solution. A type registration system (compare (very loosely) the X windows color space registration system) would allow users and libraries to allocate and deallocate a range of types. With appropriate allocate routines, you could specify alignment constraints in order to bit-encode semantics. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 10:29:05 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25120; Tue, 16 Feb 93 10:29:05 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25032; Tue, 16 Feb 93 10:27:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 10:27:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25024; Tue, 16 Feb 93 10:27:25 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA02715; Tue, 16 Feb 93 07:27:19 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA11099; Tue, 16 Feb 93 07:27:18 PST Date: Tue, 16 Feb 93 07:27:18 PST Message-Id: <9302161527.AA11099@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Continued comments on multilanguage support In response to Scott Berryman's comments: >At any rate, I'm very much for having the interface consistent >across languages. This does however, limit what we can do in C++ >and F90. We cannot, for example, use methods in C++ or optional >arguments in F90. Will such limitations cause much gnashing of teeth >in the user community? Will we hate ourselves in a year or two for >not taking advantage of the truly rightous F90 features? I'm not convinced that optional arguments are all that good for the general technical computing community, anyway, unless there is VERY GOOD interprocedural argument checking going on (which lets out most current systems). Consistency is much more important at this point in the evolution cycle. Cherri Pancake P.S. I tried to post this to mpi-lang, but got rejected. From owner-mpi-comm@CS.UTK.EDU Tue Feb 16 15:03:44 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08057; Tue, 16 Feb 93 15:03:44 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10592; Tue, 16 Feb 93 15:00:35 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Feb 1993 15:00:32 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10578; Tue, 16 Feb 93 15:00:29 -0500 Received: from s307.cs.wmich.edu by cs.wmich.edu (4.1/SMI-4.1) id AA07659; Tue, 16 Feb 93 14:53:15 EST Date: Tue, 16 Feb 93 14:53:15 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9302161953.AA07659@cs.wmich.edu> To: mpi-comm@cs.utk.edu Hi; A late pre-meeting post. I hope we can talk about the possibility of adding an io specification to MPI in Dallas. A couple of us (Leslie Hart, ... and myself) have been talking about possible MPI io specifications. Here is a brief outline of IO specifications for one option, a loosely synchronized version of multiple IO based on the ANSI-C librtary. Placing "mp_" before all names is fine (It is my preference to do that in all libraries). Leslie and others have a seek and read/write primative whuich has merit if processes were going to read from differeent "random" positions in a file. Assuming a message is sent for every call this could be high overhead, but it keeps life simple. This also allows IO request to be asynchronous (assuming the seeks are absolute), although we may wear out the disk head actuator ;-). Perhaps we should set basic goals for an IO system: 1) interaction with a terminal 2) writing and reading buffers to files, which should correspond closely to sending and recieving messages. 3) three modes of operation - single_io(), ordered_io() and unordered_io() 4) some specification of conditions for deadlock free use. Some questions: Should all IO be loosely synchronized? Should all IO "require" a message be sent with that request (implying no buffering / queueing)? If not "require" then, should the user have any control over the buffering / queueing? I'm not sure about how to address hetrogenious access to a file. I have seen a couple options. I would tend to not want to force a "binary" file compatable with all architectures. Text files should not pose as large a problem. It does appear binary conversions by the message system in pt-2-pt has supporters, if that works out then soemthing could here too. This is still a thought exercise, not a proposal. john ---------------------------------------------------------------- The Three MPI IO Modes The three levels of routines below and the three IO-modes are orthogonal. All IO routines must be executed by all members of a group in a loosely synchronized manner. All processes must invoke the same primative on the same stream. The following IO routines operate in three modes. Initializing IO and switching between modes using: single_io(group) ordered_io(group) unordered_io(group) Terminating IO from a group is done by: end_io(group) Each of these calls must be executed by all members of a group. In single_io() mode the IO routines should function from a process as described in the relevant standard. On output or in setting the file position the results from only one of the processes will be done. On input all processes receive the same values. In ordered_io() and unordered_io() the results of setting file positions are done for only one process. Input and Output from a group of size p results in p separate items being read or written. The ordered_io() mode requires the p IO operations to occur in process order. (If we have a group order this should be group order) The order of the p IO operations in unordered_io() mode is unspecified. **************************************************************** The POSIX IEEE Std 1003.1-1988 C functions From Chapter 8, 4.9 (Which is a pointer to the ANSI-C standard X3.159-1989 which is essentially the same as ISO 9899-1990) Those functions not mentioned in POSIX which were later included in the ANSI-C are still be be considered required by POSIX. Below: the functions marked with a "*" are non-ANSI-C the functions marked with a 1 provide a complete set, without access to buffer cantrol. Level-1 IO -------------------------------------------------------- 1 fopen(), FILE * fopen(const char *name, const char *mode) freopen(), FILE * fopen(const char *name, const char *mode, FILE *stream) 1 fclose(), int fclose(FILE *stream) 1 clearerr(), int clearerr(FILE *stream) 1 feof(), int feof(FILE *stream) 1 ferror(), int ferror(FILE *stream) setvbuf(), (not mentioned in POSIX, but it is in ANSI-C) int setvbuf(FILE *stream, char *buf, int smode, size_t size) fflush(), int fflush(FILE *stream) rewind(), int rewind(FILE *stream) fsetpos(), (Not in POSIX, but in ANSI-C) int fsetpos(FILE *stream, const f_pos *pos) 1 fseek(), int fseek(FILE *stream, long offset, int ptrnane) ftell(), long ftell(FILE *stream) fgetpos() (Not in POSIX, but in ANSI-C), int fgetpos(FILE *stream, fpos_t *pos) 1 fread(), int fread(char *ptr, int size, int nitems, FILE *stream); 1* sfread(), (a possible non-ANSI-C stride version) 1* ifread(), (a possible non-ANSI_C iovec version) fwrite(), int fwrite(char *ptr, int size, int nitems, FILE *stream); 1* sfwrite(), (a possible non-ANSI-C stride version) 1* ifwrite(), (a possible non-ANSI_C iovec version) Level-2 IO -------------------------------------------------------- setbuf(), void setbuf(FILE *stream, char *buff) perror(), int perror(const char *err) 1 printf(), fprintf(), sprintf(), 1 scanf(), fscanf(), sscanf(). Other ANSI-C IO -------------------------------------------------------- putc(), putchar(), puts(), fputc(), fputs(), getc(), getchar(), gets(), fgetc(), fgets(), vprintf(), vfprintf(), vsprintf(), (Not in POSIX, but in ANSI-C) remove(), rename+(), tmpfile(), tmpname(), ungetc(), ****************************************************************** Remarks: Deadlock ------------------------------------------------ Something must be said about deadlock, this needs to be addressed as part of the collective communications and pt 2 tp too. Some options are No deadlock will occur if: no messages are "active" while any IO mode is enabled (too strong but safe) no messages are "active" in the IO group. no messages are "active" in the IO group, during the execution of each IO opeation. all processes in the IO group have free receive/transmit buffers during the execution of each IO operation. Some type of inquiry function might help. All processes in the group may execute a probe to find out if it is safe to call the IO operations (humm....) Remarks: Sizes of objects ---------------------------------------- Restriction on the sizes of objects generated by the different processes could be made. Possible examples: In fread()/fwrite() size and nitems must be identical in all processes. In printf()/scanf() the same format string appears in all processes. These would make implementation faster and easier (and should be true in single_io() mode IO.) It does make slightly irregular data distribution harder. (Why can't P always divide N :-) ). Remarks: Buffering ----------------------------------------------- There are two types of buffering to consider, queueing multiple output requests before sending them and buffering the response to input requests (as the normal ANSI-C buffered IO rouitines do.) Buffering works in single_io() mode, but it may not make sense in ordered_io() or unordered_io() modes. Queueing input requests could make sense, for non-blocking requests. In our distributed graphics system we have some asynchronious commands, which are not loosely synchronized. For these commnads each process gives an independent stream to the server, which it must de-mux into a single valid stream. Any calls which change "state" in the graphics server are loosely synchronized. So the server dose not have to keep state for each process, just insure a stream can be interupted from one process at a valid place before switching to another processes stream. I don't think that type of IO is needed most. Since the most heavy use of IO would be the ordered_io() / unordered_io(), and that would seem to be large size chuncks of data, buffering multiple IO requests may not be needed. For terminal IO, the usual line buffering should be OK. What I'm really trying to figure out is, if setting up things with no buffering/queueing specified has a negitive enough impact on any applications to worry about. That boils down to how applications would use IO routines. Remarks: Higher level IO ----------------------------------------- Mapping r-dimensional data onto a logical s-dimensional process array is one fundimental high level IO operation. The ordered_io() and unordered_io() calls above allow a fixed dimension data file to be operated on by a fixed dimension array. I'm not sure how often dimensions above 3 are needed. That is, how often problems of higher dimension can't be well handled by projecting into 1-3 dimensions first (for the data/process distribution). If the size of the array was changed, a single_io() read of the file could be done, disgarding items not needed. For non-parallel disks (from a host or other single source) on an MIMD machine, I don't think this posses a serious problem. For high preformance parallel disks, or disks distributed on a network, that could be unacceptable. The only way to support that type of IO would be to define the process array and the data file and let the system sort it out - as well as choose the files internal distribution. (As in Hart's system propoasal). The EXPRESS model is successful and should be considered. Remarks: Multiple process IO ------------------------------------- What if anything to say about multiple processes operating on files? limit the total number of open files? (eg System Constant 12) limit any file to be open by only one group for writing, for defined behavior? allow any number of groups to open the same file for reading? (requires the server or group to keep state for each group). Remarks: The three buffering modes in ANSI-C. --------------------- The three buffering modes are: unbuffered block buffered line buffered by default: stdout is line buffered stderr is unbuffered all others are block buffered The setbuf(FILE *stream, char *buf) call can set buffering on a newly opened stream to unbuffered (buf == NULL) or block buffered, using a user allocated buffer. he buffer buf must be of a fixed system supplied size. Useful buffering calls, not mentioned in POSIX: setbuffer(FILE *stream, char *buf, int size), allows user specified buffer sizes to be used. setlinebuf(FILE* stream) puts a stream into line buffered mode. setvbuf(FILE *stream, char *buf, int type, int size) sets any of the three modes (using type) and allows a user sized buffer to be used. setvbuf is in ANSI-C and I suggest it be included. The tie in of the buffer and sending a message could be made, implying a message is sent when the buffer is filled. Giving the user some control. I'm not sure I would like to state something like this as a requirement. Remarks: non-C Language IO ----------------------------------------- The functions above are clearly C oriented. Compatioble calls could be available in other language bindings. No attempt to specify native Fortran IO in MPI is suggested. Remarks: The varargs forms ----------------------------------------- The varags form of the buffered output family (vprintf(), vfprintf() and vsprintf() ) is in ANSI-C (it was not mentioned in POSIX). They could be useful in avoiding extra memory copies. There are not corresponding varagr buffered input versions defined in ANSI-C. I would suggest that they would not be as useful as a stride and iovec version of the fread() and fwrite() calls. I think iovev/stride versions of fread() and fwrite() should might be the first extension to the ANSI-C standard to consider including in Level-1 IO. The format for these would be compatible with whatever the pt-2-pt calls become. Remarks: Blocking, Non-Blocking and Synchronized modes ---------- Some of these calls could be used in all 3 forms. The setvbuf() call already could give a lot of control over buffering. It might be wise to only consider "extra" modes for the fread() and fwrite(), which would correspond the modes finally in pt2pt. (though if extra modes are implemented here they should be easy for the implementer to add to calls such as fprintf() and fscanf() ) Remarks: The modes in the fopen call ---------------------------- Limiting the fopen modes to read only and write only would simplify some things. Remarks: The unbuffered IO calls. ------------------------------- Although the POSIX IEEE Std 1003.1-1988 C functions do include the usual UNIX unbuffered IO functions, I think we could take the same approach ANSI-C does and only specify the buffered IO functions. Because of the fread() and fwrite() calls no functionality is lost. If the unbuffered calls were included I would suggest the calls readv() and writev() be included as well. int readv(fd, iov, iovcnt) int fd; struct iovec *iov; /* pointer to an array of iovec structures */ int iovcnt; /* number of iovec structures */ where, struct iovec { caddr_t iov_base; int iov_len; }; From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25996; Thu, 25 Feb 93 12:19:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06338; Thu, 25 Feb 93 12:17:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06280; Thu, 25 Feb 93 12:16:46 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:32 GMT Message-Id: <2150.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:51 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26012; Thu, 25 Feb 93 12:19:51 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06370; Thu, 25 Feb 93 12:18:21 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06357; Thu, 25 Feb 93 12:18:10 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:58 GMT Message-Id: <2160.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: third part of document >-------------------------------Cut-Here-------------------------------< M3.MBW=4X7R!M"=FS[?T0"!P-9;F G&V9^JE.Y=8!$JUT,N0>,&HSGU:"W Q? M74UY1V+0@-I&:2=3'@V7AD_#)65_,*=O/LPF1:NN;XC-4J5D)^?8$M:NA3:K M"JU#[-7^+U+W_SMD+6*\V'R6?,_EX%(7H]==I.4%TIZ]]\L%WYZPI@PL;"Y# M\6[*!V29\*AQJXCLI18RE?UQ_\_4(;KYB%Q()ND9^C!Z,E7XW-2X+"<&H!H\ MAR/*J5^U0UQE3F-M5LCVZ?)IC.2I[.!4M4IV35S8F'>%K>)XH6Y8(E8X?WL[-+F'>J/_4/4R>)/\,.!6NS.9[:G]!9O.JD2DG MG&M*B9N:\W[V/SP 5 4+B).>B[H4,]%GT;BA2Q"#)PVG!F0T[.//ES MD$N_#>7*7^Z60WQ?E=\!]3H$N=;M*764C\8AF((8<4<# M:2W%D+3K[E.21LS=M1%G.*-VQMUU[IVW@ERS=)YG\PW%""J0A2XX]UCJYJX4AN*:Q-+E]]V*$8E87R73MPOSM_^ MB_&[>F)[,C:P3B.LF()*8I1;K#-P>?KVISV)/9+18UENXJ20J4ZW%<=_MJ+88R\8MKMQZ MB[EZR^=.7K$WW%QOQM!9 65U.%UGKRHYVIM$(-U6C(LX\D?*+R#UK"D-?1<# M3CNDW=YY<<$XEOKF0-Z2>Y..G.R,:?,6Z61,E%+1EO?$_&1T*7+:U19;PU9-\>H%_&#\R8\=&W!UVZLQ'7 MC&&^*<44'^ M,%>$,&@@J:L2F+H/36+"G^?&1%R[,;9CRG8BUKZF$*_/>^-5#1+5\-I]%AS[ M?8O%Z-=EI?BY(IK%_?:]7YO%043'\G)$(;9KIHE#1] O[/?Y" M.\.;U6./K+S%ZY(KR4^4V8<._D33RK_DW,)?^!7AV MCU?1;LU+HL9U(+J-V^4&5.@-/14&!\,^O1P]L8QK?Z/BVT?(&SJGSR*9Q:I3VJ M[\BJYM6+85Q.E0P##0UB&+N6V(8F\K+-]A=R;"2,'%^9XDS)J9!QD28@P@M4 M>56_: NO] '6") $>)3"%>FU_T_9Y8J79HNRJPYO=8ZQ #[4!FXXTNK9E0J# MI/=[_+$:(,OW?EMDS:EF9B>4ES=R,$[:]UF"/"1_Y:2 _\9ZE1Z8ENPKQ0XS MAVO2F$\'J;?6++L?@J>V!6* )>9A+[Q4BAS3O3>^6_D#7M_G87T3-#VQA:.B M#M=LVUM"]-Q9O9S:>TMGEV5\N&&08:.. -KIRS]OUO+2)\8.](0X+[>M^Q8N M"@?3C\)KI@E2@N:9MMVP.=:M8&F9D/10J#@:O@($ >*'.1QS+VP.L;R65B][ M13]T5N2:[F)!=??_PR\3**1_M;VHG>W/(IVMW;W>+],$0F;+=-OD6?CI ZOA MIB>?Q6-)(T3Y'(R3'B+W[9S.FZ-S%@T_ M 9S3\4.:\ALY)-R'.P>CA57!6+Y(F^XUP<<&XB\?>9=TJ^(&8,:9[:?9PU^" M/Z&30N9DW7V3SG<];0L[A?/3(-2[].%/]5>@-C0J6F'%"[[>G\L0;@#9#9J. MEP>A]&DV)6AR5#=Q!NXAAM>'P<0R,F NU3<<;;!FIC]])Q_TE+DZ]&&!.M'ZNR[F#\*@P S^KZU#:G MD:5W+KL%'DMX8?CF6>PAJS3,H,TPP1D ]\!BML@Z$J8#5]JX2EYM--SRX[4] M>^_+3]HX]7_:Q/RA)>H6"=/-M,M:GU* U,9K3+2:[X ,8]>38\KQU"Q@,=%5 MI0V&%S8[\ \ULJ,#K2:J4TV:U&?,33&5B1JL/6DZY'#2SDL4@:AR$+II;!T* MJ1O+H,MS7A[U27P&?$=W+EMOBIPQ01TU*]W2:U3_IJ74Q] ='C&RVBHP63U2 M\V"J56('Z3WA]7C,Y)Q"I>=XIYP'K\.VDS8C;=6&B>&M%+\)M;:3WRN.3JR] M" .:1%ZYW(8U_\PI[D^#>8"U643D=,;5)\CM:?VJ $W31%>XWM%UM?:P3K3> MI2O6NMDI[YU94-S?/=?6;+:!PL#F]'/ZS?#P-7UVF/V,W>HSX[IPMH?7,\:% MY7(,-]2"()LCEK-$!2(.K5F]?%>\,1(7$XWW%3S;>E4UTM3O>*I13['#C&#O/GR>WOMZHYMFQ]39^"/^M M54-QB & @5FU+;HCY@SK/7LV+,$ K\+Z^-IZPX.Z M][S5E.FO[I>:+,>WIA@;KY);VW0C,=-F\>:S9#S,[-AH#&W-[XJL7U@M7Q?/EE3 M--ESD#V6,>J:L B%CEFWKL&1XDC?9V:RGE!>!2/4F9ELW!SV+:H0!>@?[ER_ M&7R??=8:=>EZ_];C4S>O;XO&Z&)/( ,78?VQYH42 5K7L==0Z!5U<6V(F[=) M5P_)7^,09O;:VO2I"6#N*9%Q@]24<"$5 MU4O#@I< F=.9R/(KM884P_9!C; M7JN9IU"(,<$W>JME45];-AX8YA?UJEE_.<^3F>(QZJQ:,[U%?= MJ''!$.BM(>OD?MUYG6&'?1&=\.OLH8[TL)LV3E7B< '8;>-%]) 4;LP4'5O; M:PR'61/JLR4ZN<>T=C[>;*^AZ.[,=O;1I7+@+79/N"]>GPH^VG\7' .F8C_]UB$-/59.6 M?=FD!=C,L9TS#72+[N32<7FPP&,RXAYQQ?:?[@569>XWRF@JXNN7C]V!'0PJ M?P&0A]R=*$-'']W[?>3V_EBPP6/@L> T.C"#3?X22FEIS%^4WYK0>2R$1?G] M8*>_TV-7A'FC,BR/YF/GH\N_605^M%#TO=E!--C>;^38[]]D*,,SYUC_/4C_ M_9C5^M]@+K3TI4L!M,#VJQ-W;%C2-:V:E[DJ_N!N/R^-VF?D&T_-N_=%I0P! M:2O.R@)E*4=3]P=D_2KZ_CS*C^8[M7UWGAQ/[B6_>R%I8E2NL,]5,V?CBT,U=@LCW8&JQ"CBAW M8E7!ZF N=#Z-;]I@A.V.B!4GYXTC*E1(;ISI_(_F7UW$9^YV M=XF[X,S[;(*CH\@2O!M229O$\D?'-? M7BF\+&MW("X4K3VQG45^7/-I?#Z4W4V3"ENW"2'B K^21MPE-J\8^]RF"1"\ M:5:!4^SR:Q>3BAOX/1Q;K>MH[E0^ MH]&(Y^C:L0@V+#C^A6774@-SC-S ')#/.=?[Z^."U8S'@MP_(A"VX'G\?67O MHX&BLVQVM"-[VHG]W5@&MTNYTU]9;A+3P:$#M$Q;(),^H-*IZ"BR)/!(VBQ: MMU62<]^-&N37C&L2G?M",B_;-^@^'-O.?N!,C52*GW7(LQT=L %B M(N:^%S6O RD-W]@K/1V@ME?'J^TOXF_;M0W>U.+("1_9LFU$;L^OMNT9NFU; M):N2CJ^+7V!.-;@U@&+'*+>Y_ MN-?1X$WGY7+;".N$O6D.K1_;I,4;=[$4GYS37."-*W/)6%>)HS0%,0%4.XQ=W7*S M(^S#)6V$(I@[G[8W<,HU"?%S"N63F9HU3DN^MC%WE[O#TUMX''UJG'^$G[<0GH(=?/5JJ^;X\6X9;HGF9KQ=8CAYB M6I/.9FKE7><.NRHEY#3C5W&R5J)Y6XO.)P:7)=CM^ 1YF(33H:YNTDLY7;2^ M92F@L&7LW4HFZRPB?CEZY*Z.T6KUY/@T6QP30%O(,7O/2=LOIR^GD/:TJU)+ M:,O(A>1-'YP8Q/?%0VWTCP'(/N1,JO;X:/L$SGJW >5WB.(]HC'4U?@%KN)I MO!]K+P VIS:X8V@O*F=[+FY3X4AQ!S%=_;5>X1FHRK.9U[6T[%#7P M=I>WJV)I8<:Y>GV"G"QZI&>V$F*ZJ_!&2RF2)/(9<3L$#L_";5!SJBUX=>J5 M.7RM(A6QPU?24 ++9Z=_0R9B/)0WTW MMH@-O%,1=7C2D^OMT$ M!MV_<,SI9@!VLEW'-(E:MO?8_F[UMFC[CUJ.'G*'MC7;A[19]Q\R#92!E0*J M !BP?MP_*2;;=4QZ4P'D<57'5.ZS)*Y[V7WE/DLVNP>YP^U":20;][OF+O_^ M_.9>'+8G0'M3_!.8 QYC^C2_W49*;NI8*TS(WG4?CS6Y>VZ$HY\;EDWM'G$; M$F]KS]]0=F P'LW(L;X0E^O7[G MKF^SI=:TO]FMQ7A0X M*#^K?MT.9K)ARO?0I%G:+XO/"(<<-7?; GQ>+B277(/5@0F\;J8PHNK0=GI7 M% F'6<>D3YBV-' )S6P,0?/#66_,P5VW]>CU+IPRP;/9!%3D\XJ6+FJ;ME%* M1\VTB%48L0=2_PQ':W.&00W.D^4D-)Q62ZT!_()'2X?*06(6]A891RU08P@* M?9&C/>H*WFURM?9)QBJ7'!]K3H#[&A.@3UJGHWQB^0+5W6R\G/E20GD&7#P? M6\&>"N'Y 1) Q _1.VID/AO%!?="D\*?RA&U6GPI6'2)"J*P!7$/[ [I\: MPH>7B/!YF^V/GKC+0Y>&L.-U%L?;G?_81@P,/Q<2XH;AP;E_AF,)".%+7B1P.)\M^ MY3 MPI&FIW"!^*!:=9DTI/96BA'B$;UON 6O(8[W)"F\G%.\@CR/L]]S(UO.W2V' MAWNZ]G_4RLG55FO[%+GB0NYV<;97?JW6[+JR-7&N74,_-/O0VBP,=&N# M0G_:8LX?KMO#<9O55@CZ4Y38P6\F-O/QM!B]$;'@:;1J@6TI+K&8L'TL=GYC M<7MRESAFL:U2_LI$@JX%U_AP^43W'B!.0)G@$_9R]MS89^3WUVF %7*;^2Z_2\3,+^D8=3S_WNV]\O*X/VN46C]OZ,#_.TM^ M!<0 [6_9W73@* FX"8 GN*G1/3]%KG7@_JC;SC1_8@'9(+;[-P7<*>@K!;%A MA%"_NW&2*@:<=NPIU/RJ5-=KSU],MDKUE"T]=L$.8>W1\FAM=XZ-JEH:"&V# M?WVEK5SR.-"IT"V/AHFNH\GCL0A)WO-W#*!.0SINQL4.G?%"X,PY\2N+2"*$ M-.NNT4V5I&8&WCG>[G2W20?D>T<:K^47-9[P4XW_L7FWQM_7^.FXX#VBW-'1 MQH5IL%^$ #N3M;W'FPY\H^O;1;_[0@\)IPVO94:'?VFB"0[:[_\1MS:-)H!# M 7#'O-_Y5''<&\WY#61/P /<[@7GN*I OTH?'T?#J&6_XDY+)X QF>23#^%G_NPU9T:'QT_ MR#'9@NS4L7]PY'VZNY"'N6_C$?+87'3.[X\666' !R MV#2_F#G@KW*\=6PW=1#H5Y_CDKDUN:,;NDDD5Y+7-YW:M>?W*(1-0"3?='=& MQ:#DG&XIN4GT45XEIR(NR+'D?L!DSI8\=-PEKXZOQC%^R/'-[R!;ZDPFMY!S M_S#D:/+X-Y]G3<[M#KZ>H]_DMU^+*HQ<=YR5M),/TR"Y>7+@L8+WX\:G7P4'NG&_6;M!N=$FV29*G#LW0,-CIUM _9UO/&W_EF7C9?)3^9E<0ZXJ[Y&3 MN;4X*G(V(JP\V$V ''8[N+$,[\6AZ'8.R9>P/(XOQV]U+ @=^9B 1RXLQZ$V MRPOES/*]J^%P(0B^S*Z!!B*0A7)N-]AY6" :)V]3RR?FH>XM]\28O##,HN8()=[N'M_%/+9N+H(%XN30:3EXOUT?K?@-S5H!\.:V49D8CK^16P'WES]%AXE8.9FXZ?G82 M0A_F3')CN4V <[.X79B'-B3:9,<60NO;77?G,]T9O.,K37QRV+#IT?8,OF=^8$=N77U'WJKM6, M:TWG$O;)_-SN5,N70Y]&/ZUS'/C'?+)N1_Q2ZLSAV3' MRGOFY=^?.8..\3XX'0)E'SH/EPG.R;7PS0 Z5.ZIH MSD_$+)N?*L23W1VS>:V^N]]T3TMY]TJ/WLWNV\/F,Q;"/.N]MR-A-_D$M@.2 M#U/ W6DQ*M>SF_M>KDLW=.^*^&P.L>3S+7>>Y#K#S26N9(0(8+T6YET/GRV3 M5P&?"MVM6%D84HV:NXW39 >A>[R)LYZVFZN_>\S5J^"R0\\-[ P!-S??742@ M#:.TI9K^ IK.ZZ'-5/1YPPU'7RM9:$_ MBM%^35V\7IC [N<_S WOIW=I2)\ ,[^42^>4':72XBKHC6F0[$_\3XA 59Q: M!JS9?6]5)@CP%XO0QJ!;&R. ML)%H=(WGT82ER('E@,USF;8Z@B[:E@7WDBG M09NZCSF,4+!/5QG\9O;=?9V/YKV/@GG/DA88O]MB2:W8S6\L-ONU^ :M/$4O MQBGFGD_WGCMYD!>,:!@$=2C56KQVMEB4C#Y%QVHZ$TF0O 5J*?DN+#<@2!Z> M5I_ H[];GLB3.*Q\IKQIP:])*C_;W0WT*%H&[Y[^#".)-&XI3F P47-S;&#: M! T"N9Y4V[Y<*PGC:+U)"MIX0-[3UMC+G>9B\X*55UJ-M2H+W$T9>';:RM>17QO!QO%M\ ;AQK5@LX-I["! M0;O5;DIJNAY:Q1RJ53BJ+G,XX_3&M DUJL>M/ER[*2ETUFPQ7=293LW1M4/C M@L&@,Y)%BAJ50>KVKJ<_GBNX8X*Q76M.K+K/G@!2!&B9*<_[&WRW/-R8#JA; M7,O5ZC1N>+IZ>2=6)H":J!$")665J;X<1AQ3"OENS5,O9]S'7]KVQPWUG M1H/62\K\:>62RWLWSHOOM3/1CIN:C6F$*MB)5GX/QIG?HFC#>!8;T[8L-J0/ M;AGCG#@QMB>N8>C=LXR[T_&OM@9R.,?0JNRL?1K&U=7>2\LJ[[6PF8 MYY[1:@Y/-G4\<&X['L&68"&EMG+$*H?MXPI"UM$Y,RY6. MLFNPGG(?[/J-E(W*566#QTO@*O!T$Q,V/.X"WW9#,OM:U>/)7L+/%5%8B:[K MUOV(^075^AI15[=7UWS'E CK=9I^WXN;QAW JUQN:DJEE=(4.<6\0#XM7RZI M)TWC[_/HHB4,44GCY9(WR+WDDW4(N:A\ I[:KG(ONS?KZYHG\1<1>YX;%ZT/ MS/VXI[R].IY;U'A2G9>WUL."PO$I *V\3EX^<K%Y86<1^Q%1_1;U^YJWUV5"RSIXJ[39U U9OZ][ MRK_D$G#/.FQ\/@7@)OT"V'ZY[R)XTW[#O MU?6K)W)"J$*MW?ET'$@K4W,;GL MY7D;O>ZJB=1\;_DU&#)6'R97C_?71[[*[5@-8\ 6:Z(3L,//0NFX]39Y@-ZVC M' ?=^6L]]VH=V/U@U[)SV2?L0]'8.JYI0\S_Y1R'1V%,GFX6^.>0F'6^XOF)UC M"T3FF$-F[;A\O[W^=KM=UOWKEES4,:K]R XH9ZBB6L?DEG8R-T[ I'KF9JW/ MMC>!A'/<]M7++.DG:(#K@+Z>O'+B\6==8+Y;'R.*$H'@DS\SZ M6A-$47.0:!MUTVW'-I![O]NH&7,0.9%[Z7TVCZ^[SINHF[38N=OQNVQ\[_=HU+SBGMDW8B.W<]V^XW%SV^RM'

);9_=V/Y^HCEG>9?MI>VRK\0=TT=QC[:# MRM?M*C7 @D=1$^=9Y["CCMWM2O-"=M$/!CBWD>*X#TWDSNCAN3S\X^YM5W"_ MR+?L5@!(Z9M'$MP 7[GKQL?1J78SNZ4=S9YV)]ORW%7L(LG#Z^=;*.%K57#X MM6_M>9N_(=!=L5[9_AL&W)7)>_X[LYX;)[ CV#/NH?67^[L=CLG YQSVK_M!? F@/+ZH7*GF.^^*,F+V!V<]Q$ MK0?;<'7B)I?T,(YI4QPS=:;?7FSRL2\;*H??"V:#L+<$.5G>'8%3]YT$1^I1 M"ZGI4R'G7.X/['Y,W$QSB)GH._&4U]'.U'=)7TS3>4^F>D]WWYROKP=) [M; M/@_%?CHFZV-TZ(Q8JT^^HR^&1LHRY[)MT\=^OUTWO%FLAK6=M,5S">@Z%1/. MW]U]^+U!,6=2^M>3;4TS7>&[R^:8^A(PA9L/+WH3_U3JU3FQK-OSH2[76[7[ MA'UT]%W2VW:.9"E/I(UVU^W3K9NT..22>[IGHZZ)5*KF1'6]S3_TYIC#,9\Z MY'#IHH8HA!LP]>CE_*5K;D6\Y3N6'3$=0\>@V[\%U-&-RG1/[JB8GGXH=O02 M.*G.QMRH;4XY!9SW-GHW6M^6JLO\@M>W1\[;U:9W;3^]H]XS@+Q8KA3G)*C7 MJ^':2T CI#B3!9^8E<,'TV8DFIDZ?'<];)B!MGKEX:M\XDS_^\!0!5]+[*2; MP3N(. $=*CF-;>/4YJVFGP&OTN)M'7L[$1JET0&NODTXZ;>#8+3OJDZARZHC M/[?J(DJONNY[EMN)EVA':=:VQ_>"Z!T=6(DH8A[6XK]I??1A\1]=;QM(G[[/ MU8MOY6?I]_DY^Q[OV\3MXIYUO;C$P!SG,5!9(, M;R#;*DF>I(MX=.YO_[9IXX728T34L7+YH;O4D%R (4(,V" ;F%91ZW7V>.)ZE^)7WC8<1%3Z44P_O\3:[7 M#:WK>>LDIC5^)3E>N"3ZV1B>#ASM.P^Y>BT?!UT7NHB9;\\,[JEXIAGQXE@%@G_XICQPC@9-Y$31\$R!0MH M_B)Y1<;EW=-R7.B?^\E.&_VG(C6J)[<7AA<#!@&&&IJ"_S1\;!>:C[E/O&6. MO2EI\/>GY)RP#7Q40M/=:3N&/^(=\:86\=)D'3S<9Y95 =X+OKM:=;RRVN\H<:V8*GR0@O$"M]UG6F<_#J9@C=7ZRRL M/K=[2NK"ZHHW&NO.W5EWD-F&'K@>+59S,9JP95O2E1N?=^5,88!9S0AT_-*9 M6]N>3NHP:%QX=RLW5(50U,C,.V(-AZ8,H]6@V:V]-+ADE97:ULW23=8]?,- M&2,>34L9Y^J44H@"+2U3$':[4+[=:?@+.]JCI$9_.[K)-40!N^"( !!-V _1;&H9 "H"7&Z6I M19FS(U[>PK249CLR'3DGM1UVQ\S2Z>EY+SV6LWESV!Y-#0*XJ@R/7U@$_WI. M&M6<:UT:?><'..@/8.0RW'2ZU^ MUN9J.&\P^J!X>)GIJ['%T1)"9,9V,M@63;T4>%+P$\.R=P C9GW>7G>?5]C= MDY.&'&HRH,!;!3@%H/F]WD2,/$/ WO-PJ"JD 2 Z#[N"0Z)TL#/\Z.VI\VV9F46 MU\J<,W3U'.7OQBP[#+YM-A2X].ICZ_YT">I1*\$C4-_'YE7-W:TI=.9A'O0&OK]PK&NS_I9 M"6EIMK]W/9S.*DNEOX@G2./5W=70.!]U>6B!!J5^ R>8%-S'9!(Y:/SEH?>Z M4GEX:O%K4K<>9JQ [-?KB]G$Z]V?,?#V-FVQYX=^ZQ&OUM?R>0Z0$DU'E\4W ML6GQA9HA7Z&&H):+'QS_?4/1TO( 7:E>T)ZB"D3G9C6RVNSP^Y'YZK/D5 M 9@ "_"4>QN@D\UV/T87!$<"9=)"X$A@J#Z%I8KVLB?RV#LO!_I80:\^MNK& ME96-<;6!<\94;U=<[;&UYF;7U>4B)?A6T*Q=AL8VO+^#=M5D>*C->%H)F^@:,^5C=KU MR[/2@;F67G2.PV9H=0/Z(=U[;F?4N0[N(ROFKP,QTU;?O[947F'6D;Y<5JE_<2:MB738%';M$5/QOUHO[P= MA9US -DA MJ6-\NU5@]@Y^Z[QRO([> ,/.HX$Q;?H->UFN_FQGO8?UPE XO+'1[%N&54E3 MDC'?9-@-O=Y;@;QD1+ YNK?E1,U#8K+<44/_35DR.86:GGJWYHD=O_"6)-5[ M'8\J#D\1*0[_ .L6SV%_.7FRQ%XX&E?8(V\B-![8+\N*Q3RU>/9.KBE4K&TS M3?F]?G5^X&H-RY>[K^%)%FO4"[Z_(M0F!YJQOW:32,/:[^L&-%D<7NS1-)+2 M\"^7Q$?\@E,\^XI\G\6?Y$(#_,M/30GG^;[\GEKWXG7V@W2G2L^^B\W43OQ& M$A^#(%'M-U(0K294PPL:.,RX-,GOMW9S3'@1(*C>Z* %C7Q!J<.N42\$B$K> M_#R5E;2_9,^/@9V M+&!>=""]$5F&05/K PU/ .8HX&P%%Z)X8&@;$BQ1--= M+&U^9$F-)5^0E ^D1#K^\466:OO ^B419:GN7C!N!N?V,DY3YN049RF>W%E. MW-S3:_1EJXI.U"NB3R_<)^O@N6&QEOP H5<^-BLOB&,1R,$-MJFO8WC!QOX8 M,PNKR"I-Y/&TE=Z#_QE>$UWQ7RF@8.!P.YK@:.$[Y!CY:+7TJ0PQF5^=E[J? M":?(U. Q_'*TPPB'O$8*(4/CI7PC*"LP 2EB,_C@3#>->F@N=6>.4L?@Y:T3 M^%CQZ7QT/AKS*6Y1S-;?1QV'RNF&N8G><4O2M%&:%,^ 7U"_%2.FL?0]\)# M&NGYP>, \$VS%^**K]E(HIND#T&W[5X_S%]BV.+LDI9=H[?LW.WDV/NX'\X#OYO6R?6OO8*KH_YB2_"E#)3R]> M\E5>?<&PH $RZ.?9#LRMW@_KT_$UHK+.GJ_:O&0N[9^BO#FVO9O-;?_PM(IB M[[*BXNN('F 2%UFU6S G336.SL'0KF./!$U=G<>B&1_8_TKB\^Y#>[ MYJ3)I^88([CMU/KR[%UJ(]1T]^-JKL]T-:Q7%#D>##F>['LF'\O&R2>:%&Q& MTE:\#M8[&F[4XC@_1_"Z&2%VM[S4X G P]@=W%G:_@##"6"9=I^1:M<<)''B MD;.-'TK#KA760@B;@Z3M57=]D],7(0N>4!UA]NX%[+7S/^-J)EOYLVO8;P0; ML\V3OL(.\.\O.@=-].P/325$#CD&??AOO?,$GP,3#*W2EWT9868?Y=CDNX$" M4?-J9L;;7G;V?4H7RH7_;U5TLV&3\OO4?\"5AB,I (W&F:\2X &V40]S*0), M :Y>#H[E[8< ./!9"W,6\?WGGV5W[KN5N\%2Q>^+-AC\$ULCP'.:"5 %. 4R MTM#2IJ$!_]6PN+8EJ!HKG^JL=QF"'@U&MOM=@ENMY$.Y=JG789E-=*J ZD.+;?R7_4G.:!,@D.1) M<.CX;W4[?EQ=D/[J_&P$]5'10_V!:%'_*9K]GAS3&)/ZD\.E/A-7CZW='-': M'_>81\&H_FN[NTC)M^3__+#Z1\%-8!! "-#S) HZYVRE@NX+N';=,#C6C[A7 M.@N"4_ZX9*'F4SH0]62^[;6PYF.XO@//2M=)QO.66%ET83N&+H91L!I0"P-? M9-V0$L[#K$>87I@>KN%B1D.[_T9:WMT>:KR7!>)FZ;BJB\:+(9+O0<_$$U#Z MZ>*@3\+('&$>H6F4!M&QU EW(_E,^&=O.FWQ9.SV8!']B=9N?D#MTE7 M&46-AEFU<>4UCE<=;.I>Y87.^,D:*V@W"IY(MOV=6GF10.OEIUDN8CI>CO5' MU!"MC3JF,^1>5:VKOT$NJ/.W(?A^MXV?DDZBQL[.(8?S'M#JOB8NO1_I'<>R M]]-WDKWY\*>:GHR1O1(OTO"_5Z6T=D\QO\^V"?#R]]D;__T+O^11P^^8-1O& MQ9>^1T+Z9_.M*WW?1Q?G]V^![D,)?ZNPPH_5Q_!?T0G\YL3BSH?_QI_PM_;? M?V& G'H&/@3NQ]]1S(J+!FJ4\D?WJ$T0OB_C]V#RU$B5\WZ]IY%2X9].A4<" M"*;U'<68#_#;93_3I\7W;&YKH1I)'I)_IX^SM^+Z]'MRT>_K^S#>60RTY\== MXIYUI8@( ?1A?B?-T^,$]$KT:'2+_,:19C9N)?#U=86,O\)W\A="/#S8^3:[?2(JTQ=N>'?E6NI2?P? MY5'<*+_C,G$=>QS>6@[::O,H#7 P@% MB5^>GCD;,!I4X)BA-PIG1O.F8L)7D&2H\+_71UL:@_)^<3^>921OM<9Z=*@I M03OHJE$1Z#X["MY?-4:A09&J'.AEHE(7HNYO1H%FQ"]OY590JW*P'0F&^ZNA M(P.-467?7=J2N+LZ5%%27<%M2 "B6:=345! ?-([ U#B:2T]P0G]6AY/]W][3T)*B%EJ?WLSNM.^%="WRQ7;4; M.7?A2^A'0%"X7<16J'J/;XQ>YW3V=')0KGIP6CAI,%\=@'!AIT/\.>QL4G > M@#X!2S9?2CQM46*,9^9TY4+H=+)DS$+G4@M>RS;\;&YFG3N\>A]"]6N9#7YG M&'4X08QG&W4@;+1ID&E@==IRA5?U+L9W"W-!.5A8?U<7?59'93Q8<]!J4V^S M0T(;:&LK"MUBD5?Y-!4XJ4OG2.11(7D4/IHUF% ]-'WBO>O!JQ&R7 :5+C5K? M7UU\""H-7!U[#WNP2\L[N8 "0R5T+5^?>]12(7M:5$-JMSJE2_Q0)DU3-DE2 M'V(_&SM_,GW[2,!*31PL24T<(C@U>Y]HA'%7;+YX@W1($$]^C6-$?6=]9E*Z M3G9_%G/830%>E0%[?U9[ 7E2?:YW6'8$2#0_C'^+7.U-CW\-2V%]M5]'>=%+ M>7Z:3'1??30X2KU\H4P-;3]P07!S-1)&4G@C1U,"%D0Y'T5M7T^"2;AP[D:W M6,U]NW"D1E]1'SP4.K=,NUC'8$=:?![74BE/5SKJ<-K3?^'Q$]E#IH@&\>HX W38=LZ'P1?,ERE0F90TAUO#U+=4D[ M.5X0-DH;I0$V?.9+$7E=?-@.8W.7;3U\\4L%@"!LC'M&? @VYDN30P=H43&J MX8U M?H%C@("!WDPL-A-[P5LE>\-RCX'S2,1'1QR*@#M ,7U+:SU_S3UB'!\%FAI\ M=#E]?G2$8T5_@E_)8F0U=0'J@&)2?GPA2S%!@7Q6?GAV0G?R@+1HPT[A/(=1 M_$Z7-YIFY47.3H-_'')B2[A2DGSY@'!R:F[V-8E2YT6!.>E%^DZ04FY C4R? M6.U([X%R?FQWVT4O+Q\R%:J?YY$S%+ 7/U3 M^FS$>6UE^W.O3[!0$TZ&6.ML!SM'7A>")$\(?Z]B)X%K=,$THX"W1U)J3EG/ M3&)A\3IZ:+]^P410.[P\D#115B):1SH=4Z-5Z6@;4[ Z5CL3@+\^>F1C5Q0$ M-#:56$)645]%W)+^'3I-M1PJ$PG:9I(F&D0 M/5]_=GM*@6Q?^E N@-)!4(%G?\Y@4X$/ U6!25E7@==:68$">A='*UH:;UZ! M&W]A@9ADW&W[+A=49H%)@)\_2X#^?SL;S6IP@10U34I*6_-/4&J+0D50K'B:;0P[;G8]59]N+7R)@8M 5TI12B12"SS#2)2 )D#? M-FAK U*_7IJ A&^-11QM[DF/:B5Y.'K1(3M"(WMZ17 [R$F+:D!O'(+20)@1 MU%Z;4,>"J8!A?&-M8WP/:&9QK%N9#2U+(E(L&U]\CE2*2CY\*DP47"M2E501 M8YQ']2X?!:&!K0C6@GU'@FBX5#ML\E"]5$%,:E]";$R!=WY+2F@C2%)=1ZV" M:WR80$J0'=2T'/,=@55'U7^0WA<<%Z#?]9V941A /^! ME5*P;8E_6WLNIHSG4C< =NR'4@@P>"CFW7>)%NF1O8;01I)TFI?^MN MHDE)3EU8,DXD:TA&%5IP3VUEQD:O4+Y&&H+X7!R"&$J%/^=1)$^]6EEAK%'/ M@ M>#%7);+II7$WR6%5&-1_S2>9I=EU%<"53'$PS@B=EI3SX=^1PB%,<5.Q, MACSD0HY@Z$BS.ZE6\3I @KA3)&1Y:!R"56>".11"TEV4@EQ&3D89.WY[#S5- M@FM98&I+9:AYDF%E#XYMR%-T62N FIO8TV!7H+N4U&!(DII?_ML:W]6@54\ M.8 J9UJ!*&GA1FV"+4YU>N9\$55Z3V>!DV=64VJ!NU5S9D^ SFIW8#9N'45( M/X""7DG@/3>YJ1GEB1()7-G^0 DIR)6)O=B$*7&OV43!\&'UP M=AI]0!LF?TE2T5U;3/5GM(*76[:"3W@R?^Y2KCK1@[N"^V,/"D&"Z$S^5BT\ MT6$:<7"#LE,T->!=NVMY1Y9PM&AB<6)\Q4/L4T-YRD,$M50X;#-]R W5?-/4D;>0'#2+T<_H+0=TI^.7M' M?VPT@0$%@Y) 6TRC-$YXT0+6;*%%4#V0-$,ZRCBT-#QZ.41\-1<#KC50-7@Y MCS6#-4!F/65* !$T,#Q' #UZ935?(Z [; YJ;GA5WCU5($2 FS5$-8\UVH+( M.^X]QSD_ CLXQD-I.9("QC@Q4BM)@#W_@M TX6<@-D4 GP.\,Y@_MT:"-/]3 MRT (-ALZT 'G OQ$0P ?*[,YOFAAA%@@%SX..38UR#N&-1%#!#9NA#%2IA%* M958D_SM.92@ H#D09HHU5!^)-62$&#IF-3Y&'S_X-2,V&CA#(+ U2@ ]>J@- M*#9-@5E/53P4-@ $)$=6+W@YQCG4/"P['C:H4W](T3RI0PLVE#FP 9Q/!D(D M?7:$8'FH.7J$14A[&B0;P41_A'LA.3]# /I5@X27 86$;D#81GL!40**A,M/ MC82* A):D(1/9 DY(#;>-2L!C3KX ?L%_S3Y*96@:1#KN M-KN$%C9835, 7(33/!HVLSG#A)$\%#N1 UD$ GE"-3X\B@(B3AYY!S4/#'MK MAC7T'NA6S(1I/,Z$L&9-7&PTT82I2KDYUH2T/QLF7V4I+U@ 6W95 #<_5@!% M /D>%#SN/=N$UB%-0K98J W@A/Y,"#;60($4 *2A'M:! HR'4Y7 MD#4, -------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:20:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26029; Thu, 25 Feb 93 12:20:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06397; Thu, 25 Feb 93 12:18:57 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:18:56 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06309; Thu, 25 Feb 93 12:17:22 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:17:14 GMT Message-Id: <2156.9302251717@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: second part of document >-------------------------------Cut-Here-------------------------------< M(#?)K;\YTI"T."TE/@J'],.MM=75'0(M/04$ >KU:JTCI@ <-F6PZ#W6I4AC ML!+$=!=!7!UG.]+M]FK=<=/;'S? [7*SW,+VO[URC]SY]M3] 3ZC1XP74O7%+W?YV MU;UU]]M=-]9-+6OSW8?WUYULFQE^-\Q]@S?B+6]KVS0W"D #\-F6 M,*=]3HO3C70D+4T'W3GU;DAI>]-?*1Q= WC3201_4%5WVB@ QIQL]$QHU>P M'! :MH) K56SVWLW'.UUI]Y9]]7-=9?>8/?TC7H'WM=WQWUU<]^ -_;==R?> M"CV]1U^:]^F]]\=ZTW?P!\RX3;-?NMM;G?0W?1?2TD MUE- 'G"'H $;-FH#%F )Z7:HYG]SW-1W"0Y^Q]S5M@(N>B/@*S@ OH"7!@WX M#?" (Q$E@03^>H_;[S?E#7>'X)0M]/UY2]\F>,OM@A/@+3@1'H1[WZIW]VUX MO^#K=I;]W.%F;3D0#7B&0-V#:]RR M]D=!:Y_<2C@+OH07X8:W"KZ&O^'_-@.>3PLK,SAI,%Y(X#: ^ST% I3!N;- M1A\!PPE-C6>[W97WG^IT/]]D^!Q-2Q_@0S@ [^=A=TN_>G-H+SW4HX)&Z$+]N,^"H^@!_ACKC M3;)MVPMW@HI^Q[O<@@2. [C? ET?7G3W!9H""^B'6[W7PD"H1B_4/K@:#7H_ MXJ^X&JZ*B])7MP%^>C/CK;B\/8NWWF[B#(XVBB@2> [@?IO5C8&6&HQ;#$IU M,DZ"I^+,MAONC#?CU/@TKGT+X-4X.AZ/4^(Y 39^?K]">,R9D(MOX@[XSK95 M$R@;=AP-*)P%I39EW15!Q>CW/;WCAU/1<(*_EXG%T2. JFQ9--%CQJII'8465OTA*<,- M/X:ZUWQOTG/!^IU/:\BC=?^- E@%Q5XH?4L7A)Q@7#U*Y\RW-!T3:5O>FS52 M($R_TGGW0TW1\=]6=4, 6YOEC[5@*$Q/W(]UL*A'NP "=:C-T_'?M#18+@-> M!$1W' U[NXFA05ZM=TK+ON%<;7GT"^GV8PV7V]Z3M# =\3D&,=T<0%_?TA*B M I"0/PQ_M44 6ZL@70!>WK:IU8!Y0IX"C '#M!NP>3??M[2)'9=3$)NU"Q*8 M!]/+MFQ1,<3590 *8):SY"0!N"V7H]>WMS"]!)0!7<8M77$7V,'!.]=EC-(& M0F ^QW6_JT%LWF.[#5L=F/=8OZLN]2A]&7SF,T!H#GOS:YJX<,V:U]*;-6(0 M:6L*7CDI#9O?TE/&K#AO4X#"- 1D%,&+0$;^)I;Y\SR;(Y5C])F>6Z>$@K3 M6'DZ)YJGY9AL8BU"= E+B#J=F\NKN_C]&%.&L+>GT$-ODT+BK"UYP#FW=)WP$I@!E[>;Z5K.,H6_G M3?I_7I?GT3WZ8_VCN]5_:IE^.4IP?_0HG: JY?_X>7[\>>A\-'^ -HKJN?FG MS7:+VE0&J7UFG]J485Q]2[/:4L:#K4T%.@\.\4U2C]()0K2>4B_;U'JJ#5MC MZ^KT+6UC6^%+MG:N:JOG*SHI+2MNUO#Y*.V9W]*(@=)M H)Y8+5],QH@3%.#/X(7'IQGUQ%/MXX"1 C.>;6ND*_9*$"7X%+[Z=EY?IZ0 M.P:+WV9-]1$:01U>'DB3U*@YQ%YT5^MP^ACP"@;D"OE7/F]CX:/TQ5)2G]_Z=M!E/-8^NP7N$=CIJC9J4ZHB"5-OM1C>&#K6GZ(3X M#MX@%.E1@<&.L:?MFX::'KF;[K=ZI+ZBHS:4^T6P!N#EG7>:#N8Y<&QT\6ZY MA^6?A;4QH\/FHW3FQ[+SV7C,4854%^ZDN)<.1U\&F[+.AZ'_Z7])-0MX;EN<:N(1Z8+/M?KI6SBQ7!E.#V>>K2Y6:&;,> M;@_K7WJYO5V%0K39ODM;;,W MUU@[P;ZU?^47NP6/P3?7)#R8EYL[[EWXLHV%N^H*0N5^O.OPL@A2?06D &L& MF!=(L^R<.8;NF?OGU#O-C'#0U(1[UQZW/.SS.HX" M4 E]N)_NE:\H8/47;\4?\/)W1!VT0])X^6<#1S?G)[NYGEV/!D@U'\_(9_"7 M=WY%2[_ICKE]T#\@UH$\F#=*-P8H 7[^F/_G^'HZ;;8[Z;ZA=JZ_,^J;NYJN MFK:8GY <@[%V9]$%_=+,^@3_K M$_=H;O"A\1:W0@Z5YS8-?""M=@?:O=SJKEZ'XCIB[(YT^P5U-]/]^RGS(&H- MGD\OC8_!XOY8N_#F^F-ML\/P.3.>'9R3WHH\[(ZW#]/Z^YN.F<_F%8-O2Y"# MU6!YSR[% ^CD_-)]=S\K0CM;SD93Y07Y59[0V]V N;O^MD/EC[S"\9[[! MQ^^XO"!.2LOHI7S=+L7_[R"JV>U)A]OM=S>O'13B\W7%]!+[W[ZS'['IX%(?40- M5N?KY?S=711DV0B[P3ZAP^KU^DY_1]/P\@--#4F_\?"Z,J_9<-K1H0[>O9O8 M0O5V#E8?Z)!A,)V;8]#I^;V^I]\3FW4N_UB+?U9B\\V?6^D]_?M>NP_V;H!H M'FI#]+?(]'1-P@?/<8>RP_3,#Q8@+_/ MI\M!+.^UP^A]_59^?&#Q>'S$GFKCY1XY3M"[!>7Q DGN9 -U9 &W4&8T;@ @ M==V2Y]/-]#/M94_3U71-/F;?Y&;V-XUF[^04^^M-7;_9^560;CV6D%@7]G[!?2U0UZG6_2.>5+0SM_2V6NJC5#'YSR]1-^>>P<8>ESO MN0/=+0 6'HM<;YHX%W^AO^_;;3]]GQOJ@CU9OI4?ZM4(@*^V#^VT=,4MHQ?7 MK'PF$RS[$A_AU_+XN<4NOE/QON$_;^!7 MZ 0^)(V%1P:I^7>>IN75?CE:;[LO"O][A["/PX$).UB-$$CY>[9O'[5S"45\ M=2ZAHS:\<]6@OR_F=+I4 *@+\8\U2A_GZ^I5184MQ-_26W%ZWL3W]2RVG<_9 MD^AO=E[-V(OLROMI+J]CU;GY/7B'6/9V(1%?X,?18WW7GIH'^:[YI*^>8_G; M@9\H3.?R87EMDZ+3+Z#[1 TX#W'EO>HYE)?DS'U&T+K5:N?B2EZH01DNN98] M!,3DUSTTG=V+V60V3N[=Z^1($\7^>X_W0+EY/Y33V4;YG5U\J]1"=32OEW?H MTCL?;=KDXFE!;FY\/^O)]R@"@C??TGU<5>R+'>@]2% HE :5R8/SRKDB1]QH MG!8X^.HT$@%'_PB+GMGQ;1?\Q#,]#14F':EY5&!C2QE\.1NM C %9\ 8L&+S M>9+#7ZT"C/PF)PEQO20C5$+]_+@!:#_N[+9_$U M]G:Q4ROLD?G[_M7?TKL!8O#!/_HG=G NTX?U3?MP#WN/!@X[4PU T.N>^9E_-^D-GIHPV]_IDC\<3_F/#T8 :L[QOWGE0I8MQ/?_^C_[CAW K =)H^6Y MZIQ_V;\925[-EY>;:R04=U!_4X'(7D4O5A>O@U!4#>QT$CY%'?$/@[:YD]'1 M60=33@WGZH@P<9&>\Q5 M_U -, ^W7SJ9#8!_"P(\])\=KU;FJIO7J6D2PN@YE9GF3I(7E:M=V/:*,P= M_&)X9X !(!*P]49A"\Q% 2-ZW+^[VSOO3D>02_7-YW1 6@$0(!BPZ#>GL-TX M:D04V#5.CJ2FZ;<#G."4.:1^QK6QGA<.6157>Z=I4T@V%C6BW":G5#.W.>]9 MPN)=9ZLSH!F-DE;T^PQ@ B-^J(*)GPJP"! $8 )( 4B!1 "_T!"G]99_6_4= MV@QK0 &6'4\@;8=!(\@E./ 8V(V_G2:PY]=FRVQT<78V%[6G0\HK"" %. )< MO9 CK;@P5B_K)R<;5;8*!#?_-'<\AM GN!GL!I8"0PFC,*O ;2 M5)* ; ;G7W'-+]?5,]EYX%!V0< S@2[0"& -Y 5V ADUZD 00#"0XO<$6 +0 M_OIO^;0G0(0HW;9>8Z,5?9Y_ZIYJG8^H%]@'Y--D4\0.-[=^7WZEYD,(1.\- M^XY[(KECWW*O!* HRPBHF 18\0)GWY4MVE?=Z[+-Y*Y]-KDRF[I-N@;>HZ7E MXZ&0*>TY9Z=C>K'^=&PI9E,_NY M][: ,3W:'KU.5E?22]OI[&Y_2KS-7X8/%+2RX_OQZ_A[MS1%'NHN9Y:;P\*! MU?1Y< #DW6K-:)>P.[/A!C=KD@-873WOL6>&N-"9Z7)ND[L(VL$NI"=,\^=5 M[)88K+KY7TVOI1=I>_NQV"!HWJ]671;PUI>> PWB^?!N<#3>H,%.FB<;Y.W9 M_GZ#R[;@X$5OF/1MRZMA@58#Y8<[FJHND::Z\_!!^)Z#BXCH(-?NCO:;0VW( MJ\ "6$#LH 3-OS:@@Z-!]%!SWL%:7T5 IE.9,_5YYB!IK$''P?6.CZ9?T,3) MY7YW8#];G^UOA^8;BN+]['*#8,$D@)7.H]9 0+55A7Q$(#:P'&H#O!4!'*5) M]KQ\Z$'1H)H/UJ<>X!5U^)B#8;FJ@OR/;+=+*WM(9R_!= M"*& ![TE763//H"0$\RI_X)S([WC ^BN3H-\&Q$:UBH#)4*!H)H@1;C?\XG- MYUA" ;7'6F)@/O=5BP[9_9IT;SIRFE BK\:R&^LU?V!O,@$3G8CP=-?/>]>I M(\8_P 4M1)&..;AL2PS2\42#Z#\!GPW/KE?68__-]*I"'S\>GQ=0!A05I.,M MTV)X;L"LG.PN3F?HP]HM?BYOV;7P1I8.-O>J8:/-A4QY*PK4W#70["#)XP^4 M<-HV$K\4( K@"I $8 +\O48$63;4G8=OLF?\Z^GIZO( Y*CQ'VKC^( :& ND MYTZ#?D(-X'M0,:A-B^19Y3!9G;NI'J/0JX:]0YKLXX:%'CR/VA/0E-?J$Q3N MA[2 =\#-VIA@/N>KTXIX O-J$+WK'+O/8Q<[^.#AYVJ!\[]06@,01AW;@.M3$K5-NQ M[0AWD"(9G]ONL<:R6T),#-YU+"' 7+].1N>0\_I9'J1_",-O8;V0"_@"#%% M]49V]AUH8)L-PJ:T:PB.^( VF @?H"4P"(@9@8"= ?^"@0Y+VH1M"C $B.A$ M],8(!#Z2X6S.'!A'L]AYUF!K,<(=VAA0JJ2TJP8N"Q6#VC5P(*$-_X!6"\71 M,

8#F6FS0=*>=ZQ#JYKQ_B\)182&)O,>RN=N< %\YXP45(*O05;@8M-U M1<0."PY+X *OEI,2I SR:31JYS1)@2)P$)>OLP&F]P1$@<"Z&.<&(9CT 0+V M;E@G!C\C !MM4Q>TVZCU &!KE[C\"BHMRY86@@_Z! QI,#@4 (LP7M>?2PZN M!GQ#M\A!0,P/O8P>&!;&!C!K:(1X#*/C*$0:2 HN!QT!\H -O&3B.0+"- M Y^!FK^M84S 3:0 \!K*#7V!-@%$T3QP%$@$J ($W82'D#2FFBR0#-!E<$6P MT6QV=4#NH$^/[Q>["P(2541U[4!.H%2)47,^G!ZFO'Y^0;\4 #(PZX>K2ZF= M_=2#%#[GWRTM(=#.F]!-VNQ]LKH9R3*O=I@^Q!W6Q9B'[L,B0!- @?CXFA]F MV<8/S,+B&OZPU;<]!-3Q#[N$A**!G&_H66'TDQSB)P $@L-237GC;C,9+ 1^ MY"""QC[EWK,BV<=DT+*@)S2"T[TL6T=0)G=[H\EA^[A[(\'O7K>/EB:#0PGN M^T*(4"$4(K\O,@@3!"S,!*])202;(,9/X<=&8_BUV9PJ:YN?X!"1&X@DC-VI M'11%!"%X?D%P05-03(>A"Q;"!;$=#CO"8%:0D496,P+0 M_&Q^.#^=GU5N$S@6+ LB"7]V@;FFX/?O+9@" S&!<%M:D3#(+^&EH8T)/C! M!0")++_"8-3+A-_JU M!Z=R^L(*7H)POP?*\]J)[J67HR0K^>VB]EU&8Z#[RO7(4G'U!<5. ' \- ,'SVPVNJP\N<<7!B^ M"3=IHP%N7'C-:PYV M"#UP<$,0H-R0)&"[B2=.<*:()0&]8:OP[_5(+"@R>X2&H6PFR@30H[8V3:';38C@.<0=,A_&QU" MTDJ'2+$C8.IPF:BYR\UQ"#%T[T0R(+!N%Z@^A!X&#H-OO$-1X# 0>"@_Q![^ MO(B'S4 XVA,P"A@-) /.!)J'/$4#XH(#[99 K!Y>#\&!VL/9W"P0=0A' Q_Z MAN!X6+ZLG0?N A<$# R8 6V'S\-LH%E1'BA4+ + #]\,#D1(VM@06-.CNQ\" M#F!KO+S]H2=C5O<_W/HEYDR :#?GH3MP?3C!>2HF$!>(38 &HE$1@@@V-*Q- M$&.!6D7NH3#M@@BGRR#.[SB(!4&V29L&Z0-$S&P!(< GXK :0 3P#K(1- Y<5A"P(< MO3AL10"S8& N+Z*QP6 M GDH%T5RHGU/@,4DRN]1$#IY[KW3(#PO]]<_$ '!ZFQZ@SR*@&NO4/CU^_"! M_=R)!SW9WJOO".3:V]'MV3"$S<']'I]N?*BUV_5E"_EHG)M-(9C'(6?B2R7V M$W=Y83SC7>^O3N?;FR0T]O2%XD)[86?/D$:@TQ%@CYSH71N&9@NO*.1 M[UIX[L(&(+R0[<='4W"8Z'Z#4L,.'K1@-C?;X^NI!P>&[;Q1FE=Q0$=+>P+H M[1)K?KD)8WIN@KBR\6P., M[ZIXY+N;GI9Q-G=F5"76ZA1-'L8.W]<(MM8FS"A2\6!K#@)0^">Y4P_FYBB+T,0&XY"0Q(A 6]R%Y1P("+J#@3JMT_?54R7* M\Z9V L,WX]UNT3-*&^49[ )YE+\]HF*PUKAHU-4%!#L&>+F?H69F,&BEZ_RA M!E%V^3L&X9[-B'?8&QVX&3]V[@3'W^+.B\,8CGDTQQVBY4Q)F_T(U>;7_W-F.-Z#KFZ_9Z:)VE<:BFP;0.=AH%.@- M?PQIH;V?EY50J;A94^39![.-L#I2070NE <;X$V<^0)[HSC+6TW/ PA0A!,V MA\!M>:!P8UU."HCE\S8.'<@ ]+5^'6S.(1?F0T4TUQ)\"8&"X5?@PC9GA*V% M"\H Y[EGXK9NB",#I"9&=3B*6I'77S810* #%!!9#F5".#5PHAB136=/A*0I M& %S9[]SXFE0G3C;Z^K]!AN*)46[P,R1=1*2TP$J!/>A%0>+/L6Z8W.Q=TCQ^QT: XN*M#BY(E+1>.@,C!0F M#X. K0B[8P'Q'5CO&B\D$-^*146Q8?UP>>=8O"OF#]5^W,85( #1%AA\X]P0 M$&^'C,= !]:QK<@$> (, 4J!*8!D3NNM0%=!=K/#IN*LH,@0X!8B6P$!A\A ?V%DV(HL5]'VD1V4<1I.04 MD@I);I6LP6H1R_:2V[*]%FN((,'MGD@P)S>!VR&RT3IR/[F4X&Y1B.A;W-/D M;:HR*$#?34W0N)CP0RXZ$1U^4<3>35"1BO@R!"[8V&X1ZT0.VU30--)=#"3& M_,*+;T2OH!PQ@@<%* )@ : 3P I@- /!9 7[",B'=U$%IRD7R&0,&=@[/>I M.V'H;@TXZ'P7?=E5/#%^-R&48%WW;Q0^=:HT^2Q M;!!XASNX'&K#"F=,_-!) >-_-#:Y7_I/?!ADI!F"$4M],,:C3P^/-0$S]*BE M\CAY/CH-G#+(.8=[J/ZL$FF%8 T"GZ%0N+?$:]+9&FV0*$.(@09P_0=[%.:! M[IY^WR-P6TQ-,K?9<_&)!J^#LJ*HP'YH8)C+,\VY['J,"#9[G9&PI5=_?.6- M$H&#ED=$WU[1A,<3V-M1"05':0 =1(5QF&BFV].X!V]\JS4U8VQ.%LCJ\^>A M-B)$9*$U)![O*S!+] #R(+UZDX3_G68FFX=1]&UI%!N'CD"]4B0PNR8$!!Q6 M#B^!-X%,X-VQI^@)3$7N':%""@ 58#WP'HA5/-(5U]2,"KE;&NY1F,:\V,#- MYYJ0G,=U36 ?0AZ)"SR!X:1CT>@'US1J#A7U*-)UFAI$\2\8G6N?ZAY]"LJ M#^$"Z\#%HS$2&ZD 2""6 D^!08!48.BN]38$^.I)$"V/%,3(H@5QKTCWZT,R MV3J(90)3C9RBLZCP RWVV8A]R+WEHVG1^2B_N6^@D:)[5K888O5QVG=]_+)E M'[-]W3W:8O>QO$%6<\#E%L5]"D'UGEI@$T? D[C!WS9Y_,"\ !NM_I882-NU M KF C4 K&X<@3G-\O"2&-P2,PL6,FMD "5)&HO8CSXK 1 =:+>4$HX'R1,I!'/$JJ0L!M_-'NEBBOI=!JE^"\8S =9/AIBCZ!XZ MU 9WLD9QGHQPU,B98]GYAO9Z,CK4'$<@9T;0J_QA&25ZFT0YX-L0#PD9F H@ M[QQJ^CW:'[_QEJ:(%#-F,:QUNT+8FRK$8?<;G-!1'.F, +T:X4"/OY>;6R:" MU7)Y]C\IH%AQUHB\FZO-"2MX(D@J9'IN]@C#X^6]&F>#SS_H() .IYC$N\#M M#^T#?[\W35ZMPBBV2ZP)"3U] SG<'6(MUA@LE/4)AEAV2(&/WX.O8J&[$*F MYQA[M3Q'I"LQHA:SVT%<[W9W9;GK)&RM,PEP9"5F,0ARF3WB'=R1Q3AK3!3J M$@]ZF4+"85[-.0B_D]_MAWB-I+S)'.51%OF0=C\E)5!.* &^Z2PJ M..Y;)4F19/(Q'RD17"$V'Z%L,A^9CU8$,3%]Y BZ%CV"V,?8HO9QMD@25-8U M) -M#TFAG$*0M^AT)")*$B&#\4@D8DL2X8<3/.+H!'F"5QSFHN_&N:@"S$E^ M!FB"6D2?I IPAX9=Y*E9Y9B"'[^BX"N/5+=9&_FQ*_N(V4HT8O/0NZA(7$IR M!>&(7\%.85C0"1 $6"#*%]&",KKZHAY1F+A_O/4@$K^+6D&'6B$Q,'A(S*N% M)<>2=\&L)%J2D7843%+^%YU[CD&>#61PT4:M'"(ZZ"R#@#0L UY2,_A)=."% MT@I[B#\)WW[OJ# 0E$/"UG8#M,9B6<#P2$EK1&VP]9AZYSSV7(:O-"E,A-7% M_A!ZO#/?V1.(35?CTQ<*U>Z0P,2EW7R.(] X-#:N!BP\3I@]P) M$]]R]$GB'VYP:)=YXZE]Z=!V9;FU7=K.EP>%G K,_^"%1LHZY))0R-@M; Y) M$U.6,T"=H34Q5)=-]"R:(DF*_(&CC\XQ=I^NAYK2&_3](6KUJX2A,T]DA M^N) @;DVGF9O.L )XB<>";6&8[Y8X\0-*WDBBXJ)R\!C16RKJ M#*6!#HZS8OJ249/ 9"M2$5U^4H"KXNJ14VA8NRFB+F-W33K>Y&;--QE'PP/@ MHNB.#K>H8NC1LP&^?&!"'B,AGDM&)0@2CK:JQ#Q2(RF5UDBBX5,Q4SG"I"IV M*C^5D44@-I?ZN53E9W$.*+6.5O\=ZU_CR%+G ^RQ. M!B.*]T@AHC[25WE:Q-OX)--J[0.6'&M1AGBLI"$:))65",D<8FWQRT%+6R&\ M**N7TLJ5(/EQ AFO(-RD)+.5Q$4E8OO1*O=^%&?%'U^15$0SXA>1>==?A ,T M+/N/^D7P(E-RO!BPG"/V_!"+W!(I *;.^G?0TS_B*U<(IA'\(K]RO_BN9$LF M,0.,;DPV)J[2Z?@S1# B A6,&\C$HZ@.DF;VD=#U*(>V3\S;>@O1N1R>I[.MY$J^,![W<'!LOWAA' MZ[8Z5[M081\SI2@&=+?=)MU[MK]:8)E0@WEYA*WM(AN3ND9'89Q/XZC%XUEB MZ !S.[IBH]X20CE,$C52,X68PK1(X\2 OD9ZHQ02,R> ;SIA9FHR13E',Q"& M$C^#XT(^I.IN[>?F.3[(^,*6Y+O5'L:/< D(3CPYU@VA*.-+M.)N3W391J-Q9;/.^@A'85O6D2BW%?*^%BM)$4& M?;*7BD K9?<22PFBPB82Y2"!(\7RI?PQYA.F'%.J%,>.+$71(3B0L9A/#,5! M"[U_OC\OGS-QMA>GI(EQ!A>/4!M&C9O(;S-_S%,*,)F1!,SCH>%14!E\PVP0 M(].*8LV83P+QO"C!!"52'N%H%DQO9@;S#9"^XV"J,T-I9<4F#@W3&'E4,6J: M,)61@,=DX.0QHM?"Y!&^,/N*X(8 H@E06M&_7&!.<.(5"41/Y0$RASEAVV&V M&GN8'-)&+V)]MYL$HPSTT3J4F]Z2R2;-)[O44H)L\&'QD15"&: MY"HYD(22!22A'I?%?/91'Z5]U#[8HG8OC,E]3+.1,<%I/CD'7;@OC=FW6V,2 M&,V/$LCAXOK1)3G'_%8J%^^8X\KV91]S#G!__&0U( &#_,(!.,*\M.8NN&=9E/HU)* M[\Y^QTM7)A,/L\>S?-?)YA)RTP&V'4%/$>3:R\VA[FQ_Z[]=X4N/LR>:$ZI) M]=Q[CTGG(&* 8^8;>OH)(1UU7@'7GG9.G/FU<_.\!KIR@4M5G@:0K/"?Y%#^ M?IY\]LPWW9GME"FV3&9Z# >7YT)_IMN0\5?(0R=B(AMU0D*YG"<13:>XW-"- M$C61W U172#-O_F)[%Y:V68"G$J)XMHF<$-27)48#36'8LJ48ID2!6!V="FB M .27^D3ZY700/W?5+,L5&^6.6KMWW;10F%9OM!'.2$ #<\IB)/00SGFG)%>^ M#Q&;B\K%YCGR4>G"[&7",".;P4@V29P3KMD6N-UD-G&8?,I1Y<+OL\G74UPF M^!J;MS339J227(C$A$=^I72 WXW.XE>J9 &EC&*.%GN5M\V#8FXC>S?Q*U:V M%JV/R,HOYG 3AUC -^Z4=TY;9RC^E^E$G" M'Y&(>$QA8&23!0BKD2[F-!<%$"6 M%S>!Q8@I !5 O1F8LQX&YJ8 0S\')'SSUHE^?&X^*6.;!,:O9.00JX*2S'7. M<*!]DTQ!YK_2VOF4W 2^_+:=4E+$&9^LV84@WN4>@9A*/=#^YH5K[]I(?.HO>=E/$) M"94_YDG\G-=OV.EE\,#E+0^<.CY'W_B.P1FOPPVJ(%6,<3SNG>6M6RBM,-%- M!A!K+4/JW:+0")" ;!($ 7!>;ST(H4NM*"%0) #B]E8"#C8['0WK?F?C!'DB M'#A\;+];&LDS>0GB[-B-TER>'S]>8722:@G/"Y-[<+1[JHY^^R7]C& ]M-)XE_4\^V8WEN[?GB)#(V MX42%637/9?L/.O5>^!M&OUU.\^>IU@2Y_4!A*1Y@8:)MT?> G_,OU:VE-11 M /]J:0*L6L<3/4E1RZ;L*T^9<<.>W[U3O>GM#'K:];J- +V5 '_LB,?+;!18 MZ!X^";[J9$53[QEGO DF!F9LBT^Z)TVM6*C#8U*"VPIT@30C7_"R5>>U4W-: M%J^%SD_?UO&!MP;:/!+>(=%\/KKNI-M0&NG+$[GI+&=S,CZ97:Q1YIEN(T5> M\TTB6^J1Y!:&78VX.EIEBG#) /TMZC,^Z)Z30-&?">S_H M!UEL\L\X6E]2,C?WVPZ./#^8MS27IPYO3H.F.W(V"]]]UT]]H180R5CC#%S> M]@B7/TZ"W%<.<4G\4URRV(Z0A[L2Y*OR82?VPS:H'"?LK"=B%;UIM+2.(\&!=/G^@0#BY^)Z(#83FSPS(3?6HZ]! MTH:9YSPCWKKPT!?^/(#&\ H'ACKBY;RP2,G?8U[6N_YV@[P/Y@!4G(GR9 TH M.'>%+,\[&A=2 XIXC ]Y#0V>*DW%)@L"=?C)2PPM/X-%T$^"WN4S>AE\+ GX M0)V;'Z7 S>T&J#EU?+^!(HERD1! H$@RT*$#)%_>'!-$MS6;(*$3$QKLI >^ M+V5^$S9;9.6Q,IF^VT7J%<.,C\@@!8+R)FB)/!**%GQTQ3A1U?QO$&H^O'^* M,,.:-H$KCC>2"5!Z' *D'HV*#B:_)1SM,><6M)5M#2L3D[/&$R(_60?'5F M-LZ8RTV58'.S AF0;&/")=V=<4SV(T^MURD"A"(".[.;0L6<)+_&V"D/Y"*J M #%"RTX^YG81VNG2DG;V*_F+XD7SYK53]QF5' @ O3AL6@#VHGOS2-F5I(DZ M+ >>-]%*IA7Q8UG% BX.&#NBFTR39>O&%?')S&]B"L<+HDSBF]E/=$D";7/^ M&.-U;5!UVBEO":BYZ_RA.6F3/\CD0&(-<.F8$UPJ&0N7*E"[$ OT-,@!M%EN MZ%R&J,I0VB\3XLCNZW_"U%1!7$4VVHKQ.@>8NV!>"PYYLTQLXYKSZCF? _]\ MUO!U];L9$ :MRS#!TTP*1D6@&D\FG0?.+7K0U,U1+5MV_+WCH(EN#E!I.BTT M>%Z6P]"\Y9$P+/K*FTT*)C%"^,+_IST1-=>+3-CY/L^?W$G*0 [2T-?Q[#(6 M\3B&T0,[9&%TG#EWM%GN&<%M"T).T-ON?9=_4(Z.1<<$-$^.YC31HVEE2W!< MUW2:-1\:I5*S+0"BPK(105.:TIK6V\"'%!K1:QTR)G%[X :W(6#2]VCIQ"]T M([&C/$VIJ,SGIYDXU%XV.4.2#+LF#DBQ=J._J55N1YF4#L^P(YD2JFFFE&J2 M(]T*A,:1*"T-(X3Q)'!Z_;J:8\VW)MY1?3D(W"9J-_V.T=!DX#044+G6K &: M+XMY*PK+9A!'Z=A6](9*'L=T=TTVFCB4LCBIA&QV?BR5N0W!I@B3L*D?O6&Z M0_MOHDIX*)"ST.?H'&VB,R.="H(B)FKSB*G:9(IF*=^2"4&B'+\&MMG3]'0J M'T&=+$0'3J #%QCY:H@.)(.;R!&6=ND5:)R*4-^<4?6.J M'Y.((E'4(1W3UVG'/(F6-2E^)$^UPYN-N1@,3'96)[&+@3F>FG91C)AQ_"(^ MUD9^=U*C*""3X-E&;$H6,@F0V4XJ &(Q,,+ (H M2AF(@3G#X-<33\IAL_MQV/ <,%LBDV4DOF]4RE@JY".7PY#Z,_PRS$F+2UM M,3.0"KT-I+2",TC*#"5>VC*CL+5AYYA3F*;\W,HA2!&<*L\%9X@S8]@EA/NE MT99MT,ON:+YP>1=JHR=2^ 2<8SYRVO"MAL>^[I1TN43\7(;S76?V3,AA M 6>#O#U J(VQN]!0HZ7Y19=J.40L**%Q7%?^U!=>_L"@)LI1HYNPPZDX\7'X MYF1T>[TXGC0SQ8@Q9-P5/?%_ID_Z19]4B ?'L\XI,WVA&%/4H/MO_TDK/'QV M[@)YSD%^Z5KO\2F04WD&EORE>3E[YO).>(B:XWDJ($>?.KRO5 =29$KX9'T^ M(YN YL&67N_N3PC>ZI/&/M6%/TY16NX3"I H163"_*Y>D8$LFV>N^7>[N]7Y MXJJ&%,=>9MS.)S"WPVK^!I-UZ7Q5/,#2S[-P)#)>F]TZG:=3K-CHN7!DR M&EV#DR $'[L4VV@N3+5']N0!,UA\A0&FIN[??^3,C% M165V/;UVYH33#)G_C Y,/UV 8$'6HWLO %H#7(/F)]-SVL^OIP)TS=D 9:L= MTV9[H%,):$=R [I)$_4Q.@M]%U"L&HNM6IH9K1GBU,"%^L+7) F4+7KC?(NF M0'5[5D#K6A3M1.J/OR'NBY9*K]*=EH==.K)@F2JR DW>]M M&71ILST5:L547A@:\H+R2P6(W W%XS(49I-&Y8;V'4N!O=!?J. M@1IK L05PN>1T#E-N8XB.T>!T%#.YO#03UG #%1Z2,^ABB(:)BY4Y>)'[8;B M.;^A*%+&)CJ2_9EY_'.^2*4; L0YS1YU0NH3[([.-2.8E32C(NO1L+;59#&. M%8]N5SF^ICWTEO;7/(:BQB2DADHD8B,UV5D$T&R"*HV*BLXFXN_TCM8C%8?B M0XV8=;\B:0]_)Q;ALK*7"S(!D1U9(R M*W6(#,DR9@\13 J1E 26^^1]SSRA6THSU):!\T5^X)AOCKR?(8@J/FHX_ R< M -U$I,5\9:A&CCD279,V_"*9GII0#4J4BJ@2513] =VH+U$E)!H@)KHG-;J! MV#!^"3[] WW-P:E. ZM]"H^2<3:D**]TWNF4-&0.!+( -3\DP,&262:CN^6M M!6=T8 00VT.5)LJO4?EA2I6B%[AL)*J],Y.%60>$+>Q/3/$CF76 X.25>?/(3S) DOU^C\]!=@ M[V8":+HG'H%/FL=C+%!^Y4 $EI^/H=3..W!O_-GP_")SHTXVFG4Q?6?[4P[0 M$\R<>\\-W\W.C>?ZK+M-&1@1&#K8)'^/,4H^O,"I"=%T2U4"GT-N=BE!DS;& M&E^A,3OD)&=R:DE,^S'J)W&CQTR*@-F4^.=7]3-2/9V>,Z#!:JWQP;A*S/Z% MZD1] ,)C70+U,4=9/4.8ZPAW,CSOUPO0NM9)2]-A^6B<6;WK%KVN;Q2'!%'& M5G=ONCKA),LF+FIBXWJ2WO*:,4+:*:.0#D#J0Q:">5!SPS79JK9N78.#H?-= M[P)II=3](:!.40@"U#]83(6>_T00X*?PO+H A0)" Y%WH;;,Y)EM-7IH8:,) ;J'M;^6GE<@.@=6C*LI3X6,;U,J:(UP;)=0/>;A M\LQ_P\05'_0N/W@[#8/V!,9S@,M=7H8/=X 9H*FU[KJ+507B (R3LA:ZC*7A M 2145J*#JC@S-UEB7=0%1V%X%L X9&B2L=IB=$^^ZQP$=R&V7=XN7G9>LNF9-NC./A!OND DK3W([N71<&7. M$EMZ40BU!GYN]Z@/5=U%&D-I8+6$0!O22;A?'4%*Y_*E?<7;78(2[:?_B4B] M +,I=L<6I>0R?/FSB1RN9$Z 1Y7;&OE4(OD=19_V'(&J0$MS(C@3/;I.M.>= M+KV9;I7CWL_&:VA>?5UV!D.)I$I*H7) !X'Q/$_Z^=R47K_3 :2HN49ZTS_\ M!01JR9RZRR95?4@:@ <*B/:IPL"7'RU2#/ H/..M&8*CP[@7X7GPH.IS7<#DAAFMQH\S=P=(%]W,0UW7H^'&RV6VT"'%<$ M)XFLRU66J\=TUE??BZB=*%N#]M(40/:SRY#SM,XM"LVK(5,#IH#NX,D8_+6* M'8*M)D3N!D]5/JH0.99V(J-VW,M&H#J50Q#E+-5@>Q""AYJ;8Q+5?"H@?6J2 M';F<9TKQ*)N2?7I./%XN* VCBQX!(N9FT,E)G=MH-NJ<%]) :IZS=+EIE+NR M-1%)51EE*!\5"<*I?&!^*GN>U,[6&R4/]S=:X=!&. M"P6(1QS.:RPUG1<@2%0N(T^D8K]&)0MSD@JI9)%6(P&=&T2F*(@JZ$,3-!Q. M7Z.BUM==96VSM(@0!5;>;KX;@1L^#4!2B_G;G"%6^VR(LL6$9+,RFPI.PRUR M4YF;Z;UOZB;N]0;-*T.FVT*1;(YB*13J+;E6/"+6;+:8W$HUJ77S66$2_;]* M42MVHX.8J%5N,7E0734RR^0)#5C]G5OPXX=__">>)8$(.\:&:H;Q@D=2K=.T M5,F;%-5#*7IS"E!+)0+<'], @;DP@'[U!60TA<"6YPH('[\$PE%2DGE2A:,M M)L&;F!%-G%*29"A/"'C&;+;2O/&P[%>R+0L(>L'](]H(L+>#W8$.FD=.6GN2NE%H:K/P= M":M&&D!,7= .-?<"8F;^URRF7\ >; LPQ_:_NPETZY29"02E:9;M8M$@H.== M[_IUUXE(9/@P#6N!6\.&)V-S%"4USR%Y25(AWW-,(9 ^*J<*$XJIPH6CT/!+B M8;N9;[KD)YSU(B5M;'Y:[]9TF%>^995P&$=HU0/V8$.F D+I*:!P:8H-K*H= M9"]J"MF](4,1%TC>FIV):5I MOU"&JM#A'RPQUBBB;+2>\T9_>M:*G4#U%_DBG;+BT=BGYD\;J ;P](DQ?,6* M!FN/&U %X*AQC0,4>D8L>IIT"0&EP%ZQ8RJ[@_'E+U6,9A_ ZI+N;GKB&S4Z M8PN?E[[=7J0NGHF M3:(/.-XKP)UK%Y4$(C:J[3!T< -W-:E:Y!4<[=L8]G- MZ32(O,A\!O+N*$C]::Z1[DBRV()-)%0HM$-=3,DR9+$%<<+)[+$3)BN3)1RU M !>JJD/5Y^S/W0A)>PPT$,)\_$%$8Y/N@6HWK=#A31%L?3NM;%%V\2,ARLUI M,&N/4L#::@50&5LME&[PYP24)\JW[%$VV[AA=,M>8\6-C$)#:QX69$@J4%'R M:X*)W]@U'1\S'+M8Y=!5V+2#X* [6I4!Z5D#=!%. 'E\;<&)[#I3%4N0[2EJ M1B8X9T>4+,5/)=L+-(F@9W6HJD(#)#'"R_#'@QU"9@V!=L/T;.1+H<@W[ 0J M.+)W^EE584PVH]H+U.*(-IZS+]F^(Q9@/ML"O+7&;?)JC=FH(;SN9(K::*O= M @7*49R%$\ L!K(&U#L/?5U"8$/)>IOZ !I9+"A9E]S9-5!@6^6^/>0U<-B MZ"*KA5,^;$CQ[/B'5<,:[(ZJFS5!X>.T-1=5B(N6!U>)=#_QK/[NSOI_<[/V M])IT+=H]*X9.&LF')4\>"8"L3Q];QJ)JK/.JF/(^N1Y.$DSZ.(8YP _C;NV4R M8G5S407AH# -"V<^!-1X#3NQ:%IPH)HV EA<,X]BZ$J7GK_(7#QV[B@-O)@0 M(_&Q@=K6V]'KW@H_+:[539NM[5,?I^2.5L?I.XO&[A:N8->&Z]W/K,>)_;9)/&4ZUF&;)0&')B7+>C8UA!IELBO'EA-^>DZ!9<> M]"BN=3%;*,:UUL-&U84*%=FSZX)##3;T/>J*& 4N:/%KBJ+66ZTV(L61S=7& M/5VG9,A>Q(\A(7C)G'0I+G4E*; MU*@.5J_.;.IQ# =)F]929'%\C[_/9K74Y,F@C-V!U:Z0"[JE+!/RWA=1>[BN M;^Z?:KL1HW#6Z4;&G--(;7RNYUH.*$!U(=N?9==>DZ0V[UJ*WX!V)ONLN'%( M%S&VQ#-G[7PVWU-^2\?::@&;N-K>9:)5^4GU=(WN\[BUMM,YH:3#T. M<;*(:+HK9%A.8%MT:T*FVTAO&CSC+&T6=*=Y?=B> 7^QUM;LH2R0?;KI*X8V MULB39-@CK'(@LI>R=-A83(V2[5IC?-4,'QF6'UH7T %J M-NAIZM2Z0 ^UIWI"9*9R7T.==1K 3=R6LM7;W B>.@F2J*3=_Q86V;L5':@H3'+I\7:6.0Y^OE]3A+7+DXFK9S)XF$'NAJ$?5HUU31_IZO-/> M9HEWMD?B;.[Q9LG*Y#T21^&,CEC$ZBWPJ5>N93RR;P^+L%(^Y2@UN :\H[FF M 1Z@NMLFG29V4=B)_%GM@EV_(O/>?7]&#:UI((F@T%YNLV@8N\;0(D M :8 30!!:K,T15JQ(XOJ/8FRKD'A; I@#H "J$%F5J6?&L!_X(+.NB-PC=,M M"E.QI4]D9BMV:\C7:M\2%ENX[=!;ZA'T[,?#;'123^NAM]59$B4NEE/'<"V2<65;]*7J+-13LH2K9.N M*\4 "E@;+",B LOF=,'R,J6 $UC2I7?S3XJ!?>WE:!\#2S:M2+2S"#M15942 M,EFEFT 2K!/ !#N?0L%RV%2PC$87[*[6!2NUA>2RV&2P1E$:['@S]MK(S9EE M*P6#A%R7*N\6O GQ \(R$G5 *M4(W$RUDIN$I8FF_$*YJ=)4K*S2WP?&E<(N M_3*BAD.H'WXS,FGU^W.,,GVK?EDV&HTS#$L:)!3697><05CE+!J6 ]=87<-B M,QNYC[N8:S\Q#LMNG,."!\>EEM'D[!G617O*P_Z-%P!_OCEF;BW68">(E<36 M;PNQ6,!#+)TV"@FNU4UV]"YWBDG$*B1V#HF?F\3Z8]V&$=!+K&MMR,: 9.B5 M:G5 _EO\7R@6H)M/(\6Z_@:R3MCY=?MZ6*Q:LA9K#CW;F>+O41B M+?MI+L)L(:P0-?>+9>B) >%I2+4"W0BT&)M*9>CR\F)VQ]AWXVRO(#2?0[ I M:5EL;8"Z31 M&^>9-L+5ZK7QS2K[K!0^(NF QLQ6J(P5LC?2Y0(6,5>MU,*VZNRN/TFUP],A M*,&6>]FV0*&@SDHM_.F&!K%\NMGY'R#/\,?QK-@Q_L:N0D*67>LOUOB+18"" M2(K6RR8B'9G 5N+<5PUN^3BM_X\FM98A.0;!W:I!2 M2X6TQLM\;;^QV@B];-U%7*>CNU8(6^\MFR@3P$^<;?^C1QST92=7&;>CS+(- M:BFXAEK@G*(6HFGK\_#%;-FY7$L.'NVU+)>G=3+N:4UW1$,VG> M&.-\1DKTH'_PGCNJW>B.2:5R*[=^K@DJK%=)6 M:X=X"+9)+@?5M$MF/:6F:I<]_-W!XL\UT'$0=!^N:U>J$-[8+;Q6/@NM9RUMUK!J+6VN$H\?;%-<>]WIO$%MXX&S7)2K@;;-A M>"NHAM"0;;R60*NTD]8:>/&U*MM&W;[V@NO-_-?F[NYHM=&)[CDUW6:P70B: MZ&I]L]F)3JKVOQCA9;=J7.<4.5XJXH7W4"/KR/+":S>VGMF&F^T&R^O W/ ^ M:]UH9CN;!N),4M3:NHR9D>W.^XJ8I.[[MPF[R-I'Z>S5VZ%03&] V:A?EW7PJ#Z45GT>D;="'#;BT/?LU;>^H:;LQK+.QDDNU#3>: M#T-]6-MVGM:V]9:Y;6DV 0FE:UN6 M$V6NOKP6'E27/_CX.Q*V8O5W=\A;8!;QJ_FZG?:6;D6O:$U"ZMEO=;MG*]K* M$[6A-1O#IN]P=DM3\5S.;\.(<\?D:J@V_EF]*AS-!JC_%5@HM+2[Z"(*&1S-<^)U^Q' I]/8:2!KZ]U=Z#+[/6=XB]Q:]I M;Z^M%-QO+8NWRUK2A=6==,6W%U8D;?E60VAR1=^J5]6W*8@NZ?<0!KC"M@/O!%" ^ MZ%2OG5<.P='GL/C C>#*;VVW;#0&:5F4^(?7T^!R<#VXB+N[KI[N^R%M;*K]W5$E+J^'R2&^X/M+2Y@YWTIG:_.'& M; B'0ESKJ[*F"NLDY54F<4^+[9OWS:RM;BN0G*;F;:U]$E&^[?KUU1GQV.+. M.KNX=-^LTV;1P9$J/)/N.H^+]M3KIIOT)DGQHY69_/:8"]@Y;@$42>>"7;8- M-)>%%-@][@46!OLN1;5N8'M_::!>X$C V+GPI?B]\E0 0+XW '=Q5RKO-.3J M1 66F\!3X!# "G""3<&N8"6YH-\7K'(@!OL8T$[2]^Z+E])-+G_Q!ELI!>6: MYG]RQYRH7@)O*34IN*K?',G&7Y.7J-\L>ADA0HHS3*A?:+?3^ M"LFWRMS4+"#68 B,\7]/ZUR25I H="VAW%6#4/6 M:,>Y_5/;G^0.G?NU)7[^.(%[C%BYK$!WMPLKA*0%6'5S>49.8[4Q=YO/%>1A M8H5IFEB9[\TWRT9?(Z?]5VV^!-TLFT$WXX::2\4BXB!I6U^DK& NR]B,//4#ZIG;)C*5 N]$J*G[[LN=MKXI'Z2 M(O^K^L;4K,?4(]N+S;*%9'5Z_KS[;#:%41,00\BB $,#_-F5+ V8S#?H>])I M$91:.^<+ \ET08"?6]]D3$%'NARZ[2+6IZ)@UK(+S=+3XVA?O_<_?F&7-V_\G[K/I-<#.+4,\F M@@-K4E/6*B68D1@ C@2C@#>@/+S4G=J5N-L)GLT)?RRQP-,X'BG81MB(_?CJ M[/)I%4=G\#46>4>QBUO,V-AVE5$^IHK6=KG;]=\" $>YOD\A7AYPH$N?]?^! M8X]T#+OOK_U6/QC$4[5!9B.9=YN/7(+V%ZP*=K,.@Q_!R]G.Z3'X'RRYB\M> MY5BF_SW?*)S.$:RCO>UQ@%&,?;MD\%%WS=?#"#9>6L%MJ3+,)W VL.D(4K13P2!CE;:[I+EVM#C[$J!-4VAA2S(&6 M)Y^>YTF#)E*0 5S+0P*SZ/B$G5HQL%_N'QB M@U.Q"MT)XI%PO7H.EALB&$6'PE)5H1/@"4 $" )0 4AJOD\W*S=S=UO'; Y- MAK.\"EG+[^7.!#PYM0(+T\*LID:QY:()N->BO:HV"_L+)KH*HALOP\7>0/X&YKB]3<3(3,-'U MAHVYR=X)L$S7#]IVY472=X7 %('*7^OPL2M519U"Y4X>VM/(Y9;V0\ST>V+V M^T!4WLK[3:% \:I']8[N=JN[OMW)J\#2">@ME9^^&1V\*;W$Z!CU RRAO=/J M,NEYSUT;,6".>:FV2:3>9TD#?1NFW]W0;?,+SJ1V1Z>[.J!-KW571FQ8"^XR M6]6CP3GO[IQW,CD"'A6N:ZBOFE3\,'LVDVHDGB=2=\F[^;3S;DSS['>I;>\N M6T^#FUJ8I[P43LL"+L[)@&_!F#M%G2[88B)+=6 B@FN\1=MLI%:8[-?;5=IJ M%=FTY%L69Z37$EGH!= ]@>^X,CII[RM$4)RU11.3#@6TD#@F+S\*(WGQ:WT+T6:J/#5\A;G7,83M?^&0I3[TRC.*AULB*E0.'T>7;-S>78>XET P96S\;KE!/9R>C'%H3C.[:9887NUT$/2;7:3QFNZ'6!J>ZFA[M7#XY]F>GL+31 MB=FK\5Z>[P&W,5B\[;P>;].U?4<3J6(SDKKGS.$Z-OV<+E+)I@(@>XKOS=3< MBTF/IL?LK2@5Z^SK]Y;\\7@1M8G)%" M#QFX1%\(KLR8@FOFTPXN".4),1V"7,NPH]IL')NB#MZL[=*Z[M8RUD@';N/- M2%JXU-O7+0S71BK#U6&V'G>I/DQZ:-H7]WJO6OO.0'VXP-(@[N.V$/CE\-MX M:8U[NV(HJ1(WRC:S0=[,;'B;TKTMYI64FFHLWMM>4\>86=SD9NLF(SI^3/Q^ M<:_%H0'I)IJ4C"N C4F61)>+4D18K0I0-LS&!7:Z<2=W<%S,KQP7DDM&W.#9 MWG3?+?1P+<2/'M]S$KRZW*FK_O8I>_7ZYH+TO+$DW#41AM?-^ M''&9"5[R,LFS).!\V@E8-^ !QL)1 M"%]_$E?Z[J=1@MS2>P&/TF[&_UQ/+%N09SR*_0NI7;&^ES=_PBIV7(Q;5>Q* MB+N,A.$U[*&1+GO172SP%>]U3N"I;8,63CC6+?55=<=SYTQ>GOLSN*K2)4/" M#9QS+MTS9 '5WM@<-)WV!,8"-UV[GMPS!;K3/0KV=!=^]E?['G<4J6;LO;L9 M=5&3.E8JLONS8W@TY1'"[9:]3MJI;E,7-ZIJ!;(V![5R_RZ M(96RN4+'+J1W^)<-WMA1=E^'?%.M[)QP-3P"QL>J@O_ ?S5'I$!QL#@(EOF4 M%:>(?^)U[2+X/M<(3MM%@#?(D^!=H2/X<+KI>Y F"6W![6!(6C,8\CG,4YR MJ%1^HF"OWS1X#?R_305C@]-SK. GWZ!MW\A&BP7;=V?!]5D,'/<-OLIB8(=L/%@8K@XG!VUQC, ?YQ.IG$YMV&0)I,<)M3^@VN-Y<)&U'OR_E1LZ5>XV%$A?<(VWGQQF_">;82'! F$.,D&XVGCI.PA[ M@A?(JT&&<*_P "?)QUC@7J-8!RDG<#+F M5MV1A+L^G<]218ODHZEA2Q.S-.':98;1D=!6YF8J_R3 -5G%28EN::<*ZAL] M?.9J+L@;@(8 .XRO&Q"F5F_"?N-10N-/LX(,.& ?OQ_4ZQ> M8EVMR3HL';B->:HL:)=V"7(4-L;Y0%!(?-#FY8@QY,OGUU%@= M=GM]XYV[K'/0?)<#VI@^_R004D#SG\;J?=Q?Q07VYO"?JE L8'.51>MH>D9( M:%.-DT;5'U93M6QP_%IJ*-'*J($UP/MXK?SC_$+&7]%T436 <+QNL_R,K1_; MA=2%8U;X,679!F!9)J4)/WF70[8FGXAVOT==]CVP8YFQ%64U7M(T-]"_.Z&]E-8Y^5#QEXC]66W6'?XM8IPA.])[_ZUS,'#WA(#;2C=,%*2 M"IJ$*341XVAW'5MI:K[EYK">_S=8HD/NXOND/1A^/6/,R\(6@ YO50)9U0Y> M_.0YMW%-[M]W 'J84/\U\-/VGWO#7> MB9G,>,>(A^2F3M,?CM7R#>.N&;OS7-[4.UN;5<]Y:+O,?+L?ZFM590QF5A^R M.>@TH %GJ']8F?S1%2$&!O:5,37L\H#OLV:5:Q07+M]U0H8W?1&TL(0X&R%P.6Q5/61"88R8P >-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 15:16:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09733; Tue, 2 Mar 93 15:16:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24112; Tue, 2 Mar 93 15:14:04 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 15:14:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24104; Tue, 2 Mar 93 15:13:57 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA26860 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 14:13:54 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA08480; Tue, 2 Mar 93 14:13:52 CST Date: Tue, 2 Mar 93 14:13:52 CST Message-Id: <9303022013.AA08480@godzilla.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: A proposal for buffer descriptors The following postscript file contains a more detailed account of the buffer descriptors that we discussed at the last meeting. These allow one to send non-contiguous structures of mixed typed data between heterogeneous machines. Bill and Rusty %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Title: bd.dvi %%Pages: 8 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 1 1 df0 D E /Fb 46 123 df<003FE3F801F03F1C03C03E3E07C07C3E0F807C3E0F807C1C0F807C000F80 7C000F807C000F807C000F807C00FFFFFFC0FFFFFFC00F807C000F807C000F807C000F807C000F 807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C000F807C00 7FE1FFC07FE1FFC01F1D809C1C>11 D<78FCFCFCFC7806067D850D>46 D<00600001E0000FE000 FFE000F3E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000 03E00003E00003E00003E00003E00003E00003E00003E00003E0007FFF807FFF80111B7D9A18> 49 D<07F8001FFE00383F80780FC0FC07C0FC07E0FC03E0FC03E07803E00007E00007C00007C0 000F80001F00001E0000380000700000E0000180600300600600600800E01FFFC03FFFC07FFFC0 FFFFC0FFFFC0131B7E9A18>I<03F8001FFE003C1F003C0F807C07C07E07C07C07C03807C0000F 80000F80001E00003C0003F800001E00000F800007C00007C00007E03007E07807E0FC07E0FC07 E0FC07C0780F80781F001FFE0007F800131B7E9A18>I<000180000380000780000F80001F8000 3F80006F8000CF80008F80018F80030F80060F800C0F80180F80300F80600F80C00F80FFFFF8FF FFF8000F80000F80000F80000F80000F80000F8001FFF801FFF8151B7F9A18>I<000380000003 80000007C0000007C0000007C000000FE000000FE000001FF000001BF000001BF0000031F80000 31F8000061FC000060FC0000E0FE0000C07E0000C07E0001803F0001FFFF0003FFFF8003001F80 03001F8006000FC006000FC00E000FE00C0007E0FFC07FFEFFC07FFE1F1C7E9B24>65 DI<001FE02000FFF8E003F80FE007C003E00F8001E01F0000E03E0000E03E0000607E000060 7C000060FC000000FC000000FC000000FC000000FC000000FC000000FC000000FC0000007C0000 607E0000603E0000603E0000C01F0000C00F80018007C0030003F80E0000FFFC00001FE0001B1C 7D9B22>IIII<000FF008007FFE3801FC07F807E001F80F8000781F0000783F000038 3E0000387E0000187C000018FC000000FC000000FC000000FC000000FC000000FC000000FC007F FFFC007FFF7C0001F87E0001F83E0001F83F0001F81F0001F80F8001F807E001F801FC07F8007F FE78000FF818201C7D9B26>I73 D76 DII<003FE00001F07C0003C01E000F800F801F0007C01E0003C03E0003E07E0003 F07C0001F07C0001F0FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC0001F8FC00 01F87C0001F07E0003F07E0003F03E0003E03F0007E01F0007C00F800F8003C01E0001F07C0000 3FE0001D1C7D9B24>II82 D<07F8201FFEE03C07E07801E07000E0F000E0F00060F00060F80000FE0000FFE0007FFE 003FFF003FFF800FFFC007FFE0007FE00003F00001F00000F0C000F0C000F0C000E0E000E0F001 C0FC03C0EFFF0083FC00141C7D9B1B>I<7FFFFFE07FFFFFE0781F81E0701F80E0601F8060E01F 8070C01F8030C01F8030C01F8030C01F8030001F8000001F8000001F8000001F8000001F800000 1F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000001F8000 001F800007FFFE0007FFFE001C1C7E9B21>III89 D<0FF8001C1E003E0F803E07803E07C01C07C00007C0007FC007E7C01F07C03C07C07C07 C0F807C0F807C0F807C0780BC03E13F80FE1F815127F9117>97 DI<03FC000E0E001C1F003C1F00781F00780E00F80000F80000F80000F80000F80000F800 007800007801803C01801C03000E0E0003F80011127E9115>I<000FF0000FF00001F00001F000 01F00001F00001F00001F00001F00001F00001F001F9F00F07F01C03F03C01F07801F07801F0F8 01F0F801F0F801F0F801F0F801F0F801F07801F07801F03C01F01C03F00F0FFE03F9FE171D7E9C 1B>I<01FC000F07001C03803C01C07801C07801E0F801E0F801E0FFFFE0F80000F80000F80000 7800007C00603C00601E00C00F038001FC0013127F9116>I<03F8F00E0F381E0F381C07303C07 803C07803C07803C07801C07001E0F000E0E001BF8001000001800001800001FFF001FFFC00FFF E01FFFF07801F8F00078F00078F000787000707800F01E03C007FF00151B7F9118>103 D<1E003F003F003F003F001E00000000000000000000000000FF00FF001F001F001F001F001F00 1F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E>105 D107 DIII<01FC000F07801C01C03C01E07800F07800F0F800F8F800F8F800F8F800F8F8 00F8F800F87800F07800F03C01E01E03C00F078001FC0015127F9118>II114 D<1FD830786018E018E018F000FF807FE07F F01FF807FC007CC01CC01CE01CE018F830CFC00E127E9113>I<0300030003000300070007000F 000F003FFCFFFC1F001F001F001F001F001F001F001F001F001F0C1F0C1F0C1F0C0F08079803F0 0E1A7F9913>II119 DII<3FFF803C1F00303F00 303E00607C0060FC0060F80001F00003F00007E00007C1800F81801F81801F03803E03007E0700 7C0F00FFFF0011127F9115>I E /Fc 58 126 df<03800007E0000FE0001E70001C70001C7000 1C70001C77E01CE7E01DE7E00FC7000F8E000F0E001E0E003F1C007F1C00739C00E3F800E1F800 E0F1C0E0F1C071F9C07FFFC03F9F801E070013197F9816>38 D<00E001E0038007000E001C001C 0038003800700070007000E000E000E000E000E000E000E000E000E00070007000700038003800 1C001C000E000700038001E000E00B217A9C16>40 DI<01C00001C00001C00001C00071C700F9CF807FFF001F FC0007F00007F0001FFC007FFF00F9CF8071C70001C00001C00001C00001C00011127E9516>I< 387C7E7E3E0E1E1C78F060070B798416>44 D<70F8F8F8700505788416>46 D<03E0000FF8001FFC001E3C00380E00780F00700700700700E00380E00380E00380E00380E003 80E00380E00380E00380F00780700700700700780F003C1E001E3C001FFC000FF80003E0001119 7E9816>48 D<01800380038007800F807F80FF8073800380038003800380038003800380038003 80038003800380038003807FF87FFC7FF80E197C9816>I<07E0001FF8003FFC00783E00E00700 F00780F00380600380000380000380000700000700000E00001C0000380000700000E00001C000 0380000F00001E03803803807FFF80FFFF807FFF8011197E9816>I<07E0001FF8003FFC00781E 00780700300700000700000700000E00003E0007FC0007F00007FC00001E000007000003000003 80000380600380F00380E00700781E003FFC001FF80007E00011197E9816>I<007C0000FC0000 DC0001DC00039C00039C00071C000F1C000E1C001E1C003C1C00381C00781C00F01C00FFFFE0FF FFE0FFFFE0001C00001C00001C00001C00001C0001FFC001FFC001FFC013197F9816>I<07F000 1FFC003FFE007C1F00F00780E00380E00380E003807007007C1F001FFC0007F0001FFC003C1E00 700700F00780E00380E00380E00380F007807007007C1F003FFE001FFC0007F00011197E9816> 56 D<70F8F8F870000000000000000070F8F8F8700512789116>58 D<387C7C7C380000000000 00000038787C7C3C1C1C3870E0400618799116>I<7FFF00FFFF80FFFF80000000000000000000 000000000000FFFF80FFFF807FFF00110B7E9116>61 D<7FF800FFFE007FFF001C0F001C07801C 03801C03801C03801C07801C07001FFF001FFE001FFE001C1F001C03801C03C01C01C01C01C01C 01C01C01C01C03C01C07807FFF80FFFF007FFC0012197F9816>66 D<7FF800FFFE007FFF001C0F 001C07801C03C01C01C01C01C01C01E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00 E01C01C01C01C01C03C01C07801C0F807FFF00FFFE007FF8001319809816>68 D<7FFFC0FFFFC07FFFC01C01C01C01C01C01C01C01C01C00001C00001C1C001C1C001FFC001FFC 001FFC001C1C001C1C001C00001C00E01C00E01C00E01C00E01C00E07FFFE0FFFFE07FFFE01319 7F9816>I<7F1FC0FFBFE07F1FC01C07001C07001C07001C07001C07001C07001C07001FFF001F FF001FFF001C07001C07001C07001C07001C07001C07001C07001C07001C07007F1FC0FFBFE07F 1FC013197F9816>72 DI76 DI<7E1FC0FF3FE07F1FC01D07001D87001D87001D87001DC7001DC7001CC7001CC7001C E7001CE7001CE7001C67001C67001C77001C77001C37001C37001C37001C17007F1F00FF9F007F 0F0013197F9816>I<1FFC003FFE007FFF00780F00F00780E00380E00380E00380E00380E00380 E00380E00380E00380E00380E00380E00380E00380E00380E00380F00780F00780780F007FFF00 3FFE001FFC0011197E9816>I<7FF800FFFE007FFF001C0F801C03801C03C01C01C01C01C01C01 C01C03C01C03801C0F801FFF001FFE001FF8001C00001C00001C00001C00001C00001C00001C00 007F0000FF80007F000012197F9816>I<7FE000FFF8007FFC001C1E001C0F001C07001C07001C 07001C07001C0F001C1E001FFC001FF8001FFC001C1C001C0E001C0E001C0E001C0E001C0E201C 0E701C0E707F07E0FF87E07F03C014197F9816>82 D<07E3001FFF003FFF00781F00F00700E007 00E00700E00000F000007800003F80001FF00007FC0000FE00000F000007000003800003806003 80E00380E00700F80F00FFFE00FFFC00C7F00011197E9816>I<7FFFE0FFFFE0FFFFE0E0E0E0E0 E0E0E0E0E0E0E0E000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000 E00000E00000E00000E00000E00007FC000FFE0007FC0013197F9816>I86 D91 D93 D95 D<1FE0003FF0007FF800783C00300E00000E00000E00 03FE001FFE003E0E00700E00E00E00E00E00E00E00783E007FFFE03FE7E00F83E013127E9116> 97 D<7E0000FE00007E00000E00000E00000E00000E00000E3E000EFF000FFF800F83C00F00E0 0E00E00E00700E00700E00700E00700E00700E00700E00E00F01E00F83C00FFF800EFF00063C00 1419809816>I<03F80FFC1FFE3C1E780C7000E000E000E000E000E000F000700778073E0E1FFC 0FF803F010127D9116>I<003F00007F00003F0000070000070000070000070003C7000FF7001F FF003C1F00780F00700700E00700E00700E00700E00700E00700E00700700F00700F003C1F001F FFE00FE7F007C7E014197F9816>I<03E00FF81FFC3C1E780E7007E007FFFFFFFFFFFFE000E000 700778073C0F1FFE0FFC03F010127D9116>I<001F00007F8000FF8001E78001C30001C00001C0 007FFF00FFFF00FFFF0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0 0001C00001C0003FFE007FFF003FFE0011197F9816>I<03E3C007F7E00FFFE01C1CC0380E0038 0E00380E00380E00380E001C1C000FF8001FF0001BE0003800001800001FFC001FFF003FFF8078 03C0E000E0E000E0E000E0E000E07001C07C07C03FFF800FFE0003F800131C7F9116>I<7E0000 FE00007E00000E00000E00000E00000E00000E3C000EFE000FFF000F87800F03800E03800E0380 0E03800E03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01519809816> I<018003C003C0018000000000000000007FC07FC07FC001C001C001C001C001C001C001C001C0 01C001C001C001C07FFFFFFF7FFF101A7D9916>I<7E0000FE00007E00000E00000E00000E0000 0E00000E7FE00E7FE00E7FE00E0F000E1E000E3C000E78000EF0000FF0000FF8000FBC000F1E00 0E0E000E07000E07807F87F0FFCFF07F87F01419809816>107 DII<7E3C00FEFE007FFF000F87800F03800E03800E03800E03800E 03800E03800E03800E03800E03800E03800E03807FC7F0FFE7F87FC7F01512809116>I<03E000 0FF8001FFC003C1E00780F00700700E00380E00380E00380E00380E00380F00780700700780F00 3C1E001FFC000FF80003E00011127E9116>I<7E3E00FEFF007FFF800F83C00F00E00E00E00E00 700E00700E00700E00700E00700E00700E00E00F01E00F83C00FFF800EFF000E3C000E00000E00 000E00000E00000E00000E00007FC000FFE0007FC000141B809116>I114 D<0FEC3FFC7FFCF03CE01CE01C70007F801FF007F8003C600EE0 0EF00EF81EFFFCFFF8C7E00F127D9116>I<0300000700000700000700000700007FFF00FFFF00 FFFF00070000070000070000070000070000070000070000070100070380070380070380078700 03FE0001FC0000F80011177F9616>I<7E1F80FE3F807E1F800E03800E03800E03800E03800E03 800E03800E03800E03800E03800E03800E03800E0F800FFFF007FBF803E3F01512809116>I<7F 1FC0FF1FE07F1FC01C07001E0F000E0E000E0E000E0E00071C00071C00071C00071C0003B80003 B80003B80001F00001F00000E00013127F9116>II<7F1FC07F3FC07F1FC00F1C00073C0003B80003F00001F00000E00001E00001F00003B8 00073C00071C000E0E007F1FC0FF3FE07F1FC013127F9116>I<7F1FC0FF9FE07F1FC01C07000E 07000E0E000E0E00070E00071C00071C00039C00039C0003980001B80001B80000F00000F00000 F00000E00000E00000E00001C00079C0007BC0007F80003F00003C0000131B7F9116>I<3FFFC0 7FFFC07FFFC0700780700F00701E00003C0000780001F00003E0000780000F00001E01C03C01C0 7801C0FFFFC0FFFFC0FFFFC012127F9116>I<001F80007F8000FF8001E00001C00001C00001C0 0001C00001C00001C00001C00001C00001C00003C0007F8000FF0000FF00007F800003C00001C0 0001C00001C00001C00001C00001C00001C00001C00001C00001E00000FF80007F80001F801120 7E9C16>I<7C0000FF0000FF800003C00001C00001C00001C00001C00001C00001C00001C00001 C00001C00001E00000FF00007F80007F8000FF0001E00001C00001C00001C00001C00001C00001 C00001C00001C00001C00003C000FF8000FF00007C000011207E9C16>125 D E /Fd 59 123 df<007E1F0001C1B1800303E3C00703C3C00E03C1800E01C0000E01C0000E01 C0000E01C0000E01C0000E01C000FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E 01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C000 0E01C0007F87FC001A1D809C18>11 D<007E0001C1800301800703C00E03C00E01800E00000E00 000E00000E00000E0000FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01 C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07F87F8151D809C17>I<003F07E00001C0 9C18000380F018000701F03C000E01E03C000E00E018000E00E000000E00E000000E00E000000E 00E000000E00E00000FFFFFFFC000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C00 0E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C 000E00E01C000E00E01C000E00E01C007FC7FCFF80211D809C23>14 D<6060F0F0F8F868680808 08080808101010102020404080800D0C7F9C15>34 D<004000800100020006000C000C00180018 00300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000E0006000 60006000700030003000180018000C000C00060002000100008000400A2A7D9E10>40 D<800040002000100018000C000C000600060003000300038001800180018001C001C001C001C0 01C001C001C001C001C001C001C001C0018001800180038003000300060006000C000C00180010 002000400080000A2A7E9E10>I<01800180018001804182F18F399C0FF003C003C00FF0399CF1 8F4182018001800180018010127E9E15>I<60F0F0701010101020204080040C7C830C>44 DI<60F0F06004047C830C>I<00010003000600060006000C000C000C00 18001800180030003000300060006000C000C000C0018001800180030003000300060006000C00 0C000C00180018001800300030003000600060006000C000C00010297E9E15>I<03C00C301818 300C300C700E60066006E007E007E007E007E007E007E007E007E007E007E007E007E007600660 06700E300C300C18180C3007E0101D7E9B15>I<030007003F00C7000700070007000700070007 0007000700070007000700070007000700070007000700070007000700070007000F80FFF80D1C 7C9B15>I<07C01830201C400C400EF00FF80FF807F8077007000F000E000E001C001C00380070 006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B15>I<60F0F060000000000000 0000000060F0F06004127C910C>58 D<60F0F0600000000000000000000060F0F0701010101020 204080041A7C910C>I<000600000006000000060000000F0000000F0000000F00000017800000 178000001780000023C0000023C0000023C0000041E0000041E0000041E0000080F0000080F000 0180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E0004001E000C001F 001E001F00FF80FFF01C1D7F9C1F>65 DI<001F808000E0618001 801980070007800E0003801C0003801C00018038000180780000807800008070000080F0000000 F0000000F0000000F0000000F0000000F0000000F0000000F00000007000008078000080780000 80380000801C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E> II70 D72 DI77 DI80 D<07E0801C1980300580700380600180E00180E00080E0 0080E00080F00000F800007C00007FC0003FF8001FFE0007FF0000FF80000F800007C00003C000 01C08001C08001C08001C0C00180C00180E00300D00200CC0C0083F800121E7E9C17>83 D<7FFFFFC0700F01C0600F00C0400F0040400F0040C00F0020800F0020800F0020800F0020000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000000F0000000F0000000F0000000F0000001F800003FFFC001B1C7F9B 1E>IIII91 D<08081010202040404040808080808080B0B0 F8F8787830300D0C7A9C15>II<1FC000307000783800781C00301C00001C 00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F87001212 7E9115>97 DI<07E00C301878307870306000E000E000E0 00E000E000E00060007004300418080C3007C00E127E9112>I<003F0000070000070000070000 070000070000070000070000070000070000070003E7000C1700180F00300700700700600700E0 0700E00700E00700E00700E00700E00700600700700700300700180F000C370007C7E0131D7E9C 17>I<03E00C301818300C700E6006E006FFFEE000E000E000E00060007002300218040C1803E0 0F127F9112>I<00F8018C071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C0 0C38C01C3880181800381C00381C00381C00381C001818001C38000C300013C000100000300000 1800001FF8001FFF001FFF803003806001C0C000C0C000C0C000C06001803003001C0E0007F800 121C7F9215>II<18003C003C0018000000000000000000 000000000000FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C 00FF80091D7F9C0C>I107 DIII<03F0000E1C00180600300300700380600180E001C0E0 01C0E001C0E001C0E001C0E001C06001807003803003001806000E1C0003F00012127F9115>I< FC7C001D86001E03001C01801C01C01C00C01C00E01C00E01C00E01C00E01C00E01C00E01C01C0 1C01C01C01801E03001D06001CF8001C00001C00001C00001C00001C00001C00001C0000FF8000 131A7F9117>I<03C1000C3300180B00300F00700700700700E00700E00700E00700E00700E007 00E00700600700700700300F00180F000C370007C7000007000007000007000007000007000007 00000700003FE0131A7E9116>II<1F9030704030C010C010E010F8007F803FE00FF000 F880388018C018C018E010D0608FC00D127F9110>I<04000400040004000C000C001C003C00FF E01C001C001C001C001C001C001C001C001C001C101C101C101C101C100C100E2003C00C1A7F99 10>IIII<7F8FF00F03800F030007020003840001C80001D80000F000007000007800 00F800009C00010E00020E000607000403801E07C0FF0FF81512809116>II<7F FC70386038407040F040E041C003C0038007000F040E041C043C0C380870087038FFF80E127F91 12>I E /Fe 14 118 df<000FF83F00007FFDFFC001F81FE3E003E03F87E007C03F87E00F803F 07E00F803F03C00F801F00000F801F00000F801F00000F801F00000F801F00000F801F0000FFFF FFFC00FFFFFFFC000F801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F 801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F801F00000F801F0000 0F801F00000F801F00000F801F00000F801F00007FF0FFF0007FF0FFF00023237FA221>11 D<387CFEFEFE7C3807077C8610>46 D<00180000780001F800FFF800FFF80001F80001F80001F8 0001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F8 0001F80001F80001F80001F80001F80001F80001F80001F80001F8007FFFE07FFFE013207C9F1C >49 D66 D<00FF8007FFE00F83F01F03F03E03 F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00007C00007E00007E00003E00 301F00600FC0E007FF8000FE0014167E9519>99 D<0001FE000001FE0000003E0000003E000000 3E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0001FC3E0007 FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC003E00FC003E00FC003E00 FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E001E00FE000F83BE0007FF3F C001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC00F8FC00F8 FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F00300FC07003FFC000FF00 15167E951A>I<1C003E007F007F007F003E001C000000000000000000000000000000FF00FF00 1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FF E00B247EA310>105 D<00FE0007FFC00F83E01E00F03E00F87C007C7C007C7C007CFC007EFC00 7EFC007EFC007EFC007EFC007EFC007E7C007C7C007C3E00F81F01F00F83E007FFC000FE001716 7E951C>111 DI114 D<0FF3003FFF00781F00600700E00300E003 00F00300FC00007FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F007 00FC0E00EFFC00C7F00011167E9516>I<01800001800001800001800003800003800007800007 80000F80003F8000FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F 80000F80000F80000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F 16>II E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 210 105 a Fe(1.)18 b(Bu\013er)g(descriptors)210 216 y Fd(The)h (message-passing)f(routines)h(describ)q(ed)h(ab)q(o)o(v)o(e)e(are)h (su\016cien)o(t)g(to)f(write)h(relativ)o(ely)e(e\016cien)o(t)i(pro-)210 286 y(grams.)25 b(In)17 b(man)o(y)e(applications,)h(ho)o(w)o(ev)o(er,)h(the)h (data)e(to)h(b)q(e)g(sen)o(t)h(or)e(receiv)o(ed)i(is)f(not)g(in)f(consequtiv) o(e)210 355 y(lo)q(cations)11 b(in)g(memory)m(.)k(While)c(an)o(y)g(data,)g (ho)o(w)o(ev)o(er)h(laid)f(out)h(in)f(memory)m(,)e(ma)o(y)g(b)q(e)k(sen)o(t)f (or)g(receiv)o(ed)h(this)210 425 y(w)o(a)o(y)m(,)h(there)j(are)e(a)g(n)o(um)o (b)q(er)g(of)f(sp)q(ecial)i(cases)h(for)d(whic)o(h)i(sp)q(ecial)f(arrangemen) o(ts)g(ma)o(y)e(b)q(e)j(made)e(b)o(y)h(the)210 495 y(op)q(erating)e(system)g (or)f(the)i(hardw)o(are.)k(These)c(include)f(constan)o(t,)g(non-unit)g (strides)h(\(useful)f(for)g(sending)210 565 y(a)k(ro)o(w)g(from)e(a)i(F)m (ortran)g(arra)o(y\),)g(scatter/gathers)i(\(useful)f(for)e(sparse)j(v)o (ectors\),)g(and)d(non-con)o(tiguous)210 634 y(blo)q(c)o(ks)j(\(useful)g(for) g(sending)g(structures)j(with)c(p)q(oin)o(ters\).)34 b(These)20 b(op)q(erations)g(ma)o(y)d(b)q(e)i(pro)o(vided)g(b)o(y)210 704 y(the)c(lo)o(w-lev)o(el)e(hardw)o(are)i(and)g(soft)o(w)o(are)g(b)q (ecause)h(they)f(are)g(common)d(and)i(b)q(ecause)j(imp)q(ortan)o(t)c(p)q (erfor-)210 774 y(mance)d(optimizations)e(can)j(b)q(e)g(made)e(for)h(them,)g (particularly)g(the)h(elimination)d(of)i(unnecessary)i(memory)210 844 y(motion.)22 b(F)m(or)16 b(Unix)g(programmers,)e(the)j(non-con)o(tiguous) e(blo)q(c)o(k)h(case)h(is)f(simply)e(the)j(analogue)e(of)g(the)210 913 y(routines)d Fc(readv)e Fd(and)g Fc(writev)p Fd(.)16 b(Sev)o(eral)c (message-passing)e(implemen)o(tatio)o(ns)f(already)i(pro)o(vide)f(non-unit) 210 983 y(stride)h(v)o(ector)h(op)q(erations,)f(and)f(researc)o(h)j(w)o(ork)d (with)g(scatter/gather)j(op)q(erations)d(has)h(sho)o(wn)g(that)g(these)210 1053 y(are)j(also)g(of)f(great)h(p)q(oten)o(tial)f(b)q(ene\014t.)272 1123 y(MPI)e(pro)o(vides)g(access)i(to)d(all)g(of)g(these)i(in)e(an)o(y)g (com)o(bination)e(b)o(y)j(w)o(a)o(y)f(of)g(a)g(general)h(data-arrangemen)o(t) 210 1192 y(descriptor.)210 1319 y Fb(1.1.)16 b(Organization)210 1415 y Fd(A)i(data-arrangemen)o(t)e(is)i(de\014ned)h(in)e(2)g(steps:)27 b(\(1\))18 b(create)h(a)f(data-arrangemen)o(t)e(descriptor)j(and)f(\(2\))210 1485 y(add)13 b(descriptions)i(of)e(the)h(data)f(arrangemen)o(t)g(to)g(it.)k (A)d(data-arrangemen)o(t)e(descriptor)j(is)e(created)i(with)210 1555 y Fc(MPI)p 279 1555 14 2 v 15 w(NewBD)p Fd(.)e(The)i(routines)g Fc(MPI)p 741 1555 V 15 w(BDVec)p Fd(,)e Fc(MPI)p 957 1555 V 15 w(BDBlk)p Fd(,)h(and)g Fc(MPI)p 1255 1555 V 15 w(BDIdx)f Fd(add)i(descriptions)g(for)f(non-unit)210 1624 y(stride)e(v)o(ectors,)h(blo) q(c)o(ks,)e(and)h(indexed)g(v)o(ectors)g(\(these)i(are)d(describ)q(ed)j(in)d (more)f(detail)h(on)h(the)g(man)e(pages)210 1694 y(for)17 b(these)i (routines\).)29 b(A)17 b(send)h(or)f(receiv)o(e)i(op)q(eration)e(then)h(tak)o (es)g(the)f(data-arrangemen)o(t)g(descriptor)210 1764 y(instead)d(of)f(the)i (\(bu\013er,size\))g(that)f(the)h(send)g(and)e(receiv)o(e)j(routines)e(use)h (for)e(con)o(tiguous)h(data.)210 1890 y Fb(1.2.)i(V)l(ectors)210 1987 y Fd(The)d(\\v)o(ector")g(data-arrangemen)o(t)f(allo)o(ws)f(sending)i (and)g(receiving)g(of)f(non-con)o(tiguous)g(data)g(where)i(the)210 2057 y(elemen)o(ts)g(are)g(separated)h(b)o(y)f(a)f(constan)o(t)i(distance)g (or)e(stride.)272 2126 y(T)m(o)g(add)h(a)g(v)o(ector)g(to)g(the)h(data)e (arrangemen)o(t,)g(use)210 2222 y Fc(MPI_BDVec\()20 b(da,)h(buffer,)f (elmlen,)h(stride,)f(elmcount,)g(datatype)g(\))210 2318 y Fd(The)14 b(parameters)g(are:)210 2415 y Fb(bu\013er)251 b Fd(p)q(oin)o(ter)14 b(to)g(bu\013er)210 2514 y Fb(elmlen)236 b Fd(size)15 b(of)e(eac)o(h)h (elemen)o(t,)f(in)h(b)o(ytes)210 2614 y Fb(stride)254 b Fd(distance)15 b(b)q(et)o(w)o(een)g(successiv)o(e)h(elemen)o(ts,)d(in)h(b)o(ytes)p eop %%Page: 2 2 bop 210 105 a Fb(elmcoun)o(t)182 b Fd(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts) 210 208 y Fb(datat)o(yp)q(e)190 b Fd(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 932 208 14 2 v 15 w(INT)p Fd(\))272 311 y(F)m(or)h(example,)e(if)h (a)h(F)m(ortran)f(arra)o(y)h(is)g(declared)g(as)210 414 y Fc(double)21 b(precision)e(a\(10,20\))210 517 y Fd(and)14 b(the)g(ro)o(w)g Fc(a\(3,1:20\))e Fd(is)i(to)f(b)q(e)i(sen)o(t,)f(use)210 620 y Fc(bd)21 b(=)h(MPI_BDNew\(1\))210 690 y(MPI_BDVec\()e(bd,)h(a\(3,1\),)f(8,) i(8*10,20)e(\))210 759 y(MPI_bsend\()g(to,)h(tag,)g(bd,)g(context)f(\))210 862 y Fd(Here)15 b(w)o(e)f(assume)g(that)g(a)f(double)h(precision)g(quan)o (tit)o(y)f(tak)o(es)i(eigh)o(t)e(b)o(ytes.)210 990 y Fb(1.3.)j(Blo)q(c)o(ks) 210 1087 y Fd(Blo)q(c)o(ks)d(of)f(con)o(tiguous)g(data)g(ma)o(y)f(b)q(e)i (added)g(to)f(the)h(data)f(arrangemen)o(t)g(with)g Fc(MPI)p 1564 1087 V 15 w(BDBlk)p Fd(.)17 b(The)c(format)210 1156 y(is)210 1259 y Fc(MPI_BDBlk\()20 b(bd,)h(buffer,)f(len,)h(datatype)f(\))272 1362 y Fd(Multiple)12 b(uses)i(of)e Fc(MPI)p 640 1362 V 15 w(BDBlk)g Fd(ma)o(y)e(b)q(e)j(used)h(to)e(send)i(complicated)d(data)h (structures.)21 b(F)m(or)12 b(example,)210 1432 y(to)i(send)h(the)f (structure)210 1535 y Fc(struct)21 b({)297 1605 y(int)g(n[3];)297 1674 y(double)g(a[4];)297 1744 y(char)65 b(s;)297 1814 y(})22 b(S;)210 1917 y Fd(w)o(e)14 b(can)g(use)h(the)f(follo)o(wing)e(co)q(de:)210 2020 y Fc(bd)21 b(=)h(MPI_BDNew\()e(3)h(\);)210 2090 y(MPI_BDBlk\()f(bd,)h (&S.n[0],)f(3)i(*)f(sizeof\(int\),)85 b(MPI_INT)20 b(\);)210 2159 y(MPI_BDBlk\()g(bd,)h(&S.a,)86 b(4)22 b(*)f(sizeof\(double\),)e(MPI_DBL) h(\);)210 2229 y(MPI_BDBlk\()g(bd,)h(&S.s,)86 b(sizeof\(char\),)150 b(MPI_OTHER)20 b(\);)210 2299 y(MPI_bsend\()g(to,)h(tag,)g(bd,)g(context)f (\);)210 2402 y Fd(T)m(o)13 b(receiv)o(e)i(the)g(structure,)g(the)g(exact)f (same)f(co)q(de)i(is)f(used,)g(with)g(the)g Fc(MPI)p 1432 2402 V 15 w(bsend)f Fd(replaced)i(with)210 2505 y Fc(MPI_brecv\()20 b(source,)g(tag,)h(bd,)g(context,)f(recv_handle)g(\);)p eop %%Page: 3 3 bop 210 105 a Fb(1.4.)16 b(Indexed)210 202 y Fd(Another)e(t)o(yp)q(e)g(of)f (non-con)o(tiguous)g(comm)o(unicatio)o(n)e(pattern)j(that)f(is)h(common)c(in) j(applications)g(is)g(\\in-)210 271 y(dexed.")18 b(F)m(or)11 b(example,)f(if)g(elemen)o(ts)h(of)f(one)i(v)o(ector,)g(indexed)g(b)o(y)e (another,)i(are)g(used.)18 b(These)12 b(are)g(de\014ned)210 341 y(with)i Fc(MPI)p 374 341 14 2 v 15 w(BDIdx)p Fd(.)j(The)d(format)e(is) 210 444 y Fc(MPI_BDIdx\()20 b(bd,)h(buffer,)f(idxarray,)g(numitems,)g (datatype)g(\))p eop %%Page: 4 4 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(NewBD)117 b Fd(Create)15 b(a)f(new)g(bu\013er)h(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(MPI)p 397 314 13 2 v 15 w(BD)f(*MPI)p 589 314 V 15 w(NewBD\()h(n)o(umitems)c(\))314 384 y(in)o(t)i(n)o(umitems;)210 454 y Fb(INPUT)j(AR)o(GUMENTS)314 524 y Fd(n)o(umitesm)215 b(Maxim)o(um)10 b(n)o(um)o(b)q(er)j(of)h(bu\013er)h(descriptions)g(for)e(the) i(bu\013er)f(descriptor.)210 593 y Fb(DESCRIPTION)314 663 y(MPI)p 413 663 15 2 v 17 w(NewBD)h Fd(creates)h(a)e(new)g(bu\013er)i(descriptor.)k (The)15 b(v)n(alue)f(of)f Fc(numitems)g Fd(is)h(the)h(maxim)n(um)314 733 y(n)o(um)o(b)q(er)c(of)g(bu\013er)h(descriptions)h(that)e(can)h(b)q(e)g (added)g(with)g(the)g(routines)g Fc(MPI)p 1574 733 14 2 v 15 w(BDVec)p Fd(,)e Fc(MPI)p 1787 733 V 16 w(BDBlk)p Fd(,)314 802 y(and)j Fc(MPI)p 463 802 V 16 w(BDIdx)p Fd(.)210 872 y Fb(RETURN)j(V)-5 b(ALUE)314 942 y(MPI)p 413 942 15 2 v 17 w(NewBD)14 b Fd(returns)i(a)d(p)q(oin)o(ter)h(to)g(a)g(bu\013er)g(descriptor,)h(or)f(n)o (ull)f(if)g(an)h(error)g(o)q(ccurs.)p eop %%Page: 5 5 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BD)o(V)l(ec)136 b Fd(Add)14 b(a)g(stride)g(v)o(ector)h(to)f(a)f(bu\013er)i(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BD)o(V)m(ec\()i(b)q(d,)f(bu\013er,)g(elmlen,)e(stride,)i(elmcoun)o(t,) e(datat)o(yp)q(e)j(\);)314 384 y(MPI)p 397 384 V 15 w(BD)f(b)q(d;)314 454 y(c)o(har)g(*bu\013er;)314 524 y(in)o(t)f(elmlen,)f(stride,)i(elmcoun)o (t,)f(datat)o(yp)q(e;)210 593 y Fb(INPUT)j(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(elmlen)275 b(size)15 b(of)e(eac)o(h)h(elemen)o(t,)f(in)h(b)o(ytes)314 872 y(stride)293 b(distance)15 b(b)q(et)o(w)o(een)g(successiv)o(e)h(elemen)o(ts,)d(in)h(b)o (ytes)314 942 y(elmcoun)o(t)228 b(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)314 1012 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 1012 14 2 v 15 w(INT)p Fd(\))210 1081 y Fb(DESCRIPTION)314 1151 y(MPI)p 413 1151 15 2 v 17 w(BD)o(V)l(ec)d Fd(adds)g(a)g(description)h (of)f(a)g(v)o(ector)h(\(with)f(non-unit-stride\))h(to)f(a)g(previously)g (created)314 1221 y(bu\013er)15 b(descriptor.)210 1291 y Fb(RETURN)h(V)-5 b(ALUE)314 1360 y(MPI)p 413 1360 V 17 w(BD)o(V)l(ec)13 b Fd(returns)j(0,)d (or)h Fa(\000)p Fd(1)g(if)f(an)g(error)i(o)q(ccurs.)p eop %%Page: 6 6 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BDBlk)137 b Fd(Add)14 b(a)g(con)o(tiguous)f(blo)q(c)o(k)h(to)g(a)f(bu\013er)i (descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BDBlk\()h(b)q(d,)g(bu\013er,)g(len,)g(datat)o(yp)q(e)g (\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)314 454 y(c)o(har)g (*bu\013er;)314 524 y(in)o(t)f(len,)h(datat)o(yp)q(e;)210 593 y Fb(INPUT)i(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(len)341 b(n)o(um)o(b)q(er)13 b(of)h(b)o(ytes)314 872 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 872 14 2 v 15 w(INT)p Fd(\))210 942 y Fb(DESCRIPTION)314 1012 y(MPI)p 413 1012 15 2 v 17 w(BDBlk)18 b Fd(adds)h(a)f(description)i(of)e (a)h(con)o(tiguous)f(blo)q(c)o(k)h(to)f(a)h(previously)g(created)h(bu\013er) 314 1081 y(descriptor.)210 1151 y Fb(RETURN)c(V)-5 b(ALUE)314 1221 y(MPI)p 413 1221 V 17 w(BDBlk)13 b Fd(returns)i(0,)e(or)h Fa(\000)p Fd(1)g(if)f(an)h(error)h(o)q(ccurs.)p eop %%Page: 7 7 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(BDIdx)140 b Fd(Add)14 b(a)g(scatter/gather)h(to)f(a)g(bu\013er)h(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(BDIdx\()h(b)q(d,)g(bu\013er,)h(idxarra)o(y)m(,)d(n)o(umitems,)f(datat) o(yp)q(e)j(\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)314 454 y(c)o(har)g(*bu\013er;)314 524 y(in)o(t)f(idxarra)o(y[],)f(n)o(umitems,)f (datat)o(yp)q(e;)210 593 y Fb(INPUT)16 b(AR)o(GUMENTS)314 663 y Fd(b)q(d)347 b(Data)13 b(bu\013er)314 733 y(bu\013er)290 b(p)q(oin)o(ter)14 b(to)g(bu\013er)314 802 y(idxarra)o(y)242 b(p)q(oin)o(ter)14 b(to)g(index)g(arra)o(y)314 872 y(n)o(umitems)215 b(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)314 942 y(datat)o(yp)q(e)234 b(datat)o(yp)q(e)14 b(\(e.g.,)f Fc(MPI)p 1055 942 14 2 v 15 w(INT)p Fd(\))210 1012 y Fb(DESCRIPTION)314 1081 y(MPI)p 413 1081 15 2 v 17 w(BDIdx)d Fd(adds)g(a)h(description)g(of)f(a)g(scatter)i(or)e (gather)h(blo)q(c)o(k)f(to)h(a)f(previously)g(created)i(bu\013er)314 1151 y(descriptor.)19 b(\(It)14 b(is)g(a)g(gather)g(when)g(used)h(in)e(a)h (send)h(and)f(a)f(scatter)j(when)e(used)h(in)e(a)h(receiv)o(e.\))210 1221 y Fb(RETURN)i(V)-5 b(ALUE)314 1291 y(MPI)p 413 1291 V 17 w(BDBlk)13 b Fd(returns)i(0,)e(or)h Fa(\000)p Fd(1)g(if)f(an)h(error)h(o)q (ccurs.)p eop %%Page: 8 8 bop 210 105 a Fb(NAME)314 175 y(MPI)p 413 175 15 2 v 17 w(F)l(reeBD)120 b Fd(F)m(ree)15 b(a)e(bu\013er)i(descriptor)210 245 y Fb(SYNOPSIS)314 314 y Fd(in)o(t)e(MPI)p 460 314 13 2 v 15 w(F)m(reeBD\()i(b)q(d)f(\))314 384 y(MPI)p 397 384 V 15 w(BD)g(b)q(d;)210 454 y Fb(INPUT)i(AR)o(GUMENTS)314 524 y Fd(b)q(d)347 b(Data)13 b(bu\013er)210 593 y Fb(DESCRIPTION)314 663 y(MPI)p 413 663 15 2 v 17 w(F)l(reeBD)g Fd(frees)i(a)f(bu\013er)g (descriptor)i(created)f(with)f Fc(MPI)p 1354 663 14 2 v 15 w(NewBD)p Fd(.)210 733 y Fb(RETURN)i(V)-5 b(ALUE)314 802 y(MPI)p 413 802 15 2 v 17 w(NewBD)14 b Fd(returns)i(0,)d(or)h Fa(\000)p Fd(1)f(if)g(an)h(error)h(o)q(ccurs.)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:03:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10713; Tue, 2 Mar 93 16:03:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26723; Tue, 2 Mar 93 16:01:55 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:01:54 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26694; Tue, 2 Mar 93 16:01:49 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA28558 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:01:47 -0600 Received: from localhost by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00600; Tue, 2 Mar 93 15:01:45 CST Message-Id: <9303022101.AA00600@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Revised Multi-level MPI Suggestion Date: Tue, 02 Mar 1993 15:01:44 -0600 From: Rusty Lusk Before the last meeting we sent out a suggestion on how the potential complexity of the evolving MPI library could be managed by dividing the routines into layers. The idea was to provide a very simple interface at the highest layer, which we call Level 4, and provide all of the flexibility necessary at lower levels, which we called 1 and 2. Since then the February meeting has taken place, and some of our thinking has changed. The basic idea still seems like a good one to us, so here is a new proposal for multiple levels in MPI. At the February meeting, Marc Snir's detailed point-to-point proposal basically covered what we called level 1 (where there are a small number of routines, including "init" routines for operations) and level 2 (where sets of the level 1 routines are bundled together for efficiency, but almost all functionality is preserved). There also seemed to be general agreement that there was no need to have separate routines for supporting communication among heterogeneous machines; rather, contiguous buffers could be specified by (addr,datatype,numitems) instead of (addr,len). The concept of buffer-descriptor (details in another message) would be used to describe noncontiguous buffers and buffers of mixed data types. Therefore it seems unnecessary to have *two* higher levels, since one of the differences between our level 3 and level 4 was the specification of data types. At the same time, it still seems useful to us to have a level of MPI routines that are simpler than the full set specified in Marc's document. We therefore suggest here a level 3 for MPI that imposes a set of restrictions in the interest of simplicity. (So, for example, it will not use buffer descriptors, but restrict users to contiguous buffers of data of the same type.) It corresponds roughly to "current practice", but adds thread safety (no "last message" semantics). It assumes the existence of only one default "all" group and one "universal" context, so that these need not be part of the calls. It does include nonblocking sends and receives, but simplified versions of wait, status, and probe. Users requiring more flexibility can use the routines of level 2 instead. Level 3 MPI routines: (syntax only specified for definiteness) numids() returns number of processes myrank() returns caller's rank in group "all" bsend(dest,tag,buf,datatype,numitems) blocking send (until buffer is empty) brecv(from,tag,buf,datatype,maxitems, blocking receive, with separate actual_from,actual_tag, output arguments to handle wild-card actual_numitems) input arguments Note that lots of programs have been written with just these four calls. handle = nsend(dest,tag,buf,datatype,numitems) non-blocking send handle = nrecv(from,tag,buf,datatype,maxitems, non-blocking receive actual_from,actual_tag, output arguments to handle wild-card actual_numitems) input arguments status(handle) check status of non-blocking operation wait(handle) wait for a non-blocking operation probe(from,tag,actual_from,actual_tag,actual_length) look in queue for matching message, get length so can allocate buffer This is a larger set than the "four-function" interface we proposed as level 4 earlier, but not much, and has the advantage that one can write correct programs even on machines with no buffering. It corresponds roughly to current practice in that most existing programs can be easily converted to this level. It gains simplicity and ease of use by not offering access to MPI's advanced features like noncontiguous buffers of various and mixed types, explicit handle management, high-performance protocols, and elaborate status and wait calls. It has no explicit mention of groups or contexts. The collective operations seem to be getting complicated too, and would benefit from a restricted, simple level. One way to do this would be to add a (blocking) barrier and collective operations in their simplest forms, without explicit mention of group-id's. These Level 3 operations add nothing to MPI that cannot be done at lower levels. However, it seems to us that adopting these simplified versions into the standard will make it easier for many people to use and hence pave the way for its acceptance. Any comments? Bill Gropp & Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:17:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11504; Tue, 2 Mar 93 16:17:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27414; Tue, 2 Mar 93 16:16:32 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:16:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA27402; Tue, 2 Mar 93 16:16:25 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA29060 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:16:23 -0600 Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00668; Tue, 2 Mar 93 15:16:21 CST Message-Id: <9303022116.AA00668@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: Task identifiers Date: Tue, 02 Mar 93 15:16:20 CST From: Rusty Lusk One issue that has been postponed several times is that of what "process id's" will look like and what they will mean. We would like to offer a suggestion. In this short note we discuss some of the issues and make a concrete proposal. Since the word "process" is so overloaded, we use the word "task" here (possibly only a marginal improvement). We define a task as a program unit that receives a message. For most systems, this is a single process. In a multi-threaded environment, this may be a process or a thread. To allow a process to decide whether to use threads to handle some operations, we believe that a threads within a process should have the same task id; different threads can be identifed by different tags, groups, or contexts. For example, an obvious way to provide a nonblocking user-defined operation is to execute that operation in a separate thread. This should not require a separate task id. The most important issue is whether the number of tasks is fixed or not. If the number is fixed, a sequential numbering of the tasks is possible and simple. However, the trend seems to be toward allowing for the dynamic modificaiton of the number of tasks, and this is impossible if a sequential numbering is used. To avoid restricting MPI to a fixed collection of tasks, we advocate using an opaque task id. This raises the issue of how the task ids of other processes are determined. We suggest that the rank of the task in a specified group be used as a cannonical representation. This rank is required to be in the range 0 to (size of group)-1. The usual arithmetic operations can be performed on the rank. For example, you can define your right neighbor in a ring as ((myrank + 1) mod p), where p is the number of tasks in the ring. Proposal In general, task ids should be an opaque type. A routine gets its own task id with MPI_MyTid. To get the task ids of other tasks, an application may use MPI_TidFromRank(group, rank). A routine gets its own rank with MPI_MyRank(group). It is the task id that is used as a source or destination in the point-to-point message-passing routines. In Fortran it might as well be an integer. Thus the new MPI routines being proposed are: MPI_MyTid() Get caller's task id MPI_MyRank(group) Get caller's rank in group MPI_TidFromRank(group,rank) Get task id of a task from its rank in a group MPI_RankFromTid(tid,group) Get rank in a group from a task id An exception is made for the level 3 routines described in the recent earlier message; these only have one group and we propose that task id and rank in the "all" group be the same for them. This is current common practice. Discussion Note that since a task id is an opaque type, it cannot be sent to another task and used as a task id on that other task. Only the rank is meaningful across tasks. Making the task id an opaque type allows an implementation to optimize the task ids. For example, an implementation may choose to make the task ids the physical node ids used by the hardware. In addition, it encourages users to use library routines to determine neighbors rather than arithmetic on the task id (note that they can still get a neighbor by adding one to their rank). For example, the "best" neighbor in a virtual ring depends on details of the underlying hardware and software; calling a routine such as MPI_NbrInRing to get the neighbor is a more portable way of getting a good neighbor than MPI_RankToTask(MPI_MyRank(group)+1, group ). Bill Gropp and Rusty Lusk From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 16:33:15 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12253; Tue, 2 Mar 93 16:33:15 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28299; Tue, 2 Mar 93 16:32:14 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 16:32:13 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28287; Tue, 2 Mar 93 16:32:04 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA29509 (5.65c/IDA-1.4.4 for ); Tue, 2 Mar 1993 15:32:01 -0600 Received: by donner.mcs.anl.gov (4.1/GCF-5.8) id AA00744; Tue, 2 Mar 93 15:31:59 CST Message-Id: <9303022131.AA00744@donner.mcs.anl.gov> To: mpi-comm@cs.utk.edu Subject: MPI Job startup Date: Tue, 02 Mar 93 15:31:58 CST From: Rusty Lusk We feel that the MPI standard should provide enough information to specify an entire, source-code portable program. To do this, it is necessary to specify how parallel programs start and end. This is the analogue of PROGRAM MAIN ... END in Fortran and main(argc,argv) {...exit(rc); }. This proposal does not deal with methods for spawning tasks during the execution of an MPI program, or for handling explicitly MIMD models (beyond the usual way of starting a Turing machine on all nodes and sending the appropriate tape). This is not meant to suggest that MPI not have such interfaces, just that we are only proposing an SPMD model at this time. Issues It should be possible to write a program that is source-language portable to any system, particularly if it requires only a simple interface to the operating system. An example is an environment that starts up a program with a fixed number of processors. Without something like this, it is not possible to write a portable MPI program. However, no standard should restrict programs to a particular interface. For example, some applications may want to dynamically added and remove tasks during the execution of an application. We leave it to others on the committee to propse a mechanism for this case. Below, we discuss three different approaches; we note that all of these have been tried in some existing system or systems. The seemingly simplest approach is to use an explicit initialization routine, MPI_Init. There are several problems with this approach. The major one is that the parallel part of the program is the code that "follows" the MPI_Init. This is potentially error prone; further, it can be awkward to implement on some systems. This is the way p4 does it. Another approach is to insist that the entire program be run in parallel; a simple way to do this is to replace the "main" program with a routine. In C, this could be MPI_Main( argc, argv ). The linking step then provides a "main" that handles all appropriate startup (including the processing of command line arguments where appropriate for information on the number of tasks etc.) and then calls the user's MPI_main routine. Yet another approach is to provide a routine that runs a user-defined routine in parallel. For example, MPI_Call( argc, argv, routine ) can be used to run routine( argc, argv ) in parallel. A generalization of this would allow the running of a particular routine on a defined group: MPI_Call( groupid, argc, argv, routine ). This is more flexible than the MPI_Main approach but retains the simplicity of running a well-defined program unit (the routine) in parallel. All three of these approaches are currently in use. These are not the most general methods. For example, these provide no mechanism for dynamically creating (remote spawn) tasks on other processors. Also, there is no direct support for MIMD programming. Finally, there is no support for adding an already-running process to a group of running processes. Proposal We propose the MPI_Call form. A slight variation would allow the use of a general argument descriptor instead of argc/argv. This would allow the more convenient passing arguments to the routine. We have used this mechanism (in Bill's Chameleon system) to write programs for nX, the CM-5, the nCube, and networks of workstations using both p4 and PVM, and it has worked well for us. The point is that all source code is portable. The only thing that is different is the shell command by which you start your program. Note that in this case, program termination is handled by having (a) the routine exit and (b) the calling routine exit. Thus an MPI program would look something like this: main(argc,argv) int argc; char **argv; { ret_code = MPI_Call( argc, argv, worker ); exit( irc ); } int worker( argc, argv ) int argc; char **argv; { ... if (error condition) return 1; ... return 0; } In Fortran it would like something like: program main external worker irc = MPI_call(worker) stop end integer function worker ... worker = 0 return end It may be useful to have a more drastic exit. We suggest MPI_exitall(code); the code here is an exit code that may be ignored by the invoking environment (but UNIX-like environments should treat it like exit(code)). In the case of heterogeneous environments, it is necessary to specify the set of workstations to be used. Our only requirement is that this be done, in the C/POSIX case, through command-line arguments (thus maintaining source-code portability). We could specify an extensible, common format for specifying the machines, executables, and resource limits, but, to avoid additional debate, we suggest only that the command line option -mpi be reserved for future extensions. Bill & Rusty From owner-mpi-comm@CS.UTK.EDU Tue Mar 2 18:09:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15496; Tue, 2 Mar 93 18:09:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02859; Tue, 2 Mar 93 18:07:43 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 2 Mar 1993 18:07:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02851; Tue, 2 Mar 93 18:07:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA08222; Tue, 2 Mar 93 23:07:34 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA17367; Tue, 2 Mar 93 16:06:22 MST Date: Tue, 2 Mar 93 16:06:22 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303022306.AA17367@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised Multi-level MPI Suggestion A Suggestion for probe() in the New Rusty and Bill Scheme I'm a bit confused about probe() in Rusty and Bill's latest proposal: ... > probe(from,tag,actual_from,actual_tag,actual_length) > look in queue for matching message, > get length so can allocate buffer > ... > Any comments? > > Bill Gropp & Rusty Lusk > I believe that there was general agreement that probe() should provide the caller with a "handle" to avoid "hidden state" (the thread-safety issue). (The probe() in the current point-to-point proposal does this.) Should the probe() proposed above return "handle"?: handle = probe(from,tag,actual_from,actual_tag,actual_length) If so, we need to add a "recv()" that has "handle" as an argument at Level 3 (basically precv() as I understand it). (This "handle" was also occaisionally referred to as "lock" in the last meeting.) Also, the current proposals all require that a receiving process know the data type(s) in an incoming message before that message is received. This means that datatype can be included in probe() and "numitems" can be used instead of "length". Syntax is then consistent with other routines in the new "Level 3". Here's a counter-proposal (in the spirit of Rusty and Bill's syntax): handle = probe(from, tag, datatype, actual_from, actual_tag, actual_numitems) precv(handle, buf) The use of "handle" guarantees that input parameters to probe() do not need to be repeated in precv(). "maxitems" (as in brecv()) doesn't seem to be necessary since the caller will know how much memory is required before buf is specified. It might be a good idea to add "unlock(handle)" to Level 3. (OR remove probe() from Level 3.) Of course my preference is to dump the whole mess, including buffer descriptor, into an opaque data structure (as in Leslie's and my "message capsule" proposal on Jan 15 in mpi-pt2pt :-). Tom Henderson NOAA Forecast Systems Lab hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 3 16:35:49 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA11369; Wed, 3 Mar 93 16:35:49 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07168; Wed, 3 Mar 93 16:34:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Mar 1993 16:34:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07160; Wed, 3 Mar 93 16:34:22 -0500 Date: Wed, 3 Mar 93 21:34:17 GMT Message-Id: <6940.9303032134@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Task identifiers To: Rusty Lusk In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:16:20 CST Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Question: How do Rusty and Bill write so many proposals? Answer: Surely they never sleep:-) (Oh well, perhaps I won't get featured on the jokes newsgroups after all.) But seriously, fellow committee members, I recall that the contexts subcommittee is about to prepare one or more proposals which will have considerable bearing on this subject, so perhaps we should again defer "task identifiers" until this has happened. One crucial point raised (again), which I do not see evidence of having really been addressed, is whether MPI will provide a reasonable interface for a static process model, or for a dynamic process model. We have to decide what we want to do here, and I think we ought to clearly state this as potential future users whom I have spoken to are asking just this kind of question. What are the arguments for and against a dynamic process model? To get started: a) The MPP programming world has a whole lot more experience with the static model, and as such it is better aligned with common practice. b) We anticipate people moving toward use of a dynamic process model. c) There are a whole load of issues in the dynamic model which complicate implementations and imply lesser efficiency. d) In order for programs to be portable in the MPI + dynamic process model, the interface to dynamic process control has to be uniform as well as the interface to message passing. Would MPI really consider addressing the question of a uniform interface to process control at this point? At least in the static case the process creation is not part of the program and we can "ignore" it. /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 3 17:06:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12272; Wed, 3 Mar 93 17:06:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08663; Wed, 3 Mar 93 17:04:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 3 Mar 1993 17:04:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08641; Wed, 3 Mar 93 17:04:18 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA05267; Wed, 3 Mar 93 16:01:25 CST Date: Wed, 3 Mar 93 16:01:25 CST From: Tony Skjellum Message-Id: <9303032201.AA05267@Aurora.CS.MsState.Edu> To: lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk Subject: Re: Task identifiers Cc: mpi-comm@cs.utk.edu I think that we agreed to defer dynamic issues to a future MPI standard level, but I do agree that this is never guaranteed to happen, as Rusty has said. The lack of control for processes, and the lack of dynamic group syntax/semantics worries me. They will almost certainly appear as non-portable extensions. Perhaps the original N-month limit was too short to permit the best possible standard. - Tony From owner-mpi-comm@CS.UTK.EDU Thu Mar 4 08:24:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27412; Thu, 4 Mar 93 08:24:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22467; Thu, 4 Mar 93 08:22:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 4 Mar 1993 08:22:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gmdzi.gmd.de by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22379; Thu, 4 Mar 93 08:22:07 -0500 Received: from f1neuman.gmd.de (f1neuman) by gmdzi.gmd.de with SMTP id AA02215 (5.65c/IDA-1.4.4 for ); Thu, 4 Mar 1993 14:20:32 +0100 Received: by f1neuman.gmd.de id AA16175; Thu, 4 Mar 1993 14:21:56 GMT Date: Thu, 4 Mar 1993 14:21:56 GMT From: Rolf.Hempel@gmd.de Message-Id: <9303041421.AA16175@f1neuman.gmd.de> To: mpi-comm@cs.utk.edu Subject: Process ids Cc: gmap10@f1neuman.gmd.de recently there has been a discussion on process identifiers, groups, and dynamic process creation. I would like to add a few comments. 1. It is not so much the dynamic creation of new processes which calls for something like opaque process ids, but rather the dynamic removal of processes. In the former case process ids can be added at the end (n, n+1, ...) without changing the 0,..,n-1 numbering of the existing processes. If a process terminates, however, there is a gap in the numbering scheme. On the other hand, the same gap occurs in the group ranking which also Bill and Rusty want to keep. So, I don't see how the opaque process ids resolve the problem. 2. As Bill and Rusty say at the end of their note, inquiries like "give me my right neighbor" should be done by using the process topology functions, and not on the ranks themselves. Only this way an optimization of the process ordering is possible. The ranks mainly serve to define the buffer ordering in collective communication. 3. At our last meeting we already saw that the MPI time schedule is becoming tight. I don't think it would be wise to deal with dynamic process creation in MPI-1, and as far as I remember there was already some consensus on that. Of course, in the design of the static interface we have to keep in mind that dynamic extensions will be added later. It should not be seen as a problem if on the basis of MPI-1 non-standard extensions will emerge. After all, only this kind of experimenting can make a later standardization possible. Rolf Hempel From owner-mpi-comm@CS.UTK.EDU Sat Mar 6 15:38:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03415; Sat, 6 Mar 93 15:38:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12502; Sat, 6 Mar 93 15:37:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sat, 6 Mar 1993 15:37:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12470; Sat, 6 Mar 93 15:36:41 -0500 Date: Sat, 6 Mar 93 20:36:37 GMT Message-Id: <390.9303062036@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: dynamic processes and dynamic groups To: mpi-pt2pt@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Hi again Well, we certainly are having some fun now in this discussion of static vs dynamic processes in MPI. This is as much about dynamic groups as dynamic processes, and the latter part is really about MPI as a standards committee. I have a lot of sympathy with John Flower's concerns. There appears to be reasonable consensus among those contributing to the discussion that in a world process model where processes dynamically appear and dissapear in an asynchronous fashion an enumeration of processes is not a useful way of identifying processes - what sense could we make of the enumeration? An opaque type process (or task) identifier is better. The same considerations surely lead us to the observation that in a world group model where processes dynamically join and leave groups in an asynchronous fashion an enumeration of members of a group is not a useful way of identifying members - what sense could we make of the enumeration? Rolf previously pointed this out on mpi-comm - March 4, subject "Process ids". What are asynchronously created (terminated) processes to do? I guess they either join (leave) an existing group, or create (destroy) a group for their own purposes. The primitives for these two are really process group resizing (or destruction/recreation) and process group creation (termination). I guess their must be heavyweight PVM users who have other experiences of usage of dynamic process creation/termination - Al?. These thoughts/questions, however interesting and discussion provoking they may or may not be, do not take us any further on answering some of the BIG questions about what MPI will be, regarding for example: a) Process model; static v dynamic; ... b) Group model; static v dynamic; ... I've previously stated that I'd vote for static process model in (a). This "seemed to be" the intent of MPI (although I couldn't find it written down and recorded as agreed), would be useful, and could be agreed upon within the MPI time schedule. This is not to say that I think limitation to a static process model would be the most useful thing upon which to agree to describe a 'de facto' standard. I have pretty well the same opinions regarding process groups. I'd like us all agree on how we're going to move forward and get the details of the standard described, as I'm concerned that at the current rate we may not. I'll broadly support John Flower's suggestions, primarily because I don't see any others, and I'll do my best to implement whatever strategy we can agree on. Final thought, perhaps we should move the discussion of MPI strategy to mpi-comm. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 08:01:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04198; Mon, 8 Mar 93 08:01:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19241; Mon, 8 Mar 93 07:59:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 07:59:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from marge.meiko.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19232; Mon, 8 Mar 93 07:59:19 -0500 Received: from hub.meiko.co.uk by marge.meiko.com with SMTP id AA03238 (5.65c/IDA-1.4.4 for ); Mon, 8 Mar 1993 07:59:13 -0500 Received: from float.co.uk (float.meiko.co.uk) by hub.meiko.co.uk (4.1/SMI-4.1) id AA10283; Mon, 8 Mar 93 12:59:09 GMT Date: Mon, 8 Mar 93 12:59:09 GMT From: jim@meiko.co.uk (James Cownie) Message-Id: <9303081259.AA10283@hub.meiko.co.uk> Received: by float.co.uk (5.0/SMI-SVR4) id AA02380; Mon, 8 Mar 93 12:56:19 GMT To: lusk@mcs.anl.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:31:58 CST <9303022131.AA00744@donner.mcs.anl.gov> Subject: MPI Job startup Content-Length: 2990 > We feel that the MPI standard should provide enough information to > specify an entire, source-code portable program. I agree. MPI_INIT style ============== > The seemingly simplest approach is to use an explicit initialization > routine, MPI_Init. There are several problems with this approach. > The major one is that the parallel part of the program is the code > that "follows" the MPI_Init. This is potentially error prone; > further, it can be awkward to implement on some systems. This is > the way p4 does it. I agree that this is simplest. I also prefer it to the others. I don't see the "major problem" outlined above. The whole of the code is parallel even before the MPI_INIT, it's just that it can't communicate yet. I'd expect code like this (given an I/O everywhere model) to execute fine int main(int argc,char ** argv) { printf("Hello world\n"); } and print as many "Hello world" messages as the number of nodes on which I ran it. It also seems to me to be quite an easy way to allow the dynamicness everyone seems to want. The newly created process starts up, and then attaches to the existing MPI environment at the time it calls MPI_INIT. In my view, then, what happens is 1) By some unspecified means a number of processes start to execute 2) They decide to collaborate by executing MPI_INIT [If dynamic process creation is permitted 2a) New processes are forked/execed 2b) The new processes join the existing collaboration by executing MPI_INIT ] 3) They each exit (by calling exit), or you can have a drastic exit if you like. MPI_CALL style ============== I don't think I understand what this is intended to mean, which may be why I don't like it ! Therefore please can you explain it to me. 1) Is the idea that there is one user process which then executes MPI_CALL at which point all of the other processes are created,and magically arrive in a user subroutine, or do all the processes execute main ? 2) If all the processes execute main, what can they do before calling MPI_CALL ? 3) What global state is accessible to the MPI_CALLed routine ? e.g. (your example with minor change...) extern int foo; /* Of course this could be an arbitrarily complex data structure ! */ main(argc,argv) int argc; char **argv; { foo = time(); /* Something which may differ on each node ! */ ret_code = MPI_Call( argc, argv, worker ); exit( irc ); } int worker( argc, argv ) int argc; char **argv; { printf("%d: %d\n", MPI_MYTID(), foo); ... if (error condition) return 1; ... return 0; } At the moment I definitely prefer MPI_INIT ! -- Jim James Cownie Meiko Limited Meiko Inc. 650 Aztec West Reservoir Place Bristol BS12 4SD 1601 Trapelo Road England Waltham MA 02154 Phone : +44 454 616171 +1 617 890 7676 FAX : +44 454 618188 +1 617 890 5042 E-Mail: jim@meiko.co.uk or jim@meiko.com From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 09:19:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05513; Mon, 8 Mar 93 09:19:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22555; Mon, 8 Mar 93 09:18:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 09:18:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22546; Mon, 8 Mar 93 09:18:01 -0500 Date: Mon, 8 Mar 93 14:17:41 GMT Message-Id: <1439.9303081417@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Job startup To: Rusty Lusk , mpi-comm@cs.utk.edu In-Reply-To: Rusty Lusk's message of Tue, 02 Mar 93 15:31:58 CST Reply-To: lyndon@epcc.ed.ac.uk Hi there Spring has arrived in Scotland :-) For the thrid say this year we have beautiful sunshine :-) The snow on the Caringorms is meling fast :-( Now it's MPI job startup. Thanks to Bill and Rusty for starting this one off. The MPI_INIT style is familiar. It makes it explicit that the bit of the program before MPI_INIT is executing in parallel, and no MPI services can be used before MPI_INIT. As Jim correctly points out. It documents to the user what s/he is using. The trouble is, that the user will almost always end up calling MPI_INIT right after entry into main, because what can the user usefully do before MPI_INIT? We should consider providing this approach. I really don't like the MPI_CALL approach. It serves to confuse the user. It looks like there is one main running, and the function reference in MPI_CALL magically runs in parallel. What of use can the user do before MPI_CALL? It looks to the user like the static (global) data is all accessible everywhere. I don't think this is intended, and I don't think this is what should be provided. What about the other approach, in which the user provides a "virtual main" and the MPI library supplies a "actual main", which calls the virtual main. I guess the virtual main calls MPI_INIT and then calls the actual main. This removes the necessity for the user to call MPI_INIT, and means that s/he cannot make the mistake of attempting to use MPI services before they are available. We *might* be able to get this past C programmers. I believe that we would have enormous problems trying to get this past crusty fortran programmers --- "What, you mean I have to write a subroutine instead of a program?". Wait up though, this is what is happening anyway. Your process never starts in your own main. Your "actual main" is a "virtual main", and the compiler/stdlib/runtime executes a preamble to main. FOURTH suggestion ----------------- There is no MPI initialisation procedure. The user program is "attatched to MPI" and can use MPI services as soon as the user program starts to execute. This means that we require system vendors to embed any MPI initialisation his/her implementation requires in the preamble before main. Dead simple. Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 09:25:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05746; Mon, 8 Mar 93 09:25:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22903; Mon, 8 Mar 93 09:24:39 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 09:24:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22895; Mon, 8 Mar 93 09:24:31 -0500 Date: Mon, 8 Mar 93 14:24:19 GMT Message-Id: <1449.9303081424@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Job startup To: lyndon@epcc.ed.ac.uk, Rusty Lusk , mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Mon, 8 Mar 93 14:17:41 GMT Reply-To: lyndon@epcc.ed.ac.uk Ooops - small oversight. > This means that we require system vendors to embed any MPI > initialisation his/her implementation requires in the preamble before > main. > Not quite true, Lyndon, you missed something :-) On the other hand all of the MPI service routines could do one check of a static boolean to see if MPI has been initialised, and if not then call an initialisation routine which inverts the static boolean. I just prefer if it happened for sure in the preamble. Best Wishes Lyndon ps replying to my own mail, hmmm :-) /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 10:51:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09115; Mon, 8 Mar 93 10:51:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28354; Mon, 8 Mar 93 10:50:07 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 10:50:06 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sol.cs.wmich.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28346; Mon, 8 Mar 93 10:50:04 -0500 Received: from id.wmich.edu (id.cs.wmich.edu) by cs.wmich.edu (4.1/SMI-4.1) id AA00863; Mon, 8 Mar 93 10:49:41 EST Date: Mon, 8 Mar 93 10:49:41 EST From: john@cs.wmich.edu (John Kapenga) Message-Id: <9303081549.AA00863@cs.wmich.edu> To: mpi-comm@cs.utk.edu hi; To add to the "thread" on MPI init. I think there should be some form of MPI init call before any other MPI call. This allows MPI implementations over other systems, without a small but built in overhead. I don't feel religous about this though. Assuming an MPI init call of some form is added, the only questions I would have would be - 1) Does each thread need to make the call - or is one call per process OK? and 2) In the case of only requiring on call per process, Is it OK to make more than one call to the init routine - as in the case of each thread in a process making a call before starting. If there is any state information kept on a per thread basis - than one call per thread may be reasonable. I know we tried to keep all state per thread information out (which I view as very reasonable), but unless it is a 100% requirement, one MPI init call per process may not be enough. john From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 15:11:52 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17797; Mon, 8 Mar 93 15:11:52 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13204; Mon, 8 Mar 93 15:08:17 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 15:08:15 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA13156; Mon, 8 Mar 93 15:08:08 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 8 Mar 93 12:04 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA16386; Mon, 8 Mar 93 12:02:56 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA14497; Mon, 8 Mar 93 12:02:53 PST Date: Mon, 8 Mar 93 12:02:53 PST From: d39135@sodium.pnl.gov Subject: RE: MPI Job startup To: lusk@mcs.anl.gov, lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303082002.AA14497@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Here are some comments I haven't seen yet about MPI job startup suggestions. Lyndon Clarke's FOURTH suggestion: > There is no MPI initialisation procedure. The user program is > "attatched to MPI" and can use MPI services as soon as the user program > starts to execute. > > This means that we require system vendors to embed any MPI > initialisation his/her implementation requires in the preamble before > main. This approach has the disadvantage that it requires intimate collaboration with the vendor's runtime language-support library. The interfaces to those libraries are usually undocumented and subject to change without notice. This means that one needs vendor cooperation to write and maintain an MPI implementation using this approach. From the MPI user's standpoint, "FOURTH suggestion" seems indistinguishable from the earlier proposal that there be no explicit initialization call, and that initialization be done by having each MPI routine check a static boolean. A minor problem with the static boolean suggestion is that then the initialization routine does not have access to those parts of the environment passed as (argc,argv). Such access is sometimes useful. (For example... TCGMSG's static process initializer uses command-line arguments to inform each process of its place in the grand scheme of things. In C, the command-line arguments are accessible to TCGMSG because the application has to pass them to the TCGMSG initialization procedure. (In Fortran, TCGMSG's initialization procedure is called without arguments, and it has to hunt up the argc and argv that presumably stored by Fortran initialization in global variables. See above comments about undocumented interfaces.) The MPI_Main approach (MPI does the initialization, user provides a subroutine) seems to have the disadvantage that it works for only one package. If MPI takes this approach, then I don't see how any other package FOO can too, if an application wants to use both MPI and FOO. MPI_Call has the slight problem that if another package FOO adopts the MPI_Call approach, then we end up with main (argc,argv) { MPI_main (argc,argv,myMPIinitializer); } myMPIinitializer (argc,argv) { FOO_main (argc,argv,myFOOinitializer); } myFOOinitializer (argc,argv) { myapplication (argc,argv); } compared to main (argc,argv) { MPI_Init (argc,argv); FOO_Init (argc,argv); myapplication (argc,argv); } Certainly this is not an enormous problem, but I don't see where the extra complexity buys anything. I like MPI_Init (augmented if necessary for dynamic processes). --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 15:52:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA19445; Mon, 8 Mar 93 15:52:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18884; Mon, 8 Mar 93 15:51:31 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 15:51:30 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18876; Mon, 8 Mar 93 15:51:28 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA06355 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Mon, 8 Mar 1993 12:51:24 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA18286; Mon, 8 Mar 93 12:44:12 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA15256; Mon, 8 Mar 93 12:44:15 PST Date: Mon, 8 Mar 93 12:44:15 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303082044.AA15256@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: MPI_INIT To throw in my two cents worth..... The business with argc and argv (and, of course environ) has been a thorn in the side of Express for many years. We've adopted a standard two-faced approach in which C programmers get their "main" routine renamed (silently) to exp_main which we then call from our own main. In this case argc and argv get dealt with by Express and passed to the main routine. This approach has the disease mentioned by Rik, that two or more different software suppliers may want to pull the same trick causing confusion. In fortran we went the way of MPI_INIT with no arguments. This seems to work but has the difficulty that things beyond the ken of the message passing system can be affected by it. For example, the Express program program foo write(6,*) 'I have started' call kxinit -- i.e., MPI_INIT write(6,*) 'Done with kxinit' stop end is actually wrong because the I/O system needs to know parameters that don't get set till the call to kxinit. We could, of course, enforce the rule in MPI that only MPI routines are affected by the MPI_INIT call and that, therefore, you can put anything you like before the call to MPI_INIT as long as its name doesn't start with MPI_!!!!! I dont think this is going to work in the long run since it actively discourages library vendors from building their codes with MPI routines!!! Also, in Rik's example, I think the code has to look sort of like main(argc, argv) int argc; char **argv; { MPI_INIT(&argc, &argv); FOO_INIT(&argc, &argv); /* etc... */ if we want to allow for the chance that it's actually MPI_INIT that passes down the arguments....... this is pretty ugly...... My personal vote would be for making the user rename their top level entry point: MPI_main(int argc, char **argv, char **environ) in C and MPI_MAIN in fortran Perhaps some compiler level support for potentially changing the name of the entry point so that third party software vendors could build their own "front-ends" that puts their own code first?? I appreciate that this is definitately NOT current practice and has all sorts of diseases of its own with different software suppliers providing layer upon layer of "fake" compiler front-ends. Another option that I would probably support would be bagging argc/argv altogether at the MPI level. In this case a function call with no arguments that had to be the first thing you did would be OK -- if you do it wrong you probably get a core dump! Express' attempts to deal with argc and argv date back to the days when hardware vendors didn't provide these things and we wanted to ourselves. Now all the systems I know of start up C programs apropriately and the need for Express (or some other system) to get in behind them is not so crucial...?.... A slight counter-example to my own argument, however, is that some systems give you mangled argc/argv and particularly environ. This can lead to bizarre results. (Systems which permit a remote host capability sometimes give you the environment that you would have had on the local host rather than the one you actually have on the remote host.) Jon From owner-mpi-comm@CS.UTK.EDU Mon Mar 8 20:07:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23824; Mon, 8 Mar 93 20:07:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02443; Mon, 8 Mar 93 20:06:01 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 8 Mar 1993 20:06:00 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA02433; Mon, 8 Mar 93 20:05:58 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Mon, 8 Mar 93 17:01 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA17034; Mon, 8 Mar 93 17:00:40 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA15033; Mon, 8 Mar 93 17:00:38 PST Date: Mon, 8 Mar 93 17:00:38 PST From: d39135@sodium.pnl.gov Subject: Re: MPI_INIT To: jwf@lion.Parasoft.COM, mpi-comm@cs.utk.edu Cc: d39135@sodium.pnl.gov Message-Id: <9303090100.AA15033@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu Jon Flower says: > Also, in Rik's example, I think the code has to look sort of > like > > main(argc, argv) > int argc; > char **argv; > { > MPI_INIT(&argc, &argv); > FOO_INIT(&argc, &argv); > /* etc... */ Yeah, sort of. The example I showed was based on one version of current practice (TCGMSG). It doesn't let the INIT's alter the argument list. In general, this implies far worse ugliness than Jon's example, because it requires that applications know enough to ignore arguments intended for the INIT's. A better version of current practice is the X window system, one of whose initializers looks like XtInitialize ( ... &argc, argv) This allows deleting stuff from the argument list, but not adding. Jon's syntax goes farther and allows both. But maybe this is working the wrong issue. I said: > A minor problem with the static boolean suggestion is that then the > initialization routine does not have access to those parts of the > environment passed as (argc,argv). Such access is sometimes useful. It could also be said that the problem is with the (argc,argv) approach. I defended the (argc,argv) approach, sort of, but I confess that I don't like it very much precisely because it DOES prevent calling the MPI initializer from inside some other routine. I ran afoul of this just a few days ago, when I tried to emulate another package using TCGMSG, and discovered that my application called the other package's initializer from several procedures down. What I would *really* like (putting on my user's hat), is an explicit MPI_INIT with no arguments, like Jon suggests. I guess this means that implementors have to find some way to pass environment info other than through the argv list. Can anyone come up with a convincing reason why this is a big problem? --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 00:01:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27466; Tue, 9 Mar 93 00:01:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10468; Tue, 9 Mar 93 00:00:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 00:00:24 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10455; Tue, 9 Mar 93 00:00:22 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA08899; Mon, 8 Mar 93 22:56:45 CST Date: Mon, 8 Mar 93 22:56:45 CST From: Tony Skjellum Message-Id: <9303090456.AA08899@Aurora.CS.MsState.Edu> To: jwf@lion.Parasoft.COM, mpi-comm@cs.utk.edu, d39135@sodium.pnl.gov Subject: Re: MPI_INIT; environment information Guys, What about using envp? That would provide the Unix-compatible, open-ended approach to passing information on the environment? Info could be added etc, as was suggested as a need. This is a layerable-friendly approach. ie, main(argc,argv,envp) int argc; char **argv; char **envp; 1) For Fortran, the getenv_() (or equivalent) call provides the interface. C could use the direct interface to envp, or the getenv() call. We could have mpi_getenv() to avoid conflicts with general system support, if absolutely needed to avoid POSIX interactions. 2) Handling of argc/argv A) argc, argv is made accessible to C through standard access. Made accessible to Fortran through accessor functions (eg, mpi_argc(), mpi_argv(i,string,maxlen)). B) argc, argv could be obliterated for both models (ie, left empty for C, inaccessible to Fortran). All info through mpi_getenv(). May I throw in a comment about main precursors (niam() in the BBN parlance)? This is great if supported by the linker, and a pain if not supported by the linker. If niam() is absent, then the default niam() is added, which just calls main(). If niam() is present, then the user model can choose. However, what if different programs presume the abilility to use niam(). They would not stack up... Without niam(), has to assume preprocessor substitution, or macro substitution. This requires that user uses an MPI-compatible make procedure, or compiler pseudonym, other than naked cc/f**. This turns out to be a thorny problem to handle transparently. Also, the execution model of the CM-5, for instance, does not [did not] require main to exist at all. Execution of arbitrary functions (not only main) is supported by CMMD 2.0, specifically. I don't know what CMMD 3.0 chooses to do, but you get the idea. main/program main access is an assumption true most everywhere, but not everywhere. - Tony From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 15:24:37 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA13894; Tue, 9 Mar 93 15:24:37 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24607; Tue, 9 Mar 93 15:14:24 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 15:14:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24595; Tue, 9 Mar 93 15:14:18 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA24881; Tue, 9 Mar 93 20:13:49 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA06208; Tue, 9 Mar 93 13:12:34 MST Date: Tue, 9 Mar 93 13:12:34 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303092012.AA06208@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: MPI Goals and Scope: More Details Please? As a potential user of MPI, the discussion of I/O below disturbs me a bit and leads to a whole bunch of questions... > > We feel that the MPI standard should provide enough information to > > specify an entire, source-code portable program. > > Bill & Rusty > I agree. > > I'd expect code like this (given an I/O everywhere model) to execute > fine > > int main(int argc,char ** argv) > { > printf("Hello world\n"); > } > > and print as many "Hello world" messages as the number of nodes on > which I ran it. > > -- Jim I also agree with Bill and Rusty's statement about portability. Jim's example brings up a whole bunch of interesting problems. First, should the printf() really print one "Hello world" message per node (process)? I can think of several places where I wouldn't expect that behavior (Express "single" I/O mode for example). The next problem is "ordering". Suppose Jim's example is modified slightly: int main (int argc, char **argv) { int me; MPI_INIT(???); /* Yes, I'm casting my vote for MPI_INIT(). */ me = MPI_MYRANK(); /* another guess at syntax */ printf("Hello world from process %d\n", me); } Now what happens? Do we see first-come first-served? Do we see the "intuitive" ordering? Do we get an error message? At the last meeting a straw vote lead to the adoption of "one processor can do standard I/O". If we do this, then users will use vendor-supplied "parallel I/O" routines to get decent performance. We cannot claim that the MPI standard provides enough information to specify an entire, source-code portable program unless we include I/O. I think that there is general agreement that "parallel I/O" will not be included in MPI-1. Maybe we could re-state the portability goal: The ultimate goal of the MPI standardization effort is to provide enough information to specify an entire, source-code portable program. I certainly won't use MPI unless I believe that this will happen with some version of MPI. I think we should re-examine the scope of the MPI effort and write down what we want to accomplish with MPI-1, MPI-2, etc. in much more detail than is currently in the draft introduction. John Flower has proposed one prioritization. Here's another (from my perspective as a user): 1) Static process creation and identification (and associated environmental inquiry routines). 2) Contexts. 3) Heterogeneous communication semantics (I think we already have this). All communication routines should support heterogeneous communication. 4) Blocking send and receive, contiguous data. 5) Non-blocking send and receive, contiguous data (status(), wait(), cancel()?). 6) Creation/destruction of "static" groups (by "static" I mean processes cannot join/leave an existing group). Grid/toroidal process topology would go here if we decide to include it. 7) Blocking collective communication, contiguous data (broadcast, multiple-broadcast, concatenate, barrier, etc.). 8) "Simple" commonly used blocking collective operations (sum, max, min, etc.). 9) Blocking send and receive, strided data. 10) Non-blocking send and receive, strided data. 11) Blocking collective communication, strided data. 12) Blocking send and receive with buffer-descriptor. 13) Non-blocking send and receive with buffer-descriptor. 14) Blocking collective communication with buffer-descriptor. 15) "User-defined" blocking collective operations (Express "excombine()" style). 16) Probe (and associated "precv()" routines, if necessary). 17) Non-blocking collective communication, contiguous data, strided data, and buffer-descriptor (and "multi-cancel()"?). 18) Definition of parallel I/O. 19) Blocking parallel I/O. 20) Non-blocking parallel I/O. 21) Additional environmental inquiry and "setting" routines for performance optimization (suggestions for system buffering, relative node/interconnect performance, etc.). 22) Dynamic process creation and identification. 23) Dynamic groups (processes can join/leave an existing group). 24) "Safe" send and receive. 25) Ready-send and ready-receive. 26) General process topologies. 27) Even more neat stuff. Once a detailed list like this is agreed on, we need to decide where to draw the line between MPI-1 and MPI-2 (and maybe MPI-3, etc.). Various straw polls about I/O indicate that 18-20 will probably be in MPI-2. I imagine an absolute minimum might be 1-5 for MPI-1. Here's three proposals for the list above: MPI-1 MPI-2 MPI-3, etc. "Ambitious" 1-17 18-26 27 "Less Ambitious" 1-14 15-23 24-27 "Conservative" 1-11 12-19 20-27 I share John Flower's "EXTREME concern" about the complexity of the current design. I believe that the MPIF must agree on SOME prioritization list and explicitly decide what is going to be in MPI-1 as soon as possible. I also feel strongly that we should decide what will be in future MPI versions to give potential users some confidence that MPI will eventually be useful for developing source-code portable programs. Tom Henderson (a potential MPI user) NOAA Forecast Systems Laboratory hender@fsl.noaa.gov (Thanks to Leslie Hart for some helpful suggestions.) From owner-mpi-comm@CS.UTK.EDU Tue Mar 9 20:46:50 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17983; Tue, 9 Mar 93 20:46:50 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10433; Tue, 9 Mar 93 20:45:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 9 Mar 1993 20:45:19 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA10412; Tue, 9 Mar 93 20:45:16 -0500 Date: Wed, 10 Mar 93 01:45:05 GMT Message-Id: <2554.9303100145@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Goals and Scope: More Details Please? To: hender@macaw.fsl.noaa.gov (Tom Henderson), mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Tue, 9 Mar 93 13:12:34 MST Reply-To: lyndon@epcc.ed.ac.uk Hi all There seems to be something to be discussed in what Tom is saying. Here is my 2 pennies worth. Please understand my multicomputer world bias, or flame me for it, whichever your prefer. > > > We feel that the MPI standard should provide enough information to > > > specify an entire, source-code portable program. > > > Bill & Rusty > > > I agree. This is going to sound awful picky, but I'm afraid I gotta say that I disagree with this. It's a grand ambition, and its also a grand challenge, in fact I conjecture that MPI is implicitly incapable of completing the stated mission even given arbitrary time, and that it is not a valid goal for MPI. The goal requires MPI to describe a whole raft of issues which are outwith MPI. I reckon John Flowers could spend the rest of his MPI time telling us about his experiences in Parasoft and Express with the issues that MPI cannot address! So MPI needs a more restricted mission. I'm too sleepy to try right now to find a suitable modification of the above sentence which might serve as a better mission for MPI. > > At the last meeting a straw vote lead to the adoption of "one processor can do > standard I/O". If we do this, then users will use vendor-supplied > "parallel I/O" routines to get decent performance. We cannot claim that the > MPI standard provides enough information to specify an entire, source-code > portable program unless we include I/O. I think that there is general > agreement that "parallel I/O" will not be included in MPI-1. Don't you think it will be some time before the world is ready to even think about standardisation of "parallel I/O". I can't imagine trying to do such a thing at the moment. > Maybe we could > re-state the portability goal: > The ultimate goal of the MPI standardization effort is to provide enough > information to specify an entire, source-code portable program. I have the same problem with this mission as that above. > I certainly won't use MPI unless I believe that this will happen with some > version of MPI. I reckon you will, actually, because although it won't give you the portability asked for above, it will be your best way of getting some portability. > I think we should re-examine the scope of the MPI effort and write down what > we want to accomplish with MPI-1, MPI-2, etc. in much more detail than is > currently in the draft introduction. I concur, in an ambivalent fashion. The introduction is vague, I'll agree with that. I think we are currently working out what we should be doing, in a rather ad hoc fashion. > John Flower has proposed one > prioritization. Here's another (from my perspective as a user): I'm going to walk down this list and give marks out of ten, with ten as highest priority and 0 as lowest priority, from my point of view, and some justification, from my point of view. > 1) Static process creation and identification (and associated environmental > inquiry routines). 10 - bare minimum MPI can acheive without being complete waste of time. > 2) Contexts. 9 - can get away with just tags but would impede library development. > 3) Heterogeneous communication semantics (I think we already have this). > All communication routines should support heterogeneous communication. 6 - static SPMD programs are most sensible in homogeneous machines. > 4) Blocking send and receive, contiguous data. 10 - bare minimum MPI can acheive without being complete waste of time. > 5) Non-blocking send and receive, contiguous data (status(), wait(), > cancel()?). 10 - bare minimum MPI can acheive without being complete waste of time. > 6) Creation/destruction of "static" groups (by "static" I mean processes > cannot join/leave an existing group). Grid/toroidal process topology > would go here if we decide to include it. 8 - can get away with just tags but would impede library development. > 7) Blocking collective communication, contiguous data (broadcast, > multiple-broadcast, concatenate, barrier, etc.). 10 - bare minimum MPI can acheive without being complete waste of time. > 8) "Simple" commonly used blocking collective operations (sum, max, min, > etc.). 10 - bare minimum MPI can acheive without being complete waste of time. > 9) Blocking send and receive, strided data. 8 - useful, but not essential. > 10) Non-blocking send and receive, strided data. 8 - useful, bit not essential. > 11) Blocking collective communication, strided data. 5 - makes collective communications too big. > 12) Blocking send and receive with buffer-descriptor. 8 - useful, but not essential. > 13) Non-blocking send and receive with buffer-descriptor. 8 - useful, but not essential. > 14) Blocking collective communication with buffer-descriptor. 5 - makes collective communications too big. > 15) "User-defined" blocking collective operations (Express "excombine()" > style). 10 - bare minimum MPI can acheive without being complete waste of time. > 16) Probe (and associated "precv()" routines, if necessary). 0 (boy, am I biased! :-) > 17) Non-blocking collective communication, contiguous data, strided data, and > buffer-descriptor (and "multi-cancel()"?). 5 - makes collective communication to big, and has technical complications. > 18) Definition of parallel I/O. 2 - too early to think about standardisation. > 19) Blocking parallel I/O. 2 - too early to think about standardisation. > 20) Non-blocking parallel I/O. 2 - too early to think about standardisation. > 21) Additional environmental inquiry and "setting" routines for performance > optimization (suggestions for system buffering, relative > node/interconnect performance, etc.). 2 - useless without better knowledge of how to use them. > 22) Dynamic process creation and identification. 6 - process group creation and resize is tractable model > 23) Dynamic groups (processes can join/leave an existing group). 2 - asynchronous join/leave very experimental 6 - process group creation and resize is tractable model > 24) "Safe" send and receive. 10 - bare minimum MPI can acheive without being complete waste of time. > 25) Ready-send and ready-receive. 0 - what benefit except for some old machines? > 26) General process topologies. 8 - if we have topologies at all, then should do this. > 27) Even more neat stuff. ? - ??? [stuff deleted] I think you can see what my staged schedule would be from the above. > > I share John Flower's "EXTREME concern" about the complexity of the current > design. I believe that the MPIF must agree on SOME prioritization list and > explicitly decide what is going to be in MPI-1 as soon as possible. I concur, ambivalently, as I think that it will yet take a little time for a strong consensus to emerge. I applaude efforts to generate such a consensus. > I also > feel strongly that we should decide what will be in future MPI versions to > give potential users some confidence that MPI will eventually be useful for > developing source-code portable programs. I am again ambivalent. I think it important to identify what items should be suitable for future MPI, but I'm not at all confident that it will be practical to commit ourselves to a schedule for trying to standardise such items. It will be harder to agree on standards for the items which are deferred. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 08:14:38 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28523; Wed, 10 Mar 93 08:14:38 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11106; Wed, 10 Mar 93 08:13:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 08:13:22 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11098; Wed, 10 Mar 93 08:13:21 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA20589; Wed, 10 Mar 93 08:13:19 EST Received: by b125.super.org (4.1/SMI-4.1) id AA01744; Wed, 10 Mar 93 08:13:18 EST Date: Wed, 10 Mar 93 08:13:18 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303101313.AA01744@b125.super.org> To: mpi-comm@cs.utk.edu Subject: MPI directions & priorities I was in the process of composing a note to the committee about what MPI-1 should compose and was letting it sit overnight when Tom sent out his message. Since he has done a nice job I will shorten my comments. At the last meeting there were several group and private discussions about the growing complexity of MPI. Al has posted his feeling that point-to-point and collective communications get complex and untested ideas added but other areas such as dynamic processes are not considered. Jon posted about the complexity and now Tom has started a list of priorities. I add my voice to these concerns. I have been hesitant to post because I don't want it to seem that I think MPI is hopeless. However, I think it is clear that we cannot adequately complete all the current proposals in 3 more meetings. I would propose that we continue the discussion concerning priorities via e-mail and begin the next meeting with a whole group meeting to set our course. At the higher level we need to decide: 1) How long will the MPI-1 process be allowed to take? Are we going to stick to the 6 month deadline or try and extend it? 2) What is going to be in MPI-1? 3) Of the things we leave out of MPI-1 are they going to be proposed for a later version of MPI? Should we make sure that MPI-1 is compatible with the future goals so that major changes in the base routines is not needed later? 4) If we want to have MPI-#, what mechanism and time scale will be used to get it done. I think it is past time to find the forest for the trees. Global planning to focus our efforts now could significantly reduce our discussions and efforts in the short term as deadlines approach. Furthermore, with a global blueprint we might see a more even handed approach to different aspects of the standard. These are just some thoughts. I hope the discussion will continue. Steve From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 10:15:06 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02371; Wed, 10 Mar 93 10:15:06 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16993; Wed, 10 Mar 93 10:13:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:13:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16985; Wed, 10 Mar 93 10:13:18 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA03655 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 09:13:16 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05373; Wed, 10 Mar 93 09:13:13 CST Date: Wed, 10 Mar 93 09:13:13 CST Message-Id: <9303101513.AA05373@godzilla.mcs.anl.gov> To: lederman@b125.super.org Cc: mpi-comm@cs.utk.edu In-Reply-To: Steve Huss-Lederman's message of Wed, 10 Mar 93 08:13:18 EST <9303101313.AA01744@b125.super.org> Subject: MPI directions & priorities One approach that we could take is to describe a set of MPI that is extensible to the more elaborate facilities but does not yet specify them. For example, we could keep the goal of being "thread-safe" and resolve (at least in an extensible way) the group/context/process-id issues, and then settle on, for example, somethine like what Rusty and I have called level 3, plus some collective communication routines. This would not have everything that I would want or even consider important, but would be a usable set for casual use. This is a very different process than just defining a simple set of operations; it MUST be possible to extend, in a consistent way, MPI to meet new needs. I'd like to point out that many of the so-called new features are ones that are in use in some (usually only one) system; in most cases they fulfill a perceived need. For MPI to be successful, it must not be incompatible with these needs. I realize that the other side of this is that, by specifying only a subset of desired functionality, there is the danger of mutually incompatible extensions. We need to weigh this against the possibility that MPI will become too complex to be accepted. Bill From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 10:24:04 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02512; Wed, 10 Mar 93 10:24:04 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17488; Wed, 10 Mar 93 10:22:34 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:22:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA17476; Wed, 10 Mar 93 10:22:29 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA03990 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 09:22:24 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA05385; Wed, 10 Mar 93 09:22:22 CST Date: Wed, 10 Mar 93 09:22:22 CST Message-Id: <9303101522.AA05385@godzilla.mcs.anl.gov> To: lyndon@epcc.ed.ac.uk Cc: hender@macaw.fsl.noaa.gov, mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Wed, 10 Mar 93 01:45:05 GMT <2554.9303100145@subnode.epcc.ed.ac.uk> Subject: MPI Goals and Scope: More Details Please? I'd like to clarify what we mean by > We feel that the MPI standard should provide enough information to > specify an entire, source-code portable program. This is NOT intended to mean that MPI must provide a way to specify ANY (or even most) source-code portable parallel program. Rather, given restrictions strong enough to satisfy a 2/3rds majority ( :) ), a way to specify at least one source-code portable program. In the spirit of extensibility, whatever mechanism is proposed for this simple interface should not preclude non-MPI solutions for more demanding situations. Bill From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 11:33:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03903; Wed, 10 Mar 93 11:33:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21975; Wed, 10 Mar 93 11:31:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 11:31:41 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA21960; Wed, 10 Mar 93 11:31:39 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA10050; Wed, 10 Mar 93 10:27:36 CST Date: Wed, 10 Mar 93 10:27:36 CST From: Tony Skjellum Message-Id: <9303101627.AA10050@Aurora.CS.MsState.Edu> To: lederman@b125.super.org, gropp@mcs.anl.gov Subject: Re: MPI directions & priorities Cc: mpi-comm@cs.utk.edu We must agree to reconvene about extensions in future. I also agree that it would be preferable to specify a whole portable program, but I think the I/O subject is hard to address right now... - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 09:42:01 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 10:13:21 EST From: William Gropp Date: Wed, 10 Mar 93 09:13:13 CST To: lederman@b125.super.org Cc: mpi-comm@cs.utk.edu Subject: MPI directions & priorities Content-Length: 1193 One approach that we could take is to describe a set of MPI that is extensible to the more elaborate facilities but does not yet specify them. For example, we could keep the goal of being "thread-safe" and resolve (at least in an extensible way) the group/context/process-id issues, and then settle on, for example, somethine like what Rusty and I have called level 3, plus some collective communication routines. This would not have everything that I would want or even consider important, but would be a usable set for casual use. This is a very different process than just defining a simple set of operations; it MUST be possible to extend, in a consistent way, MPI to meet new needs. I'd like to point out that many of the so-called new features are ones that are in use in some (usually only one) system; in most cases they fulfill a perceived need. For MPI to be successful, it must not be incompatible with these needs. I realize that the other side of this is that, by specifying only a subset of desired functionality, there is the danger of mutually incompatible extensions. We need to weigh this against the possibility that MPI will become too complex to be accepted. Bill ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:23:22 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05096; Wed, 10 Mar 93 12:23:22 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24792; Wed, 10 Mar 93 12:22:05 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:22:04 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA24784; Wed, 10 Mar 93 12:22:01 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA26984; Wed, 10 Mar 93 17:21:17 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07764; Wed, 10 Mar 93 10:20:02 MST Date: Wed, 10 Mar 93 10:20:02 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101720.AA07764@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI Goals and Scope: More Details Please? Cc: lyndon@epcc.edinburgh.ac.uk Lyndon writes, > Don't you think it will be some time before the world is ready to even > think about standardisation of "parallel I/O". I can't imagine trying > to do such a thing at the moment. I agree. I'm not proposing we do this now. I'm proposing that we do it when it is as well understood as message passing is now. I think "eventual standardization of parallel I/O" should be a stated goal to give potential MPI users confidence that MPI won't be just another message-passing library. > [stuff deleted...] > > > I certainly won't use MPI unless I believe that this will happen with some > > version of MPI. > > I reckon you will, actually, because although it won't give you the > portability asked for above, it will be your best way of getting some > portability. I think Lyndon has a good point here. I will use MPI if: A) It gives me as much "portability" as existing solutions OR I believe that this will eventually happen. B) It gives me equal or better "performance". I will borrow/build/buy whatever additional stuff I need to make my programs more "portable", if necessary (required for my current project). I am hopeful that A will happen with some version of MPI. Vendor participation gives me some confidence that B will happen. > > John Flower has proposed one > > prioritization. Here's another (from my perspective as a user): > > I'm going to walk down this list and give marks out of ten, with ten as > highest priority and 0 as lowest priority, from my point of view, and > some justification, from my point of view. > > [list deleted...] Thanks Lyndon, this is exactly what I was looking for. At this point I feel it is extremely important that the MPIF come to some agreement about a list like this. It is a lot more important than my actually liking what's on the list! :) > [stuff deleted...] > > I am again ambivalent. I think it important to identify what items > should be suitable for future MPI, but I'm not at all confident that it > will be practical to commit ourselves to a schedule for trying to > standardise such items. It will be harder to agree on standards for the > items which are deferred. > > Best Wishes > Lyndon > I didn't mean to imply an absolute time-frame for the introduction of future MPI features. (It doesn't make sense for the more "researchy" stuff.) Prioritization is more important to me than a strict schedule. Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:32:07 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05213; Wed, 10 Mar 93 12:32:07 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25472; Wed, 10 Mar 93 12:30:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:30:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA25460; Wed, 10 Mar 93 12:30:47 -0500 Date: Wed, 10 Mar 93 17:30:30 GMT Message-Id: <3289.9303101730@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: MPI Goals and Scope: More Details Please? To: hender@macaw.fsl.noaa.gov (Tom Henderson), mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Wed, 10 Mar 93 10:20:02 MST Reply-To: lyndon@epcc.ed.ac.uk Cc: lyndon@epcc.edinburgh.ac.uk > > I'm going to walk down this list and give marks out of ten, with ten as > > highest priority and 0 as lowest priority, from my point of view, and > > some justification, from my point of view. > > > > [list deleted...] > > Thanks Lyndon, this is exactly what I was looking for. Oh, good, glad I did that then! > At this point I feel > it is extremely important that the MPIF come to some agreement about a list > like this. AGREED!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > I didn't mean to imply an absolute time-frame for the introduction of future > MPI features. (It doesn't make sense for the more "researchy" stuff.) > Prioritization is more important to me than a strict schedule. Sorry, Tom, I misread your email. It's over and out for a week from me, colleagues, as I'm visiting my folks for the weekend and a couple of work meetings - surprise surprise, one of them to talk about MPI to a potential user, urgle urgle urgle! May your mail boxes be temporarily lighter ;-) Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 12:43:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05464; Wed, 10 Mar 93 12:43:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26174; Wed, 10 Mar 93 12:41:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:41:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA26164; Wed, 10 Mar 93 12:41:55 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27092; Wed, 10 Mar 93 17:41:49 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07798; Wed, 10 Mar 93 10:40:35 MST Date: Wed, 10 Mar 93 10:40:35 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101740.AA07798@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI Goals and Scope: More Details Please? Cc: gropp@mcs.anl.gov Bill writes, > I'd like to clarify what we mean by > > > We feel that the MPI standard should provide enough information to > > specify an entire, source-code portable program. > > This is NOT intended to mean that MPI must provide a way to specify ANY (or > even most) source-code portable parallel program. Rather, given restrictions > strong enough to satisfy a 2/3rds majority ( :) ), a way to specify at least > one source-code portable program. In the spirit of extensibility, whatever > mechanism is proposed for this simple interface should not preclude non-MPI > solutions for more demanding situations. > Bill > Could you give an example of the one source-code portable program (or class of programs) you expect MPI to support in current and future versions? I think that might help me understand your statement a bit better. Also, what do you mean by "non-MPI solution" and "demanding situations"? Thanks, Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:19:43 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06396; Wed, 10 Mar 93 13:19:43 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28116; Wed, 10 Mar 93 13:18:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:18:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28057; Wed, 10 Mar 93 13:18:02 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 10:10 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA22798; Wed, 10 Mar 93 10:08:19 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA20772; Wed, 10 Mar 93 10:08:13 PST Date: Wed, 10 Mar 93 10:08:13 PST From: d39135@sodium.pnl.gov Subject: implementation assumptions To: d39135@sodium.pnl.gov, jim@meiko.co.uk, lyndon@epcc.edinburgh.ac.uk, mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, ranka@top.cis.syr.edu, tony@Aurora.CS.MsState.Edu Message-Id: <9303101808.AA20772@sodium.pnl.gov> X-Envelope-To: mpi-context@cs.utk.edu, mpi-comm@cs.utk.edu In a discussion we are having within the contexts subcommittee, Tony Skjellum commented that: > ... I want to make sure that we do > not provide illusory improvements to the standard, while letting other > parts of it exceed ours in the unscalability department, or let > implementations qualify as compliant, whose unscalability disables the > implied scalability of our syntax/semantics... I agree. This is why (I think) proposals should explicitly address performance. If a proposal assumes that the cost of a particular function is irrelevant, it should say that. If a proposal assumes that a particular function is going to be cheap, it should justify that assumption by naming a time/memory price and outlining an implementation that can achieve it. Sometimes the suggested implementation will reveal further assumptions that have wide implications. Ideally, each proposal will eventually evolve a set of bottom-line assumptions that are consistent and believable. These comments are particularly addressed to those of us beating on contexts and groups, where there seem to be lots of subtle interactions. But I am cross-posting them to the whole committee as food for thought... --Rik ("The Skeptic") Littlefield From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:22:14 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06459; Wed, 10 Mar 93 13:22:14 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28196; Wed, 10 Mar 93 13:20:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:20:18 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28188; Wed, 10 Mar 93 13:20:16 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27261; Wed, 10 Mar 93 18:20:08 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA07839; Wed, 10 Mar 93 11:18:54 MST Date: Wed, 10 Mar 93 11:18:54 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303101818.AA07839@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities Cc: lederman@b125.super.org Steve writes, > 1) How long will the MPI-1 process be allowed to take? Are we going > to stick to the 6 month deadline or try and extend it? I'd like to stick to 6 months for MPI-1. I'm willing to have less functionality in MPI-1 as long as there is a plan for further upgrade. > 2) What is going to be in MPI-1? Good question. :) > 3) Of the things we leave out of MPI-1 are they going to be proposed > for a later version of MPI? Should we make sure that MPI-1 is > compatible with the future goals so that major changes in the base > routines is not needed later? Yes and Yes. > 4) If we want to have MPI-#, what mechanism and time scale will be > used to get it done. Maybe next year we meet again for MPI-2? We should have some experience with MPI-1 by then. (...wait a minute, I only signed on for a six-month tour...) > I think it is past time to find the forest for the trees. Global > planning to focus our efforts now could significantly reduce our > discussions and efforts in the short term as deadlines approach. > Furthermore, with a global blueprint we might see a more even handed > approach to different aspects of the standard. These are just some > thoughts. I hope the discussion will continue. Me too. > Steve Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:22:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06475; Wed, 10 Mar 93 13:22:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28313; Wed, 10 Mar 93 13:21:38 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:21:37 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28305; Wed, 10 Mar 93 13:21:34 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27273; Wed, 10 Mar 93 18:21:30 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA10163; Wed, 10 Mar 93 11:24:00 MST Date: Wed, 10 Mar 93 11:24:00 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303101824.AA10163@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities Bill Gropp sez: > One approach that we could take is to describe a set of MPI that is extensible > to the more elaborate facilities but does not yet specify them. For example, > we could keep the goal of being "thread-safe" and resolve (at least in an > extensible way) the group/context/process-id issues, and then settle on, for > example, somethine like what Rusty and I have called level 3, plus some > collective communication routines. This would not have everything that I would > want or even consider important, but would be a usable set for casual use. > > This is a very different process than just defining a simple set of operations; > it MUST be possible to extend, in a consistent way, MPI to meet new needs. > I'd like to point out that many of the so-called new features are ones that > are in use in some (usually only one) system; in most cases they fulfill > a perceived need. For MPI to be successful, it must not be incompatible with > these needs. > > I realize that the other side of this is that, by specifying only a subset of > desired functionality, there is the danger of mutually incompatible extensions. > We need to weigh this against the possibility that MPI will become too complex > to be accepted. > > Bill I agree with Bill, I think whatever we produce for MPI-1 should be extensible. I think we should consider carefully our semantics and syntax to not prohibit extensibility. An approach that has been proposed by us (Tom Henderson and myself) as well as by other people is opaque messages. This allows an initial implementation to support a relatively simple to use subset while allowing vendor extensions and future upgrades to MPI-1. We could perhaps specify some core of MPI-1 (not a new idea) as well as provide semantic and syntatical guidance to vendors for their extensions (e.g. a clue about MPI-2 if you will). The opaque message object allows these extensions to be put in place without an explosion of syntax. Leslie Hart (hart@fsl.noaa.gov) P.S This may fit into Rusty & Bill's Level 1 or Level 2 proposal neatly. From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 13:26:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06583; Wed, 10 Mar 93 13:26:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28684; Wed, 10 Mar 93 13:25:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 13:25:58 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA28669; Wed, 10 Mar 93 13:25:55 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA10195 (5.65c/IDA-1.4.4 for ); Wed, 10 Mar 1993 12:25:52 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA22431; Wed, 10 Mar 93 12:25:51 CST Date: Wed, 10 Mar 93 12:25:51 CST Message-Id: <9303101825.AA22431@godzilla.mcs.anl.gov> To: hender@macaw.fsl.noaa.gov Cc: mpi-comm@cs.utk.edu In-Reply-To: Tom Henderson's message of Wed, 10 Mar 93 10:40:35 MST <9303101740.AA07798@macaw.fsl.noaa.gov> Subject: MPI Goals and Scope: More Details Please? X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 12:41:58 EST Date: Wed, 10 Mar 93 10:40:35 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Could you give an example of the one source-code portable program (or class of programs) you expect MPI to support in current and future versions? I think that might help me understand your statement a bit better. Basically, SPMD programs that have "simple" I/O requirements. For example, we could ask the following: IF fprintf (C) or write (Fortran) are supported, THEN each use writes to the selected file, with NO guarentees on the ordering of output. Something similiar for fscanf/read . Thus, we can now write the following two programs (using the MPI_main startup approach): MPI_main() { printf( "Hello world\n" ); } and MPI_main() { int i; for (i=0; i To: mpi-comm@cs.utk.edu Subject: Re: MPI directions & priorities > From: Tony Skjellum > To: lederman@b125.super.org, gropp@mcs.anl.gov > Subject: Re: MPI directions & priorities > Cc: mpi-comm@cs.utk.edu > > We must agree to reconvene about extensions in future. I also agree > that it would be preferable to specify a whole portable program, but I > think the I/O subject is hard to address right now... > > - Tony I also understood that the process would be ongoing and hope that it will continue to track existing practice. I think some parts of the I/O subject are hard to address right now, but there is some existing common practice. Parasoft, Intel, nCUBE and TMC all have some sort of parallel I/O specification. There are ways to force the effect of a printf or of an fseek/fread/fwrite to be deterministic. I believe that the common practice could be standardized, maybe not in the time frame for MPI-1 but probably for MPI-2 (which is where it appeared in Tom's posting). Regards, Leslie From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 14:03:55 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07595; Wed, 10 Mar 93 14:03:55 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01003; Wed, 10 Mar 93 14:02:53 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 14:02:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA00994; Wed, 10 Mar 93 14:02:47 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA08764 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Wed, 10 Mar 1993 11:02:44 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA26008; Wed, 10 Mar 93 10:55:24 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA26815; Wed, 10 Mar 93 10:55:34 PST Date: Wed, 10 Mar 93 10:55:34 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303101855.AA26815@lion.parasoft> To: mpi-comm@cs.utk.edu To: mpi-comm@cs.utk.edu Re: A way of prioritizing? Something Bill said seemed to strike a chord in my mind: > This is a very different process than just defining a simple > set of operations; it MUST be possible to extend, in a > consistent way, MPI to meet new needs. I'd like to point > out that many of the so-called new features are ones that > are in use in some (usually only one) system; in most cases > they fulfill a perceived need. For MPI to be successful, > it must not be incompatible with these needs. I also have the feeling that a lot of the things that we're trying to put in MPI are really only based on experience (or even need) on a single machine. Sometimes that machine may even be "pre-history". My question is: Is incorporating into MPI a feature that is only based on a single machine any better than allowing a vendor-specific extension on that machine? Take "ready-receive", for example. Adding this to MPI means that users can justifiably put it in their portable codes. If they do, however, what should they assume about it? That it's faster or slower than the non-ready receive? I assume that there will be systems in which it's slower than the non-ready recv. because of the extra protocol that the application might need to ensure that the ready-recv.'s don't crash. In this case, are we expecting users to put code in their applications that can switch between ready-recv, and non-ready-recv semantics based on the hardware they're running on? Although I think there are some users who will do this (and I would encourage library developers to do so) I think this is far beyond the lengths that most users will go to. So what am I suggesting? --------------------------------------------------------- My suggestion (which I expect to be roundly rejected) is that the MPI document ONLY contain the stuff that is current practice. This includes EVERYTHING required to run at least a common class of programs - i.e., MPI_INIT or some such. These things should be designed to maximize ease of use and learning and encourage portability. By this I am including performance level portability. I would like to not have things in the spec. that run well on some machines and poorly on others. I would include a requirement that a conforming implementation include either documents or (preferably) programs from which a user could assess the performance of the various routines. All the other things that are off-shoots of requirements for odd-machines or otherwise more advanced should go in Appendices or "Annexes" (Ada notation(!)) ***AND WOULD NOT BE REQUIRED BY A CONFORMING IMPLEMENTATION OF MPI ***. --------------------------------------------------------- The effect I am after is that the core MPI standard is simple, effective and encourages users to write code. The annexes essentially say to vendors; "If you want to support an advanced feature that will really help on your machine, please do it this way so that my programs that use it will be portable." This is similar to Bill & Rusty's multi-level scenario except that I wouldn't make the lower levels compulsory. My hope is that two things will happen if we adopt this strategy: a) Details of things that go in Annexes can be postponed or left vague, or even left to specially interested groups to define, but we will definitely have something that could be used by an application developer at the end of the six months. b) The MPI core would be smaller which will encourage users/vendors to make implementations, and to optimize the #$%$% out of them. If this concept is acceptable I would suggest that someone prepare a list of topics, in quite great detail, and we all give them grades out of ten (via e-mail), just as Lyndon did. Someone with experience with statistics could analyze the results and give us weighted rankings at the next meeting. We could then prioritize according to this list and also make guided decisions as to what should go in the Annexes. I would then suggest that we go back to the small sub-committee approach to putting the Annexes together rather than our current "committee-of-the-whole" style which seems to encourage hour-long "arguments" followed by 35-0 votes! Jon Flower p.s. Having suggested a pile of work I should say who's going to do it. In the absence of any volunteers I might be persuaded to come up with a list of issues on which we should score. I'm not sure my statistical skills amount to anything much more that simple averages, so someone else could certainly do better than I. I would be very interested in working with a group/sub-committee trying to get up a set of MPI-compliant performance evaluation standards. From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 15:32:12 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09642; Wed, 10 Mar 93 15:32:12 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07600; Wed, 10 Mar 93 15:30:40 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 15:30:38 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA07590; Wed, 10 Mar 93 15:30:35 -0500 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 10 Mar 93 12:29 PST Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA24307; Wed, 10 Mar 93 12:27:24 PST Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA21996; Wed, 10 Mar 93 12:27:21 PST Date: Wed, 10 Mar 93 12:27:21 PST From: d39135@sodium.pnl.gov Subject: RE: MPI directions & priorities To: gropp@mcs.anl.gov, lederman@b125.super.org, tony@Aurora.CS.MsState.Edu Cc: d39135@sodium.pnl.gov, mpi-comm@cs.utk.edu Message-Id: <9303102027.AA21996@sodium.pnl.gov> X-Envelope-To: mpi-comm@cs.utk.edu > We must agree to reconvene about extensions in future. I also agree > that it would be preferable to specify a whole portable program, but I > think the I/O subject is hard to address right now... > > - Tony I presume that Tony speaks of parallel I/O. I agree that parallel I/O is too hard for now. But I think that we have to specify at least some minimum serial I/O capability. I would buy off on something as simple as . there exists one process that can do serial I/O in the standard fashion for the language (e.g., printf() or WRITE(*,*) ) . the tid of that process can be obtained from MPI (in any process) Example: if (mytid == MPI_IOROOT()) { printf ("I'm the designated serial I/O process\n"); } --Rik Littlefield From owner-mpi-comm@CS.UTK.EDU Wed Mar 10 15:37:36 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA09805; Wed, 10 Mar 93 15:37:36 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08036; Wed, 10 Mar 93 15:36:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 10 Mar 1993 15:36:27 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08022; Wed, 10 Mar 93 15:36:25 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA27653; Wed, 10 Mar 93 20:36:21 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA08016; Wed, 10 Mar 93 13:35:06 MST Date: Wed, 10 Mar 93 13:35:06 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303102035.AA08016@macaw.fsl.noaa.gov> To: jwf@lion.parasoft.com Subject: Annexes, etc. Cc: mpi-comm@cs.utk.edu Jon, > My suggestion (which I expect to be roundly rejected) is > that the MPI document ONLY contain the stuff that is current > practice. This includes EVERYTHING required to run at least > a common class of programs - i.e., MPI_INIT or some such. > > These things should be designed to maximize ease of use and > learning and encourage portability. By this I am including > performance level portability. I would like to not have > things in the spec. that run well on some machines and > poorly on others. I would include a requirement that a > conforming implementation include either documents or > (preferably) programs from which a user could assess > the performance of the various routines. > > All the other things that are off-shoots of requirements > for odd-machines or otherwise more advanced should go in > Appendices or "Annexes" (Ada notation(!)) ***AND WOULD NOT > BE REQUIRED BY A CONFORMING IMPLEMENTATION OF MPI ***. Would the MPIF make any commitment to including "Annex" features in future MPI versions? Would there be any future MPI versions? Would you mind providing a bit more detail on what you feel are a "common class of programs"? (Gee, I don't ask for much... :-) > If this concept is acceptable I would suggest that someone > prepare a list of topics, in quite great detail, and we > all give them grades out of ten (via e-mail), just as > Lyndon did. Someone with experience with statistics > could analyze the results and give us weighted rankings > at the next meeting. What level of detail do you have in mind? Could you give an example? I like this idea. Hashing this out over e-mail could save us a lot of pointless wrangling at the next meeting. Tom P.S. Jon, my apologies for mis-spellings in previous messages! :) From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 05:10:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA25347; Thu, 11 Mar 93 05:10:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11372; Thu, 11 Mar 93 05:08:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 05:08:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11362; Thu, 11 Mar 93 05:08:16 -0500 Date: Thu, 11 Mar 93 10:07:58 GMT Message-Id: <3792.9303111007@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: minutes? To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk I never saw any minutes for the February meeting yet, perhaps I've been expecting them too early. Please, can someone let me know when they are expected to be circulated. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 06:02:48 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07862; Thu, 11 Mar 93 06:02:48 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18258; Thu, 11 Mar 93 06:01:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 06:01:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA18250; Thu, 11 Mar 93 06:01:20 -0500 Date: Thu, 11 Mar 93 11:01:15 GMT Message-Id: <3852.9303111101@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: regarding discussion of directions and priorities To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I am shortly to give a presentation regarding MPI to a potential end user of some imprtance within the UK. In preparation for this, I have spent the last hour of this morning looking back over the discussions on mpi-comm and various subcommittees. The single thing which most struck me was that since March 9 the content on mpi-comm has been dominated by discussion of directions and priorities for MPI. I recorded 20 mail messages in total from 8 individuals. There were preceeding discussions of this nature on mpi-pt2pt dating from March 4. The discussions indicate to me that there are some rather concerned committee members. It would appear, to my mind at least, that different committee members have different views of the extant directions and priorities of MPI, and of desired directions and priorities for MPI. I am sure that I have a different view from some other members of MPI. In order to make satisfactory progress in the process of interface definition, which is apparently not occuring at the moment, I believe that there would indeed be advantage in describing and formally agreeing the directions of MPI and a set of priorities within those directions. I applaude efforts to create consensus regarding directions and priorities. I also believe that, from the point of view of the perception of MPI of individuals and organisations not directly involved in the committee, there is advantage in publicising the agreed directions of MPI. In order to make satisfactory progress in the process of determining directions and priorities, I believe that there would be advantage in describing and formally agreeing a framework within which we can agree on such directions and priorities. Sadly, I do not observe such a framework in operation, and I am skeptical whether MPI can succeed without such a framework. I would therefore suggest that the immediate priorities for MPI as as follows: 1. Discuss and establish framework within which directions for MPI are assessed. 2. Discuss and establish directions for MPI within agreed framework. 3. Discuss and establish priorities for MPI within agreed directions. 4. Review proposals in all subcommittees within agreed priorities. 5. Extend and complete proposals as necessary within terms of 3 and 4. As a suggestion, can we all describe what we mean by "common practice", a starting point for framework discussion? Some of the 8 mentioned above have already done this, to some extent, which I also applaud. It seems to me that time demands 1,2 and 3 must occur primarily by email. I would suggest that these should be formally accepted at the beginning of the next meeting. From a practical point of view, I further suggest that the next meeting should call a meeting of the whole committee for this purpose, and we should be prepared to hold said meeting on the Wednesday morning, on order to allow the subcommittees to make progress with 4 and 5. Now my taxi is waiting, and I must go off-line for a week. I apologise to you all that I will not be able to follow up this somewhat despondent message for 6 days. Good luck! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 11:20:08 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29874; Thu, 11 Mar 93 11:20:08 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01139; Thu, 11 Mar 93 11:17:49 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 11:17:48 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01131; Thu, 11 Mar 93 11:17:47 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA13509; Thu, 11 Mar 93 10:13:56 CST Date: Thu, 11 Mar 93 10:13:56 CST From: Tony Skjellum Message-Id: <9303111613.AA13509@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: minutes? Yes, having the minutes is becoming essential. - Tony ----- Begin Included Message ----- I never saw any minutes for the February meeting yet, perhaps I've been expecting them too early. Please, can someone let me know when they are expected to be circulated. Best Wishes Lyndon ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 11:23:41 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00145; Thu, 11 Mar 93 11:23:41 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01349; Thu, 11 Mar 93 11:21:30 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 11:21:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA01340; Thu, 11 Mar 93 11:21:27 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA00895; Thu, 11 Mar 93 11:21:24 -0500 Date: Thu, 11 Mar 93 11:21:24 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303111621.AA00895@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Jon's suggestion Jon writes: > My suggestion (which I expect to be roundly rejected) is > that the MPI document ONLY contain the stuff that is current > practice. This includes EVERYTHING required to run at least > a common class of programs - i.e., MPI_INIT or some such. > > These things should be designed to maximize ease of use and > learning and encourage portability. By this I am including > performance level portability. I would like to not have > things in the spec. that run well on some machines and > poorly on others. I would include a requirement that a > conforming implementation include either documents or > (preferably) programs from which a user could assess > the performance of the various routines. > > All the other things that are off-shoots of requirements > for odd-machines or otherwise more advanced should go in I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. Al Geist From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 12:23:03 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01995; Thu, 11 Mar 93 12:23:03 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04568; Thu, 11 Mar 93 12:19:59 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:19:57 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04559; Thu, 11 Mar 93 12:19:56 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Thu, 11 Mar 1993 12:19:53 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA21933; Thu, 11 Mar 1993 12:19:51 -0500 Date: Thu, 11 Mar 1993 12:19:51 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199303111719.AA21933@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Message Passing not for the masses Al say's I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. \begin{flame} I generally agree with Al, but not this time. While I tend to think that there should be some way to figure out how to use MPI in one sitting, I certainly do not think that we should be "maximizing ease of use" as the principle design goal. If I wanted things to be simple and slow, I'd be a Linda user. Message passing is nasty. Message passing sucks. Message passing is the way to get machines to go fast, at least for the time being. \end{flame} That being said, even if MPI turns into a huge, ugly mass of routines, it should still be possible to isolate a few of these routines and present them as the "simple subset" for people who don't need to drive their communication costs down. I don't see "limiting the size of" being the same as "making easy to use". -scott berryman From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 12:32:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02217; Thu, 11 Mar 93 12:32:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05147; Thu, 11 Mar 93 12:30:50 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:30:48 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA05132; Thu, 11 Mar 93 12:30:46 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA29711; Thu, 11 Mar 93 17:30:24 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA05324; Thu, 11 Mar 93 10:32:53 MST Date: Thu, 11 Mar 93 10:32:53 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303111732.AA05324@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu, geist@msr.epm.ornl.gov Subject: Re: Jon's suggestion > Jon writes: > > My suggestion (which I expect to be roundly rejected) is > > that the MPI document ONLY contain the stuff that is current > > practice. This includes EVERYTHING required to run at least > > a common class of programs - i.e., MPI_INIT or some such. > > > > These things should be designed to maximize ease of use and > > learning and encourage portability. By this I am including > > performance level portability. I would like to not have > > things in the spec. that run well on some machines and > > poorly on others. I would include a requirement that a > > conforming implementation include either documents or > > (preferably) programs from which a user could assess > > the performance of the various routines. > > > > All the other things that are off-shoots of requirements > > for odd-machines or otherwise more advanced should go in > > I do not reject this in fact I applaud your suggestion. > This may be the only way we can get MPI-1 off the ground. > I am a big advocate of maximizing ease of use > this is only way to get the large user base necessary > for MPI to become a standard. > > Al Geist I also agree. I think that there is significant concern regarding the near term and long term success of MPI. I think some of the concern can be aleviated if we settle on a goal that can be reached by this summer. It may be wise to set up some sort of electronic straw poll procedure to help get us past some of the issues regarding what should be in the first MPI, what are our goals for MPI-1, is there going to be MPI-N (for N>1), what features are most important for MPI-1, etc. Perhaps we can use similar rules to those agreed upon in the first meeting. You must have been in Dallas for 2 of the last three meetings (1 of the first 2, I guess). Calls for vote should be made through the committee chair (or someone appointed by the committee chair). The votes will be counted over a three day period by the comittee chair (or the designee) and reported. (Only your "last" vote counts :-). These will, of course, be straw polls and will be reported again at the beginning of the next Dallas meeting. Regards, Leslie Hart (hart@fsl.noaa.gov) P.S. We may wish to use a separate mail list for voting (mpi-vote??) which includes those eligible to vote via the voter elibibilty rules. Of course, votes for a particular topic should be clearly identified and the sense of the vote should be clear (i.e yes or no). The suggestion of electronic votes is important to me, but the mechanics is not that important. > From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 13:04:39 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03015; Thu, 11 Mar 93 13:04:39 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08235; Thu, 11 Mar 93 13:00:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 13:00:08 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA08214; Thu, 11 Mar 93 13:00:05 -0500 Received: from jadoube.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA19857 (5.65c/IDA-1.4.4 for ); Thu, 11 Mar 1993 12:00:02 -0600 From: David Levine Received: by jadoube.mcs.anl.gov (4.1/GeneV4) id AA15971; Thu, 11 Mar 93 12:00:00 CST Date: Thu, 11 Mar 93 12:00:00 CST Message-Id: <9303111800.AA15971@jadoube.mcs.anl.gov> To: berryman-harry@cs.yale.edu Cc: mpi-comm@cs.utk.edu In-Reply-To: Harry Berryman's message of Thu, 11 Mar 1993 12:19:51 -0500 <199303111719.AA21933@YOGI.NA.CS.YALE.EDU> Subject: Message Passing not for the masses X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 12:19:57 EST Date: Thu, 11 Mar 1993 12:19:51 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Al say's I do not reject this in fact I applaud your suggestion. This may be the only way we can get MPI-1 off the ground. I am a big advocate of maximizing ease of use this is only way to get the large user base necessary for MPI to become a standard. \begin{flame} : Message passing is nasty. Message passing sucks. Message passing is the way to get machines to go fast, at least for the time being. \end{flame} I'd like to add my two cents in agreement with this comment. Message passing is NOT for the masses. Its just too hard and grungy. My opinion is that message passing is the assembly language of parallel programming and it will evolve that way. I think the "large user base necessary for MPI to become a standard." will be *expert* users, vendors, and HPF compiler writers. If MPI doesn`t support what they need it will not be useful (it may still, however, become yet-another-standard.) That said, I am in agreement that there should be an easy-to-use MPI layer and a nice tutorial document for novice users, but don't handicap MPI to accomodate novice users who will migrate away from message-passing as higher-level software tools (HPF, MIMDizer, BlockComm, ....) become available. --dave levine From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 13:27:29 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA04067; Thu, 11 Mar 93 13:27:29 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09727; Thu, 11 Mar 93 13:26:20 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 13:26:19 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from msr.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09719; Thu, 11 Mar 93 13:26:17 -0500 Received: by msr.EPM.ORNL.GOV (5.67/1.34) id AA01747; Thu, 11 Mar 93 13:26:15 -0500 Date: Thu, 11 Mar 93 13:26:15 -0500 From: geist@msr.EPM.ORNL.GOV (Al Geist) Message-Id: <9303111826.AA01747@msr.EPM.ORNL.GOV> To: mpi-comm@cs.utk.edu Subject: Re: Message Passing not for the masses scott berryman flames... >even if MPI turns into a huge, ugly mass of routines, There are two problems with this argument. a. at the rate we are going we will never be able to agree on (or document) MPI so there wont be a subset. b. existing packages like NX were not designed to spare performance yet it doesn't have 4 layers and 34 different send variants. Jon and my comment reflect the concern that we will not be able to define MPI-1 in any reasonable time frame if it is a huge, ugly mass of routines, Al From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 14:23:45 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05158; Thu, 11 Mar 93 14:23:45 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12286; Thu, 11 Mar 93 14:21:53 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 14:21:52 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from NA-GW.CS.YALE.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12277; Thu, 11 Mar 93 14:21:50 -0500 Received: from YOGI.NA.CS.YALE.EDU by CASPER.NA.CS.YALE.EDU via SMTP; Thu, 11 Mar 1993 14:21:48 -0500 Received: by YOGI.NA.CS.YALE.EDU (Sendmail-5.65c/res.client.cf-3.5) id AA22157; Thu, 11 Mar 1993 14:21:47 -0500 Date: Thu, 11 Mar 1993 14:21:47 -0500 From: berryman-harry@CS.YALE.EDU (Harry Berryman) Message-Id: <199303111921.AA22157@YOGI.NA.CS.YALE.EDU> To: mpi-comm@cs.utk.edu Subject: Huge ugly mass of routines Al Giest simmers... Jon and my comment reflect the concern that we will not be able to define MPI-1 in any reasonable time frame if it is a huge, ugly mass of routines, At the risk of agreeing with you, there is a real danger at this point of the huge ugly mass of routines preventing us from do the job in the time alloted. This is a different concern than ease of use. What would you suggest eliminating to simplify the standard? -scott From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 14:37:30 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA05455; Thu, 11 Mar 93 14:37:30 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12897; Thu, 11 Mar 93 14:35:33 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 14:35:31 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from chert.CS.ORST.EDU by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA12882; Thu, 11 Mar 93 14:35:29 -0500 Received: from sycamore.CS.ORST.EDU by research.CS.ORST.EDU (4.1/1.30) id AA16681; Thu, 11 Mar 93 11:35:26 PST From: pancake@chert.CS.ORST.EDU (Cherri Pancake) Received: by sycamore.CS.ORST.EDU (4.1/CS-Client) id AA06719; Thu, 11 Mar 93 11:35:25 PST Date: Thu, 11 Mar 93 11:35:25 PST Message-Id: <9303111935.AA06719@sycamore.CS.ORST.EDU> To: mpi-comm@cs.utk.edu Subject: Response to Berryman's comments Scott says (and I love his "\begin{flame}") -- While I tend to think that there should be some way to figure out how to use MPI in one sitting, I certainly do not think that we should be "maximizing ease of use" as the principle design goal. and then even if MPI turns into a huge, ugly mass of routines, it should still be possible to isolate a few of these routines and present them as the "simple subset" for people who don't need to drive their communication costs down. I don't see "limiting the size of" being the same as "making easy to use". It's certainly try that ease-of-use does not necessarily mean a small number of routines. Ease-of-use is more dependent on: a) If there are subsets, there must be subsets that can function as a reasonal mini-language, in the sense that applications can be developed using just those routines - and getting reasonable performance. b) There must be syntactic/semantic consistency both among routines in a subset and across subset boundaries. Otherwise, users will not be able to learn subsets incrementally with any ease. c) Perhaps most importantly, the names of routines, order and type of arguments, and user-level semantics (by which I mean what can go wrong with respect to the user program) must be consistent and as simple as possible. Cherri From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 15:42:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07612; Thu, 11 Mar 93 15:42:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15959; Thu, 11 Mar 93 15:40:41 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 15:40:39 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sampson.ccsf.caltech.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA15920; Thu, 11 Mar 93 15:40:35 -0500 Received: from elephant by sampson.ccsf.caltech.edu with SMTP id AA16701 (5.65c/IDA-1.4.4 for mpi-comm@cs.utk.edu); Thu, 11 Mar 1993 12:40:32 -0800 Received: from lion.parasoft by elephant (4.1/SMI-4.1) id AA19196; Thu, 11 Mar 93 12:33:06 PST Received: by lion.parasoft (4.1/SMI-4.1) id AA00810; Thu, 11 Mar 93 12:33:21 PST Date: Thu, 11 Mar 93 12:33:21 PST From: jwf@lion.Parasoft.COM (Jon Flower) Message-Id: <9303112033.AA00810@lion.parasoft> To: mpi-comm@cs.utk.edu Subject: To fan the flames.... To add another layer of hypothetical justification to my argument .......... I think that looking at what we can achieve in the next few meetings is actually only a part of the issue. Probably a larger part of my concern is how long after we produce a draft some reasonably large percentage of the vendors will actually be supporting whatever it is we come up with. Having commitments from vendors to support MPI1 is a long way from actually having an implementation that "kicks #$%$%^$%" in our hot little hands. (Cf. the vendor "acceptance" of OSF for a good example.) My biggest fear is that if MPI-1 is a massive system then the first implementations might be a year after we finish and the first "good" ones (i.e., that actually use some of the really advanced and "neat" stuff that we put into MPI-1 just for that purposes) may take two or more years!! Is that worth it? I don't think so. Of course, I'm probably slandering the vendors and if so I apologise. Perhaps they are better off than I think they are. My impression, however, is that they are struggling to get other stuff solid which they probably would be totally justified in thinking is more important than implementing MPI. Of course this argument doesn't include third parties, either commercial or not, from implementing MPI on top of other systems and giving them to the world. This could clearly happen on a shorter time-scale. However, I expect MPI to fail if this is the best we can offer because people will still find it "convenient" to use vendor specific alternatives that have higher performance. Jon p.s., Just for the sake of it someone asked for a definition of "common practice" - a term that I've bandied about without justification. Without too much thought I think a viable definition might be to take the intersection of all the systems that are currently in use to develop application codes. This is a conceptual intersection, of course, not a strict one since all systems differ in details. This differs in my mind from what we have in the current MPI spec. which might be termed the transitive closure of all the ideas in current use!!! From owner-mpi-comm@CS.UTK.EDU Thu Mar 11 18:15:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10139; Thu, 11 Mar 93 18:15:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23334; Thu, 11 Mar 93 18:13:26 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 11 Mar 1993 18:13:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from sun2.nsfnet-relay.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA23325; Thu, 11 Mar 93 18:13:18 -0500 Via: uk.ac.southampton.ecs; Thu, 11 Mar 1993 19:07:28 +0000 Via: brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:08:41 GMT From: Ian Glendinning Received: from holt.ecs.soton.ac.uk by brewery.ecs.soton.ac.uk; Thu, 11 Mar 93 18:16:56 GMT Date: Thu, 11 Mar 93 18:16:58 GMT Message-Id: <19579.9303111816@holt.ecs.soton.ac.uk> To: mpi-comm@cs.utk.edu Subject: Re: Jon's suggestion > From: hart@gov.noaa.fsl.nipmuc (Leslie Hart) > I also agree. I think that there is significant concern regarding > the near term and long term success of MPI. I think some of the > concern can be aleviated if we settle on a goal that can be reached > by this summer. It may be wise to set up some sort of electronic > straw poll procedure to help get us past some of the issues regarding > what should be in the first MPI, what are our goals for MPI-1, is there > going to be MPI-N (for N>1), what features are most important for MPI-1, etc. I agree. Just when I thought things were converging there's been a flurry of alternative proposals. For example, can we please agree *not* to address concurrent I/O in MPI1, else we'll never get finished within the allotted timescale. On the other hand, I do think that we should decide to have an MPI2, and that that would be the proper way to deal with such issues. I don't think concurrent I/O is an intractible problem, its just that to do a reasonable job on will require more time than we've got right now. Another case in point is dynamic vs static process creation. My vote is for static now, dynamic later. If we can agree that there will definitely be an MPI2, then maybe we can keep everyone happy in the sense that their favourite features *will* all be included, but in a staged fashion. Ian -- I.Glendinning@ecs.soton.ac.uk Ian Glendinning Tel: +44 703 593368 Dept of Electronics and Computer Science Fax: +44 703 593045 University of Southampton SO9 5NH England From owner-mpi-comm@CS.UTK.EDU Fri Mar 12 15:14:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29560; Fri, 12 Mar 93 15:14:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20854; Fri, 12 Mar 93 15:12:27 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Mar 1993 15:12:25 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ocfmail.ocf.llnl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA20838; Fri, 12 Mar 93 15:12:23 -0500 Received: from [134.9.250.226] (nessett.ocf.llnl.gov) by ocfmail.ocf.llnl.gov (4.1/SMI-4.0) id AA00567; Fri, 12 Mar 93 12:12:15 PST Message-Id: <9303122012.AA00567@ocfmail.ocf.llnl.gov> Date: Fri, 12 Mar 1993 12:20:03 -0800 To: mpi-comm@cs.utk.edu From: nessett@ocfmail.ocf.llnl.gov (Dan Nessett) X-Sender: nessett@ocfmail.llnl.gov Subject: Buffer descriptors and the simple program Cc: rathkopf@llnl.gov, mirin@llnl.gov, jlowens@llnl.gov, geltgroth@llnl.gov, tsw@llnl.gov, seager@llnl.gov, trj@llnl.gov, lstanberry@llnl.gov, bolstad@llnl.gov, zosel@llnl.gov, jbrown@nersc.gov, frobose@llnl.gov, hendrickson1@llnl.gov, batkinson@llnl.gov I met with a number of application and system programmers here at the Lab to get some feedback on the decisions made at the last MPI meeting. One aspect that they felt uncomfortable with was being forced to use the buffer descriptor routines (i.e., bd_create, bd_contiguous, bd_stride and bd_free) in order to just send a simple message. The application programmers want both simple send and receive primitives as well as more general primitives based on buffer descriptors. For example, the Parallel Monte Carlo and Atmospheric Chemistry codes they are working on would benefit from the more general buffer descriptor based primitives, while the Atmospheric Dynamics code uses only simple send and receive. The programmers that use only a single contiguous buffer for their codes would be upset if MPI forced them to make two bd calls to set up the descriptor and then another bd call to free it. (BTW the last call could be eliminated if there were some way to "clear" a buffer descriptor so it could be reused). Dan Nessett From owner-mpi-comm@CS.UTK.EDU Fri Mar 12 15:52:31 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA00970; Fri, 12 Mar 93 15:52:31 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22518; Fri, 12 Mar 93 15:51:15 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 12 Mar 1993 15:51:14 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22510; Fri, 12 Mar 93 15:51:09 -0500 Received: from godzilla.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA07519 (5.65c/IDA-1.4.4 for ); Fri, 12 Mar 1993 14:51:06 -0600 From: William Gropp Received: by godzilla.mcs.anl.gov (4.1/GeneV4) id AA21843; Fri, 12 Mar 93 14:51:04 CST Date: Fri, 12 Mar 93 14:51:04 CST Message-Id: <9303122051.AA21843@godzilla.mcs.anl.gov> To: nessett@ocfmail.ocf.llnl.gov Cc: mpi-comm@cs.utk.edu, rathkopf@llnl.gov, mirin@llnl.gov, jlowens@llnl.gov, geltgroth@llnl.gov, tsw@llnl.gov, seager@llnl.gov, trj@llnl.gov, lstanberry@llnl.gov, bolstad@llnl.gov, zosel@llnl.gov, jbrown@nersc.gov, frobose@llnl.gov, hendrickson1@llnl.gov, batkinson@llnl.gov In-Reply-To: Dan Nessett's message of Fri, 12 Mar 1993 12:20:03 -0800 <9303122012.AA00567@ocfmail.ocf.llnl.gov> Subject: Buffer descriptors and the simple program [ message about having to use the "bd_xxx" routines ] This is what the level 2 (or 3, depending on where you draw the line) routines are for. Level 3 contains the "simple snd and receive" with contiguous buffers, while level 1 has everything at the cost of greater complexity. Bill From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 15:54:42 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29714; Tue, 16 Mar 93 15:54:42 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19013; Tue, 16 Mar 93 15:51:37 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 15:51:36 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA19005; Tue, 16 Mar 93 15:51:34 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19519; Tue, 16 Mar 1993 15:51:33 -0500 Date: Tue, 16 Mar 1993 15:51:33 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303162051.AA19519@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: MPI test implementation An Argonne report by William Gropp and Ewing Lusk dated December 1992 ("A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47) has recently been distributed. Please note that the MPI1 routines referred to in this document are those in a draft paper that was circulated as a discussion document. Many of these routines differ from those in the final version of MPI1, or are omitted. The final version of the MPI1 draft standard may be obtained by sending the email message "send mpi1.ps from mpi" to netlib@ornl.gov. For those of you interested in correctly referencing this the bibtex item is as follows: @techreport{MPI1, author = "J. J. Dongarra and R. Hempel and A. J. G. Hey and D. W. Walker", title = "A Proposal for a User-Level, Message Passing Interface in a Distributed Memory Environment", institution = "Oak Ridge National Laboratory", number = "{TM}-12231", month = "March", year = 1993} Regards, David Walker. From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 16:43:35 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA01882; Tue, 16 Mar 93 16:43:35 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22479; Tue, 16 Mar 93 16:41:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 16:41:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from antares.mcs.anl.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22461; Tue, 16 Mar 93 16:41:41 -0500 Received: from donner.mcs.anl.gov by antares.mcs.anl.gov with SMTP id AA13171 (5.65c/IDA-1.4.4 for ); Tue, 16 Mar 1993 15:41:34 -0600 Message-Id: <199303162141.AA13171@antares.mcs.anl.gov> To: walker@rios2.epm.ornl.gov (David Walker) Cc: mpi-comm@cs.utk.edu Subject: Re: MPI test implementation In-Reply-To: Your message of "Tue, 16 Mar 1993 15:51:33 EST." <9303162051.AA19519@rios2.epm.ornl.gov> Date: Tue, 16 Mar 1993 15:41:32 -0600 From: Rusty Lusk Thanks for setting the record straight. That Tech. Report went into the Argonne mill in early December, and we had no idea that it would take it this long to be distributed and therefore how out-of-date it would be when it finally did get mailed. - Rusty From owner-mpi-comm@CS.UTK.EDU Tue Mar 16 20:54:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA06431; Tue, 16 Mar 93 20:54:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09400; Tue, 16 Mar 93 20:53:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 16 Mar 1993 20:53:01 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA09390; Tue, 16 Mar 93 20:53:00 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Tue, 16 Mar 93 17:52 PST Message-Id: Date: Tue, 16 Mar 93 17:52 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: PostScript version of the current MPI Draft Document A (compressed) PostScript version of the current MPI Draft Document is available via anonyomous ftp at cse.ogi.edu: ftp cse.ogi.edu cd pub/otto binary get draft.ps.Z quit This should get you a copy. You then need to uncompress it. All the source is there also if you are interested. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 08:39:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA10600; Thu, 18 Mar 93 08:39:18 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04620; Thu, 18 Mar 93 08:36:29 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 08:36:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA04612; Thu, 18 Mar 93 08:36:21 -0500 Date: Thu, 18 Mar 93 13:36:16 GMT Message-Id: <9166.9303181336@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: MINUTES! To: mpi-comm@cs.utk.edu In-Reply-To: David Walker's message of Wed, 17 Mar 1993 09:57:50 -0500 Reply-To: lyndon@epcc.ed.ac.uk Hi all On Thursday of last week I enquired about the minutes of the last meeting. Dave Walker writes: > The minutes of the last meeting in February are not yet available > electronically, but should be soon. I observe that subcommittee chairs have already circulated revised "working documents" for subcommittees. The minutes of previous meeting are the primary reference for results of subcommittee straw polls which must surely be the primary determinants of working document revisions. It is now past time when the minutes were essential. Please, can these be distributed. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 11:18:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14049; Thu, 18 Mar 93 11:18:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11559; Thu, 18 Mar 93 11:17:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 11:16:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA11550; Thu, 18 Mar 93 11:16:51 -0500 Date: Thu, 18 Mar 93 16:16:47 GMT Message-Id: <9298.9303181616@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: AGENDA! To: walker@rios2.epm.ornl.gov (David Walker) In-Reply-To: David Walker's message of Thu, 18 Mar 1993 10:48:16 -0500 Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu David writes: > > I don't think Lyndon's suggestion to make the timetable session the first of > the meeting is a good idea. How can a timetable be set if we don't know > what's to be included, > and if we don't know how much of the current proposals > are acceptable. > I see your point on the matter of establishing a firm timetable. It seems to me, and there has been a fair bit of traffic on it, that we need to establish boundaries for what will be in MPI. It seems to me that this needed to be the first thing to thrash out at the next meeting. I agree with you that timetables should be sorted out later. > We could move the session on contexts, groups etc. up front I guess. This seems to mean that the context subcommittee has to come to the "report and proposal to whole committee" stage without the opportunity to meet. My understanding was that we had intended to reach that stage at the next meeting, after a subcommittee meeting. Personally I can handle doing this by email over the next 10 days, but I'm not sure that the whole of the subcommittee can. Thanks for the comments, and pointing out my error in supposing that the timetable could be determined at the beginning of the meeting. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 12:54:40 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA15727; Thu, 18 Mar 93 12:54:40 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16896; Thu, 18 Mar 93 12:50:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 12:50:43 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA16880; Thu, 18 Mar 93 12:50:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA14802; Thu, 18 Mar 93 17:44:46 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA20644; Thu, 18 Mar 93 10:43:30 MST Date: Thu, 18 Mar 93 10:43:30 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303181743.AA20644@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: MPI Priorities and Scope: Opinion Survey Cc: jwf@lion.parasoft.com, lederman@b125.super.org, igl@ecs.soton.ac.uk, lyndon@epcc.ed.ac.uk Hi all, Steve Huss-Lederman writes: > I would like to propose that we have a meeting of all participants at > the start of the next meeting to settle the issue of what MPI-1 will > cover. Several mail messages have expressed concern that we cannot > conclude by June if we try to incorporate everything that everyone > wants. Since the scope and time frame will effect most of the other > discussions I think it would be best to begin with this. My only > concern is that it will take too long to decide the global picture. > I would hope it could be done in 1-2 hours and help to refocus the > group. I fully support this idea. Here's a proposal for an opinion survey that could shorten and focus the discussion. This proposal originated with Jon Flower and has had input from Ian Glendinning, Steve Huss-Lederman, Leslie Hart, myself, and others. If you think this is a good idea, please complete the survey and return to me at: hender@fsl.noaa.gov If you think this is a bad idea, please say so! We will collect responses and distribute a summary via email before the next meeting if there is sufficient interest. Tom Henderson ----------------------------------------------------------------------------- MPI PRIORITIES OPINION SURVEY Several MPI committee members have expressed concern about the current and future directions of MPI. The MPI committee began discussion using an existing document (Dongarra et al) as a starting point. This document contains background rationale and a statement of direction for the MPI standardization effort. Since these issues were consistent with the original MPI proposal, they have not recently been discussed in detail by the MPIF. The current MPI proposals are very different from the original MPI document. As a result, we feel that the original rationale and direction need to be re-examined. In particular we need to prioritize and limit the issues that MPI will try to resolve so that o the initial MPI standard document ("MPI-1") is delivered in a timely fashion and o the functions that it defines can (and will) be implemented on major platforms within a reasonable time frame. To get discussion of these issues started, we propose an informal, email "opinion survey". The objective of the survey is to find out which MPI features are most important. High-priority features would be included in MPI-1. Lower-priority features might be deferred to a later version of MPI. Some features might never be included. The compiled results of this survey could be used as a basis for discussion at the next MPI meeting. *** This survey is in no way binding and is a completely unofficial "straw poll" arranged by its authors for their own personal information. It is not intended to distract attention from the current proposals. Any data collected will be made available to the group at large if requested. *** Following is a ballot containing possible MPI features and goals. Please rate each item (or sub-item) using the following rating scheme: 0 This feature should not appear in any version of MPI ever. 1-2 MPIF should consider including this feature in some future version of MPI (after MPI-1). 3-4 This feature should be included as an Annex to MPI-1 for possible inclusion in a future version of MPI. 5-6 This feature should be included as an Annex to MPI-1 for definite inclusion in a future version of MPI. 7-10 This feature should be included in MPI-1. In this context an "Annex" is a part of the standard document that will not be required by a conforming implementation of MPI. Rather it should be taken as a recommendation by the MPIF to vendors for the way in which particular features should be implemented, if they are implemented at all. This way features that are not usefully implementable on particular hardware can be eliminated while machines that do support these features do so in a portable manner. When replying to the survey, please use the response form attached to the end of this message. (This will make compilation of the results a lot less painful!) :-) +-------------------------------------------------------+ | POSSIBLE FEATURES FOR INITIAL AND FUTURE MPI VERSIONS | +-------------------------------------------------------+ Ordering is for convenience only. This list is not meant to suggest any priority. GENERAL FEATURES 1) Routines for static process creation: A) Same program in every node. B) Host-node programming model in which there is one "special node". C) Static model but with potentially different programs in different groups of nodes. D) Static model with a way to create more than one process per processor. (Heavy duty processes, not threads). 2) Routines for identification of statically created processes. 3) Associated environmental inquiry routines. 4) Some way of avoiding message tag conflicts between independently developed libraries and user code (contexts). 5) Heterogeneous communication. A) All communication routines should support heterogeneous communication. B) A special set of communication routines should support heterogeneous communication. 6) Creation/destruction of "static" groups (processes cannot join/leave an existing group). Groups are used to limit the scope of collective operations. A) No logical process topology is associated with a group. B) Grid/toroidal process topology is associated with a group at the time the group is created. (The default "ALL" group has a default topology.) C) General graph topology is also supported. D) Provide access to all relevant information (via topological environmental inquiry routines) to allow users to define custom topologies. 7) Support for communication between processes in different groups without reference to the "ALL" group. 8) Dynamic groups (processes can join/leave an existing group while the total number of processes remains static.) 9) Dynamic process creation and identification. 10) Additional environmental inquiry and "setting" routines for performance optimization. A) Inquire/suggest system buffering resource allocation. B) Inquire relative node performance. C) Inquire relative interconnect performance. 11) Termination of programs. A) Kill all others nodes. B) Wait for all the other nodes to complete. C) Automatically de-allocate resources. 12) Standard organization. A) "Multi-level" organization of routines (users may use simple features without having to understand all of MPI). B) Simple "core" standard with (optional) Annexes. C) Simple "core" standard without Annexes. D) Should an MPI "subset" be defined that can be quickly implemented as a layer on top of existing systems? (This question was inspired by section 1.5 of Bill and Rusty's "A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47.) 13) Should there be future version(s) of MPI (MPI-2, etc.) (yes/no)? POINT-TO-POINT COMMUNICATION 14) Blocking send and receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). 15) Non-blocking receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 16) Non-blocking send. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 17) Low-level functionality (from Marc Snir's proposal) visible to users. A) Initialize. B) Start. C) Complete. D) Free. 18) "SECURE" versions of send and receive as proposed by Lyndon Clarke on 3/3/93. (Correct programs using SECURE routines are guaranteed to work regardless of system buffering resources.) 19) Ready-send. 20) Ready-receive. 21) Message handlers (a la iNTEL hrecv, TMC active messages, Express exhandle). A) Simple definition sufficient to support anticipated MPI requirements and make a self-consistent system. B) More advanced routines for remote procedure calls, etc... 22) Additional point-to-point primitives. A) Exchange between two processors. B) "Shift-exchange": read from one, send to another (superset of "A"). C) Contiguous data. D) Strided data. E) Buffer-descriptor (multiple data locations, multiple data types). F) Non-blocking versions? G) Check for completion (status()). H) Wait for completion (wait()). I) Cancel operation (cancel()). COLLECTIVE OPERATIONS 23) Blocking collective communication. A) Broadcast. B) Concatenate. C) Barrier. D) Sum. E) Extrema (max, min). F) Average. G) Sum of squared differences (error). H) Should strided data be allowed for these operations? I) Should buffer-descriptor be allowed for these operations? 24) "User-defined" blocking collective operations (Express "excombine()" style). 25) Non-blocking collective communication, collective operations, and user-defined operations (if this is desired, detailed ranking can be done later). 26) If any non-blocking collective operations are supported, what should completion/cancel options be? A) Check for completion (status()). B) Wait for completion (wait()). C) "Multiple cancel" operation (cancel()). 27) Should status()/wait()/cancel() syntax be consistent across all types of non-blocking operations? OTHER COMMUNICATION OPERATIONS 28) Probe (and associated "precv()" routines, if necessary). I/O 29) Standard I/O is supported in the standard fashion for the language on one process (no file I/O). This process can be identified by an environmental inquiry routine. (This is the current proposal.) 30) Serial I/O is supported in the standard fashion for the language on one process. This process can be identified by an environmental inquiry routine. (Rik Littlefield's proposal.) 31) Multiplexed I/O similar to that currently provided in systems such as CMMD (TMC), VERTEX (nCUBE) and Express (ParaSoft). Basically a way of defining the behavior of multi-node I/O statements which interact with a sequential device. No special considerations for parallel I/O channels. 32) Blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. 33) Non-blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. LANGUAGE BINDINGS 34) Ada 35) C 36) C++ 37) Fortran 77 38) Fortran 90 39) Other IMPLEMENTATION PERIOD There will be a delay between the generation of the standard and its implementation on various platforms. If this delay is too long users will continue to rely more and more heavily on vendor routines and specialized calls and MPI will become less and less attractive, particularly if new technology such as Fortran 90/HPF becomes effective. 40) What is the maximum delay that you think MPI can stand and still succeed (please feel free to comment if you wish)? ---------------------------------------------------------------------------- SURVEY RESPONSE FORM 1) A) B) C) D) 2) 3) 4) 5) A) B) 6) A) B) C) D) 7) 8) 9) 10) A) B) C) 11) A) B) C) 12) A) B) C) D) 13) (yes/no) 14) A) B) C) 15) A) B) C) D) E) F) 16) A) B) C) D) E) F) 17) A) B) C) D) 18) 19) 20) 21) A) B) 22) A) B) C) D) E) F) G) H) I) 23) A) B) C) D) E) F) G) H) I) 24) 25) 26) A) B) C) 27) 28) 29) 30) 31) 32) 33) 34) 35) 36) 37) 38) 39) 40) (free form): Comments? From owner-mpi-comm@CS.UTK.EDU Thu Mar 18 14:28:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17761; Thu, 18 Mar 93 14:28:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22028; Thu, 18 Mar 93 14:22:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 18 Mar 1993 14:22:45 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA22020; Thu, 18 Mar 93 14:22:43 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA23272; Thu, 18 Mar 93 11:22:16 PST Date: Thu, 18 Mar 93 11:22:16 PST Message-Id: <9303181922.AA23272@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA10070; Thu, 18 Mar 93 11:22:13 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: lyndon@epcc.ed.ac.uk Cc: mpi-comm@cs.utk.edu Subject: Re: MINUTES! In-Reply-To: <9166.9303181336@subnode.epcc.ed.ac.uk> References: <9166.9303181336@subnode.epcc.ed.ac.uk> Reply-To: knighten@SSD.intel.com (Bob Knighten) Yes, the minutes are late and yes, they will be available soon. Rusty has been traveling extensively since the meeting and now that I am supposed to finish them I have had some personal emergencies and my company has decided there are more important ways to spend my time. That is just an explanation of the delay, I am well aware of the importance of the minutes and will have them out shortly. -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:48:01 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17196; Thu, 25 Mar 93 10:48:01 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21805; Thu, 25 Mar 93 10:46:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA21766; Thu, 25 Mar 93 10:45:43 -0500 Date: Thu, 25 Mar 93 15:44:59 GMT Message-Id: <18119.9303251544@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 12:41:24 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA21413; Thu, 25 Mar 93 12:41:24 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26938; Thu, 25 Mar 93 12:40:11 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 12:40:10 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26850; Thu, 25 Mar 93 12:39:35 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07566; Thu, 25 Mar 93 11:33:39 CST Date: Thu, 25 Mar 93 11:33:39 CST From: Tony Skjellum Message-Id: <9303251733.AA07566@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Thank you for this additional work, Lyndon. Why don't we include this as the preface to our proposal? It looks good, but I will have to read it in detail, before rendering my complete view on this matter. Who are these other, new cc people... - Tony ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 14:01:25 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA23412; Thu, 25 Mar 93 14:01:25 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00787; Thu, 25 Mar 93 13:59:45 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 13:59:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA00747; Thu, 25 Mar 93 13:58:34 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA07654; Thu, 25 Mar 93 12:52:38 CST Date: Thu, 25 Mar 93 12:52:38 CST From: Tony Skjellum Message-Id: <9303251852.AA07654@Aurora.CS.MsState.Edu> To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu, lyndon@epcc.ed.ac.uk Subject: Re: Helpful Summary of Contexts Proposals Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Lyndon, as you know I was immensely under the gun yesterday, and my 9-hour trip was quite an ordeal, but I did get the sub-committee's proposal in on time, and it is basically quite good. I do feel that it is appropriate to include comparisons and even criticisms in the conclusions, but it was not done evenly, since mine was last. To this extent, I see why you did not like that. In same light, I can see why we should drop your name and Rik's name from proposal III, as this might unfairly indicate your support for proposal III (which is admittedly, an extended version of the programming model I support most, and is what Zipcode does, plus some additions). Please restrain yourself from completely dominating this subcommittee, because of the remarkable amount of time you are able to devote to it. To be fair, I cannot devote more than 2hr/day, and you are able to devote something like 12hr/day to it. People at the SIAM meeting were overwhelmed by the volume of mail you generate on different topics. You are a star performer, but you are also very demanding, and I must say that I have had some aggravations in this last weeks, but not only from you. I am trying not to push my personal viewpoint too hard. Please work from a viewpoint of cooperation with me, as we move towards the meeting next week. In this vein, see next paragraph. I will read your summary, but I already agree (note: agree) to drop all conclusion comments from all three subproposals that are "judgemental" (a loaded word). Since we are making a report out of this, the comparison should go in as latex. I will give this to Otto today (I will give it to him). Do you want me to latex your contribution, or will you (it is not too late there). Tell me. Ala Frankie & Johnny, I have to hang up now, since there is more mail from you to read :-)... I will not respond to that which is outdated by this letter. Please work on a latex version of your excellent new contribution, which also makes our joint report much better. I thank you. PS For rest of committee, the new contribution by Lyndon does not invalidate yesterday's tripartite proposal, it will merely be included in it. ----- Begin Included Message ----- From owner-mpi-comm@CS.UTK.EDU Thu Mar 25 10:07:51 1993 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Mar 1993 10:45:59 EST Date: Thu, 25 Mar 93 15:44:59 GMT From: L J Clarke Subject: Helpful Summary of Contexts Proposals To: mpi-comm@cs.utk.edu, mpi-context@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: tony@aurora.cs.msstate.edu, nbm@castle.ed.ac.uk, bobf@epcc.ed.ac.uk Content-Length: 8050 Dear MPI Colleagues I imagine that many of you have started or are about to start reading the three contexts subcommittee proposals. We have prepared a comparative, and non judgmental, summary of the three proposals which may be of some assistance to MPI. The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. Hopefully the summary will: (a) help us to discuss the important differnces between the proposals and make agreements on how we should proceed with respect to those issues; (b) help us to isolate the separable points and make separate agreements on those issues. I hope that the summary is both accurate and complete. Please make corrections and additions if you discover such. I apologise in advance for my errors, which are surely inevitable. Best Wishes Lyndon ---------------------------------------------------------------------- Summary of context subcommittee proposals ***************************************** The three proposals in the context subcommittee share common features, and have differences both in concept and detail. Two of these proposals contain features which are "separable" and could equally appear as components of one or more other proposals. This summary identifies feature of proposals as: Common Features; Separable Features; Concept Differences; Detail Differences. Common Features =============== 1. Process group management --------------------------- In each proposal groups are created dynamically and have static membership. In each proposal a group can be created as a partition of an existing group and as a permutation of an existing group. Each proposal allows (or suggests) that a group can be created as an explicit list of processes. 2. Provision for point-to-point communication within group. ----------------------------------------------------------- In each proposal point-to-point communcation of scope closed within a group can be expressed in terms of a reference to a group coupled with a process rank within the group. 3. Provision for collective communication within group. ------------------------------------------------------ In each proposal collective communication of scope closed within a group can be expressed in terms of a reference to a group. 4. Opacity of group and process description. -------------------------------------------- In each proposal the description of groups and processes is opaque. Groups and processes are referred to by a handle like object. Separable Features ================== 1. Tag usage in point-to-point communication. --------------------------------------------- Proposal III describes tag selection for Receive in a two-integer form. Proposals I and VII say nothing about tag usage. This feature can be placed in all Proposals I, III and VII. [Historical note: Tony did say to methat this would appear as an appendix with our mutual recognition that it can equally appear as a feature of any of the proposals.] 2. Tag usage in collective communication. ----------------------------------------- Proposal III suggests that tag should be used as an argument to collective communication where this will assist debugging. Proposals I and VII say nothing about usage. This feature can be placed in all Proposals I, III and VII. 3. Context or Group cache ------------------------- Proposal VII decribes a cache facility associated with contexts and groups. Proposal III describes a similar cache facility associated with groups. This feature can be placed in all Proposals I, III and VII. 4. Opaque object (descriptor) transmission. ------------------------------------------- Proposal VII suggests that opaque object transmission can be provided by integration with transmission of typed data. Proposal III suggests that opaque transmission is provided by a mechanism for flattening a descriptor into a memory buffer. These are just details of different ways of providing the feature. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. 5. Context registry. -------------------- Proposal III describes a context name registry service. Proposal VII indicates that such a service would be useful. This feature can be placed in Proposals III and VII. This feature cannot be placed in Proposal I. Concept Differences. ==================== 1. Concept of CONTEXT and GROUP ------------------------------- In Proposal I CONTEXT and GROUP are identical concepts and are not distinguished. In Proposal III CONTEXT is a lower degree concept that GROUP. The GROUP concept inherits aspects of the CONTEXT concept. In Proposal VII CONTEXT is a higher concept than GROUP. The CONTEXT concept inherits aspects of the GROUP concept. 2. Scope of point-to-point communication. ----------------------------------------- In Proposal I the scope of point-to-point communication is limited to the CONTEXT. Processes which are members of distinct groups can only communicate through a common ancestor group. In Proposals III and VII the scope of point-to-point communciation is not limited. Processes which are members of distinct groups can communicate without reference to a common ancestor group. 3. Transmission of group or context. ------------------------------------ In Proposal I the CONTEXT cannot be transmitted from one process to another. In Proposals VII and III both CONTEXT and GROUP can be transmitted from one process to another. In Proposal VII PROCESS can alo be transmitted (Proposal III suggests such but makes no specific provision, presumably a small oversight?) Detail differences. =================== 1. Manifestation of context --------------------------- In Proposals I and VII context is an opaque object. In Proposal III context is an integer(?). 2. Deletion of group. --------------------- In Proposals VII and III groups can be deleted. In Proposal I there is no provision for group deletion (possibly a small oversight?). 3. Duplication of group. ------------------------ In Proposals I and III a group there is explicit provision for duplication of an existing group to form a new (distinct, homomorphic) group. In Proposal VII there is no such provision as similar funtionality is provided by the context (although the provision for group partition, permutation and definition can be used to create a snapshot copy of a group). 4. Global shared variables. --------------------------- Proposals I and VII do not require global shared variables. Proposal III requires a global shared variable (which can be implemented as such or of course in the traditional approach as a global service process.) 3. Process identifier addressed communication. ---------------------------------------------- Proposal I does not make provision for process identifier addressed communication. Proposal III makes provision for process identifier addressed communication within multiple distinct tag spaces. Proposal VII makes provision for process identifier addressed communication within a single distinct tag space. 5. Inter-group communication. ----------------------------- Proposal I does not provide inter-group communication as it limits the scope of point-to-point communication to be closed within a group. Proposal VII provides inter-group communication in an addressing form specified by sender (receiver) group, receiver (sender) group and sender (receiver) rank. Proposal III provides inter-group communication as process identifier addressed communication. ---------------------------------------------------------------------- /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Fri Mar 26 18:57:02 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28001; Fri, 26 Mar 93 18:57:02 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17683; Fri, 26 Mar 93 18:54:46 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 26 Mar 1993 18:54:44 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from iliamna.cse.ogi.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17674; Fri, 26 Mar 93 18:54:43 -0500 Received: by iliamna.cse.ogi.edu (/\==/\ Smail3.1.25.1 #25.15) id ; Fri, 26 Mar 93 15:54 PST Message-Id: Date: Fri, 26 Mar 93 15:54 PST From: otto@iliamna.cse.ogi.edu (Steve Otto) To: mpi-comm@cs.utk.edu Subject: Draft available on netlib A PostScript version of the current MPI Draft Document is available on netlib. It is called draft326.ps.z.uu and is a postscript, compressed, uuencoded file. The size is 1830521 bytes. send draft326.ps.z.uu from mpi will get the file, or you may want to pull it over using xnetlib. --Steve Otto From owner-mpi-comm@CS.UTK.EDU Fri Mar 26 19:42:13 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA28702; Fri, 26 Mar 93 19:42:13 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19291; Fri, 26 Mar 93 19:40:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 26 Mar 1993 19:40:33 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from wk46.nas.nasa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19262; Fri, 26 Mar 93 19:40:26 -0500 Received: by wk46.nas.nasa.gov (5.61/1.34) id AA15348; Fri, 26 Mar 93 16:40:19 -0800 Date: Fri, 26 Mar 93 16:40:19 -0800 From: barszcz@nas.nasa.gov (Eric Barszcz) Message-Id: <9303270040.AA15348@wk46.nas.nasa.gov> To: mpi-comm@cs.utk.edu Subject: Multidisciplinary Position Paper %!PS-Adobe-2.0 %%Creator: dvips, version 5.4 (C) 1986-90 Radical Eye Software %%Pages: 10 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR}B /@letter{/vsize 10 N}B /@landscape{ /isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{/vsize 15.5531 N }B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0 ]N /df{/sf 1 N /fntrx FMat N df-tail}B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0 ]N df-tail}B /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[ }B /E{pop nn dup definefont setfont}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ctr 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}imagemask restore}B /D{ /cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}B /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}B /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}B /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 8 string N /v{/ruley X /rulex X V}B /V{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}B /a{moveto}B /delta 0 N /tail{ dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{S p tail}B /c{-4 M}B /d{ -3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t {p 4 w}B /w{0 rmoveto}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 300 300 @start /Fa 34 122 df45 D<000FC000106000603800801800801C01001C02201E02101E04101E04101E04101E08203C0820 3C0840380840780880F00700E00001C000030000060000180000200000C0000100000200000400 100800301000202000605F80C063FFC040FF80807F00801E0017227CA019>50 D<000FC000386000703000E03001C0380380380780380700380F00380F00380F00381E00781E00 781E00781E00F81E00F01C00F00E01F00E02F00605E00309E001F1E00003C00003C00003800007 00000700600E00F00C00F01800E0300080600041C0003F000015227BA019>57 D<0000030000000300000007000000070000000F0000000F0000001F0000002F0000002F000000 4F0000004F80000087800000878000010780000207800002078000040780000407800008078000 080780001007800030078000200780007FFF80004007C0008007C0008003C0010003C0030003C0 020003C0040003C0040003C00C0003C03C0007C0FF003FFC1E237DA224>65 D<00007F00800003808100000E00630000380027000070001F0000E0000E0001C0000E00038000 0E000700000E000F000004000E000004001E000004003C000004003C0000080078000000007800 0000007800000000F000000000F000000000F000000000F000000000F000000000E000000000E0 00002000E000002000E000004000E000004000F000008000700000800070000100003800020000 18000400001C0008000006003000000381C0000000FE000000212479A223>67 D<00FFFFFF000F000F000F0003000F0003001E0003001E0003001E0002001E0002003C0002003C 0002003C0102003C010000780200007802000078060000780E0000FFFC0000F00C0000F00C0000 F00C0001E0080001E0080001E0080001E0000003C0000003C0000003C0000003C0000007800000 0780000007800000078000000F800000FFFC000020227DA120>70 D<00007F0080000380810000 0E00630000380027000070001F0000E0000E0001C0000E000380000E000700000E000F00000400 0E000004001E000004003C000004003C00000800780000000078000000007800000000F0000000 00F000000000F000000000F000000000F0003FFC00E00001E000E00001E000E00001E000E00003 C000E00003C000F00003C000700003C0007000078000380007800018000F80001C001380000600 2300000381C1000000FE000000212479A226>I<00FFF87FFC000F000780000F000780000F0007 80001E000F00001E000F00001E000F00001E000F00003C001E00003C001E00003C001E00003C00 1E000078003C000078003C000078003C000078003C0000FFFFF80000F000780000F000780000F0 00780001E000F00001E000F00001E000F00001E000F00003C001E00003C001E00003C001E00003 C001E000078003C000078003C000078003C000078003C0000F8007C000FFF87FFC0026227DA124 >I<00FFF8000F00000F00000F00001E00001E00001E00001E00003C00003C00003C00003C0000 780000780000780000780000F00000F00000F00000F00001E00001E00001E00001E00003C00003 C00003C00003C0000780000780000780000780000F8000FFF80015227DA113>I<0007FFC00000 3C0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F00000 00F0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C00000078000 000780000007800000078000000F0000000F0000380F0000780F0000F81E0000F81E0000F03C00 00403800004070000021E000001F8000001A237CA11A>I<00FF800007FC000F80000F80000F80 001780000F80001780001780002F000013C0002F000013C0004F000013C0008F000023C0009E00 0023C0011E000023C0011E000023C0021E000043C0043C000043C0043C000043C0083C000041E0 083C000081E01078000081E02078000081E02078000081E04078000101E040F0000101E080F000 0101E100F0000101E100F0000200F201E0000200F201E0000200F401E0000200F801E0000400F8 03C0000400F003C0000400F003C0000C00E003C0001E00C007C000FFC0C07FFC002E227DA12C> 77 D<00FFFFE0000F0038000F001E000F000E001E0007001E0007001E0007001E0007003C000F 003C000F003C000F003C001E0078001E0078003C00780078007800E000F003C000FFFE0000F000 0000F0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C000000780 00000780000007800000078000000F800000FFF8000020227DA121>80 D<0001F020000E0C4000 1802C0003001C0006001C000E0018000C0018001C0018001C0018003C0010003C0010003C00000 03C0000003E0000001F8000001FF000000FFE000007FF000001FF8000003FC0000007C0000003C 0000001E0000001E0000001E0020001C0020001C0020001C002000180060003800600030007000 60007000C000C8018000C607000081FC00001B247DA21B>83 D<1FFFFFF81E03C0381803C01830 03C01820078018200780184007801040078010400F0010800F0010800F0010000F0000001E0000 001E0000001E0000001E0000003C0000003C0000003C0000003C00000078000000780000007800 000078000000F0000000F0000000F0000000F0000001E0000001E0000001E0000001E0000003E0 0000FFFF00001D2277A123>I<01020408102020404080B8FCFCF870080F76A20F>96 D<00F8C00185C00705C00E03800E03801C03803C0380380700780700780700780700F00E00F00E 00F00E00F00E10F01C20701C20703C20305C40308C400F078014157B9419>I<03C03F80038003 80038007000700070007000E000E000E000E001C001CF81D0C1E0E3C0638073807380F700F700F 700F700FE01EE01EE01EE03CE038E038607060E031C01F0010237BA216>I<007E0001C1000301 800703800E07801C07803C0000380000780000780000780000F00000F00000F00000F00000F001 00700100700200300C001830000FC00011157B9416>I<00003C0003F800003800003800003800 00700000700000700000700000E00000E00000E00000E00001C000F9C00185C00705C00E03800E 03801C03803C0380380700780700780700780700F00E00F00E00F00E00F00E10F01C20701C2070 3C20305C40308C400F078016237BA219>I<00F803840E021C023C0238027804F018FFE0F000F0 00E000E000E000E000E002E0026004701830600F800F157A9416>I<00003E0000470000CF0001 8F000186000380000380000380000700000700000700000700000700000E0000FFF0000E00000E 00000E00001C00001C00001C00001C00001C000038000038000038000038000038000070000070 0000700000700000700000E00000E00000E00000E00000C00001C00001C000718000F18000F300 006200003C0000182D82A20F>I<001F180030B800E0B801C07001C0700380700780700700E00F 00E00F00E00F00E01E01C01E01C01E01C01E01C01E03800E03800E0780060B8006170001E70000 0700000700000E00000E00000E00701C00F01800F0300060E0003F8000151F7E9416>I<00F000 0FE00000E00000E00000E00001C00001C00001C00001C000038000038000038000038000070000 071F0007218007C0C00F00E00F00E00E00E00E00E01C01C01C01C01C01C01C01C0380380380380 380380380704700708700E08700E10700610E006206003C016237DA219>I<00C001E001C001C0 000000000000000000000000000000001C002300430043008700870087000E000E001C001C001C 00380038003840708070807080710032001C000C217BA00F>I<01E01FC001C001C001C0038003 800380038007000700070007000E000E000E000E001C001C001C001C0038003800380038007000 700070007100E200E200E200E200640038000B237CA20C>108 D<1C0F80F8002610C10C004760 66060087807807008780780700870070070087007007000E00E00E000E00E00E000E00E00E000E 00E00E001C01C01C001C01C01C001C01C01C001C01C03820380380384038038070403803807080 380380308070070031003003001E0024157B9428>I<1C0F002631C04740C08780E08780E08700 E08700E00E01C00E01C00E01C00E01C01C03801C03801C03801C0704380708380E08380E103806 107006203003C017157B941B>I<007E0001C3000381800701C00E01C01C01E03C01E03801E078 01E07801E07801E0F003C0F003C0F00380F00780700700700E00700C0030180018700007C00013 157B9419>I<01C1F002621804741C08780C08700E08700E08701E00E01E00E01E00E01E00E01E 01C03C01C03C01C03C01C07803807003807003C0E003C1C0072380071E000700000700000E0000 0E00000E00000E00001C00001C00001C0000FFC000171F7F9419>I<1C1F002620804741C08783 C08703C08701808700000E00000E00000E00000E00001C00001C00001C00001C00003800003800 0038000038000070000030000013157B9415>114 D<00FC000183000200800401800C03800C03 000C00000F00000FF00007FC0003FE00003E00000F00000700700700F00600F00600E004004008 002030001FC00011157D9414>I<00C001C001C001C001C003800380038003800700FFF8070007 000E000E000E000E001C001C001C001C003800380038003810702070207040708031001E000D1F 7C9E10>I<1E00602300E04380E04381C08381C08701C08701C00703800E03800E03800E03801C 07001C07001C07001C07081C0E10180E101C0E101C1E200C262007C3C016157B941A>I<1E0030 2300704380704380E08380E08700E08700E00701C00E01C00E01C00E01C01C03801C03801C0380 1C03801C07001C07001C07001C0F000C3E0003CE00000E00000E00001C00601C00F03800F03000 E0600080C0004380003E0000151F7B9418>121 D E /Fb 1 81 df80 D E /Fc 3 21 df0 D2 D<0000000C0000003C000000F0000003C000000F0000003C000000F0000007C000001F0000 0078000001E00000078000001E00000078000000E0000000780000001E0000000780000001E000 0000780000001F00000007C0000000F00000003C0000000F00000003C0000000F00000003C0000 000C0000000000000000000000000000000000000000000000000000000000000000FFFFFFFCFF FFFFFC1E277C9F27>20 D E /Fd 3 52 df<07C018303018701C600C600CE00EE00EE00EE00EE0 0EE00EE00EE00EE00E600C600C701C30181C7007C00F157F9412>48 D<0F8030E040708030C038 E0384038003800700070006000C00180030006000C08080810183FF07FF0FFF00D157E9412>50 D<0FE030306018701C701C001C00180038006007E000300018000C000E000EE00EE00EC00C4018 30300FE00F157F9412>I E /Fe 3 106 df<07F0000FE000F0001E0000B8001E0000B8002E0000 B8004E000138005C000138009C000138011C00011C011C00021C023800021C043800021C043800 021C083800041C107000040E107000040E207000040E407000080E40E000080E80E000080F00E0 00080700E000180601C000FE040FF80023177F9622>77 D<07F007F800F000C000B8008000B800 80009C0080011C0100011E0100010E0100010E0100020702000207020002038200020382000401 C4000401C4000400E4000400E4000800780008007800080038000800380018001000FE0010001D 177F961C>I<0300038003000000000000000000000000001C002400460046008C000C00180018 00180031003100320032001C0009177F960C>105 D E /Ff 9 84 df<0F00187F00380F00380E 00700E00700E00700E00E01C00E01C01C01C01801C0380380700380600380C0038180070300070 600070C000730000FC0000F0000015157D9418>23 D<003F0001FFC00381E00400400800001000 001000001000001060000B98000FF800100000200000400000400000400000800000C000804000 80600100380E001FFC0007E00013177F9517>34 D<70F8F8F87005057C840E>58 D<70F8FCFC7404040404080810102040060F7C840E>I<00007F00400003C0C080000E00218000 1C0013800070000F8000E000070001C0000700038000070007000007000F000002000E00000200 1E000002003C000002003C00000400780000000078000000007800000000F000000000F0000000 00F000000000F000000000F0003FFF00E00000F000E00000F000E00000F000E00001E000F00001 E000F00001E000700001E000700003C000380003C000180007C0000C0009C00006001180000380 E08000007F00000022247DA226>71 D<007FC00001FF0007C00003E00007C00005E00007C00005 E00009E0000BC00009E0000BC00009E00013C00009E00023C00011E00027800011E00047800011 E00047800011E00087800021E0010F000020F0010F000020F0020F000020F0040F000040F0041E 000040F0081E000040F0081E000040F0101E000080F0203C00008078203C00008078403C000080 78803C0001007880780001007900780001007900780001007A00780002007C00F00002007C00F0 0002003800F00006003800F0000F003001F000FFE0203FFF0030227EA12F>77 D<007FC003FF0007C000780007C000600005E000200009E000400009E000400008F000400008F0 00400010F800800010780080001078008000103C008000203C010000203E010000201E01000020 1E010000400F020000400F020000400F020000400782000080078400008007C400008003C40000 8003C400010001E800010001E800010001F800010000F800020000F00002000070000200007000 06000070000F00002000FFE000200028227EA127>I<007FFFF0000007801C000007800F000007 800700000F000380000F000380000F000380000F000380001E000780001E000780001E00078000 1E000F00003C000F00003C001E00003C003C00003C007000007801E000007FFF00000078000000 007800000000F000000000F000000000F000000000F000000001E000000001E000000001E00000 0001E000000003C000000003C000000003C000000003C000000007C0000000FFFC00000021227E A11F>80 D<0003F010000E0C2000180260002001E0004000E000C000C0008000C0018000C00180 00C00380008003800080038000000380000003C0000001F0000001FE000000FFE000007FF00000 1FF8000001FC0000003C0000001C0000000E0000000E0000000E0020000C0020000C0020000C00 2000080060001800600010007000200070004000C8008000C603000081FC00001C247DA21E>83 D E /Fg 31 122 df45 D<60F0F06004047C830C>I<030007003F00C7 000700070007000700070007000700070007000700070007000700070007000700070007000700 0700070007000F80FFF80D1C7C9B15>49 D<07C01830201C400C400EF00FF80FF807F807700700 0F000E000E001C001C00380070006000C00180030006010C01180110023FFE7FFEFFFE101C7E9B 15>I<00F0030C06040C0E181E301E300C700070006000E3E0E430E818F00CF00EE006E007E007 E007E007E007600760077006300E300C18180C3003E0101D7E9B15>54 D<03C00C301818300C70 0C600EE006E006E007E007E007E007E0076007700F300F18170C2707C700060006000E300C780C 78187010203030C00F80101D7E9B15>57 D<000600000006000000060000000F0000000F000000 0F00000017800000178000001780000023C0000023C0000023C0000041E0000041E0000041E000 0080F0000080F0000180F8000100780001FFF80003007C0002003C0002003C0006003E0004001E 0004001E000C001F001E001F00FF80FFF01C1D7F9C1F>65 D<001F808000E06180018019800700 07800E0003801C0003801C00018038000180780000807800008070000080F0000000F0000000F0 000000F0000000F0000000F0000000F0000000F000000070000080780000807800008038000080 1C0001001C0001000E000200070004000180080000E03000001FC000191E7E9C1E>67 D78 D<07E0801C1980300580700380600180E00180E00080E00080E00080F00000F800007C00 007FC0003FF8001FFE0007FF0000FF80000F800007C00003C00001C08001C08001C08001C0C001 80C00180E00300D00200CC0C0083F800121E7E9C17>83 D<7FFFFFC0700F01C0600F00C0400F00 40400F0040C00F0020800F0020800F0020800F0020000F0000000F0000000F0000000F0000000F 0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000 0F0000000F0000000F0000001F800003FFFC001B1C7F9B1E>I<1FC000307000783800781C0030 1C00001C00001C0001FC000F1C00381C00701C00601C00E01C40E01C40E01C40603C40304E801F 870012127E9115>97 D<07E00C301878307870306000E000E000E000E000E000E0006000700430 0418080C3007C00E127E9112>99 D<003F00000700000700000700000700000700000700000700 00070000070000070003E7000C1700180F00300700700700600700E00700E00700E00700E00700 E00700E00700600700700700300700180F000C370007C7E0131D7E9C17>I<03E00C301818300C 700E6006E006FFFEE000E000E000E00060007002300218040C1803E00F127F9112>I<00F8018C 071E061E0E0C0E000E000E000E000E000E00FFE00E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E007FE00F1D809C0D>I<00038003C4C00C38C01C3880181800381C 00381C00381C00381C001818001C38000C300013C0001000003000001800001FF8001FFF001FFF 803003806001C0C000C0C000C0C000C06001803003001C0E0007F800121C7F9215>II<18003C003C0018000000000000000000000000000000FC001C001C 001C001C001C001C001C001C001C001C001C001C001C001C001C001C00FF80091D7F9C0C>I107 DIII<03F0000E1C00180600300300700380600180E001C0E001C0E001C0E001C0E001C0 E001C06001807003803003001806000E1C0003F00012127F9115>II 114 D<1F9030704030C010C010E010F8007F803FE00FF000F880388018C018C018E010D0608FC0 0D127F9110>I<04000400040004000C000C001C003C00FFE01C001C001C001C001C001C001C00 1C001C001C101C101C101C101C100C100E2003C00C1A7F9910>II119 D121 D E /Fh 2 122 df<040004000400C460E4E0 3F800E003F80E4E0C4600400040004000B0D7E8D11>3 D<0C000C000C000C000C000C00FFC0FF C00C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000C000A1A 7E9310>121 D E /Fi 41 122 df<0007F80FF000007FFE7FFC0001F80FF80E0003E00FE01F00 07C01FC03F000F801F803F000F801F803F000F800F801E000F800F800C000F800F8000000F800F 8000000F800F8000000F800F800000FFFFFFFFFF00FFFFFFFFFF000F800F801F000F800F801F00 0F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F 801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F000F800F801F00 0F800F801F000F800F801F000F800F801F007FF07FF0FFE07FF07FF0FFE02B237FA22F>14 D<387CFEFFFF7F3B03030706060C1C18702008117CA210>39 D<387CFEFEFE7C3807077C8610> 46 D<00180000780001F800FFF800FFF80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F80001F800 01F80001F80001F80001F80001F8007FFFE07FFFE013207C9F1C>49 D<03FC000FFF003C1FC070 07E07C07F0FE03F0FE03F8FE03F8FE01F87C01F83803F80003F80003F00003F00007E00007C000 0F80001F00003E0000380000700000E01801C0180380180700180E00380FFFF01FFFF03FFFF07F FFF0FFFFF0FFFFF015207D9F1C>I<00FE0007FFC00F07E01E03F03F03F03F81F83F81F83F81F8 1F03F81F03F00003F00003E00007C0001F8001FE0001FF000007C00001F00001F80000FC0000FC 3C00FE7E00FEFF00FEFF00FEFF00FEFF00FC7E01FC7801F81E07F00FFFC001FE0017207E9F1C> I<0000E00001E00003E00003E00007E0000FE0001FE0001FE00037E00077E000E7E001C7E00187 E00307E00707E00E07E00C07E01807E03807E07007E0E007E0FFFFFEFFFFFE0007E00007E00007 E00007E00007E00007E00007E000FFFE00FFFE17207E9F1C>I<1000201E01E01FFFC01FFF801F FF001FFE001FF8001BC00018000018000018000018000019FC001FFF001E0FC01807E01803E000 03F00003F00003F80003F83803F87C03F8FE03F8FE03F8FC03F0FC03F07007E03007C01C1F800F FF0003F80015207D9F1C>I<001F8000FFE003F07007C0F00F01F81F01F83E01F83E01F87E00F0 7C00007C0000FC0800FC7FC0FCFFE0FD80F0FF00F8FE007CFE007CFC007EFC007EFC007EFC007E 7C007E7C007E7C007E3C007C3E007C1E00F80F00F00783E003FFC000FF0017207E9F1C>I<0000 70000000007000000000F800000000F800000000F800000001FC00000001FC00000003FE000000 03FE00000003FE00000006FF000000067F0000000E7F8000000C3F8000000C3F800000183FC000 00181FC00000381FE00000300FE00000300FE00000600FF000006007F00000E007F80000FFFFF8 0000FFFFF800018001FC00018001FC00038001FE00030000FE00030000FE000600007F00060000 7F00FFE00FFFF8FFE00FFFF825227EA12A>65 D<0003FE0080001FFF818000FF01E38001F8003F 8003E0001F8007C0000F800F800007801F800007803F000003803F000003807F000001807E0000 01807E00000180FE00000000FE00000000FE00000000FE00000000FE00000000FE00000000FE00 000000FE000000007E000000007E000001807F000001803F000001803F000003801F800003000F 8000030007C000060003F0000C0001F800380000FF00F000001FFFC0000003FE000021227DA128 >67 DII72 DI76 DI80 D82 D<01FC0407FF8C1F03FC3C007C7C003C78001C78001CF8000CF8000CFC000CFC00 00FF0000FFE0007FFF007FFFC03FFFF01FFFF80FFFFC03FFFE003FFE0003FF00007F00003F0000 3FC0001FC0001FC0001FE0001EE0001EF0003CFC003CFF00F8C7FFE080FF8018227DA11F>I<07 FC001FFF803F07C03F03E03F01E03F01F01E01F00001F00001F0003FF003FDF01FC1F03F01F07E 01F0FC01F0FC01F0FC01F0FC01F07E02F07E0CF81FF87F07E03F18167E951B>97 DI<00FF8007FFE00F83 F01F03F03E03F07E03F07C01E07C0000FC0000FC0000FC0000FC0000FC0000FC00007C00007E00 007E00003E00301F00600FC0E007FF8000FE0014167E9519>I<0001FE000001FE0000003E0000 003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E0000003E00 01FC3E0007FFBE000F81FE001F007E003E003E007E003E007C003E00FC003E00FC003E00FC003E 00FC003E00FC003E00FC003E00FC003E00FC003E007C003E007C003E003E007E001E00FE000F83 BE0007FF3FC001FC3FC01A237EA21F>I<00FE0007FF800F87C01E01E03E01F07C00F07C00F8FC 00F8FC00F8FFFFF8FFFFF8FC0000FC0000FC00007C00007C00007E00003E00181F00300FC07003 FFC000FF0015167E951A>I<003F8000FFC001E3E003C7E007C7E00F87E00F83C00F80000F8000 0F80000F80000F80000F8000FFFC00FFFC000F80000F80000F80000F80000F80000F80000F8000 0F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80007FF8007FF800 13237FA211>I<03FC1E0FFF7F1F0F8F3E07CF3C03C07C03E07C03E07C03E07C03E07C03E03C03 C03E07C01F0F801FFF0013FC003000003000003800003FFF801FFFF00FFFF81FFFFC3800FC7000 3EF0001EF0001EF0001EF0001E78003C7C007C3F01F80FFFE001FF0018217E951C>II<1C003F007F007F007F003F001C 000000000000000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F00 1F001F001F001F001F001F001F001F00FFE0FFE00B247EA310>I108 DII<00FE0007FFC00F83E01E00 F03E00F87C007C7C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007EFC007E7C007C7C00 7C3E00F81F01F00F83E007FFC000FE0017167E951C>II<00FE030007FF87000FC1C7001F006F003F003F007E003F007E001F007C001F00FC001F00FC 001F00FC001F00FC001F00FC001F00FC001F00FC001F007E001F007E001F003E003F001F007F00 0FC1DF0007FF9F0001FC1F0000001F0000001F0000001F0000001F0000001F0000001F0000001F 0000001F000000FFE00000FFE01B207E951E>II<0FF3003FFF00781F00600700E00300E00300F00300FC0000 7FE0007FF8003FFE000FFF0001FF00000F80C00780C00380E00380E00380F00700FC0E00EFFC00 C7F00011167E9516>I<0180000180000180000180000380000380000780000780000F80003F80 00FFFF00FFFF000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80 000F81800F81800F81800F81800F81800F830007C30003FE0000F80011207F9F16>II119 D121 D E /Fj 2 122 df<020002000200C218F2783AE00F800F803AE0F278C2180200020002000D0E 7E8E12>3 D<06000600060006000600060006000600FFF0FFF006000600060006000600060006 000600060006000600060006000600060006000600060006000C1D7E9611>121 D E /Fk 83 124 df<001F83E000706E3000C07C780180F8780380F07807007000070070000700 7000070070000700700007007000070070000700700007007000FFFFFFC0070070000700700007 007000070070000700700007007000070070000700700007007000070070000700700007007000 070070000700700007007000070070000700700007007000070078007FE3FF801D2380A21C>11 D<001FC0000070200000C010000180380003807800070078000700300007000000070000000700 000007000000070000000700000007000000FFFFF8000700780007003800070038000700380007 003800070038000700380007003800070038000700380007003800070038000700380007003800 07003800070038000700380007003800070038007FE1FF80192380A21B>I<001FD80000703800 00C078000180780003807800070038000700380007003800070038000700380007003800070038 000700380007003800FFFFF8000700380007003800070038000700380007003800070038000700 380007003800070038000700380007003800070038000700380007003800070038000700380007 00380007003800070038007FF3FF80192380A21B>I<000FC07F00007031C08000E00B00400180 1E00E003803E01E007003C01E007001C00C007001C000007001C000007001C000007001C000007 001C000007001C000007001C0000FFFFFFFFE007001C01E007001C00E007001C00E007001C00E0 07001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00 E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E007001C00E07FF1FF CFFE272380A229>I<7038F87CFC7EFC7E743A0402040204020402080408041008100820104020 0F0F7EA218>34 D<0780000C001840001C0018200018003010007800701C00F0006013FF6000E0 0800E000E00800C000E00801C000E008038000E008030000E008070000E0080E0000E0080C0000 60101C000070101800003010380000182070000018406000000780E03C000001C04200000180C1 00000381810000070380800006030080000E030040000C070040001C0700400038070040003007 0040007007004000E007004000C007004001C00300400180030080038003808007000181000600 00C1000E000042000C00003C0022287DA429>37 D<70F8FCFC7404040404080810102040060F7C A20E>39 D<00200040008001000300060004000C000C0018001800300030003000700060006000 6000E000E000E000E000E000E000E000E000E000E000E000E000E000E000600060006000700030 0030003000180018000C000C0004000600030001000080004000200B327CA413>I<8000400020 00100018000C000400060006000300030001800180018001C000C000C000C000E000E000E000E0 00E000E000E000E000E000E000E000E000E000E000C000C000C001C00180018001800300030006 00060004000C00180010002000400080000B327DA413>I<000180000001800000018000000180 000001800000018000000180000001800000018000000180000001800000018000000180000001 80000001800000018000FFFFFFFEFFFFFFFE000180000001800000018000000180000001800000 018000000180000001800000018000000180000001800000018000000180000001800000018000 000180001F227D9C26>43 D<70F8FCFC7404040404080810102040060F7C840E>II<70F8F8F87005057C840E>I<00018000018000038000030000030000070000060000 0600000E00000C00000C00001C0000180000180000180000380000300000300000700000600000 600000E00000C00000C00001C0000180000180000380000300000300000700000600000600000E 00000C00000C00000C00001C0000180000180000380000300000300000700000600000600000E0 0000C00000C0000011317DA418>I<01F000071C000C06001803003803803803807001C07001C0 7001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001E0 F001E0F001E0F001E07001C07001C07001C07803C03803803803801C07000C0600071C0001F000 13227EA018>I<008003800F80F380038003800380038003800380038003800380038003800380 03800380038003800380038003800380038003800380038003800380038007C0FFFE0F217CA018 >I<03F0000C1C001007002007804003C04003C08003E0F003E0F801E0F801E0F801E02003E000 03E00003C00003C0000780000700000E00001C0000180000300000600000C00001800001000002 00200400200800201800603000403FFFC07FFFC0FFFFC013217EA018>I<03F8000C1E00100700 2007804007C07807C07803C07807C03807C0000780000780000700000F00000E0000380003F000 001C00000F000007800007800003C00003C00003E02003E07003E0F803E0F803E0F003C04003C0 400780200780100F000C1C0003F00013227EA018>I<000200000600000E00000E00001E00001E 00002E00004E00004E00008E00008E00010E00020E00020E00040E00040E00080E00100E00100E 00200E00200E00400E00800E00FFFFF8000E00000E00000E00000E00000E00000E00000E00001F 0001FFF015217FA018>I<1000801E07001FFF001FFE001FF80013E00010000010000010000010 000010000010000010F800130E001407001803801003800001C00001C00001E00001E00001E000 01E07001E0F001E0F001E0E001C08001C04003C04003802007001006000C1C0003F00013227EA0 18>I<007E0001C1000300800601C00E03C01C03C0180180380000380000780000700000700000 F0F800F30C00F40600F40300F80380F801C0F001C0F001E0F001E0F001E0F001E0F001E07001E0 7001E07001E03801C03801C01803801C03000C0600070C0001F00013227EA018>I<4000006000 007FFFE07FFFC07FFFC0400080C001008001008002008002000004000008000008000010000030 0000200000600000600000600000E00000C00000C00001C00001C00001C00001C00003C00003C0 0003C00003C00003C00003C00003C00003C00001800013237DA118>I<01F800060E0008030010 01802001802000C06000C06000C06000C07000C07801803E01003F02001FC4000FF80003F80003 FC00067F00083F80100F803007C06001C06000E0C000E0C00060C00060C00060C0006060004060 00C03000801803000E0E0003F00013227EA018>I<01F000060C000C0600180700380380700380 700380F001C0F001C0F001C0F001E0F001E0F001E0F001E0F001E07001E07003E03803E01805E0 0C05E00619E003E1E00001C00001C00001C0000380000380300300780700780600700C00201800 1030000FC00013227EA018>I<70F8F8F870000000000000000000000070F8F8F87005157C940E> I61 D<07E01838201C400E800FF00FF00FF00F000F000E00 1C00380030006000C000C000800080018001000100010001000100010000000000000000000000 038007C007C007C0038010237DA217>63 D<0001800000018000000180000003C0000003C00000 03C0000005E0000005E000000DF0000008F0000008F0000010F800001078000010780000203C00 00203C0000203C0000401E0000401E0000401E0000800F0000800F0000FFFF0001000780010007 80030007C0020003C0020003C0040003E0040001E0040001E00C0000F00C0000F03E0001F8FF80 0FFF20237EA225>65 DI<0007E0 100038183000E0063001C00170038000F0070000F00E0000701E0000701C0000303C0000303C00 00307C0000107800001078000010F8000000F8000000F8000000F8000000F8000000F8000000F8 000000F800000078000000780000107C0000103C0000103C0000101C0000201E0000200E000040 070000400380008001C0010000E0020000381C000007E0001C247DA223>IIII<0007F008003C0C18 00E0021801C001B8038000F8070000780F0000381E0000381E0000183C0000183C0000187C0000 087800000878000008F8000000F8000000F8000000F8000000F8000000F8000000F8000000F800 1FFF780000F8780000787C0000783C0000783C0000781E0000781E0000780F0000780700007803 8000B801C000B800E00318003C0C080007F00020247DA226>III<03FFF0001F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F 00000F00000F00000F00700F00F80F00F80F00F80E00F01E00401C0020380018700007C0001423 7EA119>IIIII<000FE00000783C0000E00E0003C007800780 03C00F0001E00E0000E01E0000F03C0000783C0000787C00007C7C00007C7800003C7800003CF8 00003EF800003EF800003EF800003EF800003EF800003EF800003EF800003EF800003E7800003C 7C00007C7C00007C3C0000783E0000F81E0000F00F0001E00F0001E0078003C003C0078000E00E 0000783C00000FE0001F247DA226>II<000FE00000783C0000E00E0003C00780078003C00F0001E00E0000E01E0000F03E0000F8 3C0000787C00007C7C00007C7800003C7800003CF800003EF800003EF800003EF800003EF80000 3EF800003EF800003EF800003EF800003E7800003C7C00007C7C00007C3C0000783C0000781E03 80F00E0420E00F0801E0078813C003C8178000E80E00007C3C02000FEC0200000C0200000C0200 000E0600000F0E000007FC000007FC000007F8000003F8000001E01F2D7DA226>II<03F0200C0C601802603001E07000E0600060 E00060E00060E00020E00020E00020F00000F000007800007F00003FF0001FFE000FFF0003FF80 003FC00007E00001E00000F00000F0000070800070800070800070800070C00060C00060E000C0 F000C0C80180C6070081FC0014247DA21B>I<7FFFFFF878078078600780184007800840078008 40078008C007800C80078004800780048007800480078004000780000007800000078000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000FC00003FFFF001E 227EA123>IIII<7FF807FF0007E001F80003C000E00003E000C00001E000800000F001000000F803 00000078020000007C040000003E0C0000001E080000001F100000000FB000000007A000000007 C000000003E000000001E000000001F000000003F80000000278000000047C0000000C3E000000 081E000000101F000000300F80000020078000004007C00000C003E000008001E000010001F000 030000F000070000F8001F8001FC00FFE007FFC022227FA125>II91 D<080410082010201040204020 8040804080408040B85CFC7EFC7E7C3E381C0F0F7AA218>II<08102020404080808080B8FCFC7C38060F7DA20E>96 D<0FE0001838003C0C003C0E0018 070000070000070000070000FF0007C7001E07003C0700780700700700F00708F00708F00708F0 0F087817083C23900FC1E015157E9418>I<0E0000FE00001E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E1F000E61C00E80600F00300E00380E003C 0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0E00380F00700C8060 0C41C0083F0017237FA21B>I<01FE000703000C07801C0780380300780000700000F00000F000 00F00000F00000F00000F00000F000007000007800403800401C00800C010007060001F8001215 7E9416>I<0000E0000FE00001E00000E00000E00000E00000E00000E00000E00000E00000E000 00E00000E00000E001F8E00704E00C02E01C01E03800E07800E07000E0F000E0F000E0F000E0F0 00E0F000E0F000E0F000E07000E07800E03800E01801E00C02E0070CF001F0FE17237EA21B>I< 01FC000707000C03801C01C03801C07801E07000E0F000E0FFFFE0F00000F00000F00000F00000 F000007000007800203800201C00400E008007030000FC0013157F9416>I<003C00C6018F038F 030F070007000700070007000700070007000700FFF80700070007000700070007000700070007 0007000700070007000700070007000700070007807FF8102380A20F>I<00007001F198071E18 0E0E181C07001C07003C07803C07803C07803C07801C07001C07000E0E000F1C0019F000100000 1000001800001800001FFE000FFFC00FFFE03800F0600030400018C00018C00018C00018600030 6000303800E00E038003FE0015217F9518>I<0E0000FE00001E00000E00000E00000E00000E00 000E00000E00000E00000E00000E00000E00000E00000E1F800E60C00E80E00F00700F00700E00 700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00 700E0070FFE7FF18237FA21B>I<1C001E003E001E001C00000000000000000000000000000000 000E00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00 0E00FFC00A227FA10E>I<01C003E003E003E001C00000000000000000000000000000000001E0 0FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000 E000E000E000E000E000E060E0F0C0F18061803E000B2C82A10F>I<0E0000FE00001E00000E00 000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E03FC0E01F00E01 C00E01800E02000E04000E08000E10000E38000EF8000F1C000E1E000E0E000E07000E07800E03 C00E01C00E01E00E00F00E00F8FFE3FE17237FA21A>I<0E00FE001E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E 000E000E000E000E000E000E00FFE00B237FA20E>I<0E1FC07F00FE60E183801E807201C00F00 3C00E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E 003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0 0E003800E0FFE3FF8FFE27157F942A>I<0E1F80FE60C01E80E00F00700F00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FF E7FF18157F941B>I<01FC000707000C01801800C03800E0700070700070F00078F00078F00078 F00078F00078F00078F000787000707800F03800E01C01C00E038007070001FC0015157F9418> I<0E1F00FE61C00E80600F00700E00380E003C0E001C0E001E0E001E0E001E0E001E0E001E0E00 1E0E001E0E003C0E003C0E00380F00700E80E00E41C00E3F000E00000E00000E00000E00000E00 000E00000E00000E00000E0000FFE000171F7F941B>I<01F8200704600E02601C01603801E078 00E07800E0F000E0F000E0F000E0F000E0F000E0F000E0F000E07000E07800E03801E01C01E00C 02E0070CE001F0E00000E00000E00000E00000E00000E00000E00000E00000E00000E0000FFE17 1F7E941A>I<0E3CFE461E8F0F0F0F060F000E000E000E000E000E000E000E000E000E000E000E 000E000E000F00FFF010157F9413>I<0F8830786018C018C008C008E008F0007F803FE00FF001 F8003C801C800C800CC00CC008E018D0308FC00E157E9413>I<02000200020002000600060006 000E001E003E00FFF80E000E000E000E000E000E000E000E000E000E000E000E040E040E040E04 0E040E040708030801F00E1F7F9E13>I<0E0070FE07F01E00F00E00700E00700E00700E00700E 00700E00700E00700E00700E00700E00700E00700E00700E00700E00F00E00F006017003827800 FC7F18157F941B>I IIII<3FFFC03803803007802007 00600E00401C00403C0040380000700000E00001E00001C0000380400700400F00400E00C01C00 80380080780180700780FFFF8012157F9416>II E /Fl 23 123 df<000003800000000007C00000000007C0000000000FE0000000000FE000000000 0FE0000000001FF0000000001FF0000000003FF8000000003FF8000000003FF80000000073FC00 00000073FC00000000F3FE00000000E1FE00000000E1FE00000001C0FF00000001C0FF00000003 C0FF80000003807F80000007807FC0000007003FC0000007003FC000000E003FE000000E001FE0 00001E001FF000001C000FF000001FFFFFF000003FFFFFF800003FFFFFF80000780007FC000070 0003FC0000700003FC0000E00001FE0000E00001FE0001E00001FF0001C00000FF0001C00000FF 00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE2F297EA834>65 D69 D77 D80 D<01FF800007FFF0000F81FC001FC0FE001FC07F001FC07F001FC03F800F80 3F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F801FE03F803F803F807F803F807F 003F80FE003F80FE003F80FE003F80FE007F80FF007F807F00FFC03F83DFFC0FFF0FFC01FC03FC 1E1B7E9A21>97 D<001FF80000FFFE0003F01F000FE03F801FC03F803F803F803F803F807F801F 007F000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF000000FF00 00007F0000007F8000003F8001C03FC001C01FC003C00FE0078003F01F0000FFFC00001FE0001A 1B7E9A1F>99 D<00003FF80000003FF80000003FF800000003F800000003F800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800001FE3F80000FFFBF80003F03FF8000FE00FF8001FC007F8003F8003F8003F8003F800 7F8003F8007F0003F800FF0003F800FF0003F800FF0003F800FF0003F800FF0003F800FF0003F8 00FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8003F8003F8003F8007F8001FC00F F8000FE01FF80003F03FFF8000FFF3FF80003FC3FF80212A7EA926>I<003FE00001FFF80003F0 7E000FE03F001FC01F803F800FC03F800FC07F000FC07F0007E0FF0007E0FF0007E0FF0007E0FF FFFFE0FFFFFFE0FF000000FF000000FF000000FF0000007F0000007F8000003F8000E03F8001E0 1FC001C00FE003C003F81F8000FFFE00001FF0001B1B7E9A20>I<0007F0003FFC00FE3E01FC7F 03F87F03F87F07F07F07F03E07F00007F00007F00007F00007F00007F00007F000FFFFC0FFFFC0 FFFFC007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F0007FFF807FFF807FFF80182A7E A915>I<00FF81F003FFE7FC0FC1FE7C1F80FC7C3F80FE7C3F007E107F007F007F007F007F007F 007F007F007F007F007F007F003F007E003F80FE001F80FC000FC1F8001FFFE00018FF80003800 00003C0000003C0000003E0000003FFFF8003FFFFF001FFFFFC00FFFFFE007FFFFF01FFFFFF07E 0007F87C0001F8F80001F8F80000F8F80000F8F80000F8FC0001F87E0003F03F0007E00FC01F80 03FFFE00007FF0001E287E9A22>I<07001FC01FE03FE03FE03FE01FE01FC00700000000000000 0000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE0F2B7DAA14>105 D108 DII<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F0 7F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007 F07F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE0001D1B7E9A22> II114 D<03FE300FFFF03E03F07800F07000F0F00070F0 0070F80070FC0000FFE000FFFE007FFFC03FFFE01FFFF007FFF800FFFC0003FC0000FCE0007CE0 003CF0003CF0003CF80078FC0078FF01F0F7FFC0C1FF00161B7E9A1B>I<007000007000007000 00700000F00000F00000F00001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F03807F038 07F03807F03807F03807F03807F03803F87001F8F000FFE0001F8015267FA51B>II120 DI<3FFFFF803FFFFF803F00FF803C00FF003801FE007803FC00 7807FC007007F800700FF000701FE000001FE000003FC000007F800000FF800000FF000001FE03 8003FC038003FC038007F803800FF007801FF007801FE007003FC00F007F801F00FF807F00FFFF FF00FFFFFF00191B7E9A1F>I E end %%EndProlog %%BeginSetup %%Feature: *Resolution 300 TeXDict begin @letter /letter where {pop letter} if %%EndSetup %%Page: 1 1 bop 432 658 a Fl(A)23 b(Mo)r(del)f(for)i(Executing)e(Multidisci)o(pl)o(i)o (nary)663 732 y(and)i(Multizonal)d(Programs)391 841 y Fk(Eric)16 b(Barszcz)658 823 y Fj(\003)792 841 y Fk(Sisira)g(K.)g(W)l(eeratunga)1247 823 y Fj(y)1383 841 y Fk(Eddy)g(Pramono)1705 823 y Fj(y)719 960 y Fk(NAS)g(Applied)f(Researc)o(h)g(Branc)o(h)731 1018 y(NASA)g(Ames)f (Researc)o(h)i(Cen)o(ter)788 1076 y(Mo\013ett)g(Field,)f(CA)h(94035)884 1240 y(Marc)o(h)g(26,)h(1993)941 1647 y Fi(Abstract)345 1733 y Fk(This)10 b(pap)q(er)i(presen)o(ts)e(the)h(mo)q(del)e(of)j(computing)d (for)i(m)o(ultidiscipli)o(nary)d(and/or)271 1793 y(m)o(ultizonal)24 b(applications)i(on)h(parallel)e(mac)o(hines)f(b)q(eing)i(pursued)h(at)f (NASA)271 1853 y(Ames)15 b(Researc)o(h)h(Cen)o(ter.)21 b(The)16 b(mo)q(del)f(describ)q(es)h(execution)f(on)i(a)g(parallel)f(ma-)271 1914 y(c)o(hine)c(with)h(some)f(n)o(um)o(b)q(er)g(of)h(pro)q(cessors,)i(eac)o (h)d(with)h(lo)q(cal)g(memory)l(,)e(connected)271 1974 y(b)o(y)20 b(a)g(net)o(w)o(ork.)31 b(It)20 b(do)q(es)g(not)g(discuss)g(pro)q(cessor)h (rates,)g(size)e(of)h(memory)l(,)d(net-)271 2034 y(w)o(ork)h(latency)l(,)g (net)o(w)o(ork)f(bandwidth,)i(net)o(w)o(ork)e(connectivit)o(y)l(,)f(disk)i (space,)g(I/O)271 2094 y(bandwidth)k(or)f(mass)f(storage.)36 b(It)20 b(also)h(do)q(es)h(not)f(discuss)g(language)g(and)h(dis-)271 2154 y(cipline)17 b(sp)q(eci\014c)g(algorithm)g(issues.)27 b(Mo)q(del)18 b(rationalit)o(y)f(and)i(functionalit)o(y)e(are)271 2214 y(presen)o(ted.)p 149 2594 719 2 v 205 2625 a Fh(\003)224 2640 y Fg(The)d(author)g(is)g(an)g(emplo)o(y)o(ee)e(of)i(NASA.)207 2675 y Fh(y)224 2690 y Fg(The)k(author)f(is)g(an)g(emplo)o(y)o(ee)f(of)h (Computer)f(Sciences)j(Co.)28 b(This)17 b(w)o(ork)g(w)o(as)g(funded)g (through)g(NASA)149 2740 y(con)o(tract)e(NAS)f(2-12961)p eop %%Page: 2 2 bop 149 42 a Fi(1)56 b(In)n(tro)r(duction)149 151 y Fk(The)18 b(problem)e(domain)g(of)i(particular)f(in)o(terest)f(to)h(NASA)f(Ames)g (Researc)o(h)g(Cen)o(ter)h(is)g(aero-)149 211 y(nautical)i(researc)o(h)g(and) h(engineering.)29 b(Some)18 b(of)h(the)g(disciplines)f(in)o(v)o(olv)o(ed)f (in)h(mo)q(deling)g(an)149 271 y(aircraft)h(though)h(its)e(\015igh)o(t)h(en)o (v)o(elop)q(e)e(are)i(\015uid)g(dynamics,)e(structural)i(dynamics,)f(thermal) 149 332 y(analysis,)i(propulsion,)g(con)o(trol)f(theory)l(,)g (electromagnetic)e(analysis,)j(and)f(acoustics.)31 b(All)18 b(of)149 392 y(the)c(computational)g(aeroscience)f(\(CAS\))h(grand)h(c)o (hallenges)e(of)i(the)f(HPCC)g(program)g(in)o(v)o(olv)o(e)149 452 y(coupling)j(t)o(w)o(o)f(or)g(more)f(of)i(these)f(disciplines.)223 512 y(In)d(addition,)h(geometrically)d(complex)h(aircraft)i (con\014gurations,)h(often)f(with)g(one)g(or)g(more)149 572 y(comp)q(onen)o(ts)d(in)f(relativ)o(e)f(motion)h(with)h(resp)q(ect)f(to)h (others,)h(require)e(comp)q(osite)f(mesh)h(metho)q(d-)149 633 y(ologies)20 b(for)f(e\016cien)o(t)e(spatial)i(discretization)e([4].)29 b(A)19 b(m)o(ultizonal)d(application)j(meshes)e(indi-)149 693 y(vidual)j(comp)q(onen)o(ts)f(of)h(the)f(con\014guration)i(separately)f(and)g (forms)f(a)h(comp)q(osite)f(mesh)f(b)o(y)149 753 y(o)o(v)o(erlapping)c(comp)q (onen)o(t)f(meshes.)19 b(The)c(o)o(v)o(erlapping)e(meshes)g(in)o(teract)g (through)j(data)f(in)o(ter-)149 813 y(p)q(olation)i(at)g(in)o(ter-mesh)d(b)q (oundary)k(p)q(oin)o(ts)e([8].)223 873 y(Curren)o(tly)l(,)22 b(there)g(are)g(no)h(applications)f(that)h(com)o(bine)d(all)h(of)i(the)f (disciplines.)38 b(Ho)o(w-)149 934 y(ev)o(er,)22 b(sev)o(eral)f(co)q(des)h (exist)f(that)h(com)o(bine)e(at)i(least)f(t)o(w)o(o)h(of)g(these)f (disciplines.)37 b(Some)20 b(ex-)149 994 y(amples)14 b(in)g(the)h (aeronautical)g(domain)f(are)g(\015uid-thermal,)f(\015uid-structures,)i (\015uid-acoustics,)149 1054 y(and)i(\015uid-electromagnetic.)j(A)o(t)15 b(NASA)h(Ames)e(Researc)o(h)i(Cen)o(ter,)f(\015uid-thermal)g([5],)h(\015uid-) 149 1114 y(structures)i([10],)e(and)i(m)o(ultizonal)d(\015uid)i(dynamics)f ([11])h(co)q(des)g(ha)o(v)o(e)g(b)q(een)g(implem)o(en)n(ted)d(on)149 1174 y(a)j(message)f(passing)h(parallel)e(computer,)g(the)h(In)o(tel)e (iPSC/860.)223 1234 y(This)21 b(pap)q(er)i(presen)o(ts)e(the)g(mo)q(del)g(of) h(computing)e(for)i(m)o(ultidiscipl)o(inary)d(and/or)k(m)o(ul-)149 1295 y(tizonal)g(applications)f(on)h(parallel)f(mac)o(hines)e(b)q(eing)j (pursued)g(at)g(NASA)e(Ames)g(Researc)o(h)149 1355 y(Cen)o(ter.)40 b(The)22 b(mo)q(del)f(describ)q(es)i(execution)e(on)i(a)g(parallel)f(mac)o (hine)e(with)i(some)g(n)o(um)o(b)q(er)149 1415 y(of)h(pro)q(cessors,)i(eac)o (h)d(with)g(lo)q(cal)g(memory)l(,)f(connected)h(b)o(y)g(a)h(net)o(w)o(ork.)39 b(It)22 b(do)q(es)h(not)g(dis-)149 1475 y(cuss)c(pro)q(cessor)h(rates,)f (size)e(of)i(memory)l(,)d(net)o(w)o(ork)i(latency)l(,)f(net)o(w)o(ork)h (bandwidth,)i(net)o(w)o(ork)149 1535 y(connectivit)o(y)l(,)e(disk)i(space,)g (I/O)g(bandwidth,)h(or)f(mass)f(storage.)33 b(It)20 b(also)g(do)q(es)g(not)h (discuss)149 1596 y(language)d(and)f(discipline)d(sp)q(eci\014c)h(algorithm)h (issues.)223 1656 y(The)g(pap)q(er)h(has)f(three)g(sections.)21 b(Section)16 b(2)g(describ)q(es)g(general)g(m)o(ultidisci)o(plinary)d(prob-) 149 1716 y(lem)24 b(c)o(haracteristics.)47 b(Section)25 b(3)h(discusses)f(p)q (ossible)h(solution)f(approac)o(hes.)50 b(Section)24 b(4)149 1776 y(presen)o(ts)16 b(three)g(computational)f(mo)q(dels)h(and)g(their)g (requiremen)o(ts.)149 1908 y Fi(2)56 b(Problem)17 b(Characteristics)149 2017 y Fk(A)i(m)o(ultidiscipli)o(nary)e(problem)g(is)i(comp)q(osed)g(of)h(a)g (collection)d(of)j(t)o(w)o(o)f(or)h(more)e(in)o(teracting)149 2078 y(disciplines.)31 b(A)19 b(giv)o(en)g(discipline)f(ma)o(y)h(in)o(teract) f(directly)g(with)i(one)g(or)g(more)f(of)h(the)g(other)149 2138 y(disciplines.)k(Also,)17 b(these)g(in)o(teractions)g(o)q(ccur)h(sim)o (ultaneously)d(across)j(all)f(in)o(terdisciplinary)149 2198 y(b)q(oundaries.)30 b(Similar)16 b(remarks)i(apply)g(to)i(comp)q(onen)o(t)d (meshes)h(in)g(a)h(m)o(ultizonal)d(computa-)149 2258 y(tion.)223 2318 y(A)11 b(m)o(ultidiscipli)o(nary)e(problem)i(is)h(naturally)g(decomp)q (osed)f(in)o(to)h(di\013eren)o(t)g(in)o(teracting)f(dis-)149 2379 y(ciplines,)17 b(eac)o(h)h(of)h(whic)o(h)f(ma)o(y)f(mo)q(del)g(widely)h (di\013eren)o(t)f(ph)o(ysical)h(phenomena.)27 b(Therefore,)149 2439 y(the)d(equations)f(used)h(to)f(describ)q(e)g(the)g(underlying)g(ph)o (ysics)g(\(PDEs/ODEs)i(and/or)g(in)o(te-)149 2499 y(gral)19 b(equations\))f(v)m(ary)g(b)q(et)o(w)o(een)f(disciplines.)25 b(As)18 b(a)h(result,)e(the)h(computational)f(tec)o(hniques)149 2559 y(used)i(for)g(computer)e(sim)o(ulations)f(ma)o(y)h(ha)o(v)o(e)h(widely) f(di\013eren)o(t)h(n)o(umerical)d(algorithm)j(c)o(har-)149 2619 y(acteristics,)c(data)h(structures,)f(and)h(computational)e(requiremen)o (ts.)18 b(In)c(addition,)g(on)h(parallel)149 2680 y(computers)e(with)i(ph)o (ysically)d(distributed)i(memory)l(,)d(the)j(ab)q(o)o(v)o(e)g(di\013erences)g (ma)o(y)e(induce)i(dis-)149 2740 y(tinct)19 b(data)h(partitioning)f (strategies)g(for)h(eac)o(h)e(discipline.)28 b(F)l(or)19 b(example,)e(some)i (disciplines)1036 2864 y(2)p eop %%Page: 3 3 bop 149 42 a Fk(ma)o(y)17 b(b)q(e)h(mo)q(deled)e(using)i(\014nite)f (di\013erence)g(discretization)g(on)h(logically)f(structured)g(meshes)149 102 y(while)i(others)g(ma)o(y)f(use)h(a)g(\014nite)g(elemen)o(t)d (discretization)i(on)h(unstructured)g(meshes.)29 b(Also,)149 162 y(the)23 b(computational)g(requiremen)n(ts)e(within)h(eac)o(h)h (discipline)e(ma)o(y)g(v)m(ary)j(dynamically)c(due)149 222 y(to)g(solution)g(adaptiv)o(e)f(mesh)g(re\014nemen)o(t/coarsening,)f(mo)o (ving)g(b)q(oundaries,)j(and)f(ev)o(olving)149 282 y(material)15 b(non-linearities,)g(etc.)149 415 y Fi(3)56 b(Solution)19 b(Approac)n(hes)149 525 y Fk(Giv)o(en)h(a)h(m)o(ultidiscipl)o(inary)d(and/or)j(m)o(ultizonal)e (problem,)g(what)j(w)o(ould)e(b)q(e)h(the)f(b)q(est)h(ap-)149 585 y(proac)o(h)g(to)f(solving)g(it)g(on)g(a)h(parallel)e(computer?)32 b(On)20 b(a)g(serial)g(computer,)f(the)h(disciplines)149 645 y(and/or)15 b(comp)q(onen)o(t)c(meshes)h(m)o(ust)f(b)q(e)i(pro)q(cessed)g (serially)l(.)19 b(This)13 b(is)g(accomplished)e(b)o(y)h(ha)o(ving)149 706 y(a)21 b(doubly)f(nested)g(lo)q(op.)34 b(The)21 b(outer)f(lo)q(op)h (iterates)f(for)g(the)g(appropriate)h(n)o(um)o(b)q(er)d(of)j(time)149 766 y(steps)c(and)g(the)f(inner)f(lo)q(op)i(iterates)f(o)o(v)o(er)f(the)h (disciplines)f(\(meshes\).)223 826 y(F)l(or)d(a)h(parallel)e(computer,)h(the) g(disciplines)e(\(meshes\))h(ma)o(y)g(b)q(e)h(executed)f(in)h(parallel)g(or)h (se-)149 886 y(quen)o(tially)l(.)19 b(In)14 b(the)g(parallel)f(execution)h (mo)q(de,)f(eac)o(h)h(discipline)e(\(mesh\))h(ma)o(y)g(b)q(e)h(partitioned) 149 946 y(across)19 b(a)f(di\013eren)o(t)f(set)h(of)g(pro)q(cessors)h(and)g (computation)e(pro)q(ceeds)h(in)f(parallel)g(within)h(dis-)149 1007 y(ciplines)i(\(meshes\))g(and)i(across)h(disciplines)c(\(meshes\).)36 b(Both)21 b(data)h(parallelism)e(and)i(task)149 1067 y(parallelism)f(are)h (used.)40 b(In)23 b(the)f(sequen)o(tial)f(execution)g(mo)q(de,)i(eac)o(h)f (discipline)f(\(mesh\))g(is)149 1127 y(partitioned)15 b(across)g(all)f(pro)q (cessors)i(and)f(the)f(computation)f(pro)q(ceeds)i(in)f(parallel)g(within)g (dis-)149 1187 y(ciplines)h(\(meshes\))g(but)h(sequen)o(tially)f(across)i (disciplines)e(\(meshes\).)k(Only)d(data)h(parallelism)149 1247 y(is)f(used.)223 1307 y(The)k(decision)g(whether)g(or)h(not)f(to)h(use)g (task)f(parallelism)e(across)k(disciplines)c(\(meshes\))149 1368 y(is)g(in\015uenced)e(b)o(y)h(sev)o(eral)f(factors:)24 b(memory)15 b(requiremen)o(t)o(s,)g(memory)f(e\016ciency)l(,)h(computa-)149 1428 y(tional)d(requiremen)o(ts,)d(Amdahl's)h(La)o(w,)j(soft)o(w)o(are)f (engineering)f(issues,)i(and)f(m)o(ultidisci)o(plinary)149 1488 y(coupling)17 b(issues.)223 1548 y(In)g(the)g(discussion)g(b)q(elo)o(w)g (when)g(comparing)g(t)o(w)o(o)g(approac)o(hes,)h(a)f(\014xed)g(n)o(um)o(b)q (er)f(of)h(pro-)149 1608 y(cessors)f(and)f(a)g(\014xed)f(problem)f(size)h (are)h(assumed.)20 b(A)14 b(\014xed)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h (is)e(a)h(v)m(alid)149 1669 y(assumption)20 b(b)q(ecause)h(at)f(an)o(y)h (instan)o(t,)f(the)g(goal)h(is)f(to)h(use)f(the)g(a)o(v)m(ailable)f (resources)i(most)149 1729 y(e\016cien)o(tly)l(.)27 b(A)18 b(\014xed)h(problem)e(size)h(is)h(appropriate)g(b)q(ecause)h(once)e(the)h (relev)m(an)o(t)f(ph)o(ysics)g(is)149 1789 y(resolv)o(ed)f(to)g(the)g (desired)g(lev)o(el)e(of)j(accuracy)l(,)e(further)h(re\014nemen)o(t)e(\\w)o (astes")k(computational)149 1849 y(resources)e(\(memory)c(and)k(CPU)f (cycles\).)149 1977 y Fi(3.1)56 b(Memory)17 b(Requirem)o(en)n(ts)149 2069 y Fk(Memory)j(requiremen)o(ts)f(include)i(the)g(size)g(of)h(the)g (executable)e(and)i(the)g(size)f(of)h(the)f(data.)149 2130 y(F)l(or)i(a)f(SIMD)g(mac)o(hine,)f(there)g(is)h(a)g(single)g(cop)o(y)f(of)i (the)f(executable)e(and)j(the)f(disciplines)149 2190 y(are)d(pro)q(cessed)g (sequen)o(tially)e(\(data)i(parallelism)d(only\).)29 b(On)18 b(a)h(MIMD)f(mac)o(hine,)f(there)h(are)149 2250 y(m)o(ultiple)g(copies)i(of)h (the)f(executable.)33 b(If)20 b(the)g(disciplines)f(are)i(pro)q(cessed)g(in)f (parallel)g(\(task)149 2310 y(and)i(data)g(parallelism\),)e(the)h(set)g(of)h (pro)q(cessors)g(asso)q(ciated)g(with)f(a)h(discipline)d(only)i(need)149 2370 y(the)c(executable)f(for)i(that)f(discipline.)23 b(If)16 b(the)h(disciplines)f(are)h(pro)q(cessed)h(sequen)o(tially)d(\(data)149 2431 y(parallelism)j(only\),)j(eac)o(h)e(pro)q(cessor)j(needs)d(a)i(cop)o(y)f (of)g(the)g(executable)f(for)h(all)g(disciplines.)149 2491 y(Dep)q(ending)g(on)f(the)g(amoun)o(t)f(of)h(lo)q(cal)g(memory)l(,)e(whether) h(or)i(not)f(virtual)f(memory)e(is)j(sup-)149 2551 y(p)q(orted,)d(and)f(the)g (parallel)f(I/O)h(bandwidth,)h(this)f(ma)o(y)e(or)i(ma)o(y)f(not)h(b)q(e)g (an)h(issue.)k(Curren)o(tly)l(,)149 2611 y(most)13 b(massiv)o(ely)d(parallel) i(pro)q(cessors)j(should)e(b)q(e)g(though)o(t)h(of)f(as)h(ph)o(ysical)e (memory)e(mac)o(hines)149 2671 y(and)17 b(are)g(considered)e(suc)o(h)h(in)g (this)g(pap)q(er.)1036 2864 y(3)p eop %%Page: 4 4 bop 223 42 a Fk(The)19 b(memory)d(required)i(to)h(store)h(data)g(also)f(v)m (aries.)30 b(On)19 b(a)h(SIMD)f(mac)o(hine,)e(since)h(the)149 102 y(amoun)o(t)d(of)g(data)h(for)g(eac)o(h)e(discipline)g(\(mesh\))f(v)m (ary)l(,)i(some)f(data)i(structures)g(ma)o(y)d(b)q(e)i(padded)149 162 y(dep)q(ending)j(on)g(sp)q(eci\014c)g(mac)o(hine)d(or)j(compiler)d (requiremen)o(ts)g([12].)25 b(Also,)18 b(temp)q(orary)f(v)m(ari-)149 222 y(ables)g(are)g(often)g(the)g(size)f(of)h(the)g(whole)f(mesh.)22 b(On)17 b(a)g(MIMD)f(mac)o(hine,)f(when)i(a)g(domain)f(is)149 282 y(partitioned)c(across)h(m)o(ultiple)c(pro)q(cessors,)14 b(some)d(data)h(are)g(often)h(duplicated)e(to)h(sa)o(v)o(e)g(comm)o(u-)149 342 y(nication)18 b(costs)g([2].)25 b(Also,)18 b(temp)q(orary)e(v)m (ariables,)i(t)o(ypically)e(scalar)i(or)g(v)o(ector)e(temp)q(oraries,)149 403 y(and)k(small)e(global)h(data)h(structures)f(\(i.e.,)f(small)g(lo)q(ok)h (up)h(tables\))f(are)g(replicated)f(to)h(a)o(v)o(oid)149 463 y(b)q(ottlenec)o(ks.)32 b(Bu\013er)20 b(space)g(is)g(also)h(needed)e(on)i (eac)o(h)e(pro)q(cessor)i(to)g(bu\013er)f(messages)g(in)f(a)149 523 y(message)d(passing)h(en)o(vironmen)o(t.)223 583 y(If)f(the)h (disciplines)f(\(meshes\))g(are)h(pro)q(cessed)h(in)e(parallel,)h(eac)o(h)f (discipline)g(\(mesh\))g(has)h(a)149 643 y(b)q(etter)g(surface)f(to)h(v)o (olume)d(ratio)j(for)g(its)f(data)h(implying)d(less)j(duplicated)e(data.)23 b(\(Assuming)149 704 y(a)18 b(\014xed)f(n)o(um)o(b)q(er)f(of)i(pro)q(cessors) g(for)g(the)f(whole)g(application.\))25 b(Also,)17 b(the)g(memory)d(required) 149 764 y(for)23 b(temp)q(orary)f(v)m(ariables)g(dep)q(ends)h(up)q(on)g(the)f (individual)g(discipline.)37 b(If)22 b(the)h(disciplines)149 824 y(\(meshes\))d(are)h(pro)q(cessed)g(sequen)o(tially)l(,)f(the)h(lo)q(cal) g(memory)d(asso)q(ciated)k(with)e(a)i(pro)q(cessor)149 884 y(m)o(ust)17 b(b)q(e)h(shared)g(b)o(y)f(all)g(disciplines)f(\(meshes\).)24 b(This)18 b(implies)d(a)j(higher)f(surface)h(to)g(v)o(olume)149 944 y(ratio)f(for)f(eac)o(h)f(discipline)f(\(mesh\))h(since)g(eac)o(h)h (discipline)e(\(mesh\))g(is)i(spread)g(o)o(v)o(er)g(more)e(pro-)149 1005 y(cessors.)21 b(This)13 b(in)g(turn)g(implies)d(more)i(duplicated)g (data)i(to)f(a)o(v)o(oid)f(comm)o(unication.)17 b(Also,)c(the)149 1065 y(memory)j(requiremen)o(t)g(for)j(temp)q(orary)f(v)m(ariables)h(is)f (dep)q(enden)o(t)h(on)g(the)f(discipline)f(\(mesh\))149 1125 y(with)g(the)f(largest)g(requiremen)o(ts.)149 1252 y Fi(3.2)56 b(Memory)17 b(E\016ciency)149 1344 y Fk(Greater)22 b(parallelism)e(implies)f (more)h(memory)f(is)j(required)f(to)h(solv)o(e)f(the)h(application)f(e\016-) 149 1405 y(cien)o(tly)l(.)f(Let)c(memory)d(e\016ciency)h(b)q(e)j(de\014ned)f (as)932 1531 y Ff(")955 1538 y Fe(M)1008 1531 y Fk(=)1072 1497 y Ff(M)1119 1504 y Fd(0)p 1065 1519 81 2 v 1065 1565 a Ff(M)1112 1572 y Fe(N)1151 1531 y Ff(;)149 1656 y Fk(where)24 b Ff(")321 1663 y Fe(M)383 1656 y Fk(is)g(the)f(memory)d(e\016ciency)l(,)j Ff(M)1002 1663 y Fd(0)1045 1656 y Fk(is)h(amoun)o(t)e(of)i(the)f(memory)e (required)h(to)i(\014t)149 1716 y(the)d(problem)d(on)j(the)f(smallest)f(n)o (um)o(b)q(er)f(of)j(pro)q(cessors)h(\()p Ff(P)1311 1723 y Fd(0)1331 1716 y Fk(\),)f(and)g Ff(M)1531 1723 y Fe(N)1585 1716 y Fk(is)f(the)g(amoun)o (t)g(of)149 1777 y(memory)12 b(used)j(on)g Ff(P)543 1784 y Fe(N)592 1777 y Fk(pro)q(cessors)h(\()p Ff(P)873 1784 y Fd(0)907 1777 y Fc(\024)d Ff(P)990 1784 y Fe(N)1024 1777 y Fk(\).)21 b(Then)15 b(the)f(claim)f(is)h(that)h(memory)d(e\016ciency)149 1837 y(decreases)19 b(as)g(the)g(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h (increase.)28 b(All)18 b(that)h(is)g(necessary)f(for)h(this)g(claim)149 1897 y(to)f(b)q(e)f(true)g(is)f(for)i(a)f(single)g(v)m(ariable/compiler)d (temp)q(orary)j(to)g(b)q(e)g(replicated)f(across)i(t)o(w)o(o)f(or)149 1957 y(more)e(pro)q(cessors.)223 2017 y(As)f(an)h(example)e(of)i(ho)o(w)g (memory)d(requiremen)o(ts)g(can)j(gro)o(w)g(with)g(increased)f(parallelism,) 149 2078 y(consider)h(a)g(32)8 b Fc(\002)g Fk(32)g Fc(\002)g Fk(32)17 b(mesh.)i(\(F)l(airly)14 b(small)f(mesh)h(sizes)g(o)q(ccur)h (frequen)o(tly)e(in)h(m)o(ultizonal)149 2138 y(applications.\))34 b(F)l(orm)19 b(an)i(indep)q(enden)o(t)e(scalar)i(tridiagonal)g(system)e(out)h (of)h(eac)o(h)f(column.)149 2198 y(Then)i(there)f(are)h(32)547 2180 y Fd(2)589 2198 y Fk(indep)q(enden)o(t)f(tridiagonal)h(systems,)f(eac)o (h)g(of)h(length)f(32.)38 b(Compare)149 2258 y(the)20 b(memory)c(requiremen)o (ts)h(to)j(solv)o(e)f(the)g(systems)f(on)i(a)g(serial)f(mac)o(hine,)f(a)i (single)f(v)o(ector)149 2318 y(pro)q(cessor,)j(and)e(a)h(parallel)e(v)o (ector)g(mac)o(hine.)30 b(The)20 b(solution)g(tec)o(hnique)f(will)f(b)q(e)j (Gaussian)149 2379 y(elimination)14 b(without)j(piv)o(oting)e(where)h(the)g (input)g(ma)o(y)f(b)q(e)h(o)o(v)o(erwritten.)223 2439 y(On)g(a)h(serial)e (mac)o(hine,)f(32)725 2421 y Fd(3)757 2439 y Fk(+)d(3)g Fc(\002)g Fk(32)17 b(w)o(ords)g(are)f(required,)f(32)1430 2421 y Fd(3)1467 2439 y Fk(for)h(the)g(righ)o(t)g(hand)h(side)149 2499 y(and)i(3)13 b Fc(\002)f Fk(32)19 b(for)g(a)g(single)e(tridiagonal)i(system.)26 b(The)18 b(systems)f(are)h(formed)f(and)i(solv)o(ed)f(one)149 2559 y(at)f(a)g(time.)223 2619 y(On)g(a)i(single)e(v)o(ector)g(pro)q(cessor,) i(32)910 2601 y Fd(3)942 2619 y Fk(+)13 b(3)f Fc(\002)g Fk(32)1128 2601 y Fd(2)1166 2619 y Fk(w)o(ords)19 b(are)f(required,)e(32)1645 2601 y Fd(3)1684 2619 y Fk(for)i(the)f(righ)o(t)149 2680 y(hand)f(side)f(and) g(3)9 b Fc(\002)f Fk(32)588 2661 y Fd(2)623 2680 y Fk(to)16 b(form)e(32)h(indep)q(enden)o(t)f(tridiagonal)i(systems.)j(A)o(t)14 b(least)h(32)h(of)f(the)149 2740 y(systems)g(are)i(formed)e(at)h(a)h(time)d (to)j(allo)o(w)f(for)g(v)o(ectorization)f(across)i(indep)q(enden)o(t)f (systems.)1036 2864 y(4)p eop %%Page: 5 5 bop 223 42 a Fk(On)13 b(a)h(32)h(no)q(de)f(parallel)f(v)o(ector)g(mac)o (hine,)e(4)6 b Fc(\002)g Fk(32)1157 23 y Fd(3)1191 42 y Fk(w)o(ords)14 b(are)g(required,)f(32)1658 23 y Fd(3)1692 42 y Fk(for)h(the)f(righ)o(t)149 102 y(hand)20 b(side)f(and)h(3)13 b Fc(\002)g Fk(32)610 84 y Fd(3)650 102 y Fk(to)19 b(form)f(32)878 84 y Fd(2)918 102 y Fk(indep)q(enden)o(t)g(tridiagonal)i(systems.)28 b(Thirt)o(y)19 b(t)o(w)o(o)g(of)149 162 y(the)d(systems)f(are)i(formed)e(at)h(a)h(time)d(on) j(eac)o(h)f(pro)q(cessor)h(to)f(allo)o(w)g(for)h(v)o(ectorization.)223 222 y(If)c(the)h(serial)g(mac)o(hine)e(has)j(a)f(memory)d(e\016ciency)h(of)j (1)p Ff(:)p Fk(0,)f(then)g(the)g(single)g(v)o(ector)f(pro)q(ces-)149 282 y(sor)h(and)f(parallel)f(v)o(ector)g(mac)o(hine)e(ha)o(v)o(e)i(memory)e (e\016ciencies)g(of)k(0)p Ff(:)p Fk(92)f(and)g(0)p Ff(:)p Fk(25)h(resp)q (ectiv)o(ely)l(.)149 410 y Fi(3.3)56 b(Computational)18 b(Requirem)o(en)n(ts) 149 502 y Fk(Computational)d(requiremen)o(ts)d(for)j(di\013eren)o(t)e (disciplines)g(and)j(di\013eren)o(t)e(meshes)f(v)m(ary)l(.)21 b(Also,)149 563 y(mesh)16 b(sizes)g(ma)o(y)g(v)m(ary)h(widely)f(in)g(m)o (ultizonal)f(computations)h(yielding)g(v)o(ery)g(di\013eren)o(t)g(com-)149 623 y(putational)f(requiremen)n(ts)c(for)j(di\013eren)o(t)f(meshes.)19 b(This)14 b(leads)f(to)h(load)g(balancing)g(issues.)21 b(F)l(or)149 683 y(example,)14 b(computational)h(requiremen)o(ts)e(in)i(the)h(con)o(trols) g(discipline)e(is)i(m)o(uc)o(h)d(smaller)h(than)149 743 y(the)i (computational)g(requiremen)o(ts)d(to)k(compute)e(a)h(\015uid)g(\015o)o(w)h (\014eld.)223 803 y(If)g(disciplines)f(\(meshes\))h(are)g(pro)q(cessed)i(in)e (parallel)g(\(task)h(and)h(data)g(parallelism\))c(then)149 864 y(the)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h(assigned)f(to)g(eac)o(h)g (discipline)d(\(mesh\))i(can)h(b)q(e)g(adjusted)g(to)g(com-)149 924 y(putationally)g(load)h(balance)f(the)g(application.)21 b(\(Sub)s(ject)16 b(to)g(memory)e(constrain)o(ts.\))223 984 y(If)i(disciplines)f(\(meshes\))g(are)i(pro)q(cessed)g(sequen)o(tially)d (\(data)k(parallelism)c(only\),)i(for)h(dis-)149 1044 y(ciplines)d (\(meshes\))h(with)g(small)f(memory)f(requiremen)n(ts)g(and)j(small)e (computational)h(require-)149 1104 y(men)o(ts,)g(trying)h(to)h(use)g(all)f(a) o(v)m(ailable)g(pro)q(cessors)i(ma)o(y)d(actually)h(slo)o(w)h(do)o(wn)g(the)f (application)149 1165 y(due)h(to)g(the)g(decrease)f(in)g(the)h(computation)f (to)h(comm)o(unicati)o(on)e(ratio.)23 b(If)16 b(the)h(computation)149 1225 y(asso)q(ciated)d(with)f(a)g(discipline)e(\(mesh\))g(is)h(unable)h(to)g (use)g(all)f(a)o(v)m(ailable)g(pro)q(cessors,)i(then)f(some)149 1285 y(pro)q(cessors)18 b(will)d(b)q(e)h(idle)g(during)g(that)h(phase)f(of)h (the)f(computation.)223 1345 y(Ho)q(c)o(kney)h(and)i(Jesshop)q(e)g([7])f (discuss)g(trade)h(o\013s)g(in)f(algorithm)f(selection.)26 b(Whic)o(h)18 b(algo-)149 1405 y(rithm)c(is)i(\\b)q(est")h(dep)q(ends)f(up)q (on)g(the)g(mac)o(hine)d(arc)o(hitecture,)h(actual)i(n)o(um)o(b)q(er)e(of)i (pro)q(cessors)149 1466 y(b)q(eing)e(used,)f(and)h(the)f(problem)e(size.)19 b(An)13 b(algorithm)f(with)h(more)f(inheren)o(t)g(parallelism)f(migh)o(t)149 1526 y(not)20 b(execute)d(as)i(fast)g(as)h(an)f(algorithm)f(with)g(less)h (inheren)o(t)e(parallelism)f(due)j(to)g(an)g(o)o(v)o(erall)149 1586 y(increase)d(in)g(op)q(erations.)149 1714 y Fi(3.4)56 b(Amdahl's)18 b(La)n(w)149 1806 y Fk(E\016ciency)c(also)i(v)m(aries)f(dep)q (ending)h(on)f(whether)g(the)h(disciplines)d(\(meshes\))h(are)h(pro)q(cessed) h(in)149 1866 y(parallel)g(or)g(sequen)o(tially)l(.)j(Amdahl's)14 b(La)o(w)j(sa)o(ys)f(that)h(sp)q(eedup)f(is)g(b)q(ounded)h(b)o(y)e(the)h (fraction)149 1926 y(of)h(the)f(co)q(de)g(that)h(is)f(serial:)847 1998 y Ff(S)h Fc(\024)1069 1965 y Ff(N)p 951 1987 280 2 v 951 2033 a Fk(\()p Ff(N)g Fc(\000)11 b Fk(1\))p Ff(\027)j Fk(+)d(1)1236 1998 y Ff(;)149 2111 y Fk(where)18 b Ff(S)i Fk(is)e(the)f(sp)q(eedup,)h Ff(N)23 b Fk(is)17 b(the)h(n)o(um)o(b)q(er)e(of)i(pro)q(cessors,)h(and)f Ff(\027)j Fk(is)c(the)h(fraction)f(of)h(the)149 2171 y(co)q(de)f(that)g(is)f (serial.)k(E\016ciency)l(,)14 b Ff(")p Fk(,)i(is)g(de\014ned)g(as)793 2302 y Ff(")d Fk(=)892 2269 y Ff(S)p 886 2291 45 2 v 886 2336 a(N)949 2302 y Fk(=)1133 2269 y(1)p 1006 2291 280 2 v 1006 2336 a(\()p Ff(N)j Fc(\000)11 b Fk(1\))p Ff(\027)k Fk(+)c(1)1290 2302 y Ff(:)149 2437 y Fk(Amdahl's)h(La)o(w)i(is)f(relev)m(an)o(t)f(since)h (these)g(discussions)g(compare)g(and)h(con)o(trast)f(the)g(pro)q(cessing)149 2498 y(of)k(a)g(\014xed)f(n)o(um)o(b)q(er)e(of)j(m)o(ultiple)c(disciplines)h (\(meshes\).)223 2558 y(If)k(the)h(disciplines)f(\(meshes\))f(are)i(pro)q (cessed)h(in)f(parallel,)f(assume)h(that)g(the)g(n)o(um)o(b)q(er)f(of)149 2618 y(pro)q(cessors)j(assigned)e(to)h(eac)o(h)e(discipline)f(\(mesh\),)h Ff(G)1190 2625 y Fe(i)1204 2618 y Fk(,)i(is)e(suc)o(h)h(that)g(there)g(is)f (near)i(p)q(erfect)149 2678 y(load)c(balance)f(across)h(disciplines)d (\(meshes\).)19 b(Giv)o(en)14 b(the)h(same)f(n)o(um)o(b)q(er)f(of)j(pro)q (cessors)g(\()p Ff(N)j Fk(=)149 2705 y Fb(P)202 2738 y Ff(G)240 2745 y Fe(i)254 2738 y Fk(\),)14 b(pro)q(cessing)g(disciplines)d(\(meshes\))h (sequen)o(tially)f(m)o(ust)h(ha)o(v)o(e)h(a)h(lo)o(w)o(er)e(e\016ciency)l(.) 18 b(Eac)o(h)1036 2864 y(5)p eop %%Page: 6 6 bop 149 42 a Fk(discipline)13 b(\(mesh\))g(is)h(spread)h(o)o(v)o(er)f(more)f (pro)q(cessors)j(and)f(Amdahl's)d(La)o(w)j(implies)d(e\016ciency)149 102 y(decreases)k(as)h(the)f(n)o(um)o(b)q(er)f(of)h(pro)q(cessors)i (increases)e(for)g(a)h(\014xed)f(problem)e(size.)223 162 y(F)l(or)g(example,) e(assume)i(that)h(a)g(m)o(ultizonal)d(application)i(has)h(t)o(w)o(o)g(iden)o (tical)e(sized)g(meshes)149 222 y(with)g(the)g(same)f(computational)g(load)i (for)f(eac)o(h.)20 b(F)l(urther,)13 b(assume)f(that)h(the)g(computation)f(on) 149 282 y(eac)o(h)k(is)f(99.9\045)h(parallel.)k(Giv)o(en)15 b(1000)i(pro)q(cessors,)g(if)e(the)g(meshes)g(are)g(pro)q(cessed)h(in)g (parallel)149 342 y(\(500)g(pro)q(cessors)f(assigned)g(to)f(eac)o(h)f(mesh\)) g(then)h(the)g(e\016ciency)d(is)j(66)p Ff(:)p Fk(7\045.)21 b(If)13 b(the)h(meshes)f(are)149 403 y(pro)q(cessed)k(sequen)o(tially)e (\(1000)j(pro)q(cessors)f(assigned)h(to)e(eac)o(h)g(mesh\))f(then)i(the)f (e\016ciency)e(is)149 463 y(50)p Ff(:)p Fk(0\045.)21 b(This)14 b(means)e(that)i(pro)q(cessing)h(the)e(meshes)f(sequen)o(tially)f(will)i(tak) o(e)g(33\045)h(longer)f(than)149 523 y(pro)q(cessing)k(them)e(in)h(parallel.) 223 583 y(Other)c(sources)h(of)g(ine\016ciency)d(are)i(an)h(increased)f(comm) o(unication)e(to)j(computation)f(ratio)149 643 y(and)24 b(an)g(increase)f(in) g(redundan)o(t)h(computations)f(as)h(the)f(p)q(ercen)o(tage)g(of)g (duplicated)g(data)149 704 y(increases.)149 831 y Fi(3.5)56 b(Soft)n(w)n(are)20 b(Engineering)149 924 y Fk(Some)d(of)i(the)e(soft)o(w)o (are)i(engineering)e(issues)h(are)g(mo)q(dularit)o(y)l(,)e(indep)q(endence,)h (extensibilit)o(y)l(,)149 984 y(mo)q(di\014abilit)o(y)l(,)d(readabilit)o(y)l (,)h(and)i(main)o(tainabilit)o(y)l(.)i(Soft)o(w)o(are)e(engineering)e(ma)o(y) g(b)q(e)i(though)o(t)149 1044 y(of)h(as)f(complexit)o(y)d(managemen)o(t.)20 b(Tw)o(o)e(primary)d(w)o(a)o(ys)i(h)o(umans)f(deal)h(with)f(complexit)o(y)e (are)149 1104 y(divide-and-conquer)22 b(and)h(abstraction.)41 b(The)23 b(trend)f(o)o(v)o(er)g(the)g(last)g(thirt)o(y)g(y)o(ears)g(in)g (soft-)149 1165 y(w)o(are)17 b(engineering)f(has)h(b)q(een)g(to)o(w)o(ards)g (greater)g(co)q(de)g(mo)q(dularit)o(y)e(and)i(greater)g(information)149 1225 y(hiding.)24 b(Structured)16 b(programming,)g(top-do)o(wn)i(design,)f (program)g(mo)q(dules,)e(abstract)j(data)149 1285 y(structures,)c(ob)s(ject)g (orien)o(ted)f(programming)f(are)i(all)g(examples)e(of)i(this)g(trend.)20 b(They)14 b(encour-)149 1345 y(age)i(the)e(partitioning)h(of)g(problems)f(in) o(to)g(simpler)f(pieces,)g(in)o(terfaces)h(to)h(b)q(e)g(de\014ned,)g(and)g (the)149 1405 y(impleme)o(n)o(tation)f(details)h(to)i(b)q(e)f(hidden)g(b)q (ehind)g(the)g(in)o(terfaces.)149 1533 y Fi(3.6)56 b(Multidisciplinary)16 b(Coupling)149 1626 y Fk(The)d(principal)e(metho)q(ds)h(for)g(coupling)h(m)o (ultiple)c(disciplines)h(are)j(global)f(explicit)f(in)o(tegration,)149 1686 y(global)18 b(implicit)c(in)o(tegration,)j(mo)q(dal)f(equation)i(in)o (tegration,)e(and)i(partitioned)f(analysis)h([6].)149 1746 y(Global)c(explicit)d(in)o(tegration)h(applies)h(an)h(explicit)d(time)f(in)o (tegration)j(algorithm)f(to)i(the)f(global)149 1806 y(set)19 b(of)g(equations.)29 b(Ho)o(w)o(ev)o(er,)17 b(stabilit)o(y)g(is)i(a)g (concern)f(and)h(the)g(time)d(step)j(m)o(ust)e(b)q(e)i(c)o(hosen)149 1866 y(based)e(on)g(the)f(discipline)e(with)i(the)g(sti\013est)h(requiremen)n (ts.)223 1926 y(Mo)q(dal)12 b(equation)f(in)o(tegration)h(represen)o(ts)f (the)g(resp)q(onse)i(of)f(a)g(system)e(b)o(y)h(mo)q(des)h(go)o(v)o(erned)149 1987 y(b)o(y)g(mo)q(dal)g(equations)h(of)f(motion.)19 b(Ho)o(w)o(ev)o(er,)11 b(it)h(is)g(hard)h(to)g(determining)d(a)j(set)f(of)h(basis)f(mo)q(des)149 2047 y(for)17 b(a)g(nonlinear)f(system.)223 2107 y(Global)j(implicit)d (coupling)i(sc)o(heme)f(is)i(used,)g(the)g(problem)f(can)h(b)q(e)g(expressed) g(in)f(blo)q(c)o(k)149 2167 y(matrix)d(form)g(where)h(the)f(main)g(diagonal)i (blo)q(c)o(ks)f(represen)o(t)f(the)h(individual)f(disciplines)f(and)149 2227 y(the)e(o\013-diagonal)i(blo)q(c)o(ks)d(represen)o(t)g(the)g(coupling)h (b)q(et)o(w)o(een)f(the)g(disciplines.)19 b(Then)11 b(the)h(whole)149 2288 y(system)18 b(can)g(b)q(e)h(solv)o(ed)f(as)i(a)f(large)f(single)g (matrix.)27 b(Ho)o(w)o(ev)o(er,)17 b(in)h(man)o(y)g(instances,)g(it)g(ma)o(y) 149 2348 y(not)c(b)q(e)g(p)q(ossible)g(to)g(directly)d(couple)i(t)o(w)o(o)h (disciplines.)k(This)c(is)f(due)h(to)f(di\016culties)f(asso)q(ciated)149 2408 y(with)k(the)g(explicit)e(ev)m(aluation)i(of)g(o\013-diagonal)i(blo)q(c) o(ks.)j(F)l(or)16 b(example,)e(it)h(w)o(ould)h(b)q(e)g(di\016cult)149 2468 y(to)j(directly)e(couple)h(a)h(particle)e(sim)o(ulation)g(of)i(a)f (rare\014ed)h(gas)g(to)g(the)f(structural)h(and)g(ther-)149 2528 y(mal)d(analysis)h(of)g(the)g(b)q(o)q(dy)h(o)o(v)o(er)e(whic)o(h)g(it)g (\015o)o(ws.)24 b(Ev)o(en)16 b(if)g(all)h(o\013-diagonal)i(coupling)d(terms) 149 2589 y(can)k(b)q(e)f(ev)m(aluated)g(explicitly)l(,)d(direct)i(in)o(v)o (ersion)f(of)j(suc)o(h)e(a)i(large)f(sparse)g(matrix)f(w)o(ould)h(b)q(e)149 2649 y(prohibitiv)o(ely)14 b(exp)q(ensiv)o(e)h(in)h(terms)e(of)j(b)q(oth)g (memory)c(and)j(CPU)h(time)c(for)k(problems)e(of)h(an)o(y)149 2709 y(consequence)i(to)h(the)f(aeronautics)h(comm)o(unit)o(y)-5 b(.)26 b(As)18 b(a)h(result,)f(it)g(is)h(necessary)f(to)h(resort)g(to)1036 2864 y(6)p eop %%Page: 7 7 bop 149 42 a Fk(some)17 b(form)f(of)h(iterativ)o(e)e(solution)j(approac)o(h)f (based)h(on)g(the)f(concept)f(of)i(sub-structuring)g(or)149 102 y(partitioned)g(analysis.)26 b(A)17 b(natural)h(form)f(of)h (sub-structuring)g(across)h(discipline)d(b)q(oundaries)149 162 y(w)o(ould)24 b(b)q(e)f(inevitable)f(due)h(to)g(widely)f(di\013eren)o(t)h (n)o(umerical)d(c)o(haracteristics)i(across)i(disci-)149 222 y(plines.)c(A)14 b(monolithic)e(iterativ)o(e)h(solv)o(er)g(is)i(unlik)o(ely)d (to)i(b)q(e)h(able)f(to)h(cop)q(e)f(with)g(all)g(the)g(div)o(erse)149 282 y(disciplines)h(in)h(an)h(e\016cien)o(t)d(manner.)223 342 y(In)h(partitioned)f(analysis,)i(the)f(e\013ect)f(of)i(o\013-diagonal)h(blo)q (c)o(ks)e(are)g(represen)o(ted)f(as)i(forcing)149 403 y(functions)23 b(on)g(the)g(righ)o(t)f(hand)i(side.)40 b(The)23 b(computation)f(of)h(these)f (forcing)h(terms)e(w)o(ould)149 463 y(induce)13 b(in)o(ter-disciplinary)e (comm)o(unication.)18 b(P)o(artitioned)13 b(analysis)g(leads)h(to)g(blo)q(c)o (k)f(iterativ)o(e)149 523 y(metho)q(ds)k(\(blo)q(c)o(k)g(Jacobi,)g(blo)q(c)o (k)g(Gauss-Seidel,)g(etc.\).)23 b(Also,)17 b(with)g(partitioned)g(analysis,)g (it)149 583 y(is)i(easier)g(to)g(extend)f(the)h(application)f(to)h(include)f (new)h(disciplines.)27 b(Ho)o(w)o(ev)o(er,)18 b(robust)h(and)149 643 y(e\016cien)o(t)c(m)o(ultidiscipl)o(inary)f(and/or)k(m)o(ultizonal)c (coupling)i(tec)o(hniques)g(is)g(an)h(activ)o(e)e(area)j(of)149 704 y(researc)o(h.)223 764 y(P)o(ark)j(and)h(F)l(elippa)e([9])h(list)g(man)o (y)e(of)j(the)f(problems)f(asso)q(ciated)i(with)g(global)f(implicit)149 824 y(coupling)16 b(and)h(the)f(adv)m(an)o(tages)i(of)e(partitioned)g (analysis.)21 b(F)l(rom)15 b(a)i(soft)o(w)o(are)f(viewp)q(oin)o(t,)f(the)149 884 y(list)i(of)h(disadv)m(an)o(tages)g(of)g(direct)f(coupling)g(and)h(adv)m (an)o(tages)h(of)e(partitioned)g(analysis)h(could)149 944 y(b)q(e)f(said)f (ab)q(out)i(monolithic)c(programming)h(v)o(ersus)h(mo)q(dular)f(programming)g (resp)q(ectiv)o(ely)l(.)149 1078 y Fi(4)56 b(Computational)18 b(Mo)r(dels)149 1187 y Fk(Giv)o(en)d(the)g(ab)q(o)o(v)o(e)h(considerations)g (\(memory)c(requiremen)o(ts,)h(computational)h(requiremen)o(ts,)149 1247 y(Amdahl's)19 b(La)o(w,)j(soft)o(w)o(are)f(engineering,)f(and)i(m)o (ultidisci)o(pli)o(nary)c(and/or)k(m)o(ultizonal)c(cou-)149 1307 y(pling\))d(the)f(most)g(\\natural")j(approac)o(h)e(to)g(implem)o(en)o (ti)o(ng)e(a)i(m)o(ultidisci)o(plinary)d(and/or)k(m)o(ul-)149 1368 y(tizonal)i(application)g(on)h(parallel)e(pro)q(cessor)i(is)f (partitioned)g(analysis)g(using)g(a)h(MIMD)e(st)o(yle)149 1428 y(of)i(execution)d(across)j(disciplines)e(\(meshes\).)24 b(This)18 b(allo)o(ws)g(n)o(umerical)e(algorithms)h(and)h(pro-)149 1488 y(grams)e(p)q(ertaining)g(to)g(individual)f(disciplines)f(to)j(b)q(e)f(dev)o (elop)q(ed)f(and)h(tested)g(indep)q(enden)o(tly)l(.)149 1548 y(After)h(eac)o(h)g(program)h(has)g(b)q(een)g(v)m(alidated,)f(an)h (additional)g(mo)q(dule)e(is)h(added)h(to)g(eac)o(h)f(that)149 1608 y(deals)g(with)f(the)g(required)f(in)o(terdisciplinary)e(\(in)o (terzonal\))j(comm)o(uni)o(cation.)223 1669 y(Sev)o(eral)g(mo)q(dels)h(of)i (computation)e(supp)q(ort)i(the)f(MIMD)f(st)o(yle)g(of)h(execution)f(across)i (dis-)149 1729 y(ciplines.)24 b(Whic)o(h)17 b(one)g(is)h(b)q(est)g(dep)q (ends)f(up)q(on)i(the)e(ph)o(ysical)g(problem,)f(solution)h(algorithm,)149 1789 y(hardw)o(are)f(resources,)g(and)g(op)q(erating)g(system)e(supp)q(ort)j (a)o(v)m(ailable.)j(Belo)o(w)15 b(w)o(e)g(presen)o(t)f(three)149 1849 y(mo)q(dels)i(and)g(describ)q(e)g(the)g(functionalit)o(y)f(that)i(is)f (desired)f(from)g(eac)o(h.)223 1909 y(The)h(discussions)g(assume)g(the)g (follo)o(wing)f(de\014nitions:)21 b Ff(N)5 b(P)24 b Fk(equals)16 b(the)g(total)g(n)o(um)o(b)q(er)f(of)149 1970 y(pro)q(cessors)g(used,)f(a)g (group)h Ff(G)714 1977 y Fe(i)742 1970 y Fk(is)f(the)g(set)f(of)h(pro)q (cessors)h(assigned)g(to)f(a)g(particular)f(discipline,)149 2030 y Ff(N)5 b(G)17 b Fk(is)e(the)h(total)g(n)o(um)o(b)q(er)e(of)i(groups,)g (and)h Ff(GS)1061 2037 y Fe(i)1091 2030 y Fk(is)e(the)h(n)o(um)o(b)q(er)e(of) i(pro)q(cessors)h(in)e(group)i Ff(G)1919 2037 y Fe(i)1933 2030 y Fk(.)223 2090 y(In)c(all)f(of)i(the)f(mo)q(dels,)f(it)h(is)g(assumed)f (that)i(the)f(user)g(has)h(the)f(option)h(to)g(con)o(trol)e(the)h(set)h(of) 149 2150 y(ph)o(ysical)g(pro)q(cessors)h(assigned)g(to)g(a)g(group)g(and)g (the)f(data)i(la)o(y)o(out)d(on)i(that)g(set)f(of)h(pro)q(cessors.)149 2278 y Fi(4.1)56 b(Static)18 b(Mo)r(del)149 2370 y Fk(In)g(the)g(static)g(mo) q(del,)f(all)h(resource)g(requiremen)o(ts)d(are)j(kno)o(wn)h(at)f(startup)h (\(i.e.,)e Ff(N)5 b(P)i Fk(,)19 b Ff(N)5 b(G)p Fk(,)149 2431 y(and)18 b Ff(GS)313 2438 y Fe(i)345 2431 y Fk(are)f(all)f(kno)o(wn\).)24 b(F)l(unctional)16 b(requiremen)o(ts)e(are)j(that)g(the)g(n)o(um)o(b)q(er)e (of)j(pro)q(cessors)149 2491 y(assigned)23 b(to)e(eac)o(h)g(discipline)f(is)h (indep)q(enden)o(t)f(of)i(the)f(other)h(disciplines,)e(a)i(di\013eren)o(t)f (exe-)149 2551 y(cutable)e(ma)o(y)f(b)q(e)i(loaded)g(in)o(to)f(eac)o(h)g (group,)i(eac)o(h)e(group)i(has)f(its)f(o)o(wn)h(logical)f(n)o(um)o(b)q (ering,)149 2611 y(global)f(op)q(erations)h(suc)o(h)e(as)h(SUM)f(or)g(MAX)g (apply)g(to)h(the)f(lo)q(cal)g(group,)h(and)g(some)e(mec)o(ha-)149 2671 y(nism)f(is)i(pro)o(vided)e(to)i(establish)f(comm)o(unication)e(b)q(et)o (w)o(een)h(an)i(individual)e(pro)q(cessor)j(in)e(one)149 2731 y(group)i(and)e(an)h(individual)e(pro)q(cessor)i(in)f(another)h(group.)1036 2864 y(7)p eop %%Page: 8 8 bop 223 42 a Fk(The)12 b(logical)g(n)o(um)o(b)q(ering)f(within)h(a)h(group,)h (global)e(op)q(erations)i(applying)f(only)f(to)h(the)f(lo)q(cal)149 102 y(group,)24 b(and)f(di\013eren)o(t)e(executables)g(all)g(supp)q(ort)i (indep)q(enden)o(t)f(dev)o(elopmen)n(t)d(and)k(testing)149 162 y(of)g(disciplines.)36 b(The)22 b(indep)q(enden)o(t)f(assignmen)o(t)g(of) h(pro)q(cessors)h(to)f(disciplines)e(allo)o(ws)i(the)149 222 y(application)f(to)g(b)q(e)f(statically)g(load)h(balanced.)34 b(P)o(oin)o(t)20 b(to)h(p)q(oin)o(t)f(comm)o(unication)e(b)q(et)o(w)o(een)149 282 y(groups)g(allo)o(w)e(coupling)h(information)e(to)i(b)q(e)f(passed)h (directly)e(b)q(et)o(w)o(een)h(the)g(pro)q(cessors)h(that)149 342 y(need)f(to)h(kno)o(w)f(the)g(information.)223 403 y(Groups)j(are)f (neither)f(created)h(nor)h(deleted)e(dynamically)l(.)24 b(Ev)o(erything)18 b(is)g(initialized)e(at)149 463 y(startup)25 b(and)f(con)o(tin)o(ues)f(un)o (til)g(application)g(termination.)42 b(Curren)o(tly)l(,)24 b(there)f(are)h(sev)o(eral)149 523 y(applications)19 b([11)q(,)f(10)q(,)h(3]) f(using)i(this)f(mo)q(del)e(at)j(NASA)d(Ames)g(Researc)o(h)i(Cen)o(ter)f (based)h(on)149 583 y(the)d(in)o(tercub)q(e)g(comm)o(uni)o(cation)e(soft)o(w) o(are)i(dev)o(elop)q(ed)f(for)i(the)f(In)o(tel)e(iPSC/860)k([1].)223 643 y(There)11 b(are)h(sev)o(eral)f(problems)f(with)i(the)g(iPSC/860)h (implem)o(en)o(tation)c(of)j(the)g(static)g(mo)q(del.)149 704 y(Debuggers)21 b(do)f(not)g(w)o(ork)f(for)h(m)o(ultiple)d(indep)q(enden)o(t)h (groups)j(of)f(pro)q(cessors.)32 b(Groups)21 b(are)149 764 y(restricted)11 b(to)i(a)f(p)q(o)o(w)o(er-of-t)o(w)o(o)h(n)o(um)o(b)q(er)e (of)h(pro)q(cessors)h(and)g(only)f(one)g(group)h(ma)o(y)e(b)q(e)h(assigned) 149 824 y(to)21 b(a)f(pro)q(cessor)i(thereb)o(y)d(limiti)o(ng)f(the)i(abilit) o(y)f(to)h(load)h(balance)f(the)f(computation.)33 b(Also,)149 884 y(existing)24 b(batc)o(h)g(job)g(submission)g(soft)o(w)o(are,)h(Net)o(w)o (ork)e(Queuing)h(System)f(\(NQS\),)g(do)h(not)149 944 y(supp)q(ort)18 b(this)e(st)o(yle)f(of)i(computation.)149 1072 y Fi(4.2)56 b(Dynamic)17 b(Mo)r(del)149 1165 y Fk(In)g(the)f(dynamic)f(mo)q(del,)g (groups)i(ma)o(y)e(b)q(e)i(created)f(or)h(deleted)e(dynamically)l(.)20 b Ff(N)5 b(P)24 b Fk(and)17 b Ff(N)5 b(G)149 1225 y Fk(v)m(ary)19 b(dynamically)l(.)24 b Ff(GS)622 1232 y Fe(i)655 1225 y Fk(is)18 b(\014xed)f(at)i(group)g(creation.)26 b(F)l(unctional)18 b(requiremen)o(ts)d (are)j(that)149 1285 y(the)c(n)o(um)o(b)q(er)e(of)i(pro)q(cessors)h(assigned) f(to)h(eac)o(h)e(discipline)f(is)h(indep)q(enden)o(t)g(of)h(the)g(other)f (disci-)149 1345 y(plines,)i(a)i(di\013eren)o(t)e(executable)g(ma)o(y)f(b)q (e)i(loaded)h(in)o(to)e(eac)o(h)h(group,)g(eac)o(h)g(group)h(has)g(its)f(o)o (wn)149 1405 y(logical)g(n)o(um)o(b)q(ering,)e(global)i(op)q(erations)h (apply)f(to)h(the)e(lo)q(cal)h(group,)h(mec)o(hanism)o(s)d(for)i(group)149 1466 y(creation)j(and)f(deletion)g(are)g(needed,)g(and)h(some)e(mec)o(hanism) e(to)k(establish)f(comm)o(unic)o(ation)149 1526 y(b)q(et)o(w)o(een)d(an)h (individual)e(pro)q(cessor)j(in)e(one)g(group)i(and)f(an)g(individual)e(pro)q (cessor)i(in)f(another)149 1586 y(group.)223 1646 y(The)k(reasoning)i(for)f (the)g(functionalit)o(y)e(is)i(the)f(same)g(as)h(in)g(the)f(static)h(mo)q (del)e(with)i(the)149 1706 y(additional)f(requiremen)n(ts)d(of)i(dynamic)f (creation)h(and)g(deletion)g(of)g(groups.)28 b(The)19 b(size)e(of)h(an)149 1766 y(individual)23 b(group)h(is)f(\014xed)g(at)h(group)g(creation.)43 b(If)22 b(a)i(group)h(needs)e(increase)f(its)i(size,)f(a)149 1827 y(new)18 b(group)h(ma)o(y)d(b)q(e)i(created)g(and)g(the)g(relev)m(an)o (t)f(information)g(copied)g(from)g(the)g(old)h(group.)149 1887 y(Then)f(the)g(old)g(group)g(name)f(is)h(reassigned)g(to)g(the)g(new)f(group) i(and)f(the)g(old)g(group)h(deleted.)149 1947 y(The)i(create,)g(cop)o(y)f (and)h(delete)e(approac)o(h)j(to)f(dynamic)e(group)i(size)f(is)h(acceptable)f (b)q(ecause)149 2007 y(disciplines)d(often)h(assume)f(a)h(particular)f(data)i (to)f(pro)q(cessor)h(la)o(y)o(out.)23 b(Adding)16 b(pro)q(cessors)j(to)149 2067 y(the)f(curren)o(t)e(group)j(w)o(ould)e(also)i(require)d(a)i(data)g (reorganization.)26 b(Mo)o(ving)17 b(to)h(a)g(new)f(set)h(of)149 2128 y(pro)q(cessors)k(ma)o(y)c(ease)i(the)g(pro)q(cess.)33 b(This)20 b(form)f(of)h(pro)q(cess)h(migration)e(requires)g(that)h(the)149 2188 y(c)o(hange)d(in)f(lo)q(cation)g(of)h(the)f(group)h(b)q(e)f(transmitted) f(to)i(all)f(groups)h(that)g(need)e(to)i(kno)o(w.)149 2316 y Fi(4.3)56 b(Hybrid)18 b(Mo)r(del)149 2408 y Fk(In)e(the)g(h)o(ybrid)g(mo)q (del,)e(the)i(total)h(n)o(um)o(b)q(er)d(of)j(pro)q(cessors)g(used)g(b)o(y)e (the)h(application,)g Ff(N)5 b(P)i Fk(,)16 b(is)149 2468 y(\014xed)k(at)f (application)h(startup.)31 b(Ho)o(w)o(ev)o(er,)18 b(the)h(n)o(um)o(b)q(er)f (of)i(groups,)h Ff(N)5 b(G)p Fk(,)20 b(and)g(their)f(sizes,)149 2528 y Ff(GS)217 2535 y Fe(i)232 2528 y Fk(,)j(v)m(ary)f(at)g(run)o(time.)33 b(F)l(unctional)21 b(requiremen)o(ts)d(are)j(that)g(the)g(n)o(um)o(b)q(er)e (of)j(pro)q(cessors)149 2589 y(assigned)c(to)g(eac)o(h)e(discipline)f(is)i (indep)q(enden)o(t)f(of)h(other)h(disciplines,)d(a)i(di\013eren)o(t)f (executable)149 2649 y(ma)o(y)k(b)q(e)i(loaded)g(in)o(to)f(eac)o(h)g(group,)i (eac)o(h)e(group)i(has)f(its)f(o)o(wn)h(logical)f(n)o(um)o(b)q(ering,)g (global)149 2709 y(op)q(erations)k(apply)f(to)g(the)f(lo)q(cal)g(group,)j (mec)o(hanisms)21 b(for)j(group)g(creation)g(and)g(deletion)1036 2864 y(8)p eop %%Page: 9 9 bop 149 42 a Fk(are)17 b(needed,)e(and)i(some)f(mec)o(hanism)d(to)k (establish)f(comm)o(unication)e(b)q(et)o(w)o(een)h(an)i(individual)149 102 y(pro)q(cessor)h(in)e(one)g(group)h(and)g(an)g(individual)e(pro)q(cessor) i(in)f(another)h(group.)223 162 y(Ha)o(ving)h(a)h(\014xed)f(n)o(um)o(b)q(er)f (of)i(pro)q(cessors)h(assigned)g(to)f(the)f(application)h(simpli\014es)d (some)149 222 y(problems)d(and)h(creates)g(others.)20 b(Keeping)14 b(the)f(group)i(information)e(consisten)o(t)g(across)i(groups)149 282 y(is)h(simpli\014ed)e(b)q(ecause)j(it)e(is)h(a)h(b)q(ounded)g(problem)d (and)j(tables)f(ma)o(y)f(b)q(e)h(k)o(ept)g(on)g(a)h(pro)q(cessor)149 342 y(basis)e(rather)f(than)h(a)g(group)g(basis.)21 b(Ho)o(w)o(ev)o(er,)12 b(what)j(happ)q(ens)g(when)f(a)h(group)g(is)f(created)f(and)149 403 y(there)19 b(are)g(no)g(idle)f(pro)q(cessors?)30 b(P)o(ossible)19 b(solutions)g(are)g(to)g(either)f(ab)q(ort)i(the)f(application,)149 463 y(inform)d(the)g(application)h(that)g(it)f(is)h(out)g(of)g(pro)q (cessors,)h(or)f(allo)o(w)g(m)o(ultiple)c(groups)18 b(p)q(er)f(pro-)149 523 y(cessor)i(and)h(time)c(slice)i(and)h(share)g(lo)q(cal)g(memory)d(\(mem)o (ory)g(slice\))i(across)h(groups.)30 b(Recall)149 583 y(that)17 b(parallel)f(pro)q(cessors)h(are)f(b)q(eing)h(considered)e(ph)o(ysical)h (memory)d(mac)o(hines.)223 643 y(The)j(most)f(\015exible)g(solution)h(is)g (informing)f(the)h(application)g(that)g(it)g(has)h(run)f(out)h(of)f(idle)149 704 y(pro)q(cessors)24 b(and)e(let)f(it)h(decide)f(if)g(it)h(w)o(an)o(ts)g (to)g(memory)d(slice)i(or)h(ab)q(ort.)40 b(When)22 b(memory)149 764 y(slicing,)16 b(the)g(user)h(should)g(ha)o(v)o(e)f(the)h(option)g(to)g (sp)q(ecify)f(a)i(logical)e(pro)q(cessor)i(la)o(y)o(out)e(and)h(the)149 824 y(set)i(of)g(ph)o(ysical)f(pro)q(cessors)i(that)f(are)g(to)g(b)q(e)g (memory)d(sliced.)28 b(This)19 b(implies)d(the)i(abilit)o(y)g(of)149 884 y(an)f(application)f(to)h(in)o(terrogate)f(the)g(state)g(of)h(an)o(y)f (pro)q(cessor.)223 944 y(With)j(mem)o(ory)e(slicing,)h(it)h(is)g(also)h (desirable)e(to)i(ha)o(v)o(e)e(group)i(migration)e(as)i(an)g(option.)149 1005 y(This)e(is)f(an)h(issue)g(where)f(solution)h(adaptiv)o(e)f(algorithms)f (are)i(used.)25 b(Assume)16 b(that)i(a)g(region)149 1065 y(needs)h (re\014nemen)o(t.)27 b(Then)19 b(a)h(new)f(grid)g(with)g(greater)g (re\014nemen)o(t)d(can)j(b)q(e)h(placed)e(o)o(v)o(er)g(the)149 1125 y(region.)32 b(The)19 b(new)h(grid)f(is)g(treated)h(as)g(a)g(logically)e (separate)i(group.)32 b(In)19 b(other)h(places,)f(the)149 1185 y(grid)k(ma)o(y)d(b)q(e)i(coarsened.)40 b(In)21 b(those)i(places,)g(groups)g (terminate.)37 b(As)22 b(groups)h(terminate,)149 1245 y(pro)q(cessors)c(ma)o (y)c(b)q(ecome)h(idle)g(while)g(other)h(pro)q(cessors)i(are)e(memory)d (slicing.)23 b(Under)16 b(these)149 1306 y(conditions,)g(the)g(user)h(ma)o(y) d(w)o(an)o(t)i(groups)i(to)e(migrate)f(to)i(the)f(idle)f(pro)q(cessors.)223 1366 y(One)d(of)h(the)g(adv)m(an)o(tages)h(of)f(the)f(h)o(ybrid)g(mo)q(del)g (is)g(that)i(it)e(could)g(b)q(e)h(submitted)e(as)j(a)f(batc)o(h)149 1426 y(job.)22 b(The)16 b(resources)h(required)e(are)h(\014xed)g(and)h(sp)q (eci\014able)e(at)i(application)f(instan)o(tiation.)149 1559 y Fi(5)56 b(Conclusion)149 1669 y Fk(Based)22 b(on)f(memory)e(requiremen)n (ts,)h(memory)e(e\016ciency)l(,)i(Amdahl's)f(La)o(w,)k(computational)149 1729 y(requiremen)o(ts,)d(computational)h(e\016ciency)l(,)f(soft)o(w)o(are)i (engineering,)g(and)g(m)o(ultidisci)o(plinary)149 1789 y(\(m)o(ultizonal\))14 b(coupling)i(a)g(computing)f(mo)q(del)g(that)h(supp)q(orts)h(a)g(MIMD)e(st)o (yle)g(of)h(computing)149 1849 y(across)i(disciplines)c(is)i(the)g(most)g (natural.)223 1909 y(Three)c(computational)h(mo)q(dels)f(w)o(ere)g(in)o(tro)q (duced.)20 b(The)13 b(static)g(mo)q(del)e(is)i(curren)o(tly)f(b)q(eing)149 1970 y(used)20 b(at)h(NASA)d(Ames)g(for)i(m)o(ultidiscipli)o(nary)d(and)k(m)o (ultizonal)c(applications.)32 b(W)l(e)20 b(exp)q(ect)149 2030 y(to)k(mo)o(v)o(e)c(to)o(w)o(ards)j(the)g(h)o(ybrid)f(mo)q(del)f(in)h(the)h (next)f(y)o(ear)g(to)h(supp)q(ort)h(solution)f(adaptiv)o(e)149 2090 y(metho)q(ds.)149 2223 y Fi(References)174 2333 y Fk([1])h(E.)15 b(Barszcz.)k(In)o(tercub)q(e)14 b(comm)o(unicati)o(on)f(for)j(the)f (ipsc/860.)21 b(In)15 b Fa(Sc)n(alable)k(High)e(Perfor-)250 2393 y(manc)n(e)h(Computing)g(Confer)n(enc)n(e)g(`92)f(pr)n(o)n(c)n(e)n(e)n (dings)p Fk(,)e(pages)i(307{313,)h(April)d(1992.)174 2495 y([2])24 b(E.)c(Barszcz,)h(T.)g(F.)f(Chan,)i(D.)f(C.)f(Jesp)q(ersen,)i(and)f(R.)g(S.)f (T)l(uminaro.)34 b(Flo52)22 b(on)f(h)o(y-)250 2555 y(p)q(ercub)q(es:)27 b(P)o(erformance)18 b(and)h(mo)q(delling)f(of)i(a)f(m)o(ultigrid)e(euler)h (co)q(de.)30 b Fa(International)250 2615 y(Journal)17 b(of)g(High)h(Sp)n(e)n (e)n(d)f(Computing)p Fk(,)g(1\(3\):481{503,)h(1989.)1036 2864 y(9)p eop %%Page: 10 10 bop 174 42 a Fk([3])24 b(E.)17 b(Barszcz,)g(S.)g(K.)h(W)l(eeratunga,)g(and)g (R.)f(L.)h(Meaking.)25 b(Dynamic)16 b(o)o(v)o(erset)h(grid)h(com-)250 102 y(m)o(unication)c(on)j(distributed)f(memory)d(parallel)j(pro)q(cessors.) 23 b(T)l(o)17 b(b)q(e)g(presen)o(ted)e(at)i(11th)250 162 y(AIAA)d (Computational)i(Fluid)g(Dynamics)e(Conference,)i(Orlando)g(Florida,)g(July)g (1993.)174 264 y([4])24 b(J.)c(A.)g(Benek,)h(P)l(.)f(G.)h(Buning,)g(and)h(J.) e(L.)h(Steger.)35 b(A)20 b(3-d)i(c)o(himera)d(grid)i(em)o(b)q(edding)250 324 y(tec)o(hnique.)k(American)15 b(Institute)j(of)g(Aeronautics)g(and)h (Astronautics)f(pap)q(er)h(85-1523,)250 384 y(July)c(1985.)174 486 y([5])24 b(R.)11 b(F.)g(V)l(an)g(der)h(Wijngaart.)i(E\016cien)o(t)c (implem)o(en)o(tation)f(of)i(a)h(3-dimensional)f(adi)h(metho)q(d)250 546 y(on)22 b(the)f(ipsc/860.)39 b(Submitted)20 b(to)i(Sup)q(erComputing)g (`93",)h(NASA)e(Ames)f(Researc)o(h)250 606 y(Cen)o(ter,)15 b(Mo\013ett)h(Field,)f(CA)h(94035.)174 708 y([6])24 b(C.)f(A.)f(F)l(elippa)h (and)g(T.)h(L.)f(Geers.)42 b(P)o(artitioned)23 b(analysis)g(for)h(coupled)e (mec)o(hanical)250 768 y(systems.)31 b(College)20 b(of)g(Engineering)g(and)h (Applied)e(Science,)g(Univ)o(ersit)o(y)e(of)k(Colorado,)250 828 y(Boulder,)15 b(CO)h(80309.)174 930 y([7])24 b(R.)19 b(W.)g(Ho)q(c)o (kney)g(and)h(C.)g(R.)f(Jesshop)q(e.)32 b Fa(Par)n(al)r(lel)22 b(Coputers)p Fk(.)31 b(Adam)19 b(Hilger,)g(Bristol)250 990 y(and)e(Philadelphia,)e(2nd)h(edition,)g(1988.)174 1092 y([8])24 b(R.)16 b(L.)i(Meakin.)23 b(A)16 b(new)i(metho)q(d)e(for)i(establishing)f(in) o(ter-grid)f(comm)o(unic)o(ation)f(among)250 1152 y(systems)k(of)i(o)o(v)o (erset)e(grids.)35 b(American)18 b(Institute)i(of)g(Aeronautics)h(and)g (Astronautics)250 1212 y(pap)q(er)c(91-1586,)h(June)e(1991.)174 1314 y([9])24 b(K.)e(C.)h(P)o(ark)h(and)f(C.)g(A.)g(F)l(elippa.)41 b(P)o(artitioned)23 b(analysis)g(of)h(coupled)e(systems.)41 b(In)250 1374 y(T.)18 b(Belytsc)o(hk)o(o)f(and)i(T.)f(J.)h(R.)f(Hughes,)h (editors,)g Fa(Computational)h(Metho)n(ds)f(for)g(T)l(r)n(an-)250 1434 y(sient)d(A)o(nalysis)p Fk(,)f(c)o(hapter)f(3,)h(pages)g(157{219.)i (Elsevier)d(Scienc)f(Publishers)h(B.V.,)f(North)250 1494 y(Holland,)i (Amsterdam,)e(New)j(Y)l(ork,)f(and)i(Oxford,)f(1983.)149 1596 y([10])25 b(E.)14 b(Pramono)g(and)i(S.)e(K.)g(W)l(eeratunga.)19 b(Aero)q(elastic)13 b(computations)h(for)h(wings)g(through)250 1656 y(direct)27 b(coupling)h(on)h(distributed-mem)o(ory)d(mim)o(d)g (parallel)h(computers.)56 b(W)l(ork)28 b(in)250 1716 y(progress)20 b(at)f(the)g(Applied)f(Researc)o(h)h(Branc)o(h,)g(NAS)f(System)g(Division,)h (NASA)f(Ames)250 1777 y(Researc)o(h)d(Cen)o(ter,)g(Mo\013ett)i(Field,)d(CA)i (94035.)149 1878 y([11])25 b(J.S.)13 b(Ry)o(an)g(and)h(S.)g(K.)f(W)l (eeratunga.)k(P)o(arallel)c(computation)g(of)h(3-d)g(na)o(vier-stok)o(es)f (\015o)o(w-)250 1939 y(\014eld)f(for)i(sup)q(ersonic)f(v)o(ehicles.)h (American)d(Institute)i(of)g(Aeronautics)g(and)h(Astronautics)250 1999 y(pap)q(er)j(93-0064,)h(Jan)o(uary)f(1993.)149 2100 y([12])25 b(Thinking)c(Mac)o(hines)g(Corp)q(oration,)j(Cam)o(bridge,)d(MA.)36 b Fa(CM)22 b(F)l(ortr)n(an)g(Pr)n(o)n(gr)n(amming)250 2161 y(Guide)p Fk(,)16 b(v)o(ersion)f(1.0)i(edition,)e(Jan)o(uary)i(1991.)1024 2864 y(10)p eop %%Trailer end userdict /end-hook known{end-hook}if %%EOF From owner-mpi-comm@CS.UTK.EDU Sun Mar 28 16:01:54 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA16813; Sun, 28 Mar 93 16:01:54 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28455; Sun, 28 Mar 93 15:59:48 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Sun, 28 Mar 1993 15:59:47 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from [129.215.56.21] by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA28447; Sun, 28 Mar 93 15:59:45 -0500 Date: Sun, 28 Mar 93 21:59:29 BST Message-Id: <23593.9303282059@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: Multidisciplinary Position Paper To: barszcz@nas.nasa.gov (Eric Barszcz), mpi-comm@cs.utk.edu In-Reply-To: Eric Barszcz's message of Fri, 26 Mar 93 16:40:19 -0800 Reply-To: lyndon@epcc.ed.ac.uk Hi Eric Thanks for mailing this. It's a good example of a "parallel module graph" program. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Mon Mar 29 09:46:10 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12851; Mon, 29 Mar 93 09:46:10 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22553; Mon, 29 Mar 93 09:43:12 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 29 Mar 1993 09:43:11 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA22526; Mon, 29 Mar 93 09:43:09 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16002; Mon, 29 Mar 1993 09:43:08 -0500 Date: Mon, 29 Mar 1993 09:43:08 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303291443.AA16002@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Revised agenda for MPI meeting Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-3:00 Intro to context sub-committee proposals - Skjellum (10 mins.) Proposal VII - Littlefield and Clarke present (30 mins.) Discussion on VII - lead by Littlefield and Clarke (15 mins.) Proposal I - Snir (30 mins.) Discussion on I - lead by Snir (15 mins.) Proposals III and VIII - Skjellum presents (30 mins.) Discussion on III and VIII - lead by Skjellum and Sears (15 mins.) Overall discussion, Recommendations, and Ranking poll (35 mins.) 3:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Continued discussion of Snir, Gropp, Lusk point-to-point proposal (Snir) OR Discussion of Snir & Geist collection communication proposal (Otto) 12:00-1:30 Lunch (provided) 1:30-3:00 Discussion of Snir & Geist collection communication proposal (Otto) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) From owner-mpi-comm@CS.UTK.EDU Mon Mar 29 18:14:27 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA27088; Mon, 29 Mar 93 18:14:27 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20945; Mon, 29 Mar 93 18:02:00 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Mon, 29 Mar 1993 18:01:59 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA20937; Mon, 29 Mar 93 18:01:56 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA03153; Mon, 29 Mar 93 23:01:48 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA01225; Mon, 29 Mar 93 16:00:29 MST Date: Mon, 29 Mar 93 16:00:29 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303292300.AA01225@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Resend of unofficial "opinion survey" Hi everyone, Some folks have requested that the unofficial "MPI Opinion Survey" be distributed again. Here it is. Please ignore if you don't need to see it again! :-) Tom ----------------------------------------------------------------------------- Hi all, Steve Huss-Lederman writes: > I would like to propose that we have a meeting of all participants at > the start of the next meeting to settle the issue of what MPI-1 will > cover. Several mail messages have expressed concern that we cannot > conclude by June if we try to incorporate everything that everyone > wants. Since the scope and time frame will effect most of the other > discussions I think it would be best to begin with this. My only > concern is that it will take too long to decide the global picture. > I would hope it could be done in 1-2 hours and help to refocus the > group. I fully support this idea. Here's a proposal for an opinion survey that could shorten and focus the discussion. This proposal originated with Jon Flower and has had input from Ian Glendinning, Steve Huss-Lederman, Leslie Hart, myself, and others. If you think this is a good idea, please complete the survey and return to me at: hender@fsl.noaa.gov If you think this is a bad idea, please say so! We will collect responses and distribute a summary via email before the next meeting if there is sufficient interest. Tom Henderson ----------------------------------------------------------------------------- MPI PRIORITIES OPINION SURVEY Several MPI committee members have expressed concern about the current and future directions of MPI. The MPI committee began discussion using an existing document (Dongarra et al) as a starting point. This document contains background rationale and a statement of direction for the MPI standardization effort. Since these issues were consistent with the original MPI proposal, they have not recently been discussed in detail by the MPIF. The current MPI proposals are very different from the original MPI document. As a result, we feel that the original rationale and direction need to be re-examined. In particular we need to prioritize and limit the issues that MPI will try to resolve so that o the initial MPI standard document ("MPI-1") is delivered in a timely fashion and o the functions that it defines can (and will) be implemented on major platforms within a reasonable time frame. To get discussion of these issues started, we propose an informal, email "opinion survey". The objective of the survey is to find out which MPI features are most important. High-priority features would be included in MPI-1. Lower-priority features might be deferred to a later version of MPI. Some features might never be included. The compiled results of this survey could be used as a basis for discussion at the next MPI meeting. *** This survey is in no way binding and is a completely unofficial "straw poll" arranged by its authors for their own personal information. It is not intended to distract attention from the current proposals. Any data collected will be made available to the group at large if requested. *** Following is a ballot containing possible MPI features and goals. Please rate each item (or sub-item) using the following rating scheme: 0 This feature should not appear in any version of MPI ever. 1-2 MPIF should consider including this feature in some future version of MPI (after MPI-1). 3-4 This feature should be included as an Annex to MPI-1 for possible inclusion in a future version of MPI. 5-6 This feature should be included as an Annex to MPI-1 for definite inclusion in a future version of MPI. 7-10 This feature should be included in MPI-1. In this context an "Annex" is a part of the standard document that will not be required by a conforming implementation of MPI. Rather it should be taken as a recommendation by the MPIF to vendors for the way in which particular features should be implemented, if they are implemented at all. This way features that are not usefully implementable on particular hardware can be eliminated while machines that do support these features do so in a portable manner. When replying to the survey, please use the response form attached to the end of this message. (This will make compilation of the results a lot less painful!) :-) +-------------------------------------------------------+ | POSSIBLE FEATURES FOR INITIAL AND FUTURE MPI VERSIONS | +-------------------------------------------------------+ Ordering is for convenience only. This list is not meant to suggest any priority. GENERAL FEATURES 1) Routines for static process creation: A) Same program in every node. B) Host-node programming model in which there is one "special node". C) Static model but with potentially different programs in different groups of nodes. D) Static model with a way to create more than one process per processor. (Heavy duty processes, not threads). 2) Routines for identification of statically created processes. 3) Associated environmental inquiry routines. 4) Some way of avoiding message tag conflicts between independently developed libraries and user code (contexts). 5) Heterogeneous communication. A) All communication routines should support heterogeneous communication. B) A special set of communication routines should support heterogeneous communication. 6) Creation/destruction of "static" groups (processes cannot join/leave an existing group). Groups are used to limit the scope of collective operations. A) No logical process topology is associated with a group. B) Grid/toroidal process topology is associated with a group at the time the group is created. (The default "ALL" group has a default topology.) C) General graph topology is also supported. D) Provide access to all relevant information (via topological environmental inquiry routines) to allow users to define custom topologies. 7) Support for communication between processes in different groups without reference to the "ALL" group. 8) Dynamic groups (processes can join/leave an existing group while the total number of processes remains static.) 9) Dynamic process creation and identification. 10) Additional environmental inquiry and "setting" routines for performance optimization. A) Inquire/suggest system buffering resource allocation. B) Inquire relative node performance. C) Inquire relative interconnect performance. 11) Termination of programs. A) Kill all others nodes. B) Wait for all the other nodes to complete. C) Automatically de-allocate resources. 12) Standard organization. A) "Multi-level" organization of routines (users may use simple features without having to understand all of MPI). B) Simple "core" standard with (optional) Annexes. C) Simple "core" standard without Annexes. D) Should an MPI "subset" be defined that can be quickly implemented as a layer on top of existing systems? (This question was inspired by section 1.5 of Bill and Rusty's "A Test Implementation of the MPI Draft Message-Passing Standard", ANL-92/47.) 13) Should there be future version(s) of MPI (MPI-2, etc.) (yes/no)? POINT-TO-POINT COMMUNICATION 14) Blocking send and receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). 15) Non-blocking receive. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 16) Non-blocking send. A) Contiguous data. B) Strided data. C) Buffer-descriptor (multiple data locations, multiple data types). D) Check for completion (status()). E) Wait for completion (wait()). F) Cancel operation (cancel()). 17) Low-level functionality (from Marc Snir's proposal) visible to users. A) Initialize. B) Start. C) Complete. D) Free. 18) "SECURE" versions of send and receive as proposed by Lyndon Clarke on 3/3/93. (Correct programs using SECURE routines are guaranteed to work regardless of system buffering resources.) 19) Ready-send. 20) Ready-receive. 21) Message handlers (a la iNTEL hrecv, TMC active messages, Express exhandle). A) Simple definition sufficient to support anticipated MPI requirements and make a self-consistent system. B) More advanced routines for remote procedure calls, etc... 22) Additional point-to-point primitives. A) Exchange between two processors. B) "Shift-exchange": read from one, send to another (superset of "A"). C) Contiguous data. D) Strided data. E) Buffer-descriptor (multiple data locations, multiple data types). F) Non-blocking versions? G) Check for completion (status()). H) Wait for completion (wait()). I) Cancel operation (cancel()). COLLECTIVE OPERATIONS 23) Blocking collective communication. A) Broadcast. B) Concatenate. C) Barrier. D) Sum. E) Extrema (max, min). F) Average. G) Sum of squared differences (error). H) Should strided data be allowed for these operations? I) Should buffer-descriptor be allowed for these operations? 24) "User-defined" blocking collective operations (Express "excombine()" style). 25) Non-blocking collective communication, collective operations, and user-defined operations (if this is desired, detailed ranking can be done later). 26) If any non-blocking collective operations are supported, what should completion/cancel options be? A) Check for completion (status()). B) Wait for completion (wait()). C) "Multiple cancel" operation (cancel()). 27) Should status()/wait()/cancel() syntax be consistent across all types of non-blocking operations? OTHER COMMUNICATION OPERATIONS 28) Probe (and associated "precv()" routines, if necessary). I/O 29) Standard I/O is supported in the standard fashion for the language on one process (no file I/O). This process can be identified by an environmental inquiry routine. (This is the current proposal.) 30) Serial I/O is supported in the standard fashion for the language on one process. This process can be identified by an environmental inquiry routine. (Rik Littlefield's proposal.) 31) Multiplexed I/O similar to that currently provided in systems such as CMMD (TMC), VERTEX (nCUBE) and Express (ParaSoft). Basically a way of defining the behavior of multi-node I/O statements which interact with a sequential device. No special considerations for parallel I/O channels. 32) Blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. 33) Non-blocking parallel I/O. Intended to take advantage of parallel I/O channels, where available. LANGUAGE BINDINGS 34) Ada 35) C 36) C++ 37) Fortran 77 38) Fortran 90 39) Other IMPLEMENTATION PERIOD There will be a delay between the generation of the standard and its implementation on various platforms. If this delay is too long users will continue to rely more and more heavily on vendor routines and specialized calls and MPI will become less and less attractive, particularly if new technology such as Fortran 90/HPF becomes effective. 40) What is the maximum delay that you think MPI can stand and still succeed (please feel free to comment if you wish)? ---------------------------------------------------------------------------- SURVEY RESPONSE FORM 1) A) B) C) D) 2) 3) 4) 5) A) B) 6) A) B) C) D) 7) 8) 9) 10) A) B) C) 11) A) B) C) 12) A) B) C) D) 13) (yes/no) 14) A) B) C) 15) A) B) C) D) E) F) 16) A) B) C) D) E) F) 17) A) B) C) D) 18) 19) 20) 21) A) B) 22) A) B) C) D) E) F) G) H) I) 23) A) B) C) D) E) F) G) H) I) 24) 25) 26) A) B) C) 27) 28) 29) 30) 31) 32) 33) 34) 35) 36) 37) 38) 39) 40) (free form): Comments? From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 00:51:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA02775; Tue, 30 Mar 93 00:51:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06540; Tue, 30 Mar 93 00:49:03 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 00:49:02 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Mail.Think.COM by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA06532; Tue, 30 Mar 93 00:49:01 -0500 Received: from Basselope.Think.COM by mail.think.com; Tue, 30 Mar 93 00:48:58 -0500 From: Alan Mainwaring Received: by basselope.think.com (4.1/Think-1.2) id AA15508; Tue, 30 Mar 93 00:48:57 EST Date: Tue, 30 Mar 93 00:48:57 EST Message-Id: <9303300548.AA15508@basselope.think.com> To: mpi-comm@cs.utk.edu In-Reply-To: <9303291443.AA16002@rios2.epm.ornl.gov>, "David Walker's message of Mon, 29 Mar 1993 09:43:08 -0500" Subject: Revised agenda for MPI meeting Date: Mon, 29 Mar 1993 09:43:08 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Provisional Agenda for MPI Meeting, March 31-April 2, 1993 Wednesday 1:30-3:00 Intro to context sub-committee proposals - Skjellum (10 mins.) Proposal VII - Littlefield and Clarke present (30 mins.) Discussion on VII - lead by Littlefield and Clarke (15 mins.) Proposal I - Snir (30 mins.) Discussion on I - lead by Snir (15 mins.) Proposals III and VIII - Skjellum presents (30 mins.) Discussion on III and VIII - lead by Skjellum and Sears (15 mins.) Overall discussion, Recommendations, and Ranking poll (35 mins.) 3:30-6:00 Discussion of Snir, Gropp, Lusk point-to-point proposal (everyone) (Snir) 6:00-7:30 Unofficial dinner break 7:30-10:30 Break up for subcommittee meetings Thursday 9:00-12:00 Continued discussion of Snir, Gropp, Lusk point-to-point proposal (Snir) OR Discussion of Snir & Geist collection communication proposal (Otto) 12:00-1:30 Lunch (provided) 1:30-3:00 Discussion of Snir & Geist collection communication proposal (Otto) 3:00-6:00 Full group meeting for presentation of process topology subcommittee ideas and proposals. (Hempel) 6:00-8:00 Dinner (attendees pay, but hotel provides transport to area restaurant) 8:00-10:00 Continued informal subcommittee meetings if necessary Friday 9:00-11:00 Full group meeting with the intent of taking binding votes on point-to-point and collective communication proposals, or sending proposals back to subcommittees for revision. (Snir) 11:00-12:00 Full group meeting for defining timetable for producing MPI (or subset) by deadline in July. (Dongarra) After reviewing this schedule, we believe that the last item should be moved so that it is the first order of business at this meeting. If it is a goal for vendors to quickly implement and release MPI, then, it is imperative that we understand what functionality must be included in MPI1, and what functionality may be deferred to subsequent versions. Without this framework we seriously doubt that this process will yield a specification, against which vendors can implement production software this year. Respectfully, Alan and Moose From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 01:16:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA03507; Tue, 30 Mar 93 01:16:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07612; Tue, 30 Mar 93 01:15:09 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 01:15:08 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from ssd.intel.com by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA07555; Tue, 30 Mar 93 01:15:05 -0500 Received: from tualatin.SSD.intel.com by SSD.intel.com (4.1/SMI-4.1) id AA11172; Mon, 29 Mar 93 22:15:02 PST Date: Mon, 29 Mar 93 22:15:02 PST Message-Id: <9303300615.AA11172@SSD.intel.com> Received: by tualatin.SSD.intel.com (4.1/SMI-4.0) id AA15164; Mon, 29 Mar 93 22:14:59 PST From: Bob Knighten Sender: knighten@SSD.intel.com To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting In-Reply-To: <9303300548.AA15508@basselope.think.com> References: <9303291443.AA16002@rios2.epm.ornl.gov> <9303300548.AA15508@basselope.think.com> Reply-To: knighten@SSD.intel.com (Bob Knighten) Alan Mainwaring writes: > Date: Mon, 29 Mar 1993 09:43:08 -0500 > From: walker@rios2.epm.ornl.gov (David Walker) > > > Provisional Agenda for MPI Meeting, March 31-April 2, 1993 > > . . . > > > After reviewing this schedule, we believe that the last item should be > moved so that it is the first order of business at this meeting. > > If it is a goal for vendors to quickly implement and release MPI, then, it > is imperative that we understand what functionality must be included in MPI1, > and what functionality may be deferred to subsequent versions. Without this > framework we seriously doubt that this process will yield a specification, > against which vendors can implement production software this year. > > Respectfully, > > Alan and Moose > I strongly support this re-ordering of the agenda. -- Bob Robert L. Knighten | knighten@ssd.intel.com Intel Supercomputer Systems Division | 15201 N.W. Greenbrier Pkwy. | (503) 629-4315 Beaverton, Oregon 97006 | (503) 629-9147 [FAX] From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:35:53 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12387; Tue, 30 Mar 93 10:35:53 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08221; Tue, 30 Mar 93 10:33:28 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:33:28 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA08206; Tue, 30 Mar 93 10:33:26 -0500 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA01022; Tue, 30 Mar 93 10:33:20 EST Received: by b125.super.org (4.1/SMI-4.1) id AA03597; Tue, 30 Mar 93 10:33:19 EST Date: Tue, 30 Mar 93 10:33:19 EST From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9303301533.AA03597@b125.super.org> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting In response to the recent message by Alan and Moose, I want to reiterate that I support an early discussion of the goals and timetable for MPI. This will allow the general framework to be set when we decide on how to deal with specific proposals. Given the fact that some of the MPI members are already on travel to the meeting, I suspect that this change in agenda would have to be dealt with at the start of the meeting tomorrow. Steve From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:42:09 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12524; Tue, 30 Mar 93 10:42:09 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09326; Tue, 30 Mar 93 10:40:32 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:40:30 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09303; Tue, 30 Mar 93 10:40:28 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA19731; Tue, 30 Mar 93 08:40:24 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0ndiQJ-0016bbC; Tue, 30 Mar 93 08:40 MST Message-Id: Date: Tue, 30 Mar 93 08:40 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I must disagree with the suggestion to change the schedule. Without any agreement to even basic, fundamental concepts how are we to decide what will go into MPI1 and what won't. I do indeed understand the desire to know the scope of MPI1, but I really believe the scope depends on an agreed method to achieve contexts, groups, etc. It is those definitions that will define how much work a given set of MPI routines will require. Stephen From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:48:58 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12768; Tue, 30 Mar 93 10:48:58 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09919; Tue, 30 Mar 93 10:47:25 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:47:23 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from rios2.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA09910; Tue, 30 Mar 93 10:47:23 -0500 Received: by rios2.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA19794; Tue, 30 Mar 1993 10:47:22 -0500 Date: Tue, 30 Mar 1993 10:47:22 -0500 From: walker@rios2.epm.ornl.gov (David Walker) Message-Id: <9303301547.AA19794@rios2.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Revised agenda for MPI meeting I don't think it's a good idea to change the agenda. There seems little point on talking aboutwhat should be in MPI until everyone knows what has been proposed, particularly for contexts. Setting a timetable will be very difficulty if we can't largely agree on point-to-point, collective communication, and contexts, which is why these topics come first. David From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 10:51:19 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA12832; Tue, 30 Mar 93 10:51:19 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10133; Tue, 30 Mar 93 10:49:36 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 10:49:34 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA10095; Tue, 30 Mar 93 10:49:32 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AB04355; Tue, 30 Mar 93 15:49:27 GMT Received: by macaw.fsl.noaa.gov (4.1/SMI-4.1) id AA02302; Tue, 30 Mar 93 08:48:08 MST Date: Tue, 30 Mar 93 08:48:08 MST From: hender@macaw.fsl.noaa.gov (Tom Henderson) Message-Id: <9303301548.AA02302@macaw.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting Alan Mainwaring writes, > After reviewing this schedule, we believe that the last item should be > moved so that it is the first order of business at this meeting. > > If it is a goal for vendors to quickly implement and release MPI, then, it > is imperative that we understand what functionality must be included in MPI1, > and what functionality may be deferred to subsequent versions. Without this > framework we seriously doubt that this process will yield a specification, > against which vendors can implement production software this year. > > Respectfully, > > Alan and Moose I agree. It would be very helpful if vendors would speak frankly about how long implementation might take during this discussion (wishful thinking? :-). Tom Henderson NOAA Forecast Systems Laboratory hender@fsl.noaa.gov From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 11:45:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA14240; Tue, 30 Mar 93 11:45:20 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12714; Tue, 30 Mar 93 11:43:42 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 11:43:40 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from fslg8.fsl.noaa.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA12705; Tue, 30 Mar 93 11:43:38 -0500 Received: by fslg8.fsl.noaa.gov (5.57/Ultrix3.0-C) id AA04547; Tue, 30 Mar 93 16:43:34 GMT Received: by nipmuc.fsl.noaa.gov (4.1/SMI-4.1) id AA08559; Tue, 30 Mar 93 09:46:14 MST Date: Tue, 30 Mar 93 09:46:14 MST From: hart@nipmuc.fsl.noaa.gov (Leslie Hart) Message-Id: <9303301646.AA08559@nipmuc.fsl.noaa.gov> To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I strongly agree with Steve (see below) on both points. I think that we need to discuss MPI priorities early at the meeting (definitely not the last agenda item!). I agree also since it is likely that a number of people are already traveling, the agenda itself should be the first agenda item. Regards, Leslie Hart (hart@fsl.noaa.gov) ----- Begin Included Message ----- From: lederman@b125.super.org (Steve Huss-Lederman) In response to the recent message by Alan and Moose, I want to reiterate that I support an early discussion of the goals and timetable for MPI. This will allow the general framework to be set when we decide on how to deal with specific proposals. Given the fact that some of the MPI members are already on travel to the meeting, I suspect that this change in agenda would have to be dealt with at the start of the meeting tomorrow. Steve ----- End Included Message ----- From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 13:55:34 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA17414; Tue, 30 Mar 93 13:55:34 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19024; Tue, 30 Mar 93 13:53:52 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 13:53:51 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from Aurora.CS.MsState.Edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA19016; Tue, 30 Mar 93 13:53:50 -0500 Received: by Aurora.CS.MsState.Edu (4.1/6.0s-FWP); id AA14792; Tue, 30 Mar 93 12:43:16 CST Date: Tue, 30 Mar 93 12:43:16 CST From: Tony Skjellum Message-Id: <9303301843.AA14792@Aurora.CS.MsState.Edu> To: amm@Think.COM, mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I have also asked David to make this change. I repeat that this step is important, and I would not mind being delayed in my context group's presentations, to make room for said discussion. Please note, however, contexts must finish by noon on Thursday, if the agenda is changed, because I have to leave (unexpectedly) at 1pm on Thursday. . . . . . . . . . . "There is no lifeguard at the gene pool." - C. H. Baldwin Anthony Skjellum, MSU/ERC, (601)325-8435; FAX: 325-8997; tony@cs.msstate.edu From owner-mpi-comm@CS.UTK.EDU Tue Mar 30 16:15:17 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20542; Tue, 30 Mar 93 16:15:17 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26319; Tue, 30 Mar 93 16:13:22 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 30 Mar 1993 16:13:20 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from cs.sandia.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA26310; Tue, 30 Mar 93 16:13:19 -0500 Received: from panther.cs.sandia.gov by cs.sandia.gov (4.1/SMI-4.1) id AA27775; Tue, 30 Mar 93 14:13:17 MST Received: by panther.cs.sandia.gov (Smail3.1.28.1 #1) id m0ndncS-0016bbC; Tue, 30 Mar 93 14:13 MST Message-Id: Date: Tue, 30 Mar 93 14:13 MST From: srwheat@cs.sandia.gov (Stephen R. Wheat) To: mpi-comm@cs.utk.edu Subject: Re: Revised agenda for MPI meeting I too have other pressing matters that were neatly integrated with the original schedule. I must leave Wed at 4pm, to go elsewhere; I will be back Thursday night. I have used the agenda for its purpose, to make plans about. These plans are worthless if I cannot be at the context discussion. I know this is my own problem for using the agenda for the purposes of what an agenda is for. Furthermore, I repeat that I believe it to be fruitless to discuss the MPI1 contents without agreement on context and group. Stephen From owner-mpi-comm@CS.UTK.EDU Tue Apr 6 10:39:59 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18806; Tue, 6 Apr 93 10:39:59 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23928; Tue, 6 Apr 93 10:38:00 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 6 Apr 1993 10:37:59 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from super.super.org by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23920; Tue, 6 Apr 93 10:37:58 -0400 Received: from b125.super.org by super.super.org (4.1/SMI-4.1) id AA28082; Tue, 6 Apr 93 10:37:56 EDT Received: by b125.super.org (4.1/SMI-4.1) id AA06018; Tue, 6 Apr 93 10:37:56 EDT Date: Tue, 6 Apr 93 10:37:56 EDT From: lederman@b125.super.org (Steve Huss-Lederman) Message-Id: <9304061437.AA06018@b125.super.org> To: mpi-comm@cs.utk.edu Subject: subset subcommittee Hello MPIers, At the last MPI meeting, a group met to discuss the possibility of creating a recommended subset of MPI for initial implementation. The idea was presented to the whole group and there were no objections. I have volunteered to coordinate this effort. At the current time I am seeking input to the process and people interested in joining/monitoring this new subcommittee (subset subcommittee for lack of a better name). Send me e-mail (lederman@super.org) to sign up (and suggest a better name :-). At the end of this week I will send all the names of people to David to create a new mail alias. When it is formed you will receive an initial message. You can, of course, join at a later time. Steve From owner-mpi-comm@CS.UTK.EDU Tue Apr 6 13:30:00 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA22321; Tue, 6 Apr 93 13:30:00 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02738; Tue, 6 Apr 93 13:28:41 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 6 Apr 1993 13:28:40 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA02729; Tue, 6 Apr 93 13:28:38 -0400 Date: Tue, 6 Apr 93 18:28:33 BST Message-Id: <754.9304061728@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: interrupt driven To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Cc: mpi-pt2pt@cs.utk.edu Dear MPI Colleagues We have, reasonably in my opinion, decided not to provide interrupt driven receive/send (as in Intel hrecv/hsend) in MPI, or active messages. I have heard, can't recall from whom, that this carries the consequence that we should not define any feature of MPI which asks for the implementor of MPI to make use of interrupt driven receives, or active messages. This seems to me to be a bogus argument. In my mind the primary reason for not providing hrecv/hsend type facilities is that it is very difficult to standardise these facilities across different operating systems, particularly in terms of what the handler procedure is and is not allowed to do (e.g., can it do I/O, can it use MPI, ...). It seems to me clear that on many (all) machines of interest the system itself will be making use of similar facilities. Who would write MPI for CM-5 without using active messages? It seems to me that for example non blocking communications really do ask for use of interrupt/active messages. Is there agreement, disagreement, or what on this point? Please do let me know. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Wed Apr 7 13:50:56 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA18878; Wed, 7 Apr 93 13:50:56 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15424; Wed, 7 Apr 93 13:49:23 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 7 Apr 1993 13:49:22 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from pnlg.pnl.gov by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA15367; Wed, 7 Apr 93 13:48:34 -0400 Received: from carbon.pnl.gov (130.20.188.38) by pnlg.pnl.gov; Wed, 7 Apr 93 10:48 PDT Received: from sodium.pnl.gov by carbon.pnl.gov (4.1/SMI-4.1) id AA06918; Wed, 7 Apr 93 10:46:30 PDT Received: by sodium.pnl.gov (4.1/SMI-4.0) id AA17058; Wed, 7 Apr 93 10:46:26 PDT Date: Wed, 7 Apr 93 10:46:26 PDT From: rj_littlefield@pnlg.pnl.gov Subject: Re: interrupt driven To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Cc: d39135@carbon.pnl.gov, mpi-pt2pt@cs.utk.edu Message-Id: <9304071746.AA17058@sodium.pnl.gov> X-Envelope-To: mpi-pt2pt@cs.utk.edu, mpi-comm@cs.utk.edu Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. > > I have heard, can't recall from whom, that this carries the consequence > that we should not define any feature of MPI which asks for the > implementor of MPI to make use of interrupt driven receives, or active > messages. This seems to me to be a bogus argument. I (Rik Littlefield) have regularly made arguments that are close enough to this to be worth clarifying or defending. Think of the MPI specification as being divided into two parts: 1) things you can implement with portable code, and 2) things you can't. My bias is to make the POTENTIALLY portable part of MPI as large as possible. (I emphasize "potentially" to make it clear that I am talking about interface definitions. I encourage implementors to do whatever they want internally.) As an example, consider groups and collective communication. All other factors being equal, I would strongly prefer a specification that permitted groups and collective communication, including group formation and disbanding, to be implemented using just the standardized MPI point-to-point capabilities. This sort of layerability potentially has several good effects. Three of these are 1) earlier widespread availability via porting, 2) better focusing of vendor effort on tuning the point-to-point layer, and 3) the capability to create portable MPI-plug-compatible implementations that are tuned for different requirements, e.g., debugging versus performance. Of course, all other factors are not equal. Strict layerability also has some potentially bad effects. Two of these (depending on where you draw the line between layers) are 1) the loss of integrated syntax (e.g., being able to use rank directly in the point-to-point calls), and 2) loss of functionality that is not supported by lower layers (e.g., asynchronous name servers). Whether the good outweighs the bad is a matter of opinion. The applications that I deal with now and anticipate dealing with over the lifetime of MPI-1 can be characterized as follows: . sensitive to performance, . intended to run on lots of platforms, . not sensitive to syntax, and . not needing name server support. Applications like these are best served by layerability. Others' mileage may differ. Lyndon continues: > In my mind the primary reason for not providing hrecv/hsend type > facilities is that it is very difficult to standardise these facilities > across different operating systems, particularly in terms of what the > handler procedure is and is not allowed to do (e.g., can it do I/O, can > it use MPI, ...). It seems to me clear that on many (all) machines of > interest the system itself will be making use of similar facilities. > Who would write MPI for CM-5 without using active messages? It seems to > me that for example non blocking communications really do ask for use of > interrupt/active messages. > > Is there agreement, disagreement, or what on this point? Please do let > me know. I agree that it is important to distinguish between exporting a standardized hrecv/hsend and and exporting a standardized asynchronous name server interface. The former seems hopeless; the latter is doable, but at the cost of potentially detracting from other features such as performance and prompt widespread availability. Thus, I do not agree that it is bogus to argue against requiring asynchronous servers et.al. in MPI-1. --Rik ---------------------------------------------------------------------- rj_littlefield@pnl.gov (alias 'd39135') Rik Littlefield Tel: 509-375-3927 Pacific Northwest Lab, MS K1-87 Fax: 509-375-6631 P.O.Box 999, Richland, WA 99352 From owner-mpi-comm@CS.UTK.EDU Wed Apr 7 14:29:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA20295; Wed, 7 Apr 93 14:29:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17859; Wed, 7 Apr 93 14:27:46 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Wed, 7 Apr 1993 14:27:44 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from arthur.cs.purdue.edu by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA17851; Wed, 7 Apr 93 14:27:43 -0400 Received: from dragomirna.cs.purdue.edu by arthur.cs.purdue.edu (5.65c/PURDUE_CS-1.2) id ; Wed, 7 Apr 1993 13:27:40 -0500 Received: by dragomirna.cs.purdue.edu (5.65c/PURDUE_CS-1.2) id ; Wed, 7 Apr 1993 13:27:39 -0500 Date: Wed, 7 Apr 1993 13:27:39 -0500 From: dcm@cs.purdue.edu (Dan C Marinescu) Message-Id: <199304071827.AA28416@dragomirna.cs.purdue.edu> To: dcm@cs.purdue.edu, lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu Subject: interrupt driven messages Lyndon Clarke writes: > We have, reasonably in my opinion, decided not to provide interrupt > driven receive/send (as in Intel hrecv/hsend) in MPI, or active > messages. Interrupt driven messages are too important to be left out. Any attempt made either by the application or by the compiler to cache data in the local storage of any compute node can only be effective if interrupt driven messages are supported. The interrupt handling procedure should allow both MPI and I/O. Indeed, this opens a Pandora box. For example how many such messages should be queued for a suspended process, should one place a limit upon the size of an interrupt message and upon the time spent in the interrupt handling procedure and so on. Yet it seems reasonable to expect that the standardization effort in OS will make this task easier, for example one could expect that more vendors will base their OS on OSF/1. Moreover, interrupt driven messages are likely to be provided by most vendors. Dan Marinescu From owner-mpi-comm@CS.UTK.EDU Thu Apr 8 07:12:11 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA07522; Thu, 8 Apr 93 07:12:11 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05594; Thu, 8 Apr 93 07:10:15 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 8 Apr 1993 07:10:14 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA05576; Thu, 8 Apr 93 07:10:10 -0400 Date: Thu, 8 Apr 93 12:10:04 BST Message-Id: <2092.9304081110@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: mpi-comm: various (long) To: mpi-comm@cs.utk.edu Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues I am sending this letter to mpi-comm as it seems to cover issues in many subcommittees and in particular: point-to-point, collective, context, environmental. This is a long, but I believe important, letter. The letter is a discussion of some issues primarily in point-to-point, collective and context, with some strong and powerful recommendations for each of these subcommittees. o--------------------o 1) In the recent meeting we discussed and narrowly accepted an amendment to the point-to-point chapter which included secure (aka synchronous, aka unbuffered) communication. Paul Pierce made the point that providing different point-to-point procedures does not seem to be the right model for this capability as it has a microscopic effect on MPI user software. This point is actually well taken (by myself at least). Paul went on to suggest that there should be application global control over whether point-to-point communications are all secure (aka synchronous, aka unbuffered) or all insecure (aka regular, aka asynchronous, aka buffered). I opposed this suggestion as it does not respect the requirements of library writers. There appeared to be a number of people who were concerned about the geometric increase in the number of point-to-point communication procedures. The discussion of secure send and secure receive became messy, and I promised to return some more sane discussion of the point. In this letter I will later (after consideration of some issues in collective communications and communication contexts) make a recommendation which: respects the well taken point of Paul; provides a mechanism for the global control suggested by Paul while respecting the independence of library writers from user program writers; avoids geometric growth in the number of point-to-point procedures; and completely side-steps the secure send and secure receive issue. 2) The question of whether collective communication procedures do or do not barrier synchronise the group has been raised a number of times. In the recent meeting this lead to Steve Otto suggesting that MPI user software should be written as though these operations do barrier synchronise although the MPI implementation is not obliged to make the operations so do. This is exactly analagous to the point-to-point insecure (aka regular, ...) communications which may or may not synchronise the sender and receiver depending on the implementation (read depending on system buffering resource availability if you prefer). In order to write safe (i.e. guaranteed portable) programs with insecure (aka regular, ...) communications the MPI user software should be written as though these communications were secure (aka synchronous, ...). In this letter I will later (after consideration of some issues in communication contexts) make a recommendation which: provides for both secure (i.e. barrier synchronous) and insecure (i.e. may or may not be barrier synchronous depending on implementation) collective communications; and avoids geometric growth in the number of collective communication procedures. 3) In the context subcommittee Mark Sears, in Proposal VIII and in presentation at the recent meeting, has pointed out that there are two classes of MPI user libraries. I will discuss this (from my point of view) briefly here for concreteness. There is a class of libraries which accept context(s) from the caller and use these context(s) for communication. Note that since the library uses the contexts(s) given by the caller then it is clearly the responsibility of the caller to ensure that there are no "dangling" communications, either in buffers or nonblocking, which will interfere with the communications of the library, and it is the responsibility of the library to ensure that there are no dangling communications between calls to the library which will interfere with communications of the user. Let me label this class as ClassA. There is a class of libraries which create context(s) for library private use and use only these context(s) for communication. Note that since the library uses private context(s) then the user does not have any concern regarding dangling communications which may interfere with the library communications, and equally the library may leave dangling communications between library calls since they cannot interfere with the user communications. Let me label this class as ClassB. ClassA libraries will generally operate within one (or more) group(s) and accept one (or more) group(s) from the caller, in addition to context(s). ClassB libraries will also generally operate within one (or more) group(s) and accept one (or more) group(s) from the caller, but do not accept context(s). This observation suggests that context and group should be different entities within MPI. (Whether we have an object which binds a group and a context is another issue to which I wish to return in a separate message of narrower circulation.) Observe that the (blocking) collective communications which are described in the MPI collective communication chapter draft are conformant with the ClassA library described and s single group. The kind of libraries which I am writing (and I suspect also Tony, although I am not sure) are conformant with the ClassB library described. 4) I now make a recommendation which ties together the points discussed above and delivers the promises made. First we observe that context is the MPI mechanism for distinguishing the communications of the user from (ClassB) libraries. The recommendation is that secure or insecure modes of operation are a a property of the communication context. This applies equally to both point-to-point communications. In point-to-point communications a secure context provides the secure point-to-point communication which I described at the recent meeting, and an insecure context provides the inseure point-to-point communications which is as discussed (regular) in the current chapter draft. (The semantics and desirability of ready-receiver communication in secure and insecure contexts is a subject to which I wish to return in a separate message of narrower circulation.) In collective communications a secure context provides barrier synchronisation across the group, and an insecure context may or may not provide barrier synchronisation which is as discussed in the recent meeting. There should be an application global default context property, either secure or insecure, which is specified by the user in an environmental management procedure (perhaps as argument to mpi_init()?). Creation of context should specify whether the created context is to have: default property; secure property; insecure property. It is possibe to provide these capabilities in terms of buffering in different ways. One possibility is that the property of a context is the buffering associated with that context, in which case a secure context is one with zero associated buffering. Another possibility is that the property of a context is whether or not it should use whatever buffering is available, in which case a secure context is one which does not use any buffering. My preference is for the latter since it does not prescribe the description and association of buffering. This recommendation does deliver the promises made. The requirements of the (ClassB) library writer are respected while providing global control for the user program writer. The capabilities required are provided without geometric growth of the number of point-to-point and/or collective communications. o--------------------o Free beer to all who can demonstrate that they have read and considered this (long long letter). Comments, questions, (flames :-) please?! Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Fri Apr 9 11:26:20 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA29024; Fri, 9 Apr 93 11:26:20 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23758; Fri, 9 Apr 93 11:24:22 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Fri, 9 Apr 1993 11:24:21 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA23750; Fri, 9 Apr 93 11:24:19 -0400 Date: Fri, 9 Apr 93 16:24:14 BST Message-Id: <3171.9304091524@subnode.epcc.ed.ac.uk> From: L J Clarke Subject: Re: mpi-comm: various (long) To: lyndon@epcc.ed.ac.uk, mpi-comm@cs.utk.edu In-Reply-To: L J Clarke's message of Thu, 8 Apr 93 12:10:04 BST Reply-To: lyndon@epcc.ed.ac.uk Dear MPI Colleagues This message is short. It is two little things which I forgot to say in the "mpi-comm: various (long)" message. We can loosely associate ClassA libraries as collections of procedures for which one can use the substitution model of evaluation, and ClassB procedures as encapsulated objects. The argument lists of of ClassA and ClassB libraries differ primarily in that ClassA libraries accept context(s) whereas ClassB libraries do not. This seems (to me) to convey just the right idea to the library user regarding the responsibilities for management of contexts and messages within those contexts. Best Wishes Lyndon /--------------------------------------------------------\ e||) | Lyndon J Clarke Edinburgh Parallel Computing Centre | e||) c||c | Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk | c||c \--------------------------------------------------------/ From owner-mpi-comm@CS.UTK.EDU Tue Apr 20 16:21:16 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08357; Tue, 20 Apr 93 16:21:16 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03609; Tue, 20 Apr 93 16:18:26 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 20 Apr 1993 16:18:25 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03601; Tue, 20 Apr 93 16:18:23 -0400 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA14309; Tue, 20 Apr 1993 16:18:20 -0400 Date: Tue, 20 Apr 1993 16:18:20 -0400 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9304202018.AA14309@gstws.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: 2nd draft of Collective Communication proposal After studying the latest point-to-point draft by Marc and the votes from the last meeting, I have revised the original collective communication draft. Comments can be directed to the mpi-collcomm committee. Here is the LaTeX version, a postscript version will follow in a separate message. Al ----------------------------- __o /\ Al Geist _`\<,_ /\/ \ Oak Ridge National Laboratory (_)/ (_) / \ (615) 574-3153 gst@ornl.gov * * * * * * * * * * ----------------------------- % MPI Authors: % This is MY version of YOUR chapter. It has a wrapper so that you % should be able to simply LaTeX this. % % Please work from this text so that we are in synch. % % --Steve Otto \documentstyle[twoside,11pt]{report} \pagestyle{headings} %\markright{ {\em Draft Document of the MPI Standard,\/ \today} } \marginparwidth 0pt \oddsidemargin=.25in \evensidemargin .25in \marginparsep 0pt \topmargin=-.5in \textwidth=6.0in \textheight=9.0in \parindent=2em % ---------------------------------------------------------------------- % mpi-macs.tex --- man page macros, % discuss, missing, mpifunc macros % % ---------------------------------------------------------------------- % a couple of commands from Marc Snir, modified S. Otto \newlength{\discussSpace} \setlength{\discussSpace}{.7cm} \newcommand{\discuss}[1]{\vspace{\discussSpace} {\small {\bf Discussion:} #1} \vspace{\discussSpace} } \newcommand{\missing}[1]{\vspace{\discussSpace} {\small {\bf Missing:} #1} \vspace{\discussSpace} } \newlength{\codeSpace} \setlength{\codeSpace}{.3cm} \newcommand{\mpifunc}[1]{\vspace{\codeSpace} {\bf #1} \vspace{\codeSpace} } % ----------------------------------------------------------------------- % A few commands to help in writing MPI man pages % \def\twoc#1#2{ \begin{list} {\hbox to95pt{#1\hfil}} {\setlength{\leftmargin}{120pt} \setlength{\labelwidth}{95pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#2} \end{list} } \outer\long\def\onec#1{ \begin{list} {} {\setlength{\leftmargin}{25pt} \setlength{\labelwidth}{0pt} \setlength{\labelsep}{0pt} \setlength{\partopsep}{0pt} \setlength{\parskip}{0pt} \setlength{\topsep}{0pt} } \item {#1} \end{list} } \def\manhead#1{\noindent{\bf{#1}}} \hyphenation{RE-DIS-TRIB-UT-ABLE sub-script mul-ti-ple} \begin{document} \setcounter{page}{1} \pagenumbering{roman} \title{ {\em D R A F T} \\ Document for a Standard Message-Passing Interface} \author{Scott Berryman, {\em Yale Univ} \\ James Cownie, {\em Meiko Ltd} \\ Jack Dongarra, {\em Univ. of Tennessee and ORNL} \\ Al Geist, {\em ORNL} \\ Bill Gropp, {\em ANL} \\ Rolf Hempel, {\em GMD} \\ Bob Knighten, {\em Intel} \\ Rusty Lusk, {\em ANL} \\ Steve Otto, {\em Oregon Graduate Inst} \\ Tony Skjellum, {\em Missisippi State Univ} \\ Marc Snir, {\em IBM T. J. Watson} \\ David Walker, {\em ORNL} \\ Steve Zenith, {\em Kuck \& Associates} } \date{April 20, 1993 \\ This work was supported by ARPA and NSF under contract number \#\#\#, by the National Science Foundation Science and Technology Center Cooperative Agreement No. CCR-8809615. } \maketitle \hfuzz=5pt %\tableofcontents %\begin{abstract} %We don't have an abstract yet. %\end{abstract} \setcounter{page}{1} \pagenumbering{arabic} \chapter{Collective Communication} \label{sec:coll} \begin{center} Al Geist \\ Marc Snir \end{center} \section{Introduction} This section is a draft of the current proposal for collective communication. Collective communication is defined to be communication that involves a group of processes. Examples are broadcast and global sum. A collective operation is executed by having all processes in the group call the communication routine, with matching parameters. Routines can (but are not required to) return as soon as their participation in the collective communication is complete. The completion of a call indicates that the caller is now free to access the locations in the communication buffer, or any other location that can be referenced by the collective operation. However, it does not indicate that other processes in the group have started the operation (unless otherwise indicated in the description of the operation). However, the successful completion of a collective communication call may depend on the execution of a matching call at all processes in the group. The syntax and semantics of the collective operations is defined so as to be consistent with the syntax and semantics of the point to point operations. The reader is referred to the point-to-point communication section of the current MPI draft for information concerning communication buffers and their manipulations. The context section describes the formation, manipulation, and query functions (such as group size) that are available for groups and group objects. The collective communication routines are built above the point-to-point routines. While vendors may optimize certain collective routines for their architectures, a complete library of the collective communication routines can be written entirely using point-to-point communication functions. We are using naive implementations of the collective calls in terms of point to point operations in order to provide an operational definition of their semantics. The following communication functions are proposed. \begin{itemize} \item Broadcast from one member to all members of a group. \item Barrier across all group members \item Gather data from all group members to one member. \item Scatter data from one member to all members of a group. \item Global operations such as sum, max, min, etc., were the result is known by all group members and a variation where the result is known by only one member. The ability to have user defined global operations. \item Simultaneous shift of data around the group, the simplest example being all members sending their data to (rank+1) with wrap around. \item Scan across all members of a group (also called parallel prefix). \item Broadcast from all members to all members of a group. \item Scatter data from all members to all members of a group (also called complete exchange or index). \end{itemize} To simplify the collective communication interface it is designed with two layers. The low level routines have all the generality of, and make use of, the communication buffer routines of the point-to-point section which allows arbitrarily complex messages to be constructed. The second level routines are similar to the upper level point-to-point routines in that they send only a contiguous buffer. \section{Group Functions} A full description of the group formation and manipulation functions can be found in the context chapter of the MPI document. Here we describe only those group functions that are used in the semantic description of the collective communication routines. An initial group containing all processes is supplied by default in MPI. MPI provides a procedure that returns the handle to this initial group. The processes in the inital group each have a unique rank in the group represented by integers (0, 1, 2, ..., [number-of-processes~-~1]. \mpifunc{MPI\_ALLGROUP(group)} Return the descriptor of the inital group containing all processes. \begin{description} \item[OUT group] handle to descriptor object of initial group. \end{description} \mpifunc{MPI\_RANK(group, rank)} Return the rank of the calling process within the specified group. \begin{description} \item[IN group] group handle \item[OUT rank] integer \end{description} \mpifunc{MPI\_GSIZE(group, size)} Return the number of processes that belong to the specified group. \begin{description} \item[IN group] group handle \item[OUT size] integer \end{description} \section{Communication Functions} The proposed communication functions are divided into two layers. The lowest level uses the same buffer descriptor objects available in point-to-point to create noncontiguous, multiple data type messages. The second level is similar to the block send/receive point-to-point operations in that it supports only contiguous buffers of data. For each communication operation, we list these two level of calls together. \section{Synchronization} \subsubsection*{Barrier synchronization} \mpifunc{MPI\_BARRIER( group )} MPI\_BARRIER blocks the calling process until all group members have called it; the call returns at any process only after all group members have entered the call. \begin{description} \item[IN group] group handle \end{description} {\tt MPI\_BARRIER( group )} is \begin{verbatim} MPI_CREATE_BUFFER(buffer_handle, MPI_BUFFER, MPI_PERSISTENT); MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); if (rank==0) { for (i=1; i < size; i++) MPI_RECV(buffer_handle, i, tag, group, return_handle); for (i=1; i < size; i++) MPI_SEND(buffer_handle, i, tag, group); } else { MPI_SEND(buffer_handle, 0, tag, group); MPI_RECV(buffer_handle, 0, tag, group, return_handle); } MPI_FREE(buffer_handle); \end{verbatim} \discuss{ since tag was voted out in the last meeting, a "safe" tag must be selected to use in the above description } \section{Data move functions} \subsubsection*{Broadcast} \mpifunc{ MPI\_BCAST( buffer\_handle, group, root )} {\tt MPI\_BCAST} broadcasts a message from the process with rank {\tt root} to all other processes of the group. It is called by all members of group using the same arguments for {\tt group, and root}. On return the contents of the buffer of the process with rank {\tt root} is contained in buffer of all group members. \begin{description} \item[INOUT buffer\_handle] Handle for buffer where from message is sent or received. \item[IN group] group handle \item[IN root] rank of broadcast root (integer) \end{description} \mpifunc{ MPI\_BCASTC( buf, len, group, root )} {\tt MPI\_BCASTC} behaves like broadcast, restricted to a block buffer. It is called by all processes with the same arguments for {\tt len, group} and {\tt root}. \begin{description} \item[INOUT buffer] Starting address of buffer (choice type) \item[IN len] Number of words in buffer (integer) \item[IN group] group handle \item[in root] rank of broadcast root (integer) \end{description} {\tt MPI\_BCAST( buffer\_handle, group, root )} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_IRECV(handle, buffer_handle, root, tag, group, return_handle); if (rank==root) for (i=0; i < size; i++) MPI_SEND(buffer_handle, i, tag, group); MPI_WAIT(handle) \end{verbatim} \subsubsection*{Circular shift} \mpifunc{MPI\_CSHIFT( inbuf, outbuf, group, shift)} Process with rank {\tt i} sends the data in its input buffer to process with rank $\tt (i+ shift) \bmod group\_size$, who receives the data in its output buffer. All processes make the call with the same values for {\tt group}, and {\tt shift}. The {\tt shift} value can be positive, zero, or negative. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFT1( buf, group, shift)} Process with rank {\tt i} sends the data in its buffer to process with rank $\tt (i+ shift) \bmod group\_size$, who receives the data in the same buffer. All processes make the call with the same values for {\tt group}, and {\tt shift}. The {\tt shift} value can be positive, zero, or negative. \begin{description} \item[INOUT buf] handle to buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFTC( inbuf, outbuf, len, group, shift)} Behaves like {\tt MPI\_CSHIFT}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[IN inbuf] initial location of input buffer \item[OUT outbuf] initial location of output buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_CSHIFTC1( buf, len, group, shift)} Behaves like {\tt MPI\_CSHIFT1}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[INOUT buf] initial location of buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} {\tt MPI\_CSHIFT( inbuf, outbuf, group, shift)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_ISEND( handle, inbuf, mod(rank+shift, size), tag, group); MPI_RECV( outbuf, mod(rank-shift,size), tag, group, return_handle) MPI_WAIT(handle); \end{verbatim} \subsubsection*{End-off shift} \mpifunc{MPI\_EOSHIFT( inbuf, outbuf, group, shift)} Process with rank {\tt i}, $\tt \max( 0, -shift) \le i < min( size, size - shift)$, sends the data in its input buffer to process with rank {\tt i+ shift}, who receives the data in its output buffer. The output buffer of processes which do not receive data is left unchanged. All processes make the call with the same values for {\tt group}, and {\tt shift}. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFT1( buf, group, shift)} Process with rank {\tt i}, $\tt \max( 0, -shift) \le i < min( size, size - shift)$, sends the data in its buffer to process with rank {\tt i+ shift}, who receives the data in the same buffer. The output buffer of processes which do not receive data is left unchanged. All processes make the call with the same values for {\tt group}, and {\tt shift}. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFTC( inbuf, outbuf, len, group, shift)} Behaves like {\tt MPI\_EOSHIFT}, with buffers restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[IN inbuf] initial location of input buffer \item[OUT outbuf] initial location of output buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \mpifunc{MPI\_EOSHIFTC1( buf, len, group, shift)} Behaves like {\tt MPI\_EOSHIFT1}, with buffer restricted to be blocks of numeric units. All processes make the call with the same values for {\tt len, group}, and {\tt shift}. \begin{description} \item[INOUT buf] initial location of buffer \item[IN len] number of entries in input (and output) buffers \item[IN group] handle to group \item[IN shift] integer \end{description} \subsubsection*{Gather} \mpifunc{MPI\_GATHER( inbuf, outbuf, group, root, return\_status) } Each process (including the root process) sends the content of its input buffer to the root process. The root process concatenates all the incoming messages in the order of the senders' rank and places the results in its output buffer. It is called by all members of group using the same arguments for {\tt group}, and {\tt root}. The input buffer of each process may have a different length. \begin{description} \item[IN inbuf] handle to input buffer descriptor \item[OUT outbuf] handle to output buffer descriptor -- significant only at root (choice) \item[IN group] group handle \item[IN root] rank of receiving process (integer) \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_GATHERC( inbuf, inlen, outbuf, group, root) } {\tt MPI\_GATHERC} behaves like {\tt MPI\_GATHER} restricted to block buffers, and with the additional restriction that all input buffers should have the same length. All processes should provided the same values for {\tt inlen, group}, and {\tt root} . \begin{description} \item[IN inbuf] first variable of input buffer (choice) \item[IN inlen] Number of (word) variables in input buffer (integer) \item[OUT outbuf] first variable of output buffer -- significant only at root (choice) \item[IN group] group handle \item[IN root] rank of receiving process (integer) \end{description} {\tt MPI\_GATHERC( inbuf, inlen, outbuf, tag, group, root) } is \begin{verbatim} MPI_GSIZE( &size, group); MPI_RANK( &rank, group); MPI_ISENDC(handle, inbuf, inlen, root, tag, group); if (rank==root) for (i=0; i < size; i++) { MPI_RECVC(outbuf, inlen, i, tag, group, return\_status); outbuf += inlen; } MPI_WAIT(handle); \end{verbatim} \subsubsection*{Scatter} \mpifunc{MPI\_SCATTER( list\_of\_inbufs, outbuf, group, root, return\_status)} The root process sends the content of its {\tt i}-th input buffer to the process with rank {\tt i}; each process (including the root process) stores the incoming message in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt root}. \begin{description} \item[IN list\_of\_inbufs] list of buffer descriptor handles \item[OUT outbuf] buffer descriptor handle \item[IN group] handle \item[IN root] rank of sending process (integer) \item[OUT return\_status] return status handle \end{description} {\tt MPI\_SCATTER( list\_of\_inbufs, outbuf, group, root, return\_status)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); MPI_IRECV(handle, outbuf, root, tag, group); if (rank=root) for (i=0; i < size; i++) MPI_SEND(inbuf[i], i, tag, group); MPI_WAIT(handle, return\_status); \end{verbatim} \mpifunc{MPI\_SCATTERC( inbuf, outbuf, len, group, root)} {\tt MPI\_SCATTERC} behaves like {\tt MPI\_SCATTER} restricted to block buffers, and with the additional restriction that all output buffers have the same length. The input buffer block of the root process is partitioned into {\tt n} consecutive blocks, each consisting of {\tt len} words. The {\tt i}-th block is sent to the {\tt i}-th process in the group and stored in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group, len}, and {\tt root}. \begin{description} \item[IN inbuf] first entry in input buffer -- significant only at root (choice). \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries to be stored in output buffer (integer) \item[IN group] handle \item[IN root] rank of sending process (integer) \end{description} {\tt MPI\_SCATTERC( inbuf, outbuf, outlen, group, root) } is \begin{verbatim} MPI_GSIZE( &size, group); MPI_RANK( &rank, group); MPI_IRECVC( handle, outbuf, outlen, root, tag, group); if (rank=root) for (i=0; i < size; i++) { MPI_SENDC(inbuf, outlen, i, tag, group, return\_status); inbuf += outlen; } MPI_WAIT(handle); \end{verbatim} \subsubsection*{All-to-all scatter} \mpifunc{MPI\_ALLSCATTER( list\_of\_inbufs, outbuf, group, return\_status)} Each process in the group sends its {\tt i}-th buffer in its input buffer list to the process with rank {\tt i} (itself included); each process concatenates the incoming messages in its output buffer, in the order of the senders' ranks. The routine is called by all members of the group using the same arguments for {\tt group}. \begin{description} \item[IN list\_of\_inbufs] list of buffer descriptor handles \item[OUT outbuf] buffer descriptor handle \item[IN group] handle \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_ALLSCATTERC( inbuf, outbuf, len, group)} {\tt MPI\_ALLSCATTERC} behaves like {\tt MPI\_ALLSCATTER} restricted to block buffers, and with the additional restriction that all blocks sent from one process to another have the same length. The input buffer block of each process is partitioned into {\tt n} consecutive blocks, each consisting of {\tt len} words. The {\tt i}-th block is sent to the {\tt it}-th process in the group. Each process concatenates the incoming messages, in the order of the senders' ranks, and store them in its output buffer. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt len}. \begin{description} \item[IN inbuf] first entry in input buffer (choice). root (integer) \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries sent from each process to each other (integer). \item[IN group] handle \end{description} {\tt MPI\_ALLSCATTERC( inbuf, outbuf, len, group)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); for (i=0; i < rank; i++) { MPI_IRECVC(recv_handles[i], outbuf, len, tag, group); outbuf += len; } for (i=0; i < size; i++) { MPI_ISENDC(send_handle[i], inbuf, len, i, tag, group); inbuf += len; } MPI_WAITALL(send_handle); MPI_WAITALL(recv_handle); \end{verbatim} \subsubsection*{All-to-all broadcast} \mpifunc{MPI\_ALLCAST( inbuf, outbuf, group, return\_status)} Each process in the group broadcasts its input buffer to all processes (including itself); each process concatenates the incoming messages in its output buffer, in the order of the senders' ranks. The routine is called by all members of the group using the same arguments for {\tt group}. \begin{description} \item[IN inbuf] buffer descriptor handle for input buffer \item[OUT outbuf] buffer descriptor handle for output buffer \item[IN group] handle \item[OUT return\_status] return status handle \end{description} \mpifunc{MPI\_ALLCASTC( inbuf, outbuf, len, group)} {\tt MPI\_ALLCASTC} behaves like {\tt MPI\_ALLCAST} restricted to block buffers, and with the additional restriction that all blocks sent from one process to another have the same length. The routine is called by all members of the group using the same arguments for {\tt group}, and {\tt len}. \begin{description} \item[IN inbuf] first entry in input buffer (choice). root (integer) \item[OUT outbuf] first entry in output buffer (choice). \item[IN len] number of entries sent from each process to each other (including itself). \item[IN group] group handle \end{description} {\tt MPI\_ALLCASTC( inbuf, outbuf, len, group)} is \begin{verbatim} MPI_GSIZE( group, &size ); MPI_RANK( group, &rank ); for (i=0; i < rank; i++) { MPI_IRECVC(recv_handles[i], outbuf, len, tag, group); outbuf += len; } for (i=0; i < size; i++) { MPI_ISENDC(send_handle[i], inbuf, len, i, tag, group); } MPI_WAITALL(send_handle); MPI_WAITALL(recv_handle); \end{verbatim} \section{Global Compute Operations} \subsubsection*{Reduce} \mpifunc{MPI\_REDUCE( inbuf, outbuf, group, root, op)} Combines the values provided in the input buffer of each process in the group, using the operation {\tt op}, and returns the combined value in the output buffer of the process with rank {\tt root}. Each process can provide one value, or a sequence of values, in which case the combine operation is executed pointwise on each entry of the sequence. For example, if the operation is {\tt max} and input buffers contains two floating point numbers, then outbuf(1) $=$ global max(inbuf(1)) and outbuf(2) $=$ global max(inbuf(2)). All input buffers should define sequences of equal length of entries of types that match the type of the operands of {\tt op}. The output buffer should define a sequence of the same length of entries of types that match the type of the result of {\tt op}. (Note that, here as for all other communication operations, the type of entries inserted in a message depend on the information provided by the input buffer descriptor, and not on the declarations of these variables in the calling program. The types of the variables in the calling program need not match the types defined by the buffer descriptor, but in such case the outcome of a reduce operation may be implementation dependent.) The operation defined by {\tt op} is associative and commutative, and the implementation can take advantage of associativity and commutativity in order to change order of evaluation. The routine is called by all group members using the same arguments for {\tt group, root} and {\tt op}. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer -- significant only at root \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} We list below the operations which are supported for Fortran, each with the corresponding value of the {\tt op} parameter. \begin{description} \item[MPI\_IMAX] integer maximum \item[MPI\_RMAX] real maximum \item[MPI\_DMAX] double precision real maximum \item[MPI\_IMIN] integer minimum \item[MPI\_RMIN] real minimum \item[MPI\_DMIN] double precision real minimum \item[MPI\_ISUM] integer sum \item[MPI\_RSUM] real sum \item[MPI\_DSUM] double precision real sum \item[MPI\_CSUM] complex sum \item[MPI\_DCSUM] double precision complex sum \item[MPI\_IPROD] integer product \item[MPI\_RPROD] real product \item[MPI\_DPROD] double precision real product \item[MPI\_CPROD] complex product \item[MPI\_DCPROD] double precision complex product \item[MPI\_AND] logical and \item[MPI\_IAND] integer (bit-wise) and \item[MPI\_OR] logical or \item[MPI\_IOR] integer (bit-wise) or \item[MPI\_XOR] logical xor \item[MPI\_IXOR] integer (bit-wise) xor \item[MPI\_MAXLOC] rank of process with maximum integer value \item[MPI\_MAXRLOC] rank of process with maximum real value \item[MPI\_MAXDLOC] rank of process with maximum double precision real value \item[MPI\_MINLOC] rank of process with minimum integer value \item[MPI\_MINRLOC] rank of process with minimum real value \item[MPI\_MINDLOC] rank of process with minimum double precision real value \end{description} \mpifunc{MPI\_REDUCEC( inbuf, outbuf, len, group, root, op)} Is same as {\tt MPI\_REDUCE}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer -- significant only at root \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} \mpifunc{MPI\_USER\_REDUCE( inbuf, outbuf, group, root, function)} Same as the reduce operation function above except that a user supplied function is used. {\tt function} is an associative and commutative function with two arguments. The types of the two arguments and of the returned value of the function, and the types of all entries in the input and output buffers all agree. The output buffer has the same length as the input buffer. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer -- significant only at root \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN function] user provided function \end{description} \mpifunc{MPI\_USER\_REDUCEC( inbuf, outbuf, len, group, root, function)} Is same as {\tt MPI\_\_USER\_REDUCE}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer -- significant only at root \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN root] rank of root process (integer) \item[IN op] operation (status) \end{description} \discuss{ Do we also want a version of reduce that broadcasts the result to all processes in the group? (This can be achieved by a reduce followed by a broadcast, but a combined function may be somewhat more efficient.) These would be respectively: \mpifunc{MPI\_GOP( inbuf, outbuf, group, op)} \mpifunc{MPI\_GOPC( inbuf, outbuf, len, group, op)} \mpifunc{MPI\_USER\_GOP( inbuf, outbuf, group, function)} \mpifunc{MPI\_USER\_GOPC( inbuf, outbuf, len, group, function)} Do we want a user provided {\em function} (two IN parameters, one OUT value), or a user provided procedure that overwrites the second input (ie. one IN param, one INOUT param, the equivalent of C {\tt a op= b} type assignment)? The second choice may allow a more efficient implementation, without changing the semantics of the MPI call. } \subsubsection*{Scan} \mpifunc{ MPI\_SCAN( inbuf, outbuf, group, op )} MPI\_SCAN is used to perform a parallel prefix with respect to an associative reduction operation on data distributed across the group. The operation returns in the output buffer of the process with rank {\tt i} the reduction of the values in the input buffers of processes with ranks {\tt 0,...,i}. The type of operations supported and their semantic, and the constraints on input and output buffers are as for {\tt MPI\_REDUCE}. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer \item[IN group] handle to group \item[IN op] operation (status) \end{description} \mpifunc{ MPI\_SCANC( inbuf, outbuf, len, group, op )} Same as {\tt MPI\_SCAN}, restricted to block buffers. \begin{description} \item[IN inbuf] first input buffer element (choice) \item[OUT outbuf] first output buffer element (choice) \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN op] operation (status) \end{description} \mpifunc{ MPI\_USER\_SCAN( inbuf, outbuf, group, function )} Same as the scan operation function above except that a user supplied function is used. {\tt function} is an associative and commutative function with two arguments. The types of the two arguments and of the returned values all agree. \begin{description} \item[IN inbuf] handle to input buffer \item[OUT outbuf] handle to output buffer \item[IN group] handle to group \item[IN function] user provided function \end{description} \mpifunc{MPI\_USER\_SCANC( inbuf, outbuf, len, group, function)} Is same as {\tt MPI\_USER\_SCAN}, restricted to a block buffer. \begin{description} \item[IN inbuf] first location in input buffer \item[OUT outbuf] first location in output buffer \item[IN len] number of entries in input and output buffer (integer) \item[IN group] handle to group \item[IN function] user provided function \end{description} \discuss{ Do we want scan operations executed by segments? (The HPF definition of prefix and suffix operation might be handy -- in addition to the scanned vector of values there is a mask that tells where segments start and end.) } \section{Correctness} \discuss{ This is still very preliminary} The semantics of the collective communication operations can be derived from their operational definition in terms of point-to-point communication. It is assumed that messages pertaining to one operation cannot be confused with messages pertaining to another operation. Also messages pertaining to two distinct occurrences of the same operation cannot be confused, if the two occurrences have distinct parameters. The relevant parameters for this purpose are {\tt group}, {\tt root} and {\tt op}. messages pertaining to another occurrence of the same operation, with different parameters. The implementer can, of course, use another, more efficient implementation, as long as it has the same effect. \discuss{ This statement does not yet apply to the current, incomplete and somewhat careless definitions I provided in this draft. The definition above means that messages pertaining to a collective communication carry information identifying the operation itself, and the values of the {\tt group} and, where relevant, {\tt root} or {\tt op} parameters. Is this acceptable? } A few examples: \begin{verbatim} MPI_BCAST(buf, len, group, 0); MPI_BCAST(buf, len, group, 1); \end{verbatim} Two consecutive broadcasts, in the same group, with the same tag, but different roots. Since the operations are distinguishable, messages from one broadcast cannot be confused with messages from the other broadcast; the program is safe and will execute as expected. \begin{verbatim} MPI_BCAST(buf, len, group, 0); MPI_BCAST(buf, len, group, 0); \end{verbatim} Two consecutive broadcasts, in the same group, with the same tag and root. Since point-to-point communication preserves the order of messages here, too, messages from one broadcast will not be confused with messages from the other broadcast; the program is safe and will execute as intended. \begin{verbatim} MPI_RANK(&rank, group) if (rank==0) { MPI_BCASTC(buf, len, group, 0); MPI_SENDC(buf, len, 2, tag, group); } elseif (rank==1) { MPI_RECVC(buf, len, MPI_DONTCARE, tag, group); MPI_BCASTC(buf, len, group, 0); MPI_RECVC(buf, len, MPI_DONTCARE, tag, group); } else { MPI_SENDC(buf, len, 2, tag, group); MPI_BCASTC(buf, len, group, 0); } \end{verbatim} Process zero executes a broadcast followed by a send to process one; process two executes a send to process one, followed by a broadcast; and process one executes a receive, a broadcast and a receive. A possible outcome is for the operations to be matched as illustrated by the diagram below. \begin{verbatim} 0 1 2 / - > receive / - send / / broadcast / broadcast / broadcast / / send - receive < - \end{verbatim} The reason is that broadcast is not a synchronous operation; the call at a process may return before the other processes have entered the broadcast. Thus, the message sent by process zero can arrive to process one before the message sent by process two, and before the call to broadcast on process one. \end{document} From owner-mpi-comm@CS.UTK.EDU Tue Apr 20 16:21:18 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA08359; Tue, 20 Apr 93 16:21:18 -0400 Received: from localhost by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03662; Tue, 20 Apr 93 16:19:33 -0400 X-Resent-To: mpi-comm@CS.UTK.EDU ; Tue, 20 Apr 1993 16:19:30 EDT Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from gstws.EPM.ORNL.GOV by CS.UTK.EDU with SMTP (5.61+IDA+UTK-930125/2.8s-UTK) id AA03654; Tue, 20 Apr 93 16:19:25 -0400 Received: by gstws.epm.ornl.gov (AIX 3.2/UCB 5.64/4.03) id AA16360; Tue, 20 Apr 1993 16:19:21 -0400 Date: Tue, 20 Apr 1993 16:19:21 -0400 From: geist@gstws.epm.ornl.gov (Al Geist) Message-Id: <9304202019.AA16360@gstws.epm.ornl.gov> To: mpi-comm@cs.utk.edu Subject: Postscript of Collective Communication chapter of MPI %!PS-Adobe-2.0 %%Creator: dvips 5.47 Copyright 1986-91 Radical Eye Software %%Title: cc.dvi %%Pages: 17 1 %%BoundingBox: 0 0 612 792 %%EndComments %%BeginProcSet: tex.pro /TeXDict 200 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{ isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10 N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{ /vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{ statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N /FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin /FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array /BitMaps X /BuildChar{CharBuilder}N /Encoding IE N end dup{/foo setfont}2 array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail} B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont setfont}B /ch-width{ch-data dup length 5 sub get}B /ch-height{ch-data dup length 4 sub get}B /ch-xoff{128 ch-data dup length 3 sub get sub}B /ch-yoff{ ch-data dup length 2 sub get 127 sub}B /ch-dx{ch-data dup length 1 sub get}B /ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0 N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0 ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image} imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256 array N 0 1 255{IE S 1 string dup 0 3 index put cvn put}for}N /p /show load N /RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X /rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0 7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1 TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1 -.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{ moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{ S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B /j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w }B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p a}B /bos{/SS save N}B /eos{clear SS restore}B end %%EndProcSet TeXDict begin 1000 400 400 @start /Fa 9 118 400 360 dfs[<7FFFF0FFFFF8FFFFF87F FFF00000000000000000000000007FFFF0FFFFF8FFFFF87FFFF0>21 12 126 148 29 61 D[<1FF0003FFC007FFE00781F00300780000380000380007F8007FF801FFF80 3FC3807C0380F00380E00380E00380E00380F007807C1F803FFFFC1FFDFC07F0FC>22 21 125 148 29 97 D[22 30 127 157 29 I[<00F8FC03FFFE07FFFE0F8F8C0E03801E03C01C01C01C01C01C01C01E03C00E03800F8F80 0FFF001FFE001CF8001C00001C00001E00000FFF801FFFF03FFFF87C00FC70001CF0001EE0000E E0000EE0000EF0001E78003C3F01F81FFFF00FFFE001FF00>23 33 127 148 29 103 D[<01F00007FC001FFF003E0F803C07807803C07001C0E000E0E000E0E000E0E000 E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF0007FC0001F000>19 21 125 148 29 111 D[22 32 127 148 29 I[22 21 126 148 29 114 D[<00C00001C00001C00001C00001C00001C00001C0007FFFE0FFFFE0FF FFE001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C07001C07001 C07001C0F001E1E000FFE0007FC0003F00>20 28 127 155 29 116 D[23 21 127 148 29 I E /Fb 7 118 400 360 dfs[<007C01C2030307070E0F1C0F3C003800780078007800F000F000F000F000F001 70037006301C18380FC0>16 21 123 148 26 99 D[<00007C0000CE00019E00039E00038C0003 00000700000700000700000700000E00000E00000E00000E0001FFF001FFF0001C00001C00001C 00001C00001C0000380000380000380000380000380000700000700000700000700000700000E0 0000E00000E00000E00001C00001C00001C00001C000038000738000F30000F300006600003C00 00>23 45 130 162 17 102 D[<00E000E001E000C00000000000000000000000000000000000 001E00330063806380C380C700C70007000E000E000E001C001C001C40386038C070C070803180 31001E00>11 34 124 161 17 105 D[<1E07803318E063B06063E070C3C070C38070C3807007 00E00700E00700E00700E00E01C00E01C00E03820E03831C03861C07061C070C1C030838031818 01E0>24 21 124 148 31 110 D[<007C0001C6000303000603800E03C01C03C03C03C03803C0 7803C07803C07803C0F00780F00780F00780F00F00F00E00701E00701C003038001860000F8000 >18 21 123 148 28 I[<006000E000E000E000E001C001C001C001C00380FFF8FFF803800700 0700070007000E000E000E000E001C001C001C101C18383038303860186018C00F00>13 31 124 158 19 116 D[<0F003011807021C07061C0E0C1C0E0C380E0C380E00381C00701C007 01C00701C00E03800E03800E03840E03860E070C0C070C0E070C0E0B1806131003E1E0>23 21 124 148 30 I E /Fc 2 61 438 432 dfs[<78FCFCFEFE7A020202020404040810102040> 7 18 123 133 17 59 D[<00000000E000000003E00000000FC00000003F00000000FC00000003 F00000000FC00000003F00000000FC00000003F00000000FC00000003F00000000FC00000003F0 0000000FC00000003F00000000FC00000000F000000000FC000000003F000000000FC000000003 F000000000FC000000003F000000000FC000000003F000000000FC000000003F000000000FC000 000003F000000000FC000000003F000000000FC000000003E000000000E0>35 35 123 159 47 I E /Fd 46 124 400 360 dfs[<000FC0000078300000E0080001803C000380 7C0007007C0007007C0007003800070000000700000007000000070000000700000007000000FF FFFC00FFFFFC0007003C0007001C0007001C0007001C0007001C0007001C0007001C0007001C00 07001C0007001C0007001C0007001C0007001C0007001C0007001C0007001C0007001C007FF1FF C07FF1FFC0>26 35 128 162 31 12 D[<000FC03F00007031E0C000E00B802001803E00F00380 7E01F007007C01F007007C01F007003C00E007001C000007001C000007001C000007001C000007 001C000007001C0000FFFFFFFFF0FFFFFFFFF007001C00F007001C007007001C007007001C0070 07001C007007001C007007001C007007001C007007001C007007001C007007001C007007001C00 7007001C007007001C007007001C007007001C007007001C00707FF1FFC7FF7FF1FFC7FF>40 35 128 162 47 14 D[<701CF83EFC3FFC3F741D04010401040104010802080210041004200840 10>16 15 126 162 28 34 D[<001000200040008001000300060004000C001800180018003000 300030007000600060006000E000E000E000E000E000E000E000E000E000E000E000E000600060 00600070003000300030001800180018000C0004000600030001000080004000200010>12 50 125 164 21 40 D[<800040002000100008000C0006000200030001800180018000C000C000 C000E0006000600060007000700070007000700070007000700070007000700070006000600060 00E000C000C000C00180018001800300020006000C0008001000200040008000>12 50 125 164 21 I[<70F8FCFC7404040404080810102040>6 15 124 132 16 44 D[12 2 127 139 19 I[<70F8F8F870>5 5 124 132 16 I[<70F8F8F870000000000000000000000070F8F8F870>5 21 124 148 16 58 D[<07F000181C00200E00400700F00780F80780F80780F80780700780000F00000E0000 1C0000380000700000600000C00000800000800001800001000001000001000001000001000001 000000000000000000000000000000000003800007C00007C00007C000038000>17 35 125 162 27 63 D[<0007F008003FFC1800FC073801F001B803C000F8078000780F0000381E 0000183E0000183C0000187C0000087C00000878000008F8000000F8000000F8000000F8000000 F8000000F8000000F8000000F8000000780000007C0000087C0000083C0000083E0000081E0000 100F0000100780002003C0004001F0018000FC0700003FFE000007F000>29 34 125 161 40 67 D[32 34 126 161 42 I[28 34 126 161 37 70 D[32 34 126 161 41 72 D[14 34 126 161 20 I[41 34 126 161 51 77 D[32 34 126 161 41 I[<000FF00000781E0000E0070003C003C0078001E00F0000F01E0000781E0000783C00003C3C 00003C7C00003E7800001E7800001EF800001FF800001FF800001FF800001FF800001FF800001F F800001FF800001F7800001E7C00003E7C00003E3C00003C3E00007C1E0000781E0000780F0000 F0078001E003C003C000E0070000781E00000FF000>32 34 125 161 43 I[28 34 126 161 38 I[<7FFFFFFC7FFFFFFC7803C03C6003C00C4003C0044003C004C003C006C003C0068003C00280 03C0028003C0028003C0020003C0000003C0000003C0000003C0000003C0000003C0000003C000 0003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000003C0 000003C0000003C0000003C00001FFFF8001FFFF80>31 34 126 161 40 84 D[32 34 126 161 41 I[<1FF000381C007C06007C07007C0380380380000380000380007F8007C3801E03803C0380 780380780380F00384F00384F00384F00784780B843C11C80FE0F0>22 21 126 148 28 97 D[<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E0000 0E00000E00000E00000E00000E1F800E60E00E80300F00380E001C0E001E0E000E0E000F0E000F 0E000F0E000F0E000F0E000F0E000F0E000E0E001E0E001C0F00380C80700C60E0081F80>24 35 127 162 31 I[<01FE000707000C0F801C0F80380F80780700700000F00000F00000F00000 F00000F00000F00000F000007000007800403800401C00800C010007060001F800>18 21 126 148 24 I[<0000700007F00007F00000F0000070000070000070000070000070000070 00007000007000007000007001F8700706700E01701C00F0380070780070700070F00070F00070 F00070F00070F00070F00070F000707000707800703800701C00F00C017807067F01F87F>24 35 126 162 31 I[<01FC000707000C03801C01C03801C07800E07000E0F000E0FFFFE0F00000 F00000F00000F00000F000007000007800203800201C00400E008007030000FC00>19 21 127 148 24 I[<003E0000E30001C780038F80030F80070700070000070000070000070000 070000070000070000070000FFF800FFF800070000070000070000070000070000070000070000 0700000700000700000700000700000700000700000700000700000700007FF8007FF800>17 35 128 162 17 I[<01F078071D9C0E0E1C1C07001C07003C07803C07803C07803C07801C0700 1C07000E0E000F1C0019F0001000001000001800001C00001FFF000FFFE00FFFF03800F8600018 40001CC0000CC0000CC0000C6000186000183800700E01C001FE00>22 32 127 148 28 I[<0E000000FE000000FE0000001E0000000E0000000E0000000E0000000E000000 0E0000000E0000000E0000000E0000000E0000000E0000000E1F80000E60E0000E8070000F0038 000F0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E00 38000E0038000E0038000E0038000E0038000E003800FFE3FF80FFE3FF80>25 35 127 162 31 I[<1C003E003E003E001C00000000000000000000000000000000000E00FE00 FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC0> 10 34 127 161 16 I[<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E00 000E00000E00000E00000E00000E03FC0E03FC0E01E00E01800E02000E04000E08000E10000E38 000EF8000F1C000E1E000E0E000E07000E07800E03C00E01C00E01E00E01F0FFE3FEFFE3FE>23 35 127 162 29 107 D[<0E00FE00FE001E000E000E000E000E000E000E000E000E000E000E00 0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FF E0FFE0>11 35 127 162 16 I[<0E1FC07F00FE60E18380FE807201C01F003C00E00F003C00E0 0E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800 E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFEFFE3FF 8FFE>39 21 127 148 47 I[<0E1F8000FE60E000FE8070001F0038000F0038000E0038000E00 38000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E 0038000E0038000E003800FFE3FF80FFE3FF80>25 21 127 148 31 I[<00FC000703800E01C0 1C00E0380070780078700038F0003CF0003CF0003CF0003CF0003CF0003CF0003C700038780078 3800701C00E00E01C007038000FC00>22 21 127 148 28 I[<0E1F80FE60E0FE80700F00380E 001C0E001E0E001E0E000F0E000F0E000F0E000F0E000F0E000F0E000F0E001E0E001E0E001C0F 00380E80700E60E00E1F800E00000E00000E00000E00000E00000E00000E00000E0000FFE000FF E000>24 31 127 148 31 I[<01F8200704600E02601C01603801E07800E07800E0F000E0F000 E0F000E0F000E0F000E0F000E0F000E07000E07800E03801E01C01E00C02E0070CE001F0E00000 E00000E00000E00000E00000E00000E00000E00000E0000FFE000FFE>23 31 126 148 29 I[<0E1E00FE6300FE87801E87800F03000F00000E00000E00000E00000E0000 0E00000E00000E00000E00000E00000E00000E00000E00000E0000FFF000FFF000>17 21 127 148 22 I[<0FC4303C600CC00CC004C004E004F0007F803FF00FF800FC001E800E8006 C006C006C004E00CD81887E0>15 21 126 148 22 I[<02000200020002000200060006000600 0E001E003FF8FFF80E000E000E000E000E000E000E000E000E000E000E040E040E040E040E040E 040708030801F0>14 31 127 158 21 I[<0E003800FE03F800FE03F8001E0078000E0038000E 0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E0038000E003800 0E0038000E0078000E0078000700BC0003833F8000FC3F80>25 21 127 148 31 I[23 21 127 148 29 I[33 21 127 148 40 I[23 21 127 148 29 I[23 31 127 148 29 I[24 1 128 140 28 123 D E /Fe 29 118 400 360 dfs[<000C0038007000E001C003C0038007800F000F001E001E003E003C003C007C007C007C00 7800F800F800F800F800F800F800F800F800F800F800F80078007C007C007C003C003C003E001E 001E000F000F000780038003C001C000E000700038000C>14 49 124 164 24 40 D[14 49 125 164 24 I[<3C007E00FF00FF00FF80FF807F803D800180018003000300070006000C001C003800 2000>9 18 124 135 18 44 D[<3C7EFFFFFFFF7E3C0000000000003C7EFFFFFFFF7E3C>8 22 124 149 18 58 D[<0001FF0040001FFFC1C0007F80F3C001FC001FC003F0000FC007E00007 C00FC00003C01FC00003C03F800001C03F800001C07F800000C07F000000C07F000000C0FF0000 0000FF00000000FF00000000FF00000000FF00000000FF00000000FF00000000FF000000007F00 0000007F000000C07F800000C03F800000C03F800001C01FC00001800FC000018007E000030003 F000060001FC001C00007F807800001FFFE0000001FF0000>34 34 125 161 46 67 D[38 34 126 161 49 I[33 34 126 161 42 I[<0001FF0020001FFFE0E0007F 8079E001FC001FE003F80007E007E00003E00FC00001E01FC00001E03F800000E03F800000E07F 800000607F000000607F00000060FF00000000FF00000000FF00000000FF00000000FF00000000 FF00000000FF0007FFFEFF0007FFFE7F00000FE07F00000FE07F80000FE03F80000FE03F80000F E01FC0000FE00FE0000FE007E0000FE003F8000FE001FC001FE0007F8073E0001FFFE1E00001FF 8060>39 34 125 161 50 71 D[ 19 34 127 161 24 73 D[47 34 125 161 60 77 D[<0007FE0000003FFFC00000FE07F00003F801FC0007F000FE000FE0007F001FC0003F801F 80001F803F80001FC03F80001FC07F00000FE07F00000FE07F00000FE0FF00000FF0FF00000FF0 FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF0FF00000FF07F00000F E07F80001FE07F80001FE03F80001FC01FC0003F801FC0003F800FE0007F0007F000FE0003F801 FC0000FE07F000003FFFC0000007FE0000>36 34 125 161 48 79 D[33 34 126 161 43 I[40 34 126 161 48 82 D[<01FE020007FFCE001F01FE003C007E003C001E0078000E0078000E00F8 000600F8000600FC000600FC000000FF000000FFF000007FFF80003FFFE0003FFFF8001FFFFC00 07FFFE0003FFFF00003FFF000001FF0000003F8000001F8000001F80C0000F80C0000F80C0000F 80E0000F00E0000F00F0001E00FC001C00FF807800E7FFF000807FC000>25 34 125 161 36 I[39 34 126 161 49 85 D[27 35 126 162 36 98 D[<00FF8007FFE00F83F01F03F03E03F07E03F07C01E07C0000FC0000FC00 00FC0000FC0000FC0000FC00007C00007E00007E00003F00301F00600FC0E007FF8000FE00>20 22 126 149 28 I[<00FE0007FF800F83E01F01E03E00F07E00F07C00F8FC00F8FC0078FFFFF8 FFFFF8FC0000FC0000FC0000FC00007E00007E00183E00381F00300F80F003FFC000FF00>21 22 126 149 29 101 D[<001F8000FFE001F1F003E3F007E3F00FC3F00FC1E00FC0000FC0000F C0000FC0000FC0000FC000FFFE00FFFE000FC0000FC0000FC0000FC0000FC0000FC0000FC0000F C0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0007FFC007FFC00> 20 35 126 162 20 I[<00FE0F8003FF9FC00F83E3C01F01F3C01E00F0003E00F8003E00F8003E 00F8003E00F8003E00F8001E00F0001F01F0000F83E0000BFF800008FE00001800000018000000 1C0000001FFFE0001FFFFC000FFFFF0007FFFF001FFFFF807C001FC078000FC0F80007C0F80007 C0F80007C07C000F803E001F001F807E000FFFFC0001FFE000>26 33 127 149 32 I[<0E003F807F807F807F807F803F800E00000000000000000000000000FF80FF801F80 1F801F801F801F801F801F801F801F801F801F801F801F801F801F801F801F801F80FFF0FFF0> 12 36 126 163 17 105 D[12 35 126 162 17 108 D[27 22 125 149 36 110 D[<00FF0007FFE00F81F01F00F83E007C7C003E7C003E7C003EFC003FFC003F FC003FFC003FFC003FFC003FFC003F7C003E7E007E3E007C1F00F80F81F007FFE000FF00>24 22 126 149 32 I[27 32 126 149 36 I[20 22 126 149 27 114 D[<07F9801FFF80380780700380F00180F00180F80000FF0000FFF8007FFE003FFF001FFF8007 FF80003FC0C007C0C003C0E003C0E003C0F00380FC0F00EFFE00C3F800>18 22 126 149 26 I[<00C00000C00000C00000C00001C00001C00003C00007C0000FC0001FC000 FFFF00FFFF000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC0000FC000 0FC1800FC1800FC1800FC1800FC18007C18007E30003FE0000FC00>17 32 127 159 24 I[27 22 125 149 36 I E /Ff 65 126 438 432 dfs[<00F0000003F8000003FC000007FC0000071E00000F0E00000E0E00000E 0E00000E0E00000E0E00000E0E00000E1E7FC00E3CFFC00E7CFFC007787FC007F0380007F03800 07E0380007C070000F8070001F8070003FC0E0007DC0E00078E0E00078E1C000F071C000F07B80 00F03B8000F01F0000F01F01C0F00E01C0781F81C0787FC3C03FFBFF803FF1FF801FE0FF000780 3C00>26 37 126 164 31 38 D[<000F001F003E007C00F801F003E007C00F800F001E001E003C 003C003C00780078007800F000F000F000F000F000F000F000F000F000F000F000780078007800 3C003C003C001E001E000F000F8007C003E001F000F8007C003F001F000F>16 47 119 169 31 40 D[<7000F8007C003E001F000F8007C003E001F000F000780078003C003C00 3C001E001E001E000F000F000F000F000F000F000F000F000F000F000F001E001E001E003C003C 003C0078007800F001F003E007C00F801F003E007C00F8007000>16 47 123 169 31 I[<000E0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000 001F0000001F0000001F00007FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF80001F0000001F00 00001F0000001F0000001F0000001F0000001F0000001F0000001F0000001F0000000E0000>26 27 126 159 31 43 D[<1C003F007F007F807F803F801F8007800F800F001F007E00FC00F80060 00>9 15 117 134 31 I[<7FFFFEFFFFFFFFFFFFFFFFFF7FFFFE>24 5 125 148 31 I[<387CFEFEFE7C38>7 7 116 134 31 I[<00000E00001F00001F00003F00003E0000 7E00007C00007C0000FC0000F80001F80001F00003F00003E00007E00007C0000FC0000F80000F 80001F80001F00003F00003E00007E00007C0000FC0000F80001F80001F00001F00003F00003E0 0007E00007C0000FC0000F80001F80001F00003F00003E00003E00007E00007C0000FC0000F800 00F80000700000>24 47 125 169 31 I[<007E0001FF8003FFC007FFE00FC3F01F00F81E0078 3E007C3C003C7C003E78001E78001E78001EF0000FF0000FF0000FF0000FF0000FF0000FF0000F F0000FF0000FF0000FF8001F78001E78001E78001E7C003E3C003C3E007C1F00F81F81F80FC3F0 07FFE003FFC001FF80007E00>24 37 125 164 31 I[<00700000700000F00000F00001F00003 F00007F0007FF000FFF000FEF000F8F00000F00000F00000F00000F00000F00000F00000F00000 F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000 F00000F0007FFFE0FFFFF0FFFFF07FFFE0>20 37 122 164 31 I[<00FE0003FFC00FFFE01FFF F83E03FC7C007C78003EF0001EF0000FF8000FF8000F70000F00000F00000F00001E00001E0000 1E00003C00007C0000F80001F00003E00007C0000F80001F00003E00007C0001F80003F00007C0 000F800F1F000F3E000F7FFFFFFFFFFFFFFFFF7FFFFF>24 37 125 164 31 I[<1C3E7F7F7F3E1C0000000000000000000000001C3E7E7F7F3F1F0F1F1E3E7CF8F060>8 34 117 153 31 59 D[<00000E00001F00007F0000FF0003FE0007FC001FF0003FE000FF8001FF 0007FC000FF8003FE0007FC000FF0000FE0000FF00007FC0003FE0000FF80007FC0001FF0000FF 80003FE0001FF00007FC0003FE0000FF00007F00001F00000E>24 31 125 161 31 I[<7FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF800000000000000000000000000000 0000000000007FFFFF80FFFFFFC0FFFFFFC0FFFFFFC07FFFFF80>26 15 126 153 31 I[<700000F80000FE0000FF00007FC0003FE0000FF80007FC0001FF0000FF80003F E0001FF00007FC0003FE0000FF00007F0000FF0003FE0007FC001FF0003FE000FF8001FF0007FC 000FF8003FE0007FC000FF0000FE0000F80000700000>24 31 125 161 31 I[<001E0000003F0000003F0000003F00000073800000738000007380000073800000738000 00F3C00000F3C00000F3C00000E1C00001E1E00001E1E00001E1E00001E1E00001E1E00003C0F0 0003C0F00003C0F00003C0F00007C0F80007FFF80007FFF80007FFF80007FFF8000F003C000F00 3C000F003C000F003C000F003C001E001E00FFC0FFC0FFE1FFC0FFE1FFC0FFC0FFC0>26 37 126 164 31 65 D[25 37 126 164 31 I[<001F81C0007FE1C001FFFBC003FFFFC007F03FC00FC01FC01F80 0FC01F0007C03E0007C03C0003C07C0003C0780003C0780003C078000000F0000000F0000000F0 000000F0000000F0000000F0000000F0000000F0000000F00000007800000078000000780003C0 7C0003C03C0003C03E0003C01F0007801F8007800FC00F0007F03F0003FFFE0001FFFC00007FF0 00001FC000>26 37 126 164 31 I[<7FFF8000FFFFE000FFFFF8007FFFFC000F00FE000F003E 000F001F000F000F800F000F800F0007800F0007C00F0003C00F0003C00F0003E00F0001E00F00 01E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0003E00F0003C00F 0003C00F0003C00F0007C00F000F800F000F800F001F000F003E000F00FE007FFFFC00FFFFF800 FFFFF0007FFF8000>27 37 127 164 31 I[27 37 126 164 31 I[26 37 126 164 31 I[<003F070000FFC70001FFEF0007FF FF0007E0FF000F807F001F003F001E001F003E001F003C000F007C000F0078000F0078000F00F8 000000F0000000F0000000F0000000F0000000F0000000F0000000F001FFC0F001FFE0F001FFE0 F801FFC078000F0078000F0078001F003C001F003E001F001E001F001F003F000F807F0007E0FF 0007FFFF0001FFEF0000FFCF00003F0F00>27 37 126 164 31 I[<7FE07FE0FFF0FFF0FFF0FF F07FE07FE00F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F000F000F000F000F000F000FFFFF000FFFFF000FFFFF000FFFFF000F000F000F000F000F 000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F000F00 0F000F007FE07FE0FFF0FFF0FFF0FFF07FE07FE0>28 37 127 164 31 I[<7FFFF8FFFFFCFFFF FC7FFFF80078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078000078000078000078000078 000078000078000078000078007FFFF8FFFFFCFFFFFC7FFFF8>22 37 124 164 31 I[<7FC07FC0FFE0FFC0FFE0FFC07FC07FC00E001C000E0038000E0078000E00F0000E00 E0000E01C0000E03C0000E0780000E0700000E0E00000E1E00000E3C00000E3E00000E7F00000E F700000EE780000FC380000FC1C0000F81C0000F00E0000E00E0000E0070000E0070000E003800 0E0038000E001C000E001C000E000E000E000E007FC01FE0FFE03FE0FFE03FE07FC01FE0>27 37 127 164 31 75 D[<7FFC0000FFFE0000FFFE00007FFC000007800000078000000780000007 800000078000000780000007800000078000000780000007800000078000000780000007800000 078000000780000007800000078000000780000007800000078000000780000007800000078001 80078003C0078003C0078003C0078003C0078003C0078003C07FFFFFC0FFFFFFC0FFFFFFC07FFF FFC0>26 37 126 164 31 I[28 37 127 164 31 I[<7F00FF80FF81FFC0FF81FFC07FC0FF800EC01C000EC01C 000EE01C000E601C000E601C000E701C000E701C000E301C000E381C000E381C000E381C000E18 1C000E1C1C000E1C1C000E0C1C000E0E1C000E0E1C000E061C000E071C000E071C000E071C000E 031C000E039C000E039C000E019C000E019C000E01DC000E00DC000E00DC007FC0FC00FFE07C00 FFE07C007FC03C00>26 37 126 164 31 I[<03FFC01FFFF83FFFFC3FFFFC7E007E7C003E7800 1E78001EF8001FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF000 0FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF0000FF8001F78001E78001E7C003E7F00 FE3FFFFC3FFFFC1FFFF803FFC0>24 37 125 164 31 I[25 37 126 164 31 I[<7FFF0000FFFFE000FFFFF000 7FFFF8000F01FC000F007E000F001E000F001F000F000F000F000F000F000F000F000F000F001F 000F001E000F007E000F01FC000FFFF8000FFFF0000FFFE0000FFFF0000F01F8000F0078000F00 7C000F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C780F003C780F 003E787FE01FF0FFF01FF0FFF00FE07FE003C0>29 37 127 164 31 82 D[<01FC1C07FF9C0FFFFC3FFFFC3E03FC7C00FC78007CF0003CF0003CF0003CF0003CF0000078 00007C00003E00003FE0001FFE0007FFC001FFF0001FF80001FC00007C00001E00001E00000F00 000F70000FF0000FF0000FF0001FF8001EFC003EFF00FCFFFFF8FFFFF0E3FFE0E0FF80>24 37 125 164 31 I[<7FFFFFC0FFFFFFC0FFFFFFC0FFFFFFC0F01E03C0F01E03C0F01E03C0F01E 03C0F01E03C0F01E03C0001E0000001E0000001E0000001E0000001E0000001E0000001E000000 1E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000 001E0000001E0000001E0000001E0000001E0000001E000003FFF00007FFF80007FFF80003FFF0 00>26 37 126 164 31 I[28 37 127 164 31 I[<7FC03FE0FFE07FF0FFE07FF07FC03FE00F000F000F000F00 0F000F000F000F0007801E0007801E0007801E0007801E0003C03C0003C03C0003C03C0003C03C 0003E07C0001E0780001E0780001E0780001E0780000F0F00000F0F00000F0F00000F0F0000070 E0000079E0000079E0000079E0000039C0000039C0000039C0000039C000001F8000001F800000 1F8000000F0000>28 37 127 164 31 I[28 37 127 164 31 I[<3FFFFF7FFFFF7FFFFF7FFFFF78001E78003C 78007C7800787800F07801F00001E00003E00003C0000780000F80000F00001E00003E00003C00 007C0000780000F00001F00001E00003C00007C00007800F0F800F0F000F1E000F3E000F3C000F 78000FFFFFFFFFFFFFFFFFFFFFFFFF>24 37 125 164 31 90 D[16 47 116 169 31 I[<700000F80000F80000FC00007C00007E00003E00003E0000 3F00001F00001F80000F80000FC00007C00007E00003E00003F00001F00001F00001F80000F800 00FC00007C00007E00003E00003F00001F00001F80000F80000F80000FC00007C00007E00003E0 0003F00001F00001F80000F80000FC00007C00007C00007E00003E00003F00001F00001F00000E >24 47 125 169 31 I[16 47 126 169 31 I[<7FFFFEFFFFFFFFFFFFFFFFFF7FFFFE>24 5 125 126 31 95 D[<07FC00001FFF00003FFFC0003FFFE0003E03F0001C01F0000000F8000000780000007800 00007800007FF80003FFF8000FFFF8003FE078007E00780078007800F0007800F0007800F00078 00F00078007800F8007E03F8003FFFFFE03FFFFFE00FFE3FE003F00FE0>27 26 125 153 31 97 D[27 37 127 164 31 I[<007FC001FFF007FFF80FFFF81F80F83E00703C00007800007800 00F80000F00000F00000F00000F00000F00000F00000F800007800007C00783E00783F00F81FC1 F00FFFE007FFE001FF80007E00>21 26 123 153 31 I[<0007FC000007FC000007FC000007FC 0000003C0000003C0000003C0000003C0000003C0000003C0000003C0000FC3C0003FF3C0007FF BC000FFFFC001F81FC003E00FC003C007C007C003C0078003C00F8003C00F0003C00F0003C00F0 003C00F0003C00F0003C00F0003C00F8003C0078007C0078007C003C00FC003E01FC001F83FC00 1FFFFFE007FFBFE003FE3FE000F83FE0>27 37 126 164 31 I[<007F0001FFC007FFE00FFFF0 1F81F83F00783C003C7C003C78001E78001EFFFFFEFFFFFEFFFFFEFFFFFEF00000F00000780000 7800007C001E3E001E1F803E1FE07C0FFFF803FFF001FFE0003F80>23 26 125 153 31 I[<0001F80007FC000FFE001FFE003E3E007C1C0078000078000078000078000078 007FFFFCFFFFFCFFFFFCFFFFFC0078000078000078000078000078000078000078000078000078 000078000078000078000078000078000078000078000078000078007FFFF87FFFF87FFFF87FFF F8>23 37 126 164 31 I[<00FC0F8003FF3FC007FFFFE00FFFFFE00F87E1C01F03E0001E01E0 003C00F0003C00F0003C00F0003C00F0003C00F0003C00F0001E01E0001F03E0000F87C0000FFF C0001FFF80001FFF00001CFC00001C0000001C0000000E0000000FFFE0001FFFF8003FFFFE003C 001F007800078070000380E00001C0E00001C0E00001C0E00001C0700003807C000F803F003F00 1FFFFE000FFFFC0003FFF000007F8000>27 40 126 153 31 I[28 37 127 164 31 I[<00300000780000FC00 00FC000078000030000000000000000000000000000000000000007FFC007FFC007FFC007FFC00 003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C007FFFFCFFFFFEFFFFFE7FFFFC>23 38 124 165 31 I[ 27 37 126 164 31 107 D[24 37 125 164 31 I[29 26 128 153 31 I[28 26 127 153 31 I[<00FC0003FF0007FF801FFFE01F87E03E01F07C00F87800787800 78F0003CF0003CF0003CF0003CF0003CF0003CF0003CF8007C7800787C00F87C00F83E01F01F87 E01FFFE007FF8003FF0000FC00>22 26 124 153 31 I[27 39 127 153 31 I[26 26 126 153 31 114 D[<03FC700FFFF03FFFF07FFFF07C03F0F801F0F000F0F000F0F000F07C 00007FE0001FFF0007FFC000FFE00003F00000F870003CF0003CF0003CF8003CFC007CFF01F8FF FFF0FFFFF0E7FFC0E1FE00>22 26 124 153 31 I[<0070000000F0000000F0000000F0000000 F0000000F0000000F000007FFFFE00FFFFFE00FFFFFE00FFFFFE0000F0000000F0000000F00000 00F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F007 8000F0078000F0078000F0078000F80F00007C1F00007FFE00003FFC00001FF8000007E000>25 33 127 160 31 I[28 26 127 153 31 I[<7FE07FE0FFF0FFF0FFF0FFF07FE07FE007000E0007000E0007801E000380 1C0003801C0003C03C0001C0380001C0380001E0780000E0700000E0700000E070000070E00000 70E0000070E0000039C0000039C0000039C000001F8000001F8000001F8000000F0000>28 26 127 153 31 I[<7FF0FFC07FF1FFE07FF1FFE07FF0FFC003C0380001E0700000E0F0000070 E0000079C000003FC000001F8000000F0000000E0000000F0000001F8000003B80000039C00000 70E00000E0F00001E0700001C0380003801C007FE07FE0FFF0FFF0FFF0FFF07FE07FE0>28 26 127 153 31 120 D[<3FFFFF807FFFFF807FFFFF807FFFFF8078003F0078007E007800FC00 7801F8000003F0000007E000000FC000001F8000003F0000007E000000FC000001F8000003F000 0007E007800FC007801F8007803F0007807E000780FFFFFF80FFFFFF80FFFFFF80FFFFFF80>25 26 126 153 31 122 D[<00007F0003FF000FFF001FFF001F00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007C0000F8007FF800FF E000FFC000FFE0007FF80000F800007C00003C00003C00003C00003C00003C00003C00003C0000 3C00003C00003C00003C00003C00003C00003C00001F00001FFF000FFF0003FF00007F>24 47 125 169 31 I[<7E0000FFC000FFF000FFF80000F800003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C00003C00003C00003C00003C00003E00001F00001FFE0007FF 0003FF0007FF001FFE001F00003E00003C00003C00003C00003C00003C00003C00003C00003C00 003C00003C00003C00003C00003C00003C0000F800FFF800FFF000FFC0007E0000>24 47 125 169 31 125 D E /Fg 47 123 438 432 dfs[<0001FF81FE00001FFFEFFF80007F80FF 87C000FC00FE0FE001F801FE0FE003F801FC0FE007F001FC0FE007F001FC07C007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC0000FFFFFFFF F800FFFFFFFFF800FFFFFFFFF80007F001FC000007F001FC000007F001FC000007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC 000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007F001FC000007 F001FC000007F001FC000007F001FC000007F001FC00007FFF1FFFE0007FFF1FFFE0007FFF1FFF E000>43 42 127 169 41 11 D[<00030007000E001C0038007000F001E003C003C007C007800F 800F001F001F003F003E003E007E007E007E007C007C00FC00FC00FC00FC00FC00FC00FC00FC00 FC00FC00FC00FC007C007C007E007E007E003E003E003F001F001F000F000F80078007C003C003 C001E000F000700038001C000E00070003>16 60 122 172 27 40 D[<8000C000E00070003800 1C001E000F000780078007C003C003E001E001F001F001F800F800F800FC00FC00FC007C007C00 7E007E007E007E007E007E007E007E007E007E007E007E007C007C00FC00FC00FC00F800F801F8 01F001F001E003E003C007C0078007800F001E001C0038007000E000C0008000>15 60 123 172 27 I[<1C007F007F00FF80FFC0FFC07FC07FC01CC000C000C00180018001800300 030006000C00180030002000>10 21 123 136 19 44 D[17 6 127 144 23 I[<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE007FFFFE7FFFFE7FFFFE>23 39 123 166 34 49 D[<000003800000000007C000000000 07C0000000000FE0000000000FE0000000000FE0000000001FF0000000001FF0000000003FF800 0000003FF8000000003FF80000000073FC0000000073FC00000000F3FE00000000E1FE00000000 E1FE00000001C0FF00000001C0FF00000003C0FF80000003807F80000007807FC0000007003FC0 000007003FC000000E003FE000000E001FE000001E001FF000001C000FF000001FFFFFF000003F FFFFF800003FFFFFF80000780007FC0000700003FC0000700003FC0000E00001FE0000E00001FE 0001E00001FF0001C00000FF0001C00000FF00FFFE001FFFFEFFFE001FFFFEFFFE001FFFFE>47 41 126 168 53 65 D[41 41 125 168 50 I[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE00007F80 03FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803FC000 0003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF8000000000 FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F8000 0000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE000000700 0FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E000000FFF FFC0000003FFFF000000003FF80000>41 41 124 168 51 I[ 45 41 125 168 54 I[39 41 125 168 46 I[36 41 125 168 44 I[<00007FE003000003FFFC0700001FFFFF0F00003FF00FFF0000FF 8001FF0001FE0000FF0003F800003F0007F000003F000FF000001F001FE000000F001FE000000F 003FC000000F003FC0000007007FC0000007007F80000007007F8000000000FF8000000000FF80 00000000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000 00FF8001FFFFF87F8001FFFFF87F8001FFFFF87FC00000FF003FC00000FF003FC00000FF001FE0 0000FF001FE00000FF000FF00000FF0007F00000FF0003F80000FF0001FE0000FF0000FF8001FF 00003FF007BF00001FFFFF1F000003FFFE0F0000007FF00300>45 41 124 168 55 I[47 41 125 168 55 I[22 41 126 168 26 I[47 41 125 168 55 75 D[34 41 125 168 42 I[58 41 125 168 66 I[47 41 125 168 55 I[<0000FFE000000007FFFC0000003FC07F8000007F001FC00001FC0007F00003F80003 F80007F00001FC000FF00001FE001FE00000FF001FE00000FF003FC000007F803FC000007F807F C000007FC07F8000003FC07F8000003FC07F8000003FC0FF8000003FE0FF8000003FE0FF800000 3FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF 8000003FE07F8000003FC07FC000007FC07FC000007FC03FC000007F803FC000007F801FE00000 FF001FE00000FF000FF00001FE0007F00001FC0003F80003F80001FC0007F00000FF001FE00000 3FC07F8000000FFFFE00000000FFE00000>43 41 124 168 53 I[ 39 41 125 168 48 I[48 41 125 168 53 82 D[<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC00 01E0FC0001E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001F FFFE000FFFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8 600001F8E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F 80F3FFFF00E0FFFE00C01FF000>29 41 124 168 39 I[<7FFFFFFFFFC07FFFFFFFFFC07FFFFF FFFFC07F803FC03FC07E003FC007C078003FC003C078003FC003C070003FC001C0F0003FC001E0 F0003FC001E0E0003FC000E0E0003FC000E0E0003FC000E0E0003FC000E0E0003FC000E000003F C0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000 00003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003F C0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC0000000003FC00000 00003FC0000000003FC00000007FFFFFE000007FFFFFE000007FFFFFE000>43 40 126 167 49 I[46 41 125 168 54 I[<7FFFF81FFFF07FFFF81FFFF07FFFF81FFFF001FF0000780000FF8000F000007FC001E00000 7FC001C000003FE003C000001FF0078000000FF80F0000000FF80E00000007FC1E00000003FE3C 00000003FE7800000001FF7000000000FFF0000000007FE0000000007FC0000000003FE0000000 001FF0000000001FF0000000001FF8000000001FFC000000003FFE000000007BFE00000000F1FF 00000000E0FF80000001E0FFC0000003C07FC0000007803FE0000007001FF000000F001FF00000 1E000FF800003C0007FC0000380003FE0000780003FE0000F00001FF0000E00000FF80FFFF801F FFFEFFFF801FFFFEFFFF801FFFFE>47 41 126 168 53 88 D[<3FFFFFFF803FFFFFFF803FFFFF FF803FF000FF003F8001FE003F0003FE003E0003FC003C0007F8007C000FF80078000FF0007800 1FE00070003FE00070003FC00070007F80007000FF80000000FF00000001FE00000003FE000000 03FC00000007F80000000FF80000000FF00000001FE00000003FE00000003FC001C0007F8001C0 00FF8001C000FF0001C001FE0001C003FE0003C003FC0003C007F80003C00FF80007800FF00007 801FE0000F803FE0001F803FC0007F807F8001FF80FFFFFFFF80FFFFFFFF80FFFFFFFF80>34 41 124 168 43 90 D[<01FF800007FFF0000F81F8001FC07E001FC07E001FC03F000F803F8007 003F8000003F8000003F8000003F80000FFF8000FFFF8007FC3F800FE03F803F803F803F003F80 7F003F80FE003F80FE003F80FE003F80FE003F807E007F807F00DF803F839FFC0FFF0FFC01FC03 FC>30 27 126 154 33 97 D[33 42 126 169 39 I[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F803F801F007F80 0E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF000000FF0000007F 0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FFFC00001FE000 >26 27 126 154 31 I[<00003FF80000003FF80000003FF800000003F800000003F800000003 F800000003F800000003F800000003F800000003F800000003F800000003F800000003F8000000 03F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F8001F8003F8003F 8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F800FF0003F800 FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003F8001F8003F8 000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80>33 42 126 169 39 I[<003FE00001FFF80003F07E0007C01F000F801F801F800F803F800FC07F000FC0 7F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F0000 007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF000>27 27 126 154 32 I[<0007F0003FFC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007F000 07F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007F000 07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F0007FFF807FFF807FFF80>24 42 126 169 21 I[<00FF81F003FFE7F80FC1 FE7C1F80FC7C1F007C383F007E107F007F007F007F007F007F007F007F007F007F007F007F003F 007E001F007C001F80FC000FC1F8001FFFE00018FF800038000000380000003C0000003E000000 3FFFF8001FFFFF001FFFFF800FFFFFC007FFFFE01FFFFFF03E0007F07C0001F8F80000F8F80000 F8F80000F8F80000F87C0001F03C0001E01F0007C00FC01F8003FFFE00007FF000>30 40 126 154 34 I[33 42 125 169 39 I[<07000FC01FE03FE03FE03FE01FE00FC007000000000000000000000000000000FFE0 FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00F E00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 20 I[31 42 126 169 37 107 D[15 42 125 169 20 I[33 27 125 154 39 110 D[<003FE00001FFFC0003F07E000FC01F801F800F C03F800FE03F0007E07F0007F07F0007F07F0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF00 07F8FF0007F8FF0007F8FF0007F87F0007F07F0007F03F800FE03F800FE01F800FC00FC01F8007 F07F0001FFFC00003FE000>29 27 126 154 34 I[33 39 126 154 39 I[24 27 126 154 28 114 D[<03FE300FFFF01E03F03800F0700070F00070F000 70F80070FC0000FFE0007FFE007FFF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE000 3CF0003CF00038F80038FC0070FF01E0F7FFC0C1FF00>22 27 126 154 27 I[<00700000700000700000700000F00000F00000F00001F00003F00003F00007F0001FFFF0 FFFFF0FFFFF007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000 07F00007F00007F03807F03807F03807F03807F03807F03803F03803F87001F86000FFC0001F80 >21 38 127 165 27 I[33 27 125 154 39 I[33 39 127 154 37 121 D[<3FFFFF803FFFFF803F007F003C00FE0038 01FE007803FC007803F8007007F800700FF000700FE000001FC000003FC000007F8000007F0000 00FF000001FE038001FC038003F8038007F803800FF007800FE007801FE007003FC00F003F801F 007F007F00FFFFFF00FFFFFF00>25 27 126 154 31 I E /Fh 25 87 438 432 dfs[<387C7EFC7C38>7 6 123 133 17 46 D[<0000C0000180000780007F8007F7800787 80000780000F00000F00000F00000F00000F00000F00001E00001E00001E00001E00001E00001E 00003C00003C00003C00003C00003C00003C0000780000780000780000780000780000780000F0 0000F00000F00000F00000F00000F00001F000FFFFE0FFFFE0>19 40 122 167 30 49 D[<0003FC00000FFF00003C0F80006007C0008003E001E003E001F003E003F003E0 03F003E001E003E000C007E0000007C0000007C000000F8000001F0000001E00000038000000E0 00001FE0000000780000001C0000001E0000000F0000000F8000000F8000000FC000000FC00000 0FC018000FC07C000FC07C000F80FC001F80FC001F80F8001F0040003E0040003E0020007C0038 00F8001E03E00007FFC00000FE0000>27 41 125 167 30 51 D[<00C000C000F0078000FFFF00 00FFFE0001FFF80001FFE000011F00000100000001000000010000000200000002000000020000 000200000002000000021F800004E0E000050070000600380004003C0000003C0000001E000000 1E0000001E0000001E0000003E0000003E0070003E00F8003E00F8003E00F8007C00F0007C0080 0078008000F8008000F000C001E0004003C00060078000381F00000FFC000007F00000>26 41 123 167 30 53 D[<0000FC000007FF00000F0180003C01C0007003C000E007C001C007C003 C0078007800000070000000F0000001F0000001E0000003E0000003E0000003C1F80007C606000 7C8038007D001C00FA001C00FA001E00FC000F00FC000F00F8000F00F8000F00F8001F00F0001F 00F0001F00F0001F00F0001F00F0001E00F0003E00F0003C00F0003C0070007800700070003800 E0003C01C0001E0780000FFE000003F80000>26 41 123 167 30 I[<080000000C0000001FFF FFC01FFFFFC01FFFFF803FFFFF8020000100600002004000040040000800800010008000200000 004000000040000000800000010000000200000006000000040000000800000018000000380000 00300000007000000060000000E0000000E0000001E0000001C0000003C0000003C0000007C000 0007800000078000000F8000000F8000000F8000000F8000001F0000001F0000001F0000000E00 0000>26 42 120 168 30 I[<0000003000000000300000000070000000007000000000F00000 0000F800000001F800000001F800000002F800000002F800000004FC0000000C7C000000087C00 0000107C000000107C000000207C000000203E000000403E000000403E000000803E000000803E 000001003F000001001F000002001F000002001F000004001F000004001F80000FFFFF80000FFF FF800010000F800010000F800020000F8000200007C000400007C000C00007C000800007C00180 0007C001000007E003800003E00FC00007E0FFF000FFFFFFF000FFFF>40 42 126 169 46 65 D[<01FFFFFF0003FFFFFFC0000FC003F0000F8000F8000F8000FC000F8000 7C000F80007E000F80007E001F00003E001F00003E001F00007E001F00007E001F00007C001F00 00FC003E0000F8003E0001F0003E0003E0003E0007C0003E001F00003FFFFC00007C000F80007C 0007C0007C0003F0007C0001F0007C0000F8007C0000F800F80000FC00F80000FC00F80000FC00 F80000FC00F80000FC00F80000F801F00001F801F00001F801F00003F001F00007E001F0000FC0 01F0001F8003F0007F00FFFFFFFC00FFFFFFE000>39 41 126 168 43 I[<00000FF00100007F FE030001FC07070007E0018E001F80005E003E00003E007C00003E00F800001E01F000001E03E0 00000C07C000000C07C000000C0F8000000C1F8000000C1F0000000C3F000000083F000000007E 000000007E000000007E000000007E00000000FC00000000FC00000000FC00000000FC00000000 FC00000000FC00000000FC00000000FC000000207C000000207C000000207C000000403E000000 403E000000801E000000801F000001000F8000020007C000040003E000180001F0003000007E01 E000003FFF80000007FC0000>40 43 122 169 44 I[<01FFFFFF800003FFFFFFE000000FC003 F800000F80007C00000F80003E00000F80001F00000F80000F80000F80000780001F000007C000 1F000007C0001F000003E0001F000003E0001F000003E0001F000003E0003E000003F0003E0000 03F0003E000003F0003E000003F0003E000003F0003E000003F0007C000003E0007C000007E000 7C000007E0007C000007E0007C000007C0007C000007C000F800000FC000F800000F8000F80000 1F8000F800001F0000F800001F0000F800003E0001F000007C0001F00000F80001F00001F00001 F00003E00001F00007C00001F0001F800003F000FE0000FFFFFFF80000FFFFFFC00000>44 41 126 168 47 I[<01FFFFFFFF03FFFFFFFF000FC0007F000F80000F000F800007000F800007 000F800003000F800003001F000003001F000003001F000001001F000001001F000801001F0008 01003E001000003E001000003E001000003E003000003E00F000003FFFF000007FFFE000007C00 E000007C006000007C006000007C002000007C00200200F800400400F800400400F800000400F8 00000800F800000800F800001801F000001001F000003001F000003001F000007001F00000E001 F00003E003F0000FE0FFFFFFFFC0FFFFFFFFC0>40 41 126 168 42 I[<01FFFFFFFE03FFFFFF FE000FC0007E000F80001E000F80000E000F800006000F800006000F800006001F000006001F00 0006001F000002001F000002001F001002001F001002003E002000003E002000003E002000003E 006000003E01E000003FFFE000007FFFC000007C01C000007C00C000007C00C000007C00400000 7C00400000F800800000F800800000F800000000F800000000F800000000F800000001F0000000 01F000000001F000000001F000000001F000000001F000000003F0000000FFFFE00000FFFFE000 00>39 41 126 168 40 I[<00000FF00100007FFE030001FC07070007E0018E001F80005E003E 00003E007C00003E00F800001E01F000001E03E000000C07C000000C07C000000C0F8000000C1F 8000000C1F0000000C3F000000083F000000007E000000007E000000007E000000007E00000000 FC00000000FC00000000FC00000000FC00000000FC0001FFFFFC0001FFFFFC000003F0FC000003 E07C000003E07C000003E07C000003E03E000003E03E000007C01F000007C01F000007C00F8000 07C007C0000FC003E00013C001F8002380007E01C180003FFF00800007FC0000>40 43 122 169 48 I[<01FFFF03FFFE03FFFF07FFFE000FC0001F80000F80001F00000F80001F00 000F80001F00000F80001F00000F80001F00001F00003E00001F00003E00001F00003E00001F00 003E00001F00003E00001F00003E00003E00007C00003E00007C00003E00007C00003E00007C00 003E00007C00003FFFFFFC00007FFFFFF800007C0000F800007C0000F800007C0000F800007C00 00F800007C0000F80000F80001F00000F80001F00000F80001F00000F80001F00000F80001F000 00F80001F00001F00003E00001F00003E00001F00003E00001F00003E00001F00003E00001F000 03E00003F00007E000FFFF81FFFF00FFFF81FFFF00>47 41 126 168 46 I[<03FFFF03FFFF000FC0000F80000F80000F80000F80000F80001F00001F00001F00001F0000 1F00001F00003E00003E00003E00003E00003E00003E00007C00007C00007C00007C00007C0000 7C0000F80000F80000F80000F80000F80000F80001F00001F00001F00001F00001F00001F00003 F000FFFFC0FFFFC0>24 41 126 168 22 I[<01FFFF800003FFFF8000000FC00000000F800000 000F800000000F800000000F800000000F800000001F000000001F000000001F000000001F0000 00001F000000001F000000003E000000003E000000003E000000003E000000003E000000003E00 0000007C000000007C000000007C000000007C000000007C000000007C00002000F800004000F8 00004000F800004000F800008000F800008000F800018001F000018001F000030001F000030001 F000070001F0000E0001F0003E0003F001FE00FFFFFFFC00FFFFFFFC00>35 41 126 168 38 76 D[<01FF80000007FF8003FFC000000BFF80000BC000000FE000000BC00000 17C000000BC0000017C0000009E0000027C0000009E0000027C0000009E0000047C0000011E000 004F80000011E000008F80000010F000010F80000010F000010F80000010F000020F80000010F0 00020F800000207800041F000000207800041F000000207800081F000000207800081F00000020 7800101F000000203C00201F000000403C00203E000000403C00403E000000403C00403E000000 401E00803E000000401E00803E000000401E01003E000000801E01007C000000801E02007C0000 00800F04007C000000800F04007C000000800F08007C000000800F08007C00000100079000F800 000100079000F80000010007A000F80000010007A000F80000010007C000F800000300038000F8 00000FC0038001F80000FFF803007FFFC000FFF803007FFFC000>57 41 126 168 56 I[<01FFC0003FFE03FFC0007FFE000FE00007E0000BF00003800009F00001000009 F80001000008F80001000008FC00010000107C00020000107E00020000103E00020000103F0002 0000101F00020000101F80020000200F80040000200FC00400002007C00400002007E004000020 03E00400002003F00400004001F00800004001F80800004000F80800004000FC08000040007C08 000040007E08000080003E10000080003F10000080001F10000080001F90000080000F90000080 0007D00001000007E00001000003E00001000003E00001000001E00001000001E00003000000E0 000FC00000C000FFF800004000FFF800004000>47 41 126 168 46 I[<00001FF0000000F03C 000003C00700000F0003C0001E0001E0003C0000F000F80000F001F000007801E000007C03C000 007C07C000003C0F8000003E0F8000003E1F0000003E1F0000003E3F0000003F3E0000003F7E00 00003F7E0000003F7E0000003F7E0000003FFC0000007EFC0000007EFC0000007EFC0000007EFC 0000007CFC000000FCFC000000FCFC000000F8FC000001F87C000001F07C000003F07C000003E0 3E000007C03E00000F801E00000F001F00001E000F80003C00078000780003C000F00000F003C0 00007C0F0000000FF80000>40 43 122 169 47 I[<01FFFFFF0003FFFFFFE0000FC003F0000F 8000F8000F80007C000F80003E000F80003F000F80003F001F00003F001F00003F001F00003F00 1F00003F001F00003F001F00007E003E00007E003E00007C003E0000F8003E0001F0003E0007E0 003E001F80007FFFFC00007C000000007C000000007C000000007C000000007C00000000F80000 0000F800000000F800000000F800000000F800000000F800000001F000000001F000000001F000 000001F000000001F000000001F000000003F0000000FFFF800000FFFF800000>40 41 126 168 42 I[<01FFFFFC000003FFFFFF8000000FC00FE000000F8001F000000F8000F800 000F80007C00000F80007C00000F80007E00001F00007E00001F00007E00001F00007E00001F00 007E00001F00007E00001F0000FC00003E0000F800003E0001F800003E0001E000003E0003C000 003E000F0000003E007C0000007FFFE00000007C00700000007C003C0000007C001E0000007C00 0F0000007C000F000000F8000F800000F8000F800000F8000F800000F8000F800000F8000F8000 00F8000F800001F0001F800001F0001F800001F0001F800001F0001F804001F0001F804001F000 1F808003F0000FC080FFFF8007C100FFFF8003C20000000000FC00>42 42 126 168 45 82 D[<0001FC020007FF06001E038E003800DC0070007C00E0003C01E0001C03C0 001C03C0001C0380000807800008078000080780000807C0000807C0000007E0000003F0000003 FE000001FFE00001FFFE0000FFFF00003FFF80000FFFC00000FFE000000FE0000003F0000001F0 000001F0000001F0200000F0200000F0200000F0200000E0600001E0600001E0700001C0700003 C0780007807C000700E6001E00E3C07C00C1FFF000803FC000>31 43 125 169 33 I[<1FFFFFFFFE3FFFFFFFFE3F003F007E38003E001E30003E000C20003E000460003E00 0440003E000440007C000440007C000480007C000480007C000480007C000480007C00040000F8 00000000F800000000F800000000F800000000F800000000F800000001F000000001F000000001 F000000001F000000001F000000001F000000003E000000003E000000003E000000003E0000000 03E000000003E000000007C000000007C000000007C000000007C000000007C000000007C00000 000FC000000FFFFFC0001FFFFFC000>39 41 121 168 44 I[<7FFFC00FFF80FFFFC01FFF8003 F00001F80003E00000E00003E00000400003E00000400003E00000400003E00000400007C00000 800007C00000800007C00000800007C00000800007C00000800007C0000080000F80000100000F 80000100000F80000100000F80000100000F80000100000F80000100001F00000200001F000002 00001F00000200001F00000200001F00000200001F00000200003E00000400003E00000400003E 00000400003E00000400003E00000800003E00000800003E00001800001E00001000001E000020 00000F00006000000F0000C0000007800180000003C00700000001F01C000000007FF800000000 1FC0000000>41 42 120 168 46 I[42 42 120 168 46 I E /Fi 3 21 438 432 dfs[35 3 123 143 47 0 D[<007C0001FF0007FFC00FFFE01FFFF03FFFF83FFFF87FFFFC7FFFFCFFFFFE FFFFFEFFFFFEFFFFFEFFFFFEFFFFFEFFFFFE7FFFFC7FFFFC3FFFF83FFFF81FFFF00FFFE007FFC0 01FF00007C00>23 25 125 154 30 15 D[<000000006000000001E000000007E00000001F8000 00007E00000001F800000007E00000001F800000007E00000001F800000007E00000001F800000 007E00000001F800000007E00000001F800000007E00000000F800000000F8000000007E000000 001F8000000007E000000001F8000000007E000000001F8000000007E000000001F8000000007E 000000001F8000000007E000000001F8000000007E000000001F8000000007C000000001E00000 0000E0000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000007FFFFFFFC0FFFFFFFFE0FFFFFFFFE0>35 48 123 165 47 20 D E /Fj 35 123 576 432 dfs[<1C003E007F00FF80FF80FF807F003E001C00>9 9 123 136 25 46 D[<000E00001E00007E0007FE00FFFE00FFFE00F8FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE 007FFFFE7FFFFE7FFFFE>23 39 123 166 45 49 D[<00FF800003FFF0000FFFFC001F03FE0038 00FF007C007F80FE003FC0FF003FC0FF003FE0FF001FE0FF001FE07E001FE03C003FE000003FE0 00003FC000003FC000007F8000007F000000FE000000FC000001F8000003F0000003E000000780 00000F0000001E0000003C00E0007000E000E000E001C001C0038001C0070001C00FFFFFC01FFF FFC03FFFFFC07FFFFFC0FFFFFF80FFFFFF80FFFFFF80>27 39 125 166 45 I[<007F800003FFF00007FFFC000F81FE001F00FF003F80FF003F807F803F807F803F807F80 1F807F800F007F800000FF000000FF000000FE000001FC000001F8000007F00000FFC00000FFF0 000001FC0000007E0000007F0000007F8000003FC000003FC000003FE000003FE03C003FE07E00 3FE0FF003FE0FF003FE0FF003FC0FF007FC07E007F807C007F003F01FE001FFFFC0007FFF00000 FF8000>27 39 125 166 45 I[<00000E0000001E0000003E0000007E000000FE000000FE0000 01FE000003FE0000077E00000E7E00000E7E00001C7E0000387E0000707E0000E07E0000E07E00 01C07E0003807E0007007E000E007E000E007E001C007E0038007E0070007E00E0007E00FFFFFF F8FFFFFFF8FFFFFFF80000FE000000FE000000FE000000FE000000FE000000FE000000FE000000 FE00007FFFF8007FFFF8007FFFF8>29 39 126 166 45 I[<0C0003000F803F000FFFFE000FFF FC000FFFF8000FFFF0000FFFE0000FFFC0000FFE00000E0000000E0000000E0000000E0000000E 0000000E0000000E7FC0000FFFF8000F80FC000E003E000C003F0000001F8000001FC000001FC0 00001FE000001FE018001FE07C001FE0FE001FE0FE001FE0FE001FE0FE001FC0FC001FC078003F 8078003F803C007F001F01FE000FFFF80003FFF00000FF8000>27 39 125 166 45 I[<0007F000003FFC0000FFFE0001FC0F0003F01F8007E03F800FC03F801FC03F801F80 3F803F801F003F8000007F0000007F0000007F000000FF000000FF0FC000FF3FF800FF707C00FF C03E00FFC03F00FF801F80FF801FC0FF001FC0FF001FE0FF001FE0FF001FE07F001FE07F001FE0 7F001FE07F001FE03F001FE03F001FC01F801FC01F803F800FC03F0007E07E0003FFFC0000FFF0 00003FC000>27 39 125 166 45 I[<380000003E0000003FFFFFF03FFFFFF03FFFFFF07FFFFF E07FFFFFC07FFFFF807FFFFF0070000E0070000E0070001C00E0003800E0007000E000E0000000 E0000001C000000380000007800000078000000F0000000F0000001F0000001F0000003F000000 3E0000003E0000007E0000007E0000007E0000007E000000FE000000FE000000FE000000FE0000 00FE000000FE000000FE000000FE0000007C000000380000>28 41 124 168 45 I[<00003FF001800003FFFE0380000FFFFF8780003FF007DF8000FF8001FF8001FE0000 7F8003FC00003F8007F000001F800FF000000F801FE0000007801FE0000007803FC0000007803F C0000003807FC0000003807F80000003807F8000000000FF8000000000FF8000000000FF800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF80000000007F 80000000007F80000000007FC0000003803FC0000003803FC0000003801FE0000003801FE00000 07000FF00000070007F000000E0003FC00001E0001FE00003C0000FF8000F800003FF007E00000 0FFFFFC0000003FFFF000000003FF80000>41 41 124 168 67 67 D[45 41 125 168 71 I[36 41 125 168 57 70 D[<00007FE003000003FFFC0700001FFFFF0F00003FF00FFF0000FF8001FF0001FE0000 FF0003F800003F0007F000003F000FF000001F001FE000000F001FE000000F003FC000000F003F C0000007007FC0000007007F80000007007F8000000000FF8000000000FF8000000000FF800000 0000FF8000000000FF8000000000FF8000000000FF8000000000FF8000000000FF8001FFFFF87F 8001FFFFF87F8001FFFFF87FC00000FF003FC00000FF003FC00000FF001FE00000FF001FE00000 FF000FF00000FF0007F00000FF0003F80000FF0001FE0000FF0000FF8001FF00003FF007BF0000 1FFFFF1F000003FFFE0F0000007FF00300>45 41 124 168 72 I[22 41 126 168 35 73 D[<0000FFE000000007FFFC0000003FC07F8000007F001FC00001FC0007F0 0003F80003F80007F00001FC000FF00001FE001FE00000FF001FE00000FF003FC000007F803FC0 00007F807FC000007FC07F8000003FC07F8000003FC07F8000003FC0FF8000003FE0FF8000003F E0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF8000003FE0FF80 00003FE0FF8000003FE07F8000003FC07FC000007FC07FC000007FC03FC000007F803FC000007F 801FE00000FF001FE00000FF000FF00001FE0007F00001FC0003F80003F80001FC0007F00000FF 001FE000003FC07F8000000FFFFE00000000FFE00000>43 41 124 168 69 79 D[<007F806003FFF0E007FFF9E00F807FE01F001FE03E0007E07C0003E07C0001E0FC00 01E0FC0001E0FC0000E0FE0000E0FE0000E0FF000000FFC000007FFE00007FFFE0003FFFFC001F FFFE000FFFFF8007FFFFC003FFFFE000FFFFE00007FFF000007FF000000FF8000007F8000003F8 600001F8E00001F8E00001F8E00001F8F00001F0F00001F0F80003F0FC0003E0FF0007C0FFE01F 80F3FFFF00E0FFFE00C01FF000>29 41 124 168 51 83 D[<01FF800007FFF0000F81F8001FC0 7E001FC07E001FC03F000F803F8007003F8000003F8000003F8000003F80000FFF8000FFFF8007 FC3F800FE03F803F803F803F003F807F003F80FE003F80FE003F80FE003F80FE003F807E007F80 7F00DF803F839FFC0FFF0FFC01FC03FC>30 27 126 154 44 97 D[33 42 126 169 51 I[<001FF80000FFFE0003F01F0007E03F800FC03F801F803F80 3F801F007F800E007F0000007F000000FF000000FF000000FF000000FF000000FF000000FF0000 00FF0000007F0000007F0000007F8000003F8001C01F8001C00FC0038007E0070003F01E0000FF FC00001FE000>26 27 126 154 41 I[<00003FF80000003FF80000003FF800000003F8000000 03F800000003F800000003F800000003F800000003F800000003F800000003F800000003F80000 0003F800000003F800000003F800001FE3F80000FFFBF80003F03FF80007E00FF8000FC007F800 1F8003F8003F8003F8007F0003F8007F0003F8007F0003F800FF0003F800FF0003F800FF0003F8 00FF0003F800FF0003F800FF0003F800FF0003F8007F0003F8007F0003F8007F0003F8003F8003 F8001F8003F8000F8007F80007C00FF80003F03BFF8000FFF3FF80003FC3FF80>33 42 126 169 51 I[<003FE00001FFF80003F07E0007C01F000F801F801F800F803F800FC07F00 0FC07F0007C07F0007E0FF0007E0FF0007E0FFFFFFE0FFFFFFE0FF000000FF000000FF0000007F 0000007F0000007F0000003F8000E01F8000E00FC001C007E0038003F81F0000FFFE00001FF000 >27 27 126 154 43 I[<0007F0003FFC00FE3E01F87F03F87F03F07F07F07F07F03E07F00007 F00007F00007F00007F00007F00007F000FFFFC0FFFFC0FFFFC007F00007F00007F00007F00007 F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007 F00007F00007F00007F0007FFF807FFF807FFF80>24 42 126 169 28 I[33 42 125 169 51 104 D[<07000FC01FE03FE03FE03FE01FE00FC0070000 00000000000000000000000000FFE0FFE0FFE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0 0FE00FE00FE00FE00FE00FE00FE00FE00FE00FE00FE0FFFEFFFEFFFE>15 43 125 170 27 I[15 42 125 169 27 108 D[53 27 125 154 80 I[33 27 125 154 51 I[<003FE00001FFFC0003F07E000FC01F801F800FC03F800FE03F0007E07F0007F07F0007F07F 0007F0FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F8FF0007F87F0007F0 7F0007F03F800FE03F800FE01F800FC00FC01F8007F07F0001FFFC00003FE000>29 27 126 154 45 I[33 39 126 154 51 I[24 27 126 154 37 114 D[<03FE300FFFF01E03F03800F0700070F00070F00070F80070FC0000FFE0007FFE007F FF803FFFE01FFFF007FFF800FFF80003FC0000FC60007CE0003CF0003CF00038F80038FC0070FF 01E0F7FFC0C1FF00>22 27 126 154 36 I[<00700000700000700000700000F00000F00000F0 0001F00003F00003F00007F0001FFFF0FFFFF0FFFFF007F00007F00007F00007F00007F00007F0 0007F00007F00007F00007F00007F00007F00007F00007F03807F03807F03807F03807F03807F0 3803F03803F87001F86000FFC0001F80>21 38 127 165 36 I[33 27 125 154 51 I[33 27 127 154 48 I[33 39 127 154 48 121 D[<3FFFFF803FFFFF803F007F003C00FE0038 01FE007803FC007803F8007007F800700FF000700FE000001FC000003FC000007F8000007F0000 00FF000001FE038001FC038003F8038007F803800FF007800FE007801FE007003FC00F003F801F 007F007F00FFFFFF00FFFFFF00>25 27 126 154 41 I E /Fk 70 124 438 432 dfs[<0007F81F80003C067060007003E0F000E007C1F001C00FC1F003C00F80E00780 078040078007800007800780000780078000078007800007800780000780078000078007800007 800780000780078000FFFFFFFF00FFFFFFFF000780078000078007800007800780000780078000 078007800007800780000780078000078007800007800780000780078000078007800007800780 000780078000078007800007800780000780078000078007800007800780000780078000078007 80000780078000078007C000FFF87FFE00FFF87FFE00>36 42 127 169 35 11 D[<0007F800003C06000070010000E0070001C00F8003C00F8007800F80078007000780 000007800000078000000780000007800000078000000780000007800000FFFFFF80FFFFFF8007 800F80078007800780078007800780078007800780078007800780078007800780078007800780 078007800780078007800780078007800780078007800780078007800780078007800780078007 800780078007800780FFF87FFCFFF87FFC>30 42 127 169 33 I[<0007F980003C0780007007 8000E00F8001C00F8003C00F800780078007800780078007800780078007800780078007800780 0780078007800780078007800780FFFFFF80FFFFFF800780078007800780078007800780078007 800780078007800780078007800780078007800780078007800780078007800780078007800780 0780078007800780078007800780078007800780078007800780078007800780FFFCFFFCFFFCFF FC>30 42 127 169 33 I[<0003F803FC00001E061E030000700138008000E003F0038001C007 E007C003C007E007C0078007C007C0078003C00380078003C00000078003C00000078003C00000 078003C00000078003C00000078003C00000078003C00000078003C00000FFFFFFFFFFC0FFFFFF FFFFC0078003C007C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0 078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003 C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0078003C003C0 078003C003C0078003C003C0078003C003C0FFFC7FFE7FFEFFFC7FFE7FFE>47 42 127 169 51 I[<78FCFCFEFE7A020202020404040810102040>7 18 123 169 17 39 D[<0004000800100020004000C0018003000300060006000E000C001C001800 38003800380030007000700070007000F000F000E000E000E000E000E000E000E000E000E000E0 00E000F000F0007000700070007000300038003800380018001C000C000E000600060003000300 018000C000400020001000080004>14 61 123 172 23 I[<800040002000100008000C000600 030003000180018001C000C000E0006000700070007000300038003800380038003C003C001C00 1C001C001C001C001C001C001C001C001C001C003C003C00380038003800380030007000700070 006000E000C001C0018001800300030006000C0008001000200040008000>14 61 125 172 23 I[<000038000000003800000000380000000038000000003800000000380000 000038000000003800000000380000000038000000003800000000380000000038000000003800 0000003800000000380000000038000000003800000000380000FFFFFFFFFEFFFFFFFFFEFFFFFF FFFE00003800000000380000000038000000003800000000380000000038000000003800000000 380000000038000000003800000000380000000038000000003800000000380000000038000000 00380000000038000000003800000000380000>39 41 125 162 47 43 D[<78FCFCFEFE7A020202020404040810102040>7 18 123 133 17 I[15 3 127 142 20 I[<78FCFCFCFC78>6 6 123 133 17 I[<00000600000E00000E00001C00001C 00001C0000380000380000380000700000700000E00000E00000E00001C00001C00001C0000380 000380000380000700000700000700000E00000E00000E00001C00001C00001C00003800003800 00700000700000700000E00000E00000E00001C00001C00001C000038000038000038000070000 0700000700000E00000E00000E00001C00001C0000380000380000380000700000700000700000 E00000E00000C00000>23 60 125 172 30 I[<007F000001C1C0000780F0000F0078000E0038 001C001C003C001E003C001E003C001E0078000F0078000F0078000F0078000F00F8000F80F800 0F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8000F80F8 000F80F8000F80F8000F80F8000F8078000F0078000F0078000F0078000F003C001E003C001E00 3C001E001C001C000E0038000F0078000780F00001C1C000007F0000>25 41 126 167 30 I[<00100000700001F0000FF000FEF000F0F00000F00000F00000F00000F000 00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F000 00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F000 00F00001F8007FFFE07FFFE0>19 40 123 167 30 I[<00FE0007FF800E07E01803F02001F820 00F840007C40007CF8007EFC007EFC003EFC003EFC003E78007E00007E00007C00007C0000F800 00F80001F00001E00003C0000780000700000E00001C0000380000700000600000C00001800203 00020600040C000418000410000C3FFFFC7FFFF8FFFFF8FFFFF8>23 40 125 167 30 I[<007F000003FFC0000701F0000C00F80010007C001C007C003E007E003E003E00 3E003E001E003E000C007E0000007C0000007C00000078000000F0000000E0000001C000000700 0000FF00000001E0000000F0000000780000003C0000003E0000001F0000001F0000001F800000 1F8030001F8078001F80FC001F80FC001F80FC001F00F8001F0040003F0040003E0030007C0018 00F8000F01F00003FFC000007F0000>25 41 126 167 30 I[<00006000000060000000E00000 01E0000001E0000003E0000003E0000005E0000009E0000009E0000011E0000021E0000021E000 0041E0000081E0000081E0000101E0000201E0000201E0000401E0000801E0000801E0001001E0 003001E0002001E0004001E000C001E000FFFFFF80FFFFFF800001E0000001E0000001E0000001 E0000001E0000001E0000001E0000001E0000003F000007FFF80007FFF80>25 40 126 167 30 I[<1800181F00F01FFFE01FFFC01FFF801FFF0011F800100000100000100000 100000100000100000100000100000107E001183801600C01800E010007000007800003C00003C 00003C00003E00003E00003E70003EF8003EF8003EF8003EF8003C80003C40007C400078200078 3000F01801E00E07C007FF0001FC00>23 41 125 167 30 I[<000FE000003FF80000F81C0001 E00C0003801E0007803E000F003E000E001C001E0000001C0000003C0000003C0000007C000000 7800000078000000F83F0000F840E000F9807000F9003800FA001C00FC001E00FC001E00FC000F 00F8000F00F8000F80F8000F80F8000F80F8000F8078000F8078000F8078000F807C000F803C00 0F003C000F001C001E001E001E000E003C000700780003C0F00001FFC000007F0000>25 41 126 167 30 I[<20000000380000003FFFFF803FFFFF803FFFFF007FFFFF00600002004000 040040000400400008008000100080002000000020000000400000008000000080000001000000 030000000200000006000000060000000C0000000C0000001C0000001C0000001C000000380000 00380000003800000078000000780000007800000078000000F8000000F8000000F8000000F800 0000F8000000F8000000F8000000F8000000700000>25 42 125 168 30 I[<007F000001FFC0000381F000060078000C003C001C001C0018000E0038000E0038000E0038 000E003C000E003C000E003E001C001F8018001FC038000FF0600007F8C00003FF800001FF0000 007FC00000FFE000030FF8000603FC001C01FE0038007E0030003F0070000F0070000780E00007 80E0000380E0000380E0000380E0000380F0000300700007007800060038000C001E0038000F80 F00003FFE000007F0000>25 41 126 167 30 I[<007F000001FFC00007C1E0000F0070001E00 38001C003C003C001C0078001E0078001E00F8000F00F8000F00F8000F00F8000F00F8000F80F8 000F80F8000F80F8000F8078000F8078001F803C001F803C001F801C002F800E004F800700CF80 03810F80007E0F8000000F0000000F0000000F0000001E0000001E0000001E0000003C001C003C 003E0078003E0070003C00E0001801C0001C0780000FFE000003F80000>25 41 126 167 30 I[<78FCFCFCFC78000000000000000000000000000078FCFCFCFC78>6 26 123 153 17 I[<78FCFCFCFC78000000000000000000000000000070F8FCFCFC7C04040404 0808081010202040>6 38 123 153 17 I[39 15 125 149 47 61 D[<000018000000001800000000 18000000003C000000003C000000003C000000007E000000007E00000000FF000000009F000000 009F000000011F800000010F800000010F8000000207C000000207C000000207C000000403E000 000403E000000403E000000801F000000801F000001801F800001000F800001000F800002000FC 000020007C00003FFFFC00007FFFFE000040003E000040003E000080001F000080001F00008000 1F000100000F800100000F800100000F8002000007C007000007C01F80000FE0FFF000FFFFFFF0 00FFFF>40 42 126 169 46 65 D[36 41 126 168 43 I[<0000FF00100007FFE030001FC07830003E000C7000F80006F001F00003F003E00001 F007C00000F00F800000700F800000701F000000303F000000303E000000303E000000107E0000 00107E000000107C00000000FC00000000FC00000000FC00000000FC00000000FC00000000FC00 000000FC00000000FC00000000FC000000007C000000007E000000007E000000103E000000103E 000000103F000000101F000000200F800000200F8000006007C000004003E000008001F0000180 00F8000300003E000E00001FC038000007FFE0000000FF8000>36 43 125 169 44 I[36 41 126 168 42 69 D[34 41 126 168 40 I[<0000FF00100007FFE030001FC07830003E000C7000F80006F001F00003F0 03E00001F007C00000F00F800000700F800000701F000000303F000000303E000000303E000000 107E000000107E000000107C00000000FC00000000FC00000000FC00000000FC00000000FC0000 0000FC00000000FC00000000FC00000000FC0000FFFF7C0000FFFF7E000003F07E000001F03E00 0001F03E000001F03F000001F01F000001F00F800001F00F800001F007C00001F003E00001F001 F00002F000F80002F0003E000C70001FC038300007FFE0100000FF8000>40 43 125 169 48 I[40 41 126 168 46 I[17 41 126 168 22 I[50 41 126 168 56 77 D[40 41 126 168 46 I[<0001FF0000000F01E000003C00 78000078003C0000E0000E0001E0000F0003C000078007800003C00F800003E01F000001F01F00 0001F03E000000F83E000000F87E000000FC7E000000FC7C0000007C7C0000007CFC0000007EFC 0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007EFC0000007E 7C0000007C7E000000FC7E000000FC7E000000FC3E000000F83F000001F81F000001F01F000001 F00F800003E007800003C007C00007C003E0000F8000F0001E000078003C00003C007800000F01 E0000001FF0000>39 43 125 169 47 I[35 41 126 168 42 I[41 42 126 168 45 82 D[<00FE010003FF83000F81E3001E0037003C001F0038000F00780007007000 0700F0000300F0000300F0000300F0000100F8000100F8000100FC0000007C0000007F0000003F E000001FFF00000FFFE00007FFF80003FFFC00007FFE000007FF0000007F0000001F8000000F80 000007C0000007C0800003C0800003C0800003C0800003C0C00003C0C0000380C0000380E00007 80F0000700F8000E00EE001C00C3C07800C1FFF000803FC000>26 43 125 169 33 I[<7FFFFFFFF87FFFFFFFF87C007C00F870007C003860007C001840007C000840007C00 08C0007C000CC0007C000C80007C000480007C000480007C000480007C000480007C000400007C 000000007C000000007C000000007C000000007C000000007C000000007C000000007C00000000 7C000000007C000000007C000000007C000000007C000000007C000000007C000000007C000000 007C000000007C000000007C000000007C000000007C000000007C000000007C000000007C0000 0000FE000000FFFFFE0000FFFFFE00>38 41 126 168 44 I[58 42 127 168 62 87 D[9 60 122 172 17 91 D[9 60 127 172 17 93 D[<01FC00000E0780001001C0003C00E0003E 00F0003E0078001C00780008007800000078000000780000007800007FF80003E078000F807800 1F0078003E0078007C00780078007820F8007820F8007820F8007820F800F8207C00F8203C013C 401F063FC007F80F00>27 26 126 153 30 97 D[<07800000FF800000FF8000000F8000000780 000007800000078000000780000007800000078000000780000007800000078000000780000007 800000078000000783F000078C1C0007B0070007A0038007C003C0078001E0078001E0078000F0 078000F0078000F8078000F8078000F8078000F8078000F8078000F8078000F8078000F0078000 F0078001F0078001E0078001C007

FK4MK MV^JW$G9V0.#:M*:M,VNK@+4>KE%KXAJT>JV-J[ZPJU0*JMB%QRM M@HS-B+XBHV>%\CK>,:\HC/SZOM*OW,;."KBVKS:C A :D*\,BOF*OKZ<]>O. M>K\"L()>G5>S"JZTGNWZO\:O#6P8@;ZZ!;0>\(JUXJ_AS /KN^:O%6RA>> 1 M%:G%1'H)-A60('#A$_@%Q!\=<$DP""C!);%I6 1M0$V0!M!U&<&VFAC0=?9! M8D#\L0%UQ!J1$:RL=80OL49\K,R@5) Z@!Q)*K:3' JM/J.WF.\1HF9#-0G% M^A/M(FMH30HK_ 44NQ*TBTXL[?D*_15>[!)K-F"A*6I#<+7B#E*L-/JQS 5/ M+,*:A!0E&0<<"SPZD5?K@0#&@I#I28]YM8HB6>R[*3[B 'DL'0NG=9'.) TP M=XJGR"G=-Q:,?B1L/7&EF,6;(?F=%(T&[!*,Y"I<1!):( MEJIA& 80?O2C$AOJJ;)X10UQ4N 5.!YY&J*,CF=#+.MTU*,?;#\JRY:@)0>. MEX2D#JULU(I7F(N^+!M+S$*OQND\:1(*-LDI*AC)Z@1(0+=:R=X!]*2@M\KJ ML<'L&P'+^G@F(1)T3M2RW*S9P,?FLC->-RO(@K/]:&\#S,H?-8@VR\X>L^FL M'@O/+J+)K+&WS-H,OJ8:"LFBL//!8'B7AA&I+!N[/XZQ\R4JZG3HI:Z73X8>>E@VM-%J%LH8PI$5;2[J>!VUPV5YFM!.M=-EA.K2S M1&.ISK:JUFPT2D[>EQ>JV6!*AIF$Y+)02_ZF5*EX^5NREP#"ZW@LUI*6*=+G M4D*/\R7F1CM&L9ZH$[N#SK*T9[X7T8IOQ*,J:T.^EH^BG%AZ(J=5*M[WS$H! MT:P\@REJBMOE6%@-GAMG86]3E'AY)RV.!YZD#BKMC)G3!K4O[1D[.LJT-J1) MB#4>)U!MG%@H#J%4K1JJ5-XX6EP-<2]\M69A5.#,HK ZK-/PSP84X$:H-\>V MJ@M!T&D]!I<.IVH)13XX&6!14ERBL:VHFGG(JI;.9$D@UI:1,FTTVM-:HQ>M M. FKU7L[+6"I@LYPOF5C>X)ZLVX!47MDMH^P): XU8H=5:VE*,E&LWZ"'. G M'GH"K3ZYV.)X>H(1F=M>M;SM1?#;GK1D[1=+W-H,QJT^^_P]LSL*]+$&Q'V? M25(@.< +[.IJA[)NMVA!-W'@5;=""N6@BQ!\>PAI)YZFLM=)W=?'E(YD*?0' MWKH*;0AK2*<)E/.ML4!^)I/.))21W[8AT&-)NV/RDXSD?QL45); &_%HX*:0 M5>C]Z-\2?,9"]JF(YIZ](PBPX,:5DJJ%"^$"N&3DJYD3++C0W[M9G"J=:"AK MB..HH0VJ9C$VDG;=XR&)3CXX*^M\NXPD@O6$E1K-6@3/RD(0$K"73VB,2_#- MN-9M?8CW7:EY*ADP%9B.L^K3.6W*#P#$CTO>UKA#;C3;LM*P?213F9_2CM\M MP??D"KG0;,G:1X8$J.L3FN7""UON\X?5N@K^@Y&KG(*3S:79Z9WX$O-MF8OW MG;E3KFXPV%Z*6D:4^5O.ESDM!ON&CJ0^;14J8N*TV^>K2>A"E^"G%MI+WH^" M;E[)DV*;9.5=.::H21N(TIN2K=6+0I[YFJU 6N%ZJAAFS5;O2=(NHFX[N099B*> M+BVN"^PZL08H"IGKSI?.917*ZTJZ]6B8*4[JN*-C*IGK,KNC8S0ZT4*[0:UI M6U*&FK=N4#MYCKANI[%G9F@T(JR&)^?!'W*?9Z ]YHP[8Y3)7."X4B64$2/4 MAT!;=ULA2 $MZ\WP2'JMCIZQ4>3]IPV"G)HTR@'N9*NH+%8()PD.D=ZFNZ$> M%))>G(R:BL^Z;B2\?LLUOSJKPD[]%1IZ&\4@#2F^\ED38O"C %2+U[&LH+ M]#:]3^?FR/.&O!POR]L^@)9?;]'K%W*]OP*,:_.B-D_ RIOO=JCX9M5K!+B] M[,JAJO2B=2DOW?L2Y'L]9]5+!.B]Z4G9Z_-NO?ANW1O-H;PXKTZ0[X:X=R\* M( 0 OE)EXQOS*K[LRJ0&]4*^:D[C>P1 OE(G+KO[@9JYT:SJ>V6VH,CKIO+2JL>JO$@0DK*GZK\D$/"["&JU*?93@" M?A91PFHP98 (@!&* +4!!1@+EAI'@AW1(E(5!P&?)P"\!&^A-TBY9!8&I;; MY1O!CM".9*L^.?^.CO4O4-%!8I1]*V.+FMIN]=X;P3M>E@'PKQ!1]K_A[4=J M'CXX!K"#,SI&E 9PE#8Y;I7'1 1:ZKZGYT0%W)'2::F'%WD<)L!EPHXK+-B_ M(J:;V0/>CAAH -Q,>I/OI@W!.W*: 3"I]CK&P"=%POENUL!JSO084TFP0%D#.\%-8M3:04K!J4C^NY0:JL-* MK2F4AH\2\%**49ZBO&S[\UE2D7NI"?QJ H^D@0:*0DZZFRFG"0W:ESIPA]D# M!Z5[<'1I!%>@'&4AF5GRCN(E%7P%&\+'G"8<7=".83"2*0=$M7>M'@D"XWIG M\+V(9[AZ>-_T2RH6MELMA6FNMI*I9AMLZ9IQH8$2.0OSCG8F,@?8JJJ+RC":Q_[J7QJ""[&[VF4:50&L+ !2SDZ6B>T^4%3NI#.J=6@N&F+]K.20&9M]TP!D("Z'='A!*&@L+S@![4N#$Q5Q]L>+:>45+4""=]@;= MXBA(W%1\6[&\$&7!.IX,8^GFY<&X?"M"W2*&X& MJ_U+XN%'_!KF+4BX\*8DD: ;,G^TJWC?KE@4H(B!(NG)+H*#0&(*2]&)Q=YB M<+CD%7/EA77X#DJ,JS#"BA CO^UP2.@5_W[^H7:L7,2_OFHU"![KFL4<]+?, MS'4'L=*I'CL37_$SZ!Y;"O!QVUD?=\?L<)6@"OL9!]YKJ1"'A-AQI?@?<\5@ M1GCLUPW(X7%;2AXCR!?KX8>+/AK@*.?R6L:6CP;ZIR4"B6%Q@(PD6HF&17QL M(-_'%^L0T!2"@:#@&M -A@L50GX\\"[(JNLYF!V7R&#&B%PI\LCRP4Y\, [) M&J&$?!90R#;RA1PX\'4FH=VR(C*#Y_'TVBGZR.+@@\P=A\>.77T<'LN, S*+ M/.0UR= O6C 9L+\LHA@LU8K(\N"/W!P'R(.BG)@B%\COQ(&\)&?(OZ@0<'S4 MR**PDJF,](!4,I\()(?'K-YC,!_#R4IR]G>Q3@:7(F$@L'[(MBV?G"97R7_R MF @H#LH6LIQL*'_))8!I>)'4 73(:Y 1- %9'^3']I5]5!_:QQOL@T$ 9F Q M/,@ZP.CW I@!=H$"DB%,!"#"16 XB!^A\M;7,.@6J\7H%RU^#.#Q;K#^>8LH M0/TP!^#*:Q_7]]6A !*BU[BGCC/R0]4 ,K0/P#*.80>>!;[RP2A 3 6SA80R M+%<:$F.AG"UOA,)R/2$P@,K)\JY\WI4>K_)K(/LARZ(RUV>6T!E<0IR<63PI MV/*GR"ENRQ0!<.$MUQ/5,I'B+0;,$V.7,"Y[BQ:"V5?UV8+2B']'@[3+?:,' M0@LNS%??BW O@HPFH8X8!L@31<$5@#J@!5MQJ_PJQ\JOP:P,?> 3MS*Z'"_O MRG,)^_$K]R3!\H2R!Q3+K\&QK#+KRF' LMPLPQO3,;3,,VLJU?*]? M$\RK<-.<,)?*#'/4C-Y1,!"S5R Q*\RF,J- A%S,3#+>=^YB!FZ#7>"RF<2? M6M"@5N@@AZ /PH,$:OE!"9=-L8XE@2,;\+*-ZL?;6.1!?V'Q>MPE2!L1&"FSG4G3@P[;\AVY]X),'JG ./SAQB( 77 &: BC _:24:@-<(AM0*A8KS?:>@,%.RK &,DFSM'QVM![_P[(Z7"\]Z2-4R1F,CR&-NV MMW;*6/+,,L^;(OG,^"Z_YRK^R>6.SJ6B8?M@1IGD;N+1V&K&#N3@W! 4SN!S MP#MWFH10@$<7_UF#(X(Q6!,TR.)@3> XGP4U09^L'DQU]T-H@2UPOQ*J@G & M=G\B -N("SH:"LH:\6N$QPC?L@P=<3:!E1SRA[ M2-#WMLRE!6O&$( $) &@<@H0MXD!1F\*D%BB""A )*A&VP5L]''C9?R\H6%S M ^9M*@!)W,*4"; T5Y!0J<#I "2&EM@!*@!*( 9C49# M 8:T6\!&+P%4 "6]R9F\9<8?'4AST71 "H &H ")=,C'2-/1:0$D+4FGT4() M&WWT!I9PM!Q=)AQZ*( =;0;@T !(1K(TP' 2G +2T$@!B.00J &='29@1'!WQ3 PI@(C"1I:GE 4NU1 M,]6 0TC]5&_3)35JU0U'1U6 Q!*=8#*&S0,#TP*< :$U7B!(1W-L=$Z M-5;M44L*' ືU1##KG8&.-:HS5H=5V/4X/1M9UC])L!2[/1T(?&5]"Q 94T+7T26!NHB!825V?71]S[FD\' < % M&J >"--OQIHA(;X!<;17MV8 V()C"N &W*N0 5@ 6P\!]9T.0E_3<6Q&!8%@ M[]>@X&E0:1('FS47D/)6=5Q X2A,)P&\P1WM%ZX9SL9G_5C;V#, ?CU*=P=D MX5F08TO4>@A*<,CU!1E!WA:H"0N9W$=YQG+7C_2-#7,YTYET-#U)4]/6-#:= M FC3W+3*.UG3U;]U518@U-$2M94=K_%IV$X-XBADUU-0$&$$C-(+]AD"&XS3 MI1IH$*F]@*HW5@ M4&:PT;^U!.?@0!E_-"2]'S#2!+75FV:LL)(U'!T$]!-^05S]6$,!/$U)]89]KV* M::8 J1N]]FN[VP*WM:UO']S"M@)0<-_; [?!77++VQZWR9URH]S%-FJC-------------------------------Cut-Here-------------------------------< ---------------------------------------------------------------------- / e||) Lyndon J Clarke Edinburgh Parallel Computing Centre e||) \ \ c||c Tel: 031 650 5021 Email: lyndon@epcc.edinburgh.ac.uk c||c / ---------------------------------------------------------------------- From owner-mpi-comm@CS.UTK.EDU Thu Feb 25 12:19:23 1993 Received: from CS.UTK.EDU by surfer.EPM.ORNL.GOV (5.61/1.34) id AA26006; Thu, 25 Feb 93 12:19:23 -0500 Received: from localhost by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06312; Thu, 25 Feb 93 12:17:23 -0500 X-Resent-To: mpi-comm@CS.UTK.EDU ; Thu, 25 Feb 1993 12:17:21 EST Errors-To: owner-mpi-comm@CS.UTK.EDU Received: from daedalus.epcc.ed.ac.uk by CS.UTK.EDU with SMTP (5.61++/2.8s-UTK) id AA06274; Thu, 25 Feb 93 12:16:42 -0500 From: lyndon@epcc.ed.ac.uk Date: Thu, 25 Feb 93 17:16:34 GMT Message-Id: <2153.9302251716@subnode.epcc.ed.ac.uk> To: mpi-comm@cs.utk.edu Subject: background information Dear MPI Colleagues I found the February meeting both enjoyable and stimulating. It seems to me that the question of process groups and communication contexts remains fairly open, and the committee has taken steps to move forward. I am sending the interface specification for the system which we have implemented in Edinburgh, called CHIMP. It doesn't contain heterogeneous support and the such, but it has a very flexible approach to process groups/contexts. We've used this extensively for both SPMD and MIMD application and library development. It's coming in three parts, first below, and it's a uuencoded compressed PostScript file. >-------------------------------Cut-Here-------------------------------< begin 644 interface.ps.Z M'YV0)4) F=(B")DW8LJTD.$"AH(2)8;(*1.&SALY.D"0L9,&SAP0-5S0R%$# MQ) WBP*9/QX!@7&],XE4B13IHW;HA4I KB2ADR((R4$0,3!@@8.73(B$LC MALT<,YQ""7.FS)R,-/)"W-OWB1PR93"""#)G3!DW9)P*>5,'C6 M;8T<-D#@H"'#*9$W8^JT>4S'"%@Z?T'L95/Q\IL64MZT">-&-M#:;FY/9H-V M-O#;2>@ ]>FT".23;5:[@?V0B)4D Z'OALSD,MFL'C6BQLJQ^O6!1^O(<9P1 M!)4R6$!0I@.GSLN8>!LRC %#!UP9.$A6QAF702$':E.404=&=)2!APMP'/A0 M" J\\!X61/CTD@PUN-73&"\A9H9[\&4((@@)$=C;"T[L(6(?(H+P@A![B'$9 M6B\Z(>,4>S@X!AI]Z/@"%GM, 8(3?2@@A(Q42+$''7+P-@=M#08I8QI3?F0& M4',4):0=@:51&HZZ,2 (C0%55)QSNB7I M'(@J"I8"8(I95*&-@I!F4$T*.N*HI5;:6Z23FMI;JF."T *KF,($ D]HO>J> M% KL!F4:>("0FAP33:?L2LV240<<&ET+ @T@]/72@96AQ:V/:(!07XC:6HMM M2=Z" "YD((% K@+G7?2RY[W9U &5& M&6?M,8=R=-0QQXJ*.6*=XF8RN34=%'@"#,,.18+\F\[)X*&#&='(TN_-K/;_1+,]+ZCCQ M2IC)*$8878(0'U-1YB&C$%')'%Y\,]91-1E#H,'4'@I$SI00CA.7F+[.^=E3 MWTFC_1B.UPILQAMOV$O'Z=/UD1/A81A>LNQVR T"&V_P9#/1]-HG8]-+;_%R MU +/,:)=0KX-Y=UKHQWUU&%4/;P9<[;Z@O&"RZA\W%M@[Y!;V&MZ!M!"/T\U M&R\7L0<<*(& M;H:E6&&=ZR_U.7JK[W\8PMWI$$&'6C8P_[(4)$P* !^4\$, M .,UASJPQ5OZ0T,+T% &EKA$@!(DH'*RA:T$GF&!W&K@ Q44P1;@ 2EFV$,, M -2G#!:0@[=[S ?+=381=DM!G')@"?. 0@RV0(-A@*$'%Y@3&[9KA3=0@ A+ M2 8\^!"(0I3A NUB1!(J;']IV$U?GOA"^-&!;&6HT-XN\T$PNB]@P&O7"X ' M/+L0ZTBR>IG_GO8".=R!CNNAXQFP9;0*C8&/:'O!$>CHO4!:3@Z8>UQB)!8& M.Q3E;'8Y$!N"8B3XO>!O@6M73F[DH!O200%&FE'BPA">=AE)C0-\87S8AZTU M0NEI VR66_9W0C.,:'\\M*6S)$A!"W[2AK1$X2[YY[\%OA&7PKS?3WYTEC)P MQ#'#[-__RK6_7K;D)1^C&= 4@"=?N< NP=0E,"68RQ%]JXA.'=P M0XV6BH:F+O0%4T@"IQKI)1"TB26748">VO &1UI$1E9P546HY2SU1(L.^*J6 MMNS2+O@9BUPVJZNVC$4L!;C@)K>C@\"B8(4^"$P*A1U4&>14%-SI+D:L-!=* M]%6&HF;5GA.[2%'F4-4[P,&=(#CJ8314H0*OD6BH^_(&!AX+U M NLD08O+#1V'0@,[P^G(3B#@4 T4L(?0&8F*!1T?_-QR-DXVJUYCL$-O0G8Z MI]F@!C? 00Q<<(/06(]718G/??.[W_X>D+IH"%.O7"8D.$BLL_J2PMJTJ:=N M!DU(0CC-AFP OO0&4@YJZF0?ZQABPQE-1G80&(BG8CC&K;B3\2FLPJRPAYZ P4G0KG(^AC(Z.H@21]UPSVXY0:>O$R#Y& 'H' 0 M!1F: QQHDX<4E"$.\L+#CVQ& P5X2\J)J3(;4. $^%#!RV"^B&$CNR7&RFJQ M73)LG?.L6,;*"S)[V,,9YL#58WGS3M]40).$1<6?3,5=(6[6BPVWYZ)(N")[ MR/ ; M7.OLRSGF>82&;S:5A!$]J1AFY!HE5MET6_6-(EYI2CR22L2NMD;9G6 M,#S=^>ENB1J?[[RSGQ5&6,H-NM!0DE)]V^ NRL +K^]""UZC$B4W&.\BS":K M65/W:IO)@0W>.:M#)GW&\7T[W-R.=+F]#6Y-G75^D_3U/37[LC#L0=L*>L/P M%KO!$9_/1=IZ 6+8L,'XN$4.^+;(RYI0)'/%C^!!?./YAMIP;$W\BGMH ;>: MD"2%D2'C9^.XP@*V$!"(7'L9M\O)7W &)YO\90%4N1SWD).5JV$/(7_9&O:P M\815Z XN1WA9\_TRJV%+XR"XP\O:L#Y?G4WI"I/JT7,"=1F]H>G=K/H+XM!T MNVA=#D4.;])?-B=L/?UE@L46M[3N1+<82>C;?ED><"YV208%6V$8ZAOFA-6. M01"$@J(HDND(10- MZ&MU3.'\(%._*"JPI#?5VDMXK$(0N!=L22#"&<7NW@I5P0UIB,/!8C][WGS\ M1>MK'RN%G>?Z H4-VGR]6T+33;N'N0S^= M>Q]M\1 7EN+52;G-U=2E'&4M\ MY'M!$Q2TDC%\!*\QVO'GW:" SOV8C#T2<[F\:,:!36= B1%H41!JY/:9E"WK M%4/!!P>&)3Q]EF>4$S87M5;,DEIR(C,P4 ,P:,R 9$(#]AH":P(3 )!@(VL$*!!"8*< ,X$4A5 M14=VD#HG]@()YE.OV!C"(B2\,3ZY. 1,@!UXU#\)\A@@8&M'4T=HD ;)V!O, M*"-300=35F-"$AP-@HTR$@8TP(TO,!5G !38^#) T"E341E1T18C!@2[Z"E[ M<(ZSIHX5Z"BZZ(LNHS! ":S%H_H6 ;TV'THAH_FF&"JHHHBR#C$:(S(8XX M)C!@LEP+"06_:(YC &X.]@(3*7:!-S)594OWHXJIDX\5LH\H%)(O<(N"8X[! M.!4"$XPKJ8]VE 8J-%WV.)/+-9/16# @AC R^8SDQ1\"Z8QID)//N),? T?Z M.$E.%(YLT"SY.#*3-'=.V6+FJ!Y-B94Q69+J095=&9,RHD7_R),'@X)_AA;[ M(C.'2 =5X&"UU7D:@Q:[U5MO"2(5T@9HT2[Z!S4:LI=?9E1=,EHG0HEX&4.G MUQ:I5YB[@0=#5"Y;%)?)0CI*=I@RX!9OE%2S@X(+=8*%28GAJ"#7J"_:&##Z M(H[DJ #Z\F-/@@?L$S@UTP9T,(1;@HIQI1')9FW+9A3#HAQN\ *?E3V:XED5 M04T8^5D E"Q,L7.W>6T(!R?+F9L)IV^&]9S8YIS5QIS,UFY$9UC,EQ-V%U#7 MF9O=>8-!09T(QXJ^XIW6QY/*5IW4UI[G&1KJ.4E>:)[,)I_DZ7KVB9[S&13/ M$G>E=I%[5P;&"21]4* :R 8@50;"N4!^4AG6J!O).2S0$CO#,FI%P7+^A =1 M(UG8A+J<3).HSRNYP:.!%QF M0!LMMQ=2)@2T,0:O-2CZ$INM&3!P@ LE!X@*78D@=F.BR %EE3L*#CXZ52 !,X<'";-A;" MTDU.&B$UV"Y/ZC1TA4/@9T/@UZ=BQZ=ZZA#M$J@G!7ZQ$E-EBI$@@*:0JHUJ M^G%LNFYPZJ%R2J<4 S)ZFA>&ZJ>>M&Z"NBENT:@C(JD= 0*/NJJ4ZGOKHZ=_ M^A*#>JB>Y*;'$D=[<*48.2=?*GSM(P,WP*EV&J:R-J:.FJ:JBG> ]JMP0"_M MDP/$VB!W*J:/1JAG$ZK"DJB;4JKC0Z@YH:UNX2VXZJVYFJR0NJSNLVF5N@?. MJJG2VFR=:BZ?.JJ$.JOENJBF>JZIFJ:MBBVOVJQQVG_8(JS3ZJE.$ZXX1*B( M"JC=JJ_"J)NH2GR;-6MBD >@$:M.(Z6Z$UGJD2=]5E]82[,]FOTIB.C$T=$FHW8QB7]N'VV M\B@C&RS8YUSL!RR9TJVSIE:#UH_6A5U0.&L+!3X*$FJQHR\>D11S8B0#IE_\ M90,'YBA((B2R!5SO@0=&P!("$[8D, 5<1093,*^GUE5U^1*+V9B9R&-WR1=C M:6LQ$AP(9[2>TE;0PAKL(V6&YIO&0P)?&;:P(2-TJY66*X*2:P8D,)7F8CR7 MV[F?^Y2A*[:8.[F&L[FIZ[GO);H?X8*.^P:02[ICT&(50K=C )6M2P)S\+JN M>[J4&S?"RY0Y!(#!TF"3.P>KJ[K"VY7T,KF@:T.MHB-T^[O"R[S'>JV3N[O9 M6RW!0K?&:T.ZFP?1Z[G:^U*3^Y4VM&C7D[/N!(Z) 3@$&K\TBQ)D8[\-5H,B MH@! H):3UY;Z8A48N3[2:[KBRP;-Z[F:N[YR(#LD.!V/.QW\29X+VJ#EPBV9 MR2W]Z2I#=U;,9[+Q9B3:>58DC&ZZ6<*I(Z!=4J!],JD,6J >''=",CIO&[.A+?SZK/YQ"8\IHBB>)J;.(GKMR.A.(H\O%F%IB/'AFJ(NQMR,K6R M*\&T2\%-XK^@N(A 48JS27"QZVP9@QIDO%QD\* 4K,9T@#&R):1P%%L*0L2C MN <3J0!V\9?!*<-C]<&ILXH'I\(V$Y'GYFY__!'@%\@HS,+U>YP7V1$+58LJ MN6B[" +\N+U% 9,'\L/("#%2A9-:.;XGU2IR<)2>3#FE7)1?.;VCC"F&I2X4 MB\FW8[K@![J,*@6&EI'%H//,OF2[T6^+)3:[RUK,#K-K$8:!+%2)$Y MH<0:/QC@JH3N,@P=1@A:,L[>U&540,S,*/03J'0/LO=[M_=[N'=_T#=\.,=_VC=_N MC=[H/:R?/=[G+5W=#0-%\-TW4 0Y< ,Q, ,^0 ,X$%0\8 02/N%S.N$4+ET8 MGN%BE>$_N$@;N$27N$B/J$#+ETMSN$][MT[CN%!_N,\/J<8;N1$CN0;KN1"?N1)/N31/ MON11#N4^+N5!+N7>K> V< -$@!=X[.+8'51" 3;^1 .B+0,XKMV??0-#<. ' M/@/J7> %'N P( 2R.!HQ8 ,-7F9$P ,W,.'29>$D/N(@WNB.#@.* M#ND7?N@2[N/\X><(7B9H^^)V4>C2)>4QH.=)+NH^+@09'@.FCN$S( 14ONJM MSNJJ_MU$[MZS/@2U?MK<3>L"KNL$WNN?[NN2/N02ON.A7NGKS=T^;NO#>NS+ M?NBA;NL2'NKJ[>Q&8.MX/ ,Q<>@R$>@O7A*%3NE(/N$[3NFB3NOEKNPF;NN? M?>SK3NOMCN[,[N-&L*+E/NSR;@3$/@2AGNSAG>[A;>NT'O /_#'+O#L3O#= M3>XD3N\"+N&V[MTRD.U&L.V"G@3;'>JRSMT\$^Z,ON]&,*ZD?MIX$;_ ?%?7Q_>NZ7O6]?O7'GO4/C_5>7_!13_ A MS^P*X.\?K_'&+NY+C^0+'A.@7?0^8/'D#NT3KNQU#_9X+_5#(%8&G^0_O]ZZ M[MZ![_#W_O#%;OCZCOB!__=:W_A[;_5=O_73KO86/OGVWO8)/O%P;_&5G_1* MW_F._^MX'/FA[_>GGOC)/OC57OBL#_B*C_BGK^&\OO7\;OHQ?_:)OOJY3^68 MK^V;O]UT;MI+?_OE7N#S?O+0?O)SVNQ\/O/++O,P_^LOW_+A??:3;^OJ/0-G M+^I)U/S,7_,5COP[+^3[GOND?NC+'MY%L.PK-/04'_='_^R2+@1*?^AXO/JT MSOXY'^_\G_].__] C^DAN@!X[F:>_UMO8D7_&<#]=P ;( /I_FXW:<;]A)N!55!-3>NJ-Y(/ #BD D-P*5GPD,@20P!9[ $DCSQ K*TX P M\/-ENS#W^PA?[OMY-I#VZ<#9QP/_WNB+?;:O]@G!'$ R&-R& QV#=JA.# M9] ,]@PSF .$P,AK@V_0#8(\.#@'C8#-*P)WT.;1/#WXV5S@S=N#?S +7L%! M* BCX.\[X29L+UM@G=6R=D M;_IO_24[4;C>OERR,X7K[0;H/U5XVMS;#-!_KS#9Q4+ U^R,0/F3<#F ^^FY M^$-[HD[[*;T :/C*1((C>ACPZ.$^M:?Q&-ZQ.WONSM^- M/;V7]\2>V#-[)FX %CMBA_3I_N"W3]$BCZN%_J^=%@0 M"^*(8XEZKB#2P\$8]NHA C2,A!$Q%L;$V/>R'Y)SC'PN_241H[@(#2+;JP%N M#R[40&6G!0DA9[2( -#=^;@_N.X4 !\DC>GM#=K!.A@'VQT<]'&OT;O%QIC@ M!F%C;92-MS$F++A6^ 53':13>4ORU&_*S?J,ISV\XZP3L"%QUSW\Q3Q?UH'F_=KOMYKD[5C$-\ED1WW\4!DVQ-6F8[;);CD*.64HA5DD'$101(X$/GIC)U8 M>8@M\N&)NA+IZWYBB.R)2/)(*DDBJ>>,7)#\'@^Q Y*Z&V?X8.+;2X<-TN'M MPY#7XKK;)@1VZA$H"KOOX?P2'H@T?N[Q['U'_H!$PIRF@W_"D"YZ1YF7_IH? MD=-P,^_?I3\)Q_X&()?DACQQYAG)0=G\"*6@]'Y0LOF%0WU7[WJ>BN1^<:Z] MP<2]:/34W^X[?%QR_27#9H"O%(;';BL81 M3@:_AX?^6"2]BV^>,"I&0VF9#:>EX,.!N*_8)3DA,.2B/S(Y(["T?NV&2*"W.KTNCEP)E9%_G>KAR( MP;)7XDQ@R3._A\X4ECTST@G-"R?TSN'[XP\!HM!-O"'P CL@W;-W(2^_9#\\ MYQB'5=4TF)$0:UK-?KBAP"9A,/XKG,MD) 6[V< M.YA^\-7MXXBZ+U,6PVY8**]AM;R M9)%2:!Q)'Y4BWB MQ=PWY(:G=ZL+L*YTRLSCER8K(Y\DEGH.";9++#W*!+?P=/B.:3H\2HS=/UU( H< 5"P")')( M?KP65_2I+[+D]O6<"I(MI;P!:N *:)[UG WV@=1*">K<[61J1 MW0?T;T@NOZ2_RBCODB+;*Q.J<.*U(J.'"+.NB7Y+"=L)IX]=(^SK^O2"&K(R#\HJ>QW^%!$Y<+A5],L(.V MT8P2 3]W"=FH=UNCE="-_L8=]R.!8T:THN23A +10Z?][)[\^XWED5]"T>HW M1:G?/W1R!'*0W@!^V43/XA!5A/"Q N)1+,H7@>@&S(6,$.$=R/0W'/DCL&ND MC31$ <@<,$IYWX!4$D-1YJE2'1G]F"@I)7)"H#SB/D*JZAA?R?)SE-3HB<,J M>/?FGO-;?DL1V3DY?CGENJ.JTW)4;F!>.:C'[,S>W;-P3Q/_1;PKZD//',R( M##6@)*R0-K?=KF+M$WY[3KJ<3!&X/=,?5#R78O)W KNP*5W.J614=:J.[?V\ MGW=.P2FO>WA!( 80@1L0!&Y";FMP#J'043EZ"D]'HQ <X)3%LA056 ( MQ'7OM)W^SL!' C'<.?UY0W%5[M,AX$]I@+)TJ.ONQN$ M!FBI"2^,XV,;APJ M/7&8YS1@216IZ@ZDACO^P!4CGD>5>P"SVB%.6Z?O6&>O>W@<)@C$(IE0%SPJ M#3B?%^_O\'54ZM4=J5]. M:+43ATRSK][5O;I(EY\7+:S<;;#R3X='X=AD"'VK-Q7XV<]@YPH+G&#MJS[1 M)][/4,E.T:EGO9^9]=.MNYYZ+5^@^/N-JA#!M:+("B\?8G76W02DC99RN8O,$GD#F*EUOJW=]@2F5:!97$?I/,^"GS*N)U;"B5Q2( M0?U;I1.'8M-K-M05*%-SGWV=H#$@M;K5X]HJ0QZ'&7:5];O95V%CD8W%LDP6R3O;']KJM.:STW/4KD,\QB:C* M(_LH'V5=#:S_+6#*U_06WG) ]GNE ;*/5K]3.37;;'S-FJ^5=,=EV.*W+9^]@932,6#(-J$ WV M1DCK!7M&I66#UUMQG+^5YL MASV3&M+4*4-9ZT2A(:T=@V2VS.;:$+5K2:DR]+4G#]CVN_7>VJ7;70$EKT"F']JOJ\FWUV M!')70:M2B5VBS;,T5M@AO?_V\+XLBXV*7O2W-MS@6OUP'WQ5?B-0USG;I4VM75S$A(3RU["_B/=;CNMD@'*0; HSRK[9+#A/RWJ=\([8X M5ADJO1]9[^2I=OQTF[7<-DFT*6<1W?]$K8HVPRH\SY<,7>&LG+!--[ 2.%?H M75UX$'LAS)VT:+:07@:\V9N+;5GU]2JW;1K\\SN'\R/ M2-3G&5/BF%]!&Y=EE!4.[WI8.FHMQ^JU3'VO#_!Z52K7544KX76*6'6J\KCT MEP#KI%/UCD]UJ3I.(TR](ZCCNC]39]K,8%D0DZ^EJ[:^5QV: MU&^KW]Y;J 6!G'2B_LJ.*V73Z=8;K.O6P?;+&(@%X^V\M7?V;LAUU@1K@#TK M!;; ZO0"JT<%F8'5*00>AU]PG]Z$5:EGL6!#U, =.$&BX J\@C$P"^[ 'UB] MB940K.U(L _ "Q#NCZ*].SG@3"*XO)G#4R-R26:9!)G@U]-UPXI=5MS5&6-I M:A'(<_]T1BK-I A_=V63J\+9D@B,PWW'_K(?OWMWBC$,2[V2&0^?G!FULI>P M+A)='Z=R[6Z&S9*9-_&M.^QZ3CEKJ$RHSE#PF>!3"4"YVQ.VA9$U^"I?5HH@ MTZ.!A',*-]_Q2--FY&@GD7S$KQ,2PTYD9XC)&W#ED=).]Y5-@*=R+6^KO)&5 MCNHRXDU(;,WM_A5WG14/4T,]+.GX\&_TPU X$$_/08C?/AL2G*^4;O_FS4$; M<'OQG\VVA,\6KMPW7#;E';V[>6GV6_(Y)+SUV.W)4\;]3@121B4L0>=DMO24 M#7<8!M,4BSF?X?>3G.30_GHW9!F+X9^X-(A5.&=6X0&G3[?PQX.*/VY= L ? M*(_[WP,$C?9XYOU%$YSGVANV6[5'EKP=Q0Q:43TF75R.OI@7_]E!RT+!*]HD M?)9552HX67Q.Z:9%-.+2;FIA?@C>GEP]AKOY^"O? MXWI,CW??5^S'5-DJ V(XR2ZU9=3,FO"3VYKB=#N7'>R\S)UO65O.8EL(B]&1X,2IFFO?CSC!(U,*(+UNF8YYYA0WB+#V([0\' MJ&7O:_P2W?K#??CS ^[/H&@066GWS)' DQ 3O]5\?S5>.3:?,6Z[_636.>)H M[D@=MOWN_*E)8V(5U"CFR9V_?ECI%F5 M_;*&3:O?EZS*9X^(??EJ]36\%A6<2M_%RY_;KW]FJDFN3+3#JTQO>_/S_(/< M>,6666(K2HD=*35\M'?VV@##%P0F=).KT+B7&E/? OM!::K95+HG-/A^Y^*[ MH?I6. M/$X_S\*35R>5'-\#=>]7'7=7 ><4U2_S/75EXG 2Z%T:BR9<:(W+%U0 G\ MS.Q(7BT\M&S2-?OE]@,VBH)^V0*F#23( !7TX-8T@# ;=@+*R("<$!-@ IIV M"V\Z31]5F""G:0:4,6T*X$W' !54$M+TG+;3'*(MH&D1-:AGP-E(TX.Z+L#I M\+(%.H39H!DB2E)+:A82J>V"4[73(>I1$.KP4A(2M:DNU:B:5*OJ1#VH75%^ MW18U8 O4F*.J #I$0^!P,L(X>(7@T *& UJ0 V/("!P/D% 2PG81DQ'A2G-Y;#*,@S&1@(G'[)<^)T2 +G<<>M!;$?%G!0L;]8U7:Z(84? M:9U33!- X[$WRA6_8 WP 37@?%I3(S ^KALW_:\[#RHB01H YWSJ$(#7#Z[0 M<;GJ>>H&=@(F 9B: M4K.Y\"*B/G7,A@DMFS2 :OE4J1_<#%!!9\/!@6HZO064-FG 8[%Z+O#LJ3T2 M9+7K.!N:S04XZJT=&EZ KK8-O1I1_.I@793.]*&.UWV;?;9K! M+6KVMF#3H-I.Y #&C9B.JM6> >.E+K@ &\"Y.?>#\]J_859N#01(. MAJ>U:9O."W'[:+?M5FV[(?>;%MIIVG&;ZKQMM&=VH7[4C1MF%^K[9KL;=Z'. MV];;;.1M[2V^61'-P--K.WV'%_6MM].WVE[?JEHH+6\\3;F7=TX(459[D6Z+ M_=*Y/7>N#MU@.S=LAXP6K.N)V6[=;MIPY^[:+;<#':(NW*KZ%<*$YKV\RXSN MQMUKVW@C:K]]M!,UI#[<39MP4VH'CKG-QNQNU-M[:9MOH82S]_;KAM]Z^WQO MBY9=8T*UYN[?-N!S?VU>[:L).-@("@MN<@LK$VXV##<-* E.6P5Q,,B-!$%U M[P[ABIK#@(1*[1#6M"RZU&_:!F3J2LVH@S;3%MIPVRZH.:M=IUFGUM[:GQIT M4XD +J'8S^F&;))-DKR!,T"(W,#DN19.02U4!/5 %E0 TE(NPH\,F(Q(9-G6 MCUX@)8EA"JB*C! $RHSBP5L* H\/!G>2$?PV'K,91 5;W !M6L(-2!P: HH< M,5"P/4 "D@ 9R @$JI^X@(F1!UB H]@O+/LNS( 7 -D "V_VS5 !WPV'2"T MQ8 :< -WI QA"Q+0!U"0 O 57PYHBZ@@ @2&F$4#X@O.JQBON7 V^(4"J]5- M' AHI0'C5;Y2#G<+;:)_>)7[\='J^&E(#=*!\J3R,2 &[@#N. -OP 40*!$4 M$ 9"03@("0%/- 33@!I4 VM00+$!"2P6L^(3# A$@ I2@2P4 2@ @&$SZWIOD?2$V?!2(,!G"!1G1 M#)4\FE-SD-#$!\PCUPZL@9X_!*' T(? 7; +[SQUH Q8 +N0 J0%W"@G[3S MK"[//<)#4 58G[LD;V3 !) 32)M,*"8]Z\7L'=F35OG.=NB#R2 _E4A)@)B MN.MN0A [&&+CK.2&N[W7Y42$*.X&ZZYYM6]R P<[8Z\A9..QN/9DK M=L*>*-8"4W#K_/HM4'8ST-B9PAUHZS9@L9/V2\(4T$!;U\RJ_2'XIO,!V6^ M"TC4FYVQEX 7<#Y.>P*P 2*A!L3VV1X]V$!MU]HE(;>O]MZ.VI'[<#\?KCT! M8.[8+D:H6!7S% D@OU#WZV'=R8L"2 O0*"S 8+^$Q+ Z(D:/"@!S 'K_H_4 M^ZRI'ZCI@##V"D$;Y$!?\.[@7;R3]S!@WF\$?$\ ]3VHV_5_!-^3!%^_))V] M&DT9_,Y9>$M\/_!^7<$G!OR>E'X$4W!A!GZ^"YPH<0<<^]^0 _A=%]&!!$ N M8CMXOP-D@,3+'^J> %!\ H !":"0Y9OO#N-=0HQ78=\=Q9>;%W_CT M%*(3:BL#7SW"@Z8)OS=22QC1\L,IHUUVJ6814-L8"/-G0<*#^#*O@]#Z43M. MCMV?>'D4\.83@&.7\TH$2IAY.U_/T( "<.S88L_#@:_^Y\<\B(_IQ9W-/Z+> M7C"*^_SA\!Z>*1AY^H3F\7QG__"2_M";=A31VC4"IO_TFOZJH?508WC.? FX M\R5%J>/@Z+0MAKA3Y19DX-0?#%!/Y\]\@Q=.]_T/089S[L^GPS''9_[BF0-M MDFT37H=7<>9>#)J#<^,5"Z4Y,G-P#D&;$Z\;D,V_TB*%,N&\*)VC.?;,S7D_ M3^>HO Z,@5\#(>BY0" (!@$A*(09T!"H^A0P BV WCL$B, 5-+IB, (3H0P@ M =\# E %4 (AJ,)7(25<'Y> E" \W["VA@,$'$64(3A4 )O WT!LR1T$'\ M08\*4R$CK/M[ A.^29GA]QG=5JQTLG #7@ .> &K$B;$ !V 7X9%&Y ,SN9S M9 :[D=39R[#:%G/J431Q%+ 4/D)E"B^L$R0DP M&-))5%@45@8QG(^;P$-6=/]Z7_M%?+,B!>D)4P@538!GR01JI"3;0 OP$[E /6X7@N+8R M0/Q;O\Y'$7;#NO(2V'^^7\5W&9A_EHSN-_,9!0K"&- &Y %K 'H-!P5: &E M M@;"$2!9A@JU 1B" N=\W,0-D!%8 XV^Q_ M6@(-(@),?6>? ,<;B !%H.=A+H@:K 'A5R%,,5:&].&$98=*8@60'\C"S&(]PD944%18"D$$9\@'/,W""FA1LY7(2@%9LS]YS3P M+])@P7 'K#"]'_[7 @B!E0J*$*2L 6G=_$< XG^X7CEH!LQ_!HTX:/<9!:I" M"S 1+'E4RH$0^24M$9^@)R?("R[ &=#_7036PD10%A03Y4+W%^ U%@\=YV(, M*AA%00\(_:D0@?Z""J%]6J!5NA:>?:>0"S 1.%0YV/\R M4P:X01;N@TY %G@&?@U;0!= H5@:1\"2H&8<#!\!"@ *#=IASK MIB%BB,?A?'@@%G$/SA'P(HJ!)>(]&&^@B 7#,_(%%@5_ WW2(M8'D-^CT1YF M-B7?=N@D;@=,HFOH8QXJ]&']N$,P"UD 8L%[G 'O "&03!R)0:)18&6F _Z?C>*IK 2 MNA-BHAM %$P2O(69:!ZZB:PAM[B%PB )A:R MD M0%!WS(6$G&)(^/Q)?P0?9W$R$ =]XF!X%MQWA=_H!\9X!ZJ# P@!2H \8F+0 M!GP$2(&[< :P!3$%I2*#: J>( 2#U[P$: ;D-Q%,&>9?#T('N(H18._G- @9 MB4%1$-!LBM+?'G#6-( /8+*(.X!XG:*T"#\()5\&"&"S7("WH@28)'2*V>+_ ML2T("\Y'7D NKHO>8@WR>KP1X$< 0G!XBY%%7.=ZC B0A 7#+I*+Q=Q TQ5 MB_LB^@#=D+EE0LEHW=0U*@.VHD:D,6 *;A95!/ M*!,%8Z^'FAQX!X$0< 6,C(R-5##E080LH\I(2@2-FH++^#68AFN 87 %/"/4 M"LUX+>(83$T8(894C)^%E$$Y 'EI7A031CB-%\$R4C4H**P>AG(S:G@CW\20 M5-@ N%[GP=>5C6T,$J,B5(Q>XE30)82)E,/S%S=N%AW#1[ .5HW301#($ Z& M=@$*P 0@ 2E 32 #U 0T 4A0$YP-+$"9L1 =U &:'#;A7H+@6U7$I $Y\1" M0 -,CI%0O;<0T'M0QDP@$C NVP8WP>9PCINCB((GV'9/1@Z0'-X,(4%.H I] MA:3CC(()!<:C0.^ICF7"Z+A(_16U8YO'.YX39P+MB">\AE#& M\)@3+(_U'K>WN16/C\A"\!J**)H9?W$[^'B:X^28 VR.;L'C^#HF.)ECS*8] M2HXJBWI0.BZ-U:!A0"4FB2 #)_A)P'IH'9Z XR.THKVZ.79C[1C<_@ZKHZ< MHVWWX. )(0'V:-MI,.JC[9C9$(^B8TEP/(8&9@N4<3RZ!7,*[4C3&(_\P>LH MBVB/OL+]*#L"=W:!KV#;Y8XA 91Q03XBR!SMN,E-CB(D\;A!AI MP/WXX%20 M#\X)R4UDD!0D0_"Q!(]VP6A /!Z//21PESY2&0KD3"@:N(XY 8Z')]![$*3K MZ$ N.--C_CA YC;XXWZ!0!*1&.3F^."L2MKC[3@^;HX2)!6Y1<:.U.,-"=RY M!=_C_YA?T8XXP>C(/W(+^:,)F4"V>>QC@^ ^5HG7BOS8Z1EZQYSY&!I@C@^. MEQ=>') @0!H9&H07FV-.(#KF!'/;](@#%)!EI$W@2#($#PZ?=DZ8+2'D BD[ MVG9#9.DXP6F0D*3,IT)NCIDD+"?TG1,VI"@I.^I[T9_K>#8XC]'?_9@[XHZ1 MI&@P/4J/A^0*$3T*D="?+MDZ3HZI)!19$FR0C.0:.4,R!%"&)7E,:H^I9&4R M.F:/W *.M[X]DZZCB!)>, 1_I.X'#=)[E^-^$$ER"%$D+1E> '>7(W#'+8R3 M461(4$/.D8]('5D&W)'PXR8H9>R1:)TE^4>^AH&D'W/,?3?UGO^82YJ3H@$G MV4N&!!0D<+=.8I$HY ))08:2&>0,^1H6DP_EH^ Z7I&>9"&I0GJ0T9_E^"CP MDM"?0?DHJ).UY!GY4;*0CP+RB$3&DOWD]5A+GI2;S>O8%FAMH@%!V4MBB*(! M0!E,[AJ2.>7T*!.RD2HE5!A%FI+!)/A82.*4O22]MT5JD;VD48DY M4I0S'DQ 1AJ2O23K6$@:D?9D(8E/EI'N)#QI),PM!).C[H@FZ),W MPRUY/YZ39.4,:45"?VVE: Z1I)Q9>=X2+8 S&,AF3K"E9LC'+E7II(M0$CP M1X*1<.5:>53ZE"U -7E8@A/LY VY5?J/)4$'>4F&DK^"(YE(TI4CI&A 2<*5 MD*-FR4*V "(E#4E7A@3#I!,)5\Z5*Z0U259^D,!D8]E5R@'OXU?)-\Z/A5X] M&5X@CR"E(%GR<0OY1;W72TZ2OF7)!T%>EJFD<"D:8)2^PFMH%Z21E:39$$Y: MD(TE],=44I707V;)14:6%N5R:=%%DCCD$3DZ5I?/I$#943*0@F5(R4!>EM ? M)-E2AI @ >>85Q:73.4&>4XZE\,D:1G]>9,KY$&Y/C*-[R1LB4?&C_,D_4@2 MPHU"8D)7)%H-H]_=2.K1)\_?$V ?:">< APP9(01!.3HB-W0COOC LDM:''G MA&0IHDB/C"1>Z4(2E]%E26DFH)=09&C 86:23"0KLED&DRTFR^92FI@@P7X1 M8JJ4F%%KB5".E?>CB))C/H\:IM*G1H8$&^:+Z5KREX:!CNC)@)6?!'@QLB * MW@$J^(RT!.#&-;$I(I47IEV@6VZ6%R9VR66RDM"?4LF*T'N=9%3I7#XXK0CM M" U.CQ>F-4E?LB+O98G98G(8)"5#<#9DF=$?ENE-VI QHQM03\R61P+ 804F9%\",$@>N@?\&(P'-N@)4Q1@ ,(@CT*#O:F*&> MK\!47IB,I"WYX'@VQ"/T)T."!+\C=ZE-RIC1I(^77+*9#$&(J4..F;MC!KEI MZIBE8Y_Y9^J1X(5=(#)&F6,E_W9*GI6>I(@Y3$:40N5U\QKFDF0FG#D]PIBT MXXTS.4J3MJ2(OX9'":U.:[ MF6:&, IG"7DSA)S,9&-)D+1DFDEL1I65I8V#7IZ9ON.,*68. MCS1AB;E5#H\B2KZ)3+(BBF4+:6WNE5ZDBLEI\IFXAY\I;AJ:9YRN0D?>F MEHE6ZG-0QKUY3@J3K,A^84UBG6FFB )M7C?E);798SJ;D65@.3E&G7 EU+EG MFIQ)YZXY3S*=4*:Y.6!^@XAF,W$^S FCG_)7W.F-6<-Z=WQL@HNF7W!EGIVT MY)9Y=LJ4:^:^:5^&F8=GR;EC,I[[YLQYW!!KV"9#X&,VF\4E*WEO-I=(9;SF M4Y9\D>>SR7;*C$MGRMETUI-FIUV0M4F=Q28I"1+@J:I^=8 M65YR;M-CP]ER6I6>)INI4=:>*J9M)T<^GOD>F\E1:AS1)B%9>UJ;9"1@*6:B MC6IFZLF* '=WY?0)?Y:1)^?;:7K&G6=>PCDY@IJ48_!XW?R;5G\8>W$EH M;HJ&9=96$NB6$^7K66I:E,%G"?I2IJ"BY^.900:?R*5%N6^>E"_E[HE*1I>V MII"93%:;)BAV&8+"GO2G\UEPWI\?J*D9;?*2).@1RD%:E&BF>-E0LB*$9#"Y M55*?M*=%.7':=IEDFIAN$H]2Z.O(?F*7)R7U&6J>FF)H$$IZ#J'0)_YI6W*: M"@Z&B5;>EK(C!IIX*BU+:+.9/$*@CZ=S>6MZF=$C-GE?NI!::.89/6J'>:8* MB53^EK>F?@EN"J$H9QKZ@2Z1V.:%&1IDF'TEK1D^4I$+Z ^Y0$Z;<>6AB"0G= M;G 94 YBI=U8=X*):,%\0&CF>\MG(>DZLIX*) 7*82:1KJ9%N7P4H]*D4\5& MGII*Y&=S3MP$YP2.QX$VF41HE'DWSHTF#2JH8$X,\L18Z8J:#1AEPSEK\FO3 MIJM960:?R*;9&7R.GT2FD(E[HI#;PO18?&Z/ALRF6%Z.!"2E \J.AIH"J4!9 M7 J;UB@SVH]NH^/FDTEHVJ)>8$)7-R*($X.S #A\B<)HE$D_PI41CQNIZ>B/ M;^A"\%;Z"N0CE$%MZI]LI>1HDC:C*21;64&NI/UHGMEB=I!Y);6959*DA*5- MZD?"E9@C3&IFHI,A3*"C/=JD[650ZGWRHS^I<[E#0I$])TOJ7-Z5V*A/FGM& M?S-I8NDZ[J!D94IZE49_6>G- .(G,5E>N*19I!8Z:S).W:6'.E8VEDBG>(A M@-F!"@L.*;!9&\(+$>FCD9&:FQPI< <\0I$,9TAJ/JJ6::14BH\"D(#I7'F2 M!H^JY>PHF *,Q:@(>93:I% D#)D?+*;WI3:)DPZE+&E'&J;PI4CI5,IX+J6& MJ5/*@'JEER516I7*!"2E/@>8HJ5IY6KJ>%:4K2E/RII6EJXI5DI(+H^]8UA* MFWJE(B=<:=OMI7.E;ZJ)9KG)V"DH ,:G\5V"#V9B<-/= M!#* ?=I9X*=A(["!"GJG[D(8$-S,%B((1VI8(G-YI6ZY$%RHOX+(>9+* )6H M"+F5ZJ29I8B*F<:DT* LR5G6I"QI\'A7TIF4:2\Y?R)SCZEHBEA^.2=J4JJB M8J@YJF<:0WZ7H"6+BJ("GYQEC8JBX@D )8W:HPZFP2-;":-VIH\(0ZHGT)31 MY_,W.#H+T4*<6!M8"B+ /W@RF $BP*;XB7*6O:.&BG4B<[WC8>$1C,@YJE^JE)R]!1RL D@2JX M,:B*J> D6[E"]J45Y><(I"JF4"JT"$[>E:RJ3EI>(G,C:F9:E5*?C>KI"*MZ MJ#Y>>$BG IH"IJ&I(GH%1,;H]P+^&'%BQ3>L[G]3W520%*R#!)^;X!),$ 1? M@5(7X@%?7:UX&?B)G,(SLEA(&YEB45 AX 'A8_XHHLQU>$ (R47ZFO.=27@Z MX'_!8I>@"%H*$\SE@@)T&=MI_D>L9C+P0L@"+S@-IZ+@^0+D 6>#3*AQ&G,# M:ZB)7[R. $.2FL5:99<'7*(@*L/:KBXC?")BL!+T,/9$"\ :5!LO08[B M%5"&12