Subject: Re: Next MPI Meeting (etc)

From: L J Clarke <[email-masked]>
Date: Thu, 10 Jun 93 15:59:53 BST

Hi David

Seems to be just me and thee on the contexts and examples front at the
moment.  I guess people are just too busy. 

> 	A more complicated example based on the convolution example could
> 	use overlapping groups. Assume for example that the parallel FFT
> 	requires the number of processes to be a power of two, and we have
> 	a total of 12 processes. Then we could put 0,1,2,..,7 in one group
> 	and 4,5,6,...,11 in the other group. I'm not sure how much this
> 	complicates things.

I was thinking, in terms of this example, of something like having
arbitrarily different distributions and numbers of nodes in each of the
two groups.  This would introduce an arbitrary communication pattern
between the two groups making sender choice more important to the
example.  I was really thinking of abritrarily different computations in
two or more groups.  Trouble is, its real hard to write a complete yet
sshort example - conceptually simple is easy, small simple code is not
so easy. 

> 	Sorry, in my corrected version there was still an extraneous
> 	mpi_safemake_comm inside the first conditional branch. This has 
> 	been corrected in the revised version below. 

Thanks.

>       The arguments to
> 	my version of mpi_safemake_comm are:
> 
> 		mpi_safemake_comm (local_group, remote_group, communicator)
> 		IN  local_group
> 		IN  remote_group
> 		OUT communicator

Well of course I support this kind of communicator constructor, as my
previous messages make obvious.  I guess the point is that this is *not*
the kind of constructor in the working document from Marc Snir - this is
something that Marc identified as left out for now.  Further this is
*not* the sort of thing that Tony/Mark were proposing to include in the
working document - from what I can gather.  Would you like me to type in
explanation of what I think is the kind of thing that they are
suggesting? Would that help anyone?

> 	In the sends the rank of the destination process is relative to the
> 	remote group, i.e., the receiving group. In the receives the rank 
> 	of the source process is relative to the remote group, i.e., the
> 	sending group. This seems the natural way to do things. 

Yes this is the natural way to do communication between different
groupings, of course, in my opinion. 

>       As you
> 	point out, the source rank in the receives can be replace by
> 	MPI_DONTCARE. 

Yes, in the example given, because the communication is a particular
pattern - i.e.  one process in A group sends to one process in B group
and vica versa.

>       It's not obvious to me why a C2 receive can't
> 	specify a particular source in the remote group. Please explain 
> 	this to me.

The comment I made was a result of confusing your example with my
understanding of the working document and what Tony/Mark are suggesting.

In particular the way in which your example assumes to use
publish/subscribe is quite incompatible with what I think is meant my
Tony/Mark.  Are you aware of this? I can send an explanation of what I
believe thes guys intend, if that will help.  (Frankly, it will mean
that you can write your example program in a half-sensible way, but more
complicated examples like those I suggested above cannot be written in
any sensible way.)

I should make a general point about your example.  When building the
communicator for intercommunication you are using the init_group and the
fft_group.  Now the init_group is a superset of the fft_group, in fact
it encloses the two different fft_group's.  This is a rather strange
thing to be doing since it means that calculation of the destination
rank in the other group refers to the enclosing group. 

It is *much* more natural and expressive to think of the destination
rank in the destination group, so that a better thing to do is to make a
communicator with the local group being the fft_group and the remote
group being the other (remote) fft_group. 

Would you like me to adapt your example program to this way of working?
It will be easy for me to do - of course I will be inventing definitions
of some things like publish and subscribe. 

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

---

From: L J Clarke <[email-masked]>
Date: Thu, 10 Jun 93 19:44:35 BST

Hi Tom

I forwarded some stuff to you personally, that I had sent out to
mpi-context last week.  I think it covers the material.  I try again in
a different form of words here, for your convenience.  

From what I can glean, what Tony/Mark suggest to incorporate into the
context subcommittee working document.  More precisely, this will be
part of what they suggest, specifically part to do with inter
communication. 

I hope this helps, or I just wasted 45 minutes of my life :-)

Best Wishes
Lyndon

----------------------------------------------------------------------

1.  The "map" should be replaced by a naked absolute group.  This is
similar to David's suggestion, I think, but the group is not relative to
a parent or the such. 

2.  There should be exactly one kind of communicator object, as opposed
to two communicator objects.  The object should be a binding of exactly
one group and exactly one context. 

3.  The communicator object is used for communication within the group
in the way we have all come to expect in MPI - the user can do send
addressing the receiver by rank in the group, and receive selecting the
sender by rank in the group. 

4.  There should be a publish/subscribe mechanism which implementa a
name service and allows a user process to obtain a communicator object
which the process did not create and moreover when the user process is
not a mamber of the communicator group.  More details follow in the
next 4 items.

5. The publish operation is:

mpi_publish(communicator, name, persistence)

IN  communicator - communicator to be published in global name space
IN  name         - name by which communicator to be published
IN  persistence  - a flag having the value MPI_PERSISTENT for a publish
                   which can be subscribed any number of times and
                   remains in the global name space indefinitely,
                   or MPI_EPHEMERAL for a publish which can be
                   subscribed exactly once and is removed by the 
                   first subscribe.

There is also the suggestion of some kind of "permissions" argument,
which is suggested to be an array of contexts which are allowed to
subscribe to the published communicator.  I dont understand this part of
the suggestion so I cant make further (guessed) description. 

6. The subscribe operation is:

mpi_subscribe(name, communicator)

IN  name          - the name by which the communicator is published
OUT communicator  - the communicator published by the given name

The communicator obtained by subscribe provides the subscriber with the
communicator context and a picture of the publisher group.

7. The publisher can use the published communicator for the following 
communications:

a.  Send to processes within the communicator group addressing the
receiver by rank within group.  This is the usual send capability. 

b.  Receive from processes within the communicator group selecting the
sender by rank within group or by wildcard.  This is the usual
receive capability. 

c.  Receive from subscriber processes selecting the sender by wildcard. 
The receiver cannot choose the sender since it does not have a rank in
the group.  The reciever cannot discover the sender - the sender will be
recorded as "unknown" through a constant such as MPI_UNKNOWN, or
MPI_ANYPRCOESS or the such. 

8.  The subscriber can use the subscribed communicator for the following
communications:

a.  Send to processes within the communicator, i.e.  publisher, group
addressing the receiver by rank within group.  This is the send
capability extended to processes outside the group. 

9.  Tony's mail of June 5 said that the communicator can be used by the
subscriber for receive selecting the sender by rank.  This is
inconsistent with the context allocation in the working document from
Marc Snir (so I assume it is a small mistake) because the subscriber
could legitimately be using the communicator context within a different
communicator. 

10.  There is mumbling about using a group union to do receives when the
sender needs to be chosen.  Tony did send me an email with some C-ish
example code for this but the example code does not work with the
context allocator of the subcommittee working document. 

----------------------------------------------------------------------

---

From: Tony Skjellum <[email-masked]>
Date: Thu, 10 Jun 93 14:08:20 CDT

Lyndon, have you sent me e-mail describing the following failures???
- Tony

----- Begin Included Message -----

---

From: L J Clarke <[email-masked]>
Date: Thu, 10 Jun 93 19:44:35 BST

Hi Tom
...

----- End Included Message -----

---

From: [email-masked] (Tom Henderson)
Date: Thu, 10 Jun 93 13:44:23 MDT

Lyndon,

> I hope this helps, or I just wasted 45 minutes of my life :-)

Yes, this helps a lot.  Thanks!  

> the suggestion so I cant make further (guessed) description. 

A few comments:  

1)

Would it be a good idea to have a way of publishing a whole bunch of 
communicators at once?  We designed a tag registration server a while back 
(and have not implemented it hoping MPI will do a better job :-) with 
registration routines that allow a calling process to specify how many "tags" 
it wants to reserve under a particular name.  This might be good for context 
name registration also.  The subscribe routine would match:  

mpi_publish(list_of_communicators, number_of_communicators, name, persistence)

IN  list_of_communicators   - list of communicator to be published in global 
                              name space
IN  number_of_communicators - number of communicators in list_of_communicators
IN  name                    - name by which communicators will be published
IN  persistence             - a flag having the value MPI_PERSISTENT for a 
                              publish which can be subscribed any number of 
                              times and remains in the global name space 
                              indefinitely, or MPI_EPHEMERAL for a publish 
                              which can be subscribed exactly once and is 
                              removed by the first subscribe.

mpi_subscribe(name, list_of_communicators, number_of_communicators)

IN  name                     - the name by which the communicator is published
OUT list_of_communicators    - the communicator published by the given name
OUT number_of_communicators  - the communicator published by the given name

2)

What happens when multiple processes call mpi_publish() with the same name?  I 
could see this happening in a distributed server for instance...  I assume 
that mpi_publish() will return a status indicating that the name has already 
been registered.  The process would then call mpi_subscribe() to find out what 
name is associated with:  

status = mpi_publish(comm, NAME, MPI_PERSISTENT)
if (status == MPI_NAME_ALREADY_PUBLISHED)
    mpi_subscribe(NAME, comm)

If this is happens a lot (I think it will with the current MIMD/SPMD model), 
we may want to modify publish() to do this for us automatically:  

mpi_publish(in_communicator, name, persistence, out_communicator)

IN  in_communicator  - communicator to be published in global name space
IN  name             - name by which communicator is to be published
IN  persistence      - a flag having the value MPI_PERSISTENT for a publish
                       which can be subscribed any number of times and
                       remains in the global name space indefinitely,
                       or MPI_EPHEMERAL for a publish which can be
                       subscribed exactly once and is removed by the 
                       first subscribe.
OUT out_communicator - if name already exists in the global name space, 
                       out_communicator is set to the communicator registered 
                       under name.  Otherwise out_communicator is ignored.

Return status indicates if in_communicator was successfully registered or if 
out_communicator was previously registered under name.  If the calling process 
cares, it can check to see if in_communicator and out_communicator are the 
same in the second case.  

3)

1 and 2 could be combined.  

4)

If we have mpi_unpublish(name) (and I think we should), then we could make the 
restriction that only the publish()'ing process can unpublish().  I'm not yet 
sure if this is a good idea...  Comments?  

Tom

---

From: Tony Skjellum <[email-masked]>
Date: Thu, 10 Jun 93 14:49:52 CDT

4)

Tom

Yes, I concur.  This was a detail we did not write down yet!
- Tony

---

From: L J Clarke <[email-masked]>
Date: Fri, 11 Jun 93 10:27:13 BST

Dear Tony

> Lyndon, have you sent me e-mail describing the following failures???

I dont see these as "failures", but the answer is "yes and yes". 

Will it help if I resend same email-s, or perhaps if I send new email
covering both points more slowly and in more detail.  I'm happy to do
either or both if it helps. 

> could legitimately be using the communicator context within a different

I do believe I sent stuff to you personally during our conversation of
last weekend and also to mpi-context this week, both of which stuff
covered this point among others.  Relevant header data of latter email
follows. 

Date: Sun, 6 Jun 93 16:39:45 BST
Subject: mpi-context; comments on "June 5 - intercommunication"
To: [email-masked]

Indeed scanning that email I can see that I asked the question whether
yourself/Mark were also suggesting that the context allocation is
changed to be like Zipcode/Proposal-X, in which case it would be fine
for the subscriber to use the subscribed communicator for receive.

> context allocator of the subcommittee working document. 

I replied to your mail containing example C-ish code making this point,
relevant email header data follows. Did this fail to reach you?

Date: Sun,  6 Jun 93 20:17:24
Subject: Re: more ideas on C2-level
To: Tony Skjellum <[email-masked]>

---

From: L J Clarke <[email-masked]>
Date: Fri, 11 Jun 93 10:29:49 BST

> Yes, I concur.  This was a detail we did not write down yet!

Yup, its worth having an unpublish(), and restrict such that publisher
process alone can do the unpublish.

Best Wishes
Lyndon