Subject: Re: suggestion for changes in 5.5

From: L J Clarke <[email-masked]>
Date: Wed, 5 Jan 94 14:41:36 GMT

Happy New Year to all!

> 5.5.2.
> The usual mechanism for binding client to server (e.g. via sockets) is that a
> client opens a connection, without specifying the remote port (client), and
> starts listening (receiving).  The client opens the connection by specifying
> both local and remote ports, and starts sending messages.
> This protocol is asymmetric: the server need not know the name/address of the
> client, whereas the client provides full information on the server.  Also, the
> server operation is nonblocking:  The server may continue serving other clients
> while waiting for a new client to join.

For concreteness this is as follows.

----------------------------------------------------------------------

Client
------

int              socket_fd;
struct sockaddr *srvname;
int              namelen;

/* create a socket */
socket_fd = socket(domain,type,protocol); 

/* connect to server "srvname, namelen" */
connect(socket_fd, srvname, namelen);

/*

In this case the name of the server socket "srvname,  namelen"  are IN
arguments for connect() and the example does  not deal  with the whole
ugly magic associated with how the client can discover them.

*/

write(socket_fd, data, nbytes);

Server
------

int              listen_fd, socket_fd;
struct sockaddr *srvname;
int             *namelen;

/* create a socket */
listen_fd = socket(domain,type,protocol);

/* start to listen */
listen(listen_fd, backlog);

/* accept connection */
socket_fd = accept(listen_fd, cliname, namelen);

/*

In this case the name of  the client socket "cliname, namelen" are OUT
arguments for accept() and  the example does not deal with  the  whole
ugly magic associated with how the server can interpret them.

*/

read(socket_fd, buf, nbytes);

----------------------------------------------------------------------

There is  another  aspect to  the  socket mechanism for  client-server
which  involves  use  of  select(2).   The server  wants  to  wait for
connections  from new  clients  and communications from (possibly also
to) existing clients.

Not an option for the server to block waiting for a new client without
regard for communications from existing clients since this  means that
existing  clients dont get serviced, possibly indefinitely if there is
no  new  client.   Not  an option for the server to block waiting  for
communications from  existing clients  without regard  for connections
from new clients since this  means that new clients dont get serviced,
possibly  indefinitely if there are no  more  requests  from  existing
clients.

The server sets listen() on its dangling socket  and waits in select()
for either communication from an existing client or  connection from a
new client.  In the above example there is a select() call in  between
the listen  and  the wait.   The accept()  call is in a program branch
after the select()  call  etc.  If the  connection was  ready then the
server  calls  accept() which  completes  the connection  to  the  new
client.  If  it was a  socket file descriptor  ready for read then the
server reads the message and does with it. I'd extend the example, and
will  do so if people want, but the select()  call is rather  messy to
use.

If we translate  this  into MPI like terms, it would be something like
the inter-request handle being suitable as an argument to MPI_WAIT_ANY
(etc) along with non blocking receive requests.  The alternative I had
thought of,  which  had  the advantage  of not  mixing up communicator
management and  point-to-point  quite  so  intimately,  for binding of
clients and servers follows.

The client send a message  within MPI_COMM_WORLD (or the copy thereof)
to the server,  such message meaning "I want to  connect".  Then  wait
for a  reply from the server such reply containing the tag to use  for
calls   to    the    intercommunicator    constructors.    Then   call
MPI_INTERCOMM_START   with   the   bridge   (peer)   communicator   as
MPI_COMM_WORLD (or  the  copy  thereof), remote leader rank is that of
server,  and  tag  is that  contained in  the reply  message from  the
server.  Then call MPI_INTERCOMM_FINISH.

The  server post nonblocking receives for each existing client in each
case using the (inter)communicator which provides a connection between
the  server  and  the  client.    The  server  also  keep  posted  one
nonblocking receive from  any member of  MPI_COMM_WORLD  (or  the copy
thereof).  The server wait in MPI_WAIT_ANY (etc) for any of the posted
receives.  The server receive either a message from an existing client
or this "I want to connect" message from  a new client.  On receipt of
the "I  want to  conect"  message  the server  reply to the client  in
MPI_COMM_WORLD  (or the copy thereof)  with  a tag which can safely be
used  for  the client  and server to bind using the  intercommunicator
constructors.  The  server then call MPI_INTERCOMM_START using tag and
in  MPI_COMM_WORLD (or  copy  thereof) and remote rank  is that of the
client which sent the message.   Then  call  MPI_INTERCOMM_FINISH. The
MPI_INTERCOMM_START and FINISH calls in  the  server block only  for a
short   time   as   the   client   is   also  calling   the   matching
MPI_INTERCOMM_START and FINISH.

The client  is  actually a group.   So before the client  group leader
sends that  first message to  the server there is a synchronisation in
the client  group  and  all other  members of  the group wait  for the
leader.  The reply containing  safe tag can be broadcast by the leader
client,   and   then   all   members   of   client   group    do   the
MPI_INTERCOMM_START etc.

The server is actually  a  group.  So the  client  send  to the server
group leader.  After the server group  leader receive the "I  want  to
connect"  message it must  send  "It wants to  connect" down  to other
servers.  The server group  synchronises before the leader  replies to
the client  with a  safe tag for the  intercommunicator  constructors.
Then all members of the server group do the MPI_INTERCOMM_START etc.

Is there a flaw in this mechanism? If  so then I would be grateful for
an explanation of  the flaw.  If not then I dont think we need to make
extensions which attempt to move toward the socket mechanism. However,
if there is a preference for a socket  like mechanism then  I am happy
to work on a  proposal  for such mechanism  (which I believe would not
make  too  much sense  unless  it  contained the  ability to wait  for
combinations  of  communication and  connect/listen  requests  -  thus
impacting on the point-to-point chapter).

> 3.  For good measure, we can add a blocking call:
> MPI_INTERCOMM_MAKE(local_comm, local_leader, peer_comm, remote_leader, tag,
>                    intercomm),
> which blocks until the intercommunicator is created.

This  suggestion  is  fine,  in  fact  was originally  there,  and  is
convenient for the usage anticipated as described  above. It increases
the number of functions by one, i.e. epsilon which imho is negligible.

> A server that can accept new clients executes a call to MPI_INTERCOMM_START.
> The server leader provides the "peer_comm" parameter and passes
> "remote_leader = MPI_ANY_SOURCE".
> The server then periodically calls MPI_INTERCOMM_TEST to check if it has pending
> requests to bind.   If it returns "flag=true" then the server starts
> communicating with the new client.

This  means  however  that  the   server   cannot  block  waiting  for
communications from existing clients, since this may mean that waiting
new clients always wait.  So the server  is  forced to be a busy loop?
(In the scenario I described the server is not a busy loop, unless the
(low quality?) implementation of MPI_WAIT_ANY is itself a busy loop.)

> 5.5.3
> 
> I propose to change the name service constructs for the following reasons:
> 
> 1.  It is an overkill to force each MPI implementation to have an (anonymous)
> name server:  
>
> When I run with 8 processes, I may not want an additional name
> server process, or have one of the eight processes perturbed by an additional
> name serving function.

Sure.   If  you dont use a name  service then  you dont want resources
consumed by it. So it is useful to consider how we can avoid such.

One thought which comes to mind, MPI demands that the program creation
environment at  least allow  SPMD  program creation,  perhaps we could
make the inclusion  of  name server(s) a mandated run time _option_ in
program creation environment (possibly also a link time option).  This
way if you dont want it you dont get it, and if you do want it you say
you want it and you get it.

> 2.  I may want in a large system multiple name servers, and perhaps
> hierarchical name servers.
> 3.  Name servers, as currently defined, may be difficult to extent to a dynamic
> MPI, where processes come and go.  To do so, we should have name servers that
> are local to a communication universe (e.g., MPI_COMM_WORLD).  The mechanism
> provided to extend a communication universe will, ipso facto, also extend the
> group of processes serviced by a name server.   Quite generally, we should want
> our standard "communication context" mechanism, namely communicators, to extend
> to name servers as well.
> 4.  Various communicators may correspond to various
> communication protocols, various protection mechanisms, etc.  One more reason to
> have the name server "local" to a communication context.

This is  fine  so far.   The name service was intended to be (in  some
sense) local to  MPI_COMM_WORLD - i.e., it did not intend to offer  to
provide intercomm  services  which  bind processes within the group of
MPI_COMM_WORLD to processes outwith that group.

We could accomadate(sp?) these considerations by adding a communicator
argument to  MPI_INTERCOMM_NAME,  this  communicator  identifying  the
scope of the  name. If there  is no name service  associated  with the
given scope then perhaps this  is a non-fatal error condition.  If  we
do  this, then  I do  hope we  mandate that  there is  a  name service
associated  with MPI_COMM_WORLD  (subject  to possibilities of program
configuration options perhaps?).

So MPI_INTERCOMM_NAME would gain one argument and then look like

MPI_INTERCOMM_NAME(comm, scope, name, inter_request)

IN  comm           local intra-communicator (handle)
IN  scope          intra-communicator scope of name service
IN  name           character string 
OUT inter_request  handle for MPI_INTERCOMM_FINISH

> 5.  It seems that the functions in 5.2.2 can be used to provide the functions of
> a name server, as well, thus reducing the number of functions.  

It decreases  the number of functions by one, i.e.  epsilon which imho
is negligible (as above).

The detailed  suggestion  does have  some abstract  elegance,  and  it
removes  the function  MPI_INTERCOMM_NAME  while  retaining  a  "name"
service capability in which "name" is a non-negative integer. However,
what does  the  proposal do for us?  It  removes the string name which
after all was more convenient for the user (loss) for what seems to be
a very small decrease in the complexity of implementation (gain). Is it
worth making this change?

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/