Subject: problems with MPI_COMM_DUP and MPI_COMM_FREE and intercommunicator

From: L J Clarke <[email-masked]>
Date: Wed, 12 Jan 94 17:10:35 GMT

Dear All

In the draft  MPI_INTERCOMM_MERGE  is a  collective  operation  in the
union  of the two  groups associated  with the intercommunicator.  The
draft  also  says  that  an  intercommunicator  can  be  created  with
MPI_COMM_DUP, and  is  destroyed with MPI_COMM_FREE.  It is not stated
that  these are collective operations  in the union  of the two groups
associated  with the intercommunicator, although  I believe that these
are assumed semantics.

Let  us  consider  two  groups  G  and  H   which  at  first  make  an
intercommunicator with  MPI_INTERCOMM_START and  MPI_INTERCOMM_FINISH.
The following  pseudocode  executed  by all members of G  and H  works
whether the intersection of G and H is empty or non-empty.

IF (IN G)
  MPI_INTERCOMM_START(Gcomm, 0, MPI_COMM_WORLD, 1, 99, &Greq)

IF (IN H)
  MPI_INTERCOMM_START(Hcomm, 0, MPI_COMM_WORLD, 0, 99, &Hreq)

IF (IN G)
  MPI_INTERCOMM_FINISH(Greq, &GHcomm)

IF (IN H)
  MPI_INTERCOMM_FINISH(Hreq, &HGcomm)

The important thing  is that where  G and H intersect the processes in
the intersection make  the calls all in  the same order, as Nathan has
pointed  out.  Notice also that processes in the  intersection end  up
with two intercommunicators, GHcomm and HGcomm.

Now imagine that G-H want  to  create another intercommunicator by use
of MPI_COMM_DUP.

IF (IN G)
  MPI_COMM_DUP(GHcomm, &GHcomm2)

IF (IN H)
  MPI_COMM_DUP(HGcomm, &HGcomm2)

This  looks extraordinarily  bad.  

Imagine that MPI_COMM_DUP is a collective operation in the union of G,
presumably the following occurs.  Processes in G and not in H complete
the  textually  first  MPI_COMM_DUP.   Processes  in  H and  not in  G
complete the textually second MPI_COMM_DUP.   Processes in G and in  H
complete the textually first  MPI_COMM_DUP  and are then  stuck in the
second MPI_COMM_DUP, i.e. deadlock.

Imagine  that  MPI_COMM_DUP is defined  such that  none  of the  calls
complete  until  the calls for all instances  of the GHcomm and HGcomm
communicators are made - it is bi-collective in  G and H.   Presumably
then the  processes  in  the  intersection cannot  make  the textually
second  MPI_COMM_DUP  since  they  are  stuck in the  textually  first
MPI_COMM_DUP, waiting for the calls on HGcomm, i.e. deadlock again.

The   same    considerations    apply   to   MPI_COMM_FREE   with   an
intercommunicator. These are  just  the  same  problems as  a combined
MPI_INTERCOM_START and  MPI_INTERCOMM_FINISH call  MPI_INTERCOMM_MAKE,
as Nathan has also pointed out.

So we  have a bit of a problem here (okay,  this is at least partly of
my making).  I suggest  that  we  might overcome  the  problem with  4
changes as follows ...

1) Change name of MPI_INTERCOMM_START to MPI_INTERCOMM_MAKE.

Trivial change in draft, for consistency with below.

2) Disallow MPI_COMM_DUP and MPI_COMM_FREE on intercommunicators.

Just means removing some (already vague) text from the draft.

3) Introduce  MPI_INTERCOMM_FREE which is a  free-start  operation and
must be finished with MPI_INTERCOM_FINISH (c.f., MPI_INTERCOMM_MAKE).

The implementation is simple and structurally similar to
MPI_INTERCOMM_MAKE.

MPI_INTERCOMM_FREE :-

mark intercommunicator as being freed

barrier in local group (can be omitted if no error tracking)

zeroth member of local group Isends "I wish free" to zeroth member of
remote group (this can use mpi provate hidden intercommunicator
contexts)

zeroth member of group Irecvs from zeroth member of remote group

done

MPI_INTERCOMM_FINISH (for free) :-

zeroth member of local group waits for isend and irecv to complete

barrier in local group (can be omitted if no error tracking)

deallocate contexts and other clean up

done

4) Introduce MPI_INTERCOMM_DUP which is a dup-start operation and must 
be finished with MPI_INTERCOMM_FINISH (c.f., MPI_INTERCOMM_MAKE).

The implementation is simple and structurally similar to
MPI_INTERCOMM_MAKE.

MPI_INTERCOMM_DUP :-

allocate context(s) in local group (may be a barrier)

zeroth member of local group Isends "I wish dup" and contexts to
zeroth member of remote group (mpi private contexts as per free)

zeroth member of group Irecvs from zeroth member of remote group

done

MPI_INTERCOMM_FINISH (for dup) :-

zeroth member of local group waits for isend and irecv to complete

bcast remote contexts in local group (may be a barrier)

munge together groups and context to make new intercommunicator

done

Finally there  is a  related issue with MPI_INTERCOMM_MERGE.  It looks
like processes  in the  intersection are expected  to call twice.  I'm
sure this is not what  Marc  intended since the above deadlock problem
arise. In order to ensure  that there are calls which cover all of the
endpoints for the intercommunicator I'd suggest the  following  change
to MPI_INTERCOMM_MERGE.

MPI_INTERCOMM_MERGE(gauche, droit, nouveau) 

- args?! Well, okay, we are in nice in a few days time :-)

IN gauche an intercommunicator or MPI_COMM_NULL
IN droit  an intercommunicator or MPI_COMM_NULL

OUT nouveau the new intracommunicator

semantics by example ...

IF (in G and not in H)
  MPI_INTERCOMM_MERGE(GHcomm, MPI_COMM_NULL, &new)

IF (in H and not in G)
  MPI_INTERCOMM_MERGE(MPI_COMM_NULL, HGcomm, &new)

IF (in H and in G)
  MPI_INTERCOMM_MERGE(GHcomm, HGcomm, &new)

This  makes  a new intracommunicator from  the  intercommunicator pair
GHcomm and HGcomm (of the  above  example) where the intercommunicator
in G must take the same arg  position for all members  of G,  and in H
for  all  members  of  H etc.  So if there  is  an  intersection  both
intercomm are given and they  must be in same order  over intersection
which determines the order elsewhere. 

In the current definition a  key is used to  order the  two groups for
the merge-union operation. This is  now achieved by argument ordering.
Its the same thing really.

I could have suggested a similar intercommunicator argument  pair  for
the _DUP and _FREE but it seemed rather less natural in that case.

Comments?

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/