Collective Communication Module
Collectives Made Easy

Two talks with a common theme

• Talks:
  • Last PTOOLS talk for UCSD/SDSC: IMPRINT
  • Today's talk: Collective Communication Module: Collectives Made Easy
• Theme: Make it easy

Coherent Cognition

• First instantiation started in 1985
  • Reason - provide extra income while attending UCSD until 1987
  • Product - Numerical Recipes package written in Pascal
  • People: Dr. Alice Vargas, Tim Kaiser
• Second instantiation started in 2002
  • Reason - work on projects (mainly, DoD funded) that can not be done at UCSD/SDSC
  • People: Dr. Alice Vargas, Dr. Tim Kaiser
  • Contracts:
    • Collective Communications Module (Easy MPI style collectives)
    • Fortran Portability Toolkit? (make moving between platforms easy)

The Problem

• MPI provides a rich set of collective communications routines but:
  • Too general
  • Too complicated
  • Not very compatible with Fortran 90 calling conventions
  • Not communicative
• There are other APIs (SHMEM, Co-Array Fortran, LAPI) but:
  • Not portable
  • Don't have robust easy to use collectives

The Solution
Collective Communications Module

• Develop a Fortran 90 module for commonly used collective communications operations
  • API
    • Make it easy
    • Make it "trans"portable
  • Designed to be used with/without other communications packages
  • Reference implementations in MPI and SHMEM
  • Test and timing suite
• CCM_MOD is available today soon
  • http://coherentcognition.com/~ccm/api/index.html
  • API and documentation
  • MPI reference implementation
  • Test suite
  • SHMEM implementation needs additional testing

The CCM user callable routines

Things were done "differently"

• Used many Fortran 90 specific technologies
• Fewer arguments: much simpler to use
• Compile time error checking
• Runtime error checking and tracing
• Utility functions
• More robust Barrier and checkin routine
• API document has good examples

CCM Uses Fortran 90 technologies

• Subroutine overloading is used to map input data type to routine
• Can get length of arrays inside of a routine
• Subroutines can have optional arguments

Fewer arguments: much simpler to use

• Size(s) not an argument
• Data type not an argument
• Error parameter is optional
• Root is optional
• Communicator not an argument

CCM has Compile time error checking

• Check to see that types match
• Check to see that ranks input make sense

CCM has Runtime error checking

• Optional because it is expensive
• Checks that the sizes of the arrays make sense
• Checks that send and receive sizes match
• Checks the operation for reduce
• Optional, automatic detailed error messages
• Optional tracing, print a message on entry to a routine

CCM has more useful utility functions

• ccm_info: returns information about the library, data types and the like
• ccm_testing: set error testing, tracing
• ccm_time: return time, down to millisecond
• ccm_unique: returns a unique string based on myid
• ccm_print_warning: prints the last warning message that was generated

CCM has more robust Barrier and Checkin routines

• ccm_barrier
  • Like MPI barrier
  • Optional: sleep for some number of seconds
  • Optional: Flush IO
• ccm_checkin
  • wait a given number of seconds for all tasks to reach a point
    and signal a timeout (print warning message)
  • Unlike barriers, checkin points can have names

CCM API document has good examples

• Shows calls in simplest form
• Shows calls with all optional arguments
• Problem with most libraries is that examples are little more than one line
• CCM_MOD API documentation at one or two complete programs for each routine
  • Most are short 10 lines or so
  • Example output is given for each program
  • Programs can be cut from the web page and run
  • Scatter and gather example is very extensive, and useful

Hello world using CCM_MOD and MPI

program ccm_init_x1 
  use ccm_mod
  call ccm_init(myid,nodes)
  write(*,*)"hello: ",myid," of ",nodes
  call ccm_close()
end program

program ccm_init_x1
  include "mpif.h"
  call mpi_init(ierr)
  call mpi_com_size(mpi_comm_world,nodes,ierr)
  call mpi_com_rank(mpi_comm_world,myid,ierr)
  write(*,*)"hello: ",myid," of ",nodes
  call mpi_finalize(ierr)
end program

• Similar, but even here CCM_MOD is a little simpler

Let's look at...

• The API/Reference manual
• Some examples compared to MPI
• How do you get it?
• Running all of the examples
• Some notes about the distribution

The API/Reference manual

• The routines
• Minimum call requirements and Required Arguments
• Optional Arguments and Specifying Optional Arguments
• Notes
• Examples
• Error conditions
• For example take ccm_scatterv

Some examples compared to MPI

• BCAST
• REDUCE
• SCATTER
• ALLTOALLV
• CHECKIN

How do you get it?

• API document:
  • http://coherentcognition.com/~ccm/api/index.html
• MPI reference implementation (includes examples and tests)
  • coherentcognition.com/~ccm/mpi_ref.tar
• Questions:
  • tkaiser@coherentcognition.com

How do you build and use it?

• Untar the distribution
• Make file for AIX, Apple OSX, SGI, T3e
  • setenv CCM_COM for include file
• Makefile builds a library and ccm_mod file
• Need to point your compiler to ccm_mod
• Need to point your linker to libccm.a

How do you run all of the examples?

• Make as shown above
• Go to the examples directory (exam)
• Compile and run
  • On machines that have mpprun:
    • do_all
    • This will compile and run on 4 processors
  • On other machines
    • make
    • Run using machine's version of mpprun

A digression:
A Blue Horizon (IBM SP) mapping to mpprun

Create mpprun:

#!/bin/csh -f
poe $3 -nodes 1 -tasks_per_node $2 -rmpool 1 -euilib ip -euidevice en0


Tar file is in ~tkaiser/mpi_ref.tar

Interactive nodes: ssh b80login.sdsc.edu

Some notes about the distribution

• There is a generic source for most routines
  • Preprocessor takes *.input -> *.f90
  • Supplements Fortran 90, lack of template capability
  • Helps in program correctness and maintainability
  • Can build for any data type and rank of arrays
  • The header for ccm_scatter_mod.input file is:
    

    Header	Meaning
    .false. scatter 1 3 0 3 sp real(b4) myreal put32 dp real(b8) mydouble put64 in integer(def_int) myint integer_put comp complex(c4) mycomp put64 dpcomp complex(c8) mydpcomp put128 logical logical mylogical logical_put character character mycharacter logical_put	Input/Output ranks must match Generic name for the routine Build for Input/Output ranks One line for each data type: suffix for routine name data type variable name for data type SHMEM routine name for this type

• Testing should be done against same data types and ranks
• Two sets of tests
  • Basic functionality
  • Error messages

SHMEM Reference Implementation

• Challenges
  • SHMEM is not standard across T3e, SV1, SGI
  • Different calls available on all three
  • Different data types
  • There may be some undocumented calls
  • Lack of atomic operations on SGI set me back some
• Status
  • Routines share a large amount with MPI reference Implementation
  • Recently rebuilt all operations in terms of CCM_SEND and CCM_RECV operations
  • Passes test suite

What still needs to be done?

• Additional testing of SHMEM implementation
• Define behavior for unallocated arrays
• Improve testing suite
• Timing suite
• Notes for implementers
• Notes for building the reference implementation

Should this be a PTOOL's Project?

• Designed to make life easier for parallel programmers
• Works on multiple platforms
• Open source
• Sponsor wants to see in used and adapted
• Other good things that may come out of this:
  • A prototype that might be adapted by vendors to build more robust implementations
  • Good starting points for student projects
  • Affect future versions of MPI
  • Affect future versions of Fortran
  • The preprocessor could be used for other projects

Summary

• Developed a Fortran 90 module for commonly used collective communications operations
• Designed to be easy to use
• CCM_MOD is available today at http://coherentcognition.com/~ccm/api/index.html
• We desire comments and suggestions
• PTOOL's project?

Feedback to :

• Tim Kaiser
• tkaiser@coherentcognition.com