README for Linpack for QS22, version 1.0a -------------------------------------- 0. Install Cell SDK according to the SDK installation instructions. 1. Install Atlas BLAS. cd /usr/local/src wget http://downloads.sourceforge.net/math-atlas/atlas3.8.1.tar.bz2 tar -xjvf atlas3.8.1.tar.bz2 mkdir atlas_build ; cd atlas_build ../ATLAS/configure -b 64 -A 5 -V 1 -m 3200 vi Make.inc # :0,$s/=970/=cell/g # :0,$s/= gcc/= ppu-gcc/g make Note: the "make" takes a *long* time! (e.g. 12 hours) make install # The libraries are installed in /usr/local/atlas/lib 2. Install OpenMPI. Here are the steps to install OpenMPI with TCP/IP as the primary transport layer. (Slightly different configure options are needed to use Infiniband as the primary transport layer.) As root: cd /usr/local/src wget http://www.open-mpi.org/software/ompi/v1.2/downloads/openmpi-1.2.8.tar.bz2 tar -xjf openmpi-1.2.8.tar.bz2 cd openmpi-1.2.8 mkdir obj64 cd obj64 ../configure --prefix=/usr/local/openmpi64-1.2.8 --disable-mpi-f90 \ --enable-ltdl-convenience --with-platform=optimized \ CC=ppu-gcc CPP=ppu-cpp CXX=ppu-g++ CXXPP=ppu-cpp \ CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 LDFLAGS=-m64 \ --with-wrapper-cflags=-m64 --with-wrapper-ldflags=-m64 --with-wrapper-fflags=-m64 make -j2 make install ln -s /usr/local/openmpi64-1.2.8 /usr/local/openmpi echo "/usr/local/openmpi/lib" > /etc/ld.so.conf.d/openmpi.conf ldconfig # Create /etc/profile.d/openmpi.sh cat > /etc/profile.d/openmpi.sh << EOD if ! echo \${PATH} | grep -q /usr/local/openmpi/bin ; then PATH=/usr/local/openmpi/bin:\${PATH} fi EOD chmod +x /etc/profile.d/openmpi.sh # Now verify the install ... # Log out and log back in, then echo $PATH # should contain /usr/local/openmpi/bin file /usr/local/openmpi/lib/libmpi.so.0.0.0 # should say 64-bit PowerPC ompi_info # should show gobs of stuff mpirun -np 2 --host $(hostname) hostname 3. Grab the HPL 1.0a version of the hpl source from the web wget http://www.netlib.org/benchmark/hpl/hpl.tgz tar -xzvf hpl.tgz 4. Patch hpl with changes to add qs22 support and build: cd hpl patch -p 0 < hpl_qs22.patch export CELL_TOP=/opt/cell/sdk make arch=qs22 5. Configure huge pages. By default, Linpack for QS22 uses huge pages to hold the matrix and other large buffer areas, but this can be changed with settings in the makefile. To configure huge pages. HUGE_TOT=448 # Pick an appropriate value here. Each huge page is 16MB. mkdir -p /huge /sbin/sysctl -w vm.nr_hugepages=$HUGE_TOT mount -t hugetlbfs nodev /huge chmod 777 /huge You can verify that the huge pages were configured as requested with cat /proc/meminfo 6. Run. cd /hpl/bin/qs22 # Set up parameters. The HPL.dat file in hpl/bin/qs22 defines the parameters # for a set of tests to be executed. Linpack for QS22 currently imposes some # restrictions on the values for these settings. In particular: # * NB must be 128 # * L1 must be not transposed (1) # * U must be transposed (0) # * memory alignment must be 64 (or some multiple of 64) mpirun --mca btl_openib_want_fork_support 0 \ -np 1 numactl --physcpubind=0 ./xhpl : \ -np 1 numactl --physcpubind=3 ./xhpl