wiki.chpc.ac.za

GADGET-2 is an N-body simulation used to model the evolution of the Universe on cosmological distance and time scales.

The benchmark constructed contains 256 ^3 (~ 16.7 million) dark matter particles in 128 Mpc^3 volume. The initial conditions were generated using the MUSIC utility. The simulation is run from a scale factor of 0.2 (z=4) to a scale factor of 0.5 (z=1), in order to reduce total run time. Only a few output snapshots are generated to minimize IO overheads.

The runtime for this benchmark on the Dell C4130 (24 Haswell cores) is 60 minutes.

Developers Site http://www.mpa-garching.mpg.de/gadget/
HPCAC Best Practices http://www.hpcadvisorycouncil.com/pdf/GADGET-2_Best_Practices.pdf

mkdir GADGET
cd GADGET
mkdir tars
mkdir libs

Note: Intel Cluster Suite 2015 causes segmentation faults - use ICS2013 instead.

vim bashrc

Add:

source /opt/intel/bin/compilervars.sh intel64
source /opt/intel/mkl/bin/mklvars.sh intel64
export PATH=/opt/openmpi-1.8.7-intel/bin:$PATH
export LD_LIBRARY_PATH=/opt/openmpi-1.8.7-intel/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/$USER/GADGET/libs/gsl/lib:$LD_LIBRARY_PATH
export CC=icc
export CXX=icpc
export CFLAGS='-O3 -xHost'
export CXXFLAGS='-O3 -xHost'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O3 -xHost'
export CPP='icc -E'
export CXXCPP='icpc -E'

source bashrc

cd ~/GADGET/tars
wget http://gnu.afri.cc/gsl/gsl-1.16.tar.gz
tar -xf gsl-1.16.tar.gz
cd gsl-1.16
./configure --prefix=/home/$USER/GADGET/libs/gsl
make -j4
make install

cd ~/GADGET/tars
wget http://www.fftw.org/fftw-2.1.5.tar.gz
tar -xf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/home/$USER/GADGET/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi
make -j4
make install
make distclean
./configure --prefix=/home/$USER/GADGET/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi --enable-float
make -j4
make install

cd ~/GADGET

Download the pre-configured code source gadget-2.0.7.tar.gz

tar -xf Gadget-2.0.7.tar.gz
cd Gadget2 
make 

Check that the dynamic libraries (Intel, GCC and GSL) are sourced (they should be from the bashrc file sourced previously).

ldd Gadget2

cd ~/GADGET/tars

Download the benchmark here: http://www.ace.chpc.ac.za/tars/dm_128n256.tar.gz

tar -xf dm_256n128.tar.gz
mv dm_256n128 ..
cd ../dm_256n128
cp ../Gadget2/Gadget2 .

Run the benchmark using the script provided, 'after editing the number of MPI ranks'

./run.sh

The script reports the total runtime (and dumps to time.log). The number of simulation time steps is available from the timings.txt file in the output directory.

tail output/timings.txt

From here, the metric 'timesteps per hour' can be determined.