wiki.chpc.ac.za

GADGET is a freely available code for cosmological N-body/SPH simulations on massively parallel computers with distributed memory. GADGET uses an explicit communication model that is implemented with the standardized MPI communication interface. The code can be run on essentially all supercomputer systems presently in use, including clusters of workstations or individual PCs. source

The benchmark constructed contains 256 ^3 (~ 16.7 million) dark matter particles in 128 Mpc^3 volume. The initial conditions were generated using the MUSIC utility. The simulation is run from a scale factor of 0.2 (z=4) to a scale factor of 0.5 (z=1), in order to reduce total run time. Only a few output snapshots are generated to minimize IO overheads.

The runtime for this benchmark on the Dell C4130 (24 Haswell cores) is 60 minutes.

Developers Site http://www.mpa-garching.mpg.de/gadget/
HPCAC Best Practices http://www.hpcadvisorycouncil.com/pdf/GADGET-2_Best_Practices.pdf

mkdir GADGET
cd GADGET
mkdir tars
mkdir libs

vim bashrc

Add:

source /opt/intel/bin/compilervars.sh intel64
source /opt/intel/mkl/bin/mklvars.sh intel64
export PATH=/opt/openmpi-1.8.7-intel/bin:$PATH
export LD_LIBRARY_PATH=/opt/openmpi-1.8.7-intel/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/$USER/GADGET/libs/gsl/lib:$LD_LIBRARY_PATH
export CC=icc
export CXX=icpc
export CFLAGS='-O3 -xHost'
export CXXFLAGS='-O3 -xHost'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O3 -xHost'
export CPP='icc -E'
export CXXCPP='icpc -E'

source bashrc

cd ~/GADGET/tars
wget http://gnu.afri.cc/gsl/gsl-1.16.tar.gz
tar -xf gsl-1.16.tar.gz
cd gsl-1.16
./configure --prefix=/home/$USER/GADGET/libs/gsl
make -j4
make install

cd ~/GADGET/tars
wget http://www.fftw.org/fftw-2.1.5.tar.gz
tar -xf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/home/$USER/GADGET/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi
make -j4
make install
make distclean
./configure --prefix=/home/$USER/GADGET/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi --enable-float
make -j4
make install

cd ~/GADGET

Download the pre-configured code source gadget-2.0.7.tar.gz

tar -xf Gadget-2.0.7.tar.gz
cd Gadget2 
make 

Check that the dynamic libraries (Intel, GCC and GSL) are sourced (they should be from the bashrc file sourced previously).

ldd Gadget2

cd ~/GADGET/tars

Download the benchmark here: dm_128n256.tar.gz

tar -xf dm_256n128.tar.gz
mv dm_256n128 ..
cd ../dm_256n128
cp ../Gadget2/Gadget2 .

Run the benchmark using the script provided, after editing the number of MPI ranks

./run.sh

The script reports the total runtime (and dumps to time.log). The number of simulation time steps is available from the timings.txt file in the output directory.

tail output/timings.txt

From here, the metric 'timesteps per hour' can be determined.