GADGET is a freely available code for cosmological N-body/SPH simulations on massively parallel computers with distributed memory. GADGET uses an explicit communication model that is implemented with the standardized MPI communication interface. The code can be run on essentially all supercomputer systems presently in use, including clusters of workstations or individual PCs. source
The benchmark constructed contains 256 ^3 (~ 16.7 million) dark matter particles in 128 Mpc^3 volume. The initial conditions were generated using the MUSIC utility. The simulation is run from a scale factor of 0.2 (z=4) to a scale factor of 0.5 (z=1), in order to reduce total run time. Only a few output snapshots are generated to minimize IO overheads.
The runtime for this benchmark on the Dell C4130 (24 Haswell cores) is 60 minutes.
Developers Site http://www.mpa-garching.mpg.de/gadget/
HPCAC Best Practices http://www.hpcadvisorycouncil.com/pdf/GADGET-2_Best_Practices.pdf
mkdir GADGET cd GADGET mkdir tars mkdir libs
vim bashrc
Add:
source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/mkl/bin/mklvars.sh intel64 export PATH=/opt/openmpi-1.8.7-intel/bin:$PATH export LD_LIBRARY_PATH=/opt/openmpi-1.8.7-intel/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/home/$USER/GADGET/libs/gsl/lib:$LD_LIBRARY_PATH export CC=icc export CXX=icpc export CFLAGS='-O3 -xHost' export CXXFLAGS='-O3 -xHost' export F77=ifort export FC=ifort export F90=ifort export FFLAGS='-O3 -xHost' export CPP='icc -E' export CXXCPP='icpc -E'
source bashrc
cd ~/GADGET/tars wget http://gnu.afri.cc/gsl/gsl-1.16.tar.gz tar -xf gsl-1.16.tar.gz cd gsl-1.16 ./configure --prefix=/home/$USER/GADGET/libs/gsl make -j4 make install
cd ~/GADGET/tars wget http://www.fftw.org/fftw-2.1.5.tar.gz tar -xf fftw-2.1.5.tar.gz cd fftw-2.1.5 ./configure --prefix=/home/$USER/GADGET/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi make -j4 make install make distclean ./configure --prefix=/home/$USER/GADGET/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi --enable-float make -j4 make install
cd ~/GADGET
Download the pre-configured code source gadget-2.0.7.tar.gz
tar -xf Gadget-2.0.7.tar.gz cd Gadget2 make
Check that the dynamic libraries (Intel, GCC and GSL) are sourced (they should be from the bashrc file sourced previously).
ldd Gadget2
cd ~/GADGET/tars
Download the benchmark here: dm_128n256.tar.gz
tar -xf dm_256n128.tar.gz mv dm_256n128 .. cd ../dm_256n128 cp ../Gadget2/Gadget2 .
Run the benchmark using the script provided, after editing the number of MPI ranks
./run.sh
The script reports the total runtime (and dumps to time.log). The number of simulation time steps is available from the timings.txt file in the output directory.
tail output/timings.txt
From here, the metric 'timesteps per hour' can be determined.