wiki.chpc.ac.za

GIZMO version 0.5 is an N-body simulation which is an evolution of the Gadget-2 code, originally released in 2005. This code adds new models and allows for optional physics modules.

The benchmark constructed contains 128 ^3 (~ 2.1 million) dark matter and 128 ^3 (~ 2.1 million) gas particles in 32 Mpc^3 volume. The initial conditions were generated using the MUSIC utility. The simulation is run from a scale factor of 0.2 (z=4) to a scale factor of 0.7 (z=0.42), in order to reduce total run time. Only a few output snapshots are generated to minimize IO overheads.

Developers Site [http://www.tapir.caltech.edu/~phopkins/Site/GIZMO.html] <br> User Guide [http://www.tapir.caltech.edu/~phopkins/Site/GIZMO_files/gizmo_documentation.html]

mkdir GIZMO
cd GIZMO
mkdir tars
mkdir libs

'Note: Intel Cluster Suite 2015 causes segmentation faults - use ICS2013 instead.'

vim bashrc

Add:

source /opt/intel/bin/compilervars.sh intel64
source /opt/intel/mkl/bin/mklvars.sh intel64  export PATH=/opt/openmpi-1.8.7-intel/bin:$PATH
export PATH=/opt/openmpi-1.8.4-intel/bin:$PATH
export LD_LIBRARY_PATH=/opt/openmpi-1.8.4-intel/lib:$LD_LIBRARY_PATH	
export LD_LIBRARY_PATH=/home/$USER/GIZMO/libs/gsl/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/$USER/GIZMO/libs/hdf5/lib:$LD_LIBRARY_PATH
export CC=icc
export CXX=icpc
export CFLAGS='-O3 -xHost -ip -no-prec-div -static-intel'
export CXXFLAGS='-O3 -xHost -ip -no-prec-div -static-intel'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O3 -xHost -ip -no-prec-div -static-intel'
export CPP='icc -E'
export CXXCPP='icpc -E'

source bashrc

cd ~/GIZMO/tars
wget http://www.hdfgroup.org/ftp/lib-external/szip/2.1/src/szip-2.1.tar.gz
tar -xf szip-2.1.tar.gz
cd szip-2.1
./configure --prefix=/home/$USER/GIZMO/libs/szip
make -j4
make install

cd ~/GIZMO/tars
wget http://zlib.net/zlib-1.2.8.tar.gz
tar -xf zlib-1.2.8.tar.gz
cd zlib-1.2.8
./configure --prefix=/home/$USER/GIZMO/libs/zlib
make 
make install

cd ~/GIZMO/tars
wget https://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.15-patch1/src/hdf5-1.8.15-patch1.tar.gz
tar -xf hdf5-1.8.15-patch1.tar.gz
cd hdf5-1.8.15-patch1
./configure --prefix=/home/$USER/GIZMO/libs/hdf5 --with-zlib=/home/mcawood/GIZMO/libs/zlib --with-szlib=/home/mcawood/GIZMO/libs/szip
make -j 12
make install

cd ~/GIZMO/tars
wget http://gnu.afri.cc/gsl/gsl-1.16.tar.gz
tar -xf gsl-1.16.tar.gz
cd gsl-1.16
./configure --prefix=/home/$USER/GIZMO/libs/gsl
make -j4
make install

cd ~/GIZMO/tars
wget http://www.fftw.org/fftw-2.1.5.tar.gz
tar -xf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/home/$USER/GIZMO/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi
make -j4
make install
make clean
./configure --prefix=/home/$USER/GIZMO/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi --enable-float
make -j4
make install

cd ~/GIZMO

Download the pre-configured code source gizmo.tar.gz.

tar -xf gizmo.tar.gz
cd GIZMO
make 

Check that the dynamic libraries (Intel, GCC and GSL) are sourced (they should be from the bashrc file sourced previously).

ldd GIZMO

Download the benchmark here: http://www.ace.chpc.ac.za/tars/gas32n128.tar.gz

cd ..
tar gas32n128.tar.gz
cd gas32n128

cp ../GIZMO/GIZMO .

Run the benchmark using the script provided.

./run.sh

The script reports the total runtime (and dumps to time.log). The number of simulation time steps is available from the timings.txt file in the output directory.

tail output/timings.txt

From here, the metric 'timesteps per hour' can be determined. The runtime for this benchmark on the Dell C4130 (24 Haswell cores) is 92 minutes.