wiki.chpc.ac.za

The simulation code GIZMO is a flexible, multi-method magneto-hydrodynamics+gravity code. The code lets you solve the hydrodynamic equations using a variety of different methods – whatever is best for the problem at hand. In particular, it introduces a couple of new, Lagrangian Godunov-type methods, that allow you to solve the fluid equations with a moving particle distribution that is automatically adaptive in resolution and avoids the advection errors, angular momentum conservation errors, and excessive diffusion problems that seriously limit the applicability of “adaptive mesh” (AMR) codes, while simultaneously avoiding the low-order errors inherent to simpler methods like smoothed-particle hydrodynamics (SPH). source

The benchmark constructed contains 128 ^3 (~ 2.1 million) dark matter and 128 ^3 (~ 2.1 million) gas particles in 32 Mpc^3 volume. The initial conditions were generated using the MUSIC utility. The simulation is run from a scale factor of 0.2 (z=4) to a scale factor of 0.7 (z=0.42), in order to reduce total run time. Only a few output snapshots are generated to minimize IO overheads.

Developers Site http://www.tapir.caltech.edu/~phopkins/Site/GIZMO.html
User Guide http://www.tapir.caltech.edu/~phopkins/Site/GIZMO_files/gizmo_documentation.html

mkdir GIZMO
cd GIZMO
mkdir tars
mkdir libs

'Note: Intel Cluster Suite 2015 causes segmentation faults - use ICS2013 instead.'

vim bashrc

Add:

source /opt/intel/bin/compilervars.sh intel64
source /opt/intel/mkl/bin/mklvars.sh intel64  export PATH=/opt/openmpi-1.8.7-intel/bin:$PATH
export PATH=/opt/openmpi-1.8.4-intel/bin:$PATH
export LD_LIBRARY_PATH=/opt/openmpi-1.8.4-intel/lib:$LD_LIBRARY_PATH	
export LD_LIBRARY_PATH=/home/$USER/GIZMO/libs/gsl/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/home/$USER/GIZMO/libs/hdf5/lib:$LD_LIBRARY_PATH
export CC=icc
export CXX=icpc
export CFLAGS='-O3 -xHost -ip -no-prec-div -static-intel'
export CXXFLAGS='-O3 -xHost -ip -no-prec-div -static-intel'
export F77=ifort
export FC=ifort
export F90=ifort
export FFLAGS='-O3 -xHost -ip -no-prec-div -static-intel'
export CPP='icc -E'
export CXXCPP='icpc -E'

source bashrc

cd ~/GIZMO/tars
wget http://www.hdfgroup.org/ftp/lib-external/szip/2.1/src/szip-2.1.tar.gz
tar -xf szip-2.1.tar.gz
cd szip-2.1
./configure --prefix=/home/$USER/GIZMO/libs/szip
make -j4
make install

cd ~/GIZMO/tars
wget http://zlib.net/zlib-1.2.8.tar.gz
tar -xf zlib-1.2.8.tar.gz
cd zlib-1.2.8
./configure --prefix=/home/$USER/GIZMO/libs/zlib
make 
make install

cd ~/GIZMO/tars
wget https://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.15-patch1/src/hdf5-1.8.15-patch1.tar.gz
tar -xf hdf5-1.8.15-patch1.tar.gz
cd hdf5-1.8.15-patch1
./configure --prefix=/home/$USER/GIZMO/libs/hdf5 --with-zlib=/home/mcawood/GIZMO/libs/zlib --with-szlib=/home/mcawood/GIZMO/libs/szip
make -j 12
make install

cd ~/GIZMO/tars
wget http://gnu.afri.cc/gsl/gsl-1.16.tar.gz
tar -xf gsl-1.16.tar.gz
cd gsl-1.16
./configure --prefix=/home/$USER/GIZMO/libs/gsl
make -j4
make install

cd ~/GIZMO/tars
wget http://www.fftw.org/fftw-2.1.5.tar.gz
tar -xf fftw-2.1.5.tar.gz
cd fftw-2.1.5
./configure --prefix=/home/$USER/GIZMO/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi
make -j4
make install
make clean
./configure --prefix=/home/$USER/GIZMO/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi --enable-float
make -j4
make install

cd ~/GIZMO

Download the pre-configured code source gizmo.tar.gz.

tar -xf gizmo.tar.gz
cd GIZMO
make 

Check that the dynamic libraries (Intel, GCC and GSL) are sourced (they should be from the bashrc file sourced previously).

ldd GIZMO

Download the benchmark here: http://www.ace.chpc.ac.za/tars/gas32n128.tar.gz

cd ..
tar gas32n128.tar.gz
cd gas32n128

cp ../GIZMO/GIZMO .

Run the benchmark using the script provided.

./run.sh

The script reports the total runtime (and dumps to time.log). The number of simulation time steps is available from the timings.txt file in the output directory.

tail output/timings.txt

From here, the metric 'timesteps per hour' can be determined. The runtime for this benchmark on the Dell C4130 (24 Haswell cores) is 92 minutes.