The simulation code GIZMO is a flexible, multi-method magneto-hydrodynamics+gravity code. The code lets you solve the hydrodynamic equations using a variety of different methods – whatever is best for the problem at hand. In particular, it introduces a couple of new, Lagrangian Godunov-type methods, that allow you to solve the fluid equations with a moving particle distribution that is automatically adaptive in resolution and avoids the advection errors, angular momentum conservation errors, and excessive diffusion problems that seriously limit the applicability of “adaptive mesh” (AMR) codes, while simultaneously avoiding the low-order errors inherent to simpler methods like smoothed-particle hydrodynamics (SPH). source
The benchmark constructed contains 128 ^3 (~ 2.1 million) dark matter and 128 ^3 (~ 2.1 million) gas particles in 32 Mpc^3 volume. The initial conditions were generated using the MUSIC utility. The simulation is run from a scale factor of 0.2 (z=4) to a scale factor of 0.7 (z=0.42), in order to reduce total run time. Only a few output snapshots are generated to minimize IO overheads.
Developers Site http://www.tapir.caltech.edu/~phopkins/Site/GIZMO.html
User Guide http://www.tapir.caltech.edu/~phopkins/Site/GIZMO_files/gizmo_documentation.html
mkdir GIZMO cd GIZMO mkdir tars mkdir libs
vim bashrc
Add:
source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/mkl/bin/mklvars.sh intel64 export PATH=/opt/openmpi-1.8.7-intel/bin:$PATH export PATH=/opt/openmpi-1.8.4-intel/bin:$PATH export LD_LIBRARY_PATH=/opt/openmpi-1.8.4-intel/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/home/$USER/GIZMO/libs/gsl/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/home/$USER/GIZMO/libs/hdf5/lib:$LD_LIBRARY_PATH export CC=icc export CXX=icpc export CFLAGS='-O3 -xHost -ip -no-prec-div -static-intel' export CXXFLAGS='-O3 -xHost -ip -no-prec-div -static-intel' export F77=ifort export FC=ifort export F90=ifort export FFLAGS='-O3 -xHost -ip -no-prec-div -static-intel' export CPP='icc -E' export CXXCPP='icpc -E'
source bashrc
cd ~/GIZMO/tars wget http://www.hdfgroup.org/ftp/lib-external/szip/2.1/src/szip-2.1.tar.gz tar -xf szip-2.1.tar.gz cd szip-2.1 ./configure --prefix=/home/$USER/GIZMO/libs/szip make -j4 make install
cd ~/GIZMO/tars wget http://zlib.net/zlib-1.2.8.tar.gz tar -xf zlib-1.2.8.tar.gz cd zlib-1.2.8 ./configure --prefix=/home/$USER/GIZMO/libs/zlib make make install
cd ~/GIZMO/tars wget https://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.15-patch1/src/hdf5-1.8.15-patch1.tar.gz tar -xf hdf5-1.8.15-patch1.tar.gz cd hdf5-1.8.15-patch1 ./configure --prefix=/home/$USER/GIZMO/libs/hdf5 --with-zlib=/home/mcawood/GIZMO/libs/zlib --with-szlib=/home/mcawood/GIZMO/libs/szip make -j 12 make install
cd ~/GIZMO/tars wget http://gnu.afri.cc/gsl/gsl-1.16.tar.gz tar -xf gsl-1.16.tar.gz cd gsl-1.16 ./configure --prefix=/home/$USER/GIZMO/libs/gsl make -j4 make install
cd ~/GIZMO/tars wget http://www.fftw.org/fftw-2.1.5.tar.gz tar -xf fftw-2.1.5.tar.gz cd fftw-2.1.5 ./configure --prefix=/home/$USER/GIZMO/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi make -j4 make install make clean ./configure --prefix=/home/$USER/GIZMO/libs/fftw CC=mpicc FC=mpifort --enable-type-prefix --enable-mpi --enable-float make -j4 make install
Check that the dynamic libraries (Intel, GCC and GSL) are sourced (they should be from the bashrc file sourced previously).
ldd GIZMO
Download the benchmark here: gas32n128.tar.gz
cd .. tar gas32n128.tar.gz cd gas32n128
cp ../GIZMO/GIZMO .
Run the benchmark using the script provided.
./run.sh
The script reports the total runtime (and dumps to time.log). The number of simulation time steps is available from the timings.txt file in the output directory.
tail output/timings.txt
From here, the metric 'timesteps per hour' can be determined. The runtime for this benchmark on the Dell C4130 (24 Haswell cores) is 92 minutes.