====== Gadget2 at CHPC ======

From its [[http://wwwmpa.mpa-garching.mpg.de/~volker/gadget/|website]]: "GADGET computes gravitational forces with a hierarchical tree algorithm (optionally in combination with a particle-mesh scheme for long-range gravitational forces) and represents fluids by means of smoothed particle hydrodynamics (SPH)... GADGET can therefore be used to address a wide array of astrophysically interesting problems, ranging from colliding and merging galaxies, to the formation of large-scale structure in the Universe. With the inclusion of additional physical processes such as radiative cooling and heating, GADGET can also be used to study the dynamics of the gaseous intergalactic medium, or to address star formation and its regulation by feedback processes."

Contents:
  - [[gadget-2#compiling_your_own_gadget2_code|Compiling your own Gadget2 code]]
  - [[gadget-2#running_gadget2|Running Gadget2]]
  - [[gadget-2#generating_initial_conditions|Generating Initial Conditions]]
===== Compiling your own Gadget2 code =====

=== Step 1 ===

Log in to the cluster, ''wget'' the source code and unpack, e.g. 
<code bash>
localuser@mylaptop:~ $ ssh username@lengau.chpc.ac.za
Last login: Fri Apr 01 15:00:00 2016 from 10.128.23.235
[username@login1 ~]$ wget http://wwwmpa.mpa-garching.mpg.de/gadget/gadget-2.0.7.tar.gz
[username@login1 ~]$ tar zxvf gadget-2.0.7.tar.gz
</code>

=== Step 2 ===

Navigate into the ''Gadget-2.0.7/Gadget2/'' directory - this is where the heart of the code is located. Edit the Makefile as required (note example Makefiles and parameter files located in ''Gadget-2.0.7/Gadget2/parameterfiles/''). Before compiling, be sure to source the following modules and paths (HDF5 excluded here for now):
<code bash>
module add gcc/5.1.0 chpc/openmpi/1.8.8/gcc-5.1.0
UTILS_DIR=/apps/chpc/astro/utils/gcc/v510-openmpi188
GSL_INSTALL=gsl-1.16-build
FFTW_INSTALL=fftw-2.1.5-build
export PATH=$UTILS_DIR/$GSL_INSTALL/include:$PATH
export PATH=$UTILS_DIR/$GSL_INSTALL/bin:$PATH
export PATH=$UTILS_DIR/$FFTW_INSTALL/include:$PATH
export LD_LIBRARY_PATH=$UTILS_DIR/$GSL_INSTALL/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$UTILS_DIR/$FFTW_INSTALL/lib:$LD_LIBRARY_PATH
</code>
as well as add the appropriate "SYSTYPE", e.g.
<code bash>
#--------------------------------------- Select target computer

SYSTYPE="CHPC"

ifeq ($(SYSTYPE),"CHPC")
CC       =  mpicc
OPTIMIZE =  -O3 -Wall
GSL_INCL =  -I/apps/chpc/astro/utils/gcc/v510-openmpi188/gsl-1.16-build/include
GSL_LIBS =  -L/apps/chpc/astro/utils/gcc/v510-openmpi188/gsl-1.16-build/lib -lgsl -lgslcblas
FFTW_INCL=  -I/apps/chpc/astro/utils/gcc/v510-openmpi188/fftw-2.1.5-build/include
FFTW_LIBS=  -L/apps/chpc/astro/utils/gcc/v510-openmpi188/fftw-2.1.5-build/lib
MPICHLIB =
HDF5INCL =
HDF5LIB  =
endif
</code>
Finally, to compile:
<code bash>
[username@login1 Gadget-2.0.7/Gadget2]$ make
</code>

===== Running Gadget2 =====

This assumes you have successfully compiled Gadget, have put together a parameter file for your particular simulation as well as generated the corresponding initial conditions. For now, let's consider one of the examples provided in ''Gadget-2.0.7/Gadget2/parameterfiles/'', say "LCDM gas".

=== Step 1 ===
Make sure that you create the necessary paths (and directories) for the storing of your input data (e.g. initial conditions) as well as for the saving of Gadget2 snapshots (and other information) somewhere **on the Lustre (scratch) filesystem** i.e. ''/mnt/lustre/users/username/''. Then, in ''Gadget-2.0.7/Gadget2/parameterfiles/lcdm_gas.param'' adjust the ''InitCondFile'' and ''OutputDir'' variables accordingly. If you intend to use a predetermined set of redshifts at which to write snapshots, also adjust ''OutputListFilename'' as required (usually a small file, but is regarded as input data). 

=== Step 2 ===
Create a PBS script for the job and then submit the job to the cluster queue (see [[quick:start?#example_job_scripts|our quick start guide]] for such information) as follows:
<code bash>
[username@login1 ~]$ qsub mypbsjobscript.sh
</code>


===== Generating Initial Conditions =====
If you require any help with generating your own initial conditions, please contact [[mailto:sfebruary@csir.co.za|Sean February]] or [[mailto:ccress@csir.co.za|Catherine Cress]].