This page describes how to use Gromacs 4.6.5 on the GPU nodes of sun.chpc.ac.za.

To use Gromacs add the following lines to your PBS job script:

source /etc/profile.d/modules.sh
module add /opt/gridware/applications/gpu/modules/gromacs/4.6.5

In this section we describe how to run a simple Gromacs job. Run this test case to verify the installation is working correctly.

Retrieve the files from the installation directory:

cp -r /opt/gridware/applications/gpu/gromacs/4.6.5/chpc/test .
cd test

Optionally, execute the following sequence of commands to configure the test interactively on a GPU node:

qsub -I -q kepla_k20
cd test
grompp -f pme_verlet_vsites.mdp -c conf.gro -p topol.top -o adh_cubic_vsites_pme.tpr
logout

Below is the PBS job script for executing the job on two of the six available GPU nodes:

gromacs.pbs

#!/bin/bash
#PBS -N test_gromacs
#PBS -l select=2:ncpus=24:mpiprocs=2
#PBS -l walltime=12:00:00
#PBS -j oe
#PBS -q kepla_k20
 
source /etc/profile.d/modules.sh
module add /opt/gridware/applications/gpu/modules/gromacs/4.6.5
 
cd $PBS_O_WORKDIR
 
export OMP_NUM_THREADS=12
 
mpirun mdrun -s adh_cubic_vsites_pme.tpr -nb gpu_cpu -gpu_id 00

NOTE: The -s flag above specifies the input file to be run. The -nb flag specifies that the non-bonded force calculations should be performed on a hybrid GPU-CPU combination which currently yields the best performance. You can optionally set this to gpu if you prefer to perform the non-bonded force calculations exclusively on the GPU or to cpu if you prefer that all calculations are run on the CPU. The parameter to -gpu_id flag denotes the index of the GPU device each MPI process should use. In the above case, there is one GPU device per node and two MPI processes per node. So we specify that each MPI process use the first GPU device by setting the -gpu_id string to 00.

Submit the job using the following command:

qsub gromacs.pbs