As of January 2014, CHPC has a new scheduler - PBSPro.

Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf

But here is a summary of how to submit jobs at CHPC.

• qsub submit a batch job
• qstat see all jobs and their status
• qdel delete a job
• qalter modify options on a pending job
• qsig send a signal to a job (eg. to terminate it)

Use the qsub command to submit a script file. PBSPro job file allows for specification of options at the beginning of the file prefaced by the #PBS delimiter followed by PBS commands. Commands will be executed in the order given in the script.

A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. See the qsub man page for a complete list of options.

qsub myscriptname

The job number will be returned if the job has been accepted by the scheduler. Use this number to check on the progress of a specific job:

qstat -f <jobnum>

To inspect jobs that have already finished:

qstat -x -f <jobnum>

This is the exit value of the top process in the job, typically the shell. This may be the exit value of the last command executed in the shell or the .logout script if the user has such a script (csh).

This means the job was killed with a signal. The signal is given by X modulo 128 (or 256). For example an exit value of 137 means the job's top process was killed with signal 9 (137 mod 128 = 9). The exit status values greater than 128 (or 256) indicate which signal killed the job. Depend- ing on the system, values greater than 128 (or on some systems 256; see man wait(2) for more information), are the value of the signal that killed the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.)

To interpret (or “decode”) the signal contained in the exit status value, subtract the base value from the exit status. For example, if a job had an exit status of 143, that indicates the job was killed via a SIGTERM (e.g. 143 - 128 = 15, signal 15 is SIGTERM). See the kill(1) manual page for a mapping of signal numbers to signal name on your operating system.

Send a signal to the job

qsig -s SIGTERM <jobnum>

See PBS-Pro Job submission examples.

Job submission scripts are shell scripts with PBS directives in comments.

PLEASE NOTE:

1. The first line should always have the hash-bang and shell path, eg. #!/bin/bash or #!/bin/sh
2. The output and error file paths (#PBS -o and #PBS -e directives) should be files in your scratch directory, or else the job will be killed automatically. (There should be a symbolic link called “scratch” in your home directory that points to the workspace on the Lustre file system.)
3. Job names (#PBS -N ) longer than 15 characters don't work

On a cluster system, run an executable on 3 nodes using all 8 cores per node with 8 MPI processes for 5 hours:

#!/bin/bash
#PBS -e scratch/error.out
#PBS -o scratch/outputfile.txt
#PBS -j oe
#PBS -l walltime=5:00:00
#PBS -l select=3:ncpus=8:mpiprocs=8
#PBS -V

source /etc/profile.d/modules.sh
module add openmpi/openmpi-1.6.5_gcc-4.7.2

cd $HOME/scratch

mpirun -np 24  /opt/gridware/bioinformatics/bin/hello

This is done using the #PBS -l select directive in the job submission script. Cluster partitions differ in how many cpu's per node there are:

• Harpertown: 8 cores per node
• Nehalem: 8 cores per node
• Westmere: 12 cores per node
• Dell: 12 cores per node

To run a 24-core job on the Dell partition, for instance, only 2 nodes are needed:

#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell:group=nodetype

The number of cores can be obtained by:

NP=`cat $PBS_NODEFILE | uniq | wc -l`

To determine the number of cores, include the following line in the submission script:

set NP = `cat $PBS_NODEFILE | wc -l`

The mpirun command can use the PBS variable $PBS_NODEFILE that contains the file name with the list of nodes:

mpirun -np $NP -machinefile $PBS_NODEFILE ~/bin/myprogram

On a non-cluster system, run an executable on 24 cores for 5 hours

#!/bin/bash
#PBS -j oe
#PBS -l walltime=5:00:00
#PBS -l ncpus=24
#PBS -V

cd $HOME/scratch5

To use the module system, add the following line to your submission script after the #PBS lines:

source /etc/profile.d/modules.sh

Then you will be able to include your required module to set up an environment:

module add openmpi/openmpi-1.6.5_gcc-4.7.2

To list modules, the MODULEPATH variables normally are set by default and can be viewed from the command line using:

module avail

For Bioinformatics users, add the additional modules in your .profile as follows:

export MODULEPATH=/opt/gridware/bioinformatics/modules:$MODULEPATH

There is no limit to the amount of time that a PBS job can run. If no time limit is specified, a default of 12 hours is assigned. Remember to check point long running jobs.

To specify a time, include the following line in the submission script:

#PBS -l walltime=hhhh:mm:ss

On cluster systems, if a job does not request all the cores on a node, it is possible another job will share the same node. To prevent this, request exclusive use.

#PBS -l place=excl

To include the shell environment in the batch job, either use the -V option with qsub or as a PBS directive

qsub -V

#PBS -V

On the CHPC cluster, Hyperthreading is disabled, except on the MIC nodes.

Use PBS Job Arrays. You can specify how many jobs to run by adding the directive

#PBS -J <range>

where range is X-Y:Z. So 2-10:2 indicates all the even jobs from 2 to 10, i.e., 2,4,6,8 and 10.

PBS defines two environment variables:

PBS_ARRAY_INDEX job array index

PBS_ARRAY_ID job array id

These variables are also defined as attributes:

array_index

array_id

Based on the job array index, different input files can be used for each job in the job array. Below is a job script example that submits two jobs, each using one processor.

NOTE: The PBS directives specify the resources that will be used by EACH individual job, NOT all the jobs together.

#!/bin/sh
#PBS -V
#PBS -l select=1:ncpus=1:mpiprocs=1,walltime=48:00
#PBS -N Job_Array_Test
#PBS -j oe -o ja.^array_index^.pbs
#PBS -J 1-2

source /etc/profile.d/modules.sh
module add openmpi/openmpi-1.6.5_gcc-4.7.2
unset echo

cd $PBS_O_WORKDIR             
echo $PBS_ARRAY_INDEX               
echo $PBS_ARRAY_ID $PBS_ARRAY_INDEX >> ja.$PBS_ARRAY_INDEX.out
echo ' '        >> ja.$PBS_ARRAY_INDEX.out
/opt/gridware/bioinformatics/bin/pi < pi.inp >> ja.$PBS_ARRAY_INDEX.out
                
exit

Look at the job status with qstat. The next to last line of the output is a comment describing the job status. Possible outputs include:

qstat -f yourjobid

• Job run at date at time on hostname

Job is running OK

• Not Running: No available resources on nodes

Job requires more memory,cores, or processors than currently available, or you have requested more resources than what physically exists on the system.

• Job held, too many failed attempts to run

Delete and resubmit job

If this persists, contact User support

• Not Running, Draining system to allow starving job to run

Job will not run because the resources are reserved for other jobs.

Try deleting the job with qdel

qdel -Wforce yourjobid

The following must be plain text files, as created by unix programs like nano or vi. If you wish to create them on your desktop and copy them over, ensure they contain no formatting and have unix line endings. Microsoft Word would be a bad choice.

Gaussian Single and Multi-Node Scripts

#!/bin/sh
#PBS -N eric.job
#PBS -l select=1:ncpus=12:mpiprocs=12
#PBS -l walltime=1:00:00
#PBS -q kepla_k20
#PBS -m be
#PBS -o /export/home/username/amber/amber12/eric.out
#PBS -e /export/home/username/amber/amber12/eric.err
#PBS -M email@mail.com

cd $PBS_O_WORKDIR

pwd
echo "My job starts here"
date
source /etc/profile.d/modules.sh
module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20
module add amber/k20

exe=pmemd.cuda.MPI
nproc=`cat $PBS_NODEFILE|wc -l`
time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd
echo "My job ends here"
date

#!/bin/sh
#PBS -N eric.job
#PBS -l select=1:ncpus=12:mpiprocs=12
#PBS -l walltime=1:00:00
#PBS -q workq
#PBS -m abe
#PBS -o /export/home/username/amber/amber12/eric.out
#PBS -e /export/home/username/amber/amber12/eric.err
#PBS -M email@mail.com

cd $PBS_O_WORKDIR

pwd

source /etc/profile.d/modules.sh
module add intel-XE/13.0 openmpi/openmpi-1.6.5-intel amber/12

echo "My job starts here"
date

exe=sander.MPI

nproc=`cat $PBS_NODEFILE|wc -l`
time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd
echo "My job ends here"
date

#!/bin/sh
###These lines are for PBSpro
#PBS -N QEspresso
#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype:excl
#PBS -l walltime=24:00:00
#PBS -q workq
#PBS -m be
#PBS -o /export/home/username/scratch/run-directory/stdout
#PBS -e /export/home/username/scratch/run-directory/stderr
#PBS -V
##### Environment Settings
source /etc/profile.d/modules.sh
module add intel-XE/13.0
module add QEspresso/5.0.2
##### Running commands
NP=`cat $PBS_NODEFILE | wc -l`
cd $HOME/scratch5
mpirun -f $PBS_NODEFILE -r ssh -ib -env I_MPI_DEBUG 2 -np $NP pw.x -npool 3  -inp inputfile.inp 

#! /bin/sh
#PBS -N <jobname>
#PBS -l select=<numberofnodes>:ncpus=<totalcores>:jobtype=<architecture>:place=free:group=nodetype
#PBS -l walltime=<hours>:<minutes>:<seconds>
#PBS -q <queue>
#PBS -m be
#PBS -o <path to your output file>
#PBS -e <path to your error file>
source /etc/profile.d/modules.sh   # so module command works
cd $PBS_O_WORKDIR   # change to working directory

#Where
#<jobname> = name of the job
#<totalcores>= total number of cores
#<hosts> = number of hosts
#<architecture> = architecture where the jobs will run (nehalem, westmere, dell)
#<queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys)
#<executable> = Binary to be run

mpirun -np <totalcores> <executable>

Note that the M9000 sparc queue is called 'spark'

Syntax

...
#PBS -l walltime=$HOURS:$MINUTES:$SECONDS
#PBS -l select=${NO_OF_HOSTs}:ncpus=${CPUs_PER_HOST}:mpiprocs=${CPUs_PER_HOST}:jobtype=$FEATURE
#PBS -q $QUEUE
...

• Various bioinformatics examples illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.
• This gromacs example shows an interesting case where mpi, openmp and the gpus are used in a single job.