As of January 2014, CHPC has a new scheduler - PBSPro.

Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf

But here is a summary of how to submit jobs at CHPC.

• qsub submit a batch job
• qstat see all jobs and their status
• qdel delete a job
• qalter modify options on a pending job

Use the qsub command to submit a script file. PBSPro job file allows for specification of options at the beginning of the file prefaced by the #PBS delimiter followed by PBS commands. Commands will be executed in the order given in the script.

A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. See the qsub man page for a complete list of options.

qsub myscriptname

On a cluster system, run an executable on 3 nodes using all 8 cores per node with 8 MPI processes for 5 hours

#!/bin/sh
#PBS -j oe
#PBS -l walltime=5:00:00
#PBS -l select=3:ncpus=8:mpiprocs=8
#PBS -V
cd $HOME/scratch5

On a non-cluster system, run an executable on 24 cores for 5 hours

#!/bin/sh
#PBS -j oe
#PBS -l walltime=5:00:00
#PBS -l ncpus=24
#PBS -V
cd $HOME/scratch5

There is no limit to the amount of time that a PBS job can run. If no time limit is specified, a default of 12 hours is assigned. Remember to check point long running jobs.

To specify a time, include the following line in the submission script:

#PBS -l walltime=hhhh:mm:ss

On cluster systems, if a job does not request all the cores on a node, it is possible another job will share the same node. To prevent this, request exclusive use.

#PBS -l place=excl

To include the shell environment in the batch job, either use the -V option with qsub or as a PBS directive

qsub -V

#PBS -V

On the CHPC cluster, Hyperthreading is disabled, except on the MIC nodes.

Use PBS Job Arrays. You can specify how many jobs to run by adding the directive

#PBS -J <range>

where range is X-Y:Z. So 1-10:2 indicates all the even jobs from 1 to 10, i.e., 2,4,6,8 and 10.

PBS defines two environment variables:

PBS_ARRAY_INDEX job array index PBS_ARRAY_ID job array id

These variables are also defined as attributes:

array_index array_id

Based on the job array index, different input files can be used for each job in the job array. Below is a job script example that submits two jobs, each using one processor.

NOTE: The PBS directives specify the resources that will be used by EACH individual job, NOT all the jobs together.

#!/bin/sh
#PBS -V
#PBS -l select=1:ncpus=1:mpiprocs=1,walltime=48:00
#PBS -N Job_Array_Test
#PBS -j oe -o ja.^array_index^.pbs
#PBS -J 1-2
cd $PBS_O_WORKDIR
#
unset echo
             
echo $PBS_ARRAY_INDEX
               
echo $PBS_ARRAY_ID $PBS_ARRAY_INDEX >> ja.$PBS_ARRAY_INDEX.out
echo ' '        >> ja.$PBS_ARRAY_INDEX.out
bin/pi < pi.inp >> ja.$PBS_ARRAY_INDEX.out
                
exit

To determine the number of cores requested, include the following line in the submission script:

set NCORES = `cat $PBS_NODEFILE | wc -l`

To determine the number of nodes requested, include the following line in the submission script:

set NNODES = `cat $PBS_NODEFILE | uniq | wc -l `

Look at the job status with qstat. The next to last line of the output is a comment describing the job status. Possible outputs include:

qstat -f yourjobid

• Job run at date at time on hostname

Job is running OK

• Not Running: No available resources on nodes

Job requires more memory,cores, or processors than currently available, or you have requested more resources than what physically exists on the system.

• Job held, too many failed attempts to run

Delete and resubmit job

If this persists, contact User support

• Not Running, Draining system to allow starving job to run

Job will not run because the resources are reserved for other jobs.

Try deleting the job with qdel

qdel -Wforce yourjobid

The following must be plain text files, as created by unix programs like nano or vi. If you wish to create them on your desktop and copy them over, ensure they contain no formatting and have unix line endings. Microsoft Word would be a bad choice.

gaussian_single.qsub

#! /bin/sh
#PBS -N eric.job
#PBS -l select=1:ncpus=12:jobtype=dell,place=free:group=nodetype
#PBS -l walltime=1:00:00
#PBS -q workq
#PBS -m abe
#PBS -o /export/home/username/scratch5/gaussian.out
#PBS -e /export/home/username/scratch5/gaussian.err
#PBS -M username@email
 
cd $PBS_O_WORKDIR
source /etc/profile.d/modules.sh
module add gaussian/g09.D01
g09 < eric.inp > eric.log

gaussian_multiple.qsub

#!/bin/bash
#PBS -N eric.job
#PBS -l select=2:ncpus=12:jobtype=dell,place=free:group=nodetype
#PBS -l walltime=12:00:00
#PBS -q workq 
#PBS -m abe
#PBS -o /export/home/embele/scratch5/eric/cow/viw.out
#PBS -e /export/home/embele/scratch5/eric/cow/viw.err
#PBS -M username@email
 
cd $PBS_O_WORKDIR
 
echo "My job start here"
date
pwd
 
source /etc/profile.d/modules.sh
module add gaussian/g09.D01
source /opt/gridware/applications/gaussian/g09/bsd/g09.profile
 
LINDA=`cat $PBS_NODEFILE | uniq | tr '\n' "," | sed 's|,$||' `
echo linda: $LINDA
cat viw.inp | sed "s/LINDA/$LINDA/" > temp$$.inp
g09 < temp$$.inp > OUTPUT_FILE.log
 
 
echo "My job ends here"
date
 
where viw.inp is my input file
 
Your input file should look like this :-
 
%chk=viw.chk
%nprocshared=12
%mem=32GB
%lindaworkers=LINDA

#! /bin/sh
#PBS -N eric.job
#PBS -l select=1:ncpus=12:mpiprocs=12
#PBS -l walltime=1:00:00
#PBS -q kepla_k20
#PBS -m be
#PBS -o /export/home/username/amber/amber12/eric.out
#PBS -e /export/home/username/amber/amber12/eric.err
#PBS -M email@mail.com

cd $PBS_O_WORKDIR

exe=$HOME/amber/amber12/bin/pmemd.MPI
###exe=$HOME/amber/amber12/bin/pmemd.cuda.MPI
source /etc/profile.d/modules.sh
module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20
module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20
nproc=`cat $PBS_NODEFILE|wc -l`
time mpirun -np $nproc -machinefile $PBS_NODEFILE $exe -O -i mdin -o mdout -p prmtop -c inpcrd

#!/bin/sh
###These lines are for PBSpro
#PBS -N QEspresso
#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype:excl
#PBS -l walltime=24:00:00
#PBS -q workq
#PBS -m be
#PBS -o /export/home/username/scratch5/run-directory/stdout
#PBS -e /export/home/username/scratch5/run-directory/stderr
#PBS -V
##### Environment Settings
source /etc/profile.d/modules.sh
module add intel-XE/13.0
module add QEspresso/5.0.2
##### Running commands
NP=`cat $PBS_NODEFILE | wc -l`
cd $HOME/scratch5
mpirun -f $PBS_NODEFILE -r ssh -ib -env I_MPI_DEBUG 2 -np $NP pw.x -npool 3  -inp inputfile.inp 

#! /bin/sh
#PBS -N <jobname>
#PBS -l select=<totalcores>:ncpus=1:select=<hosts>:jobtype=<architecture>:place=free:group=nodetype
#PBS -l walltime=<hours>:<minutes>:<seconds>
#PBS -q <queue>
#PBS -m be
#PBS -o <path to your output file>
#PBS -e <path to your error file>
source /etc/profile.d/modules.sh   # so module command works
cd $PBS_O_WORKDIR   # change to working directory
Where
<jobname> = name of the job
<totalcores>= total number of cores
<hosts> = number of hosts
<architecture> = architecture where the jobs will run (nehalem|westmere|dell)
<queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys)

Note that the M9000 sparc queue is called 'spark'

Syntax

...
#PBS -l walltime=$HOURS:$MINUTES:$SECONDS
#PBS -l select=${NO_OF_HOSTs}:ncpus=${CPUs_PER_HOST}:mpiprocs=${CPUs_PER_HOST}:jobtype=$FEATURE
#PBS -q $QUEUE
...

For more information on qsub

$ man qsub

Additional information on the job queues available

$ qstat -q

A minimal script Dell example would be

$ more mpirun_basic_example
#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell
cat $PBS_NODEFILE

submiting the script

$ qsub mpirun_basic_example

The submitted jobs output file should produce output similar to

cnode-6-33
... repeated 11 times ...
cnode-6-34
... repeated 11 times ...