This is an old revision of the document!
Table of Contents
Job submission using PBSPro
As of January 2014, CHPC has a new scheduler - PBSPro.
Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf
But here is a summary of how to submit jobs at CHPC.
Common PBS commands
qsubsubmit a batch jobqstatsee all jobs and their statusqdeldelete a jobqaltermodify options on a pending job
How to submit a batch job
Use the qsub command to submit a script file. PBSPro job file allows for specification of options at the beginning of the file prefaced by the #PBS delimiter followed by PBS commands. Commands will be executed in the order given in the script.
A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. See the qsub man page for a complete list of options.
qsub myscriptname
Example PBS scripts
On a cluster system, run an executable on 3 nodes using all 8 cores per node with 8 MPI processes for 5 hours
#!/bin/sh #PBS -j oe #PBS -l walltime=5:00:00 #PBS -l select=3:ncpus=8:mpiprocs=8 #PBS -V cd $HOME/scratch5
On a non-cluster system, run an executable on 24 cores for 5 hours
#!/bin/sh #PBS -j oe #PBS -l walltime=5:00:00 #PBS -l ncpus=24 #PBS -V cd $HOME/scratch5
How long can a job run
There is no limit to the amount of time that a PBS job can run. If no time limit is specified, a default of 12 hours is assigned. Remember to check point long running jobs.
To specify a time, include the following line in the submission script:
#PBS -l walltime=hhhh:mm:ss
Exclusive use of cluster nodes
On cluster systems, if a job does not request all the cores on a node, it is possible another job will share the same node. To prevent this, request exclusive use.
#PBS -l place=excl
How to export interactive session environment to job
To include the shell environment in the batch job, either use the -V option with qsub or as a PBS directive
qsub -V
#PBS -V
Hyper-Threading
On the CHPC cluster, Hyperthreading is disabled, except on the MIC nodes.
Suppose I want to do a parameter study. How do I submit all these jobs with a different value of parameter?
Use PBS Job Arrays. You can specify how many jobs to run by adding the directive
#PBS -J <range>
where range is X-Y:Z. So 1-10:2 indicates all the even jobs from 1 to 10, i.e., 2,4,6,8 and 10.
PBS defines two environment variables:
PBS_ARRAY_INDEX job array index PBS_ARRAY_ID job array id
These variables are also defined as attributes:
array_index array_id
Based on the job array index, different input files can be used for each job in the job array. Below is a job script example that submits two jobs, each using one processor.
NOTE: The PBS directives specify the resources that will be used by EACH individual job, NOT all the jobs together.
#!/bin/sh
#PBS -V
#PBS -l select=1:ncpus=1:mpiprocs=1,walltime=48:00
#PBS -N Job_Array_Test
#PBS -j oe -o ja.^array_index^.pbs
#PBS -J 1-2
cd $PBS_O_WORKDIR
#
unset echo
echo $PBS_ARRAY_INDEX
echo $PBS_ARRAY_ID $PBS_ARRAY_INDEX >> ja.$PBS_ARRAY_INDEX.out
echo ' ' >> ja.$PBS_ARRAY_INDEX.out
bin/pi < pi.inp >> ja.$PBS_ARRAY_INDEX.out
exit
How do I determine the number of cores I have requested in my job script?
To determine the number of cores requested, include the following line in the submission script:
set NCORES = `cat $PBS_NODEFILE | wc -l`
How do I determine the number of nodes I have requested in my job script?
To determine the number of nodes requested, include the following line in the submission script:
set NNODES = `cat $PBS_NODEFILE | uniq | wc -l `
A job submitted to the queue is not running
Look at the job status with qstat. The next to last line of the output is a comment describing the job status. Possible outputs include:
qstat -f yourjobid
- Job run at date at time on hostname
Job is running OK
- Not Running: No available resources on nodes
Job requires more memory,cores, or processors than currently available, or you have requested more resources than what physically exists on the system.
- Job held, too many failed attempts to run
Delete and resubmit job
If this persists, contact User support
- Not Running, Draining system to allow starving job to run
Job will not run because the resources are reserved for other jobs.
Queued Jobs stuck in an error state (E)
Try deleting the job with qdel
qdel -Wforce yourjobid
The following must be plain text files, as created by unix programs like nano or vi. If you wish to create them on your desktop and copy them over, ensure they contain no formatting and have unix line endings. Microsoft Word would be a bad choice.
Example 1 : Gaussian job on 1 node
- gaussian_single.qsub
#! /bin/sh #PBS -N eric.job #PBS -l select=1:ncpus=12:jobtype=dell,place=free:group=nodetype #PBS -l walltime=1:00:00 #PBS -q workq #PBS -m abe #PBS -o /export/home/username/scratch/gaussian.out #PBS -e /export/home/username/scratch/gaussian.err #PBS -M username@email cd $PBS_O_WORKDIR source /etc/profile.d/modules.sh module add gaussian/g09.D01 g09 < eric.inp > eric.log
Example 2 : Gaussian job on more than 1 node
- gaussian_multiple.qsub
#!/bin/bash #PBS -N eric.job #PBS -l select=2:ncpus=12:jobtype=dell,place=free:group=nodetype #PBS -l walltime=12:00:00 #PBS -q workq #PBS -m abe #PBS -o /export/home/embele/scratch/eric/cow/viw.out #PBS -e /export/home/embele/scratch/eric/cow/viw.err #PBS -M username@email cd $PBS_O_WORKDIR echo "My job start here" date pwd source /etc/profile.d/modules.sh module add gaussian/g09.D01 source /opt/gridware/applications/gaussian/g09/bsd/g09.profile LINDA=`cat $PBS_NODEFILE | uniq | tr '\n' "," | sed 's|,$||' ` echo linda: $LINDA cat viw.inp | sed "s/LINDA/$LINDA/" > temp$$.inp g09 < temp$$.inp > OUTPUT_FILE.log echo "My job ends here" date where viw.inp is my input file Your input file should look like this :- %chk=viw.chk %nprocshared=12 %mem=32GB %lindaworkers=LINDA
Example 2 (a) : Amber job on GPU nodes
#! /bin/sh #PBS -N eric.job #PBS -l select=1:ncpus=12:mpiprocs=12 #PBS -l walltime=1:00:00 #PBS -q kepla_k20 #PBS -m be #PBS -o /export/home/username/amber/amber12/eric.out #PBS -e /export/home/username/amber/amber12/eric.err #PBS -M email@mail.com cd $PBS_O_WORKDIR pwd echo "My job starts here" date source /etc/profile.d/modules.sh module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20 module add amber/k20 exe=pmemd.cuda.MPI nproc=`cat $PBS_NODEFILE|wc -l` time mpirun -np $nproc -machinefile $PBS_NODEFILE $exe -O -i mdin -o mdout -p prmtop -c inpcrd echo "My job ends here" date
Example 2 (b) : Amber job on normal nodes (Please use this for python,sander and other executables that do not work on GPU
#! /bin/sh #PBS -N eric.job #PBS -l select=1:ncpus=12:mpiprocs=12 #PBS -l walltime=1:00:00 #PBS -q workq #PBS -m be #PBS -o /export/home/username/amber/amber12/eric.out #PBS -e /export/home/username/amber/amber12/eric.err #PBS -M email@mail.com cd $PBS_O_WORKDIR pwd source /etc/profile.d/modules.sh module add intel-XE/13.0 openmpi/openmpi-1.6.5-intel amber/12 echo "My job starts here" date exe=sander.MPI nproc=`cat $PBS_NODEFILE|wc -l` time mpirun -np $nproc -machinefile $PBS_NODEFILE $exe -O -i mdin -o mdout -p prmtop -c inpcrd echo "My job ends here" date
Example 3 : Quantum Espresso job
#!/bin/sh ###These lines are for PBSpro #PBS -N QEspresso #PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype:excl #PBS -l walltime=24:00:00 #PBS -q workq #PBS -m be #PBS -o /export/home/username/scratch5/run-directory/stdout #PBS -e /export/home/username/scratch5/run-directory/stderr #PBS -V ##### Environment Settings source /etc/profile.d/modules.sh module add intel-XE/13.0 module add QEspresso/5.0.2 ##### Running commands NP=`cat $PBS_NODEFILE | wc -l` cd $HOME/scratch5 mpirun -f $PBS_NODEFILE -r ssh -ib -env I_MPI_DEBUG 2 -np $NP pw.x -npool 3 -inp inputfile.inp
Example 4 : empty job
#! /bin/sh #PBS -N <jobname> #PBS -l select=<totalcores>:ncpus=1:select=<hosts>:jobtype=<architecture>:place=free:group=nodetype #PBS -l walltime=<hours>:<minutes>:<seconds> #PBS -q <queue> #PBS -m be #PBS -o <path to your output file> #PBS -e <path to your error file> source /etc/profile.d/modules.sh # so module command works cd $PBS_O_WORKDIR # change to working directory Where <jobname> = name of the job <totalcores>= total number of cores <hosts> = number of hosts <architecture> = architecture where the jobs will run (nehalem|westmere|dell) <queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys)
Note that the M9000 sparc queue is called 'spark'
MPI example
Syntax
...
#PBS -l walltime=$HOURS:$MINUTES:$SECONDS
#PBS -l select=${NO_OF_HOSTs}:ncpus=${CPUs_PER_HOST}:mpiprocs=${CPUs_PER_HOST}:jobtype=$FEATURE
#PBS -q $QUEUE
...
For more information on qsub
$ man qsub
Additional information on the job queues available
$ qstat -q
A minimal script Dell example would be
$ more mpirun_basic_example #PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell cat $PBS_NODEFILE
submiting the script
$ qsub mpirun_basic_example
The submitted jobs output file should produce output similar to
cnode-6-33 ... repeated 11 times ... cnode-6-34 ... repeated 11 times ...
