This is an old revision of the document!
Table of Contents
Job submission using PBSPro
As of January 2014, CHPC has a new scheduler - PBSPro.
Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf
But here is a summary of how to submit jobs at CHPC.
Common PBS commands
qsubsubmit a batch jobqstatsee all jobs and their statusqdeldelete a jobqaltermodify options on a pending jobqsigsend a signal to a job (eg. to terminate it)
How to submit a batch job
Use the qsub command to submit a script file to the scheduler. PBS-Pro jobs can be controlled using command line switches, or directives in the job submission file prefaced by the #PBS keyword.
See the qsub man page for a complete list of options.
The first line of a script file may specify the interpreter, eg., bash is selected by using #!/bin/bash as the first line of the job script file.
A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files.
qsub myscriptname.sub
How to start an interactive job on a compute node
qsub -I -V
to start an interactive session, retaining all environment variables.
How to monitor job submissions to the queue
The job number will be returned if the job has been accepted by the scheduler. Use this number to check on the progress of a specific job:
qstat -f <jobnum>
To inspect jobs that have already finished:
qstat -x -f <jobnum>
or
qstat -xf <jobnum>
PBS-Pro Job Exit Codes
Job Exit Codes Between 0 and 128 (or 256)
This is the exit value of the top process in the job, typically the shell. This may be the exit value of the last command executed in the shell or the .logout script if the user has such a script (csh).
Job Exit Codes >= 128 (or 256)
This means the job was killed with a signal. The signal is given by X modulo 128 (or 256). For example an exit value of 137 means the job's top process was killed with signal 9 (137 mod 128 = 9). The exit status values greater than 128 (or 256) indicate which signal killed the job. Depend- ing on the system, values greater than 128 (or on some systems 256; see man wait(2) for more information), are the value of the signal that killed the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.)
To interpret (or “decode”) the signal contained in the exit status value, subtract the base value from the exit status. For example, if a job had an exit status of 143, that indicates the job was killed via a SIGTERM (e.g. 143 - 128 = 15, signal 15 is SIGTERM). See the kill(1) manual page for a mapping of signal numbers to signal name on your operating system.
Terminate a job by signal
Send a signal to the job
qsig -s SIGTERM <jobnum>
Examples: PBS-Pro job submission scripts
Examples: Gaussian jobs
Example 2 (a) : Amber job on GPU nodes (NB: For other executables e.g. python and sender, see example 2(b))
#!/bin/sh #PBS -N eric.job #PBS -l select=1:ncpus=12:mpiprocs=12 #PBS -l walltime=1:00:00 #PBS -q kepla_k20 #PBS -m be #PBS -o /export/home/username/amber/amber12/eric.out #PBS -e /export/home/username/amber/amber12/eric.err #PBS -M email@mail.com cd $PBS_O_WORKDIR pwd echo "My job starts here" date source /etc/profile.d/modules.sh module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20 module add amber/k20 exe=pmemd.cuda.MPI nproc=`cat $PBS_NODEFILE|wc -l` time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd echo "My job ends here" date
Example 2 (b) : Amber job on normal nodes (Please use this for python,sander and other executables that do not work on GPU
#!/bin/sh #PBS -N eric.job #PBS -l select=1:ncpus=12:mpiprocs=12 #PBS -l walltime=1:00:00 #PBS -q workq #PBS -m abe #PBS -o /export/home/username/amber/amber12/eric.out #PBS -e /export/home/username/amber/amber12/eric.err #PBS -M email@mail.com cd $PBS_O_WORKDIR pwd source /etc/profile.d/modules.sh module add intel-XE/13.0 openmpi/openmpi-1.6.5-intel amber/12 echo "My job starts here" date exe=sander.MPI nproc=`cat $PBS_NODEFILE|wc -l` time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd echo "My job ends here" date
Example 3 : Quantum Espresso job
#!/bin/sh ###These lines are for PBSpro #PBS -N QEspresso #PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype:excl #PBS -l walltime=24:00:00 #PBS -q workq #PBS -m be #PBS -o /export/home/username/scratch/run-directory/stdout #PBS -e /export/home/username/scratch/run-directory/stderr #PBS -V ##### Environment Settings source /etc/profile.d/modules.sh module add intel-XE/13.0 module add QEspresso/5.0.2 ##### Running commands NP=`cat $PBS_NODEFILE | wc -l` cd $HOME/scratch5 mpirun -f $PBS_NODEFILE -r ssh -ib -env I_MPI_DEBUG 2 -np $NP pw.x -npool 3 -inp inputfile.inp
Example 4 : empty job
#! /bin/sh #PBS -N <jobname> #PBS -l select=<numberofnodes>:ncpus=<totalcores>:jobtype=<architecture>:place=free:group=nodetype #PBS -l walltime=<hours>:<minutes>:<seconds> #PBS -q <queue> #PBS -m be #PBS -o <path to your output file> #PBS -e <path to your error file> source /etc/profile.d/modules.sh # so module command works cd $PBS_O_WORKDIR # change to working directory #Where #<jobname> = name of the job #<totalcores>= total number of cores #<hosts> = number of hosts #<architecture> = architecture where the jobs will run (nehalem, westmere, dell) #<queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys) #<executable> = Binary to be run mpirun -np <totalcores> <executable>
Note that the M9000 sparc queue is called 'spark'
Example 5 : MPI example
Syntax
...
#PBS -l walltime=$HOURS:$MINUTES:$SECONDS
#PBS -l select=${NO_OF_HOSTs}:ncpus=${CPUs_PER_HOST}:mpiprocs=${CPUs_PER_HOST}:jobtype=$FEATURE
#PBS -q $QUEUE
...
Example 6 : job array
Syntax
- simple_job_array.pbs
#! /bin/bash #PBS -l select=1:ncpus=1 #PBS -l walltime=00:01:00 #PBS -o /export/home/username/scratch5/simple_jobarray.stdout #PBS -e /export/home/username/scratch5/simple_jobarray.stderr #PBS -M youremail@address.com #PBS -m abe #PBS -N J_example #PBS -J 1-24 # Make sure we're in the right working directory cd /export/home/username/scratch5/ #sleep a random number of seconds (<10) sleep $[ ( $RANDOM % 10 ) + 1 ]s #see what happens when we write to a file... echo "Wawaweewaa! This is sub-job ${PBS_ARRAY_INDEX} of job ${PBS_JOBID} running on ${HOSTNAME} run by ${USER} writing to stdout!" echo "Wawaweewaa! This is sub-job ${PBS_ARRAY_INDEX} of job ${PBS_JOBID} running on ${HOSTNAME} run by ${USER} writing to stderr!" 1>&2 echo "Wawaweewaa! This is sub-job ${PBS_ARRAY_INDEX} of job ${PBS_JOBID} running on ${HOSTNAME} run by ${USER} appending to simple_jobarray.txt" >> simple_jobarray.txt
Thing to note: the stdout and stderr outputs do not all get joined together as they might with an mpi job.
Further examples
- Various bioinformatics examples illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.
- This gromacs example shows an interesting case where mpi, openmp and the gpus are used in a single job.
