wiki.chpc.ac.za

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Job submission using PBSPro ======

The CHPC system uses the PBSPro scheduler.

Main user documentation can be found at [[https://www.altair.com/pbs-works-documentation/|Altair]]

But here is a summary of how to submit jobs at CHPC.

=====Common PBS commands=====

  *     ''qsub'' submit a batch job
  *     ''qstat'' see all jobs and their status
  *     ''qdel'' delete a job
  *     ''qalter'' modify options on a pending job          
  * ''qsig'' send a signal to a job (eg. to terminate it)  


====How to submit a batch job====

Use the ''qsub'' command to submit a script file to the scheduler. PBS-Pro jobs can be controlled using command line switches, //or// directives in the job submission file prefaced by the ''#PBS'' keyword. 

See the ''qsub'' man page for a complete list of options. 

The first line of a script file may specify the interpreter, eg., bash is selected by using ''#!/bin/bash'' as the first line of the job script file.

A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. 

> Job script files can be named to anything allowed for a regular file, but it is strongly recommended that you use ''.pbs'' as the file extension to make it easy to distinguish job scripts from other shell scripts (''.sh'').


====How to start an interactive job on a compute node====

''qsub -I -V''

to start an interactive session, retaining all environment variables.


====How to monitor job submissions to the queue====

The job number will be returned if the job has been accepted by the scheduler. Use this
number to check on the progress of a specific job:

''qstat -f <jobnum>''

To inspect jobs that have already finished:

''qstat -x -f <jobnum>''

or

''qstat -xf <jobnum>''

====How to monitor job arrays====

To see the array job status on the queue, postfix the
jobnumber with empty square brackets:

''qstat -J <jobnum>[]''


To look at one specific job in the array, add ''-t'' to the qstat arguments:

''qstat -xft <jobnum>[array-index]''


eg

''qstat -xft 967890[2]''



=====PBS-Pro Job Exit Codes =====

====Job Exit Codes Between 0 and 128 (or 256)====
This is the exit value of the top process in the job, typically the shell. This may be the exit value of the last command executed in the shell or the .logout script if the user has such a script (csh).

====Job Exit Codes >= 128 (or 256)====
This means the job was killed by the PBS scheduler by sending it a signal. 

The **modulo** or **remainder** operator [[http://en.wikipedia.org/wiki/Modulo_operation]] is used to interpret (or “decode”) PBS job exit values. 


The signal is given by X modulo 128 (or 256). An exit value of 137 means the job's top process was killed with signal 9 because ''137 mod 128 = 9''. 

Depending on the system, a value greater than 128 or 256, see ''man wait(2)'', is the signal that was sent by PBS-Pro to kill the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.) 

Exit code 271 means that the job exceeded the limit requested at submission, eg. walltime or cpu usage, and was killed.

See ''man kill'' for a mapping of signal numbers to signal name on your operating system.
====Terminate a job by signal====
Send a signal to the job

'' qsig -s SIGTERM  <jobnum>  ''


===== Examples: PBS-Pro job submission scripts =====

[[howto:PBS-Pro_job_submission_examples]]

===== Examples: Gaussian jobs =====


[[howto:gaussian|Gaussian Single and Multi-Node Scripts]]
   
   





===== Examples: Amber jobs =====


[[howto:amber|Amber job submission scripts]]



===== Example: Quantum Espresso =====


[[howto:quantum_espresso|Quantum Espresso]]


===== Example 4 : empty job =====

<code>
#! /bin/sh
#PBS -N <jobname>
#PBS -l select=<numberofnodes>:ncpus=<totalcores>:jobtype=<architecture>:place=free:group=nodetype
#PBS -l walltime=<hours>:<minutes>:<seconds>
#PBS -q <queue>
#PBS -m be
#PBS -o <path to your output file>
#PBS -e <path to your error file>
source /etc/profile.d/modules.sh   # so module command works
cd $PBS_O_WORKDIR   # change to working directory

#Where
#<jobname> = name of the job
#<totalcores>= total number of cores
#<hosts> = number of hosts
#<architecture> = architecture where the jobs will run (nehalem, westmere, dell)
#<queue> = submission queue ( serial, smp, normal, ...,  accelrys)
#<executable> = Binary to be run

mpirun -np <totalcores> <executable>

</code>



===== Example 5 : MPI example =====

Syntax
<code>
...
#PBS -l walltime=$HOURS:$MINUTES:$SECONDS
#PBS -l select=${NO_OF_HOSTs}:ncpus=${CPUs_PER_HOST}:mpiprocs=${CPUs_PER_HOST}:jobtype=$FEATURE
#PBS -q $QUEUE
...
</code>
===== Further examples =====
  * Various [[howto:bioinformatics?&#basic_examples|bioinformatics examples]] illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.
  * This [[howto:gpu_gromacs#pbs_job_script|gromacs example]] shows an interesting case where mpi, openmp and the gpus are used in a single job.
wiki.chpc.ac.za

User Tools

Site Tools

Page Tools
