wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » quick » pbspro
Trace: • opensim
quick:pbspro

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
quick:pbspro [2015/04/08 16:35]
ischeepers [Examples 1: Gaussian jobs on 1 node or more than 1 node] 		quick:pbspro [2025/09/22 17:47] (current)
kevin [Example 4 : empty job]
Line 1: Line 1:
 ====== Job submission using PBSPro ====== ====== Job submission using PBSPro ======
  
-As of January 2014, CHPC has a new scheduler - PBSPro.+The CHPC system uses the PBSPro scheduler.
  
-Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf+Main user documentation can be found at [[https://www.altair.com/pbs-works-documentation/|Altair]]
  
 But here is a summary of how to submit jobs at CHPC. But here is a summary of how to submit jobs at CHPC.
Line 26: Line 26:
 A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files.  A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. 
  
-''qsub myscriptname.sub''+> Job script files can be named to anything allowed for a regular file, but it is strongly recommended that you use ''.pbs'' as the file extension to make it easy to distinguish job scripts from other shell scripts (''.sh'').
  
-or+ 
 +====How to start an interactive job on a compute node====
  
 ''qsub -I -V'' ''qsub -I -V''
  
 to start an interactive session, retaining all environment variables. to start an interactive session, retaining all environment variables.
 +
 +
 +====How to monitor job submissions to the queue====
  
 The job number will be returned if the job has been accepted by the scheduler. Use this The job number will be returned if the job has been accepted by the scheduler. Use this
Line 46: Line 50:
  
 ''qstat -xf <jobnum>'' ''qstat -xf <jobnum>''
 +
 +====How to monitor job arrays====
 +
 +To see the array job status on the queue, postfix the
 +jobnumber with empty square brackets:
 +
 +''qstat -J <jobnum>[]''
 +
 +
 +To look at one specific job in the array, add ''-t'' to the qstat arguments:
 +
 +''qstat -xft <jobnum>[array-index]''
 +
 +
 +eg
 +
 +''qstat -xft 967890[2]''
 +
  
  
Line 54: Line 76:
  
 ====Job Exit Codes >= 128 (or 256)==== ====Job Exit Codes >= 128 (or 256)====
-This means the job was killed with a signal. The signal is given by X modulo 128 (or 256). For example an exit value of 137 means the job's top process was killed with signal 9 (137 mod 128 = 9). +This means the job was killed by the PBS scheduler by sending it a signal. 
-The exit status values greater than 128 (or 256) indicate which signal killed the job. Depend- ing on the system, values greater than 128 (or on some systems 256; see man wait(2) for more information), are the value of the signal that killed the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.)+
  
-To interpret (or “decode”) the signal contained in the exit status value, subtract the base value from the exit status. For example, if a job had an exit status of 143, that indicates the job was killed via a SIGTERM (e.g. 143 - 128 = 15, signal 15 is SIGTERM). See the kill(1) manual page for a mapping of signal numbers to signal name on your operating system.+The **modulo** or **remainder** operator [[http://en.wikipedia.org/wiki/Modulo_operation]] is used to interpret (or “decode”) PBS job exit values. 
  
 +
 +The signal is given by X modulo 128 (or 256). An exit value of 137 means the job's top process was killed with signal 9 because ''137 mod 128 = 9''. 
 +
 +Depending on the system, a value greater than 128 or 256, see ''man wait(2)'', is the signal that was sent by PBS-Pro to kill the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.) 
 +
 +Exit code 271 means that the job exceeded the limit requested at submission, eg. walltime or cpu usage, and was killed.
 +
 +See ''man kill'' for a mapping of signal numbers to signal name on your operating system.
 ====Terminate a job by signal==== ====Terminate a job by signal====
 Send a signal to the job Send a signal to the job
Line 80: Line 109:
  
  
-===== Example 2 (a) :  Amber job on GPU nodes (NB: For other executables e.g. python and sender, see example 2(b)) =====+===== Examples: Amber jobs =====
  
-<code> 
-#!/bin/sh 
-#PBS -N eric.job 
-#PBS -l select=1:ncpus=12:mpiprocs=12 
-#PBS -l walltime=1:00:00 
-#PBS -q kepla_k20 
-#PBS -m be 
-#PBS -o /export/home/username/amber/amber12/eric.out 
-#PBS -e /export/home/username/amber/amber12/eric.err 
-#PBS -M email@mail.com 
  
-cd $PBS_O_WORKDIR+[[howto:amber|Amber job submission scripts]]
  
-pwd 
-echo "My job starts here" 
-date 
-source /etc/profile.d/modules.sh 
-module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20 
-module add amber/k20 
  
-exe=pmemd.cuda.MPI 
-nproc=`cat $PBS_NODEFILE|wc -l` 
-time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd 
-echo "My job ends here" 
-date 
-</code> 
  
-===== Example 2 (b) :  Amber job on normal nodes (Please use this for python,sander and other executables that do not work on GPU=====+===== Example: Quantum Espresso =====
  
-<code> 
-#!/bin/sh 
-#PBS -N eric.job 
-#PBS -l select=1:ncpus=12:mpiprocs=12 
-#PBS -l walltime=1:00:00 
-#PBS -q workq 
-#PBS -m abe 
-#PBS -o /export/home/username/amber/amber12/eric.out 
-#PBS -e /export/home/username/amber/amber12/eric.err 
-#PBS -M email@mail.com 
  
-cd $PBS_O_WORKDIR+[[howto:quantum_espresso|Quantum Espresso]]
  
-pwd 
  
-source /etc/profile.d/modules.sh +===== Example 4 : empty job =====
-module add intel-XE/13.0 openmpi/openmpi-1.6.5-intel amber/12 +
- +
-echo "My job starts here" +
-date +
- +
-exe=sander.MPI +
- +
-nproc=`cat $PBS_NODEFILE|wc -l` +
-time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd +
-echo "My job ends here" +
-date +
-</code> +
- +
-===== Example 3 : Quantum Espresso job ===== +
- +
-<code> +
-#!/bin/sh +
-###These lines are for PBSpro +
-#PBS -N QEspresso +
-#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype:excl +
-#PBS -l walltime=24:00:00 +
-#PBS -q workq +
-#PBS -m be +
-#PBS -o /export/home/username/scratch/run-directory/stdout +
-#PBS -e /export/home/username/scratch/run-directory/stderr +
-#PBS -V +
-##### Environment Settings +
-source /etc/profile.d/modules.sh +
-module add intel-XE/13.0 +
-module add QEspresso/5.0.2 +
-##### Running commands +
-NP=`cat $PBS_NODEFILE | wc -l` +
-cd $HOME/scratch5 +
-mpirun -f $PBS_NODEFILE -r ssh -ib -env I_MPI_DEBUG 2 -np $NP pw.x -npool 3  -inp inputfile.inp  +
-</code> +
- +
-===== Example 4 :  empty job =====+
  
 <code> <code>
Line 182: Line 141:
 #<hosts> = number of hosts #<hosts> = number of hosts
 #<architecture> = architecture where the jobs will run (nehalem, westmere, dell) #<architecture> = architecture where the jobs will run (nehalem, westmere, dell)
-#<queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys)+#<queue> = submission queue ( serial, smp, normal, ...,  accelrys)
 #<executable> = Binary to be run #<executable> = Binary to be run
  
Line 189: Line 148:
 </code> </code>
  
-//Note that the M9000 sparc queue is called 'spark'// 
  
-===== MPI example =====+ 
 +===== Example 5 : MPI example =====
  
 Syntax Syntax
Line 201: Line 160:
 ... ...
 </code> </code>
- 
 ===== Further examples ===== ===== Further examples =====
   * Various [[howto:bioinformatics?&#basic_examples|bioinformatics examples]] illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.   * Various [[howto:bioinformatics?&#basic_examples|bioinformatics examples]] illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.
   * This [[howto:gpu_gromacs#pbs_job_script|gromacs example]] shows an interesting case where mpi, openmp and the gpus are used in a single job.   * This [[howto:gpu_gromacs#pbs_job_script|gromacs example]] shows an interesting case where mpi, openmp and the gpus are used in a single job.
/app/dokuwiki/data/attic/quick/pbspro.1428503733.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki