wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » quick » pbspro
Trace: • opensim
quick:pbspro

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
quick:pbspro [2015/05/08 12:57]
dane Added very basic job array example. 		quick:pbspro [2025/09/22 17:47] (current)
kevin [Example 4 : empty job]
Line 1: Line 1:
 ====== Job submission using PBSPro ====== ====== Job submission using PBSPro ======
  
-As of January 2014, CHPC has a new scheduler - PBSPro.+The CHPC system uses the PBSPro scheduler.
  
-Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf+Main user documentation can be found at [[https://www.altair.com/pbs-works-documentation/|Altair]]
  
 But here is a summary of how to submit jobs at CHPC. But here is a summary of how to submit jobs at CHPC.
Line 26: Line 26:
 A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files.  A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. 
  
-''qsub myscriptname.sub''+> Job script files can be named to anything allowed for a regular file, but it is strongly recommended that you use ''.pbs'' as the file extension to make it easy to distinguish job scripts from other shell scripts (''.sh'').
  
  
Line 50: Line 50:
  
 ''qstat -xf <jobnum>'' ''qstat -xf <jobnum>''
 +
 +====How to monitor job arrays====
 +
 +To see the array job status on the queue, postfix the
 +jobnumber with empty square brackets:
 +
 +''qstat -J <jobnum>[]''
 +
 +
 +To look at one specific job in the array, add ''-t'' to the qstat arguments:
 +
 +''qstat -xft <jobnum>[array-index]''
 +
 +
 +eg
 +
 +''qstat -xft 967890[2]''
 +
  
  
Line 58: Line 76:
  
 ====Job Exit Codes >= 128 (or 256)==== ====Job Exit Codes >= 128 (or 256)====
-This means the job was killed with a signal. The signal is given by X modulo 128 (or 256). For example an exit value of 137 means the job's top process was killed with signal 9 (137 mod 128 = 9). +This means the job was killed by the PBS scheduler by sending it a signal. 
-The exit status values greater than 128 (or 256) indicate which signal killed the job. Depend- ing on the system, values greater than 128 (or on some systems 256; see man wait(2) for more information), are the value of the signal that killed the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.)+
  
-To interpret (or “decode”) the signal contained in the exit status value, subtract the base value from the exit status. For example, if a job had an exit status of 143, that indicates the job was killed via a SIGTERM (e.g. 143 - 128 = 15, signal 15 is SIGTERM). See the kill(1) manual page for a mapping of signal numbers to signal name on your operating system.+The **modulo** or **remainder** operator [[http://en.wikipedia.org/wiki/Modulo_operation]] is used to interpret (or “decode”) PBS job exit values. 
  
 +
 +The signal is given by X modulo 128 (or 256). An exit value of 137 means the job's top process was killed with signal 9 because ''137 mod 128 = 9''. 
 +
 +Depending on the system, a value greater than 128 or 256, see ''man wait(2)'', is the signal that was sent by PBS-Pro to kill the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.) 
 +
 +Exit code 271 means that the job exceeded the limit requested at submission, eg. walltime or cpu usage, and was killed.
 +
 +See ''man kill'' for a mapping of signal numbers to signal name on your operating system.
 ====Terminate a job by signal==== ====Terminate a job by signal====
 Send a signal to the job Send a signal to the job
Line 84: Line 109:
  
  
-===== Example 2 (a) :  Amber job on GPU nodes (NB: For other executables e.g. python and sender, see example 2(b)) =====+===== Examples: Amber jobs =====
  
-<code> 
-#!/bin/sh 
-#PBS -N eric.job 
-#PBS -l select=1:ncpus=12:mpiprocs=12 
-#PBS -l walltime=1:00:00 
-#PBS -q kepla_k20 
-#PBS -m be 
-#PBS -o /export/home/username/amber/amber12/eric.out 
-#PBS -e /export/home/username/amber/amber12/eric.err 
-#PBS -M email@mail.com 
  
-cd $PBS_O_WORKDIR+[[howto:amber|Amber job submission scripts]]
  
-pwd 
-echo "My job starts here" 
-date 
-source /etc/profile.d/modules.sh 
-module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20 
-module add amber/k20 
  
-exe=pmemd.cuda.MPI 
-nproc=`cat $PBS_NODEFILE|wc -l` 
-time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd 
-echo "My job ends here" 
-date 
-</code> 
  
-===== Example 2 (b) :  Amber job on normal nodes (Please use this for python,sander and other executables that do not work on GPU=====+===== Example: Quantum Espresso =====
  
-<code> 
-#!/bin/sh 
-#PBS -N eric.job 
-#PBS -l select=1:ncpus=12:mpiprocs=12 
-#PBS -l walltime=1:00:00 
-#PBS -q workq 
-#PBS -m abe 
-#PBS -o /export/home/username/amber/amber12/eric.out 
-#PBS -e /export/home/username/amber/amber12/eric.err 
-#PBS -M email@mail.com 
  
-cd $PBS_O_WORKDIR+[[howto:quantum_espresso|Quantum Espresso]]
  
-pwd 
- 
-source /etc/profile.d/modules.sh 
-module add intel-XE/13.0 openmpi/openmpi-1.6.5-intel amber/12 
- 
-echo "My job starts here" 
-date 
- 
-exe=sander.MPI 
- 
-nproc=`cat $PBS_NODEFILE|wc -l` 
-time mpirun -np $nproc -machinefile $exe -O -i mdin -o mdout -p prmtop -c inpcrd 
-echo "My job ends here" 
-date 
-</code> 
- 
-===== Example 3 : Quantum Espresso job ===== 
- 
-<code> 
-#!/bin/sh 
-###These lines are for PBSpro 
-#PBS -N QEspresso 
-#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype:excl 
-#PBS -l walltime=24:00:00 
-#PBS -q workq 
-#PBS -m be 
-#PBS -o /export/home/username/scratch/run-directory/stdout 
-#PBS -e /export/home/username/scratch/run-directory/stderr 
-#PBS -V 
-##### Environment Settings 
-source /etc/profile.d/modules.sh 
-module add intel-XE/13.0 
-module add QEspresso/5.0.2 
-##### Running commands 
-NP=`cat $PBS_NODEFILE | wc -l` 
-cd $HOME/scratch5 
-mpirun -f $PBS_NODEFILE -r ssh -ib -env I_MPI_DEBUG 2 -np $NP pw.x -npool 3  -inp inputfile.inp  
-</code> 
  
 ===== Example 4 : empty job ===== ===== Example 4 : empty job =====
Line 186: Line 141:
 #<hosts> = number of hosts #<hosts> = number of hosts
 #<architecture> = architecture where the jobs will run (nehalem, westmere, dell) #<architecture> = architecture where the jobs will run (nehalem, westmere, dell)
-#<queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys)+#<queue> = submission queue ( serial, smp, normal, ...,  accelrys)
 #<executable> = Binary to be run #<executable> = Binary to be run
  
Line 193: Line 148:
 </code> </code>
  
-//Note that the M9000 sparc queue is called 'spark'//+
  
 ===== Example 5 : MPI example ===== ===== Example 5 : MPI example =====
Line 205: Line 160:
 ... ...
 </code> </code>
- 
-===== Example 6 : job array ===== 
- 
-Syntax 
-<file bash simple_job_array.pbs> 
-#! /bin/bash 
-#PBS -l select=1:ncpus=1 
-#PBS -l walltime=00:01:00 
-#PBS -o /export/home/username/scratch5/simple_jobarray.stdout 
-#PBS -e /export/home/username/scratch5/simple_jobarray.stderr 
-#PBS -M youremail@address.com 
-#PBS -m abe 
-#PBS -N J_example 
-#PBS -J 1-24 
- 
- 
-# Make sure we're in the right working directory 
-cd /export/home/username/scratch5/ 
- 
-#sleep a random number of seconds (<10) 
-sleep $[ ( $RANDOM % 10 )  + 1 ]s 
- 
-#see what happens when we write to a file... 
-echo "Wawaweewaa! This is sub-job ${PBS_ARRAY_INDEX} of job ${PBS_JOBID} running on ${HOSTNAME} run by ${USER} writing to stdout!" 
-echo "Wawaweewaa! This is sub-job ${PBS_ARRAY_INDEX} of job ${PBS_JOBID} running on ${HOSTNAME} run by ${USER} writing to stderr!" 1>&2 
-echo "Wawaweewaa! This is sub-job ${PBS_ARRAY_INDEX} of job ${PBS_JOBID} running on ${HOSTNAME} run by ${USER} appending to simple_jobarray.txt" >> simple_jobarray.txt 
-</file> 
- 
-Thing to note: the stdout and stderr outputs do not all get joined together as they might with an mpi job. 
- 
 ===== Further examples ===== ===== Further examples =====
   * Various [[howto:bioinformatics?&#basic_examples|bioinformatics examples]] illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.   * Various [[howto:bioinformatics?&#basic_examples|bioinformatics examples]] illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.
   * This [[howto:gpu_gromacs#pbs_job_script|gromacs example]] shows an interesting case where mpi, openmp and the gpus are used in a single job.   * This [[howto:gpu_gromacs#pbs_job_script|gromacs example]] shows an interesting case where mpi, openmp and the gpus are used in a single job.
/app/dokuwiki/data/attic/quick/pbspro.1431082642.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki