wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » quick » pbspro
Trace:
quick:pbspro

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
quick:pbspro [2014/01/06 12:10]
andyr 		quick:pbspro [2025/09/22 17:47] (current)
kevin [Example 4 : empty job]
Line 1: Line 1:
 ====== Job submission using PBSPro ====== ====== Job submission using PBSPro ======
  
-As of January 2014, CHPC has a new scheduler - PBSPro.+The CHPC system uses the PBSPro scheduler.
  
-Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf+Main user documentation can be found at [[https://www.altair.com/pbs-works-documentation/|Altair]]
  
 But here is a summary of how to submit jobs at CHPC. But here is a summary of how to submit jobs at CHPC.
  
-The following must be **plain text** files, as created by unix programs like ''nano'' or ''vi''. If you wish to create them on your desktop and copy them over, ensure they contain no formatting and have unix line endings. Microsoft Word would be a bad choice.+=====Common PBS commands=====
  
-Example 1 : Gaussian job +  *     ''qsub'' submit a batch job 
-<code> +  *     ''qstat'' see all jobs and their status 
-#PBS -N eric.job +  *     ''qdel'' delete a job 
-#PBS -l select=8:ncpus=1:jobtype=dell,place=free:group=nodetype +  *     ''qalter'' modify options on a pending job           
-#PBS -l select=1: +  * ''qsig'' send a signal to a job (eg. to terminate it)  
-#PBS -l walltime=1:00:00 +
-#PBS -q workq +
-#PBS -m be +
-#PBS -o /export/home/embele/scratch5/eric.out +
-#PBS -e /export/home/embele/scratch5/eric.err +
-#PBS+
  
-cd $PBS_O_WORKDIR 
-source /etc/profile.d/modules.sh 
-module add gaussian/g09.D01 
-g09 < eric.com > sibusiso.log 
-</code> 
  
-Example 2 :  Amber job+====How to submit a batch job====
  
-<code> +Use the ''qsub'' command to submit a script file to the scheduler. PBS-Pro jobs can be controlled using command line switches, //or// directives in the job submission file prefaced by the ''#PBS'' keyword. 
-#PBS -N eric.job +
-#PBS -l select=1:mpiprocs=8 +
-#PBS -l walltime=1:00:00 +
-#PBS -q kepla_k20 +
-#PBS -m be +
-#PBS -o /export/home/embele/amber/amber12/Amber_GPU_Benchmark_Suite/GB/myoglobin/eric.out +
-#PBS -e /export/home/embele/amber/amber12/Amber_GPU_Benchmark_Suite/GB/myoglobin/eric.err +
-#PBS +
-cd $PBS_O_WORKDIR +
-exe=/export/home/embele/amber/amber12/bin/pmemd.MPI +
-###exe=/export/home/embele/amber/amber12/bin/pmemd.cuda.MPI +
-source /etc/profile.d/modules.sh +
-module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20 +
-module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20 +
-nproc=`cat $PBS_NODEFILE|wc -l` +
-time mpirun -np $nproc -machinefile $PBS_NODEFILE $exe -O -i mdin -o mdout -p prmtop -c inpcrd +
-</code>+
  
 +See the ''qsub'' man page for a complete list of options. 
  
-Example 3 :  empty job+The first line of a script file may specify the interpreter, eg., bash is selected by using ''#!/bin/bash'' as the first line of the job script file. 
 + 
 +A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files.  
 + 
 +> Job script files can be named to anything allowed for a regular file, but it is strongly recommended that you use ''.pbs'' as the file extension to make it easy to distinguish job scripts from other shell scripts (''.sh''). 
 + 
 + 
 +====How to start an interactive job on a compute node==== 
 + 
 +''qsub -I -V'' 
 + 
 +to start an interactive session, retaining all environment variables. 
 + 
 + 
 +====How to monitor job submissions to the queue==== 
 + 
 +The job number will be returned if the job has been accepted by the scheduler. Use this 
 +number to check on the progress of a specific job: 
 + 
 +''qstat -f <jobnum>'' 
 + 
 +To inspect jobs that have already finished: 
 + 
 +''qstat -x -f <jobnum>'' 
 + 
 +or 
 + 
 +''qstat -xf <jobnum>'' 
 + 
 +====How to monitor job arrays==== 
 + 
 +To see the array job status on the queue, postfix the 
 +jobnumber with empty square brackets: 
 + 
 +''qstat -J <jobnum>[]'' 
 + 
 + 
 +To look at one specific job in the array, add ''-t'' to the qstat arguments: 
 + 
 +''qstat -xft <jobnum>[array-index]'' 
 + 
 + 
 +eg 
 + 
 +''qstat -xft 967890[2]'' 
 + 
 + 
 + 
 +=====PBS-Pro Job Exit Codes ===== 
 + 
 +====Job Exit Codes Between 0 and 128 (or 256)==== 
 +This is the exit value of the top process in the job, typically the shell. This may be the exit value of the last command executed in the shell or the .logout script if the user has such a script (csh). 
 + 
 +====Job Exit Codes >= 128 (or 256)==== 
 +This means the job was killed by the PBS scheduler by sending it a signal.  
 + 
 +The **modulo** or **remainder** operator [[http://en.wikipedia.org/wiki/Modulo_operation]] is used to interpret (or “decode”) PBS job exit values.  
 + 
 + 
 +The signal is given by X modulo 128 (or 256). An exit value of 137 means the job's top process was killed with signal 9 because ''137 mod 128 = 9''.  
 + 
 +Depending on the system, a value greater than 128 or 256, see ''man wait(2)'', is the signal that was sent by PBS-Pro to kill the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.)  
 + 
 +Exit code 271 means that the job exceeded the limit requested at submission, eg. walltime or cpu usage, and was killed. 
 + 
 +See ''man kill'' for a mapping of signal numbers to signal name on your operating system. 
 +====Terminate a job by signal==== 
 +Send a signal to the job 
 + 
 +'' qsig -s SIGTERM  <jobnum>  '' 
 + 
 + 
 +===== Examples: PBS-Pro job submission scripts ===== 
 + 
 +[[howto:PBS-Pro_job_submission_examples]] 
 + 
 +===== Examples: Gaussian jobs ===== 
 + 
 + 
 +[[howto:gaussian|Gaussian Single and Multi-Node Scripts]] 
 +    
 +    
 + 
 + 
 + 
 + 
 + 
 +===== Examples: Amber jobs ===== 
 + 
 + 
 +[[howto:amber|Amber job submission scripts]] 
 + 
 + 
 + 
 +===== Example: Quantum Espresso ===== 
 + 
 + 
 +[[howto:quantum_espresso|Quantum Espresso]] 
 + 
 + 
 +===== Example 4 : empty job =====
  
 <code> <code>
 +#! /bin/sh
 #PBS -N <jobname> #PBS -N <jobname>
-#PBS -l select=<totalcores>:ncpus=1:select=<hosts>:jobtype=<architecture>:place=free:group=nodetype+#PBS -l select=<numberofnodes>:ncpus=<totalcores>:jobtype=<architecture>:place=free:group=nodetype
 #PBS -l walltime=<hours>:<minutes>:<seconds> #PBS -l walltime=<hours>:<minutes>:<seconds>
 #PBS -q <queue> #PBS -q <queue>
Line 59: Line 133:
 #PBS -o <path to your output file> #PBS -o <path to your output file>
 #PBS -e <path to your error file> #PBS -e <path to your error file>
-Where +source /etc/profile.d/modules.sh   # so module command works 
-<jobname> = name of the job +cd $PBS_O_WORKDIR   # change to working directory 
-<totalcores>= total number of cores + 
-<hosts> = number of hosts +#Where 
-<architecture> = architecture where the jobs will run (nehalem|westmere|dell) +#<jobname> = name of the job 
-<queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys)+#<totalcores>= total number of cores 
 +#<hosts> = number of hosts 
 +#<architecture> = architecture where the jobs will run (nehalem, westmere, dell) 
 +#<queue> = submission queue ( serial, smp, normal, ...,  accelrys) 
 +#<executable> = Binary to be run 
 + 
 +mpirun -np <totalcores> <executable>
  
 </code> </code>
-i 
-Note that the M9000 sparc queue is called 'spark' - sorry. 
  
 +
 +
 +===== Example 5 : MPI example =====
 +
 +Syntax
 +<code>
 +...
 +#PBS -l walltime=$HOURS:$MINUTES:$SECONDS
 +#PBS -l select=${NO_OF_HOSTs}:ncpus=${CPUs_PER_HOST}:mpiprocs=${CPUs_PER_HOST}:jobtype=$FEATURE
 +#PBS -q $QUEUE
 +...
 +</code>
 +===== Further examples =====
 +  * Various [[howto:bioinformatics?&#basic_examples|bioinformatics examples]] illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies.
 +  * This [[howto:gpu_gromacs#pbs_job_script|gromacs example]] shows an interesting case where mpi, openmp and the gpus are used in a single job.
/app/dokuwiki/data/attic/quick/pbspro.1389003031.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki