wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » quick » pbspro
Trace:
quick:pbspro

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
quick:pbspro [2014/05/14 11:07]
ischeepers 		quick:pbspro [2025/09/22 17:47] (current)
kevin [Example 4 : empty job]
Line 1: Line 1:
 ====== Job submission using PBSPro ====== ====== Job submission using PBSPro ======
  
-As of January 2014, CHPC has a new scheduler - PBSPro.+The CHPC system uses the PBSPro scheduler.
  
-Main user documentation can be found at http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf+Main user documentation can be found at [[https://www.altair.com/pbs-works-documentation/|Altair]]
  
 But here is a summary of how to submit jobs at CHPC. But here is a summary of how to submit jobs at CHPC.
Line 12: Line 12:
   *     ''qstat'' see all jobs and their status   *     ''qstat'' see all jobs and their status
   *     ''qdel'' delete a job   *     ''qdel'' delete a job
-  *     ''qalter'' modify options on a pending job +  *     ''qalter'' modify options on a pending job           
 +  * ''qsig'' send a signal to a job (eg. to terminate it)  
  
  
 ====How to submit a batch job==== ====How to submit a batch job====
  
-Use the qsub command to submit a script file. PBSPro job file allows for specification of options at the beginning of the file prefaced by the #PBS delimiter followed by PBS commands. Commands will be executed in the order given in the script.+Use the ''qsub'' command to submit a script file to the scheduler. PBS-Pro jobs can be controlled using command line switches, //or// directives in the job submission file prefaced by the ''#PBS'' keyword. 
  
-A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. See the qsub man page for a complete list of options.+See the ''qsub'' man page for a complete list of options. 
  
-''qsub myscriptname''+The first line of a script file may specify the interpreter, eg., bash is selected by using ''#!/bin/bash'' as the first line of the job script file.
  
-===Example PBS scripts=== +A script can include just about anything which you could do during a terminal session such as set environment variables, change directories, and move files. 
-Job submission scripts are shell scripts with PBS directives in comments. +
  
-**PLEASE NOTE:**  +> Job script files can be named to anything allowed for a regular file, but it is strongly recommended that you use ''.pbs'' as the file extension to make it easy to distinguish job scripts from other shell scripts (''.sh'').
-  - The first line should always have the hash-bang and shell path, eg. **#!/bin/bash** or **#!/bin/sh** +
-  - The output and error file paths (#PBS -o and #PBS -e directives) should point to your scratch directory, else the job will be killed automatically. +
-  +
-On a cluster system, run an executable on 3 nodes using all 8 cores per node with 8 MPI processes for 5 hours:+
  
-<code> 
-#!/bin/bash 
-#PBS -j oe 
-#PBS -l walltime=5:00:00 
-#PBS -l select=3:ncpus=8:mpiprocs=8 
-#PBS -V 
-cd $HOME/scratch5 
-</code> 
  
-===How to specify number of nodes and cores===+====How to start an interactive job on a compute node====
  
-This is done using the ''#PBS -l select'' directive in the job submission script.  +''qsub -I -V''
-Cluster partitions differ in how many cpu's per node there are: +
-  * Harpertown: 8 cores per node +
-  * Nehalem: 8 cores per node +
-  * Westmere: 12 cores per node +
-  * Dell: 12 cores per node +
-  +
-To run a 24-core job on the Dell partition, for instance, only 2 nodes are needed, +
-hence the directive should be:+
  
-''#PBS -l select=2:ncpus=12:mpiprocs=12 +to start an interactive session, retaining all environment variables.
-''+
  
-===Arguments to MPI jobs=== 
  
-When running MPI-based jobs, the number of cores and machinefile can be obtained by:+====How to monitor job submissions to the queue====
  
-''NP=`cat $PBS_NODEFILE | wc -l` +The job number will be returned if the job has been accepted by the scheduler. Use this 
-''+number to check on the progress of a specific job:
  
-and the mpirun command uses the PBS variable $PBS_NODEFILE:+''qstat -f <jobnum>''
  
 +To inspect jobs that have already finished:
  
-''mpirun -np $NP -machinefile $PBS_NODEFILE ~/bin/myprogram +''qstat -x -f <jobnum>''
-''+
  
 +or
  
-On a non-cluster system, run an executable on 24 cores for 5 hours+''qstat -xf <jobnum>''
  
-<code> +====How to monitor job arrays====
-#!/bin/bash +
-#PBS -j oe +
-#PBS -l walltime=5:00:00 +
-#PBS -l ncpus=24 +
-#PBS -V +
-cd $HOME/scratch5 +
-</code>+
  
-===Using Modules===+To see the array job status on the queue, postfix the 
 +jobnumber with empty square brackets:
  
-To use the module system, add the following line to your submission script after +''qstat -J <jobnum>[]''
-the #PBS lines:+
  
-''source /etc/profile.d/modules.sh 
-'' 
  
-Then you will be able to include your required module to set up an environment:+To look at one specific job in the array, add ''-t'' to the qstat arguments:
  
-''module add openmpi/openmpi-1.6.5_gcc-4.7.2 +''qstat -xft <jobnum>[array-index]''
-''+
  
-To list modules, the MODULEPATH variables normally are set by default and can be viewed  
-from the command line using: 
  
-''module avail +eg
-''+
  
-For Bioinformatics users, add the additional modules in your .profile as follows:+''qstat -xft 967890[2]''
  
  
-''export MODULEPATH=/opt/gridware/bioinformatics/modules:$MODULEPATH 
-'' 
  
-===How long can a job run===+=====PBS-Pro Job Exit Codes =====
  
-There is no limit to the amount of time that a PBS job can run. If no time limit is specified, a default of 12 hours is assigned. Remember to check point long running jobs.+====Job Exit Codes Between 0 and 128 (or 256)==== 
 +This is the exit value of the top process in the job, typically the shell. This may be the exit value of the last command executed in the shell or the .logout script if the user has such a script (csh).
  
-To specify a time, include the following line in the submission script:+====Job Exit Codes >= 128 (or 256)==== 
 +This means the job was killed by the PBS scheduler by sending it a signal. 
  
-''#PBS -l walltime=hhhh:mm:ss''+The **modulo** or **remainder** operator [[http://en.wikipedia.org/wiki/Modulo_operation]] is used to interpret (or “decode”) PBS job exit values. 
  
-===Exclusive use of cluster nodes=== 
  
-On cluster systems, if a job does not request all the cores on a node, it is possible another job will share the same node. To prevent this, request exclusive use.+The signal is given by X modulo 128 (or 256). An exit value of 137 means the job's top process was killed with signal 9 because ''137 mod 128 = 9''. 
  
-''#PBS -l place=excl''+Depending on the system, a value greater than 128 or 256, see ''man wait(2)'', is the signal that was sent by PBS-Pro to kill the job. (i.e., 265 mod 256 = 9 or 271 mod 256 = 15.) 
  
-===How to export interactive session environment to job===+Exit code 271 means that the job exceeded the limit requested at submission, eg. walltime or cpu usage, and was killed.
  
-To include the shell environment in the batch job, either use the -V option with qsub or as a PBS directive+See ''man kill'' for a mapping of signal numbers to signal name on your operating system. 
 +====Terminate a job by signal==== 
 +Send a signal to the job
  
-''qsub -V'' +'' qsig -s SIGTERM  <jobnum>  ''
-         +
-''#PBS -V''+
  
-===Hyper-Threading=== 
  
-On the CHPC cluster, Hyperthreading is disabled, except on the MIC nodes.+===== Examples: PBS-Pro job submission scripts =====
  
-===Suppose I want to do a parameter study. How do I submit all these jobs with a different value of parameter?===+[[howto:PBS-Pro_job_submission_examples]]
  
-Use PBS Job Arrays. You can specify how many jobs to run by adding the directive+===== Examples: Gaussian jobs =====
  
-#PBS -J <range> 
  
-where range is X-Y:Z. So 1-10:2 indicates all the even jobs from 1 to 10, i.e., 2,4,6,8 and 10.+[[howto:gaussian|Gaussian Single and Multi-Node Scripts]] 
 +    
 +   
  
-PBS defines two environment variables: 
  
-PBS_ARRAY_INDEX     job array index 
-PBS_ARRAY_ID        job array id 
  
-These variables are also defined as attributes: 
  
-array_index 
-array_id 
  
-Based on the job array index, different input files can be used for each job in the job array. Below is a job script example that submits two jobs, each using one processor.+===== Examples: Amber jobs =====
  
-NOTE: The PBS directives specify the resources that will be used by EACH individual job, NOT all the jobs together. 
  
-<code> +[[howto:amber|Amber job submission scripts]]
-#!/bin/sh +
-#PBS -V +
-#PBS -l select=1:ncpus=1:mpiprocs=1,walltime=48:00 +
-#PBS -N Job_Array_Test +
-#PBS -j oe -o ja.^array_index^.pbs +
-#PBS -J 1-2 +
-cd $PBS_O_WORKDIR +
-# +
-unset echo +
-              +
-echo $PBS_ARRAY_INDEX +
-                +
-echo $PBS_ARRAY_ID $PBS_ARRAY_INDEX >> ja.$PBS_ARRAY_INDEX.out +
-echo ' '        >> ja.$PBS_ARRAY_INDEX.out +
-bin/pi < pi.inp >> ja.$PBS_ARRAY_INDEX.out +
-                 +
-exit +
-</code>+
  
-===How do I determine the number of cores I have requested in my job script?=== 
  
-To determine the number of cores requested, include the following line in the submission script: 
  
-''set NCORES = `cat $PBS_NODEFILE | wc -l`''+===== Example: Quantum Espresso =====
  
-===How do I determine the number of nodes I have requested in my job script?=== 
  
-To determine the number of nodes requested, include the following line in the submission script:+[[howto:quantum_espresso|Quantum Espresso]]
  
-''set NNODES  = `cat $PBS_NODEFILE | uniq | wc -l `'' 
  
-===A job submitted to the queue is not running=== +===== Example 4 : empty job =====
- +
-Look at the job status with qstat. The next to last line of the output is a comment describing the job status. Possible outputs include: +
- +
-''qstat -f yourjobid'' +
- +
-  *  Job run at date at time on hostname +
-        Job is running OK  +
- +
-  *  Not Running: No available resources on nodes +
-        Job requires more memory,cores, or processors than currently available, or you have requested more resources than what physically exists on the system.  +
- +
-  *  Job held, too many failed attempts to run +
-        Delete and resubmit job +
- +
-If this persists, contact User support  +
- +
-  *  Not Running, Draining system to allow starving job to run +
-        Job will not run because the resources are reserved for other jobs.  +
- +
-===Queued Jobs stuck in an error state (E)=== +
- +
-Try deleting the job with qdel +
- +
-''qdel -Wforce yourjobid'' +
- +
-The following must be **plain text** files, as created by unix programs like ''nano'' or ''vi''. If you wish to create them on your desktop and copy them over, ensure they contain no formatting and have unix line endings. Microsoft Word would be a bad choice. +
- +
-===== Example 1 : Gaussian job on 1 node ===== +
-<file bash gaussian_single.qsub> +
- +
-#! /bin/sh +
-#PBS -N eric.job +
-#PBS -l select=1:ncpus=12:jobtype=dell,place=free:group=nodetype +
-#PBS -l walltime=1:00:00 +
-#PBS -q workq +
-#PBS -m abe +
-#PBS -o /export/home/username/scratch/gaussian.out +
-#PBS -e /export/home/username/scratch/gaussian.err +
-#PBS -M username@email +
- +
-cd $PBS_O_WORKDIR +
-source /etc/profile.d/modules.sh +
-module add gaussian/g09.D01 +
-g09 < eric.inp > eric.log +
-</file> +
- +
-===== Example 2 : Gaussian job on more than 1 node ===== +
-<file bash gaussian_multiple.qsub> +
- +
-#!/bin/bash +
-#PBS -N eric.job +
-#PBS -l select=2:ncpus=12:jobtype=dell,place=free:group=nodetype +
-#PBS -l walltime=12:00:00 +
-#PBS -q workq  +
-#PBS -m abe +
-#PBS -o /export/home/embele/scratch/eric/cow/viw.out +
-#PBS -e /export/home/embele/scratch/eric/cow/viw.err +
-#PBS -M username@email +
- +
-cd $PBS_O_WORKDIR +
- +
-echo "My job start here" +
-date +
-pwd +
- +
-source /etc/profile.d/modules.sh +
-module add gaussian/g09.D01 +
-source /opt/gridware/applications/gaussian/g09/bsd/g09.profile +
- +
-LINDA=`cat $PBS_NODEFILE | uniq | tr '\n' "," | sed 's|,$||' ` +
-echo linda: $LINDA +
-cat viw.inp | sed "s/LINDA/$LINDA/" > temp$$.inp +
-g09 < temp$$.inp > OUTPUT_FILE.log +
- +
- +
-echo "My job ends here" +
-date +
- +
-where viw.inp is my input file +
- +
-Your input file should look like this :- +
- +
-%chk=viw.chk +
-%nprocshared=12 +
-%mem=32GB +
-%lindaworkers=LINDA +
- +
- +
- +
-</file> +
-===== Example 2 (a) :  Amber job on GPU nodes ===== +
- +
-<code> +
-#! /bin/sh +
-#PBS -N eric.job +
-#PBS -l select=1:ncpus=12:mpiprocs=12 +
-#PBS -l walltime=1:00:00 +
-#PBS -q kepla_k20 +
-#PBS -m be +
-#PBS -o /export/home/username/amber/amber12/eric.out +
-#PBS -e /export/home/username/amber/amber12/eric.err +
-#PBS -M email@mail.com +
- +
-cd $PBS_O_WORKDIR +
- +
-pwd +
-echo "My job starts here" +
-date +
-source /etc/profile.d/modules.sh +
-module add intel-XE/c8000 cuda/5.0-c8000 mvapich2/c8000 amber/k20 +
-module add amber/k20 +
- +
-exe=pmemd.cuda.MPI +
-nproc=`cat $PBS_NODEFILE|wc -l` +
-time mpirun -np $nproc -machinefile $PBS_NODEFILE $exe -O -i mdin -o mdout -p prmtop -c inpcrd +
-echo "My job ends here" +
-date +
-</code> +
- +
-===== Example 2 (b) :  Amber job on normal nodes (Please use this for python,sander and other executables that do not work on GPU===== +
- +
-<code> +
-#! /bin/sh +
-#PBS -N eric.job +
-#PBS -l select=1:ncpus=12:mpiprocs=12 +
-#PBS -l walltime=1:00:00 +
-#PBS -q workq +
-#PBS -m be +
-#PBS -o /export/home/username/amber/amber12/eric.out +
-#PBS -e /export/home/username/amber/amber12/eric.err +
-#PBS -M email@mail.com +
- +
-cd $PBS_O_WORKDIR +
- +
-pwd +
- +
-source /etc/profile.d/modules.sh +
-module add intel-XE/13.0 openmpi/openmpi-1.6.5-intel amber/12 +
- +
-echo "My job starts here" +
-date +
- +
-exe=sander.MPI +
- +
-nproc=`cat $PBS_NODEFILE|wc -l` +
-time mpirun -np $nproc -machinefile $PBS_NODEFILE $exe -O -i mdin -o mdout -p prmtop -c inpcrd +
-echo "My job ends here" +
-date +
-</code> +
- +
-===== Example 3 : Quantum Espresso job ===== +
- +
-<code> +
-#!/bin/sh +
-###These lines are for PBSpro +
-#PBS -N QEspresso +
-#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype:excl +
-#PBS -l walltime=24:00:00 +
-#PBS -q workq +
-#PBS -m be +
-#PBS -o /export/home/username/scratch/run-directory/stdout +
-#PBS -e /export/home/username/scratch/run-directory/stderr +
-#PBS -V +
-##### Environment Settings +
-source /etc/profile.d/modules.sh +
-module add intel-XE/13.0 +
-module add QEspresso/5.0.2 +
-##### Running commands +
-NP=`cat $PBS_NODEFILE | wc -l` +
-cd $HOME/scratch5 +
-mpirun -f $PBS_NODEFILE -r ssh -ib -env I_MPI_DEBUG 2 -np $NP pw.x -npool 3  -inp inputfile.inp  +
-</code> +
- +
-===== Example 4 :  empty job =====+
  
 <code> <code>
Line 373: Line 135:
 source /etc/profile.d/modules.sh   # so module command works source /etc/profile.d/modules.sh   # so module command works
 cd $PBS_O_WORKDIR   # change to working directory cd $PBS_O_WORKDIR   # change to working directory
-Where + 
-<jobname> = name of the job +#Where 
-<totalcores>= total number of cores +#<jobname> = name of the job 
-<hosts> = number of hosts +#<totalcores>= total number of cores 
-<architecture> = architecture where the jobs will run (nehalem, westmere, dell) +#<hosts> = number of hosts 
-<queue> = submission queue ( workq , priorityq , specialq , intel_mic , spark , kepla_k20 , accelrys)+#<architecture> = architecture where the jobs will run (nehalem, westmere, dell) 
 +#<queue> = submission queue ( serial, smp, normal, ...,  accelrys) 
 +#<executable> = Binary to be run 
 + 
 +mpirun -np <totalcores> <executable>
  
 </code> </code>
  
-//Note that the M9000 sparc queue is called 'spark'// 
  
-===== MPI example =====+ 
 +===== Example 5 : MPI example =====
  
 Syntax Syntax
Line 394: Line 160:
 ... ...
 </code> </code>
- +===== Further examples ===== 
-For more information on qsub +  * Various [[howto:bioinformatics?&#basic_examples|bioinformatics examples]] illustrate some further possibilities. Specifically the blast examples illustrate job arrays and job dependencies. 
-  $ man qsub +  * This [[howto:gpu_gromacs#pbs_job_script|gromacs example]] shows an interesting case where mpi, openmp and the gpus are used in a single job.
- +
-Additional information on the job queues available  +
-  $ qstat -q +
- +
-A minimal script Dell example would be  +
-  $ more mpirun_basic_example +
-  #PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell +
-  cat $PBS_NODEFILE +
- +
- +
-submiting the script +
-  $ qsub mpirun_basic_example +
-The submitted jobs output file should produce output similar to  +
-  cnode-6-33 +
-  ... repeated 11 times ... +
-  cnode-6-34 +
-  ... repeated 11 times ...+
/app/dokuwiki/data/attic/quick/pbspro.1400058448.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki