wiki.chpc.ac.za

Job submission scripts are similar to shell scripts with PBS directives in comments.

PLEASE NOTE:

1. The first line should have the hash-bang and shell path, eg. #!/bin/bash or #!/bin/sh if you need to use shell variables and operations. It is not essential for a successful job submission.
2. The output and error file paths (#PBS -o and #PBS -e directives) should be valid paths to files, preferably in your scratch directory, else the job may be killed. There should be a symbolic link called “scratch” in your home directory that points to the workspace on the Lustre file system, i.e., /lustre/SCRATCH5/users/yourname, if not, you may create it. If you do not specify output and error file names, they will be created in your home directory using the job name and number, eg.: myjob.o765432 and myjob.e765432.
3. Job names (#PBS -N ) longer than 15 characters do not work. If you do not give a job name, the submission script file name will be used as the job name.

On a cluster system, run the hello mpi executable on 3 nodes using all 8 cores per node with 8 MPI processes for up to 5 hours wall time:

#!/bin/bash
#PBS -e scratch/error.out
#PBS -o scratch/outputfile.txt
#PBS -j oe
#PBS -l walltime=5:00:00
#PBS -l select=3:ncpus=8:mpiprocs=8
#PBS -V

source /etc/profile.d/modules.sh
module add openmpi/openmpi-1.6.5_gcc-4.7.2

cd $HOME/scratch

mpirun -np 24  /opt/gridware/bioinformatics/bin/hello

This is done using the #PBS -l select directive in the job submission script. Cluster partitions differ in how many cpu's per node there are:

• Harpertown: 8 cores per node
• Nehalem: 8 cores per node
• Westmere: 12 cores per node
• Dell: 12 cores per node

To run a 24-core job on the Dell partition, for instance, only 2 nodes are needed:

#PBS -l select=2:ncpus=12:mpiprocs=12:jobtype=dell:group=nodetype

The number of cores can be obtained by:

NP=`cat $PBS_NODEFILE | uniq | wc -l`

To determine the number of cores, include the following line in the submission script:

set NP = `cat $PBS_NODEFILE | wc -l`

The mpirun command can use the PBS variable $PBS_NODEFILE that contains the file name with the list of nodes:

mpirun -np $NP -machinefile $PBS_NODEFILE ~/bin/myprogram

On a non-cluster system, run an executable on 24 cores for 5 hours

#!/bin/bash
#PBS -j oe
#PBS -l walltime=5:00:00
#PBS -l ncpus=24
#PBS -V

cd $HOME/scratch5

To use the module system, add the following line to your submission script after the #PBS lines:

source /etc/profile.d/modules.sh

Then you will be able to include your required module to set up an environment:

module add openmpi/openmpi-1.6.5_gcc-4.7.2

To list modules, the MODULEPATH variables normally are set by default and can be viewed from the command line using:

module avail

For Bioinformatics users, add the additional modules in your .profile as follows:

export MODULEPATH=/opt/gridware/bioinformatics/modules:$MODULEPATH

There is no limit to the amount of time that a PBS job can run. If no time limit is specified, a default of 12 hours is assigned. Remember to check point long running jobs.

To specify a time, include the following line in the submission script:

#PBS -l walltime=hhhh:mm:ss

On cluster systems, if a job does not request all the cores on a node, it is possible another job will share the same node. To prevent this, request exclusive use.

#PBS -l place=excl

To include the shell environment in the batch job, either use the -V option with qsub or as a PBS directive

qsub -V

#PBS -V

On the CHPC cluster, Hyperthreading is disabled, except on the MIC nodes.

Use PBS Job Arrays. You can specify how many jobs to run by adding the directive

#PBS -J <range>

where range is X-Y:Z. So 2-10:2 indicates all the even jobs from 2 to 10, i.e., 2,4,6,8 and 10.

PBS defines two environment variables:

PBS_ARRAY_INDEX job array index

PBS_ARRAY_ID job array id

These variables are also defined as attributes:

array_index

array_id

Based on the job array index, different input files can be used for each job in the job array. Below is a job script example that submits two jobs, each using one processor.

NOTE: The PBS directives specify the resources that will be used by EACH individual job, NOT all the jobs together.

#!/bin/sh
#PBS -V
#PBS -l select=1:ncpus=1:mpiprocs=1,walltime=48:00
#PBS -N Job_Array_Test
#PBS -j oe -o ja.^array_index^.pbs
#PBS -J 1-2

source /etc/profile.d/modules.sh
module add openmpi/openmpi-1.6.5_gcc-4.7.2
unset echo

cd $PBS_O_WORKDIR             
echo $PBS_ARRAY_INDEX               
echo $PBS_ARRAY_ID $PBS_ARRAY_INDEX >> ja.$PBS_ARRAY_INDEX.out
echo ' '        >> ja.$PBS_ARRAY_INDEX.out
/opt/gridware/bioinformatics/bin/pi < pi.inp >> ja.$PBS_ARRAY_INDEX.out
                
exit

Look at the job status with qstat. The next to last line of the output is a comment describing the job status. Possible outputs include:

qstat -f yourjobid

• Job run at date at time on hostname

Job is running OK

• Not Running: No available resources on nodes

Job requires more memory,cores, or processors than currently available, or you have requested more resources than what physically exists on the system.

• Job held, too many failed attempts to run

Delete and resubmit job

If this persists, contact User support

• Not Running, Draining system to allow starving job to run

Job will not run because the resources are reserved for other jobs.

Try deleting the job with qdel

qdel -Wforce yourjobid

The following must be plain text files, as created by unix programs like nano or vi. If you wish to create them on your desktop and copy them over, ensure they contain no formatting and have unix line endings. Microsoft Word would be a bad choice.