There is a major problem with using GNU parallel directly on the PBSPro cluster: it uses the ssh transport and not momd which means that the PBS does not know about the tasks. This prevents PBSPro from properly monitoring or cleaning up node(s) when a job ends. Especially if that job ends from error or walltime ending.

Users should use the pbsdsh command to launch multiple serial processes, especially across many nodes.

pbsdsh [ -n <vnode index> ] [-s] [-v] [-o] -- <program> [<program args>]

However, pbsdsh is very simple so it's helpful to use parallel as the task scheduler and pbsdsh to launch the executable on each node.

Say you have a shell script script.sh that you want to run repeatedly with different arguments (parameters). Store those different parameter sets, one per line, in a plain text file inputs.txt.

Then you can use GNU parallel to use pbsdsh repeatedly to launch the script with a different line of parameters across 24 cores (i.e., one node) with

parallel -j 24 pbsdsh -n {%} -- bash -l -c '"'script.sh {}'"' :::: inputs.txt

Note that this works even if you have more input lines than the available CPUs (24 in one regular compute node), because parallel -j 24 will only execute 24 at a time, and the extra jobs will be executed as cores become available.

GNU parallel will replace the {%} with the job slot number, from 1 to maximum cores (24 in the example). and {} gets replaced with a line from inputs.txt.

Pbsdsh starts a very minimal shell, so we evoke bash -l to start up a full bash shell which loads all of your startup files, so that things like modules work.

The -c option to bash is so that bash separates out the arguments correctly (otherwise they're all in a single string).

#!/bin/bash
#PBS -N p2nodes
#PBS -l select=2:ncpus=24
#PBS -P YOURPROJECT
#PBS -q normal
#PBS -l walltime=0:30:00
 
CWD=/mnt/lustre/users/$USER/example
cd $CWD
echo "Working directory: " `pwd`
 
 
SCDIR=/mnt/lustre/users/$USER/scripts
 
SCRIPT=$SCDIR/script.sh     # Script to run.
INPUTS=inputs.txt           # Each line in this file is used as arguments to ${SCRIPT}
 
parallel -j ${PBS_NCPUS} pbsdsh -n {%} -- bash -l -c '"'${SCRIPT} {}'"' :::: ${INPUTS}

GNU parallel at NCI.