======pbsdsh======

There is a major problem with using GNU ''parallel'' directly on the PBSPro cluster: it uses the ssh transport and not momd which means that the PBS does not know about the tasks.  This prevents PBSPro from properly monitoring or cleaning up node(s) when a job ends.  Especially if that job ends from error or walltime ending.

Users should use the ''pbsdsh'' command to launch multiple serial processes, especially across many nodes.

=====Syntax=====

''pbsdsh'' [ ''-n'' //<vnode index>// ] [''-s''] [''-v''] [''-o''] ''-''''-'' //<program>// [//<program args>//]


=====With GNU parallel=====

However, ''pbsdsh'' is very simple so it's helpful to use ''parallel'' as the task scheduler and ''pbsdsh'' to launch the executable on each node.

====Example 1====

Say you have a shell script ''script.sh'' that you want to run repeatedly with different arguments (parameters).  Store those different parameter sets, one per line, in a plain text file ''inputs.txt''.

Then you can use GNU ''parallel'' to use ''pbsdsh'' repeatedly to launch the script with a different line of parameters across 24 cores (i.e., one node) with

<code bash>
parallel -j 24 pbsdsh -n {%} -- bash -l -c '"'script.sh {}'"' :::: inputs.txt
</code>

Note that this works even if you have more input lines than the available CPUs (24 in one regular compute node), because ''parallel -j 24'' will only execute 24 at a time, and the extra jobs will be executed as cores become available.



GNU parallel will replace the ''{%}'' with the job slot number, from 1 to maximum cores (24 in the example).
and ''{}'' gets replaced with a line from ''inputs.txt''.

Pbsdsh starts a very minimal shell, so we evoke ''bash -l'' to start up a full bash shell which loads all of your startup files, so that things like modules work.

The ''-c'' option to ''bash'' is so that bash separates out the arguments correctly (otherwise they're all in a single string).


====Example 2: job script====

<code bash>
#!/bin/bash
#PBS -N p2nodes
#PBS -l select=2:ncpus=24
#PBS -P YOURPROJECT
#PBS -q normal
#PBS -l walltime=0:30:00

CWD=/mnt/lustre/users/$USER/example
cd $CWD
echo "Working directory: " `pwd`


SCDIR=/mnt/lustre/users/$USER/scripts

SCRIPT=$SCDIR/script.sh     # Script to run.
INPUTS=inputs.txt           # Each line in this file is used as arguments to ${SCRIPT}

parallel -j ${PBS_NCPUS} pbsdsh -n {%} -- bash -l -c '"'${SCRIPT} {}'"' :::: ${INPUTS}
</code>


===Ref.===

[[https://opus.nci.org.au/display/Help/GNU+parallel|GNU parallel]] at NCI.