What this example does is that it finds a whole lot of scripts in a directory and gets gnu parallel to run them. The -j 6 options tells gnu parallel to assume that each sub-job requires 6 cores.

gnu_parallel.qsub

#!/bin/bash
#PBS -e /mnt/lustre/users/USERNAME/test_scripts/gnu_parallel.stderr.out
#PBS -o /mnt/lsutre/users/USERNAME/test_scripts/gnu_parallel.stdout.out
#PBS -V
#PBS -P PROGRAMMESHORTNAME
#PBS -M youremailaddress
#PBS -l select=2:ncpus=24:nodetype=haswell_reg
#PBS -l walltime=00:01:00
#PBS -q normal
#PBS -m be
#PBS -r n
#PBS -mb
 
module add chpc/gnu/parallel-20160422
 
WORKING_DIR=/mnt/lustre/users/USERNAME/test_scripts
echo "Hello World! Main gnu parallel test thingy running here"
 
cd ${WORKING_DIR}
 
ls ${WORKING_DIR}/gnup_scripts/* | parallel -j 6 -u --sshloginfile ${PBS_NODEFILE} "cd ${WORKING_DIR}/gnup_scripts; {} {}"

Then inside the directory

/mnt/lustre/users/USERNAME/test_scripts/gnup_scripts

you can put a whole lot of copies of the following file:

gnup.test.sh

#!/bin/bash
 
NOW=$(date +"%x %X")
echo "Hello World! The time is ${NOW} and I'm running on host: ${HOSTNAME}. I'm running task $1 :-)"
sleep 1

and make sure they are all runnable, i.e.

chmod u+x gnup.test.sh

After running

qsub gnu_parallel.qsub

if you look inside

/mnt/lsutre/users/USERNAME/test_scripts/gnu_parallel.stdout.out

you should see something like:

Hello World! Main gnu parallel test thingy running here
Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.12.sh :-)
Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.10.sh :-)
.
.
.
Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.9.sh :-)

One problem we have is that jobs fail on the cluster. If one is making use of gnu-parallel it can be simple to include some fault checking code in the form of log files. So building on the previous example we build a script that is designed to fail randomly. It logs successful runs and doesn't re-run if a failure is detected.

gnup.test.sh

#!/bin/bash
 
LOGFILE="$0.log"  # create a log file based on the executable's name
echo "$(date +'%x %X'): STARTING" >> "${LOGFILE}"  # Log the start of the run
if [ $(grep "COMPLETE" "${LOGFILE}" | wc -l) -lt 1 ]  # check if it has already been run...
then
  echo "Hello World! The time is $(date +'%x %X') and I'm running on host: ${HOSTNAME}. I'm running task $1 :-)"
  sleep 1
  if [ $(( ( RANDOM % 10 )  + 1 )) -gt 4 ]  # will run successfully something like 60% of the time
  then
    echo "SUCCESS!"
    echo "$(date +'%x %X'):COMPLETE" >> "${LOGFILE}"
  else
    echo "Failure"  # Oh boo. This will not be logged
  fi
else
  echo "$(date +'%x %X'): NOT REPEATING" >> "${LOGFILE}"  # Log that run has already been performed
fi

So if we run our same PBS script, but this time point at a directory containing, say 50, copies of the above script then we see:

[dane@login1]$ ls gnup_scripts/*.sh | wc -l; ls gnup_scripts/*.log | wc -l
50
50
[dane@login1]$ cat gnup_scripts/*.log | grep STARTING | wc -l; cat gnup_scripts/*.log | grep COMPLETE | wc -l ; cat gnup_scripts/*.log | grep "NOT REPEATING" | wc -l
50
34
0
[dane@login1]$ cat gnu_parallel.stdout.out | grep "SUCCESS" | wc -l; cat gnu_parallel.stdout.out | grep "Failure" | wc -l
34
16
[dane@login1]$ 

So we can see that it ran 50 times, and failed 16 out of 50 times. We can run again and we see:

[dane@login1]$ cat gnup_scripts/*.log | grep STARTING | wc -l; cat gnup_scripts/*.log | grep COMPLETE | wc -l ; cat gnup_scripts/*.log | grep "NOT REPEATING" | wc -l                                                                                                          
100
45
34
[dane@login1]$ cat gnu_parallel.stdout.out | grep "SUCCESS" | wc -l; cat gnu_parallel.stdout.out | grep "Failure" | wc -l
11
5

i.e. we successfully ran another 11 jobs…

So the idea around this script is that one has a working directory where you in turn have tree structure that might look something like

DATE/EXPERIMENT/*.fasta

. Here gnu parallel simply takes as input each fasta file and does the blast on each with the parameters provided.

blast.gnup.qsub

#!/bin/bash
#PBS -e /mnt/lustre/users/USERNAME/blastjobs/blast.stderr.out
#PBS -o /mnt/lustre/users/USERNAME/blastjobs/blast.stderr.out
#PBS -P PROGRAMMESHORTNAME
#PBS -M youremail@address
#PBS -l select=4:ncpus=24:nodetype=haswell_reg
#PBS -l walltime=00:60:00
#PBS -q normal
#PBS -m be
 
cd ${PBS_O_WORKDIR}
 
module add chpc/gnu/parallel-20160422
module add chpc/BIOMODULES
module add ncbi-blast/2.3.0/gcc
 
BLASTDB="/lustre/bsp/blast/nt/nt"
BLASTCMD=$(which blastn)
BLASTARGS="-evalue 0.005 -num_alignments 20 -outfmt 5 -num_threads 24"
INPUTDIRS="DATE/*"
 
ls ${INPUTDIRS}/*.fasta | parallel -j 24 -u --sshloginfile ${PBS_NODEFILE} "cd ${PBS_O_WORKDIR}; ${BLASTCMD} -db ${BLASTDB} -query {} ${BLASTARGS} -out {}.xml && gzip --best {} {}.xml"

The idea around the fault tolerance is very simple – when the blast has been successfully run, the file will be zipped and given a .gz suffix and won't be found in subsequent blasts (which explicitly look for .fasta).