What this example does is that it finds a whole lot of scripts in a directory and gets gnu parallel to run them. The -j 6 options tells gnu parallel to assume that each sub-job requires 6 cores.

gnu_parallel.qsub

#!/bin/bash
#PBS -e /mnt/lustre/users/USERNAME/test_scripts/gnu_parallel.stderr.out
#PBS -o /mnt/lsutre/users/USERNAME/test_scripts/gnu_parallel.stdout.out
#PBS -V
#PBS -P PROGRAMMESHORTNAME
#PBS -M youremailaddress
#PBS -l select=2:ncpus=24:nodetype=haswell_reg
#PBS -l walltime=00:01:00
#PBS -q normal
#PBS -m be
#PBS -r n
#PBS -mb
 
module add chpc/gnu/parallel-20160422
 
WORKING_DIR=/mnt/lustre/users/USERNAME/test_scripts
echo "Hello World! Main gnu parallel test thingy running here"
 
cd ${WORKING_DIR}
 
ls ${WORKING_DIR}/gnup_scripts/* | parallel -j 6 -u --sshloginfile ${PBS_NODEFILE} "cd ${WORKING_DIR}/gnup_scripts; {} {}"

Then inside the directory

/mnt/lustre/users/USERNAME/test_scripts/gnup_scripts

you can put a whole lot of copies of the following file:

gnup.test.sh

#!/bin/bash
 
NOW=$(date +"%x %X")
echo "Hello World! The time is ${NOW} and I'm running on host: ${HOSTNAME}. I'm running task $1 :-)"
sleep 1

and make sure they are all runnable, i.e.

chmod u+x gnup.test.sh

After running

qsub gnu_parallel.qsub

if you look inside

/mnt/lsutre/users/USERNAME/test_scripts/gnu_parallel.stdout.out

you should see something like:

Hello World! Main gnu parallel test thingy running here
Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.12.sh :-)
Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.10.sh :-)
.
.
.
Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.9.sh :-)

One problem we have is that jobs fail on the cluster. If one is making use of gnu-parallel it can be simple to include some fault checking code in the form of log files. So building on the previous example we build a script that is designed to fail randomly. It logs successful runs and doesn't re-run if a failure is detected.

gnup.test.sh

#!/bin/bash
 
LOGFILE="$0.log"  # create a log file based on the executable's name
echo "$(date +'%x %X'): STARTING" >> "${LOGFILE}"  # Log the start of the run
if [ $(grep "COMPLETE" "${LOGFILE}" | wc -l) -lt 1 ]  # check if it has already been run...
then
  echo "Hello World! The time is $(date +'%x %X') and I'm running on host: ${HOSTNAME}. I'm running task $1 :-)"
  sleep 1
  if [ $(( ( RANDOM % 10 )  + 1 )) -gt 4 ]  # will run successfully something like 60% of the time
  then
    echo "SUCCESS!"
    echo "$(date +'%x %X'):COMPLETE" >> "${LOGFILE}"
  else
    echo "Failure"  # Oh boo. This will not be logged
  fi
else
  echo "$(date +'%x %X'): NOT REPEATING" >> "${LOGFILE}"  # Log that run has already been performed
fi

So if we run our same PBS script, but this time point at a directory containing, say 50, copies of the above script then we see:

[dane@login1]$ ls gnup_scripts/*.sh | wc -l; ls gnup_scripts/*.log | wc -l
50
50
[dane@login1]$ cat gnup_scripts/*.log | grep STARTING | wc -l; cat gnup_scripts/*.log | grep COMPLETE | wc -l ; cat gnup_scripts/*.log | grep "NOT REPEATING" | wc -l
50
34
0
[dane@login1]$ cat gnu_parallel.stdout.out | grep "SUCCESS" | wc -l; cat gnu_parallel.stdout.out | grep "Failure" | wc -l
34
16
[dane@login1]$ 

So we can see that it ran 50 times, and failed 16 out of 50 times. We can run again and we see:

[dane@login1]$ cat gnup_scripts/*.log | grep STARTING | wc -l; cat gnup_scripts/*.log | grep COMPLETE | wc -l ; cat gnup_scripts/*.log | grep "NOT REPEATING" | wc -l                                                                                                          
100
45
34
[dane@login1]$ cat gnu_parallel.stdout.out | grep "SUCCESS" | wc -l; cat gnu_parallel.stdout.out | grep "Failure" | wc -l
11
5

i.e. we successfully ran another 11 jobs…

So the idea around this script is that one has a working directory where you in turn have tree structure that might look something like

DATE/EXPERIMENT/*.fasta

. Here gnu parallel simply takes as input each fasta file and does the blast on each with the parameters provided.

blast.gnup.qsub

#!/bin/bash
#PBS -e /mnt/lustre/users/USERNAME/blastjobs/blast.stderr.out
#PBS -o /mnt/lustre/users/USERNAME/blastjobs/blast.stderr.out
#PBS -P PROGRAMMESHORTNAME
#PBS -M youremail@address
#PBS -l select=4:ncpus=24:nodetype=haswell_reg
#PBS -l walltime=00:60:00
#PBS -q normal
#PBS -m be
 
cd ${PBS_O_WORKDIR}
 
module add chpc/gnu/parallel-20160422
module add chpc/BIOMODULES
module add ncbi-blast/2.3.0/gcc
 
BLASTDB="/lustre/bsp/blast/nt/nt"
BLASTCMD=$(which blastn)
BLASTARGS="-evalue 0.005 -num_alignments 20 -outfmt 5 -num_threads 24"
INPUTDIRS="DATE/*"
 
ls ${INPUTDIRS}/*.fasta | parallel -j 1 -u --sshloginfile ${PBS_NODEFILE} "cd ${PBS_O_WORKDIR}; ${BLASTCMD} -db ${BLASTDB} -query {} ${BLASTARGS} -out {}.xml && gzip --best {} {}.xml"

The idea around the fault tolerance is very simple – when the blast has been successfully run, the file will be zipped and given a .gz suffix and won't be found in subsequent blasts (which explicitly look for .fasta).

This takes the previous example one step further and is useful for large jobs with very many independent fasta files (and this individual blasts). The added step involves copying the blast database into ram disk at the beginning of the job.

blast.advanced.gnup.qsub

#!/bin/bash
#PBS -e /mnt/lustre/users/USERNAME/blastjobs/blast.advanced.stderr.out
#PBS -o /mnt/lustre/users/USERNAME/blastjobs/blast.advanced.stderr.out
#PBS -P PROGRAMMESHORTNAME
#PBS -M youremail@address
#PBS -l select=4:ncpus=24:nodetype=haswell_reg
#PBS -l walltime=00:60:00
#PBS -q normal
#PBS -m be
 
cd ${PBS_O_WORKDIR}
 
module add chpc/gnu/parallel-20160422
module add chpc/BIOMODULES
module add ncbi-blast/2.3.0/gcc
 
BLASTDBDIR="/lustre/bsp/blast/nt"
BLASTDB="nt"
BLASTCMD=$(which blastn)
BLASTARGS="-evalue 0.005 -num_alignments 20 -outfmt 5 -num_threads 24"
INPUTDIRS="2016-05-12/*"
 
NODES=$(cat ${PBS_NODEFILE} | sort | uniq)
 
# copy blast databases to ram disk
for node in ${NODES}
do
  ssh ${node} "mkdir -p /dev/shm/${USER}/BLAST && cp -r ${BLASTDBDIR}/${BLASTDB}* /dev/shm/${USER}/BLAST && echo 'successfully added DBs on ${node}' || exit 1" &
done
 
wait  # wait for parallel copies to finish
 
ls ${INPUTDIRS}/*.fa | parallel -j 1 -u --sshloginfile ${PBS_NODEFILE} "cd ${PBS_O_WORKDIR}; ${BLASTCMD} -db /dev/shm/${USER}/BLAST/${BLASTDB} -query {} ${BLASTARGS} -out {}.xml && gzip --best {} {}.xml"
 
# clean up ram disk
for node in ${NODES}
do
  ssh ${node} "rm -rf /dev/shm/${USER}/BLAST && echo 'successfully deleted DBs on ${node}' || exit 1" &
done
 
wait