This is an old revision of the document!
What this example does is that it finds a whole lot of scripts in a directory and gets gnu parallel to run them. The -j 6 options tells gnu parallel to assume that each sub-job requires 6 cores.
#!/bin/bash #PBS -e /mnt/lustre/users/USERNAME/test_scripts/gnu_parallel.stderr.out #PBS -o /mnt/lsutre/users/USERNAME/test_scripts/gnu_parallel.stdout.out #PBS -V #PBS -P PROGRAMMESHORTNAME #PBS -M youremailaddress #PBS -l select=2:ncpus=24:nodetype=haswell_reg #PBS -l walltime=00:01:00 #PBS -q normal #PBS -m be #PBS -r n #PBS -mb module add chpc/gnu/parallel-20160422 WORKING_DIR=/mnt/lustre/users/USERNAME/test_scripts echo "Hello World! Main gnu parallel test thingy running here" cd ${WORKING_DIR} ls ${WORKING_DIR}/gnup_scripts/* | parallel -j 6 -u --sshloginfile ${PBS_NODEFILE} "cd ${WORKING_DIR}/gnup_scripts; {} {}"
Then inside the directory
/mnt/lustre/users/USERNAME/test_scripts/gnup_scripts
you can put a whole lot of copies of the following file:
#!/bin/bash NOW=$(date +"%x %X") echo "Hello World! The time is ${NOW} and I'm running on host: ${HOSTNAME}. I'm running task $1 :-)" sleep 1
and make sure they are all runnable, i.e.
chmod u+x gnup.test.sh
After running
qsub gnu_parallel.qsub
if you look inside
/mnt/lsutre/users/USERNAME/test_scripts/gnu_parallel.stdout.out
you should see something like:
Hello World! Main gnu parallel test thingy running here Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.12.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.10.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.13.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.14.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.15.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.16.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.17.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.18.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.19.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.1.sh :-) Hello World! The time is 05/05/2016 15:24:26 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.20.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.11.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.21.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.23.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.22.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.24.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.25.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.26.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.27.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.28.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.29.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.2.sh :-) Hello World! The time is 05/05/2016 15:24:27 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.30.sh :-) Hello World! The time is 05/05/2016 15:24:28 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.31.sh :-) Hello World! The time is 05/05/2016 15:24:28 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.32.sh :-) Hello World! The time is 05/05/2016 15:24:28 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.33.sh :-) Hello World! The time is 05/05/2016 15:24:28 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.34.sh :-) Hello World! The time is 05/05/2016 15:24:28 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.35.sh :-) Hello World! The time is 05/05/2016 15:24:28 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.36.sh :-) Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.3.sh :-) Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.4.sh :-) Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.5.sh :-) Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.6.sh :-) Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0282. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.7.sh :-) Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.8.sh :-) Hello World! The time is 05/05/2016 15:24:29 and I'm running on host: cnode0281. I'm running task /home/dane/test_scripts/gnup_scripts/gnup.test.9.sh :-)