Contents:

1. Running OpenFOAM
2. Compiling your own OpenFOAM code
3. Using the HiSA compressible Solver for OpenFOAM

It has been necessary to change some of our MPI installations. The reason for this is that these installations were compiled without support for the PBS scheduler, which could result in orphaned processes left on the cluster if a job failed for any reason. The revised MPI installations keep all MPI ranks under full control of PBS, which means that they should be cleaned up properly in the event of a job failure. However, uers may now experience an incompatibility with the mpirun command. A typical error message may look like this:

HYDT_bsci_init (tools/bootstrap/src/bsci_init.c:175): unrecognized RMK: user

The simple solution to this error is to remove the -machinefile $PBS_NODEFILE parameter to the mpirun command.

This section describes how to run OpenFOAM on the cluster at CHPC. Versions 2.4.0, 3.0.1, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10, 11, 12, v1706, v1712, v1806, v1812, v1906, v1912, v2006, v2012, v2106, v2112, v2206, v2212, v2306, v2312, v2406 and foam-extend-3.1, 3.2, 4.0 and 5.0 are installed in /apps/chpc/compmech/CFD/OpenFOAM or /home/apps/chpc/compmech/OpenFOAM. Source the required environment from one of the files OF240, OF301, OF40, OF50, OF50mpich, OF60, OF70, OF80, OF90, OF10, OF11, OF12, OFextend31, OFextend32, OFextend40, OFextend50, OF1706, OF1712, OF1806, OF1906, OF1912, OF2006, OF2012 OF2106, OF2112, OF2206, OF2212, OF2306, OF2312 or OF2406 respectively, as illustrated in the example job scripts below. Please be aware that versions after 5.0 and 18.06 use MPICH rather than OpenMPI, and the mpirun command has to select the use of the Infiniband network interface. Also, you may need to source the file from your .bashrc file, to ensure that the compute nodes have the right environment. This should not be necessary, but it is sometimes a useful workaround if you experience MPI troubles. It is assumed that you are familiar with running OpenFOAM solvers in parallel, and have already set up the case directory. The Gmsh and cfMesh meshing utilities are also installed and are added to the executable path when any of the OFxxx files are sourced. OpenFOAM-extend-* has been installed on an experimental basis. To enable it, source the appropriate file in /apps/chpc/compmech/CFD/OpenFOAM/, such as OFextend32.

There is an additional version of OpenFOAM-5.0 that uses MPICH-3.2 rather than OpenMPI. From Open-FOAM-6.0 and v1812, the installations are only available with MPICH. The source file for the OpenFOAM-5.0 version is OF50mpich , and the mpirun command in the script must be modified to read mpirun -iface ib0, to force mpich to use the Infiniband network rather than Ethernet.

Unfortunately, the OpenFOAM development community has become rather fragmented, and the code base has been forked into several variants. This has made it impractical for CHPC to continuously keep up with installing the latest versions of all the variants. However, if you need a particular version or variant, please contact us through the helpdesk and we will take the necessary action.

Copy the case directory you want to run into your lustre folder (/mnt/lustre/users/<username>). The job will fail unless it is run from somewhere on the lustre drive.

PBSPro is used as a scheduler. Users are requested to study the example scripts carefully. In addition, users will be required to include the project ID (dummy value projectid in the example below) in the submit script Create a job file for the PBSPro scheduler containing the following text in your case directory (named, for example, runFoam):

runFoam.qsub

#!/bin/bash 
### This 72-way example calls
### for 3 (virtual) nodes, each with 24 processor cores
### Please note that it is necessary to specify both ncpus and mpiprocs, and 
### for OpenFOAM these should be identical to each other.
### For your own benefit, try to estimate a realistic walltime request.  Over-estimating the 
### wallclock requirement interferes with efficient scheduling, will delay the launch of the job,
### and ties up more of your CPU-time allocation untill the job has finished.
#PBS -P projectid
#PBS -l select=3:ncpus=24:mpiprocs=24:nodetype=haswell_reg
#PBS -q normal
#PBS -l walltime=01:00:00
#PBS -o /mnt/lustre/users/username/foamJobs/job01/stdout
#PBS -e /mnt/lustre/users/username/foamJobs/job01/stderr
#PBS -m abe
#PBS -M username@email.co.za
### Source the openFOAM environment:
. /apps/chpc/compmech/CFD/OpenFOAM/OF2112
##### Running commands
# Set this environment variable explicitly.
export PBS_JOBDIR=/mnt/lustre/users/username/foamJobs/job01
# Explicitly change to the job directory
cd $PBS_JOBDIR
nproc=`cat $PBS_NODEFILE | wc -l`
exe=simpleFoam
#### These next statements build an appropriate decomposeParDict file 
#### based on the requested nunber of nodes
echo "FoamFile" > system/decomposeParDict
echo "{"  >> system/decomposeParDict
echo "  version             2.0;" >> system/decomposeParDict
echo "  format            ascii;" >> system/decomposeParDict
echo "  class        dictionary;" >> system/decomposeParDict
echo "  object decomposeParDict;" >> system/decomposeParDict
echo "}"  >> system/decomposeParDict
echo "numberOfSubdomains " $nproc ";" >> system/decomposeParDict
echo "method scotch;" >> system/decomposeParDict
#### End of decomposeParDict file 
decomposePar -force > decompose.out
## Issue the MPIRUN command.  Omit -iface ib0 when using OpenMPI versions
mpirun -iface ib0 -np $nproc $exe -parallel > foam.out
reconstructPar -latestTime > reconstruct.out
rm -rf processor*

Notes:

• You may wish to add further options for PBS, but these are the minimal ones that are required.
• The number of parallel processes is selected by means of specifying the number of virtual nodes, each with ncpus cores and the same number of mpiprocs. So, in the example above, the case must be decomposed into 72 subdomains via the numberOfSubdomains setting in decomposeParDict. Decomposition method scotch is recommended.
• walltime should be a small over-estimate of the time for the job to run, in hours:minutes:seconds.
• exe should be set to the OpenFOAM executable that you wish to run.
• Screen output for the run will go to the filenames following the #PBS -o and #PBS -e directives. These are buffered and only written after job completion. If you would like to monitor the run as it progresses, use the normal > output redirection to send output to file explicitly, as in the example script.
• The line . /apps/chpc/compmech/CFD/OpenFOAM/OF2112 sources a script file that loads the necessary modules and sets other environment variables for running OpenFOAM-v2112. If you want to run OpenFOAM-2.4.0, change it to . /apps/chpc/compmech/CFD/OpenFOAM/OF240, for example.
• If you need to submit a large number of small jobs, when doing a parametric study, for example, please use Job Arrays. Refer to the PBS-Pro guide at http://wiki.chpc.ac.za/quick:pbspro for guidance on how to set this up.

Submit the job script with the command

qsub runFoam

• Useful commands for monitoring are qstat or qstat -u <username> (only shows your jobs).
• When the job is running you can monitor the output in the log file.
• If the log file is empty and the job has finished running, check for errors in the output files (in the example above, stdout and stderr.

• Are you running from your lustre directory?
• No extra modules or include directories should be necessary to run OpenFOAM. Try commenting out unnecessary lines in your startup files.
• Have you tested your input parameters on a small single process version of your model?

This graph should give you an indication of how well OpenFOAM scales on the cluster. Very good scaling has been achieved up to at least 1000 cores for a 60 million cell model. This ties in quite well with a general rule of thumb that parallelising down to around 50 000 cells per core is a good starting point.

However, using all 24 cores per node is not necessarily beneficial. These two graphs indicate that using 16, 20 or 24 cores per node makes little difference to the performance per node.

This page describes how to set up the environment for compiling your own OpenFOAM solvers and libraries on CHPC. It assumes you are familiar with running OpenFOAM on CHPC as detailed above, and with compiling OpenFOAM code on your own machine.

Usually, the easiest approach is to use one of the installed OpenFOAM versions as a basis for compiling your own code. First choose and source the appropriate OFxx file in /apps/chpc/compmech/CFD/OpenFOAM. You should then set the following environment variables to filepaths somewhere where you have write-permission(eg your home directory): $FOAM_USER_APPBIN and $FOAM_USER_LIBBIN. For instance, using OpenFOAM-6.0, with your solvers and libraries in userapp-6.0 and userlib-6.0 sub-directories in an OpenFOAM directory in your home dir:

customOFenv.sh

source /apps/chpc/compmech/CFD/OpenFOAM/OF60
export FOAM_USER_APPBIN=$HOME/OpenFOAM/userapps-6.0
export FOAM_USER_LIBBIN=$HOME/OpenFOAM/userlibs-6.0

Now you can use wmake to compile your code, and run it as described above. You will need to source the original OFxx file, and then set $FOAM_USER_APPBIN and FOAM_USER_LIBBIN in your submit scripts whenever you run an OpenFOAM job using your custom code.

If you need to make more significant changes to OpenFOAM that require recompiling the whole package, you should first make a local copy of the most appropriate OFxx script, then edit it carefully so that all the filepaths therein point to your custom OpenFOAM installation (again, probably in your home directory). You can then source your altered OFxx script, cd to your custom OpenFOAM installation directory, and run Allwmake. If necessary, you can also set $FOAM_USER_APPBIN and $FOAM_USER_LIBBIN as described earlier (these can be set at the end of the doctored OFxx script, if desired).

NB: OpenFOAM builds typically take several hours so, you should not run it on the login node (and the build may be killed if you do). The build should instead be submitted as a normal cluster job, or use qsub -qsmp -I to get an interactive session on a compute node.