The CHPC has an installation of the CFD-Ace suite of software, including Ace, Geom, View and VisCART, but with no license. Approved academic users can use the software on the CHPC cluster, but need to work off their own licenses. Version 2015.0 is currently available on the system. If you need an older version, please contact CHPC support through the CHPC website.

The ESI-CFD suite is installed under '/apps/chpc/compmech/CFD/ESI'

Because the cluster compute nodes do not have direct access to the internet, it is necessary to use ssh-tunneling through another node to contact the user's license server, which will need to have the license manager and vendor daemon ports open. Tunneling is done via a separate node, chpcslic1. Please apply to CHPC Support so that the CHPC firewall can be configured to allow the necessary traffic. You will need to take similar firewall configuration measures at your end.

On the CHPC clusters all simulations are submitted as jobs to the PBS Pro job scheduler which will assign your job to the appropriate queue and machine. Unlike other software packages, CFD-Ace is not compatible with the parallel Lustre scratch file system. Use your home directory instead.

Example job script:

runAce.qsub

#!/bin/bash
##### The following line will request 10 (virtual) nodes, each with 24 cores running 24 mpi processes 
##### a total of 24-way parallel.
#PBS -l select=10:ncpus=24:mpiprocs=24:mem=4GB
#PBS -q normal
##### Supply YOUR resource programme code in the next line
#PBS -P MECH0000
#PBS -l walltime=1:00:00
#### CFD-Ace will not work on scratch, work in home directory:
#PBS -o /home/username/acetesting/ace.out
#PBS -e /home/username/acetesting/ace.err
##### The following two lines will send the user an e-mail when the job aborts, begins or ends.
#PBS -m abe
#PBS -M username@email.co.za
##### Set up path.  To ensure that the compute nodes also have access to this path, 
#####  also add these lines to your .bashrc file
export ESI_HOME=/apps/chpc/compmech/CFD/ESI
export PATH=$ESI_HOME/2015.0/UTILS/bin:$PATH
export LD_LIBRARY_PATH=$ESI_HOME/2015.0/UTILS/lib:$LD_LIBRARY_PATH
##### set up ssh-tunnels to your license server. Obviously use the right port numbers. And server URL.
#### lmgrd daemon
ssh -f username@chpclic1 -L 1999:licenseserver.ac.za:1999 -N
#### vendor daemon port
ssh -f username@chpclic1 -L 1998:licenseserver.ac.za:1998 -N
#### Tell solver where to look for the license.  
####  localhost is correct here, it follows from the ssh-tunneling
export LM_LICENSE_FILE=1999@localhost
#### There is no -d option available under PBS Pro, therefore 
#### explicitly set working directory and change to that.
#### CFD-Ace will not work on scratch, work in home directory.
export PBS_JOBDIR=/home/username/acetesting
cd $PBS_JOBDIR
nproc=`cat $PBS_NODEFILE | wc -l`
#### This is a minimal run instruction, 
####  it will run the solver until reaching the stopping criterium set in the DTF file.
CFD-SOLVER --model=test01 --wd=$PBS_JOBDIR --num=$nproc --hosts=$PBS_NODEFILE --sim=1 > test01.out

There are different ways of monitoring your solution. Please refer to the page on Remote Visualization for instructions on how to get a VNC session on the visualization server. Add the path to the VirtualGL installation with the environment setting: 'export PATH=$PATH:/opt/VirtualGL/bin'. The vglrun wrapper is required to ensure that OpenGL works in the VNC session. Alternatively, as per the remote viz instructions, simply start your X-session with the vglrun command, after which it will not be necessary to use it again for OpenGL apps. In order to access your license, set up the environment (with the bash 'export' statements as per your job submission script), and create the same SSH-tunnels as per the job submission script. The command 'vglrun CFD-ACE-GUI' should allow you to open the GUI. Once you have the GUI open, you can open the .RSL and .MON files to monitor progress. Alternatively, you can use gnuplot to plot the contents of these files. In the absence of a graphics-capable environment, it is still practical to plot these files with gnuplot, using old-fashioned but very funky ASCII graphics. In the gnuplot interface, simply give the command 'set terminal dumb 120 40', where the numbers specify the size of the character terminal. Use vglrun to open CFD-VIEW for graphical post-processing and image creation.