The CHPC has an installation of Ansys-CFD along with a limited license for academic use only. The license covers use of the Fluent and CFX solvers, as well as the IcemCFD meshing code.

If you are a full time student or staff at an academic institution then you may request access to use Ansys-CFD on the CHPC clusters. Send your request along with motivation and description of the work in an email to helpdesk@chpc.ac.za.

CFX version 5 is installed in /apps/chpc/compmech/CFD/ansys_inc/v170/CFX. IcemCFD is installed under /apps/chpc/compmech/CFD/ansys_inc/v170/icemcfd.

CHPC has academic licenses for AnsysCFD. There are 5 “solver” processes, available as “cfd_base” and 4096 “HPC” licenses, available as “anshpc”. There is a license resource management system. If you request license resources (as in these example scripts), the scheduler will check for license availability before starting a job. License unavailability will result in the job being held back until the necessary licenses have become available. Although use of the license resource request is not mandatory, its use is strongly recommended. If you do not use the license resource requests, the job will fail if no licenses are available. A single cfd_base license is required to start the solver, and includes up to 4 HPC licenses. Therefore you should request ($nproc-4) ans_hpc licenses.

On the CHPC clusters all simulations are submitted as jobs to the PBS Pro job scheduler which will assign your job to the appropriate queue and machine.

Example job script:

runCFX.qsub

#!/bin/bash
#PBS -P projectid
#PBS -l select=4:ncpus=24:mpiprocs=24:mem=24GB:nodetype=haswell_reg
#PBS -l walltime=2:00:00
#PBS -q normal
#PBS -m be
#PBS -V
#PBS -o /mnt/lustre3p/users/username/testCFX/test.out
#PBS -e /mnt/lustre3p/users/username/testCFX/test.err
#### Check for license availability.  If insufficient licenses are available, job will be held back until
####  licenses are available.
#PBS -l cfd_base=1
#PBS -l anshpc=200
 
cd /mnt/lustre3p/users/username/testCFX #change to CFX case directory
export CFX5RSH=ssh #force CFX to use ssh instead of rsh
cfx=/apps/chpc/compmech/CFD/ansys_inc/v194/CFX/bin/cfx5solve #path to cfx5 solver
$cfx -batch -def testCFX.def -par-dist $(tr "\n" "," <$PBS_NODEFILE) > runcfx.log

The “standard” process assumes that the user already has a local license for the software.

• Mesh and pre-process the simulation as usual for a local simulation.
• Test it locally to ensure that everything works properly. Be cautious about absolute path file names.
• Compress the def file with gzip.
• Upload to CHPC using either scp or rsync. The advantage of rsync is that the transfer can be made persistent, to prevent network communication glitches from killing the file transfer.

If your simulations files are too large, or your internet connection too slow, consider transferring geometry and script files only. This will require careful scripting and testing, but is certainly practical.

• Neither ANSYS-Mesh nor Gambit is available on the CHPC system, but ICEMCFD is.
• If using IcemCFD, transfer the Icem .prj, .tin, .fbc and .blk (if using hexa) files, along with a recorded Icem script file for generating the mesh and exporting the file in CFX format. Watch for absolute path names in the script file. Run IcemCFD with the -batch -script options to create the mesh. A more comprehensive CFX script will be required to import the mesh and pre-process the case. Test locally!
• If your internet connection is too slow to permit easy case uploading, it will also be far too slow for downloading the results files. Consider generating post-processing images “on the fly”, or alternatively exporting only surface data on completion of the simulation.

• Give some thought to the resources being requested. Partitioning a simulation too finely will not necessarily speed it up as expected. In general, less than 100 000 cells per process is seldom justified.
• A request for a smaller number of cores may result in the job launching earlier, resulting in reduced turn-around time, even if the job takes longer to run.
• Monitoring convergence of batch jobs can be painful but necessary.
• Monitor files (such as cd or cl files) can be plotted with gnuplot even if no GUI is available. On a slow connection, consider using gnuplot with set term dumb to get funky 1970's style ASCII graphics.
• If you need to submit a large number of small jobs, when doing a parametric study, for example, please use Job Arrays. Refer to the PBS-Pro guide at http://wiki.chpc.ac.za/quick:pbspro for guidance on how to set this up.