ANSYS/CFX

The CHPC has an installation of Ansys-CFD along with a limited license for academic use only. The license covers use of the Fluent and CFX solvers, as well as the IcemCFD meshing code.

If you are a full time student or staff at an academic institution then you may request access to use Ansys-CFD on the CHPC clusters. Send your request along with motivation and description of the work in an email to helpdesk@chpc.ac.za.

CFX versions are installed in /apps/chpc/compmech/CFD/ansys_inc/v???/CFX, where ??? represents the software release version such as 242, for example. Similarly, IcemCFD is installed under /apps/chpc/compmech/CFD/ansys_inc/v???/icemcfd.

CHPC has academic licenses for AnsysCFD. There are 25 “solver” processes and the license pool is not currently under heavy pressure. There is a license resource management system to be used for Fluent, but it is not effective for older CFX versions.

On the CHPC clusters all simulations are submitted as jobs to the PBS Pro job scheduler which will assign your job to the appropriate queue and machine.

Example job script:

runCFX.qsub

#!/bin/bash
#PBS -P projectid
#PBS -l select=4:ncpus=24:mpiprocs=24
#PBS -l walltime=2:00:00
#PBS -q normal
#PBS -V
#PBS -o /mnt/lustre/users/username/testCFX/test.out
#PBS -e /mnt/lustre/users/username/testCFX/test.err
 
export LM_LICENSE_FILE=1055@login1
export ANSYSLMD_LICENSE_FILE=1055@login1
#change to CFX case directory
cd /mnt/lustre/users/username/testCFX 
#force CFX to use ssh instead of rsh
export CFX5RSH=ssh 
# Select IntelMPI-2018 instead of 2021
export CFX5_IMPI_DIR=/apps/chpc/compmech/CFD/ansys_inc/v242/commonfiles/MPI/Intel/2018.3.222/linx64
cfx=/apps/chpc/compmech/CFD/ansys_inc/v242/CFX/bin/cfx5solve #path to cfx5 solver
$cfx -batch -def testCFX.def -par-dist $(tr "\n" "," <$PBS_NODEFILE | rev | cut -c2- | rev) > runcfx.log

The “standard” process assumes that the user already has a local license for the software.

• Mesh and pre-process the simulation as usual for a local simulation.
• Test it locally to ensure that everything works properly. Be cautious about absolute path file names.
• Compress the def file with gzip.
• Upload to CHPC using either scp or rsync. The advantage of rsync is that the transfer can be made persistent, to prevent network communication glitches from killing the file transfer.

If your simulations files are too large, or your internet connection too slow, consider transferring geometry and script files only. This will require careful scripting and testing, but is certainly practical.

• Neither ANSYS-Mesh nor Gambit is available on the CHPC system, but ICEMCFD is.
• If using IcemCFD, transfer the Icem .prj, .tin, .fbc and .blk (if using hexa) files, along with a recorded Icem script file for generating the mesh and exporting the file in CFX format. Watch for absolute path names in the script file. Run IcemCFD with the -batch -script options to create the mesh. A more comprehensive CFX script will be required to import the mesh and pre-process the case. Test locally!
• If your internet connection is too slow to permit easy case uploading, it will also be far too slow for downloading the results files. Consider generating post-processing images “on the fly”, or alternatively exporting only surface data on completion of the simulation.

• Give some thought to the resources being requested. Partitioning a simulation too finely will not necessarily speed it up as expected. In general, less than 100 000 cells per process is seldom justified.
• A request for a smaller number of cores may result in the job launching earlier, resulting in reduced turn-around time, even if the job takes longer to run.
• Monitoring convergence of batch jobs can be painful but necessary.
• Monitor files (such as cd or cl files) can be plotted with gnuplot even if no GUI is available. On a slow connection, consider using gnuplot with set term dumb to get funky 1970's style ASCII graphics.
• If you need to submit a large number of small jobs, when doing a parametric study, for example, please use Job Arrays. Refer to the PBS-Pro guide at http://wiki.chpc.ac.za/quick:pbspro for guidance on how to set this up.