====== ANSYS/CFX ====== The CHPC has an installation of Ansys-CFD along with a limited license for academic use only. The license covers use of the Fluent and CFX solvers, as well as the IcemCFD meshing code. ===== Application Process ===== If you are a **full time** student or staff at an **academic** institution then you may request access to use Ansys-CFD on the CHPC clusters. Send your request along with motivation and description of the work in an email to ''helpdesk@chpc.ac.za''. ===== Installation ===== CFX versions are installed in ''/apps/chpc/compmech/CFD/ansys_inc/v???/CFX'', where ??? represents the software release version such as 242, for example. Similarly, IcemCFD is installed under ''/apps/chpc/compmech/CFD/ansys_inc/v???/icemcfd''. ===== Licensing ===== CHPC has academic licenses for AnsysCFD. There are 25 "solver" processes and the license pool is not currently under heavy pressure. There is a license resource management system to be used for Fluent, but it is not effective for older CFX versions. ===== Running a CFX Job ===== On the CHPC clusters all simulations are submitted as jobs to the PBS Pro job scheduler which will assign your job to the appropriate queue and machine. Example job script: #!/bin/bash #PBS -P projectid #PBS -l select=4:ncpus=24:mpiprocs=24 #PBS -l walltime=2:00:00 #PBS -q normal #PBS -V #PBS -o /mnt/lustre/users/username/testCFX/test.out #PBS -e /mnt/lustre/users/username/testCFX/test.err export LM_LICENSE_FILE=1055@login1 export ANSYSLMD_LICENSE_FILE=1055@login1 #change to CFX case directory cd /mnt/lustre/users/username/testCFX #force CFX to use ssh instead of rsh export CFX5RSH=ssh # Select IntelMPI-2018 instead of 2021 export CFX5_IMPI_DIR=/apps/chpc/compmech/CFD/ansys_inc/v242/commonfiles/MPI/Intel/2018.3.222/linx64 cfx=/apps/chpc/compmech/CFD/ansys_inc/v242/CFX/bin/cfx5solve #path to cfx5 solver $cfx -batch -def testCFX.def -par-dist $(tr "\n" "," <$PBS_NODEFILE | rev | cut -c2- | rev) > runcfx.log ===== Different methods of uploading a simulation ===== ==== Build and test locally, upload .def file ==== The "standard" process assumes that the user already has a local license for the software. * Mesh and pre-process the simulation as usual for a local simulation. * Test it locally to ensure that everything works properly. Be cautious about absolute path file names. * Compress the def file with gzip. * Upload to CHPC using either scp or rsync. The advantage of rsync is that the transfer can be made persistent, to prevent network communication glitches from killing the file transfer. ==== Build and test locally, upload geometry and script files only, mesh and pre-process remotely ==== If your simulations files are too large, or your internet connection too slow, consider transferring geometry and script files only. This will require careful scripting and testing, but is certainly practical. * Neither ANSYS-Mesh nor Gambit is available on the CHPC system, but ICEMCFD is. * If using IcemCFD, transfer the Icem .prj, .tin, .fbc and .blk (if using hexa) files, along with a recorded Icem script file for generating the mesh and exporting the file in CFX format. Watch for absolute path names in the script file. Run IcemCFD with the -batch -script options to create the mesh. A more comprehensive CFX script will be required to import the mesh and pre-process the case. **Test locally!** * If your internet connection is too slow to permit easy case uploading, it will also be far too slow for downloading the results files. Consider generating post-processing images "on the fly", or alternatively exporting only surface data on completion of the simulation. ===== General tips and advice ===== * Give some thought to the resources being requested. Partitioning a simulation too finely will not necessarily speed it up as expected. In general, less than 100 000 cells per process is seldom justified. * A request for a smaller number of cores may result in the job launching earlier, resulting in reduced turn-around time, even if the job takes longer to run. * Monitoring convergence of batch jobs can be painful but necessary. * Monitor files (such as cd or cl files) can be plotted with gnuplot even if no GUI is available. On a slow connection, consider using gnuplot with ''set term dumb'' to get funky 1970's style ASCII graphics. * If you need to submit a large number of small jobs, when doing a parametric study, for example, please use Job Arrays. Refer to the PBS-Pro guide at http://wiki.chpc.ac.za/quick:pbspro for guidance on how to set this up.