User Tools

Site Tools


howto:ansys

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
howto:ansys [2020/05/07 12:03]
ccrosby [Running a Fluent Job]
howto:ansys [2021/05/05 11:36] (current)
ccrosby [If a GUI is required]
Line 1: Line 1:
 ====== ANSYS/​Fluent ====== ====== ANSYS/​Fluent ======
  
-The CHPC has an installation of Ansys-CFD along with a limited license for academic use only.  The license covers use of the Fluent and CFX solvers, as well as the IcemCFD meshing code.  Versions before 17.2 have been retired, as well as 18.0 and 18.1, but Versions 17.2, 18.2, 19.0, 19.1, 19.2, 19.3, 19.4, 19.5 (aka 2019R3) and 20.1 (2020R1) ​are available. ​ Please note that the default file format has changed from version 20.1.+===== LICENSING PROBLEMS continue unabated ===== 
 +** Please bear with us while we work with Ansys on sorting out some teething problems introduced by the license system changes. ​  ** 
 + 
 + 
 +  * ** The pre-R21.1 versions also require communications on port 2325 of the license server. ​ Unfortunately,​ this port is no longer available on login1, which prevents pre-R21.1 versions of the software from checking out a license. ​ The work-around for this problem if you need the older versions of the software is to rather use the license on chpclic1 untill this problem has been resolved. ​ Please note that the license resources cfd_base and anshpc are not available on chpclic1, and should not be specified. ​ ** 
 + 
 +  
 + 
 +===== URGENT: LICENSE RESOURCE REQUEST CHANGE ===== 
 +** With the release of Ansys version 21.1, the structure of the license pool has changed dramatically. ​ The old license resources //​aa_r_cfd//​ and //​aa_r_hpc//​ are no longer relevant, and requesting them will prevent your job from running. ​ It is necessary to change these statements to request the new license resources //​cfd_base//​ and //anshpc//, as per the information given below. ​   ** 
 + 
 +===== URGENT: LICENSE SERVER CHANGE ===== 
 +**We are in the process of retiring the existing license server chpclic1. ​ The license has been moved to login1. ​ Please change your .bashrc file and job scripts to point to the new license server, as per the example scripts below.** 
 + 
 +The CHPC has an installation of Ansys-CFD along with a limited license for academic use only.  The license covers use of the Fluent and CFX solvers, as well as the IcemCFD meshing code.  ​If you are a new Ansys user on Lengau, [[https://​users.chpc.ac.za/​helpdesk/​tickets/​submit/​|submit a helpdesk ticket]], requesting access to the license. ​Versions before 17.2 have been retired, as well as 18.018.1, 19.0 and 19.1, but Versions 17.2, 18.2, 19.2, 19.3, 19.4, 19.520.1, 20.2 nd 21.1 are available. ​ Please note that the default file format has changed from version 20.1.
  
 ===== Application Process ===== ===== Application Process =====
Line 14: Line 28:
 ===== Licensing ===== ===== Licensing =====
  
-** Please note that the license has been upgraded, making more resources available. ​ We will monitor use and advise users accordingly.** 
  
-CHPC has academic licenses for AnsysCFD. ​ There are 25 "​solver"​ processes, available as aa_r_cfd ​and 2048 "​HPC"​ licenses, available as aa_r_hpc.  ​license resource management system ​has been activated. If you request license resources (as in these example scripts), the scheduler will check for license availability before starting a job.  License unavailability will result in the job being held back until the necessary licenses have become available. ​ Although use of the license resource request is not mandatory, its use is strongly recommended. ​ If you do not use the license resource requests, the job will fail if no licenses are available. ​ A single ​aa_r_cfd ​license is required to start the solver, and includes up to 16 HPC licenses. ​ Therefore you should request ($nproc-16aa_r_hpc ​licenses. ​ **Do not request more than you need, it will delay the start of your job.**+CHPC has academic licenses for AnsysCFD.  ​If you are a new Ansys user on Lengau, [[https://​users.chpc.ac.za/​helpdesk/​tickets/​submit/​|submit a helpdesk ticket]], requesting access to the license.  
 + There are 25 "​solver"​ processes and 4096 "​HPC"​ licenses.  ​Please use the license resource management system ​to ensure that your job does not start if there are no licenses available for it. 
 + 
 +=== License resource management system === 
 +If you request license resources (as in these example scripts), the scheduler will check for license availability before starting a job.  License unavailability will result in the job being held back until the necessary licenses have become available. ​ Although use of the license resource request is not mandatory, its use is strongly recommended. ​ If you do not use the license resource requests, the job will fail if no licenses are available. ​ A single ​cfd_base ​license is required to start the solver, and includes up to HPC licenses. ​ Therefore you should request ($nproc-4anshpc ​licenses. ​ **Do not request more than you need, it will delay the start of your job.**
  
  
 The Fluent licenses are in general highly utilised. ​ The consequence is that jobs may be held back due to unavailability of licenses. ​ It is possible for the CHPC to forcefully apply measures that will ensure fair use.  However, in order to avoid this situation, please stick to the following guidelines: The Fluent licenses are in general highly utilised. ​ The consequence is that jobs may be held back due to unavailability of licenses. ​ It is possible for the CHPC to forcefully apply measures that will ensure fair use.  However, in order to avoid this situation, please stick to the following guidelines:
-  * Each Ansys user shall submit only one PBS script (tying up one aa_r_cfd) at any given time.+  * Each Ansys user shall submit only one PBS script (tying up one cfd_base) at any given time.
   * Given the constraint of the number of solver licenses, do your Fluent runs sequentially. ​ Do not try to run more than one Fluent analysis at a time.   * Given the constraint of the number of solver licenses, do your Fluent runs sequentially. ​ Do not try to run more than one Fluent analysis at a time.
   * Take full advantage of the large number of cores available on Lengau to run each Fluent analysis faster. ​ Without requesting special permission, you are entitled to use 240 cores for a Fluent run.  Our testing has indicated very good parallel scaling down to 10000 grid cells per core (sometimes even less), which means that for any run over about 2 million cells, you should aim to use around 200 cores.   * Take full advantage of the large number of cores available on Lengau to run each Fluent analysis faster. ​ Without requesting special permission, you are entitled to use 240 cores for a Fluent run.  Our testing has indicated very good parallel scaling down to 10000 grid cells per core (sometimes even less), which means that for any run over about 2 million cells, you should aim to use around 200 cores.
Line 30: Line 47:
 ===== Running a Fluent Job ===== ===== Running a Fluent Job =====
  
-On the CHPC cluster all simulations are submitted as jobs to the PBS Pro job scheduler which will assign your job to the appropriate queue.+On the CHPC cluster all simulations are submitted as jobs to the PBS Pro job scheduler which will assign your job to the appropriate queue.  Below are given instructions and examples, which are also further elaborated upon {{:​howto:​writing_a_cluster_job_script_for_fluent.pdf|here}}.
  
 Example job script: Example job script:
Line 36: Line 53:
 <file bash runFluent.qsub>​ <file bash runFluent.qsub>​
 #!/bin/bash #!/bin/bash
-##### The following line will request ​(virtual) nodes, each with 24 cores running 24 mpi processes for +##### The following line will request ​(virtual) nodes, each with 24 cores running 24 mpi processes for 
-##### a total of 216-way parallel. ​ Specifying memory requirement is unlikely to be necessary, as the +##### a total of 96-way parallel. ​ Specifying memory requirement is unlikely to be necessary, as the 
 ##### compute nodes have 128 GB each. ##### compute nodes have 128 GB each.
-#PBS -l select=9:​ncpus=24:​mpiprocs=24:​mem=32GB:​nodetype=haswell_reg+#PBS -l select=4:​ncpus=24:​mpiprocs=24:​mem=32GB:​nodetype=haswell_reg
 #### Check for license availability. ​ If insufficient licenses are available, job will be held back untill ​ #### Check for license availability. ​ If insufficient licenses are available, job will be held back untill ​
-####  licenses are available.  +####  licenses are available. ​  
-#PBS -l aa_r_cfd=1 +#PBS -l cfd_base=1 
-#PBS -l aa_r_hpc=200+#PBS -l anshpc=92
 ## For your own benefit, try to estimate a realistic walltime request. ​ Over-estimating the  ## For your own benefit, try to estimate a realistic walltime request. ​ Over-estimating the 
 ## wallclock requirement interferes with efficient scheduling, will delay the launch of the job, ## wallclock requirement interferes with efficient scheduling, will delay the launch of the job,
Line 50: Line 67:
 #PBS -P myprojectcode #PBS -P myprojectcode
 #PBS -l walltime=1:​00:​00 #PBS -l walltime=1:​00:​00
-#PBS -o /home/username/scratch/​FluentTesting/​fluent.out +#PBS -o /mnt/lustre3p/users/​username/​FluentTesting/​fluent.out 
-#PBS -e /home/username/scratch/​FluentTesting/​fluent.err+#PBS -e /mnt/lustre3p/users/​username/​FluentTesting/​fluent.err
 #PBS -m abe #PBS -m abe
 #PBS -M username@email.co.za #PBS -M username@email.co.za
Line 58: Line 75:
 #### have the correct environment. ​ Ensure that any OpenFOAM-related environment #### have the correct environment. ​ Ensure that any OpenFOAM-related environment
 #### settings have been removed. ​ #### settings have been removed. ​
-####### PLEASE NOTE THAT THE LICENSE SERVER ID HAS NOW CHANGED, IT IS NOW chpclic1 +####### PLEASE NOTE THAT THE LICENSE SERVER ID HAS NOW CHANGED, IT IS NOW login1 
-export LM_LICENSE_FILE=1055@chpclic1 +export LM_LICENSE_FILE=1055@login1 
-export ANSYSLMD_LICENSE_FILE=1055@chpclic1 +export ANSYSLMD_LICENSE_FILE=1055@login1 
-# Edit this next line to select the appropriate version. V 19.2, 19.1, 19.0, 18.2 and 17.2 are available+# Edit this next line to select the appropriate version.  
-export PATH=/​apps/​chpc/​compmech/​CFD/​ansys_inc/​v192/​fluent/​bin:​$PATH+export PATH=/​apps/​chpc/​compmech/​CFD/​ansys_inc/​v211/​fluent/​bin:​$PATH
 export FLUENT_ARCH=lnamd64 export FLUENT_ARCH=lnamd64
-#### There is no -d option available under PBS Pro, therefore ​ 
 #### explicitly set working directory and change to that. #### explicitly set working directory and change to that.
-export PBS_JOBDIR=/​home/username/scratch/​FluentTesting+export PBS_JOBDIR=/​mnt/lustre3p/users/​username/​FluentTesting
 cd $PBS_JOBDIR cd $PBS_JOBDIR
 nproc=`cat $PBS_NODEFILE | wc -l` nproc=`cat $PBS_NODEFILE | wc -l`
Line 79: Line 95:
   * It is not possible to generate images during the computation. ​ Instead, these have to be created interactively afterwards.   * It is not possible to generate images during the computation. ​ Instead, these have to be created interactively afterwards.
  
-The second method allows the use of a recorded journal file and also supports "on the fly" generation of images.  ​A dummy X-windows session has to be started with Xvfb (X virtual frame buffer) and set as a virtual display.  ​This virtual session is automatically terminated at the end of the job.  The following is an example of a PBS job script using this method:+The second method allows the use of a recorded journal file and also supports "on the fly" generation of images.  ​We have previously made use of the virtual frame buffer ​"​Xvfb"​ to enable this.  ​However, ​the frame buffer method has now been deprecated, simply use the addition ​of the command line options ''​-gu -driver null''​ to enable the generation of images.  The following is an example of a PBS job script using this method:
  
 <file bash runFluent.qsub>​ <file bash runFluent.qsub>​
 #!/bin/bash #!/bin/bash
-##### The following line will request ​(virtual) nodes, each with 24 cores running 24 mpi processes for +##### The following line will request ​(virtual) nodes, each with 24 cores running 24 mpi processes for 
-##### a total of 216-way parallel. +##### a total of 96-way parallel. 
-#PBS -l select=9:​ncpus=24:​mpiprocs=24:​mem=32GB:​nodetype=haswell_reg +#PBS -l select=4:​ncpus=24:​mpiprocs=24:​mem=32GB:​nodetype=haswell_reg 
-#### License resource request +#### License resource request.  ​ 
-#PBS -l aa_r_cfd=1 +#PBS -l cfd_base=1 
-#PBS -l aa_r_hpc=200+#PBS -l anshpc=92
 ## For your own benefit, try to estimate a realistic walltime request. ​ Over-estimating the  ## For your own benefit, try to estimate a realistic walltime request. ​ Over-estimating the 
 ## wallclock requirement interferes with efficient scheduling, will delay the launch of the job, ## wallclock requirement interferes with efficient scheduling, will delay the launch of the job,
Line 95: Line 111:
 #PBS -P myprojectcode #PBS -P myprojectcode
 #PBS -l walltime=1:​00:​00 #PBS -l walltime=1:​00:​00
-#PBS -o /home/username/scratch/​FluentTesting/​fluent.out +#PBS -o /mnt/lustre3p/users/​username/​FluentTesting/​fluent.out 
-#PBS -e /home/username/scratch/​FluentTesting/​fluent.err+#PBS -e /mnt/lustre3p/users/​username/​FluentTesting/​fluent.err
 #PBS -m abe #PBS -m abe
 #PBS -M username@email.co.za #PBS -M username@email.co.za
Line 103: Line 119:
 #### have the correct environment. ​ Ensure that any OpenFOAM-related environment #### have the correct environment. ​ Ensure that any OpenFOAM-related environment
 #### settings have been removed. ​ #### settings have been removed. ​
-####### PLEASE NOTE THAT THE LICENSE SERVER ID HAS NOW CHANGED, IT IS chpclic1 +####### PLEASE NOTE THAT THE LICENSE SERVER ID HAS NOW CHANGED, IT IS login1 
-export LM_LICENSE_FILE=1055@chpclic1 +export LM_LICENSE_FILE=1055@login1 
-export ANSYSLMD_LICENSE_FILE=1055@chpclic1 +export ANSYSLMD_LICENSE_FILE=1055@login1 
-# Edit this next line to select the appropriate version.  ​V 20.1, 19.5, 19.4, 19.3, 19.2, 19.1, 19.0, 18.2 and 17.2  are available. +# Edit this next line to select the appropriate version. ​  
-export PATH=/​apps/​chpc/​compmech/​CFD/​ansys_inc/​v192/​fluent/​bin:​$PATH+export PATH=/​apps/​chpc/​compmech/​CFD/​ansys_inc/​v211/​fluent/​bin:​$PATH
 export FLUENT_ARCH=lnamd64 export FLUENT_ARCH=lnamd64
-#### There is no -d option available under PBS Pro, therefore ​ 
 #### explicitly set working directory and change to that. #### explicitly set working directory and change to that.
-export PBS_JOBDIR=/​home/username/scratch/​FluentTesting+export PBS_JOBDIR=/​mnt/lustre3p/users/​username/​FluentTesting
 cd $PBS_JOBDIR cd $PBS_JOBDIR
 nproc=`cat $PBS_NODEFILE | wc -l` nproc=`cat $PBS_NODEFILE | wc -l`
 exe=fluent exe=fluent
-### X11 required to save images. ​ Use Virtual frame buffer, which needs to be killed after completion +$exe 3d -t$nproc -pinfiniband -ssh -cnf=$PBS_NODEFILE ​-gu -driver null -i journalFile.jou > run.out 
-/bin/Xvfb :1 & +
-export DISPLAY=:​1 +
-$exe 3d -t$nproc -pinfiniband -ssh -cnf=$PBS_NODEFILE -i journalFile.jou > run.out +
-kill -9 %1+
 </​file>​ </​file>​
  
Line 153: Line 165:
 === Set up the appropriate environment === === Set up the appropriate environment ===
 <​code>​ <​code>​
-export LM_LICENSE_FILE=1055@chpclic1 +export LM_LICENSE_FILE=1055@login1 
-export ANSYSLMD_LICENSE_FILE=1055@chpclic1+export ANSYSLMD_LICENSE_FILE=1055@login1
 export PATH=/​apps/​chpc/​compmech/​CFD/​ansys_inc/​v181/​fluent/​bin:​$PATH export PATH=/​apps/​chpc/​compmech/​CFD/​ansys_inc/​v181/​fluent/​bin:​$PATH
 export FLUENT_ARCH=lnamd64 export FLUENT_ARCH=lnamd64
Line 164: Line 176:
  
 === Remote Solution Monitoring and Control === === Remote Solution Monitoring and Control ===
-Starting with Version 19.0 of the software, it is now possible to use a GUI to connect to a Fluent process that is already running. ​ The process requires that the Fluent be started with access to an X-server, therefore use a PBS script ​that contains the instructions to start up and remove a virtual frame buffer.  Here is a minimalist example of such a script:  ​+Starting with Version 19.0 of the software, it is now possible to use a GUI to connect to a Fluent process that is already running. ​ The process requires that the Fluent be started with access to an X-server, therefore use a run command ​that contains the parameters ''​-gu -driver null''​.  Here is a minimalist example of such a script:  ​
  
 <file bash runFluentWith_flremote.qsub>​ <file bash runFluentWith_flremote.qsub>​
Line 172: Line 184:
 #PBS -P MECH1234 #PBS -P MECH1234
 #PBS -l walltime=12:​00:​00 #PBS -l walltime=12:​00:​00
-#PBS -o /home/user/lustre/​FluentTest/​fluent.out +#PBS -o /mnt/lustre3p/users/​username/​FluentTest/​fluent.out 
-#PBS -e /home/user/lustre/​FluentTest/​fluent.err +#PBS -e /mnt/lustre3p/users/​username/​FluentTest/​fluent.err 
-#PBS -l aa_r_cfd=1 +#PBS -l cfd_base=1 
-#PBS -l aa_r_hpc=104 +#PBS -l anshpc=116 
-/bin/Xvfb :1 & +export LM_LICENSE_FILE=1055@login1 
-export DISPLAY=:1 +export ANSYSLMD_LICENSE_FILE=1055@login1 
-export LM_LICENSE_FILE=1055@chpclic1 +export PATH=$PATH:/​apps/​chpc/​compmech/​CFD/​ansys_inc/​v211/fluent/bin
-export ANSYSLMD_LICENSE_FILE=1055@chpclic1 +
-export PATH=$PATH:/​apps/​chpc/​compmech/​CFD/​ansys_inc/​190/fluent/bin+
 export FLUENT_ARCH=lnamd64 export FLUENT_ARCH=lnamd64
-cd /home/user/lustre/FluentTest+cd /mnt/lustre3p/users/​username/FluentTest
 nproc=`cat $PBS_NODEFILE | wc -l` nproc=`cat $PBS_NODEFILE | wc -l`
-fluent 3ddp -t$nproc -pinfiniband -ssh -mpi=intel -cnf=$PBS_NODEFILE ​ -i runCommands.txt | tee fluentrun.out +fluent 3ddp -t$nproc -pinfiniband -ssh -mpi=intel -cnf=$PBS_NODEFILE ​-gu -driver null -i runCommands.txt | tee fluentrun.out 
-kill -9 %1+
 </​file>​ </​file>​
  
/var/www/wiki/data/attic/howto/ansys.1588845829.txt.gz · Last modified: 2020/05/07 12:03 by ccrosby