wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » fds
Trace:
howto:fds

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
howto:fds [2018/07/26 14:33]
ccrosby created 		howto:fds [2022/10/05 13:35] (current)
ccrosby [Postprocessing]
Line 1: Line 1:
 ====== Running the open-source Fire Dynamics Simulator FDS on the CHPC system ====== ====== Running the open-source Fire Dynamics Simulator FDS on the CHPC system ======
 ===== Introduction ===== ===== Introduction =====
-Please refer to [[https://pages.nist.gov/fds-smv/|the FDS home page]] for more information on FDS, an open-source code targeted at fire simulation.  The current version of the code is 6.7.0, and has been installed in '''/apps/chpc/compmech/CFD/FDS'''.  There is a module for FDS installed on Lengau.  ''module load chpc/compmech/SU2/6.1'' will set up the environment for the latest version.+Please refer to [[https://pages.nist.gov/fds-smv/|the FDS home page]] for more information on FDS, an open-source code targeted at fire simulation.  There are two installed versions.  The older version of the code is 6.7.0, and has been installed in '''/apps/chpc/compmech/CFD/FDS'''.  There is a module for FDS installed on Lengau.  ''module load chpc/compmech/FDS/6.7.0'' will set up the appropriate environment.
  
-===== Sample PBS script ===== +The most recent version is FDS-6.7.9.  This is installed in ''/home/apps/chpc/compmech/FDS/FDS-6.7.9'', along with Smokeview-6.7.21.  There is also a directory containing sample cases in ''/home/apps/chpc/compmech/FDS/FDS-6.7.9/Examples''.  To use these versions, load the following modules: 
-When running in an interactive environment, SU2 makes use of python scripting to automate  +<code> 
-tasks.  For cluster running, it is necessary to ''qsub'' a PBS script.  This example executes the classic +module load chpc/compmech/FDS/FDS-6.7.9 
-Onera M6 transonic wing example from the TestCases/euler/oneram6 directory.+module load chpc/compmech/FDS/SMV-6.7.21 
 +</code> 
 + 
 +This will give you access to two fds executables: 
 +  - fds has been compiled without support for OpenMP and should be used with MPI only 
 +  - fds_openmp has been compiled with OpenMP support and can be used for single node and hybrid parallel runs 
 + 
 +If you intend using MPI parallel, it will be necessary to provide a machinefile for this version, which has not been compiled with one of the system MPI compilers.  A typical command line will thus be: 
 +<code> 
 +mpirun -np 48 -machinefile $PBS_NODEFILE fds box_burn_away1.fds > fds.out 
 +</code> 
 + 
 +If you want to run Smokeview on a compute node, it will be necessary to load a module for Mesa software rendering: 
 +<code> 
 +module load chpc/compmech/mesa/20.2.2_swr 
 +</code> 
 + 
 + 
 +===== Sample PBS script for version 6.7.9 ===== 
 + 
 +<file bash runFDS679.qsub> 
 +#!/bin/bash 
 +## Lines starting with the # symbol are comments, unless followed by ! or PBS,  
 +##  in which case they are directives 
 +## The following PBS directive requests two 24 core compute nodes 
 +##  The $PBS_NODEFILE file will contain the hostnames for 16 MPI processes (2 X 8, as per mpiprocs) 
 +##  This is only meaningful if your model contains 16 grids 
 +##  The number of grids must match the number of MPI processes 
 +#PBS -l select=2:ncpus=24:mpiprocs=8 -q normal 
 +## Specify your own project shortcode here 
 +#PBS -P MECH1234 
 +## The walltime should be a small overestimate of the expected run time 
 +##  Requesting a very long walltime may delay the start of your job 
 +##  If the requested walltime is too short, the job will be killed before it is finished 
 +#PBS -l walltime=6:00:00 
 +##  Obviously use your own paths here 
 +#PBS -e /mnt/lustre/users/jblogs/FDS_Runs/stderr.txt 
 +#PBS -o /mnt/lustre/users/jblogs/FDS_Runs/stdout.txt 
 +export PBS_JOBDIR=/mnt/lustre/users/jblogs/FDS_Runs 
 +cd $PBS_JOBDIR 
 +module load chpc/compmech/FDS/FDS-6.7.9  
 +## Assign a sensible value for OMP_NUM_THREADS 
 +##  If your number of MPI processes is a multiple of ##  24, it will be best to set it to 1, 
 +##  and not use OpenMP at all 
 +##  A value greater than 3 does not help 
 +export OMP_NUM_THREADS=3 
 +## The number of MPI processes is extracted from the length of the machinefile $PBS_NODEFILE 
 +nproc=`cat $PBS_NODEFILE | wc -l` 
 +mpirun -np $nproc -machinefile $PBS_NODEFILE fds_openmp FDS_inputFile.fds > fds.out 
 +</file> 
 + 
 +===== Sample PBS script for version 6.7.0 =====
  
 <file bash runFDS.qsub> <file bash runFDS.qsub>
-#!/bin/bash  +#!/bin/bash 
-# This example script requests 3 nodes of the cluster, each with 24 processor cores +## Lines starting with the # symbol are comments, unless followed by ! or PBS,  
-#PBS -P projectid +##  in which case they are directives 
-#PBS -l select=3:ncpus=24:mpiprocs=24:nodetype=haswell_reg +## The following PBS directive requests two 24 core compute nodes 
-#PBS -q normal +##  The $PBS_NODEFILE file will contain the hostnames for 16 MPI processes (2 X 8, as per mpiprocs) 
-#PBS -l walltime=01:00:00 +##  This is only meaningful if your model contains 16 grids 
-#PBS -o /home/username/scratch/foamJobs/job01/stdout +##  The number of grids must match the number of MPI processes 
-#PBS -e /home/username/scratch/foamJobs/job01/stderr +#PBS -l select=2:ncpus=24:mpiprocs=8 -q normal 
-#PBS -m abe +## Specify your own project shortcode here 
-#PBS -M user@wherever.co.za +#PBS -P MECH1234 
-#### Setup environment for SU2 +## The walltime should be a small overestimate of the expected run time 
-module add chpc/compmech/SU2/6.1  +##  Requesting a very long walltime may delay the start of your job 
-# The -d option used with moab no longer works.  Set this environment variable explicitly. +##  If the requested walltime is too short, the job will be killed before it is finished 
-export PBS_JOBDIR=/export/home/user/SU2/TestCases/euler/oneram6 +#PBS -l walltime=6:00:00 
-# Explicitly change to the job directory+##  Obviously use your own paths here 
 +#PBS -e /mnt/lustre/users/jblogs/FDS_Runs/stderr.txt 
 +#PBS -o /mnt/lustre/users/jblogs/FDS_Runs/stdout.txt 
 +export PBS_JOBDIR=/mnt/lustre/users/jblogs/FDS_Runs
 cd $PBS_JOBDIR cd $PBS_JOBDIR
 +module load chpc/compmech/FDS/6.7.0
 +## Assign a sensible value for OMP_NUM_THREADS
 +##  If your number of MPI processes is a multiple of ##  24, it will be best to set it to 1,
 +##  and not use OpenMP at all
 +##  A value greater than 3 does not help
 +export OMP_NUM_THREADS=3
 +## The number of MPI processes is extracted from the length of the machinefile $PBS_NODEFILE
 nproc=`cat $PBS_NODEFILE | wc -l` nproc=`cat $PBS_NODEFILE | wc -l`
-### Run the SU2_CFD solver +mpirun -np $nproc fds FDS_inputFile.fds > fds.out
-mpirun -np $nproc -machinefile $PBS_NODEFILE SU2_CFD inv_ONERAM6_JST.cfg > SU2_CFD.out +
-### Check to see if the solution_flow.dat file exists.  If it does,  +
-###  use it to generate the necessary output files.  If it doesn't,  +
-###  copy the restart file to solution_flow.dat and generate the output. +
-### Make sure that these two file names are consistent with what you have  +
-###  used in the .cfg file for the run. +
-if [ ! -e solution_flow.dat ] ; +
- then +
-  cp restart_flow.dat solution_flow.dat +
-fi +
-### Build the specified format output files +
-SU2_SOL inv_ONERAM6_JST.cfg > SU2_SOL.out+
 </file> </file>
  
-===== Postprocesing ===== +===== Postprocessing ===== 
-By default, the SU2 example cases write Tecplot format output files.  CHPC does not have a license  +Smokeview is also installed on the system, and can be accessed with the command ''smokeview'' or ''smv''.  This can be done  on one of the visualisation nodes chpcviz1 or chpclic1.  Please read the instructions on [[howto:remote_viz|setting up a VNC connection]], and run Smokeview with the VirtualGL wrapper ''/opt/VirtualGL/bin/vglrun smokeview''.    
-for Tecplot, although it is possible to read Tecplot files into Paraview.  However, in the SU2 input file  + 
-it is easy to specify Paraview format output files (VTK) directly.  Refer to [[howto:remote_viz|the instructions on Remote Visualization]] to obtain a graphics environment for post-processing with VirtualGL and [[howto:paraview|Paraview]] or [[howto:visit|VisIt]].+If you want to run Smokeview on a compute node, please read the instructions at https://wiki.chpc.ac.za/howto:remote_viz#getting_a_virtual_desktop_on_a_compute_node.  You will need to load a mesa module to enable software rendering. 
 + 
 + It may also be practical to use the very well-developed visualisation codes [[howto:paraview|Paraview]] or [[howto:visit|VisIt]].  Please experiment and provide feedback. 
 + 
 +===== Parallel Scaling ===== 
 +FDS implements two forms of parallisation, using OpenMP threads as well as MPI-based domain decomposition.   
 + 
 + 
 +OpenMP provides only modest improvement in performance, but has the advantage of also working with a single grid model.  Going from 1 OpenMP thread to 2 provides a modest but helpful improvement, and going to 3 threads will provide another very small improvement.  More than 3 OpenMP threads do not provide more improvement.  
 + 
 + 
 +MPI parallel will only work if the model has been set up in such a way that the number of grids is equal to the number of MPI processes.  Somewhat confusingly, the code will still run if this condition is not satisfied, but not efficiently.  If there are more MPI processes than grids, the extra MPI processes will start and consume CPU resources, but not do any useful work.  If there are more grids than MPI processes, the slowdown is quite dramatic.  MPI parallel scaling is very good, provided that the number of grids match the number of MPI processes, and are all similarly dimensioned.  The compute nodes in the Lengau cluster have 24 cores each.  Good MPI scaling and efficiency is therefore achieved by developing models where the number of grids is a multiple of 12 or 24.  Underloading the compute nodes, by running say 12 MPI processes, each with two OpenMP threads, will achieve the best results, at the expense of occupying more nodes.  This is a typical characteristic of the performance of any CFD code, which is strongly constrained by memory bandwidth.  Maximum performance is achieved by accessing the largest number of memory channels. 
 + 
 +=== Scaling graphs === 
 + 
 +{{:howto:fds_01.png|}} 
 +{{:howto:fds_02.png|}} 
 +{{:howto:fds_03.png|}} 
 +{{:howto:fds_04.png|}} 
 +{{:howto:fds_05.png|}} 
 +{{:howto:fds_06.png|}} 
 + 
/app/dokuwiki/data/attic/howto/fds.1532608402.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki