| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
howto:openfoam [2023/07/25 16:03]
ccrosby |
howto:openfoam [2026/02/24 13:03] (current)
ccrosby [Latest version: OpenFOAM-v2512] |
| - [[openfoam#compiling_your_own_openfoam_code|Compiling your own OpenFOAM code]] | - [[openfoam#compiling_your_own_openfoam_code|Compiling your own OpenFOAM code]] |
| - [[howto:openfoam:hisa|Using the HiSA compressible Solver for OpenFOAM]] | - [[howto:openfoam:hisa|Using the HiSA compressible Solver for OpenFOAM]] |
| | |
| | ====== Latest version: OpenFOAM-v2512 ====== |
| | The most recent available version is OpenFOAM-v2512. In contrast to previous installations, this has been compiled with Intel's OneAPI compiler and runs under IntelMPI. No special settings or parameters outside of the source script OF2512 are required. Here is an example job script: |
| | |
| | <file bash runFoam2512.qsub> |
| | #!/bin/bash |
| | ### This 72-way example calls |
| | ### for 3 (virtual) nodes, each with 24 processor cores |
| | ### Please note that it is necessary to specify both ncpus and mpiprocs, and |
| | ### for OpenFOAM these should be identical to each other. |
| | ### For your own benefit, try to estimate a realistic walltime request. Over-estimating the |
| | ### wallclock requirement interferes with efficient scheduling, will delay the launch of the job, |
| | ### and ties up more of your CPU-time allocation untill the job has finished. |
| | #PBS -P projectid |
| | #PBS -l select=3:ncpus=24:mpiprocs=24:nodetype=haswell_reg |
| | #PBS -q normal |
| | #PBS -l walltime=01:00:00 |
| | #PBS -o /mnt/lustre/users/username/foamJobs/job01/stdout |
| | #PBS -e /mnt/lustre/users/username/foamJobs/job01/stderr |
| | ### Source the openFOAM environment: |
| | . /home/apps/chpc/compmech/OpenFOAM/OF2512 |
| | # Set this environment variable explicitly. |
| | export PBS_JOBDIR=/mnt/lustre/users/username/foamJobs/job01 |
| | # Explicitly change to the job directory |
| | cd $PBS_JOBDIR |
| | # Count the number of lines in the hosts file to get the number of processors |
| | nproc=`cat $PBS_NODEFILE | wc -l ` |
| | #### These next statements build an appropriate decomposeParDict file |
| | #### based on the requested nunber of processors |
| | echo "FoamFile" > system/decomposeParDict |
| | echo "{" >> system/decomposeParDict |
| | echo " version 2.0;" >> system/decomposeParDict |
| | echo " format ascii;" >> system/decomposeParDict |
| | echo " class dictionary;" >> system/decomposeParDict |
| | echo " object decomposeParDict;" >> system/decomposeParDict |
| | echo "}" >> system/decomposeParDict |
| | echo "numberOfSubdomains " $nproc ";" >> system/decomposeParDict |
| | echo "method scotch;" >> system/decomposeParDict |
| | #### End of decomposeParDict file |
| | # Decompose the case into processor directories |
| | decomposePar -force > decompose.out |
| | # Run the solver, simpleFoam in this case |
| | mpirun -np $nproc simpleFoam -parallel > simpleFoam.out |
| | # Reconstruct the time directories from the processor directories |
| | reconstructPar > reconstruct.out |
| | # Clean up the processor directories |
| | rm -r processor* |
| | </file> |
| |
| ====== Warning: mpirun parameters changed ====== | ====== Warning: mpirun parameters changed ====== |
| |
| It has been necessary to change some of our MPI installations. The reason for this is that these installations were compiled without support for the PBS scheduler, which could result in orphaned processes left on the cluster if a job failed for any reason. The revised MPI installations keep all MPI ranks under full control of PBS, which means that they should be cleaned up properly in the event of a job failure. However, uers may now experience an incompatibility with the mpirun command. A typical error message may look like this: | It has been necessary to change some of our MPI installations. The reason for this is that these installations were compiled without support for the PBS scheduler, which could result in orphaned processes left on the cluster if a job failed for any reason. The revised MPI installations keep all MPI ranks under full control of PBS, which means that they should be cleaned up properly in the event of a job failure. However, users may now experience an incompatibility with the mpirun command. A typical error message may look like this: |
| |
| ''HYDT_bsci_init (tools/bootstrap/src/bsci_init.c:175): unrecognized RMK: user'' | ''HYDT_bsci_init (tools/bootstrap/src/bsci_init.c:175): unrecognized RMK: user'' |
| ===== Running OpenFOAM ===== | ===== Running OpenFOAM ===== |
| |
| This section describes how to run OpenFOAM on the cluster at CHPC. Versions 2.4.0, 3.0.1, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10, 11, v1706, v1712, v1806, v1812, v1906, v1912, v2006, v2012, v2106, v2112, v2206, v2212 and foam-extend-3.1, 3.2, 4.0 and 5.0 are installed in ''/apps/chpc/compmech/CFD/OpenFOAM''. Source the required environment from one of the files OF240, OF301, OF40, OF50, OF50mpich, OF60, OF70, OF80, OF90, OF10, OF11, OFextend31, OFextend32, OFextend40, OFextend50, OF1706, OF1712, OF1806, OF1906, OF1912, OF2006, OF2012 OF2106, OF2112, OF2206 or OF2212 respectively, as illustrated in the example job scripts below. Please be aware that versions after 5.0 and 18.06 use MPICH rather than OpenMPI, and the mpirun command ** has to ** select the use of the Infiniband network interface. Also, you may need to source the file from your .bashrc file, to ensure that the compute nodes have the right environment. This ** should ** not be necessary, but it is sometimes a useful workaround if you experience MPI troubles. It is assumed that you are familiar with running OpenFOAM solvers in parallel, and have already set up the case directory. The Gmsh and cfMesh meshing utilities are also installed and are added to the executable path when any of the OFxxx files are sourced. OpenFOAM-extend-* has been installed on an experimental basis. To enable it, source the appropriate file in /apps/chpc/compmech/CFD/OpenFOAM/, such as OFextend32. | This section describes how to run OpenFOAM on the cluster at CHPC. Versions 6.0, 7.0, 8.0, 9.0, 10, 11, 12, 13, v1906, v1912, v2006, v2012, v2106, v2112, v2206, v2212, v2306, v2312, v2406, v2412, v2506, v2512 and foam-extend-3.1, 3.2, 4.0 and 5.0 are installed in ''/apps/chpc/compmech/CFD/OpenFOAM'' or ''/home/apps/chpc/compmech/OpenFOAM''. Source the required environment from one of the files OF60, OF70, OF80, OF90, OF10, OF11, OF12, OF13, OFextend31, OFextend32, OFextend40, OFextend50, OF1906, OF1912, OF2006, OF2012 OF2106, OF2112, OF2206, OF2212, OF2306, OF2312, OF2406, OF2412, OF2506 and OF2512 respectively, as illustrated in the example job scripts below. Please be aware that versions after 5.0 and 18.06 use MPICH rather than OpenMPI, and the mpirun command ** has to ** select the use of the Infiniband network interface. However, as an experiment, OpenFOAM-v2412 has been compiled against OpenMPI, and for this version, ** do not specify -iface ib0**. It is assumed that you are familiar with running OpenFOAM solvers in parallel, and have already set up the case directory. The Gmsh and cfMesh meshing utilities are also installed and are added to the executable path when any of the OFxxx files are sourced. OpenFOAM-extend-* has been installed on an experimental basis. To enable it, source the appropriate file in /apps/chpc/compmech/CFD/OpenFOAM/, such as OFextend32. |
| |
| === Using MPICH instead of OpenMPI === | === Using MPICH instead of OpenMPI === |
| There is an additional version of OpenFOAM-5.0 that uses MPICH-3.2 rather than OpenMPI. The Open-FOAM-6.0, OpenFOAM-7.0, OpenFOAM-8.0, OpenFOAM-9.0, v1812, v1906, v1912, v2006, v2012, v2106 and v2112 installations are only available with MPICH. The source file for the OpenFOAM-5.0 version is OF50mpich , and the mpirun command in the script ** must ** be modified to read ''mpirun -iface ib0'', to force mpich to use the Infiniband network rather than Ethernet. | There is an additional version of OpenFOAM-5.0 that uses MPICH-3.2 rather than OpenMPI. From Open-FOAM-6.0 and v1812, the installations are only available with MPICH. The source file for the OpenFOAM-5.0 version is OF50mpich , and the mpirun command in the script ** must ** be modified to read ''mpirun -iface ib0'', to force mpich to use the Infiniband network rather than Ethernet. |
| |
| === OpenFOAM versions === | === OpenFOAM versions === |
