howto:wrf
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
howto:wrf [2019/05/20 12:16] ccrosby [Parallel scaling] |
howto:wrf [2022/10/27 18:42] (current) thlatshwayo |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Running ARW / WRF at the CHPC ====== | ====== Running ARW / WRF at the CHPC ====== | ||
| - | There are several versions of WRF, using different combinations of compiler and MPI implementation, | + | |
| + | **NEW: | ||
| + | |||
| + | There are several versions of WRF, using different combinations of compiler and MPI implementation, | ||
| + | |||
| + | ====== Warning: mpirun parameters changed ====== | ||
| + | |||
| + | It has been necessary to change some of our MPI installations. | ||
| + | |||
| + | '' | ||
| + | |||
| + | **The simple solution to this error is to remove the '' | ||
| + | |||
| ==== OpenMP ==== | ==== OpenMP ==== | ||
| - | WRF-4.0 | + | WRF-4.0, WRF-4.1.1 and WRF-4.2 have been installed with support for OpenMP. It is therefore possible to run using the same number of cores in total, but fewer MPI processes. |
| ==== WRF, Parallel NetCDF and I/O Quilting ==== | ==== WRF, Parallel NetCDF and I/O Quilting ==== | ||
| Line 43: | Line 56: | ||
| #PBS -l select=10: | #PBS -l select=10: | ||
| #PBS -l walltime=3: | #PBS -l walltime=3: | ||
| - | #PBS -o /home/username/scratch/ | + | #PBS -o /mnt/lustre/users/ |
| - | #PBS -e /home/username/scratch/ | + | #PBS -e /mnt/lustre/users/jblogs/ |
| #PBS -m abe | #PBS -m abe | ||
| - | #PBS -M username@unseenuniversity.ac.za | + | #PBS -M jblogs@unseenuniversity.ac.za |
| - | ### Source the WRF-3.8 environment: | + | ### Source the WRF-4.1.1 environment: |
| - | export WRFDIR=/ | + | export WRFDIR=/ |
| . $WRFDIR/ | . $WRFDIR/ | ||
| # Set the stack size unlimited for the intel compiler | # Set the stack size unlimited for the intel compiler | ||
| Line 54: | Line 67: | ||
| ##### Running commands | ##### Running commands | ||
| # Set PBS_JOBDIR to where YOUR simulation will be run | # Set PBS_JOBDIR to where YOUR simulation will be run | ||
| - | export PBS_JOBDIR=/ | + | export PBS_JOBDIR=/ |
| # First though, change to YOUR WPS directory | # First though, change to YOUR WPS directory | ||
| - | export WPS_DIR=/export/home/username/scratch/WPS_test | + | export WPS_DIR=/mnt/lustre/users/jblogs/WPS_test |
| cd $WPS_DIR | cd $WPS_DIR | ||
| # Clean the directory of old files | # Clean the directory of old files | ||
| Line 82: | Line 95: | ||
| nnodes=`cat hosts | wc -l` | nnodes=`cat hosts | wc -l` | ||
| # Run real.exe with one process per node | # Run real.exe with one process per node | ||
| - | exe=$WRFDIR/ | + | exe=$WRFDIR/ |
| mpirun -np $nnodes -machinefile hosts $exe &> real.out | mpirun -np $nnodes -machinefile hosts $exe &> real.out | ||
| # Run wrf.exe with the full number of processes | # Run wrf.exe with the full number of processes | ||
| - | exe=$WRFDIR/ | + | exe=$WRFDIR/ |
| - | mpirun -np $nproc | + | mpirun -np $nproc $exe &> wrf.out |
| </ | </ | ||
| Line 101: | Line 114: | ||
| #PBS -l select=10: | #PBS -l select=10: | ||
| #PBS -l walltime=3: | #PBS -l walltime=3: | ||
| - | #PBS -o /home/username/scratch/ | + | #PBS -o /mnt/lustre/users/ |
| - | #PBS -e /home/username/scratch/ | + | #PBS -e /mnt/lustre/users/jblogs/ |
| #PBS -m abe | #PBS -m abe | ||
| - | #PBS -M username@unseenuniversity.ac.za | + | #PBS -M jblogs@unseenuniversity.ac.za |
| - | ### Source the WRF-3.8 environment with parallel NetCDF: | + | ### Source the WRF-4.1.1 environment with parallel NetCDF: |
| - | export WRFDIR=/ | + | export WRFDIR=/ |
| . $WRFDIR/ | . $WRFDIR/ | ||
| # Set the stack size unlimited for the intel compiler | # Set the stack size unlimited for the intel compiler | ||
| Line 112: | Line 125: | ||
| ##### Running commands | ##### Running commands | ||
| # Set PBS_JOBDIR to where YOUR simulation will be run | # Set PBS_JOBDIR to where YOUR simulation will be run | ||
| - | export PBS_JOBDIR=/ | + | export PBS_JOBDIR=/ |
| cd $PBS_JOBDIR | cd $PBS_JOBDIR | ||
| - | exe=$WRFDIR/ | + | exe=$WRFDIR/ |
| # Clear and re-set the lustre striping for the job directory. | # Clear and re-set the lustre striping for the job directory. | ||
| # used by CHPC, a stripe size of 12 should work well. | # used by CHPC, a stripe size of 12 should work well. | ||
| Line 121: | Line 134: | ||
| ## For this example, assume that nproc_x=8, nproc_y=28, nio_tasks_per_group=4 and nio_groups=4, | ## For this example, assume that nproc_x=8, nproc_y=28, nio_tasks_per_group=4 and nio_groups=4, | ||
| ## of 16 I/O processes and 228 solver processes, therefore 240 MPI processes in total. | ## of 16 I/O processes and 228 solver processes, therefore 240 MPI processes in total. | ||
| - | mpirun -np 240 -machinefile $PBS_NODEFILE | + | mpirun -np 240 $exe &> wrf.out |
| </ | </ | ||
| Line 134: | Line 147: | ||
| #PBS -l walltime=06: | #PBS -l walltime=06: | ||
| #PBS -N WRF4-10X6X4 | #PBS -N WRF4-10X6X4 | ||
| - | #PBS -o /home/userid/ | + | #PBS -o /mnt/lustre/ |
| - | #PBS -e /home/userid/ | + | #PBS -e /mnt/lustre/ |
| ### These two stack size settings are essential for use with Intel-compiled code | ### These two stack size settings are essential for use with Intel-compiled code | ||
| ulimit -s unlimited | ulimit -s unlimited | ||
| export OMP_STACKSIZE=2G | export OMP_STACKSIZE=2G | ||
| ### Source the appropriate environment script | ### Source the appropriate environment script | ||
| - | . / | + | . / |
| - | export PBSJOBDIR=/home/userid/ | + | export PBSJOBDIR=/mnt/lustre/ |
| cd $PBSJOBDIR | cd $PBSJOBDIR | ||
| ### Get total number of MPI ranks | ### Get total number of MPI ranks | ||
| Line 150: | Line 163: | ||
| ### Issue the command line, passing the number of OpenMP threads. | ### Issue the command line, passing the number of OpenMP threads. | ||
| ### These affinity settings work OK, but may be unnecessary. YMMV. | ### These affinity settings work OK, but may be unnecessary. YMMV. | ||
| - | time mpirun | + | time mpirun -np $nproc -genv OMP_NUM_THREADS 4 -genv KMP_AFFINITY " |
| </ | </ | ||
| Line 190: | Line 203: | ||
| The bar graph explores the different parallelisation options with WRF-4.0. | The bar graph explores the different parallelisation options with WRF-4.0. | ||
| * Using 24 cores per node instead of 12 cores produces a relatively modest improvement | * Using 24 cores per node instead of 12 cores produces a relatively modest improvement | ||
| - | * Trading off MPI processes in favour of more OpenMP threads improves performance up to 4 OpenMP threads | + | * Trading off MPI processes in favour of more OpenMP threads improves performance up to 4 OpenMP threads, when using the Intel-compiled version |
| * Although there are many options for setting process and thread affinity, the defaults generally work quite well | * Although there are many options for setting process and thread affinity, the defaults generally work quite well | ||
| - | * For this particular problem set on only 10 nodes, there is a small penalty associated with using parallel I/O, but this will be reversed if more nodes are used. | + | * For this particular problem set on only 10 nodes, there is a small penalty associated with using parallel I/O, but this will be reversed if more nodes are used |
| + | * Although the PGI version is competitive with the Intel version when using MPI only, it does not benefit from using OpenMP processes as well. The GCC version is generally not competitive, | ||
| {{: | {{: | ||
/app/dokuwiki/data/attic/howto/wrf.1558347403.txt.gz · Last modified: 2021/12/09 16:42 (external edit)
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
