| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
howto:wrf [2020/06/30 09:09]
ccrosby [OpenMP] |
howto:wrf [2022/10/27 18:42] (current)
thlatshwayo |
| ====== Running ARW / WRF at the CHPC ====== | ====== Running ARW / WRF at the CHPC ====== |
| | |
| | **NEW: [[https://youtu.be/beXd1pBEOYY|Howto video]] ** |
| | |
| There are several versions of WRF, using different combinations of compiler and MPI implementation, installed on the filesystem in ''/apps/chpc/earth/''. The latest version is WRF-4.2, built with the Intel compiler. Tests have indicated a very large benefit from using the Intel compiler and MPI rather than Gnu compiler and OpenMPI or MPICH. MPICH versions need the mpirun argument ''-iface ib0'' to force it to use the Infiniband network. Please note that it is essential to set the unlimited stack size for the Intel-compiled version, as done in the script below. To set up an appropriate environment, "source" the setWRF file in the required directory with the following type of command: ''. /apps/chpc/earth/WRF-3.8-impi/setWRF'' . This command should be placed in the PBS-Pro job submission script. Users need to develop their own workflows, but it is also practical to execute the pre-processing steps ''geogrid.exe, ungrib.exe, metgrid.exe and real.exe'' in single node mode with an interactive session. Simply give the command ''qsub -I -q smp -P <AAAA0000>'', where <AAAA0000> should be replaced with **your** project code, to obtain an interactive session. Do not try to run these pre-processing steps from the login shell, as the shared login node cannot sustain a high work load. The real.exe pre-processing step for large cases may run into memory constraints. In that case, run real.exe in parallel over the requested number of nodes, but with only one process per node, as per the example script. | There are several versions of WRF, using different combinations of compiler and MPI implementation, installed on the filesystem in ''/apps/chpc/earth/''. The latest version is WRF-4.2, built with the Intel compiler. Tests have indicated a very large benefit from using the Intel compiler and MPI rather than Gnu compiler and OpenMPI or MPICH. MPICH versions need the mpirun argument ''-iface ib0'' to force it to use the Infiniband network. Please note that it is essential to set the unlimited stack size for the Intel-compiled version, as done in the script below. To set up an appropriate environment, "source" the setWRF file in the required directory with the following type of command: ''. /apps/chpc/earth/WRF-3.8-impi/setWRF'' . This command should be placed in the PBS-Pro job submission script. Users need to develop their own workflows, but it is also practical to execute the pre-processing steps ''geogrid.exe, ungrib.exe, metgrid.exe and real.exe'' in single node mode with an interactive session. Simply give the command ''qsub -I -q smp -P <AAAA0000>'', where <AAAA0000> should be replaced with **your** project code, to obtain an interactive session. Do not try to run these pre-processing steps from the login shell, as the shared login node cannot sustain a high work load. The real.exe pre-processing step for large cases may run into memory constraints. In that case, run real.exe in parallel over the requested number of nodes, but with only one process per node, as per the example script. |
| |
| ====== Warning: mpirun parameters changed ====== | ====== Warning: mpirun parameters changed ====== |
| |
| It has been necessary to change some of our MPI installations. The reason for this is that these installations were compiled without support for the PBS scheduler, which could result in orphaned processes left on the cluster if a job failed for any reason. The revised MPI installations keep all MPI ranks under full control of PBS, which means that they should be cleaned up properly in the event of a job failure. However, uers may now experience an incompatibility with the mpirun command. A typical error message may look like this: | It has been necessary to change some of our MPI installations. The reason for this is that these installations were compiled without support for the PBS scheduler, which could result in orphaned processes left on the cluster if a job failed for any reason. The revised MPI installations keep all MPI ranks under full control of PBS, which means that they should be cleaned up properly in the event of a job failure. However, users may now experience an incompatibility with the mpirun command. A typical error message may look like this: |
| |
| ''HYDT_bsci_init (tools/bootstrap/src/bsci_init.c:175): unrecognized RMK: user'' | ''HYDT_bsci_init (tools/bootstrap/src/bsci_init.c:175): unrecognized RMK: user'' |
| #PBS -l select=10:ncpus=24:mpiprocs=24 -q normal -P TEST1234 | #PBS -l select=10:ncpus=24:mpiprocs=24 -q normal -P TEST1234 |
| #PBS -l walltime=3:00:00 | #PBS -l walltime=3:00:00 |
| #PBS -o /home/username/scratch/WRFV3_test/run/stdout | #PBS -o /mnt/lustre/users/jblogs/WRFV3_test/run/stdout |
| #PBS -e /home/username/scratch/WRFV3_test/run/stderr | #PBS -e /mnt/lustre/users/jblogs/WRFV3_test/run/stderr |
| #PBS -m abe | #PBS -m abe |
| #PBS -M username@unseenuniversity.ac.za | #PBS -M jblogs@unseenuniversity.ac.za |
| ### Source the WRF-4.1.1 environment: | ### Source the WRF-4.1.1 environment: |
| export WRFDIR=/apps/chpc/earth/WRF-4.1.1-pnc-impi | export WRFDIR=/apps/chpc/earth/WRF-4.1.1-pnc-impi |
| ##### Running commands | ##### Running commands |
| # Set PBS_JOBDIR to where YOUR simulation will be run | # Set PBS_JOBDIR to where YOUR simulation will be run |
| export PBS_JOBDIR=/home/username/scratch/WRFV3_test/run | export PBS_JOBDIR=/mnt/lustre/users/jblogs/WRFV3_test/run |
| # First though, change to YOUR WPS directory | # First though, change to YOUR WPS directory |
| export WPS_DIR=/export/home/username/scratch/WPS_test | export WPS_DIR=/mnt/lustre/users/jblogs/WPS_test |
| cd $WPS_DIR | cd $WPS_DIR |
| # Clean the directory of old files | # Clean the directory of old files |
| #PBS -l select=10:ncpus=24:mpiprocs=24 -q normal -P TEST1234 | #PBS -l select=10:ncpus=24:mpiprocs=24 -q normal -P TEST1234 |
| #PBS -l walltime=3:00:00 | #PBS -l walltime=3:00:00 |
| #PBS -o /home/username/scratch/WRFV3_test/run/stdout | #PBS -o /mnt/lustre/users/jblogs/WRFV3_test/run/stdout |
| #PBS -e /home/username/scratch/WRFV3_test/run/stderr | #PBS -e /mnt/lustre/users/jblogs/WRFV3_test/run/stderr |
| #PBS -m abe | #PBS -m abe |
| #PBS -M username@unseenuniversity.ac.za | #PBS -M jblogs@unseenuniversity.ac.za |
| ### Source the WRF-4.1.1 environment with parallel NetCDF: | ### Source the WRF-4.1.1 environment with parallel NetCDF: |
| export WRFDIR=/apps/chpc/earth/WRF-4.1.1-pnc-impi | export WRFDIR=/apps/chpc/earth/WRF-4.1.1-pnc-impi |
| ##### Running commands | ##### Running commands |
| # Set PBS_JOBDIR to where YOUR simulation will be run | # Set PBS_JOBDIR to where YOUR simulation will be run |
| export PBS_JOBDIR=/home/username/scratch/WRFV3_test/run | export PBS_JOBDIR=/mnt/lustre/users/jblogs/WRFV3_test/run |
| cd $PBS_JOBDIR | cd $PBS_JOBDIR |
| exe=$WRFDIR/WRF/run/wrf.exe | exe=$WRFDIR/WRF/run/wrf.exe |
| #PBS -l walltime=06:00:00 | #PBS -l walltime=06:00:00 |
| #PBS -N WRF4-10X6X4 | #PBS -N WRF4-10X6X4 |
| #PBS -o /home/userid/lustre/WRFrun/wrf4.out | #PBS -o /mnt/lustre/users/jblogs/WRFrun/wrf4.out |
| #PBS -e /home/userid/lustre/WRFrun/wrf4.err | #PBS -e /mnt/lustre/users/jblogs/WRFrun/wrf4.err |
| ### These two stack size settings are essential for use with Intel-compiled code | ### These two stack size settings are essential for use with Intel-compiled code |
| ulimit -s unlimited | ulimit -s unlimited |
| ### Source the appropriate environment script | ### Source the appropriate environment script |
| . /apps/chpc/earth/WRF-4.1.1-pnc-impi/setWRF | . /apps/chpc/earth/WRF-4.1.1-pnc-impi/setWRF |
| export PBSJOBDIR=/home/userid/lustre/WRFrun/wrf4.out | export PBSJOBDIR=/mnt/lustre/users/jblogs/WRFrun/wrf4.out |
| cd $PBSJOBDIR | cd $PBSJOBDIR |
| ### Get total number of MPI ranks | ### Get total number of MPI ranks |
