WRF-3.7, compiled with the Intel compilers and using Intel MPI, is installed on the filesystem in /opt/gridware/non-supported/WRF-3.7-impi. Tests have indicated a useful performance benefit from using the Intel rather than Gnu compiler and OpenMPI. Please note that it is essential to set the unlimited stack size for the Intel-compiled version, as done in the script below. To set up an appropriate environment, “source” the setWRF file with the following command: . /opt/gridware/non-supported/WRF-3.7-impi/setWRF . This command should be placed in the PBS-Pro job submission script, but should also be placed in the user's .bashrc file, to ensure that the WRF environment is available when needed. Users need to develop their own workflows, but it is also practical to execute the pre-processing steps geogrid.exe, ungrib.exe, metgrid.exe and real.exe in single node mode with an interactive session. Simply give the command qsub -I to obtain an interactive session. Do not try to run these pre-processing steps from the login shell, as the shared login node cannot sustain a high work load. The real.exe pre-processing step for large cases may run into memory constraints. In that case, run real.exe in parallel over the requested number of nodes, but with only one process per node, as per the example script.

WRF-3.6.1 is installed on the filesystem in /opt/gridware/non-supported/WRF-3.6.1-gcc-ompi. Versions compiled with the Intel compiler and / or other MPI's are installed in similarly named directories. However, this GCC-compiled version with Open-MPI has been tested and is known to work.

WRF-3.6.1 with the added chemistry model is available in /opt/gridware/non-supported/WRF_chem-3.6.1-gcc-ompi. As per the above instructions, source the setWRF script in that directory to set up a suitable environment.

#!/bin/bash 
#### For the distributed memory versions of the code that we use at CHPC, mpiprocs should be equal to ncpus
#### ncpus=8 will select for Nehalem cluster, ncpus=12 will select for Dell or Westmere
#PBS -l select=4:ncpus=8:mpiprocs=8:mem=10GB
#PBS -q workq
#PBS -l walltime=3:00:00
#PBS -o /export/home/username/scratch/WRFV3_test/run/stdout
#PBS -e /export/home/username/scratch/WRFV3_test/run/stderr
#PBS -m abe
#PBS -M username@unseenuniversity.ac.za
### Source the WRF-3.7 environment:
export WRFDIR=/opt/gridware/non-supported/WRF-3.7-impi
. $WRFDIR/setWRF
# Set the stack size unlimited for the intel compiler
ulimit -s unlimited
##### Running commands
# Set PBS_JOBDIR to where YOUR simulation will be run
export PBS_JOBDIR=/export/home/username/scratch/WRFV3_test/run
# First though, change to YOUR WPS directory
export WPS_DIR=/export/home/username/scratch/WPS_test
cd $WPS_DIR
# Clean the directory of old files
rm FILE*
rm GRIB*
rm geo_em*
rm met_em*
# Link to the grib files, obviously use the location of YOUR grib files
./link_grib.csh ../DATA_test/GFS_* 
# Run geogrid.exe
mpirun -np 1 -machinefile $PBS_NODEFILE geogrid.exe &> geogrid.out
# Run ungrib.exe
mpirun -np 1 -machinefile $PBS_NODEFILE ungrib.exe &> ungrib.out
# Run metgrid.exe
mpirun -np 1 -machinefile $PBS_NODEFILE metgrid.exe &> metgrid.out
# Now change to the main job directory
cd $PBS_JOBDIR
# Link the met_em* data files into this directory
ln -s $WPS_DIR/met_em* ./
# Figure out how many processes to use for wrf.exe
nproc=`cat $PBS_NODEFILE | wc -l`
# Now figure out how many nodes are being used
cat $PBS_NODEFILE | sort -u > hosts
# Number of nodes to be used for real.exe
nnodes=`cat hosts | wc -l`
# Run real.exe with one process per node
exe=$WRFDIR/WRFV3/run/real.exe
mpirun -np $nnodes -machinefile hosts $exe &> real.out
# Run wrf.exe with the full number of processes
exe=$WRFDIR/WRFV3/run/wrf.exe
mpirun -np $nproc -machinefile $PBS_NODEFILE $exe &> wrf.out

For post-processing, ARW-Post, NCL and GrADS have been installed, and the necessary paths and environment variables set up by sourcing the setWRF file. In addition, ncview is also available as /opt/gridware/non-supported/ncview/bin/ncview. For graphics, refer to the Remote Visualization page for instructions on setting up a VNC session.