WRF-3.6.1 is installed on the filesystem in /opt/gridware/non-supported/WRF-3.6.1-gcc-ompi. Versions compiled with the Intel compiler and / or other MPI's are installed in similarly named directories. However, this GCC-compiled version with Open-MPI has been tested and is known to work. To set up an appropriate environment “source” the setWRF file with the following command: . /opt/gridware/non-supported/WRF-3.6.1-gcc-ompi/setWRF . This command should be placed in the PBS-Pro job submission script, but should also be placed in the user's .bashrc file, to ensure that the WRF environment is available when needed. Users need to develop their own workflows, but it is practical to execute the pre-processing steps geogrid.exe, ungrib.exe, metgrid.exe and real.exe in single node mode with an interactive session. Simply give the command qsub -I to obtain an interactive session. Do not try to run these pre-processing steps from the login shell, as the shared login node cannot sustain a high work load. The real.exe pre-processing step for large cases may run into memory constraints. In that case, run real.exe in parallel over the requested number of nodes, but with only one process per node, as per this example script.

WRF-3.6.1 with the added chemistry model is available in /opt/gridware/non-supported/WRF_chem-3.6.1-gcc-ompi. As per the above instructions, source the setWRF script in that directory to set up a suitable environment.

#!/bin/bash 
#PBS -l select=10:ncpus=12:mpiprocs=12:jobtype=dell,place=free:group=nodetype
#PBS -q workq 
#PBS -l walltime=1:00:00
#PBS -o stdout
#PBS -e stderr
#PBS -m abe
#PBS -M username@csir.co.za
### Source the WRF-3.6.1 environment:  
### Please ensure that these lines also appear in your .bashrc file, 
### preferably right at the end
export WRFDIR=/opt/gridware/non-supported/WRF-3.6.1-gcc-ompi
. $WRFDIR/setWRF
##### Running commands
# The -d option used with moab no longer works.  Set this environment variable explicitly.
export PBS_JOBDIR=/export/home/username/scratch5/WRFV3/run
# Explicitly change to the job directory
cd $PBS_JOBDIR
# Figure out how many processes for wrf.exe
nproc=`cat $PBS_NODEFILE | wc -l`
# Now figure out how many nodes are being used
cat $PBS_NODEFILE | sort -u > hosts
nnodes=`cat hosts | wc -l`
# Run real.exe with one process per node
exe=$WRFDIR/WRFV3/run/real.exe
mpirun --bynode -np $nnodes -machinefile $PBS_NODEFILE $exe &> real.out
# Run wrf.exe with the full number of processes
exe=$WRFDIR/WRFV3/run/wrf.exe
mpirun -np $nproc -machinefile $PBS_NODEFILE $exe &> wrf.out   

For post-processing, ARW-Post, NCL and GrADS have been installed, and the necessary paths and environment variables set up by sourcing the setWRF file. For graphics, refer to the Remote Visualization page for instructions on setting up a VNC session.