Author: M.S Mabakane Date: February, 08 2011

TAU (Tuning and Analysis Utilities) is a portable profiling and tracing toolkit that can be used to analyse the performance of parallel models written in Fortran, C, C++ and Python. It is also used to identify key areas such as mpi processes, message passing; in general; the performance of the application that occur during the execution of the model. Many various options are also available in the TAU package that can be selected to obtain more detail information about the performance of the model.

At CHPC, TAU is installed and tested in the CHPC Sun Microsystems cluster to help users improve the performance of their models. On the other hand, TAU is also anticipated to be a useful tool for developers who need to solve challenging errors of their programs. For this task, DLPOLY-2 was selected for the purpose of testing this tuning analysis system in the CHPC cluster. However, other models such as (WRF, CAM, POP and CCSM) were also tested by the TAU Developers (University of Oregon and University of Tennessee, USA) in their supercomputing systems.

Now, this is the procedure to port, compile and run the code using TAU:

Please note: typewriter characters depicts the command, directory or files in the system.

1. Login to the CHPC Sun system using -X mode: e.g. ssh -X username@sun.chpc.ac.za
2. In the command line, type the command: module add intel. The system will then add the intel compiler and open-mpi in the user's environment. To test the environment type: which ifort and the following path should be displayed: /opt/gridware/intel/Compiler/11.1/056/bin/intel64/ifort.
3. From then, type the command: module add tau. It will add the following path to tau libraries: /opt/gridware/performance/tau-2.19.1/x86_64/bin and TAU_MAKEFILE=/opt/gridware/performance/tau-2.19.1/x86_64/lib/Makefile.tau-mpi-pdt-trace in your profile. To test the TAU environment type: which tau_f90.sh, it should display these message: /opt/gridware/performance/tau-2.19.1/x86_64/bin/tau_f90.sh.
4. You are now ready to port and compile the model using TAU. Note: If your model is written in Fortran language, use the library named: tau_f90.sh to compile the code. If it is written in C language use tau_cc.sh. All TAU binaries are located in: /opt/gridware/performance/tau-2.19.1/x86_64/bin/. The current installation of TAU has been performed using Intel Fortran compiler.
5. Procedure to port & compile DLPOLY-2.18 with TAU-2.19.1:
   1. Untar the dlpoly-2.18 using the following command: untar -xzvf packagename. For this test, the working directory is: /export/home/username/scratch/dlpoly-tar/
   2. cd to dl_poly_2.18/srcmod.
   3. Edit Makefile by doing the following:

 
BINROOT = /export/home/username/scratch/dlpoly-tar/executables/   # Location of executable
CC  = icc						            # Intel compiler
EX = DLPOLY_2.18.X					            # Executable name
EXE = $(BINROOT)/$(EX)
FC=ifort						            # Fortran compiler
SHELL=/bin/sh
TYPE=par
  
Sun:

$(MAKE) LD="tau_f90.sh -o" \  # Tau mpi wrapper (e.g. In this line, users normally put mpi wrapper such as mpif90)
LDFLAGS="-O3 " \
FC="tau_f90.sh -c" \                 # Tau mpi wrapper (e.g. in this line, users normally put an mpi wrapper such as mpif90)
FCFLAGS="-O3 " \
EX=$(EX) BINROOT=$(BINROOT) $(TYPE)

1. In the same directory dl_poly_2.18/srcmod, compile the model using the following command: make Sun.
2. From then, type command: make install. It will then generate an executable in the following path (as quoted in your Makefile): /export/home/username/scratch/dlpoly-tar/executables/. Note: The only important change in your model is to replace mpi with tau wrapper. e.g. (mpif90 should be replaced by tau_f90.sh), while mpicc should be replaced by tau_cc.sh).
3. Procedure to submit the job and generate TAU output:
   1. Use the normal script to submit job into the system (e.g):

  ###These lines are for Moab
  #MSUB -l nodes=2:ppn=8
  #MSUB -l partition=ALL
  #MSUB -l walltime=168:00:00
  #MSUB -m be
  #MSUB -V
  #MSUB -o /export/home/username/scratch/dlpoly-2/out
  #MSUB -e /export/home/username/scratch/dlpoly-2/err
  #MSUB -d /export/home/username/scratch/dlpoly-2/
  #MSUB -mb
  ##### Running commands
  exe=/export/home/username/scratch/dlpoly-tar/executables/DLPOLY_2.18.X
  nproc=`cat $PBS_NODEFILE | wc -l`
  mpirun -np $nproc $exe

1. In this case, the job was submitted from the following directory: /export/home/username/scratch/dlpoly-2/.
2. After the completion of the job; there should be normal output of your model including many files named: tautrace and events.
3. Procedure to visualize the output of TAU:
4. In the same running directory of the job /export/home/username/scratch/dlpoly-2/, type: tau_treemerge.pl. Note: this command will merge all the tautrace & events files to just two files named: tau.trc and tau.edf)
5. Again in the same directory, type: tau2slog2 tau.trc tau.edf -o tau.slog2. Note: This will then converts the traces to the slog2 format. This process may take more than 40 minutes depending on the size of the model.
6. From then, issue the following command: jumpshot tau.slog2. It will then pop-up a message which state as follows “It seems this is your first time using Jumpshot-4, a setup file will be created in your home directory with the default settings”.
7. Click “OK”
8. Another message will be displayed which quote as follows “Save preferred settings to the setup file”.
9. Click “Yes”
10. It will then display the TAU Program which contains all the processes that took place during the execution of the model.
11. You may select your preferred processes that you need to track in the application e.g. Mpi_Send, Mpi_Receive, Mpi_Wait, etc.

For more information about TAU, visit the following website address: http://www.cs.uoregon.edu/research/tau/docs/newguide/bk01ch01s02.html