Author: M.S Mabakane Date: November, 30 2018

TAU (Tuning and Analysis Utilities) is a portable profiling and tracing tool that analyse the performance of parallel models written in C, C++, Python and FORTRAN. It is further used to identify performance bottlenecks incurred during the simulation of the parallel program. TAU users can obtain performance information such as execution time of the objects (e.g. modules, routines, subroutines and functions) and relationship between different parts of the application.

At CHPC, TAU is installed and tested on the CHPC’s Lengau cluster in order to help users improve the performance of their parallel programs. On the other hand, TAU is also useful tool for developers who need to solve complex optimization problems within their parallel applications.

TAU was installed using Intel Compiler 16.0.1 and Open MPI version 1.8.8. Below is the procedure on how to port, compile and run the parallel program using TAU-2.25.1:

Now, this is the procedure to port, compile and run the code using TAU:

Please note: typewriter characters depicts the command, directory or files in the system.

1. Login to the CHPC Lengau system using –X mode: e.g. ssh –X username@lengau.chpc.ac.za.
2. In the command line, type the command: module load chpc/parallel_studio_xe/16.0.1/2016.1.150. The system will then add the Intel compiler in the user's environment. To test the environment type: which ifort and the following path should be displayed: /apps/compilers/intel/parallel_studio_xe_2016/compilers_and_libraries/linux/bin/intel64/ifort.
3. Afterwards, type the command: module load chpc/openmpi/1.8.8/intel-16.0.1. to load Open MPI-1.10.2.
4. From then, type the command: module load tau/2.25.1-intel. It will add the following path to tau libraries: /apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/bin and TAU_MAKEFILE /apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/lib/Makefile.tau-icpc-mpi-pdt in your profile. To test the TAU environment type: which tau_f90.sh, it should display these message: /apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/bin/tau_f90.sh.
5. You are now ready to port and compile the model using TAU. Note: If your model is written in Fortran language, use the library named: tau_f90.sh to compile the code. If it is written in C language use tau_cc.sh. All TAU binaries are located in: /apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/bin/. The current installation of TAU has been performed using Intel Compiler 16.0.1.
6. Procedure to port & compile DLPOLY-2.18 using TAU-2.25.1:
   1. Untar the dlpoly-2.18 using the following command: untar -xzvf packagename. For this test, the working directory is: /mnt/lustre/users/username/dlpoly-tar/
   2. cd to dl_poly_2.18/srcmod.
   3. Edit Makefile by doing the following:

       
      BINROOT = /mnt/lustre/users/username/dlpoly-tar/executables/    # Location of executable
      CC  = icc						            # Intel compiler
      EX = DLPOLY_2.18.X					            # Executable name
      EXE = $(BINROOT)/$(EX)
      FC=ifort						            # Fortran compiler
      SHELL=/bin/sh
      TYPE=par
      
      $(MAKE) LD="tau_f90.sh -o" \  # Tau mpi wrapper (e.g. In this line, users normally put mpi wrapper such as mpif90)
      LDFLAGS="-O3 " \
      FC="tau_f90.sh -c" \                 # Tau mpi wrapper (e.g. in this line, users normally put an mpi wrapper such as mpif90)
      FCFLAGS="-O3 " \
      EX=$(EX) BINROOT=$(BINROOT) $(TYPE)

7. In the same directory dl_poly_2.18/srcmod, compile the model using the following command: make.
8. From then, type command: make install. It will then generate an executable in the following path (as quoted in your Makefile): /mnt/lustre/users/username/dlpoly-tar/. Note: The only important change in your model is to replace mpi with tau wrapper. e.g. (mpif90 should be replaced by tau_f90.sh), while mpicc should be replaced by tau_cc.sh).
9. Procedure to submit the job and generate TAU output:
   1. Use the normal script to submit job into the system (e.g):

        ###These lines are for PBSPro
         #!/bin/bash
         #PBS -l select=10:ncpus=24:mpiprocs=24:nodetype=haswell_reg
         #PBS -P PRJT1234
         #PBS -q normal
         #PBS -l walltime=4:00:00
         #PBS -o /mnt/lustre/users/username/dlpoly/dlpoly.out
         #PBS -e /mnt/lustre/users/username/dlpoly/dlpoly.err
         #PBS -m abe
         #PBS -M your.email@address
         ulimit -s unlimited
         cd /apss/chpc/executables/
         nproc=`cat $PBS_NODEFILE | wc -l`
         echo nproc is $nproc
         cat $PBS_NODEFILE
         time mpirun -np $nproc DLPOLY_2.18.X 

   2. In this case, the job was submitted from the following directory: /mnt/lustre/users/username/dlpoly/.
   3. After the completion of the job; there should produce normal output of your model including many files named: tautrace and events.
10. Procedure to visualize the output of TAU:
    1. In the same running directory of the job /mnt/lustre/users/username/dlpoly/, type: tau_treemerge.pl. Note: this command will merge all the tautrace & events files to just two files named: tau.trc and tau.edf)
    2. Again in the same directory, type: tau2slog2 tau.trc tau.edf -o tau.slog2. Note: This will then converts the traces to the slog2 format. This process may take more than 40 minutes depending on the size of the model.
    3. From then, issue the following command: jumpshot tau.slog2. It will then pop-up a message which state as follows “It seems this is your first time using Jumpshot-4, a setup file will be created in your home directory with the default settings”.
    4. Click “OK”
    5. Another message will be displayed which quote as follows “Save preferred settings to the setup file”.
    6. Click “Yes”
    7. It will then display the TAU Program which contains all the processes that took place during the execution of the model.
    8. You may select your preferred processes that you need to track in the application e.g. Mpi_Send, Mpi_Receive, Mpi_Wait, etc.

For more information about TAU, visit the following website address: http://www.cs.uoregon.edu/research/tau/docs/newguide/bk01ch01s02.html