wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » tune
Trace:
howto:tune

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
howto:tune [2011/02/08 16:28]
wikiadmin created 		howto:tune [2021/12/09 16:42] (current)
Line 1: Line 1:
-====== User Guide For Tuning and Analysis Tools in CHPC Sun System ======+====== User Guide For Tuning and Analysis Tools on the CHPC Lengau System ======
  
 Author: M.S Mabakane Author: M.S Mabakane
-Date: February, 08 2011+Date: November, 30 2018
  
-TAU (Tuning and Analysis Utilities) is a portable profiling and tracing toolkit that can be used to analyse the performance of parallel models written in Fortran, C, C++ and Python. It is also used to identify key areas such as mpi processes, message passing; in general; the performance of the application that occur during the execution of the model. Many various options are also available in the TAU package that can be selected to obtain more detail information about the performance of the model. +TAU (Tuning and Analysis Utilities) is a portable profiling and tracing tool that analyse the performance of parallel models written in C, C++, Python and FORTRAN. It is further used to identify performance bottlenecks incurred during the simulation of the parallel program. TAU users can obtain performance information such as execution time of the objects (e.g. modules, routines, subroutines and functions) and relationship between different parts of the application. 
  
-At CHPC, TAU is installed and tested in the CHPC Sun Microsystems cluster to help users improve the performance of their models. On the other hand, TAU is also anticipated to be a useful tool for developers who need to solve challenging errors of their programs. For this task, DLPOLY-2 was selected for the purpose of testing this tuning analysis system in the CHPC cluster. However, other models such as (WRF, CAM, POP and CCSM) were also tested by the TAU Developers (University of Oregon and University of Tennessee, USA) in their supercomputing systems. +At CHPC, TAU is installed and tested on the CHPC’s Lengau cluster in order to help users improve the performance of their parallel programs. On the other hand, TAU is also useful tool for developers who need to solve complex optimization problems within their parallel applications.  
 + 
 +TAU was installed using Intel Compiler 16.0.1 and Open MPI version 1.8.8. Below is the procedure on how to port, compile and run the parallel program using TAU-2.25.1:
  
 Now, this is the procedure to port, compile and run the code using TAU: Now, this is the procedure to port, compile and run the code using TAU:
Line 12: Line 14:
 Please note: typewriter characters depicts the command, directory or files in the system.  Please note: typewriter characters depicts the command, directory or files in the system. 
  
-  - Login to the CHPC Sun system using ''-X'' mode: e.g. ''ssh -X username@sun.chpc.ac.za'' +  - Login to the CHPC Lengau system using ''–X'' mode: e.g. ''ssh –X username@lengau.chpc.ac.za''. 
-  - In the command line, type the command: ''module add intel''. The system will then add the intel compiler and open-mpi in the user's environment. To test the environment type: ''which ifort'' and the following path should be displayed: ''/opt/gridware/intel/Compiler/11.1/056/bin/intel64/ifort''.   +  - In the command line, type the command: ''module load chpc/parallel_studio_xe/16.0.1/2016.1.150.'' The system will then add the Intel compiler in the user's environment. To test the environment type: ''which ifort'' and the following path should be displayed: ''/apps/compilers/intel/parallel_studio_xe_2016/compilers_and_libraries/linux/bin/intel64/ifort''. 
-  - From then, type the command: module add tau. It will add the following path to tau libraries: /opt/gridware/performance/tau-2.19.1/x86_64/bin and  TAU_MAKEFILE=/opt/gridware/performance/tau-2.19.1/x86_64/lib/Makefile.tau-mpi-pdt-trace in your profile. To test the TAU environment type: which tau_f90.sh, it should display these message: /opt/gridware/performance/tau-2.19.1/x86_64/bin/tau_f90.sh. +  - Afterwards, type the command: ''module load chpc/openmpi/1.8.8/intel-16.0.1.'' to load Open MPI-1.10.2.   
-  - You are now ready to port and compile the model using TAU.   Note: If your model is written in Fortran language, use the library named: tau_f90.sh to compile the code. If it is written in C language use   tau_cc.sh. All TAU binaries are located in: /opt/gridware/performance/tau-2.19.1/x86_64/bin/. The current installation of TAU has been performed   using Intel Fortran compiler.  +  - From then, type the command: ''module load tau/2.25.1-intel''. It will add the following path to tau libraries: ''/apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/bin'' and  ''TAU_MAKEFILE /apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/lib/Makefile.tau-icpc-mpi-pdt'' in your profile. To test the TAU environment type: ''which tau_f90.sh'', it should display these message: ''/apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/bin/tau_f90.sh''. 
-  - Procedure to port & compile DLPOLY-2.18 with TAU-2.19.1:  +  - You are now ready to port and compile the model using TAU.   Note: If your model is written in Fortran language, use the library named: ''tau_f90.sh'' to compile the code. If it is written in C language use   tau_cc.sh. All TAU binaries are located in: ''/apps/chpc/visualization/soft/tau-2.25.1-intel-16.0.1/x86_64/bin/''. The current installation of TAU has been performed using Intel Compiler 16.0.1.  
-    - Untar the dlpoly-2.18 using the following command: untar -xzvf packagename. For this test, the working directory is: /export/home/username/scratch/dlpoly-tar/ +  - Procedure to port & compile DLPOLY-2.18 using TAU-2.25.1:  
-    - cd to dl_poly_2.18/srcmod.  +    - Untar the dlpoly-2.18 using the following command: ''untar -xzvf packagename''. For this test, the working directory is: ''/mnt/lustre/users/username/dlpoly-tar/'' 
- +    - cd to ''dl_poly_2.18/srcmod''.  
-5.3. Edit DLPOLY Makefile by doing the following:  +    - Edit ''Makefile'' by doing the following: <code>  
- +BINROOT = /mnt/lustre/users/username/dlpoly-tar/executables/    # Location of executable 
-  BINROOT = /export/home/username/scratch/dlpoly-tar/executables/   # Location of executable +CC  = icc             # Intel compiler 
-  CC  = icc             # Intel compiler +EX = DLPOLY_2.18.X             # Executable name 
-  EX = DLPOLY_2.18.X             # Executable name +EXE = $(BINROOT)/$(EX) 
-  EXE = $(BINROOT)/$(EX) +FC=ifort             # Fortran compiler 
-  FC=ifort             # Fortran compiler +SHELL=/bin/sh 
-  SHELL=/bin/sh +TYPE=par
-  TYPE=par +
-  +
-Sun: +
- +
-  $(MAKE) LD="tau_f90.sh -o" \  # Tau mpi wrapper (e.g. In this line, users normally put mpi wrapper such as mpif90) +
-  LDFLAGS="-O3 " \ +
-  FC="tau_f90.sh -c" \                 # Tau mpi wrapper (e.g. in this line, users normally put an mpi wrapper such as mpif90) +
-  FCFLAGS="-O3 " \ +
-  EX=$(EX) BINROOT=$(BINROOT) $(TYPE) +
- +
-5.4. In the same directory (dl_poly_2.18/srcmod), compile the model using the following command: make Sun. +
- +
-9. From then, type command: make install. It will then generate an executable in the following path (as quoted in your Makefile): /export/home/username/scratch/dlpoly-tar/executables/. +
- +
-Note: The only important change in your model is to replace mpi with tau wrapper. e.g. (mpif90 should be replaced by tau_f90.sh), while mpicc should be replaced by tau_cc.sh).  +
-  +
-6. Procedure to submit the job and generate TAU output: +
- +
- +
-6.1. Use the normal script to submit job into the system (e.g): +
- +
-###These lines are for Moab +
-#MSUB -l nodes=2:ppn=8 +
-#MSUB -l partition=ALL +
-#MSUB -l walltime=168:00:00 +
-#MSUB -m be +
-#MSUB -V +
-#MSUB -o /export/home/username/scratch/dlpoly-2/out +
-#MSUB -e /export/home/username/scratch/dlpoly-2/err +
-#MSUB -d /export/home/username/scratch/dlpoly-2/ +
-#MSUB -mb +
- +
-##### Running commands +
-exe=/export/home/username/scratch/dlpoly-tar/executables/DLPOLY_2.18.X +
-nproc=`cat $PBS_NODEFILE | wc -l` +
-mpirun -np $nproc $exe +
- +
-6.2. In this case, the job was submitted from the following directory: /export/home/username/scratch/dlpoly-2/.  +
- +
-6.3. After the completion of the job; there should be normal output of your model including many files named: tautrace and events. +
- +
- +
- +
- +
-7. Procedure to visualize the output of TAU: +
- +
-7.1. In the same running directory of the job (/export/home/username/scratch/dlpoly-2/), type: tau_treemerge.pl. Note: this command will merge all the tautrace & events files to just two files named: tau.trc and tau.edf)  +
- +
-7.8. Again in the same directory, type: tau2slog2 tau.trc tau.edf -o tau.slog2. Note: This will then converts the traces to the slog2 format. This process may take more than 40 minutes depending on the size of the model. +
- +
-7.9. From then, issue the following command: jumpshot tau.slog2. It will then pop-up a message which state as follows "It seems this is your first time using Jumpshot-4, a setup file will be created in your home directory with the default settings".  +
- +
-7.10. Click "OK" +
- +
-7.11. Another message will be displayed which quote as follows "Save preferred settings to the setup file". +
- +
-7.12. Click "Yes" +
- +
-7.13. It will then display the following TAU Program which contains all the processes that took place during the execution of the model as follows: +
- +
- +
- +
-7.14. The below picture outlines the processes of the selected mpi processes (such as Mpi_Send, Mpi_Receive, Mpi_Wait, etc) that occurred during the model execution: +
- +
  
 +$(MAKE) LD="tau_f90.sh -o" \  # Tau mpi wrapper (e.g. In this line, users normally put mpi wrapper such as mpif90)
 +LDFLAGS="-O3 " \
 +FC="tau_f90.sh -c" \                 # Tau mpi wrapper (e.g. in this line, users normally put an mpi wrapper such as mpif90)
 +FCFLAGS="-O3 " \
 +EX=$(EX) BINROOT=$(BINROOT) $(TYPE)</code>
 +  - In the same directory ''dl_poly_2.18/srcmod'', compile the model using the following command: ''make''.
 +  - From then, type command: make install. It will then generate an executable in the following path (as quoted in your Makefile): ''/mnt/lustre/users/username/dlpoly-tar/''. Note: The only important change in your model is to replace ''mpi'' with tau wrapper. e.g. (''mpif90'' should be replaced by ''tau_f90.sh''), while ''mpicc'' should be replaced by ''tau_cc.sh''). 
 +  - Procedure to submit the job and generate TAU output:
 +    -Use the normal script to submit job into the system (e.g):<code>
 +  ###These lines are for PBSPro
 +   #!/bin/bash
 +   #PBS -l select=10:ncpus=24:mpiprocs=24:nodetype=haswell_reg
 +   #PBS -P PRJT1234
 +   #PBS -q normal
 +   #PBS -l walltime=4:00:00
 +   #PBS -o /mnt/lustre/users/username/dlpoly/dlpoly.out
 +   #PBS -e /mnt/lustre/users/username/dlpoly/dlpoly.err
 +   #PBS -m abe
 +   #PBS -M your.email@address
 +   ulimit -s unlimited
 +   cd /apss/chpc/executables/
 +   nproc=`cat $PBS_NODEFILE | wc -l`
 +   echo nproc is $nproc
 +   cat $PBS_NODEFILE
 +   time mpirun -np $nproc DLPOLY_2.18.X 
 +</code>
 +    - In this case, the job was submitted from the following directory: ''/mnt/lustre/users/username/dlpoly/''. 
 +    - After the completion of the job; there should produce normal output of your model including many files named: tautrace and events.
 +  - Procedure to visualize the output of TAU:
 +    - In the same running directory of the job ''/mnt/lustre/users/username/dlpoly/'', type: ''tau_treemerge.pl''. Note: this command will merge all the tautrace & events files to just two files named: tau.trc and tau.edf) 
 +    - Again in the same directory, type: ''tau2slog2 tau.trc tau.edf -o tau.slog2''. Note: This will then converts the traces to the slog2 format. This process may take more than 40 minutes depending on the size of the model.
 +    - From then, issue the following command: ''jumpshot tau.slog2''. It will then pop-up a message which state as follows "It seems this is your first time using Jumpshot-4, a setup file will be created in your home directory with the default settings". 
 +    - Click "OK"
 +    - Another message will be displayed which quote as follows "Save preferred settings to the setup file".
 +    - Click "Yes"
 +    - It will then display the TAU Program which contains all the processes that took place during the execution of the model.
 +    - You may select your preferred processes that you need to track in the application e.g. Mpi_Send, Mpi_Receive, Mpi_Wait, etc.
  
 For more information about TAU, visit the following website address: http://www.cs.uoregon.edu/research/tau/docs/newguide/bk01ch01s02.html For more information about TAU, visit the following website address: http://www.cs.uoregon.edu/research/tau/docs/newguide/bk01ch01s02.html
- 
- 
- 
  
/app/dokuwiki/data/attic/howto/tune.1297175291.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki