wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » scaling
Trace:
scaling:start

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
scaling:start [2014/12/11 17:06]
dane created 		scaling:start [2021/12/09 16:42] (current)
Line 2: Line 2:
 It is extremely important to verify that your code/problem will scale properly in the cluster before running big jobs. So here is a simple example of how one might perform such a check... It is extremely important to verify that your code/problem will scale properly in the cluster before running big jobs. So here is a simple example of how one might perform such a check...
  
-First one should start with a relatively small problem -- say one that would take 10 minutes or so on a single node((This may not always be appropriate -- sometimes large jobs are needed before parallelism can be seen properly)).+First one should start with a problem that can be completed quickly -- say one that would take 10 minutes or so on a single node. The problem should not be trivially small -- large jobs are needed before parallelism can be seen properly.  If possible, run for fewer iterations or timesteps.
  
-Then one submits the jobs on increasing numbers of nodes and observes to see if the runtime comes down as expected. As an example of how you may go about this I have two scripts -- one is the PBS job script, and the second is a simple shell script that allows me to submit the PBS script with different parameters.+Then one submits the jobs on increasing numbers of nodes and observes to see if the runtime comes down as expected. As an example of how one may go about this, here are two scripts -- one is the PBS job script, and the second is a simple shell script that allows the user to submit the PBS script with different parameters.
  
-<file bash example_script.qsub>+<file bash scaling_test.qsub>
 #!/bin/bash #!/bin/bash
 #PBS -l walltime=10:00:00 #PBS -l walltime=10:00:00
-#PBS -q specialq +#PBS -q normal 
-#PBS -M dkennedy1@csir.co.za+#PBS -M YOUR@EMAIL.ADDRES
 #PBS -m be #PBS -m be
 #PBS -V #PBS -V
-#PBS -e /export/home/username/scratch5/scaling_test/stderr.txt +#PBS -e /mnt/lustre/users/USERNAME/scaling_test/test1.out 
-#PBS -o /export/home/username/scratch5/scaling_test/stdout.txt+#PBS -o /mnt/lustre/users/USERNAME/scaling_test/test1.err
 #PBS -mb #PBS -mb
  
-MODULEPATH=/opt/gridware/bioinformatics/modules:$MODULEPATH + 
-source /etc/profile.d/modules.sh +module add ### MODULES NEEDED
-module add gromacs/4.6.5_nehalem+
  
 NP=`cat ${PBS_NODEFILE} | wc -l` NP=`cat ${PBS_NODEFILE} | wc -l`
Line 27: Line 26:
 ARGS="-deffnm md -g md.${NP}.log" ARGS="-deffnm md -g md.${NP}.log"
  
-cd /export/home/username/scratch5/scaling_test+cd /mnt/lustre/users/USERNAME/scaling_test
 mpirun -np ${NP} -machinefile ${PBS_NODEFILE} ${EXE} ${ARGS} mpirun -np ${NP} -machinefile ${PBS_NODEFILE} ${EXE} ${ARGS}
 </file> </file>
Line 36: Line 35:
 first=1 first=1
  
-for i in 1 2 4 8 10 16 20 32 40 64+for i in 1 2 4 8 10
 do do
-  select="select=${i}:ncpus=8:mpiprocs=8:jobtype=nehalem,place=excl"+  select="select=${i}:ncpus=24:mpiprocs=24:nodetype=haswell_reg,place=excl"
   name="${i}_g_scale"   name="${i}_g_scale"
   if [ ${first} -eq 1 ]   if [ ${first} -eq 1 ]
   then   then
-    previous=`qsub -l ${select} -N ${name} dane_test_scaling.qsub`+    previous=`qsub -l ${select} -N ${name} scaling_test.qsub`
     first=0     first=0
   else   else
-    current=`qsub -l ${select} -N ${name} -W depend=afterok:${previous} dane_test_scaling.qsub`+    current=`qsub -l ${select} -N ${name} -W depend=afterok:${previous} scaling_test.qsub`
     previous=${current}     previous=${current}
   fi   fi
 done done
 </file> </file>
-When I look at the results I see that the corresponding walltimes are:+ 
 +The (fake) results are represented here as a table:
 ^  Number of nodes  ^  Runtime (seconds)  ^ ^  Number of nodes  ^  Runtime (seconds)  ^
-| 1  |  2745|+| 1  |  344|
 | 2  |  181| | 2  |  181|
 | 4  |  105| | 4  |  105|
 | 8  |  55| | 8  |  55|
 | 10 |  43| | 10 |  43|
-| 16 |  30| + 
-| 20 |  25| + 
-| 32 |  21| +These results are best interpreted in the form of a graph: 
-| 40 |  15| + 
-| 64 |  13|+{{:scaling:scalegraph1.png|}} 
 + 
 +This graph looks OK (the run time comes down as the number of nodes goes up), but is actually quite difficult to interpret, in terms of finding the optimum number of nodes to use.  It is much more useful to plot the reciprocal of the runtime, such as the number of runs per hour: 
 + 
 +{{:scaling:scalegraph2.png|}} 
 + 
 +This clearly shows that the application is scaling linearly.  Adding more nodes will almost certainly make it run faster. 
 + 
 +The following two graphs illustrate a case which does **not** scale well.  The runtime graph does not look as good as the first example, but it is not immediately clear where the scaling starts worsening.  The "Runs per hour" graph provides much better insight. 
 + 
 +{{:scaling:scalegraph3.png|}} 
 + 
 +{{:scaling:scalegraph4.png|}} 
 + 
 +In this case there is definitely no point in trying to use more than 8 nodes, and the scaling is already showing diminishing returns after 2 nodes.  In this case it may be more efficient to use fewer nodes per run, but submit more simultaneous runs. 
/app/dokuwiki/data/attic/scaling/start.1418310371.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki