wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » quick » walltime
Trace: • induction • moab • gpu • compiling_octave • feedback • walltime
quick:walltime

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
quick:walltime [2021/03/26 15:31]
agill [How to Estimate Walltime] 		quick:walltime [2021/12/09 16:42] (current)
Line 3: Line 3:
 As most CHPC users will know, there is a limit on how long a job submitted to any queue on Lengau will be allowed to run for, as is specified in the user policy document. As most CHPC users will know, there is a limit on how long a job submitted to any queue on Lengau will be allowed to run for, as is specified in the user policy document.
 For instance, at present, jobs submitted to the normal queue cannot run for more than 48 hours. For instance, at present, jobs submitted to the normal queue cannot run for more than 48 hours.
 +In your script, you'd specify it something like this:
  
-It might be tempting, therefore, to specify this for the maximum allowed walltime for the queue, and not worry about how long the job will actually run for. However, this is not actually a good idea.+ **#PBS -l walltime=48:00:00** 
 + 
 +It might be tempting, therefore, to specify this for the maximum allowed walltime for the queue, and not worry about how long the job will actually run for. However, this is not actually a good idea and may waste your time by delaying the start of your jobs.
  
 When you submit a job using qsub, it gets passed to a piece of cluster-management software called the job-scheduler (we use PBS-Pro on lengau at present). This software looks at the requested walltime, cores, nodes and other resources requested for each job, and attempts to fit as many jobs into the available resources on the cluster as soon as is reasonable, so that no one job has too long a wait. In effect, you might say that it is a bit like playing tetris with the jobs. When you submit a job using qsub, it gets passed to a piece of cluster-management software called the job-scheduler (we use PBS-Pro on lengau at present). This software looks at the requested walltime, cores, nodes and other resources requested for each job, and attempts to fit as many jobs into the available resources on the cluster as soon as is reasonable, so that no one job has too long a wait. In effect, you might say that it is a bit like playing tetris with the jobs.
Line 13: Line 16:
 ===== How to Estimate Walltime ===== ===== How to Estimate Walltime =====
 So how do you make a better estimate of your walltime? So how do you make a better estimate of your walltime?
-Well, when you begin doing a new type of simulation, you should do a few tests to see how long a few representative jobs will run for. In this case, it is all right if your walltime estimate is inaccurate - because you won't have to do this often. Afterwards, you can have a much better guess of the actual walltime required. This may seem like an effort, and you may argue that you do not have the time to devote to this. However, time spent on this will be amply rewarded afterwards by time saved in waiting for queued jobs to run. It also need not be done with very great precision - as long as the estimate is reasonably close. Of course, overestimating by an hour or two is not a huge disaster and may even be advisable, given that there will always be some uncertainty on the exact runtime of a job. But over-estimating by 10 or 20 hours is another question entirely and should be avoided.+Well, when you begin doing a new type of simulation, you should do a few tests to see how long a few representative jobs will run for. In this case, it is all right if your walltime estimate is inaccurate - because you won't have to do this often. You need not run the job to completion; if it is an iterative calculation, you can get a pretty good estimate by running a few iterations and extrapolating the time take to get to the expected number of iterations. Afterwards, you will already have a much better guess of the actual walltime required, and can narrow it down further if necessary. This may seem like an effort, and you may argue that you do not have the time to devote to this. However, time spent on this will be amply rewarded afterwards by time saved in waiting for queued jobs to run. It also need not be done with very great precision - as long as the estimate is reasonably close. Of course, overestimating by an hour or two is not a huge disaster and may even be advisable, given that there will always be some uncertainty on the exact runtime of a job. But over-estimating by 10 or 20 hours is another question entirely and should be avoided.
  
 ===== Using Scaling to Decrease Walltime ===== ===== Using Scaling to Decrease Walltime =====
-Something else which can help you use shorter walltimes is to test the scaling of your code. This is dealt with elsewhere on the wiki ,but briefly speaking, are you using as many cores as you can efficiently use to speed up your simulation? And are you perhaps using too many cores, which can actually slow things down? Again, when you begin running a new piece of software or a new type of calculation on the cluster, do a few runs to experiment on how much you will benefit from using more cores, and at what point you stop benefiting by requesting more cores. Going back to our Tetris analogy, if you can turn a long shape sideways and can lay it flat, you are usually better off than trying to stand it on its end. Similarly, if you can turn your long-duration, low core-count job into a short-duration, high core-count job, you will usually benefit more fully from using HPC as your job will finish quicker, and the other cluster users will also be less impeded. But you do need to take the trouble to do some investigation beforehand to characterize how your job scales on more cores.+Something else which can help you use shorter walltimes is to test the scaling of your code. This is dealt with elsewhere on the wiki ([[scaling:start|Scaling]]),but briefly speaking, are you using as many cores as you can efficiently use to speed up your simulation? And are you perhaps using too many cores, which can actually slow things down? Again, when you begin running a new piece of software or a new type of calculation on the cluster, do a few runs to experiment on how much you will benefit from using more cores, and at what point you stop benefiting by requesting more cores. Going back to our Tetris analogy, if you can turn a long shape sideways and can lay it flat, you are usually better off than trying to stand it on its end. Similarly, if you can turn your long-duration, low core-count job into a short-duration, high core-count job, you will usually benefit more fully from using HPC as your job will finish quicker, and the other cluster users will also be less impeded. But you do need to take the trouble to do some investigation beforehand to characterise how your job scales on more cores. A supercomputer is a powerful research tool just like an expensive piece of lab equipment, so a researcher using it should take the time and trouble to make sure they know how to get the most out of it. As scientists, we can't afford to not understand our chosen tools - if we don't know what we don't know, we also don't know how it will affect our research and the results we obtain!
  
 ===== Long Walltime Jobs and Checkpointing ===== ===== Long Walltime Jobs and Checkpointing =====
/app/dokuwiki/data/attic/quick/walltime.1616765469.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki