wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » quick » walltime
Trace: • induction • moab • compiling_octave • feedback • walltime • gpu
quick:walltime

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
quick:walltime [2021/03/26 15:59]
agill 		quick:walltime [2021/12/09 16:42] (current)
Line 3: Line 3:
 As most CHPC users will know, there is a limit on how long a job submitted to any queue on Lengau will be allowed to run for, as is specified in the user policy document. As most CHPC users will know, there is a limit on how long a job submitted to any queue on Lengau will be allowed to run for, as is specified in the user policy document.
 For instance, at present, jobs submitted to the normal queue cannot run for more than 48 hours. For instance, at present, jobs submitted to the normal queue cannot run for more than 48 hours.
 +In your script, you'd specify it something like this:
  
-It might be tempting, therefore, to specify this for the maximum allowed walltime for the queue, and not worry about how long the job will actually run for. However, this is not actually a good idea.+ **#PBS -l walltime=48:00:00** 
 + 
 +It might be tempting, therefore, to specify this for the maximum allowed walltime for the queue, and not worry about how long the job will actually run for. However, this is not actually a good idea and may waste your time by delaying the start of your jobs.
  
 When you submit a job using qsub, it gets passed to a piece of cluster-management software called the job-scheduler (we use PBS-Pro on lengau at present). This software looks at the requested walltime, cores, nodes and other resources requested for each job, and attempts to fit as many jobs into the available resources on the cluster as soon as is reasonable, so that no one job has too long a wait. In effect, you might say that it is a bit like playing tetris with the jobs. When you submit a job using qsub, it gets passed to a piece of cluster-management software called the job-scheduler (we use PBS-Pro on lengau at present). This software looks at the requested walltime, cores, nodes and other resources requested for each job, and attempts to fit as many jobs into the available resources on the cluster as soon as is reasonable, so that no one job has too long a wait. In effect, you might say that it is a bit like playing tetris with the jobs.
Line 13: Line 16:
 ===== How to Estimate Walltime ===== ===== How to Estimate Walltime =====
 So how do you make a better estimate of your walltime? So how do you make a better estimate of your walltime?
-Well, when you begin doing a new type of simulation, you should do a few tests to see how long a few representative jobs will run for. In this case, it is all right if your walltime estimate is inaccurate - because you won't have to do this often. You need not run the job to completion; if it is an iterative calculation, you can get a pretty good estimate by running a few iterations and multiplying the time taken up to get to the expected number of iterations. Afterwards, you will already have a much better guess of the actual walltime required, and can narrow it down further if necessary. This may seem like an effort, and you may argue that you do not have the time to devote to this. However, time spent on this will be amply rewarded afterwards by time saved in waiting for queued jobs to run. It also need not be done with very great precision - as long as the estimate is reasonably close. Of course, overestimating by an hour or two is not a huge disaster and may even be advisable, given that there will always be some uncertainty on the exact runtime of a job. But over-estimating by 10 or 20 hours is another question entirely and should be avoided.+Well, when you begin doing a new type of simulation, you should do a few tests to see how long a few representative jobs will run for. In this case, it is all right if your walltime estimate is inaccurate - because you won't have to do this often. You need not run the job to completion; if it is an iterative calculation, you can get a pretty good estimate by running a few iterations and extrapolating the time take to get to the expected number of iterations. Afterwards, you will already have a much better guess of the actual walltime required, and can narrow it down further if necessary. This may seem like an effort, and you may argue that you do not have the time to devote to this. However, time spent on this will be amply rewarded afterwards by time saved in waiting for queued jobs to run. It also need not be done with very great precision - as long as the estimate is reasonably close. Of course, overestimating by an hour or two is not a huge disaster and may even be advisable, given that there will always be some uncertainty on the exact runtime of a job. But over-estimating by 10 or 20 hours is another question entirely and should be avoided.
  
 ===== Using Scaling to Decrease Walltime ===== ===== Using Scaling to Decrease Walltime =====
/app/dokuwiki/data/attic/quick/walltime.1616767169.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki