As most CHPC users will know, there is a limit on how long a job submitted to any queue on Lengau will be allowed to run for, as is specified in the user policy document. For instance, at present, jobs submitted to the normal queue cannot run for more than 48 hours. In your script, you'd specify it something like this:

#PBS -l walltime=48:00:00

It might be tempting, therefore, to specify this for the maximum allowed walltime for the queue, and not worry about how long the job will actually run for. However, this is not actually a good idea and may waste your time by delaying the start of your jobs.

When you submit a job using qsub, it gets passed to a piece of cluster-management software called the job-scheduler (we use PBS-Pro on lengau at present). This software looks at the requested walltime, cores, nodes and other resources requested for each job, and attempts to fit as many jobs into the available resources on the cluster as soon as is reasonable, so that no one job has too long a wait. In effect, you might say that it is a bit like playing tetris with the jobs.

Now the longer a job runs for, the more difficult it is for the job-scheduler to accommodate it (the same is true for jobs using many cores, particularly if this is for a long walltime). The only way that the job-scheduler can estimate the run-time of a job is based on the walltime requested by the user. So if your job will complete in only two hours but you requested 48, you will probably cause the job-scheduler to needlessly push your job to the back of the queue, as it has to wait until it can allocate a block of cores for the requested time. You will make your own life difficult, as well as potentially holding up other users needlessly.

So how do you make a better estimate of your walltime? Well, when you begin doing a new type of simulation, you should do a few tests to see how long a few representative jobs will run for. In this case, it is all right if your walltime estimate is inaccurate - because you won't have to do this often. You need not run the job to completion; if it is an iterative calculation, you can get a pretty good estimate by running a few iterations and extrapolating the time take to get to the expected number of iterations. Afterwards, you will already have a much better guess of the actual walltime required, and can narrow it down further if necessary. This may seem like an effort, and you may argue that you do not have the time to devote to this. However, time spent on this will be amply rewarded afterwards by time saved in waiting for queued jobs to run. It also need not be done with very great precision - as long as the estimate is reasonably close. Of course, overestimating by an hour or two is not a huge disaster and may even be advisable, given that there will always be some uncertainty on the exact runtime of a job. But over-estimating by 10 or 20 hours is another question entirely and should be avoided.

Something else which can help you use shorter walltimes is to test the scaling of your code. This is dealt with elsewhere on the wiki (Scaling),but briefly speaking, are you using as many cores as you can efficiently use to speed up your simulation? And are you perhaps using too many cores, which can actually slow things down? Again, when you begin running a new piece of software or a new type of calculation on the cluster, do a few runs to experiment on how much you will benefit from using more cores, and at what point you stop benefiting by requesting more cores. Going back to our Tetris analogy, if you can turn a long shape sideways and can lay it flat, you are usually better off than trying to stand it on its end. Similarly, if you can turn your long-duration, low core-count job into a short-duration, high core-count job, you will usually benefit more fully from using HPC as your job will finish quicker, and the other cluster users will also be less impeded. But you do need to take the trouble to do some investigation beforehand to characterise how your job scales on more cores. A supercomputer is a powerful research tool just like an expensive piece of lab equipment, so a researcher using it should take the time and trouble to make sure they know how to get the most out of it. As scientists, we can't afford to not understand our chosen tools - if we don't know what we don't know, we also don't know how it will affect our research and the results we obtain!

What if you believe that your job needs to run for longer than the maximum allowed walltime? The best way to deal with this is to checkpoint your job - that is, have it write out restart files regularly. This allows you to resubmit your job after it has ended so that it can continue from where it left off. This is also generally a good safety precaution, as even if a job is allowed to run for weeks, it is unwise to assume that nothing will occur to terminate it unexpectedly. For instance, it may become unstable and crash due to a choice of initial parameters but be savable if one returns to an earlier checkpoint and restarts it with better choices. Or load-shedding could strike, in which case the CHPC cannot run currently all the nodes off the backup generator power, therefore some would have to be shut down (which might include one on which your simulation is running). This particular situation will be resolved in future, but the point is that it is dangerous to assume that any computation will run uninterrupted for weeks on end. Some precaution against losing days or weeks of work is always advisable.

Most scientific codes have some provision for checkpointing and restarting in this fashion, and you should check the documentation of your particular code (or consider adding this functionality if you are writing your own code). There are, admittedly a few unfortunate exceptions to this rule, particularly for applications such as gene-sequencing or certain types of Machine-learning problems. Those who legitimately cannot restart a computation, but which need very long run-times, should contact us, and we will see what can be done to accommodate them.