Differences

This shows you the differences between two versions of the page.

--- quick:loadshedding [2022/07/15 10:55]
alopis [Tips]
+++ quick:loadshedding [2022/07/15 13:31] (current)
wikiadmin
@@ Line 1: / Line 1: @@
 ======Electrical Power and Load Shedding======
-Unfortunately, the CHPC is subject to load-shedding (under the City of Cape Town) when Eskom has reduced generation.
+Unfortunately, the CHPC is subject to load-shedding (under the City of Cape Town schedule) when Eskom has reduced generation capacity.
-While the CHPC has a 1000 kVA generator this is not sufficient to power the entire cluster during blackouts and only 10 of the compute racks are available.
+While the CHPC has a 1000 kVA generator, this is not sufficient to power the entire cluster during blackouts and only 10 of the compute racks will be available.
-**Please plan accordingly**
+**Please plan accordingly.**
 =====Tips=====
-There are steps you can take to make more effective use of the cluster when it is operating at reduced capacity. Many of these are also useful outside of loadshedding events as well.
+There are steps you can take to make more effective use of the cluster when it is operating at reduced capacity. Many of these are also useful outside of load-shedding events as well.
-. Please check whether each of your jobs is very necessary.
+**The following actions will help your job to start running sooner and remain queued for a shorter time:**
-The following will allow your job to start running sooner and remain queued for a shorter time:
+====1. Prioritise your most urgent jobs.====
-. Choose as short a walltime as appropriate such that your job will have passed and written checkpoint files, so that you can then submit subsequent jobs to continue your calculation. You can also adjust (increase) the frequency of checkpoint file outputs when possible. You can also choose the waltimes according to your working hours and schedule.
+Please plan your work so that only the most urgently needed jobs are submitted.  Making your workflow more efficient will help you make more effective use of the cluster. Please be cautious and check you've set up your job correctly.
-. Select fewer nodes for your job. Be aware of how well your specific jobs scale and what the sweetspot is.
-For example your job has speed S for 1 node, speed 1.8S for 2 nodes, 3S for 4 nodes, 4S for 6 nodes, 3.8S for 8 nodes. When there are a surplus of nodes on the cluster, one might choose to use 6 nodes (8 nodes would be a very poor choice as it's inefficient and wasteful). During loadshedding when the cluster is oversubscribed, choosing 2 nodes (or even 1 node) would be a good idea.
-. Make sure to be using checkpoints as well as possible.
+====2. Choose as short a walltime as appropriate.====
+Select the walltime such that your job will have passed and written checkpoint files, so that you can then submit subsequent jobs to continue your calculation. It will help if you increase the frequency of checkpoint file outputs when possible. You can also choose the walltimes according to your working hours and schedule.  **Please** put extra effort into estimating walltimes.  Scheduling becomes more efficient when walltime estimates are more accurate.
+====3. Select fewer nodes for your job.====
+Be aware of how well your specific jobs scale and what the sweetspot is.
+For example, suppose your job has
+|  speed //S// | for 1 node,  |
+|  speed 1.8//S// | for 2 nodes,  |
+|  3//S// | for 4 nodes,  |
+|  4//S// | for 6 nodes,  |
+|  3.8//S// | for 8 nodes.  |
+When there are a surplus of nodes on the cluster, one might choose to use 6 nodes (8 nodes would be a very poor choice as it's inefficient and wasteful). During load-shedding when the cluster is oversubscribed, choosing 2 nodes (or even 1 node) would be a good idea.
+====4. Use efficient checkpoints.====
+Breaking a very long running job into shorter job runs using checkpoints will help with queuing times. If you already use checkpoints, it will help if you increase the frequency of checkpoint file outputs when possible, to submit shorter jobs.
+====5. Testing of a new type of job/calculation. ====
+When testing a new job script, either use an [[quick:start#example_interactive_job_request| interactive job]], on a single node, or a very very short job (maybe 15 mins). This way you will know your job will work, and then you can submit your actual longer job thereafter. You can thus avoid waiting hours or even days to find your job failed, due to perhaps a small error in your input file. Most importantly, **test!**  Please read our section on [[howto:tipsandtricks#please_slow_down_and_work_methodically|working methodically]].
+====6. Check how well your job is running. ====
+Always check to see if your job is working correctly:
+  - Is it producing the expected output at the expected rate?
+  - Are you fully using all the compute cores that you requested?
+You can ssh into your assigned nodes once your job has started running and monitor activity with ''top'' or ''htop''.
+So to determine which node number/s your job is using:
+  qstat -u yourusername -n
+Then ssh into this node/s, for example
+  ssh cnode0123
+  htop
-. Testing of a new type of job/calculation - either use an interactive job/node or a very very short job (maybe 15 mins). This way you will know your job will work, and then you can submit your actual loner job thereafter. You can thus avoid waiting hours or even days to find your job failed, due to perhaps a small error in your input file.

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools