wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » tipsandtricks
Trace:
howto:tipsandtricks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
howto:tipsandtricks [2023/08/10 16:15]
alopis [Determining the status of your queued jobs] 		howto:tipsandtricks [2025/03/19 15:20] (current)
ccrosby [Allowing or preventing rerunning]
Line 39: Line 39:
 module load chpc/mc/4.8.17 module load chpc/mc/4.8.17
 </code> </code>
 +
 +
 +===== Compressing and uncompressing very large files =====
 +Reduce storage space occupation by making use of compression.  However, large files take a long time to compress or uncompress, but this process can be accelerated with the use of parallel compression methods.  On Lengau, there are two options:
 +
 +=== Parallel pzip2 ===
 +Get access to parallel bzip2 with the following module:
 +<code> module load chpc/compmech/pbzip2/1.1.13 </code>
 +This provides the following executables: 
 +<code> $ ls /apps/chpc/compmech/utils/bzip2/bin
 +bunzip2  bzcat  bzcmp  bzdiff  bzegrep  bzfgrep  bzgrep  bzip2  bzip2recover  bzless  bzmore  pbunzip2  pbzcat  pbzip2
 +</code> 
 +
 +''pbzip2 --help '' will provide the necessary instructions.
 +
 +=== Pigz, or parallel gzip ===
 +Get access to the parallel implementation of gzip with the following module:
 +<code> module load chpc/compmech/pigz/2.8 </code>
 +This provides the following executables:
 +<code> $ ls /home/apps/chpc/compmech/pigz-2.8
 +pigz  unpigz  </code> 
 +
 +''pigz --help'' and ''unpigz --help'' will provide the necessary instructions.
 +
  
  
Line 51: Line 75:
 </code> </code>
  
 +In general, the CHPC recommends that you do not allow automatic re-running.  It is most often better to have a process die properly, so that the fault can be traced and corrected.
  
-On the other hand, if your software is set up to resume automatically from the last data written, PBS should be permitted to rerun the process:+On the other hand, if your software is set up to resume automatically from the last data written, PBS may be permitted to rerun the process:
 <code> <code>
 #PBS -r y #PBS -r y
Line 233: Line 258:
  
 ===== Determining the status of your queued jobs ===== ===== Determining the status of your queued jobs =====
-Your job/s may be queued for various reasons. When the cluster is oversubscribed, such as when there are loadshedding cycles and we do not have sufficient generator capacity, a major reason is that there large number of users' jobs waiting in the queues. However, it is important to be aware that your job/s may be queued because your Research Programme (RP) allocation has run out and your Principal Investigator (PI) needs to provide 6 monthly feedback and/or contact your CHPC support scientist. Also possible is that you have specified a job which cannot ever run. Please check your jobs and queued jobs on the cluster using:+Your job/s may be queued for various reasons. When the cluster is oversubscribed, such as when there are loadshedding cycles and we do not have sufficient generator capacity, a major reason is that there large number of users' jobs waiting in the queues. However, it is important to be aware that your job/s may be queued because your Research Programme (RP) allocation has run out and **your Principal Investigator (PI) needs to provide 6 monthly feedback** and/or contact your CHPC support scientist. Also possible is that you have specified a job which cannot ever run. Please check your jobs and queued jobs on the cluster using:
  
 <code> <code>
Line 265: Line 290:
 The first 3 messages indicate indicate that there are not enough resources of the particular types. The last 5 messages indicate that the user (you if these your own job numbers) have other jobs in the specified queue which have reached the limit of number of jobs per user in this queue. The first 3 messages indicate indicate that there are not enough resources of the particular types. The last 5 messages indicate that the user (you if these your own job numbers) have other jobs in the specified queue which have reached the limit of number of jobs per user in this queue.
  
 +If you see "Insufficient amount of resource" then it is worth checking whether your jobs do correctly request the resources.
 +
 +<code>
 +qstat -f 4123211.sched01  |grep "List.select"
 +Resource_List.select = 1:ncpus=24:mpiprocs=24:mem=999gb
 +qstat -f 4123211.sched01  |grep "queue"
 +queue = smp
 +</code>
 +
 +This example indicates a job on the smp queue requiring one node, 24 cpu's/cores, 24 mpi processes, and memory of 999Gb. Note that the standard nodes on Lengau have either 64Gb or 128Gb, so this memory request is inappropriate and will never run. In fact in terms of memory on such nodes one should specify 56gb or 120gb since each node needs memory for system processes.
 +
 +If you are running Materials Studio jobs (accelrys) you will likely need to first look up you job number on the cluster. MS does give you a job identification name, so for example:
 +
 +<code>
 +qstat |grep My_MS_jobname
 +qstat|grep  MS_L2FXD
 +5046144.sched01   MS_L2FXD         accelrys                 0 Q accelrys  
 +</code>
 +
 +Then in this example the job number is 5046144.sched01, so thereafter please follow the procedure above.
/app/dokuwiki/data/attic/howto/tipsandtricks.1691676915.txt.gz · Last modified: 2023/08/10 16:15 by alopis

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki