Differences

This shows you the differences between two versions of the page.

--- howto:tipsandtricks [2023/08/11 09:12]
alopis [Determining the status of your queued jobs]
+++ howto:tipsandtricks [2025/03/19 15:20] (current)
ccrosby [Allowing or preventing rerunning]
@@ Line 39: / Line 39: @@
 module load chpc/mc/4.8.17
 </code>
+===== Compressing and uncompressing very large files =====
+Reduce storage space occupation by making use of compression.  However, large files take a long time to compress or uncompress, but this process can be accelerated with the use of parallel compression methods.  On Lengau, there are two options:
+=== Parallel pzip2 ===
+Get access to parallel bzip2 with the following module:
+<code> module load chpc/compmech/pbzip2/1.1.13 </code>
+This provides the following executables:
+<code> $ ls /apps/chpc/compmech/utils/bzip2/bin
+bunzip2  bzcat  bzcmp  bzdiff  bzegrep  bzfgrep  bzgrep  bzip2  bzip2recover  bzless  bzmore  pbunzip2  pbzcat  pbzip2
+</code>
+''pbzip2 --help '' will provide the necessary instructions.
+=== Pigz, or parallel gzip ===
+Get access to the parallel implementation of gzip with the following module:
+<code> module load chpc/compmech/pigz/2.8 </code>
+This provides the following executables:
+<code> $ ls /home/apps/chpc/compmech/pigz-2.8
+pigz  unpigz  </code>
+''pigz --help'' and ''unpigz --help'' will provide the necessary instructions.
@@ Line 51: / Line 75: @@
 </code>
+In general, the CHPC recommends that you do not allow automatic re-running.  It is most often better to have a process die properly, so that the fault can be traced and corrected.
-On the other hand, if your software is set up to resume automatically from the last data written, PBS should be permitted to rerun the process:
+On the other hand, if your software is set up to resume automatically from the last data written, PBS may be permitted to rerun the process:
 <code>
 #PBS -r y
@@ Line 276: / Line 301: @@
 This example indicates a job on the smp queue requiring one node, 24 cpu's/cores, 24 mpi processes, and memory of 999Gb. Note that the standard nodes on Lengau have either 64Gb or 128Gb, so this memory request is inappropriate and will never run. In fact in terms of memory on such nodes one should specify 56gb or 120gb since each node needs memory for system processes.
-If you are running Materials Studio jobs (accelrys) you will likely need to first look up you job number on the cluster. MS does give you a job identification name, so for example
+If you are running Materials Studio jobs (accelrys) you will likely need to first look up you job number on the cluster. MS does give you a job identification name, so for example:
 <code>

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools