Differences

This shows you the differences between two versions of the page.

--- howto:tipsandtricks [2023/08/11 09:13]
alopis [Determining the status of your queued jobs]
+++ howto:tipsandtricks [2025/03/19 15:20] (current)
ccrosby [Allowing or preventing rerunning]
@@ Line 39: / Line 39: @@
 module load chpc/mc/4.8.17
 </code>
+===== Compressing and uncompressing very large files =====
+Reduce storage space occupation by making use of compression.  However, large files take a long time to compress or uncompress, but this process can be accelerated with the use of parallel compression methods.  On Lengau, there are two options:
+=== Parallel pzip2 ===
+Get access to parallel bzip2 with the following module:
+<code> module load chpc/compmech/pbzip2/1.1.13 </code>
+This provides the following executables:
+<code> $ ls /apps/chpc/compmech/utils/bzip2/bin
+bunzip2  bzcat  bzcmp  bzdiff  bzegrep  bzfgrep  bzgrep  bzip2  bzip2recover  bzless  bzmore  pbunzip2  pbzcat  pbzip2
+</code>
+''pbzip2 --help '' will provide the necessary instructions.
+=== Pigz, or parallel gzip ===
+Get access to the parallel implementation of gzip with the following module:
+<code> module load chpc/compmech/pigz/2.8 </code>
+This provides the following executables:
+<code> $ ls /home/apps/chpc/compmech/pigz-2.8
+pigz  unpigz  </code>
+''pigz --help'' and ''unpigz --help'' will provide the necessary instructions.
@@ Line 51: / Line 75: @@
 </code>
+In general, the CHPC recommends that you do not allow automatic re-running.  It is most often better to have a process die properly, so that the fault can be traced and corrected.
-On the other hand, if your software is set up to resume automatically from the last data written, PBS should be permitted to rerun the process:
+On the other hand, if your software is set up to resume automatically from the last data written, PBS may be permitted to rerun the process:
 <code>
 #PBS -r y

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools