howto:tipsandtricks
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
howto:tipsandtricks [2025/01/22 10:10] ccrosby |
howto:tipsandtricks [2025/03/19 15:20] (current) ccrosby [Allowing or preventing rerunning] |
||
|---|---|---|---|
| Line 42: | Line 42: | ||
| ===== Compressing and uncompressing very large files ===== | ===== Compressing and uncompressing very large files ===== | ||
| - | High-performance computing can obviously generate very large files very easily. However, | + | Reduce storage space occupation by making use of compression. However, large files take a long time to compress or uncompress, but this process can be accelerated with the use of parallel compression methods. |
| + | |||
| + | === Parallel pzip2 === | ||
| + | Get access to parallel bzip2 with the following module: | ||
| + | < | ||
| + | This provides the following executables: | ||
| + | < | ||
| + | bunzip2 | ||
| + | </ | ||
| + | |||
| + | '' | ||
| + | |||
| + | === Pigz, or parallel gzip === | ||
| + | Get access to the parallel implementation of gzip with the following module: | ||
| + | < | ||
| + | This provides the following executables: | ||
| + | < | ||
| + | pigz unpigz | ||
| + | |||
| + | '' | ||
| Line 56: | Line 75: | ||
| </ | </ | ||
| + | In general, the CHPC recommends that you do not allow automatic re-running. | ||
| - | On the other hand, if your software is set up to resume automatically from the last data written, PBS should | + | On the other hand, if your software is set up to resume automatically from the last data written, PBS may be permitted to rerun the process: |
| < | < | ||
| #PBS -r y | #PBS -r y | ||
/app/dokuwiki/data/attic/howto/tipsandtricks.1737533440.txt.gz · Last modified: 2025/01/22 10:10 by ccrosby
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
