howto:dirisa
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
howto:dirisa [2023/09/22 13:18] msovara |
howto:dirisa [2024/07/23 14:48] (current) msovara [Important caveats] |
||
|---|---|---|---|
| Line 76: | Line 76: | ||
| === 2. You cannot use files on DIRISA === | === 2. You cannot use files on DIRISA === | ||
| The DIRISA object stores are designed for storage, not use. At best you can use the Data Deposit Tool to edit metadata and view the first few lines of files. It is recommended to avoid conducting uploads or downloads of data stored in iRODS directly within a PBS jobscript. You might be wondering, "What about automating my workflow?" | The DIRISA object stores are designed for storage, not use. At best you can use the Data Deposit Tool to edit metadata and view the first few lines of files. It is recommended to avoid conducting uploads or downloads of data stored in iRODS directly within a PBS jobscript. You might be wondering, "What about automating my workflow?" | ||
| - | * Performance Impact: iRODS may not be optimized for high-speed data transfer, especially within the context of a jobscript. Attempting to transfer large amounts of data during job execution can significantly slow down the job and potentially lead to resource contention with other jobs running on the cluster. | + | * Performance Impact: iRODS is not optimized for high-speed data transfer, especially within the context of a jobscript. Attempting to transfer large amounts of data during job execution can significantly slow down the job and potentially lead to resource contention with other jobs running on the cluster. |
| * Resource Contention: Cluster resources, including network bandwidth and I/O operations, are typically shared among multiple users and jobs. Uploading or downloading data within a jobscript can monopolize these shared resources, causing delays or conflicts with other users' jobs. | * Resource Contention: Cluster resources, including network bandwidth and I/O operations, are typically shared among multiple users and jobs. Uploading or downloading data within a jobscript can monopolize these shared resources, causing delays or conflicts with other users' jobs. | ||
| Line 90: | Line 90: | ||
| === 4. Make sure of your i's === | === 4. Make sure of your i's === | ||
| - | You are only an **i** away from deleting valuable data. Proceed with due care. | + | You are only an **i** away from permanently |
| < | < | ||
| Line 101: | Line 101: | ||
| </ | </ | ||
| will irretrievably delete your data on Lustre. | will irretrievably delete your data on Lustre. | ||
| + | |||
| + | === 5. Moving really large data files === | ||
| + | To move really large data files, consider using terminal multiplexers like **screen** or **tmux**. These tools allow you to detach from a session, leaving your tasks (such as large data transfers) running in the background. Even if you close the terminal window or your local machine is switched off, the tasks will continue to run. When you're ready, you can open a new terminal window and reattach to the still-running session. This ensures that your tasks continue uninterrupted even when you sign out of LENGAU. | ||
/app/dokuwiki/data/attic/howto/dirisa.1695381524.txt.gz · Last modified: 2023/09/22 13:18 by msovara
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
