Differences

This shows you the differences between two versions of the page.

--- howto:dirisa [2023/07/24 11:04]
ccrosby
+++ howto:dirisa [2024/07/23 14:48] (current)
msovara [Important caveats]
@@ Line 1: / Line 1: @@
-===== Using DIRISA storage =====
+`===== Using DIRISA storage =====
 High-performance computing is closely related to "big data".  Many HPC users both use and generate very large data sets.  The CHPC's default recommendation to users is:
   * Feel free to use the CHPC's lustre storage as "work space".  It is large, fast and designed for parallel computing.
@@ Line 10: / Line 10: @@
 ==== Registering for DIRISA resources ====
-DIRISA is currently working on a comprehensive registration system.  However, as an interim measure, a temporary registration system has been set up, consisting of a Google form and registration for the web-based Data Deposit Tool.
+Register for DIRISA's data storage services [[https://service.dirisa.ac.za/|here]].
-=== 1.  Application form ===
+Here is a [[https://youtu.be/9Uc_F6z1TkA|video tutorial]] of the process.
-Please complete this [[https://forms.gle/eYH73zocGGVauz527|Google form]] and submit it.
-=== 2.  Data Deposit Tool registration ===
-Please create a user account for the [[https://service.dirisa.ac.za/subscription/accounts/home|Data Deposit Tool]].  When registering, you will be asked to provide a reason for requesting a user account  Specify that you have completed the google form for getting storage and are now requiring a user account. This will ensure that the evaluators process your application and request correctly.
-=== 3.  Await confirmation on application status ===
-You will receive two confirmations. The first if your user account is approved on the Data
-Deposit Tool and the second if your application to store data is approved for the archival
-solution.
@@ Line 75: / Line 66: @@
 === 0. Here be dragons ===
-Managing large quantities of data is a specialized, responsible and difficult task that cannot be undertaken casually.  It is expensive to create a lot of data but very easy to muck it up.  Each research group must appoint an experienced full-time researcher to take custody of its valuable data.  **This is not a job for amateurs.**  If you are intimidated by a raw command line, managing big data is not for you.
+Managing large quantities of data is a specialized, mission-critical and difficult task that cannot be undertaken casually.  It is expensive to create a lot of data but very easy to muck it up.  Each research group must appoint an experienced full-time researcher to take custody of its valuable data.  **This is not a job for amateurs.**  If you are intimidated by a raw command line, managing big data is not for you.
 === 1. Avoid lots of small files ===
@@ Line 84: / Line 75: @@
 === 2. You cannot use files on DIRISA ===
-The DIRISA object stores are designed for storage, not use.  At best you can use the Data Deposit Tool to edit metadata and view the first few lines of files.
+The DIRISA object stores are designed for storage, not use.  At best you can use the Data Deposit Tool to edit metadata and view the first few lines of files. It is recommended to avoid conducting uploads or downloads of data stored in iRODS directly within a PBS jobscript. You might be wondering, "What about automating my workflow?"
+  * Performance Impact: iRODS is not optimized for high-speed data transfer, especially within the context of a jobscript. Attempting to transfer large amounts of data during job execution can significantly slow down the job and potentially lead to resource contention with other jobs running on the cluster.
+  * Resource Contention: Cluster resources, including network bandwidth and I/O operations, are typically shared among multiple users and jobs. Uploading or downloading data within a jobscript can monopolize these shared resources, causing delays or conflicts with other users' jobs.
+  * Job Portability: Jobscripts are often designed to be portable and self-contained. Embedding data transfer operations within a jobscript can make it less portable and harder to maintain, as users would need to modify the script each time they want to transfer different data.
+  * Data Integrity: Depending on the scale and complexity of data transfer, doing it within a jobscript may increase the risk of data corruption or incomplete transfers, especially if the job terminates unexpectedly.
+It is generally recommended to perform data transfers separately from jobscripts, either before or after job execution, using dedicated tools and scripts. This approach ensures better control over data transfer processes, reduces the impact on job performance, and enhances jobscript portability and reliability.
 === 3. RTFM ===
@@ Line 90: / Line 90: @@
 === 4. Make sure of your i's ===
-You are only an i away from deleting valuable data.  Proceed with due care.
+You are only an **i** away from permanently deleting valuable data.  Proceed with due care.
 <code>
 irm -r myDataDirectory
 </code>
 will delete your data collection at DIRISA.
 <code>
 rm -r myDataDirectory
 </code>
 will irretrievably delete your data on Lustre.
+=== 5. Moving really large data files ===
+To move really large data files, consider using terminal multiplexers like **screen** or **tmux**. These tools allow you to detach from a session, leaving your tasks (such as large data transfers) running in the background. Even if you close the terminal window or your local machine is switched off, the tasks will continue to run. When you're ready, you can open a new terminal window and reattach to the still-running session. This ensures that your tasks continue uninterrupted even when you sign out of LENGAU.

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools