Differences

This shows you the differences between two versions of the page.

--- howto:dirisa [2023/07/14 18:29]
ccrosby
+++ howto:dirisa [2024/07/23 14:48] (current)
msovara [Important caveats]
@@ Line 1: / Line 1: @@
-==== Using DIRISA storage ====
+`===== Using DIRISA storage =====
 High-performance computing is closely related to "big data".  Many HPC users both use and generate very large data sets.  The CHPC's default recommendation to users is:
   * Feel free to use the CHPC's lustre storage as "work space".  It is large, fast and designed for parallel computing.
@@ Line 8: / Line 8: @@
 However, the CHPC recognizes that this workflow is not universal.  Many research groups own or produce very large data sets that need to be stored safely for sharing or future re-use.  This is where the [[https://www.dirisa.ac.za/|Data Intensive Research Initiative of South Africa]] comes into play.  Like the CHPC, DIRISA is also part of South Africa's [[https://www.nicis.ac.za/|National Integrated Cyber Infrastructure System]].  DIRISA operates storage systems that **are** designed for long-term safe storage of research data.
-=== DIRISA Object Stores ===
+==== Registering for DIRISA resources ====
+Register for DIRISA's data storage services [[https://service.dirisa.ac.za/|here]].
+Here is a [[https://youtu.be/9Uc_F6z1TkA|video tutorial]] of the process.
+==== DIRISA Object Stores ====
 DIRISA operates two object stores:
   * [[http://irods.org|iRODS]] is an acronym for  Integrated Rule-Oriented Data System
@@ Line 15: / Line 23: @@
 These storage systems can be accessed by way of DIRISA's web-based [[https://www.dirisa.ac.za/deposit-data/|Data Deposit Tool]].  However, the data deposit tool is not well suited to the requirements of a CHPC cluster user.  It is intended for smaller quantities of data that are not already stored on NICIS resources.
-== The iCommands ==
+=== The iCommands ===
-The DIRISA object stores are directly accessible from the dtn node in the CHPC cluster.  The good news is that this data transfer node mounts all the cluster file systems, which makes it possible to transfer data more efficiently to the DIRISA object stores.  The bad news is that DIRISA's storage systems differ fundamentally from the cluster's storage:
+The DIRISA object stores are directly accessible from the ''dtn'' node in the CHPC cluster.  The good news is that this data transfer node mounts all the cluster file systems, which makes it possible to transfer data more efficiently to the DIRISA object stores.  The bad news is that DIRISA's storage systems differ fundamentally from the cluster's storage:
   * The DIRISA systems operate as [[https://en.wikipedia.org/wiki/Object_storage|Object Stores]]
   * The cluster storages are accessed as [[https://en.wikipedia.org/wiki/POSIX|POSIX]] file systems, even though "under the bonnet", Lustre is also actually an object storage system.
@@ Line 35: / Line 43: @@
 <code>
-[charles@dtn:~/lustre]$ ipwd
+[jblogs@dtn:~/lustre]$ ipwd
-/dirisa.ac.za/home/charles
+/dirisa.ac.za/home/jblogs
-[charles@dtn:~/lustre]$ imkdir myObjectCollection
+[jblogs@dtn:~/lustre]$ imkdir myObjectCollection
-[charles@dtn:~/lustre]$ icd myObjectCollection
+[jblogs@dtn:~/lustre]$ icd myObjectCollection
-[charles@dtn:~/lustre]$ iput SimpleBenchMarkLarge2006.tgz SimpleBenchMarkLarge2006_toiRODS.tgz
+[jblogs@dtn:~/lustre]$ iput LotsOfMyFiles.tgz LotsOfMyFiles_toiRODS.tgz
-[charles@dtn:~/lustre]$ iput -R LTA SimpleBenchMarkLarge2006.tgz SimpleBenchMarkLarge2006_toTape.tgz
+[jblogs@dtn:~/lustre]$ iput -R LTA LotsOfMyFiles.tgz LotsOfMyFiles_toTape.tgz
-[charles@dtn:~/lustre]$ ils
+[jblogs@dtn:~/lustre]$ ils
-/dirisa.ac.za/home/charles/myObjectCollection:
+/dirisa.ac.za/home/jblogs/myObjectCollection:
-  SimpleBenchMarkLarge2006_toiRODS.tgz
+  LotsOfMyFiles_toiRODS.tgz
-  SimpleBenchMarkLarge2006_toTape.tgz
+  LotsOfMyFiles_toTape.tgz
-[charles@dtn:~/lustre]$ ils -l
+[jblogs@dtn:~/lustre]$ ils -l
-/dirisa.ac.za/home/charles/myObjectCollection:
+/dirisa.ac.za/home/jblogs/myObjectCollection:
-  charles           0 dirisa_root;dirisa_replication;CapeTown;irods-resccpt02Resource   2492058420 2023-07-14.18:26 & SimpleBenchMarkLarge2006_toiRODS.tgz
+  jblogs           0 dirisa_root;dirisa_replication;CapeTown;irods-resccpt02Resource   2492058420 2023-07-14.18:26 & LotsOfMyFiles_toiRODS.tgz
-  charles           0 LTA   2492058420 2023-07-14.18:26 & SimpleBenchMarkLarge2006_toTape.tgz
+  jblogs           0 LTA   2492058420 2023-07-14.18:26 & LotsOfMyFiles_toTape.tgz
 </code>
+In the above example, a new collection called "myObjectCollection" was created and we moved into that namespace with the ''icd'' command.  Two ''iput'' commands were then executed.  In the first, a file was transferred to the default (iRODS) storage system.  The second ''iput'' command had the parameter ''-R LTA'', which means "use the resource long term archive".  When we check the contents of the collection with ''ils'' we see that both data objects are listed.  However, ''ils -l'' shows that the two objects in the same collection are stored in two totally different storage systems.
+==== Important caveats ====
+If you don't know what //caveat// means, go look it up.
+=== 0. Here be dragons ===
+Managing large quantities of data is a specialized, mission-critical and difficult task that cannot be undertaken casually.  It is expensive to create a lot of data but very easy to muck it up.  Each research group must appoint an experienced full-time researcher to take custody of its valuable data.  **This is not a job for amateurs.**  If you are intimidated by a raw command line, managing big data is not for you.
+=== 1. Avoid lots of small files ===
+It is cripplingly slow and inefficient to transfer lots of small files.  Roll small files up into a tarball and transfer that.
+<code>
+tar czf LotsOfMyFiles.tgz myDirectoryContainingLotsOfTinyFiles
+</code>
+=== 2. You cannot use files on DIRISA ===
+The DIRISA object stores are designed for storage, not use.  At best you can use the Data Deposit Tool to edit metadata and view the first few lines of files. It is recommended to avoid conducting uploads or downloads of data stored in iRODS directly within a PBS jobscript. You might be wondering, "What about automating my workflow?"
+  * Performance Impact: iRODS is not optimized for high-speed data transfer, especially within the context of a jobscript. Attempting to transfer large amounts of data during job execution can significantly slow down the job and potentially lead to resource contention with other jobs running on the cluster.
+  * Resource Contention: Cluster resources, including network bandwidth and I/O operations, are typically shared among multiple users and jobs. Uploading or downloading data within a jobscript can monopolize these shared resources, causing delays or conflicts with other users' jobs.
+  * Job Portability: Jobscripts are often designed to be portable and self-contained. Embedding data transfer operations within a jobscript can make it less portable and harder to maintain, as users would need to modify the script each time they want to transfer different data.
+  * Data Integrity: Depending on the scale and complexity of data transfer, doing it within a jobscript may increase the risk of data corruption or incomplete transfers, especially if the job terminates unexpectedly.
+It is generally recommended to perform data transfers separately from jobscripts, either before or after job execution, using dedicated tools and scripts. This approach ensures better control over data transfer processes, reduces the impact on job performance, and enhances jobscript portability and reliability.
+=== 3. RTFM ===
+Read the documentation for the iCommands to ensure that you know what these commands and their parameters do.
+=== 4. Make sure of your i's ===
+You are only an **i** away from permanently deleting valuable data.  Proceed with due care.
+<code>
+irm -r myDataDirectory
+</code>
+will delete your data collection at DIRISA.
+<code>
+rm -r myDataDirectory
+</code>
+will irretrievably delete your data on Lustre.
+=== 5. Moving really large data files ===
+To move really large data files, consider using terminal multiplexers like **screen** or **tmux**. These tools allow you to detach from a session, leaving your tasks (such as large data transfers) running in the background. Even if you close the terminal window or your local machine is switched off, the tasks will continue to run. When you're ready, you can open a new terminal window and reattach to the still-running session. This ensures that your tasks continue uninterrupted even when you sign out of LENGAU.

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools