wiki.chpc.ac.za

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
`===== Using DIRISA storage =====
High-performance computing is closely related to "big data".  Many HPC users both use and generate very large data sets.  The CHPC's default recommendation to users is:
  * Feel free to use the CHPC's lustre storage as "work space".  It is large, fast and designed for parallel computing.
  * Lustre is **not** a safe space for storing important data sets.
  * Expedite post-processing and remove raw data.
  * Store post-processed data somewhere safe.

However, the CHPC recognizes that this workflow is not universal.  Many research groups own or produce very large data sets that need to be stored safely for sharing or future re-use.  This is where the [[https://www.dirisa.ac.za/|Data Intensive Research Initiative of South Africa]] comes into play.  Like the CHPC, DIRISA is also part of South Africa's [[https://www.nicis.ac.za/|National Integrated Cyber Infrastructure System]].  DIRISA operates storage systems that **are** designed for long-term safe storage of research data.


==== Registering for DIRISA resources ====
Register for DIRISA's data storage services [[https://service.dirisa.ac.za/|here]].

Here is a [[https://youtu.be/9Uc_F6z1TkA|video tutorial]] of the process.



==== DIRISA Object Stores ====
DIRISA operates two object stores:
  * [[http://irods.org|iRODS]] is an acronym for  Integrated Rule-Oriented Data System
  * The Long Term Archive (LTA) is a tape-based system

These storage systems can be accessed by way of DIRISA's web-based [[https://www.dirisa.ac.za/deposit-data/|Data Deposit Tool]].  However, the data deposit tool is not well suited to the requirements of a CHPC cluster user.  It is intended for smaller quantities of data that are not already stored on NICIS resources.

=== The iCommands ===
The DIRISA object stores are directly accessible from the ''dtn'' node in the CHPC cluster.  The good news is that this data transfer node mounts all the cluster file systems, which makes it possible to transfer data more efficiently to the DIRISA object stores.  The bad news is that DIRISA's storage systems differ fundamentally from the cluster's storage:
  * The DIRISA systems operate as [[https://en.wikipedia.org/wiki/Object_storage|Object Stores]]
  * The cluster storages are accessed as [[https://en.wikipedia.org/wiki/POSIX|POSIX]] file systems, even though "under the bonnet", Lustre is also actually an object storage system.

All computer users are familiar with the directory and file structure of a POSIX file system.  An object store has several compelling advantages for long-term data storage, but lacks the transaction-friendly structure of a file system.  To facilitate the transfer of data to and from the object store, the [[https://docs.irods.org/4.3.0/icommands/user/|iCommands]] tools were developed.  Superficially these are simply the familiar Unix file-handling utilities, prepended with the character ''i'':

^ Unix ^ iCommand      ^ Meaning  |
| cp | icp  | copy               | 
| rm | irm  | delete (remove)    |
| ls | ils  | list files         |
| cd | icd  | change directory   |
| mkdir | imkdir | make directory |
| rmdir | irmdir | remove directory |
|       | iput   | copy file to object store |
|       | iget   | copy file from object store |

It is important to get away from thinking in terms of a directory structure.  The data objects exist in a namespace, and this namespace can be addressed with the iCommands in such a way that it resembles a directory structure.  Data objects in the same namespace can be stored in either the iRODS or the tape system.  From the perspective of the user, the command ''ils'' will display data objects in the current namespace, whether they have been stored in iRODS or tape.  For example:

<code>
[jblogs@dtn:~/lustre]$ ipwd
/dirisa.ac.za/home/jblogs
[jblogs@dtn:~/lustre]$ imkdir myObjectCollection
[jblogs@dtn:~/lustre]$ icd myObjectCollection
[jblogs@dtn:~/lustre]$ iput LotsOfMyFiles.tgz LotsOfMyFiles_toiRODS.tgz
[jblogs@dtn:~/lustre]$ iput -R LTA LotsOfMyFiles.tgz LotsOfMyFiles_toTape.tgz
[jblogs@dtn:~/lustre]$ ils
/dirisa.ac.za/home/jblogs/myObjectCollection:
  LotsOfMyFiles_toiRODS.tgz
  LotsOfMyFiles_toTape.tgz
[jblogs@dtn:~/lustre]$ ils -l
/dirisa.ac.za/home/jblogs/myObjectCollection:
  jblogs           0 dirisa_root;dirisa_replication;CapeTown;irods-resccpt02Resource   2492058420 2023-07-14.18:26 & LotsOfMyFiles_toiRODS.tgz
  jblogs           0 LTA   2492058420 2023-07-14.18:26 & LotsOfMyFiles_toTape.tgz

</code> 

In the above example, a new collection called "myObjectCollection" was created and we moved into that namespace with the ''icd'' command.  Two ''iput'' commands were then executed.  In the first, a file was transferred to the default (iRODS) storage system.  The second ''iput'' command had the parameter ''-R LTA'', which means "use the resource long term archive".  When we check the contents of the collection with ''ils'' we see that both data objects are listed.  However, ''ils -l'' shows that the two objects in the same collection are stored in two totally different storage systems.

==== Important caveats ====
If you don't know what //caveat// means, go look it up.  

=== 0. Here be dragons ===
Managing large quantities of data is a specialized, mission-critical and difficult task that cannot be undertaken casually.  It is expensive to create a lot of data but very easy to muck it up.  Each research group must appoint an experienced full-time researcher to take custody of its valuable data.  **This is not a job for amateurs.**  If you are intimidated by a raw command line, managing big data is not for you.   

=== 1. Avoid lots of small files ===
It is cripplingly slow and inefficient to transfer lots of small files.  Roll small files up into a tarball and transfer that.
<code> 
tar czf LotsOfMyFiles.tgz myDirectoryContainingLotsOfTinyFiles
</code>

=== 2. You cannot use files on DIRISA ===
The DIRISA object stores are designed for storage, not use.  At best you can use the Data Deposit Tool to edit metadata and view the first few lines of files. It is recommended to avoid conducting uploads or downloads of data stored in iRODS directly within a PBS jobscript. You might be wondering, "What about automating my workflow?"
  * Performance Impact: iRODS is not optimized for high-speed data transfer, especially within the context of a jobscript. Attempting to transfer large amounts of data during job execution can significantly slow down the job and potentially lead to resource contention with other jobs running on the cluster.

  * Resource Contention: Cluster resources, including network bandwidth and I/O operations, are typically shared among multiple users and jobs. Uploading or downloading data within a jobscript can monopolize these shared resources, causing delays or conflicts with other users' jobs.

  * Job Portability: Jobscripts are often designed to be portable and self-contained. Embedding data transfer operations within a jobscript can make it less portable and harder to maintain, as users would need to modify the script each time they want to transfer different data.

  * Data Integrity: Depending on the scale and complexity of data transfer, doing it within a jobscript may increase the risk of data corruption or incomplete transfers, especially if the job terminates unexpectedly.

It is generally recommended to perform data transfers separately from jobscripts, either before or after job execution, using dedicated tools and scripts. This approach ensures better control over data transfer processes, reduces the impact on job performance, and enhances jobscript portability and reliability.

=== 3. RTFM === 
Read the documentation for the iCommands to ensure that you know what these commands and their parameters do.

=== 4. Make sure of your i's ===
You are only an **i** away from permanently deleting valuable data.  Proceed with due care.

<code> 
irm -r myDataDirectory 
</code> 
will delete your data collection at DIRISA.

<code>
rm -r myDataDirectory
</code>
will irretrievably delete your data on Lustre.  

=== 5. Moving really large data files ===
To move really large data files, consider using terminal multiplexers like **screen** or **tmux**. These tools allow you to detach from a session, leaving your tasks (such as large data transfers) running in the background. Even if you close the terminal window or your local machine is switched off, the tasks will continue to run. When you're ready, you can open a new terminal window and reattach to the still-running session. This ensures that your tasks continue uninterrupted even when you sign out of LENGAU.
wiki.chpc.ac.za

User Tools

Site Tools

Page Tools
