This is an old revision of the document!
Table of Contents
Using DIRISA storage
High-performance computing is closely related to “big data”. Many HPC users both use and generate very large data sets. The CHPC's default recommendation to users is:
- Feel free to use the CHPC's lustre storage as “work space”. It is large, fast and designed for parallel computing.
- Lustre is not a safe space for storing important data sets.
- Expedite post-processing and remove raw data.
- Store post-processed data somewhere safe.
However, the CHPC recognizes that this workflow is not universal. Many research groups own or produce very large data sets that need to be stored safely for sharing or future re-use. This is where the Data Intensive Research Initiative of South Africa comes into play. Like the CHPC, DIRISA is also part of South Africa's National Integrated Cyber Infrastructure System. DIRISA operates storage systems that are designed for long-term safe storage of research data.
Registering for DIRISA resources
DIRISA is currently working on a comprehensive registration system. However, as an interim measure, a temporary registration system has been set up, consisting of a Google form and registration for the web-based Data Deposit Tool.
1. Application form
Please complete this Google form and submit it.
2. Data Deposit Tool registration
Please create a user account for the Data Deposit Tool. When registering, you will be asked to provide a reason for requesting a user account Specify that you have completed the google form for getting storage and are now requiring a user account. This will ensure that the evaluators process your application and request correctly.
3. Await confirmation on application status
You will receive two confirmations. The first if your user account is approved on the Data Deposit Tool and the second if your application to store data is approved for the archival solution.
DIRISA Object Stores
DIRISA operates two object stores:
- iRODS is an acronym for Integrated Rule-Oriented Data System
- The Long Term Archive (LTA) is a tape-based system
These storage systems can be accessed by way of DIRISA's web-based Data Deposit Tool. However, the data deposit tool is not well suited to the requirements of a CHPC cluster user. It is intended for smaller quantities of data that are not already stored on NICIS resources.
The iCommands
The DIRISA object stores are directly accessible from the dtn node in the CHPC cluster. The good news is that this data transfer node mounts all the cluster file systems, which makes it possible to transfer data more efficiently to the DIRISA object stores. The bad news is that DIRISA's storage systems differ fundamentally from the cluster's storage:
- The DIRISA systems operate as Object Stores
- The cluster storages are accessed as POSIX file systems, even though “under the bonnet”, Lustre is also actually an object storage system.
All computer users are familiar with the directory and file structure of a POSIX file system. An object store has several compelling advantages for long-term data storage, but lacks the transaction-friendly structure of a file system. To facilitate the transfer of data to and from the object store, the iCommands tools were developed. Superficially these are simply the familiar Unix file-handling utilities, prepended with the character i:
| Unix | iCommand | Meaning |
|---|---|---|
| cp | icp | copy |
| rm | irm | delete (remove) |
| ls | ils | list files |
| cd | icd | change directory |
| mkdir | imkdir | make directory |
| rmdir | irmdir | remove directory |
| iput | copy file to object store | |
| iget | copy file from object store |
It is important to get away from thinking in terms of a directory structure. The data objects exist in a namespace, and this namespace can be addressed with the iCommands in such a way that it resembles a directory structure. Data objects in the same namespace can be stored in either the iRODS or the tape system. From the perspective of the user, the command ils will display data objects in the current namespace, whether they have been stored in iRODS or tape. For example:
[jblogs@dtn:~/lustre]$ ipwd /dirisa.ac.za/home/jblogs [jblogs@dtn:~/lustre]$ imkdir myObjectCollection [jblogs@dtn:~/lustre]$ icd myObjectCollection [jblogs@dtn:~/lustre]$ iput LotsOfMyFiles.tgz LotsOfMyFiles_toiRODS.tgz [jblogs@dtn:~/lustre]$ iput -R LTA LotsOfMyFiles.tgz LotsOfMyFiles_toTape.tgz [jblogs@dtn:~/lustre]$ ils /dirisa.ac.za/home/jblogs/myObjectCollection: LotsOfMyFiles_toiRODS.tgz LotsOfMyFiles_toTape.tgz [jblogs@dtn:~/lustre]$ ils -l /dirisa.ac.za/home/jblogs/myObjectCollection: jblogs 0 dirisa_root;dirisa_replication;CapeTown;irods-resccpt02Resource 2492058420 2023-07-14.18:26 & LotsOfMyFiles_toiRODS.tgz jblogs 0 LTA 2492058420 2023-07-14.18:26 & LotsOfMyFiles_toTape.tgz
In the above example, a new collection called “myObjectCollection” was created and we moved into that namespace with the icd command. Two iput commands were then executed. In the first, a file was transferred to the default (iRODS) storage system. The second iput command had the parameter -R LTA, which means “use the resource long term archive”. When we check the contents of the collection with ils we see that both data objects are listed. However, ils -l shows that the two objects in the same collection are stored in two totally different storage systems.
