This is an old revision of the document!
If you are a new user to the CHPC, please watch these videos:
Tips & tricks for common tasks.
Please read this. The information is based on hard won experience.
High-performance computing is closely related to “big data”. Many HPC users both use and generate very large data sets. The CHPC's default recommendation to users is:
However, the CHPC recognizes that this workflow is not universal. Many research groups own or produce very large data sets that need to be stored safely for sharing or future re-use. This is where the Data Intensive Research Initiative of South Africa comes into play. Like the CHPC, DIRISA is also part of South Africa's National Integrated Cyber Infrastructure System. DIRISA operates storage systems that are designed for long-term safe storage of research data.
DIRISA operates two object stores:
These storage systems can be accessed by way of DIRISA's web-based Data Deposit Tool. However, the data deposit tool is not well suited to the requirements of a CHPC cluster user. It is intended for smaller quantities of data that are not already stored on NICIS resources.
The DIRISA object stores are directly accessible from the dtn node in the CHPC cluster. The good news is that this data transfer node mounts all the cluster file systems, which makes it possible to transfer data more efficiently to the DIRISA object stores. The bad news is that DIRISA's storage systems differ fundamentally from the cluster's storage:
A number of computer languages are available on the cluster. The preferred way to set up your environment to use a specific toolset is to load the appropriate modules to activate the needed libraries and set up the paths for the correct binaries.
Commercial software involves dealing with licensing.