User Tools

Site Tools


howto:start

This is an old revision of the document!


How To Guides

Video tutorials for newcomers

Tips and Tricks

Tips & tricks for common tasks. :!: Please read this. The information is based on hard won experience. :!:

Transferring data to DIRISA

High-performance computing is closely related to “big data”. Many HPC users both use and generate very large data sets. The CHPC's default recommendation to users is:

  • Feel free to use the CHPC's lustre storage as “work space”. It is large, fast and designed for parallel computing.
  • Lustre is not a safe space for storing important data sets.
  • Expedite post-processing and remove raw data.
  • Store post-processed data somewhere safe.

However, the CHPC recognizes that this workflow is not universal. Many research groups own or produce very large data sets that need to be stored safely for sharing or future re-use. This is where the Data Intensive Research Initiative of South Africa comes into play. Like the CHPC, DIRISA is also part of South Africa's National Integrated Cyber Infrastructure System. DIRISA operates storage systems that are designed for long-term safe storage of research data.

DIRISA Object Stores

DIRISA operates two object stores:

  • iRODS is an acronym for Integrated Rule-Oriented Data System
  • The Long Term Archive (LTA) is a tape-based system

These storage systems can be accessed by way of DIRISA's web-based Data Deposit Tool. However, the data deposit tool is not well suited to the requirements of a CHPC cluster user. It is intended for smaller quantities of data that are not already stored on NICIS resources.

The iCommands

The DIRISA object stores are directly accessible from the dtn node in the CHPC cluster. The good news is that this data transfer node mounts all the cluster file systems, which makes it possible to transfer data more efficiently to the DIRISA object stores. The bad news is that DIRISA's storage systems differ fundamentally from the cluster's storage:

  • The DIRISA systems operate as Object Stores
  • The cluster storage is used as

Transferring data with Globus

Compilers and Interpreters

A number of computer languages are available on the cluster. The preferred way to set up your environment to use a specific toolset is to load the appropriate modules to activate the needed libraries and set up the paths for the correct binaries.

Dealing with software licensing

Commercial software involves dealing with licensing.

Gaussian at CHPC

Computational Mechanics

Computational Electromagnetics

Bioinformatics

Cosmology and Astrophysics

GPU Computing

Gromacs

DLPoly

  • dlpoly Material Sciences application

Python at the CHPC

R for Statistical Computing

MPI Java

Co-Array Fortran

Omni Compiler

Tuning and Analysis Tools

Building Essential Tools

Remote Visualization

Earth Sciences

/app/dokuwiki/data/attic/howto/start.1689334710.txt.gz · Last modified: 2023/07/14 13:38 by ccrosby