wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » data
Trace: • quantum_espresso • practicalhpc2022
howto:data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
howto:data [2020/06/23 13:04]
ccrosby [Compressing Files] 		howto:data [2021/12/09 16:42] (current)
Line 3: Line 3:
 ===== Understand how critical data management is ===== ===== Understand how critical data management is =====
 Managing data is an integral part of doing computational analysis work.  Scientific applications generally use and create very large amounts of raw data.  The graphs and tables that eventually show up in reports, theses and research publications represent this data in post-processed form. However, the raw data generated in the process of may take up several Terabytes of data.  Once the raw data has been post-processed, it is seldom necessary to keep it, and it should be removed as soon as possible. Managing data is an integral part of doing computational analysis work.  Scientific applications generally use and create very large amounts of raw data.  The graphs and tables that eventually show up in reports, theses and research publications represent this data in post-processed form. However, the raw data generated in the process of may take up several Terabytes of data.  Once the raw data has been post-processed, it is seldom necessary to keep it, and it should be removed as soon as possible.
 +
 +===== Data management is the responsibility of the Principal Investigator =====
 +Individual users, mostly post-graduate students, come and go.  For this reason, it is the responsibility of the principal investigator to manage the data and storage occupation of their research programme. This is a big task, and we recommend that PIs of large groups appoint a person responsible for managing the compute activities of the group.  Where users need access to shared data, put in a [[https://users.chpc.ac.za/helpdesk/tickets/submit/|helpdesk request]] for a shared group directory to be created. 
 +
 +===== Pre-process - Run - Post-process - Store results - Clean-up =====
 +This mantra is critical:
 +  - Prepare your run, download and link driving data, generate grids, setup initial and boundary conditions, pepare run scripts
 +  - Execute the run
 +  - Inspect the raw results, then post-process the raw information **immediately**
 +  - Store the processed results
 +  - Remove the raw results and driving data, unless these will **immediately** be re-used
 +
 +===== Offload data with Globus =====
 +If you need to move a lot of data off the cluster to your own storage, consider using [[howto:globus|Globus]]
 +
 +===== Request long-term storage from Dirisa =====
 +If you need large amounts of data to be archived or stored in an accessible form, request this from [[https://www.dirisa.ac.za/|Dirisa]], the Data-intensive research branch of the [[http://www.nicis.ac.za|National Integrated Cyber Infrastructure System]].
 +
  
 ===== The CHPC's limited storage capacity ===== ===== The CHPC's limited storage capacity =====
 On the Lengau cluster, users have access to a total of 4 Petabytes of "scratch" storage space.  The hardware underpinning this storage is a Lustre storage cluster, consisting of meta-data servers, object storage servers (OSSs), object storage targets (OSTs), a high-speed Infiniband network and several thousand spinning hard drives.  To the user it looks like a single file system.  This 4 PB scratch space is intended purely for short term storage.  The system prioritises speed over reliability.  The CHPC's mostly unread policy document states clearly that files older than 90 days may be removed at the CHPC's discretion. On the Lengau cluster, users have access to a total of 4 Petabytes of "scratch" storage space.  The hardware underpinning this storage is a Lustre storage cluster, consisting of meta-data servers, object storage servers (OSSs), object storage targets (OSTs), a high-speed Infiniband network and several thousand spinning hard drives.  To the user it looks like a single file system.  This 4 PB scratch space is intended purely for short term storage.  The system prioritises speed over reliability.  The CHPC's mostly unread policy document states clearly that files older than 90 days may be removed at the CHPC's discretion.
 +
 +Please be pro-active about managing your data before we do it for you.  **Without** first asking your permission.  In order to get a list of files that have not been accessed in the last 90 days, use the ''find'' command:  
 +<code>
 +find  -type f -atime +90 -printf "%h/%f, %s, %AD \n"
 +</code> 
 +which will produce a csv table with the path, size in bytes and last time of access.  To make it even easier for yourself, simply delete these files automagically with find:
 +<code>
 +find  -type f -atime +90 -delete
 +</code>
 +
 +
 +
  
 =====Compressing Files===== =====Compressing Files=====
/app/dokuwiki/data/attic/howto/data.1592910253.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki