wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » data
Trace: • data
howto:data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
howto:data [2020/06/26 11:02]
ccrosby [Request long-term storage from Dirisa] 		howto:data [2021/12/09 16:42] (current)
Line 14: Line 14:
   - Store the processed results   - Store the processed results
   - Remove the raw results and driving data, unless these will **immediately** be re-used   - Remove the raw results and driving data, unless these will **immediately** be re-used
 +
 +===== Offload data with Globus =====
 +If you need to move a lot of data off the cluster to your own storage, consider using [[howto:globus|Globus]]
  
 ===== Request long-term storage from Dirisa ===== ===== Request long-term storage from Dirisa =====
Line 21: Line 24:
 ===== The CHPC's limited storage capacity ===== ===== The CHPC's limited storage capacity =====
 On the Lengau cluster, users have access to a total of 4 Petabytes of "scratch" storage space.  The hardware underpinning this storage is a Lustre storage cluster, consisting of meta-data servers, object storage servers (OSSs), object storage targets (OSTs), a high-speed Infiniband network and several thousand spinning hard drives.  To the user it looks like a single file system.  This 4 PB scratch space is intended purely for short term storage.  The system prioritises speed over reliability.  The CHPC's mostly unread policy document states clearly that files older than 90 days may be removed at the CHPC's discretion. On the Lengau cluster, users have access to a total of 4 Petabytes of "scratch" storage space.  The hardware underpinning this storage is a Lustre storage cluster, consisting of meta-data servers, object storage servers (OSSs), object storage targets (OSTs), a high-speed Infiniband network and several thousand spinning hard drives.  To the user it looks like a single file system.  This 4 PB scratch space is intended purely for short term storage.  The system prioritises speed over reliability.  The CHPC's mostly unread policy document states clearly that files older than 90 days may be removed at the CHPC's discretion.
 +
 +Please be pro-active about managing your data before we do it for you.  **Without** first asking your permission.  In order to get a list of files that have not been accessed in the last 90 days, use the ''find'' command:  
 +<code>
 +find  -type f -atime +90 -printf "%h/%f, %s, %AD \n"
 +</code> 
 +which will produce a csv table with the path, size in bytes and last time of access.  To make it even easier for yourself, simply delete these files automagically with find:
 +<code>
 +find  -type f -atime +90 -delete
 +</code>
 +
 +
 +
  
 =====Compressing Files===== =====Compressing Files=====
/app/dokuwiki/data/attic/howto/data.1593162129.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki