Differences

This shows you the differences between two versions of the page.

--- guide:filesystems [2011/04/07 15:39]
smabakane
+++ guide:filesystems [2026/05/18 13:11] (current)
ccrosby [File Systems]
@@ Line 1: / Line 1: @@
 ====== File Systems ======
-===== SUN systems =====
-The SUN system consisting of Nehalem and Harpertown clusters, the M9000 SMP machine, preprocessing nodes and visualization node has the following file systems:
-  * NFS for home directories ''/export/home///username//''
+:!: To prevent data loss the CHPC has to be strict about its 90 day life time policy for files on Lustre.  Please manage your data carefully and always keep off site backups on all needed files.
-  * Luster file system for fast scratch on ''/export/home///username///scratch''
-**Actual file systems mounted on SUN**
-For example, typing the ''df -h'' command into the SUN login node will show all the available filesystems:
+The primary file systems available to users are:
-  @login02:~> df -h
+^ Name  ^ Mount point  ^  File System ^  Size ^  Quota ^  Backup ^ Access  ^
-  Filesystem            Size  Used Avail Use% Mounted on
+| "Home"  | ''/home''  | NFS  | 80 TB  | 45 GB  | No*  | Yes  |
-  /dev/sda3             119G   62G   52G  55% /
+| "Lustre"  | ''/mnt/lustre/users''  | Lustre  | 4 PB  | none  | NO  | Yes  |
-  udev                  7.9G  900K  7.9G   1% /dev
+| "Apps"  | ''/apps''  | NFS  | 20 TB  | none  | No  | On request  |
-  /dev/sda1             130M   25M   98M  21% /boot
+| "Groups"  | ''/mnt/lustre/groups''  | Lustre  | **1 PB**  | yes*  | NO  | On request only  |
-.17.203.15:/mnt/gridware
-.6T  457G  1.1T  30% /opt/gridware
-.17.203.15:/mnt/home
-.9T  1.3T  442G  76% /export/home
-.17.203.50:/scratch/work
-.1T  2.6T  508G  84% /scratch/work
-.17.203.50:/scratch/home
-G  205G  508G  29% /scratch/home
-.17.195.21@o2ib0:172.17.195.20@o2ib0:/lfs02
-T   36T   33T  52% /lustre/SCRATCH2
-.17.195.20@o2ib0:172.17.195.21@o2ib0:/lfs03
-T   44T   25T  65% /lustre/SCRATCH3
-.17.195.21@o2ib0:172.17.195.20@o2ib0:/lfs04
-T  3.4T   65T   5% /lustre/SCRATCH4
-.17.195.20@o2ib0:172.17.195.21@o2ib0:/lfs01
-T   90G   68T   1% /lustre/SCRATCH1
+In the above table **Name** refers to the colloquial name of the file system, i.e., what it is referred to in ordinary speech or writing.
+=====Home=====
+The home file system contains all users' home directories as is usual for a Linux system.  This is where your standard and primary files reside, most especially your [[guide:connect|ssh]] keys file, your login scripts and other configuration files.
-===== IBM systems =====
+====Quota====
-There are two IBM systems (e1350 and Blue Gene/P cluster). In summary, the e1350 cluster consists of 160 compute nodes, 3 hotspares, 3 storage nodes, back-up node, management node and a single login node. These two systems are connected to the shared Storage Area Networks (SAN) of about 94 Terabytes.
+Your home directory is subject to a strict quota.  Once you exceed that quota you will not be able to write any more to any file within that directory.  This has implications  for files that log various things, e.g. your command line history, and installed applications.
-Below is the structure of the file system in IBM Systems:
+For this reason, large application codes should be installed in your Lustre directory, or you should use a pre-installed application from the Apps file system.
-  [chpcln][/root]> df -h
+To request that an application be installed in Apps please open a [[https://users.chpc.ac.za/helpdesk/tickets/submit/|Helpdesk]] ticket.
-  Filesystem            Size  Used Avail Use% Mounted on
-  /dev/sda2              68G   68G  1.6M 100% /
-  tmpfs                 8.4G  8.0K  8.4G   1% /dev/shm
-  /dev/home             2.8T  2.1T  677G  76% /CHPC/home
-  /dev/local            310G   33G  278G  11% /CHPC/usr/local
-  /dev/work              89T   78T   12T  88% /CHPC/work
-  [chpcln][/root]>
-===== GPU cluster =====
-The GPU has its own designated login node connected to the 5 compute nodes and 6 storage nodes. GPU also has storage capacity of 12 Terabytes and its structured as follows:
+=====Lustre=====
-  [root@hn.gpu ~]# df -h
+The Lustre file system is a high performance parallel file system which is to be used for all running jobs on the Lengau cluster.
-  Filesystem            Size  Used Avail Use% Mounted on
-  /dev/md0               49G   16G   31G  35% /
+This is a work space, or "scratch" file system and is **not** intended for long term storage.  Any files you store on Lustre must be required for your jobs.  Files that have not been //accessed// for more than 90 days will be considered stale and will be deleted.  The data deletion process only removes files, and not directories.  The empty directories may be removed later with a separate process when necessary to preserve meta-data space.
-  tmpfs                  12G     0   12G   0% /dev/shm
-  /dev/gpfs              12T   55G   12T   1% /GPU
+If you rely in your workflow on certain directories that may have been deleted due to the clean-up process, modify your workflow to make it more robust.  You can do this by simply having the necessary ''mkdir'' command in your jobscript.  If the directory has been removed, this will re-create it.  If the directory already exists, it will do no harm.
-  [root@hn.gpu ~]#
+Note that Lustre has a complex structure and is best suited for large files and parallel access. See the separate [[guide:lustre|Lustre guide]] for more information and important tips on use.
+====Job Scripts====
+The //working directory// for all your job scripts must be on Lustre.  Each user is allocated a directory on the Lustre file system under
+  /mnt/lustre/users/username
+where ''username'' is replaced by //your// user name.
+Large codes, data sets, and output data must be stored under your Lustre directory.  This is why we often refer to Lustre as the "work" file system---it's where all your work resides.
+>  Smaller applications, scripts, libraries, etc. can be stored in your home directory, of course.
+====Lustre is NOT Backed Up====
+We do not keep a back up of Lustre.  Any loss of files and data is entirely **unrecoverable**.  It is thus essential that you take data management seriously and ensure that you have adequate off site storage to maintain your own copies of any important files or data on Lustre.
+Lustre is designed for //performance// and capacity.
+To upload or download data to/from Lustre use the ''scp'' login node for small files (~10GB) or the Globus system for **large** files (>10GB).  See the [[guide:connect|Connection Guide]] for more details and important tips.
+=====Apps=====
+The Apps file system contains application codes, libraries, development tools, compilers, etc., installed by the CHPC for our users.
+This system is accessed through the environment modules tool ''module''.  Examples are provided in the [[quick:start|Quick Start Guide]].
+=====Groups=====
+By special arrangement research groups may store files and data that are shared by members of that group.  //However, note that this directory is also subject to the 90-day life time limit.//  This sub-directory of Lustre is limited to 1PB (leaving 2PB for the main Lustre work space) and is subject to a strict quota per group of 1TB.  Where a strong motivation is made by a large research group this quota may be increased up to a maximum of 10TB.

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools