wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » quick
Trace:
quick:start

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
quick:start [2021/05/05 14:47]
kevin 		quick:start [2026/06/11 11:17] (current)
kevin [Logging in for the First Time]
Line 2: Line 2:
  
 This guide is intended for experienced HPC users and provides a summary of the essential components of the systems available at the CHPC.  For more detailed information on the subjects below see the full [[guide:start|User Guide]]. This guide is intended for experienced HPC users and provides a summary of the essential components of the systems available at the CHPC.  For more detailed information on the subjects below see the full [[guide:start|User Guide]].
- 
-**NOTE: the new system is still under construction and information here and in the User Guide is incomplete and subject to sudden change.** 
  
 //docti cave// //docti cave//
  
  
-===== New 3 PB Lustre storage system ===== +==== Video tutorials for newcomers ==== 
- +If you are a new user to the CHPC, please watch these videos: 
-With effect from 1 May 2021, a new 3 PB Lustre storage system has been implemented.   +  * [[https://youtu.be/fCM8ZaFn4rs|Setting up a cluster password and logging in for the first time]] 
- +  * [[https://youtu.be/CIEVMlrKXF4|The CHPC's service nodes and using the module system]] 
-Please read: **[[howto:datamigration|Moving to the new 3 PB Lustre storage]]** +  * [[https://youtu.be/mQjh8J_aMYU|Test running in an interactive PBS session]] 
- +  * [[https://youtu.be/MEwi4uxM5hc|Running an application in batch mode under PBS]]
-:!: The old 4 PB Lustre storage system **will be //decommissioned// on 1 June 2021** +
- +
-====New directory path:====+
  
-The 3 PB Lustre is mounted on ''/mnt/lustre3p'' and, once you have [[howto:datamigration|copied over your files]], you will need to change the paths in your job scripts.  +====Mailing List====
  
-Search and replace your script files and change ''/mnt/lustre/'' to ''/mnt/lustre3p/''+The **[[https://lists.chpc.ac.za/sympa/info/chpc-users|chpc-users mailing list]]** is the primary communication channel for information about the status of the cluster.  [[https://lists.chpc.ac.za/sympa/subscribe/chpc-users|Subscribe here]] to receive new messages; and //please note you should check your various mail folders including **spam**.// Once subscribed and logged in, you will also be able to view the [[https://lists.chpc.ac.za/sympa/arc/chpc-users|archive]] of past messages.
  
-We will be updating all examples in this wiki to the new path, however, you may still come across an old example with the outmoded path. To make the example wor 
  
 =====Overview: 32 832 cores===== =====Overview: 32 832 cores=====
-The CHPC's brand new Dell Linux cluster is up and running.+The CHPC's Dell Linux cluster has been up and running since 2014.
  
-The new system is an homogeneous cluster, comprising Intel 5th generation CPUs. As of March 2017 it has 1368 compute nodes with 24 cores and 128 GiB* memory (360 nodes have only 64 GiB) each, and five large memory "fat" nodes with 56 cores and 1TiB* each, all interconnected using FDR 56 Gb/s InfiniBand accessing 4 PB of shared storage over the Lustre filesystem. +The new system is an homogeneous cluster, comprising Intel 5th generation CPUs. As of May 2025 it has 41 nodes with 24 cores and 256 GiB memory, 784 compute nodes with 24 cores and 128 GiB* memory, 355 nodes with 24 cores and 64 GiB memory, and four large memory "fat" nodes with 56 cores and 1TiB* each, all interconnected using FDR 56 Gb/s InfiniBand accessing 4 PB of shared storage over the Lustre filesystem. 
  
  
-* Maximum available memory on each type of node: ''mem=124gb'' (regular) or ''mem=61gb'' (regular with only 64GiB), and ''mem=1007gb'' (fat).+* Maximum available memory on each type of node: ''mem=250gb'' (regular with 256 GiB), ''mem=124gb'' (regular with 128 GiB) or ''mem=61gb'' (regular with only 64GiB), and ''mem=1007gb'' (fat).
  
  
Line 44: Line 38:
 </code> </code>
  
 +====Logging in for the First Time====
  
-The new system is running CentOS 7.0 and uses the //Bash// shell by default. +:!: You will use a password only for the **first time** you log in using ssh. And the first step will be to set up your ssh key.  After this, you **can only log in using ssh keys and your password will be disabled.**
  
-You should change your password after logging in the first time. +Please read carefully and follow the procedure to set up your [[guide:sshkeys|ssh keys]] for your first login [[guide:sshkeys|here]].
-To change your password, use the ''passwd'' command. +
-Rules are: 10 characters, with at least one of the following character types: upper and lower case, numbers, and special characters.  Use ssh keys wherever possible.+
  
-Once you have logged in, give some consideration to how you will be using your session on the login node.  If you are going to spend a long time logged in, doing a variety of tasks, it is best to get yourself [[http://wiki.chpc.ac.za/quick:start#example_interactive_job_request|an interactive PBS session]] to work in.  This way, if you need to do something demanding, it will not conflict with other users logged into the login node.+ 
 +====Long Interactive Sessions==== 
 + 
 +Once you have logged in, give some consideration to how you will be using your session on the login node.  If you are going to spend a long time logged in, doing a variety of tasks, it is best to get yourself **[[http://wiki.chpc.ac.za/quick:start#example_interactive_job_request|an interactive PBS session]]** to work in.  This way, if you need to do something demanding, it will not conflict with other users logged into the login node. 
 + 
 +The new system is running [[quick:os|CentOS]] 7.3 and uses the //Bash// shell by default. 
  
 ====Trouble Logging in?==== ====Trouble Logging in?====
Line 81: Line 79:
 From the command line on your Linux workstation: From the command line on your Linux workstation:
 <code> <code>
-scp filetocopy.tar.gz yourusername@scp.chpc.ac.za:/mnt/lustre3p/users/yourusername/run15/+scp filetocopy.tar.gz yourusername@scp.chpc.ac.za:/mnt/lustre/users/yourusername/run15/
 </code> </code>
 transfers the file //filetocopy.tar.gz// from your disk on your computer to  transfers the file //filetocopy.tar.gz// from your disk on your computer to 
 the Lustre file system on the CHPC cluster, under the //run15///             the Lustre file system on the CHPC cluster, under the //run15///            
-subdirectory of your scratch directory ///mnt/lustre3p/users/yourusername///  +subdirectory of your scratch directory ///mnt/lustre/users/yourusername///  
 (where //yourusername// is replaced by your user name on the CHPC cluster). (where //yourusername// is replaced by your user name on the CHPC cluster).
  
Line 112: Line 110:
  
 ^ Mount point  ^  File System ^  Size ^  Quota ^  Backup ^ Access  ^ ^ Mount point  ^  File System ^  Size ^  Quota ^  Backup ^ Access  ^
-| ''/home''  | NFS  | 80 TB  | **15 GB**  | NO <sup>[1]</sup>  | Yes  |  +| ''/home''  | NFS  | 80 TB  | **45 GB**  | NO <sup>[1]</sup>  | Yes  |  
-| ''/mnt/lustre3p/users''  | Lustre  | 4 PB  | none :!:  | NO  | Yes   |  +| ''/mnt/lustre/users''  | Lustre  | 4 PB  | none :!:  | NO  | Yes   |  
-| ''/apps''  | NFS  | 20 TB  | none  | Yes  | On request  |  +| ''/apps''  | NFS  | 20 TB  | none  | Yes  | No <sup>[2]</sup>  |  
-| ''/mnt/lustre3p/groups''  | Lustre  | 1 PB  | 1 TB <sup>[2]</sup>| NO  | On request only  | +| ''/mnt/lustre/groups''  | Lustre  | 1 PB  | 1 TB <sup>[3]</sup>| NO  | On request only  |  
 + 
 +:!: **IMPORTANT NOTE:** Files older than 90 days on ''/mnt/lustre/users'' will be automatically deleted without any warning or advance notice. 
 + 
 +**Note [1]** Unfortunately, at the moment the CHPC cannot guarantee any backup of the ''/home'' file system owing to hardware limitations.
  
-**Note 1:** Unfortunately, at the moment the CHPC cannot guarantee any backup of the ''/home'' file system owing to hardware limitations.+**Note [2]** Create a support ticket on Helpdesk if you would like to request us to install a new application, library or programming tool.
  
-:!: **IMPORTANT NOTE:** Files older than 90 days on ''/mnt/lustre3p/users'' will be automatically deleted without any warning or advance notice.+**Note [3]** Access to ''/mnt/lustre/groups'' is by application only and a quota will be assigned to the programme, to be shared by all members of that group.
  
-**Note 2:** Access to ''/mnt/lustre3p/groups'' is by application only and a quota will be assigned to the programme, to be shared by all members of that group.+It is essential that all files that your job script writes to be on Lustre, apart from scheduler errors you will lose performance because your home directory is on NFS which is not a parallel file system. It is also recommended that all files your jobs scripts reads, especially if large or read more than once, be on Lustre for the same reason.
  
 +It is usually okay to keep binaries and libraries on home since they are read once and loaded into RAM when your executable launches. But you may notice improved performance if they are also on Lustre.
 ====Quotas==== ====Quotas====
  
-The ''/home'' file system is managed by quotas and a strict limit of 15 GB (15 000 000 000 bytes) is applied to it.  Please take care to not fill up your home directory.  Use ''/mnt/lustre3p/users/yourusername'' to store large files.  If your project requires access to large files over a long duration (more than 60 days) then please submit a request to helpdesk.+The ''/home'' file system is managed by quotas and a strict limit of 15 GB (15 000 000 000 bytes) is applied to it.  Please take care to not fill up your home directory.  Use ''/mnt/lustre/users/yourusername'' to store large files.  If your project requires access to large files over a long duration (more than 60 days) then please submit a request to helpdesk.
  
 You can see how much you are currently using with the ''du'' command: You can see how much you are currently using with the ''du'' command:
Line 136: Line 139:
 Make sure that all jobs use a working directory on the Lustre file system.  Do not use your home directory for the working directory of your job.  Use the directory allocated to you on the fast Lustre parallel file system: Make sure that all jobs use a working directory on the Lustre file system.  Do not use your home directory for the working directory of your job.  Use the directory allocated to you on the fast Lustre parallel file system:
 <code> <code>
-/mnt/lustre3p/users/USERNAME/+/mnt/lustre/users/USERNAME/
 </code> </code>
 where ''USERNAME'' is replace by //your// user name on the CHPC cluster. where ''USERNAME'' is replace by //your// user name on the CHPC cluster.
Line 255: Line 258:
 | smp  |  24 |  24 |  20 |  10 |  96 | For single-node parallel jobs.  |  | | smp  |  24 |  24 |  20 |  10 |  96 | For single-node parallel jobs.  |  |
 ^ normal  ^  240 ^  25 ^  20 ^  10 ^  48 ^ The standard queue for parallel jobs ^  ^ ^ normal  ^  240 ^  25 ^  20 ^  10 ^  48 ^ The standard queue for parallel jobs ^  ^
-| large  |  2400 |  264 |  10 |  5 |  48 | For large parallel runs  | //Restricted//  |+| large  |  2400 |  264 |  10 |  5 |  96 | For large parallel runs  | //Restricted//  | 
 +| xlarge  |  6000 |  2424 |  2 |  1 |  96 | For extra-large parallel runs  | //Restricted//  |
 | express  |  2400 |  25 |  N/A |  100 total nodes |  96 | For paid commercial use only  | //Restricted//  | | express  |  2400 |  25 |  N/A |  100 total nodes |  96 | For paid commercial use only  | //Restricted//  |
 | bigmem  |  280 |  28 |  4 |  1 |  48 | For the large memory (1TiB RAM) nodes. | //Restricted//  | | bigmem  |  280 |  28 |  4 |  1 |  48 | For the large memory (1TiB RAM) nodes. | //Restricted//  |
Line 350: Line 354:
 <code bash> <code bash>
 #!/bin/bash #!/bin/bash
-#PBS -l ncpus=24:mpiprocs=24:nodetype=haswell_reg+#PBS -l select=1:ncpus=24:mpiprocs=24
 #PBS -P PRJT1234 #PBS -P PRJT1234
 #PBS -q smp #PBS -q smp
Line 357: Line 361:
 #PBS -e /mnt/lustre/users/USERNAME/OMP_test/test1.err #PBS -e /mnt/lustre/users/USERNAME/OMP_test/test1.err
 #PBS -m abe #PBS -m abe
-#PBS -WMail_Users=youremail@ddress+#PBS -M youremail@ddress
 ulimit -s unlimited ulimit -s unlimited
  
Line 383: Line 387:
 #PBS -e /mnt/lustre/users/USERNAME/WRF_Tests/WRFV3/run2km_100/wrf.err #PBS -e /mnt/lustre/users/USERNAME/WRF_Tests/WRFV3/run2km_100/wrf.err
 #PBS -m abe #PBS -m abe
-#PBS -WMail_Users=youremail@ddress+#PBS -M youremail@ddress
 ulimit -s unlimited ulimit -s unlimited
 . /apps/chpc/earth/WRF-3.7-impi/setWRF . /apps/chpc/earth/WRF-3.7-impi/setWRF
/app/dokuwiki/data/attic/quick/start.1620218850.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki