Differences

This shows you the differences between two versions of the page.

--- quick:start [2017/04/03 10:47]
ccrosby
+++ quick:start [2026/06/11 11:17] (current)
kevin [Logging in for the First Time]
@@ Line 2: / Line 2: @@
 This guide is intended for experienced HPC users and provides a summary of the essential components of the systems available at the CHPC.  For more detailed information on the subjects below see the full [[guide:start|User Guide]].
-**NOTE: the new system is still under construction and information here and in the User Guide is incomplete and subject to sudden change.**
 //docti cave//
+==== Video tutorials for newcomers ====
+If you are a new user to the CHPC, please watch these videos:
+  * [[https://youtu.be/fCM8ZaFn4rs|Setting up a cluster password and logging in for the first time]]
+  * [[https://youtu.be/CIEVMlrKXF4|The CHPC's service nodes and using the module system]]
+  * [[https://youtu.be/mQjh8J_aMYU|Test running in an interactive PBS session]]
+  * [[https://youtu.be/MEwi4uxM5hc|Running an application in batch mode under PBS]]
-=====Overview: 24 472 cores=====
+====Mailing List====
-The CHPC's brand new Dell Linux cluster is up and running.
-The new system is an homogeneous cluster, comprising Intel 5th generation CPUs. As of February 2016 it has 1008 compute nodes with 24 cores and 128 GiB* memory each, and five large memory "fat" nodes with 56 cores and 1TiB* each, all interconnected using FDR 56 Gb/s InfiniBand accessing 4 PB of shared storage over the Lustre filesystem.
+The **[[https://lists.chpc.ac.za/sympa/info/chpc-users|chpc-users mailing list]]** is the primary communication channel for information about the status of the cluster.  [[https://lists.chpc.ac.za/sympa/subscribe/chpc-users|Subscribe here]] to receive new messages; and //please note you should check your various mail folders including **spam**.// Once subscribed and logged in, you will also be able to view the [[https://lists.chpc.ac.za/sympa/arc/chpc-users|archive]] of past messages.
-* Maximum available memory on each type of node: ''mem=125gb'' (regular) and ''mem=1007gb'' (fat).
+=====Overview: 32 832 cores=====
+The CHPC's Dell Linux cluster has been up and running since 2014.
+The new system is an homogeneous cluster, comprising Intel 5th generation CPUs. As of May 2025 it has 41 nodes with 24 cores and 256 GiB memory, 784 compute nodes with 24 cores and 128 GiB* memory, 355 nodes with 24 cores and 64 GiB memory, and four large memory "fat" nodes with 56 cores and 1TiB* each, all interconnected using FDR 56 Gb/s InfiniBand accessing 4 PB of shared storage over the Lustre filesystem.
+* Maximum available memory on each type of node: ''mem=250gb'' (regular with 256 GiB), ''mem=124gb'' (regular with 128 GiB) or ''mem=61gb'' (regular with only 64GiB), and ''mem=1007gb'' (fat).
+====GPU nodes====
+There are 9 compute nodes that contain a total of 30 Nvidia V100 GPUs.  For more information see the [[guide:gpu|GPU guide]].
 =====Logging in=====
@@ Line 25: / Line 38: @@
 </code>
+====Logging in for the First Time====
-The new system is running CentOS 7.0 and uses the //Bash// shell by default.
+:!: You will use a password only for the **first time** you log in using ssh. And the first step will be to set up your ssh key.  After this, you **can only log in using ssh keys and your password will be disabled.**
-You should change your password after logging in the first time.
+Please read carefully and follow the procedure to set up your [[guide:sshkeys|ssh keys]] for your first login [[guide:sshkeys|here]].
-To change your password, use the ''passwd'' command.
-Rules are: 10 characters, with at least one of the following character types: upper and lower case, numbers, and special characters.  Use ssh keys wherever possible.
-====scp/rsync====
-To transfer data onto or off the CHPC cluster use **scp** or **rsync** comands and connect to or from the cluster ''lengau.chpc.ac.za''.
+====Long Interactive Sessions====
+Once you have logged in, give some consideration to how you will be using your session on the login node.  If you are going to spend a long time logged in, doing a variety of tasks, it is best to get yourself **[[http://wiki.chpc.ac.za/quick:start#example_interactive_job_request|an interactive PBS session]]** to work in.  This way, if you need to do something demanding, it will not conflict with other users logged into the login node.
+The new system is running [[quick:os|CentOS]] 7.3 and uses the //Bash// shell by default.
+====Trouble Logging in?====
+Many users have their login blocked at some point. Usually this is because an incorrect password was entered more times than permitted (5 times). This restriction was put in place to prevent brute-force attacks by malicious individuals who want to gain access to your account.
+  * If you cannot log in, the first step is to make sure that you typed your username, hostname (lengau.chpc.ac.za or scp.chpc.ac.za) and password correctly. It sounds stupid, but this is often the problem. It happens to CHPC staff too...
+  * Next, check that you are not experiencing a network problem. If you see a message along the lines of "cannot resolve hostname", then your network is probably at fault (assuming that your spelling is correct).
+  * If your network connection is fine, wait 30 minutes before attempting to log in again. After this period, the block is supposed to be automatically removed.
+  * If for some reason this does not work, you should go to your user page on users.chpc.ac.za. There is a link at that address, to the left, which allows you to change your password and also edit other details for your entry on our user database (email addresses, qualifications, institution, etc.) **Be sure that your password conforms to all requirements**
+  * If even changing the password does not help, please contact our helpdesk, and ask for our assistance.
+==== Transferring Data ====
+There are two main protocols for transferring data to and from the CHPC:
+=== Globus ===
+Globus is a set of tools built on the GridFTP protocol.  Instructions for transferring data with Globus are provided [[howto:globus|here]].
+=== scp ===
+To transfer data onto or off the CHPC cluster use **scp**, **rsync** or sftp commands and connect to or from the cluster's scp node: **''scp.chpc.ac.za''**.  **Please do not transfer files directly to lengau.chpc.ac.za.**  It is a shared login node, and in order not to overload it, file transfer should be done to the dedicated scp server.  For the same reason, when copying large files from one directory to another, it is preferable to not do this on the login node.  You can log into one of several other servers to do the copying:
+  * Use an [[http://wiki.chpc.ac.za/quick:start?s[]=interactive#example_interactive_job_request|Interactive PBS session]] on a compute node
+  * ssh into scp
+  * ssh into chpcviz1
+  * ssh into chpclic1
 ===Examples===
@@ Line 40: / Line 79: @@
 From the command line on your Linux workstation:
 <code>
-scp filetocopy.tar.gz yourusername@lengau.chpc.ac.za:/mnt/lustre/users/yourusername/run15/
+scp filetocopy.tar.gz yourusername@scp.chpc.ac.za:/mnt/lustre/users/yourusername/run15/
 </code>
 transfers the file //filetocopy.tar.gz// from your disk on your computer to
@@ Line 46: / Line 85: @@
 subdirectory of your scratch directory ///mnt/lustre/users/yourusername///
 (where //yourusername// is replaced by your user name on the CHPC cluster).
+=== Downloading files from other servers ===
+You may need to download data from a server at another site.  Do not do this on ** //login2// **!  Use ** // scp.chpc.ac.za// ** for this purpose.  The easiest way of doing this is with the **wget** command:
+<code>
+wget http://someserver.someuni.ac.za/pub/somefile.tgz
+</code>
+Very large files may be transferred more quickly by using a multi-threaded downloader. The easiest of these is **axel**, see [[https://github.com/axel-download-accelerator/axel|axel's GitHub page]].  The syntax is very simple:
+<code>
+module load chpc/compmech/axel/2.17.6
+axel -n 4 -a http://someserver.someuni.ac.za/pub/somefile.tgz
+</code>
 [[guide:connect|Read more on connecting to the CHPC...]]
@@ Line 54: / Line 110: @@
 ^ Mount point  ^  File System ^  Size ^  Quota ^  Backup ^ Access  ^
-| ''/home''  | NFS  | 80 TB  | **15 GB**  | Yes  | Yes  |
+| ''/home''  | NFS  | 80 TB  | **45 GB**  | NO <sup>[1]</sup>  | Yes  |
-| ''/mnt/lustre''  | Lustre  | 4 PB  | none  | NO  | Yes  |
+| ''/mnt/lustre/users''  | Lustre  | 4 PB  | none :!:  | NO  | Yes   |
-| ''/lustre/SCRATCH5''  | Lustre  | 1 PB  | none  | NO  | **//No longer available//**  |
+| ''/apps''  | NFS  | 20 TB  | none  | Yes  | No <sup>[2]</sup>  |
-| ''/apps''  | NFS  | 20 TB  | none  | Yes  | On request  |
+| ''/mnt/lustre/groups''  | Lustre  | 1 PB  | 1 TB <sup>[3]</sup>| NO  | On request only  |
-| ''/lustre/data''  | Lustre  | 1 PB  | none  | NO  | On request only  |
+:!: **IMPORTANT NOTE:** Files older than 90 days on ''/mnt/lustre/users'' will be automatically deleted without any warning or advance notice.
+**Note [1]** Unfortunately, at the moment the CHPC cannot guarantee any backup of the ''/home'' file system owing to hardware limitations.
+**Note [2]** Create a support ticket on Helpdesk if you would like to request us to install a new application, library or programming tool.
+**Note [3]** Access to ''/mnt/lustre/groups'' is by application only and a quota will be assigned to the programme, to be shared by all members of that group.
+It is essential that all files that your job script writes to be on Lustre, apart from scheduler errors you will lose performance because your home directory is on NFS which is not a parallel file system. It is also recommended that all files your jobs scripts reads, especially if large or read more than once, be on Lustre for the same reason.
+It is usually okay to keep binaries and libraries on home since they are read once and loaded into RAM when your executable launches. But you may notice improved performance if they are also on Lustre.
 ====Quotas====
@@ Line 121: / Line 187: @@
 The module command may be abbreviated and optionally be given a search term, eg.:
   module ava chpc/open
+Or, more flexibly, you can pipe stderr to grep, and search for a phrase, such as mpi:
+  module avail 2>&1 | grep mpi
 To see a synopsis of a particular modulefile's operations:
   module help <module_name>
@@ Line 129: / Line 197: @@
 To remove all modules:
   module purge
+To search for a module name or part of a name
+  module-search  partname
 After upgrades of software in ''/apps/'', new modulefiles are created to reflect the changes made to the environment variables.
@@ Line 179: / Line 250: @@
 ''workq'' is no longer to be used.
-The available queues are:
+The available queues with their nominal parameters are given in the following table.  Please take note that these limits may be adjusted dynamically to manage the load on the system.
 ^ Queue Name  ^ Max. cores  ^ Min. cores  ^  Max. jobs  ^^  Max. time  ^  Notes  ^ Access  ^
 ^ :::  ^  per job  ^^  in queue  ^  running  ^  hrs  ^ :::  ^ :::  ^
-| serial  |  24 |  1 |  ??? |  ??? |  48 | For single-node non-parallel jobs.  |  |
+| serial  |  23 |  1 |  24 |  10 |  48 | For single-node non-parallel jobs.  |  |
-| smp  |  24 |  1 |  20 |  10 |  96 | For single-node parallel jobs.  |  |
+| seriallong  |  12 |  1 |  24 |  10 |  144 | For very long sub 1-node jobs.  |  |
-^ normal  ^  240 ^  48 ^  20 ^  10 ^  48 ^ The standard queue for parallel jobs ^  ^
+| smp  |  24 |  24 |  20 |  10 |  96 | For single-node parallel jobs.  |  |
-| large  |  2400 |  264 |  10 |  5 |  48 | For large parallel runs  | //Restricted//  |
+^ normal  ^  240 ^  25 ^  20 ^  10 ^  48 ^ The standard queue for parallel jobs ^  ^
+| large  |  2400 |  264 |  10 |  5 |  96 | For large parallel runs  | //Restricted//  |
+| xlarge  |  6000 |  2424 |  2 |  1 |  96 | For extra-large parallel runs  | //Restricted//  |
+| express  |  2400 |  25 |  N/A |  100 total nodes |  96 | For paid commercial use only  | //Restricted//  |
 | bigmem  |  280 |  28 |  4 |  1 |  48 | For the large memory (1TiB RAM) nodes. | //Restricted//  |
-| vis  |  24 |  1 |  1 |  1 |  3 | Visualisation node  |  |
+| vis  |  12 |  1 |  1 |  1 |  3 | Visualisation node  |  |
 | test  |  24 |  1 |  1 |  1 |  3 | Normal nodes, for testing only  |  |
+| gpu_1 |  10 |  1 |    |  2 |  12 | Up to 10 cpus, 1 GPU        |  |
+| gpu_2 |  20 |  1 |    |  2 |  12 | Up to 20 cpus, 2 GPUs        |  |
+| gpu_3 |  36 |  1 |    |  2 |  12 | Up to 36 cpus, 3 GPUs        |  |
+| gpu_4 |  40 |  1 |    |  2 |  12 | Up to 40 cpus, 4 GPUs        |  |
+| gpu_long |  20 |  1 |    |  1 |  24 | Up to 20 cpus, 1 or 2 GPUs        |  //Restricted//  |
 ===Notes:===
+  * The queue limits may be adjusted dynamically in order to best manage the workload on the system.  Use the command ''qstat -Qf'' to see what the current limits are.
   * A standard compute node has 24 cores and 128 GiB of memory (RAM).
   * Each large memory node has 56 cores and 1 TiB of memory.
   * Access to the ''large'' and ''bigmem'' queues is restricted and by special application only.
+  * To obtain access to the ''large'' queue, you will need to submit satisfactory [[scaling:start|scaling]] results which demonstrate that the use of more than 10 nodes is justified.  This includes demonstrating that ** you ** are competent enough to run large cases.  This can only be done by proving that you can run smaller cases efficiently.
   * Additional restrictions:
 ^  Queue Name  ^  Max. total simultaneous running cores  ^
-| **normal**  |  **480** |
+| **normal**  |  **240** |
-| large  |  4800 |
+| large  |  2400 |
 ====PBS Pro commands====
@@ Line 213: / Line 296: @@
 <code>
- -l select=10:ncpus=24:mpiprocs=24
+ -l select=10:ncpus=24:mpiprocs=24:mem=120gb
 </code>
@@ Line 221: / Line 304: @@
 | ''ncpus=N''     | number of cores //per node//  |
 | ''mpiprocs=N''  | number of MPI ranks (processes) //per node//  |
+| ''mem=Ngb''     | amount of ram //per node// |
 <code>
@@ Line 264: / Line 348: @@
 ====Example job scripts====
+===An OpenMP program using 24 threads on 24 cores===
+Using the ''smp'' queue to run your own program called ''hello_mp.x'' which is in the path:
+<code bash>
+#!/bin/bash
+#PBS -l select=1:ncpus=24:mpiprocs=24
+#PBS -P PRJT1234
+#PBS -q smp
+#PBS -l walltime=4:00:00
+#PBS -o /mnt/lustre/users/USERNAME/OMP_test/test1.out
+#PBS -e /mnt/lustre/users/USERNAME/OMP_test/test1.err
+#PBS -m abe
+#PBS -M youremail@ddress
+ulimit -s unlimited
+cd /mnt/lustre/users/USERNAME/OMP_test
+nproc=`cat $PBS_NODEFILE | wc -l`
+echo nproc is $nproc
+cat $PBS_NODEFILE
+# Run program
+hello_mp.x
+</code>
 ===An MPI program using 240 cores===
@@ Line 277: / Line 387: @@
 #PBS -e /mnt/lustre/users/USERNAME/WRF_Tests/WRFV3/run2km_100/wrf.err
 #PBS -m abe
-#PBS -M your.email@address
+#PBS -M youremail@ddress
 ulimit -s unlimited
 . /apps/chpc/earth/WRF-3.7-impi/setWRF
@@ Line 318: / Line 428: @@
 ====Example interactive job request====
-To request an //interactive session// on a single node, the full command for qsub is:
+To request an //interactive session// on a single core, the full command for qsub is:
+<code>
+qsub -I -P PROJ0101 -q serial -l select=1:ncpus=1:mpiprocs=1:nodetype=haswell_reg
+</code>
+To request an //interactive session// on a full node, the full command for qsub is:
 <code>
@@ Line 325: / Line 441: @@
 **Note:**
+  * Please think carefully about whether you really need a full node, or if 1, 2 or 3 cores might be sufficient
   * ''-I'' selects an //interactive// job
+  * You can add ''-X'' to get X-forwarding
   * you still must specify your project
   * the queue must be ''smp'', ''serial'' or ''test''

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools