wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » workshops » hpcschool » chpchardware
Trace: • hpcschool
workshops:hpcschool:chpchardware

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

workshops:hpcschool:chpchardware [2023/02/22 17:41]
wikiadmin created 		workshops:hpcschool:chpchardware [2023/03/01 12:21] (current)
wikiadmin
Line 1: Line 1:
 ======CHPC Hardware====== ======CHPC Hardware======
  
-In Week 2 of the Supercomputing MOOC you were introduced to the EPCC's super computer //Archie//. In this section we describe the CHPC's super computer:  //Lengau//.+In Week 2 of the Supercomputing MOOC you are introduced to the EPCC's super computer //Archie//. In this section we describe the CHPC's super computer:  //Lengau//.
  
 =====Lengau===== =====Lengau=====
Line 40: Line 40:
 //Lengau// was commissioned in June 2016 and, before any users were added to the system, the **HPL** (High Performance LINPACK) benchmark as used by the [[https://www.top500.org/|Top500]] list was run.  The result (using just the general compute nodes): //Lengau// was commissioned in June 2016 and, before any users were added to the system, the **HPL** (High Performance LINPACK) benchmark as used by the [[https://www.top500.org/|Top500]] list was run.  The result (using just the general compute nodes):
  
-Rpeak 1.307 PFlops +|  R<sub>peak</sub> = 1.307 PFlops  | 
-Rmax 1.029 PFlops+^  R<sub>max</sub> = 1.029 PFlops  ^
  
-Other nodes:+====Other nodes:====
  
 In addition to the general compute nodes, //Lengau// also has special purpose nodes.  These are divided into In addition to the general compute nodes, //Lengau// also has special purpose nodes.  These are divided into
  
-    large memory compute nodes +  * large memory compute nodes 
-    GPU compute nodes +  * GPU compute nodes 
-    interactive login nodes +  * interactive login nodes 
-    interactive visualisation nodes +  * interactive visualisation nodes 
-    interactive file transfer nodes +  * interactive file transfer nodes 
-    batch file transfer nodes +  * batch file transfer nodes 
-    management nodes +  * management nodes 
-    storage nodes+  * storage nodes
  
 These are described in separate notes on the [[workshops:hpcschool:start|course page]]. These are described in separate notes on the [[workshops:hpcschool:start|course page]].
  
-Interconnect+====Interconnect====
  
 Lengau actually has three networks connecting all the nodes together: Lengau actually has three networks connecting all the nodes together:
  
-    the main interconnect is Infiniband, +  * the main interconnect is Infiniband, 
-    the second interconnect is an Ethernet network, +  * the second interconnect is an Ethernet network, 
-    and there is a management network.+  * and there is a management network.
  
-Inifiniband+===Infiniband===
  
 The Infiniband interconnect consists of a network of leaf switches connected to the main core switch. Each chassis connects its blades to an Infiniband leaf switch. Thus there are three leaf switches in a rack.  Each leaf switch has 12 connections to the cluster's core switch.  This results is a 2:1 tree network topology. The Infiniband interconnect consists of a network of leaf switches connected to the main core switch. Each chassis connects its blades to an Infiniband leaf switch. Thus there are three leaf switches in a rack.  Each leaf switch has 12 connections to the cluster's core switch.  This results is a 2:1 tree network topology.
  
 Lengau uses FDR Infiniband running with a bandwidth of 56 Gbps and a latency of 5 μs. Lengau uses FDR Infiniband running with a bandwidth of 56 Gbps and a latency of 5 μs.
-Ethernet+ 
 +===Ethernet===
  
 All nodes are also connected to a separate Ethernet network running at 1 Gbps with latency of 50 μs. All nodes are also connected to a separate Ethernet network running at 1 Gbps with latency of 50 μs.
-Management+ 
 +===Management===
  
 A special separate management network is used to control the hardware at a physical level: it is mostly used for power control and administration.  This network is invisible to the users. A special separate management network is used to control the hardware at a physical level: it is mostly used for power control and administration.  This network is invisible to the users.
-Storage+ 
 +=====Storage=====
  
 The third component of a parallel distributed memory super computer is its storage. Lengau has two storage systems that are connected over the two networks. The third component of a parallel distributed memory super computer is its storage. Lengau has two storage systems that are connected over the two networks.
-Lustre Parallel File System+ 
 +===Lustre Parallel File System===
  
 is connected over the Inifiniband network and provides high speed parallel file IO.  Lustre comprises two Meta Data Servers (MDS), 16 Object Store Servers (OSS) and 96 Object STore Targets (OST). is connected over the Inifiniband network and provides high speed parallel file IO.  Lustre comprises two Meta Data Servers (MDS), 16 Object Store Servers (OSS) and 96 Object STore Targets (OST).
Line 86: Line 90:
 The MDS provides file meta-data, the OSSs store the actual file contents in the OSTs.  Each OST is a disk array (using RAID). The MDS provides file meta-data, the OSSs store the actual file contents in the OSTs.  Each OST is a disk array (using RAID).
  
-In total there are 4 PiB of storage space in Lengau's Lustre file system. +In total there is 4 PiB of storage space in Lengau's Lustre file system. 
-NFS+ 
 +===NFS===
  
 is used for home directories and software in /apps.  This is much smaller than Lustre and slower since it is a serial file system. is used for home directories and software in /apps.  This is much smaller than Lustre and slower since it is a serial file system.
-Operating System+ 
 + 
 +=====Operating System=====
  
 All nodes run the CENTOS Linux operating system.  For this reason it is essential that you are familiar with the Linux shell commands and its command line interface (CLI).  This is covered in the next section of this course. All nodes run the CENTOS Linux operating system.  For this reason it is essential that you are familiar with the Linux shell commands and its command line interface (CLI).  This is covered in the next section of this course.
-Cluster Management Software+ 
 +====Cluster Management Software====
  
 The scheduler on the Lengau cluster is Altair's PBSPro.  This will be a major part of this course in the first week. The scheduler on the Lengau cluster is Altair's PBSPro.  This will be a major part of this course in the first week.
/app/dokuwiki/data/attic/workshops/hpcschool/chpchardware.1677080465.txt.gz · Last modified: 2023/02/22 17:41 by wikiadmin

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki