User Tools

Site Tools


quick:start

README

File name: USING CHPC SUN FUSION

File release date: 29 January 2013

File version: 6.00

Author: Technical Team

Email: helpdesk @ chpc.ac.za

Website: http://www.chpc.ac.za

Tel: +27 21 658 2740/58/60

Overview

The SUN Fusion System Linux cluster, named Tsessebe, was made possible by the Department of Science and Technology. Tsessebe entered formal production in September 2009 and supports high-end computational science research throughout South Africa.

The Tsessebe system is comprised of different architectures, that is, the Harpertown, Nehalem, Westmere and SPARC (M9000-64). It has 672 compute nodes giving a total of 6720 cores excluding the SPARC M9000 which has 256 cores with 2 threads per core giving a total of 512 threads. All these architectures are interconnected using the InfiniBand technology and share 480 TB of storage which uses Lustre as the filesystem. It has a LinPack performance of 61.5 TFlops. An overview of the Tsessebe system is shown in figure 1-1.

Tsessebe cluster at CHPC

Architecture

SUN FUSION INFRASTRUSTURE

Cluster feature= Nodes CPU Speed ppn* Performance RAM OS cnodes
HARPERTOWN harpertown 482x Xeon 3.0 GHz 8 3 TFlops 16 GB Redhat 5.8 1
NEHALEM nehalem 2882x Xeon2.93 GHz 8 24 Tflops 12 GB Centos 5.8 2,3,4
WESTMERE westmere 962x Xeon2.93 GHz 12 13.5 TFlops 24 GB Centos 5.8 5
DELL WESTMERE dell 2402x Xeon2.93 GHz 12 37.1 TFlops 36 GB Centos 5.8 6,7,8,9
SUN SPARC M9000 sparc 1Sparcv92.5 GHz 512 2 TFlops 2 TB Solaris 10 m9000
VISUALISATION SERVER ssh viz01 1 4x AMD Opteron2.3 GHz 16 64 GB Redhat 5.1 viz01

*ppn refers to number of cores per node

The cluster as a whole has a measured Linpack performance of 61.5 TFlops.

Filesystems

Tsessebe's shared storage is hosted on 10 Sun x4540 disk servers, each containing 48 integrated SATA drives giving a total of 480 TB of shared storage with Lustre as a filesystem.

Tsessebe has several different filesystems with distinct storage characteristics. There are predefined directories in these filesystems for you to store your data. Since these filesystems are shared with others, they are managed either by a quota limit or a purge policy.

Shared Filesystems

We have the NFS and the Lustre filesystems which are HOME and SCRATCH respectively. The HOME directory has a 5GB quota, backed up filesystem. SCRATCH is periodically purged and not backed up, and has a very large, 450TB quota. All filesystems also impose an inode limit, which affects the number of files allowed.

HOME

  • At login, the system automatically sets you to the home directory (/export/home/username where username is replaced by your login name).
  • Store your important files that should be backed up as home directory is backed up daily.
  • This directory has a quota limit of 5 GB.
  • The login node and any compute node can access this directory.
  • We allow users to install applications in their home directories, but it is important for users to know that CHPC will not support those applications.
  • Users are urged not to run any jobs under this directory they should run jobs in scratch, that is, on the Lustre filesystem. We have a script that automatically kills jobs that are running in home directory.

SCRATCH

  • This is NOT a local disk filesystem on each node.
  • This is a global Lustre filesystem for storing temporary files and that is accessible from compute nodes.
  • Information on Lustre user utilities are available at http://wiki.lustre.org/manual/LustreManual20_HTML/UserUtilities_HTML.html
  • Files on this system are purged when a file's access time exceeds 90 days or when the Lustre filesystem is at 80% usage.
  • A symbolic link (usually named “scratch”) in your home directory points to the Lustre storage. If this is not there, ask Helpdesk to create the symlink.
  • Use cd /export/home/username/scratch to change to SCRATCH.
  • If you require additional storage, please email your request to helpdesk (at) chpc.ac.za

NOTE: CHPC staff may delete files from scratch if the scratch filesystem becomes full, even if files are less than 90 days old. Users are advised to remove their data after a successful run completion. A full filesystem inhibits use of the filesystem for everyone. The use of programs or scripts to actively circumvent the file purge policy will not be tolerated.

Determining your filesystem usage

To determine the amount of disk space used in a filesystem, cd to the directory of interest and execute the following command:

df -h .

It is important to include the dot which represents the current directory. Without the dot, all filesystems are reported.

In the example command output below, the filesystem name appears on the left, and the used and available space (the -h, --human-readable, option prints sizes in the more useful units of G for gigabyte, etc.) appear in the middle columns followed by the percentage used and the mount point:

login01$ df -h .
Filesystem               Size  Used  Avail  Use%   Mounted on
172.17.203.15:/mnt/home  1.9T  1.4T  373G   79%    /export/home

To determine the amount of space occupied in a user-owned directory, cd to the directory and execute the du command:

du -h .

To find the largest sub directory in the current directory you can use a command line pipe like this:

du -s * | sort -n

User Environment

Many of the LINUX commands and tools, such as the compilers, debuggers, profilers, editors, look in the environment for variables that specify information they may need to access. To see the variables in your environment execute the env command:

env

The variables are listed as keyword/value pairs separated by an equal = sign, as illustrated below by the HOME and PATH variables.

HOME=/export/home/oscar
PATH=/opt/gridware/moab/7.1.3/sbin:/opt/gridware/moab/7.1.3/bin:/opt/gridware/torque/2.5.12/sbin:/opt/gridware/torque/2.5.12/bin:/opt/gridware/moab/7.1.3/sbin:/opt/gridware/moab/7.1.3/bin:/opt/gridware/torque/2.5.12/sbin:/opt/gridware/torque/2.5.12/bin:/usr/local/bin:/bin:/usr/bin

Notice that the PATH environment variable consists of a colon (:) separated list of directories. Variables exported in the environment are “carried over” to the environment of shell scripts and new shell invocations provided they are “exported” by the export command:

YEAR=1970
export YEAR

Normal shell variables (created with just an assignment) are useful only in the present shell. Only environment variables are displayed by the env (or printenv) command. Execute set to see the (normal) shell variables.

To access the value (ie, contents) of an environment variable add a $ dollar symbol to the name. For example

echo HOME

will simply display

HOME

while

echo $HOME

will display

/export/home/oscar

Application Development

Software on CHPC Resources

A collection of program libraries and software packages are supported on Tsessebe. These software products for the supercomputing environment have been selected on the basis of quality, history of performance, system compatibility, and benefit to the scientific community. An organized and customizable listing of all packages (name, version, etc.) as well as execution/loading information is available in the Software and Tools Table. The same information for individual packages can be found in the module files on the machine through the execution of the module help command (module help <module_name>).

Modules

CHPC continually updates application packages, compilers, communications libraries, tools, and math libraries. To facilitate this task and to provide a uniform mechanism for accessing different revisions of software, CHPC uses the modules utility, which manipulates your environment.

At login, modules commands set up a basic environment for the default compilers, tools, and libraries. For example: the $PATH, $MANPATH, $LIBPATH environment variables, directory locations and license paths. Therefore, there is no need for you to set them or update them when updates are made to system and application software. Each of the major CHPC applications has a modulefile that sets, unsets, appends to, or prepends to environment variables such as $PATH, $LD_LIBRARY_PATH, $INCLUDE, $MANPATH for the specific application. Each modulefile also sets functions or aliases for use with the application. You need only to invoke a single command to configure the application/programming environment properly. The general format of this command is:

login01$ module load <module_name>

where <module_name> is the name of the module to load. You can add or change the modulepath in your .profile or by setting the environment variable: eg. export MODULEPATH=/opt/gridware/bioinformatics/modules:$MODULEPATH. Most of the package directories are in /opt/gridware/application; /opt/gridware/compilers and /opt/gridware/libraries. They are named after the package. In each package directory there are subdirectories that contain the specific versions of the package. As an example, the Intel 12.1 package requires several environment variables that point to its home, libraries, include files, and documentation. These can be set up by loading the intel2012 module:

login01$ module load intel2012

To see a list of available modules, a synopsis of a particular modulefile's operations (in this case, intel2012), and a list of currently loaded modules, execute the following commands:

login01$ module avail
login01$ module help intel2012
login01$ module list

During upgrades, new modulefiles are created to reflect the changes made to the environment variables. CHPC will always announce upgrades and module changes in advance.

Compilers, libraries and modules

Compilers and libraries that CHPC support are available from the following folders:

  • compilers: /opt/gridware/compilers
  • libraries: /opt/gridware/libraries

To use our libraries , compiliers and some applications you will have to load them as modules.

Here is an example on how to use an amber at CHPC :

module list           ### this will show you modules that are currently loaded in your environment
module avail          ### this will show you modules that are available for the cluster
module add intel      ### loading an intel 2011 module
module add amber      ### loading amber2012 with ambertools module

You can also use Intel compilers by adding the following modules:

module add intel Intel 11.1
module add inteltools Intel 12.0
module add intel2012 Intel 12.1

or sunstudio and gnu

module add sunstudio XXX comment?
module add gcc/version version=4.6.3 or 4.7.2
module add clustertools XXX comment?
module add openmpi/openmpi-1.6.1-gnu compiled with GNU compiler version 4.XXX
module add openmpi/openmpi-1.6.1-intel compiled with Intel compiler version 12.1
Code Name Version Directory Notes
gcc 4.5.1 /opt/gridware/compilers with-gmp
zlib 1.2.7 /opt/gridware/compilers with gcc
ImageMagick 6.7.9 /opt/gridware/compilers with intel 2012
NCO 4.2.1 /opt/gridware/compilers with gcc-4.5.1 , intel 11 and openmpi-1.4.2-intel
netcdf-gnu 4.1.2 /opt/gridware/libraries with gcc
netcdf-intel 4.1.2 /opt/gridware/libraries with intel 2012
Mvapich2 (r5668) 1.8 /opt/gridware/libraries with intel 2012
mvapich 2.1.8 /opt/gridware/libraries with gcc
HDF5 1.8.9 /opt/gridware/compilers with intel 11.1
OpenMPI 1.6.1 /opt/gridware/compilers/OpenMPI with intel 2012
OpenMPI 1.6.1 /opt/gridware/compilers/OpenMPI with gcc
FFTW 3.3.2 /opt/gridware/libraries with intel 2012 , using mvapich2(r5668) mpi lib
FFTW 2.1.5 /opt/gridware/libraries with intel 2012 , using mvapich2(r5668) mpi lib

Applications like gaussian and amber have their own modules.

Note that CHPC only support applications that are installed in /opt/gridware/applications

We discourage users from installing any applications on their home directories. Users can install their own applications in /opt/gridware/users/ which will also not be supported by CHPC.

Logging in

To connect to the Sun systems (eg. M9000, Nehalem cluster) ssh to sun.chpc.ac.za and log in using the username and password sent to you by the CHPC.

If you wish to use the IBM system (eg. the Blue Gene/P) use ssh to go to ssh.chpc.ac.za instead.

more...

STORAGE

CHPC offer 450TB shared temporary lustre storage for users to store their output files. This is a temporary storage, users are advised to remove their data after each successful run. Data will be purged when the storage approaches 80%. Read CHPC Storage Policy: http://www.chpc.ac.za/use-policy

The home directory /export/home/ is backed up daily. Only files that need to be backed up should be stored here. Only max 5GB data per user should be stored in this directory. Applications installed on this directories will not be supported.

If there are any request for storage please send it to helpdesk@chpc.ac.za

Code Name Version Module Directory Notes
gcc 4.5.1 gcc /opt/gridware/compilers with-gmp
zlib 1.2.7 ??? /opt/gridware/compilers with gcc
ImageMagick 6.7.9 ??? /opt/gridware/compilers with Intel 2012
NCO 4.2.1 ??? /opt/gridware/compilers with gcc-4.5.1, intel 11 and openmpi-1.4.2-intel
netcdf-gnu 4.1.2 ??? /opt/gridware/libraries with gcc
netcdf-intel 4.1.2 ??? /opt/gridware/libraries with intel 2012
mvapich2 1.8 ??? /opt/gridware/libraries with intel 2012
mvapich 2.1.8 ??? /opt/gridware/libraries with gcc
HDF5 1.8.9 ??? /opt/gridware/compilers with intel 11.1
OpenMPI 1.6.1 ??? /opt/gridware/compilers/OpenMPI with intel 2012
OpenMPI 1.6.1 ??? /opt/gridware/compilers/OpenMPI with gcc
FFTW 3.3.2 ??? /opt/gridware/libraries with intel 2012, using mvapich2 (r5668) mpi lib
FFTW 2.1.5 ??? /opt/gridware/libraries with intel 2012, using mvapich2 (r5668) mpi lib

XXX WHAT ARE THE MODULE NAMES FOR THESE???

XXX WHERE IS BLAS, ATLAS, LAPACK ???

Customize the .bashrc file

The Bash Unix shell executes the commands in the ~/.bashrc file every bash is launched (i.e. when you login). This file is often used to setup the environmental variables to better suits the specific users needs, for example if your codes are going to be using python2.6.5, openmpi-1.4.2 and gcc-4.3, you can save yourself some type by adding the following line to ~/.bashrc file,

# Python 2.6.5
export PATH=/opt/gridware/python2.6.5/bin:$PATH
export LD_LIBRARY_PATH=/opt/gridware/python2.6.5/lib/

# gridware  general librarys
export LD_LIBRARY_PATH=/opt/gridware/lib/:$LD_LIBRARY_PATH

# openMPI 1.4.2
export PATH=/opt/gridware/openmpi-1.4.2-gnu/bin:$PATH
export LD_LIBRARY_PATH=/opt/gridware/openmpi-1.4.2-gnu/lib:$LD_LIBRARY_PATH
  
# Your Home Directory
export LD_LIBRARY_PATH=~/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=~/lib64:$LD_LIBRARY_PATH
export PATH=~/bin:$PATH

# GCC 4.3.4
export PATH=/opt/gridware/gcc-4.3/bin:$PATH
export LD_LIBRARY_PATH=/opt/gridware/gcc-4.3/lib/:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/gridware/gcc-4.3/lib64/:$LD_LIBRARY_PATH 

The next time you login, the Bash environmental variables will be updated. In the case of the above example, the result would instead of having to type $/opt/gridware/python2.6.5/bin/python , the user can simply

$ python --version
Python 2.6.5

NB After the August 2012 system upgrades, .bashrc is not executed for interactive shells, only .profile is. As such the following line needs to be added to the end of .profile file,

source .bashrc

Compile your code

/var/www/wiki/data/pages/quick/start.txt · Last modified: 2014/02/10 17:55 by ischeepers