User Tools

Site Tools


guide:m9000

Using the SUN M9000 SMP

This guide explains how to use the M9000 massive multi-core shared memory super computer.

Hardware

M9000
CPU 64× SPARC64 VII quad-core
CPU clock 2.88 GHz
CPU cores 256 (512 threads available)
Memory 2 048 GB (2 TB)
Peak performance 2 Tflops
Linpack performance 1.032 Tflops
Interconnect Jupiter
O.S. Solaris 10
Storage 5.3 TB XFS
Incept date September 2009

Operating System

The operating system on the M9000 is Oracle (né Sun) Solaris version 10:

m9000:~$ uname -a
SunOS m9000 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise
SunOS is the actual name of the operating system; version 5 was renamed “Solaris” and the .10 in 5.10 refers to the version of Solaris. Wikipedia explains the confusing naming history.

File Systems

Partition File System Mount point Size Usage
Scratch XFS /scratch/* 5.3 TB Scratch (local) storage
/scratch/work/ 5.3 TB Local scratch work directory
Gridware NFS /opt/gridware 1.6 TB Shared software*: compilers, tools, libraries and applications
Home NFS /export/home 1.8 TB Users' home directories from Sun cluster storage
Software /opt/software SPARC64-Solaris compatible software: compilers and libraries
  • WARNING: *Most of the software in /opt/gridware is compiled for the amd64 (x86_64) architecture of the Sun/Dell clusters and is not usable on the sparc architecture of the M9000.

The M9000 does not mount the Lustre SCRATCH* file systems. If you need fast local storage for your processing jobs, request a sub-directory be created for you on /scratch/work/ by emailing .

Compilers

Language GCC Sun Studio
C gcc cc
Fortran gfortran f90 / f95
C++ g++ CC

The recommended system compilers are Sun Studio. The installed version is:

m9000:~$ f90 -V
f90: Sun Fortran 95 8.3 SunOS_sparc Patch 127000-07 2008/10/21
Usage: f90 [ options ] files.  Use 'f90 -flags' for details

The default system GCC compilers are very old (version 3.4.x) and the newer GCC 4.6.3 is installed in /opt/software/gcc-4.6.3/. Add to your path variables:

export PATH=/opt/software/gcc-4.6.3/bin:$PATH
export LD_LIBRARY_PATH=/opt/software/gcc-4.6.3/lib:$LD_LIBRARY_PATH
export MANPATH=/opt/software/gcc-4.6.3/share/man:$MANPATH

And, when building code using make files or configure you will need to specify the compilers to use:

GCC 4.6.3

export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran

Sun Studio

export CC=cc
export CXX=CC
export FC=f90
export F77=f90

OpenMP

The above compilers include support for OpenMP. To compile an OpenMP code with the GCC compilers add the -fopenmp option. For example:

gcc -o mp_hello -fopenmp mp_hello.c

Example code from https://computing.llnl.gov/tutorials/openMP/samples/C/omp_hello.c

With the Sun Studio compilers, use the -xopenmp=parallel option or the safer -xopenmp=noopt option. See Oracle's documentation.

-xopenmp=parallel Enables recognition of OpenMP pragmas. The minimum optimization level for -xopenmp=parallel is -xO3. The compiler changes the optimization from a lower level to -xO3 if necessary, and issues a warning. Not recommended unless you know what you are doing and really want full speed.
-xopenmp=noopt Enables recognition of OpenMP pragmas. The compiler does not raise the optimization level if it is lower than -xO3. If you explicitly set the optimization level lower than -xO3, as in -xO2 -openmp=noopt the compiler will issue an error. If you do not specify an optimization level with -openmp=noopt, the OpenMP pragmas are recognized, the program is parallelized accordingly, but no optimization is done. Recommended as it does not introduce potentially risky out of order optimisations.
  • IMPORTANT Always use validation and verification test benchmarks with your parallel code after compiling with OpenMP enabled and especially if you used any compiler optimisations. Floating point arithmetic is not associative and out of order execution may introduce greater than expected approximation errors.

Libraries

See under /opt/software/ and add to paths (see above examples) as needed.

MPI

Job submission

[Courtesy of Sean February]

The queue name for the M9000 machine is spark. Your job script (see example below) must be located somewhere in /scratch/work/yourusername/ (equivalently /export/home/yourusername/m9_scratch/). Submission should be done normally via the PBS scheduler from the login node:

yourusername@login02:~/m9_scratch $ qsub jobscriptname

Example script:

#!/bin/sh
#PBS -N test
#PBS -q spark
#PBS -l select=1:ncpus=10:mpiprocs=10
#PBS -l place=free
#PBS -l walltime=01:00:00
#PBS -o /scratch/work/yourusername/stdout
#PBS -e /scratch/work/yourusername/stderr
cd $PBS_O_WORKDIR
 
HOME_DIR=/scratch/work/yourusername/
LOG_NAME=testjob_log
THIS_JOB=$HOME_DIR$LOG_NAME
 
echo "My job starts here" > $THIS_JOB
date >> $THIS_JOB
pwd >> $THIS_JOB
echo $PATH >> $THIS_JOB
echo `cat $PBS_NODEFILE` >> $THIS_JOB
 
date >> $THIS_JOB
echo "My job ends here" >> $THIS_JOB 
/var/www/wiki/data/pages/guide/m9000.txt · Last modified: 2015/02/20 15:05 by sfebruary