Table of Contents
Using the SUN M9000 SMP
This guide explains how to use the M9000 massive multi-core shared memory super computer.
Hardware
| M9000 | |
|---|---|
| CPU | 64× SPARC64 VII quad-core |
| CPU clock | 2.88 GHz |
| CPU cores | 256 (512 threads available) |
| Memory | 2 048 GB (2 TB) |
| Peak performance | 2 Tflops |
| Linpack performance | 1.032 Tflops |
| Interconnect | Jupiter |
| O.S. | Solaris 10 |
| Storage | 5.3 TB XFS |
| Incept date | September 2009 |
Operating System
The operating system on the M9000 is Oracle (né Sun) Solaris version 10:
m9000:~$ uname -a SunOS m9000 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise
SunOS is the actual name of the operating system; version 5 was renamed “Solaris” and the .10 in 5.10 refers to the version of Solaris. Wikipedia explains the confusing naming history.
File Systems
| Partition | File System | Mount point | Size | Usage |
|---|---|---|---|---|
| Scratch | XFS | /scratch/* | 5.3 TB | Scratch (local) storage |
/scratch/work/ | 5.3 TB | Local scratch work directory | ||
| Gridware | NFS | /opt/gridware | 1.6 TB | Shared software*: compilers, tools, libraries and applications |
| Home | NFS | /export/home | 1.8 TB | Users' home directories from Sun cluster storage |
| Software | /opt/software | SPARC64-Solaris compatible software: compilers and libraries |
- WARNING: *Most of the software in
/opt/gridwareis compiled for the amd64 (x86_64) architecture of the Sun/Dell clusters and is not usable on the sparc architecture of the M9000.
The M9000 does not mount the Lustre SCRATCH* file systems. If you need fast local storage for your processing jobs, request a sub-directory be created for you on /scratch/work/ by emailing 
.
Compilers
| Language | GCC | Sun Studio |
|---|---|---|
| C | gcc | cc |
| Fortran | gfortran | f90 / f95 |
| C++ | g++ | CC |
The recommended system compilers are Sun Studio. The installed version is:
m9000:~$ f90 -V f90: Sun Fortran 95 8.3 SunOS_sparc Patch 127000-07 2008/10/21 Usage: f90 [ options ] files. Use 'f90 -flags' for details
The default system GCC compilers are very old (version 3.4.x) and the newer GCC 4.6.3 is installed in /opt/software/gcc-4.6.3/. Add to your path variables:
export PATH=/opt/software/gcc-4.6.3/bin:$PATH export LD_LIBRARY_PATH=/opt/software/gcc-4.6.3/lib:$LD_LIBRARY_PATH export MANPATH=/opt/software/gcc-4.6.3/share/man:$MANPATH
And, when building code using make files or configure you will need to specify the compilers to use:
GCC 4.6.3
export CC=gcc export CXX=g++ export FC=gfortran export F77=gfortran
Sun Studio
export CC=cc export CXX=CC export FC=f90 export F77=f90
OpenMP
The above compilers include support for OpenMP. To compile an OpenMP code with the GCC compilers add the -fopenmp option. For example:
gcc -o mp_hello -fopenmp mp_hello.c
Example code from https://computing.llnl.gov/tutorials/openMP/samples/C/omp_hello.c
With the Sun Studio compilers, use the -xopenmp=parallel option or the safer -xopenmp=noopt option. See Oracle's documentation.
-xopenmp=parallel | Enables recognition of OpenMP pragmas. The minimum optimization level for -xopenmp=parallel is -xO3. The compiler changes the optimization from a lower level to -xO3 if necessary, and issues a warning. | Not recommended unless you know what you are doing and really want full speed. |
-xopenmp=noopt | Enables recognition of OpenMP pragmas. The compiler does not raise the optimization level if it is lower than -xO3. If you explicitly set the optimization level lower than -xO3, as in -xO2 -openmp=noopt the compiler will issue an error. If you do not specify an optimization level with -openmp=noopt, the OpenMP pragmas are recognized, the program is parallelized accordingly, but no optimization is done. | Recommended as it does not introduce potentially risky out of order optimisations. |
|---|
- IMPORTANT Always use validation and verification test benchmarks with your parallel code after compiling with OpenMP enabled and especially if you used any compiler optimisations. Floating point arithmetic is not associative and out of order execution may introduce greater than expected approximation errors.
Libraries
See under /opt/software/ and add to paths (see above examples) as needed.
MPI
Job submission
[Courtesy of Sean February]
The queue name for the M9000 machine is spark. Your job script (see example below) must be located somewhere in /scratch/work/yourusername/ (equivalently /export/home/yourusername/m9_scratch/). Submission should be done normally via the PBS scheduler from the login node:
yourusername@login02:~/m9_scratch $ qsub jobscriptname
Example script:
#!/bin/sh #PBS -N test #PBS -q spark #PBS -l select=1:ncpus=10:mpiprocs=10 #PBS -l place=free #PBS -l walltime=01:00:00 #PBS -o /scratch/work/yourusername/stdout #PBS -e /scratch/work/yourusername/stderr cd $PBS_O_WORKDIR HOME_DIR=/scratch/work/yourusername/ LOG_NAME=testjob_log THIS_JOB=$HOME_DIR$LOG_NAME echo "My job starts here" > $THIS_JOB date >> $THIS_JOB pwd >> $THIS_JOB echo $PATH >> $THIS_JOB echo `cat $PBS_NODEFILE` >> $THIS_JOB date >> $THIS_JOB echo "My job ends here" >> $THIS_JOB
