======Using the SUN M9000 SMP======

This guide explains how to use the M9000 massive multi-core shared memory super computer.

=====Hardware=====

| ^ M9000 |
^ CPU | 64× SPARC64 VII quad-core |
^ CPU clock | 2.88 GHz |
^ CPU cores | 256 (512 threads available) |
^ Memory | 2 048 GB (2 TB)|
^ Peak performance | 2 Tflops |
^ Linpack performance | [[wp>SPARC_Enterprise|1.032 Tflops]] |
^ Interconnect | Jupiter |
^ O.S. | Solaris 10 |
^ Storage | 5.3 TB XFS |
^ Incept date | September 2009 |

=====Operating System=====

The operating system on the M9000 is Oracle (né Sun) Solaris version 10:
<code>
m9000:~$ uname -a
SunOS m9000 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise
</code>
>SunOS is the actual name of the operating system; version 5 was renamed "Solaris" and the .10 in 5.10 refers to the version of Solaris.  Wikipedia explains the confusing naming [[wp>Solaris_%28operating_system%29#History|history]].

=====File Systems=====

^ Partition ^ File System  ^ Mount point ^ Size ^Usage |
| Scratch | XFS | ''/scratch''/* | 5.3 TB | Scratch (local) storage |
| | | ''/scratch/work/'' | 5.3 TB | Local scratch **work** directory |
| Gridware | NFS | ''/opt/gridware'' | 1.6 TB | Shared software*: compilers, tools, libraries and applications |
| Home | NFS | ''/export/home'' | 1.8 TB | Users' home directories from Sun cluster storage |
| Software | | ''/opt/software'' | | SPARC64-Solaris compatible software: compilers and libraries|

  * **WARNING:** *Most of the software in ''/opt/gridware'' is compiled for the amd64 (x86_64) architecture of the Sun/Dell clusters and is **not** usable on the sparc architecture of the M9000.

The M9000 does not mount the Lustre SCRATCH* file systems.  If you need fast local storage for your processing jobs, request a sub-directory be created for you on ''/scratch/work/'' by emailing {{:guide:helpdesk.png?nolink|}}.

=====Compilers=====

| Language ^  GCC         ^  Sun Studio     |
^ C        | ''gcc''      | ''cc''    |
^ Fortran  | ''gfortran'' | ''f90'' / ''f95''  |
^ C++      | ''g++''      | ''CC''   |

The recommended system compilers are Sun Studio.  The installed version is:
<code>
m9000:~$ f90 -V
f90: Sun Fortran 95 8.3 SunOS_sparc Patch 127000-07 2008/10/21
Usage: f90 [ options ] files.  Use 'f90 -flags' for details
</code>

The default system GCC compilers are very old (version 3.4.x) and the newer GCC 4.6.3 is installed in ''/opt/software/gcc-4.6.3/''. Add to your path variables:

<code bash>
export PATH=/opt/software/gcc-4.6.3/bin:$PATH
export LD_LIBRARY_PATH=/opt/software/gcc-4.6.3/lib:$LD_LIBRARY_PATH
export MANPATH=/opt/software/gcc-4.6.3/share/man:$MANPATH
</code>

And, when building code using make files or ''configure'' you will need to specify the compilers to use:

===GCC 4.6.3===
<code bash>
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran
</code>

===Sun Studio===
<code bash>
export CC=cc
export CXX=CC
export FC=f90
export F77=f90
</code>

====OpenMP====

The above compilers include support for OpenMP.  To compile an OpenMP code with the GCC compilers add the ''-fopenmp'' option.  For example:
<code>
gcc -o mp_hello -fopenmp mp_hello.c
</code>

Example code from [[https://computing.llnl.gov/tutorials/openMP/samples/C/omp_hello.c]]

With the Sun Studio compilers, use the ''-xopenmp=parallel'' option or the safer ''-xopenmp=noopt'' option. See Oracle's [[http://docs.oracle.com/cd/E19205-01/820-7883/820-7883.pdf|documentation]].

| ''-xopenmp=parallel'' | Enables recognition of OpenMP pragmas. **The minimum optimization level for ''-xopenmp=parallel'' is ''-xO3''**. The compiler changes the optimization from a lower level to ''-xO3'' if necessary, and issues a warning. | Not recommended unless you know what you are doing and really want full speed. |
^ ''-xopenmp=noopt'' ^ Enables recognition of OpenMP pragmas. The compiler does not raise the optimization level if it is lower than ''-xO3''. If you explicitly set the optimization level lower than ''-xO3'', as in ''-xO2 -openmp=noopt'' the compiler will issue an error. //If you do not specify an optimization level with// ''-openmp=noopt'', //the OpenMP pragmas are recognized, the program is parallelized accordingly, but no optimization is done.// ^ Recommended as it does not introduce potentially risky out of order optimisations. ^

  * **IMPORTANT** Always use validation and verification test benchmarks with your parallel code after compiling with OpenMP enabled and especially if you used any compiler optimisations.  Floating point arithmetic is **not associative** and out of order execution may introduce greater than expected approximation errors.

=====Libraries=====

See under ''/opt/software/'' and add to paths (see above examples) as needed.

====MPI====

=====Job submission=====
[Courtesy of Sean February]

The queue name for the M9000 machine is spark. Your job script (see example below) must be located somewhere in ''/scratch/work/yourusername/'' (equivalently ''/export/home/yourusername/m9_scratch/''). Submission should be done normally via the PBS scheduler from the login node:
<code>
yourusername@login02:~/m9_scratch $ qsub jobscriptname
</code>

Example script:
<code bash>
#!/bin/sh
#PBS -N test
#PBS -q spark
#PBS -l select=1:ncpus=10:mpiprocs=10
#PBS -l place=free
#PBS -l walltime=01:00:00
#PBS -o /scratch/work/yourusername/stdout
#PBS -e /scratch/work/yourusername/stderr
cd $PBS_O_WORKDIR

HOME_DIR=/scratch/work/yourusername/
LOG_NAME=testjob_log
THIS_JOB=$HOME_DIR$LOG_NAME

echo "My job starts here" > $THIS_JOB
date >> $THIS_JOB
pwd >> $THIS_JOB
echo $PATH >> $THIS_JOB
echo `cat $PBS_NODEFILE` >> $THIS_JOB

date >> $THIS_JOB
echo "My job ends here" >> $THIS_JOB 
</code>