Differences

This shows you the differences between two versions of the page.

--- guide:m9000 [2013/07/25 11:15]
kevin created
+++ guide:m9000 [2021/12/09 16:42] (current)
@@ Line 2: / Line 2: @@
 This guide explains how to use the M9000 massive multi-core shared memory super computer.
+=====Hardware=====
 | ^ M9000 |
-^ CPU | Sparc64 VI |
+^ CPU | 64× SPARC64 VII quad-core |
-^ CPU clock | 1.9 GHz |
+^ CPU clock | 2.88 GHz |
 ^ CPU cores | 256 (512 threads available) |
 ^ Memory | 2 048 GB (2 TB)|
 ^ Peak performance | 2 Tflops |
-^ Linpack peformance | TBA |
+^ Linpack performance | [[wp>SPARC_Enterprise|1.032 Tflops]] |
 ^ Interconnect | Jupiter |
+^ O.S. | Solaris 10 |
 ^ Storage | 5.3 TB XFS |
-^ Incept date | Septemer 2009 |
+^ Incept date | September 2009 |
+=====Operating System=====
+The operating system on the M9000 is Oracle (né Sun) Solaris version 10:
+<code>
+m9000:~$ uname -a
+SunOS m9000 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise
+</code>
+>SunOS is the actual name of the operating system; version 5 was renamed "Solaris" and the .10 in 5.10 refers to the version of Solaris.  Wikipedia explains the confusing naming [[wp>Solaris_%28operating_system%29#History|history]].
+=====File Systems=====
+^ Partition ^ File System  ^ Mount point ^ Size ^Usage |
+| Scratch | XFS | ''/scratch''/* | 5.3 TB | Scratch (local) storage |
+| | | ''/scratch/work/'' | 5.3 TB | Local scratch **work** directory |
+| Gridware | NFS | ''/opt/gridware'' | 1.6 TB | Shared software*: compilers, tools, libraries and applications |
+| Home | NFS | ''/export/home'' | 1.8 TB | Users' home directories from Sun cluster storage |
+| Software | | ''/opt/software'' | | SPARC64-Solaris compatible software: compilers and libraries|
+  * **WARNING:** *Most of the software in ''/opt/gridware'' is compiled for the amd64 (x86_64) architecture of the Sun/Dell clusters and is **not** usable on the sparc architecture of the M9000.
+The M9000 does not mount the Lustre SCRATCH* file systems.  If you need fast local storage for your processing jobs, request a sub-directory be created for you on ''/scratch/work/'' by emailing {{:guide:helpdesk.png?nolink|}}.
+=====Compilers=====
+| Language ^  GCC         ^  Sun Studio     |
+^ C        | ''gcc''      | ''cc''    |
+^ Fortran  | ''gfortran'' | ''f90'' / ''f95''  |
+^ C++      | ''g++''      | ''CC''   |
+The recommended system compilers are Sun Studio.  The installed version is:
+<code>
+m9000:~$ f90 -V
+f90: Sun Fortran 95 8.3 SunOS_sparc Patch 127000-07 2008/10/21
+Usage: f90 [ options ] files.  Use 'f90 -flags' for details
+</code>
+The default system GCC compilers are very old (version 3.4.x) and the newer GCC 4.6.3 is installed in ''/opt/software/gcc-4.6.3/''. Add to your path variables:
+<code bash>
+export PATH=/opt/software/gcc-4.6.3/bin:$PATH
+export LD_LIBRARY_PATH=/opt/software/gcc-4.6.3/lib:$LD_LIBRARY_PATH
+export MANPATH=/opt/software/gcc-4.6.3/share/man:$MANPATH
+</code>
+And, when building code using make files or ''configure'' you will need to specify the compilers to use:
+===GCC 4.6.3===
+<code bash>
+export CC=gcc
+export CXX=g++
+export FC=gfortran
+export F77=gfortran
+</code>
+===Sun Studio===
+<code bash>
+export CC=cc
+export CXX=CC
+export FC=f90
+export F77=f90
+</code>
+====OpenMP====
+The above compilers include support for OpenMP.  To compile an OpenMP code with the GCC compilers add the ''-fopenmp'' option.  For example:
+<code>
+gcc -o mp_hello -fopenmp mp_hello.c
+</code>
+Example code from [[https://computing.llnl.gov/tutorials/openMP/samples/C/omp_hello.c]]
+With the Sun Studio compilers, use the ''-xopenmp=parallel'' option or the safer ''-xopenmp=noopt'' option. See Oracle's [[http://docs.oracle.com/cd/E19205-01/820-7883/820-7883.pdf|documentation]].
+| ''-xopenmp=parallel'' | Enables recognition of OpenMP pragmas. **The minimum optimization level for ''-xopenmp=parallel'' is ''-xO3''**. The compiler changes the optimization from a lower level to ''-xO3'' if necessary, and issues a warning. | Not recommended unless you know what you are doing and really want full speed. |
+^ ''-xopenmp=noopt'' ^ Enables recognition of OpenMP pragmas. The compiler does not raise the optimization level if it is lower than ''-xO3''. If you explicitly set the optimization level lower than ''-xO3'', as in ''-xO2 -openmp=noopt'' the compiler will issue an error. //If you do not specify an optimization level with// ''-openmp=noopt'', //the OpenMP pragmas are recognized, the program is parallelized accordingly, but no optimization is done.// ^ Recommended as it does not introduce potentially risky out of order optimisations. ^
+  * **IMPORTANT** Always use validation and verification test benchmarks with your parallel code after compiling with OpenMP enabled and especially if you used any compiler optimisations.  Floating point arithmetic is **not associative** and out of order execution may introduce greater than expected approximation errors.
+=====Libraries=====
+See under ''/opt/software/'' and add to paths (see above examples) as needed.
+====MPI====
+=====Job submission=====
+[Courtesy of Sean February]
+The queue name for the M9000 machine is spark. Your job script (see example below) must be located somewhere in ''/scratch/work/yourusername/'' (equivalently ''/export/home/yourusername/m9_scratch/''). Submission should be done normally via the PBS scheduler from the login node:
+<code>
+yourusername@login02:~/m9_scratch $ qsub jobscriptname
+</code>
+Example script:
+<code bash>
+#!/bin/sh
+#PBS -N test
+#PBS -q spark
+#PBS -l select=1:ncpus=10:mpiprocs=10
+#PBS -l place=free
+#PBS -l walltime=01:00:00
+#PBS -o /scratch/work/yourusername/stdout
+#PBS -e /scratch/work/yourusername/stderr
+cd $PBS_O_WORKDIR
+HOME_DIR=/scratch/work/yourusername/
+LOG_NAME=testjob_log
+THIS_JOB=$HOME_DIR$LOG_NAME
+echo "My job starts here" > $THIS_JOB
+date >> $THIS_JOB
+pwd >> $THIS_JOB
+echo $PATH >> $THIS_JOB
+echo `cat $PBS_NODEFILE` >> $THIS_JOB
+date >> $THIS_JOB
+echo "My job ends here" >> $THIS_JOB
+</code>

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools