The Sun system is divided into partitions where each partition represents a separate physical system:

1. harpertown for the Intel Harpertown cluster
2. nehalem for the Intel Nahalem cluster
3. sparc for the Sparc M9000 SMP
4. viz for the AMD Opteron visualization node
5. test for the test (pre- and post-processing) nodes (also Intel Nahalem chipsets)

Users can specify different cluster partitions by adding a -l partition flag in their job submission script. For example

msub -l partition=nehalem

You are able to choose more than one partition, provided, of course, your code can execute on the requested hardware. The syntax looks like:

msub -l partition=nehalem|harpetown

The Moab scheduler has several different queues on the Sun systems for different sized and priority jobs:

1. small (min processors=1 , max processors = 8 , max walltime = 336 hours)
2. par32 (min processors=9 , max processors = 32 , max walltime = 336 hours)
3. par64 ( min processors=33 ,max processors = 64 , max walltime = 336 hours)
4. big ( min processors=65,max processors = 128 , max walltime = 336 hours)
5. test (min processors=1 , max processors = 8 , max walltime = 10 minutes)
6. interactive (max processors = 2 )
7. special ( min processors=129 , max processors = 512 , max walltime = 336hours )
8. priority ( min processors=129 , max processors = 768 , max walltime = 336hours )

NOTE: Users need to log a call at http://www.chpc.ac.za/apply-additional/ two weeks in advance if they need to use the special or priority queues.

Note that it is not necessary for users to specify queues since moab will automatically choose the correct queue for you from the information supplied in your job script e.g. using “msub -l nodes=1: ppn=8 and msub -l walltime”

Job script example:

1.MPI job

#vi mpi.job

#/bin/sh #MSUB -l nodes=1:ppn=8 #MSUB -l partition=nehalem|harpertown #MSUB -l walltime=3:00:00 #MSUB -m be #MSUB -V #MSUB -o /export/home/username/stdout #MSUB -e /export/home/username/stderr #MSUB -d /export/home/username #MSUB -mb ##### Running commands exe=/opt/gridware/usersexecutable nproc=`cat $PBS_NODEFILE | wc -l` mpirun -np $nproc -machinefile $PBS_NODEFILE $exe > users.out

To submit this job (mpi.job) #msub mpi.job by default the job will be submitted to small queue since I have requested 8 processors. 2.Interactive job You have to specify -I flag when submitting your job 3.Test job Users can use the same script and just change the walltime to 10 minutes or less and #MSUB -l partition=nehalem|harpertown to #MSUB -l partition=test. GPU CLUSTER PARTITIONS 1.C2070 [8 of this GPU cards are available] 2.C1060 [12 of this GPU cards are available] Note that GPU cluster has 5 compute nodes and each node has 16 processors and 4 GPU`s Job script example: 1.MPI job over infiniband # vi mpi.job

###These lines are for Moab #MSUB -l nodes=2:ppn=16:gpus=4 #MSUB -l partition=c2070 #MSUB -l walltime=168:00:00 #MSUB -m be #MSUB -V #MSUB -o /GPU/home/username/stdout #MSUB -e /GPU/home/username/stderr #MSUB -d /GPU/home/username #MSUB -mb ##### Running commands echo “original machine file is:” echo “++++++++++” cat $PBS_NODEFILE echo “++++++++++” cat $PBS_NODEFILE|sed -e 's/.*/&-ib/'>$PBS_STEP_OUT.hostfile echo “modified machine file is:” echo “++++++++++” cat $PBS_STEP_OUT.hostfile exe=location of the executable nproc=`cat $PBS_NODEFILE | wc -l` cd /GPU/home/username/testjob/ mpirun $nproc $exe > /GPU/home/username/testjob/OUTPUT

#msub mpi.job ###To submit your job In this example we have requested 4 C2070 GPU`s and 32 processors

Users can only use 32, 64, 128, 256 or 512 processors (ppn).

Job script example:

1.MPI job

# vi mpi.job

###These lines are for Moab #MSUB -l ppn=128 #MSUB -l walltime=168:00:00 #MSUB -m be #MSUB -V #MSUB -o /CHPC/work/username/testjob/out #MSUB -e /CHPC/work/username/testjob/err #MSUB -d /CHPC/work/username/testjob/ #MSUB -mb ##### Running commands /bgsys/drivers/ppcfloor/bin/mpirun -np 128 \ -mode VN -exe LOCATIONOFTHEEXECUTABLE

#msub mpi.job

The following CHPC HPC systems (Sun Microsytems, Blue Gene and GPU) use schedulers and resources managers to schedule and run jobs in the systems. Below is the detail information:

Sun Microsystems - Moab (Scheduler) and Torque (Resource manager).
Blue Gene/P - Loadleveler (Resource manager).
GPU - Torque (Resources manager)

SUBMITTING A JOB USING MOAB

Create a test job.

vi test the file named: job name.
include the following in your script, for more info check http://www.clusterresources.com.
To get the defination and more flags of MSUB do a command "man msub".

###These lines are for Moab
#MSUB -l nodes=4:ppn=8
#MSUB -l walltime=2:00:00
#MSUB -m be
#MSUB -o /export/home/nmonama/scratch/test/dlpoly.3.07.out
#MSUB -e /export/home/nmonama/scratch/test/dlpoly.3.07.err
#MSUB -d /export/home/nmonama/scratch/test
#MSUB -mb
#MSUB -M nmonama@csir.co.za 

##### Running commands
NP=`cat $PBS_NODEFILE | wc –l`
mpirun –np $NP –machinefile $PBS_NODEFILE DLPOLY_3.07.SPARC.Y

SCRIPT	       Description/Ntes

#MSUB -a 	       Declares the time after which the job is eligible for execution. Syntax: (brackets delimit optional items with the default being current date/time):
#MSUB -A account     Defines the account associated with the job.
#MSUB -e	       Defines the file name to be used for stderr.
#MSUB -d path	       Specifies the directory in which the job should begin executing.
#MSUB -h	       Put a user hold on the job at submission time.
#MSUB -j oe	       Combine stdout and stderr into the same output file
#MSUB -l string      Defines the resources that are required by the job
#MSUB -m options     Defines the set of conditions (a=abort,b=begin,e=end) when the server will send a mail message about the job to the user
#MSUB -N name	       Gives a user specified name to the job
#MSUB -o filename    Defines the file name to be used for stdout.
#MSUB -p priority    Assigns a user priority value to a job
#MSUB -q queue       Run the job in the specified queue (short.q,long.q,graphics.q,serial.q and interactive.q)
#MSUB -r y	       Automatically rerun the job is there is a system failure
#MSUB -S path	       Specifies the shell which interprets the job script. The default is your login shell.
#MSUB -v list	       Specifically adds a list (comma separated) of environment variables that are exported to the job
#MSUB -V    	       Declares that all environment variables in the msub environment are exported to the batch job.
#MSUB -W	       This option has been deprecated and should be ignored.

++++

submit test:

msub test -q x (where x=nehalem,harpertown or sparc)

*POST AND PRE PROCESSING*

Please use the following node for post and pre processing:
chpcsp01
chpcsp02

Below is an example of Loadleveler script in Blue Gene/P:

#@ job_type = bluegene
#@ bg_size = 64
#@ class = BGP
#@ input = /dev/null
#@ output = dlpoly.$(jobid).out
#@ error = dlpoly.$(jobid).err
#@ wall_clock_limit = 12:00:00
#@ resources = ConsumableCpus(1)
#@ node_usage = shared
#@ cluster_list = BGP
#@ notification = complete
#@ queue
/bgsys/drivers/ppcfloor/bin/mpirun -np 32 -mode VN -cwd /CHPC/work/username/test -exe EXECUTABLENAME 

There are various controls of Moab and Loadleveler that can help you to manage your job:

Some of Moab controls are as follows: