wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » guide » moab
Trace:
guide:moab

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
guide:moab [2011/08/11 18:03]
kevin 		guide:moab [2021/12/09 16:42] (current)
Line 1: Line 1:
 ======A Guide to the Moab Job Scheduler at the CHPC====== ======A Guide to the Moab Job Scheduler at the CHPC======
 +
 +To submit jobs to run on CHPC systems, you must write a shell script that will run your job, and then use ''msub'' to put it on the queue. msub needs information, like which cluster to run your code on, and where to put any errors or output from the job. This extra information can be put within the script, disguised from the shell as special comments, or put on the command line when submitting the job.
  
 ====SUN HYBRID SYSTEM==== ====SUN HYBRID SYSTEM====
Line 7: Line 9:
 The Sun system is divided into //partitions// where each partition represents a separate physical system: The Sun system is divided into //partitions// where each partition represents a separate physical system:
  
 +  - ''dell'' for the new **Dell** Intel Westmere cluster
 +  - ''westmere'' for the Sun Intel Westmere cluster
   - ''harpertown'' for the Intel Harpertown cluster   - ''harpertown'' for the Intel Harpertown cluster
   - ''nehalem'' for the Intel Nahalem cluster   - ''nehalem'' for the Intel Nahalem cluster
   - ''sparc'' for the Sparc M9000 SMP   - ''sparc'' for the Sparc M9000 SMP
   - ''viz'' for the AMD Opteron visualization node   - ''viz'' for the AMD Opteron visualization node
-  - ''test'' for the test (pre- and post-processing) nodes (also Intel Nahalem chipsets)+  - ''test'' for the test (pre- and post-processing) nodes (also Intel Nehalem chipsets)
  
-Users can specify different cluster partitions by adding a ''-l'' partition flag in their job submission script.  For example +Users can specify different cluster partitions by adding a ''-l'' feature flag in their job submission script or when running the ''msub'' command.  For example 
-  msub -l partition=nehalem+  msub -l feature=nehalem
 You are able to choose more than one partition, provided, of course, your code can execute on the requested hardware. The syntax looks like: You are able to choose more than one partition, provided, of course, your code can execute on the requested hardware. The syntax looks like:
-  msub -l partition=nehalem|harpetown+  msub -l feature=nehalem|harpertown 
 +to use both ''nehalem'' and ''harpertown'' clusters. 
 + 
 + 
 +===Sun and Dell Westmere Clusters=== 
 + 
 +The new **Westmere** clusters are now open to be used and we hope that this will reduce the waiting time and the number of queuing jobs. The new Westmere clusters have their own partitions called "''westmere''" for the Sun Westmere cluster and "''dell''" for the Dell Westmere cluster.
  
 +**NB:** The new Dell cluster runs a different O/S from the Sun clusters.  Please see the special instructions in the **//[[guide:dell|Dell]]//** section of this user guide.
 +
 +Please note that each node of Westmere has **12 cores**, so please change the processes per node value from ''ppn=8'' to ''ppn=12'' **only** when you run on the ''westmere'' or ''del'' partition.
 + 
 +To run on one of the Westmere clusters specify the ''westmere'' or ''dell'' partition as follows:
 +
 +For example, in your script
 +  #MSUB -l feature=westmere
 +  #MSUB -l nodes=1:ppn=12 
 + 
 +Or on the command line
 +  msub -l feature=westmere -l nodes=1:ppn=12 <scriptname>
 +
 +See the **//[[guide:dell|Dell]]//** section of this user guide for information on using the new Dell Westmere cluster.
  
 ===QUEUES=== ===QUEUES===
Line 45: Line 69:
 ===MPI job=== ===MPI job===
  
-Edit the file with, eg,+Edit the file with, for example,  ''vi mpi.job'' 
 +<file bash mpi.job> 
 +#/bin/sh 
 +#MSUB -l nodes=1:ppn=8 
 +#MSUB -l feature=nehalem|harpertown 
 +#MSUB -l walltime=3:00:00 
 +#MSUB -m be 
 +#MSUB -V 
 +#MSUB -o /export/home/username/scratch/stdout 
 +#MSUB -e /export/home/username/scratch/stderr 
 +#MSUB -d /export/home/username/scratch 
 +#MSUB -mb 
 +##### Running commands 
 +exe=/opt/gridware/usersexecutable 
 +nproc=`cat $PBS_NODEFILE | wc -l` 
 +mpirun -np $nproc -machinefile $PBS_NODEFILE $exe > users.out 
 +</file>
  
-  vi mpi.job 
- 
---------------- 
- 
-  #/bin/sh 
-  #MSUB -l nodes=1:ppn=8 
-  #MSUB -l partition=nehalem|harpertown 
-  #MSUB -l walltime=3:00:00 
-  #MSUB -m be 
-  #MSUB -V 
-  #MSUB -o /export/home/username/stdout 
-  #MSUB -e /export/home/username/stderr 
-  #MSUB -d /export/home/username 
-  #MSUB -mb 
-  ##### Running commands 
-  exe=/opt/gridware/usersexecutable 
-  nproc=`cat $PBS_NODEFILE | wc -l` 
-  mpirun -np $nproc -machinefile $PBS_NODEFILE $exe > users.out 
- 
- 
----------------------------- 
  
 To submit this job (the ''mpi.job'' file) use the ''msub'' command: To submit this job (the ''mpi.job'' file) use the ''msub'' command:
Line 73: Line 92:
   msub mpi.job   msub mpi.job
  
-by default the job will be submitted to small queue since it only requests 8 processors.+By default the job will be submitted to small queue since it only requests 8 processors. 
 + 
 +**NB:** Your job MUST be located on the scratch partition, otherwise it will fail to run.
  
 ===Interactive job=== ===Interactive job===
  
-You have to specify ''-I'' flag when submitting your job, ie.+You have to specify ''-I'' flag when submitting your job, i.e.
  
   msub -I   msub -I
  
 +There will be a delay while the scheduler allocates and logs you into a free compute node. You can target
 +a specific free node:
 +
 +  msub -I -l nodes=cnode-9-23
  
-===Test job===+===Test jobs===
  
 Users can use the same script and just change the walltime to 10 minutes or less and then change Users can use the same script and just change the walltime to 10 minutes or less and then change
-  #MSUB -l partition=nehalem|harpertown+  #MSUB -l feature=nehalem|harpertown
 to to
-  #MSUB -l partition=test+  #MSUB -l feature=test
 in the job script. in the job script.
  
Line 94: Line 119:
 ===PARTITIONS=== ===PARTITIONS===
  
-  - //C2070// (8 of these Nvidia GPU cards are available) +  - //C2070// (8 of these Nvidia //Fermi// GPU cards are available on two nodes) 
-  - //C1060// (12 of these Nvidia GPU cards are available)+  - //C1060// (12 of these Nvidia //Tesla// GPU cards are available on three nodes)
  
-Note that GPU cluster has 5 compute nodes and each node has 16 processors and 4 GPUs+Note that GPU cluster has 5 compute nodes and each node has 16 processors and 4 GPUs.  In addition the head node (login node) has one each of C2070 and C1060 GPU cards.
  
 ===Job script example:=== ===Job script example:===
Line 104: Line 129:
  
 Edit with ''vi mpi.job'' as before. Edit with ''vi mpi.job'' as before.
 +<file bash mpi.job>
 +###These lines are for Moab
 +#MSUB -l nodes=2:ppn=16:gpus=4
 +#MSUB -l feature=c2070
 +#MSUB -l walltime=168:00:00
 +#MSUB -m be
 +#MSUB -V
 +#MSUB -o /GPU/home/username/stdout
 +#MSUB -e /GPU/home/username/stderr
 +#MSUB -d /GPU/home/username
 +#MSUB -mb
 +##### Running commands
 +echo "original machine file is:"
 +echo "++++++++++"
 +cat $PBS_NODEFILE
 +echo "++++++++++"
 +cat $PBS_NODEFILE|sed -e 's/.*/&-ib/'>$PBS_STEP_OUT.hostfile
 +echo "modified machine file is:"
 +echo "++++++++++"
 +cat $PBS_STEP_OUT.hostfile
 +exe=location of the executable
 +nproc=`cat $PBS_NODEFILE | wc -l`
 +cd /GPU/home/username/testjob/
 +mpirun $nproc $exe > /GPU/home/username/testjob/OUTPUT
 +</file>
  
--------------------------------------------------- 
- 
-  ###These lines are for Moab 
-  #MSUB -l nodes=2:ppn=16:gpus=4 
-  #MSUB -l partition=c2070 
-  #MSUB -l walltime=168:00:00 
-  #MSUB -m be 
-  #MSUB -V 
-  #MSUB -o /GPU/home/username/stdout 
-  #MSUB -e /GPU/home/username/stderr 
-  #MSUB -d /GPU/home/username 
-  #MSUB -mb 
-  ##### Running commands 
-  echo "original machine file is:" 
-  echo "++++++++++" 
-  cat $PBS_NODEFILE 
-  echo "++++++++++" 
-  cat $PBS_NODEFILE|sed -e 's/.*/&-ib/'>$PBS_STEP_OUT.hostfile 
-  echo "modified machine file is:" 
-  echo "++++++++++" 
-  cat $PBS_STEP_OUT.hostfile 
-  exe=location of the executable 
-  nproc=`cat $PBS_NODEFILE | wc -l` 
-  cd /GPU/home/username/testjob/ 
-  mpirun $nproc $exe > /GPU/home/username/testjob/OUTPUT 
- 
-------------------------------------------------------- 
  
 Type ''msub mpi.job'' at the Linux command line to submit your job. Type ''msub mpi.job'' at the Linux command line to submit your job.
Line 137: Line 160:
 In this example we have requested four C2070 GPUs and 32 processors In this example we have requested four C2070 GPUs and 32 processors
  
-====BlueGene/P CLUSTER====+====(Obsolete) BlueGene/P CLUSTER==== 
 + 
 +The BlueGene has been decommissioned and is no longer available.
  
 Users can only use 32, 64, 128, 256 or 512 processors (''ppn''). Users can only use 32, 64, 128, 256 or 512 processors (''ppn'').
Line 146: Line 171:
  
 The ''mpi.job'' file contains: The ''mpi.job'' file contains:
 +<file bash mpi.job>
 +###These lines are for Moab
 +#MSUB -l ppn=128
 +#MSUB -l walltime=168:00:00
 +#MSUB -m be
 +#MSUB -V
 +#MSUB -o /CHPC/work/username/testjob/out
 +#MSUB -e /CHPC/work/username/testjob/err
 +#MSUB -d /CHPC/work/username/testjob/
 +#MSUB -mb
 +##### Running commands
 +/bgsys/drivers/ppcfloor/bin/mpirun -np 128 \
 +-mode VN -exe LOCATIONOFTHEEXECUTABLE
 +</file>
 +As before, use ''msub mpi.job'' to submit the job.
  
------------------------------------------------- 
  
-  ###These lines are for Moab +======Moab Summary======
-  #MSUB -l ppn=128 +
-  #MSUB -l walltime=168:00:00 +
-  #MSUB -m be +
-  #MSUB -V +
-  #MSUB -o /CHPC/work/username/testjob/out +
-  #MSUB -e /CHPC/work/username/testjob/err +
-  #MSUB -d /CHPC/work/username/testjob/ +
-  #MSUB -mb +
-  ##### Running commands +
-  /bgsys/drivers/ppcfloor/bin/mpirun -np 128 \ +
-  -mode VN -exe LOCATIONOFTHEEXECUTABLE+
  
  
--------------------------------------+====SUBMITTING A JOB USING MOAB==== 
 +  
 +Create a test job using a //text// editor: the job script file is a shell script file and should be a plain Ascii text file with Unix style line end characters.  The easiest way to create the file is to use one of the standard Linux text editors, for example, vi or vim:
  
 +  vi test.job
  
-''msub mpi.job'' to submit the job.+to create a file called ''test.job''. Note that you do not have to use the extension ''.job'' for your file but it is a very useful convention that allows you to easily distinguish job script files from other script files (which end in ''.sh'' by convention).
  
 +Include the following in your script, for more info check http://www.clusterresources.com
 +To get the definitions, and details of other flags, of the  MSUB statements run the command ''man msub'' [[http://www.adaptivecomputing.com/resources/docs/mwm/commands/msub.php|online version]].
  
 +<file bash job.msub>
 +###These lines are for Moab
 +#MSUB -l nodes=4:ppn=8
 +#MSUB -l walltime=2:00:00
 +#MSUB -m be
 +#MSUB -o /export/home/nmonama/scratch/test/dlpoly.3.07.out
 +#MSUB -e /export/home/nmonama/scratch/test/dlpoly.3.07.err
 +#MSUB -d /export/home/nmonama/scratch/test
 +#MSUB -mb
 +#MSUB -M nmonama@csir.co.za
 +  
 +##### Running commands
 +NP=`cat $PBS_NODEFILE | wc –l`
 +mpirun –np $NP –machinefile $PBS_NODEFILE DLPOLY_3.07.SPARC.Y
 +</file>
  
-The following CHPC HPC systems (Sun Microsytems, Blue Gene and GPU) use schedulers and resources managers to schedule and run jobs in the systems. Below is the detail information: +Details of each line of the job script:
- +
-  Sun Microsystems - Moab (Scheduler) and Torque (Resource manager). +
-  Blue Gene/P - Loadleveler (Resource manager). +
-  GPU - Torque (Resources manager) +
- +
-SUBMITTING A JOB USING MOAB +
-  +
-  Create a test job. +
-  vi test the file named: job name. +
-  include the following in your script, for more info check http://www.clusterresources.com. +
-  To get the defination and more flags of MSUB do a command "man msub". +
- +
-************************************************ +
- +
-  ###These lines are for Moab +
-  #MSUB -l nodes=4:ppn=8 +
-  #MSUB -l walltime=2:00:00 +
-  #MSUB -m be +
-  #MSUB -o /export/home/nmonama/scratch/test/dlpoly.3.07.out +
-  #MSUB -e /export/home/nmonama/scratch/test/dlpoly.3.07.err +
-  #MSUB -d /export/home/nmonama/scratch/test +
-  #MSUB -mb +
-  #MSUB -M nmonama@csir.co.za  +
- +
-  ##### Running commands +
-  NP=`cat $PBS_NODEFILE | wc –l` +
-  mpirun –np $NP –machinefile $PBS_NODEFILE DLPOLY_3.07.SPARC.Y +
  
   SCRIPT        Description/Ntes   SCRIPT        Description/Ntes
Line 220: Line 240:
   #MSUB -W        This option has been deprecated and should be ignored.   #MSUB -W        This option has been deprecated and should be ignored.
  
-++++ 
  
-submit test:+To submit the  test job script use the following on the Linux command line:
  
-  msub test -q x (where x=nehalem,harpertown or sparc)+  msub test.job
  
-============================== +**NB:** Your job should always write its output to scratch. It will be faster if it reads its input from there as well.
-*POST AND PRE PROCESSING* +
-  Please use the following node for post and pre processing: +
-  chpcsp01 +
-  chpcsp02+
  
-Below is an example of Loadleveler script in Blue Gene/P: 
- 
-  #@ job_type = bluegene 
-  #@ bg_size = 64 
-  #@ class = BGP 
-  #@ input = /dev/null 
-  #@ output = dlpoly.$(jobid).out 
-  #@ error = dlpoly.$(jobid).err 
-  #@ wall_clock_limit = 12:00:00 
-  #@ resources = ConsumableCpus(1) 
-  #@ node_usage = shared 
-  #@ cluster_list = BGP 
-  #@ notification = complete 
-  #@ queue 
-  /bgsys/drivers/ppcfloor/bin/mpirun -np 32 -mode VN -cwd /CHPC/work/username/test -exe EXECUTABLENAME  
- 
-   
 There are various controls of Moab and Loadleveler that can help you to manage your job: There are various controls of Moab and Loadleveler that can help you to manage your job:
      
 Some of Moab controls are as follows: Some of Moab controls are as follows:
 +
 | Command | Description |   | Command | Description |  
 ^ msub | Scheduler job submission | ^ msub | Scheduler job submission |
Line 258: Line 257:
 ^ showres | Show existing reservations | ^ showres | Show existing reservations |
 ^ showstart | Show estimate of when job can or will start | ^ showstart | Show estimate of when job can or will start |
-^ Checkjob | Provide report for specified job | +^ checkjob | Provide report for specified job |
-  +
      
 +
/app/dokuwiki/data/attic/guide/moab.1313078580.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki