wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » guide » moab
Trace: • containers • compiling_octave_3_2_4 • pbspro • dell • file_systems • induction1 • pbspro • new • connect • moab
guide:moab

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
guide:moab [2012/11/20 13:40]
oliveroxtoby 		guide:moab [2021/12/09 16:42] (current)
Line 1: Line 1:
 ======A Guide to the Moab Job Scheduler at the CHPC====== ======A Guide to the Moab Job Scheduler at the CHPC======
 +
 +To submit jobs to run on CHPC systems, you must write a shell script that will run your job, and then use ''msub'' to put it on the queue. msub needs information, like which cluster to run your code on, and where to put any errors or output from the job. This extra information can be put within the script, disguised from the shell as special comments, or put on the command line when submitting the job.
  
 ====SUN HYBRID SYSTEM==== ====SUN HYBRID SYSTEM====
Line 13: Line 15:
   - ''sparc'' for the Sparc M9000 SMP   - ''sparc'' for the Sparc M9000 SMP
   - ''viz'' for the AMD Opteron visualization node   - ''viz'' for the AMD Opteron visualization node
-  - ''test'' for the test (pre- and post-processing) nodes (also Intel Nahalem chipsets)+  - ''test'' for the test (pre- and post-processing) nodes (also Intel Nehalem chipsets)
  
 Users can specify different cluster partitions by adding a ''-l'' feature flag in their job submission script or when running the ''msub'' command.  For example Users can specify different cluster partitions by adding a ''-l'' feature flag in their job submission script or when running the ''msub'' command.  For example
Line 22: Line 24:
  
  
-===NEW! Sun and Dell Westmere Clusters Now Available===+===Sun and Dell Westmere Clusters===
  
 The new **Westmere** clusters are now open to be used and we hope that this will reduce the waiting time and the number of queuing jobs. The new Westmere clusters have their own partitions called "''westmere''" for the Sun Westmere cluster and "''dell''" for the Dell Westmere cluster. The new **Westmere** clusters are now open to be used and we hope that this will reduce the waiting time and the number of queuing jobs. The new Westmere clusters have their own partitions called "''westmere''" for the Sun Westmere cluster and "''dell''" for the Dell Westmere cluster.
  
-**NB:** The new Dell cluster runs a different OS from the Sun clusters.  Please see the special instructions in the **//[[guide:dell|Dell]]//** section of this user guide.+**NB:** The new Dell cluster runs a different O/S from the Sun clusters.  Please see the special instructions in the **//[[guide:dell|Dell]]//** section of this user guide.
  
 Please note that each node of Westmere has **12 cores**, so please change the processes per node value from ''ppn=8'' to ''ppn=12'' **only** when you run on the ''westmere'' or ''del'' partition. Please note that each node of Westmere has **12 cores**, so please change the processes per node value from ''ppn=8'' to ''ppn=12'' **only** when you run on the ''westmere'' or ''del'' partition.
Line 68: Line 70:
  
 Edit the file with, for example,  ''vi mpi.job'' Edit the file with, for example,  ''vi mpi.job''
- +<file bash mpi.job> 
-  #/bin/sh +#/bin/sh 
-  #MSUB -l nodes=1:ppn=8 +#MSUB -l nodes=1:ppn=8 
-  #MSUB -l feature=nehalem|harpertown +#MSUB -l feature=nehalem|harpertown 
-  #MSUB -l walltime=3:00:00 +#MSUB -l walltime=3:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -V +#MSUB -V 
-  #MSUB -o /export/home/username/scratch/stdout +#MSUB -o /export/home/username/scratch/stdout 
-  #MSUB -e /export/home/username/scratch/stderr +#MSUB -e /export/home/username/scratch/stderr 
-  #MSUB -d /export/home/username/scratch +#MSUB -d /export/home/username/scratch 
-  #MSUB -mb +#MSUB -mb 
-  ##### Running commands +##### Running commands 
-  exe=/opt/gridware/usersexecutable +exe=/opt/gridware/usersexecutable 
-  nproc=`cat $PBS_NODEFILE | wc -l` +nproc=`cat $PBS_NODEFILE | wc -l` 
-  mpirun -np $nproc -machinefile $PBS_NODEFILE $exe > users.out +mpirun -np $nproc -machinefile $PBS_NODEFILE $exe > users.out 
 +</file>
  
  
Line 100: Line 102:
   msub -I   msub -I
  
 +There will be a delay while the scheduler allocates and logs you into a free compute node. You can target
 +a specific free node:
 +
 +  msub -I -l nodes=cnode-9-23
  
 ===Test jobs=== ===Test jobs===
Line 123: Line 129:
  
 Edit with ''vi mpi.job'' as before. Edit with ''vi mpi.job'' as before.
- +<file bash mpi.job> 
-  ###These lines are for Moab +###These lines are for Moab 
-  #MSUB -l nodes=2:ppn=16:gpus=4 +#MSUB -l nodes=2:ppn=16:gpus=4 
-  #MSUB -l feature=c2070 +#MSUB -l feature=c2070 
-  #MSUB -l walltime=168:00:00 +#MSUB -l walltime=168:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -V +#MSUB -V 
-  #MSUB -o /GPU/home/username/stdout +#MSUB -o /GPU/home/username/stdout 
-  #MSUB -e /GPU/home/username/stderr +#MSUB -e /GPU/home/username/stderr 
-  #MSUB -d /GPU/home/username +#MSUB -d /GPU/home/username 
-  #MSUB -mb +#MSUB -mb 
-  ##### Running commands +##### Running commands 
-  echo "original machine file is:" +echo "original machine file is:" 
-  echo "++++++++++" +echo "++++++++++" 
-  cat $PBS_NODEFILE +cat $PBS_NODEFILE 
-  echo "++++++++++" +echo "++++++++++" 
-  cat $PBS_NODEFILE|sed -e 's/.*/&-ib/'>$PBS_STEP_OUT.hostfile +cat $PBS_NODEFILE|sed -e 's/.*/&-ib/'>$PBS_STEP_OUT.hostfile 
-  echo "modified machine file is:" +echo "modified machine file is:" 
-  echo "++++++++++" +echo "++++++++++" 
-  cat $PBS_STEP_OUT.hostfile +cat $PBS_STEP_OUT.hostfile 
-  exe=location of the executable +exe=location of the executable 
-  nproc=`cat $PBS_NODEFILE | wc -l` +nproc=`cat $PBS_NODEFILE | wc -l` 
-  cd /GPU/home/username/testjob/ +cd /GPU/home/username/testjob/ 
-  mpirun $nproc $exe > /GPU/home/username/testjob/OUTPUT +mpirun $nproc $exe > /GPU/home/username/testjob/OUTPUT 
 +</file>
  
  
Line 154: Line 160:
 In this example we have requested four C2070 GPUs and 32 processors In this example we have requested four C2070 GPUs and 32 processors
  
-====BlueGene/P CLUSTER====+====(Obsolete) BlueGene/P CLUSTER==== 
 + 
 +The BlueGene has been decommissioned and is no longer available.
  
 Users can only use 32, 64, 128, 256 or 512 processors (''ppn''). Users can only use 32, 64, 128, 256 or 512 processors (''ppn'').
Line 163: Line 171:
  
 The ''mpi.job'' file contains: The ''mpi.job'' file contains:
- +<file bash mpi.job> 
-  ###These lines are for Moab +###These lines are for Moab 
-  #MSUB -l ppn=128 +#MSUB -l ppn=128 
-  #MSUB -l walltime=168:00:00 +#MSUB -l walltime=168:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -V +#MSUB -V 
-  #MSUB -o /CHPC/work/username/testjob/out +#MSUB -o /CHPC/work/username/testjob/out 
-  #MSUB -e /CHPC/work/username/testjob/err +#MSUB -e /CHPC/work/username/testjob/err 
-  #MSUB -d /CHPC/work/username/testjob/ +#MSUB -d /CHPC/work/username/testjob/ 
-  #MSUB -mb +#MSUB -mb 
-  ##### Running commands +##### Running commands 
-  /bgsys/drivers/ppcfloor/bin/mpirun -np 128 \ +/bgsys/drivers/ppcfloor/bin/mpirun -np 128 \ 
-  -mode VN -exe LOCATIONOFTHEEXECUTABLE +-mode VN -exe LOCATIONOFTHEEXECUTABLE 
- +</file>
- +
 As before, use ''msub mpi.job'' to submit the job. As before, use ''msub mpi.job'' to submit the job.
  
Line 196: Line 202:
 To get the definitions, and details of other flags, of the  MSUB statements run the command ''man msub'' [[http://www.adaptivecomputing.com/resources/docs/mwm/commands/msub.php|online version]]. To get the definitions, and details of other flags, of the  MSUB statements run the command ''man msub'' [[http://www.adaptivecomputing.com/resources/docs/mwm/commands/msub.php|online version]].
  
- +<file bash job.msub> 
-  ###These lines are for Moab +###These lines are for Moab 
-  #MSUB -l nodes=4:ppn=8 +#MSUB -l nodes=4:ppn=8 
-  #MSUB -l walltime=2:00:00 +#MSUB -l walltime=2:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -o /export/home/nmonama/scratch/test/dlpoly.3.07.out +#MSUB -o /export/home/nmonama/scratch/test/dlpoly.3.07.out 
-  #MSUB -e /export/home/nmonama/scratch/test/dlpoly.3.07.err +#MSUB -e /export/home/nmonama/scratch/test/dlpoly.3.07.err 
-  #MSUB -d /export/home/nmonama/scratch/test +#MSUB -d /export/home/nmonama/scratch/test 
-  #MSUB -mb +#MSUB -mb 
-  #MSUB -M nmonama@csir.co.za+#MSUB -M nmonama@csir.co.za
      
-  ##### Running commands +##### Running commands 
-  NP=`cat $PBS_NODEFILE | wc –l` +NP=`cat $PBS_NODEFILE | wc –l` 
-  mpirun –np $NP –machinefile $PBS_NODEFILE DLPOLY_3.07.SPARC.Y +mpirun –np $NP –machinefile $PBS_NODEFILE DLPOLY_3.07.SPARC.Y 
 +</file>
  
 Details of each line of the job script: Details of each line of the job script:
Line 239: Line 245:
   msub test.job   msub test.job
  
-**NB:** Your job MUST be located on the scratch partition, otherwise it will fail to run.+**NB:** Your job should always write its output to scratch. It will be faster if it reads its input from there as well.
  
 There are various controls of Moab and Loadleveler that can help you to manage your job: There are various controls of Moab and Loadleveler that can help you to manage your job:
Line 251: Line 257:
 ^ showres | Show existing reservations | ^ showres | Show existing reservations |
 ^ showstart | Show estimate of when job can or will start | ^ showstart | Show estimate of when job can or will start |
-^ Checkjob | Provide report for specified job |+^ checkjob | Provide report for specified job |
      
  
/app/dokuwiki/data/attic/guide/moab.1353411654.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki