wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » guide » moab
Trace: • opensim
guide:moab

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
guide:moab [2013/05/10 12:25]
andyr [SUBMITTING A JOB USING MOAB] 		guide:moab [2021/12/09 16:42] (current)
Line 15: Line 15:
   - ''sparc'' for the Sparc M9000 SMP   - ''sparc'' for the Sparc M9000 SMP
   - ''viz'' for the AMD Opteron visualization node   - ''viz'' for the AMD Opteron visualization node
-  - ''test'' for the test (pre- and post-processing) nodes (also Intel Nahalem chipsets)+  - ''test'' for the test (pre- and post-processing) nodes (also Intel Nehalem chipsets)
  
 Users can specify different cluster partitions by adding a ''-l'' feature flag in their job submission script or when running the ''msub'' command.  For example Users can specify different cluster partitions by adding a ''-l'' feature flag in their job submission script or when running the ''msub'' command.  For example
Line 24: Line 24:
  
  
-===NEW! Sun and Dell Westmere Clusters Now Available===+===Sun and Dell Westmere Clusters===
  
 The new **Westmere** clusters are now open to be used and we hope that this will reduce the waiting time and the number of queuing jobs. The new Westmere clusters have their own partitions called "''westmere''" for the Sun Westmere cluster and "''dell''" for the Dell Westmere cluster. The new **Westmere** clusters are now open to be used and we hope that this will reduce the waiting time and the number of queuing jobs. The new Westmere clusters have their own partitions called "''westmere''" for the Sun Westmere cluster and "''dell''" for the Dell Westmere cluster.
  
-**NB:** The new Dell cluster runs a different OS from the Sun clusters.  Please see the special instructions in the **//[[guide:dell|Dell]]//** section of this user guide.+**NB:** The new Dell cluster runs a different O/S from the Sun clusters.  Please see the special instructions in the **//[[guide:dell|Dell]]//** section of this user guide.
  
 Please note that each node of Westmere has **12 cores**, so please change the processes per node value from ''ppn=8'' to ''ppn=12'' **only** when you run on the ''westmere'' or ''del'' partition. Please note that each node of Westmere has **12 cores**, so please change the processes per node value from ''ppn=8'' to ''ppn=12'' **only** when you run on the ''westmere'' or ''del'' partition.
Line 70: Line 70:
  
 Edit the file with, for example,  ''vi mpi.job'' Edit the file with, for example,  ''vi mpi.job''
- +<file bash mpi.job> 
-  #/bin/sh +#/bin/sh 
-  #MSUB -l nodes=1:ppn=8 +#MSUB -l nodes=1:ppn=8 
-  #MSUB -l feature=nehalem|harpertown +#MSUB -l feature=nehalem|harpertown 
-  #MSUB -l walltime=3:00:00 +#MSUB -l walltime=3:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -V +#MSUB -V 
-  #MSUB -o /export/home/username/scratch/stdout +#MSUB -o /export/home/username/scratch/stdout 
-  #MSUB -e /export/home/username/scratch/stderr +#MSUB -e /export/home/username/scratch/stderr 
-  #MSUB -d /export/home/username/scratch +#MSUB -d /export/home/username/scratch 
-  #MSUB -mb +#MSUB -mb 
-  ##### Running commands +##### Running commands 
-  exe=/opt/gridware/usersexecutable +exe=/opt/gridware/usersexecutable 
-  nproc=`cat $PBS_NODEFILE | wc -l` +nproc=`cat $PBS_NODEFILE | wc -l` 
-  mpirun -np $nproc -machinefile $PBS_NODEFILE $exe > users.out +mpirun -np $nproc -machinefile $PBS_NODEFILE $exe > users.out 
 +</file>
  
  
Line 102: Line 102:
   msub -I   msub -I
  
 +There will be a delay while the scheduler allocates and logs you into a free compute node. You can target
 +a specific free node:
 +
 +  msub -I -l nodes=cnode-9-23
  
 ===Test jobs=== ===Test jobs===
Line 125: Line 129:
  
 Edit with ''vi mpi.job'' as before. Edit with ''vi mpi.job'' as before.
- +<file bash mpi.job> 
-  ###These lines are for Moab +###These lines are for Moab 
-  #MSUB -l nodes=2:ppn=16:gpus=4 +#MSUB -l nodes=2:ppn=16:gpus=4 
-  #MSUB -l feature=c2070 +#MSUB -l feature=c2070 
-  #MSUB -l walltime=168:00:00 +#MSUB -l walltime=168:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -V +#MSUB -V 
-  #MSUB -o /GPU/home/username/stdout +#MSUB -o /GPU/home/username/stdout 
-  #MSUB -e /GPU/home/username/stderr +#MSUB -e /GPU/home/username/stderr 
-  #MSUB -d /GPU/home/username +#MSUB -d /GPU/home/username 
-  #MSUB -mb +#MSUB -mb 
-  ##### Running commands +##### Running commands 
-  echo "original machine file is:" +echo "original machine file is:" 
-  echo "++++++++++" +echo "++++++++++" 
-  cat $PBS_NODEFILE +cat $PBS_NODEFILE 
-  echo "++++++++++" +echo "++++++++++" 
-  cat $PBS_NODEFILE|sed -e 's/.*/&-ib/'>$PBS_STEP_OUT.hostfile +cat $PBS_NODEFILE|sed -e 's/.*/&-ib/'>$PBS_STEP_OUT.hostfile 
-  echo "modified machine file is:" +echo "modified machine file is:" 
-  echo "++++++++++" +echo "++++++++++" 
-  cat $PBS_STEP_OUT.hostfile +cat $PBS_STEP_OUT.hostfile 
-  exe=location of the executable +exe=location of the executable 
-  nproc=`cat $PBS_NODEFILE | wc -l` +nproc=`cat $PBS_NODEFILE | wc -l` 
-  cd /GPU/home/username/testjob/ +cd /GPU/home/username/testjob/ 
-  mpirun $nproc $exe > /GPU/home/username/testjob/OUTPUT +mpirun $nproc $exe > /GPU/home/username/testjob/OUTPUT 
 +</file>
  
  
Line 156: Line 160:
 In this example we have requested four C2070 GPUs and 32 processors In this example we have requested four C2070 GPUs and 32 processors
  
-====BlueGene/P CLUSTER====+====(Obsolete) BlueGene/P CLUSTER==== 
 + 
 +The BlueGene has been decommissioned and is no longer available.
  
 Users can only use 32, 64, 128, 256 or 512 processors (''ppn''). Users can only use 32, 64, 128, 256 or 512 processors (''ppn'').
Line 165: Line 171:
  
 The ''mpi.job'' file contains: The ''mpi.job'' file contains:
- +<file bash mpi.job> 
-  ###These lines are for Moab +###These lines are for Moab 
-  #MSUB -l ppn=128 +#MSUB -l ppn=128 
-  #MSUB -l walltime=168:00:00 +#MSUB -l walltime=168:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -V +#MSUB -V 
-  #MSUB -o /CHPC/work/username/testjob/out +#MSUB -o /CHPC/work/username/testjob/out 
-  #MSUB -e /CHPC/work/username/testjob/err +#MSUB -e /CHPC/work/username/testjob/err 
-  #MSUB -d /CHPC/work/username/testjob/ +#MSUB -d /CHPC/work/username/testjob/ 
-  #MSUB -mb +#MSUB -mb 
-  ##### Running commands +##### Running commands 
-  /bgsys/drivers/ppcfloor/bin/mpirun -np 128 \ +/bgsys/drivers/ppcfloor/bin/mpirun -np 128 \ 
-  -mode VN -exe LOCATIONOFTHEEXECUTABLE +-mode VN -exe LOCATIONOFTHEEXECUTABLE 
- +</file>
- +
 As before, use ''msub mpi.job'' to submit the job. As before, use ''msub mpi.job'' to submit the job.
  
Line 198: Line 202:
 To get the definitions, and details of other flags, of the  MSUB statements run the command ''man msub'' [[http://www.adaptivecomputing.com/resources/docs/mwm/commands/msub.php|online version]]. To get the definitions, and details of other flags, of the  MSUB statements run the command ''man msub'' [[http://www.adaptivecomputing.com/resources/docs/mwm/commands/msub.php|online version]].
  
- +<file bash job.msub> 
-  ###These lines are for Moab +###These lines are for Moab 
-  #MSUB -l nodes=4:ppn=8 +#MSUB -l nodes=4:ppn=8 
-  #MSUB -l walltime=2:00:00 +#MSUB -l walltime=2:00:00 
-  #MSUB -m be +#MSUB -m be 
-  #MSUB -o /export/home/nmonama/scratch/test/dlpoly.3.07.out +#MSUB -o /export/home/nmonama/scratch/test/dlpoly.3.07.out 
-  #MSUB -e /export/home/nmonama/scratch/test/dlpoly.3.07.err +#MSUB -e /export/home/nmonama/scratch/test/dlpoly.3.07.err 
-  #MSUB -d /export/home/nmonama/scratch/test +#MSUB -d /export/home/nmonama/scratch/test 
-  #MSUB -mb +#MSUB -mb 
-  #MSUB -M nmonama@csir.co.za+#MSUB -M nmonama@csir.co.za
      
-  ##### Running commands +##### Running commands 
-  NP=`cat $PBS_NODEFILE | wc –l` +NP=`cat $PBS_NODEFILE | wc –l` 
-  mpirun –np $NP –machinefile $PBS_NODEFILE DLPOLY_3.07.SPARC.Y +mpirun –np $NP –machinefile $PBS_NODEFILE DLPOLY_3.07.SPARC.Y 
 +</file>
  
 Details of each line of the job script: Details of each line of the job script:
Line 253: Line 257:
 ^ showres | Show existing reservations | ^ showres | Show existing reservations |
 ^ showstart | Show estimate of when job can or will start | ^ showstart | Show estimate of when job can or will start |
-^ Checkjob | Provide report for specified job |+^ checkjob | Provide report for specified job |
      
  
/app/dokuwiki/data/attic/guide/moab.1368181502.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki