wiki.chpc.ac.za

Aim: to launch a parallel job on the CHPC cluster Lengau.

The slides for the OpenMP lecture on Linux will be posted.

Download and install putty-ssh and connect to lengau.chpc.ac.za using the username (studentNN) and password provided.

Documentation for the CHPC systems starts with the CHPC Quick Start Guide. Read it, but note the differences below which are specific to the studentNN accounts you are using.

• Your user name is: studentNN
• Your project is: Wchpc
• Your queue is: R522145
• Your PBS group is: training

A user can not simply ssh onto a compute node because you don't know which ones are not in use by a currently running job. But the PBSPro scheduler does know. So we ask the scheduler for an interactive job which gives us a login shell on a compute node:

qsub -I -P Wchpc -q R522145 -W group_list=training

Note:

• -I selects an interactive job
• you must specify the project Wchpc
• the queue is R522145
• you need to specify the training group.

Advanced options:

• -l select=1:ncpus=24:mpiprocs=24:nodetype=haswell_reg
• interactive jobs only get one node: select=1
• you can request all the cores on the node: ncpus=24
• you can run MPI code: indicate how many ranks you want with mpiprocs=

If you find your interactive session timing out too soon then add -l walltime=4:0:0 to the above command line to request the maximum 4 hours.

To carry out the exercises in the first OpenMP practical you must run the examples on a compute node and not the login node.

After logging on to lengau, the first thing you will do is request an interactive session on a compute node:

qsub -I -P Wchpc

Your prompt will change to something like:

[student00@cnode0004 ~]$

Remember, this is a login shell so you can change the environment as before, for example, change the prompt with

export PS1='\h:\w\$ '

and now see

cnode0004:~$ 

Download and follow the worksheet.

Copy the OpenMP_Tut.tar.gz file from the student00 home directory to your home directory.

cp ~/../student00/OpenMP_Tut.tar.gz ~

Then untar (unpack) it.

tar -zxvf ~/OpenMP_Tut.tar.gz

And then go into the OpenMP/ tut directory and start the practical. Have fun!

Most of the prac can be done in an interactive session.

To execute longer running jobs you will need to submit a job script to the scheduler. There is an example PBSPro job script on the school00 home directory called example.sh and which can be copied to your current directory in the usual way:

cp ~/../student00/example.sh .

example.sh:

#!/bin/bash
#PBS -N example01
#PBS -l select=1:ncpus=24:mpiprocs=24
#PBS -P Wchpc
#PBS -q R522145
#PBS -W group_list=training
#PBS -l walltime=4:00:00
 
CWD=/mnt/lustre/users/$USER/example
 
 
 
cd $CWD
echo "Working directory is $CWD"
nproc=`cat $PBS_NODEFILE | wc -l`
echo "nproc is $nproc"
echo "Nodes used by this job:"
cat $PBS_NODEFILE
 
echo "Starting run..."
# Insert command to run here:
 
echo "... done."

Note that the options passed to the qsub command appear as shell comment lines that start with #PBS.

Note: the script uses example/ as the working directory. Create this before you run the script:

mkdir ~/lustre/example

or edit the script file to point to the prac directory.

NOTE: your script will fail if you don't use the Lustre file system.

In the example file the command to actually run your code is missing. Depending on which part of the prac you are attempting, this will differ. Insert the command in the indicated place.

To then submit the job script to the scheduler you simply use

qsub example.sh

The qsub command will read all its option parameters from the file.

To check the status of your jobs in the Winter School queue:

qstat | grep R522145

This command lists all the jobs in the R522145 queue. Running jobs will have a R in the status column, a Q if the job is queued and waiting.

Use the qdel command to remove a queue job from the queue. The synatx is

qdel jobid

where jobid is replaced by the number of the job (looks like 557809.sched01).