User Tools

Site Tools


Jupyter / Ipython Notebook

Jupyter is installed in most of the python modules. To make use of it you would do something like:

module add chpc/python/3.6.0_gcc-6.3.0

However using it effectively on the cluster is a bit more complicated…


First things first – you are not the only person on the system, so it is important to set up authentication on your notebook so that not everyone gets access to your notebook (and worse – your data).

So first one needs a configuration file, this can be done by passing the generate-config parameter to jupyter as follows:

[USERNAME@cnode0010 ~]$ jupyter-notebook --generate-config
Writing default config to: /home/USERNAME/.jupyter/

Next you need to generate your password (remember it – you'll need it when you connect later):

from notebook.auth import passwd
Enter password:
Verify password:

Now edit the file, specifically edit the c.NotebookApp.passwd line (remember to uncomment it, and don't just copy and paste my hash in):

c.NotebookApp.password = 'sha1:f27008fdb0eb:4c2f305d5e230edca16c7059882ba3ba63bee03b'

Starting a notebook inside a job

There might be a cleaner way of doing this… Please let me know if you have one!

Read here first. Do not add the lines below to your .ssh/config file on the cluster, you will break any attempt at parallel processing!

First you need to edit your ~/.ssh/config file on your local machine and add in the lines:

Host cnode*
    Hostname %h
    ProxyCommand ssh nc %h 22
    LocalForward 8838 localhost:8838

If your desktop system runs Windows, a simple way to deal with this is to run a unix-like environment inside Windows. You can either useCygwin Cygwin directly, or start a “Local Terminal” in MobaXterm. From this terminal you can edit the local ~/.ssh/config file as if you were working on a Linux computer.

What this will do is allow you to ssh directly to a compute node (note you can only do this to nodes where you currently have a job running).

The jobscript looks something like:

#PBS -q serial
#PBS -l select=1:ncpus=8:mpiprocs=1
#PBS -l walltime=08:00:00
#PBS -N Jupyter
#PBS -m abe
module add chpc/python/3.6.0_gcc-6.3.0
JUPYTERPORT=8838  # you could change this too, if you wanted to.
hostname > ~/
jupyter-notebook --port=${JUPYTERPORT} --no-browser

If you submit that job and wait for it to start running then you can check which host the session is running on with:

cat ~/

Then, again on your local machine, you need to connect to the compute node, i.e. ssh cnode0101. If you are working in Windows, do this from your Cygwin or MobaXterm terminal command line. You will be prompted for your Lengau login password.

Finally in your browser go to: http://localhost:8838

Starting a parallel session

/var/www/wiki/data/pages/tipsntricks/ipython_notebook.txt · Last modified: 2020/07/02 15:52 by ccrosby