wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » playground
Trace:
playground:playground

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
playground:playground [2021/02/23 14:37]
msovara [Jupyter / Ipython Notebook] 		playground:playground [2025/04/29 16:06] (current)
nmfuphi [Starting a notebook inside a job]
Line 1: Line 1:
-====== **Playground Test edits by MSovara** ======+====== **Edits by MSovara** ======
  
 ===== Jupyter / Ipython Notebook ===== ===== Jupyter / Ipython Notebook =====
Line 10: Line 10:
 However using it effectively on the cluster is a bit more complicated... However using it effectively on the cluster is a bit more complicated...
  
-After [[CHPC quickstart guide | Logging in]] your account select your preferred python version as follows:+After [[CHPC quickstart guide | Logging in]] your account check for your preferred python version as follows:
  
 <code bash> <code bash>
Line 16: Line 16:
 </code> </code>
  
-... And then proceed to load the preferred python module as follows:  +Now proceed to request for an interactive compute node where a configuration file and security measures will be established. A single core on an interactive node has proven sufficient for this exercise. Request an arbitrary interactive compute node as follows: 
-<code bash> +
-module add chpc/python/3.6.0_gcc-6.3.0 +
-</code> +
-  +
-It is important to use an interactive node for this next step. A single core on an interactive node has proven sufficient to setup your Jupyter Notebook for security measures. Request an arbitrary interactive node as follows: +
  
 <code bash> <code bash>
Line 28: Line 23:
  
 //**Note**://  //**Note**:// 
-  *//"PROJ0101" is an arbitrary project name, kindly use your research project's shortname e.g. //ERTH0859\\ +  -//"PROJ0101" is an arbitrary project name, kindly use your research project's shortname e.g. //ERTH0859\\ 
-  *//Advanced interactive node settings are found here: [[CHPC quickstart guide | Example interactive job request]].//+  -// Record the cnode ID because you will ssh into that particular compute node when setting up your password.// 
 +  -//Advanced interactive compute node settings are found here: [[CHPC quickstart guide | Example interactive job request]].//
  
 +In your interactive compute node, load the preferred python module as follows: 
 +<code bash>
 +module add chpc/python/3.6.0_gcc-6.3.0
 +</code>
 + 
 ==== Security ==== ==== Security ====
  
-First things first -- you are not the only person on the system, so it is important to set up authentication on your notebook so that not everyone gets access to your notebook (and worse -- your data).+You are not the only person on the system, so it is important to set up authentication on your notebook so that not everyone gets access to your notebook (and worse -- your data).
  
 So first one needs a configuration file, this can be done by passing the generate-config parameter to jupyter as follows: So first one needs a configuration file, this can be done by passing the generate-config parameter to jupyter as follows:
Line 39: Line 40:
 <code bash> <code bash>
 [USERNAME@cnode0010 ~]$ jupyter-notebook --generate-config [USERNAME@cnode0010 ~]$ jupyter-notebook --generate-config
-Writing default config to: /home/USERNAME/.jupyter/jupyter_notebook_config.py 
 </code> </code>
 +
 +**Note**
 +  -//This writes default config to: /home/USERNAME/.jupyter/jupyter_notebook_config.py//
 +  -//The output file "jupyter_notebook_config.py" will be listed as a hidden file. Thus, to view it do a $ ls -a //
  
 Next you need to generate your password (remember it -- you'll need it when you connect later): Next you need to generate your password (remember it -- you'll need it when you connect later):
Line 54: Line 58:
 </code> </code>
  
-Now edit the ''jupyter_notebook_config.py'' file, specifically edit the ''c.NotebookApp.passwd'' line (remember to uncomment it, and don't just copy and paste my hash in):+Use the following command to access the "jupyter_notebook_config.py" file: $ cd /home/USERNAME/.jupyter/jupyter_notebook_config.py \\ 
 +Now edit the ''jupyter_notebook_config.py'' file, specifically edit the ''c.NotebookApp.passwd'' line.  
 + 
 <code> <code>
 c.NotebookApp.password = 'sha1:f27008fdb0eb:4c2f305d5e230edca16c7059882ba3ba63bee03b' c.NotebookApp.password = 'sha1:f27008fdb0eb:4c2f305d5e230edca16c7059882ba3ba63bee03b'
 </code> </code>
 +
 +**Remember to uncomment it, don't just copy and paste my hash in**.
 +
 +==== Starting a notebook inside a job ====
 +
 +There might be a cleaner way of doing this... Please let us know if you have one!
 +
 +**VERY IMPORTANT:**  Do not add the lines below to your .ssh/config file on the cluster, you //WILL// break any attempt at parallel processing!
 +
 +Open a terminal on your local machine and create a ''.ssh/config'' file. 
 + 
 +<code>
 +touch .ssh/config
 +</code>
 +
 +If your desktop system runs Windows, a simple way to deal with this is to run a unix-like environment inside Windows.  You can either use[[http://cygwin.com|Cygwin]] Cygwin directly, or start a "Local Terminal" in [[https://mobaxterm.mobatek.net/|MobaXterm]].  From this terminal you can edit the local ''~/.ssh/config'' file as if you were working on a Linux computer. 
 +
 + 
 +Continue to edit the ''~/.ssh/config'' file on your local machine by adding in these lines:
 +<code>
 +Host cnode*
 +    Hostname %h
 +    User YOURUSERNAME
 +    ProxyCommand ssh YOURUSERNAME@lengau.chpc.ac.za nc %h 22
 +    LocalForward 8838 localhost:8838
 +</code>
 +
 +
 +**Note**
 +  -//Each session will be assigned a different compute node (cnode* changes everytime you request for an interactive compute node session)//
 +  -//Use localForward and localhost port number provided for each different session.//
 +
 +Ultimately, what this will do is allow you to ''ssh'' directly to a compute node (note you can only do this to nodes where you currently have a job running). 
 +
 +<code>
 +ssh cnode*
 +</code>
 +
 +You should be prompted for your password, twice. This is because the ''ssh'' logs in to Lengau first and then from Lengau it logs into the the compute node. 
 +
 +You are now ready to roll...
 +
 +In your browser go to: http://localhost:8838
 +
 +The jobscript will look something like:
 +<file bash jupyter.qsub>
 +#!/bin/bash
 +#PBS -P SHORTNAME
 +#PBS -q serial
 +#PBS -l select=1:ncpus=8:mpiprocs=1
 +#PBS -l walltime=08:00:00
 +#PBS -N Jupyter
 +#PBS -m abe
 +#PBS -M YOUR@EMAIL.ADDRESS
 +
 +module add chpc/python/3.6.0_gcc-6.3.0
 +
 +JUPYTERPORT=8838  # you could change this too, if you wanted to.
 +
 +hostname > ~/jupyter.host
 +
 +jupyter-notebook --port=${JUPYTERPORT} --no-browser
 +</file>
 +
 +If you submit that job and wait for it to start running then you can check which host the session is running on with:
 +<code bash>
 +cat ~/jupyter.host
 +</code>
 +
 +Then, again on your local machine, you need to connect to the compute node, i.e. ''ssh cnode0101''.  If you are working in Windows, do this from your Cygwin or MobaXterm terminal command line.  You will be prompted for your Lengau login password.
 +
  
  
  
/app/dokuwiki/data/attic/playground/playground.1614083858.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki