wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » tipsntricks » ipython_notebook
Trace:
tipsntricks:ipython_notebook

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
tipsntricks:ipython_notebook [2021/03/11 13:56]
ccrosby [Security] 		tipsntricks:ipython_notebook [2024/06/21 16:01] (current)
wikiadmin [Jupyter / Ipython Notebook]
Line 1: Line 1:
 ===== Jupyter / Ipython Notebook ===== ===== Jupyter / Ipython Notebook =====
  
-Jupyter is installed in most of the python modules. To make use of it you would do something like:+Jupyter is a commonly installed component in most Python modules, and it can be easily utilized by following a few simple steps. For instance:
 <code bash> <code bash>
 jupyter-notebook jupyter-notebook
 </code> </code>
  
-However using it effectively on the cluster is a bit more complicated...+However, using it effectively on the cluster is a bit more complicated...
  
-After [[CHPC quickstart guide | Logging in]] your account check for your preferred python version as follows:+After [[http://wiki.chpc.ac.za/quick:start#logging_in|Logging in]] to your account check for your preferred python version as follows:
  
 <code bash> <code bash>
Line 14: Line 14:
 </code> </code>
  
-Now proceed to request for an interactive compute node where a configuration file and security measures will be established. A single core on an interactive node has proven sufficient for this exercise. Request an arbitrary interactive compute node as follows: +You can now request an interactive compute node to establish a configuration file and implement security measures. For this exercise, a single core on an interactive node has been found to be sufficient. However, if you plan to open multiple notebooks, please request an appropriate number of cores, such as four, using the following command:
  
 <code bash> <code bash>
-qsub -I -P PROJ0101 -q serial -l select=1:ncpus=1:mpiprocs=1:nodetype=haswell_reg+qsub -I -P PROJ0101 -q serial -l select=1:ncpus=4:mpiprocs=4
 </code> </code>
  
 //**Note**://  //**Note**:// 
-  -//"PROJ0101" is an arbitrary project name, kindly use your research project's shortname e.g. //ERTH0859\\ +  -// Kindly replace "PROJ0101" with your research programme's short name e.g. //ERTH0859\\ 
-  -// Record the cnode ID because you will ssh into that particular compute node when setting up your password.// +  -// Make sure to record the cnode ID as you will need to ssh into that specific compute node when setting up your password..// 
-  -//Advanced interactive compute node settings are found here: [[CHPC quickstart guide | Example interactive job request]].//+  -//Advanced interactive compute node settings are found here: [[http://wiki.chpc.ac.za/howto:tipsandtricks#using_an_interactive_pbs_session | Example interactive job request]].//
  
-In your interactive compute node, load the preferred python module as follows: +Certain users may require nodes with more memory in order to post-process large data sets.  You may use a "fat" node for this purpose.  These nodes have 1 TB of memory and 56 cores each.  It does not make sense to request an entire fat node and all its memory for a Jupyter task, so it is best to share such a node with other users.  If you do not already have access to the bigmem queue, you will need to request this from the [[https://users.chpc.ac.za/helpdesk|CHPC Helpdesk]]   
 + 
 +<code bash> 
 +qsub -I -P PROJ0101 -q bigmem -l select=1:ncpus=4:mpiprocs=4 
 +</code> 
 + 
 +In your interactive node, load the preferred python module as follows: 
 <code bash> <code bash>
 module add chpc/python/3.6.0_gcc-6.3.0 module add chpc/python/3.6.0_gcc-6.3.0
Line 32: Line 38:
 ==== Security ==== ==== Security ====
  
-You are not the only person on the system, so it is important to set up authentication on your notebook so that not everyone gets access to your notebook (and worse -- your data).+You are not the only person on the system, so it is important to set up authentication on your notebook so that not everyone gets access to your notebook (and worse -- your data).  
  
 So first one needs a configuration file, this can be done by passing the generate-config parameter to jupyter as follows: So first one needs a configuration file, this can be done by passing the generate-config parameter to jupyter as follows:
Line 41: Line 47:
  
 **Note** **Note**
-  -//This writes default config to: /home/USERNAME/.jupyter/jupyter_notebook_config.py// +  -//This writes a default config file to: /home/USERNAME/.jupyter/jupyter_notebook_config.py//, where USERNAME is YOUR username. 
-  -//The output file "jupyter_notebook_config.py" will be listed as a hidden file. Thus, to view it do a $ ls -a //+  -//The output file "jupyter_notebook_config.py" will be listed as a hidden file. To display it, execute the command $ ls -a. //
  
 Next you need to generate your password (remember it -- you'll need it when you connect later): Next you need to generate your password (remember it -- you'll need it when you connect later):
Line 56: Line 62:
 </code> </code>
  
-Use the following command to access the "jupyter_notebook_config.py" file: $ cd /home/$USERNAME/.jupyter/ \\ +Your password hash will be different.  Obviously use the one in your terminal, not the one shown in this example.
-Now edit the ''jupyter_notebook_config.py'' file, specifically edit the ''c.NotebookApp.passwd'' line. +
  
  
 +Exit python and then use the following command to access the "jupyter_notebook_config.py" file:
 +<code bash>
 +  cd .jupyter/
 +</code>
 +
 +Now edit the file ''jupyter_notebook_config.py'' with your favourite editor. 
 <code> <code>
 +vim jupyter_notebook_config.py 
 c.NotebookApp.password = 'sha1:f27008fdb0eb:4c2f305d5e230edca16c7059882ba3ba63bee03b' c.NotebookApp.password = 'sha1:f27008fdb0eb:4c2f305d5e230edca16c7059882ba3ba63bee03b'
 </code> </code>
  
-**Remember to uncomment it, don't just copy and paste my hash in**.+**Remember to uncomment it, and copy and paste your own hash in**.
  
 ==== Starting a notebook inside a job ==== ==== Starting a notebook inside a job ====
Line 72: Line 84:
 **VERY IMPORTANT:**  Do not add the lines below to your .ssh/config file on the cluster, you //WILL// break any attempt at parallel processing! **VERY IMPORTANT:**  Do not add the lines below to your .ssh/config file on the cluster, you //WILL// break any attempt at parallel processing!
  
-Open a terminal on your local machine and create a ''.ssh/config'' file. +Open a terminal on your **local** workstation and create a ''.ssh/config'' file. 
    
 <code> <code>
Line 78: Line 90:
 </code> </code>
  
-If your desktop system runs Windows, a simple way to deal with this is to run a unix-like environment inside Windows.  You can either use[[http://cygwin.com|Cygwin]] Cygwin directly, or start a "Local Terminal" in [[https://mobaxterm.mobatek.net/|MobaXterm]].  From this terminal you can edit the local ''~/.ssh/config'' file as if you were working on a Linux computer. +If your desktop system runs Windows, a simple way to deal with this is to run a unix-like environment inside Windows.  You can either use[[http://cygwin.com|Cygwin]] Cygwin directly, or start a "Local Terminal" in [[https://mobaxterm.mobatek.net/|MobaXterm]].  From this terminal you can edit the local ''.ssh/config'' file as if you were working on a Linux computer. 
  
    
-Continue to edit the ''~/.ssh/config'' file on your local machine by adding in these lines:+Find,  
 +<code> 
 +ls -al 
 +</code> 
 +... and continue to edit the ''.ssh/config'' file on your local machine by adding in these lines:
 <code> <code>
 Host cnode* Host cnode*
Line 87: Line 103:
     User YOURUSERNAME     User YOURUSERNAME
     ProxyCommand ssh YOURUSERNAME@lengau.chpc.ac.za nc %h 22     ProxyCommand ssh YOURUSERNAME@lengau.chpc.ac.za nc %h 22
-    LocalForward 8838 localhost:8838+    LocalForward 8888 localhost:8888 
 +Host fat* 
 +    Hostname %h 
 +    User YOURUSERNAME 
 +    ProxyCommand ssh YOURUSERNAME@lengau.chpc.ac.za nc %h 22 
 +    LocalForward 8888 localhost:8888    
 </code> </code>
 +
 +
 +Remember to replace "YOURUSERNAME" with your own username on the cluster.
  
 At this point you may not know what the LocalForward and localhost port numbers are so, on the cluster in the interactive node you had opened earlier type:  At this point you may not know what the LocalForward and localhost port numbers are so, on the cluster in the interactive node you had opened earlier type: 
Line 94: Line 118:
 jupyter-notebook --no-browser jupyter-notebook --no-browser
 </code> </code>
-Edit given port number into LocalForward and localhost above+Edit the given port number into LocalForward and localhost above
  
 **Note** **Note**
Line 100: Line 124:
   -//Confirm the localForward and localhost port numbers provided for each different session.//   -//Confirm the localForward and localhost port numbers provided for each different session.//
  
-In your local terminal ''ssh'' directly to a compute node +In your local terminal ''ssh'' directly to the compute node that you are using.  Let us assume that it is cnode1234: 
  
 <code> <code>
-ssh cnode*+ssh cnode1234
 </code> </code>
  
-You should be prompted for your password, twice. This is because the ''ssh'' logs in to Lengau first and then from Lengau it logs into the the compute node. +You should be prompted for your CLUSTER password, twice. This is because the ''ssh'' logs in to Lengau first and then from Lengau it logs into the the compute node.  On the first login, MobaXterm will offer to remember your password, and if you allow it to do so it will not be necessary to re-enter it.
  
 You are now ready to roll... You are now ready to roll...
  
-In your browser go to: http://localhost:8838 (note you can only do this to nodes where you currently have a job running). +In your browser go to: http://localhost:8888 (note you can only do this for nodes where you currently have a jupyter job running). Your port number may be different.  Use the number that you have been assigned.
  
 The jobscript will look something like: The jobscript will look something like:
 <file bash jupyter.qsub> <file bash jupyter.qsub>
 #!/bin/bash #!/bin/bash
-#PBS -P SHORTNAME+#PBS -P ERTH1234 
 #PBS -q serial #PBS -q serial
-#PBS -l select=1:ncpus=8:mpiprocs=1+#PBS -l select=1:ncpus=8:mpiprocs=8
 #PBS -l walltime=08:00:00 #PBS -l walltime=08:00:00
 #PBS -N Jupyter #PBS -N Jupyter
Line 125: Line 149:
 module add chpc/python/3.6.0_gcc-6.3.0 module add chpc/python/3.6.0_gcc-6.3.0
  
-JUPYTERPORT=8838  # you could change this too, if you wanted to.+JUPYTERPORT=8888  # you could change this too, if you wanted to.
  
 hostname > ~/jupyter.host hostname > ~/jupyter.host
Line 137: Line 161:
 </code> </code>
  
-Then, again on your local machine, you need to connect to the compute node, i.e. ''ssh cnode0101''.  If you are working in Windows, do this from your Cygwin or MobaXterm terminal command line.  You will be prompted for your Lengau login password.+Then, again on your local machine, you need to connect to the allocated compute node, for example ''ssh cnode0101''.  If you are working in Windows, do this from your Cygwin or MobaXterm terminal command line.  You will be prompted for your Lengau login password.
  
 ==== Running Jupyter on a GPU node ==== ==== Running Jupyter on a GPU node ====
/app/dokuwiki/data/attic/tipsntricks/ipython_notebook.1615463794.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki