wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » guide » python
Trace:
guide:python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
guide:python [2014/04/22 16:50]
wikiadmin 		guide:python [2022/06/13 11:58] (current)
wikiadmin
Line 1: Line 1:
-====== Python on the Sun Cluster ======+====== Python ======
  
 +There are a number of python versions installed on Lengau compiled with either gcc or the intel compiler, including Python 2.7.11 to 2.7.15, Python 3.5-3.7 and anaconda installs for Python 2 and 3.
 +Use this command to get a full list:
 +    module avail chpc/python
  
-A fairly complete python 2.6.5 installation is available on both the login and compute nodes, at+Note that the modules including anaconda are more recent and thus recommended over the plain Python modules, i.e., use one of:
  
-''/opt/gridware/python2.6.5/bin/python''+  chpc/python/anaconda/3-2019.10 
 +  chpc/python/anaconda/3-2020.02 
 +  chpc/python/anaconda/3-2020.11 
 +  chpc/python/anaconda/3-2021.05 
 +  chpc/python/anaconda/3-2021.11
  
-I recommend you invoke your python scripts by explicitly invoking python as above --- for example:+>  To check the version of Python provided by each, load the module (after purging any already loaded modules) and then run ''python --version''.  For example (screen capture --- don't type in the ''login1:~$'', that's the shell prompt):
  
-  /opt/gridware/python2.6.5/bin/python /export/home/username/bin/myscript.py+  login1:~$ module purge 
 +  login1:~$ module load chpc/python/anaconda/3-2019.10 
 +  login1:~$ python3 --version 
 +  Python 3.7.4 
 +  login1:~$ module purge 
 +  login1:~$ module load chpc/python/anaconda/3-2021.11 
 +  login1:~$ python3 --version 
 +  Python 3.9.7 
 +  login1:~$ 
  
-Modules included with this are :- 
  
-| numpy scipy     | Comprehensive scientific processing for python | 
-| PyNGL           | Scientific visualisation | 
-| tables          | HDF file reader and writer | 
-| mpl_toolkits matplotlib    | Matplotlib --- MATLAB-like plotting functions | 
-| Crypto          | Cryptographic utilities | 
-| OpenSSL         | Secure Sockets Layer | 
-| IPython         | Interactive python, great for debugging | 
-| PyNIO           | NetCDF file reader / writer | 
-| Polygon         | Polygon manipulation routines | 
-| ANN             | kdtree-based Approximate Nearest Neighbour routines | 
-| h5py            | interface to the HDF5 binary data format [see notes below] | 
  
-Most of these modules invoke low-level C and Fortran to do the heavy lifting.+All python modules should have mpi4py, numpy and scipy installed; however if matplotlib is required, please use anaconda (which also has the biggest range of 3rd-party python modules). The module ''chpc/python/3.5.2_gcc-6.2.0'' also has a working matplotlib installation.
  
-====Notes====+To use tensorflow, the anaconda module for python3 is also recommended.
  
-===h5py===+==== User Package Installation via pip3 and virtualenv ==== 
 +//The following will **not** work with the anaconda modules, see below if you want to use those.// 
 + 
 +Because CHPC users don't have permission to modify the python installations, you will not be able to install libraries to the python installation. You can, however, install them to a different location using the --user flag when you run pip install or pip3 install(the latter for python3): 
 +  pip install --user PySomeProject 
 +**This can only be done from a node with internet access - for example chpcviz1 or chpclic1 (but not a compute node).** 
 +Of course, you will also need to replace "PySomeProject" with the name of the package that you want to install. 
 + 
 +By default this installs to the .local directory within your home directory. This can be a problem if you need to install a lot of python packages, as your home directory can only accommodate 15GB. If you think you are likely to run out of space, you should rather specify a directory on lustre, as follows: 
 + 
 +  pip install --install-option="--prefix=/mnt/lustre/users/USERNAME/.local" PySomeProject 
 + 
 +As before, replace USERNAME and PySomeProject with appropriate values. 
 +You can find more information on user installs in the [[https://packaging.python.org/tutorials/installing-packages/#installing-to-the-user-site|official python documentation]] and more information on pip user installs [[https://pip.pypa.io/en/latest/user_guide/#user-installs|here]]. 
 + 
 +As an alternative, you can also set up a python virtual environment. This essentially gives you your own python installation which you can alter as you see fit. Again, it is perhaps best done in your Lustre directory if a large number of python libraries will be installed. 
 + 
 +To create a virtual environment called //myenv//, just cd to the place where you want the virtual environment to live --- for example 
 + 
 +<code bash> 
 +cd /mnt/lustre/users/USERNAME 
 +</code> 
 + 
 +where ''USERNAME'' is replace by __your__ cluster user name --- and run the following command: 
 + 
 + 
 +  python3 -m venv myenv 
 + 
 +You only need to do this once, to set up the virtual environment. 
 + 
 +Then, to activate the virtual environment (which you will need to do whenever you use it - so put it in your submit-script or ~/.bashrc file): 
 + 
 +  source /mnt/lustre/users/USERNAME/myenv/bin/activate 
 + 
 +> Note the full path to the virtual environment is on your Lustre directory: ''/mnt/lustre/users/USERNAME/myenv/'' 
 + 
 +Again, there is more information in [[https://docs.python.org/3/tutorial/venv.html|the official python documentation]]. 
 + 
 +==== User Package Installation with Anaconda === 
 +Modules for anaconda version 2 and 3 are installed on lengau.To load these modules, use the following command (either from the command-line in an interactive session, or in a script: 
 +    module add chpc/python/anaconda/2 
 +or: 
 +    module add chpc/python/anaconda/3-2019.10 
 + 
 +Using virtualenv with anaconda is not recommended. 
 +However, Anaconda does include a similar functionality, which allows users to create an anaconda virtual environment. 
 + 
 +The following instructions to set up your virtual environment.  
 + 
 +**YOU NEED ONLY DO THEM ONCE. Please replace //USERNAME// with your actual username wherever it occurs below.** 
 + 
 +1. Log into lengau.chpc.ac.za 
 + 
 +2. ''ssh chpclic1'' 
 +//Unlike the compute nodes, chpcviz1 and chpclic1 have a connection to the internet, which is needed here.// 
 + 
 +3. Load the appropriate anaconda module  
 +    module load chpc/python/anaconda/3-2019.10 
 + 
 +You can substitute chpc/anaconda/2 if you want Python 2. 
 + 
 +4. Do the  once-off initialisation for your shell:  
 +    conda init bash 
 +You might have to log out and back in at this point to enable the  
 +shell setup. 
 + 
 +5. Create a new conda environment: 
 +    conda create --prefix /home/USERNAME/myenv python=3.6 anaconda 
 +Change ''python=3.6'' to ''python=2.7'' if you're using ''anaconda/2''. 
 + 
 +6. Activate the environment 
 +    conda activate /home/USERNAME/myenv 
 +You don't have to use "myenv", just replace it consistently with whatever name you prefer for your virtual environment, ''myenv'' will work, though. To enable conda environments, add the following to your script before activating the environment. 
 +    eval "$(conda shell.bash hook)"  
 + 
 + 
 +7. You can now install whatever packages you need, that are not installed already; for instance, for rpy2, type: 
 +   conda install rpy2 
 + 
 +8. you can now type: 
 +  conda deactivate /home/USERNAME/myenv 
 +to exit the virtual environment. 
 + 
 + 
 +Now, once you've done this, you can log out, and ready your job for submission from the login node. 
 + 
 +**You must do the following in your job-submit script or your .bashrc file in your home directory**: 
 + 
 +Put it all near the top of your script, but after the ''#PBS'' directives: 
 +<code> 
 +  module add chpc/python/anaconda/3-2019.10 
 + 
 +  conda activate /home/USERNAME/myenv 
 +</code> 
 +and at the end: 
 +<code> 
 +  conda deactivate /home/USERNAME/myenv 
 +</code> 
 +Then submit your script as normal with ''qsub''. 
 + 
 + 
 +if you ever need to install new Python packages, just repeat steps 1, 2, 3, 6, 7 and 8 from above. (omit 4 and 5). 
 + 
 + 
 +===conda bug=== 
 + 
 +There is known bug inflicting some versions of conda.  Use   
 + 
 +  source /apps/chpc/chem/anaconda3-2019.10/etc/profile.d/conda.sh  
 + 
 +instead of ''conda init'' to initialise conda. 
 + 
 +> Replace the above path ''/apps/chpc/chem/anaconda3-2019.10'' with the path to the Anaconda version you are actually using. 
 + 
 +Remember that you can find the paths used by any module using the ''module show'' command: 
 + 
 +    module show chpc/python/anaconda/3-2019.10
  
-The ''h5py'' module uses HDF5 library version 1.8.11, which is installed in ''/opt/gridware/libraries/hdf5/1.8.11/''.  
  
  
/app/dokuwiki/data/attic/guide/python.1398178228.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki