There are a number of python versions installed on Lengau compiled with either gcc or the intel compiler, including Python 2.7.11 to 2.7.15, Python 3.5-3.7 and anaconda installs for Python 2 and 3. Use this command to get a full list:

  module avail chpc/python

All python modules should have mpi4py, numpy and scipy installed; however if matplotlib is required, please use anaconda (which also has the biggest range of 3rd-party python modules). The module chpc/python/3.5.2_gcc-6.2.0 also has a working matplotlib installation.

To use tensorflow, the anaconda module for python3 is also recommended.

Because CHPC users don't have permission to modify the python installations, you will not be able to install libraries to the python installation. You can, however, install them to a different location using the –user flag when you run pip pip3 (the latter for python3): pip install –user PySomeProject This can only be done from a node with internet access - for example chpcviz1 or chpclic1 (but not a compute node). By default this installs to the .local directory within your home directory. This can be a problem if you need to install a lot of python packages, as your home directory can only accommodate 15GB.

Modules for anaconda version 2 and 3 are installed on lengau.To load these modules, use the following command (either from the command-line in an interactive session, or in a script:

  module add chpc/python/anaconda/2

or:

  module add chpc/python/anaconda/3-2019.10

Using virtualenv with anaconda is not recommended. However, Anaconda does include a similar functionality, which allows users to create an anaconda virtual environment.

The following instructions to set up your virtual environment.

YOU NEED ONLY DO THEM ONCE. Please replace USERNAME with your actual username wherever it occurs below.

1. Log into lengau.chpc.ac.za

2. ssh chpclic1 Unlike the compute nodes, chpcviz1 and chpclic1 have a connection to the internet, which is needed here.

3. Load the appropriate anaconda module

  module load chpc/python/anaconda/3-2019.10

You can substitute chpc/anaconda/2 if you want Python 2.

4. Do the once-off initialisation for your shell:

  conda init bash

You might have to log out and back in at this point to enable the shell setup.

5. Create a new conda environment:

  conda create --prefix /home/USERNAME/myenv python=3.6 anaconda

Change python=3.6 to python=2.7 if you're using anaconda/2.

6. Activate the environment

  conda activate /home/USERNAME/myenv

You don't have to use “myenv”, just replace it consistently with whatever name you prefer for your virtual environment, myenv will work, though.

7. You can now install whatever packages you need, that are not installed already; for instance, for rpy2, type:

 conda install rpy2

8. you can now type:

conda deactivate /home/USERNAME/myenv

to exit the virtual environment.

Now, once you've done this, you can log out, and ready your job for submission from the login node.

You must do the following in your job-submit script or your .bashrc file in your home directory:

Put it all near the top of your script, but after the #PBS directives:

  module add chpc/python/anaconda/3-2019.10

  conda activate /home/USERNAME/myenv

and at the end:

  conda deactivate /home/USERNAME/myenv

Then submit your script as normal with qsub.

if you ever need to install new Python packages, just repeat steps 1, 2, 3, 6, 7 and 8 from above. (omit 4 and 5).