wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » research » ml
Trace: • chemistry • cosmology
research:ml

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
research:ml [2019/07/08 15:12]
kgovender [Running Tensorflow on GPU nodes] 		research:ml [2021/12/09 16:42] (current)
Line 1: Line 1:
-If you are making use of Jupyter notebook to write your python scripts then you first need to make sure you export the .py file from Jupyter and then copy it onto the cluster 
  
-Also ensure job is copied to /mnt/lustre/users/YOURUSERNAME or subdirectories therein+If you are looking for information on Tensorflow, please go to [[guide:tensorflow|TensorFlow Guide]].
  
-===== Running Tensorflow on CPU nodes ===== +======Machine Learning======
- +
-To test a job on a compute node first get onto an interactive node with the following: +
- +
-     qsub -I -P YOURPROGRAMME(E.G. CSCI1234) -q smp -l select=1:ncpus=24 +
- +
-Once on an interactive node (cnodeNNNN) you need to load up the appropriate modules: +
- +
-     module purge +
-     module load chpc/python/3.6.1_gcc-6.3.0 +
- +
-Then  +
- +
-     cd /mnt/lustre/users/YOURUSERNAME or where ever you placed your .py file +
- +
-Finally run +
- +
-     python nameofyourfile.py +
- +
-If you import matplotlib in your python script you may end up with the following error: +
- +
-     ModuleNotFoundError: No module named '_tkinter' +
- +
-If so then add the following to your python script before resubmitting +
- +
-     import matplotlib +
-     matplotlib.use('agg') +
-===== Running Tensorflow on GPU nodes ===== +
- +
-As with CPU version you can test your python jobs on an interactive node: +
- +
-     qsub -I -P YOURPROGRAMME(E.G. CSCI1234) -q gpu_1 -l select=1:ncpus=10:ngpus=1 +
- +
-Once on an interactive node (gpuNNNN) you need to load up appropriate modules: +
- +
-     module purge +
-     module load chpc/cuda/10.0 +
-     module load chpc/python/anaconda/3 +
- +
-Then +
- +
-     cd /mnt/lustre/users/YOURUSERNAME or where ever you placed your .py file +
- +
-When running on a single GPU you need to include the following in your .py file to ensure that not all the CPU's on the node get consumed, thereby resulting in your job being killed by the scheduler +
- +
-      session_conf = tf.ConfigProto(intra_op_parallelism_threads=10,inter_op_parallelism_threads=10) +
-      sess = tf.Session(config=session_conf)  +
- +
-Finally run +
- +
-     python nameofyourfile.py +
- +
-If you wish to run jobs through the scheduler then there are scripts on the login node that can help you setup a PBS submission script +
- +
-Once you are on the login node just run **qtensorflow_cpu** or **qtensorflow_gpu** +
- +
-Examples of what is needed when running the above scripts are provided below: +
- +
-            EXAMPLE1 +
-   Enter research programme name +
-   CSCI1234 +
-   Enter python script name (with .py extension) +
-   test.py +
-   Enter total walltime (hour:minute) +
-   2:00 +
-   Enter email address +
-   testing@gmail.com +
-   Generated pbs file for test +
-   Do you wish to submit job to cluster (y/n) +
-   y +
- +
-            EXAMPLE2 +
-     PLEASE TAKE NOTE OF EMPTY SPACES +
-     THESE CORRESPOND TO THE ENTER KEY +
-   Enter research programme name +
-   CSCI1234 +
-   Enter python script name (with .py extension) +
-   test.py +
-   Enter total walltime (hour:minute) +
-    +
-   Enter email address +
-   testing@gmail.com +
-   Generated pbs file for test +
-   Do you wish to submit job to cluster (y/n) +
-   y+
  
 +The CHPC supports Machine Learning (ML) activities at the CHPC.   More than a just a simulation of AI, machine learning is now applied to modelling areas of science where traditional mathematical models struggle with the complexity and volume of data.
  
  
/app/dokuwiki/data/attic/research/ml.1562591568.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki