Differences

This shows you the differences between two versions of the page.

--- guide:tensorflow [2020/11/03 13:16]
kevin
+++ guide:tensorflow [2025/04/23 14:19] (current)
kevin
@@ Line 3: / Line 3: @@
 If you are making use of Jupyter notebook to write your python scripts then you first need to make sure you export the .py file from Jupyter and then copy it onto the cluster
-Also ensure job is copied to /mnt/lustre/users/YOURUSERNAME or subdirectories therein
+Also ensure job is copied to ''/mnt/lustre/users/YOURUSERNAME'' or subdirectories thereunder.
 ===== Running Tensorflow on CPU nodes =====
@@ Line 16: / Line 16: @@
      module load chpc/python/3.6.1_gcc-6.3.0
-Then
+Then ''cd /mnt/lustre3p/users/YOURUSERNAME'' or where ever you placed your ''.py'' file
-     cd /mnt/lustre/users/YOURUSERNAME or where ever you placed your .py file
 Finally run
@@ Line 32: / Line 30: @@
      import matplotlib
      matplotlib.use('agg')
 ===== Running Tensorflow on GPU nodes =====
 As with CPU version you can test your python jobs on an interactive node:
-     qsub -I -P YOURPROGRAMME(E.G. CSCI1234) -q gpu_1 -l select=1:ncpus=10:ngpus=1
+     qsub -I -P YOURPROGRAMME(E.G. CSCI1234) -q gpu_1 -l select=1:ncpus=9:ngpus=1
 Once on an interactive node (gpuNNNN) you need to load up appropriate modules:
      module purge
-     module load chpc/python/anaconda/3-2019.03
+     module load chpc/python/anaconda/3-2021.05
-Then
+To test that Python (and Tensorflow) sees the GPUs:
-     cd /mnt/lustre/users/YOURUSERNAME or where ever you placed your .py file
+<code>
+module purge
+module load chpc/python/anaconda/3-2021.05
+module load chpc/cuda/11.2/PCIe/11.2
+python3
+import tensorflow as tf
+print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
+</code>
+> From the [[https://www.tensorflow.org/guide/gpu|Tensorflow Guide]].
+The output will look something like this:
+<code>
+-11-03 18:17:58.640547: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
+-11-03 18:17:58.654899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
+name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
+pciBusID: 0000:3b:00.0
+-11-03 18:17:58.656022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
+name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
+pciBusID: 0000:af:00.0
+-11-03 18:17:58.657201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 2 with properties:
+name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
+pciBusID: 0000:d8:00.0
+-11-03 18:17:58.658593: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
+-11-03 18:17:58.690365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
+-11-03 18:17:58.718120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
+-11-03 18:17:58.762931: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
+-11-03 18:17:58.795721: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
+-11-03 18:17:58.824856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
+-11-03 18:17:58.870878: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
+-11-03 18:17:58.877470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1, 2
+Num GPUs Available:  3
+>>>
+</code>
+> Type ''exit()'' to quit the interactive Python session and return to the shell.
+Now you can try your Python code:
+     cd /mnt/lustre3p/users/YOURUSERNAME or where ever you placed your .py file
 When running on a single GPU you need to include the following in your .py file to ensure that not all the CPU's on the node get consumed, thereby resulting in your job being killed by the scheduler
@@ Line 59: / Line 98: @@
 Once you are on the login node just type:
+**module purge**
+**module load chpc/easy_scripts**
 **qtensorflow_cpu** (CPU version of Tensorflow)
@@ Line 96: / Line 139: @@
 Please take note of empty space in EXAMPLE2. This corresponds to the enter key
+===== Updating Tensorflow on GPU nodes =====
+First get onto a GPU node:
+    qsub -I -P YOURPROGRAMME(E.G. CSCI1234) -q gpu_1 -l select=1:ncpus=10:ngpus=1
+To update tensorflow you should use a conda environment
+    module purge
+    module load chpc/python/anaconda/3-2021.05
+Create your environment
+    conda create -n tf-gpu tensorflow-gpu python=3.8
+    conda activate tf-gpu
+The above only needs to be done once thereafter all you will need to do is:
+    module purge
+    module load chpc/python/anaconda/3-2021.05
+    conda activate tf-gpu
+in order to use your updated version of tensorflow

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools