Differences

This shows you the differences between two versions of the page.

--- howto:ultrafluidx [2022/03/29 14:03]
ccrosby [Example PBS scripts]
+++ howto:ultrafluidx [2022/03/29 16:23] (current)
ccrosby [Example PBS scripts]
@@ Line 27: / Line 27: @@
   * To view the output file uFX_log_2022-03-28_08-27-50.out <code> less uFX_log_2022-03-28_08-27-50.out </code>  Scroll down with the arrow keys or spacebar, exit with the ''q'' key.
   * To monitor the progress of the job that is writing the output file uFX_log_2022-03-28_08-27-50.out <code> tail -f uFX_log_2022-03-28_08-27-50.out </code>  To exit press CTRL-C.
-  * To view GPU usage, get the hostname of the node being used, then ssh to it.  This is only possible as long as you have a job running on it.  Then run the command <code> /apps/chpc/compmech/nvtop/bin/nvtop </code>  To exit press the ''q'' key.  Only three of the nodes have the GPUs connect with NVlink, which is a much faster and lower latency interconnect than PCIe.
+  * To view GPU usage, get the hostname of the node being used, then ssh to it.  This is only possible as long as you have a job running on it.  Then run the command <code> /apps/chpc/compmech/nvtop/bin/nvtop </code>  To exit press the ''q'' key.
+=== Relationship between the number of MPI ranks and the number of GPUs ===
+UltraFluidX requires one more MPI rank than the number of GPUs.  Please refer to this table:
+^ Number of GPUs ^ ncpus ^ mpiprocs ^ ngpus ^ queue ^ Command line  ^
+|  1  |  2  |  2  |  1  | gpu_1 | ufx -np 2 -inpFile .... |
+|  2  |  3  |  3  |  2  | gpu_2 | ufx -np 3 -inpFile .... |
+|  3  |  4  |  4  |  3  | gpu_3 | ufx -np 4 -inpFile .... |
+|  4  |  5  |  5  |  4  | gpu_4 | ufx -np 5 -inpFile .... |
@@ Line 61: / Line 73: @@
 === Performance considerations ===
-Running on a single V100 GPU card produces a performance of around 400 MNUPS.  Running on 2 cards can deliver up to 690 and on 3 cards around 1050 MNUPS is possible.  The current configuration of the scheduler does not make it straightforward to run over multiple compute nodes.  Until further notice, do not attempt to use more than one node at a time, but it is worthwhile to use multiple cards if the model size justifies it.  However, there is an important factor that **must** be considered.  Please refer to the configuration of the GPU cluster as documented [[https://wiki.chpc.ac.za/guide:gpu#gpu_nodes|here]].
+Running on a single V100 GPU card produces a performance of around 400 MNUPS.  Running on 2 cards can deliver up to 690 and on 3 cards around 1050 MNUPS is possible.  The current configuration of the scheduler does not make it straightforward to run over multiple compute nodes.  Until further notice, do not attempt to use more than one node at a time, but it is worthwhile to use multiple cards if the model size justifies it.  However, there is an important factor that **must** be considered.  Please refer to the configuration of the GPU cluster as documented [[https://wiki.chpc.ac.za/guide:gpu#gpu_nodes|here]].  Only three of the nodes have the GPUs connected with the high speed low-latency NVlink fabric.  In the remaining nodes the GPUs are connected with the PCIe system.  For an application like UltraFluidX it appears to be essential to use NVlink on multi-GPU runs.  Running with more than one GPU on the PCIe system only results in around 310 MNUPS, which is slower than using a single GPU.  In order to obtain a full 4-GPU node with NVlink, the following type of submission can be used:
+<code> qsub -l select=1:ncpus=5:mpiprocs=5:ngpus=4 -q gpu_4 -P MECH1234 -l walltime=4:00:00 </code>
+It gets more difficult if only two or three NVlinked GPUs are required.  It is then necessary to nominate a particular NVlink-equipped compute node:
+<code> qsub -l select=1:ncpus=3:mpiprocs=3:ngpus=2:host=gpu4001 -q gpu_4 -P MECH1234 -l walltime=4:00:00 </code>
+The scheduler is not currently configured in such a way that it is possible to select **any** available one of the three NVlink-equipped nodes.  This may change in future.

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools