The visualization servers are intended for single-process pre- and post-processing only, as well as GUI monitoring of running jobs. These servers are NOT intended for parallel processing or running compute tasks. The system administrators will terminate processes that do not fit the above description without warning. However, it is now possible to get a Virtual Desktop on a compute node, where you can use the full capabilities of the compute node without restriction.
There is a dedicated visualization server called chpcviz1, and an identical backup server called chpclic1. Please note that these servers should be used for pre- and post-processing only. If the servers are being overloaded, we reserve the right to kill your processes without warning. These servers mount the Lustre and NFS file systems, have 64 GB of RAM and 2 Intel Xeon E5-2640 processors, each with 6 processor cores, as well as NVidia Tesla K20m GPU cards. For security reasons, these servers do not have IP addresses visible from outside the CHPC. However, they can be accessed via the login node by means of SSH tunneling. Although it is possible to perform remote visualization by means of X-forwarding (log in with “ssh -X”), this approach is generally too slow to be of use if the user is not on the internal network. It is therefore preferable to use a remote desktop. However, standard VNC is too slow for this and does not work properly with OpenGL, the library mostly used for 3D graphics. The visualization server has been set up with TurboVNC and VirtualGL, which gets around both of these problems. Use the following process for remote visualization:
There is a default VNC server installed on chpcviz. Do not use it. Use TurboVNC, which can be started with a command like this:
/apps/chpc/compmech/TurboVNC-2.2.3/bin/vncserver :3 -geometry 1920×1080 -depth 24
TurboVNC by default sets up a startup file that selects the lightweight window manager Fluxbox. If you are running TurboVNC on chpcviz1 or chpclic1 you can also use the friendlier window managers Mate or XFCE4.
Your xstartup.turbovnc file should look like this:
#!/bin/sh unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS # Uncomment out the next line if you to use the mate desktop exec /bin/mate-session # Uncomment the next line if you want to use the xfce4 desktop #exec /bin/xfce4-session # Uncomment the next two lines if you want to use the very lightweight window manager Fluxbox. #export PATH=/apps/chpc/compmech/fluxbox-1.3.7/bin:$PATH #exec /apps/chpc/compmech/fluxbox-1.3.7/bin/startfluxbox
More than one VNC session can be run simultaneously. In this example, two other sessions are already running, therefore :3 is used to specify that this will be a session on virtual display 3. If this is unavailable, try :4, etc. Optionally specify an appropriate display resolution and colour depth, as in the above example. If you get a warning of unsupported resolution, try one of the standard resolutions. The port number used for a VNC session is 5900+n , where n is the number of the display. In this example, the VNC session will be served on port 5903.
The vncserver will continue running even after logging out. If you are no longer going to use it, please kill the server as follows:
/apps/chpc/compmech/TurboVNC-2.2.3/bin/vncserver -kill :3
where :3 should be changed to whichever display the server has been running on.
You cannot log into chpcviz1 directly from outside CHPC's network. However, it is easy to set up an ssh tunnel to it. There are a number of ways to set up such a tunnel:
This is an example of the command:
ssh -f firstname.lastname@example.org -L 5903:chpcviz1:5903 -N
Any VNC client can be used to connect to the TurboVNC server on chpcviz1. However, to take full advantage of the higher speed and configuration options of TurboVNC, use the TurboVNC client as well. It can be downloaded from http://sourceforge.net/projects/virtualgl/files/TurboVNC/ . The Windows installer includes a customized version of PuTTY. Once TurboVNC has been installed, run the PuTTY in the TurboVNC installation directory. In the left pane, expand the SSH option, and click on Tunnels. Add the port number for your local host in the source port box, and chpcviz1:5903 (use the port number that your VNC server is using) in the destination box.
Now click on “Add”.
Log in with your usual user-id and password.
If you haven't done so already, install the TurboVNC client, either from http://sourceforge.net/projects/virtualgl/files/TurboVNC/ or from a repository for your version of Linux. Start the TurboVNC Viewer client, and specify localhost::5903 (or whichever local port number you have selected) as the VNC server. The documentation recommends using double colons and the full 590* number, but this also works:
Click on connect and log in with the VNC password you provided when starting the VNC server for the first time.
You should now get a remote desktop:
Clicking on the top left corner will open an “Options” menu:
Experiment with the various settings. You can trade off quality for speed. On a slow connection, use fast low quality settings to set up the scene, then request a “Lossless refresh” to get a high quality image.
3D programs (Paraview, for example) mostly use OpenGL. In a normal VNC session, OpenGL is most likely to throw an error, or at best run with software rendering. In order to take advantage of the graphics processing hardware on the server, it is necessary to run OpenGL programs with the VirtualGL “wrapper”. For example:
/opt/VirtualGL/bin/vglrun /apps/chpc/compmech/CFD/Paraview-4.3.1-Linux-64bit/bin/paraview &
will run a recent version of Paraview. If you have started up your X-windows session with the vglrun wrapper as per the first time instructions given above, it is not necessary to use it when starting the OpenGL application. The following $HOME/.vnc/xstartup.turbovnc file makes life easier.
#!/bin/sh unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS # Uncomment out the next line if you to use the mate desktop exec /opt/VirtualGL/bin/vglrun /bin/mate-session # Uncomment the next line if you want to use the xfce4 desktop #exec /opt/VirtualGL/bin/vglrun /bin/xfce4-session # Uncomment the next two lines if you want to use the very lightweight window manager Fluxbox. #export PATH=/apps/chpc/compmech/fluxbox-1.3.7/bin:$PATH #exec /opt/VirtualGL/bin/vglrun /apps/chpc/compmech/fluxbox-1.3.7/bin/startfluxbox
/apps/chpc/compmech/TurboVNC-2.2.3/bin/vncserver -kill :3
will shut down the VNC server for display 3, freeing up resources for other users.
It is now possible to run a VNC session directly on a compute node. The advantage of this is that far more compute power is available, as the standard compute nodes have 24 compute cores and either 128 GB or 64 GB of RAM. Even a 56-core, 1 TB fat node can be used this way, if your use case justifies it. The compute cores support AVX instructions, and can consequently render 3D graphics very effectively. The process is very similar to what is described above, with some important differences.
qsub -X -I -l select=1:ncpus=24:mpiprocs=24:mem=120gb -l walltime=8:00:00 -q smp -P MECH1234
will get you a full compute node for 8 hours.
Make sure that you have a suitable $HOME/.vnc/xstartup.turbovnc file:
#!/bin/sh unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS export PATH=/apps/chpc/compmech/fluxbox-1.3.7/bin:$PATH exec /apps/chpc/compmech/fluxbox-1.3.7/bin/startfluxbox
This file loads up the minimalist window manager Fluxbox, rather than a full desktop. Fluxbox also works well on the standard visualization nodes, but the full desktop is not available on compute nodes, hence the requirement for a very lightweight window manager.
Now launch the VNC server:
/apps/chpc/compmech/TurboVNC-2.2.3/bin/vncserver :1 -depth 24 -geometry 1800x1000
Use a resolution that fits your display. Make note of the compute node hostname.
The process described above has an important vulnerability. If you lose the interactive ssh session because of a network glitch, you will also lose the VNC session. A better approach is to launch the VNC session from a normal PBS job script, which could look like this:
#!/bin/bash #PBS -l select=1:ncpus=4 #PBS -q serial #PBS -l walltime=2:00:00 #PBS -P MECH1234 #PBS -o $HOME/vncsession.out #PBS -e $HOME/vncsession.err #PBS -m abe #PBS -M email@example.com /apps/chpc/compmech/TurboVNC-2.2.3/bin/vncserver :1 -depth 24 -geometry 1600x900 sleep 2h /apps/chpc/compmech/TurboVNC-2.2.3/bin/vncserver -kill :1
Obviously customise this script to suit your own circumstances. The above script will provide 4 cores for two hours. DO NOT use more than 4 cores in the VNC session. If you intend using a full compute node, use the smp queue and request 24 cores. If you need more than 2 hours, specify the walltime as well as the sleep time accordingly. Get the compute node hostname by querying PBS:
qstat -awu username will give you the jobnumbers of all your jobs (please substitute “username” with your OWN username).
qstat -n1 jobnumber will give you the hostname of the compute node(s) that you are using for that job.
This is identical to the process as described above for a visualization node, except that you now tunnel to the compute node that you have been allocated. If you are using Windows, you can set up the tunnel with PuTTY or MobaXterm, and from Linux, use the command line as in this example:
ssh -f firstname.lastname@example.org -L 5901:cnode1234:5901 -N
As usual, use your username and compute node, not jblogs and cnode1234!
Now, as before, fire up your VNC client, preferably TurboVNC, and connect to port 5901 (or 1 in VNC shorthand) on localhost. Once connected, you will be confronted with a blank black screen. Click your right mouse button to pop up a small menu, and select xterm, which will give you a small terminal. You can resize the window by holding down the alt key and the right mouse button, and dragging the mouse. Refer to the Fluxbox homepage if you need help with other operations.
Up to this point, the process has been basically identical to using a visualization node. However, the compute nodes do not support VirtualGL, as they do not have dedicated graphics cards. All is not lost however:
export GALLIUM_DRIVER=llvmpipe. The advantage of the LLVMpipe implementation is that it supports a more recent version of OpenGL.
/apps/chpc/compmech/CFD/ParaView-4.3.1-Linux-64bit/bin/paraviewwork very well with the Mesa-19.1.2 implementation. More recent ParaView versions are built with –mesa-swr and –mesa-llvm support. Although these work well with X-forwarding, they don't work in the VNC environment. We are not sure why, but we are working on it.
-mesa -rr -rrthreads N, where N is the number of cores that should be used for graphics rendering. In this implementation, you can use up to 16 threads.
Historically, Mesa software rendering has been a way of getting OpenGL-enabled software to work, albeit very slowly, on hardware without dedicated graphics-processing capabilities. However, the OpenSWR framework makes full use of the sse and avx capabilities of modern CPUs to produce good rendering performance. Recent versions of the alternative LLVMpipe implementation have similar performance, and the
apps/chpc/compmech/mesa/19.1.2_swr module defaults to LLVMpipe, because it supports more recent OpenGL features.