wiki.chpc.ac.za

Some general background information on software licensing is provided here because it can be tricky to get access to off-site licenses right and it helps to understand some of the general principles and tools that can be used to debug the process.

Use of commercial software products is most often controlled by means of a software license. There are different types of software licensing schemes, with FlexNet the most commonly cursed used provider. Network software licenses tend to have the following features in common:

• A license daemon runs on a license server and makes a license available at a known port number. This port number is always greater than 1024 and is unprivileged, in order to prevent attacks by way of the license port.
• Some licensing systems use a single license daemon and port, most use two and in very malicious rare cases, three.
• When more than one daemon is involved, one typically uses a fixed port number and some of the other daemons may use floating port numbers.
• The client software has to be instructed where to look for the license. With some software products this can be set at install time, others use a manually-editable configuration file and most can be instructed by way of an environment variable.

If the license server is running in the same network as the clients, using the license is as simple as providing the license server and port number information to the clients, typically with an environment variable. The CHPC operates an academic license for its approved users of Ansys software, for example, and the client software can be instructed by setting the LM_LICENSE_FILE environment variable in the job script:

export LM_LICENSE_FILE=1055@login

Access to this license is controlled by listing approved users in a license option file.

It is possible to check if the license service is up and running with a telnet command, for example:

[charles@cnode0491:~]$ telnet login1 1055
Trying 172.18.0.126...
Connected to login1.
Escape character is '^]'.

There is no telnet service running on port 1055, therefore it is not possible for telnet to actually connect, but the above response is an indication that there is indeed some service running on port 1055. If you try the same command with a port where there is no service, you get this sort of response:

[charles@cnode0491:~]$ telnet login1 1047
Trying 172.18.0.126...
telnet: connect to address 172.18.0.126: Connection refused

The telnet “trick” is a quick way of checking for the existence of a service. The more sophisticated tool nmap can provide more comprehensive information:

[charles@login2:~]$ nmap -Pn -p 1055 login1

Starting Nmap 6.40 ( http://nmap.org ) at 2022-04-07 13:35 SAST
Nmap scan report for login1 (172.18.0.126)
Host is up (0.022s latency).
PORT     STATE SERVICE
1055/tcp open  ansyslmd

Nmap done: 1 IP address (1 host up) scanned in 0.77 seconds

It is somewhat atypical for the CHPC to provide in-network software licensing, the availability of such a service depending largely on the marketing strategy of the software vendor. There are a number of difficulties involved with using an off-site license server:

• The lengau compute nodes do not have direct access to the internet
• Some of the license daemons may be using random port numbers, which make it difficult to set up appropriate firewall rules
• The off-site license server is typically situated within a corporate network behind a firewall

It is possible for compute nodes to communicate with the outside world using ssh-tunnels. Users log into and transfer files to the cluster using the secure shell, secure copy and secure file transfer protocol, which all involve communication through port 22. SSH tunnels allow connections made to a local port to be forwarded to a remote machine via a secure channel. An example of accessing a remote license with an SSH tunnel through the node chpclic1 is the way that STAR-CCM+ users can check out a “power on demand” license from an open license server operated by the software vendor:

ssh -f jblogs@chpclic1 -L *:1999:flex.cd-adapco.com:1999 -N
export CDLMD_LICENSE_FILE=1999@localhost

In practice, the CHPC has many users making use of this system, and it has become easier for the CHPC to simply set up a permanent tunnel with the license server port forwarded to port 1999 on chplic1. Therefore the user can simply access the license with this environment variable setting

export CDLMD_LICENSE_FILE=1999@chpclic1 

even though the license server itself is actually on a different continent entirely. In the case of the STAR-CCM+ license, each user has a unique 22-character security key which is passed to the license server at runtime.

The difficulty with default random port numbers is that it makes it impossible to set very tight firewall rules. Strict firewall practices restrict communication to a very small and known number of ports. It is normally very easy to set a fixed port number, by editing the license file and restarting the license server. The top 2 lines of a Flexnet license file typically look like this:

SERVER login1 246e96345ca0 1055
VENDOR ansyslmd 

This Flexnet license file nominates port 1055 for the Flexnet lmgrd daemon, but leaves the ansyslmd vendor daemon free to use a random port number. Adding the port=1056 parameter enforces a fixed port number also for the vendor daemon:

SERVER login1 246e96345ca0 1055
VENDOR ansyslmd port=1056

Internet communication between the CHPC cluster and the remote license server must be possible. As a general rule, license servers at most organisations are inside corporate networks and protected by the network's firewall. In order for the license server ports to be visible to the cluster, the remote firewall must be configured to allow network traffic with the outside world on the license server ports. The security risk is minimized by limiting this traffic to the license ports, which are in any case not privileged, to and from the CHPC's public-facing IP addresses only. These addresses are :

196.24.44.124
196.24.44.6
196.24.44.129
196.24.44.83
154.114.44.7

The CHPC's firewall will also need to be configured to allow license server traffic to and from the off-site license server. You will need to provide the CHPC with the fixed public IP address(es) of your license server(s) as well as the ports that will need to be opened.

1. Contact your CHPC support scientist or engineer and explain your requirements.
2. Ensure that all your license server daemons are providing their services on fixed and known port numbers.
3. Instruct corporate IT at your end to configure the network to make the license server ports accessible. This will probably require port forwarding.
4. Instruct corporate IT at your end to configure the firewall to permit traffic to and from the CHPC's public IP addresses.
5. Provide the CHPC with the license server port numbers as well as your license server's public IP address so that we can configure our firewall appropriately.
6. Once all these configurations have been set, test the connectivity from the nodes login1 and chpclic1 in the cluster. For example, if the license service is on port 1234:

    telnet mylicenseserver.myorganisation.co.za 1234 

   or

    nmap -Pn -p 1234 mylicenseserver.myorganisation.co.za 

7. Now you can attempt to check out the license from a compute node. Preferably first test this in an interactive PBS session, but the instructions are essentially the same as you would use in the following PBS job script snippet:

JobScriptSnippet.pbs

#!/bin/bash
#PBS -l select=.... etc
 
# Set up the required number of ssh-tunnels to the license server ports
ssh -f jblogs@chpclic1 -L *:1234:mylicenseserver.myorganisation.co.za:1234 -N
ssh -f jblogs@chpclic1 -L *:1235:mylicenseserver.myorganisation.co.za:1235 -N
# Set the appropriate license server environment variable.  
# Please RTFM of the software that you are using
export LM_LICENSE_FILE=1234@localhost
# Now run your code with the normal command line 
mpirun -np $nproc ......
# Finally take down each of your ssh tunnels again
kill -9 `ps ux | grep "ssh -f" |  grep -o -E '[0-9]+' | head -1 | sed -e 's/^0\+//' `
kill -9 `ps ux | grep "ssh -f" |  grep -o -E '[0-9]+' | head -1 | sed -e 's/^0\+//' `

If the user has control over the off-site license server that they want to use, it is possible to access the license from inside the cluster by means of ssh-tunneling, without reconfiguring firewalls. A reverse ssh-tunnel is used to forward a port on the license server to the loopback device of a cluster login node. Unfortunately, this forwarded port is not visible to the cluster's internal network, therefore a second conventional ssh-tunneling command is used to connect this port on the loopback device to the internal network interface. Unfortunately, it is not possible to do this with the same port, therefore some trickery is involved with intermediate port numbers.

• The user must have the ability to run commands on the license server.
• Command line access to the license server is necessary. If the license server is running Windows, this can be done with MobaXterm or Cygwin.
• Let us assume that the license server is running the license service on port 2345.
• On the license server command line, create the ssh tunnel:

ssh -R *:2346:localhost:2345 jblogs@scp.chpc.ac.za ssh -L *:2345:localhost:2346 jblogs@login1 -N -f

• In plain English, this translates as: “Connect port 2345 on the license server to port 2346 on localhost at scp.chpc.ac.za. Then connect port 2346 on the localhost of this same machine to port 2345 on the internal network interface of the machine internally known as login1, which just happens to be the same machine.”
• Do the same for the second (or third) license port. Do not try to do this through login1 or chpclic1 with default Ansys license ports, because the Ansys ports are already being used on these servers. If you need to do this with an Ansys license, set different port numbers for all three ports.
• You can now access the license by setting the relevant environment variables as such in your job script:

export LM_LICENSE_FILE=2345@login1

• When you take down the license tunnel, kill the tunnel process on the license server. Then log into scp.chpc.ac.za (login1 internally), look for your secondary tunnel with ps ux | grep ssh and use a kill -9 command followed by the process ID to take down the secondary tunnel on login1.