With effect from 1 May 2021, there is a new 3 PB Lustre storage system available. It is mounted on the system as /mnt/lustre3p, and user directories for all current users have already been created. To make life easy for yourself, create a symbolic link in your home directory:

[jblogs@login2:~]$ USERNAME=`whoami`
ln -s /mnt/lustre3p/users/$USERNAME lustre3p

From your home directory, cd lustre3p will take you straight to your new lustre directory.

Do not use the symbolic link in job scripts.

Use the full absolute path in job scripts, e.g.,

cd /mnt/lustre3p/users/$USER/subdir

(where subdir is replaced with the actual name of the subdirectory in your Lustre directory which will be the working directory of the job script).

When you use the symbolic link this generates a file access via your home directory which is on NFS and this extra step slows down the job script for each and every file access.

As of 1 May 2021, it is already possible to do new runs directly using the new storage. Running from the original Lustre will no longer be available after 31 May 2021. The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit.

Please refer to the File Systems guide for policy information, but the following points are most relevant:

1. There is a strict 90-day lifetime applied to files on Lustre. Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning.
2. Data intended for future re-use or archiving must be stored elsewhere.

Generally speaking, large files of several hundred MB can be transferred at high speed, thanks to the inherent speed of both Lustre systems and the high-speed Infiniband networks connecting everything in the cluster. However, transferring large numbers of small files can be extremely time-consuming, because of the need to negotiate the transfer of each file individually. If you have a lot of small files to transfer, consider using tar to “archive” the files into a single file which can be copied more effectively. However, bear in mind that we have tested the utility bbcp also for this purpose, and it has turned out to be easy to use and faster. bbcp is available on all nodes in the cluster.

1. Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded. This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle. You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions here) on a compute node for this purpose. You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it.
2. Plan your data migration. Do not simply copy all of your data. This will take a very long time, and it will simply leave you with unused data that will shortly be deleted. Only move the data that you are currently working with. Anything older is either unused, or needs to be archived somewhere else.

bbcp can easily be used to copy data between different computers. However, it also works well when copying data between different directories on the same server, hence it is well suited to the purpose of migrating data from one Lustre storage to another. By default it can use multiple simultaneous streams and it can resume interrupted transfers. The syntax is described here.

This example will take the entire directory DataDirectory and copy it to the new Lustre space of the user jblogs. As ever, PLEASE use your own username and not jblogs!

bbcp -apr -s 4 -P 5 /mnt/lustre/users/jblogs/DataDirectory /mnt/lustre3p/users/jblogs/

The rather cryptic command line options mean the following: