With effect from 1 May 2021, there is a new 3 PB Lustre storage system available. It is mounted on the system as /mnt/lustre3p, and user directories for all current users have already been created. To make life easy for yourself, create a symbolic link in your home directory:

[jblogs@login2:~]$ USERNAME=`whoami`
ln -s /mnt/lustre3p/users/$USERNAME lustre3p

From your home directory, cd lustre3p will take you straight to your new lustre directory.

As of 1 May 2021, it is already possible to do new runs directly using the new storage. Running from the original Lustre will no longer be available after 31 May 2021. The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit.

Please refer to the File Systems guide for policy information, but the following points are most relevant:

1. There is a strict 90-day lifetime applied to files on Lustre. Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning.
2. Data intended for future re-use or archiving must be stored elsewhere.

Generally speaking, large files of several 100 MB can be transferred at high speed, thanks to the inherent speed of both Lustre systems and the high-speed Infiniband networks connecting everything in the cluster. However, transferring large numbers of small files can be extremely time-consuming, because of the need to negotiate the transfer of each file individually. If you have a lot of small files to transfer, consider using tar to “archive” the files into a single file which can be copied more effectively. However, bear in mind that we have tested the utility bbcp also for this purpose, and it has turned out to be easy to use and faster. bbcp is available on all nodes in the cluster.

1. Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded. This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle. You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions here) on a compute node for this purpose. You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it.
2. Plan your data migration. Do not simply copy all of your data. This will take a very long time, and it will simply leave you with unused data that will shortly be deleted. Only move the data that you are currently working with. Anything older is either unused, or needs to be archived somewhere else.