| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
howto:datamigration [2021/05/03 16:51]
ccrosby [Advice on copying data] |
howto:datamigration [2021/12/09 16:42] (current)
|
| ======Migrating data from old to new Lustre====== | ====== Moving to the new 3 PB Lustre storage ====== |
| | |
| | :!: **Do not attempt to simply copy all data over.** PLAN the transition and read this document first. |
| |
| ==== Background ==== | ==== Background ==== |
| |
| From your home directory, ''cd lustre3p'' will take you straight to your new lustre directory. | From your home directory, ''cd lustre3p'' will take you straight to your new lustre directory. |
| | |
| | ===NB: Do not use the symbolic link in job scripts.=== |
| | |
| | Use the full absolute path in job scripts, e.g., |
| | |
| | cd /mnt/lustre3p/users/$USER/subdir |
| | |
| | (where ''subdir'' is replaced with the actual name of the subdirectory in your Lustre directory which will be the working directory of the job script). |
| | |
| | When you use the symbolic link this generates a file access via your home directory which is on NFS and this extra step slows down the job script for each and every file access. Depending on the process and software, using a symbolic link instead of the full path may prevent the job from executing successfully. |
| | |
| | |
| |
| ==== Policy ==== | ==== Policy ==== |
| |
| As of 1 May 2021, it is already possible to do new runs directly using the new storage. **Running from the original Lustre will no longer be available after 31 May 2021**. The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit. | As of 1 May 2021, it is already possible to do new runs directly using the new storage. |
| | |
| | |
| | The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit. |
| |
| Please refer to the [[guide:filesystems|File Systems guide]] for policy information, but the following points are most relevant: | Please refer to the [[guide:filesystems|File Systems guide]] for policy information, but the following points are most relevant: |
| - There is a strict 90-day lifetime applied to files on Lustre. **Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning**. | |
| - Data intended for future re-use or archiving must be stored elsewhere. | * There is a strict **90-day lifetime** applied to files on Lustre.\\ **Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning**. |
| | |
| | *Data intended for future re-use or archiving must be stored elsewhere. |
| |
| ==== Small files are a problem ==== | ==== Small files are a problem ==== |
| Generally speaking, large files of several 100 MB can be transferred at high speed, thanks to the inherent speed of both Lustre systems and the high-speed Infiniband networks connecting everything in the cluster. However, transferring large numbers of small files can be extremely time-consuming, because of the need to negotiate the transfer of each file individually. If you have a lot of small files to transfer, consider using tar to "archive" the files into a single file which can be copied more effectively. However, bear in mind that we have tested the utility [[https://github.com/eeertekin/bbcp|bbcp]] also for this purpose, and it has turned out to be easy to use and faster. **bbcp** is available on all nodes in the cluster. | Generally speaking, large files of several hundred MB can be transferred at high speed, thanks to the inherent speed of both Lustre systems and the high-speed Infiniband networks connecting everything in the cluster. However, transferring large numbers of small files can be extremely time-consuming, because of the need to negotiate the transfer of each file individually. If you have a lot of small files to transfer, consider using tar to "archive" the files into a single file which can be copied more effectively. The following example shows how to do this without first saving an intermediate file. The files are concatenated into a stream, and a pipe is used to untar them into the new directory. ** Do not attempt this before testing it first.** |
| | |
| | <code> |
| | cd /mnt/lustre/users/jblogs |
| | tar cf - MyDataDirectory | tar xf - -C /mnt/lustre3p/users/jblogs |
| | </code> |
| | Of course, this assumes that your username is "jblogs". Please use your own username and directory names. If your files are uncompressed ASCII files, it will probably be faster to compress them with the following version of the process: |
| | <code> |
| | cd /mnt/lustre/users/jblogs |
| | tar czf - MyDataDirectory | tar xzf - -C /mnt/lustre3p/users/jblogs |
| | </code> |
| | Do not use data compression if you haven't tested it first and found it to be beneficial. Many types of data files, such as NetCDF files, are already compressed, and trying to compress them again will slow the process down substantially. |
| | |
| | However, bear in mind that we have tested the utility [[https://github.com/eeertekin/bbcp|bbcp]] also for this purpose, and it has turned out to be easy to use and fast. **bbcp** is available on all nodes in the cluster. |
| |
| ==== Advice on copying data ==== | ==== Advice on copying data ==== |
| - Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded. This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle. You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions [[http://wiki.chpc.ac.za/howto:tipsandtricks#using_an_interactive_pbs_session|here]]) on a compute node for this purpose. You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it. | |
| - **Plan** your data migration. **Do not** simply copy all of your data. This will take a very long time, and it will simply leave you with unused data that will shortly be deleted. **Only** move the data that you are currently working with. Anything older is either unused, or needs to be archived somewhere else. | * Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded. This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle. You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions [[http://wiki.chpc.ac.za/howto:tipsandtricks#using_an_interactive_pbs_session|here]]) on a compute node for this purpose. You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it. |
| | |
| | * **Plan** your data migration. **Do not** simply copy all of your data. This will take a very long time, and it will simply leave you with unused data that will shortly be deleted. **Only** move the data that you are currently working with. Anything older is either unused, or needs to be archived somewhere else. |
| | |
| | ==== How to use bbcp ==== |
| | |
| | bbcp can easily be used to copy data between different computers. However, it also works well when copying data between different directories on the same server, hence it is well suited to the purpose of migrating data from one Lustre storage to another. By default it can use multiple simultaneous streams and it can resume interrupted transfers. The syntax is described [[https://github.com/eeertekin/bbcp|here]]. |
| | |
| | == Example == |
| | |
| | This example will take the entire directory DataDirectory and copy it to the new Lustre space of the user jblogs. As ever, **PLEASE** use your own username and not jblogs! |
| | <code> |
| | bbcp -apr -s 4 -P 30 /mnt/lustre/users/jblogs/DataDirectory /mnt/lustre3p/users/jblogs/ |
| | </code> |
| | The rather cryptic command line options mean the following: |
| | ^ Option ^ Meaning ^ |
| | | -a | append mode to restart a previously failed copy | |
| | | -p | preserve source mode, ownership, and dates | |
| | | -r | copy subdirectories and their contents (actual files only) | |
| | | -s 4 | number of network streams to use (default is 4) | |
| | | -P 30 | produce a progress message every sec seconds (15 sec minimum) | |
| | |
| | == Other potentially useful options == |
| | ^ Option ^ Meaning ^ |
| | | -v | verbose (provide more informative output, such as transfer rate per file) | |
| | | -O | omits files that already exist at the target node (useful with -r) | |
| | | -c lvl | compress data before sending across the network (default lvl is 1) | |
| | | -t sec | sets the time limit for the copy to complete | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| |
| |
| |
