| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
howto:datamigration [2021/05/04 14:10]
ccrosby [Background] |
howto:datamigration [2021/12/09 16:42] (current)
|
| ====== Moving to the new 3 PB Lustre storage ====== | ====== Moving to the new 3 PB Lustre storage ====== |
| |
| **:!: Do not attempt to simply copy all data over.** PLAN the transition and read this document first. | :!: **Do not attempt to simply copy all data over.** PLAN the transition and read this document first. |
| |
| ==== Background ==== | ==== Background ==== |
| From your home directory, ''cd lustre3p'' will take you straight to your new lustre directory. | From your home directory, ''cd lustre3p'' will take you straight to your new lustre directory. |
| |
| **:!: Do not use the symbolic link in job scripts.** | ===NB: Do not use the symbolic link in job scripts.=== |
| |
| Use the full absolute path in job scripts, e.g., | Use the full absolute path in job scripts, e.g., |
| As of 1 May 2021, it is already possible to do new runs directly using the new storage. | As of 1 May 2021, it is already possible to do new runs directly using the new storage. |
| |
| :!: **Running from the original Lustre will no longer be available after 31 May 2021**. :!: | |
| |
| The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit. | The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit. |
| |
| Please refer to the [[guide:filesystems|File Systems guide]] for policy information, but the following points are most relevant: | Please refer to the [[guide:filesystems|File Systems guide]] for policy information, but the following points are most relevant: |
| - There is a strict 90-day lifetime applied to files on Lustre. **Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning**. | |
| - Data intended for future re-use or archiving must be stored elsewhere. | * There is a strict **90-day lifetime** applied to files on Lustre.\\ **Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning**. |
| | |
| | *Data intended for future re-use or archiving must be stored elsewhere. |
| |
| ==== Small files are a problem ==== | ==== Small files are a problem ==== |
| |
| ==== Advice on copying data ==== | ==== Advice on copying data ==== |
| - Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded. This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle. You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions [[http://wiki.chpc.ac.za/howto:tipsandtricks#using_an_interactive_pbs_session|here]]) on a compute node for this purpose. You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it. | |
| - **Plan** your data migration. **Do not** simply copy all of your data. This will take a very long time, and it will simply leave you with unused data that will shortly be deleted. **Only** move the data that you are currently working with. Anything older is either unused, or needs to be archived somewhere else. | * Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded. This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle. You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions [[http://wiki.chpc.ac.za/howto:tipsandtricks#using_an_interactive_pbs_session|here]]) on a compute node for this purpose. You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it. |
| | |
| | * **Plan** your data migration. **Do not** simply copy all of your data. This will take a very long time, and it will simply leave you with unused data that will shortly be deleted. **Only** move the data that you are currently working with. Anything older is either unused, or needs to be archived somewhere else. |
| |
| ==== How to use bbcp ==== | ==== How to use bbcp ==== |
| | |
| bbcp can easily be used to copy data between different computers. However, it also works well when copying data between different directories on the same server, hence it is well suited to the purpose of migrating data from one Lustre storage to another. By default it can use multiple simultaneous streams and it can resume interrupted transfers. The syntax is described [[https://github.com/eeertekin/bbcp|here]]. | bbcp can easily be used to copy data between different computers. However, it also works well when copying data between different directories on the same server, hence it is well suited to the purpose of migrating data from one Lustre storage to another. By default it can use multiple simultaneous streams and it can resume interrupted transfers. The syntax is described [[https://github.com/eeertekin/bbcp|here]]. |
| |
