wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » howto » datamigration
Trace: • datamigration
howto:datamigration

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
howto:datamigration [2021/05/04 14:08]
ccrosby [Small files are a problem] 		howto:datamigration [2021/12/09 16:42] (current)
Line 1: Line 1:
 ====== Moving to the new 3 PB Lustre storage ====== ====== Moving to the new 3 PB Lustre storage ======
  
-**:!: Do not attempt to simply copy all data over.**  PLAN the transition and read this document first.+:!: **Do not attempt to simply copy all data over.**  PLAN the transition and read this document first.
  
 ==== Background ==== ==== Background ====
Line 13: Line 13:
 From your home directory, ''cd lustre3p'' will take you straight to your new lustre directory. From your home directory, ''cd lustre3p'' will take you straight to your new lustre directory.
  
-**:!: Do not use the symbolic link in job scripts.**+===NB: Do not use the symbolic link in job scripts.===
  
 Use the full absolute path in job scripts, e.g., Use the full absolute path in job scripts, e.g.,
Line 21: Line 21:
 (where ''subdir'' is replaced with the actual name of the subdirectory in your Lustre directory which will be the working directory of the job script). (where ''subdir'' is replaced with the actual name of the subdirectory in your Lustre directory which will be the working directory of the job script).
  
-When you use the symbolic link this generates a file access via your home directory which is on NFS and this extra step slows down the job script for each and every file access.+When you use the symbolic link this generates a file access via your home directory which is on NFS and this extra step slows down the job script for each and every file access.  Depending on the process and software, using a symbolic link instead of the full path may prevent the job from executing successfully.
  
  
Line 29: Line 29:
 As of 1 May 2021, it is already possible to do new runs directly using the new storage.   As of 1 May 2021, it is already possible to do new runs directly using the new storage.  
  
-:!: **Running from the original Lustre will no longer be available after 31 May 2021**. :!:   
  
 The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit. The original Lustre storage will be permanently retired shortly afterwards, as it has reached its lifetime limit.
  
 Please refer to the [[guide:filesystems|File Systems guide]] for policy information, but the following points are most relevant: Please refer to the [[guide:filesystems|File Systems guide]] for policy information, but the following points are most relevant:
-  - There is a strict 90-day lifetime applied to files on Lustre.  **Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning**. + 
-  - Data intended for future re-use or archiving must be stored elsewhere.+  * There is a strict **90-day lifetime** applied to files on Lustre.\\ **Any file that has not been accessed for read or write for 90 days or more will be automatically deleted without warning**. 
 + 
 +  *Data intended for future re-use or archiving must be stored elsewhere.
  
 ==== Small files are a problem ==== ==== Small files are a problem ====
Line 54: Line 55:
  
 ==== Advice on copying data ==== ==== Advice on copying data ====
-  - Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded.  This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle.  You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions [[http://wiki.chpc.ac.za/howto:tipsandtricks#using_an_interactive_pbs_session|here]]) on a compute node for this purpose.  You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it. + 
-  - **Plan** your data migration.  **Do not** simply copy all of your data.  This will take a very long time, and it will simply leave you with unused data that will shortly be deleted.  **Only** move the data that you are currently working with.  Anything older is either unused, or needs to be archived somewhere else.+  * Do not attempt to copy your data on login2, where there is a script that protects this login node from being overloaded.  This script will kill your copy process and may leave you with partially copied files which may be awkward to disentangle.  You may use one of the other service nodes, but it is probably best just to get yourself an interactive session (instructions [[http://wiki.chpc.ac.za/howto:tipsandtricks#using_an_interactive_pbs_session|here]]) on a compute node for this purpose.  You can also create a PBS script for the purpose of copying your files, but first ensure that this works as intended before submitting it. 
 + 
 +  * **Plan** your data migration.  **Do not** simply copy all of your data.  This will take a very long time, and it will simply leave you with unused data that will shortly be deleted.  **Only** move the data that you are currently working with.  Anything older is either unused, or needs to be archived somewhere else.
  
 ==== How to use bbcp ==== ==== How to use bbcp ====
 +
 bbcp can easily be used to copy data between different computers.  However, it also works well when copying data between different directories on the same server, hence it is well suited to the purpose of migrating data from one Lustre storage to another.  By default it can use multiple simultaneous streams and it can resume interrupted transfers.  The syntax is described [[https://github.com/eeertekin/bbcp|here]]. bbcp can easily be used to copy data between different computers.  However, it also works well when copying data between different directories on the same server, hence it is well suited to the purpose of migrating data from one Lustre storage to another.  By default it can use multiple simultaneous streams and it can resume interrupted transfers.  The syntax is described [[https://github.com/eeertekin/bbcp|here]].
  
/app/dokuwiki/data/attic/howto/datamigration.1620130118.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki