User Tools

Site Tools


guide:lustre

This is an old revision of the document!


The Lustre Parallel File System

Lustre on Lengau

Lustre tools

The main tool for information on the files and directories of a Lustre file system is lfs.

Searching

Use lfs find (instead of the standard GNU find command) to search for Lustre files based on their metadata properties (size, striping, OSTs, etc.).

To find all files larger than 1GB:

lfs find <dirname> -size +1G

To find all files newer than 60 days:

lfs find <dirname> -ctime -60

Best practices

Lustre is designed for parallel access to large files. It accomplishes this by using striping to distribute the contents of a large file across several disk arrays (OSTs) to enable fast parallel IO.

Striping

The standard recommendation from the Lustre developers is:

  • Files smaller than 1GB should be left with the default striping (1)
  • Between 1GB and 10GB a small stripe should be used (2)
  • 10 to 100GB : use medium striping (2 to 4)
  • Above 100GB: use maximum striping (-c -1)
    Use only this last option with individual files that you know will be above 100GB or use highly parallelized access.

To set the striping for a file to 2

lfs setstripe -c 2 <filename>

To change the default striping applied to files in a sub-directory to 2 (and only applied to files larger than 1G)

lfs setstripe -c 2  -s 1g <dirname>
/var/www/wiki/data/attic/guide/lustre.1588236179.txt.gz · Last modified: 2020/04/30 10:42 by kevin