LUSTRE

Lustre is a High-performance parallel file system used for large-scale I/O workloads. Currently, Lustre filesystem is available on CRESCO7 and CRESCO8.

One of the main factors determining the high performance of Lustre file systems is the ability to stripe data across multiple storage targets (OST) in a round-robin fashion. In practice, files can be divided into multiple parts that will then be stored on different OSTs within the Lustre system. Any file is simply a linear sequence of bytes.

Striping offers two advantages:

  • An increase in bandwidth because multiple processes can access the same file simultaneously.

  • The possibility of storing very large files that would otherwise occupy more space than a single OST.

However, striping is not without drawbacks: - Increased overhead due to network operations and contention with servers. - Increased risk of file corruption due to hardware failures.

Users have the possibility to configure the stripe size and the number of stripes used for any file.

Default Stripe Settings

The default stripe settings vary from machine to machine, but the stripe count is generally set to 1 or 2 and the stripe size is generally 1 MB. To determine the stripe configurations for a file or a directory use the lfs getstripe command:

> lfs getstripe movie.mov
movie.mov
lmm_stripe_count:   4
lmm_stripe_size:    1048576
lmm_stripe_offset:  186
        obdidx           objid          objid            group
           186        52153455      0x31bcc6f                0
           258        53124880      0x32a9f10                0
            25        52477227      0x320bd2b                0
            97        52444876      0x3203ecc                0

In this example, the file movie.mov has been striped across 4 OSTs with a stripe size of 1 MB. The listed obdidx numbers are the indexes of the OSTs used to stripe this file. Using getstripe on a directory provides information about the directory and the files contained within it. It is possible to restrict getstripe to display only directory information using the -d option. Alternatively, the -r option can be used to recursively follow all subdirectories.

Setting Striping Configurations

The lfs setstripe command is used to specify a particular striping configuration for a file or a directory. For a file, setstripe:

  • returns an error if the file already exists (see the note below),

  • otherwise creates an empty file with the desired stripe settings.

Using setstripe on a directory:

  • changes the stripe settings for the directory,

  • any file subsequently created in the directory will inherit those settings,

  • does not affect existing files in the directory.

Note

Once a file has been written on Lustre with a particular stripe configuration, it is not possible to simply use setstripe to change it. The file must be rewritten with a new configuration. In general, if it is necessary to change the striping of a file, one of the following methods can be used:

  • using setstripe, create a new empty file with the desired stripe settings and then copy the old file into the new one, or

  • create a directory with the desired configuration and cp (not mv) the file into that directory.

The options for lfs setstripe are:

  • -c to set the stripe count; 0 means use the system default value (usually 1) and -1 means stripe across all available OSTs (up to the intrinsic system limit of 160)

  • -s to set the stripe size; 0 means use the system default value (usually 1 MB), otherwise use k, m or g for KB, MB or GB respectively

lfs setstripe -c 50 -s 32m bigdir

User Area

For each user is available a SCRATCH area on Lustre with the following details:

CLUSTER

QUOTA

VARIABLE

HOME LINK

CRESCO7

1.1 PT quota

PFS_SCRATCH0_LUSTRE

~/PFS/ltmp/

CRESCO8

6.9PT quota

$PFS_SCRATCH1_LUSTRE

~/PFS/ltmp/

Warning

The STORE area is still a work in progress.

Lustre Best Practices

Large files benefit from a higher stripe count. By striping a large file across many OSTs, the bandwidth for accessing the file increases and many processes can operate simultaneously on the same file. Conversely, a large file striped on only one or two OSTs may degrade the performance of the entire Lustre system by unnecessarily filling those OSTs. A good practice is to have dedicated directories with a high stripe count where very large files can be written.

Another scenario to avoid is having small files with a high stripe count. This can harm performance because of the unnecessary communication overhead with multiple OSTs. A good practice is to ensure that small files are written in a directory with a stripe count equal to 1, meaning without striping.

I/O Considerations

  • With a file-per-process I/O scheme, it is better not to use striping (stripe count equal to 1). This limits contention with OSTs when dealing with a large number of files/processes.

  • When accessing a single file shared by many processes, the stripe count should be equal to the number of processes, if possible. The size and position of I/O operations from processes must be handled carefully to allow maximum stripe alignment, so that each process accesses only one OST. Avoid I/O access patterns where a single process must access all the OSTs used.

  • Open files in read-only mode when possible.

  • Avoid unnecessary file operations, such as the following:

    • multiple open-write-close cycles on the same file during an application

    • many processes retrieving information (stat) from the same file

    • multiple processes reading the same small file

    • Excessive use of stdout or stderr by parallel processes for reading a small file or retrieving statistical information should be avoided; it is better for a single process to perform the I/O and then transmit the results to the other processes.