LUSTRE ======= Lustre is a High-performance parallel file system used for large-scale I/O workloads. Currently, Lustre filesystem is available on **CRESCO7** and **CRESCO8**. One of the main factors determining the high performance of Lustre file systems is the ability to stripe data across multiple storage targets (OST) in a round-robin fashion. In practice, files can be divided into multiple parts that will then be stored on different OSTs within the Lustre system. Any file is simply a linear sequence of bytes. Striping offers two advantages: - An increase in bandwidth because multiple processes can access the same file simultaneously. - The possibility of storing very large files that would otherwise occupy more space than a single OST. However, striping is not without drawbacks: - Increased overhead due to network operations and contention with servers. - Increased risk of file corruption due to hardware failures. Users have the possibility to configure the stripe size and the number of stripes used for any file. Default Stripe Settings ----------------------- The default stripe settings vary from machine to machine, but the stripe count is generally set to 1 or 2 and the stripe size is generally 1 MB. To determine the stripe configurations for a file or a directory use the ``lfs getstripe`` command: .. code-block:: bash > lfs getstripe movie.mov movie.mov lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_stripe_offset: 186 obdidx objid objid group 186 52153455 0x31bcc6f 0 258 53124880 0x32a9f10 0 25 52477227 0x320bd2b 0 97 52444876 0x3203ecc 0 In this example, the file movie.mov has been striped across 4 OSTs with a stripe size of 1 MB. The listed obdidx numbers are the indexes of the OSTs used to stripe this file. Using getstripe on a directory provides information about the directory and the files contained within it. It is possible to restrict getstripe to display only directory information using the ``-d`` option. Alternatively, the ``-r`` option can be used to recursively follow all subdirectories. Setting Striping Configurations ------------------------------- The ``lfs setstripe`` command is used to specify a particular striping configuration for a file or a directory. For a file, setstripe: - returns an error if the file already exists (see the note below), - otherwise creates an empty file with the desired stripe settings. Using setstripe on a directory: - changes the stripe settings for the directory, - any file subsequently created in the directory will inherit those settings, - does not affect existing files in the directory. .. note:: Once a file has been written on Lustre with a particular stripe configuration, it is not possible to simply use setstripe to change it. The file must be rewritten with a new configuration. In general, if it is necessary to change the striping of a file, one of the following methods can be used: - using setstripe, create a new empty file with the desired stripe settings and then copy the old file into the new one, or - create a directory with the desired configuration and ``cp`` (not ``mv``) the file into that directory. The options for ``lfs setstripe`` are: - ``-c`` to set the stripe count; 0 means use the system default value (usually 1) and -1 means stripe across all available OSTs (up to the intrinsic system limit of 160) - ``-s`` to set the stripe size; 0 means use the system default value (usually 1 MB), otherwise use ``k``, ``m`` or ``g`` for KB, MB or GB respectively .. code-block:: bash lfs setstripe -c 50 -s 32m bigdir User Area ------------------ For each user is available a **SCRATCH** area on **Lustre** with the following details: .. list-table:: :header-rows: 1 :widths: 30 25 25 30 * - CLUSTER - QUOTA - VARIABLE - HOME LINK * - CRESCO7 - 1.1 PT quota - PFS_SCRATCH0_LUSTRE - ~/PFS/ltmp/ * - CRESCO8 - 6.9PT quota - $PFS_SCRATCH1_LUSTRE - ~/PFS/ltmp/ .. warning:: The STORE area is still a work in progress. Lustre Best Practices ---------------------- **Large files** benefit from a higher stripe count. By striping a large file across many OSTs, the bandwidth for accessing the file increases and many processes can operate simultaneously on the same file. Conversely, a large file striped on only one or two OSTs may degrade the performance of the entire Lustre system by unnecessarily filling those OSTs. A good practice is to have dedicated directories with a high stripe count where very large files can be written. Another scenario to avoid is having **small files** with a high stripe count. This can harm performance because of the unnecessary communication overhead with multiple OSTs. A good practice is to ensure that small files are written in a directory with a stripe count equal to 1, meaning without striping. I/O Considerations ~~~~~~~~~~~~~~~~~~~ - With a **file-per-process I/O scheme**, it is better not to use striping (stripe count equal to 1). This limits contention with OSTs when dealing with a large number of files/processes. - When accessing a **single file shared** by many processes, the stripe count should be equal to the number of processes, if possible. The size and position of I/O operations from processes must be handled carefully to allow maximum stripe alignment, so that each process accesses only one OST. Avoid I/O access patterns where a single process must access all the OSTs used. - Open files in read-only mode when possible. - **Avoid unnecessary file operations**, such as the following: - multiple open-write-close cycles on the same file during an application - many processes retrieving information (stat) from the same file - multiple processes reading the same small file - Excessive use of stdout or stderr by parallel processes for reading a small file or retrieving statistical information should be avoided; it is better for a single process to perform the I/O and then transmit the results to the other processes. USEFUL LINKS ------------ Basic general information about Lustre for ENEAGRID/CRESCO users can be found `here `_