LSF
====

LSF (IBM Spectrum) is a workload management platform, job scheduler, for distributed high performance computing.
It can be used to execute batch jobs on **CRESCO6**, **CRESCO4F** and **CRESCO4C**


Monitoring the cluster
-----------------------

To monitor the status of the cluster compute nodes, use the following command:

.. code-block:: bash

   bqueues

Example output from the CRESCO6 cluster:

.. code-block:: bash

   QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP 

   cresco6_h4       41  Open:Inact        -    -    -    -     0     0     0     0
   cresco6_h144     40  Open:Inact        -    -    -    -   109   109     0     0

Running Application
---------------------

LSF provides two different modes to allocate resources:

Interactive
^^^^^^^^^^^^

An interactive session can be started using the ``lsrun`` command.


.. code-block:: bash

   lsrun -P my_project -q interactive /bin/bash

Bsub
^^^^^

With LSF you create a batch job which is then submitted to the scheduler.
When LSF executes a job, it sets environment variables such as:

- ``LSB_HOSTS`` → list of allocated hosts
- ``LSB_DJOB_HOSTFILE`` → path to the hostfile

A basic submission command is:

.. code-block:: bash

   bsub -n <n_procs> mpirun_wrapper.sh

Where:

- ``-n`` specifies the number of processors
- the wrapper script runs the MPI program

Example:

.. code-block:: bash

   bsub -n <n_procs> -o lsf.out -e lsf.err -q <queue_name> ./my_script.sh

The most useful flag for ``bsub`` command are:

.. list-table:: Bsub flag
   :header-rows: 1
   :widths: 20 80

   * - Flag
     - Description
   * - -q <queue>
     - Submits the job to the specified queue.
   * - -J <job_name>
     - Assigns a name to the job.
   * - -P <project>
     - Assigns the job to the specified project.
   * - -n <num_slots>
     - Requests the specified number of CPU slots.
   * - -M <memory>
     - Sets the memory limit for the job.
   * - -W <time_limit>
     - Sets the wall-clock time limit.
   * - -o <file>
     - Writes the standard output to the specified file.
   * - -e <file>
     - Writes the standard error to the specified file.
   * - -R "<resource_request>"
     - Specifies resource requirements for the job.
   * - -I
     - Runs the job interactively.

LSF commands
---------------

The table below provides a short description of the most commonly used LSF commands.

.. list-table:: LSF jobs command
   :header-rows: 1
   :widths: 20 80

   * - Command
     - Description
   * - lsid
     - Displays the LSF version, cluster name, and management host.
   * - lsinfo
     - Displays LSF configuration information, such as resource names, host types, and host models.
   * - lshosts
     - Displays static host information, such as CPU count, memory, swap, and configured resources.
   * - lsload
     - Displays dynamic load information for cluster hosts.
   * - bjobs
     - Shows the current status of submitted jobs.
   * - bhist
     - Shows historical information about completed jobs.
   * - bacct
     - Shows accounting and resource usage information for finished jobs.
   * - bqueues
     - Shows the status of LSF queues.
   * - bhosts
     - Shows the status of batch execution hosts.
   * - bsub
     - Submits a job to LSF.
   * - lsrun
     - Runs an interactive task through LSF.
   * - brequeue <job_id>
     - Kills and requeues a job.
   * - bkill <job_id>
     - Kills or sends a signal to a job.

.. note::

   Parallel jobs may write data to the local ``/tmp`` directory of compute nodes. To prevent storage saturation, temporary data is automatically cleaned files older than 10 days are removed.