LSF

LSF (IBM Spectrum) is a workload management platform, job scheduler, for distributed high performance computing. It can be used to execute batch jobs on CRESCO6, CRESCO4F and CRESCO4C

Monitoring the cluster

To monitor the status of the cluster compute nodes, use the following command:

bqueues

Example output from the CRESCO6 cluster:

QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP

cresco6_h4       41  Open:Inact        -    -    -    -     0     0     0     0
cresco6_h144     40  Open:Inact        -    -    -    -   109   109     0     0

Running Application

LSF provides two different modes to allocate resources:

Interactive

An interactive session can be started using the lsrun command.

lsrun -P my_project -q interactive /bin/bash

Bsub

With LSF you create a batch job which is then submitted to the scheduler. When LSF executes a job, it sets environment variables such as:

• LSB_HOSTS → list of allocated hosts

• LSB_DJOB_HOSTFILE → path to the hostfile

A basic submission command is:

bsub -n <n_procs> mpirun_wrapper.sh

Where:

• -n specifies the number of processors

• the wrapper script runs the MPI program

Example:

bsub -n <n_procs> -o lsf.out -e lsf.err -q <queue_name> ./my_script.sh

The most useful flag for bsub command are:

Bsub flag

Flag	Description
-q <queue>	Submits the job to the specified queue.
-J <job_name>	Assigns a name to the job.
-P <project>	Assigns the job to the specified project.
-n <num_slots>	Requests the specified number of CPU slots.
-M <memory>	Sets the memory limit for the job.
-W <time_limit>	Sets the wall-clock time limit.
-o <file>	Writes the standard output to the specified file.
-e <file>	Writes the standard error to the specified file.
-R “<resource_request>”	Specifies resource requirements for the job.
-I	Runs the job interactively.

LSF commands

The table below provides a short description of the most commonly used LSF commands.

LSF jobs command

Command	Description
lsid	Displays the LSF version, cluster name, and management host.
lsinfo	Displays LSF configuration information, such as resource names, host types, and host models.
lshosts	Displays static host information, such as CPU count, memory, swap, and configured resources.
lsload	Displays dynamic load information for cluster hosts.
bjobs	Shows the current status of submitted jobs.
bhist	Shows historical information about completed jobs.
bacct	Shows accounting and resource usage information for finished jobs.
bqueues	Shows the status of LSF queues.
bhosts	Shows the status of batch execution hosts.
bsub	Submits a job to LSF.
lsrun	Runs an interactive task through LSF.
brequeue <job_id>	Kills and requeues a job.
bkill <job_id>	Kills or sends a signal to a job.

Note

Parallel jobs may write data to the local /tmp directory of compute nodes. To prevent storage saturation, temporary data is automatically cleaned files older than 10 days are removed.