CRESCO4C
================

Cluster Features
----------------

The **CRESCO4C cluster** located in **Casaccia** is a computing system composed of **32 compute nodes**.

Each node provides:

- **2 sockets with 8 cores** Intel **E5-2670 2.6 GHz** processors and **64 GB RAM**
- **4 GB per core**, **8 instructions per clock cycle**
- **1 × 500 GB SATA II disk**
- **InfiniBand QDR interface (40 Gb/s, 32 Gb/s effective bandwidth)**
- **Two GbE network interfaces**
- **BMC/IPMI 1.8 support** and software for **remote console management**

Overall the cluster provides **512 CPU cores** interconnected through a **low-latency, high-bandwidth network** based on **InfiniBand 4×QDR (40 Gb/s)** managed by a **Qlogic 12800 switch**.

The architecture of a **CRESCO4C compute node** is represented in the following scheme:

.. image:: clusters/cresco4c_image1.png
   :align: center
   :width: 1600px

|

Front-end Nodes
---------------

Access to the cluster is performed by logging into the **front-end node**. The nodes are used for submitting jobs through **LSF**, editing job submission scripts and compiling applications.

.. code-block::

   cresco4cx001.frascati.enea.it


Access
------

To access the system via SSH, refer to the documentation at the page :doc:`Access to CRESCO System <../GENERAL_INFORMATION/access_to_systems>`.

Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page:

`Interface CRESCO-ONDEMAND <https://cresco-ondemand.portici.enea.it>`_

File Systems
------------

The file systems available on **CRESCO4F** are:

**AFS**: The geographically distributed file system shared across the entire **ENEAGRID** infrastructure.
Information about the common **HOME directory structure** for all ENEAGRID/CRESCO users can be found in the section **File Systems and Data Management** available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`.

**GPFS**: The high-performance parallel I/O file system.
Information aboutGPFS** for ENEAGRID/CRESCO users is available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`.

.. note::
   
   The **GPFS file system** available on CRESCO4C nodes is:
   
   .. code-block:: text

      /gcasi0   (~33 TB)

Compilers
---------

The table below lists the **compilers available on CRESCO4C**.

.. list-table::
   :header-rows: 1
   :widths: 20 18 18 18 18 26

   * -
     - GNU (4.8.5)
     - GNU (7.3.0)
     - Intel (17.0.4)
     - Intel (18.0.2)
     - PGI (19.7-0)

   * - PATH compilers
     - /usr/bin
     - /opt/gcc730/bin
     - /opt/intel/bin
     - /opt/intel18/bin
     - /afs/enea.it/software/pgi_compiler/pgi_19_7/build/linux86-64/19.7/bin

MPI Flavours Available
----------------------

The **MPI flavours** available on the system, grouped by compiler, are listed
in the following table.

.. list-table::
   :header-rows: 1
   :widths: 20 27 27 26

   * -
     - GNU (version 4.8.5 / 7.3.0)
     - Intel (version 17.0.4)
     - PGI (version 19.7-0)

   * - OpenMPI
     - openmpi_gcc-4.0.5
     - openmpi_intel-4.0.5
     - openmpi_pgi-4.0.5

   * - Intel MPI
     - No flavour available
     - IntelMPI
     - No flavour available

Job Submission
--------------

The scheduler that manages the resources of CRESCO4C and allows the submission of jobs is LSF.
Below the are the LSF queues defined for CRESCO4C together with some examples of submission wrappers for serial and parallel jobs.

LSF Queues on CRESCO4C
~~~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :header-rows: 1
   :widths: 20 26

   * - PARTITION
     - DESCRIPTION

   * - **cresco4c_h144**
     - | Queue for serial and parallel jobs requiring up to **32 cores**.
       | Maximum running time: **144 hours**. Open to all users.

   * - **cresco4c_16h24**
     - | Queue for parallel jobs requiring greater than **16 cores**.
       | Maximum running time for **24 hours**.

.. note::

   On the **cresco4c_16h24** queue, the number of requested cores must be a
   **multiple of 16**, corresponding to the number of cores available on each
   compute node.


Job Script Examples
~~~~~~~~~~~~~~~~~~~~~~

**OpenMPI 4.0.5 (GCC, Intel, PGI)**

.. code-block:: bash

   #!/bin/sh
   exe=/path/to/exe                            # path of your MPI program
   HOSTFILE=$LSB_DJOB_HOSTFILE                 # name of hostfile for mpirun
   N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l`    # give to mpirun same number of slots

   mpirun --mca plm_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe

**Intel MPI 4.1.2**

.. code-block:: bash

   #!/bin/sh
   exe=/path/to/exe                            # path of your MPI program
   HOSTFILE=$LSB_DJOB_HOSTFILE                 # name of hostfile for mpirun
   N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l`    # give to mpirun same number of slots

   mpirun -bootstrap rsh -n $N_procs -machinefile $HOSTFILE $exe

**Job Submission Examples**

.. code-block:: bash

   bsub -o output.o -e error.e -q cresco4c_h144 -n number_of_cores ./submission_script
   bsub -o output.o -e error.e -q cresco4c_16h24 -n number_of_cores ./submission_script