CRESCO7 ================ .. warning:: The CRESCO7 cluster does not guarantee backup, it is recommended to use only in development and testing. Cluster Characteristics ------------------------ The CRESCO7 cluster located in Portici is a computing system composed of **144 compute nodes**. Each node is equipped with: - **2 sockets**, each with **24 cores**, based on **Intel(R) Xeon(R) Platinum 8160 processors** running at **2.10 GHz** - **192 GB of RAM** - **One Mellanox EDR interface** - **Two Gigabit Ethernet (GbE) interfaces** - **BMC/IPMI support** and software for remote console management The system provides a total of **6,912 CPU cores**, interconnected through a **high-bandwidth, low-latency network** based on **Mellanox EDR at 100 Gb/s**. The architecture of a CRESCO7 compute node is illustrated in the following diagram: .. image:: clusters/cresco7_image1.png :align: center :width: 1600px | Front-end Nodes --------------- Access to the cluster is performed by logging into one of the **front-end nodes**. These nodes are intended for submitting jobs through **SLURM**, editing job submission scripts, and compiling applications. Compute nodes are accessible only through the **SLURM workload manager**. For the CRESCO7 cluster, the available front-end nodes are: .. code-block:: bash cresco7x001.portici.enea.it cresco7x002.portici.enea.it (also available for interactive applications) Access ------- To access the system via SSH, refer to the documentation at the page :doc:`Access to CRESCO System <../GENERAL_INFORMATION/access_to_systems>`. Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page: `Interface CRESCO-ONDEMAND `_ File Systems ------------ The file systems available on **CRESCO7** are: **AFS**: The geographically distributed file system shared across the entire **ENEAGRID** infrastructure. Information about the common **HOME directory structure** for all ENEAGRID/CRESCO users can be found in the section **File Systems and Data Management** available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`. **LUSTRE**: The high-performance parallel I/O file system. Information about **Lustre** for ENEAGRID/CRESCO users is available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`. Compilers and Numerical Libraries --------------------------------- The following table lists the compilers and numerical libraries available on **CRESCO7**: .. list-table:: :header-rows: 1 :widths: 20 16 16 16 16 16 * - - GNU (11.3.1) - Intel (17.0.4) - Intel (18.0.2) - Intel (OneAPI2023) - Intel (OneAPI2024) * - PATH compilers - /usr/bin - /opt/intel17/bin - /opt/intel18/bin - /afs/enea.it/software/intel_compiler/alma9_2/oneapi2023/compiler/2023.1.0/linux/bin/intel64 - /afs/enea.it/software/intel_compiler/alma9_2/oneapi2024/compiler/2024.0/bin/compiler * - PATH numerical libraries - | BLAS/LAPACK/ATLAS | /usr/lib64 | /usr/lib/atlas-sse* | /usr/lib64/atlas | /usr/lib64/atlas-sse* - | Intel MKL | /opt/intel17/mkl/lib/intel64 | - | Intel MKL | /opt/intel18/mkl/lib/intel64 | - | Intel MKL | /afs/enea.it/software/intel_compiler/alma9_2/oneapi2023/mkl/latest/lib/intel64 | | *icc and icpc are no longer available* | - | Intel MKL | /afs/enea.it/software/intel_compiler/alma9_2/oneapi2024/compiler/2024.0/lib | | *icc and icpc are no longer available* MPI Flavours Available ---------------------- The MPI flavours available on the system, grouped by compiler, are listed in the following table. .. list-table:: :header-rows: 1 :widths: 20 26 27 27 * - - GNU (version 11.3.1) - Intel (version 17.0.4/18.0.2) - Intel (version OneAPI2023) * - OpenMPI - | openmpi_gcc-1.10.4-hfi | openmpi_gcc-4.1.5 | openmpi_gcc-4.1.6 - No flavour available - openmpi_intel-4.1.5 * - Intel MPI - No flavour available - IntelMPI - IntelMPI Job Submission -------------- The scheduler that manages the resources of **CRESCO7** and allows the submission of **batch jobs** is **SLURM**. :doc:`Here <../TUTORIAL/Scheduler_and_Job_Submission>` you can find the full documentation. Below, the SLURM partition defined for **CRESCO7** is reported together with some examples of submission wrappers for **serial jobs** and **parallel jobs**, valid for both **OpenMPI** and **IntelMPI**. Examples of submission scripts for **hybrid jobs (MPI + OpenMP)** are also provided. SLURM Partition on CRESCO7 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 26 * - PARTITION - DESCRIPTION * - **cresco7** - | Main partition with **144 compute nodes** equipped with Intel CPUs. | Both serial and parallel jobs can be submitted with a maximum running time of **24 hours**. .. note:: On the **cresco8_cpu** partition, several **Quality of Service (QoS)** configurations are defined, allowing users to utilize up to **250 compute nodes**. Job priority is managed through a **fairshare mechanism**, which lowers the priority of jobs submitted by users who have already heavily used the system. Job Script Examples ~~~~~~~~~~~~~~~~~~~~ **Serial Job Example** In the following example the job uses **one core on a single node**. .. code-block:: bash #!/bin/sh #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --partition=cresco7 #SBATCH --job-name=serialJOB #SBATCH --auks=yes #SBATCH --err=%j.err #SBATCH --out=%j.out #SBATCH --account=your_account /path/to/exe **MPI Job Example** In this example the job uses **20 nodes**, launching **48 MPI processes per node**. .. code-block:: bash #!/bin/bash #SBATCH --nodes=20 #SBATCH --ntasks-per-node=48 #SBATCH --partition=cresco7 #SBATCH --job-name=jobMPI #SBATCH --auks=yes #SBATCH --err=%j.err #SBATCH --out=%j.out #SBATCH --account=your_account mpirun /path/to/exe **Hybrid Job Example (MPI + OpenMP)** In this example the job uses **20 nodes**, launching **2 MPI processes per node** (one per socket). Each MPI process spawns **24 OpenMP threads**. .. code-block:: bash #!/bin/bash #SBATCH --nodes=20 #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=24 #SBATCH --ntasks-per-socket=1 #SBATCH --partition=cresco7 #SBATCH --job-name=jobMPI #SBATCH --auks=yes #SBATCH --err=%j.err #SBATCH --out=%j.out #SBATCH --account=your_account export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} mpirun --bind-to none /path/to/exe **Job Submission** All job types can be submitted using the following command: .. code-block:: bash sbatch submission_script .. note:: To access the **AFS filesystem** during batch job execution, the token is generated on the compute nodes by **SLURM daemons** through integration with **AUKS** and the use of an **AUKS helper script**. AUKS generates the **Kerberos ticket** on the remote nodes by retrieving the user credentials from the login node cache. The helper script included with AUKS then invokes ``aklog`` to generate the corresponding **AFS token** from the Kerberos ticket. For additional information go to the following :doc:`page <../TUTORIAL/afs>` **Interactive Session Examples** Console session .. code-block:: bash srun --account=enea --nodes=1 --ntasks-per-node=2 -p cresco7 --pty /bin/bash Graphic session .. code-block:: bash srun --x11 --account=enea --nodes=1 --ntasks-per-node=2 -p cresco7 octave |