CRESCO6 ================= .. warning:: The support/ticketing services and the backup services on the GPFS filesystem related to the CRESCO 6 cluster ended on September 30, 2025; The user GPFS scratch area of CRESCO 6 (/gporq2/scratch_0/usr/) is no longer subject to synchronization or transfer to the new LUSTRE filesystem*; The CRESCO 6 cluster remains available for use, subject to the limitations indicated above. .. note:: Users interested in migrating to the LUSTRE filesystem of CRESCO 8 are advised to copy the files currently stored in the scratch area to their GPFS store area on CRESCO 6, in compliance with the available quota limits. Cluster Characteristics ------------------------ The CRESCO6 cluster located in Portici is a computing system consisting of 434 nodes. Each node is equipped with: - 2 sockets, each with 24 cores, featuring Intel® Xeon® Platinum 8160 processors with a clock frequency of 2.10 GHz and 192 GB of RAM. - One 100 Gb/s Intel Omni-Path interface. - Two Gigabit Ethernet (GbE) interfaces. - BMC/IPMI 1.8 support and software for remote console management A total of 20,832 cores are available, interconnected through a high-bandwidth, low-latency network based on 100 Gb/s Intel Omni-Path technology. The architecture of a CRESCO6 node can be summarized in the following diagram: .. image:: clusters/cresco6_image1.png :align: center :width: 1600px | Front-End Nodes ------------------------ Access to the cluster is performed by logging into one of the front-end nodes. The front-end nodes are intended solely for submitting jobs through LSF, editing job submission scripts, or compiling applications. Compute nodes can only be accessed through LSF. For the CRESCO6 cluster, the following front-end nodes are available: .. code-block:: bash cresco6x001.portici.enea.it cresco6x002.portici.enea.it (also available for interactive applications) Access ------- To access the system via SSH, refer to the documentation at the page :doc:`Access to CRESCO System <../GENERAL_INFORMATION/access_to_systems>`. Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page: `Interface CRESCO-ONDEMAND `_ File Systems ------------- The file systems available on **CRESCO6** are: **AFS**: The geographically distributed file system shared across the entire **ENEAGRID** infrastructure. Information about the common **HOME directory structure** for all ENEAGRID/CRESCO users can be found in the section **File Systems and Data Management** available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`. **GPFS**: The high-performance parallel I/O file system. Information aboutGPFS** for ENEAGRID/CRESCO users is available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`. Compilers and Numerical Libraries ---------------------------------- The table below lists the compilers and numerical libraries available on CRESCO6. .. list-table:: Compilers and Numerical Libraries on CRESCO6 :header-rows: 1 :widths: 20 20 20 20 20 20 * - - GNU (4.8.5) - GNU (7.3.0) - Intel (17.0.4) - Intel (18.0.2) - PGI (11.10-0) * - Compiler PATH - /usr/bin - /opt/gcc730/bin - /opt/intel/bin - /opt/intel18/bin - /afs/enea.it/software/pgi_11.10/linux86-64/11.10/bin * - Numerical Libraries PATH - | BLAS/LAPACK/ATLAS | /usr/lib | /usr/lib64 | /usr/lib/atlas-sse* | /usr/lib64/atlas | /usr/lib64/atlas-sse* - | BLAS/LAPACK/ATLAS | /usr/lib | /usr/lib64 | /usr/lib/atlas-sse* | /usr/lib64/atlas | /usr/lib64/atlas-sse* - | Intel MKL | /opt/intel/mkl/lib/intel64 - | Intel MKL | /opt/intel18/mkl/lib/intel64 - | PGI for AMD/ACML and CUDA | /afs/enea.it/software/pgi_11.10/linux86-64/11.10/lib | /afs/enea.it/software/pgi_11.10/linux86-64/11.10/libso Available MPI Flavours ------------------------ The available MPI flavours, grouped by compiler, are listed in the following table: .. list-table:: MPI Flavours Available on CRESCO6 :header-rows: 1 :widths: 20 25 25 25 * - - GNU (version 4.8.5 / 7.3.0) - Intel (version 17.0.4 / 18.0.2) - PGI (version 11.10-0) * - MVAPICH - mvapich2.2.2_gcc_hfi - mvapich2.2.2_intel-hfi - mvapich2.2.2_pgi-hfi * - OpenMPI - | openmpi_gcc-1.10.4-hfi | openmpi_gcc-3.1.1 | openmpi_gcc730-3.1.1 - | openmpi_intel-1.10.4-hfi | openmpi_intel-3.1.1 - | openmpi_pgi-1.10.4-hfi | openmpi_pgi-3.1.1 * - Intel MPI - None - IntelMPI - None Job Submission ---------------------- The scheduler that manages the resources of **CRESCO6** and allows the submission of batch jobs is **LSF**. Below are the submission queues defined for this cluster, along with some examples of submission wrappers for OpenMPI, MVAPICH, and IntelMPI. LSF Queues on CRESCO6 ~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 26 * - PARTITION - DESCRIPTION * - **cresco6_48h24** - | Queue for parallel jobs requiring at least **48 cores**. | Maximum running time: **24 hours**. Open to all users for testing purposes. .. note:: It accepts jobs requesting a number of cores that is a multiple of 48. A fairshare policy is enforced on this queue, reducing the priority of jobs submitted by users who have already made intensive use of the system. **OpenMPI (GCC, Intel, PGI)** .. code-block:: bash #!/bin/sh exe=/path/to/exe # path of your MPI program HOSTFILE=$LSB_DJOB_HOSTFILE # name of hostfile for mpirun N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # give to mpirun same number of slots mpirun --mca plm_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe **MVAPICH (GCC, Intel, PGI)** .. code-block:: bash #!/bin/sh exe=/path/to/exe # path of your MPI program HOSTFILE=$LSB_DJOB_HOSTFILE # name of hostfile for mpirun N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # give to mpirun same number of slots mpirun_rsh -rsh -np $N_procs -hostfile $HOSTFILE $exe **Intel MPI** .. code-block:: bash #!/bin/sh exe=/path/to/exe # path of your MPI program N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # number of slots host_numb=`echo "$N_procs/48" | bc` # number of hosts to involve export I_MPI_HYDRA_BRANCH_COUNT=$host_numb mpirun -bootstrap rsh $exe **Submission Examples** .. code-block:: bash bsub -o output.o -e error.e -q cresco6_48h24 -n ./submission_script (Parallel jobs with ≥ 48) .. code-block:: bash bsub -o output.o -e error.e -q cresco6_48h48 -n ./submission_script (Parallel jobs with ≥ 48) .. note:: Please note that on the cresco6_48h24 queue, in order for a job to be accepted, the number of requested cores must be a multiple of 48 (the number of cores per compute node), up to a maximum of 6144 cores (128 nodes). On the cresco6_48h48 queue, jobs must also be submitted in multiples of 48, up to a maximum of 3072 cores (64 nodes). It should be noted that 3072 cores also represent the total number of cores available system-wide for all users on this queue at any given time.