CRESCO7
Warning
The CRESCO7 cluster does not guarantee backup, it is recommended to use only in development and testing.
Cluster Characteristics
The CRESCO7 cluster located in Portici is a computing system composed of 144 compute nodes. Each node is equipped with:
2 sockets, each with 24 cores, based on Intel(R) Xeon(R) Platinum 8160 processors running at 2.10 GHz
192 GB of RAM
One Mellanox EDR interface
Two Gigabit Ethernet (GbE) interfaces
BMC/IPMI support and software for remote console management
The system provides a total of 6,912 CPU cores, interconnected through a high-bandwidth, low-latency network based on Mellanox EDR at 100 Gb/s.
The architecture of a CRESCO7 compute node is illustrated in the following diagram:
Front-end Nodes
Access to the cluster is performed by logging into one of the front-end nodes. These nodes are intended for submitting jobs through SLURM, editing job submission scripts, and compiling applications.
Compute nodes are accessible only through the SLURM workload manager.
For the CRESCO7 cluster, the available front-end nodes are:
cresco7x001.portici.enea.it
cresco7x002.portici.enea.it (also available for interactive applications)
Access
To access the system via SSH, refer to the documentation at the page Access to CRESCO System.
Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page:
File Systems
The file systems available on CRESCO7 are:
AFS: The geographically distributed file system shared across the entire ENEAGRID infrastructure. Information about the common HOME directory structure for all ENEAGRID/CRESCO users can be found in the section File Systems and Data Management available here.
LUSTRE: The high-performance parallel I/O file system. Information about Lustre for ENEAGRID/CRESCO users is available here.
Compilers and Numerical Libraries
The following table lists the compilers and numerical libraries available on CRESCO7:
GNU (11.3.1) |
Intel (17.0.4) |
Intel (18.0.2) |
Intel (OneAPI2023) |
Intel (OneAPI2024) |
|
|---|---|---|---|---|---|
PATH compilers |
/usr/bin |
/opt/intel17/bin |
/opt/intel18/bin |
/afs/enea.it/software/intel_compiler/alma9_2/oneapi2023/compiler/2023.1.0/linux/bin/intel64 |
/afs/enea.it/software/intel_compiler/alma9_2/oneapi2024/compiler/2024.0/bin/compiler |
PATH numerical libraries |
BLAS/LAPACK/ATLAS
/usr/lib64
/usr/lib/atlas-sse*
/usr/lib64/atlas
/usr/lib64/atlas-sse*
|
Intel MKL
/opt/intel17/mkl/lib/intel64
|
Intel MKL
/opt/intel18/mkl/lib/intel64
|
Intel MKL
/afs/enea.it/software/intel_compiler/alma9_2/oneapi2023/mkl/latest/lib/intel64
icc and icpc are no longer available
|
Intel MKL
/afs/enea.it/software/intel_compiler/alma9_2/oneapi2024/compiler/2024.0/lib
icc and icpc are no longer available
|
MPI Flavours Available
The MPI flavours available on the system, grouped by compiler, are listed in the following table.
GNU (version 11.3.1) |
Intel (version 17.0.4/18.0.2) |
Intel (version OneAPI2023) |
|
|---|---|---|---|
OpenMPI |
openmpi_gcc-1.10.4-hfi
openmpi_gcc-4.1.5
openmpi_gcc-4.1.6
|
No flavour available |
openmpi_intel-4.1.5 |
Intel MPI |
No flavour available |
IntelMPI |
IntelMPI |
Job Submission
The scheduler that manages the resources of CRESCO7 and allows the submission of batch jobs is SLURM. Here you can find the full documentation.
Below, the SLURM partition defined for CRESCO7 is reported together with some examples of submission wrappers for serial jobs and parallel jobs, valid for both OpenMPI and IntelMPI. Examples of submission scripts for hybrid jobs (MPI + OpenMP) are also provided.
SLURM Partition on CRESCO7
PARTITION |
DESCRIPTION |
|---|---|
cresco7 |
Main partition with 144 compute nodes equipped with Intel CPUs.
Both serial and parallel jobs can be submitted with a maximum running time of 24 hours.
|
Note
On the cresco8_cpu partition, several Quality of Service (QoS) configurations are defined, allowing users to utilize up to 250 compute nodes. Job priority is managed through a fairshare mechanism, which lowers the priority of jobs submitted by users who have already heavily used the system.
Job Script Examples
Serial Job Example
In the following example the job uses one core on a single node.
#!/bin/sh
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --partition=cresco7
#SBATCH --job-name=serialJOB
#SBATCH --auks=yes
#SBATCH --err=%j.err
#SBATCH --out=%j.out
#SBATCH --account=your_account
/path/to/exe
MPI Job Example
In this example the job uses 20 nodes, launching 48 MPI processes per node.
#!/bin/bash
#SBATCH --nodes=20
#SBATCH --ntasks-per-node=48
#SBATCH --partition=cresco7
#SBATCH --job-name=jobMPI
#SBATCH --auks=yes
#SBATCH --err=%j.err
#SBATCH --out=%j.out
#SBATCH --account=your_account
mpirun /path/to/exe
Hybrid Job Example (MPI + OpenMP)
In this example the job uses 20 nodes, launching 2 MPI processes per node (one per socket). Each MPI process spawns 24 OpenMP threads.
#!/bin/bash
#SBATCH --nodes=20
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=24
#SBATCH --ntasks-per-socket=1
#SBATCH --partition=cresco7
#SBATCH --job-name=jobMPI
#SBATCH --auks=yes
#SBATCH --err=%j.err
#SBATCH --out=%j.out
#SBATCH --account=your_account
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
mpirun --bind-to none /path/to/exe
Job Submission
All job types can be submitted using the following command:
sbatch submission_script
Note
To access the AFS filesystem during batch job execution, the token is generated on the compute nodes by SLURM daemons through integration with AUKS and the use of an AUKS helper script. AUKS generates the Kerberos ticket on the remote nodes by retrieving the user credentials from the login node cache. The helper script included with AUKS then invokes aklog to generate the corresponding AFS token from the Kerberos ticket. For additional information go to the following page
Interactive Session Examples
Console session
srun --account=enea --nodes=1 --ntasks-per-node=2 -p cresco7 --pty /bin/bash
Graphic session
srun --x11 --account=enea --nodes=1 --ntasks-per-node=2 -p cresco7 octave