CRESCO6
Warning
The support/ticketing services and the backup services on the GPFS filesystem related to the CRESCO 6 cluster ended on September 30, 2025; The user GPFS scratch area of CRESCO 6 (/gporq2/scratch_0/usr/<username>) is no longer subject to synchronization or transfer to the new LUSTRE filesystem*; The CRESCO 6 cluster remains available for use, subject to the limitations indicated above.
Note
Users interested in migrating to the LUSTRE filesystem of CRESCO 8 are advised to copy the files currently stored in the scratch area to their GPFS store area on CRESCO 6, in compliance with the available quota limits.
Cluster Characteristics
The CRESCO6 cluster located in Portici is a computing system consisting of 434 nodes. Each node is equipped with:
2 sockets, each with 24 cores, featuring Intel® Xeon® Platinum 8160 processors with a clock frequency of 2.10 GHz and 192 GB of RAM.
One 100 Gb/s Intel Omni-Path interface.
Two Gigabit Ethernet (GbE) interfaces.
BMC/IPMI 1.8 support and software for remote console management
A total of 20,832 cores are available, interconnected through a high-bandwidth, low-latency network based on 100 Gb/s Intel Omni-Path technology.
The architecture of a CRESCO6 node can be summarized in the following diagram:
Front-End Nodes
Access to the cluster is performed by logging into one of the front-end nodes. The front-end nodes are intended solely for submitting jobs through LSF, editing job submission scripts, or compiling applications. Compute nodes can only be accessed through LSF.
For the CRESCO6 cluster, the following front-end nodes are available:
cresco6x001.portici.enea.it
cresco6x002.portici.enea.it (also available for interactive applications)
Access
To access the system via SSH, refer to the documentation at the page Access to CRESCO System.
Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page:
File Systems
The file systems available on CRESCO6 are:
AFS: The geographically distributed file system shared across the entire ENEAGRID infrastructure. Information about the common HOME directory structure for all ENEAGRID/CRESCO users can be found in the section File Systems and Data Management available here.
GPFS: The high-performance parallel I/O file system. Information aboutGPFS** for ENEAGRID/CRESCO users is available here.
Compilers and Numerical Libraries
The table below lists the compilers and numerical libraries available on CRESCO6.
GNU (4.8.5) |
GNU (7.3.0) |
Intel (17.0.4) |
Intel (18.0.2) |
PGI (11.10-0) |
|
|---|---|---|---|---|---|
Compiler PATH |
/usr/bin |
/opt/gcc730/bin |
/opt/intel/bin |
/opt/intel18/bin |
/afs/enea.it/software/pgi_11.10/linux86-64/11.10/bin |
Numerical Libraries PATH |
BLAS/LAPACK/ATLAS
/usr/lib
/usr/lib64
/usr/lib/atlas-sse*
/usr/lib64/atlas
/usr/lib64/atlas-sse*
|
BLAS/LAPACK/ATLAS
/usr/lib
/usr/lib64
/usr/lib/atlas-sse*
/usr/lib64/atlas
/usr/lib64/atlas-sse*
|
Intel MKL
/opt/intel/mkl/lib/intel64
|
Intel MKL
/opt/intel18/mkl/lib/intel64
|
PGI for AMD/ACML and CUDA
/afs/enea.it/software/pgi_11.10/linux86-64/11.10/lib
/afs/enea.it/software/pgi_11.10/linux86-64/11.10/libso
|
Available MPI Flavours
The available MPI flavours, grouped by compiler, are listed in the following table:
GNU (version 4.8.5 / 7.3.0) |
Intel (version 17.0.4 / 18.0.2) |
PGI (version 11.10-0) |
|
|---|---|---|---|
MVAPICH |
mvapich2.2.2_gcc_hfi |
mvapich2.2.2_intel-hfi |
mvapich2.2.2_pgi-hfi |
OpenMPI |
openmpi_gcc-1.10.4-hfi
openmpi_gcc-3.1.1
openmpi_gcc730-3.1.1
|
openmpi_intel-1.10.4-hfi
openmpi_intel-3.1.1
|
openmpi_pgi-1.10.4-hfi
openmpi_pgi-3.1.1
|
Intel MPI |
None |
IntelMPI |
None |
Job Submission
The scheduler that manages the resources of CRESCO6 and allows the submission of batch jobs is LSF.
Below are the submission queues defined for this cluster, along with some examples of submission wrappers for OpenMPI, MVAPICH, and IntelMPI.
LSF Queues on CRESCO6
PARTITION |
DESCRIPTION |
|---|---|
cresco6_48h24 |
Queue for parallel jobs requiring at least 48 cores.
Maximum running time: 24 hours. Open to all users for testing purposes.
|
Note
It accepts jobs requesting a number of cores that is a multiple of 48. A fairshare policy is enforced on this queue, reducing the priority of jobs submitted by users who have already made intensive use of the system.
OpenMPI (GCC, Intel, PGI)
#!/bin/sh
exe=/path/to/exe # path of your MPI program
HOSTFILE=$LSB_DJOB_HOSTFILE # name of hostfile for mpirun
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # give to mpirun same number of slots
mpirun --mca plm_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe
MVAPICH (GCC, Intel, PGI)
#!/bin/sh
exe=/path/to/exe # path of your MPI program
HOSTFILE=$LSB_DJOB_HOSTFILE # name of hostfile for mpirun
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # give to mpirun same number of slots
mpirun_rsh -rsh -np $N_procs -hostfile $HOSTFILE $exe
Intel MPI
#!/bin/sh
exe=/path/to/exe # path of your MPI program
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # number of slots
host_numb=`echo "$N_procs/48" | bc` # number of hosts to involve
export I_MPI_HYDRA_BRANCH_COUNT=$host_numb
mpirun -bootstrap rsh $exe
Submission Examples
bsub -o output.o -e error.e -q cresco6_48h24 -n <number_of_cores> ./submission_script
(Parallel jobs with <number_of_cores> ≥ 48)
bsub -o output.o -e error.e -q cresco6_48h48 -n <number_of_cores> ./submission_script
(Parallel jobs with <number_of_cores> ≥ 48)
Note
Please note that on the cresco6_48h24 queue, in order for a job to be accepted, the number of requested cores must be a multiple of 48 (the number of cores per compute node), up to a maximum of 6144 cores (128 nodes). On the cresco6_48h48 queue, jobs must also be submitted in multiples of 48, up to a maximum of 3072 cores (64 nodes). It should be noted that 3072 cores also represent the total number of cores available system-wide for all users on this queue at any given time.