CRESCO4F
Cluster Features
The CRESCO4F cluster located in Frascati is a computing system composed of 64 compute nodes.
Each node provides:
2 sockets with 8 cores Intel E5-2670 2.6 GHz processors and 64 GB RAM
4 GB per core, 8 instructions per clock cycle
1 × 500 GB SATA II disk
InfiniBand QDR interface (40 Gb/s, 32 Gb/s effective bandwidth)
Two GbE network interfaces
BMC/IPMI 1.8 support and software for remote console management
Overall the cluster provides 1024 CPU cores interconnected through a low-latency network based on InfiniBand 4×QDR (40 Gb/s) managed by a Qlogic 12800 switch.
The architecture of a CRESCO4F compute node is represented in the following scheme:
Front-end Nodes
Access to the cluster is performed by logging into the front-end node. The nodes are used for submitting jobs through LSF, editing job submission scripts and compiling applications.
cresco4fx001.frascati.enea.it
Access
To access the system via SSH, refer to the documentation at the page Access to CRESCO System.
Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page:
File Systems
The file systems available on CRESCO4F are:
AFS: The geographically distributed file system shared across the entire ENEAGRID infrastructure. Information about the common HOME directory structure for all ENEAGRID/CRESCO users can be found in the section File Systems and Data Management available here.
GPFS: The high-performance parallel I/O file system. Information aboutGPFS** for ENEAGRID/CRESCO users is available here.
Compilers and Numerical Libraries
The table below lists the compilers and numerical libraries available on CRESCO4F.
GNU (4.4.6) |
GNU (4.8.2) |
Intel (14.0.1) |
Intel (16.0.0) |
PGI (11.10-0) |
|
|---|---|---|---|---|---|
PATH compilers |
/usr/bin |
/opt/gcc482/bin |
/opt/intel/composer_xe_2013_sp1.1.106/bin/intel64 |
/opt/intel16/compilers_and_libraries_2016.0.109/linux/bin/intel64 |
/afs/enea.it/software/pgi_11.10/linux86-64/11.10/bin |
PATH numerical libraries |
BLAS/LAPACK/ATLAS
/usr/lib
/usr/lib64
/usr/lib/atlas-sse
/usr/lib64/atlas
/usr/lib64/atlas-sse
|
BLAS/LAPACK/ATLAS
/usr/lib
/usr/lib64
/usr/lib/atlas-sse
/usr/lib64/atlas
/usr/lib64/atlas-sse
|
Intel MKL
/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/ia32
/opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64
|
Intel MKL
/opt/intel16/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin
|
PGI per AMD/ACML e CUDA
/afs/enea.it/software/pgi_11.10/linux86-64/11.10/lib
/afs/enea.it/software/pgi_11.10/linux86-64/11.10/libso
|
MPI Flavours Available
On the CRESCO cluster in Frascati, the OFED stack version 1.5.4.1 is installed.
The MPI flavours available on the system are listed below.
Open64 (version 4.5.1) |
GNU (version 4.4.6 / 4.8.2) |
Intel (version 14.0.1) |
Intel (version 12.1.3) |
PGI (version 11.10-0) |
|
|---|---|---|---|---|---|
MVAPICH |
not available |
mvapich2_gcc_qlc-1.7 |
mvapich2_intel14_qlc-1.7 |
mvapich2_intel12_qlc-1.7 |
mvapich2_pgi_qlc-1.7 |
OpenMPI |
openmpi_open64-1.4.3
openmpi_open64-1.6.5
|
openmpi_gcc-1.4.3
openmpi_gcc-1.6.5
openmpi_gcc482-1.4.3
openmpi_gcc482-1.6.5
|
openmpi_intel14-1.4.3-qlc
openmpi_intel14-1.6.5
|
openmpi_intel12-1.5.4-qlc |
openmpi_pgi-1.4.3-qlc
openmpi_pgi-1.6.5
|
Intel MPI |
No flavour available |
No flavour available |
impi-4.1.2 |
No flavour available |
No flavour available |
Job Submission
The scheduler that manages the resources of CRESCO4F and allows the submission of jobs is LSF. Below the are the LSF queues defined for CRESCO4F together with some examples of submission wrappers for serial and parallel jobs.
LSF Queues on CRESCO4F
PARTITION |
DESCRIPTION |
|---|---|
cresco4f_h6 |
Queue for serial and parallel jobs requesting up to 256 cores.
Maximum running time: 6 hours. Open to all users for testing purposes.
|
cresco4f_16h24 |
Queue for parallel jobs requesting greater than 257 cores.
Core allocation uses a granularity of 16 cores. Maximum running time: 24 hours.
|
cresco4f_h144 |
Queue dedicated to serial jobs. Maximum number of simultaneous jobs: 96.
Maximum running time: 144 hours.
|
Job Script Examples
OpenMPI 1.4.3 (GCC, Intel, PGI)
#!/bin/sh
exe=/path/to/exe
HOSTFILE=$LSB_DJOB_HOSTFILE
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l`
mpirun --mca plm_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe
OpenMPI 1.6.5 (GCC, Intel, PGI)
#!/bin/sh
exe=/path/to/exe
HOSTFILE=$LSB_DJOB_HOSTFILE
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l`
mpirun --mca orte_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe
MVAPICH2 (GCC, Intel, PGI)
#!/bin/sh
exe=/path/to/exe
HOSTFILE=$LSB_DJOB_HOSTFILE
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l`
mpirun_rsh -rsh -np $N_procs -hostfile $HOSTFILE $exe
Intel MPI 4.1.2
#!/bin/sh
exe=/path/to/exe
HOSTFILE=$LSB_DJOB_HOSTFILE
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l`
mpirun -bootstrap rsh -n $N_procs -machinefile $HOSTFILE $exe