CRESCO4C
Cluster Features
The CRESCO4C cluster located in Casaccia is a computing system composed of 32 compute nodes.
Each node provides:
2 sockets with 8 cores Intel E5-2670 2.6 GHz processors and 64 GB RAM
4 GB per core, 8 instructions per clock cycle
1 × 500 GB SATA II disk
InfiniBand QDR interface (40 Gb/s, 32 Gb/s effective bandwidth)
Two GbE network interfaces
BMC/IPMI 1.8 support and software for remote console management
Overall the cluster provides 512 CPU cores interconnected through a low-latency, high-bandwidth network based on InfiniBand 4×QDR (40 Gb/s) managed by a Qlogic 12800 switch.
The architecture of a CRESCO4C compute node is represented in the following scheme:
Front-end Nodes
Access to the cluster is performed by logging into the front-end node. The nodes are used for submitting jobs through LSF, editing job submission scripts and compiling applications.
cresco4cx001.frascati.enea.it
Access
To access the system via SSH, refer to the documentation at the page Access to CRESCO System.
Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page:
File Systems
The file systems available on CRESCO4F are:
AFS: The geographically distributed file system shared across the entire ENEAGRID infrastructure. Information about the common HOME directory structure for all ENEAGRID/CRESCO users can be found in the section File Systems and Data Management available here.
GPFS: The high-performance parallel I/O file system. Information aboutGPFS** for ENEAGRID/CRESCO users is available here.
Note
The GPFS file system available on CRESCO4C nodes is:
/gcasi0 (~33 TB)
Compilers
The table below lists the compilers available on CRESCO4C.
GNU (4.8.5) |
GNU (7.3.0) |
Intel (17.0.4) |
Intel (18.0.2) |
PGI (19.7-0) |
|
|---|---|---|---|---|---|
PATH compilers |
/usr/bin |
/opt/gcc730/bin |
/opt/intel/bin |
/opt/intel18/bin |
/afs/enea.it/software/pgi_compiler/pgi_19_7/build/linux86-64/19.7/bin |
MPI Flavours Available
The MPI flavours available on the system, grouped by compiler, are listed in the following table.
GNU (version 4.8.5 / 7.3.0) |
Intel (version 17.0.4) |
PGI (version 19.7-0) |
|
|---|---|---|---|
OpenMPI |
openmpi_gcc-4.0.5 |
openmpi_intel-4.0.5 |
openmpi_pgi-4.0.5 |
Intel MPI |
No flavour available |
IntelMPI |
No flavour available |
Job Submission
The scheduler that manages the resources of CRESCO4C and allows the submission of jobs is LSF. Below the are the LSF queues defined for CRESCO4C together with some examples of submission wrappers for serial and parallel jobs.
LSF Queues on CRESCO4C
PARTITION |
DESCRIPTION |
|---|---|
cresco4c_h144 |
Queue for serial and parallel jobs requiring up to 32 cores.
Maximum running time: 144 hours. Open to all users.
|
cresco4c_16h24 |
Queue for parallel jobs requiring greater than 16 cores.
Maximum running time for 24 hours.
|
Note
On the cresco4c_16h24 queue, the number of requested cores must be a multiple of 16, corresponding to the number of cores available on each compute node.
Job Script Examples
OpenMPI 4.0.5 (GCC, Intel, PGI)
#!/bin/sh
exe=/path/to/exe # path of your MPI program
HOSTFILE=$LSB_DJOB_HOSTFILE # name of hostfile for mpirun
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # give to mpirun same number of slots
mpirun --mca plm_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe
Intel MPI 4.1.2
#!/bin/sh
exe=/path/to/exe # path of your MPI program
HOSTFILE=$LSB_DJOB_HOSTFILE # name of hostfile for mpirun
N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` # give to mpirun same number of slots
mpirun -bootstrap rsh -n $N_procs -machinefile $HOSTFILE $exe
Job Submission Examples
bsub -o output.o -e error.e -q cresco4c_h144 -n number_of_cores ./submission_script
bsub -o output.o -e error.e -q cresco4c_16h24 -n number_of_cores ./submission_script