CRESCO4F ================ Cluster Features ---------------- The **CRESCO4F cluster** located in **Frascati** is a computing system composed of **64 compute nodes**. Each node provides: - **2 sockets with 8 cores** Intel **E5-2670 2.6 GHz** processors and **64 GB RAM** - **4 GB per core**, **8 instructions per clock cycle** - **1 × 500 GB SATA II disk** - **InfiniBand QDR interface (40 Gb/s, 32 Gb/s effective bandwidth)** - **Two GbE network interfaces** - **BMC/IPMI 1.8 support** and software for **remote console management** Overall the cluster provides **1024 CPU cores** interconnected through a **low-latency network based on InfiniBand 4×QDR (40 Gb/s)** managed by a **Qlogic 12800 switch**. The architecture of a **CRESCO4F compute node** is represented in the following scheme: .. image:: clusters/cresco4f_image1.png :align: center :width: 1600px | Front-end Nodes --------------- Access to the cluster is performed by logging into the **front-end node**. The nodes are used for submitting jobs through **LSF**, editing job submission scripts and compiling applications. .. code-block:: cresco4fx001.frascati.enea.it Access ------ To access the system via SSH, refer to the documentation at the page :doc:`Access to CRESCO System <../GENERAL_INFORMATION/access_to_systems>`. Note that the access to the ENEAGRID front-end nodes can also be performed via the web interface available on the CRESCO Open OnDemand page: `Interface CRESCO-ONDEMAND `_ File Systems ------------ The file systems available on **CRESCO4F** are: **AFS**: The geographically distributed file system shared across the entire **ENEAGRID** infrastructure. Information about the common **HOME directory structure** for all ENEAGRID/CRESCO users can be found in the section **File Systems and Data Management** available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`. **GPFS**: The high-performance parallel I/O file system. Information aboutGPFS** for ENEAGRID/CRESCO users is available :doc:`here <../TUTORIAL/File_Systems_and_Data_Management>`. Compilers and Numerical Libraries --------------------------------- The table below lists the **compilers** and **numerical libraries** available on **CRESCO4F**. .. list-table:: :header-rows: 1 :widths: 18 18 18 22 22 18 * - - GNU (4.4.6) - GNU (4.8.2) - Intel (14.0.1) - Intel (16.0.0) - PGI (11.10-0) * - PATH compilers - /usr/bin - /opt/gcc482/bin - /opt/intel/composer_xe_2013_sp1.1.106/bin/intel64 - /opt/intel16/compilers_and_libraries_2016.0.109/linux/bin/intel64 - /afs/enea.it/software/pgi_11.10/linux86-64/11.10/bin * - PATH numerical libraries - | BLAS/LAPACK/ATLAS | /usr/lib | /usr/lib64 | /usr/lib/atlas-sse | /usr/lib64/atlas | /usr/lib64/atlas-sse - | BLAS/LAPACK/ATLAS | /usr/lib | /usr/lib64 | /usr/lib/atlas-sse | /usr/lib64/atlas | /usr/lib64/atlas-sse - | Intel MKL | /opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/ia32 | /opt/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64 - | Intel MKL | /opt/intel16/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin - | PGI per AMD/ACML e CUDA | /afs/enea.it/software/pgi_11.10/linux86-64/11.10/lib | /afs/enea.it/software/pgi_11.10/linux86-64/11.10/libso MPI Flavours Available ---------------------- On the **CRESCO cluster in Frascati**, the **OFED stack version 1.5.4.1** is installed. The MPI flavours available on the system are listed below. .. list-table:: :header-rows: 1 :widths: 18 22 22 22 22 22 * - - Open64 (version 4.5.1) - GNU (version 4.4.6 / 4.8.2) - Intel (version 14.0.1) - Intel (version 12.1.3) - PGI (version 11.10-0) * - MVAPICH - not available - mvapich2_gcc_qlc-1.7 - mvapich2_intel14_qlc-1.7 - mvapich2_intel12_qlc-1.7 - mvapich2_pgi_qlc-1.7 * - OpenMPI - | openmpi_open64-1.4.3 | openmpi_open64-1.6.5 - | openmpi_gcc-1.4.3 | openmpi_gcc-1.6.5 | openmpi_gcc482-1.4.3 | openmpi_gcc482-1.6.5 - | openmpi_intel14-1.4.3-qlc | openmpi_intel14-1.6.5 - openmpi_intel12-1.5.4-qlc - | openmpi_pgi-1.4.3-qlc | openmpi_pgi-1.6.5 * - Intel MPI - No flavour available - No flavour available - impi-4.1.2 - No flavour available - No flavour available Job Submission -------------- The scheduler that manages the resources of CRESCO4F and allows the submission of jobs is LSF. Below the are the LSF queues defined for CRESCO4F together with some examples of submission wrappers for serial and parallel jobs. LSF Queues on CRESCO4F ~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 20 26 * - PARTITION - DESCRIPTION * - **cresco4f_h6** - | Queue for serial and parallel jobs requesting up to **256 cores**. | Maximum running time: **6 hours**. Open to all users for testing purposes. * - **cresco4f_16h24** - | Queue for parallel jobs requesting greater than **257 cores**. | Core allocation uses a granularity of **16 cores**. Maximum running time: **24 hours**. * - **cresco4f_h144** - | Queue dedicated to serial jobs. Maximum number of simultaneous jobs: **96**. | Maximum running time: **144 hours**. Job Script Examples ~~~~~~~~~~~~~~~~~~~~~~ **OpenMPI 1.4.3 (GCC, Intel, PGI)** .. code-block:: bash #!/bin/sh exe=/path/to/exe HOSTFILE=$LSB_DJOB_HOSTFILE N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` mpirun --mca plm_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe **OpenMPI 1.6.5 (GCC, Intel, PGI)** .. code-block:: bash #!/bin/sh exe=/path/to/exe HOSTFILE=$LSB_DJOB_HOSTFILE N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` mpirun --mca orte_rsh_agent "blaunch.sh" -n $N_procs --hostfile $HOSTFILE $exe **MVAPICH2 (GCC, Intel, PGI)** .. code-block:: bash #!/bin/sh exe=/path/to/exe HOSTFILE=$LSB_DJOB_HOSTFILE N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` mpirun_rsh -rsh -np $N_procs -hostfile $HOSTFILE $exe **Intel MPI 4.1.2** .. code-block:: bash #!/bin/sh exe=/path/to/exe HOSTFILE=$LSB_DJOB_HOSTFILE N_procs=`cat $LSB_DJOB_HOSTFILE | wc -l` mpirun -bootstrap rsh -n $N_procs -machinefile $HOSTFILE $exe |