National Computational Infrastructure
NCI National Facility
Sun Constellation cluster, vayu: System Details
Image of vayu cluster goes here

Sun Constellation Cluster Hardware Specifications of the interim (Sep-Dec 2009) Sun Constellation Cluster:
  • a high density integrated system

  • 192 nodes in Sun X6275 blades, each containing:
    • two quad-core 2.93GHz Intel Nehalem cpus with 6.4GTs QPI bus
    • L1 cache (on chip): 32KB (I) + 32KB (D)
      L2 cache (on chip): 256KB
      L3 cache (on chip): 8MB per quad-core cpu
    • 24Gbyte DDR3-1333 memory
    • 24GB Flash DIMM for swap and some job scratch
    • on-board QDR InfiniBand adapter

  • Aggregate Specfprate_base_2006 of (compute nodes only) 32000 (AC was around 20000)
    Peak theoretical performance of approximately 18TFlops.

  • Total of 4.6TB of RAM on compute nodes

  • Dual socket, quad-core Sun X4170 servers for Lustre fileserving

  • Approx 100 TBytes of global user storage using 8 Sun J4400 JBOD trays each with 24 1TB Seagate Enterprise SATA drives

  • 288 ports of a QDR Sun Datacenter InfiniBand Switch 648 for both MPI and Lustre filesystem traffic:
    Measured MPI Latency: 2.0us
    Measured MPI Bandwidth: > 2400MB/s per node
Sun Constellation Cluster Software The system software used on the vayu cluster includes:
  • CentOS 5.3 Linux distribution (based on RHEL5.3)

  • the oneSIS cluster software management system

  • the Lustre cluster file system:
    • 16 x (8+2 RAID6 8TB OST) for /short
    • 16 x (1+1 RAID1 520GB OST) for /home
    • 16 x (1+1 RAID1 140GB OST) for /apps

  • the National Facility's variant of the OpenPBS batch queuing system

        
Image of VU IB switch

Image of VU disks

Email problems, suggestions, questions to