National Computational Infrastructure
NCI National Facility
SGI XE Cluster SGI XE Cluster System Details
Image of Linux Cluster

SGI XE Cluster Hardware Specifications of the SGI XE Cluster:
  • 156 SGI Altix XE 320 nodes, each containing:
    • two quad-core 3.0GHz Intel Harpertown cpus with 1600MHz front side bus
    • L1 cache (on chip): 12 Kb(I) + 8 kbytes(D)
      L2 cache (on chip): 6MB per quad-core cpu
    • 16Gbyte DDR2-800 memory (a few nodes have 32Gbyte)
    • a range of 250GB, 500GB and 1TB SATA disks for swap and job scratch
    • on-board DDR InfiniBand adapter

  • Aggregate Specfprate_base_2006 of (compute nodes only) 12168 (AC is around 20000)
    Peak theoretical performance of almost 15 TFlops.

  • 32 Fermi GPUs (within Telsla s2050), each containing:
    • 488 CUDA cores in 14 stream multi-processors (SM), each SM containing
      • 32 CUDA cores running at 1.15GHz
      • 64Kbytes configurable L1/shared memory.
    • 2.6 Gbytes GDDR5 memory (ECC enabled)
    • 786Kbytes L2 cache
    Increased total system performance: 16 TFlops.

  • Total of 2.7TB of RAM on compute nodes

  • 18 Quad-core SGI Altix XE 210 servers for Lustre fileserving

  • Approx 54 TBytes of global user storage using 8 SGI InfiniteStorage 120 JBOD trays each with 12x 750GByte Seagate SATA drives

  • Approx 20 TBytes user scratch space for running jobs using 20x SGI TP9100 JBOD trays each with 16x 73GByte Fibre Channel drives

  • Total of over 130 TBytes of disk

  • 192-port Voltaire ISR2012 InfiniBand switch for both MPI and Lustre filesystem traffic:
    Measured MPI Latency: 3.4us
    Measured MPI Bandwidth: > 1200MB/s per node
SGI XE Cluster Software The system software used on the XE cluster includes:
  • CentOS 5.2 Linux distribution (based on RHEL5.2)

  • the oneSIS cluster software management system

  • the Lustre cluster file system:
    • 8 x (8+2 RAID6 6TB OST) for /short
    • 8 x (1+1 RAID1 520GB OST) for /home
    • 8 x (1+1 RAID1 140GB OST) for /apps

  • the National Facility variant of the OpenPBS batch queuing system

        
Image of Linux Cluster

Image of Linux Cluster

Email problems, suggestions, questions to