We don't have a user filled form for this submission. Instead we have a detailed decription of the site, the storage hardware and the compute clusters from personal conversations with the submitter. See the relevant description below.
Los Alamos National Lab (LANL) is a supercomputing site in New Mexico. We collected file statistics from this site on three different file systems labeled lanl-scratch1, lanl-scratch2 and lanl-scratch3.
All three are scratch space and use the Panasas Panfs file system. PanFS is a fault tolerant high performance cluster file system that supports per file RAID. The lanl-scratch space is a general purpose file system used mainly for I/O testing, but can be used as scratch space on preproduction clusters, i.e. consisting of user who were porting and testing their code. There is no active purge policy for this file system. It has predominately RAID5 files with 64 KB stripe size, but may have some RAID10 files on them. The storage nodes are organized as 7 shelves each with 10 OSDs (Object Store Device). A single OSD has 500 GB raw space created using two 7200 RPM 250 GB SATA drives.
The main compute cluster that uses lanl-scratch has 246 dual processor 1.6 Opteron nodes. The lanl-scratch2 and lanl-scratch3 are used by a different cluster called Flash/Gordon and the Yellow Rail system. These file system use RAID 5 with a stripe unit of 64KB. The lanl-scratch2 has 8 shelves of 800 GB OSDs and lanl-scratch3 has 16 shelves of 500GB OSDs. They're both purged periodically (60 days or older) or on a need basis (to keep capacity used below 90 \%) and users are encouraged to archive the files they want to keep. Though users can set RAID levels on a per file basis, in reality, very few do this.
NERSC (The National Energy Research Scientific Computing Center) is a scientific computing facility for the Office of Science in the US Department of Energy, located at Lawrence Berkeley National Laboratory in Berkeley, California.
We collected data from the NERSC Global File System (NGF) which provides shared storage
accessible from five supercomputers: Franklin, PDSF, Jacquard, Bassi, and DaVinci.
Franklin is a Cray XT4 with 9,660 dual PDSF is 275 nodes with a mixed collection of
dual x86 CPUs. Jacquard is a Linux Networx cluster of 356 dual Opteron nodes.
Bassi is an IBM p575 POWER5 cluster of 122 8-processor nodes. DaVinci is a SGI Altix 350
with 32 Itanium-2 processors. While all these supercomputers have local scratch disk space
of different sizes using different file system software, we had access to only the shared NGF file system.
NGF is a IBM GPFS file system with 96 volumes built on top of 2 Data Direct
Networks (DDN) S2A 9550 disk arrays and 4 IBM DS4500 disk arrays. The DDN disk arrays are
both configured as 8 data + 2 "parity" RAID-6 arrays; one has 16 RAID sets (each 8+2 disks)
using 250 GB 7 Krpm SATA disks and the other has 24 RAID sets using 300 GB 10 Krpm FC disks.
Two IBM DS4500 disk arrays each have 16 RAID sets each a RAID-5 array of 4 data + 1 parity disks
configured with 512 KB stripe units and using 250 GB 7 Krpm SATA disks. The other two IBM 4500
disk arrays each have 12 RAID sets each a RAID-5 array with 5 + 1 disks configured with 64 KB
stripe units and using 300 GB 10 Krpm FC disks. The result is about 132 TB usable space with
another 31 TB for RAID overhead (24\%). Files in NGF are not automatically archived and purged
except when a project becomes inactive.
The users of NGF are over 100 science projects funded through the Office of Science's SciDAC
and INCITE programs. Example projects include Interaction of Turbulence and Chemistry, Full
Vehicle Windnoise Simultion, Plasma Based Accelerators, Climate-Science Computation, X-Ray Free
Electron Lasers, Modeling the Earth System, Clouds in Global Clime, Thermahaline Circulation,
Magnetohydrodynamic Modeling, Quantum Simulations of Nanostructures, Turbulent Combustion, and Supernovae.
PNNL (Pacific Northwest National Laboratory), is a U.S. Department of Energy (DOE)
government research laboratory in Richland Washington.
We collected attribute data from three different file systems at PNNL, labeled as pnnl-dtemp, pnnl-nwfs and pnnl-home.
The pnnl-dtemp dataset is a general purpose global scratch space. Its purge policy deletes a file after 30 days since last use
but this policy is not currently enforced since free space is ample. The volume uses the Lustre file system.
Lustre, for this volume, runs over 32 storage servers using 64 logical LUNs. The disk array backend is a HP EVA3000 with FC
disks, configured with RAID-5 LUNs. The compute cluster attached to this volume is the mpp2 cluster of 946 dual Itanium HP RX2600
nodes used by projects in the Environmental Molecular Sciences Laboratory such as preteomics, chemistry, biology, and biochemistry.
The pnnl-home is home directories for scientists who have projects approved to run on the mpp2 cluster. Files in both of these
volumes are manually archived to the pnnl-nwfs volume on a need basis. The pnnl-nwfs volume is backed by 70 storage servers each
with 2 to 4 RAID-5 LUNs built by a 3ware 9000 series controller. These LUNs use a mix of 400GB, 750GB, and 1TB drives, all 7Krpm SATA
drives.
ARSC (Arctic Region Supercomputing Center) is part of the DoD (Department of Defense)
and the university of Alaska Fairbanks for high performance computing. Its goal is to
support computational research in science and engineering with emphasis on high latitudes
and the arctic.
We collected file systems statistics from four major file systems here,
labeled: arsc-sea-u1, arsc-sea-u2, arsc-nan-u1 and arsc-nan-projects. The volume arsc-sea-u1
is for academic user archival storage on the DoD systems, arsc-sea-u2 is for DoD user archival
storage on the DoD systems, arsc-nan-u1 is for academic user archival storage on academic (non-DoD)
systems and arsc-nan-projects is for academic user archival storage who have requested actual
shared project space, e.g. National Weather Service, Bureau of Land Management etc.
Volumes arsc-sea-u1, arsc-sea-u2
and arsc-nan-u1 usually contain data archived from scratch but may come from anywhere. Client mount
these file systems using the SUN SAM-QFS file system. SAM-QFS can hide the file's real
storage (disk or tape) from the user and so a stat on a file on tape can return a large file size (EOF)
but no capacity used. SAM-QFS inodes also track multiple copies of a file existing on various disk and
tape systems and a stat returns a sum of this information. The arsc-nan-projects is used for shared (multiple owner)
projects, not much file system information was available to us about arsc-projects. Volumes
arsc-sea-u1, arsc-sea-u2, and arsc-nan-u1 use two storage servers, seawolf and nanook, which are
configured with the same equipment. Each of these three file systems consists of 5.5TB (5 x 1.1TB LUNS,
each of which are 4+1 RAID5 300GB FC 10k/rpm drives) drive space on two separate StorageTek FLX380 drive array.
These drives use a 512KB segment size, giving a 2MB stripe size for any write. Two main compute clusters are
concurrently using this storage, Midnight and Iceberg. Midnight is a Sun cluster comprised of 2312 compute
Opteron processors in 415 shared memory nodes. Iceberg is 800 processor IBM System spread over a mix of 98 IBM servers.
PSC (Pittsburgh Supercomputing Center) is a joint effort of Carnegie Mellon University and the
University of Pittsburgh together with Westinghouse Electric Company to provide resources for
high performance computing to universities, government, and industrial researchers.
We collected file statistics from two volumes used as scratch space in the PSC clusters.
They are labeled psc-scratch and psc-bessemer. Both are used for for output from parallel
jobs running on the compute cluster. Users of psc-scratch archive files that they need
and delete those they don't. PSC deletes files from here on a
need basis, generally those older than 21 days. Whereas psc-bessemer is used for special projects that
need more disk space for a longer period of time. Files are deleted only after the project is completed.
Both file systems run Lustre, a high performance file system for shared clusters.
The psc-scratch is used by BigBen, a Cray XT3 MPP system
with 2068 compute nodes and twenty-two dedicated IO processors. The storage is backed by 1 Lustre DDN
8500 in a 8+1 configuration, serving 24 2TB luns to 8 lustre OSS nodes. The psc-bessemer used 3 Lustre
DDN 9550 in a 8+2 configuration serving 24 6TB luns to 12 lustre OSS nodes. All drives in both configurations
are 400GB 7200RPM. The psc-bessemer is also attached to BigBen, Pople: an SGI Altix 4700 with 768 processors
and 1.5 TB of shared memory, and Salk: an SGI Altix 4700 with 144 processors and 288 gigabytes memory.
We also collected data from our storage laboratory at CMU called PDL (Parallel data laboratory).
PDL is composed of a group of graduate students and scientists at CMU who work on storage systems
related research and projects. File statistics were collected from a storage server that manages
a variety of volumes storing data typical to university development and research environment.
This data is represented as pdl1 and pdl2 on the Graphs. These volumes store user home
directories, CVS repository, OS distributions, music and video for file system testing, internal
projects and trace data collected from industry as well as CMU labs.
Its backend store is a NetApp filer using WAFL with Dual
parity RAID LUNs created out of SATA disks. We also collected statistics from a volume that is
used as PDL scratch space, labeled as pdl-scratch.
PNNL
ARSC
PSC
CMU