Meltdowns zap NSA's giant new data center
Connecting state and local government leaders
Surges caused by arc fault failures have melted equipment and will delay operations at the $1.5 billion center for a year.
Operations at the National Security Agency’s massive new Utah Data Center will be delayed by a year because of powerful electrical surges that have caused explosions, melted equipment and disrupted circuits. The center had been scheduled to begin operating this month.
Ten meltdowns caused by arc fault failures — described as "a flash of lightning inside a 2-foot box" — over the last 13 months have destroyed hundreds of thousands of dollars of equipment at the center, the Wall Street Journal (subscription required) reports.
The failures occurred between Aug. 9, 2012 and Sept. 25 this year, with each causing up to $100,000 in damage, the Journal reported. It took investigators six months to determine what caused the first two failures; the causes of the other eight remained undetermined.
While investigations continue, an initial report by the Army Corps of Engineers’ Tiger Team blamed the problems overall on fast-tracking construction of the $1.5 billion center and taking shortcuts around “regular quality controls in design and construction,” the Journal said. The Army Corps’ report also said it “did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents.”
Arc faults are high-powered electrical discharges between two or more conductors. Forbes quoted an unidentified person familiar with the construction project as saying, “The problem, and we all know it, is that they put the appliances too close together. They used wiring that’s not adequate to the task. We all talked about the fact that it wasn’t going to work.”
Backup generators at the center also reportedly have failed in testing, with officials disagreeing over the cause.
The Army Corps of Engineers is overseeing the construction, with architectural firm KlingStubbins handling the electrical work under a subcontract. Three companies — Balfour Beatty Construction, DPR Construction and Big-D Construction — are the primary contractors.
The center is located at the National Guard’s Camp Williams near Bluffdale, 26 miles south of Salt Lake City. The NSA has said the data storage capacity of the center is classified, but estimates have ranged from several zettabytes (a zettabyte being 1 trillion gigabytes) to Utah Gov. Gary Herbert’s claim of a yottabyte (a thousand zettabytes).
In July, Forbes examined blueprints of the facility with Brewster Kahle, founder of the Internet Archive, and came up with a lower estimate, of about 12 exabytes.
By any measure, the center — one of the largest in the world and the largest run by the intelligence community — will have the capacity for a lot of the data being generated by the agency’s surveillance programs, including its controversial phone records collection and PRISM Web monitoring program. Delaying its operation for a year will put a crimp in NSA’s plans, but the Utah center isn’t its only means of processing data.
The agency has a data center at its Fort Meade, Md., headquarters and in the spring broke ground at the base on the $860 million High Performance Computing Center-2, a 600,000-square-foot facility scheduled to go live in 2016.