Order out of chaos

When Doug Babb first signed on as chief information technology systems architect at Hill Air Force Base, Utah, the base was running a dizzying mix of operating systems, servers and applications.There was Hewlett-Packard OpenVMS, Sun Solaris, Microsoft Windows, and varying flavors of Unix and Oracle. The whole operation was plagued by frequent downtime. Each hour of system downtime cost about $1 million, and the base had suffered through eight downtime episodes in three months.'It was like opening a closet where everything falls out,' Babb said. 'I kept wondering, do I put everything back in the closet, or do I throw it all out and start over?'The system had 42 servers in six buildings, running 80 applications for six different branches. 'Everybody had their own administrators, agendas and cultures,' Babb said.The confusion caused by the clash of systems even affected morale ' Babb described the atmosphere on the base as having 'little joy.'And the base has an important mission. Hill's Ogden Air Logistics Center provides maintenance, engineering and logistics management for some of the service's most essential aircraft and weapons, including the F-16 Falcon, the A-10 Thunderbolt II, the C-130 Hercules and the Minuteman III intercontinental ballistic missile. Downtime wasn't an option.The base employees who performed maintenance on the aircraft had to access a wide array of databases and systems. Some were for scheduling, some were for parts and replacements.'The biggest problem was that we had a lot of different hardware and programs,' said Mike Jolley, project manager and chief of the operational policy branch in the informational technology directorate at Hill. 'They were all built at different times by different people.'Over the course of 11 months, Babb, a contractor with Systems Implementers, worked on-site with Jolley and a team of 27 people ' mostly civilian employees but also some contractors ' to deploy the new system, dubbed Project Bonfire.The base had been using HP Superdome and Sun servers. 'Hill had huge metal,' Babb said. 'You name it, they had it.' And the Windows environment on the base Babb described as 'unstable.' There had been some data attacks and other security concerns.Choosing a new direction for the base's system proved to be fairly easy. 'The only vendor out there that had gone through National Security Agency security was Red Hat,' Babb said. 'That narrowed down our choices pretty well.'Based on Red Hat Enterprise Linux 5, Project Bonfire employs grid computing, linking servers to boost performance. Babb described the back of one of the servers on the grid as 'like veins on a body. It's very complex and interconnects with everything.'The Hill group also liked the fact that open systems such as Linux have a lower cost of ownership.By consolidating and standardizing on less expensive x86, 64-bit architecture and consolidating on the Red Hat Linux platform, the Air Force saved more than $5 million. To sustain the existing environment would have cost Hill $5 million per year. The Linux system cost $100,000.The base supports 22,000 on-site users and 3,000 more at remote sites. For some applications, Project Bonfire required a conversion from client-server to the Web, which took some user training. But for the most part it was a transparent conversion for users and required no training at all. 'One day, they were on the old system, then one day on the new system,' Babb said.With 25,000 users, the cost of training them all on a new system 'would have been beyond belief,' he said.One of the reasons the project was called Project Bonfire is that 'it was a sort of celebration,' Babb said. 'We consolidated servers, but also storage. We worked on [uninterruptible power sources] and cooling. It was a crazy 11 months.'Since the base implemented Project Bonfire, the turnaround time to equip an aircraft with new wheels and brakes shrank from 38 days to 16 days. The new system can run its largest data warehouse load in less than three hours, where before it took more than 12 hours. Response times decreased from 10 seconds to a little more than a second. And instead of being housed in six buildings, the system now resides in two.The Project Bonfire group is working on deploying this same technology to 59 Army sites. The group also has received inquiries from the Marine Corps and the Navy.'The bottom line is not to create a really cool IT system,' said Rich Wood, vice president at Systems Implementers, which has helped Hill implement commercial software for enterprise resource planning, among other contracts. 'The whole idea is to support the warfighter and get them the tools they need.'