One size fits all?

Scott McNealy, chairman of Sun Federal, likes to say he's never been inside a government data center and recognized what he's seen. He jokes that the systems running agency applications are a jumble of thrown-together parts that collectively resemble something more like Frankenstein's monster than an efficient, consolidated server farm.Maybe's he's been to the Census Bureau's data center.Census is largely a Unix shop, with more than 1,000 servers running its systems. The lion's share is from Sun Microsystems Inc. and Hewlett-Packard Co., but just about every manufacturer is represented, along with their associated operating systems. The heterogeneous environment grew up because each business unit was responsible for acquiring the equipment to run its applications.Tom Berti, assistant division chief for systems management and administration at Census, is responsible for most of the Unix deployments. He says the chore of managing stovepiped resources is overwhelming.'You can have so many variations, and that makes for a disjointed support staff,' Berti said. 'These people usually specialize in one area, and when you have all these different products you don't have the depth required to support them.'Berti and the bureau think utility computing could be the answer.About 18 months ago, the bureau's IT directorate mandated the Census Bureau Utility Computing Environment to simplify data center operations and increase efficiency. The plan would also shift much of the IT hardware and software budget away from separate business units within the bureau.'The capital investment would be our responsibility,' Berti said. 'That's the point of utility computing.'In utility computing, IT services are delivered from a shared pool of resources rather than from dedicated hardware and software supporting each application. Among other things, the model allows admins to provision computing resources more quickly and adjust to changing needs.To that end, Census recently installed a handful of blade servers in its Bowie, Md., data center to support the utility computing program and provide IT services to other bureau divisions.'We're in our infancy right now' with utility computing, said Berti. But the bureau is committed to the program. 'We're down the road now, and there's no turning back.'Berti's group built a dedicated module for blade servers in the 50,000-square-foot data center that supports Census operations. It then bought blade systems for about 150 servers from Egenera Inc. of Marlborough, Mass., and IBM Corp.For now, blade servers represent only about 10 percent of the center's environment. 'They're not all online,' Berti said. Census planned to cut over the first major application to the new platform last month.When it comes to IT consolidation, agencies like Census have two masters: their customers, who have various needs and preferred technologies, and themselves, who must answer to higher powers over questions of return on investment, total cost of ownership and IT-mission alignment.It's not always easy to balance the needs of both. About customers, Berti said, 'We're providing the service but they're paying for the service, so we want to accommodate them. If they were a Sun shop, they wanted Sun equipment.'Still, about two years ago the IT staff began looking for a way to standardize and improve efficiency.'The impetus for moving to utility computing was a meeting I had with Egenera,' Berti said.Egenera's approach is to strip the complexity out of the server with a blade form factor.'At the end of the day, all the customer wants is to run an application on the CPU,' said Egenera CTO Vern Brownell.Each blade contains only the processors and memory. Ancillary functions such as disk drives and network interfaces are moved to a common frame and managed by software.'Software is a lot easier to manage than hardware, and it's cheaper,' Brownell said.[IMGCAP(2)]The difficulty in trying to create a common pool of resources with traditional RISC servers is that although the processors can be fast and efficient, each server usually is dedicated to a single application.'It's very difficult to combine applications on a server,' Brownell said. 'A lot of time the software components don't work well together.'Egenera's BladeFrame system has a high-speed interconnect fabric to support up to 24 blade servers on each frame, creating a processing area network with interfaces for the storage area network and the LAN.'We're taking all of these things away, so the server has no identity,' Brownell said. 'That enables utility computing.'Census liked the approach.'Our CIO was sold on the concept and he asked us to move forward,' Berti said.Census evaluated the products of five leading blade manufacturers in proof-of-concept testing in its lab. When all products proved successful, the IT directorate bought the equipment to continue benchmarking and to sell the utility computing concept to the bureau's operational units.In August 2005 a blanket purchasing agreement was awarded for Egenera's BladeFrame and IBM's BladeCenter systems. The servers will be standardized on Red Hat Enterprise Linux and Windows operating systems, and the first applications to be served up are being moved to Oracle Database 10g with Real Application Clusters.Once a decision to buy the equipment was made, there remained the task of convincing other operational divisions within Census to use it.'We have to market it to them,' Berti said. 'The IT division doesn't write or build applications. We leave that up to our business units. It's very important that they run on platforms they are comfortable with. It was a tough sell.'Lab demonstrations and the promise of faster provisioning helped to make the sale, but the job of selecting a configuration for the software proved to be more difficult.'We wanted a standard configuration for Linux,' Berti said. 'It's tough to get a standard configuration.'A standardized configuration would allow the IT directorate to burn that configuration into its distribution servers and push it out to each server as needed, simplifying management and provisioning. But the configuration also required buy-in from each operational division and there was little consensus on what should be included. The problem is that there is no single right or wrong answer for all of the possible choices.'Some of it was driven by technology and some was driven by personal preferences,' Berti said. 'Eventually you have to pick one.'Or pick half a dozen. Berti said they managed to whittle it down to five or six configurations for the two operating systems being used, and three for the Oracle Database 10g.'That's pretty darn good,' he said.The data center now has eight BladeFrame systems and four BladeCenters from IBM.'I'm satisfied with the progress we've made up to the end of the fiscal year,' in selecting the servers and getting them on site, Berti said. 'I'm not discouraged, but it took longer than I expected to decide on standardized images for the systems.'Although the goal is to simplify, utility computing still is complex, Berti said. Over the last 18 months some 20 engineers have received training from Egenera and 20 have been sent out for Red Hat training.And although it should save money in the long run, utility computing does not come cheap. Brownell said the entry cost for setting up a system with the BladeFrame solution is around $100,000. That is money the Census IT directorate must get into its own budget. It spent a year identifying money spent by the bureau's operating units on hardware and software support.[IMGCAP(3)]'We are in the midst of trying to transfer some of that money to IT so we can support utility computing,' Berti said.The first major application to switch to the new platform will be the Master Address File/Topologically Integrated Geographic Encoding and Referencing Accuracy Improvement Project, or MAF/TIGER AIP. MAF is a complete, current list of 175 million U.S. business and residential addresses, and TIGER is a database that associates the addresses with each other and with other manmade and geographic features. Online mapping services such as MapQuest Inc. draw on MAF/TIGER for their data.'The whole system is being re-engineered to run on Egenera servers,' with four dual-core AMD Opteron processors and 32GB of memory on each blade, Berti said in early May. 'The systems are in testing right now. They're very happy with the performance they're getting.'Brownell said MAF/TIGER is the type of application utility computing is best suited to. It's a large-scale program with a high volume of data. And it's mission-critical, so it requires high availability. He said the Census installation so far is not unusually large. Some customers have 60 or more BladeFrame systems in place, each capable of handling up to 24 blades.Berti said that applications that require large amounts of batch processing might not be suitable for utility computing because of the need for flow-routing, but otherwise utility computing will be the wave of the Census Bureau's future.There is no timetable for moving everyone at the bureau to utility computing. The IT directorate will let each business unit request the services as needed, based in part on how quickly they use up current server capacity.According to Berti, 'Nobody buys equipment now except blade servers.'