Capturing the private cloud

The information technology industry is all abuzz over cloud computing, but government agencies might need to sit on the sidelines until vendors work out security and procurement issues. All those benefits that accrue with this new approach to computing — flexibility with resources and savings with consolidation — will have to wait.

Or will they? At the Federal IT on a Budget Forum in Washington recently, Jim Whitehurst, president and chief executive officer of Red Hat, urged agencies to build their own clouds.

"There is a significant amount of value in a cloud infrastructure for a single entity that is running multiple programs and multiple data centers," Whitehurst said. If an agency is running more than 1,000 servers, it could save money and become more flexible with its processor resources by building an internal cloud computing infrastructure, he said. A private cloud could offer almost all the benefits of a public cloud but without the attendant security and privacy headaches.

"What is the benefit of a public cloud? For most large agencies in the government or even medium-sized agencies, there is a not a lot of purchasing cost advantage with going to a third-party cloud," Whitehurst said. "The real benefit is getting high utilization of your existing infrastructure and flexibility around that."

Whitehurst said Red Hat has more than 50 enterprise customers with private clouds. "They are seeing huge benefits [by] running clouds themselves," Whitehurst said. One customer he spoke with, a chief information officer of a large organization running 25,000 processors, told him that public-cloud services such as Amazon.com would not provide much of a cost advantage to his organization because his company could obtain servers at almost the same cost as those providers.

In March, IT analyst firm McKinsey & Co. released a report arguing that although small and midsize organizations could save money by using public-cloud services, large organizations could save money by keeping their processing needs in-house.

The firm estimated that if an organization could consolidate servers and keep the operational costs of the resulting data center to $45 per month per CPU, it could enjoy this computational power at a rate that would be less expensive than the cost of running operations on, say, Amazon's Elastic Compute Cloud.

"The cloud is just a fancy term for globally accessible resource pool," said Maj. Carl Brodhun, a lead information architect for the Marine Corps' Systems Command, who spoke at the forum about the Marines’ private-cloud deployment. "The key to cloud computing is the business processes that allow people to leverage the technical advancements associated with virtualization."

Private cloud reporting for duty

How do you set up a cloud? Most, though not all, of the technologies are available for an organization to put together a computing-as-a-service offering.

To set up a private cloud, an agency would pool all its servers and offer the processing power to each department, Whitehurst said. Although such a step is easier said than done, some agencies are attempting this approach, or at least parts of it.

Rather than build a cloud computing infrastructure, the Army Research Laboratory is re-engineering its application layer in a cloudlike fashion.

"Although we're not doing virtualized compute resources, our application architecture represents a lot of what you'd like to see in a cloud — we're doing real-time provisioning, service-level agreements...dynamically monitoring services," said Dennis Reedy, a system architect and adviser for ARL supporting contractor Altus Engineering, speaking at the JavaOne conference held recently in San Francisco.

The lab is updating its Modular Unix-based Vulnerability Estimation Suite. Version 3 of MUVES is a complete rewrite of a general-use modeling and simulation application that ARL has used for the past 20 years. Among other uses, the agency employs the software to measure how much damage bullets, bombs and other projectiles can do to vehicles.

Unlike the previous version of MUVES, the new software will be a distributed architecture, one in which different functions of the applications are broken into tiers, which then could be run on different workstations or servers. The software will be "composed over numerous services on a local network," said ARL computer scientist Ronald Bowers, who also spoke at the JavaOne presentation. The client software, which resides on the user's workstation, interacts with gateway software, which pieces together the needed components from various services elsewhere on the network, including other workstations. To confront the persistence problem, material that is not currently needed is moved to storage.

For an organization to build a private cloud, the first step will be to consolidate the servers. They could be consolidated into a single data center for management efficiency. A second data center could be set up as backup.

Most, although not all, cloud providers use some form of server virtualization technology, such as commercial offerings from Citrix Systems, VMware or Microsoft. The general idea is that each physical server could host multiple virtual servers, or virtual containers, each running an application. The software being placed in a cloud has to be installed in virtual containers, and all the virtual containers need to be placed in a repository, where they can be called up as needed.

These virtualization tools all have management consoles, which allow administrators to keep an eye on the virtualized machines, initiate new instances of applications and move them from one server to another. Some of those applications offer some automation capabilities, though an organization would need to script the workflow for automated operations. In addition, load balancers would be needed in the front end to distribute the work across multiple virtual machines.

Cloud formations

Rue Moody, strategic products technical director at Citrix, said virtualization is critical to how an agency would set up a private cloud using the Citrix software stack.

"Instead of an application running a physical server as a contained unit, with server virtualization we are looking at workloads," Moody said. "We are separating the work of Outlook or SAP from the Windows operating system and the underlying server. I look now at the workload. It becomes portable. I can move that workload to any machine, physical server or virtual server, where I have capacity."

When installed on 64-bit x86 servers, Citrix's XenServer virtualization software can act as a base for cloud operations. Administrators can use the Citrix XenCenter console to start new instances of virtual machines and run multiple copies should the workload grow beyond the capability of a single server.

If an application being put into the cloud is a desktop application or a client/server application, it could then run on Citrix XenApp application platform software — formerly Citrix MetaFrame Server and Citrix Presentation Server. XenApp can run within a virtual server running on XenServer.

By going with XenApp, the application would be run from within the data center rather than on each client's computer. Each client has something called a Citrix Receiver, which users can access from a thin client, smart phone, or laptop or desktop PC. "Once they log in, they will get their desktop application," Moody said. "The user environment will be very similar to everything [the user] will be comfortable with." The files would be saved to network storage rather than on a user's computer. The polices are set either through Citrix's Policy Profile Manager or through Active Directory.

Moody suggests packaging only one application per virtual server. "Our provisioning technology allows you to provision out multiple users for one copy of the OS, the workload and the application," Moody said. Each virtual server gets a dedicated port on the physical machine, though they all share the same IP number. With this approach, even existing applications, such as software written in Cobol, can be packaged for use in new computers.

One downside with cloud computing is that users are accessing applications via the network, which can be slower than using an on-desk application. To help speed the delivery, an organization can use Citrix's NetScaler as the front end for servers. NetScaler acts as a load balancer for applications running across multiple servers. It also does some data caching as well.

Citrix also has a product called EdgeSite that monitors the amount of server resources a virtual instance is taking up on a given physical server. An administrator can write a script, using Citrix's Workflow Studio, that can start more instances of an application. "The administrator has to set up the thresholds at first, but then the system will monitor workloads [and spin up a new instance] automatically," Moody said.

Using Citrix is only one approach to deploying a virtualization-based private cloud, of course. Microsoft has Virtual Machine Manager (VMM), which is a component of Microsoft's Systems Center suite. The suite also includes Microsoft Operations Manager, which can commingle virtual machines and physical machines from a single view, said Susie Adams, chief technology officer at Microsoft Federal's Civilian practice.

Other virtualization management consoles include the open-source Open Nebula and VMware's VSphere. All those applications offer administrators the ability to "build up and tear down a virtualization service management layer," said Adam Rossi, president of IT consulting firm Platinum Solutions, which does work for intelligence agencies.

Also keep in mind that virtualization does not need to be used in a cloud. Instead of an infrastructure cloud, you could have a platform cloud, in which the underlying operating system is not an issue. The Google App Engine runs on this approach. You provide the code, and Google worries about what operating system it runs on.

Bill Vass, president of Sun Microsystems' federal subsidiary, said applications written for the Java Enterprise Environment (JEE) can run side by side on JEE application servers without developers needing to worry about the underlying operating system.

The service oriented architecture-based application stack "will allow you to push JEE-compliant threads onto the cloud," Vass said. A thread is the portion of a program running on the server. "You can initiate objects in memory on the cloud in threads on app servers that are balanced and policy-based."

Working the network

Networking is another aspect to work out when setting up a private cloud. Applications and servers that resided in different subnets within a large agency will now be located somewhere else within the agency's topology. A common way to address this problem is by offering each office its own virtual local-area network (VLAN). In this way, the organization can preserve the original routing information.

In many data centers, an application or group of interlinked applications typically enjoy their own subnets, said Tim Silk, Cisco systems engineer manager. Preserving this IP numbering-based system can be done through the use of VLANs.

In this scenario, a cloud provider would set up a network access control point, which a user could log in to. A workload broker, which keeps track of all the virtual machines, then can act as a go-between for the user and requested application.

Cisco offers software that it calls a soft switch, or a switch implemented entirely in software that can help organizations keep track of their subnets. The switch, the Nexus 1000V, is used as a component of VMware's VSphere application.

"In a scenario where you are actually moving work around, you have to move the network around with the machine," said Steve Picot, regional sales manager at Cisco. "The idea [with the soft switch] is to allow the network to treat a virtual machine just as it would a physical end-note."

Gaining altitude

Although most of the technology for building a private cloud is available, agencies might still have a number of issues to work out before realizing its full potential.

An office of the Environmental Protection Agency has set up a service that offers business intelligence software for other EPA offices on a fee-per-user basis. But it found that some software configuration problems initially hampered wider use.

"We provide…business intelligence tools [and] analytics tools on a software-as-a-service model, as if we were a contractor. [Users] don't have to install anything," said Timothy Hinds, program manager at the EPA Business Intelligence and Analytics Center. Hinds outlined the center's operations during a session at the Independent Oracle Users Group annual conference.

On a subscriber basis, EPA offices can use Oracle Business Intelligence Enterprise Edition, Business Objects XI, Informatica PowerCenter and SAS to prepare their reports. Users can generate reports as PDFs or Web pages. In addition to offering the software, the center also offers training, consulting and help-desk support.

Overall, this service-oriented approach should save the agency money in software licensing and support costs, Hinds said. The agency started a working capital fund that got the center up and running, though now it is funded by user fees.

"We buy [the software] on a CPU basis and sell it out to the agency on the basis of named users or concurrent users," he said. The software is purchased according to the number of processors it runs on, so it doesn't matter how many individuals use the software. "For us it doesn't matter how many users are on the CPU, as long as the CPU can handle the load."

Periodically, the center analyzes its costs of purchasing the software against the fees it gets from users. The price it charges users is then recalculated annually.

The backbone of the operation is the Oracle Business Intelligence Enterprise Edition, which is deployed on IBM HS20 blade servers that run Microsoft Windows 2003.

The architecture consists of a shared environment with development servers that users can log in to and prepare their data. Also in the mix are sets of staging servers and production servers, which only the center can access. The center also runs a public-access proxy server for reports that are issued for public access.

"We're very big on separation of duties," Hinds said. "Customers do not get to update stuff on the production servers. We do that for them."

Overall, business intelligence software works well within a shared-services architecture. However, the center ran into one glitch with the Oracle BI software.

"I think it is very easy to use and very powerful, and scales well to the enterprise," Hinds said. The downside, however, is that "it doesn't scale very well to the shared model of software-as-a-service."

"The product can be used for that, but it has several weaknesses," Hinds said.

The Oracle BI software can access only a single repository under Windows environments, so the center had to place all the user accounts in a single Rapid Database (RPD) file. And with this Oracle application, all user accounts come with full administrative rights for the software, Hinds said.

Those expansive privileges might be fine for stand-alone BI projects, but they are problematic in a shared-services environment. A user from one project using the software could potentially see all the material in other projects, which gives the agency huge risks in security, privacy and data leakage.

To resolve the problem, the center set up a workflow so that when users want to modify an RPD, the center prepares a stripped-down version of the file, using the material only from their project. When they are finished making changes, the center merges the modified file back into the master RPD. "This is a manual process for us," Hinds said.

The lesson is clear: Agencies should check to see how their software operates in a cloud architecture.

You also need to take care in configuring hardware. In general, servers that run virtual machines seem to require more memory than they do with other typical uses, said Clint Harder, a product development manager at CDW, which offers a hosted cloud service, Hosted Enterprise Infrastructure.

Administrators should find out what types of applications will be run in virtual environments and judge server capacity appropriately, Harder said. For instance, Microsoft Exchange and data warehousing in general not only tend to require lots of memory but also can go through periods of heavy use, which could potentially draw all the processing and memory capacity from a server, starving other virtual instances in the process.

Harder said developers should also keep in mind that a few spare servers should always be on hand, in case one of the primary servers fails. If that happens, they can move applications to a new environment. VMware offers this capability with its VMotion software, which can migrate a virtual machine from one server to another without interruptions in service.

Other issues that cloud architects must consider could involve certification and accreditation and automated billing.

Those who operate in a secure environment must think through what, if any, additional C&A steps need to be addressed for a cloud-based environment, Rossi said. Regulations largely haven't caught up with virtualization. "It is not as clear-cut as it should be," Rossi said.

In a secure environment, an application and its host operating system typically will be accredited, and that version will be placed on a master disk for installation on a physical server. That approach worked well until virtualization came along. In most cases, each copy of a virtual image must go through the accreditation process anew, which can slow deployment. "You see a lot of labor to go in and rescan the image," he said.

Another problematic area is billing. How does each office pay fairly for its share of the cloud? One of the advantages of using cloud services by Amazon and others is that billing is determined by a precisely defined set of usage metrics. Users pay for how much bandwidth is used, how much storage is consumed and how many CPUs are utilized.

Replicating that billing can be done in-house, though not without some work, as few commercial software products are available to help with the task at this point. Citrix's software, for instance, produces logs of all virtual-machine activity. But coordinating the use of the software with specific users would still require additional programming or scripting work, said Tom Simmons, Citrix's area vice president for government systems. "We do have some agencies use [the logs] today, more for the preparation for charge-back rather than in a live charge-back mode," he said.

"Citrix will not develop the technology that will develop a charge-back tool," Simmons said. "But the tools in our products will have logs with the data that the charge-back tools will require."

"Today, I don't think there are any commercially available products that allow," this sort of usage monitoring, Microsoft's Adams said.

The hurdles notwithstanding, the idea of setting up a private cloud could be something for agencies to consider. It can save money and make the agency more responsive to its ever-changing IT needs.