An upgrade to the plumbing

The inspector general at the Health and Human Services Department has 10 regional offices that house servers accessed by 85 field offices throughout the country. Recently, the OIG wanted to modernize its IT infrastructure and consolidate resources in data centers.'On paper, it seemed like a good idea,' said OIG chief technology officer Chris Finucane.But there were problems. The offices were linked in a hub-and-spoke configuration with dedicated frame relay connections.'On a good day, they might get 128K,' Finucane said.This WAN did not adequately support data access in its existing configuration and needed to be upgraded before any consolidation could be done. It was replaced with T1 links that provided 1.54-Mbps connections to the MCI mesh network cloud.'Every office was point-to-point with every other office,' Finucane said. But the problem did not go away. 'The performance wasn't noticeably improved by the upgrade.'The OIG considered another WAN upgrade before the consolidation. 'But doubling the amount of bandwidth wasn't going to gain us anything,' he said.The difficulty was that the transport and application algorithms on the file transfers did not work efficiently. A lot of the available bandwidth was simply being wasted, and applications were suffering from the latency.'We weren't completely saturating the T1s,' Finucane said. 'I understood enough about the philosophy of networking to know bandwidth wasn't the only element.' So he sat in on a webinar on data center consolidation and WAN acceleration. There he was introduced to the Steelhead WAN optimization appliance from Riverbed Technology Inc. of San Francisco.Steelhead is designed to address significant problems, said Alan Saldich, vice president of marketing for Riverbed Technology.First are bandwidth issues'applications that work fine over the LAN (up to 1 Gbps) can show lousy performance on the WAN (often 100 Mbps). Second is latency, which affects the performance of applications over TCP, the transport layer component of the Internet Protocols, which is sensitive to latency. Delays on a LAN often are measured in microseconds'millionths of a second'while on WAN they can be measured in milliseconds'thousandths of a second.Steelhead is a rack-mounted appliance that sits at each end of a WAN link, usually between the switch and router. It acts as a TCP proxy, intercepting TCP traffic and setting up its own session with the appliance at the other end. Each appliance reduces the volume of traffic by analyzing data being passed and storing it in short segments. As data is exchanged, only new data is passed between appliances.'Anything repetitive gets eliminated from the network,' Saldich said. 'That's usually more than 50 percent and sometimes up to 90 percent of the data.'Algorithms analyze patterns so data segments are identical at each end of the link.'If you send the same data through the system twice, the algorithm will always make the same decisions, so you always have the same segments,' Saldich said.This form of transport streamlining addresses TCP latency by maximizing the amount of data sent in each window, thereby reducing the number of round trips necessary in the TCP session. Application streamlining improves performance because apps normally require numerous exchanges between client and server to set up a session. 'This is a legacy of many years of development,' Saldich said. It often is not noticeable on a LAN, but on a slower WAN, 'it usually is not tolerable for end users.'Steelhead uses application-specific protocols to set up the session with the server over the LAN, then passes data over the WAN to the appliance on the client side.In the spring of 2005, Finucane brought in a pair of Steelhead appliances, along with similar tools from Cisco Systems Inc. and Juniper Networks Inc. of Sunnyvale, Calif.'We did a bake-off between them,' he said. 'At the end of it, Steelhead had the best performance.'The different brands worked equally well in the first exchange between client and server, when full files are transferred, Finucane said. In each case, compression provided about a 50 percent reduction in volume. But in subsequent exchanges, segment caching kicked in, further improving performance.'Steelhead seemed to be blazingly fast on the second return,' he said.But it was ease of implementation rather than speed that tilted the balance toward Steelhead, which worked better with the mesh network serving the HHS OIG and did not require any segmenting or re-engineering.'The second-rated appliance was really close. The mesh networking was the show stopper,' Finucane said.HHS began installing the appliances in August 2005 and finished up this past February. It is using the high-end 5010 model in its 10 regional offices, which offers 45 Mbps throughput and up to 4,500 TCP connections for $45,000 each. The 85 field offices have the midsize 1020 model, offering 2 Mbps throughput and up to 625 TCP connections.The installation was done without additional funds, Finucane said.'We're really happy with the devices,' Finucane said. One field office that had been doing a lot of large file transfers had complained that the T1 upgrade had not produced any results. After the Steelhead was installed, 'I got an unsolicited e-mail that said, 'What did you uys do to the network? It seems like I'm getting LAN speeds.' ' n