Data centers untangled

In many organizations, the cables that run between servers and network equipment are usually out of sight ' in closets, under the floor or in the ceiling ' and out of mind. But as data centers grow, evolve and age, the miles of cabling in the background can become a nightmare to manage.Ask the Internal Revenue Service. In 1996, when what is now the Government Accountability Office examined the agency's data centers, it criticized the agency for multiple problems with its data center cables. The agency had some exposed data cables that could be cut or tapped. Inspectors determined that the foam keeping the cables in place was a fire hazard, and data center employees could not identify some of the cables that ran through the facility.That was a prime example of what happens when a clear plan for cabling isn't included on the ground floor, as it were, of data center construction.'I start thinking about cabling from the very start,' said Albert Ramos, a facility engineer at Terremark, which offers hosting and collocation services to the government.Ramos is involved in building Terremark's new $250 million data center in Culpepper, Va.Fortunately, some simple measures can help keep your cables from growing into a gigantic, untamable monster. These best practices can make your life easier when dealing with cables, managing your data center or just overhauling a wire closet.'A data center is as good only as the weakest link,' said Chris Henri, a program manager at integrator American Systems who is setting up a data center for the Marine Corps. 'The infrastructure, if thought out, can be flexible, comprehensive and complete,' Henri said. 'It can be'the best return on investment if done properly, or it can be one of the worst.'Here are 13 tips for taming the cable monster. When servers or switches are decommissioned, facility workers often don't remove the associated cables. The shortcut is to just cut the heads off the cables and leave them in place, said Mike Clemson, vice president of facilities at Server Vault. As a result, 'over time, dead cables occupy a larger and larger volume of under-floor or overhead space.'When you take equipment out of service, remove the cables. 'When a customer decommissions a server, we remove all the cabling going to and from that location,' Ramos said.'A lot of time, when you see that spaghetti effect in data centers, about half the cables are probably dead,' he added. 'I've been to many facilities where a whole section of the floor is dead cable.' Too many facilities managers drape cables between points, laying them on the concrete under a raised floor or above the tiles of a drop ceiling. This can be a problem when the cool air is pumped through the raised floor, because when a data center is designed, estimates of how much space will be needed to force air through are based on unobstructed spaces, Clemson said. Cables filling the space will make the whole center run hotter. Vertical rack cable managers will keep cables on the data center floor tidy, said David Divins, principal engineer at Server- Vault. As one agency data center manager said, when the agency's top brass walk through the data center, it's good to have them see tidy rows of cable. Many installers routinely overestimate cable lengths on the assumption that too long is better than too short, Clemson said. Leftover cable gets stuffed under the floor and creates what he called air dams that block the flow of cool air. Have your cable cut to the lengths you need.And if you have extra Ethernet cable, don't roll it into a coil, said Rudi Van Drumen, chief technology officer at Netherlands- based integrator Competa IT, during a hardware troubleshooting presentation at the recent Usenix '08 conference in Boston. A coiled cable acts as an inductor that can pick up radio frequency interference and disrupt signals. Throughput is slow, and you're not sure why? Perhaps your communications cables are sitting adjacent to power cables. Electricity, with its attendant properties of magnetism, can interfere with the throughput of a network cable. Insulation on the power cable should stop most ' though perhaps not all ' of this interference, but it's best to keep network cables as far away from power cables as possible, Clemson said.It's also a good idea to separate fiber cables from copper ones. 'Fiber tends to be more fragile than copper,' Clemson said. Handling both types together, a technician might damage the fiber cables.Even different types of copper cable should be separated whenever possible, said Bill Powell, vice president of cabling services at American Systems. A signal going through a Category 5 cable can easily bleed onto a nearby Category 6 cable and cause interference, he said.Ramos said he uses a multiple-tier system of cables with a dedicated 12-inch rack for each tier. The top tier is used for bulk fiber from the outside. The second tier is for shielded copper cabling, and the bottom tier can be power cabling. He prefers using overhead space, allowing a foot or more between tiers. Plastic cable ties are cheap and easy to use for bundling cables, but cutting them can nick the underlying cable, Divins said. Plastic ties also make it difficult to remove old cables. Ramos uses heavy-duty string to tie cable to the rack. Another option is Velcro, Henri said. It can be pulled only so tight, and unwrapping bundles for the addition and removal of cables is easy. Network cables come in two varieties ' copper and fiber. The tradition of choosing fiber for performance and copper for price is breaking down. 'Copper has gone up in price five times in the last five years,' said Lim Goh, chief technology officer at SGI, at the High Performance Computing and Communications conference in Newport, R.I., earlier this year. 'As price goes up, it makes it practical to think about optical cables,' he added. Copper also has inherent latency limitations that make it hard to keep up with processors. When designing a new data facility wire closet, be sure to leave some room for expansion, both in space and throughput. Ron Bryson, an American Systems program manager, noted that at one government facility American Systems established, the agency installed fiber-optic OC-3 cable even though it didn't need that much throughput at the time. 'They were planning to keep this structured cabling system for at least 15 to 20 years.Obsolescence was a big deal for them ' they figured they would go through three or four generations of equipment changes,' he said. Without foresight, you could end up making cable spaghetti by running cables across different cabinets on an asneeded basis. Instead, consider running cables from all the servers to a central patching facility, Bryson said.'As blade servers or switch upgrades occur, you do not have to try to replace those backbone cables.' Instead, run a new connection to the hub, and then make connections through a series of shorter patch cables. 'If you have money, you should try to do that upfront,' Bryson said. Once you amass a certain amount of equipment, you should start thinking about the most orderly way of laying it out. Then your cabling will be orderly, too.Cisco federal sales manager Steve Picot said most data centers he sees are set up in a multitiered arrangement. An access layer switch is on top of each rack of servers. Cables go from each server to the switch. Another set of cables runs from the access nodes to aggregation layer switches located perhaps at one end of the aisle of servers. Finally, a third set of cables run from the aggregation layer to some central point in the data center where all the cables converge and connect to the outside telecommunications conduit. These are the core layer switches.Henri said data center designers should look at ANSI standard TIA-942, 'Telecommunications Infrastructure Standard for Data Centers,' which lays out placement of data center components in a hierarchy. 'It doesn't matter where you put the systems,' Henri said. 'If you need to go farther away, you step up the hierarchy and then step back down.' Any facility manager worth his or her salt can cut a length of cable, crimp it and attach plugs on either end.However, you might still want to purchase pre-terminated cable whenever possible. Pre-terminated cables, which are usually assembled in a sterile environment, typically have a lower signal loss than the homemade variety. Although such losses might be within today's acceptable range, that could change as new generations of equipment come on board. 'A technician meeting the minimum standard right now is OK, but the work would be the first point of failure down the road,' Powell said. In many facilities, cable passes under a raised floor that also provides a passage for chilled air pumped in to cool the equipment. Be sure not to put so many cables in the air flow lanes that the cool air never finds its destination and equipment overheats.David Gay, vice president of special products at American Systems, said routing cable along circuitous paths is permissible in cases where a more direct route would block airflow. Bundling cable too tightly or bending it too sharply around corners can lead to signal loss, Henri said. 'You don't want to add additional losses due to poor installation.' Proper labeling of cable is crucial, especially if the team that set up the data center is different from the operations and maintenance employees, Gay said. Clear labeling makes it easy to identify cables for removal or replacement.ANSI TIA 606 contains standards for labeling cables. You might also want to consider mapping your data center on a grid for easy identification of cable routes and endpoints. In his book 'Build the Best Data Center Facility for Your Business', Douglas Alger wrote, 'It is extremely difficult to trace cable and electrical runs if no numbering scheme exists to designate where they run from and to.'Grid coordinates can be marked on the walls or floors. One axis might be numbers while the other is letters, so it would be easy, for example, to find the server at location B26. Another option would be to number the aisles supermarket-style.In a Terremark facility, Ramos said, each cable is labeled with to and from locations ' identified with grid coordinates ' and the number of the patch panel, rack and port. Ramos uses a P-Touch electronic label-maker from Brother.Try to align your labeling and grid coordinates with best practices found elsewhere in your agency. Then any agency employee could walk into any agency data center and know the scheme, said Algers, who is a member of the Cisco infrastructure team. He said Cisco has standardized labeling procedures for its data centers worldwide. 'We have a lot of stuff documented on our internal Web site.'