Data centers untangled

 

Connecting state and local government leaders

The cabling in data centers can grow like the worst kind of weed, clogging cooling systems and hindering network performance. Here are 13 practical tips for getting control over what lies beneath the floor.

In many organizations, the cables that run between servers and network equipment are usually out of sight ' in closets, under the floor or in the ceiling ' and out of mind. But as data centers grow, evolve and age, the miles of cabling in the background can become a nightmare to manage.Ask the Internal Revenue Service. In 1996, when what is now the Government Accountability Office examined the agency's data centers, it criticized the agency for multiple problems with its data center cables. The agency had some exposed data cables that could be cut or tapped. Inspectors determined that the foam keeping the cables in place was a fire hazard, and data center employees could not identify some of the cables that ran through the facility.That was a prime example of what happens when a clear plan for cabling isn't included on the ground floor, as it were, of data center construction.'I start thinking about cabling from the very start,' said Albert Ramos, a facility engineer at Terremark, which offers hosting and collocation services to the government.Ramos is involved in building Terremark's new $250 million data center in Culpepper, Va.Fortunately, some simple measures can help keep your cables from growing into a gigantic, untamable monster. These best practices can make your life easier when dealing with cables, managing your data center or just overhauling a wire closet.'A data center is as good only as the weakest link,' said Chris Henri, a program manager at integrator American Systems who is setting up a data center for the Marine Corps. 'The infrastructure, if thought out, can be flexible, comprehensive and complete,' Henri said. 'It can be'the best return on investment if done properly, or it can be one of the worst.'Here are 13 tips for taming the cable monster. When servers or switches are decommissioned, facility workers often don't remove the associated cables. The shortcut is to just cut the heads off the cables and leave them in place, said Mike Clemson, vice president of facilities at Server Vault. As a result, 'over time, dead cables occupy a larger and larger volume of under-floor or overhead space.'When you take equipment out of service, remove the cables. 'When a customer decommissions a server, we remove all the cabling going to and from that location,' Ramos said.'A lot of time, when you see that spaghetti effect in data centers, about half the cables are probably dead,' he added. 'I've been to many facilities where a whole section of the floor is dead cable.' Too many facilities managers drape cables between points, laying them on the concrete under a raised floor or above the tiles of a drop ceiling. This can be a problem when the cool air is pumped through the raised floor, because when a data center is designed, estimates of how much space will be needed to force air through are based on unobstructed spaces, Clemson said. Cables filling the space will make the whole center run hotter. Vertical rack cable managers will keep cables on the data center floor tidy, said David Divins, principal engineer at Server- Vault. As one agency data center manager said, when the agency's top brass walk through the data center, it's good to have them see tidy rows of cable. Many installers routinely overestimate cable lengths on the assumption that too long is better than too short, Clemson said. Leftover cable gets stuffed under the floor and creates what he called air dams that block the flow of cool air. Have your cable cut to the lengths you need.And if you have extra Ethernet cable, don't roll it into a coil, said Rudi Van Drumen, chief technology officer at Netherlands- based integrator Competa IT, during a hardware troubleshooting presentation at the recent Usenix '08 conference in Boston. A coiled cable acts as an inductor that can pick up radio frequency interference and disrupt signals. Throughput is slow, and you're not sure why? Perhaps your communications cables are sitting adjacent to power cables. Electricity, with its attendant properties of magnetism, can interfere with the throughput of a network cable. Insulation on the power cable should stop most ' though perhaps not all ' of this interference, but it's best to keep network cables as far away from power cables as possible, Clemson said.It's also a good idea to separate fiber cables from copper ones. 'Fiber tends to be more fragile than copper,' Clemson said. Handling both types together, a technician might damage the fiber cables.Even different types of copper cable should be separated whenever possible, said Bill Powell, vice president of cabling services at American Systems. A signal going through a Category 5 cable can easily bleed onto a nearby Category 6 cable and cause interference, he said.Ramos said he uses a multiple-tier system of cables with a dedicated 12-inch rack for each tier. The top tier is used for bulk fiber from the outside. The second tier is for shielded copper cabling, and the bottom tier can be power cabling. He prefers using overhead space, allowing a foot or more between tiers. Plastic cable ties are cheap and easy to use for bundling cables, but cutting them can nick the underlying cable, Divins said. Plastic ties also make it difficult to remove old cables. Ramos uses heavy-duty string to tie cable to the rack. Another option is Velcro, Henri said. It can be pulled only so tight, and unwrapping bundles for the addition and removal of cables is easy. Network cables come in two varieties ' copper and fiber. The tradition of choosing fiber for performance and copper for price is breaking down. 'Copper has gone up in price five times in the last five years,' said Lim Goh, chief technology officer at SGI, at the High Performance Computing and Communications conference in Newport, R.I., earlier this year. 'As price goes up, it makes it practical to think about optical cables,' he added. Copper also has inherent latency limitations that make it hard to keep up with processors. When designing a new data facility wire closet, be sure to leave some room for expansion, both in space and throughput. Ron Bryson, an American Systems program manager, noted that at one government facility American Systems established, the agency installed fiber-optic OC-3 cable even though it didn't need that much throughput at the time. 'They were planning to keep this structured cabling system for at least 15 to 20 years.Obsolescence was a big deal for them ' they figured they would go through three or four generations of equipment changes,' he said. Without foresight, you could end up making cable spaghetti by running cables across different cabinets on an asneeded basis. Instead, consider running cables from all the servers to a central patching facility, Bryson said.'As blade servers or switch upgrades occur, you do not have to try to replace those backbone cables.' Instead, run a new connection to the hub, and then make connections through a series of shorter patch cables. 'If you have money, you should try to do that upfront,' Bryson said. Once you amass a certain amount of equipment, you should start thinking about the most orderly way of laying it out. Then your cabling will be orderly, too.Cisco federal sales manager Steve Picot said most data centers he sees are set up in a multitiered arrangement. An access layer switch is on top of each rack of servers. Cables go from each server to the switch. Another set of cables runs from the access nodes to aggregation layer switches located perhaps at one end of the aisle of servers. Finally, a third set of cables run from the aggregation layer to some central point in the data center where all the cables converge and connect to the outside telecommunications conduit. These are the core layer switches.Henri said data center designers should look at ANSI standard TIA-942, 'Telecommunications Infrastructure Standard for Data Centers,' which lays out placement of data center components in a hierarchy. 'It doesn't matter where you put the systems,' Henri said. 'If you need to go farther away, you step up the hierarchy and then step back down.' Any facility manager worth his or her salt can cut a length of cable, crimp it and attach plugs on either end.However, you might still want to purchase pre-terminated cable whenever possible. Pre-terminated cables, which are usually assembled in a sterile environment, typically have a lower signal loss than the homemade variety. Although such losses might be within today's acceptable range, that could change as new generations of equipment come on board. 'A technician meeting the minimum standard right now is OK, but the work would be the first point of failure down the road,' Powell said. In many facilities, cable passes under a raised floor that also provides a passage for chilled air pumped in to cool the equipment. Be sure not to put so many cables in the air flow lanes that the cool air never finds its destination and equipment overheats.David Gay, vice president of special products at American Systems, said routing cable along circuitous paths is permissible in cases where a more direct route would block airflow. Bundling cable too tightly or bending it too sharply around corners can lead to signal loss, Henri said. 'You don't want to add additional losses due to poor installation.' Proper labeling of cable is crucial, especially if the team that set up the data center is different from the operations and maintenance employees, Gay said. Clear labeling makes it easy to identify cables for removal or replacement.ANSI TIA 606 contains standards for labeling cables. You might also want to consider mapping your data center on a grid for easy identification of cable routes and endpoints. In his book 'Build the Best Data Center Facility for Your Business', Douglas Alger wrote, 'It is extremely difficult to trace cable and electrical runs if no numbering scheme exists to designate where they run from and to.'Grid coordinates can be marked on the walls or floors. One axis might be numbers while the other is letters, so it would be easy, for example, to find the server at location B26. Another option would be to number the aisles supermarket-style.In a Terremark facility, Ramos said, each cable is labeled with to and from locations ' identified with grid coordinates ' and the number of the patch panel, rack and port. Ramos uses a P-Touch electronic label-maker from Brother.Try to align your labeling and grid coordinates with best practices found elsewhere in your agency. Then any agency employee could walk into any agency data center and know the scheme, said Algers, who is a member of the Cisco infrastructure team. He said Cisco has standardized labeling procedures for its data centers worldwide. 'We have a lot of stuff documented on our internal Web site.'
















1. Remove old cables.





2. Use cable trays and rack cable managers.

3. Measure your cable.



4. Don't mix cable types.







5. Use resealable ties.

6. Consider fiber.

7. Future-proof your facility design.



8. Establish hubs.



9. Arrange your equipment.





10. Use pre-terminated cable.



11. Don't block airflow.



12. Don't pressure the cable.

13. Label the cable.









GCN News Editor Wilson P. Dizard III contributed to this report.

NEXT STORY: Fewer cables in your future

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.