When are supercomputers really super?

 

Connecting state and local government leaders

When the Energy Department's National Energy Research Scientific Computing Center went shopping for a new mid-sized supercomputer, processor speed was a critical factor, as expected.

When the Energy Department's National Energy Research Scientific Computing Center went shopping for a new mid-sized supercomputer, processor speed was a critical factor, as expected. But it also carefully scrutinized the interconnect speeds of the proposed systems. That's because for this supercomputing center, the speed of the conduit between processors was as important as the speed of the CPUs themselves.

How interconnects fared in the last HPC Challenge

Developed with DARPA funding, the HPC Challenge Benchmark is a set of seven tests that measure the performance of high-performance computing systems. To measure latency, the framework uses the Random Ring Benchmark. At right are the lowest latency times for systems using some of the most widely used interconnects, taken from an HPC Challenge held last November. Eighty-three organizations submitted test numbers. Results may include base and optimized runs. For more results, visit GCN.com and type 540 in the GCN.com/box.


Interconnect'Lowest measured latency (smaller number is better)


PathScale InfiniPath'1.31 microseconds


Cray RapidArray'1.63 microseconds


Quadrics'4.89 microseconds


NUMAlink'5.79 microseconds


Myrinet'19.00 microseconds


Gigabit Ethernet'42.23 microseconds


Fast Ethernet'603.15 microseconds


Source: HPC Challenge, November 2005.

LOW LATENCY: Both SGI, pictured, and Cray high-performance computers use proprietary interconnects.



'The interconnect is important because a lot of the applications rely on low latency and high bandwidth,' said Bill Kramer, NERSC general manager. 'We run highly parallel applications. One application may make use of 50 or 100 individual nodes.'


Because high-performance computing applications are increasingly spread out over so many processors, how fast they perform comes down in large part to how fast individual nodes can communicate with one another.


Not surprisingly, HPC interconnect makers are jockeying to show who has the speediest, most cost-effective technology for connecting nodes. The field is awash in different adapters, both proprietary and standards-based, and comparing them can be a challenge for even the most adept system architect. And confusing matters even more is the industry practice of tweaking performance results so speed figures look more competitive, and less indicative of what users may actually see. In short, finding the fastest HPC technology takes more than scanning the list of the Top 500 supercomputers (www.top500.org).


'You definitely have to understand what is being measured,' said Greg Thorson, principal engineer for platform development at SGI of Mountain View, Calif.
Shortly after PathScale Inc. of Mountain View, Calif., released its new InfiniPath networking adapter, it dispatched its distinguished scientist Greg Lindahl to give a presentation at a Beowulf Users Group meeting in Washington.


PathScale was not the first HPC interconnect vendor to court the group, which comprises a small but technically savvy collection of HPC system managers, including several from government agencies. SGI, Foundry Networks Inc. of San Jose, Calif., InfiniCon Systems Inc. of King of Prussia, Pa., and Voltaire Inc. of Billerica, Mass., also have presented to the Beowulf group. (Beowulf is a Linux-based clustering platform'and an Old English poem).


In addition to touting the benefits of his company's own interconnect, which is based on the InfiniBand standard, Lindahl warned of the potential pitfalls of comparing performance claims.


Hero numbers

When vendors market interconnects, Lindahl said, they typically use a pair of metrics'interconnect bandwidth and interconnect latency. Lindahl calls these 'hero numbers.' Bandwidth measures how much data a network can pass at a time; latency refers to the speed at which an interconnect can relay a packet of data.


'Things are changing in a way that makes latency more important,' Lindahl said, estimating that 30 percent of PathScale's customers and potential customers have applications that require low latency.


'Some applications only need bandwidth, but some applications send out a request for [a] piece of information, and until it comes back, they can't really do anything,' agreed Thorson.


What kind of latency do today's interconnects achieve? When PathScale started shipping its new interconnect last summer, it claimed a latency of 1.32 micro- seconds. SGI says its NUMALink has less than 1 microsecond of latency. According to the InfiniBand Trade Association, InfiniBand operates at 7 microseconds, while Gigabit Ethernet hits 66 microseconds.


The rule of thumb has been that proprietary interconnects, from the likes of Cray and SGI, have the lowest latencies. The downside to these interconnects is that they work only with Cray and SGI systems, respectively.


Interconnects built specifically for HPC environments, such as InfiniBand, Myrinet and Quadrics, also offer fast performance, but they still cost more than such slower, more widely deployed technologies as Gigabit Ethernet and the emerging 10-gigabit Ethernet.


The question that system architects must answer is how much weight they should place on performance figures when building an HPC system.


When measuring latency in fractions of milliseconds, small factors become very large. So it's not surprising that vendors look for ways to boost tested speeds by changing configurations.


'Latency numbers [can] vary dramatically depending on what you're measuring,' said Donald Becker, chief scientist at Scyld Software of Annapolis, Md. 'It's all how you can lie without saying anything that is demonstrably false.'


Apples and oranges

Latency numbers often are difficult to compare because they hide a lot of variation from interconnect to interconnect. One such variation is the type of notification the interconnect uses to alert the system that a data packet has arrived, Becker said. In an interrupt, or event-based mechanism, the adapter alerts the operating system when a new packet has arrived'an approach that is slower but uses a processor efficiently. Typically, this approach is used in 10-G Ethernet interconnects.


Another approach is called polling, in which the processor continuously monitors the memory space to check if a data transmission has been completed. Both InfiniBand and Myrinet typically use polling mechanisms.


'If your CPU is constantly polling, then you will get really low latency, but you're constantly burning up power, whereas if you're interrupt-driven, the CPU is only turned on when it receives notification that it has work to do,' Becker said.


Another source of variation is the size of the packets in a test environment. Most use zero-length packets, or packets with no payload. Lindahl insists that performance numbers based on these tiny packets do not scale evenly and may not convey true performance. Likewise, the number of nodes in a test environment can also be misleading. Most tests involve messages sent back and forth, ping-pong fashion, between two linked machines. Real-world systems have far more nodes.


Another assumption often implicit in performance tests is that neither the computer nor the network has other duties. In fact, a heavily loaded bus (the connection between the HPC card and the CPU) may cause delay as messages wait for right-of-way. Clearly, experts say, there needs to be a better way of measuring HPC performance.


DARPA to the rescue

One objective measure of performance, Lindahl and others have suggested, is the HPC Challenge Benchmark. Developed with Defense Advanced Research Projects Agency funding, the HPC Challenge Benchmark is a set of seven tests that measure the performance of supercomputing systems.


To measure latency, the framework uses the Random Ring Benchmark, which measures both bandwidth and latency. The test involves assembling a group of nodes into a ring network topology and averaging the time it takes a message to get around the ring. Multiple tests are run with random numbers of nodes, which minimizes the chance of the benchmark being gamed or equipment tweaked to one particular configuration. In this way, the test can provide a vendor-neutral way of comparing interconnects.


'If you want to quote a latency, you should publish the Random Ring Latency,' Lindahl said. (See sidebar for a rundown of recent HPC Challenge latency scores.)


All this talk of numbers, of course, obscures other, less quantitative, factors a system architect should ponder when choosing an interconnect. For instance, do you need cards that can be swapped in or out while the machine is up and running? Will the network interconnect scale to the number of nodes you plan to use? In some cases, higher latency may actually be preferable if it means reaping other benefits, such as lower power consumption.


'Interrupt mitigation deliberately adds additional latency in return for lower overall system load,' Becker said. 'That is almost always a good trade-off.'
For NERSC's new system, the decision came down to a number of factors. 'We didn't say we wanted InfiniBand or any other particular interconnect,' Kramer said.


Instead, the center put out a series of benchmarks and a test application. Vendors were asked to build a system to hit those metrics using any technology they chose, keeping in mind that the price of the system had to be as low as possible.


NERSC awarded the contract to Linux Networx Inc. of Bluffdale, Utah. The cluster system is capable of a theoretical peak performance of 3.1 teraflops, with InfiniBand interconnects running among its 722 AMD Opteron processors.


Michael Hall, senior director of customer care and fulfillment at Linux Networx, said there is no right or wrong answer when it comes to choosing an HPC interconnect. Some agencies will take the high-latency, low-cost interconnect, while others require top-notch speed at any cost. The important thing is to compare apples to apples.


'The price-performance balance is really what determines who wins the bill,' Hall said.

NEXT STORY: Let's get ready to FOSE!

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.