Virtualization breaks a high-performance computing barrier

 

Connecting state and local government leaders

A Defense Department research team introduces virtualization into the world of scientific simulations, leading to efficiency and cost savings in high-performance computing.

Virtualization is all the rage in most places, even for large organizations like federal data centers. However, for all the advantages virtualization can bring, there is one piece of the computing arena the technology has not been able to crack, until now: high-performance computing and simulation environments.

The processing requirements of most scientific computing call for  hefty hardware blocks with lots of CPUs and GPUs in massive racks supported by complicated cooling schemes. Computing tasks are carefully loaded into those systems, and competition for computing cycles can be both intense and political. Simulation applications have operating system requirements that are not often compatible with virtualized environments.

Almost nowhere are all the forces against virtualization at work quite so intensely than at the Johns Hopkins University Applied Physics Laboratory Air and Missile Defense Department's Combat Systems Development Facility.

Edmond DeMattia, a senior system engineer and virtualization architect at the facility, described its configuration, which is typical of many government simulation laboratories.

"We had two stovepipe systems with one running Windows and one running Linux," he said. "There are 1,500 cores per cluster, and everyone was sharing that computer using grid scheduling."

The tasks required of the system are intense. DeMattia explained that most simulations use the Monte Carlo method, a class of algorithms where repeated random sampling is inserted into equations to obtain concrete numerical results. That means that simulations need to be run many thousands of times in most cases.

"Some simulations take five seconds per task, and we run that same task up to a million times," DeMattia said. "While others may take 15 hours per task, but are only run 1,000 times."

The problem facing DeMattia and the lab is that while the computing requirements are intense, they are not used all the time. This is further complicated by working in different operating systems,  a necessity as some of the simulation jobs come from outside sources, and the lab has to accept their programming requirements.

However, it also means that in many instances, either the Windows or Linux stacks might be maxed out while the other was idle. To keep up with such demand or to expand capacity, most labs purchase more computers.

This leads to wasted resources when cycles aren't being used, even as more power, cooling and physical space are being called for. Faced with the situation, DeMattia believed there had to be a way to tap into those idle computing cycles, but it simply wouldn’t be possible using traditional computing methods.

To tackle the problem, DeMattia began to experiment with ESXi from VMware, a bare-metal hypervisor based on the VMkernel operating system that manages virtual machines running on top of it.

ESXi acts as the one part of the computing grid that knows its maximum limits. In using the tool, various operating systems don’t know the others exist. However, they operate as if they have double the computing nodes as they can tap into nodes normally reserved for another OS when they are available. The ESXi layer manages the entire system.

Turning a high-performance computing environment into a virtualized one was new territory for DeMattia. But with the added layer, he wasn't really expecting too much.

In fact, in doing the initial experiments, he was trying to figure out how much performance loss would be acceptable in the new system, offset by the gains of opening up the new nodes. "I had it figured that doing it this way would result in a 6 to 8 percent loss, which would have been acceptable," DeMattia said. "But I was shocked when we measured a 2 percent gain instead."

Once the optimal virtual grid configuration was established, DeMattia's team, which included lead automation engineer Irwin Reyes and security office systems administrator Valerie Simon, removed physical nodes from the stove-piped HPC grids and simultaneously incorporated them into the vGrid architecture.

The end result was a seamless migration of independent grids into a fully virtualized environment, an end result that more than doubled the usable CPU cores for each OS platform.

"My team fundamentally redesigned how high-performance scientific computing is performed in the Air and Missile Defense Department by utilizing virtualization and distributed storage as the framework for pooling resources across multiple departments," DeMattia said.

"By leveraging the ESXi abstraction layer, multiple stove-piped high-performance computing grids are aggregated into a single 3,728 core vGrid, hosting multiple operating systems and grid scheduling engines. This has allowed our engineers to achieve decreased simulation runtimes by an order of magnitude for many studies."

It took a while for others to appreciate what DeMattia's team had done: They had created a virtualized high-performance computing environment that could reduce idle computing cycles and run more efficiently at the same time. Even with a gain of just a few percentage points in performance, it represents a big jump when pushing millions of calculations through the system. Another significant advantage is the ability to tap into every node at the same time.

Back at the Air and Missile Defense Department, vGrid led to quite a cost savings. Instead of buying new computing nodes, which can run up to $10,000 each, IT managers can tap their existing infrastructure to its fullest. This has led to an estimated $504,000 savings in hardware costs as the need to buy additional computing resources to meet peak demand was eliminated. Beyond that, it would have cost another $40,000 per year to cool and power the expansion.

DeMattia said he believes that other high-performance simulation and government labs can make use of this new technology. Even though most have shied away from trying to take high-performance computing into a virtualized space, he said they are highly complementary technologies when deployed correctly.

"I get a little embarrassed when people ask me to talk about what we did," DeMattia said. "At its core, it's really just a simple process using technology in a way other than it was designed."

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.