Crowdstrike debacle underlines single-point-of-failure risk

A blue Windows error message caused by the CrowdStrike software update is displayed on a screen in a bus shelter in Washington, D.C.

A blue Windows error message caused by the CrowdStrike software update is displayed on a screen in a bus shelter in Washington, D.C. Justin Sullivan via Getty Images

 

Connecting state and local government leaders

COMMENTARY | As our dependency on technology and energy increases, state and local leaders need to take a hard look at their disaster recovery and business plans.

At the height of the summer travel season last month, thousands of flights worldwide were halted and hundreds of thousands of travelers stranded for days all due to a faulty software update to a seemingly secure area within Microsoft operating systems by the cybersecurity firm CrowdStrike. The update impacted IT systems globally, affecting around 8.5 million Windows devices. Although the affected devices represented less than 1% of all Windows machines, the disruption was significant due to the critical services it upended. Aside from airlines, the outage impacted federal, state and local government entities.

The Crowdstrike debacle is far from the only example of technology failing. Just a few weeks later, the District of Columbia’s 911 Emergency Communications Center—the nation’s fourth largest by volume—was knocked offline for up to six hours because of a faulty software update.

Given the exponential growth in the use of technology permeating every aspect of business, government and our personal lives, we have become proportionally more vulnerable to catastrophic failure. Faulty software updates are but one example. Other negative forces threaten to disrupt a rather fragile digital infrastructure, such aging power networks and the growing risk of extreme weather like flooding, heat and high winds knocking out the electric grid.

Take, for instance, the Great New York City Blackout of 1977, which was caused by a series of lightning strikes that overwhelmed the grid. Or consider the Northeast Blackout of 2003, the largest blackout in North American history. It affected millions of people across the Northeast and Midwest, and was caused by a combination of factors, including a failure to address overloaded power lines and a series of equipment failures. The West and Southwest have also had their share of power disruption, such as The Blackout of 2011, the largest in California’s history, was caused by a technician’s error.  For around 12 hours, 2.7 million Americans had no electricity.

To add to the collective worry facing IT leaders, most state and local governments have increased their reliance on third-party vendors as they seek ways to reduce costs, supplement staff expertise and hopefully gain services they could not alone afford to provide. This trend toward greater dependence on “outside” expertise is itself a challenge.

All of this leads to what every technology leader worries about the most: a single point of failure, which even has its own acronym, SPOF.

Here are some common examples of SPOFs in technology systems:

  • One server running a critical application: If the server fails, the entire application becomes unavailable.

  • A lone network switch: If the switch connects multiple servers and fails, all those servers become inaccessible.

  • A single Internet service provider: Relying on just one for internet connectivity can lead to complete loss of internet access if that provider experiences an outage.

  • Single power source: Having only one power supply for critical equipment can lead to system-wide failure if that power source goes down.

  • One database: If a critical database is not replicated and fails, it can bring down all applications and services that depend on it.

  • A lone storage device: Relying on a single storage device or drive for important data without backups creates a SPOF.

  • Single network connection: Having only one network link between critical parts of the infrastructure can lead to isolation if that link fails.

  • A single firewall: If only one firewall protects the network and fails, the entire network becomes vulnerable.

  • One domain controller: In Windows environments, having only one domain controller can cause authentication and policy issues if it fails.

  • A solitary load balancer: If all traffic is routed through a single load balancer and it fails, it can disrupt access to all backend services.

  • Single cooling system: In data centers, relying on a single cooling system can lead to overheating and system shutdowns if it malfunctions.

  • One administrator or subject matter expert: When only one person knows how to manage or troubleshoot a critical system, their unavailability can become a SPOF.

  • A single vendor responsible for “everything”: A dependency on one vendor can lead to unexpected failure if the vendor itself faces a failure in an internal system or the execution of a standard operating procedure.

To mitigate these risks and more, governmental organizations should implement redundancy and failover mechanisms, and distribute critical components across multiple systems or locations. While no responsible tech leader would disagree with these approaches, many lament that, in practice, they often fall short in this area of preparedness. This is why regular risk assessments and system audits are always recommended to help identify potential SPOFs before actual problems arise. Perhaps the old adage is appropriate here, “Don’t place all your eggs in one basket.”

Redundancy and failover are the operative words when it comes to SPOFs. An example of this is server clustering. This involves multiple servers working together to provide the same service. If one server in the cluster fails, another server can take over its workload seamlessly, ensuring continuous availability of applications and services. This redundancy helps prevent downtime and data loss that could occur if a single server fails. Other approaches include shared storage with redundancy, network redundancy and geographical distribution.

We are only beginning to learn some important lessons from the CrowdSrike debacle. But clearly over-dependence and "overtrust” in one vendor is a paramount learning. This event is just another “wake-up call” that IT leaders and managers need to do a better job of planning, testing, documenting, training and conducting lifelike simulations. As our dependency on technology and energy increases, so does the need to actively reexamine disaster recovery and continuity of business operations plans. Such plans must be tested and updated and, as importantly, practiced.

At this moment, we are in the throes of a summer that has brought several record-breaking storms, it is hurricane season and our power grids are at capacity. When was the last time your jurisdiction did an SPOF analysis?

Dr. Alan R. Shark is the executive director of the Public Technology Institute (PTI) and Associate Professor for the Schar School of Policy and Government, George Mason University, where he is also an affiliate faculty member at the Center for Advancing Human-Machine Partnership (CAHMP). Shark is a National Academy of Public Administration Fellow and Co-Chair of the Standing Panel on Technology Leadership. Shark also hosts the bi-monthly podcast Sharkbytes.net. Dr. Shark acknowledges collaboration with generative AI in developing certain materials.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.