A 'Consumer Reports' for software vulnerabilities
Connecting state and local government leaders
The Cyber Independent Testing Lab will evaluate the security of commercial software so that users and purchasers can make quantitative comparisons among different products.
For years, organizations have struggled with understanding the potential weaknesses of the software they’re using, in part because there is no unbiased measure of comparison to help guide their choices.
But with the help of one emerging government-supported nonprofit, companies and government agencies alike may soon have a better handle on how software they own or are considering purchasing measures up in terms of security. During a presentation at last week’s Black Hat conference in Las Vegas, computer scientists Peiter Zatko (better known as Mudge) and Sarah Zatko discussed the independent organization they are building to impartially benchmark commercial software security flaws.
“All the certifications and evaluations that come out, they’re not about security,” said Sarah Zatko, who is chief scientist for the Cyber Independent Testing Lab and a member of the Army's Order of Thor, which recognizes contributions of cybersecurity professionals.
Meanwhile, more arcane and technical source code reviews do not help the average corporate or independent software user understand or evaluate the potential security flaws in their software. “Legislation is well-meaning,” she added, “but it typically focuses on making it illegal to look at this problem, and that is a terrible way to solve anything.”
Mudge, a longtime hacker and vulnerability specialist, left Google last year to launch and become the director of CITL after he received a call from the White House urging him to do so. A former program manager at the Defense Advanced Research Projects Agency and author of the password-cracking L0phtCrack software, Mudge is no stranger to the public- and private-sector struggles involved with evaluating the security of the software that organizations are using.
Although they are not planning to certify or offer any seals of approval on the software they test, the Zatkos said CITL aims to use a Consumer Reports-like methodology to evaluate the security of commercial software based on metrics and measurements that will allow laypeople to quantitatively compare different products.
In the year since they began their efforts, the pair and their team at CITL have been using a range of heurisitics that attackers typically use to determine whether software targets are hard or soft, meaning difficult or easy to break into with their exploits. Their metrics and testing are a combination of popular, widely known techniques and “esoteric tradecraft,” and they test software on the three most popular operating systems. So far, the lab has tested software quality and inherent vulnerabilities in more than 100,000 binary applications across Windows, Linux and OS X platforms.
So far, their work has confirmed some basic beliefs about the security of some products, particularly when run on particular operating systems, but it has also challenged conventional wisdom in some areas. Sometimes, CITL found, “the more secure product is actually ... cheaper, and quite often the [expensive] security product is the most vulnerable.”
It’s not surprising, as Mudge pointed out, that the “price tags for exploits mapped up nicely” with how hardened a product is – meaning exploits for hardened software are more expensive. “If we do this measurement for the office suites that are available,” he said, “you start to realize the level of effort an adversary has to go through.”
Mudge also noted the Microsoft Office Suite running on OS X is much softer (and open to compromise) than the same suite running on Windows, because the security Microsoft offers for its own platform is better than what is available when its software runs on competing operating systems.
CITL will start releasing its results in early 2017, the co-founders said.
NEXT STORY: How to prepare your network for the unexpected