Intrusion protection systems: Buyer beware
Connecting state and local government leaders
A head-to-head test of IPS from major vendors found that performance often is overstated and users must tune the systems to get effective results.
How well do intrusion prevention systems really work?
A recent head-to-head evaluation of 15 IPS products against more than a thousand real-world exploits found that IPS effectiveness ranged from a high of nearly 90 percent to a low of 17.3 percent.
“I was a little surprised at how low some of the numbers are,” said Rick Moy, president of NSS Labs, an independent security research and testing organization that performed the tests.
Vendors overstated their products’ performance levels by 12 percent to 50 percent, the evaluation showed, and the systems rarely worked well out of the box with the default settings. Tuning the systems almost always produced significant improvements. “In some cases, you could get up to 44 percent improvement by tuning it,” Moy said. “The average improvement was 18 percent.”
The results were disappointing for a technology that has been mature for about five years and is often treated as a commodity.
“IPS is a critical protection layer for enterprise security,” Moy said. “This is not an indictment of IPS. It’s more of a wake-up call. Buyers need to demand more of them.”
Unlike intrusion detection systems, which — as the name implies — merely detect intrusions and attempted intrusions, intrusion prevention systems are intended to operate in-line and actively block attacks. Their ability to interfere with network traffic makes it important to tune the sensitivity of an IPS to balance false positives, produced when benign traffic is incorrectly identified as malicious, with false negatives, which occur when malicious traffic is incorrectly identified as benign, to suit the user’s needs.
No tool is perfect, and a false negative could allow an attack to penetrate a network. On the other hand, a false positive could interrupt legitimate traffic. Determining which is more critical depends on the circumstances.
NSS conducted its evaluation in the fourth quarter of last year and tested 15 products from seven vendors against 1,159 live exploits.
“There are probably 10,000 exploits we could be testing, but we are trying to focus on the ones that are the most important,” Moy said.
Vendors participating in the tests were Cisco Systems (one product), IBM (two products), Juniper Networks (three products), McAfee (two products), Sourcefire (one product), Stonesoft (three products) and TippingPoint (three products). Products were tested once at default or recommended settings and again as tuned by the vendor.
NSS has not released the individual results of the tests but sells a full report for $2,500 or product-specific reports for $600.
The general advice offered by Moy is that buyers need to consider the total cost of ownership of an IPS when selecting one. That includes the cost of installation, annual maintenance fees and the manpower it takes to keep it up and running, in addition to the upfront price. Products that are a bargain on the front-end price often have higher back-end costs that more than counter the initial bargain.
“You also need to understand the assets you are trying to protect and where you are going to put the product,” he said. Knowing the value of the assets, where they reside and how they are accessed can help you make a decision on which system will give you the most bang for your buck and how to best tune it.
And finally, “it is important to test,” he said. “If you don’t know what your IPS is doing, you could be in for a rude awakening.”
NEXT STORY: DARPA: Calling all cyber geneticists