NIST suggests areas for further security metrics research

Computer security is a difficult thing to quantify because, if done right, nothing happens. How, then, do you measure what didn’t happen?

Nevertheless, meaningful metrics are necessary so security can become a reliable, repeatable process with the necessary levels of assurance. The National Institute of Standards and Technology (NIST) doesn't have the answer for this, but scientists in its Computer Security Division have identified some areas for further research they hope might yield results.

“Security metrics is an area of computer security that has been receiving a good deal of attention lately,” the agency said in the draft of the new interagency report, titled “Directions in Security Metrics Research.” “It is not a new topic, but one which receives focused interest sporadically.”

So far, this interest has not produced many actual metrics that have proven useful in practice. “Advancing the state of scientifically sound, security measures and metrics would greatly aid the design, implementation, and operation of secure information systems,” the report states.

The scientists identified several factors that complicate the field of security metrics, some of which are merely difficult so solve, others of which might not be resolvable. Several of these factors are:

• The lack of good estimators of system security.
• The entrenched reliance on subjective, human, qualitative input.
• The protracted and delusive means commonly used to obtain measurements.
  
• The dearth of understanding and insight into the composition of security mechanisms.

The areas suggested for further research include:

Formal Models of Security Measurement and Metrics: “The absence of formal security models and other formalisms needed to improve the relevance of security metrics to deployed systems have hampered progress. Having formal models that depict security properties of operational IT systems and incorporate relevant objects of significance to system security measurement would be a useful contribution.”

Historical Data Collection and Analysis: “Predictive estimates of the security of software components and applications under consideration should be able to be drawn from historical data collected about the characteristics of other similar types of software and the vulnerabilities they experienced. At the very least, insight into security measurements would likely be gained by applying analytical techniques to such historical collections to identify trends and correlations, to discover unexpected relationships and to reveal other predictive interactions that may exist.”

Artificial Intelligence Assessment Techniques: “While the use of AI has met with both successes and defeats, its application in aspects of security metrics might prove beneficial, particularly as a means for reducing subjectivity and human involvement in performing security assessments.”

Practicable Concrete Measurement Methods: “The current practice of security assessment, best illustrated by lower level evaluations under the Common Criteria, emphasizes the soundness of the evaluation evidence of the design and the process used in developing a product over the soundness of the product implementation. The rationale is that without a correct and effective design and development process, a correct and effective implementation is not possible. While this is true, the emphasis on design and process evidence versus actual product software largely overshadows practical security concerns involving the implementation and deployment of operational systems.”

Intrinsically Measurable Components: “Development of computing components that are inherently attuned to measurement would be a significant improvement in the state of the art of security metrics.”

Comments on the report should be sent by March 27 to IR7564comments@nist.gov.

NIST also has released an online version of the annual report for the Computer Security Division (CSD) for fiscal 2008. Under the Federal Information Security Management Act, the CSD is responsible for providing agencies with standards, specifications and guidance in implementing requirements of the act. Toward that end, NIST issued 18 special publications offering management, operational and technical security guidance, and has updated several Federal Information Processing Standard publications covering hash algorithms and digital signatures.

NIST also has been collaborating with the Defense Department and the Office of the Director of National Intelligence to harmonize security requirements and programs for civilian agencies with national security systems, which to date have not fallen into NIST’s purview.

A final printed version of the report, with brighter colors, is expected to be published late this month or early next month.