Feds must weigh worth of vendors' CMM claims

Agencies often rely on Capability Maturity Model ratings touted by vendors when deciding whether to hire them for software development jobs. But are the rating claims made by many vendors legit? Maybe, maybe not.Since the Software Engineering Institute at Carnegie Mellon University began using the CMM to take the pulse of software development 15 years ago, the number of participating software shops has soared more than tenfold.SEI's 1993 software maturity profile cited 156 organizations, 65 percent of them government agencies or their contractors. The most recent profile--issued in March by the institute--cited 1,638 organizations, and only 31 percent were government agencies or their contractors.Despite the last 15 years' tremendous growth in commercial software, which now far overshadows government development, the Defense Department has always been the financial force behind SEI's model.The CMM for software ranks projects and organizations at levels 1 through 5. The higher levels are supposed to indicate fewer software defects, more repeatable development processes and better project management.DOD acquisition policy for major systems, such as weapons systems, requires contractors to have what is called "CMM Level 3 equivalence." Bidders that lack such credentials must submit risk-mitigation plans with their bids.Civilian acquisition officials are far less strict about equivalence ratings, even for projects costing hundreds of millions of dollars."Typical DOD IT projects in the $100 million range, which account for most of the problems and failures, are not covered" by the Level 3 equivalence requirement, said Lloyd K. Mosemann II, a former deputy assistant secretary of the Air Force and now a senior vice president with Science Applications International Corp. "At the very least, government [should] specify that the performing organization be Level 3" on the software CMM, Mosemann said in his keynote speech at the recent Software Technology Conference in Salt Lake City."Virtually every large DOD contractor can boast at least one organization at Level 4 or above, and several at Level 3," Mosemann said. "On the other hand, most DOD software is still being developed in less mature organizations, mainly because the program executive office or program manager doesn't demand that the part of the company that will actually build the software be Level 3."The economic pressure to obtain a prestigious Level 3, 4 or 5 rating has led to a proliferation of SEI and non-SEI models--not only for software but for acquisition, personnel, product development, systems integration and other areas.Only one software shop in 1993 had a strong enough grip on its development practices to reach the rarefied Level 5. In contrast, the latest list shows that 86 organizations say they have Level 5 certification--but SEI does not guarantee the accuracy of these claims.An SEI disclaimer, at , says, "This list of published maturity levels is by no means exhaustive."Why is that?Software quality assessment, like real estate appraisal, is partly a science and partly an art. SEI maintains a list of hundreds of appraisers, assessors and consultants who will undertake to rate software strengths and weaknesses according to the SEI model.That wide dispersion of authority, coupled with the enormous growth of the software industry, leaves SEI in the position of neither confirming nor denying the claims that are made using its model."As a federally funded research and development center, SEI must avoid any statement that might be perceived to validate or certify the assessment results that an organization chooses to make public," SEI spokesman Bill Pollak said. "The most we can do is to validate the conduct of an assessment--for example, 'An SEI-authorized lead assessor and trained team performed the assessment.' We do receive results from SEI-authorized lead assessors, but we keep those results confidential."SEI senior technical staff member Mary Beth Chrissis said there are "many different flavors of appraisals. Many other organizations have developed their own appraisals" based on SEI's public-domain model. A number of such organizations are offshore.Where does that leave agencies that want to make sure they hire competent contractors whose CMM certifications are current?A starting point is SEI's CMM for software acquisition (SA-CMM), developed in the mid-1990s by a government-industry team. Many agencies, including the General Accounting Office and the IRS, have used its methods to evaluate contracting or outsourcing practices. But there is no recommended list of SA-CMM vendors, and SEI does not qualify them. In choosing such vendors, SEI says, the key is experience: "The experience should be demonstrated and not just claimed."Mosemann, one of the instigators of the SA-CMM, said it was not meant to apply to contractors but rather to government program and acquisition offices."The problem that I perceived--and it clearly exists today--is that a gross mismatch occurs when a DOD program office that can barely spell the word 'software' oversees a Level 3 or 4 contractor organization," he said."The government program manager has no appreciation for the tools, techniques and methods--and their cost--that are necessary to develop software on a predictable schedule at a predictable cost with predictable performance results," Mosemann said. "That is why there is no list of SA-CMM contractors and why SEI has no plan to qualify them."Meanwhile, Defense is negotiating with its service acquisition executives to use SEI's newer CMM for integration, said Joe Jarzombek, deputy director for software-intensive systems in the Office of the Undersecretary of Defense for Acquisition, Technology and Logistics.Managers of major software programs are required to choose contractors that have succeeded at comparable systems and that have mature software development processes in place, Jarzombek said. DOD's present Software Development Capability Evaluation Core is equivalent to the software CMM Level 3 criteria, he said.