Without precise planning, metrics can be a minefield

In 1991, when Allan Burman was administrator of the Office of Federal Procurement Policy, he issued a bulletin starting the push for performance-based contracting. Since then, this contracting technique has had some success but proponents are still dissatisfied. In their view, agencies are missing many opportunities to use it.The basic concept is simple but powerful: Focus on results, rather than process; measure the results, and base compensation directly on these measurements.The hoped-for benefits are threefold:If it works properly, performance-based contracting is the classic win-win situation.So why aren't there more performance-based contracts?The procurement regulations go out of their way to encourage and enable them. An entire subpart of the Federal Acquisition Regulation, 37.6, deals with performance-based contracting. The FAR mandates performance-based contracting for most services acquisitions 'to the maximum extent practicable.' A standard clause for solicitations invites contractors to propose performance-based payments. And recently, the Defense Department required special approval for services buys that are not performance-based.To be sure, any new initiative takes time to implement, as people learn new ways. Understaffed procurement offices might be stretched too thin to learn and implement a new contracting technique. Performance-based contracting promises to save time and effort during contract administration, but it requires a significant up-front investment in planning and other preaward tasks.There is lots of help, however, available on government Web sites. And there is a cottage industry of consultants who offer their expertise'for a price, of course.My nominee for the biggest obstacle is the metrics themselves.Getting the measurements right is often tricky. One threshold issue is that proper metrics don't exist for many contracts. Without historical data, how does the government establish an appropriate baseline for performance? If it sets the bar too low, the government will pay a premium for mediocrity. If the standard is too high, the contractor hemorrhages losses. Either way, the incentive system won't work.So implementing performance-based contracting could involve collecting data now for some potential future contract action.Another problem is that some things are hard to measure well, especially if you aren't used to thinking numerically. Mathematician John Allen Paulos, who wrote a book titled Innumeracy: Mathematical Illiteracy and its Consequences, has documented that faulty statistics and poor computation are commonplace.A ray of hope here is the Office of Management and Budget's insistence on quantified business cases for IT projects. As I have pointed out, a byproduct of this process could be metrics that are useful in performance-based contracting.Bad metrics can be worse than none, as illustrated by the maxim, 'You get what you contract for.'One example I have seen was an ingenious, but flawed, scheme to rate contractors on collecting defaulted debts. The basic metric was determined by the money the contractor collected during a rating period, divided by the total dollar amount of bad debts it had to collect. The system not only ranked relative performance, it was used to allocate new work. Higher-ranked contractors got more new accounts.It didn't take the contractors long to realize that they were in a catch-22. If they got a high ranking, they would be swamped with new work, and their ratings for the next periods would plummet. The system actually penalized those contractors who expended time and effort to work on difficult accounts.To cope with this problem, some contractors found out which accounts would yield the most money with the least effort, and dumped the rest on their competitors by returning them to the agency. Those contractors who had learned how to play the system had better past performance ratings and won new contracts.Metrics can also break down when a contract is multifaceted. One contract for operating a large government facility has 87 separate metrics. Another such contract uses 109 separate performance measurements. Taking so many measurements doesn't necessarily save time and effort. But beyond this, it is hard to combine so many numbers in a way that provides clear and understandable incentives to the contractor. Does the contractor succeed by managing the work or by managing the numbers?And of course, institutional culture plays a role. Government officials enjoy the flexibility of subjective incentives such as award fees and the power they get from them. Contractors generally are more comfortable with a numerical formula. When they are investing in better performance, they want to be assured of a return on that investment, if it achieves a measurable success. Inserting the human element of subjectivity blunts the ability to predict a profit. And in the long run, contractors live or die by the profits they make.Which brings us to the latest policy initiative. In July, an OFPP study reported that making performance-based contracting successful required even more regulatory changes.The centerpiece of the proposal was a major redefinition of what qualifies as performance-based. Objective performance measures would no longer be necessary; subjective ones, such as customer satisfaction, would suffice. And incentives would become merely optional. The chain between objective metrics and contractor compensation would lose two links.If adopted, this proposal would sound a broad retreat from a program that promised to revolutionize the way the government does business. But the new rules may simply reflect that performance-based elements can be useful in contracting even if all of them don't fit all circumstances.