Chicago adjusts on the fly to make its contract perform

What's one way to make performance-based IT acquisition work? Keep it simple.That's the message from Chris O'Brien, the city of Chicago's CIO.'Look at the overall mission of your agency, and from that develop a couple of key performance measures'find things that are measurable, meaningful and have real impact on the bottom line,' he said.O'Brien speaks from hard-nosed experience.Four years ago, the city awarded a $75 million contract to Unisys Corp. for managed IT infrastructure services.Under the contract, Unisys assumed responsibility for help desk, desktop and network management, including desktop configuration, asset management, maintenance, remote network management and desktop training services.Unisys set up a service desk to provide a single point-of-contact for all service incidents and support requests from the city's 16,000 employees and 40 departments.Unisys also established a dedicated program-management office in Chicago to manage its on-site technicians and support its service desk, which operates out of the company's managed service center in Texas.But after the first year, all was not entirely well. O'Brien began to realize that the help-desk services weren't as effective enough. For instance, there were too many repeat trouble tickets, indicating that end-user problems weren't being completely fixed.'We were obsessed with measuring everything,' O'Brien said. 'Our original contract contained a lot of what I would now consider process measures.'Under the contract's initial service-level agreements, Chicago micromanaged Unisys' help desk performance using a meticulous problem-resolution format.Metrics included the number of calls to the help desk and the time it took the vendor's technicians to close a call or fix a problem.By monitoring the vendor's technicians on the number of calls they closed, the city created an incentive for Unisys' technicians to 'get in there, make a few quick fixes and get out as quickly as they could, perhaps not addressing the problem in its entirety,' O'Brien said.It dawned on city officials that an important outcome was missing. 'After the first year, we realized that we were not measuring the ultimate objective we had, which really was user satisfaction,' he said.On the vendor side, Unisys officials recognized the problem too.'A key performance indicator is customer satisfaction,' said Everett Dyer, general manager of global infrastructure services North American for Unisys. 'We didn't have that in the first contract. But the evolution of our understanding in this engagement is that satisfaction is important to the clients' employees.'Officials from Chicago and Unisys then got together and revamped the contract, making user satisfaction the city's principal outcome and letting Unisys worry about service-level metrics.'We retooled our measurement strategy so that we stopped worrying about a lot of nuts-and-bolts measurements and really measured outcomes,' O'Brien said. 'In terms of what we were most interested in measuring Unisys on, satisfaction was really the key indicator.'He added: 'As the CIO, I particularly don't care how long it takes to close a call, unless the customer is unhappy. Closed-call rates are more of value to the vendor; they help the vendor manage their workload.'The CIO's office routinely sends an e-mail survey to end users after each trouble ticket is closed, to determine users' level of satisfaction. O'Brien said current surveys show that about 93 percent of users are very satisfied or satisfied with the help-desk servicesUnisys' willingness to change course on measurements a year into the contract underscores how crucial flexibility is in a performance-based contract, O'Brien said.'When we changed our focus from time-based measures to satisfaction-based measures, Unisys was right there with us,' he said. 'That kind of flexibility makes it a lot easier to maintain a partnership.'In general, Dyer said, managing the relationship, not the contract per se, is the main ingredient in successful partnership.'If you manage by the contract, the relationship tends not to work,' he said. 'You need the contract'it describes certain things that everybody agrees to'but if you have to manage day-to-day by pulling out the contract, you're really not going to make a lot of progress.'Chicago and Unisys recently renewed the contract at a value of about $57 million over five years, Dyer said.'The city of Chicago has been a dream client,' Dyer said. 'In a performance-based contract, you want a client that doesn't want to get down into the bowels of how we're doing something. If that happens, then it becomes a discussion of what we're doing and not the results we're getting.'O'Brien offered this advice to managers struggling to develop outcomes and measures: Keep the number of measurements to two or three, and make them items that are practical to measure and that drive your ultimate objectives and mission.'Keep it simple, because if you don't you can bury yourself in different kinds of measurements,' he said. 'Obviously, vendors will have all kinds of measures to help manage their workload, cost and resources, but what government is doing in a performance-based contract is getting out of that business.'In performance-based contracting, 'the vendor's expertise is in the doing, and our expertise is in communicating our key objectives and holding them to that,' O'Brien said.