Inside the evaluations

During his five years with the Office of Management and Budget, fall meant two things to Tony Frater: the start of the Green Bay Packers' season and the reading of IT business cases. Many hours of reading.The Madison, Wis., native and owner of one share of Packers' stock would spend most of his week reading and evaluating agency business cases before carving out three hours on Sunday to watch his beloved team.'I would rather stay up to 3 a.m. reading business cases than miss the game,' said Frater, who now is vice president at the Dutko Group of Washington. 'The end of September through Thanksgiving is the real crunch. You just know you will be swamped.'There are about 20 full-time OMB staff members and three or five federal employees detailed to OMB from other agencies who each fall dedicate their weekdays, weekends and family lives to evaluating and scoring more than 1,000 agency IT business cases.The business cases eventually go through one of the OMB director's two or three dozen review sessions that help shape part of the president's budget request, which is submitted to Congress in February.Former and current OMB officials and staff members variously describe this time of year as 'the low point of their OMB job,' 'the most pressurized time of the year' and 'not fun.'Why do feds continue to put up with the annual demands of this work?Their answers ranged from dedication to the job, to being public policy wonks, to the idea of suffering through the little bit of bad to get to the good.'There are other things that make working at OMB worthwhile,' said a former staff member who requested anonymity. 'We understood what has to get done and we were the ones who had to do it. It's a piece to the larger puzzle.'Agencies have submitted their business cases the first week after Labor Day since 1997, and each year they have gotten more complex. The new requirements have increased the time it takes to review and grade the business cases.The new complexity also re-quires officials and staff members to better understand the intricacies of cybersecurity, acquisition management and enterprise architecture.But at the same time, the added information has made the business cases more valuable, said Jonathan Breul, a former senior adviser to OMB's deputy director for management and currently a senior fellow at IBM's Center for the Business of Government.'Every budget season has its own feel to it and each agency's submission has its own quirks and strengths,' Breul said. 'The budget analysts are trying to do a thorough job and ask a lot of challenging questions that frequently cannot be answered.'Breul described the people who read business cases as highly trained experts in various areas, whether it is management, public policy or law.'They've done graduate work or often worked in the fields they are reviewing and have a deep interest in those subjects,' he said.Even though the reviewers are experts, business cases are read by dozens, if not hundreds of people, said one former OMB official.Karen Evans, OMB administrator for e-government and IT, said each business case is divided up among five teams of reviewers, four of them specializing in cybersecurity, acquisition, enterprise architecture or performance metrics, and one focusing on overall analysis.Each team is made up of at least one full-time OMB employee and one fed detailed from another agency.Evans said the teams score their sections using a 1 to 5 scale. The overall analysis team grades each business case in six other areas. Then the analysis team tallies up the scores to give the case a final grade.The business cases that pass'scores of 3, 4 or 5 can pass if the projects also have no weaknesses in security, project management or metrics'then are sent to OMB's resource management officer, who puts together information for OMB's director.Breul added that the examiners or RMOs must be able to make the agency's case to the director in a sentence or two.'The examiner must be able to tell the director why this project matters in plain English,' Breul said. 'The director does not have the time to go through the detailed scoring.'Many officials said it is important to make sure the business case summary is clear, concise and to the point. They also said the documents should not be a 'creative writing exercise,' or 'so long-winded, you wanted to throw yourself out the window.'Frater added agencies don't realize the importance of giving budget examiners good information that is easy to translate and can be used to justify the IT investment to the OMB director.Evans said many agency managers believe one person grades all their business cases.'Before I came to OMB, I didn't realize there was this great process to ensure consistency,' Evans said. 'When comparing scores among agencies, it really is apples to apples.'Evans said OMB is considering her recommendation that would require all career Senior Executive Service members to rotate through OMB on detail sometime during their careers to better understand the budget process.Each detailee or career staff person reads two to four agencies' business cases, which average 20 to 30 for smaller agencies and 60 to 80, sometimes even more, for larger ones. For instance, Evans said when she was the Energy Department's CIO, her department submitted 107 business cases.Officials said business cases took on average between 30 minutes and 60 minutes to read the first time through.'I would score it to start a dialogue with the agencies,' Frater said. 'I always wanted to get mine done quickly so I could give agencies time to fix any problems that arose.'A former OMB official said reading the first few paragraphs of the business case would be enough to indicate the quality of it.'The really bad business cases take no time to read and the really good ones take a little more time, but it is the ones that are in-between that take the most time,' the former official said.Business cases are not the first time budget examiners and OMB staff members hear of a project.OMB staff members often have read an agency's business case from the previous year and have been in contact with the agency project managers much of the year as a part of the normal oversight process, Frater and others said.'The budget examiner monitors the project quarterly, and I would meet with agency officials often,' Frater said. 'You didn't have to read every word of the business case because many times you knew so much about them to begin with.'That could be good or bad'another former official said evaluators sometimes hold a bias even before opening the business case because of preconceived expectations.The areas agencies need to concentrate on, according to former and current OMB officials, are the alternative analysis, performance goals, and metrics and lifecycle costing sections.Breul said examiners give more credit to business cases that meet agency and administration priorities or try to reduce duplicative programs.'The business case must show that everyone is involved in the business case, not just the CIOs and IT folks,' Breul said.Evans said, 'The business case is the first step in making a good argument for your project. If we are going to fund a project, we need to see the benefits and results that will be achieved. We really are focusing the discussion and evaluation of the business cases to ensure that money is well spent and the agency has defined success.'