The 10 percent rule of system upgrades
Connecting state and local government leaders
Last week, Jeff O'Leary, who is the software development and acquisition manager for the Federal Aviation Administration, spoke at the at the 2007 SIGAda conference in Fairfax, Va.
Yes, the Ada programming language is alive and well. In fact, O'Leary was very bullish on using Ada for applications that require high margins of safety'watch out for a story GCN will run in the upcoming Dec. 11 issue on Ada.
Anyway, O'Leary oversees the software development portion of the En Route Automation Modernization system, an upgrade to the nation's air traffic control system. Lockheed Martin recently finished the software for ERAM ahead of schedule. That work involved writing over 1.2 million lines of code.
During his talk he mentioned an informal rule that could be useful for system managers in other agencies, one regarding upgrades of big systems.
The rule of thumb runs like this: If you plan to modify more than 10 percent of a system that must remain in operation, expect delays.
Generally speaking, with good planning you can change as much as 7 to 10 percent of a system and keep it running for users. Beyond 10 percent, however, the complexity rises to the point where its gets to unwieldy to manage.
O'Leary did not formulate this rule'he noted that a software consultant came up with it over 10 years ago, but it remains useful in the work that he does.
As an example, he pointed to The FAA's attempts to upgrade its systems in preparation for Y2K. In that case, about 11 percent of the system needed to be modified'a lot of the mainframe applications needed to be moved to Web servers, and the operating system needed to be changed out. The FAA completed the job, but there was one major delay--and a lot of sweaty brows--in the process.
O'Leary also applied this rule to the FAA's ill-fated Advanced Automation System. That was a totally new system, he mentioned, and switching systems like that is going to difficult at best.
After his talk, we caught up with him to have him explain this rule-of-thumb in a bit more detail. He was happy to oblige.
What goes wrong after an upgrade goes beyond 10 percent? Some of it is just technical, he explained. The more new code you have, the more bugs you will have. Some of it is organizational. If you change the user interface too much, end-users will require training.
To beat these odds, you need to upgrade your system in smaller chunks, he advised. Instead of doing one major upgrade, break it into smaller chunks. Test and implement a chunk at a time. That way, you can migrate to a new system more smoothly.
Although he was talking about major systems running on millions of lines of code that can not, under any circumstance, fail. But the rule of thumb is useful to keep in mind for systems of any size, really.
***
O'Leary also had a nifty rule on software reusability. The Defense Department has been pushing for software reusability for decades now. And it makes sense'why reuse code you already have?
Software reuse is tricky, however, simply because the secondary use of a software component is not usually exactly the same as what the software was originally written for, he explained. The component may have a few useful functions, but also may need a couple of extra functions for the second use. So the software will need additional modification. But too much modification and, in the end, it would have been more cost-effective to write the code from scratch.
What is that cut-off? About 20 to 25 percent, he said. If you have to change more than 20 percent of the code than figuring out how to modify the code in such a way that it remains safe but encompasses the additional features actually becomes more time-consuming than writing it from scratch.
NEXT STORY: DHS to hold IT science fair