NFC's preparation paid off when Katrina hit New Orleans

Hurricane Katrina was bearing down on New Orleans just as the National Finance Center prepared to add 60,000 Transportation Security Administration and Coast Guard employees to the payroll system.As the category 5 storm ap- proached the Big Easy, NFC di-rector Jerry Lohfink considered delaying the new payroll implementation'until he found out that the previous provider no longer had the ability to reconstitute those employee records in its database to process payroll.'Literally, we were at the point that if anyone was going to get these people paid, it had to be us. That was a solidifying force that our employees took to heart. It was us or nobody for these employees to get paid,' Lohfink said.This was the moment Lohfink and NFC'one of the government's four shared-service providers for payroll'had been practicing for: to keep more than a half-million federal employees' paychecks coming in the face of a crisis.Lohfink and NFC CIO Gil Hawk put the agency's continuity-of-operations plan into effect.'Having gone through various diverse scenarios in disaster recovery planning and exercises allowed us to have a firm grasp on what was needed to be done and in what order,' Lohfink said. 'It allowed us to quickly zero in on the most critical tasks and identify things that could be set aside for later. We did our single largest new customer implementation, and we only had half of a pay period to do it, because the other half was devoted to the storm.'Key staff members stayed over Saturday night and into Sunday at the NFC facility to complete payroll processing for federal employees, then shut down operations and fled the city for their backup locations before the hurricane slammed into New Orleans. Three trucks left town with 50,000 tape cassettes containing NFC back-up data. When the storm made landfall in Louisiana shortly after 6 a.m. on Monday, Aug. 29, NFC had already sent an advance deployment team to its Philadelphia and Grand Prairie, Texas, back-up sites.Late Monday night the trucks arrived at an alternate processing facility outside Philadelphia. The tapes let the New Orleans NFC staff restore many operations within two days of the devastation.'Within a 50-hour period, we had the data center recovered and began catch-up processing to bring our applications up to current state. When our customers logged on, it would be as if they had not missed any time,' said Lohfink, now located in Alexandria, La., about four hours from New Orleans.NFC had activated its disaster recovery service, which includes an alternate facility hosted by SunGard Data Systems Inc. of Wayne, Pa. Other business operations and daily customer interfaces take place at alternate sites in the District of Columbia; Kansas City, Mo.; Grand Prairie and Birmingham, Ala.NFC replicated and updated records to fill in gaps from the lag time of tapes journeying to Philadelphia in order to return systems and data to the same state they were when they turned off the lights, literally, in New Orleans.Tapes provided 95 percent of the disaster recovery. The remainder came from mirroring software from EMC Corp. of Hopkinton, Mass., as part of its hardware/software package. One segment of information, NFC's reporting center, is mirrored every two hours, and the system is completely updated every 24 hours. Other systems are updated every 48 hours, Hawk said.The toughest challenge early on was voice communication, followed closely by e-mail connectivity, Hawk said. All the disaster recovery cell phones had the New Orleans 504 area code, which was put out of commission. Lohfink had set up a 24-hour 800-number conference-call capability, which senior leaders used repeatedly each day to discuss status and planning.'As we go back over lessons learned, we're going to have to get telecommunications'not only us but FEMA and everyone else'to have cell phones out of our own area code,' he said.Personal e-mail accounts worked better than the NFC accounts. NFC may consider an Internet service provider for emergency accounts.'Not having gone through a disaster but doing the drills, you don't think of these kinds of things until the communications infrastructure gets wiped out,' Hawk said.Agriculture has had plans to move NFC's data center equipment and co-locate it at the department's National IT Center in Kansas City, Mo. Those plans cannot be accelerated any faster than they are already, Lohfink said.NFC would remotely manage its data center from New Orleans and convert its New Orleans center to a backup and disaster recovery operation. The plan is designed to remove NFC's physical assets used in daily operations from the path of frequent hurricanes and flooding.Although floodwaters approached NFC's headquarters, there was only mild damage. Many NFC employees are back at New Orleans headquarters. Lohfink is uncertain when all NFC employees will return.Technology allowed the agency to have the data to restore operations, but it was the sacrifice of NFC personnel, who traveled to several alternate locations and worked long hours apart from their families, that made the difference.'They accomplished the job regardless of the odds. Virtually every employee was impacted in some way and scattered because of the evacuation,' Lohfink said.