Data disaster: When COOP is not enough

Disasters, both natural and man-made, drive home the need for agencies to ensure that data held on IT systems and devices remains accessible in order to support mission-critical operations.'In the wake of Katrina, I think you're going to see a greater focus on system integrity and data replication,' Chas Phillips, policy counsel for the House Government Reform Committee, said at an IT conference this fall.Continuity-of-operations plans'those that keep government going in the face of emergencies'are important, but far from foolproof. The fact is, the best-laid plans could be overwhelmed by unforeseen circumstances. And when that happens, COOP turns to disaster recovery. Maybe systems are still functional, maybe not. Even if you've protected most of your agency's data at a mirror site or on backup media, you probably haven't saved everything. How do you get that information back?Data recovery is easier said than done, experts say. Once tapes, disks and hard drives have been damaged, recovery becomes dicey and expensive.'It's going to take a long time and cost a lot of money,' possibly as much as $10,000, to extract usable data from a damaged $250 hard drive, said Mark Rasch, chief security counsel for Solutionary Inc. of Omaha, Neb. Rasch used to head up the Justice Department's computer crime unit.'The easiest way to recover data is not to lose it at all,' Rasch said. 'Did you make a backup?'But as many system administrators can testify, backup is not foolproof. In a recent survey by Asigra Inc. of Toronto, 75 percent of respondents said their organizations had lost backed-up data because of unreadable, lost or stolen media. Almost two-thirds of the respondents had run into unreadable backup tapes when trying to recover data.How difficult data recovery will be depends in part on the media you are dealing with. Floppy disks and diskettes can hold relatively little data, but even when mutilated they can be reconstructed using low-tech tools such as cellophane tape and glue.'We used to run an exercise for the Transportation Department at the Federal Law Enforcement Training Center,' Rasch said. At the end of the course, students were given damaged floppy disks containing their certificates of completion. 'If you couldn't recover it, you didn't graduate.'Hard drives hold a lot more data and present a greater challenge to recovery. Data on a hard drive is electro-mechanical'magnetic and stored on a spinning disk. Disk and reader have to be properly synchronized to read the information. If either is damaged, calibration can be a hard job.'It makes it more difficult, but not impossible,' Rasch said.One of the premier government shops for this kind of data recovery is the Defense Computer Forensics Lab.'We now have a hard-drive repair capability,' said Robert Renko, special agent with the Air Force Office of Special Investigations.OSI is the executive agency for the Computer Forensics Lab supporting the criminal investigative agencies of each military service, which have their own computer crime investigators.'We were stood up to handle the strange and complex cases,' Renko said. 'We've had everything in here from the typical laptop and desktop to cell phones, BlackBerrys and personal digital assistants, and Microsoft Xboxes and diver's watches.'There are two layers of data recovery for hard drives, logical and a physical. The logical layer involves the file allocation table, which allows an operating system to locate data on a disk.Removing the table is 'the same as taking the card catalog out of a library,' Renko said. The data remains, but access is difficult. There are forensic tools that can locate the data when operating systems cannot.Dealing with physically damaged drives can be trickier. If you left your notebook on the ground floor of a building that has been flooded, you might be out of luck. The circuit boards are useless, of course. The drives are sealed, but they have pressure equalization holes in them. 'They are not waterproof,' Renko said.The first step in bringing the data back to life would be to dry out the drive.'You can use a blow-dryer, or you can use calcium carbonate or some other desiccant,' Rasch said.But the water is not the real problem. 'The problem is the residue that remains,' which can scour, corrode and otherwise damage the hardware, Renko said. 'Rust is going to set in incredibly fast.'These problems can raise the cost of recovering hard drive data beyond the value of the data you're after.Cost-effectiveness is not the primary consideration in criminal investigations, but manpower and resources are being squeezed at the DOD forensics lab.'Our biggest challenge has turned out to be the growth of the storage capacity' of the devices being examined, Renko said.The gigabytes of data that can be stored on large servers can be time-consuming to recover and analyze, and some cases can involve hundreds of drives.The FBI also does forensic data recovery.'We've had limited success here' with hard drives, said an agent in the Cleveland field office. 'It depends on what it was subjected to.'Smaller devices such as PDAs can present different challenges.'They are supposedly dust-free and airtight, so the data should still be intact on the drive' after a catastrophic event such as a flood, the agent said. 'But once it is wet, the circuitry is gone, so getting the circuits to read it is a challenge.'Whether it is worth the effort depends on the value of the data being sought, he said. 'It might be worth trying.'Interestingly, new data-handling techniques not specifically designed for disaster recovery could ultimately apply to agencies trying to reconstruct critical information.Forensics investigators often must prepare data for use in a courtroom, which means it must be verified as accurate after it is recovered. The FBI has turned to the National Institute of Standards and Technology for help with this. NIST used its expertise in measurement to create high-resolution images of magnetic data that can tell an investigator when data has been written, erased or altered, said physicist David Pappas, project lead at NIST's labs in Boulder, Colo.The technique is called second harmonic magnetoresistive microscopy, and it uses powerful magnetic readers designed for server drives to image the fields on other magnetic media, such as tapes and disks.'You're actually taking a picture of the magnetic field above it, rather than just scanning it really fast and averaging the data,' Pappas said.This system works because it uses magnetic readers, designed to read very dense hard drives, to read much less-dense media such as magnetic tapes or strips.'The system we built to look at storage data was an answer looking for a problem,' Pappas said.The FBI provided the problem when it needed a way to validate the authenticity of evidence recovered from magnetic media. The image of the magnetic field can show erasure marks, like those you might find on a penciled message.The method can be used to noninvasively examine damaged and very short pieces of tape, such as those from a crashed airplane's flight data recorder. The resulting image also can be used to reconstruct and play back analog audio files from tapes.The Library of Congress also is interested in the technology.'They have thousands of tapes that have 'sticky shed syndrome,' ' in which old audio tape sticks to and peels apart on the heads of a tape player, Pappas said. NIST is doing a test to see if the technique can be used to recover data from these tapes without replaying them conventionally.'To do this data recovery, we're going to have to run the tape very slowly,' he said.Building large arrays of magnetic sensors than can read multiple tracks simultaneously might speed up the process.Unfortunately, there is no comparable way to build magnetic images from hard drives. The only certain way of making sure data from your hard drive is available, Pappas said, is to 'back up your hard drive.'