Managing the deluge

E-mail has generated a whole new category of electronic records.The messages have enormously variable sizes and difficult-to-classify subject matter, and can carry attachments, nonstandard formats and viruses. You can have uncertainty regarding their true origins and the true intended recipients. And it exists in a paradigm in which the precise sequence and time stamps of messages could be critical to placing a message's content in proper context.All of which has also created a new category of e-records management.E-records management includes workflow management, content management, version control, archiving, access control, backup, recovery and more.It covers photographs and document images, video, sound recordings, e-mail, documents captured through optical character recognition, text files, spreadsheets, instant messages, database records and even the contents of personal digital assistants. E-records can even be indexes and links to hard-copy locations such as bar-coded paper file folders or document storage boxes.Only records stored in traditional databases have been around long enough to have powerful legacy management systems.Every other category of e-record has received hit-or-miss management that varies from agency to agency; systems within a single office can change with management policies.E-mail in particular has often been managed, or mismanaged, by users or administrators who viewed it as a nuisance to be responded to, deleted, or saved when and if they got around to it.But regulatory changes and the implications of court decisions are driving a revolution in e-records management.When high-speed and high-volume document scanners were introduced, so were management tools, but most were tightly integrated with the hardware. What's new is the emergence of more tools intended to work with an existing IM infrastructure and deal with all forms of e-records.Because agencies have different retention needs, every IT department faces a unique and growing records management nightmare, especially when it comes to e-mail and IM.Rules regarding retention and even access to e-mail and IM are changing faster than other e-records policies, and the tools used to manage other e-records are often inadequate for dealing with the surging volume and diversity of e-mail records. E-mail could be by far the biggest e-records management challenge facing many offices.The accompanying chart includes some general e-records management tools, but the emphasis is on e-mail and IM.Before you even start to write specifications for e-records management software, you need to answer some very basic questions, with an eye on a changing regulatory and legal environment.Government agencies must first decide whether they can delete any electronic messages at all.Should you retain all legitimate messages or only the 'meaningful' ones? Who should have access to e-mail records?Should you keep spam? At first glance this seems simple; just delete the junk. Obviously, you want to filter it out so users don't have to see it, though dealing with spam costs a lot of money and laws regarding spam are changing. But if you don't maintain a historical record of spam, how can you support litigation to stop spammers, gain reimbursement for costs, or help prosecute illegal operations involving child porn-ography or blatant scams?Should spam be filtered out before it reaches the mail server? That would conserve resources, but then you can't record it for possible litigation, or even internal disciplinary action if the spam or pornography was invited.Will agencies someday be held legally responsible for failing to retain records of criminal wrongdoing on the Internet?How about retaining only legitimate and 'meaningful' messages? Just which ones are they?What is meaningful in this context? Is the simple 'OK' you send acknowledging receipt of a memo meaningful or meaningless? How about attachments or text included from the previous message? If there is ever a dispute over whether you got the message, then it's certainly meaningful.Enough examples'it should be obvious that e-mail and IM management is an incredibly complex task and, with some government organizations, such as Congress, receiving millions of messages every day, management of electronic messages is rapidly becoming the biggest challenge facing many IT departments.It might seem less expensive'and more legally defensible'to save every message rather than decide which ones should be retained, and then tweak software to properly categorize and delete unnecessary messages.A cynic might even foresee the day when, as Poe suggested in 'The Purloined Letter,' embarrassing information might be more easily and effectively hidden by placing it in 'plain sight,' so to speak, among billions or even trillions of meaningless junk-mail messages, than by classifying it.But as many corporations learn when presented with a subpoena to produce archived e-mail and other e-records, the costs of deleting spam and otherwise managing e-mail can pale in comparison with the cost of restoring, searching and printing out hundreds of millions of records.What do you do first? Establish detailed retention policies. Given fast processor speed, massive data pipes and unlimited storage, you can afford to be slipshod about establishing an e-mail management policy as long as you err on the side of keeping more and deleting less. In the real world, you need to settle on a policy that can carry your organization into the next decade.Government agencies face special problems that could require them to keep every e-mail message that hits the gateway. In addition to legal retention requirements, agencies might need to retain records so they can track e-mail attacks, employee misconduct, employee performance and abuse of message policies.Because e-mail volume will probably never decrease, merely seeing a demonstration of software working with 10, 50 or 100 messages won't really tell you anything.At minimum, you should try to see a demonstration of the e-records management software operating at two or three times the present traffic volume.Unless something meaningful is done to stop spam, the volume of e-mail hitting government servers could continue to double every year.Other things to consider: