Labor captures its current Web site
Connecting state and local government leaders
Labor's effort is aimed at providing greater depth and accessibility to the department's Web pages than a similar but much broader project being led by the Library of Congress.
If a picture is worth a thousand words, how do you calculate what a snapshot of a 1.6 million-page Web site is worth?
The Labor Department doesn’t claim to know the answer, but it has resolved that its Web site is valuable enough to create a comprehensive snapshot for posterity.
The effort is aimed at providing greater depth and accessibility to the department’s Web pages than a similar but much broader project that the Library of Congress is leading, said Patrick Pizzella, chief information officer and assistant secretary for administration and management at the Labor Department.
LOC and several partners embarked on a project that began in September 2008 to preserve a snapshot of U.S. government Web sites as the Bush administration drew to a close. The group, which includes the California Digital Library, the University of North Texas Libraries and the Government Printing Office, commissioned Internet Archive to create a permanent copy of federal Web sites.
But the massive effort to crawl, capture and index between 300 million and 400 million government Web pages, each with associated images and files, will take months to complete. As Pizzella and Paul Craven, Labor’s webmaster, discovered, there are limitations on what the LOC project will capture, the timing of the capture and the extent to which users can search the pages.
In the LOC project, agencies have no control over when their sites get crawled. Labor's snapshot gives the department control over dates and areas on the site. The department’s snapshot also will allow users to search for keywords and text, in addition to URLs and domains. In the LOC project, text search can be added on after the crawl.
Labor's Web sites, like those at most federal agencies, have expanded significantly in recent years. The sites host about 1.6 million pages of information, Craven said, and drew 117 million visitors and 521.6 million page views in fiscal 2008. The traffic volume has been growing about 9 percent annually during the past three years, he said.
“When we found out [the National Archives and Records Administration] wasn’t going to be archiving [government Web sites] this year,” Labor assigned a team to find ways to capture a wide range of programs that had evolved on the department's sites, Craven said.
“Our first approach was to see if we could do this in-house,” he said. “We looked at several open-source applications. What became apparent was they didn’t include all the capabilities…and didn’t provide the kind of support we needed.” And many would place unacceptable strains on the department’s Web systems, Craven said.
Craven and project manager Mark Randazzo eventually learned about LOC’s work with San Francisco-based Internet Archive and an on-demand service it offers, called Archive-It.
Internet Archive is a nonprofit organization that has gained popularity for its efforts to collect and preserve historical snapshots of the World Wide Web. Founded in 1996, it has archived more than 115 billion URLs and 3 petabytes of data, said Kristine Hanna, director of Web archiving services at Internet Archive. The Archive-It service evolved from those efforts.
The service agreement, announced by Labor earlier this month, lets the department archive as many as 5 million Web pages and related documents at a subscription rate of less than $10,000 a year, Craven said.
Internet Archive also is working with the Energy Department, the National Institutes of Health and the National Library of Medicine, Hanna said.
Behind the scenes, Internet Archive uses Heritrix, a Web crawler that can be controlled to capture digital content behind firewalls, and Wayback Machine, an access tool that reconnects and renders the content, Hanna said.
Archive-It also uses NutchWAX, which is open-source search software that can search the text of collected material and display the results, similar to traditional search engines. Hanna said Archive-It’s search capability will allow it to collect more material in the Labor snapshot than will generally be the case for the LOC’s broader government snapshot.
The Labor snapshot will be hosted and maintained by Internet Archive but will appear to the public on the department’s Web site and as part of its Wirtz Labor Library collection at www.library.dol.gov. All pages will have a disclaimer at the top that indicates the page is an archived copy. Copies of the documents will also be available to help meet NARA records management requirements, Craven said.
The snapshot is expected to be completed and initially available by the end of January. More information on the Web Snapshot Initiative at Labor is available at www.dol.gov/Archive.
NEXT STORY: Microsoft rolls out Web Platform Installer