WhiteHouse.gov: The case of the broken links

As Donald Trump stood on the steps of the Capitol Jan. 20 to make the peaceful transition of power official, there was another presidential transfer taking place online.

As Trump took the oath of office, WhiteHouse.gov transformed to reflect the new administration -- and quite literally became a shell of the Obama-era site.

A new homepage showed the new president, the "Issues" section updated to reflect six topics central to Trump's campaign, and bio pages for President Trump, Vice President Mike Pence and their wives.  But the briefing room was empty, sections on veterans and climate change disappeared and links to hundreds if not thousands of official documents suddenly stopped working. The entire Office of Management and Budget web presence -- a critical resource for IT policies and other nuts-and-bolts government operations -- was nowhere to be found.

News stories and social media quickly pointed to the missing issue pages as evidence of Trump's hostility to climate science, the LGBTQ community and other progressive issues.  Yet while the new president's priorities are inarguably different from former President Barack Obama's, WhiteHouse.gov reboot had more to do with server configurations and content management than ideological messaging.

Here's how it played out:

A long-planned reset

In the months leading up to Jan. 20, the National Archives and Records Administration worked closely with the White House Office of Digital Strategy to archive the digital footprint of the Obama administration.  And the Obama team made clear in an Oct. 31 blog post that the entire site would migrate to ObamaWhiteHouse.archives.gov. "The incoming White House will receive the WhiteHouse.gov domain and all content that has been posted to WhiteHouse.gov during the Obama administration will be archived with NARA," then-White House Deputy Chief Digital Officer Kori Schulman wrote at the time.   She reiterated those plans in a Jan. 17 post -- although in both cases the emphasis was on how one could stay connected to the outgoing administration, not on the future of WhiteHouse.gov.

Dana Allen-Greil, the web and social media branch chief at NARA, said her agency and White House staff started thinking about archiving Obama’s WhiteHouse.gov site about a year before Inauguration Day, but the work didn’t begin in earnest until about September 2016.

That’s when NARA began to ready the archival platform and start syncing the Obama site every night, she told GCN.

WhiteHouse.gov runs on the Drupal content management system. NARA’s archive site is also on Drupal for now, but Allen-Greil said it will be changed to a flat site within a few months. Keeping it on Drupal would require constant patching and would mean more security risk, she said.

Open.whitehouse.gov had to be brought onto a Drupal based platform too. It was previously powered by Socrata, for which NARA did not have a contract and couldn’t continue to maintain. 

Those migrations, by all accounts, worked well. Every page, image and file attachment from Obama's WhiteHouse.gov remains online at obamawhitehouse.archives.gov.  The archived site's own search engine works, and Google quickly began updating as well.  For much of the content -- blog posts and individual Briefing Room announcements of presidential actions, for example -- the old WhiteHouse.gov links automatically redirect to the archival site.

For other critical documents, however -- like the Commission on Enhancing National Cybersecurity's recently issued final report -- noon on Jan. 20 rendered links across the internet broken and useless.

The limits of redirects

When a web browser tries to load content from a URL, domain name servers use the domain (ie, WhiteHouse.gov) to steer the request to the appropriate web server.  Everything after the .gov (or .com, .org, etc.) is then handled by the web server itself.  And web servers can be configured to manage outdated links on the fly -- up to a point.

Most Drupal-based sites rely on open source Apache HTTP servers, which include a module called mod_rewrite.  (Other HTTP servers offer similar functionality.)  With some careful planning and creative use of variables, entire years' worth of URLs can be forwarded to new locations with a single line in the Apache configuration file.

That appears to be what the Obama digital team did.  (GCN reached out to the lead on this project at Obama’s White House Office of Digital Strategy but did not receive a reply.)  Any URL that begins with, say, whitehouse.gov/blog/2013/ is passed along to the obamawhitehouse.archives.gov domain.   But the main whitehouse.gov/blog URL, which presumably will be used again by the Trump White House, gets no such redirect.

Similarly, PDF files like the Cyber Commission report, which lived in a generic /sites/default/files/ directory on WhiteHouse.gov, were not redirected in bulk -- presumably because the Trump administration's files might also be uploaded to that location. 

Allen-Greil said the redirects that are working are already more than she expected.

“The redirect question is an interesting one,” she said. “We knew all of the short links that went out on social media using the go.wh.gov shortener would redirect to the archived version … But we were actually surprised to see that anything from Whitehouse.gov was redirecting to the frozen site.”

The White House team didn't share those plans with NARA, she said, even though they were talking regularly.

Dave Cole, who worked with the Obama transition in 2009 and then again as part of the White House new media team until 2011, said the disappearance of issue pages, like climate change, is “either a deliberate policy choice or just lack of proficiency in the work they're doing."

"I mean, you don’t miss the climate change page, you know that exists," he told GCN. "If you have nothing to say about climate change then that’s your message on your plan.”

As for setting up redirects, Cole -- who now works for MapBox, an open source mapping platform provider -- said it's a fairly easy process, though he wasn’t directly involved with it in 2009.

“It’s not technically hard,” he said. “You can set up a rule to redirect everything that doesn’t have a page on the new site to [redirect to] the old site, if that’s what you want to do."

Setting up such a redirect-everything rule, however, would steer any broken WhiteHouse.gov link to the Obama archive, regardless of whether an archival page exists.  That would not likely sit well with any current administration -- or provide a good user experience when the end result was a 404 error from a different site than a visitor originally sought.

But a site administrator also could have created redirects for each individual PDF that was removed from the /files/ directory.  And ideally, WhiteHouse.gov would have date-specific directories for uploaded files as well -- though that was not quite so easily done for a circa-2009 Drupal site.

"Technology is never the limiting factor in these decisions," Cole said.  "It's really policy and the communications objectives.”

Adam Ziegler, the managing director of Harvard’s Library Innovation Lab, said that archiving political websites and setting up redirects has become an important conversation within the research and scholarship community.

“So if you’re citing to a government website, a whitehouse.gov page, to demonstrate that the Obama administration took some action or made some statement at some point, and then that link ceases to work, then anyone reading your analysis in the future is not going to get what you intended them to get,” Ziegler said. “It is going to undermine your message, it is going to undermine the credibility and integrity of your work.”

The work on archiving presidential websites is an important step, he said, but a better, more comprehensive approach to combat "link rot" is needed. 

Some site have cropped up, including perma.cc, which Ziegler is involved with, to allow researchers to create archived versions of the websites they cite. This allows them to cite the archived link and not a link that could change due to ownership or change in administration.

Such solutions would also guard against the risk that redirects put in place on Jan. 20 might well stop working at some future date.  Mod_rewrite rules live in the configuration of the current WhiteHouse.gov server, and that configuration file can itself be modified or replaced.  Since that server is controlled by whatever digital team is currently in the White House, future changes (deliberate or accidental) could re-break the archival links in an instant.

Given that possibility, Ziegler suggested the other obvious solution.  If a website or research paper has links to resources from Obama’s whitehouse.gov that aren’t working, readers can always find and update the link themselves.

“For people who have relied on links in the past and those links are now broken they may find that their source has been archived,” he said. “They’ll have to make a change to the URL and point to a different spot, but the content itself is out there.  It's been archived and you just have to find it.”