How to move a data center
The Utah Division of Technology Services team handing over the key to the building they vacated. Courtesy of Utah DTS
Connecting state and local government leaders
Utah’s Division of Technology Services pushed 25% of its applications to the cloud and moved 2,000 servers, the state network, databases and a mainframe from a soon-to-be demolished building into a new facility with a faster network, new equipment and environmental monitoring.
Last week, the Utah Division of Technology Services (DTS) completed a massive project that not many people noticed. It moved a data center.
DTS had less than a year – until Aug. 1 – to vacate a building behind the State Office Building, which is scheduled to be demolished, and into the Taylorsville State Office Building about 15 miles away. More than 350 employees from DTS, the Department of Government Operations and the Division of Facilities Construction and Management were involved in moving 2,000 servers, the state network, the phone system and call centers, databases and a mainframe as part of what the state dubbed the Salt Lake Data Center Exit project.
“The reason we called it ‘exit’ is because we had about 25% of our servers and workloads moved to the cloud – [Amazon Web Services] or [Google Cloud Platform],” said Dan Frei, DTS finance director and project lead. “It wasn’t just a pick up and move.”
And because it wasn’t, the team was able to build a new data center from the ground up with new equipment and many improvements. For instance, the old data center had a 10 gigabyte network connection; the new one is 100 gigabytes. Additionally, all the racks, power and voice equipment are new, plus the state added environmental monitoring that it didn’t have before.
“We didn’t back up a U-Haul, put a bunch of servers in it and move them down there,” Frei said. “We built all new, got it up and running…. Once you got everything working, it seemed to just be really slick.” Both data centers were running at the same time, until the team was sure the new one was operating as expected. Then the old data center was shut down.
What’s more, the state moved 25% of its applications to the cloud. The goal is to eventually have 75% of them in the cloud, but that will take a few years to achieve, Frei said. The applications that just moved included many related to health and human services, public safety, taxes, transportation and natural resources. One of the biggest applications handles resident eligibility for social services such as Medicaid.
Benefits of the move include better security, such as enhanced data encryption at rest, new secure private cloud and more robust firewalls. Additionally, the applications in the cloud now have better redundancy, he added.
“You usually need about six months of planning and at least 18 months to move a data center,” Frei said. “We did it in three months of planning and 10 or 11 months to move – and we did it on time and within budget,” he said on Aug. 3, two days after the team handed over the key to the old data center building at a ceremony celebrating the project’s end.
He attributes that success to top-level support and team alignment. “Our CIO that said, ‘This is our No. 1 priority and we’re going to make this happen’ and then we had all the teams come together,” Frei said.
It was also a monumental test of organizational skills, added Stephanie Weteling, DTS’ public information officer. “We had daily meetings. It was constant communication about what’s happening, what’s next, what will the impact be to the agency applications,” she said.
The effort wasn’t without its challenges, though, especially from supply-chain issues.
“We spent three months of planning this,” Frei said. “Day 1 of execution, most of our plans had to go out the window because of supply constraints.”
They worked around the items that weren’t available by selecting different models or refurbished technology.
“Security was a big challenge…. Some of the things that we had planned to buy, because it was so quick, when the security team got a better look at it, they said, ‘You know what, this is not going to work,’” Frei said, adding that the team quickly made adjustments. “In this whole move, the No. 1 thing we couldn’t have was a big security incident. It would have just sunk this project.”
Now that the move is finished, the team will assess lessons learned, but one aspect that Frei said worked particularly well was a best practice he copied from Arizona. He stood up a website that tracked each department’s progress with the move. A chart listed the percentage of the work each had completed. “We could do a little competition,” Frei said.
But overall, the results of the effort will get little notice. Applications will work faster because of speedier networks and services are less likely to go down because of redundancy, but mostly “it’s one of those projects that kind of flies under the radar. Nobody really knows the impact of it unless something goes wrong, and fortunately, we went through this really well,” Weteling said.
Stephanie Kanowitz is a freelance writer based in northern Virginia.