How a city’s SCADA upgrade turned bandwidth into revenue

When the city of Piqua, Ohio, built a control network for its municipal power system in the early 1980s, it installed fiber optic cable connecting the power plant and substations.

“I believe we were ahead of the curve,” said Dean Burch, the city’s IT director. The technology was state of the art at the time, but after 30 years it was showing its age, and the network had become the weak link in the electrical distribution control system. “As the technology in power systems changed, we realized it wasn’t going to support us much longer,” Burch said.

When replacing the SCADA (Supervisory Control and Data Acquisition) network the city decided to include enough bandwidth to serve the rest of the city’s IT needs. For a minimal additional up-front investment the city hopes to save as much as $100,000 a year by replacing commercial network links.

There was even enough bandwidth left over to let the city partner with a local carrier to sell excess capacity to local businesses, turning the network into an economic development tool and generating revenue. The first business customer has been signed up, and the city is looking forward to the revenue shared with carrier.

Piqua, a city of a little under 21,000 in Western Ohio, is a forward-look community. It is one of about 80 cities in the state operating a municipal power system, ensuring reliability and reducing costs to customers, Burch said. For four years in the ’60s it even boasted the first municipally operated nuclear power plant in the nation, a short-lived experiment on which the Energy Department pulled the plug in 1966.

Planning to replace the aging SCADA network began around 2010, and in January 2013 a $1.7 million contract for construction of a new redundant single-mode fiber optic ring was awarded to Efacec Advanced Control Systems.

Fiber optic was not the only option for a SCADA network. Wireless also was a possibility, and some in the city questioned the expense of a fiber network. But there were several advantages to fiber. The city already had the supporting infrastructure, and replacing the fiber from an existing network would be less expensive than starting from scratch. Also, the same company that had been maintaining the network was available to install the new one, which would ease the process.

Finally, there was the issue of bandwidth. The added expense of additional fiber to support the city government’s IT needs made it a cost-effective option. Not only would it allow improved service to city departments, it also would replace costly telecom connections with outside vendors. The cost of pulling 72 fiber optic strands rather than 24 was only an additional 10 to 15 percent, Burch said, and the estimated savings for the first year of operation in 2014 were $45,000.

The amount of bandwidth delivered by this infrastructure depends on the optical networking equipment used. Some sections of the city now have gigabit/sec access, and in most of the network it is 100 Mbps.

The single-mode fiber used in the new network can carry multiple frequencies of light, and because it retains the fidelity of the light pulse better than the multi-mode fiber used in the old network, it can provide higher bandwidth over longer distances. Rates up to 10 Gbps are possible over 50 miles, and higher rates are possible with amplification and shorter distances.

Because the network serves the municipal power grid, it must meet security requirements from the Energy Department and regulators including the Federal Energy Regulatory Commission (FERC) and the North American Electric Reliability Corp. (NERC) “We have virtual networks and switches in place that meet the NERC and FERC regulations,” Burch said. SCADA traffic is segregated from other users on separate strands for security.

In the first phase of the project, a redundant 72-fiber ring was constructed to connect the city’s electrical services building with power substations, via a 144-fiber transverse segment connecting the IT and public safety offices at the city hall. Fiber optic switches and two SCADA servers were installed, and SCADA equipment was upgraded from serial communications to TCP/IP. This work was completed by the end of 2013. In phase two, additional city departments have been linked to the network.

The third phase will be the connection to the network of 20 key commercial customers for economic development. For this, Piqua has partnered with Independents Fiber Network, a local carrier that had received redevelopment funds to extend fiber service into rural areas. “We didn’t want to get into the Internet service provider business,” Burch said, so the city signed an agreement to provide commercial Internet connections through the carrier and share the revenue.

With an expected lifetime for the network infrastructure of 30 to 50 years and available bandwidth in the multi-gigabit range, Burch does not expect to have to oversee another major upgrade any time soon. “I think we are pretty well set for a while,” he said.