Florida turnpike shifts gears

High-speed switching technology once used primarily by telecommunications carriers has helped Florida's Transportation Department scrap an outdated toll operations network for a new, faster model. The network, built from the ground up, can handle more applications, provide robust recovery capabilities and meet future computing demands.

The department operates the second-largest toll road system in the country with about 600 miles of road and 800 toll lanes that collect about $800 million a year, all supported by an extensive network.

'We have a far-flung network' stretching from the town of Wildwood north of Orlando to Homestead near the southern tip of the state and across the state from east to west, said Kevin Palmer, program director at the Florida Turnpike.

Ten years ago, the most effective way to provide this network was with leased circuits.

'It has been an effective solution,' Palmer said. 'However, we are integrating new tolling technologies, and we are expanding our back-office activities both vertically and horizontally.' The turnpike is supported by redundant data centers in Orlando and Boca Raton, and the data centers are supporting more applications, including video and still imaging. 'That pushed us in the direction of having our own private network,' he added.

Revenue generator

The Florida Turnpike, which operates as a separate business unit of the Transportation Department, is implementing a new 10-Gigabit Ethernet Toll Operations network with Multiprotocol Label Switching.

MPLS supports multiple services and simplifies traffic management between the new fiber-optic backbone and remaining leased circuits.

It also provides robust failover and recovery in the event of a disaster. 'In a state like Florida, that is very important,' Palmer said, 'especially since tolls are the second-largest revenue generator' behind the state lottery.

MPLS also provides an avenue for growth.

'We didn't just look at current needs,' Palmer said. 'We looked at what we would need in the future.'

Only a few years ago, MPLS was an expensive, bleeding-edge technology used primarily by large service providers to provision multiple services, including applications such as voice and video that are sensitive to latency.

But the bleeding edge is not where most enterprises want to be. A turning point came in 2005, when affordable high-performance MPLS routers became available in a variety of sizes, said Ahmed Abdelhalim, product management director of the high-end business unit at Foundry Networks.

Today, instead of the bleeding edge, 'Florida is on the cutting edge of the technology in the enterprise world,' Abdelhalim said.

MPLS is a protocol that works between Layer 2, the data layer, and Layer 3, the network layer, in the Open Systems Interconnection stack. The protocol can speed data communication over IP networks in addition to those running on Asynchronous Transfer Mode and frame relay. It's often used to support converged voice, video and data over IP systems.

Label Edge Routers, which are entry and exit points on the MPLS networks, add MPLS headers to packets entering the network containing information about the packet's protocol, priority and level of service.

Routers and switches use this data in processing traffic on the network, and the headers are stripped as packets leave the network.

Information technology employees at the Florida Turnpike wanted the ability to segregate traffic from different applications on its network for quality of service and security.

MPLS also enables the merger of different sites at Layer 2 or 3 to create virtual private networks.

Groups of hosts can be linked across the wide-area network on a Layer 2 VPN, and different sites can be linked on a Layer 3 VPN as if they were connected by dedicated routers.

Do-it-yourself network

The turnpike's decision to build its own network was prompted by the fact that the Transportation Department was building its Intelligent Traffic System, which uses embedded sensors linked to fiber-optic cable along the roadway to monitor traffic and highway conditions.

'We realized that if we were going to lay the fiber along 400 miles of highway, we ought to leverage it for other operations,' Palmer said.

When a cost/benefit analysis was done on a new network, the fiber did not have to be factored into the equation because the department had already committed to that expense.

The turnpike did not give up its leased circuits entirely. They are used for disaster recovery and backup and to link several unconnected fiber segments. The network does least-cost routing, so 'under normal operating conditions, we aren't using the leased circuits very much,' Palmer said. But the two networks still need to coexist and route to each other without conflicts, and managing this is an art unto itself, he said. 'MPLS is a more eloquent solution. It has some intelligence built into it' and simplifies managing a heterogeneous environment.

Clear pictures

But it was not just the need to tie together 200 toll collection sites statewide that sold the Florida Turnpike on MPLS. 'If it were only simple financial transactions we were moving, we wouldn't need such an advanced network,' Palmer said.

Still cameras to catch violators are being installed in each of the 800 lanes where tolls are collected, he said. 'We are going to be capturing a high-quality image of every license plate that goes through.' With nearly 2 million motorists using the system each day, 'that is a fairly significant chunk of traffic moving on the network.'

Video also is being used to keep track of what is happening to that $800 million in yearly tolls at every step of the way from the collection booths to pick-up points and counting rooms.

Security and audit employees access that video through the network. The network also is used to manage the SunPass electronic system, through which motorists pay 65 percent of the state's tolls automatically. In addition to making those transactions via the network, motorists use it to manage their SunPass accounts.

The state has more than 2 million SunPass customers.

The department also needed to provide quick backup and failover between the two data centers, either of which can support the enterprise alone in the not-too-unlikely event that a hurricane or other event takes one off-line.

The turnpike is implementing its network with Foundry's NetIron MLX Series MPLS metro routers, which run the Multi-Service IronWare operating system, and the FastIron family of Layer 2 and 3 switches, according to the company.

The routers have a redundant switch fabric architecture ' combined with hardware redundancy in management modules, power supplies and cooling systems ' designed to keep the system running even in the event of a fabric card failure or other disruption. The NetIron MLX routers include MPLS Fast Re-Route, which enables failure recovery in less than 50 milliseconds, the company said.

When completed, the network will use about 100 new routers and switches. Planning for the new network began in 2006, and it now is operational along 400 miles of the main trunk of the turnpike from Wildwood to Homestead and on the 200-mile link between data centers.

'We're in the process of building out the rest,' Palmer said. 'What we have built is a robust fiber network that takes advantage of a portion of the dark fiber that was provided for us.'
Don't overlook staffing and training needs

Making the decision to build your own network rather than leasing one becomes a lot easier if someone hands you 400 miles of fiber-optic cable.

That was the situation Florida Turnpike officials found themselves in when they were wondering how to replace the turnpike's outdated toll operations network.

But even with a built-in infrastructure, the decision was not a no-brainer, said Kevin Palmer, the turnpike's program director. You still have to maintain the network.

'Our biggest lesson learned is that you should consider your maintenance plans early in the process rather than late,' Palmer said. 'We have transitioned from a leased network that was maintained by others to a privately owned network that we are responsible for operating and maintaining.'

The staffing costs should be included in the cost/benefit analysis. Fortunately for the Florida Turnpike, it already had much of the expertise it needed to assume those responsibilities.

'It's a good fit for us because we already maintained the toll collection system,' he said.

'Other agencies might not have the staffing for it.'

Training also is an issue to consider, especially if new technologies are being adopted.

The Florida Turnpike chose to build a Multiprotocol Label Switching network to ensure it would have the security, quality of service and management capabilities it would need in the future.

But MPLS traditionally has been a technology used by large service providers.

'The challenge for Florida was in the training of the staff for running the network,' said Ahmed Abdelhalim, director of product management at Foundry Networks' high-end business unit, which provided Florida with its routers and switches. 'It's new to the enterprise,' he added, although not so new that it was a difficult hurdle for administrators.