Softphones answer the call

During the San Diego wildfires last fall, employees at the California Department of Insurance call center had to work extended hours to handle the increased volume of calls.

A year earlier, they would have had to stay in the office after-hours in a rough area of Los Angeles, a dicey proposition for personal security, said Roy Simpson, CDI's chief information officer. What did they do instead? 'We rolled out the softphones,' Simpson said, referring to software that enables voice-over- IP telephony from a PC. Operators were able to plug into DSL lines at home to receive calls and access data, essentially taking the call center home with them.

At the time, CDI was in the final phases of replacing its telephone infrastructure with a VOIP system, California's first statewide implementation of Internet telephony.

'We weren't quite ready to roll the softphones out at the time, but it was one of those things we had to do,' Simpson said.

The VOIP system, which uses Cisco Call Manager and Enterprise Call Center software and runs on a Cisco network, received final acceptance in February and is fully deployed. New interactive voice response (IVR) systems are running in the Los Angeles and Sacramento call centers, and more than 1,500 IP phones have been installed in 14 locations, '15 if you count the warehouse in West Sacramento, which has two or three people,' Simpson said.

The $3 million program is a pilot for the state government, which is weighing the feasibility of moving its entire voice infrastructure to IP.

The results look good so far, although there have been some delays.

The project slipped by 15 months and required a $482,000 upgrade of the department's IP network, even though it already was in fairly good shape, said Tom Hagan, Cisco account manager. Part of the delay came from the need to map users' business requirements in offices statewide, which was a daunting task.

With more than $118 billion in direct premiums written in the state, CDI regulates the largest U.S. insurance market.

'It took months and thousands of man-hours to validate the requirements,' Hagan said. And they had to test the functionality of all features before deploying the system. 'You're talking about thousands of features,' he said. 'That was an enormous task.'

Under the bottom line

But implementation of the VOIP system came in 2.7 percent under the $3.1 million budgeted for it. In addition to increased functionality, the system will save an estimated $80,000 annually in long-distance calling and avoid another $111,000 a year in line lease payments for the Los Angeles call center. The elimination of three T1 lines connecting the department's major offices in Sacramento, San Francisco and Los Angeles saves another $5,000 a month, Simpson said.

All in one box

There are downsides, of course. 'I can't remember what my voice-mail password is because I never use it,' Simpson said. He accesses his voice mail through a Unified Messaging interface that combines voice, e-mail and faxes on his computer.

That also means that mailboxes fill up more quickly.

But he said it's a small price for the ability to select and listen to voice mail from an onscreen menu. And the IP network upgrade has let the department reduce travel costs by putting six videoconferencing sites online at its three main locations.

The job of replacing CDI's old Executone private branch exchange system was forced on the department, Simpson said.

'The problem went back to 2004, when we realized it was [at the] end of its life,' he said.

The Executone system dated to 1989. Vendor support was no longer available, and parts had to be scavenged from other departments or bought refurbished on the Internet.

The department wanted to replace it with something that would provide a hedge against obsolescence and improve functionality.

'We had a feeling it was going to be VOIP,' Simpson said. When they looked at what the industry was producing and what customers were installing, they saw a clear trend toward IP telephony edging out older circuit-switched systems. 'When you see a trend develop, you have to ask yourself why.'

One of the recommendations in the state's 2005 Performance Review Report was for one department to act as a pilot program for a statewide deployment of VOIP.

'We saw there was a perfect opportunity for us,' Simpson said, and his department took on the role of guinea pig and began reporting on its progress and challenges to a review group under the state CIO.

Control point

The first challenge was evaluating the existing network, which used circuits leased from AT&T under the CALNET II network contract from the General Services Department's Telecommunications Division. Cisco publishes a set of standards for networking equipment and software versions necessary to support its VOIP equipment, and the result of the analysis was a network upgrade.

'The routers and switches we had did not support [the] quality of service' necessary to ensure adequate voice quality, Simpson said. One more upgrade in the bandwidth was also necessary to ensure it could handle the volume of traffic, another step in an already ongoing process. 'We watched over the years the capacity on the triangle between Los Angeles, San Francisco and Sacramento get higher and higher.'

The CDI network now consists of T3 lines between the three main locations, with T1 lines linking another 11 satellite sites statewide.

There are 22 Cisco voice gateways deployed on the system.

'The control point for the entire system is the Call Manager,' which sets up and tears down calls and ensures quality of service for voice traffic across the network, Hagan said. This provides standard calling features such as hold and conference calls along with an enterprisewide look-up directory to enable four-digit dialing to any location.

The 1,550 IP phones come in five models, from executive to low-level, with varying features depending on the user's needs and status.

High-end models, for example, have programmable softkeys and a screen that gives access to the online directory.

The call centers are the heart of the department, Simpson said. The center in Sacramento serves the insurance industry, and the Los Angeles center handles consumer calls. Each receives about 450,000 calls a year. Cisco's call center software provides enhanced IVR that includes text-to-speech to help callers interact with the system, and it queues and directs calls. It also has customer relationship management software that identifies callers, associates them with records in the system and gives call center employees access to those records.

The call center software also enables continuity of operations in the event of a disaster by replicating each site in a partition on the other center's server.

'What happens if L.A. falls into the ocean?' Simpson asked. 'Now we know.' Operators from that center could work via the VOIP network from anywhere with their calls going to the Los Angeles partition on the Sacramento server.
The contract for the California Department of Insurance's Telecommunications Infrastructure Replacement Project to replace a 17-year-old private branch exchange phone system was awarded to AT&T in June 2006, and pilot programs began in March 2007. Full deployment in three phases began in July with final sign-off and acceptance in February 2008.

Results of the program include:

1. A distributed, scalable enterprise IP call processing system with high availability that supports more than 1,300 users across 14 offices statewide.
   
2. Extension mobility, letting users take their numbers and defined features anywhere by logging on to an IP phone.
   
3. Presence capability showing status ' such as available, busy or out-of-office ' of users on a contact list before initiating a call.
   
4. Automatic failover: If the wide-area network fails, the system automatically preserves calls and diverts them to the public switched telephone network.
   
5. Remote move/add/ change capability for quicker completion of telecom service requests.
   
6. Enhanced 9-1-1 capabilities to provide the street address and phone, cubicle or office floor a call came from.
   
7. Unified messaging, the integration of e-mail, voice mail and fax at the desktop, with retrieval through telephone and graphical interfaces.
   
8. Automated call directing for call centers in Los Angeles and Sacramento, including speech recognition, position-in-queue and waittime announcements, with supervisory monitoring and reporting capabilities.
   
9. Self-service interactive voice response application with speech recognition for insurance agents' licensing status and requirements integrated with an Oracle database.
   
10. Distributed call center system design providing failover and overload functionality.
    
11. Ability for call center agents to provide on-site support during disasters from the Disaster Assistance Center established by the Office of Emergency Preparedness.
    
12. protection of CDI investment from obsolescence.