L.A. Cyber Center Hopes to Be a Model for Cities Nationwide

LOS ANGELES — Four years ago, cybersecurity operations for the city of Los Angeles were divided between four centers that didn’t regularly share information with each other. When they did communicate, it was a managed through phone calls and emailed spreadsheets.

Cybersecurity awareness among the city’s 48,000 employees was mixed. Protections at the city’s 40 departments were hit or miss. Top department officials often didn’t know all the computer systems they were running, making it impossible to defend them.

Despite these deficiencies, L.A. was a high-tech city and believed it was reasonably well defended. “We thought we were secure, but we just didn’t know,” the city’s Chief Information Security Officer Timothy Lee told Nextgov this week.

The truth, Lee said, was that city computer systems were far from secure. When the city flipped the switch on a cyber scanning tool from the company FireEye in February 2015, it turned up about 15,000 instances of malware sitting on city systems.

Now, Los Angeles has become a case study for how a city can use models developed in the federal government and industry sectors to not only protect municipal networks but also improve cyber protections for local businesses.

Las Vegas sent about 40 city officials to examine L.A. cyber protections last week, Lee said, and Chicago officials visited this week. Officials from New York also plan to visit, he said.

At the heart of LA’s cybersecurity surge is its integrated strategic operations center, or ISOC, a bank of computers and human operators located in a small chunk of downtown L.A. office space next to the Los Angeles Police Department’s emergency response division and just a few blocks from City Hall.

The ISOC processes cyber threat information from the Homeland Security Department, the FBI and various private sector and non-profit sources and feeds it out to its member operations centers and to city departments.

Those four operations centers that formerly didn’t speak to each other—at the city’s IT office, the Water and Power Department, the Port of Los Angeles and Los Angeles International Airport—now all have precisely the same picture.

They’re also far less burdened by redundant busy work. Instead of each of the centers poring through thousands of raw threat indicators separately, the ISOC only forwards a handful of indicators that it has verified pose a danger to city systems, Lee said.

“Our overall security posture and situational awareness has improved dramatically,” he said.

Know Yourself; Know Your Enemy

Lee compares the ISOC’s mission to a lesson from the 5th century B.C. Chinese military strategist Sun Tzu in his treatise “The Art of War.”

“If you want to win the battle, you need to know your enemy and you need to know yourself,” Lee said. “‘Know yourself’ applied to cybersecurity is situational awareness, and ‘know your enemy’ is threat intelligence sharing.”

A bank of display monitors at the front of the ISOC demonstrated just how well the city now knows itself.

One screen tallied digital security events. That could mean anything from a phishing email sent to a city email address to a curious request to a city system.  

The figure typically hovers between 800 million and 1 billion events every 24 hours but was only around 300 million during the Monday morning when Nextgov spoke with Lee because hackers, like everyone else, prefer to take weekends off.

Another screen listed the countries these security events originated from. The U.S., Russia and China led the list, as usual, Monday morning with the U.S. on top. Attempts from Russia and China tend to rise during normal business hours in those countries and fall during their sleeping hours, he said.

Another screen tracked activity on city websites for possible attempts to overwhelm them with distributed denial of service attacks.

There had been 4.5 million failed attempts to log into city accounts that day, according to yet another screen. When that figure rises above 6 or 7 million, Lee begins to pay attention, he said.

One of the most important screens at the ISOC tracks activity on 104 particular city assets that are considered highly critical, such as its payroll system.

“Anything that targets those, we focus on that and we’re in an elevated threat space,” Lee said. Three of those systems were being targeted that Monday morning.

The ISOC monitors city networks using a system of sensors developed for state and local governments by the Homeland Security Department and based on the federal government’s own threat detection system called Einstein. The ISOC’s system, called Albert (get it?), detects malicious traffic coming in and out of city networks.

The ISOC also continuously monitors activity on employee computers and networks and receives alerts about anomalies that suggest someone other than a city employee is inside the system. Those alerts could come when someone accesses a system late at night, for example, or copies an excessively large number of files.

Knowing itself is only half the battle, though. The ISOC also struggles to know its enemy.

Lee’s office receives streams of threat data from the Homeland Security Department’s automated indicator sharing program, which includes threat intelligence from the government’s own sensors and intelligence services as well as information companies share with the government under a 2015 law that guarantees them legal indemnification for doing so.

The center also receives threat information from a government-backed cybersecurity information sharing program for state and local governments, known as the Multi-State Information Sharing and Analysis Center, and subscribes to a feed of private sector threat data.

The Homeland Security data is, by far, the most useful and voluminous data source, Lee said. He echoed a criticism made by private companies, though, that the Homeland Security data often lacks context that would make it easier to determine which threat indicators are most important and how they apply to city systems.

Securing the Community

In August last year, Los Angeles launched a cyber threat sharing initiative with the FBI and Secret Service, which investigates many financial cyber crimes. The initiative, called Cyber Lab, also includes the University of California, Los Angeles, the University of Southern California and California State University as well as numerous large businesses including Cisco and IBM.

In addition to sharing cyber threat information with each other, the consortium produces a feed of information that other organizations can subscribe to for free, including the city’s many small and medium-sized businesses.

Eventually, Cyber Lab hopes to shift to an automated threat sharing model similar to how Homeland Security shares threat information with top national companies, Lee said, rather than compiling and emailing data files.

The Weakest Link

All this security work, however, can’t overcome insecure employees. Lee’s office sent phony phishing emails to city employees in early 2016 to test who would open them. They were disheartened when about 40 percent of employees clicked the seemingly malicious links.

After a concerted retraining effort, the percentage of people clicking the email links dropped to 20 percent and then 10 percent and then further during the course of the year.

When a threat sneaks through these defenses, such as ransomware that an employee recently downloaded from her personal AOL email account, ISOC staff has some power to remotely lock users out of systems and can immediately share information with an agency about how to prevent the threat from spreading.

In the case of that ransomware attack, the attacker was able to move within about 20 seconds from the employee’s computer to a shared system and encrypt about 270,000 files. It took Lee’s staff and the department about 24 hours to restore those files from backups. It was one of about 40 ransomware attacks across seven departments the city suffered last year, he said.

After each significant attack, Lee’s office compiles a report that it shares back with the department staff describing precisely what happened in layman’s terms, moment by moment, and how to prevent it from happening again.

“I always train my team to not just do detection and investigation and remediation,” he said. “I also want them to be able to tell the story back to the customers so they really understand what’s going on.”