Hacker tried to poison Florida city’s water supply

As an employee at a water treatment plant watched, a hacker took control of his computer and changed chemical controls to dump lye into the drinking water of Oldsmar, Fla., a city of 15,000 near Tampa.

At about 8 a.m. on Feb. 5, a worker at the Oldsmar water treatment plant noticed that his computer was being remotely accessed by TeamViewer, a popular desktop control application that allows IT staff and supervisors to monitor operations and troubleshoot enterprise computers in remote locations. The worker “didn't think much of it,” Pinellas County Sheriff Bob Gualtieri said at a Feb. 8 news conference, because such remote access was not unusual.

The intruder returned later that same day, moving the employee’s mouse to open functions that control water treatment protocols, including one that adjusts the amount of sodium hydroxide, or lye, in the water. The hacker changed that level from about 100 parts per million to 11,100 parts per million, potentially endangering Oldsmar residents. Fortunately, the operator who was watching the intruder’s movements immediately reduced the chemical to the appropriate level and notified a supervisor.

Such attacks on utility control systems are not unusual, according to Lesley Carhart, a principal threat analyst at Dragos, an industrial control system security firm. Carhart told Wired that even unsophisticated hackers can find thousands of connected systems with tools like Shodan, a search engine that lets users find specific types of internet-connected devices.

According to Carhart, water treatment and sewage plants are vulnerable targets, especially during the pandemic when some workers are remote and IT staff are under-resourced. It’s usually the complexity and redundancies built into industrial control systems that prevent hackers from causing serious consequences, she said.

Oldsmar’s water treatment plant has several redundancies in place to catch unexpected changes.

“If you change the alkalinity level, the pH changes -- that would have been an alarm throughout the entire system,” City Manager Al Braithwaite said at the news conference. “So even if we hadn't noticed it right away, it would have [alerted] all of our people.”  

“Water systems like other public utility systems are part of the nation's critical infrastructure and can be vulnerable targets when someone desires to adversely affect public safety,” Gualtieri said.

“This type of hacking of critical infrastructure is not necessarily limited to just water supply systems. It can be anything -- it could be sewer systems, it could be a whole variety of things -- it could really be problematic,” Gualtieri said. “We want to make sure that we're paying close attention to all of it because it's not just an accident when you're taking it from 100 parts per million to 11,100 parts per million with a caustic substance.”

When asked at the press conference whether he considered the incident a bioterror attack, Gaultieri said: “You can put whatever label you want on it…. Somebody hacked into the system -- not just once but twice  -- and controlled the system, took control of the mouse, moved it around and opened the program and changed the levels from 100 to 11,100 parts per million with a caustic substance.”

“In order to get into the system, somebody had to use some pretty sophisticated ways of doing it,” he added.

Gualtieri said the city is asking that “all governmental entities within the Tampa Bay area with critical infrastructure components actively review their computer security protocols and make any necessary updates that are consistent with the most up-to-date practices.”

The Pinellas County Sheriff's Office’s digital forensics unit is working with the FBI and the Secret Service to determine how the breach occurred and who is behind it, but, so far, no suspects have been identified.

Redundancies in the system may have saved Oldsmar from tainted drinking water, but hackers can bypass existing detection algorithms, according to researchers at Penn State who have been simulating injecting false data into the insufficiently sensitive data detection frameworks many utilities use. “If a water tank is empty, we can change the reading to appear like a full tank, which changes the waterflow and pumps -- causing damage downstream in the water distribution network,” Javad Khazaei, assistant professor of electrical engineering at Penn State Harrisburg, told Penn State News.

Protection of critical infrastructure from cyberattacks is top of mind for the Cybersecurity Infrastructure and Security Agency, which issued a warning in July that urged all critical infrastructure sectors to be prepared for attacks on operational technology and reduce remote access to OT networks and devices. If such access is required, plant operators should ensure networks are segmented, data encrypted and traffic limited to known IP addresses.

"What we’ve seen exemplified here is that the need to understand and baseline normal in terms of critical asset/system access is absolutely key,” said Richard Cassidy, senior director of security strategy at Exabeam.

“We’ve got to ensure we’re monitoring OT systems far more diligently by capturing all viable log data in terms of access control, system settings and maintenance,” he said. “Any abnormality -- regardless of how small -- should be investigated, triaged and managed accordingly. Relying on users alone for the protection of our [critical national infrastructure] systems does not (and will not) scale.”

This week, the Federal Energy Regulatory Commission proposed a rule change allowing the federal government to subsidize electric companies that implement cybersecurity measures beyond the minimum standards required by current regulations. The policy allows for three categories of improvements: third-party hardware, software and computing and networking services, employee training to implement the upgrades, and costs associated with the implementation "such as risk assessments by third parties or internal system reviews," according to the Federal Register.