Florida city water cyber incident allegedly caused by employee error
negaprion/Getty Images
Al Braithwaite, the former city manager of Oldsmar, Florida, said the 2021 episode was not caused by outside hackers but was instead a “non-event” sparked by user error.
In 2021, news broke of a cyberattack at the Oldsmar, Florida, water treatment plant, an event that sparked fears about the cyber vulnerabilities of critical infrastructure.
At the time, reports suggested that a worker at the plant saw his computer being remotely accessed and controlled. His mouse moved to open functions to control water treatment protocols, and then the amount of sodium hydroxide, or lye, in the water was changed from about 100 parts per million to 11,100 parts per million. The operator immediately reduced the chemical to the proper level and alerted a supervisor.
The alleged hack, which gained worldwide publicity from subsequent press conferences given by Pinellas County Sheriff Bob Gualtieri and other leading officials, prompted an investigation led by the FBI and the U.S. Secret Service, as well as a joint federal advisory warning water treatment facility operators of the dangers they faced from hackers and urging them to upgrade their security systems.
But according to one official who was with the city at the time, the incident was not a hack at all, just a case of an employee mistakenly clicking on the wrong buttons, before alerting his superiors to his error.
Former Oldsmar City Manager Al Braithwaite described it as a “non-event” that was resolved in two minutes, but said law enforcement and the media seized on the idea of a cyberattack and “ran with it.” The attention resulted in a four-month FBI investigation, which Braithwaite said reached the same conclusion that employee error was to blame.
“The FBI concluded there was nothing, no evidence of any access from the outside, and that it was likely the same employee that was purported to be a hero for catching it, was actually banging on his keyboard,” Braithwaite said in a March 20 panel discussion during the American Society for Public Administration’s Annual Conference.
A spokesperson with the FBI’s Tampa Field Office, which led the federal investigation in 2021, declined to comment on the investigation or on any conclusions it drew, citing restrictions under federal law.
Braithwaite said that the various investigations spawned by the incident, including one by the Florida Office of Information Technology, were particularly critical of the staff in Oldsmar, which he said runs its water treatment facility on a network made up of five computers and a couple of iPads.
“Our staff felt like they were being accused [by investigators] of being the criminals,” he said. Braithwaite said state officials came in “to identify our many—I admit it, many—vulnerabilities,” and to suggest ways they could remedy them. He added that the investigations were “extremely taxing” on staff.
As for the employee who made the error and then reported it to his supervisors, Braithwaite said he has not been fired, and nor should he have been.
“The employee did everything he was supposed to do,” he said. “I could say that they screwed up and that there was some kind of accountability that needed to be dished out.” Instead, Braithwaite said the responsibility rested with him as the city manager, as he was “responsible for everything that goes on there.”
“I wasn't, but I should have been held accountable for the fact that it happened, even if they discovered later on that it didn't really happen,” he added.
Other panelists said terminating employees for following standard operating procedures would set a dangerous precedent, especially given the staff shortages state and local governments have in tech and cybersecurity.
“If you think you have an employee shortage now, and it gets out that you're going to fire somebody over that, you're really going to have a big employee shortage,” said Karen Evans, executive director at the Cyber Readiness Institute and former federal CIO. “I'm not saying you don't talk to him [about mistakes], but I definitely say that you don't fire him.”
City leaders in Florida leery of copycat attacks on their systems had budget requests for more cybersecurity funding quickly approved after the incident in Oldsmar, Braithwaite said, which he said was one benefit of Gualtieri’s press conferences shining a light on the issue. But even now, Braithwaite said he still does not know how big a spend constitutes “good cybersecurity money,” as the threats are constantly changing and are less tangible than potholes or other physical infrastructure.