The Florida Water Hack: Don’t Think ‘Redundancy’ Will Prevent the Next One

Photo by Bluewater Globe on Unsplash

By now, just about everybody has seen or heard at least the headline version of the story: A hacker tried to poison the drinking water supply of a small Florida town near Tampa earlier this month.

But as was also reported, the attack was detected and blocked long before there was any damage. A supervisor monitoring the Oldsmar (population 15,000) water plant systems saw a mouse pointer move across a screen and “immediately noticed the change in dosing amounts,” which could eventually have boosted the amount of sodium hydroxide (lye) in drinking water by 100 times. That caustic chemical, at low levels, is used to control acidity in water. In high concentrations, it amounts to drain cleaner.

A joint advisory from the FBI, the Cybersecurity and Infrastructure Security Agency, the Environmental Protection Agency, and the Multi-State Information Sharing and Analysis Center said the supervisor reversed the change immediately. And even if nobody had been watching, it would have taken 24 to 36 hours to affect the water supply and other plant safeguards would have detected and blocked the changes.

As Pinellas County Sheriff Bob Gualtieri and Oldsmar Mayor Eric Seidel said at a press conference, “redundancies” built into the system would have blocked changes of that magnitude anyway.

So the security measures in place worked. No harm no foul, right? Can we all get back to trying to jump the vaccination line?

Well, yes but …

The fact that there was no poisoning of Oldsmar’s water is obviously good news, but it shouldn’t give local officials, or water system operators anywhere, a sense that its security was adequate. The circumstances of the attack should actually prompt the managers of water systems all over the country — there are about 54,000 of them — to harden their security, by a lot. It should be a warning that it could have been a nightmare.

Let us count the reasons:

• On any scale of sophistication, this attack wouldn’t even make the JV team. The intruder(s) attacked during a workday, not the middle of the night when monitoring would have been less rigorous. They made no effort to hide. Anybody looking at the screen would have seen that someone from a remote location had taken over the controls.
• The attacker changed the level of sodium hydroxide by so much that the system couldn’t have handled that level of change anyway, according to Joe Weiss, managing partner at Applied Control Solutions and a control systems expert.
• It was easy to get into the system — very easy. Officials acknowledged that the plant was using outdated Windows 7 PCs that had not installed security updates for more than a year, and the staff all used the same password for the remote desktop sharing software application TeamViewer.

As industrial security consultant Patrick C. Miller put it on Twitter, “if you put an unprotected remote access tool on the internet that has a path to your critical stuff, expect it to get pwned. Best you can hope for is a noisy attacker.”

Rehan Bashir, managing consultant with the Synopsys Software Integrity Group, said the attack was enabled more by “people and process” weaknesses than technology vulnerabilities. “Having rigorous authentication systems will not prevent such a hack if users share their credentials among themselves,” he said.

• Sophisticated attackers would have had little to no trouble getting around the technology redundancies built into the system. That is what Stuxnet did more than a decade ago. The 2010 cyber attack on Iran’s nuclear program, allegedly a joint effort by the U.S. and Israel, destroyed nearly 1,000 uranium enrichment centrifuges by tricking the system controlling them to think they were fine when in reality they were spinning out of control.

And Bashir said redundancies are only “a corrective control” that don’t prevent or detect such attacks.

“Organizations need to implement preventative security measures/controls in accordance with their risk profile. They should use standards such as NIST (National Institute of Standards and Technology) and CIS (Center for Internet Security) benchmarks to enhance the security of OT environments,” he said.

• The water industry overall remains deliberately uninformed. There is little information sharing about cyber attacks among operators of critical infrastructure (like water supply) for multiple reasons: System operators don’t want to alarm the public, they want to avoid any possible liability risk and they don’t want attackers to know that their systems are vulnerable. That last reason is a version of the “security by obscurity” philosophy.

Also, there is no law mandating information sharing. While the America’s Water Infrastructure Act of 2018 requires water systems serving more than 3,300 people “to develop or update risk assessments and emergency response plans,” there’s no requirement to report cybersecurity incidents.

Andrew Hildick-Smith, a consultant who previously managed remote access systems for the Massachusetts Water Resources Authority, told security blogger Brian Krebs of a recent ransomware attack on an unnamed water utility that never became public.

That utility “made contact with the Water ISAC (Information Sharing and Analysis Center) and the FBI, but it certainly didn’t become a press event, and any lessons learned haven’t been shared with folks,” he said.

Indeed, Weiss said the most shocking thing about the Oldsmar attack was that it became public. “All these people are claiming this is the first time for an attack like this. That’s nuts,” he said. “It wasn’t first time and it won’t be the last.”

Weiss wrote last week in a post on his Unfettered blog that he has documented almost 100 control system water/wastewater cyber incidents throughout the U.S. and internationally, although “not all cases could be identified as cyber-related.”

“Many incidents were unintentional. However, the impacts could be devastating,” he wrote, citing a water utility that inadvertently pumped water from a Superfund well site into a drinking water system.

This can be fixed

If there is any good news for the industry at large here, it is that there is no great mystery about how to improve the security of industrial control systems (ICS) that operate water plants and other critical infrastructure. In other words, this is a problem that can be fixed.

There are recommendations from all directions, both public and private, most of them falling into the “security fundamentals” category.

Among those from the joint advisory from the FBI and other federal agencies:

• Update to the latest version of the operating system — in Oldsmar’s case, Windows 10.
• Use multifactor authentication.
• Use strong passwords to protect Remote Desktop Protocol (RDP) credentials.
• Ensure anti-virus, spam filters, and firewalls are up to date, properly configured and secure.
• Audit network configurations and isolate computer systems that cannot be updated.
• Audit the network for systems using RDP, close unused RDP ports, apply multifactor authentication wherever possible, and track RDP login attempts.
• Audit logs for all remote connection protocols.
• Train users to identify and report attempts at social engineering.
• Identify and suspend access of users exhibiting unusual activity.
• Install independent cyber-physical safety systems to limit any damage if the control system is compromised. These can include the size of the chemical pump and of the chemical reservoir, gearing on valves, and pressure switches. Measures like these can prevent an attacker from raising the pH in water to dangerous levels.

To that list, Bashir adds:

• Allocate necessary budget and resources for security.
• Segment the networks of information technology (IT) and operational technology (OT) environments
• Train employees in security awareness.
• Implement network access control to block unauthorized systems to connect to OT networks.
• Enable encryption between wireless devices that are communicating sensor data back to data historian servers.
• Implement vulnerability scanning in OT environment.
• Subscribe to various threat intelligence sources to stay informed about the latest vulnerabilities and issues pertaining to operating environments.

Of course, for any of these measures to be effective, they have to be done. And what is unsettling is that while awareness of security weaknesses in water supply systems, and what to do about them, have been known for a long time, few of those “best practices” are implemented.

It was 13 years ago, in March 2008, that the Water Sector Coordinating Council (WSCC) Cyber Security Working Group issued a report titled “Roadmap to Secure Control Systems in the Water Sector.” Its description of the security situation then sounds like it could have been written today:

In today’s highly dynamic and expanding digital economy, much of the ICS that operate our current water sector infrastructure are being used in ways that were never intended. Many ICS were designed decades ago with little or no consideration of cyber security. Increasing connectivity, the proliferation of access points, escalating system complexity, and wider use of common operating systems and platforms have all contributed to heightened security risks.

That report set out a list of four major goal for the water sector that it said were within reach within a decade — by 2018.

• ICS security programs that reflect changes in technologies, operations, standards, regulations, and threat environments. Water system operators “will have a thorough understanding of their current security posture, helping them to determine where ICS vulnerabilities exist and implement timely remediation.”
• A robust portfolio of recommended ICS security analysis tools to assess risk of vulnerabilities, leading to the development and deployment of appropriate mitigation measures.
• Cost-effective security solutions for legacy systems, new architecture designs, and secure communication methods. “Close collaboration among stakeholders and a strong and enduring commitment of resources will accelerate and sustain widespread adoption of ICS security practices over the long term,” the report said.
• Water asset owners and operators will be working collaboratively with government and sector stakeholders to accelerate security advances.

The WSCC did not respond to a request for comment on the status of those goals. But Weiss is just one of many who says close collaboration doesn’t exist. “Information sharing is broken,” he said.

Bashir said he is “not really sure if any actions were taken based on the roadmap.”

All of which could mean that at some point there will be a cyberattack on a water supply system that will cause catastrophic damage. And the public outcry will include the anguished question, “How could this happen?”

This is how, and why, it could happen.