Cybersecurity In Industrial Control Systems: The Evolving Threat Landscape
So far in my IT career, I have worked in or done work for several chemical plants and factory locations. These locations all had very different operations, procedures, culture, and people. However, the one thing that was the same throughout all of these locations was the security culture and technology, or lack of it.
Usually, the places I have worked at are all owned by large, multi-national energy or manufacturing companies, and all of these have a corporate environment. In most of these places, IT is managed top-down in a waterfall fashion. You know the type: you need to submit a help desk ticket to some team 3 countries away just to get access to SharePoint or some other trivial issue.
During one of my previous jobs, I worked at chemical plant, and my main day to day duties were managing the non-control servers and infrastructure (the office network).
Because of these methods, I often had to preform, what a cybersecurity analyst would consider, trivial and unnecessary work. This included everything from using complex, clunky compliance applications that only got worse with every update, tedious and repetitive “compliance” tasks that did exactly nothing to secure systems, and lastly and most defiantly least: writing pointless comments on every task that I preformed just for the sake of “visibility”.
Also, there was little to no user or even admin training regarding threats, and what little security training was issued involved completely irrelevant or absurd threat scenarios.
Obviously, this is not the approach that companies should be taking for securing their systems, especially in environments where a network breach, or compromised control system can cost someone their life.
“As an admin of the office network, if you make a mistake, users will just lose connection to YouTube. If the control system engineer next door makes a mistake, someone might die.”
Security Starts At the Top
Now, let’s dive in to the messy inter-workings surrounding enterprise and control system security. Let’s start where cybersecurity originates: at the top.
I’m not going to get into the whole “someone isn't qualified to make security policy” debate as this post focuses on the technical aspect. This however, is a re-occurring aspect in large companies: CIOs and CISOs get hired only because of who they know, or their reputation in the company, and not necessarily because of what they know.
Security and IT knowledge should be passed from top to bottom, and everyone in the company from the CIO down to the contracted night-time janitor should be aware of all security procedures. This is often not done due to the costs required for developing programs and training personnel. However, a company can solve the cost issue by simply controlling costs. Don’t let vendors charge you premium prices just because you are a large company, use open source training platforms, and hire a few experienced security professionals instead of a large firm that is only experienced in training systems.
Blanket Policy Doesn’t Work
No matter what corporate or some security solutions vendor tells you, cybersecurity does not have a one size fits all solution. Security is different for every team, every office, and every service. You simply cannot purchase a product, set up a rule, and press “apply to all”. It doesn't work that way. Each business unit or team may have their own way of handling data, day to day operations, and their own security practices. Here is where we get into the more complex issue of standardizing operations between business units and team in various locations. That is a paradox of corporate policy that can have it’s own post. For now let’s assume that the standards in the company are not 100% the same across every site, as most are not.
Here are some examples of blanket policy mistakes to avoid:
- Rolling out the same authentication policy for every user in the company
- Locking out all removable media
- Pretty much anything else that includes the word “all”.
Bottom line: configure applications smartly, use alternate authentication, adjust the corporate mindset.
Threats Are Everywhere
A typical threat can come from any direction. Most large companies only focus on traffic ingress points in their network, the WAN connections, remote links, etc. However, a threat can come from just about anywhere.
Here are some of the most common ways that threats spread in control system networks:
- A remote network intrusion from an external threat actor that release a payload into the top level network which then makes it’s way into the CSN.
- Phishing or any form of email based delivery methods containing trojans or worms.
- A wireless network intrusion containing the initial payload to compromise a single machine and use that as a beachhead to launch other attacks.
- A payload that infects the public, internal office network, or CSN adjacent networks and is copied unintentionally via automated data transfer systems to the CSN.
- Direct internal intrusion where a threat actor (or unknowing employee) infects a machine via removable media. (This is one of the few attacks that can directly infect the CSN).
- Remote RF intrusions where radio frequency systems are compromised and used as beachheads for launching other attacks on adjacent networks. These attacks can also compromise air-gaped systems or systems that are not directly connected to other systems or networks.
- A device running Bluetooth is compromised remotely and is used as a staging point for launching attacks on the wireless network it is connected to. This attack can happen anywhere, and the victim brings their infected device to work.
- A relatively new attack that involves using sophisticated remote sensing equipment to sniff traffic flowing through UTP cables via completely wireless and off-site attacks.
Because of the varied nature of these attacks, a one size fits all solution will not work as each attack requires a different mitigation strategy. Most of these attacks do have something in common: people. People are the first and last line of defense for many of these attacks. Empowering the workforce can be separated into two categories: training and educating the users and empowering your IT staff.
Protecting Against Threats
Before we dive into the human aspects of cybersecurity, let’s look at the technical stuff.
Note that the mitigation actions for these attacks can vary. The ones listed here are just the ones I have implemented, simulated, or have seen implemented.
Here are the common mitigations that apply to all of these attacks:
- Visibility is critical. Network administrators, system administrators, and security analysts should always have visibility into all of the systems that they are responsible for. All data should flow into one large data pool which can then be mined or extracted by various tools. We need to get out of the spreadsheet and report generation era. These tools should be abandoned for “all in one place” solutions. An example of this would be to use the ELK stack along with Grafana to collect device and application logs and analyze them in real-time with applied machine learning to create information-rich dashboards that are simple to read. Most companies have some various mess of different products that all somehow attempt to talk to eachother, cost a fortune, and provide limited visibility. *cough* Solarwinds *cough*.
- Adding to the visibility aspect, administrators should be able to have visibility into every layer of the stack, from the front-end application, all the way down to the equipment providing power to the server that the application runs on. Equally as important, all of this should happen on a single display or at least in a single system.
- Apply modern technologies to security. One of the most important reasons to have visibility into the entire stack is data correlation. Machine learning can be applied to the entire dataset and software such as Kibana can then display the correlated results and give the administrator or analyst a much better picture of an issue. This can not only help in security but also in system reliability as well, and reduce the workload for system administrators at the same time. An example of this would be: the system detects a failure of a datacenter air handler, which causes a nearby server that handles storage to overheat and thermal throttle, as a result, the onboard CPU reduces it’s speed, which affects the performance of the storage platform, in turn affecting the performance of an application that is using the attached storage. The events would be displayed for each one of these failures, combined together, and produce a single root cause message for the administrator.
- Train your administrators and users. Develop custom training programs that demonstrate these attacks, what they can do, and how to protect against them.
- Use common sense and care about security. This sounds like a no brainier, however this is one of the most common attack vectors exploited by threat actors. Don’t click on links that you are unfamiliar with, don’t connect to open Wifi, always use a VPN. Secure your mobile device with a product like Malwarebytes for iOS or Android.
Now, let’s look at how to defend against the attacks mentioned above:
- The first is WAN and external connectivity security. There are all kinds of ways to protect from this attack, but one of the most common is to use a managed firewall with traffic monitoring, and integrated IDS / IPS. The firewall should also be configured by someone that knows what they are doing. It’s also a good idea to keep all of the firewall configs and rules in a central, secure system of record that is fully automated. (Don’t make your network admins do busy work by editing rules in some other system by hand just for the sake of compliance).
- Next up is phishing attacks. This can be countered in several ways. The first is educating your users. Any user within the workforce, no matter their role should be highly trained to withstand phishing and social engineering attacks. Teach them to always inspect links before clicking on them, always verifying the identity of the person on the phone before discussing sensitive information, and to report suspicious activity or emails. The other method, which should be implemented alongside, is smart email filtering. There should be some sort of appliance that sits between the MX server and the outside world that is capable of utilizing deep inspection technologies to sniff out bad links, invalid URL targets, and any other issues with incoming and outgoing email.
- Wireless network intrusions are harder to mitigate. A wireless intrusion can come from anywhere, any access point, any compromised device, and in a large number of different attack types. Use common sense when implementing wireless networking in the control environment. If possible, don’t connect anything wireless (Wifi or RF) to the PCE. Some facilities use mesh based sensor and instrument systems for their machinery and tanks. If possible, avoid this system. If wireless technology is required, ensure that the communications are encrypted and use WAP2-PSK Enterprise security and that protected wireless setup (WPS) is not just switched off, but disabled in the kernel if possible. Always monitor wireless networks and use smart algorithms to detect possible intrusions. Another countermeasure is to limit the broadcast range of the wireless equipment so that the signal stays within the perimeter of the facility. Also if possible, don’t broadcast the service set identifier (SSID) of the network.
- A copied payload attack is difficult to detect unless there security systems that sit between the CSN and adjacent connected networks that monitor inbound and outbound SMB, SCP, FTP, and SSH traffic. If this attack occurs, a part of the security chain has already failed, as the payload would have to get into the adjacent network. There are two mitigation steps to prevent or detect this attack: the first is to use the scanning method mentioned above. I used a tools which leveraged the API of several different anti-virus products to scan incoming and outgoing files that were transferred between sterile and non-sterile internal security zones. If a threat is found, the admins would be alerted and the suspected file would be intercepted and quarantined. This was also combined with an application layer firewall appliance that regulates traffic between the two zones. The other mitigation step is to make sure that the payload never makes it to the adjacent network. Secure all points of network ingress, patch your systems, and educate your people.
- Removable media can be a nightmare. On one hand, users and admins need to transfer large files (ISOs, documentation, vendor files, diagnostic data, etc), but on the other hand, CSN administrators do not want removable media to be used at all due to the massive security risk it poses. The solution is to be smart about removable media security and to use products that can mitigate security issues caused by removable media. One such product is Honeywell’s SME system, or secure media exchange.
Industrial Cyber Security Protection Against USB Threats
Secure Media Exchange (SMX) reduces the industrial cybersecurity risk and operational disruption for industrial…
Basically, it uses a portable terminal to scan removable media and secure it with a special marker indicating that it has been scanned and that no data was modified post-scan on an insecure machine. The user then inserts the secured drive into an authorized computer that has SME software installed, which authorizes the drive to be opened if it has been secured.
Interesting concept, however no system is prefect. There are still some advanced attack vectors that can be used to circumvent this system (these are advanced topics which are out of scope of this post).
- RF attacks against process control infrastructure are not very common (at least at the time of writing) in the industrial control space. They are mainly used to wireless clone security badges and run passive sniffing attacks against a facility. However, modern industrial control systems are increasingly relying on internet of things or IOT technologies and RF based mesh communication for systems such as machinery health monitoring, tank level indication, and process safety systems. The mitigation steps for RF intrusions can be complex but basically involve securing traffic between the clients, the mesh itself, and all of the endpoints in the mesh. This can be achieved via strong encryption, 2 factor authentication, and new RF security measures included in some new systems such as MAC address whitelisting. The problem is that most IOT and industrial monitoring systems do not support such security measures.
- I have used the Bluetooth attack quite a few times on jobs and security audits. Bluetooth hijacking (sometimes called Bluejacking or Bluesnarfing) is one of the simplest attacks to execute if you have the proper equipment (most of which can be easily built or purchased online). A Bluetooth device Running Windows or Linux is a breach waiting to happen. Disable your Bluetooth, people! There is no reason to have Bluetooth on an office computer, in fact don’t even order PCs with built in Bluetooth, and if you do, apply policies to disable the entire Bluetooth stack. There are many ways to remotely wake or enable a disabled or sleeping BT module. The way that the Blueooth stack works on laptops and desktops, allows an attacker to gain full access to the machine remotely, in a very short amount of time. Also, some people will say that this attack is already mitigated because of the limited range of Bluetooth. Not so. With about $80 in electronics, I can boost the range of my Blueooth module 10–20 times the normal power. So please….don’t connect these things to your control network.
- A remote cable sniffing attack is somewhat in the more theoretical realm of security, however it can happen if the conditions are right and the attacker is determined. The good thing about this type of attack is that it’s not very easy (or cheap) to preform wireless from a remote location. It requires sophisticated equipment and a trained security specialist (most of the time anyway). Mitigating this attack is fairly straightforward: don’t use UTP cabling anywhere. Also, if possible, use ESD safe, grounded network cabinets that are fully sealed. Also, keep cabinet doors and openings closed. (If you have air-gaped systems, you should already be doing all of this anyway).
IOT Security: Where is it?
Internet of things devices are becoming more and more popular in the industrial control system environment. Large vendors like Honeywell, GE and Emerson are using IOT devices in their newer industrial solutions. Before we go on, I would like to point out the fact that IOT devices that are intended for use in industrial environments are different than the IOT devices provided by a consumer oriented vendor such as Nest or Ring. The difference is usually in the software and management protocols that the devices run. For example, an industrial device will most likely not run the Zigbee control protocol but will rather use a proprietary, sometimes encrypted protocol provided by the vendor or OEM.
Let’s first look at consumer IOT products, such as the Nest Learning Thermostat. This device controls your A/C unit and allows you to manage all of your unit’s settings via the Nest app. In order to accomplish this, the unit must communicate out to the internet via Zigbee or some other protocol….wait a minute, communicate out? To the internet? No way. All of my IOT devices are behind a firewall and none of them are allowed to call home, or send traffic anywhere outside the local network. But this poses a problem: how do we get updates to those devices? How do we manage them. We adapt, and we force the vendors to adapt. Unfortunately this is a lost cause with Nest. They don’t seem to understand the concept of IOT security and use a closed protocol which I can’t do anything with. However, for other devices, I can use an on-prem IOT server which I can then control via a single app or interface. This has also has other advantages besides security. If you manage all of the data flow, you can look at the data using any software you want (Kibana, Balena IOT, etc). Another benefit to this is that you don’t need to download six million apps just to control one house or building.
Process control IOT infrastructure has similar security issues. One common thing I have seen a lot of vendors do is to put everything in the cloud. If a vendor asks you if you want “managed cloud services” or some other cloud based offering for any product that resides in or touches your PCE, DECLINE IT. Stop trying to be fancy and stick process control infrastructure into the cloud. If you need to open your PCE firewall to the outside world (anywhere outside the facility) for any reason, you're doing it wrong. There is no good reason for not having PCE infrastructure on-site. Now, does that mean you should only keep the data on-prem? Probably not, as most modern disaster recovery and BC strategies use cloud based backup, but when designing process control systems, use common sense.
You should not however, have a live data stream (read or write) going anywhere outside the local facility. Note: I know there are ways to mitigate and secure cloud traffic with AWS, encrypted private connections, etc….I know. But don’t be fancy with your PCE. Save the cloud migration for your office network.
Next, have an IOT strategy that is adopted by all of your sites. Many times I have been in facilities where to the question of where is your IOT procedures guide, the answer was….”uuummm, we don’t have one?” The document should be put together by security researchers and specialists, experts in IOT, and peer reviewed by local IT or PCE engineering teams. IT should include all policies and guidelines on rolling out new IOT systems, which vendors are approved for use, and any other procedures that should be followed when working with IOT systems.
When implementing IOT systems into your process control environment, you should always ask the following:
- Do these systems comply with the IOT guidelines for the facility?
- Can the endpoint to endpoint and endpoint to client traffic be encrypted or secured?
- Can the endpoints themselves be secured?
- Do the devices comply with corporate data handling requirements and security standards?
- Can I modify the TX and RX power of the wireless receivers?
- Can I hide the SSID or any other network identifier of the IOT mesh?
- How will the data get to the operators and people that need to see it?
- Can the devices be mounted in secure locations?
- Can I use high power HF or VLF based RF systems in my facility? (This is more of a process safety decision rather than a security one, and exists for facilities that may have a large concentration of flammable gasses in their process areas, such as refineries or chemical plants).
Listen to Your Employees
Many times I have seen management implement or not implement things simply because it was a “good idea” from the vendor or was a “part of a requirement for corporate IT”. I’m going to be blunt here: if an idea doesn't make sense, don’t implement it. No matter where is comes from. In most companies, the local admins and engineers are responsible for the facility’s process control systems. Management should always listen to, and evaluate the input of their IT and engineering staff. Keep an open mind and try to see things from the employee’s perspective. I could write an entire post on this, as this is a re-occurring problem in the enterprise and small business environments.
More Expensive Does not Mean More Secure
The cost of the solution has nothing to do with how secure the solution is. A lot of times, the mindset in the manufacturing and energy industries is that more expensive is always better and that open source has no business being anywhere in the enterprise environment. This is simply flawed thinking that results from decades of overcharging by vendors and uncontrolled pricing that exists because nobody ever pushes back on it. Large control system vendors such as Honeywell and Emerson are notorious for doing this. They take a simple technology, modify it a little, and sell it at 10 times what it cost to develop and manufacture. Security has also been affected by this. The general mindset of more expensive and professional means more secure needs to end.
Also, companies should start embracing open source software and hardware for their control systems. There is no reason why open source should not be used in the PCE environment as long as it is secured to the same high standards that everything else is. A common excuse for not using open source in the PCE is that it cannot be used alongside the systems that other vendors provide due to “qualification reasons”. While this is true in a few edge cases (where unlike systems can cause problems), installation and designed by qualified individuals should mitigate this issue. Most of the time, this is only done by vendors to squeeze more money out of customers. If the control protocols of the open source systems are alike, and a common API exists between the two, there should be little to no security or operational hurdles toward implementation and integration of open source systems.
In conclusion, be smart about security, go ahead and deploy that open source IOT platform, employ qualified people to design your security procedures, mitigate threats before they happen, and listen to / empower your workforce.
As always, thanks for reading. I value comments and questions.