IT and OT (Operational Technology) networks. Better together?
Co-Author : Alistair De Marco
Operating as a network engineer in manufacturing often has its own unique set of challenges. These challenges stem from devices sitting on a network controlling physical machinery, often procured decades ago when the networking and technological landscape was very different. Now, our manufacturing clients are recognising this growing disparity and the need to adapt to the new networking landscape.
Over the years, we have become quite accustomed to the idea of Operational Technology (OT) — networks controlling physical systems, such as factory machinery, surgical robots, or process control devices) as something completely distinct from the traditional Informational Technology (IT) (e.g. anything that provides connectivity to and between the office systems in the enterprise).
Historically, a separate stack of protocols would be developed, making them hardly compatible with IT networks. However, more and more company operations started to become more dynamic and data driven, and for organisations prioritising efficiency, integration of the IT & OT networks became the more logical choice.
This article will highlight the real-world challenges with OT connectivity faced by our manufacturing clients and offer some advice for network engineers on how to overcome them, providing practical tips for better security if you want to use a converged IT&OT network.
A whole different world
Even if you are not working directly in OT communications, you might have heard of Industrial Ethernet (IE) protocols, such as EtherCat¹ (for real-time computing requirement); EtherNet/IP² (adapts Common Industrial Protocols (CIP) to Ethernet standard); Profinet³(real-time data delivery) of Profibus⁴. If you have not, I would suggest that you fall down the rabbit hole of researching it! More generally, you may have heard of the architectural frameworks, such as the Perdue model⁵, setting out guiding principles for the design of OT networks.
AirGap
One approach historically taken is to simply separate environments operating on these protocols from the rest of the IT. Airgap. Never connect. This is because machinery running on OT Networks is often mission critical; turbine facilities, water treatments plants, surgical lasers… the list can continue forever. Unexpected downtime will have a direct impact on production and ultimately revenue, and network isolation can be seen as a mitigation. While this separation leaves the OT environment unexposed to threats from the IT (and Internet connections, as normally OT wouldn’t be connected to the public networks as well!), it brings a level of inconvenience for those who support the OT network. Updates and patches to software become extremely problematic to perform, as well as for those who operate in it. Have you ever tried to transfer a file from the external machine? USB sticks, COM ports and serial cables are your best friends.
What happens when it’s extremely difficult to update software? That’s right — it stays obsolete, justified by the environment separation in the enterprise risk register. What happens when software reaches end of life? Correct — there are no patches for vulnerabilities. Equally it is somewhat anecdotical, at some point hackers stop writing viruses for this software. This may work to your benefit, but even then, this is a situation that is far from ideal and should be avoided. To overcome this nuisance, architecture has adapted to include “anti-contamination stations” for USB sticks, file transfers etc. as well as update/patch machines to be used as a buffer zone for software staging and distribution. Operating those networks is complex, requires specialist knowledge and as a result is quite expensive. The engineering force skilled in supporting those environments was rare, and as time went on many of them naturally aged and retired without replacement, taking their knowledge of the industrial equipment with them. With these and other drivers, most of the companies started to search for a compromise, which we discuss in the next section…
Starting to converge
Another approach involves partial convergence between OT and IT environments. You may have seen use cases where a WAN network is being shared by both IT and OT environments, with the OT network plugged in via a firewall, so beyond the switch port, all you see from a networking perspective is an encrypted tunnel. Quite often this is done in companies with a separate team to support OT equipment and the wider network. Thus, enterprise IT would never see insights of the traffic or the OT application setup. However, it creates a “Russian doll” / “Matryoshka” situation, it works until it doesn’t. In most cases, this is a single switchport that creates a single point of failure between the two environments, introducing latency due to encryption overhead. Additionally, ensuring sufficient bandwidth all the way inside LAN and on WAN transport! The moment an issue arises, IT teams need to troubleshoot, whilst having no visibility and a handoff point between IT and OT teams, and thus it becomes an obstacle to the speed of resolution.
A driving force for manufacturers to reconsider their approach to OT environments was the introduction of Internet of Things (IoT), or Internet of Fridges as some people call it… Where a multitude of sensors and small devices, generate data streams that need network access. This spurred manufacturers to perform some convergence between IT and OT, with companies deploying IoT devices to gather operational data and improve efficiency on their production line. Like legacy OT systems, IoT devices interact with the physical world with various sensors to collect and transmit data, making IoT devices a natural fit for OT transformation projects. However, to be utilised effectively, the data IoT devices produced required the connectivity and functionality traditionally found in IT networks; data being transmitted to cloud analytics services (or on-prem systems) and implementation of network segmentation to address ever-present security vulnerabilities. There was an opportunity to re-use IT capabilities to manage the security of OT, utilising Next Generation firewalls with IPS/IDS and similar operating systems to those used on elements of the IT and OT controllers, allowing for additional insight and control of the OT traffic. The functionality required from both environments needed a level of convergence, and partial convergence seems like a good place to start.
Shaken. Not Stirred.
As the number of applications and business requirements have grown, networks in turn have become more complex. Office spaces need to seamlessly accommodate various populations of users, research labs, guests, facilities, and office employees; all while being managed by the same-sized network teams. In response to this, the IT world has adopted new trends to automate security policies, such as Zero Trust⁶ architectures. So, if managing the policies and keeping the IT environment which already has investment to advance capabilities — why not share these very capabilities allowing for better security with the OT?
It seems a pretty straightforward decision, though there are a few things you need to be mindful of: risk appetite versus efficiency is a key factor for companies making these decisions. When designing the network, you should always look out for resilience and redundancy to ensure the network support team is not disturbed too often.
However, let’s look at the additional factors that needs to be considered when IT and OT networks share the same infrastructure:
1. Segmentation and Isolation; controlling the impact, managing the access between networks and limiting the size of the segments critical to security. The technologies to support this can be as old and well known as static VLANs (on packet switching domains) and VRFs (on routing domains) or more advanced downloadable dynamic ACLs.
2. Management and Visibility; in most cases, these two factors come last on the list of the architectural factors, but we believe they deserve a top of the list position when it comes to sharing the infrastructure. At any given point of time, the network operator should be able to read the state and health parameters of the network and be alerted of any deviations. You want to know if your taps are leaking rather than walk in on a flood in your bathroom. This is metaphorical, but in case of utility networks, it can be a very bitter and very real.
3. Throughput and scalability; joining the traffic from office documents and calls with production data requires setting your priorities right and not letting your production struggle when the PR department is trying to download a huge video from a contractor’s website.
4. Regulation and Compliance; depending on the industry, apart from the standard ISO27001⁷, NIST⁸ requirements, there might be additional regulatory requirements or certifications to consider. Ensure your architecture complies with those standards and guidelines.
Left, right or centre? Which architecture should I follow?
It is clear OT environments are a unique and business-critical corner of the IT world and have specialised requirements. That being said, when there is a risk to life involved or a stoppage of production that could cause a significant loss to a business or its reputation — investing in bulletproof architecture and appropriate product choices is essential. The actual decision tree for a large organisation is more complex than it seems and, in most cases, has a set of constrains, such as the “heritage” solutions. For example, having a legacy system using ModBus⁹ as a protocol requires an appropriate segmentation and ring-fencing of the solution along with the tight access control, since ModBus messages are not encrypted and highly susceptible to the man-in-the-middle replay attacks. It is a continuous task of the network architect to make sure those legacy solutions are appropriately set up within the wider architecture so that the attack surface is minimised, and, if there is no wider appetite of the business to invest into the bullet proof security, the only way is a compromise. This compromise can be in a shape of sharing the WAN, or underlay network — with additional belts and braces, such as centralised policy provisioning and monitoring.
“But which architecture should I follow?” you ask, clinging to the Perdue model for OT networks. Well, with the current level of maturity for security and visibility tooling in the market, the answer for most organisations will be a hybrid model, with a manageable risk footprint and associated cost savings compared to isolated networks. However, as you already know, the right solution is…. the one that fits your organisation best.
References
[1]EtherCat:https://www.ethercat.org/en/downloads/downloads_A02E436C7A97479F9261FDFA8A6D71E5.htm
[2] EtherNet/IP: https://www.odva.org/technology-standards/key-technologies/ethernet-ip/#:~:text=EtherNet%2FIP%E2%84%A2%20is%20a,data%20anytime%2C%20anywhere.
[3] Profinet: Profinet Specification https://www.profibus.com/download/profinet-specification
[4] Profibus: PROFIBUS Standard — DP Specification https://www.profibus.com/download/profibus-standard-dp-specification
[5] Purdue model: Theodore J. Williams (1993) “The Purdue enterprise reference architecture.” Proceedings of the JSPE/IFIP TC5/WG5. 3 Workshop on the Design of Information Infrastructure Systems for Manufacturing. North-Holland Publishing Co.
[6] ZeroTrust: “Zero trust: Never trust, always verify. Security in the age of the porous perimeter” https://www2.deloitte.com/us/en/insights/focus/tech-trends/2021/zero-trust-security-framework.html
[7] ISO 27001: ISO/IEC 27001:2022 Information security, cybersecurity and privacy protection Information security management systems Requirements https://iso.org/standard/27001
[8] NIST: Cybersecurity Framework https://www.nist.gov/cyberframework
[9] ModBus: Modbus Protocol Description https://www.modbustools.com/modbus.html
Note: This article speaks only to my personal views/experiences, is not published on behalf of Deloitte LLP and associated firms and does not constitute professional or legal advice. All product names, logos, and brands are the property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.
