The Floor is Lava

Or, why a cyber incident is an inevitability in any organisation, and what practical steps you might take to improve your resilience

10 min readNov 23, 2023

This is a joint post with Ambrose Neville, Head of Information Security at the University of Surrey. The two of us presented on these topics at the JISC Security Conference 2023.

Photo by Tetiana Grypachevska on Unsplash

[Mark] In September 2006, my wife and I took a trip on the Chain of Craters Road along the south-eastern coast of Hawaiʻi Island to see the lava fields of Kīlauea. As you approach from the west, the rugged volcanic scrubland gives way to a field of solidified lava which stretches as far as the eye can see. Back in 2006 you could still steam rising in the distance as lava streamed into the sea, boiling the ocean instantly. I’m sure that is still the case if you go there now.

If you go there, and I highly recommend it if you have the chance, you’ll get to see first-hand the sheer destructive power of a volcano and how utterly defenceless humans really are to them. At the end of the road you literally cannot drive any further as the road is buried under several meters of solidified lava.

Like many security professionals we have a close group of friends and peers in other organisations who we keep in close contact with, sometimes on an almost daily basis. While our organisations may officially be in competition with each other, we have yet to meet another security professional who won’t talk about their challenges or the issues they are facing. There’s a certain camaraderie in the profession and we all share intelligence information and discuss the issues of the day openly, although often under Chatham House rules.

“Do you ever get this uneasy feeling that we’re playing some kind of ‘the floor is lava’ game, where the lava is rising and we’re running out of apps which haven’t been compromised?”

It was during one such exchange — talking about the recent Ivanti zero-day vulnerabilities — that I (Mark) remarked to Ambrose: “Do you ever get this uneasy feeling that we’re playing some kind of ‘the floor is lava’ game, where the lava is rising and we’re running out of apps which haven’t been compromised?”

This is the reality in which we as cyber professionals so often find ourselves. We might not be drowning in lava right now, but our organisations — or their suppliers — will be breached eventually, and we should prepare for that as much as we can, with a defence-in-depth mindset.

With threat groups like Cl0p and others picking off apps which until now no-one has paid much attention to, the likelihood of your organisation in some way being affected had increased considerably. Before May 2023 we suspect that no-one in the security teams of the organisations affected by the MOVEit vulnerabilities had paid the software much mind, let alone considering that it was a fundamental part of one of the line-of-business apps that their organisation was consuming from a 3rd party (a common payroll service, in this instance).

So what can we to do prepare for the inevitable lava moment? Here are some thoughts based on our own experiences as cyber security leaders:

Your Peer Network is a Force Multiplier

One element that is often overlooked is the effect you can have by simply sharing with others (in an appropriate way of course) what security incidents you’re each dealing with. As it happens we both knew that the other used the same managed printing service provider, so by letting Ambrose know that we’d been hit with the PaperCut compromise he was able to expedite the process of investigating things taking their systems offline while they waited for the service provider to schedule patching.

[Ambrose] Fortunately the n-day hadn’t been targeted yet in our environment, although we did take the decision to take service offline until it could be patched. Meanwhile we did started to look for other potentially vulnerable PaperCut instances at other UK Universities, using Shodan and some basic Google dorking.

Using this time to conduct some vulnerability hunting proved lucrative. Ambrose was able to provide a heads-up to other university cyber people in his network, and also shared this data back to JISC who in turn shared it to other potentially affected institutions. A win all-round.

Lookalike domains are another topic where we feel there’s benefit in working in common, and sharing threat intel for defensive purposes. Each university has a certain amount of commonality with each sister organisation, and threat actors understand this. Therefore, there’s real-world value in one organisation sharing with another things like the naming scheme of past domains that have been registered maliciously against it, for phishing campaigns and so on. University of Surrey and University of Southampton have therefore defensively registered a similar set of lookalike domains, based on lessons-learned from each side. A red teamer and alumnus of University of Surrey also reached out to Ambrose recently via LinkedIn to share that he’d been performing an engagement at another UK university and had looked out of curiosity to see if the University of Surrey had registered surrey.university, and was therefore kindly sharing this as a recommendation. Southampton has pro-actively registered their equivalent too.

Embrace Healthy Competition

Due to the nature of our sector, you’re unlikely to be in any direct competition with your cyber security peers in other organisations. If you can, use your network to your advantage and make those other teams your (friendly) competition.

We routinely share examples of projects, activities, board packs, and even reports we’re creating. If one of us shares something the other doesn’t have an equivalent for, it’s seen as something to strive towards. If we can create something better then we get to share it back and the cycle of improvements begins again. Everyone wins if you can embrace the spirit of this kind of collaboration.

Some examples of things you might want to share with your network include board reporting packs/decks, strategies/strategic plans, vendor experiences, job descriptions, annual reports, key risk indicators etc.

Earlier in the summer Ambrose shared a draft outline of an annual report format he’d been working on with the intent to share it with some of his internal board members. This type of reporting — looking back over the past year and summarising achievements — may not be something you often see in higher education cyber security, but if you can spare the time it’s an excellent way to demonstrate how your security program is progressing, using a format that doesn’t confuse your audience with cryptic metrics.

Layer Your Defences

A wise ogre once said “Cyber defences are like onions, they both have layers”.
OK, so Shrek didn’t really say that, but it’s not far from the truth — if you only rely on one defensive control to protect something, sooner or later it’ll fail and you’ll wish you had more layered controls.

This approach shouldn’t be a surprise to anyone. Over 1000 years ago the Normans introduced the world to motte-and-bailey fortifications with their layered defences. The same principles of defence are equally valid today.

[Mark] Earlier in the year we were hit by (we think) Cl0p trying to compromise Southampton’s PaperCut print service. In just 5.272 seconds, a server was probed, compromised, reconfigured, and attempts made to download and install the remote management AteraAgent tool.

As you can see below, Microsoft Defender was running on the Windows server, but all it did was log these alerts — it didn’t actually block anything.

Microsoft Defender Attack Story for PaperCut Compromise — April 2023

In the UK, all universities can make use of something called the Janet Network Resolver service (JNRS). This is a curated DNS resolver, provided by Jisc, which you set as your upstream DNS for your whole network, and it blocks clients from accessing known-malicious or compromised websites. This is particularly useful for blocking phishing links in emails where other controls haven’t prevented the user from visiting the domain. If the DNS hostname is in their block list, anyone trying to go to it will instead be directed to a webpage telling you why it was blocked.

The eagle-eyed among you might recognise the 193.63.72.83 IP address in the above screenshot as belonging to JISC — it’s actually the IP of the JNRS response server — as the malicious URLs were already in their block list. This is the control which prevented this attack from succeeding.

[Ambrose] If you have the capability on your perimeter firewalls, it’s also worth looking at URL filtering. Palo Alto firewalls can be licensed with a feature called Advanced URL Filtering. This allows you to block access to any URL which matches a given category in a firewall policy, giving you much more control over when any blocking is applied. The list of categories is extensive, and few organisations will want to use them all, but the set of security-focused URL categories are very well worth investigating. In particular, we took what I call a ‘ratcheting’ approach of applying a few core categories, analysing the effect on the campus network environment, then working up through additional categories to understand the effect of the categories and ensure there were no unforeseen consequences to the business.

We’ve found this to be quite a potent defence-in-depth control in combination with protected DNS, and Network Protection at the Operating System for managed devices. The beauty of a curated, protected DNS service along with network category filtering at the campus perimeter is that it benefits everything that touches your network, from students’ own devices on halls networks to crown-jewels services in the datacentre. Via your SIEM, these security controls can also be used as detective controls as well preventative ones.

If It’s Free, Use It

Finances for any organisation can vary over time. Some years you might be given more budget than others, allowing you to invest in a new piece of tech or embark on a larger project of improvement.

Security vendors with their flashy presentations and promises of disruption, AI-this, or automated-that, are a necessary part of running any cyber function, but their promises should always be viewed through the lens of return on investment — particularly so for those of us in the HE sector. Technology deployment has a number of costs associated with it — mainly time and money — so if you can reduce either of those costs, you might find a bigger bang for your buck by looking elsewhere.

Free and Open-Source tools should be thoroughly explored before you turn too far towards paid alternatives. Ambrose is a huge fan of Shodan, the CIS hosted Controls Self Assessment Tool (CIS CSAT), and the AlienVault Open Threat Exchange, all of which are free to use or, in the case of Shodan, have a free tier for academic users.

If you do have the ability to spend a little, we both highly recommend sending your team on the Security Blue Team training. For £399, there are few better ways to train your team in incident response and basic forensics. Mark sends all of his security analysts on the Level 1 training as part of their first year, and at Surrey the training has been undertaken by a range of operational technology specialists as well as those in infosec.

Talking of incident response, if you haven’t explored the Exercise in a Box or any of the other Active Cyber Defence services from the NCSC yet, you really should. Similarly, their Cyber Security Toolkit for Boards is an invaluable resource for helping executives in your organisation understand cyber risk and what it really means for their organisation.

Reading Time

One thing that is always in short supply is time. If you can spare it, there are some truly excellent books (and podcasts — Ambrose) out there which range from the instructive and thought provoking, through to those which dive deeply into certain topics.

Here’s our Winter reading list:

Rafeeq Rehman’s Cybersecurity Arm Wrestling provides a great blueprint for building your own SOC function, taking you from business case all the way through to day-to-day activities. Rafeeq is also the author of the excellent CISO MindMap.
The Ransomware Protection Playbook, by Roger Grimes, is essential reading for anyone looking to understand ransomware and how to defend against it.
The CISO Desk Reference Guides, by Bill Bonney, Gary Hayslip, and Matt Stamper are a great set of books to have on hand. The authors have many years’ experience in running cyber security functions, and their insight is well worth exploring.
On a (slightly) lighter note, Crime Dot Com and The Lazarus Heist, both by Geoff White — host of The Lazarus Heist Podcast — are excellent dives into the world of cybercrime, exploring the shady world of North Korean hackers and their links to some high-profile crimes.

And for those of you who are more into podcasts:

‎Darknet Diaries on Apple Podcasts Jack Rhysider produces audio stories specifically intended to capture, preserve, and explain the culture around hacking and cyber security in order to educate and entertain both technical and non-technical audiences.
‎Life of a CISO with Dr. Eric Cole on Apple Podcasts Dr. Eric Cole is a distinguished cybersecurity expert and keynote speaker who helps organisations curtail the risk of cyber threats. In his podcast he gives you insights on how to become an effective CISO.
‎Afternoon Cyber Tea with Ann Johnson on Apple Podcasts Ann Johnson, Corporate Vice President, Business Development, Security, Compliance & Identity at Microsoft, talks with cybersecurity thought leaders and influential industry experts about the trends shaping the cyber landscape and what should be top-of-mind for the C-suite and other key decision makers.