Lessons Learned from Supply Chain Caretaking

Why do companies care for their supply chains?

Supply chain security can be complex to manage. Even a small business can easily have dozens of vendors they depend on. Software applications, cloud services, system integrators and software-as-a-service providers tend to pile up as your business grows and evolves. No matter how hard you work to keep your own cyber security best practices, policies, and processes in great shape, vulnerabilities or bad security practices at a single vendor can jeopardize your whole business.

Your supply chain might be used to gain an entrance into your organization. A scam attempt may impersonate your vendor. A phishing attacker might ask for your credentials or passwords in the guise of a vendor’s IT person. Ransomware or other malware operators often use an unprotected vendor as a stepping stone into multiple customer organizations at the same time.

Taking regular care of your supply chain is paramount. You gain an understanding of the dependencies and risks associated in working with different vendors. You form an image of the maturity level and security posture of your vendors. You see how they deal with vulnerabilities reported to them and how they communicate during the vulnerability coordination process.

How can you take care of your supply chain?

In a Supply Chain Caretaking exercise, you first map out all of your important vendors with our assistance. You identify what key services each vendor provides, and how and why those services are used within your company. Vendors can be e.g. software manufacturers, system integrators, ISPs, SaaS providers, or cloud service providers.

Some of the identified services can also be your own internal servers, services or software applications developed in-house — in the context of Supply Chain Caretaking we can view those as “vendors” when appropriate.

Based on the information gathered together with you, we investigate each vendor in detail. The investigation is conducted using a set of open-source intelligence gathering methods and analysis made by our team of cyber security experts.

As security issues are identified during the analysis, we gather the findings into a set of vulnerability reports for the vendors. We report the issues to the affected vendors in collaboration with our customers, handling the heavy lifting of vulnerability coordination work on your behalf.

To understand your vendors, we consider basic vulnerability reporting practices. Does the communication work, does the vendor communicate about the potential issue to avoid misunderstandings, and of course, will the issue get fixed. In this process, we help the vendor any way we can — after all, the most important thing is to get the issues fixed.

What have our customers learned about their vendors?

We took a birds-eye view of 77 reports we’ve sent to 60 vendors. What common trends have emerged across all of the customers and vendors we’ve worked with?

What have we observed about the quality and speed of vendor responses and how easy it has been to communicate with them? One third of the vendors have A-class response, meaning that they react to reporting fast, and fix the issues as they should. The last two thirds react slowly, even if they want to fix the problem — or in the worst case, don’t respond at all.

• Class A: Relentless hunter — Sometimes the starting point is obscure. “This should not be possible.” Relentless hunters are not daunted by the difficulties, they hunt down the issue until the root cause is found and it can be fixed. Respect.
• Class A: Pro — They acknowledge the report immediately, say what they are going to do, and do what they said. They talk about the issue with the reporter to avoid miscommunication. Sometimes they learn new things in the process, and allow the reporter to learn too. Such beauty.
• Class A: Do it in scale — “I’ve read your report, and checked all similar places. I found 8 different places with the same issue. All of them are fixed now, thanks.”
• Class B: Slowish vendor — “We’ve investigated the issue and are waiting for our own vendor to fix this.”
• Class B: Focus on the present day — “This service was built in 2008, it can not be patched anymore, we are investigating how to migrate it to a new service.”
• Class C: There is no problem — In these cases, we get a nonsensical explanation why the problem does not exist. For example, when we find an open database, the vendor explains that it is not a problem, as the service is not used anymore.
• Class C: Our IT infra is totally broken — When your vulnerability report bounces because the vendor’s domain is not available.
• Class C: Silent treatment — This is the most problematic category. Does the vendor not respond because they don’t want to, don’t know what to say, or because the message hasn’t reached the right people? How can we activate them in a positive way?

What kinds of vulnerabilities have been found?

17% of the 343 vendors we’ve dealt with have had issues that should be fixed. Typical issues have been old unsupported operating systems running on abandoned servers, exposed databases, known vulnerabilities in Internet-facing services, subdomain takeover vulnerabilities, or vulnerable applications.

A common category of vulnerabilities we have encountered are old server installations that no longer receive security updates. A vendor has installed e.g. a server with a particular Linux distribution, and hasn’t noticed that the distribution has reached end-of-life (EOL) status. The server is no longer receiving security updates. Gradually more and more known vulnerabilities pile up, making it easy for an attacker to exploit those known vulnerabilities and to take over the server and any important business or customer data it contains. Sometimes the vendor knows that the OS version has reached end-of-life, but they do not have a clear plan or process on how to systematically upgrade their servers to a new release.

Another very common category of vendor issues comes from known vulnerabilities in software. Keeping track of new security vulnerabilities against all of the layers of a complex system can be quite a daunting task for a software integrator. Keeping all of the customer servers updated to the latest releases of important software is even more time-consuming. Reacting fast to emerging threats and updating all of the potentially affected servers whenever a new vulnerability is announced takes resources, skills and highly optimized system management processes.

Services left open to the Internet either by accident or through lazy service architecture design is also a very common finding among the vendors we’ve reviewed. No matter how well your servers are updated and new security vulnerabilities managed, if your customer database is open to the Internet without authentication, that can be an extremely damaging oversight with liabilities ranging from brand impact to personal data breach ramifications. Often we find open services such as databases, remote desktops or file sharing ports that have been left open due to some administrative process oversight: a maintenance routine or a software update requires remote access, or system configurations are accidentally overwritten and replaced with default access configurations when an update is installed.

Subdomain takeover vulnerabilities are also very common among our findings. A vendor has set up a server in the cloud, and the customer DNS has a CNAME record pointing to that server. The server may have been set up and used for a specific purpose, but that need has already ended. The vendor decommissions the server, but the DNS record is left pointing to a cloud resource. An attacker may set up their own server at the same address, and use that to deceive users that still try to access the decommissioned service.

Case: Kotkan Energia — Dealing with vendors provided insights

“Executing the supply chain caretaking playbook with Badrap proved highly useful for us”, confirms an ICT Manager from Kotkan Energia. “We were able to map out our suppliers and to understand our digital dependencies even better than before. We received valuable insights on how our vendors deal with security issues. Better yet, our suppliers now know we are assessing their security response capabilities as a standard practice. This is what others should do, too.”

Choose the right angle to cyber

Badrap’s goal is to make cyber security as easy as possible for everyone. That’s why we created Playbooks — they make completing your cyber security tasks guided, systematic, and easy. You’ll just have to pick a task and follow the guided steps!

Playbooks cover different cyber security related themes, and the execution varies: playbooks cover training, monitoring and policies. In this post, you hopefully got a comprehensive view on our Supply Chain Caretaking Playbook: how it works and what results and knowledge we have gained on the way.

What if Supply Chain Caretaking is not relevant to your organization right now? We have plenty of other playbooks to start with: check out all of our playbooks at badrap.io!