Practical Security Assessment of SD-WAN Implementations

Published in

hackingodyssey

15 min readOct 29, 2019

Software-defined networking in a wide area network (SD-WAN) quickly becomes very popular in Enterprises. Vendors promise “on-the-fly agility, simplicity, security and automation” and many other benefits. What do you know about SD-WAN? What “security” means from a hands-on perspective? Are present SD-WAN products really secure? The goal of SD-WAN New Hop project is to give answers to these questions by exploring the implementation security of SD-WAN solutions from a practical point of view.

Overview

This report provides an overview of the SD-WAN New Hop Project — a community-driven research project devoted to security analysis of SD-WAN products and solutions. The research team held several Internet-wide scans to survey the distribution of SD-WAN products and to detect known software vulnerabilities in its components. Manual and tool-aided vulnerability assessment of multiple SD-WAN products were performed. This paper describes an overview SD-WAN Threat Landscape, methodology and implementation of the Internet-wide scans, provides security checklists and vulnerabilities found during the project.

SD-WAN New Hop Team

Sergey Gordeychik
Denis Kolegov
Alex Timorin
Maxim Gorbunov
Oleg Broslavsky
Anton Nikolaev
Nikita Oleksov
Nikolay Tkachenko
Christoph Jaggi

Introduction

Nowadays SDx technologies are very popular. SD-WAN is used for Cloud platforms and Enterprise networks for branch connectivity. Vendors are developing different subcomponents such as SD-LAN, SD-CORE, SD-VPN, SD-Access and SD-DC products. At the time of writing, Metro Ethernet Forum (MEF) has unveiled its “SD-WAN Service Attributes and Services Standard”.
The SD-WAN New Hop Project targets the security of SD-WAN. It was started in December 2017, as a side project of SD-WAN solution security design and Security SDL planing. During the course of the project multiple critical vulnerabilities, such as RCE, XXE, SQLi, unpatched software, outdated packages, vulnerable multi-tenant access control mechanisms were found. It was decided to investigate the security of well-known and famous SD-WAN products. The assessment covered the following products.

Contributions

The main contributions of the project are as follows:

The SD-WAN cybersecurity threat landscape was created and used for practical security assessment.
Multiple design flaws, weaknesses and vulnerabilities in the SD-WAN products were found, investigated and disclosed to vendors.
The Internet-wide signatures and fingerprints knowledge base of SD-WAN products was created.
Basic security requirements checklist for SD-WAN systems was developed.
The SD-WAN security assessment tools for the Internet and private networks were developed and tested.

Online Resources

The full research, including actual versions of the papers, scan and assessment results, advisories, tools, is available on the following link https://github.com/sdnewhop/sdwannewhope.

Responsible Disclosure

All discovered vulnerabilities were reported to the corresponding vendors according to “Responsible Disclosure” best practices. At this time, the notified vendors have been processing and fixing the vulnerabilities. If a vulnerability is found in an older version of a product but was absent in the current version of the product, we do not notify the vendor.

Terms and Definitions

For the purposes of the article, we use the main terms and definitions of Verizon SDN-NFV and ETSI GS NFV-SWA.

SD-WAN Threats

This section briefly considers the main and the most widespread security threats related to the SD-WAN functional blocks and components according to the developed SD-WAN threat landscape.

Outdated Software

Most of SD-WAN systems use GNU/Linux-based operating systems such as Ubuntu, Debian, CentOS, etc. Very often other open source tools are used in different network functions. For example, IPsec overlay can be based on “strongSwan”, while control plane runs API on top of Apache Tomcat, Karaf, WildFly, Node.js.

The following typical issues were found in many SD-WAN systems:

Using outdated software
Insecure software update mechanisms
Using insecure cryptographic primitives (e.g., MD5, SHA1)

Insecure Web Interfaces

All analysed SD-WAN products use Web components to implement user-machine or machine-to-machine interfaces. At the same time, it was found that the existence of “low-hanging fruit” vulnerabilities in those Web interfaces is widespread and systematic.

We identified multiple vulnerabilities to the following well-known attacks:

HTTP Slow DoS-attacks
Password brute-forcing
XSS (Reflected, stored)
Command injection and RCE
XXE and SSRF
CSRF
IDOR

For instance, the security bulletin describes multiple vulnerabilities found on the Web UI of the Citrix SD-WAN appliances. The vulnerabilities could allow an unauthenticated attacker having network access to the management interface to compromise a corresponding SD-WAN network.

Non-isolated Multi-tenancy

An SD-WAN system is called multi-tenant if it has an ability to provide logical isolation of shared resources within the NFV concept. In general, this suggests the existence of a common Web UI used by different tenants to access isolated SD-WAN instances. It turned out that those Web UI also have security vulnerabilities specific to multi-tenant applications such as missing or improper access control.

In this case, an adversary having legitimate access to the Web interface can implement different attacks against other tenants (clients or providers). For example, standard or weak passwords for different tenants or weak password recovery mechanism can lead to horizontal privilege escalation. In this regard, the adversary can analyse the implementation and its possible weaknesses using an own isolated session and then exploit vulnerabilities against other clients.

During the research, the following main threats regarding multi-tenancy were identified:

Unauthorized access to provider data
Unauthorized access to tenant data
Unauthorized access to tenant VNF
Unauthorized access to the tenant stored flow data (e.g., NetFlow, IPFIX)
Denial of Service

Broken Cryptography

Cryptographic mechanisms are employed on all SD-WAN planes: data, control, management and orchestration (application). Cryptographic mechanisms are the essence of SDN/SD-WAN.

At the present vendors have to invent cryptography protocols to implement SD-WAN specific flows because there are no production-ready safe cryptography protocols developed within SD-WAN conception and taken into account all peculiarities. Most vendors approach to this problem by adopting IPsec for SD-WAN.
We also found that most SD-WAN products have the following flaws related to cryptographic provisioning:

Use of hardcoded public-key cryptography key pairs and corresponding certificates that are the same for all customers and can not be replaced
Use of self-signed certificates for generated public-key cryptography key pairs issued by the SD-WAN product
Manual installation of self-signed certificates on SD-WAN nodes without certification revocation features

Insecure Zero-touch Provisioning

Zero Touch Provisioning (ZTP) is a mechanism that allows nodes to be provisioned and configured automatically. All the main SD-WAN products support ZTP. At the same time, by design, ZTP server must accept requests from unidentified and unauthenticated devices on the Internet. This fact increases the attack surface and attacker capabilities. The following threats can be encountered regarding ZTP:

ZTP server or/and client spoofing
Unauthorized access to ZTP service and data in the cloud
Exhaustive Denial of Service on ZTP service
Privilege escalation on ZTP server
Eavesdropping
Insufficient access control on multi-tenant ZTP service

Internet-scale SD-WAN Scanning

This section briefly describes Internet-wide scanning of SD-WAN devices according to the SD-WAN Internet Census approach.

SD-WAN Enumeration

The employed methodology can be defined as follows:

Prepare fingerprints of the network interfaces of an SD-WAN product.
Define and express the signatures within a search engine query language.
Discover and enumerate devices using search engines (Shodan, Censys, etc.).
Use incremental save method to store the results into the database.
If a new SD-WAN product or interface is discovered go to step 1.
Run the steps 1–5 regularly.

The Grinder Framework

The Grinder framework is a security research intelligence framework that has been developing since 2018. It was created to automatically enumerate, fingerprint and scan hosts on the Internet using different specialised back-end systems: search engines (e.g., Shodan or Censys) for discovering, scanners (e.g., NMAP) for active fingerprinting and scanning, vulnerability databases (e.g., Vulners) for getting information related to vulnerabilities in the discovered software. The Grinder framework can be used in many different areas of security research, as a connected Python module to your project or as an independent ready to use from the box tool.

The main features are as follows:›

Collecting hosts and additional information using Shodan and Censys search engines.
Scanning ports and services with boosted multiprocessor Nmap Scanner wrapper.
Scanning vulnerabilities and additional information about them with Vulners[iv] database and Shodan CVEs database
Retrieving information about SSL certificates.
Scanning for SSL/TLS configuration and supported cipher suites.
Scanning for SSL/TLS bugs, vulnerabilities and attacks.
Custom scanning scripts support (in LUA or Python3).
Collecting Threat Intelligence Information, such as security bulletins, public exploits etc. for detected vulnerabilities and software.
Confidence filtering.
Building an interactive map with information about the hosts found.
Creating plots and tables based on the collected results.

Practical evaluation of this approach during massive Internet census projects has shown the benefits of the combination of active and passive host identification methods.

Having said this, the main purpose of Grinder is to unify different sophisticated security tools, aggregate gathered information and to help to understand collected information and exposed statistics related to the systems in the research scope. Grinder incrementally saves all scans, results and statistics to its database to compare results over time and track the statistics changes.

To visualize gathered data, Grinder provides an interactive world map containing all results. Grinder map backend is written in Flask and supports additional REST API methods to get more information about all scanned hosts or some particular host from the map. Also, it is possible to show some additional information about host interactively from the map. For example, currently open host will be automatically checked for availability with “ping” from the backend, also for every host many additional features are available: current host can be directly opened in Shodan, Censys and ZoomEye search engine web interfaces, host can be shown on Google Maps with all available information about geolocation, also it is possible to make an IP lookup, or open raw information in JSON directly in browser or from your application with provided API methods.

SD-WAN TLS Scanning

In this section, we present a scan for known vulnerabilities in the TLS implementations of SD-WAN products deployed on the Internet. We used the TLS-Attacker framework as a TLS scanning engine and the Grinder framework, developed by us, as an orchestrator.

The idea to conduct this research was born after reading the “Scalable Scanning and Automatic Classification of TLS Padding Oracle Vulnerabilities” paper. The last one evaluated the Alexa Top Million Websites for CBC padding oracle attack and revealed vulnerabilities in 1.83% of them. But the paper did not mention vulnerabilities to CBC Padding Oracle Attack in SD-WAN products. It can be explained by many reasons. For example, SD-WAN TLS interfaces are may not be vulnerable to this attack or the SD-WAN nodes are not in Alexa Top Million Websites. It should be noted that the paper considers and encounters only vulnerabilities to CBC padding attack. We thought that it is also interesting to scan SD-WAN nodes related to all main TLS vulnerabilities.

Within our research, we considered only SD-WAN nodes (controllers, Web UI, edge routers, etc.) we had already collected and stored via enumeration and fingerprinting.

We used TLS-Attacker framework as a TLS core scanning engine and the Grinder framework as an orchestrator. In this case, the TLS-Attacker was used as a back-end within the Grinder framework. To get information about TLS configuration, bugs, and possible attacks, Grinder provides a wrapper module for TLS-Scanner and TLS-Attacker tools, that is used to handle all scanning options and analyze the results. TLS-Attacker is a Java-based framework for analyzing TLS libraries. It is able to send arbitrary protocol messages in an arbitrary order to the TLS peer, and define their modifications using a provided interface. Grinder uses related TLS-Scanner module to scan TLS configuration with TLS-Attacker.

Gathered data by TLS-Scanner and TLS-Attacker helps Grinder to count unique attacks and bugs, also it is possible to count different unique types of entities for every vendor, product, protocol, and many other types and categories. To get more accurate results, Grinder firstly checks availability of every host that was found in Shodan or Censys, after that Grinder tries to detect proper port and other options to successfully start scan with TLS-Scanner, and finally Grinder saves every result from every host in separate files that later can be parsed more accurately with additional parser and analyzer.

The employed methodology is defined as follows:

Run TLS scanning engines (e.g., TLS-Attacker, SSL Labs Server Scan, etc.) on the appropriate hosts and interfaces from the SD-WAN Knowledge Database.
If vulnerabilities are found, rescan the node two times to minimize false positives.
If the vulnerabilities are still present, check them again using PoC scripts in Python.
Save the confirmed results to the database.

We scanned about 7500 SD-WAN nodes and found the following:

1873 nodes were vulnerable to Sweet 32 attack
121 nodes were vulnerable to CBC Padding Oracle attack
30 nodes were vulnerable to CRIME attack
29 nodes were vulnerable to Logjam attack
28 nodes were vulnerable to CVE-2016–2107
14 nodes were vulnerable to DROWN attack
6 nodes were vulnerable to ROBOT attack
1 node was vulnerable to Heartbleed

Vulnerabilities to the ROBOT attack were found in Viprinet product only. We found one Riverbed SteelHead node vulnerable to Heartbleed attack. The software was released in 2013.

Riverbed SteelHead nodes released prior to 2014 and Viprinet VPN Virtual Hub nodes were vulnerable to CBC Padding Oracle attack.

Scanning Tools

Within the project, we have developed and used the following tools.

SD-WAN Infiltrator is an NMAP NSE script to automatically discover SD-WAN nodes in a local network. It is based on SD-WAN Knowledge Database.

SD-WAN Harvester was initially created to automatically enumerate and fingerprint SD-WAN nodes on the Internet. It uses the Shodan search engine for discovering, NMAP NSE scripts for fingerprinting and masscan to implement some vendor-specific checks. This project is no longer maintained. It is stable and you still can use it for SD-WAN scanning, but currently, more preferable and accurate way to scan different things on the Internet (not only SD-WAN solutions) is to use the Grinder Framework.

Vulnerabilities

The following security vulnerabilities and issues were identified:

Versa Networks:

Arbitrary command execution on FlexVNF appliances connected to Versa Director by unauthenticated remote user
Vulnerability to XPath and XXE attack allows an attacker to read arbitrary files or enable SSO authentication and create SSO user to obtain root privileges
Vulnerability to OS command injections allows any authenticated user to execute arbitrary system commands with root privileges
Multiple memory management flaws can lead to DoS
Weak permissions and hardcoded passwords allow local and remote attackers to get access to sensitive data.
It is possible to directly access the database via crafted HTTP request in different scripts
Some Suricata IDPS Emerging Threat rules use regular expressions that are vulnerable to ReDoS-attack
An attacker can craft a malicious Apache Solr configuration file to execute arbitrary OS command
Apache Solr is vulnerable to XXE attack
Cryptographically unprotected control plane protocols
Versa Analytics Driver REST API Hardcoded credentials
Weak CSRF protection can be bypassed via Flash

Silver Peak Unity EdgeConnect:

Lack of protection against brute-forcing password
Web UI leaks software versions
Lack of protection against CSRF for REST API
Denial of service of Web UI via Slow HTTP attacks
REST API leaks software versions
Use of default SNMP community strings
Access to operating system interface via administrative CLI backdoor
Multiple vulnerabilities to reflected XSS
Arbitrary file reading via path traversal

Riverbed SteelConnect:

Stored XSS via user name field
Denial of service of a gateway via slow HTTP attacks
Password reset link spoofing via HTTP Host header
TLS server is vulnerable to CBC Padding Oracle Attack

Cisco (Viptela) SD-WAN:

OpenSSH leaks system version via a warning message
Incorrect protection against CSRF for REST API and Web UI

Viprinet Virtual VPN Hub:

Stored XSS in CLI via item names
TLS server is vulnerable to ROBOT attack
TLS server is vulnerable to CBC Padding Oracle Attack

Citrix NetScaler SD-WAN:

Denial of Service on Web UI via Slow HTTP attacks
Multiple stored and reflected XSS
Lack of protection against CSRF for REST API and Web UI
Absence of function level access control mechanism
Multiple command injections
Multiple SQL injections
Arbitrary file reading via path traversal
Unauthorized access to Munin web UI
MitM using a hard-coded cryptography identity

Fortinet SD-WAN:

No Rate Limiting on Web UI Authentication
Denial of Service via Slow HTTP DoS Attacks
Disclosure of sensitive configuration information for unauthenticated users
Highly insecure TLS configuration for Web UI
Cross-Site WebSocket Hijacking
Insecure password storage
Sandboxed shell escaping on FortiGate
Sandboxed shell escaping on FortiManager

Brain4Net SD-WAN:

Docker Hub credentials leakage
Cryptographically unprotected control plane protocols
Multiple stored and reflected XSS
Cross-site Websocket hijacking

All these issues were reported to corresponding vendors. Some issues were disclosed via “Full Disclosure Mailing List”.

Vulnerability Management

According to OpenSAMM, Operation Enablement is one of the crucial elements of Secure Software Development Lifecycle (SSDL). In general, a vendor implements that process by creating a Product CERT team which coordinates the vulnerability remediation process together with the software development team using best practices and international security standards like ISO/IEC 29147:2018 — Vulnerability Disclosure in Information Technology.

We believe interactions with external security researchers is an indirect sign of efficient internal processes related to SSDL. We identify the following indicators showing security maturity from a security researcher perspective:

Security contact: The official web site of the product contains web links or emails related to a product security team.
Communications: vendor complies the main business rules and norms within communications (polite interaction, acknowledgement, email encryption, etc.).
Bug confirmations and patches: vendor notifies security researchers when the state of vulnerabilities are changed (approved, fixed, tested, done, etc.).
CVE, Credits: Vendor requests and assigns CVE numbers to the reported vulnerabilities, mentions security researchers found that vulnerabilities.
Researcher friendly: Vendor made a good general impression on the security researchers within the interactions on the vulnerability management process.

It is important to note that vendors having bad reputation and negative feedback from the security community perspective will not receive responsibly disclosed vulnerabilities. This increases the risk of appearance of publicly available zero-day exploits and the overall risk of compromising.

The summarized impressions of our interactions with SD-WAN vendors within the security research are as follows.

SD-WAN Security Requirements

This section provides SD-WAN security requirements. The requirements are defined within OWASP ASVS approach and use the same semantics. These requirements can be used as checklists or security tests. It should be noted that the requirements can’t be considered as completed and based on our experience and reflect the found security issues.

Common Requirements

Verify that the SD-WAN software components use updated third-party components and libraries.
Verify that the operating system and the kernel are hardened.
Verify that known host-based vulnerability scanners (e.g., Vulners, LibScanner, etc.), web vulnerability scanners (e.g., Burp, Acunetix, etc.), specialized scanners (e.g., ssh_scan) do not detect vulnerabilities on the corresponding components and interfaces.
Verify if a vendor-controlled cloud management interface is used within the architecture.

Zero Touch Provisioning Requirements

Zero Touch Provisioning (ZTP) is a mechanism that allows nodes to be provisioned and configured automatically. ZTP service mechanism is a root of trust service. The requirements are as follows:

Verify that an edge router gets its initial configuration using a secure protocol.
Verify that an edge router discovers a controller, orchestrator and other entities using a secure protocol.
Verify that all SD-WAN entities ( edge routers, controllers, orchestrators) authenticate each other using secure cryptographic protocols.
Verify that the trust bootstrapping mechanism uses adequate procedures.
Verify that identities are stored in secure storages (TPM, HSM) or short-lived and can be revoked easily.

Secure Communications and Cryptography Requirements

Cryptographic mechanisms are the essence of modern SD-WAN technologies. Our research revealed that SD-WAN products introduce self-invented distributed protocols for control planes. Beside that, we identified multiple critical vulnerabilities related to secure provisioning. The basic cryptographic requirements as follows:

Verify that self-invented cryptographic protocols and implementations are not used in the control and data planes.
Verify that AEAD primitives are supported.
Verify that the key management protocols are cryptographically secure.
Verify that the security levels provided by cryptographic primitives are consistent.
Verify that legacy modes or primitives like DHE_EXPORT, DES, TripleDES, RC4, MD5, SHA1 or custom primitives are not employed.
Verify that it is possible to manage cryptographic mechanisms (e.g., enable TLS 1.3 only, disable unwanted ciphers, etc.).
Verify that the scanning results of all TLS-, SSH- or IPsec-enabled interfaces do not contain vulnerabilities to known attacks.
Verify that running WireShark on each node (orchestrator, controller, edge router) does not reveal unencrypted sensitive flows.
Verify that employed protocols and primitives provide the security level required by the product security policy and threat model.
Verify that long-term secrets (e.g., long-term private keys, pre-shared keys, session tokens) cannot be extracted from web interfaces or accessed by unprivileged users.
Verify that long-term secrets are stored within HSM-model.
Verify that the same hardcoded certificates are not used on different deployments or instances.
Verify that a key renewal mechanism is supported on control channels.

Web Management Interface Requirements

All examined SD-WAN products employed a web user interface (Web UI) as a main unified interface to govern all parts of the SD-WAN system. At the same time, it was found that all products have low-hanging critical vulnerabilities to the following attacks:

CSRF
HTTP Slow DoS-attacks
Password brute-forcing
XSS (Reflected, stored)
Command injection and RCE

Because of the simplicity of these attacks the following

Verify that the web interfaces are protected against CSRF attacks.
Verify that the web interfaces are accessible over HTTPS only.
Verify that the web interfaces employ modern hardening mechanisms: secure headers, CSP, CORS headers, etc.
Verify that the web interfaces are not vulnerable to Slow HTTP DoS attacks.

Security Management Requirements

Verify that the contacts of the product security team (CERT) can be found easily.
Verify that you can communicate with the CERT team.
Verify that the list of known vulnerabilities of the product is published regularly. Recommendations and all necessary information are provided.
Verify that 3rd-party software vulnerabilities are tracked and patched.
Verify that update mechanisms are satisfied with modern security requirements.