Urgent response to emerging IoT-vulnerabilities with open source tools
Emerging vulnerabilities are fresh discoveries that might require an emergency response from the end users. Need for a rapid response is painfully true for the Internet of Things (IoT) devices, some of which may already abandoned by their vendors. This is a write-up about how open source tools can be used by the end users to respond to a high profile IoT-vulnerability case.
Emerging IoT-vulnerability case — URGENT/11
URGENT/11 is a set of networking level vulnerabilities affecting a popular IoT and appliance platform called VxWorks. Wind River Systems’ VxWorks is a commercial platform for appliances and hundreds of millions of devices running it are claimed to be connected to the Internet. URGENT/11 vulnerabilities are serious since they allow remote code execution (RCE). Vulnerable devices range from home appliances to critical systems in healthcare, transport, and industry. Vulnerabilities were discovered by Armis Labs and they reported them to the public on the 29th of July, 2019.
Unlike our phones and laptops these vulnerable devices might not patch and update themselves automatically. Some of the devices may be abandonware and not eligible to receive new software or some of the potentially hundreds of vendors might be slow to react to emerging vulnerabilities. We might not even be aware which of our devices run vulnerable VxWorks on the same level as we know if we are using Windows, Linux or OSX on our laptops.
Discover the impacted devices — Nmap
Nmap is a popular network discovery tool to identify devices and services on your network. As part of the scan Nmap can do platform (operating system) fingerprinting. If you are able to run Nmap in all your network segments, subnetworks or virtual networks you will get an inventory of your network assets and a good guess of what operating systems they are running on. Nmap is not a full fledged vulnerability scanner per se and won’t be able to tell if your VxWorks based device has been patched against the vulnerability or not but at least you now have some idea what is your risk level. Especially in the case of emerging IoT-vulnerabilities it is safer to assume that the patch is either not yet available or has not been automatically installed.
Example Nmap run against a single VxWorks-based printer is shown below:
# nmap -O 10.224.1.159
Starting Nmap 7.70 ( https://nmap.org ) at 2019–08–07 15:24 EEST
Nmap scan report for 10.224.1.159
Host is up (0.0090s latency).
Not shown: 995 closed ports
PORT STATE SERVICE
21/tcp open ftp
23/tcp open telnet
80/tcp open http
515/tcp open printer
9100/tcp open jetdirect
MAC Address: 00:80:77:57:B9:07 (Brother industries)
Device type: general purpose
Running: Wind River VxWorks
OS CPE: cpe:/o:windriver:vxworks
OS details: VxWorks
Network Distance: 1 hopTo scan full network ranges, specify the network address instead of a single IP, for example to scan IP addresses 10.224.1.0 to 10.224.1.255 you can use:
nmap -O 10.224.1.0/24Operating system scan has some tunables that may improve your detection results but the defaults will carry you far.
Scanning your networks from outside is a start but there are solid reasons why discovery should be done from within. If you scan from outside of your firewall, NAT or other network apparatus, they might hide the problem out of sight. However later on your firewall rules or topology may chance, someone might gain a foothold inside and move laterally in your networks or the type of vulnerability might leave you exposed without direct connectivity from outside. In case of URGENT/11, the type of vulnerability bites us back. It allows anyone who your IoT-devices connect to or anyone who can do a man in the middle (MitM) on those connections to attack the device even without direct inbound connectivity. Many of IoT-devices connect to backend services on the Internet to share measurements, to listen for commands or in unfortunately rare cases to check for updates. Be prepared to scan from within each and every network you have.
Protect the devices on the traffic level — Suricata
Intrusion detection and prevention systems (IDS/IPS) are a great stop gap measure to some network based vulnerabilities. They are not so great if vulnerability is triggered on the application level inside strong end-to-end encryption, but many vulnerabilities are still exposed outside encrypted envelope. Two most popular open source IDS/IPS products are Snort and Suricata. We use Suricata in our example but you could use Snort instead.
If you already have IDS on you network perimeter you could move fast and before anything else deploy forensic capability as detection rules. This way you have means to know how much of potential attack traffic your networks are receiving. However, don’t let this distract you from the actual response since alerts you may not be real attacks and destination of the traffic might not be a vulnerable device. These false positives are a common case due to opportunistic scans and blind attacks, many times carried out by automated (ro)bots.
Armis Labs’ report on the URGENT/11 gives solid advice and includes four ready made Snort IDS signatures (detection rules). Suricata is mostly compatible with Snort rules. These rules, slightly modified to work with Suricata, are reproduced below for your convenience:
# 1
alert tcp any any -> any any (flags:U+; msg:”OS-VXWORKS — Use of Urgent Flag might indicate potential attempt to exploit an Urgent11 RCE vulnerability”; classtype:attempted-admin; reference:cve,2019–12255; reference:cve,2019–12260; reference:cve,2019–12261; reference:cve,2019–12263; reference:url,armis.com/urgent11; rev:1; sid:1000002;)# 2
alert tcp any any -> any any (flags:SUF+; msg:”OS-VXWORKS Illegal use of Urgent pointer — Potential attempt to exploit an Urgent11 RCE vulnerability”; classtype:attempted-admin; reference:cve,2019–12255; reference:cve,2019–12260; reference:cve,2019–12261; reference:cve,2019–12263; reference:url,armis.com/urgent11; rev:1; sid:1000001;)# 3
alert ip any any -> any any (ipopts:lsrr; msg:”OS-VXWORKS Use of LSRR option, potential attempt to exploit an Urgent11 RCE vulnerability”; reference:cve,2019–12256; classtype:attempted-admin; reference:url,armis.com/urgent11; rev:1; sid:1000003;)# 4
alert ip any any -> any any (ipopts:ssrr; msg:”OS-VXWORKS Use of SSRR option, potential attempt to exploit an Urgent11 RCE vulnerability”; reference:cve,2019–12256; classtype:attempted-admin; reference:url,armis.com/urgent11; rev:1; sid:1000004;)
These rules start with a generic case (#1) which may occur in normal traffic and move to more specific case (#2) and to cover different aspects (#3 & #4) of the URGENT/11 vulnerabilities.
Example alert from the detection rules is shown below:
{
“timestamp”: “2019–08–07T16:31:31.182402+0300”,
“flow_id”: 2157245348490508,
“in_iface”: “enp0s8”,
“event_type”: “alert”,
“src_ip”: “192.168.56.1”,
“src_port”: 21861,
“dest_ip”: “192.168.56.212”,
“dest_port”: 80,
“proto”: “TCP”,
“alert”: {
“action”: “allowed”,
“gid”: 1,
“signature_id”: 1000002,
“rev”: 1,
“signature”: “OS-VXWORKS — Use of Urgent Flag might indicate potential attempt to exploit an Urgent11 RCE vulnerability”,
“category”: “Attempted Administrator Privilege Gain”,
“severity”: 1
}
}How to turn this detection capability to mitigation or stop gap protection? If your IDS is placed in a position where it can control the traffic flowing through it then it can be used as an IPS to block traffic. Rules that alert on traffic can be changed to drop the traffic. However avoid collateral damage and be careful of not disturbing legitimate traffic and services. In an ideal case you would limit the rules to protect only those devices you have identified to be potentially vulnerable. Combining discovery and protection allows to be more radical with potentially vulnerable devices and more conservative on other services you don’t want to disturb.
Why IDS/IPS is a stop gap measure and defence in depth instead of a complete solution? Blocking the potential attack from any traffic at the perimeter firewall or IPS doesn’t eliminate the need to identify and remedy the vulnerable devices themselves. Your firewall rules or topology might change or attacker may gain some other foothold in your networks that allows attack to develop through lateral movement.
Open source based response to IoT-vulnerabilities
Numbered bullets below summarise the process you can take to respond to emerging and network based IoT-vulnerabilities. Pointers are given to illustrate this approach in the URGENT/11 case.
- Follow the news and study the emerging vulnerability. See https://armis.com/urgent11/ vulnerability report, it contains solid advice
- Identify all your networks (segments, subnets, VLANs). You have to rely on your own documentation and experts here.
- Discover impacted devices (in all your networks, from within). https://nmap.org/ is a popular discovery software.
- Protect the devices on the traffic level (configure your IDS/IPS). https://suricata-ids.org/ is a popular IDS/IPS Software and https://armis.com/urgent11/ has URGENT/11 specific IDS/IPS rules (we’ve modified the Snort rules to work with Suricata — see above)
- Find or demand for updates and patches to the devices and deploy them. See excellent collection of vendor responses from GitHub user SwitHak at https://gist.github.com/SwitHak/7e1bfa1e36a5f1f02d900062d188a8a4
Rules of thumb that we have found helpful:
- Speed over perfection
- Identify all your networks and try to be on top of your asset inventory
- Avoid collateral damage by not overdoing firewall or IPS rules
- After covering your bases, apply gentle pressure to the vendors who lag behind
All in all, open source tools are great and give you the ability to react quickly and boost your independence from commercial security tool vendors and vendors of the vulnerable devices
Kudos to Mikko Korkalo of SensorFleet Oy and Juho Juopperi of Remod Oy for reviewing and helping with this blog.
