Exploring the Ubiquiti UniFi Cloud Key Gen2 Plus
Scoping attack surface, setting up debugging for UniFi Protect and UniFi Management Portal APIs, and finding unauthenticated API vulnerabilities
Recently I’ve been researching the Ubiquiti UniFi Cloud Key Gen2 Plus. The Cloud Key products are hardware appliances that enable management of network devices both locally via the pre-installed UniFi Network Controller and remotely using a Ubiquiti cloud account. The Gen2 Plus also includes UniFi Protect network video recorder (NVR) software to manage surveillance cameras and recordings.
I decided to target the Gen2 Plus because devices and software related to surveillance are critical assets to keep secure. I also found it interesting that the UniFi Protect software was only available on a few Ubiquiti hardware devices — the Cloud Key Gen2 Plus, the UniFi Protect Network Video Recorder, and the UniFi Dream Machine Pro — and not as a standalone software installation as some other Ubiquiti UniFi applications are. Since UniFi Protect is not as readily available without a hardware investment, I surmised that it may not get as much attention from researchers and could likely use some extra scrutiny.
Attack Surface
The first thing that struck me about the Ubiquiti UniFi Cloud Key Gen2 Plus was the broad attack surface. The appliance has dozens of ports open to the network associated with over a dozen processes. Fortunately, one of those processes is sshd, so I was able to log in to the device via SSH and enumerate listeners locally. SSH credentials for the Cloud Key are the same as the credentials for UniFi Management Portal on ports 80 and 443, with ubnt/ubnt or root/ubnt as the default.
A few of the listening ports exposed common services, like NTP and SSH, but the majority were associated with the three applications installed by default on the Gen2 Plus:
- UniFi Network Controller, used for managing Ubiquiti network devices
- UniFi Management Portal, providing a GUI for managing Cloud Key devices
- UniFi Protect, used to manage surveillance cameras and recordings
Narrowing the Scope
I decided to focus on UniFi Management Portal and UniFi Protect, both node.js applications.
I enumerated the listening services and their ports using netstat and ps. Since netstat only shows the process command and not its arguments, I used “ps -p $pid -o args=” to look up the full command string for each process and piped the output through “cat” so it wouldn’t be truncated.
The node process running UniFi Management Portal listens on localhost ports 10080 and 10081 and is exposed externally via the nginx process on ports 80 and 443. This app provides a graphical user interface to manage the operating system of the Cloud Key, including firmware updates, SSH credentials, and system settings. Browser tools show that client-side JavaScript interacts with the server via API requests. The API is implemented in ump.js under /usr/share/unifi-management-portal/app/be/.
UniFi Protect is a much more complex application, with processes listening on five localhost ports and ten external ports. The UniFi Protect node process primarily listens on external ports and connects to a local postgresql database. The remaining listening ports related to UniFi Protect are associated with evostreamms, which is used to stream audio and video from connected devices.
I was able to map out labels for the listening protocols using UniFi Protect configuration files. The ems.json file found under /srv/unifi-protect/data maps each port that evostreamms listens on to a protocol label, e.g. “inboundWsJsonCli” on port 7440.
The config.json file under /usr/share/unifi-protect/app/config maps out the listening ports for the main UniFi Protect app as well as evostreamms ports that UniFi Protect interacts with.
Packet captures of the device show that these services are very chatty among themselves and with remote servers even when the device is otherwise idle.
UniFi Protect provides an interesting array of listening protocols, but I decided to focus on the server API first, implemented in server.js under /usr/share/unifi-protect/app/.
Analyzing Source Code
With my targets selected — the server APIs implemented in ump.js and server.js — my next step was to take a look at the source code, minified JavaScript. What’s fun about minified code is that it’s condensed into extremely long lines, many of the variables are a single letter, and it’s practically impossible to read or grep sanely. So the first step was to get the source code into a legible and easily searchable format.
Prettifying ump.js was easily handled with the jsbeautifier package.
UniFi Protect’s server.js was another story. I tried a handful of CLI utilities to prettify server.js but they all left the majority of the source code on a single line at the end. Ultimately, after seeing how well Google Chrome DevTools prettified the client-side source code, I found that Chrome’s DevTools for Node was able to prettify the entirety of server.js as well. (More on debugging server.js below.)
Node Debugging
As was the case with prettifying the code, setting up debugging was easier for UniFi Management Portal’s ump.js than it was for UniFi Protect’s server.js.
UniFi Management Portal
There were a few key files related to starting the unifi-management-portal service:
- Startup script: /usr/share/unifi-management-portal/bin/unifi-management-portal
- Service start location: /usr/bin/unifi-management-portal, a symlink to the above file
- Config file: /etc/default/unifi-management-portal, containing environment variables NODE_ENV and LOG_LEVEL and sourced by the startup script
To set up remote debugging for ump.js, I copied the startup script to /usr/bin/unifi-management-portal.debug and added the --inspect flag to the node command. To run the app with debugging, I stopped the unifi-management-portal service and then ran my modified startup script.
Initially, there were very few debugging logs. I looked for LOG_LEVEL in my prettified copy of ump.js and found the log level options, ranging from 0–5 with “LOG” as the highest level.
Then I made a debugging copy of the config file, changed its LOG_LEVEL to LOG, and updated the filename to source in the debugging startup script.
Starting the app with debugging then showed a lot more log activity.
Once that was done, I navigated to chrome://inspect from a Chrome browser on the same network and configured network target discovery to include <IP>:9229. The inspect link came up and I was able to connect to the remote debugger. The logs in the remote debugger included links to source code, so I could jump to ump.js, prettify the code, set breakpoints, and step through.
UniFi Protect
UniFi Protect’s server.js was a bit more trouble. The basics were the same:
- Startup script: /usr/share/unifi-protect/bin/unifi-protect
- Service start location: /usr/bin/unifi-protect, a symlink to the above file
- Config file: /etc/default/unifi-protect, containing several environment variables and sourced by the startup script
Initially, I tried just adding the --inspect flag to the node command as I had for ump.js. I stopped the default service and started the app with debugging. The output showed that the debugger was listening, but there were no logs written to the console. When I connected to the process via the Chrome remote debugger, I got an empty console and no sources shown.
I tried adding the --inspect-brk option to the node command in the startup script and tried again to run the app with debugging. The process hung, waiting to be resumed from the debugger. This time, when I opened the inspect debugger I was able to access the server.js source.
I later discovered that when the inspect console was empty like that, closing it and clicking “inspect” again to re-launch it would eventually connect successfully, though it could take a few tries. Using --inspect-brk I was able to connect reliably on the first try each time, but had to resume execution in the debugger.
Once I had access to server.js in the Chrome debugger, I was able to fully prettify server.js and saved a prettified copy as a reference.
Debugging logs for UniFi Protect weren’t printed to the console or the remote debugger by default, but I was able to view them in real-time by tailing the log files under /srv/unifi-protect/logs/. With tail -f running in the background, once the debugger was connected and the execution was resumed, logs were written to the console.
I eventually found the “enableConsole” option which can be set in the UniFi Protect configuration files. The default UniFi Protect configuration file is the config.json file mentioned earlier that maps the ports and protocols and is located at /usr/share/unifi-protect/app/config/. By default, “enableConsole” is set to false.
Fortunately, changing “enableConsole” to true did not stop the app from also logging to the files under /srv/unifi-protect/logs, so I was able to enable the setting and leave it on. In order to apply custom settings, “override” config files are used, defined in the default config.json.
Each additional file is loaded in order if it exists, and custom settings added to those files override the default configuration. I created a new file under /etc/unifi-protect to set “enableConsole” to true. I could then start the app with console debugging.
The debugging output showed the order in which the config.json files were loaded, with the new file under /etc/unifi-protect loaded last to override the others.
There were several log levels defined in server.js with the highest being “silly” level 5. I increased all log levels to 5 by adding the “levels” settings to my custom config.json.
“Silly” debugging was, as expected, quite verbose, but because the console logs were duplicated to the log files I had those as a backup to search if I missed something.
Now that I had the debugging modifications figured out for each app, I wrote scripts unifi_mp_debug_setup.sh and unifi_protect_debug_setup.sh to allow me to reapply them automatically after resets and updates.
Parsing and Testing API Endpoints
Once I had prettified code for both apps, I was able to identify the code conventions used to define API endpoints and enumerate available endpoints using grep to parse them out.
I started by opening the UI of each app in the browser and using dev tools to monitor activity. This showed the API requests to the server which included some example endpoint names, such as “/api/ump/device” and “/api/webrtc/sdp”.
I grepped the prettified source file of each app for the API endpoint examples in order to see how the endpoints were implemented. Once I had the expected pattern to look for, I was able to enumerate all of the available endpoints along with their associated HTTP methods for each app.
With those lists of endpoints, I was able to script testing each of the APIs. Testing each API endpoint without first authenticating to the app allowed me to enumerate endpoints available pre-authentication, which if vulnerable, would be most accessible for exploitation. Endpoints requiring prior authentication consistently returned an error message unique to each app which the testing scripts were able to filter out.
The remaining endpoints did not return those particular error messages which indicated that they were accessible without first authenticating to the API.
Some endpoints returned HTTP status code 200, indicating the request was successful as it was and no data was required to be sent. The remaining responses I reviewed manually and found helpful error messages indicating which fields were required for requests to each endpoint.
The endpoints that could be accessed without authentication were where I decided to focus code review efforts.
I added some proof of concept scripts to the tenable/poc repo on github:
- enumerate_unifi_mp_apis.py
- test_unifi_mp_apis.py
- enumerate_unifi_protect_apis.py
- test_unifi_protect_apis.py
I feel obliged to emphasize that these are proof of concept scripts for research purposes and aren’t intended for production devices.
What could go wrong with arbitrarily testing API endpoints? Well, most of my API testing was performed externally, but I did try running the test scripts from the command line of the Cloud Key itself to test localhost. I learned that the UniFi Management Portal API doesn’t require authentication for requests from localhost, and I also learned that arbitrarily sending requests to all of the available UMP APIs is not a very safe thing to do. In particular, the /ump/device endpoints can power off, reboot, or factory reset the device with a GET request from localhost.
At least the script hit “power-off” first and not “reset.” Silver linings!
Hunting Unauthenticated API Vulnerabilities
Using the results of the API test scripts, the prettified source code, and remote debugging, I started looking for vulnerabilities in the Cloud Key’s APIs that could be exploited without prior authentication.
Unauthenticated Hostname Modification via UniFi Management Portal “setup” Endpoints
It seemed odd to me that after setting up the UniFi Management Portal, there was still an unauthenticated API option available to set up the UniFi Management Portal. The error message from the scripted testing indicated several fields were required.
Most of the fields were obvious, but I wasn’t sure what “name” was supposed to be. I looked to the related code in the prettified ump.js and found that the name field was being passed to setHostname().
I tested out an API request with test data in each of the required fields, and then checked the device. The hostname, visible on the LCM screen on the front of the Cloud Key, was indeed updated. I logged in to the UMP UI and confirmed that the credentials and timezone had not changed, although the hostname had.
While in the UMP UI, I saw that there was a Cloud Key firmware update available and applied it. After the update, the exploit no longer worked; the /ump/setup and /umc/setup endpoints were no longer available. I prettified the new ump.js and diffed it with the previous version and found that the endpoints had been removed completely.
I found the release notes for firmware version 1.1.10, and there was a vague mention of the issue — “Removed an API endpoint for security concerns” — but no security advisory and no details. As the vendor had already patched and disclosed the issue, Tenable released security advisory TRA-2020–21 with further details and notified Ubiquiti, requesting CVE information. Ubiquiti replied promptly, confirming that the issue had been patched in 1.1.10, and followed up the next morning with a CVE assignment for reference and a detailed disclosure of their own.
Username Discovery using Multiple Methods via UniFi Protect “auth” Endpoint
Looking over the code for the UniFi Protect “auth” endpoint, I noticed that the function returns immediately if the username is wrong, or proceeds to checking the password if the username is found.
Testing on the command line with time and curl, I found that the “invalid username or password” error was returned more quickly for invalid usernames than for valid usernames with invalid passwords.
The time difference, almost an order of magnitude greater for valid usernames, was significant enough to use reliably for identifying valid vs invalid usernames.
Looking at the code again, I realized that a timing attack wasn’t even necessary — the function was passing a different HTTP status code depending on whether the username was invalid or valid with an invalid password. Since I had been testing with curl, I hadn’t seen the HTTP code with each response. Adding ‘-w “ %{http_code}”’ to the command, I confirmed the difference:
While that obviated the need for a more complex timing attack, both vulnerabilities needed to be addressed in order to remove the ability to enumerate valid usernames. I reported both issues to Ubiquiti and released security advisory TRA-2020–45 once the issues were patched.
Further Research
I’ve barely scratched the surface of the Ubiquiti UniFi Cloud Key Gen2 Plus. There’s a lot of untapped potential in researching the device, its applications, and its numerous listening services. The EvoStream application bundled with UniFi Protect is also an interesting target, apparently a fork of open source project crtmpserver which was last synced with EvoStream in 2015. My initial API testing focused on endpoints accessible without authentication, so I still haven’t explored the majority of the API endpoints enumerated for both apps. Sometime soon, there will be a new app to explore on the Cloud Key; Ubiquiti mentioned in the release notes for UniFi Protect 1.13.4 at the end of July that UniFi OS will be coming to the Cloud Key in the coming weeks. It will be interesting to see how that interface differs from the APIs explored above.
