Automating Firmware Security with FwAnalyzer
At Cruise, we aim to improve our security posture to build the most secure product possible. One major aspect of this goal is securing the various independent components of our all-electric vehicles. These components house everything from the autonomous driving functionality, external communications, and customer facing interfaces. Needless to say, securing these components is highly prioritized and requires continuous attention. The core of these components is their firmware.
In this blog post and upcoming talk, Come Join the CAFSA — Continuous Automated Firmware Security Analysis at Black Hat USA 2019, we present our efforts around continuous firmware security analysis. The source code for our FwAnalyzer tool is available on Cruise’s GitHub page. The slides will also be made available via GitHub.
Firmware Lifecycle
Firmware, like any product, has a lifecycle from inception and prototyping to full force engineering and deployment to production. Each step in the lifecycle has a different focus and will likely impact the security of the firmware and final device in various ways. In addition, many devices are developed by a third party, not in-house. As a result, you have insights and control over fewer steps in the process.
We identified three critical parts of the lifecycle to achieve the best possible outcome for security:
- Development — specifically the everyday CI-based engineering effort
- (Acceptance) Testing — especially for firmware delivered by third parties
- Production Release — security as gatekeeper for production release and signing
With this in mind, we developed processes and tools to efficiently analyze the firmware for our devices. As you will see, our processes and tools are significantly different compared to traditional methods, such as vulnerability scanners or CVE enumeration.
Introducing FwAnalyzer
We developed FwAnalyzer, a tool to analyze firmware for security issues. FwAnalyzer specifically targets filesystem images and, therefore, targets devices that have a clear separation between OS, applications, and filesystem. FwAnalyzer was built for Linux-based devices, including Android, but can be augmented to work on similar platforms.
How FwAnalyzer works
FwAnalyzer examines a filesystem image by applying a set of rules from a configuration file. The result of the analysis is returned as a machine readable report. Targeting filesystem images is the logical choice since they provide context, combine metadata such as pathnames, ownership, and permissions, along with the actual file content. Analysis of a standalone file cannot provide the same insights.
FwAnalyzer implements a file system abstraction layer through which the analyzer functionality is implemented independently from the filesystem type. Adding support for new filesystem types is fairly easy. FwAnalyzer currently supports ext2/3/4, VFat, squashfs, UBIFS, and files in a local directory.
Before FwAnalyzer can start its work, the filesystem images have to be extracted from a firmware update file. This step is not part of FwAnalyzer since it solely depends on the specific device, and thus will not be further discussed in this blog post. However, our open source release includes an unpacking and checking script as well as example configuration for Android firmware. This sample script nicely shows how to unpack and check a firmware image end-to-end.
Configuring rules with FwAnalyzer
Once the filesystem images are extracted, it is time to configure the rules that are to be applied. FwAnalyzer implements three different types of rules.
Rules that are applied to file metadata such as permissions, type and ownership. Access to file metadata allows implementing checks around access control — a fundamental security control on any system. FwAnalyzer provides support for SELinux labels and could be extended to provide support for other extended attributes.
Rules that target the content of a file. Analyzing the contents of a file is an obvious feature. In fact, a large number of our checks target the file content. Overall, we designed our file content analysis rules to be easy to understand and use. We implemented a number of checks that each provide different insights into file content. These checks will be described in more detail later in this blog post.
Rules that analyze filesystem metadata. This includes things like whether a specific file exists or not, as well as whether a directory contains specific files. Furthermore, it provides the capability to compare the metadata of two filesystem images to determine if files were added, removed, or modified. This functionality can be used to compare two different versions of the same filesystem image.
In addition to rules that perform a check of some kind, FwAnalyzer implements a data extraction subsystem. This gives FwAnalyzer the ability to include data extracted from files on the filesystem in a detailed report. The extracted data can provide valuable information such as versioning or configuration details. These details provide precise information about the specific firmware and can be used to implement more complex checks at a later phase of your security pipeline.
Rules and Examples
One of the most simple rules flags setuid (SUID) files to help identify possibly dangerous executables. SUID executables, often used to provide elevated privileges for specific actions, are notorious for local privilege escalation vulnerabilities, especially if those applications are developed in-house without going through proper security design and implementation review. The rule also allows ignoring certain known good SUID files such as common system utilities.
Similarly, checking if a file exists can reveal that debugging and development binaries are present in the filesystem. Real world examples include CVE-2018–5399, where an ssh service was found on a production firmware and was removed in later firmware version.
Analysis of file content is the core of FwAnalyzer. There are a number of types of file content checks. Below we present the three most important mechanisms:
First, you can compare the SHA-256 digest of the file with a preconfigured value. This allows us to check that a specific file has specific content. This can be useful in the context of development vs. production public keys and certificates stored on the filesystem.
Second, you can apply a regular expression to the contents of a file. This mechanism can be used to check configuration files to ensure certain settings are present. Similarly, release information stored on the filesystem can be checked to ensure the specific firmware update file was the result of a production built. One example would be Android properties such as ro.build.type that clearly indicates the type of build.
For more complex checks, FwAnalyzer supports running an external script or binary to analyze the content of a file. For this, the file is extracted to a temporary location so the external script does not need to understand or care about anything besides the file it analyzes. External scripts enable easy extension of FwAnalyzer without modifying FwAnalyzer itself.
Using FwAnalyzer to prevent security issues
Armed with these capabilities, you now have the ability to detect and prevent a wide variety of security issues. Using an external script that we provide, you can detect any non-stripped binaries, preventing leaking potentially valuable debug information.
You can ensure that files have the right permissions, so you do not end-up with issues such as CVE-2014–5457 where a password file is world readable. Further, you can use FwAnalyzer to prevent shipping firmware that contains private key files such as was seen in CVE-2017–8222.
These are just a few examples of what FwAnalyzer can do. Overall checks will be contingent on the product and its functionality, but the underlying mechanisms provided by this tool are universal.
Continuous Firmware Security
FwAnalyzer is built to integrate into the development, test, and release processes. It is intended to be used for automation and produces machine readable output.
At Cruise, we integrated FwAnalyzer into our development process as a check in CI, providing direct feedback during development and runs as one part of the standard unit tests. During CI, we use a different set of configurations for FwAnalyzer, depending on the build and signing type. This allows us to enforce more stringent security rules for production builds while not hindering development builds by being overly strict.
In addition, this also helps to keep the balance between security and ease of development. Examples of differences would include stricter firewall rules and flagging specific binaries for production builds. For example, we may allow debugging tools or ssh for development builds, but not production builds.
We also operate an analysis service for firmware images delivered to us from third parties. The service analyzes firmware and provides feedback to us and the supplier. Adding a new device to FwAnalyzer requires a one-time investigation and the creation of a corresponding FwAnalyzer configuration. Afterward, only minimal maintenance is required to support future releases of firmware for that device. Maintenance usually is required if the firmware update contains major changes. While the service is just one part of the acceptance testing process, it helps to speed it up due to being fully automated.
FwAnalyzer and production signing
Finally, we use FwAnalyzer as a gatekeeper for production signing of firmware. Production signing is an important security control since it is the step that enables a specific firmware image to be deployed into production.
FwAnalyzer guards the production signing in multiple ways. Most importantly, it analyzes the firmware image and only allows it to be signed if it passes all checks. This step should eradicate accidental signing of bad firmware, such as development builds or engineering builds created by an individual developer. After all, we are all human and humans can make mistakes, especially when deadlines are close. By only production signing firmwares that have passed analysis, we eliminate these types of errors. Further, the analysis report generated by FwAnalyzer serves as a permanent record for each specific firmware image, especially when using the data extraction subsystem to its full extent.
One main feature of FwAnalyzer’s JSON formatted report is that it can be used to further automate processes around security and security Q&A. At Cruise, we upload all FwAnalyzer reports to our central logging service. Using the logging service’s filters and triggers, we implement a variety of reporting and notifications using common off-the-shelf, cloud-based services. One simple example of this is having all analysis reports easily searchable. This allows one to easily access a report by searching using the version or the digest of the firmware in question.
Automating Firmware Security is a step in the right direction
FwAnalyzer has provided valuable results to the Cruise security teams on multiple occasions, some of which would otherwise have been very costly. It also enables us to quickly engage developers and third parties shortly after receiving new firmware revisions.
After discovering issues, we are able to quickly implement new checks to automatically flag the issue in the future and effectively prevent regressions. We strongly believe that automation for firmware security is a big step in the right direction, especially when dealing with a large number of different devices and firmware.
FwAnalyzer was designed to be easily extendable to add support for additional filesystem types, as well as adding new checks. The FwAnalyzer configuration file format is designed to allow including and sharing of rules between build types, devices, and even across organizations.
Firmware security is extremely important not only for us at Cruise, but for every one who builds products. We decided to open source FwAnalyzer to help others to build more secure products as well.
In addition to releasing FwAnalyzer completely open source under the Apache license, we will also include a number of example configuration files, scripts to perform various checks, as well as an example end-to-end unpacking and checking script for Android.
If you are interested in making bespoke tools of your own to help solve the engineering challenge of our generation and potentially open source, take a look at our open roles.
