L1TF: Impact on Intel SGX and TEEX’s Virtualization-based Defense
L1TF is a new vulnerability of Intel CPU which exists on a wide range of platforms and is easy to utilize. L1TF also breaks the security guarantee of Intel SGX (Software Guard eXtension, a hardware security extension), which affects many systems that leverage SGX for security enforcement. In this article, we first give a detailed introduction to L1TF and the official defence method provided by Intel. Then, we introduce a new virtualization-based mechanism to defend against L1TF, which is provided by TEEX team (https://teex.io). Finally, we have a brief discussion on how to leverage Intel SGX in blockchain system given that SGX is not perfect.
1. What is Intel L1TF?
The L1TF (L1 Terminal Fault) bug allows an unprivileged attacker to steal privilege data residing in the L1 data cache. It has three variants:
· L1 Terminal Fault — SGX：An user-level attacker can steal the Enclave data residing in the L1 data cache;
· L1 Terminal Fault — OS/SMM：An unprivileged attacker can steal privileged data (belong to either OS or SMM) residing in the L1 data cache;
· L1 Terminal Fault — VMM：A malicious guest VM (virtual machine) can steal the data belonging to either VMM (virtual machine monitor) or other guests VMs from L1 data cache.
L1TF exists on almost all the Intel CPUs which support speculative execution as well as a page table. The official list of the influenced CPUs includes almost all the new Intel CPUs in recent years.
1.1 Background: Page Table, Cache, Speculative Execution and Convert Channel
· Page Table: Page table is used to manage memory. When it is enabled, the CPU will use a virtual address to access the memory. The page table is used to translate the virtual address to physical address which is used to access the physical memory. Each virtual page (commonly 4KB) has its own page table entry, which records its corresponding physical page and permission bits. There are two permission bits related to L1TF, the P-bit (the present bit) and the reserved bit. These two bits will be checked when the entry is used to perform address translation.
· Cache: Cache is used to accelerate memory access. The CPU cache can be divided into three levels (from Level-1 to Level-3). L1 cache (Level-1 cache) is the smallest and fastest one and can be further divided into L1-data cache and L1-instruction cache. In a multi-core CPU, each CPU core has its own L1 cache.
· Speculative Execution: Speculative execution is used to accelerate the instruction execution. When CPU transfers the control flow and cannot immediately gets the new instruction (e.g., indirect jump, interrupt), it will speculatively execute the following instruction. If the speculation is correct, the results are committed and the performance can be improved. Otherwise, the CPU will rollback all its execution results.
· Convert Channel: Convert channel is an indirect data transfer method. Two isolated components can control the change of some system status and leverage such change to bypass isolation mechanisms and transfer message. One example is cache convert channel, which leverages cache status (hit or miss) to transfer message.
1.2 Introduction of L1TF
When the CPU accesses the memory, the virtual address will be translated to the physical address with the help of the page table. During the translation, if the corresponding page table entry is invalid (i.e., the check of the P-bit or the reserved bit fails), the CPU will immediately stop the translation and throw a Page Fault, which is also called “Terminal Fault”.
However, when a terminal fault happens, the CPU may speculatively execute the next few instructions instead of immediately jumping to the fault handler. At this time, the CPU speculatively generates a physical address (according to the content of the page table entry) and continues the execution of the memory access instruction. If the data of the target physical address reside in the L1-data cache, the data will be loaded and passed to the following instructions speculatively.
Finally, all the results of the speculative execution will be rollbacked when the CPU jumps to the fault handler. However, the CPU will not rollback the change of the cache status. Thus, the speculatively executed instruction can build a cache convert channel and pass the data to the attacker.
In the above example, the CPU speculatively generates the physical address to access physical memory after the terminal fault. So accessing this physical address will bypass many memory access checks, including:
· Bypass the Enclave memory protection of SGX. Enclave’s data will be encrypted in the memory and only be decrypted in the cache. After the address translation, CPU will check whether the target physical address belongs to current Enclave;
· Bypass the memory protection of SMM. SMM (System Manage Mode) is a high privilege mode in Intel CPU. After the address translation, CPU will check whether target physical address belongs to SMM and whether the current mode has the privilege to access it;
· Bypass the address translation of Intel EPT (Extended Page Table). In the virtualization environment, the second address translation will be performed using EPT after the first address translation (with page table in guest VM). EPT is widely used to isolate the memory between different VMs.
Thus, L1TF has three variants accordingly: L1TF-SGX, L1TF-OS/SMM and L1TF-VMM.
1.3 Requirements of L1TF
L1TF bug requires that the target data must be loaded into the L1-data cache of the CPU core where the attack code runs. Specifically, L1TF has the following three requirements and all of them need to be satisfied:
1. The target data must be loaded into the L1-data cache of the CPU core which locates the attack code. Since the size of L1-data cache is limited and the data in it are replaced frequently, the attack window is very small.
2. The attack code must be located in the CPU whose L1-data cache contains the target data. At the same time, there must be a vulnerable page table entry, which can be used to construct the physical address of the target data.
3. The attacker can only get the data with speculative execution and convert channel.
Requirement 1 means that the attacker must enforce that the target data are loaded in L1-data cache. Requirement 2 and 3 mean that the attack code must run in the CPU core whose L1-data cache contains the target data. With hyper-threading (HT), one CPU core has two physical threads which can execute instructions concurrently and share the L1-data cache. With HT enabled, the attack code is only needed to be located in one of the two physical threads of the target CPU core.
2. L1TF’s impact on Intel SGX
Intel SGX is the new hardware security feature of Intel CPU, which provides a TEE (trusted execution environment) called Enclave. Several new instructions are provided to create, manage and destroy an Enclave. All the code and data within an Enclave are stored in a specific physical memory region called EPC (Enclave Page Cache). Data in EPC are encrypted in memory and decrypted only in the cache. The Enclave can not only defend against software-based attack from the privileged mode but also the physical attack (e.g., cold boot).
2.1 How to leverage L1TF to read the Enclave data?
Although the Enclave data is encrypted in the memory, it is decrypted in the cache. L1TF can bypass the memory access check of Enclave so that the attacker can steal the Enclave data loaded in L1-data cache. There is a paper, “FORESHADOW: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution“[1] published on USENIX Security 2018 that contains more details of the attack.
The paper points out that it is easy to leverage the L1TF bug to read the Enclave data since Intel provides a new instruction eldu, which is used to load Enclave data from normal memory into the EPC. Meanwhile, the eldu also loads the data into the L1-data cache. With eldu, an attacker does not need to guess when the Enclave data will be loaded into L1-data cache. She can directly load the target data into the L1-data cache of current CPU core and then leverage L1TF-SGX to read the data.
2.2 The defence methods provided by Intel
Intel provides the microcode patch to fix the L1TF-SGX bug. The patch includes two defence methods：
· Flushing L1-data cache during Enclave enter/exit. It enforces that all Enclave data will not have resided in L1-data cache after Enclave exit so that the data cannot be stolen with L1TF bug.
· Allow Enclave to test whether hyper-threading is enabled. Intel advises the user to not trust the Enclave running on the CPU with hyper-threading enabled.
However, Intel does not mention eldu instruction in its document[2]. We do not know whether the eldu instruction is patched. If it is not, the attacker may still leverage L1TF to steal the Enclave data.
Intel’s defence method requires the user to not trust the Enclave running on the CPU which enables hyper-threading. However, only the BIOS can disable the hyper-threading feature and it cannot be performed at runtime. It means that a user is forced to make a choice between hyper-threading and SGX. Meanwhile, some hardware vendors do not support turning off hyperthreading in BIOS, which makes it impossible for users to use SGX securely on these platforms.
3. Virtualization-based Security Enhancing Method for SGX
Consider that 1) it is not clear whether the eldu instruction is patched, 2) some machines may not be patched and 3) there may be more bugs of SGX, TEEX team proposes virtualization-based security enhancing method for SGX called TXVisor. TXVisor leverages virtualization to isolate different Enclaves and the out-enclave environment, which enforces that the enclave data cannot be stolen with the L1TF-SGX. TXVisor has following advantages:
· L1 Cache Isolation: For each Enclave exit, TXVisor flushes the L1 cache of the current CPU core, so that attacker cannot leverage L1TF to steal the Enclave data, even if the CPU is not patched.
· ELDU Security Enhancement: For the eldu instruction, TXVisor will trap its execution, performs a security check and enforces that it cannot destroy the L1 cache isolation.
· Hyper-threading Security Enhancement: TXVisor controls the scheduler to forbid an attacker leveraging hyper-threading technology to perform L1TF attack. By isolating different Enclaves and out-enclave environment, TXVisor can enforce the security of Enclave while allowing the user to use hyper-threading to improve the performance out of the Enclave.
· TEE Attestation: TXVisor provides a software-hardware combined remote attestation method, which allows the user to attest whether the application is protected by SGX and TXVisor.
TXVisor can defend against attacks like L1TF, even if the platform is not patched. Meanwhile, we also provide several optimizations to reduce the performance overhead while enforcing security. TXVisor is based on many previous research findings on virtualization, TEE and isolation that are done by the TEEX team, including CHAOS [ISTR’07]、CloudVisor [SOSP’11]、HyperCoffer [HPCA’13]、TxIntro [HPCA’14] (best paper nomination) 、CrossOver [ISCA’15]、SeCage [CCS’15]、AdAttester [MobiSys’15]、Nexen [NDSS’17]、SGX Migration [DSN’17]、vTZ [USENIX Security’17]、VButton [MobiSys’18]、Fidelius [HPCA’18] and EPTI [USENIX ATC’18].
4. SGX and Blockchain
Data privacy is a key challenge of the blockchain. Currently, many researchers are focusing on how to leverage security hardware (e.g., Intel SGX) to provide privacy for blockchain. Existing works have proven that SGX can effectively improve the data and code privacy of blockchain.
However, SGX is no silver bullet. Either the L1TF-SGX bug or some precious side-channel attacks (e.g., spectre) shows the security problem of SGX. We think that a good method to leverage SGX needs to consider the imperfection of its security.
Some of the existing solutions rely on SGX excessively, for example, designing a consensus protocol based on the security of SGX. For these solutions, once there is a bug in SGX, the entire blockchain could be destroyed. TEEX team aims to leverage SGX to enforce the privacy of the user data and control logic, and combining virtualization technology and blockchain to limit the impact of SGX bugs. The main idea of TEEX is to use SGX to improve privacy while not relying on it as the trust anchor.
To sum up, we believe that the TEE (Trusted Execution Environment) will be a key technology of blockchain, same as the cryptology. TEEX team tries to provide a secure framework for both the current and the future blockchain system and promotes the developing of the eco-system, based on our rich experience of TEE. More discussion on how to combine TEE and blockchain will be presented in the future.
About TEEX
TEEX team aims to construct a decentralized trusted execution framework. Our team has proposed a variety of TEE solutions, including Intel SGX, ARM TrustZone, AMS SME/SVE, TPM/TXT, formal verification and virtualization for many years. We are also experienced in how to leverage TEE in the blockchain. More info can be found in https://teex.io.
References
1. Van Bulck J, Piessens F, Strackx R. Foreshadow: Extracting the Keys to the Intel {SGX} Kingdom with Transient Out-of-Order Execution, USENIX Security’2018.
3. Daonity-Grid Security from Two Levels of Virtualization. Haibo Chen, Jieyun Chen, Wenbo Mao, and Fei Yan. Elsevier Information Security Technical Report. (Invited Paper), Volumn 12, Issue 3, pp. 123–138. June 2007.
4. Fengzhe Zhang, Jin Chen, Haibo Chen and Binyu Zang. CloudVisor: Retrofitting Protection of Virtual Machines in Multi-tenant Cloud with Nested Virtualization. In 23rd ACM Symposium on Operating Systems Principles, SOSP, 2011.
5. Yubin Xia, Yutao Liu and Haibo Chen. Architecture Support for Guest-Transparent VM Protection from Untrusted Hypervisor and Physical Attacks. In Proceedings of 2013 Internal Symposium on High Performance Computer Architecture, February, 2013.
6. Yutao Liu, Yubin Xia, Haibing Guan, Binyu Zang, Haibo Chen. Concurrent and Consistent Virtual Machine Introspection with Hardware Transactional Memory. In Proceedings of the 20th IEEE International Symposium On High Performance Computer Architecture (Best Paper Nominee), Orlando, Florida, 2014.
7. Wenhao Li, Yubin Xia, Haibo Chen, Binyu Zang, Haibing Guan. Reducing World Switches in Virtualized Environment with Flexible Cross-world Calls. In the 42nd International Symposium on Computer Architecture, Portland, Oregon, USA. June 2015.
8. Yutao Liu, Tianyu Zhou, Kexin Chen, Haibo Chen, Yubin Xia. Thwarting Memory Disclosure with Efficient Hypervisor-enforced Intra-domain Isolation. Proceedings of the 22th ACM Conference on Computer and Communications Security, Denver, Colorado, US, October, 2015.
9. Wenhao Li, Haibo Li, Haibo Chen, Yubin Xia. AdAttester: Secure Online Advertisement Attestation on Mobile Devices Using TrustZone. In Proceedings of the 13th International Conference on Mobile Systems, Applications, and Services, Florence, Italy. May 2015.
10. Lei Shi, Yuming Wu, Yubin Xia, Nathan Dautenhahn, Haibo Chen, Binyu Zang, Haibing Guan, Jingming Li. Deconstructing Xen. The Network and Distributed System Security Symposium 2017. San Diego, California. March, 2017.
11.Jinyu Gu, Zhichao Hua, Yubin Xia, Haibo Chen, Binyu Zang, Haibing Guan, Jinming Li. Secure Live Migration of SGX Enclaves on Untrusted Cloud. The 47th IEEE/IFIP International Conference on Dependable Systems and Networks, June 26–29, 2017. Denver,CO,USA.
12. Zhichao Hua, Jinyu Gu, Yubin Xia, Haibo Chen, Binyu Zang and Haibing Guan. vTZ: Virtualizing ARM TrustZone. USENIX Security Symposium 2017. Voncouer, Canada, August, 2017.
13. Wenhao Li, Shiyu Luo, Zhichuang Sun, Yubin Xia, Long Lu, Haibo Chen, Binyu Zang, Haibing Guan. VButton: Practical Attestation of User-driven Operations in Mobile Apps. In Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services, 2018.
14. Yuming Wu, Yutao Liu, Ruifeng Liu, Haibo Chen, Binyu Zang and Haibing Guan. Comprehensive VM Protection against Untrusted Hypervisor through Retrofitted AMD Memory Encryption. The 24th IEEE International Symposium on High-Performance Computer Architecture, Vienna, Austria. Feb 2018.
15. Zhichao Hua, Dong Du, Yubin Xia, Haibo Chen, Binyu Zang. EPTI: Efficient Defence against Meltdown Attack for Unpatched VMs. USENIX ATC, 2018.
