Confidential computing solution case studies(Intel SGX, AMD SEV-SNP and ARM CCA comparison)
AMD SEV-SNP(Secure encryption virtualization-Secure nested paging). Virtual machine(VM) based confidential computing solution with integrity protection
Following threat vectors are primarily addressed by SEV-SNP architecture as depicted by below diagram based on analysis of available online documents:
- Provide per-VM data-in-use confidentiality to mitigate device DMA memory copying attack and other physical attacks like DDR link snooping, cold boot memory dump etc.
- Provide per-VM data-in-use confidentiality against unauthorized data access from high privileged components or users eg. hypervisor and administrators, cloud service provider, platform software stacks etc.
- Provide per-VM data-in-use integrity threat mitigations: Replay, data corruption, memory aliasing or re-mapping. Similar to secure storage or disk encryption threat model, confidentiality alone is not enough.
As illustrated by the SEV architecture, it relies on the hypervisor for performing virtualization functionalities such as device emulation and VM scheduling but delegates security operations to secure processor via securely managed APIs. To protect SEV enabled guest VM, SEV firmware assists in the enforcement of main security properties: platform authenticity, remote attestation of a launched guest, the guest’s data-in-use confidentiality & integrity.
SEV uses per-VM key to isolate guest VM and the hypervisor. Actual keys are managed by Secure Processor. While the hypervisor must manage/schedule the guest and its resources, the hypervisor never gains knowledge of the memory encryption keys. The SEV firmware that runs within the secure processor provides a secure key management interface to accomplish this. The hypervisor uses this interface to enable SEV for secure guests and perform common hypervisor activities such as launching, running, snapshotting, migrating, and debugging a guest.
Hardware inline AES engine in the memory controller provides per-VM private memory encryption/decryption for data-in-use confidentiality. When hypervisor assigns memory pages to launch guest VM, physical addresses tagged by SEV hardware with enCrypted bit(AXI address b47) for each VM private memory pages will be encrypted/decrypted by using per-VM key via hardware AES engine during any instruction fetching or data load/store bus transactions.
Each guest VM’s memory pages are encrypted with different AES key managed by secure processor, which provides data-in-use confidentiality. Compromised hypervisor or other high privileged entities might be able to capture and dump memory contents from guest VM address spaces, however they only can see the cipher-texts without access to corresponding VM’s AES key.
Next, what about guest VM memory integrity? Hypervisor can physically access entire SDRAM physical address space assigned for each VM instance, which means it can freely tamper, corrupt or replay memory cipher-text content. SEV’s answer is A global memory mapping table — RMP(Reverse Mapping Table) to track each assigned memory pages’ owner UID then MMU table walking will be extended to enforce access control on any memory access transactions.
This global RMP memory mapping table is maintained to identify memory page level’s owner. MMU page table walking is extended to check this table to make sure only private pages’ owner(guest VM itself) can be authorized to access. Specifically, each private guest VM page on RMP(code or data page entries) is tagged with its owner ASID and RMP tracks the owner of each PTE(page table entry). The RMP tables contains one entry for every 4k pages of DRAM requested by hypervisor for launching guest VM.
RMP tables along with MMU page tables to enforce memory restrictions and page access rights and indexed by bus physical addresses. RMP table is checked at the end of MMU or IOMMU table walks and the access will be denied if not the owner tries to access the guest private pages with raised table-walk faults.
RMP memory table itself isn’t writeable by hypervisor and privileged instructions are used to enable Hypervisor to assign pages to specific guests or dynamic reclaims pages, which also prevents hypervisor from tampering RMP entries.
Hypervisor, secure processor and RMP table works together with hardware AES engine and MMU/IOMMU based access control, which establishes hardware enforced confidential computing infrastructure and makes it possible for guest owners to deploy their computing payload to cloud based multi-tenant virtual machine instance.
Next question is how to establish trustworthiness between guest VM owner and cloud based confidential computing platform/VM instance before guest owner can really deploy its confidential payload and necessary credentials as illustrated in following diagram? Answer is remote attestation process.
Secure processor maintains TCB(trusted computing base) identity digest(eg. similar to TPM’s extended hash platform register PCR) called the ReportedTCB, which will be used to derive the VCEK(Versioned Chip Endorse Key) for signing remote attestation report.
During remote attestation process, guest VM will be challenged to request attestation reports through trusted channel with secure processor. Secure processor will collect all measures including device and TCB identity to generate remote attestation report. This signed attestation report will be received and verified by relying party(guest owner) against reference values and corresponding supply chain endorser entity eg. authentic chip vendor etc.
Wrap up the discussion of AMD-SEV-SNP confidential computing solution by capturing the summary of its key hierarchy and key agreement protocol between different participants for reference.
Intel Software Guard Extensions (SGX) provides user space process based confidential computing framework, which is different from VM based AMD-SEV solution. It uses a set of dedicated secure gateway instructions to allow user process code to define private regions of memory for SGX secure enclave isolating from other non-trusted process, enclave memory contents are encrypted and unreadable by any other processes either privilege or not.
Hardware blocks and processor features enabling SGX:
- Processor secure gateway instructions and special processor operating mode: enclave mode. For those who are familiar with ARM cortex-m33 security extension’s non-secure callable and secure gateway ABI, you might find the similarity here on how SGX transitions the processor between unsafe operating mode(NSPE in cortex-m33 term) and enclave processing mode(SPE in cortex-m33 term). Within enclave(SPE) mode, processor provides logical trusted execution environment enforced by hardware firewall such as MMU/MPU and inline memory encryption engine.
- MEE(inline Memory Encryption Engine) and MMU, memory controller
- EPC(Enclave Pages Cache) serves enclave pages requesting when application wants to create SGX enclave. EPC maintains per-enclave private address space’s page tables, access policy, memory integrity hash used for security check. SECS(SGX enclave control structure) storing real-time enclave memory content hash and TCS(Thread Control Structure) per enclave thread containing enclave pre-defined execution entry points. EPC resides inside PRM(processor reserved memory) carved out from system DRAM(can be configured via platform BIOS) as illustrated in the diagram below. EPC content is encrypted via MEE encryption engine. EPC is only accessible to processor enclave mode and immutable outside of it.
- EPCM(EPC Map) maintains a root-level look-up table against all secure enclave instances’ EPC pages inside processor trusted boundary, only visible and accessible to processor enclave mode.
How these hardware features are working together to protect data-in-use and memory integrity during launching and run-time? Please refer to the following diagram for launching interaction flow.
- As per 1A/1B steps in the diagram when a new enclave is launched, user-space application process first uses OS EPC/EPCM interfaces to request for enclave pages. SGX service in CPU enclave mode will assign DRAM pages and map these pages into enclave private address spaces, populate enclave page tables within EPC. In addition, EPCM enclave pages look-up table will also be updated to track each enclave instance’s page tables. Note: EPC per-enclave page tables hosted inside PRM memory region are MEE encrypted.
- As per 2A/2B steps once SGX successfully assigned and returned enclave pages then user space process issues processor secure gateway ABIs — ECREATE to spin off secure enclave whose ephemeral per-enclave AES-CTR-128 key is injected into MEE).
- 3A/3B steps — invokes EADD(*src, *dst) secure gateway ABI to copy encrypted enclave code/data into assigned enclave pages.
- 4A/4B steps — invokes EEXTEND(*src) to instruct MEE doing hardware measure on secure enclave pages.
- 5A/5B steps — invokes EINIT to instruct MEE to update memory measuring result(hash) into SECS control structure.
Following diagrams capture at high-level how SGX enclave is launched and attestation flow.
Note that during 3A/3B steps, SGX copy encrypted enclave code/data into assigned enclave pages. Before enclave owner passes decryption key into secure enclave, enclave owner(relying party) performs remote attestation process to capture measures(TCB firmware and SoC cryptographic identity to reflect platform authenticity and security status). Once enclave owner validates measurement report(quote or attestation token) decrypting key can be sent over to the secure enclave.
So far, SGX enclave is launched and measured, trusted as part of user space application process. Enclave pages are measured and encrypted via ephemeral per-enclave key during run-time. Whenever any ECALL gateway ABI is invoked to request for secure enclave run-time services, access control policies(configured by EPCM/EPC enclave page info) will be enforced on each enclave memory pages access(instruction fetch or data load/store) during MMU page table walking.
ARM Confidential computing architecture(CCA) allows you to deploy confidential Virtual Machines (VMs), while preventing access by more privileged software entities. ARM CCA allows the hypervisor to manage/control the VM’s resources, but removes its right for access to the code, register state, or data that is used by that VM.
As illustrated in following diagram, ARM CCA is derived and evolved from ARMv8A architecture. New processor state(Realm state) is added to host confidential VM instances.
Arm CCA allows Realms to be created, scheduled and destroyed on demand under the control of the Normal world REE hypervisor. Resources can be added or retrieved from Realms dynamically. Its physical execution is sandboxed within the Realm processor state. The execution of the Realm VM is initialized through hypervisor commands that are passed to the Monitor, and then pushed through the Monitor to the RMM.
The Monitor is the gatekeeper between the separate worlds. It controls any passage between Normal world, Secure world, and Realm world to ensure that the isolation between the worlds is maintained whilst allowing communication and control.
The initial state of a Realm VM instance and the platform upon which it executes, can be attested. Attestation allows the Realm owner(guest owner/Relying party) to establish trust in the Realm VM, before provisioning any secrets to it.
Realm world code can request for attestation reports to RMM via RSI (Realm Service Interface) at any time. The reports can then be used to authenticate the authenticity of the platform and device/TCB identity in the realm.
Memory isolation between different domain address spaces is managed through a new Granule Protection Table (GPT), an extension of MMU page tables that is controlled exclusively by ROOT Monitor in EL3. GPT’s L0 table is cached in OCRAM securely and L1 tables are typically placed in DDR SDRAM in a memory region with GPI_ROOT accessible only. GPT tables define the ranges of physical memory that each security state can access. Memory regions on GPT are tagged with one of six access properties(realm, NS, S, Root monitor and all, none). Any memory regions transferring request need to go through SMC calls and be serviced by TFA BL31 runtime firmware.
Invalid accesses raise page faults(GPF). Granule Protection Check (GPC) is enforced the translation process for stage 1 and 2 translations and checks all physical address access against the GPT entry to allow memory access or create a fault.