PETs in Advertising: Scenarios for Trusted Execution Environments

Criteo Engineering

Published in

Criteo R&D Blog

15 min readMay 25, 2023

Article by Fabian Höring, Julien NIGRI and Maxime Vono.

Source: https://unsplash.com/photos/ydArxOA1uSM

Introduction

In our first article, we introduced and compared in a high-level manner two privacy-enhancing technologies (PETs) that are currently being investigated by major browser vendors [1,2] (e.g. Google Chrome & Mozilla Firefox) to address advertising use cases while meeting privacy guarantees.

Our second article focused on detailing how some key advertising use cases such as private attribution, reporting, and campaign optimisation could be handled with secure multi-party computation (MPC), notably leveraged in Interoperable Private Attribution (IPA) [1] proposed by Meta & Mozilla.

In this article, we will investigate how and at what cost those aforementioned use cases could be addressed using trusted execution environments (TEEs), a parallel path that is also investigated by Google Chrome in their Privacy Sandbox initiative. More precisely, we will present the current architecture of the aggregated version of the Attribution Reporting API (ARA), which leverages TEEs to build so-called trusted helper servers, deep dive into the main TEE mechanisms ensuring privacy and correctness (see our first article for definitions of these terms) such as remote attestation, and highlight some limitations associated with the use of TEEs. Finally, we will provide scalability and cost analyses of private reporting and campaign optimisation using TEEs, illustrated via experiments we have done to properly assess the potential of those technologies.

TL;DR

Regarding computing and networking costs, performing reporting and campaign optimisation using TEEs seems more efficient than performing those tasks via MPC. As an example, the cost of performing reporting using a TEE compared to the campaign cost is roughly 0.004%; in comparison with 0.2% using MPC-based IPA and assuming a 1:1 ratio between display and conversion events. In a more realistic scenario, the ARA cost would be 100 times lower as attribution is performed browser-side and not server-side.
Hardware vendor-based TEE approaches (such as Intel SGX or AMD SEV-SNP) seem more promising than cloud provider-based TEE solutions (such as AWS Nitro Enclaves) because they would allow for flexibility between cloud and on-premise deployment.
VM-based TEEs seem more promising in terms of time to market as they can reuse existing (open-source) libraries. Process-based TEEs provide stronger security guarantees as they have a reduced trusted computing base (TCB).

TEE-based Privacy Sandbox proposals

Currently, trusted execution environments (TEEs) would be used in the following Privacy Sandbox proposals:

Aggregated Attribution Reporting API (ARA) and Private Aggregation API [2,3], which enable the generation of aggregated and noisy reports;
Bidding and Auctions Services API [4] which aims to implement a secure bidding server;
Protected Audience API’s key/value server, a way to fetch real-time bidding signals [5].

ARA, similar to Interoperable Private Attribution (IPA) [1], solves the problem of measuring the performance of ad campaigns, while the Bidding and Auction Services API solves the problem of real-time bidding (RTB) where ads are bought and sold in real-time in an auction.

Both proposals would leverage TEEs to implement trusted helper servers. The latter aims at processing data, such as users’ browsing history, without giving the adtech company or other actors the possibility to identify individual users across sites. Since such processing is outsourced to helper servers, enhancing the latter with TEE capabilities such as remote attestation, allows to audit the code running on these helper servers in order to obtain sufficient trust.

In this blog post series, we mainly want to assess the advantages and limitations of PETs for reporting, attribution, and campaign optimisation use cases. As such, we will focus here on ARA and leave the analysis of the Bidding and Auctions Services API for a future article. Note that ARA is, as of today, the most advanced TEE-based Privacy Sandbox proposal.

Aggregated Attribution Reporting API (ARA)

Before we deep dive into the technical details of how TEE could be used to perform private reporting and campaign optimisation, let’s start with a high-level picture of ARA.

Figure 1 below depicts a high-level overview of the architecture of the aforementioned API [3]. In contrast to IPA detailed in our previous article, attribution will be done by the browser and the TEE will only be leveraged to aggregate individual reports sent by the browser.

Figure 1 — Aggregated Attribution Reporting API (ARA) report workflow. Source: https://github.com/WICG/attribution-reporting-api/blob/main/AGGREGATION_SERVICE_TEE.md

The proposed workflow goes as follows. Each browser instance directly sends encrypted attribution reports to an adtech company willing to measure and optimise advertising campaigns. These encrypted reports, once collected, are then sent to the adtech-operated aggregation service running in a TEE. The TEE decrypts the reports, aggregates them, and adds noise to meet differential privacy requirements. Finally, the adtech company can query the noisy aggregated reports via an API.

Please note, that the current proposal of the API requires that the TEE will run in a cloud instance. However, some adtech companies might also want to run such TEE-based aggregation services on their own infrastructure for operational, economic, and strategic reasons.

One important actor in this workflow is the coordinator (see the right part of Figure 1) who owns several responsibilities:

it ensures that the adtech company runs a genuine (i.e. non-tampered) instance of a TEE; this process is also called remote attestation (see our first article);
it distributes cryptographic keys to allow encrypted communication between the client and the TEE and as such operates the key management system;
it keeps track of which reports have already been queried to prevent processing replay and reveal sensitive information.

In the current API proposal, Google owns the coordinator, but in the long term, it is preferable that an independent party takes this role [3, 25].

Campaign optimisation via Machine Learning

The use case of training a Machine Learning (ML) model to optimise campaigns is generally mentioned in public discussions [6–8] but no formally agreed proposal has been established yet. The current envisioned way of performing ML training would be to call the Aggregated Attribution APIs to collect noisy and aggregated reports and then train a ML model on such aggregated data. Unfortunately, such a task is very challenging as depicted by an ML competition we organised on this topic, see our Medium article summarising our findings.

Leveraging a TEE opens the possibility to train in a flexible and secure manner a ML model directly on event-level data inside a TEE owned by the adtech company. This solution is for instance proposed by Microsoft through MaskedLARK.

The workflow would be similar to what has been described above regarding ARA, namely:

a browser would send encrypted reports to the adtech company;
the adtech company would forward those reports to the TEE which could decrypt them;
instead of aggregating noisy event-level data, the TEE would learn a ML model on such event-level data and output the model weights;
the model would be differentially private and could be directly used for ML inference in the bid stream.

A deep-dive into important TEE mechanisms

This section provides details on the TEE main mechanisms that ensure privacy and correctness (see our first article for definitions of these terms). The final paragraph highlights some security risks that might happen by exploiting side-channel attacks.

Remote attestation

Attestation plays a major role in the security guarantees of TEEs. Notably, it ensures that the application is running inside a genuine TEE by proving its identity to an external trusted party, the coordinator (see Figure 1).

To detail the remote attestation process, we chose to describe the process associated with virtual machine (VM)-based TEEs since such TEEs are simpler and more flexible than process-based TEEs. Nevertheless, the workflow described below also applies to process-based TEEs, such as Intel SGX.

Figure 2 describes VM-based attestation whereby the hypervisor (e.g. the cloud platform or an on-premise data-center) launches an application inside a VM and the root of trust relies on hardware built-in cryptographic keys signed by the hardware vendor, referred to as certificate authority (CA).

The hypervisor launches a VM with the application (1). The TEE firmware then collects measurements, signs them (2), and provides an attestation report to the coordinator (3). The TEE firmware is a tamper-resistant piece of cryptographic hardware built onto the CPU that implements primitive cryptographic functions and includes built-in private/public keys that the authentication process relies on. The emitted report contains several measurements about the running platform (chip version, firmware & microcode), the application owner (the adtech company in our case), and a hash of the running code base (OS + application code). The coordinator then checks the reports signature (4), checks that the platform is not vulnerable, and compares the hash of the code base to what has been allowed for the associated owner (via for instance an open-source repository) (5). If all the previous checks are positive, the coordinator trusts the application and shares the secrets necessary for the application to start its workload (6).

More details can be found in this presentation describing AMD SEV-SNP, a specific type of VM-based TEE.

The attestation process depicted in Figure 2 is currently implemented via Azure attestation [9–10] and Amazon Web Services (AWS) (support has been recently added for AMD processors) [11]. One important part of this process is that the hypervisor (cloud vendor or data center) does not necessarily need to be trusted which further reduces the attack surface of those solutions.

On the other hand, we would like to highlight that the current Privacy Sandbox proposals currently only leverage solutions from AWS and Google Cloud Platform (GCP) with dedicated TEE firmwares [12–13]. As such, trust is partly delegated to these cloud providers.

Client communication

Once attestation is complete, the coordinator knows the adtech company’s application is genuine and can securely start. From that stage, the TEE can start accepting workloads from the client.

The general way to communicate with the TEE is by initiating a secure connection based on hybrid public key encryption (HPKE). The latter works in 2 steps:

doing a handshake with asymmetric ciphers to authenticate and exchange a private key;
using the symmetric private key for encrypted communication.

Figure 3 below describes the workflow to start initiating a secure communication.

Figure 3 — Secure communication between a client (browser or TEE) and the TEE.

As described previously in Figure 2, the hypervisor launches the application (1). The coordinator performs the attestation procedure (2) and if successful, creates a new private/public key pair for the exchanging the symmetric private key. The private key is sent to the application (encrypted by TEE firmware public key in order to securely provision the TEE). The public key is given to the client (the browser in ARA) on demand or is signed and then also sent to the client. The client can then use the public key to encrypt a shared secret that can only be decrypted from inside the TEE. This shared secret will be used for communication between the browser and TEE. Regular key rotation can ensure that historical communications cannot be decrypted in case the private key is revealed.

TLS also allows for both parties to authenticate to each other via exchanging public keys (for example via Diffie Hellman key exchange). This can be used for TEE-to-TEE communication.

Code update frequency

As pointed out previously, one important part of the attestation process is to validate that each adtech company runs some genuine container image compliant with privacy and security requirements (see steps 4 & 5 in Figure 2). Therefore, all container images must be shared publicly and every single code update in the adtech company’s container image would require a vetting process in order to prevent the code from being tampered with. Unfortunately, this procedure implies an additional bottleneck: the code updates would need to be released to a public code repository, and then be reviewed and validated manually by the coordinator organisation. For instance, at Criteo, we have multiple releases of our production containers every day. As such, initiating a new image review process for each of our updates does not seem practical or scalable.

To cope with this issue, one solution is to reduce the code running inside the TEE to a strict minimum that still allows it to meet privacy guarantees.

Attribution & reporting. In the current version of ARA, attribution and reporting use cases likely don’t need very frequent code update reviews while allowing the adtech company to ask for customised aggregated reports. Some details are provided hereafter.

The technical requirements of the attribution service API seem to make it possible to converge to a stable image that never changes. Simplified to the maximum, the necessary code to be used by the aggregation service is somehow similar to an SQL GROUP_BY operation. This operation would be similar for all adtech companies, as the attribution is already done on the browser and integrated into the encrypted report.

Campaign optimisation. Regarding campaign optimisation via ML training, there is unfortunately a trade-off between the scalability of the attestation process and the flexibility given to the adtech company to train a specific ML model inside a TEE. Indeed, there are many different ML algorithms to optimise advertising campaigns such as logistic regression, deep learning models, or gradient-boosted decision trees, and this area actually stands as a source of innovation for adtech companies. Auditing each specific adtech company’s code would be challenging for the coordinator. One solution could be to provide some ML training building blocks in a stable image such as feature transformation functions or some neural network layers. Customising these building blocks could be performed via user-defined functions (UDFs) in a sandboxed environment allowing each adtech company to still experiment with different approaches [14]. Note that UDFs are not covered by the TEE attestation. It would be up to the sandbox to check that UDFs indeed fulfill the prescribed privacy guarantees. Potentially, current ML frameworks like Tensorflow and Pytorch could be used to speed up the TEE development as those frameworks already provide the separation of the execution graph and high-level libraries (e.g. Keras) to define the models.

There are also other approaches that do not require ML training to run in the TEE at all and would just rely on event-level signals or aggregated counters [15–17].

Table 1 below summarises our statements regarding the scalability of the attestation procedure and the flexibility given to the adtech company for each considered use case.

*Table 1. Scalability & flexibility regarding the TEE-based attestation procedure.*

To summarize, even though the remote attestation process takes some time to execute (up to 100 ms), it will likely not be a problem for reporting because we would be able to run it with a stable container image that rarely changes. On the other hand, the answer is currently not clear for campaign optimisation via ML training.

Side-channel attacks

We end this section by pointing out major security risks associated with the use of TEEs, namely side-channel attacks. A side-channel attack exploits a leakage in the implementation of a piece of code by extracting information from a measured value that is correlated to the computation. There are two types of side-channel attacks:

Logical side-channel attacks, undertaken only using software. For instance, these attacks can exploit timing or memory access patterns.
Physical side-channel attacks, undertaken by requiring physical access to the TEE. For instance, these attacks can exploit power consumption and electromagnetic leakage [26].

Those kinds of attacks can be executed by malicious hypervisors that have access to the system or other actors by exploiting leaks in existing communication channels. Whereas the most basic attacks to extract the cryptographic secrets such as bus sniffing attacks are mitigated now because the TEE firmware is done inside the CPU and not in a separate module [18], many side-channels are regularly discovered [18–19] and patched by the hardware vendors. In most cases, those attacks can be mitigated by firmware updates or BIOS settings (like deactivating hyper threading). However, such updates might take some time to be developed and, in some cases, they might even require newer processor generations [27].

Therefore, side-channel attacks present a real risk to application security. However, the existence of such attacks does not mean we should not use TEEs. Indeed, while the latter is not perfect, they are far better than the current status quo.

In a practical scenario where a vulnerability gets detected, the coordinator could block-list the vulnerable TEE instances within a given notice period until the adtech company deploys a new node with updated firmware. As the TEE runs in a server-side scenario and not on consumer devices, such mitigations seem more practical in most cases in a reasonable timeframe (except for a vulnerability that affects the whole processor generation). Also, regular audits of the hardware should be more easily done than auditing the whole software stack of today’s ad ecosystem.

Scaling our workloads

We finally analyse the practical scalability and cost to operate reporting and campaign optimisation via ML training using a TEE, similar to what we have done in our previous article on secure multi-party computation.

Reporting using TEEs

We will try to compare the cost of reporting with what has been presented in our previous article on secure multi-party computation. Since IPA and the ARA are very different regarding the attribution stage, we will only perform the benchmark for the reporting use case by considering a similar experimental design.

As such, we consider 1,000,000 events to process as in the IPA benchmark with a 1:1 (unrealistic for ARA, see explanation below) ratio between the trigger (e.g. sales) and source (e.g. display) events. The encrypted payload involves two quantities: a bucket (referred to as breakdown key in IPA) encoded using at most 128 bytes, and a value encoded using at most 32 bytes [2]. If we also consider the number of bytes required for symmetric encryption, base64 encoding, and add up the shared_info and JSON payload, 500 bytes per event would be necessary.

Hence, the networking overhead would be 1,000,000 * (500 bytes) = 476 MB, corresponding to a $0.04 cost by assuming a networking cost of $0.08/GB. Table 2 below compares this cost with the one associated with IPA, proposed by Meta/Mozilla.

Table 2. Hypothetical cost of the Aggregated Reporting Attribution API by considering an average CPM of $2 and a networking cost of $0.08/GB. For IPA, these costs have been measured using 1,000,000 events, 16 breakdown keys, and 40-bit match keys under the malicious adversarial model. “Estimate” means that total cost has been estimated using *computing cost* = 0.5 * *networking cost*.

Note that under a more realistic scenario, the aforementioned costs would be much lower since 1,000,000 impressions might not lead to 1,000,000 sales but to 10,000 clicks and 100 sales, as an example. In this scenario, only 4.7MB of networking cost would be required compared to 476 MB by assuming a 1:1 ratio between the display and click events. This overhead reduction by ARA compared to IPA is mainly due to the fact that attribution is performed by the browser.

Campaign optimisation using TEEs

Regarding campaign optimisation using TEEs, we considered training a Machine Learning model with both process-based (Intel SGX) and VM-based (AMD SEV-SNP) enclaves.

Process-based TEE. One specificity with Intel SGX is that the secure code parts should be isolated and use specific APIs like Intel SDK (C, C++) [20]. Intel also provides a library called Privacy Preserving Machine Learning (PPML) [21] being able to run unmodified code with Gramine [22] on SGX.

We ran the PySpark sample [23] running a logistic regression with MLLib. It would be close to what we currently do in production because we have our own Spark Jobs for ML training.

We were able to run our sample successfully inside one TEE, but it was a challenge to get multi-process workloads to work. The documentation also says that up to 1000x overhead might be required which seems unsuitable for our production workloads.

VM-based TEE. Given those constraints VM-based TEEs such as AMD SEV-SNP and Intel TDX could provide a reasonable alternative for the advertising use cases we are trying to address, especially campaign optimisation. Such TEEs require increasing the TCB as a full VM, including the OS, must be started. The advantage is that this allows running unmodified workloads with a small overhead.

Therefore, we also experimented with running a logistic regression model on an AMD SEV-SNP VM on Azure. Our goal was to compare the performances of a similar machine that was either confidential or not. To achieve this, we compared the standard_DC4as_v5 (confidential) with the standard_D4as_v5 (non-confidential) virtual machines. To ensure a fair comparison, we ran the same code on each of these VMs 100 times. Our results indicated that the confidential VM (standard_DC4as_v5) was approximately 14% slower in average compared to the non-confidential VM (standard_D4as_v5). Additionally, the confidential VM was taking around 16% more memory and consumed roughly 6% more CPU.

Table 3 below summarises our findings and compares them with secure multi-party computation. One can see that performing campaign optimisation using a TEE is much more efficient than performing this task using secure multi-party computation.

Table 3. Comparison between standard logistic regression (LR) training, the shallow neural network (NN) model considered in our previous article and trained with secure multi-party computation (MPC), and logistic regression training in a vm-based TEE.

What’s Next

In our next article, we will wrap up all the analyses and conclusions made in this series of articles regarding the advantages, limitations, and costs of MPC and TEEs for advertising use cases.