Secure ESP32 Device Provisioning with Hardware Security

Alejandro Corredor
Life and Tech @ LifeOmic
13 min readJun 22, 2023
Midjourney Generated Microchip

DISCLAIMER: The examples shown in this article deal with sensitive data like claim certificates, private keys, and encryption keys. None of this data should be exposed. DO NOT commit any file containing sensitive information to version control (e.g. via .gitignore for git). If this type of file is committed, a bad actor could obtain a claim certificate and private key for a fleet of devices. This would allow the bad actor to potentially deploy an infinite number of unauthorized machines.

Intro

In one of our previous articles, we explained how to provision a medical device for use with the LifeOmic Platform. The process consists of obtaining a claim certificate and a private key for a fleet of devices, getting those into the device, and exchanging them for a device certificate on the first boot-up. However, there is one important aspect that we left out of the initial article — securely storing the claim certificates and private keys in the device. The secure storage process is complicated, so we left it out of the first article for simplicity’s sake. This article addresses securely storing the claim certificates and private key in the device.

In the first article, we stored the claim certificate and private key in plain text inside the application code. Plain text is a clear security vulnerability. An attacker with access to the device could read the claim certificate and private key from the binary. That means that the attacker could provision an infinite number of unauthorized devices. We don’t want that.

We still need to get that claim certificate and private key into the device during the manufacturing process. So, how do we securely do that? Enter Flash Encryption. Devices with the ESP32 architecture support Flash Encryption through the use of the ESP-IDF framework. In basic terms, Flash Encryption works by doing a one-time encryption activation on the device through a series of commands. After that, the contents of the flash memory are fully encrypted. But, for our provisioning use case, the story does not end there. The claim certificate and private key will not be stored in flash memory, since they would be erased as soon as the device is turned off. We’ll be storing them in something durable: Non-Volatile Storage (NVS). NVS is the type of storage that does not get wiped when the device is turned off — Exactly what we need.

But wait, NVS is not encrypted, so that would defeat the whole purpose of this story. Well, the truth is that Flash Encryption is just the first step towards NVS Encryption. We actually encrypt the claim certificate and private key with another set of encryption keys. After that happens, we’ll store the encryption keys in a new partition on the device. That partition is indeed protected by Flash Encryption and is what makes everything work together.

Overview of the NVS encryption flow

Our first step was to load our initial encrypted data (the claim certificate and private key) and their corresponding encryption keys into the device. NVS Encryption is then activated. Any subsequent read/writes made to or from the device are fully protected. For those interested in the fine-grained details, check out the Flash Encryption documentation. We’ve also published the final repository of the example to GitHub; find it here.

Here are the steps to securely provision our claim certificate and private key into the device hardware:

  1. Set up a custom partition table for our device with a CSV file. It will include a partition for NVS and a partition for securely storing NVS Encryption keys (more on this in a bit).
  2. Specify how we’ll load our claim certificate and private key binaries into NVS with a CSV file. The file will specify where our data is and in what NVS namespace it will live.
  3. Generate encrypted binaries of our claim certificate and private key. This step will also output the binary of the keys used to encrypt the data.
  4. Flash the encrypted binaries of the claim certificate and the private key into the NVS partition.
  5. Flash the binary of the keys used to encrypt the data from step 4 into the NVS keys partition. The NVS keys partition is encrypted through Flash Encryption.
  6. Flash the device for the first time. This enables Flash Encryption on the initial boot.
  7. Confirm that everything worked by inspecting the output of the application code. The code should print the decrypted claim certificate and private key.

We now have a general understanding of the encryption process. Let’s begin the step-by-step guide.

Prerequisites

Before we get into the actual setup, we need to install a few things on our local machine:

  1. Python 3
  2. The ESP-IDF Framework. To follow along, we recommend you install and use it via their VSCode plugin: https://github.com/espressif/vscode-esp-idf-extension/blob/master/docs/tutorial/install.md
  3. After installing ESP-IDF, set the $IDF_PATH in your bash/zsh profile. When installing ESP-IDF with the VSCode plugin, the path should end up looking something like this:
# Inside your bash/zsh profile file.

export IDF_PATH=$HOME/esp/esp-idf

Securely Storing the Claim Certificate and Private Key

After having access to a claim certificate and a private key, the next step is to locate where they were saved. For this article, we store them in a certs folder under the main project code. Keep in mind that we do not commit that folder to version control. We now have to configure the partition table for our device. Importantly, it specifies our NVS partition and a partition for securely storing our NVS encryption keys. The custom partition table is specified by creating a CSV file:

custom_partitions.csv

# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, , 0x6000,
storage, data, 0xff, , 0x1000,
factory, app, factory, , 1M,
nvs_key, data, nvs_keys, , 0x1000, encrypted,

The two important rows here are the first and last one. The first one specifies our NVS partition with its given size. The last one specifies an extra partition for storing our NVS encryption keys. Note that it’s marked with an encrypted flag. This is what tells ESP-IDF to protect the encryption keys using Flash Protection. It’s also worth noting that we do not specify offsets. This allows the offsets to be automatically calculated during the build process.

We can now specify the content that our NVS partition initially loads (the claim certificate and private key). This is also done by leveraging a CSV file:

main/nvs.csv

key,type,encoding,value
certs,namespace,,
claim_cert,file,string,main/certs/claim-cert.pem
private_key,file,string,main/certs/private.key

We first declare a new namespace called certs. Namespaces are a concept in NVS for dividing stored data. This single CSV file essentially defines a key/value store. The first row specifies the namespace, and the remaining rows specify the values. Our last two rows specify where our claim certificate and private key are located, as well as the name of the key we’ll use to access them.

We’re now ready to generate the encrypted data that will be flashed into the NVS storage. We can do this with the NVS partition generator tool. The tool is included with the ESP-IDF installation:

$IDF_PATH/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py \
encrypt main/nvs.csv encrypted_nvs.bin 0x6000 --keygen --keyfile nvs_keys.bin

The command takes an input file, an output file, the size of the NVS partition, and a set of options. The NVS partition size corresponds to the size specified in the partition table. The options specify the encryption of the data and the output of the encryption keys binary.

After running the above command, our ESP-IDF project tree should look like this:

├── build
│ └── (...)
├── keys
│ ├── nvs_keys.bin <---- DO NOT COMMIT TO VERSION CONTROL
├── components
│ └── (...) <---- No components for our example
├── main
│ ├── CMakeLists.txt
│ ├── certs <---- Contains claim certificate and private key
│ │ ├── claim-cert.pem DO NOT COMMIT TO VERSION CONTROL
│ │ ├── private.key
│ ├── main.c
│ └── nvs.csv <---- CSV with NVS key/value data
├── CMakeLists.txt
├── encrypted_nvs.bin <---- Encrypted claim cert + private key to be
│ Flashed to NVS storage
├── custom_partitions.csv <---- CSV with device partition table
└── sdkconfig.defaults <---- More on this in a bit

For the final step of this section, we’ll flash the two generated binaries into the device. The encrypted NVS data is flashed to the nvs partition, and the NVS encryption keys are flashed to the nvs_key partition. Before we can do that, we need to locate the offset of each partition.

First, we make sure that our sdkconfig.defaults file has the correct configuration:

# Tells the build we have a custom partition table
CONFIG_PARTITION_TABLE_CUSTOM=y

# Tells the build where our custom partition table is defined
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="custom_partitions.csv"
CONFIG_PARTITION_TABLE_FILENAME="custom_partitions.csv"

# Specifies where the partition table is stored. The default location
# is 0x8000, but when Flash Encryption is enabled, the size of the
# Bootloader increases, which requires us to move this to a higher offset.
# The size of the binary also has an impact on the end value of this
# location, so it may change depending on the size of the application.
CONFIG_PARTITION_TABLE_OFFSET=0x10000

# Tells the build that the device will use Flash Encrytpion and
# NVS Encryption.
# These values are the key to making things work.
CONFIG_NVS_ENCRYPTION=y
CONFIG_SECURE_FLASH_ENC_ENABLED=y
# This last option would only be enabled while developing. For
# a final production release, it would be off. See the
# reference links for more info about what the option does.
CONFIG_SECURE_FLASH_ENCRYPTION_MODE_DEVELOPMENT=y

Then, we build our project. Open it with VSCode and run the command palette with ESP-IDF: Build your project. This generates a binary file for the partition table under the build folder. We can now inspect that partition by running the following command:

$IDF_PATH/components/partition_table/gen_esp32part.py \ 
build/partition_table/partition-table.bin

The output should look like this:

Parsing binary partition input...
Verifying table...
# ESP-IDF Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x11000,24K,
storage,data,255,0x17000,4K,
factory,app,factory,0x20000,1M,
nvs_key,data,nvs_keys,0x120000,4K,encrypted

We’ll remember the Offset for the nvs and nvs_key.

We can now flash the encrypted binary of the claim certificate and the private key to the NVS partition:

python $IDF_PATH/components/esptool_py/esptool/esptool.py \
-p $PORT \
--before default_reset \
--after no_reset \
write_flash $OFFSET encrypted_nvs.bin

$OFFSET corresponds to the value obtained from the previous step for the nvs row. $PORT corresponds to the port that connects to the device.

We do the same for the NVS encryption keys, but this time passing the --encrypt option. If we don’t include it, we’ll encounter errors when flashing the device and encryption won’t end up working.

python $IDF_PATH/components/esptool_py/esptool/esptool.py \
-p $PORT \
--before default_reset \
--after no_reset \
write_flash --encrypt $OFFSET keys/nvs_keys.bin

$OFFSET corresponds to the value obtained from the previous step for the nvs_key row. Again, do not forget the --encrypt option when flashing the encryption keys.

Enabling Flash Encryption

At this point, the data required for securely provisioning the device is loaded. The only thing left to do is to flash the device for the first time. This step will actually enable Flash Encryption. In simple terms, the device flashes, the encryption fuse turns on, and the device resets. All flash memory is subsequently encrypted.

To do this we first open an ESP-IDF terminal within VSCode with ESP-IDF: Open ESP-IDF and then run the following command inside it:

idf.py -p $PORT flash monitor

At this point everything should be good to go. The application code within main/main.c should have printed out the chip information and chip encryption status. The chip encryption status should be ON. The code should also have printed the decrypted claim certificate and the private key. That proves that NVS encryption is working correctly. Let’s take a look at what we should see in the console:

Loaded app from partition at offset ${OFFSET}
Checking flash encryption...
Generating new flash encryption key...
flash_encrypt: Read & write protecting new key...
flash_encrypt: Setting CRYPT_CONFIG efuse to 0xF
flash_encrypt: Not disabling UART bootloader encryption
flash_encrypt: Disable UART bootloader decryption...
flash_encrypt: Disable UART bootloader MMU cache...
flash_encrypt: Disable JTAG...
flash_encrypt: Disable ROM BASIC interpreter fallback...
flash_encrypt: Flash encryption completed
boot: Resetting with flash encryption enabled...
Checking Flash Encryption Status
FLASH_CRYPT_CNT eFuse value is 1
Flash encryption feature is enabled in DEVELOPMENT mode
storage-encryption: Loading private key & certificate
storage-encryption: Loaded private key & certificate successfully!
Private Key:
------BEGIN RSA PRIVATE KEY------
...
...
...
------END RSA PRIVATE KEY------

Claim Certificate:
------BEGIN CERTIFICATE------
...
...
...
------END CERTIFICATE------

Next, let’s look at the application code more closely.

Application Code

For our example, there’s not much to it. It’s a single C file. We first have a couple of helper functions. These just print out some information about the chip and its encryption status.

The important part is where we securely initialize NVS to access the encrypted claim certificate and private key. nvs_secure_initialize finds the nvs_key partition, reads the encrypted encryption keys (yeap, encryptception) and initializes NVS. It does this by passing them along when calling nvs_flash_secure_init.

The code then just accesses the certs namespace by using the native NVS library. It retrieves the claim certificate and private key by using the nvs_load_value_if_exist helper.

Let’s look at the code:

// Inside main/main.c

#include <stdio.h>
#include <inttypes.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "soc/efuse_reg.h"
#include "esp_efuse.h"
#include "esp_chip_info.h"
#include "esp_flash.h"
#include "esp_partition.h"
#include "esp_flash_encrypt.h"
#include "esp_efuse_table.h"
#include "nvs.h"
#include "nvs_flash.h"

static void print_chip_info(void);
static void print_flash_encryption_status(void);

static const char *TAG = "storage-encryption";

#if CONFIG_IDF_TARGET_ESP32
#define TARGET_CRYPT_CNT_EFUSE ESP_EFUSE_FLASH_CRYPT_CNT
#define TARGET_CRYPT_CNT_WIDTH 7
#else
#define TARGET_CRYPT_CNT_EFUSE ESP_EFUSE_SPI_BOOT_CRYPT_CNT
#define TARGET_CRYPT_CNT_WIDTH 3
#endif

static void print_chip_info(void)
{
esp_chip_info_t chip_info;
uint32_t flash_size;
esp_chip_info(&chip_info);

printf("This is %s chip with %d CPU core(s), WiFi%s%s, ",
CONFIG_IDF_TARGET,
chip_info.cores,
(chip_info.features & CHIP_FEATURE_BT) ? "/BT" : "",
(chip_info.features & CHIP_FEATURE_BLE) ? "/BLE" : "");

unsigned major_rev = chip_info.revision / 100;
unsigned minor_rev = chip_info.revision % 100;

printf("silicon revision v%d.%d, ", major_rev, minor_rev);

if (esp_flash_get_size(NULL, &flash_size) != ESP_OK)
{
printf("Get flash size failed");
return;
}

printf("%" PRIu32 "MB %s flash\n", flash_size / (1024 * 1024),
(chip_info.features & CHIP_FEATURE_EMB_FLASH) ? "embedded" : "external");
}

static void print_flash_encryption_status(void)
{
printf("\nChecking Flash Encryption Status\n");

uint32_t flash_crypt_cnt = 0;
esp_efuse_read_field_blob(TARGET_CRYPT_CNT_EFUSE, &flash_crypt_cnt, TARGET_CRYPT_CNT_WIDTH);
printf("FLASH_CRYPT_CNT eFuse value is %" PRIu32 "\n", flash_crypt_cnt);

esp_flash_enc_mode_t mode = esp_get_flash_encryption_mode();
if (mode == ESP_FLASH_ENC_MODE_DISABLED)
{
printf("Flash encryption feature is disabled\n");
}
else
{
printf("Flash encryption feature is enabled in %s mode\n",
mode == ESP_FLASH_ENC_MODE_DEVELOPMENT ? "DEVELOPMENT" : "RELEASE");
}
}

/**
* Helper function that loads a value from NVS.
* It returns NULL when the value doesn't exist.
*/
char *nvs_load_value_if_exist(nvs_handle handle, const char *key)
{
// Try to get the size of the item
size_t value_size;
if (nvs_get_str(handle, key, NULL, &value_size) != ESP_OK)
{
ESP_LOGE(TAG, "Failed to get size of key: %s", key);
return NULL;
}

char *value = malloc(value_size);
if (nvs_get_str(handle, key, value, &value_size) != ESP_OK)
{
ESP_LOGE(TAG, "Failed to load key: %s", key);
return NULL;
}

return value;
}

esp_err_t nvs_secure_initialize()
{
static const char *NVS_TAG = "nvs";
esp_err_t err = ESP_OK;

// 1. Find partition with NVS encryption keys
const esp_partition_t *partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA,
ESP_PARTITION_SUBTYPE_DATA_NVS_KEYS,
"nvs_key");
if (partition == NULL)
{
ESP_LOGE(NVS_TAG, "Could not locate nvs_key partition. Aborting.");
return ESP_FAIL;
}

// 2. Read NVS encryptions from key partition
nvs_sec_cfg_t cfg;
if (ESP_OK != (err = nvs_flash_read_security_cfg(partition, &cfg)))
{
ESP_LOGE(NVS_TAG, "Failed to read nvs keys (rc=0x%x)", err);
return err;
}

// 3. Securely initialize NVS partition
if (ESP_OK != (err = nvs_flash_secure_init(&cfg)))
{
ESP_LOGE(NVS_TAG, "failed to initialize nvs partition (err=0x%x). Aborting.", err);
return err;
};

return err;
}

void app_main(void)
{
print_chip_info();
print_flash_encryption_status();

esp_err_t err = nvs_secure_initialize();
if (err != ESP_OK)
{
ESP_LOGE("main", "Failed to initialize nvs (rc=0x%x). Halting.", err);
while (1)
{
vTaskDelay(100);
}
}

// Open the "certs" namespace in read-only mode
nvs_handle handle;
ESP_ERROR_CHECK(nvs_open("certs", NVS_READONLY, &handle) != ESP_OK);

// Load the private key & certificate
ESP_LOGI(TAG, "Loading private key & certificate");
char *private_key = nvs_load_value_if_exist(handle, "private_key");
char *certificate = nvs_load_value_if_exist(handle, "claim_cert");

// We're done with NVS
nvs_close(handle);

// Check if both items have been correctly retrieved
if (private_key == NULL || certificate == NULL)
{
ESP_LOGE(TAG, "Private key or cert could not be loaded");
return; // You might want to handle this in a better way
}
else
{
// We're printing the keys to the console for demonstration
// purposes. This shows that NVS Encryption is actually working.
// Don't do this in production code, obviously!
ESP_LOGI(TAG, "Loaded private key & certificate successfully!");
printf("Private Key:\n %s\n", private_key);
printf("Claim Certificate:\n %s\n", certificate);
}

// At this point the private_key and claim_cert have been loaded.
// Use them to exchange them for a claim certificate and proceed to
// communicate with the LifeOmic Platform via MQTT.

// Once we've exchanged the claim certificate for a device certificate,
// we would also proceed to securely storing the device certificate to
// NVS storage. Since NVS Encryption is already enabled at this point,
// we do not have to worry about exposing it to bad actors.
}

At this point, we can do anything we want with our retrieved data. For our example, we want to be able to authenticate against the LifeOmic Platform. To do that we would follow the same steps as in our previous article. We would use the claim certificate and private key to start the MQTT exchange process for obtaining a device certificate. Once we obtain the certificate, we can securely store it in NVS, since NVS Encryption is already enabled.

Voila!

Closing Thoughts

We have seen how to securely provision an ESP32 chip with any preloaded data that we wish to include. Provisioning occurs before we ship off our device. By enabling Flash Encryption along with NVS Encryption, we also guarantee that our runtime code can securely store future data. This improves our security posture when provisioning a fleet of devices.

This is only one way of achieving secure device provisioning. Depending on the specific architecture of the chip, the process may vary. This article is based on the ESP32 architecture, which is the architecture we are using throughout our device series. Some ESP32 chips even include a separate secure hardware element, which can provide similar security capabilities. It is important to keep in mind the type of data being dealt with. You need to come up with a battle-tested strategy for securing sensitive information before letting your devices out into the wild.

Troubleshooting

When flashing the device for the first time with encryption enabled, certain eFuses are burned. This changes the eFuse values from 0 to 1 and turns on encryption. If things go wrong and encryption is not enabled, it’s possible to check the eFuse values. Run the following command:

python $IDF_PATH/components/esptool_py/esptool/espefuse.py 
--port $PORT summary

See the links in the references for more details on how the flash encryption process works.

References

--

--